Home

ConnectX-3 10GbE User Manual

1. Item Value Format Description 28 Length 7 0 LSB 0x4F 29 Length 15 8 MSB 0x00 30 VPD Keyword PN STR Add in Card Part Number 32 Length 0x15 33 Part Number PN STR_SPC 54 VPD Keyword EC STR Engineering Change Level of the card rev 56 Length 0x2 57 Revision RV STR PCB revision 59 VPD Keyword SN STR Serial Number 61 Length 0x18 62 SerialNumber SN STR_SPC 00 00X XXX XX 86 VPD Keyword VO STR Misc Information 88 Length 0x10 89 Data PCIe Gen3 x8 STR SPC 105 VPD Keyword RV STR 107 Length 0x1 108 Data 0 107 96 50 109 Large Resource Type VPD W 0x91 Tag 0x11 110 Length 7 0 LSB Ox8F 111 Length 15 8 MSB OxF 112 VPD Keyword VI STR EFI Driver version 114 Length 0x6 115 Data N A STR SPC 121 VPD Keyword YA STR Asset Tag 123 Length 0x18 124 Data N A STR_SPC N A 148 VPD Keyword RW STR Remaining read write area 150 Length 0x69 151 Data STR_ZERO Reserved 0x00 256 VPD Keyword RW STR Remaining read write area 258 Length Oxfd 259 Data STR_ZERO Reserved 0x00 512 VPD Keyword RW STR Remaining read write area 514 Length Oxfd 515 Data STR_ZERO Reserved 0x00 768 VPD Keyword RW STR Remaining read write area 770 Length Oxfd 771 Data STR_ZERO Reserved 0x00 1024 VPD Keyword RW STR Remaining read write area 1026 Length Oxfd 1027 Data STR_ZERO Reserved 0x00 1280 VPD Keywor
2. 38 A 3 Board Mechanical Drawing and Dimensions 39 Regulatory Statements esis a ea aa 41 Appendix B Interface Connectors Pinout 42 DC compatible Connector Pinout 25241428 nde 224 ode da des hasse eg dd 42 B 2 PCI Express x8 Connector Pinout 42 B 3 SFP Connector en ne en nn eee nee 44 Appendix C Replacing Tall Bracket With a Short Bracket 46 Replacing a Bracket crs oce a A AAA 46 C 2 Removing the Existing Bracket from the Adapter Card 46 C 3 Installinp the New Bracket s 422 47 283 0 224243 AA e 47 Appendix D Avertissements de s curit d installation Warnings in French 49 Appendix E Sicherheitshinweise Warnings in German 51 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 53 4 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 List of Tables Table 1 Revision History Table 7 Table 2 7 Documents Lista teta tete a tt o Rr 8 Table 3 Single and Dual port 10 Gigabit Ethernet Adapter Card 11 Table 4 Physical and Logical Lin
3. 27 3 1 HardwareRequirements 27 3 2 Installation Instructions s possades Sele vos dp ria ads de da ely xu 27 3 3 Identify the Card in Your System 27 3 3 17 On Windows 4 4 2 da wid 27 3 32 On DIBUX iR hr RR xe AA te me iru Kae 28 3 4 Cablesand Modules ata cals bea ote das dada veg en a 29 3 4 1 Inserting the Optical Transceiver Module 29 3 42 Gable Installation 422 4223 552243 2 at naa dai 29 Chapter 4 Driver Software and Firmware 31 Aol NN bee eats e Saute eh Ip dde We D NN 31 414 TUX spared 31 41 2 Windows ia DAR dev mild mb Ms Mh beta 31 4 2 tare e eee AA A AA 33 43 over Converged Ethernet 33 4 3 1 Hardware and Software Requirements 34 4 4 Updating Adapter Card Firmware 34 Appendix A Specifications 0 ccc ccc cece ce ce cece cece ence cere 37 AA MCX311A XCAT Specifications 564 72 Free 37 Mellanox Technologies 3 Rev 1 7 A 2 MCX312 AB XCBT Specifications
4. 17 Figure 9 I2C compatible Connector 18 Figure 10 PCH Devices bea tee ie te gee Rated e a e tes 28 Figure 11 Module With Locking Mechanism Closed 29 Figure 12 Module With Locking Mechanism Open 29 Rig re I3 Device Manager ci CREER en een ann Due Be duet 32 Figure 14 Support Download Assistant 35 Figure 15 Mechanical Drawing of the Dual port MCX311A XCAT Adapter Card 39 Figure 16 Mechanical Drawing of the Dual port MCX312A XCBT Adapter Card 40 Figure 17 Mechanical Drawing of the Dual port MCX312B XCBT 40 Figure 18 Compatible Connector Plug and Pinout 42 Figure 19 PCIe Connector Pinout 43 Figure 20 Rear View of Module With Pin Placement 44 Figure 215 Bracket Screws vga A iD 46 Figure 22 Placing the Bracket on the Card 2 47 Figure 23 Placing the Bracket on the Card MCX312B XCBT 48 Figure 24 Placing the Bracket on the Card MCX311A XCAT 48 6 Mellanox Technologies ConnectX 3 Ethernet Single Dual SFP Port Adapter Card User Manual Rev 1 7 Revision Histo
5. 111 Length 15 8 MSB OxF 112 VPD Keyword VI STR EFI Driver version 114 Length 0x6 115 Data N A STR_SPC 121 VPD Keyword YA STR Asset Tag 123 Length 0x18 124 Data N A STR_SPC N A 148 VPD Keyword RW STR Remaining read write area 150 Length 0x69 151 Data STR_ZERO Reserved 0x00 256 VPD Keyword RW STR Remaining read write area 258 Length Oxfd 259 Data STR_ZERO Reserved 0x00 512 VPD Keyword RW STR Remaining read write area 24 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Table 8 MCX311A XCAT VPD Layout OR Item Value Format Description 514 Length Oxfd 515 Data STR ZERO Reserved 0x00 768 VPD Keyword RW STR Remaining read write area 770 Length Oxfd 771 Data 5 ZERO Reserved 0x00 1024 VPD Keyword RW STR Remaining read write area 1026 Length Oxfd 1027 Data STR_ZERO Reserved 0x00 1280 VPD Keyword RW STR Remaining read write area 1282 Length Oxfd 1283 Data STR_ZERO Reserved 0x00 1536 VPD Keyword RW STR Remaining read write area 1538 Length Oxfd 1539 Data STR_ZERO Reserved 0x00 1792 VPD Keyword RW STR Remaining read write area 1794 Length Oxfd 1795 Data STR_ZERO Reserved 0x00 2048 VPD Keyword RW STR Remaining read write area 2050 Length Oxfd 2051 Data STR_ZERO Reserved 0x00 2304 VPD Keywor
6. 5 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur Mellanox Technologies 49 Rev 1 7 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 50 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Appendix E Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 Ubertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 55 C 131 F betrieben werden Es ist ein Luft A strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mind
7. 12 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 44 CLASS 1 LASER PRODUCT and reference to the most recent laser standards Mellanox Technologies 13 Rev 1 7 Adapter Card Interfaces 2 Adapter Card Interfaces 2 1 I O Interfaces Each adapter card includes the following interfaces SFP port s PCI Express PCIe x4 or x8 edge connector I O panel LEDs PC compatible connector for debug Figure 3 MCX312A XCBT Card 14 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Figure 4 MCX312B XCBT Card Figure 5 MCX311A XCAT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables Mell
8. Ethernet adapter under Network adapters list and left click Properties 32 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Step 3 Select the Information tab from the Properties sheet example Details Events Power Management General Advanced Information Pertomance Driver AM Adaptes Information Mellanox Information Value Driver Version 42111650 Famware Version 211 500 Port Number 1 Bus Type PCI E 5 0 Gbps x8 Link Speed Part Number MOX3544 F CBT levice Id 4033 Revision Id 0 Current MAC Address 00 02 9 35 9 Permanent Address 00 02 C9 35 SE Network Status Disconnected Adapter Name Ethernet 3 IPv4 Address 163 254 27 228 Adapter User Name Save To Fie Cora 280 To save this information for debug purposes click Save To File and provide the output file name 42 FlexBoot FlexBoot enables remote boot over Ethernet Boot over Ethernet BoE or Boot over iSCSI Bo iSCSI This technology is based on the Preboot Execution Environment PXE standard specifica tion and FlexBoot software is based on the open source iPXE project see www ipxe org For more information go to http www mellanox com_ gt Products gt Software Drivers gt Ethernet Software Drivers gt FlexBoot gt Download 4 3 RDMA over Converged Ethernet ConnectX 3 connected to an Ethernet fabric pro
9. Bracket on the Card MCX312A XCBT Mellanox Technologies 47 Rev 1 7 Figure 23 Placing the Bracket on the Card MCX312B XCBT 3 Make sure that the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 lbs in torque on the screws 48 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM LE cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles en cuivre en les branchant et en les d branchant des connecteurs Consultez le fabricant des Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention c bles pour conna tre les mises en garde et les instructions sp ciales
10. Hub Control Status and RAS Registers 3423 PU Intel R 5520 5500 X58 VO Hub GPIO and Scratch Pad Registers 3422 OM tel 5520 5500 x58 VO Hub PCI Express Root Port 1 3408 PU Intel R 5520 5500 X58 VO Hub PCI Express Root Port 2 3409 P Intel Fo 5520 5500 x58 Hub PCI Express Root Port 3 3404 PU intel f 5520 5500 X58 WO Hub PCI Express Root Port 7 340 PE te O 5520 5500 x 58 VO Hub System Management Registers 2428 pO Intel R 5520 5500 X58 Hub Throttle Registers 3438 PEN detel Fo 5520 5500 X58 VO Hub to ESI Port 3406 PU IntellF 5520 5500 X58 Physical and Link Layer Registers Port O 3425 PM intel 520 5500 X530 Routing and Protocol Layer Registers Port O 3426 SHE detel F 5520 5500 X58 Trusted Execution Technology Registers 342F Intel 82801 PCI Bridge 244E 188 intel Chipset Quick Data Technology device 3429 SER intel Chipset QuickOsts Technology device 342A ER inte Fo Chipset Quick Data Technology device 3428 PMA Chipset Quick Data Technology device PER intel Chipset Quick Technology device 29430 ME intel Chipset QuickData Technology device 3431 PU Intel R Chipset QuickDete Technology device 3432 JU Chipset QuickData Technology device 3433 pO intel ICHIO Farnity PCI Express Root Port 1 3 40 PER ICH10 PCI Express Root Port 5 3448 188 deste ICH10 Family SMBus Controller 3430 077777 Step 2 Right click a Mellanox
11. PCI device does not have a Mellanox adapter ID return to Step 4 to check another device 5 The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 3 3 2 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt lspci grep i Mellanox 27 00 0 Network controller Mellanox Technologies MT27500 Fam ily ConnectX 3 28 Mellanox Technologies ConnectX 3 Ethernet Single Dual SFP Port Adapter User Manual Rev 1 7 3 4 Cables and Modules Please refer to Mellanox Products Approved Cable Lists at http www mellanox com related docs user_manuals Mellanox_approved_cables pdf 3 4 1 Inserting the Optical Transceiver Module To insert the module into the cage 1 Open the module s locking mechanism see Figure 11 and Figure 12 2 Make sure that the male connectors on the module will align with the female connectors inside of the cage Also check that there is no dirt or foreign matter in the module or in the cage Mellano sv 8 MEMITOZA S 1 f Mellanox artcen o osiee 0071 1 mrmrreznse whe DE praestet Wi aw Class 1 13 1 MADE IN MALA YA 3 Insert the module into the adapter card module cage 4 Close the locki
12. Products gt Software Drivers gt Ethernet SW Drivers gt ConnectX EN 10GigE Windows Driver gt Download Follow the installation instructions included in the download package also available from the download page To ensure that communication has been established follow the instructions below Mellanox Technologies 31 Rev 1 7 Driver Software and Firmware To display a summary of network adapter software firmware and hardware related information such as driver version firmware version bus interface adapter identity and network port link information perform the following steps Step 1 Display the Device Manager Figure 13 Device Manager Eile Action View Help m Kd nm gt PY Mice and other posting devices b BE Monitors Nemwork adapters BCMS709C NetXtreme Il Gig NDIS VED Client WP Broadcom BCMS702C NetXtr if GigE NDIS VBD Client 49 N M USB Remote NDIS Network Device Ports COM amp LPT dee Print queues ES Processors Security devices lt gt Storage controllers DEM System devices SER ACPI Fixe di Feature Button ME Brosdcorn BCNSTO9C Netxtreme GigE 248 DM Broadcom BCMS700C NetXtreme Gigk 49 jM Composite Bus Enumerstor PER Direct memory access controller PU intel 5520 5500 Physical and Link Layer Registers Port 1 3427 PW intel 5520 5500 Routing and Protocol Layer Register Port 1 2420 PEN detel R 5520 5500 X58
13. is measured 1 from the heat sink between the heat sink and the cooling air inlet Mellanox Technologies 37 Rev 1 7 A2 MCX312 AB XCBT Specifications Table 10 MCX312A XCBT Specifications Table Size 2 71in x 5 6in 68 90 mm x 142 25 mm Physical Connector SFP 10Gb s Ethernet 10GBASE CX4 10GBASE R and 1000BASE R Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 0 8 0GT s 4 lanes 2 0 and 1 1 compatible PCI Express Gen2 SERDES 0 5 0GT s 8 lanes 1 1 compatible Voltage 12V 3 3V Typ Power PCIe x4 Gen3 Passive Cables 3 83W PCIe x8 Gen2 Passive Cables 5 31W Max Power PCIe x4 Gen3 Power and Passive Cables 4 55W Environmental PCIe x8 Gen2 Passive Cables 6 20W Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 2001 EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user manuals Mellanox approved cables pdf a For both operational and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 38 Mellanox Tech
14. x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification 42 Mellanox Technologies Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Figure 19 PCIe Connector Pinout GND PERNO GND RSVD GND PERp1 GND GND PERp2 PERN2 GND GND PERp3 PERMA GND RSVD RSVD GND PERp4 PERNA GND GND PERNS GND GND PERNS GND GND PERp7 PERNT GND Mellanox Technologies 43 Rev 1 7 B 3 SFP Connector Pinout Figure 20 Rear View of Module With Pin Placement SFP Module at 75 lt lt EN a 4 E a 8 E 8 5 4 8 8 5 5 6 B 3 N 0854 XL vas 199A une XL Sv dow 507 Xul VAR VooT VeeT TD RS 5 SDA VeeT VocR RS1 Rx LOS TA Disable TX Fault MOD ABS 44 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Table 12 SFP Connector Pinout Pin Symbol Name Description 1 VeeT Transmitter Ground Common with Receiver Ground 2 TX_Fault Transmitter Fault P 3 TX_Disable Transmitter Disable Laser output disabled on high or open 4 SDA 2 wire Serial Interface Data Line Y 5 SCL 2 wire Serial Interface Clock Line 4 6 MOD_ABS Module Absent Grounded within the mod
15. 0 256 VPD Keyword RW STR Remaining read write area 258 Length Oxfd 259 Data STR_ZERO Reserved 0x00 512 VPD Keyword RW STR Remaining read write area 514 Length Oxfd 515 Data STR_ZERO Reserved 0x00 768 VPD Keyword RW STR Remaining read write area 770 Length Oxfd 771 Data 5 ZERO Reserved 0x00 ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Table 7 MCX312B XCBT VPD Layout TOR Item Value Format Description 1024 VPD Keyword RW STR Remaining read write area 1026 Length Oxfd 1027 Data STR ZERO Reserved 0x00 1280 VPD Keyword RW STR Remaining read write area 1282 Length Oxfd 1283 Data STR ZERO Reserved 0x00 1536 VPD Keyword RW STR Remaining read write area 1538 Length Oxfd 1539 Data STR ZERO Reserved 0x00 1792 VPD Keyword RW STR Remaining read write area 1794 Length Oxfd 1795 Data STR ZERO Reserved 0x00 2048 VPD Keyword RW STR Remaining read write area 2050 Length Oxfd 2051 Data STR ZERO Reserved 0x00 2304 VPD Keyword RW STR Remaining read write area 2306 Length Oxfd 2307 Data STR ZERO Reserved 0x00 2560 VPD Keyword RW STR Remaining read write area 2562 Length Oxfd 2563 Data STR ZERO Reserved 0x00 2816 VPD Keyword RW STR Remaining read write area 2818 Length Oxfd 2819 Data STR ZERO Reserved 0x00 3072 VPD Keyword RW STR Remaining read write
16. 8 Technical Support ii 245 252222 A LEN ge MX ea daa 8 Firmware and Software Updates 9 Chapter I Overview i c verre E o E ers ese 10 1 1 Supported Network Protocol Standards 10 1 1 1 10 Gigabit Ethernet IRI e 10 1 2 Adapter Cards Covered in this Manual 11 1 3 Finding the MAC and Serial Number on the Adapter Card 11 14 Safety in RR 20h Kerken 12 Chapter 2 Adapter Card Interfaces 14 ZO tac o nb 14 2 1 1 Bth rnet SEP Interface 17 2123 BELExpress Interfacer Senn etienne st dan gel ois tisse 17 2 13 A ieee ae des M eam eU bets 17 2 1 4 I2C compatible Interface 18 2 2 ALONE n Uta IR A ARA n Pr eu dau 18 2 3 SEP PowerLb yel ucc Fed UOS be EA we eU en a ee ES 18 2 1 Memoty A I A A p obi s 18 24 1 System Memory a RE SSS TESS tag Rr dor e a UR 19 DA D SPI Mars 424 Are ee spur vs 19 2 4 3 BEPROM nr die HS Wee A Gide debate 19 Chapter 3 Ethernet Adapter Card Installation
17. Mellanox TECHNOLOGIES ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual P N MCX312A XCBT MCX312B XCBT MCX311A XCAT Rev 1 7 www mellanox com Rev 1 7 ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING I
18. N ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Copyright 2012 Mellanox Technologies All rights reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB FabricIT MLNX OS ScalableHPC TestX Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3755 ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Table of Contents Table of Contents sente A rr a lens alle is ets le ere Se ss oe ee Listof Tables scene ee Up eR e qe Last ot F19ur 6S A asa Revision History senes a E About this Manual I IH nd Intended Audiencecs us NL ELLE ee 8 Related Documentation A eye e EATER ICE UIS Rn ee e 8 Document Conventions
19. an Ethernet network protocol standard for LANs specifying data transfer rate of 10Gb s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover 10 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 1 2 Adapter Cards Covered in this Manual Table 3 lists the single and dual port 10 Gigabit Ethernet adapter cards described in this manual Table 3 Single and Dual port 10 Gigabit Ethernet Adapter Card 3 Data Transmis IC s 4 sion Rate RoHS Part Number Device ID decimal of ports MCX312A XCAT PCIe 3 0 x4 x8 10 GigE dual port R 6 MT27518A0 FCCR XE 4099 for Physical Legacy product 8GT s SFP Function 4100 for Virtual Function MCX311A XCAT PCIe 3 0 x4 10 GigE single R 6 MT27518A1 FCCR XE 4099 for non virtu 8GT s port SFP alized servers 4100 for virtual machines MCX312A XCBT PCIe 3 0 x4 x8 10 GigE dual port R 6 MT27518A1 FCCR XE 4099 for Physical 8GT s SFP Function 4100 for Virtual Function MCX312B XCBT PCIe 3 0 x4 x8 10 GigE dual port R 6 MT27518A1 FCCR XE 4099 for Physical 8GT s SFP Function 4100 for Virtual Function 1 3 Finding the MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a label on the print side that shows the card serial
20. anox Technologies 15 Rev 1 7 Adapter Card Interfaces Figure 6 Dual port Bracket for MCX312A XCBT Figure 7 Dual port Bracket for MCX312B XCBT 16 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Figure 8 Single port Bracket For dual port cards Port 1 connects to connector 1 of the device while Port 2 connects to connec tor 2 of the device 2 1 1 Ethernet SFP Interface The network ports of the ConnectX 3 adapter cards are compliant with the IEEE 802 3 Ethernet standards listed in Section 1 1 1 10 Gigabit Ethernet on page 10 The SFP port has one Tx Rx pair of SerDes Ethernet traffic is transmitted through the cards SFP connectors 2 1 2 PCI Express Interface The ConnectX 3 adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x4 or x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations 2 1 3 LED Assignment There are two 1 0 LEDs per port in dual port designs and one bicolor LED in single port designs green and yellow located on the 1 panel See Table 4 for different LED functions Mellanox Technologies 17 Rev 1 7 Adapter Card Interfaces Table 4 Physical and Logical Link Indication LED Function Constant on indicates a good physical link Ifneither LED is lit then the physical link has not been e
21. area 3074 Length Oxfd 3075 Data STR ZERO Reserved 0x00 3328 VPD Keyword RW STR Remaining read write area 3330 Length Oxfd 3331 Data STR ZERO Reserved 0x00 3584 VPD Keyword RW STR Remaining read write area 3586 Length Oxfd 3587 Data STR ZERO Reserved 0x00 3840 VPD Keyword RW STR Remaining read write area 3842 Length Oxfc 3843 Data STR ZERO Reserved 0x00 4095 Small Resource Type END 0x78 Tag 0x11 Mellanox Technologies 23 Rev 1 7 Adapter Card Interfaces Table 8 MCX311A XCAT VPD Layout TOO Item Value Format Description 0 Large Resource Type ID 0x82 String Tag 0x02 1 Length 7 0 LSB 0x18 2 Length 15 8 MSB 0x0 3 Data CX311A Con STR nectX 3 SFP 27 Large Resource Type VPD R 0x90 Tag 0x10 28 Length 7 0 LSB 0x4F 29 Length 15 8 MSB 0x00 30 VPD Keyword PN STR Add in Card Part Number 32 Length 0 15 33 Part Number PN STR_SPC 54 VPD Keyword EC STR Engineering Change Level of the card rev 56 Length 0x2 57 Revision RV STR PCB revision 59 VPD Keyword SN STR Serial Number 61 Length 0x18 62 SerialNumber SN STR_SPC 00 00X XXX XX 86 VPD Keyword VO STR Misc Information 88 Length 0 10 89 Data PCIe x4 STR SPC 105 VPD Keyword RV STR 107 Length 0x1 108 Data 0 107 96 50 109 Large Resource Type VPD W 0x91 Tag 0x11 110 Length 7 0 LSB 0 8
22. ble 7 MCX312B XCBT VPD Layout Pon Item Value Format Description 0 Large Resource Type ID 0x82 String Tag 0x02 1 Length 7 0 LSB 0x18 2 Length 15 8 MSB 0x0 Mellanox Technologies 21 Rev 1 7 Adapter Card Interfaces Table 7 MCX312B XCBT VPD Layout 22 Mellanox Technologies RR Item Value Format Description 3 Data CX312B Con STR nectX 3 SFP 27 Large Resource Type VPD R 0x90 Tag 0x10 28 Length 7 0 LSB Ox4F 29 Length 15 8 MSB 0x00 30 VPD Keyword PN STR Add in Card Part Number 32 Length 0 15 33 Part Number PN STR_SPC 54 VPD Keyword EC STR Engineering Change Level of the card rev 56 Length 0x2 57 Revision RV STR PCB revision 59 VPD Keyword SN STR Serial Number 61 Length 0x18 62 SerialNumber SN STR_SPC 00 00X XXX XX 86 VPD Keyword VO STR Misc Information 88 Length 0 10 89 Data PCIe Gen3 x8 STR SPC 105 VPD Keyword RV STR 107 Length 0x1 108 Data 0 107 96 50 109 Large Resource Type VPD W 0x91 Tag 0x11 110 Length 7 0 LSB 0 8 111 Length 15 8 MSB OxF 112 VPD Keyword VI STR EFI Driver version 114 Length 0x6 115 Data N A STR_ SPC 121 VPD Keyword YA STR Asset Tag 123 Length 0x18 124 Data N A STR_SPC N A 148 VPD Keyword RW STR Remaining read write area 150 Length 0x69 151 Data STR_ZERO Reserved 0x0
23. ctor output Should be pulled up with 4 7 10 on host board to a voltage between 2 0V and 3 6V Logic 0 indicates normal operation logic 1 indicates loss of signal Rev 1 7 Appendix C Replacing a Tall Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Mellanox Tech nologies adapter card and replace it with a short one C 1 Replacing a Bracket To replace the bracket you will need the following parts the new bracket of the proper height one new square gasket for each of the ports the 2 screws saved from the removal of the bracket the 2 fiber washers saved from the removal of the bracket C 2 Removing the Existing Bracket from the Adapter Card Figure 21 Bracket Screws Screws Remove the two screws holding the bracket in place 2 The bracket comes loose from the card Be careful not to put stress on the LEDs 3 Save the two screws and the two fiber washers 46 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 C 3 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the procedure above step 1 Figure 22 Placing the
24. d RW STR Remaining read write area 20 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Item Value Format Description 1282 Length Oxfd 1283 Data STR_ZERO Reserved 0x00 1536 VPD Keyword RW STR Remaining read write area 1538 Length Oxfd 1539 Data STR_ZERO Reserved 0x00 1792 VPD Keyword RW STR Remaining read write area 1794 Length Oxfd 1795 Data STR_ZERO Reserved 0x00 2048 VPD Keyword RW STR Remaining read write area 2050 Length Oxfd 2051 Data STR_ZERO Reserved 0x00 2304 VPD Keyword RW STR Remaining read write area 2306 Length Oxfd 2307 Data STR_ZERO Reserved 0x00 2560 VPD Keyword RW STR Remaining read write area 2562 Length Oxfd 2563 Data STR_ZERO Reserved 0x00 2816 VPD Keyword RW STR Remaining read write area 2818 Length Oxfd 2819 Data STR_ZERO Reserved 0x00 3072 VPD Keyword RW STR Remaining read write area 3074 Length Oxfd 3075 Data STR_ZERO Reserved 0x00 3328 VPD Keyword RW STR Remaining read write area 3330 Length Oxfd 3331 Data STR_ZERO Reserved 0x00 3584 VPD Keyword RW STR Remaining read write area 3586 Length Oxfd 3587 Data STR_ZERO Reserved 0x00 3840 VPD Keyword RW STR Remaining read write area 3842 Length Oxfc 3843 Data STR ZERO Reserved 0x00 4095 Small Resource Type END 0x78 Tag 0x11 Ta
25. d RW STR Remaining read write area 2306 Length Oxfd 2307 Data STR_ZERO Reserved 0x00 2560 VPD Keyword RW STR Remaining read write area 2562 Length Oxfd 2563 Data STR_ZERO Reserved 0x00 2816 VPD Keyword RW STR Remaining read write area 2818 Length Oxfd 2819 Data STR_ZERO Reserved 0x00 3072 VPD Keyword RW STR Remaining read write area 3074 Length Oxfd 3075 Data STR_ZERO Reserved 0x00 3328 VPD Keyword RW STR Remaining read write area 3330 Length Oxfd 3331 Data STR_ZERO Reserved 0x00 3584 VPD Keyword RW STR Remaining read write area 3586 Length Oxfd 3587 Data STR_ZERO Reserved 0x00 3840 VPD Keyword RW STR Remaining read write area 3842 Length Oxfc 3843 Data STR ZERO Reserved 0x00 4095 Small Resource Type END 0x78 Tag 0x11 Mellanox Technologies 25 Rev 1 7 Adapter Card Interfaces 26 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 3 Ethernet Adapter Card Installation 3 1 Hardware Requirements A system with a PCI Express x4 or x8 slot is required for installing the card 3 2 Installation Instructions Read all installation instructions before connecting the equipment to the power source y Bracket on page 46 5 To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short The adapter cards listed above are standard PCI Express cards each with a standard x8 edge con nector The cards require a PCI Exp
26. echnologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos gt 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable gt 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales gt Mellanox Technologies 53 Rev 1 7 7 C digos el ctricos l
27. er mst start f Verify that the card firmware was updated successfully gt flint d dev mst mt4099 pci Image type ConnectX FW Version 2 9 4100 Device ID 4099 36 Mellanox Technologies ConnectX 3 Ethernet Single Dual SFP Port Adapter User Manual Rev 1 7 Appendix A Specifications A 1 MCX311A XCAT Specifications Table 9 MCX311A XCAT Specifications Table Size 2 11in x 4 18 in 53 59mm x 106 06 mm Physical Connector SFP 10Gb s Ethernet XAUV10GBASE KX4 4X 10GigE port XFI SFI 10GBASE KR 1X 10GigE port SGMII 1000BASE KX 1X 1GigE port Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 4 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 3 47W Max Power Passive Cables 4 84W Power and Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 2001 EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user_manuals Mellanox_approved_cables pdf a For both operational and non operational states b Air flow
28. estens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschlie en Trennen von Kupferkabel Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die A gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden Mellanox Technologies 51 Rev 1 7 This equipment should be installed in compliance with local and national elec trical codes 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt fiihren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 52 Mellanox T
29. k Indication 18 Table 5 Jumper Configuration 19 Table 6 MCX312A XCBT VPD Layout ues ee quet mace etes eer tute eu pe s 19 Table 7 2 VED Layout a AER 21 Table 8 MCX311A XCAT VPD RRA 24 Table 9 MCX311A XCAT Specifications Table 37 Table 10 MCX312A XCBT Specifications Table 38 Table 11 I2C compatible Connector Pinout 42 Table 12 SFP Connector Pinout 45 Mellanox Technologies 5 Rev 1 7 List of Figures Figure 1 Product Label Example 11 Figure 2 Board Labels for MCX312B XCBT 12 Figure 3 MEX312A XCBT Card rennes oy dave SEEN SUE du 14 12 a oeiee Ree Heep acd bus e EY Ih ee EN PEERS 15 Figure 5 Card 2 15 Figure 6 Dual port Bracket for MCX312A XCBT 16 Figure 7 Dual port Bracket for MCX312B XCBT 16 Figure 8 Single port Bracket
30. ng Mechanism To remove the module from the cage 1 Unlock the locking mechanism by opening the handle 2 Pull the module out of the cage 3 4 2 Cable Installation 1 All cables can be inserted or removed with the unit powered on To insert a cable press the connector into the port receptacle until the connector is firmly seated 2 After inserting a cable into a port the GREEN LED indicator will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port 3 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the YELLOW LED will come on When data is being transferred the yellow led will blink Mellanox Technologies 29 Rev 1 7 Ethernet Adapter Card Installation gt When installing cables make sure that the latches engage 5 Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 4 should be taken as not to impede the air exhaust flow through the ventilation holes Cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack should be used 5 remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will t
31. nologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 A 3 Board Mechanical Drawing and Dimensions All dimensions are in millimeters All the mechanical tolerances are 0 1mm Figure 15 Mechanical Drawing of the Dual port MCX311A XCAT Adapter Card PC Connector Flash Jumper 142 24 E iri CE a al 7 CINE ce Fe ur ge 49 09 ML SE 1 5 0 O Mellanox Technologies 39 Rev 1 7 Figure 16 Mechanical Drawing of the Dual port MCX312A XCBT Adapter Card 2 Connector Flash Jumper 142 25 qui 8 E p 45 95 40 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Regulatory Statements For regulatory statements for all ConnectX 3 cards please refer to http www mellanox com related docs user_manuals Regulatory and Compliance Guide pdf Mellanox Technologies 41 Rev 1 7 Appendix B Interface Connectors Pinout B 1 l2C compatible Connector Pinout Figure 18 Compatible Connector Plug and Pinout Table 11 12C compatible Connector Pinout Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC B 2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x4 or
32. number and the card MAC for the Ethernet protocol Figure 1 Card Product Label Example Model No CX312A ConnectX 3 EN 10 Gigabit Ethernet P N MCX312A XCAT Rev 1 LULU SIN MT0806X01504 TI MAC 00 02 C9 27 05 1C Bes Made in Israel Mellanox Technologies 11 Rev 1 7 Overview Figure 2 Board Labels for MCX312B XCBT Model No CX312B 7 ConnectX 3 EN 10 Gigabit Ethernet Made in Israel 5 QUE HU O PS 22 ER ev SIN 806 01504 R 4 MAC 00 02 C9 27 051C 1 4 Safety Warnings Below is a list of safety warnings in English For safety warnings in other languages please refer to the appendices in this user manual 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting attached to or detached from the connectors Refer to the cable manufacturer for spe Some copper cables are heavy and not flexible as such they should be carefully A cial warnings and instructions
33. ocales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes TEC 60825 1 2007 03 y EN 60825 1 2007 54 Mellanox Technologies
34. ress x4 or x8 Please consult the host machine documentation for instructions on how to install a PCI Express card 3 If the card is installed in a PCI slot with less lanes than the card requires then the adapter card will not provide the optimum data transfer aa 3 3 Identify the Card in Your System 3 3 1 On Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 adapter card 3 Select Properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or Hardware Ids Windows 2008 R2 from the Property pull down menu Mellanox Technologies 27 Rev 1 7 Ethernet Adapter Card Installation Figure 10 PCI Device Mellanox MT27500 4099 ConnectX 3 YPI Network Adapter Pr General Port Protocol Driver Details Resources f A Mellanox MT27500 4099 3 Network Adapter 1 2 Property Hardware Ids Value 15B3 amp DEV 1003 amp 5UBSYS 00515B35REV 00 5 Inthe Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string PCINVEN 15B3 amp DEV 1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID If the
35. roelectronics accessible via the SPI interface of the ConnectX 3 EN device When a jumper is inserted into the drill holes in the adapter it indicates to the device that the on board Flash device should not be used for boot instructions Contact your Mellanox support repre sentative should you need to use this jumper Table 5 provides information on this jumper The jumper location on the board is illustrated in Appendix A 3 Board Mechanical Drawing and Dimensions on page 39 Table 5 Jumper Configuration 4 Card Default Description Option Flash connection open Flash present connection open not present Flash present connection shorted Flash not present 2 4 3 EEPROM Each board incorporates an EEPROM that is accessible through the I C compatible interface The EEPROM capacity is 4Kb The EEPROM is used for storing the Vital Product Data VPD The PCI VPD Vital Product Data layout for each of the adapter cards complies with the format defined in the PCI 3 0 Specification Appendix 1 Table 6 MCX312A XCBT VPD Layout Offset uH Decimal Item Value Format Description 0 Large Resource Type ID 0x82 String Tag 0x02 1 Length 7 0 LSB 0x18 2 Length 15 8 MSB 0x0 3 Data CX312A Con STR nectX 3 SFP 27 Large Resource Type VPD R 0x90 Tag 0x10 Mellanox Technologies 19 Rev 1 7 Adapter Card Interfaces
36. rsion found on the Mella nox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm ware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Support gt Download Firmware Tools A firmware binaries table lists a binary file per adapter card The file name of each such binary is composed by combining the firmware name the firmware release version and the card part num ber Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at http www mellanox com gt Products gt Software Drivers gt Ethernet Software Drivers gt Firmware Tools Make sure to down load the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci cr0 d Get the PSID firmware identification and programmed firmware version using the command 34 Mellanox Technologies ConnectX 3 E
37. ry This document was printed on November 14 2012 Table 1 Revision History Table Date Rev Comments Changes November 2012 1 7 e Added MCX312B XCBT to the following locations Section 1 2 Adapter Cards Covered in this Manual on page 11 Figure 4 MCX312B XCBT Card on page 15 Figure 7 Dual port Bracket for MCX312B XCBT on page 16 Table 7 MCX312B XCBT VPD Layout on page 21 Figure 17 Mechanical Drawing of the Dual port MCX312B XCBT on page 40 October 2012 1 6 Added single port SFP card to all necessary locations Fixed Table 5 Jumper Configuration on page 19 e Updated figures in Section 4 1 2 Windows on page 31 by pro viding ConnectX 3 examples October 2012 1 5 Removed all references to 40 Gigabit Ethernet from the User Manual August 2012 1 4 Added operational and non operational temperature and humid ity level to the following tables Table 10 MCX312A XCBT Specifications Table on page 38 January 2012 1 2 Minor edits Updated LED functions in Section 2 1 3 LED Assignment on page 17 Formatted specification tables in Appendix A Specifications on page 37 October 2011 1 1 Added new OPNs based on ConnectX 3 Step A1 devices July 2011 1 0 Minor edits July 2011 0 10 First Release Mellanox Technologies 7 Rev 1 7 About this Manual This User Manual describes Mellanox Technologie
38. s ConnectX 3 10 Gigabit Ethernet Single Dual SFP port PCI Express x4 or x8 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant docu mentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet networks and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual User Manual describing the set of MFT firmware management tools for a Document no 2204UG single node See http www mellanox com gt Support gt Download Firmware Tools IEEE Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechanical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits In this document PCIe is used to mean PCI Express Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Suppor
39. stablished Green physical link Yellow A blinking yellow indicates activity data transfer logical data activity Stays off when there is no activity link The short bracket has the same port and LED footprints as the tall bracket 2 1 4 PC compatible Interface 2 2 2 3 2 4 A three pin header on the adapter cards is provided as the I2C compatible interface See Figure 16 Mechanical Drawing of the Dual port MCX312A XCBT Adapter Card on page 40 for the loca tion on the board Figure 9 Connector Power All adapter cards receive 12V and 3 3V power from the PCI Express edge connector All other required power voltages are generated by on board switch mode regulators See Specifications on page 37 SFP Power Level The adapter cards support power levels according to SFF Committee SFF 8431 Specification for SFP Small Form factor Pluggable Transceiver Memory The adapter cards support multiple memory devices through the PCIe SPI Flash and inter faces 18 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 2 4 1 System Memory The adapter cards utilize the PCI Express interface to store and access Ethernet fabric connection information and packet data on the system memory 2 4 2 SPI Each of the adapter cards includes one 16MB SPI Flash device 25 16 device by ST Mic
40. t Services please see your contract for details regarding Technical Support 8 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database informa tion for Mellanox products Access the data base from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Rev 1 7 Overview 1 Overview This document is a User Manual for Mellanox Technologies Ethernet cards based on the Con nectX 3 EN integrated circuit device The cards described in this manual have the following main features IEEE Std 802 3 compliant PCI Express 3 0 1 1 and 2 0 compatible through an x4 or x8 edge connector up to 8GT s 10 Gb s Ethernet Single and Dual SFP ports for connection Ethernet traffic RDMA over Converged Ethernet RoCE End to end QoS and congestion control Hardware based 1 virtualization TCP UDP IP stateless off load RoHS R6 compliant Two bracket heights short and tall 1 1 Supported Network Protocol Standards 1 1 1 10 Gigabit Ethernet 10 Gigabit Ethernet is
41. thernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 gt flint d dev mst mt4099 pci Image type ConnectX FW Version 2 9 4000 Device ID 4099 Chip Revision 0 Description Node Porti Port2 Sys image GUIDs 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT_1020110019 VSD PSID MT_1020110019 2 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 14 b Enter your card PSID to display the latest firmware Figure 14 Support Download Assistant Mellanox TECHNOLOGIES Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home MT_1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Sel Switches Gateways 3 Ifanewer firmware version exists for your adapter card update the firmware as follows a Download the firmware image zip file from the Download Center see Step 2a above b Unzip the firmware image c Burn the firmware image Enter Mellanox Technologies 35 Rev 1 7 Driver Software and Firmware gt flint d dev mst mt4099 pci i binary image burn d Reboot the computer e Ent
42. ule Y 7 RSO No connection required 8 RX_LOS Loss of Signal indication Logic 0 indicates normal operation 9 RS1 No connection required 10 VeeR Receiver Ground Common with Transmitter Ground VeeR Receiver Ground Common with Transmitter Ground 12 RD Receiver Inverted DATA out AC Coupled 13 RD Receiver Non inverted DATA out AC Coupled 14 VeeR Receiver Ground Common with Transmitter Ground 15 VccR Receiver Power Supply 16 VecT Transmitter Power Supply 17 VeeT Transmitter Ground Common with Receiver Ground 18 TD Transmitter Non Inverted DATA in AC Coupled 19 TD Transmitter Inverted DATA in AC Coupled 20 VeeT Transmitter Ground Common with Receiver Ground Mellanox Technologies 45 a Circuit ground is internally isolated from chassis ground b Tpaurr is an open collector drain output which should be pulled up with a 4 7k 10k Ohms resistor on the host board if intended for use Pull up voltage should be between 2 0V to Vcc 0 3V A high output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed ing the preset alarm thresholds A low output indicates normal operation In the low state the output is pulled to lt 0 8V Laser output disabled on TDIS gt 2 0V or open enabled on TDIS lt 0 8V d Should be pulled up with 4 7kQ 10kQ on host board to a voltage between 2 0V and 3 6V MOD ABS pulls line low to indicate module is plugged in e LOS is open colle
43. urn off when the cable is unseated Cables especially long copper cables can weigh a substantial amount Make sure that LA the weight of the cable is supported on its own and is not hanging from the adapter card 30 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 7 4 Driver Software and Firmware 4 1 Driver Software 4 1 1 Linux For Linux download and install the latest MLNX_EN driver software package available via the Mellanox web site at http www mellanox com gt Products gt Software Drivers gt Ethernet SW Drivers gt ConnectX 3 EN 10GigE Linux Driver gt Download Follow the installation instructions included in the download package also available from the download page To ensure that communication has been established follow the instructions below Check the link status First check the network interface name by running the ifconfig a command To assign an IP address to the interface run a ifconfig eth lt x lt ip gt where x is the OS assigned interface number To check driver and device information run gt ethtool i eth lt x gt Example Gum i GENZ driver mlx4_en MT_1020110019_CX 3 version 2078 firmware version 2 10 0000 bus info 0000 07 00 0 4 1 2 Windows For Windows download the MLNX EN Ethernet driver Download this package from the Mella nox web site at http www mellanox com gt
44. vides all of the basic NIC functionality plus RDMA over Converged Ethernet RoCE utilizes advances in Data Center Bridging DCB to enable efficient and low cost implementations of RDMA over Ethernet supporting the entire breadth of RDMA and low latency features This includes reliable connected service datagram service RDMA and send receive semantics atomic operations user level multicast user level I O access kernel bypass and zero copy This capability is enabled when using the Mellanox OFED or WinOF VPI drivers Mellanox Technologies 33 Rev 1 7 Driver Software and Firmware ConnectX 3 EN with RoCE based network management is the same as that for any Ethernet and DCB based network management eliminating the need for IT managers to learn new technolo gies 4 3 1 Hardware and Software Requirements Software Mellanox OFED 1 5 1 WinOF 2 1 2 or later To download these packages go to http www mellanox com gt Products gt Software Drivers gt InfiniBand amp VPI SW Driver Hardware ConnectX 3 EN adapter card 4 4 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com gt Support gt Download Firmware Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest ve

ConnectX-3 10GbE User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents