Home

User Manual - Advanced HPC

1. 43 Table 16 MCX354A FCBT Specifications Table 44 Table 17 I2C compatible Connector Pinout 47 Table 18 Connector Pin Number Name to Signal Name Map 50 Mellanox Technologies 5 ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Revision History This document was printed on November 14 2012 Table 1 Revision History Table Date Rev Comments Changes November 2012 1 6 Updated Web links in the following locations Section 4 5 1 Hardware and Software Requirements on page 36 Table 11 MCX353A QCBT Specifications Table on page 39 Table 12 MCX354A QCBT Specifications Table on page 40 Table 13 MCX353A TCBT Specifications Table on page 41 Table 14 MCX354A TCBT Specifications Table on page 42 Table 15 MCX353A FCBT Specifications Table on page 43 Table 16 MCX354A FCBT Specifications Table on page 44 October 2012 1 56 Fixed Table 9 Jumper Configuration on page 21 Updated figures in Section 4 1 2 Windows on page 32 by pro viding ConnectX 3 examples August 2012 1 4 Added operational and non operational temperature and humid ity level to the following tables Table 11 MCX353A QCBT Specifications Table on page 39 Table 12 MCX354A QCBT Specif
2. Mellanox TECHNOLOGIES ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual P N MCX353A FCBT 53 53 MCX354A FCBT MCX354A TCBT MCX354A QCBT Rev 1 6 www mellanox com Rev 1 6 ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIAB
3. cat proc driver nvidia version gt version 195 36 15 or later 4 5 RDMA over Converged Ethernet ConnectX 3 connected to an Ethernet fabric provides all of the basic NIC functionality plus RDMA over Converged Ethernet RoCE RoCE utilizes advances in Data Center Bridging DCB to enable efficient and low cost implementations of RDMA over Ethernet supporting the entire breadth of RDMA and low latency features This includes reliable connected service datagram service RDMA and send receive semantics atomic operations user level multicast user level I O access kernel bypass and zero copy This capability is enabled when using the Mellanox OFED or WinOF VPI drivers ConnectX 3 EN with RoCE based network management is the same as that for any Ethernet and DCB based network management eliminating the need for IT managers to learn new technolo gies 4 5 1 Hardware and Software Requirements Software Mellanox OFED 1 5 1 WinOF 2 1 2 or later To download these packages go to http www mellanox com gt Products gt Software Drivers gt InfiniBand amp VPI SW Driver 36 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Hardware ConnectX 3 card 4 6 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obta
4. ARR LES 18 2 1 5 LED Assigninent 22 2 2 A Et Re phon lie neh RE 19 2 1 6 12C compatible Interface 4 255 224 vns eis aaa et a 20 22 POWOD Este ar Me ae se elle ie 20 273 OSEP Power Level E Rae RE mace e ert aN ale es 20 s ers ea A ede At ee ER Be dee 20 24 1 System Memory tn aan RNA are 20 24 2 SPL SS I AA rn A A A 20 27453 WHE PROM e tod a do 21 Chapter 3 Adapter Card 24 3 1 Hardware Requirements 22222 ads seated odd dels vad daly ve 24 3 2 Installation Instructions 24 3 3 Identify the Card in Your System 24 3521 On Wind OWS ee Uwe Eq den an mi ees ee 24 O dh eh ge eet een ee Gow eee era tate 25 3 4 Cables and Modules sa A AS pa ee Aan ea ed 26 321 Cable Installation leal 26 3 4 1 1 Inserting a Cable into the Adapter Card 26 3 4 1 2 Removing a Cable from the Adapter Card 27 Chapter 4 Driver Software and Firmware 28 A Driver Software edt o ex TR tata eto te ptt eren Serie en 28 AW nue an AS
5. List of Figures Figure 1 Product Label 1 13 Figure 2 MCX354A QFTICBT 16 Figure 3 D l port er nes PEN SUE god 17 Figure 4 Single port 17 Figure 5 Physical and Logical Link Indications InfiniBand mode 19 Figure 6 I2C compatible Connector 20 Figure 7 PGID 6VICe ois RE CU A Ede e d s 25 Figure 8 Connector Orientation 27 Figure 9 Support Download 38 Figure 10 Mechanical Drawing of the Single port MCX353A QFT CBT Adapter Card 45 Figure 11 Mechanical Drawing of the Dual port MCX354A QFT CBT Adapter Card 46 Figure 12 Compatible Connector Plug and Pinout 47 Figure 132 PCle Connector u gen pe emn set Le een ran 48 Figure 14 Connector and Cage 49 Figure 15 Bracket Screws 5 es a 51 Figure 16 Gasket Location on Adapter 52 Figure 17 Gasket Installation epi pO rast
6. C 131 F betrieben werden Es ist ein Luft A strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschlie en Trennen von Kupferkabel Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die A gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 56 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 This equipment should be installed in compliance with local and national elec trical codes 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gefahrlic
7. Download Firmware Tools IBTA Specification Release 1 2 1 InfiniBand Architecture Specification IEEE Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechanical Specifications Document Conventions When discussing memory sizes MB and MBytes used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits IB is used in this document to mean InfiniBand In this document PCIe is used to mean PCI Express Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 8 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database informa tion for Mellanox products Access the database from the Mellanox Support web page http www mellanox com gt
8. QSFP port s PCI Express PCIe x8 edge connector I O panel LEDs PC compatible connector for debug Figure 2 MCX354A QFT CBT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables For dual port cards Port 1 connects to connector 1 of the device while Port 2 connects to connec tor 2 of the device See Figure 3 for the mechanical drawing of a dual port bracket and Figure 4 for the mechanical drawing of a single port bracket 16 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Figure 3 Dual port Bracket Figure 4 Single port Bracket Mellanox Technologies 17 Rev 1 6 Adapter Card Interfaces 2 1 1 InfiniBand Interface The network ports of the ConnectX 3 adapter cards are compliant with the nfiniBand Architecture Specification Release 1 2 1 InfiniBand traffic is transmitted through the cards QSFP connectors 2 1 2 Ethernet Interface The network ports of the ConnectX 3 adapter cards are compliant with the IEEE 802 3 Ethernet standards listed in Section 1 1 4 10 and 40 Gigabit Ethernet on page 11 Ethernet traffic is transmitted through the cards QSFP connectors For connecting to an SFP interface you can use Mellanox QSA QSFP to SFP adapter module to be ordered separately 2 1 3 VPI Port Configuratio
9. gt Host ethtool ethO Settings for eth0 Supported ports TP Supported link modes 1000baseT Full 10000baseT Full Supports auto negotiation Yes Advertised link modes 1000baseT Full 10000baseT Full 28 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 To check the IB link status for IB and VPI cards run ibstat and focus on the Physical state attributes Example Mellanox Technologies 29 Rev 1 6 Driver Software and Firmware Physical state LinkUp Rate 56 Base lid 8 LMC 0 121615 12 Capability mask 0x02514868 Port GUID 0x0002c903002fefe2 Link layer InfiniBand Check the OFED version To get the version of the running Mellanox OFED BXOFED run the following command ofed info head 1 21200 INS 00 18 OE ED ASS O Troubleshooting MLNX_OFED Installation For troubleshooting driver installation please check Mellanox OFED driver user manual at http www mellanox com gt Support gt InfiniBand Software and Drivers gt Mellanox OFED Loading the Ethernet Driver By default the stack loads mlx4 en Run lsmod to verify that the module is listed Example Host 18 4 grep mlx4 mlx4 en 114184 0 mlx4 core 156512 2 mlx4 ib mlx4 en If you don t see the mlx4 en driver run modprobe mlx4 en Another option is to use the command below to see which mod
10. ees eee FE EUN 52 Figure 18 Placing the Bracket on the Card 53 6 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 List of Tables Table 1 Revision History Table 7 Fable 2 7 Documents 486 Rhein 8 Table 3 Single and Dual port QDR Adapter Cards 12 Table 4 Single and Dual port FDR10 Adapter 12 Table 5 Single and Dual port FDR Adapter 13 Table 6 Supported Port Configurations 18 Table 7 Unsupported Port Configurations 18 Table 8 Physical and Logical Link Indication Ethernet mode 19 Table 9 Jumper Configuration 21 1 1 7 21 Table 10 MCX35 34 A QFT CBT VPD Layout 21 Table 11 MCX353A QCBT Specifications Table 39 Table 12 MCX354A QCBT Specifications Table 40 Table 13 53 Specifications Table 41 Table 14 MCX354A TCBT Specifications Table 42 Table 15 MCX353A FCBT Specifications Table
11. related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a The ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC2207310 0xx x For both operational and non operational states d Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet eo 44 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 A 7 Board Mechanical Drawing and Dimensions 2 dimensions millimeters All the mechanical tolerances are 0 1mm Figure 10 Mechanical Drawing of the Single port MCX353A QFT CBT Adapter Card J1 Connector Flash Jumper 142 24 I CE 88 a oem Hm 7 CINE ce O ge 49 09 ML SE al 5 1 is Mellanox Technologies 45 Rev 1 6 Figure 11 Mechanical Drawing of the Dual port MCX354A QFT CBT Adapter Card J1 PC Connector Flash Jumpe
12. with the card 4 should be taken as not to impede the air exhaust flow through the ventilation holes Cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack should be used 5 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated the weight of the cable is supported on its own and is not hanging from the adapter Cables especially long copper cables can weigh a substantial amount Make sure that A card 3 4 1 1 Inserting a Cable into the Adapter Card 1 Support the weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack 2 Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card 3 Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card 26 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 4 Make sure that the connector locks in place Figure 8 Connector Orientation Example Tie the Cable to the Rack 3 4 1 2 Removing a Cable from the Adapter Card 1 Pull
13. Adapter Cards x Data Transmis EN ns ung d sion Rate RoHS Adapter IC Part Number Device ID decimal of ports MCX353A QCBT PCIe 3 0 x8 40 Gb s QDR IB R6 MT27508A1 FCCR QV 4099 for Physical 8GT s 10GigE single Function port QSFP 4100 for Virtual Function MCX354A QCAT PCIe 3 0 x8 40 Gb s QDR IB R6 MT27508A0 FCCR QV Legacy product 8GT s 10 GigE dual port QSFP MCX354A QCBT PCIe 3 0 x8 40 Gb s QDR IB R6 MT27508A1 FCCR QV 8GT s 10GigE dual port QSFP Table 4 Single and Dual port FDR10 Adapter Cards Ordering Part Number PCI Express Da us OPN SERDES Speed sion Rate RoHS Adapter IC Part Number Device ID decimal of ports MCX353A TCBT PCIe 3 0 x8 40 Gb s FDR10 R6 MT27508A1 FCCR TV 4099 for Physical 8GT s IB 10GigE sin Function gle port QSFP 4100 for Virtual Function MCX354A TCAT PCIe 3 0 x8 40 Gb s FDR10 R6 MT27508A0 FCCR TV Legacy product 8GT s IB 10GigE dual port QSFP MCX354A TCBT PCIe 3 0 x8 40 Gb s FDR10 R6 MT27508A1 FCCR TV 8GT s IB 10GigE dual port QSFP 12 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Table 5 Single and Dual port FDR Adapter Cards Data Transmis De 4 sion Rate RoHS Adapter IC Part Number Device ID decimal of ports MCX353A FCBT PCIe 3 0 x8 56 Gb s FDR IB R6 MT27508A1 FCCR FV 4099 for Physical 8GT s 40GigE si
14. CH10 Family SMBus Controller 3430 Intel R LPC Interface Controller 3418 Mellanox MT27500 4099 ConnectX 3 Network Adapter Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties Select the Port Protocol tab from the Properties sheet Note The Port Protocol tab is displayed only if the NIC is a VPI IB and ETH If the NIC is either only IB or ETH the tab will not be shown Note The figure below is an example of the displayed Port Protocol sheet for a dual port VPI adapter card Mellanox Technologies 33 Rev 1 6 Driver Software and Firmware 4 3 4 4 Mellanox ConnectX VPI MT26428 PCIe 2 0 5GT s IB QDR 7 1 General Port Piolocel Driver Details Resources Choose ths port protocol that best suits your needs Mellanox Pert 1 ETH Pert2 ETH Active ULPs ND SDP Port Protocol Configuration Choose the network protocols for the network adapter ports Configure both ports as Ethemet ETH ONLY s Configure both ports as InfiniBand ONLY Configure Port 1 as InfiniBand and Port 2 as Ethernet COMBO Step 3 In this step you can perform two different functions a Choose the desired port protocol for the available port s and b activate or deactivate the WSD ND and or SDP ULPs Note IB must be always the first port in Port 1 If you choos
15. H a uo rtp eee AM MS 28 2 1 25 WindOWS tire ie eel ects ee EU US ola toa 32 4 2 Port Type Management Via Driver SW 32 Mellanox Technologies 3 Rev 1 6 A Ded EE 222 32 422 Windows sn a DA repere dX aes 32 43 2 cad Seah ona 34 4 4 NVIDIA GPUDirect Support 222344 haa ur t lye said 34 4 4 1 Hardware and Software Requirements 35 4 4 25 Installation 441352 A ee as GA AS SS a hs 35 4 4 2 1 Installation 35 4422 MLNX Drivers Installation 35 4 4 2 3 NVIDIA Driver Installation 36 45 over Converged Ethernet 36 4 5 1 Hardware and Software Requirements 36 4 6 Updating Adapter Card Firmware 37 Appendix 39 MCX353A QCBT Specifications 39 A2 MCX354A QCBT Specifications 40 3 MCX353A TCBT Specifications 24 2 de pie nes Dette dr HG ds Bean ri 41 4 MCX354A TCBT Spe
16. ILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Copyright 2012 Mellanox Technologies All rights reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB FabricIT MLNX OS ScalableHPC TestX Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3754 ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Table of Contents Table of Contents sente rr rr a lens Rr ets le ere Se ss Listof Tables een ee Last ot PI GUVs ise Re In RE Revision History senes 0 E About this Manual I IH nd Intended Audiencecs us NL ELLE ert sut I dct ch Eee ee E Weck Red ee 8 Related Documentation
17. If the card is installed in a PCI slot with less lanes than the card requires then the adapter card will not provide the optimum data transfer aa 3 3 Identify the Card in Your System 3 3 1 On Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 adapter card 3 Select Properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or Hardware Ids Windows 2008 R2 from the Property pull down menu 24 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Figure 7 PCI Device Mellanox MT27500 4099 ConnectX 3 YPI Network Adapter Pr General Port Protocol Driver Details Resources f A Mellanox MT 27500 4099 ConnectX 3 Network Adapter Property Hardware Ids Value PCISVEN_15B3 amp DEV_1003 amp 5UBSY S_O00615B3REV_00 5 Inthe Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string PCI VEN_15B3 amp DEV_ 1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID gt If the PCI device does not have a Mellanox adapter ID return to Step 4 to check another device 27 The li
18. Keyword RW STR Remaining read write area 2306 Length Oxfd 2307 Data STR_ZERO Reserved 0x00 2560 VPD Keyword RW STR Remaining read write area 2562 Length Oxfd 2563 Data STR_ZERO Reserved 0x00 2816 VPD Keyword RW STR Remaining read write area 2818 Length Oxfd 2819 Data STR_ZERO Reserved 0x00 3072 VPD Keyword RW STR Remaining read write area 3074 Length Oxfd 3075 Data STR_ZERO Reserved 0x00 3328 VPD Keyword RW STR Remaining read write area 3330 Length Oxfd 3331 Data STR_ZERO Reserved 0x00 3584 VPD Keyword RW STR Remaining read write area 3586 Length Oxfd 3587 Data STR_ZERO Reserved 0x00 3840 VPD Keyword RW STR Remaining read write area 3842 Length 0 3843 Data STR_ZERO Reserved 0x00 4095 Small Resource Type END 0x78 Tag 0x11 Mellanox Technologies 23 Rev 1 6 Adapter Card Installation 3 Adapter Card Installation 3 1 Hardware Requirements A system with a PCI Express x8 slot is required for installing the card 3 2 Installation Instructions Read all installation instructions before connecting the equipment to the power source To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short y Bracket on page 51 5 The adapter cards listed above are standard PCI Express cards each with a standard x8 edge con nector The cards require a PCI Express x8 Please consult the host machine documentation for instructions on how to install a PCI Express card 3
19. Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Rev 1 6 Overview 1 Overview This document is a User Manual for Mellanox Technologies VPI adapter based on the Con nectX 3 VPI integrated circuit device The cards described in this manual have the following main features Virtual Protocol Interconnect VPI InfiniBand Architecture Specification v1 2 1 compliant IEEE Std 802 3 compliant PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Single and Dual port options available ESD protection on the network connector Compliant with QSFP MSA spec Rev 1 0 Compatible with copper cables and optical cables with the use of QSFP connectors Support for SFP cables is available through QSA Quad to Serial CORE Direct application off load GPUDirect application off load RDMA over Converged Ethernet RoCE End to end QoS and congestion control Hardware based I O virtualization TCP UDP IP stateless off load Ethernet encapsulation EoIB RoHS R6 compliant Two bracket heights short and tall 1 1 Supported Network Protocol Standards 1 1 1 InfiniBand FDR FDR is a pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified p
20. and Dual QSFP Port Adapter Card User Manual Rev 1 6 Appendix C Replacing a Tall Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Mellanox Tech nologies adapter card and replace it with a short one C 1 Replacing a Bracket To replace the bracket you will need the following parts the new bracket of the proper height one new square gasket for each of the ports the 2 screws saved from the removal of the bracket the 2 fiber washers saved from the removal of the bracket 2 Removing the Existing Bracket from the Adapter Card Figure 15 Bracket Screws Screws 1 Remove the two screws holding the bracket in place 2 The bracket comes loose from the card A Be careful not to put stress on the LEDs 3 Save the two screws and the two fiber washers Mellanox Technologies 51 Rev 1 6 C 3 Installing the New Bracket Remove the paper to expose the adhesive on the gasket 2 Place the gasket for each port onto the new bracket Make sure to correctly align the gasket with the hole in the bracket 3 Ifthe old gaskets are still on the card remove them before installing the new bracket Make sure that only one gasket per port is used Figure 16 Gasket Location on Adapter Card Gasket Figure 17 Gasket Installation 1 Place the bracket onto the card until the screw holes line up Do not force the bracket onto the card You
21. at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting attached to or detached from the connectors Refer to the cable manufacturer for spe Some copper cables are heavy and not flexible as such they should be carefully Ay cial warnings and instructions 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal 4 Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical LA codes 14 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 2 2001 and EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 15 Rev 1 6 Adapter Card Interfaces 2 Adapter Card Interfaces 2 1 I O Interfaces Each adapter card includes the following interfaces
22. ays InfiniBand auto Link sensing mode detect the port type based on the attached network type If no link is detected the driver retries link sensing every few seconds The port link type can be configured for each device in the system at run time using the sbin connectx port config script This utility will prompt for the PCI device to be modified if there is only one it will be selected automatically At the next stage the user will be prompted for the desired mode for each port The desired port configuration will then be set for the selected device Note This utility also has a non interactive mode sbin connectx port config d device PCI device ID gt c conf lt port1 port2 gt 4 2 2 Windows After installing Mellanox WinOF VPI for Windows on your machine you can change a port s pro tocol configuration The following steps describe how to configure the port type Step 1 Display the Device Manager and expand System Devices 32 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 ACPI Fixed Feature Button amp amp Composite Bus Enumerator 1 Direct memory access controller amp Intel R 5520 5500 Physical and Link Layer Registers Port 1 3427 Intel R 5520 5500 Routing and Protocol Layer Register Port 1 3428 Intel R 5520 5500 X58 I O Hub Control Status and RAS Registers 3423 a
23. cifications 42 5 MCX353A FCBT Specifications 43 6 MCX354A FCBT Specifications 44 A 7 Board Mechanical Drawing and Dimensions 45 A 8 Regulatory Statements 46 Appendix Interface Connectors 47 B 1 I2C compatible Connector Pinout 47 B2 PCLExpress x8 Connector Pinout sus use dw ote odes res bed da esi ea cae bei d ei 47 B 3 OSEP Connector Pinout sorses a Be ee kan ie a Lost ete ii 49 Appendix Replacing Tall Bracket With a Short 51 Cal R placimg a Brackets ns Meh ee LIS S ga ouk eoe es 51 C 2 Removing the Existing Bracket from the Adapter 51 7 3 Installing the New Bracket os rasade eu RE eI eet eise d Gach 52 Appendix D Avertissements de s curit d installation Warnings in French 54 Appendix E Sicherheitshinweise Warnings in 56 Appendix Advertencias de seguridad para la instalaci n Warnings Spanish 58 4 Mellanox Technologies Rev 1 6
24. e ETH as your first port in Port 1 then the second port in Port 2 can be only ETH Note WSD is not supported in Windows 7 Consequently on this OS the WSD checkbox is grayed out and cannot be selected FlexBoot FlexBoot enables remote boot over Ethernet Boot over Ethernet BoE Boot over InfiniBand BoIB or Boot over iSCSI Bo iSCSD This technology is based on the Preboot Execution Envi ronment PXE standard specification and FlexBoot software is based on the open source iPXE project see www ipxe org For more information go to gt Products gt Software Drivers gt InfiniBand amp VPI Software Drivers gt FlexBoot gt Download NVIDIA GPUDirect Support Utilizing the high computational power of the Graphics Processing Unit GPU the GPU to GPU method has proven valuable in various areas of science and technology The Mellanox Con nectX 3 based adapter cards provide the required high throughput and low latency for GPU to GPU communications 34 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 4 4 1 Hardware and Software Requirements Software Operating system RHELS 4 2 6 18 164 615 x86 64 or later Mellanox OFED with GPUDirect support NVIDIA Development Driver for Linux version 195 36 15 or later Hardware Mellanox ConnectX 3 adapter card e NVIDIA Tesla series 4 4 2 Installation For complete installation instructions visit www mella
25. e a todas las leyes y reglamentaciones nacionales gt 58 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los codigos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precaucion el uso de controles o ajustes o la realizacion de procedimientos distintos de los que aqui se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes TEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 59
26. e and Firmware 41 2 Windows For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com gt Products gt Software Drivers gt InfiniBand amp VPI Software Drivers gt Mellanox WinOF VPI gt Download Follow the installation instructions included in the download package also available from the download page To ensure that communication has been established follow the instructions below Displaying the Device Manager will show the Mellanox adapter devices and an IPoIB network device for each port Note that MLNX_VPI configures the ports of VPI cards at installation time to run the InfiniBand protocol 4 2 Port Type Management Via Driver SW 4 2 1 Linux If you wish to change the port type use the connectx_port_config script after the driver is loaded The script is installed as part ofthe Mellanox OFED for Linux package under sbin See the Mel lanox OFED for the Linux User Manual available at http www mellanox com gt Support gt InfiniBand Products gt Mellanox OFED Running sbin connectx_port_config s will show the current port configuration for all ConnectX 3 devices Port configuration is saved in the file etc infiniband connectx conf This saved configuration is restored at driver restart only if done via etc init d openibd restart Possible port types are e eth Always Ethernet ib Alw
27. e interface The EEPROM capacity is 4Kb The EEPROM is used for storing the Vital Product Data VPD The PCI VPD Vital Product Data layout for each of the adapter cards complies with the format defined in the PCI 3 0 Specification Appendix I Table 10 MCX35 34 A QFT CBT VPD Layout Offset It ipti Decimal em Value ormat Description 0 Large Resource Type ID 0x82 String Tag 0x02 1 Length 7 0 LSB 0x18 2 Length 15 8 MSB 0x0 3 Data CX354A Con STR nectX 3 QSFP 27 Large Resource Type VPD R 0x90 Tag 0x10 28 Length 7 0 LSB Ox4F 29 Length 15 8 MSB 0x00 30 VPD Keyword PN STR Add in Card Part Number Example MCX354A FCBT 32 Length 0 15 33 Part Number PN STR_SPC 54 VPD Keyword EC STR Engineering Change Level of the card rev Mellanox Technologies 21 Rev 1 6 Adapter Card Interfaces Table 10 MCX35 34 A QFT CBT VPD Layout 22 Mellanox Technologies M Item Value Format Description 56 Length 0x2 57 Revision RV STR PCB revision Example 1 59 VPD Keyword SN STR Serial Number 61 Length 0x18 62 SerialNumber SN STR_SPC 00 00X XXX XX 86 VPD Keyword VO STR Misc Information 88 Length 0x10 89 Data PCIe Gen3 x8 STR SPC 105 VPD Keyword RV STR 107 Length 0x1 108 Data 0 107 96 50 109 Large Re
28. eee 2a en AUREAS UIS eue e 8 Document 8 Technical Support Les ER IA iid iu eed nnus 8 Firmware and Software Updates 9 Chapter L Overview u Ter RS FCRI ES a 11 Supported Network Protocol Standards 10 1121 JafiniBand FDR rin ae SATB CAR Ee a AES 10 1 1 2 InfiniB nd EDR10 Ree ps IUE Rem NER e 10 12 3 InfiniBand ODR cocina a I 11 1 1 4 10 and 40 Gigabit 1 11 1 2 Adapter Cards Covered in this Manual 12 13 Finding the GUID MAC and Serial Number on the Adapter Card 13 1 47 Safety Walnin es dis 211 23 8 husdpesq Ug rendue 13 Chapter 2 Adapter Card 16 2 1 JO Interfaces ii enm eR Si pclae Dein 16 22121 InfiniBand Interface cs sn een 18 2A2 Ethernet Interface corre tee ese EUCH VEU en 18 2 1 3 VPI Port Configuration 18 2 154 terface ur Dose RR ar TVs
29. en the heat sink and the cooling air inlet eo 42 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 A 5 MCX353A FCBT Specifications Table 15 MCX353A FCBT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000 5 40GBASE R4 Protocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 71W Active Cables 8 41 WP Max Power Passive Cables 8 13W Power and Active Cables 10 13W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user manuals Mellanox approved cables pdf a T
30. he ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC2207310 0xx x For both operational and non operational states d Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet eo Mellanox Technologies 43 Rev 1 6 A 6 MCX354A FCBT Specifications Table 16 MCX354A FCBT Specifications Table Size 2 71in x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000 5 40GBASE R4 Protocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 7 94W Active Cables 11 34WP Max Power Passive Cables 9 35W Power and Active Cables 11 35W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com
31. hem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 57 Rev 1 6 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos gt 4 Conexion y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec tores y su desconexion se deben efectuar con mucho cuidado Para ver advertencias instrucciones especiales consultar al fabricante del cable gt 5 Instalaci n de equipos La instalacion el reemplazo y el mantenimiento de este equipo estaran a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conform
32. hysical layer using different block types deskew mechanism and framing rules 1 1 2 InfiniBand FDR10 FDR10 is an InfiniBand transport and link layer running over a physical layer using the PCS encoding of 64b 66b and a bit rate of 10 3125Gb s per lane Consequently the effective bandwidth is 25 higher than the InfiniBand specification standard QDR rate which uses an 8b 10b encod ing The FDR10 implementation on the adapter device can use the optional Forward Error Correc tion FEC supported by the FDR pre standard 10 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 1 1 3 InfiniBand QDR QDR is a standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10Gb s 1 1 4 10 and 40 Gigabit Ethernet 10 and 40 Gigabit Ethernet are Ethernet network protocol standards for LANs specifying data transfer rates of 10 and 40Gb s respectively Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet EEE Std 802 3ba 40 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover Mellanox Technologies 11 Rev 1 6 Overview 1 2 Adapter Cards Covered in this Manual Table 3 lists the single and dual port QDR adapter cards Table 4 lists the single and dual port FDR10 adapter cards Table 5 lists the single and dual port FDR adapter cards Table 3 Single and Dual port QDR
33. i binary image burn d Reboot the computer e Enter mst start f Verify that the card firmware was updated successfully gt flint d dev mst mt4099 pci Image type ConnectX FW Version 2 941100 Device ID 4099 38 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Appendix A Specifications A 1 MCX353A QCBT Specifications Table 11 MCX353A QCBT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP 40Gb s InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane Ethernet 10GBASE CX4 10GBASE R and 1000 5 Protocol Support Data Rate 40Gb s QDR InfiniBand 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 5 63 Active Cables 6 28W Max Power Passive Cables 6 61 W Power and Active Cables 7 36W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 PPlease refer
34. ications Table on page 40 Table 13 MCX353A TCBT Specifications Table on page 41 Table 14 MCX354A TCBT Specifications Table on page 42 Table 15 MCX353A FCBT Specifications Table on page 43 Table 16 MCX354A FCBT Specifications Table on page 44 January 2012 1 2 Minor edits Updated LED functions in Section 2 1 5 LED Assignment on page 19 Formatted specification tables in Appendix A Specifications on page 39 October 2011 1 1 Added new OPNs based on ConnectX 3 Step Al devices July 2011 1 0 Minor edits July 2011 0 10 First Release Mellanox Technologies 7 Rev 1 6 8 Mellanox Technologies Rev 1 6 About this Manual This User Manual describes Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP port PCI Express x8 adapter cards It provides details as to the interfaces of the board specifica tions required software and firmware for operating the board and relevant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with InfiniBand and Ethernet networks and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual User Manual describing the set of MFT firmware management tools for a Document no 2204UG single node See http www mellanox com gt Support gt
35. ined from gt Support gt Download Firmware Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware be updated on the stand alone single card using the flint tool of the Mellanox Firm ware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Support gt Download Firmware Tools A firmware binaries table lists a binary file per adapter card The file name of each such binary is composed by combining the firmware name the firmware release version and the card part num ber Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at http www mellanox com gt Products gt Software Drivers gt InfiniBand amp VPI Software Drivers gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the fo
36. l InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane Ethernet 10GBASE CX4 10GBASE R and 1000BASE R Protocol Support Data Rate 40Gb s FDR10 InfiniBand 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 84W Active Cables 8 14WP Max Power Passive Cables 7 82W Power and Active Cables 9 32W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a The ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC22062xx 0xx x For both operational and non operational states d Air flow is measured 1 from the heat sink betwe
37. low 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a The ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC22062xx 0xx x For both operationsal and non operational states d Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet eo 40 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 A 3 MCX353A TCBT Specifications Table 13 MCX353A TCBT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP 40Gb s InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1 2 4 SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane Ethernet 10GBASE CX4 10GBASE R 1000BASE R Protocol Support Data Rate 40Gb s FDR 10 InfiniBand 1 10Gb s Ethe
38. may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the procedure above step 1 52 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Figure 18 Placing the Bracket on the Card 3 Make sure that the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 lbs in torque on the screws Mellanox Technologies 53 Rev 1 6 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM LE cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les cables 4 Branchement d branchement des cables en cuivre en les branchant et en les d branchant des connecteurs Consultez le fabricant des Les cables en cuivre sont l
39. mp Intel R 5520 5500 X58 I O Hub GPIO and Scratch Pad Registers 3422 amp Intel R 5520 5500 X58 I O Hub PCI Express Root Port 1 3408 amp Intel R 5520 5500 X58 I O Hub PCI Express Root Port 2 3409 Intel R 5520 5500 58 I O Hub PCI Express Root Port 3 3404 amp Intel R 5520 5500 X58 Hub PCI Express Root Port 7 340E amp Intel R 5520 5500 58 I O Hub System Management Registers 342E amp Intel R 5520 5500 X58 I O Hub Throttle Registers 3438 Intel R 5520 5500 X58 Hub ta ESI Port 3406 amp Intel R 5520 5500 58 Physical and Link Layer Registers Port 0 3425 44K Intel R 5520 5500 X58 Routing and Protocol Layer Registers Port 0 3426 Intel R 5520 5500 58 Trusted Execution Technology Registers 342F amp Intel R 5520 58 I O Hub PCI Express Root Port 5 340C Intel R 82801 PCI Bridge 244E amp Intel R Chipset QuickData Technology device 3429 amp Intel R Chipset QuickData Technology device 342A amp Intel R Chipset QuickData Technology device 3428 88 Intel R Chipset QuickData Technology device 342 Intel R Chipset QuickData Technology device 3430 288 Intel R Chipset QuickData Technology device 3431 amp Intel R Chipset QuickData Technology device 3432 amp Intel R Chipset QuickData Technology device 3433 Intel R ICH10 Family PCI Express Root Port 1 3440 Intel R ICH10 Family PCI Express Root Port 5 3448 Intel R I
40. n VPI ports are auto sensing but can be manually configured using a script Table 6 lists the configurations supported by dual port VPI adapter cards Table 6 Supported Port Configurations Port 1 Port 2 Ethernet Ethernet IB IB auto sensing auto sensing IB Ethernet IB auto sensing auto sensing Ethernet Table 7 lists the configurations not supported by dual port VPI adapter cards Table 7 Unsupported Port Configurations Port 1 Port 2 Ethernet IB Ethernet auto sensing auto sensing IB For port type configuration instructions please refer to Section 4 2 Port Type Management Via Driver SW on page 32 2 1 4 Express Interface The ConnectX 3 adapter cards support PCI Express 3 0 1 1 and 2 0 compatible throughanx8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations 18 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 2 1 5 LED Assignment There are two I O LEDs per port in dual port designs See Figure 5 for different LED functions in InfiniBand mode and Table 8 for different LED functions in Ethernet mode Figure 5 Physical and Logical Link Indications InfiniBand Mode LED Function Green Physical Constant on indicates a good phys link ical link Blinking indicates a problem with the physical link Port 1 Ifneither LED is lit then the phys ical link has not been e
41. ngle Function port QSFP 4100 for Virtual Function MCX354A FCAT PCIe 3 0 x8 56 Gb s FDR IB R6 MT27508A0 FCCR FV Legacy product 8GT s 40GigE dual port QSFP MCX354A FCBT PCIe 3 0 x8 56 Gb s FDR IB R6 MT27508A1 FCCR FV 8GT s 40GigE dual port QSFP 1 3 Finding the GUID MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a label on the print side that shows the card serial number the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID Figure 1 Card Product Label Example Model No CX354A ConnectX 3 FDR InfiniBand 40GigE P N MCX354A FCAT Rev 1 O S N MT1106X01504 GUID 00 02 C9 03 00 27 05 1C MAC 00 02 C9 27 05 1C Made in Israel Port 1 uses the GUID or MAC described on the label To obtain the GUID or MAC for Port 2 add 1 to that of Port 1 1 4 Safety Warnings Below is a list of safety warnings in English For safety warnings in other languages please refer to the appendices in this user manual Mellanox Technologies 13 Rev 1 6 Overview 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 55 C 131 F To guarantee proper air flow allow
42. nox com gt Products gt Software Drivers gt InfiniBand amp VPI SW Driver gt Mellanox OFED GPUDirect gt Download 4 4 2 1 Kernel Installation Use Red Hat Package Manager RPM to install the Kernel RPM files Install the required RPMs for example run rpm foree rpm Modify the boot loader configuration file if needed e g edit etc grub conf Reboot the machine with the new kernel reboot 4 4 2 2 MLNX Drivers Installation The MLNX driver is called MLNX_OFED_LINUX Nvidia 1 5 1 Mount the ISO file mount o ro loop MLNX lt version gt iso mnt Run the installation script mnt mlnxofed install Restart the driver etc init d openibd restart To make sure that GPUDirect is enabled run cat sys module ib core parameters gpu direct enable gt Mellanox Technologies 35 Rev 1 6 Driver Software and Firmware The number of shared pages by GPUDirect is reported under cat sys module ib core parameters gpu direct shares 4 4 2 3 NVIDIA Driver Installation Install NVIDIA Development Driver for Linux x86 64 available under http developer nvidia com object cuda 3 0 downloads html For example R aes cevchilver 3 0 lilas 64 196 26015 cm Follow the installation wizard instructions To make sure that the NVIDIA driver was installed successfully Load nvidia driver modprobe nvidia Check the driver version for example
43. on the latch release mechanism thereby unlatching the connector and pull the connector out of the cage Do not apply torque to the connector when removing it from the adapter card 3 Remove any cable supports that were used to support the cable s weight Mellanox Technologies 27 Rev 1 6 Driver Software and Firmware 4 4 1 4 1 1 Driver Software and Firmware Driver Software For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox web site at http www mellanox com gt Products gt Soft ware Drivers gt InfiniBand amp VPI Software Drivers gt Mellanox OFED gt Download Follow the installation instructions included in the download package also available from the download page To ensure that communication has been established follow the instructions below Check the link status First check the network interface name by running the ifconfig a command gt ifconfig a etho Link encap Ethernet HWaddr 5C F3 FC 5F 40 EA ame 140 7 45 307 0 iexernEsdi0 7 75255 Wesike255 22623 5 0 inet6 addr fe80 5ef3 fcff fe5f 40ea 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 20009 errors 0 dropped 0 overruns 0 frame 0 TX packets 4492 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 7005902 6 6 MiB TX bytes 399581 390 2 KiB run ethtool interface
44. ourds et ne sont pas flexibles il faut donc faire tr s attention cables pour connaitre les mises en garde et les instructions sp ciales 5 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L elimination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 54 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 55 Rev 1 6 Appendix E Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Gerat an die Stromversorgung anschlieBen 2 Ubertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 55
45. r A 8 Regulatory Statements For regulatory statements for all ConnectX 3 cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf 46 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Appendix B Interface Connectors Pinout B 1 12 Connector Pinout Figure 12 Compatible Connector Plug and Pinout Table 17 I2C compatible Connector Pinout Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC B 2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Mellanox Technologies 47 revi fo Figure 13 PCle Connector Pinout GND PERpO PERMO GND RSVD A20 1 HAs Pernt HAs A23 T GND ads 2 AS 2 A GND PERp3 A ra RSVD Aie any Rsvp AS GND PERS ang PERNA GND 31 22 Th PERPS aan PERNS 42 GND Y T PERTE 7 GNO Pag 7 PERN7 A48 GND 48 Mellanox Technologies Rev 1 6 ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual QSFP Connector Pinout B 3 Connector and Cage Views Figure 14 18 35 Top View into Rea
46. r of Module GND Tp GND TX3n TX3p GND LPMode Vcc1 VccTx IntL ModPrsL GND RX4p RX4n GND RX2p RX2n GND GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL SCL SDA GND RX3p RX3n GND RX1p RX1n GND 18 35 To View into Front of Cage QN9 sid POW UEXY dexy AND vas 195 XH99A q s y AND AND dzXL ucxL AND 49 Mellanox Technologies Table 18 Connector Pin Number and Name to Signal Name Map Rev 1 6 Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN 2 Tx2n 3 2 2 4 GND GND 5 TXN 4 Tx4n 6 4 Tx4p 7 GND GND 8 ModSelL_Port0 ModSelL 9 ResetL_Port0 ResetL 10 11 SCL SCL 12 SDA SDA 13 GND GND 14 3 Rx3p 15 RXN 3 Rx3n 16 GND GND 17 1 18 RXN_1 19 GND GND 20 GND GND 21 RXN 2 Rx2n 22 2 Rx2p 23 GND GND 24 RXN 4 Rx4n 25 4 Rx4p 26 GND GND 27 ModPrsl Port0 Mod PrsL 28 IntL IntL 29 VecTx 30 31 LPMode_Port0 LPMode 32 GND GND 33 TXP_3 Tx3p 34 TXN_3 Tx3n 35 GND GND 36 1 37 TXN 1 38 GND GND 50 Mellanox Technologies ConnectX 3 VPI Single
47. rm dev mst mt4099 pci cr0 d Get the PSID firmware identification and programmed firmware version using the command gt flint 4 dev mst mt4099 pci Image type ConnectX FW Version 2 9 4000 Device ID 4099 Chip Revision 0 Description Node Porti Port2 Sys image GUIDs 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT_1020110019 VSD PSID MT_1020110019 2 Compare the programmed firmware version with the latest available Mellanox Technologies 37 Rev 1 6 Driver Software and Firmware a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 9 b Enter your card PSID to display the latest firmware Figure 9 Support Download Assistant Mellanox TECHNOLOGIES Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home MT_1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Se Switches Gateways 3 If a newer firmware version exists for your adapter card update the firmware as follows a Download the firmware image zip file from the Download Center see Step 2a above b Unzip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci
48. rnet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 5 69W Active Cables 6 34W Max Power Passive Cables 6 67W Power and Active Cables 7 42W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a The ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC22062xx 0xx x For both operational and non operational states d Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet eo Mellanox Technologies 41 Rev 1 6 A 4 MCX354A TCBT Specifications Table 14 MCX354A TCBT Specifications Table Size 2 71n x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP 40Gb s InfiniBand Copper and optica
49. source Type VPD W 0x91 Tag 0x11 110 Length 7 0 LSB Ox8F 111 Length 15 8 MSB OxF 112 VPD Keyword VI STR EFI Driver version 114 Length 0x6 115 Data N A STR_SPC 121 VPD Keyword YA STR Asset Tag 123 Length 0x18 124 Data N A STR_SPC N A 148 VPD Keyword RW STR Remaining read write area 150 Length 0x69 151 Data STR_ZERO Reserved 0x00 256 VPD Keyword RW STR Remaining read write area 258 Length Oxfd 259 Data STR_ZERO Reserved 0x00 512 VPD Keyword RW STR Remaining read write area 514 Length Oxfd 515 Data STR_ZERO Reserved 0x00 768 VPD Keyword RW STR Remaining read write area 770 Length Oxfd 771 Data STR_ZERO Reserved 0x00 1024 VPD Keyword RW STR Remaining read write area 1026 Length Oxfd 1027 Data STR_ZERO Reserved 0x00 1280 VPD Keyword RW STR Remaining read write area 1282 Length Oxfd 1283 Data STR_ZERO Reserved 0x00 1536 VPD Keyword RW STR Remaining read write area 1538 Length Oxfd 1539 Data STR_ZERO Reserved 0x00 ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 Table 10 MCX35 34 A QFT CBT VPD Layout aes Item Value Format Description 1792 VPD Keyword RW STR Remaining read write area 1794 Length Oxfd 1795 Data STR_ZERO Reserved 0x00 2048 VPD Keyword RW STR Remaining read write area 2050 Length Oxfd 2051 Data STR_ZERO Reserved 0x00 2304 VPD
50. st of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 ha 3 3 2 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt lspci grep i Mellanox 27 00 0 Network controller Mellanox Technologies MT27500 Fam ily ConnectX 3 Mellanox Technologies 25 Rev 1 6 Adapter Card Installation 3 4 Cables and Modules Please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user manuals Mellanox approved cables pdf 3 4 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on To insert a cable press the connector into the port receptacle until the connector is firmly seated 2 After inserting a cable into a port the GREEN LED indicator will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port 3 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the YELLOW LED will come on When data is being transferred the yellow led will blink When installing cables make sure that the latches engage gt Always install and remove cables by pushing or pulling the cable and connector in a straight line
51. stablished i Yellow A constant yellow indicates a valid Logical data logical data activity link without activity link data transfer A blinking yellow indicates a valid logical link with data transfer Ifonly the green LED is lit and the yellow LED is off then the logical link has not been established Table 8 Physical and Logical Link Indication Ethernet Mode LED Function Green physical link Constant on indicates a good physical link Ifneither LED is lit then the physical link has not been established Yellow A blinking yellow indicates activity data transfer logical data activity Stays off when there is no activity link Mellanox Technologies 19 Rev 1 6 Adapter Card Interfaces gt The short bracket has the same port and LED footprints as the tall bracket 2 1 6 12C compatible Interface A three pin header on the adapter cards is provided as the I C compatible interface See Figure 11 Mechanical Drawing of the Dual port MCX354A QFT CBT Adapter Card on page 46 for the location on the board Figure 6 Connector 2 2 Power All adapter cards receive 12V and 3 3V power from the PCI Express edge connector All other required power voltages are generated by on board switch mode regulators See Specifications on page 39 2 3 QSFP Power Level The adapter cards support power levels according
52. to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a The ConnectX 3 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox InfiniBand product b These power numbers were measured using Mellanox cable MC22062xx 0xx x For both operational and non operational states d Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet e Mellanox Technologies 39 Rev 1 6 A2 MCX354A QCBT Specifications Table 12 MCX354A QCBT Specifications Table Size 2 71n x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP 40Gb s InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane Ethernet 10GBASE CX4 10GBASE R and 1000BASE R Protocol Support Data Rate 40Gb s QDR InfiniBand 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 78W Active Cables 8 08W Max Power Passive Cables 7 76W Power and Active Cables 9 26W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air F
53. to SFF Committee SFF 84366 Specification for QSFP Quad Small Form factor Pluggable Transceiver 2 4 Memory The adapter cards support multiple memory devices through the PCIe SPI Flash and inter faces 2 4 1 System Memory The adapter cards utilize the PCI Express interface to store and access InfiniBand and or Ethernet fabric connection information and packet data on the system memory 2 4 2 SPI Each of the adapter cards includes one 16MB SPI Flash device 25 16 device by ST Microelectronics accessible via the SPI interface of the ConnectX 3 VPI device 20 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 When a jumper is inserted into the drill holes in the adapter it indicates to the device that the on board Flash device should not be used for boot instructions Contact your Mellanox support repre sentative should you need to use this jumper Table 9 provides information on this jumper The jumper location on the board is illustrated in Appendix A 7 Board Mechanical Drawing and Dimensions on page 45 Table 9 Jumper Configuration Card Default n escription Option Flash present connection open Flash present connection open not present Flash present connection shorted Flash not present 2 4 3 EEPROM Each board incorporates an EEPROM that is accessible through the PC compatibl
54. ules are active For example Edit etc infiniband openib conf which modules needs to load from a service 4 Load MLX4 EN module 30 Mellanox Technologies ConnectX 3 VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 6 MLX4 EN LOAD yes The Usage openibd start stop restart status command to modify this file and thereby control the drivers Ethernet Driver Usage and Configuration To assign an IP address to the interface run where n is the OS assigned interface number gt ifconfig eth n ip To check driver and device information run d ethtool i eth lt n gt Example GENEL i GENO driver mlx4 en MT_1020110019_CX 3 Versions 1 5 6 323 ei 2014 firmware version 2 10 0000 bus info 0000 07 00 0 The mlx4 en parameters can be found under sys module mlx4 en or sys module mlx4 en parameters depending on the OS and can be listed using the command d modinfo mlx4 en To set non default values to module parameters the following line should be added to the file etc modprobe conf options mlx4 en lt param_name gt lt value gt lt param_name gt lt value gt Ethernet Network Tuning To improve network performance by tuning your network see the Mellanox Performance Tuning Guide located at http www mellanox com gt Products gt Software Drivers gt Ethernet SW Drivers gt Linux Driver Mellanox Technologies 31 Rev 1 6 Driver Softwar

User Manual - Advanced HPC

Contents

Download Pdf Manuals

Related Search

Related Contents