Mellanox Technologies
Contents
1. which is organized under the Ministry of Information and Communications MIC EMC testing includes electromag netic emissions EMI and susceptibility EMS Certified equipment is labeled with the MIC mark and certification number Class A Statement 9 77S Si 9 AAW SER ERR 9b 71719 9 4 BHA Xu aeae ol BE 7 2 3 7 pep bep um tm TE FYBRS wor AELE REA gu Translation Class A Device This device is registered for EMC requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced with a residential use type Rev 0 11 d Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual A 3 MHEH28 XSC and MHEH28 XTC Specifications Table 14 Specifications for MHEH28 XSC XTC Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3V 200LFM 55 C Typ Power 11 01W Air Flow InfiniBand Copper current rating 0 5A Maximum Power 12 62W 4X 10Gb s Connector max with active media adapter support Temperature 0 C to 55 C Protocol Support Regulatory InfiniBand IBTA v1 2 Auto Negotiation 10Gb s MIC Certification 2 5Gb s Safety IEC EN 60950 1 2001 QoS 8 InfiniBand Virtual Lanes for each port ETSI EN 300 019 2 2 RDMA Support Yes All Ports Data Rate Single Environmental IEC 60068 2 64 29 32 PCI2. 3 Gently place the label as shown in Figure 13 Make sure to align the center hole of the label with the drilled hole in the board Figure 13 Place Label on Print Side With Label s and Card s Holes Aligned 4 After placing the label complete the process by gently sweeping your thumb on the label to assure the label is well attached onto board See Figure 14 Figure 14 Ensure That Label is Well attached See also Figure 22 Print Side View After Short Bracket Assembly With Kapton Label on page 36 Now your card is ready for a short bracket assembly Rev 0 11 a Mellanox Technologies Confidential ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 33 C 3 Assembling Short Bracket The short bracket can now be assembled onto the HCA card See Figure 15 Figure 15 HCA Card Ready for Short Bracket Step 1 Place short bracket onto card 5 Gently place the bracket onto the card fitting the connectors through the bracket connector holes Make sure the LEDs are aligned into their intended bracket holes Step 2 Attach Figure 16 Place Short Bracket onto Card LED Holes Mellanox Technologies Confidential Rev 0 11 34 short bracket to cardInsert a screw along with a washer into each of the two holes on the card intended for holding the bracket Use a torque screwdriver to apply up to 2 Ibs in torque on each screw Figure 17 Attach Bracket onto Card using Screws
3. 4 4 1 System Memory Each of the HCA adapter cards utilizes the PCI Express interface to store and access IB fabric connection information and packet date on the system memory 4 4 2 Flash Each of the HCA adapter cards includes one 2MB SPI Flash device P N M25P16 VME6G by ST Microelectronics accessible via the Flash interface of the MT25408 ConnectX IB device There is a jumper on each adapter card that indicates to the device whether an on board Flash device exists or is to be used Table 7 provides information on this jumper See Figure 4 on page 22 for the jumper location Table 7 Jumper Configuration Card Default Comments Configuration Description Option Flash present not present connection open Flash present Header 1x2 connection shorted Flash not present connection open Flash present CLOSED FLASH NOT PRESENT 4 4 5 EEPROM Each board incorporates an EEPROM that is accessible through the I2C compatible interface The EEPROM is used for storing the Vital Product Data VPD The VPD format adheres to the PCI Local Bus specification rev 2 3 VPD definition see VPDs on page 15 The EEPROM capacity is 512 bytes 4 5 VPDs The PCI VPD Vital Product Data layout for each of the described Mellanox Technologies Connectx TM IB HCA Adapter cardscomply with the format defined in the PC7 2 3 A1 was used as the HCA card PCB revi sion Later revisions of the HCA
4. Note A1 was used as the HCA card PCB revision Later revisions of the HCA card will have the same format Table 9 VPD Layout for MHEH28 XTC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHEH28 XTC STR_SPC Rev 0 11 Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 9 VPD Layout for MHEH28 XTC Continued I D Item Value Format Description 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCle x8 STR_SPC 90 VPD Keyword RV STR 92 Length Ox1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 10
5. 2 EMC Certification Statements Table 13 lists the approved EMC certification status per HCA card in different regions of the world Table 13 HCA Cards EMC Certification Status HCA Card P N FCC Class USA EN Class Europe ICES Class Canada MHEH28 XSC A A A MHEH28 XTC A A A MHGH28 XSC Rev 0 11 Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 13 HCA Cards EMC Certification Status HCA Card P N FCC Class USA EN Class Europe ICES Class Canada MHGH28 XTC A A A MHGH29 XTC TBD TBD TBD A 2 1 FCC A 2 2 Statements USA Class A Statements 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compli ance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful inter ference to radio communications Operation of this equipment
6. Nnm Gams Step 3 Install Connector Clips 1 Gently push one clip onto the connector Make sure to slide both clip hooks sides around the connector evenly as shown in Figure 18 Figure 18 Sliding Connector Clip Evenly Rev 0 11 Mellanox Technologies Confidential ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 35 2 Usea small flat head screwdriver to gently slide the clip s hook towards the connector s base side as shown in Figure 19 Figure 19 Fix Clip Hooks into Place Using Screwdriver 3 Repeat this step for the second clip See Figure 21 for the assembled short bracket side view and Figure 22 for the Print Side view showing the Kapton label Figure 20 Assembled Short Bracket View Mellanox Technologies Confidential Rev 0 11 36 Figure 22 Print Side View After Short Bracket Assembly With Kapton Label Rev 0 11 Mellanox Technologies Confidential
7. Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 29 Appendix C Replacing a Tall Bracket with a Short Bracket on HCA Cards This appendix provides instructions on how to remove a tall bracket of a Mellanox Technologies HCA card and replace it with a short one It includes the following sections e Removing Tall Bracket e Placing a Kapton Polyimide Label e Assembling Short Bracket Figure 7 shows the bracket side view of a dual port HCA card Figure 7 Tall Bracket of a Dual IB Port HCA Card C 1 Removing Tall Bracket Step 1 Remove connector clips Figure 8 shows a connector retention clip and the designated names of its sec tions Figure 8 Connector Retention Clip Clip Top Clip Center embraces the connector Clip Hook grips onto the connector Usinga small flat head screwdriver gently push up one hook of a connector s clip toward the connector s top side as shown in Figure 9 a on page 30 2 Then push the other hook each of the two clip s hook towards the connector s top side see Figure 9 b Finally pull the clip away from its center see Figure 9 c Mellanox Technologies Confidential Rev 0 11 30 Figure 9 Extracting Connector Clip a Gently Push One Hook of Clip b Gently Push Other Hook of Clip c Pull Clip Away 3 Repeat the above actions for the second connector s clip Step 2 Unscrew bracket screws 1 Unscrew both screws f
8. card will have the same format Table 8 VPD Layout for MHEH28 XSC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHEH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 Mellanox Technologies Confidential Rev 0 11 16 Table 8 VPD Layout for MHEH28 XSC Continued en D Item Value Format Description 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00X XXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCle x8 STR_SPC 90 VPD Keyword RV STR 92 Length Ox1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR_SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78
9. mellanox com under Firmware downloads InfiniBand Administration IBADM Package User s Manual Document no 2130UM User s Manual describing the utilities included in the IBADM tools package for system administration of an InfiniBand cluster See http www mellanox com under Management Tools IB Specifications Release 1 0 a PCI Express 2 0 Specifications Infiniband Architecture Specifications Industry Standard PCI Express 2 0 Card Electromechanical Specifica tion Rev 1 3 Online Resources Mellanox Technologies Web pages http www mellanox com Mellanox Technologies Firmware download Web page http www mellanox com under Firmware downloads Mellanox Technologies Document Distribution System DDS http docs mellanox com requires a customer login account Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits Mellanox Technologies Confidential Rev 0 11 8 1 Overview This document is a User 5 Manual for Mellanox Technologies host channel adapter HCA Cards based on the MT25408 ConnectX IB integrated circuit device The cards described in this manual have the following main fea tures IBTA v1 2 compliant Two 4X InfiniBand copper ports for connecting InfiniBand traffic 4X IB connectors Two 1OGBASE CX4 copper p
10. 9 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Note A1 was used as the HCA card PCB revision Later revisions of the HCA card will have the same format Table 10 VPD Layout for MHGH28 XSC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number Rev 0 11 Mellanox Technologies Confidential 18 Table 10 VPD Layout for MHGH28 XSC Continued a D Item Value Format Description 17 Length 0x15 18 PN MHGH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length Ox1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver ver
11. ANN Mellanox TECHNOLOGIES ConnectX M IB Dual Port InfiniBandAdapter Cards User s Manual P N MHEH28 XSC MHEH28 XTC MHGH28 XSC MHGH28 XTC MHGH29 XTC Rev 0 11 Mellanox Technologies Confidential O Copyright 2007 Mellanox Technologies Inc All Rights Reserved Mellanox Technologies InfiniHost and ConnectX are registered trademarks for Mellanox Technologies Inc All other marks and names mentioned herein may be trademarks of their respective companies ConnectX IBDual Port InfiniBandAdapter Cards With PCI Express User s Manual Document Number 2802 Mellanox Technologies Inc 2900 Stender Way Santa Clara CA 95054 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam 20692 Israel Tel 972 4 909 7200 Fax 972 4 959 3245 Rev 0 11 Mellanox Technologies Confidential ConnectX InfiniBand Dual Port HCA Adapter Cards With PCI Express x8 User s Manual Table of Contents Revision History About this Manual Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 AppendixA AppendixB AppendixC Overview 1 1 Adapter Cards 1 2 Mellanox Part Numbering Legend HCA Card Installation 2 1 Hardware and Software Requirements 2 2 Installation Instructions Driver Software and Firmware 3 1 Driver Software 3 2 Updating HCA Card Firmware 3 3 Single HCA Card Firmware Update 3 4 HCA Card Firmware Update a
12. Express 2 0 SERDES 2 5 GT s RoHS RoHS R5 Mellanox Technologies Confidential Rev 0 11 26 A 4 MHGH28 XSC and MHGH28 XTC Specifications Table 15 Specifications for MHGH28 XSC XTC Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3V 200LFM 55 C Typ Power 11 48W Air Flow InfiniBand Copper current rating 0 5A Maximum Power 13 09W 4X 20Gb s Connector max with active media adapter support Temperature 09C to 55 C Protocol Support Regulatory InfiniBand BTA v1 2 Auto Negotiation 20Gb s EMC FCC 47 CFR part 15 2005 subpart B class 5Gb s or 10Gb s 2 5Gb s A QoS 8 InfiniBand Virtual Lanes for each port RDMA Support yes All Ports ICES 003 2004 Issue 4 class A Double Data Rate PCI Express 2 0 SERDES 2 5 GT s VCCI V 3 2005 04 class A EN 55022 1998 A1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A1 2001 EN 55024 1998 A1 2001 A2 2003 stan dards harmonized under EMC Directive 89 336 EEC MIC Certification Safety IEC EN 60950 1 2001 ETSI EN 300 019 2 2 Environmental IEC 60068 2 64 29 32 RoHS RoHS R5 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specification Release 1 2 Rev 0 11 Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards Wit
13. PCle x8 STR_SPC 90 VPD Keyword RV STR 92 Length Ox1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 12 VPD Layout for MHGH29 XTC a Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle DDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 Rev 0 11 Mellanox Technologies Confidential 20 Table 12 VPD Layout for MHGH29 XTC Continued Pu Item Value Format Description 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHGH29 STR_SPC XSC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00X XXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCle Gen2 STR_SPC x8 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 96CSO 94 Large Resource Typ
14. See http www mellanox com under Firmware downloads A firmware binaries table lists a binary file per HCA card The file name of each such binary is composed by combin ing the firmware name the firmware release version and the card part number Note Please contact your assigned Field Application Engineer if you cannot find the firmware binary for your adapter card This may happen if the product is not yet available for general distribution 3 4 HCA Card Firmware Update as Part of a Cluster Firmware Update If the HCA card is part of an InfiniBand cluster its firmware can be updated as part of the entire cluster firmware update using the ibfwmer tool of the IB administration IBADM tools package IBADM is available for download from http www mellanox com through the Management Tools download link Check the Firmware downloads link for cluster update instructions 1 Currently only the Linux distributions support updating firmware for an entire InfiniBand cluster Rev 0 11 Mellanox Technologies Confidential ConnectX BDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 4 Adapter Card Interfaces 4 1 I O Interfaces Each HCA card includes the following interfaces Two 4X InfiniBand copper connectors PCI Express x8 edge connector O panel LEDs Pc compatible connector for debug 4 1 1 InfiniBand CXA Interface The ConnectX TM B MT25408 device is compliant with t
15. al 13 14 14 22 27 27 29 29 30 30 31 31 32 32 33 33 34 34 35 35 35 36 Rev 0 11 5 6 Revision History This document was printed on 2 4 08 Table 1 Revision History Table Date Rev Comments Changes May 2007 0 10 Preliminary first revision Rev 0 11 Mellanox Technologies Confidential ConnectX InfiniBand Dual Port HCA Adapter Cards With PCI Express x8 User s Manual About this Manual This User s Manual describes Mellanox Technologies ConnectX B 10GBase CX4 Ethernet PCI Express HCA cards It provides details as to the interfaces of the card specifications required software and firmware for operating the card and relevant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Infiniband networks Related Documentation Table 2 Documents List ConnectX IB Programmer 5 Reference Manual Document no 2623PM Reference describing the interface used by developers to write a device driver ConnectX 1B MT25208 Hardware Reference Manual Document no 2715HM Reference for hardware engineers responsible for designing systems and boards incorporating ConnectXTM IB components Mellanox Firmware Tools MFT User s Manual Document no 2204UG User s Manual describing the set of MFT firmware management tools for a single InfiniBand node See http www
16. e VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Rev 0 11 Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 5 Connectivity This adapter card can be connected to switches and routers using 20 meters of passive copper CX4 cables for SDR Adapter cards and 10 meters of passive copper CX4 cables for DDR Adapter cards These lengths are greatly increased with active copper CX4 cables or with fiber cables with external adapters Mellanox Technologies Confidential Rev 0 11 21 22 Appendix A Specifications A 1 Board Mechanical Drawing and Dimensions All the HCA cards covered in this User 5 Manual have the same mechanical drawing and share the same dimensions as depicted in Figure 4 Note All dimensions are in millimeters Figure 4 Schematic of the InfiniBand Adapter Card With CX4 Connectors 136 47 CD WEN HI um y emasa 5 T9 ERS SS GOS psp D ammm B KS gia i TT e 8b ima DS UZ U a TH P 172p s i Ces J5 I2C Connector J6 Flash Jumper A
17. ecifications for MHGH28 XSC XTC Specifications for MHGH29 XSC XTC I2C compatible Connector Pinout Connector Pin To Port Signal Name Mellanox Technologies Confidential 0 Nl A 14 15 15 16 17 18 19 22 25 25 26 27 28 ConnectX InfiniBand Dual Port HCA Adapter Cards With PCI Express x8 User s Manual List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 Figure 18 Figure 19 Figure 20 Figure 21 Figure 22 Figure 23 MHGH28 XTC Port Numbering Physical and Logical Link Indications I2C Connector Schematic of the InfiniBand Adapter Card With CX4 Connectors I2C compatible Connector ConnectX CX4 Copper Connector Pinout Tall Bracket of a Dual IB Port HCA Card Connector Retention Clip Extracting Connector Clip Unscrew Bracket Screws Rotate the Bracket to Detach it From the Card Hold Kapton Label With Pincers Place Label on Print Side With Label s and Card s Holes Aligned Ensure That Label is Well attached HCA Card Ready for Short Bracket Place Short Bracket onto Card Attach Bracket onto Card using Screws Sliding Connector Clip Evenly Fix Clip Hooks into Place Using Screwdriver Assembled Short Bracket View Assembled Short Bracket View Print Side View After Short Bracket Assembly With Kapton Label Mellanox Technologies Confidenti
18. ements Before installing the HCA Adapter card please make sure that the system meets the hardware and software require ments listed in Table 5 Table 5 Hardware and Software Requirements Requirement Description Hardware PCI Express x8 slot or x4 slot with x8 connector PCI Express x8 or x16 slots Software For Windows see WinIB ReadMe at https docs mellanox com dm WinIB ReadMe html Operating For Linux see OpenFabrics Enterprise Distribution OFED software package available via the OpenFab Systems Distributions rics Web site http www openfabrics org 2 2 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 2 1 Safety Warnings Over temperature The HCA Adapter cardshould not be operated in an area with an ambient temperature exceeding the maximum recommended temperature of 55 C Moreover it requires an airflow velocity of 200LFM linear feet per minute at this maximum ambient temperature During Lightning During periods of lightning activity do not work on the equipment or connect or disconnect cables Copper InfiniBand Cable Connecting Disconnecting Copper InfiniB cables are heavy and not very flexible As such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warnings instructions Rev 0 11 a Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Ca
19. h PCI Express x8 User s Manual A 5 MHGH29 XTC Specifications Table 16 Specifications for MHGH29 XSC XTC Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3V 200LFM 55 C Typ Power 12 6W Air Flow InfiniBand Copper current rating 0 5A Maximum Power TBD W 4X 20Gb s Connector max with active media adapter support Temperature 0 C to 55 C Protocol Support Regulatory InfiniBand IBTA v1 2 Auto Negotiation 20Gb s EMC TBD 5Gb s or 10Gb s 2 5Gb s E QoS 8 InfiniBand Virtual Lanes for each port RDMA Support Yes All Ports Double Data Rate PCI Express 20 SERDES 5 0 GT s Safety Environmental RoHS 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specification Release 1 2 Mellanox Technologies Confidential Rev 0 11 ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 27 Appendix B Interface Connectors Pinout B 1 IZC compatible Connector Pinout Figure 5 I2C compatible Connector Table 1 I2C compatible Connector Pinout A Connector Pin Number 1 HCA Signal Name SPSDA N A 0o en f SPSCL GND NC NC B 2 InfiniBand Connector Pinout Figure 6 ConnectX CX4 Copper C
20. he InfiniBand Architecture Specification Release 1 2 It has two compliant 4X InfiniBand ports 1 and 2 each having four Tx Rx pairs of SerDes Each of the HCA cards listed in Table 3 on page 8 based on this device provides access to these ports by means of two 4X InfiniBand con nectors for external InfiniBand copper cables also compliant with the IBTA specification 1 2 Connector 1 connects to port 1 of the device while connector 2 connects to port 2 Figure 2 Port Numbering Port 1 Port 2 4 1 2 Each of the HCA cards is embedded with a media detect circuit that supports active cables and external InfiniBand fiber solutions to be connected to the InfiniBand port connectors Fiber Solutions require the use of active media converters Mellanox Technologies Confidential Rev 0 11 14 4 1 3 PCI Express Interface The ConnectX M IB HCA Adapter cards support the PCI Express 2 0 x8 interface 1 1 compatible The card can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations 4 1 4 LED Assignment The board has four LEDs located on the I O panel 2 LEDs per 4X port The green LED when lit indicates that the InfiniBand driver is running and a valid physical connection between nodes exists The yellow LED when lit indi cates a valid data activity link this is the logical link The yellow LED illuminates when the InfiniBand network is discovered over the physical link A valid da
21. in a residential area is likely to cause harmful interfer ence in which case the user will be required to correct the interference at his own expense A 2 3 EN Statements Europe EN55022 Class A Statement RF Emissions Control Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures A 2 4 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada Mellanox Technologies Confidential Rev 0 11 23 24 A 2 5 VCCI Statements Japan Class A Statement OB FHRUBRESERESAEMABBS VCCI OBE dO 72AAT amp sSRPHIIEIEE CY CORE TREERE CHATS CBR Gees AROTFCEPHVETF COBRCMsHAAPBIGHReBS SEFBKANSATCEPHV ET Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 2 6 MIC Statement Republic of Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and cer tification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL
22. onnector Pinout J2 Conn 1X S10 iBixop 3 IBtxIp 3 S9 IBixOn 3 IBtxin 3 S S12 IBxOp 2 IBbip 2 25 S11 BwOn 2 IBtxin 2 22 S14 iBtop 1 IBtxip 1 S3 13 iBtxOn 1 IBbdn 1 S16 IBtxOp 0 IBip 0 S S15 IBtxOn 0 IBtxin 0 2 Mellanox Technologies Confidential 28 B 3 PCI Express x8 Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the Table 2 Connector Pin To Port Signal Name Connector Pin Connector Pin IB Port A Signal IB Port B Signal Number Name Name Name S1 IBtxIp 0 Rx_Al Rx Bl S2 IBtxIn 0 Rx AO Rx B0 S3 IBtxIp 1 Rx A3 Rx B3 S4 IBtxIn 1 Rx A2 Rx B2 S5 IBtxIp 2 Rx A5 Rx B5 S6 IBtxIn 2 Rx A4 Rx B4 S7 IBtxIp 3 Rx A7 Rx B7 S8 IBtxIn 3 Rx A6 Rx B6 S9 IBtxOn 3 Tx A6 Tx B6 S10 IBtxOp 3 Tx A7 Tx B7 S11 IBtxOn 2 Tx_A4 Tx B4 S12 IBtxOp 2 Tx_A5 Tx B5 S13 IBtxOn 1 Tx_A2 Tx_B2 S14 IBtxOp 1 Tx A3 Tx B3 S15 IBtxOn 0 Tx A0 Tx B0 S16 IBtxOp 0 Tx_Al Tx_Bl G1 G6 G9 H1 H2 Signal Ground GND GND G7 Sense 3 3V SENSE PI SENSE P2 G8 Vec MC POWER PI MC POWER P2 1 The Sense 3 3V signal is used to enable the Vcc power supply pin G8 used to provide power to the active media adapter PCI Express 2 0 specification Rev 0 11 Mellanox Technologies Confidential ConnectX IB Dual
23. orts for connecting Ethernet traffic PCI Express 2 0 1 1 compatible through an x8 edge connector up to 5GT s Media detect circuit with powered connectors supporting the use of active cables and external PHY fiber solu tions EU Restriction of Hazardous Substances RoHS compliant The cards differ in IB10Gb s SDR or 20Gb s DDR Bracket height short or tall PCI Express 2 0 with SerDes speed 2 5 GT s or 5 0 GT s 1 1 Adapter Cards Table 3 lists the InfiniBand HCA cards described in this manual Table 3 HCA Cards PCI hort RoH Ordering Part Express IB SDR es 4 ES a HCA IC a Number OPN SERDES DDR p Part Number HCA Card Photo Bracket ce Speed MHEH28 XSC 2 5 GT s SDR Short RoHS R5 MT25408A0 FCC Figure 1 MHGH28 XTC with SI exemp tion MHEH28 XTC 2 5 GT s Tall MHGH28 XSC 2 5 GT s DDR Short RoHS R5 MT25408A0 FCC with DI exemp tion MHGH28 XTC 2 5 GT s Tall Short Tall Bracket Bracket MHGH29 XTC 5 0 GT s DDR Tall MT25408A0 FCC XSC XTC GI 1 The HCA cards have a similar form and fit The main visible difference is in the bracket height Rev 0 11 a Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 1 2 Mellanox Part Numbering Legend Table 4 describes the Mellanox Technologies adapter cards part numbering legend Table 4 Mellanox HCA Cards Part Numbering Ke
24. rds With PCI Express x8 User s Manual Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regulations 2 2 2 Installation Instructions The adapter cards listed in Table 3 on page 8 are standard PCI Express x8 cards each with a standard x8 edge connec tor Please consult the host machine documentation for instructions on how to install a PCI Express card Mellanox Technologies Confidential Rev 0 11 M 12 3 Driver Software and Firmware 3 1 Driver Software For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the OpenFabrics Web site at http www openfabrics org Follow the installation instructions included in the download package For Windows download the appropriate software from https docs mellanox com dm WinIB ReadMe html 3 2 Updating HCA Card Firmware Each HCA card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com through the Firmware downloads link HCA InfiniBand cluster firmware update 3 3 Single HCA Card Firmware Update Firmware can be updated on the standalone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user s manual from the single HCA card firmware update page
25. rom the card using a torque screwdriver as shown in Figure 10 Figure 10 Unscrew Bracket Screws Step 3 Detach bracket Grip the bracket as shown in Figure 11 placing your thumb on the LED component In a rotating move toward the component side of the card slide the bracket out of the connectors Figure 11 c 1 2 3 Gently hold your thumb on the LED component 4 At the same time extract the bracket as shown in Figure 11 c Make sure to protect the LED while extracting the bracket Rev 0 11 a Mellanox Technologies Confidential ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Figure 11 Rotate the Bracket to Detach it From the Card a Card without Clips and Screws b Grip the Card in preparation for Detachment c Rotate the bracket toward the Component Side C 2 Placing a Kapton Polyimide Label Prior to assembling the short bracket you need to apply a Kapton polyimide round label on the board s Print Side Note Check to see if the label is already installed as this label may have been installed at the factory Note The label can be provided by Mellanox Technologies P N MEC000821 The following steps are instructions for placing the polyimide label 1 Make sure your working area is ESD protected 2 Hold the label with light pincers See Figure 12 Figure 12 Hold Kapton Label With Pincers Rev 0 11 Mellanox Technologies Confidential 32
26. s Part of a Cluster Firmware Update Adapter Card Interfaces 4 1 I O Interfaces 4 2 DC Compatible Interface 4 3 Power 4 4 Memory 4 5 VPDs Connectivity Specifications A l Board Mechanical Drawing and Dimensions A 2 EMC Certification Statements A 3 MHEH28 XSC and MHEH28 XTC Specifications A 4 MHGH28 XSC and MHGH28 XTC Specifications A 5 MHGH29 XTC Specifications Interface Connectors Pinout B 1 I2C compatible Connector Pinout B 2 InfiniBand Connector Pinout B 3 PCI Express x8 Connector Pinout Replacing a Tall Bracket with a Short Bracket on HCA Cards C 1 Removing Tall Bracket C 2 Placing a Kapton Polyimide Label C 3 Assembling Short Bracket Mellanox Technologies Confidential coco 00 N A 10 10 12 12 12 12 12 13 13 14 14 14 15 21 22 22 22 25 25 26 27 21 27 28 29 29 31 33 Rev 0 11 4 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Rev 0 11 Revision History Table Documents List HCA Cards Mellanox HCA Cards Part Numbering Key Hardware and Software Requirements LEDs Jumper Configuration VPD Layout for MHEH28 XSC VPD Layout for MHEH28 XTC VPD Layout for MHGH28 XSC VPD Layout for MHGH28 XTC VPD Layout for MHGH29 XTC HCA Cards EMC Certification Status Specifications for MHEH28 XSC XTC Sp
27. sion 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Note A1 was used as the HCA card PCB revision Later revisions of the HCA card will have the same format Table 11 VPD Layout for MHGH28 XTC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F Rev 0 11 Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 11 VPD Layout for MHGH28 XTC Continued E D Item Value Format Description 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHGH28 XTC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00X XXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data
28. ta activity link without data transfer is designated by a constant yellow LED indication A valid data activity link with data transfer is designated by a blinking yellow LED indication If the LEDs are not active either the physical link or the logical link or both connections have not been established Table 6 LEDs Port Number LED Name Figure 3 Physical and Logical Link Indications Port 1 Physical Link Green Data Activity Yellow Port 1 Blinking indicates Data Transfer Constant on indicates no Data Transfer Port 2 Physical Link Green Data Activity Yellow Blinking indicates Data Transfer Constant on indicates no Data Transfer Note The short bracket has the same port and LED footprint as the tall bracket 4 2 PC Compatible Interface A three pin header on the HCA card is provided as the PC compatible interface See Figure 4 on page 22 for the loca tion on the board Figure 3 PC Connector 4 3 Power All adapter cards receive power from the PCI Express Edge connector All other required power voltages are gener ated by on board switch mode regulators For power consumption see Specifications starting on page 22 4 4 Memory The HCA adapter cards support multiple memory devices through the PCI Express Flash and I2C compatible inter faces Rev 0 11 a Mellanox Technologies Confidential ConnectX IBDual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 15
29. y passivi YAR Field Decoder M Mellanox Technologies H Adapter Type H InfiniBand Host Channel Adapter N Ethernet Network Interface Card S Express Module T Media E 10GBASE CX4 G 10GBASE CX4 K 10GBASE SR XFP T 10GBASE T with powered connector S Silicon H ConnectX ports 1 1 2 2 I Host Interface 8 PCIe x8 9 PCIe SerDes 5 0 GT s G Generation lt blank gt Initial product generation Separator X Memory Size X MemFree B Bracket S Short T Tall N None R RoHS lt blank gt non RoHS C RoHS w Exemption R RoHS Lead Free For example the part number MHGH28 XSC describes Mellanox Technologies ConnectX IB HCA card with dual CX4 ports a PCIe2 0 x8 2 5GT s interface no on board memory mem free a short PCI bracket and RoHS R5 compliance Using the legend field M M to indicate a Mellanox Technologies product field H H to indicate an InfiniBand Adapter Card field T G to indicate Cu DDR field S H to indicate the ConnectX family e field 2 to indicate two ports e field I 8 to indicate PCI Express 2 0 x8 running at 2 5GT s field X X to indicate no on board memory field B S to indicate a short bracket and field R C to indicate RoHS R5 w Exemptions compliance Mellanox Technologies Confidential Rev 0 11 10 2 HCA Card Installation 2 1 Hardware and Software Requir
Download Pdf Manuals
Related Search