InfiniHost III Lx PCI Express x8 HCA Adapter Card User's Manual
Contents
1. Les c bles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fab ricant des c bles pour connaitre les mises en garde et les instructions sp ciales 5 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 33 Mellanox Technologies Rev 1 052. RoHS R5 Mellanox Technologies Rev 1 05 26 Appendix B Interface Connectors Pinout B 1 P C Compatible Connector Pinout Figure 7 I2C Compatible Connector Plug and Pinout Table 12 I2C compatible Connector Pinout Connector Pin Number HCA Signal Name 1 SPSDA 2 SPSCL GND oe fmi EA NC n O co 4 5 NC B 2 InfiniBand Connector Pinout Figure 8 InfiniBand CX4 Connector Pinout Conn 4X S10 iBixop 3 IBtxip 3 S9 IBtxOn 3 IBtxin g S12 Btxop 2 IBtxIp 2 lt 2 S L iBixon e IBtxin 2 8 S14 IBtxOp 1 IBtxIp 1 lt 3 13 iBixon t IBtxin 1 SE S16 IBtxOp 0 IBtxlp o _S15 IBtxOn 0 IBtxIn 0 Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 13 Connector Pin Name and Number to Signal Name Correspondence Connector Pin Connector Pin IB Port A Signal Number Name Name SI IBtxIp 0 Rx_Al S2 IBtxIn 0 Rx A0 S3 IBtxIp 1 Rx A3 S4 IBtxIn 1 Rx A2 S5 IBtxIp 2 Rx A5 S6 IBtxIn 2 Rx A4 S7 IBtxIp 3 Rx A7 S8 IBtxIn 3 Rx A6 S9 IBtxOn 3 Tx A6 S10 IBtxOp 3 Tx A7 S11 IBtxOn 2 Tx A4 S12 IBtxOp 2 Tx A5 S13 IBtxOn 1 Tx A2 S14 IBtxOp 1 Tx_A3 S15 IBtxOn 0 Tx AO S16 IBtxOp 0 Tx Al G1 G6
3. This eguipment should be installed replaced or serviced only by trained and gualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes gt Mellanox Technologies Rev 1 05 Driver Software and Firmware 14 3Driver Software and Firmware 3 1 Driver Software For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox OpenFabrics Web site at http www mellanox com products ofed php Follow the installation instructions included in the download package For Windows download the appropriate software from https docs mellanox com dm WinIB ReadMe html 3 2 Updating HCA Card Firmware Each HCA card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com through the Firmware downloads link 3 3 Single HCA Card Firmware Update Firmware can be updated on the standalone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user s manual from the single HCA card firmware update page See http www mellanox com under Fi
4. s Manual 15 3 5 Customized HCA Card Firmware Update OEM Only Note The procedure described in this section is normally not needed and applies to OEMs only To create a customized firmware binary the firmware image in MLX format needs to be downloaded along with the MFT tools package See http www mellanox com under Firmware Downloads for customized firmware update instructions Mellanox Technologies Rev 1 05 Adapter Card Interfaces 16 4Adapter Card Interfaces 4 1 I O Interfaces Each HCA card includes the following interfaces e One 4X InfiniBand copper connector e PCI Express x8 edge connector e I O panel LEDs Cc compatible connector for debug 4 1 1 InfiniBand Interface The InfiniHost IIISLX MT25204 Third Generation device is compliant with the InfiniBand Architecture Specifica tion Release 1 2 It has one compliant 4X InfiniBand port having four Tx Rx pairs of SerDes Each of the HCA cards listed in Table 3 on page 9 based on this device provides access to this port by means of an InfiniBand connec torfor external InfiniBand copper cables These cards are also compliant with the IBTA specification 1 2 Figure 3 InfiniHost III CX4 interface Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 17 Each of the HCA cards is embedded with a media detect circuit that supports active cables and external I
5. G9 HI H2 Signal Ground GND G7 Sense 3 3V SENSE_P1 G8 Vee MC_POWER_P1 1 The Sense 3 3V signal is used to enable the Vcc power supply pin G8 used to provide power to the active media adapter B 3 PCI Express x8 Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 1 0a specification Mellanox Technologies Rev 1 05 28 Appendix C Replacing a Tall Bracket With a Short Bracket on HCA Cards This appendix provides instructions on how to remove the tall bracket of a standard Mellanox Technologies HCA card and replace it with a short one It includes the following sections e Removing Tall Bracket e Assembling Short Bracket Figure 10 shows the bracket side view of an HCA card Figure 9 CX4 Connector in a Tall Bracket CE C 1 Removing Tall Bracket Figure 10 shows a connector retention clip and the designated names of its sections Figure 10 Connector Retention Clip Clip Top Clip Center embraces the connector Clip Hook grips onto the connector Using a small flat head screwdriver gently push up one hook of a connector s clip toward the connector s top side as shown in Figure 11 on page 29 2 2 Then push the other hook With both hooks unlatched push the clip towards the connector s top side see Figure 11 b Finally pull the clip away from the bracket see Figure 11 c Rev 1 0
6. June 2007 Corrected the Maximum Power number for MHES18 XSC in Section A 3 MHES18 X ST C IB SDR MHGS18 X ST C IB DDR Specifications on page 25 as follows Old Maximum Power 4 2 W New Maximum Power 3 5 W 1 01 December Added EMC VCCI statements to Appendix A Specifications on page 22 2006 1 00 November First revision 2006 Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual About this Manual This User s Manual describes Mellanox Technologies MHES18 X TS C InfiniHost III Lx PCI Express x8 HCA Adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Infiniband networks and architecture specifications Related Documentation Table 2 Documents List InfiniHost III Programmer s Reference Manual Document no 2248PM A reference describing the interface used by developers to write a driver for MemFree Mellanox InfiniHost III devices InfiniHost IIl Lx MT25204 Hardware Reference Manual Document no 2131HM Reference for hardware engineers responsible for designing systems and boards incorporating InfiniHost III Lx components Mellanox Firmware Tools MFT User s Manual Document no 2204UG
7. Updating HCA Card Firmware 3 3 Single HCA Card Firmware Update 3 4 HCA Card Firmware Update as Part of a Cluster Firmware Update 3 5 Customized HCA Card Firmware Update OEM Only Adapter Card Interfaces 4 1 I O Interfaces 42 Power 4 3 Memory 44 VPDs Specifications A 1 Board Mechanical Drawing and Dimensions A 2 EMC Certification Statements A 3 MHES18 X ST C IB SDR MHGS18 X ST C IB DDR Specifications Interface Connectors Pinout B 1 12C Compatible Connector Pinout B 2 InfiniBand Connector Pinout B 3 PCI Express x8 Connector Pinout Replacing a Tall Bracket With a Short Bracket on HCA Cards C 1 Removing Tall Bracket C 2 Assembling Short Bracket Avertissements de s curit d installation Mellanox Technologies Rev 1 05 8 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Rev 1 05 Revision History Table Documents List HCA Cards Mellanox HCA Cards Part Numbering Key Hardware and Software Requirements Firmware Images for InfiniHost III Lx PCI Express HCA Cards LEDs Jumper Configuration VPD Layout for MHES18 X ST C VPD Layout for MHGS18 X ST C HCA Cards Certification Status D2C compatible Connector Pinout Connector Pin Name and Number to Signal Name Correspondence Mellanox Technologies 10 12 14 17 18 19 20 23 26 27 ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Expre
8. the LEDs are not active either the physical link or the logical link or both connections have not been established Figure 4 Physical and Logical Link Indications Table 7 LEDs Port Number LED Name Port 1 Physical Link Green Constant on indicates a good physical link Blinking indicates a problem with the Physical link Data Activity Yellow Blinking indicates Data Transfer Constant on indicates no Data Transfer Note The short bracket has the same port and LED footprint as the tall bracket 4 1 4 PC Compatible Interface A three pin header on the HCA card is provided as the re compatible interface See Figure 7 on page 22 for the loca tion on the board Figure 5 C Connector Mellanox Technologies Rev 1 05 Adapter Card Interfaces 18 4 2 Power All adapter cards receive power from the PCI Express Edge connector All other required power voltages are gener ated by on board switch mode regulators See Specifications on page 22 4 3 Memory The HCA cards support multiple memory devices through the PCIExpress Flash and I2C compatible interfaces 4 3 1 System Memory Each of the HCA cards utilizes the PCI Express interface to store and access IB fabric connection information and packet data on the system memory 4 3 2 Flash Each of the HCA cards includes one 2MB SPI Flash device P N M25P16 by ST Microelectronics accessible via the Flash interface of the MT25408 InfiniHost III L
9. 5 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Figure 11 Extracting Connector Clip a Gently Push One Hook of Clip b Gently Push Other Hook of Clip c Pull Clip Away 3 Unscrew both screws from the card using a torque screwdriver as shown in Figure 13 EU gt ae E r a Tit lt gt E ts Figure 12 Unscrew Bracket Screws Ss ee These two screws hold on the bracket Grip the bracket as shown in Figure 13 placing your thumb on the LED component In a rotating move toward the component side of the card slide the bracket out of the connector Figure 13 b Gently hold your thumb on the LED component oe XR At the same time extract the bracket as shown in Figure 13 b Make sure to protect the LED while extracting the bracket Mellanox Technologies Rev 1 05 30 Figure 13 Rotate the Bracket to Detach it From the Card a Grip the Card in preparation for Detachment b Rotate the bracket toward the Component Side c Bracket Separated From Clips and Screws C 2 Assembling Short Bracket The short bracket can now be assembled onto the HCA card 1Gently place the bracket onto the card fitting the connectors through the bracket connector holes The tab on the bracket should be pointing in the same direction as the PCI connector Make sure the LEDs are aligned into their intended bracket holes 2 Inserta screw along with a washer in
10. AM Mellanox TECHNOLOGIES InfiniHost III Lx PCI Express x8 HCA Adapter Card User s Manual P N MHES18 XSC MHES18 XTC MHGS18 XSC MHGS18 XTC Rev 1 05 Mellanox Technologies Copyright 2008 Mellanox Technologies Inc All Rights Reserved Mellanox Technologies InfiniHost and Connectx 2 are registered trademarks for Mellanox Technologies Inc All other marks and names mentioned herein may be trademarks of their respective companies InfiniHost III Ex PCI Express Low Profile Adapter Card User s Manual Document Number 2311 Mellanox Technologies Inc 2900 Stender Way Santa Clara CA 95054 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam 20692 Israel Tel 972 4 909 7200 Fax 4972 4 959 3245 Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table of Contents Table of Contents List of Tables List of Figures Revision History About this Manual Chapter 1 Chapter 2 Chapter 3 Chapter 4 AppendixA AppendixB AppendixC AppendixD Overview 1 1 Adapter Cards 1 2 Mellanox Part Numbering Legend 1 3 Finding the GUID and Serial Number on the Adapter Cards HCA Card Installation 2 1 Hardware and Software Requirements 2 2 Installation Instructions 2 3 Safety Warnings Driver Software and Firmware 3 1 Driver Software 3 2
11. N EUR ONSE EU XT EIE BEDPBRKENSCECPHVET Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual A 3 MHES18 X ST C IB SDR MHGS18 X ST C IB DDR Specifications Physical Power and Environmental Size Air Flow 10Gb s Connector 54mm x 102mm 2 13 in X 4 in 200LFM 55C Amphenol InfiniBand MicroGigaCN Emcore Fiber Channel Voltage Maximum Power Temperature 3 3V 3 5W 0 to 55 Celsius Protocol Support Regulatory InfiniBand Qos RDMA Support Auto Negotiation 10Gb s 2 5Gb s 4 InfiniBand Virtual Lanes for the 4X port Yes EMC Safety Environmental RoHS FCC 47 CFR part 15 2005 subpart B class A ICES 003 2004 Issue 4 class A VCCI V 3 2005 04 class A MIC class A EN 55022 1998 A 1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A1 2001 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Direc tive 89 336 EEC AS NZS 3548 TEC EN 60950 1 2001 ETSI EN 300 019 2 2 UL 60950 1 2003 R7 06 CAN CSA C22 2 No 60950 1 03 IEC 60068 2 64 29 32
12. S18 X ST C Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x82 0x02 1 Length 7 0 LSB 0xB 2 Length 15 8 MSB 0x0 3 Data Cheetah DDR STR 14 Large Resource Type VPD R Tag 0x90 0x10 15 Length 7 0 LSB Ox4F 16 Length 15 8 MSB 0x00 17 VPD Keyword PN STR Add in Card Part Number 19 Length Ox15 20 PN PN STR_SPC 41 VPD Keyword EC STR Engineering Change Level of the card rev 43 Length 0x2 44 Revision RV STR PCB revision 46 VPD Keyword SN STR Serial Number 48 Length 0x18 49 SerialNumber STR_SPC 00 00X XXX XX 73 VPD Keyword vo STR Misc Information 75 Length 0x10 76 Data PCle x8 STR SPC 92 VPD Keyword RV STR 94 Length 0x1 95 Data 0 94 CSO 96 Large Resource Type VPD W Tag 0x91 0x11 97 Length 7 0 LSB 0x9C 98 Length 15 8 MSB 0x00 99 VPD Keyword VI STR EFI Driver version 101 Length 0x6 102 Data N A STR SPC 108 VPD Keyword YA STR Asset Tag 110 Length 0x20 111 Data N A STR_SPC N A 143 VPD Keyword RW STR Remaining read write area 145 Length 0x6D Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 10 VPD Layout for MHGS18 X ST C Offset Decimal Item Value Format Description 146 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x78 0x11 Mellanox Technologies Rev 1 05 22 Appendix A Specifications A 1 Board Mechanical Drawing and Di
13. User s Manual describing the set of MFT firmware management tools for a single InfiniBand node See http www mellanox com under Firmware downloads Mellanox MST User s Manual Document no 2125SM This manual describes various tools and utilities included in the Mel lanox Software Tools MST package for accessing burning firmware and tracing Mellanox silicon devices InfiniBand Administration IBADM Package User s Manual Document no 2130UM User s Manual describing the utilities included in the IBADM tools package for system administration of an InfiniBand cluster See http www mellanox com under Management Tools VAPI HCA Device Driver Release Notes Version 4 0 0 or later InfiniHost III device driver release notes IB Specifications Release 1 0 a Infiniband Architecture Specifications PCI Express 2 0 Specifications Industry Standard PCI Express 2 0 Card Electromechanical Specifica tion Rev 1 3 Online Resources e Mellanox Technologies Web pages http www mellanox com e Mellanox Technologies Firmware download Web page http www mellanox com under Firmware downloads Mellanox Technologies Document Distribution System DDS http docs mellanox com requires a customer login account Mellanox Technologies Rev 1 05 8 Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega byte
14. e ID String Tag 0x82 0x02 1 Length Ox7 3 Data Cheetah Alphanumeric 10 Large Resource Type VPD R Tag 0x90 0x10 11 Length Ox4F 13 VPD Keyword PN Numbers Add in Card Part Number 15 Length Ox15 16 Data PN 37 VPD Keyword EC Alphanumeric Engineering Change Level of the card rev 39 Length 0x2 40 Data B1 PCB revision 42 VPD Keyword SN Alphanumeric Serial Number 44 Length 0x18 45 Data MTYYW according to the board label WPSSSSS 69 VPD Keyword V0 Misc Information 71 Length 0x10 72 Data PCIe x8 88 VPD Keyword RV 90 Length 0x1 91 Data Checksum 92 Large Resource Type VPD W Tag 0x91 0x11 93 Length OxAO 95 VPD Keyword V1 EFI Driver version 97 Length 0x6 98 Data N A Number 104 VPD Keyword YA Asset Tag 106 Length 0x20 107 Data N A Alphanumeric N A 139 VPD Keyword RW Remaining read write area 141 Length 0x71 142 Data Reserved 0x00 255 Small Resource Type END Tag 0x78 Ox11 Mellanox Technologies Rev 1 05 Adapter Card Interfaces 20 Table 9 VPD Layout for MHES18 X ST C Offset Decimal Item Value Format Description 256 Mellanox Read Only Mask 0x0 0 Numbers 351 Mellanox Read Write Mask Ox1 1 Numbers 511 Mellanox Read Only Mask 0x0 Numbers Table 10 VPD Layout for MHG
15. e PCI Express 2 0 x8 running at 2 5GT s e field X X to indicate no on board memory e field B S to indicate a short bracket and e field R C to indicate RoHS RS w Exemptions compliance Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 1 3 Finding the GUID and Serial Number on the Adapter Cards All mellanox HCA adapter cards have a label on the printed side of the adapter card that has the card serial number and the card GUID Figure 2 Card Product Label S N MT0710X01085 OLANLA YANMA A ABEL LU REV X1 P N MHGH28 XTC Ulu W MU 0 NN EL ULT I GUID 0002C90300000424 OL LL LU OR EL LE AE LE LE LE RE _ Made In IL Mellanox Technologies Rev 1 05 HCA Card Installation 12 2 HCA Card Installation 2 1 Hardware and Software Requirements Before installing the HCA Adapter card please make sure that the system meets the hardware and software require ments listed in Table 5 Table 5 Hardware and Software Requirements Requirement Description Hardware PCI Express x8 slot or x4 slot with x8 connector Software e For Windows see WinIB ReadMe at https docs mellanox com dm WinIB ReadMe html Operating e For Linux see Mellanox OpenFabrics Enterprise Distribution OFED software package available via the Systems Distributions Mellanox OpenFabrics Web site http www mellanox com products ofed php 2 2 Installat
16. egend Table 4 describes the Mellanox Technologies adapter cards part numbering legend Table 4 Mellanox HCA Cards Part Numbering Key M e Field Decoder M Mellanox Technologies H Adapter Type H InfiniBand Host Channel Adapter N Ethernet Network Interface Card S Express Module T Media E CX4 SDR G CX4 DDR J CX4 QDR K XFP SR M SFP SR N SFP LRM O SFP LR Q QSFP QDR R QSFP DDR T UTP S Silicon H ConnectX S InfiniHost III Lx T InfiniHost A InfiniHost III Ex Arbel S Infini Host III Lx Sinai T InfiniHost Tavor ports 1 1 2 2 I Host Interface X PCI X 4 PCle x4 8 PCle x8 9 PCle SerDes 5 0 GT s G Generation lt blank gt Initial product generation Separator X Memory Size X MemFree 1 128MB 2 256MB 3 512MB B Bracket S Short T Tall N None R RoHS lt blank gt non RoHS C RoHS w Exemption R RoHS Lead Free For example the part number MHES 14 XSC describes Mellanox Technologies InfiniHost III IB HCA card with single CX4 port a PCIe2 0 x4 2 5GT s interface no on board memory mem free a short PCI bracket and RoHS R5 compliance Using the legend e field M M to indicate a Mellanox Technologies product e field H H to indicate an InfiniBand Adapter Card e field T E to indicate 1OGBASE CX4 e field S S to indicate InfiniHostIII e field to indicate one port e field I 8 to indicat
17. ion Instructions Read all installation instructions before connecting the equipment to the power source Please consult the host machine documentation for instructions on how to install a PCI Express card 2 2 1 Installation Instructions as per Host Machine The adapter cards listed in Table 3 on page 9 are standard PCI Express 8 cards each with a standard x8 edge connec tor Please consult the host machine documentation for instructions on how to install a PCI Express card Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 2 3 Safety Warnings 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Over temperature This eguipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 55 C 131 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions gt 5 Eguipment Installation
18. mensions All of the HCA cards covered in this User s Manual have the same mechanical drawing and share the same dimen sions as depicted in Figure 7 Note All dimensions are in millimeters Figure 7 Schematics for MH EG 18 X ST C red 101 55 S Ue fU2 88888 ol en 80 A N am me Jo O C12 B AE C E Rp Af LCS z n Apa U8 a n beli Mk Eee D Q C ET 44 56 n E 7 4 2885 ge 3 65 wd 3385 8 99 88 89 at 94 05 J1 I2C Connector J2 Flash Jumper Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual A 2 EMC Certification Statements A 2 1 Table 11 lists the approved certification status per HCA card in different regions of the world Table 11 HCA Cards Certification Status C Tick ICES Australia HCA Card PN ao a er Class and New e un CB Report CTUVUS P Canada Zealand P Class A Class A Class A Class A Class A MHES18 XSC C d ut Class A Class A Class A Class A Class A MHES18 XTC S T di Class A Class A Class A Class A Class A MHGS18 XSC 4 Class A Class A Class A Class A Class A MHGS18 XTC C di T d A 2 2 FCC Statements USA Class A Statements 8 15 21 Statement Warning Changes or modifications to this equipment not expressly appr
19. nfiniBand fiber solutions to be connected to the InfiniBand port connectors Fiber Solutions require the use of active media con verters The ConnectX InfiniHost III IB HCA adapter cards support the PCI Express 1 0a x4 interface 1 0a compatible The cards can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations 4 1 2 PCI Express Interface The InfiniHost III Lx MT25204 device has eight Tx Rx pairs of SerDes providing for a PCI Express x8 interface version 1 0a compatible The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The PCI Express bus can connect to either a host CPU in an HCA application or to an I O device such as Gigabit Ethernet when used as a Target Channel Adapter 4 1 3 LED Assignment The board has two LEDs located on the I O panel The green LED when lit indicates that the InfiniBand driver is running and a valid physical connection between nodes exists If the green LED is blinking it indicates a problem with the physical link The yellow LED when lit indicates a valid data activity link this is the logical link The yel low LED illuminates when the InfiniBand network is discovered over the physical link A valid data activity link without data transfer is designated by a constant yellow LED indication A valid data activity link with data transfer is designated by a blinking yellow LED indication If
20. oved by the party responsi ble for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equip ment generates uses and can radiate radio frequency energy and if not installed and used in accor dance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense Mellanox Technologies Rev 1 05 24 A 2 3 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures A 2 4 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian I CES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada A 2 5 VCCI Statements Japan Class A Statement CORB TESRAVEEXE IE SS TER C B SEN RE VCCI DH EKT lt 772A RME CT COREE RE CEAT 5 CER Hesse TL E
21. rmware downloads A firmware binaries table lists a binary file per HCA card The file name of each such binary is composed by combin ing the firmware name the firmware release version and the card part number See Table 6 Please contact your Table 6 Firmware Images for InfiniHost III Lx PCI Express HCA Cards PCI DevID Decimal amp Firmware Image Name Type of HCA Card Examples PCI DevID ai fw 25204 1 1 0 MHESIS XSC A2 bin zip is the firmware binary of firmware fw MERS 25204 version 1 1 0 for the HCA card MHES18 XSC Rev A2 Firmware fw 25204 assigned Field Application Engineer if you cannot find the firmware binary for your adapter card This may happen if the product is not yet available for general distribution 3 4 HCA Card Firmware Update as Part of a Cluster Firmware Update If the HCA card is part of an InfiniBand cluster its firmware can be updated as part of the entire cluster firmware update using the ibfwmgr tool of the IB administration IBADM tools package IBADM is available for download as part of IB stack distributions such as IB Gold and OFED available via http www mellanox com See Firmware Downloads under the same webpage for cluster update instructions 1 Currently only the Linux distributions support updating firmware for an entire InfiniBand cluster Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User
22. s The use of Mb or Mbits small b indicates size in mega bits Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 1 Overview This document is a User s Manual for Mellanox Technologies host channel adapter HCA Cards based on the MT25408 InfiniHost III Lx HCA integrated circuit device The cards described in this manual have the following main features e IBTA v1 2 compliant e Single 4X InfiniBand copper ports for connecting InfiniBand traffic 4X IB connectors e PCI Express expansion board with an x8 edge connector compatible to the PCI Express 1 0a specification e Media detect circuit with powered connectors supporting the use of active cables and external PHY fiber solu tions e EU Restriction of Hazardous Substances RoHS compliant e SDR DDR4X IB10Gb s SDR or 20Gb s DDR Fiber solutions run at SDR only e Two bracket heights short or tall 1 1 Adapter Cards Table 3 lists the InfiniBand HCA cards described in this manual Table 3 HCA Cards Short Ordering Part IB SDR RoHS HCA IC Part Number OPN DDR Tall Compliance Number HCA Card Photo Bracket MHES18 XSC SDR Short RoHS R5 MT25204A0 FCC Figure 1 HCA Card Component Side MHES18 XTC Tall MHGS18 XSC DDR Short RoHS R5 MT25204A0 FCC D MHGS18 XTC Tall Mellanox Technologies Rev 1 05 Overview 10 1 2 Mellanox Part Numbering L
23. ss x8 User s Manual List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 HCA Card Component Side Card Product Label InfiniHost III CX4 interface Physical and Logical Link Indications I2C Connector Flash Jumper Schematics for MH EG 18 X ST C I2C Compatible Connector Plug and Pinout InfiniBand CX4 Connector Pinout CX4 Connector in a Tall Bracket Connector Retention Clip Extracting Connector Clip Unscrew Bracket Screws Rotate the Bracket to Detach it From the Card Attach Bracket onto Card using Screws Sliding Connector Clip Evenly Assembled Short Bracket View Mellanox Technologies 11 16 17 17 18 22 26 26 28 28 29 29 30 30 31 31 Rev 1 05 6 Revision History This document was printed on August 12 2008 3 48 pm Table 1 Revision History Table Rev Date Comments Changes 1 05 August 2008 Removed watermark Removed Confidential from footer Added copyright info to page 2 Replaced safety warnings with more encompassing warnings Added translation of warnings into French in appendix 1 04 May 2008 Added MIC C Tick Cb report and CTUVUS Certs to Appendix A Cert Table 1 03 March 2008 Added MIC statement and new graphics for I2C and Flash Jumper Added 2 columns to the Cert Table in Appendix A 1 02
24. to each of the two holes on the card intended for holding the bracket 3 Use a torque screwdriver to apply up to 2 Ibs in torque on each screw Figure 14 Attach Bracket onto Card using Screws Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 31 4 Gently push the clip onto the connector Make sure to slide both clip hooks sides around the connector evenly as shown in Figure 15 Figure 15 Sliding Connector Clip Evenly 5 Use a small flat head screwdriver to gently slide the clip hooks towards the connector s base side as shown in Figure 15 Figure 16 Assembled Short Bracket View Mellanox Technologies Rev 1 05 32 Appendix D Avertissements de s curit d installation 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 45 C 113 F En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles InfiniBand en cuivre
25. x device There is a jumper on each adapter card that indicates to the device whether an on board Flash device exists or is to be used Table 8 provides information on this jumper See Figure 7 on page 22 for the jumper location Table 8 Jumper Configuration Card Default Description Option Comments p Configuration Flash present not present connection open Flash present connection open Header 1x2 connection shorted Flash not present Flash present Figure 6 Flash Jumper Metal Soldered lead Connection Insulating Spacer Soldered Metal Connection Jead 4 3 3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C compatible interface The EEPROM is used for storing the Vital Product Data VPD The VPD format adheres to the PCI Local Bus specification rev 2 3 VPD definition The EEPROM capacity is 512 bytes Rev 1 05 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 4 4 VPDs The PCI VPD Vital Product Data layout for each of the described Mellanox Technologies InfiniHost III Lx cards complies with the format defined in the PCI 2 3 Specification Appendix I All InfiniHost III Lx cards share the same PCI VPD layout Table 9 VPD Layout for MHES18 X ST C Offset Decimal Item Value Format Description 0 Large Resource Typ
Download Pdf Manuals
Related Search