ConnectX® -2 Dual-Port VPI Mezzanine I/O Cards for Dell
Contents
1. Socket for Fabric B Socket for Fabric C Socket for Fabric B 7 Replace the blade server cover 8 Replace the blade server into the chassis Fabric and Slot Connections The Chassis can connect to 16 Servers each server is connected to Fabric B and Fabric C through the switch and out to external fabric connections These cards can be used as DDR IB QDR IB 10 Gb s Ethernet switches 20 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 With DDR switches in slots B1 C1 C2 B2 the cards will work as 20 Gb s IB HCA cards With Ethernet switches in slots BI C1 C2 B2 the cards will work as 10 Gb s NIC cards With QDR switches in slots BI and C1 C2 and B2 the cards will work as 40 Gb s IB HCA cards In this configuration only one port for each card will be active This section has diagrams that show the possibilities regarding fabric and slot combinations that are allowed This is especially important when using an M3601Q QDR switch The M3601Q QDR switch physically takes up two slots The two possible places for locating the M3601Q switch is in slots Bl and C1 or in slots C2and B2 Even though the QDR switch takes up two slots physically it is only connected to the fabric on the right side slot of the two slots The implications of this are that the left side fabric of the two slots wherein the QDR switch is inserted is not connected to any of the servers The diagrams below show possi2. 96 Data N A Number 102 VPD Keyword YA Asset Tag 104 Length 0x20 105 Data N A Alphanumeric 137 VPD Keyword RW Remaining read write area 139 Length 0x73 140 Data Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 256 MellanoxReadOnlyMsk 00 0 Numbers 349 Mellanox Read Write Mask Ox1 1 Numbers 511 Mellanox Read Only Mask 0x0 Numbers 14 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 3 Driver Software and Firmware 3 1 Drivers and Clustering Software Download and install the Mellanox OpenFabric software package for Linux Windows or other operating systems from the Mellanox software website on all nodes of the cluster Go to http www mellanox com gt Products gt InfiniBand VPI SW drivers This software package provides server drivers enabling connectivity for server and storage sys tems utilizing High Performance Computing HPC or enterprise data center EDC applications across InfiniBand and Ethernet fabrics It also provides a Subnet Manager for simple InfiniBand network configuration and network administration and diagnostic tools for network management 3 2 Updating the Mezzanine I O Card Firmware The Mezzanine I O card is shipped with the latest version of qualified firmware at the time of man ufacturing New firmware versions will be posted on the Mellanox web site Go to http www mellanox com gt Support gt Dell Make sure that you downl
3. A 2 1 FCC Statements USA Class A Statements 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 2 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be A 2 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme a la norme NMB 003 du Canada A 2 4 VCCI Statements Japan Class A Statement CORB PHUBRES ES RRS A ERER VCCI OR ESC TFAZA
4. Australia New Zealand C Tick Class A Data Rate DDR PCI Express Base 2 0 compliant backwards compatible with 1 1 Mellanox Technologies 39 Rev 1 2 Appendix B Avertissements de s curit d installa tion French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambi ante d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouvertures de ventilation gt 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut provoquer des blessures corporelles et des d gradations de biens 4 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel gt 5 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr caut ions sp ciales pour s assurer que le systeme reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut 6 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form
5. C1 Bl C2 and 1 EN switch in slot B2 Back plane connections to each server 12 12 12 12 6 6 2 C B C B C B Server 1 Server 16 Mellanox Technologies 31 Rev 1 2 VO Card Installation Configuration for 2 IB switches in slots C1 B1 and 2 EN switches in slots B2 and C2 Back plane connections to each server Server 1 Server 16 32 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for 3 IB switches in slots C1 B1 and B2 and 1 EN switch in slot C2 Ethernet Back plane connections to each server PR Be C slot 2 un EEE EN EN EN I wi sallis 6 0 121112 fell c B c B IC B EE Server 1 Server 16 4 4 Safety Warnings 1 Installation Instructions Read all installation instructions before connecting the equipment to the power N source 2 Over Temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings Mellanox Technologies 33 Rev 1 2 O Card Installation 3 Stacking the Chassis The chassis should not be stacked on any other equipment If
6. Dell logo to blue badge June 2010 1 1 Formatting April 2010 1 0 Initial Release 6 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 About this Manual This User Manual describes ConnectX 2 Dual Port mezzanine I O cards for Dell PowerEdge M1000e series Blade Servers It provides details as to the interfaces of the board specifications required software and firmware for operating the cards and relevant documentation Intended Audience This manual is intended for the installer and user of the mezzanine I O cards listed in ConnectX 2 Mezzanine I O Card Details on page 10 The manual assumes basic familiarity with the Infiniband architecture specifications Related Documentation Table 2 Documents List InfiniBand Architecture Specification Volume 1 Release 1 2 and Volume 2 release 1 2 1 Infiniband Architecture Specifications Descriptions PCI Local Bus Specification Rev 2 3 IEEE Std 802 3This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 2 0 Specifications Industry Standard PCI Express 2 0 Card Electrome chanical Specification Rev 1 3 Mellanox Firmware Tools MFT User s Manual Document Number 2204UG Online Resources e Mellanox Technologies Web pages http www mellanox com e Dell Support Web pages http support dell com Mellanox Technologies 7 Rev 1 2 Document Conventions A These symb
7. et qualifi gt 40 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 7 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gisl ations et r glementations nationales en vigueur 8 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 41 Rev 1 2 Appendix C Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromver sorgung anschlie en gt 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 55 C 131 F betrieben werden Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ft ungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten gt 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunterf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren gt 4 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t gt 5 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind beson dere Vorsichtsma nahmen zu ergreifen um die Stabilit t de
8. swing the clip up and back until the stop This action moves the cover back from the blade body b Grasp the latch and lift the cover from the blade Figure 5 Lift Off Cover Mellanox Technologies 17 Rev 1 2 O Card Installation Avoid touching the DIMM components A Allow sufficient time for the blade components to cool before starting this pro cedure 3 Open the retaining clip Figure 6 Open the Clip The card can be placed in either of two available slots 4 Expose the socket to be used for the new card a When replacing an existing card remove the card from the socket Grab the card on the edge on the side with UPC number and pull up while gently rocking the card back and forth b For a new installation remove the protective cover enclosing the socket for the card Figure 7 Protective Cover in Place 18 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Figure 8 Remove the Cover A Hold the mezzanine I O card by the edges only Line up the adapter card so that the pins of the adapter card are over the sockets in the blade server 5 Plug the card into the socket by placing your thumb over the UPC symbol see sample UPC below and pressing down until the card reaches the bottom Figure 9 Press on the Card 6 Close the retaining clip Mellanox Technologies 19 Rev 1 2 O Card Installation 4 3 Figure 10 Single Card in Slot C Socket for Fabric C
9. the chassis falls it can cause bodily injury and equipment damage 4 Lightening Electrical Hazard During periods of lightning activity do not work on the equipment gt 5 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equipment starting from the bottom to the top gt 6 Installation of Equipment This equipment should be installed replaced or serviced only by trained and qualified personnel gt 7 Disposal of Equipment Disposal of this equipment should be in accordance to all national laws and regulations gt 8 Compliance with Local and National Codes This equipment should be installed in compliance with local and national elec trical codes gt 34 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Appendix A Specifications A 1 Board Mechanical Drawing and Dimensions The ConnectX 2 VPI Mezzanine I O card mechanical drawing is depicted in Figure Note All dimensions are in millimeters J1 is the I2C Connector J2 is the flash jumper Figure 12 ConnectX 2 MCQH29B XCC SFF Mezzanine I O card 1 655 _ J2 is the I2C Connector t J1 is the Flash Jumper 17 5 CD D3 51 15 g 35 89 83 99 v Mellanox Technologies 35 Rev 1 2 A 2 EMC Certification Statements
10. 2 Mellanox Technologies 27 Rev 1 2 I O Card Installation Configuration for 1 IB switch in slot B1 1 EN switch in slot C1 and an IB QDR switch in slots C2 and B2 i Ethemet Back plane connections to each server OQ I 121112 60 00 121112 60 00 121112 C B C B C B Serveri Zn Server 16 An IB DDR Switch is in slot Bl When using an M3601Q QDR switch there is no con An Ethernet Switch is in slot C1 nection to the left side fabric of the slot When Slot 2 Contains an IB switch the corre The M3601Q switch physically takes up two slots but sponding Slot 1 for the same Fabric Must be is only connected to the right side slot IB With an M3601Q in slots C2 B2 Fabric C is unavail able in slot 2 28 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for 4 IB switches one in each slot C1 B1 C2 and B2 Back plane connections to each server Server 1 Server 16 Mellanox Technologies 29 Rev 1 2 I O Card Installation Configuration for 4 EN switches one in each slot C1 B1 C2 and B2 Back plane connections to each server 12 12 C B C B C B Server 1 Server 16 30 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for 3 IB switches in slots
11. GRRMHRECS CORE RE CERT S CBR HERZILEELTZLENBSNEF TOBARRA EU ZNR ET BELAPBKENSCECPHVET 36 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 2 5 KCC Notice Republic of Korea Only The KCC label may be located separately from the other regulatory markings applied to your prod uct Class A devices are for business purposes Class A Device 13 A 3 A yg Az 7121 ol 21712 4782922 AXA SSS rs 44547171 17024 HAR E ASAE 0 amp Z21847 aA wy AR Boy E TUHAS HHS 79822 WHS Al 7 BFC KCC Class A Regulatory Label If the regulatory label includes the following marking your device is a Class A product 2121 38248 2ISHS amp A ABS 79 43 MzZAUU M XH M 22F 9 nawy Mellanox Technologies 37 Rev 1 2 A 3 Specifications Table 8 ConnectX 2 MCQH29B XCC Specifications Physical Power and Environmental Size 88 00mm X 83 99 mm Voltage 12V 3 3V Typ Power 10 704 W Maximum Power 12 276 W Temperature 0 C to 55 C Protocol Support Regulatory InfiniBand IBTA v1 2 Auto Negotiation 10 Safety US Canada
12. Mellanox TECHNOLOGIES Dell and the Dell logo are trademarks of Dell Inc ConnectX 2 Dual Port VPI Mezzanine I O Cards for Dell PowerEdge M1000e series Blade Servers User Manual P N ConnectX 2 MCGH29B XCC MCQH29B XCC Rev 1 2 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PRO VIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUS TOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CAN NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MER CHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEM PLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCURE MENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISI
13. NG IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2010 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale and InfiniPCI are registered trademarks of Mellanox Technologies Ltd CORE Direct FabricIT PhyX and Virtual Protocol Interconnect are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners All other marks and names mentioned herein may be trademarks of their respective companies ConnectX 2 Adapter Card User Manual 2 Mellanox Technologies Document Number 3238 ConnectX 2 Adapter Card User Manual Rev 1 2 Table of Contents Revision History 6 About this Manual Chapter 1 Overview 1 1 Mezzanine I O Card 10 Chapter 2 Mezzanine I O Card Interfaces 11 2 1 InfiniBand Interface 11 2 2 Ethernet Interface 11 2 3 PCI Express Interface 11 2 4 Flash 11 2 5 Memory 12 2 5 1 System Memory 12 2 5 2 Flash 12 2 5 3 EEPROM VPD 12 Chapter 3 Driver Software and Firmware 15 3 1 Drivers and Clustering Software 15 3 2 Updating the Mezzan
14. ble configurations of switches fabrics and servers The view is of the back of the chassis Mellanox Technologies 21 Rev 1 2 I O Card Installation Configuration for QDR IB switch in slots Bl and C1 and 2 IB switches in Slots C2 and B2 Eihemet Back plane connections to each server 2 fn J ae Server 1 Server 16 When using an M3601Q QDR switch there is no connec An IB DDR Switch is in slot C2 tion to the left side fabric of the slot An IB DDR Switch is in slot B2 The M3601Q switch physically takes up two slots but is only connected to the right side slot With an M3601Q in slots B1C1 Fabric B is unavailable in slot 1 22 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for QDR IB switch in slots C1 and Bland 2 EN switches in slots C2 and B2 Ethernet Back plane connections to each server Fabric B slot 2 EE mE mE cu un En En Fabric C slot 2 mE mE it it i a a 000 CHERE CE B Server 1 Server 16 When using an M3601Q QDR switch there is no connec An Ethernet DDR Switch is in slot C2 tion to the left side fabric of the slot An Ethernet DDR Switch is in slot B2 The M3601Q switch physically takes up two slots but is only connected to the right side slot With an M3601Q in slots B1C1 Fabric B
15. cTUVus 20 or 40Gb s per port EU IEC60950 International CB Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet EMC Emissions USA FCC Class A IEEE Std 802 3ap Backplanes Canada ICES Class A including FEC EU CE Mark EN55022 Class A EN50024 EN61000 3 2 EN61000 3 QoS Multicast and Jumbo Frame Sup 3 port Japan VCCI Class A Korea KCC Class A RDMA Support 8 InfiniBand Virtual Lanes for each Australia New Zealand C Tick port Class A Data Rate Yes All Ports PCI Express QDR Base 2 0 compliant backwards compatible with 1 1 Table 9 ConnectX 2 MCGH29B XCC Specifications Physical Power and Environmental Size 88 00mm X 83 99 mm Voltage 12V 3 3V Typ Power TBD W Maximum Power TBD W Temperature 0 C to 55 C Protocol Support Regulatory 38 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Table 9 ConnectX 2 MCGH29B XCC Specifications Physical Power and Environmental InfiniBand IBTA v1 2 Auto Negotiation 10 Safety US Canada cTUVus 20 or 40Gb s per port EU IEC60950 International CB Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet EMC Emissions USA FCC Class A IEEE Std 802 3ap Backplanes Canada ICES Class A including FEC EU CE Mark EN55022 Class A EN50024 EN61000 3 2 EN61000 3 QoS 8 InfiniBand Virtual Lanes for each 3 port Japan VCCI Class A Korea KCC Class A RDMA Support Yes All Ports
16. ce The EEPROM is used for storing the Vital Product Data VPD and FRU The VPD format adheres to the PCI Local Bus specification rev 2 3 VPD definition The EEPROM capacity is 4Kbytes Table 5 MCQH29B XCC ah Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 0x16 3 Data DELL SFF Alphanumeric Short description ID MEZZ VPI QDR 25 Large Resource Type VPD R Tag 0x10 0x90 Read Only Area 26 Length 0x43 28 VPD Keyword PN Numbers Add in Card Part Number 30 Length 0x6 31 Data OX24WC 37 VPD Keyword EC Alphanumeric Engineering Change Level of the card rev 39 Length 0x3 40 Data X01 Customer revision 43 VPD Keyword SN Alphanumeric Serial Number 45 Length 0x14 46 Data OO0X24WC according to the board label MMM MMYMDSSSS 66 VPD Keyword V0 Misc Information 68 Length 0x16 12 Mellanox Technologies J ConnectX 2 Adapter Card User Manual Rev 1 2 Table 5 MCQH29B XCC en Item Value Format Description 69 Data 10G QDR Mezzanine card 91 VPD Keyword RV 93 Length 0x1 94 Data Checksum 95 Large Resource Type VPD W Tag 0x11 0x91 Read Write Area 96 Length 0x9D 98 VPD Keyword V1 Driver version 100 Length 0x6 101 Data N A Number 107 VPD Keyword YA Asset Tag 109 Length 0x20 110 Data N A Alphanumer
17. ht side slot With an M3601Q in slots B1C1 Fabric B is unavailable in slot 1 Mellanox Technologies 25 Rev 1 2 O Card Installation Configuration for QDR IB switch in slots C1 and B1 and another QDR IB switch in slots C2 and B2 Back plane connections to each server 2 Server 16 i When using an M3601Q QDR switch there is no con An IB QDR Switch is in Slots C2 B2 nection to the left side fabric of the slot With an M3601Q in slots C2 B2 Fabric C is The M3601Q switch physically takes up two slots but unavailable in slot 2 is only connected to the right side slot With an M3601Q in slots B1C1 Fabric B is unavail Serveri 26 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for 1 IB switch in slot C1 1 IB switch in slot Bland an IB QDR switch in slots C2 and B2 Ethernet Back plane connections to each server S lt lt a l g I 1 121112 12 12 60 60606 Cc B G B C B Server 1 Server 16 An IB DDR Switch is in slot Bl When using an M3601Q QDR switch there is no con An IB DDR Switch is in slot C1 nection to the left side fabric of the slot The M3601Q switch physically takes up two slots but is only connected to the right side slot With an M3601Q in slots C2 B2 Fabric C is unavail able in slot
18. ic 142 VPD Keyword RW Remaining read write area 144 Length Ox6E 145 Data Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 256 Mellanox Read Only Mask 0x0 0 Numbers 354 Mellanox Read Write Mask 0x1 1 Numbers 511 Mellanox Read Only Mask 0x0 Numbers Table 6 MCGH29B XCC E Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 0x15 3 Data DELL SFF Alphanumeric Short description ID MEZZ VPI DDR 24 Large Resource Type VPD R Tag 0x10 0x90 Read Only Area 25 Length 0x3F 27 VPD Keyword PN Numbers Add in Card Part Number 29 Length 0x6 30 Data O9DCC6 36 VPD Keyword EC Alphanumeric Engineering Change Level of the card rev 38 Length 0x3 39 Data X01 Customer revision Mellanox Technologies 13 Rev 1 2 Mezzanine I O Card Interfaces Table 6 MCGH29B XCC A Item Value Format Description 42 VPD Keyword SN Alphanumeric Serial Number 44 Length 0x14 45 Data QO09DCC6M according to the board label MMMMYMDS SSS 65 VPD Keyword V0 Misc Information 67 Length 0x12 68 Data 10G Mezza nine card 86 VPD Keyword RV 88 Length 0x1 89 Data Checksum 90 Large Resource Type VPD W Tag 0x11 0x91 Read Write Area 91 Length 0xA2 93 VPD Keyword V1 Driver version 95 Length 0x6
19. ice can be either a master ini tiating the PCI Express bus operations or a slave responding to PCI bus operations 2 4 Flash The I O card includes one SPI Flash device accessible via the Flash interface of the MT25408B0 ConnectX 2 device There is a jumper on the card that indicates to the device whether an on board Flash device exists or is to be used Table 4 provides information on this jumper See the schematic in Figure 12 on page 35 for the jumper location Table 4 Jumper Configuration Card Default Description Option Configueation Comments Flash present connection open Flash present connection open Header 1x2 not present connection shorted Flash not present Flash present Figure 2 Flash Jumper CLOSED FLASH NOT_PRESENT ae ng y 2 em Mellanox Technologies 11 Rev 1 2 Mezzanine I O Card Interfaces 2 5 Memory The Mezzanine I O card supports multiple memory devices through the PCI Express Flash and I2C compatible interfaces 2 5 1 System Memory The Mezzanine I O card utilizes the PCI Express interface to store and access network fabric con nection information on the system memory 2 5 2 Flash The Mezzanine I O card includes one 16MB SPI Flash device P N M25P16 VME6G by ST Microelectronics accessible via the Flash interface of the MT25408 ConnectX 2 device 2 5 3 EEPROM VPD Each board incorporates an EEPROM that is accessible through the I2C compatible interfa
20. ies 9 Rev 1 2 Overview 1 1 Mezzanine I O Card Table 3 ConnectX 2 Mezzanine I O Card Details Ordering Part Number OPN Link Speed RoHS Compliance MCGH29B XCC InfiniBand 20 Gb s DDR RoHS R6 with exemption or 10 GigE MCQH29B XCC InfiniBand 40 Gb s QDR RoHS R6 with exemption or 10 GigE poocoocoo Booooooeo Boooneeeo 10 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 2 Mezzanine I O Card Interfaces The Mezzanine I O card attaches to the blade using a press fit connector which connects both the network and PCI Express interfaces 2 1 InfiniBand Interface The ConnectX 2 MT25408 device is compliant with the InfiniBand Architecture Specification Release 1 2 1 It has two compliant 4X InfiniBand ports ports 1 and 2 each having four Tx Rx pairs of SerDes The Mezzanine I O card based on this device provides access to these ports through a board to board press fit connector 2 2 Ethernet Interface The ConnectX 2 MT25408 device is compliant with the IEEE Std 802 3 Specification It has two compliant ports each having a one Tx Rx pair of SerDes The Mezzanine I O card based on this device provides access to these ports through a board to board press fit connector 2 3 PCI Express Interface The Mezzanine I O card has eight Tx Rx pairs of SerDes providing for a PCI Express x8 interface version 2 0 compliant and compatible with base 1 1 chipsets The dev
21. ine I O Card Firmware 15 3 3 FlexBoot 15 Chapter 4 T O Card Installation 16 4 1 Hardware and Software Requirements 16 4 2 Installation Instructions 16 4 3 Fabric and Slot Connections 20 4 4 Safety Warnings 33 Appendix A Specifications 35 A l Board Mechanical Drawing and Dimensions 35 A 2 EMC Certification Statements 36 A 3 Specifications 38 Mellanox Technologies 3 Rev 1 2 List of Tables Table 1 Revision History Table Table 2 Documents List Table 3 ConnectX 2 Mezzanine I O Card Details 10 Table 4 Jumper Configuration 11 Table 5 MCQH29B XCC 12 Table 6 MCGH29B XCC 13 Table 7 Hardware and Software Requirements 16 Table 8 ConnectX 2 MCQH29B XCC Specifications 38 Table 9 ConnectX 2 MCGH29B XCC Specifications 38 4 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 List of Figures Figure 1 Dell SFF Mezzanine Card 10 Figure 2 Flash Jumper 11 Figure 3 Grounding Clips 16 Figure 4 DIMMS can be hot 17 Figure 5 Lift Off Cover 17 Figure 6 Open the Clip 18 Figure 7 Protective Cover in Place 18 Figure 8 Remove the Cover 19 Figure 9 Press on the Card 19 Figure 10 Single Card in Slot C 20 Figure 11 Two Cards Installed 20 Figure 12 ConnectX 2 MCQH29B XCC SFF Mezzanine I O card 35 Mellanox Technologies 5 Rev 1 2 Revision History This document was first printed on 14 11 2010 Table 1 Revision History Table Date Rev Comments Changes June 2010 1 2 Changed
22. is unavailable in slot 1 Mellanox Technologies 23 Rev 1 2 O Card Installation Configuration for QDR IB switch in slots C1 and B1 and 1 EN switch in slot C2 and 1 IB switch in slot B2 Ethernet Fabric C slot 2 mE a a a a EI Back plane connections to each server Z a ae D fr tf U al 121112 00 12 12 6 60 C B Ic B 6i ER EE pa Server 1 Server 16 When using an M3601Q QDR switch there is no connec An Ethernet DDR Switch is in slot C2 tion to the left side fabric of the slot An IB DDR Switch is in slot B2 The M3601Q switch physically takes up two slots but is only connected to the right side slot With an M3601Q in slots B1C1 Fabric B is unavailable in slot 1 24 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 Configuration for QDR IB switch in slots C1 and B1 and 1 IB switch in slot C2 and 1 EN switch in slot B2 Ethernet Back plane connections to each server ZY TS Fabric B slot 2 EE mE m tt En a SSS IE 2 eee eco E B ce B C B aed Server 1 Server 16 When using an M3601Q QDR switch there is no connection to An IB DDR Switch is in slot C2 the left side fabric of the slot An Ethernet Switch is in slot B2 The M3601Q switch physically takes up two slots but is only con nected to the rig
23. oad the FW according to the part number You will need the Mellanox Firmware Tools package available in the Mellanox OpenFabrics soft ware package to update firmware for this switch It can also be downloaded from the Management Tools section from the Mellanox Download web site In the directory that holds the latest firm ware run the following commands Ispci grep Mellanox identifies PCI ID to be used in next command mstflint d lt PCI ID for example 05 00 0 gt i lt bin file gt b 3 3 FlexBoot FlexBoot enables remote boot over Ethernet or InfiniBand using Boot over InfiniBand BolB Boot over Ethernet BoE or Boot over iSCSI Bo iSCSD This technology is based on the Pre boot Execution Environment PXE standard specification and FlexBoot software is based on the open source EtherBoot gPXE project see www etherboot org For more information go to http www mellanox com gt Products gt InfiniBand VPI SW Drivers gt FlexBoot Mellanox Technologies 15 Rev 1 2 O Card Installation 4 NO Card Installation 4 1 Hardware and Software Requirements Before installing the VPI Mezzanine I O card please make sure that the system meets the hard ware and software requirements listed in Table 7 Hardware and Software Requirements Table 7 Hardware and Software Requirements Requirement Description Hardware Used with Dell PowerEdge M1000e series Software Operating Refer to the PowerEdge M1000e serie
24. ols indicate a situation status or condition that may cause harm to people or damage to the equipment When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits 8 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 1 Overview This document is a User Manual for the Mellanox ConnectX 2 Mezzanine I O cards for Dell Pow erEdge M1000e series Blade Servers with Virtual Protocol Interconnect VPI supporting Infini Band and Ethernet connectivity VPI supported adapters make it possible for Dell PowerEdge M1000e series Blade Servers to operate with either InfiniBand or Ethernet switches With auto sensing capability each ConnectX 2 port can identify the attached switch and bring up the link automatically ConnectX 2 with VPI makes it easier for IT managers to deploy infrastructure that meets the challenges of a dynamic data center ConnectX 2 with VPI can accommodate the following port configurations Both ports InfiniBand e Both ports Ethernet e Port 1 InfiniBand port 2 Ethernet The cards described in this manual see Table 3 ConnectX 2 Mezzanine I O Card Details have the following main features e lus MPI ping latency e Selectable InfiniBand or Ethernet per port e CPU offload of transport operations e End to end QoS and congestion control e Hardware based I O virtualization Mellanox Technolog
25. s Manuals Systems Distributions Software Stacks Mellanox OpenFabric software package either MLNX_OFED for Linux or MLNX_WinOF for Microsoft Windows 4 2 Installation Instructions Installation of this mezzanine I O card should only be done by a properly qualified technician or engineer Installation or service not authorized by Dell or performed by unqualified personnel may void guarantees and warranties Read and follow all safety precautions specified in this document and in the Power Edge documen tation Use the documentation supplied with the PowerEdge M1000e series Blade Servers to remove and replace the blade from the chassis The card shown below has grounding clips that must come in contact with the chassis after the card is installed Figure 3 Grounding Clips Grounding Clip BE 16 Mellanox Technologies ConneciX 2 Adapter Card User Manual Rev 1 2 Make sure that you are properly grounded Make sure that the equipment both while in the chassis and while removed from the chassis is properly grounded to prevent ESD Figure 4 DIMMS can be hot Socket for Fabric C Retaining Clip Socket for Fabric B DIMMS can be hot A Allow sufficient time for the blade components to cool before starting this pro cedure The card can be placed in either of two available slots 1 Remove the blade from the chassis 2 Remove the cover from the blade a With the blade sitting on the work table
26. s Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen gt 6 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausgetauscht oder gewartet werden gt 42 Mellanox Technologies ConnectX 2 Adapter Card User Manual Rev 1 2 7 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen 8 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden Mellanox Technologies 43
Download Pdf Manuals
Related Search