Home

User Manual

Contents

1. 72 Mellanox Technologies 19 18 Table 13 InfiniBand QSFP Connector Pinout 17 nnector uc TU Signal Description m 1 GND Ground i 2 Tx2n Transmitter Inverted Data Input T 3 Tx2p Transmitter Non Inverted Data 10 Input 9 4 GND Ground 8 5 Tx4n Transmitter Inverted Data Input 7 6 Tx4p Transmitter Non Inverted Data 6 Input 5 T GND Ground 8 ModSelL Module Select 2 9 ResetL Module Reset 1 10 Vcc Rx 3 3 V Power supply receiver 11 SCL _ 2 wire serial interface clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Out put 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Out put 18 Rxin Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Out put 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Out put 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vcc Tx 3 3 V Power supply transmitter 30 Vecl 43 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground Rev 3 4 QDR InfiniBand Switch Platform User Manual 18 35 Top View into Rear of Connector GND 138 AX n la7 TX1p l36 GND 35 TX3n 34 TX3p 33 GND 132 LPMode
2. A 1 EMC Certifications The list of approved certifications per switch in different regions of the world is located on the Mellanox Website at www mellanox com related docs user manuals Regulatory and Compliance Guide pdf A 2 EMC Statements A 2 1 FCC Statements USA Class A Statements 8 15 19 a 4 This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 This device must accept any interference received including interference that may cause undesired operation 5 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digi tal device pursuant to Part 15 of the FCC Rules These limits are designed to provide reason able protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interfer ence to radio communications Operation of this equipment in a residential area is likely to Mellanox Technologies 57 Rev 3 4
3. 2222 ooo 69 Figure 47 Screwing on the Bracket saure een tees 69 Figure 48 Making Room for the Power Cord 70 Figure 49 Screwing on the Rak esee EX b eher ihr 70 Figure 50 Power Cord Through the Bracket ro 70 Figure 51 Power Side View Hr 5s rogi A IS x ANA AAA 71 Figure 52 Connect Bracket to Rack Vertical support 71 6 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 List of Tables Table 1 Revision History Table 23 00 src nS ERES hib e pK ie x sai EE LALA 8 Table 2 Reference Documents eere 10 Table 3 Switch Management 2 su ose neg pr kenn 14 Table 4 System Status LED Configurations 17 Table 5 Connector Physical and Logical Link Indications 17 Table 6 PSU Status LED Configurations ssi RR RR mangue 18 Table 7 PSU Status LED Configurations 22448288 due arbre ei ae Rhin 19 Table 8 Fan Status LED Configurations 2054 noie bte a REX ES 19 Table 9 Air Flow Direction nues ee a e E 21 Table 10 IS5025 Specification Data 2 2 e ette ertet nete ni 52 Table 11 185030 Specification Dalai ee ne On e Rs tac oe ai 54 Table 12 185035 Specification Data 56 Table 13 InfiniBand QSFP Connector Pinout 72 Table 14 RJ45 CONSOLE Pinout Loricariidae aes 74
4. 9 Equipment Disposal pa A Disposal of this equipment should be in accordance to all national laws and regula Ah tions 10 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical KA codes 11 Do Not Use The Switch As A Shelf Or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space 3 6 2 Mechanical Installation The procedure for installing the switch in a full rack while bringing the power cord across along side of the switch can be found in See Passing the Power Cord From the Connector Side to the Power Side on page 61 The switch platform can be rack mounted and is designed for installation in a standard 19 rack The power side of the switch includes a hot swap power supply module a blank cover for an optional second PSU for redundancy and a hot swap fan tray There are two possible air flow directions Be sure that the switch air flow direction is compatible with your system rack and 26 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 PSUs The connector side of the switch has the QSFP ports system LEDs and management con nection ports The switch platform contains auto sensing 100 240 VAC connections for all possible PSUs The installer should use a rack capable of supporting the mechanical and environmental charac teristics of a fully populated platform
5. ual for instructions to install the license key 1 5030 and 185035 management stack license is located on the bottom side of the pull out tab Figure 2 Pull Out Tab S N MT0924X00266 UMMM MM T Tt AA A IYA EU UE OVIN LOIN GUID 0002C9020040DDBO E E ME 0 T Made in IL Pull out tab 1 1 3 License Key This switch comes with a license for FabricIT The license key to activate the license is found on the bottom side of the pull out tab Figure 3 License Key Label A MT1134x00560 License Key LK2 EFM CONFIG 5G2L S5HG LOKA L64Q K6D2 Q60Q 3ADH G88A 1H30 7FT9 5M Mellanox Technologies 13 Rev 3 4 Internally Managed vs Externally Managed 2 internally Managed vs Externally Managed The following table shows which switches come with a management CPU and which do not Unmanaged switches are plug and play out of the box All switches come with the latest FW burned on the Flash and EEPROM Update the FW on unmanaged switches in band only When new FW is available you will receive an e mail with the link to the Mellanox FW download site The download site has the Mellanox FW tool package and full instructions for updating FW All managed switches have internal chassis management and can manage up to 108 nodes FabricIT can support IB fabrics of up to 648 nodes Managed switches need an initial configura tion before they will start working See the Installation Guide for initial configuration
6. cause harmful interference in which case the user will be required to correct the interference at his own expense A 2 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be A 2 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada 58 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 A 3 China CCC Warning Statement China CCC Warning statements EENE EENAA English translation disconnect all power sources before service Q For non tropical use REM FFE RAITESER river Kid 9 208 mrt wie od a aly MR C SH nte s Dan hab yungh youq gij digih haibaz 3 2000m CC haenx ancienz saw h gado ASAL san Woo ig 35 pina 2000 3253 gt Warning for Class A B HA A RP NS RUE WA A TIGER ITR English translation of above statement This is a Class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures A4 VCCI Statements Japan Class A Statement Mellanox Technologies 59 Rev 3 4 COREL PHUBRES TER HER EHE MRA VCCI DH CESTA A RARE CT CORE RE CERT S CBR D esse TL EN EURE COMSICMLHAA
7. dante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeck ung nicht sicher festsitzt oder entfernt wurde 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 6 Anschlie en Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers gt 7 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind beson dere Vorsichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen gt 8 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausgetauscht oder gewartet werden gt 78 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden Mellanox Technologies 79 Rev 3 4 Appendix H Advertencias de seguridad para
8. 81 Vect 80 VecTx leo IntL lea ModPrsL 57 GND 26 RX4p 25 RX4n 24 GND 23 RX2p 22 RX2n 21 GND po gt a M ModSelL View into Front of Cage 4 E and wzxu dexx AND upXu d7XH AND 1SIdPON N e gt ol Oo N ee CRE X199A bz LIDA 03 9POWAT ka QNO kas 1 AND n dx AND UEXY dexu AND Vas 10S XH99A Tieseu TISSPON ANO dex UPXL ANO dzXL uzZXL ANO 73 Mellanox Technologies Rev 3 4 Appendix D RJ45 CONSOLE Interface The RJ45 CONSOLE interface uses the EIA 568A standard wiring color coding CONSOLE J34 Table 14 RJ45 CONSOLE Pinout Connection Signal Pin Color TD TXD sa Fin TXD RJ45 RTS 1 G W Ko I RJ45 DTR GND ERO DD TXD RJ45_DTR 2 G GND END RXD GND RXD RJ45_TXD 3 O W OND E RJ45 RXD GND GND GND GND 4 BI GND Pin 8 GND GND 5 BI W RXD RJ45 RXD 6 O GND GND T Br w Looking into the Socket em END i T FC J34 RR gt TXD in GND GND AN GND Le GND an GND Exe RJ45 SCL GND GND Pin 8 Looking into the Socket 74 Mellanox Technologies Table 15 Replacement Parts Ordering Numbers QDR InfiniBand Switch Platform User Manual Rev 3 4 Appendix E Replacement Parts Ordering Numbers
9. Rev 3 4 9 Figure 38 Caged Nut Spacing The caged nuts are separated by a single space Using two of the pan head screws and one washer for each rail install the rails to the other end of the rack Place the rail behind the holes in the rack and screw the screws through the holes in the washer then through the rails Tighten the pan head screws that hold the rails to 9 2 Nm or 81 5 pound inches Figure 39 Screwing in the Rails Place the four bolts for the caged nuts within reach Slide the switch into the rails Put the switch into place and screw the bolts into the nuts from step 4 Tighten the bolts to 9 2 Nm or 81 5 pound inches Tighten all of the screws 10 Plug in the power cables 11 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation 12 You can start connecting all of the cables to the switch Make sure that the Rail kit is compatible with your rack For short depth switches rail kit MIS00083 is to be used for racks from 50cm to 80cm deep and rail kit MIS00079 is to be used for racks from 38cm to 50cm deep including the iDataPlex rack For standard depth switches rail kit MIS00085 is to be used for racks from 60cm to 80cm deep 64 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Parts included in the rail kit 2 rails 18 recessed flat head screws 2 rail slides 8 caged nuts 2 b
10. corresponding to each QSFP port will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connec tor plugged into a functioning port After plugging in a cable lock the connector using the latch ing mechanism particular to the cable vendor When a logical connection is made the yellow light will come on When data is being transferred the yellow light will blink Cables in the bottom row should be inserted upside down in relation to the how A the cables are inserted in the top row Mellanox Technologies 37 Rev 3 4 Basic Operation and Installation Figure 28 Top and Bottom Ports 03 E 5 E m To remove disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Care should be taken not to impede the air exhaust flow through the ventilation holes next to the InfiniBand ports Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 3 6 7 Extracting and Inserting the Fan Unit This switch can operate indefinitely with one of the three fans in the fan module inoperable so long as the ambient temperature is below 45 Celsius Operation without a fan unit should not exceed two minutes During fan hot swap if both
11. 1 1 Redundant Power Supplies 1 44Tb s for 36 ports QDR 720 Gb s for 36 ports DDR 720 Gb s for 18 ports QDR 360 Gb s for 18 ports DDR Mellanox Technologies 53 Rev 3 4 Table 11 IS5030 Specification Data Physical Power and Environmental Size 185030 B short IS5030 S Standard i e long Weight Short Standard Mounting Air Flow SerDes Speeds Connector Types 1 716 1U H x 17 17 W x16 84 D 43 6mm X 436 2mm X 427 7mm 1 716 1U H x 17 17 W x24 71 D 43 6mm X 436 2mm X 627 7 mm 15 3 lbs 6 94kg 1 PSU 17 2 lbs 7 80kg 2 PSUs 19 711bs 8 94kg 1 PSU 21 6lbs 9 797kg 2 PSUs 19 Rack mount 94 0 CFM 10 20 or 40 Gb s per port Input Voltage Power Consumption 405EXR management Passive cables Active cables QSFP Max Power per port for active cables Temperature Humidity Shock and Vibration 100 240 VAC 50 60Hz Typical Max in W 118 30 147 46 190 59 243 86 For second power supply add 7 6 W 2 0W 10 to 45 Celsius 10 90 non condensing ETSI EN 300 019 2 2 1999 09 InfiniBand In Band Management QDR Environmental Acoustic Sound power level QSFP Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 40Gb s Safety US Canada cTU Vus 20Gb s 10Gb s EU IEC60950 QoS International CB 8 InfiniBand Virtual Lanes for all EMC Emissions Manage
12. 36 ports Passive Cables Active Cables QSFP Max Power per port for active cables Temperature Humidity Shock and Vibration 100 240 VAC 50 60Hz Typical Max in W 122 85 153 02 196 14 249 41 For second power supply add 7 6 W 2 0W 10 to 45 Celsius 10 90 non condensing ETSI EN 300 019 2 2 1999 09 Device management Agents for full InfiniBand In Band Management QDR Environmental Acoustic Sound power level QSFP Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 40Gb s Safety US Canada cTU Vus _ 20Gb s 10Gb s EU IEC60950 Qos International CB Management 8 InfiniBand Virtual Lanes for all EMC Emissions ports USA FCC Class A Canada ICES Class A Baseboard Performance and EU EN55022 Class A Data Rate EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 76 dB A 56 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 CHUTE ets Reliability Availability and Serviceability Features Switching Performance Simultaneous wire speed any port to Hot Swappable Fan Module and NT any port Power Supplies Switching Capacity yP 1 1 Redundant RP 1 44Tb s for 36 ports QDR 720 Gb s for 36 ports DDR
13. Figure 7 155030 2BRC QSFP Power Side Panel 18 Figure 9 PSU Status LEDS gt su Ies na eb Oe diel ma We e Dos ee OS 18 Figure Os Reset Bullen nA NA erano a dos A WA Baca SE que die 20 Figur 10 Port NumbBring ex eek ey Rex xt y me Heaters Rae te 22 Figure 11 Top and Bottom Ports Upa oe act RU te e Ot ar 22 Figure 12 Management Interfaces 22 Figure 13 RJ45 IPC Gonneclor xxr a Gi ek e e dure 23 Figure 14 Rack Rail Kit Parts MIS00083 85 0 0 00 cee nenn 28 Figure 15 Rack Rail Kit Parts MIS000079 29 Figure 16 Installation Options Available for Short Switches 30 Figure 17 Installation Options Available for Standard Length Switches 30 Figure 18 For Short Switches Which Side of the Rack Do You Want the Connectors 31 Figure 19 Making Room for the Power Cord 32 Figure 20 Screwing on the Bracket een nennen 32 Figure 21 Screwing on the Rail orita pee toa dh he WSS eh 33 Figure 22 Inserting the Caged Nuts 33 Figure 23 Connect Bracket to Rack Vertical support 34 Figure 24 Ground Connections rer Peeters Sate bates cor ete eee eee ago etd 35 Figure 25 Two Power Inlets Electric Caution Notifications 36 Figure 26 Power Supply U
14. Install the rail slides placing the power cords through the slots Figure 45 Placing the Power Cords 68 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 46 Power Cord Through the Rail Slide Slot in the rail slide o 3 Tighten the bolts that hold the rail slides onto the rack to 9 2 Nm or 81 5 pound inches 4 Make sure the cords come out the other side ofthe rack 5 Place both brackets on the switch so that they extend forward of the switch Figure 47 Screwing on the Bracket In this position the front of the switch is even with the rack vertical suppo In this position the front of the switch is 5cm behind the rack vertical support Place to put the power cable In this position you will use 3 N flat head screws flat head screws This configuration also allows you to put the power cord through the bracket Mellanox Technologies 69 Rev 3 4 Figure 48 Making Room for the Power Cord Power cord Note The side of the switch with these brackets will be the side that is very close to the vertical rack support 1 Screw the rails onto the switch Use 5 flat head screws to connect each rail to the switch Figure 49 Screwing on the Rail 5 screws 2 Clip 4 more caged nuts into the holes in the rack you will be using to connect the brackets Check that
15. Section 6 May 2011 Rev2 2 e Changed Grounding section Removed note from Table 4 Changed unmanaged FW update procedure Oct 2010 Rev2 1 Added information regarding the MIS5031 port numbering and port activation Oct 2010 Rev2 0 e New power numbers Cable power level changed to 2 0W level 2 July 2010 Rev 1 9 1 Added QSFP Cable Power Budget Classification June 2010 Rev 1 9 e Added section Getting the License Added Note Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10 100M auto negotiation 8 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Table 1 Revision History Table Date Revision Description April 2010 Rev 1 8 Added appendix with instructions for bringing the power cord from one side of the switch to the other in a full rack Added Safety Warnings in Spanish April 2010 Rev 1 7 Removed DDR switches January 2010 Rev 1 6 Added instructions in the installation section regarding putting the power cord through the bracket Figure 15 and note on page 26 Removed DDR switches January 2010 Rev 1 5 Updates figures to include the power side I2C connector Added Section 3 3 2 5 DC Interface on the power side of the switch December 2009 Rev 1 4 Added Chapter 6 regarding FabricIT EFM licensing information Added Chapter 2Management modules and CPU connections Section 4 3 on unmana
16. Table 15 Replacement Parts Ordering Numbers o 75 Mellanox Technologies 7 Rev 3 4 Revision History Table 1 Revision History Table Date Revision Description November 2012 Rev3 4 e Added Note to Section 3 6 2 2 regarding lack of support for standard length switch installation with power side next to the vertical support Added Figure 17 and Figure 18 and text regarding lack of support for Stan dard installation with the connector side deeper in the rack Added CCC China Warning October 2012 Rev 3 3 Added Note to Section 4 3 regarding unmanaged switch July 2012 Rev3 2 e Removed references to CD and Quick Start Guide Added reference to Thank You page February 2012 Rev3 1 e Fixed view FW revision command Section 4 3 2 Added pull out tab section for externally managed switches December 2011 Rev3 0 Minor punctuation and grammar fixes June 2011 Rev2 9 e Removed references to paper clip Changes to reset button Section 3 1 2 Added warning do not use the switch as a shelf June 2011 Rev2 8 Changed three minute boot up time to 5 minutes June 2011 Rev2 7 Removed Flashing green from Table 4 June 2011 Rev2 6 e Removed products scheduled for End of Life Removed Flashing green from Table 4 May 2011 Rev2 5 Fixed power numbers May 2011 Rev2 4 Changes to Section 1 1 and Section 2 May 2011 Rev2 3 Changes to the management sections Removed
17. The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take pre cautions to guarantee proper ventilation in order to maintain good airflow at ambient temperature Cable routing in particular should not impede the air exhaust from the chassis 3 6 2 1 Minimum and Maximum Rack Depth for this Switch The short switch with the MIS000079 Rail kit can only go into a 19 rack whose vertical sup ports are between 380mm and 500mm apart dard depth or order the MIS000083 rail kit The both of these solutions will allow you to install the switch in a 19 rack whose vertical supports are between 500mm aa and 800mm apart To use the IS50XX in a rack deeper than 500mm order the IS50XX with the stan The standard depth switch uses the MIS000085 rail kit for installation in a 19 rack whose verti cal supports are between 500mm and 800mm apart Mellanox Technologies 27 Rev 3 4 Basic Operation and Installation 3 6 2 2 Installing the Switch in the Rack Tools and Customer Supplied Parts Phillips Screwdrivers 1 and 2 Grounding screw ESD strap Grounding wire sufficient to reach a valid ground ESD mat Figure 14 Rack Rail Kit Parts MIS00083 85 Bracket Md 4 o o 8 Rail C SL Rail slide e rn d Ing mg 2929 iu Ic In g kng ju em Make sure that the Rail kit is compatible with your rack For short depth switches rail kit MIS00083 is to
18. be used for racks from 50cm to 80cm deep and rail kit MIS00079 is to be used for racks from 38cm to 50cm deep including the iDataPlex rack For standard depth switches rail kit MIS00085 is to be used for racks from 60cm to 80cm deep Parts included in the rail kit 2 rails 18 recessed flat head screws 2 rail slides 8 caged nuts 2 brackets 8 pan head screws M6 28 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 15 Rack Rail Kit Parts MIS000079 o Oo o Rail Ba ESA Bracket A E o _ e d f Rail slide 3 y g amp 13 An Br Br Br Br Br c Cg ils D B dg B lo li es A ln Ga in Br Bing Br 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the rack vertical support Either the side with the power supply units or the side with the IB connectors can be even with one of the vertical rack supports Short switches have the OPN character B and are 1 716 IU Hx 17 17 W x16 84 D 43 6mm X 436 2mm X 427 7mm Standard switches have the OPN character S and are 1 716 IU H x 17 17 W x24 71 D 43 6mm X 436 2mm X 627 7 mm Things to consider before choosing where to mount the rails and rail slides Arr flow Configura
19. both sides of the switch power side and connector side are at the same level in the rack 3 Place the four bolts for the caged nuts within reach 4 Slide the switch into the rack and catch the rail slides into the rails Note Before putting the switch in place guide the power cords through the slots in the brackets Figure 50 Power Cord Through the Bracket 70 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 51 Power Side View y 1 Tighten the bolts to 9 2 Nm or 81 5 pound inches Figure 52 Connect Bracket to Rack Vertical support 2 Ground the switch 3 Plug in the power cables 4 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow status LEDs is cause for concern and must be dealt with immediately It can take up to 3 minutes to boot up during which time the status LED may indicate red 5 You can start connecting all of the cables to the switch Mellanox Technologies 71 Rev 3 4 N Appendix C QSFP Interface 20 21 22 23 24 25 26 27 28 29 30 81 32 33 34 38 36 37 EE el GND Rx2n Rx2p GND Rx4n Rx4p GND ModPrsL IntL VccTx Vec1 LPMode GND Tx3p Tx3n GND Txip Txin GND GND Rx1n Rx1p GND Rx3n Rx3p GND SDA SCL Vcc Rx ResetL ModSelL GND Tx4p Tx4n GND Tx2p Tx2n GND
20. connector side of the switch to the power side of the switch in a full rack Tools and Customer Supplied Parts Phillips Screwdrivers 1 and 2 Grounding screw ESD Strap Grounding wire sufficient to reach a valid ground ESD mat Parts included in the installation kit 2 rails 8recessed flat head screws 2 rail slides 4 caged nuts 8 pan head screws M6 2 metal washers Figure 34 Rack Installation Kit Parts Mellanox Technologies 61 Rev 3 4 Figure 35 Rack Rail Kit Parts MIS00083 85 Bracket Lc 4 o o o a Rail es Y 2 Q Rail slide Dp d B NG NG hy Lp Lp yy Before you install your new MTS3600 switch unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have occurred during shipping Note If anything is damaged or missing contact your customer representative immediately Procedure 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the vertical rack support Either the side with the power supply units or the side with the IB connectors can be even with the vertical rack support The other side of the switch will be furth
21. doit pas fonctionner dans une zone avec une temp rature ambi ante d passant le maximum recommand de 45 C 113 F En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouvertures de ventilation gt 3 Empilage du ch ssis Le chassis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut provoquer des blessures corporelles et des d gradations de biens gt 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lectrique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solide ment fix ou s il est enlev gt 5 Orages dangers electriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles gt 6 Branchement debranchement des c bles InfiniBand en cuivre Les cables InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour connaitre les mises en garde et les instructions sp ciales gt 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cau tions sp ciales pour s assurer que le syst me reste stable En g n ral il faut remplir la b
22. indicators are OFF then the fan unit is discon nected There are two possible air flows for the fan unit The air flow depends on the switch model An R in the model number indicates a reverse air flow See Air Flow on page 20 for an explanation of the model numbers and labels Make sure that the fans have the air flow that matches the model number An air flow opposite to the switch design will cause the switch to operate at a higher less than optimal temperature Figure 29 Air Flow Labels Mellanox These air flow labels must be the same To extract a Fan Unit 1 Using two hands push both latch releases simultaneously while pulling the fan module out of the switch As the fan unseats the fan status indicator will turn off 38 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 30 Fan Module Latches These two latches must be pushed towards each other at the same time while the module is pulled out To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely 3 7 Disassembly of the Switch from the Rack The green fan status indicator should light If not extract the fan unit and reinsert it After two unsuccessful attempts to install the fan unit power off the switch before a
23. instruc tions See Table 3 for details Table 3 Switch Management Managed Family Unmanaged Management Connections IS5025 Unmanaged Plug and play All firmware updates should be done in band using Mellanox Firm ware Management Tools 12C port access using MTUSB 1 device is required for firmware updates if in band burning is not possible IS5030 Managed RS232 cable DB9 to RJ45 included in the box to connect to host PC for initial configuration of the switch After initial configuration the switch can be managed through the ethernet using a remote connec tion 185035 Managed RS232 cable DB9 to RJ45 included in the box to connect to host PC for initial configuration of the switch After initial configuration the switch can be managed through the ethernet using a remote connec tion 14 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 3 Basic Operation and Installation 3 1 Switch Platform Hardware Overview Figure 4 shows the power side panel and connector side panel views of the switch The figure shows port configurations for the switch systems the dual hot swap power supplies and hot swap fan module Ethernet RJ45 connector RJ45 connector for connecting to a host PC USB connector and various status LEDs Unmanaged switches have an I2C RJ45 connector Figure 4 QSFP Switch System Power and Connector Side Panels Power Side Panel MGT CONSOLE AY AVAY AY AV
24. is a reset button This reset button requires a tool to be pressed DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object to press this reset button Mellanox Technologies 19 Rev 3 4 Basic Operation and Installation Figure 9 Reset Button Press the reset button to reset the main and management CPUs and to delete the existing password This button resets both the CPU of the switch device and the CPU of the management module It thereby resets all of the ports by bringing them down and powering them up when the button is pushed A quick push of this button performs this reset When the button is held down for 15 sec onds the switch is reset and the password is deleted You will then be able to enter without a pass word and make a new password for the user admin In the externally managed switch the reset button resets the CPU of the switch device 3 2 Air Flow These switches can come with two air flow patterns The two patterns are Connector side inlet to power side outlet Power side inlet to connector side outlet The air flow is specified in the product model number See Mellanox Part Numbering Legend on page 11 On the switch and fan modules the air flow direction can be seen on the power side panel 20 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Table 9 Air Flow Direction
25. la insta laci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Instalaci n en un lugar con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 3 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n gt 4 Apilamiento del chasis Los chasis no se deben apilar sobre otros equipos La ca da del chasis podr a causar lesiones corporales as como da os al equipo gt 5 Conexi n de fuente de alimentaci n redundante peligro de descarga el ctrica Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa est quitada o no est bien cerrada gt 6 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos gt 7 Montaje y mantenimiento de bastidores Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En gener
26. n a PSI D MT 0D00110012 Other tools like flint can also be useful 4 3 2 1 Instructions for Reprogramming Over the InfiniBand Network To update an InfiniScale IV switch device having a specific GUID for example 0x00000006660abcd0 or LID the following are the recommended steps to update the device firmware 1 Make sure all subnet ports are in the active state One way to check this is to run opensm the Subnet Manager root mymach gt etc init d opensmd start opensm start OK 2 Make sure the local ports are active by running ibv_devinfo 3 Obtain the device LID 1 Use the mst ib add command The mst ib add runs the ibdiagnet ibdiscover tool to discover the InfiniBand fabric and then lists the discovered IB nodes as an mst device under dev mst directory These devices can be used for access by other MFT tools rootemymach gt mst ib add Mellanox Technologies 45 Rev 3 4 Management and Tools Overview Discovering the fabric Running ibdiagnet skip all Added 10 in band devices 2 List the discovered mst inband devices run mst status root mymach gt mst status MST modules MST PCI module loaded MST PCI configuration module loaded nba nd devices dev mst CA MT25418 sw005 HCA 1 lid 0x0001 dev mst SW MT47396 lid 0x0011 dev mst SW MT48438 lid 0x0003 Identify the switch you want to update according to its description or its LID in the examples below we will u
27. A OPN T Picture De bonus Description Connector side inlet to power side R outlet Power side inlet to connector side outlet F 3 3 SFP Cable Power Budget Classification All MIS50XX QSFP switches are designed for a active cables with a max power per module of 2 W This is power level 3 according to the QSFP Public Specification 3 4 Interfaces 3 4 4 Port Connector Interfaces 3 4 1 1 36 Port Switches The Connector side of the switch has 36 QSFP ports These are placed in two rows 18 ports to a row The ports are labelled as shown in Figure 10 The bottom row ports are flipped from the top row See Figure 11 for bottom row top row port orientation Mellanox Technologies 21 Rev 3 4 Basic Operation and Installation Figure 10 Port Numbering 1 9 S y a 13 15 17 19 21 23 25 27 29 31 33 35 MSS PA aaa ee a Bus RSS EAE aaa aaa aaa cmc 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 3 4 1 2 Top and Bottom Orientation Figure 11 Top and Bottom Ports 3 4 2 Management and FW updating Interfaces There are three interfaces to connect to the IS50XX They are lor2 connectors labelled MGT Use these connectors to connect to the Ethernet The 185035 switch has two MGT connectors whereas the IS5030 has one 1 USB port that is labelled This interface can be used to update software or firm ware connector that is labelled CONSOLE Use this conne
28. AY AVY AVAY AY AVAY AV Y A AV AVvav ELEC RSS BE i URSS DNS PEA Connector Side Panel All InfiniBand connectivity is via the connector side panel All connectors can support active cables 3 1 1 Status LEDs 3 1 1 1 System Status Indicators The System Status Indicators are located to the left of the QSFP connectors on the connector side panel and labeled STATUS and on the power side at the far right Both of these LEDs give identical information Power side Connector side status LED status LED The system status indicators should display as follows When the switch is plugged in within five minutes the STATUS LED should light up green The PSU LED for the plugged in PSU should light up green The PSU LED for the second PSU should light up green only if a second PSU is installed in the switch for redundancy and Hot Swap ability and it is connected to a power source If two PSUs are installed and only one PSU is connected to a power sup Mellanox Technologies 15 Rev 3 4 Basic Operation and Installation ply the second PSU LED will be red If only one PSU is installed in the switch the sec ond PSU LED will be off As long as there is power to the switch one PSU is connected and the switch is booted up and running the status LED will be green 5 The FAN LED should light up green If the STATUS LED shows red after five minutes unplug the switch and call your Mel lanox representative for assi
29. DE Seide Ses eee 57 A 2 1 FCC Statements USA 57 A 2 2 EN Statements Europe lees 58 A 2 3 ICES Statements Canada nennen nn 58 A 3 China CCC Warning Statement 59 A 4 VCCI Statements Japan anne eee 59 A 5 MIC Certification Korea oo 60 Appendix B Passing the Power Cord From the Connector Side to the Power Side 61 Appendix C QSFP Interface 72 Appendix D RJ45 CONSOLE Interface nenne 74 Appendix E Replacement Parts Ordering Numbers 75 Appendix F Avertissements de s curit d installation French 76 Appendix G Installation Sicherheitshinweise German 78 Appendix H Advertencias de seguridad para la instalaci n Spanish 80 Appendix I Special Regulations Regarding Finland Sweden Denmark and Norway 83 4 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 List of Figures Figure te Bull Out Tab 213 sues d eek nat ates bord c eg dense OR ey ae E 12 Figure 2 Pull OulTaD amp 4u35 ot a ALA Ba PREMO E SI d Re S LHP eine 13 Figure 3 License Key Label urn Siege a RR x RITE E REY Hs 13 Figure 4 QSFP Switch System Power and Connector Side Panels 15 Figure 5 Power Fan and System LEDs 16 Figure 6 System Health LED rs 22a 2a King 17
30. InfiniBand Software and Drivers Click Mellanox OpenFabrics Enterprise Distribution for Linux MLNX OFED Select the Linux User s Manual The embedded OS and tools on the CPU in the management module is a subset of the Mellanox OFED stack FabricIT Enterprise Fabric Management Software CLI User s Manual See http www mellanox com gt Products gt Management Software Select FabricIT EFM InfiniBand Fabric Management Click Software and Documentation Download center select User Manual Mellanox Firmware The MFT Mellanox Firmware Tools package is a set of firmware tools Tools MFT The manual supplied with this package provides an overview of the firm User s Manual ware its installation and replacement The MFT can be downloaded with its Document 2329 documentation at http www mellanox com gt Support gt Download Firmware Tools Conventions Throughout this manual the name IS50XX and the term switch are used to describe both the 36 port QSFP 40Gb s InfiniBand Switch and the 36 port QSFP 20Gb s InfiniBand Switch unless explicitly indicated otherwise 10 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies IS System Type InfiniScale Switch 50 Model Family FF Form factor 25 36 Ports Unmanaged 30 36 Ports and Chassis Management 31 18 ports
31. Mellanox TECHNOLOGIES InfiniScale IV 36 Port QSFP 40 Gb s InfiniBand Switch User Manual P N IS5025D 1BRC IS5025Q 1BFC IS5025Q 1SRC IS5025Q 1SFC IS5025Q 1BRC IS5025Q 2SRC IS5025Q 2BRC 1S5030Q 1BFC IS5030Q 1SFC IS5030Q 1BRC IS5030Q 2BFC IS5030Q 2SFC IS5030Q 2BRC IS5035Q 1BFC IS5035Q 1BRC IS5035Q 2BFC IS5035Q 2BRC IS5035Q 1SFC IS5035Q 2SFC Rev 3 4 www mellanox com Rev 3 4 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTE
32. P KU AR A BEAPBKENSCECPHVET A 4 1 Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 5 MIC Certification Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and certification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL which is organized under the Ministry of Informa tion and Communications MIC EMC testing includes electromagnetic emissions EMI and susceptibility EMS Certified equipment is labeled with the MIC mark and certification num ber 9 NAT PELE AAA SES amp 71719 214 PAA EE NG AS e RE FIAN repu NY 3X Ga ET TATAE AAE 8822 AA HAYY Translation Class Device This device is registered for EMC requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced with a residential use type 60 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Appendix B Passing the Power Cord From the Connector Side to the Power Side This appendix will demonstrate the installation procedure to follow should you need to bring the power cords from the
33. Part Description OPN IS503X Power Supply Unit PSU Connector side to Power side air MIS000053 flow This Replacement part is for both the PSU 1 and PSU 2 IS503X Power Supply Unit PSU Power side to Connector side air MIS000054 flow This Replacement part is for both the PSU 1 and PSU 2 IS503X Power supply blank MIS000055 IS503X Rack installation kit standard depth switch MIS000085 IS503X Rack installation kit standard MIS000083 IS503X Rack installation kit short MIS000079 IS503X Fan Unit for Power side to Connector side air flow MIS000051 IS503X Fan Unit for Connector side to Power side air flow MIS000082 RS232 Cable RJ45 to DB9 Harness HAR000028 I2C DB9 or RJ45 to USB Adapter MTUSB I Power cord Type C13 C14 ACC000251 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany Netherlands Russia ACC000207 Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 Mellanox Technologies 75 Rev 3 4 Appendix F Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne
34. RRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TEEHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2012 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB FabricIT MLNX OSTM ScalableHPCTM Unbreakable LinkTM UFMTM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3133 QDR InfiniBand Switch Platform User Manual Rev 3 4 Table of Contents Table of Contents x 24 Sera tote lata sede goo echa ace d P e Bande s 3 LIST OF Figures coi reae Ed le CX RU alos tox crie E rca cdd tentes st 5 Listof Tables iaa AAA e HAKA 7 Revision History e Ee a A RR 8 About this Manual a ar xr aria a ne QE a 10 Intended Audience eres 10 Related Documentation 0000 00s 10 CONVENTIONS ca a a a sd LUN EN 10 Mellan
35. Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based The InfiniBand subnet manager OpenSM assigns Local IDentifiers LIDs to each port con nected to the InfiniBand fabric and develops a routing table based on the assigned LIDs A typical InfiniBand installation using the OFED package will run the OpenSM subnet manager at system start up after the OpenIB drivers are loaded This automatic OpenSM is resident in memory and sweeps the InfiniBand fabric approximately every 5 seconds for new InfiniBand adapters to add to the subnet routing tables Updating Firmware on Managed Switches When you buy the switch it comes with the latest firmware burned on the board All firmware updates should be done through the management software Go to the Mellanox Website and con firm that the firmware is the latest If not reburn the latest firmware from the download site New firmware versions will be posted on the Mellanox firmware download page http www mellanox com gt Support gt Download Center You will also need to download and unzip the firmware binary image This is provided in the Mellanox Web site at http www mellanox com gt Support gt Download Center and go to the InfiniScale IV Switch systems Click in the Table for the firmware image that you need 48 Mellanox Technologies QDR InfiniBand Switch Platform User Ma
36. aie avec du mat riel de bas en haut gt 76 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 8 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 9 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gis lations et r glementations nationales en vigueur 10 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 77 Rev 3 4 Appendix G Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromvers orgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel f tungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten gt 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunterf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redun
37. al en un bastidor los equipos se deben insta lar comenzando desde abajo hacia arriba gt 80 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 8 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente gt 9 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incendio gt 10 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales gt 11 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales gt 12 Cable de alimentaci n homologado por UL y con certificaci n CSA Fuga gt 3 5 mA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG terminado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros gt En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora ai
38. and Chassis Management 35 36 ports and Fabric Management C nfiniBand Port Config Q QDR D DDR ISeparator P Power Supplies 0 0 1 1 2 2 Depth of the Unit S standard depth B short depth Y Air Flow direc R Connector side to PSU side airflow tion F PSU side to Connector side airflow R RoHS C RoHS5 X RoHS6 Mellanox Technologies 11 Rev 3 4 Overview 1 Overview Mellanox IS50XX switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers High Performance Computing and Embedded environments Networks built with IS50XX systems can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 4th generation InfiniScale IV InfiniBand switch device IS50XX systems provide up to 40Gb s full bidirectional bandwidth per port With 36 ports these systems are among the densest switch ing systems available These stand alone switches are an ideal choice for top of rack leaf connec tivity or for building small to medium size clusters The switch comes pre installed with all necessary firmware and is configured for standard opera tion within an InfiniBand fabric This switch requires an InfiniBand compliant Subnet Manager running from one of the hosts or Fabric Management software running on the switch All that is required for normal operation is to follow the usual precautions for installat
39. as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables connected to the switch Do you want the connector side recessed in the rack to allow for a larger cable bending radius Will the connector side be recessed past other equipment in the rack and will this be prob lematic Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Note The distance between the vertical supports in the rack must be at least 10cm longer than the switch 1 Determine which side of the rack front or back will need the plug end and which side of the rack will need the socket end If you have only one power supply unit determine which side of the rack left or right will need the power cord Note The primary PSU is on the right side of the power side of the switch 1 Put the power cables through the 1U space for the switch Figure 42 Full Rack Note Make sure that they are extending from both sides of the rack 1 Clip the 4 caged nuts into the holes in the rack you will be using to connect the rail slides Check that both sides of the switch left and right are the same level in the rack Figure 43 Inserting the Caged Nuts a x The caged nuts are separated by e a single space Mellanox Technologies 67 Rev 3 4 Figure 44 Put the Cables Through the 1U Space 2
40. ction allows access to the switch for updating FW See Section 4 3 3 4 2 3 RJ 45 Ethernet Connector MGT Internally Managed Switches The Ethernet connection labelled MGT provides access for remote management The IS50XX can be connected to an Ethernet switch or the Ethernet port of a computer The switch with the FabricIt management module has an extra Ethernet connector below the MGT connector shown in Figure 12 Note These connector s are not found in unmanaged externally managed switches Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10 100M auto negotiation The unmanaged switches are Plug and Play and all firmware updates should be done in band The I2C connection should only be used if the FW image was corrupted to the point that the regular FW tools cannot successfully reburn the correct image Mellanox Technologies 23 Rev 3 4 Basic Operation and Installation 3 4 2 4 3 4 2 5 3 5 3 6 USB interface FabricIT EFM Web User Interface WebUT or FabricITTM EFM Command Line Interface CLI There is a single USB connector This connector can be used to install software and or firmware upgrades using a disc on key or similar memory device that has a USB connector This connector is not found on unmanaged externally managed switches 12C Interface There is an 12C connector on the far left of the power side of the switch This interfac
41. ctor side away from the door This configuration has more room for the cables and a larger bending radius This configuration is not supported for standard length switches 24 71 or 627 6 mm deep The figure below shows the connector side next to the door and the power side away from the door This configuration may be necessary to conform to your rack configuration 3100000000000000000000000000090 oooooooooookd 3 Screw the brackets onto the switch Use the flat head screws to connect the bracket There are two options for mounting the bracket One option will place the switch even with the vertical support of the rack and the second option will recess the switch further into the rack If you Mellanox Technologies 31 Rev 3 4 Rev 3 4 Basic Operation and Installation are using the second option insert the power cable before screwing the bracket to the vertical support If you need to bring the Power cord from the other side of the rack recess the switch and run the Power cord through the bracket also using the slot in the YN rail slide Go to Passing the Power Cord From the Connector Side to the Power Side on page 61 for detailed instructions on routing the power cord from the power side to the connector side Figure 19 Making Room for the Power Cord Power cord Figure 20 Screwing on the Bracket Option A Option B Option A puts the front of the switch Scm Option B puts
42. ctor to connect to the host PC Figure 12 Management Interfaces ONSO ONSO P NN P MN HM Internally Managed Internally Managed with chassis manager with FabricIT manager 3 4 2 1 RJ 45 Connector CONSOLE Internally Managed Switches only The port labelled CONSOLE is for a local host connection to the management module This is used the first time the switch is connected An HAR 000028 harness is included in the package to connect to a DB9 connection on a host PC Connecting to a local host PC and following the instructions in the Installation Guide Configuring the Switch for the First Time must be done before any remote management is available For the Socket pinout see RJA5 CONSOLE Inter face on page 74 This connector is not found in unmanaged externally managed switches 22 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 3 4 2 2 RJ 45 Connector I2C Externally Managed Switches only Figure 13 RJ45 I2C Connector All firmware updates should be done in band using Mellanox Firmware Management Tools This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only You will need to order an MTUSB 1 USB to I2C adapter to make use of the I2C interface An HAR 000022 harness is included in the MTUSB 1 package to connect the switch for updating FW The I2C connection provides access to Flash and EEPROMs This conne
43. ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 15 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el ctricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos A deshacerse de este producto y de todas sus partes h galo de una manera res ponsable y respetuosa con el medio ambiente 82 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Appendix Special Regulations Regarding Finland Swe den Denmark and Norway Denmark This unit is class I and must be connected with an AC cord compliant with all national electrical codes in Denmark The AC cord shall have an integral ground wire and can only be plugged into a fully grounded outlet Do not connect this unit to any outlet that 1s not fully grounded Finland Laite on liitett v suojamaadoituskoskettimilla var ustettuun pistorasiaan Norway Apparatet ma tilkoples jordet stikkontakt Unit is intended for connection to IT power systems for Norway only Sweden Apparaten skall anslutas till jordat uttag Mellanox Technologies 83
44. d in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 3 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 4 Redundant Power Supply Connection Electrical Hazard This product includes a blank cover over the space for the redundant power supply Do not operate the product 1f the blank cover is not securely fastened or if it is removed 5 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables Mellanox Technologies 25 Rev 3 4 Basic Operation and Installation 6 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions 7 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 8 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel
45. e PSU airflow direction is different from the fan module airflow direction the system will shut down Figure 26 Power Supply Unit Extraction Latch Release 36 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 27 PSU Pulled Out Power Side SE Fan Unit Hi PSU1 To extract a PSU 1 Remove the power cord from the power supply unit 2 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PSU unseats the PSU status indicators will turn off 3 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Do not attempt to insert a PSU with a power cord connected to it 2 Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation 4 Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The green PSU indicator should light If not repeat the whole procedure to extract the PSU and re insert it 3 6 6 InfiniBand Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable press the con nector into the port receptacle until the connector is firmly seated The GREEN LED indicator
46. e fan unit is properly and completely inserted Figure 5 on page 16 shows the explanation of the PSU Status LED colors 18 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Table 7 PSU Status LED Configurations LED Color Status Green OK The Power supply is delivering the correct voltage 12VDC Red Error The PSU is not operational Off Off There is no power to the system neither PSU is receiving power If one PSU is showing green and the second PSU is unplugged it will show a red indica tion 3 1 1 5 Fan Status Indicators The indicator labeled Fan is located to the left of the QSFP connectors on the connector side panel The following fan status conditions are possible Table 8 Fan Status LED Configurations LED Configuration FAN LED Green OK The system is up and running Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan module Off Off The fan unit is not receiving any power Check that the fan unit is properly and completely inserted All fans must be operating while the power supply is plugged in y If the switch shuts down due to over temperature unplug the switch wait 5 minutes A and replug in the switch For more information See Troubleshooting on page 49 A 3 1 2 Reset Button On the connector side panel under the system LEDs
47. e is for Debug and Troubleshooting only This connector can be used to install firmware upgrades should the FW image be damaged and cannot be upgraded through a host PC or remotely This interface is for FAEs and advanced users only Package Contents Before you install your new IS50XX switch unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping The switch comes packed with the following items e 1 switch 1 power cable for each PSU Type B 6ft US 125V 10A chord See Replacement Parts Ordering Numbers on page 75 to order power cords for various countries A single power cord for each power supply unit can be ordered at no extra charge e 1 rail kit there are three options for the kit e MIS000079 kit for a short switch in a shallow rack 38cm to 50cm e MIS000083 kit for a short switch in a standard rack 50cm to 80cm e MIS000085 kit for a standard switch in standard rack 50cm to 80cm Note Kit MIS000085 is for standard depth switches only 1 harness HAR000028 Harness for 185025 185030 and 185035 switches Thank you page with pointers to relevant documents on the web and support page Note If anything is damaged or missing contact your customer representative immediately Switch Platform Installation and Operation Installation and initialization of the switch platform are
48. em Status LED for the Fans is Yellow The link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA and the switch 4 If media adapters are used check that the all connections are good tight and secure The activity LED does not come on Check that the Subnet Manager has been started The switch is off 1 Unplug the switch 2 Wait 5 minutes 3 Plug in the switch 4 If the switch does not come on check the power supplies Mellanox Technologies 49 Rev 3 4 Troubleshooting 5 6 7 If the switch comes on Use the FabricIT management CLI or Web GUI to determine the cause of the Shutdown Check the temperature Check the Fan status The switch is not working and unresponsive 1 Reset the switch If resetting the switch does not work 1 Qno e PAIS Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies Ifthe switch comes on use the FabricIT management CLI or Web GUI to determine the cause of the shutdown The last software update did not succeed l Connect the RS232 connector CONSOLE to a laptop 2 Push the reset button on the switch or management module 3 4 Choose the image to upload Only use image 1 or image 2 You will have 5 seconds t
49. er inside of the rack Things to consider before choosing where to mount the rails and rail slides 62 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 36 Which Side of the Rack Do You Want the Connectors bs The distance between the rack and the door can be as little as 4 cm on one side of the rack and as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables connected to the switch Do you want the connector side recessed in the rack to allow for larger cable bending radius Will the connector side be recessed past other equipment in the rack and will this be prob lematic 3 Install the rail slides onto the switch Place the end of the rail slide with the 90 angle on the side of the switch that you want to be even with the vertical support of the 19 rack Use four of the flat head screws for each rail slide There are two sets of holes in the rail slide Select the set of holes to either mount the switch closer or farther away from the rack vertical sup port Tighten the screws to 3Nm or 26 5 pound inches Figure 37 Screwing the Rail Slide onto the switch This side of the y switch will be next to the verti cal rack support 4 Clip the nuts into the holes in the rack you will be using to connect the rail slides Check that both sides of the switch are in the same position number on the rack Mellanox Technologies 63
50. g the Power Supply Unit 36 3 6 6 InfiniBand Cable Installation 37 3 6 7 Extracting and Inserting the Fan Unit 38 3 7 Disassembly of the Switch from the Rack 39 3 8 DISDOSal scu sa eene eth qu de DA eR ERAT Eas 40 Chapter 4 Management and Tools Overview 41 4 1 Network Management and Clustering Software 41 4 2 Internally Managed Switch System 41 4 2 1 Subnet Management 41 4 2 2 Fabric Management with UFM 42 4 2 3 Configuring the Switch for the First Time 43 Mellanox Technologies 3 Rev 3 4 4 2 4 Starting a Remote Connection to the Switch 43 4 2 5 Downloading Firmware llle 43 4 3 Unmanaged Externally Managed Switch 44 4 3 111 12G Connector 2 4 2200 rar AG 44 4 3 2 Current Firmware Revision eese 44 4 3 3 How to Get Mellanox Firmware Tools MFT 48 4 3 4 OPEeN SM na A es SLE 48 4 4 Updating Firmware on Managed Switches 48 Chapter 5 Troubleshooting 49 Appendix A Specification i uses nx ERR das ue ete dede 52 1 EMG Certifications e e Rh RU eon RAE 57 A 2 EMC Statements 2 naa ns as ne age a ed
51. ged switches including The I2C connector should only be used when the FW cannot be updated in band October 2009 Rev 1 3 Fixed Depth of Standard switch in the Spec sheets Added RJ45 console pinout to appendix Fixed Status Fan LED info on page 13 Added to Troubleshooting section Updated power numbers Added IS5025Q 2SRC IS5025Q 2BRC model numbers September 2009 Rev 1 2 Added Spec sheets for IS5025 and IS5035 updated revision table September 2009 Rev 1 1 Updated power numbers September 2009 Rev 1 0 Initial Release Mellanox Technologies 9 About this Rev 3 4 Manual This manual describes the installation and basic use of the Mellanox IS50XX switch which is based on the InfiniScale IV InfiniBand switch device Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above The manual assumes familiarity with the InfiniBand Architecture Specification Related Documentation Additional Documentation available from Mellanox Table 2 Reference Documents Switch Firmware and Firmware Update Tools See http www mellanox com gt Support gt Download Firmware Tools Note that the Switch System described in this manual is based on Mellanox Technologies InfiniScale IV switch device Mellanox OFED Stack for Linux User s Manual See http www mellanox com gt Support gt
52. he single switch in the fabric the switch requires a host based management solution gt I2C Connector The I2C connection provides access to Flash and EEPROMs This connection allows access to the switch for updating FW when in band FW updating is impossible Unmanaged Externally managed switches that is the IS5025 switches do not get configured On unmanaged switches the CONSOLE Ethernet and USB connectors are not found Instead there is an I2C connector The I2C connector should only be used when the FW cannot be updated in band The unmanaged switches are Plug and Play and all firmware updates should be done in band The I2C connection should only be used if the FW image was corrupted to the point that the regular FW tools cannot successfully reburn the correct image Note The RJ45 to DB9 harness 22 in the box with the switch is for use with the MTUSB 1 adapter This is necessary only when updating FW cannot be updated in band This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only When you install the switch it comes with the latest firmware burned on the board All firmware updates should be done in band This is only done when you receive an email that a newer FW version for your switch is available Download the latest FW from http www mellanox com gt Downloads gt Firmware For instructions on downloading FW see http www mellanox com gt Downloads gt Firmwa
53. hes that is the IS5025 switches do not get configured On unmanaged switches the CONSOLE Ethernet and USB connectors are not found Instead there is an I2C connector See the Installation Guide of the ISSOXX switch Configuring the Switch for the First Time The port labelled CONSOLE must be connected to a local host PC This must be used the first time the switch is connected This must be done before any remote management is available Hook up the supplied harness cable HAR00028 from the connector labelled CONSOLE to the DB9 connector of the local host PC Figure 31 Host Connection A v CONSOLE Connect the host PC to here E 4 2 4 Starting a Remote Connection to the Switch 4 2 4 1 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet the WEB 4 2 5 Downloading Firmware Firmware for this switch system can be found at and downloaded from http www mellanox com gt Downloads gt Firmware Be sure to read and follow all of the instructions regarding the updating of the firmware on your switch system Firmware for the HCA cards connected to this switch system can be downloaded from the same site Mellanox Technologies 43 Rev 3 4 Management and Tools Overview 4 3 4 3 1 4 3 2 Unmanaged Externally Managed Switch d When using Mellanox 5025 Unmanaged switch as t
54. iBand SW Drivers This software package provides connectivity for server and storage systems utilizing High Per formance Computing HPC or enterprise data center EDC applications across an InfiniBand fabric It also provides a subnet manager for simple network configuration and network adminis tration and diagnostic tools for network management 4 2 internally Managed Switch System The managed switches come standard with management software module for chassis manage ment IS5030 1 switches are pre installed with a license of up to 108 nodes 185035 switch is pre installed with a license of up to 648 nodes If there is a need to support more than 648 nodes an optional license of Unified Fabric Manager UFMTM is needed Management modules will have the capability to allow remote monitoring and remote manage ment of the chassis from any host connected to the fabric The managed switch system includes a CPU which contains e embedded OS secure in band out band access chassis manager and system BIST SNMP agent 3rd party tool integration e GUI e Subnet Manager SM The chassis manager will give the user access to switch temperatures power supply voltages fanunit information power unit information Flash memory The manager also has the ability to burn new firmware on the switch 4 2 4 Subnet Management The Subnet management features include Mellanox Technologies 41 Rev 3 4 Management and Tools O
55. ion and to connect the switch to the HCAs Once connected the Subnet Management software automatically configures and begins utilizing the switch It is recommended that the Mellanox OpenFabrics software package be installed on all nodes connected to the ISS0XX The software package provides a subnet manager and network man agement tools as well as connectivity software for servers and storage and is available on the Mellanox web site See Chapter 3 for more information Basic installation hot swapping components and hardware maintenance is covered in Basic Operation and Installation on page 15 1 1 Serial Number and Product Version Information 1 1 1 Externally Managed Switches The Serial number and GUID for the switch are found on the pull out tab below the I2C RJ 45 connector Figure 1 Pull Out Tab S N MT0924X00355 Rev X1 JAN A P N MIS5025Q 1BFC UN AAN UN AU GUID 0002C9020040DDBO ae NANAMAN III Made in IL Pull out tab 1 1 2 Internally Managed Switches The Serial number and GUID for the switch and the MAC for the Management PC are found on the pull out tab below the USB interface connection The key for access to the software manage ment is pre installed on your system however in the event the factory defaults reset button is pressed this key may need to be re installed Please see the FabricIT EFM Software User Man 12 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4
56. me national and or local codes may require IT components to be bonded and exter 4 nally grounded not including the power cord ground You must follow all national n and local codes when installing this equipment 34 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 24 Ground Connection O O BS 3 6 4 Power Connections and Initial Power On The switch platform ships with one or two Power Supply Units For switches with only one unit installed a second PSU may be installed at a later time Each supply has a separate AC recepta cle The input voltage is auto adjusting for 100 240 VAC 50 60Hz power connections The power cords should be standard 3 wire AC power cords including a safety ground and rated for 15A or higher Caution The switch platform will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable N Caution After inserting a power cable and confirming the green system status B LED light is on make sure that the Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the switch make sure ALL LEDS are off to en
57. ment ports USA FCC Class A Canada ICES Class A Baseboard Performance and EU EN55022 Class A DR Device management Agents for full EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCL Class A EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 76 dB A 54 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Reliability Availability and Serviceability Scalability and Performance Features Switching Performance Simultaneous wire speed any port to Hot Swappable Fan Module and any port Power Supplies Switching Capacity 1 1 Redundant 1 44Tb s for 36 ports QDR 720 Gb s for 36 ports DDR 720 Gb s for 18 ports QDR 360 Gb s for 18 ports DDR Mellanox Technologies 55 Rev 3 4 Table 12 185035 Specification Data Physical Power and Environmental Size 185035 B short IS5035 S Standard i e long Weight Short Standard Mounting Air Flow SerDes Speeds Connector Types 1 716 1U Hx 17 17 W x16 84 D 43 6mm X 436 2mm X 427 7mm 1 716 1U Hx 17 17 W x24 71 D 43 6mm X 436 2mm X 627 7 mm 15 3 Ibs 6 94kg 1 PSU 17 2 lbs 7 80kg 2 PSUS 19 711bs 8 94kg 1 PSU 21 6lbs 9 797kg 2 PSUs 19 Rack mount 94 0 CFM 10 20 or 40 Gb s per port Input Voltage Power Consumption running
58. nit Extraction 36 Figure 27 PSU Pulled DUE s het 1 Sore Sethe NLANG ded 37 Figure 28 Top and Bottom Ports etorri PA wad a gene seins PANIG PpREe 5i 38 Figure 29 Air Flow Labels voies e isle Kae rod t mih de e eV ten eee RUE NAGA 38 Figure 30 Fan Module Latches cor xy ERR EIER ARE 39 Figure 31 HOSE CONMECHOMN oce Lem reo ee Ri 43 Figure 32 MTUSB 1 with Gables rs ria a 47 Figure 33 12C Cable Connected to IS5025 Haan nn nn 47 Figure 34 Rack Installation Kit Parts sce er eR Rr RR RR 61 Figure 35 Rack Rail Kit Parts MIS00083 85 62 Figure 36 Which Side of the Rack Do You Want the Connectors 63 Mellanox Technologies 5 Rev 3 4 Figure 37 Screwing the Rail Slide onto the switch 63 Figure 38 Caged Nut Spacing eee set RR eed re ER UR ad Pes 64 Figure 39 Serewing in the Rails aed RE ER ERU ERAT AREE SS Ru d 64 Figure 40 Rack Rail Kit Parts MIS000079 eee 65 Figure 41 Which Side of the Rack Do You Want the Connectors 66 Figure 42 Full Hack pesa RE dec eds A oe devo ERE INO YU 67 Figure 43 Inserting the Caged Nuts H nenn eee nenn 67 Figure 44 Put the Cables Through the 1U Space 68 Figure 45 Placing the Power Cords 68 Figure 46 Power Cord Through the Rail Slide
59. nual Rev 3 4 5 Troubleshooting As soon as a switch is plugged in make sure that the green power LEDs on the PSUs are on Status LED and or Status Health LED If either of these two LEDs is red unplug the switch and call your Mellanox representative Power supply unit If the LED on the PSU is not lit or is red check that the power cable is plugged into a working outlet 1 Check that the power cable has a voltage within the range of 100 240 volts AC 2 Check that the air flow direction of the PSUs are consistent with the Fan module air flow 3 Remove and reinstall the power cable 4 Remove and reinstall the PSU The power LED for the switch shuts off 1 Check that the there is adequate ventilation Are the fan LEDs showing that the fans are all up and running 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner The green power LED for the fans does not come on 1 Check that the Power LEDs are on 2 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 3 6 7 Extracting and Inserting the Fan Unit on page 38 Caution Do not run the switch if the Syst
60. o stop the U Boot by pressing Control B U Boot 2009 01 minx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location 12C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No 12C ready DRAM 2 GB ECC enabled 400 MHz CL3 FLASH 16 MB NAND 1024 MiB PCI Bus Dev Venld Devid Class Int PCIEO link is not up PCIE1 successfully set as root complex 01 00 15b3 bd34 0c06 00 Net ppc 4xx ethO ppc 4xx eth1 Hit Ctrl B to stop autoboot 0 Mellanox Fabricl T Boot Menu 1 EFM PPC M460EX EFM 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM PPC M460EX EFM 1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt Choice 50 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 5 Select the image to boot Mellanox Technologies 51 Rev 3 4 Appendix A Specification Table 10 IS5025 Specification Data Physical Power and Environmental Size IS5025 B short IS5025 S Standard i e long Weight Short Standard Mounting 1 716 1U Hx 17 17 W x16 84 D 43 6mm X 436 2mm X 427 7mm 1 716 10 Hx 17 17 W x24 71 D 43 6mm X 436 2mm X 627 7 mm 15 091 Ibs 1 PSU 6 845 kg 16 981 Ibs 2 PSUs 7 702 kg 19 5 Ibs 1 PSU 8 845kg 21 39 Ibs 2 PSUs 9 702kg Input Voltage Power Consumption one power supply 36 p
61. ort QDR no management CPU Passive Cables Active Cables QSFP Max Power per port for active cables 100 240 VAC 50 60Hz Typ ical Max 127 94 136 35W 200 23 232 74W For second power supply Air Flow 55 19 Rack mount REN add 7 6 W SerDes Speeds Humidity 94 0 CFM 2 0W Connector Types Shock and 10 20 or 40 Gb s per port Vibration 10 to 45 Celsius QSFP 10 90 non condensing ETSI EN 300 019 2 2 1999 09 Protocol Support Regulatory Compliance 52 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 InfiniBand Auto Negotiation of 40Gb s Safety US Canada cTU Vus 20Gb s 10Gb s EU IEC60950 QoS International CB 8 InfiniBand Virtual Lanes for all EMC Emissions Management ports USA FCC Class A Canada ICES Class A Baseboard Performance and Device EU EN55022 Class A Data Rate management Agents for full Infini EU EN55024 Class A Band In Band Management EU EN61000 3 2 Class A QDR Environmental EU EN61000 3 3 Class A Japan VCCI Class A EU IEC 60068 2 64 Ran Acoustic dom Vibration EU IEC 60068 2 29 Sound power level Shocks Type I MI EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 76 dB A Reliability Availability and Serviceability Scalability and Performance Features Switching Performance Simultaneous wire speed any port to Hot Swappable Fan Module and any port Switching Capacity
62. ox Part Numbering Legend 11 Chapter 1 Overview icc cece eee dax 12 1 1 Serial Number and Product Version Information 12 1 1 1 Externally Managed Switches 12 1 1 2 Internally Managed Switches 12 TIS License Key os senec ee Ire LSU Ure e RE rer EE RU 13 Chapter 2 Internally Managed vs Externally Managed 14 Chapter 3 Basic Operation and Installation 15 3 1 Switch Platform Hardware Overview 15 Sh Stalus LEDS nex Tur o esten eee ema s AER Vase sch ae 15 3 152 Reset Button oem baled airada uui 19 32 SAI EIOW aan erre a tp b EXP PUER ed e ees 20 3 3 QSFP Cable Power Budget Classification 21 3 4 Interfaces nase me UR EROR ARE at Re 21 3 4 1 Port Connector Interfaces 21 3 4 2 Management and FW updating Interfaces 22 3 5 Package Contents os ase ee TES WE Rer ERE ERE 24 3 6 Switch Platform Installation and Operation 24 3 6 1 Installation Safety Warnings 25 3 6 2 Mechanical Installation 26 3 6 3 Grounding the Switch 34 3 6 4 Power Connections and Initial Power On 35 3 6 5 Extracting and Insertin
63. rackets 8 pan head screws M6 Figure 40 Rack Rail Kit Parts MIS000079 o O O o gt o o o n 2 _ Rail 0 Bracket gt Y po Mo MT d m a Rail slide a ly 2 D fir Py Py les 2 G fen NG Dir Dis Drs Ds na a NG l 9 Er la Pr a Bing by g by g Er B1 A 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want close to the rack vertical support Either the side with the power supply units or the side with the IB connectors can be close to one of the ver tical rack supports Things to consider before choosing where to mount the rails and rail slides Mellanox Technologies 65 Rev 3 4 6 6 Figure 41 Which Side of the Rack Do You Want the Connectors The figure above shows the power side next to the door and the connector side away from the door This configuration has more room for the cables and a larger bending radius The figure below shows the connector side next to the door and the power side away from the door This configuration may be necessary to conform to your rack configura 00000000 fWiIkfQ0000000000000000000000000009 The distance between the rack and the door can be as little as 4 cm on one side of the rack and
64. re Tools Be sure to download the user manual appropriate to your OS Read the instructions in the User manual for the FW update pro cedure Current Firmware Revision The user can query for the currently loaded firmware revision by using the command root eaglel mst start Starting MST Mellanox Software Tools driver set Loading MST PCI module Success Loading MST PCI configuration module Success Create devices root eaglel mst ib add Discovering the fabric Running ibdiagnet skip all mst l Added 14 in band devices root eaglel mst status MST modules 44 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 MST PCI module loaded MST PCI configuration module loaded MST devices dev mst mt26428 pciconfO PCI configuration cycles access bus dev fnz04 00 0 addr regz88 data reg 92 Chip revision is BO dev mst mt26428 pci crO PCI direct access bus dev fn 04 00 0 bar 0xca200000 size 0x100000 Chip revision is BO Inband devices dev mst CA MT25418 eagle8 HCA 1 lid 0x0068 dev mst SW MT48438 155030 Ranana 2 lid 0x0001 dev mst SW MT48438 155030 Ranana lid 0x0002 root eaglel flint d dev mst SW MT48438 155030 Ranana lid 0x0002 qq q W Running quick query Skipping full image integrity checks Image type FS2 FW Version 7 4 2200 Device ID 48438 Description Node Sys image GUI Ds 0002c902004429e8 0002c902004429eb Board ID n a MT 0D00110012 VSD
65. ription Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed data transfer speed Orange Physical link up Flashing Orange A problem with the physical link Usually the SM is down or not enabling the port Mellanox Technologies 17 Rev 3 4 Basic Operation and Installation 3 1 1 4 Power Supply Status Indicators The IS50XX 36 Port Switch is available with one or two factory installed Power Supply Units For switches with only one unit installed a second Power Supply Unit can be added to increase security hot swap ability and to add redundancy See Section E Replacement Parts Ordering Numbers on page 75 for ordering part numbers Figure 7 IS5030 2BRC QSFP Power Side Panel PSU status LED Fan status LED PSU status LED s A PSU 2 PSU 1 Secondary Power Fan Unit Primary Power Supply Unit Supply Unit The primary power supply unit PSU1 is located on the right side of the power side panel with PSU2 on the left side Each PSU has a single 2 color LED on the right side of the PSU that indi cates the internal status of the unit Figure 8 PSU Status LEDs Table 6 PSU Status LED Configurations LED Configuration FAN LED Green OK The system is up and running Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan module Off Off The fan unit is not receiving any power Check that th
66. se SW MT48438 lid 0x0003 as our target switch Query the switch to determine the current firmware version root mymach gt flint d dev mst SW MT48438 lid 0x0003 q Image type FS2 firmware Version 7 3 0 Device ID 48438 Chip Revision AO Description Node Sys image GUI Ds 0002090200417 7d8 0002c902004177db Board ID n a MT 0D00110012 VSD n a PSID MT 0D00110012 Burn the Infiniscale IV switch flint d dev mst SW MT48438 lid 0x0003 i fw 1S4 rel 7 4 0 Q Al bin qq b 4 3 2 2 Reprogramming the Switch Through The I2C Port This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only P The I2C connector should only be used when the firmware cannot be updated in band hs Use the MTUSB 1 adapter available from Mellanox to reprogram the switch firmware using a server or PC running MFT 46 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Figure 32 MTUSB 1 with Cables ee The MTUSB adapter comes with three cables but has only two connections for cables One cable is an I2C cable with DB9 female connectors on both ends One cable is a I2C cable with DB9 female connector to RJ45 male Use the cable that matches your switch The other is a USB cable with one type A host connection to be connected to the server or PC and a type B connec tion to be connected to the MTUSB adapter Figure 33 I2C Cable Connected to 185025 With the
67. slante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nomi nal 10 A ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n Mellanox Technologies 81 Rev 3 4 13 A adir conexi n a tierra N Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suministrar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encar gara de instalar un CONDUCTOR DE CONEXI N A TIERRA DE PROTEC CI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 14 C digos de instalaci n Este dispositivo se debe instalar conforme a la versidn mas reciente de los c digos el
68. stance If the FAN LED shows red troubleshoot the fan module If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See Troubleshooting on page 49 If the PSU LEDs are not green this indicates a problem with the power supplies Only run the switch if at least one of the PSU LEDs is green Figure 5 Power Fan and System LEDs 16 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 Table 4 System Status LED Configurations LED Configuration STATUS System Health LED Green OK The system is up and running Yellow Error A fault in the system most likely the firmware did not BOOT properly Red Major Error Possible damage can result to the switch Turn off immedi ately for example bad FW can t boot overheated Off Off The system has no power 3 1 1 2 Power Side Panel System LED On the right side of the power side panel is a system LED that displays the health of the switch This indicator is the same as the system status indicator on the other side of the switch Figure 6 System Health LED 3 1 1 3 Port Connector LED assignment Above the ports are two LEDs one for the upper port and one for the lower port V The fol lowing table shows the port status according to the LED indication Table 5 Connector Physical and Logical Link Indications LED Desc
69. straightforward processes requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment The unmanaged externally managed switch platform does not require any programming or con figuration to operate as a basic InfiniBand switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software 24 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 The managed switch platform requires initial configuration to operate as an InfiniBand switch All internally managed switches come with an internal PPC based management board This board allows for internally managing the switch through a host PC or remotely through the Ether net 3 6 1 Installation Safety Warnings For Safety Warnings in French see Section FE Avertissements de s curit d installation French on page 76 for German see Section G Installation Sicherheitshinweise German on page 78 and for Spanish see Section H Advertencias de seguridad para la instalaci n Span ish on page 80 For special regulations regarding Finland Sweden Denmark and Norway see Section I Special Regulations Regarding Finland Sweden Denmark and Norway on page 83 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operate
70. sure a powered down status Mellanox Technologies 35 Rev 3 4 Basic Operation and Installation Do not hot swap the power supply if your switch has only one power supply You must power down the system to replace the power supply unit when there is only one PSU in the switch Figure 25 Two Power Inlets Electric Caution Notifications CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Risque de choc et de danger hazard The two PSUs are inde Schocks Entferrnen des e lectriques Le de branch pendent Netzsteckers elnes Netz ment d une seule alimentation teils spannungsfrei Um stabilise e ne de branch Disconnect all power supplies to alle Einhieten spannungs uniquement qu un module ensure a powered down state frei zu machen sind die Alimentation Stabilise e inside of the switch platform Netzstecker aller Netzteile Pour isoler completement le zu entfernen module en cause Il faut de brancher toutes les alimen tations stabilise es 3 6 5 Extracting and Inserting the Power Supply Unit With both power supplies installed in the redundant configuration either PSU may be extracted without bringing down the system Make sure that the PSU that you are NOT replacing is showing all green for both the PSU and status indicators Power supply units have directional air flows similar to the fan module The Fan module airflow must coincide with the airflow of all of the PSUs If th
71. switch connected to a computer through the MTUSB it is now possible to reprogram the SPI Flash memory 4 3 2 3 Instructions for Reprogramming Through the 12C Port 1 Make sure that the MTUSB I2C cable is connected to the switch and USB cable is connected to a com puter MFT is running on the computer mtusb 1 mst device appears in mst status 2 Set the switch I2C switch to allow access by running gt mlxi2c d dev mst mtusb 1 s IS5025 p IS4 3 Run the burn command flint d dev mst mtusb 1 i fw IS4 rel 7 0 142 Q Al bin qq b Mellanox Technologies 47 Rev 3 4 Management and Tools Overview 4 3 3 4 3 4 4 4 4 Power cycle the switch by unplugging and re plugging the power cord to load the new firm ware How to Get Mellanox Firmware Tools MFT Mellanox Firmware Tools MFT and documentation are available for download via http www mellanox com gt Downloads gt Firmware Tools The MFT kit includes mlxburn flint spark Bspark debug utilities See Related Documentation on page 7 Open SM To manage the Mellanox switch system using OFED download Mellanox Open Fabrics from http www mellanox com gt Downloads gt InfiniBandS W Drivers Be sure to read and follow all of the instructions regarding the installation and use of these tools Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand network requires a
72. the front of the switch behind the rack vertical support even with the rack vertical support Place to put the power cable Option A will require 3 flat head Screws Option B will require 4 flat head Screws This configuration also allows you to put the power cord through the bracket 32 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 The side of the switch with these brackets will be the side that is even with or very close to the vertical rack support 4 Screw the rails onto the switch Use 5 flat head screws to connect each rail to the switch Figure 21 Screwing on the Rail 5 screws 5 Clip the 4 caged nuts into the holes in the rack you will be using to connect the rail slides Check that both sides of the switch left and right are the same level in the rack Figure 22 Inserting the Caged Nuts 2 The caged nuts are separated by a single space e 6 Clip 4 more caged nuts into the holes in the rack you will be using to connect the brackets Check that both sides of the switch power side and connector side are at the same level in the rack 7 Using two of the bolts for each rail slide install the rail slides 8 Tighten the bolts to 9 2 Nm or 81 5 pound inches If the power cable is on this side of the switch feed the power cable into the slot in the rail slide before screwing it to
73. the vertical sup port Mellanox Technologies 33 Rev 3 4 Basic Operation and Installation Figure 23 Connect Bracket to Rack Vertical support 9 Place the four bolts for the caged nuts within reach 10 Slide the switch into the rails 11 Put the switch into place and screw the bolts into the nuts from step 6 If the power cable is on this side of the switch feed the power cable into the slot in the bracket before screwing it to the vertical support Tighten the bolts to 9 2 Nm or 81 5 pound inches 12 Tighten all of the screws to 9 2 Nm or 81 5 pound inches 13 Ground the switch 14 Plug in the power cables 15 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow status LEDs is cause for concern and must be dealt with imme diately It can take up to 5 minutes to boot up during which time the status LED may indicate red 16 You can start connecting all of the cables to the switch 3 6 3 Grounding the Switch Check to determine if your local or national electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter So
74. tion of already installed equipment Cable bending radius The distance between the rack and the door can be as little as 4 cm on one side of the rack and as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables con nected to the switch Do you want the connector side recessed in the rack to allow for a larger cable bending radius Will the connector side be recessed past other equipment in the rack and will this be prob lematic Mellanox Technologies 29 Rev 3 4 Basic Operation and Installation Figure 16 Installation Options Available for Short Switches Doors Doors y Connector side Spine side f 19 482 6mm 19 482 6mm Switch Spine side Doors Doors Figure 17 Installation Options Available for Standard Length Switches 24 71 or 627 6mm deep Doors Door Connector side Spine side Switch Spine side See Connector side Doors Doors Installation A Installation B Standard switches only support installation with the connector side of the switch near the N rack door as per installation B in Figure 17 30 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Figure 18 For Short Switches Which Side of the Rack Do You Want the Connectors The figure above shows the power side next to the door and the conne
75. ttempting any system debug To disassemble the switch from the rack 1 Unplug and remove all connectors Unplug all power cords Remove the ground wire Unscrew the 4 bolts from the side of the switch with the bracket bracket qe p i 1 W Support the weight of the switch when you remove the screws so that the switch does N not fall 5 Slide the switch from the rack 6 Remove the rail slides from the rack 7 Remove the eight caged nuts Mellanox Technologies 39 Rev 3 4 Basic Operation and Installation 3 8 Disposal For proper disassembly instructions to comply with the WEEE Directive 2002 96 EC see the Mellanox website According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not dis posed of with regular household waste Dispose of this product and all of its parts in a responsible and environ mentally friendly way 40 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 4 Management and Tools Overview The Ethernet ports for remote management connect to Ethernet switches These switches must be configured to 10 100M auto negotiation 4 1 Network Management and Clustering Software Download and install on all nodes the Mellanox OpenFabric software package for Linux Win dows or other operating systems from the Mellanox software website http www mellanox com gt Downloads gt Infin
76. verview upgrading drivers upgrading software monitoring of AC power to the PSUs DC power out from the PSUs board temperature fan module unit failure in the switch system system failure in the switch system querying for board serial numbers and their revisions In addition the tools enable firmware management capabilities such as querying for existing firmware versions burning new firmware from scratch or for recovery from damaged firmware querying for and changing system GUIDs checking for duplicate or bad GUIDs 4 2 2 Fabric Management with UFM For an added licensing fee you can purchase Mellanox s Unified Fabric UFM is a powerful plat form for managing scale out computing environments it enables data center operators to effi ciently monitor and operate the entire fabric boost application performance and maximize fabric resources utilization UFM provide a central management console including the following main features Fabric dashboard including Fabric Congestion Map Advance Fabric wide real time and history monitoring Quality of Service Traffic Aware Routing Algorithm TARA Central Device Management Task automation Threshold based alerts Fabric segmentation isolation Logging High Availability 42 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 3 4 4 2 3 Configuring the Switch for the First Time Unmanaged Externally managed switc

Download Pdf Manuals

Related Search

Related Contents

Serie 627 - Welcome to Emerson Process Management  Proyector Híbrido Láser LED Full HD de 1080p  MXSP-WP2000 取扱説明書  RASS-R Radar Comparator Mono    HD Media Wonder  to the User Manual.  IEEE 802.11b PCI Adapter User's Guide  ソキア 三次元測定システム MONMOS NET1200  BIP 129 du 18 Janvier au 1er Fu00E9vrier 2015  

Copyright © All rights reserved. Failed to retrieve file