Home

SwitchX Piranha 36 port switch.book

Contents

1. Mellanox Technologies 15 Rev 1 0 Basic Operation and Installation 4 Basic Operation and Installation 4 1 Switch Platform Hardware Overview Figure 3 shows the power side panel and connector side panel views of the managed switch The figure shows port configurations for the switch systems the dual hot swap power supplies and hot swap fan module 2 Ethernet RJ45 connectors for management 1 RJ45 connector for con necting to a host PC 1 USB connector and various status LEDs Unmanaged switches only have a single I2C RJ45 connector Figure 3 QSFP Switch System Power and Connector Side Panels A V SA V AV V A AC Connector Side Panel All connectivity except for power cords is via the connector side panel All connectors can support active cables 4 1 1 Status LEDs 4 1 1 1 System Status Indicators The System Status Indicators P ND are located to the left of the connectors on the connector side panel and on the power side at the far right Both of these LEDs give identical information 16 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Figure 4 System Status Indicators Green OK Yellow a fault in the sys tem Red Major Fault or Fatal Error Led Color shows the Switch status Power side Connector side status LED status LED The system status indicators should display as follows When the switch is plugged in within five minutes the STA
2. For special regulations regarding Finland Sweden Denmark and Norway see Section on page 70 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air b flow allow at least 8cm 3 inches of clearance around the ventilation openings 3 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 4 Redundant Power Supply Connection Electrical Hazard This product includes a blank cover over the space for the redundant power supply Do not operate the product if the blank cover is not securely fastened or if it is removed 5 During Lightning Electrical Hazard connect cables h During periods of lightning activity do not work on the equipment or connect or dis 6 Copper Cable Connecting Disconnecting Copper cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warnings and instructions 28 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 7 Rack Mounting and Servicing N 1 When this product is mounted or serviced in a rack
3. The following fan status conditions are possible Table 7 Fan Status LED Configurations LED Configuration FAN LED Green OK The system is up and running Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan module Off Off The fan unit is not receiving any power Check that the fan unit is properly and com pletely inserted All fans must be operating while the power supply is plugged in If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See Troubleshooting on page 48 Mellanox Technologies 21 Rev 1 0 Basic Operation and Installation 4 1 1 5 Bad PortLED Figure 11 Bad Port LED The Bad Port indicator is located on the left side of the connector side panel of the unit The fol lowing Bad Port conditions are possible Table 8 Bad Port LED Configurations LED Configuration Description Off OK All ports are up and running Flashing Orange Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are badcable bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 4 1 1 6 UID LED Switch Identifier The UID LED is a debug featur
4. 186W 104W 234W 129W 224W 119W MAX 2W TYP 1 5W 0 to 45 Celsius 5 95 non condensing Mellanox Technologies 53 Rev 1 0 Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 56Gb s 40Gb s Safety US Canada cTUVus 20Gb s 10Gb s EU IEC60950 International CB QoS 8 InfiniBand Virtual Lanes for all ports Russia GOST R Argentina S mark Management Baseboard Performance and Device China CCC for power supply management Agents for full InfiniBand Korea KN22 2209 In Band Management KN24 2009 Data Rate FDR FDR10 QDR DDR SDR EMC Emissions USA FCC Class A Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A Australia C Tick Environmental EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test Acoustic ISO 7779 ETS 300 753 Scalability and R Reliability Availability and Serviceability Features Switching Perfor mance Simultaneous wire speed any port to any Hot Swappable Fan Module port 1 1 Redundant Power Supplies Switching Capacity 2 016 Tb s for 36 ports FDR 1 65 Tb s for 36 ports FDR 10 1 44 Tb s for 36 ports QDR 720 Gb s for 36 ports DDR A 1 EMC Certification Statements See the Regulatory and Compliance Guide A 1 1 FCC Statements USA Class A Stateme
5. 2 bertemperatur f Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben Ah werden Auferdem sollten mindestens 8 cm 3 in Freiraum um die Bel f tungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Stapeln des Chassis A Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis Bi herunterf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redun dante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeck LA ung nicht sicher festsitzt oder entfernt wurde 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t L schlie en Sie keine Kabel an oder ab 6 AnschlieDen Trennen von Kupferkabel MN Kupferkabel sind schwer und nicht flexible Deshalb miissen sie vorsichtig an p DN die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziel y N len Warnungen und Anleitungen des Kabelherstellers 7 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind beson dere Vorsichtsmafnahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f ll
6. During fan hot swap if both indicators are OFF then the fan unit is disconnected There are two possible air flows for the fan unit The air flow depends on the switch model See Air Flow on page 23 for an explanation of the model numbers and labels Make sure that the fans have the air flow that matches the model number An air flow opposite to the switch design will cause the switch to operate at a higher less than optimal temperature Mellanox Technologies 39 Rev 1 0 Basic Operation and Installation Figure 26 Air Flow Labels IX 11 II II pI II 11 XI II IX 11 xr II Ir XX xr IX 11 II II xr Li XI II XX II XX IX xx xx xx XX XI II XX XI IX II II II XX XI IX II 139 II II II M XI XI XXI XI XI XI XI XI IX II IX IX XX II To extract a Fan Unit lellanox These air flow labels must be the same 1 Using two fingers push both latch releases towards each other simultaneously while pulling the fan module out of the switch As the fan unseats the fan status indicator will turn off Figure 27 II XI XI XI II II II XI XI 11 X r M M 2 XX II XI II XX 11 II II XX 11 Ff ER Le 2 xx 22 XXXI nm a a XI II XI XI II xr XI XI XI IX Xr XI Fan Module Latches XI XI XI XI XI XI XI XI II 11 11 II II II 4 XX II ED EE Le me 1 xx II XX 1 127 2 127 XX xr xx
7. On the PSU and fan modules the air flow direction can be seen on the power side panel m N N A All PSUs must have the same air flow as the fan module 4 N Mellanox Technologies 23 Rev 1 0 Basic Operation and Installation Table 9 Air Flow Direction Direction Label CEN Description Designation R Connector side inlet to power side outlet Power side inlet to connector side F outlet 4 3 QSFP Cable Power Budget Classification All SwitchX QSFP switches are designed for active cables with a max power per module of 2W Typical power per port is 1 5W 4 4 Interfaces 4 4 1 Port Connector Interfaces 4 4 1 1 36 Port Switches The Connector side of the switch has 36 QSFP ports These are placed in two rows 18 ports to a row The ports are labelled as shown in Figure 13 The bottom row ports are flipped from the top row See Figure 14 for bottom row top row port orientation 24 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Figure 13 Port Numbering 19 21 23 25 27 29 31 33 35 E ial ii En E Ee EE Eee 20 22 24 26 28 30 32 34 36 4 4 1 2 Top and Bottom Orientation Figure 14 Top and Bottom Ports a 4 4 2 Management and Firmware Updating Interfaces There are four interfaces to connect to the SX6036 2X 100M IGb Ethernet connectors labelled MGT USB port that is labelled This interface can
8. 1 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rx1n Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vcc Tx 3 3 V Power supply transmitter 30 Vec 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground 60 Mellanox Technologies Rev 1 0 SwitchX FDR InfiniBand Switch User Manual A GND fa n TX1p i GND j TX3n TX3p GND LPMode Vcc1 VccTx Intl ModPrsL 18 35 Top GND RX4p RX4n Connector GND View into Rear of RX2n sz GND N s S M GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL VccRx SCL spa GND RX3p RX3n GND RX1p RX1n GND HP 18 38 View into Front of Cage zo eh ek D Q N es gt O1 92 N CRE E 61 Mellanox Technologies Rev 1 0 Appendix D RJ45 CONSOLE and Ethernet Inter
9. Chapter 6 Troubleshooting 48 Appendix A Specification 51 EMC Certification Statements 54 VCCI Statements Japan 56 MIC Certification Korea 56 Appendix B Transferring the Power Cord 58 Appendix C QSFP Interface 60 Appendix D RJ45 CONSOLE and Ethernet Interfaces 62 Appendix E Replacement Parts Ordering Numbers 63 Appendix F Avertissements de s curit d installation French 64 Appendix G Installation Sicherheitshinweise German 66 Appendix H Advertencias de seguridad para la instalaci n Spanish 68 4 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 List of Figures Figure 1 Connector Side View of the Switch 11 Figure 2 Pull Out Tab 13 Figure 3 QSFP Switch System Power and Connector Side Panels 16 Figure 4 System Status Indicators 17 Figure 5 Power Fan and System LEDs 18 Figure 6 Port LEDs 18 Figure 7 Power Side Panel 19 Figure 8 PSU Status LEDs 20 Figure 9 Fan Status LED Connector Side 20 Figure 10 Fan Status LED Power side 21 Figure 11 Bad Port LED 22 Figure 12 Reset Button 23 Figure 13 Port Numbering 25 Figure 14 Top and Bottom Ports 25 Figure 15 Management Interfaces 25 Figure 16 RJ45 I2C Connector 26 Figure 17 Rack Rail Kit Parts 31 Figure 18 Mounting Options 33 Figure 19 Screwing on the Rail 34 Figure 20 Inserting the Caged Nuts 34 Figure 21 Slide the Rail into the Rail Slide 35 Figure 22 Two Power Inlets Electric Caution Notifications 37 Figure 23 Power Supply U
10. Technologies 29 Rev 1 0 Basic Operation and Installation 4 5 2 Mechanical Installation The switch platform can be rack mounted and is designed for installation in a standard 19 rack The power side of the switch includes a hot swap power supply module a blank cover for an optional second PSU for redundancy and a hot swap fan tray There are two possible air flow directions Be sure that the switch air flow direction is compatible with your system rack and PSUs The connector side of the switch has the QSFP ports system LEDs and management con nection ports The switch platform contains auto sensing 100 240 VAC connections for all possible PSUs The installer should use a rack capable of supporting the mechanical and environmental character istics of a fully populated platform The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take pre cautions to guarantee proper ventilation in order to maintain good airflow at ambient temperature Cable routing in particular should not impede the air exhaust from the chassis 4 5 2 1 Installation Kits There are two Installation kit options One long and one short Both the long and the short switches can be mounted using the long rail kit Table 10 Installation Kit According to Rack Size Kit OPN Rack Size MSX60 BKIT 40 60 cm MSX60 SKIT 60 80 cm 4 6 Package Contents and Installation Before you install your new SX60XX switch unpa
11. a los c digos el ctricos locales y nacio nales 12 Cable de alimentaci n homologado por UL y con certificaci n CSA Fuga 3 5 mA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG terminado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nomi nal 10 A ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n Mellanox Technologies 69 Rev 1 0 13 Anadir conexi n a tierra Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si
12. all national Some national and or local codes may require IT components to be bonded and exter and local codes when installing this equipment gt 4 6 3 Power Connections and Initial Power On The switch platform ships with one or two Power Supply Units For switches with only one unit installed a second PSU may be installed at a later time Each supply has a separate AC receptacle The input voltage is auto adjusting for 100 240 VAC 50 60Hz power connections The power cords should be standard 3 wire AC power cords including a safety ground and rated for 15A or higher Caution The switch platform will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable Caution After inserting a power cable and confirming the green system status LED light is on make sure that the Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the switch make sure ALL Connector LEDS are off to ensure a powered down status 36 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Do not hot swap the power supply if your switch has only one power supply You must power down th
13. firmly seated The GREEN LED indicator corresponding to each data port will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connec tor plugged into a functioning port After plugging in a cable lock the connector using the latch 38 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 ing mechanism particular to the cable vendor When a logical connection is made the yellow light will come on When data is being transferred the green light will blink Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row Figure 25 Top and Bottom Ports EE To remove disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Care should be taken not to impede the air exhaust flow through the ventilation holes next to the connector ports Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 4 6 6 Extracting and Inserting the Fan Unit This switch can operate indefinitely with one of the three fans in the fan module inoperable so long as the ambient temperature is below 45 Celsius Operation without a fan unit should not exceed two minutes
14. is based on the SwitchX IB switch device Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above The manual assumes familiarity with the InfiniBand Architecture Specification and Related Documentation Additional Documentation available from Mellanox Table 2 Reference Documents Document Name Description InfiniBand Architecture Specification Vol 1 Release 1 2 1 The InfiniBand Architecture Specification that is provided by IBTA Switch Hardware Installation Guide Each Mellanox Technologies switch platform is shipped with an Installation Guide document that contains instruc tions for Installation in a rack and initialization of the switch platform Mellanox Firmware Tools MFT User s Manual The MFT Mellanox Firmware Tools package is a set of firmware tools The manual supplied with this package pro vides an overview of the firmware its installation and replacement The MFT can be downloaded with its documentation according to the operating system at http www mella nox com gt Support gt Download Firmware Tools Unified Fabric Manager UFM The user manual can be downloaded together with the soft ware package upon license purchase Mellanox MLNX OS SwitchX Software User Manual This document contains information regarding configuring and managing Mellanox Technolo
15. la toma el ctrica de la que se suministrar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el TECNICO DE SERVICIO se encar gar de instalar un CONDUCTOR DE CONEXI N A TIERRA DE PROTEC CI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 14 C digos de instalaci n A Este dispositivo se debe instalar conforme a la versi n m s reciente de los p c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo 9 El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 15 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el ctricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos mm Al deshacerse de este producto y de todas sus partes h galo de una manera responsable y respetuosa con el medio ambiente 70 Mellanox Technologies
16. orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 6 Branchement d branchement des c bles en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour connaitre les mises en garde et les instructions sp ci ales 7 Montage et entretien sur baie f Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cau 2 tions sp ciales pour s assurer que le syst me reste stable En g n ral il faut d remplir la baie avec du mat riel de bas en haut 64 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 8 Installation du mat riel IN Ce mat riel ne doit tre install remplac ou entretenu que par du personnel B form et qualifi 9 Elimination du mat riel f L limination de ce mat riel doit s effectuer dans le respect de toutes les l gis lations et r glementations nationales en vigueur 10 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et L nationaux Mellanox Technologies 65 Rev 1 0 Appendix G Installation Sicherheitshinweise German 1 Installationsanleitungen N Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromver 24 sorgung anschlieBen
17. 7 Remove the eight caged nuts 4 9 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and elec tronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way PEERS Follow the instructions found in the Mellanox Web site at mellanox com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive Mellanox Technologies 41 Rev 1 0 Management and Tools Overview 5 Management and Tools Overview The switch to be managed either remotely or Out of Band using MLNX OS 5 1 Chassis Management Using the MLNX OS Software The SX6036 switches come standard with a management software module for chassis manage ment called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX based managed switch systems MLNX OS includes a CLI WebUI SNMP and chassis manage ment software and IB management software OpenSM The Ethernet ports for remote management connect to Ethernet switches These switches must be configured to 100M 1G auto negotiation The IB Subnet Manager running on the X6036 switches supports up to 648 nodes If the fabric includes more than 648 nodes you may need to purchase Mellanox s Unified Fabric Manager UFM software package The managed switch system includes the following software components embed
18. C Connector The I2C connection provides access to Flash and EEPROMs This connection allows access to the switch for updating firmware when in band firmware updating is impossible The I2C connector should only be used when the firmware cannot be updated in band The unmanaged switches are Plug and Play and all firmware updates should be done in band The I2C connection should only be used if the firmware image LA was corrupted to the point that the regular firmware tools cannot successfully reburn the correct image Note The RJ45 to DB9 harness 22 in the box with the switch is for use with the MTUSB 1 adapter This is necessary only when updating firmware cannot be updated in band This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only 5 5 2 Current Firmware Revision For Externally Managed Switches In order to obtain the Firmware version of the externally managed switch you need to run the fol lowing command from a host vendstat N Switch LID gt 5 5 3 Reprogramming the Switch Through The I2C Port This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only Mellanox Technologies 45 Rev 1 0 Management and Tools Overview The I2C connector should only be used when the firmware cannot be updated in band 2 Use the MTUSB 1 adapter available from Mellanox to reprogram the switch firmware using a server or PC running MF
19. FDRIO supports Forward Error Correction FEC 12 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 1 2 Serial Number and Product Version Information The Serial number and GUID for the switch and the MAC for the Management PC are found on the pull out tab below the USB interface connection Figure 2 Pull Out Tab S N MT1117X00014 ADA UN X P N MSX6036F 1BFR LOL LL GUID 0002C9020040DDB0 Rev A1 AIA Made in IL Z LOW T MINA MA 10 2 V LL 69 20 00 LOO0St6StLZLLL Pull out tab Mellanox Technologies 13 Rev 1 0 2 FDR FDR Mellanox SX60XX switch systems supports FDR a pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified physical layer using advanced block types deskew mechanism and framing rules for suporting higher speeds and increased reliability SX60XX switches also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with a 64b 66b encoding resulting in an effective band width of 40Gb s FDR10 supports 20 more bandwidth over QDR using the same QSFP cables connectors Both FDR and FDRIO supports Forward Error Correction FEC as described in IEEE 802 3ap chapter 74 FDR and FDR10 are only guaranteed to work with approved Mellano
20. I II Loo EX xx II XX XI XX II CIC CIC CIC CIC XI II XI II nm x XI XI XI XI II II XI IX IX II Latch Release Mellanox Technologies 37 Rev 1 0 Basic Operation and Installation Figure 24 PSU Pulled Out Power Side Fan Unit To extract a PSU 1 Remove the power cord from the power supply unit 2 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PSU unseats the PSU status indicators will turn off 3 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Do not attempt to insert a PSU with a power cord connected to it Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirm ing the proper installation Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The green PSU indicator should light If not repeat the whole procedure to extract the PSU and re insert it 4 6 5 Cable Installation FDR and FDR10 are only guaranteed to work with approved Mellanox Cables 5 All cables can be inserted or removed with the unit powered on To insert a cable press the con nector into the port receptacle until the connector is
21. Mellanox TECHNOLOGIES SwitchX 36 Port QSFP FDR InfiniBand System User Manual P N MSX6025F 1SFR MSX6025F 1SRR MSX6025T 1SFR MSX6025T 1SRR MSX6025F 1BFR MSX6025F 1BRR MSX6025T 1BFR MSX6025T 1BRR MSX6036F 1SRR MSX6036T 1SFR MSX6036T 1SRR MSX6036F 1SFR MSX6036F 1BRR MSX6036T 1BFR MSX6036T 1BRR MSX6036F 1BFR Rev 1 1 www mellanox com Rev 1 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEO
22. PR Data Transfer Protocol 60 InfiniBand FF Management Capabilities 25 36 Ports externally managed 36 36 Ports managed internally C Data Rate F EDR T FDR10 Q QDR D DDR Separator P Power Supplies 0 0 1 1 2 2 M Depth of the Unit S standard depth B short depth Y Air Flow direction Rz Connector side to PSU side airflow F PSU side to Connector side airflow X RoHS C RoHS5 R RoHS6 10 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 1 Overview Mellanox SX60XX 4 032 Tb s switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Performance Computing HPC and Embedded environments Networks built with SX60XX systems can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 5 generation switch device 1 Generation SwitchX VPI switch device SX60XX systems provide up to 56Gb s full bidirectional bandwidth per port With FDR ports these systems are among the fastest switching systems available These stand alone switches are an ideal choice for top of rack leaf connectivity or for building small to large size clusters The switch comes pre installed with all necessary firmware and is configured for standard opera tion within an InfiniBand fabric This switch requires an InfiniBand compliant Subne
23. RY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS and SwitchX are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners SwitchX FDR InfiniBand Switch User Manual 2 Mellanox Technologies Document Number 3410 SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Table of Contents Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience 8 Related Documentation 8 Conventions 9 Mellanox Part Numbering Legend 10 Chapter 1 Overview 11 1 1 Features 12 1 1 1 Full Feature List 12 1 2 Serial Number and Product Version Information 13 Chapter 2 FDR 14 Chapter 3 Internally Managed Vs External
24. Ste AS mes suc CLASS A device l This device has been approved by EMC registration commercial broadcasting and Distributors or users pay attention to this point This device is communication equipmen usually aimed to be used in other area except at home e Remark Class A device operated in a commercial area 56 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Translation Class A Device This device is registered for EMC requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced with a residential use type Mellanox Technologies 57 Rev 1 0 Appendix B Transferring the Power Cord To use the rail kit to transfer the power cord from the connector side to the power side follow these directions Do you want to install power cords on both sides of the switch or only one side For each power cord you want to transfer 1 Once you have decided which side of the switch will be next to the vertical support 2 Make sure that you place the cord so that the correct end of the cord will be at the power side of the switch This will depend on which of the four mounting options you choose Figure 31 Transfer Power Cord 3 Put the power cord in the switch slide channel Push the cord end through the hole in the switch slide Leave 7 of cord hanging out from the hole 5 Screw the switch slides to
25. T Figure 29 MTUSB 1 with Cables rw The MTUSB adapter comes with three cables but has only two connections for cables One cable is an I2C cable with DB9 female connectors on both ends One cable is a I2C cable with DB9 female connector to RJ45 male Use the cable that matches your switch The other is a USB cable with one type A host connection to be connected to the server or PC and a type B connection to be connected to the MTUSB adapter Figure 30 I2C Cable Connected lo SX6025 With the switch connected to a computer through the MTUSB it is now possible to reprogram the SPI Flash memory 5 5 3 1 Instructions for Reprogramming Through the I2C Port 1 Make sure that 46 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 the MTUSB I2C cable is connected to the switch and USB cable is connected to a com puter MFT is running on the computer mtusb 1 mst device appears in mst status 2 Set the switch I2C switch to allow access by running gt mlxi2c d dev mst mtusb 1 s SX6025 p SX 3 Run the burn command flint d dev mst mtusb 1 i fw SX rel 7 4 0 Q Al bin qq b 4 Power cycle the switch by unplugging and re plugging the power cord to load the new firm ware 5 5 4 How to Get Mellanox Firmware Tools MFT Mellanox Firmware Tools MFT and documentation are available for download via http www mellanox com gt Downloads gt Firmware Tools The MFT kit includes mlxbur
26. TUS LED should light up green The PSUI LED for the plugged in PSU should light up green The PSU2 LED for the second PSU should light up green only if a second PSU is installed in the switch for redundancy and Hot Swap ability and it is connected to a power source If two PSUs are installed and only one PSU is connected to a power supply the second PSU2 LED will be red If only one PSU is installed in the switch the PSU2 LED will be off As long as there is power to the switch one PSU is connected and the switch is booted up and running the status LED should be green 5 The FAN LED should light up green A If the STATUS LED shows red after five minutes unplug the switch and call your N Mellanox representative for assistance SS If the FAN LED shows red troubleshoot the fan module See See Troubleshooting on page 48 N A If the switch shuts down due to over temperature unplug the switch wait 5 minutes N and replug in the switch For more information See Troubleshooting on page 48 If the PSU LEDs are not green this indicates a problem with the power supplies The switch is operational only if at least one of the PSU LEDs is green Mellanox Technologies 17 Rev 1 0 Basic Operation and Installation Figure 5 Power Fan and System LEDs Y V A Y SA W Table 4 System Status LED Configurations LED Configuration STATUS System Health LED Green OK The system is up and run
27. be used to update software or firmware e 1 connector that is labelled CONSOLE Use this connector to connect to the host PC There is one I2C interface to connect to the SwitchX SX6025 Figure 15 Management Interfaces SX6036 in Externally Managed Internally Managed 4 4 2 1 RJ 45 Connector CONSOLE Internally Managed Switches only The port labelled CONSOLE is for a local host connection to the management module This is used the first time the switch is connected A harness is included in the package to connect to a DB connection on a host PC Connecting to a local host PC and following the instructions in the Installation Guide Configuring the Switch for the First Time must be done before any remote Mellanox Technologies 25 Rev 1 0 Basic Operation and Installation management is available For the Socket pinout see RJA5 CONSOLE and Ethernet Interfaces on page 62 gt This connector is not found in externally managed switches 4 4 2 2 RJ 45 Connector I2C Externally Managed Switches only Figure 16 RJ45 I2C Connector All firmware updates should be done in band using Mellanox Firmware Management Tools This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only You will need to order an MTUSB 1 USB to I2C adapter to make use of the I2C interface An HAR 000022 harness is included in the MTUSB 1 package to connect the switch for updati
28. ck the system and check to make sure that all the parts have been sent check this against the parts list below Check the parts for visible damage that may have occurred during shipping The switch comes packed with the following items e 1 X switch e X installation kit e X power cable for each PSU Type B 6ft US 125V 10A chord See Replacement Parts Ordering Numbers on page 63 to order power cords for various countries A single power cord for each power supply unit can be ordered at no extra charge 30 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 e X Harness HARO000028 Harness RS232 2M cable DB9 to RJ45 for SX6036 switches If anything is damaged or missing contact your customer representative immediately 4 6 1 Installing the Switch in the Rack Tools and Customer Supplied Parts Phillips Screwdrivers 1 and 2 grounding screw ESD strap grounding wire sufficient to reach a valid ground ESD mat For racks from 60cm to 80cm deep either use the standard depth switches with the long rail kit or the short switches with the long rail kit Parts included in the rail kit e 2 rails 14 recessed flat head screws for the standard switch e 2rail slides e 10 caged nuts 2switch slides e 10 pan head screws M6 Figure 17 Rack Rail Kit Parts Switch slide Rail Mellanox Technologies 31 Rev 1 0 Basic Operation and Ins
29. d refer the MLNX OS Software Command Reference Guide 5 4 2 Upgrading Software The new software and firmware update is available from the Mellanox Support website Copy the update to a known location on a Remote server within the user s LAN Use the CLI or the GUI in order to perform Software upgrade For further information please refer to the MLNX OS user manual Be sure to read and follow all of the instructions regarding the updating of the software on your switch system 5 5 Unmanaged Externally Managed Switch The switch comes with the latest firmware burned on the board All firmware updates should be done in band Go to the Mellanox Website and confirm that the firmware is the latest If not 44 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 reburn the latest firmware from the download site New firmware versions will be posted on the Mellanox firmware download page http www mellanox com gt Support gt Download Center You will need the Mellanox Firmware Tools package to update firmware for this switch It can also be downloaded from http www mellanox com gt Support gt InfiniBand Software and Drivers You will also need to download and unzip the firmware binary image This is provided in the Mel lanox Web site at http www mellanox com gt Support gt Download Center and go to the SwitchX Switch systems Click in the Table for the firmware image that you need 5 5 1 12
30. ded OS chassis manager and system BIST SNMP agent 3rd party tool integration GUI subnet manager remote logging SSH telnet secured access in band and out band IPv4 IPv6 network stack 5 2 Chassis Management The chassis management includes the following features upgrading SwitchX firmware upgrading software monitoring of AC power to the PSUs DC power out from the PSUs board temperature fan module status 42 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 chassis failures querying for switch serial numbers and their revisions software version and SwitchX FW ver sion 5 2 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It discovers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS routing partitioning to the fabric devices You can view and configure the Subnet Parameters SM via the CLI WebUI Each subnet needs one subnet manager to discover activate and manage the subnet Each network requires a Subnet Manager to be running in either the switch itself switch based or on one of the nodes which is connected to the fabric host based Y No more than two subnet managers are recommended for any single fabric 5 3 Fabric Diagnostics with UFM Diag UFM Diag is a plug amp play software for fee wi
31. e D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 Installation Kit for short switches in racks 40 60 cm deep MSX60 BKIT Installation Kit for short or standard switches in racks 60 80 cm deep MSX60 SKIT Mellanox Technologies 63 Rev 1 0 Appendix F Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive N Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambi AN ante d passant le maximum recommand de 45 C 113 F En outre pour Ah garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouvertures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le chassis tombe il peut provoquer des blessures corporelles et des d gradations de biens gt 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lectrique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solide ment fix ou s il est enlev 5 Orages dangers lectriques Pendant un
32. e de alimentaci n redundante peligro de descarga el ctrica YIN Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el pro ducto si su tapa est quitada o no est bien cerrada 6 Cuando hay rayos peligro de descarga el ctrica Mo No utilizar el equipo ni conectar o desconectar cables durante per odos de acti vidad de rayos 7 Montaje y mantenimiento de bastidores Al instalar o realizar el mantenimiento de este aparato en un bastidor es pre A N ciso adoptar precauciones especiales para garantizar que el sistema se man y tenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 68 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 8 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente 9 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incendio 10 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 11 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme
33. e of the switch This interface is for Debug and Troubleshooting only This connector can be used to install firmware upgrades should the firm ware image be damaged and cannot be upgraded through a host PC or remotely This interface is for support personnel and advanced users only 4 5 Switch Platform Installation and Operation Installation and initialization of the switch platform are straightforward processes requiring atten tion to the normal mechanical power and thermal precautions for rack mounted equipment The unmanaged switch platform does not require any programming or configuration to operate as a basic InfiniBand switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software The managed switch platform requires initial configuration to operate as an InfiniBand switch All internally managed switches come with an internal PPC based management board This board allows for internally managing the switch through a host PC or remotely through the Ethernet 4 5 1 Installation Safety Warnings For Safety Warnings in French see Section E Avertissements de s curit d installation French on page 64 for German see Section G Installation Sicherheitshinweise German on page 66 Mellanox Technologies 27 Rev 1 0 Basic Operation and Installation and for Spanish see Section H Advertencias de seguridad para la instalaci n Spanish on page 68 66
34. e switch or management module 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 mlnx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No I2C ready Mellanox Technologies 49 Rev 1 0 Troubleshooting 5 Select the image to boot 50 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Appendix A Specification Table 11 SX6025 Specification Data Physical Power and Environmental Size SX6025 B short SX6025 S Standard Weight Short Standard Center of Gravity Short IPSU 2PSUs Long 1PSU 2PSUs Mounting Air flow Heat dissipation SerDes Speeds Connector Types 1 716 1U H x 16 8 W x16 8 D 43 6mm X 427 8mm X 428 9mm 1 716 1U H x 16 8 W x24 75 D 43 6mm X 427 8mm X 628 9 mm 6 98kg 1 PSU 7 73kg 2 PSUs 7 92kg 1 PSU 8 67kg 2 PSUs As measured from the Connector side left side bottom corner 21 6mm X 193mm X 209 8mm 21 6mm X 214mm X 218 8mm 21 6mm X 201 mm X 337mm 21 6mm X 214mm X 348mm 19 Rack mount 62CFM 798 BTUs hr 10 20 40 or 56 Gb s per port QSFP Input Voltage P
35. e system to replace the power supply unit when there is only one PSU in the switch Figure 22 Two Power Inlets Electric Caution Notifications CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Risque de choc et de danger hazard The two PSUs are indepen Schocks Entferrnen des e lectriques Le de branchment dent Netzsteckers elnes Netzteils d une seule alimentation stabi spannungsfrei Um alle Einhi lise e ne de branch uniquement Disconnect all power supplies to eten spannungsfrei zu machen qu un module Alimentation Sta ensure a powered down state inside sind die Netzstecker aller bilise e Pour isoler com of the switch platform Netzteile zu entfernen pletement le module en cause Il faut de brancher toutes les alimen tations stabilise es 4 6 4 Extracting and Inserting the Power Supply Unit With both power supplies installed in the redundant configuration either PSU may be extracted without bringing down the system Make sure that the PSU that you are NOT replacing is showing all green for both the PSU and status indicators Power supply units have directional air flows similar to the fan module The Fan mod ule airflow must coincide with the airflow of all of the PSUs If the PSU airflow direc tion is different from the fan module airflow direction the system will shut down Figure 23 Power Supply Unit Extraction XI XI XI XI IX II II II II II X
36. e that will become available to customers in the near future For details please contact Mellanox Technologies support Reset Button On the connector side panel under the system LEDs is a reset button This reset button requires a tool to be pressed 22 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 N DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset A a N button Sharp objects can cause damage use a flat object to push the reset button Figure 12 Reset Button Press the reset button to reset the main and management CPUs and to delete the existing password This button resets both the CPU of the switch device and the CPU of the management module It thereby resets all of the ports by bringing them down and powering them up when the button is pushed A quick push of this button performs this reset When the button is held down for 15 sec onds the switch is reset and the password is deleted You will then be able to enter without a pass word and make a new password for the user admin In the externally managed switch the reset button resets the CPU of the switch device 4 2 Air Flow These switches can come with two air flow patterns The two patterns are Connector side inlet to power side outlet Power side inlet to connector side outlet The air flow is specified in the product model number See Mellanox Part Numbering Legend on page 10
37. en 66 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 8 Ger teinstallation FN Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert B ausgetauscht oder gewartet werden 9 Ger teentsorgung A Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze A D Bestimmungen erfolgen 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen L Bestimmungen installiert werden Mellanox Technologies 67 Rev 1 0 Appendix H Advertencias de seguridad para la insta laci n Spanish 1 Instrucciones de instalaci n a A q Antes de conectar el equipo a la fuente de alimentaci n leer todas las instruc ciones de instalaci n 2 Instalaci n en un lugar con acceso restringido Y N Esta unidad ha sido ideada para instalar en lugares de acceso restringido 3 Sobrecalentamiento N No se debe utilizar el equipo en un rea con una temperatura ambiente superior A a la m xima recomendada 45 C Adem s para garantizar una circulaci n de YN aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas C alrededor de las aberturas de ventilaci n 4 Apilamiento del chasis oN Los chasis no se deben apilar sobre otros equipos La ca da del chasis podr a causar lesiones corporales as como da os al equipo 5 Conexi n de fuent
38. faces The RJ45 CONSOLE and Ethernet interfaces uses the EIA 568A standard wiring color coding CONSOLE J34 Looking into the Socket C J34 Looking into the Socket 62 Mellanox Technologies Pin 1 RJ45_RTS RJ45_DTR RJ45_TXD GND GND RJ45_RXD GND GND Pin 8 Pin 1 RJ45_SDA GND GND GND GND RJ45_SCL GND GND Pin 8 Table 14 RJ45 Pinout Connection Signal Pin Color TXD RJ45_RTS 1 G W TXD RJ45_DTR 2 G RXD RJ45_TXD 3 O W GND GND d Bl GND GND 5 BW RXD RJ45_RXD 6 O GND GND 7 Br w GND GND 8 Br SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Appendix E Replacement Parts Ordering Numbers Table 15 Replacement Parts Ordering Numbers Part Description OPN I2C DB9 or RJ45 to USB Adapter MTUSB 1 Fan module with power supply side to connector side air flow MSX60 FF Fan module with connector side to power supply side air flow MSX60 FR 300W Power Supply w Power Supply Side to Connector side air flow MSX60 PF 300W Power Supply w Connector side to Power Supply side air flow MSX60 PR Power cord Type C13 C14 ACC000251 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany Netherlands Russia ACC000207 Power cord Type G for UK ACC000208 Power cord Typ
39. gies SwitchX Switch Platforms MLNX OS Software WebUI User s Manual WebUI user s manual for MLNX OS MLNX OS Software Command Reference Guide Command Reference Guide for MLNX OS listing all of the commands available through MLNX OS with explanations and examples MLNX OS Software Configuration Guide Configuration Guide for MLNX OS displaying different configuration scenarios 8 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 All of these documents can be found on the Mellanox Website They are available either through the product pages or through the support page with a login and password Conventions Throughout this manual the name SX60XX and the term switch are used to describe the 36 port QSFP 56Gb s InfiniBand Switch unless explicitly indicated otherwise The following pictures are used throughout this document to indicate information that is important to the user v This symbol makes recommendations to the user This symbol indicates information that is helpful to the user This symbol indicates a situation that can potentially cause damage to hardware or software BEWARE This symbol indicates a situation that can potentially cause personal injury or damage to hardware or software Mellanox Technologies 9 Rev 1 0 Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies SX System Type SwitchX Switch
40. hich case the user may be A 1 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian I CES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada Mellanox Technologies 55 Rev 1 0 A 2 VCCI Statements Japan Class A Statement CORE TESRAVERXE IE SS TER REL ER BRS VCCI OHS CE2 722AAITSSSIINIEIB C CORE REZAR CHEATS CBR MEAS ARMOFCEPHV ETF COBARDE AS ZBEDBRKENZScCEPHVET A 2 1 Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 3 MIC Certification Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and certification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL which is organized under the Ministry of Information and Communications MIC EMC testing includes electromagnetic emissions EMI and susceptibil ity EMS Certified equipment is labeled with the MIC mark and certification number English Translation Device User s information 0 71712 APT S Ag e s elaul iere m S Pr des u amp 547 7l 71710 2 LU ZONAS FS ASAE Ol SE FSAI HeH 7 H89l9 AISA A
41. ly Managed 15 Chapter 4 Basic Operation and Installation 16 4 1 Switch Platform Hardware Overview 16 4 1 1 Status LEDs 16 4 1 2 Reset Button 22 4 2 Air Flow 23 4 3 QSFP Cable Power Budget Classification 24 4 4 Interfaces 24 4 4 1 Port Connector Interfaces 24 4 4 2 Management and Firmware Updating Interfaces 25 4 5 Switch Platform Installation and Operation 27 4 5 1 Installation Safety Warnings 27 4 5 2 Mechanical Installation 30 4 6 Package Contents and Installation 30 4 6 1 Installing the Switch in the Rack 31 4 6 2 Grounding the Switch 35 4 6 3 Power Connections and Initial Power On 36 4 6 4 Extracting and Inserting the Power Supply Unit 37 4 6 5 Cable Installation 38 4 6 6 Extracting and Inserting the Fan Unit 39 4 7 Switch Shut Down Procedure 40 4 8 Disassembly of the Switch from the Rack 41 4 9 Disposal 41 Chapter 5 Management and Tools Overview 42 5 1 Chassis Management Using the MLNX OS Software 42 5 2 Chassis Management 42 5 2 1 InfiniBand Subnet Manager 43 5 3 Fabric Diagnostics with UFM Diag 43 5 4 Configuring the Switch for the First Time 43 5 4 1 Starting a Remote Connection to the Switch 44 5 4 2 Upgrading Software 44 5 5 Unmanaged Externally Managed Switch 44 5 5 1 D2C Connector 45 5 5 2 Current Firmware Revision For Externally Managed Switches 45 Mellanox Technologies 3 Rev 1 0 5 5 3 Reprogramming the Switch Through The I2C Port 45 5 5 4 How to Get Mellanox Firmware Tools MFT 47
42. n flint spark debug utilities See Related Documentation on page 7 Mellanox Technologies 47 Rev 1 0 Troubleshooting 6 Troubleshooting As soon as a switch is plugged in make sure that the green power LEDs on the PSUs are on Status LED and or Status Health LED If either of these two LEDs is red unplug the switch and call your Mellanox representative Power supply unit If the LED on the PSU is not lit or is red check that the power cable is plugged into a working out let 1 Check that the power cable has a voltage within the range of 100 240 volts AC 2 Check that the air flow direction of the PSUs are consistent with the Fan module air flow 3 Remove and reinstall the power cable 4 Remove and reinstall the PSU The power LED for the switch shuts off 1 Check that the there is adequate ventilation Are the fan LEDs showing that the fans are all up and running 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner The green power LED for the fans does not come on 1 Check that the Power LEDs are on 2 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and
43. ng firmware The I2C connection provides access to Flash and EEPROMs This connection allows access to the switch for updating firmware See Section 5 5 4 4 2 3 RJ 45 Ethernet Connector MGT Internally Managed Switches The Ethernet connection labelled MGT provides access for remote management The switch can be connected to any Ethernet port These connector s are not found in unmanaged externally managed switches 5 Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M 1G auto negotiation gt 26 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 The unmanaged switches are Plug and Play and all firmware updates should be done in band The I2C connection should only be used if the firmware image was corrupted to the point that the regular firmware tools cannot successfully reburn the correct image 4 4 2 4 USB interface dl The USB interface can be used to update the MLNX OS Web User Interface WebUI or MLNX OS Command Line Interface CLI 5 There is a single USB connector This connector can be used to install software and or firmware upgrades using a memory device that has a USB connector This connector is USB 2 0 compliant gt The USB connector is not found on unmanaged externally managed switches ae 4 4 2 5 12C Interface On Managed Switches There is an I2C connector on the far left of the power sid
44. ning Flashing Green The system is booting up Yellow Error A fault in the system most likely the firmware did not BOOT properly Red Major Error Possible damage can result to the switch Turn off immediately e g bad firmware can t boot overheated Off Off The system has no power 4 1 1 2 Port Connector LED Indicators Above the ports are two LEDs one for the upper port A and one for the lower port V The fol lowing table shows the port status according to the LED indication Figure 6 Port LEDs Port LEDs AW A Y AY 18 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Table 5 Connector Physical and Logical Link Indications LED Status LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed data transfer speed Orange Physical link up Flashing Orange A problem with the physical link Usually the SM is down or not enabling the port 4 1 1 3 Power Supply Status Indicators The SX60XX switch is available with one or two factory installed Power Supply Units For switches with only one unit installed a second Power Supply Unit can be added to support hot swap ability and to add redundancy See Section E Replacement Parts Ordering Numbers on page 63 for ordering part numbers Figure 7 Power Side Panel PSU2 status LED Fan status LED PSUI status LED Secondary Po
45. nit Extraction 37 Figure 24 PSU Pulled Out 38 Figure 25 Top and Bottom Ports 39 Figure 26 Air Flow Labels 40 Figure 27 Fan Module Latches 40 Figure 28 Host Connection 44 Figure 29 MTUSB 1 with Cables 46 Figure 30 I2C Cable Connected to SX6025 46 Figure 31 Transfer Power Cord 58 Figure 32 Install the Power Cord Clip 59 Figure 33 Transfer Power Cord Finished 59 Mellanox Technologies 5 Rev 1 0 List of Tables Table 1 Revision History Table 7 Table2 Reference Documents 8 Table3 Switch Management 15 Table4 System Status LED Configurations 18 Table 5 Connector Physical and Logical Link Indications 19 Table 6 PSU Status LED Configurations 20 Table 7 Fan Status LED Configurations 21 Table 8 Bad Port LED Configurations 22 Table 9 Air Flow Direction 24 Table 10 Installation Kit According to Rack Size 30 Table 11 SX6025 Specification Data 51 Table 12 SX6036 Specification Data 53 Table 13 QSFP Connector Pinout 60 Table 14 RJ45 Pinout 62 Table 15 Replacement Parts Ordering Numbers 63 6 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Revision History Table 1 Revision History Table Date Revision Description July 2011 1 1 Changes reset button section removed reference to paper clip July 2011 1 0 Initial Release Mellanox Technologies 7 Rev 1 0 About this Manual This manual describes the installation and basic use of the Mellanox switch which
46. nts 15 19 a 4 This device complies with Part 15 of the FCC Rules 54 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 This device must accept any interference received including interference that may cause undesired operation 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance Mellanox Technologies could void the user s authority to operate the equipment 815 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 1 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in w
47. or 36 ports FDR 10 1 44 Tb s for 36 ports QDR 720 Gb s for 36 ports DDR Features Switching Perfor mance Simultaneous wire speed any port to any Hot Swappable Fan Module port 1 1 Redundant Power Supplies Switching Capacity 52 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Table 12 SX6036 Specification Data Physical Power and Environmental Size SX6036 B short SX6036 S Standard Weight Short Standard Center of Gravity Short IPSU 2PSUs Long 1PSU 2PSUs Mounting Airflow Heat dissipation SerDes Speeds Connector Types 1 716 1U H x 16 8 W x16 8 D 43 6mm X 427 8mm X 428 9mm 1 716 1U H x 16 8 W x24 75 D 43 6mm X 427 8mm X 628 9 mm 7 08kg 1 PSU 7 82kg 2 PSUs 8 02kg 1 PSU 8 76kg 2 PSUs As measured from the Connector side left side bottom corner 21 6mm X 193mm X 209 8mm 21 6mm X 214mm X 218 8mm 21 6mm X 201mm X 337mm 21 6mm X 214mm X 348mm 19 Rack mount 62CFM 798 BTUs hr 10 20 40 or 56 Gb s per port QSFP Input Voltage Management CPU Power Consump tion TypOTyp FDR Active Cables Passive cables QDR FDRI10 Active Cables Passive cables Max Max FDR Active Cables Passive cables QDR FDRI10 Active Cables Passive cables QSFP Temperature Humidity 100 240 VAC 50 60Hz PowerPC 460EXr PS unit fan is always at 7096 207W 126W
48. or obstacles See Section 4 6 6 Extracting and Inserting the Fan Unit on page 39 A Caution Do not run the switch if the System Status LED for the Fans is Yellow The link LED for the connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest firmware version is installed on both the HCA and the switch 4 If media adapters are used check that the all connections are good tight and secure The activity LED does not come on Check that the Subnet Manager has been started 48 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 The switch is off 1 Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies If the switch comes on Use the MLNX OS management CLI or Web GUI to determine the cause of the Shutdown ASTRO Check the temperature Check the Fan status The switch is not working and unresponsive 1 Reset the switch If resetting the switch does not work 1 Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies If the switch comes on use the MLNX OS management CLI or Web GUI to determine the cause of the shutdown ER Saito The last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop 2 Push the reset button on th
49. ower Consump tion TypETyp FDR Active Cables Passive cables QDR FDRI10 Active Cables Passive cables Max Max FDR Active Cables Passive cables QDR FDRI10 Active Cables Passive cables QSFP Temperature Humidity 100 240 VAC 50 60Hz PS unit fan is always at 7096 194W 112W 172W 91W 221W 116W 211W 106W MAX 2W TYP 1 5W 0 to 45 Celsius 596 9596 non condensing Mellanox Technologies 51 Rev 1 0 Protocol Support Regulatory Compliance InfiniBand QoS Management Data Rate Auto Negotiation of 56Gb s 40Gb s 20Gb s 10Gb s 8 InfiniBand Virtual Lanes for all ports Baseboard Performance and Device management Agents for full InfiniBand In Band Management FDR FDR10 QDR DDR SDR Safety EMC Emissions Environmental Acoustic US Canada cTUVus EU IEC60950 International CB Russia GOST R Argentina S mark China CCC for power supply Korea KN22 2209 KN24 2009 USA FCC Class A Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A Australia C Tick EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 Scalability and Performance Reliability Availability and Serviceability 2 016 Tb s for 36 ports FDR 1 65 Tb s f
50. r Manual Rev 1 0 Figure 21 Slide the Rail into the Rail Slide This side of the rail kit goes on the side of the rack you will slide the switch into This is the same side of the switch that will be next to the vertical support 9 Slide the switch into the rails 10 Tighten the bolts to 9 2 Nm or 81 5 pound inches 11 Put the switch into place and screw the bolts into the nuts Tighten the bolts to 9 2 Nm or 81 5 pound inches 12 Ground the switch 13 Plugin the power cables 14 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LEDs are cause for concern and must be dealt with immediately It can take up to 5 minutes to boot up during which time the status LED may indicate red 15 You can start connecting all of the cables to the switch 4 6 2 Grounding the Switch Check to determine if your local or national electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the ground screw make sure that the rack is properly Mellanox Technologies 35 Rev 1 0 Basic Operation and Installation grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter nally grounded not including the power cord ground You must follow
51. side 3 Decide which mounting option you want to use To use the rail kit to transfer the power cord from the connector side to the power side go now to Transferring the Power Cord on page 58 4 Screw the switch slides onto the switch Use 5 flat head screws for short switches and 7 screws for standard depth switches to connect each switch slide Mellanox Technologies 33 Rev 1 0 Basic Operation and Installation Figure 19 Screwing on the Rail P 22 d Za s 7 y 4 Dx o B P j oO Ome L I Ly A 3 9 4 EF o 7 7 Lf 7 Fi 5 screws per side are needed for the short switch 7 screws per side are needed for the standard switch Figure 20 Inserting the Caged Nuts Side you will slide the swich into Second side lo Moll _lolloMe of el i Jelole 5 Clip 6 caged nuts into the holes in the rack on the side of the rack you will be sliding the switch into Check that both sides of the switch power side and connector side are at the same level in the rack 6 Clip 4 more caged nuts into the holes on the opposite side of the rack Check that both sides of the switch left and right are the same level in the rack 7 Slide the rail into the rail slide 8 Using two of the bolts for each corner install the rails and rail slides in the rack Do not tighten the bolts yet 34 Mellanox Technologies SwitchX FDR InfiniBand Switch Use
52. special precautions must be taken YN to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top S Equipment Installation N This equipment should be installed replaced or serviced only by trained and qualified N personnel 9 Equipment Disposal g Disposal of this equipment should be in accordance to all national laws and regula isl tions 10 Local and National Electrical Codes YN This equipment should be installed in compliance with local and national electrical codes 11 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 12 UL Approved AC Power Cords For North American power connection select a power supply cord that is UL Listed MN and CSA Certified Ah 3 conductor 18 AWG terminated in a molded on plug cap rated at 125 V 15 A EA with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 0 75 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded on plug cap rated 250 V 10 A Mellanox
53. t Manager running from one of the hosts or Fabric Management software running on the switch All that is required for normal operation is to follow the usual precautions for installation and to connect the switch to the adapter cards Once an unmanaged switch is connected the Subnet Management software automatically configures and begins utilizing the switch Managed switches need initial configuration before they will start working It is recommended that the Mellanox OpenFabrics software package be installed on all nodes con nected to the SX60XX switch The software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is available on the Mellanox web site See Section 3 on page 15 for more information Installation hot swapping components and hardware maintenance is covered in Basic Operation and Installation on page 16 Figure 1 Connector Side View of the Switch Mellanox Technologies 11 Rev 1 0 Overview 1 4 Features 1 1 1 Full Feature List 36 FDR 56Gb s ports in a 1U switch BTA 1 3 and 1 21 compliant SDR DDR QDR FDR10 FDR link speed Congestion control Adaptive routing Port mirroring FDRIO supports 20 more bandwidth over QDR using the same cables connec tors Upto 8 multiple switch partitions Flexible combinations of FDR and Gateway ports This feature will be available to customers in the near future FDR
54. tallation 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the rack vertical support Either the side with the power supply units or the side with the QSFP connectors can be even with one of the vertical rack supports Things to consider before choosing where to mount the rails and rail slides The distance between the rack and the door can be as little as 4 cm on one side of the rack and as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables connected to the switch Do you want the connector side recessed in the rack to allow for a larger cable bending radius It is possible to recess the connector side by 5cm by optional placement of the switch rails See Figure 18 Mounting Options Will the connector side be recessed past other equipment in the rack and will this be prob lematic The installation kit allows for a 2 recess of the switch past the vertical support 32 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Figure 18 Mounting Options Vertical Support Connector side Rack door Power sid Connector side ower side Power side Connector side B Rack door Power side Connector
55. the switch Make sure to use two cord clamps per switch slide These will be needed to hold the cord during the switch installation process 58 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Figure 32 Install the Power Cord Clip 6 Push the power cord into the cord clamps Figure 33 Transfer Power Cord Finished ern 7 Return to step 4 on page 33 and continue the installation Mellanox Technologies 59 Rev 1 0 Appendix C QSFP Interface _20 QGND anp _ 19 21 non Rxin 18 Table 13 QSFP Connector Pinout 22 Rx2p Rx1p p17 C i 23 GND GND 16 __ onnector Connector ae 24 15 Pin Pin N Signal Description Rx4n Rx3n Number UE 25 Rx3 L4 BN p 1 GND Ground _ 26 GND _ CRD 2 Tx2n Transmitter Inverted Data Input 2 MooPrsL SDA 2 28 41 3 Tx2p Transmitter Non Inverted Data Input gt EM o 4 GND Ground 30 ER un 9 5 Tx4n Transmitter Inverted Data Input Vec1 ResetL m Wa 8 6 Tx4p Transmitter Non Inverted Data Input 1 PMode ModSelL 5 MB na 32 7 GND ND 8 ModSelL Module Select 33 Tx Tx4p 34 Tx3n TXAN 5 9 ResetL Module Reset 35 GND GND 4 10 VccRx 3 3 V Power supply receiver al Txip Tx2p 3 11 SCL 2 wire serial interface clock gt Txin Tx2nl 2 ND a
56. thin MLNX OS displaying and filtering all identi fied systems and nodes within the fabric UFM Diag includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections based on traffic patterns and user assigned system names GUIDs 5 4 Configuring the Switch for the First Time Unmanaged Externally managed switches do not get configured The CONSOLE Ethernet and USB connectors are not found Instead there is an I2C connector See the Installation Guide of the SX60XX switches Configuring the Switch for the First Time The port labelled CONSOLE must be connected to a local host PC This must be used the first time the switch is connected This must be done before any remote management is available Mellanox Technologies 43 Rev 1 0 Management and Tools Overview Hook up the supplied harness cable HAR00028 from the connector labelled CONSOLE to the DB connector of the local host PC Figure 28 Host Connection Connect the host PC to here SX6036 i 5 4 1 Starting a Remote Connection to the Switch There are two ways to access the management CPU via the Ethernet connector for remote access SSH Telnet Web SNMP XML via KVM connections locally For details on the SNMP commands and MIBs supporte
57. wer Supply Unit Primary Power Supply Unit Fan Unit The primary power supply unit PSU1 is located on the right side of the power side panel with PSU2 on the left side Each PSU has a single 2 color LED on the right side of the PSU that indi cates the internal status of the unit Mellanox Technologies 19 Rev 1 0 Basic Operation and Installation Figure 8 PSU Status LEDs This is a two color LED COS I Green OK Red Major fault Off No power to the system Fan air flow indicator A Table 6 PSU Status LED Configurations LED Color Status Green OK The Power supply is delivering the correct voltage 12VDC Red Error The PSU is not operational Off Off There is no power to the system neither PSU is receiving power If one PSU is showing green and the second PSU is unplugged it will show a red indication 4 1 1 4 Fan Status Indicators The indicator e is located to the left of the connectors on the connector side panel Figure 9 Fan Status LED Connector Side This is a two color LED Green OK Red Major fault Off No power to the fans The LED indicator on the Fan Module is on the right side of the module 20 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 Figure 10 Fan Status LED Power side This is a two color LED pu i Green OK Red Major fault Off No power to the fans Fan air flow indicator pri
58. x Cables 5 gt FDRIO is only guaranteed to work with approved Mellanox ConnectX 3 adapters As 14 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 3 Internally Managed Vs Externally Managed The following table shows which switches come with a management CPU and which do not Externally Managed Unmanaged switches are plug and play out of the box All switches come with the latest firmware burned on the flash Updating the firmware stack for the unmanaged switches is done in band only When a new firmware release is available e mail notification you can upgrade the device through a link to the Mellanox web page firmware download site All internally managed switches have internal chassis management and can support IB fabric of up to 648 nodes Internally managed switches need an initial configuration before they will start working Table 3 Switch Management Family M Management Connections Unmanaged SX6025 Externally Man Plug and Play aged All firmware updates should be done in band using Mellanox firmware Unmanaged management tools I2C port access using MTUSB 1 device is required for firmware updates if in band burning is not possible SX6036 Internally Man RS232 cable DB9 to RJ45 included in the box to connect to host PC for aged initial configuration of the switch After initial configuration the switch can be managed through the Ethernet port using a remote con nection
59. xx XI XX XI II 3631 II II Mellanox XY II XI I1 xk x Ir XI XI XI XI XI IX II IX IX XI II These two latches must be pushed towards each other at the same time while the module is pulled out To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely 4 7 The green fan status indicator should light If not extract the fan unit and reinsert it After two unsuccessful attempts to install the fan unit power off the switch before attempting any system debug Switch Shut Down Procedure To shut down the switch run the command Reload halt noconfirm The switch cannot be restarted remotely To restart the switch you must physically go to the switch and unplug and replug in the power cord 40 Mellanox Technologies SwitchX FDR InfiniBand Switch User Manual Rev 1 0 4 8 Disassembly of the Switch from the Rack To disassemble the switch from the rack 1 Unplug and remove all connectors 2 Unplug all power cords 3 Remove the ground wire 4 Unscrew the 2 bolts from the side of the switch with the bracket Support the weight of the switch when you remove the screws so that the switch does not fall 5 Slide the switch from the rack 6 Remove the rail slides from the rack

Download Pdf Manuals

Related Search

Related Contents

AP MAX User Manual.book  User Manual: M5000 Flow Meter  EPSON ESNSB2 取扱説明書  Carrier 48GX User's Manual  E-58H, E-68, E-78 & E-88 power unit service manual  Lenovo IdeaPad U330 Touch  Corr.d.montagg.estintore anticen.polvere  Informazioni sul presente manuale  takeMS Mobile Drive 2in1  V7 CAT6 UTP Network Cable 10 (RJ45m/m) black 5m  

Copyright © All rights reserved. Failed to retrieve file