Home

324 Port InfiniBand FDR Switch Platform Hardware User Manual

1. Rev 1 0 Hardware 2 6 2 2 6 3 2 6 4 2 6 5 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure MGT Management The MGT port is an Ethernet port for remote management Any remote terminal connected to the Ethernet port can then be used to manage the fabric and chassis Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M 1G auto negotiation N Initial configuration must be done on all of the management modules The first man agement module you configure will be the master U o B The USB port can be used to upload new SW using any storage device that has a USB connector This interface is USB 2 0 Reset RST The Reset button resets the chassis management module when the button is pushed When the but ton is held down for 15 seconds the management module is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip 32 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 14 Reset Button CONSOLE MGT L gt RST PSU S FANS L FANS MASTER lu III
2. Se g 1 4 Grasping the handle with one hand push the black latch release while pulling the handle out ward As the PSU unseats the PSU status indicators will turn off 5 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation Insert the power cord into the supply connector on the other side of the chassis 5 Replace the cover over the PSUs The green indicators should light If not extract the PSU and re insert it again Mellanox Technologies 65 Rev 1 0 Insertion and or Hot swap Extraction 5 2 Leaf Boards When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions The leaf boards are numbered from top to bottom with corresponding numbers displayed to the outside of the leafs vertically along the side panel Figure 47 Leaf Board Numbering 5 2 1 Extracting a Leaf Board Each leaf board has an ejector that locks the board in place and serves as a lever for seating or extracting see Figu
3. 17 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera respon EEN sable y respetuosa con el medio ambiente Mellanox Technologies 109
4. Each spine has a pair of ejectors that lock the board in place and serve as levers for seating or extracting see Figure 51 Management board 1 is connected to spine board 1 and management board 2 is connected to spine board 2 When a slave management board is not installed or not working the spine board connected to the master management board cannot be hot swapped All of the spine boards can be hot swapped when two management boards installed and working 5 3 1 Extracting a Spine Board When a slave management board is not installed or not working hot swapping the spine board connected to the master management board will cause the chassis to crash Neither the CLI nor the GUI management tools will allow you to shut down spine 1 or spine 2 as the management modules are connected to the chassis components through these spines KR 5 3 1 1 Extracting Spine Board 1 or 2 Spine board 1 is connected to management board 1 and spine board 2 is connected to manage ment board 2 Figure 50 Management Module Numbering mu ie D a Il a care WWE T Im i ME MNG2 management module 2 bottom X i al m CONSOLE MGE Dres a aa TT MNGI management module 1 top 68 Mellanox Technologies Rev 1 0 324 Port InfiniBand FDR Switch Platform Hardware User Manual gt N Removing spine 1 causes management module 1 to reset Removing spine 2 causes management module 2 to reset d Warning If the
5. This button resets the CPU of the management module A quick push of this button performs this reset When the reset button is pushed on the master management module this management mod ule is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin Mellanox Technologies 33 Rev 1 0 Installation 3 Installation 4 This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports of the rack As Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the openings are populated with blanks All of the leafs spines and management modules come shipped in a separate package The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instruc tions to make the initial configu
6. 2 wire serial interface clock ModPrsL SDA 28 11 12 SDA 2 wire serial interface data Intl SCL gt 29 10 13 GND Ground VccTx Vcc Rx 30 Vect zs 14 Rx3p PERSIUS Non Inverted Data Output 21 LPMode ModSell 8 15 Rx3n Receiver Inverted Data Output 32 GND GND 7 16 GND Ground 33 Tx3p Tx4p 6 17 Rxlp Receiver Non Inverted Data Output 34 Tx3n Tx4n 5 18 RxIn Receiver Inverted Data Output 35 4 so MEN EM 19 GND Ground z BM E 20 GND Ground Txin Tx2n 38 GND GND 1 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vcc Tx 3 3 V Power supply transmitter 30 Vcc 1 43 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground Mellanox Technologies 95 Rev 1 0 lews QSFP Connector Male and Female Vi Figure 60 Top 18 35 View into Rear of Connector GND AXin TXtp GND TX3n TX3p GND LPMode Vcc1 VccTx IntL ModPrsL GND RX4p RX4n GND RX2p RX2n GND Ny h M E GND TX2n TX2p GND TX4n TX4p GN
7. frei Um alle Einhieten spannungsfrei zu machen sind die Netzstecker aller Netzteile zu Gafahr des elektrischen Schocks Entferrnen des Netzsteckers elnes Netzteils spannungs A entfernen Mellanox Technologies 103 Rev 1 0 Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch plat form 8 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 9 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 10 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 11 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden gt 12 Richtigen Schutz sicherstellen Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen gt 13 Strahlenkontakt Achtung Nutzung von Steuerungen oder Einstellungen oder Ausfiihrung von Proze duren die hier nicht spezifiziert sind kann zu
8. Capacity 48K Unicast Addresses Max per Subnet 16K Multicast Addresses per Subnet Switching Switching Capacity 36 25Tb s Performance Simultaneous wire speed any port to any port 90 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 A 2 Approved Cables For a list of all approved cables see www mellanox com related docs user_manuals Mellanox_approved_cables pdf A 3 EMC Certifications The list of approved certifications per chassis in different regions of the world is located on the Mellanox Website at www mellanox com related docs user_manuals Regulatory_and_Compliance_Guide pdf EMC Statements are also in the Regulatory and Compliance Guide Mellanox Technologies 91 Rev 1 0 Appendix B Thermal Threshold Definitions There are three thermal threshold definitions for the SwitchX switch device which impact the overall switch system operation state Warning Critical and Emergency 1 Warning 100 C On managed systems only When the SwitchX device crosses the 100 C threshold a Warning Threshold message will be issued by the MLNX OS management SW indicating to system admin istration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any action by hardware such as switch shutdown 2 Critical 1209C When the SwitchX device crosses this temperature the firmware will automatically
9. to 45 Celsius 40 to 70 Celsius QSFP Humidity peat MAX 2W Operating 5 95 non condensing eL TYP 1 5W Mellanox Technologies 89 Rev 1 0 Protocol Support InfiniBand Management Speed protocol Auto Negotiation of 56Gb s 40Gb s Managed using MLNX OS P p 20Gb s 10Gb s Baseboard Performance and Device Management ue management Agents for full Infini Band In Band Management Data Rate QoS Data Rate QoS 8 InfiniBand Virtual Lanes for all 56Gb s per port ports Regulatory Compliance Safety EMC US Canada cTUVus Emissions USA FCC Class A EU IEC60950 Canada ICES Class A International CB EU EN55022 Class A im d Russia GOST R EU EN55024 Class A pu Argentina S mark EU EN61000 3 2 Class A China CCC for power supply EU EN61000 3 3 Class A Korea KN22 2209 KN24 2009 Japan VCCI Class A Australia C Tick Tnvironmentat Environmental Vibration bie EU IEC 60068 2 64 Random EU IEC 60068 2 29 Shocks Type I Vibration u EU IEC 60068 2 32 Fall Test Acoustic Acoustic 76DbA Shock and Shock and ISO 7779 Vibration ETSI EN 300 019 2 2 1999 09 Vibration ETS 300 753 Scalability and Performance Leafs Power Supplies Up to 18 leafs 18 ports each 6 Power Supplies standard Spines 9 spine boards Management All spines are needed for non block Modules 2 available 1 required for operation ing configuration Switching Performance Addressing Leaf fans 4
10. 3 Check that the power cable has a voltage within the range of 100 240 volts AC 4 Remove and reinstall the power cable 5 Check the circuit breakers to be sure that the breaker has not tripped 6 Check that the power cable is good Replace the power cable 7 Ifthe AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU 8 2 Leaf Board The power LED for the Leaf board is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall the Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one Should any of the boards shut down due to over temperature wait 5 minutes and then follow the procedure starting with Step 2 gt The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure Mellanox Technologies 83 Rev 1 0 TroubleShooting 5 8 3 8 3 1 8 3 2 84 Mellanox Technologies Replace the cable The A
11. Center of Gravity tion CoGh 90 kg 199 LBS empty configuration Measured from 311 8 mm from bottom of chassis 112 kg 247LBS As shipped config the bottom left Weight uration corner as one CoGw Center of faces the switch 222 mm from left side surface Gravity ports CoGd 328 14 mm from port end surface Measured from the bottom left corner as one faces the switch ports Air Flow Max Air Flow Max Air Flow Through leafs 16 59M3 min 585 9 CFM Through spines 9 48 M3 min 334 8 CFM SerDes Speeds SerDes Speeds Connector Types Connector Types 10 20 40 Gb s or 56 Gb s per port QSFP Sound Level Sound Level Ports Ports 76 db A 324 ports Power and Environmental Input Voltage Input Voltage Management Management 100 240 VAC 50 60Hz CPU PowerPC 460EX Total Power Con Typical FDR14 4467W including Power Consump sumption QSEP at 1 5W tion Leaf Typical 91W not including QSFP power Typical FDR10 3234W including Max 94W QSFP at 1 5W not including QSFP power Max FDR14 4812W including QSFP at 2W Power numbers N Max FDR10 4411W including QSFP at 2W Power Consump Max Heat Output tion Spine Typical 108W include 2 Fans Max 117W include 2 Fans Power Consump tion for Active cables QSFP 16 597BTUs hr 2 0W MAX Power level 2 88 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Operating Non operating Temperature 0
12. Code und dem Canadian Electrical Code 18 Zusammenschaltung von EINHEITEN Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal La tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 19 Zweipolige bzw Neutralleiter Sicherung im Netzteil Achtung Zweipolige bzw Neutralleiter Sicherung im Netzteil Netzstecker ziehen um sicher zustellen da keine Spannung am Ger t anliegt Entfernen Sie alle Netzkabel vor dem ffnen der Abdeckung dieses Produkts oder dem Ber hren der Innenteile Mellanox Technologies 105 Rev 1 0 20 lt 40 lbs 40 70 lbs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 106 Mellanox Technologies Rev 1 0 324 Port InfiniBand FDR Switch Platform Hardware User Manual Appendix I Advertencias de seguridad para la insta laci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Lesi n corporal por peso LA e b 40 lbs 40 70 Ibs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Lesi n corporal por peso Y N This equipment is very heavy and should be moved using a mechanical lift to avoid injuri
13. Each Power Supply Unit has the following indicator LEDs Figure 5 Power Supply Unit Status Indications p DC AC OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis 5 Mellanox Technologies 23 Rev 1 0 Hardware Figure 6 PSU Cover On and Off PSU Covers On PSU Covers Off O o o D HA O e o 8 68 BB D SX6518 mmm A O o o Y HM o e CH Ba 8 08 8 I SX6518 i i iE up ED A D A A D A D up ED up D D D D D A aem up D D D D D D EK x AD O ADA D D O ep D D D D D D D D 2 3 2 Leaf Board LED Indicators Figure 7 Leaf Board Led Indicators MEN 2 2 3 2 1 Status LED Each leaf board has a Status LED on the far left of the leaf Table 5 shows the leaf status according to the LED condition 24 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 The Leaf Status indicator LED A has the following LED assignment Table 5 Leaf Status LED LED Condition LED Description Off No power to
14. Install and tighten the 12 M6 bolts into the caged nuts 46 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 24 Shelf Brackets The spine side of the shelf has two slots for the exten sion SHELL 07 AP dE AP AP D P P P AP GP A4 brackets 9 Tighten the 16 hex head bolts on the adjustable brackets Figure 25 U Numbering Above Shelf The first cage nut on the spine side goes here The rest of the caged nuts go every 4 U units up from here 10 Place 8 caged nuts into the vertical rack support at the locations corresponding to the holes in the chassis faceplate The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support There are 4 holes on each side Mellanox Technologies 47 Rev 1 0 Installation 3 6 3 6 Inserting the Chassis 1 Putthe container on the fork lift Figure 26 Putting the Chassis in the Rack The chassis is screwed down to the pallet on both the spine side and the leaf side 48 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 27 Place the Upper Bracket Offset 2 Screw on the two Upper Bracket offsets onto the chassis using nine flat head screws for each
15. The holes in the bracket are offset to one side The holes must be closer to the top of the shelf Figure 21 Top of Shelf Bracket The holes in the bracket are offset to one side The holes must be closer to the top of the shelf op of shelf Bottom of shelf is even with this side of the bracket the greater distance between the holes and the edge of the bracket show that this is the bottom side 5 Move the shelf brackets on the shelf Place the shelf bracket so that it measures slightly more than the measured distance between the vertical rack supports Make sure that there is enough 44 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 play in the bracket so that it can move to the vertical support Do not tighten the hex head bolts You may need to remove the brackets and reinstall them bs Figure 22 Placing the Shelf Bracket Adjusting slots Mellanox Technologies 45 Rev 1 0 Installation Figure 23 Caged Nut Locations for the Shelf Brackets 6 Insert three caged nuts into the rack for each of the four shelf brackets the 2 brackets that are adjustable and the 2 that are fixed Make sure that the caged nuts are all at the same height on the vertical supports 7 Put the Shelf into the rack and twist the shelf into place Make sure the spine side of the shelf is closest to the rack doors See Figure 24 to identify the spine side of the shelf 8
16. alimentation stabilise e ne de branch unique ment qu un module Alimentation Stabilise e Pour isoler completement le module en cause Il faut de brancher toutes les alimentations stabilise es Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch plat form gt gt 9 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cautions sp ciales pour s assurer que le syst me reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut 10 Fuite gt 3 5mA Leakage gt 3 5mA ATTENTION La connexion la terre des forts courants de fuite est essentielle avant le branchement de l alimentation Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent 11 Forts Courants de Fuite High Leakage Current Attention Forts courants de fuite Il est essentiel de relier a la terre avant de brancher l alimentation 12 Ajouter une information de connexion la masse Connect a Valid Ground to this Device Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b
17. and proper equipment is used for transporting and moving the LA chassis The fully loaded chassis weighs 212 36 kg 468 LBS full configuration 90 kg 199 LBS empty configuration 112 kg 247 LBS shipped configuration The switch platform uses 17U of rack pace in a standard 19 rack 16U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chassis and there are two lock down bars to secure the chassis to the shelf The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have between 65cm and 80cm between the vertical supports of the rack Make sure that a fully populated rack including cables will have suf ficient air flow for cooling L Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis 2 36 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 3 6 1 Starting with the Rack 3 6 1 1 Rack Recommendation Due to the space required by up to 324 connector cables Mellanox recommends a rack that is 100 cm long and 80cm wide This will allow for proper cable management and enough ventilation to properly cool the chassis 3 6 1 2 Preparing the Rack The rack may need to be modified to accommodat
18. as long as no more than one PSU is bad When using the 1000W PSUs the minimum complement of power supply units is 6 and the chas sis will continue to run at full capacity with only 5 PSUs 2 2 2 1600W Power Supply Units When using the 1600W PSUs the minimum complement of power supply units is 3 and the chas sis will continue to run at full capacity with only 3 PSUs The 1600W PSUs can implement two redundancy options e N 1 configuration The chassis when supplied by a single power grid can be configured for N 1 times 3 This means that all 6 PSUs are supplied from a single power source usually the commercial electric grid This enables 3 power supply redundancy N N configuration N N with 1600W PSUs will allow for N N redundancy This requires a second power grid supplied bya backup power supply grid generator battery backup system any combination of the above Connecting 3 power supplies to one power supply grid and the remaining 3 power supplies to a sec ondary power supply grid will create N N redundancy This is High Availability Under these condi tions should a power supply fail an electric company power failure or blackout for example power grid High Availability will continue to keep the chassis running at full capacity through the secondary or backup power supply grid With N N optional PSU grid redundancy the chassis can run on 1 2 of the full complement of PSUs N N allows the chassis to run on 3 PS
19. failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it will be necessary to reboot the chassis once the defective PSU has been replaced With 1600W optional PSU supplies the chassis can run on as little as 1 2 of the full complement of PSUs N N allows the chassis to run on 3 PSUs supplied from one power grid while 3 are con nected to a second power grid 1 Check all FRUs for proper insertion and seating before connecting the AC power cords Boards Power supplies Leaf fan modules Spine fan modules Insert all leafs that you plan to use in the chassis Insert all spines in the chassis Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all leaf and spine mounting screws Connect the power cords to the PSUs au PWN Connect the power cords to grounded electrical outlets With N 1 PSU redundancy do not power up the chassis with less than all PSUs installed 62 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 4 1 Power Supply and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes As the power is turned on you should observe the following conditions for normal operat
20. fan unit located on Spine Side SX6536 SX6518 MSX6512 MSX 6506 Modular MTF005005 Switch Series Spine Fan Unit DB9 to RJ45 Harness HAR000028 Power cord 250V 15A 2 0M C14 to C13 ACC000334 Power cord 125V 15A 2 0M C14 TO C13 ACCO00242 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Mellanox Technologies 97 Rev 1 0 Table 17 Replacement Parts Ordering Numbers Continued Part Description OPN Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 98 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Appendix G Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 45 C 113 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l ai
21. front panel of the leaf 2 Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated Perform the initial configuration on the management module See the Installation Guide On switch systems with dual management systems first connect the cable and configure the master management module CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and slave roles see the MLNX OS Software UM section High Availability amp All management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure KR 5 6 Switch Shut Down Procedures To shut down the chassis run the following command twice once for each MM Reload halt noconfirm MN The chassis cannot be restarted remotely N To restart the chassis you must physically go to the switch and unplug all of the power cords to the chassis and then replug in all of the power cords to the chas sis The first time you run the command it shuts down the master management module and the second time shuts down the slave management module Mellanox Technologies 75 Rev 1 0 Insertion and or Hot swap Extraction To shut down a
22. includes the following Table 2 Reference Documents and Web Sites Document Name Description InfiniBand Architecture Specification Vol The InfiniBand Architecture Specification that is provided by IBTA 1 Release 1 2 1 Switch Hardware Errata For any possible errata due to hardware issues see the switch support product page This requires a customer support login Mellanox MLNX OS SwitchX Soft WebUI Overview for MLNX OS software ware WebUI User s Manual Mellanox MLNX OS SwitchX Soft This document contains information regarding configuring and managing Mella ware User Manual nox Technologies SwitchX Switch Platforms MLNX OS Software Command Refer Command Reference Guide for MLNX OS listing all of the commands available ence Guide through MLNX OS with explanations and examples MLNX OS Software Configuration Configuration Guide for MLNX OS displaying different configuration scenarios Guide Conventions Throughout this manual the name X6518 and the terms chassis and switch are used to describe the 324 port QSFP InfiniBand chassis unless explicitly indicated otherwise 10 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 The following icons are used throughout this document to indicate information that is important to the user This symbol makes recommendations to the user This symbol indicates information that is helpf
23. leaf run the following command no power enable lt module gt To shut down a spine run the following command no power enable lt module gt To shut down a management module run the following command no power enable lt module gt 76 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 6 Disassembly and Disposal 6 1 Disassembling the Chassis Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Remove all leafs Remove all spines AD EA Ge Pes po Remove all management modules Use enough people to safely lift this product a lt 40 lbs 40 70 lbs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs This product and all of its parts are NOT to be disposed of with household waste This product contains printed circuit boards cables and batteries According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and disposed of according to the directive 8 Goto the Mellanox website for detailed instructions for disassembly of the FRUs and chassis according to the WEEE Directive 9 Dispose of these pieces in a legal and environmentally friendly way 6 1 1 Removing the Chassis Remove the screws connecting the upper brackets to the rack 2 Remove the screws holding the chassis to the rack These
24. of connections from each leaf is 2 If the number of leafs x the maximum number of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF The status LEDs for the spine and their descriptions are shown in Table 8 The LEDs indicate as follows 26 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 8 Spine Status LEDs Spine Status Fan Status Bad Port UID E A dul 000000 imi mimm mm 6 8 10 12 14 16 18 Spine Leaf IB Connection Status 2 3 3 1 Status LED A Table 8 shows the spine status according to the LED condition Table 8 Spine Status LED LED Condition LED Description Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Solid Orange Non fatal error this spine needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 2 3 3 2 Fan LED amp The spine fan indicator LED has the following LED assignment Table 9 Spine Fan Status LED LED Color LED Description Solid Green Spine Fan is OK Solid Yellow One or more of the fans in this spine is not working Each spine has two fans in the fan module Mellanox Technologies 27 Rev 1 0 Hardware 2 3 3 3 Spine to Leaf IB Connection Status LEDs The leaf connection status on each spine displays the condition
25. screws are located in the faceplate on the spine side of the chassis 3 Remove the upper brackets from the chassis Mellanox Technologies 77 Rev 1 0 Disassembly and Disposal Figure 59 Mechanical Lifting Device Put the container base saved from the installation on the fork lift Raise the container base to 1 cm below the level of the shelf Move the chassis away from the rack and onto the container base Put wedges around the chassis to prevent the chassis from rolling off of the container base Move the chassis away from the rack NADA Dispose of the Chassis in a legal and environmental way 10 Go to the Mellanox website for detailed instructions for disassembly of the chassis according to the WEEE Directive 6 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack 2 Remove the shelf 3 Remove all of the caged nuts 78 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 6 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Go to the Mellanox website for detailed instructions for disassembly according
26. the Leaf Solid Green Leaf is up and running Flashing Green Leaf is powering up Solid Orange Non fatal error this leaf needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 2 3 2 2 Bad Port LED Kg The Bad Port indicator is located on the left side of the leaf The following Bad Port conditions are possible Table 6 Bad Port LED Configurations LED Configuration Description Off OK all ports are up and running Flashing Orange Error one or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are e bad cable e bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 2 3 2 3 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support Mellanox Technologies 25 Rev 1 0 Hardware 2 3 2 4 Leaf Board Port Connector LED Assignment 2 3 3 Above the ports are two LEDs one for the upper port and one for the lower port V Each port has a single 2 color LED Table 7 shows the link status according to the LED condition Table 7 Connector Physical and Logical Link Indications LED Condition LED Description Off No power to the port Solid Green Logical link up F
27. the face plate of the chassis touches the vertical rack sup port 17 Make sure the upper brackets are installed in their correct location 18 Screw the 8 bolts through the faceplate into the caged nuts DO NOT TIGHTEN Figure 35 Offset to Upper Bracket Bolts 19 For each offset and upper bracket screw six bolts through the offset into the upper bracket See Figure 35 Use one flat washer and one lock washer for each of these bolts DO NOT TIGHTEN Figure 36 Location for the Lock down Bar k L e Ji Bottom of the ll M chassis 54 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 37 Lock Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf 20 Place the front lock down bar at the edge of the chassis and align the holes in the lock down bar with the holes in the shelf Screw down the lock down bar You will need the 10 hex head screws and washers for each lock down bar DO NOT TIGHTEN 21 Screw down the second lock down bar Figure 38 Lock Down Bars Bolt and Washer Order 22 Remove the shelf extension and save for removal of the chassis 23 Connect a valid ground See Section 3 6 5 Ground Connections on page 57 for detailed instructions 24 Tighten the faceplate screws See Figure 32 Mellanox Technologies 55 Rev 1 0 Installation 25 Tighten
28. timent 13 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 100 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 14 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 15 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 16 Codes d installation L appareil doit tre install selon l ancienne version des codes lectriques nationaux du pays Pour l Am rique du Nord l quipement doit tre install conform ment aux sp cifications du Code Electrique National Am ricain et du Code Electrique Canadien 17 Interconnexion des unites Les c bles de branchement l unit RS232 et les interfaces Ethernet doivent tre certi fi s UL de type DP 1 ou DP 2 Note lorsqu il existe dans un circuit non LPS Protection contre la surintensit Un appareil de protection r pertori facilement accessible contre la surintensit du circuit de branchement et calibr 20A doit tre incorpor dans le c blage lectrique du b timent 18 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui
29. to the WEEE Directive Mellanox Technologies 79 Rev 1 0 Switch Management Tools 7 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via MLNX OS There are 2 Ethernet ports 1 for each management module that get connected to Eth ernet switches These switches must be configured to 10 100M auto negotiation 5 The X6518 switch comes standard with a management software module for chassis management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX based managed switch systems management modules MLNX OS includes a CLI WebUI SNMP and chassis management features for software and IB management software OpenSM You can get more information via the Mellanox MLNX OSTM SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual The managed switch system includes the following software components e Embedded Subnet Manager SM e Chassis manager and system BIST e SNMP agent 3rd party tool integration GUI e Remote logging e SSH telnet e Secured access in band and out band e PvA IPv6 network stack The chassis manager will give the user access to e Switch temperatures e Power supply voltages e Fan unit information e Power unit information e Flash memory e Monitoring of AC power to the PSUs DC power out from the PSUs chassis failures querying
30. wellbeing of all of the people involved in the installation Installation and Cable Management Kit Parts e Parts included in the installation kit 1 shelf with brackets pre installed 1 shelf extension 1 left side upper bracket offset 1 right side upper bracket offset 2 lock down bars 2 upper bracket washers 8 lock washers for the upper brackets to verti 12 lock washers for the upper brackets to upper cal support bracket offsets hex head bolts 8 for the upper 20 caged nuts 12 for the shelf and 8 for the chas brackets and for the shelf extension sis faceplate Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 20 pan head screws for the caged nuts 12 for the shelf and 8 for the chassis faceplate 32 lock washers 20 for the lock down bars 20 flat washers 8 for the lock down bars and 12 for the upper brackets 32 hex head bolts 20 for the lock down bars and 12 for the upper brackets to upper bracket and 12 for the upper brackets to upper bracket offsets offsets 2 upper brackets 12 flat head screws with locktite for the upper bracket offsets 1 open end ratchet wrench 10mm 1 socket nut driver 8mm 1 socket wrench 1 socket nut driver 10mm 2 extensions for the socket driver 1 socket nut driver 13mm flat washers for the shelf extension 1 box wrench e Parts included in the cable management installation kit lcable management rack RH l
31. 0W Power Supply units 21 2 2 2 1600W Power Supply Units 21 2 3 LED Status Indicators 23 2 3 1 Power Supply Unit LEDs 23 2 3 2 Leaf Board LED Indicators 24 2 3 3 Spine Board LED Indicators 26 2 3 4 Spine Side Panel Display LED Indicators 28 2 3 5 Management Module LED Indicators 30 2 3 6 Port Connector Interfaces 30 2 4 AirFlow 31 2 5 QSFP Cable Power Budget Classification 31 2 6 Management Module Interfaces 31 2 6 1 RC 31 2 6 2 CONSOLE 32 2 6 3 MGT Management 32 2 6 4 USB 32 2 6 5 Reset RST 32 Chapter 3 Installation 34 3 1 Environmental and Safety Recommendations 34 3 2 Chassis Package Contents 35 3 3 Leaf Package Contents 36 3 4 Spine Package Contents 36 3 5 Management Package Contents 36 3 6 Physical Installation 36 3 6 1 Starting with the Rack 37 3 6 2 ESD Connection 37 3 6 3 Installation Procedure 38 3 6 4 Installing the Cable Holder 56 3 65 Ground Connections 57 Mellanox Technologies 3 Rev 1 0 3 7 PowerConnections 57 3 7 1 Powering Up the Switch Platform 58 3 8 InfiniBand QSFP Cable Installation 60 3 8 1 Supported Approved Cables 60 3 8 2 Cable Power Classes 61 Chapter 4 Chassis Power Up 62 4 1 Power Supply and Spine Board Indicator Status at Power ON 63 Chapter 5 Insertion and or Hot swap Extraction 64 5 1 Power Supply Units 64 5 1 1 Extracting and Inserting the Power Supply Unit 64 5 2 LeafBoards 66 5 2 1 Extracting a Leaf Board 66 5 22 Inserting a Leaf Board 67 5 3 Spine Boar
32. 2010 14 08 15 CPU Board FDEF I2C DRAM FLASH NAND PCI PCIEO PCIEL Net AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Mellanox PPC460EX Board No ready 2 GB ECC enabled 400 MHz CL3 16 MB 1024 MiB Bus Dev VenId DevId Class Int link is not up successfully set as root complex 01 00 15b3 bd34 0c06 00 ppc 4xx eth0 ppc 4xx ethl Hit Ctrl B to stop autoboot 0 Mellanox FabricIT Boot Menu 1 EFM PPC M460EX EFM 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM PPC M460EX EFM 1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt 86 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Choice 5 Select the image to boot For more detailed instructions concerning Mellanox OS software see the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual Mellanox Technologies 87 Rev 1 0 Appendix A Specification Data A 1 Specification Data Sheet Table 14 Switch Specification Sheet SX6518 Physical Size 28 x 19 x 27 inches Mounting Size Mounting 711 mm x 482 6mm x 685 mm 19 Rack mount The shelf adds 44 5mm to the height Weight 212 36 kg 468 LBS full configura
33. 45 C 113 F Moreover to guarantee proper air This equipment should not be operated in an area with an ambient temperature exceed h flow allow at least 8cm 3 inches of clearance around the ventilation openings 5 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage Mellanox Technologies 17 Rev 1 0 Hardware 6 Redundant Power Supply Connection Electrical Hazard This product includes a redundant power or a blank in its place In case of a blank power supply do not operate the product with the blank cover removed or not securely fastened 7 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 8 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 9 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 10 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should
34. D ModSelk ResetL VccRx SCL o spa E GND 18 35 RX3p RX3n GND RX1p RX1n GND View into Front of Cage aNd ne diXH QNO UEXH dexu aNd vas 10s XH99A Tieseu I9SPOWN aNd dpx 1 UPXL QN9 dzXL uZXL AND 4 4 iU 9 8 7 6 5 4 3 2 EF Mellanox Technologies 96 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Appendix F Replacement Parts Ordering Numbers Table 17 Replacement Parts Ordering Numbers Part Description OPN Power supply unit PSU 1000W for Modular Switch MTP005001 Family Power Supply unit PSU 1600W for Modular Switch MTP006002 Family Power supply blank MTM005001 Leaf board unit 18 port FDR MSX6001FR Leaf board unit 18 port FDR 10 MSX6001TR Modular Switch Family Leaf Blank MTM005004 Spine board unit for FDR system MSX6002FLR Spine board unit for FDR10 QDR system MSX6002TBR Spine board blank MTM005002 PPC460 Management Module MSX6000MAR x86 Management Module MSX6000MBR Modular Switch Family Management Blank MTM005003 Rack Installation Kit for IS5600 SX6536 Series MTR005600 Cable Management Device Set for IS5600 SX6536 MTR005601 Cables Set For IS5600 SX6536 Series MTR005602 IS5600 SX6536 IS5300 SX6518 Series Modular MTF005001 Switch leaf cooling chassis fan unit located on Leaf Side IS5600 SX6536 IS5300 SX6518 Modular Switch MTF005002 leaf cooling chassis
35. DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com Tel 972 0 4 909 7200 Tel 408 970 3400 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 O Copyright 2012 Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners FDR InfiniBand Switch System User Manual 2 Mellanox Technologies Document Number 3726 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Table of Contents Table of Contents 3 List of Figures 6 List of Tables 8 Revision History 9 About this Manual 10 Intended Audience 10 Related Documentation 10 Conventions 10 Mellanox Part Numbering Legend 11 Chapter 1 Overview 12 1 1 Product Information 12 1 1 1 Serial Number and Product Version Information 12 1 1 2 Management Module MAC 13 1 1 3 Product Physical Specifications and Power 14 1 2 Features List 16 1 3 InfiniBand FDR and FDR10 Overview 16 Chapter 2 Hardware 17 2 1 Installation Safety Warnings 17 2 2 Power Supply Redundancy 21 2 2 1 100
36. FDR Switch Platform Hardware User Manual Rev 1 0 Figure 9 Spine Side Panel Display Status Indications AAA AA O STATUS PSU SPINE LEAF MNG 1 MNG 2 STATUS FANS FANS STATUS STATUS Table 12 LEDs Display for Normal Operation qu ERAS Normal LED Condition Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MNGI Off Green MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master MNG2 Off Off MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operati
37. MT0806X01504 in iii MM 1 ozcsozoozrosac MU Rev A1 Made in Israel The GUID is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management SW look for this GUID in addition to their own Node GUID Figure 2 Product Label Tab Location SX6518 Here is the tab with the product label 1 1 2 Management Module MAC Each management module has a label with its MAC address See below for the location of this label Figure 3 Management Module MAC Address Location MAC address location Mellanox Technologies 13 Rev 1 0 Overview 1 1 3 Product Physical Specifications and Power Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis Rack Rack of leafs about Height Rack Width Weight Max Power 324 x QSFP Ports E Depth E Temp Required 1t of Spines 17U 19 25 7 18 9 212 36 kg 468 LBS Typical 4791W including QSFP Max 45 C 755 mm EIA 310 685 mm Full configuration active cables at 2W Min 10 C 16 U for the 482 6 mm Typical 4467W including QSFP chassis and 90 kg 199 LBS active cables at IW 1U for the Empty configuration Typical 3728W passive cables shelf Up to 6 Power Supplies of 100 112 kg 247 LBS 240VAC 50 60Hz 6 3 13A Shipped configuration The switch ships in a minimum base configuration plus add
38. Mellanox TECHNOLOGIES 324 Port InfiniBand FDR Switch Platform Hardware User Manual PN MSX6518 NR MSX6518 6R Rev 1 0 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCU MENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRON MENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGE MENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PROD UCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
39. Spine Module JA CICR ee ME MEN CODOOODOD rrr Status Fan Bad Port and UID LEDs Leaf connection LEDs Mellanox Technologies 59 Rev 1 0 Installation 3 8 InfiniBand QSFP Cable Installation The switch uses industry standard QSEP InfiniBand cables which are available from Mellanox Technologies The Mellanox proprietary QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 See Section 2 3 6 for port numbering If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase FDR BandWidth is supported only by MLNX QSFP cables FDR and FDR10 are only guaranteed to work with approved Mellanox Cables SN cs SN All cables can be inserted or removed with the unit powered on To insert a cable press the con nector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 seconds for link to get up To remove dis engage the lock and slowly pull the connector away from the port receptacle For a valid physical connection both ends of the cabl
40. Us supplied from one power grid while 3 are con nected to a second power grid Mellanox Technologies 21 Rev 1 0 Hardware 99 With power grid A charged with current and power grid B not charged there is only grid redundancy and not PSU redundancy gt MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot command in the CLI or reboot through the Web GUI The form is identical between the 1600W and the 1000W PSUs The standard PSU weighs 0 3kg more than the 1600W PSU N N redundancy ONLY works with a supply voltage of 220V Table 4 OPNs for Power Supply Units PSU E OPN Wattage Description MTP005001 1000 W Supplies N 2 redundancy for SX6536 and SX6518 chassis Supplies N 1 redundancy for SX6512 and SX6506 chassis MTP006002 1600 W Supplies N N redundancy for all switch chassis at 220 Volts 22 Mellanox Technologies Rev 1 0 324 Port InfiniBand FDR Switch Platform Hardware User Manual 2 3 LED Status Indicators The LEDs are placed on the chassis for the convenience of the IT manager All chassis conditions and management options are available and controllable through the management SW either CLI or Web GUI SY It is recommended that all of the chassis sub systems be maintained and managed through the management software 2 3 1 Power Supply Unit LEDs
41. agement kit e 1 Installation Guide Before you install your new SX6518 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis Mellanox Technologies 35 Rev 1 0 Installation 3 3 3 4 3 5 3 6 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order Spine Package Contents The spines are shipped 3 to a box The customer will receive as many boxes as needed to fill the chassis with a full complement of spines Management Package Contents The package includes e all of the management modules ordered by the customer e 1 RJ45 to DB9 harness for each management module received Physical Installation Warning This equipment is very heavy Safety is the first concern Make sure that adequate manpower
42. ased on traffic patterns and user assigned system names GUIDs 7 3 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH e Telnet WEB SNMP XML 7 4 Upgrading Software The new software and firmware update is available to the user from the Mellanox Support website Copy the update to a known location on a Remote server within your LAN Mellanox Technologies 81 Rev 1 0 Switch Management Tools Use the CLI or the GUI in order to perform the Software upgrade For further information please refer to the Mellanox OS user manual oy an earlier version than the minimum that the new version of the software can work with then the chassis management system may require up to 45 minutes to update all of the Be FW images in all of the leafs and spines A If Mellanox OS is updated and the FW image in the leafs and spines of the chassis is 82 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 8 TroubleShooting 8 1 Power Supply Unit hl If the Power supplies cannot supply enough power the management module may shut down the system 5 As each PSU is plugged in make sure that the green power LEDs on the PSU comes on If the AC power LED is off 1 Check that the power cable is the correct power cable for your country 2 Check that the power cable is plugged into a working outlet
43. ble3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 14 Table 4 OPNSs for Power Supply Units 22 Table 5 Leaf Status LED 25 Table 6 Bad Port LED Configurations 25 Table 7 Connector Physical and Logical Link Indications 26 Table 8 Spine Status LED 27 Table 9 Spine Fan Status LED 27 Table 10 Spine to Leaf IB Link Status 28 Table 11 Bad Port LED Configurations 28 Table 12 LEDs Display for Normal Operation 29 Table 13 LEDs Display for Normal Operation 30 Table 14 Switch Specification Sheet SX6518 88 Table 15 Power Consumption of Chassis Parts 94 Table 16 InfiniBand QSFP Connector Pinout 95 Table 17 Replacement Parts Ordering Numbers 97 8 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Revision History Table 1 Revision History of this User s Manual Revision Date Details 1 0 February Initial release 2012 Mellanox Technologies 9 Rev 1 0 About this Manual This manual provides an overview of the SwitchX based SX6518 modular InfiniBand chassis switch and guidelines for its operation Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniB and architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform
44. cable management rack LH 14 caged nuts M6 14 M6 bolts Figure 16 Installation Kit Parts Shelf Lock down bar Shelf extension Upper bracket Upper bracket washer Upper bracket offset d0o YU L d n Phillips pan head hex head bolt Screw Flat washers Lock washers Caged nuts Flat head screw Mellanox Technologies 39 Rev 1 0 Installation 3 6 3 3 Container Mishandling The container has shock and tilt stickers applied These will turn red if the container has been mis handled or roughly handled Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped If one or more are red notify the shipper and Mella nox This on its own does not indicate damage to the contents But be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped Figure 17 Shock and Tilt Stickers ger Stickers unaffected by ship 3 6 3 4 Opening the Container Stickers showing excessive shock or tilt 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 40 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 18 Container Screws GS D S A gt A Container screws 2 Remove the four sides of the container by unscrewing t
45. ck down Bar 51 Figure 31 Fork Lift 51 Figure 32 Face Plate Mounting Bolt Locations 52 Figure 33 Upper Bracket Installation 53 Figure 34 Upper Brackets and Offset Location 53 6 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 35 Offset to Upper Bracket Bolts 54 Figure 36 Location for the Lock down Bar 54 Figure 37 Lock Down Bar 55 Figure 38 Lock Down Bars Bolt and Washer Order 55 Figure 39 Cable Holder 56 Figure 40 Ground Connection 57 Figure 41 Multiple Power Inlets Electric Caution Notification 59 Figure 42 Spine Module 59 Figure 43 Spine Side Panel Display Status Indications 63 Figure 44 Management Module Status Indications for Normal Operation 63 Figure 45 Power Cord Top Bar 65 Figure 46 Power Supply 65 Figure 47 Leaf Board Numbering 66 Figure 48 Ejector Latch 67 Figure 49 Leaf Release 67 Figure 50 Management Module Numbering 68 Figure 51 Ejector Latch 69 Figure 52 Spine Board Extraction 69 Figure 53 Fan Locations on the Chassis 71 Figure 54 Leaf Fan Module Extraction 72 Figure 55 Spine Fan Module 12 Figure 56 Fan Status LED on the Spine Module 73 Figure 57 Ejector Latch 74 Figure 58 Management Module 45 Figure 59 Mechanical Lifting Device 78 Figure 60 QSFP Connector Male and Female Views 96 Mellanox Technologies 7 Rev 1 0 List of Tables Table 1 Revision History of this User s Manual 9 Table 2 Reference Documents and Web Sites 10 Ta
46. ctivity indication does not come on Check that the Subnet Manager has been started Management Module Yellow Status LED for the Chassis on the Management Module is Lit nu pw Check the MLNX OS management for confirmation and possible explanation of the alert Reset the master management module by pushing the rest button If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the management module is removed Make sure the S Fans and L Fans LEDs are green Make sure that the spine and the leafs both have the same version of FW Reburn the FW and remove and reinstall the management module If you are running the chassis with only one management module remove and reinstall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 5 5 on page 73 If you are running the chassis with only one management module replace the management module Yellow LED for the Leaf Fan on the Management Module is Lit Check the MLNX OS management for confirmation and possible explanation of the alert Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over th
47. deado con capuch n de 250 voltios nominal 10 A 15 Afiadir conexi n a tierra N Antes de conectar el dispositivo ala linea de alimentaci n los tornillos del terminal de j AN la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de YN protecci n de la instalaci n del edificio ET Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DUCTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 16 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad
48. ds 68 5 3 1 Extracting a Spine Board 68 5 3 2 Inserting a Spine Board 70 5 4 Fan Modules 70 5 4 1 Leaf Fan Module 71 5 4 2 Spine Fan Modules 72 5 5 Management Module 73 5 5 1 Extracting a Management Module 74 5 5 2 Inserting a Management Module 75 5 6 Switch Shut Down Procedures 75 Chapter 6 Disassembly and Disposal 77 6 1 Disassembling the Chassis 77 6 1 1 Removing the Chassis 77 6 1 2 Removing the Bottom Shelf 78 6 2 Disposal 79 Chapter7 Switch Management Tools 80 7 1 InfiniBand Subnet Manager 81 7 2 Fabric Inspector Diagnostics 81 7 3 Accessing the CPU via the Ethernet Connector 81 7 4 Upgrading Software 81 Chapter 8 TroubleShooting 83 8 1 Power Supply Unit 83 8 2 LeafBoard 83 8 3 Management Module 84 8 3 1 Yellow Status LED for the Chassis on the Management Module is Lit 84 8 3 2 Yellow LED for the Leaf Fan on the Management Module is Lit 84 8 3 3 Yellow LED for the Spine Fan on the Management Module is Lit 85 8 4 Spine Board 85 8 5 MLNX OS Software 86 Appendix A Specification Data 88 A 1 Specification Data Sheet 88 A 2 Approved Cables 91 A 3 EMC Certifications 91 Appendix B Thermal Threshold Definitions 92 Appendix C Calculating the Weight of a Customized Chassis 93 Appendix D Calculating the Power of a Chassis 94 Appendix E QSFP Interface 95 Appendix F Replacement Parts Ordering Numbers 97 4 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Append
49. e fan module Management modules Power Supply Units For N 1 configuration a PSU Power Supply Unit can be extracted without bringing down the system as long as no more than one PSU is bad For N N configuration up to half of the PSUs can go down and the system will continue to run The power required to run the switch system is equally divided between all of the working PSUs 5 1 1 Extracting and Inserting the Power Supply Unit With all of the N 1 1000W power supplies installed the system is in N 1 redundant configura tion any single PSU may be extracted without bringing down the system With all of the N N 1600W power supplies installed the system is in N N redundant configura tion half of the PSUs may be extracted without bringing down the system To extract a PSU 1 Determine which AC connector on the connector side of the chassis corresponds to the defec tive PSU Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering 64 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 45 Power Cord Top Bar 3 On the spine side of the chassis remove the cover to the power supply unit for the non work ing PSU Odd numbered PSUs are on the left side and even numbered PSUs are on the right side facing the spines There are four phillips head screws for the cover plate Figure 46 Power Supply Handle Latch
50. e holes If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Determine which fan module is problematic by checking the status LED on each fan module Remove and reinstall the problematic fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 4 on page 70 Replace the Leaf fan module There are two non interchangeable types of leaf fan modules If the new leaf fan does not go into the chassis confirm that it is the correct fan module 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 YN Replace defective leaf fan modules as soon as they are identified Should any of the boards shut down due to over temperature follow the procedure starting in Step 2 8 3 3 Yellow LED for the Spine Fan on the Management Module is Lit Check the MLNX OS management for confirmation and possible explanation of the alert Determine which spine has a defective fan by checking the Fan LEDs on all of the spines 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 5 Remove and rein
51. e must be connected to working devices Take care to not impede the air flow through the ventilation holes next to the Infini Band ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 3 8 1 Supported Approved Cables For a list of approved cables for this switch see the Mellanox approved cable list www mellanox com related docs user_manuals Mellanox_approved_cables pdf 60 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 3 8 2 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX 65XX series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables Mellanox Technologies 61 Rev 1 0 Chassis Power Up 4 Chassis Power Up Before starting any procedure on the SX6518 series chassis system put an ESD pre vention wrist strap on your wrist and connect to the SX6518 series chassis With N 1 PSU redundancy the chassis must be started with a full complement of possible PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines pro vides AC
52. e the SX6518 chassis The distance between the vertical supports must be between 65 and 80 cm Figure 15 Distance Between the Vertical Supports Vertical Supports peooo50oc5o5on H This edge of the shelf must be as close as possible to the inside of the door when closed Y 3 6 2 ESD Connection Before starting any procedure on the SX6518 series switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit Mellanox Technologies 37 Rev 1 0 Installation 3 6 3 3 6 3 1 3 6 3 2 3 8 Installation Procedure AS Installing the chassis will be a lot easier if you can remove the sides of the rack LA You will need a fork lift to move and insert the chassis into the rack Requirements You will need e 2 phillips screwdriver 3 phillips screwdriver a fork lift e a grounding lug measuring tape ground wire of sufficient length and gauge to properly ground the Y chassis The installation will be much easier with a power screwdriver Y It is recommended to use AWG6 or 4mm diameter wire for grounding purposes SY It is recommended to have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and
53. erheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft LA strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeckung nicht sicher Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redundante A festsitzt oder entfernt wurde 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 6 Anschliefen Trennen von InfiniBand Kupferkabel an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig A Warnungen und Anleitungen des Kabelherstellers 7 Gafahr des elektrischen Schocks
54. es 4 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adec uada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 5 Apilamiento del chasis Los chasis no se deben apilar sobre otros equipos La ca da del chasis podr a causar lesiones corporales as como da os al equipo 6 Dos fusibles uno en el polo y otro en el neutro N Dos fusibles uno en el polo y otro en el neutro Quitar los cables de corriente antes de A abrir la tapa de este producto o tocar cualquier componente interno Mellanox Technologies 107 Rev 1 0 7 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos 8 Conexi n y desconexi n del cable Copper InfiniBand N Dado que los cables de cobre InfiniBand son pesados y no son flexibles su conexi n a a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver adver tencias o instrucciones especiales consultar al fabricante del cable 9 Montaje y mantenimiento de bastidores A Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adop a tar precauciones especiales para garantizar que el sistema se mantenga estable En gen eral en un bastidor los equipo
55. est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A 1 1996 A2 2001 19 S assurer que les enceintes sont appropri es Des enceintes lectriques m caniques et incendie adapt es doivent tre fournies par le fabricant du produit final ou par l utilisateur final Mellanox Technologies 101 Rev 1 0 20 Cordons lectriques CA homologu s UL N Pour les prises lectriques en Am rique du Nord choisissez un cordon lectrique homologu UL et certifi CSA 3 conducteurs 18 AWG termin par une fiche moul e d une tension nominale de 125 V 15 A avec une longueur minimale de 1 5 m 6 pieds et d une longueur maxi male de 4 5 m 18 pieds Pour les prises lectriques en Europe choisissez un cordon lectrique harmonis inter nationalement et marqu lt HAR gt 3 conducteurs d un diam tre de fil minimum de 0 75 mm2 d une tension nominale de 300 V avec une gaine isol e en PVC Le cordon doit avoir une fiche moul e d une tension nominale de 250 V et d une intensit nominale de 10 A 21 40 lbs 40 70 Ibs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 102 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Anhang H Installation Sich
56. evention wrist strap on your wrist and connect to the X6518 series chassis Mellanox Technologies 73 Rev 1 0 Insertion and or Hot swap Extraction 5 5 1 Extracting a Management Module Management modules are located on the leaf side above the leafs There are two places to install the management modules P Only one management module is required to run the switch system 5 Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 58 1 Run the shut down command no power enable module For example to shut down management module 2 run the command below switch 1 Orca648 master config no power enable MGMT2 Disconnect all cables connected to the management module Push outward on the latches to unlock the ejectors from the chassis Open the ejectors until they are 45 degrees from the module HN PON Pull out on the module using both ejectors Figure 57 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Push the latch m The board is short therefore do not let go of it while sliding it out 74 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 58 Management Module em Il Ejectors 5 5 2 Inserting a Management Module Start with the latch fully open that is 45 degrees to the
57. fill the rack with equip ment starting from the bottom to the top 11 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 12 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user gt 18 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 13 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula 4 tions 14 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 15 UL Listed and CSA Certified Power Supply Cord Leakage gt 3 5mA For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at YN 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 16 High Leakage Current LA Warning High leakage current Earth connection essential before connecting supply 17 Add GND connection information Be
58. for switch serial numbers revisions 80 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 software version SwitchX FW version switch temperatures The manager also has the ability to burn new firmware and upgrade software on the switch 7 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It discovers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS routing partitioning to the fabric devices You can view and configure the Subnet Parameters SM via the CLI WebUI Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which 1s connected to the Infiniband fabric host based 7 2 Fabric Inspector Diagnostics Fabric Inspector is a plug amp play software for fee within MLNX OS displaying and filtering all identified systems and nodes within the fabric Fabric Inspector includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections b
59. fore connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation LA GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room Mellanox Technologies 19 Rev 1 0 Hardware 18 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 19 Interconnection of units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent pr
60. gefahrlichem Strahlenkontakt fiihren Klasse Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 104 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 14 UL und CSA Certified Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied LA 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 15 Ableitstrom 3 5mA LEAKAGE gt 3 5mA WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 16 Add GND Verbindung Informationen Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation 17 Installation Codes Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimmung mit den gel YN tenden Vorschriften in der US amerikanischen National Electrical
61. he chassis are MTF005002 Figure 53 Fan Locations on the Chassis a 5 amp mor JM amp ad s d Le CDA 1 b Mellano 3 n 5 4 1 Leaf Fan Module There are four leaf fan modules Two leaf fan modules are found on the leaf side and two leaf fan modules are found on the spine side The leaf side fans are not interchangeable with the spine side fans Air flow through the leafs is independent of the air flow through the spines 5 4 1 1 Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 54 2 Slowly pull out the fan module using the handle Mellanox Technologies 71 Rev 1 0 Insertion and or Hot swap Extraction Figure 54 Leaf Fan Module Extraction Handle Latch 5 4 1 2 Inserting the Leaf Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module If the fan module stops before it goes in all of the way it is inserted incorrectly or it may be for the other side of the chassis 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 5 4 2 Spine Fan Modules Each spine module has a fan module with two individual fans built in When a fan module is not functioning the Fan Status LED on the
62. he grounding post Make sure the con nections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 40 Ground Connection o O e AD LI o o SX6518 Ferrer ye B oboooco00 Donnnnoon Donsonoon ee padod pOD00o000 OOOO ono Dopoooooo panoan 100 e ESD connector Ground Post 3 7 Power Connections The switch includes integrated hot swap power supplies which support up to 6 load sharing 1000W or 1600W supply units The slots for the power supply units PSUs are on the spine side The left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has a dedicated AC inlet This design enables the optional use of separate main and backup AC feeds Mellanox Technologies 57 Rev 1 0 Installation The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be standard 3 wire AC power cords including a safety ground 3 7 1 Powering Up the Switch Platform Make sure that the power cords are compatible with your outlets Power cords for dif ferent countries can be ordered from Mellanox The chassis in N 1 mode must be started with a full complement of PSUs thereafter it can run on one less than the total number of PSUs This fi
63. he phillips head screws holding the sides on Remove and put aside the box containing the cables Remove and put aside the box containing the cable management kit Remove and put aside the box containing the installation kit Nn Pw Visually inspect the chassis make sure that there is no visible damage 6 PSUs are installed all4 fans are installed 7 Remove all protective plastic film from all sides and top of the chassis The leafs spines and management modules are shipped separately gt Mellanox Technologies 41 Rev 1 0 Installation AN Do not remove any of the blocks or either of the lock down bars at this time 3 6 3 5 Installing the Shelf of the pallet Should the chassis start to fall move out of the way The chassis can The chassis is on ball bearings and can roll easily Be aware that the chassis can roll off Ay cause grave bodily harm should it fall on or near you 1 Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low It is recommended to set the top of the shelf at US Try and keep the chassis as low as possible Y vA Placing the shelf too low may make it impossible to tighten the shelf brackets in step The maximum height that the chassis can be mounted will be determined by the maximum height that the fork lift can raise the pallet 2 Open the Installation kit and make sure that all of the parts are in the bo
64. ified physical layer using different block types deskew mechanism and framing rules The SX6518 switch also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with a 64b 66b encoding resulting in an effective band width of 40Gb s FDR10 supports 20 more bandwidth over regular QDR using the same QSFP cables connectors Both FDR and FDRIO support Forward Error Correction FEC as described in IEEE Std 802 3apTM 2007 Amendment to IEEE Std 802 3 2005 chapter 74 P FDR and FDR10 are only guaranteed to work with approved Mellanox Cables A FDR10 is only guaranteed to work with approved Mellanox ConnectX 3 adapters 1 Features for a future release 16 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 2 Hardware 2 1 Installation Safety Warnings These safety warnings are in English For French Spanish and German see the Appendixes 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product LA o lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Installation in Restricted Access Location YN This unit is intended for installation in a Restricted Access Location 4 Over temperature ing the maximum recommended
65. ion 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Checkthe Spine LEDs and make sure they coincide with Figure 43 Figure 43 Spine Side Panel Display Status Indications CES NEM EM IN O STATUS PSU SPINE LEAF MNG 1 MNG 2 STATUS FANS FANS STATUS STATUS 5 Check the Management Module LEDs and make sure they coincide with Figure 44 Figure 44 Management Module Status Indications for Normal Operation AIT Aa ar h h 9 Mellanox Technologies 63 Rev 1 0 Insertion and or Hot swap Extraction 5 Insertion and or Hot swap Extraction Before starting any procedure on the SX6518 switch system put an ESD prevention wrist strap on your wrist and connect to the SX6518 chassis When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions This switch platform supports hot swap capabilities for the parts listed below 5 1 Power supply units Leaf boards Spine boards Leaf fan module Spin
66. itional modules depending on the cho sen customer configuration Optional modules included e Leaf boards e Management modules e Spine boards Figure 4 shows the connector and spine sides fully populated 14 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 4 SX6518 Switch Views Leaf Side gam PR e I e 9 o 9 o e e mo AC ACS Ace ACA Spine Side Mes nox da monnonnn Le mm realen 4 Green pondong 5 paaa anaa Em CENE Up to 6 Power Supply Units 0000000 TIODODDODD 7 ml Amann esee Free 5 naa cpl cn Bonnano A 9 spine modules 9 ET ovocnsgon Banana Bononcoon 6 Two leaf fan modules on each side of the chassis 18 Leaf modules 2 management modules Mellanox Technologies 15 Rev 1 0 Overview 1 2 Features List 324 FDR 56Gb s InfiniBand ports in a 16U switch e 36 36 Tb s aggregate data switching capacity with ultra low latency IBTA 1 3 and 1 21 compliant e SDR DDR QDR FDRIO FDR link speed e Congestion control l e Adaptive routing e Port mirroring e Chassis High Availability e sMB High Availability 1 3 InfiniBand FDR and FDR10 Overview The Mellanox SX6518 switch system supports FDR a pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA spec
67. ix G Avertissements de s curit d installation French 99 Appendix H Installation Sicherheitshinweise German 103 Appendix I Advertencias de seguridad para la instalaci n Spanish 107 Mellanox Technologies 5 Rev 1 0 List of Figures Figure 1 Generic Product Label 13 Figure 2 Product Label Tab Location 13 Figure3 Management Module MAC Address Location 13 Figure4 SX6518 Switch Views 15 Figure 5 Power Supply Unit Status Indications 23 Figure 6 PSU Cover On and Off 24 Figure 7 Leaf Board Led Indicators 24 Figure8 Spine Status LEDs 27 Figure9 Spine Side Panel Display Status Indications 29 Figure 10 Management Module Status Indications 30 Figure 11 Port Numbering 30 Figure 12 Top and Bottom Ports 31 Figure 13 Management Module Interfaces 31 Figure 14 Reset Button 33 Figure 15 Distance Between the Vertical Supports 37 Figure 16 Installation Kit Parts 39 Figure 17 Shock and Tilt Stickers 40 Figure 18 Container Screws 41 Figure 19 Vertical Rack Supports Dimension 43 Figure 20 Adjustable Shelf Brackets 44 Figure 21 Top of Shelf Bracket 44 Figure 22 Placing the Shelf Bracket 45 Figure 23 Caged Nut Locations for the Shelf Brackets 46 Figure 24 Shelf Brackets 47 Figure 25 U Numbering Above Shelf 47 Figure 26 Putting the Chassis in the Rack 48 Figure 27 Place the Upper Bracket Offset 49 Figure 28 Attach the Shelf Extension 49 Figure 29 Caged Nut Placement by U Numbers 50 Figure 30 Location for the Lo
68. lashing Green Data activity flashing speed is proportional to data transfer speed Solid Orange Physical link up Flashing Orange A problem with the physical link The LED indicator corresponding to each data port will light orange when the physical connec tion is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a logical connection is made the LED will change to green When data is being transferred the light will blink green The switch does not provide a visual means to indicate the port speed configuration A SDR DDR QDR or FDR and or the link width 1X or 4X The speed and link ad width configurations can be retrieved using management software Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has the following LEDs e One status LED for the spine health e One status LED for the spine fan modules e 36 status LEDs showing the existence of the leaf to spine connection e One Bad Port Led showing symbol errors in the data stream e One UID LED that can be lit to identify an individual spine The 36 LEDs on each spine are divided by the number of leafs and the result N is the number of connections from each leaf that are connected to the spine 9 spines must be installed and working to ensure that full BW exists between nodes The maximum number
69. le The switch platform comes pre installed with all necessary firmware for standard operation within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system An initial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal Operation can proceed See the installation guide for details regarding the initial configuration Once connected to the network the Subnet Management software automatically discovers and configures the fabric and begins utilizing the switch The Mellanox Operating System MLNX OSTM software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is avail able on the Mellanox web site Basic installation is covered in Chapter 3 Installation on page 34 Hot swapping components and hardware maintenance is covered in Chapter 5 Insertion and or Hot swap Extraction on page 64 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID Identifier and product version information are found on the label that is attached to the pull out tab that is below the Mellanox logo on the spine side of the chassis 12 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 1 Generic Product Label O S N
70. nal PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection When using the 1600W PSUs the minimum complement of power supply units to start the chassis is 3 and the chassis will continue to run at full capacity with only 3 PSUs The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot command in the CLI or reboot through the Web GUI The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cords into grounded outlets Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 58 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 41 Multiple Power Inlets Electric Caution Notifica
71. ng as a master Mellanox Technologies 29 Rev 1 0 Hardware 2 3 5 Management Module LED Indicators Figure 10 Management Module Status Indications LT N Status The management module LEDs display the switch system operating conditions Table 13 LEDs Display for Normal Operation LED Condition Description Normal State STATUS Off No Power Green This LED shows Green System is up and running the status of the Yellow System warning Attention needed such as overheating chassis Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MASTER Off this management module is not the master MM Green Green Management module is operating as a master i e the other management module is a slave 2 3 6 Port Connector Interfaces The connector side of the swi
72. of the connection between the spine and each leaf There is a minimum of one LED per leaf per spine and a maximum of 2 LEDs per leaf These LEDs indicate a valid connection between a leaf and a spine Table 10 shows the leaf to spine status according to the LED condition Table 10 Spine to Leaf IB Link Status LED Condition LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Solid Orange Physical connection 2 3 3 4 Bad Port LED EN The Bad Port indicator is located on the left side of the spine The following Bad Port conditions are possible Table 11 Bad Port LED Configurations LED Condition Description Off OK No ports have received symbol errors recently Flashing Orange Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are e bad cable e bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 2 3 3 5 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support 2 3 4 Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition 28 Mellanox Technologies 324 Port InfiniBand
73. offset Figure 28 Attach the Shelf Extension 3 Screw on the shelf extension using two bolts with flat washers 4 If you are installing the cable management system place 10 caged nuts on the other side of the rack in the vertical supports 5 Place caged nuts in all of the locations specified in Figure 15 The black holes show where to place caged nuts the red holes show the location of the upper brackets on the leaf side Mellanox Technologies 49 Rev 1 0 Installation Figure 29 Caged Nut Placement by U Numbers For the 324 port chassis switches the upper brackets go here These are for the cable holder Place 5 caged nuts Spine side Leaf Side Faceplate caged nuts go on this side 50 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 30 Location for the Lock down Bar Bottom of the chassis 6 Unscrew both lock down bars 7 Place the chassis in front of the rack and as close as possible to the rack Figure 31 Fork Lift Container floor Hinged extension 8 Raise the chassis so that it is 5cm above the shelf in the rack 9 Raise the hinged extension on the container floor 10 Lower the chassis until the hinged extension is sitting on the shelf extension 11 Push the chassis from the container to the shelf SY Save the c
74. ontainer pallet to remove this chassis It will be necessary for the removal and disposal procedures Mellanox Technologies 51 Rev 1 0 Installation Figure 32 Face Plate Mounting Bolt Locations 4 TY ing on the rack sxssan B g BH B g SX6518 Mellanox ec Holes in the faceplate for mount e The holes are separated by 4U units Dong la CL dD D X D D D A EK X GD O AAA 0O D ape U GD dD D CD D 4D D E Mel EE X dD EDEL o D D o ED D 1 D dD Pp ED D ED 12 Slide the chassis onto the shelf and determine the location of the upper brackets on the vertical supports on the connector side of the chassis Do not move the forklift away from the rack just yet 13 Confirm that the caged nuts for the faceplate are in the correct location 14 Slide the chassis out just far enough to install the upper brackets See Figure 33 52 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 33 Upper Bracket Installation This hole is 60cm up from the top of the shelf Figure 34 Upper Brackets and Offset Location ON 15 Screw the upper brackets onto the vertical supports and tighten in place See Figure 34 You will need the socket wrench with both extensions to tighten these bolts Mellanox Technologies 53 Rev 1 0 Installation 16 Slide the chassis onto the shelf until
75. otec tive device rated 20 A must be incorporated in the building wiring 20 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60825 1 2007 03 and EN 60825 1 2007 21 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space ra 22 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way gt lt 23 Country of Norway Power Restrictions This unit is intended for connection to a TN power system and an IT power system of Norway only 20 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 2 2 Power Supply Redundancy The MSX6518 platform comes standard with six power supplies and has two order options e MSX6518 6R N 1 1000W output per PSU e MSX6518 NR N N 1600W output per PSU 2 2 1 1000W Power Supply units With 1000W PSUs the only redundancy option is N 1 In this configuration a PSU Power Supply Unit can be extracted without bringing down the system
76. r laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut pro voquer des blessures corporelles et des d gradations de biens 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lec trique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solidement fix ou s il est enlev 5 Systeme de fusible neutre double p le Avertissement Syst me de fusible neutre double p le Veuillez debrancher tous les cordons d alimentation avant d ouvrir le boitier de ce produit ou de toucher un de ses composants internes 6 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 7 Branchement d branchement des c bles InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire LA tr s attention en les branchant et en les d branchant des connecteurs Consultez le fab ricant des c bles pour conna tre les mises en garde et les instructions sp ciales Mellanox Technologies 99 Rev 1 0 8 Risque de choc et de danger Risque de choc et de danger e lectriques Le de branchment d une seule
77. ration This unit is intended for installation in a Restricted Access Location A restricted access area can be accessed only through the use of a special tool lock and key or other means of security 3 1 Environmental and Safety Recommendations The following are Mellanox recommendations Y Recommended ambient temperature in the System room is 20 5 C Recommended humidity range is 40 15 without condensing SY It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference SY It is highly recommended that the installation site building be equipped with a light ning rod 34 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 3 2 Chassis Package Contents The package includes e 1 chassis with the following modules installed 4 fans 6PSUs e 1 installation kit e 1 box containing 6 power cords 250v 15a 2 0M C14 to C13 USA UL Standard e 1 cable man
78. re 58 1 Run the shut down command no power enable module For example to shut down leaf 16 run the command below switch 1 Orca648 master config 4 no power enable L16 Disconnect all cables connected to the leaf Push the latch to unlock the ejector from the chassis Open the ejector until it is 45 degrees from the leaf w e pp Pull out on the leaf 66 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 48 Ejector Latch Pull the leaf out of the chassis Leaf Leaf E Push the latch ee release RS Ejector Ah The board is short therefore do not let go of it while sliding it out Figure 49 Leaf Release Push here to release the latch Pull the ejector to remove the leaf 5 2 2 Inserting a Leaf Board Start with the latch fully open that is 45 degrees to the front panel of the leaf Carefully set the leaf board into the side guide rails and slowly slide the board into the chassis Catch the hooks onto the vertical bar and push the latch shut Mellanox Technologies 67 Rev 1 0 Insertion and or Hot swap Extraction 5 3 Spine Boards When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions
79. s 5 3 2 Inserting a Spine Board Start with the latches fully open that is 45 degrees to the front panel of the leaf Hold the spine board by the sides with both latches fully open Insert the board into its rail The board should slide in without resistance Push the board in until the hooks on the latches are touching the vertical bar n ode ow ob or Now press both latches inward simultaneously You should feel the latches are hooked into the front panel 6 Press both latches applying some force to close them completely The locking latch will engage 5 4 Fan Modules There are four fan modules on the chassis for the leafs They are located on both sides of the chas sis two on the leaf side corners and two on the spine side When a fan module is not functioning the status LED on the fan will light up Each spine has an individual fan module that contains two individual fans Should a single fan fail the Fan Status LED on the spine and the S Fan LED on the management module will light indi cating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs 70 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 There are two different fan modules for the chassis These fans have different model numbers and are not interchangeable The fans on the leaf side of the chassis are MTF005001 whereas the fans on the spine side of t
80. s se deben instalar comenzando desde abajo hacia arriba 10 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 11 Asegurar confinamientos adecuados un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen El fabricante del producto final o el usuario final deber n suministrar A dio 12 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 13 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales 108 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 14 Cable de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe mol
81. shut down the device 3 Emergency 130 C In case the firmware fails to shut down the SwitchX device upon crossing the Critical threshold the SwitchX device will auto shutdown upon crossing the Emergency 130 C threshold 92 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Appendix C Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculated for any possible customization as follows Take the weight of a chassis with the following FRUs installed e All fans modules e All power supplies The weight of the SX6518 chassis configured above is 112kg To this add the weight of installed FRUs e Spines e Leafs e Leaf blanks e Management modules Fill in the table below to calculate the weight of your system Number of FRUs FRU Type Weight of 1 FRU Total Weight FRU Type Chassis as shipped of Spines 3 77 of leafs 2 76 of leaf blanks 0 764 of management modules 13 15 Total This total is in kilograms Multiply the total by 2 2 to get the total weight in pounds Mellanox Technologies 93 Rev 1 0 Appendix D Calculating the Power of a Chassis To calculate the power consumption of a chassis take the base power and add the power of the fans spines management modules and leafs Table 15 Power Consumption of Chassis Parts Part Typ Po
82. spine will light up Figure 55 Spine Fan Module Pet 4 OOOOOU0UL ALLE JODDDODUDD seg CODOODOOU LILA rir rr ASA REE SE PERE SE Fan Module When a fan module is removed the indicator light will reset 72 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 56 Fan Status LED on the Spine Module ELLE aln alalaln 10 12 14 E Ao oo Fan Status 5 4 2 1 Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out 5 4 2 2 Inserting the Spine Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new spine fan module 3 Push the fan module as far as it will go make sure the locking latches engage IN y N If the Fan LED continues to show red remove the fan module and check the pins on the connector inside if the spine to make sure that none of them are bent 5 5 Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing Ab the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions Before starting any procedure on the X6518 series switch system put an ESD pr
83. spine you want to hot swap is connected to the master management module the management module will reboot when you take out the spine If you have only one management module the chassis will crash Figure 51 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Push the latch lee release Figure 52 Spine Board Extraction Pull the latches to remove the spine EBEN In EG 000000000 HANNA g ODO BEN Re OOOOO0000 8 Jo ET Mellanox Technologies 69 Rev 1 0 Insertion and or Hot swap Extraction If you need to hot swap spine 1 or spine 2 1 Check to see if the spine you need to remove is connected to the master management module See Figure 50 Warning If you have only one management module the chassis will crash 2 Push outward on the latches to unlock the ejectors from the chassis 3 Open the ejectors until they are 45 degrees from the module 4 Pullout on the module using both ejectors 5 3 1 2 Extracting Spine Boards Except 1 or 2 1 Run the shut down command no power enable module For example to shut down spine 06 run the command below switch 1 Orca648 master config 4 no power enable S06 Push outward on the latches to unlock the ejectors from the chassis Open the ejectors until they are 45 degrees from the module 4 Pullout on the module using both ejector
84. stall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 4 on page 70 6 Replace the spine fan module LA Replace defective spine fan modules as soon as they are identified 8 4 Spine Board The yellow LED on the Spine board is lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 5 3 on page 68 5 Make sure that the spine and the Leafs both have the same version of FW 6 Reburn the FW and remove and reinstall the spine 7 Replace the spine board Mellanox Technologies 85 Rev 1 0 TroubleShooting 8 5 MLNX OS Software The last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop Push the reset button on the switch or management module 2 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 mlnx1 4 May 12
85. tch has 18 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 11 The bottom row ports are flipped from the top row See Figure 12 Figure 11 Port Numbering 1 3 5 7 9 W 1 13 1s 7 A A A E A AA A A AA a ES a M A 2 4 6 8 10 12 14 16 18 30 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 12 Top and Bottom Ports Ses E 2 4 Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side 2 5 QSFP Cable Power Budget Classification All SX6518 QSFP switches are designed for active cables with a max power per module of 2 0W This is power level 2 according to the QSFP Public Specification 2 6 Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the SX6518 They are e I2C port e 1 CONSOLE port this is an RS232 connector for connecting to a host machine e 1 MGT this is an Ethernet connector e 1 USB port e 1 RST reset button Figure 13 Management Module Interfaces ME CONSOLE MGT LD RST PSU x ih Phu Tr I O I only A This interface is for Debug and Troubleshooting only This interface is for FAEs Mellanox Technologies 31
86. the upper bracket to upper bracket offset screws See Figure 35 26 Tighten the lock down bars See Figure 38 27 Go to Section 4 Chassis Power Up on page 62 to power up the switch system 3 6 4 Installing the Cable Holder Y Now is the time to install the cable holder Figure 39 Cable Holder A 0 A P e Nes ve P T Left side rack D ye P _ Top side A y w P d e i i P d e 8 La 7 A e 14 X M6 caged nut e 14 X M6 bolt for caged e nut e 1 Place the Cable holder next to the rack on the connector side of the chassis and identify the holes where the caged nuts were placed in step 4 The cable holders should go to the outside of the vertical supports aa 2 Set the bottom of the cable holder at the level of the shelf 3 Screw the cable holder onto the rack using the screws provided 4 Repeat steps 1 3 for the second cable holder 56 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 3 6 5 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect t
87. tion CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Schocks Risque de choc et de danger hazard Entferrnen des Netzsteckers elnes e lectriques Le de branchment The PSUs are all independent Netzteils spannungsfrei Um alle d une seule alimentation stabilise e Einhieten spannungsfrei zu machen ne de branch uniquement qu un Disconnect all power supplies to sind die Netzstecker aller Netzteile module Alimentation Stabilise e ensure a powered down state inside zu entfernen Pour isoler completement le module of the switch platform en cause Il faut de brancher toutes les alimentations stabilise es 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation 4 N Any yellow or red status LEDs on any of the management modules is cause for con 4 W cern and must be dealt with immediately 4 Check that none of the LEDs on the spines are yellow pn N It can take up to 5 minutes to boot up the system Turn off the system if any LEDs W remain red for more than 5 minutes 5 Check that the Leaf status LEDs Fan status LED and Spine status LED in the spines are all green iN The maximum number of connections from each leaf is 2 If the number of leafs x the A maximum number of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF Figure 42
88. ul to the user This symbol indicates a situation that can potentially cause damage to hardware or software Warning This symbol indicates a situation that can potentially cause personal injury and or damage to hardware or software Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies SX System Type SwitchX Switch PR Data Transfer Protocol 1 2 3 ZEN 5 6 7 8 2 IB G Module Generation 5 6 7 8 FF Number of leafs 36 18 12 6 Separator P Power Supplies N N N redundant 020 1 1 2 2 R RoHS C RoHS5 R RoHS6 Mellanox Technologies 11 Rev 1 0 Overview 1 Overview This User Manual provides an overview of the SX6518 QSFP Modular InfiniBand Switch Plat form known in this document as the chassis or switch and its operational environment Mellanox SX6518 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Performance Computing HPC and Embedded environments Networks built with the SX6518 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mel lanox s 5 generation SwitchX VPI switch device SX6518 systems provide up to 56Gb s full bidirectional bandwidth per port With up to 324 ports in a 17U high form factor these systems are among the densest switching systems availab
89. wer Passive Cables AY EONEE Le ae prp ii IB FDR leaf 91W 125 Spine 108W Not applicable Fan 117W Not applicable Management Module 14W Not applicable To calculate the chassis power at typical conditions fill in the table below Typ Power of 1 Total Power FRU Number of FRUs FRU Type FRU Type of Spines 108 of leafs passive cables 9 of leafs active cables 125 of management modules 14 for the PC 460 management module Chassis Fans 15 Total not including Power efficiency of 85 Total 0 85 Divide the total by 0 85 the power supply efficiency 94 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Appendix E QSFP Interface Table 16 InfiniBand QSFP Connector Pinout Connector Connecto Pin r Pin Signal Description Number Name 1 GND Ground 2 Tx2n Transmitter Inverted Data Input J3 and J6 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 1 Al GND GND NA 5 Tx4n Transmitter Inverted Data Input Giant dus 6 Tx4p Transmitter Non Inverted Data Input _ 22 Rx2p Rxip t 23 GNE GND 16 T GND Ground 24 15 8 ModSelL Module Select Rx4 Rx3n IT 25 2e us 14 9 ResetL Module Reset Rx4p Rx3p 26 E an 13 10 Vcc Rx 3 3 V Power supply receiver 27 12 11 SCL
90. x See Figure 16 Installation Kit Parts The chassis will sit in the rack with the spine side as close as possible to the door This leaves max imum room for the cable management system The distance between the vertical supports must be between 65 cm 25 6 and 80 cm 31 5 If the distance in your rack is different move the vertical supports so that one pair is 10 cm 4in away from the door The side of the chassis with the spine units will sit flush with the vertical supports of the rack The side of the chassis with the QSFP connectors will sit at the edge of the shelf closer to the center of the rack 42 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware User Manual Rev 1 0 Figure 19 Vertical Rack Supports Dimension Vertical Supports 3 Measure the distance between the vertical supports and adjust the shelf brackets according to this distance This chassis can be installed in a rack with a distance between the vertical sup ports of from 65cm 25 6in to 80cm 31 5in 4 The shelf comes with the four brackets pre installed Two of the brackets should be loose They may need to be removed and repositioned to meet the distance between the vertical sup ports of the rack Mellanox Technologies 43 Rev 1 0 Installation Figure 20 Adjustable Shelf Brackets These two brackets will need to be adjusted They may even need to be removed and repositioned

324 Port InfiniBand FDR Switch Platform Hardware User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents