Home

648 Port InfiniBand FDR Switch Platform User Manual

1. Part Description OPN 648 port FDR capable modular chassis includes fans MSX6536 10R and power supplies RoHS6 Power supply unit PSU MTP005001 Power supply blank MTM005001 Leaf board unit 18 port QDR MSX6001QR Leaf board unit 18 port FDR MSX6001FR Leaf board unit 18 port FDR 10 MSX6001TR Modular Switch Family Leaf Blank MTM005004 Spine board unit for FDR system MSX6002FLR Spine board unit for FDR10 QDR system MSX6002TBR Spine board blank MTM005002 PPC460 Management Module MSX6000MAR x86 Management Module MSX6000MBR Modular Switch Family Management Blank MTM005003 Rack Installation Kit for IS5600 SX6536 Series MTR005600 Cable Management Device Set for IS5600 SX6536 MTR005601 Cables Set For IS5600 SX6536 Series MTRO05602 IS5600 S X6536 IS5300 SX6518 Series Modular MTF005001 Switch leaf cooling chassis fan unit located on Leaf Side 155600 5X65336 IS5300 SX6518 Modular Switch MTFO05002 leaf cooling chassis fan unit located on Spine Side IS5600 SX6536 IS5300 SX6518 Modular Switch MTF005005 Series Spine Fan Unit Fan unit for spines MTF005003 DB9 to RJ45 Harness HAR000028 Power cord 250V 15A 2 0M C14 to C13 ACC000334 Power cord 125V 15A 2 0M C14 TO C13 ACC000242 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Mellanox Technologies 91 Rev 1 0 Part Description OPN Power cord Type H for Israel ACC000205 Power cord
2. Anhang G Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 Ubertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft LA strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeckung nicht sicher Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redundante A festsitzt oder entfernt wurde 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlieBen Sie keine Kabel an oder ab 6 Anschlie en Trennen von InfiniBand Kupferkabel an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb miissen sie vorsichtig Ay Warnungen und Anleitungen des Kabelherstellers 7
3. berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 98 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Appendix H Advertencias de seguridad para la insta laci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instruc ciones de instalaci n 2 Lesi n corporal por peso Dado que el equipo es muy pesado se debe mover nicamente mediante un elevador mec nico para evitar lesiones gt 3 Instalaci n en un lugar con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 4 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C 113 F Adem s para garantizar una circu laci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pul gadas alrededor de las aberturas de ventilaci n gt 5 Apilamiento del chasis a Los chasis no se deben apilar sobre otros equipos La caida del chasis podria causar lesiones corporales as como da os al equipo 6 Conexi n de fuente de alimentaci n redundante peligro de descarga el ctrica Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n
4. The list of approved certifications per chassis in different regions of the world is located on the Mellanox Website at www mellanox com related docs user_manuals Regulatory_and_Compliance_Guide pdf EMC Statements are also in the Regulatory and Compliance Guide 86 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Appendix B Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculated for any possible customization as follows Take the weight of a chassis with e all fans modules e all power supplies The weight of the SX6536 chassis configured above is 157 57kg To this add of spines 3 73 of leafs 2 67 of leaf blanks 0 764 3 15 if you will have two management modules 1 15 if you will have only one management module weight of the chassis plus the leaf boards spines PSUs and Management modules installed in the chassis Mellanox Technologies 87 Rev 1 0 Appendix C Calculating the Power of a Customized Chas sis To calculate the power consumption of a customized chassis take the base power and add the power of the fans spines management modules and leafs Table 15 Power Consumption of Chassis Parts Part typ power passive cables Sp hover E CANIS IB FDR leaf 91W 125 Spine 108W Not applicable Fan 15W Not applicable Management Module 14W Not applicable To calculate the
5. Mellanox Technologies 17 Rev 1 0 Hardware 7 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 8 Copper InfiniBand Cable Connecting Disconnecting attached to or detached from the connectors Refer to the cable manufacturer for spe Copper InfiniBand cables are heavy and not flexible as such they should be carefully A cial warnings instructions 9 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken LA to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 10 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 11 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user 12 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula LA tions 13 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical 4 codes 18 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 14 UL Listed and CSA Certified Power Supply Cord Leakage gt 3 5mA For North American power connection select a pow
6. Table 14 SX6536 Switch Specification Sheet Physical Power and Environmental InfiniBand QoS Management Auto Negotiation of 56Gb s 40Gb s 20Gb s 10Gb s 8 InfiniBand Virtual Lanes for all ports Baseboard Performance and Device management Agents for full InfiniBand In Band Management Safety EMC Environmental Accoustic US Canada cTUVus EU IEC60950 International CB Russia GOST R Argentina S mark China CCC for power supply Korea KN22 2209 KN24 2009 USA FCC Class A Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A Australia C Tick EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 Scalability and Performance Replaceable Parts Switching Performance Addressing Switching Capacity Simultaneous wire speed any port to any port 48K Unicast Addresses Max per Subnet 16K Multicast Addresses per Subnet 72 5 Tb s Leafs Power Supplies Leaf Fan Modules Spine Fan Modules Spines Management Modules Up to 36 leafs 18 ports each 10 Power Supplies standard with 2 optional redundant power supplies available 8 Leaf Fan Modules 18 Spine Fan Modules 18 2 available 1 required for oper ation Mellanox Technologies 85 Rev 1 0 A 2 EMC Certifications
7. remove the cover to the power supply unit for the non work ing PSU Odd numbered PSUs are on the left side and even numbered PSUs are on the right side facing the spines There are four phillips head screws for the cover plate 62 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 46 Power Supply Handle Latch CD r DGS AC 3 lt r Ee FAIL N E 3 Grasping the handle with one hand push the black latch release while pulling the handle out ward As the PSU unseats the PSU status indicators will turn off 4 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation Insert the power cord into the supply connector on the other side of the chassis 5 Replace the cover over the PSUs Y N The green indicators should light If not extract the PSU and re insert it again 6 2 Leaf Boards When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sion
8. 2 for redundancy 18 spine modules 285mm our leaf fan modules on this side of the chassis A EA D GE D E D 1 amp e D e U Ih D Ed E 14 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 2 Features List e 648 FDR 56Gb s ports in a 29U switch e 72 52 Tb s aggregate data switching capacity with ultra low latency e IBTA 1 3 and 1 21 compliant SDR DDR QDR FDR10 FDR link speed e Congestion control e Adaptive routing e Port mirroring FDR10 supports 20 more bandwidth over regular QDR using the same cables connectors e chassis High Availability e sMB High Availability FDR10 supports Forward Error Correction FEC Mellanox Technologies 15 Rev 1 0 FDR 2 FDR The Mellanox SX6536 switch system supports FDR a pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified physical layer using different block types deskew mechanism and framing rules The SX6536 switch also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with a 64b 66b encoding resulting in an effective band width of 40Gb s FDR10 supports 20 more bandwidth over regular QDR using the same QSFP cables connectors Both FDR and FDRI10 supports Forward Error Co
9. 20A doit tre incorpor dans le c blage lectrique du b timent gt 16 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 17 S assurer que les enceintes sont appropri es Des enceintes lectriques m caniques et incendie adapt es doivent tre fournies par le fabricant du produit final ou par l utilisateur final gt 18 Cordons lectriques CA homologu s UL Pour les prises lectriques en Am rique du Nord choisissez un cordon lectrique homologu UL et certifi CSA 3 conducteurs 18 AWG termin par une fiche moul e d une tension nominale de 125 V 15 A avec une longueur minimale de 1 5 m 6 pieds et d une longueur maxi male de 4 5 m 18 pieds Pour les prises lectriques en Europe choisissez un cordon lectrique harmonis inter nationalement et marqu lt HAR gt gt 3 conducteurs d un diam tre de fil minimum de 0 75 mm2 d une tension nominale de 300 V avec une gaine isol e en PVC Le cordon doit avoir une fiche moul e d une tension nominale de 250 V et d une intensit nominale de 10 A Mellanox Technologies 95 Rev 1 0
10. chassis power 108W of spines 14W of management modules 15W of fans 91W of leafs for passive cables or 125W of leafs for active cables 0 85 Where 0 85 is the power supply efficiency 88 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Appendix D QSFP Interface 0___ en eno Y Table 16 InfiniBand QSFP Connector Pinout 21 non Rxin 18 _ 22 Rx2p Rxip 17 _ Connector Connecto 23 GND anp 16 Pin r Pin Signal Description 24 15 Number Name Rx4n Rx3n IT 25 Rxdp Rx3p 14 1 GND Ground 26 GND GND 13 2 Tx2n Transmitter Inverted Data Input 27 ModPrsL SDA 12 3 Tx2p Transmitter Non Inverted Data Input 28 Int SCL 11 4 GND Ground 29 VocTx Veo Rx 10 5 Tx4n Transmitter Inverted Data Input 30 Veet Reset __2 6 Tx4p Transmitter Non Inverted Data Input 21 LPMode ModSelL 1 GND Ground 32 GND GND 7 8 ModSelL Module Select 3 Tx3p Tx4p 9 ResetL Module Reset a En A 10 Vcc Rx 3 3 V Power supply receiver 36 ae gt 3 11 SCL 2 wire serial interface clock p P 37 Dis BAr 2 12 SDA 2 wire serial interface data 38 GND GND 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxin R
11. designed for active cables with a max power per module of 2 0W This is power level 2 according to the QSFP Public Specification 3 4 Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the SX6536 They are e 1 12C port e 1 CONSOLE port this is an RS232 connector for connecting to a host machine e 1 MGT this is an Ethernet connector e 1 USB port e 1 RST reset button Figure 13 Management Module Interfaces CONSOLE MGT gt RST PSU S FANS L h h ON Mellanox Technologies 29 Rev 1 0 Hardware 3 4 1 12C This interface is for Debug and Troubleshooting only This interface is for FAEs only 3 4 2 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure 3 4 3 MGT Management The MGT port is an Ethernet port for remote management Any remote terminal connected to the Ethernet can then be used to manage the fabric and chassis Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10M 100M 1G auto negotiation gt gt Initial configuration must be done on all of the management modules gt 3 4 4 USB The USB port can be used to upload new SW using any storage device th
12. driver 4 flat washers for the shelf extension 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 shelf extension 1 right side upper bracket offset 2 upper bracket washers 12 lock washers for the upper brackets to upper bracket offsets 34 caged nuts 20 for the shelf and 14 for the chassis faceplate 20 flat washers 8 for the lock down bars and 12 for the upper brackets 32 hex head bolts 20 for the lock down bars and 12 for the upper brackets to upper bracket offsets 12 flat head screws with locktite for the upper bracket offsets 1 socket nut driver 8mm 1 socket nut driver 10mm 1 socket nut driver 13mm Parts included in the cable management installation kit 1 cable management rack RH 16 caged nuts M6 1 cable management rack LH 16 M6 bolts Mellanox Technologies 37 Rev 1 0 Installation Figure 16 Installation Kit Parts Shelf Lock down bar Flat head screws with Shelf extensi i locktite men 6 hex head bolt Upper bracket Washer Phillips pan head screw A 8 hex head bolt Caged nuts g Lock washers a amp Lock washers i E Flat washers NEY Upper bracket o Flat washers Upper bracket offset 4 6 3 2 Opening the Container 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 2 Remove the four sides of the container by unscrewing the phillips head screws holding the
13. failure Turn the chassis off and then troubleshoot LEAF FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot MGT 1 STATUS Off no power or this management module is not installed Green Green Management module is operating as a master Yellow Management module is operating as a slave i e the other man agement module is a master MGT 2 STATUS Off no power or this management module is not installed Yellow Green Management module is operating as a master Yellow Management module is operating as a slave i e the other man agement module is a master Mellanox Technologies 27 Rev 1 0 Hardware 3 1 5 Management Module LED Indicators Figure 10 Management Module Status Indications tatus The management module LEDs display the switch system operating conditions Table 13 LEDs Display for Normal Operation Normal LED Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE
14. hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A Mellanox Technologies 97 Rev 1 0 14 Ableitstrom gt 3 5mA LEAKAGE gt 3 5mA WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 15 Add GND Verbindung Informationen Add GND connection information Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation 16 INSTALLATION CODES INSTALLATION CODES Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in Ubereinstimmung mit den gel LA tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 17 Zusammenschaltung von EINHEITEN INTERCONNECTION OF UNITS Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal LA tung
15. management module will start a new cycle checking through the leafs and spines for the FW ver sions A PSU Power Supply Unit can be extracted without bringing down the system as long as no more than one PSU is bad The power required to run the switch system is equally divided between all of the working PSUs 6 1 1 Extracting and Inserting the Power Supply Unit With all of the power supplies installed the system is in the redundant configuration any single PSU may be extracted without bringing down the system To extract a PSU 1 Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering Mellanox Technologies 61 Rev 1 0 Insertion and or Hotswap Extraction Figure 44 Power Cord Top Bar Figure 45 PSU Locations jg ala AEN ar ji PSU 1 a PSU 2 al lnlallallalelinlalatenmiallale UNE Het AO PSU 3 PSU 4 I earth er H Hl nn lol ol oll ITE o o an a i nin alalla p y el num PSU 5 EN PSU 6 PSU 7 PSU 8 PSU 9 PSU 10 2 On the spine side of the chassis
16. the holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 6 4 on page 67 LA Replace defective leaf fan modules as soon as they are identified Should any of the boards shut down due to over temperature follow the procedure start ing in Step 1 9 3 3 Yellow LED for the Spine Fan is Lit 1 Determine which spine has a defective fan 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit 1s free of any dirt and or obstacles See Section 6 4 on page 67 Mellanox Technologies 81 Rev 1 0 TroubleShooting LA Replace defective spine fan modules as soon as they are identified 9 4 Spine Board The yellow LED on the Spine board is lit 1 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over t
17. AMA Mellanox TECHNOLOGIES 648 Port InfiniBand FDR Switch Platform User Manual PN SX6536 10R Rev 1 0 www mellanox com Rev 1 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCU MENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRON MENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGE MENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PROD UCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
18. DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 O Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE Direct InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale InfiniPCIO PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS and SwitchX are trademarks of Mellanox Technologies Ltd All other marks and names mentioned herein may be trademarks of their respective companies FDR InfiniBand Switch System User Manual 2 Mellanox Technologies Document Number 3490 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table of Contents Table of Contents 3 Table of Figures 5 Table of Lists 7 Revision History 8 About this Manual 9 Intended Audience 9 Related Documentation 9 Conventions 10 Chapter 1 Overview 11 1 1 Product Information 11 1 1 1 Serial Number and Product Version Information 11 1 1 2 Management Module MAC 12 1 2 Features List 15 Chapter 2 FDR 16 Chapter 3 Hardware 17 3 1 LED Status Indicators 21 3 1 1 Power Supply Unit LEDs 21 3 1 2 Leaf Board LED Indicators 22 3 1 3 Spine Board LED Indicators 24 3 1 4 Spin
19. FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot LEAF FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot MASTER Off no power or this management module is not installed Green Green Management module is operating as a master i e the other management module is a slave Yellow Management module is operating as a slave i e the other man agement module is a master 3 1 6 Port Connector Interfaces The connector side of the switch has leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 11 The bottom row ports are flipped from the top row See Figure 12 28 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 11 Port Numbering 1 3 5 7 9 11 13 is 17 ARA A A AA O EA PEA ROS VA AAN AAA AA AO SAS AE AO AE SAA 2 4 6 8 1 0 E 14 16 18 Figure 12 Top and Bottom Ports eee 3 2 Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side 3 3 QSFP Cable Power Budget Classification All SX6536 QSFP switches are
20. NTION Risk of electric shock and Gafahr des elektrischen Schocks Risque de choc et de danger energy hazard Entferrnen des Netzsteckers e lectriques Le de branchment The PSUs are all independent elnes Netzteils spannungsfrei d une seule alimentation stabi Um alle Einhieten spannungsfrei lise e ne de branch uniquement Disconnect allpower suppliesto zu machen sind die Netzstecker qu un module Alimentation ensure a powered down state aller Netzteile zu entfernen Stabilise e Pour isoler com inside of the switch platform pletement le module en cause Il faut de brancher toutes les ali mentations stabilise es 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Any yellow or red status LEDs on any of the management modules is cause for con cern and must be dealt with immediately It can take up to 5 minutes to boot up the system During this time the status LEDs may be RED Turn off the system if any LEDs remain red for more than 5 minutes 5 Check that the Leaf status LED Fan status LED and Spine status LED in the spines are all green 56 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 40 Spine Module DODODO000 ODO0000000 ODO 000000 QC Status Fan Bad Port and UID LEDs Leaf connection LEDs 4 8 InfiniBand QSFP Cable Installation The switch use
21. Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 96 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 8 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden gt 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen gt 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 11 Richtigen Schutz sicherstellen Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen gt 12 Strahlenkontakt Achtung Nutzung von Steuerungen oder Einstellungen oder Ausfiihrung von Proze duren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkontakt fiihren Klasse Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 13 UL und CSA Certified Netzkabel UL Listed and CSA Certified Power Supply Cord F r Nordamerika Stromanschluss w
22. Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 92 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Appendix F Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 45 C 113 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut pro voquer des blessures corporelles et des d gradations de biens gt 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lec trique redondante Ne pas faire fonctionner le produit si l
23. a valid ground Make sure that this is a tight fit 4 6 3 Installation Procedure Installing the chassis will be a lot easier if you can remove the sides of the rack LA You will need a fork lift to move and insert the chassis into the rack 4 6 3 1 Requirements You will need e 2 phillips screwdriver e 3 phillips screwdriver e a fork lift e a grounding lug e measuring tape e ground wire of sufficient length and gauge to properly ground the chassis The installation will be much easier with a power screwdriver It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation 36 Mellanox Technologies Parts included in the installation kit 1 shelf with brackets pre installed 1 left side upper bracket offset 2 lock down bars 8 lock washers for the upper brackets to vertical support 12 hex head bolts 8 for the upper brackets and 4 for the shelf extension 34 pan head screws for the caged nuts 20 for the shelf and 14 for the chassis face plate 32 lock washers 20 for the lock down bars and 12 for the upper brackets to upper bracket offsets 2 upper brackets 1 open end ratchet wrench 10mm 1 socket wrench 2 extensions for the socket
24. allet 2 Open the Installation kit and make sure that all of the parts are in the box See Figure 16 Installation Kit Parts The chassis will sit in the rack with the spine side as close as possible to the door tance in your rack is different move the vertical supports so that one pair is 10 gt The distance between the vertical supports must be between 63 and 80 cm If the dis cm 4in away from the door the rack The side of the chassis with the QSFP connectors will sit at the edge of the d The side of the chassis with the spine units will sit flush with the vertical supports of shelf closer to the center of the rack 3 Measure the distance between the vertical supports and adjust the shelf bracket according to this distance This chassis can be installed a rack with a distance between the vertical supports of from 65cm 25 6in to 80cm 31 5in 40 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 18 Vertical Rack Supports Dimension 4 The shelf comes with the four brackets pre installed Two of the brackets should be loose They may need to be removed and repositioned to meet the distance between the vertical sup ports of the rack Figure 19 Adjustable Shelf Brackets These two brackets will need to be adjusted They may even need to be removed and repositioned The holes in the bracket are offset to one side The holes must be closer to the t
25. and a fire alarm warning system The system requires a KVA rated UPS system The heat dissipation is calculated Power Dissipation x 3 42 Heat Dissipation It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 4 2 Chassis Package Contents The package includes e 1 chassis with the following modules installed 8 fans 10PSUs e 1 installation kit e 1 box containing 10 power cords 250v 15a 2 0M C14 to C13 USA UL Standard e 1 cable management kit Before you install your new SX6536 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have occurred during shipping gt The leafs spines and management modules are shipped separately Mellanox Technologies 33 Rev 1 0 Installation 4 3 4 4 4 5 4 6 If anything is damaged or missing contact your customer representative immediately 4 The rack mounting holes conform to the IEA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In part
26. at has a USB connector This interface is USB 2 0 3 4 5 Reset RST The Reset button resets the chassis management module when the button is pushed When the but ton is held down for 15 seconds the management module is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip 30 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 14 Reset Button CONSOLE MGT a RST PSU S FANS L FANS MASTER h Mh This button resets the CPU of the management module A quick push of this button performs this reset When the reset button is pushed on the master management module this management mod ule is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin Mellanox Technologies 31 Rev 1 0 Installation 4 Installation d This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports of the rack A Installation an
27. ations Holes in the faceplate for mount ing on the rack Mellanox TECHNOLOGIES SX6536 The holes are separated by 4U units S S13 E ES CR a A D g N H 11 Slide the chassis onto the shelf and determine the location of the upper brackets on the vertical supports on the connector side of the chassis 12 Slide the chassis out just far enough to install the upper brackets 50 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 32 Upper Brackets and Offset Location Figure 33 Upper Bracket Installation This hole is 60cm up from the top of the shelf Mellanox Technologies 51 Rev 1 0 Installation 13 Screw the upper brackets onto the vertical supports and tighten in place You will need the socket wrench with both extensions to tighten these bolts 14 Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack sup port 15 Make sure the upper brackets are installed in their correct location 16 Screw the bolts through the faceplate into the caged nuts and tighten all 14 bolts and nuts 17 For each offset and upper bracket screw six bolts through the offset into the upper bracket Use one flat washer and one lock washer for each of these bolts Figure 34 Offset to Upper bracket Bolts 18 Place the front lock down bar at the edge of the
28. c Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections based on traffic patterns and user assigned system names GUIDs Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH e Telnet WEB e SNMP XML Upgrading Software The new software and firmware update is available to the user from the Mellanox Support website Copy the update to a known location on a Remote server within your LAN Use the CLI or the GUI in order to perform the Software upgrade For further information please refer to the MLNX OS user manual If MLNX OS is updated and the FW image in the leafs and spines of the chassis is an gt 4 earlier version than the minimum that the new version of the software can work with then the chassis management system may require up to 45 minutes to update all of the FW images in all of the leafs and spines 78 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 8 4 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It discovers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS routing partitioning to the fabric devices You can view and configure the Subnet Paramet
29. chassis and align the holes in the lock down bar with the holes in the shelf Screw down the lock down bar 52 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 35 Lock Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf 19 Screw down the second lock down bar Figure 36 Bolt and Washer Order 20 Remove the shelf extension and save for removal of the chassis 21 Connect a valid ground See Section 4 6 5 Ground Connections on page 55 for detailed instructions 22 Go to Section 5 Chassis Power Up on page 59 to power up the switch system Mellanox Technologies 53 Rev 1 0 Installation 4 6 4 Installing the Cable Holder Now is the time to install the cable holder Figure 37 Cable Holder Left side rack Top side Place the Cable holder next to the rack on the connector side of the chassis and identify the 1 holes where the caged nuts were placed in step 5 d The cable holders should go to the outside of the vertical supports 2 Set the bottom of the cable holder at the level of the shelf 54 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 3 Screw the cable holder onto the rack using the screws provided 4 Repeat steps 1 3 for the second cable holder 4 6 5 Ground Connections Make sure to connect the ground post to a valid elect
30. codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 17 Interconnection of units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit YN Overcurrent Protection A readily accessible Listed branch circuit overcurrent protec tive device rated 20 A must be incorporated in the building wiring 18 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure Mellanox Technologies 19 Rev 1 0 Hardware CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60825 1 2007 03 and EN 60825 1 2007 19 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 20 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 3 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of the IT manager NV 4 W
31. ctar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DUCTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 17 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe 4 instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 18 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de A tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una cor riente nominal de 20 A Mellanox Technologies 101 Rev 1 0 19 Exposici n a niveles de radiaci n pe
32. d initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the openings are populated with blanks All of the leafs spines and managmement modules come shipped in a separate pakage The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instruc tions to make the initial configuration This unit is intended for installation in a Restricted Access Location A restricted A access area can be accessed only through the use of a special tool lock and key or 4 other means of security 4 1 Environmental and Safety Recommendations The following are Mellanox recommendations Y Recommended ambient temperature in the System room is 20 5 C Recommended humidity ranges is 40 15 without condensing It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference V 32 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 It is highly recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors
33. ding to the LED indication Table 7 Connector Physical and Logical Link Indications LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed data transfer speed Orange Physical link up Flashing Orange A problem with the physical link A SDR DDR QDR or FDR and or the link width 1X or 4X The speed and link gt The switch does not provide a visual means that indicates the port speed configuration M width configurations can be retrieved using management software 3 1 3 Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has one status LED for the spine and 36 status LEDs The 36 LEDs on each spine are divided by the number of leafs and the result X is the number of ports from each leaf that is connected to the spine spines must be installed and working to ensure that full BW exists between nodes Should a leaf be removed then X number of LEDs will turn off on each of the spines The status LEDs for the spine and their descriptions are shown in Table The LEDs indicate as fol lows 24 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 8 Spine Status LEDs LLILISGELLILIE afefafalafaralafa AA 000000000 LETI rrr Spine Status Fan Status Spine Leaf IB Connection Status 3 1 3 1 Status LED The Spine Status ind
34. e Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one 99 Should any of the boards shut down due to over temperature wait 5 minutes and then fol low the procedure starting with Step 2 gt The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 80 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable The Activity indication does not come on Check that the Subnet Manager has been started 9 3 Management Module 9 3 1 Yellow LED for the Management Module is Lit 1 Remove and reinstall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 6 5 on page 69 Make sure that the spine and the leafs both have the same version of FW Reburn the FW and remove and reinstall the management module Replace the management module 9 3 2 Yellow LED for the Leaf Fan is Lit 1 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over
35. e Module 57 Figure 41 Top and Bottom Ports 58 Figure 42 Spine Side Panel Display Status Indications 60 Figure 43 Management Module Status Indications for Normal Operation 60 Figure 44 Power Cord Top Bar 62 Figure 45 PSU Locations 62 Figure 46 Power Supply 63 Figure 47 Leaf Board Numbering 64 Figure 48 Ejector Latch 64 Figure 49 Leaf Release 65 Figure 50 Ejector Latch 66 Figure 51 Spine Board Extraction 66 Figure 52 Fan Locations on the Chassis 67 Figure 53 Leaf Fan Module Extraction 68 Figure 54 Spine Fan Module 69 Figure 55 Fan Status LED on the Spine Module 69 Figure 56 Ejector Latch 70 Figure 57 Management Module 70 Figure 58 lock down bar 73 Figure 59 Face Plate Mounting Bolt Locations 74 Figure 60 Mechanical Lifting Device 75 Figure 61 QSFP Connector Male and Female Views 90 6 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table of Lists Table 1 Revision History of this User s Manual 8 Table 2 Reference Documents and Web Sites 9 Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 13 Table 4 Leaf Board Status Indications 23 Table 5 Leaf Status LED 23 Table 6 Bad Port LED Configurations 23 Table 7 Connector Physical and Logical Link Indications 24 Table 8 Spine Status LED 25 Table 9 Spine to Leaf IB Link Status 25 Table 10 Spine Fan Status LED 26 Table 11 Bad Port LED Configurations 26 Table 12 LEDs Display for Normal Op
36. e Side Panel Display LED Indicators 26 3 1 5 Management Module LED Indicators 28 3 1 6 Port Connector Interfaces 28 3 2 AirFlow 29 3 3 QSFP Cable Power Budget Classification 29 3 4 Management Module Interfaces 29 3 4 1 RC 30 3 4 2 CONSOLE 30 3 4 3 MGT Management 30 3 44 USB 30 3 4 5 Reset RST 30 Chapter 4 Installation 32 4 1 Environmental and Safety Recommendations 32 4 2 Chassis Package Contents 33 4 3 Leaf Package Contents 34 4 4 Spine Package Contents 34 4 5 Management Package Contents 34 4 6 Physical Installation 34 4 6 1 Starting with the Rack 35 4 6 2 ESD Connection 36 4 6 3 Installation Procedure 36 4 6 4 Installing the Cable Holder 54 4 6 5 Ground Connections 55 4 7 PowerComnections 55 4 7 1 Powering Up the Switch Platform 55 4 8 InfiniBand QSFP Cable Installation 57 4 8 1 Cable Power Classes 58 Chapter 5 Chassis Power Up 59 Mellanox Technologies 3 Rev 1 0 5 1 Power Supply and Spine Board Indicator Status at Power ON 60 Chapter 6 Insertion and or Hotswap Extraction 61 6 1 Power Supply Units 61 6 1 1 Extracting and Inserting the Power Supply Unit 61 6 2 Leaf Boards 63 6 2 1 Extracting a Leaf Board 64 6 2 2 Inserting a Leaf Board 65 6 3 Spine Boards 65 6 3 1 Extracting a Spine Board 65 6 3 2 Inserting a Spine Board 66 6 4 Fan Modules 67 6 4 1 Leaf Fan Module 67 6 4 2 Spine Fan Modules 68 6 5 Management Module 69 6 5 1 Extracting a Management Module 70 6 5 2 Inserting a Mana
37. e and four leaf fan modules are found on the spine side The leaf side fans are not interchangeable with the spine side fans Air flow through the leafs is independent of the air flow through the spines Mellanox Technologies 67 Rev 1 0 Insertion and or Hotswap Extraction 6 4 1 1 Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 53 2 Slowly pull out the fan module using the handle Figure 53 Leaf Fan Module Extraction Handle Latch 6 4 1 2 Inserting the Leaf Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module A Y N Tf the fan module stops before it goes in all of the way it is inserted incorrectly 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 6 4 2 Spine Fan Modules Each spine fan module contains two individual fans Should a single fan fail the Fan Status LED on the spine and on the management module will light indicating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs Each spine module has a fan module with two individual fans built in When a fan module is not functioning the status LED on the fan will light up 68 Mellanox Technolog
38. e couvercle transparent n est pas solidement fix ou s il est enlev gt 5 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les cables gt 6 Branchement d branchement des cables InfiniBand en cuivre Les cables InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fab ricant des cables pour connaitre les mises en garde et les instructions sp ciales gt 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cautions sp ciales pour s assurer que le syst me reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut Mellanox Technologies 93 Rev 1 0 8 Fuite gt 3 5mA Leakage gt 3 5mA ATTENTION La connexion la terre des forts courants de fuite est essentielle avant le branchement de l alimentation LA Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent 9 Forts Courants de Fuite High Leakage Current Attention Forts courants de fuite Il est essentiel de relier a la terre avant de brancher l alimentation 10 Ajouter une information de connexion la mas
39. e lock down bar onto the shelf at the connector side See Figure 23 to identify the spine side of the shelf Figure 23 Place the Lock Down Bar on the Shelf eee ee the lock down bar here The spine side of the shelf has two slots for the exten sion 11 Place 14 caged nuts into the vertical rack support at the locations corresponding to the holes in the chassis faceplate The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support There are 7 holes on each side 44 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 24 U Numbering Above Shelf The first cage nut on the spine side goes here The next six caged nuts go every 4 U units up from here 4 6 3 4 Inserting the Chassis 1 Putthe container on the fork lift Mellanox Technologies 45 Rev 1 0 Installation Figure 25 Putting the Chassis in the Rack The chassis is screwed down to the pallet on both the spine side and the leaf side Figure 26 Location for the Lock down Bar Bottom of the chassis 46 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Unscrew the lock down bars 3 Screw on the two Upper Bracket offsets onto the chassi
40. e recommend that all of the chassis sub systems be maintained and managed through the management software 3 1 1 Power Supply Unit LEDs Each Power Supply Unit has the following indicator LEDs Figure 5 Power Supply Unit Status Indications De AC OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC A The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis As Mellanox Technologies 21 Rev 1 0 Hardware Figure 6 PSU Cover On and Off l D lh e E E E O 285mm PSUs covered marani ai ob Cover off al cf i ys Eee ae D D GD D Y ED D D D o O 2 T 3 0 E ESD connection Ground connection 3 1 2 Leaf Board LED Indicators Figure 7 Leaf Board Led Indicators Each leaf board has a Status LED on the far left of the leaf This LED has the following meanings according to the LED status 22 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table 4 Leaf Board Status Indications LED Description Off No power to the leaf Solid Green Normal operation Flashing Green Power up in prog
41. eceiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vcc Tx 3 3 V Power supply transmitter 30 Vcc 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground Mellanox Technologies 89 Rev 1 0 1ews QSFP Connector Male and Female Vi Figure 61 18 35 Top View into Rear of Connector GNB AX n 1Xtp GND TX3n TX3p GND LPMode Vcc1 VecTx IntL ModPrsL GND RX4p RX4n GND RX2p RX2n GND N i GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL VecRx SCL 18 35 View into Front of Cage N Co gt O1 OO NS 00 io aNd me diXH aNd UEXH dexu aNd vas 198 Xy99A Osoy TISSPON IND dex upXL IND dzXL uzXL aNd Mellanox Technologies 90 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Appendix E Replacement Parts Ordering Numbers Table 17 Replacement Parts Ordering Numbers
42. ected to the network the Subnet Management software automatically discovers and configures the fabric and begins utilizing the switch The Mellanox Operating System MLNX OS software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is avail able on the Mellanox web site Basic installation is covered in Chapter 4 Installation on page 32 Hot swapping components and hardware maintenance is covered in Chapter 6 Insertion and or Hotswap Extraction on page 61 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID Identifier and product version information are found on the label that is attached to the pull out tab that is below the Mellanox logo on the spine side of the chassis Mellanox Technologies 11 Rev 1 0 Overview Figure 1 Generic Product Label S N MTO806X01504 D IN il i man oocsozcozrosac NUE Rev A1 Made in IL This is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management SW look for this GUID in addition to their own Node GUID Figure 2 Location of the Product Label Tab SX6536 Here is the tab with the product label Mellanox TECHNOLOGIES 1 1 2 Management Module MAC Each management module has a label with its MAC address See below for the location of thi
43. er supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at LA 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A LA Warning High leakage current Earth connection essential before connecting supply 15 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation LA GND Connection Information The building installation shall provide a means for connection to protective earth and the equipment shall to be connected to that means permanently by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 16 Installation
44. eration 27 Table 13 LEDs Display for Normal Operation 28 Table 14 SX6536 Switch Specification Sheet 84 Table 15 Power Consumption of Chassis Parts 88 Table 16 InfiniBand QSFP Connector Pinout 89 Table 17 Replacement Parts Ordering Numbers 91 Mellanox Technologies 7 Rev 1 0 Revision History Table 1 Revision History of this User s Manual Revision Date Details 1 0 July 2011 Initial Release 8 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 About this Manual This manual provides an overview of the SX6536 modular InfiniBand chassis switch and guide lines for its operation Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniB and architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Web Sites Mellanox OFED Stack for Linux User s Manual See http www mellanox com gt Support gt InfiniBand Software and Drivers Click Mellanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED Select the Linux User s Manual The embedded OS and tools on the CPU in the management module is a subset of the Mellanox OFED stack Switch Hardware Installation Guide Each Mellan
45. ers SM via the CLI WebUI Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based Mellanox Technologies 79 Rev 1 0 TroubleShooting 9 9 1 9 2 TroubleShooting As each PSU is plugged in make sure that the green power LEDs on the PSU come on Power Supply Unit 99 Vf the Power supplies cannot supply enough power the management module may shut down the system gt If the AC power LED is off 1 Check that the power cable is the correct power cable for your country 2 Check that the power cable is plugged into a working outlet 3 Check that the power cable has a voltage within the range of 100 240 volts AC 4 Remove and reinstall the power cable 5 Check the circuit breakers to be sure that the breaker has not tripped 6 Check that the power cable is good Replace the power cable 7 Ifthe AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU Leaf Board The power LED for the Leaf board is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall in the Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling th
46. es Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales gt 100 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 14 Cable de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi LA nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 15 Fuga gt 3 5 mA ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 16 A adir conexi n a tierra Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de LA protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber cone
47. f the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling d Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis gt 4 6 1 Starting with the Rack 4 6 1 1 Rack Recommendation Due to the space required by up to 648 connector cables Mellanox recommends a rack Y that is 120 cm long and 80cm wide This will allow for proper cable management and enough ventilation to properly cool the chassis 4 6 1 2 Preparing the Rack The rack may need to be modified to accommodate the SX6536 chassis The distance between the vertical supports must be between 63 and 80 cm Figure 15 Distance Between the Vertical Supports This edge of the shelf must be as close as possible to the inside of the door when closed Mellanox Technologies 35 Rev 1 0 Installation The vertical supports may have to be moved so that the vertical support closest to the door is 6 cm from the inside of the door This may be required so that the shelf does not protrude from the rack gt If you move the vertical supports make sure that you reinstall them vertically plumb gt 4 6 2 ESD Connection Before starting any procedure on the SX6536 series Switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to
48. gement Module 71 6 6 Switch Shut Down Procedure 71 Chapter 7 Disassembly and Disposal 72 7 1 Disassembling the Chassis T2 7 1 1 Removing the Chassis 74 7 1 2 Removing the Bottom Shelf 75 7 2 Disposal 76 Chapter 8 Switch Management Tools 77 8 1 Fabric Diagnostics with UFM Diag 78 8 2 Accessing the CPU via the Ethernet Connector 78 8 3 Upgrading Software 78 8 4 InfiniBand Subnet Manager 79 Chapter 9 TroubleShooting 80 9 1 Power Supply Unit 80 9 2 LeafBoard 80 9 3 Management Module 81 9 3 1 Yellow LED for the Management Module is Lit 81 9 3 2 Yellow LED for the Leaf Fan is Lit 81 9 3 3 Yellow LED for the Spine Fan is Lit 81 9 4 Spine Board 82 9 5 MLNX OSSW 82 Appendix A Specification Data 84 A 1 Specifications 84 A 2 EMC Certifications 86 Appendix B Calculating the Weight of a Customized Chassis 87 Appendix C Calculating the Power of a Customized Chassis 88 Appendix D QSFP Interface 89 Appendix E Replacement Parts Ordering Numbers 91 Appendix F Avertissements de s curit d installation French 93 Appendix G Installation Sicherheitshinweise German 96 Appendix H Advertencias de seguridad para la instalaci n Spanish 99 4 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table of Figures Figure 1 Generic Product Label 12 Figure 2 Location of the Product Label Tab 12 Figure 3 Management Module MAC Address Location 12 Figure 4 SX6536 Switch Views 14 Figure 5 Power Sup
49. hassis Catch the hooks onto the vertical bar and push the latch shut 6 3 Spine Boards When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management i module will start a new cycle checking through the leafs and spines for the FW ver ae sions Each spine has a pair of ejectors that lock the board in place and serve as levers for seating or extracting see Figure 50 6 3 1 Extracting a Spine Board 1 Push outward on the latches to unlock the ejectors from the chassis 2 Open the ejectors until they are 90 degrees from the module 3 Pull out on the module using both ejectors Mellanox Technologies 65 Rev 1 0 Insertion and or Hotswap Extraction Figure 50 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Push the latch pompe pe release 7 P Figure 51 Spine Board Extraction Pull the latches to remove the spine LELET EEHEHE IENEN 7 550000006 MANERA 9700000000 4 ODO 000000 SN DODOD0ODO 6 3 2 Inserting a Spine Board 1 Start with the latches fully open that is perpendicular to the front panel of the leaf Hold the spine board by the sides with both latches fully open Insert the board into its rail The board should slide in without resistance Push the board in until the hooks
50. hassis management features for software and IB management software OpenSM The IB Subnet Manager running on the SX6536 switches supports up to 648 nodes If the fabric includes more than 648 nodes you may need to purchase Mellanox s Unified Fabric Manager UFM software package The managed switch system includes the following software components Embedded OS e Chassis manager and system BIST SNMP agent 3rd party tool integration e GUI e Subnet Manager SM e Remote logging e SSH telnet e Secured access in band and out band IPv4 IPv6 network stack The chassis manager will give the user access to e Switch temperatures e Power supply voltages e Fan unit information e Power unit information e Flash memory e Monitoring of AC power to the PSUs DC power out from the PSUs chassis failures Mellanox Technologies 77 Rev 1 0 Switch Management Tools 8 1 8 2 8 3 e querying for switch serial numbers revisions software version SwitchX FW version switch temperatures The manager also has the ability to burn new firmware and upgrade software on the switch Fabric Diagnostics with UFM Diag UFM Diag is a plug amp play software for fee within MLNX OS displaying and filtering all identi fied systems and nodes within the fabric UFM Diag includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabri
51. he holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 6 3 on page 65 Make sure that the spine and the Leafs both have the same version of FW 5 Reburn the FW and remove and reinstall the spine Replace the spine board 9 5 MLNX OS SW The last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop 2 Push the reset button on the switch or management module 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 mlnx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No I2C ready DRAM 2 GB ECC enabled 400 MHz CL3 82 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 5 Select the image to boot Mellanox Technologies 83 Rev 1 0 Appendix A Specification Data A 1 Specifications Table 14 SX6536 Switch Specification Sheet Physical Power and Environ
52. icates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the Spine LEDs and make sure they coincide with Table 42 Figure 42 Spine Side Panel Display Status Indications AAA AA O STATUS PSU SPINE LEAF MGT 1 MGT 2 STATUS FANS FANS STATUS STATUS 5 Check the Spine LEDs and make sure they coincide with Table 43 Figure 43 Management Module Status Indications for Normal Operation iT o Mn NN 60 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 6 Insertion and or Hotswap Extraction Before starting any procedure on the SX6536 series Switch system put an ESD prevention wrist strap on your wrist and connect to the SX6536 series Chassis When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sions This switch platform supports Hotswap capabilities for the e Power supply units e Leaf boards e Spine boards e Leaf fan module e Spine fan module e Management modules 6 1 Power Supply Units When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the
53. icator LED has the following led assignment Table 8 Spine Status LED LED Description Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Orange Non fatal error Red Fatal error The leaf board status on each spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per each leaf per each spine Each spine has leaf status LEDs The leaf LEDs on each spine indicate as follows The Spine to leaf board Status indicator LED has the following led assignment Table 9 Spine to Leaf IB Link Status LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Orange Physical connection Mellanox Technologies 25 Rev 1 0 Hardware 3 1 3 2 Fan LED amp The Spine Fan indicator LED amp has the following led assignment Table 10 Spine Fan Status LED LED Description Green Spine Fan is OK yellow One or more of the fans in this spine is not working Each spine has two fans in the fan module 3 1 3 3 Bad Port LED 1 The Bad Port indicator is located on the left side of the spine The following Bad Port conditions are possible Table 11 Bad Port LED Configurations LED Configuration Description Off OK All ports are up and running Flashing Orange Error One or possibly more ports has just received a symb
54. icular route cables such that they do not impede the air into or out of the chassis 4 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order Spine Package Contents The spines are ordered by the customer and are shipped 3 to a box The customer will receive as many boxes as needed to fill the order Management Package Contents The package includes e all of the management modules ordered by the customer e 1 RJ45 to DB9 harness for each management module received Physical Installation chassis weighs 740 lbs 335 kg Make sure that proper manpower and equipment Warning This equipment is very heavy Safety is the first concern The fully loaded A is used for transporting and moving the chassis The switch platform uses 31U of rack space in a standard 19 rack 29U for the chassis and 2U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chassis and there are two lock 34 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 down bars to secure the chassis to the shelf The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports o
55. ies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 54 Spine Fan Module a OOOOOOOCE OOO OO0000 000000000 HANNA QO0CO000C MN Tr Fan Module When a fan module is removed the indicator light will reset Figure 55 Fan Status LED on the Spine Module ODODO TE OOO 00000 10 12 14 16 18 6 4 2 1 Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out 6 4 2 2 Inserting the Spine Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module Slowly slide in the new spine fan module 3 Push the fan module until as far as it will go make sure the locking latches engage 6 5 Management Module When hotswapping any of the units it is necessary to wait 1 minute after removing the N defective part before inserting the new part This is necessary so that the management N module will start anew cycle checking through the leafs and spines for the FW ver SS sions Before starting any procedure on the SX6536 series Switch system put an ESD prevention wrist strap on your wrist and connect to the SX6536 series Chassis Mellanox Technologies 69 Rev 1 0 Insertion and or Hotswap Extraction 6 5 1 Extracting a Management Module Management modules are located on the leaf side above the leafs There are two
56. level of the shelf Move the chassis away from the rack and onto the container base Put wedges around the chassis to prevent the chassis from rolling off of the container base Move the chassis away from the rack wa NR Dispose of the Chassis in a legal and environmental way 7 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack 2 Remove the shelf 3 Remove all of the caged nuts Mellanox Technologies 75 Rev 1 0 Disassembly and Disposal 7 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 76 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 8 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via MLNX OS d There are 2 Ethernet ports 1 for each management module that each get connected to Ethernet switches These switches must be configured to 10 100M auto negotiation gt The SX6536 switch comes standard with a management software module for chassis management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX based managed switch systems MLNX OS includes a CLI WebUI SNMP and c
57. ligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 20 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera responsable y respetuosa con el medio ambiente 102 Mellanox Technologies
58. may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced 1 Check all Boards Power supplies Leaf fan modules Spine fan modules for proper insertion and seating before connecting the AC power cords Insert all leafs that you plan to use in the chassis Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all mounting screws Connect the power cords to the PSUs Nu P wD Connect the power cords to grounded electrical outlets Do not power up the chassis with less than all PSUs installed Populate the chassis with the leafs before you power up the chassis A fully loaded SX6536 series switch system can draw 8 kW of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or a 3 phase supply Mellanox Technologies 59 Rev 1 0 Chassis Power Up 5 1 Power Supply and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system During this time the status LEDs may be RED Turn off the system if any LEDs remain red for more than 5 minutes As the power is turned on you should observe the following conditions for normal operation 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that ind
59. mental HxWxD Weight Mounting Max Air Flow Through leafs Max Air Flow Through spines SerDes Speeds 50 629 x 19 x 27 inches 1286 mm x 482 6mm x 685 mm The shelf adds 100mm to the height 128 97kg 284 33lbs empty weight no leafs spines or management modules 327 15kg 721 24 lbs full configuration 157 57 kg 347 38 lbs shipped configu ration 19 Rack mount EIA 310 16 59M3 min 585 9 CFM 9 48 M3 min 334 8 CFM 20 40 and 56 Gb s per port Input Voltage Temperature Humidity Power Consump tion Leaf Power Consump tion Spine Power Consump 100 240 VAC 50 60Hz 5 10A per inlet 10 to 45 Celsius 10 90 non condensing Typical 91W not including QSFP power Max 94W not including QSFP power Typical 108W include 2 Fans Max 115W include 2 Fans 2 0W Power level 2 Connector Types QSFP tion for Active cables QSFP Typical 7792 54W including Center of Gravity QSFP at 2W CoGh 637 mm from bottom of chassis Power Consump tion Typical 7754W including CoGw 222 mm from left side surface QSFP at 1 W CoGd 342 5 mm from port end surface Max 9711 W including QSFP Measured from the bottom left corner as at 2W one faces the switch ports Max Heat Output 33 135 BTUs hr Protocol Support Regulatory Compliance 84 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0
60. n holes next to the InfiniBand ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row gt Figure 41 Top and Bottom Ports _ E 7 E 4 8 1 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX 65XX series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables 58 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 5 Chassis Power Up Before starting any procedure on the SX6536 series chassis system put an ESD prevention wrist strap on your wrist and connect to the SX6536 series chassis The chassis must be started with a full complement of PSUs that is10 Thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS
61. ol error This LED shows symbol errors Possible causes for this are e bad cable e bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 3 1 3 4 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support 3 1 4 Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition 26 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 9 Spine Side Panel Display Status Indications CS A O O STATUS PSU SPINE LEAF MGT 1 MGT 2 STATUS FANS FANS STATUS STATUS Table 12 LEDs Display for Normal Operation nae Normal LED Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal
62. on the latches are touching the vertical bar Now press both latches inward simultaneously You should feel the latches are hooked into the front panel 6 Press both latches applying some force to close them completely The locking latch will engage AA 66 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 6 4 Fan Modules When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sions gt There are eight fan modules on the chassis for the leafs They are located on both sides of the chas sis four on the leaf side corners and four on the spine side corners When a fan module is not func tioning the status LED on the fan will light up There are two different fan modules for the chassis These fans have different model numbers and are not interchangeable The fans on the leaf side of the chassis are MTF005001 whereas the fans on the spine side of the chassis are MTF005002 gt Figure 52 Fan Locations on the Chassis E a 2 el O e AC2 AC3 AC4 AC5 1 1 ACI ut i CON 8 E AC6 AC7 1 C8 1 ha 1 te la el A i i 6 4 1 Leaf Fan Module There are eight Leaf fan modules Four leaf fan modules are found on the leaf sid
63. op of the shelf Mellanox Technologies 41 Rev 1 0 Installation 5 Move the shelf brackets on the shelf Place the shelf bracket so that it measures slightly more than the measured distance between the vertical rack supports Make sure that there is enough play in the bracket so that it can move to the vertical support Do not tighten the hex head bolts gt d You may need to remove the brackets and reinstall them gt Figure 20 Placing the Shelf Bracket Adjusting slots 6 Insert five caged nuts into the rack for each of the four shelf brackets the 2 brackets that are adjustable and the 2 that are fixed Make sure that the caged nuts are all at the same height on the vertical supports Do not use the top hole in the bracket 42 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 21 Caged Nut Locations for the Shelf Brackets 7 Put the Shelf into the rack and twist the shelf into place Make sure the spine side of the shelf is closest to the rack doors See Figure 23 to identify the spine side of the shelf 8 Install and tighten the 20 M6 bolts into the caged nuts Mellanox Technologies 43 Rev 1 0 Installation Figure 22 Shelf Brackets The spine side of the shelf has two slots for the extension vs Adjustable she brackets ME E EEE IN IBIS IN IIS 9 Tighten the 16 hex head bolts on the adjustable brackets 10 Connect th
64. ox Technologies switch platform is shipped with an Installation Guide document that contains instructions for Instal lation in a rack and initialization of the switch platform IB Switch Platforms Embedded Management Tools Describes the Embedded Management Tools EMGT package Document no 2141UM which provides Out of Band management capabilities for InfiniBand switches Available from http docs mellanox com requires a Mellanox Technologies customer account Switch Firmware and Firmware Update Tools See http www mellanox com content pages php pg management_tools amp menu_section 34 Note that the Switch System described in this manual is based on Mellanox Technologies SwitchX switch device Unified Fabric Manager UFM The user manual can be downloaded together with the software package upon license purchase Mellanox Technologies 9 Rev 1 0 Conventions Throughout this manual the name SX6536 and the term switch are used to describe the 648 port QSFP InfiniBand chassis unless explicitly indicated otherwise The following pictures are used throughout this document to indicate information that is important to the user Y This symbol makes recommendations to the user gt This symbol indicates information that is helpful to the user be This symbol indicates a situation that can potentially cause damage to hardware or soft ware BEWARE This symbol indicates a sit
65. places to install the management modules Note Only one management module is required to run the switch system Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 57 1 Disconnect all cables connected to the management module 2 Push outward on the latches to unlock the ejectors from the chassis 3 Open the ejectors until they are 90 degrees from the module 4 Pull out on the module using both ejectors Figure 56 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Bm Push the latch OS 1 release D Note The board is short therefore do not let go of it while sliding it out Figure 57 Management Module TT e EEE u war LITT Ejectors 70 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 6 5 2 Inserting a Management Module Start with the latch fully open that is perpendicular to the front panel of the leaf 2 Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated On switch systems with dual management systems first connect the cable and configure the master management module CPU and only then config
66. ply Unit Status Indications 21 Figure 6 PSU Cover On and Off 22 Figure 7 Leaf Board Led Indicators 22 Figure 8 Spine Status LEDs 25 Figure 9 Spine Side Panel Display Status Indications 27 Figure 10 Management Module Status Indications 28 Figure 11 Port Numbering 29 Figure 12 Top and Bottom Ports 29 Figure 13 Management Module Interfaces 29 Figure 14 Reset Button 31 Figure 15 Distance Between the Vertical Supports 35 Figure 16 Installation Kit Parts 38 Figure 17 Container Screws 39 Figure 18 Vertical Rack Supports Dimension 41 Figure 19 Adjustable Shelf Brackets 41 Figure 20 Placing the Shelf Bracket 42 Figure 21 Caged Nut Locations for the Shelf Brackets 43 Figure 22 Shelf Brackets 44 Figure 23 Place the Lock Down Bar on the Shelf 44 Figure 24 U Numbering Above Shelf 45 Figure 25 Putting the Chassis in the Rack 46 Figure 26 Location for the Lock down Bar 46 Figure 27 Place the Upper Bracket Offset 47 Figure 28 Attach the Shelf Extension 47 Figure 29 Place Caged Nuts 48 Figure 30 Fork Lift 49 Figure 31 Face Plate Mounting Bolt Locations 50 Figure 32 Upper Brackets and Offset Location 51 Figure 33 Upper Bracket Installation 51 Figure 34 Offset to Upper bracket Bolts 52 Mellanox Technologies 5 Rev 1 0 Figure 35 Lock Down Bar 53 Figure 36 Bolt and Washer Order 53 Figure 37 Cable Holder 54 Figure 38 Ground Connection 55 Figure 39 Multiple Power Inlets Electric Caution Notification 56 Figure 40 Spin
67. ress Yellow Non fatal error This color LED needs trou bleshooting but does not require chassis shutdown Red Fatal error 3 1 2 1 Status LED The Leaf Status indicator LED A has the following led assignment Table 5 Leaf Status LED LED Description Off No power to the Leaf Solid Green Leaf is up and running Flashing Green Leaf is powering up Orange Non fatal error Red Fatal error 31 2 2 Bad Port LED The Bad Port indicator is located on the left side of the spine The following Bad Port conditions are possible Table 6 Bad Port LED Configurations LED Configuration Description Off OK All ports are up and running Flashing Orange Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are e bad cable e bad connection bad connector Mellanox Technologies 23 Rev 1 0 Hardware This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 3 1 2 3 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support 3 1 2 4 Leaf Board Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port V The fol lowing table shows the port status accor
68. rical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the con nections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 38 Ground Connection Grounding post 4 7 Power Connections The switch includes integrated hot swap power supplies which support up to 10 load sharing 1000W supply units The slots for the power supply units PSUs are on the spine side The left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has a dedi cated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be standard 3 wire AC power cords including a safety ground Before starting any procedure on the SX6536 series switch system put an ESD prevention wrist strap on your wrist and connect to the SX6536 series Chassis 4 7 1 Powering Up
69. rrection FEC as described in IEEE 802 3ap chapter 74 gt d FDR and FDR10 are only guaranteed to work with approved Mellanox Cables A FDR10 is only guaranteed to work with approved Mellanox ConnectX 3 adapters 16 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 3 Hardware Installation Safety Warnings 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product LA lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Installation in Restricted Access Location YN This unit is intended for installation in Restricted Access Location 4 Over temperature ing the maximum recommended 45 C 113 F Moreover to guarantee proper air This equipment should not be operated in an area with an ambient temperature exceed A flow allow at least 8cm 3 inches of clearance around the ventilation openings 5 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 6 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redundant power or a blank in its place In case of a blank A fastened
70. s The leaf boards are numbered from top to bottom with corresponding numbers displayed to the outside of the leafs vertically along the side panel Mellanox Technologies 63 Rev 1 0 Insertion and or Hotswap Extraction Figure 47 Leaf Board Numbering L1 1 SS zes L2 L3 EEE a a E L4 EA A EEE EEE A L6 ee 4 L34 PA PO OCTO al E L36 6 2 1 Extracting a Leaf Board Each leaf board has an ejector that locks the board in place and serves as a lever for seating or extracting see Figure 57 1 Disconnect all cables connected to the leaf 2 Push the latch to unlock the ejector from the chassis 3 Open the ejector until it is 90 degrees from the leaf 4 Pull out on the leaf Figure 48 Ejector Latch Pull the leaf out of the chassis Leaf 0 Leaf pa Push the latch 3 release si E Ejector Note The board is short therefore do not let go of it while sliding it out 64 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 49 Leaf Release Push here to release the latch Pull the ejector to remove the leaf en AAAs zz 6 2 2 Inserting a Leaf Board Start with the latch fully open that is perpendicular to the front panel of the leaf Carefully set the leaf board into the side guide rails and slowly slide the board into the c
71. s label Figure 3 Management Module MAC Address Location i MAC address location 12 Mellanox Technologies Product Physical Specifications and Power Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Rack of leafs Height Rack Width RE Weight doi sue A oe eee Required P of Spines P P 31U 19 2557 36 18 327 15kg 721 24 lbs FDR14 7792 54W including Max 45 C 1385 mm EIA 310 685 mm full configuration QSFP at 2W Min 10 C 482 6 mm 128 97kg 284 331bs empty configuration Up to 10 Power Supplies of 100 240VAC 50 60Hz 6 3 13A Max heat dissipation 33 135 BTUs hr The switch itself is 29U tall but requires an extra 2U for installing the shelf The switch ships in a minimum base configuration plus additional modules depending on the cho sen customer configuration Optional modules include Leaf boards e Management modules e Spine boards Figure 4 shows the connector and spine sides fully populated Mellanox Technologies 13 Rev 1 0 Overview Figure 4 SX6536 Switch Views Leaf Fan Module with four internal fans RU pl Ml Two Management Modules Leafs Each leaf has 18 Ports and up to 36 leafs can fit in the chassis AC9 AC10 1 if of H D H D D pto 10 Power Supply Units 8 Required and
72. s industry standard QSFP InfiniBand cables which are available from Mellanox Technologies The Mellanox propriety QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports The standard QSFP cables support full 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connec tions are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase FDR BandWidth is supported only by MLNX QSFP cables FDR and FDR10 are only guaranteed to work with approved Mellanox Cables Kl RAI IW All cables can be inserted or removed with the unit powered on To insert a cable press the con nector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 seconds for link to get up To remove dis engage the lock and slowly pull the connector away from the port receptacle For a valid physical connection both ends of the cable must be connected to working devices Mellanox Technologies 57 Rev 1 0 Installation Take care to not impede the air flow through the ventilatio
73. s using six flat head screws for each offset Figure 27 Place the Upper Bracket Offset 4 Screw on the shelf extension using two bolts with flat washers Figure 28 Attach the Shelf Extension 5 If you are installing the cable management system place 14 caged nuts on the other side of the rack in the vertical supports Mellanox Technologies 47 Rev 1 0 Installation 6 7 8 9 Figure 29 Place Caged Nuts lt 25 A mM Place 8 caged nuts between here and here for the cable holders Place the chassis in front of the rack and as close as possible to the rack Raise the chassis so that it is 5cm above the shelf in the rack Raise the hinged extension on the container floor Lower the chassis until the hinged extension is sitting on the shelf extension 10 Push the chassis from the container to the shelf 48 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 30 Fork Lift Container floo Hinged extension Save the container pallet until you want to remove this chassis It will be necessary for the removal and disposal procedures Mellanox Technologies 49 Rev 1 0 Installation Figure 31 Face Plate Mounting Bolt Loc
74. se Connect a Valid Ground to this Device terre du terminal de l appareil doivent tre appliqu es l installation de protection la Avant de brancher l appareil la conduite d alimentation les vis de protection la A Terre du b timent 11 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 12 Elimination du mat riel T limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 13 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 14 Codes d installation pays Pour l Am rique du Nord l quipement doit tre install conform ment aux L appareil doit tre install selon l ancienne version des codes lectriques nationaux du A sp cifications du Code Electrique National Am ricain et du Code Electrique Canadien 94 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 15 Interconnexion des unites Les c bles de branchement l unit RS232 et les interfaces Ethernet doivent tre certi fies UL de type DP 1 ou DP 2 Note lorsqu il existe dans un circuit non LPS Protection contre la surintensit Un appareil de protection r pertori facilement accessible contre la surintensit du circuit de branchement et calibr
75. sides on 38 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 17 Container Screws Se CJ amp CAN E amp AN NA Ye Container screws Remove and put aside the box containing the cables Remove and put aside the box containing the cable management kit Remove and put aside the box containing the Installation kit Ne Visually inspect the chassis make sure that there is no visible damage 10 PSUs are installed all 8 fans are installed 7 Remove all protective plastic film from all sides and top of the chassis gt The leafs spines and management modules are shipped separately Ad LA Do not remove any of the blocks or either of the lock down bars at this time Mellanox Technologies 39 Rev 1 0 Installation 4 6 3 3 Installing the Shelf of the pallet Should the chassis start to fall move out of the way The chassis can The chassis is on ball bearings and can roll easily Be aware that the chassis can roll off A cause grave bodily harm should it fall on or near you 1 Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low It is recommended to set the shelf at US Try and keep the chassis as low as possible The maximum height that the chassis can be mounted will be determined by the maximum height that the fork lift can raise the p
76. the Switch Platform Note Make sure that the power cords are compatible with your outlets Power cords for differ ent countries can be ordered from Mellanox 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cord into a grounded outlets Mellanox Technologies 55 Rev 1 0 Installation Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit The chassis must be started with a full compliment of PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs Figure 39 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTE
77. the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and disposed of according to the directive 8 Using a socket wrench remove the hex head screws holding down the lock down bar on the spine side of the chassis 72 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 58 lock down bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf 9 Remove the lock down bar 10 Dispose of these pieces in a legal and environmentally friendly way Mellanox Technologies 73 Rev 1 0 Disassembly and Disposal Figure 59 Face Plate Mounting Bolt Locations SEE Holes in the faceplate for mount TECHNOLOGIES mg on the rack The holes are separated by 4U units a 1 p 7 1 1 Removing the Chassis 1 Remove the screws connecting the upper brackets to the rack 2 Remove the screws holding the chassis to the rack These screws are located in the faceplate on the spine side of the chassis 3 Remove the upper brackets from the chassis 74 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 60 Mechanical Lifting Device Put the container base saved from the installation on the fork lift Raise the container base to 1 cm below the
78. uation that can potentially cause personal injury or damage to hardware or software 10 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 Overview This User Manual provides an overview of the SX QSFP Modular InfiniBand Switch Platform known in this document as the switch and its operational environment Mellanox SX6536 switch system provides the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Performance Computing HPC and Embedded environments Networks built with the SX6536 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mel lanox s 5 generation 1 generation SwitchX VPI switch device SX6536 systems provide up to 56Gb s full bidirectional bandwidth per port With up to 648 ports in a 31U rack space these sys tems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard operation within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system An initial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configuration Once conn
79. ure the slave By default the master CPU is the top management module For further information on the master and slave roles see the MLNX OS Software UM section High Availability for more information Initial configuration must be done on all of the management modules See the Installa tion Guide for the Initial Configuration procedure 99 All management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure y 6 6 Switch Shut Down Procedure To shut down the switch run the command Reload halt noconfirm The switch cannot be restarted remotely To restart the switch you must physically go to the switch and unplug and replug in all of the power cords to the chassis Mellanox Technologies 71 Rev 1 0 Disassembly and Disposal 7 Disassembly and Disposal 7 1 Disassembling the Chassis Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Remove all leafs Remove all spines AN OS ne Remove all management modules Use enough people to safely lift this product An lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs This product and all of its parts are NOT to be disposed of with household waste This gt d product contains printed circuit boards cables and batteries According to
80. vac a no utilizar el producto si su tapa est quitada o no est bien cerrada 7 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de activ idad de rayos Mellanox Technologies 99 Rev 1 0 8 Conexi n y desconexi n del cable Copper InfiniBand Dado que los cables de cobre InfiniBand son pesados y no son flexibles su con exi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 9 Montaje y mantenimiento de bastidores Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 10 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente 11 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen dio 12 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 13 C digos el ctricos locales y nacional

648 Port InfiniBand FDR Switch Platform User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents