Home

HW User Manual

1. 8 9 InfiniBand
2. 16 UL u CSA UL CSA 3 16 AWG 1 5 M 6 4 5 125 13 A lt HAR gt 3 1 0 MM 300
3. 20 2002 96 WEEE G 9 Avertismente privind siguran a la instalare Romanian 1 Instruc iuni de instalare N Cititi toate instructiunile de instalare inainte de a conecta 116 Mellanox Technologies Rev 2 0 2 Accidentare cauzat de greutate Apelati la un num r suficient de persoane pentru a ridica in sigurant acest produs e lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Echipament greu Acest echipament este greu i trebuie s fie mutat folosind un dispozitiv mecanic de ridicare pentru a evita producerea de leziuni 4 Risc de soc electric Risc de oc electric Odat ce modulul ventilator este ndep rtat pinii ele
4. 114 Mellanox Technologies Rev 2 0 11 12 H KBATHHIHpPOBAHHEIH 15
5. 250 10 Mellanox Technologies 115 Rev 1 3 17 TOK TOK K 4 NN 18 AN RS232 Ethernet DP 1 y DP 2 UL 4 N B 20 19
6. Eos Eh 45 C 113 F HA A T URRESRURLE AE 8 3 RM WEEE 6 NETS A 100 Mellanox Technologies Rev 2 0 7 EXEREERW ER fo E AAA A LIBE AR N DIR e Jin DAS RR E ABR 8 BER he SH tek LA PSU 9 NERNERER LA FH ZI NE Be C FAR 10 InfiniBand HAE RK IN LN InfiniBand 584848 RAR AKA DE TER L K BR ZEE BR 12 ARES Mellanox Technologies 101 14
7. Now 2 21 WEEE 29 WEEE 2002 96 EC DINA Town D lt 22 TN IT mon gt Mellanox Technologies 9 9 Rev 1 3 G4 Chinese 1 2 e lt 40 lbs 40 70 lbs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 4 4 A FRA AR n 22 E Pa ERS
8. 30 Figure 11 Inserting the Caged Nuts for the Shelf 30 Figure 12 Connect Rail Slide to Rack Vertical support 31 Figure 13 Inserting the Caged Nuts for the Faceplate 31 Figure 14 Chassis on Shelf with Filler Panel 32 Figure 15 Cable Holders Hr es 33 Figure 16 Inserting the Caged Nuts for the Cable Holder 34 Figure 17 Cable Holder Placement 35 Figure 18 Chassis Rails and Rail Slides 36 Figure 19 Installing the Handles a ann 37 Figure 20 Screw the Handles Onto the Chassis 37 Figure 21 The Rails are Already Connected Onto the Chassis 38 Figure 22 Raising the Chassis Using a Mechanical Lift 39 Figure 23 Put on the Rail Slide 39 Figure 24 SX6506 Chassis on the Shelf 40 Figure 25 Face Plate Mounting Bolt Locations 40 Figure 26 Ground Connection 41 Figure 27 Do Not Use AC Power Socket 1 with N N PSUs 42 Figure 28 Multiple Power Inlets Electric Caution Notification
9. nio 10 4 PN Mellanox Technologies 97 Rev 1 3 11 InfiniBand on gt 12 NINI MNT gt 13 14 DINI 15 gt 16 NASI
10. 91 Mellanox Technologies 8 J InfiniBand FDR Switch Platform Hardware User Manual Revision History Table 1 Revision History of this User s Manual Revision Date Details 2 0 Nov 30 2015 Updated 1600W PSU power from 1600W to 1670W 1 9 Nov 26 2015 Updated Appendix F Replacement Parts Ordering Numbers on page 92 spine and leaf module OPNs 1 8 Nov 18 2015 Updated Table 12 Management LED Display for Normal Operation on page 68 1 June 06 2015 Updated Section 2 3 Chassis Package Contents on page 23 Section G 13 Japan VCCI Class A Statement on page 130 1 6 Feb 24 2015 Updated Table 12 Management LED Display for Normal Operation on page 68 with master LED status indicators Appendix A Specification Data on page 82 with FDR10 power numbers and max heat output Appendix D Calculating the Power of a Chassis on page 87 with FDR10 power numbers 1 5 Nov 27 2014 Updated Section 1 4 2 1670W Power Supply Units on page 16 Section 2 7 1 3 Container Mishandling on page 27 Appendix A Specification Data on page 82 1 4 Oct 27 2014 Updated Air Flow numbers in Appendix A Specification Data on page 82 Nt configuration in Section 1 4 2 1670W Power Supply Units on page 16 1 3 July 17 2014 Updated Section 2 7 4
11. 235 Y N UL CSA nonon 3 16 AWG 13A 125 1 5 6 4 5 235 gt HAR gt 3 DITA 1 0 V 2 300 by PVC 250 10A V 98 Mellanox Technologies Rev 2 0 17 s gt pa 18 MINI Ethernet RS232 UL DP 1 non DP 2 mo 19 333 DYSON NIN 204 20 PN PN
12. 57 Figure 48 Management Module 58 Figure 49 Intact vs Defected Signal Connectors 58 Figure 50 Intact vs Defected Power Pin Holders 59 Figure 51 Power Supply Unit Status Indications 61 Figure 52 PSU Cover and Off 62 Figure 53 Leaf Board Led Indicators 63 Figure 54 Spine Status LEDS is sat on ab reed ete 65 Figure 55 Spine Side Panel Display Status Indications 67 Figure 56 Management Module Status Indications 68 Figure 57 Management Module 69 Figure58 Port Numberng 2 5 Ete hn aber 70 Figure 59 Top and Bottom Ports 70 Figure 60 Management Module Interfaces 71 Figure i Reset sacarlos darias rs a ea ds cra 72 Figure 62 Do not use AC power Socket 1 with N N PSUs 73 Figure 63 Spine Side Panel Display Status Indications 74 Figure 64 Management Module Status Indications for Normal Operation 74 Figure 65 InfiniBand Connector Symbol 90 Figure 66 Connector Male and Female Views
13. 15 UL IZA CSA RAER PEAS UL IAM CSA 29 8 82888 16 AWG 125 15 4 5 BEA ARH AA AERA lt HAR gt FR 182888 1 0 300 V KA PVC 08 ER 250 V 10 16 A CAATHREBE 17 18 Bien A RS232 SEAN ZAMAN RARA UL DP 1 DP 2 A LPS BERRE 20 A 19 BA 102 Mellanox Technologies 20 WEEE 5
14. 80 ee 81 Appendix A Specification Data 82 A 1 EMI Certification 84 A 2 Approved Cables 84 A3 EMC Certifications pirita te bado eic ba piena 84 Appendix B Thermal Threshold Definitions 85 Appendix C Calculating the Weight of a Customized Chassis 86 Appendix D Calculating the Power of a Chassis 87 Appendix Interface 89 Appendix F Replacement Parts Ordering Numbers 92 Appendix G Safety Warnings Multiple Languages 93 G 1 Nordic Countries Notices 93 G 2 Installation Safety Warnings English 93 DAY nn 6 G4 Chinese han dans ous 100 G 5 Avertissements de s curit pour l installation French 103 G 6 Installation Sicherheitshinweise German 106 G 7 Advertencias de seguridad de instalaci n Spanish 110 G 8 Russian 113 9 Avertismente privind siguranta la instalare Romanian 116 G 10 Sigurnosna upozorenja za instaliranje Croatian 120 G 11 Avvertenze d
15. won dw e re 12 1 1 Product Information 2 12 1 1 1 Serial Number and Product Version Information 12 1 1 2 Management Module MAC 13 1 1 3 Product Physical Specifications and Power 13 1 27 Features iS iria aan 15 1 3 InfiniBand FDR and FDR10 Overview 15 1 4 Power Supply Redundancy 15 1 4 1 1000W Power Supply Units 16 1 4 2 1670W Power Supply Units 16 Chapter 2 Installation re 18 2 1 Installation Safety Warnings 18 2 2 Environmental and Safety Recommendations 22 2 3 Chassis Package Contents 23 24 Leaf Package 24 2 5 Spine Package Contents 24 2 6 Management Package Contents 24 2 7 Physical Installation A A A A A mE 24 2 7 1 ESD Connection 25 2 7 2 Installing the Cable Holder 33 2 7 3 Installing the Chassis 36 2 7 4 Ground Connections 41 2 8 Power Connections
16. 41 2 8 1 Powering Up the Switch Platform 42 2 9 InfiniBand Cable Installation 43 2 9 1 Supported Approved Cables 44 2 9 2 Cable Power Classes 44 2 10 Hot Swap Insertion and Extraction 45 2 10 1 Power Supply Units 45 2 10 2 Leaf Boards 47 2 10 3 Spine Boards 51 2 10 4 Fan Modules 55 2 10 5 Management Module 57 2 10 6 Switch Shut Down Procedures 60 Chapter 3 Interfacess u ss a DD en A E AREE 61 3 1 LED Status Indicators 61 3 1 1 Power Supply Unit LEDs 61 3 1 2 Leaf Board LED Indicators 63 3 1 3 Spine Board LED Indicators 64 3 1 4 Spine Side Panel Display LED Indicators 66 3 1 5 Management Module LED Indicators 68 Mellanox Technologies 3 J InfiniBand FDR Switch Platform Hardware User Manual 3 1 6 Port Connector Interfaces 70 32 AIT FOW s o Nee tea AUR ORT a 70
17. 10 Cable de conexi n y desconexi n InfiniBand de cobre Dado que los cables de cobre InfiniBand son pesados y no son flexibles su conexi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver adver tencias o instrucciones especiales consultar al fabricante del cable 11 Montaje y mantenimiento del bastidor Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 12 Instalaci n del equipo N La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica A mente de personal capacitado y competente La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y N reglamentaciones nacionales Mellanox Technologies 111 Rev 1 3 14 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio 4 males P 15 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 16 Cab
18. 4 Elektrik Carpmas1 Riski Bu ekipman nerilen maksimum ortam s cakl n asan alanlarda calistirilmamalidir 45 C 113 F Ayr ca d zg n hava ak s saglamak i in havaland rma deliklerinin etraf nda en az 8 cm 3 in a kl k b rak lmal d r 5 s nma y Bu ekipman nerilen maksimum ortam sicakli ini asan alanlarda calistirilmamalidir A 45 C 113 F Ayrica d zg n hava ak s saglamak i in havalandirma deliklerinin 4 etraf nda en az 8 cm 3 in a kl k b rak lmal d r 6 Sasi Istif k Sasinin diger herhangi bir ekipman n zerine istiflenmemesi gerekir Sasi d serse fiziksel yaralanmalara ve ekipmanda hasara neden olabilir Mellanox Technologies 127 Rev 1 3 7 Yedekli G Kayna y Ba lantysy Elektrik arpma Tehlikesi Bu r n yedekli g kayna i veya onun yerine bos elektrik kutusu icerir G kaynag N i in bos elektrik kutusu varsa kutunun kapag a ikken veya tam olarak kapat lmam ken r n al t rmay n 8 oklu G Giri leri Elektrik arpmas riski ve enerji tehlikesi B t n PSU lar G Kaynag niteleri ayr d r Anahtar platformundaki g c kapatmak i in tim g kaynaklar n n baglant lar n kesin 9 Simsek Elektrik Carpma Tehlikesi G ky z nde simsek gakt g zamanlarda ekipman zerinde alismayin veya kablo baglamay n ya da kablo baglant s kesmeyin 10 Bak r nfin
19. 69 Table 15 Management Module L Fan LED Configurations 69 Table 16 Management Module MASTER LED Configurations 70 Table 17 Switch Specification SX6506 3R NR 82 Table 18 Switch System Weight Calculation 86 Table 19 Power Consumption of Chassis Parts 87 Table 20 Cable Power Consumption 87 Table 21 Calculating Total Switch System Power Consumption 87 Table 22 InfiniBand Connector Pinout 89 Table 23 Replacement Parts Ordering Numbers 92 Mellanox Technologies 6 GN InfiniBand FDR Switch Platform Hardware User Manual List of Figures Figure 1 Product Label 5 E 12 Figure 2 Management Module MAC Address Location 13 Figure 3 SX6506 Switch Views i 14 Figure 4 Power Supplies for SX6506 3R and SX6506 NR 16 Figure 5 Shelf Installation Kit Parts 26 Figure 6 Chassis Installation Parts 27 Figure 7 Shock Stickers e de a U ns 28 Figure 8 TIU Sticker uoces RE ARA 28 Figure 9 Opening the Container 29 Figure 10 Placement of Chassis in Rack
20. Mellanox Technologies 47 J InfiniBand FDR Switch Platform Hardware User Manual Figure 33 Leaf Board Numbering EI X L1 A L2 L3 L4 L5 L6 2 10 2 1 Extracting a Leaf Board Each leaf board has a ejector handle that locks the board in place and serves as a lever for seating or extracting see Figure 34 1 Run the shut down command no power enable module For example to shut down leaf 16 run the command below switch master config no power enable L16 2 Disconnect all cables connected to the leaf 3 Push the ejector handle to unlock the ejector from the chassis 4 Open the ejector until it is 45 degrees from the leaf Figure 34 Ejector Handle Pull the leaf out of the chassis Push the ejec tor handle 3 Ejector handle 5 Pull out the module half way through the guiding rails using the ejector handle 6 Lock the ejector handle 7 Hold the body of the leaf on both sides and remove it from the chassis Ah The board is short therefore do not let go of it while sliding it out Figure 35 Leaf Release Push here to release the ejector handle Pull the ejector handle to remove the leaf RI Dax DAA ES Y VER Vo gr BA SA Vs Mellanox Technologies 48 InfiniBand FDR Switch Platform Hardware User Manual 2 10 2 2 Inserting a Leaf Board When hot swapping any ofthe units it is necessary to wait 1 minute after removing the defectiv
21. 10 20 40 Gb s or 56 Gb s per port Connector QSFP Types Sound Level 67 5 db A of Ports 108 Power and Environmental Input Voltage 100 240 VAC 50 60Hz Typical Passive 1015W Active 1262W includes QSFP at 1 5W Total Power Optical 1295W includes QSFP at 1 7W Consumption FDR 10 Maximum Passive 1185W Active 1464W includes QSFP at 1 5W Optical 1513W includes QSFP at 1 7W FDR 10 Max 5 163 BTUs hr Heat Output ellanox Technologies 82 J InfiniBand FDR Switch Platform Hardware User Manual M Table 17 Switch Specification SX6506 3R NR Physical Total Power Typical Passive 1241W Active 1488W includes QSFP at 1 5W Optical 1521W includes QSFP at 1 7W Consumption FDR Maximum Passive 1314W Active 1593W includes QSFP at 1 7W Optical 1643W includes QSFP at 2 0W FDR Max Heat 5 606 BTUs hr Output Operating 0 to 45 Celsius 2 Non operating 40 to 70 Celsius Humidity Operating 10 85 non condensing Protocol Support Speed Protocol InfiniBand Auto negotiation of 56Gb s 40Gb s 20Gb s 10Gb s MLNX OS and baseboard performance and device management agents for full Infini Management Band in band management Data Rate 56Gb s per port QoS 8 InfiniBand virtual lanes for all ports Regulatory Compliance US Canada cTUVus certificate International CB report and certificate Saft Russia GOS
22. Ground Connections on page 41 Appendix A Specification Data on page 82 Appendix D Calculating the Power of a Chassis on page 87 1 2 January 2014 Rearranged document Consolidated installation sections under Chapter 2 Installation on page 18 updated Chapter 1 Overview on page 12 and re ordered sections Added Figure 57 Management Module LEDs on page 69 Updated Section 1 4 Power Supply Redundancy on page 15 Section 2 7 1 5 Installing the Shelf on page 29 Section 2 7 2 Installing the Cable Holder on page 33 FDR note in Section 2 9 InfiniBand QSFP Cable Installation on page 43 Section 2 10 Hot Swap Insertion and Extraction on page 45 Section 7 1 Disassembling the Chassis on page 80 Table 17 Switch Specification SX6506 3R NR on page 82 Appendix D Calculating the Power of a Chassis on page 87 1 1 May 2013 New graphics Changed PSU input power numbers to match the chassis power numbers both are measured at AC voltage not the DC side Added N N 2 PSUs as per NR not 3 as per 3R Added filler panel 1 0 October 2012 Initial release Mellanox Technologies 9 J InfiniBand FDR Switch Platform Hardware User Manual About this Manual This manual provides an overview of the SwitchX based SX6506 modular InfiniBand chassis switch and guidelines for its operation
23. Con el m dulo del ventilador quitado se obtiene acceso a las clavijas de alimentaci n desde dentro de la cavidad del m dulo NO introducir herramientas ni partes del cuerpo en la cavidad del m dulo del ventila dor 5 Sobretemperatura No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adec uada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 6 Apilamiento del chasis A IN Los chasis no se deben apilar sobre otros equipos La caida del chasis podria causar N lesiones corporales as como da os al equipo 110 Mellanox Technologies Rev 2 0 7 Conexi n redundante de fuente de alimentaci n peligro de descarga Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a A est quitada o no est bien cerrada 8 Tomas de alimentaci n m ltiples Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes Desconecte todas las fuentes de alimentaci n para asegurar que no haya corriente alguna dentro de la plataforma de conmutaci n gt 9 Alhaber rayos peligro de descarga No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de LA rayos
24. 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 2 10 4 2 Spine Fan Modules Each spine module has a fan module with two individual fans built in When a fan mod ule is not functioning the Fan Status LED on the spine will light up Figure 46 Spine Fan Module Fan module When a fan module is removed the indicator light will reset Mellanox Technologies 56 J InfiniBand FDR Switch Platform Hardware User Manual M Figure 47 Fan Status LED on the Spine Module H Fan status LED Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out Inserting the Spine Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new spine fan module 3 Push the fan module as far as it will go make sure the locking latches engage gt Ifthe Fan LED continues to show red remove the fan module and check the pins on the connector inside of the spine to make sure t
25. 43 Figure 29 Spine Module 43 Figure 30 Power Cord 46 Figure ST PSU Locations E rit e qui er ats us ida 46 Figure 32 Power Supply siria ARE Be De A A 47 Figure 33 Leaf Board Numbering 48 Figure 34 Ejector Handle sua ak A EA 48 Figure 39 Lear Releases vi a hte la ceh o ii sa ues 48 Figure 36 Intact vs Defected Signal Connectors 49 Figure 37 Intact vs Defected Power Pin Holders 50 Mellanox Technologies 7 J InfiniBand FDR Switch Platform Hardware User Manual Figure 38 Management Module Numbering 52 Figure 39 Spine Board Extraction 52 Figure 40 Intact vs Defected Mechanics 53 Figure 41 Intact vs Defected Side Signal Connectors 54 Figure 42 Intact vs Defected Middle Signal Connectors 54 Figure 43 Intact vs Defected Power Pin Holders 54 Figure 44 Spine Board Insertion Caution 55 Figure 45 Leaf Fan Module Extraction 56 Figure 46 Spine Fan Module sor a ran 56 Figure 47 Fan Status LED on the Spine Module
26. 5 Slide the chassis further into the rack 6 Slide the chassis all the way into the rack until the faceplate is touching the vertical rack support The caged nuts placed in the last procedure should line up with the holes in the faceplate 7 Screw the 8 screws through the faceplate and into the caged nuts Mellanox Technologies 39 J InfiniBand FDR Switch Platform Hardware User Manual 8 Slide the rail slides to the vertical rails Figure 24 SX6506 Chassis on the Shelf Figure 25 Face Plate Mounting Bolt Locations Holes in the faceplate for mounting on the rac 9 Place the 4 holed spacer on the outside of the vertical support and screw in the 4 screws for each rail slide Use the split lock washers here 10 Remove all four handles and save for future use 11 Ground the switch 12 Install all of the spines leafs and management modules 13 Install any power supply units and fan units removed at the start of the installation procedure 14 Insert all leafs that you plan to use in the chassis Start at the bottom of the chassis and work your way up 15 Insert the first two spines in the top two slots 16 Insert the rest of the spines from the bottom of the chassis up to slot 3 17 Insert thermal blanks in unused leaf slots to maintain balanced air flow 4 Mellanox Technologies 0 J InfiniBand FDR Switch Platform Hardware User Manual 18 Tighten all leaf and spine mounting screws 19 Connect t
27. ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 O Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX CloudX logo Connect IB ConnectX CoolBox CORE Direct GPUDirect InfiniHost InfiniScale Kotura Kotura logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox ScalableHPC Mellanox Connect Accelerate Outperform logo Mellanox Virtual Modular Switch MetroDX MetroX MLNX OS Open Ethernet logo PhyX SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd AccelioTM CyPUTM FPGADirectTM HPC XTM InfiniBridgeTM LinkXTM Mellanox CareTM Mellanox CloudXTM Mellanox Multi Host Mellanox NEO Mellanox PeerDirect Mellanox Socket Direct Mellanox Spectrum NVMeDirectTM StPU Spectrum logo Switch IB Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3885 InfiniBand FDR Switch Platform Hardware User Manual Table of Contents Revision History co a dea a sizes wales 9 About this Manual Aca AAA 10 Chapter 1 Overview
28. Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniBand architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Websites Document Name Description InfiniBand Architecture Specification The InfiniBand Architecture Specification that is provided by Vol 1 Release 1 2 1 IBTA Switch Product Release Notes For possible hardware issues see the switch support product page This requires a customer support login Look up the rele vant SwitchX based switch system series release note file Mellanox MLNX OS User Manual This document contains information regarding configuring and for VPI managing Mellanox Technologies SwitchX switch platforms listing all ofthe commands available through MLNX OS with explanations and examples Conventions Throughout this manual the name SX6506 and the terms chassis and switch are used to describe the 108 port QSFP InfiniBand chassis unless explicitly indicated otherwise The following icons are used throughout this document to indicate information that is important to the user Y This symbol makes recommendations to the user 4 This symbol indicates information that is helpful to the user 5 Mellano
29. along the top of the chassis When a fan module is not functioning the status LED on the fan will light up Each spine has an individual fan module that contains two individual fans Should a sin gle fan fail the Fan Status LED on the spine and the S Fan LED on the management module will light indicating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs 2 10 4 1 Leaf Fan Module There are 4 leaf fan modules found on the spine side Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 45 2 Slowly pull out the fan module using the handle Mellanox Technologies 55 J InfiniBand FDR Switch Platform Hardware User Manual Figure 45 Leaf Fan Module Extraction Handle Latch Release Inserting the Leaf Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module If the fan module stops before it goes in all of the way it is inserted incorrectly or it may be for the other side of the chassis
30. nu conecta i sau deconectati cablurile 10 Conectarea deconectarea cablului din cupru InfiniBand N a Cablurile InfiniBand din cupru sunt grele si inflexibile de aceea trebuie s fie atasate N sau detasate de conectori cu grij Consultati produc torul de cabluri pentru avertis y mente instructiuni speciale 11 Montarea sau depanarea intr un rack y N C nd acest produs este montat sau depanat intr un rack trebuie s fie luate m suri de 4 N precautie speciale pentru a se asigura c sistemul r m ne stabil In general trebuie s y umpleti rack ul cu echipamente incep nd de jos n sus 12 Instalarea echipamentului Acest echipament trebuie s fie instalat inlocuit si sau depanat numai de c tre personal instruit si calificat 13 Eliminarea echipamentului Eliminarea acestui echipament trebuie s se realizeze in conformitate cu toate legile si regulamentele nationale 118 Mellanox Technologies Rev 2 0 15 Codurile ed instalare Acest dispozitiv trebuie s fie instalat in conformitate cu ultima versiune a codurilor electrice nationale ale t rii in cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerintelor aplicabile din Codul electric national al SUA si Codul electric canadian 16 Cordon de alimentare electric nregistrat UL si certificat CSA A Pentru conectarea la o surs de alimentare pentru America de Nord selectati un cordon N
31. o S 2 Grasping the handle with one hand push the black latch release while pulling the handle outward As the PSU unseats the PSU status indicators will turn off Remove the PSU To insert a PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles Insert the PSU by sliding it into the opening until a slight resistance is felt ed e EEE Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation B Insert the power cord into the supply connector on the other side of the chassis 5 Replace the cover over the PSUs 4 The green indicators should light If not extract the PSU and re insert it again 2 10 2 Leaf Boards Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation y When hot swapping any of the units it is necessary to wait 1 minute after removing PA N the defective part before inserting the new part This is necessary so that the manage N ment module will start a new cycle checking through the leafs and spines for the FW versions The leaf boards are numbered from top to bottom with corresponding numbers dis played to the outside of the leafs vertically along the side panel
32. tre install remplac et maintenu que par un personnel form et qualifi mise au rebut de cet quipement doit se faire conform ment toutes les lois et N reglementations nationales 14 Codes lectriques locaux et nationaux A IN Cet quipement doit tre install conform ment aux codes lectriques locaux et nation aux 15 Codes d installation Cet appareil doit tre install conform ment la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 16 Cordon d alimentation UL Listed et certifi CSA Pour le branchement lectrique en Am rique du Nord utiliser un cordon d alimenta tion UL Listed et CSA Certified 3 conducteurs calibre 16 AWG avec une prise moul e 125 V 13 A faisant au moins 1 5 m de long six pieds et au plus 4 5 m Pour le branchement lectrique en Europe utiliser un cordon d alimentation au format international harmonis marqu lt HAR gt 3 conducteurs d au moins 1 mm2 de sec tion 300 V avec une gaine isolante en PVC Le cordon doit avoir une prise moul e 250 V 10 A 17 Courant de fuite lev Avertissement courant de fuite lev une connexion la terre est indispensable avant de brancher l alimentation Mellanox Technologies 105 Rev 1 3 18 Inter
33. 5th generation SwitchX VPI switch device SX6506 systems provide up to 56Gb s full bidirectional bandwidth per port With up to 108 ports in a 6U high form factor these systems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard opera tion within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system The ini tial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configuration Once connected to the network the Sub net Management software automatically discovers and configures the fabric and begins utilizing the switch The Mellanox Operating System MLNX OS software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is available on the Mellanox web site Basic installation is covered in Chapter 2 Installation on page 18 Hot swapping components and hardware maintenance is covered in Chapter 2 10 Hot Swap Insertion and Extraction on page 45 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID identifier and product version information are found on the label attached to
34. A PSU SFANS LFANS Master m os 200 cosy III Ejectors 2 10 5 1 Inserting a Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a management board 1 Check for foreign objects in or mechanical damage to the chassis management slots 2 Check management mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 49 Figure 49 Intact vs Defected Signal Connectors Fail Mellanox Technologies 58 InfiniBand FDR Switch Platform Hardware User Manual 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 50 Figure 50 Intact vs Defected Power Pin Holders 5 Start with the ejector handles fully open that is at 45 degrees to the front panel ofthe management 6 Holding the management by its sides carefully set the management board into the chassis 7 Using only the management ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar Do not apply excessive force to slide in the management board I
35. CORBIS 772 A BER CI COREE XRRR CHATS 5 02 6 9 3 DIE 758 5 416 LUK VCCI A 130 Mellanox Technologies
36. Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Tx 3 3 V Power supply transmitter ellanox Technologies 89 J InfiniBand FDR Switch Platform Hardware User Manual Table 22 InfiniBand QSFP Connector Pinout Connector Cote Pin Signal Description Pin Name Number 30 Vcc 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txln Transmitter Inverted Data Input 38 GND Ground Figure 65 InfiniBand QSFP Connector Symbol J3 and J6 20 GND GND _ 19 21 Rx2n Rx1n 18 22 Rx2p Rx1p 7 23 GND 16 24 15 Rx4n Rx3n 14 25 Rx4p Rx3p 13 28 GND GND i ModPrsL SDA 12 11 28 IntL SCL 10 29 VecIx Vcc Rx su Veci ResetL 9 31 LPMode ModSelL 32 GND GND 7 33 Tx3p Tx4p 34 Tx3n Tx4n 5 35 GND 36 Tx1p Tx2p 3 3 Tx2n 2 38 GND GND 1 Mellanox Technologies 90 Ne InfiniBand FDR Switch Platform Hardware User Manual Figure 66 QSFP Connector Male and Female Views Top View into Rear of 18 35 Connector Le 2 lt 2 9 889 229 7 s Es 8 5 X 9 x 9 G38 o 5 6
37. Sie das Ger t an die Stromversorgung N anschlie en 106 Mellanox Technologies Rev 2 0 2 Verletzungsgefahr wegen des Gewichts da Um das Produkt sicher anzuheben gen gend Personen einsetzen Mal lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Schweres Ger t Dieses Ger t ist schwer und muss mit einem mechanischen Hebeger t verschoben werden um Verletzungen zu vermeiden 4 Stromschlagrisiko Stromschlagrisiko Bei abgenommenem Ventilatormodul sind die Stromkontakte in der Modulvertiefung zug nglich Es d rfen KEINE Werkzeuge oder K rperteile in die Vertiefung des Ventilatormoduls gelangen 5 bertemperatur A Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der A maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem SS sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 6 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunt erf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 7 Mehrere Stromeing nge Risiko eines Stromschlags und Stomgefahr Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgesch
38. WEEE 2002 96 EC ERIN EREI ES Ex EEE ERE 8B ZA 55 TIR TA ARE FE nn 21 TN E EX tA IT SRA Lo G 5 Avertissements de s curit pour l installation French 1 Instructions d installation Veuillez lire la totalit des instructions d installation avant de relier l quipement au secteur 2 Blessures cause du poids A Pr voyez assez de personnel pour soulever ce produit en toute s curit lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 quipement lourd Cet quipement est lourd et doit tre d plac avec un systeme de levage m canique pour viter les blessures Mellanox Technologies 1 0 3 Rev 2 0 Rev 1 3 4 Danger d lectrocution AN Danger d electrocution V Lorsque le module de ventilation est retir les broches d alimentation sont expos es A dans l emplacement du module NE PAS ins rer d outils ou la main dans l emplacement du module 5 Surchauffe Cet quipement ne doit pas tre en service dans un local dont la temp rature d passe le N maximum recommand de 45 C 113 F En outre et pour garantir une circulation y d air correcte laisser un espace d au moins 8 cm 3 autour des orifices de ventilation 6 Ch ssis empil sur d autres quipements L
39. as needed to fill the order 2 5 Spine Package Contents The spines are shipped 3 to a box The customer will receive as many boxes as needed to fill the chassis with a full complement of spines Insert the bottom spine board first 2 6 Management Package Contents The package includes e all of the management modules ordered by the customer e RJ45 to DB9 harness for each management module received 2 7 Physical Installation Warning This equipment is very heavy Safety is the first concern Make sure that adequate manpower and proper equipment is used for transporting and moving the chassis The fully loaded chassis weighs 77kg 170 LBS full configuration 32 KG 71 LBS empty configuration 54 7kg 120 LBS shipped configuration The switch platform uses 7U of rack space in a standard 19 rack 6U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine Mellanox Technologies 24 ST gt InfiniBand FDR Switch Platform Hardware User Manual side There are upper brackets to connect the leaf side to the rack near the top of the chas sis The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have between 65cm and 80cm between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling Choose a rack w
40. down command no power enable lt module gt For example to shut down spine 06 run the command below switch master config no power enable S06 2 Push outward on the ejector handles to unlock the ejectors from the chassis Mellanox Technologies 52 InfiniBand FDR Switch Platform Hardware User Manual 3 Open the ejectors until they are at a 45 degree angle from the module y N Do not use the fan FRU handle to extract the spine board 4 Pull out the module half way through the guiding rails using both ejectors 5 Lock the ejector handles 6 Hold the body of the board on both sides and remove it from the chassis 2 10 3 2 Inserting a Spine Board When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a spine board 1 Check for foreign objects in or mechanical damage to the chassis spine slots 2 Check spine mechanics for any noticeable damage or deviations from the pass crite rion shown in Figure 40 Figure 40 Intact vs Defected Mechanics Mellanox Technologies 53 J InfiniBand FDR Switch Platform Hardware User Manual 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criteria shown in Figure 41 and Figure 42 Fi
41. f F 2 4 TVS 5 6 lt 3 O O m 0 lt J 0 H H 5 5 3 55 E O 3 5 0 o gt 20 G 5 Vs 35 U 2 HP 18 35 View into Front of Cage 8 5 Mellanox Technologies 91 J InfiniBand FDR Switch Platform Hardware User Manual Appendix F Replacement Parts Ordering Numbers Table 23 Replacement Parts Ordering Numbers Part Description OPN Power supply unit PSU 1000W for Modular Switch 1 Family Power Supply unit PSU 1670W for Modular Switch MTP006002 Family Power supply blank MTM005001 Leaf board unit 18 port FDR MSX6001FS Leaf board unit 18 port FDR10 MSX6001TS Modular Switch Family Leaf Blank MTM005004 Spine board unit for FDR system MSX6002FLS Spine board unit for FDR10 QDR system MSX6002TBS Spine board blank MTM005002 PPC460 Management Module MSX6000MAR x86 Management Module MSX6000MBR Modular Switch Family Management Blank MTM005003 Rack Installation Kit for ISS600 SX6536 Series MTR005600 SX6536 SX6518 SX6512 MSX 6506 Modular Switch MTF005005 Series Spine Fan Unit DB9 to RJ45 Harness HAR000028 Power cord 250V 15A 2 0M C14 to C13 ACC000334 Power cord 125V 15A 2 0M C14 TO C13 ACC000242 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Ty
42. gt 55 kg 3 Apparecchiatura pesante 4 Questa apparecchiatura molto pesante e va spostata mediante un sollevatore meccanico per evitare lesioni Mellanox Technologies 1 2 3 Rev 2 0 Rev 1 3 4 Rischio di scosse elettriche N Rischio di scosse elettriche Con il modulo ventola rimosso i pin di alimentazione sono accessibili all interno della 4 N cavit del modulo NON inserire strumenti o parti del corpo nella cavit del modulo della ventola 5 Temperatura eccessiva AN Questa apparecchiatura non va utilizzata in un area con una temperatura ambiente y superiore a quella massima consigliata 45 C 113 F Inoltre per assicurare un N flusso d aria adeguato lasciare almeno 8 cm 3 pollici di spazio attorno alle aperture di ventilazione 6 Impilare lo chassis Kuciste se ne bi trebalo slagati na drugu opremu Ako kuciste padne moze izazvati tje lesne ozljede i o te enje opreme 7 Collegamento di alimentazione ridondante Pericoli elettrici installato di uno spazio vuoto Qualora l alimentatore non sia installato non utilizzare l Questo prodotto dotato di un alimentatore ridondante o qualora esso non sia il prodotto con il coperchio rimosso o non fissato correttamente 8 Prese di alimentazione multiple Rischio e pericolo di scosse elettriche Gli alimentatori sono tutti indipendenti Scollegare tutti gli alimentatori per assicurarsi che il commuta
43. in the US National Electrical Code and the Canadian Electri cal Code 23 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring Mellanox Technologies 21 J InfiniBand FDR Switch Platform Hardware User Manual M 24 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 25 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user 26 Do Not Use the Switch as a Shelf or Work Space Y Caution Slide rail mounted equipment is not to be used as a shelfor a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance 27 WEEE Directive X According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of
44. link ad width configurations can be retrieved using management software Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has the following LEDs One status LED for the spine health One status LED for the spine fan modules 36status LEDs showing the existence of leaf to spine connections One Bad Port Led showing symbol errors in the data stream OneUID LED that can be lit to identify an individual spine The 36 LEDs on each spine are divided by the number of leafs and the result N is the number of connections from each leaf that are connected to the spine 3 spines must be installed and working to ensure that full BW exists between nodes The maximum num ber of connections from each leaf is 6 If the number of leafs x the maximum number ellanox Technologies 64 InfiniBand FDR Switch Platform Hardware User Manual of connections per leaf is less than 36 then some of the 1081 to spine connection LEDs may be OFF The status LEDs for the spine and their descriptions are shown in Table 7 The LEDs indicate as follows Figure 54 Spine Status LEDs Spine Status LED Fan Status LED Bad Port LED UID LED uy SO Ki 6 8 10 12 14 16 18 Spine leaf IB connection status 3 1 3 1 Status LED N Table 7 shows the spine status according to the LED condition Table 7 Spine Status LED LED Condition LED Description
45. of a special tool lock and key or This unit is intended for installation in a Restricted Access Location A restricted other means of security 5 2 1 Installation Safety Warnings These safety warnings are in English For French German Spanish Russian and Roma nian see the Appendixes 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product LA e lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Heavy Equipment k This equipment is very heavy and should be moved using a mechanical lift to avoid injuries Mellanox Technologies 18 InfiniBand FDR Switch Platform Hardware User Manual 4 Installation in Restricted Access Location This unit is intended for installation in a Restricted Access Location 5 Risk of Electric Shock Risk of Electric Shock LA With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 6 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 7 Stacking the Chassis The chassis should
46. php pg cables for the cable class rating of Mellanox cables Mellanox Technologies 44 InfiniBand FDR Switch Platform Hardware User Manual 2 10 Hot Swap Insertion and Extraction Before starting any procedure on the SX6506 switch system put an ESD prevention wrist strap on your wrist and connect to the SX6506 chassis 4 Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start anew cycle checking through the leafs and spines for the FW versions This switch platform supports hot swap capabilities for the parts listed below Power supply units Leaf boards Spine boards Leaf fan module Spine fan module Management modules 2 10 1 Power Supply Units The 1000W power supplies deliver 1000W at 48V The input to each of these power sup plies requires 1176 5W in order to output 1000W The 1670W power supplies deliver 1670W at 48V The input to each of these power sup plies requires 1882 4W in order to output 1670W For N 1 configuration PSUs Power Supply Unit can be extracted without brin
47. se nalazi u krugu bez LPS vodi a Za tita od strujnog preoptere enja Uvijek dostupni odobreni za titni ure aji od strujnog preoptere enja nazivne struje od 20 A moraju se ugraditi u o i enje zgrade 19 Nemojte koristiti preklopnik kao policu ili radnu povr inu Pozor Oprema montirana na kliza e vodilice ne bi se trebala koristiti kao polica ili radna povr ina Vodilice nisu namijenjene za povla enje ure aja iz ormari a Slu e samo za trajnu instalaciju na kona nom polo aju a ne za servisiranje i odr avanje 122 Mellanox Technologies 20 WEEE direktiva Sukladno WEEE direktivi 2002 96 EZ sav elektri ni i elektroni ki otpad EEE trebao bi se prikupljati zasebno i ne bi se trebao odlagati kao obi an ku anski otpad Odlaganje ovog proizvoda i svih njegovih dijelova vr ite na odgovoran i ekolo ki na in 21 Elektri na ograni enja dr ave Norve ke Ovaj je ure aj namijenjen samo za spajanje na elektri ni sustav s TN uzemljenjem i na elektri ni sustav s IT uzemljenjem dr ave Norve ke G 11 Avvertenze di sicurezza per l installazione italiano 1 Istruzioni di installazione Leggere tutte le istruzioni di installazione prima di collegare l apparecchiatura all alimentazione 2 Lesionia causa del peso h Usare un numero di persone sufficiente per sollevare in sicurezza guesto prodotto lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs
48. should be green Table 14 Management Module S Fan LED Configurations LED Condition Description Green OK All Spine fans are working at configured speed Red Error One or more of the spine fans within a spine fan module is bad Check the spines for a flashing green fan LED 3 1 5 3 Management Module L FANS LED Indicator The management module L FANS indicator should be green Table 15 Management Module L Fan LED Configurations LED Condition Description Green OK All Chassis Leaf fans are working at configured speed Red Error One or more of the leaf fans for the chassis is bad Check each chassis fan for a red LED Mellanox Technologies 69 AAA InfiniBand FDR Switch Platform Hardware User Manual 3 1 5 4 Management Module MASTER LED Indicator Table 16 Management Module MASTER LED Configurations LED Condition Description Green This management module is the master Off This management module is the slave 3 1 6 Port Connector Interfaces 3 2 3 3 3 4 The connector side of the switch has 6 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 58 The bottom row ports are flipped from the top row See Figure 59 Figure 58 Port Numbering Figure 59 Top and Bottom Ports Air Flow These switches come with the air flow pa
49. the leafs spines and management modules in the chas sis Remove all leafs 7 Remove all spines 8 Remove all management modules 9 Use enough people to safely lift this product LA E Lil lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs product contains printed circuit boards cables and batteries According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be Ad collected separately and disposed of according to the directive P This product and all of its parts are NOT to be disposed of with household waste This Go to the Mellanox website for detailed instructions for disassembly of the FRUs and chassis according to the WEEE Directive 10 Dispose of these pieces in a legal and environmentally friendly way 7 1 1 Removing the Chassis 1 2 CP Remove the screws connecting the upper brackets to the rack Remove the screws holding the chassis to the rack These screws are located in the faceplate on the spine side of the chassis Remove the upper brackets from the chassis Remove the lock down bars Dispose of the Chassis in a legal and environmental way Go to the Mellanox website for detailed instructions for disassembly of the chassis according to the WEEE Directive 7 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack Mellanox
50. the pull out tab below the Mellanox logo on the spine side of the chas sis Figure 1 Product Label 2 S N MT0806X01504 I MSX6506 3R utp 0002c9020027051c EA en MAC 0002C9020040 III di Made in Israel Mellanox Technologies InfiniBand FDR Switch Platform Hardware User Manual The GUID is the System Image GUID according to the IB spec Itis burned on the board which is in the chassis All the boards and the management software look for this GUID in addition to their own Node GUID Management Module MAC Here is the tab with the product label Each management module has a label with its MAC address See Figure 2 for the loca tion of this label Figure 2 Management Module MAC Address Location MAC address location CONSOLE RST PSU SFANS LFANS Master k DIE Product Physical Specifications and Power The switch ships in a minimum base configuration plus additional modules depending on the chosen customer configuration Optional modules included Leaf boards Management modules Spine boards Mellanox Technologies 13 J InfiniBand FDR Switch Platform Hardware User Manual The following figure shows the connector and spine sides fully populated Figure 3 SX6506 Switch Views Leaf Side 3 1172 5W PSUs power supply units or 2 1882 4W PSUs 3 spine modules 4 leaf fan modules 6 leaf modules 2 management modules 14 Infin
51. 00509599909 PSU 1 0555555065 25985550606 PSU 2 055559000 000000000 55550505050 Oooo oc Blank 55955550565 5585550605 PSU 2 5555590050 2555000595 5555550500 5585550605 Mellanox Technologies 62 InfiniBand FDR Switch Platform Hardware User Manual 3 1 2 Leaf Board LED Indicators Figure 53 Leaf Board Led Indicators 3 1 2 1 Status LED Each leaf board has a Status LED on the far left ofthe leaf Table 4 shows the leaf status according to the LED condition The Leaf Status indicator LED A has the following LED assignment Table 4 Leaf Status LED LED Condition LED Description Off No power to the Leaf Solid Green Leaf is up and running Flashing Green Leaf is powering up Solid Orange Non fatal error this leaf needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 2 2 Bad Port LED The Port indicator is located on the left side of the leaf The following Bad Port con ditions are possible Table 5 Bad Port LED Configurations LED Configuration Description Off OK all ports are up and running Flashing Orange Error one or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are bad cable bad connection bad connector Mellanox Technologies 63 J InfiniBand FDR Switch Platform Hardware User Ma
52. 1 Place the cable holder next to the rack on the connector side of the chassis and iden tify the holes where the caged nuts are to be placed The cable holders should go to the outside of the vertical supports A lt 2 Setthe bottom of the cable holder at the level of the shelf If the cable holder is not set properly the upper bracket will not line up with the cable holder Mellanox Technologies 34 InfiniBand FDR Switch Platform Hardware User Manual Figure 17 Cable Holder Placement Notch for upper bracket washer Set this even with the top of the shelf Screw the cable holder onto the rack using the screws provided Repeat steps 1 3 for the second cable holder Install the shelves The cable support shelves just sit on the cable holders Nam dq M Use the holes in the shelves to tie down the cables individually or in bundles 7 V Use Velcro ties to tie down the cables Electric cable ties are not recommended Mellanox Technologies 35 J InfiniBand FDR Switch Platform Hardware User Manual 2 7 3 Installing the Chassis Figure 18 Chassis Rails and Rail Slides Rail slides y N Do not use the handles on the fans for lifting or moving the chassis in the rack 1 Screw the handles onto the chassis Use the 8 allen head screws and the allen wrench provided Two handles go on the connector side and two go on the power side of the chassis Do not use the locknuts They are extras not ne
53. 3 3 Cable Power Budget Classification 70 3 4 Management Module Interfaces 70 93 4 d 412655 E yel te e ee 71 34 2 CONSOLE fic tos ae dest et edo seb da iaia 71 3 4 3 MGT Management 71 SAA USB isis a ii ERA a ES Res 71 3 4 5 Reset a a Re 72 Chapter 4 Chassis Power 73 4 1 Power Supply and Spine Board Indicator Status at Power ON 74 Chapter 5 Switch Management Tools 75 5 1 InfiniBand Subnet 76 5 2 Fabric Inspector Diagnostics 76 5 3 Accessing the CPU via the Ethernet Connector 76 5 4 Upgrading Software sa si Hama ra De dr 76 Chapter 6 Troubleshooting 77 6 1 Power Supply Unit 77 D2 Leat Board see a d dob POT NC e ed gu 77 6 3 Management Module 78 6 4 Spine Board ul ea ARCU RR A 79 6 5 MLNX OS Software 79 Chapter 7 Disassembly and Disposal 80 7 1 DisassemblingtiheChassis 80 7 1 1 RemovingtheChassis 80 7 1 2 RemovingtheBottomShelf
54. 40 Ibs 40 70 Ibs 70 121 Ibs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Mellanox Technologies 113 Rev 1 3 4 TOKOM CHAT 5 45 113 F 8 3 6
55. 6 3 Management Module Issue 1 Yellow Status LED for the Chassis on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Reset the master management module by pushing the rest button If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master N Ifthere is only one management module in the chassis all ofthe leafs and ports are reset by bringing them down and powering them up when the management y module is removed Make sure the S Fans and L Fans LEDs are green Make sure that the spine and the leafs both have the same version of FW Reburn the FW and remove and reinstall the management module SD If you are running the chassis with only one management module remove and rein stall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 2 10 5 on page 57 7 If you are running the chassis with only one management module replace the man agement module Issue 2 Yellow LED for the Leaf Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust bloc
56. Bad Port conditions are possible Table 10 Bad Port LED Configurations LED Condition Description Off OK No ports have received symbol errors recently Flashing Orange Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are bad cable bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED imme diately goes off until the next symbol error is received UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition ellanox Technologies 66 J InfiniBand FDR Switch Platform Hardware User Manual Figure 55 Spine Side Panel Display Status Indications STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER Table 11 Spine Side LED Display for Normal Operation 8 Normal LED Condition Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PS
57. Mellanox TECHNOLOGIES Connect Accelerate Outperform 108 Port InfiniBand FDR SwitchX Switch Platform Hardware User Manual PN MSX6506 3R MSX6506 NR Rev 2 0 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF
58. Mellanox Technologies Rev 2 0 5 nio gt PR 0525 6 Y N 110 PN 45 C 113 F YT nov DYP 8 3 DAV 7 PY PN gt bawn 8 MI PINI IN PN 4 9 MID 4 TON MIN MIN pin DITO NIWAN
59. Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Solid Orange Non fatal error this spine needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 3 2 Fan LED The spine fan indicator LED has the following LED assignment Table 8 Spine Fan Status LED LED Color LED Description Solid Green Spine Fan is OK Flashing Green Spine Fan needs replacing Mellanox Technologies 65 Ri InfiniBand FDR Switch Platform Hardware User Manual M 3 1 3 3 3 1 3 4 3 1 3 5 Table 8 Spine Fan Status LED LED Color LED Description Solid Yellow One or more of the fans in this spine is not working Each spine has two fans in the fan module Spine to Leaf IB Connection Status LEDs The leaf connection status on each spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per leaf per spine and a maximum of 6 LEDs per leaf These LEDs indicate a valid connection between a leaf and a spine Table 9 shows the leaf to spine status according to the LED condition Table 9 Spine to Leaf IB Link Status LED Condition LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Solid Orange Physical connection Bad Port LED 4 The Bad Port indicator is located on the left side of the spine The following
60. T R certificate x Argentina S mark certificate DoC BSMI certificate for the power supply only CE EMC amp safety report and certificate USA FCC part 15 class A ms Canada ICES 003 class A missions Australia New Zealand AS NZS CISPR 22 class A Japan VCCI Class A Australia C Tick certificate EU IEC 60068 2 64 Random Vibration Environmental EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test 76 DbA Acoustic ISO 7779 ETS 300 753 ellanox Technologies 83 J InfiniBand FDR Switch Platform Hardware User Manual Table 17 Switch Specification SX6506 3R NR Physical Shock and ETSI EN 300 019 2 2 1999 09 Vibration Scalability and Performance Switching capacity 12 1Tb s Switching performance Simultaneous wire speed any port to any port Addressing 48K unicast addresses max per subnet and 15 5K multicast addresses per subnet Leafs Up to 6 leafs 18 ports each Switching Spines 3 spine boards All spines are needed for non blocking configuration Performance Capacity Management modules 2 available 1 required for operation Management CPU PowerPC 460EX Power Sup 3 1000W Power Supplies for SX6506 3R plies 2 1670W Power Supplies for SX6506 NR Leaf fans 4 Spine fans 3 A 1 EMI Certification EMI certification on the fully populated chassis was performed with the chassis installed in a closed two door rack using the c
61. Technologies 80 ei InfiniBand FDR Switch Platform Hardware User Manual 2 Remove the filler panels 3 Remove the shelf 4 Remove all of the caged nuts 7 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Go to the Mellanox website for detailed instructions for disassembly according to the WEEE Directive Mellanox Technologies 81 AA RECI InfiniBand FDR Switch Platform Hardware User Manual Appendix A Specification Data Table 17 Switch Specification SX6506 3R NR Physical 10 43 x 19 x 27 inches Size 265mm x 482 6mm x 685mm The shelf adds 44 5mm to the height Mounting 19 Rack mount of Spines 3 of Leafs 6 32 0kg 70 55 Ibs empty configuration Weight 61 8kg 136 25 lbs shipped configuration 77 0kg 169 76 lbs full configuration CoGh 311 80mm from bottom of chassis Center of CoGw 222 00mm from left side surface Gravity CoGd 328 14mm from port end surface Measured from the bottom left corner as one faces the switch ports Max Air Flow Through Leafs 6 789M min 239 9 Max Air Flow Through Spines 1 585M min 56 0 CFM Total Max Air Flow 8 374M min 295 9 CFM SerDes Speeds
62. U STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MNGI Off Green MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master MNG2 Off Off MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master Mellanox Technologies 67 om InfiniBand FDR Switch Platform Hardware User Manual 3 1 5 Management Module LED Indicators Figure 56 Management Module Status Indications The management module LEDs display the switch system operating conditions Table 12 Management LED Display for Normal Operation sti ga Normal LED Condition Description State STATUS Off No Power Green This LED Green System is up and running shows the sta Yellow System warning Attention needed such as overhea
63. VC yal t ml bir g kayna kablosu se in Kablonun 250 V 10 A de erinde bir kaliplanmis fi i olmas gerekmektedir 17 Y ksek Kacak Ak m Uyary Y ksek ka ak akim varsa g kaynag na baglanmadan nce mutlaka toprak lama baglant s yap lmal d r 18 Unitelerin Ara Ba lantisi RS232 nitesini ve Ethernet Arabirimlerini baglayacak olan kablolar n UL onayl DP 1 veya DP 2 tipi olmas gerekir Not LPS olmayan devreye aitse A r Ak m Korumas Kolayca eri ilebilecek 20 V Kay tl devre par as a r ak m koruma cihaz n n bina elektrik ebekesinde kurulu olmas gerekir 19 Anahtar Raf veya al ma Alan olarak kullanmay nl Dikkat S rg l rayl ekipman raf veya al ma alan olarak kullan lamaz Raylar niteyi iskeleden uza a kayd rmak i in yap lmam t r Sadece ekipman n son olarak duraca yerdeki kal c montaj i indir servis veya bak m i in kullan lamaz Mellanox Technologies 129 Rev 1 3 20 WEEE Y nergesi WEEE Y nergesi 2002 96 EC uyar nca t m elektrikli ve elektronik ekipman at klar EEE ayr olarak toplanmal ve evsel at klarla birlikte pe at lmamal d r x Bu r n ve t m pargalar evreye dost ve sorumlu bir sekilde imha edilmelidir 21 Norve G K s tlamalar Bu nite bir TN g sistemine ve sadece Norve in TT g sistemine ba lanmak i indir Sp G 13 Japan VCCI Class A Statement
64. X OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot command in the CLI or reboot through the WebUI The form is identical between the two PSU types while the 1000W PSU weighs 0 3kg more than the 1670W PSU Table 3 OPNs for Power Supply Units PSU SI OPN Wattage Description MTP005001 1000W Supplies N 1 redundancy MTP006002 1670W Supplies N N redundancy for all switch chassis at 220 Volts Mellanox Technologies 17 J InfiniBand FDR Switch Platform Hardware User Manual 2 Installation 4 This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports of the rack Ad Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the open ings are populated with blanks All of the leafs spines and management modules come shipped in a separate package The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instructions to make the initial configuration access area can be accessed only through the use
65. a omogu ite najmanje 8 cm 3 in a razmaka oko otvora ventilatora 6 Slaganje ku i ta i Ku i te se ne bi trebalo slagati na drugu opremu Ako ku i te padne mo e izazvati tje lesne ozljede i o te enje opreme 120 Mellanox Technologies Rev 2 0 7 Redundantno napajanje Opasnost od elektri ne energije U slu aju praznog prostora za napajanje nemojte rukovati proizvodom ako je pok a Ovaj proizvod uklju uje redundantno napajanje ili prazan prostor na njegovu mjestu A lopac uklonjen ili ako nije dobro pri vr en 8 Vi estruki ulazi za napajanje Rizik od strujnog udara i opasnost od elektri ne energije PSU jedinice su neovisne Odspojite sva napajanja kako biste osigurali stanje bez napajanja unutar platforme preklopnika 9 Tijekom udara munje Opasnost od elektri ne energije A Tijekom djelovanja munja nemojte raditi na opremi ili spajati ili odspajati kabele 10 Spajanje Odspajanje bakrenog kabela InfiniBand Bakreni kabeli InfiniBand su teski 1 nesavitljivi i kao takvi se moraju pazljivo prikl juciti na ili iskljuciti iz konektora Obratite se proizvoda u kabela za posebna upo zorenja upute 11 Montaza ormari a i servisiranje opreza kako bi se osiguralo da sustav ostane stabilan Opcenito trebali biste ispunja Kad se proizvod montira ili se servisira u ormaricu moraju se poduzeti posebne mjere vati ormari s opremom po ev i od dna prema vrhu 12 Insta
66. ables IB FDR or FDR10 leaf with active cables Mellanox Technologies 87 J InfiniBand FDR Switch Platform Hardware User Manual Table 21 Calculating Total Switch System Power Consumption Part Quantity Power Quantity x Power IB FDR or FDR10 leaf with optical cables Spine FDR or FDR10 Chassis fans 1 Management module Total power N N redundancy ONLY works with a supply voltage of 220V Mellanox Technologies 88 J InfiniBand FDR Switch Platform Hardware User Manual Appendix E QSFP Interface M Table 22 InfiniBand QSFP Connector Pinout Connector Een da Xi Pin Name Signal Description 1 GND Ground 2 Tx2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 5 Tx4n Transmitter Inverted Data Input 6 Tx4p Transmitter Non Inverted Data Input 7 GND Ground 8 ModSelL Module Select 9 ResetL Module Reset 10 Rx 43 3 V Power supply receiver 11 SCL 2 wire serial interface clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxln Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted
67. alteten Zustand im Inneren der Switch Plattform sicherzustellen Mellanox Technologies 107 Rev 1 3 8 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 9 Anschlie en Trennen von InfiniBand Kupferkabel N InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig N an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen y Warnungen und Anleitungen des Kabelherstellers 10 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 11 Ger teinstallation A Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert aus N getauscht oder gewartet werden 12 Ger teentsorgung N 4 N Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 4 13 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bes timmungen installiert werden 14 Installationscodes installiert weden In Nodamerika muss das Ger t gem den geltenden Anforderungen EN N Dieses Ger t muss entsprechend der akt
68. ardware User Manual 4 Chassis Power Up Before starting any procedure on the SX6506 system put an ESD prevention wrist strap on your wrist and connect to the SX6506 chassis With N 1 PSU redundancy the chassis must be started with a full complement of possi ble PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it will be necessary to reboot the chassis once the defec tive PSU has been replaced With 1670W optional PSU supplies the chassis can run on as little as 1 2 of the full com plement of PSUs N N allows the chassis to run on 1 PSUs supplied from one power grid while 1 is connected to a second power grid 1 Check all FRUs for proper insertion and seating before connecting the AC power cords Boards Power supplies Leaf fan modules Spine fan modules 2 Insert all leafs that you plan to use in the chassis Start at the bottom of the chassis and work your way up 3 Insert the first two spines in the top two slots 4 Insert the rest of the spines from the bottom of the chassis up to slot 3 5 Insert thermal blanks in unused leaf slots to maintain ba
69. chiatura va installata in conformit con i requisiti applicabili del codice elettrico nazionale USA e del codice elettrico q pp canadese 16 Cavo di alimentazione UL e munito di certificazione CSA Per una connessione di alimentazione nordamericana selezionare un cavo di alimentazione di tipo UL e munito di certificazione CSA a 3 conduttori 16 AWG terminato con una spina stampata con tensione nominale pari a 125 V 13 A di lunghezza minima pari a 1 5 m sei piedi ma non piu lunga di 4 5 m Per una connessione europea selezionare un cavo di alimentazione armonizzato a livello internazionale e contrassegnato da lt HAR gt a 3 conduttori minimo 1 0 mm2 fili con guaina isolante in PVC Il cavo deve disporre di una spina stampata di potenza nominale pari a 250 V 10 A 17 Corrente di dispersione elevata Avvertenza corrente di dispersione elevata il collegamento a terra essenziale prima di collegare l alimentazione Mellanox Technologies 125 Rev 1 3 18 Interconnessione delle unita A I cavi per il collegamento all unit RS232 e alle interfacce Ethernet devono 4 disporre della certificazione UL ed essere del tipo DP 1 o DP 2 Nota in caso y di installazione su un circuito la cui potenza non limitata Protezione contro le sovracorrenti la cablatura dell edificio deve integrare un dispositivo di protezione contro le sovracorrenti di potenza nominale pari a 20 19 Non utili
70. configuration must be done on all of the management modules The first man agement module you configure will be the master 4 3 4 4 USB The USB port can be used to upload new SW using any storage device that has a USB connector This interface is USB 2 0 Mellanox Technologies 71 J InfiniBand FDR Switch Platform Hardware User Manual 3 4 5 Reset RST The Reset button resets the chassis management module when the button is pushed When the button is held down for 15 seconds the management module is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip This button resets the CPU of the management module A quick push of this button per forms this reset When the reset button is pushed on the master management module this management module is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin 2 Mellanox Technologies 7 J InfiniBand FDR Switch Platform H
71. connexion des unites A Les c bles de connexion aux interfaces RS232 et Ethernet de l appareil doivent tre certifi UL de type DP 1 ou DP 2 Note en cas d installation sur un circuit dont la N puissance n est pas limit e Protection contre les surintensit s le c blage de l immeuble doit int grer un dispositif certifi de protection contre les surintensit s calibr 20 A et ais ment accessible 19 Ne pas utiliser comme tag re ou plan de travail Attention un quipement coulissant ou mont sur rail ne doit pas servir d tag re ni de plan de travail Les rails ne sont pas destin s faire coulisser l unit hors du rack Ils sont destin s a une installation permanente a l emplacement final pas pour l entretien ni la maintenance 20 Directive DEEE Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doivent tre collect s s par ment et ne pas tre mis au rebut avec les d chets m nagers habituels Ce produit et toutes ses pi ces doivent tre mis au rebut d une mani re responsable respectant l environnement 21 Restrictions concernant l alimentation pour la Norv ge Cet appareil est pr vu pour tre reli un systeme d alimentation TN et un systeme d alimentation informatique de Norvege uniquement G 6 Installation Sicherheitshinweise German 1 Installationsanleitungen N Lesen Sie alle Installationsanleitungen bevor
72. ctrici sunt accesibili n cavitatea modulului NU introduce i instrumente sau p r i din corp n cavitatea modulului ventilator 5 Temperatura n exces Acest echipament nu trebuie sa fie actionat intr o zona unde temperatura ambianta dep seste valoarea maxima recomandat 45 C 113 F In plus pentru a asigura un flux de aer adecvat lasati un spatiu liber de cel putin 8 cm 3 inchi in jurul fantelor de ventilare 6 Suprapunerea cadrului a A N Cadrul nu trebuie s fie suprapus peste niciun alt echipament In cazul in care cadrul N cade poate cauza leziuni corporale si deterior ri ale echipamentului 7 Conexiunea la o surs de alimentare electric suplimentar pericol electric N Acest produs include o surs de alimentare suplimentar sau un spatiu gol in locul prox p pat u g 4 acesteia In cazul n care spatiul pentru sursa de alimentare este gol nu operati produ y sul c nd capacul orb este indep rtat sau nu este fixat in mod sigur Mellanox Technologies 117 Rev 1 3 8 Multiple mufe electrice Risc de soc electric si pericol electric Toate aparatele cu alimentare de la retea sunt independente Deconectati toate sursele de alimentare cu energie pentru a asigura decuplarea in inter iorul platformei de comutare 9 n timpul desc rc rilor electrice pericol electric In timpul perioadelor desc rc ri electrice luminoase nu lucrati cu echipamentul sau
73. d user assigned system names GUIDs 5 3 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet WEB SNMP XML 5 4 Upgrading Software The new software and firmware updates are available to the user from the Mellanox Sup port website Copy the update to a known location on a Remote server within your LAN Use the CLI or the GUI in order to perform the Software upgrade For further informa tion please refer to the MLNX OS user manual earlier version than the minimum that the new version of the software can work with then the chassis management system may require up to 45 minutes to update all of the Da FW images in all ofthe leafs and spines If MLNX OS is updated and the FW image in the leafs and spines of the chassis is an Mellanox Technologies 76 Ne InfiniBand FDR Switch Platform Hardware User Manual 6 Troubleshooting 6 1 Power Supply Unit Ifthe Power supplies cannot supply enough power the management module may shut down the some leafs As each PSU is plugged in make sure that green power LEDs on PSU comes Issue 1 Ifthe AC power LED is off Check that the power cable is the correct power cable for your country Check that the power cable is plugged into a working outlet Check that the power cable has a voltage within the range of 100 240 volts AC Remov
74. de alimentare care este inregistrat UL si certificat CSA cu 3 conductoare 16 AWG terminat cu o fis turnat cu putere nominal egal cu 125 V 13 A cu o lungime de minimum 1 5 m sase picioare dar nu mai lung de 4 5 m Pentru conectarea la o sursa de alimentare in Europa selectati un cordon de alimentare care este armonizat la nivel international i marcat lt HAR gt cu 3 conductoare cu minimum 2 fire de 1 0 mm cu putere nominal egal cu 300 V si cu o manta izolant din PVC Cordonul de alimentare trebuie s fie prev zut cu o fis turnat cu putere nominal egal cu 250 V 10 A 17 Curent de scurgere de inalt frecvent Avertisment Curent de scurgere de inalt frecvent Imp m ntarea este esential inainte de a conecta sursa de alimentare 18 Interconectarea unit tilor Cablurile pentru conectarea la unitatea RS232 si la interfetele Ethernet trebuie s fie de tipul DP 1 sau DP 2 certificate UL Not c nd se reg sesc intr un circuit non LPS Protec ie la supracurent Un dispozitiv de protec ie la supracurent nregistrat n cir cuitul de ramificare usor accesibil si cu o putere nominal egal cu 20 A trebuie s fie integrat in cablajul cl dirii 19 Nu utilizati comutatorul ca raft sau spatiu de lucru Atentie Echipamentul montat pe o linie de alunecare sin nu va fi utilizat ca raft sau spatiu de lucru Scopul sinelor nu este de a glisa unitatea de pe rack Acestea s
75. de protecci n contra sobrecargas de circuito derivado de f cil acceso con una corriente nominal de 20 A 112 Mellanox Technologies Rev 2 0 20 No utilizar el conmutador como estante ni como espacio de trabajo Cuidado Equipos montados en deslizadores o rieles no se deben utilizar como estantes ni como espacio de trabajo La finalidad de los rieles no es deslizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o mantenimiento 21 Directiva WEEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera respons able y respetuosa con el medio ambiente 6 8 Rus sian 1 2 AR Tre
76. dle onto the board Mellanox Technologies 50 InfiniBand FDR Switch Platform Hardware User Manual 2 10 3 Spine Boards Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation P Do NOT mix replacement parts based on different generations of chip When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sions Each spine has a pair of ejectors that lock the board in place and serve as levers for seat ing or extracting see Figure 39 Management board 1 is connected to spine board 1 and management board 2 is con nected to spine board 2 When a slave management board is not installed or not working the spine board connected to the master management board cannot be hot swapped All of the spine boards can be hot swapped when two management boards are installed and working When more than one spine slot is empty always insert the lowest spine board first then work your way up If you need to replace the bottom spine and it does not go in try removing one or two spines above it and then insert the spines from the bottom up 2 10 3 1 Extracting a Spine Board A A Wh
77. e and reinstall the power cable Check the circuit breakers to be sure that the breaker has not tripped Check that the power cable is good Replace the power cable Ifthe AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU Y hae uU REA 6 2 Leaf Board Issue 2 The power LED for the Leaf board is off 1 Make sure that all ofthe PSUs are showing DC OK 2 Uninstall and reinstall the Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one 4 Should any of the boards shut down due to over temperature wait 5 minutes and then follow the procedure starting with step 2 Be Issue 3 The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable Issue 4 The Activity indication does not come on Check that the Subnet Manager has been started Mellanox Technologies 77 Ne InfiniBand FDR Switch Platform Hardware User Manual
78. e ch ssis ne doit pas tre empil sur d autres quipements S il tombe il peut endom mager l quipement ou entra ner des blessures 7 Connexion de l alimentation redondante danger d lectrocution Ce produit est quip d une alimentation redondante ou d un cache si elle est absente Dans ce dernier cas ne pas faire fonctionner le produit si le cache est retir ou mal fix S Plusieurs prises d alimentation Risque et danger d lectrocution Les alimentations sont toutes ind pendantes Pour s assurer que le commutateur est bien hors tension d branchez toutes les alimen tations 9 En cas d orage danger d lectrocution Pendant un orage ne pas travailler sur l quipement ni brancher ou d brancher des c bles 10 Connexion et d connexion du c ble InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et peu flexibles Par cons quent il faut proc der avec soin pour les brancher ou les d brancher des connecteurs Consulter le fabricant du c ble pour obtenir des instructions ou des avertissements sp cifiques 104 Mellanox Technologies Rev 2 0 11 Montage en rack et maintenance ment attention pour s assurer que l ensemble reste stable En r gle g n rale le rack doit Lors du montage ou de la maintenance de ce produit dans un rack il faut faire sp ciale A etre rempli en commengant par le bas 12 Installation de l quipement N Cet quipement ne doit
79. e part before inserting the new part This is necessary so that the manage ment module will start anew cycle checking through the leafs and spines for the FW versions To insert the leaf board 1 Check for foreign objects in or mechanical damage to the chassis leaf slots 2 Check leaf mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 36 Figure 36 Intact vs Defected Signal Connectors Pass Fail Mellanox Technologies 49 ey InfiniBand FDR Switch Platform Hardware User Manual 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion show in Figure 37 Figure 37 Intact vs Defected Power Pin Holders 5 Start with the ejector handle fully open that is at 45 degrees to the front panel of the leaf 6 Holding the leaf by its sides carefully set the leaf board into the chassis 7 Applying equal pressure on both sides of the leaf board slowly slide the board into the chassis until the ejector handle reaches the vertical bar Do not apply excessive force to slide in the leaf board If you feel resistance remove the leaf board and double check both the chassis and leaf for any damage 8 Catch the hook onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector han
80. e top 14 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 15 Equipment Disposal N A Disposal of this equipment should be in accordance to all national laws and regula tions 16 Local and National Electrical Codes N This equipment should be installed in compliance with local and national electrical codes 17 Installation Codes A This device must be installed according to the latest version of the country 4 N national electrical codes For North America equipment must be installed in A accordance to the applicable requirements in the US National Electrical Code 7 and the Canadian Electrical Code Mellanox Technologies 20 Ne InfiniBand FDR Switch Platform Hardware User Manual 18 Battery Replacement A Warning Replace only with UL Recognized battery certified for maximum abnormal charging current not less than 4mA There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispose of used batteries according to the instructions 19 UL Listed and CSA Certified Power Supply Cord For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a pow
81. eaf with passive cables 109 5 111 4 IB FDR leaf with 18 active cables 150 6 157 9 IB FDR leaf with 18 optical cables 156 1 166 2 Spine FDR 128 6 135 2 IB FDR10 leaf with passive cables 84 4 97 1 IB FDR 10 leaf with 18 active cables 125 5 143 6 IB FDR 10 leaf with 18 optical cables 131 0 151 8 Spine FDR10 103 5 120 8 Chassis fans 163 3 188 2 Management module 17 6 25 9 Table 19 assumes the QSFP cable power consumption specified in Table 20 Please note that QSFP module power is related to the module itself and is not referenced to AC plane Table 20 QSFP Cable Power Consumption QSFP Cable Type Typical W Maximum W Active 1 5 1 7 Optical 1 7 2 0 To calculate the total power consumption of the switch system at a certain configuration fill out Table 21 in the following manner 1 In the Power column fill in the desired typical or maximum values Be careful not to mix FDR and FDR10 power numbers for the particular calculation 2 In the Quantity column fill in the desired configuration Please note that the number of fans is not configurable and is always 1 3 Get the total power per each FRU by multiplying values in adjusted columns in the same row 4 Get the total switch system power consumption system power by summing up all multiplication products Table 21 Calculating Total Switch System Power Consumption Part Quantity Power Quantity x Power IB FDR or FDR10 leaf with passive c
82. eded for this installation Mellanox Technologies 36 J InfiniBand FDR Switch Platform Hardware User Manual Figure 19 Installing the Handles Do not use the locknuts They are extras not needed for this installation A lt Figure 20 Screw the Handles Onto the Chassis Do not use the handles on the fans for lifting or moving the chassis in the rack PEERI The handles go here SX6506 q q DU NY KE 2 4 2 af db ab ar ij 299999909 Mellanox Technologies 37 J InfiniBand FDR Switch Platform Hardware User Manual Figure 21 The Rails are Already Connected Onto the Chassis lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 2 Lift the chassis and slide it onto the shelf Use a mechanical lift or enough people to safely lift the chassis It weighs 55kg 120165 Y N Do not use the handles on the fans for lifting or moving the chassis in the rack Mellanox Technologies 38 InfiniBand FDR Switch Platform Hardware User Manual Figure 22 Raising the Chassis Using a Mechanical Lift y d O 5 3 Push the chassis into the rack until the faceplate is 20cm 8 from the vertical support 4 Put the rail slides onto the rails Figure 23 Put on the Rail Slide
83. ems ad unmanaged switch systems need SwitchX firmware version 9 2 6000 or higher P FDR and FDR10 are only guaranteed to work with approved Mellanox Cables All cables can be inserted or removed with the unit powered on To insert a cable press the connector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 sec onds for link to get up To remove disengage the lock and slowly pull the connector away from the port receptacle For a valid physical connection both ends of the cable must be connected to working devices Take care to not impede the air flow through the ventilation holes next to the Infini Band ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 2 9 1 Supported Approved Cables For a list of approved cables for this switch see the Mellanox approved cable list http www mellanox com related docs user manuals Mellanox approved cables pdf 2 9 2 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX65xx series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages
84. en a slave management board is not installed or not working hot swapping the spine board connected to the master management board will cause the chassis to crash or spine 2 as the management modules are connected to the chassis components Neither CLI nor the GUI management tools will allow you to shut down spine 1 ad through these spines Extracting Spine Board 1 or 2 Spine board 1 is connected to management board 1 and spine board 2 is connected to management board 2 Mellanox Technologies 51 w o InfiniBand FDR Switch Platform Hardware User Manual Figure 38 Management Module Numbering MNGI management module 1 top MNG2 management module 2 bottom Removing spine 1 causes management module 1 to reset Removing spine 2 causes management module 2 to reset Warning Ifthe spine you want to hot swap is connected to the master management module the management module will reboot when you take out the spine If you have only one management module the chassis will crash Figure 39 Spine Board Extraction Pull the ejector handles out to remove the spine If you need to hot swap spine 1 or spine 2 1 Check to see ifthe spine you need to remove is connected to the master management module See Figure 38 Y N If you have only one management module the chassis will crash 2 Follow the steps in Section below Extracting Spine Boards Except 1 or 2 1 Run the shut
85. en wrench 8 lock washers 4 lock nuts for handles These nuts are not to be used 8 caged nuts for the faceplate 8 M6 bolts for the caged nuts 8 for the faceplate Parts for installing the filler panel 1 Bottom filler panel e 2 M6 bolts 2 caged nuts M6 Parts for installing the cable manager 1 RH cable holder e 1 LH cable holder 8 caged nuts M6 8 M6 bolts 3 cable holder shelves You will need 7U of space in the rack 6U for the chassis and 1U for the shelf Figure 5 Shelf Installation Kit Parts Angled Shelf Bracket Rail M6X 16 Rail Slide M6 X 12 bolts caged nuts and split lock washers Mellanox Technologies 26 InfiniBand FDR Switch Platform Hardware User Manual Figure 6 Chassis Installation Parts 48 amp Rail slide 0 M 5 Pan head ge screws and lock washers 4 hole washer Handles Screws for the handles Caged nuts gt lt 2 7 1 3 Container Mishandling The container has shock and tilt stickers applied The shock stickers turn red if the con tainer has been mishandled or roughly handled Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped If a shock sticker or more are red or if the tiltwatch sticker indicates more than 40 notify the ship per and Mellanox This on its own does not indicate damage to the contents But be sure to carefully inspect the con
86. ent should be in accordance to all national laws and regula 4 N tions 14 Local and National Electrical Codes y N This equipment should be installed in compliance with local and national electrical 4 codes 15 UL Listed and CSA Certified Power Supply Cord A For North American power connection select a power supply cord that is UL Listed N and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 16 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 17 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 19 Do Not Use the Switch as a Shelf or Work Space Caution Slide ra
87. eplace the spine fan module LA Replace defective spine fan modules as soon as they are identified 6 4 Spine Board Issue 1 The yellow LED on the Spine board is lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 2 10 3 on page 51 5 Make sure that the spine and the Leafs both have the same version of FW 6 Reburn the FW and remove and reinstall the spine 7 Replace the spine board 6 5 MLNX OS Software For more detailed instructions concerning MLNX OS software see the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual Mellanox Technologies 79 J InfiniBand FDR Switch Platform Hardware User Manual 7 Disassembly and Disposal 7 1 Disassembling the Chassis A MES en Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Loosen all locking screws for
88. er options SX6506 3R N 1 redundancy SX6506 NR N N redundancy 1 Features for a future release Mellanox Technologies 15 InfiniBand FDR Switch Platform Hardware User Manual M 1 4 1 1 4 2 Figure 4 Power Supplies for SX6506 3R and SX6506 NR SX6506 SX6506 N 1 PSU Covers Off 1 PSU 1 i e PSU 2 BOLSA SX6506 N N PSU Covers Off e o e Pii Y KIALALAL A ee vc dd 8 SX6506 PSU 2 1000W Power Supply Units The 1000W power supply units PSUs deliver 1000W at 48V The AC source to each PSU must be able to provide 1176 5W in order to output 1000W in DC With 1000W PSUS the only redundancy option is N 1 In this configuration 1PSU can be extracted without bringing down the system When using the 1000W PSUs the mini mum complement of PSUs at start up is 3 and the chassis will continue to run at full capacity with only 2 PSUs 1670W Power Supply Units The 1670W power supply units PSUs deliver 1670W at 48V The AC source to each PSU must be able to provide 1882 4W in order to output 1670W in DC When using the 1670W PSUs the minimum complement of PSUs which allows the chassis to run at full capacity is 1 The 1670W PSUs can implement the following two redundancy options No redundancy combined mode N 1 configuration ps redundant mode ellanox Technologies 16 J InfiniBand FDR Switch Platform Ha
89. er supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 20 High Leakage Current A Warning High leakage current Earth connection essential before connecting supply 21 Add GND connection information da Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 22 Installation codes A This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements
90. f you feel resistance y N remove the management and double check both the chassis and management for any damage 8 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handles onto the board the master management module CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and A slave roles see the MLNX OS Software UM section High Availability On switch systems with dual management systems first connect the cable and configure 2 All management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure A Mellanox Technologies 59 om InfiniBand FDR Switch Platform Hardware User Manual 2 10 6 Switch Shut Down Procedures To shut down the chassis run the following command twice once for each MM Reload halt noconfirm A The chassis cannot be restarted remotely To restart the chassis you must physically go to the switch and unplug all of the 1 V power cords to the chassis and then replug in all of the power cords to the chas UN The first time you run the command it shuts down the master management module and the second time shuts down the slave management module To shut down a leaf run the following command no power enable lt module gt To
91. flistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 19 Switch nicht als Regal oder Arbeitsplatz nutzen Achtung Auf Schieber Schienen montiertes Ger t ist nicht als Regal oder Arbeitsbere ich zu nutzen Die Schienen sind nicht daf r bestimmt die Einheit aus dem Gestell weg zu ziehen Sie sind nur f r die permanente Installation an einem endg ltigen Stan dort gedacht nicht f r Instandhaltung und Wartung 20 WEEE Direktive Gem WEEE Directive 2002 96 EC m ssen alle elektrischen und elektronischen Abfallger te EEE separat gesammelt und nichit mit normalem Haushaltsm ll ensorgt werden Dieses Produkt und alle seine Teile in verantwortungsvoller und umweltfreundlicher Art und Weise entsorgen Mellanox Technologies 109 Rev 1 3 G 7 Advertencias de seguridad de instalaci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Lesi n corporal a causa de peso Recurra a suficientes personas para levantar este producto sin A se lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Equipos pesados mec nico para evitar lesiones k Dado que el equipo es pesado se debe mover nicamente mediante un elevador 4 Riesgo de descarga el ctrica iRiesgo de descarga el ctrica
92. ging down the system For N N configuration up to half of the PSUs can go down and the system will continue to run The power required to run the switch system is equally divided between all ofthe work ing PSUs 2 10 1 1 Extracting and Inserting the Power Supply Unit With all ofthe 1000W power supplies installed the system is in N 1 redundant configu ration With all ofthe N N 1670W power supplies installed the system is in N N redundant configuration half of the PSUs may be extracted without bringing down the system gt To extract a PSU 1 Determine which AC connector on the connector side of the chassis corresponds to the defective PSU 2 Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering Mellanox Technologies 45 J InfiniBand FDR Switch Platform Hardware User Manual ring Power Cord Numbe Figure 30 Ions PSU Locati SX6506 with 3 IkW PSUs Figure 31 SX6506 with 2 1 6kW PSUs 3 On the spine side of the chassis remove the cover to the power supply unit for the non working PSU Odd numbered PSUs are on the left side and even numbered PSUs 46 Mellanox Technologies InfiniBand FDR Switch Platform Hardware User Manual are on the right side facing the spines There are four phillips head screws for the cover plate Figure 32 Power Supply Handle O may D P S
93. gure 41 Intact vs Defected Side Signal Connectors Pass Premere bn te mi ers NUN Figure 42 Intact vs Defected Middle Signal Connectors 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 43 Figure 43 Intact vs Defected Power Pin Holders Mellanox Technologies 54 InfiniBand FDR Switch Platform Hardware User Manual 5 Note the following sticker placed on the top of the spine board Figure 44 Spine Board Insertion Caution A INSERT SLOWLY A USE EJECTOR HANDLES ONLY EN Y A 6 Start with the ejector handles fully open that is at 45 degrees to the front panel of the spine s IPN N Do not use the fan FRU handle to insert the spine board f 7 Holding the spine by its sides carefully set the spine board into the chassis 8 Using only the ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar 4 N Do not apply excessive force to slide in the spine board If you feel resistance remove the spine and double check both the chassis and spine for any damage p 9 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 10 Lock the ejector handles onto the board 2 10 4 Fan Modules There are four fan modules on the chassis for the leafs They are located on the spine side
94. hassis installation kit supplied by Mellanox Tech nologies A 2 Approved Cables For a list of all approved cables see http www mellanox com related docs user_manuals Mellanox_approved_cables pdf A 3 EMC Certifications The list of approved certifications per chassis in different regions of the world is located on the Mellanox Website at http www mellanox com related docs user_manuals Regulatory_and_Compli ance_Guide pdf EMC Statements are also in the Regulatory and Compliance Guide Mellanox Technologies 84 InfiniBand FDR Switch Platform Hardware User Manual Appendix B Thermal Threshold Definitions There are three thermal threshold definitions for the SwitchX switch device which impact the overall switch system operation state Warning Critical and Emergency 1 Warning 100 C On managed systems only When the SwitchX device crosses the 100 C threshold a Warning Threshold message will be issued by the MLNX OS management SW indicat ing to system administration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any action by hardware such as switch shutdown 2 Critical 120 C When the SwitchX device crosses this temperature the firmware will automatically shut down the device 3 Emergency 130 C In case the firmware fails to shut down the SwitchX device upon crossing the Critical thre
95. hat none of them are bent 2 10 5 Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start anew cycle checking through the leafs and spines for the FW versions gt Extracting a Management Module Management modules are located on the leaf side above the leafs There are two places to install the management modules Only one management module is required to run the switch system ellanox Technologies 57 InfiniBand FDR Switch Platform Hardware User Manual Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 48 1 Run the shut down command no power enable lt module gt For example to shut down management module 2 run the command below switch master config no power enable MGMT2 Disconnect all cables connected to the management module Push outward on the ejector handles to unlock the ejectors from the chassis Open the ejectors until they are 45 degrees from the module Pull out the module half way through the guiding rails using the ejector handle Lock the ejector handle aD Hold the body of the board on both sides and remove it from the chassis y N The board is short therefore do not let go of it while sliding it out Figure 48 Management Module CONSOLE MGT
96. he power cords to the PSUs 20 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow status LEDs is cause for concern and must be dealt with imme diately LA It can take up to 5 minutes to boot up during which time the status LED may indicate flashing green 21 You can start connecting all of the cables to the switch 2 7 4 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge The grounding post is M 6 with Imm pitch threads A ground wire of AWG 6 or Amm diam eter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the connections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground con nection between the chassis of the switch and the rack LA Warning System grounding must comply with local electrical code Figure 26 Ground Connection ESD Post Ground Post 2 8 Power Connections The switch includes integrated hot swap power supplies which support up to 3 load shar ing 1000W or 2 load sharing 1670W supply units The s
97. hich is able to support the mechanical and environmental characteris tics ofa fully populated switch chassis 2 7 1 ESD Connection Before starting any procedure on the SX6506 switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit 2 7 1 1 Installation ProcedureRequirements You will need 2 phillips screwdriver 3 phillips screwdriver a grounding lug ground wire to properly ground the chassis Y The installation will be much easier with a power screwdriver Y It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation Mellanox Technologies 25 InfiniBand FDR Switch Platform Hardware User Manual 2 7 1 2 Installation and Cable Management Kit Parts Parts for installing the shelf 1 shelf 8 M6 bolts for the caged nuts 4 caged nuts e 2 Shelf rail slides 4 lock washers 2 long 2 hole flat washers Parts for installing the chassis 2 chassis rail slides 4 handles e 8 M 5 pan head bolts e 8 allen head screws 2 flat 4 hole metal spacers 1 all
98. i sicurezza per l installazione italiano 123 Mellanox Technologies 4 InfiniBand FDR Switch Platform Hardware User Manual 2 MontajG venlikUyar lar T rk e 127 3 Japan VCCI Class A Statement 130 G G Mellanox Technologies 5 J InfiniBand FDR Switch Platform Hardware User Manual List of Tables Table 1 Revision History of this User s Manual 9 Table 2 Reference Documents and Websites 10 Table 3 OPNs for Power Supply Units 17 Table 4 leafStatuslED 63 Table 5 Bad Port LED Configurations 63 Table 6 Connector Physical and Logical Link Indications 64 Table 7 Spine Status 65 Table 8 Spine Fan Status LED 65 Table 9 Spine to Leaf IB Link Status 66 Table 10 Bad Port LED 66 Table 11 Spine Side LED Display for Normal Operation 67 Table 12 Management LED Display for Normal Operation 68 Table 13 Management Module PSU LED Configurations 69 Table 14 Management Module S Fan LED Configurations
99. iBand FDR Switch Platform Hardware User Manual 1 2 Features List 108 FDR 56Gb s InfiniBand ports in a 6U switch 12 12 Tb s aggregate data switching capacity with ultra low latency IBTA 1 3 and 1 2 1 compliant SDR DDR QDR FDR10 FDR link speed N N power supply Congestion control Adaptive routing Port mirroring Chassis High Availability SMB High Availability 1 3 InfiniBand FDR and FDR10 Overview The Mellanox SX6506 switch system supports FDR standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified physical layer using different block types deskew mechanism and framing rules The SX6506 switch also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with a 64b 66b encoding resulting in an effective bandwidth of 40Gb s FDR10 supports 20 more bandwidth over regular QDR using the same QSFP cables connectors Both FDR and FDR10 support Forward Error Correction FEC as described in IEEE Std 802 3ap 2007 Amendment to IEEE Std 802 3 2005 chapter 74 FDR and 10 are only guaranteed to work with approved Mellanox cables A lt FDR 10 is only guaranteed to work with approved ConnectX 3 adapters A 1 4 Power Supply Redundancy The SX6506 platform has two ord
100. iband Kablo Ba lama Ba lant y Kesme Bak r nfiniband kablolar a rd r ve esnemezler Bu nedenle ba lant lara ok dikkatli bir ekilde tak lmalar veya kar lmalar gerekir zel uyar lar talimatlar i in kablo reticinize ba vurun 11 skele Montaj ve Bak m Bu r n bir iskelede monte edildiyse veya bir iskele ile sunulduysa sistemin sabit kalmas i in zel nlemler al nmal d r Genelde ekipmanlar iskeleye a a dan yukar do ru doldurman z gerekir 12 Ekipman Montaj Ekipman n yaln zca e itimli ve nitelikli personel taraf ndan monte edilmesi de i tirilmesi ve veya bak m n n yap lmas gerekir 128 Mellanox Technologies Rev 2 0 14 Yerel ve Ulusal Elektrik Kodlar Bu cihaz n lke ulusal elektrik kodlar n n son s r m ne g re monte edilmesi W gerekir Kuzey Amerika i in ekipmanin ABD Ulusal Elektrik Kodu ve Kanada Elektrik Kodu nun uygulama kosullar na g re monte edilmesi gerekir 16 UL Kay tl ve CSA Onayl G Kayna i Kablosu A Kuzey Amerika da g g baglantisi icin UL Kayitli ve CSA Onayli bir g kaynag 4 kablosu secin 3 iletken 16 AWG 125 V degerinde kal planm s bir fisle biten 13 A en az 1 5 m alti fit uzunlugunda fakat 4 5 m den uzun olmayan bir kablo Avrupa da g baglant s i in uluslararas uyumlu ve lt HAR gt isaretli 3 iletken en az 1 0 mm tel 300 V degerinde ve P
101. il mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance Mellanox Technologies 95 Rev 1 3 20 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 21 Country of Norway Power Restrictions Norway only A This unit is intended for connection to a TN power system and an IT power system of G3 nay 1 A NDN MANN mr 2 N A PN 32 nm 3 4 4 A 6
102. king the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Determine which fan module is problematic by checking the status LED on each fan module 5 Remove and reinstall the problematic fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 10 4 on page 55 6 Replace the Leaf fan module YN Replace defective leaf fan modules as soon as they are identified Should any of the boards shut down due to over temperature follow the procedure starting in step 2 Issue 3 Yellow LED for the Spine Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert Mellanox Technologies 78 J InfiniBand FDR Switch Platform Hardware User Manual 2 Determine which spine has a defective fan by checking the Fan LEDs on all of the spines 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 5 Remove and reinstall the fan unit ofthe spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 10 4 on page 55 6 R
103. lanced air flow 6 Tighten all leaf and spine mounting screws 7 Connect the power cords to the PSUs 8 Connect the power cords to grounded electrical outlets Figure 62 Do not use AC power Socket 1 with N N PSUs SX6506 3R SX6506 NR Mellanox Technologies 73 J InfiniBand FDR Switch Platform Hardware User Manual installed A With N 1 PSU redundancy do not power up the chassis with less than all PSUs 4 1 Power Supply and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes gt As the power is turned on you should observe the following conditions for normal oper ation 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the Spine LEDs and make sure they coincide with Figure 63 Figure 63 Spine Side Panel Display Status Indications STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER 5 Check the Management Module LEDs and make sure they coincide with Figure 64 Figure 64 Management Module Status Indications for Normal Operation OA A Mellanox Techno
104. le de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud minima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como minimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 17 Alta corriente de fuga ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 18 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 19 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo
105. liranje opreme Ovu bi opremu trebalo instalirati zamjenjivati i ili servisirati samo obu eno i kvalifici rano osoblje Mellanox Technologies 121 Rev 1 3 14 Lokalni i nacionalni elektri ni kodovi Ova oprema trebala bi se instalirati u skladu s lokalnim i nacionalnim elektri nim kodovima 15 Instalacijski kodovi A Ovaj se ure aj mora instalirati sukladno najnovijoj verziji nacionalnih elek AR tri nih kodova dr ave U Sjevernoj Americi oprema se mora instalirati suk N ladno va e im zahtjevima navedenim u US National Electrical Code i Canadian Electrical Code 16 UL CSA kabel napajanja IN Za sjevernoameri ku mre u odaberite kabel napajanja koji je na UL listi 1 sa MA CSA certifikatom 3 ilni 16 AWG 16 AWG koji zavr ava lijevanim uti N ka em nazivnog napona od 125 V 13 A minimalne duljine od 1 5 m six feet Sest stopa ali ne dulji od 4 5 m Za europsku mreZu odaberite kabel napajanja koji je medunarodno uskladen 1 oznacen lt HAR gt 3 Zilni s najmanje 1 0 mm Zice nazivnog napona od 300 V s PVC izolacijom Kabel mora imati lijevani utika nazivnog napona od 250 V nazivne struje od 10 A 17 Veliko curenje struje k Upozorenje Veliko curenje struje Prije spajanja napajanja nu no je spojiti uzeml jenje 18 Interkonekcija ure aja Kabeli za spajanje na jedinicu RS232 1 Ethernet su elja moraju biti s UL cer tifikatom vrste DP 1 ili DP 2 Napomena kad
106. logies 74 J InfiniBand FDR Switch Platform Hardware User Manual 5 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via MLNX OS There are 2 Ethernet ports 1 for each management module that get connected to Ethernet switches These switches must be configured to 10 100M auto negotiation 5 The SX6506 switch comes standard with a management software module for chassis management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX amp based managed switch systems management modules MLNX OS includes a CLI WebUI SNMP and chassis management features for software and IB management software OpenSM You can get more information via the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual The managed switch system includes the following software components Embedded Subnet Manager SM Chassis manager and system BIST SNMP agent 3rd party tool integration GUI Remote logging SSH telnet Secured access in band and out band IPv4 IPv6 network stack The chassis manager will give the user access to Switch temperatures Power supply voltages Fanunit information Power unit information Flash memory Monitoring of AC power to the PSUs DC power out from the PSUs chassis failures querying for switch serial number
107. lots for the power supply units PSUs are on the spine side The left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has a dedicated AC inlet This design enables the Mellanox Technologies 41 J InfiniBand FDR Switch Platform Hardware User Manual optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be standard 3 wire AC power cords including a safety ground 2 8 1 Powering Up the Switch Platform ferent countries can be ordered from Mellanox Y N Make sure that the power cords are compatible with your outlets Power cords for dif The chassis in N 1 mode must be started with a full complement of PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection When using the 1670W PSUs the minimum complement of power supply units to start the chassis is 1 and the chassis will continue to run at full capacity with only 1 PSUs The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of current so that the total cur
108. mediately 4 Check that none of the LEDs on the spines are yellow It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes 5 Check that the leaf status LEDs fan status LED and spine status LED in the spines are all green The maximum number of connections from each leaf is 6 If the number of leafs x the AA maximum number of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF Figure 29 Spine Module Leaf connection LEDs Status Fan Bad Port and UID LEDs 2 9 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from Mellanox Technologies The Mellanox proprietary QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand connections are Mellanox Technologies 43 J InfiniBand FDR Switch Platform Hardware User Manual made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 See Section 3 1 6 for port numbering 4 If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase must be programmed with management software MLNX OS 3 3 4304 or higher and To raise link at FDR 56Gb s with 3rd party FDR cables managed switch syst
109. nd put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Make sure you have the spine side close to the doors to ensure maximum cable bend ing room Mellanox Technologies 29 SN InfiniBand FDR Switch Platform Hardware User Manual Figure 10 Placement of Chassis in Rack Doors RACK Cable with recommended bending radius Chassis E Spinelside Insufficient room for recommended bending radius Doors You will most likely need extra room for cable bending on the connector side of the chas sis and should plan to keep the spine side as close to the rack door as possible thus hav ing more room for the cables See Figure 1 on page 36 3 Clip 4 caged nuts into the holes in the rack you will be using to connect the shelf brackets Check that both sides of the shelf are at the same level in the rack Figure 11 Inserting the Caged Nuts for the Shelf Top of shelf e Bottom of shelf Do not use this hole for the shelf This hole is for the filler panel 4 Insert 2 caged nuts for the filler panel 5 Using two of the bolts for each rail slide install the rail slides onto the rack Check that both sides of the switch left and right are the same level in the rack Mellanox Technologies 30 Ne InfiniBand FDR Switch Platform Hardware User Manual Figure 12 Co
110. nnect Rail Slide to Rack Vertical support 6 Place the four bolts for the caged nuts within reach 7 Put the shelf into place and screw the bolts into the nuts from step 3 8 Extend the rail slides to the rack vertical support and screw them to the rack using the 2 hole washers and lock washers See Figure 12 9 Tighten all ofthe screws to 7 7 Nm or 68 pound inches 10 Insert 8 caged nuts for the faceplate in the exact locations shown in Figure 13 Inserting the Caged Nuts for the Faceplate 11 Install the filler panel directly below the shelf Figure 13 Inserting the Caged Nuts for the Faceplate Place caged nuts here Shelf Mellanox Technologies 31 J InfiniBand FDR Switch Platform Hardware User Manual Figure 14 Chassis on Shelf with Filler Panel Shelf Filler panel Mellanox Technologies 32 J InfiniBand FDR Switch Platform Hardware User Manual 2 7 2 Installing the Cable Holder Y Now is the time to install the cable holder Figure 15 Cable Holders Right Side Cable Holder gt Left Side Cable Holder gt 8x MG xX 8 Caged Nu 8X M6 bolt EK Cable support shelf For each model of chassis there is the one support shelf for every two leafs Mellanox Technologies 33 J InfiniBand FDR Switch Platform Hardware User Manual Figure 16 Inserting the Caged Nuts for the Cable Holder Place to connect the chassis rail slide Place caged nuts here
111. not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 8 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redundant power or a blank in its place In case of a blank A fastened 9 Double Pole Neutral Fusing This system has double pole neutral fusing Remove all power cords before opening the cover of this product or touching any internal parts 10 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform Mellanox Technologies 19 InfiniBand FDR Switch Platform Hardware User Manual 11 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 12 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 13 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken LA to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to th
112. nual M 3 1 2 3 3 1 2 4 This LED lights up when one or more ports is receiving a symbol error The LED imme diately goes off until the next symbol error is received UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support Leaf Board Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port v Each port has a single 2 color LED Table 6 shows the link status according to the LED condition Table 6 Connector Physical and Logical Link Indications LED Condition LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed is proportional to data transfer speed Solid Orange Physical link up Flashing Orange A problem with the physical link The LED indicator corresponding to each data port will light orange when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a log ical connection is made the LED will change to green When data is being transferred the light will blink green The switch does not provide a visual means to indicate the port speed configuration ZA SDR DDR QDR or FDR and or the link width 1X or 4X The speed and
113. pe G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 2 Mellanox Technologies 9 J Rev 2 0 Appendix G Safety Warnings Multiple Languages G 1 Nordic Countries Notices Finland Laite on liitett v suojakoskettimilla varustettuun pistorasiaan Norway Apparatet m tilkoples jordet stikkontakt y Sweden Apparaten skall anslutas till jordat uttag G2 Installation Safety Warnings English 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product 3 Heavy Equipment gp This eguipment is heavy and should be moved using a mechanical lift to avoid injuries 4 Risk of Electric Shock Risk of Electric Shock A With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 5 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper allow at least 8cm 3 inches of clearance around the ventilation openings Mellanox Technologies 93 J Rev 1 3 6 Stacking the Chassis The chas
114. rdware User Manual When supplied from a single 220V power grid the chassis supports up to 1 redundant units Thereby the minimum required number of PSUs equals 1 When supplied by a single 110V power grid the number of redundant power supplies is only 1 NT N configuration grid redundant mode ZN y N N N redundancy ONLY works with a supply voltage of 220V The chassis PSUs are fed from two power grids for high availability The second power grid can be supplied by any of the following a backup power supply grid a generator a battery backup system any combination of the above Connecting 1 power supply to one power supply grid and the remaining 1 power supply to a secondary power supply grid will create N N redundancy This is High Availability Under these conditions should a power grid fail an electric company power failure or blackout for example power grid High Availability will continue to keep the chassis running at full capac ity through the secondary or backup power supply grid With N N optional PSU grid redundancy the chassis can run on 1 2 ofthe full complement of PSUs N N allows the chassis to run on 1 PSU supplied from one power grid while 1 is con nected to a second power grid 4 With power grid A charged with current and power grid B not charged there is only grid redundancy and not PSU redundancy When the power drops below the required minimum due to power supply failure MLN
115. rent is always divided by the number of working PSUs 1 Plug in the power cords to the PSUs Figure 27 Do Not Use AC Power Socket 1 with N N PSUs SX6506 3R SX6506 NR 2 Plug the other end of the power cords into grounded outlets at least two or three circuits or use a 3 phase circuit Make sure that the outlets and circuits will not be overloaded Spread out the load over Mellanox Technologies 42 InfiniBand FDR Switch Platform Hardware User Manual Figure 28 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and Gafahr des elektrischen Schocks Risque de choc et de danger energy hazard Entferrnen des Netzsteckers e lectriques Le de branchment The PSUs are all independent elnes Netzteils spannungsfrei d une seule alimentation stabi Um alle Einhieten spannungsfrei lise e ne de branch uniquement Disconnect all power supplies to zu machen sind die Netzstecker qu un module Alimentation ensure a powered down state aller Netzteile zu entfernen Stabilise e Pour isoler comple inside of the switch platform tement le module en cause Il faut de brancher toutes les ali mentations stabilise es 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Any yellow or red status LEDs on any of the management modules is cause for con cern and must be dealt with im
116. s revisions software version SwitchX FW version Switch temperatures Mellanox Technologies 75 gt Xi InfiniBand FDR Switch Platform Hardware User Manual The manager also has the ability to burn new firmware and upgrade software on the switch 5 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It dis covers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS rout ing partitioning to the fabric devices You can view and configure the Subnet Parameters SM via the CLI WebUI Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand amp network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based 5 2 Fabric Inspector Diagnostics Fabric Inspector is a plug amp play software for fee within MLNX OS displaying and fil tering all identified systems and nodes within the fabric Fabric Inspector includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections based on traffic patterns an
117. shold the SwitchX device will auto shutdown upon crossing the Emergency 130 C threshold Mellanox Technologies 85 A A InfiniBand FDR Switch Platform Hardware User Manual Appendix C Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculated for any possible customization as follows Take the weight 01 a chassis with the following FRUs installed e All fans modules All power supplies The weight of the SX6506 chassis configured above is kg To this add the weight of installed FRUs Spines Leafs Leaf blanks Management modules Fill in the Table 18 to calculate the weight of your system Table 18 Switch System Weight Calculation Number of FRUs FRU Type Weight of 1 FRU Total Weight FRU Type Chassis as shipped of Spines 377 of leafs 2 76 of leaf blanks 0 764 of management modules 3 15 Total This total is in kilograms Multiply the total by 2 2 to get the total weight in pounds Mellanox Technologies 86 AIR InfiniBand FDR Switch Platform Hardware User Manual Appendix D Calculating the Power of a Chassis To calculate the power consumption of a chassis add the power of the fans spines leafs and management modules Table 19 Power Consumption of Chassis Parts Part Typical W Maximum W IB FDR l
118. shut down a spine run the following command no power enable lt module gt To shut down a management module run the following command no power enable lt module gt Mellanox Technologies 60 J InfiniBand FDR Switch Platform Hardware User Manual 3 Interfaces 3 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of the IT manager All chassis conditions and management options are available and controllable through the manage ment software either CLI or WebUI Y It is recommended that all of the chassis sub systems be maintained and managed through the management software 3 1 1 Power Supply Unit LEDs Each Power Supply Unit has the following indicator LEDs Figure 51 Power Supply Unit Status Indications m DC AC OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC A The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis Mellanox Technologies 61 J InfiniBand FDR Switch Platform Hardware User Manual Figure 52 PSU Cover On and Off PSU Covers On OOO ma mm mia 55055500 LLN I IE Inn BG 0555005550 I I ES Ooococooon a oooooooo 0050050000 00000000
119. sis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 7 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redundant power or a blank in its place In case of a blank A fastened 8 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 9 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 10 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 11 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with equip When this product is mounted or serviced in a rack special precautions must be taken A ment starting from the bottom to the top 12 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 94 Mellanox Technologies Rev 2 0 13 Equipment Disposal N N Disposal of this equipm
120. tents if any of the shock and tilt stickers have tripped Mellanox Technologies 27 J InfiniBand FDR Switch Platform Hardware User Manual Figure 7 Shock Stickers WARNIN Stickers unaf fected by shipping WARNIN WARNING Stickers showing excessive shock Figure 8 Tilt Sticker 2 7 1 4 Opening the Container 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 2 Open the crate by unscrewing the sides and top of the crate Mellanox Technologies 28 J InfiniBand FDR Switch Platform Hardware User Manual Figure 9 Opening the Container Container screws Y It is highly recommended to have a screw gun or electric screwdriver for this step 3 Unscrew the sides of the crate 4 Remove and put aside the box 5 Visually inspect the chassis make sure that there is no visible damage 3 PSUs are installed for theSX6506 3R chassis and 2 PSUs are installed for the SX6506 NR chassis all 4 fans are installed 6 Remove all protective plastic film from all sides and top of the chassis 7 If you wish you may reduce the weight of the chassis by removing all of the power supply units and fan units and put aside for reinstallation after the chassis is installed in the rack The leafs spines and management modules are shipped separately 22 2 7 1 5 Installing the Shelf 1 Place the ESD mat on the floor where you will be working a
121. ting tus of the chas Red System not operational Diagnostics fail CPU hang HW fail sis Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS 08 No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individ ual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MASTER Off Green if master no power Off if slave this management module is not the master Green Management module is operating as a master Mellanox Technologies 68 InfiniBand FDR Switch Platform Hardware User Manual Figure 57 Management Module LEDs LFANS Master 3 1 5 1 Management Module PSU LED Indicator The management module PSU indicator should be green Table 13 Management Module PSU LED Configurations LED Condition Description Green OK All PSUs are working at correct input and output voltages Red Error One or more of the PSUs for the chassis is bad Check each PSU for a red LED 3 1 5 2 Management Module S FANS LED Indicator The management module S FANS indicator
122. tore non sia sotto tensione 9 Durante i temporali pericolo di scosse elettriche Durante i temporali non effettuare interventi sull apparecchiatura e non collegare o scollegare i cavi 10 Collegamento scollegamento del cavo di rame InfiniBand I cavi di rame InfiniBand sono pesanti e non flessibili Di conseguenza vanno collegati o scollegati con cura dai connettori Per avvertenze istruzioni speciali rivolgersi al produttore di cavi 124 Mellanox Technologies Rev 2 0 11 Montaggio su rack e manutenzione Quando questo prodotto viene montato sottoposto manutenzione su un rack A necessario adottare delle precauzioni speciali per assicurarsi che il sistema resti stabile In generale il rack va riempito con apparecchiature procedendo dal basso verso l alto 12 Installazione delPapparecchiatura N Questa apparecchiatura va installata sostituita e o sottoposta a manutenzione solo da N pp p N personale addestrato e qualificato a Lo smaltimento di questa apparecchiatura va effettuato in conformit con tutte le leggi e le normative nazionali 14 Codici elettrici locali e nazionali AB Questa apparecchiatura va installata in conformit con le norme elettriche locali e N 5 A nazionali 15 Codici di installazione y Questo dispositivo va installato in conformit con l ultima versione dei codici elettrici nazionali del Paese Per il Nord America l apparec
123. ttern of air entering through the spine side and exiting through the connector side QSFP Cable Power Budget Classification All SX6506 QSFP switches are designed for active cables with a max power per module of 2 0W This is power level 2 according to the QSFP Public Specification Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the SX6506 They are 112C port 1 CONSOLE port this is an RS232 connector for connecting to a host machine MGT this is an Ethernet connector e 1 USB port RST reset button Mellanox Technologies 70 J InfiniBand FDR Switch Platform Hardware User Manual Figure 60 Management Module Interfaces 3 4 1 12C This interface is for Debug and Troubleshooting only This interface is for FAEs only 3 4 2 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure 3 4 3 MGT Management The MGT port is an Ethernet port for remote management Any remote terminal con nected to the Ethernet port can then be used to manage the fabric and chassis Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M 1G auto negotiation 4 Initial
124. uellsten Version des National Electrical Code des US National Electrical Code und des Canadian Electrical Code installiert werden AAA 108 Mellanox Technologies Rev 2 0 15 UL und CSA zertifiziertes Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 16 Hoher Ableitstrom WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 17 Installationscodes Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimmung mit den gel tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 18 Verbindung der Ger te untereinander Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal tung berstromschutz Eine leicht zug ngliche Au
125. unt des tinate instal rii permanente numai la punctul final de oprire si nu vor fi folosite pentru depanare si ntre inere 20 Directiva DEEE n conformitate cu Directiva DEEE 2002 96 CE toate de eurile de echipamente elec trice i electronice EEE trebuie colectate separat i nu trebuie eliminate mpreun cu de eurile menajere obi nuite Eliminati acest produs i toate componentele sale n mod responsabil i ecologic Mellanox Technologies 119 Rev 1 3 G 10 Sigurnosna upozorenja za instaliranje Croatian 1 Upute za instaliranje Pa ljivo pro itajte upute za instaliranje prije spajanja opreme na izvor elektri ne energije 2 Tjelesne ozljede uslijed te ine Kako biste sigurno podignuli ovaj proizvod koristite dovoljan broj ljudi lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 Te ka oprema Ova oprema je vrlo te ka i treba se premje tati pomo u mehani kog dizala kako bi se izbjegle ozljede 4 Rizik od strujnog udara Rizik od strujnog udara S uklonjenim modulom ventilatora perima napajanja se mo e pristupiti u otvoru mod ula NEMOJTE umetati alat ili dijelove tijela u otvor modula ventilatora 5 Pregrijavanje y Ovom se opremom ne bi trebalo rukovati u podru jima s temperaturom okoline koja y premasuje najvise preporucene vrijednosti 45 C 113 F Osim toga kako bi se osigu 4 N rao odgovaraju i protok zrak
126. with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 28 Country of Norway Power Restrictions This unit is intended for connection to a TN power system and an IT power system of Norway only 2 2 Environmental and Safety Recommendations The following are Mellanox recommendations V ellanox Technologies Recommended ambient temperature in the System room is 20 5 C Recommended humidity range is 40 15 without condensing 22 InfiniBand FDR Switch Platform Hardware User Manual It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference It is highly recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit lt 2 3 Chassis Package Contents The package includes 1 box containing installation kit parts cable management system power cords 1 shelf e 1 chassis
127. with the following modules installed 4 fans 3 PSUs for the SX6506 3R or 2 PSUs for the SX6506 NR If you wish you may reduce the weight of the chassis by removing all of the power sup ply units and fan units and put aside for reinstallation after the chassis is installed in the rack installation guide Inside the single box is 3 smaller boxes l installation kit box 1 box containing power cords 250v 158 2 0M C14 to C13 USA UL Standard cable management kit Mellanox Technologies 23 J InfiniBand FDR Switch Platform Hardware User Manual 6 cable supporter boxes Before you install your new SX6506 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear ofthe switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis 4 2 4 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes
128. x Technologies 10 InfiniBand FDR Switch Platform Hardware User Manual This symbol indicates a situation that can potentially cause damage to hardware or software Warning This symbol indicates a situation that can potentially cause personal injury and or damage to hardware or software Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies SX System Type SwitchX Switch PR Data Transfer Protocol 1 2 3 4 Ethernet 5 6 7 8 InfiniBand G Module Generation 5 6 7 8 Number of leafs 36 18 12 06 C Data Rate B 40Gb s Ethernet F FDR T FDR10 QDR D DDR Separator P Power Supplies N N N redundant 1 1PSU 2 2PSUs R Chip Generation R SwitchX S SwitchX 2 Mellanox Technologies 11 ey InfiniBand FDR Switch Platform Hardware User Manual 1 Overview This User Manual provides an overview of the SX6506 QSFP Modular InfiniBand Switch Platform known in this document as the chassis or switch and its operational environment Mellanox SX6506 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Per formance Computing HPC and Embedded environments Networks built with the SX6506 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s
129. zzare lo switch come scaffale o piano di lavorol Attenzione un apparecchiatura scorrevole o montata su binari non va utilizzata come scaffale o piano di lavoro I binari non sono progettati per far scorrere e allontanare l unit dal rack Essi sono destinati all installazione permanente solo nel luogo di lavoro e non vengono utilizzati per assistenza e manutenzione 20 Direttiva RAEE Secondo la direttiva RAEE 2002 96 EC tutti 1 rifiuti da apparecchiature elettriche ed elettroniche RAEE vanno raccolti separatamente e non smaltiti nei normali rifiuti domestici Smaltire questo prodotto e tutte le sue parti in modo responsabile e rispettoso dell ambiente LA 21 Limitazioni relative all alimentazione per la Norvegia y N Questa apparecchiatura progettata esclusivamente per il collegamento a un sistema di N alimentazione TN e a un sistema di alimentazione IT 126 Mellanox Technologies Rev 2 0 G 12 Monta G venlik Uyar lar T rkge 1 Montaj Talimatlar Ekipman g kaynag na baglamadan nce t m montaj talimatlar n okuyun 2 A rl k Nedeniyle Fiziksel Yaralanma Bu r n g venli bir ekilde kald rabilmek i in yeterli say da insandan yard m al n lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 A r Ekipman Bu ekipman ok agirdir ve yaralanmalar nlemek i in ekipman n mekanik asans r kullan larak tas nmas gerekir

HW User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents