Home

User Manual

1. G 9 Avertismente privind siguran a la instalare Romanian 1 Instruc iuni de instalare Cititi toate instruc iunile de instalare nainte de a conecta 116 Mellanox Technologies Rev 2 5 2 Accidentare cauzat de greutate Apelati la un num r suficient de persoane pentru a ridica in siguranta acest produs e e i 40 Ibs 40 70 Ibs 70 121 Ibs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Echipament greu Acest echipament este greu si trebuie s fie mutat folosind un dispozitiv mecanic de ridicare pentru a evita producerea de leziuni 4 Risc de soc electric Risc de soc electric Odat ce modulul ventilator este indep rtat pinii electrici sunt accesibili in cavitatea modulului NU introduceti instrumente sau p rti din corp in cavitatea modulului ventilator 5 Temperatur in exces Acest echipament nu trebuie s fie actionat intr o zon unde temperatura ambiant dep seste valoarea maxim recomandat 45 C 113 F n plus pentru a asigura un flux de aer adecvat l sati un spatiu liber de cel putin 8 cm 3 inchi in jurul fantelor de ventilare 6 Suprapunerea cadrului a A N Cadrul nu trebuie s fie su
2. DINA Town D lt 22 TN IT mon gt Mellanox Technologies 9 9 Rev 1 3 G4 Chinese 1 2 lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 4 4 A FRA RA 46 84 AB 22 E Pa RY ERS Eos Eh 45 C 113 F HA A T URRESRURLE AE 8 3 RM WEEE 6 NETS AA 9 2 5 E MRA BSR 100
3. CSA UL CSA 3 16 AWG 1 5 M 6 4 5 125 13 A lt HAR gt 3 1 0 MM 300 250 10 Mellanox Technologies 115 Rev 1 3 17 TOK K 4 18 AN RS232 Ethernet
4. 12 H 15 16 UL
5. lt gt 3 DITA 1 0 V 2 300 DY PVC 250 10 98 Mellanox Technologies Rev 2 5 17 s gt pa 18 MINI Ethernet RS232 UL DP 1 non DP 2 mo 19 333 DYSON NIN 20A 20 PN PN Now 2 21 WEEE 29 WEEE 2002 96 EC
6. 8 9 InfiniBand 114 Mellanox Technologies Rev 2 5 11
7. CHAT 5 45 113 F 8 3 6
8. 250 V 10 16 A ASR BES REE 17 18 Bien A RS232 SEAN ZAMAN RARA UL DP 1 DP 2 A LPS ERRE 20 A 19 BA 102 Mellanox Technologies Rev 2 5 20 WEEE WEEE 2002 96 EC PAREN BSS Tixi EEE 0 8 ARE FE 21 TN RK tA IT SRA Lo G 5 Avertissements de s curit pour l installation French 1 Instructions d installation Veuillez lire la totalit des instructions d installation avant de relier l quipement au secteur 2 Blessures a cause du poids A Pr
9. 70 32 AIT FOW s o Nee de asan a 70 3 3 OSFP Cable PowerBudgetClassification 70 3 4 Management Module Interfaces 70 93 4 d 412655 o o pm yel e ee 71 34 2 CONSOLE nue od pest dest et ARE Aa ol sod 71 3 4 3 MGT 71 SAA USB a mea een A BIER 71 RS 72 Chapter 4 Chassis Power 73 4 1 Power Supply and Spine Board Indicator Status at Power ON 74 Chapter 5 Switch Management Tools 75 5 1 InfiniBand Subnet 76 5 2 Fabric Inspector Diagnostics 76 5 3 Accessing the CPU via the Ethernet Connector 76 5 4 Upgrading Software sta si Hama ala ae AE 76 Chapter 6 Troubleshooting 77 6 1 Power Supply Unit 77 D2 Leat Board an does en ENS NC see ed Ped gu 77 6 3 Management Module 78 6 4 Spine Board se a ARCU RR A 79 6 5 MLNX OS Software 79 Chapter 7 Disassembly and Disposal 80 7 1 DisassemblingtiheChassis 80 7 1 1 RemovingtheChassis
10. 6 Y N 110 PN 45 C 11390 nov 8 3 DAV 7 399 PN 4 bawn 8 MI IN PN 4 9 MID 4 TON MIN MIN pin DITO nio 10 4 PN Mellanox Technologies 97 Rev 1 3 11 Infi
11. 41 2 8 Power Connections 42 2 8 1 Powering Up the Switch Platform 42 2 9 InfiniBand Cable Installation 43 2 9 1 Supported Approved Cables 44 2 9 2 Cable Power Classes 44 2 10 Hot Swap Insertion and Extraction 44 2 10 1 Power Supply Units 45 2 10 2 Leaf Boards 47 2 10 3 Spine BoardS 2 2424 WE 50 2 10 4 Fan Modules 55 2 10 5 Management Module 57 2 10 6 Switch Shut Down Procedures 60 Chapter 3 Interfaces bd awe Lu Ru MERE EIE EE Ed RARE 61 3 1 LED Status Indicators 61 3 1 1 Power Supply Unit LEDs 61 3 1 2 Leaf Board LED Indicators 63 3 1 3 Spine Board LED Indicators 64 3 1 4 Spine Side Panel Display LED Indicators 66 3 1 5 Management Module LED Indicators 68 Mellanox Technologies 3 InfiniBand FDR Switch Platform Hardware User Manual 3 1 6 Port Connector Interfaces
12. 69 Table 14 Management Module S Fan LED Configurations 69 Table 15 Management Module L Fan LED Configurations 69 Table 16 Management Module MASTER LED Configurations 70 Table 17 Switch Specification SX6512 4NIR 82 Table 18 Switch System Weight Calculation 86 Table 19 PowerConsumptionofChassisParts 87 Table 20 OSFPCablePowerConsumption 87 Table 21 Calculating Total Switch System Power Consumption 87 Table 22 InfiniBandosFPConnectorPinout 89 Table 23 Replacement Parts Ordering Numbers 92 Mellanox Technologies 6 InfiniBand FDR Switch Platform Hardware User Manual List of Figures Figure 1 Product Label her poda rn e Me a b e 13 Figure 2 Product Label Tab Location 14 Figure 3 Management Module MAC Address Location 14 Figure 4 SX6512 Switch Views 15 Figure 5 Shelf Installation Kit Parts 27 Figure 6 Chassis Installation Parts 28 Figure Shock Stickers starti nD RES de re nr Si 29 Figure 8 TIU Sticker occas
13. 57 Figure 47 Management Module 58 Figure 48 Intact vs Defected Signal Connectors 58 Figure 49 IntactvsDefectedPowerPinHolders 59 Figure 50 Power Supply Unit Status Indications 61 Figure 51 PSU Cover and Off eu ran ER a 62 Figure 52 Leaf Board Led Indicators 63 Figure 53 Spine Status LEDS 65 Figure 54 Spine Side Panel Display Status Indications 67 Figure 55 Management Module Status Indications 68 Figure 56 Management Module 69 Fig re 57 gt a ence a ERGO RI 10 Figure 58 Top and Bottom Ports 70 Figure 59 Management Module Interfaces 71 Figure 60 Reset Buttons semasi serus de en neue rei nein 72 Figure 61 Spine Side Panel Display Status Indications 74 Figure 62 Management Module Status Indications for Normal Operation 74 Figure 63 InfiniBand Connector Symbol 90 Figure 64 Connector Male and Female Views 91 Mellanox Technologies 8 InfiniBand FDR Switch P
14. 1 3 August 2012 Minor changes due to changes in container 1 2 July 2012 Fixed broken link 1 1 July 2012 Added figure to show use of 2nd type of mechanical lift Added EMI explanation to Appendix Added explanation to N 2 Power Numbers 1 0 May 2012 Initial release About this Manual This manual provides an overview of the SwitchX based SX6512 modular InfiniBand chassis switch and guidelines for its operation Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniBand architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Websites Document Name Description InfiniBand Architecture Specification The InfiniBand Architecture Specification that is provided by Vol 1 Release 1 2 1 IBTA Mellanox Technologies 10 Me RR Fes InfiniBand FDR Switch Platform Hardware User Manual Table 2 Reference Documents and Websites Document Name Description Switch Product Release Notes For possible hardware issues see the switch support product page This reguires a customer support login Look up the rele vant SwitchX based switch system series release note file Mellanox MLNX OS User Manual This document
15. DP 1 y DP 2 UL 4 20 19 20 2002 96 EC WEEE
16. 80 7 1 2 RemovingtheBottomShelf 80 ee 81 Appendix A Specification Data 82 A 1 EMI Certification 84 A2 Approved Cables 84 A3 EMC Certifications ins nv renk te tethered ta icerik ila 84 Appendix Thermal Threshold Definitions 85 Appendix C Calculating the Weight of a Customized Chassis 86 Appendix D Calculating the Power of a Chassis 87 Appendix E Interface 89 Appendix F Replacement Parts Ordering Numbers 92 Appendix G Safety Warnings Multiple Languages 93 G 1 Nordic Countries Notices 4 93 G 2 Installation Safety Warnings English 93 NAY eee 6 G4 Chinese cover Re Rn die 100 G 5 Avertissements de s curit pour l installation French 103 G 6 Installation Sicherheitshinweise German 106 G 7 Advertencias de seguridad de instalaci n Spanish 110 G 8 Russian 113 9 Avertismente privind siguranta la instalare Romanian 116 G 10 Sigurnosna upozo
17. Do not use the handles on the fans for lifting or moving the chassis in the rack Mellanox Technologies 38 InfiniBand FDR Switch Platform Hardware User Manual Figure 21 Raising the Chassis Using a Mechanical Lift y d O 5 3 Remove the two handles from the sides of the chassis 4 Push the chassis into the rack until the faceplate is 20cm 8 from the vertical support 5 Putthe rail slides onto the rails Figure 22 Put on the Rail Slide 6 Slide the chassis further into the rack 7 Slide the chassis all the way into the rack until the faceplate is touching the vertical rack support The caged nuts placed in the last procedure should line up with the holes in the faceplate Mellanox Technologies 39 InfiniBand FDR Switch Platform Hardware User Manual M Holes in the faceplate for mounting on the 8 Screw the 8 screws through the faceplate and into the caged nuts 9 Slide the rail slides to the vertical rails Figure 23 SX6512 Chassis on the Shelf 10 Place the 4 holed spacer on the outside of the vertical support and screw in the 4 screws for each rail slide Use the split lock washers here 11 Remove the two handles from the spine side and save all four handles for future use 12 Ground the switch 13 Install all of the spines leafs and mana
18. InfiniBand FDR Switch Platform Hardware User Manual 18 Battery Replacement A Warning Replace only with UL Recognized battery certified for maximum abnormal charging current not less than 4mA There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispose of used batteries according to the instructions 19 UL Listed and CSA Certified Power Supply Cord For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 20 High Leakage Current A Warning High leakage current Earth connection essential before connecting supply 21 Add GND connection information da Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON
19. S SwitchX 2 Mellanox Technologies 12 InfiniBand FDR Switch Platform Hardware User Manual 1 Overview This User Manual provides an overview of the SX6512 QSFP Modular InfiniBand Switch Platform known in this document as the chassis or switch and its operational environment Mellanox SX6512 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Per formance Computing HPC and Embedded environments Networks built with the SX6512 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 5th generation SwitchX VPI switch device SX6512 systems provide up to 56Gb s full bidirectional bandwidth per port With up to 216 ports in a 9U high form factor these systems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard opera tion within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system The ini tial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configuration Once connected to the network the Sub net Managemen
20. Status Fan Bad Port and UID LEDs Leaf connection LEDs 2 9 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from Mellanox Technologies The Mellanox proprietary QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 See Section 3 1 6 for port numbering If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase 5 must be programmed with management software MLNX OS 3 3 4304 or higher and P To raise link at FDR 56Gb s with 3rd party FDR cables managed switch systems yo unmanaged switch systems need SwitchX firmware version 9 2 6000 or higher Mellanox Technologies 43 m Mi E InfiniBand FDR Switch Platform Hardware User Manual FDR and FDR10 only guaranteed to work with approved Mellanox Cables cables can be inserted or removed with the unit powered on To insert a cable press the connector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 sec onds for lin
21. a Rail slide 0 M 5 Pan head ge screws and lock washers 4 hole washer Handles Screws for the handles Caged nuts gt lt 2 7 1 3 Container Mishandling The container has shock and tilt stickers applied The shock stickers turn red if the con tainer has been mishandled or roughly handled Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped If a shock sticker or more are red or if the tiltwatch sticker indicates more than 40 notify the ship per and Mellanox This on its own does not indicate damage to the contents But be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped Mellanox Technologies 28 InfiniBand FDR Switch Platform Hardware User Manual Figure 7 Shock Stickers WARNIN Stickers unaf fected by shipping WARNIN WARNING Stickers showing excessive shock Figure 8 Tilt Sticker 2 7 1 4 Opening the Container 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 2 Open the crate by unscrewing the sides and top of the crate Mellanox Technologies 29 InfiniBand FDR Switch Platform Hardware User Manual Figure 9 Opening the Container Container screws Y It is highly recommended to have a screw gun or electric screwdriver for this step Unscrew the sides of the cra
22. on page 87 1 7 January 2014 Rearranged document Consolidated installation sections under Chapter 2 Installation on page 19 updated Chapter 1 Overview on page 13 and re ordered sections Added Figure 56 Management Module LEDs on page 69 Updated e Section 1 4 Power Supply Redundancy on page 16 e Section 2 7 1 5 Installing the Shelf on page 30 e Section 2 7 2 Installing the Cable Holder on page 34 FDR note in Section 2 9 InfiniBand Cable Installation on page 43 e Section 2 10 Hot Swap Insertion and Extraction on page 44 Section 7 1 Disassembling the Chassis on page 80 Table 17 Switch Specification SX6512 4N R on page 82 Appendix D Calculating the Power of a Chassis on page 87 Mellanox Technologies 9 m M InfiniBand FDR Switch Platform Hardware User Manual Table 1 Revision History of this User s Manual Revision Date Details 1 6 June 2013 New graphics Changed PSU input power numbers to match the chassis power numbers both are measured at AC voltage not the DC side Added filler panel rearranged parts lists Add Graphic showing the chassis with the filler panels New Safety warnings 1 5 October 2012 Added Warning for handles 1 4 September 2012 Added figure 23 Added information regarding inserting the bottom spine first Insert spines from bottom to top
23. InfiniBand FDR Switch Platform Hardware User Manual 3 1 2 Leaf Board LED Indicators Figure 52 Leaf Board Led Indicators 3 1 2 1 Status LED Each leaf board has a Status LED on the far left ofthe leaf Table 4 shows the leaf status according to the LED condition The Leaf Status indicator LED A has the following LED assignment Table 4 Leaf Status LED LED Condition LED Description Off No power to the Leaf Solid Green Leaf is up and running Flashing Green Leaf is powering up Solid Orange Non fatal error this leaf needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 2 2 Bad Port LED The Port indicator is located on the left side of the leaf The following Bad Port con ditions are possible Table 5 Bad Port LED Configurations LED Configuration Description Off OK all ports are up and running Flashing Orange Error one or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are bad cable bad connection bad connector Mellanox Technologies 63 InfiniBand FDR Switch Platform Hardware User Manual M 3 1 2 3 3 1 2 4 This LED lights up when one or more ports is receiving a symbol error The LED imme diately goes off until the next symbol error is received UID LED Switch Identifier The UID LED is a debug feature that will become av
24. According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 21 Country of Norway Power Restrictions Norway only A This unit is intended for connection to a TN power system and an IT power system of G3 nmay 1 A MANN mr 2 PN A 32 nm 3 1900 19 4 4 A 6 Mellanox Technologies Rev 2 5 5 nio gt PR 0525 222
25. En general en un bastidor los eguipos se deben instalar comenzando desde abajo hacia arriba 12 Instalaci n del equipo La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica A mente de personal capacitado y competente La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales Mellanox Technologies 111 Rev 1 3 14 C digos el ctricos locales nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio 4 males ___ 15 C digos de instalaci n Este dispositivo se debe instalar conforme a la version mas reciente de los c digos el ctricos nacionales del pais en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canada 16 Cable de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificacion CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud minima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2
26. by 43 Figure 28 Power Cord Numbering 2 22 2 2 a cc2rcrrrrracacrrraacrris 45 Figure 29 PSU Locations sia LR ark Laurie 46 Figure 30 Power Supply 46 Figure 31 Leaf Board Numbering 47 Figure 32 Ejector Handle Ausser e eq Saree eae A 48 Figure 33 Lear Release ur mpini i ne a 48 Figure 34 Intact vs Defected Signal Connectors 49 Figure 35 Intact vs Defected Power Pin Holders 49 Figure 36 Management Module Numbering 51 Figure 37 Spine Board Extraction 51 Mellanox Technologies 7 InfiniBand FDR Switch Platform Hardware User Manual Figure 38 Intact vs Defected Mechanics 53 Figure 39 Intact vs Defected Side Signal Connectors 53 Figure 40 Intact vs Defected Middle Signal Connectors 54 Figure 41 Intact vs Defected Power Pin Holders 54 Figure 42 Spine Board Insertion Caution 54 Figure 43 Leaf Fan Locations on the Chassis 55 Figure 44 Leaf Fan Module Extraction 56 Figure 45 Spine Fah Module ii an 56 Figure46 FanStatusLEDontheSpineModule
27. 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 25 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user 26 Do Not Use the Switch as a Shelf or Work Space Y Caution Slide rail mounted equipment is not to be used as a shelfor a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance 27 WEEE Directive X According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 28 Country of Norway Power Restrictions This unit is intended for connection to a TN power system and an IT power system of Norway only 2 2 Environmental and Safety Recommendations The following are Mellanox recommendations V ellanox Technologies Recommended ambient temperature in the System room is 20 5 C Recommended humidity range is 40 15 without condensing 23 InfiniBand FDR Switch Platform Hardware User Manual It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference It is highl
28. AR 29 Figure 9 Opening the Container 30 Figure 10 Placement of Chassis Rack 31 Figure 11 Inserting the Caged Nuts for the Shelf 31 Figure 12 Connect Rail Slide to Rack Vertical support 32 Figure 13 Inserting the Caged Nuts for the Faceplate 32 Figure 14 Chassis on Shelf with Filler Panel 33 Figure 15 Cable Holders 22 2 R34 er ae he 34 Figure 16 Cable Holder Placement 35 Figure 17 Chassis Rails and Rail Slides 36 Figure 18 Installing the Handles ee 37 Figure 19 Screw the Handles Onto the Chassis 37 Figure 20 The Rails are Already Connected Onto the Chassis 38 Figure 21 Raising the Chassis Using a Mechanical Lift 39 Figure 22 Put on the Rail Slide rei RE u ne rk a na 39 Figure 23 SX6512 Chassis on the Shelf 4 5 40 Figure 24 Face Plate Mounting Bolt Locations 40 Figure 25 Ground Connection 41 Figure 26 Multiple Power Inlets Electric Caution Notification 42 Figure 27 pine Module t2 these A le e at
29. Chassis fam ily and switch chip generation Mellanox Technologies 44 e m InfiniBand FDR Switch Platform Hardware User Manual When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions This switch platform supports hot swap capabilities for the parts listed below Power supply units Leafboards Spine boards Leaf fan module Spine fan module Management modules 2 10 1 Power Supply Units The 1000W power supplies deliver 1000W at 48V The input to each of these power sup plies requires 1176 5W in order to output 1000W The 1670W power supplies deliver 1670W at 48V The input to each of these power sup plies requires 1882 4W in order to output 1670W For N 1 configuration PSUs Power Supply Unit can be extracted without bringing down the system For configuration up to half of the PSUs can go down and the system will continue to run The power required to run the switch system is equally divided between all of the work ing PSUs 2 10 1 1 Extracting and Inserting the Power Supply Unit With all of the 1000W power supplies installed the system is in N 1 redundant configu ration With all of the N N 1670W power supplies installed the system is in N N redundant configuration half of th
30. InfiniBand FDR Switch Platform Hardware User Manual 4 Installation in Restricted Access Location This unit is intended for installation in a Restricted Access Location 3 Risk of Electric Shock Risk of Electric Shock LA With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 6 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 7 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 8 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redundant power or a blank in its place In case of a blank A fastened 9 Double Pole Neutral Fusing This system has double pole neutral fusing Remove all power cords before opening the cover of this product or touching any internal parts 10 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform Mellanox Technologies 20 e HEN Infi
31. PSU status indicators will turn off Remove the PSU To insert a PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles Insert the PSU by sliding it into the opening until a slight resistance is felt Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation Insert the power cord into the supply connector on the other side of the chassis Mellanox Technologies 46 InfiniBand FDR Switch Platform Hardware User Manual M 5 Replace the cover over the PSUs The green indicators should light If not extract the PSU and re insert it again 2 10 2 Leaf Boards Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start anew cycle checking through the leafs and spines for the FW versions The leaf boards are numbered from top to bottom with corresponding numbers dis played to the outside of the leafs vertically along the side panel Figure 31 Leaf Board Numbering L1 L2 L3 L4 L11 112 2 10 2 1 Extra
32. Sigurnosna upozorenja za instaliranje Croatian 1 Upute za instaliranje Pa ljivo pro itajte upute za instaliranje prije spajanja opreme na izvor elektri ne energije 2 Tjelesne ozljede uslijed te ine Kako biste sigurno podignuli ovaj proizvod koristite dovoljan broj ljudi lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 Te ka oprema Ova oprema je vrlo te ka i treba se premje tati pomo u mehani kog dizala kako bi se izbjegle ozljede 4 Rizik od strujnog udara Rizik od strujnog udara S uklonjenim modulom ventilatora perima napajanja se mo e pristupiti u otvoru mod ula NEMOJTE umetati alat ili dijelove tijela u otvor modula ventilatora 5 Pregrijavanje y Ovom se opremom ne bi trebalo rukovati u podru jima s temperaturom okoline koja y N prema uje najvi e preporu ene vrijednosti 45 C 113 F Osim toga kako bi se osigu 4 rao odgovaraju i protok zraka omogu ite najmanje 8 cm 3 in a razmaka oko otvora ventilatora 6 Slaganje ku i ta i Ku i te se ne bi trebalo slagati na drugu opremu Ako ku i te padne mo e izazvati tje lesne ozljede i o te enje opreme 120 Mellanox Technologies Rev 2 5 7 Redundantno napajanje Opasnost od elektri ne energije U slu aju praznog prostora za napajanje nemojte rukovati proizvodom ako je pok a Ovaj proizvod uklju uje redundantno napajanje ili prazan pr
33. User Manual Table 22 InfiniBand OSFP Connector Pinout Connector O Pin Signal Description Pin Name Number 30 Vecl 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txln Transmitter Inverted Data Input 38 GND Ground Figure 63 InfiniBand QSFP Connector Symbol J3 and J6 20 GND GND 12 21 Rx2n Rx1n 18 22 Rx2p Rx1p 17 23 GND 16 24 15 Rx4n Rx3n 14 23 Rx4p Rx3p 13 28 GND GND i ModPrsL SDA 12 11 28 IntL SCL 10 29 VECIX Vcc Rx su Veci ResetL 9 31 LPMode ModSelL 3 32 GND GND 7 33 Tap Tx4p 34 Tx3n Tx4n 5 35 GND 36 Tx1p Tx2p 3 Tx2n 2 38 GND GND 1 Mellanox Technologies 90 gt om InfiniBand FDR Switch Platform Hardware User Manual Figure 64 QSFP Connector Male and Female Views Top View into Rear of 18 35 Connector E nn 8 lt y 9 889 229 7 5 8 8 8 5 9 x 9 G38 o 5 O ff lt 4 0 8 5 6 se O a oo 0 lt J 0 1 25 28 F E O 3 5 0 o gt 20 on 5 5 Vs 35 U 2 HP 18 35 View into Front of Cage 8 50 Mellanox Technologies 91 InfiniBand FDR Switch Platform Hardware User Manual Appendix F Replacem
34. bringing down the system When using the 1000W PSUs the mini mum complement of PSUs at start up is 4 and the chassis will continue to run at full capacity with only 3 PSUs 1 4 2 1670W Power Supply Units The 1670W power supply units PSUs deliver 1670W at 48V The AC source to each PSU must be able to provide 1882 4W in order to output 1670W in DC When using the 1670W PSUs the minimum complement of PSUs which allows the chassis to run at full capacity is 2 The 1670W PSUs can implement the following two redundancy options e No redundancy combined mode e N 1 configuration ps redundant mode When supplied from a single 220V power grid the chassis supports up to 2 redundant units Thereby the minimum required number of PSUs equals 2 When supplied by a single 110V power grid the number of redundant power supplies is only 1 configuration grid redundant mode A y N NN redundancy ONLY works with a supply voltage of 220V The chassis PSUs are fed from two power grids for high availability The second power grid can be supplied by any of the following e 8 backup power supply grid generator e a battery backup system any combination of the above Connecting 2 power supplies to one power supply grid and the remaining 2 power supplies to a secondary power supply grid will create N N redundancy This is High Availability Under these conditions should a power grid fail an electric company power failure or
35. contains information regarding configuring and for VPI managing Mellanox Technologies SwitchX switch platforms listing all ofthe commands available through MLNX OS with explanations and examples Conventions Throughout this manual the name SX6512 and the terms chassis and switch are used to describe the 216 port QSFP InfiniBand chassis unless explicitly indicated otherwise The following icons are used throughout this document to indicate information that is important to the user Y This symbol makes recommendations to the user This symbol indicates information that is helpful to the user This symbol indicates a situation that can potentially cause damage to hardware or software Warning This symbol indicates a situation that can potentially cause personal injury and or damage to hardware or software Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies SX System Type SwitchX amp Switch PR Data Transfer Protocol 1 2 3 4 Ethernet 5 6 7 8 InfiniBand G Module Generation 5 6 7 8 Mellanox Technologies 11 NN gt InfiniBand FDR Switch Platform Hardware User Manual Place Field Decoder FF Number of leafs 36 18 12 06 C Data Rate 40Gb s Ethernet F FDR T FDR10 QDR D DDR Separator P Power Supplies NIN redundant 1 1PSU 2 2PSUs R Chip Generation R SwitchX
36. mantenimiento 21 Directiva WEEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera respons able y respetuosa con el medio ambiente 6 8 Rus sian 1 2 AR Tre 40 Ibs 40 70 Ibs 70 121 Ibs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 BO Mellanox Technologies 113 Rev 1 3 4 TOKOM
37. no power enable lt module gt For example to shut down spine 06 run the command below switch master config no power enable S06 2 Push outward on the ejector handles to unlock the ejectors from the chassis 3 Open the ejectors until they are at a 45 degree angle from the module IN N y N Do not use the fan FRU handle to extract the spine board 4 Pull out the module half way through the guiding rails using both ejectors 5 Lock the ejector handles 6 Hold the body of the board on both sides and remove it from the chassis Mellanox Technologies 52 InfiniBand FDR Switch Platform Hardware User Manual 2 10 3 2 Inserting a Spine Board When hot swapping any of the units it is necessary to wait 1 minute after removing y N the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a spine board 1 Check for foreign objects in or mechanical damage to the chassis spine slots 2 Check spine mechanics for any noticeable damage or deviations from the pass crite rion shown in Figure 38 Figure 38 Intact vs Defected Mechanics 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criteria shown in Figure 39 and Figure 40 Figure 39 Intact vs Defected Side Signal Connectors Pass 2
38. of the chassis 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 2 10 4 2 Spine Fan Modules Each spine module has a fan module with two individual fans built in When a fan mod ule is not functioning the Fan Status LED on the spine will light up Figure 45 Spine Fan Module Fan module When a fan module is removed the indicator light will reset Mellanox Technologies 56 InfiniBand FDR Switch Platform Hardware User Manual M Figure 46 Fan Status LED on the Spine Module Fan status LED Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out Inserting the Spine Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location ofthe connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new spine fan module 3 Push the fan module as far as it will go make sure the locking latches engage gt Ifthe Fan LED continues to show red remove the fan module and check the pins on the connector inside of the
39. onayl DP 1 veya DP 2 tipi olmas gerekir Not LPS olmayan devreye aitse A r Ak m Korumas Kolayca eri ilebilecek 20 V Kay tl devre par as a r ak m koruma cihaz n n bina elektrik ebekesinde kurulu olmas gerekir 19 Anahtar Raf veya al ma Alan olarak kullanmay nl Dikkat S rg l rayl ekipman raf veya al ma alan olarak kullan lamaz Raylar niteyi iskeleden uza a kayd rmak i in yap lmam t r Sadece ekipman n son olarak duraca yerdeki kal c montaj i indir servis veya bak m i in kullan lamaz Mellanox Technologies 129 Rev 1 3 20 WEEE Y nergesi WEEE Y nergesi 2002 96 EC uyar nca t m elektrikli ve elektronik ekipman at klar EEE ayr olarak toplanmal ve evsel at klarla birlikte pe at lmamal d r X Bu r n ve t m par alar evreye dost ve sorumlu bir ekilde imha edilmelidir Le 21 Norve G K s tlamalar Bu nite bir TN g sistemine ve sadece Norve in TT g sistemine ba lanmak i indir Sp G 13 Japan VCCI Class A Statement CORBIS 772 Cd COREE XRRR CHATS 5 02 6 9 3 DIE 758 5 416 08 0 3 VCCI A 130 Mellanox Technologies
40. or more of the spine fans is bad User should check individual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MNGI Off Green MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master MNG2 Off Off MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master Mellanox Technologies 67 InfiniBand FDR Switch Platform Hardware User Manual 3 1 5 Management Module LED Indicators Figure 55 Management Module Status Indications The management module LEDs display the switch system operating conditions Table 12 Management LED Display for Normal Operation sti ga Normal LED Condition Description State STATUS Off No Power Green This LED Green System is up and running shows the sta Yellow System warning Attention needed such as overheating tus of the chas Red System not operational Diagnostics fail CPU hang HW fail sis Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operati
41. or not working hot swapping the spine board connected to the master management board will cause the chassis to crash or spine 2 as the management modules are connected to the chassis components Neither the CLI nor the GUI management tools will allow you to shut down spine 1 gt through these spines Extracting Spine Board 1 or 2 Spine board 1 is connected to management board 1 and spine board 2 is connected to management board 2 MNGI management module 1 top MNG2 management module 2 bottom Removing spine 1 causes management module 1 to reset Removing spine 2 causes management module 2 to reset Warning Ifthe spine you want to hot swap is connected to the master management module the management module will reboot when you take out the spine If you have only one management module the chassis will crash Figure 37 Spine Board Extraction Pull the ejector handles out to remove the spine ade ep 15 17 1 21 23 25 27 29 31 33 35 12 14 16 18 2g 22 24 26 28 30 32 34 35 Mellanox Technologies 51 InfiniBand FDR Switch Platform Hardware User Manual If you need to hot swap spine 1 or spine 2 1 Check to see ifthe spine you need to remove is connected to the master management module See Figure 36 A If you have only one management module the chassis will crash a 2 Follow the steps in Section below Extracting Spine Boards Except 1 or 2 1 Run the shut down command
42. si cu o manta izolant din PVC Cordonul de alimentare trebuie sa fie prevazut cu o fi turnat cu putere nominal egal cu 250 V 10 A 17 Curent de scurgere de inalt frecvent Avertisment Curent de scurgere de inalt frecvent Imp m ntarea este esential inainte de a conecta sursa de alimentare 18 Interconectarea unit tilor Cablurile pentru conectarea la unitatea RS232 si la interfetele Ethernet trebuie s fie de tipul DP 1 sau DP 2 certificate UL Not c nd se reg sesc intr un circuit non LPS Protectie la supracurent Un dispozitiv de protectie la supracurent inregistrat in cir cuitul de ramificare usor accesibil si cu o putere nominal egal cu 20 A trebuie s fie integrat in cablajul cladirii 19 Nu utilizati comutatorul ca raft sau spatiu de lucru Atentie Echipamentul montat pe o linie de alunecare sin nu va fi utilizat ca raft sau spatiu de lucru Scopul sinelor nu este de a glisa unitatea de pe rack Acestea sunt des tinate instal rii permanente numai la punctul final de oprire si nu vor fi folosite pentru depanare si intretinere 20 Directiva DEEE n conformitate cu Directiva DEEE 2002 96 CE toate de eurile de echipamente elec trice i electronice EEE trebuie colectate separat si nu trebuie eliminate mpreun cu deseurile menajere obisnuite Eliminati acest produs si toate componentele sale in mod responsabil si ecologic Mellanox Technologies 119 Rev 1 3 G 10
43. spine to make sure that none of them are bent 2 10 5 Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start anew cycle checking through the leafs and spines for the FW versions gt Extracting a Management Module Management modules are located on the leaf side above the leafs There are two places to install the management modules Only one management module is required to run the switch system ellanox Technologies 57 InfiniBand FDR Switch Platform Hardware User Manual Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 47 1 Run the shut down command no power enable lt module gt For example to shut down management module 2 run the command below switch master config no power enable MGMT2 Disconnect all cables connected to the management module Push outward on the ejector handles to unlock the ejectors from the chassis Open the ejectors until they are 45 degrees from the module Pull out the module half way through the guiding rails using the ejector handle Lock the ejector handle aD Hold the body of the board on both sides and remove it from the chassis Y N The board is short therefore do not let go of it while sliding it out Figure 47 Managemen
44. the PSU grounds It is absolutely necessary to connect the grounding post Make sure the connections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground con nection between the chassis of the switch and the rack Warning System grounding must comply with local electrical code Figure 25 Ground Connection ESD connector Ground Post Mellanox Technologies 41 UU InfiniBand FDR Switch Platform Hardware User Manual M 2 8 2 8 1 Power Connections The switch includes integrated hot swap power supplies which support up to 4 load shar ing 1000W or 1670W supply units The slots for the power supply units PSUs are on the spine side The left side has odd numbered PSUs and the right side has even num bered PSUs Each PSU has a dedicated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be stan dard 3 wire AC power cords including a safety ground Powering Up the Switch Platform A Make sure that the power cords are compatible with your outlets Power cords for dif ferent countries can be ordered from Mellanox The chassis in N 1 mode must be started with a full complement of PSUs thereafter it can run on one less than the total number of PSUs This fin
45. the leaf board 1 Check for foreign objects in or mechanical damage to the chassis leaf slots 2 Check leaf mechanics for any noticeable damage Mellanox Technologies 48 InfiniBand FDR Switch Platform Hardware User Manual 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 34 Figure 34 Intact vs Defected Signal Connectors m Fail 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion show in Figure 35 Figure 35 Intact vs Defected Power Pin Holders 5 Start with the ejector handle fully open that is at 45 degrees to the front panel of the leaf 6 Holding the leaf by its sides carefully set the leaf board into the chassis Mellanox Technologies 49 InfiniBand FDR Switch Platform Hardware User Manual 7 Applying equal pressure on both sides of the leaf board slowly slide the board into the chassis until the ejector handle reaches the vertical bar AN N Do not apply excessive force to slide in the leaf board If you feel resistance remove the leaf board and double check both the chassis and leaf for any damage 8 Catch the hook onto the vertical bar ofthe chassis and push the ejector handle shut 9 Lock the ejector handle onto the board 2 10 3 Spine Boards Do NOT mix replacement parts bas
46. trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 16 Cordon d alimentation UL Listed et certifi CSA Pour le branchement lectrique en Am rique du Nord utiliser un cordon d alimenta tion UL Listed et CSA Certified 3 conducteurs calibre 16 AWG avec une prise moul e 125 V 13 A faisant au moins 1 5 m de long six pieds et au plus 4 5 m Pour le branchement lectrique en Europe utiliser un cordon d alimentation au format international harmonis marqu lt HAR gt 3 conducteurs d au moins 1 mm2 de sec tion 300 V avec une gaine isolante en PVC Le cordon doit avoir une prise moul e 250 V 10 A 17 Courant de fuite lev Avertissement courant de fuite lev une connexion la terre est indispensable avant de brancher l alimentation Mellanox Technologies 105 Rev 1 3 18 Interconnexion des unites A Les c bles de connexion aux interfaces RS232 et Ethernet de l appareil doivent tre certifi UL de type DP 1 ou DP 2 Note en cas d installation sur un circuit dont la puissance n est pas limit e Protection contre les surintensit s le c blage de l immeuble doit int grer un dispositif certifi de protection contre les surintensites calibre a 20 A et aisement accessible 19 Ne pas utiliser comme tag re ou plan de travail Attention un quipemen
47. ventilaci n 6 Apilamiento del chasis A IN Los chasis no se deben apilar sobre otros eguipos La caida del chasis podria causar lesiones corporales asi como da os al equipo 110 Mellanox Technologies Rev 2 5 7 Conexi n redundante de fuente de alimentaci n peligro de descarga Si se dispone de una fuente de alimentaci n vacia no utilizar el producto si su tapa Este producto incluye una fuente de alimentaci n redundante o en su lugar una vacia A esta quitada o no esta bien cerrada 8 Tomas de alimentaci n multiples Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes Desconecte todas las fuentes de alimentaci n para asegurar que no haya corriente alguna dentro de la plataforma de conmutaci n gt 9 Al haber rayos peligro de descarga No utilizar el equipo ni conectar o desconectar cables durante periodos de actividad de LA rayos 10 Cable de conexi n y desconexi n InfiniBand de cobre Dado que los cables de cobre InfiniBand son pesados y no son flexibles su conexi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver adver tencias o instrucciones especiales consultar al fabricante del cable 11 Montaje y mantenimiento del bastidor Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable
48. voyez assez de personnel pour soulever ce produit en toute s curit lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 quipement lourd Cet quipement est lourd et doit tre d plac avec un systeme de levage m canique pour viter les blessures Mellanox Technologies 103 Rev 1 3 4 Danger d lectrocution AN Danger d electrocution 4 Lorsque le module de ventilation est retir les broches d alimentation sont expos es A dans l emplacement du module NE PAS ins rer d outils ou la main dans l emplacement du module 5 Surchauffe Cet quipement ne doit pas amp tre en service dans un local dont la temp rature d passe le maximum recommand de 45 C 113 F En outre et pour garantir une circulation y d air correcte laisser un espace d au moins 8 cm 3 autour des orifices de ventilation 6 Ch ssis empil sur d autres quipements Le ch ssis ne doit pas tre empil sur d autres quipements S il tombe il peut endom mager l quipement ou entra ner des blessures 7 Connexion de l alimentation redondante danger d lectrocution Ce produit est quip d une alimentation redondante ou d un cache si elle est absente Dans ce dernier cas ne pas faire fonctionner le produit si le cache est retir ou mal fix gt 8 Plusieurs prises d alimentation Risque et danger d lectrocution Les alimentations sont toutes ind
49. 1 Overview wsdl sss ster ei COs lele ERE 13 1 1 Product Information 2 13 1 1 1 Serial Number and Product Version Information 13 1 1 2 Management Module MAC 14 1 1 3 Product Physical Specifications and Power 14 1 27 Features List near a Roe er Dot a au Da m ears 16 1 3 InfiniBand and FDR10 Overview 16 1 4 Power Supply Redundancy 16 1 4 1 1000W Power Supply Units 17 1 4 2 1670W Power Supply Units 17 Chapter 2 Installation ss seke ea sles 19 2 1 Installation Safety Warnings 19 2 2 Environmental and Safety Recommendations 23 2 3 Chassis Package 24 24 Leaf Package 25 2 5 Spine Package Contents se ar anne 25 2 6 Management PackageContenis 5 6 25 2 7 1 5 25 2 7 1 ESD Connection 26 2 7 2 InstallingtheCableHolder 34 2 7 3 Installing the Chassis 36 2 7 4 Ground Connections
50. 1 for each management module that get connected to Ethernet switches These switches must be configured to 10 100M auto negotiation 5 The SX6512 switch comes standard with a management software module for chassis management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX amp based managed switch systems management modules MLNX OS includes a CLI WebUI SNMP and chassis management features for software and IB management software OpenSM You can get more information via the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual The managed switch system includes the following software components Embedded Subnet Manager SM Chassis manager and system BIST SNMP agent 3rd party tool integration GUI Remote logging e SSH telnet e Secured access in band and out band IPv4 IPv6 network stack The chassis manager will give the user access to Switch temperatures Power supply voltages Fanunit information Power unit information Flash memory Monitoring of AC power to the PSUs DC power out from the PSUs chassis failures querying for switch serial numbers revisions software version SwitchX FW version Switch temperatures Mellanox Technologies 75 O om InfiniBand FDR Switch Platform Hardware User Manual The manager also has the ability to burn new firmware and upgrade software on the
51. 5 120 8 Chassis fans 163 3 188 2 Management module 17 6 25 9 Table 19 assumes the QSFP cable power consumption specified in Table 20 Please note that QSFP module power is related to the module itself and is not referenced to AC plane Table 20 QSFP Cable Power Consumption QSFP Cable Type Typical W Maximum W Active 1 5 1 7 Optical 1 7 2 0 To calculate the total power consumption of the switch system at a certain configuration fill out Table 21 in the following manner 1 Inthe Power column fill in the desired typical or maximum values Be careful not to mix FDR and FDR10 power numbers for the particular calculation 2 In the Quantity column fill in the desired configuration Please note that the number of fans is not configurable and is always 1 3 Get the total power per each FRU by multiplying values in adjusted columns in the same row 4 Get the total switch system power consumption system power by summing up all multiplication products Table 21 Calculating Total Switch System Power Consumption Part Quantity Power Quantity x Power IB FDR or FDR10 leaf with passive cables IB FDR or FDR10 leaf with active cables Mellanox Technologies 87 InfiniBand FDR Switch Platform Hardware User Manual Table 21 Calculating Total Switch System Power Consumption Part Quantity Power Quantity x Power IB FDR or FDR10 leaf with optical cables
52. 94 III 1 1 2 Management Module MAC Each management module has a label with its MAC address See Figure 3 for the loca tion of this label Figure 3 Management Module MAC Address Location MAC address location RST PSU SFANS LFANS Mester mE MAC A I 1 1 3 Product Physical Specifications and Power The switch ships in a minimum base configuration plus additional modules depending on the chosen customer configuration Optional modules included Leafboards Management modules Spine boards Mellanox Technologies 14 InfiniBand FDR Switch Platform Hardware User Manual The following figure shows the connector and spine sides fully populated Figure 4 SX6512 Switch Views Spine Side FRONT SIDE ORCA SX6512 Up to 4 power supply units 6 spine modules 4 leaf fan modules 12 leaf modules 2 management modules 1 Mellanox Technologies 5 InfiniBand FDR Switch Platform Hardware User Manual 1 2 Features List 216 FDR 56Gb s InfiniBand ports in a 9U switch 24 2 Tb s aggregate data switching capacity with ultra low latency IBTA 1 3 and 1 2 1 compliant SDR DDR QDR FDR10 FDR link speed e NN power supply Congestion control Adaptive routing Port mirroring Chassis High Availability sMB High Availability 1 3 InfiniBand FDR and FDR10 Overview The Mellanox SX6512 switch system supports FDR standard InfiniBand data rate where each lane
53. EE Directive 2002 96 EC m ssen alle elektrischen und elektronischen Abfallger te EEE separat gesammelt und nichit mit normalem Haushaltsm ll ensorgt werden Dieses Produkt und alle seine Teile in verantwortungsvoller und umweltfreundlicher Art und Weise entsorgen Mellanox Technologies 109 Rev 1 3 G7 Advertencias de seguridad de instalaci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalacion 2 Lesi n corporal a causa de peso Recurra a suficientes personas para levantar este producto sin A se lt 40 Ibs 40 70 lbs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Equipos pesados mec nico para evitar lesiones 4 Dado que el equipo es pesado se debe mover nicamente mediante un elevador 4 Riesgo de descarga el ctrica Riesgo de descarga el ctrica Con el m dulo del ventilador quitado se obtiene acceso a las clavijas de alimentaci n desde dentro de la cavidad del m dulo NO introducir herramientas ni partes del cuerpo en la cavidad del m dulo del ventila dor 5 Sobretemperatura No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adec uada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de
54. IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 O Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo Bridge XQ CloudX logo Connect IB ConnectX CoolBox CORE Direct GPUDirect InfiniHost InfiniScale Kotura Kotura logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox ScalableHPC Mellanox Connect Accelerate Outperform logo Mellanox Virtual Modular Switch MetroDX MetroX MLNX OS Open Ethernet logo PhyX SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd Accelio CyPUTM FPGADirect HPC XTM InfiniBridgeTM LinkX Mellanox Care Mellanox CloudXTM Mellanox Multi Host Mellanox NEOTM Mellanox PeerDirect Mellanox Socket Direct Mellanox Spectrum NVMeDirectTM StPUTM Spectrum logo Switch IBTM Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3835 InfiniBand FDR Switch Platform Hardware User Manual Table of Contents Revision History co a es 9 About this Manual Din 10 Chapter
55. Ibs 40 70 lbs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs product contains printed circuit boards cables and batteries According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be Ad collected separately and disposed of according to the directive P This product and all of its parts are NOT to be disposed of with household waste This Go to the Mellanox website for detailed instructions for disassembly of the FRUs and chassis according to the WEEE Directive 10 Dispose of these pieces in a legal and environmentally friendly way 7 1 1 Removing the Chassis 1 2 CP Remove the screws connecting the upper brackets to the rack Remove the screws holding the chassis to the rack These screws are located in the faceplate on the spine side of the chassis Remove the upper brackets from the chassis Remove the lock down bars Dispose of the Chassis in a legal and environmental way Go to the Mellanox website for detailed instructions for disassembly of the chassis according to the WEEE Directive 7 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack Mellanox Technologies 80 ei InfiniBand FDR Switch Platform Hardware User Manual 2 Remove the filler panels 3 Remove the shelf 4 Removeall ofthe caged nuts 7 2 Disposal According to the WEEE Directive 2002 96 EC al
56. Mellanox TECHNOLOGIES Connect Accelerate Outperform 216 Port InfiniBand FDR SwitchX Switch Platform Hardware User Manual PN MSX6512 4R MSX6512 NR Rev 2 5 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN
57. Mellanox Technologies Rev 2 5 7 EXEREERW BEER AA EEE Y MA AR N MARES MAREE ABR 8 BER he SH tek LA PSU 9 NERNERER LA FH ZI NE FAR 10 InfiniBand HAE RK IN LN InfiniBand HE REA NANE AKA DEERE L K ER BR ZEE BR MERE 12 ARES Mellanox Technologies 101 14 15 UL IZA CSA BERR PEAS UL IAM CSA 29 8 82888 16 AWG 125 15 4 5 2 BRR BEA ARH AA lt HAR gt FR 182888 EKRANE 1 0 EEA 300 V KA PVC 08 ER
58. Multiple Languages G 1 Nordic Countries Notices Finland Laite on liitett v suojakoskettimilla varustettuun pistorasiaan Norway Apparatet m tilkoples jordet stikkontakt Sweden Apparaten skall anslutas till jordat uttag G2 Installation Safety Warnings English 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product 3 Heavy Eguipment This eguipment is heavy and should be moved using a mechanical lift to avoid injuries 4 Risk of Electric Shock Risk of Electric Shock With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 5 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper allow at least 8cm 3 inches of clearance around the ventilation openings Mellanox Technologies 93 Rev 1 3 6 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 7 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redunda
59. Spine FDR or FDR10 Chassis fans 1 Management module Total power N N redundancy ONLY works with a supply voltage of 220V Mellanox Technologies 88 InfiniBand FDR Switch Platform Hardware User Manual Appendix E QSFP Interface M Table 22 InfiniBand QSFP Connector Pinout Connector Een A da Xi Pin Name Signal Description 1 GND Ground 2 Tx2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 5 Tx4n Transmitter Inverted Data Input 6 Tx4p o Transmitter Non Inverted Data Input 7 GND Ground 8 ModSelL Module Select 9 ResetL Module Reset 10 Rx 3 3 V Power supply receiver 11 SCL 2 wire serial interface clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxln Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Tx 3 3 V Power supply transmitter ellanox Technologies 89 InfiniBand FDR Switch Platform Hardware
60. The spine fan indicator LED has the following LED assignment Table 8 Spine Fan Status LED LED Color LED Description Solid Green Spine Fan is OK Flashing Green Spine Fan needs replacing Mellanox Technologies 65 e MAN InfiniBand FDR Switch Platform Hardware User Manual M 3 1 3 3 3 1 3 4 3 1 3 5 Table 8 Spine Fan Status LED LED Color LED Description Solid Yellow One or more of the fans in this spine is not working Each spine has two fans in the fan module Spine to Leaf IB Connection Status LEDs The leaf connection status on cach spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per leaf per spine and a maximum of 3 LEDs per leaf These LEDs indicate a valid connection between a leaf and a spine Table 9 shows the leaf to spine status according to the LED condition Table 9 Spine to Leaf IB Link Status LED Condition LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Solid Orange Physical connection Bad Port LED 4 The Bad Port indicator is located on the left side of the spine The following Bad Port conditions are possible Table 10 Bad Port LED Configurations LED Condition Description Off OK No ports have received symbol errors recently Flashing Orange Error One or possibly more ports has just received a s
61. U sa m REA 6 2 Leaf Board Issue 2 The power LED for the Leaf board is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall the Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one 4 Should any of the boards shut down due to over temperature wait 5 minutes and then follow the procedure starting with step 2 Issue 3 The Physical link LED for the InfiniBand connector does not come 1 Check that both ends of the cable are connected 2 Checkthat the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable Issue 4 The Activity indication does not come on Check that the Subnet Manager has been started Mellanox Technologies 77 ei e InfiniBand FDR Switch Platform Hardware User Manual 6 3 Management Module Issue 1 Yellow Status LED for the Chassis on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Reset the master management module by pushing t
62. ZS CISPR 22 class A Japan VCCI Class A Australia C Tick certificate EU IEC 60068 2 64 Random Vibration Environmental EU IEC 60068 2 29 Shocks Type I EU IEC 60068 2 32 Fall Test 76 4 dB A Acoustic ISO 7779 ETS 300 753 ellanox Technologies 83 InfiniBand FDR Switch Platform Hardware User Manual Table 17 Switch Specification SX6512 4N R Physical Shock and ETSIEN 300 019 2 2 1999 09 Vibration Scalability and Performance Switching capacity 24 2 Tb s Switching performance Simultaneous wire speed any port to any port Addressing 48K unicast addresses max per subnet and 15 5K multicast addresses per subnet Leafs Up to 12 leafs 18 ports each De Spines Up to 6 spine boards All spines are needed for non blocking configuration Capacity Management modules 2 available 1 required for operation Management CPU PowerPC 460EX Power supplies 4 power supplies standard Leaf fans 4 Spine fans 3 A 1 EMI Certification EMI certification on the fully populated chassis was performed with the chassis installed in a closed two door rack using the chassis installation kit supplied by Mellanox Tech nologies A 2 Approved Cables For a list of all approved cables see http www mellanox com related docs user manuals Mellanox approved cables pdf A 3 EMC Certifications The list of approved certifications per chassis in differ
63. ailable to customers in the near future For details please contact Mellanox Technologies support Leaf Board Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port Each port has a single 2 color LED Table 6 shows the link status according to the LED condition Table 6 Connector Physical and Logical Link Indications LED Condition LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed is proportional to data transfer speed Solid Orange Physical link up Flashing Orange A problem with the physical link The LED indicator corresponding to each data port will light orange when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a log ical connection is made the LED will change to green When data is being transferred the light will blink green The switch does not provide a visual means to indicate the port speed configuration ZA SDR DDR ODR or FDR and or the link width 1X or 4X The speed and link ad width configurations can be retrieved using management software Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has the following LEDs One status LED for the spine health e One sta
64. al PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection When using the 1670W PSUS the minimum complement of power supply units to start the chassis is 2 and the chassis will continue to run at full capacity with only 2 PSUs The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot com mand in the CLI or reboot through the Web GUI The power system will divide the current consumption by the number of working PSUs Should one of the PSUS fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of current so that the total current is always divided by the number of working PSUs 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cords into grounded outlets m A N Make sure that the outlets and circuits will not be overloaded Spread out the load over 7 N OC at least two or three circuits or use a 3 phase circuit Figure 26 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and Gafahr des el
65. are and firmware updates are available to the user from the Mellanox Sup port website Copy the update to a known location on a Remote server within your LAN Use the CLI or the GUI in order to perform the Software upgrade For further informa tion please refer to the MLNX OS user manual earlier version than the minimum that the new version of the software can work with then the chassis management system may require up to 45 minutes to update all of the Da FW images in all ofthe leafs and spines If MLNX OS is updated and the FW image in the leafs and spines of the chassis is an Mellanox Technologies 76 InfiniBand FDR Switch Platform Hardware User Manual 6 Troubleshooting 6 1 Power Supply Unit A If the Power supplies cannot supply enough power the management module may shut down the some leafs As each PSU is plugged in make sure that green power LEDs on PSU comes on Issue 1 Ifthe AC power LED is off Check that the power cable is the correct power cable for your country Check that the power cable is plugged into a working outlet Check that the power cable has a voltage within the range of 100 240 volts AC Remove and reinstall the power cable Check the circuit breakers to be sure that the breaker has not tripped Check that the power cable is good Replace the power cable Ifthe AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PS
66. assis with less than all PSUs installed Mellanox Technologies 73 InfiniBand FDR Switch Platform Hardware User Manual M 4 1 Power Supply and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes As the power is turned on you should observe the following conditions for normal oper ation 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the Spine LEDs and make sure they coincide with Figure 61 Figure 61 Spine Side Panel Display Status Indications 1 E STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER 5 Check the Management Module LEDs and make sure they coincide with Figure 62 Figure 62 Management Module Status Indications for Normal Operation rs Do ES e IN ellanox Technologies 74 Re InfiniBand FDR Switch Platform Hardware User Manual 5 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via MLNX OS There are 2 Ethernet ports
67. ation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 2 10 3 on page 50 5 Make sure that the spine and the Leafs both have the same version of FW 6 Reburn the FW and remove and reinstall the spine 7 Replace the spine board 6 5 MLNX OS Software For more detailed instructions concerning MLNX OS software see the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual Mellanox Technologies 79 InfiniBand FDR Switch Platform Hardware User Manual 7 Disassembiy and Disposal 7 1 Disassembling the Chassis MES ds en Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Loosen all locking screws for the leafs spines and management modules in the chas sis Remove all leafs 7 Remove all spines 8 Remove all management modules 9 Use enough people to safely lift this product LA E Li lt 40
68. blackout for example power grid High Availability will continue to keep the chassis running at full capac ity through the secondary or backup power supply grid With N N optional PSU grid redundancy the chassis can run on 1 2 of the full complement of PSUs N N allows the chassis to run on 2 PSUs supplied from one power grid while 2 are connected to a second power grid A With power grid A charged with current and power grid B not charged there is only grid redundancy and not PSU redundancy Mellanox Technologies 17 gt gt o InfiniBand FDR Switch Platform Hardware User Manual When the power drops below the required minimum due to power supply failure MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot command in the CLI or reboot through the WebUI The form is identical between the two PSU types while the 1000W PSU weighs 0 3kg more than the 1670W PSU Table 3 OPNs for Power Supply Units PSU AOA OPN Wattage Description MTP005001 1000W Supplies N 1 redundancy MTP006002 1670W Supplies N N redundancy for all switch chassis at 220 Volts Mellanox Technologies 18 InfiniBand FDR Switch Platform Hardware User Manual 2 Installation 4 This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertica
69. bles Electric cable ties are not recommended m2 Mellanox Technologies 35 InfiniBand FDR Switch Platform Hardware User Manual 2 7 3 Installing the Chassis Figure 17 Chassis Rails and Rail Slides Rail slides y N Do not use the handles on the fans for lifting or moving the chassis in the rack 1 Screw the handles onto the chassis Use the 8 allen head screws and the allen wrench provided Two handles go on the connector side and two go on the sides of the chas sis Do not use the locknuts They are extras not needed for this installation Mellanox Technologies 36 InfiniBand FDR Switch Platform Hardware User Manual Figure 18 Installing the Handles Do not use the locknuts They are extras not needed for this installation Figure 19 Screw the Handles Onto the Chassis These handles get installed here Do not use the handles on the fans for lifting or moving the chassis in the rack 7 Mellanox Technologies 3 InfiniBand FDR Switch Platform Hardware User Manual Figure 20 The Rails are Already Connected Onto the Chassis lt 40 lbs 40 70 lbs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 2 Lift the chassis and slide it onto the shelf Use a mechanical lift or enough people to safely lift the chassis A full chassis weighs 120 kgs an empty chassis weighs 50 kgs
70. ck you will be using to connect the shelf brackets Check that both sides of the shelf are at the same level in the rack Figure 11 Inserting the Caged Nuts for the Shelf Top of shelf Bottom of shelf Do not use this hole for the shelf This hole is for the filler panel 4 Insert 2 caged nuts for the filler panel 5 Using two of the bolts for each rail slide install the rail slides onto the rack Check that both sides of the switch left and right are the same level in the rack Mellanox Technologies 31 lt _ lt om InfiniBand FDR Switch Platform Hardware User Manual Figure 12 Connect Rail Slide to Rack Vertical support 6 Place the four bolts for the caged nuts within reach 7 Putthe shelfinto place and screw the bolts into the nuts from step 3 8 Extend the rail slides to the rack vertical support and screw them to the rack using the 2 hole washers and lock washers See Figure 12 9 Tighten all of the screws to 7 7 Nm or 68 pound inches 10 Insert 8 caged nuts for the faceplate in the exact locations shown in Figure 13 Inserting the Caged Nuts for the Faceplate 11 Install the filler panel directly below the shelf Figure 13 Inserting the Caged Nuts for the Faceplate Leaf Side Spine Side Place the caged nuts Place the here caged nuts here Shelf Mellanox Technologies 32 InfiniBand FDR Switch Pla
71. como minimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 17 Alta corriente de fuga ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 18 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pais en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del Codigo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 19 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una corriente nominal de 20 A 112 Mellanox Technologies Rev 2 5 20 No utilizar el conmutador como estante ni como espacio de trabajo Cuidado Equipos montados en deslizadores o rieles no se deben utilizar como estantes ni como espacio de trabajo La finalidad de los rieles no es deslizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o
72. connector This interface is USB 2 0 Mellanox Technologies 71 InfiniBand FDR Switch Platform Hardware User Manual 3 4 5 Reset RST The Reset button resets the chassis management module when the button is pushed When the button is held down for 15 seconds the management module is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip This button resets the CPU of the management module A quick push of this button per forms this reset When the reset button is pushed on the master management module this management module is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin 2 Mellanox Technologies 7 InfiniBand FDR Switch Platform Hardware User Manual 4 Chassis Power Up A A N Before starting any procedure on the SX6512 system put an ESD prevention wrist Y strap on your wrist and connect to the SX6512 chassis With N 1 PSU redundancy the chassis
73. cting a Leaf Board Each leaf board has a ejector handle that locks the board in place and serves as a lever for seating or extracting see Figure 32 1 Run the shut down command no power enable lt module gt For example to shut down leaf 16 run the command below switch master config no power enable L16 2 Disconnect all cables connected to the leaf 3 Push the ejector handle to unlock the ejector from the chassis 4 Open the ejector until it is 45 degrees from the leaf ellanox Technologies 47 ad A InfiniBand FDR Switch Platform Hardware User Manual Figure 32 Ejector Handle Ejector handle gt gt Pull the leaf out of the chassis Push the ejec tor handle 5 Pull out the module half way through the guiding rails using the ejector handle 6 Lock the ejector handle 7 Hold the body of the leaf on both sides and remove it from the chassis The board is short therefore do not let go of it while sliding it out Figure 33 Leaf Release Push here to release the ejector handle Pull the ejector handle to remove the leaf c UP 2 X A A vo vo Vu UA Vs UA Vs 2 10 2 2 Inserting a Leaf Board When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert
74. d for any possible customization as follows Take the weight of a chassis with the following FRUs installed e All fans modules All power supplies The weight of the SX6512 chassis configured above is kg To this add the weight of installed FRUs Spines Leafs Leaf blanks Management modules Fill in the Table 18 to calculate the weight of your system Table 18 Switch System Weight Calculation Number of FRUs FRU Type Weight of 1 FRU Total Weight FRU Type Chassis as shipped of Spines 377 of leafs 2 76 011081 blanks 0 764 of management modules 3 15 Total This total is in kilograms Multiply the total by 2 2 to get the total weight in pounds Mellanox Technologies 86 E InfiniBand FDR Switch Platform Hardware User Manual Appendix D Calculating the Power of a Chassis To calculate the power consumption of a chassis add the power of the fans spines leafs and management modules Table 19 Power Consumption of Chassis Parts Part Typical W Maximum W IB FDR leaf with passive cables 109 5 111 4 IB FDR leaf with 18 active cables 150 6 157 9 IB FDR leaf with 18 optical cables 156 1 166 2 Spine FDR 128 6 135 2 IB FDR10 leaf with passive cables 84 4 97 1 IB FDR10 leaf with 18 active cables 125 5 143 6 IB FDR10 leaf with 18 optical cables 131 0 151 8 Spine FDR10 103
75. dule is bad Check the spines for a flashing green fan LED 3 1 5 3 Management Module L FANS LED Indicator The management module L FANS indicator should be green Table 15 Management Module L Fan LED Configurations LED Condition Description Green OK All Chassis Leaf fans are working at configured speed Red Error One or more of the leaf fans for the chassis is bad Check each chassis fan for a red LED Mellanox Technologies 69 Xi InfiniBand FDR Switch Platform Hardware User Manual 3 1 5 4 Management Module MASTER LED Indicator Table 16 Management Module MASTER LED Configurations LED Condition Description Green This management module is the master Off This management module is the slave 3 1 6 Port Connector Interfaces 3 2 3 3 3 4 The connector side of the switch has 12 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 57 The bottom row ports are flipped from the top row See Figure 58 Figure 57 Port Numbering Figure 58 Top and Bottom Ports Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side QSFP Cable Power Budget Classification All SX6512 QSFP switches are designed for active cables with a max power per module of 2 0W This is p
76. dule is not functioning the status LED on the fan will light up Each spine has an individual fan module that contains two individual fans Should a sin gle fan fail the Fan Status LED on the spine and the S Fan LED on the management module will light indicating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs 2 10 4 1 Leaf Fan Module There are 4 leaf fan modules found on the spine side Figure 43 Leaf Fan Locations on the Chassis Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 44 2 Slowly pull out the fan module using the handle Mellanox Technologies 55 InfiniBand FDR Switch Platform Hardware User Manual Figure 44 Leaf Fan Module Extraction Handle Latch Release Inserting the Leaf Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module If the fan module stops before it goes in all of the way it is inserted incorrectly or it may be for the other side
77. dulo della ventola 5 Temperatura eccessiva AN Questa apparecchiatura non va utilizzata in un area con una temperatura ambiente y V superiore a quella massima consigliata 45 C 113 F Inoltre per assicurare un N flusso d aria adeguato lasciare almeno 8 cm 3 pollici di spazio attorno alle aperture di ventilazione 6 Impilare lo chassis Ku i te se ne bi trebalo slagati na drugu opremu Ako ku i te padne mo e izazvati tje lesne ozljede i o te enje opreme 7 Collegamento di alimentazione ridondante Pericoli elettrici installato di uno spazio vuoto Qualora l alimentatore non sia installato non utilizzare l Questo prodotto dotato di un alimentatore ridondante o qualora esso non sia il prodotto con il coperchio rimosso o non fissato correttamente 8 Prese di alimentazione multiple Rischio e pericolo di scosse elettriche i Gli alimentatori sono tutti indipendenti Scollegare tutti gli alimentatori per assicurarsi che il commutatore non sia sotto tensione 9 Durante i temporali pericolo di scosse elettriche Durante i temporali non effettuare interventi sull apparecchiatura e non collegare scollegare i cavi 10 Collegamento scollegamento del cavo di rame InfiniBand I cavi di rame InfiniBand sono pesanti e non flessibili Di conseguenza vanno collegati o scollegati con cura dai connettori Per avvertenze istruzioni speciali rivolgersi al produttore di cavi 124 Mellano
78. e PSUs may be extracted without bringing down the system gt To extract a PSU 1 Determine which AC connector on the connector side of the chassis corresponds to the defective PSU 2 Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering Figure 28 Power Cord Numbering Mellanox Technologies 45 InfiniBand FDR Switch Platform Hardware User Manual Figure 29 PSU Locations gt PIA A Fan 1 e AM e so eo e Mellanox 9 se es ee O Fan 3 Fan LES O 050000000 000550000 550055060 000550000 5500550060 000550000 050050000 0005560000 095559906 000905000 4 A 000900000 0090005500 pal 000500000 LI On the spine side of the chassis remove the cover to the power supply unit for the non working PSU Odd numbered PSUs are on the left side and even numbered PSUs are on the right side facing the spines There are four phillips head screws for the cover plate Figure 30 Power Supply e DE 5 AC Latch DD SM Handle Grasping the handle with one hand push the black latch release while pulling the handle outward As the PSU unseats the
79. e in the management board If you feel resistance Y N remove the management and double check both the chassis and management for any damage 8 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handles onto the board the master management module CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and slave roles see the MLNX OS Software UM section High Availability On switch systems with dual management systems first connect the cable and configure gt All management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure A Mellanox Technologies 59 om InfiniBand FDR Switch Platform Hardware User Manual 2 10 6 Switch Shut Down Procedures To shut down the chassis run the following command twice once for each MM Reload halt noconfirm A The chassis cannot be restarted remotely To restart the chassis you must physically go to the switch and unplug all of the 1 V power cords to the chassis and then replug in all of the power cords to the chas BB The first time you run the command it shuts down the master management module and the second time shuts down the slave management module To shut down a leaf run the following command no p
80. ed on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sions Each spine has a pair of ejectors that lock the board in place and serve as levers for seat ing or extracting see Figure 37 Management board 1 is connected to spine board 1 and management board 2 is con nected to spine board 2 When a slave management board is not installed or not working the spine board connected to the master management board cannot be hot swapped All of the spine boards can be hot swapped when two management boards are installed and working 4 V When more than one spine slot is empty always insert the lowest spine board first then 4 work your way up A A N If you need to replace the bottom spine and it does not go in try removing one or two NS spines above it and then insert the spines from the bottom up GC Mellanox Technologies 50 e mm InfiniBand FDR Switch Platform Hardware User Manual 2 10 3 1 Extracting a Spine Board When a slave management board is not installed
81. ektrischen Schocks Risque de choc et de danger energy hazard Entferrnen des Netzsteckers e lectriques Le de branchment The PSUs are all independent elnes Netzteils spannungsfrei d une seule alimentation stabi Um alle Einhieten spannungsfrei lise e ne de branch uniquement Disconnect all power supplies to zu machen sind die Netzstecker qu un module Alimentation ensure a powered down state aller Netzteile zu entfernen Stabilise e Pour isoler comple inside of the switch platform tement le module en cause Il faut de brancher toutes les ali mentations stabilise es ellanox Technologies 42 InfiniBand FDR Switch Platform Hardware User Manual 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Any yellow or red status LEDs on any ofthe management modules is cause for con cern and must be dealt with immediately 4 Check that none ofthe LEDs on the spines are yellow It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes 5 Check that the leaf status LEDs fan status LED and spine status LED in the spines are all green maximum number of connections per leaf is less than 36 then some ofthe leafto spine The maximum number of connections from each leaf is 3 Ifthe number of leafs x the MA connection LEDs may be OFF Figure 27 Spine Module
82. elettriche ed elettroniche RAEE vanno raccolti separatamente e non smaltiti nei normali rifiuti domestici Smaltire questo prodotto e tutte le sue parti in modo responsabile e rispettoso dell ambiente LA 21 Limitazioni relative all alimentazione per la Norvegia y Questa apparecchiatura progettata esclusivamente per il collegamento a un sistema di N alimentazione TN e a un sistema di alimentazione IT 126 Mellanox Technologies Rev 2 5 G12 Montaj G venlik Uyar lar T rk e 1 Montaj Talimatlar Ekipman g kayna na ba lamadan nce t m montaj talimatlar n okuyun 2 A rl k Nedeniyle Fiziksel Yaralanma Bu r n g venli bir ekilde kald rabilmek i in yeterli say da insandan yard m al n lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 A r Ekipman Bu ekipman ok a rd r ve yaralanmalar nlemek i in ekipman n mekanik asans r kullan larak ta nmas gerekir 4 Elektrik arpmas Riski Bu ekipman nerilen maksimum ortam s cakl n a an alanlarda calistirilmamalidir 45 C 113 F Ayr ca d zg n hava ak sa lamak i in havaland rma deliklerinin etraf nda en az 8 cm 3 in a kl k b rak lmal d r 5 s nma y Bu ekipman nerilen maksimum ortam s cakl n a an alanlarda calistirilmamalidir A V 45 C 113 F Ayr ca d zg n hava ak sa lama
83. emove and reinstall the problematic fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 10 4 on page 55 6 Replace the Leaf fan module YN Replace defective leaf fan modules as soon as they are identified Should any of the boards shut down due to over temperature follow the procedure starting in step 2 Issue 3 Yellow LED for the Spine Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert Mellanox Technologies 78 InfiniBand FDR Switch Platform Hardware User Manual 2 Determine which spine has a defective fan by checking the Fan LEDs on all of the spines 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 5 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 10 4 on page 55 6 Replace the spine fan module LA Replace defective spine fan modules as soon as they are identified 6 4 Spine Board Issue 1 The yellow LED on the Spine board is lit 1 Check the MLNX OS management for confirmation and possible explan
84. ent Parts Ordering Numbers Table 23 Replacement Parts Ordering Numbers Part Description OPN Power supply unit PSU 1000W for Modular Switch 1 Family Power Supply unit PSU 1670W for Modular Switch MTP006002 Family Power supply blank MTM005001 Leaf board unit 18 port FDR MSX6001FS Leaf board unit 18 port FDR10 MSX6001TS Modular Switch Family Leaf Blank MTM005004 Spine board unit for FDR system MSX6002FLS Spine board unit for FDR10 QDR system MSX6002TBS Spine board blank MTM005002 PPC460 Management Module MSX6000MAR x86 Management Module MSX6000MBR Modular Switch Family Management Blank MTM005003 Rack Installation Kit for IS5600 SX6536 Series MTR005600 SX6536 SX6518 SX6512 MSX 6506 Modular Switch MTF005005 Series Spine Fan Unit DB9 to RJ45 Harness HAR000028 Power cord 250V 15A 2 0M C14 to C13 ACC000334 Power cord 125V 15A 2 0M C14 TO C13 ACC000242 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 2 Mellanox Technologies 9 Rev 2 5 Appendix G Safety Warnings
85. ent regions of the world is located on the Mellanox Website at http www mellanox com related docs user manuals Regulatory and Compli ance Guide pdf EMC Statements are also in the Regulatory and Compliance Guide Mellanox Technologies 84 InfiniBand FDR Switch Platform Hardware User Manual Appendix B Thermal Threshold Definitions There are three thermal threshold definitions for the SwitchX switch device which impact the overall switch system operation state Warning Critical and Emergency 1 Warning 100 C On managed systems only When the SwitchX device crosses the 100 C threshold a Warning Threshold message will be issued by the MLNX OS management SW indicat ing to system administration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any action by hardware such as switch shutdown 2 Critical 120 C When the SwitchX device crosses this temperature the firmware will automatically shut down the device 3 Emergency 130 C In case the firmware fails to shut down the SwitchX device upon crossing the Critical threshold the SwitchX device will auto shutdown upon crossing the Emergency 130 C threshold Mellanox Technologies 85 A InfiniBand FDR Switch Platform Hardware User Manual Appendix C Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculate
86. f momo m ma iti A LE un Mellanox Technologies 53 ST e EA InfiniBand FDR Switch Platform Hardware User Manual Figure 40 Intact vs Defected Middle Signal Connectors 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 41 5 Note the following sticker placed on the top of the spine board Figure 42 Spine Board Insertion Caution A INSERT SLOWLY A USE EJECTOR HANDLES ONLY 4 Mellanox Technologies 5 InfiniBand FDR Switch Platform Hardware User Manual 6 Start with the ejector handles fully open that is at 45 degrees to the front panel of the spine y N Do not use the fan FRU handle to insert the spine board 7 Holding the spine by its sides carefully set the spine board into the chassis 8 Using only the ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar Do not apply excessive force to slide in the spine board If you feel resistance remove the spine and double check both the chassis and spine for any damage 9 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 10 Lock the ejector handles onto the board 2 10 4 Fan Modules There are four fan modules on the chassis for the leafs They are located on the spine side along the top of the chassis When a fan mo
87. g w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 16 Hoher Ableitstrom WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 17 Installationscodes Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code Nordamerika m ssen bereinstimmung mit den gel tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 18 Verbindung der Ger te untereinander Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 19 Switch nicht als Regal oder Arbeitsplatz nutzen Achtung Auf Schieber Schienen montiertes Ger t ist nicht als Regal oder Arbeitsbere ich zu nutzen Die Schienen sind nicht daf r bestimmt die Einheit aus dem Gestell weg zu ziehen Sie sind nur f r die permanente Installation an einem endg ltigen Stan dort gedacht nicht f r Instandhaltung und Wartung 20 WEEE Direktive Gem WE
88. gement modules 14 Install any power supply units and fan units removed at the start of the installation procedure 15 Insert all leafs that you plan to use in chassis Start at the bottom of the chassis and work your way up 16 Insert the first two spines in the top two slots 17 Insert the rest of the spines from the bottom of the chassis up to slot 3 18 Insert thermal blanks in unused leaf slots to maintain balanced air flow 19 Tighten all leaf and spine mounting screws ellanox Technologies 40 InfiniBand FDR Switch Platform Hardware User Manual 2 7 4 20 Connect the power cords to the PSUS 21 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow status LEDs is cause for concern and must be dealt with imme diately LA It can take up to 5 minutes to boot up during which time the status LED may indicate flashing green 22 You can start connecting all of the cables to the switch Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge The grounding post is M 6 with Imm pitch threads A ground wire of AWG 6 or 4mm diam eter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUS Only connect the PSU cords to properly grounded outlets Do not rely on
89. genommenem Ventilatormodul sind die Stromkontakte in der Modulvertiefung zug nglich Es d rfen KEINE Werkzeuge oder K rperteile in die Vertiefung des Ventilatormoduls gelangen 5 bertemperatur A Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der A maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 6 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunt erf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 7 Mehrere Stromeing nge Risiko eines Stromschlags und Stomgefahr Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgeschalteten Zustand im Inneren der Switch Plattform sicherzustellen Mellanox Technologies 107 Rev 1 3 8 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 9 Anschlie en Trennen von InfiniBand Kupferkabel N InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen y Warnungen und Anleitu
90. guration The switch platform uses 10U of rack space in a standard 19 rack 9U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine Mellanox Technologies 25 NN em InfiniBand FDR Switch Platform Hardware User Manual side There are upper brackets to connect the leaf side to the rack near the top of the chas sis The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have between 65cm and 80cm between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis 5 2 7 1 ESD Connection Before starting any procedure on the SX6512 switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit 2 7 1 1 Installation ProcedureRequirements You will need 2 phillips screwdriver 3 phillips screwdriver agrounding lug ground wire to properly ground the chassis Y The installation will be much easier with a power screwdriver Y It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to
91. have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation Mellanox Technologies 26 InfiniBand FDR Switch Platform Hardware User Manual 2 7 1 2 Installation and Cable Management Kit Parts Parts for installing the shelf 1 shelf 8 M6 bolts for the caged nuts 4 caged nuts e 2 Shelf rail slides 4 lock washers 2 long 2 hole flat washers Parts for installing the chassis e 2 chassis rail slides e 4 handles e 8 5 pan head bolts e 8 allen head screws 2 flat 4 hole metal spacers 1 allen wrench 8 lock washers 4 lock nuts for handles These nuts are not to be used 8 caged nuts for the faceplate e 8 M6 bolts for the caged nuts 8 for the faceplate Parts for installing the filler panel 1 Bottom filler panel e 2 M6 bolts 2caged nuts M6 Parts for installing the cable manager RH cable holder 1 LH cable holder 12 caged nuts M6 e 12 M6 bolts 6 cable holder shelves You will need 10 U of space in the rack Nine U for the chassis and one U for the shelf Figure 5 Shelf Installation Kit Parts Angled Shelf Bracket Rail M6 X 16 Rail Slide M6 X 12 bolts caged nuts and split lock washers Mellanox Technologies 27 InfiniBand FDR Switch Platform Hardware User Manual Figure 6 Chassis Installation Parts
92. he rest button If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master AN If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the management y module is removed Make sure the S Fans and L Fans LEDs are green Make sure that the spine and the leafs both have the same version of FW Reburn the FW and remove and reinstall the management module NN aa If you are running the chassis with only one management module remove and rein stall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 2 10 5 on page 57 7 If you are running the chassis with only one management module replace the man agement module Issue 2 Yellow LED for the Leaf Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Determine which fan module is problematic by checking the status LED on each fan module 5 R
93. iale pentru a se asigura ca sistemul r m ne stabil In general trebuie s y umpleti rack ul cu echipamente incep nd de jos n sus 12 Instalarea echipamentului Acest echipament trebuie s fie instalat nlocuit i sau depanat numai de c tre personal instruit i calificat 13 Eliminarea echipamentului Eliminarea acestui echipament trebuie s se realizeze n conformitate cu toate legile i regulamentele na ionale 118 Mellanox Technologies Rev 2 5 15 Codurile ed instalare Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor electrice nationale ale rii n cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerin elor aplicabile din Codul electric na ional al SUA i Codul electric canadian 16 Cordon de alimentare electric nregistrat UL i certificat CSA A Pentru conectarea la o surs de alimentare pentru America de Nord selectati un cordon de alimentare care este nregistrat UL si certificat CSA cu 3 conductoare 16 AWG terminat cu o fi turnat cu putere nominal egal cu 125 V 13 A cu o lungime de minimum 1 5 m sase picioare dar nu mai lung de 4 5 m Pentru conectarea la o sursa de alimentare in Europa selectati un cordon de alimentare care este armonizat la nivel international i marcat lt HAR gt cu 3 conductoare cu minimum 2 fire de 1 0 mm cu putere nominal egal cu 300 V
94. k i in havaland rma deliklerinin 4 etraf nda en az 8 cm 3 in a kl k b rak lmal d r 6 asi stif 4 Sasinin di er herhangi bir ekipman n zerine istiflenmemesi gerekir asi d erse fiziksel yaralanmalara ve ekipmanda hasara neden olabilir Mellanox Technologies 127 Rev 1 3 7 Yedekli G Kayna y Ba lantysy Elektrik arpma Tehlikesi Bu r n yedekli g kayna veya onun yerine bo elektrik kutusu i erir G kayna gp i in bo elektrik kutusu varsa kutunun kapa a kken veya tam olarak kapat lmam ken r n al t rmay n 8 oklu G Giri leri Elektrik arpmas riski ve enerji tehlikesi B t n PSU lar G Kayna niteleri ayr d r Anahtar platformundaki g c kapatmak i in t m g kaynaklar n n ba lant lar n kesin 9 im ek Elektrik arpma Tehlikesi G ky z nde im ek akt zamanlarda ekipman zerinde al may n veya kablo ba lamay n ya da kablo ba lant s kesmeyin 10 Bak r nfiniband Kablo Ba lama Ba lant y Kesme Bak r nfiniband kablolar a rd r ve esnemezler Bu nedenle ba lant lara ok dikkatli bir ekilde tak lmalar veya kar lmalar gerekir zel uyar lar talimatlar i in kablo reticinize ba vurun 11 skele Montaj ve Bak m Bu r n bir iskelede monte edildiyse veya bir iskele ile sunulduysa sistemin sabit kalmas i in zel nlemler al
95. k to get up To remove disengage the lock and slowly pull the connector away from the port receptacle A For a valid physical connection both ends of the cable must be connected to working A devices EN Take to not impede the air flow through the ventilation holes next to the Infini Band ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 2 9 1 Supported Approved Cables For a list of approved cables for this switch see the Mellanox approved cable list http www mellanox com related docs user_manuals Mellanox_approved_cables pdf 2 9 2 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX65xx series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables 2 10 Hot Swap Insertion and Extraction Before starting any procedure on the SX6512 switch system put an ESD prevention wrist strap on your wrist and connect to the SX6512 chassis Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the
96. l supports of the rack Ad Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the open ings are populated with blanks All of the leafs spines and management modules come shipped in a separate package The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instructions to make the initial configuration access area can be accessed only through the use of a special tool lock and key or This unit is intended for installation in a Restricted Access Location A restricted other means of security gt 2 1 Installation Safety Warnings These safety warnings are in English For French German Spanish Russian and Roma nian see the Appendixes 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product LA e lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Heavy Equipment 4 This equipment is very heavy and should be moved using a mechanical lift to avoid injuries Mellanox Technologies 19
97. l waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Go to the Mellanox website for detailed instructions for disassembly according to the WEEE Directive Mellanox Technologies 81 AA RECI Appendix A InfiniBand FDR Switch Platform Hardware User Manual Specification Data Table 17 Switch Specification SX6512 4N R Physical 15 75 x 19 x 27 inches Size 400mm x 482 6mm x 685 8mm The shelf adds 44 5mm to the height Mounting 19 rack mount of spines 6 of leafs 12 48kg 105 lbs empty configuration Weight 83 94kg 185 Ibs shipped configuration 147kg 324 lbs full configuration CoGh 311 80mm from bottom of chassis Center of CoGw 222 00mm from left side surface Gravity CoGd 328 14mm from port end surface Measured from the bottom left corner as one faces the switch ports Max Air Flow Through Leafs 9 186M min 324 6 Max Air Flow Through Spines 2 533M min 89 5 CFM Total Max Air Flow 11 719M3 min 414 0 SerDes Speeds 10 20 40 Gb s or 56 Gb s per port Connector QSFP Types Sound Level 76 db A of Ports 216 Power and Environmental Input Voltage 100 240 VAC 50 60Hz Typical Passive 1832W Active 2325W include
98. latform Hardware User Manual Revision History Table 1 Revision History of this User s Manual Revision Date Details 2 5 Nov 30 2015 Updated 1600W PSU power from 1600W to 1670W 24 Nov 26 2015 Updated Appendix F Replacement Parts Ordering Numbers on page 92 spine and leaf module OPNs 2 3 Nov 18 5 Updated Table 12 Management LED Display for Normal Operation on page 68 2 2 June 30 2015 Updated e Section 2 3 Chassis Package Contents page 24 e Section G 13 Japan VCCI Class A Statement on page 130 2 1 Feb 24 2015 Updated Table 12 Management LED Display for Normal Operation on page 68 with master LED status indicators Appendix A Specification Data on page 82 with FDR10 power numbers and max heat output Appendix D Calculating the Power of a Chassis on page 87 with FDR10 power numbers 2 0 Nov 17 2014 e Section 1 4 2 1670W Power Supply Units on page 17 e Section 2 7 1 3 Container Mishandling on page 28 Appendix A Specification Data on page 82 1 9 Oct 27 2014 Updated Max Air Flow numbers in Appendix A Specification Data on page 82 N 1 configuration in Section 1 4 2 1670W Power Supply Units on page 17 1 8 July 18 2014 Updated e Section 2 7 4 Ground Connections on page 41 Appendix A Specification Data on page 82 Appendix D Calculating the Power of a Chassis
99. list Check the parts for visible damage that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear ofthe switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis 4 2 4 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order 2 5 Spine Package Contents The spines are shipped 3 to a box The customer will receive as many boxes as needed to fill the chassis with a full complement of spines Insert the bottom spine board first 2 6 Management Package Contents The package includes e all of the management modules ordered by the customer e RJ45 to DB9 harness for each management module received 2 7 Physical Installation Warning This equipment is very heavy Safety is the first concern Make sure that adequate manpower and proper equipment is used for transporting and moving the chassis The fully loaded chassis weighs 120 kg 265 LBS full configuration 48 KG 105 LBS empty configuration 83 94kg 185 LBS shipped confi
100. ma pari a 1 5 m sei piedi ma non pi lunga di 4 5 m Per una connessione europea selezionare un cavo di alimentazione armonizzato a livello internazionale e contrassegnato da lt HAR gt a 3 conduttori minimo 1 0 mm2 fili con guaina isolante in PVC Il cavo deve disporre di una spina stampata di potenza nominale pari a 250 V 10 A 17 Corrente di dispersione elevata Avvertenza corrente di dispersione elevata il collegamento a terra essenziale prima di collegare l alimentazione Mellanox Technologies 125 Rev 1 3 18 Interconnessione delle unit A I cavi per il collegamento all unit RS232 e alle interfacce Ethernet devono 4 disporre della certificazione UL ed essere del tipo DP 1 o DP 2 Nota in caso 4 y di installazione su un circuito la cui potenza non limitata Protezione contro le sovracorrenti la cablatura dell edificio deve integrare un dispositivo di protezione contro le sovracorrenti di potenza nominale pari a 20 19 Non utilizzare lo switch come scaffale o piano di lavorol Attenzione un apparecchiatura scorrevole o montata su binari non va utilizzata come scaffale o piano di lavoro I binari non sono progettati per far scorrere e allontanare l unit dal rack Essi sono destinati all installazione permanente solo nel luogo di lavoro e non vengono utilizzati per assistenza e manutenzione 20 Direttiva RAEE Secondo la direttiva RAEE 2002 96 EC tutti i rifiuti da apparecchiature
101. must be started with a full complement of possi ble PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it will be necessary to reboot the chassis once the defec tive PSU has been replaced With 1670W optional PSU supplies the chassis can run on as little as 1 2 of the full com plement of PSUs N N allows the chassis to run on 2 PSUs supplied from one power grid while 2 are connected to a second power grid 1 Check all FRUs for proper insertion and seating before connecting the AC power cords Boards Power supplies e Leaf fan modules Spine fan modules 2 Insert all leafs that you plan to use in the chassis Start at the bottom of the chassis and work your way up Insert the first two spines in the top two slots Insert the rest of the spines from the bottom of the chassis up to slot 3 Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all leaf and spine mounting screws Connect the power cords to the PSUs Ol cU e M Connect the power cords to grounded electrical outlets 2 4 N With N 1 PSU redundancy do not power up the ch
102. ngen des Kabelherstellers 10 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 11 Ger teinstallation A V Diese Ger t sollte nur von geschultem und gualifiziertem Personal installiert aus getauscht oder gewartet werden 12 Ger teentsorgung N N A Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 4 13 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bes timmungen installiert werden 14 Installationscodes installiert weden In Nodamerika muss das Ger t gem den geltenden Anforderungen EN 4 N Dieses Ger t muss entsprechend der aktuellsten Version des National Electrical Code des US National Electrical Code und des Canadian Electrical Code installiert werden SE 108 Mellanox Technologies Rev 2 5 15 UL und CSA zertifiziertes Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhan
103. niBand on gt PINI 12 NINI gt 13 14 DINI 15 gt 16 NASI 919 Y N UL CSA 3 16 AWG 13A 125 1 5 6 4 5 235
104. niBand FDR Switch Platform Hardware User Manual 11 During Lightning Electrical Hazard During periods of lightning activity do not work on the eguipment or connect or dis connect cables 12 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 13 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken LA to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 14 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 15 Equipment Disposal AN A Disposal of this equipment should be in accordance to all national laws and regula tions 16 Local and National Electrical Codes N This equipment should be installed in compliance with local and national electrical codes 17 Installation Codes A This device must be installed according to the latest version of the country A national electrical codes For North America equipment must be installed in A accordance to the applicable requirements in the US National Electrical Code 7 and the Canadian Electrical Code Mellanox Technologies 21
105. nmal d r Genelde ekipmanlar iskeleye a a dan yukar do ru doldurman z gerekir 12 Ekipman Montaj Ekipman n yaln zca e itimli ve nitelikli personel taraf ndan monte edilmesi de i tirilmesi ve veya bak m n n yap lmas gerekir 128 Mellanox Technologies Rev 2 5 14 Yerel ve Ulusal Elektrik Kodlar Bu cihaz n lke ulusal elektrik kodlar n n son s r m ne g re monte edilmesi hh V gerekir Kuzey Amerika i in ekipman n ABD Ulusal Elektrik Kodu ve Kanada Elektrik Kodu nun uygulama kosullarina gore monte edilmesi gerekir 16 UL Kay tl ve CSA Onayl G Kayna Kablosu A Kuzey Amerika da g ba lant s i in UL Kay tl ve CSA Onayl bir g kayna 4 kablosu se in 3 iletken 16 AWG 125 V de erinde kal planm bir fi le biten 13 A en az 1 5 m alt fit uzunlu unda fakat 4 5 m den uzun olmayan bir kablo Avrupa da g ba lant s i in uluslararas uyumlu ve lt HAR gt i aretli 3 iletken en az 1 0 mm tel 300 V de erinde ve PVC yal t ml bir g kayna kablosu se in Kablonun 250 V 10 A de erinde bir kal planm fi i olmas gerekmektedir 17 Y ksek Ka ak Ak m Uyary Y ksek ka ak ak m varsa g kayna na ba lanmadan nce mutlaka toprak lama ba lant s yap lmal d r 18 nitelerin Ara Ba lant s RS232 nitesini ve Ethernet Arabirimlerini ba layacak olan kablolar n UL
106. nt power or a blank in its place In case of a blank A fastened 8 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 9 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 10 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 11 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with eguip When this product is mounted or serviced in a rack special precautions must be taken A ment starting from the bottom to the top 12 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 94 Mellanox Technologies Rev 2 5 13 Eguipment Disposal N N Disposal of this equipment should be in accordance to all national laws and regula 4 tions 14 Local and National Electrical Codes y V This eguipment should be installed in compliance with local and national electrical 4 codes 15 UL Listed and CSA Certified Power Supply Cord A For North American po
107. of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified physical layer using different block types deskew mechanism and framing rules The SX6512 switch also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with a 64b 66b encoding resulting in an effective bandwidth of 40Gb s FDR10 supports 20 more bandwidth over regular QDR using the same QSFP cables connectors Both FDR and FDR10 support Forward Error Correction FEC as described in IEEE Std 802 3ap 2007 Amendment to IEEE Std 802 3 2005 chapter 74 FDR 10 are only guaranteed to work with approved cables FDR 10 is only guaranteed to work with approved Mellanox ConnectX 3 adapters di 1 4 Power Supply Redundancy The SX6512 platform comes standard with four power supplies and has two order options e SX6512 4R N 1 redundancy SX6512 NR redundancy 1 Features for a future release Mellanox Technologies 16 0 Mi an InfiniBand FDR Switch Platform Hardware User Manual 1 4 1 1000W Power Supply Units The 1000W power supply units PSUs deliver 1000W at 48V The AC source to each PSU must be able to provide 1176 5W in order to output 1000W in DC With 1000W PSUs the only redundancy option is N 1 In this configuration IPSU can be extracted without
108. onal Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individ ual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MASTER Off Green if master no power Off if slave this management module is not the master Green Management module is operating as a master Mellanox Technologies 68 e Xi IE InfiniBand FDR Switch Platform Hardware User Manual Figure 56 Management Module LEDs LFANS Master 3 1 5 1 Management Module PSU LED Indicator The management module PSU indicator should be green Table 13 Management Module PSU LED Configurations LED Condition Description Green OK All PSUs are working at correct input and output voltages Red Error One or more of the PSUs for the chassis is bad Check each PSU for a red LED 3 1 5 2 Management Module S FANS LED Indicator The management module S FANS indicator should be green Table 14 Management Module S Fan LED Configurations LED Condition Description Green OK All Spine fans are working at configured speed Red Error One or more of the spine fans within a spine fan mo
109. ostor na njegovu mjestu N lopac uklonjen ili ako nije dobro pri vr en 8 Vi estruki ulazi za napajanje Rizik od strujnog udara i opasnost od elektri ne energije PSU jedinice su neovisne Odspojite sva napajanja kako biste osigurali stanje bez napajanja unutar platforme preklopnika 9 Tijekom udara munje Opasnost od elektri ne energije A Tijekom djelovanja munja nemojte raditi na opremi ili spajati ili odspajati kabele 10 Spajanje Odspajanje bakrenog kabela InfiniBand Bakreni kabeli InfiniBand su teski 1 nesavitljivi i kao takvi se moraju pazljivo prikl juciti na ili isklju iti iz konektora Obratite se proizvo a u kabela za posebna upo zorenja upute 11 Monta a ormari a i servisiranje opreza kako bi se osiguralo da sustav ostane stabilan Op enito trebali biste ispunja Kad se proizvod montira ili se servisira u ormari u moraju se poduzeti posebne mjere vati ormari s opremom po ev i od dna prema vrhu 12 Instaliranje opreme Ovu bi opremu trebalo instalirati zamjenjivati i ili servisirati samo obu eno i kvalifici rano osoblje Mellanox Technologies 121 Rev 1 3 14 Lokalni i nacionalni elektri ni kodovi Ova oprema trebala bi se instalirati u skladu s lokalnim i nacionalnim elektri nim kodovima 15 Instalacijski kodovi A Ovaj se ure aj mora instalirati sukladno najnovijoj verziji nacionalnih elek PA N tri nih kodova dr ave U Sjevernoj Americi oprema se mo
110. ower enable lt module gt To shut down a spine run the following command no power enable lt module gt To shut down a management module run the following command no power enable lt module gt Mellanox Technologies 60 InfiniBand FDR Switch Platform Hardware User Manual 3 nterfaces 3 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of the TT manager All chassis conditions and management options are available and controllable through the manage ment software either CLI or WebUI Y It is recommended that all of the chassis sub systems be maintained and managed through the management software 3 1 1 Power Supply Unit LEDs Each Power Supply Unit has the following indicator LEDs Figure 50 Power Supply Unit Status Indications DC AC OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC A The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis Mellanox Technologies 61 InfiniBand FDR Switch Platform Hardware User Manual PSU Cover On and Off PSU Covers On Figure 51 FRONT SIDE ORCA SX6512 PSU Covers Off 62 Mellanox Technologies
111. ower level 2 according to the Public Specification Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the SX6512 They are 112C port 1 CONSOLE port this is an RS232 connector for connecting to a host machine MGT this is an Ethernet connector e 1 USB port e 1 RST reset button Mellanox Technologies 70 InfiniBand FDR Switch Platform Hardware User Manual Figure 59 Management Module Interfaces 3 4 1 12C This interface is for Debug and Troubleshooting only This interface is for FAEs als only 3 4 2 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure 3 4 3 MGT Management The MGT port is an Ethernet port for remote management Any remote terminal con nected to the Ethernet port can then be used to manage the fabric and chassis Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M 1G auto negotiation 4 Initial configuration must be done on all of the management modules The first man agement module you configure will be the master 4 3 44 USB The USB port can be used to upload new SW using any storage device that has a USB
112. pendantes Pour s assurer que le commutateur est bien hors tension d branchez toutes les alimen tations 9 En cas d orage danger d lectrocution Pendant un orage ne pas travailler sur l quipement ni brancher ou d brancher des c bles 10 Connexion et d connexion du c ble InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et peu flexibles Par cons quent il faut proc der avec soin pour les brancher ou les d brancher des connecteurs Consulter le fabricant du c ble pour obtenir des instructions ou des avertissements sp cifiques 104 Mellanox Technologies Rev 2 5 11 Montage en rack et maintenance ment attention pour s assurer que l ensemble reste stable En r gle g n rale le rack doit Lors du montage ou de la maintenance de ce produit dans un rack il faut faire sp ciale N amp tre rempli en commengant par le bas 12 Installation de l quipement Cet quipement ne doit tre install remplac et maintenu que par un personnel form et qualifie mise au rebut de cet quipement doit se faire conform ment a toutes les lois et r glementations nationales 14 Codes lectriques locaux et nationaux A IN Cet quipement doit tre install conform ment aux codes lectriques locaux et nation aux 15 Codes d installation Cet appareil doit tre install conform ment la version la plus r cente des codes lec
113. prapus peste niciun alt echipament In cazul in care cadrul cade poate cauza leziuni corporale i deterior ri ale echipamentului 7 Conexiunea la o surs de alimentare electric suplimentar pericol electric Acest produs include o surs de alimentare suplimentar sau un spatiu gol n locul prox p pat u g A acesteia n cazul n care spatiul pentru sursa de alimentare este gol nu operati produ y sul c nd capacul orb este indep rtat sau nu este fixat in mod sigur Mellanox Technologies 117 Rev 1 3 8 Multiple mufe electrice Risc de oc electric i pericol electric y Toate aparatele cu alimentare de la re ea sunt independente Deconectati toate sursele de alimentare cu energie pentru a asigura decuplarea n inter iorul platformei de comutare 9 n timpul desc rc rilor electrice pericol electric n timpul perioadelor cu desc rc ri electrice luminoase nu lucrati cu echipamentul sau nu conecta i sau deconectati cablurile 10 Conectarea deconectarea cablului din cupru InfiniBand N a Cablurile InfiniBand din cupru sunt grele i inflexibile de aceea trebuie s fie ata ate 4 sau deta ate de conectori cu grij Consulta i produc torul de cabluri pentru avertis y mente instructiuni speciale 11 Montarea depanarea intr un rack y N C nd acest produs este montat sau depanat ntr un rack trebuie s fie luate m suri de A precautie spec
114. ra instalirati suk ladno va e im zahtjevima navedenim u US National Electrical Code i Canadian Electrical Code 16 UL CSA kabel napajanja IN Za sjevernoameri ku mre u odaberite kabel napajanja koji je na UL listi 1 sa MA CSA certifikatom 3 Zilni 16 AWG 16 AWG koji zavr ava lijevanim uti ka em nazivnog napona od 125 V 13 A minimalne duljine od 1 5 m six feet est stopa ali ne dulji od 4 5 m Za europsku mre u odaberite kabel napajanja koji je me unarodno uskla en i ozna en lt HAR gt 3 ilni s najmanje 1 0 mm ice nazivnog napona od 300 V s PVC izolacijom Kabel mora imati lijevani utika nazivnog napona od 250 V nazivne struje od 10 A 17 Veliko curenje struje 4 Upozorenje Veliko curenje struje Prije spajanja napajanja nuZno je spojiti uzeml jenje 18 Interkonekcija ure aja Kabeli za spajanje na jedinicu RS232 1 Ethernet su elja moraju biti s UL cer tifikatom vrste DP 1 ili DP 2 Napomena kad se nalazi u krugu bez LPS vodi a Za tita od strujnog preoptere enja Uvijek dostupni odobreni za titni ure aji od strujnog preoptere enja nazivne struje od 20 A moraju se ugraditi u o i enje zgrade 19 Nemojte koristiti preklopnik kao policu ili radnu povr inu Pozor Oprema montirana na kliza e vodilice ne bi se trebala koristiti kao polica ili radna povr ina Vodilice nisu namijenjene za povla enje ure aja iz ormari a Slu e samo za trajnu instalacij
115. renja za instaliranje Croatian 120 G 11 Avvertenze di sicurezza per l installazione italiano 123 Mellanox Technologies 4 InfiniBand FDR Switch Platform Hardware User Manual 2 MontajG venlikUyar lar T rk e 127 3 Japan VCCI Class Statement 130 _ G G Mellanox Technologies 5 InfiniBand FDR Switch Platform Hardware User Manual List of Tables Table 1 RevisionHistoryofthisUser sManual 9 Table 2 Reference Documents and Websites 10 Table 3 OPNs for Power Supply Units 18 Table 4 leafStatuslED een en 63 Table 5 Bad Port LED 63 Table 6 Connector Physical and Logical Link Indications 64 Table 7 SpineStatuslRD tee 65 Table 8 Spine Fan Status LED 65 Table 9 Spine to Leaf IB Link Status 66 Table 10 Bad Port LED 66 Table 11 Spine Side LED Display for Normal Operation 67 Table 12 Management LED Display for Normal Operation 68 Table 13 Management Module PSU LED Configurations
116. s QSFP at 1 5W Total Power Optical 2391 W includes at 1 7W Consumption FDR 10 Maximum Passive 2129W Active 2688W includes QSFP at 1 7W Optical 2787W includes QSFP at 2 0W FDR 10 Max 9 509 BTUs hr Heat Output ellanox Technologies 82 InfiniBand FDR Switch Platform Hardware User Manual M Table 17 Switch Specification SX6512 4N R Physical Total Power Typical Passive 2284W Active 2777W includes QSFP at 1 5W Optical 2843W includes QSFP at 1 7W Consumption FDR 14 Maximum Passive 2388W Active 2946W includes QSFP at 1 7W Optical 3046W includes QSFP at 2 0W Max Heat 10 393 BTUs hr Output eur Operating 0 to 45 Celsius Non operating 40 to 70 Celsius Humidity Operating 10 85 non condensing Protocol Support Speed Protocol InfiniBand Auto negotiation of 56Gb s 40Gb s 20Gb s 10Gb s MLNX OS and baseboard performance and device management agents for full Infini Management Band in band management Data Rate 56Gb s per port QoS 8 InfiniBand virtual lanes for all ports Regulatory Compliance US Canada cTUVus certificate International CB report and certificate Saft Russia GOST R certificate x Argentina S mark certificate DoC BSMI certificate for the power supply only CE EMC amp safety report and certificate USA FCC part 15 class A ms Canada ICES 003 class A missions Australia New Zealand AS N
117. shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 22 Installation codes A This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 23 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 22 Mellanox Technologies InfiniBand FDR Switch Platform Hardware User Manual M 24 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards TEC 60
118. switch 5 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It dis covers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS rout ing partitioning to the fabric devices You can view and configure the Subnet Parameters SM via the CLI WebUI Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand amp network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based 5 2 Fabric Inspector Diagnostics Fabric Inspector is a plug amp play software for fee within MLNX OS displaying and fil tering all identified systems and nodes within the fabric Fabric Inspector includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections based on traffic patterns and user assigned system names GUIDs 5 3 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet WEB SNMP XML 5 4 Upgrading Software The new softw
119. t Module CONSOLE ci PSU SFANS LFANS Master m os oo Ejectors 2 10 5 1 Inserting a Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a management board 1 Check for foreign objects in or mechanical damage to the chassis management slots 2 Check management mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 48 Figure 48 Intact vs Defected Signal Connectors Fail Mellanox Technologies 58 InfiniBand FDR Switch Platform Hardware User Manual 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 49 Figure 49 Intact vs Defected Power Pin Holders 5 Start with the ejector handles fully open that is at 45 degrees to the front panel of the management 6 Holding the management by its sides carefully set the management board into the chassis 7 Using only the management ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar Do not apply excessive force to slid
120. t coulissant ou monte sur rail ne doit pas servir d tag re ni de plan de travail Les rails ne sont pas destin s a faire coulisser l unit hors du rack Ils sont destin s a une installation permanente a l emplacement final pas pour l entretien ni la maintenance 20 Directive DEEE Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doivent tre collect s s par ment et ne pas tre mis au rebut avec les d chets m nagers habituels Ce produit et toutes ses pieces doivent tre mis au rebut d une mani re responsable respectant l environnement ran 21 Restrictions concernant l alimentation pour la Norv ge Cet appareil est prevu pour tre reli a un systeme d alimentation TN et un systeme d alimentation informatique de Norv ge uniquement G6 Installation Sicherheitshinweise German 1 Installationsanleitungen N Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung N anschlie en 106 Mellanox Technologies Rev 2 5 2 Verletzungsgefahr wegen des Gewichts da Um das Produkt sicher anzuheben gen gend Personen einsetzen Mal lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Schweres Ger t Dieses Ger t ist schwer und muss mit einem mechanischen Hebeger t verschoben werden um Verletzungen zu vermeiden 4 Stromschlagrisiko Stromschlagrisiko Bei ab
121. t software automatically discovers and configures the fabric and begins utilizing the switch The Mellanox Operating System MLNX OS software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is available on the Mellanox web site Basic installation is covered in Chapter 2 Installation on page 19 Hot swapping components and hardware maintenance is covered in Chapter 2 10 Hot Swap Insertion and Extraction on page 44 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID identifier and product version information are found on the label attached to the pull out tab below the Mellanox logo on the spine side of the chas sis Figure 1 Product Label 2 S N MT0806X01504 AIT MSX6506 3R curo 0002c9020027051c EA en MAC 0002C9020040 UN di Made in Israel Mellanox Technologies 13 InfiniBand FDR Switch Platform Hardware User Manual The GUID is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management software look for this GUID in addition to their own Node GUID Figure 2 Product Label Tab Location ETS ane de de wi AAA A d t a MZ L Here is the tab with the product label TENTE maka dh KRA 5
122. te 4 Remove and put aside the box 5 Visually inspect the chassis make sure that there is no visible damage 4PSUs are installed for the chassis all4 fans are installed 6 Remove all protective plastic film from all sides and top of the chassis 7 If you are not using a mechanical lift to install the chassis reduce the weight of the chassis by removing all of the power supply units and fan units and put aside for reinstallation after the chassis is installed in the rack P The leafs spines and management modules are shipped separately 2 7 1 5 Installing the Shelf 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground Make sure you have the spine side close to the doors to ensure maximum cable bend ing room Mellanox Technologies 30 InfiniBand FDR Switch Platform Hardware User Manual Figure 10 Placement of Chassis in Rack Doors RACK Cable with recommended bending radius Chassis E Spinelside Insufficient room for recommended bending radius Doors You will most likely need extra room for cable bending on the connector side of the chas sis and should plan to keep the spine side as close to the rack door as possible thus hav ing more room for the cables See Figure 1 on page 36 3 Clip 4 caged nuts into the holes in the ra
123. tform Hardware User Manual Figure 14 Chassis on Shelf with Filler Panel Shelf Filler panel Mellanox Technologies 33 InfiniBand FDR Switch Platform Hardware User Manual 2 7 2 Installing the Cable Holder Y Now is the time to install the cable holder Figure 15 Cable Holders tl side cable holder gt 12 X M6 caged nut 1 12 X M6 bolt for caged nut pu Left side cable holder b 1 Place the cable holder next to the rack on the connector side of the chassis and iden tify the holes where the caged nuts are to be placed EU Cable support shelf For each model of chassis there is the one support shelf for every two leafs The cable holders should go to the outside of the vertical supports Mellanox Technologies 34 InfiniBand FDR Switch Platform Hardware User Manual 2 Set the bottom of the cable holder at the level of the shelf A If the cable holder is not set properly the upper bracket will not line up with the cable holder Figure 16 Cable Holder Placement Notch for upper bracket washer Set this even with the top of the shelf Screw the cable holder onto the rack using the screws provided Repeat steps 1 3 for the second cable holder Install the shelves The cable support shelves just sit on the cable holders ds 209 Use the holes in the shelves to tie down the cables individually or in bundles A Use Velcro ties to tie down the ca
124. tus LED for the spine fan modules 36status LEDs showing the existence of leaf to spine connections One Bad Port Led showing symbol errors in the data stream OneUID LED that can be lit to identify an individual spine The 36 LEDs on each spine are divided by the number of leafs and the result N is the number of connections from each leaf that are connected to the spine 6 spines must be installed and working to ensure that full BW exists between nodes The maximum num ber of connections from each leaf is 3 If the number of leafs x the maximum number ellanox Technologies 64 ei InfiniBand FDR Switch Platform Hardware User Manual of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF The status LEDs for the spine and their descriptions are shown in Table 7 The LEDs indicate as follows Figure 53 Spine Status LEDs Spine Status LED Fan Status LED Bad Port LED UID LED uy SO 3e LS HERE 6 8 10 12 14 16 18 Spine leaf IB connection status 3 1 3 1 Status LED N Table 7 shows the spine status according to the LED condition Table 7 Spine Status LED LED Condition LED Description Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Solid Orange Non fatal error this spine needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 3 2 Fan LED
125. u na kona nom polo aju a ne za servisiranje i odr avanje 122 Mellanox Technologies 20 WEEE direktiva Sukladno WEEE direktivi 2002 96 EZ sav elektri ni i elektroni ki otpad EEE trebao bi se prikupljati zasebno i ne bi se trebao odlagati kao obi an ku anski otpad Odlaganje ovog proizvoda i svih njegovih dijelova vr ite na odgovoran i ekolo ki na in 21 Elektri na ograni enja dr ave Norve ke Ovaj je ure aj namijenjen samo za spajanje na elektri ni sustav s TN uzemljenjem i na elektri ni sustav s IT uzemljenjem dr ave Norve ke G 11 Avvertenze di sicurezza per l installazione italiano 1 Istruzioni di installazione Leggere tutte le istruzioni di installazione prima di collegare l apparecchiatura all alimentazione 2 Lesionia causa del peso Ah Usare un numero di persone sufficiente per sollevare in sicurezza questo prodotto 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kes 18 32 kc3 32 55 kgs gt 55 kg 3 Apparecchiatura pesante 4 Questa apparecchiatura molto pesante e va spostata mediante un sollevatore meccanico per evitare lesioni Mellanox Technologies 1 2 3 Rev 2 5 Rev 1 3 4 Rischio di scosse elettriche N Rischio di scosse elettriche Con il modulo ventola rimosso i pin di alimentazione sono accessibili all interno della 4 N cavit del modulo NON inserire strumenti o parti del corpo nella cavit del mo
126. wer connection select a power supply cord that is UL Listed N and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m LO For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 16 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 17 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 19 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance Mellanox Technologies 95 Rev 1 3 20 WEEE Directive
127. x Technologies Rev 2 5 11 Montaggio su rack e manutenzione Quando questo prodotto viene montato o sottoposto manutenzione su un rack Wm necessario adottare delle precauzioni speciali per assicurarsi che il sistema resti stabile In generale il rack va riempito con apparecchiature procedendo dal basso verso l alto 12 Installazione dell apparecchiatura A Ouesta apparecchiatura va installata sostituita sottoposta a manutenzione solo da N pp p N personale addestrato e qualificato a Lo smaltimento di questa apparecchiatura va effettuato in conformit con tutte le leggi e le normative nazionali 14 Codici elettrici locali e nazionali PA N Questa apparecchiatura va installata in conformit con le norme elettriche locali e N 5 A nazionali 15 Codici di installazione y Questo dispositivo va installato in conformit con l ultima versione dei codici elettrici A V nazionali del Paese Per il Nord America l apparecchiatura va installata in conformit con i reguisiti applicabili del codice elettrico nazionale USA e del codice elettrico q pp canadese 16 Cavo di alimentazione UL e munito di certificazione CSA Per una connessione di alimentazione nordamericana selezionare un cavo di alimentazione di tipo UL e munito di certificazione CSA a 3 conduttori 16 AWG terminato con una spina stampata con tensione nominale pari a 125 V 13 A di lunghezza mini
128. y recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit gt 2 3 Chassis Package Contents The package includes 1 box containing installation kit parts cable management system power cords 1 shelf 1 chassis with the following modules installed 4 fans 4PSUs If you are not using a mechanical lift to install the chassis reduce the weight of the chas sis by removing all of the power supply units and fan units and put aside for reinstalla tion after the chassis is installed in the rack 1 installation guide Inside the single box is 3 smaller boxes installation kit box 1 box containing 4 power cords 250v 158 2 0M C14 to C13 USA UL Standard cable management kit Mellanox Technologies 24 InfiniBand FDR Switch Platform Hardware User Manual 6 cable supporter boxes Before you install your new SX6512 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts
129. ymbol error This LED shows symbol errors Possible causes for this are bad cable e bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED imme diately goes off until the next symbol error is received UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition ellanox Technologies 66 InfiniBand FDR Switch Platform Hardware User Manual Figure 54 Spine Side Panel Display Status Indications STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER Table 11 Spine Side LED Display for Normal Operation Normal LED Condition Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One

User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents