Home

648-Port InfiniBand QDR SwitchX® Switch Platform

1. 22
2. Mellanox Technologies 118 23 RS232 Ethernet 1 DP 2 UL B 20 24
3. 9 Ha H 10 Mellanox Technologies 116 11
4. G G 7 Avertismente privind siguran a la instalare Romanian 1 Instruc iuni de instalare A SLE 4 Citi i toate instruc iunile de instalare nainte de a conecta ellanox Technologies 119 2 Accidentare cauzat de greutate Apelati la un num r suficient de persoane pentru ridica n siguran acest produs LA lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Echipament greu Acest echipament este foarte greu i trebuie s fie mutat folosind un dispozitiv meca nic de ridicare pentru a evita producerea de leziuni 4 Instalarea ntr un loc cu acces limitat Aceast unitate este destinat instal rii ntr un loc cu acces limitat 5 Risc de soc electric A Risc de soc electric y Odat ce modulul ventilator este ndep rtat electrici sunt accesibili n cavitatea A N modulului J NU introduce i instrumente sau p r i din corp n cavitatea modulului ventilator 6 Temperatur n exces AN Acest echipament nu trebuie s fie ac ionat ntr o zon unde temperatura ambiant dep e te valoarea maxim re
5. UL CSA 3 16 AWG 1 5 6 4 5 125 13 lt HAR gt 3 c 1 0 300 250 10 20 21
6. TOKOM InfiniBand H
7. 65 Table 9 LEDs Display for Normal 66 Table 10 Management Module PSU LED Configurations 67 Table 11 Management Module S Fan LED Configurations 67 Table 12 Management Module L Fan LED Configurations 68 Table 13 Management Module MASTER LED Configurations 68 Table 14 155600 Switch Specification 82 Table 15 Power Consumption of Chassis Parts 86 Table 16 InfiniBand Connector Pinout 89 Table 17 Replacement Parts Ordering Numbers 91 Mellanox Technologies 5 InfiniBand Switch Platform Hardware User List of Figures Figure 1 Product Label set ka E thaw Earl A a rat Tu die 11 Figure 2 License Key 12 Figure 3 Product Label Tab Location 12 Figure 4 Management Module MAC Address Location 12 Figure 5 155600 Switch Views ci see e pl e ee aa ERR 13 Figure 6 Distance Between the Vertical Supports 22 Figure 7 Installation Kit Parts 24 Figure 8 Shock and Tilt Stickers be iR
8. 52 Figure 47 Intact vs Defected Side Signal Connectors 53 Figure 48 Intact vs Defected Middle Signal Connectors 53 Figure 49 Intact vs Defected Power Pin Holders 53 Figure 50 Leaf Fan Locations on the Chassis 55 Figure 51 Leaf Fan Module Extraction 55 Figure 52 Spine Fan Module isi i eere REA ERR Rd 56 Figure 53 Fan Status LED on the Spine Module 56 Figure 54 Management Module 58 Figure 55 Intact vs Defected Signal Connectors 58 Figure 56 Intact vs Defected Power Pin Holders 59 Figure 57 Power Supply Unit Status Indications 61 Figure 58 PSU Cover On and Off 62 Figure 59 Leaf Board Led Indicators 62 Figure 60 Spine Status LEDs t te et isn 64 Figure 61 Spine Side Panel Display Status Indications 65 Figure 62 Management Module Status Indications 66 Figure 63 Management Module LEDS 67 Figure 64 Port Numbering 1 5 x nero Pot Vein was COU oe Bk ta eee eR eta 68 Figure 65 Top and B
9. 1 IEC 60 825 1 1993 A1 1997 A2 2001 EN 60825 1 1994 A1 1996 A2 2001 25 26 27 WEEE 2002 96 WEEE
10. 12 1 1 4 Product Physical Specifications and Power 12 1 2 Features MIS Rise a a 14 1 3 Power Supply Redundancy 14 Chapter 2 Installation a uam 15 2 1 Installation Safety Warnings 15 2 2 Environmental and Safety Recommendations 19 2 3 Chassis Package Contents 20 2 4 Leaf Package Contents 21 2 5 Management Package Contents 21 2 6 Physical Installation nee aa ise 21 2 6 1 Starting with the Rack 22 2 6 2 ESD Connection 23 2 6 3 Installation Procedure 23 2 6 4 Installing the Cable Holder 40 2 6 5 Ground Connections 41 2 7 Power Connections ns IAN 41 2 7 1 Powering Up the Switch 42 2 8 InfiniBand QSFP Cable Installation 43 2 8 1 Supported Approved Cables 44 2 8 2 Cable Power Classes 44 2 9 Hot Swap Insertion and Extraction 44 2 9 1 Power Supply
11. 17 Technologies 117 18 UL 4 LA 19 UL
12. amp 18 UL 4mA LA EIRETREREN Mellanox Technologies 99 o 19 UL CSA AEREA UL CSA 8 8 888 16 AWG 125 V 13 A ERED 5 5 4 5 lt HAR gt 1 0 300 V PVC 250 V 10 Ao 20 21 GND RER E AUR A RR GND KREZ ETE te DOS SERERE 75 A REA
13. Mellanox Technologies 72 A mr InfiniBand Switch Platform Hardware User M 5 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via FabricIT EFM A There are 2 Ethernet ports 1 for each management module that get connected to Ethernet switches These switches must be configured to 10 100M auto negotiation 5 The 155600 switch comes standard with a management software module for chassis management called FabricIT FabricIT is installed on all management modules for the managed switch systems based on InfiniScaleIV amp technology FabricIT includes a CLI WebUI SNMP and chassis management features for software and IB management soft ware OpenSM You can get more information via the Mellanox FabricIT EFM Software WebUI User s Manual or the Mellanox FabricIT EFM Software User Manual The managed switch system includes the following software components Embedded Subnet Manager SM e Chassis manager and system BIST SNMP agent 3rd party tool integration GUI Remote logging SSH telnet e Secured access in band and out band Pv4 IPv6 network stack The chassis manager will give the user access to Switch temperatures Power supply voltages Fan unit information Power unit information Flash memory Monitoring of AC power to the PSUs DC power out from the PSUs
14. 45 2 9 2 Leaf Boards sc A pg ee eti e een 46 2 93 Spine Boards ios a HEHE EE XE 50 2 9 4 Fan Modules ema 54 2 9 5 Management Module 57 2 9 6 Switch Shut Down Procedures 59 Chapter 3 Interfaces Rx Oe d RN ean e total out 61 3 1 LED Status lt eet a 61 3 1 1 Power Supply Unit LEDs 61 3 1 2 Leaf Board LED Indicators 62 3 1 3 Spine Board LED Indicators 63 3 1 4 Spine Side Panel Display LED Indicators 65 3 1 5 Management Module LED Indicators 66 Mellanox Technologies 3 InfiniBand Switch Platform Hardware User 3 1 6 Port Connector Interfaces 68 3 2 LAIT FOW aa ot ee er en a D Lac le ia ar 68 3 3 Cable Power Budget Classification 68 3 4 Management Module Interfaces 69 9 b I der aT hd ue fes 69 3 42 CONSOLE n a Yu ae Seg DEW en 69 3 4 3 MGT 69 3 44 USB eset ipsae Eire E Ema nr po fairs 70 3 4 5 Reset RST a0 a De Riga 70 Chapter 4 Chassis Power Up
15. 8 3 RM MIE LA 8 EEE RE E RARE E RU A ES ATE aa E BER A RAF RAMA RS VIRF ZE mo 9 Si META 10 YN PSU PERES LA FH ZI asi MIBK Mellanox Technologies 98 A 12 InfiniBand HEE RF InfiniBand ARENA ALMA IAE ES E LR EX BANMANEH 13 14 16 A UM O 17 L
16. PSU SFANS L FANS Master CONSOLE m The management module LEDs display the switch system operating conditions Table 9 LEDs Display for Normal Operation a NN Red or more of the spine fans is bad User should check individual spine fan LEDs for fault indications au Ata Normal LED Condition Description State STATUS Off No Power Green This LED Green System is up and running shows the sta Yellow System warning Attention needed such as overheating tus of the chas Red System not operational Diagnostics fail CPU hang sis HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational ellanox Technologies 66 Table 9 LEDs Display for Normal Operation Normal LED Condition Description a LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications MASTER Off this management module is not the master MM Green Green Management module is operating as a master 1 e the other management module is a slave Figure 63 Manage
17. Stickers unaffected by shipping Stickers showing excessive shock or tilt Mellanox Technologies 25 InfiniBand Switch Platform Hardware User 2 6 3 4 Opening the Container 1 Before starting the procedure put the ESD strap on and connect it to a valid ground Figure 9 Container Screws Container screws v It is highly recommended to have a screw gun or electric screwdriver for this step 2 Remove the four sides of the container by unscrewing the phillips head screws hold ing the sides on 3 Remove and put aside the box 4 Visually inspect the chassis make sure that there is no visible damage all 18 spines are installed 10 PSUs are installed for the chassis e all 8 fans are installed 1 management module is installed 5 Remove all protective plastic film from all sides and top of the chassis Mellanox Technologies 26 Q OOoRV SES InfiniBand Switch Platform Hardware User 4 The leafs are shipped separately A second redundant management module when ordered is shipped separately gt LA Do not remove any of the blocks or either of the lock down bars at this time 2 6 3 5 Installing the Shelf of the pallet Should the chassis start to fall move out of the way The chassis can The chassis is on ball bearings and can roll easily Be aware that the chassis can roll off A cause grave bodily harm should it fall on or near you
18. B for USA Canada Mexico Tai ACC000204 wan Power cord Type H for Israel ACC000205 Mellanox Technologies 91 InfiniBand Switch Platform Hardware User Table 17 Replacement Parts Ordering Numbers Continued Part Description OPN Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 Mellanox Technologies 92 G Installation Safety Warnings G 1 Installation Safety Warnings English 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight YN Use enough people to safely lift this product 3 Heavy Equipment LA This equipment is very heavy and should be moved using a mechanical lift to avoid injuries 4 Installation in Restricted Access Location This unit is intended for installation in a Restricted Access Location 5 Risk of Electric Shock Risk of Electric Shock LA With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 6 Over temperature This equipment should not be operated in a
19. Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low v It is recommended to set the top of the shelf at U5 Try and keep the chassis as low as possible Set the shelf from U3 to U5 Placing the shelf too low may make it impossible to tighten the shelf brackets in step 11 The maximum height that the chassis can be mounted will be determined by the maxi mum height that the fork lift can raise the pallet 2 Open the Installation kit and make sure that all of the parts are in the box See Figure 7 Installation Kit Parts The chassis should sit in the rack with the spine side as close as possible to the door This leaves maximum room for the cable management system The distance between the vertical supports must be between 65 cm 25 6 and 80 cm 31 5 If the distance in your rack is different move the vertical supports so that one pair is 10 cm 4in away from the door lt The side of the chassis with the spine units will sit flush with the vertical supports of the rack The side of the chassis with the QSFP connectors will sit at the edge of the shelf closer to the center of the rack LI Technologies 27 InfiniBand Switch Platform Hardware User Figure 10 Vertical Rack Supports Dimension Vertical supports 3 Decide which side of the rack you want the data connectors Figure 11 Placement of Chassis
20. TITE TELNET TT u a e E amp Er Mellanox Technologies 87 InfiniBand Switch Platform Hardware User Manual Figure 74 Rear Panel Leaf Side of the Switch LT 1 2 ACA ACS I ni it 1 LJ CONSOLE MOT sys AST PSU al TT RE RE 1 ii Management Modules A Leafs Each Leaf has 18 connector ports The switch can hold up to 36leafs Leaf Blanks must be in place when a slot is not populated with a Leaf This is necessary to ensure proper air flow for cooling the leafs and spines Mellanox Technologies 88 InfiniBand Switch Platform Hardware User Appendix E QSFP Interface M Table 16 InfiniBand QSFP Connector Pinout Connector Signal Description 1 GND Ground 2 Tx2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 5 Tx4n Transmitter Inverted Data Input 6 Tx4p Transmitter Non Inverted Data Input 7 GND Ground 8 ModSelL Module Select 9 ResetL Module Reset 10 Rx 3 3 V Power supply receiver 11 SCL 2 wire serial interface clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxin Rec
21. Interface 89 Appendix Replacement Parts Ordering Numbers 91 Appendix Installation Safety Warnings 93 G 1 Installation Safety Warnings English 93 G 2 Chinese 97 G 3 Avertissements de s curit pour l installation French 102 G 4 Installation Sicherheitshinweise German 107 G 5 Advertencias de seguridad de instalaci n Spanish 111 G 6 Russian 115 G 7 Avertismente privind siguran a la instalare Romanian 119 G 8 Sigurnosna upozorenja za instaliranje Croatian 124 Mellanox Technologies 4 InfiniBand Switch Platform Hardware User List of Tables Table 1 Revision History of this User s Manual 8 Table 2 Reference Documents and Websites 10 Tables Lear Status LED user are eke aa PRO DERE S EISE PROS 62 Table 4 Connector Physical and Logical Link Indications 63 Table 5 Spine Status LED ue GRE EEE ERE 64 Table 6 Spine Fan status LED ek Gs re ea as RC a 64 Table 7 Spine to Leaf IB Link Status 65 Table 8 LEDs Display for Normal
22. chassis failures querying for switch serial numbers revisions software version SwitchX FW version e switch temperatures ellanox Technologies 73 o InfiniBand Switch Platform Hardware User The manager also has the ability to burn new firmware and upgrade software on the switch 5 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It dis covers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as rout ing partitioning to the fabric devices You can view and configure the Subnet Parameters SM via the CLI WebUI Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based 5 2 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet WEB SNMP XML 5 3 Upgrading Software The new software and firmware updates are available to the user from the Mellanox Sup port website Copy the update to a known location on a Remote server within your LAN Use the CLI or the GUI in order to p
23. 4 Open the ejector until it is 45 degrees from the leaf Figure 40 Ejector Handle Ejector handle gt Pull the leaf out of the chassis Push the ejec tor handle 5 Pull out the module half way through the guiding rails using the ejector handle 6 Lock the ejector handle Mellanox Technologies 47 InfiniBand Switch Platform Hardware User Manual 7 Hold the body of the leaf both sides and remove it from the chassis The board is short therefore do not let go of it while sliding it out Figure 41 Leaf Release Push here to release the ejector handle Pull the ejector handle to remove the leaf 17 OAD 0 E NA Y Vo YA We 3 SA Vu 2 9 2 2 Inserting Leaf Board When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert the leaf board 1 Check for foreign objects in or mechanical damage to the chassis leaf slots 2 Check leaf mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 42 Figure 42 Intact vs Defected Signal Connectors Mellanox Technologies 48 InfiniBand Switch Platform Hard
24. When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 14 Equipment Installation Mellanox Technologies This equipment should be installed replaced and or serviced only by trained and qual ified personnel 94 Re 15 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula N tions 16 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 17 Installation Codes A This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electrical Code 18 Battery Replacement Warning Replace only with UL Recognized battery certified for maximum L abnormal charging current not less than 4mA There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispose of used batteries according to the instructions 19 UL Listed and CSA Certified Power Supply Cord A For North American power connection select a power supply cord that is UL Listed PA N and CSA Certified 3 conductor 16 AWG term
25. Congestion control Adaptive routing e Port mirroring Chassis High Availability e sMB High Availability 1 3 Power Supply Redundancy This chassis has N 1 power supply redundancy With a full complement of power sup plies one power supply can be removed and replaced without shutting down the chassis or stopping the flow of data 1 Features for a future release Mellanox Technologies 14 InfiniBand Switch Platform Hardware User 2 Installation A This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports of the rack Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your Chassis will be shipped in two packages one with the all of the leafs leaf package that are ordered and an extra management module should it be ordered and the second pack age chassis package with all of the rest of the parts The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instructions to make the initial configuration This unit is intended for installation in a Restricted Access Location A restricted ZA access area can be accessed only through the use of a special tool lock and key or othe
26. Figure 44 Management Module Numbering MNGI management module 1 top MNG2 management module 2 bottom Removing spine 1 causes management module 1 to reset Removing spine 2 causes management module 2 to reset Warning If the spine you want to hot swap is connected to the master management module the management module will reboot when you take out the spine If you have only one management module the chassis will crash Figure 45 Spine Board Extraction Pull the latches to remove the spine TESTO RE LEE 921 2325 33 35 2000095906 000000000 0000000 55 5 299009000 If you need to hot swap Spine 1 or Spine 2 1 Check to see if the spine you need to remove is connected to the master management module See Figure 44 If you have only one management module the chassis will crash 2 Follow the steps in Section below Extracting Spine Boards Except 1 or 42 1 Run the shut down command no power enable lt module gt For example to shut down spine 06 run the command below switch master config no power enable 506 2 Push outward on the ejector handles to unlock the ejectors from the chassis ellanox Technologies 51 InfiniBand Switch Platform Hardware User Manual 3 Open the ejectors until they are at a 45 degree angle from the module Do not use the fan FRU handle to extract the spine board 4 Pull out the module half way through the guiding rails using both eject
27. 2 6 1 2 Preparing the Rack The rack may need to be modified to accommodate the 155600 chassis The distance between the vertical supports must be between 65 and 80cm Figure 6 Distance Between the Vertical Supports Vertical supports This edge of the shelf must be as close as possible to the inside of the door when closed Between 65 and 80 cm The vertical supports may have to be moved so that the vertical support closest to the door is a minimum of 6 cm from the inside of the door This is required so that the shelf will not hit the door If you move the vertical supports make sure that you reinstall them vertically plumb A Mellanox Technologies 22 InfiniBand Switch Platform Hardware User 2 6 2 ESD Connection Before starting any procedure on the 155600 switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit 2 6 3 Installation Procedure Installing the chassis will be a lot easier if you can remove the sides of the rack LA You will need a fork lift to move and insert the chassis into the rack 2 6 3 1 Requirements You will need 2 phillips screwdriver 3 phillips screwdriver a fork lift e a grounding lug measuring tape ground wire of sufficient length and gauge to properly ground
28. 32 hex head bolts 20 for the lock down bars and 12 for the upper brackets to upper bracket offsets 12 flat head screws for the upper bracket offsets 1 socket nut driver 8mm 1 socket nut driver 10mm 1 socket nut driver 14mm 1 box wrench 6 two hole lock washers for lock down bars Top filler panel Parts included in the cable management installation kit 1 cable management rack RH 16 caged nuts M6 9 cable management shelves 1 cable management rack LH 16 M6 bolts Figure 7 Installation Kit Parts Shelf locktite Lock down bar Flat head screws with Shelf extension Upper bracket washer lt Upper bracket Upper bracket offset 2 6 3 3 Container Mishandling 6 hex head bolt Phillips pan head screw 8 hex head bolt Caged nuts Lock washers Lock washers ANS 5 Flat washers Flat washers The container has shock and tilt stickers applied These will turn red if the container has been mishandled or roughly handled Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped If one or more are red Mellanox Technologies 24 InfiniBand Switch Platform Hardware User Manual notify shipper and Mellanox This on its own does not indicate damage to the con tents But be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped Figure 8 Shock and Tilt Stickers
29. 71 4 1 Power Supply and Spine Board Indicator Status at Power ON 72 Chapter 5 Switch Management Tools 73 5 1 InfiniBand Subnet 74 5 2 Accessing the CPU via the Ethernet Connector 74 5 3 Upgrading 5 74 Chapter 6 Troubleshooting 75 6 1 Power Supply Unit Ok 75 6 2 Leaf Board canes here Ata en wee Re ced Be 75 6 3 Management 76 Spinei BORNE sur is Re sd ped ee ese rece eda de 77 6 5 E Dui ls bee 77 Chapter 7 Disassembly and Disposal 78 7 1 Disassembling the Chassis 78 7 1 1 Removing the Chassis cea 80 7 1 2 Removing the Bottom Shelf 81 7 2 Disposal eean eda dak g is dp a ce ate Th 81 Appendix A Specification Data 82 EMI Certification Sade e e Rex e 83 A Approved Gables ads ae eat 83 84 Appendix Calculating the Weight of a Customized Chassis 85 Appendix C Calculating the Power of a Chassis 86 Appendix D Mechanical Drawing 87 Appendix
30. E 22 23 RS232 UL DP 1 DP 2 LPS 20 24 ID Pe Adie 3648 BA E m CLASS 1 872 SRRA IEC 60 825 1 1993 A1 1997 A2 2001 EN 60825 1 1994 1 1996 A2 2001 FR 3648 E m 25 HEAR f im E SCAR m A MER APE Mellanox Technologies 100 26 BR E 27 WEEE Ef WEEE 2002 96 EC EEE H ARENE ERE EEES 28
31. Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Y okneam 20692 USA Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 04 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BidgeX ConnectX Connect IB CORE Direct InfiniBridge In fini Host InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendXTM FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3178 InfiniBand Switch Platform Hardware User Table of Contents Listiof Tables rn ae age ao 5 List Of Figures coo a AAA ew Ew 6 Revision History lt a xe eee eee aa i 8 About this Manual akc en 9 Chapter 1 Overview isis a 11 Tl Prod ct Information s ess eserse e 11 1 1 1 Serial Number and Product Version Information 11 14 2 License Key uses io eem Y DR ANA 12 1 1 3 Management Module MAC
32. h This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 23 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 24 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 25 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user 26 Do Not Use the Switch as a Shelf or Work Space ellanox Technologies Caution Slide rail mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance 96 27 WEEE Directive According
33. 26 lt 3 Top of shelf 4 If you are installing the cable management system place 14 caged nuts on the other side of the rack in the vertical supports Figure 22 Location for the Lock Down Bar Bottom of the chassis 5 Unscrew both lock down bars With both lock down bars removed the chassis can roll Be very careful that the chassis does not roll off of the pallet 6 Place the chassis in front of the rack and as close as possible to the rack Figure 23 Fork Lift Container floor Hinged extension Mellanox Technologies 34 InfiniBand Switch Platform Hardware User Figure 24 Mechanical Lift 7 Raise the chassis so that it is 5cm above the shelf in the rack 8 Raise the hinged extension on the container floor 9 Lower the chassis until the hinged extension is sitting on the shelf extension 10 Push the chassis from the container to the shelf Save the container pallet to remove this chassis It will be necessary for the removal and disposal procedures Mellanox Technologies 35 InfiniBand Switch Platform Hardware User Figure 25 Face Plate Mounting Bolt Locations Holes in the faceplate for i TET mounting on HAE pi the rack The holes are separated by 4U units KS 11 Slide the chassis
34. aa ae kan a RS 25 Figure 9 Container Screws sue VERTES per eb ER ERE RE AE de eke 26 Figure 10 Vertical Rack Supports Dimension 28 Figure 11 Placement of Chassis in Rack 28 Figure 12 Adjustable Shelf Brackets 29 Figure 13 Placing the Shelf Bracket 29 Figure 14 Caged Nut Locations for the Shelf Brackets 30 Figure 15 Shelf Brackets uses Reb eR ER Ar a RR e Bee Eee e age 30 Figure 16 U Numbering Above Shelf 31 Figure 17 Putting the Chassis in the Rack Using a Fork Lift 32 Figure 18 Putting the Chassis in the Rack Using a Mechanical Lift 32 Figure 19 Place the Upper Bracket Offset 33 Figure 20 Attach the Shelf Extension 33 Figure 21 Place Caged Nuts tr wk ERI bp et ms ES 34 Figure 22 Location for the Lock Down Bar 34 Figure 28 Fork Lit ss alta dno va vno Bi A etc tud dau rb tat OE 34 Figure 24 Mechanical Lift ee us Rete Fa Race E Nace ea gota Aen E 35 Figure 25 Face Plate Mounting Bolt Locations 36 Figure 26 Upper Bracket Installation 37 Figure 27 Upper Brackets a
35. conform ment la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 18 Remplacement de la batterie Avertissement ne remplacer qu avec une batterie UL certifi e pour accepter un cou rant de charge anormal maximal sup rieur ou gal 4 mA LA Si la batterie n est pas remplac e par un type correct il y a un risque d explosion Les batteries usag es doivent tre mises au rebut conform ment aux instructions 19 Cordon d alimentation UL Listed et certifi CSA A Pour le branchement lectrique en Am rique du Nord utiliser un cordon d alimenta y N tion UL Listed et CSA Certified 3 conducteurs calibre 16 AWG avec une prise N moul e 125 V 13 A faisant au moins 1 5 m de long six pieds et au plus 4 5 m m Pour le branchement lectrique en Europe utiliser un cordon d alimentation au format international harmonis marqu lt HAR gt 3 conducteurs d au moins 1 mm2 de sec tion 300 V avec une gaine isolante en PVC Le cordon doit avoir une prise moul e 250 V 10 A 20 Courant de fuite lev Avertissement courant de fuite lev une connexion la terre est indispensable avant de brancher l alimentation 21 Information d ajout de branchement la terre GND Avant de relier cet appareil au secteu
36. 1 minute after removing the defective part before inserting the new part This is necessary so that the manage MN N A 4 N ment module will start anew cycle checking through the leafs and spines for the FW versions Extracting a Management Module Management modules are located on the leaf side above the leafs There are two places to install the management modules Only one management module is required to run the switch system A Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 54 1 Run the shut down command no power enable lt module gt For example to shut down management module 2 run the command below switch master config no power enable MGMT2 Disconnect all cables connected to the management module Push outward on the ejector handles to unlock the ejectors from the chassis Open the ejectors until they are 45 degrees from the module Pull out the module half way through the guiding rails using the ejector handle mm EA Lock the ejector handle ellanox Technologies 57 InfiniBand Switch Platform Hardware User Manual 7 Hold the body of the board on both sides and remove it from the chassis The board is short therefore do not let go of it while sliding it out Figure 54 Management Module nom ee SEAN Ejector handles 2 9 5 1 Inserting a Management Module When hot swapping any of
37. 55 kgs gt 55 kgs 3 Equipos pesados Dado que el equipo es muy pesado se debe mover nicamente mediante un elevador mec nico para evitar lesiones 4 Instalaci n en lugares con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 5 Riesgo de descarga el ctrica N Riesgo de descarga el ctrica Con el m dulo del ventilador quitado se obtiene acceso a las clavijas de alimentaci n desde dentro de la cavidad del m dulo NO introducir herramientas ni partes del cuerpo en la cavidad del m dulo del ventila dor 6 Sobretemperatura AA No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 Adem s para garantizar una circulaci n de adec se debe dejar como minimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 7 Apilamiento del chasis Los chasis se deben apilar sobre otros equipos La ca da del chasis podr a causar lesiones corporales as como da os al equipo 8 Conexi n redundante de fuente de alimentaci n peligro de descarga Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa est quitada o no est bien cerrada 9 Fusible neutro o de polo doble Dos fusibles uno en el polo y otro en el neutro Quitar lo
38. Leafs Leaf blanks e Management modules Fill in the table below to calculate the weight of your system Number of FRUs FRU Type Weight of 1 FRU Total Weight FRU Type Chassis as shipped of Spines 3 71 of leafs 2 76 of leaf blanks 0 764 of management modules 3 15 Total This total is in kilograms Multiply the total by 2 2 to get the total weight in pounds Mellanox Technologies 85 io InfiniBand Switch Platform Hardware User Appendix C Calculating the Power of a Chassis To calculate the power consumption of a chassis add the power of the fans spines leafs and management modules Table 15 Power Consumption of Chassis Parts Part Power Passive Cables Drin Pen ja any IB QDR leaf 82 28W 121 45W Spine 102 86W Not applicable Fan 15 1W Not applicable Management module 15W Not applicable To calculate the chassis power at typical conditions fill in the table below Number of FRUs FRU Type de OL Total Power FRU Type of spines 102 86 W of leafs passive cables 82 28 W of leafs optical cables 121 45W of management modules 15W for the PPC 460 manage ment module 8 Chassis fan 15 1W 120 8W Total DC power Total AC power Total DC power efficiency 85 Mellanox Technologies 86 InfiniBand Switch
39. Technologies 126 SN m 21 Dodaj informaciju GND spoju Prije spajanja ovog uredaja na elektri ni vod vijci terminala zastitnog uzemljenja uredaja moraju se spojiti na zastitno uzemljenje u instalacijama zgrade Informacija o GND spajanju Instalacije zgradi omogucit ce sredstva za spajanje na zastitno uzemljenje a servisno osoblje ce trajno odrZavati spoj opreme s uzemljenjem SERVISNO OSOBLJE e provjeriti omogu uje li ne zidna uti nica napajanja za opremu spoj na za titno uzemljenje zgrade Ako ne SERVISNO OSOBLJE e orga nizirati instaliranje VODI A ZA TITNOG UZEMLJENJA iz posebnog terminala za titnog uzemljenja na icu za titnog uzemljenja u zgradi Oprema e se instalirati u podru je gdje postoje ekvipotencijalni spojevi poput telekomunikacijskog centra ili namjenske ra unalne sobe 22 Instalacijski kodovi Ovaj se ure aj mora instalirati sukladno najnovijoj verziji nacionalnih elektri nih kodova dr ave U Sjevernoj Americi oprema se mora instalirati sukladno va e im zahtjevima navedenim u US National Electrical Code i Canadian Electrical Code 23 Interkonekcija ure aja Kabeli za spajanje na jedinicu RS232 i Ethernet su elja moraju biti s UL certi fikatom vrste DP 1 ili DP 2 Napomena kad se nalazi u krugu bez LPS vodi a Za tita od strujnog preoptere enja Uvijek dostupni odobreni za titni ure aji od strujnog preoptere enja nazivne struje od 20 A moraju se
40. UL certified type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection readily accessible Listed branch circuit overcurrent protective device rated 20 must be incorporated in the building wiring Mellanox Technologies 18 InfiniBand Switch Platform Hardware User Manual 24 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards 4 IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 25 Proper Enclosure 4 A suitable electrical mechanical and fire enclosure shall be provided by the end prod N uct manufacturer and or the end user 26 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance 27 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way ama 28 Country of Norway Power Restrictions Th
41. baterie recunoscut UL certificat pentru curent de inc rcare anormal maxim de minimum 4 mA LA Exist risc de explozie n cazul n care bateria este nlocuit cu o baterie necorespunz toare Eliminati bateriile folosite n conformitate cu instruc iunile 19 Cordon de alimentare electric nregistrat UL i certificat CSA A Pentru conectarea la o surs de alimentare pentru America de Nord selecta i un cordon A N de alimentare care este nregistrat UL i certificat CSA cu 3 conductoare 16 AWG y W terminat cu o fi turnat cu putere nominal egal cu 125 V 13 A cu o lungime de EROT minimum 1 5 m ase picioare dar nu mai lung de 4 5 m Pentru conectarea la o surs de alimentare n Europa selecta i un cordon de alimentare care este armonizat la nivel interna ional i marcat lt gt cu 3 conductoare cu minimum 2 fire de 1 0 mm cu putere nominal egal cu 300 V i cu o manta izolant din PVC Cordonul de alimentare trebuie s fie prev zut cu o fi turnat cu putere nominal egal cu 250 V 10 A 20 Curent de scurgere de nalt frecven Avertisment Curent de scurgere de nalt frecven Imp m ntarea este esen ial nainte de a conecta sursa de alimentare 21 Ad ugarea de informa ii privind legarea la p m nt nainte de a conecta acest dispozitiv la o linie electric de energie uruburile bornelor de protec ie de legare la p m nt ale acestui
42. de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 20 Alta corriente de fuga ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n Mellanox Technologies 113 A os 21 Agregar informaci n de conexi n a tierra GND 5 Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DUCTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 22 C digos de instalaci n gt Este dispositivo se debe instalar conforme a la
43. de l immeuble doit int grer un dispositif certifi de protection contre les surintensit s calibr 20 A et ais ment accessible 24 Exposition des rayonnements dangereux Attention l utilisation de commandes l ajustement ou la conduite de proc dures autres que celles indiqu es ici pourraient conduire une exposition dangereuse des rayonnements Pour les produits disposant de ports optiques PRODUIT LASER DE CLASSE 1 et r f rence aux plus r centes normes relatives aux lasers IEC 60 825 1 1993 A1 1997 A2 2001 et EN 60825 1 1994 A1 1996 A2 2001 Pour les produits disposant de ports optiques 25 Armoire de protection appropri e Une enceinte appropri e aux niveaux lectrique m canique et incendie doit tre four nie par le constructeur du produit fini ou par l utilisateur 26 Ne pas utiliser comme tag re ou plan de travail Attention un quipement coulissant ou mont sur rail ne doit pas servir d tag re ni de plan de travail Les rails ne sont pas destin s faire coulisser l unit hors du rack Ils sont destin s une installation permanente l emplacement final pas pour l entretien ni la maintenance 27 Directive DEEE X Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doivent tre collect s s par ment et ne pas tre mis au rebut avec les d chets m nagers habituels Ce produit et toutes ses pi ces doive
44. dispozitiv trebuie s fie conectate la priza de p m nt de protec ie din instala ia cl dirii Informa ii privind legarea la p m nt Instala ia cl dirii va asigura un mijloc de conectare la priza de p m nt iar echipamen tul va fi n mod permanent conectat la priza de p m nt de un agent de ntre inere UN AGENT DE NTRE INERE va verifica dac priza borna de ie ire prin interme diul c reia va fi alimentat echipamentul cu energie electric asigur o conexiune la priza de p m nt a cl dirii n caz contrar AGENTUL DE NTRE INERE va dispune instalarea unui CONDUCTOR DE PROTEC IE DE LEGARE LA P M NT de la borna separat de protec ie de legare la p m nt la cablul de protec ie de legare la p m nt al cl dirii Echipamentul va fi instalat ntr o zon unde exist leg turi echipotentiale precum un centru de telecomunica ii sau o camer a computerelor dedicat 122 22 Codurile de instalare Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor IN IN Y N electrice nationale ale rii n cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerin elor aplicabile din Codul electric na ional al SUA i Codul electric canadian 23 Interconectarea unit ilor f Cablurile pentru conectarea la unitatea RS232 si la interfetele Ethernet trebuie s fie de A tipul DP 1 sau DP 2 certificate UL Not c nd se reg sesc ntr
45. elektritne energije Ovaj proizvod ukljucuje redundantno napajanje ili prazan prostor na njegovu mjestu U slu aju praznog prostora za napajanje nemojte rukovati proizvodom ako je pok lopac uklonjen ili ako nije dobro pri vr en gt 9 Dvopolni neutralni osigura i Ovaj sustav raspola e dvopolnim neutralnim osigura ima Uklonite sve kabele napa janja prije otvaranja poklopca proizvoda ili dodirivanja unutarnjih dijelova 10 Vi estruki ulazi za napajanje Rizik od strujnog udara i opasnost od elektri ne energije PSU jedinice su neovisne gt Odspojite sva napajanja kako biste osigurali stanje bez napajanja unutar platforme preklopnika 11 Tijekom udara munje Opasnost od elektritne energije y Y Tijekom djelovanja munja nemojte raditi na opremi ili spajati ili odspajati kabele 12 Spajanje Odspajanje bakrenog kabela InfiniBand Bakreni kabeli InfiniBand su te ki i nesavitljivi 1 kao takvi se moraju paZljivo priklju iti na ili isklju iti iz konektora Obratite se proizvo a u kabela za posebna upo zorenja upute 13 Monta a ormari a i servisiranje opreza kako bi se osiguralo da sustav ostane stabilan Op enito trebali biste ispunja Kad se proizvod montira ili se servisira u ormari u moraju se poduzeti posebne mjere A vati ormari s opremom po ev i od dna prema vrhu Mellanox Technologies 125 SN ESA 14 Instaliranje Ovu bi opremu trebalo instalirati zamje
46. fatal error this spine needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 3 2 Fan LED The spine fan indicator LED has the following LED assignment Table 6 Spine Fan Status LED LED Color LED Description Solid Green Spine Fan is OK Flashing Green Spine Fan needs replacing Solid Yellow One or more of the fans in this spine is not working Each spine has two fans in the fan module 3 1 3 3 Spine to Leaf IB Connection Status LEDs The leaf connection status on each spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per leaf per spine and a maximum of 1 LED per leaf These LEDs indicate a valid connection between a leaf and a spine 4 Mellanox Technologies 6 Table 7 shows leaf to Spine status according to LED condition Table 7 Spine to Leaf IB Link Status LED Condition LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Solid Orange Physical connection 3 1 4 Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition Figure 61 Spine Side Panel Display Status Indications m m m El El m STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER Table 8 LEDs Display for Normal Operation Normal LED Condition Description State STATUS Off No Power Green Gr
47. found on the leaf side and four leaf fan modules are found on the spine side The leaf side fans are not interchangeable with the spine side fans Air flow through the leafs is independent of the air flow through the spines Mellanox Technologies 54 InfiniBand Switch Platform Hardware User Figure 50 Leaf Fan Locations on the Chassis Fans Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 51 2 Slowly pull out the fan module using the handle Figure 51 Leaf Fan Module Extraction Latch Release Mellanox Technologies 55 InfiniBand Switch Platform Hardware User Manual Inserting the Leaf Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module Tf the fan module stops before it goes in all of the way it is inserted incorrectly or it may be for the other side of the chassis 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 2 9 4 2 S
48. ind pendantes gt Pour s assurer que le commutateur est bien hors tension d branchez toutes les alimen tations 11 En cas d orage danger d lectrocution Pendant un orage ne pas travailler sur l quipement ni brancher ou d brancher des c bles gt 12 Connexion et d connexion du c ble InfiniBand en cuivre A N Les c bles InfiniBand en cuivre sont lourds et peu flexibles Par cons quent il faut N proc der avec soin pour les brancher ou les d brancher des connecteurs Consulter le fabricant du c ble pour obtenir des instructions ou des avertissements sp cifiques vr 13 Montage en rack et maintenance Lors du montage ou de la maintenance de ce produit dans un rack il faut faire speciale ment attention pour s assurer que l ensemble reste stable En r gle g n rale le rack doit etre rempli en commengant par le 14 Installation de l quipement Cet quipement ne doit tre install remplac et maintenu que par un personnel form et qualifi 15 Mise au rebut de l quipement IN 4 N x N La mise au rebut de cet quipement doit se faire conform ment toutes les lois et 4 N r glementations nationales Mellanox Technologies 104 Me MEN 16 Codes lectriques locaux et nationaux Cet quipement doit tre install conform ment aux codes lectriques locaux et nation V aux 17 Codes d installation N Cet appareil doit tre install
49. module IN There are two non interchangeable types of leaf fan modules If the new leaf fan does b not go into the chassis confirm that it is the correct fan module LA Replace defective leaf fan modules as soon as they are identified Mellanox Technologies 76 InfiniBand Switch Platform Hardware User Should any of the boards shut down due to over temperature follow the procedure starting in step 2 Issue 3 Yellow LED for the Spine Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Determine which spine has a defective fan by checking the Fan LEDs on all of the spines 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 5 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 9 4 on page 54 6 Replace the spine fan module LA Replace defective spine fan modules as soon as they are identified 6 4 Spine Board Issue 1 The yellow LED on the Spine board is lit 1 Check the MLNX OS management for confirmation and possible explanation of the al
50. onto the shelf and determine the location of the upper brackets on the vertical supports on the connector side of the chassis Y Do not move the forklift away from the rack just yet 12 Confirm that the caged nuts for the faceplate are in the correct location 13 Slide the chassis out just far enough to install the upper brackets See Figure 26 Mellanox Technologies 36 InfiniBand Switch Platform Hardware User Manual Figure 26 Upper Bracket Installation 117cm from the shelf to this hole Use the socket wrench with both extensions to tighten these bolts 9 9 AAA Figure 27 Upper Brackets and Offset Location 14 Screw the upper brackets onto the vertical supports and tighten in place See Figure 27 You will need the socket wrench with both extensions to tighten these bolts 15 Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack support 16 Make sure the upper brackets are installed in their correct location 17 Screw the 14 bolts through the faceplate into the caged nuts DO NOT TIGHTEN Figure 28 Offset to Upper Bracket Bolts Mellanox Technologies 37 InfiniBand Switch Platform Hardware User 18 For each offset and upper bracket screw six bolts through the offset into the upper bracket See Figure 28 Use one flat washer and one lock washer for each of these bolts DO NOT TIGHTEN Figure 29 Location for the Lock Do
51. rack These screws are located in the faceplate on the spine side of the chassis 3 Remove the upper brackets from the chassis 4 Remove the lock down bars Mellanox Technologies 80 ee InfiniBand Switch Platform Hardware User Figure 72 Mechanical Lifting Device Put the container base saved from the installation on the fork lift Raise the container base to 1 cm below the level of the shelf Move the chassis away from the rack and onto the container base DADA Put wedges around the chassis to prevent the chassis from rolling off of the container base 9 Move the chassis away from the rack 10 Dispose of the Chassis in a legal and environmental way 11 Go to the Mellanox website for detailed instructions for disassembly of the chassis according to the WEEE Directive 7 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack 2 Remove the filler panels 3 Remove the shelf 4 Remove all of the caged nuts 7 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Go to the Mellanox website for detailed instructions for disassembly according to the WEEE Directive Mellanox Technologies 81 InfiniBand Switch Pla
52. the password is deleted You will then be able to enter without a password and make a new password for the user admin Mellanox Technologies 70 Mey 4 Chassis Power Up N N A Before starting any procedure on the IS5600 system put an ESD prevention wrist strap on your wrist and connect to the IS5600 chassis With N 1 PSU redundancy the chassis must be started with a full complement of possi ble PSUs thereafter it run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running FabricIT EFM may power down some leafs If this happens it will be necessary to reboot the chassis once the defec tive PSU has been replaced Check all FRUs for proper insertion and seating before connecting the AC power cords Boards Power supplies Leaf fan modules Spine fan modules Insert all leafs that you plan to use in the chassis Start at the bottom of the chassis and work your way up Insert the first two spines in the top two slots Insert the rest of the spines from the bottom of the chassis up to slot 43 Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all leaf and spine mounting screws Con
53. the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a management board 1 Check for foreign objects in or mechanical damage to the chassis management slots 2 Check management mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criterion shown in Figure 55 Figure 55 Intact vs Defected Signal Connectors 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Mellanox Technologies 58 ee InfiniBand Switch Platform Hardware User Figure 56 Figure 56 Intact vs Defected Power Pin Holders 5 Start with the ejector handles fully open that is at 45 degrees to the front panel of the management 6 Holding the management by its sides carefully set the management board into the chassis 7 Using only the management ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar Do not apply excessive force to slide in the management board If you feel resistance remove the management and double check both the chassis and management for any damage 8 Catch the hooks onto t
54. to insert the spine board 6 Holding the spine by its sides carefully set the spine board into the chassis 7 Using only the ejector handles slowly slide the board into the chassis until the hooks reach the vertical bar AN Do not apply excessive force to slide in the spine board If you feel resistance remove A N the spine and double check both the chassis and spine for any damage 8 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handles onto the board Fan Modules There are eight fan modules on the chassis for the leafs They are located on both sides of the chassis four on the leaf side corners and four on the spine side corners When a fan module is not functioning the status LED on the fan will light up Each spine has an individual fan module that contains two individual fans Should a sin gle fan fail the Fan Status LED on the spine and the S Fan LED on the management module will light indicating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs numbers and are not interchangeable The fans on the leaf side of the chassis are PP There are two different fan modules for the chassis These fans have different model A MTF005001 whereas the fans on the spine side of the chassis are MTF005002 Leaf Fan Module There are eight Leaf fan modules Four leaf fan modules are
55. un circuit non LPS Y N Protec ie la supracurent Un dispozitiv de protec ie la supracurent nregistrat n circui AA tul de ramificare u or accesibil i cu o putere nominal egal cu 20 A trebuie s fie integrat n cablajul cl dirii SS 24 Expunerea la radiatii periculoase Atentie Utilizarea elementelor de comanda sau ajustarea sau executarea altor proce duri decat a celor indicate in acest document poate avea ca rezultat expunerea la radiatii periculoase PRODUS LASER CLASA 1 si referire la cele mai recente standarde pentru laser IEC 60 825 1 1993 A1 1997 A2 2001 si EN 60825 1 1994 A1 1996 A2 2001 25 Carter adecvat SS IN A N Un carter electric mecanic si ignifug adecvat va fi furnizat de fabricantul produsului N final si sau utilizatorul final 26 Nu utilizati comutatorul ca raft sau spatiu de lucru Atentie Echipamentul montat pe o linie de alunecare sin nu va fi utilizat ca raft sau spa iu de lucru Scopul sinelor nu este de a glisa unitatea de pe rack Acestea sunt des tinate instal rii permanente numai la punctul final de oprire i nu vor fi folosite pentru depanare i ntre inere E 27 Directiva DEEE conformitate cu Directiva DEEE 2002 96 CE toate deseurile de echipamente elec trice i electronice EEE trebuie colectate separat i nu trebuie eliminate mpreun cu de eurile menajere obi nuite Eliminati acest produs i toate componentele sale n mod res
56. versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 23 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una corriente nominal de 20 A 24 Exposici n a radiaci n peligrosa Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligro 05 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes 60825 1 2007 03 y EN 60825 1 2007 25 Recinto adecuado ellanox Technologies El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen dio 114 26 No utilizar el conmutador como estante ni espacio de trabajo Cuidado Equipos montados en deslizadores o rieles no se deben utilizar como estantes ni como espacio de trabajo La finalidad de los rieles no es de
57. von unten nach oben mit Ger ten f llen 12 Ger teinstallation N IN 4 Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge N tauscht oder gewartet werden Mellanox Technologies 108 SAA 13 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 14 Regionale und nationale elektrische Bestimmungen Bestimmungen installiert werden N N Y N Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen N 15 Installationscodes Dieses Ger t muss entsprechend der aktuellsten Version des National Electrical Code j installiert weden In Nodamerika muss das Ger t gem den geltenden Anforderungen y des US National Electrical Code und des Canadian Electrical Code installiert werden 16 Akkuaustausch Warnung Nur durch von UL anerkannten Akkus ersetzen die f r maximalen anor l malen Ladestrom von nicht weniger als 4mA zertifiziert sind Es besteht Explosiionsgefahr wenn der Akku durch einen Akku eines falschen Typs ersetzt wird Akkus gem den Anweisungen entsorgen 17 UL und CSA zertifiziertes Netzkabel A F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi g fed y N 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit a einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F
58. 86W includ ing 2 fans Max 110 32W including 2 fans Typ 15 1W Max 135W Typ 15W Max 22W 2 0W power level 2 648 ports Passive 5841W Active 6678W including QSFP at 1W Optical 7499W including QSFP at 2W Optical 9119W including QSFP at 2W Mellanox Technologies 82 InfiniBand Switch Platform Hardware User Table 14 155600 Switch Specification Sheet Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 20Gb s 40Gb QoS Management 5 8 InfiniBand Virtual Lanes for all ports Baseboard Performance and Device management Agents for full Infini Band In Band Management Safety EMC ENVIRONMEN TAL ACCOUSTIC SOUND POWER LEVEL US Canada cTUVus EU IEC60950 International CB USA FCC Class A Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 76 4 dB A Scalability and Performance Replaceable Parts Switching Perfor Simultaneous wire speed any port to Leafs Up to 36 leafs 18 ports mance Addressing any port 48K Unicast Addresses Max per Subnet 16K Multicast Addresses per Subnet Power Supplies Leaf Fan Mod each 10 Power Supplies standard with 2 optional re
59. Band Switch Platform Hardware User 1 Overview This User Manual provides an overview of the IS5600 QSFP Modular InfiniBand Switch Platform known in this document as the chassis or switch and its operational environ ment Mellanox 155600 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers EDC High Per formance Computing HPC and Embedded environments Networks built with the 155600 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 4 generation InfiniScale InfiniBand switch device 155600 systems provide up to 40Gb s full bidirectional band width per port With up to 648 ports in a 31U rack space these systems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard opera tion within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system The ini tial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configuration Once connected to the network the Sub net Management software automatically discovers and configures the fabri
60. Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one A Should any of the boards shut down due to over temperature wait 5 minutes and then follow the procedure starting with step 2 Issue 3 The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable Issue 4 The Activity indication does not come on Check that the Subnet Manager has been started Mellanox Technologies 75 or InfiniBand Switch Platform Hardware User 6 3 Management Module Issue 1 Yellow Status LED for the Chassis on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Reset the master management module by pushing the rest button If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master AA If there is only one managem
61. Mellanox TECHNOLOGIES 648 Port InfiniBand QDR SwitchX Switch Platform Hardware User Manual PN MIS5600Q 10DNC Rev 2 7 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE
62. PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running FabricIT EFM may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two simple ways to reboot is to use the reboot com in the CLI or reboot through the Web GUI The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of current so that the total current is always divided by the number of working PSUs 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cords into grounded outlets A fully loaded 155600 switch system can draw 8 kW 67A 120V or 34A 240V of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit Figure 35 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and Gafahr des elektrischen Schocks Risque de choc et de danger energy hazard Entferrnen des Netzsteckers e lectriques Le de branchment The PSUs are all independent e
63. Platform Hardware User Appendix D Mechanical Drawing Figure 73 Front Panel en Leaf Fan Module HH 155600 Te lalele dale nnn N VE ba 000000000 000000000 00000000 a IYI TT CD TTT AN ee DA A AO TT mm m E ng 000000000000000000000 A s nl lla AS a NA A 56 covers for power suppl N ee imis PPY JE Inlialullalulalalln nn 00000 0000 00001 AT lalalalalala al ah NUD 0000000000000 PSU 1 PSU 3 7 0000000 OO 100 240 We it MATITE LETI TINITINI TTA 1285mm AA OA ROM REN Mes H o a TTF ITE EFE ER ER 511 fesse lof lala dom 8 J i PSU 5 0100110010000000100100210000001000000000 II TATTET 612 nll lla ll ll la al ll smin PTT a TITI T AAA 13 0000000000000 iTe A ITI EIER RT II TITI ee ei oll lm lalele eee nOn T en 15 h LR let al VITEETTETTTURETETTYTE LE TT Ca 18 spines W Sa ao ele loa T T PSU 7 PSU 9 LITTI TEO LETITE TTTE TINA TI 17 ally In il lla Ih ll NI En adobe n ofere ete n 8 E 00000 D 00000000 00000000 000000 000000000
64. The cord must have a molded plug rated at 250 V 10 A 20 High Leakage Current LA Warning High leakage current Earth connection essential before connecting supply 21 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation LA GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 22 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 23 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be
65. Y N EEE TN IT Mellanox Technologies 101 29 China CCC Warning Statement English translation disconnect all power sources before service For non tropical use Dan hab yungh youq gij dienheiq diuzgen dwg ad haenx ancienz saw 2000m Warning for Class A AP URB RUE English translation of above statement This is a Class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures G 3 Avertissements de s curit pour l installation French 1 Instructions d installation Veuillez lire la totalit des instructions d installation avant de relier l quipement au secteur Mellanox Technologies 1 0 2 2 Blessures cause du poids Pr voyez assez de personnel pour soulever ce produit en toute s curit he e gt 121 Ibs gt 55 kgs 70 121 lbs lt 40 lbs 40 70 Ibs 32 55 kgs lt 18 kgs 18 32 kgs 3 quipement lourd Cet quipement est tr s lourd et doit tre d plac avec un syst me de levage m ca nique pour vi
66. c and begins utilizing the switch It is recommended that the Mellanox OpenFabrics software package be installed on all nodes connected to the switch The software package provides a subnet manager and net work management tools as well as connectivity software for servers and storage and is available on the Mellanox web site Basic installation is covered in Chapter 2 Installation page 15 Hot swapping components and hardware maintenance is covered in Chapter 2 9 Hot Swap Insertion and Extraction on page 44 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID identifier and product version information are found on the label attached to the pull out tab below the Mellanox logo on the spine side of the chas 515 Figure 1 Product Label P S N MTO806X01504 E TI o ooozesozo027051c Rev A1 GUI TUM MAC 0002C9020040 EA ies Made in Israel Mellanox Technologies 11 InfiniBand Switch Platform Hardware User The GUID is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management software look for this GUID in addition to their own Node GUID 1 1 2 License Key This chassis comes with a license for FabricIT Chassis Manager The license key to acti vate the license is found on the bottom side of the pull o
67. ces The connector side of the switch has 36 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 64 The bottom row ports are flipped from the top row See Figure 65 Figure 64 Port Numbering 3 2 Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side 3 3 QSFP Cable Power Budget Classification 155600 switches are designed for active cables with a max power per module of 2 0W This is power level 2 according to the QSFP Public Specification Mellanox Technologies 68 3 4 Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the 155600 They are 112C port 1 CONSOLE port this is an RS232 connector for connecting to a host machine MGT this is an Ethernet connector e 1 USB port RST reset button Figure 66 Management Module Interfaces pio CONSOLE MGT gt 3 4 1 12C 3 4 2 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure 3 4 3 MGT Management The MGT port is an Ethernet port for remote manag
68. comandat 45 113 In plus pentru a asigura un flux de aer adecvat l sa i un spa iu liber de cel pu in 8 cm 3 inchi n jurul fantelor de ventilare 7 Suprapunerea cadrului Cadrul nu trebuie s suprapus peste niciun alt echipament n cazul n care cadrul cade poate cauza leziuni corporale i deterior ri ale echipamentului 8 Conexiunea la o surs de alimentare electric suplimentar pericol electric Acest produs include o surs de alimentare suplimentar sau un spa iu gol n locul acesteia In cazul n care spa iul pentru sursa de alimentare este gol nu produ sul c nd capacul orb este ndep rtat sau nu este fixat n mod sigur Mellanox Technologies 120 MN 9 Siguran fuzibil bipolar neutr Acest sistem este prev zut cu siguran fuzibil bipolar neutr Indep rtati toate cor doanele de alimentare nainte de a deschide capacul acestui produs sau nainte de a atinge orice componente interne 10 Multiple mufe electrice Risc de oc electric i pericol electric Toate aparatele cu alimentare de la re ea sunt independente Deconectati toate sursele de alimentare cu energie pentru a asigura decuplarea n inte riorul platformei de comutare gt 11 In timpul desc rc rilor electrice pericol electric A In timpul perioadelor cu desc rc ri electrice luminoase nu lucra i cu echipamentul sau N nu conecta i sau deconec
69. ds to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the ground ing post Make sure the connections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 34 Ground Connection Grounding post 2 7 Power Connections The switch includes integrated hot swap power supplies which support up to 10 load sharing 1000W The slots for the power supply units PSUs are on the spine side The Mellanox Technologies 41 InfiniBand Switch Platform Hardware User left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has dedicated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 50Hz or 60Hz The output voltage for the PSUs is 48 The power cords should be standard 3 wire AC power cords including a safety ground 2 7 1 Powering Up the Switch Platform Make sure that the power cords are compatible with your outlets Power cords for dif ferent countries can be ordered from Mellanox The chassis in N 1 mode must be started with a full complement of PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a
70. dundant power supplies available ules 8 Leaf Fan Modules Switching Capacity 52 Tb s Spine Fan Modules 18 Spine Fan Modules Spines 18 Management Modules 2 available 1 required for operation 1 Certification EMI certification on the fully populated chassis was performed with the chassis installed in a closed two door rack using the chassis installation kit supplied by Mellanox Tech nologies A 2 Approved Cables For a list of all approved cables see Mellanox Technologies 83 AA InfiniBand Switch Platform Hardware User http www mellanox com related docs user manuals Mellanox approved cables pdf EMC Certifications The list of approved certifications per chassis in different regions of the world is located on the Mellanox Website at http www mellanox com related docs user_manuals Regulatory and Compliance Guide pdf EMC Statements are also in the Regulatory and Compliance Guide Mellanox Technologies 84 InfiniBand Switch Platform Hardware User Appendix B Calculating the Weight of a Customized Chassis The weight of customized chassis can be calculated for any possible customization as follows Take the weight of a chassis with the following FRUs installed All fans modules All power supplies The weight of the 155600 chassis configured above is 157 57kg To this add the weight of installed FRUs e Spines
71. e a todas las leyes y reglamentaciones nacionales 16 C digos el ctricos locales y nacionales N Este equipo se debe instalar conforme a los c digos el ctricos locales nacio nales 17 C digos de instalaci n AN Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos N el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 18 Cambio de bater a II y a danger d explosion s il y a remplacement incorrect de la batterie Remplacer uniquement avec une batterie du m me type ou d un type quivalent recommand par le constructeur Mettre au rebut les batteries usag es conform ment aux instructions du fabricant 19 Cable de alimentaci n homologado por UL y con certificaci n CSA N En conexiones de America del Norte seleccionar un cable de alimentaci n y homologado UL con certificaci n CSA de tres conductores 16 AWG termi A W nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante
72. e brackets and reinstall them Figure 13 Placing the Shelf Bracket Adjusting slots Mellanox Technologies 29 sl InfiniBand Switch Platform Hardware User Figure 14 Caged Nut Locations for the Shelf Brackets 7 Insert five nuts into the rack for each of the four shelf brackets the 2 brackets that are adjustable and the 2 that are fixed Make sure that the caged nuts are all at the same height on the vertical supports Do not use the top hole in the bracket 8 Insert 4 caged nuts two on right and 2 on the left of the spine side for the bottom filler panel 9 Put the Shelf into the rack and twist the shelf into place Make sure the spine side of the shelf is closest to the rack doors See Figure 15 to identify the spine side of the shelf 10 Install and tighten the 20 M6 bolts into the caged nuts Figure 15 Shelf Brackets The spine side of the shelf has two slots for the extension Adjustable shelf brackets 11 Tighten the 16 hex head bolts on the adjustable brackets Mellanox Technologies 30 gt InfiniBand Switch Platform Hardware User Figure 16 U Numbering Above Shelf lt 1 2 The first cage nut on the spine side goes here The Test of the caged nuts go every 4 U units up from here Top of 12 Place 14 caged nuts int
73. een System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking green System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the spine fans is bad User should check individual spine fan LEDs for fault indications LEAF FANS Off No power to fan Green STATUS Green Nominal operational Red One or more of the leaf fans is bad User should check individual leaf fan LEDs for fault indications Mellanox Technologies 65 Me MON Wl ra NE Status Table 8 LEDs Display for Normal Operation this management module is not installed this management module is not the master Green Management module is operating as a master qu en Normal LED Condition Description ar MNGI Off Green MASTER no power STATUS this management module is not installed this management module is not the master Green Management module is operating as a master MNG2 Off Off MASTER no power STATUS Management Module LED Indicators Figure 62 Management Module Status Indications MGT
74. eighs 327 15kg 721 24 LBS full configuration 128 97kg 284 33 LBS empty configuration 157 57kg 347 LBS shipped configuration The switch platform uses 31U of rack space in a standard 19 rack 29U for the chassis and 2U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chas sis and there are two lock down bars to secure the chassis to the shelf The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have between 65 and 80cm between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling A Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis gt Mellanox Technologies 21 InfiniBand Switch Platform Hardware User Manual 2 6 1 Starting with the Rack 2 6 1 1 Rack Recommendation that is 120 cm long and 80cm wide This will allow for proper cable management and enough ventilation to properly cool the chassis Mellanox recommends that you remove both sides of the rack to make the installation easler v Due to the space required by up to 648 connector cables Mellanox recommends a rack You will need a fork lift to move and insert the chassis into the rack
75. eiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Tx 3 3 V Power supply transmitter 30 Vcc 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground ellanox Technologies 89 InfiniBand Switch Platform Hardware User Figure 75 InfiniBand QSFP Connector Symbol J3 and J6 20 GND 19 21 Rx2n Rxin 18 ___22 Rx2p 1 17 2 eis GND 16 _ 24 15 4 gt 14 Rx4p Rx3p 13 m GND GND I 2 ModPrsL spa 12 11 aad IntL Sem 28 VecTx Rx 10 30 Vcc1 ResetL 9 31 LPMode ModSelL 32 GND GND 7 33 ne 34 3 Tx4n 5 35 GND GND 4 36 2 3 3 Tx1n Tx2n 2 38 GND 1 Figure 76 QSFP Connector Male and Female Views View into Rear of Connector Pd TI pe lt 2 Dv 2 2 5 O 4 4 x 9 8 amp 9 EO 9582882 JE 6 gt 05 5 0
76. ement Any remote terminal con nected to the Ethernet port can then be used to manage the fabric and chassis 4 Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M auto negotiation d Initial configuration must be done on all of the management modules The first man agement module you configure will be the master La Mellanox Technologies 69 3 4 4 USB The USB port be used to upload new SW using any storage device that has a USB connector This interface is USB 2 0 3 4 5 Reset RST The Reset button resets the chassis management module when the button is pushed When the button is held down for 15 seconds the management module is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip Figure 67 Reset Button This button resets the CPU of the management module A quick push of this button per forms this reset When the reset button is pushed on the master management module this management module is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and
77. ent module in the chassis all of the leafs and ports Ah are reset by bringing them down and powering them up when the management module is removed Make sure the S Fans and L Fans LEDs are green 3 4 Make sure that the spine and the leafs both have the same version of FW 5 Reburn the FW and remove and reinstall the management module 6 If you are running the chassis with only one management module remove and rein stall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 2 9 5 on page 57 7 If you are running the chassis with only one management module replace the man agement module Issue2 Yellow LED for the Leaf Fan on the Management Module is Lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Determine which fan module is problematic by checking the status LED on each fan module 5 Remove and reinstall the problematic fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 9 4 on page 54 6 Replace the Leaf fan
78. entation set accompanying the Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Websites Document Name Description InfiniBand Architecture Specification The InfiniBand Architecture Specification that is provided by Vol 1 Release 1 2 1 IBTA Switch Product Release Notes For possible hardware issues see the switch support product page This requires a customer support login Look up the rele vant SwitchX based switch system series release note file Mellanox MLNX OS User Manual This document contains information regarding configuring and for VPI managing Mellanox Technologies SwitchX switch platforms listing all of the commands available through MLNX OS with explanations and examples Conventions Throughout this manual the name 155600 and the terms chassis and switch are used to describe the 648 port QSFP InfiniBand chassis unless explicitly indicated otherwise The following icons are used throughout this document to indicate information that is important to the user This symbol makes recommendations to the user This symbol indicates information that is helpful to the user A This symbol indicates a situation that can potentially cause damage to hardware or software mama Warning This symbol indicates a situation that can potentially cause personal injury and or damage to hardware or software Mellanox Technologies 10 Infini
79. erform the Software upgrade For further informa tion please refer to the FabricIT EFM user manual an earlier version than the minimum that the new version of the software can work with then the chassis management system may require up to 45 minutes to update all of the lt FW images in all of the leafs and spines P If FabricIT EFM is updated and the FW image in the leafs and spines of the chassis is da Mellanox Technologies 74 u InfiniBand Switch Platform Hardware User 6 Troubleshooting 6 1 Power Supply Unit A If the Power supplies cannot supply enough power the management module may shut down the some leafs gt As each PSU is plugged in make sure that the green power LEDs on the PSU comes on Issue 1 If the AC power LED is off 1 Check that the power cable is the correct power cable for your country Check that the power cable is plugged into a working outlet Check that the power cable has a voltage within the range of 100 240 volts AC Remove and reinstall the power cable Check the circuit breakers to be sure that the breaker has not tripped Check that the power cable is good Replace the power cable If the AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU au gt 6 2 Leaf Board Issue 2 The power LED for the Leaf board is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall the
80. ert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 2 9 3 on page 50 5 Make sure that the spine and the Leafs both have the same version of FW 6 Reburn the FW and remove and reinstall the spine 7 Replace the spine board 6 5 FabriclT Software For more detailed instructions concerning MLNX OS software see the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual Mellanox Technologies 77 InfiniBand Switch Platform Hardware User 7 7 1 Disassembly and Disposal Disassembling the Chassis D m Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Loosen all locking screws for the leafs spines and management modules in the chas 15 Remove all leafs 7 Remove all spines 8 Remove all management modules Use enough people to safely lift this product lt 40 Ibs 40 70 Ibs 70 121 lb
81. exe amp 8 FETS 6 Mellanox Technologies 90 InfiniBand Switch Platform Hardware User Appendix F Replacement Parts Ordering Numbers Table 17 Replacement Parts Ordering Numbers Part Description OPN 648 port QDR capable modular chassis includes fans MIS5600Q 10DNC and power supplies RoHS5 Power supply unit PSU 1000W for Modular Switch MTP005001 Family Power supply blank MTIM005001 Leaf board unit 18 port QDR MIS5001QC Modular Switch Family Leaf Blank MTM005004 Spine board unit for QDR Switch system MIS5002QC Spine board blank MTM005002 PPC460 Management Module MIS5600MDC Management module blank MTM005003 Fan unit for chassis and leafs MTF005002 Fan unit for Chassis Leaf side MTF005001 Fan unit for Chassis Spine side MTF005002 Modular Switch Family Management Blank MTM005003 Rack Installation Kit for IS5600 SX6536 Series MTR005600 Cable Management Device Set for IS5600 SX6536 MTR005601 Cables Set For IS5600 SX6536 Series MTR005602 155600 5 6536 IS5300 SX6518 Series Modular MTF005001 Switch leaf cooling chassis fan unit located on Leaf Side IS5600 SX6536 IS5300 SX6518 Modular Switch MTF005002 leaf cooling chassis fan unit located on Spine Side MIS5X00 Modular Switch Series Spine Fan Unit MTF005003 DB to RJ45 Harness HAR000028 Power cord 250V 15A 2 0M C14 to C13 ACC000334 Power cord 125V 15 2 0M C14 TO C13 ACC000242 Power cord
82. h see the Mellanox approved cable list http www mellanox com related docs user manuals Mellanox approved cables pdf 2 8 2 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox IS5x00 series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables 2 9 Hot Swap Insertion and Extraction Before starting any procedure on 155600 switch system put an ESD prevention wrist strap on your wrist and connect to the 155600 chassis Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam ily and switch chip generation When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions This switch platform supports hot swap capabilities for the parts listed below e Power supply units Leaf boards Spine boards Leaf fan module Spine fan module Management modules Mellanox Technologies 44 Infin
83. he vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handles onto the board the master management module CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and slave roles see the FabricIT EFM Software UM section High Availability y On switch systems with dual management systems first connect the cable and configure 2 management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure 5 2 9 6 Switch Shut Down Procedures To shut down the chassis run the following command twice once for each MM Reload halt noconfirm Mellanox Technologies 59 MU A InfiniBand Switch Platform Hardware User AN The chassis cannot be restarted remotely A To restart the chassis you must physically go to the switch and unplug all of the power cords to the chassis and then replug in all of the power cords to the chas sis The first time you run the command it Shuts down the master management module and the second time shuts down the slave management module To shut down a leaf run the following command no power enable lt module gt To shut down a spine run the following command no power enable lt module gt To shut down a management module run the following command no power e
84. iBand Switch Platform Hardware User 2 9 1 Power Supply Units 2 9 1 1 Extracting and Inserting the Power Supply Unit gt To extract a PSU 1 Determine which AC connector on the connector side of the chassis corresponds to the defective PSU 2 Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering Figure 37 Power Cord Numbering 3 On the spine side of the chassis remove the cover to the power supply unit for the non working PSU Odd numbered PSUs are on the left side and even numbered PSUs Mellanox Technologies 45 InfiniBand Switch Platform Hardware User 2 9 2 are on the right side facing the spines There are four phillips head screws for the cover plate Figure 39 Power Supply Abc AC Latch D p E EN D SON OK FAL 71 Handle B Grasping the handle with one hand push the black latch release while pulling the handle outward As the PSU unseats the PSU status indicators will turn off Remove the PSU To insert a PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles Insert the PSU by sliding it into the opening until a slight resistance is felt Ua cpu gen Vom Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation B Insert the power cord into the supply c
85. igure 33 Cable Holder Left side rack Top side 16 X M6 caged nut 16 X M6 bolt for caged nut Cable support shelf For each model of chassis there is the one support shelf for every two leafs Mellanox Technologies 40 InfiniBand Switch Platform Hardware User Manual 1 Place the cable holder next to the rack on the connector side of chassis and iden the holes where the caged nuts were placed step 4 The cable holders should go to the outside of the vertical supports A 2 Set the bottom of the cable holder at the level of the shelf If the cable holder is not set properly the upper bracket will not line up with the cable holder Screw the cable holder onto the rack using the screws provided Repeat steps 1 3 for the second cable holder Install the shelves The cable support shelves just sit on the cable holders su Ro Use the holes in the shelves to tie down the cables individually or in bundles Use Velcro ties to tie down the cables Electric cable ties are not recommended 2 6 5 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge The grounding post is M 8 with 1 25mm pitch threads A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cor
86. iir die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 18 Hoher Ableitstrom WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 19 Infomationen zum Erdungsanschluss Bevor Sie dieses Ger t das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation Mellanox Technologies 109 A s 20 Installationscodes Dieses muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code Fiir Nordamerika miissen in Ubereinstimmung mit den gel LA tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 21 Verbindung der Ger te untereinander Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal tung Uberstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden miissen in dem Geb ude Verkabelung 22 Gef hrliche Strahlung Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Proze duren die hier nicht spe
87. in um einen einwandfreien Luftstrom zu gew hrleisten Mellanox Technologies 107 7 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terfallt kann es zu Verletzungen und Besch digungen an Ger ten f hren 6 Zweipolig Neutrale Sicherung Achtung Zweipolige bzw Neutralleiter Sicherung im Netzteil Netzstecker ziehen um sicher zustellen da keine Spannung am Ger t anliegt Entfernen Sie alle Netzkabel vor dem ffnen der Abdeckung dieses Produkts oder dem Ber hren der Innenteile 8 Mehrere Stromeing nge Risiko eines Stromschlags und Stomgefahr h Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgeschalteten Zustand im Inneren der Switch Plattform sicherzustellen 9 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 10 AnschlieBen Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb miissen sie vorsichtig an die Anschliisse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers 11 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell
88. in Rack Doors Cable with recommended bending radius Chassis um E Insufficient room for recommended bending radius Doors You will most likely need extra room for cable bending on the connector side of the chassis and should plan to keep the spine side as close to the rack door as possible thus having more room for the cables 4 Measure the distance between the vertical supports and adjust the shelf brackets according to this distance This chassis can be installed in a rack with a distance between the vertical supports of from 65cm 25 6in to 80cm 31 5in Mellanox Technologies 28 InfiniBand Switch Platform Hardware User 5 The shelf comes with the four brackets pre installed Two of the brackets should be loose They may need to be removed and repositioned to meet the distance between the vertical supports of the rack Figure 12 Adjustable Shelf Brackets These two brackets need to be adjusted They may even need to be removed and repositioned The holes in the bracket are offset to one side The holes must be closer to the top of the shelf 6 Move the shelf brackets on the shelf Place the shelf bracket so that it measures slightly more than the measured distance between the vertical rack supports Make sure that there is enough play in the bracket so that it can move to the vertical support Do not tighten the hex head bolts You may need to remove th
89. inated with a molded plug rated at A W 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm Wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 10 A 20 High Leakage Current LA Warning High leakage current Earth connection essential before connecting supply Mellanox Technologies 95 im 21 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 22 Installation codes
90. ion The Leaf Status indicator LED has the following LED assignment Table 3 Leaf Status LED LED Condition LED Description Off No power to the Leaf Mellanox Technologies 62 Table 3 Leaf Status LED LED Condition LED Description Solid Green Leaf is up and running Flashing Green Leaf is powering up Solid Orange Non fatal error this leaf needs troubleshooting but does not require chassis shutdown Solid Red Fatal error 3 1 2 2 Leaf Board Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port Each port has single 2 color LED Table 4 shows the link status according to the LED condition Table 4 Connector Physical and Logical Link Indications LED Condition LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed is proportional to data transfer speed Solid Orange Physical link up Flashing Orange A problem with the physical link The LED indicator corresponding to each data port will light orange when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a log ical connection is made the LED will change to green When data is being transferred the light will blink green The sw
91. is unit is intended for connection to a TN power system and an IT power system of Norway only 2 2 Environmental and Safety Recommendations The following are Mellanox recommendations v Recommended ambient temperature in the System room is 20 5 C Recommended humidity range is 4096 15 without condensing Mellanox Technologies 19 InfiniBand Switch Platform Hardware User Manual It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference It 1s highly recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure A fully loaded 155600 switch system can draw 8 kW 67A E 120V or 34A 240V of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit gt 2 3 Chassis Package Contents The package includes 1 chassis with the following modules installed 18 spines 1 management module 8 fans e 10 PSUs e installation guide e 1 box containing various other parts installation kit box e 1 box containing 10 power co
92. itch does not provide a visual means to indicate the port Speed configuration ZA SDR DDR QDR and or the link width 1 or 4X The speed and link width config ad urations can be retrieved using management software 3 1 3 Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has the following LEDs One status LED for the spine health e One status LED for the spine fan modules 36 status LEDs showing the existence of leaf to spine connections The 36 LEDs on each spine are divided by the number of leafs and the result N is the number of connections from each leaf that are connected to the spine 18 spines must be installed and working to ensure that full BW exists between nodes The maximum num ber of connections from each leaf is 1 If the number of leafs x the maximum number Mellanox Technologies 63 of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF The status LEDs for the spine and their descriptions are shown in Table 5 The LEDs indicate as follows Figure 60 Spine Status LEDs Spine Status Fan Status Spine leaf IB connection status 3 1 3 1 Status LED Table 5 shows the spine status according to the LED condition Table 5 Spine Status LED LED Condition LED Description Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Solid Orange Non
93. lanox Technologies 115 4 5 6 45 113 F 8 3 7
94. lnes Netzteils spannungsfrei dune seule alimentation stabi Um alle Einhieten spannungsfrei lise e ne de branch uniquement Disconnect all power supplies to zu machen sind die Netzstecker qu un module Alimentation ensure a powered down state aller Netzteile zu entfernen Stabilise e Pour isoler com inside of the switch platform pletement le module en cause Il faut de brancher toutes les ali mentations stabilise es Mellanox Technologies 42 InfiniBand Switch Platform Hardware User Manual 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Any yellow or red status LEDs on any of the management modules is cause for con cern and must be dealt with immediately 4 Check that none of the LEDs on the spines are yellow It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes 5 Check that the leaf status LEDs fan status LED and spine status LED in the spines are all green maximum number of connections per leaf is less than 36 then some of the leaf to spine The maximum number of connections from each leaf is 1 If the number of leafs x the MA connection LEDs may be OFF Figure 36 Spine Module Fan LED 0000000 Status LED Leaf connection LEDs 2 8 InfiniBand QSFP Cable Installation The switch uses industry standard Infi
95. ment Module LEDs 3 1 5 1 Management Module PSU LED Indicator The management module PSU indicator should be green Table 10 Management Module PSU LED Configurations LED Condition Description Green OK All PSUs are working at correct input and output voltages Red Error One or more of the PSUs for the chassis is bad Check each PSU for a red LED 3 1 5 2 Management Module S FANS LED Indicator The management module S FANS indicator should be green Table 11 Management Module S Fan LED Configurations LED Condition Description Green OK All Spine fans are working at configured speed Red Error One or more of the spine fans within a spine fan module is bad Check the spines for a flashing green fan LED Mellanox Technologies 67 om 3 1 5 3 Management Module L FANS LED Indicator The management module L FANS indicator should be green Table 12 Management Module L Fan LED Configurations LED Condition Description Green OK All Chassis Leaf fans are working at configured speed Red Error One or more of the leaf fans for the chassis is bad Check each chassis fan for a red LED 3 1 5 4 Management Module MASTER LED Indicator Table 13 Management Module MASTER LED Configurations LED Condition Description Green This management module is the master Off This management module is the slave 3 1 6 Port Connector Interfa
96. n area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 7 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage Mellanox Technologies 93 8 Redundant Power Supply Connection Electrical Hazard This product includes a redundant power or a blank in its place In case of a blank power supply do not operate the product with the blank cover removed or not securely fastened 9 Double Pole Neutral Fusing M This system has double pole neutral fusing Remove all power cords before opening the cover of this product or touching any internal parts 10 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 11 During Lightning Electrical Hazard A During periods of lightning activity do not work on the equipment or connect or dis connect cables 12 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 13 Rack Mounting and Servicing h
97. nable lt module gt Mellanox Technologies 60 3 Interfaces 3 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of the IT manager All chassis conditions and management options are available and controllable through the manage ment SW either CLI or Web GUI Y It is recommended that all of the chassis sub systems be maintained and managed through the management software 3 1 1 Power Supply Unit LEDs Each Power Supply Unit has the following indicator LEDs Figure 57 Power Supply Unit Status Indications r 0 AC FAIL OK When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC A The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis 5 Mellanox Technologies 61 Figure 58 PSU Cover On and Off En ES O DE gr 285mm PSUs covered lt A Ge lt Ge A lt CYC lt D A EA gt A A A an CD D A Y O O Cover off 4 1 ll D ESD connection Ground connection 3 1 2 Leaf Board LED Indicators 3 1 2 1 Status LED Each leaf board has a Status LED on the far left of the leaf Table 3 shows the leaf status according to the LED condit
98. nd Offset Location 37 Figure 28 Offset to Upper Bracket Bolts 37 Figure 29 Location for the Lock Down Bar 38 Figure 30 Lock Down Bar 4 225 b t Ad here 38 Figure 31 Lock Down Bars Bolt and Washer Order 38 Figure 32 Spine Side with Filler Panels 39 Figure 33 Cable Holder 40 Figure 34 Ground Connection 5 rA Name RAE ma 41 Figure 35 Multiple Power Inlets Electric Caution Notification 42 Figure 36 Spine Module i 43 Figure 37 Power Cord Numbering 45 Mellanox Technologies 6 InfiniBand Switch Platform Hardware User Figure 38 PSU and Outlet Locations 45 Figure 39 Power Supply 46 Figure 40 Ejector Handle 47 Figure 41 Leaf Release dese pate TRA Gd XI ERES 48 Figure 42 Intact vs Defected Signal Connectors 48 Figure 43 Intact vs Defected Power Pin Holders 49 Figure 44 Management Module Numbering 51 Figure 45 Spine Board Extraction 51 Figure 46 Intact vs Defected Mechanics
99. nect the power cords to the PSUs Connect the power cords to grounded electrical outlets With N 1 PSU redundancy do not power up the chassis with less than all PSUs installed A fully loaded IS5600 series switch system can draw 9 7 kW of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or a 3 phase supply Mellanox Technologies 71 4 1 Power Supply and Spine Board Indicator Status at Power ON remain red for more than 5 minutes A It can take up to 5 minutes to boot up the system Turn off the system if any LEDs As the power is turned on you should observe the following conditions for normal oper ation 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the Spine LEDs and make sure they coincide with Figure 68 Figure 68 Spine Side Panel Display Status Indications E u E m STATUS PSU SPINE LEAF MNG1 MNG2 FANS FANS MASTER MASTER 5 Check the Management Module LEDs and make sure they coincide with Figure 69 Figure 69 Management Module Status Indications for Normal Operation ml
100. niBand cables which are available from sev eral vendors The standard cables support full 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All Infini Band connections are made to the leaf boards Each leaf has 18 InfiniBand connectors in two rows which are numbered 1 18 See Section 3 1 6 for port numbering A If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase cables can be inserted or removed with the unit powered on To insert a cable press the connector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 sec Mellanox Technologies 43 InfiniBand Switch Platform Hardware User onds for link to get up To remove disengage the lock and slowly pull the connector away from the port receptacle b For a valid physical connection both ends of the cable must be connected to working devices Take care to not impede the air flow through the ventilation holes next to the Infini Band ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 2 8 1 Supported Approved Cables For a list of approved cables for this switc
101. njivati i ili servisirati obuceno 1 kvalifici rano osoblje Ova oprema trebala bi se instalirati u skladu s lokalnim i nacionalnim elektri nim kodovima 17 Instalacijski kodovi N Ovaj se uredaj mora instalirati sukladno najnovijoj verziji nacionalnih y elektri nih kodova dr ave U Sjevernoj Americi oprema se mora instalirati 4 N sukladno va e im zahtjevima navedenim u US National Electrical Code i Canadian Electrical Code 18 Zamjena baterije Upozorenje Bateriju zamijenite samo baterijom iz serije UL koja je certifici rana za maksimalnu nepravilnu struju punjenja ne manju od 4 mA LA Postoji rizik od eksplozije ako se baterija zamijeni neodgovaraju om vrstom OdloZite prazne baterije sukladno uputama 19 UL CSA napajanja Za sjevernoameri ku mre u odaberite kabel napajanja koji je na UL listi i sa CSA certifikatom 3 ilni 16 AWG 16 AWG koji zavr ava lijevanim utika em nazivnog napona od 125 V 13 A minimalne duljine od 1 5 m six feet est stopa ali ne dulji od 4 5 m Za europsku mre u odaberite kabel napajanja koji je me unarodno uskla en i ozna en lt HAR gt 3 ilni s najmanje 1 0 mm ice nazivnog napona od 300 V s PVC izolacijom Kabel mora imati lijevani utika nazivnog napona od 250 V nazivne struje od 10 A 20 Veliko curenje struje h Upozorenje Veliko curenje struje Prije spajanja napajanja nu no je spojiti uzem ljenje Mellanox
102. nt tre mis au rebut d une mani re responsable respectant l environnement 28 Restrictions concernant l alimentation pour la Norv ge ellanox Technologies 10 6 G 4 Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 Verletzungsgefahr wegen des Gewichts Um das Produkt sicher anzuheben gen gend Personen einsetzen e e e lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Schweres Ger t Dieses Ger t ist sehr schwer und muss mit einem mechanischen Hebeger t verschoben werden um Verletzungen zu vermeiden 4 Installation an Bereich mit eingeschr nktem Zugang Diese Ger t ist f r die Installation in einem Bereich mit beschr nktem Zugang vorge sehen 5 Stromschlagrisiko Stromschlagrisiko Bei abgenommenem Ventilatormodul sind die Stromkontakte in der Modulvertiefung zug nglich Es d rfen KEINE Werkzeuge oder K rperteile in die Vertiefung des Ventilatormoduls gelangen 6 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 113 F betrieben werden Es ist ein Luftstrom von 200 bei maximaler Umgebungstemperatur erforderlich Au er dem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen se
103. o the vertical rack support at the locations corresponding to the holes in the chassis faceplate The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support There are 7 holes on each side Mellanox Technologies 31 CITIES InfiniBand Switch Platform Hardware User 2 6 3 6 Inserting the Chassis of the pallet Should the chassis start to fall move out of the way The chassis can The chassis is on ball bearings and can roll easily Be aware that the chassis can roll off A cause grave bodily harm should it fall on or near you 1 Put the container on the fork lift Figure 17 Putting the Chassis in the Rack Using a Fork Lift The chassis is Screwed down to the pallet on both the spine side and the leaf side Figure 18 Putting the Chassis the Rack Using a Mechanical Lift Mellanox Technologies 32 InfiniBand Switch Platform Hardware User Figure 19 Place the Upper Bracket Offset YLF LT E IE XIII II IX 2 Screw on the two Upper Bracket offsets onto the chassis using six flat head screws for each offset Figure 20 Attach the Shelf Extension Mellanox Technologies 33 InfiniBand Switch Platform Hardware User Manual 3 Screw on the shelf extension using four bolts with flat washers Figure 21 Place Caged Nuts Place 8 caged nuts between here and here for the cable holders
104. onnector on the other side of the chassis 5 Replace the cover over the PSUs The green indicators should light If not extract the PSU and re insert it again Leaf Boards Do NOT mix replacement parts based on different generations of chip Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa All Replacement modules must be consistent with the Chassis fam and switch chip generation AAN When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This 15 necessary so that the 4 N ment module will start a new cycle checking through the leafs and spines for the FW versions The leaf boards are numbered from top to bottom with corresponding numbers dis played to the outside of the leafs vertically along the side panel Mellanox Technologies 46 im InfiniBand Switch Platform Hardware User 2 9 2 1 Extracting a Leaf Board L33 L35 Each leaf board has a ejector handle that locks the board in place and serves as a lever for seating or extracting see Figure 40 1 Run the shut down command no power enable lt module gt For example to shut down leaf 16 run the command below switch master config no power enable L16 2 Disconnect all cables connected to the leaf 3 Push the ejector handle to unlock the ejector from the chassis
105. ors 5 Lock the ejector handles 6 Hold the body of the board on both sides and remove it from the chassis 2 9 3 2 Inserting a Spine Board When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a spine board 1 Check for foreign objects in or mechanical damage to the chassis spine slots 2 Check spine mechanics for any noticeable damage or deviations from the pass crite rion shown in Figure 46 Figure 46 Intact vs Defected Mechanics Mellanox Technologies 52 ee InfiniBand Switch Platform Hardware User 3 Check back signal connectors integrity Look for any broken signal dividers or any deviations from the pass criteria shown in Figure 47 and Figure 48 Pass r m mmm mmm mmm i gt ll mm ee 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 49 Figure 49 Intact vs Defected Power Pin Holders Mellanox Technologies 53 InfiniBand Switch Platform Hardware User 2 9 4 2 9 4 1 5 Start with the ejector handles fully open that is at 45 degrees to the front panel of the spine A MA Do not use the fan FRU handle
106. ottom PONS 68 Figure 66 Management Module Interfaces 69 Figure 67 Reset BUllO tin ta pide a a Meee 70 Figure 68 Spine Side Panel Display Status Indications 72 Figure 69 Management Module Status Indications for Normal Operation 72 Figure 70 Lock down Bar nic ei ee pax RE Pets eese Ae 79 Figure 71 Face Plate Mounting Bolt Locations 80 Figure 72 Mechanical Lifting Device 81 Figure 73 Front Panel x cisa abria ds a Sede 87 Figure 74 Rear Panel Leaf Side of the Switch 88 Figure 75 InfiniBand Connector Symbol 90 Figure 76 Connector Male and Female Views 90 Mellanox Technologies 7 InfiniBand Switch Platform Hardware User Revision History Table 1 Revision History of this User s Manual Revision Date Details 2 7 January 2014 Rearranged document Consolidated installation sections under Chapter 2 Installation on page 15 updated Chapter 1 Overview on page 11 and re ordered sections Updated e Section 1 3 Power Supply Redundancy on page 14 Section 2 6 4 Installing the Cable Holder on page 40 e Section 2 9 Hot Swap Insertion and Extraction on
107. page 44 e Section 7 1 Disassembling the Chassis on page 78 e Appendix Installation Safety Warnings on page 93 Table 14 IS5600 Switch Specification Sheet on page 82 e Appendix C Calculating the Power of a Chassis on page 86 2 6 Feb 2012 Modified Section 7 to point to the SW UMs 2 5 July 2011 Changed name to Hardware User Manual Fixed incorrect features list in Section 1 2 2 4 July 2011 Changes to Reset button section 2 3 June 2011 Changes to installtion kit parts and procedure 2 2 May 2011 Removed InfiniBand Fabric Initialization and Management Updated Fabric Management Software Section 2 1 Nov 2010 Added Barcode last page 2 0 Oct 2010 Added air flow direction 1 9 Oct 2010 Added procedure step to screw in 9 screws through the offset into the upper bracket Fixed Port Connector LED Assignment 1 8 Aug 2010 Added weight lifting warning Added Remove all protective plastic film from all sides and top of the chassis Added FCC Statement 15 19 Added QSFP Cable Power Budget Classification Added Center of gravity numbers Added Max Heat output numbers 1 7 June 2010 Fixed FabricIT screen grab Mellanox Technologies 8 InfiniBand Switch Platform Hardware User Table 1 Revision History of this User s Manual Revision Date Details 1 6 June 2010 Added section on getting licenses Added note
108. pine Fan Modules Each spine module has a fan module with two individual fans built in When a fan mod ule is not functioning the Fan Status LED on the Spine will light up Figure 52 Spine Fan Module 19 21 2325 27 29 33 35 000000000 000005050 Fan module When a fan module is removed the indicator light will reset Figure 53 Fan Status LED on the Spine Module Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out Mellanox Technologies 56 InfiniBand Switch Platform Hardware User M 2 9 5 Inserting the Spine Fan Module A When hot swapping any of the units it is necessary to wait 1 minute after removing 8 the defective part before inserting the new part This is necessary so that the manage b ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new spine fan module 3 Push the fan module as far as it will go make sure the locking latches engage A gt If the Fan LED continues to show red remove the fan module and check the pins on the 4 connector inside of the spine to make sure that none of them are bent Management Module When hot swapping any of the units it is necessary to wait
109. ponsabil i ecologic ellanox Technologies 123 G 8 Sigurnosna upozorenja za instaliranje Croatian 1 za instaliranje Pa ljivo pro itajte upute za instaliranje prije spajanja opreme na izvor elektri ne ener gije 2 Tjelesne ozljede uslijed te ine Kako biste sigurno podignuli ovaj proizvod koristite dovoljan broj ljudi LA lt 40 lbs 40 70 lbs 70 121 lbs gt 121 Ibs lt 18 kg 18 32 kg 32 55 kg gt 55 kg 3 Te ka oprema Ova oprema je vrlo te ka i treba se premje tati pomo u mehani kog dizala kako bi se izbjegle ozljede 4 Instaliranje na lokaciji s ograni enim pristupom Ovaj ure aj je namijenjen za instaliranje na lokaciju s ograni enim pristupom 5 Rizik od strujnog udara Rizik od strujnog udara S uklonjenim modulom ventilatora perima napajanja se mo e pristupiti u otvoru mod ula NEMOJTE umetati alat ili dijelove tijela u otvor modula ventilatora 6 Pregrijavanje Ovom se opremom ne bi trebalo rukovati u podru jima s temperaturom okoline koja prema uje najvi e preporu ene vrijednosti 45 C 113 F Osim toga kako bi se osig urao odgovaraju i protok zraka omogu ite najmanje 8 cm 3 in a razmaka oko otvora ventilatora Mellanox Technologies 124 7 Slaganje kucista Kuciste se ne bi trebalo slagati na drugu opremu Ako kuciste padne moze izazvati tjelesne ozljede i ostecenje opreme 8 Redundantno napajanje Opasnost od
110. pply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes redundant power or blank in its place In case of a blank A fastened 9 Double Pole Neutral Fusing This system has double pole neutral fusing Remove all power cords before opening the cover of this product or touching any internal parts 10 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform Mellanox Technologies 16 InfiniBand Switch Platform Hardware User 11 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 12 Copper InfiniBand Cable Connecting Disconnecting A N Copper InfiniBand cables are heavy and not flexible as such they should be carefully A attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 13 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken LA to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 14 Equipment Installation This equipment should be installed replaced and or serviced only by trained and
111. qual ified personnel Disposal of this equipment should be in accordance to all national laws and regula tions 16 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 5 17 Installation Codes A This device must be installed according to the latest version of the country A N national electrical codes For North America equipment must be installed in A accordance to the applicable requirements in the US National Electrical Code C and the Canadian Electrical Code Mellanox Technologies 17 xl InfiniBand Switch Platform Hardware User 18 Battery Replacement Warning Replace only with UL Recognized battery certified for maximum L abnormal charging current not less than 4mA There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispose of used batteries according to the instructions 19 UL Listed and CSA Certified Power Supply Cord For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket
112. r ses vis des bornes de la terre de protection doi vent tre reli es la terre de l installation lectrique de l immeuble Informations de LA branchement la terre L installation lectrique doit fournir une connexion la terre et un technicien de main tenance doit y relier l quipement de fa on permanente UN TECHNICIEN DE MAINTENANCE doit v rifier si la prise laquelle sera bran ch l quipement fournit une connexion la terre Si ce n est pas le cas le TECH NICIEN DE MAINTENANCE doit arranger l installation d un CONDUCTEUR DE MISE A LA TERRE allant de la borne de mise la terre jusqu au conducteur de terre de l immeuble Cet quipement doit tre install dans une zone disposant de liaisons quipotentielles comme un centre de t l communications ou une salle informatique d di e Mellanox Technologies 105 Re E 22 Codes d installation Cet appareil doit Gtre install conform ment a la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 23 Interconnexion des unit s Les c bles de connexion aux interfaces RS232 et Ethernet de l appareil doivent tre certifi UL de type DP 1 ou DP 2 Note en cas d installation sur un circuit dont la puissance n est pas limit e Protection contre les surintensit s le c blage
113. r e Wait 1 minute before putting in a Hotswap part New power numbers Added appendix Calculate Custom chassis weight Added appendix Calculate Custom chassis power Added a note that if EFM is updated and the FW image in the leafs and spines is earlier than the minimum that the new version of EFM can work with then the chassis management system will require 45 min utes to update all of the FW images in the leafs and spines Replaced Directive WEEE graphic Added a note switches must be configured to 10 100M auto negoti ation Moved Revision above About 1 5 April 2010 Minor formatting 1 4 April 2010 changed Appendix E OPNs 1 3 Mar 2010 Updated Installation Section New Installation kit Updated Cable holder figure 1 2 Jan 2010 Updated Installation Parts Kit figure and parts table Updated Cable holder figure 1 1 Jan 2010 Changed figure of the spine module 1 0 Dec 2009 Initial Release About this Manual This manual provides an overview of the SwitchX based 155600 modular InfiniBand chassis switch and guidelines for its operation Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniB and architecture specification Mellanox Technologies 9 th InfiniBand Switch Platform Hardware User Related Documentation The docum
114. r means of security 5 2 1 Installation Safety Warnings These safety warnings are in English For French German Spanish Russian and Roma nian see the Appendixes 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight M Use enough people to safely lift this product lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Heavy Equipment h This equipment is very heavy and should be moved using a mechanical lift to avoid injuries Mellanox Technologies 15 InfiniBand Switch Platform Hardware User 4 Installation in Restricted Access Location A N This unit is intended for installation in a Restricted Access Location 5 Risk of Electric Shock Risk of Electric Shock LA With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 6 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 7 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 8 Redundant Power Su
115. rds 250v 15a 2 0M C14 to C13 USA UL Stan dard e 1 cable management kit Before you install your new IS5600 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for Mellanox Technologies 20 InfiniBand Switch Platform Hardware User 2 4 2 5 2 6 visible damage that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately N The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order Management Package Contents When a second management module is ordered the package includes 1 management module ordered by the customer e RJA5 to DB9 harness Physical Installation Warning This equipment is very heavy Safety is the first concern Make sure that adequate manpower and proper equipment is used for transporting and moving the LA chassis The fully loaded chassis w
116. s gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs This product and all of its parts are NOT to be disposed of with household waste This product contains printed circuit boards cables and batteries According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and disposed of according to the directive 9 Goto the Mellanox website for detailed instructions for disassembly of the FRUs and chassis according to the WEEE Directive 10 Using a socket wrench remove the hex head screws holding down the lock down bar on the spine side of the chassis Mellanox Technologies 78 InfiniBand Switch Platform Hardware User Figure 70 Lock down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf 11 Remove the lock down bar 12 Dispose of these pieces in a legal and environmentally friendly way Mellanox Technologies 79 InfiniBand Switch Platform Hardware User Manual Figure 71 Face Plate Mounting Bolt Locations F3 Ei Helesin the faceplate ounting on the XX lt an 285mm KA Ge S A CS a 3 3 8 ES 3 ES a 8 po gt gt lt e m Ga e die sb E T ES 7 1 1 Removing the Chassis 1 Remove the screws connecting the upper brackets to the rack 2 Remove the screws holding the chassis to the
117. s cables de corriente antes de abrir la tapa de este producto o tocar cualquier componente interno gt 10 Tomas de alimentaci n m ltiples Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes Desconecte todas las fuentes de alimentaci n para asegurar que no haya corriente alguna dentro de la plataforma de conmutaci n 11 Al haber rayos peligro de descarga No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos gt 12 Cable de conexi n desconexi n InfiniBand de cobre AN Dado que los cables de cobre InfiniBand son pesados no son flexibles su 1 a rer los conectores y su desconexi6n se deben efectuar con mucho cuidado Para ver adver N tencias o instrucciones especiales consultar al fabricante del cable N 13 Montaje mantenimiento del bastidor Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adop tar precauciones especiales para garantizar que el sistema se mantenga estable En gen eral en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba Mellanox Technologies 112 14 Instalaci n del equipo La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente La eliminaci n definitiva de este equipo se debe efectuar conform
118. slizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o mantenimiento 27 Directiva WEEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera respon sable y respetuosa con el medio ambiente G 6 Russian 1 ycTaHOBKe 2 M e e lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Mel
119. tart a new cycle checking through the leafs and spines for the FW ver sions Each spine has a pair of ejectors that lock the board in place and serve as levers for seat ing or extracting see Figure 45 Management board 1 is connected to spine board 1 and management board 2 1 con nected to spine board 42 When a slave management board is not installed or not working the spine board connected to the master management board cannot be hot swapped of the spine boards can be hot swapped when two management boards are installed and working IN IN 4 When more than one spine slot is empty always insert the lowest spine board first then f N work your way up A N If you need to replace the bottom spine and it does not go in try removing one or two N spines above it and then insert the spines from the bottom up A When a slave management board is not installed or not working hot swapping the spine board connected to the master management board will cause the chassis to crash or Spine 2 as the management modules are connected to the chassis components Neither the CLI nor the GUI management tools will allow you to shut down spine 1 through these spines Extracting Spine Board 1 2 Spine 1 15 connected to management board 1 Spine board 2 15 connected to management board 2 Mellanox Technologies 50 i InfiniBand Switch Platform Hardware User Manual M
120. tati cablurile 12 Conectarea deconectarea cablului din cupru InfiniBand AN A Cablurile InfiniBand din cupru sunt grele i inflexibile de aceea trebuie s fie ata ate A W sau detasate de conectori cu grij Consultati produc torul de cabluri pentru avertis 4 y mente instructiuni speciale 13 Montarea sau depanarea ntr un rack C nd acest produs este montat sau depanat ntr un rack trebuie s fie luate m suri de precau ie speciale pentru a se asigura c sistemul r m ne stabil In general trebuie s umpleti rack ul cu echipamente ncep nd de jos n sus 14 Instalarea echipamentului Acest echipament trebuie s fie instalat nlocuit i sau depanat numai de c tre personal instruit i calificat 15 Eliminarea echipamentului Eliminarea acestui echipament trebuie s se realizeze n conformitate cu toate legile i regulamentele na ionale Mellanox Technologies 121 16 Codurile electrice locale i na ionale Acest echipament trebuie s fie instalat conform codurilor electrice locale i na ionale 17 Codurile ed instalare IN Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor N electrice nationale ale rii n cauz Pentru America de Nord echipamentul trebuie s N fie instalat conform cerintelor aplicabile din Codul electric national al SUA si Codul electric canadian 18 nlocuirea bateriei Avertisment Inlocuiti numai cu o
121. ter les blessures 4 Installation dans un emplacement acc s limit Cette unit doit tre install e dans un emplacement acc s limit 5 Danger d lectrocution A Danger d lectrocution Lorsque le module de ventilation est retir les broches d alimentation sont expos es dans l emplacement du module 4 NE PAS ins rer d outils ou la main dans l emplacement du module 6 Surchauffe Cet quipement ne doit pas tre en service dans un local dont la temp rature d passe le maximum recommand de 45 C 113 F En outre et pour garantir une circulation d air correcte laisser un espace d au moins 8 cm 3 autour des orifices de ventilation 7 Chassis empil sur d autres quipements Le ch ssis ne doit pas tre empil sur d autres quipements S il tombe il peut endom LA mager l quipement ou entra ner des blessures 8 Connexion de l alimentation redondante danger d lectrocution Ce produit est quip d une alimentation redondante ou d un cache si elle est absente LA Dans ce dernier cas ne pas faire fonctionner le produit si le cache est retir ou mal fix Mellanox Technologies 103 9 Fusibles phase neutre Ce syst me dispose de fusibles phase neutre D branchez tous les cordons d alimenta tion avant d ouvrir le capot ou de toucher tout l ment a l int rieur 10 Plusieurs prises d alimentation Risque et danger d lectrocution Les alimentations sont toutes
122. tform Hardware User Appendix A Specification Data Table 14 155600 Switch Specification Sheet Physical Power and Environmental H x W x D Weight Mounting Max Air Flow Through leafs Max Air Flow Through spines SerDes Speeds Connector Types Center of Gravity CoGh CoGw Max Heat Output 50 629 x 19 x 27 inches 1286mm x 482 6mm x 685mm The shelf adds 100mm to the height 126 4kg 278 7 LBS empty weight no leafs spines or management modules 328 6 kg 724 4 LBS full configura tion 126 4 kg 279 LBS empty configu ration 19 Rack mount EIA 310 16 59M3 min 585 9 CFM 9 48 M3 min 334 8 CFM 20 and 40 Gb s per port QSFP 637 mm from bottom of chassis 222 mm from left side surface 342 5 mm from port end surface Measured from the bottom left cor ner as one faces the switch ports 30635 56 BTUs hr Input Voltage Temperature Operating Non operating Humidity operating Power Con sumption Leaf Power Con sumption Spine Power Con sumption Fan Power Con sumption Man agement Power Con sumption for optical cables QSFP IS5600 Power consump tion Typ Max 100 240 VAC 50 60Hz 5 10A per inlet 45 Celsius 40 to 70 Celsius 10 85 non condensing Typical 82 28W including QSFP passive power Max 89 74W including QSFP passive power Typical 102
123. the chassis Y The installation will be much easier with a power screwdriver v It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation NG 2 6 3 2 Installation and Cable Management Kit Parts e Parts included in the installation kit 1 shelf with brackets pre installed 1 shelf extension 1 left side upper bracket offset 1 right side upper bracket offset 2 lock down bars 2 upper bracket washers Mellanox Technologies 23 InfiniBand Switch Platform Hardware User 8 lock washers for the upper brackets to vertical support 12 hex head bolts 8 for the upper brackets and 4 for the shelf extension 34 pan head screws for the caged nuts 20 for the shelf and 14 for the chassis face plate and 6 for the filler panels 32 lock washers 20 for the lock down bars and 12 for the upper brackets to upper bracket offsets 2 upper brackets 1 open end ratchet wrench 10mm 1 socket wrench 2 extensions for the socket driver 4 flat washers for the shelf extension Bottom filler panel 1 nut Size M 8 for ground post 34 caged nuts 20 for the shelf and 14 for the chassis faceplate and 6 for the filler panels
124. to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 5 Country of Norway Power Restrictions This unit is intended for connection to a TN power system and an IT power system of Norway only G 2 Chinese 1 2 e lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 LA RER MIRE DARE DER ASAS 4 Mellanox Technologies 97 s BREWER LA 6 45 113 F
125. ugraditi u o i enje zgrade 24 Izlo enost opasnom zra enju Pozor Kori tenje kontrola prilagodba ili izvo enje postupaka koji se razli kuju od onih koji su ovdje navedeni mo e izazvati izlaganje opasnom zra enju LASERSKI PROIZVOD KLASE 1 i referenca na najnovije laserske standarde IEC 60 825 1 1993 A1 1997 A2 2001 i EN 60825 1 1994 A1 1996 A2 2001 25 Odgovaraju a okolina Odgovaraju e elektri no mehani ko i protupo arno okru enje trebao bi omogu iti proizvo a krajnjeg proizvoda i ili krajnji korisnik 26 Nemojte koristiti preklopnik kao policu ili radnu povr inu ellanox Technologies Pozor Oprema montirana na kliza e vodilice ne bi se trebala koristiti kao polica ili radna povr ina Vodilice nisu namijenjene za povla enje ure aja iz ormari a Slu e samo za trajnu instalaciju na kona nom polo aju a ne za servisiranje i odr avanje 12 7 27 WEEE direktiva Sukladno WEEE direktivi 2002 96 EZ sav elektri ni i elektroni ki otpad EEE trebao bi se prikupljati zasebno i ne bi se trebao odlagati kao obi an ku anski otpad Odlaganje ovog proizvoda i svih njegovih dijelova vr ite na odgovoran i ekolo ki na in nn 28 Elektri na ograni enja dr ave Norve ke Ovaj je ure aj namijenjen samo za spajanje na elektri ni sustav s TN uzemljenjem i na elektri ni sustav s IT uzemljenjem dr ave Norve ke D Mellanox Technologies 128
126. ut tab Figure 2 License Key Label SAN KUN MT1134x 00560 License Key LK2 EFM CONFIG 5G2L S5HG LOKA L640 K6D2 Q600 3ADH G88A 1H3Q 7FT9 5M 1 1 3 Management Module MAC Each management module has a label with its MAC address See Figure 4 for the loca tion of this label Figure 4 Management Module MAC Address Location PSU S FANS LFANS Master 1 1 4 Product Physical Specifications and Power The switch itself is 29U tall but requires an extra 2U for installing the shelf The switch ships in a minimum base configuration plus additional modules depending on the chosen customer configuration Optional modules included Leaf boards Management modules Mellanox Technologies 12 InfiniBand Switch Platform Hardware User The following figure shows the connector and spine sides fully populated Figure 5 155600 Switch Views Leaf fan module with four internal fans Leafs Each leaf has 18 ports and up to 36 9 leafs can fit in the chassis Power supply units Up to 10 power supply units 18 spine modules 36 leaf modules 2 management modules Four leaf fan modules on each side of the chassis Mellanox Technologies 13 EEE InfiniBand Switch Platform Hardware User 1 2 Features List e 648 QDR 40Gb s InfiniBand ports in 29U switch e IBTA 1 2 1 compliant SDR DDR QDR link speed power supply
127. ware User 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion show in Figure 43 Figure 43 Intact vs Defected Power Pin Holders 5 Start with the ejector handle fully open that is at 45 degrees to the front panel of the leaf 6 Holding the leaf by its sides carefully set the leaf board into the chassis 7 Applying equal pressure on both sides of the leaf board slowly slide the board into the chassis until the ejector handle reaches the vertical bar Do not apply excessive force to slide in the leaf board If you feel resistance remove the leaf board and double check both the chassis and leaf for any damage 8 Catch the hook onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handle onto the board Mellanox Technologies 49 InfiniBand Switch Platform Hardware User 2 9 3 Spine Boards Do not install InfiniScale IV based replacement parts within a SwitchX based chas sis and vice versa Replacement modules must be consistent with the Chassis fam Do NOT mix replacement parts based on different generations of chip ily and switch chip generation N When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the management 4 N module will s
128. wn Bar Bottom of the chassis Figure 30 Lock Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf 19 Place the front lock down bar at the edge of the chassis and align the holes in the lock down bar with the holes in the shelf Screw down the lock down bar You will need the 10 hex head screws and washers for each lock down bar DO NOT TIGHTEN 20 Screw down the second lock down bar Figure 31 Lock Down Bars Bolt and Washer Order Mellanox Technologies 38 InfiniBand Switch Platform Hardware User 21 Remove the shelf extension and save for removal of the chassis 22 Connect a valid ground See Section 2 6 5 Ground Connections on page 41 for detailed instructions 23 Tighten the faceplate screws See Figure 25 on page 36 24 Tighten the upper bracket to upper bracket offset screws See Figure 28 25 Tighten the lock down bars See Figure 31 26 Connect the bottom filler panel below the shelf bend facing out 27 Connect the top filler panel directly above the top of the chassis Figure 32 Spine Side with Filler Panels 28 Go to Chapter 4 Chassis Power Up on page 71 to power up the switch system 29 Insert 2 caged nuts for the filler panel Mellanox Technologies 39 InfiniBand Switch Platform Hardware User Manual 2 6 4 Installing the Cable Holder Y Now is the time to install the cable holder F
129. zifiziert sind kann zu gef hrlichem Strahlenkontakt fiihren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 23 Geeignetes Geh use Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen 24 Switch nicht als Regal oder Arbeitsplatz nutzen Achtung Auf Schieber Schienen montiertes Ger t ist nicht als Regal oder Arbeitsbe reich zu nutzen Die Schienen sind nicht daf r bestimmt die Einheit aus dem Gestell weg zu ziehen Sie sind nur f r die permanente Installation an einem endg ltigen Stan dort gedacht nicht fiir Instandhaltung und Wartung 25 WEEE Direktive Gem WEEE Directive 2002 96 EC m ssen alle elektrischen und elektronischen Abfallger te EEE separat gesammelt und nichit mit normalem Haushaltsm ll ensorgt werden Dieses Produkt und alle seine Teile in verantwortungsvoller und umweltfreundlicher Art und Weise entsorgen ellanox Technologies 110 G 5 Advertencias de seguridad de instalaci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Lesi n corporal a causa de peso 5 Recurra suficientes personas para levantar este producto sin lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32

648-Port InfiniBand QDR SwitchX® Switch Platform

Contents

Download Pdf Manuals

Related Search

Related Contents

648-Port InfiniBand QDR SwitchX&reg; Switch Platform

Contents

Download Pdf Manuals

Related Search

Related Contents

648-Port InfiniBand QDR SwitchX® Switch Platform