Home

108 Port InfiniBand QDR Switch Platform User Manual

1. e parts list 28 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Before you install your new IS5100 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have occurred during shipping gt If anything is damaged or missing contact your customer representative imme diately The rack mounting holes conform to the IEA 310 standard for 19 inch racks Guarantee proper ventilation by leaving 8cm 3 of space to the front and rear A of the switch This will ensure proper air flow through the chassis This is cru re cial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis 3 2 Physical Installation loaded chassis weighs 170 Ibs 77 kg Make sure that proper manpower and Warning This equipment is very heavy Safety is the first concern The fully A equipment is used for transporting and moving the chassis The switch platform uses 7U of rack space in a standard 19 rack 6U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chassis The weight of the switch is supported from underneath the unit by the shelf This chassis can be ins
2. 198 Xy99A Osoy TISSPON GND dex upXL IND dzXL uzXL aNd 75 Mellanox Technologies Rev 1 7 Appendix E Ethernet RJ 45 Connector Figure 47 Ethernet RJ45 Connector 76 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Appendix F Replacement Parts Ordering Numbers Table 16 Replacement Parts Ordering Numbers Part Description OPN Power supply unit PSU MTP005001 Power supply blank MTMO005001 Leaf board unit 18 port QDR MIS5001QC Leaf Thermal blank MTM005004 Spine board unit for QDR Switch system MIS5002QC Spine board blank MTM005002 PPC460 Management Module MIS5600MDC Management module blank MTM005003 Fan unit for leafs MTF005004 Fan unit for spines MTF005003 DB9 to RJ45 Harness HAR000028 Power cord 250v 15a 2 0M C14 to C13 ACC000242 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 Mellanox Technologies 77 Rev 1 7 Appendix G Avertissements de s curit d installation French 1
3. Management Module Status Indications 24 Figure 10 Port Numbering 25 Figure 11 Top and Bottom Ports 25 Figure 12 Management Module Interfaces 25 Figure 13 Reset Restore Button 21 Figure 14 Shelf Installation Kit Parts 31 Figure 15 Chassis Installation Parts 32 Figure 16 Inserting the Caged Nuts for the Shelf 33 Figure 17 Use These Holes for the Rail Slides 34 Figure 18 Connect Rail Slide to Rack Vertical support 34 Figure 19 Inserting the Caged Nuts for the Faceplate 35 Figure 20 Cable Holders 36 Figure 21 Inserting the Caged Nuts for the Cable Holder 37 Figure 22 Cable Holder Placement 38 Figure 23 Chassis Rails and Rail Slides 39 Figure 24 Screw the Handles Onto the Chassis 39 Figure 25 The Rails are Already Connected Onto the Chassis 40 Figure 26 Put on the Rail Slide 41 Figure 27 185100 Chassis on the Shelf 42 Figure 28 Face Plate Mounting Bolt Locations 42 Figure 29 Move the Rail Slide to the Vertical Support 43 Figure 30 Ground Connection 44 Figure 31 Multiple Power Inlets Electric Caution Notification 45 Figure 32 Spine Module 46 Figure 33 Top and Bottom Ports 47 Figure 34 Power Cord Top Bar 51 Mellanox Technologies 5 Rev 1 7 Figure 35 Power Supply 51 Figure 36 Leaf Board Numbering 52 Figure 37 Ejector Latch 52 Figure 38 Leaf Release 53 Figure 39 Ejector Latch 54 Figure 40 Spine Board Extraction 54 Figure 41 Spine Fan Module 55 Figure 42 Leaf Fan Module Extraction 56 Figure 43 Fan Status
4. ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 20 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera responsable y respetuosa con el medio ambiente lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs Mellanox Technologies 87 Rev 1 7 Mellanox TECHNOLOGIES USA Mellanox Technologies Inc 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 Tel 408 970 3400 Fax 408 970 3403 www mellanox com E mail info mellanox com Map and Directions Tech Support 408 916 0055 8 00AM to 5 00 PM Pacfic 108Port InfiniBand QDR Switch Platform User Manual Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam Israel 20692 Telephone 972 4 909 7200 AC C 0 0 0 3 0 8 Fax 972 4 959 3245 E mail info mellanox com Tech Support 972 4 9097200 ext 250 8 00AM to 5 00 PM Israel Time www mellanox com Mellanox Technologies
5. AS SE SS er des 48541771 717 0 L Holz FE ASAE O HE FHI a Het 779210 RIAA ASE Ag SAO sch CLASS A device l i 3 This device has been approved by EMC registration commercial broadcasting and Distributors or users pay attention to this point This device is communication equipmen usually aimed to be used in other area except at home O Remark Class A device operated in a commercial area Mellanox Technologies 71 Rev 1 7 Appendix B Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculated for any possible customization as follows Take the weight of a chassis with all spines all leaf fan modules 0 leafs 0 leaf blanks all power supplies 1 management modules The weight of the IS5100 chassis configured above is 54 7kg To this add of leafs 2 76 of leaf blanks 0 764 3 15 if you will have two management modules 1 15 if you will have only one management module weight of the chassis plus the leaf boards PSUs and Management modules installed in the chassis 72 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Appendix C Calculating the Power of a Customized Chas sis To calculate the power consumption of a customized chassis take the base power and add the power of the fans spines management modules and leafs Table 14 Power Consumption of Chassis Parts Part typ p
6. AV A Av av ay amp Four leaf fan modules 0 i on the spine side of the L it j LE chassis r Three spine modules on this side of the ee ERLERNEN chassis 155100 3 Power Supply Units 14 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 2 Hardware Installation Safety Warnings 1 Installation Instructions Read all installation instructions before connecting the equipment to the power LA source 2 Bodily Injury Due to Weight Use enough people to safely lift this product LA lt 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Installation in Restricted Access Location LA This unit is intended for installation in Restricted Access Location 4 Over temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 5 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 6 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not This product includes a redundant power or a blank in its place In case of a blank A securely fastened Mellanox
7. BAr 2 12 SDA 2 wire serial interface data 38 GND GND 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 RxIn Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vcc Tx 3 3 V Power supply transmitter 30 Vcc 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground 74 Mellanox Technologies Rev 1 7 108 Port InfiniBand QDR Switch Platform User Manual 1ews QSFP Connector Male and Female Vi Figure 46 18 35 Top View into Rear of Connector GNB AX n 1Xtp GND TX3n TX3p GND LPMode Vcc1 VecTx IntL ModPrsL GND RX4p RX4n GND RX2p RX2n GND Bas N e kel GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL VecRx SCL 18 35 View into Front of Cage N Co gt O1 OO N aNd me diXH aNd UEXH dexu aNd vas
8. LED on the Spine Module 56 Figure 44 Ejector Latch 58 Figure 45 Management Module 58 Figure 46 QSFP Connector Male and Female Views 75 Figure 47 Ethernet RJ45 Connector 76 6 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Table of Lists Table 1 Revision History of this User s Manual Table 2 Reference Documents and Web Sites 9 Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 13 Table4 Leaf Board Status Indications 20 Table 5 Connector Physical and Logical Link Indications 21 Table 6 Spine Status LED 22 Table 7 Spine to Leaf IB Link Status 22 Table 8 Spine Fan Status LED 22 Table 9 LEDs Display for Normal Operation 23 Table 10 LEDs Display for Normal Operation 24 Table 11 LEDs Display for Normal Operation 49 Table 12 Switch Specification Sheet IS5100 68 Table 13 Certifications 69 Table 14 Power Consumption of Chassis Parts 73 Table 15 InfiniBand QSFP Connector Pinout 74 Table 16 Replacement Parts Ordering Numbers 77 Mellanox Technologies 7 Rev 1 7 Revision History Table 1 Revision History of this User s Manual Revision Date Details 1 7 May 2011 Added last page with barcode Removed InfiniBand Fabric Initialization and Management Updated Fabric Management Software Section 1 6 Oct 2010 Added air flow direction 1 5 Oct 2010 Removed Chassis Rails from installation Kit and Installation instructi
9. Luft LA strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terfallt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redundante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeckung nicht sicher festsitzt oder entfernt wurde 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 6 Anschlie en Trennen von InfiniBand Kupferkabel an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig A Warnungen und Anleitungen des Kabelherstellers 7 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen Mellanox Technologies 81 Rev 1 7 8 Ger teinstallation Diese Ger t sollte nur
10. Max 89 745W not including QSFP power Typical 102 86W include 2 Fans Max 110 32W include 2 Fans 2 0W Power level 2 108 ports Typical 1333 27W including QSFP at 2W Typical 1225 27W including QSFP at 1W Typical 1056 81W passive cables Max 1515 79W including QSFP at 2W 68 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Table 12 Switch Specification Sheet IS5100 Physical Power and Environmental Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 20Gb s 40Gb s Safety US Canada cTUVus 8 InfiniBand Virtual Lanes for all ports EU IEC60950 QoS International CB Baseboard Performance and Device Management management Agents for full InfiniBand USA FCC Class A In Band Management Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EMC EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I H ENVIRONMEN EU IEC 60068 2 32 Fall TAL Test ACCOUSTIC ISO 7779 ETS 300 753 Scalability and Performance Replaceable Parts Switching Performance Simultaneous wire speed any port to any leafs Up to 6 leafs 18 ports each port Power Supplies 3 Power Supplies standard one Addressing 48K Unicast Addresses Max per Subnet of which is a redundant power 16K Multicast Addresses per Subnet s
11. Technologies 15 Rev 1 7 Hardware 7 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 8 Copper InfiniBand Cable Connecting Disconnecting attached to or detached from the connectors Refer to the cable manufacturer for spe Copper InfiniBand cables are heavy and not flexible as such they should be carefully A cial warnings instructions 9 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with When this product is mounted or serviced in a rack special precautions must be taken A equipment starting from the bottom to the top 10 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qualified personnel 11 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end product manufacturer and or the end user 12 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula LA tions 13 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical 4 codes 16 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 14 UL Listed and CSA Certified Power Supply Cord Leakage gt 3 5mA For North American power connection select a power supply cord
12. The board is short therefore do not let go of it while sliding it out Figure 38 Leaf Release Push here to release the latch Pull the ejector to remove the leaf 5 2 2 Inserting a Leaf Board Start with the latch fully open that is perprndicula to the front panel of the leaf Carefully set the leaf board into the side guide rails and slowly slide the board into the chassis Catch the hooks onto the vertical bar and push the latch shut 5 3 Spine Boards When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so 4 that the management module will start a new cycle checking through the leafs and spines for the FW versions Each spine has a pair of ejectors that lock the board in place and serve as levers for seating or extracting see Figure 39 5 3 1 Extracting a Spine Board 1 Push outward on the latches to unlock the ejectors from the chassis 2 Open the ejectors until they are 90 degrees from the module 3 Pull out on the module using both ejectors Mellanox Technologies 53 Rev 1 7 Insertion and or Hotswap Extraction Figure 39 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Push the latch pompe pe release 7 P Figure 40 Spine Board Extraction Pull the latches to re
13. a license supporting of up to 648 nodes If there is a need to support more than 648 nodes an optional license of Unified Fabric Manager UFMTM is needed Management modules have the capability to allow remote monitoring and remote management of the chassis from any host connected to the fabric The managed switch system includes a CPU which contains e embedded OS secure in band out band access e chassis manager and system BIST e SNMP agent 3rd party tool integration e GUI e Subnet Manager SM The chassis manager will give the user access to e switch temperatures e power supply voltages e fan unit information e power unit information Flash memory The manager also has the ability to burn new firmware on the switch 7 1 Subnet Management The Subnet management features include In addition the tools enable firmware management capabilities such as e upgrading drivers e upgrading software e monitoring of AC power to the PSUs Mellanox Technologies 61 Rev 1 7 Switch Management Tools 7 2 7 3 7 4 n DC power out from the PSUs board temperature fan module unit failure in the switch system system failure in the switch system e querying for board serial numbers and their revisions Fabric Management with UFM For an added licensing fee you can purchase Mellanox s Unified Fabric Manager UFM is a power ful platform for managing scale out computing environments it enables data cente
14. appropriees Des enceintes lectriques m caniques et incendie adapt es doivent tre fournies par le fabricant du produit final ou par l utilisateur final 18 Cordons lectriques CA homologu s UL Pour les prises lectriques en Am rique du Nord choisissez un cordon lectrique homologu UL et certifi CSA 3 conducteurs 18 AWG termin par une fiche moul e d une tension nominale de 125 V 15 A avec une longueur minimale de 1 5 m 6 pieds et d une longueur maxi male de 4 5 m 18 pieds Pour les prises lectriques en Europe choisissez un cordon lectrique harmonise inter nationalement et marqu lt HAR gt 3 conducteurs d un diam tre de fil minimum de 0 75 mm2 d une tension nominale de 300 V avec une gaine isol e en PVC Le cordon doit avoir une fiche moul e d une tension nominale de 250 V et d une intensit nominale de 10 A lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 80 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Anhang H Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein
15. are 4 Leaf fan modules found on the Spine side The leaf side fans are not interchangeable with the spine side fans Air flow through the leafs is independent of the air flow through the spines 5 4 1 1 Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 42 2 Slowly pull out the fan module using the handle Mellanox Technologies 55 Rev 1 7 Insertion and or Hotswap Extraction Figure 42 Leaf Fan Module Extraction Handle Latch 5 4 1 2 Inserting the Leaf Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new leaf fan module LA If the fan module stops before it goes in all of the way it is inserted incorrectly 3 Push the fan module until the latch engages 4 Make sure that the green leaf LED on the module comes on indicating that fan is running 5 4 2 Spine Fan Modules Each spine fan module contains two individual fans Should a single fan fail the Fan Status LED on the spine and on the management module will light indicating the necessity to replace the fan module Air flow through the spines is independent of the air flow through the leafs When a fan module is removed the indicator light will reset Figure 43 Fan Status LED on the Spine Module 56 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform U
16. autoboot 0 Mellanox FabriclT Boot Menu 1 EFM_PPC_M460EX EFM_ 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM_PPC_M460EX EFM_1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt Choice 66 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 5 Select the image to boot Mellanox Technologies 67 Rev 1 7 Appendix A Specification Data A 1 Certifications and Statements Table 12 Switch Specification Sheet IS5100 Physical Power and Environmental HxWxD Weight Mounting SerDes Speeds Connector Types Center of Gravity CoGh CoGw CoGd 10 43 x 19 x 27 inches 265 mm x 482 6mm x 685 mm The shelf adds 44 5mm to the height 61 8 kg 136 lbs minimum configuration 77 kg 170 lbs full configuration 32 kg 71 lbs empty configuration 19 Rack mount EIA 310 20 and 40 Gb s per port QSFP 115 58 mm from bottom of chassis 227 2 mm from left side surface 338 09 mm from port end surface Measured from the bottom left corner as one faces the switch ports Input Voltage Temperature Max Heat Output Humidity Power Consump tion per Leaf Power Consump tion per Spine Power Consump tion for Active cables QSFP 155100 Power consump tion full chassis 90 264 VAC 50 60Hz 13A 100V 6 3A0240V 10 to 45 Celsius 5 172 1 BTUs hr 10 90 non condensing Typical 82 28W not includ ing QSFP power
17. cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi LA nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud minima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como minimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 15 Fuga gt 3 5 mA ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 16 Anadir conexi n a tierra Antes de conectar el dispositivo ala linea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de YN protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DU
18. can potentially cause personal injury or damage to hardware or software 10 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 1 Overview This User s Manual provides an overview of the IS5100 QSFP DDR QDR Modular InfiniBand Switch Platform known in this document as the switch and its operational environment Mellanox IS5100 switch system provides the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers High Performance Computing and Embed ded environments Networks built with the IS5100 system can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 4th gen eration InfiniScale IV InfiniBand switch device IS5100 system provides up to 40Gb s full bidi rectional bandwidth per port With up to 108 ports in a 6U high form factor these systems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard operation within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system An initial configuration procedure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configuration Once co
19. chassis system put an ESD prevention wrist strap on your wrist and connect to the IS5100 series chassis The chassis must be started with a full complement of PSUs that is 3 Thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running FabricIT EFM may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced 1 Check all boards power supplies leaf fan modules spine fan modules for proper insertion and seating before connecting the AC power cords All leafs that you plan to use insert in the chassis Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all mounting screws Connect the power cords to the PSUs DN JE Connect the power cords to grounded electrical outlets LA Do not power up the chassis with less than all PSUs installed 4 Populate the chassis with the leafs before you power up the chassis 4 1 Power Supply and Spine Board Indicator Status at Power ON LEDs may be RED Turn off the system if any LEDs remain red for more than 3 It can take up to 3 minutes to boot up the system during this time the status A minutes As
20. isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 82 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 14 Ableitstrom gt 3 5mA LEAKAGE gt 3 5mA WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 15 Add GND Verbindung Informationen Add GND connection information Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation 16 INSTALLATION CODES INSTALLATION CODES Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimmung mit den gel LA tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 17 Zusammenschaltung von EINHEITEN INTERCONNECTION OF UNITS Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal LA tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs Mellanox Technologies 83 Rev 1 7 Appendix I Advertencias de seguridad para la insta laci n
21. les mises en garde et les instructions sp ciales gt 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cautions sp ciales pour s assurer que le systeme reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut 78 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 8 Fuite gt 3 5mA Leakage gt 3 5mA ATTENTION La connexion la terre des forts courants de fuite est essentielle avant le branchement de l alimentation LA Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent 9 Forts Courants de Fuite High Leakage Current Attention Forts courants de fuite Il est essentiel de relier a la terre avant de brancher l alimentation 10 Ajouter une information de connexion la masse Connect a Valid Ground to this Device terre du terminal de l appareil doivent tre appliqu es l installation de protection la Avant de brancher l appareil la conduite d alimentation les vis de protection la A Terre du b timent 11 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 12 Elimination du mat riel T limination de ce mat riel doi
22. must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 17 Interconnection of units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified h type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protec tive device rated 20 A must be incorporated in the building wiring 18 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure Mellanox Technologies 17 Rev 1 7 Hardware CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60825 1 2007 03 and EN 60825 1 2007 19 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 18 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 2 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of the IT manager NV 4 We recommend that all
23. of the chassis sub systems be maintained and managed through the FabricIT EFM management software 2 1 1 Power Supply Unit LEDs Each Power Supply Unit has the following indicator LEDs Figure 5 Power Supply Unit Status Indications De AC OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC Note The PSUs are on the spine side of the chassis behind two cover panels The plugs for these PSUs are on the leaf side of the chassis Mellanox Technologies 19 Rev 1 7 Hardware Figure 6 PSU Cover On and Off AOOO ne PU 155100 DOOODDDDE 000000000 s2 OOOOOOOCO FAN maaa ajalna onooooonn DOoooooan s3 DOODO0odO FAN minaaa aafaa Doonoooon poooooooo o OOOO 155100 000000000 DOO000000 s2 anadodoa 000000000 oooooo000 ooooooo00 Wa Anaconda 000000000 oooooo0o0 Oooooooo0o Grounding post ESD Connector 2 1 2 Leaf Board LED Indicators Each leaf board has a Status LED on the far left of the leaf This LED has the following meanings according to the LED status Table 4 Leaf Board Status Indications LED Description Off No power to the leaf Solid Green Normal operation 20 Mellanox Technologies 108 Port InfiniBand QDR Switch
24. radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 2 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures 70 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 A 2 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada A 2 4 VCCI Statements Japan Class A Statement LORIE TIAZARRNRE CT CORE ERR CHEAT SEB RBS ES E ESTER COBBICMEAAY BOS WREBITSLIBKENSCEPHWETF VCCI A Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 2 5 KCC Certification Korea English Translation Device User s information As 7 7 AR amOo2 HA zane st ol 7l7lE ITS
25. the connector is firmly seated The green LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 seconds for link to get up To remove disen gage the lock and slowly pull the connector away from the port receptacle For a valid physical connection both ends of the cable must be connected to working devices Warning Take care to not impede the air flow through the ventilation holes next to the InfiniBand ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack Il Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row t 46 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 33 Top and Bottom Ports _i E 3 4 1 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox IS5X00 series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables Mellanox Technologies 47 Rev 1 7 Chassis Power Up 4 Chassis Power Up Before starting any procedure on the IS5100 series
26. AMA Mellanox TECHNOLOGIES 108 Port InfiniBand QDR Switch Platform User Manual PN MIS5100Q 3DNC Rev 1 7 www mellanox com Rev 1 7 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCU MENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRON MENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGE MENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PROD UCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SU
27. CH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale InfiniPCI PhyX and Virtual Protocol Interconnect are registered trademarks of Mellanox Technologies Ltd CORE Direct and FabricIT are trademarks of Mellanox Technologies Ltd All other marks and names mentioned herein may be trademarks of their respective companies QDR InfiniBand Switch Platform User Manual 2 Mellanox Technologies Document Number 3242 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Table of Contents Table of Contents 3 Table of Figures 5 Table of Lists 7 Revision History 8 About this Manual 9 Intended Audience 9 Related Documentation 9 Chapter 1 Overview 11 1 1 ProductInformation 11 1 1 1 Serial Number and Product Version Information 11 1 1 2 Management Module MAC 12 Chapter 2 Hardware 15 2 1 LED Status Indicators 19 2 1 1 Power Supply Unit LEDs 19 2 1 2 Leaf Board LED Indicators 20 2 1 3 Spine Board LED Indicators 21 2 1 4 Spine Side Panel Display LED Indicators 23 2 1 5 Management Module LED Indicators 24 2 1 6 P
28. CTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 17 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe YN instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 18 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de LA tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una cor riente nominal de 20 A 86 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 19 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqui se especifican podrian causar exposici n a niveles de radiaci n peligrosos lt lt PRODUCTO LASER DE CLASE 1 y referencia a las normas de l
29. Connections The switch includes integrated hot swap power supplies which support up to 3 load sharing 1000W supply units The slots for the power supply units PSUs are on the spine side The left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has a dedi cated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be standard 3 wire AC power cords including a safety ground Before starting any procedure on the IS5100 series switch system put an ESD prevention wrist strap on your wrist and connect to the IS5100 series Chassis 3 3 1 Powering Up the Switch Platform Note Make sure that the power cords are compatible with your outlets Power cords for differ ent countries can be ordered from Mellanox 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cord into a grounded outlets Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 44 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 The Chassis must be started with a full compliment of PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to
30. Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 45 C 113 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut provoquer des blessures corporelles et des d gradations de biens gt 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour alimentation lec trique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solidement fix ou s il est enlev 5 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les cables 6 Branchement d branchement des cables InfiniBand en cuivre Les cables InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fab ricant des cables pour connaitre
31. Platform User Manual Rev 1 7 Table 4 Leaf Board Status Indications LED Description Flashing Green Power up in progress Yellow Non fatal error Red Fatal error 2 1 2 1 Leaf Board Port Connector LED Assignment Above the ports are two LEDs one for the upper port and one for the lower port V The fol lowing table shows the port status according to the LED indication Table 5 Connector Physical and Logical Link Indications LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed gt data transfer speed Orange Physical link up Flashing Orange A problem with the physical link The switch does not provide a visual means that indicates the port speed configuration A SDR DDR or QDR and or the link width 1X or 4X The speed and link width con re figurations can be retrieved using FabricIT management software 2 1 3 Spine Board LED Indicators Each leaf board is connected by links to each spine module Each spine has one status LED for the spine and 36 status LEDs The 36 LEDs on each spine are divided by the number of leafs and the result X is the number of ports from each leaf that is connected to the spine 3 spines must be installed and working to ensure that full BW exists between nodes Should a leaf be removed then X number of LEDs will turn off on each of the spines The status LEDs for the spine a
32. S STATUS Table 9 LEDs Display for Normal Operation aya Normal LED Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot LEAF FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot MGT1 STATUS Off no power or this management module is not installed Green Green Management module is operating as a master Yellow Management module is operating as a slave i e the other man agement module is a master MGT2 STATUS Off no power or this management module is not installed Yellow Green Management module is operating as a master Yellow Management module is operating as a slave i e the other man agement module is a master Mellanox Technologies 23 Rev 1 7 Hardware 2 1 5 Management Module LED Indicators Figure 9 Management Mod
33. Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instruc ciones de instalaci n 2 Lesi n corporal por peso Dado que el equipo es muy pesado se debe mover nicamente mediante un elevador mec nico para evitar lesiones gt 3 Instalaci n en un lugar con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 4 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C 113 F Adem s para garantizar una circu laci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pul gadas alrededor de las aberturas de ventilaci n gt 5 Apilamiento del chasis AR Los chasis no se deben apilar sobre otros equipos La caida del chasis podria causar lesiones corporales as como da os al equipo 6 Conexi n de fuente de alimentaci n redundante peligro de descarga el ctrica Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa est quitada o no est bien cerrada 7 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de activ idad de rayos 84 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform Us
34. agement modules and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LEDs on any of the management modules is cause for concern and must be dealt with immediately 4 Check that none of the LEDs on the spines are yellow LEDs may be RED Turn off the system if any LEDs remain red for no more It can take up to 3 minutes to boot up the system during this time the status A than 3 minutes 5 Check that the Leaf status LED Fan status LED and Spine status LED in the spines are all green Mellanox Technologies 45 Rev 1 7 Installation Figure 32 Spine Module Fan status LEDs 19 21 2325 2729 31 33 35 DODODO OOO 2000000000000000 092000200 0000000000000000 Status LED Leaf connection LEDs 3 4 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from several ven dors The standard QSFP cables support full 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase All cables can be inserted or removed with the unit powered on To insert a cable press the con nector onto the port receptacle until
35. different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running FabricIT EFM may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs Figure 31 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and Gafahr des elektrischen Schocks Risque de choc et de danger energy hazard Entferrnen des Netzsteckers e lectriques Le de branchment The PSUs are all independent elnes Netzteils spannungsfrei d une seule alimentation stabi Um alle Einhieten spannungsfrei lise e ne de branch uniquement Disconnect all power supplies to zu machen sind die Netzstecker qu un module Alimentation ensure a powered down state aller Netzteile zu entfernen Stabilise e Pour isoler com inside of the switch platform pletement le module en cause Il faut de brancher toutes les ali mentations stabilise es 3 Check the Status LEDs on all of the man
36. e Turn the chassis off and then troubleshoot MGTI Off no power or this management module is not installed Green STATUS Green Management module is operating as a master 1 e the other management module is a slave Yellow Management module is operating as a slave i e the other management module is a master MGT2 Off no power or this management module is not installed Yellow STATUS Green Management module is operating as a master 1 e the other management module is a slave Yellow Management module is operating as a slave i e the other management module is a master Mellanox Technologies 49 Rev 1 7 Insertion and or Hotswap Extraction 5 Insertion and or Hotswap Extraction Before starting any procedure on the IS5100 series Switch system put an ESD prevention wrist strap on your wrist and connect to the IS5100 series Chassis When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so LA that the management module will start a new cycle checking through the leafs and spines for the FW versions This switch platform supports Hotswap capabilities for the e Power supply units e Leaf boards e Spine boards e Leaf fan module e Spine fan module e Management modules 5 1 Power Supply Units When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before
37. e Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one Note Should any of the boards shut down due to over temperature follow the procedure start ing in Step 2 The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected Check that the locks on the ends are secured Make sure that the latest FW version is installed on both the HCA card and the switch If media adapters are used check that the all connections are good tight and secure SR Replace the cable 64 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 The Activity indication does not come on Check that the Subnet Manager has been started 8 3 Management Module 8 3 1 Yellow LED for the Management Module is Lit 1 Remove and reinstall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 5 5 on page 57 Make sure that the spine and the leafs both have the same version of FW 3 Reburn the FW and remove and reinstall the management module Replace the management module 8 3 2 Yellow LED for the Leaf Fan is Lit 1 Make sure that there is nothing blocking the front or rear o
38. er Manual Rev 1 7 8 Conexi n y desconexi n del cable Copper InfiniBand Dado que los cables de cobre InfiniBand son pesados y no son flexibles su con exi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 9 Montaje y mantenimiento de bastidores Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 10 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente 11 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen dio 12 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 13 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales gt Mellanox Technologies 85 Rev 1 7 14 Cable de alimentaci n homologado por UL y con certificacion CSA En conexiones de America del Norte seleccionar un
39. f the bolts for each rail slide install the rail slides onto the rack Check that both sides of the switch left and right are the same level in the rack Figure 18 Connect Rail Slide to Rack Vertical support Place the four bolts for the caged nuts within reach Slide the shelf into the rail slides Put the shelf into place and screw the bolts into the nuts from step 2 Tighten all of the screws to 9 2 Nm or 81 5 pound inches 2 AED A Insert 8 caged nuts for the faceplate in the exact locations shown in the Figure 19 Inserting the Caged Nuts for the Faceplate 34 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 19 Inserting the Caged Nuts for the Faceplate Place caged nuts here Shelf Mellanox Technologies 35 Rev 1 7 Installation 3 2 3 Installing the Cable Holder Now is the time to install the cable holder Figure 20 Cable Holders Right Side Cable Holder Left Side Cable aD E M6 X 8 Caged Nut tl M6 X 8 bolt 1 Place the Cable holder next to the rack on the connector side of the chassis and identify the holes where the caged nuts are to be placed 36 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 21 Inserting the Caged Nuts for the Cable Holder O Place to connect the chassis rail slide p a x Place caged nuts here O H The cable holders should go to t
40. f the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 4 on page 55 LA Replace defective leaf fan modules as soon as they are identified Should any of the boards shut down due to over temperature follow the procedure start ing in Step 1 8 3 3 Yellow LED for the Spine Fan is Lit Determine which spine has a defective fan 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 4 on page 55 LA Replace defective spine fan modules as soon as they are identified Mellanox Technologies 65 Rev 1 7 TroubleShooting 8 4 Spine Board The yellow LED on the Spine board is lit 1 Make sure that there is nothing blocking the front or rear of
41. he outside of the vertical supports o 2 Set the bottom of the cable holder at the level of the shelf Note If the cable holder is not set properly the upper bracket will not line up with the cable holder Mellanox Technologies 37 Rev 1 7 Installation Figure 22 Cable Holder Placement Notch for upper bracket washer Set this 3 top of the shelf 3 Screw the cable holder onto the rack using the screws provided 4 Repeat steps 1 3 for the second cable holder 38 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 3 2 4 Installing the Chassis Figure 23 Chassis Rails and Rail Slides Rail Slide ALi yyy yyy Tyry gt eA S 1 Screw the handles onto the chassis Use the 8 allen head screws and the allen wrench pro vided Two handles go on the connector side and two go on the power side of the chassis Figure 24 Screw the Handles Onto the Chassis The handles go here ANO GAN A IL l DU y Ml uses oe eas e E NT Te an HA 185100 J Mellanox Technologies 39 Rev 1 7 Installation Figure 25 The Rails are Already Connected Onto the Chassis When lifting e use a legs bend your knees and not your back MAN lt 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 2 Lift the chassis and slide it onto the shelf Use a mechanical lift or enough people to safely lift the chass
42. ible damage all 3 spines are installed 3 PSUs are installed all 4 fans are installed management module is installed 8 Remove all protective plastic film from all sides and top of the chassis 32 Mellanox Technologies J 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 9 If you wish you may reduce the weight of the chassis by removing all of the spines power supply units fan units and management modules and put aside for reinstallation after the chassis is installed in the rack gt The leafs are shipped separately gt 3 2 2 3 Installing the Shelf 10 11 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Clip 4 caged nuts into the holes in the rack you will be using to connect the brackets Check that both sides of the shelf are at the same level in the rack Figure 16 Inserting the Caged Nuts for the Shelf Top of shelf a O Bottom of shelf TT MX Do not use this hole 19 3 Clip 4 caged nuts into the holes in the rack you will be using to connect the rail slides Check that both sides of the shelf will be at the same level in the rack Mellanox Technologies 33 Rev 1 7 Installation Figure 17 Use These Holes for the Rail Slides G 4 Using two o
43. inserting the new part This is necessary so LA that the management module will start a new cycle checking through the leafs and spines for the FW versions A PSU Power Supply Unit can be extracted without bringing down the system as long as no more than one PSU is bad The power required to run the switch system is equally divided between all of the working PSUs 5 1 1 Extracting and Inserting the Power Supply Unit With all of the power supplies installed the system is in the redundant configuration any single PSU may be extracted without bringing down the system To extract a PSU 1 Remove the power cord from the power supply unit Note which power cord it is according to the AC numbering 50 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 34 Power Cord Top Bar AC2 2 On the spine side of the chassis remove the cover to the power supply unit for the non work ing PSU Figure 35 Power Supply Handle Latch 3 Grasping the handle with one hand push the black latch release while pulling the handle out ward As the PSU unseats the PSU status indicators will turn off 4 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap in
44. is It weighs 77kg 170 lbs 3 Push the chassis into the rack until the faceplate is 20cm 8 from the vertical support Put the rail slides onto the rails 40 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 26 Put on the Rail Slide 5 Slide the chassis further into the rack Place the caged nuts in the rack in the places corresponding to their places in the faceplate 7 Slide the chassis all the way into the rack until the faceplate is touching the vertical rack sup port The caged nuts placed in the last procedure should line up with the holes in the face plate 8 Screw the 8 screws through the faceplate and into the caged nuts Slide the rail slides to the vertical rails Mellanox Technologies 41 Rev 1 7 Installation Figure 27 155100 Chassis on the Shelf Figure 28 Face Plate Mounting Bolt Locations Holes in the faceplate for mount ing on the rack e pd pq ee i eo a B oO 185100 Ep ur u S ODODDOODO y Fan A ggogogggg o 0000000 E ooooo0000 Oooo0ooo0 000000000 s pogona EE a Gooooooo0 DOODODOOO DoooooooG 42 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 29 Move the Rail Slide to the Vertical Support 10 Place the 4 holed spacer on the outside of the vertical support and screw in the 4 screws for each rail slide 11 Remove all four handles and save f
45. management Connect this port to a local host using the harness supplied with the chassis See the Installation Guide for the initial configuration procedure 2 4 3 MGT Management The MGT port is an Ethernet port for remote management Any remote terminal connected to the Ethernet can then be used to manage the fabric and chassis Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10 100M auto negotiation hb gt d Initial configuration must be done on all of the management modules bs 2 4 4 USB The USB port can be used to upload new SW using any storage device that has a USB connector 2 4 5 RST Reset Restore There is a Reset Restore button This button resets the chassis when the button is pushed When the button is held down for 5 seconds the chassis is reset and the password is reset to the original default password DO NOT use a sharp pointed object such as needle or push pin for pressing the 4 Reset button Sharp objects can cause damage use a flat object such as a paper clip 26 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 13 Reset Restore Button CONSOLE MGT RST PSU S FANS L FANS MASTER Mh Mh YU UU Mellanox Technologies 27 Rev 1 7 Installation 3 3 1 Installation oy This chassis can be installed in standard 19 racks that have depths between 650mm and 800mm between the vertical s
46. mellanox com gt Downloads gt InfiniBandSW Drivers Be sure to read and follow all of the instructions regarding the installation and use of these tools Each InfiniBand subnet needs one subnet manager to discover activate and manage the subnet An InfiniBand network requires a Subnet Manager to be running in either the Infiniband switch itself switch based or on one of the nodes which is connected to the Infiniband fabric host based The InfiniBand subnet manager OpenSM assigns Local IDentifiers LIDs to each port con nected to the InfiniBand fabric and develops a routing table based on the assigned LIDs A typical InfiniBand installation using the OFED package will run the OpenSM subnet manager at system start up after the OpenIB drivers are loaded This automatic OpenSM is resident in mem ory and sweeps the InfiniBand fabric approximately every 5 seconds for new InfiniBand adapters to add to the subnet routing tables earlier version than the minimum that the new version of EFM can work with then the chassis management system may require up to 45 minutes to update all of the FW be images in all of the leafs and spines P If FabricIT EFM is updated and the FW image in the leafs and spines of the chassis is an On IS5X00 systems when having dual management systems first connect the cable and configure the master management module CPU and only then configure the slave gt By default the master CPU is the top management
47. module For further information on the master and slave roles see the FabricIT Software UM section High Availability for more information Initial configuration must be done on all of the management modules See the Installa tion Guide for the Initial Configuration procedure 99 All management modules in the chassis must go through an initial configuration proce dure See the Installation Guide for the initial configuration procedure gt Mellanox Technologies 63 Rev 1 7 TroubleShooting 8 TroubleShooting As each PSU is plugged in make sure that the green power LEDs on the PSU come on 8 1 Power Supply Unit Note Ifthe Power supplies cannot supply enough power the management module may shut down the system If the AC power LED is off 1 Check that the power cable is the correct power cable for your country 2 Check that the power cable is plugged into a working outlet 3 Check that the power cable has a voltage within the range of 100 240 volts AC 4 Remove and reinstall the power cable 5 Check the circuit breakers to be sure that the breaker has not tripped 6 Check that the power cable is good Replace the power cable 7 If the AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU 8 2 Leaf Board The power LED for the Leaf board is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall in the Leaf board 3 When th
48. move the spine 1921 2325 27 29 31 33 35 ooog 4 OODODODOO 20 22 242628 30 32 34 36 5 3 2 Inserting a Spine Board 1 Hold the spine board by the sides with both latches fully open Insert the board into its rail The board should slide in without resistance Push the board in until the hooks on the latches are touching the vertical bar E PUN Now press both latches inward simultaneously You should feel the latches are hooked into the front panel 5 Press both latches applying some force to close them completely The locking latch will engage 54 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 5 4 Fan Modules When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so LA that the management module will start a new cycle checking through the leafs and spines for the FW versions There are four fan modules on the chassis for the leafs They are located on the spine side along the top of the chassis When a fan module is not functioning the status LED on the fan will light up Each spine module has a fan module with two individual fans built in When a fan module is not functioning the status LED on the fan will light up Figure 41 Spine Fan Module 1921 2325 27 29 31 33 35 oono gogoo 20 22 2426 28 30 32 34 36 Fan Module 5 4 1 Leaf Fan Module There
49. nagement MAC IS5100 1 1 2 Management Module MAC Each management module has a label with its MAC address See below for the location of this label Figure 3 Management Module MAC Address Location LICA o 00 00 pee TT m MAC address location III m 12 Mellanox Technologies Product Physical Specifications and Power Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Rack of leafs a Height Rack Width en Weight E ern nee Required of Spines 6U 19 EIA 25 7 6 3 77kg 170lbs full con Typical 1333 27W including Max 45 C 265 mm 310 685 mm figuration QSFP at 2W Min 10 C Plus 1U for 482 6 mm 32 KG 71 lbs empty Typical 1225 27W including the Shelf configuration QSFP at 1W Typical 1056 81 W passive cables 3 Power Supplies of 1000W 100 240VAC 50 60Hz 6 3 13A The switch ships in a minimum base configuration plus additional modules depending on the cho sen customer configuration Optional modules include Leaf boards e Management modules Figure 4 shows the connector and spine sides fully populated Mellanox Technologies 13 Rev 1 7 Overview Figure 4 155100 Switch Views Two Management ML AA ne B A Ilm il Kele ee e1e 00100 DINK XOX XX ES CARLA il G M acy a vay AY AV ay Aw Away B T RUE d BANS OA OO y AV AV AY AY
50. nd their descriptions are shown in Table The LEDs indicate as fol lows Mellanox Technologies 21 Rev 1 7 Hardware Figure 7 Spine Status LEDs 19 21 2325 27 29 31 33 35 DDODOODOO ODODODODO 20 22 2426 28 30 32 34 36 Spine Status Fan Status Spine Leaf IB Connection Status Table 6 Spine Status LED LED Description Off No power to the spine Solid Green Spine is up and running Flashing Green Spine is powering up Orange Non fatal error Red Fatal error The leaf board status on each spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per each leaf per each spine Each spine has leaf status LEDs The leaf LEDs on each spine indicate as follows Table 7 Spine to Leaf IB Link Status LED Description Off Link is down Solid Green Logical connection Flashing Green Data activity Orange Physical connection Table 8 Spine Fan Status LED LED Description Green Spine Fan is OK yellow One or more of the fans in this spine is not working 22 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 2 1 4 Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition Figure 8 Spine Side Panel Display Status Indications BL EEE O STATUS PSU SPINE LEAF MGT 1 MGT 2 STATUS FANS FANS STATU
51. nnected to the network the Subnet Management software automatically discovers and configures the fabric and begins utilizing the switch It is recommended that the Mellanox OpenFabrics software package be installed on all nodes con nected to the switch The software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is available on the Mellanox web site Basic installation is covered in Chapter 3 Installation on page 28 Hot swapping components and hardware maintenance is covered in Chapter 5 Insertion and or Hotswap Extraction on page 50 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID Identifier and product version information are found on the label that is attached to the pull out tab that is below the Mellanox logo on the spine side of the chassis FabricIT CM license is located on the bottom side of the pull out tab Mellanox Technologies 11 Rev 1 7 Overview Figure 1 Generic Product Label N MT0949X001 5100Q Made in IL This is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management SW look for this GUID in addition to their own Node GUID Figure 2 Product Label Tab Location Top right hand side of the spine side of the chassis Tab location with ee Product label and ma
52. nt 61 7 2 Fabric Management with UFM 62 7 3 Accessing the CPU via the Ethernet Connector 62 7 4 Downloading Firmware 62 7 5 OpenSM 63 Chapter 8 TroubleShooting 64 8 1 Power Supply Unit 64 8 2 LeafBoard 64 8 3 Management Module 65 8 3 1 Yellow LED for the Management Module is Lit 65 8 3 2 Yellow LED for the Leaf Fan is Lit 65 8 3 3 Yellow LED for the Spine Fan is Lit 65 8 4 SpineBoard 66 8 5 FabricIT SW 66 8 5 1 The last software update did not succeed 66 Appendix A Specification Data 68 A 1 Certifications and Statements 68 A 2 EMC Statements USA 70 Appendix B Calculating the Weight of a Customized Chassis 72 Appendix C Calculating the Power of a Customized Chassis 73 Appendix D QSFP Interface 74 Appendix E Ethernet RJ 45 Connector 76 Appendix F Replacement Parts Ordering Numbers 77 Appendix G Avertissements de s curit d installation French 78 Appendix H Installation Sicherheitshinweise German 81 Appendix I Advertencias de seguridad para la instalaci n Spanish 84 4 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Table of Figures Figure 1 Generic Product Label 12 Figure 2 Product Label Tab Location 12 Figure 3 Management Module MAC Address Location 12 Figure 4 1S5100 Switch Views 14 Figure 5 Power Supply Unit Status Indications 19 Figure 6 PSU Cover On and Off 20 Figure 7 Spine Status LEDs 22 Figure 8 Spine Side Panel Display Status Indications 23 Figure 9
53. ons Fixed Port Connector LED Assignment 1 4 Aug 2010 Added QSFP Cable Power Budget Classification Added weight lifting warning Added Remove all protective plastic film from all sides and top of the chassis Added FCC Statement 15 19 Added CoG and Max heat to Specs in Appendix A 1 3 July 2010 Fixed graphic call out in Figure 4 1 2 July 2010 Added mote that the initial configuration must be performed on all of the man agement modules in the chassis Master before Slave 1 1 June 2010 Added section on getting licenses Added note r e Wait 1 minute before putting in a Hotswap part New power numbers Added appendix Calculate Custom chassis weight Added appendix Calculate Custom chassis power Added a note that if EFM is updated and the FW image in the leafs and spines is earlier than the minimum that the new version of EFM can work with then the chassis management system will require 45 minutes to update all of the FW images in the leafs and spines Replaced Directive WEEE graphic Added a note switches must be configured to 10 100M auto negotiation Moved Revision above About 1 0 May 2010 Initial release 8 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 About this Manual This manual provides an overview of the 5100 modular InfiniBand chassis switch and guidelines for its operation Intended Audience This manual is intended for
54. or future use 12 Ground the switch 13 Replace all of the spines power supply units fan units and management modules removed at the start of the installation procedure 14 Plug in the power cables 15 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow status LEDs is cause for concern and must be dealt with immediately It can take up to 3 minutes to boot up during which time the status LED may indicate red 16 You can start connecting all of the cables to the switch 3 2 5 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or Amm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the con nections are solid and permanent If you choose to not use the ground screw make sure that the Mellanox Technologies 43 Rev 1 7 Installation rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 30 Ground Connection TDR DE DED 155100 Ground Post ESD Post 3 3 Power
55. ort Connector Interfaces 24 2 2 AirFlow 25 2 3 QSFP Cable Power Budget Classification 25 2 4 Management Module Interfaces 25 2 4 1 RC 26 2 42 CONSOLE 26 24 3 MGT Management 26 2 44 USB 26 2 4 5 RST Reset Restore 26 Chapter 3 Installation 28 3 1 Chassis Package Contents 28 3 2 Physical Installation 29 3 2 1 ESD Connection 29 3 2 2 Installation Procedure 30 3 2 3 Installing the Cable Holder 36 3 2 4 Installing the Chassis 39 3 2 5 Ground Connections 43 3 3 PowerConnections 44 3 3 1 Powering Up the Switch Platform 44 3 4 InfiniBand QSFP Cable Installation 46 3 4 1 Cable Power Classes 47 Chapter 4 Chassis Power Up 48 4 1 Power Supply and Spine Board Indicator Status at Power ON 48 Chapter 5 Insertion and or Hotswap Extraction 50 5 1 Power Supply Units 50 5 1 1 Extracting and Inserting the Power Supply Unit 50 5 2 Leaf Boards 52 5 2 1 Extracting a Leaf Board 52 5 2 2 Inserting a Leaf Board 53 Mellanox Technologies 3 Rev 1 7 5 3 Spine Boards 53 5 3 1 Extracting a Spine Board 53 5 3 2 Inserting a Spine Board 54 5 4 Fan Modules 55 5 4 1 Leaf Fan Module 55 5 4 2 Spine Fan Modules 56 5 5 Management Module oT 5 5 1 Extracting a Management Module ST 5 5 2 Inserting a Management Module 58 Chapter 6 Disassembly and Disposal 59 6 1 Disassembling the Chassis 59 6 1 1 Removing the Chassis 59 6 1 2 Removing the Bottom Shelf 59 6 2 Disposal 60 Chapter 7 Switch Management Tools 61 7 1 Subnet Manageme
56. ould be collected separately and disposed of according to the directive 8 Dispose of these pieces in a legal and environmentally friendly way 6 1 1 Removing the Chassis Remove the screws connecting the upper brackets to the rack 2 Remove the screws holding the chassis to the rack These screws are located in the faceplate on the spine side of the chassis 3 Remove the upper brackets from the chassis 4 Dispose of the Chassis in a legal and environmental way 6 1 2 Removing the Bottom Shelf 1 Remove all of the bolts that are holding the shelf to the rack 2 Remove the shelf 3 Remove all of the caged nuts Mellanox Technologies 59 Rev 1 7 Disassembly and Disposal 6 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and elec tronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 60 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 7 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via FabricIT EFM d There are 2 Ethernet ports 1 for each management module that each get connected to Ethernet switches These switches must be configured to 10 100M auto negotiation 2 The IS5100 switch is pre installed with
57. ower passive cables SR hover ie a eye IB QDR leaf 82 28W 121 45 Spine 103W Not applicable Fan 14 Not applicable Management Module 20 Not applicable The base power consumption of the chassis including full power supplies with O fans O spines 0 management modules and 0 leafs is 254 55W To calculate the chassis power 254 55W 103W of spines QOW of management modules 14W of fans 82 28W of leafs for passive cables or 121 45W of leafs for active cables Mellanox Technologies 73 Rev 1 7 Appendix D QSFP Interface 0___ en eno Y Table 15 InfiniBand QSFP Connector Pinout 21 non Rxin 18 _ 22 Rx2p Rxip 17 Connector Connecto 23 GND anp 16 Pin r Pin Signal Description 24 15 Number Name Rx4n Rx3n IT 25 Rxdp Rx3p 14 1 GND Ground 26 GND GND 13 2 Tx2n Transmitter Inverted Data Input 27 ModPrsL SDA 12 3 Tx2p Transmitter Non Inverted Data Input 28 Int SCL 11 4 GND Ground 29 VocTx Veo Rx 10 5 Tx4n Transmitter Inverted Data Input 30 Veet Reset __2 6 Tx4p Transmitter Non Inverted Data Input 21 LPMode ModSelL 1 GND Ground 32 GND GND 7 8 ModSelL Module Select 3 113 Tx4p 9 ResetL Module Reset a En A 10 VccRx 3 3 V Power supply receiver 36 Es gt 3 11 SCL 2 wire serial interface clock p P 37 Dis
58. r operators to efficiently monitor and operate the entire fabric boost application performance and maximize fab ric resources utilization UFM provides a central management console including the following main features e querying for existing firmware versions e burning new firmware from scratch or for recovery from damaged firmware e querying for and changing system GUIDs e checking for duplicate or bad GUIDs Fabric dashboard including Fabric Congestion Map Advance Fabric wide real time and history monitoring Quality of Service e Traffic Aware Routing Algorithm TARA e Central Device Management e Task automation e Threshold based alerts e Fabric segmentation isolation e Logging e High Availability Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through e SSH e Telnet the WEB Downloading Firmware Firmware for this switch system can be found at and downloaded from http www mellanox com gt Downloads gt Firmware 62 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Be sure to read and follow all of the instructions regarding the updating of the firmware on your switch system Firmware for the HCA cards connected to this switch system can be downloaded from the same site 7 5 Open SM To manage the Mellanox switch system using OFED download Mellanox Open Fabrics from http www
59. row The ports are labelled as shown in Figure 10 The bottom row ports are flipped from the top row See Figure 11 24 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 10 Port Numbering 1 3 5 7 9 11 13 is 17 ARA A A A O EA PEA ROS VA AA A AAA AA A 2 4 6 8 1 0 E 14 16 18 Figure 11 Top and Bottom Ports eee 2 2 Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side 2 3 QSFP Cable Power Budget Classification All 185100 QSFP switches are designed for a active cables with a max power per module of 2 0W This is power level 2 according to the QSFP Public Specification 2 4 Management Module Interfaces The switch system requires at least one management module The management module has five interfaces to connect to the 185100 They are e 1 T2C port e 1 CONSOLE port this is an RS232 connector for connecting to a host machine e 1 MGT this is an Ethernet connector e 1 USB port e 1 RST reset button Figure 12 Management Module Interfaces CONSOLE MGT gt RST PSU S FANS L h h ON Mellanox Technologies 25 Rev 1 7 Hardware 2 4 1 12C This interface is for Debug and Troubleshooting only This interface is for FAEs only 2 4 2 CONSOLE The CONSOLE port is used during the installation process to configure the chassis for remote
60. s for installing the shelf e shelf e 8 M6 bolts for the caged nuts e 8 of caged nuts e 2 Shelf rail slides Parts for installing the chassis e 2 Chassis rail slides e 8 M4 Screws e 8 sets of caged nuts and bolts 12 Flat head screws e 2 Flat 4 hole metal spacers e 8 Allen head screws e 4 Handles e 1 allen wrench Parts for installing the cable manager e 1 RH cable holder e 1 LH cable holder e 8 sets of caged nuts and bolts You will need 7U of space in the rack 6U for the chassis and 1U for the shelf 30 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Figure 14 Shelf Installation Kit Parts Angled S Bracket ar NS Rail j e Rail Slide gt M6 X4 Caged nut Z y M6 X4 bolt Mellanox Technologies 31 Rev 1 7 Installation Figure 15 Chassis Installation Parts Rail Slide is N Flat head Washer saree M6 Pan head I SCrews Before starting the procedure put the ESD strap on connect it to a valid ground 3 2 2 2 Opening the Container Before starting the procedure put the ESD strap on and connect it to a valid ground Open the crate by opening the latches Unscrew the sides of the crate Remove and put aside the box containing the cables Remove and put aside the box containing the cable management kit Remove and put aside the box containing the Installation kit SON OR eis Visually inspect the chassis make sure that there is no vis
61. ser Manual Rev 1 7 5 4 2 1 Extracting the Spine Fan Module 1 Push the two blue latch buttons together while pulling the fan module out 5 4 2 2 Inserting the Spine Fan Module 1 Make sure the fan module is oriented correctly top side up Confirm that the location of the connector in the chassis will line up with the connector in the fan module 2 Slowly slide in the new spine fan module 3 Push the fan module until as far as it will go make sure the locking latches engage 5 5 Management Module When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so LA that the management module will start a new cycle checking through the leafs and spines for the FW versions Before starting any procedure on the IS5100 series Switch system put an ESD prevention wrist strap on your wrist and connect to the IS5100 series Chassis 5 5 1 Extracting a Management Module Management modules are located on the leaf side above the leafs There are two places to install the management modules Note Only one management module is required to run the switch system Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 45 1 Disconnect all cables connected to the management module 2 Push outward on the latches to unlock the ejectors from the chassis 3 Open the ejectors un
62. t s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 13 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 14 Codes d installation pays Pour l Am rique du Nord l quipement doit tre install conform ment aux L appareil doit tre install selon l ancienne version des codes lectriques nationaux du A sp cifications du Code Electrique National Am ricain et du Code Electrique Canadien Mellanox Technologies 79 Rev 1 7 15 Interconnexion des unites gt Les c bles de branchement l unit RS232 et les interfaces Ethernet doivent tre certi fies UL de type DP 1 ou DP 2 Note lorsqu il existe dans un circuit non LPS Protection contre la surintensit Un appareil de protection r pertori facilement accessible contre la surintensit du circuit de branchement et calibr 20A doit tre incorpor dans le c blage lectrique du b timent 16 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 17 S assurer que les enceintes sont
63. talled in standard 19 racks that have depths between 650mm and 800mm between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling Y Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis gt Mellanox recommends a rack that is at least 80 cm long 3 2 1 ESD Connection Before starting any procedure on the IS5100 series Switch system Mellanox Technologies 29 Rev 1 7 Installation 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit 3 2 2 Installation Procedure 3 2 2 1 Requirements You will need e 2 phillips screwdriver e 3 phillips screwdriver e a grounding lug e ground wire to properly ground the chassis The installation will be much easier with a power screwdriver It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to have at least two people for the duration of the installation procedure Use a mechanical lift to raise this chassis If not use enough man power to ensure the safety and wellbeing of all of the people involved in the installation e Parts included in the installation kit e Parts included in the cable management installation kit Part
64. that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at LA 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A LA Warning High leakage current Earth connection essential before connecting supply 15 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation LA GND Connection Information The building installation shall provide a means for connection to protective earth and the equipment shall to be connected to that means permanently by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of aPROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the protective earth wire in the building The equipment shall be installed in area where equipotential bonding exists such as a telecommunication centre or a dedicated computer room 16 Installation codes This device
65. the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 5 3 on page 53 Make sure that the spine and the Leafs both have the same version of FW Reburn the FW and remove and reinstall the spine Replace the spine board 8 5 FabricIT SW 8 5 1 The last software update did not succeed 1 2 3 4 Connect the RS232 connector CONSOLE to a laptop Push the reset button on the switch or management module You will have 5 seconds to stop the U Boot by pressing Control B Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 minx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location 12C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No 12C ready DRAM 2 GB ECC enabled 400 MHz CL3 FLASH 16 MB NAND 1024 MiB PCI Bus Dev Venid Devid Class Int PCIEO link is not up PCIE1 successfully set as root complex 01 00 15b3 bd34 0c06 00 Net ppc_4xx_eth0 ppc_4xx_ethl Hit Ctrl B to stop
66. the power is turned on you should observe the following conditions for normal operation 48 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the management module LEDs and make sure they coincide with Table 11 Table 11 LEDs Display for Normal Operation Normal LED Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheat ing Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power sup plies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot LEAF FANS _ Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failur
67. til they are 90 degrees from the module 4 Pull out on the module using both ejectors Mellanox Technologies 57 Rev 1 7 Insertion and or Hotswap Extraction Figure 44 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine Push the latch gt release Note The board is short therefore do not let go of it while sliding it out Figure 45 Management Module Il CONSOLE MGT A i me TT Mil ih oe ae a Ejectors 5 5 2 Inserting a Management Module 1 Carefully seat the management module into the slot and slowly slide the board into the chas sis until the ejectors begin to engage on the chassis edge 2 Press the ejectors inward until the locks snap This indicates that the board is fully seated 58 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 6 Disassembly and Disposal 6 1 Disassembling the Chassis Power down the chassis Remove all power cables Remove all connector cables Disconnect the ground lug from the ground post Remove all leafs Remove all spines SOD QUE DE Remove all management modules This product and all of its parts are NOT to be disposed of with household waste This product contains printed circuit boards cables and batteries Accord YN ing to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE sh
68. to place confirm ing the proper installation Insert the power cord into the supply connector on the other side of the chassis 5 Replace the cover over the PSUs LA The green indicators should light If not extract the PSU and re insert it again Mellanox Technologies 51 Rev 1 7 Insertion and or Hotswap Extraction 5 2 Leaf Boards When hotswapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so LA that the management module will start a new cycle checking through the leafs and spines for the FW versions The leaf boards are numbered from top to bottom with corresponding numbers displayed to the outside of the leafs vertically along the side panel Figure 36 Leaf Board Numbering L 1 Lo L3 L4 L5 L6 5 2 1 Extracting a Leaf Board Each leaf board has an ejector that locks the board in place and serves as a lever for seating or extracting see Figure 45 1 Disconnect all cables connected to the leaf 2 Push the latch to unlock the ejector from the chassis 3 Open the ejector until it is 90 degrees from the leaf 4 Pull out on the leaf Figure 37 Ejector Latch Pull the leaf out of the chassis Leaf J Leaf Bi Push the latch Ejector 52 Mellanox Technologies 108 Port InfiniBand QDR Switch Platform User Manual Rev 1 7 Note
69. ule Status Indications tatus The management module LEDs display the switch system operating conditions Table 10 LEDs Display for Normal Operation Normal LED Description State STATUS Off No Power Green Green System is up and running Yellow System warning Attention needed such as overheating Red System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow System booting Restore factory defaults in progress PSU STATUS Off No power Green Green Normal operational Red PS fault detected User should check individual power supplies for fault indications SPINE FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot LEAF FANS Off No power to fan Green STATUS Green Nominal operational Yellow Non fatal failure Red Fatal failure Turn the chassis off and then troubleshoot MASTER Off no power or this management module is not installed Green Green Management module is operating as a master i e the other management module is a slave Yellow Management module is operating as a slave i e the other man agement module is a master 2 1 6 Port Connector Interfaces The connector side of the switch has 6 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a
70. upply Leaf Fan Modules 4 Leaf Fan Modules Switching Capacity 8 6 Tb s Spine Fan Mod 3 Spine Fan Modules with 2 ules integral fans per module Spines 3 Management Modules 2 available 1 required for oper ation Table 34 lists the approved certification status per chassis in different regions of the world Table 13 Certifications TPN FCC VCCI EN ICES CE CB cTUVus KCC C TICK CCC GOST R S MARK MIS5100Q 3DNC Mellanox Technologies 69 Rev 1 7 A2 EMC Statements USA A 2 1 FCC Statements USA Class A Statements 8 15 19 a 4 This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 This device must accept any interference received including interference that may cause undesired operation 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate
71. upports of the rack A Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your Chassis will be shipped in two packages one with the all of the leafs leaf package that are ordered and an extra management module should it be ordered and the second package chassis package with all of the rest of the parts The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management See the Installation Guide that is packed in the box for the instruc tions to make the initial configuration This unit is intended for installation in a Restricted Access Location A L restricted access area can be accessed only through the use of a special tool lock M and key or other means of security Chassis Package Contents The package includes 20 1 chassis with the following moduled installed 3 spines O leafs 1 management module 4 fans 3 PSUsIf you wish you may reduce the weight of the chassis by removing all of the spines power supply units fan units and management modules and put aside for reinstalla tion after the chassis is installed in the rack e installation kit e 1 box containing 3 power cords 125V 10A C14 to C13 6 feet long USA UL Standard and 1 RJ45 to DB9 harness e 1 cable management kit e 1 installation guide e user manual this document
72. users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniB and architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Web Sites Switch Firmware and Firmware Update Tools See http www mellanox com gt Downloads gt Firmware Tools Note that the Switch System described in this manual is based on Mellanox Technologies InfiniScale IV switch device FabricIT Enterprise Fabric Management Software See CLI User s Manual http www mellanox com gt Products gt Management Software Select FabricIT EFM InfiniBand Fabric Management Click Software and Documentation Download center select User Manual IB Switch Platforms Embedded Management Tools Describes the Embedded Management Tools EMGT package Document no 2141UM which provides Out of Band management capabilities for InfiniBand switches Available from http docs mellanox com requires a Mellanox Technologies customer account Y This symbol makes recommendations to the user AP This symbol indicates information that is helpful to the user gt Mellanox Technologies 9 This symbol indicates a situation that can potentially cause damage to hardware or soft Ware BEWARE This symbol indicates a situation that
73. von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden gt 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen gt 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 11 Richtigen Schutz sicherstellen Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen gt 12 Strahlenkontakt Achtung Nutzung von Steuerungen oder Einstellungen oder Ausfiihrung von Proze duren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkontakt fiihren Klasse Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 13 UL und CSA Certified Netzkabel UL Listed and CSA Certified Power Supply Cord F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel

108 Port InfiniBand QDR Switch Platform User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents