Home

User Manual

1. Ta UEM EE RR mu 1 r 3 4 6 f 8 9 10 REE 13_ 134 15 sr nnn CN RR GRO ES he te 13 14 15 umm E 1 2 4 5 6 T B g 10 hipi Sre nnn nn nn DURS 1 2 4 6 Dn B8 9 10 hec CH 1 2 3 4 zi 6 PR 8 x 10 1 BEA Li cono d Management Modules Each Leaf has 18 QSFP connector ports The switch can hold up to 18 leafs Leaf Blanks must be in place when a slot is not populated with a Leaf This is necessary to ensure proper air flow for cooling the leafs and spines 54 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Appendix 5 Interface 20 np 12 __ 21__ gt 18 __ Table 8 InfiniBand QSFP Connector 22 Rxtp __ Pinout 23 GND 16 24 15 Connector Connecto Te sun Pin Signal Description H Rx3p Number Name CND END TN 1 GND Ground E Modern SPA arcum 2 Tx2n Transmitter Inverted Data Input m mit Sec 10 3 Tx2p Transmitter Non Inverted Data Input 30 Vee Rx 4 GND Ground PA 8 5 Tx4n Transmitter Inverted Data Input 32 GND exe 7 6 Tx4p Transmitter Non Inverted
2. When the Yellow board status LED labelled ATTN is on this indicates that there is a fault in the leaf board If the Yellow LED is on uninstall and reinstall the leaf board For more information see Section 9 TroubleShooting on page 47 Figure 4 Leaf Board Status Indications Fault When lit there is a fault in the leaf board wp p t dq t pap dp DD cod gp d bi pil 1 2 3 Power When lit there is a proper power supply to the leaf board Each of the 18 connectors on the Leaf board has two LEDs that indicate if there is a physical and logical connection The figure below explains the connector status according to its LED display Mellanox Technologies 15 Rev 1 7 Safety Warnings and LED Indications Figure 5 Connector Physical and Logical Link Indications Table 4 Connector LEDs Port number LED Description Physical Link Green Constant on indicates a good physical link Blinking indicates a problem with the Physical link Off indicates No Physical link Cable may be unseated or unconnected Data Activity Yellow Blinking indicates Data Transfer Blinking rate is proportional to traffic rate Constant on indicates no Data Transfer Note The switch does not provide a visual means that indicates the port speed configuration SDR DDR or QDR and or the link width 1X or 4X The speed and link width configura tions can be retrieved using FabricIT management software
3. Figure 6 LED Indications When the Leaf is Working Properly ap p t d i ap pap dp d op be pil 7 2 This LED should b off The green connector The yellow connector LEDs should be on LEDs should be blinking This LED should be on 2 2 3 Spine Board LED Indicators Each spine has two status LEDs for the spine and 72 status LEDs for the leaf boards Each leaf board is connected by two links to each spine module These are displayed on the spine as link A and link B The two status LEDs for the spine have the configuration shown in Figure 7 The LEDs indicate as follows 16 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 e ATTN Yellow indicates fault in spine PWR Green indicates that the spine is receiving the proper voltage Figure 7 Spine Board Status Indications ATTN Yellow indicates fault in spine normally off d MTS3610 Links A and B to each leaf ATTN PWR Leaf numbers Green indicates Power OK Yellow indicates that there is a logical connection to the spin through this link Green indicates that there is a physical connection to the spines through this link The 72 status LEDs for the leafs have the configuration shown in Figure 7 The leaf number is fol lowed by four LEDs These LEDs indicate the condition of the leaf as follows Link A and Link B Each of the leafs is connected to each of the spines throug
4. _ Holes in the faceplate for mount ing on the rack 4 eh y 6 1 1 Removing the Chassis 1 Screw the four Eye bolts to the four corners of the top of the chassis 2 Connect the eye bolts to a mechanical lifting device Figure 29 Mechanical Lifting Device 3 Raise the chassis 2 cm 1 above the level of the shelf 4 Move the chassis away from the rack 5 Remove the mechanical lifting device 6 Remove the ten caged nuts from the vertical rack support Mellanox Technologies 43 Rev 1 7 Disassembly and Disposal 6 1 2 Removing the Bottom Shelf 1 Remove the four screws on the same side of the rack that are holding the shelf 2 Slide the shelf out from the slide rails Figure 30 Shelf Removal OO Em Ros 3 Remove the four caged nuts 4 Remove the rail slides from the rack Figure 31 Rail Slide Removal Remove these screws and their caged nuts 5 Remove the four caged nuts 6 2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not dis posed of with regular household waste Dispose of this product and all of its parts in a responsible and environ B mentally friendly way 44 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 7 Switch Management Tools This chapter describes the management tools The management module all
5. operational Red PS fault detected User should check individual power supplies for fault indications FAN SPINE off No power to fan Green Green operational Red Fault detected fan broken can t reach full RPMs 18 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Normal LED Description State FAN LEAF off No power to fan Green Green operational Red Fault detected fan broken can t reach full RPMs MASTER off no power SLAVE Green Management module is operating as a master i e the other management module is a Slave Yellow Management module is operating as a slave other management module is the master When only one management is present in the system default is Green On Mellanox Technologies 19 Rev 1 7 Installation 3 3 1 3 2 20 Mellanox Technologies Installation Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment When first installed the chassis needs an initial configuration of the management module See the Installation Guide for instructions on Initail configuration and remote management This ensures that OpenSM will be running on the system After the inital configuration the chassis will start up using an InfiniBand fabric standard compliant Subnet Managemen
6. B to each leaf ATTN PWR Leaf numbers Green indicates Power OK Yellow indicates that there is a logical connection to the spin through this link Green indicates that there is a physical connection to the e spines through this link 3 2 6 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from several ven dors The standard cables support full 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors which are numbered 1 18 from left to right Note If maximum cable lengths are exceeded data transfer will be reduced and Berr will increase All cables can be inserted or removed with the unit powered on To insert a cable press the con nector onto the port receptacle until the connector is firmly seated The green LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices To remove disengage the lock and slowly pull the con nector away from the port receptacle For a valid physical connection both ends of the cable must be connected to working devices tilation holes next to the InfiniBand ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending Warning
7. CSA Certified Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certified 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das interna tional harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 10 A 13 Ableitstrom gt 3 5mA WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung N von wesentlicher Bedeutung werden Mellanox Technologies 71 Rev 1 7 14 Add GND Verbindung Informationen Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben N dieses Ger t muss an den Schutzleiter in der Geb udeinstallation 15 INSTALLATION CODES Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimm ung mit den geltenden Vorschriften in der US amerikanischen National Elec trical Code und dem Canadian Electrical Code 16 Zusammenschaltung von EINHEITEN Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohn sitz in LPS Schaltung berstromschutz
8. Data Input 33 Es D 6 7 GND Ground 34 Tx3n 5 8 ModSelL Select 35 GND 4 9 ResetL Module Reset 1 2 10 3 3 V Power supply receiver 1 11 SCL Ze Sn clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxin Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Vee Tx 3 3 V Power supply transmitter 30 Vee 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input Mellanox Technologies 55 lews 18 35 Top QSFP Connector Male and Female Vi View into Rear of Connector Figure 34 Rev 1 7 GND Ta n TXip GND 2 TX2n N9 TX2p GND i GND TX3n upxu TX3p dpxu TX4p GND LPMode GND ModSell ResetL 11SIdPON VccTx u VocRx 9 Lal nt SCL XL99 id ModPrsL M 2 Loon MS i GND GND amp 001
9. Eine leicht zug ngliche Auflistung Abzweigleitung bers trom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verka belung 72 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Appendix I Special Regulations Regarding Finland Sweden mark and Norway 1 Denmark Unit is class I unit shall be used with an AC cord set suitable with Denmark deviations Cord shall including an earthing conductor Unit shall be plugged into a wall socket outlet which jis connected to protective earth Socket outlets which are not connected to earth shall not be used Do not connect this unit to any outlet that is not fully grounded gt 2 Finland Laite on liitett v suojamaadoituskoskettimilla varustettuun pistora siaan gt 3 Norway Apparatet m tilkoples jordet stikkontakt Unit is intended for connection to IT power systems for Norway only 4 Sweden Apparaten skall anslutas till jordat uttag Mellanox Technologies 73
10. Switch Platform User s Manual Rev 1 7 5 Insertion and or Hotswap Extraction Before starting any procedure on the MTS3610 Switch system put an ESD prevention wrist strap on your wrist and connect to the MTS3610 Chassis This switch platform supports Hotswap capabilities for the Power supply units Leaf boards Spine boards Leaf fan module Spine fan module Management modules 5 1 Power Supply Units A PSU Power Supply Unit can be extracted without bringing down the system At least five PSUs must remain working during a hotswap The power required to run the switch system is equally divided between all of the working PSUs When initially populating the system make sure that the five vertically oriented PSUs are installed before the horizontally oriented PSUs 5 2 Extracting and Inserting the Power Supply Unit With more than six power supplies installed the system is in the redundant configuration any sin gle PSU may be extracted without bringing down the system To extract a PSU 1 Remove the power cord from the power supply unit 2 Unscrew the cable clip and put the clip and screws aside for later Mellanox Technologies 33 Rev 1 7 Insertion and or Hotswap Extraction 5 3 5 3 1 Figure 21 Power Supply with Cable Attached Handle Cable clip and screws Latch 3 Grasping the handle with one hand push the black latch release while pulling the handle out ward As the PSU unseats the PS
11. Table 2 Revision History of this User s Manual Revision Date Details 1 7 Dec 2009 Imported new template Added Fabric IT Management Chapter 8 Added When first installed the chassis needs an initial configuration of the management module See the Installation Guide for instructions on Initail con figuration and remote management After the inital configuration the chassis will start up using an InfiniBand fabric standard compliant Subnet Manage ment software in Ch 3 on page 20 Added power cable type to parts list 1 6 May 2009 Added WEEE Disposal section Added Disassembly instructions Explanation of the Mammoth Chassis GUID Minor formatting changes Removed line 5 Make sure the Leaf board and the HCA cards have the same version of the FW 1 5 May 2009 Updated management module LEDs and Reset Restore button 1 4 May 2009 Removed parts of section 6 and replaced with Fabric IT link Photoshopped in new management module graphics Added Cable holder installation 1 3 April Updated installation with new parts 2009 1 2 Mar 2009 Updated installation with new parts 1 1 Feb 2009 added max time a fan module can be removed and LED reset of the fan leds 1 0 Jan 2009 Added Installation instructions and added info on EMGT 8 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 1 Overview This User s Manual provides
12. and equipment damage 5 Redundant Power Supply Connection Electrical Hazard This product includes a redundant power or a blank in its place In case of a blank power supply do not operate the product with the blank cover removed or not securely fastened 6 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or disconnect cables gt 12 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 7 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warnings instructions gt 8 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equipment starting from the bottom to the top N 9 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qualified personnel 10 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regulations 11 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes gt 12 Battery Replaceme
13. both its upper and lower handles Mellanox Technologies 37 Rev 1 7 Insertion and or Hotswap Extraction Figure 24 Leaf Fan Module Extraction Retaining Screws Handles 5 5 1 2 Inserting the Leaf Fan Module 1 Slowly slide in the new leaf fan module while holding both its upper and lower handles 2 Push the fan module until the front of the fan module is even with the surrounding panel 3 Make sure that the yellow leaf LED on the management module is off indicating that all fans are rotating 4 Screw back the four captive screws 5 5 2 Spine Fan Modules The spine fan module contains four individual fans Should a single fan fail the LED on the man agement module will light indicating the necessity to replace the fan module The air flow through the module is outside air is brought in through vent holes below the spines pushed through the spines pushed out of the chassis by the spine fan module above the leafs Air flow through the spines is independent of the air flow through the leafs 38 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 5 5 2 1 Extracting the Spine Fan Module 1 Unscrew the four captive screws that hold the spine fan module to the chassis 2 Slowly pull out the fan module while holding both its handles Figure 25 Spine Fan Module Extraction Handles Retaining Screws 5 5 2 2 Inserting the Spine Fan Module 1 Sl
14. sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits Figure 19 Electrical Cable Clamp Clamp to remove strain on the plug The switch system requires at least six PSUs to operate a fully populated platform The seventh and eighth PSUs provide a failover protection mechanism such that the system will continue oper ating if one or more of the PSUSs fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU even if it has only six PSUs When only six PSUs are installed five of them must be installed vertically along the right side of the chassis The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs 28 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Figure 20 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Risque de choc et de danger hazard Schocks Entferrnen des electriques Le de branch The PSUs are all independent Netzste
15. 1 7 Insertion and or Hotswap Extraction Figure 26 Management Module 09090900900000000000000000000000000000000 111 LAJ nee LJ 70003200699 ir Tees TI AA 75361 0 1 ad retaining screws red buttons ejectors 5 6 2 Inserting a Management Module 5 1 1 Verify the ejectors are unlocked by pressing the red buttons and pulling the ejectors outward on both sides of the board slot 2 Carefully seat the management module into the side guide rails and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated 4 Screw in the captive retaining screws on the outside of each ejector to lock the management module in place Replacing the Battery The battery is only used for the clock within the management module If the Battery is dead the time on the management module will not be updated The battery is a lithium 3V CR2032 Replace the battery by 1 Remove the Management module from the chassis 2 Put on an ESD strap and ground the strap to a valid ground 3 Remove the battery from the holder 4 Insert the new battery side up 40 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 5 Reinsert the management module in the chassis 5 8 Over temperature Shutdown Mechanism Each leaf boar
16. 2 Physical Installation 20 3 2 1 ESD Connection 21 3 2 2 Installing the Chassis 21 3 2 3 Ground Connections 26 3 24 Installing the Cable Holder 27 3 2 5 Power Connections 27 3 2 6 InfiniBand QSFP Cable Installation 30 Chapter 4 Switch Power Up 31 4 1 Power Supply and Spine Board Indicator Status at Power ON 31 4 2 InfiniBand Fabric Initialization and Management 31 Chapter 5 Insertion and or Hotswap Extraction 33 5 1 Power Supply Units 33 5 2 Extracting and Inserting the Power Supply Unit 33 5 3 Leaf Boards 34 5 3 1 Extracting a Leaf Board 34 5 3 2 Inserting a Leaf Board 35 5 4 Spine Boards 35 5 4 1 Extracting a Spine Board 35 5 4 2 Inserting a Spine Board 36 5 5 Fan Modules 37 5 5 1 Leaf Fan Module 37 5 5 2 Spine Fan Modules 38 5 6 Management Module 39 5 6 1 Extracting a Management Module 39 5 6 2 Inserting a Management Module 40 5 7 Replacing the Battery 40 5 8 Over temperature Shutdown Mechanism 41 Chapter 6 Disassembly and Disposal 42 6 1 Disassembling the Chassis 42 Mellanox Technologies 3 Rev 1 7 6 1 1 Removing the Chassis 43 6 1 2 Removing the Bottom Shelf 44 6 2 Disposal 44 Chapter 7 Switch Management Tools 45 7 1 Chassis Management 45 Chapter 8 FabricIT Management 46 8 1 Downloading FabricIT Software and Documents 46 8 2 Installing Licenses 46 8 2 1 FabricIT Management and Inspection License 46 Chapter 9 TroubleShooting 47 9 1 Power Supply Unit 47 9 2 LeafBoard 47 9 3 Management Mod
17. 8438_lid 0x0013 dev mst SW_MT48438_lid 0x0014 dev mst SW_MT48438_lid 0x0015 dev mst SW_MT48438_lid 0x0016 dev mst SW_MT48438_lid 0x0017 dev mst SW_MT48438_lid 0x0018 dev mst SW_MT48438_lid 0x0019 dev mst SW_MT48438_lid OxO01A dev mst SW_MT48438_lid 0x001C dev mst SW_MT48438_lid 0x001D dev mst SW_MT48438_lid Ox001F mtlsqt131 home folder flint d dev mst SW_MT48438_lid 0x0002 q Image type FS2 FW Version 7 1 903 Device ID 48438 Chip Revision AO Description Node Sys image GUIDs 0002c90200405a10 0002c90200405a10 Board ID MT_0C60000003 VSD PSID MT 0C60000003 62 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Choose a lid which you want to burn and the following command line mtlsqt131 home folder mlxburn d dev mst SW MT48438 lid 0x0002 fw mswg release fw IS4 last_revision fw IS4 mlx I Querying device I Using auto detected configuration file mswg release fw IS4 last_revision MTS3611QC ini PSID MT_0C60000003 I Using auto detected hash file mswg release fw IS4 last revision hash csv Generating image Current FW version on flash 7 1 903 New FW version 7 1 918 Burning second FW image without signatures OK Restoring second signature OK I Image burn completed successfully Mellanox Technologies 63 Rev 1 7 Appendix F Replacement Parts Ordering Numbers Table 9 Replacemen
18. Band Switch Platform User s Manual Rev 1 7 Appendix G Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambi ante d passant le maximum recommand de 45 C 113 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouvertures de ventilation gt 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut provoquer des blessures corporelles et des d gradations de biens 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lectrique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solide ment fix ou s il est enlev 5 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles gt 6 Branchement d branchement des c bles InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention
19. Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 13 Codes d installation L appareil doit tre install selon l ancienne version des codes lectriques nationaux du pays Pour l Am rique du Nord l quipement doit tre install conform ment aux sp cifications du Code Electrique National Am ricain et du Code Electrique Canadien gt 14 Interconnexion des unites Les c bles de branchement l unit RS232 et les interfaces Ethernet doivent tre certifi s UL de type DP 1 ou DP 2 Note lorsqu il existe dans un circuit non LPS Protection contre la surintensit Un appareil de protection r pertori facile ment accessible contre la surintensit du circuit de branchement et calibr 20A doit tre incorpor dans le c blage lectrique du b timent gt 68 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 15 Remplacement des piles Il existe un risque d explosion si les piles sont remplac es par des piles d un type incorrect Jetez les piles us es selon les instructions 16 Cordons lectriques CA homologu s UL Pour les prises lectriques en Am rique du Nord choisissez un cordon lec trique homologu UL et certifi CSA 3 conducteurs 18 AWG termin par une fiche moul e d une tension nom inale de 125 V 15 A avec une longueur minimale de 1 5 m 6 pieds et d une longueur maximale d
20. Mellanox TECHNOLOGIES 324 PORT QDR InfiniBand Switch Platform User s Manual MTS3610Q 6UNC MTS3610D 6UNC Rev 1 7 www mellanox com Rev 1 7 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF S
21. Press the ejectors outward so that the board pops out then slowly slide the board out Note The board is short therefore do not let go of it while sliding it out Note The print side of the leaf board does not have a base plate it is covered with a plastic Nomex sheet be careful not to fold or damage the Nomex Please handle the board carefully Figure 22 Retaining Screw and Red Button Retaining Screws Red Buttons 5 3 2 Inserting a Leaf Board 1 Verify the ejectors are unlocked by pressing the red buttons and pulling the ejectors outward on both sides of the board 2 Carefully seat the leaf board into the side guide rails and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated 4 Screw in the captive retaining screws on the outside of each ejector to lock the leaf board in place Note The leaf board does not have a base plate thus the print side components are exposed Please handle the board carefully 5 4 Spine Boards Each switch spine board has a pair of ejectors that lock the board in place and serve as levers for seating or extracting see Figure 23 5 4 1 Extracting a Spine Board 1 Unscrew the locking screws on both of the latches 2 Press the latch button on each of the spine ejectors and pull open both latches Open both latches simultaneously until they are 90 d
22. Take care to not impede the air exhaust flow through the ven upward or downward in the rack 30 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 4 Switch Power Up Before starting any procedure on the MTS3610 Switch system put an ESD prevention wrist strap on your wrist and connect to the MTS3610 Chassis 1 Check all boards power supplies e Leaf fan module Spine fan module for proper insertion and seating before connecting the AC power cords 2 Insert thermal blanks in unused leaf slots to maintain balanced air flow 3 Tighten all mounting screws 4 Connect the power cords to the PSUs 5 Clamp down the cord using the electrical cable clamps 6 Connect the power cords to grounded electrical outlets Note A fully loaded MTS3610 switch system can draw 5 2 kW of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits 4 1 Power Supply and Spine Board Indicator Status at Power ON As the power is turned on you should observe the following conditions for normal operation 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existin
23. Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 4 Move the chassis over the shelf Place the chassis so that the face plate of the chassis is a few inches from the vertical rack support 5 Lower the chassis onto the shelf 6 Remove the mechanical lifting device 7 Place ten caged nuts into the vertical rack support at the locations corresponding to the holes in the chassis faceplate Figure 16 Face Plate Mounting Bolt Locations oles in the faceplate for mounting on the rack 8 Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack sup port 9 Screw the bolts through the faceplate into the caged nuts and tighten all ten bolts and nuts 10 Place the lock down bar over the lip of the MTS3610 and align the holes in the lock down bar with the holes in the shelf Mellanox Technologies 25 Rev 1 7 Installation Figure 17 Lock Down Bar The lock down bar goes over the lip on the bottom of the chassis 11 Place a washer on each Allen screw and screw down the lock down bar to the shelf using the Allen wrench supplied Use all 5 Allen screws 12 Connect a valid ground to the grounding post see Section 3 2 3 13 Go to Section 4 Switch Power Up on page 31 to power up the switch system 3 2 3 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of suffic
24. U status indicators will turn off 4 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation 4 Insert the power cord into the supply connector 5 Install the cable clip using the two screws Note The green indicators should light If not extract the PSU and re insert it again Leaf Boards The leaf boards are numbered from top to bottom with corresponding numbers displayed to the left of the leafs vertically along the side panel Extracting a Leaf Board Switch leafs are located on the back of the system Each switch leaf board has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 22 The ejec tors snap inward to lock the board into place and include a red button to disengage the lock and electrically notify the system that an extraction has been requested 1 Disconnect all cables connected to the leaf board 2 Unscrew the retaining screws on the outside of the connectors These screws are captive and will remain hanging in place See Figure 22 34 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 3 Press the red buttons to unlock the ejectors 4
25. UCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2009 Mellanox Technologies Inc All Rights Reserved Mellanox ConnectX InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale and InfiniPCI are registered trademarks of Mellanox Technologies Ltd BridgeX PhyX and Virtual Protocol Interconnect are trademarks of Mellanox Technologies Ltd All other marks and names mentioned herein may be trademarks of their respective companies InfiniScale IV 324 Port QDR InfiniBand Switch User s Manual 2 Mellanox Technologies Document Number 2981 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Table of Contents Table of Contents 3 Table of Figures 5 Table of Lists 6 About this Manual 7 Intended Audience 7 Related Documentation 7 Revision History 8 Chapter 1 Overview 9 1 1 Serial Number and Product Version Information 9 Chapter 2 Safety Warnings and LED Indications 12 2 1 Installation Safety Warnings 12 2 2 LED Status Indicators 15 2 2 1 Power Supply Unit LEDs 15 2 2 2 Leaf Board LED Indicators 15 2 2 3 Spine Board LED Indicators 16 2 2 4 Management Module LED Indicators 18 Chapter 3 Installation 20 3 1 Package Contents 20 3
26. an overview of the 19U QSFP DDR QDR Modular InfiniBand Switch Platform the switch and the operational environment Mellanox MTS3610 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers High Performance Computing and Embedded environments Networks built with MTS3610 systems can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 4th generation InfiniScale IV InfiniBand switch device MTS3610 systems provide up to 40Gb s full bidirectional bandwidth per port With 324 ports these systems are among the densest switching systems available The switch platform comes pre installed with all necessary firmware and configured for standard operation within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager run ning from one of the hosts All that is required for normal operation is to follow the usual precau tions for installation and to connect the switch to the HCAs Once connected the Subnet Management software automatically configures and begins utilizing the switch It is recommended that the Mellanox OpenFabrics software package be installed on all nodes con nected to the MTS3610 The software package provides a subnet manager and network manage ment tools as well as connectivity software for servers and storage and is available on the Mellanox web site Basic ins
27. anox Technologies 51 Rev 1 7 Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equip ment generates uses and can radiate radio frequency energy and if not installed and used in accor dance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 1 1 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate A 1 2 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada A 1 3 VCCI Statements Japan Class A Statement ZNE 75 TT LORE RER CERT ET SNRA EAEN EN WREBTSLIBRKSENSATEPHVWETF VCCI A Translation This is a Class A product based on the standard of the Voluntary Control Council for Inter ference by Informatio
28. b page FabricIT http www mellanox com content pages php pg fabric it login Note that you will need to enter the License Entitlement number and the chassis S N to obtain the software and documentation and also to generate an EFM license that enables extended FabricIT management and inspection features on your switch system 8 2 Installing Licenses A license for extended FabricIT management and inspection features with FabricIT EFM software is available for an additional licensing fee 8 2 1 FabriciT Management and Inspection License To activate the extended FabricIT management and inspection features you need to install the license that was purchased along with the switch system Without the license you will not be able to run a Subnet Manager on the switch nor run advanced fabric diagnostics For information on installation please refer to the FabricIT Enterprise Fabric Management Soft ware CLI User s Manual 46 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 9 TroubleShooting As each PSU is plugged in make sure that the green power LEDs on the PSU come on 9 1 Power Supply Unit Note If the Power supplies cannot supply enough power the management module may shut down the system If the AC power LED is off 1 Check that the power cable is the correct power cable for your country 2 Check that the power cable is plugged into a working outlet 3 Check that the power
29. cable has a voltage within the range of 100 240 volts AC 4 Remove and reinstall the power cable 5 Check the circuit breakers to be sure that the breaker has not tripped 6 Check that the power cable is good Replace the power cable 7 If the AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU 9 2 Leaf Board The power LED for the Leaf board is off 1 Make sure that at least six PSUs are showing DC OK 2 Uninstall and reinstall in the Leaf board 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the Leaf board 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board 5 Replace the Leaf board with a new one Note Should any of the boards shut down due to over temperature follow the procedure start ing in Step 2 The Physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable Mellanox Technologies 47 Rev 1 7 TroubleShooting 9 3 9 4 The Activity LED does not come on Check that the Subnet Manager has been started Management Mod
30. ckers elnes Netz ment d une seule alimentation teils spannungsfrei Um stabilise e de branch Disconnect all power supplies to alle Einhieten spannungs uniquement qu un module ensure a powered down state frei zu machen sind die Alimentation Stabilise e inside of the switch platform Netzstecker aller Netzteile Pour isoler completement le zu entfernen module en cause Il faut de brancher toutes les ali mentations stabilise es 1 As soon as a switch is plugged in make sure that the green power LEDs on the PSUs are on OK FAIL OK 2 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LEDs on any of the management modules is cause for concern and must be dealt with immediately 3 Check that none of the ATTN status LEDs on the spines are yellow 4 Go around to the other side of the switch chassis and check the leaf board status LEDs ap 0 qeu a pap dp d op prog Do Pm a oodd i Ip T 2 This LED should b off The green connector The yellow connector LEDs should be on LEDs should be blink This LED should be on Ing Mellanox Technologies 29 Rev 1 7 Installation 5 Check that the leaf physical and data link status LEDs in the spines are lit ATTN Yellow indicates fault in spine normally off E 7 MTS3610 Links A and
31. d and each spine board has its own on board thermal sensor to track PCB tempera ture If a PCB s temperature exceeds 70 C the thermal sensor shuts down the over heated card and notifies the management module Mellanox Technologies 41 Rev 1 7 Disassembly and Disposal 6 Disassembly and Disposal 6 1 Disassembling the Chassis 1 Power down the chassis 2 Remove all power cables 3 Remove all connector cables 4 Disconnect the Ground lug from the ground post 5 Remove all leafs See Section 5 3 1 6 Remove all spines See Section 5 4 1 7 Remove all management modueles See Section 5 6 1 This product and all of its parts are NOT to be disposed of with house hold waste This product contains printed circuit boards cables and bat teries According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and disposed of according to the directive 8 Using an Allen wrench remove the Allen screws holding down the loek down bar on the rear side of the chassis Figure 27 Lock Down Bar The lock down bar goes over the lip on the bottom of the chassis 9 Remove the lock down bar 10 Remove the 10 screws holding the chassis to the rack These screws are located in the faceplate on the rear of the chassis 42 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Figure 28 Face Plate Mounting Bolt Locations
32. e 4 5 m 18 pieds Pour les prises lectriques en Europe choisissez un cordon lectrique harmon is internationalement et marqu lt HAR gt 3 conducteurs d un diam tre de fil minimum de 0 75 mm2 d une tension nominale de 300 V avec une gaine isol e en PVC Le cordon doit avoir une fiche moul e d une tension nominale de 250 V et d une intensit nominale de 10 A Mellanox Technologies 69 Rev 1 7 Anhang Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromver N sorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemper atur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleis ten gt 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunterf llt Kann es zu Verletzungen und Besch digungen an Ger ten f hren gt 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redun dante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeck ung nicht sicher festsitzt
33. egrees from the face of the spine Mellanox Technologies 35 Rev 1 7 Insertion and or Hotswap Extraction 3 Keeping the latches fully open pull the spine out of the chassis at least 2 inches 6cm Expect some resistance before the board pops out Note If the spine is not far enough out of the chassis the top latch may fall slightly and catch the top of the spine to the chassis 4 Hold the board with one hand supporting its bottom and the other grasping its middle Then slide the board all the way out Note Do not damage the back plane connectors on the board and or the EMI gasket of the spine slot in the system Figure 23 Spine Board Extraction Latch button on the spine ejector Two latches OL9ES LIN Locking screw Model number indicating the top of the spine 5 4 2 Inserting a Spine Board The top of the spine is the side with the model number and the two spine LEDS 1 Hold the spine board with the top up with one hand supporting its bottom and the other grasping its middle 2 Insert the board into its rail The board should slide in without resistance 3 Once the board is about half way inside its rail move your hands so that they hold the spine ejectors and push the board in until it cannot move 4 Make sure the spine ejectors are open all the way Now press both ejectors inward simultane ously You should feel the ejectors are hooked into the front panel 5 Press both ejectors applying some force
34. en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour connaitre les mises en garde et les instructions sp ciales gt 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr amp cau tions sp ciales pour s assurer que le syst me reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut gt Mellanox Technologies 67 Rev 1 7 8 Fuite gt 3 5mA ATTENTION La connexion 4 la terre des forts courants de fuite est essen tielle avant le branchement de l alimentation Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent gt 9 Ajouter une information de connexion la masse Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent gt 10 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi gt 11 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gis lations et r glementations nationales en vigueur 12 Codes lectriques locaux et nationaux
35. g leaf Boards should be ON 4 There is a yellow Status LED per leaf board and per spine board Make sure they are all off 5 Check the management module LEDs and make sure they coincide with Table 5 on page 18 4 2 InfiniBand Fabric Initialization and Management Subnet Management and Subnet Administration are standard components defined by the Infini Band Specification which implement fabric initialization discovery and configuration The switch s switching fabric can be managed by any third party Subnet Management software running on an InfiniBand Host that is connected to any switch port The switch is also compatible with the Open Source Subnet Manager OpenSM The switch is basically an InfiniBand fabric built out of individual InfiniBand switch devices and appears so to external Subnet Managers The switch supports the SystemImage GUID feature of the InfiniBand specification which means that software management tools that support this feature can recognize all the internal devices as a single switch system Mellanox Technologies 31 Rev 1 7 Switch Power Up A fully populated 324 port system features up to 25 9 Tbit sec IB ports in QDR mode Constant Bisectional Bandwidth CBB Fat Tree topology that the Subnet Manager will discover and con figure to enable a full cluster to operate For more information refer to the documentation of the Subnet Manager being used 32 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand
36. h two links Yellow indicates that there is a logical connection to the spines through this link Green indicates that there is a physical connection to the spines through this link Mellanox Technologies 17 Rev 1 7 Safety Warnings and LED Indications 2 2 4 Management Module LED Indicators The switch system requires at least one management module The management module has three interfaces to connect to the MTS3610 They are e RS232 CONSOLE connector USB port Ethernet There is also a Reset Restore button This button resets all of the leafs in the chassis by bringing them down and powering them up when the button is pushed When the button is held down for 15 seconds the chassis is reset and the factory defualt values for the configuration are implemented Figure 8 Management Module Status Indications LOI M M M eee NN YY STATUS PS FAN MASTER RESET LEAF SLAVE The management module LEDs display the switch system operating conditions Table 5 LEDs Display for Normal Operation Normal LED Description State STATUS Off No Power Green Green Unit is operating well all temp and voltages are OK Yellow System warning Attention needed such as overheating Red z System not operational Diagnostics fail CPU hang HW fail Overheat critical Blinking Yellow Restore factory defaults in progress PS off No power Green Green
37. he other side of the rack Figure 11 Vertical Rails to Cabinet Measurement Note The side of the chassis with the power supply units will sit flush with the vertical supports of the rack The side of the chassis with the QSFP connectors will sit at the edge of the shelf closer to the center of the rack Note place the chassis as low as possible in the rack to keep the center of gravity as low as pos sible 2 Insert four caged nuts into the rack on the side you plan to have the connectors Note the U num ber on the rack Figure 12 Caged Nut Insertion 3 Screw the rail slides onto the rack Mellanox Technologies 23 Rev 1 7 Installation Figure 13 Rail Slide Installation The caged nuts are separated by a single space 4 Insert four caged nuts into the rack on the side you plan to have the powers supplies and spines Use the same U number on this side of the rack as you used on the rail side of the rack 5 Prepare four bolts and keep them within reach 6 Slide the shelf onto the rail slides 7 Connect the shelf to the rack Figure 14 Shelf Installation EN 8 Tighten all eight bolts into the caged nuts 3 2 2 2 Inserting the Chassis 1 Screw the four Eye bolts to the four corners of the top of the chassis 2 Connect the eye bolts to a Mechanical lifting device Figure 15 Mechanical Lifting Device 3 Raise the chassis 2 cm 1 above the level of the shelf 24 Mellanox
38. ient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the con nections are solid and permanent If the chassis is rack mounted and you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground con nection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 18 Ground Connection Grounding post Grounding lug 26 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 3 2 4 Installing the Cable Holder Left side rack Top side Am EN Holes for con necting the cable holder to the rack 8 X8 D e Right side rack Bottom side 1 Place the Cable holder next to the rack and identify the holes for the caged nuts The numbers on the cable holders should face each other 2 Put the caged nuts into the previously identified holes in the rack 3 Screw the cable holder onto the rack using the screws provided 4 Repeat steps 1 to 3 for the second cable holder 3 2 5 Power Connections The switch includes integrated hot swap po
39. ight fit Figure 9 ESD Connector NEE ESD connec tor 3 2 2 Installing the Chassis You will need 2 phillips screwdriver a mechanical lift a grounding lug Mellanox Technologies 21 Rev 1 7 Installation ground wire of sufficient length and gauge to properly ground the chassis Note It is recommended to use AWG6 or 4mm diameter wire for grounding purposes Parts included in the installation kit 1 shelf 5 Allen head screws e 1 lock down bar 5 washers 4eye bolts 1 Allen wrench 2rail slides 26 sets of caged nuts and bolts 1 right hand cable holder 1 left hand cable holder Figure 10 Installation Kit Parts Lock down bar De Rail slide Eye bolts 8 Caged nuts bolts amp LU 8 a or Beh X26 S 7 22 Right Left Cable holders Before starting the procedure put the ESD strap on connect it to a valid ground 22 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 3 2 2 1 Installing the Bottom Shelf 1 Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low Choose which way you want the chassis to face in the rack Keep in mind that one side of the chas sis can have up to 324 cables connected The other side of the chassis will have power cables The distance between the rack and the door will be small on one side of the rack as opposed to t
40. llow LED labelled ATTN on the Spine board is lit 1 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 5 4 on page 35 4 Make sure that the spine and the Leafs both have the same version of FW 48 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 5 Reburn FW remove reinstall the 6 Replace board 9 5 Management Module 1 Check that there is power to the management module 2 Remove and reinstall management module Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 5 6 on page 39 3 Make sure that the spine and the Leafs both have the same version of FW 4 Reburn the FW and remove and reinstall the management module 5 Replace the Management module Mellanox Technologies 49 Rev 1 7 Appendix A 190 DDR ODR IB Switch Specification Data 1 Certification Statements Table 34 lists the approved certification status per HCA card in different regions of
41. n Technology Equipment VCCT If this equipment is used in a domestic environ ment radio interference may occur in which case the user may be required to take corrective actions 52 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Appendix Mechanical Drawing Figure 32 Front Panel _ kx IR IE le lt gt a 0 8 AES 1 MTS3610 mrssio Mmrsasio Mrsseto 1 ATMTS3610 mrsssio t t Lee we ee oe ee eo ve ee oe BM 11111 Eight Power Modules 100 240 VAC 50 60 Hz eo e800 e000 e000 e200 oo eee o Nine spines Air inlets to Spines Mellanox Technologies 53 Rev 1 7 Figure 33 Rear Panel Leaf Side of the Switch LIE I ET N Spine Fan o Module snanu seaman m rs we 152 Kar Hot 120 445 15 p dn TEEGEE 10 12 13 ee er EE EE En SE E ees nuca ss
42. n a standard 19 rack The switch ships from the fac tory with mounting racks on the front side There are four threaded connectors on the top of the chassis for connecting eye bolts The four eye bolts are included in the installation kit The chassis can be lifted using these four eye bolts and a mechanical lift The switch is supported from under neath the unit by a shelf Choose a rack which is able to support the mechanical and environmental characteristics of a fully populated switch chassis as listed in Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis Note A 20U mounting space is required for proper installation of the switch platform The rack mounting holes conform to the IEA 310 standard for 19 inch racks Guarantee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis 3 2 4 ESD Connection Before starting any procedure on the MTS3610 Switch system 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap 2 Plug the other end of the wrist strap to the MTS3610 chassis ESD connector on the front of the switch system or connect it to a valid ground Make sure that this is a t
43. ng up the switch platform The manual assumes familiarity with the InfiniB and architecture specification Related Documentation The documentation set accompanying the 19U 18 slot QSFP Chassis InfiniBand Switch platform includes the following Table 1 Reference Documents and Web Sites Switch Firmware and Firmware Update Tools See http www mellanox com gt Downloads gt Firmware Tools Note that the Switch System described in this manual is based on Mellanox Technologies InfiniScale IV switch device FabricIT Enterprise Fabric Management Software Talk to your Mellanox representative for information regarding CLI User s Manual licensing and implementation of the FabricIT Enterprise Fabric Management Software System Mellanox OFED Stack for Linux User s Manual See http www mellanox com gt Downloads gt InfiniBand SW Driv ers Click Mellanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED Select the Linux User s Manual The embedded OS and tools on the CPU in the management module is a subset of the Mellanox OFED stack IB Switch Platforms Embedded Management Tools Describes the Embedded Management Tools EMGT package Document no 2141UM which provides Out of Band management capabilities for Infini Band switches Available from http docs mellanox com requires a Mellanox Technologies customer account Mellanox Technologies 7 Rev 1 7 Revision History
44. nt There is a risk of explosion should the battery be replaced with a battery of an incorrect type gt Dispose of used batteries according to the instructions Mellanox Technologies 13 Rev 1 7 Safety Warnings and LED Indications 13 UL Listed and CSA Certified Power Supply Cord Listed and CSA Certified 3 conductor 18 AWG terminated with a molded plug rated at 125 V 15 A with a minimum length of 1 5m six feet but no For North American power connection select a power supply cord that is UL longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 0 75 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 14 Leakage gt 3 5mA WARNING High leakage current Earth connection essential before connect ing supply gt 15 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation gt 16 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electrical Code gt 17 Interconnection of unit
45. o RX4p GND E3 RX4n RX3n 5 dex 8 GND GND 8 UEXL ve RX2p GE x iioi RXin dix Ie X GND EN QNS Mellanox Technologies 56 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Appendix D Ethernet RJ 45 Connector Figure 35 Ethernet RJ45 Connector Mellanox Technologies 57 Rev 1 7 Appendix Sample In band FW Upgrade mtlsqt131 home folder mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded MST devices dev mst mt25418_pciconf0 PCI configuration cycles access bus dev fn 07 00 0 addr reg 88 data reg 92 Chip revision is AO dev mst mt25418 pci crO PCI direct access bus dev fn 07 00 0 bar 0xe2000000 size 0x 100000 Chip revision is dev mst mt25418 pci msixO PCI direct access bus dev fn 07 00 0 bar 0x00000000 size 0x0 dev mst mt25418_pci_uar0 PCI direct access bus dev fn 07 00 0 bar 0xde000000 size 0x2000000 dev mst mtusb 1 USB to I2C adapter as I2C master Running the mst ib add for adding the in band devices you should run opensm before this opera tion 58 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 mtlsqt131 home folder mst ib add I Running ibdiagnet to discover the fabric Loading IBDIAGNET from usr lib64 ibdiagnet1 2 W Top
46. oder entfernt wurde gt 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 6 Anschlie en Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers 7 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind beson dere Vorsichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 70 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 8 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausgetauscht oder gewartet werden 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 11 Richtigen Schutz sicherstellen Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen 12 UL und
47. odules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded MST devices dev mst mt25418_pciconf0 PCI configuration cycles access bus dev fn 07 00 0 addr reg 88 data reg 92 Chip revision is AO dev mst mt25418 pci crO PCI direct access bus dev fn 07 00 0 bar 0xe2000000 size 0x 100000 Chip revision is AO dev mst mt25418 pci msixO PCI direct access bus dev fn 07 00 0 bar 0x00000000 size 0x0 dev mst mt25418_pci_uar0 PCI direct access bus dev fn 07 00 0 bar 0xde000000 size 0x2000000 dev mst mtusb 1 USB to I2C adapter as I2C master Inband devices dev mst CA MT25418 mtlsqt127 HCA 1 lid 0x0020 dev mst CA MT25418 1111159128 1 lid 0x000A dev mst CA MT25418 mtlsqt131 1 lid 0x001E dev mst CA MT25418 mtlsqt135 1 lid 0x001B dev mst CA MT25418 mtlsqt137 1 lid 0x0001 dev mst CA MT25418 mtlsqt138 HCA 1 lid 0x000F dev ms SW MT48438 lid 0x0002 dev ms SW 48438 lid 0x0003 dev mst SW_MT48438_lid 0x0004 dev mst SW_MT48438_lid 0x0005 dev mst SW_MT48438_lid 0x0006 Mellanox Technologies 61 Rev 1 7 dev ms SW MT48438 lid 0x0007 dev mst SW 48438 lid 0x0008 dev mst SW 48438 lid 0x0009 dev mst SW MT48438 lid 0x000B dev mst SW 48438 lid 0x000C dev mst SW MT48438 lid 0x000D dev mst SW MT48438 lid 0x000E dev mst SW_MT48438_lid 0x0010 dev mst SW_MT48438_lid 0x0011 dev mst SW_MT48438_lid 0x0012 dev mst SW_MT4
48. ology file is not specified Reports regarding cluster links will use direct routes Loading IBDM from usr lib64 ibdm1 2 Using port 1 as the local port I Discovering 32 nodes 26 Switches amp 6 CA s discovered Te ENTE MER I Bad Guids LIDs Info Toernee eeen venen ESE sana I skip option set no report will be issued EE SEN I Links With Logical State INIT el Reale NN a Du DT d skip option set no report will be issued Dee un E UE eeen PM Counters Info eene emp et ee AT skip option set no report will be issued Mellanox Technologies 59 Rev 1 7 S PMID EED Fabric Partitions Report see ibdiagnet pkey for a full hosts list RE I skip option set no report will be issued EEE I IPoIB Subnets Check nte MESS skip option set no report will be issued 5 Ee I Bad Links Info I No bad link were found dsten I Stages Status Report STAGE Errors Warnings Bad GUIDs LIDs Check 00 Link State Active Check 0 0 Performance Counters Report 0 0 Partitions Check 0 0 IPoIB Subnets Check 0 0 Please see tmp ibdiagnet log for complete log I Done Run time was 14 seconds I Added 32 in band devices 60 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Now you can see mammoth switches as in band devices on mst status output mtlsqt131 home folder mst status MST m
49. otiation of 20Gb s 40Gb s Safety US Canada cTUVus 8 InfiniBand Virtual Lanes for all ports EU IEC60950 QoS International CB Baseboard Performance and Device Management management Agents for full InfiniBand EMC USA FCC Class A In Band Management Canada ICES Class A EU EN55022 Class A EU EN55024 Class A EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A ENVIRONMEN EU IEC 60068 2 64 Ran TAL dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ACCOUSTIC ISO 7779 ETS 300 753 Scalability and Performance Replaceable Parts Switching Performance Simultaneous wire speed any port to any leafs to 18leafs 18 ports each port Power Supplies 6 Power Supplies standard Addressing 48K Unicast Addresses Max per Subnet with 2 optional redundant 16K Multicast Addresses per Subnet power supplies available Switching Capacity Leaf Fan Modules 1 Leaf Fan Module with 7 25 9 Tb s integral fans Spine Fan Mod 1 Spine Fan Modules with 4 ules integral fans per module 9 Spines Management 2 available 1 required for oper Modules ation A 1 FCC Statements USA Class A Statements amp 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsi ble for compliance Mellanox Technologies could void the user s authority to operate the equipment 815 105 Mell
50. otification 29 Figure 21 Power Supply with Cable Attached 34 Figure 22 Retaining Screw and Red Button 35 Figure 23 Spine Board Extraction 36 Figure 24 Leaf Fan Module Extraction 38 Figure 25 Spine Fan Module Extraction 39 Figure 26 Management Module 40 Figure 27 Lock Down Bar 42 Figure 28 Face Plate Mounting Bolt Locations 43 Figure 29 Mechanical Lifting Device 43 Figure 30 Shelf Removal 44 Figure 31 Rail Slide Removal 44 Figure 32 Front Panel 53 Figure 33 Rear Panel Leaf Side of the Switch 54 Figure 34 QSFP Connector Male and Female Views 56 Figure 35 Ethernet RJ45 Connector 57 Mellanox Technologies 5 Rev 1 7 Table of Lists Table 1 Reference Documents and Web Sites 7 Table 2 Revision History of this User s Manual 8 Table3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis 10 Table 4 LEDs Display for Normal Operation 18 Table 5 Cards Certification Status 50 Table 6 Switch Specification Sheet MTS3610D and MTS3610Q 50 Table 7 InfiniBand Connector Pinout 55 Table 8 Replacement Parts Ordering Numbers 64 6 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 About this Manual This manual provides an overview of the 19U 18 slot QSFP Chassis InfiniBand Switch Platform and guidelines for its operation Intended Audience This manual is intended for users and system administrators responsible for installing and setti
51. owly slide in the new spine fan module while holding both its handles 2 Push the fan module until as far as it will go 3 Make sure that the yellow spine LED on the management module is off indicating that all fans are rotating 4 Screw back the four captive screws 5 6 Management Module Before starting any procedure on the MTS3610 Switch system put an ESD prevention wrist strap on your wrist and connect to the MTS3610 Chassis 5 6 1 Extracting a Management Module Management modules are located on the rear side at the bottom of the switch system There are two places to install the management modules Note Only one management module is required to run the switch system Each management module has a pair of ejectors that lock the board in place and serve as a lever for seating or extracting see Figure 26 The ejectors snap inward to lock the board into place and each ejector includes a red button to disengage the lock and electrically notify the system that an extraction has been requested 1 Disconnect all cables connected to the management module 2 Unscrew the retaining screws on the outside of the connectors These screws are captive and will remain hanging in place 3 Press the red buttons to unlock the ejectors 4 Press the ejectors outward so that the board pops out then slowly slide the board out Note The board is short therefore do not let go of it while sliding it out Mellanox Technologies 39 Rev
52. ows the switch to be managed either locally or remotely The chassis can also be managed using Out of Band manage ment via FabricIT EFM The chassis comes with a module that will completely manage your chas sis n optional management module in combination with the licensing fee for full access to the FabricIT EFM software can manage a fabric of up to 2000 nodes 7 1 Chassis Management The chassis manager will give the user access to switch temperatures power supply voltages fan unit information power unit information Flash memory The manager also has the ability to burn new FW on the switch system Mellanox Technologies 45 Rev 1 7 FabriclT Management 8 FabricIT Management FabricIT EFM is a software based management system that can be run with either a command line interface or with a GUI interface The GUI interface can be run through the Web A license is required for full use of FabricIT The standard management module without a license allows for managing the switch system using the Web interface or the command line interface and software supplied on the management chip Make sure to register your FabricIT license to enable all of the available commands and functions See the FabricIT EFM User Manual for instructions and commands available to manage the switches and fabric 8 1 Downloading FabriciT Software and Documents To download FabricIT software and documentation please visit the following We
53. s Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit gt Overcurrent Protection A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 18 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not dis posed of with regular household waste Dispose of this product and all of its parts in a responsible and environ mentally friendly way 14 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 2 2 LED Status Indicators 2 2 1 Power Supply Unit LEDs Each Power Supply Unit has three indicator LEDs Figure 3 Power Supply Unit Status Indications OK FAIL OK AC When lit this LED indicates input voltage between 100 and 240 Volts DC FAIL When lit this LED indicates a fault in the power supply DC OK When lit this LED indicates that the output from the power supply is 48 VDC 2 2 2 Leaf Board LED Indicators Each leaf board has two Status LEDs Green and Yellow and 36 Connector LEDs two per con nector When the Green board status LED labelled PWR is on this indicates that there is power to the leaf board If the Green LED is off uninstall and reinstall the leaf board
54. shows the front and rear sides fully populated 10 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Figure 2 Switch Views Spine Fan Module with four internal fans _ Leafs Each leaf has 18 Ports and up to 18 leafs can fit in the chassis Two management Modules Up to 8 Power Supply Units 6 Required and 2 for redundancy to allow hotswapping The chassis needs all nine spines for the switch system to run properly Four spines to the right of the fan module and five spines to the left of the fan mod ule The leaf fan module contains 7 internal fans Mellanox Technologies 11 Rev 1 7 Safety Warnings and LED Indications 2 Safety Warnings and LED Indications 2 1 Installation Safety Warnings 1 Installation Instructions Read all installation instructions before connecting the equipment to the N power source 2 Bodily Injury Due to Weight This equipment is very heavy and should be moved using a mechanical lift to avoid injuries gt 3 Over temperature This equipment should not be operated in an area with an ambient tem perature exceeding the maximum recommended 45 C 113 F More over to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings A 4 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury
55. t Parts Ordering Numbers Part Description OPN Power supply unit PSU MTP000037 Power supply blank MTM001343 Leaf board unit for QDR Switch system MTS3611QC Leaf board unit for DDR Switch system MTS3611DC Leaf Thermal blank MTM001214 Spine board unit for QDR Switch system MTS3610QSC Spine board unit for DDR Switch system MTS3610DSC Management module without CPU MTS3610U CPU management module with PPC MTS3610MDC Management module blank MTM003610 Rack installation kit MTR003610 Fan unit for leafs MTF001234 Fan unit for spines MTF001254 DB to RJ45 Harness HAR000028 Power cord Type B for USA Canada Mexico Taiwan 00204 Power cord Type B right angle chord for USA Can ACC000241 ada Mexico Taiwan Power cord 250v 15a 2 0M C14 to C13 ACCO00242 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany ACC000207 Netherlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 64 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Part Description OPN Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 Mellanox Technologies 65 Rev 1 7 66 Mellanox Technologies 19U QSFP SDR DDR QDR Infini
56. t software Package Contents The package includes 1 Chassis 1 18 leafs according to the amount ordered 18 X leaf blanks X the amount ordered 1 leaf fan module 1 spine fan module 9 spines 1 2 management modules according to the amount ordered If only 1 management module is ordered then one management module blank will be in the package 1 power cable for each PSU Type B 13ft US 125V 15 Right angle chord 6 8 PSUs depending upon your specific order 8 X PSU blanks X the amount ordered RJ45 to DB9 cable Installation kit CD nstallation guide Before you install your new MTS3610 Chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping Note If anything is damaged or missing contact your customer representative immediately Physical Installation A Warning This equipment is very heavy Safety is the first concern The fully loaded chassis weighs 350 lbs 155 kg Make sure that proper manpower and equipment is used for transporting and moving the chassis Note You will need a mechanical lift to move and insert the chassis into the rack 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 This chassis can be installed in 19 racks with a maximum depth of 81cm The switch platform uses 20U of rack space i
57. tallation hot swapping components and hardware maintenance is covered in Installa tion on page 20 1 1 Serial Number and Product Version Information The serial number and product version information are found on the label seen in the figure below Figure 1 Generic Product label S N MTYYWWFSSSSS DIN Rev XX PIN MTS36000 1UNC DIL GUID HHHHHHHHHHHHHHHH INN Made In IL Also on this label is the GUID identifier for the switch This is the System Image GUID according to the IB spec It is burned on the board which is in the chassis the boards and the management SW looks for this GUID in addition to their own Node GUID Mellanox Technologies 9 Rev 1 7 Overview Table 3 Switch Rack Mechanical and Environmental Requirements Fully Populated Chassis eler Rack Width Sun Weight Max Power 324 x QSFP Ports 190 19 EIA 25 7 105 68 kg min configu DDR 4765 W 220VAC Max 45 C 842 mm 310 652 8mm ration QDR 5170 W 220VAC Min 10 C 482 6 mm 233 Ibs min configura Up to eight Power Supplies of 100 tion 240VAC 50 60Hz 13 6 3A 153 kg full configura tion 347 Ibs full configura tion The switch ships in a minimum base configuration plus additional modules depending on the cho sen customer configuration Optional modules include Leaf boards Power supply units Management modules Figure 2
58. the world Table 6 HCA Cards Certification Status FCC EN ICES P N Class Class Class cTUVus CB USA Europe Canada MTS3610D A A A A MTS3610Q A A A A Table 7 Switch Specification Sheet MTS3610D and MTS3610Q Physical Power and Environmental HxWxD Weight Mounting leafs spines SerDes Speeds Connector Types Max Air Flow Through Max Air Flow Through 33 15 x 19 x 25 7 inches 842 mm x 482 6mm x 652 8mm 105 68 kg minimum configuration 233 Ibs minimum configuration 153 kg full configuration 337 Ibs full configuration 19 Rack mount EIA 310 16 59M3 min 585 9 CFM 9 48 M3 min 334 8 CFM 20 and 40 Gb s per port QSFP Input Voltage Temperature Humidity MTS3610D Power Consump tion Dissipated power Power through QSFP MTS3610Q Power Consump tion Dissipated power Power through 90 264 VAC 50 60Hz 13A 100V 6 3A 240V 10 to 45 Celsius 10 90 non condensing 4765W 220VAC fully pop ulated 3415W 1350W 3 5W available per port 5170W 220VAC fully pop ulated Dissipated power 3820W 1350W 3 5W available per port Protocol Support Regulatory Compliance 50 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Table 7 Switch Specification Sheet MTS3610D and MTS3610Q InfiniBand Auto Neg
59. to close them completely 6 Screw in the locking screws on each latch 36 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 5 5 Fan Modules There are two fan modules on the switch one leaf fan module and one spine fan module The leaf fan module of the switch platform is located in the center of the front and includes seven fans See Figure 24 on page 38 The spine fan module is located on the rear side panel at the top of the switch system Each spine fan module includes four fans Operation without either fan unit should exceed one minute Caution If there is a possibility that either fan module will be out of the chassis for more than one minute shut down the switch before removing the fan module When a fan module is removed the indicator light will reset 5 5 1 Leaf Fan Module The leaf fan module contains seven individual fans Should a single fan fail the LED on the man agement module will light indicating the necessity to replace the fan module The air flow through the module is outside air is brought through the fan module pushed through the leafs pushed out of the holes in the front of the leafs Air flow through the leafs is independent of the air flow through the spines 5 5 1 1 Extracting the Leaf Fan Module 1 Unscrew the four captive screws that hold the leaf fan module to the chassis See Figure 24 2 Slowly pull out the fan module while holding
60. ule Yellow LED for the Fan Drawer Leaf is Lit 1 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 5 on page 37 Caution Do not run the switch without the fan module for more than one min A ute Immediately replace the Leaf fan module Note Should any of the boards shut down due to over temperature follow the procedure start ing in Step 1 Yellow LED for the Fan Drawer Spine is Lit 1 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes 2 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 3 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 5 on page 37 utes Immediately replace the spine fan module A Caution Do not run the switch without the fan module for more than two min Spine Board The ye
61. ule 48 9 4 Spine Board 48 9 5 Management Module 49 Appendix A 19U QSFP DDR QDR IB Switch Specification Data 50 Appendix B Mechanical Drawing 53 Appendix C QSFP Interface 55 Appendix D Ethernet RJ 45 Connector 57 Appendix E Sample In band FW Upgrade 58 Appendix F Replacement Parts Ordering Numbers 64 Appendix G Avertissements de s curit d installation French 67 Appendix H Installation Sicherheitshinweise German 70 Appendix I Special Regulations Regarding Finland Sweden Denmark and Norway 73 4 Mellanox Technologies 19U QSFP SDR DDR QDR InfiniBand Switch Platform User s Manual Rev 1 7 Table of Figures Figure 1 Generic Product label 9 Figure 2 Switch Views 11 Figure 3 Power Supply Unit Status Indications 15 Figure 4 Leaf Board Status Indications 15 Figure 5 Connector Physical and Logical Link Indications 16 Figure 6 LED Indications When the Leaf is Working Properly 16 Figure 7 Spine Board Status Indications 17 Figure 8 Management Module Status Indications 18 Figure 9 ESD Connector 21 Figure 10 Installation Kit Parts 22 Figure 11 Vertical Rails to Cabinet Measurement 23 Figure 12 Caged Nut Insertion 23 Figure 13 Rail Slide Installation 24 Figure 14 Shelf Installation 24 Figure 15 Mechanical Lifting Device 24 Figure 16 Face Plate Mounting Bolt Locations 25 Figure 17 Lock Down Bar 26 Figure 18 Ground Connection 26 Figure 19 Electrical Cable Clamp 28 Figure 20 Multiple Power Inlets Electric Caution N
62. wer supplies which support up to eight load sharing 1000W supply units The slots for the eight power supply units PSUs are on the front panel and are labelled 1 8 Three are placed horizontally across the top and five are mounted vertically along the right side of the chassis Each PSU has a dedicated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords should be standard 3 wire AC power cords including a safety ground Before starting any procedure on the MTS3610 Switch system put an ESD prevention wrist strap on your wrist and connect to the MTS3610 Chassis Mellanox Technologies 27 Rev 1 7 Installation Powering Up the Switch Platform Note Make sure that the power cords are compatible with your outlets Power cords for differ ent countries can be ordered from Mellanox 1 Plug in the power cord to the PSU 2 Clamp the power cord to the chassis Each PSU has a clamp next to it for supporting the power cord Use this clamp to remove possible strain on the PSU The length of power cord between the clamp and the PSU must be less than 6 15cm See Figure 19 Electrical Cable Clamp 3 Plug the other end of the power cord into a grounded outlet 4 Repeat steps 1 to 3 for the remaining power supply units Note A fully loaded MTS3610 switch system can draw 5 2 kW of power Make

User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents