Home

648-Port EDR InfiniBand Switch-IB™ Switch Platform Hardware

1. 9 InfiniBand 108 Mellanox Technologies Rev 1 1 11 B 12
2. 5 45 113 F 8 3 6 8
3. 18 Bien A RS232 SEAN ZAMAN RARA UL DP 1 DP 2 A LPS ERRE 20 A 19 BA 96 Mellanox Technologies 20 WEEE 5 WEEE 2002 96 EC ERIN BSS T EEE RoR ERE 5 1 55 78 RARI 75 ARIE FE 21 TN E EX tA IT Lo G 5 Avertissements de s curit pour l installation French 1 Instructions d installation Veuillez lire la totalit des instructions d installation avant de relier l quipement au secteur 2 Blessures cause du poids A Pr voyez assez de personnel pour soulever ce produit en toute s curit lt 40 Ibs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 quipement lourd Cet quipement est lourd et do
4. B B 20 19 20 2002 96 WEEE
5. H PuaApOBaHHPIN 15 16 UL u CSA
6. UL CSA 3 16 AWG 1 5 M 6 4 5 Ha 125 13 A lt HAR gt 3 1 0 MM 300 250 10 Mellanox Technologies 109 Rev 1 1 17 K 18 AN RS232 Ethernet DP 1 y DP 2 UL 4
7. 01932 gt 13 14 DINI 15 gt 16 NASI mp3 919 Y N UL CSA 3 16 AWG 13A 125 1 5 6 4 5 235 gt HAR gt 3 DITA 1 0 V 2 300 DY PVC 250 10 92 Mellanox Technologies Rev 1 1
8. 34 Figure 19 Bracket Weld Assembly 35 Figure 20 Placement of Chassis in Rack Top View 35 IEA EME puo sense ee ORTA ARR ee 36 Figure 22 Mechanical rire ES AIR e A 36 Figure 23 Rack and Chassis Alignment 37 Figure 24 Rotated Strip and Bracket Weld 37 Figure 25 Sliding Chassis in Rack 38 Figure 26 Screwing the Chassis to the Rack 39 Figure 27 Screwing the 4 Oval Washer Plates 40 Figure 28 Leaf Side Fixing Bracket With Chassis 40 Figure 29 Spine Side Fixing Bracket With Chassis 41 Figure 30 Leaf Side Fixing Bracket With Shelf 41 Figure 31 Spine Side Fixing Bracket With Shelf 42 Figure 32 Installing Cable Holders 43 Figure 33 Cable Supporter Placement 44 Figure 34 Grounding Posts nase nad ees rasen 45 Figure 35 AC Inl ts eek ie er e uva edet e dut 45 Mellanox Technologies 7 Rev 1 0 Figure 36 Power Unit AC Inlet Numbering
9. 9 NERNERER LA DE ERL NE Be C FAR 10 InfiniBand HAE RK IN LN InfiniBand 584848 NANE L K ER MRZE MES 12 ais fe ER ARES Mellanox Technologies 95 14 15 UL IZA CSA HEEE RAR PEAS UL CSA 29 8 82888 16 AWG 125 15 4 5 lt HAR gt FR 182888 1 0 300 KA PVC BERR ER 250 V 10 16 A CAATHREBE 17
10. G 9 Avertismente privind siguran a la instalare Romanian 1 Instruc iuni de instalare Cititi toate instruc iunile de instalare nainte de a conecta 110 Mellanox Technologies Rev 1 1 2 Accidentare cauzat de greutate Apelati la un num r suficient de persoane pentru a ridica n sigurant acest produs e i 40 Ibs 40 70 Ibs 70 121 lbs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Echipament greu Acest echipament este greu si trebuie s fie mutat folosind un dispozitiv mecanic de ridicare pentru a evita producerea de leziuni 4 Risc de soc electric Risc de soc electric Odat ce modulul ventilator este indep rtat pinii electrici sunt accesibili in cavitatea modulului NU introduceti instrumente sau p rti din corp in cavitatea modulului ventilator 5 Temperatura in exces Acest echipament nu trebuie s fie actionat intr o zon unde temperatura ambiant dep seste valoarea maxim recomandat 45 C 113 F In plus pentru a asigura un flux de aer adecvat l sati un spatiu liber de cel putin 8 cm 3 inchi in jurul fantelor de ventilare 6 Suprapunerea cadrului A N Cadrul nu trebuie s fie suprapus peste niciun alt echipament In cazul in care cadrul cade poate cauza leziuni corporale si deterior ri ale echipamentului 7 Conexiunea la o surs de alimentare electric suplimentar pericol ele
11. Mellanox Technologies 85 Table 22 Replacement Parts Ordering Numbers Rev 1 0 Appendix F Replacement Parts Ordering Numbers Part Description OPN 2 5KW AC Power Supply w P2C air flow MTDF PS A Switch IBTM EDR InfiniBand leaf blade 36 QSFP28 ports RoHS R6 MSB7510 E MCS75xx director system Leaf Blank MTDF BLK C Switch IBTM EDR InfiniBand spine blade 36 ports RoHS R6 MSB7520 E MCS75xx director system Spine Blank MTDF BLK A x86 dual core chassis management module RoHS R6 MMB7500 MCS75xx director system Management Blank MTDF BLK B MCS75xx 648 port director system rail kit MTDF KIT A MCS75xx 216 and 324 port director system rail kit MTFD KIT B MCS75xx director system cable holder MTDF CH A MCS75xx director system cable supporter MTDF CP A Director system fan unit located on the spine side MTDF FAN A Director system fan unit located on the spine fan unit MTDF FAN B DB to RJ45 Harness HAR000028 Power cord black 250V 16A 2000MM C19 to C20 VDE ACC000734 86 Mellanox Technologies Rev 1 1 Appendix G Safety Warnings Multiple Languages G 1 Nordic Countries Notices Finland Laite on liitett v suojakoskettimilla varustettuun pistorasiaan Norway Apparatet m tilkoples jordet stikkontakt Sweden Apparaten skall anslutas till jordat uttag G 2 Installation Safety Warnings English 1 Installati
12. 47 Ligure 37 Power Supply Unit ou e RARO RE anna al CEA NES 48 Figure 38 Releasing Leaf Ejector Handles ees 49 Figure 39 Intact Leaf Signal Connectors 49 Figure 40 Intact Leaf Power Pins 0 00 50 Figure 41 Management Module Numbering 51 Figure 42 Releasing Spine Ejector Handles 52 Figure 43 Intact Spine Signal Connectors 52 Figure 44 Intact Spine Power Pin Holders 53 Figure 45 Leaf Fan Locations on the Chassis 54 Figure 46 Leaf Fan Module Extraction 54 Figure47 Spine Fan Modules 4 ee e e ene sn hr eme e bh s 55 Figure 48 Fan Status LED on the Spine Module 55 Figure 49 Releasing Management Ejector Handles 57 Figure 50 Intact Signal Connectors 57 Figure 51 Intact Management Power Pins 57 Figure 52 Power Supply Unit Status Indications 59 Fig re 33 60 7900 PSU Cover mes eee ttes tete ed dame te ss 60 Figure 54 Leaf Modu
13. 17 29 gt pa 18 MINI Ethernet RS232 UL DP 1 non DP 2 mo 19 DYSON NIN 20A 20 PN PN Now 2 21 WEEE 29 WEEE 2002 96 EC DINA D lt
14. 3 DAV 7 PY PN qn 4 bawn 8 MI PINI IN PN 4 9 MID 4 TON MIN MIN pin DITO nio 10 4 PN Mellanox Technologies 91 Rev 1 1 11 InfiniBand on gt v PINI 12
15. Rev 1 0 Appendix B Thermal Threshold Definitions There are three thermal threshold definitions for the Switch IBTM switch device which impact the overall switch system operation state Warning Critical and Emergency l Warning 100 C On managed systems only When the Switch IBTM device crosses the 100 C threshold a Warning Threshold message will be issued by the MLNX OS management SW indicating to system admin istration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any action by hardware such as switch shutdown Critical 120 C When the Switch IBTM device crosses this temperature the firmware will automatically shut down the device Emergency 130 C In case the firmware fails to shut down the Switch IBTM device upon crossing the Critical threshold the Switch IBTM device will auto shutdown upon crossing the Emergency 130 C threshold 80 Mellanox Technologies Rev 1 0 Appendix C Calculating the Weight of a Customized Chassis The weight of a customized chassis can be calculated for any possible customization as follows Sum up the following weights Spines Leafs blanks Management modules Management blanks P Chassis weight already includes pre installed 20 fan FRUs and 10 PS units Rail kit cable supporters and cable management kits weight is not included and may be added separately i
16. 82 Appendix E 83 Appendix Replacement Parts Ordering Numbers 86 Appendix Safety Warnings Multiple Languages 87 G 1 Nordic Countries Notices 87 G 2 Installation Safety Warnings English 87 G3 nus 90 G 4 Chinese A Eo selinin 94 G 5 Avertissements de s curit pour l installation French 97 G 6 Installation Sicherheitshinweise German 100 G 7 Advertencias de seguridad de instalaci n Spanish 104 G 8 Russian 107 G 9 Avertismente privind siguran a la instalare Romanian 110 G 10 Sigurnosna upozorenja za instaliranje Croatian 114 G 11 Avvertenze di sicurezza per l installazione italiano 117 4 Mellanox Technologies Rev 1 0 G 12 Montaj G venlik Uyar lar T rk e 121 G 13 Japan VCCI Class A Statement 124 Mellanox Technologies 5 Rev 1 0 List of Tables Table 1 User Manual Revision History i 9 Table 2 Reference Documents and Websites 10 Table
17. Spine fan modules Management modules 2 9 1 Power Supply Units The power supplies deliver 2500W at 48V pc The input to each of these power supplies requires 12A current in order to output 2500W For N N configuration up to half ofthe PSUs can go down and the system will continue to run The power required to run the switch system is equally divided between all ofthe working PSUs 2 9 1 1 Extracting and Inserting the PSU With all ofthe N N 2500W power supplies installed the system is in N N redundant configura tion half of the PSUs may be extracted without bringing down the system gt To extract a PSU 1 Determine which AC connector on the connector side of the chassis corresponds to the defec tive PSU 2 Remove the power cord from the PSU s inlet Note which power cord it is according to the AC numbering Figure 36 Power Unit AC Inlet Numbering 3 On the spine side of the chassis remove the cover to the PSUs There are four Phillips head screws for the cover plate Mellanox Technologies 47 Rev 1 0 Installation Figure 37 Power Supply Unit 2 9 2 2 9 2 1 NEN gt B Handle Latch Grasping the handle with one hand push the black latch release while pulling the handle out ward As the PSU unseats the PSU status indicators will turn off Remove the PSU To insert a power supply unit Make sure the mating connector of the new unit is free of any dirt and or obstacles
18. A Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der A maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem SS sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 6 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunt erf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 7 Mehrere Stromeing nge Risiko eines Stromschlags und Stomgefahr Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgeschalteten Zustand im Inneren der Switch Plattform sicherzustellen Mellanox Technologies 101 Rev 1 1 8 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 9 Anschlie en Trennen von InfiniBand Kupferkabel N InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen y Warnungen und Anleitungen des Kabelherstellers 10 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewart
19. Off No power to the spine Green Solid Spine is up and running Green Flashing Spine is powering up or being restored to factory default Amber Solid Fatal error 3 1 4 2 Fan LED For each fan drawer in the spine there is a status LED The spine fan indicator LED has the fol lowing LED assignment Table 11 Spine Fan Status LED LED Color Description Green Solid Spine fan is OK Amber Solid One of the fans is not working and needs replacement 3 1 4 3 Spine to Leaf IB Connection Status LEDs The leaf connection status on each spine displays the condition of the connection between the spine and each leaf There is a minimum of one LED per leaf per spine and a maximum of 1 LED per leaf These LEDs indicate a valid connection between a leaf and a spine Table 12 shows the leaf to spine status according to the LED condition Table 12 Spine to Leaf IB Link Status LED Color Description Off Link is down Green Solid Logical connection Green Flashing Data activity Amber Solid Physical connection established Mellanox Technologies 63 Rev 1 0 Interfaces 3 1 4 4 Bad Port LED The Bad Port LED is located on the left side of the spine The following Bad Port LED condi tions are possible Table 13 Bad Port LED Configurations LED Condition Description Off OK No ports have received
20. switch master config no power enable mgmt2 2 Disconnect all cables connected to the management module 3 Push the ejector handles in the direction of the red arrows in Figure 49 and pull them out wards to unlock the ejectors from the chassis 56 Mellanox Technologies Rev 1 0 Figure 49 Releasing Management Ejector Handles Push in this direction Push in this direction ET vo doa Open the ejectors until they 45 degrees from the module Pull out the module half way through the guiding rails using the ejector handle Lock the ejector handle E Hold the body of the module on both sides and remove it from the chassis A The module is short therefore do not let go of it while sliding it out Inserting a Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a management module 1 Check for foreign objects in or mechanical damage to the chassis management slots 2 Check management mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any devia tions from the pass criterion shown in Figure 50 Figure 50 Intact Signal Connectors 4 Check power connector s integrity Look f
21. Mellanox TECHNOLOI Connect Accelerate Outperform 648 Port EDR InfiniBand Switch IB Switch Platform Hardware User Manual PN MCS7500 Rev 1 0 www mellanox com Rev 1 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUME
22. l emplacement final pas pour l entretien ni la maintenance 20 Directive DEEE Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doivent tre collect s s par ment et ne pas tre mis au rebut avec les d chets m nagers habituels Ce produit et toutes ses pi ces doivent tre mis au rebut d une mani re responsable respectant l environnement pem 21 Restrictions concernant l alimentation pour la Norv ge Cet appareil est pr vu pour tre reli un systeme d alimentation TN et un systeme d alimentation informatique de Norv ge uniquement G 6 Installation Sicherheitshinweise German 1 Installationsanleitungen N Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung N anschlie en 100 Mellanox Technologies Rev 1 1 2 Verletzungsgefahr wegen des Gewichts da Um das Produkt sicher anzuheben gen gend Personen einsetzen Mal 40 lbs 40 70 Ibs 70 121 lbs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Schweres Ger t Dieses Ger t ist schwer und muss mit einem mechanischen Hebeger t verschoben werden um Verletzungen zu vermeiden 4 Stromschlagrisiko Stromschlagrisiko Bei abgenommenem Ventilatormodul sind die Stromkontakte in der Modulvertiefung zug nglich Es d rfen KEINE Werkzeuge oder K rperteile in die Vertiefung des Ventilatormoduls gelangen 5 bertemperatur
23. HH HH HH HH HH HH HH HH Rev i Made in COO 12 Mellanox Technologies Rev 1 0 The GUID is the System Image GUID according to the IB spec It is burned on the board which is in the chassis All the boards and the management software look for this GUID in addition to their own Node GUID Figure 2 Product Label Tab Location The product label tab is located here on the spine side Mellanox 1 1 2 Management Module MAC Each management module has a label with its MAC address See Figure 3 for the location of this label Figure 3 Management Module MAC Address Location 1 1 3 Product Physical Specifications and Power The switch itself is 28U tall but requires an extra 1U to install the shelf The switch ships in a minimum base configuration plus additional modules depending on the chosen customer configuration Optional modules included e Management modules Spine modules Leaf modules Mellanox Technologies 13 Rev 1 0 Overview The following figure shows the leaf and spine sides fully populated Figure 4 CS7500 AC inlets Fuse box Do not tamper with leafs can fit in the chassis 4 Power supplies Pa Chassis fan modules Spines AC inlets 10 BEE Power supply units 10 BEE Management modules 2 BEE Spine modules 8 BEE Leaf modules 18 Chassis fan modules 20 14 Mellanox Technologies Rev 1 0 1 2 Fe
24. Insert the PSU by sliding it into the opening until a slight resistance is felt Continue pressing the PSU until it seats completely The latch will snap into place confirming the proper installation Insert the power cord into the supply connector on the other side of the chassis Replace the cover over the PSUs A The green indicators should light If not extract the PSU and re insert it again Leaf Modules When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions Leaf module slots are numbered from top to bottom from L1 to L18 on both sides of the chassis Extracting a Leaf Module Each leaf module has 2 ejector handles that locks the module in place and serve as a lever for seating or extracting see Figure 38 1 Run the shut down command no power enable module For example to shut down leaf 16 run the command below switch master config no power enable L16 Disconnect all cables connected to the leaf Push the ejector handles in the direction of the red arrows in Figure 38 and pull them out wards to unlock the ejectors from the chassis 48 Mellanox Technologies Rev 1 0 Figure 38 Releasing Leaf Ejector Handles Push in this direction Push in this direction Open the ejectors un
25. Rev 1 0 12 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully A attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 13 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken y to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 14 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 15 Equipment Disposal A Disposal of this equipment should be in accordance to all national laws and regula tions 16 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical A codes 17 Installation Codes This device must be installed according to the latest version of the country national A electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 18 Battery Replacement Warning Replace only with UL Recognized battery certified for maximum abnormal charging current not less than There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispos
26. con guaina isolante in PVC Il cavo deve disporre di una spina stampata di potenza nominale pari a 250 V 10 A 17 Corrente di dispersione elevata Avvertenza corrente di dispersione elevata il collegamento a terra essenziale prima di collegare l alimentazione Mellanox Technologies 119 Rev 1 1 18 Interconnessione delle unit A I cavi per il collegamento all unit RS232 e alle interfacce Ethernet devono 4 disporre della certificazione UL ed essere del tipo DP 1 o DP 2 Nota in caso 4 y di installazione su un circuito la cui potenza non limitata Protezione contro le sovracorrenti la cablatura dell edificio deve integrare un dispositivo di protezione contro le sovracorrenti di potenza nominale pari a 20 19 Non utilizzare lo switch come scaffale o piano di lavorol Attenzione un apparecchiatura scorrevole o montata su binari non va utilizzata come scaffale o piano di lavoro I binari non sono progettati per far scorrere e allontanare l unit dal rack Essi sono destinati all installazione permanente solo nel luogo di lavoro e non vengono utilizzati per assistenza e manutenzione 20 Direttiva RAEE Secondo la direttiva RAEE 2002 96 EC tutti i rifiuti da apparecchiature elettriche ed elettroniche RAEE vanno raccolti separatamente e non smaltiti nei normali rifiuti domestici Smaltire questo prodotto e tutte le sue parti in modo responsabile e rispettoso dell ambiente LA 21 Limitazioni rela
27. 15 seconds the management module is reset and the password deleted Do not use a sharp pointed object such as needle or push pin for pressing the Reset but ton Sharp objects can cause damage use a flat object such as a paper clip This button resets the CPU of the management module A quick push of this button performs this reset When the reset button is pushed on the master management module this management mod ule is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin 68 Mellanox Technologies Rev 1 0 4 Chassis Power Up Before starting any procedure on the CS7500 system put an ESD prevention wrist strap on your wrist and connect to the chassis With N N PSU redundancy the chassis must be started with a full complement of possible PSUs thereafter it can run on half of the total number of PSUs Connecting the PSUs to different AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs I
28. 3 Chassis Installation Kit Items 11 Table 4 Power BuppYOPN 2 eos eae eee eee S 16 Tables Chassis Installation Kit Items u 2 pere an li 26 Table 6 Cable Management Installation Kit Parts 28 Table 7 Leaf Status LED er cs e sais E E SRA 61 Table 8 Bad Port LED 61 Table 9 Connector Physical and Logical Link Indications 61 Table 10 Spine Status LED 23324 e a ta ee 63 Table 11 Spine Fan Status 63 Table 12 Spine to Leaf IB Link Status isolare a as 63 Table 13 Bad Port LED Configurations sesl v ne nee 64 Table 14 Spine Side Panel LED Display for Normal Operation 64 Table 15 Management LED Display for Normal Operation 66 Table 16 Switch Specification CS7500 77 Table 17 Switch System Weight Calculation 81 Table 18 Power Consumption of Chassis Parts 82 Table 19 QSFP Cable Power Consumption 82 Table 20 Calculating Total Switch System Power Consumption 82 Table 21 InfiniBand Connector Pin out 83 Table 22 Replacement Parts Ordering Numbers 86 6 Mellanox Technol
29. About this Manual px N ER 0 Chapter T OyervieW essc se annee Ras nd ca a p tao TZ 1 1 Product Information sure ime aan nenn a ta sacs 12 1 1 1 Serial Number and Product Version 12 1 1 2 Management Module MAC 13 1 1 3 Product Physical Specifications and Power 13 1 2 Features Li Stas ia i 0 15 1 3 InfiniBand EDR Overview 15 14 Power Supply Redundancy 15 1 4 1 AC Source Requirements 15 1 42 Power Redundancy ritor A ki 15 Chapter 2 17 2 1 Installation Safety Warnings 17 2 2 Environmental and Safety Recommendations 21 2 3 Switch Components 22 24 Unpacking the Switch 22 2 4 1 Verifying Crate Condition 22 2 4 2 Opening the Chassis Crate 23 2 5 Physical Installation oc en neh 24 2 5 1 Installation Kit Parts 26 2 5 2 Installation Procedure aa an en nu 30 2 6 Ground Connections i iyi 44 2 7 Power C
30. FLAT WASHER SPRING WASHER HEX SCREW 10 32 3 28 3 58 Nm BRACKET WELD FIXING BRACKET Step 24 Screw the fixing bracket on the leaf side to the shelf at 9 Nm torque Figure 30 Figure 30 Leaf Side Fixing Bracket With Shelf HEX SCREW M6 SPRING WASHER M6 FLAT WASHER M6 Mellanox Technologies 41 Rev 1 0 Installation Step 25 Screw the fixing bracket on the spine side to the shelf at 9 Nm torque Figure 31 Figure 31 Spine Side Fixing Bracket With Shelf HEX SCREW M6 SPRING WASHER M6 FLAT WASHER M6 2 Mellanox Technologies Rev 1 0 Step 26 Screw the cable holders to the leaf side rack using 14 M6 pan head screws at 9 Nm torque Figure 32 Figure 32 Installing Cable Holders BLANK COVERS lal m i 1 p gt n 1 te AAW m A w Ba pm mY iw iw I a AN 1 CT SCREW PATCH M6 D gm MA 2 RIGHT QSFP HOLDER SCREW PATCH M6 LEFT QSFP HOLDER Mellanox Technologies 43 Rev 1 0 Installation Step 27 Place the 18 cable supporters onto the cable holders Figure 33 Figure 33 Cable Supporter Placement 2 6 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge The grounding post is M 8 with 1 25mm pitch
31. Installation 2 3 2 4 2 4 1 Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit Switch Components The system s components are shipped as specified below Chassis Crate The switch system chassis is shipped in a dedicated crate which includes the following boxes 1 CS7500 chassis 2 Chassis installation kit see Table 5 Chassis Installation Kit Items on page 26 3 Cable holders see Table 6 Cable Management Installation Kit Parts on page 28 4 6 boxes containing cable shelves and 1 box containing 10 power cords 250V 16A 2m C19 to C20 USA UL Standard The management spine and leaf modules are shipped separately To completely populate the CS7500 chassis one requires 2 management module d boxes 18 spine module boxes and 18 leaf module boxes Spine Module Boxes Each box contains 1 spine module with its fan modules already installed This unit is intended for installation in a restricted access location A restricted access area can be accessed only through the use of a special tool lock and key or other ae means of security Leaf Module Boxes Each box contains 1 leaf module Management Module Boxes Each box contains 1 management module and 1 RJ45 to DB9 cable Unpacking the Switch System Verifying Crate Condition The create has a shock sticker applied When receiving the create look
32. N flusso d aria adeguato lasciare almeno 8 cm 3 pollici di spazio attorno alle aperture di ventilazione 6 Impilare lo chassis Kuciste se ne bi trebalo slagati na drugu opremu Ako kuciste padne moze izazvati tje lesne ozljede i o te enje opreme 7 Collegamento di alimentazione ridondante Pericoli elettrici installato di uno spazio vuoto Qualora l alimentatore non sia installato non utilizzare l Questo prodotto dotato di un alimentatore ridondante o qualora esso non sia il prodotto con il coperchio rimosso o non fissato correttamente 8 Prese di alimentazione multiple Rischio e pericolo di scosse elettriche i Gli alimentatori sono tutti indipendenti Scollegare tutti gli alimentatori per assicurarsi che il commutatore non sia sotto tensione 9 Durante i temporali pericolo di scosse elettriche Durante i temporali non effettuare interventi sull apparecchiatura e non collegare scollegare i cavi 10 Collegamento scollegamento del cavo di rame InfiniBand I cavi di rame InfiniBand sono pesanti e non flessibili Di conseguenza vanno collegati o scollegati con cura dai connettori Per avvertenze istruzioni speciali rivolgersi al produttore di cavi 118 Mellanox Technologies Rev 1 1 11 Montaggio su rack e manutenzione Quando questo prodotto viene montato sottoposto manutenzione su un rack Wm necessario adottare delle precauzioni speciali per assicurarsi che il sis
33. S rg l rayl ekipman raf veya al ma alan olarak kullan lamaz Raylar niteyi iskeleden uza a kayd rmak i in yap lmam t r Sadece ekipman n son olarak duraca yerdeki kal c montaj i indir servis veya bak m i in kullan lamaz Mellanox Technologies 123 Rev 1 1 20 WEEE Y nergesi WEEE Y nergesi 2002 96 EC uyar nca t m elektrikli ve elektronik ekipman at klar EEE ayr olarak toplanmali ve evsel at klarla birlikte pe at lmamal d r X Bu r n ve t m par alar evreye dost ve sorumlu bir ekilde imha edilmelidir 21 Norve G K s tlamalar Bu nite bir TN g sistemine ve sadece Norve in TT g sistemine ba lanmak i indir 3 G 13 Japan VCCI Class A Statement CORBIS 772 C COREE XRRR CHATS FC 02 6 ES DIE RR 5 416 LUK VCCI A 124 Mellanox Technologies
34. and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Remove and reinstall the spine module Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 2 9 3 on page 50 5 Make sure that the spine and the Leafs both have the same version of FW 6 Reinstall the FW and remove and reinstall the spine 7 Replace the spine module 6 5 MLNX OS Software Issue 1 The last software update did not succeed 1 Connect the RS232 console connector to a laptop 2 Push the reset button on the switch or management module 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 mlnx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No 2 0 ready DRAM 2 GB ECC enabled 400 MHz CL3 FLASH 16 MB NAND 1024 MiB Bus Dev Venid Devid Class Int PCIEO
35. cargo nica A mente de personal capacitado y competente La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y N reglamentaciones nacionales Mellanox Technologies 105 Rev 1 1 14 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio 4 males NN 15 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 16 Cable de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 17 Alta corriente de fuga ADVERTENCIA Alta c
36. down bars at this time 2 5 Physical Installation The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This ensures proper airflow through the chassis This 1s crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis Warning This equipment is very heavy 257kg 5671b Safety is the first concern y N Make sure that adequate manpower and proper equipment is used for transporting and moving the chassis 24 Mellanox Technologies Rev 1 0 This chassis can be installed in standard 19 racks that have 715 850mm between the vertical supports of the rack Figure 7 Distance Between the Vertical Supports The vertical supports may have to be moved so that the vertical support closest to the door is a minimum of 6cm from the inside of the door This is required so that the shelf does not hit the door If you move the vertical supports make sure that you reinstall them vertically plumb The switch platform uses 29U ofrack space 28U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chassis and there are two lock down bars to secure the chassis to the shelf The weight
37. eine angespritztem Stecker bewertet bei 250 V 10 A 16 Hoher Ableitstrom WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 17 Installationscodes Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimmung mit den gel tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 18 Verbindung der Ger te untereinander Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 19 Switch nicht als Regal oder Arbeitsplatz nutzen Achtung Auf Schieber Schienen montiertes Ger t ist nicht als Regal oder Arbeitsbere ich zu nutzen Die Schienen sind nicht daf r bestimmt die Einheit aus dem Gestell weg zu ziehen Sie sind nur f r die permanente Installation an einem endg ltigen Stan dort gedacht nicht f r Instandhaltung und Wartung 20 WEEE Direktive Gem WEEE Directive 2002 96 EC m ssen alle elektrischen und elektronischen Abfallger te EEE separat gesammelt und nichit mit normalem Haushaltsm ll ensorgt werden Dieses Produkt und alle seine Teile in ve
38. for and inspect the shock sticker to confirm that it has not triggered IMPORTANT NOTE The shock sticker does not necessarily indicate damage to the contents However be sure to carefully inspect the contents of a crate with a triggered sticker 22 Mellanox Technologies Rev 1 0 The shock stickers turn red if the create has been mishandled or handled roughly If one of the shock stickers is red notify the shipper and Mellanox Figure 5 Shock Stickers Unaffected stickers MWARNING pe Triggered stickers 2 4 2 Opening the Chassis Crate 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 2 Unscrew and remove the top cover of the crate 3 Unscrew and remove the sides of the crate It is highly recommended to have a screw gun or electric screwdriver for this step Mellanox Technologies 23 Rev 1 0 Installation Figure 6 Crate Screws 4 Remove the nylon covers from the chassis 5 Visually inspect the chassis make sure that There is no visible damage e 10 PSUs are installed 6 Remove all protective plastic film from the sides and top of the chassis Unpack the system and check against the parts list to make sure that all the parts have been sent Check the parts for visible damage that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately A Do not remove any 01 the blocks or the lock
39. of lightning activity do not work on the equipment or connect or dis connect cables 10 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings instructions 11 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with equip When this product is mounted or serviced in a rack special precautions must be taken A ment starting from the bottom to the top 12 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qual ified personnel 88 Mellanox Technologies Rev 1 1 13 Equipment Disposal N N Disposal of this equipment should be in accordance to all national laws and regula 4 tions 14 Local and National Electrical Codes y N This equipment should be installed in compliance with local and national electrical 4 codes 15 UL Listed and CSA Certified Power Supply Cord A For North American power connection select a power supply cord that is UL Listed N and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m LO For European connection select a power supply cord that is internationally har
40. of the switch is supported from underneath the unit by the shelf Due to the space required by up to 648 connector cables it is recommended to use a rack that is 120cm long and 80cm wide This should allow for proper cable manage ment and enough ventilation to properly cool the chassis Before starting any procedure on the CS7500 switch system 1 Putan ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap Mellanox Technologies 25 Rev 1 0 Installation 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit A You may need a fork lift to move and insert the chassis into the rack 2 5 1 Installation Kit Parts It is recommended to use an AWG6 or 4mm diameter wire for grounding purposes Chassis Installation Kit It is recommended to have at least three people for the duration of the installation pro cedure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation Table 5 Chassis Installation Kit Items Item No a Quantity Description Item No Quantity Description 1 1 Shelf assembly 11 26 M6 spring washer 2 1 Riveted right chassis slide 12 8 M6 flat washer 3 1 Riveted left chassis slide 13 1 Bracket weld 4 2 4 oval washer plate 14 12 10 32 hex head screw 5 8 MS spring
41. ozljede uslijed te ine Kako biste sigurno podignuli ovaj proizvod koristite dovoljan broj ljudi 40 Ibs 40 70 Ibs 70 121 lbs lt lbs 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 Te ka oprema Ova oprema je vrlo te ka i treba se premje tati pomo u mehani kog dizala kako bi se izbjegle ozljede 4 Rizik od strujnog udara Rizik od strujnog udara S uklonjenim modulom ventilatora perima napajanja se mo e pristupiti u otvoru mod ula NEMOJTE umetati alat ili dijelove tijela u otvor modula ventilatora 5 Pregrijavanje y Ovom se opremom ne bi trebalo rukovati u podru jima s temperaturom okoline koja y N premasuje najvise preporu ene vrijednosti 45 C 113 F Osim toga kako bi se osigu 4 rao odgovaraju i protok zraka omogu ite najmanje 8 cm 3 in a razmaka oko otvora ventilatora 6 Slaganje ku i ta Kuciste se ne bi trebalo slagati na drugu opremu Ako kuciste padne moze izazvati tje lesne ozljede i o te enje opreme 114 Mellanox Technologies Rev 1 1 7 Redundantno napajanje Opasnost od elektri ne energije U slu aju praznog prostora za napajanje nemojte rukovati proizvodom ako je pok a Ovaj proizvod uklju uje redundantno napajanje ili prazan prostor na njegovu mjestu N lopac uklonjen ili ako nije dobro pri vr en 8 Vi estruki ulazi za napajanje Rizik od strujnog udara i opasnost od elektri ne energije PSU jedinice su neovisne Od
42. rack Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment The chassis comes only with the power supplies and fans pre installed The rest of the openings are populated with blanks All of the leafs spines and management modules are shipped in a separate package The chassis requires initial configuration to get the chassis and fabric management up and run ning through remote management area can be accessed only through the use of a special tool lock and key or other gt This unit is intended for installation in a restricted access location A restricted access rz means of security 2 1 Installation Safety Warnings These safety warnings are in English For Hebrew Chinese French German Spanish Russian Romanian Croatian Italian or Turkish see the appendixes 1 Installation Instructions A Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight Use enough people to safely lift this product e 40 Ibs 40 70 Ibs 70 121 lbs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Heavy Equipment This equipment is very heavy and should be moved using a mechanical lift to avoid AA injuries Mellanox Technologies 17 Rev 1 0 Installation 4 Installation in Restricted Access Location A This unit is intended for in
43. your fingers during insertion Step 17 Slide in the left and right chassis slides Figure 25 Figure 25 Sliding Chassis in Rack Step 18 Slide the switch system chassis all the way into the rack until it stops A Beware not to injure your fingers during insertion 38 Mellanox Technologies Rev 1 0 Step 19 Screw the chassis to the rack on the spine side using 12 M6 screws where the cage nuts have been placed Figure 26 Figure 26 Screwing the Chassis to the Rack 6 M6 SCREWS ON EACH SIDE 9 Nm 5 eee E g 222 gel one Mellanox Technologies 39 Rev 1 0 Installation Step 20 Screw the 2 4 oval washer plates to the left and right chassis slides Figure 27 Figure 27 Screwing the 4 Oval Washer Plates OVAL WASHER PLATE M5 SPRING WASHER M5 SCREW 3 23 3 58 Nm Step 21 Place the fixing bracket on the shelf from the leaf side and screw using 10 32 6 hex Screws and 6 10 spring washers to the chassis Figure 28 Figure 28 Leaf Side Fixing Bracket With Chassis FLAT WASHER SPRING WASHER HEX SCREW 10 32 3 28 3 58 Nm FIXING BRACKET 40 Mellanox Technologies Rev 1 0 Step 22 Remove the bracket weld on the spine side Figure 29 Step 23 Place the fixing bracket on the shelf from the spine side and screw using 10 32 6 hex screws and 6 10 spring washers to the chassis Figure 29 Figure 29 Spine Side Fixing Bracket With Chassis
44. 00 CFM Airflow SerDes Speeds 10Gb s 20Gb s 40Gb s 56Gb s 100Gb s per port Connector QSFP28 Types Power and Environmental Input Voltage 180 240 VAC 50 60 Hz Typical Passive 10264W Total Power Optical 13159W includes QSFP at 3 5W Consumption EDR Maximum Passive 10635W Optical 13531W includes QSFP at 3 5W Max Heat 46 170 BTUs hr Output Operating 0 to 40 C Pu Non operating 40 to 70 C Humidity Operating 10 85 non condensing Mellanox Technologies 77 Rev 1 0 Table 16 Switch Specification CS7500 Physical Protocol Support Speed Protocol InfiniBand Auto negotiation of 100Gb s 56Gb s 40Gb s 20Gb s and 10Gb s Management MLNX OS and baseboard performance and device management agents for full InfiniBand in band management Data Rate Data Rate 100Gb s per port QoS 8 InfiniBand virtual lanes for all ports Regulatory Compliance Safety US Canada cTUVus certificate International CB report and certificate Europe CE report amp DoC Russia Belarus Kazakhstan CU certificate Argentina S mark certificate Korea KC safety for the power supply only Taiwan BSMI Registration certificate for the power supply only EMC Emissions Europe CE EMC report amp DoC USA FCC part 15 class A Canada ICES 003 class A Australia New Zealand AS NZS CISPR 22 class A amp RCM SDoC Jap
45. 22 TN IT mon gt Mellanox Technologies 9 3 Rev 1 1 G4 Chinese 1 2 0 40 lbs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 4 4 A FRA AKEL ERA 22 E A ERS nn 45 C 113 F BT RER A 8 3 RM WEEE 6 NETS AA WERE EAI MRA WERE BSR 94 Mellanox Technologies Rev 1 1 7 EXEREERW BEER AA RRE A A MA AR N MARES 5 E ABR 8 BER he SH fe 8 LA PSU
46. Check the Management Module LEDs and make sure they coincide with Figure 62 Figure 62 Management Module Status Indications for Normal Operation Master Figure 63 Management Module Status Indications for Normal Operation Slave 70 Mellanox Technologies Rev 1 0 5 Switch Management Tools This chapter describes the management module and tools available for Out of Band management of the switch system via MLNX OS There 4 1Gb s Ethernet ports 2 for each management module that get connected to Ethernet switches which must be configured to 10 100M auto negotiation Ati The CS7500 switch is pre installed with MLNX OS management software MLNX OS is installed on all Switch IBTM based managed switch systems MLNX OS includes a CLI WebUI SNMP and XML interfaces which facilitate configuring chassis management features and InfiniBand management software OpenSM For more information please refer to the MLNX OS User Manual The managed switch system includes the following software components Embedded Subnet Manager SM Chassis manager and system BIST SNMP agent 3rd party tool integration GUI Remote logging SSH telnet Secured access in band and out band IPv4 IPv6 network stack The chassis manager will give the user access to Switch temperatures Power supply voltages Fanunit information Power unit information Flash memory Monitoring of AC power to the PSUs DC
47. EAF SIDE SUPPORT BRACKET 3mm Step3 Place the shelf in U3 of the rack by inserting it at a tilt Figure 14 Figure 14 Placing Shelf 32 Mellanox Technologies Rev 1 0 Step 4 Tighten the spine side M6X25 pan head screws of the brackets to 9Nm torque Figure 15 Tightening Shelf Cage Nut Screws on Spine Side Torque 9 Nm BEAM 2 Step 5 Tighten the leaf side M6X25 pan head screws of the brackets to torque Figure 16 Tightening Shelf Cage Nut Screws on Leaf Side BEAM 3 TORQUE 3 Nm TORQUE 3 Nm TORQUE 3 Nm BEAM 4 Mellanox Technologies 33 Rev 1 0 Installation Step 6 Tighten the hex head screws of the brackets on both sides to 9Nm torque Figure 17 Tightening Shelf Bracket Screws A ol soo y o oe a pp ra D a 10 HEX M6 SCREWS TORQUE 9 Nm Step 7 Tighten the 12 M6X25 pan head screws of the shelf rear arm bracket to its final torque of 9 Nm torque now that the shelf has been placed Figure 18 Tighten Shelf Back Support Screws 34 Mellanox Technologies Rev 1 0 Step 8 Assemble the bracket weld to the shelf on the spine side and tighten with 2 screws 2 spring washers and 2 flat washers Figure 19 Figure 19 Bracket Weld Assembly 8 Tt EG 89 6 SCREW 5 Nm SPRING WASHER FLAT WASHER Step 9 Before placing the chassis in the rack make sure to leave more room on the leaf port cable
48. Module When a slave management module is not installed or not working hot swapping the spine module connected to the master management module will cause the chassis to crash gt Neither the CLI nor the GUI management tools will allow you to shut down spine 1 or spine 2 as the management modules are connected to the chassis components through these spines Extracting Spine Module 1 2 Spine module 1 is connected to management module 1 and spine module 2 is connected to management module 2 Figure 41 Management Module Numbering Removing spine 1 causes management module 1 to reset Removing spine 2 causes management module 2 to reset Warning If the spine you want to hot swap is connected to the master management A module the management module will reboot when you take out the spine If you have only one management module the chassis will crash If you need to hot swap spine 1 or spine 2 1 Check to see if the spine you need to remove is connected to the master management module See Figure 41 2 Follow the steps in the section below Extracting Spine Modules Except 1 or 2 1 Run the shut down command no power enable lt module gt For example to shut down spine 06 run the command below switch master config no power enable S06 2 Push the ejector handles in the direction of the red arrows in Figure 42 and pull them out wards to unlock the ejectors from the chassis Mel
49. NTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Hakidma 26 Sunnyvale CA 94085 Ofer Industrial Park U S A Yokneam 2069200 www mellanox com Israel Tel 408 970 3400 www mellanox com Fax 408 970 3403 Tel 972 0 74 723 7200 Fax 972 0 4 959 3245 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct GPUDirect InfiniBridge InfiniHost InfiniScale Kotura Kotura logo Mellanox Connect Accelerate Outperform logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroX MetroDX MLNX OS Open Ethernet logo PhyX ScalableHPC SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd CyPUTM ExtendXTM FabricIT FPGADirectTM HPC X Mellanox Care Mellanox CloudX Mellanox NEO Mellanox Open Ethernet Mellanox PeerDirect NVMeDirect StPUTM Switch IBTM Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 4748 Rev 1 0 Table of Contents Revision History yoku y leale een
50. an VCCI Class A Korea KC certificate Taiwan BSMI DoC certificate China CQC certificate for the power supply only Acoustic ISO 7779 ETS 300 753 Scalability and Performance Switching Performance Capacity Switching capacity 130 Tb s Switching performance Simultaneous wire speed any port to any port Addressing 48K unicast addresses max per subnet and 15 5K multicast addresses per subnet Leafs Up to 18 leafs 18 ports each Spines Up to 18 spine boards All spines are needed for full bi sectional configuration Management modules 2 available 1 required for operation Management CPU x86 Power supplies 10 power supplies standard Leaf and management module fans 20 Spine fans 36 78 Mellanox Technologies Rev 1 0 A 1 EMI Certification EMI testing on a fully populated chassis has been performed with the chassis installed in a closed two door rack using the chassis installation kit supplied by Mellanox Technologies A 2 Approved Cables For a list of all approved cables please refer to http www mellanox com page cables mtag cable overview A 3 EMC Certifications The list of EMC certifications per chassis model type for different markets in the world is located on the Mellanox Website at http www mellanox com page environmental_compliance gt Boards Systems gt Boards Systems Certification Matrix Mellanox Technologies 79
51. an FRU handle to insert spine module 5 Holding the spine by its sides carefully set the spine module into the chassis 6 Using only the ejector handles slowly slide the module into the chassis until the hooks reach the vertical bar Do not apply excessive force to slide in the spine module If you feel resistance remove the spine and double check both the chassis and spine for any damage 7 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 8 Lock the ejector handles onto the module 2 9 4 Fan Modules 2 9 4 1 Leaf Fan Module There are 20 fan modules on the chassis for the leaf and management modules They are located on the spine side of the chassis with 10 on the right of the spines and 10 on the left When a fan module is not functioning the status LED on the fan will light up in amber Airflow through the spines is independent of the airflow through the leafs Mellanox Technologies 53 Rev 1 0 Installation Figure 45 Leaf Fan Locations on the Chassis Extracting the Leaf Fan Module 1 Push and hold the blue latch release See Figure 46 2 Slowly pull out the fan module using the handle Figure 46 Leaf Fan Module Extraction Latch Release Handle 54 Mellanox Technologies Rev 1 0 Inserting the Leaf Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new pa
52. an module is problematic by checking the status LED on each fan module 5 Remove and reinstall the problematic fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 9 4 on page 53 6 Replace the Leaf fan module A Replace defective leaf fan modules as soon as they are identified Should any of the modules shut down due to over temperature follow the procedure A starting in Step 2 Issue 3 Amber LED for the spine fan on the management module is lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Determine which spine has a defective fan by checking the Fan LEDs on all of the spines 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 74 Mellanox Technologies Rev 1 0 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels ofthe switch using a vacuum cleaner 5 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 2 9 4 on page 53 6 Replace the spine fan module y Replace defective spine fan modules as soon as they are identified 6 4 Spine Module Issue 1 Amber LED on the spine module is lit 1 Check the MLNX OS management for confirmation
53. ati zasebno i ne bi se trebao odlagati kao obi an ku anski otpad Odlaganje ovog proizvoda i svih njegovih dijelova vr ite na odgovoran i ekolo ki na in 21 Elektri na ograni enja dr ave Norve ke Ovaj je ure aj namijenjen samo za spajanje na elektri ni sustav s TN uzemljenjem i na elektri ni sustav s IT uzemljenjem dr ave Norve ke G 11 Avvertenze di sicurezza per l installazione italiano 1 Istruzioni di installazione Leggere tutte le istruzioni di installazione prima di collegare l apparecchiatura all alimentazione 2 Lesionia causa del peso h Usare un numero di persone sufficiente per sollevare in sicurezza guesto prodotto 40 Ibs 40 70 Ibs 70 121 lbs gt 121 lbs 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 Apparecchiatura pesante 4 Questa apparecchiatura molto pesante e va spostata mediante un sollevatore meccanico per evitare lesioni Mellanox Technologies 1 1 7 Rev 1 1 Rev 1 1 4 Rischio di scosse elettriche N Rischio di scosse elettriche Con il modulo ventola rimosso i pin di alimentazione sono accessibili all interno della 4 N cavit del modulo NON inserire strumenti o parti del corpo nella cavit del modulo della ventola 5 Temperatura eccessiva AN Questa apparecchiatura non va utilizzata in un area con una temperatura ambiente y superiore a quella massima consigliata 45 C 113 F Inoltre per assicurare un
54. atures List 648 EDR 100Gb s InfiniBand ports in a 28U switch 130 Tb s aggregate data switching capacity with ultra low latency e 1 3 and 1 2 1 compliant Upto EDR link speeds e N N power supply Chassis high availability sMB high availability 1 3 InfiniBand EDR Overview The Mellanox CS7500 switch system supports EDR standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 25 78125Gb s The EDR physical layer is an IBTA specified physical layer using different block types deskew mechanism and framing rules Both FDR and EDR support Forward Error Correction FEC as described in IEEE Std 802 3ap 2007 Amendment to IEEE Std 802 3 2005 chapter 74 EDR is only guaranteed to work with approved Mellanox cables A 1 4 Power Supply Redundancy The MCS7500 platform comes standard with 10 power supplies Each power supply is capable to provide 48Vpc 2500W output power 1 4 1 AC Source Requirements MCS7500 system supports 220V c operation only No support for 110VAc source Each AC source should be capable of providing at least 12A input current 1 4 2 Power Redundancy The minimum complement of PSUs which allows the chassis to run at full capacity is 5 The fol lowing two redundancy options are available No redundancy combined mode configuration grid redundant mode A N N redundancy only works with a supply voltage of 220V The chassis PSUs are fed from two power
55. aximum values 2 In the Quantity column fill in the desired configuration Please note that the number of fans is not configurable and is always 1 3 Get the total power per each FRU by multiplying values in adjusted columns in the same row 4 Get the total switch system power consumption system power by summing up all multiplica tion products Table 20 Calculating Total Switch System Power Consumption Part Quantity Power Quantity x Power IB EDR leaf with passive cables IB EDR leaf with optical cables Spine Chassis fans 1 Management module Total power 82 Mellanox Technologies Rev 1 0 Appendix E QSFP Interface Table 21 InfiniBand QSFP Connector Pin out Connector Donner Pin 2 SA Signal Description Pin Name Number 1 GND Ground 2 Tx2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 5 Tx4n Transmitter Inverted Data Input 6 Tx4p Transmitter Non Inverted Data Input 7 GND Ground 8 ModSelL Module Select 9 ResetL Module Reset 10 Rx 43 3 V Power supply receiver 11 SCL 2 wire serial interface clock 12 SDA _ 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 Rxln Receiver Inverted Data Ou
56. con residuos dom sticos A deshacerse de este producto y de todas sus partes h galo de una manera respons able y respetuosa con el medio ambiente 6 8 Rus sian 1 2 AR Tre 40 Ibs 40 70 Ibs 70 121 Ibs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Mellanox Technologies 107 Rev 1 1 4 CHAT
57. ct to the CS7500 They are 1 PC CONSOLE port this is an IZC RS232 connector for connecting to a host machine 2MGMT this is an Ethernet connector 1 USB port 1 RST reset button 1 QSFP port for debug purposes only Figure 60 Management Module Interfaces 51 350 10101 o e FAR Reset 8 Mgmtl Mgmt Console 3 4 1 PC Console RS232 The console port is used during the installation process to configure the chassis for remote man agement Connect this port to a local host using the harness supplied with the chassis See the MLNX OS User Manual for the initial configuration procedure RS232 connection should be configured for 119Kb baud rate Other than initial configuration this interface is for debug and troubleshooting only Only FAEs are authorized to connect through it Mellanox Technologies 67 Rev 1 0 Interfaces 3 4 2 Management The management ports are 1Gb s Ethernet 1GbE ports for remote management Any remote terminal connected to the Ethernet port can then be used to manage the fabric and chassis Initial configuration must be done on all of the management modules The first man agement module you configure will be the master P 3 4 3 USB The USB 2 0 port can be used to upload new software using any storage device that has a USB connector 3 4 4 Reset RST The Reset button resets the chassis management module when the button is pushed When the button is held down for
58. ctive earthing terminal to the pro tective earth wire in the building The equipment shall be installed in area where equi potential bonding exists such as a telecommunication centre or a dedicated computer room 22 Installation Codes This device must be installed according to the latest version of the country national A electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 23 Interconnection of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified A type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protec tive device rated 20 A must be incorporated in the building wiring 24 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 25 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user 20 Mellanox Technologies Rev 1 0 26 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not t
59. ctor handles reach the vertical bar Do not apply excessive force to slide in the leaf module If you feel resistance remove the leaf module and double check both the chassis and leaf for any damage 8 Catch the ejector hooks onto the vertical bar ofthe chassis and push the ejector handles shut 9 Lock the ejector handles onto the module 2 9 3 Spine Modules All replacement modules must be consistent with the chassis family and switch chip 9 Do NOT mix replacement parts based on different generations of chip generation When hot swapping any of the units it is necessary to wait 1 minute after removing the A defective part before inserting the new part This is necessary so that the management module will start a new cycle checking through the leafs and spines for the FW ver sions Each spine has a pair of ejectors that lock the module in place and serve as levers for seating or extracting see Figure 42 Management module 1 is connected to spine module 1 and management module 2 is connected to spine module 2 All of the spine modules can be hot swapped when these two management modules are installed and working When a slave management module is not installed or not working the spine module connected to the master management module cannot be hot swapped Spine module slots are numbered from top to bottom from S1 to S18 on both sides of the chassis 50 Mellanox Technologies Rev 1 0 2 9 3 1 Extracting a Spine
60. ctric NN Acest produs include o surs de alimentare suplimentar sau un spatiu gol n locul prox p pat u g A acesteia In cazul n care spatiul pentru sursa de alimentare este gol nu operati produ y sul c nd capacul orb este indep rtat sau nu este fixat in mod sigur Mellanox Technologies 111 Rev 1 1 8 Multiple mufe electrice Risc de soc electric si pericol electric y Toate aparatele cu alimentare de la re ea sunt independente Deconectati toate sursele de alimentare cu energie pentru a asigura decuplarea n inter iorul platformei de comutare 9 n timpul desc rc rilor electrice pericol electric In timpul perioadelor cu desc rc ri electrice luminoase nu lucrati cu echipamentul sau nu conecta i sau deconectati cablurile 10 Conectarea deconectarea cablului din cupru InfiniBand N a Cablurile InfiniBand din cupru sunt grele si inflexibile de aceea trebuie s fie atasate 4 sau detasate de conectori cu grij Consulta i produc torul de cabluri pentru avertis y mente instructiuni speciale 11 sau depanarea intr un rack N C nd acest produs este montat sau depanat ntr un rack trebuie s fie luate m suri de A precautie speciale pentru a se asigura c sistemul r m ne stabil In general trebuie s y umpleti rack ul cu echipamente incep nd de jos n sus 12 Instalarea echipamentului Acest echipament trebuie s fie instalat nlocui
61. d configuration SDR DDR QDR FDR or EDR and or the link width 1X or 4X The speed and n link width configurations can be retrieved using management software Spine Module LED Indicators Each leaf module is connected by links to each spine module Each spine has the following LEDs One status LED for the spine health Two status LED for the spine fan modules one per module 36 status LEDs showing the existence of leaf to spine connections One Bad Port LED showing symbol errors in the data stream OneUID LED that can be lit to identify an individual spine The 36 LEDs on each spine are divided by the number of leafs and the result N is the number of connections from each leaf that are connected to the spine 18 spines must be installed and work ing to ensure that full BW exists between nodes The maximum number of connections from each leaf is 1 If the number of leafs x the maximum number of connections per leaf is less than 36 then some of the leaf to spine connection LEDs may be OFF The status LEDs for the spine and their descriptions are shown in Table 10 The LEDs indicate as follows Figure 55 Spine Status LEDs Leaf Fan 1 LED Leaf Fan 2 LED Spine Status LED Bad Port LED UID LED Leaf connection status 62 Mellanox Technologies Rev 1 0 3 1 4 1 Status LED Table 10 shows the spine status according to the LED condition Table 10 Spine Status LED LED Condition Description
62. e de nalt frecven Imp m ntarea este esen ial nainte de a conecta sursa de alimentare 18 Interconectarea unit ilor Cablurile pentru conectarea la unitatea RS232 i la interfe ele Ethernet trebuie s fie de tipul DP 1 sau DP 2 certificate UL Not c nd se reg sesc ntr un circuit non LPS Protec ie la supracurent Un dispozitiv de protec ie la supracurent nregistrat n cir cuitul de ramificare u or accesibil i cu o putere nominal egal cu 20 A trebuie s fie integrat n cablajul cl dirii 19 Nu utiliza i comutatorul ca raft sau spa iu de lucru Aten ie Echipamentul montat pe o linie de alunecare in nu va fi utilizat ca raft sau spa iu de lucru Scopul inelor nu este de a glisa unitatea de pe rack Acestea sunt des tinate instal rii permanente numai la punctul final de oprire i nu vor fi folosite pentru depanare i ntre inere 20 Directiva DEEE n conformitate cu Directiva DEEE 2002 96 CE toate de eurile de echipamente elec trice i electronice EEE trebuie colectate separat i nu trebuie eliminate mpreun cu de eurile menajere obi nuite Eliminati acest produs i toate componentele sale n mod responsabil i ecologic Mellanox Technologies 113 Rev 1 1 G 10 Sigurnosna upozorenja za instaliranje Croatian 1 Upute za instaliranje Pa ljivo pro itajte upute za instaliranje prije spajanja opreme na izvor elektri ne energije 2 Tjelesne
63. e of used batteries according to the instructions 19 UL Listed and CSA Certified Power Supply Cord For North American power connection select a power supply cord that is UL Listed A and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A Mellanox Technologies 19 Rev 1 0 Installation 20 High Leakage Current y Warning High leakage current Earth connection essential before connecting supply 21 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of AA this device must be connected to the protective earth in the building installation GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of a PROTECTIVE EARTHING CONDUCTOR from the separate prote
64. e shelf at U3 since the chassis would roll off easily on it from the pallet b Insert 6 M6 cage nuts from the inside of beam 3 and 4 meant to support the shelf Figure 11 c Place the top end of the support bracket at the top of U4 using the M6x25 screw on beams 3 and 4 Do not tighten the screw completely at this point Figure 11 Cage Nut Placement for Shelf i nom LEAF SIDE M6 SPRING WASHER SUPPORT RACKET CAGE NUT M6 30 Mellanox Technologies Rev 1 0 d Insert 6 M6 cage nuts from the inside of the spine side beams of the rack in the slots indicated in Figure 12 e Insert 6 cage nuts from the inside of the leaf side beams of the rack in the slots indicated in Figure 12 The indexes in Figure 12 count from the first rack hole above the shelf WU Figure 12 Cage Nut Placement for Switch System Chassis Mall CAGE NUT B Mellanox Technologies 31 Rev 1 0 Installation Step 2 Measure the distance between the spine side and leaf side beams of the rack and adjust the shelf s leaf side brackets to that distance while allowing some wiggle room Figure 13 Figure 13 Adjusting the Leaf Side Shelf Brackets 715 850 3mm SHELF LEAF 715 850mm BEAMS 1 2 lt gt BEAMS 3 4 BEAMS 1 2 SPINE SIDE RELEASE 10 HEX M6 SCREWS ONLY TO ADJUST SHELF ARMS BEAMS 3 4 LEAF SIDE BEAMS 3 4 L
65. et wird sind besondere Vor A Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 11 Ger teinstallation A Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert aus getauscht oder gewartet werden 12 Ger teentsorgung N N 4 Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 4 13 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bes timmungen installiert werden 14 Installationscodes installiert weden In Nodamerika muss das Ger t gem den geltenden Anforderungen EN N Dieses Ger t muss entsprechend der aktuellsten Version des National Electrical Code des US National Electrical Code und des Canadian Electrical Code installiert werden SE 102 Mellanox Technologies Rev 1 1 15 UL und CSA zertifiziertes Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss
66. f needed Fill in the Table 17 to calculate the weight of your system Table 17 Switch System Weight Calculation Number of FRUs FRU Type Weight of 1 FRU Total Weight FRU Type Chassis as shipped 214 6 of Spines 3 015 of leafs 5 045 of leaf blanks 1 303 Il of management modules 4 877 of management module blanks 1 473 Total This total is in kilograms Multiply the total by 2 2 to get the total weight in pounds Mellanox Technologies 81 Rev 1 0 Appendix D Calculating the Power of a Chassis To calculate the power consumption of a chassis add the power of the fans spines leafs and management modules Table 18 Power Consumption of Chassis Parts Part Typical W Maximum W IB EDR leaf with passive cables 322 335 IB EDR leaf with 36 optical cables 483 496 Spine 163 170 Chassis fans 740 740 Management module 250 250 Table 18 assumes the QSFP cable power consumption specified in Table 19 Please note that QSFP module power is related to the module itself and is not referenced to AC plane Table 19 QSFP Cable Power Consumption QSFP Cable Type Typical W Maximum W Optical 3 5 3 5 To calculate the total power consumption of the switch system at a certain configuration fill out Table 20 in the following manner 1 In the Power column fill in the desired typical or m
67. f this happens it will be necessary to reboot the chassis once the defective PSU has been replaced 1 Check all FRUs for proper insertion and seating before connecting the AC power cords Power supplies e Leaf fan modules Spine modules Spine fan modules Insert all leafs that you plan to use in the chassis Insert thermal blanks in unused leaf and management slots to maintain balanced air flow Connect the power cords to the PSUs DA OA Connect the power cords to grounded electrical outlets A fully loaded CS7500 series switch system can draw 12 5kW of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or a 3 phase supply 4 1 System Status After Power Up It can take around 5 minutes for the system to boot up Turn off the system if any LEDs remain red for more than 7 minutes As system boot been completed you should observe the following 1 PSU AC OK and DC OK indicators are green 2 That the Status LED on each spine leaf and management module are green which indicates that the power supplies are good 3 Spine module indicators display the status of internal links to the installed leaf modules All PHY links to existing leafs should be ON 4 Check the spine LEDs and make sure they coincide with Figure 61 Mellanox Technologies 69 Rev 1 0 Chassis Power Up Figure 61 Spine Side Panel Display Status Indications 5
68. faces 67 341 DE Console RS232 5i e a era eR Re EIUS as 67 342 Management A A A A dar 68 68 344 Reset RST si Bears Brass 68 Chapter 4 Chassis Power Ron RT DR RR s li sata ODS 69 4 1 System Status After Power Up 69 Chapter 5 Switch Management Tools 71 5 1 InfiniBand Subnet Manager 72 5 2 Fabric Inspector Diagnostics 72 5 3 Accessing the CPU via the Ethernet Connector 72 5 4 Upgrading Softwares a et er A ed e ied ANS 72 Chapter 6 Troubleshooting 73 6 1 Power Supply Unit eli ing een ut 73 6 2 1 0 73 6 3 Management Module 74 6 4 Spine Module ua e tele SA Gp UNE RE len 75 6 5 MLNX OS 75 Appendix A Specification Data rnm nnn 77 AT EMI Certification 79 A2 Approved Cables ser garia ee en dte esten 79 79 Appendix Thermal Threshold Definitions 80 Appendix C Calculating the Weight of a Customized Chassis 81 Appendix D Calculating the Power of a Chassis
69. ffic patterns and user assigned system names GUIDs For more information please refer to the MLNX OS User Manual 5 3 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet WEB SNMP XML 5 4 Upgrading Software The new software updates are available to the user from the Mellanox Support website Copy the update to a known location on a remote server within your LAN and follow the steps listed in MLNX OS User Manual to perform the software upgrade 4 If MLNX OSQ is updated and the FW image in the leafs and spines of the chassis is an earlier version from the minimum requirement then the chassis management system may require up to 30 minutes to update all of the FW images in all of the leafs and spines 72 Mellanox Technologies Rev 1 0 6 Troubleshooting 6 1 Power Supply Unit Ifthe Power supplies cannot supply enough power the management module shut down the some leafs pre As each PSU is plugged in make sure that the green power LEDs on the PSU comes on Issue 1 If the AC power LED is off Check that the power cable is the correct power cable for your country Check that the power cable is plugged into a working outlet Check that the power cable has a voltage within the range of 180 240 volts AC Remove and reinstall the power cable Check the circuit breakers to be sure that the breaker
70. for users and system administrators responsible for installing and setting up the chassis platform The manual assumes familiarity with the InfiniBand architecture specification Related Documentation The documentation set accompanying the QSFP28 Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Websites Document Name Description InfiniBand Architecture Specification The InfiniBand Architecture Specification that is provided by Vol 1 Release 1 2 1 IBTA Switch Product Release Notes For possible hardware issues see the switch support product page This requires a customer support login Look up the rele vant Switch IBTM based switch system series release note file MLNX OS User Manual for VPI This document contains information regarding configuring and managing Mellanox Technologies SwitchX switch platforms listing all of the commands available through MLNX OS with explanations and examples Dismantling Procedures Dismantling user guides for Mellanox Technologies products organized by product category http www mellanox com page dismantling_procedures Conventions Throughout this manual the name CS7500 and the terms chassis and switch are used to describe the port QSFP28 InfiniBand chassis unless explicitly indicated otherwise The following icons are used throughout this document to indicate information that is important to the user This symbo
71. grids for high availability The second power grid can be supplied by any of the following a backup power supply grid generator Mellanox Technologies 15 Rev 1 0 Overview a battery backup system any combination of the above Connecting 5 power supplies to one power supply grid and the remaining 5 power supplies to a sec ondary power supply grid will create N N redundancy This is high availability Under these condi tions should a power grid fail an electric company power failure or blackout for example power grid High Availability will continue to keep the chassis running at full capacity through the secondary or backup power supply grid The system cannot run with less than 5 PSUS thus if one of the power grids is down there is no power or PSU redundancy in the system When the power drops below the required minimum due to power supply failure MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced Two ways to reboot are to use the reboot command in the CLI or reboot through the WebUI Table 4 Power Supply OPN PSU OPN Wattage Description MTDF PS A 2500W Supplies N N redundancy for all switch chassis at 220 Volts 16 Mellanox Technologies Rev 1 0 2 Installation This chassis can be installed in standard 19 racks that have depths between 715 850mm between the vertical supports of the
72. has not tripped Check that the power cable is good Replace the power cable If the AC power LED is green but the DC power LED red replace the PSU gi AR ES 6 2 Leaf Module Issue 1 The power LED for the leaf module is off 1 Make sure that all of the PSUs are showing DC OK 2 Uninstall and reinstall the Leaf module 3 When the amber LED is on this indicates a fault in the module uninstall and reinstall the leaf module 4 If uninstalling and reinstalling the leaf module does not work burn the latest FW on the leaf module and uninstall and reinstall the leaf module 5 Replace the leaf module with a new one Should any of the modules shut down due to over temperature wait 5 minutes and then follow the procedure starting with Step 2 Issue 2 The physical link LED for the InfiniBand connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest FW version is installed on both the HCA card and the switch 4 If media adapters are used check that the all connections are good tight and secure 5 Replace the cable Issue 3 The activity indication does not come on Check that the Subnet Manager has been started Mellanox Technologies 73 Rev 1 0 Troubleshooting 6 3 Management Module Issue 1 Amber Status LED for the chassis on the management module is lit 1 Check the MLNX OS management for confirmat
73. hnologies Rev 1 1 14 Yerel ve Ulusal Elektrik Kodlar Bu cihaz n lke ulusal elektrik kodlar n n son s r m ne g re monte edilmesi hh gerekir Kuzey Amerika i in ekipman n ABD Ulusal Elektrik Kodu ve Kanada Elektrik Kodu nun uygulama kosullarina g re monte edilmesi gerekir 16 UL Kayith ve CSA Onayl G c Kaynagi Kablosu A Kuzey Amerika da g g baglantisi icin UL Kayitli ve CSA Onayli bir g kaynag 4 kablosu secin 3 iletken 16 AWG 125 V degerinde kaliplanmis bir fisle biten 13 A en az 1 5 m alti fit uzunlugunda fakat 4 5 m den uzun olmayan bir kablo Avrupa da g c baglantisi i in uluslararas uyumlu ve lt HAR gt isaretli 3 iletken en az 1 0 mmo tel 300 V de erinde ve PVC yal t ml bir g kayna kablosu se in Kablonun 250 V 10 A de erinde bir kaliplanmis fi i olmas gerekmektedir 17 Y ksek Kacak Ak m Uyary Y ksek ka ak ak m varsa g c kaynag na baglanmadan nce mutlaka toprak lama baglant s yap lmal d r 18 nitelerin Ara Baglantisi RS232 nitesini ve Ethernet Arabirimlerini baglayacak olan kablolar n UL onayl DP 1 veya DP 2 tipi olmas gerekir Not LPS olmayan devreye aitse A r Ak m Korumas Kolayca eri ilebilecek 20 V Kay tl devre par as a r ak m koruma cihaz n n bina elektrik ebekesinde kurulu olmas gerekir 19 Anahtar Raf veya al ma Alan olarak kullanmay nl Dikkat
74. ictions Norway only A This unit is intended for connection to a TN power system and an IT power system of G3 1 A 999 MANN mr 2 N A men PN 32 ma nm 3 1900 4 4 v A 90 Mellanox Technologies Rev 1 1 5 nio gt nya PR DID 6 Y N PN 45 C 113 F nov 8
75. impede the airflow through the ventilation holes next to the InfiniBand A ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 2 8 1 Supported Approved Cables For a list of approved cables for this switch see the Mellanox approved cable list http www mellanox com page cables mtag cable overview 2 8 2 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox CS75x0 series chassis are rated for cables up to class 4 as per the SFF committee classification SFF 8436 PDF 2 9 Hot Swap Insertion and Extraction Before starting any procedure on the CS7500 switch system put an ESD prevention wrist strap on your wrist and connect to the chassis 74 Do not mix replacement parts based on different generations of chip All replacement modules must be consistent with the chassis family and switch chip generation 46 Mellanox Technologies Rev 1 0 When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions This switch platform supports hot swap capabilities for the following parts Power supply units Leaf modules Spine modules e Leaf fan modules
76. ion and possible explanation of the alert 2 Reset the master management module by pushing the rest button If you have two manage ment modules installed this will convert the master management module to the slave and con vert the slave to the master If there is only one management module in the chassis all of the leafs and ports A are reset by bringing them down and powering them up when the management module is removed Make sure the S Fans and L Fans LEDs are green Make sure that the spine and the leafs both have the same version of FW Reburn the FW and remove and reinstall the management module moo If you are running the chassis with only one management module remove and reinstall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 2 9 5 on page 56 7 If you are running the chassis with only one management module replace the management module Issue 2 Amber LED for the leaf fan on the management module is lit 1 Check the MLNX OS management for confirmation and possible explanation of the alert 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner 4 Determine which f
77. ircuits are not overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 2 8 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from Mellanox Technologies The Mellanox proprietary QSFP cables support full 100 100Gb s EDR 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connections are made to Mellanox Technologies 45 Rev 1 0 Installation the leaf modules Each leaf has 18 InfiniBand QSFP connectors in two rows which are num bered 1 18 See Section 3 1 7 for port numbering If maximum cable lengths exceeded data transfer will be reduced and the bit error rate will increase Au EDR is only guaranteed to work with approved Mellanox cables ae All cables can be inserted or removed with the unit powered on To insert a cable press the con nector onto the port receptacle until the connector is firmly seated The orange LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices Allow 15 seconds for link to get up To remove dis engage the lock and slowly pull the connector away from the port receptacle A For a valid physical connection both ends of the cable must be connected to working devices Careful to not
78. it tre d plac avec un systeme de levage m canique pour viter les blessures Mellanox Technologies 9 7 Rev 1 1 Rev 1 1 4 Danger d lectrocution AN Danger d electrocution 4 Lorsque le module de ventilation est retir les broches d alimentation sont expos es A dans l emplacement du module NE PAS ins rer d outils ou la main dans l emplacement du module 5 Surchauffe Cet quipement ne doit pas tre en service dans un local dont la temp rature d passe le maximum recommand de 45 C 113 F En outre et pour garantir une circulation y d air correcte laisser un espace d au moins 8 cm 3 autour des orifices de ventilation 6 Ch ssis empil sur d autres quipements Le ch ssis ne doit pas tre empil sur d autres quipements S il tombe il peut endom mager l quipement ou entra ner des blessures 7 Connexion de l alimentation redondante danger d lectrocution Ce produit est quip d une alimentation redondante ou d un cache si elle est absente Dans ce dernier cas ne pas faire fonctionner le produit si le cache est retir ou mal fix gt 8 Plusieurs prises d alimentation Risque et danger d lectrocution Les alimentations sont toutes ind pendantes Pour s assurer que le commutateur est bien hors tension d branchez toutes les alimen tations 9 En cas d orage danger d lectrocution Pendant un orage ne pas travailler sur l quipeme
79. l indicates information that is helpful to the user This symbol indicates a situation that can potentially cause damage to hardware or soft ware 10 Mellanox Technologies Rev 1 0 This symbol makes recommendations to the user Mellanox Part Numbering Legend Table3 Chassis Installation Kit Items Place Field Decoder M Company name Mellanox Technologies CS MB SB Switch system type Chassis Management Board Switch IBTM in leaf or spine module A Data rate 7 EDR InfiniBand System type 5 director switch XX Number of ports 00 648 ports 10 324 ports 20 216 ports Mellanox Technologies 11 Rev 1 0 Overview 1 Overview This chapter provides an overview of the CS7500 QSFP28 InfiniBand director switch platform referenced in this document as the chassis or the switch and its operational environment Mellanox CS7500 switch system provides the highest performing fabric solution by delivering high bandwidth and low latency to enterprise data centers EDC high performance computing HPC and embedded environments Networks built with the CS7500 system can carry con verged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 6th generation of Switch IBTM switch devices CS7500 systems provide up to 100Gb s full bidirectional bandwidth per port With up to 648 ports in a 28U rack space this sys tem is amo
80. lanox Technologies 51 Rev 1 0 Installation Figure 42 Releasing Spine Ejector Handles Push in this direction Push in this direction 3 Open the ejectors until they are at a 45 degree angle from the module A Do not use the fan FRU handle to extract the spine module 4 Pull out the module half way through the guiding rails using both ejectors 5 Re lock the ejector handles 6 Hold the body of the module on both sides and remove it from the chassis 2 9 3 2 Inserting a Spine Module When hot swapping any of the units it is necessary to wait 1 minute after removing the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert a spine module 1 Check for foreign objects in or mechanical damage to the chassis spine slots 2 Check back signal connectors integrity Look for any broken signal dividers or any devia tions from the pass criteria shown in Figure 43 Figure 43 Intact Spine Signal Connectors 2 Mellanox Technologies Rev 1 0 3 Check power connector s integrity Look for any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 44 Figure 44 Intact Spine Power Pin Holders 4 Start with the ejector handles fully open that is at 45 degrees to the front panel of the spine Do not use the f
81. le LED Indicators 60 Figure 55 Spine Status LEDs 62 Figure 56 Spine Side Panel Display Status Indications 64 Figure 57 Management Module Indications 65 Figure 58 Port Numbering eee E e euren 66 Figure 59 Top and Bottom Ports 66 Figure 60 Management Module Interfaces 67 Figure 61 Spine Side Panel Display Status Indications 70 Figure 62 Management Module Status Indications for Normal Operation Master 70 Figure 63 Management Module Status Indications for Normal Operation Slave 70 Figure 64 InfiniBand Connector Symbol 84 Figure 65 Connector Male and Female Views 85 8 Mellanox Technologies Rev 1 0 Revision History Table 1 User Manual Revision History Rev Date Description 1 0 19 August 2015 First release Mellanox Technologies 9 Rev 1 0 About this Manual This manual provides an overview of the Switch IBTM based InfiniBand CS7500 director switch and guidelines for its operation Intended Audience This manual is intended
82. link is not up Mellanox Technologies 75 Rev 1 0 Troubleshooting PCIEl successfully set as root complex OT 5007715537 0006 00 Nee ppc 4xx eth0 ppc 4xx ethl Hit Ctrl B to stop autoboot 0 Mellanox FabricIT Boot Menu 1 EFM M460EX EFM 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM M460EX EFM 1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt Cinsle 5 Select the image to boot For more detailed instructions concerning MLNX OS software see the Mellanox MLNX OS Switch IBTM Software WebUI User s Manual or the Mellanox MLNX OS Switch IBTM Soft ware User Manual 76 Mellanox Technologies Rev 1 0 Appendix A Specification Data Table 16 Switch Specification CS7500 Physical 49 x 19 x 30 inches Size 1243 8 mm x 482 6 mm x 761 2 mm The shelf adds 44 45 mm to the height Mounting 19 Rack mount of spines 18 of leafs 18 214 6 kg 473 1 Ib empty configuration no leafs spines or management modules Weight 369 5 kg 814 6 Ib full configuration 256 2 kg 564 9 Ib shipped configuration Rail kit 11 5 kg 25 4 Ib in Cable supporter 18 shelves 10 kg 22 0 Ib 6 Cable management 5 kg 11 0 Ib CoGh 595 9 mm from bottom of chassis Center of CoGw 241 3 mm from left side surface Gravity CoGd 359 19 mm from port end surface Measured from the bottom left corner as one faces the switch system Total Max 56 6 M min 20
83. monized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 16 Installation codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 17 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when residing in non LPS circuit A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 19 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance Mellanox Technologies 89 Rev 1 1 20 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way pum 21 Country of Norway Power Restr
84. n olabilir Mellanox Technologies 121 Rev 1 1 7 Yedekli G c Kayna y Ba lantysy Elektrik Carpma Tehlikesi Bu r n yedekli g kaynagi veya onun yerine bos elektrik kutusu icerir G kaynag N i in bos elektrik kutusu varsa kutunun kapag a ikken veya tam olarak kapat lmam ken r n al t rmay n 8 oklu G Giri leri Elektrik arpmas riski ve enerji tehlikesi B t n PSU lar G Kayna niteleri ayr d r Anahtar platformundaki g c kapatmak i in t m g kaynaklar n n ba lant lar n kesin 9 im ek Elektrik arpma Tehlikesi G ky z nde simsek gakt g zamanlarda ekipman zerinde alismayin veya kablo baglamay n ya da kablo baglant s kesmeyin 10 Bak r nfiniband Kablo Ba lama Ba lant y Kesme Bak r nfiniband kablolar a rd r ve esnemezler Bu nedenle ba lant lara ok dikkatli bir ekilde tak lmalar veya kar lmalar gerekir zel uyar lar talimatlar i in kablo reticinize ba vurun 11 skele Montaj ve Bak m Bu r n bir iskelede monte edildiyse veya bir iskele ile sunulduysa sistemin sabit kalmas i in zel nlemler al nmal d r Genelde ekipmanlar iskeleye a a dan yukar do ru doldurman z gerekir 12 Ekipman Montaj Ekipman n yaln zca e itimli ve nitelikli personel taraf ndan monte edilmesi de i tirilmesi ve veya bak m n n yap lmas gerekir 122 Mellanox Tec
85. ng a symbol error The LED immediately goes off until the next symbol error is received 3 1 3 3 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support 3 1 3 4 Leaf Module Port Connector LED Assignment Above the ports are two LEDs one for the upper port A and one for the lower port V Each port has a single 2 color LED Table 9 presents the link status according to the LED condition Table 9 Connector Physical and Logical Link Indications LED Condition Description Off No power to the port Green Solid Logical link up Green Flashing Data activity Flashing frequency is proportional to data transfer speed Mellanox Technologies 61 Rev 1 0 Interfaces Table 9 Connector Physical and Logical Link Indications LED Condition Description Amber Solid Physical link up Amber Flashing A problem with the physical link The LED indicator corresponding to each data port will light orange when the physical connec tion is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a logical connection is made the LED will change to green When data is being transferred the light will blink green The switch does not provide a visual means to indicate the port spee
86. ng remotely To shut it down the power plugs need to be physically pulled from the power grid To reboot any of the modules accessible through CLI the command reload must be run on them in configuration mode Config as shown in the following switch config reload 58 Mellanox Technologies Rev 1 0 3 Interfaces 3 1 LED Status Indicators The LEDs are placed on the chassis for the convenience of IT manager chassis conditions and management options are available and controllable through the management software either CLI or WebUI Y It is recommended that all of chassis subsystems be maintained and managed through the management software 3 1 1 Power Supply Unit LEDs The PSUs are on the spine side of the chassis at its upper side and are behind two cover panels The plugs for these PSUs are on the leaf side of the chassis Each power supply unit has the following LED indicators Figure 52 Power Supply Unit Status Indications ACOK LED indicator e Green when input voltage is 85Vac gt Vin gt 270Vac e Red when input voltage is 85Vac gt Vin or Vin gt 270Vac DC OK LED indicator e Green when output voltage above 90 5 of 48Vpc Red when output voltage fallen below 90 5 of 48V pc Mellanox Technologies 59 Rev 1 0 Interfaces Figure 53 6057500 PSU Cover CS7500 Mellanox TECHNOLOGIES 3 1 2 System AC Power Interface Each PSU in the switch sy
87. ng the densest switching systems available The switch platform comes pre installed with all necessary firmware for standard operation within an InfiniBand fabric and requires an InfiniBand compliant Subnet Manager running from one of the hosts or the management module of the switch system The initial configuration proce dure should be followed to initialize the switch before connecting it to the network after which normal operation can proceed See the installation guide for details regarding the initial configu ration Once connected to the network the Subnet Management software automatically discov ers and configures the fabric and begins utilizing the switch The Mellanox Operating System MLNX OS software package provides a subnet manager and network management tools as well as connectivity software for servers and storage and is available on the Mellanox website Basic installation is covered in Chapter 2 Installation on page 17 Hot swapping components and hardware maintenance is covered in Section 2 9 Hot Swap Insertion and Extraction on page 46 1 1 Product Information 1 1 1 Serial Number and Product Version Information The serial number GUID identifier and product version information are found on the label attached to the pull out tab below the Mellanox logo on the spine side of the chassis Figure 1 Product Label 648 port EDR InfiniBand chassis SIN MTYYWWSZZZZZ YYYY MM DD 0 P N 0
88. nt ni brancher ou d brancher des c bles 10 Connexion et d connexion du c ble InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et peu flexibles Par cons quent il faut proc der avec soin pour les brancher ou les d brancher des connecteurs Consulter le fabricant du c ble pour obtenir des instructions ou des avertissements sp cifiques 98 Mellanox Technologies Rev 1 1 11 Montage en rack et maintenance ment attention pour s assurer que l ensemble reste stable En r gle g n rale le rack doit Lors du montage ou de la maintenance de ce produit dans un rack il faut faire sp ciale N tre rempli en commen ant par le bas 12 Installation de l quipement Cet quipement ne doit tre install remplac et maintenu que par un personnel form et qualifie mise au rebut de cet quipement doit se faire conform ment toutes les lois et r glementations nationales 14 Codes lectriques locaux et nationaux N A IN Cet quipement doit tre install conform ment aux codes lectriques locaux et nation aux 15 Codes d installation Cet appareil doit tre install conform ment la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 16 Cordon d alimentation UL Listed e
89. o be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent instal lation at final resting place only not used for service and maintenance 27 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Country of Norway Power Restrictions 28 This unit is intended for connection to a TN power system and an IT power system of Norway only 2 2 Environmental and Safety Recommendations The following are Mellanox recommendations Recommended ambient temperature in the System room is 20 5 C ES Recommended humidity range is 40 15 without condensing It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference It is highly recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure ES Mellanox Technologies 21 Rev 1 0
90. odule will start a new cycle checking through the leafs and spines for the FW versions 1 Make sure the fan module is oriented correctly with the release latch on the right Confirm that the location of the connector in the chassis will line up with the connector in the fan mod ule 2 Slowly slide in the new spine fan module 3 Push the fan module as far as it will go make sure the locking latches engage If the fan LED continues to show amber remove the fan module and check the pins on the connector inside of the spine to make sure that none of them are bent Management Module When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions Extracting a Management Module Management modules are located on the leaf side above the leafs There are two slots to install the management modules Only one management module is required to run the switch system A If only one management module is used the other management slot should be covered d using a management blank unit Each management module has a pair of ejectors that lock the module in place and serve as a lever for seating or extracting see Figure 49 1 Run the shut down command no power enable module For example to shut down man agement module 2 run
91. ogies Rev 1 0 List of Figures Figure Product Label acri vA E ate wee in een Re e exo o a 12 Figure 2 Product Label Tab Location 13 Figure 3 Management Module MAC Address Location 13 Figured ESTO ida hte Cet gern Oe e ea Pare sid ol e alata 14 Figures Shock Stickers lak al MERE ala 23 Figure Crate Screws olores sn een le m ac eh ee 24 Figure 7 Distance Between the Vertical Supports 25 Figure 8 Chassis Installation Kit 27 Figure 9 Cable Management Installation 28 Figure 10 Shelf Installation Kit 29 Figure 11 Cage Nut Placement for Shelf 30 Figure 12 Cage Nut Placement for Switch System Chassis 31 Figure 13 Adjusting the Leaf Side Shelf Brackets 32 Figure 14 Placing ies ae Base ee VIE nani 32 Figure 15 Tightening Shelf Cage Nut Screws on Spine Side 33 Figure 16 Tightening Shelf Cage Nut Screws on Leaf Side 33 Figure 17 Tightening Shelf Bracket Screws 34 Figure 18 Tighten Shelf Back Support Screws
92. oji je na UL listi 1 sa MA CSA certifikatom 3 Zilni 16 AWG 16 AWG koji zavr ava lijevanim uti nazivnog napona od 125 V 13 A minimalne duljine od 1 5 m six feet Sest stopa ali ne dulji od 4 5 m Za europsku mrezu odaberite kabel napajanja koji je medunarodno uskladen i oznacen lt HAR gt 3 Zilni s najmanje 1 0 mm Zice nazivnog napona od 300 V s PVC izolacijom Kabel mora imati lijevani utika nazivnog napona od 250 V nazivne struje od 10 A 17 Veliko curenje struje 4 Upozorenje Veliko curenje struje Prije spajanja napajanja nuZno je spojiti uzeml jenje 18 Interkonekcija uredaja Kabeli za spajanje na jedinicu RS232 1 Ethernet sucelja moraju biti s UL cer tifikatom vrste DP 1 ili DP 2 Napomena kad se nalazi u krugu bez LPS vodita Za tita od strujnog preoptere enja Uvijek dostupni odobreni za titni ure aji od strujnog preoptere enja nazivne struje od 20 A moraju se ugraditi u o i enje zgrade 19 Nemojte koristiti preklopnik kao policu ili radnu povr inu Pozor Oprema montirana na kliza e vodilice ne bi se trebala koristiti kao polica ili radna povr ina Vodilice nisu namijenjene za povla enje ure aja iz ormari a Slu e samo za trajnu instalaciju na kona nom polo aju a ne za servisiranje i odr avanje 116 Mellanox Technologies 20 WEEE direktiva Sukladno WEEE direktivi 2002 96 EZ sav elektri ni i elektronicki otpad EEE trebao bi se prikuplj
93. ologies Rev 1 1 7 Conexi n redundante de fuente de alimentaci n peligro de descarga Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a A esta quitada o no esta bien cerrada 8 Tomas de alimentaci n multiples Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes Desconecte todas las fuentes de alimentaci n para asegurar que no haya corriente alguna dentro de la plataforma de conmutaci n gt 9 Alhaber rayos peligro de descarga No utilizar el equipo ni conectar o desconectar cables durante periodos de actividad de LA rayos 10 Cable de conexi n y desconexi n InfiniBand de cobre Dado que los cables de cobre InfiniBand son pesados y no son flexibles su conexi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver adver tencias o instrucciones especiales consultar al fabricante del cable 11 Montaje y mantenimiento del bastidor Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 12 Instalaci n del equipo La instalaci n el reemplazo y el mantenimiento de este equipo estar n a
94. on Instructions Read all installation instructions before connecting the equipment to the power source 2 Bodily Injury Due to Weight dp Use enough people to safely lift this product 3 Heavy Equipment gp This eguipment is heavy and should be moved using a mechanical lift to avoid injuries 4 Risk of Electric Shock Risk of Electric Shock With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 5 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper allow at least 8cm 3 inches of clearance around the ventilation openings Mellanox Technologies 87 Rev 1 1 6 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 7 Redundant Power Supply Connection Electrical Hazard power supply do not operate the product with the blank cover removed or not securely This product includes a redundant power or a blank in its place In case of a blank A fastened 8 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 9 During Lightning Electrical Hazard During periods
95. onnections 45 2 8 InfiniBand Cable Installation 45 2 8 1 Supported Approved Cables 46 2 8 2 Cable Power Classes ana ee ne ee 46 2 9 Hot Swap Insertion and Extraction 46 2 9 1 Power Supply Units aan ad enden 47 2 92 Leaf Modules oo ee de Oi elia ee 48 2 9 3 Spine Modules la ale een ernten 50 2 9 4 Fan Modules veru P CENE re 53 2 9 5 Management Module 56 2 9 6 Switch Shut Down Procedures 58 Chapter 3 Interfaces 2244 ox e ii O 3 1 LED Status Indicators 2 See en en a km eml Geli ek ei 3 59 3 1 1 Power Supply Unit LEDs 59 3 1 2 System AC Power Interface 60 3 1 3 Leaf Module LED Indicators 60 3 1 4 Spine Module LED Indicators 62 3 1 5 Spine Side Panel Display LED Indicators 64 Mellanox Technologies 3 Rev 1 0 3 1 6 Management Module LED Indicators 65 3 1 7 Port Connector Interfaces oves bs x Eu Rs Nat header 66 BZ A A 67 3 3 Cable Power Budget Classification 67 3 4 Management Module Inter
96. ons Spine Fan Off No power Green Status Green Normal operation Amber Fatal error in one or more of the spine fans check individual spine fan LEDs for fault indications Leaf Fan Status Off No power Green Green Normal operation Amber Fatal error in one or more of the leaf fans check individual leaf fan LEDs for fault indications MNGI Master Off No power Green Status Management module is not installed e Management module is not the master Green Management module is up and operating as master Amber Slave management module not installed MNG2 Master Off Management module is not the master Off Status No power Green Management module is up and operating as master 3 1 6 Management Module LED Indicators Figure 57 Management Module Indications Status Master PSU Status Leaf Fan Spine Fan Status Status The management module LEDs display the switch system operating conditions Mellanox Technologies 65 Rev 1 0 Interfaces Table 15 Management LED Display for Normal Operation qu ER Normal LED Condition Description State Status LED Off No power Green Shows the sta tus of the chas Green System is up and running are Green Flashing System booting restore factory default in progress Amber System error attention needed e g overheating diagnostics fail CPU hang HW fail overheat critical PSU Status Off No power Green Green Normal
97. operation Amber PS fault detected check individual power supplies for fault indications Spine Fan Off No power Green Status Green Normal operation Amber Fatal error in one or more of the spine fans check individual spine fan LEDs for fault indications Leaf Fan Status Off No power Green Green Normal operation Amber Fatal error in one or more of the leaf fans check individual leaf fan LEDs for fault indications Master Off This management module is not the master Green Green Management module is operating as a master 3 1 7 Port Connector Interfaces The connector side of the switch has 18 leaf modules and each leaf module has 36 QSFP28 ports The ports on each leaf module are placed in two rows 18 ports to a row The ports are labeled as shown in Figure 58 The bottom row ports are flipped from the top row See Figure 59 Figure 58 Port Numbering Figure 59 Top and Bottom Ports 66 Mellanox Technologies Rev 1 0 3 2 Air Flow These switches come with the air flow pattern of air entering through the spine side and exiting through the connector side 3 3 QSFP Cable Power Budget Classification All CS7500 switch systems are designed for active cables with a max power per module of 3 5W This is power level 4 according to the QSFP Public Specification 3 4 Management Module Interfaces The switch system requires at least one management module The management module has 6 interfaces to conne
98. or any damage on the power connector casing or blades damage or any deviations from the pass criterion shown in Figure 51 Figure 51 Intact Management Power Pins Mellanox Technologies 57 Rev 1 0 Installation 5 Start with the ejector handles fully open that is at 45 degrees to the front panel of the man agement 6 Holding the management by its sides carefully set the management module into the chassis 7 Using only the management ejector handles slowly slide the module into the chassis until the hooks reach the vertical bar Do not apply excessive force to slide in the management module If you feel resis A tance remove the management and double check both the chassis and management for any damage 8 Catch the hooks onto the vertical bar of the chassis and push the ejector handle shut 9 Lock the ejector handles onto the module On switch systems with dual management systems first connect the cable and configure 4 the master management module and only then configure the slave By default the master is the top management module For further information on the master and slave roles see section Subnet Manager SM High Availability HA of the MLNX OS User Manual All management modules in the chassis must go through an initial configuration proce d dure See the Installation Guide for the initial configuration procedure 2 9 6 Switch Shut Down Procedures The chassis cannot be shut down usi
99. orriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 18 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n mas reciente de los c digos el ctricos nacionales del pais en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canada 19 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una corriente nominal de 20 A 106 Mellanox Technologies Rev 1 1 20 No utilizar el conmutador como estante ni como espacio de trabajo Cuidado Equipos montados en deslizadores o rieles no se deben utilizar como estantes ni como espacio de trabajo La finalidad de los rieles no es deslizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o mantenimiento 21 Directiva WEEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto
100. power out from the PSUs chassis failures querying for switch serial numbers revisions software version e Switch IBTM FW version switch temperatures The manager also has the ability to burn new firmware and upgrade software on the switch Mellanox Technologies 71 Rev 1 0 Switch Management Tools 5 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch It discovers and configures all the InfiniBand fabric devices to enable traffic flow between those devices The SM applies network traffic related configurations such as QoS routing partitioning to the fabric devices Each InfiniBand subnet needs one SM to discover activate and manage the subnet The network requires an SM to be running in either the InfiniBand switch itself switch based or on one of the nodes which is connected to the InfiniBand fabric host based To learn about SM configuration please refer to the MLNX OS User Manual 5 2 Fabric Inspector Diagnostics Fabric Inspector is a plug amp play software for fee within MLNX OS displaying and filtering all identified systems and nodes within the fabric Fabric Inspector includes a complete set of tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creating filtering rules on a system wide basis between nodes or port connections based on tra
101. rantwortungsvoller und umweltfreundlicher Art und Weise entsorgen Mellanox Technologies 103 Rev 1 1 G 7 Advertencias de seguridad de instalaci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Lesi n corporal a causa de peso Recurra a suficientes personas para levantar este producto sin A se 40 Ibs 40 70 Ibs 70 121 lbs 2121 lbs 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Equipos pesados mec nico para evitar lesiones 4 Dado que el equipo es pesado se debe mover nicamente mediante un elevador 4 Riesgo de descarga el ctrica iRiesgo de descarga el ctrica Con el m dulo del ventilador quitado se obtiene acceso a las clavijas de alimentaci n desde dentro de la cavidad del m dulo NO introducir herramientas ni partes del cuerpo en la cavidad del m dulo del ventila dor 5 Sobretemperatura No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adec uada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 6 Apilamiento del chasis A IN Los chasis no se deben apilar sobre otros equipos La caida del chasis podria causar lesiones corporales asi como da os al equipo 104 Mellanox Techn
102. rt This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions 1 Confirm that the location of the connector in the chassis lines up with the connector in the fan module 2 Slowly slide in the new leaf fan module Ifthe fan module stops before it goes in all ofthe way it is inserted incorrectly 3 Push the fan module until the latch engages 4 Make sure that the green fan LED on the module comes on indicating that fan is running 2 9 4 2 Spine Fan Modules Each spine has 2 individual fan modules that contain 2 individual fans each Should a single fan fail the fan status LED on the spine and the fan status LED on the management module light up indicating the necessity to replace the fan module Airflow through the spines is independent of the airflow through the leafs Figure 47 Spine Fan Modules 19 22 25 2831 34 When a fan module is removed the indicator light will reset Figure 48 Fan Status LED on the Spine Module Fan LED 1 Fan LED 2 Extracting the Spine Fan Module To extract the spine fan module push the blue latch button while pulling the fan module out Mellanox Technologies 55 Rev 1 0 Installation 2 9 5 Inserting the Spine Fan Module When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment m
103. s side to allow room for cable management Figure 20 Placement of Chassis in Rack Top View Rack doors Leaf side Chassis 4 Spine side Rack doors Mellanox Technologies 35 Rev 1 0 Installation Step 10 Put the crate on the fork lift or a mechanical lift Figure 21 Fork Lift Wooden floor Hinged extension Figure 22 Mechanical Lift Step 11 Position the chassis in front of the installation rack Figure 23 Step 12 Unscrew both lock down bars The chassis is on ball bearings and can roll easily Beware that the chassis can roll off A of the pallet possibly causing grave bodily harm should it fall on someone 36 Mellanox Technologies Rev 1 0 Step 13 Using a fork or mechanical lift position the middle of the leaf side of the chassis in front of the spine side of the rack Figure 23 Figure 23 Rack and Chassis Alignment LEAF SIDE SPINE SIDE RACK BRACKET WELD Step 14 Place the rotated strip on top of the bracket weld Figure 24 Figure 24 Rotated Strip and Bracket Weld ROTATED STRIP ON TOP OF BRACKET WELD BRACKET WELD Step 15 Unscrew the lock down bar facing the rack Note that the chassis is on ball bearings and with the lock down bars removed the chassis can roll off Mellanox Technologies 37 Rev 1 0 Installation Step 16 Slide the switch system chassis half way into the rack Figure 25 A Beware not to injure
104. spojite sva napajanja kako biste osigurali stanje bez napajanja unutar platforme preklopnika 9 Tijekom udara munje Opasnost od elektri ne energije A Tijekom djelovanja munja nemojte raditi na opremi ili spajati ili odspajati kabele 10 Spajanje Odspajanje bakrenog kabela InfiniBand Bakreni kabeli InfiniBand su teski 1 nesavitljivi i kao takvi se moraju pazljivo prikl juciti na ili iskljuciti iz konektora Obratite se proizvoda u kabela za posebna upo zorenja upute 11 MontaZa ormarica i servisiranje opreza kako bi se osiguralo da sustav ostane stabilan Op enito trebali biste ispunja Kad se proizvod montira ili se servisira u ormaricu moraju se poduzeti posebne mjere vati ormari s opremom po ev i od dna prema vrhu 12 Instaliranje opreme Ovu bi opremu trebalo instalirati zamjenjivati i ili servisirati samo obu eno i kvalifici rano osoblje Mellanox Technologies 115 Rev 1 1 14 Lokalni i nacionalni elektri ni kodovi Ova oprema trebala bi se instalirati u skladu s lokalnim i nacionalnim elektri nim kodovima 15 Instalacijski kodovi A Ovaj se ure aj mora instalirati sukladno najnovijoj verziji nacionalnih elek MA tri nih kodova drZave U Sjevernoj Americi oprema se mora instalirati suk ladno va e im zahtjevima navedenim u US National Electrical Code i Canadian Electrical Code 16 UL CSA kabel napajanja IN Za sjevernoameri ku mre u odaberite kabel napajanja k
105. stallation in a Restricted Access Location 5 Risk ofElectric Shock Risk of Electric Shock With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity 6 Over temperature This equipment should not be operated in an area with an ambient temperature exceed A ing the maximum recommended 40 C 104 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 7 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can 4 cause bodily injury and equipment damage 8 Redundant Power Supply Connection Electrical Hazard This product includes a redundant power or a blank in its place In case of a blank y power supply do not operate the product with the blank cover removed or not securely fastened 9 Double Pole Neutral Fusing This system has double pole neutral fusing Remove all power cords before opening AA the cover of this product or touching any internal parts 10 Multiple Power Inlets Risk of electric shock and energy hazard 4 The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 11 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 18 Mellanox Technologies
106. stem has a dedicated AC inlet The inlets are numbered 1 through 10 corresponding to the numbers on the PSU covers The system utilizes C19 C20 AC inlet standard where C20 socket is installed on the system A C19 to C20 power cord is provided with the sys tem An appropriate capacitor discharge unit CDU should be installed to interface with the sys tem 3 1 3 Leaf Module LED Indicators Figure 54 Leaf Module LED Indicators Leaf Status LED Bad Port LED UID LED 3 1 3 1 Status LED The Status LED is located in the middle of the leaf to allow easier visibility when all cables are installed 60 Mellanox Technologies Rev 1 0 The 1081 Status indicator LED has the following LED assignment Table 7 Leaf Status LED LED Condition Description Off No power to the leaf Green Solid Leaf is up and running Green Flashing Leaf is booting or being restored to factory default Amber Solid Fatal error 3 1 3 2 Bad Port LED The Bad Port LED is located on the left side of the leaf The following Bad Port LED conditions are possible Table 8 Bad Port LED Configurations LED Condition Description Off OK All ports are up and running Amber Flashing Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are badcable badconnection bad connector This LED lights up when one or more ports is receivi
107. symbol errors recently Amber Flashing Error One or possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are badcable badconnection e bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 3 1 4 5 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support 3 1 5 Spine Side Panel Display LED Indicators The spine side panel display has LEDs that show the chassis condition Figure 56 Spine Side Panel Display Status Indications m EJ m m A e o Status PSU Status MNG2 Leaf Fan Spine Fan Master Status Status Status MNGI Master Status Table 14 Spine Side Panel LED Display for Normal Operation LED Condition Description Normal State Status Off No power Green Green System is up and running Green Flashing System booting restore factory default in progress Amber Fatal error 64 Mellanox Technologies Rev 1 0 Table 14 Spine Side Panel LED Display for Normal Operation PR AVA Normal LED Condition Description State PSU Status Off No power Green Green Normal operational Amber PS fault detected check individual power supplies for fault indicati
108. t certifi CSA Pour le branchement lectrique en Am rique du Nord utiliser un cordon d alimenta tion UL Listed et CSA Certified 3 conducteurs calibre 16 AWG avec une prise moul e 125 V 13 A faisant au moins 1 5 m de long six pieds et au plus 4 5 m Pour le branchement lectrique en Europe utiliser un cordon d alimentation au format international harmonis marqu lt HAR gt 3 conducteurs d au moins 1 mm2 de sec tion 300 V avec une gaine isolante en PVC Le cordon doit avoir une prise moul e 250 V 10 A 17 Courant de fuite lev Avertissement courant de fuite lev une connexion la terre est indispensable avant de brancher l alimentation Mellanox Technologies 99 Rev 1 1 18 Interconnexion des unites A Les c bles de connexion aux interfaces RS232 et Ethernet de l appareil doivent tre certifi UL de type DP 1 ou DP 2 Note en cas d installation sur un circuit dont la puissance n est pas limit e LO Protection contre les surintensit s le c blage de l immeuble doit int grer un dispositif certifi de protection contre les surintensit s calibr 20 A et ais ment accessible 19 Ne pas utiliser comme tag re ou plan de travail Attention un quipement coulissant ou mont sur rail ne doit pas servir d tag re ni de plan de travail Les rails ne sont pas destin s faire coulisser l unit hors du rack Ils sont destin s une installation permanente
109. t si sau depanat numai de c tre personal instruit si calificat 13 Eliminarea echipamentului Eliminarea acestui echipament trebuie s se realizeze in conformitate cu toate legile si regulamentele nationale 112 Mellanox Technologies Rev 1 1 15 Codurile ed instalare Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor electrice nationale ale rii n cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerin elor aplicabile din Codul electric na ional al SUA i Codul electric canadian 16 Cordon de alimentare electric nregistrat UL i certificat CSA A Pentru conectarea la o surs de alimentare pentru America de Nord selectati un cordon de alimentare care este nregistrat UL si certificat CSA cu 3 conductoare 16 AWG terminat cu o fis turnat cu putere nominal egal cu 125 V 13 A cu o lungime de minimum 1 5 m sase picioare dar nu mai lung de 4 5 m Pentru conectarea la o sursa de alimentare in Europa selectati un cordon de alimentare care este armonizat la nivel international i marcat lt HAR gt cu 3 conductoare cu minimum 2 fire de 1 0 mm cu putere nominal egal cu 300 V i cu o manta izolant din PVC Cordonul de alimentare trebuie s fie prev zut cu o fi turnat cu putere nominal egal cu 250 V 10 A 17 Curent de scurgere de nalt frecven Avertisment Curent de scurger
110. tema resti stabile In generale il rack va riempito con apparecchiature procedendo dal basso verso l alto 12 Installazione dell apparecchiatura N Questa apparecchiatura va installata sostituita e o sottoposta manutenzione solo da N pp p N personale addestrato e qualificato a Lo smaltimento di questa apparecchiatura va effettuato in conformit con tutte le leggi e le normative nazionali 14 Codici elettrici locali e nazionali AB Questa apparecchiatura va installata in conformit con le norme elettriche locali e N 5 A nazionali 15 Codici di installazione y Questo dispositivo va installato in conformit con l ultima versione dei codici elettrici A nazionali del Paese Per il Nord America l apparecchiatura va installata in conformit con i requisiti applicabili del codice elettrico nazionale USA e del codice elettrico q pp canadese 16 Cavo di alimentazione UL e munito di certificazione CSA Per una connessione di alimentazione nordamericana selezionare un cavo di alimentazione di tipo UL e munito di certificazione CSA a 3 conduttori 16 AWG terminato con una spina stampata con tensione nominale pari a 125 V 13 A di lunghezza minima pari a 1 5 m sei piedi ma non pi lunga di 4 5 m Per una connessione europea selezionare un cavo di alimentazione armonizzato a livello internazionale e contrassegnato da lt HAR gt a 3 conduttori minimo 1 0 mm2 fili
111. threads A ground wire of AWG 6 or Amm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only con nect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the connections are solid and permanent If 44 Mellanox Technologies Rev 1 0 you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack A Warning System grounding must comply with local electrical code Figure 34 Grounding Posts Grounding post Grounding post 2 7 Power Connections The switch includes 10 hot swap power supply units PSUs Each PSU has its dedicated AC inlet The system uses C20 type inlets The design enables optional use of separate main and backup AC feeds The AC power source should comply with 180 265VAC 50Hz or 60Hz charac teristics C19 to C20 type power cords 15A rated should be used to connect system to AC power The maximum power consumption of each power supply is 12A Ensure all 10 power cords are firmly connected to inlets on the one side and to the AC source on the other Figure 35 AC Inlets Make sure that the power cords are compatible with your outlets Power cords for dif ferent countries can be ordered from Mellanox Make sure that the outlets and c
112. til it is 45 degrees from the leaf Pull out the module half way through the guiding rails using the ejector handles Re lock the ejector handles Hold the body of the leaf on both sides and remove it from the chassis A The module is short therefore do not let go of it while sliding it out 2 9 2 2 Inserting a Leaf Module When hot swapping any of the units it is necessary to wait 1 minute after removing A the defective part before inserting the new part This is necessary so that the manage ment module will start a new cycle checking through the leafs and spines for the FW versions To insert the leaf module 1 Check for foreign objects in or mechanical damage to the chassis leaf slots 2 Check leaf mechanics for any noticeable damage 3 Check back signal connectors integrity Look for any broken signal dividers or any devia tions from the pass criterion shown in Figure 39 Figure 39 Intact Leaf Signal Connectors Mellanox Technologies 49 Rev 1 0 Installation 4 Check power connector s integrity Look for any damage on the power connector casing or blades damage Figure 40 Intact Leaf Power Pins 5 Start with the ejector handles fully open that is at 45 degrees to the front panel of the leaf 6 Holding the leaf by its sides carefully set the leaf module into the chassis 7 Applying equal pressure on both sides of the leaf module slowly slide the module into the chassis until the eje
113. tive all alimentazione per la Norvegia y Questa apparecchiatura progettata esclusivamente per il collegamento a un sistema di N alimentazione TN e a un sistema di alimentazione IT 120 Mellanox Technologies Rev 1 1 G 12 Montaj G venlik Uyarilari T rkge 1 Montaj Talimatlar Ekipman g c kaynag na baglamadan nce t m montaj talimatlar n okuyun 2 A rl k Nedeniyle Fiziksel Yaralanma Bu r n g venli bir ekilde kald rabilmek i in yeterli say da insandan yard m al n 40 Ibs 40 70 Ibs 70 121 lbs lt lbs 18 kes 18 32 kc3 32 55 kgs gt 55 kge 3 A r Ekipman Bu ekipman cok agirdir ve yaralanmalar nlemek igin ekipman n mekanik asans r kullan larak tas nmas gerekir 4 Elektrik Carpmasi Riski Bu ekipman nerilen maksimum ortam s cakl n a an alanlarda calistirilmamalidir 45 C 113 F Ayr ca d zg n hava ak sa lamak i in havaland rma deliklerinin etraf nda en az 8 cm 3 in a kl k b rak lmal d r 5 s nma y Bu ekipman nerilen maksimum ortam s cakl n a an alanlarda calistirilmamalidir A 45 C 113 F Ayrica d zg n hava ak s saglamak i in havaland rma deliklerinin 4 etraf nda en az 8 cm 3 in a kl k b rak lmal d r 6 Sasi Istif 4 Sasinin diger herhangi bir ekipman n zerine istiflenmemesi gerekir Sasi d serse fiziksel yaralanmalara ve ekipmanda hasara nede
114. tput 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present Mellanox Technologies 83 Rev 1 0 Table 21 InfiniBand QSFP Connector Pin out Connector Pin Connector Signal Description Pin Name Number 28 IntL Interrupt 29 Vec Tx 3 3 V Power supply transmitter 30 Vec 1 43 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground Figure 64 InfiniBand QSFP Connector Symbol J3 and J6 20 GND 1 _ 21 Rx2n Rx1n 18 22 Rx2p Rxtp 17 23 GND GND 16 _ 24 15 Rx4n Rx3nm 14 Rx4p Rx3p 13 2 E GND 27 ModPrsL spa 2 11 m scL gt 10 VccTx Vcc 30 Vcc1 ResetL 9 31 LPMode ModSelL 3 GND GND 7 33 Tx3p Tx4p 34 Tx3n Tx4n 9 35 GND 4 36 Tx2p 3 mon Tx2n 2 38 GND 1 84 Mellanox Technologies Rev 1 0 Figure 65 QSFP Connector Male and Female Views View into Rear of Connector
115. washer 15 12 Washer spring 10 6 8 5 pan head screw 16 18 M6X25 pan head screw 7 12 M6X16 pan head screw 17 4 Shelf rear arm bracket 8 30 M6 cage nut 18 2 Support bracket 9 2 Perforated fixing bracket 19 12 Washer 10 10 8 M6X16 hex head screw a Use the item number to identify the corresponding item in Figure 8 Each item number appears in a call out bubble 26 Mellanox Technologies Rev 1 0 Figure 8 Chassis Installation Kit 27 Mellanox Technologies Rev 1 0 Installation Cable Management Installation Kit Table 6 Cable Management Installation Kit Parts Item No a Quantity Description Item No Quantity Description 1 1 Left riveted cable holder 3 14 Pan head screw 2 1 right riveted cable holder 4 14 M6 cage nut a Use the item number to identify the corresponding item in Figure 9 Each item number appears in a call out bubble Figure 9 Cable Management Installation 8 Mellanox Technologies Rev 1 0 Shelf Installation Kit The shelf installation kit includes 18 cable supporters only Figure 10 Shelf Installation Kit Mellanox Technologies 29 Rev 1 0 Installation 2 5 2 Installation Procedure Step 1 Insert all the cage nuts necessary for the following steps of the installation procedure in the rack a Insert 3 M6 cage nuts from the inside of beam 1 and 2 meant to support the shelf Figure 11 v It is recommended to set th

648-Port EDR InfiniBand Switch-IB™ Switch Platform Hardware

Contents

Download Pdf Manuals

Related Search

Related Contents

648-Port EDR InfiniBand Switch-IB&trade; Switch Platform Hardware

Contents

Download Pdf Manuals

Related Search

Related Contents

648-Port EDR InfiniBand Switch-IB™ Switch Platform Hardware