Home

SwitchX 36-Port QSFP FDR InfiniBand System Hardware User Manual

image

Contents

1. 22 IN B 23
2. Mellanox Technologies 103 Rev 2 2 27 WEEE WEEE 2002 96 EC PPAR SRL ERES TA EEE EDR MAETER RRERFEHN ERE HUA ALEE FE MAS ES 28 AE PRE HER MEAN TN IT 104 Mellanox Technologies
3. Mellanox Technologies 91 Rev 2 2 14 H 17 AN C AR
4. lt HAR gt 3 c 1 0 MM 300 250 10 92 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 20 TOK TOK 21 IN A
5. 4 2 QQ uw 40 lbs 40 70 Ibs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 4 5 A CHAT N B
6. 18 UL 4 19 UL CSA f B UL CSA 3 16 AWG 1 5 6 4 5 125 13
7. RS232 Ethernet DP 1 DP 2 UL 20 24 1 IEC 60 825 1 1993 A1 1997 A2 2001 u EN 60825 1 1994 A1 1996 A2 2001 25
8. H 10 InfiniBand
9. Mellanox Technologies 93 Rev 2 2 26 27 WEEE 2002 96 EC WEEE 94 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix K Avertismente privind siguranfa la instalare Romanian 1 Instruc
10. 49 3 9 Internally Managed Switch Configuration 49 Chapter un AA Rao e aa 56 4 1 Cable Installation 56 4 2 Splitting the Data Stream Using Breakout Fanout Cables 57 Chapter 5 Switch Management 60 5 1 InfiniBand Subnet Manager 60 5 2 Fabric Inspector Diagnostics 60 3 3 Upgrading Software ss lA RR pU T p AERE 61 Mellanox Technologies 3 Rev 2 2 5 4 Updating Firmware Internally Managed Switch 61 5 5 Updating Firmware Externally Managed Switch 61 Chapter 6 Troubleshooting ran sie eres nen nennen 64 Appendix A Specification oo 67 A1 Approved Cables fod os 68 2 EMC Certifieations ye ee oe En 68 China CCC Warning Statement 69 Appendix B Thermal Threshold Definitions 70 Appendix C QSFP Interface eee eue nn er ann 71 Appendix D RJ 45 CONSOLE and DC Interface 73 Appendix Accessory and Replacement Parts 74 Appendix Disassembly of the Switch from the Rack 75 E l Disassembly Procedure ame 75 E2 Disposal A en ea eet a 75 App
11. 6 A A 45 C 113 F A N roro He 8 cM 3 90 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 7 9 Ha
12. 15 Table 4 Management Software 16 Table 5 Switch Management 18 Table 6 Systemstatus LEDS o wee A ee da ne A wee 20 Table 7 Switch Status LED Assignments 22 Table 8 Fan Status LED Assignments 23 Table 9 PS Unit Status LED Assignments 25 Table 10 Bad Port LED Assignments 26 Table 11 Port Connector Physical and Logical Link Assignments for Ethernet Mode 27 Table 12 Connector Physical and Logical Link Assignments for IB Mode 27 Table 13 Air Flow Direction ee IRR RR 9h ehe 29 Table 14 Installation Kit According to Rack Size 38 Table 15 Serial Terminal Program Configuration 50 Table 16 Configuration Wizard Session DHCP Configuration Example 51 Table 17 Configuration Wizard Session Static IP Configuration 52 Table 18 Configuration Wizard Session Zeroconf Configuration 53 Table 19 Port Splitting Options 58 Table 20 SX60 25 36 Specification Data 67 Tab
13. BEBBEBNE SX6025 BA W324 5A We LE BE S E NB a WA CONSOLE A2 8 8 u u S N MT1117X00014 A NN P N MSX6036F 1SFR UTI AMA GUID 0002C9020040DDB0 Rev B6 LL LL M Made in China im x 322X6519220002 N Pull out tab 1 3 FDR Transmission Rate Mellanox switch systems support FDR fourteen data rate an InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625 Gb s with a 64b 66b encoding resulting in an effec tive bandwidth of 56 25 Gb s The FDR physical layer is an IBTA specified physical layer using advanced block types deskew mechanism and framing rules for supporting higher speeds and increased reliability The switches also supports FDR 10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125 Gb s with a 64b 66b encoding resulting in an effective bandwidth of 40 Gb s FDR10 supports 20 more bandwidth over QDR using the same QSFP cables connectors Both FDR and FDR10 support Forward Error Correction FEC as described in IEEE 802 3ap chapter 74 FDR 10 are only guaranteed to work with approved Mellanox Cables Mellanox Technologies 17 Rev 2 2 Overview gt FDR10 is only guaranteed to work with approved Mellanox ConnectX 3 adapters A 1 4 Internally Managed vs Externally Managed All internally managed switches can support IB fabric of up to 648 nodes Internally managed switches n
14. License required This feature will be available to customers in the near future Mellanox Technologies 15 Rev 2 2 Overview Table 4 Management Software Feature Description Management Software MLNX OS Software Management Dual software image Software and firmware updates File management FTP TFTP SCP Logging Event history log SysLog support Management Interface DHCP Zeroconf IPv6 Chassis Management Monitoring environmental controls Power management Auto temperature control High availability Network Management Inter faces SNMP vl v2c v3 REST interfaces XML Gateway Security SSH Telnet RADIUS TACACS Date and Time NTP Cables amp Transceivers Transceiver info Virtual Port Interconnect VPI Ethernet InfiniBand 16 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 1 2 Serial Number and Product Version Information The serial number GUID and MAC for the switches are found on the pull out tab on the right side of the Connector side panel below the USB connector On managed switches only the MACs for the Management PC are also placed on this tab This MAC is for the mgmt0 manage ment interface the top Ethernet port The second interface mgmt has the next consecutive MAC Externally managed switches do not have a MAC on the tab Figure 2 Pull Out Tab
15. which ports can be split into 4 which ports can be split into 2 which ports will be unmapped due to a 1X4 split Mellanox Technologies 57 Rev 2 2 Cabling When using this feature you MUST go into the MLNX OS chassis management sys tem and reconfigure the individual ports to split 2 or split 4 See the MLNX OS CLI User Manual for instructions on port configuration Table 19 Port Splitting Options Port Can be split Turnsoffport Can be split Port Can be split Turns off port Can be split to 4 to 2 to 4 to 2 1 19 2 20 Y 3 21 Y 4 Y 1 22 Y 19 5 Y 6 23 Y 24 6 24 7 25 8 26 9 Y 12 27 Y 30 10 Y 7 28 Y 25 11 29 12 30 13 31 14 SZ 13 32 Y 31 15 Y 18 33 Y 36 16 34 17 35 Y 18 36 58 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 39 3 Examples of Port Mapping Assignment 64 X10 GbE DO NOT USE THESE PORTS x 12 x 40GbE ports blocked 00 10GbE ports 0000 12 x 40GbE ports split to 4 10GbE ports 48 10GbE ports 4 x 40GbE ports split to 2 10GbE ports 8 10GbE ports 8 x 40GbE ports used as 1OGbE ports 8 10GbE ports Total 64 x 10GbE ports 56 x 10GbE and 8 x 40GbE
16. Care should be taken not to impede the air flow through the ventilation holes next to the connec tor ports Cable lengths should be used which allow for routing horizontally around to the side of the switch before bending upward or downward in the rack Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 4 2 Splitting the Data Stream Using Breakout Fanout Cables Only internally managed switches support splitting of the data stream 5 Specific ports can be split by using a QSFP 1X4 breakout cable to split one 40 GbE port into 4 lanes 4 SFP connectors 10 GbE each These 4 lanes then go one lane to each of the 4 SFP connectors Some ports can be split into 2 10 GbE ports using lanes 1 and 2 only When a QSFP port is split into 2 10 GbE ports then only SFP connectors 1 and 2 are used Connectors 3 and 4 are left unconnected Using Figure 38 and Table 19 with breakout cables you can achieve 64 ports of 10 Gb s Figure 37 Breakout or Fanout Cable mu This port can be split into 4 10 Gb s SFP mmm This port can be split into 2 10 Gb s SFP x This port is unmapped according to the Port Splitting Table The maximum 10 Gb s Ethernet ports configurable with this switch is 64 When using a port to split a data stream into 4 10 Gb s data streams four lanes one of the other ports on the switch will be disabled unmapped See Table 19 to determine
17. Rev 2 2 14 15 RERE SRI Rais IS AR 16 17 18 UL ATEN AmA RE REB Swe TER A SE IBI ES 19 UL CSA RE RS BRR A ALE EEE LE UL CSA ZE 16 AWG 125 V 13A 1 5 AR 8845 AAA lt gt Stk FARM 1 0 300 V KA PVC 250 10 Ao 20 7 AREA RR 102 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 21 8138 GND EHEM
18. Rev 2 2 Switch Management 2 Compare the results of this command with the latest version for your switch posted on the www mellanox com gt Support gt Firmware Downloads gt SwitchX SwitchX 2 Switch System page 3 If the current version is not the latest version follow the directions in the MFT User manual to burn the new firmware inband 4 2 Connector I2C connection provides access to Flash EEPROMs This connection is for FAE use only F The externally managed switches are Plug and Play and all firmware updates should be done in band This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only 5 5 2 Reprogramming the Switch Through The 12C Port This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only The I2C connector should only be used when the firmware cannot be updated in band Use the MTUSB 1 adapter available from Mellanox to reprogram the switch firmware using a server or PC running MFT Figure 40 MTUSB 1 with Cables 62 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 The MTUSB adapter comes with three cables but has only two connections for cables One cable is an I2C cable with DB9 female connectors on both ends One cable is a I2C cable with DB9 female connector to RJ 45 male Use the cable that matches your switch The
19. Table 7 Switch Status LED Assignments LED Configuration STATUS System Health LED Solid Green OK The system is up and running Flashing Green The system is booting up This assignment is valid on managed systems only Solid Yellow To be implemented in a future SW release Solid Red Major Error Possible damage can result to the switch Turn off immediately e g bad firmware can t boot overheated If the system is booting up it can take up to 5 minutes for the status led to change to green Off Off The system has no power 2 2 3 Fan Status LED Y N All fans must be operating while the power supply is plugged in A IN Ifthe switch shuts down due to over temperature unplug the switch wait 5 minutes 4 and replug in the switch For more information See page 64 22 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 7 Fan Status LED Connector Side The LED indicator on the Fan Module is on the right side of the module Figure 8 Fan Status LED Power Side This is a two color LED Table 8 shows the fan status LED assignment Table 8 Fan Status LED Assignments LED Configuration FAN LED Solid Green OK All fans are up and running Solid Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan module Off Off The fan unit is not
20. Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch gt 5 Before attempting a remote for example SSH connection to the switch check the mgmt0 interface configuration Specifically verify the existence of an IP address To check the cur rent mgmt0 configuration enter the following commands switch gt enable switch configure terminal switch config show interfaces mgmt0 The following is an example of the output Interface mgmt0 state Admin up yes Link up yes IP address 192 168 10 43 Netmask ANNA Speed 1000Mb s auto Duplex full auto Interface type ethernet Interface source physical MTU 1500 EW address 00 02 C9 11 2A AE Comment Mellanox Technologies 53 Rev 2 2 Installation RX bytes 1343502058 TX bytes 313920869 RX packets 17589211 TX packets 992 Tn RX mcast packets 0 TX discards 0 RX discards 0 TX errors 0 RX errors 0 TX overruns 0 RX overruns 0 TX carrier 0 RX frame 0 TX collisions 0 TX queue len 1000 Confirm that the latest SW revision is installed on this device Use the CLI to get the currently installed revision and then go to Mellanox com to check the revision of the latest version 6 Run the command show version 7 Compare the results of this command with the latest version for your switch posted on the su
21. EE 5 AEN EH VERUS LAB E 88716 T B dL LES EN 6 MA 45 C 113 F 8 3 100 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 7 a E THEN AREAS o 8 RE ES A PE aa ZE SE rn EUR L gt KARAA SEA PHM ARAS RR FITI BIZ E mH ei CE AN SBS HF e gt 10 PSU gt TEES SHE TERRA NE RER IK FAR 12 InfiniBand HAER KT InfiniBand 845 ARAM 2 o SE To E S EDGE ERR EA JET FRHERRER 13 BRRR Ba MERE Mellanox Technologies 101
22. These switches can come with two air flow patterns The two patterns are Connector side inlet to power side outlet Power side inlet to connector side outlet Mellanox Technologies 27 Rev 2 2 Basic Operation The air flow is specified in the product model number See Mellanox Part Numbering Legend on page 12 On the power side panel the PS unit and fan modules air flow direction can be seen All compo nents need to have the same air flow A mismatch in the air flow will affect the heat dissipation All PS units must have the same air flow as the fan module Figure 15 Component Air Flow T i i In niii Wis These air flow labels must be the same Red is connector side inlet to power side outlet ee LL Ll ues A A El LL Blue is power side inlet to connector side outlet Figure 16 shows a mismatch in component airflow Figure 16 Component Airflow Mismatch Wrong me T UEU Inm ea These air flow labels must be the same Any mismatch of these labels will affect the switch temperature Table 13 shows the icons that indicate airflow using arrows and colors 28 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 2 5 2 6 Table 13 Air Flow Direction All components need the same colored airflow icons OP
23. GND RE RAY 75 Be COR BEAR ER IT o 22 gt gt 23 HERE HS RS232 UL ARAE 1 DP 2 LPS 20 A 24 44 ut N RAN m CLASS 1 SRRA 8112 IEC 60 825 1 1993 A1 1997 A2 2001 EN 60825 1 1994 A1 1996 A2 2001 E m Rin EE B A ARAS NEAR A LO 26
24. The Power supply is delivering the correct voltage 12VDC Solid Red Error The PS unit is not operational Off Off There is no power to the system neither PS unit is receiving power If one PS unit is showing green and the second PS unit is unplugged it will show a red indication 2 2 5 Bad Port LED Figure 12 Bad Port LED The Bad Port indicator is located on the left side of the connector side panel of the unit Table 10 shows the bad port status LED assignment Mellanox Technologies 25 Rev 2 2 Basic Operation 2 2 6 2 3 Table 10 Bad Port LED Assignments LED Configuration Description Off OK No symbol errors have been received in last few seconds Flashing Orange Error One or possibly more ports have just received a symbol error Possible causes are Bad cable Bad connection Bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received UID LED Switch Identifier The UID LED is a debug feature that lights a blue LED on the switch connector side panel for ease in finding a particular switch within a cluster Figure 13 Identifier LED Port Connector LEDs Above the downlink ports are two LEDs one for the upper port A and one for the lower port V Table 11 shows the port status according to the LED indication for Ethernet protocol Table 12 shows the port stat
25. U Boot 2009 01 mlnx1 4 date GB UE AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No I2C ready DRAM 2 GB ECC enabled 400 MHz CL3 FLASH 16 MB AND 1024 MiB ACIER Bus Dev Venld DevId Class Int PCIEO link is not up 1 successfully set as root complex 01 00 1553 bd34 0 06 00 et ppc_4xx_eth0 ppc_4xx_ethl Hit Ctrl B to stop autoboot 0 ellanox MLNX OS Boot Menu Mellanox Technologies 65 Rev 2 2 Troubleshooting 1 SX 60 3 2 0000 dev HA 2013 04 10 12 02 49 ppc 2 SX M460EX 3 2 0000 dev HA 2013 04 11 14 05 39 ppc 3 U Boot prompt Choice 5 Select the image to boot 66 Mellanox Technologies Table 20 SX60 25 36 Specification Data SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix A Specification Physical Size SX60 25 36 B 1 716 IU Hx 16 85 W x SX6036S 1 716 1U H x 16 85 W Size short 16 8 D Standard iie x24 75 D 43 6mm X 427 9mm X 428 9mm long 43 6mm X 427 9mm X 628 9 mm Weight Center of Grav Short 7 08kg 1 PS unit ity As measured from the Connector 7 82kg 2 PS units side left side bottom corner Standard Wna i 8 02kg 1 PS unit Short 21 6mm X
26. CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DUCTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 22 C digos de instalaci n A Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 23 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado del edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una corriente nominal de 20 A 24 Exposici n a radiaci n peligrosa Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se
27. Ce produit est quip d une alimentation redondante ou d un cache si elle est absente Dans ce dernier cas ne pas faire fonctionner le produit si le cache est retir ou mal fixe gt 9 Fusibles phase neutre Ce systeme dispose de fusibles phase neutre D branchez tous les cordons d alimenta tion avant d ouvrir le capot ou de toucher tout l ment l int rieur gt 10 Plusieurs prises d alimentation Risque et danger d lectrocution Les alimentations sont toutes ind pendantes gt Pour s assurer que le commutateur est bien hors tension d branchez toutes les alimen tations 11 En cas d orage danger d lectrocution Pendant un orage ne pas travailler sur l quipement ni brancher ou d brancher des c bles gt 12 Connexion et d connexion du cable InfiniBand en cuivre Les cables InfiniBand en cuivre sont lourds et peu flexibles Par cons quent il faut proc der avec soin pour les brancher ou les d brancher des connecteurs Consulter le fabricant du c ble pour obtenir des instructions ou des avertissements sp cifiques 13 Montage en rack et maintenance ment attention pour s assurer que l ensemble reste stable En r gle g n rale le rack doit Lors du montage ou de la maintenance de ce produit dans un rack il faut faire sp ciale A tre rempli en commengant par le bas Mellanox Technologies 77 Rev 2 2 14 Installation de l quipement Cet quipement ne doit tre
28. China RoHS statement If anything is damaged or missing contact your customer representative immediately For customer support go to www mellanox com gt Support gt Customer Support Portal Login 3 4 Mechanical Installation Tools and Customer Supplied Parts Phillips Screwdrivers 1 and 2 Grounding screw ESD strap Grounding wire sufficient to reach a valid ground ESD mat For racks from 60cm to 80cm deep either use the standard depth switches with the long rail kit or the short switches with the long rail kit Parts included in the rail kit 2 rails 16 recessed flat head screws You will have extras 2 rail slides 10 caged nuts 2 switch slides 10 pan head screws M6 Mellanox Technologies 39 Rev 2 2 Installation Figure 22 Rack Rail Kit Parts Switch slide x2 Rail x2 Rail slide x2 aaa TTTTT Ia 2222 od 13 11111 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the rack vertical support Either the side with the power supply units or the side with the QSFP connectors can be even with one ofthe vertical rack supports 40 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 23 Placement of Sw
29. hors du rack Ils sont destin s a une installation permanente a l emplacement final pas pour l entretien ni la maintenance 27 Directive DEEE Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doivent tre collect s s par ment et ne pas tre mis au rebut avec les d chets m nagers habituels Ce produit et toutes ses pi ces doivent tre mis au rebut d une mani re responsable respectant l environnement 28 Restrictions concernant l alimentation pour la Norv ge Cet appareil est pr vu pour tre reli a un syst me d alimentation TN et un syst me d alimentation informatique de Norvege uniquement 80 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix H Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 Verletzungsgefahr wegen des Gewichts A Um das Produkt sicher anzuheben gen gend Personen einsetzen lt 40 lbs 40 70 Ibs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Schweres Ger t Dieses Ger t ist sehr schwer und muss mit einem mechanischen Hebeger t verschoben werden um Verletzungen zu vermeiden 4 Installation an Bereich mit eingeschr nktem Zugang Diese Ger t ist f r die Installation in einem Bereich mit beschr nktem
30. 193mm X 209 8mm GHI 8 76kg 2 PS units 1PS unit 21 6mm X 214mm X 218 8mm 2PS units Standard 21 6mm X 201mm X 337mm 1PS unit 21 6mm X 214mm X 348mm 2PS units Mounting Mounting 19 Rack mount SerDes Speeds Connector SerDes Speeds 10 20 40 or 56 Gb s per port Types QSFP Connector Types Can use QSFP to SFP adapter modules Nemos Air Flow Heat dissipa ORAE 62 CFM tion 798 BTUs hr Heat dissipation Maximum Power and Environmental Input Voltage Input Voltage Management Management 100 240 VAC 50 60Hz CPU PowerPC 460EX CPU CPU Power Typ Typ 56 Gb s Consumption PS unit fan is always at 70 Active Cables 207W 5 7W port Passive cables 126W 3 5W port Power numbers Max Max 56 Gb s Cable power Temperature Shock and Vibration Humidity Active Cables Passive cables QSFP Shock and Vibration 234W 6 5 W port 129W 3 6W port MAX 2 0W TYP 1 5W ETSI EN 300 019 2 2 1999 09 Temperature Operating Non operating Humidity Operating 0 to 45 Celsius 40 to 70 Celsius 5 90 non condensing Mellanox Technologies 67 Rev 2 2 Protocol Support Power Supplies Speed protocol IB Management QoS Auto Negotiation of 56 Gb s Managed using MLNX OS Management 40 Gb s 10 Gb s 1 Gb s Data Rate Data Rate 56 Gb s Regulatory Compliance Safety US Canada cTUVus EMC USA FCC Class A EU IEC60950 Emiss
31. InfiniBand Architecture Specification Related Documentation Additional Documentation available from Mellanox Table 2 Reference Documents and Web Sites Document Name Description InfiniBand Architecture Specifica The InfiniBand Architecture Specification that is provided by IBTA tion Volume 1 Release 1 2 1 SwitchX 2 Switch Product Hardware For possible hardware issues see the switch support product page This Release Notes document can be found on the support web page for this product This requires a customer support login Look up the relevant SwitchX based switch system series release note file Mellanox MLNX OS SwitchX 2 This document contains information regarding configuring and manag Software User Manual ing Mellanox Technologies SwitchX 2 Switch Platforms MLNX OS Software Command Ref Command Reference Guide for MLNX OS listing all of the commands erence Guide available through MLNX OS with explanations and examples All of these documents can be found on the Mellanox Website at www mellanox com They are available either through the product pages www mellanox com Products gt Select your prod uct or through the support page with a login and password Conventions Throughout this manual the name SX60XX and the term switch are used to describe the 36 port QSFP 56 Gb s InfiniBandswitch unless explicitly indicated otherwise The following icons are used throug
32. OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2013 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale MLNX OS PhyX SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IBTM FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroXTM MetroDX ScalableHPC Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3410 SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Table of Contents Fable of Contents Ga so ee en nee RRR ORES AAA 3 Eist of Hours AA E 5 Dist Of Tabs ODE NE E RARA AER 7 R vision History AAA AAA Aa 8 About this Manual 10 Intended Audience 1 on 10 Related Docum
33. Pentru conectarea la o surs de alimentare in Europa selectati un cordon de alimentare care este armonizat la nivel international i marcat lt HAR gt cu 3 conductoare cu minimum 2 fire de 1 0 mm cu putere nominala egala cu 300 V si cu o manta izolanta din PVC Cordonul de alimentare trebuie sa fie prevazut cu o fisa turnata cu putere nominala egala cu 250 V 10 A 20 Curent de scurgere de inalta frecventa Avertisment Curent de scurgere de inalt frecventa Imp m ntarea este esential inainte de a conecta sursa de alimentare Mellanox Technologies 97 Rev 2 2 21 Ad ugarea de informatii privind legarea la pamant Inainte de a conecta acest dispozitiv la o linie electric de energie suruburile bornelor de protec ie de legare la p m nt ale acestui dispozitiv trebuie s fie conectate la priza de p m nt de protec ie din instala ia cl dirii Informa ii privind legarea la p m nt Instala ia cl dirii va asigura un mijloc de conectare la priza de p m nt iar echipamen tul va fi n mod permanent conectat la priza de p m nt de un agent de ntre inere UN AGENT DE NTRE INERE va verifica dac priza borna de ie ire prin interme diul c reia va fi alimentat echipamentul cu energie electric asigur o conexiune la priza de p m nt a cl dirii n caz contrar AGENTUL DE NTRE INERE va dispune instalarea unui CONDUCTOR DE PROTEC IE DE LEGARE LA P M NT de la borna separat de protec ie
34. SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 3 Over temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation gt openings 4 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 5 Double Pole Neutral Fusing This system has double pole neutral fusing Remove all power cords before opening the cover of this product or touching any internal parts 6 Multiple Power Inlets Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the switch platform 7 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 8 Copper Cable Connecting Disconnecting Copper cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warn ings instructions 9 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with When this product is mounted or serviced in a rack special precautions must be
35. Type C13 C14 ACC000500 Installation Kit for short switches in racks 40 60 cm deep MSX60 BKIT Installation Kit for short or standard switches in racks 60 80 cm deep MSX60 SKIT 74 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix F Disassembly of the Switch from the Rack F 1 Disassembly Procedure gt To disassemble the switch from the rack 1 Unplug and remove all connectors 2 Unplug all power cords 3 Remove the ground wire 4 Unscrew the 2 center bolts from the side of the switch with the bracket Support the weight of the switch when you remove the screws so that the switch does not fall 5 Slide the switch from the rack 6 Remove the rail slides from the rack 7 Remove the ten caged nuts F2 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way EEE Follow the instructions found in the Mellanox Web site at mellanox com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive Mellanox Technologies 75 Rev 2 2 Appendix G Avertissements de s curit pour l installa tion French 1 Instructions d installation Veuillez lire la totalit des instructions d i
36. Zugang vorge sehen 5 Stromschlagrisiko y Stromschlagrisiko Bei abgenommenem Ventilatormodul sind die Stromkontakte in der Modulvertiefung zug nglich Es diirfen KEINE Werkzeuge oder K rperteile in die Vertiefung des Ventilatormoduls gelangen 6 Ubertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der PA NN maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft Ah strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Mellanox Technologies 81 Rev 2 2 7 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terf llt kann es zu Verletzungen und Besch digungen an Ger ten fihren 8 Zweipolig Neutrale Sicherung Achtung Zweipolige bzw Neutralleiter Sicherung im Netzteil Netzstecker ziehen um sicher zustellen da keine Spannung am Ger t anliegt Entfernen Sie alle Netzkabel vor dem ffnen der Abdeckung dieses Produkts oder dem Ber hren der Innenteile 9 Mehrere Stromeing nge Risiko eines Stromschlags und Stomgefahr Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgeschalteten Zustand im Inneren der Switch Plattform sicherzustellen 10 Bei Gewitter Elektrische Ge
37. especifican podr an causar exposici n a niveles de radiaci n peligro sos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes TEC 60825 1 2007 03 y EN 60825 1 2007 25 Recinto adecuado El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen dio 88 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 26 No utilizar el conmutador como estante ni como espacio de trabajo Cuidado Equipos montados en deslizadores o rieles no se deben utilizar como estantes ni como espacio de trabajo La finalidad de los rieles no es deslizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o mantenimiento 27 Directiva WEEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos Al deshacerse de este producto y de todas sus partes h galo de una manera respon D sable y respetuosa con el medio ambiente Mellanox Technologies 89 Rev 2 2 Appendix J no npu ycraHoBke Russian 1
38. labelled CONSOLE must be connected to a local host PC This must be used the first time the switch is connected This must be done before any remote management is available Hook up the supplied harness cable HAR00028 from the connector labelled CONSOLE to the DB9 connector of the local host PC Figure 35 Console Port MU SA SX6036 CONSOLE WA T Connect the host PC to this port o Make sure to connect to the Console RJ 45 port of the switch and not to the Ether net MGT port No remote IP connection is available at this stage 2 2 Configure a serial terminal program for example HyperTerminal minicom or Tera Term on your host PC with the settings described in Table 15 Table 15 Serial Terminal Program Configuration Parameter Setting Baud Rate 9600 Data bits 8 50 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Table 15 Serial Terminal Program Configuration Continued Parameter Setting Stop bits 1 Parity None Flow Control None 3 Login from a serial terminal program as admin and use admin as password This starts the Mellanox configuration wizard 4 Go through the Mellanox configuration wizard Table 16 shows an example of a wizard ses sion Table 16 Configuration Wizard Session DHCP Sheet 1 of 2 Configuration Example Wizard Session Display Comments Mellano
39. receiving any power Check that the fan unit is properly and completely inserted Risk of Electric Shock Wn With the fan module removed power pins are accessible within the module cavity DO NOT insert tools or body parts into the fan module cavity Mellanox Technologies 23 Rev 2 2 Basic Operation 2 2 4 Power Supply Status LEDs Figure 9 Power Status LED Connector Side This switch comes with two power supply units for redundancy The SX60XX switch is available with one factory installed Power Supply Unit A second Power Supply Unit can be added to support hot swap ability and to add redundancy See Accessory and Replacement Parts on page 74 for ordering part numbers Figure 10 Power Side Panel PS unit 1 status LED WII Fan Unit Primary Power Supply Unit The primary power supply unit PS1 is located on the right side of the power side panel with PS2 on the left side Each PS unit has a single 2 color LED on the right side of the PS unit that indicates the internal status of the unit 24 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 11 PS Unit Status LEDs This is a two color LED Power supply fan air flow direction indicator Table 9 shows the power supply unit status LED assignment Table 9 PS Unit Status LED Assignments LED Color Status Solid Green OK
40. residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protective device rated 20 A must be incorporated in the building wiring 19 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser stan dards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 20 Proper Enclosure A suitable electrical mechanical and fire enclosure shall be provided by the end prod uct manufacturer and or the end user gt 21 Do Not Use the Switch as a Shelf or Work Space Caution Slide rail mounted equipment is not to be used as a shelf or a work space The rails are not intended for sliding the unit away from the rack It is for permanent installation at final resting place only not used for service and maintenance 2 Mellanox Technologies 37 Rev 2 2 Installation 22 WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way 23 Country of Norway Power Restrictions LA This unit is intended for connection to a TN power system and an IT powe
41. 0 0 0 255 255 255 0 Step 6 Default gateway for example 192 168 10 1 10 10 10 255 Step 7 Primary DNS server Step 8 Domain name Step 9 Enable IPv6 yes Step 10 Enable IPv6 autoconfig SLAAC on mgmt0 interface no Step 11 Admin password Enter to leave unchanged To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch gt 52 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 The table below shows an example of a Zeroconf wizard session Table 18 Configuration Wizard Session Zeroconf Configuration Wizard Session Display IP Zeroconf Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration yes Step 1 Hostname switch Step 2 Use DHCP on mgmt0 interface yes no Step 3 Use zeroconf on mgmt0 interface no yes Step 4 Default gateway For example 192 168 10 1 Step 5 Primary DNS server Step 6 Domain name Step 7 Enable IPv6 yes Step 8 Enable IPv6 autoconfig SLAAC on mgmt0 interface no Step 9 Admin password Enter to leave unchanged To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit
42. 2 2 Basic Operation 2 2 Status LEDs The System Status LEDs are located to the left of the connectors on the connector side panel and on the power side at the far right 2 2 1 System Status LEDs Both of these LEDs give identical information Table 6 System Status LEDs Symbol Name Description Normal Conditions A Switch Status LED Shows the health of the switch Solid green after 5 minutes Fan Status LED Shows the health of the fans Green SH Power supply 1 Shows the health of the right side power supply unit Green PS2 Power supply 2 Shows the health of the left side power supply unit Depends on switch Bad Port LED lights up when a symbol error is detected Off pin Unit Identifier LED Lights up on command through the CLI Off Figure 4 Externally Managed Switch System Power and Connector Side Panels Power Side Panel These two LEDs indicate the status of the switch as assigned in Table 7 on page 22 ESEEREEEEUNESGSHHENZSEHEEUHLnZSESESEHUEZSZHESEUSBEURNEHRSENESSESSSEEEHZEBEZEESUBEUERBEREREZEEERERNSHEES 14 V2 Va Yo 7 Ys 94 Viola Vi 134 Fis 154 Vic 174 Vis 194 7Wo 214 Vaz 234 254 W26 274 Vas 294 30 312 9 Wi 354 V36 Connector Side Panel 20 Mellanox Technologies Rev 2 2 SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Figure 5 Switch Status LED Both of these LEDs show the switch status Power side Connector side sta
43. 6 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 14 Instalarea echipamentului Acest echipament trebuie s fie instalat inlocuit si sau depanat numai de c tre personal instruit si calificat Eliminarea acestui echipament trebuie s se realizeze in conformitate cu toate legile si regulamentele nationale Acest echipament trebuie s fie instalat conform codurilor electrice locale si nationale 17 Codurile ed instalare Acest dispozitiv trebuie sa fie instalat in conformitate cu ultima versiune a codurilor electrice nationale ale tarii in cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerintelor aplicabile din Codul electric national al SUA si Codul electric canadian 18 Inlocuirea bateriei Avertisment nlocuiti numai cu o baterie recunoscut UL certificat pentru curent de inc rcare anormal maxim de minimum 4 mA Exist risc de explozie in cazul in care bateria este inlocuita o baterie necorespunz toare Eliminati bateriile folosite in conformitate cu instructiunile 19 Cordon de alimentare electric inregistrat UL si certificat CSA Pentru conectarea la o sursa de alimentare pentru America de Nord selectati un cordon de alimentare care este inregistrat UL si certificat CSA cu 3 conductoare 16 AWG terminat cu o fis turnat cu putere nominal egal cu 125 V 13 A cu o lungime de minimum 1 5 m sase picioare dar nu mai lung de 4 5 m
44. DO NOT USE THESE PORTS x 12 x 40GbE ports blocked 00 10GbE ports 00 40GbE ports 0000 12 x 40GbE ports split to 4 10GbE ports 48 10GbE ports du 4 x 40GbE ports split to 2 10GbE ports 8 IOGDE ports 8 x 40GbE ports used as 40GbE ports 8 x 40GbE ports Total 56x 10 and 8 40GbE ports 48 x 10GbE and 12 x 40GbE DO NOT USE x THESE PORTS 12 x 40GbE ports blocked 00 10GbE ports 00 40GbE ports 12 x 40GbE ports split to 4 10GbE ports 48 10GbE ports 12 x 40GbE ports used as 40GbE ports 12 40GbE ports Total 48 x 10GbE and 12 x 40GbE Mellanox Technologies 59 Rev 2 2 Switch Management 5 Switch Management Managed switches come with a management chip and MLNX OS software The MLNX OS switch management package can be downloaded from http support mellanox com SupportWeb software products fm products mlnx os For addi tional information on the MLNX OS refer to MLNX OS User Manual and MLNX OS Com mand Reference Guide located on Mellanox support web 5 1 InfiniBand Subnet Manager The InfiniBand Subnet Manager SM is a centralized entity running in the switch The SM applies network traffic related configurations such as QoS routing partitioning to the fabric devices You can view and configure the Subnet Manager parameters via the CLI WebUI Each subnet needs one subnet manager to discover activate and manage the subnet Each network requires a Subnet Manager to be running in either t
45. EEPROMS This connection allows access to the switch for updating firmware Figure 18 RJ 45 I2C Connector A Va SA Vs 12C 2 6 6 Reset Button On the connector side panel under the system LEDs is a reset button D RST This reset button requires a tool to be pressed 32 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object to push the reset button Figure 19 Reset Button Press the reset button to reset the main and management CPUs and to delete the existing password This button resets both the CPU of the switch device and the CPU of the management module It thereby resets all of the ports by bringing them down and powering them up when the button is pushed A quick push of this button performs this reset When the button is held down for 15 sec onds the switch is reset and the password is deleted You will then be able to enter without a pass word and make a new password for the user admin In the externally managed switches the reset button resets the CPU of the switch device 2 6 7 Port Connector Interfaces The connector side of the switch has 36 QSFP ports These are placed in two rows 18 ports to a row The ports are labelled as shown in Figure 20 The bottom row ports are flipped from
46. Inverted Data Output 3 38 GND GND 1 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Tx 3 3 V Power supply transmitter 30 1 3 3 Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground Mellanox Technologies 71 Rev 2 2 lews QSFP Connector Male and Female Vi Figure 42 Top 18 35 View into Rear of Connector GND AX n 7 GND TX3n TX3p GND LPMode Vcc1 VccTx Int ModPrsL GND RX4p RX4n GND RX2p RX2n GND N S 2 lt GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL LO VecRx co SCL SDA GND a o E RX3p RX3n GND RX1p RX1n GND 8 50 View into Front of Cage 4 4 0 9 8 7 6 5 4 3 2 UND qo diXH AND UEXH dexu IND vas 10s XH99A eseu TISSPON aNd dyXL UpXL AND dzXL uZXL AND Mellanox Technologies 72 SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix D RJ 45 CONSOLE and I2C Interface The RJ 45 CONSOLE a
47. Issue 1 Switch Status health LED If this LED is red unplug the switch and call your Mellanox representative If this LED is off 1 Check that the there is adequate ventilation Are the fan LEDs showing that the fans are all up and running 2 Make sure that there is nothing blocking the front or rear of the switch and that the fan modules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Issue 2 Power supply unit If the LED on the PS unit is not lit or is red check that the power cable is plugged into a work ing outlet 1 Check that the power cable has a voltage within the range of 100 240 volts AC 2 Check that the air flow direction of the PS units are consistent with the Fan module air flow 3 Remove and reinstall the power cable 4 Remove and reinstall the PS unit Issue3 The green power LED for the fans does not come on 1 Check that at least one Power LED is green 2 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 Switch Management on page 60 3 Check that the mating connector pins are not bent Do not run the switch 1f the System Status LED for the Fans is Red Issue 4 The link LED for the connector does not come on 1 Check tha
48. Mellanox TECHNOLOGIES SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual P N MSX6025T 1SFR MSX6025T 1BRR MSX6025F 1SFR MSX6025F 1BRR MSX6036T 1SFR MSX6036T 1BRR MSX6036F 1SFR MSX6036F 1BRR MSX6025T 1SFS MSX6025T 1BRS MSX6025F 1SFS MSX6025F 1BRS MSX6036T 1SFS MSX6036T 1BRS MSX6036F 1SFS MSX6036F 1BRS Rev 2 2 www mellanox com NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY
49. N Direction Label Designation Description Connector side inlet to power side R outlet Power side inlet to connector side outlet F QSFP Cable Power Budget Classification All SwitchX 2 QSFP switches are designed for fiber cables with a max power per module of 2W Typical power per port is 1 5 W Interfaces Management and Firmware Updating Interfaces The following interfaces connect to the switch e 2 100M 1 Gb Ethernet connectors labeled MGT Mellanox Technologies 2 9 Rev 2 2 Basic Operation 2 6 1 30 Mellanox Technologies e USB port that is labeled This interface can be used to update software or firmware This is only on the managed switch 1 connector that is labeled CONSOLE Use this connector to connect to the host PC reset button 2 banana connector on the power supply side is for FAE use only This is only on the managed switch Figure 17 Management Interfaces LE EE GS AvcY A WA ES 354 W3 ja 9M 9 CONSOLE ea Externally Managed Internally Managed For Externally managed switches there is one I2C RJ 45 interface to connect to the SX6025 RJ 45 Connector CONSOLE Internally Managed Switches only The port labeled CONSOLE is for a local host connection to the management module This is used the first time the switch is connected A harness is includ
50. ag es conform ment aux instructions du fabricant 19 Cable de alimentaci n homologado por UL y con certificaci n CSA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A 20 Alta corriente de fuga ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n Mellanox Technologies 87 Rev 2 2 21 Agregar informaci n de conexi n a tierra GND Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T
51. cy 130C In case the firmware fails to shut down the SwitchX device upon crossing the Critical thresh old the SwitchX device will auto shutdown upon crossing the Emergency 130C threshold 70 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Appendix C QSFP Interface Connector Connector Sienal Description Pin Number Pin Name SM 1 GND Ground 2 Tx2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground J3 and J6 5 Tx4n Transmitter Inverted Data Input 20 ae o 6 4 Transmitter Non Inverted Data Input 21 Rxin 18 7 GND Ground 22 Rx2p Rxtp 17 8 ModSelL Module Select 23 16 GND GND 9 ResetL Module Reset m REGIT Rx3n 10 Rx 3 3 V Power supply receiver m 13 11 SCL 2 wire serial interface clock ane SAR 12 12 SDA 2 wire serial interface data ModPrsL SDA 28 11 13 GND Ground Intl SCL 29 Ve mM 10 14 Rx3p Receiver Non Inverted Data Output 30 Voci ResetLL_9 15 Rx3n Receiver Inverted Data Output 31 pMode ModSell 2 16 GND Ground 32 GND GND 7 17 Rxlp Receiver Non Inverted Data Output 33 Tx3p Tx4p S 18 Rxin Receiver Inverted Data Output 34 ty3n Tx4n 5 19 GND Ground 35 GND 36 2 _3 20 GND Ground E Txan __2 21 Rx2n Receiver
52. de legare la p m nt la cablul de protec ie de legare la p m nt al cl dirii Echipamentul va fi instalat ntr o zon unde exist leg turi echipotentiale precum un centru de telecomunica ii sau o camer a computerelor dedicat 22 Codurile de instalare Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor electrice na ionale ale rii n cauz Pentru America de Nord echipamentul trebuie s fie instalat conform cerin elor aplicabile din Codul electric na ional al SUA i Codul electric canadian 23 Interconectarea unit ilor Cablurile pentru conectarea la unitatea RS232 si la interfetele Ethernet trebuie s fie de tipul DP 1 sau DP 2 certificate UL Nota cand se reg sesc intr un circuit non LPS Protectie la supracurent Un dispozitiv de protectie la supracurent inregistrat in circui tul de ramificare usor accesibil si cu o putere nominal egal cu 20 A trebuie s fie integrat in cablajul cl dirii 24 Expunerea la radiatii periculoase Atentie Utilizarea elementelor de comand sau ajustarea sau executarea altor proce duri dec t a celor indicate in acest document poate avea ca rezultat expunerea la radiatii periculoase PRODUS LASER CLASA 1 si referire la cele mai recente standarde pentru laser IEC 60 825 1 1993 A1 1997 A2 2001 si EN 60825 1 1994 A1 1996 A2 2001 25 Carter adecvat Un carter electric mecanic si ignifug adecvat va fi furnizat de fabr
53. ded for the short switch 7 screws per side are needed for the standard switch There are 16 screws in the kit you will have left over screws 42 Mellanox Technologies SwitchX 36 Port FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 26 Inserting the Caged Nuts Side you will slide the switch into Second side 3 Clip 6 caged nuts into the holes in the rack on the side of the rack you will be sliding the switch into Check that both sides of the switch power side and connector side are at the same level in the rack 4 Clip 4 more caged nuts into the holes on the opposite side of the rack Check that both sides of the switch left and right are the same level in the rack 5 Slide the rail into the rail slide 6 Using two of the bolts for each corner install the rails and rail slides in the rack Do not tighten the bolts yet Figure 27 Slide the Rail into the Rail Slide This side of the rail kit goes on the side of the rack you will slide the switch into This is the same side of the switch that will be next to the vertical support Mellanox Technologies 43 Rev 2 2 Installation 7 Slide the switch into the rails 8 Put the switch into place and screw the bolts into the nuts Tighten the bolts to 9 2 Nm or 81 5 pound inches 9 Ground the switch 10 Plug in the power cables Wait at least 5 seconds between inserting the first power supply cord and inserting the second power s
54. e dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n Mellanox Technologies 85 Rev 2 2 7 Apilamiento del chasis Los chasis no se deben apilar sobre otros equipos La ca da del chasis podr a causar lesiones corporales as como da os al equipo 8 Conexi n redundante de fuente de alimentaci n peligro de descarga Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Ay esta quitada o no esta bien cerrada 9 Fusible neutro o de polo doble Dos fusibles uno en el polo y otro en el neutro Quitar los cables de corriente antes de abrir la tapa de este producto o tocar cualquier componente interno 10 Tomas de alimentaci n m ltiples Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes Desconecte todas las fuentes de alimentaci n para asegurar que no haya corriente alguna dentro de la plataforma de conmutaci n 11 Al haber rayos peligro de descarga No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de LA rayos 12 Cable de conexi n y desconexi n InfiniBand de cobre Dado que los cables de cobre InfiniBand son pesados y no son flexibles su conexi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver adver t
55. e latch will snap into place confirm ing the proper installation 4 Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The green PS unit indicator should light If not repeat the whole procedure to extract the PS unit and re insert it 3 7 2 Extracting and Inserting the Fan Units This switch can operate indefinitely with less than a full compliment of fans in the fan modules operating properly so long as the ambient temperature 1s below 45 Celsius access area can be accessed only through the use of a special tool lock and key or This unit is intended for installation in a Restricted Access Location A restricted Wm other means of security Operation without a fan unit should not exceed two minutes During fan hot swap if the LED indicator is OFF then the fan unit is disconnected There are two possible air flows for the fan unit The air flow depends on the switch model See Air Flow on page 27 for an explanation of the model numbers and labels Make sure that the fans have the air flow that matches the model number An air flow opposite to the switch design will cause the switch to operate at a higher less than optimal temperature Figure 33 Air Flow Labels HH 22 These air flow labels must be the same gt To extract a Fan Unit 1 Using two fingers push both latch releases towards each other s
56. ed in the package to connect to a DB9 connection on a host PC Connecting to a local host PC and following the instructions in the Installation Guide Configuring the Switch for the First Time must be done before any remote management is available For the Socket pinout see RJ 45 CONSOLE and I2C Interface on page 73 This connector is not found in externally managed switches A SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 2 6 2 45 Ethernet Connector MGT Internally Managed Switches Only The 2 Ethernet connections labeled MGT provides access for remote management The switch can be connected to any Ethernet port The Ethernet port on the connector side of the switch 15 mgmt0 These connector s are not found in unmanaged externally managed switches bel Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 100M 1G auto negotiation gt 4 Use only FCC compliant Ethernet cables A The externally managed switches are Plug and Play and all firmware updates should N be done in band The I2C connection should only be used if the firmware image was i corrupted to the point that the regular firmware tools cannot successfully reburn the correct image 2 6 3 USB Connector Internally Managed Switches Only gt The USB interface can be used to update the MLNX OS Web User Interface WebUI or MLNX OS Command L
57. eed an initial configuration before they will start working Externally Managed Unmanaged switches are plug and play out of the box Updating the firm ware stack for the externally managed switches is done in band only You will need to confirm that your switch has the latest firmware For externally managed switches confirm that the latest FW revision is installed on this device Use the CLI to get the currently installed revision and then go to Mellanox com to check the revision of the latest version 1 Externally managed switches can only run in an IB fabric Externally managed f switches cannot run Ethernet protocol The following table shows which switches come with a management CPU and which do not Table 5 Switch Management Externally Family Internally Management Connections Managed SX6025 Externally Plug and Play Managed All firmware updates should be done in band using Mellanox firmware man Unmanaged agement tools I2C port access using MTUSB 1 device is required for firm ware updates if in band burning is not possible SX6036 Internally RS232 cable DB9 to RJ 45 included in the box to connect to host PC for ini Managed tial configuration of the switch After initial configuration the switch can be managed through the Ethernet port using a remote connection Gateway functionality is not supported in the externally managed SwitchX switches 18 Mellanox Technologies SwitchX 36 P
58. eet Australia New Zealand AS NZS CISPR 22 class A August 2012 1 7 e Converted graphics from Visio to use of mechanical drawings Added China CCC warning statement July 2012 1 6 Updated commands for Updating FW through the I2C port on the front of un managed switches July 2012 1 5 Replaced graphics with mechanical drawings Updated Specs in Appendix A May 2012 1 4 Added comments showing that MSX6025 switches do not support FRU LEDs in current revision December 2011 1 3 Changed Specifications table formatting 4 column to 2 column Fixed port LED assignment October 2011 1 2 e Removed reference to spring clip in Appendix Transferring the Power Cord Added Appendix for Thermal Threshold Definitions e Changed SwitchX to a Registered trademark 8 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Table 1 Revision History Table Date Revision Description July 2011 1 1 Changes reset button section removed reference to paper clip July 2011 1 0 Initial Release Mellanox Technologies 9 Rev 2 2 About this Manual This manual describes the installation and basic use of the Mellanox switch which is based on the SwitchX IB switch device Intended Audience This manual is intended for users and system administrators responsible for installing running and managing switch platforms The manual assumes familiarity with the
59. em rack and PS units The connector side of the switch has the QSFP ports system LEDs and management connection ports The switch platform contains auto sensing 100 240 VAC connections for all possible PS units The installer should use a rack capable of supporting the mechanical and environmental charac teristics of a fully populated platform The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take precautions to guarantee proper ventilation in order to maintain good airflow at ambi ent temperature Cable routing in particular should not impede the air flow through the switch 3 1 Installation Safety Warnings For Safety Warnings in French see Avertissements de s curit pour l installation French on page 76 for German Installation Sicherheitshinweise German on page 81 for Spanish see Advertencias de seguridad de instalaci n Spanish on page 85 for Russian see on page 90 for Roma nian see Avertismente privind siguranta la instalare Romanian on page 95 and for Chinese see page 100 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Installation in Restricted Access Location This unit is intended for installation in a Restricted Access Location 34 Mellanox Technologies
60. encias o instrucciones especiales consultar al fabricante del cable 13 Montaje y mantenimiento del bastidor Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adop tar precauciones especiales para garantizar que el sistema se mantenga estable En gen eral en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 86 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 14 Instalaci n del equipo La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales 17 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 18 Cambio de bater a II y a danger d explosion s il y a remplacement incorrect de la batterie Remplacer uniquement avec une batterie du m me type ou d un type quivalent recommand par le constructeur Mettre au rebut les batteries us
61. endix G Avertissements de s curit pour l installation French 76 Appendix H Installation Sicherheitshinweise German 81 Appendix I Advertencias de seguridad de instalaci n Spanish 85 Appendix J 90 Appendix Avertismente privind siguran a la instalare Romanian 95 Appendix L Chinese 100 4 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 List of Figures Figure 1 Connector Side View of the Switch 14 Figure 2 PulbOut Tab bates sate tee bee bsp 17 Figure 3 Switch System Power and Connector Side Panels 19 Figure 4 Externally Managed Switch System Power and Connector Side Panels 20 Figure 5 Switch Status LED ccc ene eee eee 21 Figure 6 Switch Status LED A ce I 22 Figure 7 Fan Status LED Connector Side 23 Figure 8 Fan Status LED Power Side 23 Figure 9 Power Status LED Connector Side 24 Figure 10 Power Side Panel 24 Figure Lil PS Un
62. entation 10 Conventions ars aces is ii 10 ADDreViatiOnS di dat SR tala 11 Mellanox Part Numbering Legend 12 Chapter L Qverview ou cs I a ak 13 Lol ESAS ee ER 15 1 2 Serial Number and Product Version Information 17 1 3 FDR Transmission Rate 17 1 4 Internally Managed vs Externally Managed 18 Chapter 2 Basic 19 2 1 Switch Platform Hardware Overview 19 2 20 2 3 Port Comnector LEDS os sn ana 26 2 42 iia A AA a are os SUL PU ote eas 27 2 5 Cable Power Budget Classification 29 2 6 A os RO a Rub hale ean hele We n aa 29 Chapter 3 Installation cscs sso teu o RR re EXER E EE eens See RES 34 3 1 Installation Safety Warnings 34 3 2 installation Kris ans a at a a ee 38 3 3 Package Contents a cena we usus evade sedi eR oda d as 39 3 4 Mechanical Installation 39 3 5 Grounding the Switch sp ee ata Pa qu ERU RECON 44 3 6 Power Connections and Initial Power On 45 3 7 Extracting and Inserting FRUs 46 3 8 Externally Managed Switches
63. fahr Arbeiten Sie wahrend eines Gewitters und Blitzschlag nicht am Ger t schlieBen Sie keine Kabel an oder ab 11 AnschlieBen Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb miissen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers 12 Rack Montage und Wartung sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor N Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen 13 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 82 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 14 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 16 Installationscodes Dieses Ger t muss entsprechend der aktuellsten Version des National Electrical Code installiert weden In Nodamerika muss das Ger t gem den geltenden Anforderungen des US National Electrical Code und des Canadian Electrical Code installiert werden 17 Akkua
64. g filtering rules on a system wide basis between nodes or port connections based on traffic patterns and user assigned system names GUIDs 60 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 5 3 Upgrading Software Software and firmware updates are available from the Mellanox Support website Check that your current revision 1s the same one that is on the Mellanox website If not Upgrade your soft ware Copy the update to a known location on a Remote server within the user s LAN Use the CLI or the GUI in order to perform software upgrades For further information please refer to the MLNX OS Software User Manual section Upgrading MLNX OS Software Be sure to read and follow all of the instructions regarding the updating of the software on your switch system 5 4 Updating Firmware Internally Managed Switch Managed switches do not require Firmware updating Firmware updating is done through the MLNX OS management software The switch comes standard with a management software module for switch management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX 2 based managed switch systems MLNX OS includes a CLI WebUI SNMP switch management software and IB management software OpenSM The Ethernet ports for remote management connect to Ethernet switches These switches must be configured to 100Mb 1 Gb auto negotiation 5 5 Updating Firmware External
65. he switch itself switch based or on one of the nodes which is connected to the fabric host based Y No more than two subnet managers are recommended for any single fabric The IB Subnet Manager running on the switch supports up to 648 nodes If the fabric includes more than 648 nodes you may need to purchase Mellanox s Unified Fabric Manager UFM software package Each subnet needs one subnet manager to discover activate and manage the subnet Each network requires a Subnet Manager to be running in either the switch itself switch based or on one of the nodes which is connected to the fabric host based The subnet manager OpenSM assigns Local IDentifiers LIDs to each port connected to the fabric and develops a routing table based on the assigned LIDs A typical installation using the OFED package will run the OpenSM subnet manager at system start up after the OpenIB drivers are loaded This automatic OpenSM is resident in memory and sweeps the fabric approximately every 5 seconds for new adapters to add to the subnet routing tables 5 2 Fabric Inspector Diagnostics Fabric Inspector is a plug amp play licensed software within MLNX OS displaying and filtering all identified systems and nodes within the fabric Fabric Inspector includes a complete set of InfiniBand tools for fabric wide diagnostics to check node node and node switch connectivity and to verify routes within the fabric Advanced filtering allows creatin
66. hit mit normalem Haushaltsm ll ensorgt werden Dieses Produkt und alle seine Teile in verantwortungsvoller und umweltfreundlicher Art und Weise entsorgen gt 84 Mellanox Technologies Rev 2 2 SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Appendix Advertencias de seguridad de insta lacion Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Lesi n corporal a causa de peso rN Recurra a suficientes personas para levantar este producto sin lt 40 Ibs 40 70 lbs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Equipos pesados mecanico para evitar lesiones Lh Dado que el equipo es muy pesado se debe mover nicamente mediante un elevador 4 Instalaci n en lugares con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 5 Riesgo de descarga el ctrica Riesgo de descarga el ctrica Con el m dulo del ventilador quitado se obtiene acceso a las clavijas de alimentaci n desde dentro de la cavidad del m dulo NO introducir herramientas ni partes del cuerpo en la cavidad del m dulo del ventila dor 6 Sobretemperatura 2 No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adec uada se deb
67. hout this document to indicate information that is important to the user Y This icon makes recommendations to the user 10 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 This icon indicates information that is helpful to the user This icon indicates a situation that can potentially cause damage to hardware or soft ware BEWARE This icon indicates a situation that can potentially cause personal injury or damage to hardware or software Abbreviations FDR Fourteen Data Rate Used to indicate a 4X port running at 56 Gb s FRU Field Replaceable Unit Mellanox Technologies 11 Rev 2 2 Mellanox Part Numbering Legend Place Field Decoder M Mellanox Technologies SX System Type SwitchX switch Family P Data Transfer Protocol 6 InfiniBand R Size of box 0 10 1 1 50 2 2U FF Management Capabilities 05 12 Ports externally managed 12 12 Ports internally managed 15 18 Ports externally managed 18 18 Ports internally managed 25 36 Ports externally managed 36 36 Ports internally managed C Data Rate F FDR T FDR10 Q QDR D DDR Separator P Power Supplies 1 1 2 2 M Depth of the Unit S standard depth B short depth Y Air Flow direction R Connector side to Power side airflow F Power Side side to Connector side airflow R Chip Generation R SwitchX S SwitchX 2 12 Mel
68. icantul produsului final si sau utilizatorul final 26 Nu utilizati comutatorul ca raft sau spatiu de lucru Atentie Echipamentul montat pe o linie de alunecare sin nu va fi utilizat ca raft sau spatiu de lucru Scopul sinelor nu este de a glisa unitatea de pe rack Acestea sunt des tinate instalarii permanente numai la punctul final de oprire Si nu vor fi folosite pentru depanare si intretinere 98 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 27 Directiva DEEE In conformitate cu Directiva DEEE 2002 96 CE toate deseurile de echipamente elec trice si electronice EEE trebuie colectate separat si nu trebuie eliminate impreuna cu deseurile menajere obisnuite Eliminati acest produs si toate componentele sale in mod responsabil si ecologic Mellanox Technologies 99 Rev 2 2 Appendix L Chinese 1 Ah 2 lt 40 lbs 40 70 Ibs 70 121 Ibs gt 121 Ibs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 ERIR En 4 Ah
69. ice Q For non tropical use ERA DIE dl aa aras Dan hab yungh youq gij dienheiq diuzgen mbouj dwg diegndat haenx ancienz saw h Dan hab yungh youq gij digih 2000m me haenx ancienz sawj h gy as es Ms IF E 2000 PR p Warning for Class A A AP IRE RAE EA English translation of above statement This is a Class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures Mellanox Technologies 69 Rev 2 2 Thermal Threshold Definitions There are three thermal threshold definitions for the SwitchX switch device which impact the overall switch system operation state Warning Critical and Emergency 1 Warning 100C On managed systems only When the SwitchX device crosses the 100C threshold a Warning Threshold message will be issued by the MLNX OS management SW indicating to system administration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any action by hardware such as switch shutdown 2 Critical 120C When the SwitchX device crosses this temperature the firmware will automatically shut down the device 3 Emergen
70. imultaneously while pulling the fan module out of the switch As the fan unseats the fan status indicator will turn off 1 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the fan unit unseats the fan unit status LEDs will turn off 48 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 2 Remove the fan unit Figure 34 Fan Module Latchs Bu sy ERINI These two latches must be pushed towards each other at the same time while the module is pulled out gt To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely The green fan status indicator should light If not extract the fan unit and reinsert it After two unsuccessful attempts to install the fan unit power off the switch before attempting any system debug 3 8 Externally Managed Switches Externally managed switches that is the SX6025 switches do not have a management chip or management board within the switch These switches do not get configured On externally man aged switches the CONSOLE Ethernet and USB connectors are not found Instead there is an I2C connector The externally managed switches are Plug and Play and all firmware updates
71. in locul acesteia In cazul in care spatiul pentru sursa de alimentare este gol nu operati produ sul c nd capacul orb este indep rtat sau nu este fixat in mod sigur 9 Siguran fuzibil bipolar neutr doanele de alimentare inainte de a deschide capacul acestui produs sau inainte de a Acest sistem este prev zut cu sigurant fuzibil bipolar neutr Indep rtati toate cor atinge orice componente interne 10 Multiple mufe electrice Risc de soc electric si pericol electric Toate aparatele cu alimentare de la retea sunt independente Deconectati toate sursele de alimentare cu energie pentru a asigura decuplarea n inte riorul platformei de comutare 11 In timpul desc rc rilor electrice pericol electric In timpul perioadelor cu descarcari electrice luminoase nu lucrati cu echipamentul sau nu conectati sau deconectati cablurile 12 Conectarea deconectarea cablului din cupru InfiniBand Cablurile InfiniBand din cupru sunt grele si inflexibile de aceea trebuie sa fie atasate sau detasate de conectori cu grij Consultati produc torul de cabluri pentru avertis mente instructiuni speciale 13 Montarea sau depanarea intr un rack Cand acest produs este montat sau depanat intr un rack trebuie sa fie luate masuri de precautie speciale pentru a se asigura ca sistemul ramane stabil In general trebuie sa umpleti rack ul cu echipamente incepand de jos in sus 96 Mellanox Technologies SwitchX 3
72. including the power cord ground You must follow all national al and local codes when installing this equipment 3 6 Power Connections and Initial Power On The switch platform ships with one or two power supply units For switches with only one unit installed a second PS unit may be installed at a later time Each supply has a separate AC recep tacle The input voltage is auto adjusting for 100 240VAC 50 60Hz power connections The power cords should be standard 3 wire AC power cords including a safety ground and rated for 15A or higher Caution The switch platform will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable access area can be accessed only through the use of a special tool lock and key or i This unit is intended for installation in a Restricted Access Location A restricted Wm other means of security AD Caution After inserting a power cable and confirming the green system status LED a light is on make sure that the Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the switch make sure ALL Connector LEDS are off to ensure a powered down status Mellanox Tech
73. ine Interface CLI 5 There is a single USB connector This connector can be used to install software and or firmware upgrades using a memory device that has a USB connector This connector is USB 2 0 compli ant The USB connector is not found on externally managed switches gt gt USB 1 0 is not supported 5 Mellanox Technologies 31 Rev 2 2 Basic Operation Do NOT use excessive force when inserting or extracting the USB from the connec tor 5 2 6 4 12C Connector On Internally Managed Switches There is an I2C connector on the far left of the power side of the switch This interface is for Debug and Troubleshooting only This connector can be used to install firmware upgrades should the firmware image be damaged and cannot be upgraded through a host PC or remotely This interface 1s for support personnel and advanced users only 2 6 5 12 Connector On Externally Managed Switches Only The I2C RJ 45 connection provides access to Flash and EEPROMs This connection is for FAE use only The externally managed switches are Plug and Play and all firmware updates should be done in band This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only All firmware updates should be done in band using Mellanox Firmware Management Tools You will need to order an MTUSB 1 USB to I2C adapter to make use of the I2C interface The I2C connection provides access to Flash and
74. install remplac et maintenu que par un personnel form et qualifi La mise au rebut de cet quipement doit se faire conform ment toutes les lois et r glementations nationales Cet quipement doit tre install conform ment aux codes lectriques locaux et nation aux 17 Codes d installation Cet appareil doit tre install conform ment la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit tre install en respectant les exigences de l US National Electrical Code et du Code canadien de l lectricit 18 Remplacement de la batterie Avertissement ne remplacer qu avec une batterie UL certifi e pour accepter un cou rant de charge anormal maximal sup rieur ou gal 4 mA Si la batterie n est pas remplac e par un type correct il y a un risque d explosion Les batteries usag es doivent tre mises au rebut conform ment aux instructions 19 Cordon d alimentation UL Listed et certifi CSA AD Pour le branchement lectrique en Am rique du Nord utiliser un cordon d alimenta tion UL Listed et CSA Certified a 3 conducteurs calibre 16 AWG avec une prise moul e 125 V 13 A faisant au moins 1 5 m de long six pieds et au plus 4 5 m Pour le branchement lectrique en Europe utiliser un cordon d alimentation au format international harmonis marqu lt HAR gt a 3 conducteurs dau moins 1 mm2 de sec tion 300 V avec une gaine isola
75. ions Canada ICES Class A International CB EU EN55022 Class A Safety Russia GOSTR EU EN55024 Class A EMC Argentina S mark DoC EU EN61000 3 2 Class A Emissions EU EN61000 3 3 Class A Japan VCCI Class A Australia New Zealand AS NZS CISPR 22 class A Environmental Acoustic EU IEC 60068 2 64 Random ISO 7779 Environmental Vibration ETS 300 753 Acoustic EU IEC 60068 2 29 Shocks 71dB A 7 1 Bel Type I II EU IEC 60068 2 32 Fall Test Scalability and Performance Switching Switching Switching Performance Performance Simultaneous wire speed any port Capacity 4Tb s Capacity to any port Reliability Availability and Serviceability Features Hot Swappable Hot swapability Fan Module Redundancy 1 1 Redundant A 1 Approved Cables For a list of all approved cables see http www mellanox com related docs user_manuals Mellanox_approved_cables pdf A 2 EMC Certifications The list of approved certifications per switch in different regions of the world is located on the Mellanox Website at http www mellanox com related docs user_manuals Regulatory_and Compliance Guide pdf EMC statements are also in the Regulatory and Compliance Guide 68 Mellanox Technologies SwitchX 36 Port FDR InfiniBand System Hardware User Manual Rev 2 2 China CCC Warning Statement English translation disconnect all power sources before serv
76. it Status LEDS eei Ar rue ere aue qt aule 25 Ris re 12 Bad Port LED cialis a aa LE ee ONE EP EIN CEA ER 25 Figure 13 Identifier EED sun A Run EE Ux RN E Es 26 Figure I4 PortilEDS ecc een Eve RE v Eee e vise e re E 27 Figure 15 Component Air Flow s 28 Figure 16 Component Airflow Mismatch Wrong 28 Figure 17 Management Interfaces 30 Figure T8 RIAS PC Connector se ne es e Pet art a de e e aue dg 32 Figur 19 Reset Button cupa tei Lime Apa SG Gap ke Rete ok a DE 33 Figure 205 Port Numbering mi ein ws 33 Figure 21 Top and Bottom Port Orientation 33 Figure 22 Rack Rail Rit Parts oi sa seu A a ge 40 Figure 23 Placement of Switch in Rack 41 Figure 24 Mounting Options stss pep parent se pate pu OX ER A panne lan 42 Figure 25 Screwing on Ra l 42 Figure 26 Inserting the Caged Nuts 43 Figure 27 Slide the Rail into the Rail Slide 43 Figure 28 Status LEDs 5 Minutes After Power On 44 Figure 29 Two Power Inlets Electric Caution Notifications 46 Figure 30 Fan and PS u
77. itch in Rack Doors Te Connector side Cable with recom mended bending radius 19 482 6mm Switch This edge of the switch is even with the rack vertical support Insufficient room for recommended bend ing radius i b Power Spine side Doors Consider the following when deciding where and how to mount the rails and rail slides The distance between the rack and the door can be as little as 1 57 4 cm on one side of the rack and as much as 7 18 cm on the other side of the rack Keep in mind that there can be as many as 36cables connected to the switch Do you want the connector side recessed in the rack to allow for a larger cable bending radius It is possible to recess the connector side by 2 5cm by optional placement of the switch rails See Figure 24 Mounting Options Will the connector side be recessed past other equipment in the rack and will this be problematic The installation kit allows for a 2 5cm recess of the switch past the vertical support Mellanox Technologies 41 Rev 2 2 Installation Figure 24 Mounting Options Installing the switch slides on the switch Decide which mounting option you want to use Screw the switch slides onto the switch Use 5 flat head screws for short switches and 7 screws for standard depth switches to connect each switch slide Figure 25 Screwing on the Rail 5 screws per side are nee
78. lanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 1 Overview SX6036 switch system provides the highest performing fabric solution in a 1U form factor by delivering up to 4Tb s of non blocking bandwidth with 170 ns port to port latency Faster servers based on PCle 3 0 combined with high performance storage and applications that use increasingly complex computations are causing data bandwidth requirements to spiral upward As servers are deployed with next generation processors High Performance Computing HPC environments and Enterprise Data Centers EDC will need every last bit of bandwidth delivered with Mellanox s next generation of FDR InfiniBand high speed smart switches FDR InfiniBand technology moves from 8b 10b encoding to a more efficient 64 66 encoding while increasing the per lane signaling rate to 14 Gb s Mellanox end to end systems can also take advantage of the efficiency of 64 66 encoding using Mellanox FDR 10 supporting 20 more bandwidth over QDR using the same cables connectors designed for 40 GbE Built with Mellanox s sixth generation SwitchX InfiniBand switch device the SX6036 pro vides up to thirty six 56 Gb s full bi directional bandwidth per port These stand alone switches are an ideal choice for top of rack leaf connectivity or for building small to extremely large sized clusters Virtual Protocol Interconnect VPI flexibility enables any standard networki
79. le 21 gt RI 45 CONSOLE Pifiout cui ee VC ale r d ae aa 73 Table 22 PINOUT rast Sond toa ia 73 Table 23 OPNs for Replacement Parts 74 Mellanox Technologies 7 Rev 2 2 Revision History Table 1 Revision History Table Date Revision Description August 2013 2 2 Added note to wait 5 seconds between plugging in PSU 1 and PSU2 July 2013 2 1 Updated Pull tab Changed Figure 3 to IPSU Added note concerning USB insertion Added note concerning RJ45 use of FCC compliant cables Added warning Restricted Access Location e Added warning do not stick body parts into switch cavities Removed 0 0 from the Mellanox Part numbering legend Removed caution notes regarding fixed bugs Added information regarding confirming you have the latest SW and FW Added Sections on Fanout cables Fixed issue in the Troubleshooting section Added caution to wait 2 seconds between connecting Power supplies Updated Features list Reorganized document order Added new Safety Warnings November 2012 2 0 Fixed Figure 1 October 2012 1 9 Added caution note not to insert cables with excessive force Figure 29 with caution note September 2012 1 8 Reduce the OPNs on the title page to those in the Price book e Added to Safety in spec sheet Russia GOST R Argentina S mark DoC Added to EMC in spec sh
80. low direction PS unit Air flow direction 46 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 See Section 2 4 on page 27 for more information on air flow direction N Power supply units have directional air flows similar to the fan module The Fan mod ule airflow must coincide with the airflow of all of the PS units If the PS unit airflow direction is different from the fan module airflow direction the switch internal temper ature will be affected Figure 31 Power Supply Unit Extraction Push Latch release If you are not immediately installing a new PSU you must install a blank so that the air flow for cooling the switch is not compromised Figure 32 PS Unit Pulled Out gt To extract a PS unit 1 Remove the power cord from the power supply unit 2 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PS unit unseats the PS unit status LEDs will turn off 3 Remove the PS unit gt To insert a PS unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Do not attempt to insert a PS unit with a power cord connected to it 2 Insert the PS unit by sliding it into the opening until a slight resistance is felt Mellanox Technologies 47 Rev 2 2 Installation 3 Continue pressing the PS unit until it seats completely Th
81. ly Managed Switch All firmware updates should be done in band Go to the Mellanox Website and confirm that the firmware is the latest If not reburn the latest firmware from the download site New firmware versions will be posted on the Mellanox firmware download page http www mellanox com gt Support gt Download Firmware Access to this page requires a login and password You will need the Mellanox Firmware Tools package to update firmware for this switch It can also be downloaded from http www mellanox com gt Support gt InfiniBand Software and Drivers You will also need to download and unzip the firmware binary image This is provided in the Mellanox Web site at http www mellanox com gt Support gt Download Firmware and go to the SwitchX 2 Switch systems Click in the Table for the firmware image that you need In order to get information regarding the externally managed switch you need to download the Mellanox MFT tools from the Mellanox website www mellanox com gt Products gt Software gt Firmware tools Select and download the release that matches your system Follow the instructions in the User Manual http www mellanox com pdf MFT MFT_user_manual pdf to get the tools 5 5 1 Current Firmware Revision For Externally Managed Switches gt In order to obtain the Firmware version of the externally managed switch 1 Run the following command from a host flint d lid number qq Mellanox Technologies 61
82. nd I2C interface uses the 568A standard wiring color coding CONSOLE Table 21 RJ 45 CONSOLE Pinout J34 Connection Signal Pin Color uma Pin 1 TXD RJ 45 RTS 1 G W RAD m RI45 rs CIN nm RJAS DTR an TXD TXD 45 DTR4 2 G ew G HA RXD RI 45 TXD O W A in Pin 8 GND GND 4 Bl Looking into the Socket GND GND 5 BI W RXD RJ 45 RXD 6 O GND GND 7 Br W GND GND 8 Br Table 22 RJ 45 I2C Pinout Connection Signal Pin Color TXD RJ 45 1 G W TXD GND 2 G T 12C RXD GND 3 O W GND GND 4 TXD Pin 1 A GND GND 5 GND ND GND RXD RXD RJ 45 SCL 6 O a GND GND 7 Br W au Pin 8 GND GND 8 Br Looking into the Socket Mellanox Technologies 73 Rev 2 2 Appendix E Accessory and Replacement Parts Table 23 OPNs for Replacement Parts Part Description OPN I2C DB9 or RJ 45 to USB Adapter MTUSB 1 Harness RS232 2M cable DB9 to RJ 45 for managed switches only HAR000028 Fan module with power supply side to connector side air flow MSX60 FF Fan module with connector side to power supply side air flow MSX60 FR 300W Power Supply w Power Supply Side to Connector side air flow MSX60 PF 300W Power Supply w Connector side to Power Supply side air flow MSX60 PR Power cord
83. ng clustering stor age and management protocol to seamlessly operate over any converged network leveraging a consolidated software stack VPI simplifies I O system design and makes it easier for IT manag ers to deploy infrastructure that meets the challenges of a dynamic data center SX6036 comes with an onboard subnet manager enabling simple out of the box fabric bring up for up to 648 nodes The SX6036 MLNX OS software delivers complete chassis management to manage the firmware power supplies fans ports and other interfaces SX6036 can also be coupled with Mellanox s Unified Fabric Manager UFM software for managing scale out InfiniBand computing environments UFM enables data center operators to efficiently provision monitor and operate the modern data center fabric UFM boosts application performance and ensures that the fabric is up and running at all times The switch comes pre installed with all necessary firmware and is configured for standard opera tion within an InfiniBand fabric This switch requires an InfiniBand compliant Subnet Manager running from one of the hosts or Fabric Management software running on the fabric All that is required for normal operation is to follow the usual precautions for installation and to connect the switch to the adapter cards Once an unmanaged switch is connected the Subnet Management software automatically configures and begins utilizing the switch Managed switches need initial configurati
84. nit Air Flow Direction 46 Figure 31 Power Supply Unit Extraction 47 Figure 325 PS Unit Pulled Qut ner ee a ein n 47 Figure 33 Air Flow Labels 48 Figure 34 Fan Module Latchs 49 Figure 352 Console Port ies E E PE arn gaa 50 Mellanox Technologies 5 Rev 2 2 Figure 36 Top and Bottom Cable Orientation 56 Figure 37 Breakout or Fanout Cable 57 Figure 38 Port Splitting OptionsPort Splitting Options 54 Figure 39 3 Examples of Port Mapping Assignment 59 Figure 40 MTUSB 1 with Cables 62 Figure 41 I2C Cable Connected to SX6025 63 Figure 42 Connector Male and Female Views 72 6 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 List of Tables Table 1 Revision History Table 8 Table 2 Reference Documents and Web Sites 10 Table 3 General System Features
85. nologies 45 Rev 2 2 Installation Do not hot swap the power supply if your switch has only one power supply You must power down the system to replace the power supply unit when there is only one PS unit in the switch Figure 29 Two Power Inlets Electric Caution Notifications CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Risque de choc et de danger hazard The two PS units are Schocks Entferrnen des e lectriques Le de branch independent Netzsteckers elnes Netz ment d une seule alimentation teils spannungsfrei Um stabilise e ne de branch Disconnect all power supplies to alle Einhieten spannungs uniquement qu un module ensure a powered down state frei zu machen sind die Alimentation Stabilise e inside of the switch platform Netzstecker aller Netzteile Pour isoler completement le zu entfernen module en cause Il faut de brancher toutes les alimen tations stabilise es 3 7 Extracting and Inserting FRUs Never run the switch for more than 5 minutes with an uncovered open PSU or Fan slot 3 7 1 Extracting and Inserting the Power Supply Unit With both power supplies installed in the redundant configuration either PS unit may be extracted without bringing down the system Make sure that the PS unit that you are NOT replacing is showing all green for both the PS unit and status LEDs Figure 30 Fan and PS unit Air Flow Direction ome Fan Air f
86. nstallation avant de relier l quipement au secteur 2 Blessures cause du poids rN Pr voyez assez de personnel pour soulever ce produit en toute s curit lt 40 lbs 40 70 lbs 70 121 lbs gt 121 los lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 quipement lourd Cet quipement est tr s lourd et doit tre d plac avec un syst me de levage m ca nique pour viter les blessures 4 Installation dans un emplacement acc s limit 7 Cette unit doit tre install e dans un emplacement acc s limit 5 Danger d lectrocution Danger d lectrocution Lorsque le module de ventilation est retir les broches d alimentation sont expos es dans l emplacement du module NE PAS ins rer d outils ou la main dans l emplacement du module 6 Surchauffe maximum recommand de 45 C 113 F En outre et pour garantir une circulation N Cet quipement ne doit pas tre en service dans un local dont la temperature d passe le MA d air correcte laisser un espace d au moins 8 cm 3 autour des orifices de ventilation 76 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 7 Ch ssis empil sur d autres quipements Le ch ssis ne doit pas tre empil sur d autres quipements S il tombe il peut endom mager l quipement ou entra ner des blessures gt 8 Connexion de l alimentation redondante danger d lectrocution
87. nte en PVC Le cordon doit avoir une prise moul e 250 V 10 A 20 Courant de fuite lev Avertissement courant de fuite lev une connexion 4 la terre est indispensable avant de brancher I alimentation 78 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 21 Information d ajout de branchement la terre GND Avant de relier cet appareil au secteur ses vis des bornes de la terre de protection doi vent tre reli es la terre de l installation lectrique de l immeuble Informations de AA branchement a la terre L installation lectrique doit fournir une connexion 4 la terre et un technicien de main tenance doit y relier l quipement de fa on permanente UN TECHNICIEN DE MAINTENANCE doit v rifier si la prise laquelle sera bran ch l quipement fournit une connexion la terre Si ce n est pas le cas le TECH NICIEN DE MAINTENANCE doit arranger l installation d un CONDUCTEUR DE MISE A LA TERRE allant de la borne de mise la terre jusqu au conducteur de terre de l immeuble Cet quipement doit tre install dans une zone disposant de liaisons quipotentielles comme un centre de t l communications ou une salle informatique d di e 22 Codes d installation trique nationaux En Am rique du Nord l quipement doit tre install en respectant Cet appareil doit tre install conform ment la version la plus r cente des codes lec A les e
88. on before they will start working It is recommended that the Mellanox OpenFabrics software package be installed on all nodes connected to the switch The software package provides a subnet manager and network manage ment tools as well as connectivity software for servers and storage and is available on the Mella nox web site See Section 5 on page 60 for more information Installation hot swapping components and hardware maintenance is covered in Basic Opera tion on page 19 Mellanox Technologies 13 Rev 2 2 Overview Figure 1 Connector Side View of the Switch 427 9 mm Standard depth 628 9 mm Short depth 428 9 mm 14 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 1 1 Features The switch includes the following features Table 3 General System Features Feature Description Hardware features Network Interfaces 36 QSPF VPI interfaces Interface type InfiniBand or Ethernet InfiniBand speed FDR 56Gb s FDR10 QDR 40Gb s DDR 20Gb s SDR 10Gb s Ethernet speed 10 40 56 GbE Switch options InfiniBand In box Ethernet InfiniBand to Ethernet Gateway Compliance Compliant with IBTA 1 2 1 and 1 3 RoHS 6 Advanced Features Congestion control Adaptive routing Port mirroring Forward Error Correction FEC Link Layer Retransmission LLR Multiple SWIDs
89. ort QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 2 Basic Operation 2 1 Switch Platform Hardware Overview Figure 3 shows the power side panel view of the switch The figure shows the hot swap power supply locations and the hot swap fan module and an 12C banana connector The figure below shows the connector side panel views of the internally managed and externally managed switches The figure shows port configurations for the switch systems Managed sys tems have 2 Ethernet RJ 45 connectors for management 1 RJ 45 connector for connecting to a host PC 1 USB connector Externally managed switches only have a single I2C RJ 45 connector All switches have various status LEDs for on site status information Figure 3 Switch System Power and Connector Side Panels AN S INNE Lo Power side panel EESIESSEETSZBSESSOSSEUSESSSRBASEERSSUSSESESSESUBUSEZISESSTMSASSEISSSSZESZSZSUESSSSSSSSEESSSS A SIR TA Vi VA Viola VD DA Vu Sa Vic VA Vis Sa Vi la Wi BA YH 254 Vu VA Vis MA Vo Vn BA Va SA Connector side panel internally managed DOI IO OOOO OOOO O OOOO l V2 SS 7 Fs 9 Folla 134 Via 154 Vic 174 Vis 194 7W20 214 V2 234 Coa 254 Y26274 Vis 294 W30 DESEES Connector side panel externally managed All connectivity except for power cords is via the connector side panel All connectors can sup port fiber cables Mellanox Technologies 19 Rev
90. other is a USB cable with one type A host connection to be connected to the server or PC and a type B con nection to be connected to the MTUSB adapter Figure 41 I2C Cable Connected to SX6025 bh 5605 Vs SA VW 12C With the switch connected to a computer through the MTUSB it is now possible to reprogram the SPI Flash memory 5 5 2 1 Instructions for Reprogramming Through the I2C Port 1 Make sure that the MTUSB I2C cable is connected to the switch and USB cable is connected to a com puter e is running on the computer e mtusb 1 mst device appears in mst status 2 Open the bus by running the unlock command gt i2c dev mst mtusb 1 w 0x22 Oxla Oxfb 3 Open 0X48 by running the following command gt i2c dev mst mtusb 1 w 0x70 0x0 0x1 4 Run the burn command ilime Cl SS Ms i Eel 7 0 0 ASS 5 Power cycle the switch by unplugging and re plugging the power cord to load the new firm Ware 5 5 3 How to Get Mellanox Firmware Tools MFT Mellanox Firmware Tools MFT and documentation are available for download via http www mellanox com gt Downloads gt Firmware amp Documents The MFT kit includes mlxburn flint spark debug utilities See Related Documentation on page 10 Mellanox Technologies 63 Rev 2 2 Troubleshooting 6 Troubleshooting As soon as a switch is plugged in make sure that the green power LEDs on the PS units are on
91. pport page for your switch This page will need a support login and password Rerunning the Wizard 8 If you want to rerun the wizard run the following commands switch enable switch configure terminal switch config configuration jump start 3 9 2 Starting a Remote Connection to the Switch There are two ways to access the management CPU Via the Ethernet connector for remote access SSH Telnet Web SNMP XML For additional information on the MLNX OS refer to MLNX OS User Manual and MLNX OS Command Reference Guide located on Mellanox support web gt To Start an SSH Connection to the Switch perform the following steps 1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connector 2 Connect to the remote machine rem_machl is used as an example 54 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 3 Start a remote shell to the switch using the following command switch IP address gt is the IP address of the switch or its DNS name rem machl gt ssh 1 username switch ip address gt Mellanox MLNX 0S Switch Management Password Last login March 13 2013 from 192 168 10 1 Mellanox Switch switch gt 4 You can enter any supported command now Mellanox Technologies 55 Rev 2 2 Cabling 4 4 1 56 Cabling Cable Ins
92. r system of Norway only 3 1 1 Battery Replacement Mellanox Technologies does not support battery replacement Customer removal of the switch cover will void the warrantee Only Remove the cover to comply with WEEE directives or to dis assemble for environmentally approved disposal Customer removal of the switch cover will void the warrantee There is a danger of explosion if the battery is incorrectly replaced DO NOT replace the battery Dispose of used batteries according to manufacturer s instructions 3 2 Installation Kits There are two Installation kit options One long and one short Both the standard and the short switches can be mounted using the long rail kit Table 14 Installation Kit According to Rack Size Kit OPN Rack Size MSX60 BKIT 40 60 cm deep MSX60 SKIT 60 80 cm deep 38 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 3 3 Package Contents Before you install your new switch unpack the system and check to make sure that all the parts have been sent check this against the parts list below Check the parts for visible damage that may have occurred during shipping The switch comes packed with the following items 1 switch installation kit e power cable for each PS unit Type C13 C14 Harness DB9 to RJ 45 for connecting to the CONSOLE port for managed switches only Documentation QSG e
93. s to the machine please type a pass lt Enter gt to leave unchanged and then press Enter Then confirm the password lt new_password gt by re entering it Step 6 Confirm admin pass word lt new_password gt Note that password characters are not printed You have entered the following The wizard displays a summary of your choices and then information asks you to confirm the choices or to re edit them lt A summary of the configura Either press Enter to save changes and exit or enter the tion is now displayed gt configuration step number that you wish to return to To change an answer enter the step number to return to or hit Note lt enter gt to save changes and exit To re run the configuration wizard run the command Choice lt Enter gt configuration jump start in Config mode Configuration changes saved The table below shows an example of static IP configuration for mgmt0 interface Table 17 Configuration Wizard Session Static IP Configuration Wizard Session Display Static IP Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration yes Step 1 Hostname switch Step 2 Use DHCP on mgmt0 interface yes no Step 3 Use zeroconf mgmt0 interface no Step 4 Primary IP address for example 192 168 10 4 10 10 10 10 Mask length may not be zero if address is not zero interface eth0 Step 5 Netmask 0
94. should be done in band The I2C connection should only be used if the firmware image was corrupted to the point that the regular firmware tools cannot successfully reburn the correct image You will need to confirm that your switch has the latest firmware For externally managed switches confirm that the latest FW revision is installed on this device Use the CLI to get the cur rently installed revision and then go to Mellanox com to check the revision of the latest version For more information on how to get the current firmware version see Section 5 5 1 Current Firm ware Revision For Externally Managed Switches on page 61 3 9 Internally Managed Switch Configuration 3 9 1 Configuring the Switch for the First Time The procedures described in this chapter assume that you have already installed and powered on your switch according to the instructions in this document Mellanox Technologies 49 Rev 2 2 Installation Internally managed switches must be configured before they will work Follow the procedures below to configure the switch To configure the switch A Externally managed switches do not get configured The CONSOLE Ethernet and Ah USB connectors are not found on externally managed switches Instead there is an K I2C connector 1 Connect the host PC to the Console RJ 45 port of the switch system using the supplied cable The Console ports for systems are shown below as examples The port
95. t both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest firmware version is installed on all of the HCA cards and the switch 4 If media adapters are used check that the all connections are good tight and secure Issue 5 The activity LED does not come on When running IB check that the Subnet Manager has been started Issue 6 The switch is off 1 Unplug the switch 2 Wait 5 minutes 3 Plug in the switch 64 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 4 Ifthe switch does not come on check the power supplies 5 If the switch comes on Use the MLNX OS management CLI or Web GUI to determine the cause of the Shutdown 6 Check the temperature 7 Check the Fan status Issue 7 The switch is not working and unresponsive 1 Reset the switch If resetting the switch does not work 1 Unplug the switch 2 Wait 5 minutes 3 Plug in the switch 4 If the switch does not come on check the power supplies 5 If the switch comes on use the MLNX OS management CLI or Web GUI to determine the cause of the shutdown Issue 8 The last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop 2 Push the reset button on the switch or management module 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image or image 2
96. taken AA equipment starting from the bottom to the top Mellanox Technologies 35 Rev 2 2 Installation 10 Equipment Installation This equipment should be installed replaced and or serviced only by trained and qualified personnel 11 Equipment Disposal Y Disposal of this equipment should be in accordance to all national laws and regula tions 12 Local and National Electrical Codes 7 This equipment should be installed in compliance with local and national electrical A codes 13 Installation Codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electrical Code 14 Battery Replacement Warning Replace only with UL Recognized battery certified for maximum abnormal charging current not less than 4mA There is a risk of explosion should the battery be replaced with a battery of an incorrect type Dispose of used batteries according to the instructions 15 UL Listed and CSA Certified Power Supply Cord For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 16 AWG terminated with a molded plug rated at 125 V 13 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord
97. tallation FDR and FDR 10 are only guaranteed to work with approved Mellanox Cables For a list of approved cables for these switches see http www mellanox com related docs user_manuals jo Mellanox_approved_cables pdf All cables can be inserted or removed with the unit powered on To insert a cable press the con nector into the port receptacle until the connector is firmly seated The LED indicator corre sponding to each data port will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the LED will change to green for IB and orange for Eth When data is being transferred the light will blink green Do not force the cable into the cage with more than 40 newtons 9 0 pounds force 4 kg force Greater insertion force may cause damage to the cable or the cage Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row gt gt Figure 36 Top and Bottom QSFP Cable Orientation iz gt S NEN To remove disengage the locks and slowly pull the connector away from the port receptacle The LED indicator for that port will turn off when the cable is unseated
98. that is internationally harmonized and marked lt HAR gt 3 conductor minimum 1 0 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded plug rated at 250 V 10 A 16 High Leakage Current A Warning High leakage current Earth connection essential before connecting supply 36 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 17 Add GND connection information Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation N GND Connection Information The building installation shall provide a means for a connection to protective earth and the equipment shall be permanently connected to that by a service person A SERVICE PERSON shall check whether or not the socket outlet from which the equipment is to be powered provides a connection to the building protective earth If not the SERVICE PERSON shall arrange for the installation of aPROTECTIVE EARTHING CONDUCTOR from the separate protective earthing terminal to the protective earth wire in the building The equipment shall be installed in area where equipotential bonding exists such as a telecommunication centre or a dedicated computer room 18 Interconnection Of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified type DP 1 or DP 2 Note when
99. the top row See Figure 21 for bottom row top row port orientation See page 56 for more information regarding breakout fanout cables Figure 20 Port Numbering Mellanox Technologies 33 Rev 2 2 Installation 3 Installation Installation and initialization of the switch platform are straightforward processes requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment The externally managed switch platform does not require any programming or configura tion to operate as a basic InfiniBand switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software The managed switch platform requires initial configuration to operate as an InfiniBand device All internally managed switches come with an internal PPC based management board This board allows for internally managing the switch through a host PC or remotely through the Ethernet The switch platform can be rack mounted and is designed for installation in a standard 19 rack The power side of the systemgatewayswitch includes a hot swap power supply module a blank cover for an optional second PS unit for redundancy and a hot swap fan tray There are two pos sible air flow directions Be sure that the switch air flow direction is compatible with your sys t
100. tional Electrical Code und dem Canadian Electrical Code gt 22 Verbindung der Ger te untereinander Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung 23 Gef hrliche Strahlung Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Proze duren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkontakt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 24 Geeignetes Geh use Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts oder dem Endbenutzer bereitzustellen 25 Switch nicht als Regal oder Arbeitsplatz nutzen Achtung Auf Schieber Schienen montiertes Ger t ist nicht als Regal oder Arbeitsbe reich zu nutzen Die Schienen sind nicht daf r bestimmt die Einheit aus dem Gestell weg zu ziehen Sie sind nur f r die permanente Installation an einem endg ltigen Stan dort gedacht nicht f r Instandhaltung und Wartung 2 26 WEEE Direktive Gem WEEE Directive 2002 96 EC m ssen alle elektrischen und elektronischen Abfallger te EEE separat gesammelt und nic
101. tiuni de instalare Cititi toate instructiunile de instalare inainte de a conecta 2 Accidentare cauzat de greutate A Apelati la un num r suficient de persoane pentru a ridica in sigurant acest produs 40 lbs 40 70 lbs 70 121 lbs gt 121 lbs lt 18 kgs 18 32 kgs 32 55 kgs gt 55 kgs 3 Echipament greu Acest echipament este foarte greu si trebuie s fie mutat folosind un dispozitiv meca nic de ridicare pentru a evita producerea de leziuni 4 Instalarea intr un loc cu acces limitat Aceast unitate este destinat instal rii intr un loc cu acces limitat 5 Risc de soc electric Risc de soc electric Odat ce modulul ventilator este indepartat pinii electrici sunt accesibili in cavitatea modulului NU introduceti instrumente sau parti din corp in cavitatea modulului ventilator 6 Temperatura in exces A Acest echipament nu trebuie s fie ac ionat ntr o zon unde temperatura ambiant N dep seste valoarea maxima recomandat 45 C 113 F In plus pentru a asigura un flux de aer adecvat l sati un spatiu liber de cel putin 8 cm 3 inchi in jurul fantelor de ventilare Mellanox Technologies 95 Rev 2 2 7 Suprapunerea cadrului Cadrul nu trebuie s fie suprapus peste niciun alt echipament n cazul in care cadrul cade poate cauza leziuni corporale si deterior ri ale echipamentului Acest produs include o surs de alimentare suplimentar sau un spatiu gol
102. tus LED status LED The system status LEDs should display as follows when the switch is plugged in within five minutes the switch status LED should light up green the FAN LED should light up green If the FAN LED shows red troubleshoot the fan module See page 64 the PSI LED for the plugged in PS unit should light up green the PS2 LED for the second PS unit should light up green only if a second PS unit is installed in the switch for redundancy and it is connected to a power source if two PS units are installed and only one PS unit is connected to a power supply the PS2 LED will be red if only one PS unit is installed in the switch the PS2 LED will be off As long as there is power to the switch one PS unit is connected and the switch is booted up and running the switch status LED should be green 5 If the status LED shows red after five minutes unplug the switch and call your Mella 4 representative for assistance If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See page 64 If the PS LEDs are not green this indicates a problem with the power supplies The switch is operational only if at least one of the PS LEDs is green Mellanox Technologies 21 Rev 2 2 Basic Operation 2 2 2 Switch Status LED Figure 6 Switch Status LED BN B NE B BN 7 VW Ew E EE SM RH p
103. upply cord Wait at least 5 seconds between inserting the first power supply cord and inserting the second power supply cord If you do not wait long enough before inserting the second power cord the LED for the second PSU may light up red 11 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Figure 28 Status LEDs 5 Minutes After Power On Mellanox Mellanox B A ma Se IR Ps 1 BE PS2 PS2 un O E up UID RST RST Switch with 2 PSUs Switch with 1 PSU Any yellow or red status LEDs are cause for concern and must be dealt with immedi ately It can take up to 5 minutes to boot up during which time the status LED may indicate red 12 You can start connecting all of the cables to the switch 3 5 Grounding the Switch Check to determine if your local or national electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the ground screw make sure that the rack is properly 44 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Some national and or local codes may require IT components to be bonded and exter ZA nally grounded not
104. us according to the LED indication for InfiniBand protocol 1 This feature will be available in a coming software release 26 Mellanox Technologies SwitchX 36 Port QSFP FDR InfiniBand System Hardware User Manual Rev 2 2 Figure 14 Port LEDs TPP RPE REESE 1A V2 Ws Mellanox E ZZ SM Table 11 Port Connector Physical and Logical Link Assignments for Ethernet Mode LED Configuration LED Description Off Physical link is down Default Solid Green Physical link is up with no traffic Flashing Green Physical link is up with traffic Flashing Orange Physical errors Table 12 Connector Physical and Logical Link Assignments for IB Mode LED Status LED Description Off No power to the port Solid Green Logical link is up Flashing Green Data activity flashing speed is proportional to data transfer speed Solid Orange Physical link is up Flashing Orange A problem with the physical link Usually the SM is down or not enabling the port In IB mode the LED indicator corresponding to each data port will light orange when the phys ical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a logical connec tion is made the LED will change to green When data is being transferred the light will blink green 2 4 Air Flow
105. ustausch Warnung Nur durch von UL anerkannten Akkus ersetzen die f r maximalen anor malen Ladestrom von nicht weniger als AmA zertifiziert sind Es besteht Explosiionsgefahr wenn der Akku durch einen Akku eines falschen Typs ersetzt wird Akkus gem den Anweisungen entsorgen 18 UL und CSA zertifiziertes Netzkabel F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A 19 Hoher Ableitstrom WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 20 Infomationen zum Erdungsanschluss Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation Mellanox Technologies 83 Rev 2 2 21 Installationscodes Dieses Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code Fiir Nordamerika miissen in Ubereinstimmung mit den gel tenden Vorschriften in der US amerikanischen Na
106. x configuration wizard You must perform this configuration the first time you Do you want to use the wizard operate the switch or after resetting the switch Type for initial configuration yes and then press lt Enter gt Step 1 Hostname switch If you wish to accept the default hostname then press lt Enter gt Otherwise type a different hostname and press lt Enter gt Step 2 Use DHCP on mgmt0 Perform this step to obtain an IP address for the switch interface no yes mgmt0 is the management port of the switch If you wish the DHCP server to assign the IP address type yes and press lt Enter gt If you type no no DHCP then you will be asked whether you wish to use the zeroconf configuration or not If you enter no no Zeroconf then you need to enter a static IP and the session will continue Step 3 Enable IPv6 yes The management interface will be able to use IPv6 addresses Step 4 Enable IPv6 auto config This turns on auto configuration of the IPv6 addresses SLAAC on mgmt0 interface This is unsuitable for DHCPv6 no Step 5 Enable DHCPv6 on To enable DHCPv6 on the MGMTO interface mgmt0 interface no Mellanox Technologies 51 Rev 2 2 Installation Table 16 Configuration Wizard Session DHCP Sheet 2 of 2 Configuration Example Wizard Session Display Comments Step 6 Admin password Press To avoid illegal acces
107. xigences de l US National Electrical Code et du Code canadien de l lectricit 23 Interconnexion des unit s Les c bles de connexion aux interfaces RS232 et Ethernet de l appareil doivent tre certifi UL de type DP 1 ou DP 2 Note en cas d installation sur un circuit dont la puissance n est pas limit e Protection contre les surintensit s le c blage de l immeuble doit int grer un dispositif certifi de protection contre les surintensit s calibr 20 A et ais ment accessible 24 Exposition des rayonnements dangereux Attention l utilisation de commandes l ajustement ou la conduite de proc dures autres que celles indiqu es ici pourraient conduire a une exposition dangereuse a des rayonnements Pour les produits disposant de ports optiques lasers IEC 60 825 1 1993 A1 1997 A2 2001 et EN 60825 1 1994 A1 1996 A2 2001 Pour les produits disposant de ports optiques te PRODUIT LASER DE CLASSE 1 et reference aux plus r centes normes relatives aux 25 Armoire de protection appropri e Une enceinte appropri e aux niveaux lectrique m canique et incendie doit tre four nie par le constructeur du produit fini ou par l utilisateur Mellanox Technologies 79 Rev 2 2 26 Ne pas utiliser comme tag re ou plan de travail Attention un quipement coulissant ou mont sur rail ne doit pas servir d tag re ni de plan de travail Les rails ne sont pas destin s a faire coulisser l unit

Download Pdf Manuals

image

Related Search

Related Contents

EnjoyTone W30C User Manual English  Specific Product Documentation  Traulsen RBC200RT Blast Chiller Roll Through Unit  User Manual PZ143E - Physik Instrumente  FROSTIG GB  - VAC Ulta  

Copyright © All rights reserved.
Failed to retrieve file