BridgeX GT Series BX5020 Gateway IB to Ethernet User Manual
Contents
1. Port protocol and speed Figure 4 Example of a Setup Showing a Port Group Ethernet T 2 3 4 5 6 7 1 2 4 5 6 7 8 mummu mmm o o muija uu u O m mmm r ma um EA Downlink Gateway Uplink IB 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 EEE EERE 8 8 8 5 1505 8 185 05 HS s E EN L NND NS EE UM EN NO UL URL UR ILL InfiniBand Swich 1 2 3 4 5 6 g 8 9 10 12 13 14 15 16 17 18 QSFP Cable Rev 1 4 Installation and Basic Operation 2 2 1 2 2 Installation and Basic Operation Unpacking the Gateway Before you install your new BX5020 unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping p If anything is damaged or missing contact your customer representative immediately See the package contents in Section 2 3 BridgeX Gateway Hardware Overview The figures below show dual hot swap power supplies 1 I2C connector various status LEDs and the hot swap fan module on the power side The RS232 CONSOLE 2 Management GigE connec tors one USB connector and various status LEDs on the connector IB ETH side Figure 5 Gateway System Connector and Power Side Panels Power Side STATUS PSU1 E PSU L FAN RST2. O System Status Indicators AA STATUS PSU 1 PSU 2 FAN RST PSU Status Indicators 2 2 5 1 System Status Indicators Status indicators are located on the right side of power side panel and on the left side of the con nector IB ETH side panel Both of these system status indicators are three color LEDs display ing the same status on both sides of the gateway The following status conditions are possible 16 Mellanox Technologies Rev 1 4 Table 6 System Status LED Configurations LED Configuration STATUS System Health LED Green OK The system is up and running Yellow Error A fault in the system most likely the firmware did not BOOT properly Red Major Error Possible damage can result to the gateway Turn off immediately For example bad FW can t boot or overheated Note When the system is turned on the red LED will light up for up to two minutes until the CPU is up and running Off Off The system has no power If the STATUS LED shows red after five minutes unplug the gateway and call your Mellanox representative for assistance If the FAN LED shows red troubleshoot the fan module If the gateway shuts down due to over temperature unplug the gateway wait 5 minutes and replug in the gateway 2 2 5 2 Fan Status Indicators The fan unit is located in the center of the power side panel The fan unit has a single 2 color LED to the right of the fa
3. Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 4972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE DirectG InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale InfiniPCI PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricITTM and SwitchXTM are trademarks of Mellanox Technologies Ltd BridgeX BX5020 Gateway IB to Ethernet 2 Mellanox Technologies Document Number 3276 Rev 1 4 Table of Contents Table of Contents 3 List of Tables 5 List of Figures 6 Revision History 7 About this Manual 8 Intended Audience 8 Related Documentation 8 Conventions 8 Gateway Products Covered in this User Manual 9 Mellanox Part Numbering Legend 9 Chapter 1 Overview 10 1 1 Features 11 1 2 Serial Number and Product Version Information 12 1 3 Gateway Port Groups 12 Chapter 2 Installation and Basic Operation 14 2 1 Unpacking the Gateway 14 2 2 BridgeX Gateway Hardware Overview 14 2 2 1 Downlink Ports 14 2 2 2 Uplink Ports 15 2 2 3 Configuring the Port Gateway Groups 15 2 2 4 Making Connections to Other Formats 15 2 2 5 Status LEDs 16 2 2 6 Port Connector LEDs 19
4. 2 2 1 Connector IB ETH Side All connectivity is via the connector panel All connectors can support active cables Downlink Ports There are four downlink ports per gateway port group These ports have SFP connectors that sup port both direct attach copper cables and optical cables by using SR or LR modules When configured for Ethernet only 3 of the 4 downlink ports per gateway group are used and all downlink ports within the gateway group must have the same speed that is 1Gbps or 10Gbps See Table 5 for the port numbers of the active Ethernet ports 14 Mellanox Technologies Rev 1 4 2 2 2 Uplink Ports There is one uplink port per gateway port group This port has a QSFP Connector These connec tors have support for powered cables and media adapter circuits All of the Uplink ports must be InfiniBand 2 2 8 Configuring the Port Gateway Groups See the FabricIT BXM Management Software CLI User Manual for CLI commands to configure the gateway You can also configure the gateway using the WebUI 2 2 4 Making Connections to Other Formats Hybrid CX4 to QSFP cables are supported on the uplink side SR and LR modules can be used on the downlink side The following downlink options are also supported SFP modules for 1 10GbE 2 2 4 1 SFP Transceiver Module The gateways are shipped without optical modules Approved Mellanox modules must be pur chased from Mellanox The OPNs for the approved Mellanox modu
5. 2 2 7 Reset Button 19 2 2 8 AirFlow 20 2 2 9 Management Interfaces 21 2 3 Package Contents 22 24 Gateway Installation and Operation 22 2 4 1 Installation Safety Warnings 23 2 4 2 Mechanical Installation 26 2 4 3 Grounding the Gateway 30 2 4 4 Disassembling the Gateway from the Rack 30 2 4 5 Power Connections and Initial Power On 31 2 4 6 Shut Down Procedure 32 2 4 7 Extracting and Inserting the Power Supply Unit 32 2 4 8 Cable Installation 33 2 4 9 Extracting and Inserting the Fan Unit 35 Chapter 3 Management and Software 37 3 1 FabricIT 37 Chapter 4 Troubleshooting 38 Appendix A Specifications 41 Appendix B EMC Certifications 43 B l EMC Certifications 43 Appendix C Interface Connector Pinouts 44 El SFP Interface 44 C2 RS232 to RJ45 Connector Interface 46 Mellanox Technologies 3 Rev 1 4 C 3 I2C RJA5 Connector Interface Pinout 47 C 4 QSFP Connector Pinout 48 Appendix D Replacement Parts Ordering Numbers 50 Appendix E Avertissements de s curit d installation French 52 Appendix F Installation Sicherheitshinweise German 55 Appendix G Advertencias de seguridad para la instalaci n Spanish 58 Appendix H SFP Modules 62 H 1 Product Features 62 H 2 Applications 62 H 3 Dimensions 62 H 4 Pin Descriptions 63 H 5 Power Dissipation 64 4 Mellanox Technologies Rev 1 4 List of Tables Table 1 Revision History Table 7 Table2 Reference Documents 8 Table3 Gateway Products Covered by this User s
6. Flashing Orange A problem with the physical link 2 2 7 Reset Button On the connector IB ETH side panel under the system LEDs is a reset button This reset button requires a tool to be pressed a paper clip will do DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip Mellanox Technologies 19 Rev 1 4 Installation and Basic Operation 2 2 8 20 Figure 12 Reset Button Reset button to reset the main and manage ment CPUs and to reset the existing password back to the default The password is returned to the default password admin This button resets both the CPU of the gateway device and the CPU of the management module It thereby resets all of the ports by bringing them down and powering them up when the button is pushed A quick push of this button performs this reset When the button is held down for 5 sec onds the gateway is reset and the password is changed back to the default password admin Air Flow These gateways can come with two air flow patterns The two patterns are connector IB ETH side inlet to power side outlet Power side inlet to connector IB ETH side outlet The air flow is specified in the product model number See Mellanox Part Numbering Legend on page 9 On the PSUs and fan modules the air flow direction can be seen on the power side panel Table 10 A
7. Transmitter Non Inverted DATA in AC Coupled 19 TD Transmitter Inverted DATA in AC Coupled 20 VeeT Transmitter Ground Common with Receiver Ground a Circuit ground is internally isolated from chassis ground b Traurr is an open collector drain output which should be pulled up with a 4 7k 10k Ohms resistor on the host board if intended for use Pull up voltage should be between 2 0V to Vcc 0 3V A high output indicates a transmitter fault caused by either the TX bias cur rent or the TX output power exceeding the preset alarm thresholds A low output indicates normal operation In the low state the output is pulled to 0 8V Laser output disabled on TDIS gt 2 0V or open enabled on TDIS 0 8V d Should be pulled up with 4 7kQ 10kQ on host board to a voltage between 2 0V and 3 6V MOD ABS pulls line low to indicate module is plugged in e LOS is open collector output Should be pulled up with 4 7kQ 10kQ on host board to a voltage between 2 0V and 3 6V Logic 0 indicates normal operation logic 1 indicates loss of signal e Mellanox Technologies 45 Rev 1 4 C 2 RS232 to RJ45 Connector Interface Figure 26 RJ45 Connector Pinout CONSOLE Pin 1 RJ45_RTS RJ45_DTR RJ45_TXD GND GND RJ45_RXD GND GND Pin 8 Looking into the Socket 46 Mellanox Technologies Rev 1 4 C 3 I2C RJ45 Connector Interface Pinout Figure 27 I2C Pinouts PC Pin 1 RJ45 SDA GND GN
8. e d une tension nominale de 125 V 15 A avec une longueur minimale de 1 5 m 6 pieds et d une longueur maxi male de 4 5 m 18 pieds Pour les prises lectriques en Europe choisissez un cordon lectrique harmonis inter nationalement et marqu lt HAR gt 3 conducteurs d un diam tre de fil minimum de 0 75 mm2 d une tension nominale de 300 V avec une gaine isol e en PVC Le cordon doit avoir une fiche moul e d une tension nominale de 250 V et d une intensit nominale de 10 A 54 Mellanox Technologies Rev 1 4 Anhang F Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Stapeln des Chassis Das Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herun terf llt kann es zu Verletzungen und Besch digungen an Ger ten f hren 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die
9. quipement doit tre install conform ment aux sp cifications du Code Electrique National Am ricain et du Code Electrique Canadien Mellanox Technologies 53 Rev 1 4 15 Interconnexion des unites INTERCONNECTION OF UNITS Les c bles de branchement l unit RS232 et les interfaces Ethernet doivent tre certi fi s UL de type DP 1 ou DP 2 Note lorsqu il existe dans un circuit non LPS Protection contre la surintensit Un appareil de protection r pertori facilement accessible contre la surintensit du circuit de branchement et calibr 20A doit tre incorpor dans le c blage lectrique du b timent 16 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 17 S assurer que les enceintes sont appropri es Des enceintes lectriques m caniques et incendie adapt es doivent tre fournies par le fabricant du produit final ou par l utilisateur final 18 Cordons lectriques CA homologu s UL Pour les prises lectriques en Am rique du Nord choisissez un cordon lectrique homologu UL et certifi CSA 3 conducteurs 18 AWG termin par une fiche moul
10. 20 A must be incorporated in the building wiring 12 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 24 Mellanox Technologies Rev 1 4 13 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 14 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical LA codes 15 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 16 UL Approved AC Power Cords For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 18 AWG terminated in a molded on plug cap rated YN at 125 V 15 A with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt HAR gt 3 conductor minimum 0 75 mm wire rated at 300 V with a PVC insulated jacket The cord must have a molded on plug cap rated 250 V 10 A 17 Do Not Use The Switch As A Shelf Or Work Space S Caution Slide rail mounted equipment is not t
11. 56 50 5 z gt de 2 WN 14 80 13 60 j E Mellanox Technologies 13 70 E MTSFPP SR Module 11 85 62 Mellanox Technologies J Rev 1 4 H 4 Pin Descriptions Figure 30 Rear View of Module With Pin Placement Top 13 70 Ib D w gt a N E gt 2 a 2l Table 15 SFP Pinout Pin Symbol Name Description l VeeT Transmitter Ground Common with Receiver Ground 2 TX Fault Transmitter Fault 3 TX Disable Transmitter Disable Laser output disabled on high or open 4 SDA 2 wire Serial Interface Data Line 3 SCL 2 wire Serial Interface Clock Line Y 6 MOD_ABS Module Absent Grounded within the module Y 7 RSO No connection required 8 RX_LOS Loss of Signal indication Logic O indicates normal operation 9 RSI No connection required 10 VeeR Receiver Ground Common with Transmitter Ground H VeeR Receiver Ground Common with Transmitter Ground Mellanox Technologies 63 Rev 1 4 Pin Symbol Name Description 12 RD Receiver Inverted DATA out AC Coupled 13 RD Receiver Non inverted DATA out AC Coupled 14 VeeR Receiver Ground Common with Transmitter Ground 15 VccR Receiver Power Supply 16 Vc
12. Ger t muss installiert sein entsprechend auf die neueste Version des Landes National Electrical Code F r Nordamerika m ssen in bereinstimmung mit den gel LA tenden Vorschriften in der US amerikanischen National Electrical Code und dem Canadian Electrical Code 17 Zusammenschaltung von EINHEITEN INTERCONNECTION OF UNITS Kabel f r den Anschluss an das Ger t RS232 und Ethernet Schnittstellen m ssen UL zertifiziert Typ DP 1 oder DP 2 Hinweis wenn nicht mit Wohnsitz in LPS Schal LA tung berstromschutz Eine leicht zug ngliche Auflistung Abzweigleitung berstrom Schutzeinrichtung 20 A bewertet werden m ssen in dem Geb ude Verkabelung Mellanox Technologies 57 Rev 1 4 Appendix G Advertencias de seguridad para la insta laci n Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instruc ciones de instalaci n 2 Instalaci n en un lugar con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido 3 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alre dedor de las aberturas de ventilaci n 4 Apilamiento del chasis Los chasis no se deben apilar sobre otros equipos La ca da del chasis
13. Manual 9 Table4 Part Numbering Legend 9 Table5 BX5020 Gateway Port Grouping Protocols and Speeds 13 Table6 System Status LED Configurations 17 Table 7 Fan LED Configurations 17 Table 8 PSU Status LED Configurations 18 Table 9 Port Connector LED Assignment 19 Table 10 Air Flow Direction 20 Table 11 Gateway and Installation Kit Options 26 Table 12 BX5020 1S Specification Data 41 Table 13 SFP Pinout 44 Table 14 Replacement Parts Ordering Numbers 50 Table 15 SFP Pinout 63 Mellanox Technologies 5 Rev 1 4 List of Figures Figure 1 MBX 5020 Gateway 10 Figure 2 Port Numbering 11 Figure3 Generic Product Label 12 Figure4 Example of a Setup Showing a Port Group 13 Figure5 Gateway System Connector and Power Side Panels 14 Figure 6 SFP Transceiver Module 15 Figure 7 Module with Locking Mechanism Closed 16 Figure 8 Module with Locking Mechanism Open 16 Figure9 Power and System LEDs 16 Figure 10 BX Gateways Power Side Panel 18 Figure 11 Connector Port Identification 19 Figure 12 Reset Button 20 Figure 13 Management Interfaces 21 Figure 14 Installation Kit Parts 27 Figure 15 Making Room for the Power Cord 28 Figure 16 Screwing on the Bracket 28 Figure 17 Screwing on the Rail 28 Figure 18 Clipping in the Caged Nuts 29 Figure 19 Caged Nut Locations 29 Figure 20 Ground Connection 30 Figure 21 Two Power Inlets Electric Caution Notifications 32 Figure 22 PSU Pulled Out 33 Figure 23 Top and Bottom Ports
14. Technologies 23 Rev 1 4 Installation and Basic Operation 6 Copper InfiniBand Cable Connecting Disconnecting attached to or detached from the connectors Refer to the cable manufacturer for spe Copper InfiniBand cables are heavy and not flexible as such they should be carefully A cial warnings and instructions 7 Rack Mounting and Servicing to ensure that the system remains stable In general you should fill the rack with eguip When this product is mounted or serviced in a rack special precautions must be taken A ment starting from the bottom to the top 8 Leakage gt 3 5mA WARNING High leakage current Earth connection essential before connecting sup A ply 9 Add GND Connection Before connecting this device to the power line the protective earth terminal screws of this device must be connected to the protective earth in the building installation 10 Installation Codes This device must be installed according to the latest version of the country national electrical codes For North America equipment must be installed in accordance to the applicable requirements in the US National Electrical Code and the Canadian Electri cal Code 11 Interconnection of Units Cables for connecting to the unit RS232 and Ethernet Interfaces must be UL certified L type DP 1 or DP 2 Note when residing in non LPS circuit Overcurrent Protection A readily accessible Listed branch circuit overcurrent protec tive device rated
15. at www mellanox com related docs user manuals Regulatory and Compliance Guide pdf EMC Statements are also in the Regulatory and Compliance Guide Mellanox Technologies 43 Rev 1 4 Appendix C Interface Connector Pinouts C 1 SFP Interface Figure 25 SFP Connector Pinout Rear View of Module With Pin Placement VeeR o co O Lo 0 N mc ier an s S a Z Table 13 SFP Pinout Pin Symbol Name Description 1 VeeT Transmitter Ground Common with Receiver Ground 2 TX Fault Transmitter Fault 3 TX Disable Transmitter Disable Laser output disabled on high or open 4 SDA 2 wire Serial Interface Data Line Y 5 SCL 2 wire Serial Interface Clock Line d 6 MOD ABS Module Absent Grounded within the module Y 7 RSO No connection required 8 RX LOS Loss of Signal indication Logic 0 indicates normal operation 9 RSI No connection required 10 VeeR Receiver Ground Common with Transmitter Ground 11 VeeR Receiver Ground Common with Transmitter Ground 12 RD Receiver Inverted DATA out AC Coupled 13 RD Receiver Non inverted DATA out AC Coupled 14 VeeR Receiver Ground Common with Transmitter Ground 15 VccR Receiver Power Supply 44 Mellanox Technologies Rev 1 4 Table 13 SFP Pinout Pin Symbol Name Description 16 VccT Transmitter Power Supply 17 VeeT Transmitter Ground Common with Receiver Ground 18 TD
16. bring down the system before you replace the PSU 32 Mellanox Technologies Rev 1 4 Power supply units have directional air flows similar to the fan module The Fan mod ule airflow must coincide with the airflow of all of the PSUs If the PSU airflow direc tion is different from the fan module airflow direction the system will not cool properly 2 4 7 1 To extract a PSU Figure 22 PSU Pulled Out Power Side Remove the power cord from the power supply unit 2 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PSU unseats the PSU status indicators will turn off S Remove the PSU 2 4 7 2 ToinsertaPSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Do not attempt to insert the PSU with a power cord connected to it Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirm ing proper installation Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The PSU indicator should light green If not or if it is red repeat the procedure starting with Section 2 4 7 1 2 4 8 Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable press the con nector into
17. connect the rail slides Check that both sides of the gateway left and right are the same level in the rack 28 Mellanox Technologies J Rev 1 4 Figure 18 Clipping in the Caged Nuts Qu EE The caged nuts are separated by pur a single space 5 Clip 4 more caged nuts into the holes in the rack you will be using to connect the brackets Check that both sides of the gateway power side and connector IB ETH side are at the same level in the rack 6 Using two of the bolts for each rail slide install the rail slides Figure 19 Caged Nut Locations The caged nuts are separated by a single space The rail slides are to be installed on the side of the rack where the gateway will be recessed into the rack 7 Tighten the bolts to 9 2 Nm or 81 5 pound inches If the power cable is on this side of the gateway feed the power cable into the slot in the rail slide before screwing it to the vertical support 8 Slide the gateway catching the rail slides into the rails on the gateway If you extended the bracket past the gateway to use the slot for the power cord now is the time to put the power cord through the slot 9 Screw in the bolts into the caged nuts and tighten the bolts to 9 2 Nm or 81 5 pound inches 10 Ground the gateway See Section 2 4 3 for grounding instructions 11 Plug in the power cord 12 You can start connecting all of the cables to the gateway There is no On Off s
18. of EoIB protocols EoIB capable ConnectX and ConnectX 2 adapters present standard Ethernet Sockets software interfaces to server applications providing transpar ent connectivity for existing Enterprise Data Center EDC and High Performance Computing HPC applications to Ethernet LAN ConnectX and ConnectX 2 adapters also present a standard InfiniBand Verbs software interface to server applications for Inter Process Communication IPC This allows Ethernet and InfiniBand payloads to be carried over a common converged high performance 40Gbps InfiniBand fabric without any changes to existing server applications Data on the converged fabric requiring access to Ethernet LAN is bridged by the gateway preserving existing LAN investments and management practices The BridgeX gateway along with ConnectX and ConnectX 2 adapters converge different networks onto a single fabric reducing the number of adapters cables and switch ports by a factor of three and reducing capital expenditure The high port density of the gateway within a 1U form factor low power consumption and ease of management also reduces operating expenses Figure 1 MBX 5020 Gateway A A D MGT CONSOLE j i lt A VA A JA a lt B CA A A a E YA A A A lt EM E MR 2 i i E ES E C O O Port grou ort groups Product version information tab location 10 Mellanox Technologies Rev 1 4 Figure 2 Port Numbering 1 4 Features The BridgeX
19. redundante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeckung nicht sicher festsitzt oder entfernt wurde 5 BeiGewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 6 AnschlieDen Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers 7 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind besondere Vor sichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen Mellanox Technologies 55 Rev 1 4 5 6 8 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 11 Richtigen Schutz sicherstellen Geeigneter elekrischer mechanischer und Feuerschutz sind vom Hersteller des Endprodukts o
20. the port receptacle until the connector is firmly seated The indicator above each port will light green when the physical connection is established that is when the unit is powered on Mellanox Technologies 33 Rev 1 4 Installation and Basic Operation and a cable is plugged into the port with the other end of the connector plugged into a functioning port After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When there is data transfer the light will blink green When installing cables make sure that the latch engages gt 4 Always install and remove cables by pushing or pulling the cable and connector in a straight line with the gateway ha To remove disengage the locks and slowly pull the connector away from the port receptacle The LED indicator will turn off when the cable is unseated Care should be taken not to impede the air exhaust flow through the ventilation holes next to the ports Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack Cables especially long copper cables can weigh a substantial amount Make sure that the weight of the cable is supported on its own and not hanging from the gateway These gateways are able to support direct attached copper cables optical FC cables and cables with media adapters at various lengths and gauges Check www mellanox c
21. 1 II II XI II II II II II IX II IX II xx xx XX 11 11 11 yy yy L o DI el II II II II II II II II II II II 11 11 11 II II II II II II II II II II 11 II II II 11 II Figure 24 Fan Unit Pulled Out Power Side Fan Unit 1 Using two hands push both latch releases in simultaneously while pulling the fan module out of the gateway As the fan unseats the fan status indicator will turn off 2 Pull the fan unit out Mellanox Technologies 35 Rev 1 4 Installation and Basic Operation To insert a FAN Unit Make sure that the new fan unit has the correct air flow direction This can be verified by checking the label on the fan unit and comparing it to the air flow label on the PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely The green fan status indicator should light If not extract the fan unit and reinsert it After two unsuccessful attempts to install the fan unit power off the gateway before attempting any system debug 36 Mellanox Technologies Rev 1 4 3 Management and Software The BX 5020 series gateway contains a management module that gives the user the ability to man age the gateway This management module runs Mellanox proprietary software For information re
22. 1 and PSU 2 Power supply blank MIS000055 Rack installation kit standard depth gateway standard MIS000085 rack Rack installation kit Short gateway standard rack MIS000083 Rack installation kit short gateway short rack MIS000079 Harness RS232 to DB9 2M long cable HAR000028 Fan Unit for Power side to connector IB ETH side air MIS000051 flow Fan Unit for connector IB ETH side to Power side air MIS000082 flow Power cord Type C13 C14 ACC000251 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany Neth ACC000207 erlands Russia Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 50 Mellanox Technologies p T Rev tt Mellanox Technologies 51 Rev 1 4 Appendix E Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 45 C 113 F Un flux d air de 200LFM cette temp rat
23. 34 Figure 24 Fan Unit Pulled Out 35 Figure 25 SFP Connector Pinout Rear View of Module With Pin Placement 44 Figure 26 RJ45 Connector Pinout 46 Figure 27 I2C Pinouts 47 Figure 28 QSFP Connector Pinout 48 Figure 29 Pinout Looking Into the Rear of the Connector and the Front of the Cage 49 Figure 30 Rear View of Module With Pin Placement 63 6 Mellanox Technologies Rev 1 4 Revision History Table 1 Revision History Table Date Revision Description August 2011 1 4 Updated Power numbers Added Power as heat in BTUs hr June 2011 1 3 Removed references to FC June 2011 1 2 Minor fixes Feb 2011 1 1 New power numbers New I2C section Sept 2010 1 0 Initial release Mellanox Technologies 7 Rev 1 4 About this Manual This manual describes the installation and basic use of Mellanox BridgeX MBX5020 Gateway products and development platforms based on the MT68102 BridgeX gateway device Intended Audience This manual is intended for users and system administrators responsible for installing and setting up BridgeX gateways from InfiniBand to Ethernet fabrics and networks The manual assumes familiarity with the InfiniBand Architecture Specification as well as the Ethernet Architecture Related Documentation Additional documentation available from Mellanox is provided in Table 2 Table 2 Reference Documents BridgeX Programmer s Reference Manual User guide describing the interfa
24. D GND GND RJ45 SCL GND GND Pin 8 Looking into the Socket Mellanox Technologies 47 Rev 1 4 C 4 QSFP Connector Pinout Figure 28 QSFP Connector Pinout 20 GND GND 12 _ 21 Rx2n Rxin L 18 X 22 Rx2p Rxip 17 2 EN GND 16 24 15 Rx4n Rx3n I 14 Rx4p Rx3p 13 Ium GND 27 ModPrsL spa 2 11 MI IntL Se 10 VccTx Vcc Rx 39 Vcc1 ResetL 9 31 PMode ModSelL 33 Tx3p Tx4p 6 Tx3n Tx4n 5 35 GND GND 36 Txip Tx2p 3 E Tx1n Tx2n 2 48 Mellanox Technologies Rev 1 4 Figure 29 Pinout Looking Into the Rear of the Connector and the Front of the Cage Top 18 35 View into Rear of Connector GND TX2n TX2p GND TX4n TX4p GND ModSelk ResetL VecRx SCL SDA GND RX3p RX3n GND RX1p RX1n GND View into Front of Cage GN poWd1 AND AA UEXL N dixi wa 49 Mellanox Technologies Rev 1 4 Appendix D Replacement Parts Ordering Numbers Table 14 Replacement Parts Ordering Numbers Part Description Product Number Power Supply Unit PSU connector IB ETH side to Power MIS000053 side airflow This Replacement part is for both the PSU 1 and PSU 2 Power Supply Unit PSU Power side to connector IB ETH MIS000054 side airflow This Replacement part is for both the PSU
25. Gateway series has the following features Uplink ports 4 ports of 40 G InfiniBand QSFP Downlink ports 12 ports of 1 GigE 10 GigE Ethernet Down link ports are configured in groups of 3 Down link ports within a group must run the same protocol down link port group running Ethernet must run the same speed on all ports within the group 10Gbps or 1Gbps 1024 virtual NICs per Ethernet port Total of 16 000 MAC addresses Ethernet ports IEEE 802 3ae 10Gigabit Ethernet support IEEE 802 3z Gigabit Ethernet IEEE 802 1D Spanning Tree IEEE 802 1p QoS COS IEEE 802 1Q VLAN Tagging IEEE 802 1AB Link Layer Discovery IEEE 802 3ad Link Aggregation with LACP EEE 802 3x Flow Control Per Priority Flow Control Virtual lanes support Jumbo Frames up to 9K support This feature will be available to customers in the near future Connectors and Cabling SFP connectors for downlink 10GigE ports Optical transceiver modules for SR and LR for Ethernet Optical modules for FC Mellanox Technologies 11 Rev 1 4 Overview 1 2 1 3 Indicators Per port status LEDs Link Activity System status LEDs system fans power supply Ethernet indicators for downlink port groups Management Options 2 1000BASE T Ethernet ports Power Supply Dual redundant slots e 1 PSU is required for device functioning Serial Number and Product Version Information The serial number an
26. Mellanox TECHNOLOGIES BridgeX GT Series BX5020 Gateway IB to Ethernet User Manual P N MBX5020 1SFR Rev 1 4 www mellanox com Rev 1 4 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE
27. Schocks Entferrnen des e lectriques Le de branchment dent Netzsteckers elnes Netzteils d une seule alimentation stabi spannungsfrei Um alle Einhi lise e ne de branch uniquement Disconnect all power supplies to eten spannungsfrei zu machen qu un module Alimentation Sta ensure a powered down state inside sind die Netzstecker aller bilise e Pour isoler com of the gateway Netzteile zu entfernen pletement le module en cause Il faut de brancher toutes les alimen tations stabilise es 2 4 6 Shut Down Procedure If the Gateway is powered off for any reason without using the proper shut down pro cedure the Flash memory may be corrupted If the Flash memory is corrupted the Gateway may fail to boot Should this happen call your Mellanox representative for assistance To run the gateway through a power cycle Run config reload halt 2 Wait long enough to let the gateway halt and verify there is no ping to it 3 Unplug the power cord Should the gateway fail to boot upon reinsertion of the power cord burn the latest FW image to the Flash memory 2 4 7 Extracting and Inserting the Power Supply Unit With both power supplies installed in the redundant configuration either PSU may be extracted without bringing down the system Make sure that the PSU that you are NOT replacing is showing all green for both the PSU and status indicators If your system does not have a redundant power supply you must
28. a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suminis trar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el T CNICO DE SERVICIO se encargar de instalar un CON DUCTOR DE CONEXI N A TIERRA DE PROTECCI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecci n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 15 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c digos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 16 Interconexi n de unidades Los cables para la conexi n con las interfaces RS232 y Ethernet de la unidad deben estar homologados por UL tipo DP 1 o DP 2 Nota cuando residen en circuito no de tipo LPS Protecci n contra sobrecargas Al cableado de
29. boards power supplies and fan tray modules for proper Caution The gateway will automatically power on when AC power is applied There A insertion before plugging in a power cable 2 4 1 Installation Safety Warnings For Safety Warnings in French see Section E Avertissements de s curit d installation French on page 52 for German see Section F Installation Sicherheitshinweise German on page 55 and for Spanish see Section G Advertencias de seguridad para la instalaci n Spanish on page 58 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature ing the maximum recommended 45 C 113 F Moreover to guarantee proper air This equipment should not be operated in an area with an ambient temperature exceed A flow allow at least 8cm 3 inches of clearance around the ventilation openings 3 Stacking the Chassis The chassis should not be stacked on any other eguipment If the chassis falls it can cause bodily injury and eguipment damage 4 Redundant Power Supply Connection Electrical Hazard This product includes a blank cover over the space for the redundant power supply Do not operate the product if the blank cover is not securely fastened or if it is removed 5 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables Mellanox
30. cT Transmitter Power Supply 17 VeeT Transmitter Ground Common with Receiver Ground 18 TD Transmitter Non Inverted DATA in AC Coupled 19 TD Transmitter Inverted DATA in AC Coupled 20 VeeT Transmitter Ground Common with Receiver Ground P Circuit ground is internally isolated from chassis ground b Traurr is an open collector drain output which should be pulled up with a 4 7k 10k Ohms resistor on the host board if intended for use Pull up voltage should be between 2 0V to Vcc 0 3V A high output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed ing the preset alarm thresholds A low output indicates normal operation In the low state the output is pulled to 0 8V c Laser output disabled on TDIS gt 2 0V or open enabled on TDIS 0 8V d Should be pulled up with 4 7kQ 10kQ on host board to a voltage between 2 0V and 3 6V MOD ABS pulls line low to indicate module is plugged in e LOS is open collector output Should be pulled up with 4 7kQ 10k on host board to a voltage between 2 0V and 3 6V Logic 0 indicates normal operation logic 1 indicates loss of signal H 5 Power Dissipation Max power dissipation 1 00 W 64 Mellanox Technologies
31. ce used by developers to write a driver Document 2936PM between system software and the Mellanox BridgeX device FabricIT BX Management Software CLI User manual describing the software interface including examples for using User s Manual the BX manager and for installing EoIB and FCoE Host stacks www mellanox com Products Gateway Software Software and Docu mentation Download You will need a valid Mellanox Gateway S N Mellanox Firmware Tools MFT User s The MFT Mellanox Firmware Tools package is a set of firmware tools The Manual manual supplied with this package provides an overview of the firmware its Document 2329 installation and replacement The MFT can be downloaded with its documen tation at www mellanox com gt Support gt Download Firmware Tools Conventions The terms uplink internal and downlink external are used throughout the document Uplink refers to the server switch facing ports where InfiniBand IB is used Downlink refers to the Local Area Network LAN switch facing ports where Ethernet EN are used LA Caution This symbol indicates the possibility of physical injury to the user or installer 8 Mellanox Technologies Rev 1 4 This symbol indicates information that is helpful to the user gt Gateway Products Covered in this User Manual Table 3 Gateway Products Covered by this User s Manual Family Product Number Description BridgeX IB Gate
32. d product version information are found on the label seen in the figure below The product version information tab location is on the connector IB ETH side of the gateway on the bottom right hand side See Figure 3 on page 12 for a sample of the product label See Figure 1 on page 10 for the product label tab location Figure 3 Generic Product Label N MT9054X00 i III II il i dci LI Ii VII u AIV I 02642 FC il T Til TUL Ti I Made in IL lt e IL N o N o o o O lt 5 H O 5 Gateway Port Groups There are four gateway groups in the BX5020 identified as A B C and D Each gateway group consists of one OSFP uplink port and four SFP down link ports Each gateway group can be inde pendently configured Within each gateway group the OSFP uplink ports can be either 10Gbps 20Gbps or 40Gbps InfiniBand and the SFP downlink ports can be either 1Gbps or 10Gbps Ethernet 12 Mellanox Technologies Rev 1 4 EN 10Gbps 10Gbps 10Gbps unused EN 10Gbps 10Gbps 10Gbps unused EN 1Gbps 1Gbps 1Gbps unused EN 1Gbps 1Gbps 1Gbps unused Uplink Uplink 18 10 20 40 Gbps 18 10 20 40 Gbps Gateway Group B Gateway Group D EN 10Gbps 10Gbps 10Gbps unused EN 10Gbps 10Gbps 10Gbps unused EN 1Gbps 1Gbps 1Gbps unused EN 1Gbps 1Gbps 1Gbps unused Uplink Uplink 18 10 20 40 Gbps 18 10 20 40 Gbps Mellanox Technologies 13 Table 5 BX5020 Gateway Port Grouping Protocols and Speeds Gateway Group A Gateway Group C
33. der dem Endbenutzer bereitzustellen 12 Strahlenkontakt 13 UL und CSA Certified Netzkabel UL Listed and CSA Certified Power Supply Cord Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Proze duren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkontakt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 F r Nordamerika Stromanschluss w hlen Sie ein Netzkabel das UL und CSA Certi fied LA 3 Leiter 18 AWG mit einem angespritztem Stecker bewertet bei 125 V 15 mit einer Mindestl nge von 1 5 m Six Feet aber nicht mehr als 4 5 m F r die europ ischen Zusammenhang w hlen Sie ein Netzkabel das international harmonisiert und der Aufschrift lt HAR gt 3 Leiter mindestens 0 75 mm2 Draht bewertet mit 300 V mit einem PVC Mantel isoliert Das Kabel muss eine angespritztem Stecker bewertet bei 250 V 10 A Mellanox Technologies Rev 1 4 14 Ableitstrom gt 3 5mA LEAKAGE gt 3 5mA WARNUNG Hohe Ableitstrom Earth Verbindung bevor Sie die Verbindung von wesentlicher Bedeutung werden 15 Add GND Verbindung Informationen Add GND connection information Bevor Sie dieses Ger t an das Stromnetz die Schutzerde Terminal Schrauben dieses Ger t muss an den Schutzleiter in der Geb udeinstallation 16 INSTALLATION CODES INSTALLATION CODES Dieses
34. e gt 3 5mA Leakage gt 3 5mA ATTENTION La connexion la terre des forts courants de fuite est essentielle avant le branchement de l alimentation gt LA Avant de brancher l appareil la conduite d alimentation les vis de protection la terre du terminal de l appareil doivent tre appliqu es l installation de protection la Terre du b timent 9 Forts Courants de Fuite High Leakage Current Attention Forts courants de fuite Il est essentiel de relier a la terre avant de brancher l alimentation 10 Ajouter une information de connexion la masse Connect a Valid Ground to this Device terre du terminal de l appareil doivent tre appliqu es l installation de protection la Avant de brancher l appareil la conduite d alimentation les vis de protection la A Terre du b timent 11 Installation du mat riel k Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 12 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 13 Codes lectriques locaux et nationaux AN Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 14 Codes d installation INSTALLATION CODES L appareil doit tre install selon l ancienne version des codes lectriques nationaux du pays Pour l Am rique du Nord l
35. es Rev 1 4 The management module does not start 1 Check that the OpenSM has been started Power Cycle the Gateway If the Gateway is powered off for any reason without using the proper shut down procedure the Flash memory may be corrupted If the Flash memory is corrupted the Gateway may fail to boot If this happens talk to your Mellanox representative To run the gateway through a power cycle Run config reload halt 2 Wait long enough to let the gateway halt and verify there is no ping to it 3 Unplug the power cord 4 Plug in the power cord Gateway Fails to Boot 1 Reburn the firmware to the Flash memory using the MFT tools The FabricIT BXM last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop 2 Push the reset button on the gateway or management module 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 minx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location 12C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No 12C ready DRAM 2 GB ECC enabled 400 MHz CL3 FLASH 16 MB NAND 1024 MiB PCI Bus Dev Venld Devid Class Int PCI EO link is not up PCIE1 successfully set as root compl
36. ex 01 00 15b3 bd34 0c06 00 Net ppc 4xx ethO ppc 4xx ethl Hit Ctrl B to stop autoboot 0 Mellanox Fabricl T Mellanox Technologies 39 Rev 1 4 Troubleshooting Boot Menu 1 EFM PPC M460EX EFM 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM_PPC_M460EX EFM 1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt Choice 5 Select the image to boot 40 Mellanox Technologies Rev 1 4 Appendix A Specifications Table 12 BX5020 1S Specification Data Physical Power and Environmental Size Weight Mounting SerDes Speeds Connectors and 1 74 1U H x 19 W x 21 26 D 44mm X 436mm X 629 mm 9 828kg 21 671bs 19 Rack mount 10 20 40 Gb s per port Input Voltage Typ Power Passive cables Active cables optical cables Max Power 100 240VAC 50 60Hz 62 44W or 213 05BTUs hr 76 82W or 262 12BTUs hr 79 69W or 271 91BTUs hr IEEE 802 3x Flow Control Per Priority Flow Control Virtual lanes support Jumbo Frames support Cabling SFP Passive cables 76 15W or 259 83BTUs hr QSFP Active cables 95 76W or 326 75BTUs hr optical cables 99 68W or 340 12BTUs hr Optical Cable Port Types SFP SR Temperature 0 to 45 Celsius SFP LR Humidity 1096 90 non condensing Dissipated Power through SFP 1 5W QSFP 2 0W Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 40Gb s 20Gb s Safety UL60950 10Gb s C UL to CAN CSA 222 OoS 8 Inf
37. garding the software see the FabricIT BX Management Software CLI User s Manual www mellanox com gt Products gt Gateway Software gt Software and Documentation Download gt You will need a valid Mellanox Gateway S N To access this document download the User Manual from This chapter describes the management module and tools available for Out of Band management of the gateway system via FabricIT gt d Each Ethernet port gets connected to Ethernet switches These gateways must be configured to 10 100 1000M auto negotiation 3 1 FabriciT FabricIT is a software based management system that can be run with either a command line inter face or with a GUI interface The GUI interface can be run through the Web See the FabricIT BXM User Manual for instructions and commands available to manage the gate ways and fabric Mellanox Technologies 37 Rev 1 4 Troubleshooting 4 Troubleshooting As soon as a gateway is plugged in make sure that the green power LEDs on the PSU are on Power supply unit 1 3 If the AC power LED is off check that the power cable is plugged into a working outlet Check that the power cable has a voltage within the range of 100 260 volts AC Remove and reinstall the power cable Replace the PSU The Status green power LED for PSU1 or PSU2 does not come on 1 2 3 4 If the LED on the PSU is off check that the Sys is off Remove and reinstall the power cable C
38. heck that the power outlet in the wall is working Remove and reinstall the PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles Replace the PSU The power LED for the gateway shuts off l 2 Check that the there is adeguate ventilation Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes Use the FabricIT BXM manager to check the temperature of the gateway If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the gateway using a vacuum cleaner The green power LED for the fans does not come on 1 2 Check that the Power LEDs are on Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles LA Caution Do not run the gateway if the System Status LED for the Fans is Red The link LED for the connector does not come on 1 2 3 4 Check that both ends of the cable are connected Check that the locks on the ends are secured Make sure that the latest FW version is installed on both the HCA NIC cards and the gate way If media adapters are used check that the all connections are good tight and secure The activity LED does not come on Check that the Subnet Manager has been started 38 Mellanox Technologi
39. ight side of the power side panel with PSU2 on the left side Each PSU also has a single 2 color LED to the right of the PSU that indi cates the internal status of the unit PSU status indicators are also located on the left side of the connector IB ETH side panel and labeled PSU1 and PSU2 Table 8 Shows the explanation of the PSU Status LED colors Table 8 PSU Status LED Configurations LED Configuration Fan LED Green OK The Power supply is delivering the correct voltage 12VDC Red Error The PSU is not operational Replace the PSU 18 Mellanox Technologies Rev 1 4 Table 8 PSU Status LED Configurations LED Configuration Fan LED Off Off There is no power to the system neither PSU is receiving power If one PSU is showing green and the second PSU is unplugged it will show a red indication 2 2 06 Port Connector LEDs Figure 11 Connector Port Identification D ISTATUSIE 4 4 4 PSU1 Ej PSU2 FAN D RST Port numbers Port group labels Above the ports are two LEDs one for the upper port A and one for the lower port v The fol lowing table shows the port status according to the LED indication Table 9 Port Connector LED Assignment LED LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed data transfer speed Orange Physical link up
40. iniBand Virtual Lanes for all ports No 60950 1 TUV GS to EN 60950 1 Management Baseboard Performance and Device Amendment A1 A4 A11 management Agents for full InfiniBand CB IEC60950 1 all country In Band Management deviations EMC Emissions CC 47CFR Part 15 Class A ETHERNET PORTS Ethernet over InfiniBand EoIB proto EN 55022 Class A col encapsulation ICES 003 Class A IEEE 802 3ae 10 Gigabit Ethernet VCCI Class A IEEE 802 3z Gigabit Ethernet AS NZS CISPR 22 Class A IEEE 802 1D Spanning Tree CISPR 22 Class A IEEE 802 1p QoS COS EN 55024 IEEE 802 1Q VLAN Tagging EN 300386 IEEE 802 1AB Link Layer Discovery CE IEEE 802 3ad Link Aggregation with LACP Environmental EU IEC 60068 2 64 Random Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test Mellanox Technologies 41 Rev 1 4 Table 12 BX5020 1S Specification Data SEDARE tonte Reliability Availability and Serviceability Features Switching Perfor Hot Swappable Fan Module and Power Sup mance Simultaneous wire speed any port to any 1 1 Redundant plies port Power Supplies Addressing 48K Unicast Addresses Max per Subnet Bridging Capacity 16K Multicast Addresses per Subnet 320Gb s 42 Mellanox Technologies Rev 1 4 Appendix B EMC Certifications B 1 EMC Certifications The list of approved certifications per gateway in different regions of the world is located on the Mellanox Website
41. ir Flow Direction Picture OPN Designation Description connector IB ETH side inlet to power side outlet Mellanox Technologies Rev 1 4 Table 10 Air Flow Direction Picture OPN Designation Description Power side inlet to connector IB ETH side outlet 2 2 9 Management Interfaces Figure 13 Management Interfaces MGT CONSOLE ls 2 E USB Connector Product Label Ethernet Ethernet Connection 2 Connection 1 The following interfaces allow access to the management module 2 2 9 1 CONSOLE RS232 RJ45 Connector This RS232 connection provides access for local management through connection to a host The BX series gateways can be connected to the RJ45 connector of a computer This connection is needed for the initial configuration of the gateway For the initial configuration procedure for the gateway see the Installation Guide 2 2 9 2 GigE Ethernet Connector The gateway has two Ethernet connectors This Ethernet connection provides access for remote management The BX series gateways can be connected to a network through the RJ45 connectors There are two GigE connectors Their MAC addresses can be found on the product version label Mellanox Technologies 21 Rev 1 4 Installation and Basic Operation tab See Figure 3 on page 12 for the location of the tab The two MAC addresses are consecutively numbered Each Ethernet port gets connected to Ethernet
42. l edificio se debe incorporar un disposi tivo de protecci n contra sobrecargas de circuito derivado de f cil acceso con una cor riente nominal de 20 A 17 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 60 Mellanox Technologies Rev 1 4 18 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el c tricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos A deshacerse de este producto y de todas sus partes h galo de una manera responsable y respetuosa con el medio ambiente Mellanox Technologies 61 Rev 1 4 Appendix H SFP Modules H 1 Product Features Hot pluggable SFP footprint Maximum link length of 300m on 2000 MHZ km MMF Supports 9 95 to 10 3 Gb s bit rates Power dissipation 1W Commercial temperature range 0 C to 70 C Single 3 3V power supply Uncooled 850nm VCSEL laser Receiver limiting electrical interface Duplex LC connector Built in digital diagnostic functions RoHS 6 compliant lead free H 2 Applications 10GBASE SR SW 10G Ethernet H 3 Dimensions
43. les are MFMITO2A SR and MFMITO2A LR The figure below shows the Mellanox approved SFP module SR and LR modules not recommended by Mellanox may not work with the adapter A lt Figure 6 SFP Transceiver Module ANA Mellanox 850 nm G TECHNOLOGIES 08 21 t MFM1T02A SR 5 FTLX8571D3BCL ME l Class 1 21CFR1040 10 L NES N UDLOSUC LUE TELE ET 07 01 MADE IN MALAYSIA 2 2 4 8 Inserting the Optical Transceiver Module To insert the module into the cage 1 Open the module s locking mechanism see Figure 7 and Figure 8 2 Make sure that the male connectors on the module will align with the female connectors inside of the cage Also check that there is no dirt or foreign matter in the module or in the cage Mellanox Technologies 15 Rev 1 4 Installation and Basic Operation Figure 7 Module with Locking Mechanism Closed Mellanox 850 nm e TECHNOLOGIES 08 21 MFM1TO2A SR t FTLxasTiD3BCL ME Class 1 21CFR1040 10 LN050 7 01 e n 03JC NIMM MADE IN MALAYSIA Mellanox sso nm e TECHNOLOGIES 08 21 MTSFPP SR i FTLX857103BCL ME Class 1 21CFR1040 10 LIS t an D LT em MADE IN MALAYSIA 3 Insert the module into the adapter card module cage 4 Close the locking Mechanism To remove the module from the cage 1 Unlock the locking mechanism by opening the handle 2 Pullthe module out of the cage 2 2 5 Status LEDs Figure 9 Power and System LEDs
44. n that indicates the internal status of the unit An identical indicator labeled Fans is located on the left side of the connector IB ETH side panel The following fan status conditions are possible Table 7 Fan LED Configurations LED Configuration Fan LED Green OK The system is up and running Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan module Off Off The fan unit is not receiving any power Check that the fan unit is properly and completely inserted Mellanox Technologies 17 Rev 1 4 Installation and Basic Operation All fans must be operating while the power supply is plugged in If the gateway shuts down due to over temperature unplug the gateway wait 5 minutes and replug in the gateway 2 2 5 3 Power Supply Status Indicators The gateway is available with one or two factory installed Power Supply Units For gateways with only one unit installed a second Power Supply Unit can be added to increase security hot swap ability and to add redundancy See Section E Replacement Parts Ordering Numbers on page 25 for ordering part numbers PSU LED Secondary Power Supply Unit Figure 10 BX Gateways Power Side Panel Fan LED PSU LED O Fan unit PSU 1 Primary Power Supply Unit The primary power supply unit PSU1 is located on the r
45. nstallation kit The Installation Kit must be compatible with the gateway standard or short gateway and with the rack short rack or standard rack Table 11 shows the Rack installation kit options There are three possible rack options Make sure that you have the Rack installation kit that fits your needs Table 11 Gateway and Installation Kit Options Gateway and Rack Description Ordering Part Number Standard depth gateway standard rack MIS000085 Short gateway standard rack MIS000083 Short gateway short rack MIS000079 2 4 2 2 Installing the Gateway in the Rack Tools required and customer supplied parts Phillips Screwdrivers 1 and 2 Grounding screw ESD Strap Grounding wire sufficient to reach a valid ground ESD mat Parts included in the rail kit 2 rails 18 recessed flat head screws 2rail slides 8 caged nuts 26 Mellanox Technologies Rev 1 4 2 brackets 8 pan head screws M6 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground Figure 14 Installation Kit Parts Ar a lm Rep Ps i if Bj Before you install your new gateway unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have
46. o be used as a shelf or a work space 18 W WEEE Directive According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Mellanox Technologies 25 Rev 1 4 Installation and Basic Operation 2 4 2 Mechanical Installation The gateway can be rack mounted for installation in a standard 19 rack Front and back are arbi trary the rack kit can be mounted so that one side is even with the vertical rack support and the other side is recessed into the rack The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take pre cautions to guarantee proper ventilation for air intake at the front of the chassis and exhaust at the rear in order to maintain good airflow at ambient temperature Cable routing in particular should not impede the air exhaust from the chassis The gateway can be either front or rear mounted The notion of front and back is arbitrary This document uses the terms power side and connector IB ETH side to Ad reduce ambiguity 2 4 2 1 Minimum and Maximum Rack Depth for this Gateway This gateway can go into 19 racks whose vertical supports are between 400mm and 800mm apart Be sure to order the installation kit to match your rack Be sure you have the correct i
47. occurred during shipping A N If anything is damaged or missing contact your customer representative immediately 4 N 2 Screw the brackets onto the gateway Use 3or 4 depending on the bracket position flat head screws to connect each bracket to the gateway If you need room to bring the power cord from the other side of the rack recess the gateway and run the power cord through the bracket The installation kit can be reversed so that the bracket can be installed on either side of the gate way the power side or the connector IB ETH side L Mellanox Technologies 27 Rev 1 4 Installation and Basic Operation Figure 15 Making Room for the Power Cord Power cord Figure 16 Screwing on the Bracket In this position the front of the gateway is In this position the front of the gate 5cm behind the rack vertical support way is even with the rack vertical sup port S Gg In this position you will In this position you will use 3 flat head screws use 4 flat head screws Place to ruh the power cable The side of the gateway with these brackets will be the side that is even with or close to the vertical rack support 3 Screw the rails onto the gateway Use 5 flat head screws to connect each rail to the gateway Figure 17 Screwing on the Rail o o e e e O D SCrews 4 Clip 4 caged nuts into the holes in the rack you will be using to
48. om gt Products gt Cables for cable recommendations regarding Mellanox approved cables and recommended maxi mum cable lengths Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row Figure 23 Top and Bottom Ports 2 34 Mellanox Technologies Rev 1 4 2 4 9 Extracting and Inserting the Fan Unit AN N Operation without a fan unit should not exceed one minute A N During fan hot swap if the LED indicators are OFF then the fan unit is disconnected The gateway can run on three of the four fans at an ambient temperature of 45 C or less indefi nitely There are two possible air flows for the fan unit The air flow depends on the gateway model An R in the model number indicates a connector IB ETH side to power side air flow See See Part Numbering Legend on page 9 for an explanation of the model numbers and labels Make sure that the fans have the air flow that matches the power units An air flow A N Y N opposite to the PSUs will cause the gateway to operate at a higher less than optimal y temperature To extract a Fan Unit Fan Module Latches IX II II II II IX II II II II II II 11 11 11 11 11 II II II II II IX II II II IX II II II I X XX II IX II 11 II II II XX 11 II II XI II XI II IX II ZZ II 11 II XX IXI 11 II 11 II XI II 11 1
49. podr a causar lesiones corporales as como da os al equipo 5 Conexi n de fuente de alimentaci n redundante peligro de descarga el ctrica Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa est quitada o no est bien cerrada 6 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de activ idad de rayos 7 Conexi n y desconexi n del cable Copper InfiniBand Dado que los cables de cobre InfiniBand son pesados y no son flexibles su con exi n a los conectores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 58 Mellanox Technologies Rev 1 4 8 Montaje y mantenimiento de bastidores Al instalar o realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben instalar comenzando desde abajo hacia arriba 9 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente 10 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministra
50. r un confinamiento adecuado para componentes el ctricos y mec nicos y contra incen dio P 11 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 12 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales P 13 Cable de alimentaci n homologado por UL y con certificaci n CSA Fuga gt 3 5 mA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG termi nado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima de 1 5 metros pero no m s de 4 5 metros En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt HAR gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nominal 10 A ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n Mellanox Technologies 59 Rev 1 4 14 A adir conexi n a tierra A Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta
51. rd ground You must follow all national Some national and or local codes may require IT components to be bonded and exter N and local codes when installing this eguipment 2 4 4 Disassembling the Gateway from the Rack To disassemble the gateway from the rack 1 Putonan ESD strap and connect to a valid ground 2 Shut down the gateway using the procedure in Section 2 4 6 30 Mellanox Technologies Rev 1 4 Unplug all power supplies Disconnect all cables Disconnect any grounds if hard wired Qu dew Unscrew three of the four screws connecting the gateway brackets to the rack Support the gateway while you unscrew the last screw The gateway will drop and could become damaged or it could damage other equipment in the rack 7 While supporting the gateway unscrew the last screw 8 Slide the gateway out of the rails This is easier with two people 9 Remove the rail slides from the other side of the rack See the Mellanox Website for instructions to disassemble the gateway for proper WEEE disposal 2 4 5 Power Connections and Initial Power On The gateway ships with one Power Supply Unit A second PSU can be ordered for redundancy Each supply has a separate AC receptacle The input voltage is auto adjusting for 100 240 VAC 50 60Hz power connections The power cords should be standard 3 wire AC power cords includ ing a safety ground See Table 14 Replacement Parts Ordering Numbers for ordering the cables compa
52. switches Ethernet switches must be configured to 10 100 1000M auto negotiation 2 2 9 3 12C Connector There is an I2C connector on the far left of the power side of the gateway This connector is for FAES only LA This interface is for troubleshooting and debugging for FAEs only 2 3 Package Contents The package should include all of the following parts BridgeX Gateway Installation kit for 19 Rack mounting 1 power cable for each PSU 125V 10A 1 83M USA UL STANDARD Type B RS232 RJA5 to DB9 2 meter cable Installation Guide 2 4 Gateway Installation and Operation Installation and initialization of the gateway are straightforward processes requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment The gateway requires programming and configuration to operate as a basic InfiniBand to Ethernet gateway The gateway includes all of the necessary functionality to operate with external standard Subnet Management software This section describes the installation process and basic operation of the gateway Please first read the warnings sub section carefully before carrying on with installation procedures If the Gateway is powered off for any reason without using the proper shut down pro LA cedure the Flash memory may be corrupted If the Flash memory is corrupted the Gateway may fail to boot 22 Mellanox Technologies Rev 1 4 is no power switch Check all
53. tible with the electrical system of your country Caution The gateway will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable Warning Any yellow or red status LED is cause for concern and must be dealt with immediately It can take up to 5 minutes to boot up during which time the status LED may indicate red Caution After inserting a power cable and confirming the green system status LED is on make sure that the Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the gateway use the proper shut down procedure see Sec tion 2 4 6 and make sure ALL LEDS are off to ensure a powered down status gt P P gt Mellanox Technologies 31 Rev 1 4 Installation and Basic Operation You must power down the system to replace the power supply unit when there is only Do not hot swap the power supply if your gateway has only one working power supply A one working PSU in the gateway Figure 21 Two Power Inlets Electric Caution Notifications CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Risque de choc et de danger hazard The two PSUs are indepen
54. ure ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut pro voquer des blessures corporelles et des d gradations de biens 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lec trique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solidement fix ou s il est enlev 5 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 6 Branchement d branchement des c bles InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fab ricant des c bles pour connaitre les mises en garde et les instructions sp ciales 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr cautions sp ciales pour s assurer que le systeme reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut 52 Mellanox Technologies Rev 1 4 8 Fuit
55. ways BridgeX IB system dual controllers 4 QSFP uplink ports and 16 MBX5020 SFP downlink ports with CPU RoHS6 Mellanox Part Numbering Legend Table 4 Part Numbering Legend Place Field Decoder M Mellanox Technologies BX System Type BridgeX based Bridge System 1 Gateway protocols 1 Ethernet Uplinks 5 InfiniBand Uplinks 9 VPI Ethernet and InfiniBand Uplinks 0 For future use 2 of BridgeX devices 1 For future use 2 2 BridgeX devices 0 For future use _ Separator 1 Power Supplies 121 222 Depth of the Unit S standard depth B For future use Y Air Flow direction R connector IB ETH side to PSU side airflow F PSU side to connector IB ETH side airflow R RoHS C RoHS5 R RoHS6 Mellanox Technologies 9 Rev 1 4 Overview 1 Overview The Mellanox BridgeX gateway series is a 1U top of rack gateway that provides server I O con solidation over an InfiniBand network The product supports 40Gb s InfiniBand to 1GigE 10GigE Ethernet bridging The BridgeX Gateway is a multi protocol bridge that enables InfiniBand connectivity to native Ethernet networks The gateway implements stateless bridging protocols by encapsulating Ether net over InfiniBand EoIB acting as a packet relay based on read only context Bridging with the gateway requires the use of Mellanox ConnectX or ConnectX 2 adapters on the server to imple ment server side processing
56. witch on the gateway The gateway will come on when one plug is plugged in Mellanox Technologies 29 Rev 1 4 Installation and Basic Operation When the gateway is plugged in the status LED may be RED for up to two minutes until the system completes booting up 13 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LED is cause for concern and must be dealt with immediately 2 4 3 Grounding the Gateway Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the con nections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the gateway and the rack Test the ground using an Ohm meter Figure 20 Ground Connection SS Check to determine if your local or national electrical codes require an external ground to all IT components nally grounded not including the power co
Download Pdf Manuals
Related Search