Home

Preliminary Draft

1. Connector Type Signal Description 28 SMA Calibration Calibration of length 2X 29 SMA Calibration Calibration of length 2X 30 Mini SMP Calibration Calibration of length X 31 Mini SMP Calibration Calibration of length X 32 Fans 33 SMA Calibration Calibration of length 2X 34 SMA Calibration Calibration of length 2X 35 Main I2C 36 Mini SMP Calibration Calibration of length 2X 37 Mini SMP Calibration Calibration of length 2X 38 Mini SMP Calibration Calibration of length X 39 Mini SMP Calibration Calibration of length X 40 CX4 Clock and Reset 41 QSFP RX Data 42 QSFP TX Data 43 SMA Clock Option for ext clock 106 25MHz 43 RJ45 Ethernet management conncetion 44 SMA Calibration Calibration of length X 45 SMA Calibration Calibration of length X 47 SMA Calibration Calibration of length X 48 SMA Calibration Calibration of length X 49 Mini SMP Reference Connects to Port 35 Ref to compliance port 33 50 Mini SMP Reference Connects to Port 35 Ref to compliance port 33 51 Mini SMP Reference Connects to Port 36 Ref to compliance port 34 52 Mini SMP Reference Connects to Port 36 Ref to compliance port 34 53 Mini SMP Reference Connects to Port 36 Ref to compliance port 34 54 Mini SMP Reference Connects to Port 36 Ref to compliance port 34 55 Mini SMP Reference Connects to Port 35 Ref to compliance port 33 56 Mini SMP Reference Connects to Port 35 Ref
2. QCN 1 6 Serial Number and Product Version Information The Serial number and GUID for the switch and the MAC for the Management PC are found on the pull out tab below the USB interface connection Figure 2 Pull Out Tab S N MT1117X00014 VAT INA P N MT51336 EN EVB VINTON TAT OD Rev A1 10 27 11 69 20 00 2 LOIN 0002C92705 20 27 11 69 20 00 L LOIN 1C Made IL Mellanox Technologies 13 2 40 Gb s Ethernet 40 Gb ETH is only guaranteed to work with approved Mellanox Cables Each QSFP port is capable of up to 40GigE and with a QSA the ports can run at IOGigE IGigE Up to 24 QSFP ports are capable of 1 2 4 8G Fibre Channel with a QSA All ports are capable of virtual switching VEPA All usable virtual switching ports are grouped together The port protocol can only be obtained from the management interface 14 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 3 FDR The SwitchX EVB supports FDR a pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s The FDR physical layer is an IBTA specified physical layer using different block types deskew mechanism and framing rules SwitchX also supports FDR10 a non standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10 3125Gb s with 64b 66b en
3. Zak 9 Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 1 5 KCC Certification Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and certification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL which is organized under the Ministry of Information and Communications KCC EMC testing includes electromagnetic emissions EMI and susceptibil ity EMS Certified equipment is labeled with the KCC mark and certification number 68 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Device User s information AS 717 017171 UP SAB OB 55 S 0 HS 2015147 as 4 17171 6551 78 RIAA 96 WS 22402 sc CLASS A device This device has been approved by EMC registration commercial broadcasting and Distributors or users pay attention to this point This device is communication equipmen usually aimed to be used in other area except at home e Remark Class A device operated in a commercial area Translation Class A Device This device is registered for EMC
4. 5 8 Disassembly of the Switch from the Rack To disassemble the switch from the rack 1 Unplug and remove all connectors Mellanox Technologies 49 Rev 1 0 Installing the PCIE 1GbE Server Side board Unplug all power cords 3 Remove the ground wire Unscrew the 2 bolts from the side of the switch with the bracket Support the weight of the switch when you remove the screws so that the switch does not fall 3 Slide the switch from the rack Remove the rail slides from the rack 7 Remove the eight caged nuts 5 9 Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of this product and all of its parts in a responsible and environmentally friendly way Follow the instructions found in the Mellanox Web site at mellanox com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive 5 10 Resetting the Switch Should the switch stop working try resetting the switch On the connector side panel under the system LEDs is a reset button This reset button requires a tool to be pressed Figure 36 Reset Button Press the rest button This button resets the switch when the button is pushed A quick push of this button resets the whole switch When the button is held down for 15 seconds the switch is reset and the password is deleted T
5. N N 4 N Do not attempt to insert a PSU with a power cord connected to it 4 SS Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirm ing the proper installation Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The green PSU indicator should light If not repeat the whole procedure to extract the A PSU and re insert it Mellanox Technologies 35 Rev 1 0 Basic Operation and Installation 4 6 5 Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable press the con nector into the port receptacle until the connector is firmly seated The GREEN LED indicator corresponding to each data port will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connec tor plugged into a functioning port After plugging in a cable lock the connector using the latch ing mechanism particular to the cable vendor When a logical connection is made the yellow light will come on When data is being transferred the green light will blink Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row Figure 22 Top and Bottom Port
6. O O O a S e ala a a zo un a 9 zm un mi iun 9 ao amon a SAS 9 NI m e S in in Lo ul 69 ala Table 14 Connector Pinout number Siena Signal 1 GND 2 12V GND 3 GBE1_MDI1 RX0 4 12V GBEO_MDI1 RX0 5 GBE1_MDI1 RX0 6 12V GBEO_MDI1 RX0 7 factory restore GND 8 X GND 9 GP_LV GBE1_MDI0 TX0 10 3 3V GBEO MDIO TXO0 11 GP HV GBE1_MDI0 TX0 12 3 3V GBEO_MDI0 TX0 13 x GND 14 3 3V GND 15 Sys_Err 2 5VA_GBE 16 X 2 5VA_GBE 17 GBE1_ACT 18 x GBEO_ACT 19 CPU_TDO GBE1_LINK 20 RS232_DCD GBEO LINK 2 CPU_TD1 INT3 22 RS232_RX CPU_Present 25 GND GND Mellanox Technologies 73
7. 2 Insert the server side board into a X8 or X16 PCIE slot in the server Figure 34 Connecting the EVB to the Server Side Board 3 Connect the cables from the daughter board to the server board connect green to green and blue to blue Connect the CX4 cable from the daughter board to the server board 5 Tie down the cables using the cable ties in the EVB Mellanox Technologies 45 Rev 1 0 Installing the PCIE 1GbE Server Side board 5 6 Board Connectors Numbering and Locations Figure 35 EVB Board Connections View 1 NNNNNNNRBPBBRBBB BBB gt gt qom 9 aom aor aor B 1 gt qom or 1 amp 3 t eU 6 6 61 9 lt 6 6 6 6 2 6 6 0 6 6 0 6 9 6 0 51 5 6 6 TEE gt 4 46 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Table 7 Port Connector Numbering Sheet 1 of 3 Connector amp Type Signal Description 1 Mini SMP Calibration Calibration of length 2X 2 Mini SMP Calibration Calibration of length 2X 3 SMA Reference Connects to Port 2 Ref to compliance port 4 4 SMA Reference Connects to Port 2 Ref to compliance port 4 5 SMA Refere
8. 2 Put the co tors for the extender daughter board through the holes in the frame Figure 32 Holes for the Extender Board Interfaces Set the connectors through these holes Mellanox Technologies 43 Rev 1 0 Installing the PCIE 1GbE Server Side board 4 Figure 33 Install the Extender Board Push the daughter board down into the board conne Screw in the four screws Remove the blank panel from the power sid he EVB Put the wire ties onto the bottom of t e Ce gt d of the EVB The cables are for 10 100 1000Mb AA fana gement RJ45 Ethernet cable Connect the RJ45 cable to a switch or server Ethernet port This is 10 100 1000Mb DN rol 1 CX4 ables blue to blue and green to green Run the cables through the power Sid Connect all 3 connectors blue green and Clock from the Extender to the Server Select only one of these two options Either use the RJ45 Ethernet connection option or the PCIE option DO NOT TRY AND USE BOTH OPTIONS AT THE SAME TIME When running the SwitchX EVB without the top cover use portable fans to keep the heat sinks cool 4 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 5 5 Installing the Server Side Board 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground
9. 2009 2011 Mellanox Technologies Inc Mellanox Technologies 61 Rev 1 0 Connecting to the Switch Platform 7 2 1 Managing the System For information and instructions regarding the use of the Embedded Management Tools see the Embedded Management Tools User Guide on the enclosed CD 7 2 2 12C Connector The I2C connection provides access to Flash and EEPROMs This connection allows access to the switch for updating firmware when in band firmware updating is impossible Note The RJ45 to DB9 harness 22 in the box with the switch is for use with the MTUSB 1 adapter This is necessary only when updating firmware cannot be updated in band This interface is for Debug and Troubleshooting only This interface is for FAEs and advanced users only When you install the switch it comes with the latest firmware burned on the board AII firmware updates should be done in band This is only done when you receive an e mail that a newer firm ware version for your switch is available Download the latest firmware from http www mellanox com gt Downloads gt Firmware For instructions on downloading firmware see http www mellanox com gt Downloads gt Firmware Tools Be sure to download the user manual appropriate to your OS Read the instructions in the User manual for the firmware update procedure 7 2 3 How to Get Mellanox Firmware Tools Mellanox Firmware Tools MFT and documentation are available for download via ht
10. 59 Make sure to connect to CONSOLE 45 port of switch and not to the Ether net MGT port No remote IP connection is available at this stage Step 2 Configure a serial terminal program for example HyperTerminal minicom or Tera Term on your host PC with the settings described in Table 8 Table 8 Serial Terminal Program Configuration Parameter Setting Baud Rate 9600 Data bits 8 Stop bits 1 54 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Table 8 Serial Terminal Program Configuration Continued Parameter Setting Parity None Flow Control None Step 3 Login from a serial terminal program as admin and use admin as password This starts the Mella nox configuration wizard Step 4 Go through the Mellanox configuration wizard Table 9 shows an example of a wizard session Table 9 Configuration Wizard Session IP Configuration by DHCP Sheet 1 of 2 Wizard Session Display Example Comments Mellanox configuration wizard Do you want to use the wizard for initial configura tion yes You must perform this configuration the first time you operate the switch or after resetting the switch Type y and then press Enter Step1 Hostname switch 1 If you wish to accept the default hostname then press lt Enter gt Otherwise type a different hostname and press lt Enter gt Step 2 Use DHCP
11. 1 46 Mellanox Technologies 5 Rev 1 0 Figure 36 Reset Button 50 Figure 37 Management Interfaces 54 Figure 38 Transfer Power Cord 71 Figure 39 Install the Power Cord Clip 72 Figure 40 Transfer Power Cord Finished 72 Figure 41 Connector Pinout 73 6 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 List of Tables Table 1 Revision History 8 Table2 System Status LED Configurations 18 Table3 Connector Physical and Logical Link Indications 19 Table4 PSU Status LED Configurations 20 Table 5 Fan Status LED Configurations 21 Table 6 Port LED Configurations 22 Table 7 Connector Numbering 47 Table 8 Serial Terminal Program Configuration 54 Table9 Configuration Wizard Session IP Configuration by DHCP 55 Table 10 Configuration Wizard Session IP Zeroconf Configuration 57 Table 11 Configuration Wizard Session Static IP Configuration 58 Table 12 Adapter Cards Certification Status 67 Table 13 Specifications for MT51336 NUDT 69 Table 14 Connector Pinout 73 Mellanox Technologies 7 Rev 1 0 Revision Table 1 Revision History Revision Description 1 0 Initial Release 0 7 Added the correct H x W x D and weight Added Power numbers Added verified compliance port to use 0 6 Added the word Hardware to the name Added correct graphics of the server side card 0 5 Added tie down the cables Added Caution Note do not us
12. 2011 from 192 168 10 1 Mellanox Switch sx 43 standalone master gt Step 4 You can enter any supported command now 7 2 Starting a WebUI Connection to the Switch Step1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connector Note The Ethernet supported link speeds are 10 100 Mbps Step 2 Start a Web browse Google Chrome Microsoft Internet Explorer 7 0 or Mozilla Firefox 3 0 60 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Make sure the screen resolution is set to 1024 768 or higher Step 3 Enter as URL the following http switch IP address where switch address is the IP address of the switch or its DNS name On Ethernet systems log into the BIP if you configured it for your system For infor mation on how to configure the BIP see Box IP Centralized Location section in the SW user manual Step 4 You will receive the login window for remote management of the switch The following figure shows an example Note that the default username is admin Mellanox MLNX OS Management Console Mellanox TECHNOLOGIES Host Piranha SX 1 User not logged in Login Please enter your username and password then click Login Account Password Mellanox MLNX OS Switch Management Best viewed using Firefox Chrome IE 7 or higher at 1024x768 resolution or higher
13. 3 You will have 5 seconds to stop the U Boot by pressing Control B 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 mlnx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option H Boot ROM Location I2C Addr 0x52 Internal PCI arbiter disabled 32 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No I2C ready DRAM 2 GB ECC enabled 400 MHz CL3 Mellanox Technologies 65 Rev 1 0 Troubleshooting 5 Select the image to boot 66 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Appendix A Specifications A 1 EMC Certification Statements Table 12 lists the approved certification status per card in different regions of the world Table 12 Adapter Cards Certification Status Adapter Card P N FCC VCCI EN ICES CE CB cTUVus KCC MT51336 EN EVB MT51336 VPI EVB 1 1 FCC Statements USA Class A Statements 8 15 19 a 4 This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 This device must accept any interference received including interference that may cause undesired operation 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for complian
14. PCIE 1GbE server side board allows for a direct connection of an external server directly to the SwitchX chip It allows for a PCIE slot in the external server to be connected to a PCIE port in the SwitchX EVB This will support a PCIE 2 0 1X connection An alternative is to select to use the RJ45 1GbE connection to the SwitchX EVB The user can connect this RJ45 connector to any Ethernet 10 100 1000 port for management 5 1 PCIEAGDbE Extender Kit parts Part Quantity For PCIE PCB server board 1 Server side board EVB Daughter board 1 System side board MT PCI HS QSFP 2 26AWG 2m cable for the PCIE fabric MCC4L30 002 1 30AWG 2m cable for the PCIE clock and Reset The EVB daughter board has 4 interfaces three for PCIE and one for Ethernet use Do not use both PCIE and Ethernet at one time You must select either the PCIE or the Ethernet connection The interfaces are Interface For PCIE CX4 clocking and reset of the PCI Express QSFP for connection to the server side board QSFP for connection to the server side board Interface For Ethernet RJ45 for remote Ethernet connection 38 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 25 Server Side Card Power connection The PCB server board has 3 interfaces Interface For PCIE 4 clocking and reset QSFP for connection to the server sid
15. PSU has a single 2 color LED on the right side of the PSU that indicates the internal status of the unit Mellanox Technologies 19 Rev 1 0 Basic Operation and Installation Figure 8 PSU Status LEDs This is a two color LED 7 Green OK Red Major fault Off No power to the system Fan air flow indicator Table 4 PSU Status LED Configurations LED Color Status Green OK The Power supply is delivering the correct voltage 12VDC Red Error The PSU is not operational Off Off There is power to the system 41 1 5 Fan Status Indicators The indicator is located to the left of the connectors the connector side panel Figure 9 Fan Status LED Connector Side This is a two color LED Green OK Red Major fault Off No power to the system The LED indicator on the Fan Module is on the right side of the module 20 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 10 Fan Status LED Power side This is a two color LED pu i Green OK Red Major fault Off No power to the system Fan air flow indicator pra i The following fan status conditions are possible Table 5 Fan Status LED Configurations LED Configuration FAN LED Green OK The system is up and running Red Error One or more fans is not operating properly The system should be powered down and troubleshoot the fan modu
16. SMAports and ports 35 and36 are mini SMP ports These reference traces are designed to have identical insertion loss as the corresponding compli ance ports and those traces are routed in the same geometry and length as the traces of the compli ance ports Mellanox Technologies 11 Rev 1 0 Overview 1 5 Features Full Feature List Reference traces SMA and mini SMP Compliance ports SMA and mini SMP e 2 ports connect to mini SMP connectors for SI testing 24 QSFP ports e 1 PSU 12 SFP ports RADIUS TACACS LDAP SSH DHCP CLI IPv6 MNG Telnet FTP RSTP PFC Static MAC SCP SNTP AAA Syslog Dual SW image Auto temprature control Upload download configuration System alarms VLAN aware ETS Flow Control 36 FDR 56Gb s ports in a 1U switch IBTA 1 3 and 1 21 complaint SDR DDR QDR FDR10 FDR link speed Congestion control Adaptive routing 12 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Port mirroring FDRIO supports 20 more bandwidth over regular QDR using the same cables connectors Up to 8 multiple switch partitions All ports can be set to gateway ports connecting to 40GigE over QSFP or 1 10GigE over QSA 4 032 Tb s aggregate data switching capacity with ultra low latency CBF DCBX LAG LACP
17. on mgmt0 interface yes Perform this step to obtain an IP address for the switch is the management port of the switch If you wish the DHCP server to assign the IP address type yes and press Enter If you type no no DHCP then you will be asked whether you wish to use the zeroconf configura tion or not If you enter yes yes Zeroconf the ses sion will continue as shown in Table 10 If you enter Zeroconf then you need to enter a static IP and the session will continue as shown in Table 11 Step 3 Admin password Press lt Enter gt to leave unchanged lt new_password gt Step 4 Confirm admin password lt new_password gt To avoid illegal access to the machine please type a password and then press lt Enter gt Then confirm the password by re entering it Note that password characters are not printed Mellanox Technologies 55 Rev 1 0 Management and Tools Overview Table 9 Configuration Wizard Session IP Configuration by DHCP Sheet 2 of 2 Wizard Session Display Example Comments You have entered the following information 1 Hostname lt switch name gt 2 Use DHCP on mgmt0 interface yes 3 Admin password Enter to leave unchanged CHANGED To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice lt Enter gt Configuration changes saved To
18. requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced with a residential use type 2 2 MT51336 Specifications Table 13 Specifications for MT51336 NUDT Physical Power and Environmental Size Lx WxH 629 mm x 428 mm x 43 6 mm Voltage 12V 3 3V 24 76 in x 16 85 in x 1 71 in Weight 9 31kg 20 52 165 Typ Typ 50 40GigE Air Flow 62CFM Active Cables 182W 5W port Passive cables 100W 2 8W port Typ Typ 40GigE Active Cables 186W 5 2 W port Passive cables 104W 2 9 W port Max Max 40GigE Active Cables 224W 6 2 W port Passive cables 119W 3 3W port Temperature 0 to 45 C Protocol Support Regulatory Mellanox Technologies 69 Rev 1 0 Table 13 Specifications for MT51336 NUDT Physical Power and Environmental PCI Express 2 0 SERDES 5 0GT s EMC FCC 47 CER part 15 2006 subpart B class A ICES 003 2004 Issue 4 class A VCCI V 3 2007 04 class A EN 55022 1998 1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A2 2005 EN 55024 1998 1 2001 2 2003 standards harmonized under Directive 2004 108 EC Article 6 2 AS NZS 3548 Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 60068 2 64 29 32 RoHS RoHS R5 70 Mellanox Technologies SwitchX Switch EVB Hardware User Ma
19. slide the swich into Second side leg el Clip 6 caged nuts into the holes in the rack on the side of the rack you will be sliding the switch into Check that both sides of the switch power side and connector side are at the same level in the rack Clip 4 more caged nuts into the holes on the opposite side of the rack Check that both sides of the switch left and right are the same level in the rack Slide the rail into the rail slide 32 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 9 Using two of the bolts for each corner install the rails and rail slides in the rack Do not tighten the bolts yet Figure 19 Slide the Rail into the Rail Slide This side of the rail kit goes on the side of the rack you will slide the switch into This is the same side of the switch that will be next to the vertical support 10 Slide the switch into the rails 11 Tighten the bolts to 9 2 Nm or 81 5 pound inches 12 Put the switch into place and screw the bolts into the nuts Tighten the bolts to 9 2 Nm or 81 5 pound inches 13 Ground the switch 14 Plug in the power cables 15 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LEDs are cause for concern and must be dealt with immediately It can take up to 5 minutes to boot up during which time the status LED may in
20. to compliance port 33 48 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Table 7 Port Connector Numbering Continued Sheet 3 of 3 Connector amp Type Signal Description 57 Mini SMP Data Connects to HW port 36 on SwitchX 58 Mini SMP Data Connects to HW port 36 on SwitchX 59 Mini SMP Data Connects to HW port 36 on SwitchX 60 Mini SMP Data Connects to HW port 36 on SwitchX 61 Mini SMP Data Connects to HW port 36 on SwitchX 62 Mini SMP Data Connects to HW port 36 on SwitchX 63 Mini SMP Data Connects to HW port 36 on SwitchX 64 Mini SMP Data Connects to HW port 36 on SwitchX 65 Mini SMP Data Connects to HW port 36 on SwitchX 66 Mini SMP Data Connects to HW port 36 on SwitchX 67 Mini SMP Data Connects to HW port 36 on SwitchX 68 Mini SMP Data Connects to HW port 36 on SwitchX 69 Mini 5 Data Connects to HW port 36 on SwitchX 70 Mini SMP Data Connects to HW port 36 on SwitchX 71 Mini SMP Data Connects to HW port 36 on SwitchX 72 Mini SMP Data Connects to HW port 36 on SwitchX 73 SMA Clock Option for ext clock 156 25MHz 5 7 Switch Shut Down Procedure To shut down the switch run the command Reload halt noconfirm The switch cannot be restarted remotely Ah To restart the switch you must physically go to the switch and unplug and replug in all of the power cords to the chassis
21. 5 1 1993 1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 2001 13 Do Not use as a Shelf NS Caution Slide rail mounted equipment is not to be used as a shelf or a work space 14 UL Approved AC Power Cords For North American power connection select a power supply cord that is UL Listed and CSA Certified 3 conductor 18 AWG terminated a molded on plug cap rated at 125 15 b with a minimum length of 1 5m six feet but no longer than 4 5m For European connection select a power supply cord that is internationally harmonized and marked lt gt 3 conductor minimum 0 75 mm2 wire rated at 300 V with a PVC insulated jacket The cord must have a molded on plug cap rated 250 V 10 A For Shark 4 5 2 Mechanical Installation The switch platform can be rack mounted and is designed for installation in a standard 19 rack The power side of the switch includes a hot swap power supply module a blank cover and a hot swap fan tray There are two possible air flow directions Be sure that the switch air flow direction is compatible with your system rack and PSU The connector side of the switch has the data ports system LEDs and management connection ports The switch platform contains auto sensing 100 240 VAC connections for the PSU 28 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 The installer should use a rack capable of supporting the mechanical and
22. AMA Mellanox TECHNOLOGIES SwitchX Evaluation Board Hardware User Manual PN MT51336 EN EVB MT51336 VPI EVB Rev 1 0 www mellanox com NS NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PRO VIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUS TOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CAN NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MER CHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEM PLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCURE MENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILI
23. Extracting and Inserting the Fan Unit on page 36 A Caution Do not run the switch if the System Status LED for the Fans is Yellow The link LED for the connector does not come on 1 Check that both ends of the cable are connected 2 Check that the locks on the ends are secured 3 Make sure that the latest firmware version is installed on both the HCA and the switch 4 If media adapters are used check that the all connections are good tight and secure The activity LED does not come on Check that the Subnet Manager has been started 64 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 The switch is off 1 Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies If the switch comes on Use the MLNX OS management CLI or Web GUI to determine the cause of the Shutdown pode cea Check the temperature Check the Fan status The switch is not working and unresponsive 1 Reset the switch If resetting the switch does not work 1 Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies If the switch comes on use the MLNX OS management CLI or Web GUI to determine the cause of the shutdown Sate The last software update did not succeed 1 Connect the RS232 connector CONSOLE to a laptop 2 Push th reset button on the switch or management module
24. Firmware Tools MFT 62 7 3 Updating Firmware 63 Chapter 8 Troubleshooting 64 4 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 List of Figures Figure 1 Connector and Power Side Panels 9 Figure 2 Pull Out Tab 13 Figure 3 Connector and Power Side Panel 16 Figure 4 Power Side Panel 16 Figure 5 Power Fan and System LEDs 18 Figure 6 System Health LED 18 Figure 7 Power Side Panel 19 Figure 8 PSU Status LEDs 20 Figure 9 Fan Status LED Connector Side 20 Figure 10 Fan Status LED Power side 21 Figure 11 Reset Button 22 Figure 12 Port Numbering 23 Figure 13 Top and Bottom Ports 24 Figure 14 Management Interfaces 24 Figure 15 Rack Rail Kit Parts 30 Figure 16 Mounting Options 31 Figure 17 Screwing on the Rail 32 Figure 18 Inserting the Caged Nuts 32 Figure 19 Slide the Railinto the Rail Slide 33 Figure 20 Power Supply Unit Extraction 35 Figure 21 PSU Pulled Out 35 Figure 22 Top and Bottom Ports 36 Figure 23 Air Flow Labels 37 Figure 24 Fan Module Latches 37 Figure 25 Server Side Card 39 Figure 26 PCIE 1GbE Extender 40 Figure 27 Screw Locations 41 Figure 28 Slide the Top 41 Figure 29 Management Board Installed 42 Figure 30 Unscrew the Management Board 42 Figure 31 Remove the Management Board 43 Figure 32 Holes for the Extender Board Interfaces 43 Figure 33 Install the Extender Board 44 Figure 34 Connecting the EVB to the Server Side Board 45 Figure 35 EVB Board Connections View
25. French see Section E Avertissements de s curit d installation French on page 53 for German see Section G Installation Sicherheitshinweise German on page 55 and for Spanish see Section H Advertencias de seguridad para la instalaci n Spanish on page 57 For special regulations regarding Finland Sweden Denmark and Norway see Section on page 59 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 45 C 113 F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings 3 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage 26 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 4 Redundant Power Supply Connection Electrical Hazard This product includes a blank cover over the space for the redundant power supply Do not operate the product if the blank cover is not securely fastened or if it is removed 5 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 6 Copper Cable Connecting Disconnecting A Copper cables are heavy and
26. TY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS and SwitchX are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners SwitchX Switch EVB Hardware User Manual 2 Mellanox Technologies Document Number 3493 SwitchX Switch EVB Hardware User Manual Rev 1 0 Table of Contents Revision 8 Chapter 1 Overview 9 1 1 Management CPU 9 1 2 PCIE 1GbE Extender Kit 10 1 2 1 PCIE Extender Kit 10 1 2 0 1GbE Extender Kit 10 1 3 Compliance Ports 11 1 4 Reference Traces 11 1 5 Features 12 1 6 Serial Number and Product Version Information 13 Chapter 2 40 Gb s Ethernet 14 Chapter 3 FDR 15 Chapter 4 Basic Operation and Installation 16 4 1 Switch Platform Hardware Overview 16 4 1 1 Status LEDs 17 4 1 2 Reset Button 22 4 2 Cable Power Budget Classification 23 4 3 SFP Cable Power Budget Classification 23 4 4 Interfaces 23 4 4 1 Port Connector Inte
27. The ports are labelled as shown in Figure 12 The bottom row ports are flipped from the top row See Figure 13 for bottom row top row port orientation Figure 12 Port Numbering Reference Traces 1 2 Reference Traces 35 36 One Trace and compliance pair 1 3 gt One Trace and compliance pair 33 35 A MCN Z 3S Mi 223 25 27 29 31 33 gt ERN eee Eee EL 4e 8 10 12 14 i6 i8 20 22 24 26 28 30 3 84136 2 WIE i One Trace and compliance pair 2 4 NU One Trace and compliance pair 34 36 Compliance Ports 3 4 Compliance Ports 33 34 Mellanox Technologies 23 Rev 1 0 Basic Operation and Installation 4 4 1 2 and Bottom Orientation Figure 13 Top and Bottom Ports 4 4 2 Management and Firmware Updating Interfaces NONE OF THE MANAGEMENT INTERFACES ON THE CONNECTOR SIDE WILL WORK IF THE MANAGEMENT BOARD WAS REMOVED AND OR REPLACED WITH THE PCIE IGbE EXTENDER There are four interfaces to connect to the EVB 2X 10 100Mgb IGb Ethernet connectors labelled USB port that is labelled This interface can be used to update software or firmware connector that is labelled CONSOLE Use this connector to connect to the host PC There is one I2C interfacesto connect to the SwitchX MT51336 on the power side but this is for Mellano
28. bject such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip Figure 11 Reset Button Use an opened paper clip to press the reset button for 15 seconds to reset the main and management CPUs and to delete the existing password This button resets both the CPU of the switch device and the CPU of the management module It thereby resets all of the ports by bringing them down and powering them up when the button is pushed A quick push of this button performs this reset When the button is held down for 15 sec onds the switch is reset and the password is deleted You will then be able to enter without a pass word and make a new password for the user admin 22 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 If the PCIE IGbE Extender is installed the reset button resets the CPU of the switch device only 4 2 QSFP Cable Power Budget Classification The SwitchX EVB QSFP ports are designed for active cables with a max power per module of 2W Typical power per port is 1 5W 4 3 SFP Cable Power Budget Classification The SwitchX EVB SFP ports are designed for active cables with amax power per module of 1 5W Typical power per port is 1W 4 4 Interfaces 4 4 1 Port Connector Interfaces 4411 36 Port Switches The Connector side of the switch EVB has 24 QSFP ports and 12 SFP ports These are placed in two rows 18 ports to a row
29. ce Mellanox Technologies could void the user s authority to operate the equipment 515 105 Statement NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense 8 1 2 Statements Europe EN55022 Class A Statement Mellanox Technologies 67 Rev 1 0 Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures A 1 3 ICES Statements Canada Class A Statement This Class digital apparatus complies with Canadian I CES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada Class B Statement A 1 4 VCCI Statements Japan Class A Statement TORE TER 5 VCCI ESQ COREE CERT 2 CBR EAS 5 2 6 ET
30. coding resulting in an effective bandwidth of 40Gb s FDR1O0 supports 20 more bandwidth over regular QDR using the same cables connectors Both FDR and FDR10 support Forward Error Correction FEC as described in IEEE 802 3ap chapter 74 FDR10 is only guaranteed to work with approved ConnectX 3 adapters gt Mellanox Technologies 15 Rev 1 0 Basic Operation and Installation 4 Basic Operation and Installation 4 1 Switch Platform Hardware Overview Figure 4 shows the power side panel and connector side panel views of the switch EVB Figure 3 Connector and Power Side Panel The figure shows port configurations for the switch systems including 2 Ethernet RJ45 connectors e 1 RJ45 connector for connecting to a host PC CONSOLE USB connector 24 QSFP ports 12 SFP ports various switch port fan and power status LEDs Figure 4 Power Side Panel hot swap fan module power supply unit e 12 banana connector various switch port fan and power status LEDs All connectivity except for power cords is via the connector side panel All connectors can support active cables 16 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 4 1 1 Status LEDs 4111 System Status Indicators The System Status Indicators PAN are located to the left of the connectors on the connector side panel and on the power side at the far ri
31. dicate flashing green 16 Youcan start connecting all of the cables to the switch 4 6 2 Grounding the Switch Check to determine if your local or national electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the ground screw make sure that the rack is properly Mellanox Technologies 33 Rev 1 0 Basic Operation and Installation grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Some national and or local codes may require IT components to be bonded and exter L nally grounded not including the power cord ground You must follow all national n and local codes when installing this equipment 4 6 3 Power Connections and Initial Power On The switch platform ships with one Power Supply Unit The input voltage 18 auto adjusting for 100 240 VAC 50 60Hz power connections The power cord should be standard 3 wire AC power cord including a safety ground and rated for 15A or higher Caution The switch platform will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable Caution After inserting a power cable and confirming the green system status LED light is on make sure that t
32. e as a shelf 0 4 fixed graphics 0 3 More fixes 0 2 Added Initial configuration section 0 1 Preliminary Release 8 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 1 Overview This EVB is intended to be used as a platform to evaluate the SwitchX silicon SW engineers can use it as a means for SW developement using the SwitxchX SDK SW developers can implement initial driver development integration into non PPC CPU environments Signal integrity engineers can use it to evaluate SwitchX SI performance and compliance to spec This EVB can be used to evaluate and qualify Ethernet SerDes IB SerDes FC SerDes Initial Ethernet and IB switch testing for L2 L2 This EVB can be used both while rack mounted standard 19 rack and in a tabletop setup for Signal Integrity evaluation Figure 1 Connector and Power Side Panels 1 4 Management CPU The EVB comes with a Power PC 460E CPU for management It also comes with a PCIE IGbE Extender that allows the user bypass the management CPU and to connect directly to the chip When the PCIE 1GbE Extender is used the Power PC is removed from the system and none of the management interfaces on the connector side will work When the Power PC is removed the switch EV B becomes unmanaged and does NOT need Initial configuration Mellanox Technologies 9 Rev 1 0 Overview 1 2 PCIE 1GbE Extender Kit The Power PC management module ca
33. e board QSFP for connection to the server side board Mellanox Technologies 39 Rev 1 0 Installing the PCIE 1GbE Server Side board Figure 26 PCIE 1GbE Extender RJ45 1GbE Connector for remote management CX4 Clock and Reset Connector QSFP RX Green Connector QSFP TX Blue Connector 5 2 Removing the Top Cover 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground Remove the 6 screws from the power side of the top 3 Slide the top towards the power side Ah With the cover off you will need a clip on fan cooling the chip heat sink 40 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 27 Screw Locations Mellanox Technologies 41 Rev 1 0 Installing the PCIE 1GbE Server Side board 5 3 Removing the Management CPU Figure 29 Management Board Installed 42 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 31 Remove the Management Board Pull up the management card 5 4 Installing the ren xtender 1 Place the ESD m floor where you will be working and put on the ESD strap Make sure the E ap is touching your skin and that the other end is connected to a verified ground
34. environmental character istics of a fully populated platform The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take pre cautions to guarantee proper ventilation in order to maintain good airflow at ambient temperature Cable routing in particular should not impede the air exhaust from the chassis 4 6 Package Contents and Installation Before you install your new EVB board unpack the system and check to make sure that all the parts have been sent check this against the parts list below Check the parts for visible damage that may have occurred during shipping The switch comes packed with the following items Part Quantity Description EVB Switch 1 Switch EVB PCIE 1GbE 1 a kit that replaces the management module and allows for external man Extender kit agement Power cable 1 Type B 6ft US 125V 10A chord See Replacement Parts Ordering Num bers on page 52 to order power cords for various countries harness 1 For connecting from the RJ45 CONSOLE connector to a DB9 connector of a laptop host MTUSB 1 MTUSB For use by Mellanox FAEs only QSA 2 QSFP to SFP Adapter modules MC2207130 002 2 QSFP FDR 30AWG 2m cables MCCI1S30A 002 2 SFP 30AWG 2m cables bumpers 5x Rubber feet for using the EVB on a table top If anything is damaged or missing contact your customer representative immediately 4 6 1 Installing the Switch in the Rack Tools and Custo
35. ght Both of these LEDs give identical information Green OK Yellow a fault in the sys tem Red Major Fault or Fatal Error Led Color shows the switch status Power side Connector side status LED status LED The system status indicators should display as follows When the switch is plugged in within three minutes the STATUS LED should light up green The PSU LED should light up green As long as there is power to the switch and the switch is booted up and running the status LED should be green The FAN LED should light up green If the STATUS LED shows red after three minutes unplug the switch and call your Mellanox representative for assistance If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See Troubleshooting on page 64 If the PSU LED is not green this indicates a problem with the power supply The switch is opera tional only if the PSU LED is green Mellanox Technologies 17 Rev 1 0 Basic Operation and Installation Figure 5 Power Fan and System LEDs Table 2 System Status LED Configurations LED Configuration STATUS System Health LED Green OK The system is up and running Flashing Green The system is booting up Yellow Error A fault in the system most likely the firmware did not BOOT properly Red Major Error Possible damage can result to the s
36. he Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the switch make sure to use the shut down procedure 4 6 4 Extracting and Inserting the Power Supply Unit Power supply unit has directional air flow similar to the fan module The Fan module airflow must coincide with the airflow of the PSU If the PSU airflow direction is dif ferent from the fan module airflow direction the system will shut down 34 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 20 Power Supply Unit Extraction XI II II II XX II II II xx 64 XXI II IX II XX II XI XI XI XI II II XI Ir xr II Latch Release Figure 21 PSU Pulled Out Power Side Fan Unit To extract a PSU 1 If possible run the shut down procedure See Switch Shut Down Procedure on page 49 2 Remove the power cord from the power supply unit 3 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PSU unseats the PSU status indicators will turn off 4 Remove the PSU To insert a PSU 1 sure the mating connector of the new unit is free of any dirt and or obstacles
37. he LEDs will flicker for a fraction of a second to indicate reset is occurring You will then be able to enter without a password and make a new password for the user admin 50 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 6 Management and Tools Overview The switch to be managed either remotely or Out of Band using MLNX OS 6 1 Chassis Management Using the MLNX OS Stack The EVB comes standard with a management software module for chassis management called Mellanox Operating System MLNX OS MLNX OS is installed on all SwitchX based managed switch systems There is no need for a license or a key for IB SM functionality MLNX OS also includes a CLI WebUI and chassis management features Fan status PS status temp etc by default The Ethernet ports for remote management connect to Ethernet switches These switches must be configured to 10M 100M 1G auto negotiation switch EVB support up to 648 nodes If there is a need to support more than 648 nodes a license for Unified Fabric Manager UFM is needed Management modules have the capability to allow remote monitoring and remote management of the chassis from any host connected to the fabric The managed switch system includes a CPU which contains embedded OS secure in band out band access chassis manager and system BIST SNMP agent 3rd party tool integration GUI subnet manager The chassis manager will give the user access
38. le Off Off The fan unit is not receiving any power Check that the fan unit is properly and com pletely inserted All fans must be operating while the power supply is plugged in If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See Troubleshooting on page 64 4 1 1 6 BadPortLED The Bad Port indicator is located on the left side of the connector side panel of the unit The fol lowing Bad Port conditions are possible Mellanox Technologies 21 Rev 1 0 Basic Operation and Installation Table 6 Bad Port LED Configurations LED Configuration Description Off OK All ports are up and running Flashing Orange Error possibly more ports has just received a symbol error This LED shows symbol errors Possible causes for this are bad cable bad connection bad connector This LED lights up when one or more ports is receiving a symbol error The LED immediately goes off until the next symbol error is received 4 1 1 7 UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future For details please contact Mellanox Technologies support Reset Button On the connector side panel under the system LEDs is a reset button This reset button requires a tool to be pressed a paper clip will do DO NOT use a sharp pointed o
39. mer Supplied Parts Phillips Screwdrivers 1 and 2 grounding screw ESD strap grounding wire sufficient to reach a valid ground Mellanox Technologies 29 Rev 1 0 Basic Operation and Installation ESD mat Parts included in the rail kit e 2 rails 14 recessed flat head screws for the standard switch e 2rail slides 10 caged nuts 2switch slides 10 pan head screws M6 Figure 15 Rack Rail Kit Parts Switch slide Rail Rail slide 3 1 Place the ESD mat the floor where you will be working and put the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the rack vertical support Either the side with the power supply units or the side with the IB connectors can be even with one of the vertical rack supports To Transfer the power cord from the connector side to the power side through the rail kit see Section B Transferring the Power Cord on page 71 Things to consider before choosing where to mount the rails and rail slides 30 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 The distance between the rack and the door can be as little as 4 cm on one side of the rack and as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables connected to the switch yo
40. n be removed and in its place a PCIE 1GbE extender can be installed When the Management module is removed the EVB becomes externally managed The PCIE 1GbE extender kit contains the system side board and the server side board The board has four connectors Interface Description connector for clocking and reset 2 QSFP connectors one is blue one is green The green one goes from the green labeled connector on the EVBduaghter board to the green label on the server side card included in the kit The blue one goes from the blue labeled connector on the EVBduaghter board to the blue label on the server side card included in the Kit Server Side board a card that gets inserted into the server X8 PCI slot Use only one option the PCIE or the Ethernet do not use both RJ 45 connector for management through an ethernet port The server side board in the kit gets inserted in a server Connect the card into a 8X or 16X PCI slot This card has 3 ports Interface Description CX4 connector for clocking and reset 2 QSFP connectors one is blue one is green The green one goes from the green labeled connector on the EVBduaghter board to the green label on the server side card included in the kit The blue one goes from the blue labeled connector on the EVBduaghter board to the blue label on the server side card included in the kit 1 2 1 PCIE Extender Kit This option allow
41. nce Connects to Port 1 Ref to compliance port 2 6 SMA Reference Connects to Port 1 Ref to compliance port 2 7 SMA Reference Connects to Port 1 Ref to compliance port 2 8 SMA Reference Connects to Port 1 Ref to compliance port 2 9 SMA Reference Connects to Port 2 Ref to compliance port 4 10 SMA Reference Connects to Port 2 Ref to compliance port 4 Mini SMP Data Connects to HW port 27 on SwitchX 12 Mini SMP Data Connects to HW port 27 on SwitchX 13 Mini SMP Data Connects to HW port 27 on SwitchX 14 Mini SMP Data Connects to HW port 27 on SwitchX 15 Mini SMP Data Connects to HW port 27 on SwitchX 16 Mini SMP Data Connects to HW port 27 on SwitchX 17 Mini SMP Data Connects to HW port 27 on SwitchX 18 Mini SMP Data Connects to HW port 27 on SwitchX 19 Mini SMP Data Connects to HW port 27 on SwitchX 20 Mini SMP Data Connects to HW port 27 on SwitchX 21 Mini SMP Data Connects to HW port 27 on SwitchX 22 Mini SMP Data Connects to HW port 27 on SwitchX 23 Mini SMP Data Connects to HW port 27 on SwitchX 24 Mini SMP Data Connects to HW port 27 on SwitchX 25 Mini SMP Data Connects to HW port 27 on SwitchX 26 Mini SMP Data Connects to HW port 27 on SwitchX 27 12V 1 Mellanox Technologies 47 Rev 1 0 Installing the PCIE 1GbE Server Side board Table 7 Port Connector Numbering Continued Sheet 2 of 3
42. not flexible as such they should be carefully attached to p N or detached from the connectors Refer to the cable manufacturer for special warnings 7 N and instructions Support the weight of copper by tying them to the rack or to a cable management system 7 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equip ment starting from the bottom to the top 8 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 9 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula M tions 10 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical hk codes Mellanox Technologies 27 Rev 1 0 Basic Operation and Installation 11 Battery Replacement There is a risk of explosion should the battery be replaced with a battery of an incorrect type 7 Dispose of used batteries according to the instructions For Shark and Mammoth 12 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 82
43. nual Rev 1 0 Appendix B Transferring the Power Cord To use the rail kit to transfer the power cord from the connector side to the power side follow these directions Do you want to install power cords on both sides of the switch or only one side For each power cord you want to transfer 1 Once you have decided which side of the switch will be next to the vertical support 2 sure that you place the cord so that the correct end of the cord will be at the power side of the switch This will depend on which of the four mounting options you choose Figure 38 Transfer Power Cord 3 power cord in the switch slide channel Push the cord end through the hole in the switch slide Leave 7 of cord hanging out from the hole 5 Screw the switch slides to the switch Make sure to use two cord clamps per switch slide These will be needed to hold the cord during the switch installation process Mellanox Technologies 71 Rev 1 0 Figure 39 Install the Power Cord Clip 6 7 Return to step 4 on page 29 and continue the installation 72 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Appendix C Samtec Connector Pinout Figure 41 Connector Pinout soleo lt ex st O ed rco O lt oo ole x 0 5 2 0 5 sss m S ass S EXE ES HAA AA ao mS 6 OO
44. o the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 gt Step 5 Before attempting a remote for example SSH connection to the switch check the inter face configuration Specifically verify the existence of an IP address To check the current configuration enter the following commands sx 43 standalone master gt enable sx 43 standalone master show interfaces mgmt0 58 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 The following is an example of the output Mellanox Technologies 59 Rev 1 0 Connecting to the Switch Platform 7 Connecting to the Switch Platform When the Power PC is removed the switch EVB becomes externally managed unmanaged and can only be managed through the host server management or a through the remote Ethernet connector 7 1 Starting an SSH Connection to the Switch CLI Step1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connector Step 2 Connect to the remote machine rem mach1 is used as an example Step 3 Start a remote shell to the switch using the following command switch IP address is the IP address of the switch or its DNS name rem machl gt ssh 1 username ip address gt Mellanox MX OS Switch Management Password Last login Thu Apr 28 11 24 13
45. return to the wizard from the CLI enter the figuration jump start command from configuration mode Launching CLI lt switch name gt gt The wizard displays a summary of your choices and then asks you to confirm the choices or to re edit them Either press lt Enter gt to save changes and exit or enter the configuration step number that you wish to return to Note To run the command configuration jump start you must be in Config mode 56 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Table 10 Configuration Wizard Session IP Zeroconf Configuration Wizard Session Display IP Zeroconf Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on mgmt0 interface no Step 3 Use zeroconf on mgmt0 interface no yes Step 4 Default gateway 192 168 10 1 Step 5 Primary DNS server Step 6 Domain name Step 7 Admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on interface no Use zeroconf on mgmt0 interface yes Default gateway 192 168 10 1 Primary DNS server Domain name Admin password Enter to leave unchanged unchanged RU To change an answer enter the step number to return to Otherwise hit lt enter gt to sa
46. rfaces 23 4 4 2 Management and Firmware Updating Interfaces 24 4 5 Switch Platform Installation and Operation 26 4 5 1 Installation Safety Warnings 26 4 5 2 Mechanical Installation 28 4 6 Package Contents and Installation 29 4 6 1 Installing the Switch in the Rack 29 4 6 2 Grounding the Switch 33 4 6 3 Power Connections and Initial Power On 34 4 6 4 Extracting and Inserting the Power Supply Unit 34 4 6 5 Cable Installation 36 4 6 6 Extracting and Inserting the Fan Unit 36 Chapter 5 Installing the PCIE 1GbE Server Side board 38 5 1 PCIE 1GbE Extender Kit parts 38 5 2 Removing the Top Cover 40 5 3 Removing the Management CPU 42 5 4 Installing the PCIE 1GbE Extender 43 5 5 Installing the Server Side Board 45 5 6 Board Connectors Numbering and Locations 46 5 7 Switch Shut Down Procedure 49 5 8 Disassembly of the Switch from the Rack 49 5 9 Disposal 50 5 10 Resetting the Switch 50 Chapter 6 Management and Tools Overview 51 6 1 Chassis Management Using the MLNX OS Stack 51 6 2 Network Management and Clustering Software 51 6 3 Chassis Management 52 6 3 1 Open SM 52 6 4 Fabric Management with UFM 53 6 5 Configuring the Switch for the First Time 54 Chapter 7 Connecting to the Switch Platform 60 Mellanox Technologies 3 Rev 1 0 7 1 Starting an SSH Connection to the Switch CLI 60 7 2 Starting a WebUI Connection to the Switch 60 7 2 1 Managing the System 62 7 2 2 120 Connector 62 7 2 3 How to Get Mellanox
47. rformance and maxi mize fabric resource utilization All fabric diagnostics capabilities are provided via UFM There are two options for UFM Advanced and Standard Advanced Standard run on hosts only and includes advanced fabric management and diagnostics UFM Diags provides IBDiagnet like functionality and runs as a plug in module to MLNX OS on the embedded processor for managed SX based switches A license is required for all three UFM options To purchase UFM Diags enter a key into the MLNX OS dashboard of the managed switch to activate the feature Provides a central management console including the following main features Fabric Dashboard including Fabric Congestion Map Advanced Fabric wide Real time and History Monitoring Quality of Service Traffic Aware Routing Algorithm TARA Central Device Management Task automation Threshold based alerts Fabric segmentation isolation Subnet Manager Configuration Logging High Availability Mellanox Technologies 53 Rev 1 0 Management and Tools Overview 6 5 Configuring the Switch for the First Time When the Power PC is removed the switch EVB becomes unmanaged and does NOT need Initial configuration Step 1 Connect the host PC to the CONSOLE 45 port of the switch system using the supplied cable The CONSOLE ports for the EVB is shown below Figure 37 Management Interfaces MED MT51336EN EVB 336 EN EVB leen rez
48. rn off Figure 24 Fan Module Latches II XI XI XI II II II XI XI XI XI xr XI XI XI XI XI II II II II II II x 4 Ir II Ff Xx be 74 xx 1 XXII XY II XI II v5 64 xx II pxx 59 1 91 1 IX II xXx xx XI XI II 3631 II II 2 II II XI II XI II xr II XI II xr XI XI XI IX XIX XI IJ xk x Ir XI XI XI XI XI IX IX IX IX XI II These twodatches must be pushed towards each other at the same time while the module is pulled out To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely AN N The green fan status indicator should light If not extract the fan unit and reinsert it 7 After two unsuccessful attempts to install the fan unit power off the switch before 0 attempting any system debug Mellanox Technologies 37 Rev 1 0 Installing the PCIE 1GbE Server Side board 5 Installing the PCIE 1GbE Server Side board The PCIE 1GbE server side board disables the internal Management capabilities and disconnects the management interfaces When installed the system side board allows for direct contact with the SwitchX device for external management and SW development purposes The
49. s checking for duplicate or bad GUIDs Open SM To manage the Mellanox switch system using OFED download Mellanox Open Fabrics from http www mellanox com gt Downloads gt InfiniBand SW Drivers Be sure to read and follow all of the instructions regarding the installation and use of these tools Each subnet needs one subnet manager to discover activate and manage the subnet Each network requires a Subnet Manager to be running in either the switch itself switch based or on one of the nodes which is connected to the fabric host based The subnet manager OpenSM assigns Local IDentifiers LIDs to each port connected to the fab ric and develops a routing table based on the assigned LIDs A typical installation using the OFED package will run the OpenSM subnet manager at system start up after the OpenIB drivers are loaded This automatic OpenSM is resident in memory and 52 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 sweeps the fabric approximately every 5 seconds for new adapters to add to the subnet routing tables 4 No more than two subnet managers are recommended for any single fabric 6 4 Fabric Management with For an added licensing fee you can purchase Mellanox s Unified Fabric Manager UFM is a powerful platform for managing scale out computing environments It enables data center opera tors to efficiently monitor and operate the entire fabric boost application pe
50. s M a m To remove disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Care should be taken not to impede the air exhaust flow through the ventilation holes next to the connector ports Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 4 6 6 Extracting and Inserting the Fan Unit This switch can operate indefinitely with one of the three fans in the fan module inoperable so long as the ambient temperature is below 45 Celsius Operation without a fan unit should not exceed two minutes During fan hot swap if both indicators are OFF then the fan unit is disconnected 36 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Figure 23 Air Flow Labels IX IX xr XX xr XI IX XX II II xr II Mellanox pI Li XI II xx II XI II xXx IX II II II 4 IX II 9 II II II 79 644 xx xx II II IX II II II XI XI XXI XI XI XI XI XI IX II xr IX II These air flow labels must be the same To extract a Fan Unit 1 Using two fingers push both latch releases towards each other simultaneously while pulling the fan module out of the switch As the fan unseats the fan status indicator will tu
51. s a single USB connector This connector can be used to install software and or firmware upgrades using a memory device that has a USB connector _ This connector will not work if the management module is removed That is if you using the PCIE 1GbE Extender 4 4 2 4 12 Interface There is I2C connector banana type connector on far left of the power side of the switch This interface is for Debug and Troubleshooting only by a certified Mellanox FAE This con Mellanox Technologies 25 Rev 1 0 Basic Operation and Installation nector can be used to install firmware upgrades should the firmware image be damaged and can not be upgraded through a host PC or remotely This interface is for support personnel and advanced users only There are also three more three pin I2C interfaces on the board Access to these connectors is only available with the top cover removed 4 5 Switch Platform Installation and Operation Installation and initialization of the switch platform are straightforward processes requiring atten tion to the normal mechanical power and thermal precautions for rack mounted equipment The switch EVB platform requires initial configuration to operate as an InfiniBand switch All switches come with an internal PPC based management board This board allows for managing the switch through a host PC or remotely through the Ethernet 4 5 1 Installation Safety Warnings For Safety Warnings in
52. s the EVB to be managed through the server This is done with three cables con necting the EVB to the Server One cable is for clocking and reset and two cables are for data TX and RX 1 2 2 1GbE Extender Kit The PCIE extender allows for remote management through the 10 100 1000Mb RJ45 Ethernet connection on the EVB Extender side board This is connected to any standard Ethernet port fabric management port 10 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 1 3 Compliance Ports Ports 3 and 4 and 33 and 34 are connected to the chip These are compliance ports and go to the SMA mini SMP interfaces Ports 3 and 4 are SMA ports and ports 33 and34 are mini SMP ports Compliance ports are the worst case loss for all of the on board connectors through the channel They are designed to have an insertion loss of 5 5dB as defined in SFF 8431 See Figure 12 on page 23 for identifying the trace and compliance ports See Figure 35 on page 46 and Table 7 on page 47 for locations and numbering of the trace and compliance port board con nections 4 As of this revision only port 4 is verified at the proper designed insertion loss Use port 4 for compliance 1 4 Reference Traces Ports 1 and 2 and 35 and 36 are not connected to the chip These are reference ports and go to the SMA mini SMP interfaces Figure 35 on page 46 and Table 7 on page 47 For locations and numbering of the board connections Ports 1 and 2 are
53. to switch temperatures power supply voltages fan unit information power unit information Flash memory The manager also has the ability to burn new firmware on the switch 6 2 Network Management and Clustering Software Download and install on all nodes the Mellanox OpenFabric software package for Linux Win dows or other operating systems from the Mellanox software website Mellanox Technologies 51 Rev 1 0 Management and Tools Overview 6 3 6 3 1 http www mellanox com gt Downloads gt InfiniBandAdapter Ethernet SW Drivers This software package provides connectivity for server and storage systems utilizing High Perfor mance Computing HPC or enterprise data center EDC applications across a fabric It also pro vides a subnet manager for simple network configuration and network administration and diagnostic tools for network management Chassis Management The chassis management features include upgrading drivers upgrading software monitoring of AC power to the PSUs DC power out from the PSUs board temperature fan module unit failure in the switch system system failure in the switch system querying for board serial numbers and their revisions In addition the tools enable firmware management capabilities such as querying for existing firmware versions burning new firmware from scratch or for recovery from damaged firmware querying for and changing system GUID
54. tp www mellanox com gt Downloads gt Firmware Tools The MFT kit includes mlxburn flint spark debug utilities See Related Documentation on page 7 62 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 7 3 Updating Firmware When you buy the switch it comes with the latest firmware burned on the board All firmware updates should be done through the management software Go to the Mellanox Website and con firm that the firmware is the latest If not reburn the latest firmware from the download site New firmware versions will be posted on the Mellanox firmware download page http www mellanox com gt Support gt Download Center You will need the MLNX OS package or Mellanox UFM to update firmware for this switch The MLNX OS package can be downloaded from http www mellanox com gt Products gt Management Software gt MLNX OS You will also need to download and unzip the firmware binary image This is provided in the Mel lanox Web site at http www mellanox com gt Support gt Download Center and go to the SwitchX Switch systems Click in the Table for the firmware image that you need The Mellanox UFM Management software is also an option to more closely manage your fabric switches and nodes Be sure to read and follow all of the instructions regarding the updating of the firmware on your switch system Firmware for the HCA cards connected to this switch s
55. u want the connector side recessed in the rack to allow for a larger cable bending radius It is possible to recess the connector side by 5cm by optional placement of the switch rails See Figure 16 Mounting Options Will the connector side be recessed past other equipment in the rack and will this be prob lematic M The installation kit allows for a 2 recess of the switch past tical support Figure 16 Mounting Options Support Connector side Rack door Power sid Connector side ower side Power side Connector side Rack door Power side Connector side 3 Decide which mounting option you want to use Mellanox Technologies 31 Rev 1 0 Basic Operation and Installation 4 To use the rail kit to transfer the power cord from the connector side to the power side go now to Transferring the Power Cord on page 47 Screw the switch slides onto the switch Use 5 flat head screws for short switches and 7 screws for standard depth switches to connect each switch slide Figure 17 Screwing on the Rail P 5 pur 7 d a F Lome i U mo LC 8 7 F4 F Y 7 screws side are needed for the standard switch Figure 18 Inserting the Caged Nuts Side you will
56. ve changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 gt Mellanox Technologies 57 Rev 1 0 Management and Tools Overview Table 11 Configuration Wizard Session Static IP Configuration Wizard Session Display Static IP Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on mgmt0 interface yes n Step 3 Use zeroconf mgmt0 interface no Step 4 Primary IP address 192 168 10 4 Mask length may not be zero if address is not zero interface mgmt0 Step 5 Netmask 0 0 0 0 255 255 255 0 Step 6 Default gateway 192 168 10 1 Step 7 Primary DNS server Step 8 Domain name Step 9 Admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on interface no Use zeroconf on mgmt0 interface Primary IP address 192 168 10 4 Netmask 255 255 255 0 Default gateway 192 168 10 1 Primary DNS server Domain name Admin password Enter to leave unchanged unchanged To change an answer enter the step number to return to Otherwise hit enter to save changes and exit Choice Configuration changes saved To return t
57. witch Turn off immediately e g bad firmware can t boot overheated Off Off The system has no power 4 1 1 2 Power Side Panel System LED On the right side of the power side panel is a single system LED that displays the health of the switch This indicator is the same as the system status indicator on the other side of the switch The LED assignment is as in Table 2 Figure 6 System Health LED his is a three color LED Green OK Yellow Fault in the system Red Major fault 4 1 1 3 Port Connector LED Assignment Above the ports are two LEDs one for the upper port A and one for the lower port V The fol lowing table shows the port status according to the LED indication 18 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 Table 3 Connector Physical and Logical Link Indications LED Status LED Description Off No power to the port Solid Green Logical link up Flashing Green Data activity flashing speed data transfer speed Orange Physical link up Flashing Orange A problem with the physical link Usually the SM is down ornot enabling the port 4 1 1 4 Power Supply Status Indicators The switch EVB is only available with one installed Power Supply Unit Figure 7 Power Side Panel Fan status LED PSUI status LED Power Supply Unit Fan Unit per The power supply unit PSU is located on the right side of the power side panel The
58. x FAEs andvadvances users only Figure 14 Management Interfaces MT51336 EN EVB MGTV 4 4 2 1 RJ 45 Connector CONSOLE The port labelled CONSOLE is for a local host connection to the management module This is used the first time the switch is connected An HAR 000028 harness is included in the package to 24 Mellanox Technologies SwitchX Switch EVB Hardware User Manual Rev 1 0 connect to a DB9 connection on a host PC Connecting to a local host PC and following the instructions in the Installation Guide Configuring the Switch for the First Time must be done before any remote management is available For the Socket pinout see RJ45 CONSOLE and Ethernet Interfaces on page 51 gt This connector will not work if the management module is removed That is if you are using the PCIE 1GbE Extender Lil gt 4 4 2 2 45 Ethernet Connector MGT The Ethernet connection labelled MGT provides access for remote management The switch can be connected to any Ethernet port This connector will not work if the management module is removed That is if you are using the PCIE 1GbE Extender 4 Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10M 100M 1G auto negotiation 4 4 4 2 3 USB interface The USB interface can be used to update the MLNX OS SX Web User Interface WebUI or MLNX OS SX Command Line Interface CLI aa There i
59. ystem can be downloaded from the same site Mellanox Technologies 63 Rev 1 0 Troubleshooting 8 Troubleshooting As soon as a switch is plugged in make sure that the green power LEDs on the PSUs are on Status LED and or Status Health LED If either of these two LEDs is red unplug the switch and call your Mellanox representative Power supply unit If the LED on the PSU is not lit or is red check that the power cable is plugged into a working out let 1 Check that the power cable has a voltage within the range of 100 240 volts AC 2 Check that the air flow direction of the PSUs are consistent with the Fan module air flow 3 Remove and reinstall the power cable 4 Remove and reinstall the PSU The power LED for the switch shuts off 1 Check that the there is adequate ventilation Are the fan LEDs showing that the fans are all up and running 2 sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner The green power LED for the fans does not come on 1 Check that the Power LEDs are on 2 Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 4 6 6

Preliminary Draft

Contents

Download Pdf Manuals

Related Search

Related Contents