SX6536 Installations Guide
Contents
1. A f VN The board is short therefore do not let go of it while sliding it out Figure 25 Management Module mo aap Sees the iat Age as a Geena UUA T S a a san UNNI Ejectors 1 9 Inserting a Spine Board 1 Start with the latches fully open that is perpendicular to the front panel of the leaf 2 Hold the spine board by the sides with both latches fully open 3 Insert the board into its rail The board should slide in without resistance 31 Rev 1 0 Chassis Installation Push the board in until the hooks on the latches are touching the vertical bar 5 Now press both latches inward simultaneously You should feel the latches are hooked into the front panel 6 Press both latches applying some force to close them completely The locking latch will engage Figure 26 Ejector Latch Pull the module out of the chassis Spine Push the latch l release E Q Figure 27 Spine Board Insertion Open the Ejector Latches CEELI Danann nn BEEBE Goooooggg HE Tr Al Innan na sie mna nan EERE E EEE 32 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 2 Installing the Cable Holder WY Now is the time to install the cable holder Figure 28 Cable Holders 1 Place the cable holder next to the rack on the connector side of the chassis and identify the holes where2. Rev 1 0 enough play in the bracket so that it can move to the vertical support Do not tighten the hex head bolts You may need to remove the brackets and reinstall them gt Figure 6 Placing the Shelf Bracket Adjusting slots 6 Insert five caged nuts into the rack for each of the four shelf brackets the 2 brackets that are adjustable and the 2 that are fixed Make sure that the caged nuts are all at the same height on the vertical supports Do not use the top hole in the bracket Rev 1 0 Chassis Installation Figure 7 Caged Nut Locations for the Shelf Brackets 7 Put the Shelf into the rack and twist the shelf into place Make sure the spine side of the shelf is closest to the rack doors See Figure 8 to identify the spine side of the shelf 8 Install and tighten the 20 M6 bolts into the caged nuts 20 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual M Figure 8 Shelf Brackets The spine side of the shelf has two slots for the extension _ Adjustable shef brackets EE E EET EE EEES Av ee ee AV dh LT a 2 sasn A E E E E E E EA 9 Tighten the 16 hex head bolts on the adjustable brackets 10 Connect the lock down bar onto the shelf at the connector side Figure 9 Place the Lock Down Bar on the Shelf ee a the lock dowi here 2 spine side of shelf has two ts for the exten n 11 Place 14 caged nuts into the vertical
3. 1 3 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order 1 4 Spine Package Contents The spines are ordered by the customer and are shipped 3 to a box The customer will receive as many boxes as needed to fill the order 1 5 Management Package Contents The package includes e all of the management modules ordered by the customer e 1 RJ45 to DB9 harness for each management module received 1 6 Physical Installation chassis weighs 740 lbs 335 kg Make sure that proper manpower and equipment Warning This equipment is very heavy Safety is the first concern The fully loaded A is used for transporting and moving the chassis The switch platform uses 31U of rack space in a standard 19 rack 29U for the chassis and 2U for the shelf The switch ships from the factory with mounting holes on the spine side There are upper brackets to connect the leaf side to the rack near the top of the chassis and there are two lock down bars to secure the chassis to the shelf The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have depths between 65cm and 80cm between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling Choose a rack which is able to support the mechanical
4. 5 InfiniBand QSFP Cable Installation 38 5 1 Cable Power Classes 39 Chapter 6 Chassis Power Up 40 6 1 Power Supply Management Board and Spine Board Indicator Status at Power ON 41 Chapter 7 InfiniBand Fabric Initialization and Management 42 7 1 Configuring the Switch Management Modules for the First Time 42 7 2 Rerunning the Wizard 47 Chapter 8 MLNX OS Management Software 48 8 1 Downloading MLNX OS Software and Documents 48 8 2 Starting an SSH Connection to the Switch 48 8 3 Starting a WebUI Connection to the Switch 48 Chapter 9 Resetting the Switch RST 50 Rev 1 0 List of Figures Figure 1 Distance Between the Vertical Supports 13 Figure 2 Installation Kit Parts 15 Figure 3 Container Screws 16 Figure 4 Vertical Rack Supports Dimension 18 Figure 5 Adjustable Shelf Brackets 18 Figure 6 Placing the Shelf Bracket 19 Figure 7 Caged Nut Locations for the Shelf Brackets 20 Figure 8 Shelf Brackets 21 Figure 9 Place the Lock Down Bar on the Shelf 21 Figure 10 U Numbering Above Shelf 22 Figure 11 Putting the Chassis in the Rack 23 Figure 12 Location for the Lock down Bar 23 Figure 13 Place the Upper Bracket Offset 24 Figure 14 Attach the Shelf Extension 24 Figure 15 Place Caged Nuts 25 Figure 16 Fork Lift 26 Figure 17 Face Plate Mounting Bolt Locations 27 Figure 18 Upper Brackets and Offset Location 28 Figure 19 Upper Bracket Installation 28 Figure 20 Offset to Upper bracket Bolts 29 Figure 21 Lock Down Bar 29
5. AMN Mellanox Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS and SwitchX are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 648 Port InfiniBand FDR Switch Platform Installation Guide 2 Mellanox Technologies Document Number 3504 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table of Contents Chapter 1 Chassis Installation 10 1 1 Environmental and Safety Recommendations 10 1 2 Chassis Package Contents 11 1 3 Leaf Package Contents 12 1 4 Spine Package Contents 12 1 5 Management Package Contents 12 1 6 Physical Installation 12 1 6 1 Starting with the Rack 13 1 6 2 ESD Connection 13 1 6 3 Installation Procedure 14 1 7 Installing the Leafs 30 1 8 Inserting a Management Module 31 1 9 Inserting a Spine Board 31 Chapter 2 Installing the Cable Holder 33 Chapter 3 Ground Connections 35 Chapter 4 Power Connections 36 4 1 Powering Up the Switch Platform 36 Chapter
6. and environmental characteris tics of a fully populated switch chassis gt 12 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 6 1 Starting with the Rack 1 6 1 1 Rack Recommendation that is 120 cm long and 80cm wide This will allow for proper cable management and Y Due to the space required by up to 648 connector cables Mellanox recommends a rack enough ventilation to properly cool the chassis 1 6 1 2 Preparing the Rack The rack may need to be modified to accommodate the SX6536 chassis The distance between the vertical supports must be between 65 and 80 cm Figure 1 Distance Between the Vertical Supports This edge of the shelf must be as close as possible to the inside of the door when closed eo ENNENENENENENENEN EEE E A E Em ENEE E The vertical supports may have to be moved so that the vertical support closest to the door is 6 cm from the inside of the door This may be required so that the shelf does not pro trude from the rack If you move the vertical supports make sure that you reinstall them vertically plumb gt 1 6 2 ESD Connection Before starting any procedure on the SX6536 series switch system 1 Putan ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap Rev 1 0 Chassis Installation 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a
7. gt is the IP address of the switch or its DNS name 4 You will receive the login window for remote management of the switch 48 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 37 Web UI Login Page Mellanox TECHNOLOGIES bes iA i j 4 4 IB SM i PER setup m System amp ra Security Ports g Status bed MGMT DA Please enter your username and password then click Login 5 Use admin for both the login Account and the password Rev 1 0 Resetting the Switch RST 9 Resetting the Switch RST The Reset button resets the chassis management module when the button is pushed When the but ton is held down for 15 seconds the chassis is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button Sharp objects can cause damage use a flat object such as a paper clip Figure 38 Reset Button CONSOLE MGT A RST PSU S FANS L FANS MASTER MGT M im O Yuu mn This button resets the CPU of the management module A quick push of this button performs this reset When the reset button is pushed on the master management module this management mod ule is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button
8. is pushed When the button is held down for 15 seconds the management module is reset and the password is deleted You will then be able to enter without a password and make a new password for the user admin For more information and documentation on the SX6536 Series go to the Mellanox Website http www mellanox com 50 Mellanox Technologies Rev 1 0 Mellanox TECHNOLOGIES USA Mellanox Technologies Inc 350 Oakmead Parkway Suite 1 Sunnyvale CA 94085 Tel 408 970 3400 Fax 408 970 3403 www mellanox com E mail info mellanox com Map and Directions Tech Support 408 916 0055 8 00AM to 5 00 PM Pacfic 648 Port InfiniBand FDR Switch Platform Installation Guide A CC00 0 3 8 3 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam Israel 20692 Telephone 972 4 909 7200 Fax 972 4 959 3245 E mail info mellanox com Tech Support 972 4 9097200 ext 250 8 00AM to 5 00 PM Israel Time www mellanox com 51 Mellanox Technologies
9. requires a KVA rated UPS system The heat dissipation is calculated Power Dissipation x 3 42 Heat Dissipation It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit 1 2 Chassis Package Contents The package includes e 1 chassis with the following modules installed 8 fans 10 PSUs e 1 installation kit e 1 box containing 10 power cords 125V 10A C14 to C13 6 feet long USA UL Standard e 1 cable management kit Before you install your new SX6536 series chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have occurred during shipping The leafs spines and management modules are shipped separately 4 If anything is damaged or missing contact your customer representative immediately 4 The rack mounting holes conform to the IEA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis 4 Rev 1 0 Chassis Installation
10. the caged nuts Mellanox Technologies 33 Sn G Rev 1 0 Installing the Cable Holder P The cable holders should go to the outside of the vertical supports gt 2 Set the bottom of the cable holder at the level of the shelf 3 Screw the cable holder onto the rack using the screws provided 4 Repeat steps 1 3 for the second cable holder 34 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 3 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the connections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figure 29 Ground Connection Grounding post Rev 1 0 Power Connections 4 Power Connections The chassis includes integrated hot swap power supplies which support up to 10 load sharing 1000W supply units The slots for the power supply units PSUs are o
11. 2 for the upper brackets 14 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 32 lock washers 20 for the lock down bars 32 hex head bolts 20 for the lock down and 12 for the upper brackets to upper bracket bars and 12 for the upper brackets to upper offsets bracket offsets 2 upper brackets 12 flat head screws with locktite for the upper bracket offsets 1 open end ratchet wrench 10mm 1 socket nut driver 8mm 1 socket wrench 1 socket nut driver 10mm 2 extensions for the socket driver 1 socket nut driver 13mm 4 flat washers for the shelf extension e Parts included in the cable management installation kit 1 cable management rack RH 1 cable management rack LH 16 caged nuts M6 16 M6 bolts Figure 2 Installation Kit Parts Shelf Lock down bar Flat head screws with locktite Shelf extension 6 hex head bolt Phillips pan head screw 8 hex head bolt Upper bracket Washer Caged nuts Lock washers X8 Lock washers X6 Flat washers amp T oO at AN Nf 2 O Ai Upper bracket Flat washers Upper bracket offset 1 6 3 2 Opening the Container 1 Before starting the procedure put the ESD strap on and connect it to a valid ground 2 Remove the four sides of the container by unscrewing the phillips head screws hold ing the sides on Rev 1 0 Chassis
12. Figure 22 Bolt and Washer Order 30 Figure 23 Ejector Latch 30 Figure 24 Ejector Latch 31 Figure 25 Management Module 31 Figure 26 Ejector Latch 32 Figure 27 Spine Board Insertion 32 Figure 28 Cable Holders 33 Figure 29 Ground Connection 35 Figure 30 Multiple Power Inlets Electric Caution Notification 37 Figure 31 Spine Module 37 Figure 32 37 Figure 33 Top and Bottom Ports 39 Figure 34 Spine Side Panel Display Status Indications 41 Figure 35 Management Module Status Indications for Normal Operation 41 Figure 36 Connecting the Harness to the Console 42 Figure 37 Web UI Login Page 49 4 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 38 Reset Button 50 Rev 1 0 List of Tables Table 1 Revision History of this Installation Guide Table 2 Reference Documents and Web Sites Table 3 Serial Terminal Program Configuration 43 Table 4 Configuration Wizard Session IP Configuration by DHCP 43 Table 5 Configuration Wizard Session IP Zeroconf Configuration 45 Table 6 Configuration Wizard Session Static IP Configuration 46 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Revision History Table 1 Revision History of this Installation Guide Revision Date Details 1 0 July 2011 Initial Release Rev 1 0 About this Manual This manual provides installation and set up instructions for the SX6536 QSFP Chassis InfiniBand Switch Platfo
13. Installation Figure 3 Container Screws as FA 25 SA ee Container screws Remove and put aside the box containing the cables Remove and put aside the box containing the cable management kit Remove and put aside the box containing the Installation kit ON ES a Visually inspect the chassis make sure that there is no visible damage 10 PSUs are installed all 8 fans are installed management module is installed 7 Remove all protective plastic film from all sides and top of the chassis P The leafs spines and management modules are shipped separately gt 4 Do not remove any of the blocks or either of the lock down bars at this time 16 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 6 3 3 Installing the Shelf of the pallet Should the chassis start to fall move out of the way The chassis can The chassis is on ball bearings and can roll easily Be aware that the chassis can roll off A cause grave bodily harm should it fall on or near you 1 Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low v4 It is recommended to set the shelf at U5 Try and keep the chassis as low as possible The maximum height that the chassis can be mounted will be determined by the maxi mum height that the fork lift can raise the pallet 2 Open the Instal
14. Mellanox TECHNOLOGIES 648 Port InfiniBand FDR Switch Platform Installation Guide PN MSX6536 10R Rev 1 0 www mellanox com Rev 1 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE
15. Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 6 1 Power Supply Management Board and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system During this time the status LEDs may be RED Turn off the system if any LEDs remain red for more than 5 minutes As the power is turned on you should observe the following conditions for normal opera tion 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON 4 Check the Spine LEDs and make sure they coincide with Table 34 Figure 34 Spine Side Panel Display Status Indications Hf HE S amp S 8 amp G O STATUS PSU SPINE LEAF MGT 1 MGT 2 STATUS FANS FANS STATUS STATUS 5 Check the Spine LEDs and make sure they coincide with Table 35 Figure 35 Management Module Status Indications for Normal Operation EE i CONSOLE MGT Ae RST PSU S FANS L FANS MASTER T LL AL 3 FA 100 NN EE AA AN TA O FN Mellanox Technologies 41 O gen Rev 1 0 InfiniBand Fabric Initialization and Management 7 InfiniBand Fabric Initialization and Management Subnet Management and Subnet Administration are standard compon
16. ackets on the vertical supports on the connector side of the chassis 12 Slide the chassis out just far enough to install the upper brackets 27 Rev 1 0 Chassis Installation Figure 18 Upper Brackets and Offset Location Figure 19 Upper Bracket Installation This hole is 60cm up from the top of the shelf 13 Screw the upper brackets onto the vertical supports and tighten in place You will need the socket wrench with both extensions to tighten these bolts 28 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 14 Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack support 15 Make sure the upper brackets are installed in their correct location 16 Screw the bolts through the faceplate into the caged nuts and tighten all 14 bolts and nuts 17 For each offset and upper bracket screw six bolts through the offset into the upper bracket Use one flat washer and one lock washer for each of these bolts Figure 20 Offset to Upper bracket Bolts 18 Place the front lock down bar at the edge of the chassis and align the holes in the lock down bar with the holes in the shelf Screw down the lock down bar Figure 21 Lock Down Bar These holes are to screw the bar into the Chassis These holes are to screw the bar into the shelf Mellanox Technologies 29 Rev 1 0 Chassis Installation 19 Screw down
17. cm between the vertical supports of the rack Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the openings are populated with blanks All of the leafs spines and managmement modules come shipped in a separate pakage The chassis requires initial configuration to get the chassis and Fabric management up and running through remote management This unit is intended for installation in a Restricted Access Location A restricted access area can be accessed only through the use of a special tool lock and key or other means of security t 1 1 Environmental and Safety Recommendations The following are Mellanox recommendations V Recommended ambient temperature in the System room is 20 5 C Recommended humidity ranges is 40 15 without condensing It is highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference lt VY It is highly recommended that the installation site building be equipped with a light ning rod 10 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system The system
18. echnologies 45 e 7 Rev 1 0 InfiniBand Fabric Initialization and Management Table 6 Configuration Wizard Session Static IP Configuration Wizard Session Display Static IP Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on eth0 interface yes n Step 3 Use zeroconf on eth0 interface no Step 4 Primary IP address 192 168 10 4 Mask length may not be zero if address is not zero interface eth0 Step 5 Netmask 0 0 0 0 255 255 255 0 Step 6 Default gateway 192 168 10 1 Step 7 Primary DNS server Step 8 Domain name Step 9 Admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on eth0 interface no Use zeroconf on eth0 interface no Primary IP address 192 168 10 4 Netmask 255 255 255 0 Default gateway 192 168 10 1 Primary DNS server Domain name Admin password Enter to leave unchanged unchanged OANNKDNFKWNK To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 gt 5 Before attempting a remote for example SSH connection to the switch ch
19. eck the ethO interface configuration Specifically verify the existence of an IP address To check the cur rent ethO configuration enter the following commands sx 43 standalone master gt enable sx 43 standalone master show interfaces mgmt0 The following is an example of the output Interface mgmt0 state 46 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Admin up yes Link up yes IP address 192 168 10 43 Netmask 255 255 255 0 Speed 1000Mb s auto Duplex full auto Interface type ethernet Interface source physical MTU 1500 HW address 00 02 C9 11 2A AE Comment RX bytes 1343502058 TX bytes 313920869 RX packets 17589211 TX packets 992717 RX mcast packets 0 TX discards 0 RX discards 0 TX errors 0 RX errors 0 TX overruns 0 RX overruns 0 TX carrier 0 RX frame 0 TX collisions 0 TX queue len 1000 6 On SX65xx systems configure the Box IP BIP centralized location for chassis High Availability according to the steps in the MLNX OS Software UM section Box IP Centralized Location 7 2 Rerunning the Wizard If you want to rerun the wizard run the following commands sx 43 standalone master gt enable sx 43 standalone master configure terminal sx 43 standalone master configuration jump start Mellanox Technologies 47 eee Rev 1 0 MLNX OS Management Software 8 MLNX OS Management Software MLNX OS is a s
20. ents defined by the InfiniBand Specification which implement fabric initialization discovery and configuration party Subnet Management software running on an InfiniBand Host that is connected to any switch port The switch is also compatible with the Open Source Subnet Manager ba OpenSM p The switch s switching fabric can be managed by MLNX OS as well as any third The chassis is basically an InfiniBand fabric built out of individual InfiniBand switch devices and appears so to external Subnet Managers The chassis supports the SystemImage GUID feature of the InfiniBand specification which means that software management tools that support this feature can recognize all the internal devices as a single switch system 7 1 Configuring the Switch Management Modules for the First Time On SX65xx systems when having dual management systems first connect the cable and configure the master management module CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and slave roles see the MLNX OS Software UM section High Availabil ity for more information Initial configuration must be done on all of the management modules 1 Connect the host PC RS232 connector to the CONSOLE port of the switch using the sup plied cable Figure 36 Connecting the Harness to the Console Connect the host PC to here CONSOLE MGT qe RST PSU S FANS L FANS MASTER Trio o
21. ered on To insert a cable press the con nector onto the port receptacle until the connector is firmly seated The green LED indicator above the port will light when the physical connection is established when both ends of the cable are properly connected to working devices To remove disengage the lock and slowly pull the con nector away from the port receptacle For a valid physical connection both ends of the cable must be connected to working devices Warning Take care to not impede the air flow through the ventilation holes next to the InfiniBand ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack Cables in the bottom row of the leafs should be inserted up side down in relation to the how the cables are inserted in the top row of the leafs aa 38 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 33 Top and Bottom Ports Se 5 1 Cable Power Classes Chassis and switches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX65xx series chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables Rev 1 0 Chassis Power Up 6 Chassis Power Up Before starting any procedu
22. ertical supports 24 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 15 Place Caged Nuts o lt 26 als 3 25 lt E Place 8 caged nuts between here and here for the cable holders Unscrew the lock down bars Place the chassis in front of the rack and as close as possible to the rack Raise the chassis so that it is 5cm above the shelf in the rack Raise the hinged extension on the container floor Lower the chassis until the hinged extension is sitting on the shelf extension em Nnnm 10 Push the chassis from the container to the shelf Rev 1 0 Chassis Installation Figure 16 Fork Lift Container floo Hinged extension YY Save the container pallet until you want to remove this chassis It will be necessary for the removal and disposal procedures 26 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 17 Face Plate Mounting Bolt Locations e e TR Holes in the faceplate for mount TECHNOLOGIES ing on the rack SX6536 The holes are separated by 4U units mig m g Val il 11 Slide the chassis onto the shelf and determine the location of the upper br
23. lation kit and make sure that all of the parts are in the box See Figure 2 Installation Kit Parts The chassis will sit in the rack with the spine side as close as possible to the door The side of the chassis with the spine units will sit flush with the vertical supports of A the rack The side of the chassis with the QSFP connectors will sit at the edge of the ai shelf closer to the center of the rack 3 Measure the distance between the vertical supports and adjust the shelf bracket according to this distance This chassis can be installed a rack with a distance between the vertical supports of from 65cm 25 6in to 80cm 31 5in Rev 1 0 Chassis Installation Figure 4 Vertical Rack Supports Dimension 4 The shelf comes with the four brackets pre installed Two of the brackets should be loose They may need to be removed and repositioned to meet the distance between the vertical supports of the rack Figure 5 Adjustable Shelf Brackets These two brackets will need to be adjusted They may even need to be removed and repositioned The holes in the bracket are offset to one side The holes must be closer to the top of the shelf 5 Move the shelf brackets on the shelf Place the shelf bracket so that it measures slightly more than the measured distance between the vertical rack supports Make sure that there is 18 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual
24. n asks you to confirm the choices or to re edit them Either press lt Enter gt to save changes and exit or enter the configuration step number that you wish to return to Note To run the command configuration jump start you must be in Config mode 44 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table 5 Configuration Wizard Session IP Zeroconf Configuration Wizard Session Display IP Zeroconf Configuration Example Mellanox configuration wizard Do you want to use the wizard for initial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on eth0 interface no Step 3 Use zeroconf on eth0 interface no yes Step 4 Default gateway 192 168 10 1 Step 5 Primary DNS server Step 6 Domain name Step 7 Admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on eth0 interface no Use zeroconf on eth0 interface yes Default gateway 192 168 10 1 Primary DNS server Domain name NADNDNFWNH Admin password Enter to leave unchanged unchanged To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 gt Mellanox T
25. n the spine side The left side has odd numbered PSUs and the right side has even numbered PSUs Each PSU has a dedi cated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Autorange 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords are standard 3 wire AC power cords including a safety ground Before starting any procedure on the SX6536 series switch system put an ESD prevention wrist strap on your wrist and connect to the series chassis 4 1 Powering Up the Switch Platform Make sure that the power cords are compatible with your outlets Power cords for dif ferent countries can be ordered from Mellanox See the user manual appendix for ordering part numbers for power cords 1 Plug in the power cords to the PSUs 2 Plug the other end of the power cord into a grounded outlets Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit The Chassis must be started a full compliment of possible PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may
26. ng Any yellow or red status LEDs on any of the management modules is cause A N for concern and must be dealt with immediately SS 4 Check that none of the LEDs on the spines are yellow N It can take up to 5 minutes to boot up the system during this time the status LEDs may A be RED Turn off the system if any LEDs remain red for more than 5 minutes SS 5 Check that the leaf status LEDs in the spines are lit Figure 31 Spine Module IB BBG LER TTT sjalen elen nn INN EE T ddd 000000 oono090000 EENES goo00000o0o mm 16 di 30 32 34 3 Spine Status Fan Status Spine Leaf IB Connection Status Figure 32 37 Rev 1 0 InfiniBand QSFP Cable Installation 5 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from Mellanox Technologies The standard QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidirectional wire speed of the switch ports All InfiniBand QSFP connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are numbered 1 18 If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase FDR BandWidth is supported only by MLNX QSFP cables FDR and FDR10 are only guaranteed to work with approved Mellanox Cables WW WW WW All cables can be inserted or removed with the unit pow
27. oftware based management system that can be run with either a command line interface or with a WebUI interface Chassis management comes with the switch system 8 1 Downloading MLNX OS Software and Documents 8 2 Starting an SSH Connection to the Switch SSH connection also supports TelNet gt 1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connector 2 Connect to the remote machine 3 Start a remote shell to the switch using the following command ssh l admin lt your ip address gt When prompted for a password enter admin Note that the IP address used above is the same IP address that was assigned to the Mellanox configura tion wizard in the Configuring the Switch Management Modules for the First Time section 4 You can enter any supported command now A For a complete reference of commands please see Mellanox MLNX OSManagement Software User s Manual gt 8 3 Starting a WebUI Connection to the Switch 1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connector 2 Start a Web browse Google Chrome Internet Explorer 7 0 or Mozilla Firefox 3 0 4 Make sure the screen resolution is set to 1024 768 or higher gt 3 Enter as URL the following http lt switch_IP_address gt where lt switch_IP_address
28. oo HP No remote IP connection is available at this stage gt 2 Configure a serial terminal program for example HyperTerminal Minicom or Tera Term on your host PC with the settings described in Table 3 42 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Table 3 Serial Terminal Program Configuration Parameter Setting Baud Rate 9600 Data bits 8 Stop bits 1 Parity None Flow Control None 3 Login from serial terminal program as admin and use admin as password This starts the Mellanox configuration wizard 4 Go through the Mellanox configuration wizard Table 4 shows an example of a wiz ard session Table 4 Configuration Wizard Session IP Configuration by DHCP Sheet 1 of 2 Wizard Session Display Example Comments Mellanox configuration wizard You must perform this configuration the first time you Do you want to use the wizard for initial configura operate the switch or after resetting the switch Type tion yes y and then press lt Enter gt Step1 Hostname switch 1 If you wish to accept the default hostname then press lt Enter gt Otherwise type a different hostname and press lt Enter gt Step 2 Use DHCP on eth0 interface yes Perform this step to obtain an IP address for the switch ethO is the management port of the switch If you wish the DHCP server to assign the IP address type yes and pres
29. power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs 36 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 30 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Schocks Risque de choc et de danger hazard Entferrnen des Netzsteckers elnes e lectriques Le de branchment The PSUs are all independent Netzteils spannungsfrei Um alle d une seule alimentation stabilise e Einhieten spannungsfrei zu machen ne de branch uniquement qu un Disconnect all power supplies to sind die Netzstecker aller Netzteile module Alimentation Stabilise e ensure a powered down state inside zu entfernen Pour isoler completement le module of the switch platform en cause II faut de brancher toutes les alimentations stabilise es 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Warni
30. rack support at the locations corresponding to the holes in the chassis faceplate The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support There are 7 holes on each side ellanox Technologies Rev 1 0 21 Rev 1 0 Chassis Installation Figure 10 U Numbering Above Shelf 2 lt i Ali The first cage nut on the spine side goes here A The next six caged nuts go every 4 U units up gt from here gt Top of i Shelf lt 1 6 3 4 Inserting the Chassis 1 Put the container on the fork lift 22 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Figure 11 Putting the Chassis in the Rack The chassis is screwed down to the pallet on both the spine side and the leaf side Figure 12 Location for the Lock down Bar Bottom of the chassis 2 Screw on the two Upper Bracket offsets onto the chassis using six flat head screws for each offset Rev 1 0 Chassis Installation Figure 13 Place the Upper Bracket Offset 3 Screw on the shelf extension using two bolts with flat washers Figure 14 Attach the Shelf Extension 4 If you are installing the cable management system place 14 caged nuts on the other side of the rack in the v
31. re on the SX6536 series chassis system put an ESD prevention wrist strap on your wrist and connect to the SX6536 series chassis The chassis must be started with a full complement of PSUs that is Thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to different AC lines provides AC failover protection A system should continue to run and allow a hotswap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it may be necessary to reboot the chassis once the defective PSU has been replaced 1 Check all boards power supplies leaf fan modules spine fan modules for proper insertion and seating before connecting the AC power cords Insert all leafs that you plan to use in the chassis Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all mounting screws Connect the power cords to the PSUs ONO Pe a Connect the power cords to grounded electrical outlets Do not power up the chassis with less than all PSUs installed Populate the chassis with the leafs before you power up the chassis A fully loaded SX6536 series switch system can draw 8 kW of power Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or a 3 phase supply 40
32. rm Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platform The manual assumes familiarity with the InfiniBand architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Web Sites Switch Firmware and Firmware Update Tools See http www mellanox com content pages php pg management_tools amp menu_section 34 Note that the Switch System described in this manual is based on Mellanox Technologies SwitchXswitch device MLNX OS Management Software User Manual See http www mellanox com Management Software gt MLNX OS UFM management Software User Manual See http www mellanox com Management Software gt Unfied Fabric Manager Document Conventions Throughout this manual the name SX6536 and the term switch are used to describe the switch unless explicitly indicated otherwise The following pictures are used throughout this document to indicate information that is important to the user Y This symbol makes recommendations to the user 8 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 Rev 1 0 Chassis Installation 1 Chassis Installation This chassis can be installed in standard 19 racks that have depths between 65cm and 80
33. s lt Enter gt If you type no no DHCP then you will be asked whether you wish to use the zeroconf configuration or not If you enter yes yes Zeroconf the session will continue as shown in Table 5 If you enter no no Zeroconf then you need to enter a static IP and the session will continue as shown in Table 6 43 Rev 1 0 InfiniBand Fabric Initialization and Management Table 4 Configuration Wizard Session IP Configuration by DHCP Sheet 2 of 2 Wizard Session Display Example Comments Step 3 Admin password Press lt Enter gt to leave unchanged lt new_password gt Step 4 Confirm admin password lt new_password gt To avoid illegal access to the machine please type a password and then press lt Enter gt Then confirm the password by re entering it Note that password characters are not printed You have entered the following information 1 Hostname lt switch name gt 2 Use DHCP on eth0 interface yes 3 Admin password Enter to leave unchanged CHANGED To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice lt Enter gt Configuration changes saved To return to the wizard from the CLI enter the con figuration jump start command from configuration mode Launching CLI lt switch name gt gt The wizard displays a summary of your choices and the
34. the second lock down bar Figure 22 Bolt and Washer Order fama 20 Remove the shelf extension and save for removal of the chassis 21 Connect a valid ground See Section 3 Ground Connections instructions 22 Go to Section 6 Chassis Power Up on page 40 to power up the switch system LJ on page 35 for detailed 1 7 Installing the Leafs The leaf boards can be installed in any of the available leaf slots Figure 23 Ejector Latch Leaf J Leaf _ Push the latch HS release e Ejector Start with the latch fully open that is perpendicular to the front panel of the leaf 2 Carefully set the leaf board into the side guide rails and slowly slide the board into the chas sis 3 Catch the hooks onto the vertical bar and push the latch shut 30 Mellanox Technologies 648 Port InfiniBand FDR Switch Platform User Manual Rev 1 0 1 8 Inserting a Management Module Start with the latches fully open that is perpendicular to the front panel of the leaf 2 Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated Figure 24 Ejector Latch Pull the module out of the chassis Management Module Spine Management Module Spine _ Push the latch LH release e
35. tight fit 1 6 3 Installation Procedure VY Installing the chassis will be a lot easier if you can remove the sides of the rack 4 You will need a fork lift to move and insert the chassis into the rack 1 6 3 1 Requirements You will need e 2 phillips screwdriver e 3 phillips screwdriver e a fork lift e a grounding lug e measuring tape e ground wire of sufficient length and gauge to properly ground the chassis YY The installation will be much easier with a power screwdriver v It is recommended to use AWG6 or 4mm diameter wire for grounding purposes dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure v4 It is recommended to have at least two people for the duration of the installation proce the safety and wellbeing of all of the people involved in the installation e Parts included in the installation kit 1 shelf with brackets pre installed 1 shelf extension 1 left side upper bracket offset 1 right side upper bracket offset 2 lock down bars 2 upper bracket washers 8 lock washers for the upper brackets to verti 12 lock washers for the upper brackets to cal support upper bracket offsets 12 hex head bolts 8 for the upper 34 caged nuts 20 for the shelf and 14 for the brackets and 4 for the shelf extension chassis faceplate 34 pan head screws for the caged nuts 20 for 20 flat washers 8 for the lock down bars the shelf and 14 for the chassis faceplate and 1
Download Pdf Manuals
Related Search