324 Port InfiniBand FDR Switch Platform Hardware
Contents
1. Mellanox Technologies Ltd Beit Mellanox PO Box 586 Yokneam 20692 Israel Tel 972 0 4 909 7200 972 0 74 723 7200 Fax 972 0 4 959 3245 Tech Support 972 4 9097200 ext 250 8 00AM to 5 00 PM Israel Time www mellanox com 54 Mellanox Technologies J2. Enter to leave unchanged new password Step 6 Confirm admin password new password To avoid illegal access to the machine please type a password and then press Enter Then confirm the password by re entering it Note that password characters are not printed You have entered the following information 1 Hostname switch name 2 Use DHCP on mgmt0 interface yes 3 Enable IPv6 yes 4 Enable IPv6 autoconfig SLAAC on mgmt0 interface no 5 Admin password Enter to leave unchanged CHANGED To change an answer enter the step number to return to Otherwise hit enter to save changes and exit Choice Enter Configuration changes saved To return to the wizard from the CLI enter the con figuration jump start command from configuration mode Launching CLI lt switch name gt gt The wizard displays a summary of your choices and then asks you to confirm the choices or to re edit them Either press lt Enter gt to save changes and exit or enter the configuration step number that you wish to return to Note To run the command configuration jump start you must be in Config mode ellanox Technologies 43 oC Rev 1 0 InfiniBand Fabric Initialization and Management Table 5 Configuration Wizard Session IP Zeroconf Configuration Wizard Session Display IP Zeroconf Configuration Example Mellanox configuration wizard Do yo
3. Package Contents The chassis package includes e 1 chassis with the following modules installed 4 fans 16 PSUs e 1 installation kit 1 box containing 6 power cords 250v 15a 2 0M C14 to C13 USA UL Standard cable management kit 1 Installation Guide Before you install your new SX6518 chassis unpack the system and check to make sure that all the parts have been sent check this against the parts list see Installation and Cable Management Kit Parts on page 14 Check the container for visible damages that may have occurred during shipping The leafs spines and management modules are shipped separately If anything is damaged or missing contact your customer representative immediately The rack mounting holes conform to the EIA 310 standard for 19 inch racks Guaran tee proper ventilation by leaving 8cm 3 of space to the front and rear of the switch This will ensure proper air flow through the chassis This is crucial for maintaining good airflow at ambient temperature In particular route cables such that they do not impede the air into or out of the chassis Revi 0 Chassis Installation 1 3 Leaf Package Contents The leafs are ordered by the customer and are shipped 4 to a box The customer will receive as many boxes as needed to fill the order 1 4 Spine Package Contents The spines are shipped 3 to a box The customer will receive as many boxes as needed to complete a full complement
4. Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 2 Installing the Cable Holder v Now is the time to install the cable holder Figure 29 Cable Holder Leftside rack Top side 10X M6 caged nut 10 X M6 bolt for caged nut Step 1 Place the Cable holder next to the rack on the connector side of the chassis and identify the holes where the caged nuts were placed The cable holders should go to the outside of the vertical supports Step 2 Set the bottom of the cable holder at the level of the shelf Step 3 Screw the cable holder onto the rack using the screws provided Step 4 Repeat steps 1 3 for the second cable holder 33 Rev 1 0 Ground Connections 3 Ground Connections Make sure to connect the ground post to a valid electrical ground Use a grounding lug and a ground wire of sufficient capacity to safely convey a potential discharge A ground wire of AWG 6 or 4mm diameter is recommended for grounding this device The chassis is concurrently grounded through each of the PSUs Only connect the PSU cords to properly grounded outlets Do not rely on the PSU grounds It is absolutely necessary to connect the grounding post Make sure the con nections are solid and permanent If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Figu
5. Status Indications FLEA pag STATUS PSU SPINE LEAF MNG 1 MNG 2 STATUS FANS FANS STATUS STATUS Step 5 Check the Spine LEDs and make sure they coincide with Figure 36 Figure 36 Management Module Status Indications for Normal Operation CONSOLE MGT M RST PSU S FANS LFANS MASTER ETT be eee EE e a Qe poo anita g 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 7 InfiniBand Fabric Initialization and Management Subnet Management and Subnet Administration are standard components defined by the InfiniBand Specification which implement fabric initialization discovery and configura tion The switch s switching fabric can be managed by MLNX OS as well as any third party Subnet Management software running on an InfiniBand Host that is connected to any switch port The switch is also compatible with the Open Source Subnet Manager Ls OpenSM The chassis is basically an InfiniBand fabric built out of individual InfiniBand switch devices and appears so to external Subnet Managers The chassis supports the System Image GUID feature of the InfiniBand specification which means that software manage ment tools that support this feature can recognize all the internal devices as a single switch system 7 1 Configuring the Switch Management Modules for the First Time On SX65xx systems when having dual management systems first connect the cable and configure the master management modu
6. and Manual managing Mellanox Technologies SwitchX Switch Platforms MLNX OS Software Command Reference Guide Command Reference Guide for MLNX OS listing all of the commands available through MLNX OS with explanations and examples MLNX OS Software Configuration Guide Configuration Guide for MLNX OS displaying different config uration scenarios Document Conventions Throughout this manual the name X6518 and the terms chassis and switch are used to describe the switch unless explicitly indicated otherwise The following pictures are used throughout this document to indicate information that is important to the user Y This symbol makes recommendations to the user 8 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Rev 1 0 Chassis Installation 1 Chassis Installation TThis chassis can be installed in standard 19 racks that have depths between 65cm 25 6in and 80cm 31 5in between the vertical supports of the rack 5 Installation and initialization of the chassis is a simple process requiring attention to the normal mechanical power and thermal precautions for rack mounted equipment Your chassis comes only with the power supplies and fans pre installed The rest of the openings are populated with blanks All of the leafs spines and management modules come shipped in separate packages The chassis requires initial configu
7. and powering them up when the management module is removed Step 3 Make sure the S Fans and L Fans LEDs are green Step 4 Make sure that the spine and the leafs both have the same version of FW Step 5 Reburn the FW and remove and reinstall the management module Step 6 If you are running the chassis with only one management module remove and reinstall the management module Make sure the mating connectors of the unit are free of any dirt and or obstacles See Section 5 5 on page 73 Step 7 If you are running the chassis with only one management module replace the management module 10 3 2 Yellow LED for the Leaf Fan on the Management Module is Lit Step 1 Check the MLNX OS management for confirmation and possible explanation of the alert Step 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes Step 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Step 4 Determine which fan module is problematic by checking the status LED on each fan module Step 5 Remove and reinstall the problematic fan unit Make sure the mating connec tor of the new unit is free of any dirt and or obstacles See Section 5 4 on page 70 Step 6 Replace the Leaf fan module There are two non interchangeable
8. by U Numbers These are for the cable holder Place 5 caged nuts here pine side Leaf Side Step 7 Place the chassis in front of the rack and as close as possible to the rack Rev 1 0 Chassis Installation Figure 17 Fork Lift O O bed Ww Container floor Hinged extension Step 8 Raise the chassis so that it is 5cm above the shelf in the rack Step 9 Raise the hinged extension on the container floor Step 10 Lower the chassis until the hinged extension is sitting on the shelf extension Step 11 Push the chassis from the container to the shelf 4 Save the container pallet to remove this chassis It will be necessary for the removal and disposal procedures Figure 18 Face Plate Mounting Bolt Locations e e o e e 9 LEM 8 M SX6518 Holes in the faceplate for mount lanox ing on the rack The holes are separated by 4U units 26 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Step 12 Slide the chassis onto the shelf and determine the location of the upper brack ets on the vertical supports on the connector side of the chassis Do not remove the fork lift with the pallet yet Step 13 Confirm that the caged nuts for the faceplate are in the correct location Step 14 Slide the chassis back out far enough to install the upp
9. command line interface or with a WebUI interface Chassis management comes with the switch sys tem 8 1 Starting an SSH Connection to the Switch SSH connection also supports TelNet A Step 1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connec tor Step 2 Connect to the remote machine Step 3 Start a remote shell to the switch using the following command ssh l admin your ip address gt Step 4 When prompted for a password enter admin Note The IP address used above is the same IP address that was assigned to the Mella nox configuration wizard in the Configuring the Switch Management Modules for the First Time section Step 5 You can enter any supported command now For a complete reference of commands please see Mellanox MLNX OS Management Software User s Manual 8 2 Starting a WebUI Connection to the Switch Step 1 Set up an Ethernet connection between the switch and a local network machine the remote machine henceforth using a standard RJ 45 connec tor Step 2 Start a Web browse Google Chrome Internet Explorer 7 0 or Mozilla Firefox 3 0 Make sure the screen resolution is set to 1024 768 or higher aa Rev 1 0 MLNX OS Management Software Step 3 Enter the following URL http switch IP address where switch IP address is the IP address of the switch or its DN
10. of spines 1 5 Management Package Contents The package includes all of the management modules ordered by the customer e 1 RJ45 to DB9 harness for each management module received 1 6 Physical Installation Warning This equipment is very heavy Safety is the first concern Make sure that proper manpower and equipment are used for transporting and moving the chassis 4 The fully loaded chassis weighs 212 36 kg 468 LBS full configuration 90 kg 199 LBS empty configuration 112 kg 247LBS As shipped configuration The switch platform uses 17U of rack space in a standard 19 rack 16U for the chassis and 1U for the shelf The switch ships from the factory with mounting holes on the spine side only There are upper brackets to connect the leaf side to the rack near the top of the chassis and there are two lock down bars to secure the chassis to the shelf The weight of the switch is supported from underneath the unit by the shelf This chassis can be installed in standard 19 racks that have between 65cm 25 6in and 80cm 31 5in between the vertical supports of the rack Make sure that a fully populated rack including cables will have sufficient air flow for cooling Choose a rack which is able to support the mechanical and environmental characteris tics of a fully populated switch chassis gt 1 6 1 Starting with the Rack The rack may need to be modified to accommodate the X6518 chassis The distance betwee
11. the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes Step 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Step 4 Remove and reinstall the spine board Make sure the mating connectors of the unit is free of any dirt and or obstacles See Section 5 3 on page 68 Step 5 Make sure that the spine and the Leafs both have the same version of FW Step 6 Reburn the FW and remove and reinstall the spine Step 7 Replace the spine board 52 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 10 5 MLNX OS Software For a instructions concerning software updates see the Mellanox MLNX OS SwitchX Software WebUI User s Manual or the Mellanox MLNX OS SwitchX Software User Manual For more information and documentation on the SX6518 Series go to the Mellanox Website http www mellanox com Rev 1 0 Mellanox TECHNOLOGIES USA Mellanox Technologies Inc 350 Oakmead Parkway Suite 1 Sunnyvale CA 94085 Tel 408 970 3400 Fax 408 970 3403 www mellanox com E mail info mellanox com Map and Directions Tech Support 408 916 0055 8 00AM to 5 00 PM Pacfic 324 Port InfiniBand FDR Switch Platform Installation Guide A cc000382
12. Mellanox TECHNOLOGIES 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide PN MSX6518 6R MSX6518 NR Rev 1 0 www mellanox com Rev 1 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILI
13. PSUs Step 2 Plug the other end of the power cord into grounded outlets Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit With N 1 PSU redundancy the chassis must be started with a full compliment of possible PSUS thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUS to dif ferent AC lines provides AC fail over protection N N will allow for either of the following 2 scenarios Connecting 3 power supplies to a different power supply grid will create N N grid redundancy this is High Availability Under these conditions should a power supply fail an electric company power failure or blackout for example power grid High Availabil ity will continue to keep the chassis running at full capacity through a secondary or backup power supply grid or generator or battery backup system The chassis can also be configured for N 1 times 3 This is all 6 PSUs are supplied from a single power source usually the commercial electric grid This enables 3 power supply redundancy With N N optional PSU grid redundancy the chassis can run on 1 2 of the full comple ment of PSUs N N allows the chassis to run on 3 PSUs supplied from one power grid while 3 are connected to a second power grid Mellanox Technologies 35 ee Rev 1 0 Power Co
14. S name Step 4 You will receive the login window for remote management of the switch Figure 38 Web UI Login Page Mellanox TECHNOLOGIES e Ports E Status titi IB SN am a 1 MGMT System Security EXE Please enter your username and password then click Login Password Account Step 5 Use admin for both the login Account and the password 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 9 Resetting the Switch RST The Reset button resets the chassis management module when the button is pushed When the button is held down for 15 seconds the chassis is reset and the password deleted DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset Y button Sharp objects can cause damage use a flat object such as a paper clip Figure 39 Reset Button CONSOLE MGT AA RST PSU S FANS L FANS MASTER hi YU i This button resets the CPU of the management module A quick push of this button per forms this reset When the reset button is pushed on the master management module this management module is reset becoming the slave and the other management module becomes the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down and powering them up when the reset button is pushed When the button is held down for 15 seconds the management module is reset and the password
15. TY OF SUCH DAMAGE ANN Mellanox Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com Tel 972 0 4 909 7200 Tel 408 970 3400 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2012 Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide 2 Mellanox Technologies Document Number 3805 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Table of Contents Chapter 1 Chassis Installation 10 1 4 Environmental and Safety Recommendations 10 1 2 Chassis Package Contents 11 1 3 Leaf Package Contents 12 1 4 Spine Package Contents 12 1 5 Management Package Contents 12 1 6 PhysicalInstallation 12 1 6 1 Starting with the Rack 12 1 6 2 ESD ElectroStatic Discharge Connection 13 1 6 3 Installation Procedure 13 1 7 Installing the Leafs 30 1 8 Inserting a Management Module 31 1 9 Inserting a Spine Board 31 Chapter 2 Installing the Cable Holder 33 Chapter 3 Groun
16. assis can roll off A cause grave bodily harm should it fall on or near you Step 1 Put the container on the fork lift Figure 12 Putting the Chassis in the Rack The chassis is screwed down to the pallet on both the spine side and the leaf side l 22 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Figure 13 Place the Upper Bracket Offset Step 2 Screw on the two Upper Bracket offsets onto the chassis using nine flat head screws for each offset The offsets are right and left handed Place the offsets so that the flat head screws go into the bevelled side of the offset Figure 14 Attach the Shelf Extension Step 3 Screw on the shelf extension using two bolts with flat washers Step 4 If you are installing the cable management system place 10 caged nuts on the other side of the rack in the vertical supports Mellanox Technologies 23 Rev 1 0 Chassis Installation Figure 15 Location for the Lock down Bar L L og i ka D l Ji Bottom of the e ce 9 chassis Step 5 Unscrew the lock down bars Step 6 Place caged nuts in all of the locations specified in Figure 16 The black holes show where to place caged nuts the red holes show the location of the upper brackets on the leaf side 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Figure 16 Caged Nut Placement
17. bar and push the latch shut 1 8 Inserting a Management Module Step 1 Start with the latches fully open that is 45 to the front panel of the leaf Step 2 Carefully seat the management module into the slot and slowly slide the board into the chassis until the ejectors begin to engage on the chassis edge Step 3 Press the ejectors inward until the locks snap This indicates that the board is fully seated Figure 26 Ejector Latch Management Module Spine Management Module Spine uf Push the latch release 1 9 Inserting a Spine Board Figure 27 Spine Board Insertion Open the Ejector Latches QOOUQUO00U EBENEN Ooooooooo Tr A OOoooooo00 sem OOOO000o00 Step 1 Start with the latches fully open that is 45 to the front panel of the leaf Step 2 Hold the spine board by the sides with both latches fully open Step 3 Insert the board into its rail The board should slide in without resistance Step 4 Push the board in until the hooks on the latches are touching the vertical bar Step 5 Now press both latches inward simultaneously You should feel the latches are hooked into the front panel Step 6 Press both latches applying some force to close them completely The locking latch will engage Mellanox Technologies 31 HMM MP Rev 1 0 Chassis Installation Figure 28 Ejector Latch Push the latch release 32 Mellanox Technologies 324
18. contents But be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped Figure 3 Shock and Tilt Stickers 9 ELA Stickers unaffected by shipping Stickers showing excessive shock or tilt 1 6 3 4 Opening the Container Step 1 Check the shock and tilt stickers on the container to see if it was subjected to excessive shock or tilting If any of these stickers are tripped contact your Mellanox representa tive Step 2 Before starting the procedure put the ESD strap on and connect it to a valid ground 16 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Figure 4 Container Screws ZA GS S Y A O i Container screws Step 3 Remove the four sides of the container by unscrewing the phillips head screws holding the sides on Step 4 Remove and put aside the box containing the cables Step 5 Remove and put aside the box containing the cable management kit Step 6 Remove and put aside the box containing the Installation kit Step 7 Visually inspect the chassis make sure that X there is no visible damage X 6 PSUs are installed X all 4 fans are installed Step 8 Remove all protective plastic film from all sides and top of the chassis The leafs spines and management modules are shipped separately 5 4 Do not remove any of the blocks or either of t
19. d Connections 34 Chapter 4 Power Connections 35 4 1 Powering Up the Switch Platform 35 Chapter 5 InfiniBand QSFP Cable Installation 37 5 1 Port Connector Interfaces 37 5 2 Supported Approved Cables 38 5 3 Cable Power Classes 38 Chapter 6 Chassis Power Up 39 6 1 Power Supply Management Board and Spine Board Indicator Status at Power ON 40 Chapter 7 InfiniBand Fabric Initialization and Management 41 7 Configuring the Switch Management Modules for the First Time 41 7 2 Rerunning the Wizard 46 Chapter 8 MLNX OS Management Software 47 8 1 Starting an SSH Connection to the Switch 47 8 2 Starting a WebUI Connection to the Switch 47 Chapter 9 Resetting the Switch RST 49 Chapter 10 TroubleShooting 50 10 1 Power Supply Unit 50 10 2 Leaf Board 50 10 3 Management Module 51 10 3 1 Yellow Status LED for the Chassis on the Management Module is Lit 51 10 3 2 Yellow LED for the Leaf Fan on the Management Module is Lit 51 10 3 3 Yellow LED for the Spine Fan on the Management Module is Lit 52 10 4 Spine Board 52 10 5 MLNX OS Software 53 Rev 1 0 List of Figures Figure 1 Distance Between the Vertical Supports 13 Figure 2 Installation Kit Parts 15 Figure3 Shock and Tilt Stickers 16 Figure 4 Container Screws 17 Figure 5 Vertical Rack Supports Dimension 18 Figure 6 Adjustable Shelf Brackets 19 Figure 7 Top of Shelf Bracket 19 Figure 8 Placing the Shelf Bracket 20 Figure9 Caged Nut Locations for the Shelf Brackets 20 Figu
20. e Rev 1 0 Step 3 Measure the distance between the vertical supports and adjust the shelf brack ets according to this distance This chassis can be installed in a rack with a distance between the vertical supports of from 65cm 25 6in to 80cm 31 5in Step 4 The shelf comes with the four brackets pre installed Two of the brackets should be loose They may need to be removed and repositioned to meet the distance between the vertical supports of the rack Figure 6 Adjustable Shelf Brackets These two brackets will need to be adjusted They may even need to be removed and repositioned The holes in the bracket are offset to one side The holes must be closer to the top of the shelf Figure 7 Top of Shelf Bracket The holes in the bracket are offset to one side The holes must be closer to the top of the shelf op of shelf Bottom of shelf is even with this side of the bracket the greater distance between the holes and the edge of the bracket show that this is the bottom side Mellanox Technologies 19 UU Rev 1 0 Chassis Installation Figure 8 Placing the Shelf Bracket Adjusting slots Step 5 Insert three caged nuts into the rack for each of the four shelf brackets the 2 brackets that are adjustable and the 2 that are fixed Make sure that the caged nuts are all at the same height on the vertical supports Step 6 Put the Shelf into the rack and twist the shelf into place Make su
21. e of an IP address To check the current ethO configuration enter the following com mands Sx 43 standalone master enable sx 43 standalone master show interfaces mgmt0 Mellanox Technologies 45 e E E e Rev 1 0 InfiniBand Fabric Initialization and Management The following is an example of the output Interface mgmtO0 state Admin up yes Link up yes IP address 192 168 10 43 Netmask 255 255 255 0 Speed 1000Mb s auto Duplex full auto Interface type ethernet Interface source physical MTU 1500 HW address 00 02 C9 11 2A AE Comment RX bytes 1343502058 TX bytes 313920869 RX packets 17589211 TX packets 992717 RX mcast packets 0 TX discards 0 RX discards 0 TX errors 0 RX errors 0 TX overruns 0 RX overruns 0 TX carrier 0 RX frame 0 TX collisions 0 TX queue len 1000 Step 6 On SX65xx systems configure the Box IP BIP centralized location for chassis High Availability according to the steps in the MLNX OS Software UM section Box IP Centralized Location 7 2 Rerunning the Wizard If you want to rerun the wizard run the following commands Sx 43 standalone master enable sx 43 standalone master configure terminal Sx 43 standalone master configuration jump start 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 8 MLNX OS Management Software MLNX OS is a software based management system that can be run with either a
22. eaf board Step 4 If uninstalling and reinstalling the Leaf board does not work burn the latest FW on the Leaf board and uninstall and reinstall the Leaf board Step 5 Replace the Leaf board with a new one Should any of the leaf boards shut down due to over temperature it may be necessary to reboot the chassis gt The Physical link LED for the InfiniBand connector does not come on Step 1 Check that both ends of the cable are connected Step 2 Check that the locks on the ends are secured Step 3 Make sure that the latest FW version is installed on both the HCA card and the switch Step 4 If media adapters are used check that the all connections are good tight and secure Step 5 Replace the cable The Activity indication does not come on 50 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Check that the Subnet Manager has been started 10 3 Management Module 10 3 1 Yellow Status LED for the Chassis on the Management Module is Lit Step 1 Check the MLNX OS management for confirmation and possible explanation of the alert Step 2 Reset the master management module by pushing the rest button If you have two management modules installed this will convert the master management module to the slave and convert the slave to the master If there is only one management module in the chassis all of the leafs and ports are reset by bringing them down
23. en them yet You may have to wiggle the chassis to catch the screws in the threaded holes You will need the 10 hex head screws and washers for each lock down bar Step 22 Catch the screws for the second lock down bar but do not tighten them yet You may have to wiggle the chassis to catch the screws in the threaded holes Mellanox Technologies 29 CMM MN Rev 1 0 Chassis Installation Figure 24 Bolt and Washer Order Step 23 Remove the shelf extension and save for removal of the chassis Step 24 Connect a valid ground See Section 3 Ground Connections on page 34 for detailed instructions Step 25 Tighten the faceplate screws See Figure 18 Step 26 Tighten the upper bracket to upper bracket offset screws See Figure 21 Step 27 Tighten the lock down bars See Figure 23 Step 28 Go to Section 6 Chassis Power Up on page 39 to power up the switch system 1 7 Installing the Leafs The leaf boards can be installed in any of the available leaf slots Figure 25 Ejector Latch Leaf Leaf Push the latch release p gt Step 1 Start with the latch fully open that is 45 to the front panel of the leaf 30 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Step 2 Carefully set the leaf board into the side guide rails and slowly slide the board into the chassis Step 3 Catch the hooks onto the vertical
24. er brackets Figure 19 Upper Bracket Installation This hole is 60cm up from the top of the shelf Rev 1 0 Chassis Installation Figure 20 Upper Brackets and Offset Location d AN NS a a eeeeeeoeeceecoeoeeeoeeeoeeoe Step 15 Screw the upper brackets onto the vertical supports and tighten in place You will need the socket wrench with both extensions to tighten these bolts Step 16 Slide the chassis onto the shelf until the face plate of the chassis touches the vertical rack support Step 17 Make sure the upper brackets are installed in their correct location Step 18 Screw the 8 bolts through the faceplate into the caged nuts DO NOT TIGHTEN Figure 21 Offset to Upper bracket Bolts Step 19 Now you can remove the fork lift and pallet l 28 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Step 20 For each offset and upper bracket screw six bolts through the offset into the upper bracket Use one flat washer and one lock washer for each of these bolts Figure 22 Location for the Lock down Bar Bottom of the chassis These holes are to screw the bar into the chassis These holes are to screw the bar into the shelf Step 21 Place the front lock down bar at the edge of the chassis and align the holes in the lock down bar with the holes in the shelf Catch all of the screws but do not tight
25. g before connecting the AC power cords Boards Power supplies 1 eaf fan modules Spine fan modules Populate the chassis with the leafs before you power up the chassis Insert all leafs that you plan to use in the chassis Insert all spines in the chassis Insert thermal blanks in unused leaf slots to maintain balanced air flow Tighten all leaf and spine mounting screws Connect the power cords to the PSUs SO Uh dm cO 9 Connect the power cords to grounded electrical outlets With N 1 1000 W PSU redundancy do not power up the chassis with less than all PSUs installed Mellanox Technologies 39 sa MEM Rev 1 0 Chassis Power Up 6 1 Power Supply Management Board and Spine Board Indicator Status at Power ON It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes As the power is turned on you should observe the following conditions for normal operation Step 1 Power Supply Unit s AC OK and DC OK indicators are ON and FAIL indicators are OFF Step 2 There is a green Status LED per spine board per leaf board and per management module that indicates power supplies are good Step 3 Spine Board indicators will display status of internal links to the installed leaf boards All PHY links to existing leaf Boards should be ON Step 4 Check the Spine LEDs and make sure they coincide with Figure 35 Figure 35 Spine Side Panel Display
26. gs gt 55 kgs You will need to have the following tools and equipment e 2 phillips screwdriver a fork lift e a grounding lug measuring tape Vv Vv Vv chassis 3 phillips screwdriver The installation will be much easier with a power screwdriver Installation and Cable Management Kit Parts e Parts included in the installation kit 1 shelf with brackets pre installed 1 left side upper bracket offset 2 lock down bars 8 lock washers for the upper brackets to vertical support hex head bolts 8 for the upper brackets and for the shelf extension 1 shelf extension 1 right side upper bracket offset 2 upper bracket washers ground wire of sufficient length and gauge to properly ground the It is recommended to use AWG6 or 4mm diameter wire for grounding purposes It is recommended to have at least two people for the duration of the installation proce dure Use a mechanical lift to raise this chassis If not use enough manpower to ensure the safety and wellbeing of all of the people involved in the installation 12 lock washers for the upper brackets to upper bracket offsets 20 caged nuts 12 for the shelf and 8 for the chassis faceplate M ellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 20 pan head screws for the caged nuts 12 for the shelf and 8 for the chassis faceplate 32 lock washers 20 for the lock down bars and 12 fo
27. guration Wizard Session IP Configuration by DHCP 42 Table 5 Configuration Wizard Session IP Zeroconf Configuration 44 Table 6 Configuration Wizard Session Static IP Configuration 45 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Revision History Table 1 Revision History of this Installation Guide Revision Date Details 1 0 March 2012 Initial release Rev 1 0 About this Manual This manual provides installation and set up instructions for the SX6518 QSFP Chassis InfiniBand Switch Platform Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platform The manual assumes familiarity with the InfiniBand architecture specification Related Documentation The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the following Table 2 Reference Documents and Web Sites InfiniBand Architecture Specification Vol 1 The InfiniBand Architecture Specification that is provided by Release 1 2 1 IBTA Switch Hardware Errata For any possible errata due to hardware issues see the switch support product page This requires a customer support login Mellanox MLNX OS SwitchX Software WebUI WebUI Overview for MLNX OS software User s Manual Mellanox MLNX OS SwitchX Software User This document contains information regarding configuring
28. he lock down bars at this time 1 6 3 5 Installing the Shelf Step 1 Select the location in the rack Try to keep the chassis as low as possible to keep the center of gravity low Mellanox Technologies 17 se MN Rev 1 0 Chassis Installation Y It is recommended to set the top of the shelf at U5 Try and keep the chassis as low as possible The maximum height that the chassis can be mounted will be determined by the maximum height that the fork lift can raise the pallet Step 2 Open the Installation kit and make sure that all of the parts are in the box See Figure 2 Installation Kit Parts Installing the chassis in the rack with the spine side as close as possible to the door leaves maxi mum room for the cable management system 31 5 If the distance in your rack is different move the vertical supports so that one pair is 10 cm 4in away from the door and that the vertical supports are between 65 Az cm 25 6 and 80 cm 31 5 apart The distance between the vertical supports must be between 65 cm 25 6 and 80 cm the rack The side of the chassis with the QSFP connectors will sit at the edge of the The side of the chassis with the spine units will sit flush with the vertical supports of shelf closer to the center of the rack Figure 5 Vertical Rack Supports Dimension Vertical Supports 8 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guid
29. is deleted You will then be able to enter without a password and make a new password for the user admin Mellanox Technologies 49 CMM MNT Rev 1 0 TroubleShooting 10 TroubleShooting 10 1 Power Supply Unit If the Power supplies cannot supply enough power the management module may shut down the leafs until the power being supplied is sufficient to run the operating leafs If the management module does shut down some leafs it may be necessary to reboot the chassis to get the shut down leafs up and running again As each PSU is plugged in make sure that the green power LEDs on the PSU comes on If the AC power LED is off Step 1 Check that the power cable is the correct power cable for your country Step 2 Check that the power cable is plugged into a working outlet Step 3 Check that the power cable has a voltage within the range of 100 240 volts AC Step 4 Remove and reinstall the power cable Step 5 Check the circuit breakers to be sure that the breaker has not tripped Step 6 Check that the power cable is good Replace the power cable Step 7 If the AC power LED is green but the OK power LED is off or the FAIL LED is on Replace the PSU 10 2 Leaf Board The power LED for the Leaf board is off Step 1 Make sure that all of the PSUs are showing DC OK Step 2 Uninstall and reinstall the Leaf board Step 3 When the Yellow LED is on this indicates a fault in the board uninstall and reinstall the L
30. le CPU and only then configure the slave By default the master CPU is the top management module For further information on the master and slave roles see the MLNX OS Software UM section High Availabil ity for more information Initial configuration must be done on all of the management modules Step 1 Connect the host PC RS232 connector to the CONSOLE port of the manage ment module using the supplied cable Figure 37 Connecting the Harness to the Console Connect the host PC to here uU 12C CONSOLE MGT gt RST PSU S FANS LFANS MASTER E W M 4 Wa No remote IP connection is available at this stage 5 Step 2 Configure a serial terminal program for example HyperTerminal Minicom or Tera Term on your host PC with the settings described in Table 3 Mellanox Technologies 41 x HE Rev 1 0 InfiniBand Fabric Initialization and Management Table 3 Serial Terminal Program Configuration Parameter Setting Baud Rate 9600 Data bits 8 Stop bits 1 Parity None Flow Control None Step 3 Login from serial terminal program as admin and use admin as password This starts the Mellanox configuration wizard Step 4 session Go through the Mellanox configuration wizard Table 4 shows an example of a wizard Table 4 Configuration Wizard Session IP Configuration by DHCP Sheet 1 of 2 Wizard Session Display Example Comments Mellanox configuration wizard D
31. must be dealt with immediately Step 4 Check that none of the LEDs on the spines are yellow It can take up to 5 minutes to boot up the system Turn off the system if any LEDs remain red for more than 5 minutes Step 5 Check that the leaf status LEDs in the spines are lit Figure 32 Spine Module ILIIS INNEN 535555556 TETU eren rrr EERE LIRE a Spine Status Fan Status Spine Leaf IB Connection Status l 36 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 5 InfiniBand QSFP Cable Installation The switch uses industry standard QSFP InfiniBand cables which are available from Mel lanox Technologies The standard QSFP cables support full 56 56Gb s FDR 40 40Gb s FDR10 40 40Gb s QDR 20 20Gb s DDR and 10 10Gb s SDR bidi rectional wire speed of the switch ports All InfiniBand QSFP connections are made to the leaf boards Each leaf has 18 InfiniBand QSFP connectors in two rows which are num bered 1 18 If maximum cable lengths are exceeded data transfer will be reduced and the bit error rate will increase FDR BandWidth is supported only by MLNX QSFP cables FDR and FDR10 are only guaranteed to work with approved Mellanox Cables c cu cu All cables can be inserted or removed with the unit powered on To insert a cable press the connector onto the port receptacle until the connector is firmly seated The LED indicator corres
32. n the vertical supports must be between 65 and 80 cm 12 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide M 1 6 2 1 6 3 Figure 1 Distance Between the Vertical Supports Vertical Supports This edge of the shelf must be as close as possible to the inside of the door when closed The vertical supports may have to be moved so that the vertical support closest to the door is 6 cm from the inside of the door This may be required so that the shelf does not pro trude from the rack If you move the vertical supports make sure that you reinstall them vertically plumb 5 ESD ElectroStatic Discharge Connection Before starting any procedure on the X6518 switch system do the following Step 1 Put an ESD prevention wrist strap on your wrist and make sure there is good contact between your body and the strap Step 2 Plug the other end of the wrist strap to a valid ground Make sure that this is a tight fit Installation Procedure Y Installing the chassis will be a lot easier if you can remove the sides of the rack ellanox Technologies Rev 1 0 13 Rev 1 0 Chassis Installation 1 6 3 1 1 6 3 2 La You will need a fork lift to move and insert the chassis into the rack Use enough people to safely lift this product 40 Ibs 40 70 Ibs 18 kgs 18 32 kgs Requirements Lir 70 121 Ibs gt 121 Ibs 32 55 k
33. nitial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on mgmt0 interface yes n Step 3 Use zeroconf on mgmt0 interface no Step 4 Primary IP address for example 192 168 10 4 Mask length may not be zero if address is not zero interface ethO Step 5 Netmask 0 0 0 0 255 255 255 0 Step 6 Default gateway for example 192 168 10 1 Step 7 Primary DNS server Step 8 Domain name Step 9 Enable IPv6 yes yes Step 10 Enable IPv6 autoconfig SLAAC on mgmt0 interface no no Step 11 Admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on mgmt interface no Use zeroconf on mgmt0 interface no Primary IP address 192 168 10 4 Netmask 255 255 255 0 Default gateway 192 168 10 1 Primary DNS server Domain name 9 Enable IPv6 yes 10 Enable IPv6 autoconfig SLAAC on mgmt0 interface no 11 Admin password Enter to leave unchanged unchanged coo 1o g tn AUNE To change an answer enter the step number to return to Otherwise hit enter to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 gt Step 5 Before attempting a remote for example SSH connection to the switch check the ethO interface configuration Specifically verify the existenc
34. nnections With power grid A charged with current and power grid B not charged there is only grid redundancy and not PSU redundancy The power system will divide the current consumption by the number of working PSUs Should one of the PSUs fail the total current consumption will then be divided by the remaining working PSUs When the failed PSU is hot swapped the new PSU will ramp up and pass its share of cur rent so that the total current is always divided by the number of working PSUs Figure 31 Multiple Power Inlets Electric Caution Notification CAUTION ACHTUNG ATTENTION Risk of electric shock and energy Gafahr des elektrischen Schocks Risque de choc et de danger hazard Entferrnen des Netzsteckers elnes e lectriques Le de branchment The PSUs are all independent Netzteils spannungsfrei Um alle d une seule alimentation stabilise e Einhieten spannungsfrei zu machen ne de branch uniquement qu un Disconnect all power supplies to sind die Netzstecker aller Netzteile module Alimentation Stabilise e ensure a powered down state inside zu entfernen Pour isoler completement le module of the switch platform en cause Il faut de brancher toutes les alimentations stabilise es Step 3 Check the Status LEDs on all of the management modules and confirm that all of the LEDs show status lights consistent with normal operation Warning Any yellow or red status LEDs on any of the management modules is cause for concern and
35. o you want to use the wizard for initial configura tion yes You must perform this configuration the first time you operate the switch or after resetting the switch Type y and then press Enter Step1 Hostname switch 1 If you wish to accept the default hostname then press lt Enter gt Otherwise type a different hostname and press lt Enter gt Step 2 Use DHCP on mgmt0 interface no Perform this step to obtain an IP address for the switch ethO is the management port of the switch If you wish the DHCP server to assign the IP address type yes and press Enter If you type no no DHCP then you will be asked whether you wish to use the zeroconf configuration or not If you enter yes yes Zeroconf the session will continue as shown in Table 5 If you enter no no Zeroconf then you need to enter a static IP and the session will continue as shown in Table 6 Step 3 Enable IPv6 yes yes The management interface will be able to use IPv6 addresses Step 4 Enable IPv6 autoconfig SLAAC on mgmt0 interface no no This turns on autoconfiguration of the IPv6 addresses This is unsuitable for DHCPv6 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Table 4 Configuration Wizard Session IP Configuration by DHCP Sheet 2 of 2 Wizard Session Display Example Comments Step 5 Admin password Press
36. ponding to each data port will light orange when the physical connection is estab lished that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port When a logical connection is made the LED will change to green When data is being transferred the light will blink green To remove disengage the lock and slowly pull the connector away from the port receptacle Warning Take care to not impede the air flow through the ventilation holes next to the InfiniBand ports Use cable lengths which allow routing horizontally around to the side of the chassis before bending upward or downward in the rack 5 1 Port Connector Interfaces The connector side of the switch has 18 leaf boards and each leaf board has 18 QSFP ports The ports on each leaf board are placed in two rows 9 ports to a row The ports are labelled as shown in Figure 33 The bottom row ports are flipped from the top row See Figure 34 Rev 1 0 InfiniBand QSFP Cable Installation Figure 33 Port Numbering 1 3 7 9 AD 18 15 1y 5 M Se Pied Bis See 2 4 6 8 10 12 14 16 18 Figure 34 Top and Bottom Ports ii m 5 2 Supported Approved Cables For a list of approved cables for this switch see the Mellanox approved cable list www mellanox com related docs user_manuals Mellanox_approved_cables pdf 5 3 Cable Power Classes Chassis and swi
37. r the upper brackets to upper bracket offsets 2 upper brackets 1 open end ratchet wrench 10mm 1 socket wrench 2 extensions for the socket driver flat washers for the shelf extension 20 flat washers 8 for the lock down bars and 12 for the upper brackets 32 hex head bolts 20 for the lock down bars and 12 for the upper brackets to upper bracket offsets 12 flat head screws with locktite for the upper bracket offsets 1 socket nut driver 8mm 1 socket nut driver 10mm 1 socket nut driver 13mm 1 box wrench e Parts included in the cable management installation kit 1 cable management rack RH 10 caged nuts M6 1 cable management rack LH 10 M6 bolts Figure 2 Installation Kit Parts Lock down bar Shelf Shelf extension Upper bracket Upper bracket washer Upper bracket offset 0 sof 4 al Phillips pan head hex head bolt screw Flat washers Lock washers Caged nuts Flat head screw Mellanox Technologies 15 MM M Rev 1 0 Chassis Installation 1 6 3 3 Container Mishandling The container has shock and tilt stickers applied These will turn red if the container has been mis handled or roughly handled Upon receipt of the container look for and inspect the shock and tilt stickers to confirm that they have not tripped If one or more are red notify the shipper and Mella nox This on its own does not indicate damage to the
38. ration to get the chassis and Fabric management up and running through remote management see Configuring the Switch Management Modules for the First Time on page 41 This unit is intended for installation in a Restricted Access Location A restricted access area can be accessed only through the use of a special tool lock and key or other means of security 1 1 Environmental and Safety Recommendations The following are Mellanox recommendations Recommended ambient temperature in the System room is 20 5 C Recommended humidity ranges is 40 15 without condensing Itis highly recommended that the installation sites be as isolated as possible from all sources of radio transmissions and electrical interference It is highly recommended that the installation site building be equipped with a light ning rod It is highly recommended that the installation site be equipped with smoke detectors and a fire alarm warning system SSS C3 10 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 The system requires a KVA rated UPS system It is recommended that a UPS system be installed to protect the equipment in the event of unexpected power failure Make sure that the outlets and circuits will not be overloaded Spread out the load over at least two or three circuits or use a 3 phase circuit N N grid redundancy requires 2 independent power supply sources 1 2 Chassis
39. re 10 Shelf Brackets 21 Figure 11 U Numbering Above Shelf 21 Figure 12 Putting the Chassis in the Rack 22 Figure 13 Place the Upper Bracket Offset 23 Figure 14 Attach the Shelf Extension 23 Figure 15 Location for the Lock down Bar 24 Figure 16 Caged Nut Placement by U Numbers 25 Figure 17 Fork Lift 26 Figure 18 Face Plate Mounting Bolt Locations 26 Figure 19 Upper Bracket Installation 27 Figure 20 Upper Brackets and Offset Location 28 Figure 21 Offset to Upper bracket Bolts 28 Figure 22 Location for the Lock down Bar 29 Figure 23 Lock Down Bar 29 Figure 24 Bolt and Washer Order 30 Figure 25 Ejector Latch 30 Figure 26 Ejector Latch 31 Figure 27 Spine Board Insertion 31 Figure 28 Ejector Latch 32 Figure 29 Cable Holder 33 Figure 30 Ground Connection 34 Figure 31 Multiple Power Inlets Electric Caution Notification 36 Figure 32 Spine Module 36 Figure 33 Port Numbering 38 Figure 34 Top and Bottom Ports 38 Figure 35 Spine Side Panel Display Status Indications 40 Figure 36 Management Module Status Indications for Normal Operation 40 Figure 37 Connecting the Harness to the Console 41 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Figure 38 Web UI Login Page 48 Figure 39 Reset Button 49 Rev 1 0 List of Tables Table 1 Revision History of this Installation Guide 7 Table 2 Reference Documents and Web Sites Table 3 Serial Terminal Program Configuration 42 Table 4 Confi
40. re 30 Ground Connection ri oooooooo S2 mm uboguonon DOG pomno Dess S3 pooo oaa Dnnnnonn DDDOODOOD S4 aei aaae DDODDDDDH Cooooo0o0 Cogoooo00 DUDDUDODU DODDUDDDU aa a a a DODDOGUDH OOoooooo ooooooon poao DIUDGDEROD oonooooo Ooo00o000 Ss s BEES ooonnooon 200000000 Aaaa ES perum s9 Coooo0000 DDDDEDOOD DDDDODDRL DIDBDDDDBD ESD Post Ground Post 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 4 Power Connections The switch includes integrated hot swap power supplies which support up to 6 load shar ing 1000W or 1600W supply units The slots for the power supply units PSUs are on the spine side The left side has odd numbered PSUS and the right side has even numbered PSUs Each PSU has a dedicated AC inlet This design enables the optional use of separate main and backup AC feeds The input voltage is Auto range 100 240 VAC 50Hz or 60Hz The output voltage for the PSUs is 48V The power cords are standard 3 wire AC power cords including a safety ground Before starting any procedure on the SX6518 switch system put an ESD prevention wrist strap on your wrist and connect to the chassis 4 1 Powering Up the Switch Platform ferent countries can be ordered from Mellanox See the user manual appendix for Make sure that the power cords are compatible with your outlets Power cords for dif ordering part numbers for power cords Step 1 Plug in the power cords to the
41. re the spine side of the shelf is closest to the rack doors See Figure 10 to identify the spine side of the shelf Step 7 Install and tighten the 12 M6 bolts into the caged nuts l 20 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Figure 10 Shelf Brackets The spine side of the shelf has two slots for the extension P dE dI AP AP AP P AP AP AP P d y o z Adjustable shef brackets EE E EEUE EE AP AP D P P P AP AP 4 Step 8 Tighten the 16 hex head bolts on the adjustable brackets Figure 11 U Numbering Above Shelf ll These numbers are num bers relative to the top of IIR shelf The first cage nut on the spine side goes here The next six caged nuts go every 4 U units up from here Step 9 Place 8 caged nuts into the vertical rack support at the locations corresponding to the holes in the chassis faceplate The holes in the faceplate start at 2U above the shelf and are spaced every 4U centered as you go up the vertical rack support There are 4 holes on each side Mellanox Technologies 21 HMM MP Rev 1 0 Chassis Installation 1 6 3 6 Inserting the Chassis of the pallet Should the chassis start to fall move out of the way The chassis can a The chassis is on ball bearings and can roll easily Be aware that the ch
42. tches need to be able to dissipate the heat generated by high power I O cables and modules The Mellanox SX65xx chassis are rated for cables up to class 2 as per the SFF committee classification SFF 8436 PDF See http www mellanox com content pages php pg cables for the cable class rating of Mellanox cables l 38 Mellanox Technologies 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 6 Chassis Power Up Before starting any procedure on the SX6518 series chassis system put an ESD pre vention wrist strap on your wrist and connect to the SX6518 series chassis With N 1 PSU redundancy the chassis must be started with a full complement of possible PSUs thereafter it can run on one less than the total number of PSUs This final PSU is redundant and allows for hot swapping a PSU should one fail Connecting the PSUs to dif ferent AC lines provides AC failover protection The system should continue to run and allow a hot swap of a defective PSU Should there not be enough power to keep all of the leafs running MLNX OS may power down some leafs If this happens it will be necessary to reboot the chassis once the defective PSU has been replaced With 1600W optional PSU supplies the chassis can run on as little as 1 2 of the full com plement of PSUs N N allows the chassis to run on 3 PSUs supplied from one power grid while 3 are connected to a second power grid 1 Check all FRUs for proper insertion and seatin
43. types of leaf fan modules If the new leaf fan does not go into the chassis confirm that it is the correct fan module Mellanox Technologies 51 er Revi 0 TroubleShooting 4 Replace defective leaf fan modules as soon as they are identified Should any of the leaf boards shut down due to over temperature it may be necessary to reboot the chassis 10 3 3 Yellow LED for the Spine Fan on the Management Module is Lit Step 1 Check the MLNX OS management for confirmation and possible explanation of the alert Step 2 Determine which spine has a defective fan by checking the Fan LEDs on all of the spines Step 3 Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked especially dust over the holes Step 4 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner Step 5 Remove and reinstall the fan unit of the spine Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 5 4 on page 70 Step 6 Replace the spine fan module 4 Replace defective spine fan modules as soon as they are identified 10 4 Spine Board The yellow LED on the Spine board is lit Step 1 Check the MLNX OS management for confirmation and possible explanation of the alert Step 2 Make sure that there is nothing blocking
44. u want to use the wizard for initial configuration y Step 1 Hostname switch 112126 Step 2 Use DHCP on mgmt0 interface yes no Step 3 Use zeroconf on mgmt0 interface no yes Step 4 Default gateway For example 192 168 10 1 Step 5 Primary DNS server Step 6 Domain name Step 7 Enable IPv6 yes yes Step 8 Enable IPv6 autoconfig SLAAC on mgmt0 interface no no Step 9 Admin password Enter to leave unchanged Step 9 Confirm admin password Enter to leave unchanged You have entered the following information Hostname switch 112126 Use DHCP on mgmtO interface no Use zeroconf on mgmtO interface no yes Default gateway 192 168 10 1 Primary DNS server Domain name Enable IPv6 yes Enable IPv6 autoconfig SLAAC on mgmtO interface no Admin password Enter to leave unchanged unchanged wo 00 10 Un UC r2 To change an answer enter the step number to return to Otherwise hit enter to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI switch 1 324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1 0 Table 6 Configuration Wizard Session Static IP Configuration Wizard Session Display Static IP Configuration Example Mellanox configuration wizard Do you want to use the wizard for i
Download Pdf Manuals
Related Search