Sun Blade 6048 InfiniBand QDR Switched Network Express Module
Contents
1. CMM Blade 0 Versions Session Time Out Components l Identification Information li Blade 1 de Component Management Blade 3 View component information from this page To view further details click on a Component Name Blade 4 SE aida Management Statu Blade 6 Component Name Type Henog j ACH Chassis lt JCH CMM Chassis Monitoring Module pes i CH CMMNETO Netwokinetace Blade 10 CH CMM SP Service Processor Blade 11 i CH CMMA MB Motherboard Blade 12 CH BLO Blade Blade 13 cueros oo Service Processor 0000 Blade 14 CH BLO SEEPROM Blade 15 CH L1 Blade 16 cHmBL16P Service Processor Blade 17 CH LVSEEPROM prom Blade 18 cHBL2 Blade Blade 19 f CH BL2 SP Service Processor Blade 20 CH BL2 SEEPROM PROM mccoy ene 0 0 0 S Ble Blade 22 CH BLS SP Service Processor TEN CH BL3 SEEPROM PROM CH BL4 Blade rm CH BLA SP Service Processor CH FMSE Rear Fan cH FMG Rear Fan E ICHMT Rear Fan CH PSO Power Supply CHPS1 Power Supply Blade 23 CHEN Network Express Module CHINEMT SEEPROM PROM Blade 18 Blade 19 Blade 20 Blade 21 Blade 22 Service Processor 4 Scroll down to CH NEMI and select the NEM1 component name The ILOM page showing the IB ONEM status details appears 16 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 20092. 2 Start the CLI by typing the command shown in the example gt start CH NEM1 SP cli Are you sure you want to start CH NEM1 SP cli y n y Sun Blade 6048 QDR InfiniBand Network Express Module gt help Special characters used in the help command are 1 encloses optional keywords or options lt gt encloses a description of the keyword If lt gt is not present an actual keyword is indicated indicates a choice of keywords or options Valid proxy CLI commands are show lt property gt set lt property gt lt value gt reset resetswitch A B all version help lt command gt Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 19 exit 3 Type the show command to display the IB ONEM hardware and software information show NEM1 Targets Properties fpga_version 03 cpld_version 08 cpld_id 5 full_serial_no 0110SJC 0904 S N CA0037 P N 5413378 HW_rev OL FRUID specno 885 1445 01 ENG change tag mfg_time Mon Jan 19 19 12 59 2009 cable_present 0 4 2 3 4 b 6 T7 8 9 l1 Lod ooi 1 L ports A External 4x IB ports on switch A AO Al A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14 Up Up Up Up Up Up Up Up Up Up Up Up Up Up Up ports belonging to server blades on switch A BLO O BLOS1 BL1 0 BL1 1 BL2 0 BL2 1 BL3 0 BL3 1 BL4 0 BL4 1 BL540 BLS 1 down down down down down down down down Up Up Up d
3. Sun Integrated Lights Out Manager View component name and information CH NEM1 li Property Value type fru name Network Express Module ASSY SB6048 QDR NEM 541 3378 01 i fru serial numbe 01105JC 0904CAO0036 fru part number 5 If you are physically near the IB ONEM you can examine its LEDs to verify that it has returned the expected feedback See To Verify Component Status Using the LEDs on page 18 V To Verify Installation Using the ILOM CLI 1 Log in to the ILOM CLI 2 Find the NEMI in your system type gt show CH 3 To verify that the NEM1 is installed type gt show CH NEM1 CH NEM1 Targets SEEPROM SP SAS Properties type Network Express Module fru_name ASSY SB6048 QDR NEM Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 17 fru part number 541 3378 01 fru serial number 0110SJC 0904CA0037 Commands cd show 4 If you are physically near the IB QNEM examine its LEDs to verify that it has returned the expected feedback See To Verify Component Status Using the LEDs on page 18 To Verify Component Status Using the LEDs Use the LED combinations to determine the status of the IB ONEM Blue LED Amber LED Green LED Top Middle Bottom IB QNEM Status Off Off Off Ready to remove Off On Off Service attention required Off Off Slow blink
4. ASHRAE Environmental Class1 Altitude 3 000m derate 1 degree C 300m above 900m Humidity 20 80 TABLE 1 5 Physical Characteristics Height 155mm 6 1 in Width 509mm 20 in Depth 108 6mm 4 28 in Weight 5 73 kg 12 7 Ibs Regulatory Compliance The regulatory compliance certifications of the Sun Blade 6048 Modular System cover the Sun Blade 6048 InfiniBand QDR Switched Network Express Module as an installed component To comply with legal EMI emissions regulations the Sun Blade 6048 Modular Server must be provided with either the standard door or cooling rear door assembly when used with the Sun Blade 6048 InfiniBand QDR Switched Network Express Module The rear door should remain closed during normal operation and opened only during product servicing or when power is removed from the product Chapter 1 Sun Blade 6048 InfiniBand QDR Switched Network Express Module Introduction 9 Commonly Used Terms TABLE 1 6 identifies some terms commonly used in this guide TABLE 1 6 Common Terms Term Chassis CMM IB I4 ILOM IB ONEM Server module SP SDR DDR QDR Definition Sun Blade 6048 modular system hardware For additional information about the Sun Blade 6048 Modular System go to http docs sun com app docs coll blade6048 Chassis Monitoring Module CMM hardware InfiniBand Mellanox Infiniscale IV I4 36 port IB switch There are two I4s switches on the IB QMEN I4A and I4B Sun In
5. QDR Switched NEM I4A base port 0 lid 3 1mc 0 CA type MT26428 Sun Blade 6048 InfiniBand Sun Blade 6048 InfiniBand WV To List All IB Hosts in the IB Network Usethe ibhosts command to list all hosts in the IB network ibhosts Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca Ca 0x50800200008e4974 0x50800200008e4970 0x50800200008e4954 0x50800200008e4950 0x50800200008e4ccc 0x50800200008e48 4 0x50800200008e48 0 0x50800200008e4934 0x50800200008e4930 0x50800200008e4924 0x50800200008e4920 0x50800200008e4a7c 0x50800200008e4a78 0x50800200008e4964 0x50800200008e4960 0x50800200008e5424 0x50800200008e5420 0x50800200008e4914 0x50800200008e4910 0x50800200008e4d3c 0x50800200008e48e4 0x50800200008e48e0 0x50800200008e4d38 por por por por por por por por por por por por por por por por por por por por por por por Appendix B CS 1 CS 1 US 1 CS 1 US 1 CS 1 US 1 US 1 US L CS 1 CS 1 CS 1 CS 1 US 1 CS 1 CS 1 CS 1 CS L US 1 CS 1 US 1 US 1 ES L ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ib 1 ip 1 ib 1 ib 1 ib 1 at 5 1 ib 1 ib 1 10 11 21 20 07 15 14 19 18 23 22 17 16 08 09 01 00 05 04 03 13 12 02 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1 CA 1
6. power consumption wattage 3 product features 3 Q ODR cables 3 R regulatory compliance 9 RoHS 5 of 6 compliant 3 S server modules 4 specifications environment 9 physical 9 state of the NEM to blade connections determining 18 switch Mellanox InfiniHost IV 3 42 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009
7. 25 down 17 26 down 18 27 down 19 28 down 20 29 down 21 30 down 22 36 down 23 35 down 24 34 down 25 33 down 26 32 down 27 31 down 28 3 Up 29 2 Up 30 1 Up 31 19 Up 32 20 Up 33 21 Up 34 22 Up 35 23 Up 36 24 Up 22 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 temperature 34 degrees leds Locate Off OK On Service Off R2R Off ipass0 2612BER 0904XB0006 10m 84 Circuit iPass Stacked Active Optical Cable ipass1 2612BER 0905XA0011 10m 84 Circuit iPass Stacked Active Optical Cable ipass2 2612BER 0905XB0003 10m 84 Circuit iPass Stacked Active Optical Cable ipass3 2612BER 0905XB0004 10m 84 Circuit iPass Stacked Active Optical Cable ipass4 2612BER 0904XB0025 10m 84 Circuit iPass Stacked Active Optical Cable ipass5 2612BER 0904XA0006 10m 84 Circuit iPass Stacked Active Optical Cable ipass6 2612BER 0905XB0011 10m 84 Circuit iPass Stacked Active Optical Cable ipass7 2612BER 0905XA0003 10m 84 Circuit iPass Stacked Active Optical Cable ipass8 2612BER 0905XA0004 10m 84 Circuit iPass Stacked Active Optical Cable ipass9 2612BER 0904XA0025 10m 84 Circuit iPass Stacked Active Optical Cable Commands show set reset resetswitch version help exit Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 23 24 Sun Blade 6048 InfiniBand Q
8. the ejector levers and damage the chassis Once the IB ONEM is disconnected from the midplane the green OK led will turn off 5 Pull both ejector levers away from the NEM slot to slide the IB ONEM out until you can comfortably remove the assembly with both hands Caution Do not support the IB QNEM by the two ejector levers 12 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 V To Install the IB ONEM in a Powered On Chassis 1 Inspect the midplane connectors in the chassis NEM slots for bent or damaged pins before installing the IB ONEM Do not install the IB ONEM if you find any bent or damaged pins Align the IB QNEM with the top NEM slot and the chassis guidance system Ensure that the IB ONEM aligns with the chassis guidance system Failure to align the IB ONEM correctly can result in damage to the IB QNEM s connections to the chassis midplane Slide the IB ONEM into the NEM guide rails until the ejector levers touch the guide rails Pull both ejector levers out to the sides and position them into their latch on the rails The levers can be closed slightly to ensure that they are aligned correctly With both hands on the rear panel push the IB ONEM firmly into the midplane connectors Caution Do not push the IB QNEM in at an angle or slam it into the midplane connectors Such actions might result in bent or damaged connector pins 6 Pus
9. 3 KYUN microsystems Sun Blade 6048 InfiniBand QDR Switched Network Express Module Users Guide Sun Microsystems Inc www sun com Part No 820 6705 10 April 2009 Revision A Submit comments about this document at http www sun com hwdocs feedback Copyright 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 U S A All rights reserved Sun Microsystems Inc has intellectual property rights relating to ee is described in this document In particular and without limitation these intellectual property rights may include one or more of the U S patents listed at http www sun com patents and one or more additional patents or pending patent applications in the U S and in other countries This document and the product to which it pertains are distributed under licenses restricting their use copying distribution and decompilation No part of the product or of this document may be reproduced in any form by any means without prior written authorization of Sun and its licensors if any Third party software including font technology is copyrighted and licensed from Sun suppliers Parts of the poe may be derived from Berkeley BSD systems licensed from the University of California UNIX is a registered trademark in the U S and in other countries exclusively licensed through X Open Company Ltd Sun Sun Microsystems the Sun logo Sun Blade Java docs sun com and Solaris are trademarks or re
10. 3 Updating the Sun Blade 6048 InfiniBand ODR Switched Network Express Module Firmware 25 Downloading the IB ONEM Firmware 25 Updating IB ONEM Firmware Using Linux Host 26 v To Update the IB QNEM Integrated Switch Firmware 26 v To Verify the IB ONEM Switch Firmware 27 A InfiniBand QDR Cables 29 IB ONEM Cables 29 B InfiniBand Command Examples 31 Configuration Scenario Used for Command Examples 31 v To Display the Local Host s IB Device Status 31 v To Check the Presence of an IB QNEM 33 To List All IB Hosts in the IB Network 33 To Display the Network Topology for the Shelf 34 To Query the Local Host s IB Firmware 37 To Check the Status of the Subnet Manager 38 To Verify IPoIB 38 4 4 4 4 Index 41 iv Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Preface This guide provides an overview installation instructions and related information for the Sun Blade 6048 InfiniBand QDR Switched Network Express Module IB ONEM These instructions are designed for system administrators with InfiniBand network experience Related Documentation The documents listed as online are available at http docs sun com app prod blade 6048 Part Document Number Available Sun Blade 6048 InfiniBand QDR Switched Network Express Module 820 6706 Online Product Notes Where to Find Sun Blade 6048 Modular System Documentation 820 2311 Printed and online Sun Blade 6048 Modular System S
11. 75 QDR IB 1 8 V To Check the Status of the Subnet Manager Usethe sminfo command to display the status of the Subnet Manager swinfo sminfo sm lid 13 sm guid 0x50800200008e4921 activity count 194341 priority 0 state 3 SMINFO MASTER V To Verify IPoIB Use the ifconfig and ping commands in the example to verify IPoIB ib0 lo ifconfig Link encap InfiniBand HWaddr 80 00 00 48 FE 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 inet addr 100 1 1 102 Bcast 100 1 1 255 Mask 255 255 255 0 inet6 addr fe80 5280 200 8e 4d39 64 Scope Link UP BROADCAST RUNNING MULTICAST MTU 65520 Metric 1 RX packets 0 errors 0 dropped 0 overruns 0 frame 0 TX packets 51 errors 0 dropped 9 overruns 0 carrier 0 collisions 0 txqueuelen 256 RX bytes 0 0 0 b TX bytes 8823 8 6 KiB Link encap Local Loopback inet addr 127 0 0 1 Mask 255 0 0 0 inet6 addr 1 128 Scope Host UP LOOPBACK RUNNING MTU 16436 Metric 1 RX packets 8 errors 0 dropped 0 overruns 0 frame 0 TX packets 8 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 0 RX bytes 560 560 0 b TX bytes 560 560 0 b ping 100 1 1 100 PING 100 1 1 100 100 1 1 100 56 84 bytes of data 64 bytes from 100 1 1 100 icmp seq 1 ttl 64 time 2 64 ms 64 bytes from 100 1 1 100 icmp seq 2 ttl 64 time 0 093 ms 64 bytes from 100 1 1 100 icmp seq 3 ttl 64 time 0 105 ms 100 1 1 100 ping statistics Sun Blade 6048 InfiniBand QDR Switched Network Expre
12. CA 1 Could be 5 0 Gbps 3 24 1 1 ib 120 HCA 1 Could be 5 0 Gbps 3 25 15 1 ib 121 HCA 1 Could be 5 0 Gbps 3 26 6 TC ib 111 HCA 1 Could be 5 0 Gbps 3 27 19 1 ib 110 HCA 1 InfiniBand QDR Switched NEM I4B Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4B Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4B Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4B Gbps Active LinkUp gt InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Switched inkUp gt Gbps Active 14B Switch 0x002128183ea40050 Sun Blade InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active InfiniBand QDR Gbps Active bi GI bi bi bi Swi Li Swi Li Swi Li Swi Li Swi Li Swi Li Swi Li Swi Li Swi Li Switched NEM I4B Switched NEM I4B Switched NEM I4B Switched gt nkUp nkUp gt nkUp gt NEM nkUp gt NEM TAB TAB tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A nkUp gt tched NEM I4A n
13. CA 1 CA 1 InfiniBand Command Examples 33 V To Display the Network Topology for the Shelf Usethe iblinkinfo command to list the network topology for the shelf iblinkinfo pl R Switch 0x002128183ea40040 Sun Blade 6048 InfiniBand QDR Switched NEM I4A InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 1 4X 10 0 Gbps Active LinkUp gt 4 1 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 2 4X 10 0 Gbps Active LinkUp gt 4 2 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 3 4X 10 0 Gbps Active LinkUp gt 4 3 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 4 4X 10 0 Gbps Active LinkUp gt 4 4 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 5 4X 10 0 Gbps Active LinkUp gt 4 5 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 6 4X 10 0 Gbps Active LinkUp gt 4 6 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 7 4X 10 0 Gbps Active LinkUp gt 4 7 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 8 4X 10 0 Gbps Active LinkUp gt 4 8 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 9 4X 10 0 Gbps Active LinkUp gt 4 9 Sun Blade 6048 InfiniBand QDR Switched
14. DR Switched Network Express Module User s Guide April 2009 CHAPTER 3 Updating the Sun Blade 6048 InfiniBand QDR Switched Network Express Module Firmware This chapter provides information on updating the IB QNEM firmware on Linux Consult the Sun Blade 6048 Series Product Notes for the most recent information about the availability of firmware updates This chapter contains the following sections m Downloading the IB ONEM Firmware on page 25 m Updating IB ONEM Firmware Using Linux Host on page 26 Downloading the IB QNEM Firmware The firmware version on your IB ONEM should be ready to use and should not require updating However if you want to update the firmware for any reason you must use vendor specific and generally OS specific firmware updating tools The IB ONEM firmware for this release and any future firmware releases can be downloaded from the Sun Download Center SDLC at http www sun com download 25 Updating IB ONEM Firmware Using Linux Host The IB ONEM switch firmware can be updated in band from any node in the fabric The node or blade server must have the following software installed m Supported Linux operating system m OpenFabrics Enterprise Distribution OFED m Mellanox Firmware Tools MFT V To Update the IB QNEM Integrated Switch Firmware The IB ONEM switch firmware can be updated in band from any node in the fabric Note Always use the instructions provided with the do
15. NEM I4B Could be 5 0 Gbps 3 10 4X 10 0 Gbps Active LinkUp gt 4 10 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 11 4X 10 0 Gbps Active LinkUp gt 4 11 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 12 4X 10 0 Gbps Active LinkUp gt 4 12 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 13 4X 10 0 Gbps Active LinkUp gt 4 13 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 14 4X 10 0 Gbps Active LinkUp gt 4 14 Sun Blade 6048 InfiniBand QDR Switched NEM I4B Could be 5 0 Gbps 3 15 4X 10 0 Gbps Active LinkUp gt 4 15 Sun Blade 6048 Could be 5 0 Gbps 3 16 4X 10 0 Gbps Active LinkUp gt 13 1 ib 122 HCA 1 Could be 5 0 Gbps 3 17 4X 10 0 Gbps Active LinkUp gt 14 1 ib 123 HCA 1 Could be 5 0 Gbps 3 18 4X 10 0 Gbps Active LinkUp gt 5 1 ib 118 HCA 1 Could be 5 0 Gbps 3 19 4X 10 0 Gbps Active LinkUp gt 8 1 ib 119 HCA 1 Could be 5 0 Gbps 3 20 4X 10 0 Gbps Active LinkUp gt 16 1 ib 114 HCA 1 Could be 5 0 Gbps 3 21 4X 10 0 Gbps Active LinkUp gt 10 1 ib 115 HCA 1 34 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Could be 5 0 Gbps 3 22 67 1 ib 222 HCA 1 Could be 5 0 Gbps 3 23 98 1 ib 107 H
16. SD wrist strap from any reputable electronics store or from Sun as part number 250 1007 Replacing IB QNEM Hardware If an IB QNEM fails or if you choose to change the I O configuration you must replace the IB QNEM You can replace an IB QNEM in a powered on Sun Blade 6048 Series Chassis using a hot plug operation A A If you are removing but not replacing the IB QNEM you must install both NEM slot filler panels to meet FCC limits for electromagnetic interference EMI and to ensure proper airflow and cooling If you encounter a problem replacing the IB QNEM see Managing the IB ONEM on page 18 Note If you are installing a IB QNEM in a Sun Blade 6048 Series Chassis that has not been powered on see the Sun Blade 6048 Series Installation Guide 820 2312 The IB ONEMs are customer replaceable units CRUs To Remove the IB ONEM in a Powered On Chassis 1 Identify which IB ONEM to replace If the amber Service Action Required indicator is lit on a specific IB QNEM there is a problem with that IB ONEM Otherwise you can choose any IB ONEM to replace if for example you want to change the I O configuration 2 Remove all cables from the IB ONEM 3 Squeeze the two tabs on both ejector levers to unlatch the levers from the IB ONEM rear panel 4 Pull both ejector levers simultaneously out to the sides until the IB ONEM disengages from the chassis midplane Caution Be careful not to over rotate
17. Standby Off Off On Links connected to blades 18 Managing the IB QNEM Because the IB QNEM are shared resources all Sun Blade server modules must respond favorably to the PCI hot remove request However a blade might not relinquish the link to a IB QNEM if for instance there are busy NFS mounted volumes file transfers and so on Note The IB ONEM does not need any user level configurations IB ONEM functionality is predefined by the switches firmware Advance network and InfiniBand configurations can be performed using the Subnet Manager on a supported server module To determine the state of the NEM to blade connections you can use the ILOM command line interface as described in the following procedure Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 V To Manage IB ONEM Using the ILOM CLI 1 Log in to the ILOM CMM CLI interface For example ssh root cmm ip address The authenticity of host xx xx xx xxx xx xx xx xxx can t be established RSA key fingerprint is a2 71 bd bc ce 74 4c 24 50 5f fb c5 c3 01 e6 58 Are you sure you want to continue connecting yes no yes Warning Permanently added 10 60 32 197 RSA to the list of known hosts Password Sun TM Integrated Lights Out Manager Version 2 0 3 13 Copyright 2008 Sun Microsystems Inc All rights reserved Use is subject to license terms Warning password is set to factory default gt
18. e has 15 ports connected to 5 CXP external IB connectors I4A has 12 ports connected to the two 4x ports from iTRAC 0 through 5 and I4B has 12 ports connected from iTRAC 6 through 11 The remaining nine ports are intralink connections between the two I4 devices Chapter 1 Sun Blade 6048 InfiniBand QDR Switched Network Express Module Introduction 7 FIGURE 1 3 IB ONEM InfiniBand Port Mapping Rear Cable Side A0 1 2 A3 4 5 A6 7 8 A9 10 11 A12 13 14 B0 1 2 B3 4 5 B6 7 8 B9 10 11 B12 13 14 ca SI 2 E i e zm e n a oa n n n a au n a X X x x X X x X x x Oo o O O Oo Oo Oo Oo O 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 3 14 15 3 3 3 3 3 3 3 3 3 3 28 28 29 29 30 30 31 31 I4A 32 9 Intraswitch 32 14B 33 Links 33 34 34 35 35 36 36 2 2 2 2 2 2 2 2 2 2 2 2 16 17 18 19 20 21 22 23 24 25 26 27 16 17 18 19 20 21 22 23 24 25 26 27 ITRAC_O ITRAC 1 ITRAC 2 ITRAC 3 ITRAC 4 ITRAC 5 ITRAC 6 ITRAC 7 ITRAC 8 ITRAC 9 TRAC 10 ITRAC 11 To To To To To To To To To To To To Slot 0 Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Slot 9 Slot 10 Slot 11 Front To Midplane 8 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Specifications TABLE 1 4 and TABLE 1 5 list the specifications for the IB ONEM in the Sun Blade 6048 Series Modular System TABLE 1 4 Operating Environment Voltage 12V Maximum power 240 watts Temperature 15 to 32 C
19. f the node board is installed in the shelf slots 0 5 reset the I4B switch before resetting the I4A switch If the node board is installed in the shelf slots 6 11 reset the I4A switch before resetting the I4B switch Ensure that the reset completes before resetting the other switch A switch reset could take up to 90 seconds to complete flint d lid 4 swreset Resetting device lid 4 flint d lid 3 swreset Resetting device lid 3 V To Verify the IB QNEM Switch Firmware Check that the FW Version and PSID are consistent with the downloaded firmware Use flint tool to query the switch firmware on both switches flint d lid 3 q Image type FS2 FW Version 7 1 948 Device ID 48438 Chip Revision AO Description Node Sys image Chapter 3 Updating the Sun Blade 6048 InfiniBand QDR Switched Network Express Module Firmware 27 GUIDs 002128183ea40040 002128183ea40043 Board ID n a QNEM 04 1I4A 1 9 SD n a BSID ONEM 04 14A 1 9 flint d lid 4 q Image type FS2 FW Version 7 1 948 Device ID 48438 Chip Revision AO Description Node Sys image GUIDs 002128183ea40050 002128183ea40053 Board ID n a QNEM 04 I4B 1 9 SD n a PSID QNEM 04 I4B 1 9 28 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 APPENDIX A InfiniBand QDR Cables This appenix provides information on the InfiniBand QDR cables This appendix
20. gistered trademarks of Sun Microsystems Inc or its subsidiaries in the U S and in other countries The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems Inc for its users and licensees Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry Sun holds a non exclusive license from Xerox to the Xerox Graphical User Interface which license also covers Sun s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun s written license agreements U S Government Rights Commercial use Government users are subject to the Sun Microsystems Inc standard license agreement and applicable provisions of the FAR and its supplements DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS REPRESENTATIONS AND WARRANTIES INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR NON INFRINGEMENT ARE DISCLAIMED EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID Copyright 2009 Sun Microsystems Inc 4150 Network Circle Santa Clara Californie 95054 Etats Unis Tous droits r serv s Sun Microsystems Inc poss de les droits de propri t intellectuels relatifs 4 la technologie d crite dans ce document En particulier et sans limitation ces droits de propri t intellectuels peuvent inclure un ou TRU des brevets am ricains list s s
21. h Coul 13 Coul 14 Coul 15 ld be 5 0 Gbps d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade Sun Blade 4 604 4 604 4 604 4 604 4 604 4 604 8 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active Coul 28 Coul 29 Coul 30 Coul 31 Coul 32 Coul 33 d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade d be 5 0 Gbps Sun Blade 604 604 604 8 8 L8 L8 L8 8 28 29 30 31 32 33 LinkUp LinkUp AX 1 AX 1 AX 1 AX 1 AX 1 LinkUp LinkUp LinkUp LinkUp pie LinkUp LinkUp LinkUp LinkUp LinkUp LinkUp AX 1 AX 1 AX 1 AX 1 AX 1 AX 1 36 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 InfiniBand QDR Switched NEM I4A Could be 5 0 Gbps 4 34 Gbps Active LinkUp gt 3 34 Sun Blade 6048 Inf
22. h both ejector levers simultaneously toward the center so that the IB ONEM engages fully with the connectors on the midplane The levers should eventually lock into their latch openings on the IB ONEM rear panel Caution Locking one lever at a time might result in seating problems that can lead to symbol errors The IB ONEM will automatic power on once the IB ONEM is fully seated in the chassis The green OK LED will blink during power up and eventually remain lit once the power on completes successfully If the amber Service Action Required indicator is on try re installing the IB ONEM If the problem persists initiate a service call Connect the InfiniBand cables to the IB ONEM port connectors Ensure that the connectors are properly engaged The connectors click when locked Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 13 the cable near the connectors and avoid sharp cable bends of more than Caution Avoid putting unnecessary stress on the connection Do not bend or twist 90 degrees 8 Verify that the IB QNEM is working properly See To Verify Hardware Installation on page 14 Note If you are replacing an IB ONEM you might not need to install the InfiniBand software packages on the server modules The appropriate software package will have been installed and configured as part of the initial IB QNEM installation Refer to the server module s documentatio
23. hwdocs feedback Please include the title and part number of your document with your feedback vi Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide Sun part number 820 6705 10 Preface vii viii Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 CHAPTER 1 Sun Blade 6048 InfiniBand ODR Switched Network Express Module Introduction This chapter provides an overview of the Sun Blade 6048 InfiniBand ODR Switched Network Express Module IB ONEM This chapter includes the following sections m Product Features on page 1 m Sun Blade 6048 Platform Support on page 4 m IB ONEM Indicators Buttons and Ports on page 4 m IB ONEM Port Mapping on page 7 m Specifications on page 9 m Regulatory Compliance on page 9 m Commonly Used Terms on page 10 You can order additional Sun Blade 6048 InfiniBand ODR Switched Network Express Modules from Sun Microsystems using the following Marketing part number X5500A Z Product Features The Sun Blade 6048 InfiniBand QDR Switched Network Express Module FIGURE 1 1 has the following features m 30 InfiniBand ports for external connectivity m 214 on board switches both providing 2 Infiniband ports for each slot in a Sun Blade 6048 modular system shelf m 9 intraswitch links between two switches within IB QNEM m 2 pass
24. includes the following section m IB QNEM Cables on page 29 IB ONEM Cables TABLE A 1 contains the list of the IB QNEM cables available at the time of release Contact your Sun representative for availability of additional cables Caution Ensure that the IB QDR cables are being used The IB DDR cables are similar and can be mistaken for IB ODR cables Always check that the keying on the cable connector matches the keying on the IB ONEM connector before installing a new cable TABLE A 1 IB ONEM Cables Length and Type Part Sun IB QDR 12x Optical Cable 10 meters X2880 10m Sun IB QDR 12x Optical Cable 20 meters X2880 20m Sun IB QDR 12x Optical Splitter Cable 20 meters X2881 20m Sun IB ODR 12x Copper Jumper Cable 0 4 meters X2882 0 4m 29 30 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 APPENDIX B InfiniBand Command Examples This section provide some commands and typical outputs used to verify an InfiniBand IB network and the presence of each component in a Sun Blade 6048 Series Modular System shelf This appendix includes the following section m Configuration Scenario Used for Command Examples on page 31 v Configuration Scenario Used for Command Examples The IB ONEM is installed in a Sun Blade 6048 Series Modular System shelf The node or blade server has the following software installed m Supported Linux operating system m OpenFabrics Enterprise Dist
25. iniBand QDR Switched NEM I4A Could be 5 0 Gbps 4 35 Gbps Active LinkUp gt 3 35 Sun Blade 6048 InfiniBand QDR Switched NEM I4A Could be 5 0 Gbps 4 36 10 0 Gbps Active LinkUp gt 3 36 Sun Blade 6048 4x 10 0 4x 10 0 V To Query the Local Host s IB Firmware Use the mst and flint commands display the host s firmware information mst status MST modules MST devices data reg 92 size 0x100000 size 0x0 size 0x800000 MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded dev mst mt26428 pciconf0 dev mst mt26428 pci cr0 dev mst mt26428 pci msixO0 dev mst mt26428 pci uar0 PCI configuration cycles access bus dev fnz07 00 0 addr reg 88 Chip revision is A0 PCI direct access bus dev fnz07 00 0 bar 0xfaf00000 Chip revision is A0 PCI direct access bus dev fn 07 00 0 bar 0x00000000 PCI direct access bus dev fn 07 00 0 bar 0xf3800000 Image type FW Version Rom Info Device ID Chip Revision Description image ConnectX 2 6 0 type GPXE 26428 AO Node 50800200008e4d3bGUIDs 50800200008e4d3a flint d dev mst mt26428 pci cr0 q version 1 9 972 devid 26428 Porti Port2 Sys 50800200008e4d38 50800200008e4d39 Appendix B InfiniBand Command Examples 37 38 MACs 00144fca3806 00144fca3807 Board ID n a X6275 QDR IB 1 8 VSD n a BSID X62
26. ite Planning Guide 820 0426 Online Sun Blade 6048 Modular System Unpacking Guide 820 2987 Printed and online Sun Blade 6048 Modular System Setup Poster 820 2310 Printed and online Sun Blade 6048 Modular System Installation Guide 820 2312 Printed and online Sun Blade 6048 Modular System Service Manual 820 2863 Online Part Document Number Available Sun Blade 6048 Modular System Safety and Compliance Guide 820 0053 Online Sun Blade 6048 Modular System Product Notes 820 2309 Online Sun Integrated Lights Out Manager ILOM 2 0 User s Guide 820 1188 Online Documentation Support and Training Sun Function URL Documentation http docs sun com Support http www sun com support Training http www sun com training Third Party Web Sites Sun is not responsible for the availability of third party web sites mentioned in this document Sun does not endorse and is not responsible or liable for any content advertising products or other materials that are available on or through such sites or resources Sun will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content goods or services that are available on or through such sites or resources Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions You can submit your comments by going to http www sun com
27. kUp gt tched NEM I4A nkUp gt 4 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active 4X 10 0 Gbps Active Could 28 Could 29 Could 30 Could 31 Could 32 Could 33 Could 34 Could 35 Could 36 Could 1 Could 2 Could 3 Could A Could 5 Could 6 Could 7 Could 8 Could 9 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 be 5 0 Gbps 3 Sun Blade 604 604 8 8 8 8 8 8 28 29 30 31 32 33 34 35 36 LinkUp gt LinkUp gt LinkUp gt LinkUp gt LinkUp gt LinkUp gt 6048 InfiniBand QDR Switched NEM be 5 0 Gbps 4 be 5 0 Gbps 4 Sun Blade be 5 0 Gbps 4 Sun Blade 604 be 5 0 Gbps 4 Sun Blade 604 be 5 0 Gbps 4 Sun Blade 604 be 5 0 Gbps 4 Sun Blade be 5 0 Gbps 4 Sun Blade be 5 0 Gbps 4 Sun Blade be 5 0 Gbps 4 Sun Blade 604 604 604 Sun Blade 604 8 8 6048 6048 1 4X 1 AX 1 AX 1 AX 1 AX 1 AX 1 AX 1 AX 1 AX 1 4X 1 4X 1 4X 1 4X 1 4X 1 4X 1 4X 1 Appendix B InfiniBand Command Exam
28. n Lights steadily when IB ONEM is operating normally Slow blink Blinks slowly when IB ONEM is transitioning from one state to the next Off IB QNEM has no power or one of the other LEDs is lit Provides the following indications Steady on Lights when there is a fault associated with the IB QNEM Off The IB QNEM has no fault condition This indicator is not used on the IB QNEM RJ 45 Ethernet passthrough ports that connect to the server blades through the midplane Note Both PO and P1 Ethernet ports are available for the Sun Blade X6275 server module and only the P1 Ethernet port is available for the Sun Blade X6270 server module Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 TABLE 1 3 Sun Blade 6048 IB ONEM Back Panel Components and Functions Continued Number Component Name Color Description 6 Physical Link Green Illuminates when the port is electrically active that indicator is when a driver is attached and a physical link to InfiniBand a remote switch or possibly an HCA has been established Each InfiniBand port has one LED indicator 7 CXP connectors CXP connectors combine three 4x IB links per connector 8 Ejector levers To remove the IB ONEM open the ejector levers IB ONEM Port Mapping The IB ONEM has two 36 port InfiniBand switches I4A and I4B Each port is capable of negotiating with a peer node up to 4x ODR speed Each switch I4 devic
29. n for more information Verifying Installation If you have not installed the IB ONEM in the chassis and connected it to an operational InfiniBand switch or HCA do so before you attempt to verify the installation The InfiniBand switch should automatically recognize InfiniBand servers when the servers are connected to the fabric V To Verify Hardware Installation 1 Once you have physically installed the IB ONEM and ensured that the cables are connected to the IB ONEM and other IB devices switches or HCA ensure that an IB Subnet Manager is running on the connected InfiniBand fabric network If the green Physical Link indicator is illuminated you have successfully completed the hardware installation and you can proceed to verification through the ILOM interfaces Step 2 The green Physical Link indicator lights to show that the port is enabled That is a physical link to a remote switch or possibly an HCA has been established There might be a short delay before the indicator lights If the port LEDs are not illuminated one possible cause might be that the InfiniBand drivers are not installed You cannot verify a complete installation on Linux until you install these drivers 14 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 2 Examine hardware status through one of the ILOM Integrated Lights Out Manager interfaces Use one of the following procedures m To Verify Installati
30. on Using the ILOM Web Interface on page 15 m To Verify Installation Using the ILOM CLI on page 17 For a description of the possible states of the IB QNEM LEDs see To Verify Component Status Using the LEDs on page 18 V To Verify Installation Using the ILOM Web Interface 1 Log in to the ILOM web interface using the IP address of the CMM The initial page of the ILOM web interface appears providing visual verification of successful hardware installation Note the image of the installed IB QNEM in the view of the back of the chassis ABOUT REFRESH Role User Administrator root CMM Hostname SUN CMMOO144F9A371E Sun Integrated Lights Out Manager Blade O Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade 7 Blade 8 Blade 9 Blade 10 Blade 11 Blade 12 Blade 13 i Chassis View CMM To manage a Blade or Chassis Monitoring Module click on it in the left navigation pane or in the image below D Es er Ex mes uc TD es Isl csl ens Hs En 2 In the left navigation pane select CMM The ILOM Version Information page appears 3 Select the System Information tab and then select the Components tab The Component Management page appears Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 15 Role User Administrator root CMM Hostname SUNCMMOO144F9A3 Sun Integrated Lights Out Manager 5 Chassis
31. our l industrie informatique Sun d tient une license non exclusive de Xerox sur l interface utilisateur graphique Xerox cette licence couvrant Sgalement les licenci s de Sun impl mentant les interfaces utilisateur graphiques OPEN LOOK et se conforment en outre aux licences crites de un LA DOCUMENTATION EST FOURNIE EN L TAT ET TOUTES AUTRES CONDITIONS D CLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES DANS LA LIMITE DE LA LOI APPLICABLE Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE LA QUALITE MARCHANDE A L APTITUDE UNE UTILISATION PARTICULIERE OU A L ABSENCE DE CONTREFA ON oy Please CA Recycle Adobe PostScript Contents Preface v Sun Blade 6048 InfiniBand QDR Switched Network Express Module Introduction 1 Product Features 1 Sun Blade 6048 Platform Support 4 IB ONEM Indicators Buttons and Ports 4 IB ONEM Port Mapping 7 Specifications 9 Regulatory Compliance 9 Commonly Used Terms 10 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 11 Replacing IB QNEM Hardware 11 v To Remove the IB QNEM in a Powered On Chassis 12 v To Install the IB QNEM in a Powered On Chassis 13 Verifying Installation 14 v To Verify Hardware Installation 14 v To Verify Installation Using the ILOM Web Interface 15 v To Verify Installation Using the ILOM CLI 17 v To Verify Component Status Using the LEDs 18 Managing the IB ONEM 18 v To Manage IB QNEM Using theILOM CLI 19
32. own ports on switch A connected to the peer switch 28 29 30 31 32 33 34 35 36 Up Up Up Up Up Up Up Up Up State Logical p Phys pi 20 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 1 1 Up 2 2 Up 3 3 Up 4 4 Up 5 5 Up 6 6 Up 7 7 Up 8 8 Up 9 9 Up 10 10 Up 11 11 Up 12 12 Up 13 18 Up 14 17 Up 15 16 Up 16 19 down 17 20 down 18 21 down 19 22 down 20 23 down 21 24 down 22 25 down 23 26 down 24 27 Up 25 28 Up 26 29 Up 27 30 down 28 15 Up 29 14 Up 30 13 Up 31 31 Up 32 32 Up 33 33 Up 34 34 Up 35 35 Up 36 36 Up ports_B External 4x IB ports on switch B BO Bl B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14 Up Up Up Up Up Up Up Up Up Up Up Up Up Up Up ports belonging to server blades on switch B BL6 0 BL641 BL7 0 BL7 1 BL8 0 BL8 1 BL9 0 BL9O 1 BL10 0 BL10 1 BL11 0 BL11 1 Chapter 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module 21 down down down down down down down down down down down down ports on switch B connected to the peer switch 28 29 30 31 32 33 34 35 36 Up Up Up Up Up Up Up Up Up Logical p Phys pit State 1 4 Up 2 5 Up 3 6 Up 4 7 Up 5 8 Up 6 9 Up 7 10 Up 8 11 Up 9 12 Up 10 18 Up 11 17 Up 12 16 Up 13 15 Up 14 14 Up 15 13 Up 16
33. ples 35 InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt Could be 5 0 Gbps 4 16 20 1 ib 112 HCA 1 Could be 5 0 Gbps 4 17 2 Tif ib 113 HCA 1 Could be 5 0 Gbps 4 18 7 1 ib 102 HCA 1 Could be 5 0 Gbps 4 19 84 1 ib 103 HCA 1 Could be 5 0 Gbps 4 20 24 1 ib 104 HCA 1 Could be 5 0 Gbps 4 21 28 1 ib 105 HCA 1 Could be 5 0 Gbps 4 22 9 1 ib 100 HCA 1 Could be 5 0 Gbps 4 23 82 1 ib 101 HCA 1 Could be 5 0 Gbps 4 24 17 1 ib 109 HCA 1 Could be 5 0 Gbps 4 25 18 1 ib 108 HCA 1 Could be 5 0 Gbps 4 26 2 1 ib 116 HCA 1 Could be 5 0 Gbps 4 27 22 1 ib 117 HCA 1 InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt InfiniBand QDR Switched NEM I4A Gbps Active LinkUp gt 3 Coul 10 Coul np Coul L
34. ribution OFED m Mellanox Firmware Tools MFT To Display the Local Host s IB Device Status Use any of the three commands in the example to display the local Host s IB device status ibstat CA mlx4_0 CA type MT26428 Number of ports 1 Firmware version 2 6 0 31 32 Hardware version a0 Node GUID 0x50800200008e4d38 System image GUID 0x50800200008e4d3b Port 1 State Active Physical state LinkUp Base lid 7 Rate 40 LMC 0 SM lid 13 Capability mask 0x02510868 Port GUID 0x50800200008e4d39 ibv devinfo hca id mlx4 0 fw ver 2 6 000 node guid 5080 0200 008e 4d38 sys_image_guid 5080 0200 008e 4d3b vendor_id 0x02c9 vendor_part_id 26428 hw_ver OxAO board id X6275 QDR IB 1 8 phys port cnt 1 port 1 state PORT ACTIVE 4 max mtu 2048 4 active mtu 2048 4 sm lid T3 port lid 7 port lmc 0x00 ibstatus Infiniband device mlx4 0 port 1 status default gid fe80 0000 0000 0000 5080 0200 008e 4d39 base lid 0x7 sm lid Oxd state 4 ACTIVE phys state 5 LinkUp rate 40 Gb sec 4X QDR Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 V To Check the Presence of an IB DONEM Use the ibswitch command to verify that an IB ONEM is installed in the shelf Switch Switch ibswitch 0x002128183ea40050 ports 36 QDR Switched NEM I4B base port 0 lid 4 lmc OCA mlx4 0 0x002128183ea40040 ports 36
35. server module 1 536 Gbps aggregate throughput Version 1 2 10 CXP connectors three 4x ODR DDR SDR IB ports per connector 24 4x IB links 24 1 Gbps passthrough ports Note Two 1 Gbps ports are available on the Sun Blade X6275 server module and one 1 Gbps port is available on the Sun Blade X6270 server module 2 Mellanox Infiniscale IV 14 36 port IB switches See IB ONEM Cables on page 29 One green LED per 4x port showing IB physical link state Locate button and LED white OK LED green Service Action Required LED amber Ready to Remove LED blue Sun proprietary form factor 5 of 6 compliant 240 watts maximum Protected from electrostatic discharge ESD and handling damage Chapter 1 Sun Blade 6048 InfiniBand QDR Switched Network Express Module Introduction 3 Sun Blade 6048 Platform Support TABLE 1 2 lists supported platforms for the IB ONEM in the Sun Blade 6048 Series Modular System TABLE 1 2 Hardware Requirements for the IB ONEM Requirements Hardware Sun Blade Server Module Sun Blade 6275 Server Module Sun Blade 6270 Server Module with Sun InfiniBand Dual Port 4x DDR PCle Fabric Expansion Modules For information about the supported operating systems for a server module go to http www sun com servers blades os jsp IB ONEM Indicators Buttons and Ports The IB QNEM provides a standard set of status indicators buttons and ports on the back panel FIGURE 1 2
36. shows the back panel Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 FIGURE 1 2 IB ONEM Back Panel Indicators Buttons and Ports A Se AN O SS eue 1000 Mbps See TABLE 1 3 for a description of the components Chapter 1 Sun Blade 6048 InfiniBand QDR Switched Network Express Module Introduction 5 TABLE 1 3 Sun Blade 6048 IB ONEM Back Panel Components and Functions Number Component Name Color 1 Locate button White indicator 2 OK indicator Green 3 Service Action Amber Required indicator 4 Ready to Remove Blue indicator 5 Ethernet ports Description The Locate indicator is a white button with an associated indicator that blinks 4 Hz when initiated remotely from the ILOM web interface The Locate indicator enables a system operator to easily locate an IB ONEM in a system within a large data center Once activated this indicator times out after 30 minutes When pressed the Locate button lights the Locate indicator The Locate indicator provides these indications e Fast blink Identifies a specific IB ONEM in the chassis Lights when the LED is initiated from the web interface remotely or from a press of the Locate button locally Off IB ONEM locator function has not been selected Displays the different states of module initialization Provides the following indications Steady o
37. ss Module User s Guide April 2009 3 packets transmitted 3 received 0 packet loss time 2000ms rtt min avg max mdev 0 093 0 948 2 648 1 202 ms root ib 102 Appendix B InfiniBand Command Examples 39 40 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Index blade server interface 3 C cables IB ONEM 29 InfiniBand QDR 29 changing the I O configuration 11 commands examples IB 31 show CH 17 show CH NEM1 17 start CH NEM1 SP cli 19 compliance 9 Component Management page ILOM 15 E EMI emissions regulations 9 F features IB DNEM 3 filler panel installing to prevent overheating 12 H handling instructions 11 hot plug operations supported 3 hot remove request blade does not respond 18 l I O configuration changing 11 IB uplink interface 3 IB QNEM installed graphic view 15 handling instructions 11 ILOM CMM CLI 19 log in 15 ILOM Component Management page 15 Indicators status 4 indicators 4 InfiniBand transfer rate 3 InfiniBand Trade Association IBTA interoperability Version 1 2 supported 3 interface blade server 3 interoperability IBTA Version 1 2 3 M Marketing part number 1 Mellanox InfiniHost IV switch 3 N NEM to blade connections determining 18 O operating system support 4 9 P PCI hot remove request blade does not respond 18 41 platform support 4 9
38. tegrated Lights Out Manager ILOM is the embedded management software on the server module SP and CCM SP that enables you to manage your system For additional information about ILOM refer to the Sun Integrated Lights Out Manager 2 0 User s Guide Sun Blade 6048 InfiniBand QDR Switched Network Express Module Sun Blade X6xxx server module hardware Embedded Service Processor SP on server module and Chassis Monitoring Module CMM Single data rate Double data rate Quad data rate Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 CHAPTER 2 Replacing the Sun Blade 6048 InfiniBand QDR Switched Network Express Module This chapter describes how to replace a Sun Blade 6048 InfiniBand QDR Switched Network Express Module IB ONEM in a powered on Sun Blade 6048 Series Chassis This chapter also includes instructions to verify that the IB QNEM has been installed correctly This chapter contains the following sections m Replacing IB ONEM Hardware on page 11 m Verifying Installation on page 14 m Managing the IB ONEM on page 18 Caution Damage to the IB QNEM can occur as the result of careless handling or electrostatic discharge ESD Always handle an IB QNEM with care to avoid damage to electrostatic sensitive components To minimize the possibility of ESD related damage Sun strongly recommends using both a workstation antistatic mat and an ESD wrist strap You can get an E
39. through Gigabit Ethernet ports for each slot of IB QNEM s respective a Sun Blade 6048 modular system shelf m Each IB ONEM InfiniBand port can provide full duplex data transfers of up to 40 Gbps ODR m Each InfiniBand port is backward compatible with 20 Gbps DDR and 10 Gbps SDR devices m The IB ONEM logically installs inthe NEM 1 slot of a Sun Blade 6048 modular system shelf but occupies both NEM slots due to physical form factor m Up to 12 blade servers can connect to a single IB ONEM Each blade server is connected to the IB ONEM through two 4x ODR IB interfaces and two 1 GbE interface FIGURE 1 1 Sun Blade 6048 InfiniBand QDR Switched Network Express Module 2 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 TABLE 1 1 lists additional features of the Sun IB ONEM TABLE 1 1 Sun IB QNEM Features Feature Hot plug operations supported IB transfer rate maximum InfiniBand Trade Association IBTA interoperability IB uplink interface Blade server interface Ethernet interface IB switch chip IB QDR cables LED indicators Form factor RoHS Power consumption Protection Description Enables system administrators to easily add or remove IB ONEMSs as needed without powering down the system 40 Gbps ODR per 4x IB port for the Sun Blade X6275 server module and 20 Gbps DDR per 4x IB port for the Sun Blade X6270 server module There are two 4x IB ports per
40. ur le site http www sun com patents un ou les plusieurs brevets suppl mentaires ainsi que les demandes de brevet en attente aux les tats Unis et dans d autres pays Ce document et le produit auquel il se rapporte sont prot g s par un copyright et distribu s sous licences celles ci en restreignent l utilisation la copie la distribution etla d compilation Aucune partie de ce produit ou document ne peut tre reproduite sous aucune forme par quelque moyen que ce soit sans l autorisation pr alable et crite de Sun et de ses bailleurs de licence s il y en a Tout logiciel tiers sa technologie relative aux polices de caract res comprise est prot g par un copyright et licenci par des fournisseurs de Sun Des parties de ce produit peuvent d river des syst mes Berkeley BSD licenci s par l Universit de Californie UNIX est une marque d pos e aux Etats Unis et dans d autres pays licenci e exclusivement par X Open Company Ltd Sun Sun Microsystems le logo Sun Sun Blade Java docs sun com et Solaris sont des marques de fabrique ou des marques d pos es de Sun Microsystems Inc ou ses filiales aux tats Unis et dans d autres pays L interface utilisateur graphique OPEN LOOK et Sun a t d velopp e par Sun Microsystems Inc pour ses utilisateurs et licenci s Sun reconnait les efforts de pionniers de Xerox dans la recherche et le developpement du concept des interfaces utilisateur visuelles ou graphiques p
41. wnloaded firmware bundle These instruction are provided as an example 1 Download the firmware from the Sun Download Ceneter SDLC 2 Obtain the LIDs from the fabric for the Infiniscale IV devices ibswitches Switch 0x002128183ea40050 ports 36 Sun Blade 6048 InfiniBand QDR Switched NEM I4B base port 0 lid 4 l1mc 0 Switch 0x002128183ea40040 ports 36 Sun Blade 6048 InfiniBand QDR Switched NEM I4A base port 0 lid 3 Imc 0 Example shows that I4B s LID is 4 and I4A s LID is 3 3 Use flint tool to update the switch firmware on both switches flint d lid 3 i QNEM 04 I4A 1 9 BIN b Current FW version on flash 7 1 8 New FW version 7 1 80 Note The new FW version is not newer than the current FW version on flash Do you want to continue y n n y 26 Sun Blade 6048 InfiniBand QDR Switched Network Express Module User s Guide April 2009 Burning first FW image without signatures OK Restoring first signature OK Burning primary image OK Verifying primary image OK flint d lid 4 i QNEM 04 I4B 1 9 BIN b Current FW version on flash 7 1 8 New FW version 7 1 80 Note The new FW version is not newer than the current FW version on flash Do you want to continue y n n y Burning first FW image without signatures OK Restoring first signature OK Burning primary image OK Verifying primary image OK 4 Reset the switches I
Download Pdf Manuals
Related Search