Home

BX900/BX400 InfiniBand Modules V5.0 Connection Blade and

Contents

1. ee eee 79 4 User Manual BX900 BX400 InfiniBand Modules 40Gb 1 Introduction This manual describes the configuration installation and basic use of the InfiniBand connection blade and Mezzanine card software The package includes tools for general and advanced management functionality It also describes the installation of OFED OpenFabrics Enterprise Distribution for Linux the identification of the Mezzanine card and Win OpenFabrics instal lation for Windows The hardware installation of the InfiniBand connection blade is described in the BX900 S1 or BX400 S1 System Unit Operation manual The hardware instal lation of the Mezzanine card is described in the Service Supplements or Upgrade and Maintenance Manual of the BX92x S3 server blades 1 1 Notational Conventions The following notational conventions are used in this manual Caution This symbol points out hazards that can lead to personal injury loss of data or damage to equipment i This symbol highlights important information and tips This symbol refers to a step that you must carry out in order to continue with the procedure italic Commands menu items names of buttons options file names and path names are written in italic letters in the text variable Marked variables that must be replaced by current values fixed font System output is written using a fixed font semi bold Co
2. User Manual English FUJITSU PRIMERGY 900 BX400 InfiniBand Modules 40Gb Connection Blade and Mezzanine Card V7 0 Linux Windows Edition October 2012 Comments Suggestions Corrections The User Documentation Department would like to know your opinion of this manual Your feedback helps us optimize our documentation to suit your individual needs Feel free to send us your comments by e mail to manuals ts fujitsu com Certified documentation according to DIN EN ISO 9001 2008 To ensure a consistently high quality standard and user friendliness this documentation was created to meet the regulations of a quality management system which complies with the requirements of the standard DIN EN ISO 9001 2008 cognitas Gesellschaft f r Technik Dokumentation mbH www cognitas de Copyright and Trademarks Copyright Fujitsu Technology Solutions GmbH 2012 All rights reserved Delivery subject to availability right of technical modifications reserved All hardware and software names used are trademarks of their respective manufacturers Contents 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 8 1 1 8 2 1 9 1 9 1 1 9 2 2 2 1 2 2 2 3 2 3 1 2 3 2 2 3 3 2 3 4 2 4 2 4 1 2 4 2 2 4 3 2 5 2 6 2 6 1 Y 5 Notational Conventions 5 Tager GOUD idus ook amp ux Gade
3. 1 5 PRIMERGY BX900 The PRIMERGY BX900 Blade Server system is a modular server system that integrates up to 18 server modules eight connection blade modules and two Management Modules MMB The InfiniBand connection blade module provides networking and Switch functions to PRIMERGY BX900 Blade Server system The Management Module offers a single point of control for the PRIMERGY BX900 Blade Server system Figure 1 PRIMERGY BX900 Blade Server 8 User Manual BX900 BX400 InfiniBand Modules 40Gb Introduction Inserting Connection Blade into PRIMERGY BX900 1 6 Inserting Connection Blade into PRIMERGY BX900 In order to support 4x QDR InfiniBand 4 lanes per one port are required in backplane Due to this reason the form factor of connection blade is double wide single height The eject lever of the connection blade has a green touch button The places at which all components may be touched to prevent them from being damaged when they are assembled or disassembled are also marked green Rear view of enclosure Fabric 1 Fabric 2 Fabric 3 Fabric 4 Figure 2 Rear view of enclosure BX900 BX400 InfiniBand Modules 40Gb User Manual 9 Connecting Mezzanine Cards on Server Blade Introduction 1 7 Connecting Mezzanine Cards on Server Blade The figure below indicates the connection relation between connection blade CB and Mezzanine Card on server blade InfiniBand IB Mezzanine Cards can be s
4. 7 Mellanox IPoIB Adapter 2 Figure 27 Network status 14 Since no IP address is assigned to this new IPoIB adapter assign an IP address to it 15 Run the command prompt as the Administrator privilege and execute the following command to enable Network Direct gt ndinstall i Administrator Command Prompt L ninis tor ndinstall i Installing IBAL provider s rovider 01 2 Provider 0000001010 ice Provider 0000001011 OpenFabrics Netw ect Provider Figure 28 Installation of Network Direct 1 48 User Manual 9 400 InfiniBand Modules 40Gb Windows Installation Installation of WinOF 16 In order to check if Network Direct is enabled properly execute the same command again The message Installing IBAL provider already installed is shown if it is enabled properly ministrator Command Prompt inistrator ndinstall i Installing IBAL provider already installed 900010 FD Tepip 00 Te MSAFD dminist rator Figure 29 Installation of Network Direct 2 BX900 BX400 InfiniBand Modules 40Gb User Manual 49 Installation of WinOF Windows Installation 50 User Manual BX900 BX400 InfiniBand Modules 40Gb 4 Connection Blade Management 4 1 Connection In this chapter the connection of IB connection blade via Management Blade is explained Follow the procedures below Configurations for connecting to IB connection blade are explained in sec
5. 01 HCA 1 02 HCA 1 Mellanox release mlnx405ex 1 0 2 build 2009 07 19 BX900 BX400 InfiniBand Modules 40Gb User Manual 33 Network Configuration Information Linux Installation root bx900 02 01 ibnetdiscover node name map root mmap txt Topology file generated on Sat Aug 1 01 27 33 2009 Max of 3 hops discovered Initiated from node 0002c903000413c0 port 0002c903000413c2 vendid 0x2c9 devid 0xbd36 sysimgguid 0x2c9020040c7a8 switchguid 0x2c9020040c7a8 2c9020040c7a8 Switch 36 S bx900 02 IB SW CB3 enhanced port 0 0 23KC o002c9020040c790 23 bx900 04 IB SW CB3 lid 9 4xODR 22 5 0002 9020040 790 22 8 bx900 04 IB 5W CB3 lid 9 4xODR 21 5 0002 9020040 790 21 bx900 04 IB SW CB3 lid 9 4xODR 20 5 0002c9020040c790 20 bx900 04 IB SW CB3 lid 9 4xODR 19 5 0002c9020040c790 19 bx900 04 IB SW CB32 lid 9 4xQDR 18 H 0002c9030003bf82 2 2c9030003bf84 ft bx900 02 18 1 lid 7 4xQDR 17 H 0002c903000413fc 2 2c903000413fe bx900 02 17 HCA 1 lid 22 4xQDR 16 H 0002c903000412fc 2 2c903000412fe bx900 02 16 HCA 1 lid 21 4xQDR 15 0002 90300041460 2 2 90300041462 8 bx900 02 15 HCA 1 lid 5 4xODR 14 H 0002c9030003bf96 2 2c9030003bf98 bx900 02 14 HCA 1 lid 11 4xQDR 13 H 0002c9030003d138 2 2c9030003d13a bx900 02 13 lid 4 4xODR 12 H 0002c9030003bf92 2 2c9
6. 6 Documentation Overview 6 ke ee 7 PRIMERGY BX900 22 8 Sa xx eee Se SR ORES 8 Inserting Connection Blade into PRIMERGY BX900 9 Connecting Mezzanine Cards on Server Blade 10 Description of InfiniBand Connection Blade 11 11 ED d eb edendo e 12 Technical Data 12 IB Connection llle 12 IB Mezzanine Card 13 Linux Installation 1 2 et 15 15 Software Requirements 16 Native InfiniBand Installation RHEL 17 Setup InfiniBand during Linux Installation 17 Loading InfiniBand Drivers at System Startup 18 Installation Results s sx 644464482 3 RR 3 19 Post installation Notes 19 Installation by means of SVIM 20 Inthe event of RHEL5 4 0 00005 ee 20 Inthe event of RHEL6 0 2 000 24 Post installation Notes 25 Updating Firmware after Installation 27 Subnet Manager 27 EOIN LL uou xk ose aUe bob he 27 BX900 BX400 InfiniBand Modules 40Gb User Manual 3 Contents 2 6 2 Confirmation of OpenSM running 28
7. E Device Manager File Action View Help WIN P85KACABF7C 9 88 Computer Disk drives Display adapters 05 Human Interface Devices lt 2 Keyboards n Mice and other pointing devices Monitors H E Network adapters Other devices Qa Infiniband Controller Ports amp LPT D Processors lt gt Storage controllers System devices System management devices Universal Serial Bus controllers m b G4 eee Figure 18 Display of InfiniBand devices before installation of WinOF 2 Log on to Windows as administrator and then double click the file of WinOF in ServerView Suite DVD1 D DRV InfiniBand Mellanox MT26428 x64W2K8Ra 3 Click Next gt after the following window appears BX900 BX400 InfiniBand Modules 40Gb User Manual 39 Installation of WinOF Windows Installation LNX_VPI InstallShield Wizard Welcome to the InstallShield Wizard for MLNX VPI The InstallShield R Wizard will install MLNX_VPI on your computer To continue dick Next WARNING This program is protected by copyright law and international treaties Figure 19 InstallShield Wizard 40 User Manual 9 400 InfiniBand Modules 40Gb Windows Installation Installation of WinOF 4 After the following window appears read the License Agreement carefully If you accept click accept the terms in the licens
8. 2 6 3 Confirmation of the master SM 28 2 7 Network Configuration Information 28 2 7 1 Network configuration diagram 29 2 7 2 Configuration sheet 1 29 2 7 9 The result of the command ibnetdiscover 32 2 8 Identifying the 35 3 Windows Installation 37 3 1 OvervieoW 6 bcc eb Re 37 3 1 1 Software 37 3 2 Installation of WinOF 38 4 IB Connection Blade Management 51 4 1 Connection sa sacosa EROR 51 4 1 1 Telnet cornnectlOH E 52 4 1 2 Save and restore the configuration 52 4 2 CONNGUIGUON 245 RU OR EUR qo as 54 4 2 1 Configuration of LAN interface 54 4 2 2 Ethernet Connection via Management Blade 57 4 3 Basic Operation ss 58 4 3 1 How to check System Information 58 4 3 2 SNMP Trap Settings 2 0 42 ce dd eee Ree we 59 44 Users and Privileges 60 5 Troubleshooting Hees 61 5 11 Check the Log File of OpenSM Linux only 61 5 2 IB Mezzanine Card eee 64 5 3 IB Connection Blade 76 Nder
9. gpgkey file media RHEL_6 0 x86_64 Disc 1 RPM GPG KEY redhat release Execute the following command on the terminal the repository name is dvd root localhost yum clean all Croot localhost yum disablerepo enablerepo dvd makecache root localhost yum disablerepo enablerepo dvd groupinstall Infiniband OFED is installed 2 4 3 Post installation Notes This section describes how to set mix4 parameters Add a line to a file If you set to RHEL6 1 MellanoxOFED please add the line to the etc modprobe d mIx4 conf If there is no mix4 conf file please create it How to set mix4 parameters options m1x4 core parameter lt value gt BX900 BX400 InfiniBand Modules 40Gb User Manual 25 Installation by means of SVIM Linux Installation How to change MR log_num_mtt Log maximum number of memory translation table segments per HCA default is 20 max is 20 log_mtts_per_seg Log number of MTT entries per segment default is 3 max is 7 The following table schows the recommended values of og_num_mtt and log_num_per_seg The values change with the main memory capacity main memory 16 GB 32 GB 64 GB capacity log_num_mtt 18 19 20 log_mtts_per_seg 4 4 4 After the parameter has been changed you will need to reboot How to change QP number of HCA log num qp Log maximum number of number per default is 18
10. please consult 27 Couldn t restore Mezzanine card PCI the system engineer Express Link control register aborting 28 Couldn t restore Mezzanine card reg x aborting 29 Couldn t restore Mezzanine card Table 6 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux note1 When a message occurs frequently confirm the following No Confirmation contents Judgment To do 1 Is the IB Mezzanine card YES mounted rigidly The IB Mezzanine card must be replaced When the message is displayed after the IB Mezzanine cards are exchanged itis necessary to replace the server blade NO Place the IB Mezzanine card again 68 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card Work after IB Mezzanine card has been replaced Linux 1 The server blade is mounted on the main body of BX900 and the power supply is turned on 2 Linux is started sbin Ispci command is executed by the root account and it is confirmed that the IB Mezzanine card is recognized Execution result 10 00 0 InfiniBand Mellanox Technologies MT26428 ConnectX IB QDR PCIe 2 0 5GT s rev a0 Subsystem Mellanox Technologies MT26428 ConnectX IB QDR PCIe 2 0 5GT s 30 00 0 InfiniBand Mellanox Technologies MT26428 LConnectX IB QDR PCIe 2 0 5GT s rev a0 Subsystem Mellanox Technologies 26428 ConnectX IB QDR PCIe 2 0 5061 51 The enhancing
11. LID information in the error messages and the output of ibnetdiscover For example Jul 30 23 34 18 600287 4CDOD940 0x02 gt osm_drop_mgr_remove_port Removed port with GUID 0x0002c90300041306 LID range 2 2 of node localhost HCA 2 In this case Error Card has the GUID 0x0002c90300041306 and the LID is 2 of Mezzanine Card IB connection blade You need to change the relevant IB connection blade You will get the relevant IB connection blade from GUID or LID information of the error messages and output of ibnetdiscover For example Jul 30 23 34 18 600424 4CDOD940 0x02 gt osm_drop_mgr_remove_port Removed port with GUID 0x0002c9020040c790 LID range 21 21 of node none SW 1 In this case Error IB connection blade has the GUID 0x0002c9020040c790 and the LID is 21 62 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting Check the Log File of OpenSM Linux only For details of how to specify the relevant IB connection blade refer to section The result of the command ibnetdiscover on page 32 3 IB cable If you have changed the above IB connection blade and you still receive the messages all IB cables connected to the IB connection blade need to be changed Reporting Generic Notice type 1 num 128 Link state change from LID LID number GID GID number the port of IB connection blade was disconnected in the subnet How to solve the problem The following changes are
12. Set value is the value of the powers of two Example 18 default 26 2144 256K 21 209 7152 2M Example etc modprobe d mlx4 conf options mIlx4 core log mtt 20 log mtts per seg 4 log num qp 18 After the parameter has been changed you will need to reboot IPoIB adapters parameter limitation If you create edit ifcfg ib files to configure an IPoIB the parameter NM CONTROLLED should be set to The limitation is applied to RHEL6 only Example etc sysconfig network scripts ifcfg ibO DEVICE ibO IPADDR 192 168 210 100 NETMASK 255 255 255 0 26 User Manual X BX900 BX400 InfiniBand Modules 40Gb Linux Installation Updating Firmware after Installation BROADCAST 192 168 210 255 NM_CONTROLLED no NOBOOT yes Since there is a problem peculiar to RHEL6 1 communication by IPoIB may not be able to be performed Please add the following parameters PREF IX lt value gt Set to the value corresponding to the NETMASK Example NETMASK 255 255 255 0 PREFIX 24 2 5 Updating Firmware after Installation If you wish to burn newer firmware you have to download it from Fujitsu Technologies Web site Attp support ts fujitsu com com support downloads html gt Driver amp Downloads If you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads 2 6 Subnet Manager At least one Subnet Manager SM is present on each
13. TX packets 12 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 1121747 1 0 MiB TX bytes 601 601 0 b Interrupt 25 56 User Manual BX900 BX400 InfiniBand Modules 40Gb IB Connection Blade Management Configuration 4 2 2 Ethernet Connection via Management Blade IB connection blade provides the function of Ethernet connection via the Management LAN port on a Management Blade Ethernet connection enables the user to connect to IB connection blade via telnet ssh with CLI An example that an IP address 192 168 1 100 24 is assigned to management console and 192 168 1 10 24 to IB connection blade is illustrated below The back face BX900 51 IP address 192 168 1 10124 manage ment ij console L 3 8 e o 00O D 83 gt i i e aS OOOO E OHNE aa example 900 chassis Figure 33 Ethernet connection BX900 BX400 InfiniBand Modules 40Gb User Manual 57 Basic Operation IB Connection Blade Management 4 3 Basic Operation This chapter describes the basic operation by the commands and how to get the information for the maintenance 4 3 1 Howto check System Information This section describes how to check system information for this product The firmware version can be checked for this product swinfo command This command shows GUID InfiniScalelV Firmware version CPU Firmw
14. back of the IB Mezzanine Card BX900 BX400 InfiniBand Modules 40Gb User Manual 35 Identifying the GUID Linux Installation How to identify the node GUID gt Log in to the server blade as root gt Run the open terminal or telnet ssh gt Execute the ibstat command and identify the node GUID Node GUID Ox The ibstat command is available after installation of OFED Stack for Linux The node GUID is 0002c903000413b8 in the table below root8bx900 02 03 1 ibstat CA mlx4 0 CA type 26428 Number of ports 2 Firmware version 2 6 922 Hardware version a0 Node GUID 0x0002c903000413b8 System image GUID 0x0002c903000413bb Port 1 State Down Physical state Polling Rate 10 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02510868 Port GUID 0x0002c903000413b9 Port 2 State Active Physical state LinkUp Rate 40 Base lid 71 LMC 0 SM lid 1 Capability mask 0x02510868 Port GUID 0x0002c903000413ba CA 1 4_1 CA type 26428 Number of ports 2 Firmware version 2 6 922 Hardware version a0 Node GUID 0x0002c903000446f0 System image GUID 0x0002c903000446f3 Port 1 State Down Physical state Polling Rate 10 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02510868 Port GUID 0x0002c903000446f1 Port 2 State Active Physical state LinkUp Rate 40 Base lid 91 LMC 0 SM lid 1 Capability mask 0x02510868 Port GUID 0x0002c9030
15. blade 2 Host interface x8 PCI Express Gen2 Interface to midplane 2 4x10Gbps QDR Consider restrictions mentioned in section Inserting Connection Blade into PRIMERGY 900 on page 9 and section Connecting Mezzanine Cards on Server Blade on page 10 BX900 BX400 InfiniBand Modules 40Gb User Manual 13 Technical Data Introduction 14 User Manual BX900 BX400 InfiniBand Modules 40Gb 2 Linux Installation 2 1 Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed A blade can be properly installed with all required InfiniBand drivers and software during Red Hat Enterprise Linux installation If you purchased the products from Fujitsu Japan you can download it from http jp fujitsu com platform server primergy downloads Please note that only Red Hat Enterprise Linux Version 5 4 or higher is supported The chapter includes the following sections Identifying the GUID on page 35 e Native InfiniBand Installation in RHEL on page 17 OFED Installation by means of SVIM on page 20 e Updating Firmware after Installation on page 27 e Subnet Manager on page 27 e Network Configuration Information on page 28 BX900 BX400 InfiniBand Modules 40Gb User Manual 15 Software Requirements Linux Installation 2 2 Software Requirements Required Disk Space for Linux Installation 400 MB Operating System Linux operating syst
16. board slot where this card is mounted be confirmed by the value of Bus 10 00 0 Enhancing board slot 1 30 00 0 Enhancing board slot 2 3 Refer to section Network configuration diagram on page 29 update a GUID table BX900 BX400 InfiniBand Modules 40Gb User Manual 69 IB Mezzanine Card Troubleshooting IB Mezzanine card driver messages Windows Server 2008 R2 Confirm the message of the IB Mezzanine card driver by referring to the event log You refer to the event log using the Event Viewer Event ID Classification To do Error Level 0x0041 Failed to open Error Channel Adapter 0x0042 xxx Failed to allocate Error Protection Domain 0x0043 xxx Failed to create Error receive Completion Queue 0x0044 xxx Failed to create send Error Completion Queue 0x0045 xxx Failed to create Error Queue Pair The IB Mezzanine card must be 0x0046 xxx Failed to get Queue 8 eplaced Error Pair number 0x0047 xxx Failed to create DMA dec Error Memory Region technical support ts fujitsu com 0x0048 xxx Failed to create If you purchased the products from Error receive descriptor pool Fujitsu Japan please consult the 0x0049 xxx Failed to create system engineer Error 5 PACKET pool to receive indications 0 004 xxx Failed to create Error NDIS_BUFFER pool to receive indications 0x004B xxx Failed to create Error NDIS_PACKET pool t
17. desired tune the value unlimited to a specific amount of RAM Subnet Manager required For your machine to be part of the InfiniBand fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM see Subnet Manager on page 27 IPoIB function The IPoIB function shouldn t be set to the IB port which is the port1 on the slot1 The limitation is applied to RHEL5 and RHEL6 IPoIB adapters parameter limitation In case the IPoIB adapters are set by using ifcfg ib files the parameter NM CONTROLLED should be set to The limitation is applied only to RHEL6 24 OFED Installation by means of SVIM The installation of OFED gets decided on the package selection in ServerView Installation manager SVIM When all packages were selected OFED is installed within OS installation When the default package groups were selected OFED is not installed Please install OFED by the following steps after the OS installation 241 Inthe event of RHEL5 4 This chapter describes the installation of OFED for Red Hat Enterprise Linux Version 5 4 1 Insert OS DVD in the DVD drive and make sure of the mount of DVD 20 User Manual 9 400 InfiniBand Modules 40Gb Linux Installation OFED Installation by means of SVIM 2 6 File Edit View Places
18. driven firmware commands aborting The IB Mezzanine card must 13 NOP command failed to generate interrupt 2 oe COSME IRQ 96d aborting technical support ts fujitsu com 14 Failed to initialize completion queue table lf you purchased the products aborting from Fujitsu Japan please 15 Failed to initialize shared receive queue consult the system engineer table aborting 16 Failed to initialize queue pair table aborting 17 Failed to initialize multicast group table aborting 18 Failed to reset Mezzanine card aborting 19 Failed to init command interface aborting 20 Only UAR pages need more than 128 note1 21 Profile requires Ox llx bytes won t fit in note1 Ox llx bytes of context memory Table 6 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux BX900 BX400 InfiniBand Modules 40Gb User Manual 67 IB Mezzanine Card Troubleshooting COMMAND aborting No Classification To do 22 Couldn t allocate memory to save Mezzanine card PCI header aborting 23 Couldn t save Mezzanine card PCI header aborting 24 Couldn t map Mezzanine card reset register aborting The IB Mezzanine card must be 25 PCI device did not come back after reset replaced Please consult aborting technical support ts fujitsu com 26 Couldn t restore Mezzanine card If you purchased the products Express Device Control register aborting from Fujitsu Japan
19. required Please contact the repair center 1 IB cable You need to change the IB cable You specify the IB cables by the following procedure You specify the relevant IB connection blade from the LID information in the error messages For example Jul 31 22 56 46 264171 42E93940 Ox02 gt osm report notice Reporting Generic Notice type 1 num 128 Link state change from LID 32 GID fe80 2 c902 40 c790 In this case LID 32 s IB connection blade is corresponding For details of how to specify the relevant IB connection blade refer to section The result of the command ibnet discover on page 32 The target port is the one that removed the port number from original ibnetdiscover command output when the system was set up normally 2 connection blade If you have changed the above IB cable and still receive the same messages you need to change the IB connection blade BX900 BX400 InfiniBand Modules 40Gb User Manual 63 IB Mezzanine Troubleshooting 5 2 IB Mezzanine Card You can confirm place and time the error occured by referring to the following items on a server blade No Confirmation item 1 Active link LED for IB Mezzanine card on the front side of the blade 2 IB Mezzanine card driver message Active link LED for IB Mezzanine card When an active link LED for IB Mezzanine card on the front side of the server blade goes dark you can confirm the place wh
20. subnet Each SM resides on a port of IB Mezzanine card or IB connection blade When there are multiple SMs on a subnet the master SM is one and the remaining SMs become standby SMs The master SM has a role to initialize and configure an Infiniband subnet OpenSM is an Infiniband compliant SM The following sections describe how to use OpenSM 2 6 4 OpenSM OpenSM is one Subnet Manager distributed with RedHat BX900 BX400 InfiniBand Modules 40Gb User Manual 27 Network Configuration Information Linux Installation This chapter describes how to configure which server becomes the master SM Please set up the server which will be the master using the following reference Regarding the other settings of OpenSM please refer to the command help of opensmd Basically a server with the lowest GUID will be the master SM within a subnet if you intend to choose the master SM you can configure the priority which range is from 0 to 15 0 is the lowest and 15 is the highest Example for RHEL5 etc init d opensmd start 15 2 6 2 Confirmation of OpenSM running Enter the following command to confirm that OpenSM is running etc init d opensmd status RHEL5 etc init d opensm status RHEL6 OpenSM is running properly if the result of this command displays running enter the following command to run OpenSM if the result displays stopped etc init d opensmd start RHEL5 etc init d opensm start RHEL6 2 6
21. 0 Slo 11 Slo 12 Slo 13 Slo 14 Slo 15 Slo 16 Slo 17 Slo 18 CB3 GUID host name Static DHCP 5 MMB1 Agent MMB1 Service Table 3 Configuration sheet IP Address Root password How to fill in XX Please fill in the GUID of the Mezzanine Card installed in Mezzanine slot in the column of mezz 1 GUID Similarly please fill in the GUID of the Mezzanine Card installed in Mezzanine II slot in the column of mezz 2 GUID Please fill in the host name in the column of hostname when you assign the host name to each server blade Please fill in the IP addresses XX in the column of ethO eth1 eth2 eth3 ibO ib1 162 and ib3 when you use the static IP IP Address 192 168 11 XX 30 User Manual BX900 BX400 InfiniBand Modules 40Gb Linux Installation Network Configuration Information dle of description mezz 1 GUID mezz 2 GUID Hostname Static iRMC ethO eth2 Bima ib2 3 DHCP 0002c9030003bf7a 0002c9030003bf6 RHEL5 4 bx920 02 01 static 101 102 103 104 105 0002 903000413 0 0002 90300041514 RHEL5 4 bx920 02 02 static 106 107 108 109 110 Static DHCP hostname Figure 14 Exampl
22. 00446f2 root8bx900 02 03 I 36 User Manual BX900 BX400 InfiniBand Modules 40Gb 3 Windows Installation 3 1 Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed A server can be properly installed with all required InfiniBand drivers and software during Windows HPC Server 2008 R2 installation or afterwards by using the Win OpenFabrics installation or OpenSM setup Please note that only Windows Server 2008 R2 and Windows HPC Server 2008 R2 are supported 3 1 1 Software Requirements Required Disk Space for Windows Installation 100 MB Operating System Windows HPC Server 2008 R2 Installer Privileges Installation requires administrator privileges on the target machine BX900 BX400 InfiniBand Modules 40Gb User Manual 37 Installation of WinOF Windows Installation e reach best performance the Enhanced Idle Power State option in the BIOS setup Advanced menu must be disabled BIOS SETUP UTILITY CPU Mismatch Detection Enabled IL Figure 17 BIOS Setup Utility 3 2 Installation of WinOF When WinOF was not installed by Server View Installation Manager SVIM please follow the following steps If WinOF was installed please jump to step 10 38 User Manual BX900 BX400 InfiniBand Modules 40Gb Windows Installation Installation of WinOF 1 Make sure that the InfiniBand device is shown in the Device Manager
23. 030003bf94 bx900 02 12 HCA 1 lid 10 4xQDR 11 H 0002c9030003bfba 2 2c9030003bfbc bx900 02 11 HCA 1 lid 17 4xQDR 10 H 0002c903000413e8 2 2c903000413ea bx900 02 10 HCA 1 lid 18 4xQDR 9 Hi 0002c9030003bfa6 2 2c9030003bfa8 bx900 02 09 lid 13 4xQDR 8 H 0002c9030003bf86 2 2c9030003bf88 bx900 02 08 HCA 1 lid 8 4xQDR 7 H 0002c9030003bf9e 2 2c9030003bfa0 bx900 02 07 HCA 1 lid 12 4xQDR 6 0002 lt 903000412 2 2 903000412 bx900 02 06 HCA 1 lid 19 4xODR 5 H 0002c90300041414 2 2c90300041416 bx900 02 05 HCA 1 lid 3 4xODR 4 0002 90300041408 2 2 9030004140 bx900 02 04 2 lid 2 4xQDR 3 _ H 0002c903000413b8 2 2c903000413ba bx900 02 03 1 lid 16 4xQDR 2 WH 0002c9030003bF7a 2 2c9030003bf7c bx900 02 02 HCA 1 lid 6 4xQDR 1 1 0002c903000413c0 2 2c903000413c2 bx900 02 01 HCA 1 lid 1 4xQDR vendid 0x2c9 devid 0xbd36 sysimgguid 0x2c9020040c790 Switchguid 0x2c9020040c790 2c9020040c790 Switch 36 5 0002c9020040c790 bx900 04 IB 5W CB3 enhanced port 0 lid 9 Imc 0 2 H 0002 903000447b0 2 2c903000447b2 bx900 04 02 HCA 1 lid 15 4xQDR 1 H 0002c903000413ec 2 20903000413ee 4 bx900 04 01 HCA 1 lid 20 4xQDR 23 S 0002c9020040c7a8 23 bx900 02 IB SW CB3 lid 14 4xQDR 22 S 0002c9020040c7a8 22 bx900 02 IB SW CB3 lid 14 4xQDR 21 S 0002c9020040c7a8 21 bx900 02 IB SW
24. 3 Confirmation of the master SM Enter the ibstat command on the system server then Base LID and SM LID are displayed The IB Mezzanine card or IB connection blade on which these two LIDs are the same is the master SM 2 7 Network Configuration Information The following information 1 6 is needed to maintain and troubleshoot an InfiniBand network The information should be saved when configuring the InfiniBand network 1 Network configuration diagram 28 User Manual BX900 BX400 InfiniBand Modules 40Gb Linux Installation Network Configuration Information 2 7 1 Example Configuration sheet The server on which the MASTER is found The server on which the STANDBY is found The result of the ibnetdiscover command Network configuration diagram GUID list and arrangement drawing of device location plan BX900 54 900 1 IB cable x5 Figure 13 Network configuration diagram 2 7 2 Configuration sheet Server 22 mezz 5 host Static iRMC eth1 eth2 eth3 160 ib1 ib2 ib3 blade 1 GUID 2 GUID name DHCP Slot1 Slot2 Slot3 Slot4 Slot5 Slot6 Slot7 Slot8 Table 3 Configuration sheet BX900 BX400 InfiniBand Modules 40Gb User Manual 29 Network Configuration Information Linux Installation Slo Slo 1
25. 40Gb User Manual 65 IB Mezzanine Card Troubleshooting IB Mezzanine card driver messages Linux Confirm the error message of IB Mezzanine card driver by referring to the var log messages og file You refer to the log file using the text editor etc No Classification To do 1 Installed FW has unsupported command interface revision Installed FW version is d d 03d The IB Mezzanine card must This driver version supports only revisions be replaced Please consult to d technical support ts fujitsu com 2 Uis version supports only revisions If you purchased the products ed to E from Fujitsu Japan please 3 Internal error detected 02 08x consult the system engineer Failed to initialize user access region table aborting Couldn t map EQ doorbell for EQN 0x 06x Couldn t map interrupt clear register aborting note1 7 Couldn t allocate FW area aborting Failed to map MCG context memory aborting Table 6 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux 66 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card No Classification To do 9 Failed to allocate driver access region aborting 10 Failed to initialize memory region table aborting 11 Failed to initialize event queue table aborting 12 Failed to switch to event
26. 900 BX400 InfiniBand Modules 40Gb User Manual 17 Native InfiniBand Installation RHEL Linux Installation RED HAT ENTERPRISE LINUX 5 Desktop Environments Applications Base Development Servers Dialup Networking Support Java Legacy Software Support System Tools X Window System Cluster Storage apop e Languages 13 of 14 optional packages selected Optional packages Notes 4 Back Next Figure 8 Red Hat package group selection Step 3 To install OFED select the Administrator Tools and press the Optional packages button while the OFED list item is selected and mark the All packages entry in the new window Step 4 Continue the installation process as normal 2 3 2 Loading InfiniBand Drivers at System Startup The complete OFED driver stack can be loaded automatically at system startup Alternatively only the Core modules will run The openibd script residing in etc init d is responsible for this task It can be configured by editing the etc infiniband openib conf To execute this script at startup it is added as a service at the appropriate run level and activated host1 chkconfig add openibd host1 chkconfig openibd on After rebooting the system all important InfiniBand modules will be loaded into the kernel 18 User Manual 9 400 InfiniBand Modules 40Gb Linux Installation Native InfiniBand Installation in RHEL 2 3 3 Install
27. CB3 lid 14 4xQDR 20 S 0002c9020040c7a8 20 bx900 02 IB SW CB3 lid 14 4xQDR 19 S 0002c9020040 7a8 19 bx900 02 IB SW CB3 lid 14 4xQDR Figure 15 ibnetdiscover command GUID list and arrangement drawing of devices 34 User Manual X BX900 BX400 InfiniBand Modules 40Gb Linux Installation Identifying the GUID ibnetdiscover format 1 Provide the GUID list file mmap txt in advance 2 Search for the GUID obtained by the command swinfo in the result of ibnetdiscover S indicates IB connection blade see a 3 There is a string lid 14 in the same line as the GUID referred above 14 indicates the LID of IB connection blade see 2 4 The leftmost figure in each line 23 to 1 indicate the port number which is connected to a device The maximum number of the port is 36 Thus if all ports are connected 1 to 36 is displayed In this example there are connections as follows see c 5 4x QDR indicates the Quad Data Rate connection In case this value is SDR or DDR or QDR there is something wrong with the cable connection see GUID list of IB connection blade the layout chart and rack loading list is useful for the maintenance 2 8 Identifying the GUID The IB Mezzanine card has 4 different GUIDs Globally Unique Identifier Node GUID System GUID GUID for Port1 and GUID for Port2 The Node GUID is listed on the label on the back of the product Figure 16 Label on the
28. Help Cluster ClusterStorage images isolinux Server VT END U END U DEI H RED H The f compi conti eula en_US EULA GPL m T 2 CDROM v 62 items Free space 0 bytes Figure 9 0 DVD It is assumed that the mount position of the DVD is media CDROM in the following procedures 2 Check the directories in the OS DVD which include the rpm files In the following example the rpm files are included in Cluster ClusterStorage Server and VT directories 3 Run Package Manager and click Edit Repositories BX900 BX400 InfiniBand Modules 40Gb User Manual 21 Installation by means of SVIM Linux Installation i Package Manager Edit View Help All packages Installedpackages Ayailable packages 1 a2ps 4 13b 57 2 e15 x86_64 Converts text and other types of files to PostScript TM 1 aci 2 2 39 6 e15 x86 64 Access control list utilities Z acpid 1 0 4 9 el5 4 2 x86 64 Event Daemon adjtimex 1 20 2 1 x86 64 A utility for adjusting kernel time variables 1 agg 2 4 2 1 x86 64 Anti Grain Geometry agg 2 4 2 1 1386 Anti Grain Geometry alde 0 13 1 6 e15 x86 64 Intrusion detection environment 1 alacarte 0 10 0 1 fc6 noarch Simple menu editor for GNOME alchemist 1 0 36 2 e15 x86 64 A multi sourced configuration back end GI I 8 6666666 6 6 lt Package Details Figure 10 Package Manager 4 Select A
29. Remote Desktop Remote De Started Started Started Started Startup Manual Manual Manual Manual Automatic Manual Manual Manual Automatic Automatic Automatic Automatic Automatic Manual Manual Manual Manual Manual Manual X Extended Standard 7 On As a Local System Local System Local System Local Service Local System Network S Local System Local Service Network S Local Service Local System Local Service Local Service Local System Local System Local System Local System Local System Local System Local System Local System Local System Local Service Local System zl Figure 26 Configuration of OpenSM 11 Change Startup type from Manual to Automatic Delay Start 12 Click the Start button to start the OpenSM service BX900 BX400 InfiniBand Modules 40Gb User Manual 47 Installation of WinOF Windows Installation 13 Open the Network Connections Window to check if IPoIB networks are linked up Disable the unused IPoIB adapters F network Connections umet JC LE Control Panel Network and Internet Network Connections Search Network Cor Organize jy Private jy Enterprise Y 1 Local Area Connection 4 Unidentified network Network Disabled 7 Intel R 82575 Gigabit Backplane 1 7 Intel R 82575EB Gigabit Backplane 1 7 Mellanox IPoIB Adapter A jy Application Identifying
30. Reporting Generic Notice type 1 num 128 Link state change from LID LID number GUID GUID number SM port is down SubnetManager s master port was disconnected How to solve the problem The following changes are required Please contact the repair center 1 IB Mezzanine card You need to change the IB Mezzanine card that ran as the Master of the SubnetManager For the method of how to specify the exact IB Mezzanine card please refer to section The result of the command ibnetdis cover on page 32 2 ServerBlade If you have changed IB Mezzanine card and you still see the same messages you need to change the server blade that installed the IB Mezzanine card BX900 BX400 InfiniBand Modules 40Gb User Manual 61 Check the Log File of OpenSM Linux only Troubleshooting 3 IB connection blade If you have changed the above server blade and still get the same messages you need to change IB connection blade For details of how to specify the exact IB connection blade please refer to section The result of the command ibnetdiscover on page 32 Removed port with GUID GUID number LID range LID number LID number of node IB Mezzanine card IB connection blade or IB cable was discon nected How to solve the problem The following changes are required Please contact the repair center 1 IB Mezzanine card You need to change the relevant IB Mezzanine card You obtain the relevant IB Mezzanine card from GUID or
31. User Manual 71 IB Mezzanine Card Troubleshooting To do The IB Mezzanine card must be replaced Please consult technical support Q ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Event ID Classification Error Level 0x005B xxx Pkey index not found Error for partition change switch pkey configuration 0x005C xxx Connected Mode Error failed to initialize disabled Interface will use default UD QP transport 0 005 SetDeviceRegistra Error tionAttributes failed 0 005 SetAdapterRegistra Error tionAttributes failed 0x0060 xxx SetOffloadAttributes Error failed 0x0061 ipoib_create_adapter Error failed 0x0062 ipoib_start_adapter Error failed Table 7 source ipoib 72 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card Event ID Classification To do Error Level 0x0003 The message changes in Error each event 0x0004 mlx4 bus has started in non Error operational mode 0x0005 mlx4_bus has failed to start Error even in non operational mode n Look into the the previous error messages 0 0007 MAP_FA command failed with Error error 2 The adapter card is non The IB Mezzanine card must be functional 9en replaced Most likely a FW problem n Please consult Please burn the last FW and tec
32. ance Recom mended and then click Next gt MLNX_VPT InstallShield Wizard Maximum Performance Configure your system for maximum 10GigE performance Figure 22 Maximum performance BX900 BX400 InfiniBand Modules 40Gb User Manual 43 Installation of WinOF Windows Installation 7 Click Install after the following window appears i MLNX_VPI InstallShield Wizard Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings dick Back Click Cancel to exit the wizard InstallShield Figure 23 Installation 44 User Manual BX900 BX400 InfiniBand Modules 40Gb Windows Installation Installation of WinOF 8 The following window appears after the installation process is finished Click Finish without checking any boxes i MLNX VPI InstallShield Wizard InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPI Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C Windows System32 LogFiles Per formanceTu ning log Enable WinSock Direct protocol WSD Enable Socket Direct Protocol SDP Enable NetworkDirect interface ND Show IB release notes Show ETH release notes Figure 24 Installation completed BX900 BX400 InfiniBan
33. ant to change this configuration Ly N y 11 Configure IP Address Netmask respectively then enter when the message Start Device On Boot is displayed If IP Address is 192 168 1 30 and Netmask 255 255 255 0 enter the input values as follows Enter the Netmask 255 255 255 0 Start Device On Boot Y n y 12 Check the configured values and enter y to save the configuration DEVICE ethO0 IPADDR 192 168 1 30 ETMASK 255 255 255 0 HWADDR 00 02 C9 11 23 56 ONBOOT yes Do you want to save the selected configuration Y n y BX900 BX400 InfiniBand Modules 40Gb User Manual 55 Configuration IB Connection Blade Management 13 After displaying the following message select 4 Exit from Network Config uration Utility to finish the configuration 1 Configure Host Networking 2 Configure IP Address 3 Configuration for Host Networking and IP Address 4 Exit from Network Configuration Utility Enter Your Selection 4 14 Enter to enable the contents of configuration Do you want to enable the new network configuration Y nly 15 After displaying the prompt enter ifconfig to check if the configured IP address is displayed lt BX900S1 CB3 gt i fconfig Link encap Ethernet HWaddr 00 02 9 11 23 56 inet addr 192 168 1 30Bcast 192 168 14 255 Mask 255 255 255 0 UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 9805 errors 0 dropped 0 overruns 0 frame 0
34. are version of the IB connection blade BX900S1 CB3 swinfo lt BX900S1 CB3 Information GUID 0x0002c9020040c7a8 MAC 00 02 c9 11 20 fc 154 Firmware revision 7 2 326 CPU Firmware revision Mellanox release 1 405 1 0 2 build 2009 07 19 ifconfig command This command shows and configures IP address subnetmask broadcast address The setting values by this command will disappear after IB connection blade reboots If you would like to keep the values you can use config net command lt BX900S1 CB3 gt ifconfig ethO 192 168 12 33 24 lt BX900S1 CB3 gt i fconfig ethOLink encap Ethernet HWaddr 00 02 C9 11 20 FC inet addr 192 168 12 33 Bcast 192 168 12 255 Mask 255 255 255 0 UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 5553 errors 0 dropped 0 overruns 0 frame 0 TX packets 228 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 745872 728 3 KiB TX bytes 39013 38 0 KiB Interrupt 25 loLink encap Local Loopback inet addr 127 0 0 1 Mask 255 0 0 0 UP LOOPBACK RUNNING MTU 16436 Metric 1 RX packets 28 errors 0 dropped 0 overruns 0 frame 0 58 User Manual BX900 BX400 InfiniBand Modules 40Gb IB Connection Blade Management Basic Operation 4 3 2 SNMP Trap Settings This chapter describes the SNMP trap settings for IB connection blade 1 IP address setting The IP address of IB connection blade is set up via Console Redirection of Management Blade 2 Enab
35. ation Results Kernel modules The kernel modules are installed under InfiniBand subsystem lib modules uname r kernel drivers infiniband mlx4 driver Under lib modules uname r kernel drivers net mIx4 you will find mlx4 core ko mlx4 en ko and mlx4 fc if you ran the installation with with fc RDS ib modules uname r kernel net rds rds ko Bonding module ib modules uname r kernel drivers net bonding bonding ko Startup script The script openibd is installed under etc init d This script can be used to load and unload the software stack Configurations files The etc ofed directory is created with the file openib conf The openib conf file contains the list of modules that are loaded when the openibd script is used If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc ofed Other files The 90 ib rules file is installed under etc udev rules d Man pages will be installed under usr share man 2 3 4 Post installation Notes User space Memory Limitations The script adds the following lines to etc security limits conf for the user space components such as soft memlock unlimited hard memlock unlimited BX900 BX400 InfiniBand Modules 40Gb User Manual 19 Installation by means of SVIM Linux Installation These settings make the amount of memory that can be pinned by a user space application unlimited If
36. d Modules 40Gb Windows Installation Installation of WinOF 10 After WinOF installation OpenSM should be set up OpenSM is an InfiniBand compliant Subnet Manager SM At least one OpenSM is present on the InfiniBand network OpenSM has a role to initialize and configure an InfiniBand network If HPC Cluster will consist of up to 16 nodes or less OpenSM can run on the head node computer The OpenSM service is registered to the Services Window after WinOF installation Double click OpenSM to open properties la x Fie Action View 9 111 amp Services Local Services Local Name Gh Microsoft iSCSI Initi OpensM Manages I Enables re Stop the service Microsoft Software Manages s Pause the service Ti Multimedia Class Sc Enables rel Restart the service Port Sharin Provides a X Maintains a Description TheNetwo OpenSM Gi Network Connections Manages Network List Service Identifies t Collects an This servic Performan Enables a GhPnP X IP Bus Enum ThePnP X Gi Portable Device Enforces g Manages p Loads files This servic X Protected Storage Provides pr Remote Access Aut Creates a Remote Access Manages di Access Qu Removes v
37. d Modules 40Gb User Manual 45 Installation of WinOF Windows Installation 9 Make sure that InfiniBand devices are recognized properly in the device manager File Action View Help E Server Manager e 2 3 E Roles ogi Features El Diagnostics Event Viewer g Performance CEN B Configuration E a Storage Em Server Manager WIN 8VG547H7FI 7775 E DVD CD ROM drives W Human Interface Devices P InfiniBand Channel Adapters P Mellanox ConnectX Virtual Channel Adapter Mellanox ConnectX Virtual Channel Adapter 91 20 Keyboards Monitors Network adapters Intel R 82575 Gigabit Backplane Connection amp Intel R 82575 Gigabit Backplane Connection 2 amp Intel R 82575EB Gigabit Backplane Connection 3 Intel R 82575 Gigabit Backplane Connection 4 n Mice and other pointing devices z Mellanox IPoIB Adapter Mellanox IPoIB Adapter 2 Mellanox IPoIB Adapter 3 Mellanox IPoIB Adapter 4 Ports amp Processors lt gt Storage controllers System devices FFE ae Universal Serial Bus controllers Figure 25 InfiniBand devices 46 User Manual BX900 BX400 InfiniBan
38. dapter i a Mellanox IPoIB Adapter 3 IPoIB Adapter 4 H P Ports COM Processors lt gt Storage controllers 19 088 System devices 9 Universal Serial Bus controllers T A Figure 34 Device manager 3 Refer to section Configuration sheet on page 29 update GUID table BX900 BX400 InfiniBand Modules 40Gb User Manual 75 IB Connection Blade Troubleshooting 5 3 IB Connection Blade IB Connection Blade Message Connect to IB connection blade via MMB Console Redirection The reference to the var log messages log file uses the text editor etc No Classification To do 1 Installed FW has unsupported command interface revision d The IB connection blade must Installed FW version is d d 03d be replaced This driver version supports only Please consult revisions d to d technical support Q ts fujitsu com 2 This driver version supports only revisions 95d to 95d If you purchased the products from Fujitsu Japan please 3 Internal error detected consult the system engineer buf 9602x 08 Table 10 IB connection blade messages snmptrap messages When SNMP Trap of the IB connection blade is setto enable following snmp trap is transmitted to SNMP Manager No SNMP Trap OID Content and Action 1 ASIC chip down 1 3 6 1 4 1 Content 33049 2 2 1 2 1 The ASIC of the IB connection blade was downed Action Reboot of the IB c
39. dd to add the repository Add New Repository x Name DVD Cluster Description DVD Cluster Location file media CDROM Cluster Mirror List 7 Advanced GPG Key Fite media CDROM PRM GPG KEY redhat releasd X Cancel OK Figure 11 Add new Repository 22 User Manual BX900 BX400 InfiniBand Modules 40Gb Linux Installation OFED Installation by means of SVIM Item Name Name Arbitrary name Description Arbitrary explanation Location L The format is as follows file lt mount position of DVD media gt lt directory name here the inclusion of rpm file gt Here the mount position of DVD media is assumed to be media CDROM The directory names where the rpm files are included are Cluster ClusterStorage Server and VT in Red Hat Enterprise Linux Version 5 4 GPG Key G Path to GPG key file In the following example GPG key file RPM GPG KEY redhat release is assumed to be included in the OS DVD go sure that RPM GPG KEY redhat release is included the OS DVD Enter the path to the file in GPG Key Set the repository for all the directories Make sure that the directories created above are ticked and other reposi tories are not ticked in repository manager BX900 BX400 InfiniBand Modules 40Gb User Manual 23 Installation by means of SVIM Linux Installation Repository Manage
40. e ae SMS MS ae a vo oe Se oe ae Se Se ae MOES PET LLL T C29 CO P CA L CA 2 CA L CA 2 CA 2 CA 1 CA 2 CA 1 CA 2 CA 2 CA 1 32 User Manua BX900 BX400 InfiniBand Modules 40Gb Linux Installation Network Configuration Information 0x0002c9030004 0x0002c9030004 0x0002c9030004 0x0002c9030004 0x0002c9030004 0x0002c9030003bf82 0x0002c9030003bf8e BX900 02 IB 0x0002c9020040c7a8 BX900 04 IB 0x0002c9020040c790 bx900 04 SWITCH SWITCH bx900 04 CARD 0x0002c9030004 0x0002c9030004 ibnetdiscover command execution procedure 1 Connect to the CLI of IB connection blade via MMB MMB Console Redirection 11748 bx900 02 1 112 bx900 02 1 11274 bx900 02 1 113 bx900 02 1 11738 bx900 02 1 bx900 02 1 bx900 02 1 1 bx900 04 2 Execute the command swinfo lt BX900S1 CB3 gt swinfo lt BX900S1 CB3 Information GUID 0x0002c9020040c7a8 00 02 c9 11 20 5a IS4 Firmware revision 7 2 326 CPU Firmware revision 3 In the example above the GUID of IB connection blade is 0x0002c9020040c7a8 4 Excute the command ibnetdiscover on the server blade of slot1 14760 bx900 04 CA 2 CA 1 CA 2 CA 1 CA 2 CA 1 CA 2 40 bx900 02 IB SW CB3 IB SW CB3
41. e agreement and then click Next gt ie MLNX InstallShield Wizard License Agreement Please read the following license agreement carefully Copyright c 2005 2008 Mellanox Technologies All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer ae ws heres Ac in les Farm mna mannan Annn tha nhara zl 1 accept the terms in the license agreement Print C Ido not accept the terms in the license agreement InstallShield Figure 20 License Agreement BX900 BX400 InfiniBand Modules 40Gb User Manual 41 Installation of WinOF Windows Installation 5 Click Next gt after the following window appears if you want to change the installation folder specify the folder you want to install on by clicking Change 51 MLNX_VPI InstallShield Wizard Destination Folder Click Next to install to this folder or dick Change to install to a different Figure 21 Destination Folder 42 User Manual BX900 BX400 InfiniBand Modules 40Gb Windows Installation Installation of WinOF 6 After the following window appears remove the check mark from Check this box to configure your system for maximum 10GigE perform
42. e of configuration sheet BX900 BX400 InfiniBand Modules 40Gb User Manual 31 Network Configuration Information Linux Installation 2 7 3 The result of the command ibnetdiscover GUID list It is useful to provide the text file txt which associates a GUID with the name of the device so that the ibnetdiscover command displays the result more meaningfully format GUID hex number Hostname and Mezzanine slot written in the Configuration sheet bx900 02 CARD 0x0002c903000413c0 0x0002c90300041514 0x0002c9030003bf7a 0x0002c9030003bf6a 0x0002c903000413b8 0x0002c90300041 0x0002c90300041 0x0002c903000412f0 0x0002c90300041 0x0002c90300041348 0x0002c903000412ec 0x0002c9030004139c 0x0002c9030003bf9e 0x0002c9030004141c 0x0002c9030003bf86 0x0002c9030003bfca 0x0002c9030003bfa6 0x0002c90300041314 0x0002c903000413e8 0x0002c903000413f0 0x0002c9030003bfba 0x0002c9030003bf7e 0x0002c9030003bf92 0x0002c9030003bfa2 0x0002c9030003d138 0x0002c90300041304 0x0002c9030003bf96 0x0002c9030003bf72 0x0002c90300041460 bx900 02 01 bx900 02 01 bx900 02 02 bx900 02 02 bx900 02 03 bx900 02 03 bx900 02 04 bx900 02 04 bx900 02 05 bx900 02 05 bx900 02 06 bx900 02 06 bx900 02 07 bx900 02 07 bx900 02 08 bx900 02 08 bx900 02 09 bx900 02 09 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 1 bx900 02 14 bx900 02 14 bx900 02 1
43. em x86_64 only Installer Privileges The installation requires administrator privileges on the target machine To reach best performance the Enhanced Idle Power State option in the BIOS setup Advanced menu must be disabled BIOS SETUP UTILITY CPU Mismatch Detection Enabled Figure 6 BIOS Setup Utility 16 User Manual BX900 BX400 InfiniBand Modules 40Gb Linux Installation Native InfiniBand Installation in RHEL 2 3 Native InfiniBand Installation in RHEL This chapter describes the installation method of OFED for Red Hat Enterprise Linux Version 5 4 2 3 1 Setup InfiniBand during Linux Installation All required InfiniBand drivers and tools can be set up during the installation of Red Hat Enterprise Linux Step 1 Run the installation routine until the Package Installation Defaults screen appears This screen varies depending on the version of Red Hat Enterprise Linux you are installing RED HAT ENTERPRISE LINUX 5 The det You can further customize the software selection now or after install via the software management application Customize later 3 Customize now Release Notes 4 Back Next Figure 7 Red Hat package installation defaults Step 2 Select the Customize now option on the screen The next screen named Package Group Selection shows a list of groups containing additional packages Selecting the list item Base Systems displays the detailed view on the right side BX
44. ere the error occured using the following confirmation contents 64 User Manual 9 400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card No Confirmation item Judgment To do Is the server blade turned on YES Go to No 2 NO Turn on the server blade Are the LEDs of all server blades turned off YES Confirm OpenSM running on the MASTER OpenSM server NO Go to No 3 Is the LED of one server blade in the system turned off YES Go to No 4 NO Go to No 5 Is the IB Mezzanine card mounted rigidly YES 1 Replace the IB Mezzanine card 2 If the LED is still turned off after replacing the IB Mezzanine card replace the server blade 3 If the LED is still turned off after replacing the server blade replace the IB connection blade 4 If the LED is still turned off please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer NO Mount the IB Mezzanine card more rigidly Does IB connection blade work normally Please refer to section IB Connection Blade on page 76 YES Go to No 4 Replace the IB connection blade Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 5 Indication lamps BX900 BX400 InfiniBand Modules
45. et to both Mezzanine slot and Mezzanine II slot When Mezzanine Card is installed in Mezzanine II slot the dual port of the card will be available and one of two ports will be connected to CB in CB5 6 the other one will be CB7 8 While installed in Mezzanine slot 1 only one port will be utilized and connected to the CB in CB3 4 When the two connection blades are installed in CB5 6 and CB 7 8 other types of connection blades like SB11 SB11A can be used in CB1 2 3 4 and SBAX2 and Brocade FC connection blade in CB3 4 in parallel with IB connection blade Server blade 929 52 BX922 52 Bx924 52 D wide SW n CB7 8 IB QOR pier 811 58 11 Ow 7 SBAX2 8G FC SW 61361 Serene be d Figure 3 Connection relation between server blade and connection blade 10 User Manual BX900 BX400 InfiniBand Modules 40Gb Introduction Description of InfiniBand Connection Blade 1 8 Description of InfiniBand Connection Blade The PRIMERGY 900 InfiniBand connection blade module features 18 internal and 18 external 4x10Gbps QDR ports The below figure shows the front view with the 18 external ports whereas the internal ones are connected to the server blades over Midplane Figure 4 InfiniBand connection blade 1 8 1 Cables For best performance use the InfiniBand connection blade with QSFP Quad Small Form factor Pluggable connector
46. gure IP address see the detail operation in Configuration below 2 Copy ibsw conf file to usr ftproot of IB connection blade by ftp or scp command 3 Execute restoreconf command from IB connection blade BX900 BX400 InfiniBand Modules 40Gb User Manual 53 Configuration IB Connection Blade Management BX900S1P00131 CB5 scp root 192 168 1 100 ibswb conf usr ftproot root 192 168 1 100 s password ibswb conf 100 4505 4 4KB s 00 00 BX900S1P00131 CB5 restoreconf ibswb conf Configuration was restored successfully i The configuration will be valid after IB connection blade reboots 4 2 Configuration In order to configure the function of IB connection blade using CLI Command Line Interface the user needs to either connect the management console to Management Blade and run a terminal software or connect to Management Blade via telnet After finishing configuring the network configuration e g IP address it is possible to login via telnet 4 2 1 Configuration of LAN interface 1 Confirm that the prompt is displayed In case the prompt isn t displayed press the Enter key to display the prompt Example configure CB3 Press Ctrl Q Return Console Menu BX900S1 CB3 config net 2 Enter config net and press the Enter key to display the menu below BX900S1 CB3 config net Welcome to The Network Configuration Utility 1 Configure Host Networking 2 Conf
47. hnical supportQ ts fujitsu com restart the mlx4 bus driver 0x0008 RUN FW command failed the oom Enor with enor 52 a ut The adapter card is non y 9 functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0 0009 QUERY_FW command failed Error with error 2 The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver Table 8 source mlx4_bus BX900 BX400 InfiniBand Modules 40Gb User Manual 73 IB Mezzanine Troubleshooting Event ID Error Level 0x000B Error 0x000C Error Classification QUERY_DEV_CAP command failed with error 2 N The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver QUERY_ADAPTER command failed with error 2 N The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x000D Error Too few QPs were requested requested 2 reserved for FW 3 The adapter card is non functional n Please increase the Registry LogNumQp parameter under HKLM System CurrentCon trolSet Services m x4_bus Parameters To do The IB Mezzanine card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system eng
48. igure IP Address 3 Configuration for Host Networking and IP Address 4 Exit from Network Configuration Utility Enter Your Selection 3 Select 3 Configuration for Host Networking and IP Address on the menu Enter Your Selection 3 4 After displaying the following message enter y Do you want to change this configuration 54 User Manual BX900 BX400 InfiniBand Modules 40Gb IB Connection Blade Management Configuration 5 After displaying the following message enter y to enable NETWORKING support Enable NETWORKING support Y nl y 6 After displaying each following message configure Gateway Address hostname respectively If Gateway Address is 192 168 1 1 and hostname ibsw1 enter the input values as follows Enter a Gateway Address 192 168 1 1 Enter a hostname localhost localdomain ibswl 7 Checkthe configured values and enter y to save the configuration Selected configuration ETWORKING yes GATEWAY 192 168 1 1 HOSTNAME i bsw1 Do you want to save the selected configuration Y nl y 8 Enter y to display the IP address configuration menu Do you want to configure IP Address Y nl y 1 Configure by DHCP 2 Static Configuration 3 Exit from Configuration System Utility Enter Your Selection 9 Select 2 Static Configuration on the menu Enter Your Selection 2 10 After displaying the following message enter y Do you w
49. ineer Table 8 source mlx4_bus Event ID Classifi To do Error Level cation 0x0003 The The IB Mezzanine card must be replaced Error message Please consult changes in technical support ts fujitsu com each event If you purchased the products from Fujitsu Japan please consult the system engineer Table 9 source ibbus 74 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card Work after IB Mezzanine card has been exchanged Windows Server 2008 R2 1 Turn on the server 2 Make sure that the InfiniBand devices are recognized properly in Device Manager E Server Manager File Action View Help 4 D Server Manager E gj Roles m aj Features B a Diagnostics Aami amp Event Viewer is iei ETT fy Display adapters Performa e A Device Manager 5 DVD CD ROM drives m gl i Human Interface Devices a Es Bun E A InfiniBand Channel Adapters Mellanox ConnectX Virtual Channel Adapter Mellanox ConnectX Virtual Channel Adapter lt gt Keyboards n Mice and other pointing devices Monitors AP Network adapters Intel R 82575EB Gigabit Backplane Connection e Intel R 82575EB Gigabit Backplane Connection 2 amp Intel R 82575 Gigabit Backplane Connection 3 Intel R 82575 Gigabit Backplane Connection 4 IPoIB A
50. is displayed lt BX900S1 CB5 gt ibstat Switch is4 0 Switch type MT48436 umber of ports 0 Firmware version 7 2 326 Hardware version a0 ode GUID 0x0002c9020040bbb8 System image GUID 0x0002c9020040bbb8 Port 0 State Active Physical state LinkUp Rate 40 Base lid 28 LMC 0 SM lid 16 Capability mask 0x02500848 Port GUID 0x0002c9020040bbb8 9 Refer to section Network Configuration Information on page 28 update a GUID table 78 User Manual BX900 BX400 InfiniBand Modules 40Gb Auto Stop 60 B 400 1 5 900 1 5 C CLI 60 Command line interface 60 configuration files 19 configuration sheet 29 CPU module boot 60 D disk Space for installation 16 disk space for Windows installation 37 E ethernet connection 57 G GUID 35 ibnetdiscover 29 ibstat 28 ifconfig 58 InfiniBand connection blade connection 51 installer privileges 16 Windows 37 K kernel 18 kernel modules 19 L LEDs 12 Linux 16 log file 61 M Management Blade 51 login 51 master 28 memory limitations 19 Mezzanine card exchange 68 GUID 35 hardware installation 5 technical data 13 N network configuration diagram 28 notational conventions 5 OFED 5 SVIM 20 OpenFabrics Enterprise Distribution 5 OpenSM 28 61 installation 18 opensm 20 operating system 16 P PRIMERGY BX400 5 PRIMERGY BX900 5 PRIMERGY BX900 Blade Server 8 Q QSFP 11 Quad Small Form factor Pluggab
51. le 11 R Red Hat Enterprise Linux 17 repair center 61 Root password 60 BX900 BX400 InfiniBand Modules 40Gb User Manual 79 5 Server View Installation Manager 20 SM 28 software requirements Windows 37 standby 29 Subnet Manager 28 subnet manager 20 27 OpenSM 61 support 61 SVIM 20 swinfo 58 switch LEDs 12 T technical support 61 telnet connection 52 U user maintenance 60 Windows installation 37 80 User Manual BX900 BX400 InfiniBand Modules 40Gb
52. ling SNMP Trap SNMP Trap is disabled by default How to enable the setting is as follows a Command ibswsnmptrap b Viewing the SNMP trap information default BX900S1P00007 ibswsnmptrap Trap is Disabled Authentication Failure is Disabled Address 0 0 0 0 Community c How to set The details of a parameter of the ibswsnmptrap command are as follows BX900S1P00007 ibswsnmptrap help Usage ibswsnmptrap enabled disabled enabled disabled i IP address Community t Send SNMP traps is enabled or disabled a When SNMP trap is enabled authentification failure trap is enabled or disabled 1 The destination of SNMP Trap receiver Set community for snmptraps SNMP Trap send address 192 168 11 253 Community name public Examples of the above settings are as follows BX900S1P00007 ibswsnmptrap t enabled i 192 168 11 253 c public BX900S1P00007 ibswsnmptrap Trap is Enabled Authentication Failure is Disabled Address 192 168 11 253 Community public BX900 BX400 InfiniBand Modules 40Gb User Manual 59 Users and Privileges IB Connection Blade Management 3 Disabling SNMP Trap How to disable the SNMP Trap is as follows BX900S1P00007 ibswsnmptrap t disabled BX900S1P00007 ibswsnmptrap Trap is Disabled Authentication Failure is Enabled Address Community 4 4 0 0 0 0 Users and Privileges The CLI is entered via the Telnet
53. mmands to be entered through the keyboard are written fixed font in a semi bold fixed font Key symbols Keys are presented according to their representation on the keyboard If capital letters are to be entered explicitly then the Shift key is shown e g SHIFT for A If two keys need to be pressed at the same time then this is indicated by placing a hyphen between the two key symbols BX900 BX400 InfiniBand Modules 40Gb User Manual 5 Target Group Introduction Fujitsu Support Please contact your Fujitsu Technical Support if you require assistance http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult your system engineer 1 2 Target Group This manual is intended for users developers and system administrators responsible for setting up and maintaining switch system platforms using InfiniBand fabrics and networks The switch system platforms must have a management server or client in order for the package to work The manual assumes familiarity with the InfiniBand Architecture Specification as well as Ethernet Architecture Specification 1 3 Documentation Overview PRIMERGY manuals are available in PDF format on the ServerView Suite DVD 2 The ServerView Suite DVD 2 is part of the ServerView Suite supplied with every server If you no longer have the ServerView Suite DVDs you can obtain the relevant curre
54. nt version using the order number U15000 C289 the order number for Fujitsu Japan please refer to the configurator of the server http jp fujitsu com platform server primergy system The PDF files of the manuals can also be downloaded free of charge from the internet The overview page showing the online documentation available on the internet can be found using the URL for EMEA market http manuals ts fujitsu com The PRIMERGY server documentation can be accessed using the Industry standard servers navigation option If you purchased the products from Fujitsu Japan please use the URL http jp fujitsu com platform server primergy manual 6 User Manual BX900 BX400 InfiniBand Modules 40Gb Introduction Overview 1 4 Overview This manual explains the use of the software management Managed devices must have a CPU and a management board This board has an operating system and a toolset to operate manage and upgrade the device Further information is provided on the PRIMERGY ServerBooks DVD PRIMERGY BX900 51 Blade Server System Operating Manual PRIMERGY BX400 S1 Blade Server System Operating Manual PRIMERGY Server Systems RemoteDeploy PRIMERGY BX Blade Server Systems LAN Connection Blade ServerView User Guide For further information on updating BX components please refer to the Operating Manual of your BX system BX900 BX400 InfiniBand Modules 40Gb User Manual 900 Introduction
55. o send processing 0x004C xxx Failed to create Error NDIS_BUFFER pool to send processing Table 7 source ipoib 70 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Mezzanine Card Event ID Classification Error Level 0x004D xxx Failed to allocate Error receive indication array 0 004 xxx Subnet Administrator Error query for port information timed out Make sure the SA is functioning properly Increasing the number of retries and retry timeout adapter parameters may solve the problem 0 004 xxx Subnet Administrator Error failed the query for port information Make sure the SA is functioning properly and compatible 0x0050 xxx Subnet Administrator Error query for port information failed 0x0055 xxx Subnet Administrator Error failed query for broadcast group information 0 0056 xxx Subnet Administrator Error failed request to joining broadcast group 0x0057 xxx The local port rate is Error too slow for the existing broadcast MC group 0x0058 xxx Incorrect value or Error non existing registry for the required IPoIB parameter 3 overriding it by default value 4 To do The IB Mezzanine card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 7 source ipoib BX900 BX400 InfiniBand Modules 40Gb
56. onnection blade When it doesn t restore even if rebooted the IB connection blade must be replaced When trap is frequent even if restore after Reboot the IB connection blade must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 11 snmp trap 76 User Manual BX900 BX400 InfiniBand Modules 40Gb Troubleshooting IB Connection Blade No SNMP Trap OID Content and Action 2 ASIC overtemp 1 3 6 1 4 1 Content 33049 2 2 1 2 2 The ASIC of the IB connection blade detected temperature abnormality Action Confirm that ambient temperature of BX900 or BX400 system does not have abnormality When ambient temperature does not have abnormality the IB connection blade must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer 3 Internal error 1 3 6 1 4 1 Content 33049 2 1 2 1 The Internal I2C error occurred Action Reboot of the IB connection blade If it doesn t restore even if rebooted the IB connection blade must be replaced When trap is frequent even if restore after reboot the IB connection blade must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer 4 cold sta
57. r Repositories DVD Cluster DVD Cluster DVD ClusterStrage Remove DVD ClusterStorage DVD Server DVD Server DVD VT DVD VT rhel debuginfo Red Hat Enterprise Linux 5Server x86 64 De rhel debuginfo beta Red Hat Enterprise Linux 5Server Beta x86 6 Close Figure 12 Repository Manager 9 Close the Repository Manager and the Package Manager 10 Run the terminal and execute the following command yum repolist 11 Run the Package Manager 12 Tick OpenFabrics Enterprise Distribution group in Base System 13 Tick all the packages in the group 14 Click Apply OFED is installed 2 4 2 Inthe event of RHEL6 0 This chapter describes the installation of OFED for Red Hat Enterprise Linux Version 6 0 Please install OFED by means of OS DVD after the OS installation 1 Insert OS DVD in the DVD drive and make sure that the DVD is mounted 24 User Manual X BX900 BX400 InfiniBand Modules 40Gb Linux Installation OFED Installation by means of SVIM 2 Add the following statement to etc yum conf group_package_types mandatory default optional Make the repo file then save the file to etc yum repos d In the following examples DVD media is mounted to media RHEL 6 0 x86 64 Disc 1 the repository name is dvd Example etc yum repos d repo Ldvd baseurl file media RHEL_6 0 x86 64N Disc 1 enabled 1 gpgcheck 1
58. rt MIB II standard Content trap The cold start occurred Action When cold start not intended happens frequently the IB connection blade must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 11 snmp trap BX900 BX400 InfiniBand Modules 40Gb User Manual 77 IB Connection Blade Troubleshooting Work after the exchange 1 IB connection blade is mounted on the main body of BX900 BX400 and the power supply is turned on 2 Confirm that the Status LED of the IB connection blade lights to green It takes about 90 seconds till the CPU of the IB connection blade starts 3 Connect to the console of the IB connection blade via telnet and wait for the prompt of the IB connection blade to be displayed When no prompt is displayed push Enter and confirm that a prompt is displayed 4 Set ip address which there was it before exchange Example ifconfig ethO 192 168 1 100 24 5 Restore composition definition information Please refer to section Save and restore the configuration on page 52 6 Connect IB Cable to the same port as before 7 Confirm that LED of P and L A lights It might take about one minute to lighting LED 8 ibstat is executed from the console of the IB connection blade and comfirm that State Active Physical state LinkUp Rate 40 Base lid figures other than 0
59. s Copper and optical variants can be used Figure 5 QSFP connectors BX900 BX400 InfiniBand Modules 40Gb User Manual 11 Technical Data Introduction 1 8 2 LEDs Connection Blade LEDs To indicate overall status of the connection blade two LEDs are located at the left side of the front panel LED Position Color Function LED A Left side top Blue Identify active Identify LED Off Identify inactive LED B Left side bottom Green Power on Health LED Amber blinking Error Off Power off Table 1 Connection blade LEDs Port LEDs For every external port a pair of LEDs exists LED Position Color Function LED A Above port right Green Link up Port Physical Link Off No Link LED B Above port left Yellow Link up Port Logical Link Yellow blinking Link active Off Link down Table 2 Port LEDs 1 9 Technical Data 1 9 1 IB Connection Blade Internal Ports 18 External Ports 18 Lanes per Port 4 4x Max Speed per Lane 10Gbps QDR 12 User Manual BX900 BX400 InfiniBand Modules 40Gb Introduction Technical Data Product Weight 9 4 Lbs 4 3 kg Form Factor Double wide Height 27 9 mm 1 1 Width 388 2 mm 15 3 Depth 267 9 mm 10 6 1 9 2 IB Mezzanine Card Form factor BX900 Standard form factor Server blades supported BX920 S2 BX922 S2 BX924 S2 BX92 S3 BX924 S3 Max number of Mezzanine Cards per
60. ssh interface of the management blade Three user levels are supported to execute CLI commands No User name Authority Description level 1 root 55ib qdr sw root Only Factory people and R amp D use The password is not usually used 2 maintenance yrotcaf maintenance Our maintenance and system engineer use 3 ibsw ibsw user For Read only users Table 4 Default user User maintenance runs as user root Or maintenance Root password can be changed User names and passwords are enforced when logging in via UART When CPU module boots up Auto Stop is done by pressing Ctrl B Pressing Ctrl B enters the boot console 60 User Manual BX900 BX400 InfiniBand Modules 40Gb 5 Troubleshooting This chapter describes what to do if IB Mezzanine card and IB connection blade is not working properly If you cannot solve the problem please contact the repair center http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult your system engineer 51 Check the Log File of OpenSM Linux only Check the log file in the server var log opensm log that started up the Subnet Manager OpenSM as Master gt Check list Are any of the following logs contained in the var log opensm log SM port is down Removed port with GUID GUID number LID range LID number LID number of node
61. t 14 53 on Monday 24 August 2009 BX900S1P00131 CB3 login maintenance Password 4 1 2 Save and restore the configuration The configuration data can be saved and restored by ftp command from PC Caution Look out for the following during the operation of save and restore Do NOT turn off the power Check that there is no traffic between the nodes Operate with NOT doing any setting from a management terminal 52 User Manual BX900 BX400 InfiniBand Modules 40Gb IB Connection Blade Management Connection The back face 900 51 IP address 192 16 1 10 24 save the condi poate nbyfip comana FTP server 192 168 1 100 24 example BX900 chassis Figure 32 Save configuration How to save the configuration If you would like to save the configuration file as ibswb conf file for example do the following 1 Configure IP address see section Configuration on page 54 2 Execute saveconf command from IB connection blade 3 Copy ibsw conf file to another PC or server by ftp or scp command because under usr ftproot files will disappear after IB connection blade reboots BX900S1P00007 CB3 saveconf ibswb conf Saving configuration into usr ftproot ibswb conf Done BX900S1P00007 CB3 scp usr ftproot ibswb conf root 192 168 1 100 ibswb conf root 192 168 1 100 s password ibswb conf 100 4505 4 4KB s 00 00 How to restore the configuration 1 Confi
62. tion Configuration on page 54 1 Login to Management Blade CLI via telnet Please refer to Server View Management Blade to see about the connection and the booting up of Management Blade CLI 2 Select 3 Console Redirection and press the key 3 Select 1 Console Redirect Connection Blade and press the Enter key 1 11 3 Se ite 1 Figure 30 Console redirection 4 Enter the slot number on which IB connection blade is installed and press the key Example If IB connection blade is installed on slot3 4 enter and press the key BX900 BX400 InfiniBand Modules 40Gb User Manual 51 Connection IB Connection Blade Management 18 Mellanox Mellanox Figure 31 Console redirect connection blade 4 1 1 telnet connection The user can excute the commands via telnet from the management console The method of connecting to IB connection blade via telnet is described below gt Runa terminal software on the management console gt Enter the command including the IP address referred to above telnet 192 168 1 10 and press the Enter key The user connects to IB connection blade via telnet gt Enter the username maintenance and the password yrotcaf and press the key LibuserGlocalhost 1 telnet 192 168 1 10 Trying 192 168 1 10 Connected to 192 168 1 10 192 168 1 10 Escape character is Linux 2 6 27 localhos

Download Pdf Manuals

Related Search

Related Contents

Zanussi 641700 Brochure  L`Accueil Habitat Adapté deux ans après… Bilan au 30 juin    Plantronics APC-4 User's Manual  Acer Aspire AZC-606-MB27  vestax_pbs-4_bedienungsanleitung (link is - Dj  GP Batteries Lithium Cell GPCR1216-C5 non-rechargeable battery  Kenmore 580.72184 User's Manual  301R IX71 User Guide  1 - Vox-Cam Associates  

Copyright © All rights reserved. Failed to retrieve file