Home

BX900/BX400 InfiniBand Modules V4.0 Connection Blade and

Contents

1. Computer Disk drives Display adapters Human Interface Devices lt Keyboards A Mice and other pointing devices amp Monitors Network adapters Other devices Infiniband Controller YI Ports COM amp LPT Processors lt gt Storage controllers 2 System devices System management devices Universal Serial Bus controllers BB m EE 8 8 9 Figure 12 Display of InfiniBand devices before installation of WinOF 2 Double click the Unknown device Click Details tab and select Hardware Ids in Property Make sure there are VEN 153B and DEV 1003 in Value which shows the device is an InfiniBand card 3 Log on to Windows as administrator and then double click the file of downloaded WinOF 36 User Manual BX900 BX400 InfiniBand Modules Windows Installation Installation of WinOF 4 Click Next gt after the following window appears InstallShield Wizard Welcome to the InstallShield Wizard for MLNX_VPI The InstallShield R Wizard will install MLNX_VPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Figure 13 InstallShield Wizard BX900 BX400 InfiniBand Modules User Manual 37 Installation of WinOF Windows Installation 5 After the following window appears read License Agreement carefully If you accept tick accept the
2. RO 25 2 6 3 identifying the GUID 2 424604 RR ORO Ee RR 26 2 6 4 The result of the command ibnetdiscover 30 3 Windows Installation 35 3 1 QVOFVIOW 0 Rau 35 3 1 1 Software Requirements 35 3 2 Installation of WinOF 35 4 IB Connection Blade Management 47 4 1 CONNECHON a ud i e uo qx Read Sue eg 47 4 1 1 Configuring the Switch for the First Time 48 4 1 2 Ethernet Connection via Management Blade aaa 52 4 1 3 ssh Conneccion an oh ae o 4d 3 wo os RU AC AC UU de EORR ROG 52 4 1 4 WebUl Conneclion_ xo odo mom aaa Ro opo RR 59 4 2 Confdguralol sss RUE SERO 54 4 2 1 Configuration of LAN interface 54 4 2 2 Save and restore the configuration 56 4 3 Basic Operation aa aaa RR XR 58 4 3 1 How to check System Information 58 4 4 Users and Privileges 59 4 5 CLIMOGES ae eG oo RSE aes Be ee 60 5 Troubleshooting 4 252292 xk 61 5 1 Check the Log File of OpenSM Linux only 61 5 2 IB Mezzanine 64 5 3 Connection Blade 76 79 4 User Manual BX900 BX400 InfiniBand Modules 1 Introduction This manual describes the
3. The PDF files of the manuals can also be downloaded free of charge from the internet The overview page showing the online documentation available on the internet can be found using the URL for EMEA market http manuals ts fujitsu com The PRIMERGY server documentation can be accessed using the Industry standard servers navigation option If you purchased the products from Fujitsu Japan please use the URL http jp fujitsu com platform server primergy manual 6 User Manual BX900 BX400 InfiniBand Modules Introduction Overview 1 4 Overview This manual explains the use of the software management Managed devices must have a CPU and a management board This board has an operating system and a toolset to operate manage and upgrade the device Further information is provided on the PRIMERGY ServerBooks DVD PRIMERGY BX900 52 Blade Server System Operating Manual PRIMERGY 400 51 Blade Server System Operating Manual PRIMERGY Server Systems RemoteDeploy PRIMERGY Blade Server Systems LAN Connection Blade ServerView User Guide For further information on updating BX components please refer to the Operating Manual of your BX system BX900 BX400 InfiniBand Modules User Manual 7 PRIMERGY 900 Introduction 1 5 PRIMERGY BX900 The PRIMERGY BX900 Blade Server system is a modular server system that integrates up to 18 server modules eight connection blade modules and two Management Modules
4. 5 Unmount mnt directory Lroot localhost mnt cd Croot localhost 1 umount mnt Croot localhost 1 6 Restart the OS root localhost 1 reboot 7 Check the modification of limits conf file root localhost 1 tail 2 etc security limits conf soft memlock unlimited hard memlock unlimited Lroot localhost 1 Please re install from step3 when the 2 messages above were not displayed 2 3 8 Update of This chapter describes the updating of OFED Only with RHEL5 7 or RHEL6 1 may be updated by the following steps gt Download OFED from Fujitsu Technology Solutions Web site http support ts fujitsu com com support downloads html gt Driver amp Downloads gt f you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads 2 3 8 4 Updating of OFED on RHEL5 7 1 Mount iso file and move to mnt In the following example the OFED is MLNX OFED LINUX 1 5 3 1 0 0 2 rhel5 7 x86_64 iso Croot localhost J mount o ro loop MLNX OFED LINUX 1 5 3 1 0 0 2 rhe15 7 x86 64 iso mnt root localhost 1 cd mnt root localhost 1 2 Force libibverbs and libibverbs devel packages to be uninstalled BX900 BX400 InfiniBand Modules User Manual 21 Installation Linux Installation root localhost mnt rpm e nodeps allmatches libibverbs devel root localhost mnt rpm e 5 allma
5. Execute Installer and enter y 1 1 1 mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y The following messages are displayed Add an option nodeps Delete scsi target utils from the package currently displayed and uninstall a package Please remove OFED RPMs coming from the Distribution Run rpm e allmatches rdma 1 0 14 e16 noarch libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs dapl rds tools ibutils infinipath psm libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs compat opensm libs dapl rds tools infinipath psm opensm libs libipathverbs dapl libibcm libibmad libibumad ibsim ibutils ibutils libs fcoe utils scsi target utils compat openmpi compat openmpi psm fcoe utils scsi target utils Some RPMs may depend on the RPMs above Please uninstall them manually BX900 BX400 InfiniBand Modules User Manual 19 Installation Linux Installation Uninstall the packages 1 1 mntl Croot localhost mnt rpm e nodeps allmatches rdma 1 0 14 e16 noarch li
6. PCI bus PCI bus E E E Figure 19 InfiniBand devices BX900 BX400 InfiniBand Modules User Manual 43 Installation of WinOF Windows Installation 11 After WinOF installation OpenSM should be set up The OpenSM service is registered to the Services Window after WinOF installation Double click OpenSM to open properties File Action View Ele 5 0 Services Local OpenSM Name ipion Status Startup 7 On As Microsoft iSCSI Initi Manages I Manual Local System Stop the service Gk Microsoft Software Manages s Manual Local System Pause the service 4 Multimedia Class Sc Enables rel Manual Local System SEM Net Tcp Port Sharin Provides Started Manual Local Service Maintains Started Automatic Local System Description Network Access Pr The Netwo Manual Network S OpensM Network Connections Started Manual Local System 10 Network List Service Identifies t Started Manual Local Service J Collects an Started Automatic Network 5 This servic Started Automatic Local Service Automatic Local System Automatic D Loc Manual Local Service Performance Logs Manual Local Service 3G Plug and Play Started A
7. Inserting Connection Blade into PRIMERGY 900 on page 9 and section Connecting Mezzanine Cards on Server Blade on page 10 14 User Manual BX900 BX400 InfiniBand Modules 2 Linux Installation 2 1 Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed A blade can be properly installed with all required InfiniBand drivers and software during Red Hat Enterprise Linux installation If you purchased the products from Fujitsu Japan you can download it from http jp fujitsu com platform server primergy downloads Please note that only Red Hat Enterprise Linux Version 5 7 or higher and 6 1 or higher are supported The chapter includes the following sections e Software Requirements on page 15 e OFED Installation on page 16 Updating Firmware after Installation on page 27 Subnet Manager on page 27 e Network Configuration Information on page 29 2 2 Software Requirements Required Disk Space for Linux Installation 400 MB Operating System Linux operating system x86 64 only Installer Privileges The installation requires administrator privileges on the target machine BX900 BX400 InfiniBand Modules User Manual 15 Installation Linux Installation 2 3 OFED Installation Please install OFED by the following steps after the OS installation by SVIM Server Installation Manager gt Download OFED from Fujit
8. co User Manual English FUJITSU PRIMERGY BX900 BX400 InfiniBand Modules 56Gb Connection Blade and Mezzanine Card V4 0 Linux Windows Edition April 2013 Comments Suggestions Corrections The User Documentation Department would like to know your opinion of this manual Your feedback helps us optimize our documentation to suit your individual needs Feel free to send us your comments by e mail to manuals ts fujitsu com Certified documentation according to DIN EN ISO 9001 2008 To ensure a consistently high quality standard and user friendliness this documentation was created to meet the regulations of a quality management system which complies with the requirements of the standard DIN EN ISO 9001 2008 cognitas Gesellschaft f r Technik Dokumentation mbH www cognitas de Copyright and Trademarks Copyright Fujitsu Technology Solutions GmbH 2013 All rights reserved Delivery subject to availability right of technical modifications reserved All hardware and software names used are trademarks of their respective manufacturers Contents 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 8 1 1 8 2 1 9 1 9 1 1 9 2 2 1 2 2 2 3 2 3 1 2 3 1 1 2 3 1 2 2 3 2 2 4 2 5 8 5 1 2 5 2 2 5 3 Introduction NR PIT 5 Notational Conventions 5 Terget Group 6 Documentation Overview 6 OVGIVIG
9. ipoib_start_adapter Error failed Table 8 source ipoib 72 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card Event ID Classification To do Error Level 0x0003 The message changes in Error each event 0 0004 mlx4 bus has started in Error non operational mode 0x0005 mlx4_bus has failed to start Error even in non operational mode n Look into the the previous error messages 0 0007 MAP_FA command failed with Error error 2 n The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x0008 RUN_FWcommand failed f you purchased the products Error with error 2 n Lae The adapter card is non 9 d functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0 0009 QUERY_FW command failed Error with error 2 n The adapter card is non functional 9en The IB Mezzanine card must be replaced Please consult technical support ts fujitsu com Most likely a FW problem n Please burn the last FW and restart the mlx4 bus driver Table 9 source mlx4 bus BX900 BX400 InfiniBand Modules User Manual 73 IB Mezzanine Card Troubleshooting Event ID Error Level Classification To do 0x000B Error QUERY_DEV_CAP command failed with error 962 9on The adapter card is non funct
10. necessary to replace the server blade NO Place the IB Mezzanine card again 68 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card Work after IB Mezzanine card has been replaced Linux 1 The server blade is mounted on the main body of BX900 and the power supply is turned on 2 Linux is started sbin Ispci command is executed by the root account and it is confirmed that the IB Mezzanine card is recognized Execution result 10 00 0 InfiniBand Mellanox Technologies MT27500 Family ConnectX 3 Subsystem Mellanox Technologies MT27500 Family LConnectX 31 30 00 0 InfiniBand Mellanox Technologies MT27500 Family LConnectX 31 Subsystem Mellanox Technologies MT27500 Family ConnectX 3 The enhancing board slot where this card is mounted can be confirmed by the value of Bus 10 00 0 Enhancing board slot 1 30 00 0 Enhancing board slot 2 3 Refer to section Configuration sheet on page 30 update a GUID table BX900 BX400 InfiniBand Modules User Manual 69 IB Mezzanine Troubleshooting IB Mezzanine card driver messages Windows Server 2008 R2 SP1 Confirm the message of the IB Mezzanine card driver by referring to the event log You refer to the event log using the Event Viewer Event ID Classification To do Error Level 0x0041 Failed to open Error Channel Adapter 0x0042 xxx Failed to alloca
11. 4 3 1 Howto check System Information This section describes how to check system information for this product The firmware version can be checked for this product Requires running in enable or config mode Prompt of a command in Enable mode hostname Prompt of a command in Config mode hostname config show system guid This command shows GUID lt BX900S2 CB3 gt standalone master show system guid 00 02 C9 03 00 5D 0C A0 show asic version This command shows Switch X Firmware version lt BX900S2 CB3 gt standalone master show asic version SX 9 0 3052 show version This command shows System Information lt BX900S2 CB3 gt standalone master show version Product name SX_PPC_M460EX Product release SX_3 1 0936 Build ID 1 dev Build date 2011 10 02 15 06 46 58 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Users and Privileges Target arch Target hw Built by Uptime Product model Host ID System memory Swap umber of CPUs CPU load averages ppc m460ex alia fit15 22m 20 100s ppc 0002C95EB028 75 MB used 1952 MB free 2027 MB total 0 MB used 0 MB free 0 MB total 1 0 00 0 00 0 00 4 4 Users and Privileges The CLI is entered via the Telnet ssh interface of the management blade There are two user roles or account types admin and monitor As admin the user is privileged to run all the available commands A
12. MMB The InfiniBand connection blade module provides networking and Switch functions to PRIMERGY BX900 Blade Server system The Management Module offers a single point of control for the PRIMERGY BX900 Blade Server system Figure 1 PRIMERGY BX900 Blade Server 8 User Manual BX900 BX400 InfiniBand Modules Introduction Inserting Connection Blade into PRIMERGY BX900 1 6 Inserting Connection Blade into PRIMERGY BX900 In order to support 4x FDR InfiniBand 4 lanes per one port are required in backplane Due to this reason the form factor of connection blade is double wide single height The eject lever of the connection blade has a green touch button The places at which all components may be touched to prevent them from being damaged when they are assembled or disassembled are also marked green Rear view of enclosure Fabric 1 Fabric 2 Fabric 3 Fabric 4 Figure 2 Rear view of enclosure BX900 BX400 InfiniBand Modules User Manual 9 Connecting Mezzanine Cards on Server Blade Introduction 1 7 Connecting Mezzanine Cards on Server Blade The figure below indicates the connection relation between connection blade CB and Mezzanine Card on server blade InfiniBand IB Mezzanine Cards can be set to both Mezzanine slot and Mezzanine slot When Mezzanine Card is installed in Mezzanine II slot the dual port of the card will be available and one of two ports will be connected to CB in CB5 6 th
13. buf 02x 08 Failed to initialize user access region table aborting technical support Q ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Couldn t map EQ doorbell for EQN 0x 06x Couldn t map interrupt clear register aborting note1 7 Couldn t allocate FW area aborting Failed to map MCG context memory aborting Table 7 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux 66 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card No Classification To do 9 Failed to allocate driver access region aborting 10 Failed to initialize memory region table aborting 11 Failed to initialize event queue table aborting 12 Failed to switch to event driven firmware commands aborting The IB Mezzanine card must 13 NOP command failed to generate interrupt don consul IRQ 95d aborting technical support ts fujitsu com 14 Failed to initialize completion queue table f you purchased the products aborting from Fujitsu Japan please 15 Failed to initialize shared receive queue consult the system engineer table aborting 16 Failed to initialize queue pair table aborting 17 Failed to initialize multicast group table aborting 18 Failed to reset Mezzanine card aborting 19 Failed to init command interface aborting 2
14. configuration installation and basic use of the InfiniBand connection blade and Mezzanine card software The package includes tools for general and advanced management functionality It also describes the installation of OFED OpenFabrics Enterprise Distribution for Linux the identification of the Mezzanine card and Win OpenFabrics instal lation for Windows The hardware installation of the InfiniBand connection blade is described in the BX900 S2 or BX400 S1 System Unit Operation manual The hardware installation of the Mezzanine card is described in the Upgrade and Maintenance Manual of the BX92x S3 server blades 1 1 Notational Conventions The following notational conventions are used in this manual A Caution This symbol points out hazards that can lead to personal injury loss of data or damage to equipment i This symbol highlights important information and tips This symbol refers to a step that you must carry out in order to continue with the procedure italic Commands menu items names of buttons options file names and path names are written in italic letters in the text variable Marked variables that must be replaced by current values fixed font System output is written using a fixed font semi bold Commands to be entered through the keyboard are written in a fixed font semi bold fixed font Key symbols Keys are presented according to their representation on the
15. ib0 in order to avoid this problem 26 User Manual BX900 BX400 InfiniBand Modules Linux Installation Updating Firmware after Installation 2 4 Updating Firmware after Installation If you wish to burn newer firmware you have to download it from Fujitsu Technology Solutions Web site http support ts fujitsu com com support downloads html gt Driver amp Downloads If you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads 2 5 Subnet Manager At least one Subnet Manager SM is present on each subnet Each SM resides on a port of IB Mezzanine card or IB connection blade When there are multiple SMs on a subnet the master SM is one and the remaining SMs become standby SMs The master SM has a role to initialize and configure an Infiniband subnet OpenSM is an InfiniBand compliant SM The following sections describe how to use OpenSM 2 5 1 5 OpenSM is one Subnet Manager distributed with RedHat This chapter describes how to configure which server becomes the master SM Please set up the server which will be the master using the following reference Regarding the other settings of OpenSM please refer to the command help of opensmd Basically a server with the lowest GUID will be the master SM within a subnet if you intend to choose the master SM you can configure the priority which range is from 0 to 15 0 is the lowest and 15 is the hig
16. loading list is useful for the maintenance 38 User Manual BX900 BX400 InfiniBand Modules 3 Windows Installation 3 1 Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed A server can be properly installed with all required InfiniBand drivers and software during Windows HPC Server 2008 R2 SP1 installation or afterwards by using the Win OpenFabrics installation or OpenSM setup Please note that only Windows Server 2008 R2 SP1 and Windows HPC Server 2008 R2 SP1 are supported 3 1 1 Software Requirements Required Disk Space for Windows Installation 100 MB Operating System Windows HPC Server 2008 R2 SP1 Installer Privileges Installation requires administrator privileges on the target machine 3 2 Installation of WinOF Please install WinOF by the following steps after the OS installation Download OFED from Fujitsu Technology Solutions Web site Attp support ts fujitsu com com support downloads htm gt Driver amp Downloads If you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads BX900 BX400 InfiniBand Modules User Manual 35 Installation of WinOF Windows Installation 1 After inserting InfiniBand cards make sure that Other devices gt Unknown device is shown in the Device Manager Device Manager File Action View Help 9110 19
17. sure that the InfiniBand devices are recognized properly in Device Manager E Server Manager File Action View Help 2112110 15116 8 gj Roles 8 gj Features Diagnostics amp Event Viewer E Performance 19 088 Computer Ega Disk drives Display adapters 43 DVD CD ROM drives mM Human Interface Devices di 95 DAH InfiniBand Channel Adapters E Storage Mellanox ConnectX Virtual Channel Adapter Mellanox ConnectX Virtual Channel Adapter 8 20 Keyboards B n Mice and other pointing devices H Monitors E A Network adapters e Intel R 82575EB Gigabit Backplane Connection Intel R 82575 Gigabit Backplane Connection 2 Intel R 82575EB Gigabit Backplane Connection 3 Intel R 82575EB Gigabit Backplane Connection 4 Mellanox IPoIB Adapter 2 gt IPoIB Adapter 3 Mellanox IPoIB Adapter 4 amp LP Ej Processors lt gt Storage controllers 19 088 System devices E Universal Serial Bus controllers Figure 29 Device manager 3 Refer to section Configuration sheet on page 30 update a GUID table BX900 BX400 InfiniBand Modules User Manual 75 IB Connection Blade Troubleshooting 5 3 IB Connection Blade IB Connection Blade Message Connect to IB connection blade via MMB Console Redirection The reference to the var log messages log file uses the text editor et
18. to connect to IB connection blade via telnet ssh with CLI An example that an IP address 192 168 1 100 24 is assigned to management console and 192 168 1 10 24 to IB connection blade is illustrated below IP address 192 168 1 10 24 5 onsole 900 52 chassis back face example BX900 52 chassis Figure 26 Ethernet connection 4 1 3 ssh connection The user can execute the commands via ssh from the management console The method of connecting to IB connection blade via ssh is described below Runaterminal software on the management console gt Enter the command including the IP address referred to above ssh I admin 192 168 1 10 and press the Enter key The user connects to IB connection blade via 557 gt Enter the password admin and press the Enter key 52 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Connection LibuserGlocalhost 1 ssh 192 168 1 10 Mellanox MLNX OS Switch Management Password Mellanox Switch BX900S2 CB3 standalone master gt 4 1 4 WebUI Connection gt Start a Web browser Google Chrome Microsoft Internet Explorer 7 0 or Mozilla Firefox 3 0 or above gt Enter as URL the following http lt switch_IP_address gt where lt switch_IP_address gt is the IP address of the switch or its DNS name gt You will receive the login window for remote management of the switch The following
19. to specify the relevant IB connection blade refer to section The result of the command ibnetdiscover on page 35 The target port is the one that removed the port number from original ibnetdiscover command output when the system was set up normally 2 connection blade If you have changed the above IB cable and still receive the same messages you need to change the IB connection blade BX900 BX400 InfiniBand Modules User Manual 63 IB Mezzanine Card Troubleshooting 5 2 IB Mezzanine Card You can confirm place and time the error occured by referring to the following items on a server blade No Confirmation item 1 Active link LED for IB Mezzanine card on the front side of the blade 2 IB Mezzanine card driver message Active link LED for IB Mezzanine card When an active link LED for IB Mezzanine card on the front side of the server blade goes dark you can confirm the place where the error occured using the following confirmation contents 64 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card No Confirmation item Judgment To do Is the server blade turned 5 Go to No 2 on NO Turn on the server blade Are the LEDs of all server YES Confirm OpenSM running on 2 blades turned off the MASTER OpenSM server NO Go to No 3 Is the LED of one server YES Go to No 4 3 blade in the system turned
20. two LEDs are located at the left side of the front panel Figure 6 LED ID Status LED Position Color Function ID Left side top Blue Identify active Identify LED Off Identify inactive Status Left side bottom Green Power on Health LED Amber blinking Error Off Power off Table 1 Connection blade LEDs User Manual BX900 BX400 InfiniBand Modules Introduction Description of InfiniBand Connection Blade Port LEDs For every external port a pair of LEDs exists Figure 7 Port Leds LED Position Color Function P Above port right Green Link up Port Physical Link Off No Link L A Above port left Yellow Link up Port Logical Link Yellow blinking Link active Off Link down Table 2 Port LEDs BX900 BX400 InfiniBand Modules User Manual 13 Technical Data Introduction 1 9 Technical Data 1 9 1 IB Connection Blade Internal Ports 18 External Ports 18 Lanes per Port 4 4x Max Speed per Lane 14Gbps FDR Product Weight 9 4 Lbs 4 3 kg Form Factor Double wide Height 27 9 1 1 Width 388 2 mm 15 3 Depth 267 9 mm 10 6 1 9 2 IB Mezzanine Card Form factor BX900 Standard form factor Server blades supported BX920 S3 BX924 S3 Max number of Mezzanine Cards per blade 2 Host interface x8 PCI Express Gen3 Interface to midplane 2 4x14Gbps FDR Consider restrictions mentioned in section
21. 0 4 Enter ip default gateway command to configure the default gateway BX900S2 CB3 standalone master config ip default gateway 192 168 1 1 BX900S2 CB3 standalone master config show ip default gateway Active default gateways 192 168 1 1 interface mgmt0 54 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Configuration 5 Enter show command to check the IP address BX900S2 CB3 standalone master config show interfaces Admin up Link up P address Netmask enabled Autoconf enabled Autoconf route Autoconf privacy addresses address Speed Duplex nterface type nterface source MTU HW address Comment RX bytes RX packets RX mcast packets RX discards RX errors RX overruns RX frame nterface ifindex 5 2 oo oa nterface mgmtO state yes yes 192 168 1 10 255 255 255 0 5 5 1 fe80 202 c9ff febe b028 64 100Mb s auto full auto ethernet physical 1500 00 02 9 5 28 350137 TX bytes 2472267 4020 TX packets 8360 TX discards TX errors TX overruns TX carrier TX collisions TX queue le 1000 BX900 BX400 InfiniBand Modules User Manual 55 Configuration IB Connection Blade Management 4 2 2 Save and restore the configuration The configuration data can be saved and restored by configuratio
22. 0 InfiniBand Modules User Manual 49 Connection IB Connection Blade Management Wizard Session Display Example Comments You have entered the following infor mation 1 Hostname lt switch name gt 2 Use DHCP interface yes 3 Admin password Enter to leave unchanged CHANGED To change an answer enter the step number to return to Otherwise hit lt enter gt to save changes and exit Choice lt Enter gt Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configuration mode Launching CLI lt switch name gt standalone master gt The wizard displays a summary of your choices and then asks you to confirm the choices or to re edit them Either press lt Enter gt to save changes and exit or enter the configuration step number that you wish to return to Table 4 Configuration Wizard Session IP Configuration by DHCP Example Mellanox configuration wizard Do you want to use the wizard for initial configuration y Step 1 Hostname switch 5eb028 BX900S2 CB3 Step 2 Use DHCP mgmtO interface no Step 3 Use zeroconf on mgmtO interface no Step 4 Primary IPv4 address and masklen 0 0 0 0 0 192 168 1 10 Step 5 Netmask or mask length 255 255 255 0 Step 6 Default gateway Step 7 Primary DNS server 50 User Manual BX900 BX400 InfiniBand Modules IB
23. 0 Only d UAR pages need more than 128 note1 21 Profile requires Ox llx bytes won t fit in note1 Ox llx bytes of context memory Table 7 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux BX900 BX400 InfiniBand Modules User Manual 67 IB Mezzanine Card Troubleshooting COMMAND aborting No Classification To do 22 Couldn t allocate memory to save Mezzanine card PCI header aborting 23 Couldn t save Mezzanine card header aborting 24 Couldn t map Mezzanine card reset register aborting The IB Mezzanine card must be 25 PCI device did not come back after reset replaced Please consult aborting technical support Q ts fujitsu com 26 Couldn t restore Mezzanine card PCI If you purchased the products Express Device Control register aborting from Fujitsu Japan please consult 27 Couldn t restore Mezzanine card PCI the system engineer Express Link control register aborting 28 Couldn t restore Mezzanine card reg aborting 29 Couldn t restore Mezzanine card Table 7 Error messages of IB Mezzanine card driver Red Hat Enterprise Linux note1 When a message occurs frequently confirm the following No Confirmation contents Judgment To do 1 Is the IB Mezzanine card YES mounted rigidly The IB Mezzanine card must be replaced When the message is displayed after the IB Mezzanine cards are exchanged it is
24. 03bfa8 H 0002c9030003bf86 2 2c9030003bf89 H 0002c9030003bf9e 2 2 c9030003bfa0 H 0002c903000412ec 2K2c903000412ee H 0002c90300041414 2 2c90300041416 f H 0002c90300041408 2 2c90300041 40a H 0002c903000413b8 2 2c903000413ba H 0002c9030003bf7a 2 2c9030003bf7c 0002 903000413 0 282 903000413 2 vendid Ox2c9f devid 0xbd36f sysimgguid 0x2c9020040c7901 switch 0 2 9020040 790 2 9020040 790 1 Switch 36 S 0002c9020040c790 Figure 11 ibnetdiscover command GUID list and arrangement drawing of devices 8 0002c9020040c7a8 21 8 0002c9020040c7a8 120 8 0002c9020040c7a8 19 H 0002c903000447b0 2 2c903000447t2 H 0002c90300041 EL 5 0002 9020040 23 S 0002c9020040c7a8 22 3 7 bx900 04 IB SW CB3 lid 9 4xFDRT 3 bx900 04 IB SW CB3 lid 9 4xFDRT bx900 04 IB SW CB3 1id 9 4xFDRY 7 bx900 02 17 HCA 1 4id 224xFD bx900 02 16 HCA 1 1id 21 4xFD 3 7 bx900 02 14 HCA 1 4id 11 4xFD bx900 02 11 HC 1 lidAA T 4AxFDR 3 bx900 02 10 HCA 1 1id 18 4xFD bx900 02 09 HCA 1 1id 13 4xFD bx900 02 07 HCA 1 Jid 12 4xFD 3t bx900 02 06 HCA 1 lidAA94xFD bx900 02 05HCA 1 lid 3 4xFD 70 900 02 04 2 4110 2 4 bx900 02 03 HCA 1 1i
25. 101 22 1 mezz 2 GUID Hostname Static iRMC ethO ett eth2 il DHCP 0002c9030003bf7a 0002c9030003bf6 1 5 4 bx920 02 01 static 510102 0002 903000413 0 0002c90300041514 RHEL5 4 bx920 02 02 static 106 107 108 109 110 510103 510104 510105 510106 510107 510108 510109 510110 510111 510112 510113 510114 510115 510116 510117 510118 Static DHCP static static 34 hostname CBS MMB1 Agent 1 5 Figure 9 Example of configuration sheet 2 6 3 Identifying the GUID The IB Mezzanine card has 4 different GUIDs Globally Unique Identifier Node GUID System GUID GUID for Port1 and GUID for Port2 The Node GUID is listed on the label on the back of the product BX900 BX400 InfiniBand Modules User Manual 31 Network Configuration Information Linux Installation Figure 10 Label on the back of the IB Mezzanine Card How to identify the node GUID gt Log in to the server blade as root gt Run the open terminal or telnet ssh gt Execute the ibstat command and identify the node GUID Node GUID 0 The ibstat command is available after installation of OFED or WinOF In case one Mezzanine Card is installed Mezz slot1 or slot2 the Mezzanine Card is named mlx4 0 In case two Mezzanine Cards are installed
26. 6 7 Connect IB Cable to the same port as before 8 Confirm that LED of P and L A lights It might take about one minute to lighting LED 9 Execute show interface ib0 command from the console of the IB connection blade and comfirm that Link up yes Example 90052 standalone master config show interfaces 1 0 Interface ibO state Admin up yes Link up no IP address Netmask IPv6 enabled yes Autoconf enabled no Autoconf route yes Autoconf privacy no Speed 10 Gb sec 1X QDR Duplex full Interface type ib Interface ifindex 4 Interface source physical MTU 2044 HW address 00 00 00 02 00 00 00 00 00 00 00 00 00 02 c9 03 00 5d 0c a0 Comment RX bytes 0 TX bytes 0 RX packets 0 TX packets 0 mcast packets 0 TX discards 0 RX discards 0 TX errors 0 RX errors 0 TX overruns 0 RX overruns 0 TX carrier 0 RX frame 0 TX collisions 0 TX queue len 256 78 User Manual BX900 BX400 InfiniBand Modules Index B 400 51 5 900 2 5 C CLI 59 Command line interface 59 configuration sheet 24 D disk Space for installation 15 disk space for Windows installation 35 E ethernet connection 52 G GUID 26 1 ibnetdiscover 24 ibstat 23 InfiniBand connection blade connection 47 installer privileges 15 Windows 35 L LEDs 12 Linux 15 log file 61 M Management Blade 47 login 47 master 23 Mezzanine card exchange 68 G
27. B Console Redirection Execute the command swinfo lt BX900S2 CB3 gt swinfo lt BX900S2 CB3 Information GUID 0x0002c9020040c7a8 MAC 00 02 c9 11 20 5a 154 Firmware revision 7 2 326 CPU Firmware revision Mellanox release 1 405 1 0 2 build 2009 07 19 In the example above the GUID of IB connection blade is 0x0002c9020040c7a8 Excute the command ibnetdiscover on the server blade by use of mmap txt 36 User Manual BX900 BX400 InfiniBand Modules Linux Installation Network Configuration Information po gtt bx900 02 01 FF ibnetdiscover node name map rootimmap txt 3t Topology file generated on SatAug 1 01 27 33 2009 1 Max of3 hops discovered Initiated from node0002c90300041 3c0 port 0002c903000413c2 p nj 1 2 bx900 04 IB SW 53169442084 3e bx900 04 IB SW CB3 lid 9 4xF DRI S 0002c9020040c790 21 5 S 0002c9020040c790 20 5 0002 9020040 790 19 0002 9030003082 1212 90900030184 26 H 0002c903000413fc 2 2c903000413fe H 0002c903000412fc 2 2c903000412fe H 0002c90300041460 2 2c90300041482 H 0002c9030003bf96 2 2c9030003bf98 H 0002c9030003d138 2 2c9030003d13a bx900 02 13HCA 1 lid 4 4xFDR H 0002c9030003bf92 2 2c9030003bf94 H 0002 9030003bfba 2 2c9030003bfbc H 0002c9030004136 8 2 2c903000413ea H 0002c9030003bfa6 2 2c90300
28. Connection Blade Management Connection Step 8 Domain name Step 9 Enable IPv6 yes Step 10 Enable IPv6 autoconfig SLAAC on mgmtO interface no Step 11 Admin password Enter to leave unchanged You have entered the following information 1 Hostname 900 2 Use DHCP mgmtO interface no Use zeroconf on mgmtO interface no Primary IPv4 address 192 168 1 10 2 3 4 5 Netmask or mask length 255 255 255 0 6 Default gateway 7 Primary DNS server 8 Domain name 9 Enable IPv6 yes 10 Enable IPv6 autoconfig SLAAC on mgmtO interface 11 Admin password Enter to leave unchanged unchanged To change an answer enter the step number to return to Otherwise hit enter to save changes and exit Choice Configuration changes saved To return to the wizard from the CLI enter the configuration jump start command from configure mode Launching CLI If you want to rerun the wizard run the following commands BX900S2 CB3 standalone master gt enable BX900S2 CB3 L standalone master configure terminal BX900S2 CB3 Lstandalone master config configuration jump start BX900 BX400 InfiniBand Modules User Manual 51 Connection IB Connection Blade Management 4 1 2 Ethernet Connection via Management Blade IB connection blade provides the function of Ethernet connection via the Management LAN port on a Management Blade Ethernet connection enables the user
29. NO Go to No 5 off 4 Is the IB Mezzanine card YES 1 Replace the IB Mezzanine mounted rigidly card 2 If the LED is still turned off after replacing the IB Mezzanine card replace the server blade 3 If the LED is still turned off after replacing the server blade replace the IB connection blade 4 If the LED is still turned off please consult technical supportQ ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer NO Mount the IB Mezzanine card more rigidly 5 Does IB connection blade YES Go to No 4 work normally Please refer NO Replace the IB connection to section IB Connection blade Please consult Blade on page 76 technical supportQ ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 6 Indication lamps BX900 BX400 InfiniBand Modules User Manual 65 IB Mezzanine Troubleshooting IB Mezzanine card driver messages Linux Confirm the error message of IB Mezzanine card driver by referring to the var log messages og file You refer to the log file using the text editor etc No Classification To do 1 Installed FW has unsupported command interface revision d Installed FW version is d d 03d The IB Mezzanine card must This driver version supports only revisions be replaced Please consult d to 2 This driver version supports only revisions to 96d Internal error detected
30. P address to it 16 Run the command prompt as the Administrator privilege and execute the following command to enable Network Direct ndinstall i Administrator Command Prompt inistrator gt ndinstal i Installing IBAL prov Current dminist rator Figure 22 Installation of Network Direct 1 BX900 BX400 InfiniBand Modules User Manual 45 Installation of WinOF Windows Installation 17 In order to check if Network Direct is enabled properly execute the same command again The message Installing IBAL provider already installed is shown if it is enabled properly ninist rator Figure 23 Installation of Network Direct 2 46 User Manual BX900 BX400 InfiniBand Modules 4 Connection Blade Management 4 1 Connection In this chapter the connection of IB connection blade via Management Blade is explained Follow the procedures below Configurations for connecting to IB connection blade are explained in section Configuration on page 54 1 Login to Management Blade via ssh Please refer to Server View Management Blade to see about the connection and the booting up of Management Blade CLI 2 Select 3 Set Console Redirection Timeout and press the Enter key 3 Select 1 Console Redirect Connection Blade and press the Enter key 13 Console Redirect Connection Blade 2 Set Return Hotkey Ctritfa character 0 3 Set Console Redirecti
31. UID 26 hardware installation 5 technical data 14 N network configuration diagram 24 notational conventions 5 5 installation 16 OpenFabrics Enterprise Distribution 5 OpenSM 22 61 operating system 15 P PRIMERGY 400 5 PRIMERGY 900 5 PRIMERGY BX900 Blade Server 8 Q QSFP 11 Quad Small Form factor Pluggable 11 R repair center 61 BX900 BX400 InfiniBand Modules User Manual 79 show asic version 58 show system guid 58 SM 22 software requirements Windows 35 ssh 47 ssh connection 52 standby 24 Subnet Manager 22 subnet manager 22 OpenSM 61 support 61 switch LEDs 12 technical support 61 Windows installation 35 80 User Manual BX900 BX400 InfiniBand Modules
32. W Gee AR ee an 7 PRIMERGY 900 8 Inserting Connection Blade into PRIMERGY 900 9 Connecting Mezzanine Cards on Server Blade 10 Description of InfiniBand Connection Blade 11 GA 11 EEDS uou domu Rue Wu BOE Wen 12 Jecnnical Data SKS eR EET Xx aoro y 14 IB Connection Blade 1 14 IB Mezzanine Card 14 Linux Installation ln 15 DUGIVIBI bee eek OM eee RES oe RRR 15 Software Requirements 15 OFED Installation 16 Update or OFED pay eh eek T 16 Updating of on RHEL5 7 16 Updating of on BRHELG 1 18 Post installation notes ai comaa ee ds de eee 20 Updating Firmware after Installation 22 Subnet Manager 3 x x3 22 uio dk x ouk RGR SSSR OX UR EUR Rew OR 22 Confirmation of OpenSM running 23 Confirmation of the master SM 23 BX900 BX400 InfiniBand Modules User Manual 3 Contents 2 6 Network Configuration Information 24 2 6 1 Network configuration diagram 24 2 6 2 Configuration
33. and Accounting security services such as authentication methods and authorization RADIUS is used to manage default RADIUS Remote Authentication Dial In User Service settings and RADIUS servers TACACS is used to manage default TACACS Terminal Access Controller Access Control System Plus settings and TACACS servers LDAP is used to manage default LDAP Light weight Directory Access Protocol settings and LDAP servers 4 6 7 PORTS MENU _ The PORTS menu displays port state and enables some Mss configuration attributes of a selected port by left clicking on the desired port It also enables the modification of the port configuration such as port enabled disabled port speed MTU size and port VLs A graphical display of traffic over time last hour or last day through the port is also available 68 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Starting MLNX OS Web User Interface 4 6 8 STATUS MENU The Status menu makes available the following sub menus listed in order of appearance from ges top to bottom Summary displays general information about the switch system and the MLNX OS SwitchX Profile and Capabilities image including current date and time Temperature hostname uptime of system installed MLNX Power Supplies OS SwitchX image version system memory ENS CPU load averages etc CPU Load System Capabiliti
34. ation 4 Execute Installer again and enter y Croot localhost 1 mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y Starting MLNX OFED LINUX 1 5 3 3 0 0 installation Installing mInx ofa kernel RPM Preparing HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE mlnx ofa kernel fHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installing kmod mlnx ofa kernel RPM 3 Installation finished successfully Configuring etc security limits conf Please reboot your system for the changes to take effect root localhost 1 5 Unmount mnt directory Croot localhost 1 cd Croot localhost 14 umount mnt Croot localhost 1 6 Restart the OS Croot localhost 1 reboot 7 Check the modification of limits conf file rootalocalhost 1 tail 2 etc security limits conf soft memlock unlimited hard memlock unlimited root localhost 1 Please re install from step3 when the 2 messages above were not displayed 18 User Manual BX900 BX400 InfiniBand Modules Linux Installation OFED Installation 2 3 2 OFED installation on RHEL6 2 1 Mount iso file and move to mnt Croot localhost 14 mount ro loop MLNX OFED LINUX 1 5 3 3 0 0 rhel6 2 x86 64 150 mnt root localhost 1 rootaGlocalhost 1 4 mnt
35. bibverbs libibverbs utils libmthca 116 1 4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs dapl rds tools ibutils infinipath psm libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs compat opensm libs dapl rds tools infinipath psm opensm libs libipathverbs dapl libibcm libibmad libibumad ibsim ibutils ibutils libs fcoe utils compat openmpi compat openmpi psm fcoe utils Execute Installer again and enter y Croot localhost 1 mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y Starting MLNX OFED LINUX 1 5 3 3 0 0 installation Installing mInx ofa kernel RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI mInx ofa kernel THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installing kmod mlnx ofa kernel RPM Installation finished successfully Configuring etc security limits conf Please reboot your system for the changes to take effect Following RPMs coming with the Linux Distribution were removed due to dependencies please reinstall them if required scsi target utils root localhost 1 20 User Manual BX900 BX400 InfiniBand Modules Linux Installation OFED Installation
36. both of Mezz slot1 and slot2 the Mezzanine Cards are named below Mezz slot Mezzanine Card name 1 mlx4 0 2 mlx4 1 32 User Manual BX900 BX400 InfiniBand Modules Linux Installation Network Configuration Information In the following example the node GUID of Mezz slot1 is 0002c903002ff3e0 and Mezz slot2 is 0002c903002ff2a0 Lroot localhost ibstat CA mlx4 O CA type 4099 Port 1 Port 2 umber of ports 2 Firmware version 2 9 8220 Hardware version 0 ode GUID 0x0002c903002ff3e0 System image GUID 0x0002c903002ff3e3 State Down Physical state Polling Rate 40 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02514868 Port GUID 0x0002c903002ff3el State Initializing Physical state LinkUp Rate 40 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02514868 Port GUID 0x0002c903002ff3e2 BX900 BX400 InfiniBand Modules User Manual 33 Network Configuration Information Linux Installation CA mlx4 1 CA type 4099 Number of ports 2 Firmware version 2 9 8220 Hardware version 0 Node GUID 0x0002c903002ff2a0 System image GUID 0x0002c903002ff2a3 Port 1 State Down Physical state Polling Rate 40 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02514868 Port GUID 0x0002c903002ff2al Port 2 State Down Physical state Polling Rate 40 Base lid 0 LMC 0 SM lid 0 Capability mask 0x02514868 Port GUID 0x0002c903002ff2a2 Lro
37. c No Classification To do 1 Installed FW has unsupported command interface revision d The IB connection blade must Installed FW version is d d 03d be replaced This driver version supports only Please consult revisions d to d technical support ts fujitsu com 2 This driver version supports only revisions d to d If you purchased the products from Fujitsu Japan please 3 Internal error detected consult the system engineer buf 02x 08 Table 11 IB connection blade messages SNMP Trap messages When SNMP Trap of the IB connection blade is set to enable following snmp trap is transmitted to SNMP Manager No SNMP Trap OID Content and Action 1 asicOverTemp 1 3 6 1 4 1 33049 2 2 2 3 Content The temperature of asic is too high Action Confirm that ambient temperature of BX900 or BX400 system does not have abnormality When ambient temperature does not have abnormality the IB connection blade must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 12 snmp trap 76 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Connection Blade No SNMP Trap OID Content and Action lowPower 1 3 6 1 4 1 33049 2 2 2 4 Content The MLNX OS management appliance has a low power budget Action Confirm that PSU or Power S
38. d hard memlock unlimited root localhost 1 Please re install from step3 when the 2 messages above were not displayed 2 3 4 Post installation notes How to change MR This section describes how to set mlx4 parameters Add line to file If you set to RHEL6 1 MellanoxOFED please add line to etc modpobe d mlx4 conf If there is mix4 conf please create the file How to set mlx4 parameters options mlx4 core parameter value BX900 BX400 InfiniBand Modules User Manual 25 Installation Linux Installation mlx4_core Parameters gt log_num_mtt Log maximum number of memory translation table segments per HCA default is 20 max is 20 gt log_mtts_per_seg Log number of MTT entries per segment 1 7 int default is 3 max is 7 Example of etc modpobe d mlx4 conf Add the following parameters options mlx4 core log num mtt 20 log mtts per seg 4 After the parameter has been changed you will need to reboot IPoIB adapters parameter limitation If you create or edit the ifcfg ib files to configure an IPoIB the parameter CONTROLLED should be set to The limitation is applied only to RHEL6 Example etc sysconfig network scripts ifcfg ibO DEVICE ibO IPADDR 192 168 210 100 PREFIX2255 255 255 0 BROADCAST 192 168 210 255 NM CONTROLLED no ONBOOT yes IPoIB function gets unavailable if ib0 is configured on BX92xS3 servers in BX900 S2 chassis Don t configure ifcfg
39. d libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel dapl dapl devel dapl utils perftest rds tools ibutils infiniband diags qperf libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel compat dapl utils dapl dapl devel dapl utils perftest rds tools infiniband diags qperf ofed docs openib opensm libs openmpi openmpi libs ibutils ibutils libs mpitests openmpi ofed docs 1 4 1 2 e15 Some RPMs may depend on the RPMs above Please uninstall them manually Uninstall the Packages Croot localhost 1 rpm e nodeps allmatches openmpi libcxgb4 ofed docs 1 4 1 2 e15 openib 1 4 1 6 e15 libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel dapl dapl devel dapl utils perftest rds tools ibutils infiniband diags qperf libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel compat dapl utils dapl dapl devel dapl utils perftest rds tools infiniband diags qperf ofed docs openib opensm libs openmpi openmpi libs ibutils ibutils libs mpitests openmpi ofed docs 1 4 1 2 e15 BX900 BX400 InfiniBand Modules User Manual 17 Installation Linux Install
40. d 16 4xFD 3 7 bx900 02 01 HCA 1 lid 1 4xFD IDEO eres Bae bx900 04 01 HCA 1 id 20 4xQDRY bx900 02 IB SW CB3 1id 14 4xQDRq bx900 02 IB SW CB3 id 14 4xQDRG bx900 02 IB SW CB3 lid 14 4xQDRTI 3 bx900 02 IB SW CB3 lid 14 4xQDRTI 3 bx900 02 IB SW CB3 lid 14 4xQDRTI 3 bx900 02 18 HCA 1 lid 7 4xFDRfI 3 7 bx900 02 15 HCA 1 lid 54xFDR px900 02 12 HCA 1 lid 10 4xFDR bx900 02 08 HCA 1 lid 8 4xFDRI bx900 02 02 HCA 1 1id 6 4xF DRI BX900 BX400 InfiniBand Modules Us er Manual 37 Network Configuration Information Linux Installation The result format of ibnetdiscover 1 2 Provide the GUID list file mmap txt in advance Search for the GUID obtained by the command swinfo in the result of ibnetdiscover S indicates IB connection blade see RED underline There is a string lid 14 in the same line as the GUID referred above 14 indicates the LID of IB connection blade see GREEN marker The leftmost figure in each line 23 to 1 indicate the port number which is connected to a device The maximum number of the port is 36 Thus if all ports are connected 1 to 36 is displayed In this example there are connections as follows see a 4 QDR indicates the Quad Data Rate connection In case this value is SDR or DDR or QDR there is something wrong with the cable connection see b GUID list of IB connection blade the layout chart and rack
41. e other one will be CB7 8 While installed in Mezzanine slot 1 only one port will be utilized and connected to the CB in CB3 4 When the two connection blades are installed in CB5 6 and CB 7 8 other types of connection blades like SB11 SB11A can be used in CB1 2 3 4 and SBAX2 and Brocade FC connection blade in CB3 4 in parallel with IB connection blade Server blade like BX920 S3 BX924 S3 _ D wde IB SW 78 _ 4 4 5 11 5811 SBAX2 8GFC SW et ensena Figure 3 Connection relation between server blade and connection blade 10 User Manual BX900 BX400 InfiniBand Modules Introduction Description of InfiniBand Connection Blade 1 8 Description of InfiniBand Connection Blade The PRIMERGY 900 InfiniBand connection blade module features 18 internal and 18 external 4x14Gbps FDR ports The below figure shows the front view with the 18 external ports whereas the internal ones are connected to the server blades over Midplane Figure 4 InfiniBand connection blade 1 8 4 Cables For best performance use the InfiniBand connection blade with QSFP Quad Small Form factor Pluggable connectors Copper and optical variants can be used Figure 5 QSFP connectors BX900 BX400 InfiniBand Modules User Manual 11 Description of InfiniBand Connection Blade Introduction 1 8 2 LEDs Connection Blade LEDs To indicate overall status of the connection blade
42. eb User Interface 4 6 4 SETUP MENU The Setup menu makes available the following sub menus listed in order of appearance from Setup top to bottom Interfaces is used to obtain the status of configure or disable interfaces to the InfiniBand fabric Thus you HA can set or clear the IP address and netmask of an interface enable DHCP to dynamically assign the IP ti address and netmask and set interface attributes ES such as MTU speed duplex etc Hostname EE HA is used to create join or modify an IB Subnet osts ARP Routing is used to set remove or display the default iub gateway and the static and dynamic routes of the fabric DNS is used to set remove modify or display Virtual Switch Mgmt static and dynamic name servers 1 Hostname is used to set or modify the hostname SNM Hosts is used to set or delete static hosts Email Alerts XML gateway ARP is used to add static and dynamic ARP entries Logs and to clear the dynamic ARP cache Configurations Virtual Switch Mgmt is used to set the system profile Date and Time Web is used to configure Web user interface and NTP proxy settings Licensing SNMP is used to configure SNMP attributes SNMP admin user and trap sinks Email Alerts is used to define the destination of email alerts and the recipients to be notified XML gateway provides an XML request response protocol to get and set
43. es displays general infor Memory mation about the switch system capabilities Network such as the enabled profiles e g IB ETH and Logs their corresponding values Maintenance Temperature provides a graphical display of Alerts the switch module sensors temperature levels over time 1 hour It is possible to display either the temperature level of one module s sensor or the temperature levels of all the module sensors together Power Supplies provides a graphical display of one of the switch s power supplies voltage level over time 1 hour This function is not supported Fans provides a graphical display of fan speeds over time 1 hour The display is per fan unit within a fan module This function is not supported CPU Load provides a graphical display of the management CPU load over time 1 hour Memory provides a graphical display of memory utilization over time 1 day BX900 BX400 InfiniBand Modules User Manual 69 Starting MLNX OS Web User Interface IB Connection Blade Management Network provides a graphical display of network usage transmitted and received packets over time 1 day It also provides per interface statistics Logs displays the system log messages It is possible to display either the currently saved system log or a continuous system log Maintenance is used to perform specific maintenance operations automatically on a predefined schedule Aler
44. figure shows an example Note that the default username is admin Mellanox MLNX OS Management Console MIN EQHNOLOGIE Host BXSUUS2 Ce2 User not ed Lagin Please enter your username and password then click Login Account admin Password Mellanox MLNX OS Switch Management Gest viewed using Firefox Chrome 1E 7 or higher at 1024x760 resolution or higher ip 2009 2011 Mellanox Technologies Inc Figure 27 WebUI login window BX900 BX400 InfiniBand Modules User Manual 53 Configuration IB Connection Blade Management 4 2 Configuration In order to configure the function of IB connection blade using CLI Command Line Interface the user needs to either connect the management console to Management Blade and run a terminal software or connect to Management Blade via ssh After finishing configuring the network configuration e g IP address it is possible to login via 557 4 2 1 Configuration of LAN interface 1 Change to Config mode Enter BX900S2 CB3 standalone master gt enable BX900S2 CB3 standalone master configure terminal 2 Disable setting IP addresses using the DHCP using the following command BX900S2 CB3 standalone master config no interface mgmtO dhcp 3 Enter interface command to configure the IP address and netmask BX900S2 CB3 standalone master config interface mgmtO ip address 192 168 1 10 255 255 255
45. hardware management infor mation Logs is used to set up system log files remote log sinks and log formats Configurations is used to manage activate save and import MLNX OS SwitchX configuration files and to execute CLI commands Date and Time is used to set the date time and time zone of the switch system BX900 BX400 InfiniBand Modules User Manual 65 Starting MLNX OS Web User Interface IB Connection Blade Management NTP is used to set NTP Network Time Protocol and NTP servers Licensing is used to manage MLNX OS SwitchX licenses 66 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Starting MLNX OS Web User Interface 4 6 5 SYSTEM MENU The System menu makes available the following sub menus listed in order of appearance from top to bottom Modules Inventory Power Management MLNX OS Upgrade Login Messages Reboot Modules displays a photo graphic illustration of the system modules front and rear views By moving the mouse over the ports in the front view a pop up caption is displayed to indicate the status of the port The port state active down is differentiated by a color scheme green for active gray black for down By moving the mouse over the rear view a pop up caption is displayed to indicate the leaf part infor mation Inventory displays a table with the following infor mation about the s
46. hest Example for RHEL5 etc init d opensmd start p 15 BX900 BX400 InfiniBand Modules User Manual 27 Subnet Manager Linux Installation 2 5 2 Confirmation of OpenSM running Enter the following command to confirm that OpenSM is running etc init d opensmd status OpenSM is running properly if the result of this command displays running enter the following command to run OpenSM if the result displays stopped etc init d opensmd start 2 5 3 Confirmation of the master SM Enter the ibstat command on the system server then Base LID and SM LID are displayed The IB Mezzanine card or IB connection blade on which these two LIDs are the same is the master SM 28 User Manual BX900 BX400 InfiniBand Modules Linux Installation Network Configuration Information 2 6 Network Configuration Information The following information 1 6 is needed to maintain and troubleshooting an InfiniBand network The information should be saved when configuring the InfiniBand network 1 o 0 Network configuration diagram Configuration sheet The server on which the MASTER is found The server on which the STANDBY is found The result of the ibnetdiscover command GUID list and arrangement drawing of device location plan 2 6 1 Network configuration diagram Example 900 2 900 2 IB cable 5 Figure 8 Network configuration d
47. iagram BX900 BX400 InfiniBand Modules User Manual 29 Network Configuration Information Linux Installation 2 6 2 Configuration sheet Server blade mezz 1 GUID mezz 2 GUID oS host name Static DHCP iRMC eth1 eth2 eth3 ib3 Slot Slo Slo Slo Slo Slo Slo 2 3 4 Slot5 6 7 8 9 Slo Slot10 Slot11 Slot12 510113 510114 510115 510116 510117 510118 CB3 GUID host name Static DHCP ethO ibO snmp 5 MMB1 Agent MMB1 Service Table 3 Configuration sheet IP Address XX Root password How to fill in 30 User Manual BX900 BX400 InfiniBand Modules Linux Installation Network Configuration Information Please fill in the GUID of the Mezzanine Card installed in Mezzanine slot in the column of mezz 1 GUID Similarly please fill in the GUID of the Mezzanine Card installed in Mezzanine II slot in the column of mezz 2 GUID Please fill in the host name in the column of hostname when you assign the host name to each server blade Please fill in the IP addresses XX in the column of ethO eth1 eth2 eth3 ibO ib1 162 and ib3 when you use the static IP IP Address 192 168 11 XX Server Blade 510
48. iguration upload command BX900S2 CB3 standalone master config configuration upload myconf scp user password 192 168 1 10 tmp confl How to restore the configuration By default or after a system reset the system loads the default initial config uration file To load a different configuration file and make it the active configu ration change to Config mode and enter 1 2 Copy configuration file myconf to server by configuration fetch command It cannot download when a configuration file already exists in an IB connection blade BX900S2 CB3 standalone master gt enable BX900S2 CB3 standalone master configure terminal BX900S2 CB3 standalone master config configuration fetch scp user password 192 168 1 10 tmp conf1 myconf To display the available configuration files and the active file enter BX900S2 CB3 standalone master config show configuration files initial active initial bak yyconf Execute configuration switch to myconf command from IB connection blade BX900S2 CB3 standalone master config configuration switch to myconf BX900S2 CB3 standalone master config show configuration files initial initial bak yconf active BX900 BX400 InfiniBand Modules User Manual 57 Basic Operation IB Connection Blade Management 4 3 Basic Operation This chapter describes the basic operation by the commands and how to get the information for the maintenance
49. imits conf soft memlock unlimited hard memlock unlimited root localhost 1 Please re install from step3 when the 2 messages above were not displayed 2 3 3 2 Updating of on RHEL6 1 1 Mount iso file and move to mnt In the following example the OFED is MLNX OFED LINUX 1 5 3 1 0 0 rhel6 1 x86_64 iso Croot localhost J mount o ro loop MLNX OFED LINUX 1 5 3 1 0 0 rhe16 1 x86 64 iso mnt root localhost 1 cd mnt root localhost 1 Install glibc devel 2 12 1 25 el6 i686 package by using RHEL6 1 installation media Croot localhost mntl cd mnt RHEL 6 14 x86 644 Disc 1 Packages root localhost Packages rpm ivh glibc devel 2 12 1 25 16 1686 warning glibc devel 2 12 1 25 e16 1686 rpm Header V3 RSA SHA256 Signature key ID fd431d51 NOKEY Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE 210021 1 glibc devel JHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE 100 root localhost Packages 1 BX900 BX400 InfiniBand Modules User Manual 23 Installation Linux Installation 3 Move to root and force the following packages to be uninstalled libibverbs libibcm librdmacm infinipath psm fcoe utils root localhost Packages cd root localhost rpm e nodeps libibverbs libibcm librdmacm infinipath psm fcoe utils root localhost 1 If these packages are not installed the following error message is returned Croot localh
50. ional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x000C Error QUERY_ADAPTER command failed with error 2 N The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x000D Error Too few QPs were requested requested 2 reserved for FW 3 n The adapter card is non functional n Please increase the Registry LogNumQp parameter under HKLM System CurrentCon trolSet Services m x4_bus Parameters Ox001 1 Error Failed to move location string 262 status 3 0x0012 Error WdfDeviceAllocAndQue ryProperty failed status 962 The IB Mezzanine card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 9 source mlx4 bus 74 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card Event ID Classifi To do Error Level cation 0x0003 The The IB Mezzanine card must be replaced Error message Please consult changes technical support ts fujitsu com each event If you purchased the products from Fujitsu Japan please consult the system engineer Table 10 source ibbus Work after IB Mezzanine card has been exchanged Windows Server 2008 R2 SP1 1 Turn on the server 2 Make
51. is box to configure your system for maximum 10GigE performance Recommended and then click Next gt ji MLNX_VPI InstallShield Wizard Maximum Performance Configure your system for maximum 10GigE performance Check this box to configure your system for maximum 10GigE performance Recommended Note This step will require rebooting the machine at the end of installation InstallShield lt Back Next gt Cancel Figure 16 Maximum performance 40 User Manual BX900 BX400 InfiniBand Modules Windows Installation Installation of WinOF 8 Click Install after the following window appears i MLNX_VPI InstallShield Wizard Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings dick Back Click Cancel to exit the wizard InstallShield Figure 17 Installation BX900 BX400 InfiniBand Modules User Manual 41 Installation of WinOF Windows Installation 9 The following window appears after the installation process is finished Click Finish without checking any boxes iz MLNX_ PI InstallShield Wizard InstallShield Wizard Completed The InstallShield Wizard has successfully installed VPI Click Finish to exit the wizard You chose not to run performance tuning Please note that it is highly recommended to run this tu
52. keyboard If capital letters are to be entered explicitly then the Shift key is shown e g SHIFT for A If two keys need to be pressed at the same time then this is indicated by placing a hyphen between the two key symbols BX900 BX400 InfiniBand Modules User Manual 5 Target Group Introduction Fujitsu Support Please contact your Fujitsu Technical Support if you require assistance http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult your system engineer 1 2 Target Group This manual is intended for users developers and system administrators responsible for setting up and maintaining switch system platforms using InfiniBand fabrics and networks The switch system platforms must have a management server or client in order for the package to work The manual assumes familiarity with the InfiniBand Architecture Specification as well as Ethernet Architecture Specification 1 3 Documentation Overview PRIMERGY manuals are available in PDF format on the ServerView Suite DVD 2 The ServerView Suite DVD 2 is part of the ServerView Suite supplied with every server If you no longer have the ServerView Suite DVDs you can obtain the relevant current version using the order number U15000 C289 the order number for Fujitsu Japan please refer to the configurator of the server http jp fujitsu com platform server primergy system
53. lade If you have changed IB Mezzanine card and you still see the same messages you need to change the server blade that installed the IB Mezzanine card BX900 BX400 InfiniBand Modules User Manual 61 Check the Log File of OpenSM Linux only Troubleshooting 3 IB connection blade If you have changed the above server blade and still get the same messages you need to change IB connection blade For details of how to specify the exact IB connection blade please refer to section The result of the command ibnetdiscover on page 35 Removed port with GUID GUID number PRIx64 LID range LID number LID number of node IB Mezzanine card IB connection blade or IB cable was disconnected How to solve the problem The following changes are required Please contact the repair center 1 IB Mezzanine card You need to change the relevant IB Mezzanine card You obtain the relevant IB Mezzanine card from GUID or LID information in the error messages and the output of ibnetdiscover For example Jul 30 23 34 18 600287 4CDOD940 0x02 gt osm drop mgr remove port Removed port with GUID 0x0002c90300041306 PRIx64 LID range 2 2 of node localhost HCA 2 In this case Error Card has the GUID 0x0002c90300041306 and the is 2 of Mezzanine IB connection blade You need to change the relevant IB connection blade You will get the relevant IB connection blade from GUID or LID information of the error me
54. n command Caution Look out for the following during the operation of save and restore NOT turn off the power Check that there is no traffic between the nodes Operate with NOT doing any setting from a management terminal IP address 192 168 1 10 24 Save the configuration by scp sftp command 900 52 chassis back face Restore the configuration by sep or sftp command SSH server 192 168 1 100 24 example BX 900 52 chassis Figure 28 Save configuration How to save the configuration To save the current configuration to the active configuration file you can either use the configuration write command requires running in Config mode 1 Execute configuration write to myconf no switch command from IB connection blade The following command saves the configuration to a user specified file without making the new file the active configuration file BX900S2 CB3 standalone master gt enable BX900S2 CB3 standalone master configure terminal BX900S2 CB3 standalone master config configuration write to myconf switch 56 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Configuration To display the available configuration files and the active file enter BX900S2 CB3 standalone master config show configuration files initial active initial yyconf Copy configuration file myconf to server by conf
55. n the admin This mode has a full unrestricted set of commands to view anything take any action or change any configuration Its commands are a superset of those in Enable mode To return to Enable mode enter exit or no configure Note that moving directly from to Standard mode to from Config mode is not possible 4 6 Starting MLNX OS Web User Interface This chapter describes how to log into the Web User Interface WebUI of Mellanox MLNX OSSwitchX based managed switches 4 6 1 Connecting to the Switch Platform Please perform the following steps to start a WebUI connection to the switch platform 60 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Starting MLNX OS Web User Interface Step1 Set up an Ethernet connection between the switch and a local network machine Step2 Open a Web browse connection between the switch and a local network mac i Make sure the screen resolution is set to 1024 768 or higher Step3 Type in the IP address of the switch or its DNS name in the following format http switch IP address Step4 Enter your username and password to log in Step5 Click Login Mellanox MLNX OS FDR IB Switch FUJITSU Host FOR CB User not legged in Login Please enter your username and password then click Login Account imm Password Figure 29 Login Mellanox MLNX OS There are two user roles or account types admin and monitor As admin the user is p
56. ning For optimum performance You can run it later by selecting the Performance Tuning tab in the Mellanox 10GBit Adapter properti r 5 IB release notes Enable NetworkDirect interface ND Show ETH release notes E Gancel Figure 18 Installation completed 42 User Manual BX900 BX400 InfiniBand Modules Windows Installation Installation of WinOF 10 Make sure that InfiniBand devices are recognized properly in Device Manager evice Manager File Action View gil Network adapters x Intel R I350 Gigabit Network Connection Intel R 1350 Gigabit Network Connection 2 Mellanox IPoIB Adapter Mellanox IPoIB Adapter 2 5 Other devices 737 Ports COM amp LPT a Processors lt gt Storage controllers Wa System devices Wi ACPI Fixed Feature Button ACPI Module Device ACPI Module Device 288 Composite Bus Enumerator 288 Direct memory access controller i Generic Bus li High precision event timer Intel R 82801 PCI Bridge 244E LSI RAID Virtual Device Mellanox Connectx Virtual Channel Adapter Mellanox 27500 4099 Connectx 3 VPI Network Adapter Microsoft ACPI Compliant Power Meter Device Microsoft ACPI Compliant System Microsoft Generic IPMI Compliant Device a Microsoft System Management BIOS Driver Microsoft Virtual Drive Enumerator Driver Numeric data processor PCI bus
57. on Timeout 900 Enter selection ar type 10 to quit 1 Figure 24 Console redirection 4 Enter the slot number on which IB connection blade is installed and press the key Example If IB connection blade is installed on slot3 4 enter 3 and press the key BX900 BX400 InfiniBand Modules User Manual 47 Connection IB Connection Blade Management 13 DBI PY Eth Switch IBP 1Gb 36 12 Switch Stackinz N A 3 DB8 PY IB Switch 56Gb 18 18 Stackinz 5 DB5 PY CB IB Switch 56Gb 18 18 Stackinz Enter selection ar type 0 to quit 3 Figure 25 Console redirect connection blade 4 1 1 Configuring the Switch for the First Time 1 Login from MMB page 3 1 as admin and use admin as password This starts the configuration wizard However the wizard may not start depending on a platform 2 Go through the configuration wizard Table 4 shows an example of a wizard session Wizard Session Display Example Comments Mellanox configuration wizard You must perform this configuration the first time you operate the switch or after resetting the switch to the factory defaults Type y and then press yes Enter Do you want to use the wizard for initial configuration Step1 Hostname switch 1 If you wish to accept the default hostname then press Enter Otherwise type a different hostname and press Enter Table 4 Configuration Wizard Session IP C
58. onfiguration by DHCP 48 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Connection Wizard Session Display Example Comments Step 2 Use DHCP mgmtO interface yes Perform this step to obtain an IP address for the switch is the management port of the switch If you wish the DHCP server to assign the IP address type yes and press Enter If you type no DHCP then you will be asked whether you wish to use the zeroconf configuration or not If you enter no no Zeroconf then you need to enter a static IP Step 3 Enable IPv6 yes Perform this step to enable IPv6 on management ports If you wish to enable IPv6 type yes and press Enter If you enter no no IPv6 then you will automatically be refered to Step 5 Step 4 Enable IPv6 autoconfig SLAAC on mgmt0 interface Perform this step to enable StateL ess address autoconfig on external management port If you wish to enable it type yes and press Enter If you wish to disable it enter no Step 5 Admin password Press Enter to leave unchanged new password Step 6 Confirm admin password new password To avoid illegal access to the machine please type a password and then press Enter Then confirm the password by re entering it Table 4 Configuration Wizard Session IP Configuration by DHCP BX900 BX40
59. ost mnt rpm e nodeps allmatches libibverbs devel error package libibverbs devel is not installed rootalocalhost mnt rpm e nodeps allmatches libibverbs error package libibverbs is not installed root localhost 1 Move to mnt and execute Installer and enter y In the following example the Installer is minxofedinstall root localhost 1 cd mnt Croot localhost mnt mlnxofedinstall without 32bit This program will install the MLNX OFED LINUX package on your machine Note that 11 other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue Ly N1 y Uninstalling the previous version of OFED Starting MLNX OFED LINUX 1 5 3 1 0 0 installation Installing kernel ib RPM 24 User Manual BX900 BX400 InfiniBand Modules Linux Installation OFED Installation Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE kernel ib THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installing kernel ib devel RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE kernel ib devel fHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installation finished successfully root localhost 1 5 Unmount mnt directory rootalocalhost 1 cd root localhost 1 umount mnt root localhost 1 6 Restart the OS root localhost 1 reboot 7 Check the modification of limits conf file root localhost tail 2 etc security limits conf soft memlock unlimite
60. ot localhost 34 BX900 BX400 InfiniBand Modules Linux Installation Network Configuration Information 2 6 4 The result of the command ibnetdiscover GUID list It is useful to provide the text file txt which associates a GUID with the name of the device so that the ibnetdiscover command displays the result more meaningfully format of the text GUID hex number Hostname and Mezzanine slot Regarding the Mezzanine slot please refer to Identifying the GUID on page 31 For example mmap txt bx900 02 IB Mezzanine 0 0002 903000413 0 bx900 02 01 Mezz 1 0x0002c90300041514 bx900 02 01 Mezz 2 0x0002c9030003bf7a bx900 02 02 Mezz 1 0x0002c9030003bf6a bx900 02 02 Mezz 2 0x0002c903000413b8 bx900 02 03 Mezz 1 0x0002c90300041488 bx900 02 03 Mezz 2 0x0002c90300041408 bx900 02 04 Mezz 1 0x0002c903000412f0 bx900 02 04 Mezz 2 0x0002c90300041414 bx900 02 05 Mezz 1 0x0002c90300041348 bx900 02 05 Mezz 2 BX900 02 IB Connection Blade 0x0002c9020040c7a8 bx900 02 IB SW CB3 BX900 04 IB Connection Blade 0x0002c9020040c790 bx900 04 IB SW CB3 bx900 04 Mezzanine 0x0002c903000413ec bx900 04 01 HCA 1 0x0002c903000447b0 bx900 04 02 HCA 1 BX900 BX400 InfiniBand Modules User Manual 35 Network Configuration Information Linux Installation ibnetdiscover command execution procedure 1 4 Connect to the CLI of IB connection blade via MMB MM
61. rivileged to execute all the available operations that are permitted by the installed license As monitor the user can execute operations that display system configuration and status or set terminal settings User Role Default Password admin admin monitor monitor BX900 BX400 InfiniBand Modules User Manual 61 Starting MLNX OS Web User Interface IB Connection Blade Management 4 6 2 After Log in Display After you log in to MLNX OS a default status summary window will be displayed containing the following information FUJITSU Mellanox MLNX OS FDR IB Switch Host Me User admin Logout Standalone Virtual IP Active node ES Subnet Manager is not running summary Profile and Capabilities Temperature Date and Time 2012 10 18 20 33 12 Power Supplies Hostname FDR CB Fans Uptime 4h 12m 2s CPU Load Version SX PPC M460EX SX 3 2 0328 2012 07 01 20 26 02 ppc Memory Model ppc WE Host ID 0002C9621D98 System memory 101 MB used 1926 MB free 2027 MB total Logs CPU load averages 0 56 0 12 0 04 Maintenance Alerts Active alerts Figure 30 MLNX OS status summary Figure 31 Active alerts 62 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Starting MLNX OS Web User Interface Icon Description Mellanox MLNX OS FDR IB Switch System Information Banner Contains information on the switch system type Standalone IP Acti
62. s monitor the user can run commands that show system configuration and status or set terminal settings No User name Password Authority Description level 1 admin admin Adminis admin is privileged to run all trator the available commands 2 monitor monitor user monitor can run commands that show system configu ration and status or set terminal settings Table 5 Default user BX900 BX400 InfiniBand Modules User Manual 59 CLI modes IB Connection Blade Management 4 5 CLI modes The CLI can be in one of three modes and each mode makes available a certain group or level of commands for execution The three command modes are 1 Standard mode When the CLI is launched it begins in Standard mode This is the most restrictive mode and only has commands to query a restricted set of state information Users cannot take any actions that directly affect the system nor can they change any configuration 2 Enable mode The enable command moves the user to Enable mode This mode offers commands to view all state information and take actions like rebooting the system but it does not allow any configuration to be changed Its commands are a superset of those in Standard mode To return to Standard mode enter disable 3 Config mode admin only The configure terminal command moves the user from Enable mode to Config mode Config mode is allowed only for user accounts i
63. ssages and output of ibnetdiscover For example Jul 30 23 34 18 600424 4CDOD940 0x02 gt osm drop mgr remove port Removed port with GUID 0x0002c9020040c790 PRIx64 LID range 21 21 of node none SW 1 In this case Error IB connection blade has the GUID 0x0002c9020040c790 and the LID is 21 For details of how to specify the relevant IB connection blade refer to section The result of the command ibnetdiscover on page 35 62 User Manual BX900 BX400 InfiniBand Modules Troubleshooting Check the Log File of OpenSM Linux only 3 IB cable If you have changed the above IB connection blade and you still receive the messages all IB cables connected to the IB connection blade need to be changed Reporting Generic Notice type 1 num 128 Link state change from LID LID number GUID GUID number the port of IB connection blade was disconnected in the subnet How to solve the problem The following changes are required Please contact the repair center 1 IB cable You need to change the IB cable You specify the IB cables by the following procedure You specify the relevant IB connection blade from the LID infor mation in the error messages For example Jul 31 22 56 46 264171 42E93940 0x02 gt osm report notice Reporting Generic Notice type 1 num 128 Link state change from LID 32 GUID fe80 2 c902 40 c790 In this case LID 32 s IB connection blade is corresponding For details of how
64. su Technology Solutions Web site http support ts fujitsu com com support downloads html gt Driver amp Downloads gt f you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads Warning This software is the driver package of ConnectX 3 IB Mezzanine card 56Gbps This package driver does not supported ConnectX 2 IB Mezzanine card 40Gbps 231 OFED installation on RHEL5 8 1 Mount iso file and move to mnt In the following example the OFED is MLNX OFED LINUX 1 5 3 3 3 0 0 rhel5 8 x86_64 iso Croot localhost J mount ro loop MLNX OFED LINUX 1 5 3 3 0 0 rhe15 8 x86 64 iso mnt root localhost 1 cd mnt root localhost 1 2 Execute Installer and enter y Croot localhost 14 mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y 3 The following messages are displayed Add an option nodeps and uninstall the package Please remove OFED RPMs coming from the Distribution 16 User Manual BX900 BX400 InfiniBand Modules Linux Installation OFED Installation Run rpm e allmatches openmpi libcxgb4 ofed docs 1 4 1 2 e15 openib 1 4 1 6 e15 libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibuma
65. tches libibverbs root localhost mnt If these packages are not installed the following error message is returned Croot localhost mnt rpm e nodeps allmatches libibverbs devel error package libibverbs devel is not installed Croot localhost 1 rpm e nodeps allmatches libibverbs error package libibverbs is not installed root localhost 1 Execute Installer and enter y In the following example the Installer is minxofedinstall Croot localhost 1 mlnxofedinstall without 32bit This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue Ly N1 y Uninstalling the previous version of OFED Starting MLNX OFED LINUX 1 5 3 1 0 0 2 installation Installing kernel ib RPM Preparing fHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE kernel ib THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installing kernel ib devel RPM Preparing fHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installation finished successfully root localhost mnt 22 User Manual BX900 BX400 InfiniBand Modules Linux Installation OFED Installation Unmount mnt directory Croot localhost mntl cd root localhost 1 umount mnt root localhost 1 Restart the OS localhost reboot Check the modification of limits conf file root localhost tail 2 etc security l
66. te Error Protection Domain 0x0043 xxx Failed to create Error receive Completion Queue 0x0044 xxx Failed to create send Error Completion Queue 0x0045 xxx Failed to create Error Queue Pair E e ezzanine card must be 0x0046 xxx Failed to get Queue replaced Error Pair number 0x0047 xxx Failed to create DMA a im uno Error Memory Region technica Suppor s fuji SU COM 0x0048 xxx Failed to create If you purchased the products from Error receive descriptor pool Fujitsu Japan please consult the 0x0049 xxx Failed to create system engineer Error NDIS_PACKET pool to receive indications 0 004 Failed to create Error NDIS_BUFFER pool to receive indications 0x004B xxx Failed to create Error NDIS_PACKET pool to send processing 0x004C xxx Failed to create Error NDIS_BUFFER pool to send processing Table 8 source ipoib 70 User Manual BX900 BX400 InfiniBand Modules Troubleshooting IB Mezzanine Card the required IPoIB parameter 3 overriding it by default value 4 Event ID Classification To do Error Level 0x004D xxx Failed to allocate Error receive indication array 0 004 xxx Subnet Administrator Error query for port information timed out Make sure the SA is functioning properly Increasing the number of retries and retry timeout adapter parameters may solve the problem 0 004 xxx Subnet Administrator Error failed the quer
67. terms in the license agreement and then click Next gt i MLNX_YPI InstallShield Wizard License Agreement Please read the following license agreement carefully Copyright 2005 2011 rights reserved Mellanox Technologies All Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer in h mirr Farm minnt tha cloro Print C 1 not accept the terms in the license agreement Installshield Back cael Figure 14 License Agreement 38 User Manual BX900 BX400 InfiniBand Modules Windows Installation Installation of WinOF 6 Click Next gt after the following window appears if you want to change the installation folder specify the folder you want to install on by clicking Change ji MLNX_VPI InstallShield Wizard Destination Folder Click Next to install to this folder or dick Change to install to a di din Install MLNX VPI to C Program Files Mellanox MLNX_VPI Change InstallShield lt Back Cael Figure 15 Destination Folder BX900 BX400 InfiniBand Modules User Manual 39 Installation of WinOF Windows Installation 7 After the following window appears untick Check th
68. ts is used to display a list of the recent health alerts and enables the user to configure health daemon settings 4 6 9 Installing the License This function is not supported 70 User Manual BX900 BX400 InfiniBand Modules 5 Troubleshooting This chapter describes what to do if IB Mezzanine card and IB connection blade is not working properly If you cannot solve the problem please contact the repair center http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult your system engineer 5 1 Check the Log File of OpenSM Linux only Check the log file in the server var log opensm log that started up the Subnet Manager OpenSM as Master gt Check list Are any of the following logs contained in the var log opensm log SM port is down Removed port with GUID GUID number PRIx64 LID range LID number LID number of node Reporting Generic Notice type 1 num 128 Link state change from LID LID number GUID GUID number SM port is down SubnetManager s master port was disconnected How to solve the problem The following changes are required Please contact the repair center 1 IB Mezzanine card You need to change the IB Mezzanine card that ran as the Master of the SubnetManager For the method of how to specify the exact IB Mezzanine card please refer to section The result of the command ibnetdis cover on page 35 2 ServerB
69. upply source of BX900 or BX400 system does not have abnormality Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer lowPower 1 3 6 1 4 1 33049 2 1 2 8 Content Recover The MLNX OS management appliance has been restored to its normal power Action lowPowerRecover indicates a recovery from low power situation Table 12 snmp trap Work after the exchange 1 IB connection blade is mounted on the main body of BX900 BX400 and the power supply is turned on Confirm that the Status LED of the IB connection blade lights to green It takes about 90 seconds till the CPU of the IB connection blade starts Connect to the console of the IB connection blade via ssh and wait for the prompt of the IB connection blade to be displayed When no prompt is displayed push Enter and confirm that a prompt is displayed Change to config mode Example BX900S2 CB3 standalone master gt enable BX900S2 CB3 standalone master configure terminal Set ip address and netmask which there was it before exchange Example BX900S2 CB3 L standalone master config interface mgmtO ip address 192 168 1 10 255 255 255 0 BX900 BX400 InfiniBand Modules User Manual 77 IB Connection Blade Troubleshooting 6 Restore composition definition information Please refer to section Save and restore the configuration on page 5
70. utomatic Local System The PnP X Disabled Local System Enforces 0 Manual Local System Manages Started Automatic Local System Loads files Started Automatic Local System This servic Manual Local System Protected Storage Provides pr Manual Local System XQ Remote Access Aut Creates a Manual Local System iGkRemote Access Co Manages di Started Manual Local System 1C Remote Access Qu Removes v Manual Local Service Desktop Remote De Manual Local System X Extended A Standard Figure 20 Configuration of OpenSM 12 Change Startup type from Manual to Automatic Delay Start 13 Click the Start button to start the OpenSM service 44 User Manual BX900 BX400 InfiniBand Modules Windows Installation Installation of WinOF 14 Open the Network Connections Window to check if IPoIB networks are linked up Disable the unused IPoIB adapters I Network Connections E JC E Control Panel Network and Internet Network Connections K search Network Cor Organize v Private Enterprise Local Area Connection 4 A Unidentified network A Network Ks 7 Intel R 82575 Gigabit Backplane 7 Intel R 82575EB Gigabit Backplane 3 7 Mellanox IPoIB Adapter Application gt Identifying 7 Mellanox IPoIB Adapter 2 Figure 21 Network status 15 Since no IP address is assigned to this new IPoIB adapter assign an I
71. ve node E3 Subnet Manager is not running information on the System Information Banner Contains OpenSM state Chassis and SM HA role Host FDR CB User admin Logout User credentials and Host name Srn Ses Pare Je 085 Action Pane A display of the MLNX OS menu icons Temperature Power Supplies Fans CPU Load Memory 1 Network Logs Maintenance Alerts Left Side Pane Displays a list of the MLNX OS submenus Dote end Time 2900 04 01 14 17 22 Hostname switch 113f2c Uptime 54 24m 65 Version EFM PPC EFM 1 1 1707 DEMO 2010 11 09 13 31 47 poc Model Most D 95503249551 System memory 77 MB used 1950 MB free 2027 M8 total CPU load averages 0 00 0 02 0 00 Information Summary provides detailed information on the chosen menu A Save button used to save system s changes A grayed out Save button suggests no system changes were performed hence the button is disabled and cannot be clicked BX900 BX400 InfiniBand Modules User Manual 63 Starting MLNX OS Web User Interface IB Connection Blade Management 4 6 3 MLNX OS WebUI The WebUI of MLNX OS makes available the following menus listed in order of appearance from left to right SETUP SYSTEM SECURITY PORTS STATUS 64 User Manual BX900 BX400 InfiniBand Modules IB Connection Blade Management Starting MLNX OS W
72. y for port information Make sure the The IB Mezzanine card must be SA is functioning properly replaced and compatible PI It 0x0050 xxx Subnet Administrator ET Error query for port information failed If you purchased the products from 0x0055 xxx Subnet Administrator Fujitsu Japan please consult the Error failed query for broadcast system engineer group information 0x0056 xxx Subnet Administrator Error failed request to joining broadcast group 0x0057 xxx The local port rate is Error too slow for the existing broadcast MC group 0x0058 xxx Incorrect value or Error non existing registry for Table 8 source ipoib BX900 BX400 InfiniBand Modules User Manual 71 IB Mezzanine Troubleshooting Event ID Classification To do Error Level 0x005B xxx Pkey index not found Error for partition change switch pkey configuration 0x005C xxx Connected Mode Error failed to initialize disabled Interface will use default The IB Mezzanine card must be UD QP transport replaced 5 SetDeviceRegistra Please consult Error tionAttributes failed technical support ts fujitsu com 0 005 SetAdapterRegistra Error tionAttributes failed If you purchased the products from Fujitsu Japan please consult the 0x0060 XXX SetOffloadAttributes system engineer Error failed 0x0061 ipoib_create_adapter Error failed 0x0062
73. ystem modules module name type serial number ordering part number and Asic firmware version Power Management displays a table with the following information about the system power supplies power supply name power voltage level current consumption and status OK or NOT PRESENT A total power summary table is also displayed providing the power W used the power capacity and the power available MLNX OS Upgrade is used to view the installed MLNX OS images and the active partition to upload a new image and to install a new image Login Messages is used to edit the login messages Message of the Day MOTD Remote Login message and Local Login message Reboot is used to reboot the system Make sure that you save your configuration prior to clicking reboot Rebooting the system may take several minutes BX900 BX400 InfiniBand Modules User Manual 67 Starting MLNX OS Web User Interface IB Connection Blade Management 46 6 SECURITY MENU bo The Security menu makes available the following sub menus listed in order of appearance from top to bottom Users Admin Password SSH RADIUS TACACS LDAP Users is used to manage setting up removing modifying user accounts Admin Password is used to modify the system administrator password SSH is used to display and generate host keys AAA is used to configure AAA Authentication Authorization

Download Pdf Manuals

Related Search

Related Contents

Instrucciones  Sapphire 2000 operators manual  777 Music Emotion review  Synology RackStation RS815+  Little LoungerTM  Samsung MS32F303TAW Manual de Usuario  

Copyright © All rights reserved. Failed to retrieve file