Mellanox OFED Linux User`s Manual
Contents
1. all Install all kernel modules libibverbs libibumad librd macm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector perftest sdpnetstat and libsdp srptools rds tools static and dynamic libraries hpc Install all kernel modules libibverbs libibumad librd macm mft mstflint diagnostic tools OpenSM ib bonding Mellanox Technologies 27 Mellanox Technologies Confidential Rev 1 5 2 Installation MVAPICH Open MPI MPI tests MPI selector dynamic librar ies basic Install all kernel modules libibverbs libibumad mft mstflint dynamic libraries msm Install all kernel modules libibverbs libibumad mft mstflint diagnostic tools OpenSM ib bonding dynamic libraries NOTE With msm flag the OpenSM daemon is configured to run upon boot v vv vvv Set verbosity level pfc lt 0 bitmask gt Priority based Flow Control policy on TX and RX 7 0 q Set quiet no messages will be printed 2 3 2 1 mlnxofedinstall Return Codes Table 5 lists the m1nxofedinstall script return codes and their meanings Table 5 minxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 3 Failed to start the mst driver 28 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 2 32. Software The OFED and MFT packages are installed under the usr directory The kernel modules are installed under InfiniBand subsystem Mellanox Technologies 33 J Mellanox Technologies Confidential Rev 1 5 2 Installation lib modules uname r updates kernel drivers infiniband e mlx4 driver Under lib modules uname r updates kernel drivers net mlx4 you willfind mlx4 core ko mlx4 en ko mlx4 ib ko mlx4 vnic ko and mlx4 fc ko IPoIB lib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko SDP lib modules uname r updates kernel drivers infiniband ulp sdp ib sdp ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko lib modules uname r updates kernel drivers infiniband ulp srpt ib srpt ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko lib modules uname r updates kernel net rds rds tcp ko The package kernel ib devel include files are placed under usr src ofa_kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel The raw package un backported source files are placed under usr src ofa kernel ver The script openibd is installed under etc init d This script can be used to load and unload the software
3. Mellanox OFED for Linux User s Manual Rev 1 5 2 2 Installation This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox InfiniBand and or Ethernet adapter hardware installed The chapter includes the following sections Section 2 1 Hardware and Software Requirements on page 25 Section 2 2 Downloading Mellanox OFED on page 26 Section 2 3 Installing Mellanox OFED on page 26 Section 2 5 Uninstalling Mellanox OFED on page 37 2 4 Hardware and Software Requirements 2 1 4 Hardware Requirements Platforms A server platform with an adapter card based on one of the following Mellanox Technologies InfiniBand HCA devices e MT25408 ConnectXe 2 VPI IB EN FCoE firmware fw ConnectX2 e MT25408 ConnectXe VPI IB EN FCoE firmware fw 25408 MT25208 InfiniHoste III Ex firmware fw 25218 for Mem Free cards and fw 25208 for cards with memory MT25204 InfiniHoste III Lx firmware fw 25204 MT23108 InfiniHoste firmware fw 23108 Note For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for Installation 400 MB Device ID Note For the latest list of device IDs please visit Mellanox website For InfiniBand Cards go to www mellanox com gt Products gt InfiniBand Cards gt Overview For Ethernet Cards go to www mellanox com gt Products gt
4. openmpi intel libmverbs opensm libs libmthca libmlx4 libibcm libibmad libsdp dapl dapl ibutils2 libmverbs ofed scripts libibverbs devel libibverbs devel static libibverbs devel static libibverbs utils libmthca libmthca devel static libmthca devel static libmlx4 libmlx4 devel libmlx4 devel libmverbs devel libmverbs devel libmqe libmqe libibcm libibcm devel libibcm devel libibumad devel libibumad static libibumad static libibmad devel libibmad devel libibmad static libibmad static ibsim librdmacm utils librdmacm devel librdmacm devel 30 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 libsdp libsdp devel libsdp devel opensm opensm devel opensm devel opensm static opensm static compat dapl compat dapl compat dapl devel compat dapl devel dapl devel dapl devel dapl devel static dapl devel static dapl utils perftest mstflint mft sdpnetstat srptools rds tools ibutils cc mgr ibdump infiniband diags qperf mlnxofed docs mvapich gcc mvapich pgi openmpi gcc openmpi pgi mpitests mvapich gcc mpitests mvapic
5. Products gt Ethernet Cards gt Overview 2 1 2 Software Requirements Operating System Linux operating system Note For the list of supported operating system distributions and kernels please refer to the Mellanox OFED Release Notes file Mellanox Technologies 25 Mellanox Technologies Confidential Rev 1 5 2 Installation Installer Privileges The installation requires administrator privileges on the target machine 2 2 Downloading Mellanox OFED Step 1 Verify that the system has a Mellanox network adapter HCA NIC installed by ensuring that you can see ConnectX or InfiniHost entries in the display The following example shows a system with an installed Mellanox HCA hostl lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT25418 ConnectX IB DDR PCIe 2 0 2 5GT s rev a0 Step2 Download the ISO image to your host The image s name has the format MLNX OFED LINUX lt ver gt lt OS label gt iso You can download it from http www mellanox com gt Products gt IB SW Drivers Step3 Use the md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result to the value provided on the download page hostl md5sum MLNX OFED LINUX ver OS label gt iso 2 3 Installing Mellanox OFED The installation script n1lnxofedinstall performs the following Discovers the currently installed kernel Uninstalls any
6. software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX OFED LINUX binary RPMs if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automatically upgrades the firmware 2 3 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Note Pre existing configuration files will be saved with the extension conf saverpm f you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel support sh script located under the docs directory Usage 1 The firmware will not be updated if you run the install script with the without fw update option 26 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 mlnx add kernel support sh i iso mlnx iso t tmpdir local work dir v verbose Example The follo
7. stack e The script connectx port config is installed under sbin This script can be used to configure the ports of ConnectX network adapter cards to Ethernet and or InfiniBand For details on this script please see Section 4 1 Port Type Management The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED installation infor mation The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file is used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc f IPoIB configuration files are included 1ifcfg ib n files will be installed under etc sysconfig network scripts ona RedHat machine e etc sysconfig network ona SuSE machine The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man 34 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 Firmware The firmware of existing network adapter devices will be updat
8. 3 Installation Procedure Step1 Login to the installation machine as root Step2 Mount the ISO image on your machine hostl mount o ro loop MLNX OFED LINUX ver OS label gt iso mnt Note After mounting the ISO image mnt will be a Read Only folder Step3 Run the installation script hostl mnt mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of OFED Starting MLNX OFED LINUX 1 5 2 0 0 5 installation Installing kernel ib RPM Preparing kernel ib Installing kernel ib devel RPM Preparing kernel ib devel Installing kernel mft RPM Preparing kernel mft Installing mpi selector RPM Preparing mpi selector Install user level RPMs Preparing libibverbs libibumad libibverbs libibumad libibumad devel librdmacm opensm libs librdmacm libibmad Mellanox Technologies 29 Mellanox Technologies Confidential Rev 1 5 2 Installation mvapich intel libibverbs devel
9. ed if the following two condi tions are fullfilled 1 You run the installation script in default mode that is without the option without fw update 2 The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image Note If an adapter s Flash was originially programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image Incase your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Mellanox Technologies 35 Mellanox Technologies Confidential Rev 1 5 2 Installation Error message I Querying device E Can t auto detect fw configuration file 2 3 5 Post installation Notes 2 4 Most of the Mellanox OFED components can be configured or reconfigured after the installa tion by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file Updating Firmware After Installation In case you ran the mn1nxofedinstall script with the without fw update option and now you wish to manually update firmware on you adapter card s you need to perform the fol lowing steps Note If you need to b
10. elf test ofed utility to verify whether or not the InfiniBand link is up The utility also checks for and displays additional information such as HCA firmware version Kernel architecture Driver version Number of active HCA ports along with their states Node GUID Note For more details on hca self test ofed see the file hca self test readme under docs hostl usr bin hca self test ofed Performing InfiniBand HCA Self Test Number of HCAs Detected 1 POE Device Cheki aii d vr en er eren WERT Eech PASS Kernel Arh bibe DS RAN ER eias x86 64 Host Driver Verslon 64 ett oa deus LNX OFED LINUX 1 5 2 0 0 5 OFED 1 5 2 20101014 1355 1 5 2 2 6 32 12 0 7 default Host Driver RPM Check ees PASS HCA Firmware on HCA 40 v2 8 0000 HCA Firmware Check on HCA 40 PASS Host Driver Initialization x0 93 PASS umber of HCA Ports Active 1 Port State of Port 1 on HCA 0 UP 4X DDR Port State of Port 2 on HCA 0 INIT Error Counter Check on HCA 0 PASS Kernel Syslog Check senge ges es ern PASS Node GUID on HCA 0 eee eee ee ee ee eee 00 02 c9 03 00 00 10 e0 Note After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters can be retrieved by running the command etc infiniband info 2 3 4 Installation Results
11. h pgi mpitests mvapich intel mpitests openmpi gcc mpitests openmpi pgi mpitests openmpi intel Device 15b3 634a 02 00 0 InfiniBand Mellanox Technologies MT25418 ConnectX VPI PCIe 2 0 2 5GT s IB DDR 10GigE rev a0 Link Width 8x Mellanox Technologies 31 Mellanox Technologies Confidential Rev 1 5 2 Installation Link Speed 2 5Gb s Installation finished successfully Programming HCA firmware for dev mst mt25418 pci cr0 device Running mlxburn d dev mst mt25418 pci cr0 fw mnt firmware fw 25408 2 8 0000 fw 25408 rel mlx dev type 25408 no I Querying device I Using auto detected configuration file mnt firmware fw 25408 2 8 0000 MHGH28 XTC A4 A7 ini PSID MT 04A0140005 I Generating image Current FW version on flash 2 7 0 New FW version 2 8 0 Burning FW image without signatures OK Restoring signature OK I Image burn completed successfully Please reboot your system for the changes to take effect warning etc infiniband openib conf saved as etc infiniband openib conf rpm save Note In case your machine has the latest firmware no firmware update will occur and the instal
12. lation script will print at the end of installation a message similar to the follow ing Installation finished successfully The firmware version 2 8 0000 is up to date Note To force firmware update use force fw update flag Note In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file Step4 In case the installation script performed firmware updates to your network adapter hardware it will ask you to reboot your machine Step 5 The script adds the following lines to etc security limits conf for the userspace components such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a user space application If desired tune the value unlimited to a specific amount of RAM Step6 For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the 32 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 OpenSM Subnet Manager on your machine For details on starting OpenSM see Chapter 7 OpenSM Subnet Manager Step7 InfiniBand only Run the hca s
13. r docs 2 Burning a firmware image from a mlx file using the mlxburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 hostl mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408 fw 25408 rel mlx Warning Make sure that you have the correct device name firmware path and firmware file name before running this command For help please refer to the Mellanox Firm ware Tools MFT User s Manual under mnt docs Step 4 Reboot your machine after the firmware burning is completed 2 5 Uninstalling Mellanox OFED Use the script usr sbin ofed uninstall sh to uninstall the Mellanox OFED package The script is part ofthe ofed scripts RPM Mellanox Technologies 37 J Mellanox Technologies Confidential
14. urn an Expansion ROM image please refer to Burning the Expansion ROM Image on page 188 Note The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site http www mellanox com gt Downloads gt Firmware Step 1 Start mst hostl mst start Step 2 Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine hostl mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded MST devices dev mst mt25418 pciconf0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 dev mst mt25418 pci Grp PCI direct access bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is A0 dev mst mt25418 pci meist PCI direct access 36 Mellanox Technologies Mellanox Technologies Confidential Mellanox OFED for Linux User s Manual Rev 1 5 2 bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix pci er In the example listed above this will be dev mst mt25418 pci crO0 Step3 Burn firmware 1 Burning a firmware binary image using mst flint that is already installed on your machine Please refer to MSTFLINT README txt unde
15. wing command will create a MLNX OFED LINUX ISO image for RedHat 5 4 under the tmp directory MLNX OFED LINUX 1 5 1 rhel5 4 docs mlnx add kernel support sh i mnt MLNX OFED LINUX 1 5 1 rhel5 4 iso All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 1 rhel5 4 iso 2 3 2 Installation Script Mellanox OFED includes an installation script called mlinxofedinstall Its usage is described below You will use it during the installation procedure described in Section 2 3 3 Installation Pro cedure on page 29 Usage mnt mlnxofedinstall OPTIONS Note If no options are provided to the script then all available RPMs are installed Options c config packages config file Example of the configuration file can be found under docs n net network config file Example of the network configuration file can be found under docs p print available Print available packages for the current platform and cre ate a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update force fw update Force firmware update force Force installation without querying the user
Download Pdf Manuals
Related Search