Mellanox OFED Linux User`s Manual
Contents
1. b Your InfiniBand device is the one with the postfix pci cr0 In the example listed above this will be dev mst mt25418 pci cr0 Step 3 Burn firmware a Burning a firmware binary image using mstflint that is already installed on your machine Please refer to MSTFLINT README txt under docs b Burning a firmware image from a mlx file using the mlxburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408 fw 25408 rel mlx Warning Make sure that you have the correct device name firmware path and firmware file name before running this command For help please refer to the Mellanox Firm ware Tools MFT User s Manual under mnt docs Step 3 Reboot your machine after the firmware burning is completed Uninstalling Mellanox OFED Use the script usr sbin ofed uninstall sh to uninstall the Mellanox OFED package The script is part of the ofed scripts RPM Mellanox Technologies 152. MT25408 ConnectX 2 VPI IB EN FCoE firmware fw ConnectX2 MT25408 ConnectXg VPI IB EN FCoE firmware fw 25408 MT25208 InfiniHost III Ex firmware fw 25218 for Mem Free cards and fw 25208 for cards with memory MT25204 InfiniHost III Lx firmware fw 25204 MT23108 InfiniHost firmware fw 23108 Note For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for Installation 400 MB Software Requirements Operating System e Linux operating system Note For the list of supported operating system distributions and kernels please refer to the Mellanox OFED Release Notes file Installer Privileges e The installation requires administrator privileges on the target machine Mellanox Technologies 3 J Rev 1 5 1 Downloading Mellanox OFED Step 1 Verify that the system has a Mellanox network adapter HCA NIC installed by ensuring that you can see ConnectX or InfiniHost entries in the display The following example shows a system with an installed Mellanox HCA hostl lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT25418 ConnectX IB DDR PCIe 2 0 2 5GT s a0 Step 2 Download the ISO image to your host The image s name has the format MLNX_OFED_LINUX lt ver gt lt OS label gt iso You can download it from http www mellanox com gt Products gt IB SW Drivers Step 3 Use the md5sum
3. libibumad devel 00 Preparing 00 libibmad devel 00 Preparing 00 libibmad devel 00 Preparing 00 ofed scripts 00 Preparing 00 libibverbs 00 Preparing 00 libibverbs 00 Preparing 00 libibverbs devel 00 Preparing 00 libibverbs devel 00 Preparing 00 libibverbs devel static 00 Preparing 00 libibverbs devel static 00 Preparing 00 libibverbs utils 00 Preparing 00 libmthca 00 Mellanox Technologies 7 Rev 1 5 1 Preparing 00 libmthca 00 Preparing 00 libmthca devel static 00 Preparing 00 libmthca devel static 00 Preparing 00 libm1x4 00 Preparing 00 libm1x4 00 Preparing 00 libmlx4 devel 00 Preparing 00 libmlx4 devel 00 Preparing 00 libibcm 00 Preparing 00 libibcm 00 Preparing 00 libibcm devel 00 Preparing 00 libibcm devel 00 Preparing 00 libibumad static 0
4. installation script exits after creating ofed conf with fc Install FCoE support Available on RHEL5 2 ONLY with 32bit Install 32 bit libraries default This is relevant for x86 64 and ppc64 platforms without 32bit Skip 32 bit libraries installation without ib bonding Skip ib bonding RPM installation without depcheck Skip Distro s libraries check without fw update Skip firmware update force fw update Force firmware update force Force installation without querying the user all Install all kernel modules libibverbs libibumad librd macm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector perftest sdp netstat and libsdp srptools rds tools static and dynamic libraries Mellanox Technologies 5 Rev 1 5 1 hpc basic msm v vv vvv q Install all kernel modules libibverbs libibumad librd macm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector dynamic librar ies Install all kernel modules libibverbs libibumad mft mstflint dynamic libraries Install all kernel modules libibverbs libibumad mft mstflint diagnostic tools OpenSM ib bonding dynamic libraries NOTE With msm flag the OpenSM daemon is configured to run upon boot Set verbosity level Set quiet no messages will be printed minxofedinstall Return Codes Table 1 lists the mlnxofedinstall
5. script return codes and their meanings Table 1 mlnxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 3 Failed to start the mst driver Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount the ISO image on your machine hosti mount o ro loop MLNX OFED LINUX lt ver gt lt 0S label gt iso mnt Note After mounting the ISO image mnt will be a Read Only folder Step 3 Run the installation script hostl mnt mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of OFED 6 Mellanox Technologies Mellanox OFED for Linux User s Manual Rev 1 5 1 Starting MLNX OFED LINUX 1 5 1 rc6 installation Installing kernel ib RPM Preparing 00 l kernel ib 00 Installing kernel ib devel RPM Preparing 00 1 kernel ib devel 00 Installing mft RPM Preparing 00 l imft 00 Installing mpi selector RPM Preparing 00 l impi selector 00 Install user level RPMs Preparing 00 libibumad 00 Preparing 00 libibumad 00 Preparing 00 libibmad 00 Preparing 00 libibmad 00 Preparing 00 libibumad devel 00 Preparing 00
6. ske ess oie ete se oie ete ets one ele x86 64 HOSE Driver VErSTOM ax MLNX OFED LINUX 1 5 1 OFED 1 5 1 mlnx9 1 5 1 2 6 9 89 ELlargesmp Host Driver REM Check uri ieee aw oe ew exe PASS HCA Firmware on HCA 0 2 7 000 HCA Firmware Check on HCA 0 PASS Host Driver Initialization PASS Number of HCA Ports Active 0 Port State of Port 0 on HCA 0 NIT Port State of Port 0 on HCA 0 DOWN Error Counter Check on HCA 0 PASS ernel Syslog Check sedicio t u sie usus PASS Node GUID on HCA 0 ec eee see see ee eee 00 02 c9 03 00 00 10 e0 S sie DON Note After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters can be retrieved by running the com mand etc infiniband info Installation Results Software e The OFED and MFT packages are installed under the usr directory e The kernel modules are installed under InfiniBand subsystem lib modules uname r updates kernel drivers infiniband mlx4 driver Under lib modules uname r updates kernel drivers net mlx4 you will find mlx4 core ko mlx4 en ko mlx4 ib ko and mlx4 fc if you ran the installation script with with fc RDS lib modules uname r updates kernel net rds rds ko 12 Mellanox Technologies Mellanox OFED for Linux User s Manual Rev 1 5 1 Bonding module lib modules
7. 0 Preparing 00 libibumad static 00 Preparing 00 libibmad static 00 Preparing 00 libibmad static 00 Preparing 00 ibsim 00 Preparing 00 librdmacm 00 Preparing 00 librdmacm 00 Preparing 00 librdmacm utils 00 Preparing 00 librdmacm devel 00 Preparing 00 librdmacm devel 00 Preparing 00 libsdp 00 Preparing 00 libsdp 00 Preparing 00 libsdp devel 00 Preparing 00 libsdp devel 00 8 Mellanox Technologies Mellanox OFED for Linux User s Manual Rev 1 5 1 Preparing 00 l opensm libs 00 Preparing 00 l opensm libs 00 Preparing 00 l opensm 00 opensmd O off soft 20Ff Soft dioff brott Groff Preparing 00 opensm devel 00 Preparing 00 opensm devel 00 Preparing 00 opensm static 00 Preparing 00 opensm static 00 Preparing 00 compat dapl 00 Preparing 00 compat dapl 00 Preparing 00 compat dapl devel 00 Preparing 00 compat dapl devel 00 Preparing 00 dapl 00 Preparing 00 dapl 00 Preparing 00 dapl devel 00 Preparing 00 dapl devel 00 Preparing 00 dapl devel static 00 Preparing 00 dapl devel static 00 Preparing 00 dapl utils 00 Preparing 00 perftest 00 Preparing 00 imstflint 00 Preparing 00 sdpnetstat 00 Preparing 00 srptools 00 Preparing 00 rds tools 00 Preparing 00 irnfs u
8. AA Mellanox TECHNOLOGIES Mellanox OFED for Linux Installation Guide Rev 1 5 1 www mellanox com Rev 1 5 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PRO VIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUS TOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CAN NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MER CHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEM PLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCURE MENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellano
9. ctory Usage mlnx add kernel support sh i iso lt mlnx iso gt t tmpdir lt local work dir gt v verbose 1 The firmware will not be updated if you run the install script with the without fw update option 4 Mellanox Technologies Mellanox OFED for Linux User s Manual Rev 1 5 1 Example The following command will create a MLNX OFED LINUX ISO image for RedHat 5 4 under the tmp directory MLNX OFED LINUX 1 5 1 rhel5 4 docs mlnx add kernel support sh i mnt MLNX_OFED_LINUX 1 5 1 rhel5 4 iso All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 1 rhel5 4 iso Installation Script Mellanox OFED includes an installation script called minxofedinstall Its usage is described below You will use it during the installation procedure described in Section Installation Proce dure on page 6 Usage mlnxofedinstall OPTIONS Note If no options are provided to the script then all available RPMs are installed Options c config lt packages config file gt Example of the configuration file can be found under docs n net lt network config file gt Example of the network configuration file can be found under docs p print available Print available packages for the current platform and cre ate a corresponding ofed conf file The
10. ile Step 4 In case the installation script performed firmware updates to your network adapter hard ware it will ask you to reboot your machine Step 5 The script adds the following lines to etc security limits conf for the userspace components such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a user space application If desired tune the value unlimited to a specific amount of RAM Step 6 For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be run ning on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM see Chapter 12 OpenSM Subnet Manager Step 7 InfiniBand only Run the hca self test ofed utility to verify whether or not the InfiniBand link is up The utility also checks for and displays additional information such as Mellanox Technologies 11 Rev 1 5 1 e HCA firmware version e Kernel architecture Driver version e Number of active HCA ports along with their states e Node GUID Note For more details on hca self test ofed see the file hca_self test readm under docs hostl usr bin hca self test ofed Performing InfiniBand HCA Self Test Number of HCAs Detected 1 PET Device Check 294442465565 2555 u kis PASS Enel Arch rene ees
11. lly upon boot can be found in the etc infiniband openib conf file Updating Firmware After Installation In case you ran the mlnxofedinstall script with the without fw update option and now you wish to manually update firmware on you adapter card s you need to perform the fol lowing steps Note If you need to burn an Expansion ROM image please refer to Burning the Expansion ROM Image on page 191 Note The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site http www mellanox com gt Downloads gt Firmware Step 1 Start mst host1 mst start Step 2 Identify your target InfiniBand device for firmware update a Get the list of InfiniBand device names on your machine hostl mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded MST devices dev mst mt 25418 pciconf0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 dev mst mt25418 pci cr0 PCI direct access bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is AO dev mst mt25418 pci msix0 PCI direct access bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 14 Mellanox Technologies Mellanox OFED for Linux User s Manual Rev 1 5 1
12. mming HCA firmware for dev mst mt25418 pci cr0 device Running mlxburn d dev mst mt25418 pci cr0 fw tmp MLNX OFED LINUX 1 5 1 MLNX OFED LINUX 1 5 1 sles1ll firmware fw 25408 2_7 000 fw 25408 rel mlx dev type 25408 no I Querying device I Using auto detected configuration file tmp MLNX OFED LINUX 1 5 1 MLNX OFED LINUX 1 5 1 sles11 firmware fw 25408 2 7 000 MHGH28 XTC A4 A7 ini PSID MT 0440140005 10 Mellanox Technologies J Mellanox OFED for Linux User s Manual Rev 1 5 1 I Generating image Current FW version on flash 2 6 0 New FW version 2 7 0 Burning FW image without signatures OK Restoring signature OK I Image burn completed successfully Please reboot your system for the changes to take effect warning etc infiniband openib conf saved as etc infiniband openib conf rpmsave Note In case your machine has the latest firmware no firmware update will occur and the installation script will print at the end of installation a message similar to the following Installation finished successfully The firmware version 2 7 000 is up to date Note To force firmware update use force fw update flag Note In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration f
13. tils 00 Preparing 00 Mellanox Technologies 9 Rev 1 5 1 ibutils 00 Preparing 00 ibutils2 00 Preparing 00 ibdump 00 Preparing 00 infiniband diags 00 Preparing 00 qperf 00 Preparing 00 imlnxofed docs 00 Preparing 00 imvapich gcc 00 Preparing 00 imvapich pgi 00 Preparing 00 imvapich intel 00 Preparing 00 openmpi gcc 00 Preparing 00 openmpi pgi 00 Preparing 00 openmpi intel 00 Preparing 00 impitests mvapich gcc 00 Preparing 00 impitests mvapich pgi 00 Preparing 00 impitests mvapich intel 00 Preparing 00 impitests openmpi gcc 00 Preparing 00 impitests openmpi pgi 00 Preparing 00 impitests openmpi intel 00 Device 15b3 634a 02 00 0 InfiniBand Mellanox Technologies MT25418 ConnectX IB DDR PCIe 2 0 2 5GT s rev a0 Link Width 8x Link Speed 2 5Gb s Installation finished successfully Progra
14. uded ifcfg ib lt n gt files will be installed under etc sysconfig network scripts ona RedHat machine etc sysconfig network on a SuSE machine e The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 e Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two condi tions are fullfilled 1 You run the installation script in default mode that is without the option without fw update 2 The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image Note Ifan adapter s Flash was originially programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image Incase your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message Mellanox Technologies 13 Rev 1 5 1 I Querying device E Can t auto detect fw configuration file Post installation Notes Most of the Mellanox OFED components can be configured or reconfigured after the installa tion by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatica
15. uname r updates kernel drivers net bonding bonding ko e The package kernel ib devel include files are placed under usr src ofa_kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel The raw package un backported source files are placed under usr src ofa_kernel lt ver gt The script openibd is installed under etc init d This script can be used to load and unload the software stack The script connectx port config is installed under sbin This script can be used to con figure the ports of ConnectX network adapter cards to Ethernet and or InfiniBand For details on this script please see Section 3 1 Port Type Management e The directory etc infiniband is created with the files info and openib conf and con nectx conf The info script can be used to retrieve Mellanox OFED installation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ether net and or InfiniBand This file is used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d e If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc If IPoIB configuration files are incl
16. utility to confirm the file integrity of your ISO image Run the following command and compare the result to the value provided on the download page host1 md5sum MLNX OFED LINUX lt ver gt lt OS label gt iso Installing Mellanox OFED The installation script minxofedinstal11 performs the following e Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack e Installs the MLNX_OFED LINUX binary RPMs if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automatically upgrades the firmware Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Note Pre existing configuration files will be saved with the extension conf saverpm Ifyou need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the minx add kernel support sh script located under the docs dire
17. x TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2010 Mellanox Technologies Inc All Rights Reserved Mellanox BridgeX ConnectX InfiniBlast InfiniBridge InfiniHost InfiniPCI InfiniRISC InfiniScale and Virtual Protocol Interconnect are registered trademarks of Mellanox Technologies Ltd CORE Direct FabricIT and PhyX are trademarks of Mellanox Technologies Ltd All other marks and names mentioned herein may be trademarks of their respective companies 2 Mellanox Technologies Document Number 2914 Mellanox OFED for Linux User s Manual Rev 1 5 1 Installation This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox InfiniBand and or Ethernet adapter hardware installed The chapter includes the following sections Hardware and Software Requirements page 3 Downloading Mellanox OFED page 4 Installing Mellanox OFED page 4 Uninstalling Mellanox OFED page 15 Hardware and Software Requirements Hardware Requirements Platforms A server platform with an adapter card based on one of the following Mellanox Technologies InfiniBand HCA devices
Download Pdf Manuals
Related Search