Mellanox OFED Linux User`s Manual
Contents
1. a aia Crete kai ae uw QR RE AR Roni 4 Document Revision History a 5 this lt usss yii eo hs uve ERR uses 6 Chapter 1 5555555 8 ET OMADE yu u tese Ie era a er aaa Prud An a Pe eb RIA Puto ien d 8 1 2 Ethernet EN Management 8 1 3 Mellanox Firmware Tools 8 1 4 Mellanox NATIVE ESX Package 9 1 4 1 Software Compo ente rnr idee ea e 9 1 5 Module Paraieters ug eR Xe EE REN dain Seb SN 9 1 5 1 nmlx4 Module Parameters 9 1 6 Device Capabilities y s uu isuu hupu ern 11 Chapter2 Installation seeks euer Ra RR A OPER CREARE REPRE 12 2 Hardware and Software Requirements 12 2 2 Installing Mellanox NATIVE ESX Driver for VMware vSphere 12 2 3 Removing Mellanox OFED Driver 13 2 4 Loading Unloading Driver Kernel Modules 13 2 5 Firmware Programming 14 Chapter3 Features Overview and Configuration 15 3 1 VXLAN Hardware Offload 15 3 1 12. Please uninstall any previous Mellanox driver packages prior to installing the new ver 2d Sion gt To install the driver l 2 Log into the ESXi server with root permissions Install the driver gt esxcli software vib install d path bundle file Example gt esxcli software vib install d path MLNX NATIVE ESX ConnectX 3 3 2 0 15 10EM 600 0 0 2768847 zip Reboot the machine Verify the driver was installed successfully esxcli software vib list grep mlx nmlx4 core 3 2 0 15 10EM 600 0 0 2768847 MEL PartnerSupported 2015 11 15 nmlx4 en 3 2 0 15 10EM 600 0 0 2768847 MEL PartnerSupported 2015 11 15 nmlx4 rdma 3 2 0 15 10EM 600 0 0 2768847 MEL PartnerSupported 2015 11 15 After the installation process all kernel modules are loaded automatically upon boot A 12 Mellanox Technologies Rev 3 2 0 15 2 3 Removing Mellanox OFED Driver Please unload the driver before removing it To remove all the drivers 1 Log into the ESXi server with root permissions 2 Listthe existing NATIVE ESX driver modules see Step 5 in Section 2 2 on page 12 3 Remove each module gt esxcli software vib remove n nmlx4 rdma gt esxcli software vib remove n nmlx4 en gt esxcli software vib remove n nmlx4 core remove the modules command must be run in the same order as shown in example above A 4 Reboot the server 2 4 Loading Unloading Driver Kerne
3. Configuring VXLAN Hardware Offload 15 Chapter 4 Troubleshooting 16 4 General Related Iss es nias ien a baha 16 4 2 Ethernet Related Issues 16 4 3 Installation Related Issues _ 17 Mellanox Technologies 3 J I Rev 3 2 0 15 List of Tables Table 1 Document Revision History 5 Table 2 Abbreviations and Acronyms 6 Table 3 Reference Documents 7 Table 4 nmlx4_ core Parameters uu za aspa cece nen en en eee nen 10 Table 5 nmlx4 en Parameters 11 Table 6 Software and Hardware Requirements 12 Table 7s General Related Issues Back Svea anton e area v Ste Ts 16 Table8 Ethernet Related Issues 00 0 lisse 16 Table 9 Installation Related Issues 17 4 Mellanox Technologies J Rev 3 2 0 15 Document Revision History Table 1 Document Revision History Release Date Description 3 2 0 15 November 2015 Updated the following sections Section 2 2 Installing Mellanox NATIVE ESX Driver for VMware vSphere on page 12 Sec
4. Mellanox TE CHNOL OGIES Connect Accelerate Outperform Mellanox NATIVE ESX Driver for VMware vSphere 6 0 User Manual Rev 3 2 0 15 www mellanox com Rev 3 2 0 15 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION E
5. VEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 O Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox amp Mellanox logo BridgeX amp CloudX logo Connect IB ConnectX CoolBox CORE Direct amp GPUDirect InfiniHost InfiniScale amp Kotura Kotura logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox ScalableHPC Mellanox Connect Accelerate Outperform logo Mellanox Virtual Modular Switch MetroDX MetroX MLNX OS Open Ethernet logo PhyX SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd Accelio CyPU FPGADirect M HPC X InfiniBridge LinkX Mellanox Care Mellanox CloudX Mellanox Multi Host Mellanox NEO Mellanox PeerDirect Mellanox Socket Direct Mellanox Spectrum NVMeDirect StPU Spectrum logo Switch IB Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 4896 Rev 3 2 0 15 Table of Contents Table of Contents ias RE X ERs 3 List of Fables
6. cli system module parameters set m nmlx4 core p parameter value and or esxcli system module parameters set m nmlx4 en p parameter value To show all parameters which were set until now esxcli system module parameters list m module name Parameters which are not set by the user remain on default value The following sections list the available nm1x4 parameters Mellanox Technologies 9 J Rev 3 2 0 15 Introduction 1 5 1 1 4 Parameters Table 1 nmlx4 core Parameters Name Description Values enable_64b_cqe_eqe Enables 64 byte CQEs EQEs when the enabled firmware supports this 0 disabled Default 0 enable dmfs Enables Device Managed Flow Steering enabled 0 disabled Default 1 enable qos Enables Quality of Service support in the enabled HCA 0 disabled Default 0 enable rocev2 Enables RoCEv2 mode for all devices 1 enabled 0 disabled Default 0 enable vxlan offloads Enables VXLAN offloads when supported 1 enabled by NIC 0 disabled Default 1 log mtts per seg Log2 number of MTT entries per segment 1 7 Default 3 log num mgm entry size Log2 MGM entry size that defines the 9 12 number of QPs per MCG for example Default 12 value 10 results in 248 QP per MGM entry msi x Enables MSI X l enabled 0 disabled Default 1 mst recovery Enables recovery mode only NMST mod 1 enabled ule is load
7. e of the break out Use supported ports on the switch cable cable or misconfiguration with proper configuration For fur of the switch s split ports ther information please refer to the MLNX OS User Manual Make sure the QSFP break out cable side is connected to the SwitchX Physical link fails to The adapter is running an Install the latest firmware on the negotiate to maximum outdated firmware adapter supported rate 16 Mellanox Technologies Rev 3 2 0 15 Table 5 Ethernet Related Issues Issue Cause Solution Physical link fails to The cable is not connected Ensure that the cable is connected on come up to the port or the port onthe both ends or use a known working cable other end of the cable is disabled 4 3 Installation Related Issues Table 6 Installation Related Issues Issue Cause Solution Driver installation fails The install script may fail Uninstall the previous driver before for the following reasons installing the new one Failed to uninstall the Usea supported operating system previous installation due and kernel to dependencies being used The operating system is not supported Mellanox Technologies 17 J
8. ed 0 disabled Default 0 rocev2 udp port Destination port for RoCEv2 1 65535 for RoCEv2 Default 4791 1 The following module parameter is not supported in this version 10 Mellanox Technologies Rev 3 2 0 15 1 5 1 2 nmlx4 en Parameters Table 2 nmlx4_en Parameters Name Description Value num rings per rss Enables RSS 2 4 queue 0 disabled Default 0 When this value is 0 RSS is enabled with 1 RSS Queue that manages num rings per rss queue Rx Rings Note The value must be a power of 2 pfcrx Priority based Flow 0 255 Control policy on RX Default 0 Itis a 8 bits bit mask each bit indicates priority 0 7 respects incoming pause frames on the specified priority O ignore incoming pause frames the specified priority pfctx Priority based Flow 0 255 Control policy on TX Default 0 Itis a 8 bits bit mask each bit indicates priority 0 7 generates pause frames according to the RX buffer threshold on the specified priority 0 never generates pause frames on the specified priority vlan filter Enables VLAN filter 1 enabled 0 disabled Default 0 1 6 Device Capabilities Normally an application needs to query the device capabilities before attempting to create a resource It is essential for the application to be able to operate over different devices with differ ent capabilities Specif
9. face access layer provides a set of accessors functions to driver objects and it 1s provided as a development source package tarball H and C files The user should include the H file and compile and link against the C file 1 3 Mellanox Firmware Tools The Mellanox Firmware Tools MFT package is a set of firmware management tools for a single node MFT can be used for Generating a standard or customized Mellanox firmware image Burning a firmware image to a single node MFT includes the following tools flint burns a firmware binary image or an expansion ROM image to the Flash device of a Mellanox network adapter bridge switch device It includes query functions to the burnt firmware image and to the binary image file Debug utilities A set of debug utilities e g itrace mlxdump mstdump mlxmcg wqdump mcra 12 mget temp and pckt drop 8 Mellanox Technologies J Rev 3 2 0 15 For additional details please refer to the MFT User s Manual www mellanox com gt Products gt InfiniBand VPI Driver Firmware Tools 1 4 Mellanox NATIVE ESX Package 1 4 4 Software Components MLNX NATIVE ESX contains the following software components Mellanox Host Channel Adapter Drivers nmlx4 which is split into multiple modules nmlx4 core low level helper nmlx4 en Ethernet EN Management Interface development package tarball 1 5 Module Parameters 1 5 1 nmlx4 Module Parameters To set nm1x4 parameters esx
10. ically when creating a QP the user needs to specify the maximum number of outstanding work requests that the QP supports This value should not exceed the queried capabilities How ever even when you specify a number that does not exceed the queried capability the verbs can still fail since some other factors such as the number of scatter gather entries requested or the size of the inline data required affect the maximum possible work requests Hence an application should try to decrease this size halving is a good new value and retry until it succeeds Mellanox Technologies 11 J Rev 3 2 0 15 Installation 2 nstallation This chapter describes how to install and test the Mellanox NATIVE ESX package on a single host machine with Mellanox Ethernet adapter hardware installed 2 1 Hardware and Software Requirements Table 3 Software and Hardware Requirements Requirements Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies HCA devices e MT27508 ConnectX 3 VPI EN firmware fw ConnectX3 MT4103 ConnectX 3 Pro VPI EN firmware fw ConnectX3Pro Device ID For the latest list of device IDs please visit Mellanox website Operating System ESXi 2015 operating system Installer Privileges The installation requires administrator privileges on the target machine 2 2 Installing Mellanox NATIVE ESX Driver for VMware vSphere
11. l Modules Tounload the driver esxcfg module u nmlx4 rdma esxcfg module u nmlx4 en esxcfg module u nmlx4 core To load the driver etc init d sfcbd watchdog stop esxcfg module nmlx4 core esxcfg module nmlx4 en esxcfg module nmlx4 rdma etc init d sfcbd watchdog start kill POLL cat var run vmware vmkdevmgr pid gt To restart the driver etc init d sfcbd watchdog stop esxcfg module u nmlx4_rdma esxcfg module u nmlx4 en esxcfg module u nmlx4 core esxcfg module nmlx4 core esxcfg module nmlx4 en esxcfg module nmlx4 rdma etc init d sfcbd watchdog start kill POLL cat var run vmware vmkdevmgr pid Mellanox Technologies 13 J Rev 3 2 0 15 Installation 25 Firmware Programming 1 Download the VMware bootable binary images v3 8 0 from the Mellanox Firmware Tools MFT site File mft 3 8 0 56 TOEM 600 0 0 2295424 x86 64 vib MDSSUM 083baec399de55a181f5b26613ae0829 File nmst 3 8 0 56 10EM 600 0 0 2295424 x86_64 vib MDSSUM 0426a9ab6e759ad44942d506 la6e9cfe 2 Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downloading mounting and boot ing from a USB device 14 Mellanox Technologies Rev 3 2 0 15 3 Features Overview and Configuration 3 1 VXLAN Hardware Offload VXLAN hardware offload enables the traditional offloads to be performed on the encapsulated traffic With ConnectX 3 Pro data cente
12. m Sockets SDP Sockets Direct Protocol SL Service Level MPI Message Passing Interface QoS Quality of Service ULP Upper Level Protocol 6 Mellanox Technologies J Rev 3 2 0 15 Table 2 Abbreviations and Acronyms Sheet 2 of 2 Abbreviation Acronym Whole Word Description vHBA Virtual SCSI Host Bus adapter uDAPL User Direct Access Programming Library Related Documentation Table 3 Reference Documents Document Name Description IEEE Std 802 3aeTM 2002 Part 3 Carrier Sense Multiple Access with Colli Amendment to IEEE Std 802 3 2002 sion Detection CSMA CD Access Method and Document PDF 5594996 Physical Layer Specifications Amendment Media Access Control MAC Parameters Physical Layers and Management Parameters for 10 Gb s Operation Firmware Release Notes for Mellanox See the Release Notes PDF file relevant to your adapter devices adapter device For further information please refer to the Mellanox website www mellanox com gt Support gt Firmware Download MFT User Manual Mellanox Firmware Tools User s Manual For further information please refer to the Mellanox website www mellanox com gt Products gt InfiniBand VPI Driver gt Firmware Tools MFT Release Notes Release Notes for the Mellanox Firmware Tools For further information please refer to the Mella nox website www mellanox com gt Products gt InfiniBand VPI Driver gt Firmwa
13. n Mellanox Technologies 15 J Rev 3 2 0 15 Troubleshooting 4 Troubleshooting You may be able to easily resolve the issues described in this section If a problem persists and you are unable to resolve it yourself please contact your Mellanox representative or Mellanox Support at support mellanox com 4 1 General Related Issues Table 4 General Related Issues Issue Cause Solution The system panics when it is booted with a failed adapter installed Malfunction hardware com ponent Remove the failed adapter 2 Reboot the system Mellanox adapter is not identified as a PCI device PCI slot or adapter PCI connector dysfunctionality Run lspci 2 Reseat the adapter in its PCI slot or insert the adapter to a different PCI slot If the PCI slot confirmed to be func tional the adapter should be replaced Mellanox adapters are not installed in the sys tem Misidentification of the Mellanox adapter installed Run the command below to identify the Mellanox adapter installed lspci grep Mellanox 4 2 Ethernet Related Issues Table 5 Ethernet Related Issues Issue Cause Solution No link Mis configuration of the switch port or using a cable not supporting link rate Ensure the switch port is not down Ensure the switch port rate is config ured to the same rate as the adapter s port No link with break out Misus
14. r operators can decouple the overlay network layer from the physical NIC performance thus achieving native performance in the new network architecture 3 1 1 Configuring VXLAN Hardware Offload VXLAN hardware offload includes TX Calculates the Inner L3 L4 and the Outer L3 checksum RX Checks the Inner L3 L4 and the Outer L3 checksum Maps VXLAN traffic to an RX queue according to Inner destination MAC address Outer destination MAC address VXLAN ID VXLAN hardware offload is enabled by default However if it was disable and you want to re enable it enable the nml1x4 core module parameters enable vxlan offloads and enable dmfs setting the parameters to 1 gt enable VXLAN hardware offload esxcli system module parameters set m nmlx4 core p enable vxlan offloads 1 esxcli system module parameters set m nmlx4 core p enable dmfs 1 To disable VXLAN hardware offload esxcli system module parameters set m nmlx4 core p enable vxlan offloads 0 Except for the module parameters set above the rest of VXLAN configuration is done in the ESX environment via VMware NSX manager For additional NSX information please refer to VMware documentation see http pubs vmware com NS X 62 index jsp topic 62Fcom vmware nsx install doc 92FGUID D18A11DF 3D85 4B80 8713 D611648D43F4 html Additional information can be found at http dailyhypervisor com vmware nsx for vsphere 6 1 step by step installatio
15. re Tools VMware vSphere 6 0 Documentation VMware website Center Mellanox Technologies 7 J Rev 3 2 0 15 Introduction 1 Introduction Mellanox NATIVE ESX is a software stack which operates across Mellanox s ConnectX 3 and ConnectX 3 Pro network adapter solutions supporting up to 40Gb s Ethernet ETH and 2 5 or 5 0 GT s PCI Express 2 0 and 3 0 uplinks to servers The following sub sections briefly describe the various components of the Mellanox NATIVE ESX stack 1 1 4 Driver nmlx4 is the low level driver implementation for ConnectX 3 and ConnectX 8 3 Pro adapt ers designed by Mellanox Technologies ConnectX family adapters can operate as an Infini Band adapter or as an Ethernet NIC The NATIVE ESX driver supports Ethernet NIC configurations To accommodate the supported configurations the driver is split into the follow ing modules nmix4 core Handles low level functions like device initialization and firmware commands processing Also controls resource allocation nmix4 en A 10 40GigE driver that handles Ethernet specific functions and plugs into the ESX uplink layer 1 2 Ethernet EN Management Interface EN Management Interface provides user space application means to access EN driver data simi lar to ethtool interface in Linux The kernel space management interface layer is implemented as part of the EN driver and released as part of nmlx4 en The user space management inter
16. tion 2 3 Removing Mellanox OFED Driver on page 13 Section 2 4 Loading Unloading Driver Kernel Modules on page 13 3 2 0 November 2015 Updated section Section 3 1 VXLAN Hardware Offload on page 15 September 2015 Initial release of this MLNX NATIVE ESX version Mellanox Technologies 5 J I Rev 3 2 0 15 About this Manual This preface provides general information concerning the scope and organization of this User s Manual Intended Audience This manual is intended for system administrators responsible for the installation configuration management and maintenance of the software and hardware of VPI in Ethernet mode and Ethernet adapter cards It is also intended for application developers Common Abbreviations and Acronyms Table 2 Abbreviations and Acronyms Sheet 1 of 2 Abbreviation Acronym Whole Word Description B Capital B is used to indicate size in bytes or multiples of bytes e g 1 1024 bytes and 1MB 1048576 bytes b Small b is used to indicate size in bits or multiples of bits e g IKb 1024 bits FW Firmware HCA Host Channel Adapter HW Hardware LSB Least significant byte Isb Least significant bit MSB Most significant byte msb Most significant bit NIC Network Interface Card SW Software VPI Virtual Protocol Interconnect PR Path Record RDS Reliable Datagra
Download Pdf Manuals
Related Search