User Manual v4.15.2.0
Contents
1. Mellanox TECHNOLO Connect Accelerate Outperform Mellanox ConnectX 4 NATIVE ESX Driver for VMware vSphere 6 0 User Manual Rev 4 15 2 0 www mellanox com Rev 4 15 2 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATI2. Raid Rn DACH eee AG Ron br d 4 Document Revision History ue 9 RR Er ow EE REY Pe 5 this Manual E 6 Chapter 1 Introduction 8 Il bcos 8 12 Mellanox NATIVE ESX ConnectX 4 Package 8 12 1 Software Components E aY US AARE E SE 8 1 3 Module Parameters 8 1 3 1 nmlx5 Module Parameters 8 Chapter 2 Installation 10 2 1 Hardware and Software Requirements 10 2 2 Installing Mellanox ConnectX 4 NATIVE ESX Driver for VMware vSphere 10 2 3 Removing Mellanox OFED Driver 11 2 4 Loading Unloading Driver Kernel 11 2 5 Firmware Programming 12 Chapter3 Troubleshooting 13 3 l General Related Iss088 ree teet eee Pts Paste ng 13 3 2 Ethernet Related Issues zoe RC EORR ER Re a 13 3 3 Installation Related Issues 14 Mellanox Technologies 3 J Rev 4 15 2 0 List of Tables Table 1 Document Revision History 5 Table 2 Abbreviations and Acronyms 6 T
3. www mellanox com gt Support gt Firmware Download MFT User Manual Mellanox Firmware Tools User s Manual For further information please refer to the Mellanox website www mellanox com gt Products Ethernet Drivers gt Firmware Tools MFT Release Notes Release Notes for the Mellanox Firmware Tools For further information please refer to the Mella nox website www mellanox com gt Products Ethernet Drivers gt Firmware Tools VMware vSphere 6 0 Documentation VMware website Center Mellanox Technologies 7 J Rev 4 15 2 0 Introduction 1 1 1 1 2 1 2 1 1 3 1 3 1 Introduction Mellanox ConnectX amp 4 NATIVE ESX is a software stack which operates across all Mellanox network adapter solutions supporting up to 100Gb s Ethernet ETH and 2 5 or 5 0 GT s PCI Express 2 0 and 3 0 uplinks to servers The following sub sections briefly describe the various components of the Mellanox Con nectX 4 NATIVE ESX stack nmlx5 Driver nm1x5 is the low level driver implementation for the ConnectX 4 adapter cards designed by Mellanox Technologies ConnectX 4 adapter cards can operate as an InfiniBand adapter or as an Ethernet NIC The ConnectX 4 NATIVE ESX driver supports Ethernet NIC configurations To accommodate the supported configurations the driver consist of mlnx5_core module nmlx5 core 10 25 40 50 100GigE driver that handles Ethernet specific functions and plugs into t
4. L X Device ID For the latest list of device IDs please visit Mellanox website Operating System ESXi 6 0 operating system Installer Privileges The installation requires administrator privileges on the target machine 2 2 Installing Mellanox ConnectX 4 NATIVE ESX Driver for VMware vSphere Please uninstall any previous Mellanox driver packages prior to installing the new ver sion Pd To install the driver 1 Log into the ESXi server with root permissions 2 Install the driver gt esxcli software vib install d path bundle file Example gt esxcli software vib install d path bundle file 3 Reboot the machine 4 Verify the driver was installed successfully esxcli software vib list grep mlx nmlx5 core 4 15 2 0 10EM 600 0 0 2768847 MEL PartnerSupported 2015 09 17 After installation process all kernel modules loaded automatically upon boot PI 10 Mellanox Technologies Rev 4 15 2 0 23 Removing Mellanox OFED Driver Please unload the driver before removing it A gt To remove all the drivers 1 Log into the ESXi server with root permissions 2 List the existing ConnectX 4 NATIVE ESX driver modules see Step 4 in Section 2 2 on page 10 3 Remove each module gt esxcli software vib remove nmlx5 core remove modules command must run in same order shown in example above 4 Re
5. ON EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 O Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX amp ConnectX Connect IB CoolBox amp CORE Direct amp GPUDirect InfiniBridge InfiniHost InfiniScale amp Kotura amp Kotura logo Mellanox Connect Accelerate Outperform logo Mellanox Federal Systems Mellanox Open Ethernet amp Mellanox Virtual Modular Switch MetroDX MetroX amp MLNX OS Open Ethernet logo PhyX ScalableHPC SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire amp and Voltaire logo are registered trademarks of Mellanox Technologies Ltd CyPU ExtendX FabricIT FPGADirect HPC X Mellanox Care CloudX NEO Mellanox Open Ethernet PeerDirect NVMeDirect StPU Spectrum Switch IBTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 4896 Rev 4 15 2 0 Table of Contents Contents s i ARE REX RE 3 List of Tales oss ACA HOS Re
6. Related Issues Issue Cause Solution Physical link fails to The cable is not connected Ensure that the cable is connected on come up to the port port onthe both ends use a known working cable other end of the cable is disabled 3 3 Installation Related Issues Table 5 Installation Related Issues Issue Cause Solution Driver installation fails The install script may fail Uninstall the previous driver before for the following reasons installing the new one Failed to uninstall the Usea supported operating system previous installation due and kernel to dependencies being used The operating system is not supported 14 Mellanox Technologies
7. Remove the failed adapter Reboot the system adapter installed Mellanox adapter is not PCI slot or adapter PCI identified as a PCI connector dysfunctionality device ka Run 1spci Reseat the adapter in its PCI slot or insert the adapter to a different PCI slot If the PCI slot confirmed to be func tional the adapter should be replaced Mellanox adapters are Misidentification of the not installed in the sys Mellanox adapter installed tem Run the command below to identify the Mellanox adapter installed lspci grep Mellanox 3 2 Ethernet Related Issues Table 4 Ethernet Related Issues Issue Cause Solution No link Mis configuration of the switch port or using a cable not supporting link rate Ensure the switch port 1s not down Ensure the switch port rate 15 config ured to the same rate as the adapter s port No link with break out Misuse of the break out cable cable or misconfiguration of the switch s split ports Use supported ports on the switch with proper configuration For fur ther information please refer to the MLNX_OS User Manual Make sure the QSFP break out cable side is connected to SwitchX Physical link fails to The adapter is running an negotiate to maximum outdated firmware supported rate Install the latest firmware on the adapter Mellanox Technologies 13 J Rev 4 15 2 0 Troubleshooting Table 4 Ethernet
8. able 3 Reference Documents 7 Table4 5 Parameters asam e bere bina ele Ruan Ea avalon dee 9 Table 5 Software and Hardware Requirements 10 Table 6 General Related Issues 13 Table 7 Ethernet Related Issues 13 Table 8 Installation Related Issues 14 4 Technologies J Rev 4 15 2 0 Document Revision History Table 1 Document Revision History Release Date Description Rev 4 15 2 0 September 2015 Initial release of this MLNX NATIVE ESX ConnectX 4 version Mellanox Technologies 5 J Rev 4 15 2 0 About this Manual This preface provides general information concerning the scope and organization of this User s Manual Intended Audience This manual is intended for system administrators responsible for the installation configuration management and maintenance of the software and hardware of VPI in Ethernet mode and Ethernet adapter cards It is also intended for application developers Common Abbreviations and Acronyms Table 2 Abbreviations and Acronyms Sheet 1 of 2 Abbreviation Acronym Whole Word Description B Capital is used to indi
9. boot the server 2 4 Loading Unloading Driver Kernel Modules Tounload the driver esxcfg module u nmlx5 core To load the driver etc init d sfcbd watchdog stop esxcfg module nmlx5 core etc init d sfcbd watchdog start kill POLL cat var run vmware vmkdevmgr pid gt To restart the driver etc init d sfcbd watchdog stop esxcfg module u nmlx5 core esxcfg module nmlx5 core etc init d sfcbd watchdog start kill POLL cat var run vmware vmkdevmgr pid Mellanox Technologies 11 J 4 15 2 0 Installation 2 5 Firmware Programming 1 Download the VMware bootable binary images v4 1 0 from the Mellanox Firmware Tools site File mft 4 1 0 34 TOEM 610 0 0 2770837 x86 64 vib MDSSUM 7b8552c1a22b554c4a280d679dfd528f 2 Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downloading mounting and boot ing from a USB device po 12 Mellanox Technologies Rev 4 15 2 0 3 Troubleshooting You may be able to easily resolve the issues described in this section If a problem persists and you are unable to resolve it yourself please contact your Mellanox representative or Mellanox Support at support mellanox com 3 1 General Related Issues Table 3 General Related Issues Issue Cause Solution The system panics when Malfunction hardware com it is booted with a failed ponent
10. cate size in bytes or multiples of bytes e g IKB 1024 bytes and 1048576 bytes b Small b is used to indicate size in bits or multiples of bits e g IKb 1024 bits FW Firmware HCA Host Channel Adapter HW Hardware LSB Least significant byte Isb Least significant MSB Most significant byte msb Most significant NIC Network Interface Card SW Software VPI Virtual Protocol Interconnect PR Path Record RDS Reliable Datagram Sockets SDP Sockets Direct Protocol SL Service Level MPI Message Passing Interface 005 Quality of Service ULP Upper Level Protocol 6 Mellanox Technologies J Rev 4 15 2 0 Table 2 Abbreviations and Acronyms Sheet 2 of 2 Abbreviation Acronym Whole Word Description vHBA Virtual SCSI Host Bus adapter uDAPL User Direct Access Programming Library Related Documentation Table 3 Reference Documents Document Name Description IEEE Std 802 3aeTM 2002 Part 3 Carrier Sense Multiple Access with Colli Amendment to IEEE Std 802 3 2002 sion Detection CSMA CD Access Method and Document PDF 5594996 Physical Layer Specifications Amendment Media Access Control MAC Parameters Physical Layers and Management Parameters for 10 Gb s Operation Firmware Release Notes for Mellanox See the Release Notes PDF file relevant to your adapter devices adapter device For further information please refer to the Mellanox website
11. he ESX uplink layer Mellanox NATIVE ESX ConnectX 4 Package Software Components MLNX NATIVE ESX ConnectX 4 contains the following software components Mellanox Host Channel Adapter Drivers 5 core Ethernet Module Parameters 5 Module Parameters To set nm1x5 parameters esxcli system module parameters set m nmlx5 core p parameter value To show all parameters which were set until now esxcli system module parameters list m module name Parameters which are not set by the user remain on default value The following sections list the available nm1x5 parameters 8 Technologies J Rev 4 15 2 0 1 3 1 1 nmlx5 core Parameters Table 1 nmlx5 core Parameters Name Description Values enable nmlx debug Enables debug prints for nmlx 1 enabled e 0 disabled Default 0 Mellanox Technologies 9 J Rev 4 15 2 0 Installation 2 Installation This chapter describes how to install and test the Mellanox ConnectX 4 NATIVE ESX package on a single host machine with Mellanox Ethernet adapter hardware installed 2 1 Hardware and Software Requirements Table 2 Software and Hardware Requirements Requirements Description Platforms server platform with an adapter card based on one of the following Mellanox Technologies devices e MT27700 ConnectX 4 VPI EN firmware fw ConnectX4 e MT27630 ConnectX 4 LX EN firmware fw ConnectX4
Download Pdf Manuals
Related Search