Mellanox MLX4_EN Driver for VMware ESXi 5.1 and ESXi 5.5 User
Contents
1. enable gos Enables Quality of Service support in the HCA 1 enabled e 0 disabled default 0 log mtts per seg Log2 number of MTT entries per segment 1 7 default 3 log num mgm entry size Log2 MGM entry size that defines the number of OPs per MCG Not in use with device managed flow steering 9 10 11 12 default 12 msi x Enables MSI X e 1 enabled 0 disabled default 1 mtu 4k Enables configuration of 4k MTU 1 enabled e 0 disabled default 0 Table 2 mlx4 en Module Parameters Parameter Description Values inline thold Threshold for using inline data 0 104 default 104 netg Uses netgueue 1 enabled e 0 disabled default 1 netg num rings per rss Number of rings per RSS netgueue 0 2 4 default 0 pferx Priority based Flow Control policy on RX Per priority bit 0 255 mask It is 8 bits bit mask each bit indicates priority 0 7 default 0 Bit value 1 respect incoming pause frames on the specified priority 0 ignore incoming pause frames on the specified priority pfctx Priority based Flow Control policy on TX Per priority bit 0 255 mask It s 8 bits bit mask each bit indicates priority 0 7 default 0 Bit value 1 generate pause frames according to the RX buffer threshold on the specified priority 0 never generate pause frames on the specified priority udp rss Per2. esxcli software vib remove n net ib sa esxcli software vib remove n net ib mad esxcli software vib remove n net ib core esxcli software vib remove n net mlx4 en esxcli software vib remove n net mlx4 core NAP NANTES ENNE EW To remove the offline bundle driver the command must be run in the same order as W shown in the example above Mal 2 3 Loading Unloading Driver Kernel Modules gt To unload the driver gt opt mellanox bin openibd sh stop gt To load the driver gt opt mellanox bin openibd sh start gt To restart the driver gt opt mellanox bin openibd sh restart 2 4 Driver Default Values The below are mlx4 en and mlx4 core module parameters Some of these values can be changed by using module parameters which can be obtained by run ning gt esxcli system module parameters list m lt module name gt For further information please refer to Section 4 1 Changing the Driver s Module Parameters Default Values on page 7 Table 1 mlx4_core Module Parameters Parameter Description Values debug_level Enables debug tracing e 1 enabled e 0 disabled default 0 enable 64b cge ege Enables 64 byte CQEs EQEs when it is supported by the e 1 enabled firmware 0 disabled default 0 Mellanox Technologies 4 Table 1 mlx4 core Module Parameters Rev 1 9 10 0 Driver Software Parameter Description Values
3. 1 Paths 2 Click Refresh The iSCSI Software adapter and the Mellanox iSER adapters one for each port will be displayed For example Device iSCSI Software Adapter vmhba37 iSCSI ign 1998 01 com vmware vsa24 7b35526 Patsburg 6 Port SATA AHCI Controller O vmhba0 Block SCSI 3 vmhba32 Block SCSI O vmhba33 Block SCSI O vmhba34 Block SCSI O vmhba35 Block SCSI O vmhba36 Block SCSI Patsburg 4 Port SATA Storage Control Unit O vmhba1 SCSI MT27500 Family ConnectX 3 3 vmhba mlx4 0 1 1 SCSI vmhba mlx4 0 2 1 SCSI Mellanox iSCSI over RDMA iSER Adapter O vmhba38 iSCSI ign 1998 01 com vmware vsa24 lab mtl com vmla O vmhba39 iSCSI ign 1998 01 com vmware vsa24 lab mtl com vmla Each port of the HCA is a vmnic in ESXi and each vmhba adapter can be bound to a single vmnic Clicking the adapter iSCSI alias will display to which vmnic it can be bound Mellanox iSCSI over RDMA iSER Adapter vmhba38 iSCSI iqn 1998 01 com O vmhba39 iSCSI iqn 1998 01 com Details vmhba39 Model Mellanox iSCSI over RDMA iSER Adapter iSCSI Name ign 1998 0 1 com vmware vsa24 lab mtl com vmlab r iSCSI Alias iser vmnic2 Connected Targets 2 Devices 3 Paths 4 Mellanox Technologies 10 Rev 1 9 10 0 Additional Driver Settings 4 4 5 Scanning for Targets Prior to scanning for targets verify a vSwitch is created with VMKernel for the vmnic needed 1 Choose the correct vmhba for the same vmni
4. Mellanox TE CHNOL OGIES Connect Accelerate Outperform Mellanox MLX4_EN Driver for VMware ESXi 5 1 and ESXi 5 5 User Manual Rev 1 9 10 0 www mellanox com Rev 1 9 10 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST OUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEOUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUME
5. NTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE AMA Mellanox Connect Accelerate Outperform Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 4 909 7200 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2012 Mellanox Technologies All rights reserved Mellanox amp Mellanox Logo BridgeX ConnectX amp CORE Direct InfiniBridge amp InfiniHost InfiniScale PhyX amp SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners Mellanox MLX4_EN Driver for VMware ESXi 5 1 and ESXi 5 5 User Manual 2 Mellanox Technologies Document Number 3527 Rev 1 9 10 0 Table of Contents Chapter I OVERVIEW ess s kceovien shee odes SKI enc SSA Kes KKK neeh Kennes caw 2 1 1 Main Features Overview sossun ne 2 Chapter 2 Driver Software comte ter breeds eran trede EAN HUS ANK 3 2 1 Installing and Running the offline bundle Driver on ESXi 5 1 and ESXi 5 5 3 2 2 Removing the offline bundle Driver eee een 4 2 3 Loading Unloading Driver Kernel Modules oen 4 2 4 Dri
6. c Go to Properties gt Network Configuration gt Add Click OK Wait several seconds for ESXi to refresh itself and go to Dynamic Discovery gt Add Enter the IP address of the target machine Click OK After several seconds the targets are shown in the Static Discovery Bi oe 0 DO 6 Close the window and choose Yes to refresh If the targets are not shown when clicking the vmhba adapter right click the vmhba adapter and rescan Mellanox iSCSI over RDMA iSER Adapter O vmhba38 iSCSI ign 1998 01 com vmware vsa24 lab mtl com vm ign 1998 01 com vmware vsa24 lab mtl com vm vmhba39 Model Mellanox iSCSI over RDMA iSER Adapter iSCSI Name ign 1998 0 1 com vmware vsa24 lab mtl com vmlab mtl com yok mtl com mtl com 1 iSCSI Alias iser vmnic2 Connected Targets 2 Devices 3 Paths 4 View Devices Paths Name Mellanox iSCSIRAID Ctlr t10 1234000000000000000000000000000000000000000000000000000000000 MellanoxiSCSI Disk t10 IET 000200010000000000000000000000000000000000000000 Mellanox iSCSI Disk t10 IET 000100010000000000000000000000000000000000000000 4 4 6 Enabling Flow Control in an Ethernet Switch Working with RoCE and Ethertnet switch requires enabling Flow Control on the ports Addition ally working with RoCE and using an Ethernet switch requires having Flow Control enabled in the switch 1 Connect to the switch using an SSH connection and enter the configuration mode en configure ter
7. ctX amp family based network adapter cards in a VMware ESXi 5 1 and ESXi 5 5 server environment The ConnectX amp family adapters identify on the PCI bus as 25408 25418 25448 26418 26428 26438 26448 26468 26478 26488 25400 27500 27510 27511 27520 27521 27530 27531 27540 27541 27550 27551 27560 27561 1 1 Main Features Overview Single Dual port MSI X Int X e NetOueue support e Multiple Tx Rx rings e Hardware Tx Rx checksum offload e Large Send Offload TCP Segmentation Offload e VLAN Tx Rx acceleration HW VLAN stripping insertion Ethtool support e NAPI support e Wake on Lan WoL only on supported hardware e NC SI e Auto moderation e Net dump Only for ESXi5 5Ux e RSS Queues e Fixed Pass Through e 10G e 40G Only for ESXi5 5Ux iSER e RoCE Mellanox Technologies 2 J Rev 1 9 10 0 Driver Software 2 2 1 3 Mellanox Technologies J Driver Software VMware uses a file package called a VIB VMware Installation Bundle as the mechanism for installing or upgrading software packages on an ESXi server MLX4 EN driver consists of several dependant kernel modules each with its own vib file In order to install the driver the VIBs need to be installed together For this MLX4_EN driver provides a bundle file a zip file that contain each module VIB file and metadata file that describes the dependencies between them The following steps describe how to download in
8. ement Assistant VMA from https my vmware com web vmware info slug datacenter cloud infrastructure vmware vsphere 5 5 2 Use the command vicfg vswitch from the VMA environment For additional documents please refer to the VMware site https www vmware com support pubs vsphere esxi vcenter server pubs html 4 3 3 Renaming the uplink Name The uplink naming format is in an increasing order e g when working in either Multifunction Mode or Flex10 vmnic0 to vmnic7 If the order is disrupted inconsistent and you wish to correct please follow the procedure below Log into the ESXi server with root permissions Open the vi etc vmware esx conf file Locate the device lt PCi device gt vmkname vmnicX Change the vmnic numbers to the desired order Save the file Reboot the server N OV ge 19 R 7 Two vmnics with the same number cannot exist in the same ESXi server ys Mellanox Technologies 8 J Rev 1 9 10 0 Additional Driver Settings 4 4 Configuring ESXi iSER 4 4 1 Installing the Driver Bundle 1 Activate the SSH in ESXi Configuration gt Security Profile gt Services Properties gt SSH gt Options gt Start 2 Copy the driver bundle using SCP into the machine 3 Install the MLNX OFED ESX driver gt esxcli software vib install d lt bundle file gt For example esxcli software vib install d MLNX OFED ESX 1 9 8 1 zip 4 Reboot the machine 5 Verify the ib iser module i
9. forms RSS for incoming UDP traffic e 1 enable e 0 disable default 0 use rx frags Uses RX frags e 1 enable e 0 disable default 0 5 Mellanox Technologies Rev 1 9 10 0 3 Firmware Programming 1 Download the bootable binary image md5sum e7b3e9357ca4045fabe2e8a95d951343 from the Mellanox Firmware Tools MFT site 2 Install the image according to the steps described in the README file 4 The following procedure reguires custom boot image downloading mounting and boot ing from a USB device Mellanox Technologies 6 J Rev 1 9 10 0 Additional Driver Settings 4 Additional Driver Settings 4 1 Changing the Driver s Module Parameters Default Values gt To change the module parameters default values 1 Ouery the available module parameters gt esxcli system module parameters list m lt module name gt 2 Set the driver with the required parameters gt esxcli system parameters set m lt module name gt p lt parameters list gt For example gt esxcli system module parameters set m mlx4 en p netg 1 udp rss 1 3 Verify that the parameters are set correctly gt esxcli system module parameters list m lt module name gt 4 Restart the driver module to apply new default kernel module parameters see Section 2 3 4 2 Disabling Enabling Automatic Load of the Driver upon Boot 1 Query the driver auto load status gt esxcli system module list 2 Disab
10. le auto load on boot gt esxcli system module set enabled false m lt module name gt For example gt esxcli system module set enabled false m mlx4 core 3 Enable auto load on boot gt esxcli system module set enabled true m lt module name gt For example gt esxcli system module set enabled true m mlx4 core 4 3 Adding the Device as an uplink to an Existing Vswitch using the CLI 4 3 1 Locally 1 Log into the ESXi server with root permissions 2 Find your device uplink_name under the name column gt esxcli network nic list 3 Add an uplink from a vSwitch gt esxcli network vswitch standard uplink add u lt uplink name gt v lt vswitch name gt 7 Mellanox Technologies J Rev 1 9 10 0 Po Once you add a device via the CLI it is visible in the vSphere client console thus d removing it can be performed via the UI 4 Check that the uplink was added successfully gt esxcli network vswitch standard list v lt vswitch_name gt gt To remove the device locally 1 Log into the ESXi server with root permissions 2 Remove an uplink from a vSwitch gt esxcli network vswitch standard uplink remove u lt uplink_name gt v lt vswitch name gt For additional documents please refer to the VMware site https pubs vmware com vsphere 50 index jsp topic 2Fcom vmware vcli ref doc 50 2Fesxcli network html 4 3 2 Remotely 1 Download and install VMware vSphere Manag
11. minal 2 Enable Flow Control for each port interface ethernet 1 22 shutdown flowcontrol receive on flowcontrol send on exit 11 Mellanox Technologies Rev 1 9 10 0 4 4 7 Using LUN as a VMFS Datastore 1 Adda storage Go to Configuration gt Storage gt Add Storage gt Disk LUN Choose LUN Choose the VMFS version Enter the datastore name Start using the VMFS Edit the VM and add a disk that is saved on the newly created datastore CAL G A DD or e Create migrate a VM to that datastore Mellanox Technologies 12
12. s loaded vmkload mod 1 grep ib iser gt To load the ib iser driver vmkload mod ib iser 4 4 2 Verifying the Mellanox Adapters are Recognized e vmnic ibX means the device port is in InfiniBand mode e vmnicx means the device port is in Ethernet mode EEA ETT Configuration METEN ESET ENEN T Network Adapters Device Speed Configured Health Status Intel Corporation 1350 Gigabit Network Connection Processors EH vmnici Down Negotiate Memory EH vmnic0 1000 Full Negotiate Storage Mellanox Technologies MT27500 Family ConnectX 3 Networking ERA vmnic ib1 56252 F Negotiate Storage Adapters EH vmnic ibo 56252 F Negotiate gt Network Adapters Advanced Settings Power Management 4 4 3 Changing the Port Mode to Ethernet RoCE 1 Change the Port Mode to Ethernet gt esxcli system parameters set m lt module name gt p lt port type gt For example esxcli system module parameters set m mlx4 core p port type array 2 2 2 Reboot the machine 3 Verify the network adapters are now set as vmnicx and not as vmnic_ibx 9 Mellanox Technologies Rev 1 9 10 0 4 4 4 AddingiSCSI Storage Adapter 1 Add iSCSI Storage Adapter Go to Configuration gt Storage Adapters gt Add gt OK 3 vmhba1 SCSI MT27500 Family ConnectX 3 3 vmhba mlx4 0 1 1 SCSI 2 vmhba mlx4 0 2 1 SCSI Add Storage Adapter x Add Software iSCSI Adapter c Ce ee T Targets 1 Devices
13. stall and run the driver Installing and Running the offline_bundle Driver on ESXi 5 1 and ESXi 5 5 1 Copy the offline bundle zip file to the ESXi 5 1 or ESXi 5 5 machine 2 Extract its contents 3 Install the driver a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version see Section 2 2 Removing the offline bundle Driver on page 4 b Install the MLX4 EN driver offline bundle package gt esxcli software vib install d lt path gt lt bundle file gt For example gt esxcli software vib install d lt path gt MLNX OFED ESX 1 9 10 0 10EM 550 0 0 1331820 zip c Reboot ESXi server The driver will be loaded automatically gt To verify that the driver is loaded gt esxcli system module list grep lt module name gt For example gt esxcli system module list grep mlx4 core gt To query network uplinks installed on your machine gt esxcli network nic list The number of uplinks claimed by MLX4 EN module should be displayed Rev 1 9 10 0 2 2 Removing the offline bundle Driver gt To remove the offline bundle driver package from the ESXi server machine gt esxcli software vib remove n net mlx4 ib esxcli software vib remove n scsi ib iser esxcli software vib remove n net rdma cm esxcli software vib remove n net ib addr esxcli software vib remove n net ib cm esxcli software vib remove n net ib umad
14. ver Default Values oss 4 Chapter 3 Firmware Programming 0 essen eee es eneen eee ene eee 6 Chapter 4 Additional Driver Settings es ese senen ee ee ene eee 7 4 1 Changing the Driver s Module Parameters Default Values 7 4 2 Disabling Enabling Automatic Load of the Driver upon Boot 7 4 3 Adding the Device as an uplink to an Existing Vswitch using the CLI 7 AST T0ca ly nr anar tus sig re ANTS EEI SEKIN STINE ER IK RR 7 432 Remotely 2 ee oetan ate deerde SESTA AE AIN SIENI oie Mena EE Ld 8 4 3 3 Renaming the uplink Name aaan en eneen eeen 8 4 4 Configuring ESXiiSER onee eee eee 9 4 4 1 Installing the Driver Bundle 0 0 cect tte 9 4 4 2 Verifying the Mellanox Adapters are Recognized oon enen 9 4 4 3 Changing the Port Mode to Ethernet ROCE 00 00 cece ee eee 9 4 4 4 Adding iSCSI Storage Adapter oa oen eneen 10 4 4 5 Scanning for Targets onee eeen 11 4 4 6 Enabling Flow Control in an Ethernet Switch aaan 11 4 4 7 Using LUN asa VMFS Datastore oossoo on eneen 12 Mellanox Technologies 3 Rev 1 9 10 0 List of Tables Table 1 mlx4 core Module Parameters oo oee eee eee 4 Table 2 mlx4 en Module Parameters oaeen eee ee 5 1 Mellanox Technologies Rev 1 9 10 0 1 Overview This document provides instructions for installing the MLX4 EN drivers for Mellanox Technol ogies Conne
Download Pdf Manuals
Related Search