Mellanox OFED Linux User's Manual
Contents
1. Abbreviation Acronym Whole Word Description B Capital B is used to indicate size in bytes or multiples of bytes e g IKB 1024 bytes and 1 1048576 bytes b Small b is used to indicate size in bits or multiples of bits e g Kb 1024 bits FW Firmware HCA Host Channel Adapter HW Hardware IB InfiniBand LSB Least significant byte Isb Least significant bit MSB Most significant byte msb Most significant bit NIC Network Interface Card SW Software IPoIB IP over InfiniBand PFC Priority Flow Control PR Path Record SL Service Level SRP SCSI RDMA Protocol Mellanox Techologies 7 Rev 1 8 0 Table 3 Abbreviations and Acronyms Sheet 2 of 2 Abbreviation Acronym Whole Word Description ULP Upper Level Protocol VL Virtual Lanes 8 Mellanox Techologies Rev 1 8 0 Glossary The following is a list of concepts and terms related to InfiniBand in general and to Subnet Manag ers in particular It is included here for ease of reference but the main reference remains the nfini Band Architecture Specification Table 4 Glossary Sheet 1 of 2 Channel Adapter An IB device that terminates an IB link and executes transport CA Host Channel functions This may be an HCA Host CA or a TCA Target Adapter HCA CA HCA Card A network adapter card based on an InfiniBand channel adapter device IB Devic2. Itis the administrator s responsibility to make sure that all the machines in the network support and work with the same MTU value For operating systems other than VMware ESXi Server the default value is set to 2044 bytes 18 Mellanox Techologies Rev 1 8 0 The procedure for changing the MTU may vary depending on the OS For example to change it to 1500 bytes On Linux if the IPoIB interface is named ib0 run ifconfig ibO mtu 1500 On Microsoft Windows execute the following steps a Open Network Connections b Select the IPoIB adapter and right click on it Select Properties a 9 Press Configure and then go to the Advanced tab Select the payload MTU size and change it to 1500 Make sure that the firmware of HCAs and the switches supports MTU you wish to set g Configure your Subnet Manager SM to set the MTU value in the configuration file The SM configuration for MTU value is per Partition Key PKey For example to enable 4K MTUs on a default PKey using the OpenSM SM6 log into the Linux machine running OpenSM and perform the following commands h Edit the file usr local ofed etc opensm partitions conf and include the line key0 0x7fff ipoib mtu 5 ALL full i Restart OpenSM etc init d opensmd restart To enable 4k mtu support run esxcli system module parameters set A m mlx4_core p mtu_4k 1 n Changes will take effect after the reboot 5 2
3. 4 Driver Features 4 1 IP over InfiniBand 4 1 1 IPoIB Overview The IP over IB IPoIB driver is a network interface implementation over InfiniBand IPoIB encapsulates Datagram transport service The IPoIB driver ib ipoib exploits the following Con nect X ConnectX 2 ConnectX 3 capabilities Uses any CX IB ports one or two Inserts IP UDP TCP checksum on outgoing packets e Calculates checksum on received packets Support net device TSO through CX LSO capability to defragment large datagrams to MTU quantas Unreliable Datagram IPoIB also supports the following software based enhancements Large Receive Offload Ethtool support 4 1 2 IPolB Configuration 1 Install the Mellanox OFED driver for VMware 2 Verify the driver VIBs are installed correctly Run esxcli software vib list grep i Mellanox a k a esxcfg nics 1 3 Verify the uplinks state is up Run esxcli network nic list grep i Mellanox a k a esxcfg nics 1 See your VMware distribution documentation for additional information about configuring IP addresses Mellanox Techologies 15 Rev 1 8 0 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere 5 5 1 5 2 5 2 1 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere Configuring an Uplink To configure an Uplink 1 Add the device as an uplink to an Existing vSwitch using the CLI a Log into the ESXi server with root perm
4. TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies Inc All Rights Reserved Mellanox BridgeX ConnectX CORE Direct amp InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale InfiniPCIG PhyX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT and SwitchX are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3464 Rev 1 8 0 Table of Contents Tabl of Contents irre reed oh nt me dye on n ee Reo d je Fee ib Sica ne ayers e Listof oie per e rte nere ar esi a ri prre Rr da P PS Revision History eer xU e hs aea Pe ee aia era aoa About this Manual ccc cece cece e eee e eee e cree eee nn n n n nn nnn sf Intended Audience sxsw bag vas vasa tale sas tents boas fee 6 Typographical Conventions 0 eee eect m e 6 Common Abbreviations and Acronyms 9 7 GlOSSaby seco sesh Lae aes Cee aA ORR du et d ce SS 9 Related Documentation sss das cae eas ey Mo
5. with major operating system distributions 1 2 Introduction to Mellanox InfiniBand Adapters Mellanox InfiniBand IB adapters which are based on Mellanox ConnectX ConnectX 2 and ConnectX 3 adapter devices provide leading server and storage I O performance with flexibility to support the myriad of communication protocols and network fabrics over a single device with out sacrificing functionality when consolidating I O For example IB enabled adapters can sup port Connectivity to 10 20 40 and 56Gb s InfiniBand switches A unified application programming interface with access to communication protocols including Networking TCP IP UDP sockets Storage NFS CIFS 1SCSI SRP and Clustered Storage Clustering MPI DAPL RDS sockets and Management SNMP SMI S Communication protocol acceleration engines including networking storage cluster ing virtualization and RDMA with enhanced quality of service 1 3 Mellanox OFED Package 1 3 1 Software Components MLNX OFED VMware contains the following software components CPU architectures x86 64 ESXi Hypervisor ESXi5 0 with BUILD ID 469512 5 15 0 UI with BUILD ID 623860 Firmware versions ConnectX 2 2 9 1000 and above ConnectX 3 2 10 0700 and above Mellanox Techologies 11 Rev 1 8 0 Mellanox InfiniBand OFED Driver for VMware vSphere Overview 14 mlx4 InfiniBand Driver MLNX OFED ESXi package contains e MLNX OFED ESX
6. 5 High Availability High Availability is supported for both InfiniBand network and storage adapters A failover port can be located on the same HCA card or on a different HCA card on the same system for hard ware redundancy To define a failover policy for InfiniBand networking and or storage follow the instructions in the ESXi Server Configuration Guide document Mellanox Techologies 19
7. Mellanox TECHNOLOGIES Mellanox InfiniBand OFED Driver for VMware vSphere 5 X User Manual Rev 1 8 0 www mellanox com THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox
8. ai e cay REG UR wae vale vas 10 Support and Updates Webpage s 10 Chapter 1 Mellanox InfiniBand OFED Driver for VMware vSphere Overview 11 1 1 Introduction to Mellanox InfiniBand OFED for VMware 11 1 2 Introduction to Mellanox InfiniBand Adapters 11 13 Mellanox OFED Package 11 131 Software Componetits esee Mie Ae 11 1 4 mlx4 InfiniBand 12 LAN a pedea arly gered area Se hdd Seated ddan dees ely wee 12 1 5 Supported Hardware Compatibility 12 Chapter 2 Installing Mellanox InfiniBand OFED Driver for VMware vSphere 13 Chapter 3 Uninstalling Mellanox InfiniBand OFED Driver 14 Chapter 4 Driver Features cece ccc cece reece mnn nn nnn nns 15 4 1 IP over InfiniBand 15 4l l IPolBiOverview aia tock as oe 15 4 L 2 TRolB Configuration Say 1 04ns CeECRYTRE AU AL E ito indu 15 Chapter 5 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere 16 5 1 Configuring an Uplin
9. bnet The maximum number of Partition Keys available on Mellanox HCAs is 128 for ConnectX IB family Check with IB switch documentation for the number of supported partition keys Guest Operating System Configuration For mode VLANs need to be configured in the installed guest operating system This proce dure may vary for different operating systems See your guest operating system manual on VLAN configuration In addition for each new interface created within the virtual machine at least one packet should be transmitted For example Create a new interface e g lt eth1 gt with IP address lt ip1 gt To guarantee that a packet is transmitted from the new interface run arping I lt ethl gt lt ipl gt c 1 Virtual Switch Configuration For VST mode the virtual switch using an InfiniBand uplink needs to be configured For further information see the ESXi Server 3 Configuration Guide and ESXi Server 3 802 10 VLAN Solutions documents 5 2 4 Maximum Transmit Unit MTU Configuration On VMware ESXi Server machines the MTU is set to 1500 bytes by default IPoIB supports larger values and allows Jumbo Frames JF traffic up to 4052 bytes on VI3 and 4092 bytes on vSphere 4 The maximum value of JF supported by the InfiniBand device is 2044 bytes for the InfiniHost family 4052 4092 bytes for ConnectX IB family vSphere 4 P Running a datagram IPoIB MTU of 4092 requires that the InfiniBand MTU is set to 4k
10. es Integrated circuit implementing InfiniBand compliant commu nication IB Cluster Fabric Subnet A set of IB devices connected by IB cables In Band A term assigned to administration activities traversing the IB connectivity only LID An address assigned to a port data sink or source point by the Subnet Manager unique within the subnet used for directing packets within the subnet Local Device Node The IB Host Channel Adapter HCA Card installed on the System machine running IBDIAG tools Local Port The IB port of the HCA through which IBDIAG tools connect to the IB fabric Master Subnet Man The Subnet Manager that is authoritative that has the refer ager ence configuration information for the subnet See Subnet Manager Multicast Forward ing Tables A table that exists in every switch providing the list of ports to forward received multicast packet The table 1s organized by MLID Network Interface A network adapter card that plugs into the PCI Express slot Card NIC and provides one or more ports to an Ethernet network Standby Subnet Man A Subnet Manager that is currently quiescent and not in the ager role of a Master Subnet Manager by agency of the master SM See Subnet Manager Subnet Administra tor SA An application normally part of the Subnet Manager that implements the interface for querying and manipulating subnet management data Mellan
11. i 1 6 8 zip Hypervisor bundle which contains the following ker nel modules 1 4 1 ULPs IPoIB mlx4 core ConnectX family low level PCI driver mlx4 ib ConnectX family InfiniBand driver 10 core ib sa ib mad ib umad Ib ipoib The IP over IB IPoIB driver is a network interface implementation over InfiniBand IPoIB encapsulates IP datagrams over an InfiniBand connected or datagram transport service IPoIB pre appends the IP datagrams with an encapsulation header and sends the outcome over the Infini Band transport service The interface supports unicast multicast and broadcast For details see Chapter 4 1 IP over InfiniBand P On VMware ESXi Server IPoIB supports Unreliable Datagram UD mode only note that Reliable Connected RC mode is not supported 1 5 Supported Hardware Compatibility For the supported hardware compatibility list HCL please refer to http communities vmware com cshwsw jspa vendor mellanox 12 Mellanox Techologies Rev 1 8 0 2 Installing Mellanox InfiniBand OFED Driver for VMware vSphere This chapter describes how to install and test the Mellanox InfiniBand OFED Driver for VMware vSphere package on a single host machine with Mellanox InfiniBand hardware installed The InfiniBand OFED driver installation on VMware ESXi Server 5 0 is done using VMware s VIB bundles d Please uninstall any previous Mellanox driver packages prior to installing the new ver sion T
12. issions b Add an uplink to a vSwitch gt esxcli network vSwitch standard uplink add lt uplink name gt v lt vswitch name gt 2 Verify the uplink is added successfully gt esxcli network vSwitch standard list vswitch name Toremove the device locally 1 Log into the ESXi server with root permissions 2 Remove an uplink from a vSwitch gt esxcli network vSwitch standard uplink remove lt uplink name gt v lt vswitch name gt For further information please refer to http pubs vmware com vsphere 50 topic com vmware vcli migration doc_50 cos upgrade technote 1 9 html 1024629 Configuring VMware ESXi Server Settings VMware ESXi Server settings can be configured using the vSphere Client Once the InfiniBand OFED driver is installed and configured the administrator can make use of InfiniBand software available on the VMware ESXi Server machine The InfiniBand package provides networking and storage over InfiniBand The following sub sections describe their configuration This section includes instructions for configuring various module parameters From ESXi 5 0 use esxcli system module parameters list m module name for viewing all the available module parameters and default settings When using ESXi use or remote CLI vicfg module pl to configure the module parameters in a similar way to what is done in the Service Console COS for ESXi Subnet Manager The driver package requires InfiniBand Subne
13. k 16 5 22 Configuring VMware ESXi Server Settings 16 32 p Subnet Manager is ates la ie opere dss Pc Pas PHP Ng 16 5 22 SNetwotkitig ee Lee ULL te E 17 5 2 3 Virtual Local Area Network VLAN Support sssleeeeeeeeeeA 17 5 24 Maximum Transmit Unit MTU Configuration esee eee 18 5 2 5 High Availability ersin 544 Ag eG 19 Mellanox Techologies 3 List of Tables RevisiomHistory ete a edad ane MERCADO RH e EE 5 Table 2 Typographical Conventions 0 ccc cece teen n en enn 6 Table 3 Abbreviations and Acronyms 7 Tabl 4 Glossary ei eR Lee beter cd aed e re LE C arden is d Eo ERR dca d 9 Table 5 Reference Documents 00 ccc een hh 10 4 Mellanox Techologies Rev 1 8 0 Revision History Table 1 Revision History Revision Date Change Description 1 6 8 June 2012 1 4 1 2 0 000 May 2011 Initial release Mellanox Techologies 5 Rev 1 8 0 About this Manual This document provides instructions for installing and using drivers for Mellanox Technologies ConnectX 2 and ConnectX 3 based network adapter cards in a VMware ESXi Server environ ment Intended Audience This manual is intended for system administ
14. o install the driver 1 Log into the 15 0 server with root permissions gt esxcli software vib install d bundle file gt 2 Reboot the machine 3 Verify the driver was installed successfully gt esxcli software vib list grep Mellanox net ib core 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net ib ipoib 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net ib mad 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net ib sa 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net ib umad 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net mlx4 core 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 net mlx4 ib 1 6 8 0EM 500 0 0 472560 Mellanox PartnerSupported 2012 05 09 P After the installation process all kernel modules are loaded automatically upon boot Ad Mellanox Techologies 13 J Uninstalling Mellanox InfiniBand OFED Driver 3 Uninstalling Mellanox InfiniBand OFED Driver To uninstall the driver 1 Log into the ESX15 0 server with root permissions esxcli software vib remove n net ib ipoib esxcli software vib remove n net mlx4 ib esxcli software vib remove n net ib umad esxcli software vib remove n net ib sa esxcli software vib remove n net ib mad esxcli software vib remove n net ib core esxcli software vib remove n net mlx4 core NU NP 2 Reboot the server 14 Mellanox Techologies Rev 1 8 0
15. ox Techologies 9 Rev 1 8 0 Table 4 Glossary Sheet 2 of 2 Subnet Manager SM One of several entities involved in the configuration and con trol of the subnet Unicast Linear For A table that exists in every switch providing the port through warding Tables LFT which packets should be sent to each LID Related Documentation Table 5 Reference Documents Document Name Description MFT User s Manual Mellanox Firmware Tools User s Manual See under docs folder of installed package MFT Release Notes Release Notes for the Mellanox Firmware Tools See under docs folder of installed package Support and Updates Webpage Please visit http www mellanox com gt Products gt Adapter IB VPI SW VMware Drivers for down loads FAQ troubleshooting future updates to this manual etc 10 Mellanox Techologies 1 8 0 1 Mellanox InfiniBand OFED Driver for VMware vSphere Overview 1 4 Introduction to Mellanox InfiniBand OFED for VMware Mellanox OFED is a single Virtual Protocol Internconnect VPI software stack based on the OpenFabrics OFED Linux stack adapted for VMware and operates across all Mellanox network adapter solutions supporting 10 20 and 40Gb s InfiniBand IB and 2 5 or 5 0 GT s PCI Express 2 0 and 3 0 uplinks to servers All Mellanox network adapter cards are compatible with OpenFabrics based RDMA protocols and software and are supported
16. ported by InfiniBand 4 port uplinks e g VMware amp VMotionTM NIC teaming and High Availability and their setting is performed transparently 5 2 3 Virtual Local Area Network VLAN Support To support VLAN for VMware ESXi Server users one of the elements on the virtual or physical network must tag the Ethernet frames with an 802 1Q tag There are three different configuration modes to tag and untag the frames as virtual machine frames 1 Virtual Machine Guest Tagging Mode 2 ESXi Server Virtual Switch Tagging VST Mode 3 External Switch Tagging EST Mode P EST is supported for Ethernet switches and can be used beyond IB Eth Gateways trans parently to VMware ESXi Servers within the InfiniBand subnet To configure VLAN for InfiniBand networking the following entities may need to be configured according to the mode you intend to use Mellanox Techologies 17 Rev 1 8 0 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere Subnet Manager Configuration Ethernet VLANs are implemented on InfiniBand using Partition Keys See RFC 4392 for informa tion Thus the InfiniBand network must be configured first This can be done by configuring the Subnet Manager SM on your subnet Note that this configuration is needed for both VLAN config uration modes VGT and VST For further information on the InfiniBand Partition Keys configuration for IPoIB see the Subnet Manager manual installed in your su
17. rators responsible for the installation configuration management and maintenance of the software and hardware of InfiniBand adapter cards It is also intended for application developers Typographical Conventions Table 2 Typographical Conventions Description Convention Example File names file extension Directory names directory Commands and their parameters command 1 Optional items Mutually exclusive parameters p1 p2 p3 Optional mutually exclusive parame ters 1 p2 p3 Prompt of a user command under bash shell hostnames Prompt of a root command under bash hostnam shell Prompt of a user command under tesh tcsh shell Environment variables VARIABLE Code example if a b Comment at the beginning of a code line n Characters to be typed by users as is bold font Keywords bold font 6 Mellanox Techologies Rev 1 8 0 Table 2 Typographical Conventions Description Convention Example Variables for which users supply spe Italic font cific values Emphasized words Italic font These are emphasized words Pop up menu sequences menul gt menu2 gt gt item Note Note Warning Warning Common Abbreviations and Acronyms Table 3 Abbreviations and Acronyms Sheet 1 of 2
18. t Manager SM to run on the subnet The driver package does not include an SM 16 Mellanox Techologies Rev 1 8 0 If your fabric includes a managed switch gateway please refer to the vendor s user s guide to acti vate the built in SM If your fabric does not include a managed switch gateway an SM application should be installed on at least one non ESXi Server machine in the subnet You can download an InfiniBand SM such as OpenSM from www openfabrics org under the Downloads section 5 2 2 Networking The InfiniBand package includes a networking module called IPoIB which causes each Infini Band port on the VMware ESXi Server machine to be exposed as one or more physical network adapters also referred to as uplinks or vmnics To verify that all supported InfiniBand ports on the host are recognized and up perform the following steps 1 Connect to the VMware ESXi Server machine using the interface of VMware vSphere Cli ent 2 Select the Configuration tab 3 Click the Network Adapters entry which appears in the Hardware list A Network Adapters list is displayed describing per uplink the Device it resides on the port Speed the port Configured state and the vSwitch name it connects to To create and configure virtual network adapters connected to InfiniBand uplinks follow the instructions in the ESXi Server Configuration Guide document features supported by Ethernet adapter uplinks are also sup
Download Pdf Manuals
Related Search