Home

Mellanox OFED for Linux Release Notes

1. Index Issue Description Workaround 59 The known PCI BDFs for all VFs in kernel com mand line should be specified by adding xen pciback hide For further information please refer to http wiki xen org wiki Xen_PCI_Passthrough 60 mlx4 core module parameter port type array cannot specify 3 auto as a port type value because of a mistake in the modinfo mlx4 core description of this parameter 61 Port Type Manage OpenSM must be stopped prior to changing ment port protocol from InfiniBand to Ethernet 62 After changing port type using Use udev rules for persis connectx port config interface ports names tent naming configuration can be changed For example ib1 gt 160 if port For further information changed to be Ethernet port and port2 left IB please refer to the User Manual 63 Flow Steering Flow Steering is disabled by default To enable it set the parame ter below as follow log num mgm entry siz e should set to 1 64 IPv4 rule with source IP cannot be created in SLES 11 65 RFS is not supported in SLES11 66 Quality of Service QosS is not supported in XenServer Debian 6 0 and 6 2 with uek kernel 67 Driver Uninstall A Kernel panic occurs if you uninstall the driver Remove the midule after without deleting the SR IOV module params uninstalling and prior to mlx4 core s num vfs inthe file etc mod restarti
2. Default Oxffff ipoib mtu 5 ALL full Where mtu 5 indicates that all IPoIB ports in the fabric are using 4k MTU mtu 4 indicates 2k MTU 12 Occasionally when using IPoIB in Connected Use the Datagram mode mode the connection might get closed and recov ered only after several minutes 13 Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring https bugs openfabrics org show_bug cgi id 1990 14 Whenever the IOMMU parameter is enabled in To avoid such issue the kernel it can decrease the number of child Decrease the amount of the interfaces on the device according to resource RX receive buffers module limitation parameter the default is The driver will stuck after unknown amount of 512 Decrease the number of RX rings sys fs or ethtool in new kernels child interfaces creation For further information please see Avoid using IOMMU if not https access redhat com site articles 66747 required http support citrix com article CTX 136517 http www novell com support kb For KVM users doc php id 70 12337 Run echo 1 gt sys module kvm parameters allow_unsafe_assigned _interrupts To make this change persist across reboots add the fol lowing to the etc mod probe d kvm conf file or create this file if it does not exist options kvm allow_unsafe_assigned _interrupts 1 kernel parameters Mellanox Technologies 15
3. 6 Table2 Supported Platforms and Operating Systems 7 Table3 Additional Software Packages 8 Table4 MLNX OFED Rev 2 2 1 0 1 Compatibility MatriX 9 Table 5 New Features Changes and Fixes in 2 2 1 0 1 10 Table 6 New Features Changes and Fixes in 2 1 1 0 6 10 Table 7 New Features Changes and Fixes in 2 1 1 0 0 11 Table 8 New Features Changes and Fixes in 2 0 3 0 0 12 Table 9 Known Issues x gee yen hea ae E e e eie ra ene aree o Rev Saw a e D en 14 Table 10 API Changes in MLNX OFED Rev 2 2 1 0 1 26 Table 11 Changes in MLNX OFED Rev 2 1 1 0 0 27 Table 12 Changes in MLNX OFED Rev 2 0 3 0 0 27 Table 13 Changes in MLNX_OFED Rev 2 0 2 0 5 28 Table 14 Fixed Bugs Eist had ed eat E wea eine LA PL QUE RS NS 29 4 Mellanox Technologies J Rev 2 2 1 0 1 1 Overview These are the release notes of Mellanox OFED for Linux Driver Rev 2 2 1 0 1 Mellanox OFED Is a single Virtual Protocol Interconnect VPI software stack and operates across all Mellanox network adapter solutions supporting the following uplinks to
4. Added non virtual environment support Hypervisor support KVM and XenServer Mellanox Technologies 11 J I Rev 2 2 1 0 1 Change Log History 84 Changes Rev 2 0 3 0 0 From Rev 2 0 2 0 5 Table 8 Features Changes v2 0 3 0 0 Category Description Operating Systems Additional OS support SLES11SP3 Fedoral6 Fedoral7 Drivers Added Connect IB support Installation Added ability to install MLNX_OFED with SR IOV support Added Yum installation support EoIB EoIB at beta level is supported only in SLESIISP2 and RHEL6 4 mlx4 core Modified module parameters to associate configuration values with specific PCI devices identified by their bus device function value format mlx4 en Reusing DMA mapped buffers major performance improvements when IOMMU 1s enabled Added Port level QoS support IPoIB Reduced memory consumption Limited the number TX and RX queues to 16 Default IPoIB mode is set to work in Datagram except for Connect IBTM adapter card which uses IPoIB with Connected mode as default Storage ISER at GA level 8 5 Features in Rev 2 0 2 0 5 SR IOV for both Ethernet and InfiniBand at Beta level RoCE over SR IOV at Beta level el PoIB to enable IPoIB in a Para Virtualized environment at Alpha level Contiguous pages Internal memory allocation improvements Register shared
5. InfiniBand interface Open MPI stack 1 6 5 and later supporting the InfiniBand interface MPI benchmark tests OSU benchmarks Intel MPI benchmarks Presta PGAS ScalableSHMEM v2 2 supporting InfiniBand MXM and FCA ScalableUPC v2 2 supporting InfiniBand MXM and FCA HPC Acceleration packages Mellanox MXM v3 0 p2p transport library acceleration over Infiniband Mellanox FCA v2 5 MPI PGAS collective operations acceleration library over InfiniBand KNEM Linux kernel module enabling high performance intra node MPI PGAS communication for large messages Extra packages 1000152 ibdump MFT Sources of all software modules under conditions mentioned in the modules LICENSE files except for MFT OpenSM plugins ibutils2 and ibdump Documentation 6 Mellanox Technologies J Rev 2 2 1 0 1 4 Supported Platforms and Operating Systems The following are the supported OSs in MLNX_OFED Rev 2 2 1 0 1 Table 2 Supported Platforms and Operating Systems Operating System Platform RHEL CentOS 6 3 x86 64 RHEL CentOS 6 4 x86 64 PPC RHEL CentOS 6 5 x86 64 PPC RHEL CentOS 7 0 x86 64 SLES11 SPI x86 64 SLES11 SP2 x86 64 PPC SLES11 SP3 x86 64 PPC OEL 6 3 x86 64 OEL 6 4 x86 64 OEL 6 5 x86 64 Citrix XenServer Host 6 2 1686 Fedora 19 x86 64 Ubuntu 12 04 x86 64 Ubuntu 13 04 x86 64 Ubuntu 14 04 x86 64 Debian 6 0
6. J Rev 2 2 1 0 1 Known Issues Table 9 Known Issues Continued Index Issue Description Workaround 15 System might crash in skb_checksum_help Use UD mode in ipoib while performing TCP retransmit involving pack ets with 64k packet size A similar out to the below will be printed kernel BUG at net core dev c 1707 invalid opcode 0000 1 SMP RIP 0010 lt ffffffff81448988 skb_checksum_help 0x148 0x160 Call Trace lt IRQ gt ffffff8144883835 dev hard start xmit4 0x3e3 0x530 ffffff8144Cc8055 dev queue 0 205 0 550 fffffff8145247d 5 neigh connected output 0xbd 0x1 16 Changing the GUID of a specific SR IOV guest after the driver has been started causes the ping to fail Hence no traffic can go over that Infini Band interface 17 send queue size over Connect IBTM adapter cards cannot be larger than 1024 18 Ethernet Ethernet PV VLAN Guest transparent Tagging VGT is only supported in openvswitch and not in standard Linux vBridges and libvirt For more information please see http lib virt org formatnetwork html Setting VLAN tag section 19 Changing the ring size on 32 bit system may result in failure due to lack of memory Therefore mlx4 en will not be able to vmap enough mem ory and the below message will be printed in dmesg vmap allocation for size 528384 failed use vmalloc size to increase size In this
7. ibv exp create_dct struct context context struct exp dct init attr attr e int exp destroy dct struct dct dct e int ibv exp query dct struct ibv dct dct struct ibv exp dct attr attr Verbs Extension API jbv post task Verbs extension API defines OFA ibv query values ex APIs extension scheme to detect ABI 16 query device ex compatibility and enable backward ibv create flow ALS ibv destroy flow and forward compatibility support DV Y P y SUPP ibv_poll_cq ex ibv_reg_shared mr ex ibv_open xrcd ibv_close xrcd ibv_modify_cq ibv_create srq ex ibv_get_srq num ibv_create_qp_ex ibv_create_cq_ex ibv_open_qp ibv_modify_qp_ex Verbs Experimental ibv_exp create qp Verbs experimental API defines ibv exp query device MLNX OFED APIs extension ibv exp create dct ibv exp destroy dct ibv exp query dct scheme which is similar to the Verbs extension This extension pro vides a way to introduce new features before they are integrated into the for mal OFA API and to the upstream ker nel and libs 10 3 API Changes in MLNX OFED Rev 2 0 3 0 0 The following are the API changes in MLNX OFED Rev 2 0 3 0 0 Table 12 API Changes in MLNX OFED Rev 2 0 3 0 0 Name Description XRC The following verbs have become deprecated struct ibv xrc domain ibv open xrc domain Struct ibv srq ibv create xrc
8. memory Control objects QPs CQs Ethernet Performance Enhancements NUMA related and others for 10G and 40G OFED VMA integration to a single branch Ethernet Time Stamping at Beta level Flow Steering for Ethernet and InfiniBand at Beta level Raw Eth QPs Checksum TX RX 1 SR IOV Ethernet Time Stamping and Flow Steering are ConnectX 3 HCA capability 12 Mellanox Technologies Rev 2 2 1 0 1 Flow Steering Errata Kernel upgrade support YUM update support Storage iSER at Beta level and SRP 64bit wide counters port xmit recv data packets unicast mcast VERSION query API library and headers Mellanox Technologies 13 J Rev 2 2 1 0 1 Known Issues 9 Known Issues The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release Table 9 Known Issues Index Issue Description Workaround 1 IPoIB When user increases receive send a buffer it might consume all the memory when few child s interfaces are created 2 The hardware address suffix of IPoIB interfaces in MLNX OFED v2 0 3 0 0 is a instead of 8 to indicate the TSS support 3 The size of send queue in Connect IB cards cannot exceed 1K 4 In 32 bit devices the maximum number of child interfaces that can be created is 16 Creating more that might cause out of memory issues 5 The default I
9. mlx4 en Fixed ALB bonding mode failure when enslaving 2 0 3 0 0 2 1 1 0 0 Mellanox interfaces 15 Fixed leak of mapped memory 2 0 3 0 0 2 1 1 0 0 16 Fixed TX timeout in Ethernet driver 2 0 2 0 5 2 0 3 0 0 17 Fixed ethtool stats report for Virtual Functions 2 0 2 0 5 2 0 3 0 0 18 Fixedan issue of VLAN traffic over Virtual Machine 2 0 2 0 5 2 0 3 0 0 in paravirtualized mode 19 Fixed ethtool operation crash while interface down 2 0 2 0 5 2 0 3 0 0 20 IPoIB Fixed memory leak in Connected mode 2 0 2 0 5 2 0 3 0 0 21 Fixed an issue causing IPoIB to avoid pkey value 0 2 0 2 0 5 2 0 3 0 0 for child interfaces Mellanox Technologies 29 J
10. servers 10 20 40 and 56 Gb s InfiniBand IB 10 40 and 56 Gb s Ethernet 2 5 or 5 0 GT s PCI Express 2 0 8GT s PCI Express 3 0 2 Main Features in This Release MLNX_OFED Rev 2 2 1 0 1 provides the following new features Signature Verbs T10 PI Connect IB DC support Single port Virtual Function Multi GUID support for InfiniBand in SR IOV e MSI X vector allocation for the SR IOV hypervisor and guest Ethernet VXLAN Hardware Stateless offload ConnectX 3 Pro ONLY Ethernet Power Management Quality of Service Ethernet PTP Hardware Clock support query operations for privileged Virtual Functions OVS support for eIPoIB Support for up to 126 Virtual Functions pending OS and Server support 1 56 GbE is a Mellanox propriety link speed and can be achieved while connected to Mellanox SX10XX switch series Mellanox Technologies 5 J Rev 2 2 1 0 1 Content of Mellanox OFED for Linux 3 Content of Mellanox OFED for Linux Mellanox OFED for Linux software contains the following components Table 1 Mellanox OFED for Linux Software Components Components Description OpenFabrics core and ULPs IB HCA drivers mlx4 mlx5 core Upper Layer Protocols IPoIB SRP and ISER Initiator OpenFabrics utilities OpenSM IB Subnet Manager with Mellanox proprietary Adaptive Routing Diagnostic tools Performance tests MPI e OSU MPI mvapich2 1 9 1 stack supporting the
11. single ported on port 2 RoCE connections with will fail in a VM When detaching a VF without shutting down the driver from a VM and reattaching it to another VM with the same IP address for the Mellanox NIC RoCE connections will fail Shut down the driver in the VM before detaching the VF Enabling SR IOV requires appending the intel iommu on option to the relevant OS in file boot grub grub conf Without that SR IOV cannot be loaded rdma cm does not support UD QPs SR IOV can be enabled only when using the firmware version embedded in the MLNX OFED v2 0 3 0 0 and above On various combinations of Hypervisor OSes and Guest OSes an issue might occur when attaching detaching VFs to a guest while that guest is up and running Attach detach VFs to from a VM only while that VM 15 down When SR IOV is disabled in the system BIOS a PCI issue is noticed in Ubuntu v12 04 3 with Linux kernel v3 8 which affects NICs of several manufacturers including Mellanox s preventing them from operating Enable SR IOV in the BIOS When working with SR IOV in Xen 4 2 virtual ization platform only the built in xen pciback driver should be loaded The xen pciback module in dom0 should not be loaded as loading them simultaneously may cause interrupts loss and cause the driver to enter the reset flow 20 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued
12. srq int ibv close xrc domain int ibv create xrc rcv qp int ibv modify xrc rcv qp int ibv query xrc rcv qp int reg rcv int ibv unreg xrc rcv qp Mellanox Technologies 27 J Rev 2 2 1 0 1 API Changes 10 4 Changes MLNX_OFED Rev 2 0 2 0 5 The following the changes in MLNX_OFED v2 0 2 0 5 Table 13 Changes MLNX_OFED Rev 2 0 2 0 5 Name Description Libibverbs Extended speeds Missingtheext active speed attribute from the struct ibv port attr Removed function ext rate to int Added functions ibv rate to mbpsandmbps to ibv rate Raw QPs QP types IBV_QPT RAW PACKET and IBV RAW ETH are not sup ported Contiguous pages Added Contiguous pages support Addedfunctionibv reg shared mr Libmverbs The enumeration IBV M WR CALC was renamed to IBV M WR CALC SEND Theenumeration IBV M WR WRITE WITH IMM was added Inthe structure m send wr the union wr send was renamed to wr calc sendand wr rdma was added The enumerations IBV WOE CAP CALC RDMA WRITE WITH IMM was added following enumerations were renamed From IBV M WQE SQ ENABLE CAP to IBV M WOE CAP SQ ENABLE From IBV M WQE RQ ENABLE CAP to IBV M WOE CAP RO ENABLE e From IBV M WQE WAIT CAP to IBV M WOE CAP COE WAIT e From IBV CALC CAP to IBV CAP CA
13. 7 x86 64 Debian 7 1 x86 64 Debian 7 2 x86 64 kernel 3 10 28 kernel 3 11 10 kernel 3 12 9 kernel 3 13 1 a This kernel is supported only when using the Operating Systems stated in the table above If you wish to install OFED on a different kernel you need to create a new ISO image 7 using mlnx add kernel support sh script See the MLNX OFED User Guide for instructions Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest version as well A 41 Supported Hypervisors The following are the supported hypervisors in MLNX_OFED Rev 2 2 1 0 1 KVM Xen Mellanox Technologies 7 J Rev 2 2 1 0 1 Hardware and Software Requirements 5 Hardware Software Requirements The following are the hardware and software requirements of MLNX_OFED Rev 2 2 1 0 1 Linux operating system Administrator privileges on your machine s Disk Space 1GB For the OFED Distribution to compile on your machine some software packages of your operat ing system OS distribution are required To install the additional packages run the following commands per OS Table 3 Additional Software Packages Operating System Required Packages Installation Command RHEL OEL yum install pciutils python gcc gfortran libxml2 python tesh libnl i686 libnl Fedora expat glib2 tcl libstdc bc tk XenServer yum install pciutils python libxml2 python libnl expat glib2 tcl be libstde tk
14. LC SEND 28 Mellanox Technologies Rev 2 2 1 0 1 11 Bug Fixes History Table 14 lists the bugs fixed in this release Table 14 Fixed Bugs List Ae Discovered Fixed in a Duc Deseripuon in Release Release l Ethernet Fixed kernel panic on Debian 6 0 7 which occurred 2 1 1 0 0 2 2 1 0 1 when the number of TX channels was set above the default value 2 Fixed a crash incidence which occurred when 2 0 2 0 5 2 2 1 0 1 enabling Ethernet Time stamping and running VLAN traffic 3 XRC XRC over ROCE in SR IOV mode is not functional 2 0 3 1 0 2 2 1 0 1 4 mlx4 en Fixed wrong calculation of packet true size reporting 2 1 1 0 0 2 2 1 0 1 in LRO flow 5 IB Core Fixed the QP attribute mask upon smac resolving 2 1 1 0 0 2 1 1 0 6 6 mlx5 16 Fixed a send WQE overhead issue 2 1 1 0 0 2 1 1 0 6 7 Fixed a NULL pointer dereference on the debug 2 1 1 0 0 2 1 1 0 6 print 8 Fixed arguments to kzalloc 2 1 1 0 0 2 1 1 0 6 9 mlx4 core Fixed the locks around completion handler 2 1 1 0 0 2 1 1 0 6 10 mlx4 core Restored port types as they were when recovering 2 0 2 0 5 2 1 1 0 0 from an internal error 11 Added an N A port type to support port type array 2 0 2 0 5 2 1 1 0 0 module param in an HCA with a single port 12 SR IOV Fixed memory leak in SR IOV flow 2 0 2 0 5 2 0 3 0 0 13 Fixed communication channel being stuck 2 0 2 0 5 2 0 3 0 0 14
15. Mellanox TE Connect CHNOLOGIES Accelerate Outperform Mellanox OFED for Linux Release Notes Rev 2 2 1 0 1 Last Modified July 13 2014 www mellanox com Rev 2 2 1 0 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION E
16. PoIB operating mode in ConnectX family adapter cards is UD and CM in Connect IB 6 Changing the IPoIB mode CM vs UD requires the interface to be in down state 7 In RHEL7 0 the Network Manager can detect when the carrier of one of the IPoIB interfaces is OFF and can decide to disable its IP address 8 IPoIB interface does not function properly if a third party application changes the PKey table We recommend modifying PKey tables via OpenSM 9 When creating a new child interface in an over loaded kernel a dmesg print is displayed advising the user to try again in a few seconds 10 Out of memory issue might occur due to overload To calculate the allowed of interfaces created memory per each IPoIB interface check the follow ing Num rings min num cores on that device 16 Ring size 512 by default it is module parameter UD memory 2 num rings ring size 8K e CM memory ring size 64k Total memory UD mem CM mem 14 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued Index Issue Description Workaround 11 The physical port MTU indicates the port capa In order to change the IPoIB bility default value was changed to 4k whereas MTU to 4k edit the the IPoIB port MTU logical MTU default OpenSM partition file in the value is 2k as it is set by the OpenSM section of IPoIB setting as follow
17. SLES 11 SP1 zypper install pciutils python libxml2 python tesh libnl libstde 43 expat glib2 tcl bc libnl i586 tk SLES 11 SP2 zypper install pciutils python libnl 32bit libxml2 python tesh libnl lib stdc 46 expat glib2 tcl bc tk SLES 11 SP3 zypper install pciutils python libnl 32bit libxml2 python tesh libstde 43 libnl expat glib2 tcl bc tk Ubuntu Debian apt get install dpkg autotools dev autoconf libtool automake1 10 automake m4 dkms debhelper tcl tc18 4 chrpath swig graphviz tcl dev tcl8 4 dev tk dev tk8 4 dev bison flex dpatch zlib1g dev curl libcurl4 gnutls dev python libxml2 libvirt bin libvirtO libnl dev libglib2 0 dev libgfortran3 6 Supported HCAs MLNX OFED Rev 2 2 1 0 1 supports the following Mellanox network adapter cards Connect IBG Rev 10 10 3000 and above ConnectX 3 Pro Rev 2 31 5050 and above ConnectX 3 Rev 2 31 5050 and above ConnectX 2 Rev 2 9 1200 and above For official firmware versions please see http www mellanox com content pages php pg firmware download 1 ConnectX 2 does not support all the new functionality of MLNX OFED 2 0 3 XXX For the complete list of the supported features per HCA please refer to the MLNX OFED User Manual 8 Mellanox Technologies J Rev 2 2 1 0 1 7 Compatibility MLNX OFED Rev 2 2 1 0 1 is compatible with the following Table 4 MLNX OFED Rev 2 2 1 0 1 Compatibility Matrix Mel
18. VEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 2 2 1 0 1 Table of Contents Table of Contents uuo Sue cae oe 3 List Of Tables 54k 4e oe Sa Che hh Ke hmm RE Roe a x Roo el Sadan Oe 4 Chapter 1 OVOEVIEW o u ox sree on e a Ue Deine Se abre oU Relates 5 Chapter2 Main Features in This 5 Chapter3 Content of Mellanox OFED for Linux 6 Chapter 4 Supported Platforms and Operating Syste
19. b iser i 106 In Ubuntu12 04 need to update user space open iscsi package to version 2 0 873 107 Trying to disconnect a session while the session is Restart iscsid undergoing a reconnect flow may result in discon Note Please be aware that nection hang doing so might cause rmmod process to hang as the 16 iser module will not be unloaded 108 Unloading ib iser during session disconnect event may result in kernel panic 109 Ubuntul2 requires update of user space open iscsi to v2 0 873 110 The initiator does not respect interface parameter Configure each interface on while logging in a different subnet 111 When connecting to a target with no exposed LUNs the following message may be reported Unexpected RDMA CM event The message can be safely ignored 112 The initiator may take a long time to stop trying to log in via a non capable interface 113 ZFS appliance Connection establishment occurs twice which may cause iSER to log a stack trace 14 interop Ora Connection establishment occurs twice which cle Sun ZFS storage may cause iSER to log a stack trace 7420 Mellanox Technologies 25 J Rev 2 2 1 0 1 API Changes 10 10 1 Changes MLNX_OFED Rev 2 2 1 0 1 The following the API changes MLNX_OFED Rev 2 2 1 0 1 Table 10 Changes MLNX_OFED Rev 2 2 1 0 1 Name Description libibverbs The following verbs changed to align with
20. case user can enlarge the vmalloc memory by adding vmal loc lt size gt to grub conf Default vmalloc setting is 128M It is recommended to add each time 64M of memory until desired ring size can be allocated Please note that in case vmalloc size is too big the OS will fail to boot so please use caution when adding additional memory For more info refer to http www mythtv org wiki Common Problem vmalloc too small 16 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued Index Issue Description Workaround 20 21 22 23 24 25 26 On OEL6 1 with uek1 2 6 32 x x x el6uek ker nel when the number of RX ring is smaller than TX rings kernel issue the following call trace will be shown in the kernel log WARNING at net core dev c 2077 get rps cpu 0x70 0x2b9 Kernel panic might occur during traffic over IPv6 on kernels between 3 12 rc7 and 3 13 rc1 kernel issue Kernel panic might occur during fio splice in ker nels before 2 6 34 rc4 Use kernel v2 6 34 rc4 which provides the follow ing solution baff42a net Fix oops from tcp 11 when using splice When creating more than 125 VLANs and SR IOV mode is enabled a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic kernel warning m1x4 core 0000 07 00 0 vhcr c
21. ed by the fabric driver and that returned by fabric driver utilities that utilities not Oxfffff read the GUID via device firmware e g using Ibstat MIxburn flint return Oxffff as GUID while the utilities return a value derived from the MAC address For all driver firmware software pur poses the latter value should be used 22 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued Index Issue Description Workaround 81 Uplinks On rare occasions ConnectX 3 Pro adapter Restart the driver card may fail to link up when performing parallel detect to 40GbE 82 Resources Limitation The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace 83 Occasionally a user process might experience To free memory to allow it some memory shortage and not function properly to be allocated in a user pro due to Linux kernel occupation of system s cess run the drop caches free memory for its internal cache procedure below Performing the following steps will cause the kernel to flush and free pages den tries and inodes caches from memory causing that mem ory to become free Note As this is a non destructive operation and dirty objects are not free able run sync first free the pagecache echo 1 proc sys vm drop cache
22. f maximum sup ported VLs is 4 If port type is ETH then the maximum priority for VLAN tagged is 3 mlx4 core 0000 0d 00 0 command SET PORT 0 failed in param 0x120064000 in mod 0x2 op mod 0x0 fw status 0x40 47 Performance Tools perftest package in MLNX OFED v2 2 1 0 1 does not work with older versions of the driver Mellanox Technologies 19 J Rev 2 2 1 0 1 Known Issues Table 9 Known Issues Continued Index Issue Description Workaround 48 49 50 5 52 53 54 55 56 ST 58 SR IOV When using legacy VMs with OFED 2 0 2 0 5 hypervisor the enable 64b cqe eqe parameter must be set to zero on the hypervisor It should be set in the same way that other module parameters are set for mlx4 core at module load time For example add options mlx4 core enable 64b eqe 0 asa line in the file etc modprobe d mlx4 core conf When working with a bonding device to enslave the Ethernet devices in active backup mode and failover MAC policy in a Virtual Machine VM establishment of RoCE connections may fail Unload the module mlx4 ib and reload it in the VM Trying to establish RoCE connection with RDMA CM may fail in a VM Unload the module mlx4 ib and reload it in the VM When the number of single port VFs is not the same for port 1 and port 2 and the relevant VF is
23. ing the openibd script when no InfiniBand vendor module is selected for example mlx4 ib may cause the execution of the sbin start udev script In RedHat 6 x and OEL6 x this may change the local system time 41 ConnectX 3 Adapter Cards Family Using RDMA READ with a higher value than 30 SGEs in the WR might lead to local length error Donot set the value of SGEs higher than 30 when RDMA READ is used 42 Verbs Using libnll 1 3 26 or earlier requires Ibv create ah protection by a lock for multi threaded applications 43 44 45 46 Driver Start Out of memory issues may rise during drivers load depending on the values of the driver module parameters set e g log num cq When reloading starting the driver using the etc init d openibd the following messages are displayed if there 1s a third party RPM or driver installed Module mlx4 core does not belong to MLNX OFED or Module mlx4 core belong to lt rpm name gt which is not a part of MLNX OFED Remove the third party RPM non MLNX_OFED drivers directory run depmod and then rerun etc init d openibd restart Occasionally when trying to repetitively reload the nes hardware driver on SLES11 SP2 a soft lockups occurs that required reboot In ConnectX 2 if the driver load succeeds the informative message below is presented convey ing the below limitations If port type is IB the number o
24. lanox Product Description Version MLNX OS MSX6036 w w MLNX OS version 3 3 4304 Grid Director 4036 w w Grid Director version 3 9 1 985 FabricIT EFM 155035 w w FabricIT EFM version 1 1 3000 FabricIT BXM MBX5020 w w FabricIT BXM version 2 1 2000 Unified Fabric Manager UFM v4 6 MXM v3 0 ScalableUPC v2 2 ScalableSHMEM v2 2 FCA v2 5 OMPI v1 6 5 MVAPICH v1 9a CD v1 0 a MLNX OFED v2 2 1 0 1 was tested with this switch however additional switches might be supported as well Mellanox Technologies 9 J Rev 2 2 1 0 1 Change Log History 8 Change Log History 8 1 Changes in Rev 2 2 1 0 1 From Rev 2 1 1 0 6 Table 5 New Features Changes and Fixes in v2 2 1 0 1 Category Description mlnxofedinstall 32 bit libraries are no longer installed by default 64 bit OS To install 32 bit libraries use the with 32bit installation parameter openibd Added pre post start stop scripts support For further information please refer to section openibd Script in the MLNX_OFED User Manual Reset Flow Reset Flow is not activated by default It is controlled by the mlx4 core internal err reset module parameter InfiniBand Core Asymmetric MSI X vectors allocation for the SR IOV hypervisor and guest instead of allocating 4 default MSI X vectors The maximum number of MSI X vectors Is num cpu for port ConnectX 3 has 1024 MSI X vectors 28 MSI X vectors are reserved Physical Function ge
25. ms 7 4 1 Supported Hypervisors sarip ine EU Ee EOS bre 7 Chapter 5 Hardware and Software Requirements 8 Chapter 6 Supported HCAS 8 Chapter 7 Compatibility 9 Chapter 8 Change Log History 10 8 1 Changes in Rev 2 2 1 0 1 From Rev 2 1 1 0 6 10 8 2 Changes in Rev 2 1 1 0 6 From Rev 2 1 1 0 0 10 8 3 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 11 84 Changes in Rev 2 0 3 0 0 From Rev 2 0 2 0 5 12 8 5 New Features in Rev 2 0 2 0 5 12 Chapter9 Known Issues 14 Chapter 10 APIT Changes ise suc susya eS REUS econ isin wi ane We Wise bie maw TR w wai 26 10 1 API Changes in MLNX OFED Rev 2 2 1 0 1 26 10 2 API Changes in MLNX OFED Rev 2 1 1 0 0 27 10 3 API Changes in MLNX OFED Rev 2 0 3 0 0 27 10 4 API Changes in MLNX OFED Rev 2 0 2 0 5 28 Chapter 11 Bug Fixes History 29 Mellanox Technologies 3 J Rev 2 2 1 0 1 List Tables Table 1 Mellanox OFED for Linux Software Components
26. ng the driver probe d mlx4 core conf On the next boot you will get the panic and machine will boot up 68 Installation When upgrading from an earlier Mellanox OFED Stop the old OFED stack version the installation script does not stop the etc init d openibd earlier version prior to uninstalling it stop before upgrading to this new version 69 Upgrading from the previous OFED installation Reboot after installing the to this release does not unload the kernel module driver Ipoib helper 70 Installation using Yum does not update HCA See Updating Firmware firmware After Installation in OFED User Manual 71 On SLES11 1 the package libnl i586 is required Perform one of the follow to install MLNX_OFED ing Install the libnl 1586 RPM from the SLES11 1 32bit installation disk Install MLNX OFED with the following flag with out 32bit Mellanox Technologies 21 J Rev 2 2 1 0 1 Known Issues Table 9 Known Issues Continued Index Issue Description Workaround 72 total vfs lt 0 63 gt installation parameter Use enable sriov is no longer supported installation parameter to burn firmware with SR IOV support The number of vir tual functions VFs will be set to 16 For further information please refer to the User Manual 73 When using bonding on Ubuntu OS the ifenslave package must be installed 74 Driver Unload openibd stop can sometime fail with
27. ns of rescan bus sh may not rec If encountering such issues ognize some newly created LUNs it is recommended to use the c flag 98 SRP Reconnecting to a target during host reset stage rescan scsi may result in devices going Offline bus sh r 99 MLNX_OFED SRP installation breaks the ibmvstgt and ibmvscsi symbol resolution in RHEL7 0 100 SRP Interop The driver is tested with Storage target vendors recommendations for multipath conf extensions ZFS DDN TMS Nimbus NetApp 24 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued Index Issue Description Workaround 101 DDN Storage Fusion DDN does not accept non default Key connec 10000 target tion establishment 102 Oracle Sun ZFS stor Occasionally the first command to a LUN may age 7420 not be serviced aborted and cause a successful re connection to the target 103 ZFS does not accept non default P Key connec tion establishment 104 Ungraceful power cycle of an initiator connected with Targets DDN Nimbus NetApp may result in temporary stale connection messages when initiator reconnects 105 SER On SLES11 the ib iser module does not load Add a dummy interface on boot using iscsiadm e iscsiadm m iface I ib iser O new e iscsiadm m iface I ib iser o update n iface transport na me v i
28. ommand ALLOC RES Oxf00 slave 0 in param 0 7 in mod 0x107 op 0 1 failed with error 0 sta tus 28 In PPC systems when QoS is enabled a harmless Kernel DMA mapping error messages might appear in kernel log iommu related issue On kernels that do not support multi queues the number of TX channels represents the number of TX rings The maximal number of TX channels is 16 Transmit timeout might occur on RH6 3 as a result of lost interrupt OS issue In this case the following message will be shown in dmesg do IRQ 0 203 No irq handler for vec tor irq 1 Mellanox Technologies 17 J Rev 2 2 1 0 1 Known Issues Table 9 Known Issues Continued Index Issue Description Workaround 27 28 29 30 31 32 33 eIPoIB On rare occasions upon driver restart the follow ing message is shown in the dmesg cannot create duplicate filename class net eth ipoib interfaces No indication is received when eIPoIB is non functional Run ps ef grep ipoibd to verify its func tionality eIPoIB requires libvirtd python eIPoIB supports only active backup mode for bonding eIPoIB supports only VLAN Switch Tagging VST mode on guests IPv6 is currently not supported in eIPoIB eIPoIB cannot run when Flow Steering is enabled 34 35 36 37 XRC Legacy API is deprecated thus
29. s free dentries and inodes echo 2 proc sys vm drop caches free pagecache dentries and inodes echo 3 proc sys vm drop caches 84 Setting more IP addresses than the available GID entries in the table results in failure and the update gid table error message is displayed GID table of port 1 is full Can t add lt address gt message 85 Ethernet Perfor In a system with more than 61 VFs the 62nd VF mance Counters and onwards is assigned with the SINKQP coun ter and as a result will have no statistics and loopback prevention functionality for SINK coun ter 86 Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics in a system with less than 61 VFs if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet In this case RoCE statistics is not available Mellanox Technologies 23 J Rev 2 2 1 0 1 Known Issues Table 9 Known Issues Continued Index Issue Description Workaround 87 In ConnectX 3 when we enable function based loopback prevention for Ethernet port by default i e based on the QP counter index the dropped self loopback packets increase the IfRxError Frames Octets counters 88 RoCE Not configuring the Ethernet devices or indepen Restart the driver dent VMs with a unique IP address in the physical port may result in RoCE GID table corr
30. the Re run openibd stop error Unloading ib cm FAILED ERROR Module ib cm is in use by ib ipoib 75 Fork Support Fork support from kernel 2 6 12 and above is available provided that applications do not use threads fork is supported as long as the parent process does not run before the child exits or calls exec The former can be achieved by calling wait childpid and the latter can be achieved by application specific means The Posix system call is supported 76 ISCSI over IPoIB When working with ISCSI over IPoIB LRO must be disabled even if IPoIB is set to connected mode due to a a bug in older kernels which causes a kernel panic 77 MLNX_OFED MLNX OFED includes OFED source RPM Sources packages used as a build platform for kernel code but does not include the sources of Mellanox pro prietary packages 78 InfiniBand Utilities When running the ibdiagnet check nodes info Runibdiagnet skip on the fabric a warning specifying that the card nodes info does not support general info capabilities for all the HCAs in the fabric will be displayed 79 mlx5 Driver Atomic Operations in Connect IB are fully sup ported on big endian machines e g PPC Their support 1s limited on little endian machines e g x86 80 General On ConnectX 2 ConnectX 3 Ethernet adapter N A cards there is a mismatch between the GUID Please use the GUID value value returned by firmware management tools return
31. ts the number of MSI X vectors according to the pf msix table size multiple of 4 1 INI parameter Virtual Functions the remaining MSI X vectors are spread equally between all VFs according to the num v s ml1x4 core module parameter Ethernet Ethernet VXLAN support for kernels 3 12 10 or higher Power Management Quality of Service when the traffic is active the Power Man agement QoS is enabled by disabling the CPU states for maximum performance Ethernet PTP Hardware Clock support on kernels OSes that support it Verbs Added additional experimental verbs interface This interface exposes new features which are not integrated yet in to the upstream libibverbs The Experimental API is an extended API therefor it is backward com patible meaning old application are not required to be recompiled to use MLNX OFED v2 2 1 0 1 Performance Out of the box performance improvements Use of affinity hints based on NUMA node of the device to indicate the IRQ balancer daemon on the optimal IRQ affinity Improvement in buffers allocation schema based on the hint above Improvement in the adaptive interrupt moderation algorithm 8 2 Changes in Rev 2 1 1 0 6 From Rev 2 1 1 0 0 Table 6 New Features Changes and Fixes in v2 1 1 0 6 Category Description IB Core Added allocation success verification process to ib alloc device dapl dapl is recompiled with no FCA support 10 Mellano
32. upstream libibverbs ibv_reg mr ibv_access flags changed ibv post send opcodes and send flags changed and wr fields removed task op dc and bind mw ibv query device capability flags changed ibv poll cq opcodes and wc flags changed ibv modify qp mask bits changed ibv create qp ex create flags field removed The following verbs removed to align with upstream libibverbs ibv bind mw ibv post task ibv query values ex ibv query device ex ibv poll cq ex ibv reg shared mr ex ibv reg shared mr ibv modify cq ibv create cq ex ibv modify qp ex Verbs Experimental API The following experimental verbs added replacing the removed extended verbs jbv exp bind mw jbv exp post task 16 exp query values ibv exp query device 16 exp poll cq 16 exp reg shared mr 10 exp modify cq jbv exp create cq 16 exp modify qp New experimental verbs 16 exp arm dct 16 exp query port 16 exp create flow ibv exp destroy flow ibv exp post send 16 exp reg mr ibv exp get provider func 26 Mellanox Technologies Rev 2 2 1 0 1 10 2 API Changes in MLNX_OFED Rev 2 1 1 0 0 The following are the API changes in MLNX_OFED Rev 2 1 1 0 0 Table 11 API Changes in MLNX_OFED Rev 2 1 1 0 0 Name Description Dynamically Connected DC The following verbs were added e struct dct
33. uption 89 If RDMA CM is not used for connection man agement then the source and destination GIDs used to modify a QP or create AH should be of the same type IPv4 or IPv6 90 On rare occasions the driver reports a wrong GID table read from sys class infiniband mlx4 ports gids This may cause communication problems 91 Since the number of GIDs per port is limited to 128 there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port Allowed number is 127 for a single function machine 15 for a hypervisor in a multifunction machine for a guest in a multifunction machine where n is the number of virtual functions 92 A working IP connectivity between the RoCE devices is required when creating an address han dle or modifying a QP with an address vector 93 MLNX OFED v2 1 1 0 0 is not interoperable with older versions of MLNX OFED 94 When configuring the single port per VF fea ture if the number of VFs on port 1 is different from port 2 RoCE rdma cm on single ported VFs which use port 2 may not work 95 Unloading mlx4 en while a rdma cm session is established can cause a kernel panic 96 Storage SLES11 SP1 When running multipath rescan while new devices are added to mpath tables multipath may not find all the device mappers 97 Older versio
34. when recompiling applications over MLNX OFED v2 0 3 x x warnings such as the below are displayed rdma c 1699 warning ibv open xrc domain is deprecated declared at usr include infiniband ofa verbs h 72 rdma c 1706 warning ibv create srq is deprecated declared at usr include infiniband ofa verbs h 89 These warnings can be safely ignored XRC is not functional in heterogeneous clusters containing non Mellanox HCAs XRC options do not work when using qperf tool Use perftest instead Out of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created XRC QPs do not have receive queues 38 mlx4 ib module The dev assign str module parameter is not backward compatible In the current version this parameter is using dec imal number to describe the InfiniBand device and not hexadecimal number as it was in previous versions in order to uniform the mapping of device function numbers to InfiniBand device numbers as defined for other module parameters e g num v s and probe vf 39 ABI Compatibility MLNX OFED Rev 2 2 1 0 1 is not ABI compati ble with previous MLNX OFED OFED versions Recompile the application over the new MLNX OFED version 18 Mellanox Technologies Rev 2 2 1 0 1 Table 9 Known Issues Continued Index Issue Description Workaround 40 System Time Loading the driver us
35. x Technologies Rev 2 2 1 0 1 Table 6 New Features Changes and Fixes in v2 1 1 0 6 Category Description openibd Added the ability to bring up child interfaces even if the parent s ifcfg file is not configured libmlx4 Unmapped the hca_clock_page parameter from mlx4_uninit_context scsi_transport_srp scsi_transport_srp cannot be cleared up when rport reconnecting fails minxofedinstall Added support for the following parameters umad dev na without package Content Packages Updates The following packages were updated to v2 2 407 mstflint to v3 5 0 1 1 g76e4acf perftest to v2 0 0 76 gbf9a463 hcoll to v2 0 472 1 Openmpi to v1 6 5 440ad47 daplto v2 0 40 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 Table 7 New Features Changes and Fixes in v2 1 1 0 0 Category Description EoIB EoIB is supported only in SLES11SP2 and RHEL6 4 eIPoIB eIPoIB is currently at GA level Connect IB Added the ability to resize CQs IPoIB Reusing DMA mapped SKB buffers Performance improvements when IOMMU is enabled minx en Added reporting autonegotiation support Added Transmit Packet Steering XPS support Added reporting 56Gbit s link speed support Added Receive Flow Steering RFS support in UDP Added Low Latency Socket LLS support Added check for dma mapping errors eIPoIB

Mellanox OFED for Linux Release Notes

Contents

Download Pdf Manuals

Related Search

Related Contents