Home

Mellanox OFED for Linux Release Notes

1. Rev 2 3 1 0 1 Known Issues 3 3 General Known Issues Table 12 General Known Issues Index Description Workaround 1 On ConnectX 2 ConnectX 3 Ethernet adapter cards N A there is a mismatch between the GUID value Please use the GUID value returned by the fab returned by firmware management tools and that ric driver utilities not Oxfffff returned by fabric driver utilities that read the GUID via device firmware e g using ibstat MIxburn flint return Oxffff as GUID while the utilities return a value derived from the MAC address For all driver firmware software purposes the latter value should be used 3 4 VGT Known Issues Table 13 VGT Known Issues Index Description Workaround l Before adding a VLAN on the VM the parent inter face should be brought up Otherwise VLAN cre ation will fail and the following message will be presented Fail to register network rule 2 Bringing down and up the parent interface with VLANS configured over it may result in traffic over VLANs being lost 3 On some of the OSes the callback ndo vlan rx add kill vidreturns void there fore ethX Y is created However if VLAN Y is not listed in the set of the allowed VLANs no traffic will pass 4 When untagged traffic is not allowed the below message will appear on Dom0 after driver restart on DomU mlx4 core 0000 16 00 0 vhcr command ALLOC RES 0xf00 slave 1 in param 0x0 in mod
2. 128 Byte Completion Queue Entry CQE Non Linux Virtual Added Windows Virtual Machine over Linux KVM Hypervisor SR IOV with Machines InfiniBand only support 2 1 API Changes in MLNX OFED Rev 2 3 1 0 1 The following are the API changes in MLNX OFED Rev 2 3 1 0 1 Table 9 API Change Log History Release Name Description Rev2 3 1 0 1 libibverbs e ibv exp rereg mr Added new API for memory region re reintegration For futher information please refer to MLNX OFED User Manual e Added to the experimental API ibv exp post send the following opcodes e IBV EXP WR EXT MASKED ATOMIC CMP AND SWP e IBV EXP WR EXT MASKED ATOMIC FETCH AND ADD e IBV EXP WR NOP and these completion opcodes e IBV EXP WC MASKED COMP SWAP e IBV EXP WC MASKED FETCH ADD 14 Mellanox Technologies Rev 2 3 1 0 1 3 Known Issues The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release 3 1 IPoIB Known Issues Table 10 IPoIB Known Issues Index Description Workaround l When user increases receive send a buffer it might consume all the memory when few child s interfaces are created The size of send queue in Connect IB cards cannot exceed 1K In 32 bit devices the maximum number of child interfaces that can be created is 16 Creating more that might cause out of memory issues In R
3. 1 Overview These are the release notes of Mellanox OFED for Linux Driver Rev 2 3 1 0 1 Mellanox OFED is a single Virtual Protocol Interconnect VPI software stack and operates across all Mellanox network adapter solutions supporting the following uplinks to servers 10 20 40 and 56 Gb s InfiniBand IB 10 40 and 56 Gb s Ethernet 2 5 or 5 0 GT s PCI Express 2 0 8 GT s PCI Express 3 0 1 4 Main Features in This Release MLNX_OFED Rev 2 3 1 0 1 provides the following new features Secure Host Virtual Guest Tagging VGT User Mode Memory Registration UMR Masked Atomics On Demand Paging ODP Reset Flow for ConnectX 3 SR IOV RoCE v2 Checksum offload for packets without L4 header Flow Steering AO simplified steering Memory re registration Cable EEPROM reporting 128 Byte Completion Queue Entry CQE Explicit Congestion Notification ECN Disable Enable ethernet RX VLAN tag striping offload via ethtool Windows Virtual Machine over Linux KVM Hypervisor SR IOV with InfiniBand only 1 2 Content of Mellanox OFED for Linux Mellanox OFED for Linux software contains the following components Table 2 Mellanox OFED for Linux Software Components Components Description OpenFabrics core and ULPs IB HCA drivers mlx4 mlx5 core e Upper Layer Protocols IPoIB SRP and iSER Initiator 1 56 GbE is a Mellanox propriety link speed and can be achieved while connected to Mellanox SX10XX switch
4. MLNX OFED or Module mlx4 core belong to rpm name gt which is not a part of MLNX OFED 3 Occasionally when trying to repetitively reload the nes hardware driver on SLES11 SP2 a soft lockups occurs that required reboot 4 In ConnectX 2 if the driver load succeeds the informative message below is presented conveying the below limitations If port type is IB the number of maximum sup ported VLs is 4 fporttype is ETH then the maximum priority for VLAN tagged is 3 mlx4 core 0000 0d 00 0 command SET PORT 0xc failed in param 0x120064000 in mod 0x2 op_mod 0x0 fw status 0x40 5 openibd start unloads kernel modules that were loaded from initrd initramfs upon boot This affects only kernel modules which come with MLNX_OFED and are included in initrd initramfs 3 13 Performance Tools Known Issues Table 22 Performance Tools Known Issues Index Description Workaround l perftest package in MLNX OFED v2 2 1 0 1 and onwards does not work with older versions of the driver Mellanox Technologies 21 J Rev 2 3 1 0 1 Known Issues 3 14 Performance Known Issues Table 23 Performance Known Issues Index Description Workaround l On machines with irqbalancer daemon turned off To avoid this issue copy the following script to the default InfiniBand interrupts will be routed toa etc infiniband post start hook sh single core which m
5. 0x207 op mod 0x1 failed with error 0 status 1 3 5 elPolB Known Issues Table 14 elPoIB Known Issues Index Description Workaround l On rare occasions upon driver restart the following message is shown in the dmesg cannot create duplicate filename class net eth ipoib interfaces 2 No indication is received when eIPoIB is non func Run ps ef grep ipoibd to verify its tional functionality 3 eIPoIB requires libvirtd python 18 Mellanox Technologies Rev 2 3 1 0 1 Table 14 elPoIB Known Issues Continued Index Description Workaround 4 eIPoIB supports only active backup mode for bond ing 5 eIPoIB supports only VLAN Switch Tagging VST mode on guests 6 IPv6 is currently not supported in eIPoIB 7 eIPoIB cannot run when Flow Steering is enabled 3 6 XRC Known Issues Table 15 XRC Known Issues Index Description Workaround l Legacy API is deprecated thus when recompiling applications over MLNX OFED v2 0 3 x x warn ings such as the below are displayed rdma c 1699 warning ibv open xrc domain is deprecated declared at usr include infiniband ofa verbs h 72 rdma c 1706 warning ibv create xrc srq is deprecated declared at usr include infiniband ofa verbs h 89 These warnings can be safely ignored 2 XRC is not functional in heterogeneous clusters containing n
6. 15 Connection Manager CM Known Issues 0000 e eee eee eee 22 3 16 SR IOV Known Isuues 2 0 eect nen neee 23 3 17 Port Type Management Known Issues 0 00 eee eens 24 3 18 Flow Steering Known Issues lsseeeeee n 25 3 19 Quality of Service Known Issues 0 000 ccc ees 25 3 20 Installation Known Issues 0 0 0 cee ee 25 3 21 Driver Upload Known Issues llle 26 3 22 InfiniBand Counters Known Issues llle ees 26 3 23 UEFI Secure Boot Known Issues 0 0000 cece eee 26 3 24 Fork Support Known Issues 00 0 cece ene 26 Mellanox Technologies 3 J Rev 2 3 1 0 1 3 25 ISCSI over IPoIB Known Issues eens 27 3 26 MLNX OFED Sources Known Issues 0 00 eee eee eens 27 3 27 InfiniBand Utilities Known Issues lesse ees 27 3 28 mlx5 Driver Known Issues lsssseeeeeeee een 27 3 29 Ethernet Performance Counters Known Issues 02 0000 cece eens 27 3 30 Uplinks Known Issues lessen 28 3 3 Resources Limitation Known Issues ees 29 3 32 ROCE Known Issues peee oona ii i a e nen ene 29 3 33 Storage Known Issues 0 ccc eee ence a 31 3 34 SRP Known Issues 0 0 0 ccc cece ehh hh hr 31 3 35 SRP Interop Known Issues 0 e 31 3 36 DDN Storage Fusion 10000 Target Known Issues 000000 ee eee 31 3 37 Oracle Sun ZFS storage 7420 Known Issues 0 0000 cee eee 31 3 38 ISER Known Is
7. Operating Systems 0 00 0 cece 9 1 3 1 Supported Hypervisors 00 eh 10 1 3 2 Supported Non Linux Virtual Machines esee 10 1 4 Hardware and Software Requirements 00 00 c cece eee eens 11 1 5 Supported HCAs Firmware Versions 0 0 e eee ence eh 11 1 6 Compatibility sssri n cas ale IE IS Lbs eL M LU 12 1 7 RoCE Mode s Matrix oo oe et ep te de tede csi 12 Chapter 2 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 0 ccc eee eee eee 13 2 API Changes in MLNX OFED Rev 23 1 0 1 2 0 0 0 cee 14 Chapter 3 Known ISSU6S 65 uso re aie ime wha Ata sone RACK Ron ds Wale ods RR AE wate 15 3 1 IPoIB Known Issues ic eas e arean e aE eh 15 3 2 Ethernet Known Issues sse 17 3 3 General Known Issues ie wee dace eU pa sd ae ee ae edu 18 3 4 VGT Knownlssue8 v3 nne p S E pU ERATES s 18 3 5 eIPoIB Known Issues n aa a AE e 18 3 67 XRC Known Issues s ecco ph ene pue whe T espe gx ep Da guo 19 3 7 ABI Compatibility Known Issues sse ee 19 3 8 System Time Known Issues 00 0 cece n 20 3 9 ConnectX 3 Adapter Cards Family Known Issues 0 0055 20 3 10 Verbs Known Issues ssrin aaa ene een n ene nes 20 3 11 Resiliency Known Issues 0 0 ce cece eee nee nes 20 3 12 Driver Start Known Issues 0 0 0 0 cece ne 21 3 13 Performance Tools Known Issues 1 0 00 000 cece ees 21 3 14 Performance Known Issues l l 22 3
8. series 8 Mellanox Technologies J Rev 2 3 1 0 1 Table 2 Mellanox OFED for Linux Software Components Components Description OpenFabrics utilities OpenSM IB Subnet Manager with Mellanox proprietary Adaptive Routing Diagnostic tools Performance tests MPI OSU MPI mvapich2 1 9 1 stack supporting the InfiniBand interface Open MPI stack 1 6 5 and later supporting the InfiniBand interface MPI benchmark tests OSU benchmarks Intel MPI benchmarks Presta PGAS ScalableSHMEM v2 2 supporting InfiniBand MXM and FCA ScalableUPC v2 2 supporting InfiniBand MXM and FCA HPC Acceleration packages Mellanox MXM v3 0 p2p transport library acceleration over Infini band Mellanox FCA v2 5 MPI PGAS collective operations acceleration library over InfiniBand KNEM Linux kernel module enabling high performance intra node MPI PGAS communication for large messages Extra packages ibutils2 ibdump MFT Sources of all software modules under conditions mentioned in the modules LICENSE files except for MFT OpenSM plugins ibutils2 and ibdump Documentation 1 3 Supported Platforms and Operating Systems The following are the supported OSs in MLNX_OFED Rev 2 3 1 0 1 Table 3 Supported Platforms and Operating Systems Operating System Platform RHEL CentOS 6 3 x86 64 RHEL CentOS 6 4 x86 64 PPC RHEL CentOS 6 5 x86 64 PPC RHEL CentO
9. 6 7 0 SLES 12 Ubuntu 14 04 12 Mellanox Technologies Rev 2 3 1 0 1 2 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 Table 8 Changes in v2 3 1 0 1 Category Description OpenSM Added Routing Chains support with Minhop UPDN FTree DOR Torus 2QoS Added double failover elimination When the Master SM is turned down for some reason the Standby SM takes own ership over the fabric and remains the Master SM even when the old Master SM is brought up to avoid any unnecessary reregistartions in the fabric To enable this feature set the master sm priority parameter to be greater than the sm priority parameter in all SMs in the fabric Once the Standby SM becomes the Master SM its priority becames equal to the master sm priority So that additional SM handover is avoided Default value of the master sm priority is 14 To disable this feature set the master sm priority in opensm conf to 0 Added credit loop free unicast multicast updn ftree routing Added multithreaded Minhop UPDN DOR routing RoCE Added IP routable RoCE modes For further information please refer to the MLNX OFED User Manual Installation Added apt get installation support Ethernet Added support for arbitrary UDP port for VXLAN From upstream 3 15 rc1 and onward it is possible to use arbitrary UDP port for VXLAN This feature requires firmware version 2 32 5100 or higher Additionally the following kernel configurat
10. Experimental API Verbs experimental API defines MLNX OFED APIs ibv exp create dct extension scheme which is ibv exp destroy dct similar to the Verbs exten e bv exp query dct sion APT This extension provides a way to introduce new features before they are integrated into the formal OFA API and to the upstream kernel and libs Rev 2 0 3 0 0 XRC The following verbs have become deprecated struct ibv xrc domain ibv open xrc domain ibv exp create qp ibv exp query device e struct ibv srq ibv create xrc srq int ibv close xrc domain int ibv create xrc rcv qp int ibv modify xrc rcv qp int ibv query xrc rcv qp e int ibv reg xrc rcv qp int ibv unreg xrc rcv qp 40 Mellanox Technologies Rev 2 3 1 0 1 Table 51 API Change Log History Release Name Description Rev 2 0 2 0 5 Libibverbs Extended speeds Missing the ext active speed attribute from the struct ibv port attr e Removed function ibv ext rate to int e Added functions ibv rate to mbps and mbps to ibv rate Libibverbs Raw QPs QP types IBV OPT RAW PACKET and IBV_QPT RAW ETH are not supported Libibverbs Contiguous e Added Contiguous pages support pages e Added function ibv_reg shared mr Libmverbs The enumeration IBV M WR CALC was renamed to IBV M WR CALC SEND Theenumeration IBV M WR WRITE WITH IMM was added Inthe structure ibv m send wr the union wr send was renamed to wr calc sen
11. HEL7 0 the Network Manager can detect when the carrier of one of the IPoIB interfaces is OFF and can decide to disable its IP address Set ignore carrier for the corresponding device in NetworkManager conf For further information please refer to man NetworkManager conf IPoIB interface does not function properly if a third party application changes the PKey table We recommend modifying PKey tables via OpenSM Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring https bugs open fabrics org show_bug cgi id 1990 Out of memory issue might occur due to overload of interfaces created To calculate the allowed memory per each IPoIB interface check the following e Num rings min num cores on that device 16 Ring size 512 by default it is module parameter UD memory 2 num rings ring size 8K CM memory ring size 64k Total memory UD mem CM mem Connect IB does not reach the bidirectional line rate Optimize the IPoIB performance in Connect IB cat sys class net interface device local cpus sys class net interface queues rx 0 rps cpus Ifthe CONNECTED MODE no parameter is set to no or missing from the ifcfg file for Connect IB IPoIB interface then the service network restart will hang Setthe CONNECTED MODE yes parameter in the ifcfg file for Connect IBG interface Joining a multicast group in the SM using th
12. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox OFED for Linux Release Notes Rev 2 3 1 0 1 www mellanox com Rev 2 3 1 0 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF
13. OFED Rev 2 3 1 0 1 Linux operating system e Administrator privileges on your machine s Disk Space IGB For the OFED Distribution to compile on your machine some software packages of your operat ing system OS distribution are required To install the additional packages run the following commands per OS Table 4 Additional Software Packages Operating System Required Packages Installation Command RHEL OEL yum install perl pciutils python gcc gfortran libxml2 python tesh libnl 1686 Fedora libnl expat glib2 tcl libstdc bc tk gtk2 atk cairo numactl XenServer yum install perl pciutils python libxml2 python libnl expat glib2 tcl bc lib stdce tk SLES 11 SP2 zypper install perl pciutils python libnl 32bit libxml2 python tesh libnl lib stdc 46 expat glib2 tcl bc tklibcurl4 gtk2 atk cairo SLES 11 SP3 zypper install perl pciutils python libnl 32bit libxml2 python tesh lib stdc 43 libnl expat glib2 tcl bc tk libcurl4 gtk2 atk cairo Ubuntu Debian apt get install perl dpkg autotools dev autoconf libtool automake1 10 auto make m4 dkms debhelper tcl tcl8 4 chrpath swig graphviz tcl dev tcl8 4 dev tk dev tk8 4 dev bison flex dpatch zlib g dev curl libcurl4 gnutls dev python libxml2 libvirt bin libvirtO libnl dev libglib2 0 dev libgfortran3 auto make m4 1 5 Supported HCAs Firmware Versions MLNX OFED Rev 2 3 1 0 1 supports the following Mellanox network adapter cards firmware versio
14. R Known Issues Index Description Workaround l SM LID reassignmet during traffic on OEL6 4 uek kernel with a Virtual Function generates soft lockup trace 2 On SLES OSs the ib iser module does not load Add a dummy interface using iscsiadm on boot e 4 iscsiadm m iface I ib iser o new e d iscsiadm m iface I ib iser o update n iface transport name v ib iser 3 DIF When running IO over FS over DM during unstable ports block layer BIOs merges may cause false DIF error 4 Ubuntul2 requires update of user space open iscsi to v2 0 873 5 The initiator does not respect interface parameter Configure each interface on a different subnet while logging in 6 ISCSID v2 0 873 can enter an endless loop on bind error 7 DIX Under heavy IO stress with large block size Disable block merges the HCA card might generate error Completions in the log and traffic might be affected 8 iSCSID may hang if target crashes during logout sequence reproducible with TCP 3 39 ZFS Appliance Known Issues Table 48 ZFS Appliance Known Issues Index Description Workaround 1 Connection establishment occurs twice which may cause ISER to log a stack trace 32 Mellanox Technologies Rev 2 3 1 0 1 4 Bug Fixes History Table 49 lists the bugs fixed in this release Table 49 Fixed Bugs List Discovered Fixed in 2 Ius Description in Release Relea
15. S 7 0 x86 64 PPC SLES11 SP2 x86 64 PPC SLES11 SP3 x86 64 PPC OEL 6 3 x86 64 OEL 6 4 x86 64 OEL 6 5 x86 64 Citrix XenServer Host 6 2 1686 Fedora 19 x86 64 Fedora 20 x86 64 Ubuntu 12 04 4 x86 64 Ubuntu 14 04 x86 64 PPCAle Debian 7 2 x86 64 Mellanox Technologies 9 J Rev 2 3 1 0 1 Overview Table 3 Supported Platforms and Operating Systems Operating System Platform Debian 7 4 x86 64 Debian 7 5 x86 64 kernel 3 10 48 kernel 3 11 10 kernel 3 12 24 kernel 3 13 1 kernel 3 14 12 kernel 3 15 5 a This kernel is supported only when using the Operating Systems stated in the table above using mlnx add kernel support sh script gt If you wish to install OFED on a different kernel you need to create a new ISO image See the MLNX OFED User Guide for instructions 7 Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest version as well P 1 3 1 Supported Hypervisors The following are the supported hypervisors in MLNX_OFED Rev 2 3 1 0 1 KVM Xen 1 3 2 Supported Non Linux Virtual Machines The following are the supported Non Linux InfiniBand only Virtual Machines in MLNX OFED Rev 2 3 1 0 1 e Windows Server 2012 R2 Windows Server 20012 e Windows Server 2008 R2 10 Mellanox Technologies Rev 2 3 1 0 1 1 4 Hardware and Software Requirements The following are the hardware and software requirements of MLNX_
16. SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS TestX PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT HPC X Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 2 3 1 0 1 Table of Contents Table of Contents oes Kien a RR CR EX eo ee ea RE A OEE BRS oR ee EGER GEES CaS 3 List OF Tables iy de Cais Bead ACRIOR n CS ica nece CR Na oboe Sols OR CE Ma tae n 5 Release Update History 54 06 ike EAR EE IP ox ERAS OR SEE PR t E EFE bud E Chapter 1 OvVervieweuo siue dx ox Ee OR Doe CC RICERCHE aUe a 8 1 1 Main Features in This Release 0 0 0 ee cette tenes 8 1 2 Content of Mellanox OFED for Linux 0 0 00 8 1 3 Supported Platforms and
17. Ses and Guest OSes an issue might occur when attaching detaching VFs to a guest while that guest is up and running Attach detach VFs to from a VM only while that VM is down When working with SR IOV in Xen 4 2 virtualiza tion platform only the built in xen pciback driver should be loaded The xen pciback module in dom0 should not be loaded as loading them simultaneously may cause interrupts loss and cause the driver to enter the reset flow Mellanox Technologies 23 J Rev 2 3 1 0 1 Known Issues Table 25 SR IOV Known Issues Continued Index Description Workaround 11 The known PCI BDFs for all VFs in kernel com mand line should be specified by adding xen pci back hide For further information please refer to http wiki xen org wiki Xen_PCI Passthrough 12 The qemu version 2 0 provided inbox with Ubuntu 14 04 does not work properly when more than 2 VMs are run over an Ubuntu 14 04 Hypervisor 13 SR IOV UD QPs are forced by the Hypervisor to use the base GID 1 e the GID that the VF sees in its GID entry at its paravirtualized index 0 This is needed for security since UD QPs use Address Vec tors and any GID index may be placed in such a vector including indices not belonging to that VF 14 Attempting to attach a PF to a VM when SR IOV is already enabled on that PF may result in a kernel panic 15 osmtest on the Hypervisor fails when SR IOV is enabled H
18. _help 0x148 0x160 Call Trace lt IRQ gt fffff81448d83 5 dev hard start xmit4 0x3e3 0x530 fffff8144Cc805 5 dev queue xmit 0x205 0x550 ffff8145247d neigh connected output 0xbd 0x1 13 When InfiniBand ports are removed from the host To avoid it and have persistent IPoIB network e g when changing port type from IB to Eth or devices names for ConnectX ports add to the removing a card from the PCI bus the remaining etc udev rules d 70 persistent IPoIB interface might be renamed net rules file SUBSYSTEM net ACTION add DRIVERS ATTR address Port GID NAME ibN Where N is the IPoIB required interface index 14 After releasing a bond interface that contains IPoIB slaves a call trace might be printed into the dmesg 16 Mellanox Technologies Rev 2 3 1 0 1 Table 10 IPoIB Known Issues Continued Index Description Workaround 15 The LRO feature cannot be disabled via ethtool on kernels 3 9 3 2 Ethernet Known Issues Table 11 Ethernet Known Issues Index Description Workaround l When creating more than 125 VLANs and SR IOV mode is enabled a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic kernel warning m1x4 core 0000 07 00 0 vher command ALLOC RES 0xf00 slave 0 in param 0x7e in mod 0x107
19. aches procedure below Performing the following steps will cause the kernel to flush and free pages dentries and inodes caches from memory causing that memory to become free Note As this is a non destructive operation and dirty objects are not freeable run sync first To free the pagecache echo 1 proc sys vm drop caches To free dentries and inodes echo 2 proc sys vm drop caches To free pagecache dentries and inodes echo 3 proc sys vm drop caches 3 32 RoCE Known Issues Table 41 RoCE Known Issues Index Description Workaround l Not configuring the Ethernet devices or independent VMs with a unique IP address in the physical port may result in RoCE GID table corruption Restart the driver If RDMA CM is not used for connection manage ment then the source and destination GIDs used to modify a QP or create AH should be of the same type IPv4 or IPv6 Mellanox Technologies 29 J Rev 2 3 1 0 1 Known Issues Table 41 RoCE Known Issues Continued Index Description Workaround 3 On rare occasions the driver reports a wrong GID table read from sys class infiniband mlx4_ ports gids This may cause communication problems 4 MLNX OFED v2 1 1 0 0 and onwards is not interoperable with older versions of MLNX OFED 5 Since the number of GIDs per port is limited to 128 there cannot be more than the allowed IP addresses con
20. ay cause overload and software and execute it as root hardware lockups usr bin perl use strict if lt 0 print This script must be run as root n exit 0 open F proc interrupts or die SI my n cat proc cpuinfo grep processor wc 1 chomp n print Spreading over n cpus Mn while F print my Sirqg chan dev if d m1x5_comp d irq chan dev 1 2 m1x5 elsif d m1x4 ib d d Sirg chan dev 1 3 2 else next my place chan n my mask 1 lt lt Splace printf dev irg d chan d bit d mask 0x n Sirg chan Splace mask my cmd sprintf echo 0x gt proc irg Sirg smp_affinity Smask print t cmd n system cmd 3 15 Connection Manager CM Known Issues Table 24 Connection Manager CM Known Issues Index Description Workaround l When 2 different ports have identical GIDs the CM All ports must have different GIDs might send its packets on the wrong port 22 Mellanox Technologies Rev 2 3 1 0 1 3 16 SR IOV Known Isuues Table 25 SR IOV Known Issues Index Description Workaround 1 When using legacy VMs with MLNX_OFED 2 x hypervisor you may need to set the enable 64b cqe eqe parameter to zero on the hypervisor It should be set in the same way that other module parameters ar
21. c MSI X vectors allocation for the SR IOV hypervisor and guest instead of allocating 4 default MSI X vectors The maxi mum number of MSI X vectors is num cpu for port ConnectX 3 has 1024 MSI X vectors 28 MSI X vectors are reserved Physical Function gets the number of MSI X vectors accord ing to the pf msix table size multiple of 4 1 INI parameter e Virtual Functions the remaining MSI X vectors are spread equally between all VFs according to the num v s mlx4 core module parameter Ethernet Ethernet VXLAN support for kernels 3 12 10 or higher Power Management Quality of Service when the traffic 1s active the Power Management QoS is enabled by disabling the CPU states for maximum performance Ethernet PTP Hardware Clock support on kernels OSes that sup port it Verbs Added additional experimental verbs interface This interface exposes new features which are not integrated yet in to the upstream libibverbs The Experimental API is an extended API therefor it is backward compatible meaning old application are not required to be recompiled to use MLNX OFED v2 2 1 0 1 Performance Out of the box performance improvements e Use of affinity hints based on NUMA node of the device to indicate the IRQ balancer daemon on the optimal IRQ affinity Improvement in buffers allocation schema based on the hint above Improvement in the adaptive interrupt moderation algorithm Mellanox Techno
22. d and wr rdma was added The enumerations IBV M WQE CAP CALC RDMA WRITE WITH IMM was added The following enumerations were renamed From IBV M WQE SQ ENABLE CAP to IBV M WOE CAP SO ENABLE From IBV M WQE RQ ENABLE CAP to IBV M WOE CAP RO ENABLE e From IBV M WQE CQE WAIT CAP to IBV M WOE CAP COE WAIT From IBV M WQE CALC CAP to IBV M WQE CAP CALC SEND Mellanox Technologies 41 J
23. d oi odd al owas a 19 Table 16 ABI Compatibility Known Issues ene e eens 19 Table 17 System Time Known Issues 00 ccc cette tence eens 20 Table 18 ConnectX 3 Adapter Cards Family Known Issues 0 000 c eee eee 20 Table 19 Verbs Known Issues 0 00 ect e re 20 Table 20 Resiliency Known Issues 20 Table 21 Driver Start Known Issues 0 0 0 cc eee teen teen nea 21 Table 22 Performance Tools Known Issues 00 00 ccc en 21 Table 23 Performance Known Issues 22 Table 24 Connection Manager CM Known Issues 00 eee cee eh 22 Table 25 SR IOV Known Issues lisse re 23 Table 26 Port Type Management Known Issues 000 e eee eee eee eee eens 24 Table 27 Flow Steering Known Issues 0 0 00 n en ens 25 Table 28 Quality of Service Known Issues 0 00 cc ete en eens 25 Table 29 Installation Known Issues 0 eee ene n tne e ene ees 25 Table 30 Driver Upload Known Issues 0 0 00 0c cect n 26 Table 31 InfiniBand Counters Known Issues 1 0 0 0000 ect ene ens 26 Table 32 UEFI Secure Boot Known Issues 0 00 0 cece ene r e 26 Table 33 Fork Support Known Issues 00 00 ccc cette nent n enna 26 Table 34 ISCSI over IPoIB Known Issues 0 0 00 cette teens 27 Table 35 MLNX OFED Sources Known Issues 00 00 cece cece een teens 27 Mellanox Technologies 5 J Rev 2 3 1 0 1 Table 36 InfiniBand Utilit
24. e RDMA CM API requires IPoIB to first join the broadcast group Mellanox Technologies 15 J Rev 2 3 1 0 1 Known Issues Table 10 IPoIB Known Issues Continued Index Description Workaround 11 Whenever the IOMMU parameter is enabled in the To avoid such issue kernel it can decrease the number of child interfaces e Decrease the amount of the RX receive buf on the device according to resource limitation fers module parameter the default is 512 The driver will stuck after unknown amount of child Decrease the number of RX rings sys fs or interfaces creation ethtool in new kernels e Avoid using IOMMU if not required For further information please see https access redhat com site articles 66747 For KVM users http support citrix com article CTX 136517 Run http www novell com support kb echo 1 gt sys module kvm parame doc php id 7012337 ters allow_unsafe_assigned_interrupts To make this change persist across reboots add the following to the etc modprobe d kvm conf file or create this file if it does not exist options kvm allow_unsafe_assigned_interrupts 1 kernel parameters 12 System might crash in skb_checksum_help Use UD mode in ipoib while performing TCP retransmit involving packets with 64k packet size A similar out to the below will be printed kernel BUG at net core dev c 1707 invalid opcode 0000 1 SMP RIP 0010 lt ffffffff81448988 gt skb_checksum
25. e 36 InfiniBand Utilities Known Issues Index Description Workaround l When running the ibdiagnet check nodes infoon Run ibdiagnet skip nodes info the fabric a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed 3 28 mlx5 Driver Known Issues Table 37 mlx5 Driver Known Issues Index Description Workaround l Atomic Operations in Connect IB are fully sup ported on big endian machines e g PPC Their support 1s limited on little endian machines e g x86 3 29 Ethernet Performance Counters Known Issues Table 38 Ethernet Performance Counters Known Issues Index Description Workaround 1 In a system with more than 61 VFs the 62nd VF and onwards is assigned with the SINKQP counter and as a result will have no statistics and loopback pre vention functionality for SINK counter Mellanox Technologies 27 J Rev 2 3 1 0 1 Known Issues Table 38 Ethernet Performance Counters Known Issues Continued Index Description Workaround 2 Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics in a system with less than 61 VFs if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet In this case RoCE statistics is not available 3 In ConnectX 3 when we enable function based loo
26. e set for mlx4 core at module load time For example add options mlx4 core enable 64b cqe eqe 0 asa line in the file etc modprobe d mlx4 core conf InfiniBand counters are not available in the VM mlx4 portl1 mtu sysfs entry shows a wrong MTU number in the VM When at least one port is configured as InfiniBand and the num vfs is provided but the probe_vf is not HCA initialization fails Use both the num_vfs and the probe vf in the modprobe line When working with a bonding device to enslave the Ethernet devices in active backup mode and failover MAC policy in a Virtual Machine VM establish ment of RoCE connections may fail Unload the module mlx4 1b and reload it in the VM Attaching or detaching a Virtual Function on SLES11 SP3 to a guest Virtual Machine while the mlx4 core driver is loaded in the Virtual Machine may cause a kernel panic in the hypervisor Unload the m1x4 core module in the hypervi sor before attaching or detaching a function to or from the guest When detaching a VF without shutting down the driver from a VM and reattaching it to another VM with the same IP address for the Mellanox NIC RoCE connections will fail Shut down the driver in the VM before detach ing the VF Enabling SR IOV requires appending the intel iommu on option to the relevant OS in file boot grub grub conf Without that SR IOV cannot be loaded On various combinations of Hypervisor O
27. ernal memory allocation improvements Register shared memory e Control objects QPs CQs Installation YUM update support VMA OFED VMA integration to a single branch Mellanox Technologies 37 J Rev 2 3 1 0 1 Change Log History Table 50 Change Log History Release Category Description Storage iSER at Beta level and SRP Operating Systems Errata Kernel upgrade support API VERSION query API library and headers Counters 64bit wide counters port xmit recv data packets unicast mcast a SR IOV Ethernet Time Stamping and Flow Steering are ConnectX 3 HCA capability 38 Mellanox Technologies Rev 2 3 1 0 1 6 API Change Log History Table 51 API Change Log History Release Name Description Rev2 2 1 0 1 libibverbs The following verbs changed to align with upstream libib verbs ibv reg mr ibv access flags changed ibv post send opcodes and send flags changed and wr fields removed task op dc and bind mw ibv query device capability flags changed ibv poll cq opcodes and wc flags changed e ibv modify qp mask bits changed ibv create qp ex create flags field removed The following verbs removed to align with upstream libib verbs ibv bind mw ibv post task ibv query values ex ibv query device ex ibv poll cq ex ibv reg shared mr ex ibv reg shared mr ibv modify cq ibv create cq
28. es 33 J Rev 2 3 1 0 1 Bug Fixes History Table 49 Fixed Bugs List Discovered Fixed in iue D seription in Release Release 15 mlx4 en Fixed ALB bonding mode failure when enslaving 2 0 3 0 0 2 1 1 0 0 Mellanox interfaces 16 Fixed leak of mapped memory 2 0 3 0 0 2 1 1 0 0 17 Fixed TX timeout in Ethernet driver 2 0 2 0 5 2 0 3 0 0 18 Fixed ethtool stats report for Virtual Functions 2 0 2 0 5 2 0 3 0 0 19 Fixed an issue of VLAN traffic over Virtual Machine 2 0 2 0 5 2 0 3 0 0 in paravirtualized mode 20 Fixed ethtool operation crash while interface down 2 0 2 0 5 2 0 3 0 0 21 IPoIB Fixed memory leak in Connected mode 2 0 2 0 5 2 0 3 0 0 22 Fixed an issue causing IPoIB to avoid pkey value 0 2 0 2 0 5 2 0 3 0 0 for child interfaces 34 Mellanox Technologies Rev 2 3 1 0 1 5 Change Log History Table 50 Change Log History Release Category Description Rev 2 2 1 0 1 minxofedinstall 32 bit libraries are no longer installed by default on 64 bit OS To install 32 bit libraries use the with 32bit installation parameter openibd Added pre post start stop scripts support For further information please refer to section openibd Script in the MLNX OFED User Manual Reset Flow Reset Flow is not activated by default It is controlled by the mlx4 core internal err reset module parameter InfiniBand Core Asymmetri
29. et to 16 For further information please refer to the User Manual 5 When using bonding on Ubuntu OS the ifenslave package must be installed 6 On PPC systems the ib srp module is not installed If your system does not require the ibmvscsi by default since it breaks the ibmvscsi module module run the mlnxofedinstall script with the with srp flag Mellanox Technologies 25 J Rev 2 3 1 0 1 Known Issues 3 21 Driver Upload Known Issues Table 30 Driver Upload Known Issues Index Description Workaround l openibd stop can sometime fail with the error Re run openibd stop Unloading ib_cm FAILED ERROR Module ib cm is in use by ib ipoib 3 22 InfiniBand Counters Known Issues Table 31 InfiniBand Counters Known Issues Index Description Workaround l Occasionally port rcv data and port xmit data counters may not function prop erly 3 23 UEFI Secure Boot Known Issues Table 32 UEFI Secure Boot Known Issues Index Description Workaround l On RHEL7 and SLES12 the following error is dis For further information please refer to the played in dmesg if the Mellanox s x 509 Public Key User Manual section Enrolling Mellanox s is not added to the system x 509 Public Key On your Systems 4671958 383506 Request for unknown module key Mellanox Technologies sign ing key 61feb074f c7292 958419386ffdd9d5ca999e4 03 err LI T
30. ex e ibv modify qp ex Verbs Experimental API The following experimental verbs added replacing the removed extended verbs e jbv exp bind mw e jbv exp post task e ibv exp query values e ibv exp query device e ibv exp poll cq e jbv exp reg shared mr e ijbv exp modify cq e ibv exp create cq e ijbv exp modify qp New experimental verbs e ijbv exp arm dct e jbv exp query port e ibv exp create flow e ibv exp destroy flow e ibv exp post send e ijbv exp reg mr e ibv exp get provider func Mellanox Technologies 39 J Rev 2 3 1 0 1 API Change Log History Table 51 API Change Log History Release Name Description Rev 2 1 1 0 0 Dynamically Connected DC The following verbs were added e struct ibv dct ibv exp create dct struct ibv context context struct ibv exp dct init attr attr int ibv exp destroy dct struct ibv dct dct int ibv exp query dct struct ibv dct dct struct ibv exp dct attr attr Verbs Extension API Ibv post task Verbs extension API defines ibv query values ex OFA APIs extension scheme e iby query device ex to detect ABI compatibility Ibv create flow and enable backward and for ibv destroy flow ward compatibility support ibv poll cq ex e ibv reg shared mr ex e ibv open xrcd e ibv close xrcd e ibv modify cq e ibv create srq ex e iby get srq num e ibv create qp ex e jibv create cq ex ibv open qp e jbv modify qp ex Verbs
31. figured to Ethernet devices that are associated with the port Allowed number is e 127 for a single function machine e 15 for a hypervisor in a multifunction machine e 127 15 n for a guest in a multifunction machine where n is the number of virtual func tions 6 A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector Te IPv4 multicast over RoCE requires the MGID for mat to be as follow ffff Multicast IPv4 Address 8 IP routable RoCE does not support Multicast Lis tener Discovery MLD therefore multicast traffic over IPv6 may not work as expected 9 Using GID index 0 the default GID is possible only if the matching IPv6 link local address is con figured on the net device of the port This behavior is possible even though the default GID is config ured regardless the presence of the IPv6 address 10 Using IPv6 link local address GIDO when VLANs are configured is not supported 30 Mellanox Technologies Rev 2 3 1 0 1 3 33 Storage Known Issues Table 42 Storage Known Issues Index Description Workaround l Older versions of rescan_scsi_bus shmay not If encountering such issues it is recommended recognize some newly created LUNs to use the c flag 2 RHEL7 0 The rescan scsi bus sh script does Use older version of the script from not rediscove
32. g Known Issues 3 19 Index Description Workaround l Flow Steering is disabled by default in firmware ver To enable it set the parameter below as follow sion 2 32 5100 log num mgm entry size should set to 1 2 IPv4 rule with source IP cannot be created in SLES 11 x OSes 3 RFS does not support UDP Quality of Service Known Issues Table 28 Quality of Service Known Issues Index Description Workaround l QoS is not supported in XenServer Debian 6 0 and 6 2 with uek kernel 2 When QoS features are not supported by the kernel mlnx qos tool may crash 3 20 Installation Known Issues Table 29 Installation Known Issues Index Description Workaround l When upgrading from an earlier Mellanox OFED Stop the old OFED stack etc init d version the installation script does not stop the ear openibd stop before upgrading to this new lier version prior to uninstalling it version 2 Upgrading from the previous OFED installation to Reboot after installing the driver this release does not unload the kernel module Ipoib helper 3 Installation using Yum does not update HCA firm See Updating Firmware After Installation in ware OFED User Manual 4 total vfs lt 0 63 gt installation parameter is Use enable sriov installation parame no longer supported ter to burn firmware with SR IOV support The number of virtual functions VFs will be s
33. his error can be safely ignored as long as Secure Boot is disabled on the system 2 Ubuntu12 requires update of user space open iscsi to v2 0 873 3 The initiator does not respect interface parameter Configure each interface on a different subnet while logging in 3 24 Fork Support Known Issues Table 33 Fork Support Known Issues Index Description Workaround l Fork support from kernel 2 6 12 and above is avail able provided that applications do not use threads fork is supported as long as the parent process does not run before the child exits or calls exec The former can be achieved by calling wait childpid and the latter can be achieved by application specific means The Posix system call is supported 26 Mellanox Technologies Rev 2 3 1 0 1 3 25 ISCSI over IPoIB Known Issues Table 34 ISCSI over IPoIB Known Issues Index Description Workaround l When working with ISCSI over IPoIB LRO must be disabled even if IPoIB is set to connected mode due to a a bug in older kernels which causes a kernel panic 3 26 MLNX OFED Sources Known Issues Table 35 MLNX OFED Sources Known Issues Index Description Workaround 1 MLNX_OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox propri etary packages 3 27 InfiniBand Utilities Known Issues Tabl
34. ies Known Issues 0 0 ccc cect ee 27 Table 37 mlx5 Driver Known Issues 0 00 ccc cece n 2 Table 38 Ethernet Performance Counters Known Issues eee eee 27 Table 39 Uplinks Known Issues 0 0 0 0 ccc e 28 Table 40 Resources Limitation Known Issues sssseeeeeee eh 29 Table 41 RoCE Known Issues 0 00 ea 29 Table 42 Storage Known Issues llis 31 Table43 SRP Known lssuesz co direc ep Saas kee Sages gare Oe Raa Series 31 Table 44 SRP Interop Known Issues 0 31 Table 45 DDN Storage Fusion 10000 Target Known Issues 0 000 e eee eee eee 31 Table 46 Oracle Sun ZFS storage 7420 Known Issues 0000 cece eee eh 31 Table 47 iSER Knownlssues llle 32 Table 48 ZFS Appliance Known Issues 0 0 0 0 0c eee en ene eens 32 Table497 Fixed Bugsilist on bte ese re day eR a Y are a ee ae eR UR 33 Table 50 Change Log History 20e ooi er UR PANE ELE PARCO ACE I ERATES 35 Table 51 API Change LogHistory sese 39 6 Mellanox Technologies J Rev 2 3 1 0 1 Release Update History Table 1 Release Update History Release Date Description 2 3 1 0 1 19 October 2014 Added Section 1 7 RoCE Moded Matrix on page 12 23 September 2014 Updated the Known Issue section Added SR IOV and Resources Limitation Known Issues September 2014 Initial version Mellanox Technologies 7 J Rev 2 3 1 0 1 Overview
35. ion option CONFIG MLX4 EN VXLAN y must be enabled MLNX OFED no longer changes the OS sysctl TCP parameters Added Explicit Congestion Notification ECN support Added Flow Steering A0 simplified steering support Added RoCE v2 support Mellanox Technologies 13 J Rev 2 3 1 0 1 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 Table 8 Changes in v2 3 1 0 1 Category Description InfiniBand Network Added Secure host to enable the device to protect itself and the subnet from mali cious software Added User Mode Memory Registration UMR to enable the usage of RDMA operations and to scatter the data at the remote side through the definition of appro priate memory keys on the remote side Added On Demand Paging ODP a technique to alleviate much of the shortcom ings of memory registration Added Masked Atomics operation support Added Checksum offload for packets without L4 header support Added Memory re registration to allow the user to change attributes of the mem ory region Resiliency Added Reset Flow for ConnectX 3 SR IOV support SR IOV Added Virtual Guest Tagging VGT an advanced mode of Virtual Guest Tag ging VGT in which a VF is allowed to tag its own packets as in VGT but is still subject to an administrative VLAN trunk policy Ethtool Added Cable EEPROM reporting support Disable Enable ethernet RX VLAN tag striping offload via ethtool
36. logies 35 J Rev 2 3 1 0 1 Change Log History Table 50 Change Log History Release Category Description Rev 2 1 1 0 6 IB Core Added allocation success verification process to ib_alloc_device dapl dapl is recompiled with no FCA support openibd Added the ability to bring up child interfaces even if the parent s ifcfg file is not configured libmlx4 Unmapped the hca clock page parameter from mlx4 uninit context scsi transport srp scsi transport srp cannot be cleared up when rport reconnecting fails minxofedinstall Added support for the following parameters e umad dev na e without package Content Packages The following packages were updated Updates bupc to v2 2 407 mstflint to v3 5 0 1 1 g76e4acf perftest to v2 0 0 76 gbf9a463 e hcoll to v2 0 472 1 Openmpi to v1 6 5 440ad47 e daplto v2 0 40 Rev 2 1 1 0 0 EoIB EoIB is supported only in SLES11SP2 and RHEL6 4 eIPoIB eIPoIB is currently at GA level Connect IB Added the ability to resize CQs IPoIB Reusing DMA mapped SKB buffers Performance improvements when IOMMU is enabled minx_en Added reporting autonegotiation support Added Transmit Packet Steering XPS support Added reporting 56Gbit s link speed support Added Low Latency Socket LLS support Added check for dma_mapping errors eIPoIB Added non virtual environment support 36 Mellanox Technologie
37. multi threaded applications 3 11 Resiliency Known Issues Table 20 Resiliency Known Issues Index Description Workaround 1 Reset Flow can run on XenServer 6 only after the active user space applications running verbs are ter minated 2 SR IOV non persistent configuration such as VGT Reset Admin configuration post Reset Flow VST Host assigned GUIDs and QPO enabled VFs may be lost upon Reset Flow 3 Upon Reset Flow or after running restart driver Reset the VLANs using the ifup command Ethernet VLANs are lost 4 Restarting the driver or running connectx port config when Reset Flow is run ning might result in a kernel panic 5 Networking configuration e g VLANs IPv6 should be statically defined in order to have them set after Reset Flow as of after restart driver 20 Mellanox Technologies Rev 2 3 1 0 1 3 12 Driver Start Known Issues Table 21 Driver Start Known Issues Index Description Workaround l Out of memory issues may rise during drivers load depending on the values of the driver module parameters set e g log num cq 2 When reloading starting the driver using the etc Remove the third party RPM non init d openibd the following messages are dis MLNX OFED drivers directory run played if there is a third party RPM or driver depmod and then rerun etc init d installed openibd restart Module mlx4 core does not belong to
38. ns Table 5 Supported HCAs Firmware Versions HCA Recommended Firmware Rev Additional Firmware Rev Supported Connect IB Rev 10 10 4020 Rev 10 10 3000 ConnectX 3 Pro Rev 2 32 5100 Rev 2 31 5050 ConnectX 3 Rev 2 32 5100 Rev 2 31 5050 ConnectX 2 Rev 2 9 1000 Rev 2 9 1000 For official firmware versions please see http www mellanox com content pages php pg firmware download Mellanox Technologies 11 J Rev 2 3 1 0 1 Overview 1 6 Compatibility MLNX_OFED Rev 2 3 1 0 1 is compatible with the following Table 6 MLNX_OFED Rev 2 3 1 0 1 Compatibility Matrix Mellanox Product Description Version MLNX OS MSX6036 w w MLNX OS version 3 3 4304 Grid Director 4036 w w Grid Director version 3 9 1 985 FabricIT EFM 1S5035 w w FabricIT EFM version 1 1 3000 FabricIT BXM MBX5020 w w FabricIT BXM version 2 1 2000 Unified Fabric Manager UFM v4 8 MXM v3 2 HPC X UPC v2 18 0 HPC X OpenSHMEM v1 8 3 FCA v2 5 and v3 1 HPC X MPI v1 8 3 MVAPICH v2 0 a MLNX OFED v2 3 1 0 1 was tested with this switch however additional switches might be supported as well 1 7 RoCE Modes Matrix The following is RoCE modes matrix Table 7 RoCE Modes Matrix Software Stack RoCE IP Based Layer 2 RoCEv2 Layer 3 Inbox Distribution Supported as of Version Supported as of Version MLNX OFED 2 1 x x x 2 3 X x x Kernel org 3 14 RHEL 6
39. on Mellanox HCAs 3 XRC options do not work when using qperf tool Use perftest instead 4 Out of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created XRC QPs do not have receive queues 3 7 ABI Compatibility Known Issues Table 16 ABI Compatibility Known Issues Index Description Workaround 1 MLNX_OFED Rev 2 3 1 0 1 is not ABI compatible Recompile the application over the new with previous MLNX_OFED OFED versions MLNX_OFED version Mellanox Technologies 19 J Rev 2 3 1 0 1 Known Issues 3 8 System Time Known Issues Table 17 System Time Known Issues Index Description Workaround l Loading the driver using the openibd script when no InfiniBand vendor module is selected for example mlx4_ib may cause the execution of the sbin start_udev script In RedHat 6 x and OEL6 x this may change the local system time 3 9 ConnectX 3 Adapter Cards Family Known Issues Table 18 ConnectX 3 Adapter Cards Family Known Issues Index Description Workaround l Using RDMA READ with a higher value than 30 Do not set the value of SGEs higher than 30 SGEs in the WR might lead to local length error when RDMA READ is used 3 10 Verbs Known Issues Table 19 Verbs Known Issues Index Description Workaround l Using libnll 1 3 26 or earlier requires Ibv create ah protection by a lock for
40. op _mod 0x1 failed with error 0 status 28 Kernel panic might occur during fio splice in kernels before 2 6 34 rc4 Use kernel v2 6 34 rc4 which provides the following solution baff42a net oops from tcp collapse when using splice Fix In PPC systems when QoS is enabled a harmless Kernel DMA mapping error messages might appear in kernel log iommu related issue Transmit timeout might occur on RH6 3 as a result of lost interrupt OS issue In this case the follow ing message will be shown in dmesg do IRQ 0 203 No irq handler for vector irq 1 The default priority to TC mapping assigns all prior ities to TCO This configuration achieves fairness in transmission between priorities but may cause undesirable PFC behavior where pause request for priority n affects all other priorities Run mlnx qos i dev p 0 1 2 3 4 5 6 7 s ets ets ets ets ets ets ets ets t 12 13 12 13 12 13 12 13 This needs to be applied every time after load ing the m1x4 en driver Mixing ETS and strict QoS policies for TCs in 40GbE ports may cause inaccurate results in band width division among TCs Creating a VLAN with user priority gt 4 on ConnectX 2 HCA is not supported Affinity hints are not supported in Xen Hypervisor an irqblancer issue This causes a non optimal IRQ affinity To overcome this issues run set_irq_affinity sh eth lt x gt Mellanox Technologies 17 J
41. owever only the test fails OpenSM will operate correctly with the host The failure reason is that if an mcg is already joined by the host a subse quent join request for that group succeeds automati cally even if the join parameters in the request are not correct This success does no harm 3 17 Port Type Management Known Issues Table 26 Port Type Management Known Issues Index Description Workaround l OpenSM must be stopped prior to changing the port protocol from InfiniBand to Ethernet 2 After changing port type using Use udev rules for persistent naming configu connectx port config interface ports names ration can be changed For example ibl gt ib0 if port For further information please refer to the changed to be Ethernet port and port2 left IB User Manual 3 A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector 4 IPv4 multicast over RoCE requires the MGID for mat to be as follow ffff Multicast IPv4 Address 5 IP routable RoCE does not support Multicast Lis tener Discovery MLD therefore multicast traffic over IPv6 may not work as expected 6 DIF When running IO over FS over DM during unstable ports block layer BIOs merges may cause false DIF error 24 Mellanox Technologies Rev 2 3 1 0 1 3 18 Flow Steering Known Issues Table 27 Flow Steerin
42. pback prevention for Ethernet port by default i e based on the QP counter index the dropped self loopback packets increase the IfRxErrorFrames Octets counters 3 30 Uplinks Known Issues Table 39 Uplinks Known Issues Index Description Workaround l On rare occasions ConnectX 3 Pro adapter card Restart the driver may fail to link up when performing parallel detect to 40GbE 28 Mellanox Technologies Rev 2 3 1 0 1 3 31 Resources Limitation Known Issues Table 40 Resources Limitation Known Issues Index Description Workaround 1 The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace mlx4 core can allocate up to 64 MSI X vectors an MSI X vector per CPU Setting more IP addresses than the available GID entries in the table results in failure and the update gid table error message is displayed GID table of port 1 is full Can t add address message Registering a large amount of Memory Regions MR may fail because of DMA mapping issues on RHEL 7 0 Occasionally a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system s free mem ory for its internal cache To free memory to allow it to be allocated in a user process run the drop c
43. r provisioned LUNs both on iSER and RHEL6 4 SRP e iSER Use iscsiadm m session rescan 3 34 SRP Known Issues Table 43 SRP Known Issues Index Description Workaround 1 In a high stress IO with unstable links SRP Initiator may generate a call trace that can be safely ignored 2 MLNX_OFED SRP installation breaks the ibmvstgt and ibmvscsi symbol resolution in RHEL7 0 3 35 SRP Interop Known Issues Table 44 SRP Interop Known Issues Index Description Workaround l The driver is tested with Storage target vendors rec ommendations for multipath conf extensions ZFS DDN TMS Nimbus NetApp 3 36 DDN Storage Fusion 10000 Target Known Issues Table 45 DDN Storage Fusion 10000 Target Known Issues Index Description Workaround l DDN does not accept non default P Key connection establishment 3 37 Oracle Sun ZFS storage 7420 Known Issues Table 46 Oracle Sun ZFS storage 7420 Known Issues Index Description Workaround l Occasionally the first command to a LUN may not be serviced aborted and cause a successful re con nection to the target 2 Ungraceful power cycle of an initiator connected with Targets DDN Nimbus NetApp may result in temporary stale connection messages when initia tor reconnects Mellanox Technologies 31 J Rev 2 3 1 0 1 Known Issues 3 38 iSER Known Issues Table 47 iSE
44. s Rev 2 3 1 0 1 Table 50 Change Log History Release Category Description Rev 2 0 3 0 0 Operating Systems Additional OS support e SLES11SP3 Fedoral6 Fedoral7 Drivers Added Connect IB support Installation Added ability to install MLNX_OFED with SR IOV support Added Yum installation support EoIB EoIB at beta level is supported only in SLES11SP2 and RHEL6 4 mlx4 core Modified module parameters to associate configuration values with specific PCI devices identified by their bus device function value format mlx4 en Reusing DMA mapped buffers major performance improvements when IOMMU is enabled Added Port level QoS support IPoIB Reduced memory consumption Limited the number TX and RX queues to 16 Default IPoIB mode is set to work in Datagram except for Con nect IBTM adapter card which uses IPoIB with Connected mode as default Storage ISER at GA level Rev 2 0 2 0 52 Virtualization SR IOV for both Ethernet and InfiniBand at Beta level Ethernet Network RoCE over SR IOV at Beta level eIPoIB to enable IPoIB in a Para Virtualized environment at Alpha level Ethernet Performance Enhancements NUMA related and others for 10G and 40G Ethernet Time Stamping at Beta level Flow Steering for Ethernet and InfiniBand at Beta level Raw Eth QPs e Checksum TX RX Flow Steering InfiniBand Net Contiguous pages work Int
45. se l IPoIB Changing the GUID ofa specific SR IOV guest after 2 1 1 0 0 2 3 1 0 1 the driver has been started causes the ping to fail Hence no traffic can go over that InfiniBand inter face 2 Ethernet Fixed kernel panic on Debian 6 0 7 which occurred 2 1 1 0 0 2 2 1 0 1 when the number of TX channels was set above the default value 3 Fixed a crash incidence which occurred when 2 0 2 0 5 22 1 0 1 enabling Ethernet Time stamping and running VLAN traffic 4 XRC XRC over ROCE in SR IOV mode is not functional 2 0 3 1 0 2 2 1 0 1 5 mlx4 en Fixed wrong calculation of packet true size reporting 2 1 1 0 0 2 2 1 0 1 in LRO flow 6 IB Core Fixed the QP attribute mask upon smac resolving 2 1 1 0 0 2 1 1 0 6 7 mlx5 ib Fixed a send WQE overhead issue 2 1 1 0 0 2 1 1 0 6 8 Fixed a NULL pointer dereference on the debug 2 1 1 0 0 2 1 1 0 6 print 9 Fixed arguments to kzalloc 2 1 1 0 0 2 1 1 0 6 10 mlx4 core Fixed the locks around completion handler 2 1 1 0 0 2 1 1 0 6 11 mlx4 core Restored port types as they were when recovering 2 0 2 0 5 2 1 1 0 0 from an internal error 12 Added an N A port type to support port type array 2 0 2 0 5 2 1 1 0 0 module param in an HCA with a single port 13 SR IOV Fixed memory leak in SR IOV flow 2 0 2 0 5 2 0 3 0 0 14 Fixed communication channel being stuck 2 0 2 0 5 2 0 3 0 0 Mellanox Technologi
46. sues 00 0 0 ent nee e eens 32 3 39 ZFS Appliance Known Issues en 32 Chapter 4 Bug Fixes HIstOry lt a costes sacs e Ser easier eh 9 n PRONTI ROT RERO SpA TR 33 Chapter 5 Change Log History cc cece ccc cece ee ee eee cece nnn 35 Chapter 6 API Change Log History ccc ccc ccc e eee w cere e eee eees 39 4 Mellanox Technologies J Rev 2 3 1 0 1 List Of Tables Table 1 Release Update History 0 0 etc nent s 7 Table 2 Mellanox OFED for Linux Software Components 0 0 cece eee eens 8 Table 3 Supported Platforms and Operating Systems 0 0 cece cece een ene 9 Table 4 Additional Software Packages 0 0 ccc cece eh 11 Table 5 Supported HCAs Firmware Versions lessseeeee e 11 Table 6 MLNX OFED Rev 2 3 1 0 1 Compatibility Matrix noana anaana eee 12 Table 7z ROGE Modes Matii iure TELECOM EV A E EE EAE dan Bae 12 Table 8 Changesam V231 0I ekon eus eee Tec e UR ERN ELS EM 13 Table9 API Change Log History 1 0 0 00 cece eet n ene r int 14 Table 10 IPoIB Known Issues 0 00 0c ee en 15 Table 11 Ethernet Known Issues 0 00 ccc cette hn 17 Table 12 General Known Issues 1 0 0 0 0c eee nen en ene e nen 18 Table 13 lt VGI Known ISSUES ei o Vaden whee Y arent aye RAI ae Vane D Dane Boe Senne Ags 18 Table I4 el PoIB Known lssuess es ux e RRCEEIR wa NE REDE A at HANDED Ua ig 18 Tabled5 XRC Knowjlssuesi ielise ce ehe eee M ed wee

Mellanox OFED for Linux Release Notes

Contents

Download Pdf Manuals

Related Search

Related Contents