Mellanox OFED for Linux Release Notes
Contents
1. 26 Table 28 Installation Known Issues 00 0 cece m 26 Table 29 Driver Upload Known Issues 5 27 Table 30 InfiniBand Counters Known Issues 1 0 0 0 0 00 ee 27 Table 31 UEFI Secure Boot Known Issues 27 Table 32 Fork Support Known Issues 0000 ccc cece tenet ene 27 Table 33 ISCSI over IPoIB Known 1 5 28 Table 34 MLNX OFED Sources Known 155 28 Table 35 InfiniBand Utilities Known Issues 28 Mellanox Technologies 5 J Rev 2 3 1 0 1 Table 36 mlx5 Driver Known Issues ireccio nen teen S 28 Table 37 Ethernet Performance Counters Known 155 28 Table 38 Uplinks Known Issues 0 0 0 0 ccc e an 29 Table 39 Resources Limitation Known 158 30 Table 40 1 e 30 Table 41 Storage Known 155 32 Table42 SRP Known ISSUES 2 24 eme ie ee arth Maan anise BAG See e Se la area aA 32 Table 43 SRP Interop Known Issues teens 32 Table 44 DDN Storage Fusion 10000 Target Known 1582. 9 37 Oracle Sun ZFS storage 7420 Known Issues Table 45 Oracle Sun ZFS storage 7420 Known Issues Index Description Workaround l Occasionally the first command to a LUN may not be serviced aborted and cause a successful re con nection to the target 2 Ungraceful power cycle of an initiator connected with Targets DDN Nimbus NetApp may result in temporary stale connection messages when initia tor reconnects 32 Mellanox Technologies Rev 2 3 1 0 1 9 38 iSER Known Issues Table 46 iSER Known Issues Index Description Workaround 1 SM LID reassignmet during traffic OEL6 4 uek kernel with a Virtual Function generates soft lockup trace 2 On SLES OSs the iser module does not load Add a dummy interface using iscsiadm on boot e iscsiadm m iface I ib iser o new e d iscsiadm m iface I ib iser o update n iface transport name v ib iser 3 DIF When running IO over FS over DM during unstable ports block layer BIOs merges may cause false DIF error 4 Ubuntul2 requires update of user space open iscsi to v2 0 873 5 The initiator does not respect interface parameter Configure each interface on a different subnet while logging in 6 ISCSID v2 0 873 can enter an endless loop on bind error 7 DIX Under heavy IO stress with large block size Disable block merges the HCA card might generate error Completions in the
3. 1 4 core module in the hypervi sor before attaching or detaching a function to or from the guest When detaching a VF without shutting down the driver from a VM and reattaching it to another VM with the same IP address for the Mellanox NIC RoCE connections will fail Shut down the driver in the VM before detach ing the VF Enabling SR IOV requires appending the intel iommu on option to the relevant OS in file boot grub grub conf Without that SR IOV cannot be loaded On various combinations of Hypervisor OSes and Guest OSes an issue might occur when attaching detaching VFs to a guest while that guest is up and running Attach detach VFs to from a VM only while that VM is down When working with SR IOV in Xen 4 2 virtualiza tion platform only the built in xen pciback driver should be loaded The xen pciback module in dom0 should not be loaded as loading them simultaneously may cause interrupts loss and cause the driver to enter the reset flow 24 Mellanox Technologies Rev 2 3 1 0 1 Table 24 SR IOV Known Issues Continued Index Description Workaround 11 The known PCI BDFs for all VFs in kernel com mand line should be specified by adding xen pci back hide For further information please refer to http wiki xen org wiki Xen_PCI Passthrough 12 The qemu version 2 0 provided inbox with Ubuntu 14 04 does not work properly when more than 2 VMs are run over
4. Chapter 4 Supported Platforms and Operating 10 4 1 Supported Hypervisors 0 III 11 4 2 Supported Non Linux Virtual 11 Chapter 5 Hardware and Software 12 Chapter 6 Supported HCAs Firmware 12 Chapter 7 Compatibility er eR NCC hon m n MR RR Re RC RR RR 13 Chapter 8 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 14 8 1 API Changes in MLNX OFED Rev 2 3 1 0 1 15 Chapter 9 Known 16 9 1 JPoIB Known Issues i erates eis an ELE TUR ak web ee ah 16 9 2 Ethernet Known Issues a 18 9 3 General Known 1 19 94 VOTE Known 15906830 bte rene pee we SA bee ed Laer Soe 19 9 5 eIPoIB Known Issues eee 19 9 6 XRC Known a dra 20 9 7 ABI Compatibility Known 1 20 9 8 System Time Known 1 5 21 9 9 ConnectX 3 Adapter Cards Family Known 1550 5 21 9 10 Verbs Known ISSUES S usce eene oth e ee EE Re RR a 21 9 11 Resiliency Known Issues 21 9 12 Driver Start Known I
5. erly 9 23 UEFI Secure Boot Known Issues Table 31 UEFI Secure Boot Known Issues Index Description Workaround l On RHEL7 and SLES12 the following error is dis For further information please refer to the played in dmesg if the Mellanox s x 509 Public Key User Manual section Enrolling Mellanox s is not added to the system x 509 Public Key On your Systems 4671958 383506 Request for unknown module key Mellanox Technologies sign ing key 61feb074f c7292 958419386ffdd9d5ca999e4 03 err LI This error can be safely ignored as long as Secure Boot is disabled on the system 2 Ubuntu12 requires update of user space open iscsi to v2 0 873 3 The initiator does not respect interface parameter Configure each interface on a different subnet while logging in 9 24 Fork Support Known Issues Table 32 Fork Support Known Issues Index Description Workaround l Fork support from kernel 2 6 12 and above is avail able provided that applications do not use threads fork is supported as long as the parent process does not run before the child exits or calls exec The former can be achieved by calling wait childpid and the latter can be achieved by application specific means The Posix system call is supported Mellanox Technologies 27 J Rev 2 3 1 0 1 Known Issues 9 25 ISCSI over IPoIB Known Issues Table 33 ISCSI over IPoIB Known Issues
6. 0 EoIB EoIB is supported only in SLES11SP2 and RHEL6 4 eIPoIB eIPoIB is currently at GA level Connect IB Added the ability to resize CQs IPoIB Reusing DMA mapped SKB buffers Performance improvements when IOMMU is enabled Added reporting autonegotiation support Added Transmit Packet Steering XPS support Added reporting 56Gbit s link speed support Added Low Latency Socket LLS support Added check for dma_mapping errors eIPoIB Added non virtual environment support Mellanox Technologies 37 J Rev 2 3 1 0 1 Change Log History Table 49 Change Log History Release Category Description Rev 2 0 3 0 0 Operating Systems Additional OS support e SLES11SP3 Fedoral6 Fedoral7 Drivers Added Connect IB support Installation Added ability to install MLNX_OFED with SR IOV support Added Yum installation support EoIB EoIB at beta level is supported only in SLES11SP2 and RHEL6 4 mlx4 core Modified module parameters to associate configuration values with specific PCI devices identified by their bus device function value format mlx4 en Reusing DMA mapped buffers major performance improvements when IOMMU is enabled Added Port level QoS support IPoIB Reduced memory consumption Limited the number TX and RX queues to 16 Default IPoIB mode is set to work in Datagram except for Con nect IBTM adapter ca
7. 3 Adapter Cards Family Known Issues Table 17 ConnectX 3 Adapter Cards Family Known Issues Index Description Workaround 1 Using READ with a higher value than 30 Do not set the value of SGEs higher than 30 SGEs in the WR might lead to local length error when READ is used 9 10 Verbs Known Issues Table 18 Verbs Known Issues Index Description Workaround l Using libnll 1 3 26 or earlier requires 16 create ah protection by a lock for multi threaded applications 9 11 Resiliency Known Issues Table 19 Resiliency Known Issues Index Description Workaround 1 Reset Flow can run on XenServer 6 only after the active user space applications running verbs are ter minated 2 SR IOV non persistent configuration such as VGT Reset Admin configuration post Reset Flow VST Host assigned GUIDs and QPO enabled VFs may be lost upon Reset Flow 3 Upon Reset Flow or after running restart driver Reset the VLANs using the ifup command Ethernet VLANs are lost 4 Restarting the driver or running connectx port config when Reset Flow is run ning might result in a kernel panic 5 Networking configuration e g VLANs IPv6 should be statically defined in order to have them set after Reset Flow as of after restart driver Mellanox Technologies 21 J Rev 2 3 1 0 1 Known Issues 9 12 Driver Start Known Issues Table 20 D
8. 32 Table 45 Oracle Sun ZFS storage 7420 Known Issues 32 Table 46 iSER Known Issues 0 0 0 0 00 ec hh 33 Table 47 ZFS Appliance Known Issues 0 0 0 0 00 ccc cette en ene eas 33 Table 48 Fixed Bugs Lists i use Rex een DE EO EO ERS NEM SR ee RID 34 Table 497 Change ogiHistorys i5 cues e ay Se Nee eR C D alae el weed aa eae anes 36 Table 50 API Change Log History ene E 40 6 Mellanox Technologies Rev 2 3 1 0 1 Release Update History Table 1 Release Update History Release Date Description 2 3 1 0 1 23 September 2014 Updated the Known Issue section Added SR IOV and Resources Limitation Known Issues September 2014 Initial version Mellanox Technologies 7 Rev 2 3 1 0 1 Overview 1 Overview These are the release notes of Mellanox OFED for Linux Driver Rev 2 3 1 0 1 Mellanox OFED is a single Virtual Protocol Interconnect VPI software stack and operates across all Mellanox network adapter solutions supporting the following uplinks to servers 10 20 40 and 56 Gb s InfiniBand IB 10 40 and 56 Gb s Ethernet 2 5 or 5 0 GT s PCI Express 2 0 8GT s PCI Express 3 0 2 Main Features in This Release MLNX OFED Rev 2 3 1 0 1 provides the following new features Secure Host Virtual Guest Tagging VGT e User Mode Memory Registration UMR Masked Atomics On Dem
9. allow the user to change attributes of the mem ory region Resiliency Added Reset Flow for ConnectX 3 SR IOV support SR IOV Added Virtual Guest Tagging VGT an advanced mode of Virtual Guest Tag ging VGT in which a VF is allowed to tag its own packets as in VGT but is still subject to an administrative VLAN trunk policy Ethtool Added Cable EEPROM reporting support Disable Enable ethernet RX VLAN tag striping offload via ethtool 128 Byte Completion Queue Entry CQE Non Linux Virtual Added Windows Virtual Machine over Linux KVM Hypervisor SR IOV with Machines InfiniBand only support 8 1 Changes in MLNX OFED Rev 2 3 1 0 1 The following are the API changes in MLNX OFED Rev 2 3 1 0 1 Table 8 API Change Log History Release Name Description Rev2 3 1 0 1 libibverbs e ibv exp rereg mr Added new API for memory region re reintegration For futher information please refer to MLNX OFED User Manual e Added to the experimental API exp post send the following opcodes e IBV EXP WR EXT MASKED ATOMIC CMP AND SWP e IBV EXP WR EXT MASKED ATOMIC FETCH AND ADD e IBV EXP WR and these completion opcodes e IBV EXP WC MASKED COMP SWAP e IBV EXP WC MASKED FETCH ADD Mellanox Technologies 15 J Rev 2 3 1 0 1 Known Issues 9 Known Issues The following is a list of general limitations and known issues of the various components of
10. glib2 tcl bc lib stde tk SLES 11 SP2 zypper install perl pciutils python libnl 32bit libxml2 python tesh libnl lib stdc 46 expat glib2 tcl bc tklibcurl4 gtk2 atk cairo SLES 11 SP3 zypper install perl pciutils python libnl 32bit libxml2 python tesh lib 501 43 libnl expat glib2 tcl be tk libcurl4 gtk2 atk cairo Ubuntu Debian apt get install perl dpkg autotools dev autoconf libtool automakel1 10 auto make m4 dkms debhelper tcl tcl8 4 chrpath swig graphviz tcl dev tcl8 4 dev tk dev tk8 4 dev bison flex dpatch zlib g dev curl libcurl4 gnutls dev python libxml2 libvirt bin libvirtO libnl dev libglib2 0 dev libgfortran3 auto make m4 6 Supported HCAs Firmware Versions MLNX OFED Rev 2 3 1 0 1 supports the following Mellanox network adapter cards firmware versions Table 5 Supported HCAs Firmware Versions HCA Recommended Firmware Rev Additional Firmware Rev Supported Connect IB Rev 10 10 4020 Rev 10 10 3000 ConnectX 3 Pro Rev 2 32 5100 Rev 2 31 5050 ConnectX 3 Rev 2 32 5100 Rev 2 31 5050 ConnectX 2 Rev 2 9 1000 Rev 2 9 1000 For official firmware versions please see http www mellanox com content pages php pg firmware download 12 Mellanox Technologies Rev 2 3 1 0 1 7 Compatibility MLNX_OFED Rev 2 3 1 0 1 is compatible with the following Table 6 MLNX_OFED Rev 2 3 1 0 1 Compatibility Matrix Mellanox Product D
11. should be of the same type IPv4 or IPv6 30 Mellanox Technologies Rev 2 3 1 0 1 Table 40 Known Issues Continued Index Description Workaround 3 On rare occasions the driver reports a wrong GID table read from sys class infiniband mlx4_ ports gids This may cause communication problems 4 MLNX OFED v2 1 1 0 0 and onwards is not interoperable with older versions of MLNX OFED 5 Since the number of GIDs per port is limited to 128 there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port Allowed number is e 127 for a single function machine e 15 for a hypervisor in a multifunction machine e 127 15 n for a guest in a multifunction machine where n is the number of virtual func tions 6 A working IP connectivity between the RoCE devices is required when creating an address handle or modifying a QP with an address vector Te IPv4 multicast over requires the MGID for mat to be as follow ffff Multicast IPv4 Address 8 IP routable RoCE does not support Multicast Lis tener Discovery MLD therefore multicast traffic over IPv6 may not work as expected 9 Using GID index 0 the default GID is possible only if the matching IPv6 link local address is con figured on the net device of the port This behavior is possible even though the default GID is config ured
12. this Mellanox OFED for Linux release 9 1 IPoIB Known Issues Table 9 IPoIB Known Issues Index Description Workaround 1 When user increases receive send a buffer it might consume all the memory when few child s interfaces are created The size of send queue in Connect IB cards cannot exceed IK In 32 bit devices the maximum number of child interfaces that can be created is 16 Creating more that might cause out of memory issues In RHEL7 0 the Network Manager can detect when the carrier of one of the IPoIB interfaces is OFF and can decide to disable its IP address Set ignore carrier for the corresponding device in NetworkManager conf For further information please refer to man NetworkManager conf IPoIB interface does not function properly if a third party application changes the PKey table We recommend modifying PKey tables via OpenSM Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring https bugs open fabrics org show_bug cgi id 1990 Out of memory issue might occur due to overload of interfaces created To calculate the allowed memory per each IPoIB interface check the following e Num rings min num cores on that device 16 Ring size 512 by default it is module parameter UD memory 2 num rings ring size 8K CM memory ring size 64k Total memory UD mem CM mem Connect IB does not reach
13. to be applied every time after load ing the m1x4 en driver Mixing ETS and strict QoS policies for TCs in 40GbE ports may cause inaccurate results in band width division among TCs Creating a VLAN with user priority gt 4 on ConnectX 2 HCA is not supported Affinity hints are not supported in Xen Hypervisor an irqblancer issue This causes a non optimal IRQ affinity To overcome this issues run set_irq_affinity sh eth lt x gt 18 Mellanox Technologies Rev 2 3 1 0 1 9 3 General Known Issues Table 11 General Known Issues Index Description Workaround 1 On ConnectX 2 ConnectX 3 Ethernet adapter cards N A there is a mismatch between the GUID value Please use the GUID value returned by the fab returned by firmware management tools and that ric driver utilities not Oxfffff returned by fabric driver utilities that read the GUID via device firmware e g using ibstat MIxburn flint return as GUID while the utilities return a value derived from the MAC address For all driver firmware software purposes the latter value should be used 9 4 VGT Known Issues Table 12 VGT Known Issues Index Description Workaround l Before adding a VLAN on the VM the parent inter face should be brought up Otherwise VLAN cre ation will fail and the following message will be presented Fail to register network rule 2 Bringing down and up the p
14. 0 API Change Log History Release Name Description Rev 2 0 2 0 5 Libibverbs Extended speeds Missing the ext active speed attribute from the struct ibv port attr e Removed function ext rate to int e Added functions rate to mbps and mbps to ibv rate Libibverbs Raw QPs QP types IBV RAW PACKET and IBV_QPT RAW ETH are not supported Libibverbs Contiguous e Added Contiguous pages support pages e Added function shared mr Libmverbs The enumeration IBV M WR CALC was renamed to IBV M WR CALC SEND Theenumeration IBV M WR WRITE WITH IMM was added Inthe structure m send union wr send was renamed to wr calc send and wr rdma was added The enumerations IBV M WQE CAP CALC RDMA WRITE WITH IMM was added The following enumerations were renamed From IBV M WQE SQ ENABLE CAP to IBV M WOE CAP SO ENABLE From IBV M WQE RQ ENABLE CAP to IBV M WOE CAP RO ENABLE e From IBV WQE WAIT CAP to IBV M WOE CAP COE WAIT From IBV M WQE CALC CAP to IBV M WQE CAP CALC SEND 42 Mellanox Technologies
15. 5 The LRO feature cannot be disabled via ethtool on kernels gt 3 9 9 2 Ethernet Known Issues Table 10 Ethernet Known Issues Index Description Workaround 1 When creating more than 125 VLANs SR IOV mode is enabled a kernel warning message will be printed indicating that the native VLAN is created but will not work with RoCE traffic kernel warning 1 4 0000 07 00 0 vher command ALLOC RES 0 00 slave 0 in param 0 7 in mod 0x107 op _mod 0x1 failed with error 0 status 28 Kernel panic might occur during fio splice in kernels before 2 6 34 rc4 Use kernel v2 6 34 rc4 which provides the following solution baff42a net oops from tcp collapse when using splice Fix In PPC systems when QoS is enabled a harmless Kernel DMA mapping error messages might appear in kernel log iommu related issue Transmit timeout might occur on RH6 3 as a result of lost interrupt OS issue In this case the follow ing message will be shown in dmesg do IRQ 0 203 No irq handler for vector irq 1 The default priority to TC mapping assigns all prior ities to TCO This configuration achieves fairness in transmission between priorities but may cause undesirable PFC behavior where pause request for priority n affects all other priorities Run mlnx qos i dev p 0 1 2 3 4 5 6 7 5 ets ets ets ets ets ets ets ets t 12 13 12 13 12 13 12 13 This needs
16. Index Description Workaround 1 When working with ISCSI over IPoIB LRO must be disabled even if IPoIB is set to connected mode due to a a bug in older kernels which causes a kernel panic 9 26 MLNX OFED Sources Known Issues Table 34 MLNX OFED Sources Known Issues Index Description Workaround 1 MLNX_OFED includes OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox propri etary packages 9 27 InfiniBand Utilities Known Issues Table 35 InfiniBand Utilities Known Issues Index Description Workaround l When running the ibdiagnet check nodes infoon Run ibdiagnet skip nodes info the fabric a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed 9 28 5 Driver Known Issues Table 36 mlx5 Driver Known Issues Index Description Workaround 1 Atomic Operations in Connect IB fully sup ported on big endian machines e g PPC Their support is limited on little endian machines e g x86 9 29 Ethernet Performance Counters Known Issues Table 37 Ethernet Performance Counters Known Issues Index Description Workaround 1 In a system with more than 61 VFs the 62nd VF and onwards is assigned with the SINKQP counter and as a result will have no statistics and loopback pre vention function
17. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox OFED for Linux Release Notes Rev 2 3 1 0 1 www mellanox com Rev 2 3 1 0 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF
18. SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 2 3 1 0 1 Table of Contents Table of Contenis 2445s Veo bike eck eens Deke REM AX TE EE E 3 List OF Tables ius dco aC Dade X CX olo e LC os ni Ra Rd i e RO eA Ba OR d os 5 Release Update History EAR E SSR A ERAS GR SER PR t EE Edd 7 Chapter do ha oa CR RR RACER RD USD C bates 8 Chapter 2 Main Features in This 8 Chapter 3 Content of Mellanox OFED for 9
19. ality for SINK counter 28 Mellanox Technologies Rev 2 3 1 0 1 Table 37 Ethernet Performance Counters Known Issues Continued Index Description Workaround 2 Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics in a system with less than 61 VFs if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet In this case RoCE statistics is not available 3 In ConnectX 3 when we enable function based loopback prevention for Ethernet port by default i e based on the QP counter index the dropped self loopback packets increase the IfRxErrorFrames Octets counters 9 30 Uplinks Known Issues Table 38 Uplinks Known Issues Index Description Workaround l On rare occasions ConnectX 3 Pro adapter card Restart the driver may fail to link up when performing parallel detect to 40GbE Mellanox Technologies 29 J Rev 2 3 1 0 1 Known Issues 9 31 Resources Limitation Known Issues Table 39 Resources Limitation Known Issues Index Description Workaround 1 The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace mlx4 core can allocate up to 64 MSI X vectors an MSI X vector per CPU Setting more IP addres
20. an Ubuntu 14 04 Hypervisor 13 SR IOV UD QPs are forced by the Hypervisor to use the base GID 1 GID that the VF sees in its GID entry at its paravirtualized index 0 This is needed for security since UD QPs use Address Vec tors and any GID index may be placed in such a vector including indices not belonging to that VF 14 Attempting to attach a PF to a VM when SR IOV is already enabled on that PF may result in a kernel panic 15 osmtest on the Hypervisor fails when SR IOV is enabled However only the test fails OpenSM will operate correctly with the host The failure reason is that if an mcg is already joined by the host a subse quent join request for that group succeeds automati cally even if the join parameters in the request are not correct This success does no harm 9 17 Port Type Management Known Issues Table 25 Port Type Management Known Issues Index Description Workaround l OpenSM must be stopped prior to changing the port protocol from InfiniBand to Ethernet 2 After changing port type using Use udev rules for persistent naming configu connectx port config interface ports names ration can be changed For example 161 gt ib0 if port For further information please refer to the changed to be Ethernet port and port2 left IB User Manual 3 A working IP connectivity between the RoCE devices is required when creating an address handle or modify
21. and Paging ODP Reset Flow for ConnectX 3 SR IOV RoCEv2 e Checksum offload for packets without L4 header Flow Steering AO simplified steering Memory re registration Cable EEPROM reporting 128 Byte Completion Queue Entry CQE Explicit Congestion Notification ECN e Disable Enable ethernet RX VLAN tag striping offload via ethtool Windows Virtual Machine over Linux KVM Hypervisor SR IOV with InfiniBand only 1 56 GbE is a Mellanox propriety link speed and can be achieved while connected to Mellanox SX10XX switch series 8 Mellanox Technologies Rev 2 3 1 0 1 3 Content of Mellanox OFED for Linux Mellanox OFED for Linux software contains the following components Table 2 Mellanox OFED for Linux Software Components Components Description OpenFabrics core and ULPs IB HCA drivers mlx4 mlx5 core Upper Layer Protocols IPoIB SRP and iSER Initiator OpenFabrics utilities e OpenSM IB Subnet Manager with Mellanox proprietary Adaptive Routing Diagnostic tools Performance tests MPI e OSU MPI mvapich2 1 9 1 stack supporting the InfiniBand interface Open MPI stack 1 6 5 and later supporting the InfiniBand interface e MPI benchmark tests OSU benchmarks Intel MPI benchmarks Presta PGAS ScalableSHMEM v2 2 supporting InfiniBand MXM and FCA ScalableUPC v2 2 supporting InfiniBand MXM and FCA HPC Acceleration packages e Mellanox MXM v3 0 p2p transport library a
22. ange Log History 1 0 0 0 ccc teen e 15 Table 9 IPoIB Known Issues e ra 16 Table 10 Ethernet Known Issues e 18 Table 11 General Known Issues 1 0 0 0 eet nen 19 Table 12 WGT Known Issues re 19 Table 13 eIPoIB Known Issues cnt as 19 Table XRG Kino wa 58U63 c oorr er ee RR ER bk CRUS CORDIA URS ELE ag 20 Table 15 ABI Compatibility Known 1 8 20 Table 16 System Time Known Issues 00 ccc e 21 Table 17 ConnectX 3 Adapter Cards Family Known Issues 21 Table 18 Verbs Known Issues porn miseries ccc ee es 21 Table 19 Resiliency Known 15 n ene e nen 21 Table 20 Driver Start Known Issues 0 ccc eee a 22 Table 21 Performance Tools Known 1 5 22 Table 22 Performance Known Issues 0 0 0 0 0c ce eee een en en eens 23 Table 23 Connection Manager CM Known Issues 23 Table 24 SR IOV Known Issues a 24 Table 25 Port Type Management Known 1 5 25 Table 26 Flow Steering Known Issues 0 0 0 0 cece eens 26 Table 27 Quality of Service Known Issues
23. arent interface with VLANS configured over it may result in traffic over VLANs being lost 3 On some of the OSes the callback ndo vlan rx add kill vidreturns void there fore ethX Y is created However if VLAN Y is not listed in the set of the allowed VLANs no traffic will pass 4 When untagged traffic is not allowed the below message will appear on Dom0 after driver restart on DomU mlx4 core 0000 16 00 0 vhcr command ALLOC RES 0xf00 slave 1 in param 0x0 in mod 0x207 op mod 0x1 failed with error 0 status 1 9 5 elPolB Known Issues Table 13 elPoIB Known Issues Index Description Workaround l On rare occasions upon driver restart the following message is shown in the dmesg cannot create duplicate filename class net eth ipoib interfaces 2 No indication is received when eIPoIB is non func Run ps ef grep ipoibd to verify its tional functionality 3 eIPoIB requires libvirtd python Mellanox Technologies 19 J Rev 2 3 1 0 1 Known Issues Table 13 elPoIB Known Issues Continued Index Description Workaround 4 eIPoIB supports only active backup mode for bond ing 5 eIPoIB supports VLAN Switch Tagging VST mode on guests 6 IPv6 is currently not supported in eIPoIB 7 eIPoIB cannot run when Flow Steering is enabled 9 6 XRC Known Issues Table 14 XRC Known Issues Index Descr
24. ation Known 155 30 9 32 ROCE Known Issues peaa oera ai i aa nen s 30 9 33 Storage Known Issues pinea eua e 32 9 34 SRP Known Issues a naa a ehh hh hh 32 9 35 SRP Interop Known Issues 32 9 36 DDN Storage Fusion 10000 Target Known 1550 32 9 37 Oracle Sun ZFS storage 7420 Known 1550 32 9 38 iSER Known Issues 0 00 hh 33 9 39 ZFS Appliance Known 1 33 Chapter 10 Bug Fixes 34 Chapter 11 Change Log amp 36 Chapter 12 API Change Log 40 4 Mellanox Technologies Rev 2 3 1 0 1 List Of Tables Table 1 Release Update History 0 0 0 hh eens 7 Table 2 OFED for Linux Software 9 Table 3 Supported Platforms and Operating Systems 10 Table 4 Additional Software Packages 0 0 cece eh 12 Table 5 Supported HCAs Firmware Versions 12 Table 6 MLNX OFED Rev 2 3 1 0 1 Compatibility 13 Table 7 Changes in v2 371 0 c sek cer vr EE o EORR E Rr Re UR RANA bea 14 Table8 Ch
25. cceleration over Infini band e Mellanox FCA v2 5 MPI PGAS collective operations acceleration library over InfiniBand e KNEM Linux kernel module enabling high performance intra node MPI PGAS communication for large messages Extra packages 160152 ibdump MFT Sources of all software modules under conditions mentioned in the modules LICENSE files except for MFT OpenSM plugins ibutils2 and ibdump Documentation Mellanox Technologies 9 Rev 2 3 1 0 1 Supported Platforms and Operating Systems 4 Supported Platforms and Operating Systems The following are the supported OSs in MLNX_OFED Rev 2 3 1 0 1 Table 3 Supported Platforms and Operating Systems Operating System Platform RHEL CentOS 6 3 x86 64 RHEL CentOS 6 4 x86 64 PPC RHEL CentOS 6 5 x86 64 PPC RHEL CentOS 7 0 x86 64 PPC SLES11 SP2 x86 64 PPC SLES11 SP3 x86 64 PPC OEL 6 3 x86 64 OEL 6 4 x86 64 OEL 6 5 x86 64 Citrix XenServer Host 6 2 1686 Fedora 19 x86 64 Fedora 20 x86 64 Ubuntu 12 04 4 x86 64 Ubuntu 14 04 x86 64 PPCA4le Debian 7 2 x86 64 Debian 7 4 x86 64 Debian 7 5 x86 64 kernel 3 10 48 kernel 3 11 10 kernel 3 12 24 kernel 3 13 1 kernel 3 14 12 kernel 3 15 5 a This kernel is supported only when using the Operating Systems stated in the table above If you wish to install OFED on a different kernel yo
26. dit loop free unicast multicast updn ftree routing Added multithreaded Minhop UPDN DOR routing RoCE Added IP routable RoCE modes For further information please refer to the MLNX OFED User Manual Installation Added apt get installation support Ethernet Added support for arbitrary UDP port for VXLAN From upstream 3 15 rc1 and onward it is possible to use arbitrary UDP port for VXLAN This feature requires firmware version 2 32 5100 or higher Additionally the following kernel configuration option CONFIG MLX4 EN VXLAN y must be enabled MLNX OFED no longer changes the OS sysctl TCP parameters Added Explicit Congestion Notification ECN support Added Flow Steering AO simplified steering support Added RoCE v2 support 14 Mellanox Technologies Rev 2 3 1 0 1 Table 7 Changes v2 3 1 0 1 Category Description InfiniBand Network Added Secure host to enable the device to protect itself and the subnet from mali cious software Added User Mode Memory Registration UMR to enable the usage of RDMA operations and to scatter the data at the remote side through the definition of appro priate memory keys on the remote side Added On Demand Paging ODP a technique to alleviate much of the shortcom ings of memory registration Added Masked Atomics operation support Added Checksum offload for packets without L4 header support Added Memory re registration to
27. e following to the etc modprobe d kvm conf file or create this file if it does not exist options kvm allow_unsafe_assigned_interrupts 1 kernel parameters 12 System might crash in skb_checksum_help Use UD mode in ipoib while performing TCP retransmit involving packets with 64k packet size A similar out to the below will be printed kernel BUG at net core dev c 1707 invalid opcode 0000 1 SMP RIP 0010 lt ffffffff81448988 gt skb_checksum_help 0x148 0x160 Call Trace lt IRQ gt fffff81448d83 5 dev hard start xmit4 0x3e3 0x530 lt 8144 805 gt dev_queue_xmit 0x205 0x550 ffff8145247d neigh connected output 0xbd 0x1 13 When InfiniBand ports are removed from the host To avoid it and have persistent IPoIB network e g when changing port type from IB to Eth or devices names for ConnectX ports add to the removing a card from the PCI bus the remaining etc udev rules d 70 persistent IPoIB interface might be renamed net rules file SUBSYSTEM net ACTION add DRIVERS ATTR address Port GID NAME ibN Where N is the IPoIB required interface index 14 After releasing a bond interface that contains IPoIB slaves a call trace might be printed into the dmesg Mellanox Technologies 17 Rev 2 3 1 0 1 Known Issues Table 9 IPoIB Known Issues Continued Index Description Workaround 1
28. escription Version MLNX OS MSX6036 w w MLNX OS version 3 3 4304 Grid Director 4036 w w Grid Director version 3 9 1 985 FabricIT EFM 155035 w w FabricIT EFM version 1 1 3000 FabricIT BXM MBX5020 w w FabricIT BXM version 2 1 2000 Unified Fabric Manager UFM v4 8 MXM v3 2 HPC X UPC v2 18 0 HPC X OpenSHMEM v1 8 3 FCA v2 5 and v3 1 HPC X MPI v1 8 3 MVAPICH v2 0 a MLNX OFED v2 3 1 0 1 was tested with this switch however additional switches might be supported as well Mellanox Technologies 13 J Rev 2 3 1 0 1 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 8 Changes in Rev 2 3 1 0 1 From Rev 2 2 1 0 1 Table 7 Changes in v2 3 1 0 1 Category Description OpenSM Added Routing Chains support with Minhop UPDN FTree DOR Torus 2QoS Added double failover elimination When the Master SM is turned down for some reason the Standby SM takes own ership over the fabric and remains the Master SM even when the old Master SM is brought up to avoid any unnecessary reregistartions in the fabric To enable this feature set the master sm priority parameter to be greater than the sm priority parameter in all SMs in the fabric Once the Standby SM becomes the Master SM its priority becames equal to the master sm priority So that additional SM handover is avoided Default value of the master sm priority is 14 To disable this feature set the master sm priority in opensm conf to 0 Added cre
29. exposes new features which are not integrated yet in to the upstream libibverbs The Experimental API is an extended API therefor it is backward compatible meaning old application are not required to be recompiled to use MLNX OFED v2 2 1 0 1 Performance Out of the box performance improvements e Use of affinity hints based on NUMA node of the device to indicate the IRQ balancer daemon on the optimal IRQ affinity Improvement in buffers allocation schema based on the hint above Improvement in the adaptive interrupt moderation algorithm 36 Mellanox Technologies Rev 2 3 1 0 1 Table 49 Change Log History Release Category Description Rev 2 1 1 0 6 IB Core Added allocation success verification process to ib_alloc_device dapl dapl is recompiled with no FCA support openibd Added the ability to bring up child interfaces even if the parent s ifcfg file is not configured libmlx4 Unmapped the hca clock page parameter from mlx4 uninit context scsi transport srp scsi transport srp cannot be cleared up when rport reconnecting fails minxofedinstall Added support for the following parameters e umad dev na e without package Content Packages The following packages were updated Updates to v2 2 407 mstflint to v3 5 0 1 1 g76e4acf perftest to v2 0 0 76 gbf9a463 e hcoll to v2 0 472 1 Openmpi to v1 6 5 440ad47 daplto v2 0 40 Rev 2 1 1 0
30. g History 11 Change Log History Table 49 Change Log History Release Category Description Rev 2 2 1 0 1 minxofedinstall 32 bit libraries are no longer installed by default on 64 bit OS To install 32 bit libraries use the with 32bit installation parameter openibd Added pre post start stop scripts support For further information please refer to section openibd Script in the MLNX OFED User Manual Reset Flow Reset Flow is not activated by default It is controlled by the mlx4 core internal err reset module parameter InfiniBand Core Asymmetric MSI X vectors allocation for the SR IOV hypervisor and guest instead of allocating 4 default MSI X vectors The maxi mum number of MSI X vectors is num cpu for port ConnectX 3 has 1024 MSI X vectors 28 MSI X vectors are reserved Physical Function gets the number of MSI X vectors accord ing to the pf msix table size multiple of 4 1 INI parameter e Virtual Functions the remaining MSI X vectors are spread equally between all VFs according to the v s mlx4 core module parameter Ethernet Ethernet VXLAN support for kernels 3 12 10 or higher Power Management Quality of Service when the traffic 1s active the Power Management QoS is enabled by disabling the CPU states for maximum performance Ethernet PTP Hardware Clock support on kernels OSes that sup port it Verbs Added additional experimental verbs interface This interface
31. ing a QP with an address vector 4 IPv4 multicast over requires the MGID for mat to be as follow ffff Multicast IPv4 Address 5 IP routable RoCE does not support Multicast Lis tener Discovery MLD therefore multicast traffic over IPv6 may not work as expected 6 DIF When running IO over FS over DM during unstable ports block layer BIOs merges may cause false DIF error Mellanox Technologies 25 J Rev 2 3 1 0 1 Known Issues 9 18 Flow Steering Known Issues Table 26 Flow Steering Known Issues Index Description Workaround l Flow Steering is disabled by default in firmware ver To enable it set the parameter below as follow sion 2 32 5100 log num mgm entry size should set to 1 2 IPv4 rule with source IP cannot be created in SLES 11 x OSes 3 RFS does not support UDP 9 19 Quality of Service Known Issues Table 27 Quality of Service Known Issues Index Description Workaround 1 QoS is not supported in XenServer Debian 6 0 and 6 2 with uek kernel 2 When QoS features are not supported by the kernel mlnx_qos tool may crash 9 20 Installation Known Issues Table 28 Installation Known Issues Index Description Workaround l When upgrading from an earlier Mellanox OFED Stop the old OFED stack etc init d version the installation script does not stop the ear openibd stop before upgradi
32. iption Workaround 1 Legacy API is deprecated thus when recompiling applications over MLNX_OFED v2 0 3 x x warn ings such as the below are displayed rdma c 1699 warning ibv open xrc domain is deprecated declared at usr include infiniband ofa verbs h 72 rdma c 1706 warning ibv create srq is deprecated declared at usr include infiniband ofa verbs h 89 These warnings can be safely ignored 2 XRC is not functional in heterogeneous clusters containing non Mellanox HCAs 3 XRC options do not work when using qperf tool Use perftest instead 4 Out of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created XRC QPs do not have receive queues 9 7 ABI Compatibility Known Issues Table 15 ABI Compatibility Known Issues Index Description Workaround 1 MLNX_OFED Rev 2 3 1 0 1 is not ABI compatible Recompile the application over the new with previous MLNX_OFED OFED versions MLNX_OFED version 20 Mellanox Technologies Rev 2 3 1 0 1 9 8 System Time Known Issues Table 16 System Time Known Issues Index Description Workaround 1 Loading the driver using the openibd script when no InfiniBand vendor module is selected for example mlx4 ib may cause the execution of the sbin start_udev script In RedHat 6 x and OEL6 x this may change the local system time 9 9 ConnectX
33. lign with upstream libib verbs ibv reg mr ibv access flags changed ibv post send opcodes and send flags changed and wr fields removed task op dc and bind mw ibv query device capability flags changed ibv poll cq opcodes and wc flags changed e ibv modify mask bits changed ibv create qp ex create flags field removed The following verbs removed to align with upstream libib verbs ibv bind mw ibv post task ibv query values ex ibv query device ex ibv poll cq ex ibv reg shared mr ex ibv reg shared mr ibv modify cq ibv create cq ex e ibv modify ex Verbs Experimental API The following experimental verbs added replacing the removed extended verbs e jbv exp bind mw e jbv exp post task ibv exp query values ibv exp query device 16 exp poll cq jbv exp reg shared mr 16 exp modify cq ibv exp create cq ijbv exp modify qp New experimental verbs e ijbv exp arm dct jbv exp query port ibv exp create flow ibv exp destroy flow ibv exp post send ijbv exp reg mr ibv exp get provider func 40 Mellanox Technologies Rev 2 3 1 0 1 Table 50 API Change Log History Release Name Description Rev 2 1 1 0 0 Dynamically Connected DC The following verbs were added e struct dct ibv_exp_create_dct struct ibv_context context struct ibv exp dct init a
34. log and traffic might be affected 8 iSCSID may hang if target crashes during logout sequence reproducible with TCP 9 39 ZFS Appliance Known Issues Table 47 ZFS Appliance Known Issues Index Description Workaround 1 Connection establishment occurs twice which may cause ISER to log a stack trace Mellanox Technologies 33 J Rev 2 3 1 0 1 Bug Fixes History 10 Bug Fixes History Table 48 lists the bugs fixed in this release Table 48 Fixed Bugs List Discovered Fixed in 2 Description in Release Release l IPoIB Changing the GUID ofa specific SR IOV guest after 2 1 1 0 0 2 3 1 0 1 the driver has been started causes the ping to fail Hence no traffic can go over that InfiniBand inter face 2 Ethernet Fixed kernel panic on Debian 6 0 7 which occurred 2 1 1 0 0 2 2 1 0 1 when the number of TX channels was set above the default value 3 Fixed a crash incidence which occurred when 2 0 2 0 5 22 1 0 1 enabling Ethernet Time stamping and running VLAN traffic 4 XRC XRC over in SR IOV mode is not functional 2 0 3 1 0 2 2 1 0 1 5 mlx4 en Fixed wrong calculation of packet true size reporting 2 1 1 0 0 2 2 1 0 1 in LRO flow 6 IB Core Fixed the QP attribute mask upon smac resolving 2 1 1 0 0 2 1 1 0 6 7 mlx5 16 Fixed a send WQE overhead issue 2 1 1 0 0 2 1 1 0 6 8 Fixed a NULL pointe
35. n Workaround 1 perftest package MLNX OFED 2 2 1 0 1 onwards does not work with older versions of the driver 22 Mellanox Technologies Rev 2 3 1 0 1 9 14 Performance Known Issues Table 22 Performance Known Issues Index Description Workaround 1 On machines with irqbalancer daemon turned off avoid this issue copy the following script to the default InfiniBand interrupts will be routed toa etc infiniband post start hook sh single core which may cause overload and software and execute it as root hardware lockups usr bin perl use strict if lt 1 0 print This script must be run as root n exit 0 open F proc interrupts or die SI my n cat proc cpuinfo grep processor wc 1 chomp n print Spreading over n cpus Mn while lt F gt print _ my Sirqg chan dev if d m1x5_comp d irq chan dev 1 2 m1x5 elsif d m1x4 ib d d Sirg chan dev 51 53 52 else next my 1 chan n my mask 1 lt lt Splace printf dev irg d chan d bit d mask 0x n Sirg chan Splace mask my cmd sprintf echo 0x gt proc irg Sirg smp_affinity Smask print t cmd n system cmd 9 15 Connection Manager CM Known Issues Table 23 Connection Manager CM Known Issues Index Description Workar
36. ng to this new lier version prior to uninstalling it version 2 Upgrading from the previous OFED installation to Reboot after installing the driver this release does not unload the kernel module Ipoib helper 3 Installation using Yum does not update HCA firm See Updating Firmware After Installation in ware OFED User Manual 4 total vfs lt 0 63 gt installation parameter 15 Use enable sriov installation parame no longer supported ter to burn firmware with SR IOV support The number of virtual functions VFs will be set to 16 For further information please refer to the User Manual 5 When using bonding on Ubuntu OS the ifenslave package must be installed 6 On PPC systems the module is not installed If your system does not require the ibmvscsi by default since it breaks the ibmvscsi module module run the mlnxofedinstall script with the with srp flag 26 Mellanox Technologies Rev 2 3 1 0 1 9 21 Driver Upload Known Issues Table 29 Driver Upload Known Issues Index Description Workaround 1 openibd stop can sometime fail with the error Re run openibd stop Unloading ib cm FAILED ERROR Module ib cm is in use by ib ipoib 9 22 InfiniBand Counters Known Issues Table 30 InfiniBand Counters Known Issues Index Description Workaround l Occasionally port rcv data and port xmit data counters may not function prop
37. ound l When 2 different ports have identical GIDs the CM All ports must have different GIDs might send its packets on the wrong port Mellanox Technologies 23 J Rev 2 3 1 0 1 Known Issues 9 16 SR IOV Known Isuues Table 24 SR IOV Known Issues Index Description Workaround 1 When using legacy VMs with MLNX_OFED 2 x hypervisor you may need to set the enable 64b eqe parameter to zero on the hypervisor It should be set in the same way that other module parameters are set for mlx4 core at module load time For example add options mlx4 core enable 64b eqe 0 asa line in the file etc modprobe d mlx4 core conf InfiniBand counters are not available in the VM mlx4 portl1 mtu sysfs entry shows a wrong MTU number in the VM When at least one port is configured as InfiniBand and num vfs is provided but the probe_vf is not HCA initialization fails Use both the num_vfs and the probe vf in the modprobe line When working with a bonding device to enslave the Ethernet devices in active backup mode and failover MAC policy in a Virtual Machine VM establish ment of RoCE connections may fail Unload the module mlx4 1b and reload it in the VM Attaching or detaching a Virtual Function on SLES11 SP3 to a guest Virtual Machine while the mlx4 core driver is loaded in the Virtual Machine may cause a kernel panic in the hypervisor Unload the
38. r dereference on the debug 2 1 1 0 0 2 1 1 0 6 print 9 Fixed arguments to kzalloc 2 1 1 0 0 2 1 1 0 6 10 mlx4 core Fixed the locks around completion handler 2 1 1 0 0 2 1 1 0 6 11 mlx4 core Restored port types as they were when recovering 2 0 2 0 5 2 1 1 0 0 from an internal error 12 Added an N A port type to support port type array 2 0 2 0 5 2 1 1 0 0 module param in an HCA with a single port 13 SR IOV Fixed memory leak in SR IOV flow 2 0 2 0 5 2 0 3 0 0 14 Fixed communication channel being stuck 2 0 2 0 5 2 0 3 0 0 34 Mellanox Technologies Rev 2 3 1 0 1 Table 48 Fixed Bugs List Discovered Fixed in iue D seription in Release Release 15 mlx4 en Fixed ALB bonding mode failure when enslaving 2 0 3 0 0 2 1 1 0 0 Mellanox interfaces 16 Fixed leak of mapped memory 2 0 3 0 0 2 1 1 0 0 17 Fixed TX timeout in Ethernet driver 2 0 2 0 5 2 0 3 0 0 18 Fixed ethtool stats report for Virtual Functions 2 0 2 0 5 2 0 3 0 0 19 Fixed an issue of VLAN traffic over Virtual Machine 2 0 2 0 5 2 0 3 0 0 in paravirtualized mode 20 Fixed ethtool operation crash while interface down 2 0 2 0 5 2 0 3 0 0 21 IPoIB Fixed memory leak in Connected mode 2 0 2 0 5 2 0 3 0 0 22 Fixed an issue causing IPoIB to avoid pkey value 0 2 0 2 0 5 2 0 3 0 0 for child interfaces Mellanox Technologies 35 J Rev 2 3 1 0 1 Change Lo
39. rd which uses IPoIB with Connected mode as default Storage ISER at GA level Rev 2 0 2 0 52 Virtualization SR IOV for both Ethernet and InfiniBand at Beta level Ethernet Network RoCE over SR IOV at Beta level eIPoIB to enable IPoIB in a Para Virtualized environment at Alpha level Ethernet Performance Enhancements NUMA related and others for 10G and 40G Ethernet Time Stamping at Beta level Flow Steering for Ethernet and InfiniBand at Beta level Raw Eth QPs e Checksum TX RX Flow Steering InfiniBand Net Contiguous pages work Internal memory allocation improvements Register shared memory e Control objects QPs CQs Installation YUM update support VMA OFED VMA integration to a single branch 38 Mellanox Technologies Rev 2 3 1 0 1 Table 49 Change Log History Release Category Description Storage iSER at Beta level and SRP Operating Systems Errata Kernel upgrade support API VERSION query API library and headers Counters 64bit wide counters port xmit recv data packets unicast mcast a SR IOV Ethernet Time Stamping and Flow Steering are ConnectX 3 HCA capability Mellanox Technologies 39 J Rev 2 3 1 0 1 API Change Log History 12 Log History Table 50 API Change Log History Release Name Description Rev 2 2 1 0 1 libibverbs The following verbs changed to a
40. regardless the presence of the IPv6 address 10 Using IPv6 link local address GIDO when VLANs are configured is not supported Mellanox Technologies 31 J Rev 2 3 1 0 1 Known Issues 9 33 Storage Known Issues Table 41 Storage Known Issues Index Description Workaround 1 Older versions of rescan_scsi_bus shmay not If encountering such issues it is recommended recognize some newly created LUNs to use the c flag 2 RHEL7 0 The rescan scsi bus sh script does Use older version of the script from not rediscover provisioned LUNs both on iSER and RHEL6 4 SRP iSER Use iscsiadm m session rescan 9 34 SRP Known Issues Table 42 SRP Known Issues Index Description Workaround 1 In a high stress IO with unstable links SRP Initiator may generate a call trace that can be safely ignored 2 MLNX_OFED SRP installation breaks the ibmvstgt and ibmvscsi symbol resolution in RHEL7 0 9 35 SRP Interop Known Issues Table 43 SRP Interop Known Issues Index Description Workaround 1 The driver is tested with Storage target vendors rec ommendations for multipath conf extensions ZFS DDN TMS Nimbus NetApp 9 36 DDN Storage Fusion 10000 Target Known Issues Table 44 DDN Storage Fusion 10000 Target Known Issues Index Description Workaround 1 DDN does not accept non default P_Key connection establishment
41. river Start Known Issues Index Description Workaround 1 Out of memory issues may rise during drivers load depending on the values of the driver module parameters set e g log num cq 2 When reloading starting the driver using the etc Remove the third party RPM non init d openibd the following messages are dis MLNX OFED drivers directory run played if there is a third party RPM or driver depmod and then rerun etc init d installed openibd restart Module mlx4 core does not belong to MLNX OFED or Module mlx4 core belong to rpm name gt which is not a part of MLNX OFED 3 Occasionally when trying to repetitively reload the nes hardware driver on SLES11 SP2 a soft lockups occurs that required reboot 4 In ConnectX 2 if the driver load succeeds the informative message below is presented conveying the below limitations If port type is IB the number of maximum sup ported VLs is 4 fporttype is ETH then the maximum priority for VLAN tagged is 3 mlx4 core 0000 0d 00 0 command SET PORT 0 failed in param 0x120064000 in mod 0x2 op_mod 0x0 fw status 0x40 5 openibd start unloads kernel modules that were loaded from initrd initramfs upon boot This affects only kernel modules which come with MLNX_OFED and are included in initrd initramfs 9 13 Performance Tools Known Issues Table 21 Performance Tools Known Issues Index Descriptio
42. ses than the available GID entries in the table results in failure and the update gid table error message is displayed GID table of port 1 is full Can t add address message Registering a large amount of Memory Regions MR may fail because of DMA mapping issues on RHEL 7 0 Occasionally a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system s free mem ory for its internal cache To free memory to allow it to be allocated in a user process run the drop caches procedure below Performing the following steps will cause the kernel to flush and free pages dentries and inodes caches from memory causing that memory to become free Note As this is a non destructive operation and dirty objects are not freeable run sync first To free the pagecache echo 1 proc sys vm drop caches free dentries and inodes echo 2 proc sys vm drop caches free pagecache dentries and inodes echo 3 proc sys vm drop caches 9 32 RoCE Known Issues Table 40 RoCE Known Issues Index Description Workaround l Not configuring the Ethernet devices or independent VMs with a unique IP address in the physical port may result in RoCE GID table corruption Restart the driver If RDMA CM is not used for connection manage ment then the source and destination GIDs used to modify a QP or create AH
43. ssues 22 9 13 Performance Tools Known 1 22 9 14 Performance Known 1 23 9 15 Connection Manager CM Known Issues 23 9 16 SR IOV Known Isuues oo caedi ra 24 9 17 Port Type Management Known Issues 25 9 18 Flow Steering Known 1 5 26 9 19 Quality of Service Known 15 26 9 20 Installation Known 1 26 9 2 Driver Upload Known 15 5 27 9 22 InfiniBand Counters Known 1 27 9 23 UEFI Secure Boot Known Issues 27 9 24 Fork Support Known 1 5 2 Mellanox Technologies 3 Rev 2 3 1 0 1 9 25 ISCSI over IPoIB Known 1 28 9 26 MLNX OFED Sources Known 1 28 9 27 InfiniBand Utilities Known 15 28 9 28 mlx5 Driver Known Issues 28 9 29 Ethernet Performance Counters Known 1550 5 28 9 30 Uplinks Known Issues lesse n 29 9 31 Resources Limit
44. the bidirectional line rate Optimize the IPoIB performance in Connect IB cat sys class net interface device local cpus sys class net interface queues rx 0 rps cpus Ifthe CONNECTED MODE no parameter is set to no or missing from the ifcfg file for Connect IB IPoIB interface then the service network restart will hang Setthe CONNECTED MODE yes parameter in the ifcfg file for Connect IBG interface Joining a multicast group in the SM using the RDMA CM API requires IPoIB to first join the broadcast group 16 Mellanox Technologies Rev 2 3 1 0 1 Table 9 IPoIB Known Issues Continued Index Description Workaround 11 Whenever the IOMMU parameter is enabled in the To avoid such issue kernel it can decrease the number of child interfaces Decrease the amount of the RX receive buf on the device according to resource limitation fers module parameter the default is 512 The driver will stuck after unknown amount of child Decrease the number of RX rings sys fs or interfaces creation ethtool in new kernels Avoid using IOMMU if not required For further information please see https access redhat com site articles 66747 For KVM users http support citrix com article CTX 136517 Run http www novell com support kb echo 1 gt sys module kvm parame doc php id 7012337 ters allow_unsafe_assigned_interrupts To make this change persist across reboots add th
45. ttr attr int ibv exp destroy dct struct ibv dct dct int ibv exp query dct struct ibv dct dct struct ibv exp dct attr attr Verbs Extension API 16 post task Verbs extension API defines ibv query values ex OFA APIs extension scheme e iby query device ex to detect ABI compatibility Ibv create flow and enable backward and for ibv destroy flow ward compatibility support ibv poll cq ex ibv reg shared mr ex 16 open xrcd 16 close xrcd ibv modify cq ibv create srq ex e 10 get srq num ibv create qp ex 16 create cq ex ibv open qp e jbv modify qp ex Verbs Experimental API Verbs experimental API defines MLNX OFED APIs 16 exp create dct extension scheme which is ibv exp destroy dct similar to the Verbs exten e 16 exp query dct sion APT This extension provides a way to introduce new features before they are integrated into the formal OFA API and to the upstream kernel and libs Rev 2 0 3 0 0 XRC The following verbs have become deprecated struct ibv xrc domain ibv open xrc domain 16 exp create qp 16 exp query device e struct srq ibv create xrc srq int ibv close xrc domain int ibv create xrc rcv qp int ibv modify xrc rcv qp int ibv query xrc rcv qp e int reg rcv int ibv unreg xrc rcv qp Mellanox Technologies 41 Rev 2 3 1 0 1 API Change Log History Table 5
46. u need to create a new ISO image using mlnx add kernel support sh script See the MLNX OFED User Guide for instructions Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest version as well 10 Mellanox Technologies Rev 2 3 1 0 1 41 Supported Hypervisors The following are the supported hypervisors in MLNX OFED Rev 2 3 1 0 1 KVM e Xen 4 2 Supported Non Linux Virtual Machines The following are the supported Non Linux InfiniBand only Virtual Machines in MLNX OFED Rev 2 3 1 0 1 Windows Server 2012 R2 Windows Server 20012 Windows Server 2008 R2 Mellanox Technologies 11 Rev 2 3 1 0 1 Hardware and Software Requirements 5 Hardware and Software Requirements The following are the hardware and software requirements of MLNX_OFED Rev 2 3 1 0 1 Linux operating system Administrator privileges on your machine s Disk Space IGB For the OFED Distribution to compile on your machine some software packages of your operat ing system OS distribution are required To install the additional packages run the following commands per OS Table 4 Additional Software Packages Operating System Required Packages Installation Command RHEL OEL yum install perl pciutils python gcc gfortran libxml2 python tesh libnl 1686 Fedora libnl expat glib2 tcl libstdc bc tk gtk2 atk cairo numactl XenServer yum install perl pciutils python libxml2 python libnl expat
Download Pdf Manuals
Related Search