Home

Mellanox OFED for Linux Release Notes

1. From IBV M WQE RQ ENABLE CAP to IBV M WOE CAP RO ENABLE e From IBV M WOE CQE WAIT CAP to IBV M WOE CAP COE WAIT e From IBV M WOE CALC CAP to IBV M WOE CAP CALC SEND 26 Mellanox Technologies Rev 2 1 1 0 6 11 Bug Fixes History Table 13 lists the bugs fixed in this release Table 13 Fixed Bugs List Roe Discovered Fixed in a its Descnpton in Release Release l IB Core Fixed the QP attribute mask upon smac resolving 2 1 1 0 0 2 1 1 0 6 25 mlx5 ib Fixed a send WQE overhead issue 2 1 1 0 0 2 1 1 0 6 3 Fixed a NULL pointer dereference on the debug 2 1 1 0 0 2 1 1 0 6 print 4 Fixed arguments to kzalloc 2 1 1 0 0 2 1 1 0 6 5 mlx4 core Fixed the locks around completion handler 2 1 1 0 0 2 1 1 0 6 6 mlx4 core Restored port types as they were when recovering 2 0 2 0 5 2 1 1 0 0 from an internal error Ts Added an N A port type to support port type array 2 0 2 0 5 2 1 1 0 0 module param in an HCA with a single port 8 SR IOV Fixed memory leak in SR IOV flow 2 0 2 0 5 2 0 3 0 0 9 Fixed communication channel being stuck 2 0 2 0 5 2 0 3 0 0 10 mlx4 en Fixed ALB bonding mode failure when enslaving 2 0 3 0 0 2 1 1 0 0 Mellanox interfaces 11 Fixed leak of mapped memory 2 0 3 0 0 2 1 1 0 0 12 Fixed TX timeout in Ethernet driver 2 0 2 0 5 2 0 3 0 0 13 Fixed ethtool stats report for Virtual Functions 2
2. Name Description XRC The following verbs have become deprecated Struct ibv xrc domain ibv open xrc domain Struct ibv srq ibv create xrc srq int ibv close xrc domain int ibv create xrc rcv qp int ibv modify xrc rcv qp int ibv query xrc rcv qp int ibv reg xrc rcv gp int ibv unreg xrc rcv qp 10 3 API Changes in MLNX OFED Rev 2 0 2 0 5 The following are the API changes in MLNX OFED v2 0 2 0 5 Table 12 API Changes in MLNX OFED Rev 2 0 2 0 5 Name Description Libibverbs Extended speeds Missingtheext active speed attribute from the struct ibv port attr Removed function ibv ext rate to int Addedfunctions ibv rate to mbpsandmbps to ibv rate Raw QPs QP types TBV OPT RAW PACKET and IBV QPT RAW ETH are not sup ported Contiguous pages Added Contiguous pages support Addedfunctionibv reg shared mr Libmverbs The enumeration IBV M WR CALC was renamed to IBV M WR CALC SEN The enumeration IBV M WR WRITE WITH IMM was added Inthe structure ibv m send wr the union wr send was renamed to wr calc sendand wr rdma was added The enumerations IBV M WOE CAP CALC RDMA WRITE WITH IMM was added Mellanox Technologies 25 J Rev 2 1 1 0 6 API Changes Table 12 API Changes in MLNX_OFED Rev 2 0 2 0 5 Name Description The following enumerations were renamed From IBV M WQE SQ ENABLE CAP to IBV M WOE CAP SQ ENABLE I
3. mance Counters and onwards is assigned with the SINKQP coun ter and as a result will have no statistics and loopback prevention functionality for SINK coun ter 78 Since each VF tries to allocate 2 more QP counter for its RoCE traffic statistics in a system with less than 61 VFs if there is free resources it receives new counter otherwise receives the default counter which is shared with Ethernet In this case RoCE statistics is not available 79 In ConnectX 3 when we enable function based loopback prevention for Ethernet port by default i e based on the OP counter index the dropped self loopback packets increase the IfRxError Frames Octets counters 80 RoCE Not configuring the Ethernet devices or indepen Restart the driver dent VMs with a unique IP address in the physical port may result in RoCE GID table corruption 81 If RDMA CM is not used for connection man agement then the source and destination GIDs used to modify a OP or create AH should be of the same type IPv4 or IPv6 82 Since the number of GIDs per port is limited to 128 there cannot be more than the allowed IP addresses configured to Ethernet devices that are associated with the port Allowed number is e 127 for a single function machine e 15 for a hypervisor in a multifunction machine n for a guest in a multifunction machine where n is the number of virtual functions 83 A workin
4. 8 Change Log History 8 1 Changes in Rev 2 1 1 0 6 From Rev 2 1 1 0 0 Table 5 New Features Changes and Fixes in v2 1 1 0 6 Category Description IB Core Added allocation success verification process to ib alloc device dapl dapl is recompiled with no FCA support openibd Added the ability to bring up child interfaces even if the parent s ifcfg file is not configured libmlx4 Unmapped the hca clock page parameter from mlx4 uninit context Scsi transport srp Scsi transport srp cannot be cleared up when rport reconnecting fails minxofedinstall Added support for the following parameters e umad dev na e without package Content Packages The following packages were updated Updates bupc to v2 2 407 e mstflint to v3 5 0 1 1 g76e4acf e perftest to v2 0 0 76 gbf9a463 hcoll to v2 0 472 1 e Openmpi to v1 6 5 440ad47 e daplto v2 0 40 8 2 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 Table 6 New Features Changes and Fixes in v2 1 1 0 0 Sheet 1 of 2 Category Description EoIB EoIB is supported only in SLESIISP2 and RHEL6 4 Connect IB Added the ability to resize CQs IPoIB Reusing DMA mapped SKB buffers Performance improvements when IOMMU is enabled minx en Added reporting autonegotiation support Added Transmit Packet Steering XPS support Added reporting 56Gbit s link speed support Added Receive Flow Steering RFS support in UDP Added Low Latency S
5. Operating Systems se se sesse se se sees ese 7 Chapter 5 Hardware and Software Reguirement se se sesse se se sesse ses 8 Chapter 6 Supported HCAS ce er ok ERR RR se OR BAER bg CRT ERR Re 9 Chapter 7 Compatibility iex Int A es So Ws ala ee LACE ee RC AC eae 9 Chapter 8 Change Log History se see see see see see see see see ses cece ntn 10 8 1 Changes in Rev 2 1 1 0 6 From Rev 2 1 1 0 0 ee ese cece eee ee eee 10 8 2 Changes in Rev 2 1 1 0 0 From Rev 2 0 3 0 0 0 se eee ee eee 10 8 3 Changes in Rev 2 0 3 0 0 From Rev 2 0 2 0 5 cc cece eee 11 84 New Features in Rev 2 0 2 0 5 0 0 00 cece eee eens 11 Chapter9 Known Issues Lio Wees IRR Knee n RC SR COCHIN Rb ce e S RC KORR oe AR wie 13 Chapter 10 API Cian Ge us cech en enc ele a xo E eral es RUE RO QC al RR 24 10 1 API Changes in MLNX OFED Rev 2 1 1 0 0 2 0 0 2 0c eee ee eee 24 10 1 1 Verbs Extension and Verbs Experimental APIs 00 0 00 see 24 10 2 API Changes in MLNX OFED Rev 2 0 3 0 0 0 0 2 0 2c cee eee ee eee 25 10 3 API Changes in MLNX OFED Rev 2 0 2 0 5 2 0 00 00 nannaa 25 Chapter 11 Bug Fixes History sis siese soe hen Rh sie SEN GR Ep RR Wise Sa eie esi es 27 Mellanox Technologies 3 J Rev 2 1 1 0 6 List Of Tables Table 1 Mellanox OFED for Linux Software Components ss ss es se se ss eee eee 6 Table 2 Supported Platforms and Operating Systems nnna nnana cece Ge ee ee ese 7 Tabl
6. by default To enable it set the parame ter below as follow log num mgm entry siz e should set to 1 55 IPv4 rule with source IP cannot be created in SLES 11 56 RFS is not supported in SLES11 57 Quality of Service QoS is not supported in XenServer Debian 6 0 and in OEL6 1 and 6 2 with uek kernel 58 Driver Uninstall A Kernel panic occurs if you uninstall the driver Remove the midule after without deleting the SR IOV module params uninstalling and prior to mlx4 core s num vfs in the file etc mod restarting the driver probe d mlx4 core conf On the next boot you will get the panic and machine will boot up 59 Installation When upgrading from an earlier Mellanox OFED Stop the old OFED stack version the installation script does not stop the etc init d openibd earlier version prior to uninstalling it stop before upgrading to this new version 60 Upgrading from the previous OFED installation Reboot after installing the to this release does not unload the kernel module driver ipoib_helper 61 Installation using Yum does not update HCA See Updating Firmware firmware After Installation in OFED User Manual 62 On SLES11 1 the package libnl i586 is required Perform one of the follow to install MLNX_OFED ing Install the libnl i586 RPM from the SLES11 1 32bit installation disk Install MLNX OFED with the following flag with out 32bit 63 When using bonding on Ubuntu OS th
7. issue the following call trace will be shown in the kernel log WARNING at net core dev c 2077 get rps cpu 0x70 0x2b9 20 Kernel panic might occur during traffic over IPv6 on kernels between 3 12 rc7 and 3 13 rel kernel issue 21 Kernel panic might occur during fio splice in ker Use kernel v2 6 34 rc4 nels before 2 6 34 rc4 which provides the follow ing solution baff42a net Fix oops from tcp collapse when using splice 22 On Debian 6 0 7 kernel panic may occur when changing the number of TX channels above the default value 8 23 On kernels that do not support multiqueues the number of TX channels represents the number of TX rings The maximal number of TX channels is 16 Mellanox Technologies 15 J Rev 2 1 1 0 6 Known Issues Table 8 Known Issues Continued Index Issue Description Workaround 24 Transmit timeout might occur on RH6 3 as a result of lost interrupt OS issue In this case the following message will be shown in dmesg do_IRQ 0 203 No irq handler for vec tor irg 1 25 26 27 28 29 30 eIPoIB On rare occasions upon driver restart the follow ing message is shown in the dmesg cannot create duplicate filename class net eth ipoib interfaces No indication is received when eIPoIB is non functional Run ps ef grep ipoibd to verify its func ti
8. large messages Extra packages e ibutils2 ibdump MFT Sources of all software modules under conditions mentioned in the modules LICENSE files except for MFT OpenSM plugins ibutils2 and ibdump Documentation 6 Mellanox Technologies J Rev 2 1 1 0 6 4 Supported Platforms and Operating Systems The following are the supported OSs in MLNX_OFED Rev 2 1 1 0 6 Table 2 Supported Platforms and Operating Systems Operating System Platform RHEL CentOS 6 3 x86 64 RHEL CentOS 6 4 x86 64 RHEL CentOS 6 5 x86 64 SLES11 SPI x86 64 SLES11 SP2 x86 64 SLES11 SP3 x86 64 OEL 6 1 x86 64 OEL 6 2 x86 64 OEL 6 3 x86 64 OEL 6 4 x86 64 OEL 6 5 x86 64 Citrix XenServer Host 6 x 1686 Fedora 18 x86 64 Fedora 19 x86 64 Ubuntu 12 04 x86 64 Ubuntu 13 04 x86 64 Ubuntu 13 10 x86 64 Debian 6 0 7 x86 64 Debian 6 0 8 x86 64 Debian 7 1 x86 64 Debian 7 2 x86 64 kernel 3 10 kernel 3 11 kernel 3 12 using mlnx add kernel support sh script gt If you wish to install OFED on a different kernel you need to create a new ISO image ae See the MLNX OFED User Guide for instructions GP Upgrading MLNX OFED on your cluster requires upgrading all of its nodes to the newest version as well Mellanox Technologies 7 J Rev 2 1 1 0 6 Hardware and Software Requirements 5 Hardware and Software Requirements The following a
9. 0 2 0 5 2 0 3 0 0 14 Fixed an issue of VLAN traffic over Virtual Machine 2 0 2 0 5 2 0 3 0 0 in paravirtualized mode 15 Fixed ethtool operation crash while interface down 2 0 2 0 5 2 0 3 0 0 16 IPoIB Fixed memory leak in Connected mode 2 0 2 0 5 2 0 3 0 0 17 Fixed an issue causing IPoIB to avoid pkey value 0 2 0 2 0 5 2 0 3 0 0 for child interfaces Mellanox Technologies 27 J
10. 6 Mellanox Technologies Rev 2 1 1 0 6 Table 8 Known Issues Continued Index Issue Description Workaround 37 ABI Compatibility MLNX_OFED Rev 2 1 1 0 6 is not ABI compati Recompile the application ble with previous MLNX_OFED OFED versions over the new MLNX_OFED version 38 System Time Loading the driver using the openibd script when no InfiniBand vendor module is selected for example mlx4_ib may cause the execution of the sbin start_udev script In RedHat 6 x and OEL6 x this may change the local system time 39 Verbs Verbs for the following features are subject to change e Core Direct e Shared memory region Contiguous pages Flow steering Verbs subject to changes are iby post task e ibv query values ex e iby query device ex e iby poll cq ex e ibv reg shared mr ex iby reg shared mr e ibv modify cq ibv create cq ex e ibv modify qp ex ibv reg mr ibv post send e jibv dealloc mw e ibv alloc mw ibv bind mw iby query device e ibv poll cq e ibv create qp ex e ibv modify qp 40 Using libnll 1 3 26 or earlier requires ibv create ah protection by a lock for multi threaded applications 41 Driver Start When reloading the driver using the etc init d openibd restart command on XenServer6 1 loading of mlx4 en driver might fail with Unresolved Symbols errors This message can safely be ignored 42 Out of memor
11. It should be set in the same way that other module parameters are set for mlx4 core at module load time For example add options mlx4 core enable 64b cqe eqe 0 asa line in the file etc modprobe d mlx4 core conf 48 Enabling SR IOV requires appending the intel iommu on option to the relevant OS in file boot grub grub conf Without that SR IOV cannot be loaded 49 rdma cm does not support UD QPs 50 SR IOV can be enabled only when using the firmware version embedded in the MLNX OFED v2 0 3 0 0 driver 51 When SR IOV is disabled in the system BIOS a Enable Sr IOV in the BIOS PCI issue is noticed in Ubuntu v12 04 3 with Linux kernel v3 8 which affects NICs of several manufacturers including Mellanox s preventing them from operating 18 Mellanox Technologies Rev 2 1 1 0 6 Table 8 Known Issues Continued Index Issue Description Workaround 52 Port Type Manage OpenSM must be stopped prior to changing the l ment port protocol from InfiniBand to Ethernet 53 After changing port type using Use udev rules for persis connectx_port_config interface ports names tent naming configuration can be changed For example ib1 gt ib0 if port For further information changed to be Ethernet port and port2 left IB please refer to the User Manual 54 Flow Steering Flow Steering is disabled
12. Mellanox TE Connect CHNOLOGIES Accelerate Outperform Mellanox OFED for Linux Release Notes Rev 2 1 1 0 6 Last Updated May 13 2014 www mellanox com Rev 2 1 1 0 6 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVE
13. N IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox amp Mellanox logo BridgeX ConnectX Connect IB amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 2 1 1 0 6 Table of Contents Table of Contemts 2 0534 ske si ER A seks ORE EER BED GEND EER EE ER ODE DE ee wes 3 Last Of Tables EE ER EE ehm CK Kx one t EE EE EE OE N RE oen e 4 Chapter 1 Overview ee Oe POWER Re oa eu ia ox d PRU acu d opa rd os dolia aos 5 Chapter 2 Main Features in This Release ss se sesse se se se see se se se eee se se 5 Chapter 3 Content of Mellanox OFED for LiNUX sesse se sesse se se sesse se se 6 Chapter 4 Supported Platforms and
14. O must be disabled even if IPoIB is set to connected mode due to a a bug in older kernels which causes a kernel panic 69 MLNX OFED sources MLNX_OFED includes the OFED source RPM packages used as a build platform for kernel code but does not include the sources of Mellanox pro prietary packages 70 InfiniBand Utilities When running the ibdiagnet check nodes_info on the fabric a warning specifying that the card does not support general info capabilities for all the HCAs in the fabric will be displayed Run ibdiagnet skip nodes_info 71 mlx5 Driver Atomic Operations over Connect IBTM are not supported T2 General On ConnectX 2 ConnectX 3 Ethernet adapter cards there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric driver utilities that read the GUID via device firmware e g using ibstat Mlxburn flint return Oxffff as GUID while the utilities return a value derived from the MAC address For all driver firmware software pur poses the latter value should be used N A Please use the GUID value returned by the fabric driver utilities not Oxfffff 73 Uplinks On rare occasions ConnectX 3 Pro adapter card may fail to link up when performing parallel detect to 40GbE Restart the driver 20 Mellanox Technologies Rev 2 1 1 0 6 Table 8 Known Issues Continued Index Issue Descr
15. a level e RoCE Time Stamping e PeerDirect nline Receive Ethernet Performance Counters Memory Window VMA bundled with MLNX OFED DCT support at beta level eIPoIB multicast support 1 56 GbE is a Mellanox propriety link speed and can be achieved while connected to Mellanox SX10XX switch series Mellanox Technologies 5 J Rev 2 1 1 0 6 Content of Mellanox OFED for Linux 3 Content of Mellanox OFED for Linux Mellanox OFED for Linux software contains the following components Table 1 Mellanox OFED for Linux Software Components Components Description OpenFabrics core and ULPs IB HCA drivers mlx4 mlx5 core Upper Layer Protocols IPoIB SRP and ISER Initiator OpenFabrics utilities OpenSM IB Subnet Manager with Mellanox proprietary Adaptive Routing Diagnostic tools e Performance tests MPI e OSU MPI mvapich2 1 9 1 stack supporting the InfiniBand interface e Open MPI stack 1 6 5 and later supporting the InfiniBand interface MPI benchmark tests OSU benchmarks Intel MPI benchmarks Presta PGAS ScalableSHMEM v2 2 supporting InfiniBand MXM and FCA ScalableUPC v2 2 supporting InfiniBand MXM and FCA HPC Acceleration packages Mellanox MXM v2 1 p2p transport library acceleration over Infiniband e Mellanox FCA v2 5 MPI PGAS collective operations acceleration library over InfiniBand KNEM Linux kernel module enabling high performance intra node MPI PGAS communication for
16. e ifenslave package must be installed 64 libnl devel is required by libvma RPM There Remove the libvma from the fore installing MLNX OFED LINUX 2 1 1 0 6 list of installed packages on RHEL6 x HPC requires downloading libnl mlnxofedinstall devel The library can be downloaded from without libvma rhn redhat com or from the corresponding Cen tOS download page Mellanox Technologies 19 J Rev 2 1 1 0 6 Known Issues Table 8 Known Issues Continued Index Issue Description Workaround 65 total vfs lt 0 63 gt installation parameter is no longer supported Use enable sriov installation parameter to burn firmware with SR IOV support The number of vir tual functions VFs will be set to 16 For further information please refer to the User Manual 66 Driver Unload Openibd stop can sometime fail with the error Unloading ib cm FAILED ERROR Module ib cm is in use by ib ipoib Re run openibd stop 67 Fork Support Fork support from kernel 2 6 12 and above is available provided that applications do not use threads fork is supported as long as the parent process does not run before the child exits or calls exec The former can be achieved by calling wait childpid and the latter can be achieved by application specific means The Posix system call is supported 68 ISCSI over IPoIB When working with ISCSI over IPoIB LR
17. e 3 Additional Software Packages 0 0 ccc SG Ge eg en 8 Table4 MLNX OFED Rev 2 1 1 0 6 Compatibility Matrix 0 0 0 0 ee ee ee es eee 9 Table 5 New Features Changes and Fixes in V2 1 1 0 6 EE cece cece ee ee es ee 10 Table 6 New Features Changes and Fixes in V2 1 1 0 0 EE ce cece cece es ee 10 Table 7 New Features Changes and Fixes in v2 0 3 0 0 0 00 cece cece cece ee ee 11 Tabl 8 Krnown lssuess4 ix De ER Sa RE RISE Iun ekle Me ROG bbe We ch ote 13 Table 9 API Changes in MLNX OFED Rev 2 1 1 0 0 ese Se ES Ge eee eee 24 Table 10 Verbs Extension and Verbs Experimental APIs 00 0 0 ccc cece eee ene nee 24 Table 11 API Changes in MLNX OFED Rev 2 0 3 0 0 ees ee ee ee ee See ee 25 Table 12 API Changes in MLNX OFED Rev 2 0 2 0 5 2 00 0 oes 25 Table 13 Fixed Bugs hist oe ote bat hated dd ai dd ard BEA Ra OLR eee 27 4 Mellanox Technologies J Rev 2 1 1 0 6 1 Overview These are the release notes of Mellanox OFED for Linux Driver Rev 2 1 1 0 6 Mellanox OFED is a single Virtual Protocol Interconnect VPI software stack and operates across all Mellanox network adapter solutions supporting the following uplinks to servers 10 20 40 and 56 Gb s InfiniBand IB 10 40 and 56 Gb s Ethernet 2 5 or 5 0 GT s PCI Express 2 0 e 8 GT s PCI Express 3 0 2 Main Features in This Release MLNX_OFED Rev 2 1 1 0 6 provides the following new features Signature Verbs T10 PI at bet
18. ecific SR IOV guest after the driver has been started causes the ping to fail Hence no traffic can go over that Infini Band interface 14 Mellanox Technologies Rev 2 1 1 0 6 Table 8 Known Issues Continued Index Issue Description Workaround 16 send queue size over Connect IBTM adapter cards cannot be larger than 1024 17 Ethernet Ethernet PV VLAN Guest transparent Tagging VGT is only supported in openvswitch and not in standard Linux vBridges and libvirt For more information please see http lib virt org formatnetwork html Setting VLAN tag section 18 Changing the ring size on 32 bit system may result in failure due to lack of memory Therefore mlx4 en will not be able to vmap enough mem ory and the below message will be printed in dmesg vmap allocation for size 528384 failed use vmalloc size to increase size In this case user can enlarge the vmalloc memory by adding vmal loc lt size gt to grub conf Default vmalloc setting is 128M It is recommended to add each time 64M of memory until desired ring size can be allocated Please note that in case vmalloc size is too big the OS will fail to boot so please use caution when adding additional memory For more info refer to http www mythtv org wiki Common Problem vmalloc too small 19 On OEL6 1 with uek1 2 6 32 x x x el6uek ker nel when the number of RX ring is smaller than TX rings kernel
19. f ipoib mtu 5 ALL full Where mtu 5 indicates that all IPoIB ports in the fabric are using 4k MTU mtu 4 indicates 2k MTU 11 Occasionally when using IPoIB in Connected Use the Datagram mode mode the connection might get closed and recov ered only after several minutes 12 Fallback to the primary slave of an IPoIB bond does not work with ARP monitoring https bugs openfabrics org show_bug cgi id 1990 13 Whenever the IOMMU parameter is enabled in To avoid such issue the kernel it can decrease the number of child Decrease the amount of the interfaces on the device according to resource RX receive buffers module limitation parameter the default is The driver will stuck after unknown amount of 512 Decrease the number of RX rings sys fs or ethtool in new kernels Avoid using IOMMU if not required child interfaces creation 14 System might crash in skb_checksum_help Use UD mode in ipoib while performing TCP retransmit involving pack ets with 64k packet size A similar out to the below will be printed kernel BUG at net core dev c 1707 invalid opcode 0000 1 SMP RIP 0010 ffffffff81448988 5 skb_checksum_help 0x148 0x160 Call Trace lt IRQ gt ffffff814488383 5 dev hard start xmit4 0x3e3 0x530 fffffff8144c8055 dev queue xmit0X205 0X550 ffffff8145247d 5 neigh connected_output 0xbd 0x1 15 Changing the GUID of a sp
20. g IP connectivity between the RoCE devices is required when creating an address han dle or modifying a QP with an address vector 84 MLNX OFED v2 1 1 0 0 is not interoperable with older versions of MLNX OFED 85 Unloading mlx4 en while a rdma cm session is established can cause a kernel panic 86 Storage SLES11 SP1 When running multipath rescan while new devices are added to mpath tables multipath may not find all the device mappers 87 Older versions of rescan_scsi_bus sh may not rec If encountering such issues ognize some newly created LUNs it is recommended to use the c flag 88 SRP Reconnecting to a target during host reset stage Run rescan scsi may result in devices going Offline bus sh r 89 SRP Interop The driver is tested with Storage target vendors recommendations for multipath conf extensions ZFS DDN TMS Nimbus NetApp 22 Mellanox Technologies Rev 2 1 1 0 6 Table 8 Known Issues Continued on boot Index Issue Description Workaround 9 DDN Storage Fusion DDN does not accept non default P Key connec 10000 target tion establishment 91 Oracle Sun ZFS stor Occasionally the first command to a LUN may age 7420 not be serviced aborted and cause a successful re connection to the target 92 ZFS does not accept non default P Key connec tion establishment 93 Ungraceful power cycle of an init
21. iator connected with Targets DDN Nimbus NetApp may result in temporary stale connection messages when initiator reconnects 94 iSER On SLES11 the ib iser module does not load Add a dummy interface using iscsiadm e Hd iscsiadm m iface I ib iser o new iscsiadm m iface I ib iser o update n iface transport na me v ib iser In SLES10 SP3 and Ubuntu12 04 need to update user space open iscsi package to version 2 0 873 Trying to disconnect a session while the session is undergoing a reconnect flow may result in discon nection hang Restart iscsid Note Please be aware that doing so might cause rmmod process to hang as the ib_iser module will not be unloaded Unloading ib iser during session disconnect event may result in kernel panic ISER interop Ora cle Sun ZFS storage 7420 Connection establishment occurs twice which may cause ISER to log a stack trace Mellanox Technologies 23 J Rev 2 1 1 0 6 API Changes 10 API Changes 10 1 API Changes in MLNX_OFED Rev 2 1 1 0 0 The following are the API changes in MLNX_OFED Rev 2 1 1 0 0 Table 9 API Changes in MLNX_OFED Rev 2 1 1 0 0 Name Description Dynamically Connected DC The following verbs were added e struct ibv dct ibv exp create dct struct ibv context context struct ibv exp dct init attr attr e int ibv exp destroy dct struct ibv dct dct int ibv exp query dct struct ibv dct dct s
22. interfaces that can be created is 16 Creating more that might cause out of memory issues 5 The default IPoIB operating mode in ConnectX family adapter cards is UD and CM in Connect IBTM 6 Changing the IPoIB mode CM vs UD requires the interface to be in down state 7 IPoIB interface does not function properly if a third party application changes the PKey table We recommend modifying PKey tables via OpenSM 8 When creating a new child interface in an over loaded kernel a dmesg print is displayed advising the user to try again in a few seconds 9 Out of memory issue might occur due to overload To calculate the allowed of interfaces created memory per each IPoIB interface check the follow ing e Num rings min num cores on that device 16 Ring size 512 by default it is module parameter UD memory 2 num rings ring size 8K e CM memory ring size 64k Total memory UD mem CM mem Mellanox Technologies 13 J Rev 2 1 1 0 6 Known Issues Table 8 Known Issues Continued Index Issue Description Workaround 10 The physical port MTU indicates the port capa In order to change the IPoIB bility default value was changed to 4k whereas MTU to 4k edit the the IPoIB port MTU logical MTU default OpenSM partition file in the value is 2k as it is set by the OpenSM section of IPoIB setting as follow Default Oxfff
23. iption Workaround 74 75 76 Resources Limitation The device capabilities reported may not be reached as it depends on the system on which the device is installed and whether the resource is allocated in the kernel or the userspace Occasionally a user process might experience some memory shortage and not function properly due to Linux kernel occupation of the system s free memory for its internal cache To free memory to allow it to be allocated in a user pro cess run the drop_caches procedure below Performing the following steps will cause the kernel to flush and free pages den tries and inodes caches from memory causing that mem ory to become free Note As this is a non destructive operation and dirty objects are not free able run sync first To free the pagecache echo 1 proc sys vm drop caches To free dentries and inodes echo 2 proc sys vm drop caches To free pagecache dentries and inodes echo 3 proc sys vm drop caches Setting more IP addresses than the available GID entries in the table results in failure and the update gid table error message is displayed GID table of port 1 is full Can t add address message Mellanox Technologies 21 J Rev 2 1 1 0 6 Known Issues Table 8 Known Issues Continued Index Issue Description Workaround 71 Ethernet Perfor In a system with more than 61 VFs the 62nd VF
24. n a Para Virtualized environment at Alpha level Contiguous pages Internal memory allocation improvements Register shared memory Control objects QPs CQs Ethernet Performance Enhancements NUMA related and others for 10G and 40G 1 SR IOV Ethernet Time Stamping and Flow Steering are ConnectX 3 HCA capability Mellanox Technologies 11 J Rev 2 1 1 0 6 Change Log History OFED VMA integration to a single branch Ethernet Time Stamping at Beta level Flow Steering for Ethernet and InfiniBand at Beta level Raw Eth QPs Checksum TX RX Flow Steering Errata Kernel upgrade support e YUM update support Storage ISER at Beta level and SRP e 64bit wide counters port xmit recv data packets unicast mcast VERSION query API library and headers 12 Mellanox Technologies Rev 2 1 1 0 6 9 Known Issues The following is a list of general limitations and known issues of the various components of this Mellanox OFED for Linux release Table 8 Known Issues Index Issue Description Workaround 1 IPoIB When user increases receive send a buffer it might consume all the memory when few child s interfaces are created 2 The hardware address suffix of IPoIB interfaces in MLNX OFED v2 0 3 0 0 is a instead of 8 to indicate the TSS support 3 The size of send queue in Connect IB cards cannot exceed IK 4 In 32 bit devices the maximum number of child
25. ocket LLS support Added check for dma mapping errors eIPoIB Added non virtual environment support 10 Mellanox Technologies Rev 2 1 1 0 6 Table 6 New Features Changes and Fixes in v2 1 1 0 0 Sheet 2 of 2 Category Description Hypervisor support KVM and XenServer 8 3 Changes in Rev 2 0 3 0 0 From Rev 2 0 2 0 5 Table 7 New Features Changes and Fixes in v2 0 3 0 0 Category Description Operating Systems Additional OS support e SLES11SP3 Fedoral6 Fedoral7 Drivers Added Connect IB support Installation Added ability to install MLNX OFED with SR IOV support Added Yum installation support EoIB EoIB at beta level is supported only in SLESIISP2 and RHEL6 4 mlx4 core Modified module parameters to associate configuration values with specific PCI devices identified by their bus device function value format mlx4 en Reusing DMA mapped buffers major performance improvements when IOMMU is enabled Added Port level QoS support IPoIB Reduced memory consumption Limited the number TX and RX queues to 16 Default IPoIB mode is set to work in Datagram except for Connect IBTM adapter card which uses IPoIB with Connected mode as default Storage ISER at GA level 8 4 New Features in Rev 2 0 2 0 5 e SR IOV for both Ethernet and InfiniBand at Beta level e RoCE over SR IOV at Beta level e eIPoIB to enable IPoIB i
26. onality eIPoIB requires libvirtd python eIPoIB supports only active backup mode for bonding eIPoIB supports only VLAN Switch Tagging VST mode on guests IPv6 is currently not supported in eIPoIB 3 32 33 34 35 XRC Legacy API is deprecated thus when recompiling applications over MLNX OFED v2 0 3 x x warnings such as the below are displayed rdma c 1699 warning ibv open xrc domain is deprecated declared at usr include infiniband ofa verbs h 72 rdma c 1706 warning ibv create xrc srq is deprecated declared at usr include infiniband ofa verbs h 89 These warnings can be safely ignored XRC is not functional in heterogeneous clusters containing non Mellanox HCAs XRC options do not work when using qperf tool Use perftest instead XRC over ROCE in SR IOV mode is not func tional Out of memory issue might occur due to overload of XRC receive QP with non zero receive queue size created XRC QPs do not have receive queues 36 mlx4 ib module The dev assign str module parameter is not backward compatible In the current version this parameter is using dec imal number to describe the InfiniBand device and not hexadecimal number as it was in previous versions in order to uniform the mapping of device function numbers to InfiniBand device numbers as defined for other module parameters e g num vfs and probe vf 1
27. re the hardware and software requirements of MLNX_OFED Rev 2 1 1 0 6 Linux operating system Administrator privileges on your machine s Disk Space IGB For the OFED Distribution to compile on your machine some software packages of your operat ing system OS distribution are required To install the additional packages run the following commands per OS Table 3 Additional Software Packages Operating System Required Packages Installation Command RHEL OEL Fedora yum install pciutils python gcc gfortran libxml2 python tesh libnl 1686 libnl libnl devel expat glib2 tcl libstdc bc tk XenServer yum install pciutils python libxml2 python libnl expat glib2 tcl bc libstdc tk OpenSUSE zypper install glib2 tools pciutils python libxml2 python tesh libnl 1 1 32bit libstdc 46 expat libnl 1 1 devellibnl 1 1 tcl be tk SLES 11 SP1 zypper install pciutils python libxml2 python tesh libnl libstde 43 libnl devel expat glib2 tcl bc libnl 1586 tk SLES 11 SP2 zypper install pciutils python libnl 32bit libxml2 python tesh libnl libnl devel libstdc 46 expat glib2 tcl bc tk SLES 11 SP3 zypper install pciutils python libnl 32bit libxml2 python tesh libstde 43 libn libnl devel expat glib2 tcl bc tk Ubuntu Debian apt get install dpkg autotools dev autoconf libtool automake1 10 automake m4 dkms debhelper tcl tcl8 4 chrpath swig graphviz tcl dev tcl8 4 dev tk dev tk8 4 dev bison flex dpatch zlib1 g dev curl libcu
28. rl4 gnutls dev python libxml2 libvirt bin libvirtO libnl dev libglib2 0 dev libgfortran3 8 Mellanox Technologies Rev 2 1 1 0 6 6 Supported HCAs MLNX_OFED Rev 2 1 1 0 6 supports the following Mellanox network adapter cards Connect IB Rev 10 10 2000 and above e ConnectX 3 Pro Rev 2 30 8000 and above e ConnectX 3 Rev 2 30 8000 and above ConnectX 2 Rev 2 9 1200 and above For official firmware versions please see http www mellanox com content pages php pg firmware download 7 Compatibility MLNX OFED Rev 2 1 1 0 6 is compatible with the following Table 4 MLNX OFED Rev 2 1 1 0 6 Compatibility Matrix Mellanox Product Description Version MLNX OS MSX6036 w w MLNX OS version 3 3 4304 Grid Director 4036 w w Grid Director version 3 9 2 992 FabricIT EFM 1S5035 w w FabricIT EFM version 1 1 3000 FabricIT BXM MBX5020 w w FabricIT BXM version 2 1 2000 Unified Fabric Manager UFM v4 6 MXM v2 1 ScalableUPC v2 2 ScalableSHMEM v2 2 FCA v2 5 OMPI v1 6 5 MVAPICH v1 9a CD v1 0 a MLNX OFED v2 1 1 0 6 was tested with this switch however additional switches might be supported as well 1 ConnectX 2 does not support all the new functionality of MLNX OFED 2 0 3 XXX For the complete list of the supported features per HCA please refer to the MLNX OFED User Manual Mellanox Technologies 9 J Rev 2 1 1 0 6 Change Log History
29. truct ibv exp dct attr attr 10 1 1 Verbs Extension and Verbs Experimental APIs e Verbs Extension API Verbs extension API defines OFA APIs extension scheme to detect ABI compatibility and enable backward and forward compatibility support Verbs Experimental API Verbs experimental API defines MLNX OFED APIs extension scheme which is similar to the Verbs extension APT This extension provides a way to introduce new features before they are integrated into the formal OFA API and to the upstream kernel and libs The following are the Verbs Extension and Verbs Experimental APIs in MLNX_OFED Rev 2 1 1 0 0 Table 10 Verbs Extension and Verbs Experimental APIs API Type APIs Verbs Extension API ibv_post_task ibv_query_values_ex ibv_query_device_ex ibv_create_flow ibv_destroy_flow ibv poll cq ex ibv reg shared mr ex ibv open xrcd ibv close xrcd ibv modify cq ibv create srq ex ibv get srq num ibv create qp ex ibv create cq ex ibv open qp ibv modify qp ex 24 Mellanox Technologies Rev 2 1 1 0 6 Table 10 Verbs Extension and Verbs Experimental APIs API Type APIs Verbs Experimental API e jbv exp create_qp e ibv_exp_query_device e jbv exp create dct e ibv exp destroy dct e ibv exp query dct 10 2 API Changes in MLNX OFED Rev 2 0 3 0 0 The following are the API changes in MLNX OFED Rev 2 0 3 0 0 Table 11 API Changes in MLNX OFED Rev 2 0 3 0 0
30. y issues may rise during drivers load depending on the values of the driver module parameters set e g log num cq Mellanox Technologies 17 J Rev 2 1 1 0 6 Known Issues Table 8 Known Issues Continued Index Issue Description Workaround 43 When reloading starting the driver using the Remove the third party etc init d openibd the following messages RPM non MLNX OFED are displayed if there is a third party RPM or drivers directory run driver installed depmod and then rerun Module mlx4 core does not belong to etc init d openibd MLNX OFED restart or Module mlx4 core belong to rpm name gt which is not a part of MLNX OFED 44 Occasionally when trying to repetitively reload the nes hardware driver on SLES11 SP2 a soft lockups occurs that required reboot 45 In ConnectX 2 if the driver load succeeds the informative message below is presented convey ing the below limitations If port type is IB the number of maximum sup ported VLs is 4 If port type is ETH then the maximum priority for VLAN tagged is 3 mlx4 core 0000 0d 00 0 command SET PORT 0xc failed in param 0x120064000 in mod 0x2 op mod 0x0 fw status 0x40 46 Operating Systems RHEL 5 X and SLES 10 SPX are currently not supported 47 SR IOV When using legacy VMs with OFED 2 0 2 0 5 hypervisor the enable 64b cqe eqe parameter must be set to zero on the hypervisor

Mellanox OFED for Linux Release Notes

Contents

Download Pdf Manuals

Related Search

Related Contents