Mellanox WinOF VPI Release Notes
Contents
1. 25 6 1 Generic SSeS AA SRO UT Ses ERES 25 6 2 InfiniBand Issues driest kadar HA 27 0 37 Ethernet IS eed esee coats ser solute wees whos avant ahd tee alate ire ats 29 6 4 Quality of Service Issues 31 6 5 ROCENVGRE Issues zeureberesv area ea ed WA te 31 6 6 Performance UE INA eae Bae eae Ba Wa 32 6 7 IPoIB Issues ches mo P eet aa e Vadis ain Gates arg yeaa 32 6 5 Hyper V ISSues oper AB UAR HEU ese k Mace we hod what ess 33 6 9 Installation Upgrade Issues 34 6 10 Utilities Issues 2 cect RI e 37 6 11 CIM WMI Issues seston MU AI e nee Mkaa 38 Chapter 7 APIChanges 39 Mellanox Technologies 3 J 7 1 API Changes in WinOF Rev 4 80 50000 39 7 2 API Changes in WinOF Rev 4 70 39 7 3 API Changes in WinOF Rev 4 60 39 74 API Changes in WinOF Rev4 55 40 7 5 API Changes in WinOF Rev 4 40 40 4 Mellanox Technologies Rev 4 80 50000 List of Tables Table Is Firmware Versions yeo edu ance EA Ae eee eee nar e ce e ura 8 Table 2 New Features Changes and Fixes in v4 80 50000 9 Table 3 New Features Changes and Fixes in v4 70 50040 10 Table2. WMI does not work due to lack of permissions Change the execution policy Run Set ExecutionPolicy AllSigned WinOF 4 40 and above IPoIB driver is not IPv6 com patible with earlier driver versions including Win dow 8 Inbox driver If WinOF 4 50 IPoIB node receives an icmpv6 mes sage from the legacy IPoIB node the following event will appear in the event log IPOIB driver detected a possible incom patibility with Windows 8 inbox IPv6 support due to which there is no IPv6 connectivity To enable compatibility mode add Win8InboxCompatibilityMode REG SZ registry key with the value of 1 in the IPoIB interface reg Istry Note All IPoIB nodes must use the same mode to enable IPv6 compatibility with earlier driver versions We recommend upgrading all hosts to the new driver version from http mellanox com or use Windows 8 compatibility mode Functionality such as CQ to EQ Auto Sensing QoS are not supported in ConnectX 2 adapter cards On rare occasions as a result of port configuration change IB ETH the UI may get stuck for up to a few minutes This effect does not require any user action The UI returns to its proper functionality after a few minutes WinOF Inbox driver does not support upgrade When installing WinOF v4 40 and above on a Windows Server 2012 and above machine the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost The
3. 6 4 6 5 Mellanox Technologies 31 J Table 16 Ethernet Issues Issue Workaround Disabling the ethernet adapter with more than 25 VLANs configured over windows 2008 R2 windows 7 may result with a non responding server Reduce the number of configured VLANs prior to disabling the ethernet adapter Virtual Ethernet Interfaces created by vea_man are not tuned by the automatic performance tuning script For optimal performance need to follow the perfor mance tuning guide and apply relevant changes to the VEA interface In SR IOV 40Gbps interfaces of vSwitch interface on HyperV and the VMNIC are identified as 10Gbps Despite the incorrect speed identification they still achieve 40Gbps performance Quality of Service Issues Table 17 Quality of Service Issues Issue Workaround Running Quality of Service QoS commands without the parameter PolicyStore ActiveStore may cause machines to load without Quality of Ser vice policy Store the QoS policy in the ActiveStore RoCE NVGRE Issues Table 18 ROCE NVGRE Issues Issue Workaround RoCE does not support Traffic cannot go through the router It works in the same subnet only Multicast traffic VLAN e Layer 3 feature In machines with heterogeneous NICs a NIC which supports RoCE v2 and a NIC which does not support RoCE v2 the following issus might raise Connect X 3 Pro is loaded
4. API Changes in WinOF Rev 4 40 Name Description VERBS MINOR VER Increased its value 0009 gt 000a enum eth link speeds Added enumerated values struct ib port attr t The mtu field was separated into two fields max mtu maximum MTU supported by the port active mtu actual MTU which the port is configured with e Added the eth link speed field WR SEND INV Added enumerated values struct ib send wr t The type of invalidate rkey was changed from net32 t gt ib net32 t IB SEND OPT SKIP DOORBELL Added the send Write flag 40 Mellanox Technologies
5. Inbox driver will be reinstalled automatically when the new driver is uninstalled Without separate ports for each stream WinSock multiplexes every packet to every subscriber socket and then filters it out Use different UDP ports to get higher performance when using multicast packets A virtual IPoIB interface created by the part man utility reports an Active state when the physical link is in the Initializing state and OpenSM is not running in the subnet The Packets Received Discarded and Packets Received Errors counter may display wrong results Mellanox Technologies 27 J Rev 4 80 5009 Known Issues Table 15 InfiniBand Issues Issue Workaround Firmware upgrade may fail during installation if there Upgrade the firmware manually was a prior firmware upgrade on the machine without a reboot after it A firmware upgrade failure does not fail the whole installation The drivers functionality is limited up to 128 cores Connection failure on ND tests while machine A have IBAL provider and machine B have MLX4ND pro vider Hibernate and Sleep are not functional when user space is using its resources Calling ib join mcast with timeout ms 0 Avoid calling this function with timeout ms 0 may result in a BSOD IPoIB does not support MAC address change e QoS packet priority e Load balancing and fail over LBFO Connected Mode
6. Partition Memory registration on 32bit machines is limited to up to 256GB In an interoperability environment that has both Linux and Windows OSs the MTU value must be the same otherwise packets larger than the minimum will not go through The default MTU for Linux is 2K and for Windows is 4K OpenSM does not run as a service during installation To run OpenSM as a service assuming the package since the SM requires the GUID parameter to decide was installed in the default path use which port to work on Setting it on setup causesitto sc create opensm binPath c Program work only on the first port and not the others Files Mellanox MLNX_VPI IB Tools opensm exe To start the service run sc start opensm Tools issues e ibportstate does not work on RoCE ports e ibdiagpath may crash on Hyper V machines If an application which uses InfiniBand runs while Stop all InfiniBand applications including OpenSM the driver is being restarted a bluescreen or an NMI upon driver restart may occur If OpenSM is up during driver restart on the same To resume operation stop OpenSM and restart the machine it might stop working when the driver 1s driver while OpenSM is down being backed up 28 Mellanox Technologies Table 15 InfiniBand Issues Issue Workaround Sometimes during traffic the latency of the IPoIB can get higher this behavior is inconsistent Set the following re
7. The new driver advertises its TSS capa bility using multiple TX rings This feature will not be used when communi cating with older version drivers such as the inbox driver This way IPv4 communication in a mixed environment that includes the inbox driver should work IPv6 communication may be problematic in such an environment Eliminated some race conditions in IPoIB mcast handling Added mcast addresses according to the Windows mcast table in addition to mcast addresses from igmp mld queries in order to connect to such mcasts that are not reported by the queries Removed redundant LID CHANGE events This fixes the redundant recon nection of all IPoIB mcasts e Added support for VM migration When a VM is migrated from one host to another it does not send a new ARP to hosts it already knew when it was sitting on the first host In order to prevent communication interruption to such hosts the driver will initiate an ARP to learn them again Fixed IPoIB VMQ affinity update bugs Fixed IPoIB VMQ parent queue management race during reset Fixeda bug in passing DHCP packets from a Linux VM Fixed bug in RDMA statistics for NDK Removed affinity restriction to group 0 for NDK e Added support for the part man tool It can create only one additional IPoIB interface per port GUID This feature can be utilized to use different IPoIB interfaces for SMB and Hyper V The virtual ports configuration 1s removed during uninstall Ins
8. creating extra copy for each packet This may lead to packet drops on the link Pcap file issues for RoCE IP Based The packets capture time field is wrong may be 0 or negative numbers For every captured packet an additional 0 size flame is added This appears in Wireshark as a malformed Ethernet packet Sniffing over IB ports is currently not supported Mellanox Technologies 37 J Rev 4 80 5009 Known Issues 6 11 CIM WMI Issues Table 24 CIM WMI Issues Issue Workaround Running Microsoft CIM cmdlets operations and their Use DriverCoreSettings instead derived classes on classes MLNX NetAdapterSettingData and MLNX NetAdapterRoceSettingData is not sup ported Calling those commands may cause the debugger 1f connected to the machine to assert For PCI Gen3 PcieLinkSpeed is reported as Unknown when running Get NetAdapterHard wareInfo Powershell cmdlet 38 Mellanox Technologies 7 API Changes 7 1 API Changes in WinOF Rev 4 80 50000 The following are the API changes in WinOF Rev 4 80 50000 Table 26 API Changes in WinOF Rev 4 80 Name Description RDMA TRANSPORT RDMAOE 1 It is an alias to RDMA_TRANSPORT_RDMAOE RDMA TRANSPORT RDMAOE 1 25 Added enumerated values is roce is mac based roce Added new functions is ip based roce is rroce or ip based roce struct ib wc t p next was replaced with an anonymous union
9. disabling QoS Enabled MaxRssProcessirs support of the following values 1 2 4 8 16 32 64 Network Direct Fixed a crash occurred when more than 4 SGEs elements were used in an ND write operation Network Direct Fixed the swap of InboundReadLimit and OutboundRead Limit when creating an EndPoint and in Connector GetConnectionData Network Direct Fixed disallowing creation of EndPoint with zero attributes in the Receive Queue Network Direct Removed the option of NDK registration failure requiring a reboot of the machine to register it again Network Direct Fixed a failure when creating an EndPoint with zero attributes in the Receive Queue e Network Direct Added the option of sensing the incoming Read messages according to the device capabilities when creating an EndPoint limit e Network Direct Fixed a failure of ND connectivity between VMs on the same host e Added Transmit Side Scaling TSS Added Ethernet QoS proprietary counters diagnostics and traffic for monitor ing using Windows perfmon utility e Added to the MTU size the IP header size 1500 gt 1514 9600 gt 9614 Thus the minimum Jumbo frame size is 614 Interrupt moderation supports the following profiles Low Latency Moderate Aggressive In addition to old values that are not supported anymore Mellanox Technologies 17 J Rev 4 80 5000d Changes and Major New Features Table 10 New Features Changes and Fixes in v4 40 Shee
10. e Fixed an issue that prevented receiving ARP traffic in NVGRE mode 12 Mellanox Technologies 2 4 New Features Changes and Fixes in Version 4 60 2 4 1 New Features Changes and Fixes in v4 60 17738 from v4 60 17718 This package version is 4 60 17738 The package contains the following versions of components Bus and eth driver version 4 60 17718 y WA The CIM provider version is 4 60 17718 The mux driver version is 4 60 17729 The IPoIB driver version is 4 60 17736 Table 7 New Features Changes and Fixes in v4 60 17738 Category Description IPoIB Fixed using CQ after VMQ is closed Fixed bad completion of VMQ QP that was caused by malformed WR Ethernet LBFO Fixed the team s MAC address uniqueness in the subnet of the team in Windows 2008 R2 2 4 New Features Changes and Fixes in v4 60 17718 from v4 55 This package version is 4 60 17718 The package contains the following versions of components Bus eth IPoIB and mux drivers version is 4 60 17718 The CIM provider version is 4 60 17718 Table 8 New Features Changes and Fixes in v4 60 17718 Category Description Installation Upgrade Enabled configuration changes saving upon Inbox and previous releases upgrade Enabled CIM installation as a standalone package e Fixed an issue occurred when uninstalling and reinstalling the driver The Con nectX 3 Pro Ethernet device was displayed in the Dev
11. failure occurred when setting the IPoIB adapter value to SA Query Timeout Fixed propagation of the physical link disconnection to virtual part man interface Fixed BSOD caused by calling ib join mcast with timeout ms 0 Performance improvements in latency Ethernet Added DSCP support over IPv4 e Added traffic profile Added IRQ dynamic moderation Modified the CQ size to prevent CQ overrun Changed the report link speed zero in case of disconnected network adapter LBFO Fixed port channel teaming with CISCO switch and Fabric Extenders traffic loose in Windows Server 2008 R2 e Fixed an issue related to packets sent with corrupted VLAN header when they were meant to be untagged Fixed unexpected behavior upon QP asynchronous event Fixed the ability to disable Wake on Lan WoL on NICs which supports it Stability fixes Performance improvements 14 Mellanox Technologies Table 8 New Features Changes and Fixes in v4 60 17718 Category Description WMI CIM Added ControlledBy association to IBPort Fixed ConformsToProfiles association for Softwareldentity and Driverldentity Fixed execution of all tests which were running when executing Diagnostic tests on one instance Fixed a failure occurred when running MLNX Card Fixed the printing of diagnostics log e Fixed an issue preventing from get event to show information after disabling the PCI device e Removed support for the
12. is associated with the default classification to these egress packets The default traffic classifica tion has the following attributes e It has a traffic classification condition of type NDIS QOS CONDITION DEFAULT Itis the first traffic classification defined in the array of NDIS QOS CLASSIFICATION ELEMENT structures RDMA Activity counters do not count during Net workDirect RoCE traffic In SR IOV mode all ports are in pure Ethernet mode and RDMA is not supported on either port In SR IOV mode enabling SR IOV in the BIOS may change the interfaces names If any VSwitch is bounded to an interface whose name was changed there will not be any way to perform any operation on them Unbond all Vswitches from the NIC s interfaces before enabling SR IOV in the BIOS GRE traffic steering by inner MAC and by outer MAC simultaneously is currently not supported Configure steering or by inner MAC or by outer MAC If VMQ set filter requests are accepted without a GRE flag i e requested steering by outer MAC the GRE packets do not reach that VMQ Set the bus driver registry keyAcceptGREbyOuterMAC P1 2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering rule to L2 TUNNELING rule for each VMQ set filter request without GRE flag Note For regular NVGRE Hyper V scenarios the value of the registry key below must be set to 0 AcceptGREbyOuterMAC P1 2 30 Mellanox Technologies
13. multiple processor groups e Set an adequate number of MTTs to map all physical memory e Allocated firmware and ICM memory in chunks of non paged memory instead of using contiguous physical memory Fixed RSS indirection table mapping building when there are less RX rings than RSS cores Fixed a bug preventing standard work with BAR value more than 4GB Fixed memory leaks Fixed error flows causing a Bluescreen in driver startup unload Fixed a Bluescreen occurrence upon shutdown due to leak in active resources Changeddevice names in device manager and their hardware IDs The changes were made to distinguish between ConnectX 2 and ConnectX 3 for ConnectX 2 MLX4 ConnectX 2_Eth and IBA ConnectX 2_ IPoIB for ConnectX 3 MLX4 ConnectX 3_Eth and IBA ConnectX 3_IPoIB Set QoS settings only for ConnectX 3 Changing the hardware ID forces the OS to install new device and re build the registry keys Added an event log to indicate driver failure upon start if there are two HCA burned with the same GUID Added firmware upgrade support as part of the setup process The setup burns the new firmware only on Mellanox cards Firmware burning failure does not prevent the driver s installation therefore it will show a warning In this case it is recommended to update the firmware manually Enabled configuration of TxRingNum registry key from the UI Improved the Port Protocol dialog Added Registry key documentation to
14. optimizes 4 60 and above or on servers with no internet access loading PowerShell DLLs the first PowerShell command might be stuck for 2 SEnv PATH Runtime InteropSer 3 minutes before its completed vices RuntimeEnvironment GetRuntime According to the following content this issue is Directory l related to Net framework version or an issue with the APPDomain CurrentDomain GetAssem blies gt internet access ant L ti der pt Location http www minasi com forum if 1 pt continue topic asp TOPIC ID 39253 if Scn na Split Path Leaf pt Write Host ForegroundColor Yellow NGENing na ngen install pt On ConnectX 3 cards only when upgrading from Reconfigure the RoCE Mode setting manually Windows 2012R2 Inbox driver to WinOF the RoCE mode setting in the registry is not properly transferred to the new driver In case a non default value was used it will not be configured following the upgrade Uninstall from the Device Manager is currently not supported 6 10 Utilities Issues Table 23 Utilities Issues Issue Workaround ibdump may encounter packet drops upon a burst of more than 4096 or 2 max burst packets Packets loss is not reported by ibdump Running ibdump on a RoCE Ethernet port may Verify Ethernet flow control is enable to ensure a decrease the functional bandwidth due to the over lossless link head of
15. response by sender when using DSCP untag priority tag mode with ETS enabled 10 Mellanox Technologies 2 2 3 New Features Changes and Fixes in v4 70 50000 from v4 61 This package version is 4 70 50000 The package contains mixed versions of components Bus eth IPoIB and mux drivers version is 4 70 10126 The CIM provider version is 4 70 10130 Table 5 New Features Changes and Fixes in v4 70 50000 Category Description Installation Upgrade Fixed removal of virtual IPoIB ports in uninstallation All user space binaries are now signed Fixed restoration process of DNS servers during upgrade Fixed popping windows during installation upgrade Fixed missing 32 bit files in the catalog files Generic Changed Ethernet and IPoIB event log messages to be more clear Ported SDK project to Visual Studio 2013 Fixed an issue which caused Mellanox miniport devices to be listed in Devices and Printers Fixed Ethernet and IPoIB deadlock in power state change during shutdown reboot Fixed stability issues IPoIB Added support for IPoIB SR IOV Virtual Function VF over KVM Hypervisor Beta level Added support for non default pkey as queried from OpenSM on IPoIB SR IOV VF over KVM Added IPoIB QoS proprietary counters diagnostics and traffic for monitoring using Windows perfmon utility Fixed part man exit with return value 0 in case of error Ethernet Added support f
16. the vSwitch to the ETH IPoIB device Reattach the vSwitch to sync with the value set in the changing the Jumbo Packet registry key on the ETH IPoIB device ETH IPoIB device does not affect the vSwitch con figuration and vice versa For example if the user sets the Jumbo Packet on the ETH IPoIB device to X and the Jumbo Packet on the vSwitch to X Y X Y sized packets will be passed from NDIS down to the driver and they will be dropped by it Unexpected behavior might occur when running ina virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM When moving an IPoIB interface ina VM from non VMQ to VMQ or from VMQ to non VMQ a reset to the NIC may occur and in the event log the following message will appear device reports a CQE error on cgn number gpn number Status number Therefore the HCA Nic will be reset The issue is reported in Function function For more information refer to details In IPoIB when using long Multicast traffic from a Virtual Machine VM to an external host there might be up to 0 5 loss in 5 bursts Hyper V is at low bandwidth on LBFO vS witch Win dows 2012 In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts when using MS MUX over the Ethernet driver use the no
17. with the transport type RoCE v2 Connect X 3 is loaded with the transport type RoCE When NVGRE off load is enabled the GRE traffic cannot be accepted as a regular L2 traffic and requires special L2 TUNNELING steering rules In such case the GRE packets are dropped or directed to promiscu ous queue Rev 4 80 5009 Known Issues 6 6 6 7 32 Mellanox Technologies Table 18 ROCE NVGRE Issues Issue Workaround Using different versions of RoCE in your cluster is not supported Use the same RoCE version in all the cluster in the Ethernet ports Performance Issues Table 19 Performance Issues Issue Workaround When using WinOF 4 40 or above low throughput will be seen on 40GbE adapters when QoS is enabled Disable QoS when it is not in use e Open a PowerShell prompt Run Disable NetAdapterQos name lt Inter face Name gt where lt Interface Name gt is e g Ethernet 1 perf tuning is supported only when one of the two NUMA nodes are in use Running performance benchmarks for a short period of time 1 sec may provide bad latency in IPoIB and Ethernet Set Rx Interrupt Moderation Profile and Tx Interrupt Moderation Profile to Low Latency to avoid bad latency Note This may increase CPU utilization The driver uses optimal interrupt moderation values for 10 GbE SR IOV VF scenario For other scenarios the optimal value
18. 1 Firmware Versions ol DRE RR EDI HERR CURE ERREUR ERES 8 Chapter2 Changes and Major New Features 9 2 1 New Features Changes and Fixes in v4 80 50000 from v4 70 9 2 2 New Features Changes and Fixes in Version 4 70 10 2 2 1 New Features Changes and Fixes in v4 70 50050 from v4 70 50040 10 2 2 2 New Features Changes and Fixes in v4 70 50040 from v4 70 50000 10 2 2 3 New Features Changes and Fixes in v4 70 50000 from v4 61 11 2 3 New Features Changes and Fixes in v4 61 From v4 60 12 2 4 New Features Changes and Fixes in Version 4 60 13 2 4 1 New Features Changes and Fixes in v4 60 17738 from v4 60 17718 13 2 4 2 New Features Changes and Fixes in v4 60 17718 from v4 55 13 2 5 New Features Changes and Fixes in v4 55 From v4 40 15 2 6 New Features Changes and Fixes in v4 40 From v4 30 16 2 7 New Features Changes and Fixes in v4 30 From v4 20 19 2 8 New Features Changes and Fixes in v4 2 From v3 2 0 20 Chapter3 Features Support to be Deprecated in the Next Release 22 Chapter 4 Beta Features AAA IKAWA 23 Chapter 5 Unsupported Functionality Features 24 5 1 ConnectX 2 Adapter Limitations 24 Chapter 6 Knownlssues
19. 22 Installation Upgrade Issues 34 Table 23 Utilities Issues atune exe ee d eae doe ee ala e dats e le aa 37 Table 24 CIM WMI Issues ooto can nian Tan kina mm ras 38 Table 25 WinOF Known Issues 38 Table 26 API Changes in WinOF Rev 4 80 39 Table 27 API Changes in WinOF Rev 4 70 39 Table 28 API Changes in WinOF Rev 4 60 39 Table 29 API Changes in WinOF Rev 4 55 40 Table 30 API Changes in WinOF Rev 4 40 40 Mellanox Technologies 5 J Rev 4 80 5009 Overview 1 1 1 1 2 Overview These are the release notes for Mellanox WinOF Rev 4 80 50000 VPI drivers The driver provides improved performance and additional functionality compared to the Inbox driver provided in Windows 2012 and Windows 2012 R2 Therefore Mellanox strongly recom mends updating to its latest driver Mellanox WinOF Rev 4 80 50000 New Features Added support for the following InfiniBand IPoIB multiple Pkey support beta level Ethernet RoCE RoCEIP Based compatible with the MLNX OFED Linux driver RoCE mode Readactive RoCE configuration from hardware RSS in UDP enabled by default 56 Gb s Ethernet e SR IOV over InfiniBand S
20. 4 New Features Changes and Fixes in v4 70 50040 10 Table 5 New Features Changes and Fixes in v4 70 50000 11 Table 6 New Features Changes and Fixes in v4 61 12 Table 7 New Features Changes and Fixes in v4 60 17738 13 Table 8 New Features Changes and Fixes in v4 60 17718 13 Table9 New Features Changes and Fixes in v4 55 15 Table 10 New Features Changes and Fixes in v4 40 16 Table 11 New Features Changes and Fixes in V4 3 19 Table 12 New Features Changes and Fixes in V4 2 20 Table 13 Beta Eeat tesc iue tars epus Sept ak aedes asd edu ace each ede bes 23 Table T4 Generic Issues oi cec Pg UERPLRE IGI E MERE S GIN OUS eI eS 25 Table 15 InfiniBand Issues erria ienen pin a eh a 27 Table lo EthemetISsues 5 ue ge e RERO E POE UEBER OE ee Maas SCENE LUEQU S 29 Table 17 Quality of Service Issues a rere ccr cere WAN ieir ene Dere ra 31 Table 18 ROCENV GRE ASSUCS scs ev ok EI D EYE UON A Oe Se Raw eee ESTO 31 Tabl 19 Performance ISsues HE oe ed PREMO ERES 32 Table 20 IPoIB Issues 32 Table 21 Hyper VISU ected fauce bes eee Re SONS Ope ee ee RUE Sd Non 33 Table
21. EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanoz Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX Connect KO Connect IB CoolBox amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS TestX PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT HPC X Mellanox Open Ethernet Mellanox PeerDirect Mellanox Virtual Modular Switch MetroDX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Rev 4 80 50000 Table of Contents Wableof AI KA aa 3 List f Tables eek AA AAA AA aa 5 Chapter OVOEVIEW oves IA AA d aate 6 1 1 Mellanox WinOF Rev 4 80 50000 New Features 6 1 2 WinOF VPI Package Contents 6 1 3 Supported Native Operating System Versions 7 1 4 Supported Network Adapter Cards 8 1 4
22. JM Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox WinOF VPI Release Notes Rev 4 80 50000 Last Updated November 23 2014 www mellanox com Rev 4 80 50000 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION
23. R IOV InfiniBand over KVM Hypervisor IPv6 support of IPoIB in an SR IOV guest OS over KVM is at beta level Non default P Key in VM Reset Flow SR IOV Generic Windows 7 Chent 64 bit only Windows 8 1 Client 64 bit only WinOF VPI Package Contents The Mellanox WinOF Rev 4 80 50000 for Windows package contains the following compo nents Core and ULPs IB HCA low level drivers mlx4 IB Access Layer IBAL Ethernet driver ETH IP over InfiniBand IPoIB NetworkDirect ND Mellanox LBFO driver for Windows 2008 R2 1 Requires firmware v2 32 5100 6 Mellanox Technologies J Utilities e OpenSM InfiniBand Subnet Manager is provided as a sample code The sample code is intended to allow users to test or bring up the InfiniBand fabric without a management console switch to get started For cluster production environments Mellanox s recommendation is to use a Managed Switch or the UFM SDN Appliance Low level performance tools nfiniBand Diagnostics tools e CIM PowerShell and WMI support Software Development Kit SDK Documentation 1 3 Supported Native Operating System Versions The following describes the supported operating systems and their roles in a virtualization envi ronment Native Windows Server 2008 R2 64 bit only Windows Server 2012 64 bit only Windows Server 2012 R2 64 bit only Windows 7 Client 64 bit only Windows 8 1 Client 64 bit only H
24. When working with LBFO the teamed interface dis appears after machine reboot The issue applies to Windows Server 2012 R2 Delete the existing teamed interface and create a new one WakeOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC When the ports of the device are configured as Ether net only ibstat vstat may display wrong information Mellanox Technologies 29 J Table 16 Ethernet Issues Rev 4 80 5000d Known Issues Issue Workaround High multicast drop rate on multicast storming Use Multicast traffic tuning option under the per formance tab For further information please refer to section Tun able Performance Parameters in the User Manual Driver installation requires deletion of the mlx4 bus sys file in Windows Server 2008 R2 and WLH OSs when using the PXE package Delete the m1x4 bus sys file and reboot the machine to install the driver When there is a stress in TCP connection establish ments some of those connections may fail Increase the Ring queue sizes e ReceiveBuffers controls the receive ring size TransmitBuffers controls the transmit ring size The DCB component specifies a default traffic classi fication that is applied to all egress packets that do not match other classification conditions In this case the network adapter assigns the IEEE 802 1p priority level that
25. ame PCI slot Configuration can be restored only in Windows Server 2012 and above IPv6 configuration restore is not supported Upon upgrade the following Registry Key values will be overwritten with the following e ReceiveBuffers 512 e MaxRssProcessors 8 RssBaseProcNumber 0 NumRSSQueues 8 e RssMaxProcNumber 63 RssProfile 1 DefaultRecvRingProcessor 1 TxInterruptProcessor 1 e TxForwardingProcessor 1 e RxIntModerationProfile 1 TxIntModerationProfile 1 e RecvCompletionMethod 1 SingleStream 0 TxRingNum 8 Upon upgrade the following Ethernet Registry Keys will be deleted e SendCompletionMethod e UseRSSForRawIP e UseRSSForUDP Upon upgrade the SendCompletionMethod IPoIB Registry Key value will be modified as follow e SendCompletionMethod 0 Mellanox Technologies 35 J Rev 4 80 5009 Known Issues Table 22 Installation Upgrade Issues Issue Workaround Upon upgrade the following IPoIB Registry Keys will be deleted e UseRSSForRawIP e UseRSSForUDP Uninstalling the driver on Windows Server 2008 R2 Choose Do not close applications This action with LBFO configuration results in the appearance of allows the uninstallation of the driver A Reboot may a pop up window requesting to close several running be required applications Rebooting the server before uninstalling the driver when LBFO is co
26. ded support for LID change event Addedenhancementsin part man forthe multiple Pkey support Changed IPv6 all dhcp servers mcast to be persistent Fixed rare cases of driver hang following a Subnet Manager failover event Fixed stability issues Ethernet Added RSS in UDP enabled by default Added 56 GbE Please refer to the Infiniband Switch User Guide for further details Changed DSCP configuration to be per port instead of global Network Direct Fixed race in NDK between handling of incoming connection and destruction of a listener Network Direct Fixed race between NDK object creation and usage Improved TCB Transmission Control Block management on send Improved transmit and receive in multi stream scenarios Enabled hardware checksum offload for non TCP UDP traffic with ConnectX3 Pro Improved stability when handling OIDs during driver reset Fixed performance tuning for 1GbE link Fixed possible reset of driver during migration of large number of VMs at the same time Fixed stability issues RoCE Added RoCE IP based ND Fixed wrong return value in IND2Adapter QueryAddressList Mellanox Technologies 9 J Rev 4 80 5009 Changes and Major New Features Table 2 New Features Changes and Fixes in v4 80 50000 Category Description Infiniband Added non default PKey in VM Performance Optimized interrupt moderation values in SR IOV VF mode for IPoIB Impro
27. detected Improved resiliency on error flow in Ethernet IPoIB and bus drivers Fixed an issue which caused Mellanox devices to be listed in Devices and Printers and had Safe Removal UI Performance e Added support OF IPv6 to all nd tests Enabled optimal interrupt moderation values in SR IOV VF mode e Stopped using NdisQueryNetBufferPhysicalCount to improve CPU utilization IPoIB Enabled searching for IBAT routes based on dest only instead of src dest and added a mechanism preventing memory growth in IBAT entries Allowed any number of RSS processors not only a power of 2 Ensured SR IOV mode is not enabled for IPoIB ports which resulted in con fusing message in event log Fixed error statistics collection which could cause false error report Fixeda connectivity problem between Hyper V VMs on the same host Fixed loopback issues in the virtualization environment Fixed stability issues Ethernet e Added support for unknown link state indication e Added support for DMA checks by driver verifier on SR IOV Virtual Function e Added support for NVGRE over LBFO Team Improved performance of handling change receive ring affinity request In SR IOV mode improved resiliency to driver failures in the Virtual Machine which could result in driver load failure in VM In SR IOV mode improved resilience in VF to PF communication Improved structure of INF file for SR IOV Physical and Virtual Functions
28. e network interface s state incor rectly even after disabling and re enabling the dis connected network interfaces To see the correct state and link speed perform one of the following RunGet netadapter Powershell cmdlet or e Right click on that interface from Network Con nections and click on status SR IOV should be enabled only when both ports are Ethernet If SR IOV is enabled when both ports are in auto sensing mode the ports type is forced to be Ethernet WoL is not supported on Windows 2008 R2 Performing the following scenario causes memory leak of 64 bytes on the Virtual Machine Creating VFs Restarting VM associated to VF s Disabling MLNX driver on the Hypervisor Do not disable the MLNX driver on the Hypervisor iSCSI boot over Windows 2008 R2 is not supported Clearing the Mellanox device counters through perfmon does not always work Restart the driver When running in SRIOV mode and Hyper V the same driver version should be used on both the Hypervisor and the virtual machine 26 Mellanox Technologies InfiniBand Issues Table 15 InfiniBand Issues Issue Workaround InfiniBand application that was compiled with an SDK version earlier than WinOF v4 40 is not binary compatible Recompile InfiniBand application with WinOF v4 40 and above ND application is backward compatible and older applications over ND do not have to be recompiled
29. following configuration e ModeFlags SingleMsixNum MultiMsixNum SingleEqNum MultiEqNum MaxContQuant SlaveNum DebugLevel DebugFlags UsePrio NumFcExch EnableQoS BlockMcastLoopBack nterruptFromFirstPacket ProbeVf a Requires firmware v2 30 8000 and above 2 5 New Features Changes and Fixes in v4 55 From v4 40 Table 9 New Features Changes and Fixes in v4 55 Sheet 1 of 2 Category Description Generic e Added support for Windows Server 2012 R2 Operating System Addedthe ParentBusPath option to each port registry key e Added a new hardware ID for ConnectX 3 Pro NICs The QP numbers allocation is now round robin manner e RecvCompletionMethod as Interrupt is no longer supported Removed the LsoV1IPva4 from the registry UI Removed from the bus driver configuration the Non DMA option Removed the TXRingNum option from the UI NVGRE Added NVGRE hardware off load support for ConnectX 3 Pro cards only e Added to the UI the EncapsulatedPacketTaskOffload option when using ConnectX 3 Pro NICs Performance Added the nd send bwandnd send lat ND benchmarking tools e Fixed nd_ bw to achieve better performance memory buffer alignment and consistent results Mellanox Technologies 15 J Rev 4 80 5000d Changes and Major New Features Table 9 New Features Changes and Fixes in v4 55 Sheet 2 of 2 Category Description E
30. g WinOF may result in installation failure Delete Mellanox components from HKEY LOCAL MACHINENSOFTWAREMMicro soft Windows CurrentVersion DIFx App Components The Mellanox components are mlx4eth63 ipoib6x andmlx4 bus Canceling the installation process may leave the bus driver in a disable state The driver appears in a yel low bang containing the following error message Windows cannot start this hardware device because its configuration infor mation in the registry is incomplete or damaged Code 19 Scan for new hardware and reboot the machine Downgrade is not supported Uninstall the current version and install the older one 34 Mellanox Technologies Table 22 Installation Upgrade Issues Issue Workaround Occasionally upon driver upgrade due to mismatch of driver versions the following error message is dis played There is an interface mismatch between ETH driver and the bus driver The ETH driver interface version is 50 while the bus driver interface version is 17 Asa result the ETH driver has failed to start This happened due to setup failures or partial update of the drivers In order to resolve the issue please reboot the computer The message can be safely ignored Configuration is not restored when replacing a Clean the old network adapter configuration prior to ConnectX 2 ConnectX 3 NIC with a ConnectX 3 Pro upgrade NIC located on the s
31. gistry in the IPoIB network inter face RecvIntModTime 0 Please note this action increases the CPU utilization No communication between the physical interface and a VM that uses vSwitch created over virtual IPoIB Pkey and vice versa Calling ib_query_cq on a CQ which was created with entry indicates that there are 0 entries in the CQ An SR IOV Virtual Machine is loaded in InfiniBand mode when no OpenSM is available in the subnet The following event appears in the event log detected a null port GUID for port lt NUMBER gt A Virtual Function device may have a null port GUID if there is no OpenSM instance on its network Please make sure the network has an active OpenSM and restart the driver Start the OpenSM and restart the driver For InfiniBand SRIOV guest OpenSM Assigned GUIDs are not supported and may cause unexpected behavior Work only with Administrator assigned GUIDs For InfiniBand SRIOV guest ND traffic does not work on an SRIOV adapter when a Para Virtualiza tion adapter configured on the same virtual machine and the same subnet as IPoIB Disable the Para Virtualization adapter Ethernet Issues Table 16 Ethernet Issues Issue Workaround Disabling the Priority amp VLAN tag in the UI which VLANID is configured may result in sending packets with the configured VLANID Remove the VLANID before disabling the Priority amp VLAN tag
32. ice Manager with a yel low bang e Fixed an issues enabling the package s execution in modify mode resulting in driver being disabled Generic Added support for a new report for bad cables Fixed random parsing failures of string registry entries Fixed compilation failure of Hello world in the SDK Fixed the return value of ib query ca if failed to allocate resources for opera tion Mellanox Technologies 13 J Rev 4 80 5009 Changes and Major New Features Table 8 New Features Changes and Fixes in v4 60 17718 Category Description Performance e Added support to IPv6 to all nd tests Fixed CPU utilization report in nd tests Fixed correct bandwidth peak results in ibv send bw with UD QP Fixed sync problems of bidirectional mode in ibv read bw ibv write bw e Fixed an issue reporting incorrect adapter type in performance tuning log file RoCE Fixed RoCE mode parsing ND Added the ability to rearm a CQ in the kernel Added the ability to handle LID changes Changed connection timeout behavior Added the STATUS CONNECTION REFUSED return value upon connection timeout Fixed missing completions when working with Completion Queue with single entry IPoIB Added the ability to handle LID changes e Added support for iSCSI boot over IPoIB Fixed unexpected behavior upon QP asynchronous event Fixed bad completions of VMQ and NonVMQ modes in IPoIB Fixed a
33. l number of any resource that can be created may be limited by the machine configuration the amount of host memory user permissions and the amount of resources already in use by other users processes Running Ntttcp without the a x flag X gt 1 ina NIC configured with 1OGbE may cause low band width in TCP single stream Run Ntttcp with a 8 for best performance Active links disappear after changing the cable con nectivity from Ethernet to InfiniBand or vice versa Disable and enable the mlx4 bus interface from the Device Manager Mellanox Technologies 25 J Table 14 Generic Issues Rev 4 80 5000d Known Issues Issue Workaround On ConnectX 2 ConnectX 3 Ethernet adapter cards there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric driver utilities that read the GUID via device firmware e g using ibstat MIxburn flint return Oxffffas GUID while the utilities return a value derived from the MAC address For all driver firm ware software purposes the latter value should be used Please use the GUID value returned by the fabric driver utilities not Oxfffff Changing the default ND providers may cause ran dom errors such as 0xC0000238 NT_STATUS ADDRESS ALREADY ASSOCIATED on Connect or with 0xC0000236 NT_STATUS CONNECTION REFUSED on Accept These errors can be safely ignored The UI displays th
34. n VMQ mode for the VMs Mellanox Technologies 33 J Table 21 Hyper V Issues Rev 4 80 5000d Known Issues Issue Workaround After disabling and enabling a port on a guest a ping to it may be renewed after a minute The ARP requests sent by Windows are less frequent as the time passes If the guest port was down for a while it could take time until Windows decides to send another ARP request to it When VMQ is enabled after reset the driver loads all the VMQs that existed before the reset However it is not guaranteed that each VMQ will receive the same QP number it had before the reset This can cause some delay as a result of resetting before connectivity is reestablished The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP The IPoIB non VMQ mode is supported only when the VMQ is enabled according to the registry values To use the non VMQ mode for a VM change its set tings as follow Press Settings on the VM Go to Network Adapter gt Hardware Acceleration e Un check the Enable virtual machine queue Installation Upgrade Issues Table 22 Installation Upgrade Issues Issue Workaround Upgrading the driver while the UI is opened with the Connect NIC device may cause the installation process to never end Close the UI before driver upgrade Rebooting the machine reboot while uninstallin
35. nfigured will eliminate this pop up completely Running a downgrade in silent mode is not supported Upon downgrade the return code will always be 0 Uninstalling the driver after upgrade won t remove the directory ProgramFiles Mellanox Uninstalling the driver when multiple of VLANs are Remove the VLANs before uninstallation configured never ends The installation process does not close any applica It is recommended to close all running applications tions running in the background and may cause a prior to upgrading the driver BSOD as a result of a stuck cmd Installation upgrade fails due to PNP failure to copy Reboot the machine and reinstall the driver files to the driver store and the following text is printed in the event logs Fault bucket type 0 Event Name PnPDriverImportError Response Not available Attached files c Users lt user gt App Data Local Temp DMI151A tmp log xml C Program Files Mella nox MLNX_VPI ETH ml1x4eth63 inf Installation upgrade fails due to failure to stop the Kill the WMIPrvSE exe tasks in the task manager WMI service and the following text is printed inthe and installation log reinstall CustomAction StopWMIService returned actual error code 1603 36 Mellanox Technologies Table 22 Installation Upgrade Issues Issue Workaround Following the upgrade of Mellanox driver to WinOF Run the following script on the server to
36. ng over it The keywords added to the registry in NDIS support Windows 2012 are e RssMaxProcNumber e NumRSSQueues RSSProfile The keywords added to the registry in NDIS support Windows 2008 R2 are RssBaseProcNumber e MaxRssProcessors e RssBaseProcGroup The rest of the keywords are added in all versions of NDIS This change is based on http msdn microsoft com en us library windows hardware ff5 70864 v vs 85 aspx Ethernet e RoCE MTU value is no longer set to 1024 by default All options stay as they are and can only be chosen if they were selected explic itly in the Ul registry The current default state is as follows The value is now derived from the MTU or MaxFramSize or Jumbo Packets value and they are all aliases for the same value The value is aligned to 256 512 1024 2048 in a way that it will be rounded down to the nearest power of two of the ETH MTU InfiniBand e Added ibdiagnet utility support Mellanox Technologies 19 J Rev 4 80 5009 Changes and Major New Features 2 8 New Features Changes and Fixes in v4 2 From v3 2 0 Table 12 New Features Changes and Fixes in v4 2 Sheet 1 of 2 Category Description Generic Modified RSS cores and changed VMQ affinity on the fly Fixed restart issue when there are not enough MSI X vectors for each machine core Added support for K GROUPS processors more than 64 processors support to allow assignment of MSI X affinity for
37. niBand or Ethernet is not supported e ibdump is not supported QoS is not supported 24 Mellanox Technologies 6 Known Issues 6 1 Generic Issues Table 14 Generic Issues Issue Workaround Pinning all the physical memory used by RDMA operations such as register memory pin user mem ory on the machine on Operating Systems prior to Windows Server 2012 may cause the machine to hang Avoid pining the whole machine memory in those Operating Systems When the tracer tool traces many events it may con sume a large amount of memory up to several GB RAM Reduce the verbosity level The displayed MAC address in the DHCP server of virtual IPoIB I F may display a wrong data FFOO although the I F is still fully functional When running applications that use ND or libibumad such as OpenSM the system might get to an unsta ble state when trying to shutdown restart hibernate it Close all applications that use ND or libibumad before performing shutdown restart hibernate Activating NC SI in WinOF v0 52 may cause driver s loading failure when using an older firmware version than 2 30 8000 Do not enable NC SI in machines that WinOF v0 52 is installed in The maximum values returned by the ib query ca function for example max qp max mr are the upper limits of the supported resources by the device However it may be impossi ble to use these maximum values since the actua
38. or Ethernet SR IOV over Windows Hyper V Hypervisor over Windows 2012 R2 Added Virtual Ethernet Adapter support which enables using SMB Direct and HyperV VMQ and NVGRE over ConnectX 3 Pro on the same port Added lossless TCP buffer management when no receive WQE are available RoCE Added ConnectX 3 Pro support for RoCEv2 Changed the transport name in vstat and ibstat to be RoCE v2 0 Fixed ibstat behavior on devices with RoCE enabled Fixed releasing of RDMA resources and reacquire them on power down and up Fixed RDMA Activity counters which didn t increase for ND traffic ND Fixed hard coded limitation of 4 SGEs InfiniBand Fixed vstat printing of counters for Ethernet ports Performance Improved perf tuning setting in single CPU machines to avoid TX collision a Requires firmware v2 30 8000 and above b Requires firmware v2 31 5050 and above Mellanox Technologies 11 J Rev 4 80 5009 Changes and Major New Features 2 3 New Features Changes and Fixes in v4 61 From v4 60 WinOF VPI version 4 61 was released as an intermediate release P Table 6 New Features Changes and Fixes in v4 61 Category Description Installation Upgrade Fixed an issue preventing JumboPackets registry key to be restored correctly Ensured that uninstallation of Mellanox package in Virtual Machine leaves the system clean Generic Improved information in event log when a bad cable is
39. oving the USB Performance Set 512 RX buffers by default Removed TXRingNum Changedthe perf tuning setting to achieve a better performance tuning e Addedthe nd write bw nd write lat andnd read bw nd read lat tools Fixed the perf tuning indication of the last chosen tuning scenarios e Fixeda crash in the ib send 1at bw utilities caused when the port link was down Fixed the Restore to defaults option in the perf tuning tool Now the default values are being restored 16 Mellanox Technologies Table 10 New Features Changes and Fixes in v4 40 Sheet 2 of 3 Category Description Ethernet e Added Transmit Side Scaling TSS Added Ethernet QoS proprietary counters diagnostics and traffic for monitor ing using Windows perfmon utility e Added to the MTU size the IP header size 1500 gt 1514 9600 gt 9614 Thus the minimum Jumbo frame size is 614 Interrupt moderation supports the following profiles Low Latency Moderate Aggressive In addition to old values that are not supported anymore Made mlx4 bus and Ethernet devices removable Network Direct Added support for NDv2 Network Direct Set the default ND provide value to mlx4nd2 Fixed WoL support on NIC with a single port Fixed the default RoCE configuration on NICs with a single ports Fixed the values for the MTU and rate of the CM REQ Fixed miniport reset on sending scenarios Removed the QoS attributes when
40. s HCA Recommended Firmware Rev Additional Firmware Rev Supported ConnectX 3 Pro Rev 2 32 5100 Rev 2 31 5050 ConnectX 3 Pro EN ConnectX 3 Rev 2 32 5100 Rev 2 31 5050 ConnectX 3 EN ConnectX 2 Rev 2 9 1000 Rev 2 9 1000 ConnectX 2 EN 8 Mellanox Technologies 2 Changes and Major New Features 2 4 New Features Changes and Fixes in v4 80 50000 from v4 70 This package version is 4 80 50000 The package contains the following versions of components Bus eth IPoIB and mux drivers version is 4 80 10388 The CIM provider version is 4 80 10388 Table 2 New Features Changes and Fixes in v4 80 50000 Category Description Installation Upgrade Added check for administrator privileges during installation Added support for installation in silent mode without execution of perf tune Fixed installation stuck when Remote Desktop Session Host Windows Installer RDS compatibility is enabled Generic Changed Reset Flow SR IOV enabled only if no user space application is running and depends on the registry key AllowResetOnError setting Changed the number of supported QPs in a multicast group from hard coded value to firmware capabilities dependent Fixed driver load failure in machines with 1 TB memory and above IPoIB Added multiple P Key support beta level Added IPoIB SR IOV over KVM and ESX Hypervisors for both full and partial mem bership Ad
41. s yet to be found In WinOF 4 80 when using the performance tuning tool on a single port device the tool crashes Manually configure the desired parameters according to the Performance Tuning Guide IPoIB Issues Table 20 IPoIB Issues Issue Workaround In VMQ and IPoIB when performing VM Migration or modifying VMQ configuration during traffic with a large number of active VMs the system may experi ence temporary network connectivity issues or failure of the VM migration When using non default Pkey for IPoIB in an SR IOV guest OS using the same partition with different membership might cause connectivity problems Work only with full membership PKeys as the non default Pkey vIPoIB ports which are created via the part man tool over the same partition with different member ship might cause connectivity problems Use part man with the full membership value the partitioning will be enforced by SM configuration IPoIB interfaces teaming is currently not supported 6 8 Hyper V Issues Table 21 Hyper V Issues Issue Workaround When the vSwitch is detached from the ETH IPoIB device while the driver is disabled the device does not reacquire the static IP it had before the attachment of the vSwitch When the vSwitch is attached to the ETH IPoIB device while there is no link it will not receive the device IP when the link is back up After attaching
42. t 3 of 3 Category Description Made mlx4 bus and Ethernet devices removable Network Direct Added support for NDv2 e Network Direct Set the default ND provide value to mlx4nd2 Fixed WoL support on NIC with a single port Fixed the default RoCE configuration on NICs with a single ports Fixed the values for the MTU and rate of the CM REQ e Fixed miniport reset on sending scenarios Removed the QoS attributes when disabling QoS Enabled MaxRssProcessirs support of the following values 1 2 4 8 16 32 64 Network Direct Fixed a crash occurred when more than 4 SGEs elements were used in an ND write operation Network Direct Fixed the swap of InboundReadLimit and OutboundRead Limit when creating an EndPoint and in Connector GetConnectionData Network Direct Fixed disallowing creation of EndPoint with zero attributes in the Receive Queue Network Direct Removed the option of NDK registration failure requiring a reboot of the machine to register it again Network Direct Fixed a failure when creating an EndPoint with zero attributes in the Receive Queue e Network Direct Added the option of sensing the incoming Read messages according to the device capabilities when creating an EndPoint limit e Network Direct Fixed a failure of ND connectivity between VMs on the same host InfiniBand e Onrare occasions depends on the GUID assignment the IPoIB MAC address can be assigned with a multicas
43. t MAC the least significant bit of the most sig nificant address octet is set to 1 In that case all of the traffic over the IPoIB I F is dropped If you experience this issue please contact Mellanox support e Addedactive mtu fieldto struct ib port attr t Added the option of vstat displaying the active mtu ofthe ports Allowed registration of a large Memory Region which is splitted to many seg ments Fixed a bluescreen issue that occurred when disabling the interface after a TX stress over the VMQ Fixed a failure of MPI ND over InfiniBand e Added the option of ibv devinfo displaying the correct MTU value after it was changed Added the option of part man printing the adapter name when the Port GUID is set to zero Added the option of part man printing the leading zeroes of port GUID Installation Upgrade Prevented displaying a message to upgrade the firmware for OEM NICs if it has the latest firmware version Removed portsetting registry key during uninstallation 18 Mellanox Technologies 2 7 New Features Changes and Fixes in v4 30 From v4 20 WinOF VPI version 4 3 was released as an intermediate release hus Table 11 New Features Changes and Fixes in v4 3 Category Description Generic e Added support for a new provider called MLX4ND which supports both NDvI1 and NDv2 interfaces Performance Enabled performance tuning running according to the operating systems that are runni
44. talled ND by default The following tools were added to the setup package e ibv_asyncwatch exe e jbv atomic bw exe e jbv atomic lat exe e ibv_devinfo exe e jbv rc pingpong exe e ibv_rdma_bw exe iby rdma lat exe ijbaddrexe e ijbcacheedit exe e iblinkinfo exe e jbqueryerrors exe e jbsysstat exe saquery exe smpdump exe Mellanox Technologies 21 J Rev 4 80 5000d Features Support to be Deprecated in the Next Release 3 Features Support to be Deprecated in the Next Release The following support will be deprecated in the next release e ConnectX 2 adapter cards 22 Mellanox Technologies 4 Beta Features Table 13 Beta Features Category Description WinVerbs WinVerbs is currently at beta level ibdump ibdump is currently at beta level IPoIB IPv6 support of IPoIB in an SR IOV guest OS over KVM is at beta level Multiple PKeys is at beta level Mellanox Technologies 23 J Rev 4 80 5000d Unsupported Functionality Features 5 Unsupported Functionality Features The following are the unsupported functionality features in WinOF Rev 4 80 50000 ND over WinVerbs provider e SRP 5 1 ConnectX 2 Adapter Limitations Limited NDK support for details contact Mellanox Support CQ to EQ feature is not supported consequently Multi core performance may be non optimal Mellanox perfmon counters are not supported Auto Sensing of link protocol Infi
45. the setup package Performance Optimized code performance Increased send parallelism Memory used in receive flow is now allocated with the same affinity of the handling processor for faster access Statistics parameters are now directly read from hardware instead of being cal culated by software e Added support for BlueFlame BlueFlame is now the default working mode for all packets that have a descriptor which fits into a BF register currently 256 bytes Use BlueFlame registry key to enable Wisable this feature Added support for RSS functionality on available processors numbers Used to be restricted to start at the first processor Changed RSS registry defaults to give better out of the box performance e Added a performance UI to tune performance under various scenarios e Added a tool to tune performance under various scenarios 20 Mellanox Technologies Table 12 New Features Changes and Fixes in v4 2 Sheet 2 of 2 Category Description Ethernet e Added support for multiple TX rings e Added an option to verify that the number of multicast groups used is no higher than the firmware limits Improved performance in virtualization when using VMQ IPoIB Fixed a bug that prevented IBAL applications from working on machines with 2 cards Fixed a bug that caused packet drop on remote node when the first sent packet is not an ARP or a multicast packet e Added support for multiple TX rings
46. thernet Fixed the issue preventing messages to be sent in VLAN 0 when using many VMQ rings Added IP IP checksum off load support Added Ports TX arbitration Bandwidth allocation per port The following ND providers MLX4ND and MLXAND2 are installed by default Fixed setting the correct SL in UD traffic over RoCE InfiniBand IPoIB performance improvements e Fixed a part man issue related to wrong statistics over virtual partman inter faces RoCE Enabled roce mode value overwrite in case it exists during installation Fixed in ibv devinfo the display of correct transport RoCE mode Added Sniffer for RoCE packets The used RoCE mode set upon driver load is printed into event log message 2 6 New Features Changes and Fixes in v4 40 From v4 30 Table 10 New Features Changes and Fixes in v4 40 Sheet 1 of 3 Category Description Generic e Added a notification in the event log in case SMB is not supported in Con nectX 2 firmware e Added the trace tool for WPP tracing Added copyright to the SDK files Added WMI Powershell support Fixed an issue causing the setup to fail upon perf tuning failure during the installation An error message will be printed in the installation log upon perf tuning failure Removed port setting registry key during uninstall Fixed and issue with the Mellanox adapter being shown on the USB removal menu which caused the removal of the Mellanox adapter once rem
47. ved perf tuning detection for the first port Improved performance in packet forwarding scenarios Decreased dropped packets rate for Ethernet significantly Changed default perf tuning scenario to be Balanced configuration Various performance improvements WMI CIM Added ability to read active RoCE configuration from hardware Added support for RoCE IP Based 2 2 New Features Changes and Fixes in Version 4 70 2 2 1 New Features Changes and Fixes in v4 70 50050 from v4 70 50040 This package version is 4 70 50050 The package contains the following versions of components Bus eth IPoIB and mux drivers version is 4 70 10143 a The CIM provider version is 4 70 10143 Table 3 New Features Changes and Fixes in v4 70 50040 Category Description IPoIB Fixed SM fail over causing the driver to hang 2 2 New Features Changes and Fixes in v4 70 50040 from v4 70 50000 This package version is 4 70 50040 The package contains the following versions of components y Bus eth IPoIB and mux drivers version is 4 70 10141 The CIM provider version is 4 70 10141 Table 4 New Features Changes and Fixes in v4 70 50040 Category Description Generic Optimized handling of affinity change on OID RECEIVE FILTER QUEUE PARAMETERS Added the ability to control the number of retries and timeout to check the device health before performing reset Ethernet Fixed missing pause
48. which contains two fields p next and qp context 7 2 API Changes in WinOF Rev 4 70 The following are the API changes in WinOF Rev 4 70 Table 27 API Changes in WinOF Rev 4 70 Name Description ib get port spl qp Added a new function ib get mad inner Changed API one more input parameter was added ib get mad Changed API one more input parameter was added VERBS MINOR VER Increased its value 000a gt 000c UNBOUND PORT NUM Added a new macro 7 3 API Changes in WinOF Rev 4 60 The following are the API changes in WinOF Rev 4 60 Table 28 API Changes in WinOF Rev 4 60 Name Description IB MOD QP CHANGE COUNTER INDEX Added a new macro struct ib qp mod t Added the field state rtr counter index Mellanox Technologies 39 J Rev 4 80 5000d API Changes 7 4 API Changes in WinOF Rev 4 55 The following are the API changes in WinOF Rev 4 55 Table 29 API Changes in WinOF Rev 4 55 Name Description RDMA TRANSPORT RRDMAOE 1 5 Added enumerated values RDMA TRANSPORT RRDMAOE 2 0 Added enumerated values RDMA TRANSPORT RRDMAOE It is an alias to RDMA TRANSPORT RRDMAOE 1 5 is rroce is xroce Added new functions IB AC SNIFFER Added a new macro struct ib qp mod t Added the field state init flags 7 5 API Changes in WinOF Rev 4 40 The following are the API changes in WinOF Rev 4 40 Table 30
49. yper V non SR IOV Hypervisor OS Windows Server 2008 R2 64 bit only Virtual machine OS Windows Server 2008 R2 64 bit only Hypervisor OS Windows Server 2012 64 bit only Virtual machine OS Windows Server 2008 R2 64 bit only and Windows Server 2012 64 bit only Hypervisor OS Windows Server 2012 R2 64 bit only e Virtual machine OS Windows Server 2008 R2 64 bit only Windows Server 2012 64 bit only and Windows Server 2012 R2 64 bit only SR IOV Ethernet Hypervisor OS Windows Server 2012 R2 64 bit only Virtual machine OS Windows Server 2012 64 bit only and Windows Server 2012 R2 64 bit only SR IOV InfiniBand Hypervisor OS KVM with MLNX_OFED 2 3 or later e Virtual machine OS Windows Server 2008 R2 64 bit only Windows Server 2012 64 bit only and Windows Server 2012 R2 64 bit only Mellanox Technologies 7 J Rev 4 80 5000d Overview 1 4 Supported Network Adapter Cards Mellanox WinOF Rev 4 80 50000 supports the following Mellanox network adapter cards ConnectX 3 Pro and ConnectX 3 Pro EN e 10 40 and 56 Gb s InfiniBand IB e 10 40 and 56 Gb s Ethernet ConnectX 3 and ConnectX 3 EN 10 40 and 56 Gb s InfiniBand IB e 10 40 and 56 Gb s Ethernet ConnectX 2 and ConnectX 2 EN e 10 and 40 Gb s InfiniBand IB 10 Gb s Ethernet 1 4 1 Firmware Versions Mellanox WinOF Rev 4 80 50000 provides the following firmware for Mellanox NICs Table 1 Firmware Version
Download Pdf Manuals
Related Search