Mellanox WinOF VPI Release Notes
Contents
1. 14 Mellanox Technologies Rev 4 60 Table 5 New Features Changes and Fixes in v4 2 Sheet 3 of 3 Category Description IPoIB Fixed a bug that prevented IBAL applications from working on machines with 2 cards Fixeda bug that caused packet drop on remote node when the first sent packet is not an ARP or a multicast packet e Added support for multiple TX rings The new driver advertises its TSS capa bility using multiple TX rings This feature will not be used when communi cating with older version drivers such as the inbox driver This way IPv4 communication in a mixed environment that includes the inbox driver should work IPv6 communication may be problematic in such an environment Eliminated some race conditions in IPoIB mcast handling e Added mcast addresses according to the Windows mcast table in addition to mcast addresses from igmp mld queries in order to connect to such mcasts that are not reported by the queries Removed redundant LID CHANGE events This fixes the redundant recon nection of all IPoIB mcasts Added support for VM migration When a VM is migrated from one host to another it does not send a new ARP to hosts it already knew when it was sitting on the first host In order to prevent communication interruption to such hosts the driver will initiate an ARP to learn them again Fixed IPoIB VMQ affinity update bugs Fixed IPoIB VMQ parent queue management race during reset Fix2. Fixed CPU utilization report in nd tests Fixed correct bandwidth peak results in ibv send bw with UD QP Fixed sync problems of bidirectional mode in ibv read bw ibv write bw e Fixed an issue reporting incorrect adapter type in performance tuning log file RoCE Fixed RoCE mode parsing ND Added the ability to rearm a CQ in the kernel e Added the ability to handle LID changes Changed connection timeout behavior Added the STATUS CONNECTION REFUSED return value upon connection timeout Fixed missing completions when working with Completion Queue with single entry Mellanox Technologies 7 J Rev 4 60 Changes and Major New Features Table 1 New Features Changes and Fixes in v4 60 Category Description IPoIB e Added the ability to handle LID changes e Added support for iSCSI boot over IPoIB Fixed unexpected behavior upon QP asynchronous event Fixed bad completions of VMQ and NonVMQ modes in IPoIB Fixed a failure occurred when setting the IPoIB adapter value to SA Query Timeout Fixed propagation of the physical link disconnection to virtual part man interface Performance improvements in latency Fixed using CQ after VMQ is closed Fixed bad completion of VMQ QP that was caused by malformed WR Ethernet Modified the CQ size to prevent CQ overrun Added DSCP support over IPv4 Added traffic profile Changed the report link speed zero in case of disconnecte
3. Hyper V is at low bandwidth on LBFO vSwitch Win dows 2012 In Ethernet to achieve better iperf TCP performance between a Linux VM and a Windows VM on different hosts when using MS MUX over the Ethernet driver use the non VMQ mode for the VMs Mellanox Technologies 25 J Rev 4 60 Known Issues Table 7 WinOF Known Issues Sheet 9 of 11 Category Installation Upgrade 26 Mellanox Technologies Description After disabling and enabling a port on a guest a ping to it may be renewed after a minute The ARP requests sent by Windows are less frequent as the time passes If the guest port was down for a while it could take time until Windows decides to send another ARP request to it Workaround When VMQ is enabled after reset the driver loads all the VMQs that existed before the reset However it is not guaranteed that each VMQ will receive the same QP number it had before the reset This can cause some delay as a result of resetting before connectivity is reestablished The delay is caused by the time it takes for the ARP table to update after initiating the Gratuitous ARP The IPoIB non VMQ mode is supported only when the VMQ is enabled according to the registry values Upgrading the driver while the UI is opened with the ConnectX NIC device may cause the installation pro cess to never end To use the non VMQ mode for a VM change its settings as follow e P
4. TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfimScale MLNX OS PhyX SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IBTM ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroX MetroDX ScalableHPCTM Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies J Rev 4 60 Table of Contents Table of Contents isss tei sides E EAR an ERE EAEK ESIOAEETERAS E EX RAM Fn 3 Eistot Lables coa ERA EROR GER EC HOE EN GI OER ADEE SEW GC AC EA e ER ERE 4 Chapter P OyervieW sexe exa xU RON ERR eR eee ea Re arabe we 5 1 1 Mellanox WinOF Rev 4 60 New Features 00 00 cece ee ee eee 5 1 2 WinOF VPI Package Contents 0 0 0 00 es 5 1 3 Supported Operating Systems Versions sssssreseereeerererr rr eee eee 6 1 4 Supported Network Adapter Cards 0 rer rer erna 6 1 4 1 Package Supplie
5. it is recommended to update the firmware manually Enabled configuration of TxRingNum registry key from the UI Improved the Port Protocol dialog Added Registry key documentation to the setup package Mellanox Technologies 13 J Rev 4 60 Changes and Major New Features Table 5 New Features Changes and Fixes in v4 2 Sheet 2 of 3 Category Description Performance Optimized code performance Increased send parallelism Memory used in receive flow is now allocated with the same affinity of the handling processor for faster access Statistics parameters are now directly read from hardware instead of being cal culated by software e Added support for BlueFlame BlueFlame is now the default working mode for all packets that have a descriptor which fits into a BF register currently 256 bytes Use BlueFlame registry key to enable disable this feature e Added support for RSS functionality on available processors numbers Used to be restricted to start at the first processor Changed RSS registry defaults to give better out of the box performance Added a performance UI to tune performance under various scenarios e Added a tool to tune performance under various scenarios Ethernet e Added support for multiple TX rings Added an option to verify that the number of multicast groups used is no higher than the firmware limits Improved performance in virtualization when using VMQ
6. reacquire the static IP it had before the attachment of the vSwitch When the vSwitch is attached to the ETH IPoIB device while there is no link it will not receive the device IP when the link is back up After attaching the vSwitch to the ETH IPoIB device Reattach the vSwitch to sync changing the Jumbo Packet registry key on the with the value set in the ETH IPoIB device does not affect the vSwitch configu ETH IPoIB device ration and vice versa For example if the user sets the Jumbo Packet on the ETH IPoIB device to X and the Jumbo Packet on the vSwitch to X Y X Y sized packets will be passed from NDIS down to the driver and they will be dropped by it Unexpected behavior might occur when running in a virtualized environment and creating two virtual switches bound to each of the ports of a dual port NIC and then using both of them with two vNICs from the same VM When moving an IPoIB interface in a VM from non VMQ to VMQ or from VMQ to non VMQ a reset to the NIC may occur and in the event log the fol lowing message will appear device reports a CQE error on cqn num ber qpn number Status number There fore the HCA Nic will be reset The issue is reported in Function lt function gt For more information refer to details In IPoIB when using long Multicast traffic from a Vir tual Machine VM to an external host there might be up to 0 596 loss in 596 bursts
7. The installation process does not close any applications running in the background and may cause a BSOD as a result of a stuck cmd Workaround It is recommended to close all running applications prior to upgrading the driver InfiniBand application that was compiled with an SDK version earlier than WinOF v4 40 is not binary compati ble Recompile InfiniBand appli cation with WinOF v4 40 and above ND application is backward compatible and older applications over ND do not have to be recompiled WMI does not work due to lack of permissions Change the execution policy Run Set ExecutionPolicy AllSigned WinOF 4 40 and above IPoIB driver is not IPv6 compat ible with earlier driver versions including Window 8 Inbox driver If WinOF 4 50 IPoIB node receives an icmpv6 message from the legacy IPoIB node the following event will appear in the event log TPoIB driver detected a possible incom patibility with Windows 8 inbox IPv6 sup port due to which there is no IPv6 connectivity Functionality such as CQ to EQ Auto Sensing QoS are not supported in ConnectX 2 adapter cards To enable compatibility mode add Win8InboxCompatibilityMo de REG SZ registry key with the value of 1 in the IPoIB interface registry Note All IPoIB nodes must use the same mode to enable IPv6 compatibility with ear lier driver versions We recommend upgrading all hosts to the new driver ver sion from http
8. egress packets The default traffic classification has the following attributes It has a traffic classification condition of type NDIS QOS CONDITION DEFAULT Itis the first traffic classification defined in the array of NDIS QOS CLASSIFICATION ELEMENT structures RDMA Activity counters do not count during Network Direct RoCE traffic NVGRE When NVGRE off load is enabled the GRE traffic can not be accepted as a regular L2 traffic and requires spe cial L2 TUNNELING steering rules In such case the GRE packets are dropped or directed to promiscuous queue GRE traffic steering by inner MAC and by outer MAC Configure steering or by simultaneously is currently not supported inner MAC or by outer MAC Mellanox Technologies 23 J Rev 4 60 Known Issues Table 7 WinOF Known Issues Sheet 7 of 11 Category Description Workaround If VMQ set filter requests are accepted without a GRE Set the bus driver registry flag i e requested steering by outer MAC the GRE keyAcceptGREbyOuterMAC packets do not reach that VMQ _P1 2 per port to accept GRE traffic by outer MAC and to duplicate L2 steering tule to L2 TUNNELING rule for each VMQ set filter request without GRE flag Note For regular NVGRE Hyper V scenarios the value of the registry key below must be set to 0 AcceptGREbyOuterMAC P1 2 Quality of Running Quality of Service QoS commands without Store the QoS policy in
9. ETH driver and the bus driver The ETH driver interface version is 50 while the bus driver interface version is 17 As a result the ETH driver has failed to start This happened due to setup failures or partial update of the drivers In order to resolve the issue please reboot the com puter The message can be safely ignored Configuration is not restored when replacing a Clean the old network adapter ConnectX 2 ConnectX 3 NIC with a ConnectX 3 Pro configuration prior to NIC located on the same PCI slot upgrade Configuration can be restored only in Windows Server 2012 and above IPv6 configuration restore is not supported Upon upgrade the following Registry Key values will be overwritten with the following e ReceiveBuffers 512 e MaxRssProcessors 8 e RssBaseProcNumber 0 NumRSSQueues 8 e RssMaxProcNumber 63 RssProfile 1 e DefaultRecvRingProcessor 1 TxInterruptProcessor 1 e TxForwardingProcessor 1 RxIntModerationProfile 1 TxIntModerationProfile 1 e RecvCompletionMethod 1 SingleStream 0 TxRingNum 8 Mellanox Technologies 27 Rev 4 60 Known Issues Table 7 WinOF Known Issues Sheet 11 of 11 Category Utilities Description Upon upgrade the following Ethernet Registry Keys will be deleted e SendCompletionMethod e UseRSSForRawIP e UseRSSForUDP Workaround Upon upgrad
10. Mellanox Technologies 11 Rev 4 60 Changes and Major New Features Table 3 New Features Changes and Fixes in v4 40 Sheet 3 of 3 Category Description InfiniBand e Onrare occasions depends on the GUID assignment the IPoIB MAC address can be assigned with a multicast MAC the least significant bit of the most sig nificant address octet is set to 1 In that case all of the traffic over the IPoIB I F is dropped If you experience this issue please contact Mellanox support e Added active mtu fieldto struct ib port attr t Added the option of vstat displaying the active mtu of the ports Allowed registration of a large Memory Region which is splitted to many seg ments Fixed a bluescreen issue that occurred when disabling the interface after a TX stress over the VMQ e Fixed a failure of MPI ND over InfiniBand Added the option of ibv_devinfo displaying the correct MTU value after it was changed Added the option of part man printing the adapter name when the Port GUID is set to zero Added the option of part man printing the leading zeroes of port GUID Installation Upgrade Prevented displaying a message to upgrade the firmware for OEM NICs if it has the latest firmware version Removed portsetting registry key during uninstall 2 4 New Features Changes and Fixes in Version 4 3 From Version 4 2 3 p WinOF VPI version 4 3 was released as an intermediate release ra Table 4 New Featur
11. and 56 Gb s InfiniBand IB 40 Gb s Ethernet Firmware v2 30 8000 10 Gb s InfiniBand IB 10 Gb s Ethernet ConnectX 3 and ConnectX 3 EN firmware v2 30 8000 ConnectX 2 and ConnectX 2 EN firmware v2 9 1200 1 The previous package of WinOF 4 60 included ConnectX 3 Pro firmware v2 30 8000 6 Mellanox Technologies Rev 4 60 2 Changes and Major New Features 2 1 New Features Changes and Fixes in Version 4 60 From Version 4 55 This package version is 4 60 17738 The package contains mixed versions of drivers Bus and eth version 4 60 17718 The mux driver version is 4 60 17729 The IPoIB driver version is 4 60 17736 Table 1 New Features Changes and Fixes in v4 60 Category Description Installation Upgrade Enabled configuration changes saving upon Inbox and previous releases upgrade Enabled CIM installation as a standalone package e Fixed an issue occurred when uninstalling and reinstalling the driver The Con nectX 3 Pro Ethernet device was displayed in the Device Manager with a yel low bang e Fixed an issues enabling the package s execution in modify mode resulting in driver being disabled Generic Fixed random parsing failures of string registry entries Fixed compilation failure of Hello world in the SDK Fixed the return value of ib query ca if failed to allocate resources for opera tion Performance e Added support to IPv6 to all nd tests
12. code The sample code is intended to allow users to test or bring up the InfiniBand fabric without a management console switch to get started For cluster production environments Mellanox s recommendation is to use a Managed Switch or the UFM SDN Appliance Low level performance tools InfiniBand Diagnostics tools Software Development Kit SDK e Documentation 1 Requires firmware v2 30 8000 and above Mellanox Technologies 5 Rev 4 60 Overview 1 3 Supported Operating Systems Versions Supported Operating Systems and Service Packs Windows Server 2008 R2 64 bit only Windows Server 2012 64 bit only Windows Server 2012 R2 64 bit only 1 4 Supported Network Adapter Cards Mellanox WinOF Rev 4 60 supports the following Mellanox network adapter cards ConnectX 3 Pro and ConnectX 3 Pro EN firmware v2 30 5000 and above 10 40 and 56 Gb s InfiniBand IB e 10 and 40 Gb s Ethernet ConnectX 3 and ConnectX 3 EN firmware v2 11 1250 and above e 10 40 and 56 Gb s InfiniBand IB 10 and 40 Gb s Ethernet ConnectX 2 and ConnectX 2 EN firmware v2 9 1200 and above 10 40 and 56 Gb s InfiniBand IB 10 and 40 Gb s Ethernet ConnectX adapter cards are not supported by this release of WinOF aa 1 4 4 Package Supplied Firmware Mellanox WinOF Rev 4 60 provides the following firmware for Mellanox NICs ConnectX 3 Pro and ConnectX 3 Pro EN Firmware v2 30 8050 40
13. mella nox com or use Windows 8 compatibility mode On rare occasions as a result of port configuration change IB ETH the UI may get stuck for up to a few minutes This effect does not require any user action The UI returns to its proper functionality after a few minutes WinOF Inbox driver does not support upgrade When installing WinOF v4 40 and above on a Windows Server 2012 and above machine the Inbox driver is uninstalled prior to starting the new installation and any previous configurations is lost The Inbox driver will be rein stalled automatically when the new driver is uninstalled Without separate ports for each stream WinSock multi plexes every packet to every subscriber socket and then filters it out Use different UDP ports to get higher performance when using multicast packets 20 Mellanox Technologies Rev 4 60 Table 7 WinOF Known Issues Sheet 4 of 11 Category Description A virtual IPoIB interface created by the part man utility reports an Active state when the physical link is in the Initializing state and OpenSM is not running in the subnet Workaround The Packets Received Discarded and Packets Received Errors counter may display wrong results Firmware upgrade may fail during installation 1f there was a prior firmware upgrade on the machine without a reboot after it A firmware upgrade failure does not fail the whole installation The dr
14. size is 614 Interrupt moderation supports the following profiles Low Latency Moderate Aggressive In addition to old values that are not supported anymore Made mlx4 bus and Ethernet devices removable e Network Direct Added support for NDv2 e Network Direct Set the default ND provide value to mlx4nd2 Fixed WoL support on NIC with a single port Fixed the default RoCE configuration on NICs with a single ports Fixed the values for the MTU and rate of the CM REQ Fixed miniport reset on sending scenarios Removed the QoS attributes when disabling QoS Enabled MaxRssProcessirs support of the following values 1 2 4 8 16 32 64 Network Direct Fixed a crash occurred when more than 4 SGEs elements were used in an ND write operation e Network Direct Fixed the swap of InboundReadLimit and OutboundRead Limit when creating an EndPoint and in Connector GetConnectionData Network Direct Fixed disallowing creation of EndPoint with zero attributes in the Receive Queue Network Direct Removed the option of NDK registration failure requiring a reboot of the machine to register it again Network Direct Fixed a failure when creating an EndPoint with zero attributes in the Receive Queue e Network Direct Added the option of sensing the incoming Read messages according to the device capabilities when creating an EndPoint limit e Network Direct Fixed a failure of ND connectivity between VMs on the same host
15. API Changes in WinOF Rev 4 60 The following are the API changes in WinOF Rev 4 60 Table 8 API Changes in WinOF Rev 4 60 Name Description IB MOD QP CHANGE COUNTER INDEX Added a new macro struct ib qp mod t Added the field state rtr counter index 6 2 API Changes in WinOF Rev 4 55 The following are the API changes in WinOF Rev 4 55 Table 9 API Changes in WinOF Rev 4 55 Name Description RDMA TRANSPORT RRDMAOE 1 5 Added enumerated values RDMA TRANSPORT RRDMAOE 2 0 Added enumerated values RDMA TRANSPORT RRDMAOE It is an alias to RDMA TRANSPORT RRDMAOE 1 5 is rroce is xroce Added new functions IB AC SNIFFER Added a new macro struct ib qp mod t Added the field state init flags 6 3 API Changes in WinOF Rev 4 40 The following are the API changes in WinOF Rev 4 40 Table 10 API Changes in WinOF Rev 4 40 Name Description VERBS MINOR VER Increased its value 0009 gt 000a enum eth link speeds Added enumerated values struct ib port attr t The mtu field was separated into two fields max mtu maximum MTU supported by the port active mtu actual MTU which the port is configured with Addedtheetn link speed field WR SEND INV Added enumerated values struct ib send wr t The type of invalidate rkey was changed from net32 t gt ib net32 t Mellanox Technologies 29 J Rev 4 60 A
16. Mellanox TECHNOLOGIES Mellanox WinOF VPI Release Notes Rev 4 60 Last Modified 26 March 2014 www mellanox com Rev 4 60 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT CPRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF AN Y KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox
17. PI Changes Table 10 API Changes in WinOF Rev 4 40 Name Description IB SEND OPT SKIP DOORBELL Added the send Write flag 30 Mellanox Technologies
18. ation in the event log in case SMB is not supported in Con nectX 2 firmware e Added the trace tool for WPP tracing Added copyright to the SDK files Added WMI Powershell support e Fixed an issue causing the setup to fail upon perf tuning failure during the installation An error message will be printed in the installation log upon perf tuning failure Removed port setting registry key during uninstall Fixed and issue with the Mellanox adapter being shown on the USB removal menu which caused the removal of the Mellanox adapter once removing the USB Performance Set 512 RX buffers by default Removed TXRingNum Changed the perf tuning setting to achieve a better performance tuning Addedthend write bw nd write latandnd read bw nd read lat tools Fixedthe perf tuning indication of the last chosen tuning scenarios Fixedacrashinthe ib send lat bw utilities caused when the port link was down Fixed the Restore to defaults option in the per tuning tool Now the default values are being restored 10 Mellanox Technologies Rev 4 60 Table 3 New Features Changes and Fixes in v4 40 Sheet 2 of 3 Category Description Ethernet Added Transmit Side Scaling TSS e Added Ethernet QoS proprietary counters diagnostics and traffic for monitor ing using Windows perfmon utility Added to the MTU size the IP header size 1500 gt 1514 9600 gt 9614 Thus the minimum Jumbo frame
19. d VMQ packets with the wrong CPUs Mellanox perfmon counters are not supported e Auto sensing is not supported e ibdump is not supported QoSis not supported Mellanox Technologies 17 Rev 4 60 Known Issues 5 Known Issues Table 7 WinOF Known Issues Sheet 1 of 11 Category General 18 Mellanox Technologies Description Pinning all the physical memory used by ROMA opera tions such as register memory pin user memory on the machine on Operating Systems prior to Windows Server 2012 may cause the machine to hang Workaround Avoid pining the whole machine memory in those Operating Systems When the tracer tool traces many events it may con sume a large amount of memory up to several GB ram Reduce the verbose level The displayed MAC address in the DHCP server of vir tual IPoIB I F may display a wrong data FF00 although the I F is still fully functional When running applications that use ND or libibumad such as OpenSM the system might get to a unstable state when trying to shutdown restart hibernate it Close all applications that uses ND or libibumad before performing shutdown restart hibernate Activating NC SI in WinOF v4 60 may cause driver s loading failure when using an older firmware version than 2 30 8000 Do not enable NC SI in machines that WinOF v4 60 Is installed in When trying to use the following WMI classes and oper atio
20. d Firmware 00 cece rr rer rer rr eens 6 Chapter 2 Changes and Major New Features ssesessses osseoooorcrrr 7 2 1 New Features Changes and Fixes in Version 4 60 From Version 4 55 7 2 2 New Features Changes and Fixes in Version 4 55 From Version 4 40 9 2 3 New Features Changes and Fixes in Version 4 40 From Version 4 3 10 2 4 New Features Changes and Fixes in Version 4 3 From Version 4 2 12 2 5 New Features Changes and Fixes in Version 4 2 From Version 3 2 0 13 Cliapter 3 Beta Features ivo e o RC o ORI ee E CRURA CR CR CHOR 16 Chapter 4 Unsupported Functionality Features eee 17 4 ConnectX 2 Adapter Limitations 0 00 00 cece eee eee ee 17 Chapter 5 KuownlssueS oss 6 6546448 6 X3 oo hee SSRN ER ELAR EM REI EE ESE kee 18 Chapter 6 API Changes 8r ved k ow CERES OR ERA a Ree ewe 29 6 1 API Changes in WinOF Rev 4 60 0 cece cette nee 29 6 2 API Changes in WinOF Rev 4 55 0 0 0 cette ene 29 6 3 API Changes in WinOF Rev 4 40 0 0 0 ccc rer rer ere rena 29 Mellanox Technologies 3 Rev 4 60 List of Tables Table 1 New Features Changes and Fixes in V4 60 2 2 0 0 cc cece cece eee teen nee 7 Table 2 New Features Changes and Fixes in v4 55 2 0 ccc eee teen nee 9 Table 3 New Features Changes and Fixes in V4 40 2 0 ccc cee ere ers res 10 Table 4 New Features Changes and Fixes in V4 3 2 0 cece r
21. d network adapter LBFO Fixed the team s MAC address uniqueness in the subnet of the team in Windows 2008 R2 Fixed port channel teaming with CISCO switch and Fabric Extenders traffic loose in Windows Server 2008 R2 e Fixed an issue related to packets sent with corrupted VLAN header when they were meant to be untagged Fixed unexpected behavior upon QP asynchronous event Stability fixes e Performance improvements WMI CIM Added ControlledBy association to IBPort Fixed ConformsToProfiles association for Softwareldentity and Driverldentity Fixed execution of all tests which were running when executing Diagnostic tests on one instance Fixed a failure occurred when running MLNX Card Fixed the printing of diagnostics log e Fixed an issue preventing from get event to show information after disabling the PCI device Removed support for the following configuration e ModeFlags SingleMsixNum MultiMsixNum SingleEqNum MultiEq Num MaxContQuant SlaveNum DebugLevel DebugFlags e UsePrio NumFcExch EnableQoS BlockMcastLoopBack nterruptFromFirstPacket ProbeVf 8 Mellanox Technologies J Rev 4 60 2 2 New Features Changes and Fixes in Version 4 55 From Version 4 40 Table 2 New Features Changes and Fixes in v4 55 Category Description Generic e Added support for Windows Server 2012 R2 Operating System Added the ParentBusPath option to each port registry ke
22. d restart issue when there are not enough MSI X vectors for each machine core e Added support for K GROUPS processors more than 64 processors support to allow assignment of MSI X affinity for multiple processor groups e Set an adequate number of MTTs to map all physical memory Allocated firmware and ICM memory in chunks of non paged memory instead of using contiguous physical memory Fixed RSS indirection table mapping building when there are less RX rings than RSS cores Fixed a bug preventing standard work with BAR value more than 4GB Fixed memory leaks Fixed error flows causing a Bluescreen in driver startup unload Fixed a Bluescreen occurrence upon shutdown due to leak in active resources Changed device names in device manager and their hardware IDs The changes were made to distinguish between ConnectX 2 and ConnectX 3 for ConnectX 2 MLX4 ConnectX 2_Eth and IBA ConnectX 2_IPoIB for ConnectX 3 MLX4 ConnectX 3_Eth and IBA ConnectX 3_IPoIB Set QoS settings only for ConnectX 3 Changing the hardware ID forces the OS to install new device and re build the registry keys Added an event log to indicate driver failure upon start if there are two HCA burned with the same GUID Added firmware upgrade support as part of the setup process The setup burns the new firmware only on Mellanox cards Firmware burning failure does not prevent the driver s installation therefore it will show a warning In this case
23. e the SendCompletionMethod IPoIB Reg istry Key value will be modified as follow SendCompletionMethod 0 Upon upgrade the following IPoIB Registry Keys will be deleted e UseRSSForRawIP e UseRSSForUDP The empty directory ProgramFiles Mella nox MLNX_VPI Endure DLLs is being created during the driver installation Uninstalling the driver on Windows Server 2008 R2 with LBFO configuration results in the appearance of a pop up window requesting to close several running applications Choose Do not close appli cations This action allows the uninstallation of the driver A Reboot may be required Rebooting the server before uninstalling the driver when LBFO is configured will eliminate this pop up com pletely Upgrading the driver while perfmon is open and moni toring the ConnectX NIC device may end up with adapters having yellow bang with code 10 ibdump may encounter packet drops upon a burst of more than 4096 or 2 max burst packets Close the perfmon before driver upgrade Packets loss is not reported by ibdump Running ibdump on a RoCE Ethernet port may decrease the functional bandwidth due to the overhead of creating extra copy for each packet This may lead to packet drops on the link Sniffing over IB ports is currently not supported Verify Ethernet flow control is enable to ensure a lossless link 28 Mellanox Technologies Rev 4 60 6 API Changes 6 1
24. eda bug in passing DHCP packets from a Linux VM Fixed bug in RDMA statistics for NDK Removed affinity restriction to group 0 for NDK e Added support for the part man tool It can create only one additional IPoIB interface per port GUID This feature can be utilized to use different IPoIB interfaces for SMB and Hyper V The virtual ports configuration is removed during uninstall Installed ND by default The following tools were added to the setup package e ibv_asyncwatch exe bv atomic bw exe bv atomic lat exe e jbv devinfo exe e iby rc pingpong exe ibv rdma bw exe ibv_rdma_lat exe ijbaddrexe e jbcacheedit exe jblinkinfo exe jbqueryerrors exe ibsysstat exe saquery exe smpdump exe Mellanox Technologies 15 J Rev 4 60 Beta Features 3 Beta Features Table 6 Beta Features Category Description WinVerbs WinVerbs is currently at beta level ibdump ibdump is currently at beta level SR IOV Ethernet SR IOV Ethernet is currently at beta level 16 Mellanox Technologies Rev 4 60 4 Unsupported Functionality Features The following are the unsupported functionality features in WinOF Rev 4 60 ND over WinVerbs provider SRP 4 1 ConnectX 2 Adapter Limitations Limited NDK support prevents SMB Direct support for details contact Mellanox Sup port CQ to EQ feature is not supported consequently the driver may associate RSS an
25. es Changes and Fixes in v4 3 Category Description General e Added support for a new provider called MLX4ND which supports both NDv1 and NDv2 interfaces Performance Enabled performance tuning running according to the operating systems that are running over it The keywords added to the registry in NDIS support Windows 2012 are e RssMaxProcNumber NumRSSQueues RSSProfile The rest of the keywords are added in all versions of NDIS This change is based on http msdn microsoft com en us library windows hardware ff5 70864 v vs 85 aspx 12 Mellanox Technologies Rev 4 60 Table 4 New Features Changes and Fixes in v4 3 Category Description Ethernet e RoCE MTU value is no longer set to 1024 by default All options stay as they are and can only be chosen if they were selected explic itly in the Ul registry The current default state is as follows The value is now derived from the MTU or MaxFramSize or Jumbo Packets value and they are all aliases for the same value The value is aligned to 256 512 1024 2048 in a way that it will be rounded down to the nearest power of two of the ETH MTU InfiniBand e Added ibdiagnet utility support 2 5 New Features Changes and Fixes in Version 4 2 From Version 3 2 0 Table 5 New Features Changes and Fixes in v4 2 Sheet 1 of 3 Category Description General Modified RSS cores and changed VMQ affinity on the fly e Fixe
26. ese rr reser 12 Table 5 New Features Changes and Fixes in V4 2 0 0 cece cee rss reser 13 Table 6 Pen Feres esos e ELS e f rd ng f rre bee Iebescutee ht erri 16 Table7 WinOF Knownlssues n r best ring fred Reed RR e e a 18 Table 8 API Changes in WinOF Rev 4 60 0 ccc cee n 29 Table 9 API Changes in WinOF Rev 4 85 1 0 ccc eee ene nen 29 Table 10 API Changes in WinOF Rev 4 40 0 0 0 ccc cece a 29 4 Mellanox Technologies Rev 4 60 1 Overview This is the release notes for Mellanox WinOF Rev 4 60 VPI drivers The driver intends to improve performance and functionality of the Inbox driver Therefore Mel lanox strongly recommends updating to its latest driver to achieve optimal performance and addi tional functionality 1 1 Mellanox WinOF Rev 4 60 New Features Added support for the following SR IOV Ethernet beta level CIM installation as a standalone package DSCP priority over IPv4 IRQ Dynamic moderation Configuration changes saving upon Inbox and previous releases upgrade e New report for bad cables e iSCSI boot over IPoIB 1 2 WinOF VPI Package Contents The Mellanox WinOF Rev 4 60 for Windows package contains the following components Core and ULPs IB HCA low level drivers mlx4 B Access Layer IBAL Ethernet driver ETH IP over InfiniBand IPoIB e NetworkDirect ND Utilities e OpenSM InfiniBand Subnet Manager is provided as a sample
27. ivers functionality is limited up to 1TB of mem ory and up to 128 cores Upgrade the firmware manu ally Connection failure on ND tests while machine A have IBAL provider and machine B have MLXAND provider Hibernate and Sleep are not functional when user space is using its resources Calling ib join mcast with timeout ms 0 may result in a BSOD Avoid calling this function with timeout ms 0 IPoIB does not support MAC address change e QoS packet priority Load balancing and fail over LBFO Connected Mode e Partition Memory registration on 32bit machines is limited to up to 256GB In an interoperability environment that has both Linux and Windows OSs the MTU value must be the same otherwise packets larger than the minimum will not go through The default MTU for Linux is 2K and for Win dows is 4K Old InfiniBand application which was complied with WinOF earlier than 4 40 must be recompiled with the latest SDK version Mellanox Technologies 21 Rev 4 60 Known Issues Table 7 WinOF Known Issues Sheet 5 of 11 Category Description OpenSM does not run as a service during installation since the SM requires the GUID parameter to decide which port to work on Setting it on setup causes it to work only on the first port and not the others Workaround To run OpenSM as a service assuming the package was installed in the default path use
28. keOnMagicPacket registry key is not added to the registry although WoL is supported by the driver and by the NIC Delete the existing teamed interface and create a new one When the ports of the device are configured as Ethernet only ibstat vstat may display wrong information 22 Mellanox Technologies Rev 4 60 Table 7 WinOF Known Issues Sheet 6 of 11 Category Description Workaround High multicast drop rate on multicast storming Use Multicast traffic tun ing option under the perfor mance tab For further information please refer to section Tun able Performance Parame ters in the User Manual Driver installation requires deletion of the Delete the m1x4 bus sys mlx4 bus sys file in Windows Server 2008 R2 and file and reboot the machine to WLH OSs when using the PXE package install the driver When there is a stress in TCP connection establish Increase the Ring queue ments some of those connections may fail Sizes e ReceiveBuffers controls the receive ring size TransmitBuffers con trols the transmit ring size Wake on Lan WoL cannot be disabled on NICs which supports it The DCB component specifies a default traffic classifi cation that is applied to all egress packets that do not match other classification conditions In this case the network adapter assigns the IEEE 802 1p priority level that is associated with the default classification to these
29. ns an error or empty messages will be displayed MLNX Card LNX FirmwareIdentity MI MLNX FirmwareInstallationServic MI LNX Realizes The maximum values returned by the ib query ca 0 function for example max qp max mr arethe upper limits of the supported resources by the device How ever it may be impossible to use these maximum values since the actual number of any resource that can be cre ated may be limited by the machine configuration the amount of host memory user permissions and the amount of resources already in use by other users pro cesses Running Ntttcp without the a x flag X 21 ina NIC configured with IO0GbE may cause low bandwidth in TCP single stream Run Ntttcp with a 8 for best performance Active links disappear after changing the cable connec tivity from Ethernet to InfiniBand or vice versa Disable and enable the mlx4 bus interface from the Device Manager Rev 4 60 Table 7 WinOF Known Issues Sheet 2 of 11 Category Description Workaround On ConnectX 2 ConnectX 3 Ethernet adapter cards Please use the GUID value there is a mismatch between the GUID value returned returned by the fabric driver by firmware management tools and that returned by fab utilities not Oxfffff ric driver utilities that read the GUID via device firm ware e g using ibstat MIxburn flint return Oxffff as GUID while the utilities return a value derived from
30. ress Settings on the VM Goto Network Adapter gt Hardware Acceleration e Un check the Enable vir tual machine queue Close the UI before driver upgrade Rebooting the machine reboot while uninstalling WinOF may result in installation failure Delete Mellanox components from HKEY LOCAL MACHINENSOF TWARE Microsoft Win dows CurrentVer sion DIFxApp Component S The Mellanox components are mlx4eth63 ipoib6x and mlx4 bus Canceling the installation process may leave the bus driver in a disable state The driver appears in a yellow bang containing the following error message Windows cannot start this hardware device because its configuration information in the registry is incomplete or damaged Code 19 Scan for new hardware and reboot the machine Downgrade is not supported Uninstall the current version and install the older one Rev 4 60 Table 7 WinOF Known Issues Sheet 10 of 11 Category Description Workaround Installation is stuck when Remote Desktop Session Host Disable the Remote Desktop Windows Installer RDS Compatibility is enabled Session Host Windows Installer for the duration of the installation according to the procedure described in http support microsoft com kb 2655192 Occasionally upon driver upgrade due to mismatch of driver versions the following error message is dis played There is an interface mismatch between
31. sc create opensm bin Path c Program Files Mella nox MLNX VPI IB Tools opensm exe To start the service run sc start opensm Tools issues ibportstate does not work on RoCE ports ibdiagpath may crash on Hyper V machines If an application which uses InfiniBand runs while the driver is being restarted a bluescreen or an NMI may occur If OpenSM is up during driver restart on the same machine it might stop working when the driver is being backed up Stop all InfiniBand applica tions including OpenSM upon driver restart To resume operation stop OpenSM and restart the driver while OpenSM is down When configuring one of the ports to be IPoIB Mella nox device appears in the Eject Devices Sometimes during traffic the latency of the IPoIB can get higher this behavior is inconsistent No communication between the physical interface and a VM that uses vSwitch created over virtual IPoIB Pkey and vice versa Set the following registry in the IPoIB network interface RecvIntModTime 0 Please note this action increases the CPU utilization Ethernet Disabling the Priority amp VLAN tag in the UI which VLANID is configured may result in sending packets with the configured VLANID Remove the VLANID before disabling the Priority amp VLAN tag When working with LBFO the teamed interface disap pears after machine reboot The issue applies to Windows Server 2012 R2 Wa
32. the Service the parameter PolicyStore ActiveStore may ActiveStore cause machines to load without Quality of Service pol icy RoCE RoCE does not support Traffic cannot go through the router It works in the same subnet only Multicast traffic VLAN e Layer 3 feature Using different versions of RoCE in your cluster isnot Use the same RoCE version supported in all the cluster in the Ether net ports Performance When using WinOF 4 40 or above low throughput will Disable QoS when it is not in be seen on 40GbE adapters when QoS is enabled use Open a PowerShell prompt Run Disable NetAdapter Qos name Inter face Name where Interface Name is e g Ethernet 1 perf tuning is supported only when one of the two NUMA nodes are in use Execution of nd write lat over mlx4ndv1 or Press CTRL C and kill the mlx4ndv2 may never end test 24 Mellanox Technologies Rev 4 60 Table 7 WinOF Known Issues Sheet 8 of 11 Category Description Workaround Running performance benchmarks for a short period of Set Rx Interrupt Mod time lt 1 sec may provide bad latency in IPoIB and eration Profile and Ethernet Tx Interrupt Modera tion Profile to Low Latency to avoid bad latency Note This may increase CPU utilization Hyper v When the vSwitch is detached from the ETH IPoIB device while the driver is disabled the device does not
33. the MAC address For all driver firmware software pur poses the latter value should be used When working with the default NDv1 and NDv2 pro viders the following error message might be displayed 0xC0000120 NT STATUS CANCELLED This error does not affect any functionality and can be safely ignored Changing the default ND providers may cause random errors such as 0xC0000238 NT STATUS ADDRESS ALREADY ASSOCIATED on Connect or with 0xC0000236 NT STATUS CONNECTION REFUSED on Accept These errors can be safely ignored The UI displays the network interface s state incorrectly To see the correct state and even after disabling and re enabling the disconnected link speed perform one ofthe network interfaces following RunGet netadapter Powershell cmdlet or Right click on that inter face from Network Con nections and click on status Upon bad Cable Recoverable the driver will print in the event log Unsupported cable message and upon unsupported cable the message be Bad Cable winctxtune dll the user interface module of the driver is not digitally signed The winctxtune dll version may be different from the driver s version 4 60 17718 or 4 60 17736 Mellanox Technologies 19 J Rev 4 60 Known Issues Table 7 WinOF Known Issues Sheet 3 of 11 Category InfiniBand Description
34. y e Added a new hardware ID for ConnectX 3 Pro NICs The QP numbers allocation is now round robin manner e RecvCompletionMethod as Interrupt is no longer supported Removed the LsoV1IPv4 from the registry UI Removed from the bus driver configuration the Non DMA option Removed the TXRingNum option from the UI NVGRE e Added NVGRE hardware off load support Added to the UI the EncapsulatedPacketTaskOffload option when using ConnectX 3 Pro NICs Performance e Added the nd send bwand nd send lat ND benchmarking tools e Fixed nd gt bw to achieve better performance memory buffer alignment and consistent results Ethernet Fixed the issue preventing messages to be sent in VLAN 0 when using many VMQ rings Added IP IP checksum off load support Added Ports TX arbitration Bandwidth allocation per port The following ND providers MLX4ND and MLX4ND2 are installed by default InfiniBand IPoIB performance improvements e Fixed a part man issue related to wrong statistics over virtual partman inter faces RoCE Added Sniffer for RoCE packets The used RoCE mode set upon driver load is printed into event log message Mellanox Technologies 9 Rev 4 60 Changes and Major New Features 2 3 New Features Changes and Fixes in Version 4 40 From Version 4 3 Table 3 New Features Changes and Fixes in v4 40 Sheet 1 of 3 Category Description General Added a notific
Download Pdf Manuals
Related Search