Home

Mellanox WinOF-2 User Manual

1. 7 viejo LEER 7 Intended Audiences u Gers wet epa ea e ehe ede Das e vta 7 Documentation Conventions 7 Common Abbreviations and 8 Related Documents odas e rc peg ER NU aida weds ES Us 9 Chapter 1 Introduction ovr ERR RE ERE aD E e EE Cre aeg 10 1 1 SupplhedPackages cece enn eens 10 1 2 WinOF 2 Set of Documentation 10 13 Windows MPI MS MPI 11 Chapter2 Installation 12 2 Hardware and Software Requirements 12 2 2 Downloading Mellanox WinOF 2 12 2 3 Installing Mellanox WinOF 2 Driver 13 2 3 1 Attended Installation llle 13 2 3 2 Unattended Installation yr uy a s lupus pa hA eee nee 18 2 4 MnstallationsResultsy siri She p Od haa es oles ed E CLER 19 2 5 Extracting Files Without Running Installation 19 2 6 Uninstalling Mellanox WinOF 2 21 2 6 1 Attended Uninstallation 0 cee ec ete eens 21 2 6 2 Unattended Uninstallation 0 0 0c ec eect eee nee 22 21 Firmwares Uperades vided dace dali eee oa en SERRA NB e ER
2. Install Uninstall Description Allows your computer to access resources on a Microsoft network Cancel Step 3 Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties Mellanox Technologies 24 Rev 1 21 Step 4 Select the Use the following address radio button and enter desired information General Alternate Configuration You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator for the appropriate IP settings Obtain an IP address automatically Use the following IP address Obtain DNS server address automatically Use the following DNS server addresses Preferred DNS server Alternate DNS server Validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from a CMD console gt ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix Jp Address Sube WEE 6 6 co 252550000 Default Gateway 3 1 3 over Converged Ethernet RoCE Remote Direct Memory Access RDMA is the remote memory management capability that allows server to server data movement directly between a
3. per V Installed Network Policy and Access Services Print and Document Services Remote Access Remote Desktop Services lt Previous Insta Cancel Mellanox Technologies 45 Rev 1 21 fs Add Roles and Features Wizard Select server ro m Add features that are required for Hyper V The following tools are required to manage this feature but do not have to be installed on the same server 4 Remote Server Administration Tools 4 Role Administration Tools 4 Hyper V Management Tools Tools Hyper V Module for Windows PowerShell Tools Hyper V GUI Management Tools Include management tools if applicable DESTINATION SERVER LAB GO75HLITJNQ mtl labs minx x jon provides the services that use to create and manage and their resources ual machine is a virtualized Pr system that operates in an execution environment This to run multiple operating simultaneously lt Previous Next gt Step3 Install Hyper V Management Tools Features gt Remote Server Administration Tools gt Role Administration Tools gt Hyper V Administration Tool Select features Befcre You Begin Select one or more features to install on the selected server Installation Features r Sel LL wurung C Multipath 1 0 Network Load Balancing Peer Name Reso
4. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox WinOF 2 User Manual Rev 1 21 SW version 1 21 51000 www mellanox com Rev 1 21 THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE
5. Step 5 Click Install DESTINATION SERVER Confirm installation selections speed nis i Before You Begin To install the following roles role services or features on selected server click Install Installation Type Restart the destination server automatically if required Optional features such as administration tools might be displayed on this page because they have been selected automatically If you do not want to install these optional features click Previous to clear their check boxes Hyper V Remote Server Administration Tools Role Administration Tools Migration Hyper V Management Tools Default Stores Hyper V GUI Management Tools Hyper V Module for Windows PowerShell Export configuration settings Specify an alternate source path Install Step 6 Reboot the system 3 3 3 2 3 Verifying SR IOV Support within the Host Operating System SR IOV Ethernet Only gt To verify that the system is properly configured for SR IOV Step 1 to Start gt Windows Powershell Mellanox Technologies 47 Rev 1 21 Step2 Run the following PowerShell commands PS Get VmHost lovSupport PS Get VmHost lovSupportReasons In case that SR IOV is supported by the OS the output in the PowerShell is as in the figure below Figure 5 Operating System Supports SR IOV tion All rights IovSupport Note If BIOS was updated according to BIOS vendor instr
6. error on lt Z gt qpn lt q gt cqe_error gt syndrome lt K gt cqe_error gt vendor_error_syndrome lt L gt Opcode lt F gt Therefore the NIC will be reset The issue is reported in Function lt M gt For more information refer to details Mellanox ConnectX 4 VPI Adapter X device reports an EQ stuck on EQn Y Attempting recovery e Mellanox ConnectX 4 VPI Adapter X device reports a send completion handling time out on TxQueue lt Y gt Attempting recovery The issue is reported in Function lt Z gt Mellanox ConnectX 4 VPI Adapter X device reports a receive completion handling timeout on RxQueue lt gt Attempting recovery The issue 1s reported in Function lt 7 gt Mellanox Technologies 81 J Rev 1 21 Flow control on the device Mellanox ConnectX 4 VPI Adapter lt X gt wasn t enabled Therefore RoCE cannot function properly To resolve this issue please make sure that flow control is configured on both the hosts and switches in your network For more details please refer to the user manual Mellanox ConnectX 4 VPI Adapter lt X gt device is configured with a MAC address desig nated as a multicast address lt Y gt Please configure the registry value NetworkAddress with another address then restart the driver The miniport driver initiates reset on device Mellanox ConnectX 4 VPI Adapter X e NDIS initiates reset on device Mellanox ConnectX 4 VPI Adapt
7. e E E Yam ed ise ta 35 Table 8 RDMACACHVILY nose MCA aed CMS 72 Table9 Fabric Performance Utilities __ 73 Table 10 Management Utilities an 75 Table L1 Diagnostic Utilities i ouo eer ve apt Rb ORT are testa 75 Table 12 Installation Related Issues 76 Table 13 Setup Return Codes cei o et arth Maeno a eine AV nae RC tta aka ae 76 Table 14 Firmware Burning Warning Codes 76 Table 15 Restore Configuration Warnings 0 0 cee eee cee 77 Table 16 Ethernet Related Issues a luys yaaa uusha uay 77 Table 17 Performance Related Issues 79 Table 18 Virtualization Related Issues 80 Table 19 Events Causing Automatic State Dumps 84 Mellanox Technologies 5 J Rev 1 21 Document Revision History Table 1 Document Revision History Document Revision Date Changes Rev 1 21 November 2015 Fixed the default value of RoCE to be RoCE v1 25 instead of RoCE v2 September 2015 Added the following section Section 3 3 3 Single Root I O Virtualization SR IOV on page 42 Updated the version number format The UM version for mat was composed of
8. NIC Get NetAdapter EthInterfaceName PS New NetVirtualizationProviderAddress InterfaceIndex SNIC InterfaceIndex Provid erAddress HypervisorInterfaceIPAddress PrefixLength 24 PS New NetVirtualizationProviderRoute InterfaceIndex NIC InterfaceIndex Destina tionPrefix 0 0 0 0 0 NextHop HypervisorInterfaceIPAddress Step 8 Configure the Virtual Subnet ID on the Hyper V Network Switch Ports for each Virtual Machine on each Hyper V Host Host 1 and Host 2 PS Get VMNetworkAdapter VMName lt VMName gt where MacAddress eq lt VMmacaddress1 gt Set VMNetworkAdapter VirtualSubnetID virtualsubnetID Please repeat steps 5 to 8 on each Hyper V after rebooting the Hypervisor 3 3 2 4 Verifying the Encapsulation of the Traffic Once the configuration using PowerShell is completed verifying that packets are indeed encap sulated as configured is possible through any packet capturing utility If configured correctly an encapsulated packet should appear as a packet consisting of the following headers Outer ETH Header Outer IP GRE Header Inner ETH Header Original Ethernet Payload 3 3 2 5 Removing NVGRE configuration Step 1 Set VSID back to 0 on each Hyper V for each Virtual Machine where VSID was set PS Get VMNetworkAdapter lt VMName gt a where MacAddress eq lt VMMacAddress gt b Set VMNetworkAdapter VirtualSubnetID 0 VMName the name of Virtual machine VMMacAddress
9. Appendix A NVGRE Configuration Scripts Examples 86 A l Adding NVGRE Configuration to Host 14 1 86 2 Adding NVGRE Configuration to Host 15 1 87 Appendix B Windows MPI MS MPI 89 Bi 5 Overview vum wee Me eu exem STR RE ND RIS S 89 B 1 1 System Requirements 89 B2 R nhting MPE LAS yana Wb elated 4b Glad Vike Su ERU 89 Directing MSMPI Traffic 89 B 4 Running MSMPI on the Desired 89 B 5 Configuring MPI gresit Re ecd xs wile ata ne s E 90 B 5 1 PFC Example a asa usuyqa hu usya AGO Eq EE NER 90 B 5 2 Running MPI Command 1 91 Mellanox Technologies 4 J Rev 1 21 List of Tables Table 1 Document Revision History 6 Table 2 Documentation Conventions 7 Table 3 Abbreviations and Acronyms 8 Table 4 Related Documents csse pepe ubere pusa DER ed E angled RI UR De 9 Table 5 Hardware and Software Requirements 12 Table 6 Registry Key Parameters 30 Table 7z Registry Keys Setting ios
10. MatchCondition TCP is being used for MPI control channel smpd while UDP is being used for other 3 services such as remote desktop PI Arista switches forwards the pcp bits e g 802 1p priority within the vlan tag from ingress to egress to enable any two End Nodes in the fabric as to maintain the priority along the route In this case the packet from the sender goes out with priority X and reaches the far end node with the same priority X The priority should be losslessin the switches ae gt To force MSMPI to work over ND and not over sockets add the following in mpiexec com mand env MPICH DISABLE ND 0 env MPICH DISABLE SOCK 1 B 5 Configuring MPI Step 1 Configure all the hosts in the cluster with identical PFC see the PFC example below Step 2 Run the WHCK ND based traffic tests to Check PFC ndrping ndping ndrpingpong ndpingpong Step3 Validate PFC counters during the run time of ND tests with Mellanox Adapter QoS Counters in the perfmon Step 4 Install the same version of HPC Pack in the entire cluster NOTE Version mismatch in HPC Pack 2012 can cause MPI to hung Step 5 Validate the MPI base infrastructure with simple commands such as hostname B 5 1 PFC Example In the example below ND and NDK go to priority 3 that configures no drop in the switches The TCP UDP traffic directs ALL traffic to priority 1 Install dcbx Install WindowsFeature Data Center Bridging Rem
11. When interrupt moderation is enabled the system accumulates interrupts and sends a single interrupt rather than a series of interrupts An interrupt is gener ated after receiving 5 packets or after the passing of 10 micro seconds from receiving the first packet The valid values are 0 disable 1 enable RxIntModeration 2 Sets the rate at which the controller moderates or delays the generation of interrupts making it possible to optimize network throughput and CPU utilization The default setting Adaptive adjusts the interrupt rates dynamically depending on traffic type and net work usage Choosing a different setting may improve network and system performance in certain configura tions The valid values are e 1 static 2 adaptive The interrupt moderation count and time are configured dynamically based on traffic types and rate Mellanox Technologies 62 J Rev 1 21 Value Name Default Value Description RSS 1 Sets the driver to use Receive Side Scaling RSS mode to improve the performance of handling incom ing packets This mode allows the adapter port to uti lize the multiple CPUs in a multi core system for receiving incoming packets and steering them to their destination RSS can significantly improve the number of transactions per second the number of connections per second and the network throughput This parameter can be set to one of two values 1 enable default Se
12. Hum 69 2 915 4 gh dev l vrt 058 b W Computer b Disk drives 8 Display adapters b IDE ATA ATAPI controllers 4 IEEE 1394 host controllers Texas Instruments 1394 OHCI Compliant Host Controller p Keyboards b Mice and other pointing devices p Monitors amp Network adapters Broadcom NetXtreme Gigabit Ethernet Broadcom NetXtreme Gigabit Ethernet 2 amp Broadcom NetXtreme Gigabit Ethernet 3 Broadcom NetXtreme Gigabit Ethernet 24 EP Mellanox ConnectX 4 VPI Adapter 3 0 Mellanox ConnectX 4 VPI Adapter 4 5 Other devices Base System Device p 9 Ports COM amp LPT p m Print queues b Processors Storage controllers jl System devices b Universal Serial Bus controllers 2 5 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Windows Server 2012 R2 Click Start gt Task Manager gt File gt Run new task gt and enter CMD Step 2 Extract the driver and the tools gt MINX WinOF2 1 20 All x64 a Toextract only the driver files gt MLNX WinOF2 1 20 All x64 a vMT DRIVERS ONLY 1 Mellanox Technologies 19 Rev 1 21 Step 3 Click Next to create a server image Welcome to the InstallShield Wizard for MLNX VPI The InstallShield R Wizard will install MLNX VPI on your computer To continue click Next WARNING This program is pr
13. the MAC address of VM s network interface associated with vSwitch that was connected to Mellanox device Step2 Remove all lookup records same command on all Hyper V hosts PS Remove NetVirtualizationLookupRecord Step3 Remove customer route same command on all Hyper V hosts PS Remove NetVirtualizationCustomerRoute Step 4 Remove Provider address same command on all Hyper V hosts PS Remove NetVirtualizationProviderAddress Mellanox Technologies 41 Rev 1 21 Step 5 Remove provider routed for a Hyper V host PS Remove NetVirtualizationProviderRoute Step 6 For HyperV running Windows Server 2012 only disable network adapter binding to ms netwnv service PS Disable NetAdapterBinding lt EthInterfaceName gt a ComponentID ms netwnv EthInterfaceName Physical NIC name 3 3 3 Single Root I O Virtualization SR IOV Single Root I O Virtualization SR IOV is a technology that allows a physical PCIe device to present itself multiple times through the PCIe bus This technology enables multiple virtual instances of the device with separate resources Mellanox adapters are capable of exposing in ConnectX 4 ConnectX 4 Lx adapter cards up to 32 virtual instances called Virtual Functions VFs These virtual functions can then be provisioned separately Each VF can be seen as an addition device connected to the Physical Function It also shares resources with the Physical Function SR IOV is commo
14. thus shortening the execution latency For best performance it is recommended to use the BlueFlame when the HCA is lightly loaded For high bandwidth scenarios it is recommended to use regular posting without BlueFlame The valid values are 0 disable 1 enable Note This registry value is not exposed via the UI MaxRSSProcessors 8 The maximum number of RSS processors Note This registry key is only in Windows Server 2012 and above 3 4 5 Ethernet Registry Keys The following section describes the registry keys that are only relevant to Ethernet driver Value Name Default Value Description RoceMaxFrameSize 1024 The maximum size of a frame or a packet that can be sent by the RoCE protocol a k a Maximum Transmission Unit MTU Using larger MTU will improve the performance however one must ensure that the entire system including switches supports the defined MTU Ethernet packet uses the general MTU value whereas the packet uses the RoCE MTU The valid values are 256 512 1024 2048 Note This registry key is supported only in Ethernet drivers PriorityVLANTag 3 Packet Pri Enables sending and receiving IEEE 802 3ac tagged frames ority amp which include VLAN 802 1p QoS Quality of Service tags for priority tagged pack Enabled ets 802 10 tags for VLANs When this feature is enabled the Mellanox driver supports sending an
15. 22 Chapter 3 Features Overview and Configuration 23 3d Ethernet Network 2 6 ou hhc Rede tuse Bae we cepacia 23 311 Mode Contiguration PENA ORE WEN CA PY LER 23 3 1 2 Assigning Port IP After Installation 23 3 1 3 over Converged Ethernet RoCE 25 3 1 4 Teaming and VLAN hh e 30 3 1 5 Configuring Quality of Service QoS 32 3 1 6 Configuring the Ethernet Driver 0 0 0 cece cece eee nee 35 3 1 7 Receive Side Scaling RSS 35 3 2 Storage Protocols peer ex tek D Seg on ev ates 36 3 2 1 Deploying SMB Direct 0 0 cece ete 36 3 3 soe eese n le pies bon Reine Sep CER RH 38 3 1 Hyper V with WEE BT TUS A COME 38 3 3 2 Network Virtualization using Generic Routing Encapsulation NVGRE 38 3 3 3 Single Root I O Virtualization 1 42 3 4 Configuration Using Registry Keys 56 3 4 1 Finding the Index Value of the Network Interface 56 Mellanox Technologies 3 J Rev 1 21 34 2 Basic Registry Keys in RR HU RIED ee e D buba 58 3 4 3 Off load Registry e 60 3 4 4 Performance Registry Ke
16. GT s Link Speed 56 0 Gbps 40 0Gbps 10 0Gbps 100 Gbps Issue 2 To determine if the Mellanox NIC and PCI bus can achieve their maximum speed it s best to run nd_send_bw in a loopback On the same machine 1 Run start b affinity 0 1 nd_send_bw S lt IP_host gt where lt IP_host gt is the local IP 2 Run start b affinity 0x2 nd_send_bw C lt IP_host gt 3 Repeat for port 2 with the appropriate IP 4 On PCI Gen3 the expected result is around 5700MB s On PCI Gen2 the expected result is around 3300MB s Any number lower than that points to bad configuration or installation on the wrong PCI slot Malfunctioning QoS settings and Flow Control can be the cause as well Issue 3 To determine the maximum speed between the two sides with the most basic test 1 Run nd send bw S lt IP_host1 gt on machine 1 where lt IP_host1 gt is the local IP 2 Run nd send bw C lt IP_host1 gt on machine 2 3 Results appear in Gb s Gigabits 2 30 and reflect the actual data that was trans ferred excluding headers 4 If these results are not as expected the problem is most probably with one or more of the following Old Firmware version Misconfigured Flow control Global pause or PFC is configured wrong on the hosts routers and switches See Section 3 1 3 RDMA over Converged Ethernet RoCE on page 25 e CPU power options are not set to Maximum Performance Mellanox Technologies 79 J Rev 1 21 5 4
17. I OAT supported systems it is highly recommended to install and enable the latest I OAT driver download from www intel com With I OAT enabled sending 256 byte messages or larger will activate I OAT This will cause a significant latency increase due to I OAT algorithms On the other hand throughput will increase significantly when using I OAT 3 5 3 Tunable Performance Parameters The following is a list of key parameters for performance tuning Jumbo Packet The maximum available size of the transfer unit also known as the Maximum Transmission Unit MTU The MTU of a network can have a substantial impact on performance A 4K MTU size improves performance for short messages since it allows the OS to coalesce many small messages into a large one Valid MTU values range for an Ethernet driver is between 614 and 9614 4 All devices on the same physical network or the same logical network must have 3 the same MTU Receive Buffers The number of receive buffers default 1024 Send Buffers The number of sent buffers default 2048 Performance Options Configures parameters that can improve adapter performance Interrupt Moderation Moderates or delays the interrupts generation Hence optimizes network throughput and CPU uti lization default Enabled Mellanox Technologies 69 J Rev 1 21 When the interrupt moderation is enabled the system accumulates interrupts and sends a single inter
18. Rev 1 21 License file User Manual this document MLNX WinOF 2 Release Notes 1 3 Windows MPI MS MPI Message Passing Interface MPI is meant to provide virtual topology synchronization and com munication functionality between a set of processes MPI enables running one process on several hosts Windows MPI runs over the following protocols Sockets Ethernet Network Direct ND For further details on MPI please refer to Appendix B Windows MPI MS MPI on page 89 Mellanox Technologies 11 Rev 1 21 2 Installation 2 1 Hardware and Software Requirements Table 5 Hardware and Software Requirements Description Package Windows Server 2012 R2 64 bit only MLNX WinOF2 1 21 All x64 exe Windows Server 2012 64 bit only MLNX WinOF2 1 21 All x64 exe a The Operating System listed above must run with administrator privileges 2 2 Downloading Mellanox WinOF 2 Driver To download the exe according to your Operating System please follow the steps below Step 1 Obtain the machine architecture For Windows Server 2012 2012 R2 1 To go to the Start menu position your mouse in the bottom right corner of the Remote Desktop of your screen 2 Open a CMD console Click Task Manager gt File gt Run new task and enter CMD 3 Enter the following command gt echo PROCESSOR ARCHITECTURES On an x64 64 bit machine the output will be AMD64 Step 2 Go to the Mellan
19. Technologies 42 Rev 1 21 3 3 3 2 Configuring SR IOV Host Machines The following are the necessary steps for configuring host machines 3 3 3 2 1 Enabling SR IOV in BIOS Depending on your system perform the steps below to set up your BIOS The figures used in this section are for illustration purposes only For further information please refer to the appropriate BIOS User Manual gt To enable SR IOV in BIOS Step 1 Make sure the machine s BIOS supports SR IOV Please consult BIOS vendor website for SR IOV supported BIOS versions list Update the BIOS version if necessary Step2 Follow BIOS vendor guidelines to enable SR IOV according to BIOS User Manual For example a Enable SR IOV BLOS SETUP UTILITY Advanced PCI PnP Settings WARNING Setting wrong values in below sections Disabled may cause system to malfunction Enabled Mellanox Technologies 43 J Clear NUR H Plug amp Play 0 5 FCI Latency Timer PCI IDE Sloti PCI X OPROH Slot2 PCI X OPROH Slot3 PCI X OPROM Slot4 PCI E OPROH Slot5 PCI E OPROH Slot6 PCI E OPROH Nol Yes 641 Disabled Enabled Enabled Enabled Enabled Enabled Enabled Load Onboard LAN 1 Option ROM Enabled Load Onboard LAN 2 Option RUM Disabled Onboard LAN Option Rom Select PHE Roots Graphic Adapter Priority Onboard UGAl Select Screen Select Item Change Option General Help F10 Save and Exit FSC Exit v02 6
20. Technologies 49 Rev 1 21 Figure 8 Connect Virtual Hard Disk Connect Virtual Hard Disk Before You Begin 4 virtual machine requires storage so that you can install an operating system You can specify the storage now or configure it later by modifying the virtual machine s properties Specify Name and Location 9 g Y ying prop Assign Memory Create a virtual hard disk Configure Networking Use this option to create a dynamically expanding virtual hard disk with the default format VHDX Connect Virtual Hard Disk vm1 vhdx Summary C YLIsersiPubliciDocumentsiHyper ViVirtual Hard Disks 127 GB Maximum 64 TB Use an existing virtual hard disk Use this option to attach an existing virtual hard disk either YHD or VHDX Format Location XWin8Srv DC x64 fre 9200 vm1 vhd Browse D Attach a virtual hard disk later Use this option to skip this step now and attach an existing virtual hard disk later lt Previous ini Cancel 3 3 3 3 Configuring Mellanox Network Adapter for SR IOV The following are the steps for configuring Mellanox Network Adapter for SR IOV 3 3 3 3 1 Enabling SR IOV in Firmware For non Mellanox OEM branded cards you may need to download and install the new firm ware For the latest OEM firmware please go to http www mellanox com page oem_firmware_download As of firmware version 2 31 5000 SR IOV can be enabled and managed by usi
21. Virtualization Related Troubleshooting Table 18 Virtualization Related Issues Issue Cause Solution When enabling the Might be missing critical Please refer to VMQ incase NVGRE Microsoft updates http support microsoft com kb offload is enabled anda 2975719 teaming of two virtual August 2014 update rollup for Win ports is performed no dows server RT 8 1 Windows server ping is detected 8 1 and Windows server 2012 R2 between the VMs and specifically fixes or ping is detected but no establishing of TCP connection is possible Mellanox Technologies 80 J Rev 1 21 5 5 Reported Driver Events The driver records events in the system log of the Windows server event system which can be used to identify diagnose and predict sources of system problems To see the log of events open System Event Viewer as follows Right click on My Computer click Manage and then click Event Viewer OR 1 Click start gt Run and enter eventvwr exe 2 In Event Viewer select the system log The following events are recorded Mellanox ConnectX 4 VPI Adapter lt X gt device has been requested for lt Y gt Virtual Func tions VFs while it only supports lt Z gt VFs Therefore only lt q gt VFs will be allowed e Mellanox ConnectX 4 VPI Adapter X device has been configured to use RSS while Windows TCP RSS is disabled This configuration prevents the initialization and enabling of the po
22. adding DumpE ventsNum DWORD32 parameter under HKLM System CurrnetControlSet Services mlx4_ bus Parameters and setting it to another value Mellanox Technologies 85 J Rev 1 21 Appendix A NVGRE Configuration Scripts Examples The setup 15 as follow for both examples below Hypervisor mtlael4 Port1 192 168 20 114 24 VM on mtlael4 mtlae14 005 172 16 14 5 16 Mac 00155D720100 on mtlael4 mtlae14 006 172 16 14 6 16 00155D720101 Hypervisor mtlael5 Port1 192 168 20 115 24 VM on mtlae15 mtlae15 005 172 16 15 5 16 Mac 00155D730100 VM on mtlae15 mtlae15 006 172 16 15 6 16 Mac 00155D730101 A 1 Adding NVGRE Configuration to Host 14 Example The following is an example of adding NVGRE to Host 14 On both sides vSwitch create command Note that vSwitch configuration is persistent no need to configure it after each reboot New VMSwitch VSwMLNX NetAdapterName Port1 AllowManagementOS true Shut down VMs Stop VM Name mtlael4 005 Force Confirm Stop VM Name mtlael4 006 Force Confirm Connect VM to vSwitch maybe you have to switch off VM before doing manual does also work Connect VMNetworkAdapter VMName mtlael4 005 SwitchName VSwMLNX Add VMNetworkAdapter VMName mtlael4 005 SwitchName VSwMLNX StaticMacAddress 00155D720100 Add VMNetworkAdapter VMName mtlael4 006 SwitchName VSwMLNX StaticMacAddress 00155D720101 The commands from Step 2 4 are
23. an interrupt is generated on the send side default 0 Interrupt Moderation TX Packet Time Maximum elapsed time in usec between the sending of a packet and the generation of an inter rupt even if the moderation count has not been reached default 0 Offload Options Allows you to specify which TCP IP offload settings are handled by the adapter rather than the operating system Enabling offloading services increases transmission performance as the offload tasks are per formed by the adapter hardware rather than the operating system Thus freeing CPU resources to work on other tasks IPv4 Checksums Offload Enables the adapter to compute IPv4 checksum upon transmit and or receive instead of the CPU default Enabled TCP UDP Checksum Offload for IPv4 packets Enables the adapter to compute TCP UDP checksum over IPv4 packets upon transmit and or receive instead of the CPU default Enabled TCP UDP Checksum Offload for IPv6 packets Enables the adapter to compute TCP UDP checksum over IPv6 packets upon transmit and or receive instead of the CPU default Enabled Large Send Offload LSO Allows the TCP stack to build a TCP message up to 64KB long and sends it in one call down the stack The adapter then re segments the message into multiple TCP packets for transmission on the wire with each pack sized according to the MTU This option offloads a large amount of kernel processing time from the host CPU to the adapter 3
24. byte msb Most significant bit NIC Network Interface Card NVGRE Network Virtualization using Generic Routing Encapsulation SW Software VPI Virtual Protocol Interconnect IPoIB IP over InfiniBand PFC Priority Flow Control PR Path Record RDS Reliable Datagram Sockets RoCE RDMA over Converged Ethernet SL Service Level Mellanox Technologies 8 J Rev 1 21 Table 3 Abbreviations and Acronyms Sheet 2 of 2 Abbreviation Acronym Whole Word Description MPI Message Passing Interface QoS Quality of Service Related Documents Table 4 Related Documents Document Description MFT User Manual Describes the set of firmware management tools for a single Infini Band node MFT can be used for Generating a standard or customized Mellanox firmware image Querying for firmware information Burning a firmware image to a single InfiniBand nodeEnabling changing card configuration to support SRIOV WinOF 2 Release Notes For possible software issues please refer to WinOF 2 Release Notes ConnectX 4 Firmware For possible firmware issues please refer to ConnectX 4 Firm Release Notes ware Release Notes Mellanox Technologies 9 J Rev 1 21 1 Introduction This User Manual describes installation configuration and operation of WinOF 2 driver Rev 1 21 package Mellanox WinOF 2 is composed of several software modules that contain Ethernet d
25. can be offloaded to a physical network adapter Boot From CD Virtual machine queue Beda Virtual machine queue requires physical network adapter that supports 6 MB this Feature Pi n ene Enable virtual machine queue Virtual pre IDE Controller 0 Hard Drive IPsec task offloading Win8Srv DC x64 fre 920 Support from a physical network adapter and the quest operating system is E IDE Controller 1 required to offload IPsec tasks amp DVD Drive When sufficient hardware resources are not available the security associations None are not offloaded and are handled in software by the guest operating system SCSI Controller 9 Network Adapter Enable IPsec task offloading Not connecte d Select the maximum number of offloaded security associations from a range of 1 to amp Network Adapter 4096 Internal Virtaal Maximum number 512 Offloaded 5A S 4 Network Adapter Mellanox SRIOV Virtual Switch Hardware Acceleration Single root I O virtualization Advanced Features Single root I O virtualization SR IOV requires specific hardware It also might 79 1 require drivers to be installed in the quest operating system None When sufficient hardware resources are not available network connectivity is com2 provided through the virtual switch None v Enable SR IOV Diskette Drive None amp Management Name
26. devices and allows applica tions to leverage advances of RDMA Both RoCE and InfiniBand IB can implement NDI For further information please refer to http msdn microsoft com en us library cc904397 v vs 85 aspx Mellanox Technologies 67 J Rev 1 21 3 5 Performance Tuning and Counters For further information on WinOF 2 performance please refer to the Performance Tuning Guide for Mellanox Network Adapters This section describes how to modify Windows registry parameters in order to improve performance Please note that modifying the registry incorrectly might lead to serious problems including the loss of data system hang and you may need to reinstall Windows As such it is recommended to 3 back up the registry on your system before implementing recommendations included in this sec tion If the modifications you apply lead to serious problems you will be able to restore the original registry state For more details about backing up and restoring the registry please visit www micro soft com 3 5 1 General Performance Optimization and Tuning To achieve the best performance for Windows you may need to modify some of the Windows registries 3 5 1 1 Registry Tuning The registry entries that may be added changed by this General Tuning procedure are Under HKEY LOCAL MACHINENSSYSTEM CurrentControlSetServicesVTcpip Parameters Disable TCP selective acks option for better cpu utilization SackOpts type
27. in the following order Not configured v J PowerShell scripts require at least Windows 7 or Windows Server 7 2008 R2 Show Files OK Cancel 5 Click Add The script should include only the following commands PS Remove NetQosTrafficClass PS Remove NetQosPolicy Confirm False PS set NetQosDcbxSetting Willing 0 PS New NetQosPolicy SMB Policystore Activestore NetDirectPortMatchCondition 445 PriorityValue8021Action 3 Mellanox Technologies 34 Rev 1 21 PS New NetQosPolicy DEFAULT Policystore Activestore Default PriorityVal ue8021Action 3 PS New NetQosPolicy TCP Policystore Activestore IPProtocolMatchCondition TCP PriorityValue8021Action 1 PS New NetQosPolicy UDP Policystore Activestore IPProtocolMatchCondition UDP PriorityValue8021Action 1 PS Disable NetQosFlowControl 0 1 2 4 5 6 7 PS 5 Enable NetAdapterQos InterfaceAlias port1 PS Enable NetAdapterQos InterfaceAlias port2 PS Enable NetQosFlowControl Priority 3 PS New NetQosTrafficClass name SMB class priority 3 bandwidthPercentage 50 Algorithm ETS 6 Browse for the script s location 7 Click OK 8 To confirm the settings applied after boot run PS get netgospolicy policystore activestore 3 1 6 Configuring the Ethernet Driver The following steps describe how to configure advanced features Step 1 Display the Device Manager Step2 Right click a M
28. netsh stack and the Mellanox int tcp set global rss adapter enabled or a less recommended suggestion as it will cause low performance Disable RSS on the adapter run netsh int tcp set global rss no dynamic balancing Mellanox Technologies 77 Table 16 Ethernet Related Issues Rev 1 21 Issue Cause Solution The driver fails to start and a yellow sign appears near the Mel lanox ConnectX 4 Adapter lt X gt in the Device Manager dis play Code 10 Look into the Event Viewer to view the error If the failure occurred due to unsup ported mode type refer to Section 3 1 1 Mode Configuration for the solution If the solution isn t mentioned in event viewer disable and re enable Mellanox ConnectX 4 Adapter lt x gt from the Device Manager dis play If the failure resumes please refer to Mellanox support at support mellanox com No connectivity to a Fault Tolerance team while using network capture tools e g Wireshark The network capture tool might have captured the network traffic of the non active adapter in the team This is not allowed since the tool sets the packet filter to promiscuous thus causing traffic to be trans ferred on multiple inter faces Close the network capture tool on the physical adapter card and set it on the team interface instead No Ethernet connectiv ity on 10Gb adapters after activating Perfor m
29. three numbers major minor and sub minor The sub minor version was removed from the UM Rev 1 20 September 2015 Added the following sections Section 2 2 Downloading Mellanox WinOF 2 Driver on page 12 Section 3 3 Virtualization on page 38 Section 5 4 Virtualization Related Troubleshooting on page 80 Appendix A NVGRE Configuration Scripts Exam ples on page 86 Section 3 1 1 Mode Configuration Section 4 2 Management Utilities Rev 1 10 July 8 2015 Updated the following sections Section 1 Introduction on page 10 Section 3 1 3 1 IP Routable RoCEv2 on page 26 Section 3 1 3 6 Configuring the RoCE Mode on page 30 Rev 1 10 June 2015 Beta Release Mellanox Technologies 6 J Rev 1 21 About this Manual Scope Mellanox WinOF 2 is the driver for adapter cards based on the Mellanox ConnectX 4 family of adapter IC devices It does not support earlier Mellanox adapter generations The document describes WinOF 2 Rev 1 21 features performance diagnostic tools content and configuration Additionally this document provides Information on various performance tools supplied with this version Intended Audience This manual is intended for system administrators responsible for the installation configuration management and maintenance of the software and hardware of Ethernet adapter cards It is also intended
30. vmi Step 7 Start and connect to the Virtual Machine Select the newly created Virtual Machine and go to Actions panel gt Connect In the virtual machine window go to Actions gt Start Step 8 Copy the WinOF driver package to the VM using Mellanox VMNIC IP address Step 9 Install WinOF driver package on the VM Step 10 Reboot the VM at the end of installation Step 11 Verify that Mellanox Virtual Function appears in the device manager Mellanox Technologies 55 Rev 1 21 Figure 12 Virtual Function in the VM File Action View Help E E g 1612 S 4 dev l vrt 227 005 b amp Audio inputs and outputs b Computer b Disk drives b 8 Display adapters gt fj DVD CD ROM drives gt e Floppy disk drives gt Floppy drive controllers b amp Human Interface Devices gt IDE ATA ATAPI controllers gt amp Keyboards b n Mice and other pointing devices K Monitors 4 xv Network adapters Intel 21140 Based PCI Fast Ethernet Adapter Emulated Mellanox ConnectX 4 Virtual Adapter AP Microsoft Hyper V Network Adapter b Y Ports COM amp LPT b Print queues b B Processors b Storage controllers b gil System devices 3 4 Configuration Using Registry Keys 3 4 4 Finding the Index Value of the Network Interface To find the index value of your Network Interface from the Device Manager please perform the following steps Step 1 Open Device Manager and go to Network Ada
31. 1 gt EQ STUCK TXCQ STUCK and RXCQ STUCK events can be disabled by adding Dis ableDumpOnEqStuck DisableDumpOnTxCqStuck and DisableDumpOnRxCqStuck parameters as follows DisableDumpOnTxCqStuck i DisableDumpOnTxCqStuck 1 DisableDumpOnRxCqStuck i The set consists of the following files 3 consecutive mstdump files These files are created in the SystemRoot temp directory and should be sent to Mellanox Support for analysis when debugging WinOF2 driver problems Their names have the following format Driver mode of work card location event tag name event num ber event name file type file gt where Driver mode of work The mode of driver work For example SingleFunc card location In form bus device function For example 4 0 0 event tag name One symbol tag See in Table 19 Events Causing Automatic State Dumps on page 84 event number The index of dump files set and created for this event This number is restricted by the hidden Registry parameter DumpEventsNum event name A short string naming the event For example eth down 1 Ethernet port passed to DOWN state file type Type of file in the set For example crspace fwtrace eq dump and eq print file index The file number of this type in the set Example Name SingleFunc 4 0 0 p000 eth down 1 eq dump O log The default number of sets of files for each event is 20 It can be changed by
32. 1 5 6 State Dumping Upon several types of events the drivers can produce a set of files reflecting the current state of the adapter Automatic state dumps are done upon the following events Table 19 Events Causing Automatic State Dumps Event Type Description Provider ii Tag CMD ERR Command failure or timeout on a command MIx5 On EQ STUCK Driver decided that an event queue is stuck 5 TXCQ STUCK Driver decided that a transmit completion queue MIx5 On t is stuck RXCQ STUCK Driver decided that a receive completion queue 15 MIx5 On r stuck PORT_STATE Adapter passed to port down state or port MIx5 Off p unknown state ON_OID User application asked to generate dump files MIx5 N A o where Provider The driver creating the set of files Default Whether or not the state dumps are created by default upon this event Tag Part of the file name used to identify the event that has triggered the state dump Dump events can be enabled disabled by adding DWORD32 parameters into HKLM Sys tem CurrnetControlSet Services m1x5 Parameters Diag as follows gt Dump events can be disabled by adding MstDumpMode parameter as follows MstDumpMode 0 gt PORT STATE events can be enabled by adding EnableDumpOnUnknownLink and EnableDumpOnPortDown parameters as follows EnableDumpOnUnknownLink EnableDumpOnPortDown a Mellanox Technologies 84 J Rev 1 2
33. 2 1 2 Verifying SMB Events that Confirm RDMA Connection gt To confirm RDMA connection verify the SMB events Step 1 Open a PowerShell window on the SMB client Step 2 Run the following cmdlets NOTE Any RDMA related connection errors will be displayed as well PS Get WinEvent LogName Microsoft Windows SMBClient Operational Message match RDMA P For further details on how to configure the switches to be lossless please refer to 2d https community mellanox com 1 The NETSTAT command confirms if the File Server is listening on the RDMA interfaces Mellanox Technologies 37 J Rev 1 21 3 3 Virtualization 3 3 4 Hyper V with VMQ 3 3 1 1 System Requirements Operating Systems Windows Server 2012 and Windows Server 2012 R2 3 3 1 2 Using Hyper V with VMQ Mellanox WinOF 2 Rev 1 21 includes a Virtual Machine Queue VMQ interface to support Microsoft Hyper V network performance improvements and security enhancement VMQ interface supports Classification of received packets by using the destination MAC address to route the packets to different receive queues NIC ability to use DMA to transfer packets directly to a Hyper V child partition s shared memory Scaling to multiple processors by processing packets for different virtual machines on different processors gt To enable Hyper V with VMQ using UI Step 1 Open Hyper V Manager Step2 Right click the desired Virtual Machine VM an
34. 5 4 Adapter Proprietary Performance Counters Proprietary Performance Counters are used to provide information on Operating System applica tion service or the drivers performance Counters can be used for different system debugging purposes help to determine system bottlenecks and fine tune system and application perfor mance The Operating System network and devices provide counter data that the application can consume to provide users with a graphical view of the system s performance quality WinOF counters hold the standard Windows CounterSet API that includes Network Interface activity SMB Direct Connection Mellanox Technologies 71 J Rev 1 21 3 5 4 0 1 RDMA Activity RDMA Activity counter set consists of NDK performance counters These performance counters allow you to track Network Direct Kernel activity including traffic rates errors and control plane activity Table 8 RDMA Activity RDMA Activity Counters Description MA Accepted Connections The number of inbound RDMA connections established The number of active RDMA connections RD RDMA Active Connections RD MA Completion Queue This counter is not supported and always is set to zero Errors RDMA Connection Errors The number of established connections with an error before a consumer disconnected the connection RDMA Failed Connection The number of inbound and outbound RDMA connection At
35. 8 OCupuright 1985 2009 imer ican Meyatrends Inc f Rev 1 21 b Enable Intel Virtualization Technologhy Support BIOS SETUP UTILITY I ng ization Tech Hit For further details please refer to the vendor s website Mellanox Technologies 44 Rev 1 21 3 3 3 2 2 Installing Operating System SR IOV Ethernet Only gt To install Hypervisor Operating System Step 1 Install Windows Server 2012 R2 Step 2 Install Hyper V role Go to Server Manager gt Manage gt Add Roles and Features and set the following Installation Type gt Role based or Feature based Installation Server Selection gt Select a server fro the server pool Server Roles gt Hyper V see figures below DESTINATION SERV Select server roles tier Select one or more roles to install on the selected server Roles Description Hyper V provides the services that you can use to create and manage Active Directory Domain Services virtual machines and their resources Active Directory Certificate Services Active Directory Federation Services Each virtual machine is a virtualized computer system that operates in an isolated execution environment This Active Directory Rights Management Services allows you to run multiple operating Active Directory Lightweight Directory Services Application Server systems simultaneously DHCP Server DNS Server Fax Server
36. InterfaceIndex ProviderAd dress 192 168 20 114 PrefixLength 24 New NetVirtualizationProviderRoute InterfaceIndex NIC InterfaceIndex Destination Prefix 0 0 0 0 0 NextHop 192 168 20 1 Step 5 Configure the Virtual Subnet ID on the Hyper V Network Switch Ports for each Virtual Machine on each Hyper V Host Host 1 and Host 2 Run the command below for each VM on the host the VM is running on it i e the for mtlael4 005 mtlael4 006 on host 192 168 20 114 and for VMs mtlae15 005 mtlael5 006 on host 192 168 20 115 mtlael4 only Get VMNetworkAdapter VMName mtlae14 005 where MacAddress eq 00155D720100 Set VMNetworkAdapter VirtualSubnetID 5001 Get VMNetworkAdapter VMName mtlael4 006 where MacAddress eq 00155D720101 Set VMNetworkAdapter VirtualSubnetID 5001 2 Adding NVGRE Configuration to Host 15 Example The following is an example of adding NVGRE to Host 15 On both sides vSwitch create command Note that vSwitch configuration is persistent no need to configure it after each reboot New VMSwitch VSwMLNX NetAdapterName Portl AllowManagementOS true Shut down VMs Stop VM Name mtlael15 005 Force Confirm Stop VM Name mtlae15 006 Force Confirm Connect VM to vSwitch maybe you have to switch off VM before doing manual does also work Connect VMNetworkAdapter VMName mtlael4 005 SwitchName VSwMLNX Add VMNetworkAdapter VMName mtlael5 005 SwitchName VS
37. MASK 11 0 0 0 255 0 0 0 env MPICH ND ZCOPY THRESHOLD 1 env MPICH DISABLE ND 1 env MPICH DISABLE SOCK 0 affinity c testl exe Mellanox Technologies 91 J
38. Name VM Name Force Confirm Step 4 Configure the Virtual Subnet ID on the Hyper V Network Switch Ports for each Virtual Machine on each Hyper V Host Host 1 and Host 2 PS Add VMNetworkAdapter VMName VMName SwitchName vSwitchName StaticMacAddress StaticMAC Address Step 5 Configure a Subnet Locator and Route records on all Hyper V Hosts same command on all Hyper V hosts PS New NetVirtualizationLookupRecord CustomerAddress VMInterfaceIPAddress 1 n ProviderAddress lt HypervisorInterfaceIPAddress1 gt VirtualSubnetID lt virtualsubnetID gt MACAddress VMmacaddressi Rule TranslationMethodEncap PS New NetVirtualizationLookupRecord CustomerAddress VMInterfaceIPAddress 2 n ProviderAddress HypervisorInterfaceIPAddress2 VirtualSubnetID virtualsubnetID MACAddress lt VMmacaddress2 gt Rule TranslationMethodEncap a This is the VM s MAC address associated with the vSwitch connected to the Mellanox device Mellanox Technologies 40 J Rev 1 21 Step 6 Add customer route on all Hyper V hosts same command on all Hyper V hosts PS New NetVirtualizationCustomerRoute RoutingDomainID 11111111 2222 3333 4444 000000005001 VirtualSubnetID virtualsubnetID DestinationPrefix VMInterfaceIPAd dress Mask NextHop 0 0 0 0 Metric 255 Step 7 Configure the Provider Address and Route records on each Hyper V Host using an appro priate interface name and IP address PS
39. OF 2 and allows a performance boost for Hyper V VMs For more details about VMQ please refer to Microsoft web site http msdn microsoft com en us library windows hardware ff571034 v vs 85 aspx Value Name Default Value Description VMQ 1 The support for the virtual machine queue VMQ features of the network adapter The valid values are 1 enable e 0 disable Mellanox Technologies 66 J Rev 1 21 Value Name Default Value Description RssOrVmaPrefer 0 Specifies whether VMQ capabilities should be enabled ence instead of receive side scaling RSS capabilities The valid values are 0 Report RSS capabilities 1 Report VMQ capabilities Note This registry value is not exposed via the UI VMQVlanFiltering Specifies whether the device enables or disables the ability to filter network packets by using the VLAN identifier in the media access control MAC header The valid values are 0 disable 1 enable 3 4 6 Network Direct Interface The Network Direct Interface NDI architecture provides application developers with a net working interface that enables zero copy data transfers between applications kernel bypass I O generation and completion processing and one sided data transfer operations NDI is supported by Microsoft and is the recommended method to write an RDMA application NDI exposes the advanced capabilities of the Mellanox networking
40. POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX CloudX logo Connect IB ConnectX CoolBox CORE Direct GPUDirect InfiniHost InfiniScale Kotura Kotura logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox ScalableHPC Mellanox Connect Accelerate Outperform logo Mellanox Virtual Modular Switch MetroDX MetroX MLNX OS Open Ethernet logo PhyX SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd Accelio CyPU FPGADirect HPC X InfiniBridge LinkX Mellanox Care Mellanox CloudX Mellanox Multi Host Mellanox NEO Mellanox PeerDirect Mellanox Socket Direct Mellanox Spectrum NVMeDirect StPU Spectrum logo Switch IB Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 3280 Rev 1 21 Table of Contents Document Revision History a 6 About this Manual aeaea a o
41. REG DWORD value set to 0 Under HKEY LOCAL MACHINE SYSTEM CurrentControlSet Services AFD Parameters Enable fast datagram sending for UDP traffic FastSendDatagramThreshold type REG DWORD value set to 64K Under LOCAL MACHINENSSYSTEM CurrentControlSetServicesNdis Parameters Set RSS parameters RssBaseCpu type REG DWORD value set to 1 3 5 1 2 Enable RSS Enabling Receive Side Scaling RSS is performed by means of the following command netsh int tcp set global rss enabled 3 5 1 3 Improving Live Migration In order to improve live migration over SMB direct performance please set the following regis try key to 0 and reboot the machine HKEY LOCAL MACHINE VSystemVCurrentControlSetV Services LanmanServerV Parameters RequireSecuritySig nature Mellanox Technologies 68 J Rev 1 21 3 5 2 Application Specific Optimization and Tuning 3 5 2 1 Ethernet Performance Tuning The user can configure the Ethernet adapter by setting some registry keys The registry keys may affect Ethernet performance gt Toimprove performance activate the performance tuning tool as follows Step1 Start the Device Manager open a command line window and enter devmgmt msc Step 2 Open Network Adapters Step 3 Right click the relevant Ethernet adapter and select Properties Step 4 Select the Advanced tab Step 5 Modify performance parameters properties as desired 3 5 2 1 1 Performance Known Issues On Intel
42. Shell cmdlets to verify Network Direct is globally enabled and that you have NICs with the RDMA capability Run on both the SMB server and the SMB client PS Get NetOffloadGlobalSetting Select NetworkDirect PS Get NetAdapterRDMA PS Get NetAdapterHardwareInfo 3 2 1 1 2 Verifying SMB Configuration Use the following PowerShell cmdlets to verify SMB Multichannel is enabled confirm the adapters are recognized by SMB and that their RDMA capability is properly identified Mellanox Technologies 36 J Rev 1 21 On the SMB client run the following PowerShell cmdlets PS Get SmbClientConfiguration Select EnableMultichannel PS Get SmbClientNetworkInterface On the SMB server run the following PowerShell cmdlets PS Get SmbServerConfiguration Select EnableMultichannel PS Get SmbServerNetworkInterface PS netstat exe xan 5 match 445 3 2 1 1 3 Verifying SMB Connection gt To verify the SMB connection on the SMB client Step1 Copy the large file to create a new session with the SMB Server Step 2 Open a PowerShell window while the copy is ongoing Step 3 Verify the SMB Direct is working properly and that the correct SMB dialect is used PS Get SmbConnection PS Get SmbMultichannelConnection PS netstat exe xan 5 match 445 P If you have no activity while you run the commands above you might get an empty list due to session expiration and absence current connections ae 3
43. Task Manager gt File gt Run new task gt and enter CMD Step 2 Display the MAC address as Physical Address gt ipconfig all Configuring a static IP is the same for Ethernet adapters gt To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices T lt Network and Inter Network Connections Organize Name Device Name Ethernet 11 49 Ethernet 12 Ethemet 13 Ethernet 15 Ethernet 9 Ethernet 3 8 Local Area Connection 9 work Broadcom NetXtreme Gigabit Et work cable unplugged Broadcom NetXtreme Gigabit Et etwork cable unplugged Broadcom NetXtreme Gigabit Et ork cable unplugged roade Unidentified network om NetXtreme Gigabit Et Mellanox ConnectX 4 VPI Adapt Network cable unplugged Mellanox ConnectX 4 VPI Adapt lt 7 items Step 2 Right click a Mellanox Local Area Connection and left click Properties Networking Sharing Connect using EP Mellanox ConnectX 4 VPI Adapter 4115 6 This connection uses the following items Client for Microsoft Networks File and Printer Sharing for Microsoft Networks 23005 Packet Scheduler 4 NDISTest600F041 Prot instance 4 NDISAPITest f041Prot instance NDISTest6003d41Prot instance NDISAPITest03d411Prot instance III
44. Tttcp is a Windows base testing application that sends and receives TCP data between two or more endpoints It is a Winsock based port of the ttcp tool that measures networking performance bytes second To download the latest version of NTttcp 5 28 please refer to Micro soft website following the link below http gallery technet microsoft com NTttcp Version 528 Now f8b12769 NOTE This tool should be run from cmd only The following InfiniBand performance tests are deprecated and might be removed in future releases Mellanox Technologies 74 J Rev 1 21 42 Management Utilities The management utilities described in this chapter are intended to be used to manage devices and providers Table 10 Management Utilities Utility Description ndinstall This utility is used to remove install and list NetworkDirect providers 43 Diagnostic Utilities The diagnostic utilities described in this chapter provide means for debugging the connectivity and status of devices in a fabric Table 11 Diagnostic Utilities Utility Description MiIxStat This utility displays information of Mellanox NIC attributes It is the equivalent utility to ibstat and vstat utilities in WinOF Mellanox Technologies 75 J Rev 1 21 5 Troubleshooting You may be able to easily resolve the issues described in this section If a problem persists and you are unable to resolve it please cont
45. act your Mellanox representative or Mellanox Support at support mellanox com 5 1 Installation Related Troubleshooting Table 12 Installation Related Issues Issue Cause Solution The installation of An incorrect driver version Use the correct driver package accord WinOF 2 fails with the might have been installed ing to the CPU architecture following error mes e g you are trying to install sage a 64 bit driver on a 32 bit This installation machine or vice versa package is not sup ported by this pro cessor type Contact your product ven dor 5 1 1 Installation Error Codes and Troubleshooting 5 1 1 1 Setup Return Codes Table 13 Setup Return Codes Error Code Description Troubleshooting 1603 Fatal error during installation Contact support 1633 The installation package is not supported Make sure you are installing the on this platform right package for your platform For additional details on Windows installer return codes please refer to http support microsoft com kb 229683 5 1 1 2 Firmware Burning Warning Codes Table 14 Firmware Burning Warning Codes Error Code Description Troubleshooting 1004 Failed to open the device Contact support 1005 Could not find an image for at The firmware for your device was not least one device found Please try to manually burn the firmware 1006 Found one device that has multiple Burn the fir
46. ader TCP TCP user data PA Mellanox Technologies 39 Rev 1 21 3 3 2 3 Enabling Disabling NVGRE Offloading To leverage NVGRE to virtualize heavy network IO workloads the Mellanox ConnectX 4 net work NIC provides hardware support for GRE off load within the network NICs by default gt To enable disable NVGRE off loading Step 1 Open the Device Manager Step 2 Go to the Network adapters Step 3 Right click Properties on Mellanox ConnectX 4 Ethernet Adapter card Step 4 Go to Advanced tab Step 5 Choose the Encapsulate Task Offload option Step 6 Set one of the following values Enable GRE off loading is Enabled by default Disabled When disabled the Hyper V host will still be able to transfer NVGRE traffic but TCP and inner IP checksums will be calculated by software that significant reduces performance 3 3 2 3 1 Configuring the NVGRE using PowerShell Hyper V Network Virtualization policies can be centrally configured using PowerShell 3 0 and PowerShell Remoting Step 1 Windows Server 2012 Only Enable the Windows Network Virtualization binding on the physical NIC of each Hyper V Host Host 1 and Host 2 PS Enable NetAdapterBinding EthInterfaceName a ComponentID ms_netwnv lt EthInterfaceName gt Physical NIC name Step 2 Create a vSwitch PS New VMSwitch vSwitchName NetAdapterName EthInterfaceName AllowManagementOS Strue Step 3 Shut down the VMs PS Stop VM
47. ance Tuning part of the installation A TepWindowSize registry value might have been added Remove the value key under HKEY LOCAL MACHINE SYSTEM Cur rentControlSet Ser vices Tcpip Parameters TcpWind owSize Or Set its value to 0xFFFF Packets are being lost The port MTU might have been set to a value higher than the maximum MTU supported by the switch Change the MTU according to the maxi mum MTU supported by the switch NVGRE changes done on a running VM are not propagated to the VM The configuration changes might not have taken effect until the OS is restarted Stop the VM and afterwards perform any NVGRE configuration changes on the VM connected to the virtual switch Mellanox Technologies 78 J Rev 1 21 5 3 Performance Related Troubleshooting Table 17 Performance Related Issues Issue Cause Solution Low performance issues The OS profile might not be 1 Go to Power Options in the Con configured for maximun trol Panel Make sure Maximum performace Performance is set as the power scheme 2 Reboot the machine 5 3 1 General Diagnostic Issue 1 Go to Device Manager locate the Mellanox adapter that you are debugging right click and choose Properties and go to the Information tab PCI Gen I should appear as PCI E 2 5 GT s PCI Gen 2 should appear as PCI E 5 0 GT s PCI Gen 3 should appear as PCI E 8 0
48. andled faster Interrupt Method Optimizes the CPU as it uses interrupts for handling incoming messages However in certain scenarios it can decrease the network throughput Adaptive Default Settings A combination of the interrupt and polling methods dynamically depending on traffic type and network usage Choosing a different setting may improve network and or system performance in certain configu rations Interrupt Moderation RX Packet Count Number of packets that need to be received before an interrupt is generated on the receive side default 5 Interrupt Moderation RX Packet Time Maximum elapsed time in usec between the receiving of a packet and the generation of an inter rupt even if the moderation count has not been reached default 10 Rx Interrupt Moderation Type Sets the rate at which the controller moderates or delays the generation of interrupts making it pos sible to optimize network throughput and CPU utilization The default setting Adaptive adjusts the interrupt rates dynamically depending on the traffic type and network usage Choosing a differ ent setting may improve network and system performance in certain configurations Send completion method Mellanox Technologies 70 J Rev 1 21 Sets the completion methods of the Send packets and it may affect network throughput and CPU utilization Interrupt Moderation TX Packet Count Number of packets that need to be sent before
49. d Packages on page 10 for details Mellanox Technologies 21 Rev 1 21 2 6 2 Unattended Uninstallation If no reboot options are specified the installer restarts the computer whenever necessary without displaying any prompt or warning to the user Use the norestart or forcerestart standard command line options to control reboots hax Touninstall MLNX WinOF2 in unattended mode Step 1 Open a CMD console Windows Server 2012 R2 Click Start gt Task Manager gt File gt Run new task and enter CMD Step 2 Uninstall the driver Run gt MLNX WinOF2 1 21 All win2012 x64 exe S x v qn 2 7 Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version the firmware will be automatically updated as part of the WinOF 2 package installation For information on how to upgrade firmware manually please refer to MFT User Manual www mellanox com gt Products gt InfiniBand VPI Drivers gt Firmware Tools Mellanox Technologies 22 J Rev 1 21 3 Features Overview and Configuration Once you have installed Mellanox WinOF 2 package you can perform various modifications to your driver to make it suitable for your system s needs Changes made to the Windows registry happen immediately and no backup is automati Hm cally made Do not edit the Windows registry unless you are confident regarding the changes 3 1 Ethernet Network 3 1 1 Mode Configuration WinOF 2 R
50. d left click Settings in the pop up menu Step 3 In the Settings window under the relevant network adapter select Hardware Accelera tion Step 4 Check uncheck the box Enable virtual machine queue to enable disable VMQ on that spe cific network adapter gt To enable Hyper V with VMQ using PowerShell Step 1 Enable VMQ on a specific VM Set VMNetworkAdapter VM Name VmqWeight 100 Step2 Disable on a specific VM Set VMNetworkAdapter VM Name Vmq Weight 0 3 3 2 Network Virtualization using Generic Routing Encapsulation NVGRE Network Virtualization using Generic Routing Encapsulation NVGRE off load is currently sup ported in Windows Server 2012 R2 with the latest updates for Microsoft 3 3 2 1 System Requirements Operating Systems Windows Server 2012 R2 3 3 2 2 Using NVGRE Network Virtualization using Generic Routing Encapsulation NVGRE is a network virtualiza tion technology that attempts to alleviate the scalability problems associated with large cloud computing deployments It uses Generic Routing Encapsulation GRE to tunnel layer 2 packets Mellanox Technologies 38 J Rev 1 21 across an IP fabric and uses 24 bits of GRE key as a logical network discriminator which is called a tenant network ID Configuring the Hyper V Network Virtualization requires two types of IP addresses Provider Addresses PA unique IP addresses assigned to each Hyper V host that are routabl
51. d receiving a packet with VLAN and QoS tag Mellanox Technologies 65 J Rev 1 21 Value Name Default Value Description PromiscuousVlan 0 Specifies whether a promiscuous VLAN is enabled or not When this parameter is set all the packets with VLAN tags are passed to an upper level without executing any filtering The valid values are 0 disable 1 enable Note This registry value is not exposed via the UI 3 4 5 1 Flow Control Options This group of registry keys allows the administrator to control the TCP IP traffic by pausing frame transmitting and or receiving operations By enabling the Flow Control mechanism the adapters can overcome any TCP IP issues and eliminate the risk of data loss Value Name Default Value Description FlowControl 0 When Rx Pause is enabled the receiving adapter generates a flow control frame when its received queue reaches a pre defined limit The flow control frame is sent to the sending adapter When TX Pause is enabled the sending adapter pauses the transmission if it receives a flow control frame from a link partner The valid values are 0 Flow control is disabled 1 Tx Flow control is Enabled 2 Rx Flow control is enabled 3 Rx amp Tx Flow control is enabled 3 4 5 2 VMQ Options This section describes the registry keys that are used to control the NDIS Virtual Machine Queue VMQ VMQ is supported by Win
52. d to measure performance in user envi ronment Documentation contains the User Manual and Release Notes Management tools installation tools used for management such as mlxstat Custom Setup Select the program features you want installed Click on an icon in the list below to change how a feature is installed 1E Performace Tools Feature Description configurations Network Direct InfiniBand This feature requires 496KB on your hard drive Install to C Program Files Mellanox MLNX_WinOF2 Performance tools InstallShield b Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Mellanox Technologies 16 Rev 1 21 Step 9 Click Finish to complete the installation MLNX WinOF2 InstallShield Wizard InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX WinOF2 Click Finish to exit the wizard Mellanox Technologies 17 Rev 1 21 2 3 2 Unattended Installation If no reboot options are specified the installer restarts the computer whenever necessary without displaying any prompt or warning to the user Use the norestart or forcerestart standard comma
53. e The valid values are MTU 1024 up to 64000 Note This registry key 1s not exposed to the user via the UI If LSOSize is smaller than MTU 1024 LSO will be dis abled LSOMinSegment 2 The minimum number of segments that a large TCP packet must be divisible by before the transport can offload it to a NIC for segmentation The valid values are 2 up to 32 Note This registry key is not exposed to the user via the UI LSOTcpOptions 1 Enables that the miniport driver to segment a large TCP packet whose TCP header contains TCP options The valid values are 0 disable 1 enable Note This registry key is not exposed to the user via the UI Mellanox Technologies 60 J Rev 1 21 Value Name Default Value Description LSOIpOptions 1 Enables its NIC to segment a large TCP packet whose IP header contains IP options The valid values are 0 disable 1 enable Note This registry key is not exposed to the user via the UI PChecksumOff 3 Specifies whether the device performs the calculation of IPv4 loadIPv4 checksums The valid values are 0 disable 1 Tx Enable 2 Rx Enable 3 Tx and Rx enable TCPUDPChecksu 3 Specifies whether the device performs the calculation of TCP mOffloadIPv4 or UDP checksum over IPv4 The valid values are 0 disable 1 Tx Enable 2 Rx Enable 3 Tx and Rx enable TCPUDPChecksu 3 Specif
54. e Flow Control Configuration Device manager gt Network adapters gt Mellanox ConnectX 4 Ethernet Adapter gt Properties gt Advanced tab 4 Device Manager ce es File Action View Help 5 a NIS 4 gd I dev w072 p jS Computer b Ca Disk drives p 8 Display adapters Details Events _ Resources I _ Power Management General Advanced b ia Human Interface Devices b 9 IEEE 1394 host controllers b Z2 Keyboards b A Mice and other pointing devices Ki Monitors 4 Network adapters Intel R 1350 Gigabit Network Connection EP Intel R 1350 Gigabit Network Connection 2 La Intel R 1350 Gigabit Network Connection 3 EP Intel R 1350 Gigabit Network Connection 4 KP Mellanox ConnectX 4 VPI Adapter MT4115 3 Mellanox ConnectX 4 VPI Adapter MT4115 4 Microsoft Kernel Debug Network Adapter 5 Other devices PCI Memory Controller Information Performance Driver The following properties are available for this network adapter Click the property you want to change on the left and then select its value on the right Property Value Interrupt Moderation IPV4 Checksum Offload Jumbo Packet Large Send Offload V2 IPv4 Large Send Offload V2 IPv6 Maximum number of RSS Processc Maximum Number of RSS Queues Network Address NetworkDirect Functionality Preferred NUMA Node Priority amp Vian Tag p Ports COM amp LPT p Print queues Processors b Storage co
55. e across the physical network infrastructure Each Hyper V host requires at least one PA to be assigned Customer Addresses CA unique IP addresses assigned to each Virtual Machine that participate on a virtualized network Using NVGRE multiple CAs for VMs running on a Hyper V host can be tunneled using a single PA on that Hyper V host CAs must be unique across all VMs on the same virtual network but they do not need to be unique across virtual networks with different Virtual Subnet ID The VM generates a packet with the addresses of the sender and the recipient within the CA space Then Hyper V host encapsulates the packet with the addresses of the sender and the recip ient in PA space PA addresses are determined by using Virtualization table Hyper V host retrieves the received packet identifies recipient and forwards the original packet with the CA addresses to the desired VM NVGRE can be implemented across an existing physical IP network without requiring changes to physical network switch architecture Since NVGRE tunnels terminate at each Hyper V host the hosts handle all encapsulation and de encapsulation of the network traffic Firewalls that block GRE tunnels between sites have to be configured to support forwarding GRE IP Protocol 47 tunnel traffic For further details on configuring NVGRE please refer to Appendix A NVGRE Configuration Scripts Examples on page 86 Figure 4 NVGRE Packet Structure Outer IP GRE He
56. ellanox network adapter under Network adapters list and left click Properties Select the Advanced tab from the Properties sheet Step 3 Modify configuration parameters to suit your system Please note the following For help on a specific parameter option check the help button at the bottom of the dialog Ifyou select one of the entries Off load Options Performance Options or Flow Control Options you ll need to click the Properties button to modify parameters via a pop up dialog 3 4 7 Receive Side Scaling RSS RSS settings can be set per individual adapters as well as globally gt To do so set the registry keys listed below For instructions on how to find interface index in registry lt nn gt please refer to Section 3 4 1 Finding the Index Value of the Network Interface on page 56 Table 7 Registry Keys Setting Sub key Description HKLM SYSTEM CurrentControlSet Con Maximum number of CPUs allotted trol Class 4d36e972 e325 11ce bfc1 Sets the desired maximum number of 08002be10318 lt nn gt MaxRSSProcessors processors for each interface The num ber can be different for each interface Note Restart the network adapter after you change this registry key Mellanox Technologies 35 J Rev 1 21 Table 7 Registry Keys Setting Sub key Description HKLM SYSTEM CurrentControlSet Con Base CPU number Sets the desired trol Class 4d3 6e972 e325 11ce bfc1 base CPU number fo
57. er X Reset on device Mellanox ConnectX 4 VPI Adapter X has finished Mellanox ConnectX 4 VPI Adapter X has got e vendor Y device id lt subvendor id lt F gt subsystem id lt L gt HWrevision lt M gt e FW version lt R gt lt G gt lt Q gt porttype lt N gt Mellanox ConnectX 4 VPI Adapter X QUERY CAP command fails with error Y The adapter card 15 dysfunctional Most likely a FW problem Please burn the last FW and restart the Mellanox ConnectX device Mellanox ConnectX 4 VPI Adapter X QUERY ADAPTER command fails with error Y The adapter card 15 dysfunctional Most likely a FW problem Please burn the last FW and restart the Mellanox ConnectX device Mellanox ConnectX 4 VPI Adapter X The number of allocated MSI X vectors is less than recommended This may decrease the network performance The number of requested MSI X vectors is Y while the number of allocated MSI X vectors is lt Z gt Mellanox ConnectX 4 VPI Adapter X FW command fails op 0x lt Y gt status 0x lt Z gt errno lt F gt syndrome 0x lt L gt Too many IPs in use for RRoCE Mellanox ConnectX 4 VPI Adapter X RRoCE supports only lt gt IPs per port Please reduce the number of IPs to use the new IPs Mellanox ConnectX 4 VPI Adapter X Driver startup fails because an insufficient num ber of Event Queues EQs is available lt Y gt are requ
58. ev 1 21 supports Ethernet mode only If the driver fails to start and a yellow sign appears near the Mellanox ConnectX 10Gb Ether net Adapter in the Device Manager display Code 10 and the event viewer shows that the driver has failed to start due to unsupported mode the correct configuration shoulf be performed For configuring the port types to Ethernet mode on a device use the mlxconfig exe utility which is part of the MFT package available at www mellanox com 1 Install the MFT package 2 Retrieve the device name a In command prompt run mst status v mst status v MST devices mt4099 pci cr0 bus dev fn 04 00 0 mt4099 pciconf0 bus dev fn 04 00 0 mt4103 pci cr0 bus dev fn 21 00 0 mt4103 pciconf0 bus dev fn 21 00 0 mt4115 pciconf0 bus dev fn 24 00 0 b Identify the desired device by its bus dev fn address 3 Execute the following command with the appropriate device name mlxconfig d mt4115 pciconf0 set LINK TYPE P1 2 LINK TYPE P2 2 4 Reboot system For further information please refer to the MFT User Manual 3 1 2 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine gt To obtain the MAC address Mellanox Technologies 23 J Rev 1 21 Step 1 Open CMD console Windows Server 2012 R2 Click Start gt
59. ew NetVirtualizationProviderAddress InterfaceIndex NIC InterfaceIndex ProviderAd dress 192 168 20 115 PrefixLength 24 New NetVirtualizationProviderRoute InterfaceIndex NIC InterfaceIndex Destination Prefix 0 0 0 0 0 NextHop 192 168 20 1 Step 5 Configure the Virtual Subnet ID on the Hyper V Network Switch Ports for each Virtual Machine on each Hyper V Host Host 1 and Host 2 Run the command below for each VM on the host the VM is running on it i e the for mtlael4 005 mtlael4 006 on host 192 168 20 114 and for VMs mtlae15 005 mtlael5 006 on host 192 168 20 115 mtlael5 only Get VMNetworkAdapter VMName mtlael5 005 where 5 MacAddress eq 00155D730100 Set VMNetworkAdapter VirtualSubnetID 5001 Get VMNetworkAdapter VMName mtlael5 006 where MacAddress eq 00155D730101 Set VMNetworkAdapter VirtualSubnetID 5001 mn Mellanox Technologies 88 J Rev 1 21 Appendix B Windows MPI MS MPI B 1 Overview Message Passing Interface MPI is meant to provide virtual topology synchronization and com munication functionality between a set of processes With MPI you can run one process on several hosts Windows MPI run over the following protocols Sockets Ethernet Network Direct ND B 1 1 System Requirements nstall HPC Build 4 0 3906 0 Validate traffic ping betwee
60. face Devices b IEEE 1394 host controllers p Keyboards b n Mice and other pointing devices Monitors 4 amp Network adapters S Intel R 1350 Gigabit Network Connection Intel R 1350 Gigabit Network Connection 2 E Intel R 1350 Gigabit Network Connection 3 KP Intel R 1350 Gigabit Network Connection 4 Mellanox ConnectX 4 VPI Adapter MT4115 3 Mellanox ConnectX 4 VPI Adapter MT4115 4 Microsoft Kernel Debug Network Adapter Other devices fy PCI Memory Controller b Ports COM amp LPT p Print queues BB Processors b Storage controllers 4 5 088 System devices b Universal Serial Bus controllers Details Events Resources Power Management General Advanced Information I Performance I Driver The following properties are available for this network adapter Click the property you want to change on the left and then select its value on the right Value Rx amp Tx Enabled Y Large Send Offload V2 IPv4 Large Send Offload V2 IPv6 Maximum number of RSS Processc Maximum Number of RSS Queues Network Address Network Direct Functionality Prefered NUMA Node Priority amp Vian Tag 3 1 3 3 Configuring SwitchX Based Switch System gt To enable RoCE the SwitchX should be configured as follows Ports facing the host should be configured as access ports and either use global pause or Port Control Protocol PCP for pr
61. for RDMA Read with minimum latency and runs over Micro soft s NetworkDirect standard The level of customizing for the user is relatively high User may choose to run with a customized message size customized number of iterations or alternatively customized test duration time nd read lat runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation nd send bw This test is used for performance measuring of Send requests in Micro soft Windows Operating Systems nd send bw is performance ori ented for Send with maximum throughput and runs over Microsoft s NetworkDirect standard The level of customizing for the user is rela tively high User may choose to run with a customized message size customized number of iterations or alternatively customized test dura tion time nd send bw runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation nd send lat This test is used for performance measuring of Send requests in Micro soft Windows Operating Systems nd send lat is performance oriented for Send with minimum latency and runs over Microsoft s NetworkDi rect standard The level of customizing for the user is relatively high User may choose to run with a customized message size customized number of iterations or alternatively customized test duration time nd send lat runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation NTttcp N
62. for application developers Documentation Conventions Table 2 Documentation Conventions Description Convention Example File names file extension Directory names directory Commands and their parameters command param 1 mts3610 1 gt show hosts Required item lt gt Optional item Mutually exclusive parameters pl p2 p3 or pl p2 p3 Optional mutually exclusive parameters pl p2 p3 Variables for which users supply specific values Italic font enable Emphasized words Italic font These are emphasized words Note lt text gt This is a note Mellanox Technologies 7 Rev 1 21 Table 2 Documentation Conventions Description Convention Example Warning lt text gt May result in system T instability Common Abbreviations and Acronyms Table 3 Abbreviations and Acronyms Sheet 1 of 2 Abbreviation Acronym Whole Word Description B Capital B is used to indicate size in bytes or multiples of bytes e g IKB 1024 bytes and 1 1048576 bytes b Small b is used to indicate size in bits or multiples of bits e g Kb 1024 bits FW Firmware HCA Host Channel Adapter HW Hardware IB InfiniBand LSB Least significant byte Isb Least significant bit MSB Most significant
63. ies whether the device performs the calculation of TCP mOffloadIPv6 or UDP checksum over IPv6 The valid values are 0 disable 1 Tx Enable 2 Rx Enable 3 Tx and Rx enable Mellanox Technologies 61 J Rev 1 21 3 4 4 Performance Registry Keys This group of registry keys configures parameters that can improve adapter performance Value Name Default Value Description RecvCompletionMethod 1 Sets the completion methods of the receive packets and it affects network throughput and CPU utilization The supported methods are Polling increases the CPU utilization because the sys tem polls the received rings for incoming packets how ever it may increase the network bandwidth since the incoming packet is handled faster Adaptive combines the interrupt and polling methods dynamically depending on traffic type and network usage The valid values are 0 polling 1 adaptive InterruptModeration 1 Sets the rate at which the controller moderates or delays the generation of interrupts making it possible to optimize network throughput and CPU utilization When disabled the interrupt moderation of the system generates an interrupt when the packet is received In this mode the CPU utilization is increased at higher data rates because the system must handle a larger number of interrupts However the latency is decreased since that packet is processed more quickly
64. iority flow control Ports facing the network should be configured as trunk ports and use Port Control Pro tocol PCP for priority flow control For further information on how to configure SwitchX please refer to SwitchX User Manual Mellanox Technologies 28 Rev 1 21 3 1 3 4 Configuring Arista Switch Step 1 Set the ports that face the hosts as trunk config interface et10 config if Et10 switchport mode trunk Step 2 Set VID allowed on trunk port to match the host VID config if Et10 switchport trunk allowed vlan 100 Step 3 Set the ports that face the network as trunk config interface et20 config if Et20 switchport mode trunk Step 4 Assign the relevant ports to LAG config interface et10 config if Et10 dcbx mode ieee config if Et10 speed forced 40gfull config if Et10 channel group 11 mode active Step 5 Enable PFC on ports that face the network config interface et20 config if Et20 load interval 5 config if Et20 speed forced 40gfull config if Et20 switchport trunk native vlan tag config if Et20 switchport trunk allowed vlan 11 config if Et20 switchport mode trunk config if Et20 dcbx mode ieee config if Et20 priority flow control mode on config if Et20 priority flow control priority 3 no drop 3 1 3 4 1 Using Global Pause Flow Control gt To enable Global Pause on ports that face the hosts perform the following config i
65. ired lt Z gt are recommended lt M gt are available e Mellanox ConnectX 4 VPI Adapter X Execution of FW command fails op 0x lt Y gt errno lt Z gt Mellanox Technologies 82 J Rev 1 21 e Mellanox ConnectX 4 VPI Adapter X Driver startup has failed due to unsupported port type lt Y gt configured on the device The driver supports Ethernet mode only please refer to the Mellanox WinOF 2 User Man ual for instructions on how to configure the correct mode e Mellanox ConnectX 4 VPI Adapter X Driver startup fails because minimal driver requirements are not supported by FW lt Y gt lt Z gt lt L gt FW reported e rss ind tbl cap lt Q gt vlan cap lt M gt max rqs lt gt max sqs lt N gt max tirs lt gt Please burn a firmware that supports the requirements and restart the Mellanox ConnectX device For additional information please refer to Support information on http mellanox com e Mellanox ConnectX 4 VPI Adapter X Driver startup fails because maximum flow table size that is supported by FW lt Y gt lt Z gt lt L gt is too small lt K gt entries Please burn a firmware that supports a greater flow table size and restart the Mellanox ConnectX device For additional information please refer to Support information on http mellanox com e Mellanox ConnectX 4 VPI Adapter X Driver startup fails because required receive size is greater than the maximum WQEs si
66. lid values are 0 up to 256 Note This registry value is not exposed via the UI AveragePollThreshold 10 The average threshold polling number when using polling completion method for receiving If the aver age number 15 higher than this value the adapter con tinues to poll The valid values are 0 up to 1000 Note This registry value is not exposed via the UI ThisPollThreshold 100 The threshold number of the last polling cycle when using polling completion method for receiving If the number of packets received in the last polling cycle is higher than this value the adapter continues to poll The valid values are 0 up to 1000 Note This registry value is not exposed via the UI 0 Enables packets with It is used when no team intermediate driver is used The valid values are 0 disable No Vlan Id is passed 1 4095 Valid Vlan Id that will be passed Note This registry value is only valid for Ethernet NumRSSQueues 8 The maximum number of the RSS queues that the device should use Note This registry key is only in Windows Server 2012 and above Mellanox Technologies 64 J Rev 1 21 Value Name Default Value Description BlueFlame 1 The latency critical Send WQEs to the device When BlueFlame is used the WQEs are written directly to the PCI BAR of the device in addition to memory so that the device may handle them without having to access memory
67. lution Protocol Quality Windows Audio Video Experience C Remote Assistance Remote Differential Compression Remote Server Administration T Feature Administration Tools Role AC stration ols AD DS and AD LDS Tools per V Management Tools Installed Remote Desktop Services Tools Windows Server Update Services Tools lt Previous Mellanox Technologies 46 RAS Connection Manager Administration Kit CMA _ DESTINATION SERVER I dev w072 Description Hyper V Management Tools includes GUI and command line tools for managing Hyper V Rev 1 21 Step 4 Confirm the installation DESTINATION SERVER Confirm installation selections Before You Begin To install the following roles role services or features on selected server click Install Installation Typ Restart the destination server automatically if required Optional features such as administration tools might be displayed on this page because they have been selected automatically If you do not want to install these optional features click Previous to clear their check boxes Hyper V Remote Server Administration Tools K Role Administration Tools Mi ton Hyper V Management Tools Defau 5 Hyper V GUI Management Tools Hyper V Module for Windows PowerShell Export configuration settings Specify an alternate source path
68. memory The valid values are 256 up to 4096 SpeedDuplex 7 The Speed and Duplex settings that a device supports This registry key should not be changed and it can be used to query the device capability Mellanox ConnectX device is set to 7 meaning 10Gbps and Full Duplex Note Default value should not be modified Mellanox Technologies 58 J Rev 1 21 Value Name Default Value Description RxIntModerationPro 2 Enables the assignment of different interrupt moderation pro file files for receive completions Interrupt moderation can have a great effect on optimizing network throughput and CPU utilization The valid values are 0 Low Latency Implies higher rate of interrupts to achieve better latency or to handle scenarios where only a small number of streams are used 1 Moderate Interrupt moderation is set to midrange defaults to allow maxi mum throughput at minimum CPU utilization for common sce narios 2 Aggressive Interrupt moderation is set to maximal values to allow maxi mum throughput at minimum CPU utilization for more inten sive multi stream scenarios TxIntModerationPro 1 Enables the assignment of different interrupt moderation pro file files for send completions Interrupt moderation can have great effect on optimizing network throughput and CPU utili zation The valid values are 0 Low Latency Implies higher rate of interrupts to achieve better latency o
69. mware manually and select images the image you want to burn 1007 Found one device for which force Burn the firmware manually with the update is required force flag Mellanox Technologies 76 J Rev 1 21 Table 14 Firmware Burning Warning Codes Error Code Description Troubleshooting 1008 Found one device that has mixed The firmware version or the expansion versions rom version does not match For additional details please refer to the MFT User Manual http www mellanox com gt Products gt Firmware Tools 5 1 1 3 Restore Configuration Warnings Table 15 Restore Configuration Warnings Error Code Description Troubleshooting 3 Failed to restore the configu Please see log for more details and contact the ration support team 5 2 Ethernet Related Troubleshooting For further performance related information please refer to the Performance Tuning Guide and to Section 3 5 Performance Tuning and Counters on page 68 Table 16 Ethernet Related Issues Issue Cause Solution Low performance Non optimal system con See section Performance Tuning and figuration might have Counters on page 68 to take advantage occurred of Mellanox 10 40 56 GBit NIC perfor mance The driver fails to start There might have been an 1 Open the event log and look under RSS configuration mis System for the mlx5 source match between the TCP 2 If found enable RSS run
70. n the whole MPI Hosts Every MPI client need to run smpd process which open the mpi channel MPI Initiator Server need to run mpiexec If the initiator is also client it should also run smpd B 2 Running MPI Step 1 Run the following command on each mpi client start smpd d p lt port gt Step2 Install ND provider on each MPI client in MPI ND Step 3 Run the following command on MPI server mpiexec exe p smpd port hosts num of hosts hosts ip list env MPICH NETMASK network ip subnet env MPICH ND ZCOPY THRESHOLD 1 env MPICH DISABLE ND 0 1 env MPICH DISABLE SOCK 0 1 affinity process B 3 Directing MSMPI Traffic Directing MPI traffic to a specific QoS priority may delayed due to Except for NetDirectPortMatchCondition the QoS powershell CmdLet for NetworkDi rect traffic does not support port range Therefore NetwrokDirect traffic cannot be directed to ports 1 65536 The MSMPI directive to control the port range namely MPICH PORT RANGE 3000 3030 15 not working for ND and MSMPI chose a random port B 4 Running MSMPI on the Desired Priority Step 1 Set the default QoS policy to be the desired priority Note this prio should be lossless all the way in the switches Step2 Set SMB policy to a desired priority only if SMD Traffic running Mellanox Technologies 89 J Rev 1 21 Step3 Recommended Direct ALL TCP UDP traffic to a lossy priority by using the IPProtocol
71. nd line options to control reboots The following is an example of an unattended installation session Step 1 Open a CMD console Windows Server 2012 R2 Click Start gt Task Manager gt File gt Run new task gt and enter CMD Step 2 Install the driver Run gt MLNX WinOF2 1 21 All x64 exe S v qn Step 3 Optional Manually configure your setup to contain the logs option MLNX WinOF2 1 21 All x64 exe S v qn v 1 vx LogFile Step 4 Optional if you want to control whether to install ND provider or gt MLNX WinOF2 1 21 All win2012 x64 exe vMT NDPROPERTY 1 Step 5 Optional If you do not wish to upgrade your firmware version gt MLNX VPI WinOF 1 21 All win2012 x64 exe v MT SKIPFWUPGRD 1 4 Applications that hold the driver files such as ND applications will be closed during the unat tended installation A 1 MT NDPROPERTY default value is True 2 MT SKIPFWUPGRD default value is False Mellanox Technologies 18 J Rev 1 21 2 4 Installation Results Upon installation completion you can verify the successful addition of the network card s through the Device Manager Upon installation completion the inf files can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers lt OS gt To see the Mellanox network adapters display the Device Manager and pull down the Net work adapters menu Figure 1 Installation Results File Action View Help 9g 5
72. ng the mlxconfig too For older firmware versions use the flint tool gt To enable SR IOV using mlxconfig mlxconfig is part of MFT tools used to simplify firmware configuration The tool is available with MFT tools 3 6 0 or higher Step 1 Download MFT for Windows www mellanox com gt Products gt Software gt Firmware Tools Step 2 Get the device ID look for the pcicont string in the output mst status Mellanox Technologies 50 Rev 1 21 Example MST devices mt4115 pciconf0 Step 3 Check the current SR IOV configuration gt mlxconfig d mt4115 pciconf0 q Example Device 1 Device type ConnectX4 PCI device mt4115 pciconf0 Configurations Current SRIOV EN N A NUM OF VFS N A WOL MAGIC EN P2 N A LINK TYPE Pl N A LINK TYPE P2 N A Step 4 Enable SR IOV with 16 VFs gt mlxconfig d mt4115 pciconf0 s SRIOV EN 1 NUM OF VFS 16 Warning Care should be taken in increasing the number of VFs All servers A are guaranteed to support 16 VFs More VFs can lead to exceeding the BIOS limit of MMIO available address space Example Device 1 Device type ConnectX3Pro PCI device mt4115 pciconf0 Configurations Current New SRIOV EN N A 1 NUM_OF VFS N A 16 WOL MAGIC EN P2 N A N A LINK TYPE Pl N A N A LINK TYPE P2 N A N A Apply new Configuration y n n y Applying Done I Please reboot machine to load new configurations Step 5 Power cycle the machine After the reboot c
73. nly used in conjunction with an SR IOV enabled hypervisor to provide virtual machines direct hardware access to network resources hence increasing its performance This guide demonstrates the setup and configuration of SR IOV using Mellanox ConnectX VPI adapter cards family SR IOV VF is a single port device Mellanox device is a dual port single PCI function Virtual Functions pool belongs to both ports To define how the pool is divided between the two ports use the Powershell SriovPortiNum VFs command see Step 3 in Section 3 3 3 3 2 Enabling SR IOV in Mellanox WinOF 2 Pack age Ethernet SR IOV Only on page 52 SR IOV is disabled in Mellanox WinOF 2 package by default Please refer to Section 3 3 3 3 2 Enabling SR IOV in Mellanox WinOF 2 Package Ethernet SR IOV Only on page 52 for enabling it 3 3 3 1 SR IOV Ethernet over Hyper V 3 3 3 1 1System Requirements server and BIOS with SR IOV support BIOS settings might need to be updated to enable virtualization support and SR IOV support Hypervisor OS Windows Server 2012 R2 Virtual Machine VM OS The VM OS can be either Windows Server 2012 and above Mellanox ConnectX 4 VPI Adapter Card family with SR IOV capability Mellanox WinOF 2 1 20 or higher Firmware version 2 30 8000 or higher 3 3 3 1 2 Feature Limitations SR IOV is supported only in Ethernet ports RDMA i e RoCE capability is not available in SR IOV mode Mellanox
74. not persistent Its suggested to create script is running after each OS reboot Step 2 Configure a Subnet Locator and Route records on each Hyper V Host Host 1 Host 2 mtlael4 amp mtlael5 New NetVirtualizationLookupRecord CustomerAddress 172 16 14 5 ProviderAddress 192 168 20 114 VirtualSubnetID 5001 MACAddress 00155D720100 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 14 6 ProviderAddress 192 168 20 114 VirtualSubnetID 5001 MACAddress 00155D720101 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 15 5 ProviderAddress 192 168 20 115 VirtualSubnetID 5001 MACAddress 00155D730100 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 15 6 ProviderAddress 192 168 20 115 VirtualSubnetID 5001 MACAddress 00155D730101 Rule TranslationMetho dEncap Add customer route New NetVirtualizationCustomerRoute RoutingDomainID 11111111 2222 3333 4444 000000005001 VirtualSubnetID 5001 DestinationPrefix 172 16 0 0 16 NextHop 0 0 0 0 Metric 255 Mellanox Technologies 86 J Rev 1 21 Step 3 Configure the Provider Address and Route records Hyper V Host 1 Host 1 Only mtlael4 SNIC Get NetAdapter Port1 New NetVirtualizationProviderAddress InterfaceIndex NIC
75. nterface et10 config if Et10 flowcontrol receive on config if Et10 flowcontrol send on 3 1 3 4 2 Using Priority Flow Control PFC gt To enable Global Pause on ports that face the hosts perform the following config interface et10 config if Et10 dcbx mode ieee config if Et10 priority flow control mode on config if Et10 priority flow control priority 3 no drop 3 1 3 5 Configuring Router PFC only The router uses L3 s DSCP value to mark the egress traffic of L2 PCP The required mapping maps the three most significant bits of the DSCP into the PCP This is the default behavior and no additional configuration is required Mellanox Technologies 29 J Rev 1 21 3 1 3 5 1 Copying Port Control Protocol PCP between Subnets The captured PCP option from the Ethernet header of the incoming packet can be used to set the PCP bits on the outgoing Ethernet header 3 1 3 6 Configuring the RoCE Mode Configuring the RoCE mode requires the following RoCE mode is configured per driver and is enforced on all the devices in the system The supported RoCE modes depend on the firmware installed If the firmware does not support the needed mode the fallback mode would be the maximum supported RoCE ae mode of the installed NIC RoCE mode can be enabled and disabled either via the registry key or the PowerShell gt RoCE is enabled by default To enable it using the registry key Set the roce mode as follo
76. ntrollers b gli System devices p Universal Serial Bus controllers To install the Data Center Bridging using the Server Manager Step 1 Open the Server Manager Step 2 Select Add Roles and Features Step3 Click Next Step 4 Select Features on the left panel Step 5 Check the Data Center Bridging checkbox Step 6 Click Install To install the Data Center Bridging using PowerShell Step 1 Enable Data Center Bridging DCB PS Install WindowsFeature Data Center Bridging Mellanox Technologies 32 Rev 1 21 gt To configure QoS on the host The procedure below is not saved after you reboot your system Hence we recom mend you create a script using the steps below and run it on the startup of the local machine Please see the procedure below on how to add the script to the local machine startup scripts As Step 1 Change the Windows PowerShell execution policy PS Set ExecutionPolicy AllSigned Step 2 Remove the entire previous QoS configuration PS Remove NetQosTrafficClass PS Remove NetQosPolicy Confirm False Step 3 Set the DCBX Willing parameter to false as Mellanox drivers do not support this feature PS set NetQosDcbxSetting Willing 0 Step 4 Create a Quality of Service QoS policy and tag each type of traffic with the relevant prior ity In this example TCP UDP use priority 1 SMB over TCP use priority 3 PS New NetQosPolicy DEFAULT st
77. ontinue to Section 3 3 3 3 2 Enabling SR IOV in Mellanox WinOF 2 Package Ethernet SR IOV Only on page 52 Mellanox Technologies 51 Rev 1 21 3 3 3 3 2 Enabling SR IOV in Mellanox WinOF 2 Package Ethernet SR IOV Only gt To enable SR IOV in Mellanox WinOF 2 Package Step 1 Install Mellanox WinOF 2 package that supports SR IOV Step2 Set the Execution Policy PS Set ExecutionPolicy AllSigned Step 3 Enable SR IOV through Powershell New NetAdapterAdvancedProperty Name Network Adapter RegistryKeyword SRIOV Regis tryValue 1 RegistryDataType REG SZ 3 3 3 4 Configuring Operating Systems 3 3 3 4 1 Configuring Virtual Machine Networking Ethernet SR IOV Only gt To configure Virtual Machine networking Step 1 Create an SR IOV enabled Virtual Switch over Mellanox Ethernet Adapter Go to Start gt Server Manager gt Tools gt Hyper V Manager In the Hyper V Manager Actions gt Virtual SwitchManager gt External gt Create Virtual Switch Step 2 Set the following Name External network Enable single root I O virtualization SR IOV Mellanox Technologies 52 J Rev 1 21 Figure 9 Virtual Switch with SR IOV Virtual Switches YA New virtual network switch sa Virtual Switch Properties E Internal Virtual Switch Name Internal only Mellano
78. ore Activestore Default PriorityValue8021Action 3 PS New NetQosPolicy TCP store Activestore IPProtocolMatchCondition TCP Priority Value8021Action 1 PS New NetQosPolicy UDP store Activestore IPProtocolMatchCondition UDP Priority Value8021Action 1 New NetQosPolicy SMB SMB PriorityValue8021Action 3 Step 5 Create a QoS policy for SMB over SMB Direct traffic on Network Direct port 445 PS New NetQosPolicy SMBDirect store Activestore NetDirectPortMatchCondition 445 PriorityValue8021Action 3 Step 6 Optional If VLANs are used mark the egress traffic with the relevant VlanID The NIC is referred as Ethernet 4 in the examples below PS Set NetAdapterAdvancedProperty Name Ethernet 4 RegistryKeyword VlanID Reg istryValue 55 Step 7 Optional Configure the IP address for the NIC If DHCP is used the IP address will be assigned automatically PS Set NetIPInterface InterfaceAlias Ethernet 4 DHCP Disabled PS Remove NetIPAddress InterfaceAlias Ethernet 4 AddressFamily IPv4 Con firm Sfalse PS New NetIPAddress InterfaceAlias Ethernet 4 IPAddress 192 168 1 10 Prefix Length 24 Type Unicast Step 8 Optional Set the DNS server assuming its IP address 15 192 168 1 2 PS Set DnsClientServerAddress InterfaceAlias Ethernet 4 ServerAddresses MV 1 2 Mellanox Technologies 33 J Rev 1 21 After establishing the priorities of ND NDK traffic the priorities must ha
79. otected by copyright law and international treaties Step 4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location for the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX VPI at the specified network location or click Cancel to exit the wizard Network location mea Change InstallShield Mellanox Technologies 20 Rev 1 21 Step 5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX_VPI at the specified network location or click Cancel to exit the wizard Network location Change InstallShield Step 6 To complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPT Click Finish to exit the wizard 2 6 Uninstalling Mellanox WinOF 2 Driver 2 6 1 Attended Uninstallation gt To uninstall MLNX WinOF2 a single node Click Start gt Control Panel gt Programs and Features gt MLNX_WinOF2 gt Uninstall NOTE This requires elevated administrator privileges see Section 1 1 Supplie
80. ove the entire previous settings Remove NetQosTrafficClass Remove NetQosPolicy Confirm False Setthe DCBX Willing parameter to false as Mellanox drivers do not support this feature Set NetQosDcbxSetting Willing 0 Mellanox Technologies 90 J Rev 1 21 Create a Quality of Service QoS policy and tag each type of traffic with the relevant priority In this example we used TCP UDP priority 1 ND NDK priority 3 New NetQosPolicy SMB NetDirectPortMatchCondition 445 PriorityValue8021Action 3 New NetQosPolicy DEFAULT Default PriorityValue8021Action 3 New NetQosPolicy TCP IPProtocolMatchCondition TCP PriorityValue8021Actionl New NetQosPolicy UDP IPProtocolMatchCondition UDP PriorityValue8021Action 1 Enable PFC on priority 3 Enable NetQosFlowControl 3 Disable Priority Flow Control PFC for all other priorities except for 3 Disable NetQosFlowControl 0 1 2 4 5 6 7 Enable QoS on the relevant interface Enable netadapterqos Name B 5 2 Running MPI Command Examples Running MPI pallas test over ND gt mpiexec exe p 19020 hosts 4 11 11 146 101 11 21 147 101 Sh 11 11 145 101 env MPICH NETMASK 11 0 0 0 255 0 0 0 env MPICH ND ZCOPY THRESHOLD 1 env MPICH DISABLE ND 0 env MPICH DISABLE SOCK 1 affinity c testl exe Running MPI pallas test over ETH gt exempiexec exe p 19020 hosts 4 11 11 146 101 11 21 147 101 S 11 11 145 101 MPICH NET
81. ox WinOF 2 web page at http www mellanox com gt Products gt InfiniBand VPI Drivers gt Windows SW Drivers Step 3 Download the exe image according to the architecture of your machine see Step 1 The name of the exe is in the following format MLNX_WinOF2 lt version gt _ lt arch gt exe Installing the incorrect exe file is prohibited If you do so an error message will be dis played For example if you try to install a 64 bit exe on a 32 bit machine the wizard will display the following or a similar error message As Windows Installer x This installation package is not supported by this processor type Contact your product vendor OK Mellanox Technologies 12 Rev 1 21 23 Installing Mellanox WinOF 2 Driver WinOF 2 supports adapter cards based on the Mellanox ConnectX 4 family of adapter IC devices only If you have ConnectX 3 and ConnectX 3 Pro on your server you will need to install WinOF driver For details on how to install WinOF driver please refer to WinOF User Manual hall This section provides instructions for two types of installation procedures Attended Installation An installation procedure that requires frequent user intervention Unattended Installation An automated installation procedure that requires no user intervention Both Attended and Unattended installations require administrator privileges 2 3 1 Attended Installation The following i
82. pplication memory without any CPU involvement RDMA over Converged Ethernet RoCE is a mechanism to provide this efficient data transfer with very low latencies on loss less Ethernet networks With advances in data center convergence over reliable Ethernet ConnectX EN with RoCE uses the proven and efficient transport to provide the platform for deploying technology in mainstream data center application at 10GigE and 40GigE link speed ConnectX EN with its hardware offload support takes advantage of this efficient RDMA transport InfiniBand services over Ethernet to deliver ultra low latency for performance critical and transaction intensive applications such as financial database storage and content delivery networks RoCE encapsulates IB transport and GRH headers in Ethernet packets bearing a dedicated ether type While the use of GRH is optional within InfiniBand subnets it is mandatory when using RoCE Applications written over IB verbs should work seamlessly but they require provisioning of GRH information when creat Mellanox Technologies 25 Rev 1 21 ing address vectors The library and driver modified to provide mapping from GID to MAC addresses required by the hardware 3 1 3 1 IP Routable RoCEv2 RoCE has two addressing modes MAC based GIDs and IP address based GIDs In RoCE IP based if the IP address changes while the system is running the GID for the port will automati cally be updated with
83. pters Step2 Right click gt Properties on Mellanox Connect X Ethernet Adapter Step 3 to Details tab Step 4 Select the Driver key and obtain the nn number Mellanox Technologies 56 Rev 1 21 In the below example the index equals 0010 Device Manager File Action View Hep 9 m S B m S Fas b M Computer Events Resources PowerM b Disk drives 5 Driver p e Display adapters b Cg IDE ATA ATAPI controllers Mellanox ConnectX 4 VPI Adapter 3 4 IEEE 1394 host controllers Texas Instruments 1394 OHCI Compliant Host Cont Bus p amp Keyboards b A Mice and other pointing devices Driver key b Ki Monitors Value 4 Network adapters AP Broadcom NetXtreme Gigabit Ethernet 44435 972 325 11 1 080025 1031880018 Broadcom NetXtreme Gigabit Ethernet 2 AP Broadcom NetXtreme Gigabit Ethernet 3 EP Broadcom NetXtreme Gigabit Ethernet 4 Mellanox ConnectX 4 VPI Adapter 3 Mellanox ConnectX 4 VPI Adapter 24 js Other devices Base System Device b YP Ports COM amp LPT b Print queues b B Processors b Storage controllers Ml System devices ACPI Fixed Feature Button Mellanox Technologies 57 Rev 1 21 3 4 2 Basic Registry Keys This group contains the registry keys that control the basic operations of the NIC Value Name Default Value Description JumboPacke
84. r each interface 08002be10318 lt nn gt RssBaseProcNumber The number can be different for each interface This allows partitioning of CPUs across network adapters Note Restart the network adapter when you change this registry key HKLM SYSTEM CurrentControlSet Con NUMA node affinitization trol Class 4d36e972 e325 11ce bfc1 08002be10318 lt nn gt NumaNodeID HKLM SYSTEM CurrentControlSet Con Sets the RSS base processor group for trol Class 4d3 6e972 e325 11ce bfc1 systems with more than 64 processors 08002be10318 lt nn gt RssBaseProcGroup 3 2 Storage Protocols 3 2 1 Deploying SMB Direct The Server Message Block SMB protocol is a network file sharing protocol implemented in Microsoft Windows The set of message packets that defines a particular version of the protocol is called a dialect The Microsoft SMB protocol is a client server implementation and consists of a set of data pack ets each containing a request sent by the client or a response sent by the server SMB protocol is used on top of the TCP IP protocol or other network protocols Using the SMB protocol allows applications to access files or other resources on a remote server to read create and update them In addition it enables communication with any server program that is set up to receive an SMB client request 3 2 1 1 SMB Configuration Verification 3 2 1 1 1 Verifying Network Adapter Configuration Use the following Power
85. r to handle scenarios where only a small number of streams are used 1 Moderate Interrupt moderation is set to midrange defaults to allow maxi mum throughput at minimum CPU utilization for common sce narios 2 Aggressive Interrupt moderation is set to maximal values to allow maxi mum throughput at minimum CPU utilization for more inten sive multi stream scenarios Mellanox Technologies 59 J Rev 1 21 3 4 3 Off load Registry Keys This group of registry keys allows the administrator to specify which TCP IP offload settings are handled by the adapter rather than by the operating system Enabling offloading services increases transmission performance Due to offload tasks such as checksum calculations performed by adapter hardware rather than by the operating system and therefore with lower latency In addition CPU resources become more available for other tasks Value Name Default Value Description LsoV1IPv4 1 Large Send Offload Version 1 IPv4 The valid values are 0 disable 1 enable LsoV2IPv4 1 Large Send Offload Version 2 IPv4 The valid values are 0 disable 1 enable LsoV2IPv6 1 Large Send Offload Version 2 The valid values are 0 disable 1 enable LSOSize 64000 The maximum number of bytes that the TCP IP stack can pass to an adapter in a single packet This value affects the memory consumption and the NIC per formanc
86. rivers only InfiniBand drivers are not supported yet It supports 10 25 40 50 or 100 Gb s Ethernet net work ports The port speed is determined upon boot based on card capabilities and user settings The Mellanox WinOF 2 driver release introduces the following capabilities Support for ConnectX 4 and ConnectX 4 Lx single and dual port adapter cards Up to 16 Rx queues per port Dedicated PCI function per physical port Rxsteering mode RSS Hardware Tx Rx checksum calculation Large Send off load 1 e TCP Segmentation Off load Receive Side Coalescing RSC or LRO in Linux Hardware multicast filtering Adaptive interrupt moderation Support for MSI X interrupts NDK with SMB Direct NDvl and v2 API support in user space for Hypervisor Hardware VLAN filtering over Converged Ethernet RoCE vl RoCE v2 NVGRE hardware off load in ConnectX 4Enhanced Transmission Selection ETS SR IOV Ethernet on Windows Server 2012 R2 Hypervisor with Windows Server 2012 and above guests Quality of Service QoS Priority Flow Control PFC 1 1 Supplied Packages Mellanox WinOF 2 driver Rev 1 21 includes the following package e MLNX WinOF2 1 21 All x64 exe 1 2 WinOF 2 Set of Documentation Under installation directory Documentation 1 WinOF 2 does not support earlier Mellanox adapters For earlier adapters the Windows driver is MLNX WinOF Mellanox Technologies 10 J
87. rt You need to either enable Windows TCP RSS or configure the adapter s port to disable RSS For further details see the README file under the documentation folder e Mellanox ConnectX 4 VPI Adapter X Jumbo packet value read from registry lt Y gt is greater than the value supported by FW lt 7 gt Therefore use the maximum value sup ported by FW q Mellanox ConnectX 4 VPI Adapter X Maximum MTU supported by FW lt L gt lt Y gt lt Z gt lt q gt is smaller than the minimum value lt K gt Mellanox ConnectX 4 VPI Adapter lt X gt device is successfully stopped Mellanox ConnectX 4 VPI Adapter X device startup fails due to less than minimum MSI X vectors available Mellanox ConnectX 4 VPI Adapter lt X gt device detects that the link is up and has initi ated a normal operation Mellanox ConnectX 4 VPI Adapter lt X gt device detects that the link is down This may occur if the physical link is disconnected or damaged or if the other end port is down Mellanox ConnectX 4 VPI Adapter lt X gt Mellanox ConnectX 4 VPI Adapter X device configures not to use RSS This configuration may significantly affect the network perfor mance e Mellanox ConnectX 4 VPI Adapter X device reports an Error event on CQn lt Y gt Since the event type is lt Z gt the NIC will be reset The issue is reported in Function lt K gt Mellanox ConnectX 4 VPI Adapter X device reports a send lt Y gt
88. rupt rather than a series of interrupts An interrupt is generated after receiving 5 packets or after 10ms from the first packet received It improves performance and reduces CPU load however it increases latency When the interrupt moderation is disabled the system generates an interrupt each time a packet is received or sent In this mode the CPU utilization data rates increase as the system handles a larger number of interrupts However the latency decreases as the packet is handled faster Receive Side Scaling RSS Mode Improves incoming packet processing performance RSS enables the adapter port to utilize the multiple CPUs in a multi core system for receiving incoming packets and steering them to the des ignated destination RSS can significantly improve the number of transactions the number of con nections per second and the network throughput This parameter can be set to one of the following values Enabled default Set RSS Mode Disabled The hardware is configured once to use the Toeplitz hash function and the indirection table is never changed IOAT is not used while in RSS mode IN Receive Completion Method Sets the completion methods of the received packets and can affect network throughput and CPU utili zation Polling Method Increases the CPU utilization as the system polls the received rings for the incoming packets However it may increase the network performance as the incoming packet is h
89. s an example of an installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX WinOF2 Step 2 Optional Manually configure your setup to contain the logs option gt MLNX WinOF2 1 21 All x64 exe v l vx LogFile Step 3 Optional If you do not want to upgrade your firmware version MLNX VPI WinOF 1 21 All win2012 x64 exe v MT SKIPFWUPGRD 1 Step 4 Click Next in the Welcome screen Welcome to the InstallShield Wizard for MLNX WinOF2 The InstallShield R Wizard will install MLNX_WinOF2 on your computer To continue dick Next WARNING This program is protected by copyright law and international treaties 1 MT SKIPFWUPGRD default value is False Mellanox Technologies 13 Rev 1 21 Step 5 Read then accept the license agreement and click Next Please read the following license agreement carefully Copyright c 2005 2015 Mellanox Technologies All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer Mads Ml em nnna Sw hinas Fawr mmm mannadnan tha nahara I accept the terms in the license agreement do not accept the terms in the license agreement v InstallShield lt we ome Step 6 Select
90. t 1514 The maximum size of a frame or a packet that can be sent over the wire This is also known as the maximum transmis sion unit MTU The MTU may have a significant impact on the network s performance as a large packet can cause high latency However it can also reduce the CPU utilization and improve the wire efficiency The standard Ethernet frame size is 1514 bytes but Mellanox drivers support wide range of packet sizes The valid values are Ethernet 600 up to 9600 Note All the devices across the network switches and rout ers should support the same frame size Be aware that differ ent network devices calculate the frame size differently Some devices include the header i e information in the frame size while others do not Mellanox adapters do not include Ethernet header informa tion in the frame size i e when setting JumboPacket to 1500 the actual frame size is 1514 ReceiveBuffers 512 The number of packets each ring receives This parameter affects the memory consumption and the performance Increasing this value can enhance receive performance but also consumes more system memory In case of lack of received buffers dropped packets or out of order received packets you can increase the number of received buffers The valid values are 256 up to 4096 TransmitBuffers 2048 The number of packets each ring sends Increasing this value can enhance transmission performance but also consumes system
91. tempts attempts that failed RDMA Inbound Bytes sec The number of bytes for all incoming RDMA traffic This includes additional layer two protocol overhead RDMA Inbound Frames sec The number in frames of layer two frames that carry incoming RDMA traffic RDMA Initiated Connections The number of outbound connections established RDMA Outbound Bytes sec The number of bytes for all outgoing RDMA traffic This includes additional layer two protocol overhead RDMA Outbound Frames sec The number in frames of layer two frames that carry outgoing RDMA traffic Mellanox Technologies 72 J Rev 1 21 4 Utilities 4 1 Fabric Performance Utilities The performance utilities described in this chapter are intended to be used as a performance micro benchmark They support both InfiniBand and RoCE For further information on the following tools please refer to the help text of the tool by running the help command line parameter hax Table 9 Fabric Performance Utilities Utility Description nd write bw This test is used for performance measuring of RDMA Write requests in Microsoft Windows Operating Systems nd write bw is perfor mance oriented for RDMA Write with maximum throughput and runs over Microsoft s NetworkDirect standard The level of customizing for the user is relatively high User may choose to run with a customized message size customized number of iterations or alternativel
92. the Property window Step 6 Set its value in the Value window zh Device Manager Sri File Action View Help lt m E Kd m V amp Rs 4 gj l dev w072 p pi Computer _ D Disk drives Details Events Resources Power Management p y Display adapters General Advanced Information Performance Driver b UN Hernan Dewites The following properties are available for this network adapter Click p IEEE 1394 host controllers the property you want to change on the left and then select its value p Keyboards on the right p PW Mice and other pointing devices p Monitors Receive Side Scaling 4 amp Network adapters Intel R 1350 Gigabit Network Connection Intel R 1350 Gigabit Network Connection 2 P Intel R 1350 Gigabit Network Connection 3 a Intel R 1350 Gigabit Network Connection 4 Mellanox ConnectX 4 VPI Adapter MT4115 3 Mellanox ConnectX 4 VPI Adapter MT4115 4 Microsoft Kernel Debug Network Adapter iy Other devices PCI Memory Controller Ports COM amp LPT gb Print queues BB Processors lt Storage controllers pl System devices Universal Serial Bus controllers vvvvvv Mellanox Technologies 31 Rev 1 21 3 1 5 Configuring Quality of Service QoS 3 1 5 1 QoS Configuration Prior to configuring Quality of Service you must install Data Center Bridging using one of the following methods gt To Disabl
93. the new IP address using either IPv4 or IPv6 RoCE IP based allows RoCE traffic between Windows and Linux systems which use IP based GIDs by default A straightforward extension of the RoCE protocol enables traffic to operate in layer 3 environ ments This capability is obtained via a simple modification of the RoCE packet format Instead of the GRH used in RoCE routable RoCE packets carry an IP header which allows traversal of IP L3 Routers and a UDP header that serves as a stateless encapsulation layer for the Transport Protocol Packets over IP Figure 2 RoCE and RoCE v2 Frame Format Differences EtherType indicates that packet is RoCE i e next header is IB GRH EIE DEED gt p t x EtherType indicates that packet is IP UDP dport number Indicates i e next header is IP ip protocol number that next header is IB BTH indicates that packet is UDP The proposed RoCEv2 packets use a well known UDP destination port value that unequivocally distinguishes the datagram Similar to other protocols that use UDP encapsulation the UDP source port field is used to carry an opaque flow identifier that allows network devices to imple ment packet forwarding optimizations e g ECMP while staying agnostic to the specifics of the protocol header format The UDP source port is calculated as follows UDP SrcPort SrcPort XOR DstPort OR 0xc000 where SrcPort and DstPort are the ports used to establish the connection For e
94. the target folder for the installation Destination Folder Click Next to install to this folder or click Change to install to a differer Install MLNX WinOF2 to C Program Files Mellanox MLNX_WinOF2 InstallShield Mellanox Technologies 14 Rev 1 21 Step 7 The firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card In this case the firmware will not be displayed Ifthe user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly However if the user has both and OEM card and a Mellanox card only the Mellanox card will be updated V Upgrade the HCA s firmware version Recommended Upgrading the firmware version will reboot your machine Note One or more of your HCA adapters has an old firmware version We recommend upgrading to a newer firmware version to enable improved functionality and support driver s capabilities InstallShield Setup Type Choose the setup type that best suits your needs Please select a setup type Complete All program features will be installed Requires the most disk space Custom Choose which program features you want installed and where they will be installed Recommended for advanced users InstallShield Mellanox Technologies 15 Rev 1 21 a Select the desired feature to install Performances tools install the performance tools that are use
95. ts RSS Mode 0 disable The hardware is configured once to use the Toeplitz hash function and the indirection table is never changed Note the I O Acceleration Technology IOAT is not functional in this mode ReturnPacketThreshold 341 The allowed number of free received packets on the rings Any number above it will cause the driver to return the packet to the hardware immediately When the value is set to 0 the adapter uses 2 3 of the received ring size The valid values are 0 to 4096 Note This registry value is not exposed via the UI NumTcb 16 The number of send buffers that the driver allocates for sending purposes Each buffer is in LSO size if LSO is enabled or in MTU size otherwise The valid values are 1 up to 64 Note This registry value is not exposed via the UI ThreadPoll 10000 The number of cycles that should be passed without receiving any packet before the polling mechanism stops when using polling completion method for receiving Afterwards receiving new packets will gen erate an interrupt that reschedules the polling mecha nism The valid values are 0 up to 200000 Note This registry value is not exposed via the UI Mellanox Technologies 63 J Rev 1 21 Value Name Default Value Description AverageFactor 16 The weight of the last polling in the decision whether to continue the polling or give up when using polling completion method for receiving The va
96. tual switch BW Memory Not connected 4096 MB 7 Processor Hee 1 Virtual processor Enable virtual LAN identification 5 IDE Controller 0 The VLAN identifier specifies the virtual LAN that this virtual machine Cs Hard Drive ccena paqana EE Wi v DC x64 fre 920 5 E IDE Controller 1 amp DVD Drive None Bandwidth Management SCSI Controller Enable bandwidth management 8 0 Network Adapter Not connected 4 i kh E 1 muli 9 0 Network Adapter Internal Virtual Switch Minimum bandwidth 0 Mbps Network Adapter 0l o To leave the minimum or maximum unrestricted specify 0 a value Mone 7 COM 2 To remove the network adapter from this virtual machine click Remove None Diskette Drive None I Use a legacy network adapter instead of this network adapter to perform network based installation of the guest operating system or when integration i Name f services not installed in the guest operating system vmi Management Integration Services Step 6 Enable the SR IOV for Mellanox VMNIC 1 Open VM settings Wizard 2 Open the Network Adapter and choose Hardware Acceleration 3 Tick the Enable SR IOV option 4 Click OK Mellanox Technologies 54 Rev 1 21 Figure 11 Enable SR IOV on VMNIC vmi 9 amp Hardware Hardware Acceleration H Add Hardware i BIOS Specify networking tasks that
97. uctions and you see the mes sage displayed in the figure below update the registry configuration as described in the Get VmHost lovSupportReasons message Figure 6 SR IOV Support Step 3 Reboot Step 4 Verify the system is configured correctly for SR IOV as described in Steps 1 2 3 3 3 2 4 Creating a Virtual Machine SR IOV Ethernet Only To create a virtual machine Step 1 to Server Manager gt Tools gt Hyper V Manager Step 2 Go to New gt Virtual Machine and set the following Name lt name gt Startup memory 4096 MB Connection Not Connected Mellanox Technologies 48 Rev 1 21 Figure 7 Hyper V Manager File Action View Help e alm Ele Ba Hyper V Manager LAB NALABJSSSEE Virtual Machines Name State CPU Usage Assigr New Virtual Machine No virtual machines were found on this serve Import Virtual Machine Hard Disk Hyper V Settings Floppy Disk Virtual Switch Manager Virtual SAN Manager Edit Disk Inspect Disk Stop Service virtual machine selected Remove Server 1 Refresh View i Help No item selected lt Displays the New Virtual Machine Wizard Step3 Connect the virtual hard disk in the New Virtual Machine Wizard Step 4 Goto Connect Virtual Hard Disk gt Use an existing virtual hard disk Step 5 Select the location of the vhd file Mellanox
98. ure PFC on Mellanox ConnectXTM cards There are multiple configuration steps required all of which may be performed via Power Shell Therefore although we present each step individually you may ultimately choose to write a PowerShell script to do them all in one step Note that administrator privileges are required for these steps Mellanox Technologies 27 Rev 1 21 3 1 3 2 1 Configuring Windows Host Since PFC is responsible for flow controlling at the granularity of traffic priority it is As per RoCE configuration all ND NDK traffic is assigned to one or more chosen pri 4 necessary to assign different priorities to different types of network traffic orities where PFC is enabled on those priorities Configuring Windows host requires configuring QoS To configure QoS please follow the pro cedure described in Section 3 1 5 Configuring Quality of Service on page 32 3 1 3 2 1 1 Global Pause Flow Control gt To use Global Pause Flow Control mode disable QoS and Priority PS Disable NetQosFlowControl PS Disable NetAdapterQos lt interface name gt gt To confirm flow control is enabled in adapter parameters Device manager gt Network adapters gt Mellanox ConnectX 4 Ethernet Adapter gt Properties Device Manager gt Advanced tab File Action View Help 9 m o His Fas 4 l dev w072 1 Computer b Disk drives b Display adapters b ia Human Inter
99. ve PFC enabled on them Step 9 Disable Priority Flow Control PFC for all other priorities except for 3 PS Disable NetQosFlowControl 0 1 2 4 5 6 7 Step 10 Enable QoS on the relevant interface PS Enable NetAdapterQos InterfaceAlias Ethernet 4 Step 11 Enable PFC on priority 3 PS Enable NetQosFlowControl Priority 3 To add the script to the local machine startup scripts Step 1 From the PowerShell invoke gpedit msc Step2 In the pop up window under the Computer Configuration section perform the following 1 Select Windows Settings 2 Select Scripts Startup Shutdown 3 Double click Startup to open the Startup Properties 4 Move to PowerShell Scripts tab Local Group Policy Editor zl M File Action View Help e zm E 15 Local Computer Policy 4 Computer Configuration Name Software Settings Windows Settings 1 2 Display Properties b Name Resolution Poli ver E Shutdown Scripts Startup Shutdown Description Security Settings Contains computer startup scripts b Policy based QoS 23 Administrative Templates 4 88 User Configuration 23 Software Settings Windows Settings 23 Administrative Templates Scripts PowerShell Scripts Windows PowerShell Startup Scripts for Local Computer Name Parameters Up C Program Files PFC P Add Remove For this GPO run scripts
100. wMLNX StaticMacAddress 00155D730100 Add VMNetworkAdapter VMName mtlael5 006 SwitchName VSwMLNX StaticMacAddress 00155D730101 Mellanox Technologies 87 J Rev 1 21 The commands from Step 2 4 are not persistent Its suggested to create script is running after each 05 reboot Step 2 Configure a Subnet Locator and Route records on each Hyper V Host Host 1 and Host 2 mtlael4 amp mtlael5 New NetVirtualizationLookupRecord CustomerAddress 172 16 14 5 ProviderAddress 192 168 20 114 VirtualSubnetID 5001 MACAddress 00155D720100 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 14 6 ProviderAddress 192 168 20 114 VirtualSubnetID 5001 MACAddress 00155D720101 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 15 5 ProviderAddress 192 168 20 115 VirtualSubnetID 5001 MACAddress 00155D730100 Rule TranslationMetho dEncap New NetVirtualizationLookupRecord CustomerAddress 172 16 15 6 ProviderAddress 2 168 20 115 VirtualSubnetID 5001 MACAddress 00155D730101 Rule TranslationMetho ncap Add customer route New NetVirtualizationCustomerRoute RoutingDomainID 11111111 2222 3333 4444 000000005001 VirtualSubnetID 5001 DestinationPrefix 172 16 0 0 16 NextHop 0 0 0 0 Metric 255 Step 4 Configure the Provider Address and Route records on Hyper V Host 2 Host 2 Only mtlae15 SNIC Get NetAdapter Port1 N
101. ws HKEY LOCAL MACHINEVSYSTEMVCurrentControlSetVServices mlx5 Parameters VRoce For changes to take effect please restart the network adapter after changing this registry key Aaa 3 1 3 6 1 Registry Key Parameters The following are per driver and will apply to all available adapters Table 6 Registry Key Parameters Parameters Name non Description Allowed Values and Default roce mode DWORD Sets the RoCE mode The following RoCE MAC Based 0 are the possible RoCE modes RoCEv2 2 RoCEMAC Based NoRoCE 4 RoCE v2 Default No RoCE NoRoCE 3 1 4 Teaming and VLAN Windows Server 2012 and above supports Teaming as part of the operating system Please refer to Microsoft guide NIC Teaming in Windows Server 2012 following the link below http www microsoft com en us download confirmation aspx id 403 19 Note that the Microsoft teaming mechanism is only available on Windows Server distributions Mellanox Technologies 30 J Rev 1 21 3 1 4 1 Configuring a Network Interface to Work with VLAN in Windows Server 2012 and Above In this procedure you DO NOT create a VLAN rather use an existing VLAN ID P To configure a port to work with VLAN using the Device Manager Step 1 Open the Device Manager Step2 Go to the Network adapters Step3 to the properties of Mellanox ConnectX 4 Ethernet Adapter card Step 4 to the Advanced tab Step 5 Choose the VLAN ID in
102. x SRIOV Virtual Switch sp Mellanox SRIOV Virtual Switch Mellanox ConnectX 3 Etherne Notes amp Global Network Settings A 0 MAC Address Range 00 15 5D 21 AC 00 to 00 15 5D 2 v Connection type What do you want to connect this virtual switch to External network Mellanox Connectx 3 Ethernet Adapter v v Allow management operating system to share this network adapter v Enable single root I O virtualization 5R IOV Internal network Private network VLAN ID Enable virtual LAN identification for management operating system The VLAN identifier sp stem will use For setting does not affect rtual LAN that the r unications thr 2 SR IOV can only be configured when the virtual switch is created An external virtual switch with SR IOV enabled cannot be converted to an internal or private switch Step 3 Click Apply Step 4 Click OK Mellanox Technologies 53 Rev 1 21 Step 5 Adda VMNIC connected to a Mellanox vSwitch in the VM hardware settings Under Actions go to Settings gt Add New Hardware gt Network Adapter gt OK In Virtual Switch dropdown box choose Mellanox SR IOV Virtual Switch Figure 10 Adding a VMNIC to a Mellanox v Switch vmi v amp Hardware 0 Network Adapter 9 Add Hardware IK BIOS Specify the configuration of the network adapter or remove the network adapter Boot from CD Vir
103. xample in a Network Direct application when connecting to a remote peer the destination IP address and the destination port must be provided as they are used in the calculation above The source port provision is optional Furthermore since this change exclusively affects the packet format on the wire and due to the fact that with RDMA semantics packets are generated and consumed below the AP applications can seamlessly operate over any form of RDMA service including the routable version of RoCE as shown in Figure 2 RoCE and v2 Frame Format Differences in a completely trans parent way 1 Standard APIs are IP based already for all existing technologies Mellanox Technologies 26 Rev 1 21 Figure 3 RoCE and RoCEv2 Protocol Stack RDMA Application mx ND NDK API RDMA API Verbs RoCE v1 RoCE v2 go Tcr oc Y The fabric must use the same protocol stack in order for nodes to communicate The default RoCE mode is RoCE v1 3 1 3 2 RoCE Configuration In order to function reliably RoCE requires a form of flow control While it is possible to use global flow control this is normally undesirable for performance reasons The normal and optimal way to use RoCE is to use Priority Flow Control PFC To use PFC it must be enabled on all endpoints and switches in the flow path In the following section we present instructions to config
104. y cus tomized test duration time nd write bw runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation nd write lat This test is used for performance measuring of RDMA Write requests in Microsoft Windows Operating Systems nd write lat 1s perfor mance oriented for RDMA Write with minimum latency and runs over Microsoft s NetworkDirect standard The level of customizing for the user is relatively high User may choose to run with a customized mes sage size customized number of iterations or alternatively customized test duration time nd write lat runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation nd read bw This test is used for performance measuring of RDMA Read requests in Microsoft Windows Operating Systems nd read bw is performance oriented for RDMA Read with maximum throughput and runs over Microsoft s NetworkDirect standard The level of customizing for the user is relatively high User may choose to run with a customized mes sage size customized number of iterations or alternatively customized test duration time nd read bw runs with all message sizes from 1B to 4MB powers of 2 message inlining CQ moderation Mellanox Technologies 73 J Rev 1 21 Utility Description nd read lat This test is used for performance measuring of RDMA Read requests in Microsoft Windows Operating Systems nd read lat is performance oriented
105. ys lees 62 3445 Ethernet Registry Keys u u usss X UPON SUEYYS PS 65 3 4 6 Network Direct Interface 67 3 5 Performance Tuning and Counters 68 3 5 1 General Performance Optimization and 68 3 5 2 Application Specific Optimization and Tuning 69 3 5 3 Tunable Performance 69 3 5 4 Adapter Proprietary Performance Counters 71 Chapter4 Utilities 44 5 saw oes ERAS EX RR be Sees oe TO 4 1 Fabric Performance 73 4 2 Management Utilities 0 0 0 ya apu een 75 43 Diagnostic Utilities us mex LER ae dial uqa q neat 75 Chapter 5 10 5 1 Installation Related Troubleshooting 76 5 1 1 Installation Error Codes and Troubleshooting 76 5 2 Ethernet Related Troubleshooting T 5 3 Performance Related Troubleshooting 79 5 3 1 General Diagnostics i vt PAS et ONSE PUDE 79 5 4 Virtualization Related Troubleshooting 80 5 5 Reported Driver Events 81 5 6 State Dumping eee DOS PPS DENS ROS Day OA Sen 84
106. ze supported by FW lt Y gt lt Z gt lt M gt lt F gt are required lt O gt are supported Mellanox ConnectX 4 VPI Adapter X Driver startup fails because maximum size that is supported by FW lt Y gt lt L gt lt M gt is too small lt K gt Please burn a firmware that supports a greater WQE size and restart the Mellanox Con nectX device For additional information please refer to Support information on http mellanox com e Mellanox ConnectX 4 VPI Adapter X CQ moderation is not supported by FW lt Y gt lt Z gt lt L gt e Mellanox ConnectX 4 VPI Adapter X CQ to EQ remap is not supported by FW lt Y gt lt Z gt lt L gt e Mellanox ConnectX 4 VPI Adapter X VPort counters are not supported by FW lt Y gt lt Z gt lt L gt e Mellanox ConnectX 4 VPI Adapter X LSO is not supported by FW lt Y gt lt Z gt lt L gt Mellanox ConnectX 4 VPI Adapter X Checksum offload is not supported by FW lt Y gt lt Z gt lt L gt NDIS initiated reset on device Mellanox ConnectX 4 VPI Adapter X has failed Mellanox ConnectX 4 VPI Adapter lt X gt mstdump System Root Temp lt Y gt _ lt Z gt lt L gt _ lt M gt _ lt F gt _ lt O gt log was created after fatal error Mellanox ConnectX 4 VPI Adapter lt X gt mstdump System Root Temp lt Y gt _ lt Z gt lt L gt _ lt M gt _ lt F gt _ lt O gt log was created after OID request Mellanox Technologies 83 J Rev 1 2

Mellanox WinOF-2 User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents