Home

MLNX VPI Windows user manual

1. 9 2 3 Supported Network Adapter Cards and Firmware Versions 9 2 4 Managing Firmware 10 2 4 1 Downloading the Firmware Tools Package 1 0 0 0 0 cee eens 10 2 4 2 Downloading the Firmware Image of the Adapter Card 0 0 0 0 0 ce eee 10 2 4 3 Updating Adapter Card Firmware dd creer ca kk eee een teens 11 Chapter 3 Dryer Features ou ov eu os AA A AAA dd Be ws IA 12 3 1 RDMA over Converged Ethernet RoCE 12 oL ROCE Overview Qu descrita und dra anal alent a dpa Uu asa rr dank iad Beate ea eee 12 35 12 Pored AppMCaliOnS us nor rb ene Lae dee a DS eae ed ae eme gore UR KI SS 13 3 1 3 Reading Port Counters Statistics da iara 13 S lve Settine ROCE 24 22 rei deba ade A 13 SS etung ROCE M DU scr di eri 13 32 Hyper V with VMQ 14 3 2 1 Enabling Virtual Machine Queue on Windows 2008 R2 o oo o oooooooooooo o 14 3 3 Header Data Split 14 3 4 Receive Side Scaling RSS 15 3 5 Port Configuration 15 Sol aiio SeTISTEIO cuo A Ng ace a air ees Ur Pu trc iA SENS 15 33 2 Port Protocol COM SUTA MOT dsd aro medo d
2. 34 Chapter 6 OpenSM Subnet Manager cc ccc ccc ccc cece eee hh hm hn nnn 38 Chapter 7 luhniBand EabEIC isis pas dada 39 7 1 Network Direct Interface 39 Mellanox Technologies 3 J Rev 3 0 0 7 2 InfiniBand Fabric Diagnostic Utilities 39 vp Ult Sas era con eder dd Aa Rin eA ee oe 39 4 2 2 MDDORISISIG 6625 42 3 2 x ucts sis damit d ita ah loe 45 PoP ADEO O a tant O d 49 TI SIBDQUELV a teu UO ERI UA Ed oe dibus bas EpL at oe ek 5 A A ESI IIT t T TTL DIEM 55 VEU Mb RR CEDE 59 U2 ADMCIMISCONCE acta snp nta Hd Rt ise da dades iode 60 LS ADAC red oet deb A Ara sd eui LAC aU dele Gas AA ede este 64 IA AMO do Do Wie yarns aa oido om Pelton eared cao te 65 TAO MOCKS A CTL ONS uae co a o Ea UR dra dba ra d ibd pide ua AAA adea Stn 67 JACET A d d iiie d od quen acte dub UR AE ru ue ac tosses bubus ted toil m USER rd ie 67 Ll A attend deae ide A et al cae aA ctp rp copa A 68 NZD Watt dile ee ateos a dde idee mites e Bad Bak 69 CP NE uU DC LET 69 7 3 InfiniBand Fabric Performance Utilities 71 Pa IMO CM A r a a a E O h Wesel MDE TCAs L estat ned ati ab EE T2 hor AD SENG OWiss dd s paid hd Ld iieri batis icc HAs di Lud one Eel 73 7 9 TO Send db uec osse onto bo Ee de esL ir egal A sd 74 iocos ID TNL DULL dt s aaa o aei to
3. SYNOPSYS on page 44 e Section 7 2 2 ibportstate on page 45 e Section 7 2 3 ibroute on page 49 e Section 7 2 4 smpquery on page 51 e Section 7 2 5 perfquery on page 55 e Section 7 2 6 ibping on page 59 e Section 7 2 7 ibnetdiscover on page 60 e Section 7 2 8 ibtracert on page 64 e Section 7 2 9 sminfo on page 65 e Section 7 2 10 ibclearerrors on page 67 e Section 7 2 11 ibstat on page 67 e Section 7 2 12 vstat on page 68 e Section 7 2 13 part man on page 69 e Section 7 2 14 osmtest on page 69 7 2 1 Utilities Usage This section first describes common configuration interface and addressing for all the tools in the package Then it provides detailed descriptions of the tools themselves including operation syn opsis and options descriptions error codes and examples Mellanox Technologies 39 J Rev 3 0 0 InfiniBand Fabric 7 2 1 1 Common Configuration Interface and Addressing Topology File Optional An InfiniBand fabric is composed of switches and channel adapter HCA TCA devices To iden tify devices in a fabric or even in one switch system each device is given a GUID a MAC equivalent Since a GUID is a non user friendly string of characters it is better to alias it to a meaningful user given name For this objective the IB Diagnostic Tools can be provided with a topology file which is an optional configurat
4. Step 1 Display the Device Manager E Device imt eimi TH m au E Monitors Network adapters Broadcom BCMS709C Netxtreme TE GigE NDIS VBD Cent E Erasdocin BCMB7U9C MetXtreme II GigE NDIS VBD Client 2 Y Ports moet UT D Processors HP Security Devices lt gt Storage controllers 2 System devices A ACPI Fixed Feature Button Ll Broadcom BCIB7USC Metxtreme IE tage M Broadcom BIMS709C Metxtreme II Gage JV Composite Bus Enumerator E Direct memory access controller Mi Intel R 5520 5500 Physical and Link Layer Registers Port 1 3427 A Intel R 5520 5500 Routing and Protocol Layer Register Port 1 3428 M Intel R 5520 5500 x58 1 0 Hub Control Status and RAS Registers 3423 A Intel R 5520 5500 453 1 0 Hub GPIO and Scratch Pad Registers 3422 j Intel R 5520 5500 59 1 0 Hub PCI Express Root Port 1 3408 M Intel R 5520 5500 x58 1 0 Hub PCI Express Root Port Z 3409 JW Intel R 5520 5500 58 1 0 Hub PCI Express Root Port 3 340A E Intel R 5520 5500 58 1 0 Hub PCI Express Root Port 7 340E ME Intel R 5520 5500 458 1 0 Hub System Management Registers 342E A IntekR 5520 5500 58 T O Hub Throttle Registers 3438 E Intel R 5520 5500 58 L O Hub to ESI Port 3406 WE Intel R 5520 5500 58 Physical and Link Layer Registers Port 0 3425 M Intel R 5520 5500 X58 Routing and Protocol Layer Registers Port O 3426 A InEekR 5520 S 500 58 Trusted Execution Technology Registers 342F eM Intel R 82601
5. in usec between the receiving of a packet and the generation of an interrupt even if the moderation count has not been reached default 10 Rx Interrupt Moderation Type Sets the rate at which the controller moderates or delays the generation of interrupts making it possible to optimize network throughput and CPU utilization The default setting Adaptive adjusts the interrupt rates dynamically depending on the traffic type and network usage Choosing a different setting may improve network and system performance in certain configurations Send completion method Sets the completion methods of the Send packets and it may affect network throughput and CPU utilization Mellanox Technologies 35 J Rev 3 0 0 Performance Interrupt Moderation TX Packet Count Number of packets that need to be sent before an interrupt is generated on the send side default 0 Interrupt Moderation TX Packet Time Maximum elapsed time in usec between the sending of a packet and the generation of an interrupt even 1f the moderation count has not been reached default 0 Bus master DMA Operations Sets the addressing type NDIS DMA addressing UseDma Enabled or physical addressing UseDma Disabled default Disabled e Offload Options Allows you to specify which TCP IP offload settings are handled by the adapter rather than the oper ating system Enabling offloading services increases transmission performance as the offload tasks
6. lt op gt lt value gt 7 2 2 3 Options The table below lists the various flags of the command Table 7 ibportstate Flags and Options penne Print Print the help menu Print the help menu menu d debug Raise the IB debug level May be used several times for higher debug levels ddd or d d d Mellanox Technologies 45 J Rev 3 0 0 InfiniBand Fabric Table 7 ibportstate Flags and Options Continued Flag Description Show send and receive errors timeouts and others Increase verbosity level May be used several times for additional verbosity vvv or v v v D Direct Use directed path address arguments The path is a comma separated list of out ports Examples 0 self port 0 1 2 1 4 out via port 1 then 2 L Lid Use Use Lid address argument address Use Lid address argument G Guid Use GUID address argument In most cases it is the Port GUID Example 0x08f1040023 In case of multiple channel adapters CAs or multiple ports without a CA port being specified a port 1s chosen by the utility according to the following criteria 1 The first ACTIVE port that is found 2 If not found the first port that is UP physical link state is LinkUp Examples I Query the status of Port I of CA mlx4 0 using ibstatus and use its output the LID 3 in this case to obtain additional link information using ibportstate gt ibstatus mlx4 0 1 Infiniband
7. use events n iteration num g num of qps in mcast group a al r Sesa V ersion C report cycles H report histogram U report unsorted F CPU freq fail 7 3 10 2 Options The table below lists the various flags of the command Table 29 ibv_send_lat Flags and Options Flag Description Mellanox Technologies 81 J Rev 3 0 0 InfiniBand Fabric Table 29 ibv send lat Flags and Options Flag Description S sl lt sl gt The service level default 0 Cocu rm F CPU freq The maximum size of message to be sent in inline mode default 0 M MGID lt multicast_gid gt In case of multicast uses lt multicast_gid gt as the group MGID The format must be 255 1 X X X X X X X X X X X X X X where X is a vlaue within 0 255 You must specify a different MGID on both sides to avoid loopback Connect QPs with rdma_cm and run test on those QPs Communicate with rdma cm module to exchange data use regular QPs 7 3 11 ibv_write_bw This is a more advanced version of ib write bw and contains more flags and featurs than the older version and also improved algorithms ibv write bw calculats the BW of RDMA write between a pair of machines One acts as a server and the other as a client The client RDMA writes to the server memory and calculate the BW by sampling the CPU each time it receive a successfull completion The test supports a large variety of features as described below and has better p
8. 3 3 1 Synopsys ib send bw i b port ib port c onnection type RC UC UD m tu mtu size s ize Wexiselorer eras ted size n iteration num p ort PDT port b idirectional a 11 V ersion 7 3 3 2 Options The table below lists the various flags of the command Table 22 ib send bw Flags and Options PS Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 7 3 4 ib send lat ib send lat calculats the latency of sending a packet in message sizeB between a pair of machines One acts as a server and the other as a client They perform a ping pong benchmark on which you send packet only if you receive one Each of the sides samples the CPU each time they receive a packet in order to calculate the latency 14 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 3 4 1 Synopsys ib send lat i b port ib port c onnection type RC UC UD m tu mtu size s ize meo aoa Le nies ien Used o n iteration num p ort BDI per Eee SA SR OU C report cycles H report histogram U report unsorted 7 3 4 2 Options The table below lists the various flags of the command Table 23 ib send lat Flags and Options p Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 7 3 5 ib write bw ib w
9. Dedi Rad ue a d lee 16 3 6 Load Balancing Fail Over LBFO and VLAN 17 SOs Adapter eate td IU Bee ae ARM e eaeque den eras 18 3 6 2 Creating a Load Balancing and Fail Over LBFO Bundle o o ooooooo 18 00 Creaune a Port VIGA Neiscustas la is da 22 310 4 Removinea Pon LAN rt ea Euer e ha e Eds EP ee 25 Chapter 4 Driver Configuration vuees ves OR re a A A A a Rica cac ere aes 26 4 1 Configuring the InfiniBand Driver 26 4 1 1 Modifying Mellanox HCA Configuration 0 0 0 00 cee es 26 4 1 2 Modifying IPoIB Configuration 26 4 1 3 Displaying Adapter Related Information 0 0 0000s 27 4 2 Configuring the Ethernet Driver 29 Chapters Performanc oia ie dines jas ER A tite ana wie dd add Vb DU aq ee e vd 32 5 1 General Performance Optimization and Tuning 32 Sa PRECISE Tun aoi decet et O et eS ad Oo ED denuo luc does ats 32 Delis AA das 32 5 13 Tuning the Network Adapter i be ee id qoe oen ler ree SUE e edat 32 5 2 Application Specific Optimization and Tuning 33 5 2 1 Ethernet Performance T ntnp astro y MPH Re PN ICREWORRUPQEen es Res RR 33 2 2 2 I PolB Performance 1 niBlg sid ose wind odd HEE ee DS qe ue RUN ES AS SM 33 5 3 Tunable Performance Parameters
10. Flags and Options uuu A AA ERE 34 smpguery Flaes and ODLIOHS deua e eK dad ovk Sands unb Pd Vr ds 37 perfquery Flags and Options o o o oooooooo ss rr rr rr rr rr Rss 41 ibpine Flaes and Options e yu A TREES 44 ibnetdiscover Flas aid ODpUOS diede HERR ERR PSP HIE PP RE deg es 45 ibtracert Places and ODLUOflss oe dba Da DER P Ee da e audet 49 sminto Flags and OPNS orda an s bd ee E wn ta ood RUN ATIS ERU SS e 50 ibelearerrors Flags and Options lt a Dn UNES RE RE RE PUES E 51 ibstat Flags and OPUONS scada leia 22 ibstat Flags and Options sierra EU ERA RESP AR EE E IPEA EU ia dE 53 part man Flags and OpHUOnS oos IA AA AES 53 osmtest Flaps and ODUOPBS x aor d etue abet pibe edens de voe hod ab et Soh cael baad een 54 ib read bw Flags and Options o rr rr rr rr rr rss rss reses 57 ib read lat Flags and Options o o o o o rr rr rss rss rss ss 58 ib send bw Flags and Opu0ns s ous care ARA a SS qn tib Re 58 ib send lat Flags and ODDIOBS isis quies dad eoe E o e RR LR ee CE Re eds 59 ib write bw Flags and Options o ooooooooorrrr es 60 ib write lat Elags and Options 2o no y A A 61 iby Tead bw Flags and ODLIOTS aacra sos anoncin aia Spo t Ah Yee E RA deae 62 ibv read lat Flags and Options o o o oo ooo rss ee ee ees 63 ibv send bw Flags and Options 0 0 0 0000 ee eee eens 64 iby Send lat Flags and OpuOnS 4 4 2 ayia heey dat eee Se De UE ERAS 65 iby write bw Flags and Opuons 4 ec
11. PCI Bridge 244E i Inkek RE Chipset QuickData Technology device 3429 Me Intel R Chipset QuickData Technology device 342A IE IntekR Chipset QuickData Technology device 3428 vi Intel R Chipset QuickData Technology device 342C E IntekR Chipset QuickData Technology device 3430 pf Intel R Chipset QuickData Technology device 3431 E Intel R Chipset QuickData Technology device 3432 UE Intel Chipset QuickData Technology device 3433 Mi Intel R ICH10 Family PCI Express Root Port 1 3440 ME InkekR ICHIO Family PCI Express Root Port 5 E Intel ICH10 Family SMBus Controller 3430 HE intelR ICHIO LPC Interface Controler 3418 Ld Mellanox Connects MEM Channel tad Step 2 Right click a Mellanox ConnectX VPI adapter under System devices list and left click Proper ties Mellanox Technologies 21 J Rev 3 0 0 Driver Configuration Step 3 Select the Information tab from the Properties sheet Mellanox Connect 10Gb Ethernet Adapter Properties Ed VLAN LEFO Driver Details Power Management General Information Diagnostics Advanced Performance Adapter Information Mellanox Driver Version 3 10 3283 0 Firmware Version 2 9 8350 Fort Humber 2 Bus Type PCI E 5 0 Gbps 8 Link Speed 10 0 Gbps Full Duplex Part Number MHUHZSB XTR Device Id 264285 Revision Id BU Current MAC Address 00 02 9 06 4 41 Permanent MAL Address DOU D2 D9 DB 4 1 Network Status Conne
12. Ree Oe fone GENI a a ee es 67 ibv write lat Flags and Options o ooooooooooooonornnanoooo ees 68 Rev 3 0 0 Document Revision History Table 1 Document Revision History Document Revision Date Changes 0000 Rev 3 0 0 February 08 2012 Rev 2 1 2 Rev 2 1 1 1 July 14 2010 Added section RDMA over Converged Ethernet RoCE and its subsections Added section Hyper V with VMQ Added section Network Driver Interface Specification NDIS Added section Header Data Split Added section Auto Sensing Added section Adapter Teaming Added section Port Protocol Configuration Added section Advanced Configuration for InfiniBand Driver Added section Advanced Configuration for Ethernet Driver Added section Updated section Tunable Performance Parame ters Added section Merged Ethernet and InfiniBand features sec tions Removed section Sockets Direct Protocol and its subsections Removed section Winsock Direct and Protocol and its subsec tions Removed section Added ConnectX 3 support Removed section IPoIB Drivers Overview Removed section Booting Windows from an iSCSI Target Removed section Debug Options Updated Section 3 Uninstalling Mellanox VPI Driver on page 11 Added Section 6 InfiniBand Fabric on page 38 and its sub sections Added Section 6 3 InfiniBand Fabric Performance Utilities on page 71 and its subsections Removed all references of InfiniHost adapter since it is not sup
13. Suggestion This can happen if the network capture tool captures the network traffic of the non active adapter in the bundle This is not allowed since the tool sets the packet filter to promis cuous thus causing traffic to be transferred on multiple interfaces Close the network capture tool on the physical adapter card and set it on the LBFO interface instead Issue 7 No Ethernet connectivity on 1Gb 100Mb adapters after activating Performance Tuning part of the installation Suggestion This can happen due to adding a TcpWindowSize registry value To resolve this issue remove the value key under HKEY LOCAL MACHINE SYSTEM CurrentControl SetiServices Tcpip Parameters TcpWindowsS ize or set its value to OXFFFF Issue 8 System reboots on an I OAT capable system on Windows Server 2008 Suggestion This may occur if you have an Intel I OAT capable system with Direct Cache Access enabled and 9K jumbo frames enabled To resolve this issue disable 9K jumbo frames Issue 9 Packets are being lost Suggestion This may occur if the port MTU has been set to a value higher than the maximum MTU supported by the switch Issue 4 10 Issue s not listed above Suggestion The MLNX EN for Windows driver records events in the system log of the Win dows event system Using the event log you ll be able to identify diagnose and predict sources of system problems To see the log of events open System Event Viewer as follows 1 Right
14. a switch this line also contains whether switch port 0 is base or enhanced and the LID and LMC of port 0 Subsequent lines pertaining to this node show the connectivity On the left 1s the port number of the current node On the right is the peer node node at other end of link It is identified in quotes with nodetype followed by followed by NodeGUID with the port number in square brackets Further on the right is a Mellanox Technologies 61 J Rev 3 0 0 InfiniBand Fabric comment What follows the comment is dependent on the node type If it it a switch node it is followed by the NodeDescription in quotes and the LID of the peer node If it is a CA or router node it is followed by the local LID and LMC and then followed by the NodeDescription in quotes and the LID of the peer node The active link width and speed are then appended to the end of this output line Example Topology file generated on Tue Jun 5 14 15 10 2007 Max of 3 hops discovered Initiated from node 0008f10403960558 port 0008 10403960559 Non Chassis Nodes When grouping is used IB nodes are organized into chasses which are numbered Nodes which cannot be determined to be in a chassis are displayed as Non Chassis Nodes External ports are also shown on the connectivity lines vendid 0x8f1 devid 0x5a06 sysimgguid 0x5442ba00003000 switchguid 0x5442ba00003080 5442ba00003080 Switch 24 S 005442ba00003080 ISR9024 Voltaire base p
15. are performed by the adapter hardware rather than the operating system Thus freeing CPU resources to work on other tasks J Pv4 Checksums Offload Enables the adapter to compute IPv4 checksum upon transmit and or receive instead of the CPU default Enabled TCP UDP Checksum Offload for IPv4 packets Enables the adapter to compute TCP UDP checksum over IPv4 packets upon transmit and or receive instead of the CPU default Enabled TCP UDP Checksum Offload for IPv6 packets Enables the adapter to compute TCP UDP checksum over IPv6 packets upon transmit and or receive instead of the CPU default Enabled Large Send Offload LSO Allows the TCP stack to build a TCP message up to 64KB long and sends it in one call down the stack The adapter then re segments the message into multiple TCP packets for transmission on the wire with each pack sized according to the MTU This option offloads a large amount of kernel processing time from the host CPU to the adapter IB Options Configures parameters related to InfiniBand functionality SA Query Retry Count Sets the number of SA query retries once a query fails The valid values are 1 64 default 10 SA Query Timeout Sets the waiting timeout in millisecond of an SA query completion The valid values are 500 60000 default 1000 ms This document describes how to modify Windows registry parameters in order to improve performance 4 Please note that modifyin
16. click on My Computer click Manage and then click Event Viewer OR 1 Click start gt Run and enter eventvwr exe 2 In Event Viewer select the system log Mellanox Technologies 87 J Rev 3 0 0 Troubleshooting The following events are recorded A Mellanox ConnectX EN 10Gbit Ethernet Adapter X has been successfully initialized and enabled Failed to initialize Mellanox ConnectX EN 10Gbit Ethernet Adapter Mellanox ConnectX EN 10Gbit Ethernet Adapter X has been successfully initialized and enabled The port s network address is MAC Address The Mellanox ConnectX EN 10Gbit Ethernet was reset Failed to reset the Mellanox ConnectX EN 10Gbit Ethernet NIC Try disabling then re enabling the Mellanox Ethernet Bus Driver device via the Windows device manager Mellanox ConnectX EN 10Gbit Ethernet Adapter X has been successfully stopped Failed to initialize the Mellanox ConnectX EN 10Gbit Ethernet Adapter X because it uses old firmware version old firmware version You need to burn firmware version new firmware version gt or higher and to restart your computer Mellanox ConnectX EN 10Gbit Ethernet Adapter X device detected that the link connected to port Y is up and has initiated normal operation Mellanox ConnectX EN 10Gbit Ethernet Adapter X device detected that the link connected to port lt Y gt is down This can occur if the physical link is disconnected or damaged or if the o
17. option specifies the name of the inventory file Normally osmtest expects to find an inventory file which osmtest uses to validate real time information received from the SA during testing If 1 is not specified osmtest defaults to the file osmtest dat See c option for related information S Stress This option runs the specified stress test instead of the normal test suite Stress test options are as fol lows OPT Description s2 Multi MAD RMPP response SA queries s3 Multi MAD RMPP Path Record SA queries s4 Single MAD non RMPP get Path Record SA queries Without s stress testing is not performed 70 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 19 osmtest Flags and Options Flag Description M Multicast_Mode This option specify length of Multicast test OPT Description Short Multicast Flow default single mode M2 Short Multicast Flow multiple mode M3 Long Multicast Flow single mode M4 Long Multicast Flow multiple mode e Single mode Osmtest is tested alone with no other apps that interact with OpenSM MC Multiple mode Could be run with other apps using MC with Opens M Without M default flow testing 1s performed t This option specifies the time in milliseconds used for transaction timeouts Specifying t 0 disables timeouts Without t OpenSM defaults to a timeout value of 200 milliseconds log file This option defines the log to be
18. the given file By default the log goes to stdout This option increases the log verbosity level The v option may be specified multiple times to further increase the verbosity level See the vf option for more information about log verbosity This option sets the maximum verbosity level and forces log flushing The V is equivalent to vfOxFF d 2 See the vf option for more information about log verbosity vf This option sets the log verbosity level A flags field must follow the D option A bit set clear in the flags enables disables a specific log level as follows BIT LOG LEVEL ENABLED 0x01 ERROR error messages 0x02 INFO basic messages low volume 0x04 VERBOSE interesting stuff moderate volume 0x08 DEBUG diagnostic high volume 0x10 FUNCS function entry exit very high volume 0x20 FRAMES dumps all SMP and GMP frames 0x40 ROUTING dump FDB routing information 0x80 currently unused Without vf osmtest defaults to ERROR INFO 0x3 Specifying vf 0 disables all messages Spec ifying vf OxFF enables all messages see V High verbosity levels may require increasing the transaction timeout with the t option h help Display this usage info then exit T 3 InfiniBand Fabric Performance Utilities The performance utilities described in this chapter are intended to be used as a performance micro benchmark The tools are e Section 7 3 1 ib read bw on page 72 e Section 7 3 2 ib read l
19. 1 GUID 0x0008f1 7 2 8 ibtracert 0x0008f1 0x0008f1 04003f125c 04003 125d 04003 10d2 04003 10d3 04003 10bf 0400400e2d 0400400e2e 0400400e2f 0400400e31 0400400e32 Node Name 0400411208 0400411a28 0x0008 10400411a34 0400411980 PEEN arl STEI TE TIBI SW SW2 SW3 SW4 Rack Rack Rack 1 Rack 11 Rack Rack Rack Rack Rack 1 Rack 1 Rack 1 Rack 1 Rack 11 Rack 11 3 ISR9024 Voltaire 9024D 3 ISR9024 Vol 3 ISR9024 Voltaire 9024D 3 ISR9024 Voltaire 9024D 1 slot 1 ISR9288 ISR9096 Voltaire sLB 24D 1 slot I ISR9288 ISR9096 Voltaire sLB 24D 1 slot 2 ISR9288 ISR9096 Voltaire sLB 24D 1 slot 2 ISR9288 ISR9096 Voltaire sLB 24D 1 slot 12 ISR9288 ISR9096 Voltaire sLB 24D I spine TSR9209 Voltaire ER ISR9288 Voltaire sFB 12D ISR9288 Voltai spine 2 ISR9288 Voltaire sFB 12D spine 2 ISR9288 Voltai taire 9024D re sFB 12D re sFB 12D ibtracert uses SMPs to trace the path from a source GID LID to a destination GID LID Each hop along the path is displayed until the destination 1s reached or a hop does not respond By using the m option multi
20. 1 gt lt pkey2 gt 7 2 14 osmtest osmtest is a test program to validate InfiniBand subnet manager and administration SM SA Default is to run all flows with the exception of the QoS flow osmtest provides a test suite for opensm osmtest has the following capabilities and testing flows e It creates an inventory file of all available Nodes Ports and PathRecords including all their fields t verifies the existing inventory with all the object fields and matches it to a presaved one A Multicast Compliancy test An Event Forwarding test A Service Record registration test An RMPP stress test A Small SA Queries stress test It is recommended that after installing opensm the user should run osmtest f c to generate the inventory file and immediately afterwards run osmtest f a to test OpenSM Additionally it is recommended to create the inventory when the IB fabric is stable and occasion ally run osmtest v to verify that nothing has changed Mellanox Technologies 69 J Rev 3 0 0 InfiniBand Fabric 7 2 14 1 Synopsys osmtest f low cla v s le f m q t w ait trap wait time gt d ebug lt number gt Made nu S CUIDA eo om vento ere a gt stress M ulticas Mode a esmellbsegonds 1 EE evt ags h elp 7 2 14 2 Options The table below lists the various flags of the command Table 19 osmtest Flags and Options This option directs osmtest to run a specific fl
21. 601 PCI Bridge 244 M Intel p Chipset QuickData Technology device 3429 E Intel Chipset QuickData Technology device le Intel R Chipset QuickData Technology device 3428 E Intel R Chipset QuekData Technology device 3420 Al Intel R Chipset QuickData Technology device 343 le Intel R Chipset QuickData Technology device E Intel R Chipset QuickData Technology device 34 ME Inte R Chipset QuickData Technology device A Intel ICHIO Family PCI Express Root Port 1 E Intel A ICHIO Family PCT Express Root Port 5 A Intel R ICHIO Family SMBus Controller 3430 OE Intel R ICHIO LPC Interface Controller 3418 E Melanox Connect QM Rasa Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties Select the Port Protocol tab from the Properties sheet The Port Protocol tab is displayed only if the NIC is a VPI IB and ETH 16 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 The figure below is an example of the displayed Port Protocol sheet for a dual port VPI adapter card Mellanox ConnectX YPI MT26428 PCIe 7 0 5GT s IB QDR 10G 3 General Fort Protocol Driver Details Resources Choose the part protocol that best suits your needs Mellanox v Hw Defaults Port 1 C IE C ETH AUTO RoCE Port CI ETH AUTO RaCE Active ND Port Protocol Configuration This menu enables you to set the network protoco
22. A ATAPI controllers IEEE 1394 Bus host controllers X Keyboards Pl Mice and other pointing devices L Monitors Network adapters E Broadcom BCMS708C NetXtreme II GigE NDIS VBD Client ing E Broadcom BCMS708C Netxtreme II GigE MDIS YBD Client 2 e E Mellanox Connects MT25418 DDR Channel Adapte st E Mellanox Connectx 10b Ethernet Adapter EX Mellanox Connect 10Gb Ethernet Adapter 2 EX Mellanox Virtual Miniport Driver Team A eee NN E fs Other devices Virtual Bundle Team Physical Adapters n Base System Device 0 Ports COM amp LPT m m Processors Ee Storage controllers m i System devices El 9 Universal Serial Bus controllers 22 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Step 2 Right click a Mellanox network adapter under Network adapters list and left click Properties Select the VLAN tab from the Properties sheet Physical Adapter Mellanox Connect 10Gb Ethernet Adapter Properties LBFO Driver Details General Information Advanced Performance VLAN Virtual Lanz Mellanox VLANs associated with this adapter WLAN Name ID Priority Status New Remove Modify This dialog allows you ta configure Virtual LANs VLANs for the adapter HOTE After configuring a VLAN the adapter associated with the VLAN may experience a momentary loss of connectivity The list views has four columns VLAH Hame Displays the
23. G DEV IDX 7 2 1 3 Addressing This section applies to the ibdiagpath tool only A tool command may require defining the destination device or port to which it applies be 40 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 The following addressing modes can be used to define the IB ports Using a Directed Route to the destination Tool option d This option defines a directed route of output port numbers from the local port to the destination e Using port LIDs Tool option 1 In this mode the source and destination ports are defined by means of their LIDs If the fabric is configured to allow multiple LIDs per port then using any of them is valid for defining a port Using port names defined in the topology file Tool option n This option refers to the source and destination ports by the names defined in the topology file Therefore this option is relevant only if a topology file is specified to the tool In this mode the tool uses the names to extract the port LIDs from the matched topology then the tool operates as in the I option Mellanox Technologies 41 J Rev 3 0 0 InfiniBand Fabric 7 2 1 4 SYNOPSYS Todan ee eco M a o pe dud t lt topo file gt s lt sys name gt i lt dev index gt p lt port num gt pm pc P lt lt PM counter gt lt Trash Limit gt gt FE os e Doce le Pas ld skip dup guids zero guids pm
24. Mellanox TECHNOLOGIES WinOF VPI for Windows User Manual Rev 3 0 0 www mellanox com Rev 3 0 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellano
25. Suggestion To troubleshoot this issue follow the steps bellow p Check that the IB driver is running on all nodes by using vstat The vstat utility located at installation directory gt tools displays the status and capabilities of the network adaptor card s On the command line enter vstat use h for options to retrieve information about one or more adapter ports The field port state will be equal to PORT DOWN when there is no InfiniBand cable no link PORT INITIALIZED when the port is connected to some other port physical link PORT ACTIVE when the port is connected and OpenSM is running logical link PORT ARMED when the port is connected to some other port physical link N K ut XH KH W Run OpenSM see OpenSM operation instructions in the OpenSM section above Verify the status of ports by using vstat All connected ports should report PORT ACTIVE State 9 2 Ethernet Troubleshooting Issue 1 The installation of MLNX VPI for Windows fails with the following or a similar error message This installation package is not supported by this processor type Contact your product vendor Suggestion This message is printed if you have downloaded and attempted to install an incor rect MSI for example if you are trying to install a 64 bit MSI on a 32 bit machine or vice versa Issue 2 The performance is low Suggestion This can be due to non optimal system config
26. aise the IB debug level May be used several times for higher debug levels ddd or d d d Show all LIDs in range including invalid entries v verbose Increase verbosity level May be used several times for additional verbosity vvv or v v v D Direct Use directed path address arguments The path is a comma separated list of out ports Examples 0 self port 0 1 2 1 4 out via port 1 then 2 G Guid Use GUID address argument In most cases it is the Port GUID Example 0x08f1040023 M Multieast 0000 Show multicast forwarding tables The parameters lt startlid gt and lt endlid gt specify the MLID range multicast Show multicast forwarding tables The parameters lt startlid gt and lt endlid gt specify the MLID range tables The parameters lt startlid gt and lt endlid gt specify the MLID range Mellanox Technologies 49 J Rev 3 0 0 InfiniBand Fabric Table 8 ibroute Flags and Options Flag Description Override the default timeout for the solicited MADs msec Examples 1 Dump all Lids with valid out ports of the switch with Lid 2 gt ibroute 2 Unicast lids 0x0 0x8 of switch Lid 2 guid 0x0002c902fffff00a MT47396 Infiniscale III Mellanox Technologies Lid Out Destination Port INEO 0x0002 000 Switch portguid 0x0002c902fffff00a MT47396 Infiniscale III Mellanox Technolo gies 0x0003 021 Switch portguid Ox000b8cffff004016 MT47396 Infiniscal
27. and enable the latest I OAT driver download from www intel com e With I OAT enabled sending 256 byte messages or larger will activate I OAT This will cause a significant latency increase due to I OAT algorithms On the other hand throughput will increase significantly when using I OAT 5 2 2 IPolB Performance Tuning The user can configure the IPoIB adapter by setting some registry keys The registry keys may affect IPoIB performance For the complete list of registry entries that may be added changed by the performance tuning pro cedure see the IPoIB registry values pdf file To improve performance activate the performance tuning tool as follows I Start the Device Manager open a command line window and enter devmgmt msc 2 Open Network Adapters Mellanox Technologies 33 J Rev 3 0 0 Performance 3 Right click the relevant IPoIB adapter and select Properties 4 Select the Advanced tab 5 Modify performance parameters properties as desired 5 3 Tunable Performance Parameters The following is a list of key parameters for performance tuning e Jumbo Packet The maximum available size of the transfer unit also known as the Maximum Transmission Unit MTU For IPoIB the MTU should not include the size of the IPoIB header 4B For example if the network adapter card supports a 4K MTU the upper threshold for payload MTU 1s 4092B and not 4096B The MTU of a network can have a substantial impact on
28. assigned VLAN name displayed Mellanox Technologies 23 J Virtual Bundle Team Mellanox Virtual Miniport Driver Team A Properties 7 General YLAN Driver Details Virtual Lanz Mellanox VLANs associated with this adapter WLAN Name ID Priority Status Mew Remove Modif This dialog allows you to configure Virtual LANs WLANs for the adapter HOTE After configuring a VLAN the adapter associated with the VLAN may experience a momentary lass of connectivity The list views has four columns VL AH Hame Displays the assigned VLAN name If a physical adapter has been added to a bundle team then the VLAN tab will not be Rev 3 0 0 Driver Features Step 3 Click New to open a VLAN dialog window Enter the desired VLAN Name and VLAN ID and select the VLAN Priority MLNX EN VLAN Ed VLAN Name VLAN ID WLAN Priority 2 This dialog allows you to enter or modify the following WLAN properties VLAH Hame The name can be any unique alphanumeric string VLAH ID The ID is a number between 1 and 4095 VLAH Priority The priority is a number between 0 and 7 lowest 7 highest HOTE After creating a neve WYLAN the adapter associated with the VLAN may experience a momentary loss of connectivity OK Cancel After installing the first virtual adapter VLAN on a specific port the port becomes dis abled This means that it is n
29. at on page 72 e Section 7 3 3 1b send bw on page 73 e Section 7 3 4 1b send lat on page 74 e Section 7 3 5 ib write bw on page 75 e Section 7 3 6 ib write lat on page 76 e Section 7 3 7 ibv read bw on page 77 e Section 7 3 8 ibv read lat on page 78 e Section 7 3 9 ibv send bw on page 80 e Section 7 3 10 ibv send lat on page 81 e Section 7 3 11 ibv write bw on page 82 e Section 7 3 12 ibv write lat on page 83 Mellanox Technologies 71 J Rev 3 0 0 InfiniBand Fabric 7 3 1 ib read bw ib read bw calculats the BW of RDMA read between a pair of machines One acts as a server and the other as a client The client RDMA reads the server memory and calculate the BW by sampling the CPU each time it receive a successfull completion The test supports features such as Bidirec tional in which they both RDMA read from each other memory s at the same time change of mtu size tx size number of iteration message size and more Read is availible only in RC connection mode as specified in IB spec 7 3 1 1 Synopsys ea PO POR SN tU sis denne message size ete ratio e p ort PDT port b idirectional o uts outstanding reads a 11 V ersion 7 3 1 2 Options The table below lists the various flags of the command Table 20 ib read bw Flags and Options 7 3 2 ib read lat ib read lat calculats the latency of RDMA read operation of message sizeB between a pair of mac
30. cast path tracing can be performed between source and destination nodes 7 2 8 1 Synopsys iburacert alebu l wierbose Mare aa errors ileuvsageh Rias o o A is Ss O Pre mare e a co e V ersion node name map lt node name map gt h elp lt dest dr path lid guid gt lt startlid gt lt endlid gt 7 2 8 2 Options The table below lists the various flags of the command 64 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Most OpenIB diagnostics take the following common flags The exact list of supported flags per utility can be found in the usage message and can be shown using the util name h syntax Table 13 ibtracert Flags and Options Flag Description Simple format do not show additional information mlid m lt mlid gt Show the multicast trace of the specified mlid node name map lt node name map gt Specify a node name map The node name map file maps GUIDs to more user friendly names See Topology File Format on page 61 Use GUID address argument In most cases it is the Port GUID Example 0x08f1040023 Examples e Unicast examples ibtracert 4 16 show path between lids 4 and 16 ibtracert n 4 16 same but using simple output format ibtracert G 0x8f1040396522d 0x002c9000100d051 use guid addresses Multicast example ibtracert m Qxc000 4 16 show multicast path of mlid 0xc000 between lids 4 and 16 7 2 9 sminfo Optionally sets and
31. cing lE Dinamic Link Aggregation 802 Jad Staus Role Primary I Failback to P Select the adapte Adapter Name O Melanos Connect 10Gb Ethernet Adapter 3 O Melanos Connect 10Gb Ethernet Adapter 4 Commit Cancel LAF stands for Load Balancing and Fail Over The administrator can configure a bundle of adapters and associate up to 8 Mellanax Connectx adapters to this bundle LOFO should be used co increase the system reliability upon a link failure and ta balance the workload SA A e 2X2 ea e The newly created virtual Mellanox adapter representing the bundle will be displayed by the Device Manager under Network adapters in the following format see figure below Mellanox Virtual Miniport Driver Team bundle name Device Manager Ol Ea File Action wiew Help e m sim i computer Disk drives Display adapters 4 DVDICD ROM drives 3 Floppy drive controllers AA Human Interface Devices jg IDE ATA ATAPI controllers Es IEEE 1394 Bus host controllers Keyboards A Mice and okher pointing devices Monitors Network adapters PE Broadcom BCMS708C Netxtreme II GigE DIS vBD Client i e xt Broadcom BIMS708C Nebkxkreme II GigE DIS vBED client 2 E e a Melanos Connectx MT25415 DDR Channel Adapter i e E Melanos Connects 10Gb Ethernet Adapter co EX Mellanox Connects 10Gb Ethernet Adapter 2 y Other devices oe hy Base System Device JT P
32. cted Adapter Friendly Mame Local 4rea Connection 4 IPy4 Address 11412122 Save To File OF Cancel Help To save this information for debug purposes click Save To File and provide the output file name 28 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 4 2 Configuring the Ethernet Driver The following steps describe how to configure advanced features Step 1 Display the Device Manager i Device Manager al File Action View Help es cit IDE ATAJATAPI controllers a T IEEE 1394 Bus host controllers A Texas Instruments 1394 OHCI Compliant Host Controller 4 Keyboards gn Mice and ather painting devices B AL Monitors E e Network adapters e Kr Broadcom BCMS709C Mebxtreme IT GigE NDIS VED Client Z Kr Broadcom BCMS709C Netxtreme IT GigE NDIS VBD Client 2 a e IBM USB Remate NDIS Network Device Mellanox Connects 10Gb Ethernet Adapter a e Mellano Connects 10Gb Ethernet Adapter 2 H Ports COM amp LPT B B Processors H p Security Devices El AT Storage controllers E M System devices fA y Universal Serial Bus controllers Mellanox Technologies 29 J Rev 3 0 0 Driver Configuration Step 2 Right click a Mellanox network adapter under Network adapters list and left click Properties Select the Advanced tab from the Properties sheet Mellanox Connectx 10Gb Ethernet Adapter Properties Xx VLAN LBFO Driver Details Power Management G
33. d colecta Den Sig event o 75 723 0 MDW Lats a aid 76 HOE A Era GR Bord ao uon aperit d D US E 77 1 9 9 MOV redd lA ss A ia EAS A do 78 Poe A A I A Me kee 80 IU ADV Sends lacada dia dta perder eut 8l EA DW 4s ach imp Sted ee ete Wats oan daret esie utate tur RE dot Bt at d re 82 Zo OVE SWING la e e iEn a ciue s 83 Chapter 8 Software Development Kit oooooooocooooooooccr rro 85 Chapter 9 Troublesho0tilg sarrada a IRIS RUN ER a EN Quee WA doe o aho d re ie 86 9 1 InfiniBand Troubleshooting 86 9 2 Ethernet Troubleshooting 86 Chapter 10 Documentation ic rep rt E Chro id Sannin eee eeu 89 4 Mellanox Technologies J WinOF VPI for Windows User Manual Rev 3 0 0 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Table 22 Table 23 Table 24 Table 25 Table 26 Table 27 Table 28 Table 29 Mellanox Technologies 9 J Typoeraphical Conventos serres ee Rade OR RO anasto S Redon re Roe O ches 3 Abbreviations and Acronyms 0 0 00 ens 4 ibditaenet or butis Output Dues cup se HERI ERROR ARR PERRA ad 26 ibdiagpathi Output Piles oit reo dac eor idas 29 ibportstate Flags and Options o 30 ibportstate
34. der to calculate latency 76 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 3 6 1 Synopsys ib write lat i b port ib port c onnection type RC UC UD m tu mtu size s ize meo a a lt deol nets bdo n iteration num p ort BDI per Eee SA SR NOU C report cycles H report histogram U report unsorted 7 3 6 2 Options The table below lists the various flags of the command Table 25 ib write lat Flags and Options p Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 7 3 7 ibv read bw This is a more advanced version of ib read bw and contains more flags and featurs than the older version and also improved algorithms ibv read bw Calculats the BW of RDMA read between a pair of machines One acts as a server and the other as a client The client RDMA reads the server memory and calculate the BW by sampling the CPU each time it receive a successfull completion The test supports a large variety of features as described below and has better performance than ib send bw in Nahelem systems Read is availible only in RC connection mode as specified in the InfiniBand spec Mellanox Technologies Tf J Rev 3 0 0 InfiniBand Fabric 7 3 7 1 Synopsys ibv read bw i b port ib port d ib device o uts outstanding reads m tu mtu size e Message size Sede pe AER n iterat
35. device mlx4 0 port 1 status default gid e80 0000 0000 0000 0000 0000 9289 3895 base lid 0x3 sm lid 0x3 Ses 2H LUNE phys state 5 LinkUp rate 20 Gb sec 4X DDR gt ibportstate C mlx4 0 3 1 query 46 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 2 Query the status of two channel adapters using directed paths Mellanox Technologies 47 J Rev 3 0 0 InfiniBand Fabric 3 Change the speed of a port 48 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 dos 2 5 Gbps or 5 0 Gbps A liic dodle ips be C ERE ER 5 0 Gbps IBA extension hwnkspeedAetrvescton etus vate qu ER 5 0 Gbps 7 2 3 ibroute Uses SMPs to display the forwarding tables for unicast LinearForwardingTable or LFT or multi cast MulticastForwardingTable or MFT for the specified switch LID and the optional lid mlid range The default range is all valid entries in the range of 1 to FDBTop 7 2 3 1 Applicable Hardware InfiniBand switches 7 2 3 2 Synopsis ibroute h d v V a n D 6 M L e u s lt smlid gt c lt ca name gt EP Canmore i t timeout ms gt dest dr path lid guid gt lt star tlid gt lt endlid gt 7 2 3 3 Options The table below lists the various ibroute flags of the command Table 8 ibroute Flags and Options EA Print Prine the help menu Prine the help menu menu d debug R
36. displays the output of a sminfo query in a readable format The target SM is the one listed in the local port info or the SM specified by the optional SM lid or by the SM direct routed path Mellanox Technologies 65 J Rev 3 0 0 InfiniBand Fabric Using sminfo for any purposes other then simple query may result in a mal function of the target SM 7 2 9 4 Synopsys smato v Este eoe eS a on cens sues cest ore Ese ete os oe rb edere E nl las usage ec uid else ca mane ise ica porte nie ow timeout mo e ser Sn OR MT neto sm lid sm dr path modifier 7 2 9 2 Options The table below lists the various flags of the command Most OpenIB diagnostics take the following common flags The exact list of supported flags per utility can be found in the usage message and can be shown using the util name h syntax Table 14 sminfo Flags and Options state s Set SM state e notactive e discovering e 2 standby e 3 master debug d ddd d d d Raise the IB debugging level Direct D Use directed path address arguments The path is a comma separated list of out ports Examples e 0 self port e 0 1 2 1 4 out via port 1 then 2 Show send and receive errors timeouts and others Use GUID address argument In most cases it is the Port GUID Example 0x08f1040023 66 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Examples sminfo local ports sminf
37. e A special diff output for ibnetdiscover output will be displayed showing differences between the old and current fabric By default the following are compared for differences switches channel adapters routers and port connections diffcheck lt key s gt Specify what diff checks should be done in the diff option above Comma separate multiple diff check key s The available diff checks are sw switches ca channel adapters router rout ers port port connections lid lids nodedesc node descriptions Note that port lid and nodedesc are checked only for the node types that are specified e g sw ca router If port is specified alongside lid or nodedesc remote port lids and node descriptions will also be com pared Obtain a ports report which is a list of connected ports with relevant information like LID port num GUID width speed and NodeDescription 7 2 7 3 Topology File Format The topology file format is largely intuitive Most identifiers are given textual names like ven dor ID vendid device ID device ID GUIDs of various types sysimgguid caguid switch guid etc PortGUIDs are shown in parentheses For switches this is shown on the switchguid line For CA and router ports it is shown on the connectivity lines The IB node is identified fol lowed by the number of ports and the node GUID On the right of this line is a comment 7 fol lowed by the NodeDescription in quotes If the node is
38. e III Mellanox Technolo gies 0x0006 007 Channel Adapter portguid 0x0002c90300001039 sw137 HCA 1 0x0007 021 Channel Adapter portguid 0x0002c9020025874a sw157 HCA 1 0x0008 008 Channel Adapter portguid 0x0002c902002582cd sw136 HCA 1 5 valid lids dumped 2 Dump all Lids in the range 3 to 7 with valid out ports of the switch with Lid 2 ibroute 2 3 7 Unicast lids 0x3 0x7 of switch Lid 2 guid 0x0002c902fffff00a MT47396 Infiniscale III Mellanox Technologies Lid Out Destination Port Info 0x0003 021 Switch portguid Ox000b8cffff004016 MT47396 Infiniscale III Mellanox Technolo gies 0x0006 007 Channel Adapter portguid 0x0002c90300001039 sw137 HCA 1 0x0007 021 Channel Adapter portguid 0x0002c9020025874a sw157 HCA 1 3 valid lids dumped 3 Dump all Lids with valid out ports of the switch with portguid 0x000b8cf fff004016 gt ibroute G Ox000b8cffff004010 Unicast lids 0x0 0x8 of switch Lid 3 guid Ox000b8cffff004016 MT47396 Infiniscale III Mellanox Technologies 50 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 4 Dump all non empty mlids of switch with Lid 3 7 2 4 smpquery Provides a basic subset of standard SMP queries to query Subnet management attributes such as node info node description switch info and port info Mellanox Technologies 51 J Rev 3 0 0 InfiniBand Fab
39. e are al 7 3 8 2 Options The table below lists the various flags of the command Table 27 ibv read lat Flags and Options PS Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 F CPU freq Mellanox Technologies 79 J Rev 3 0 0 InfiniBand Fabric 7 3 9 ibv_send_bw This is a more advanced version of ib send bw and contains more flags and featurs than the older version and also improved algorithms ibv send bw calculats the BW of SEND between a pair of machines One acts as a server and the other as a client The server receive packets from the client and they both calculate the throughput of the operation The test supports a large variety of features as described below and has better performance than ib send bw in Nahelem systems 7 3 9 1 Synopsys ibv send bw i b port ib port d ib device c onnection type RC UC UD m tu mtu size ss ze message size i x depthi tx size s epo Opeti rx size n uterationyaum sp tort PDT port Ea o al co ES SEI sl type x gid index e vents use events N o peak use peak calc Arona Ec TTC gps in mcast group M mcast gid b idirectional a 11 V ersion 7 3 9 2 Options The table below lists the various flags of the command Table 28 ibv send bw Flags and Options PS Listens Listens on connect to port lt port gt default 18515 to port
40. e click the desired HCA parameter and modify it Repeat this step for all the parameters you wish to modify Step 6 Close the Registry editor after completing all modifications Step 7 Open Device Manager and expand the correct InfiniBand Channel Adapters entry 1 e the adapter with modified parameters Step 8 Right click the expanded HCA entry and left click Disable This disables the device Step 9 Right click the expanded HCA entry and left click Enable This re enables the device For the changes to take effect you must disable and re enable the HCA Steps 8and 9 above 4 1 2 Modifying IPoIB Configuration To modify the IPoIB configuration after installation perform the following steps Step 1 Open Device Manager and expand Network Adapters in the device display pane Step 2 Right click the Mellanox IPoIB Adapter entry and left click Properties 26 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Step 3 Click the Advanced tab and modify the desired properties The IPoIB network interface is automatically restarted once you finish modifying IPoIB parameters Consequently it might affect any running traffic 4 1 3 Displaying Adapter Related Information To display a summary of network adapter software firmware and hardware related information such as driver version firmware version bus interface adapter identity and network port link information perform the following steps
41. e distributed for Windows Server 2008 x86 and x64 Windows Server 2008 R2 x64 and Windows 7 x86 and x64 Mellanox WinOF VPI is composed of several software modules that contain an InfiniBand and Ethernet driver The Mellanox WinOF VPI driver supports Infiniband and 10GB Ethernet ports The port type is determined upon boot based on card s capability and user setting 2 1 Mellanox VPI Package Contents The Mellanox WinOF for Windows package contains the following components e Core and ULPs B network adapter cards low level drivers mlx4 IB Access Layer IBAL Ethernet driver ETH IP over InfiniBand IPoIB Upper Layer Protocols ULPs NetworkDirect ND e Utilities SW Development Kit SDK e Documentation 2 2 Hardware and Software Requirements Administrator privileges on your machine s Disk Space for installation 100MB 2 3 Supported Network Adapter Cards and Firmware Versions Mellanox WinOF VPI Rev 3 0 0 supports the following Mellanox network adapter cards IB e ConnectX 2 EN IB SDR DDR QDR fw 25408 Rev 2 9 1000 VPI Ethernet e ConnectX ConnectX 2 ConnectX EN IB SDR DDR QDR fw 25408 Rev 2 9 1000 ConnectX 3 FDR SDR QDR fw 25408 Rev 2 10 0000 and higher Mellanox Technologies 9 J Rev 3 0 0 Introduction L 2 4 Managing Firmware We recommend upgrading ConnectX and ConnectX 2 adapter cards to firmware 2 9 1000 or higher to enable improved functionali
42. ed Ethernet frames into separate buffers The feature is disabled by default and can be enabled in the Advanced tab Performance Options from the Properties sheet For further information please refer to the MSDN library http msdn microsoft com en us library windows hardware ff553723 v VS 85 aspx 14 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 3 4 Receive Side Scaling RSS Mellanox WinOF Rev 3 0 0 IPoIB and Ethernet drivers use NDIS 6 2 new RSS capabilities The main changes are e Supports unlimited number of processors previously 64 Individual network adapter RSS configuration usage To set the RSS capability for individual adapter instead of global setting and to improve RSS on Windows 2008 R2 and Windows 7 set the registry keys listed in the table below Table 4 Registry Keys Setting HKLMISYSTEMVCurrentControlSetiControl Clas XXXXX72 XXX network Maximum number of CPUs allotted Sets the desired maxi adapter number MaxRSSProcessors mum number of processors for each interface The number can be different for each interface Note Restart the network adapter when you change this regis try key HKLM SYSTEM CurrentControlSet Control Class XXXXX72 XXX network Base CPU number Sets the desired base CPU number for each adapter number RssBaseProcNumber interface The number can be different for each interface This allows partitioning of CPUs across network adapters N
43. eme II GigE NDIS BD Client 2 E Mellanox Connects MT25418 DDR Channel Adapter A Er Mellanox Connects 10Gb Ethernet Adapter E Mellanox Connects 10Gb Ethernet Adapter 2 Ela Other devices y Base System Device B Ports COM amp LPT H Processors ERA Storage contrallers HE System devices E E Universal Serial Bus controllers D M s A Step 2 Right click a Mellanox ConnectX 10Gb Ethernet adapter under Network adapters list and left click Properties Select the LBFO tab from the Properties sheet It is not recommended to open the Properties sheet of more than one adapter simultane ously Mellanox Technologies 19 J Rev 3 0 0 Driver Features Mellanox Connect 10Gb Ethernet Adapter 3 Properties ES General Information Advanced Performance VLAN LBFO Driver Details MA Load Balancing and Fall O er LBFO Sethnge Mellanox Bundle Mame Ao Bundle Type Fault Tolerance a Fault Tolerance Switch Fault Tolerance Send Load Balancing Load Balancing Send amp Receive Adaptive Load Balancing Primary v Failback to P Select the adapta lE Dinamic Link Aggregation S02 3ad ipi mecs O Melanos Connect 10Gb Ethernet Adapter 3 O Melanos Comectz 10Gb Ethernet Adapter 4 Commit Cancel LBFGO stands for Load Balancing and Fail Over The administrator can configure a bundle of adapters and associate up t
44. eneral Information Diagnostics Advanced Performance Advanced Adapter Settings Mellanox Settings Value FOO Receive Butters Send Butters Flow Control Options Performance Options Offload Options YMO Options RoCE Options Use Default For All Maximum Frame Size MTU set the maximum size of a frame or packet that can be sent over the wire This is also known as the maximum transmission unit MTU The MTU of a network can have a large impact on performance The range of valid MTU values is 600 through Sent H TE All devices on the same physical network or an the same logical nete ark if using VLAN tagging must have the same MTU Cancel Help Step 3 Modify configuration parameters to suit your system Please note the following a For help on a specific parameter option check the help window at the bottom of the dialog b If you select one of the entries Offload Options Performance Options or Flow Con trol Options you ll need to click the Properties button to modify parameters via a pop up dialog See example in the two figures below c A Use Default for All button appears on the Advanced dialog Click this button to set all entries and their sub entries to the Mellanox Ethernet driver default values You will be prompted to approve this action 30 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 d If you press Cancel then the last settings wi
45. erfor mance than ib send bw in Nahelem systems 7 3 11 1 Synopsys ibv write bw i b port ib port d ib device c onnection type RC UC UD m tu mtu size Gtia mes sagets ze Se pc MXN n iteration num p ort bibe To eT hell ine satz s fe valente seals u gp timeout S 1 sl type x gid index Pevenus sere e o wedi tse speakecalel y F CPU freq fail g num of posts q num of gps b idirectional a 11 V ersion 7 3 11 2 Options The table below lists the various flags of the command Table 30 ibv write bw Flags and Options Listens on connect to port lt port gt default 18515 82 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 30 ibv_write_bw Flags and Options Flag Description The maximum size of message to be sent in inline mode default 0 Generate Cqe only after lt cq mod gt completion 7 3 12 ibv_write_lat This is a more advanced version of ib write lat and contains more flags and featurs than the older version and also improved algorithms ibv write lat calculats the latency of RDMA write opera tion of message sizeB between a pair of machines One acts as a server and the other as a client They perform a ping pong benchmark on which one side RDMA writes to the other side memory only after the other side wrote on his memory Each of the sides samples the CPU clock each time they write to the other side memory to calculate latency Mel
46. ering them to the designated destina tion RSS can significantly improve the number of transactions the number of connections per second and the network throughput This parameter can be set to one of the following values 34 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Enabled default Set RSS Mode Disabled The hardware is configured once to use the Toeplitz hash function and the indirection table is never changed d IOAT is not used while in RSS mode 5 Receive Completion Method Sets the completion methods of the received packets and can affect network throughput and CPU utiliza tion Polling Method Increases the CPU utilization as the system polls the received rings for the incoming packets However it may increase the network performance as the incoming packet is handled faster nterrupt Method Optimizes the CPU as it uses interrupts for handling incoming messages However in certain scenarios it can decrease the network throughput Adaptive Default Settings A combination of the interrupt and polling methods dynamically depending on traffic type and network usage Choosing a different setting may improve network and or system performance in certain configurations Interrupt Moderation RX Packet Count Number of packets that need to be received before an interrupt is generated on the receive side default 5 Interrupt Moderation RX Packet Time Maximum elapsed time
47. extended speeds 52 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Examples 1 Query PortInfo by LID with port modifier Mellanox Technologies 53 J Rev 3 0 0 InfiniBand Fabric 2 Query SwitchInfo by GUID o4 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 3 Query Nodelnfo by direct route 7 2 5 perfquery Queries InfiniBand ports performance and error counters Optionally it displays aggregated coun ters for all ports of a node It can also reset counters after reading them or simply reset them 7 2 5 1 Applicable Hardware All InfiniBand devices Mellanox Technologies 55 J Rev 3 0 0 InfiniBand Fabric 7 2 5 2 Synopsys pesrauerv el ied eG ames a T at D aee Sl Ove SE e E o a e o sd a a SE i C lt caname gt e e ERE a Ee E quie port reset ED The table below lists the various flags of the command Table 10 perfquery Flags and Options helpa helpa Print Printihehelp men A menu Raise the IB debug level May be used several times for higher debug levels ddd or d d d Use GUID address argument In most cases 1t 1s the Port GUID Example 0x08f1040023 56 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 10 perfquery Flags and Options Flag Description Examples FA perfquery r 32 read performance counters and reset perfquery e r 321 read ex
48. g the registry incorrectly might lead to serious problems A including the loss of data system hang and you may need to reinstall Windows As such it is recommended to back up the registry on your system before implementing recommendations included in this document If the modifications you apply lead to seri ous problems you will be able to restore the original registry state For more details about backing up and restoring the registry please visit www microsoft com 36 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Mellanox Technologies 37 J Rev 3 0 0 OpenSM Subnet Manager 6 OpenSM Subnet Manager OpenSM v3 3 11 is an InfiniBand Subnet Manager For Mellanox WinOF VPI to operate OpenSM must be running on at least one host machine in the InfiniBand cluster gt OpenSM can either run as a Windows service which starts automatically during boot or can be started manually from the following directory installation directory gt tools Please use the embedded OpenSM in the WinOF package for testing purpose and small cluster Otherwise we recommend using OpenSM from FabricIT EFMTM or UFM M To start OpenSM automatically please perform the following Right click on My computer and select Manage Go to Services and Applications and select Services Right click OpenSM and select Properties Change Startup type to Automatic p um X 39 x Change service to start mode OpenSM as a se
49. hieve the best performance for Windows using 10GigE adapters you may need to modify some of the Windows registries Registry Tuning The registry entries that may be added changed by this General Tuning procedure are Under HKEY LOCAL MACHINE SYSTEM CurrentControlSet Services Tepip Parameters Disable TCP selective acks option for better cpu utilization SackOpts type REG DWORD value set to 0 Under HKEY LOCAL MACHINE SYSTEM CurrentControlSet Services AFD Parameters Enable fast datagram sending for UDP traffic FastSendDatagramThreshold type REG DWORD value set to 64K Under HKEY LOCAL MACHINE SSYSTEM CurrentControlSet Services Ndis Parameters e Set RSS parameters RssBaseCpu type REG DWORD value set to 1 Enable RSS Enabling Receive Side Scaling RSS is performed by means of the following command netsh int tcp set global rss enabled Tuning the Network Adapter The Network Adapter tuning can be performed either during installation by modifying some of Windows registries as explained in section Registry Tuning on page 32 or can be set post instal lation manually To improve the network adapter performance activate the performance tuning tool as follows I Start the Device Manager open a command line window and enter devmgmt msc Open Network Adapters Select Mellanox Ethernet adapter right click and select Properties Select the Performance tab vv emn pire cbe Click on Ge
50. hines One acts as a server and the other as a client They perform a ping pong benchmark on which one side RDMA reads the memory of the other side only after the other side have read his memory Each of the sides samples the CPU clock each time they read the other side memory in order to calculate latency Read is availible only in RC connection mode as specified in IB spec 72 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 3 2 1 Synopsys Tee e a E sie cT a le e SINE a size n iteration num p ort PDT port o uts outstanding reads a 11 V ersion C report cycles H report histogram U report unsorted 7 3 2 2 Options The table below lists the various flags of the command Table 21 ib read lat Flags and Options PS Listens Listens on connect to port lt port default 18515 to port lt port gt Listens on connect to port lt port default 18515 18515 7 3 3 ib send bw ib send bw calculats the BW of SEND between a pair of machines One acts as a server and the other as a client The server receive packets from the client and they both calculate the throughput of the operation The test supports features such as Bidirectional on which they both send and receive at the same time change of mtu size tx size number of iteration message size and more Using the a provides results for all message sizes Mellanox Technologies 73 J Rev 3 0 0 InfiniBand Fabric 7
51. ion file specifying the IB fabric topology in user given names For diagnostic tools to fully support the topology file the user may need to provide the local sys tem name if the local hostname is not used in the topology file To specify a topology file to a diagnostic tool use one of the following two options 1 On the command line specify the file name using the option t topology file name gt 2 Define the environment variable IBDIAG TOPO FILE To specify the local system name to an diagnostic tool use one of the following two options I On the command line specify the system name using the option s lt local system name gt 2 Define the environment variable IBDIAG SYS NAME 7 2 1 2 IB Interface Definition The diagnostic tools installed on a machine connect to the IB fabric by means of an HCA port through which they send MADs To specify this port to an IB diagnostic tool use one of the follow ing options I On the command line specify the port number using the option p local port number gt see below 2 Define the environment variable IBDIAG PORT NUM In case more than one HCA device 1s installed on the local machine it is necessary to specify the device s index to the tool as well For this use on of the following options 1 On the command line specify the index of the local device using the following option i index of local device gt 2 Define the environment variable IBDIA
52. ion num p ort PDT port u gp timeout S 1 sl type x gid index e vents use events cree sans direction pea RTT tersion 7 3 7 2 Options The table below lists the various flags of the command Table 26 ibv read bw Flags and Options p Listens Listens on eomnect to port lt port gt default 18515 to port lt port gt Listens on eomnect to port lt port gt default 18515 18515 F CPU freq Generate Cqe only after lt cq mod gt completion 7 3 8 ibv read lat This is a more advanced version of ib read lat and contains more flags and featurs than the older version and also improved algorithms ibv read lat calculats the latency of RDMA read operation of message sizeB between a pair of machines One acts as a server and the other as a client 78 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 They perform a ping pong benchmark on which one side RDMA reads the memory of the other side only after the other side have read his memory Each of the sides samples the CPU clock each time they read the other side memory to calculate latency Read is availible only in RC connection mode as specified in InfiniBand spec 7 3 8 1 Synopsys lov Pead slat E e e a e a e les ze mess e E E e Ei ee e Eo e t AM E e name x gid index e ODDS OBS outstanding reads e vents use events p ort a sash ie Sy e cam C report cycles H report histogram U report unsorted FA OB aris
53. ity between IB nodes On exit IP ping like output is shown ibping is run as client server however the default is to run it as a client Note also that in addition to ibping a default server is implemented within the kernel 7 2 6 1 Synopsys e So erase cd T petere einen eee ostrea sonia Anson Mero a sos E a Alco cos cun S erver h elp dest lid guid 7 2 6 2 Options The table below lists the various flags of the command Table 11 ibping Flags and Options PS c num Stops Stops after count packets 0 count Stops after count packets 0 f flood Floods destination send packets back to back without delay Mellanox Technologies 59 J Rev 3 0 0 InfiniBand Fabric Table 11 ibping Flags and Options Flag Description 7 2 7 ibnetdiscover ibnetdiscover performs IB subnet discovery and outputs a readable topology file GUIDs node types and port numbers are displayed as well as port LIDs and NodeDescriptions All nodes and links are displayed full topology Optionally this utility can be used to list the current connected nodes by node type The output is printed to standard output unless a topology file is specified 7 2 7 1 Synopsys nerds corea E e e e a a Eso ls SOULS Alice micas TER E e a e E a c ae a ae oa e V ersion outstanding smps o lt val gt u sage node name map lt node name map gt cache lt filename gt load cache lt filename gt p orts m ax h
54. lanox Technologies 83 J Rev 3 0 0 InfiniBand Fabric 7 3 12 1 Synopsys ibv write lat i b port ib port c onnection type RC UC UD m tu mtu size s ize Messagersizel Student sizes e ss a ME u qp timeout S L sl type d ib device name x gid index n iteration num Ea a os V ersion C report cycles H report histogram U report unsorted 7 3 12 2 Options The table below lists the various flags of the command Table 31 ibv_write_lat Flags and Options p por spor gt Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 F CPU freq The maximum size of message to be sent in inline mode default 0 84 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 8 Software Development Kit Software Development Kit SDK a set of development tools that allows the creation of InfiniBand applications for MLNX VPI software package The SDK package contains header files libraries and code examples To open the SDK package you must run the sdk exe file and get the complete list of files SDK package can be found under installation directory VBNSDK Mellanox Technologies 85 Rev 3 0 0 Troubleshooting 9 Troubleshooting 9 1 InfiniBand Troubleshooting Issue 1 The IB interfaces is not up after the first reboot after the installation process is com pleted
55. ll be restored Mellanox Connectx 10Gb Ethernet Adapter Properties VLAM LEFO Driver Details Power Management General Information Diagnostics Advanced Performance Advanced Adapter Settings Mellanox Settings Jumbo Packet Properties Receive Bulters Send Butters Flow Control Options Performance Options Offload Options Settings Value 14 Checksum Offload ITs amp Hx Enabled Use Default for Al TCP UDP IPV4 Checksum Offload TCP LUDP IFE Checksum Offload Large Send Offload L50 YMO Options RoCE Options Offload Options Allows you to specify which TCPAP offload settings are handled by the adapter rather than the aperating system Enabling offloading services increases transmission performance The performance increases because offload tasks such as Use Default checksum calculations are performed by adapter hardware rather than the operating system and therefore with lover latency and also CPU resources become more available for other tasks OK Cancel Help IPv4 checksums Offload This option enables the adapter to compute IP 3 checksum upon transmit and or receive instead of the CPU By default this parameter is enabled for transmit and receive IP 4 checksum offloading OF Cancel Mellanox Technologies 31 J Rev 3 0 0 Performance D 5 1 5 1 1 Performance General Performance Optimization and Tuning To ac
56. logical state gt 7 2 1 5 OPTIONS C eseou Min Min number of packets to be sent across each link default 10 of Min number of packets to be sent across each link default 10 to be sent across each link default 10 i lt dev index gt Specifies the index of the device of the port used to connect to the IB fabric in case of multiple devices on the local system skip lt skip option s gt Skip the executions of the selected checks Skip options one or more can be specified dup guids zero guids pm logical state part ipoib all 7 2 1 6 Output Files Table 5 ibdiagnet of ibutils Output Files 42 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 5 ibdiagnet of ibutils Output Files ibdiagnet mcg A dump of the multicast groups their properties and member host ports ibdiagnet db A dump of the internal subnet database This file can be loaded in later runs using the load db option In addition to generating the files above the discovery phase also checks for duplicate node port GUIDs in the IB fabric If such an error 1s detected it is displayed on the standard output After the discovery phase is completed directed route packets are sent multiple times according to the c option to detect possible problematic paths on which packets may be lost Such paths are explored and a report of the suspected bad links is displayed on the standard output After scanning the fabric if the
57. ls for the network adapter ports To enable Auto Sensing please choose AUTO If the MIC supports 4uto Sensing the port protocol will be determined according to the link type Otherwise according to the NIC s Hardware Defaults port type Step 3 In this step you can perform the following functions X Choose HW Defaults option If you choose the HW Defaults option the port protocols will be determine according to the NIC s hardware default values X Choose the desired port protocol for the available port s If you choose IB or ETH both ends of the connection must be of the same type IB or ETH X Enable Auto Sensing by checking the AUTO checkbox If the NIC does not support Auto Sensing the AUTO option will be grayed out X Enable InfiniBand application over Ethernet by checking the RoCE checkbox RoCE can be selected on the second port Port2 only if Port 1 1s set as either IB or Ethernet with RoCE enabled X Installing Network Direct ND by checking the Active ND checkbox gt 5 IB must be always the first port in Port 1 If you choose ETH as your first port in Port 1 then the second port in Port2 can be only ETH 3 6 Load Balancing Fail Over LBFO and VLAN Mellanox Technologies 17 J Rev 3 0 0 Driver Features 3 6 1 Adapter Teaming Adapter teaming can group a group of ports inside a network adapters or a number of physical net work adapters into virtual adapters that provide the fault tole
58. lt port gt Listens on connect to port lt port gt default 18515 18515 F CPU freq The maximum size of message to be sent in inline mode default 0 80 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 28 ibv_send_bw Flags and Options Flag Description g mcg lt num of qps gt Sends messages to multicast group with num of qps gt qps attached to it M MGID lt multicast_gid gt In case of multicast uses multicast gid gt as the group MGID The format must be 255 1 X X X X X X X X X KS X X X where X is a vlaue within 0 255 Connect QPs with rdma cm and run test on those QPs Communicate with rdma cm module to exchange data use regular QPs 7 3 10 ibv send lat This is a more advanced version of ib send lat and contains more flags and featurs than the older version and also improved algorithms ibv send lat calculats the latency of sending a packet in message sizeB between a pair of machines One acts as a server and the other as a client They perform a ping pong benchmark on which you send packet only after you receive one Each of the sides samples the CPU clock each time they receive a send packet in order to calculate the latency 7 3 10 1 Synopsys ibv send lat i b port ib port c onnection type RC UC UD d ib device name m tu mtu size s ize message size ee MTM Mess Ze esse u gp timeout S L sl type x gid index e events
59. neral Tuning button 32 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Clicking the General Tuning button will change several registry entries described below and will check for system services that may decrease network performance It will also generate a log including the applied changes Users can view this log to restore the previous values The log path is SHOMEDRIVESNWindowsNSystem32NLogFilesNPerformanceTunning log This tuning is required to be performed only once after the installation is completed and on one adapter only as long as these entries are not changed directly in the registry or by some other installation or script Please note that a reboot may be required for the changes to take effect 5 2 Application Specific Optimization and Tuning 5 2 1 Ethernet Performance Tuning The user can configure the Ethernet adapter by setting some registry keys The registry keys may affect Ethernet performance To improve performance activate the performance tuning tool as follows start the Device Manager open a command line window and enter devmgmt msc Open Network Adapters l 2 3 Right click the relevant Ethernet adapter and select Properties 4 Select the Advanced tab and select Performance Options 9 Modify performance parameters properties as desired 5 2 1 1 Performance Known Issues e On Intel I OAT supported systems it is highly recommended to install
60. ng channels the traffic between the other team adapter 6 Dynamic Link Aggregation 802 3ad Provides dynamic link aggregation allowing creation of one or more channel groups using same speed or mixed speed server adapters 7 Static Link Aggregation 802 3ad Provides increased transmission and reception throughput in a team comprised of two to eight adapter ports through static configuration If the switch connected to the HCA supports 802 3ad the recommended setting is teaming mode 6 3 6 2 Creating a Load Balancing and Fail Over LBFO Bundle LBFO is used to balance the workload of packet transfers by distributing the workload over a bun dle of network instances and to set a secondary network instance to take over packet indications and information requests 1f the primary network instance fails 18 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 The following steps describe the process of creating an LBFO bundle Step 1 Display the Device Manager ont Device Manager ml File Action View Help gt Fa is Fl gay Sw263 fg Computer i Sy Disk drives Display adapters S5 DYDICD ROM drives sem Floppy drive controllers d Human Interface Devices IDE ATA ATAPI controllers M IEEE 1394 Bus hast controllers X Keyboards A Mice and other pointing devices L Monitors Network adapters aes is Broadcom BCMS708C Netxtreme II GigE NDIS YBO Client 2 eA Er Broadcom BCMS708C Netktr
61. o Mellanox Connects adapters to this bundle LOFO should be used 0 increase the system reliability upon a link failure and to balance tha workload os limo di Am cocer ta Step 3 The LBFO dialog enables creating modifying or removing a bundle 4 Only Mellanox Technologies adapters can be part of the LBFO To create a new bundle perform the following Click the Create button Enter a unique bundle name 1 2 3 Select a bundle type 4 Select the adapters to be included in the bundle that have not been associated with a VLAN 5 Optional Select Primary Adapter An active passive scenario used for data transfer of link disconnecting In such scenario the system uses one of the other interfaces When the primary link comes up the LBFO interface returns to transfer data using the primary interface If the primary adapter is not selected the primary interface 1s selected randomly 6 Optional Failback to Primary 7 Click the Commit button 20 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Mellanox Connect 10Gb Ethernet Adapter 3 Properties gt General Information Advanced Performance VLAM LBFO Driver Details Load Balancing and F ail D ver LBFO Sethngs Mellanox Bundle Mame l Bundle Type Fault Talerance Fault Tolerance Switch Fault Tolerance Send Load Balancing Load Balancing Send amp Heceree Adaptive Load Balan
62. o sminfo 32 show sminfo of lid 32 sminfo G 0x8f1040023 same but using guid address 7 2 10 ibclearerrors ibclearerrors is a script which clears the PMA error counters in PortCounters by either waking the IB subnet topology or using an already saved topology file 7 2 10 1 Synopsys belearertors mI no colon T lt topology file gt s e camnane Pica port timeout HIDE ms 7 2 10 2 Options The table below lists the various flags of the command Table 15 ibclearerrors Flags and Options C ca name Use the specified ca name P ca port gt Use the specified ca port t timeout ms Override the default timeout for the solicited mads 7 2 11 ibstat ibstat is a binary which displays basic information obtained from the local IB driver Output includes LID SMLID port state link width active and port physical state 7 2 11 1 Synopsys star ebrio sor pone ers On sii ccanhane DOT num 7 2 11 2 Options The table below lists the various flags of the command Most OpenIB diagnostics take the following common flags The exact list of supported flags per utility can be found in the usage message and can be shown using the util name h syntax Table 16 ibstat Flags and Options list of cas List all IB devices Mellanox Technologies 67 J Rev 3 0 0 InfiniBand Fabric Table 16 ibstat Flags and Options usage u usage message Examples ibstat display status of all port
63. o be the source EI NN 1 lt dev index gt Specifies the index of the device of the port used to connect to the IB fabric in case of multiple devices on the local system AAA 44 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 2 1 10 Output Files Table 6 ibdiagpath Output Files ibdiagpath log A dump of all the application reports generated according to the provided flags ibdiagnet pm A dump of the Performance Counters values of the fabric links 7 2 1 11 ERROR CODES 1 The path traced is un healthy 2 Failed to parse command line options 3 More then 64 hops are required for traversing the local port to the Source port and then to the Destination port 4 Unable to traverse the LFT data from source to destination 5 Failed to use Topology File 6 Failed to load required Package 7 2 2 ibportstate Enables querying the logical link and physical port states of an InfiniBand port It also allows adjusting the link speed that is enabled on any InfiniBand port If the queried port is a swich port then ibportstate can be used to disable enable or reset the port e validate the port s link width and speed against the peer port 7 2 2 1 Applicable Hardware All InfiniBand devices 7 2 2 2 Synopsis ibportstate d e v V D 1 6 s lt smlid gt e lt cautiame P lt calpore gt Su eres time cuts lt dest dr path lid guid gt lt portnum gt
64. ops h elp lt topology file gt 7 2 7 2 Options The table below lists the various flags of the command Most OpenIB diagnostics take the following common flags The exact list of supported flags per utility can be found in the usage message and can be shown using the util name h syntax Table 12 ibnetdiscover Flags and Options list List of connected nodes g grouping Show grouping Grouping correlates IB nodes by different vendor specific schemes It may also show the switch external ports correspondence List of connected switches S Switch list R Router list List of connected routers 60 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Table 12 ibnetdiscover Flags and Options Flag Description Show progress information during discovery node name map lt node name map gt Specify a node name map The node name map file maps GUIDs to more user friendly names See Topology File Format on page 61 cache lt filename gt Cache the ibnetdiscover network data in the specified filename This cache may be used by other tools for later analysis load cache lt filename gt Load and use the cached ibnetdiscover data stored in the specified filename May be useful for outputting and learning about other fabrics or a previous state of a fabric diff lt filename gt Load cached ibnetdiscover data and do a diff comparison to the current network or another cach
65. ormance tests 3 1 3 Reading Port Counters Statistics RoCE port statistics are not shown in the Windows network counters associated with Etherent interface It is possible to read port statistics in the same way it is done for regular InfiniBand ports The information is available by running vstat c 3 1 4 Setting RoCE To set the RoCE please perform the following Step 1 Open Device Manager and expand System devices display pane Step 2 Right click the Mellanox ConnectX VPI MT26428 PCIe 2 0 5GT s IB QDR 10GigE Net work Adapter entry and left click Properties Step 3 Click the Port Protocol tab and check RoCE check box Step 4 Click OK It s will cause to driver restart 3 1 5 Setting RoCE MTU Ethernet packet uses the general MTU value whereas the RoCE packet uses the RoCE MTU All devices that run the RoCE protocol must have the same MTU otherwise packets larger than the minimum MTU are dropped and not transferred When RoCE is enabled you can configure the MTU that can be sent by the RoCE protocol e The valid RoCE MTU values are 256 512 1024 2048 When using MTU 2048 the administrator should configure the switches to support MTU 2048 or higher e The default MTU is 1024 To set the RoCE MTU please perform the following Step 1 Open Device Manager and expand Network Adapters in the device display pane Step 2 Right click the Mellanox ConnectX 10Gb Ethernet Adapter entry and left click Properties Ste
66. ort gt Firmware Download 10 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 To identify your adapter card please visit http www mellanox com gt Support gt Firmware Downloads gt Identifying Adapter Cards 2 4 3 Updating Adapter Card Firmware Using a card specific binary firmware image file enter the following command gt ELE dede vice id pel crll ama ge mane db burn For additional details please check the MFT user s manual under http www mellanox com gt Products gt Adapter IB VPI SW Mellanox Technologies 11 J Rev 3 0 0 Driver Features 3 Driver Features The Mellanox VPI WinOF driver release introduces the following capabilities e One or two ports e Upto 16 Rx queues per port e Rx steering mode RSS Hardware Tx Rx checksum calculation Large Send Offload 1 e TCP Segmentation Offload Hardware multicast filtering Adaptive interrupt moderation e MSI X support only on Windows Server 2008 and higher Auto Sensing e RoCE Ethernet Only e High Availability HA between ports and Mellanox NICs Load Balancing between ports and Mellanox NICs HW VLAN filtering e Hyper V Header Data Split For the complete list of Ethernet and InfiniBand Know Issues and Limitation see MLNX WinVPI ReleaseNotes txt 3 1 RDMA over Converged Ethernet RoCE 3 1 1 RoCE Overview Remote Direct Memory Access RDMA is the remote memory management capability that allows
67. ort 0 lid 6 Imc 0 22 H 0008 10403961354 1 8 10403961355 MT23108 InfiniHost Mellanox Technolo gies lid 4 4xSDR 10 0008 10400410015 1 SW 61B4 Voltaire lid 3 4xSDR 8 H 0008 10403960558 2 8 1040396055a MT23108 InfiniHost Mellanox Technolo gies lid 14 4xSDR 6 0008 10400410015 3 SW 61B4 Voltaire lid 3 4xSDR 12 H 0008 10403960558 1 8 10403960559 MT23108 InfiniHost Mellanox Technolo gies lid 10 4xSDR vendid 0x8f1 devid 0x5a05 switchguid 0x8f10400410015 8 10400410015 Switch 8 S 0008f10400410015 SW 6IBA Voltaire base port 0 lid 3 Imc 0 6 H 0008 10403960984 1 8 10403960985 MT23108 InfiniHost Mellanox Technolo gies lid 16 4xSDR 4 H 005442b100004900 1 5442b100004901 MT23108 InfiniHost Mellanox Technolo gies lid 12 4xSDR 1 S 005442ba00003080 10 ISR9024 Voltaire lid 6 1xSDR 5 S 005442ba00003080 6 ISR9024 Voltaire lid 6 4xSDR 62 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Node Name Map File Format The node name map is used to specify user friendly names for nodes in the output GUIDs are used to perform the lookup Mellanox Technologies 63 J Rev 3 0 0 InfiniBand Fabric Example IB1 Line cards 0x0008f1 0x0008f1 0x0008f1 0x0008f1 0x0008f1 Spines 0x0008f1 0x0008f1 0x0008f1 0x0008f1 0x0008
68. orts COM amp LPT Processors Storage controllers 1 System devices Universal Serial Bus controllers Mellanox Technologies 21 J Rev 3 0 0 Driver Features To modify an existing bundle perform the following 1 Select the desired bundle and click Modify 2 Modify the bundle name its type and or the participating adapters in the bundle 3 Click the Commit button To remove an existing bundle select the desired bundle and click Remove You will be prompted to approve this action Notes on this step a Each adapter that participates in a bundle has two properties X Status Connected Disconnected Disabled X Role Active or Backup b Each network adapter that is added or removed from a bundle gets refreshed 1 e disabled then enabled This may cause a temporary loss of connection to the adapter c In case a bundle loses one or more network adapters by a create or modify oper ation the remaining adapters in the bundle are automatically notified of the change 3 6 3 Creating a Port VLAN You can create a Port VLAN either on a physical Mellanox ConnectX EN adapter or a virtual bun dle team The following steps describe how to create a port VLAN Step 1 Display the Device Manager Device Manager iz File Action View Help e BH qh Computer ics Disk drives Ae Display adapters 5 DVDICD ROM drives eS Floppy drive controllers d Human Interface Devices cg IDE AT
69. ot possible to bind to this port until all the virtual adapters gt associated with it are removed OF When using a VLAN the network address is configured using the VLAN ID There A fore the VLAN ID on both ends of the connection must be the same Pe Step 4 Verify the new VLAN s by opening the Device Manager window or the Network Connections window The newly created VLAN will be displayed in the following format Mellanox Virtual Miniport Driver VLAN lt name gt 24 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Device Manager iof File Action View Help eig li mE H B E B de H B E E B 1 Computer Disk drives E Display adapters y DVD CD ROM drives cx Floppy drive controllers i 3j Human InterFace Devices C IDE ATA ATAPI controllers g IEEE 1394 Bus host controllers zm Keyboards PI Mice and ather pointing devices IS Monitors L1 Metwork adapters be a Broadcom BCMS708C NetXtreme II GigE DIS YBO Client M ue Broadcom BCMS708C MeEXEreme II GigE NDIS VED Client 2 be amp Melanos Connect MT25418 DDR Channel Adapter EX Mellanox Connect 10Gb Ethernet Adapter EX Mellanox Connect 10Gb Ethernet Adapter 2 i Mellanox virtual Miniport Driver Team 4 CE Melinox virtual Miniport Driver VLAN 1 E js Other devices im Base System Device FT Ports COM amp LPT H Processors lt gt Storage controllers ge System devices H y Universal Serial Bu
70. ote Restart the network adapter when you change this regis try key HKLMISYSTEMVCurrentControlSetiControl Class XXXXX72 XXX lt network NUMA node affinitization adapter number NumaNodeID HKLMISYSTEMVCurrentControlSetiControl Clas XXX XX72 XXX lt network Sets the RSS base processor group for systems with more than adapter number gt RssBaseProcGroup 64 processors 3 5 Port Configuration After MLNX VPI installation it is possible to modify the network protocol that runs on each port of VPI adapter cards Each port can be set to run as InfiniBand Ethernet or Auto Sensing 3 5 1 Auto Sensing Auto Sensing enables the NIC to automatically sense the link type InfiniBand or Ethernet based on the cable connected to the port and load the appropriate driver stack InfiniBand or Ethernet For example if the first port is connected to an InfiniBand switch and the second to Ethernet switch the NIC will automatically load the first port as InfiniBand and the second as Ethernet Auto Sensing is performed only when rebooting the machine or after disabling enabling the mlx4 bus interface from the Device Manager Hence if you replace cables during the runtime the NIC will not perform Auto Sensing For further information on how to configure it please refer to Section 3 5 2 Port Protocol Con figuration on page 16 Mellanox Technologies 15 J Rev 3 0 0 Driver Features 3 5 2 Port Protocol Configuration Ste
71. ow The following is the flow s description c create an inventory file with all nodes ports and paths a run all validation tests expecting an input inventory v only validate the given inventory file s run service registration deregistration and lease test e run event forwarding test f flood the SA with queries according to the stress mode m multicast flow q QoS info dump VLArb and SLtoVL tables t run trap 64 65 flow this flow requires running of external tool default is all flows except QoS This option specifies the wait time for trap 64 65 in seconds It is used only when running f t the trap 64 65 flow default to 10 sec This option specifies a debug option These options are not normally needed The number follow ing d selects the debug option to enable as follows OPT Description d1 Force single threaded dispatching d2 Force log flushing after each log message d3 Disable multicast support m max lid This option specifies the maximal LID number to be searched for during inventory file build default to 100 This option specifies the local port GUID value with which OpenSM should bind OpenSM may be bound to 1 port at a time If GUID given is 0 OpenSM displays a list of possible port GUIDs and waits for user input Without g OpenSM trys to use the default port This option displays a menu of possible local port GUID values with which osmtest could bind i Inventory This
72. p 1 Display the Device Manager and expand Network adapters Device Manager E Ed Network adapters i Broadcom BCMS703C Netxtreme Il GigE NDIS VED Chent iF Broadcom BCMS709C Netitreme II GE NDIS VBD Clen 42 a 1 Ports eem amp LPT aD Processors a IP Security Devices 8 lt gt Storage controllers E AM System devices A ACPI Fixed Feature Button A Broadcom BCMS709C Netxtreme II GigE Mi Broadcom BCMSTUSC Netitreme II Gage A Composite Bus Enumerator Direct memory access controller M Intel 5520 5500 Physical and Link Layer Registers Port I 3427 A Intel R 5520 8500 Routing and Protocol Layer Register Port 1 3428 jE Intel R 5520 8500 458 1 0 Hub Control Status and RAS Registers 3423 A Intel R 5520 5500 458 1 0 Hub GPIO and Scratch Pad Registers 3422 M intel 5520 5500 x58 1 0 Hub PCT Express Root Port 1 3408 M Intel R 5520 5500 453 1 0 Hub PCI Express Root Port 2 3409 AM Intel 5520 5500 x58 1 0 Hub PCT Express Root Port 3 3404 A Intel R 5520 5500 458 1 0 Hub PCI Express Root Port 7 HOE le Intel R 5520 5500 458 1 0 Hub System Management Registers 342 A Intel R 5520 5500 458 1 0 Hub Throttle Registers 3438 M Intel R 5520 5500 458 1 0 Hub to ESI Port 3406 AME Intel 5520 5500 458 Physical and Link Layer Registers Port 0 3425 M Intel R 5520J5500 458 Routing and Protocol Layer Registers Port 0 3426 Al Intel R 5520 5500 458 Trusted Execution Technology Registers 342F e Intel R 62
73. p 3 Click the Advanced tab and modify the desired properties Mellanox Technologies 13 J Rev 3 0 0 Driver Features Step 4 Select ROCE Options and click Properties to modify the settings as needed Step 5 Click OK 3 2 Hyper V with VMQ Mellanox WinOF Rev 3 0 0 includes a virtual machine queue VMQ interface to support Micro soft Hyper V network performance improvements and security enhancement VMQ interface supports e Classification of received packets by using the destination MAC address to route the packets to different receive queues e NIC ability to use DMA to transfer packets directly to a Hyper V child partition s shared memory e Scaling to multiple processors by processing packets for different virtual machines on different processors VMQ is disabled by default for Windows 2008 R2 5 3 2 1 Enabling Virtual Machine Queue on Windows 2008 R2 To enable VMQ on Windows 2008 R2 with 10 Gbps physical network adapters set the registry keys as follow Step 1 Open Command Prompt window Click Start gt All Programs Step 2 Click Accessories right click Command Prompt and then click Run as administrator Step 3 Type reg add HKEY LOCAL MACHINE SYSTEM CurrentControlSet ser vices VMSMP Parameters TenGigV mqEnabled t REG DWORD d 1 f Step 4 Click ENTER Step 5 Reboot 3 3 Header Data Split The header data split feature improves network performance by splitting the headers and data in receiv
74. performance A 4K MTU size improves performance for short messages since NDIS can coalesce a small message into a larger one Valid MTU values range is between 600 and 9600 5 All devices on the same physical network or on the same logical network must have the same MTU Receive Buffers The number of receive buffers default 1024 Send Buffers The number of sent buffers default 2048 Performance Options Configures parameters that can improve adapter performance Interrupt Moderation Moderates or delays the interrupts generation Hence optimizes network throughput and CPU utilization default Enabled When the interrupt moderation is enabled the system accumulates interrupts and sends a single inter rupt rather than a series of interrupts An interrupt is generated after receiving 5 packets or after 10ms from the first packet received It improves performance and reduces CPU load however it increases latency When the interrupt moderation is disabled the system generates an interrupt each time a packet is received or sent In this mode the CPU utilization data rates increase as the system handles a larger number of interrupts However the latency decreases as the packet is handled faster Receive Side Scaling RSS Mode Improves incoming packet processing performance RSS enables the adapter port to utilize the multiple CPUs in a multi core system for receiving incoming packets and ste
75. ported starting with WinOF VPI v2 1 1 6 Mellanox Technologies J WinOF VPI for Windows User Manual Rev 3 0 0 1 About this Manual 1 1 Scope The document describes WinOF Rev 3 0 0 features content and configuration Additionally this document provides information on various performance tools supplied with this version 1 2 Intended Audience This manual is intended for system administrators responsible for the installation configuration management and maintenance of the software and hardware of VPI InfiniBand Ethernet adapter cards It is also intended for application developers 1 3 Documentation Conventions Table 2 Documentation Conventions LX CON NN Gee mm T e RN LOI 2 Dessen TEN KEEN SS Emphasized words These are emphasized words lt text gt This is a note lt text gt May result in system instability Mellanox Technologies T Rev 3 0 0 About this Manual 1 3 1 Common Abbreviations and Acronyms Table 3 Abbreviations and Acronyms Abbreviation Acronym Whole Word Description Capital B is used to indicate size in bytes or multiples of bytes e g IKB 1024 bytes and IMB 1048576 bytes Small b is used to indicate size in bits or multiples of bits e g IKb 1024 bits a D 8 Mellanox Technologies J WinOF VPI for Windows User Manual Rev 3 0 0 2 Introduction This User Manual addresses the Mellanox WinOF VPI driver Rev 3 0 0 packag
76. r option is provided a full report of the fabric qualities is dis played This report includes e SM report e Number of nodes and systems Hop count information maximal hop count an example path and a hop count histo gram e All CA to CA paths traced Credit loop report e mgid mlid HCAs multicast group and report Partitions report IPoIB report In case the IB fabric includes only one CA then CA to CA paths are not gt reported Furthermore if a topology file is provided ibdiagnet uses the names af defined in it for the output reports Mellanox Technologies 43 J Rev 3 0 0 InfiniBand Fabric 7 2 1 7 ERROR CODES 1 Failed to fully discover the fabric 2 Failed to parse command line options 3 Failed to intract with IB fabric 4 Failed to use local device or local port 5 Failed to use Topology File 6 Failed to load requierd Package 7 2 1 8 SYNOPSYS ibdiagpath zm ehsresnamssjotcmamsoxediese opos ed domare duel 07 ni n e lt count gt o out Sm t lt topo file gt s lt sys name gt i lt dev index gt p lt port num gt pm pc P lt lt PM counter gt lt Trash Limit gt gt lw lt 1x 4x 12x gt 1s lt 2 515 10 gt sl lt service level gt 7 2 1 9 OPTIONS n lt src name dst name gt Names of the source and destination ports as defined in the topology file source may be omitted gt local port is assumed t
77. rance and load balancing functions Depending on the teaming mode one or more interfaces can be active The non active interfaces in a team are in a standby mode and will take over the network traffic in the event of a link failure in the active interfaces All of the active interfaces in a team participate in load balancing operations by sending and receiving a portion of the total network traffic 3 6 1 1 Teaming Bundle Modes I Fault Tolerance Provides automatic redundancy for the server s network connection If the primary adapter fails the secondary adapter currently in a standby mode takes over Fault Tolerance 1s the basis for each of the following teaming types and 1s inherent in all teaming modes 2 Switch Fault Tolerance Provides a failover relationship between two adapters when each adapter 1s connected to a separate switch 3 Send Load Balancing Provides load balancing of transmit traffic and fault tolerance The load balancing is perform only on the send port 4 Load Balancing Send amp Receive Provides load balancing of transmit and receive traffic and fault tolerance The load balancing splits the transmit and receive traffic statically among the team adapters without changing the base of the traffic loading based on the source destination MAC and IP addresses 5 Adaptive Load Balancing The same functionality as Load Balancing Send amp Receive In case of traffic load in one of the adapters the load balanci
78. ric 7 2 4 1 Applicable Hardware All InfiniBand devices 7 2 4 2 Synopsys smpquery nq eas ise v Eae TE en v cameos Poca port St lt timeout_ms gt node name map lt node name map gt lt op gt lt dest dr path lid guid gt op params 7 2 4 3 Options The table below lists the various flags of the command Table 9 smpquery Flags and Options penne Print Print the help menu Print the help menu menu Increase verbosity level May be used several times for additional verbosity vvv or v v v D Direct Use directed path address arguments The path is a comma separated list of out ports Examples 0 self port 0 1 2 1 4 out via port 1 then 2 G Guid Use GUID address argument In most cases it is the Port GUID Example 0x08f 1040023 s sm 000 a 000 lt smlid gt Use Use lt smlid gt as the target LID for SM SA queries as the Use lt smlid gt as the target LID for SM SA queries LID for SM SA queries Supported operations e NodelInfo NI lt addr gt NodeDesc ND lt addr gt PortInfo PI lt addr gt lt portnum gt SwitchInfo SI lt addr gt PKeyTable PKeys lt addr gt lt portnum gt SL2VLTable SL2VL lt addr gt lt portnum gt VLArbitration VLArb lt addr gt lt portnum gt GUIDInfo GI lt addr gt dest dr path lid guid Destination s directed path LID or GUID node name map lt file gt Node name map file x extended Use
79. rite bw calculats the BW of RDMA write between a pair of machines One acts as a server and the other as a client The client RDMA writes to the server memory and calculate the BW by sampling the CPU each time it receive a successfull completion The test supports features such as Bidirectional in which they both RDMA write to each other at the same time change of mtu size tx size number of iteration message size and more Using the a flag provides results for all message sizes Mellanox Technologies 19 J Rev 3 0 0 InfiniBand Fabric 7 3 5 1 Synopsys ib write bw q num of gps c onnection type RC UC UD i b port ib port m tu mtu size Este message sizel t x deptn o exo Sue enteraron pm p lore PDTEDODE e cba eos tional a 11 V ersion 7 3 5 2 Options The table below lists the various flags of the command Table 24 ib write bw Flags and Options PS Listens Listens on connect to port lt port gt default 18515 to port lt port gt Listens on connect to port lt port gt default 18515 18515 7 3 6 ib write lat ib write lat calculats the latency of ROMA write operation of message sizeB between a pair of machines One acts as a server and the other as a client They perform a ping pong benchmark on which one side ROMA writes to the other side memory only after the other side wrote on his mem ory Each of the sides samples the CPU clock each time they write to the other side memory in or
80. rvice will use the first port which is not in down state To run OpenSM manually enter on the command line opensm exe For additional run options enter opensm exe h Notes Forlong term running please avoid using the v verbosity option to avoid exceeding disk quota e Running OpenSM on multiple servers may lead to incorrect OpenSM behavior Please do not run more than a single instance of OpenSM in the subnet BDiagnet cannot run on the same IB port that OpenSM is running on 38 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 InfiniBand Fabric 7 1 Network Direct Interface The Network Direct Interface NDI architecture provides application developers with a network ing interface that enables zero copy data transfers between applications kernel bypass I O genera tion and completion processing and one sided data transfer operations NDI is supported by Microsoft and 1s the recommended method to write InfiniBand application NDI exposes the advanced capabilities of the Mellanox networking devices and allows applica tions to leverage advances of InfiniBand For further information please refer to http msdn microsoft com en us library cc904397 v vs 85 aspx 7 2 InfiniBand Fabric Diagnostic Utilities The diagnostic utilities described in this chapter provide means for debugging the connectivity and status of InfiniBand IB devices in a fabric The tools are e Section 7 2 1 8
81. s controllers eee eee 3 6 4 Removing a Port VLAN To remove a port VLAN perform the following steps Step 1 In the Device Manager window right click the network adapter from which the port VLAN was created Step 2 Left click Properties Step 3 Select the VLAN tab from the Properties sheet Step 4 Select the VLAN to be removed Step 5 Click Remove and confirm the operation Mellanox Technologies 25 J Rev 3 0 0 Driver Configuration 4 Driver Configuration Once you have installed Mellanox WinOF VPI package you can perform various modifications to your driver to make it suitable for your system s needs Changes made to the Windows registry happen immediately and no backup is automati cally made Do not edit the Windows registry unless you are confident regarding the changes 4 1 Configuring the InfiniBand Driver 4 1 1 Modifying Mellanox HCA Configuration To modify HCA configuration after installation perform the following steps Step 1 Open the Registry editor by clicking Start gt Run and entering regedit Step 2 In the navigation pane expand HKEY LOCAL MACHINE gt SYSTEM gt CurrentControlSet gt Services Step 3 Expand in the navigation pane the HCA driver service entry mtcha for the InfiniHost family mlx4 hca and mlx4 bus for the ConnectX family Step 4 Click the Parameters entry in the expanded driver service entry to display HCA parameters Step 5 Doubl
82. s on all IB devices ibstat 1 list all IB devices ibstat p show port guids ibstat mthcad 2 show status of port 2 of mthcal 7 2 12 vstat vstat is a binary which displays information on the HCA attributes 7 2 12 1 Synopsys vysrat el 7 2 12 2 Options The table below lists the various flags of the command Table 17 ibstat Flags and Options pN repeat every N sec 68 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 7 2 13 part_man part manis an application which allows creating deleting and viewing existing host partitions 7 2 13 1 Synopsys part man exe lt show add rem gt port guid lt pkeyl pkey2 gt 7 2 13 2 Options The table below lists the various flags of the command Table 18 part man Flags and Options Shows the existing partitions The output format is port guidl pkeyl pkey2 pkey3 pkey4 pkey5 pkey6 pkey7 pkey8 where port guid is a port guid in hexadecimal format and pkeys are the values of the partition key in hex format of this port The default partition key OxFFFF is not shown and cannot be created by the part man exe Creates new partition s on the specified port The output format is e part man add port guid gt lt pkey gt lt peky gt Port guid is in the format of e XXXX XXXX XXXX XXXX Pkey format e 0X8XXX or 8xxx Removes partition key of the specified port The output format is part man exe rem port guid lt pkey
83. server to server data movement directly between application memory without any CPU involvement RDMA over Ethernet RoCE is a mechanism to provide this efficient data transfer with very low latencies on loss less Ethernet networks With advances in data center convergence over reliable Ethernet ConnectX 2 ConnectX 3 EN with RoCE uses the proven and efficient RDMA transport to provide the platform for deploying RDMA technology in mainstream data center application at 10GigE and 40GigE link speed ConnectX 2 ConnectX 3 EN with its hardware offload support takes advantage of this efficient RDMA transport InfiniBand services over Ethernet to deliver ultra low latency for performance critical and transaction intensive appli cations such as financial data base storage and content delivery networks RoCE encapsulates IB transport and GRH headers in Ethernet packets bearing a dedicated ether type While the use of GRH is optional within IB subnets it is mandatory when using RoCE Applications written over IB verbs should work seamlessly but they require provisioning of GRH 12 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 information when creating address vectors The library and driver are modified to provide map ping from GID to MAC addresses required by the hardware 3 1 2 Ported Applications The following applications are ported with RoCE e Network Direct ND applications work without any change e Perf
84. tended performance counters and reset perfquery R 0x20 1 reset performance counters of port 1 only perfquery e R 0x20 1 reset extended performance counters of port 1 only perfquery R a 32 reset performance counters of all ports perfquery R 32 2 Ox0fff reset only error counters of port 2 perfquery R 32 2 Oxf000 reset only non error counters of port 2 I Read local port s performance counters gt perfquery Port counters Lid 6 port 1 RCSB OG Ke Cia oi TAE I OU BSEC t E EU DRE TM TEL 0x1000 SRI To NOR di OVES eer E MADE 0 A n e DRE x 0 A A Sa ee Ned Ars oa 0 DO ENAS SAT E NT 0 hevhemopePDVSBEEOES Saas ein soni a 0 RETO MRE ECOS E tanda 0 A E puc 0 MMECONS EE aso 0 REVCOMS O aaa 0 LAN IEO IE ELE E eene ium 0 Mellanox Technologies 57 J Rev 3 0 0 InfiniBand Fabric 2 Read performance counters from LID 2 all ports 3 Read then reset performance counters from LID 2 port 1 58 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 TINRELE E BA DES 0 Duk O es oed SAP UEM EE 0 RO o MU ITI TCI 0 REVRSMO ESAS e 0 ReVoWReISVEEPOESI ao 0 EDES a O PE 3 E ONS ENL OES e 0 ROUCONS TALES ae 0 MS IEEE OE Se eL PT 0 EXerutO Ver TUNED TORS osas 0 Ad 0 AMD aia MED an 0 Reus EN er ES 0 FIER MP M RETE REV ooo dd cse NNI tee 0 7 2 6 ibping ibping uses vendor MADS to validate connectiv
85. ther end port is down Mismatch in the configurations between the two ports may affect the performance When Using MSI X both ports should use the same RSS mode To fix the problem configure the RSS mode of both ports to be the same in the driver GUI Mellanox ConnectX EN 10Gbit Ethernet Adapter X device failed to create enough MSI X vec tors The Network interface will not use MSI X interrupts This may affects the performance To fix the problem configure the number of MSI X vectors in the registry to be at least Y 88 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 10 Documentation Under installation directory gt Documentation License file User Manual this document MLNX VPI Installation Guide MLNX VPI Release Notes Mellanox Technologies 89 J
86. ty while using this WinOF release For further information see Section 2 4 1 Downloading the Firmware Tools Pack age on page 10 The adapter card may not have been shipped with the latest firmware version This section describes how to update firmware 2 4 1 Downloading the Firmware Tools Package 1 Download Mellanox Firmware Tools Please download the current firmware tools package MFT from http www mellanox com gt Prod ucts gt Software Drivers gt InfiniBand amp VPI SW Drivers gt Firmware Tools The tools package to download is MFT Software for Windows x86 for x86 architecture and MFT Software for Windows x64 for x64 architecture 2 Install and Run WinMFT To install the WinMFT package double click the MSI or run it from the command prompt Install the WinMFT package from the command line with administrator privileges 5 Enter nsiexec exe i WinMFT arch version msi 3 Check the Device Status start stop mst is automatically done by the tools gt CAUsersherod Desktop mst start To check device status run gt mst status If no card installation problems occur the status command should produce the following output mt lt device id pciconf0 mt lt device id pci cro where device ID will be one of the supported PCI device IDs 2 4 2 Downloading the Firmware Image of the Adapter Card To download the correct card firmware image please visit http www mellanox com gt Supp
87. uration See the section Performance Tuning to take advantage of Mellanox 10 GBit NIC performance Issue 3 The driver does no start Suggestion 1 This can happen due to an RSS configuration mismatch between the TCP stack and the Mellanox adapter To confirm this scenario open the event log and look under System for the mlx4eth5 or mlx4eth6 source If found enable RSS as follows 1 Run the following command netsh int tcp set global rss enabled Suggestion 2 This is a less recommended suggestion and will cause low performance Disable RSS on the adapter To do this set RSS mode to No Dynamic Rebalancing 86 Mellanox Technologies WinOF VPI for Windows User Manual Rev 3 0 0 Issue 4 The Ethernet driver fails to start In the Event log under the mlx4 bus source the fol lowing error message appears RUN FW command failed with error 22 Suggestion The error message indicates that the wrong firmware image has been programmed on the adapter card See http www mellanox com gt Support gt Firmware Download Issue 5 The Ethernet driver fails to start A yellow sign appears near the Mellanox ConnectX 10Gb Ethernet Adapter in the Device Manager display Suggestion This can happen due to a hardware error Try to disable and re enable Mellanox ConnectX Adapter from the Device Manager display Issue 6 No connectivity to a Fault Tolerance bundle while using network capture tools e g Wireshark
88. x Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 4 909 7200 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2012 Mellanox Technologies All rights reserved Mellanox Mellanox Logo BridgeX ConnectX CORE Direct InfiniBridge amp InfiniHost InfiniScale PhyX SwitchX amp Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd FabricIT MLNX OS Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3280 WinOF VPI for Windows User Manual Rev 3 0 0 Table of Contents Ch pter 1 About this Manual iq e Doc eade eo RE ER Rua A Ep 7 1 1 Scope 7 1 2 Intended Audience 7 1 3 Documentation Conventions 7 1 3 1 Common Abbreviations and Acronyms o 8 Chapter2 Introductio sida eee aaah Sato Ea ua D RC ue 9 2 Mellanox VPI Package Contents 9 2 2 Hardware and Software Requirements

MLNX VPI Windows user manual

Contents

Download Pdf Manuals

Related Search

Related Contents