Mellanox OFED for FreeBSD User Manual
Contents
1. 0 0000s 9 TAT Tarball Package x6 oet CR YAO RS RES LES 9 112 Software Components 5 oat t gae ere alas qe gea eS 9 Chapter 2 Installation 4 65 60 cus RR a a OR EHS E REYERXERTRE A Dek OS 10 2 1 Software Dependencies o sesede gia niu suea pidana ia EA 10 2 2 Downloading Mellanox Driver for FreeBSD sossoseeeseser esse rer ere 10 2 3 Installing Mellanox Driver for FreeBSD soseseeereeererer nnau 10 2 4 Firmware Programming sees e 12 2 4 1 Installing Firmware Tools ssseeeeseseresererrrerrrrr ere I 12 2 4 2 Downloading Firmware sssseresrereerrrererrererrer rer rer rr rss rr sa 12 2 4 3 Undating Firmware 0 0 ect ene a as 12 Chapter 3 Ethernet Driver 1 42 tbe et 3 beo do ORE ERE etek EE 13 3 1 Usage and Configuration 0 0 0 0 ete eens 13 Chapter 4 Driver Features s s esessssses e ooco c rr Ro RR al ane 16 4 1 RDMA over Converged Ethernet RoCE 0 000 cece ee rr rer 16 4 2 Packet Pacing Beta Version 16 ADA ak ca Oe ee ee ERI eT ESE a RNE UR LAE RP ME 16 4 2 2 Feature Characteristics 05 00 cece een eee E ee eet ee ees 17 Chapter 5 Performance Tuning cc ccc cece ec cece eee reer nnn n 18 5 1 Interrupt Moderation 0 00 n 18 5 2 Tuning for NUMA Architecture 2 0 0 eee ees 18 5 2 1 Single NUMA Architecture 5 18 5 2 2 Dual NUMA Architecture 0 0 0 ee cc ee 19 Mellanox Technologies 3 Rev 2 1 5 List of Tables2. This document provides information on the Mellanox driver for FreeBSD and instructions for installing the driver on Mellanox ConnectX adapter cards supporting 10Gb s and 40Gb s Ether net and InfiniBand The driver rele ase exposes the following capabilities Single Dual port e Up to 16 Rx queues per port Up to 32 Tx queues per port according to number of CPUs e MSI X or INTx Adaptive interrupt moderation HW Tx Rx checksum calculation e Large Send Offload i e TCP Segmentation Offload Large Receive Offload VLAN Tx Rx acceleration HW VLAN stripping insertion e Net device statistics 1 1 Mellanox OFED for FreeBSD Package Contents 1 1 4 Tarball Package Mellanox OFED for FreeBSD package includes the following directories e modules contains the relevant Makefiles e ofed source code 1 1 2 Software Components Mellanox OFED for FreeBSD contains the following software components Table 5 Mellanox OFED for FreeBSD Software Components Components Description mlx4 driver mlx4 is the low level driver implementation for the ConnectX adapters designed by Mellanox Technologies The ConnectX can operate as an InfiniBand adapter and as an Ethernet NIC To accommodate the two flavors the driver is split into modules mlx4 mlx4 en mlx4 core Handles low level functions like device initialization and firmware commands processing Also controls resource allocation so that the InfiniBand Ethernet and
3. FC functions can share a device without interfering with each other mlx4 en Handles Ethernet specific functions and plugs into the netdev mid layer mlx4 ib Handles InfiniBand specific functions supplied by ib core in order to interact with verbs and ULPs Documentation Release Notes User Manual 9 Mellanox Technologies Rev 2 1 5 2 Installation This chapter describes how to install and test the Mellanox driver for FreeBSD package on a sin gle host machine with Mellanox InfiniBand and or Ethernet adapter hardware installed 2 4 Software Dependencies To install the driver software kernel sources must be installed on the machine To run the Packet Pacing feature the attached kernel patch and firmware must be installed on the machine 2 2 Downloading Mellanox Driver for FreeBSD Step 1 Verify that the system has a Mellanox network adapter HCA NIC installed The following example shows a system with an installed Mellanox HCA pciconf lv grep Mellanox C 3 mlx4 core08pci0 7 0 0 class 0x028000 card 0x000615b3 chip 0x100315b3 rev 0x00 hdr 0x00 vendor Mellanox Technologies device MT27500 Family ConnectX 3 class network Step 2 Download the tarball image to your host The image name has the format MLNX OFED FreeBSD ver tgz You can download it from http www mellanox com gt Products gt Software gt Ethernet Drivers gt FreeBSD or http www mellanox com g
4. The driver defaults to the following parameters Both ports are activated 1 e a net device 1s created for each port The number of Rx rings for each port is the nearest power of 2 of number of CPU cores limited by 16 LRO is enabled with 32 concurrent sessions per Rx ring 15 Mellanox Technologies Rev 2 1 5 4 Driver Features 4 1 RDMA over Converged Ethernet RoCE RoCE allows InfiniBand IB transport applications to work over an Ethernet network RoCE encapsulates the InfiniBand transport and the GRH headers in Ethernet packets bearing a dedi cated ether type 0x8915 Thus any VERB application that works in an InfiniBand fabric can also work in an Ethernet fabric RoCE is enabled only for drivers that support VPI currently only mlx4 When working with RDMA applications over Ethernet link layer the following points should be noted The presence of a Subnet Manager SM is not required in the fabric Thus operations that require communication with the SM are managed in a different way in RoCE This does not affect the API Since the SM is not present querying a path is impossible Therefore the path record structure must be filled with the relevant values before establishing a connection Hence it is recommended working with RDMA CM to establish a connection as it takes care of filling the path record structure Since LID is a layer 2 attribute of the InfiniBand protocol stack it is not set for a p
5. Turn ON the interruption moderation Run gt sysctl hw mlxen lt xa gt conf adaptive rx coal 1 Step 3 Set the threshold values for packet rate limits and for moderation time Run gt sysctl hw mlxen lt xa gt conf pkt_rate_low N gt sysctl hw mlxen lt xa gt conf rx usecs low N gt sysctl hw mlxen lt xa gt conf pkt_rate_high gt sysctl hw mlxen lt x gt conf rx usecs high N N Above an upper limit of packet rate adaptive moderation will set the moderation time to its high est value Below a lower limit of packet rate the moderation time will be set to its lowest value P Tuning for NUMA Architecture Single NUMA Architecture When using a server with single NUMA no tuning is required Also make sure to avoid using core number 0 for interrupts and applications 1 Find a CPU list sysctl a grep group level 2 A 1 group level 2 cache level 2 gt Kom con wask eMncei 0 I 2 3 d Sy Gy Ty By 9 10 Ji cpu 2 Tune Mellanox NICs to work on desirable cores Find the NIC s PCI location pciconf lv grep mlx mlx4 core08pci0 2 0 0 class 0x028000 card 0x000315b3 chip 0x100715b3 rev 0x00 hdr 0x00 Find the NIC s device name by its PCI location Sysctl a grep pci2 dev mlx4 core 0 parent pci2 Mellanox Technologies 18 J Rev 2 1 5 Performance Tuning This means the NIC on PCI number 2 has a logic device called mlx4 core0 e Find the device interrupts vmstat i
6. 0002c92ffccl VSD PSID MT 1020120019 gt To check driver version gt dmesg Example gt dmesg mlx4 core Mellanox ConnectX core driver v2 1 Aug 21 2014 mlx4 en Mellanox ConnectX HCA Ethernet driver v2 1 Aug 18 2014 gt To query stateless offload status gt ifconfig mlxen lt x gt Note lt x gt is the OS assigned interface number gt To set stateless offload status gt ifconfig mlxen lt x gt rxcsum rxcsum txcsum txcsum tso tso lro lro Note lt x gt is the OS assigned interface number gt To query interrupt coalescing settings gt sysctl a grep adaptive Example gt sysctl a grep adaptive hw mlxen0 conf adaptive rx coal 1 hw mlxenl conf adaptive rx coal 1 gt To enable disable adaptive interrupt moderation gt sysctl hw mlxen lt x gt conf adaptive rx coal 1 0 Note lt x gt is the OS assigned interface number 13 Mellanox Technologies Rev 2 1 5 By default the driver uses adaptive interrupt moderation for the receive path which adjusts the moderation time to the traffic pattern To query values for packet rate limits and for moderation time high and low gt syeCiell oM emea Wie rere gt sysctl a grep rx usecs gt To set the values for packet rate limits and for moderation time high and low sysctl hw mlxenXx conf pkt rate low N sysctl hw mlxenXx conf pkt rate high N sysctl hw mlxenXx conf rx usecs low N sysctl hw mlxen lt x gt
7. 4 User will be able to modify the rate limit value 5 When closing the TCP socket user will need to destroy the relevant ring 4 2 2 Feature Characteristics e Supports up to 30 000 rate limited TCP connections Each TCP connection is mapped to a specific QP User interface SIOCARATECTL Create rate limit ring QP SIOCSRATECTL Modify rate limit value for a specific ring SIOCGRATECTL Query rate limit value for a specific ring SIOCDRATECTL Destroy rate limited ring QP SIOCSTXRINGID Associate ring id with a TCP socket All Rate limited rings statistics will be available via Sysctl rate limit val packets bytes Limitations e Max rate limit rings 30000 e Min rate 500Kbps Maxrate HCA s max BW Different rate limits to be configured per NIC port 1024 17 Mellanox Technologies Rev 2 1 5 5 5 1 5 2 5 2 1 Performance Tuning Interrupt Moderation Interrupt moderation is used to decrease the frequency of network adapter interruptions to the CPU Mellanox network adapters use an adaptive interrupt moderation algorithm by default The algorithm checks the transmission Tx receives Rx packet rates and modifies the Rx interrupt moderation settings accordingly In order to manually set Rx interrupt moderation use sysctl Step 1 Turn OFF the interrupt moderation Run gt sysctl hw mlxen lt x gt conf adaptive rx coal 0 Note lt x gt is the OS assigned interface number Step 2
8. G tools connect to the IB fabric Master Subnet Manager Multicast Forwarding Tables The Subnet Manager that is authoritative that has the reference con figuration information for the subnet See Subnet Manager A table that exists in every switch providing the list of ports to for ward received multicast packet The table is organized by MLID Network Interface Card NIC A network adapter card that plugs into the PCI Express slot and pro vides one or more ports to an Ethernet network Unicast Linear For warding Tables LFT A table that exists in every switch providing the port through which packets should be sent to each LID Virtual Protocol Inter connet VPI A Mellanox Technologies technology that allows Mellanox channel adapter devices ConnectX to simultaneously connect to an Infini Band subnet and a 10GigE subnet each subnet connects to one of the adpater ports Related Documentation Table 4 Reference Documents Document Name Description Release 1 2 1 InfiniBand Architecture Specification Vol 1 The InfiniBand Architecture Specification that is pro vided by IBTA Support and Updates Webpage Please visit http www mellanox com gt Products gt Software gt Ethernet Drivers gt FreeBSD Driv ers for downloads FAQ troubleshooting future updates to this manual etc Mellanox Technologies 8 Rev 2 1 5 Overview 1 Overview
9. GE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 4972 0 4 959 3245 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB amp CoolBox CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS TestX PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT HPC X Mellanox Open Ethernet Mellanox PeerDirect Mellanox Virtual Modular Switch MetroDX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 2950 Rev 2 1 5 Table of Contents Table of Contents sess Gi s 6 aee e eRREDUEREEEEEETEREEERREEERRTERANESEEEKERE ETE 3 List Of Tables rasere ane iy ak Ses Ait EE dea oh ee BBR RR BR VR ee 4 Document Revision History ik E n Ra x RAE ERE UD EA RA VERRE EE 5 About this Manual sce eek eee uoa uw KR KRA vertene tekei ues ives RNE 6 Chapter Overview eie oU RE T se sneda RENAN ORS OREN BIR TRONS EER TE SS 9 1 1 Mellanox OFED for FreeBSD Package Contents
10. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox OFED for FreeBSD User Manual Rev 2 1 5 www mellanox com Rev 2 1 5 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMA
11. Table 1 Document Revision History ssssmsressrererrrerrrererrrrre rer e 5 Table2 Abbreviations and Acronyms 0 0 00 ccc een 7 Pable 32 Glossary c cese epe EPOR OL OL RR SA ae Ree ee aed RR 8 Table 4 Reference Documents 0 00 ccc rer rer rer nen eben teens 8 Table 5 Mellanox OFED for FreeBSD Software Components 000 cee cece eee 9 Mellanox Technologies 4 Rev 2 1 5 Document Revision History Table 1 Document Revision History Release Date Description 2 1 2 1 5 January 15 2015 e Added the following sections Section 4 Driver Features on page 16 Updated the following sections Section 1 Overview on page 9 Section 2 Installation on page 10 Section 5 Performance Tuning on page 18 5 Mellanox Technologies Rev 2 1 5 About this Manual This Preface provides general information concerning the scope and organization of this User s Manual Intended Audience This manual is intended for system administrators responsible for the installation configuration management and maintenance of the software and hardware of VPI InfiniBand Ethernet adapter cards It is also intended for application developers Mellanox Technologies 6 J Rev 2 1 5 Common Abbreviations and Acronyms Table 2 Abbreviations and Acronyms M ries Whole Word Description B Ca
12. a grep mlx4 cored awk print 1 sed s irg sed s 285 286 287 Bindeach interrupt to a desirable core cpuset x 299 cii i cpuset x 286 12 Gpuset x OU ol 3 Bind the application to the desirable core cpuset l 1 11 app name gt sever flag cpuset l 1 11 app name gt client flag IP Specifying a range of CPUs when using the cpuset command will allow the application to choose any of them This is important for applications that execute on multiple threads The range argument is not supported for interrupt binding 5 2 2 Dual NUMA Architecture 1 Find the CPU list closest to the NIC e Find the NIC s PCI location pciconf lv grep mlx mlx4 core08pci0 4 0 0 class 0x028000 card 0x000315b3 chip 0x100715b3 rev 0x00 hdr 0x00 Usually low PCI locations are closest to NUMA number 0 and high PCI locations are closest to NUMA number 1 Here is how to verify the locations Findthe NIC s pcib by PCI location in this example try PCI 4 sysctl a grep pci 4 paren dev pci 4 parent pcib3 Find the NIC s pcib location sysctl a grep pcib 3 location dev pcib 3 10cation slot 2 function 0 handle X SB PCIO NPE3 In handle PCIO is the value for locations near NUMAO and PCII is the value for locations near NUMAI 19 Mellanox Technologies Rev 2 1 5 Find the cores list of the closest NUMA sysctl a grep group level 2 A 1 group level 2 cache level 2 gt
13. adapter card firmware 2 4 14 Installing Firmware Tools Step 1 Download the current Mellanox Firmware Tools package MFT from www mellanox com gt Products gt Adapter IB VPI SW gt Firmware Tools The tools package to download is MFT SW for FreeBSD tarball name is mft X X X tgz Step 2 Extract the tarball and run the installation script 2 4 Downloading Firmware Step 1 Retrieve device s PCI slot i e pci0 x 0 0 Run mst status Step 2 Verify your card s PSID Sflint d pei q Step3 Download the desired firmware from the Mellanox website http www mellanox com page firmware download 2 4 3 Undating Firmware Step 1 Unzip the firmware binary file run Srlime 0l speis a lt img bim 19 Step 2 Reboot the server Mellanox Technologies 12 J Rev 2 1 5 Ethernet Driver 3 Ethernet Driver 3 1 Usage and Configuration To assign an IP address to the interface gt ifconfig mlxen lt x gt ip Note lt x gt is the OS assigned interface number gt To check driver and device information gt pciconf lv grep mlx 2 ible cel presb ADXQWBRZ Gl Example gt pciconf lv grep mlx mlx4 core08pci0 7 0 0 class 0x028000 card 0x003715b3 chip 0x100315b3 rev 0x00 hdr 0x00 2 liume xol Terent e ge e mage type FS2 FW Version Zio ASTA Device ID 4099 Description Node Porti Port2 Sys image GUIDs 0002c903002ffcc0 0002c903002ffccl 0002c903002ffcc2 0002c903002ffcc3 ACS 0002c92ffcc0
14. conf rx usecs high N AS STAT BEST Note x is the OS assigned interface number Example gt eyst a grep jake rate hw mlxen0 conf pkt_rate_low 400000 hw mlxen0 conf pkt_rate_high 450000 hw mlxen1 conf pkt_rate_low 400000 hw mlxenl conf pkt rate high 450000 sysctl a grep rx usecs hw mlxenO conf rx usecs low 0 hw mlxen0 conf rx usecs high 128 hw mlxenl conf rx usecs low 0 hw mlxenl conf rx usecs high 128 Above an upper limit of packet rate adaptive moderation will set the moderation time to its highest value Below a lower limit of packet rate the moderation time will be set to its lowest value To query pause frame settings gt ifconfig m mlxen lt x gt Note lt x gt is the OS assigned interface number gt To set pause frame settings gt ifconfig m mlxen lt x gt mediaopt rxpause txpause Note lt x gt is the OS assigned interface number gt To query ring size values gt Syscul a cme Size Example Sys a ome suu hw mlxen0 conf rx size 1024 hw mlxen0 conf tx size 1024 hw mlxenl conf rx size 1024 hw mlxenl conf tx size 1024 To modify rings size gt sysctl hw mlxen lt x gt conf rx size N gt sysctl hw mlxen lt x gt conf tx size N Note lt x gt is the OS assigned interface number gt To obtain additional device statistics gt sysctl a grep mlx grep stat Mellanox Technologies 14 Rev 2 1 5 Ethernet Driver
15. er above ib core a Go to the ib core directory Run cd modules ibcore b Clean any previous dependencies Run make m SHEAD share mk SYSDIR SHEAD sys clean cleandepend c Compile the ib core module Run make m SHEAD share mk SYSDIR SHEAD sys d Install the ib core module Run make m SHEAD share mk SYSDIR SHEAD sys install e Load the ib core module Run kldload ibcore mlx4 ib a Go to the mlx4 ib directory Run cd modules mlx4ib b Clean any previous dependencies Run make m SHEAD share mk SYSDIR SHEAD sys clean cleandepend 11 Mellanox Technologies Rev 2 1 5 c Compile the mlx4 ib module Run make m SHEAD share mk SYSDIR SHEAD sys d Install the mlx4 ib module Run make m SHEAD share mk SYSDIR SHEAD sys install e Load the mlx4 ib module Run kldload mlx4ib ipoib Run the same steps specified above for HEAD sys modules ipoib To load a module on reboot add mlx4_core YES mlxen_load YES ibcore_load YES d mlx4ib load YES to the boot loader conf file create if does not exist Run kdlstat in order to verify which modules are loaded on your server Ad 2 4 Firmware Programming The adapter card was shipped with the most current firmware available This section is intended for future firmware upgrades and provides instructions for 1 installing Mellanox firmware update tools MFT 2 downloading FW and 3 updating
16. ort and is displayed as zero when querying the port With RoCE APM is not supported The GID table for each port is populated with N 1 entries where N is the number of IP addresses that are assigned to all network devices associated with the port including VLAN devices The first entry in the GID table at index 0 for each port is always present and equal to the link local IPv6 address of the net device that is associated with the port Note that even if the link local IPv6 address is not set index 0 is still populated GID format can be of 2 types IPv4 and IPv6 IPv4 GID is a IPv4 mapped IPv6 address while IPv6 GID is the IPv6 address itself Load the following modules for RoCE support mlx4 core ib core mlx4 ib and mlx 4 en 4 2 Packet Pacing Beta Version Packet pacing also known as the Rate limit defines a maximum bandwidth allowed for a TCP connection Limitation is done by HW where each QP transmit queue has a rate limit value from which it calculates the delay between each packet sent 4 2 1 Flow 1 User will create a rate limited ring according to desired rate Upon success the ring ID is returned 2 User will associate the ring ID to a specific TCP socket 1 For the IPv4 address A B C D the corresponding IPv4 mapped IPv6 address is ffff A B C D Mellanox Technologies 16 Rev 2 1 5 Driver Features 3 Rate Limited TCP traffic will be transmitted when using the TCP socket
17. pital B is used to indicate size in bytes or multiples of bytes e g IKB 1024 bytes and IMB 1048576 bytes b Small b is used to indicate size in bits or multiples of bits e g 1Kb 1024 bits FW Firmware HCA Host Channel Adapter HW Hardware IB InfiniBand LSB Least significant byte Isb Least significant bit MSB Most significant byte msb Most significant bit NIC Network Interface Card SW Software VPI Virtual Protocol Interconnect PFC Priority Flow Control PR Path Record RDS Reliable Datagram Sockets RoCE RDMA over Converged Ethernet SL Service Level QoS Quality of Service ULP Upper Level Protocol VL Virtual Lane Glossary The following is a list of concepts and terms related to InfiniBand in general and to Subnet Man agers in particular It 1s included here for ease of reference but the main reference remains the 7 Mellanox Technologies Rev 2 1 5 InfiniBand Architecture Specification Table 3 Glossary Channel Adapter CA An IB device that terminates an IB link and executes transport func Host Channel Adapter tions This may be an HCA Host CA or a TCA Target CA HCA HCA Card A network adapter card based on an InfiniBand channel adapter device IB Devices Integrated circuit implementing InfiniBand compliant communica tion In Band A term assigned to administration activities traversing the IB connec tivity only Local Port The IB port of the HCA through which IBDIA
18. roy keount 029 sers iniri di 2 3 4d By Gy Wh Bp Oy WO tik lt cowe group level 2 cache level 2 gt cpuscount ili mask irr 0001 2 le TA SS GENET Lolo 2 02 PRESS GTI Note Each list of cores refers to a different NUMA 2 Tune Mellanox NICs to work on desirable cores Step 1 Pin both interrupts and application processes to the relevant cores Step 2 Find the closest NUMA to the NIC e Find the NIC s device name by its PCI location sysctl a grep pci4 dev mlx4 core 0 sparent pci4 This means the NIC on PCI number 4 has a logic device called mlx4 core0 Find the device interrupts vmstat ia grep mlx4 coret awk print 1 sed s irg sed s 338 339 340 Bindeach interrupt to a core from the closest NUMA cores list Note It s best to avoid core number 0 Comseic x 398 JL i Guset x 339 L 2 cpuset x 340 1 3 Bind the application to the closest NUMA cores list Note It s best to avoid core number 0 cpuset l 1 11 app name gt sever flag cpuset l 1 11 app name gt client flag IP For best performance change CPU s BIOS configuration to performance mode Mellanox Technologies 20 J
19. t Products gt Software gt InfiniBand VPI drivers gt FreeBSD Step 3 Use the md5sum utility to confirm the file integrity of your tarball image 2 3 Installing Mellanox Driver for FreeBSD Prior to installing the driver please re compile and install the kernel with NO OFED options devices enabled P Step 1 Extract the tarball Step 2 Compile and load needed modules in the following order of dependencies Ethernet Driver mlx4 core a Go to the mlx4 directory Run cd modules mlx4 b Clean any previous dependencies Run make m SHEAD share mk SYSDIR SHEAD sys clean cleandepend Mellanox Technologies 10 J Rev 2 1 5 Installation c Compile the mlx4 core module Run make m SHEAD share mk SYSDIR SHEAD sys d Install the mlx4 core module Run make m SHEAD share mk SYSDIR SHEAD sys install e Load the mlx4 core module Run kldload mlx4 mixen a Go to the mlxen directory Run cd modules mlxen b Clean any previous dependencies Run make m SHEAD share mk SYSDIR SHEAD sys clean cleandepend c Compile the mlxen module Run make m SHEAD share mk SYSDIR SHEAD sys Note For packet pacing support add CONFIG_RATELIMIT yes This option has a kernel patch dependency d Install the mlxen module Run make m SHEAD share mk SYSDIR SHEAD sys install e Load the mlxen module Run kldload mlxen InfiniBand Driver mlx4 core Run the same steps as specified for Ethernet driv
Download Pdf Manuals
Related Search