Home

10GbE User Manual

1. dapl deve L Preparing dapl devel static Preparing S HH HH uud HH HH H HH HH H HH H HHHH HH H HH H HH HH dapl deve l static ng dapl utils HH 4444 HH HH H H HH H H HH HH Preparing A HH HH HH HH HH H H HH H HH H HH HH perftest Preparing mstflint Preparing HH A RAER HERRE EHRRRHHHRHH mft HH 4444 HH H H HH H H HH HH Preparing srptools HH HH 4444 HH H H HH H H HH
2. libmqe devel Preparing HH A A A ARARA REER REEERE HEERE libibcm Preparing HH NE libibem 28 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Preparing libibcn devel HH HH H H HH HH Preparing HH M M libibcn devel HH 4444 HH HH HH Preparing libibumad Preparing HH RR M libibumad
3. libibmad devel THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing libibmad static Preparing LA HH HH HH HH HH HH HH HH HH HH libibmad static Preparing ibsim HH 4444 HH H H HH HH Preparing ibacm HH 4444 HH HH HH Preparing librdmacm HH RR Preparing librdmacm HH A R AHRR RE EHRRRFHHRHH Preparing librdmacm utils Preparing
4. librdmacm devel HH 4444 HH HH HH Preparing THHHBHHHHBHHHHHBHHHHHHHHBHHHBHHHHBHHHHHHBHBHHHBHRHHHI librdmacm devel HH HHHH HH 4 HH H H HH HH Prepar ng HH HAE AA EEE AE AE A EEE HE AE EH BEA EH opensm libs Preparing THHHHHHHBHHHBHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBSHRHHHI opensm libs Preparing opensm H HH 4444 HH H H HH H H HH HH Mellanox Technologies 29 Driver Installation Preparing sm devel HH HH HH H H HH H H HH HH Preparing opensm devel HH A A HARAR R REEERE EHHRR HEHEH Prepari
5. libibverbs devel Preparing libibverbs devel static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHHHRHHRHRI Preparing libibverbs devel static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHRHHBHHRHRI Preparing THHHHHHHHHHHHHBHHHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHHE libibverbs utils HH ARAARA HERRE EHER RFHHRHH Preparing HH libmverbs HH H RAR EEE HAE EE HE EE HA EH Prepar 1 ng HH 4444 HH HH 4 H H HH H H HH HH libmverbs HH AA EEE HE AE AE A EH HE AE HA EH Preparing HH HHH HH libmlx4 HH HH HH HH HH Preparing HH RMA libmlx4 HH HH He 41 4E 4L HH HH HH HH Preparing libmlx4 deve Preparing libmlx4 devel
6. Preparing libibumad devel Preparing HH A RAAHAA HERRE EHER EEHEEHE libibumad devel Preparing libibumad static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHRHBHHRHRHI Preparing libibumad static Preparing libibmad Preparing libibmad Preparing libibmad devel HH RR Preparing
7. Preparing HH A R AHRR HERRE EHER REHHRHH libmlx5 Preparing libmlx5 HH HH HH HH HH HH HH Preparing HH A RA HRR HERRE EHRE EHHRHH libmlx5 devel Preparing Do libmlx5 devel Preparing HH A A R AHRR HERRE EHER EEHEEHE libmverbs devel THHHHHHHBHHHHHBHHHHHHHHBHHHHHHHBHBHHHHHHHBHBHHEHHE Preparing THHHBHHHHHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE libmverbs devel THHHBHHHBHHHBHHBHHHHHHHHBHHHBHHHHBHHHHHHBHBHHHBRHRHHI Preparing libmge HH 4 HH HH HH 4 HH HH HH Preparing KAHRAMAN libmge HH HH 4444 HH HH HH HH HH Preparing HH AA EEE HAE AE A HAE HE EE EH HAE EH libmge devel Preparing
8. 23 4 1 8 3 mlnxofedinstall Return Codes 25 4 1 8 4 Installation Procedure 26 4 1 9 Installation cub usto bass basa aka 33 4 1 10 Post installation Notes 35 4 1 11 Updating Firmware After Installation 35 4 1 12 Uninstalling Mellanox OFED 37 4 1 13 Port Type Management on Linux 37 Al Auto Sensin sns ve an ann 37 4 1 14 1 Enabling Auto Sensing 38 42 Windows Driver ccce an BRS Re pay 38 4 2 1 Downloading MLNX WinOF 38 422 Extracting Files Without Running Installation 39 4 2 3 Installing MLNX WinOF 41 4 2 3 1 Attended Installation 41 4 2 3 2 Unattended Installation 47 4 2 4 Upgrading MLNX WinOF 47 4 2 5 Installation Resuli erica lens 48 4 2 6 OpenSM Activation 48 4 2 7 Uninstalling MLNX WinOF 49 4 271 Attended Uninstall
9. 49 4 2 7 2 Unattended Uninstall 49 4 2 8 Assigning Port IP After Installation 49 429 Port Type Management on Windows 51 4 3 Performance Tuning ea Kama nn UR 53 44 VMware Driver bere bh ERAESREVPRM R BA br BARES 53 4 4 1 Installing and Running the VBI Driver on ESXi 5 x 53 442 Installing and Running the offline bundle Driver on 1 5 54 443 Removing the VIB offline bundle Driver 54 4 55 FIX BOOTS ER beurette pasa are case 55 Chapter 5 Updating Adapter Card Firmware 56 Chapter 6 Troubleshooting 58 Ord Generals o oss it ud d ccu dura 58 6 2 Linux xe A on tee nie ane eae eS ees 59 6 3 WANK OWS scien e e kel 60 Chapter 7 OL 7 1 MCX312B XCCT Specifications 61 72 MCX312C XCCT Specifications 62 4 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 7 3 MCX311A XCCT Specifications 63 7 4 Adapt
10. 1 5 10 gt install sh The package consists of several source 5 The install script rebuilds the source RPMs and then installs the created binary RPMs The created kernel module binaries are placed under lib modules lt kernel ver gt updates kernel drivers net mlx4 mlnx en installer supports 2 modes of installation The install scripts selects the mode of driver installation depending of the running OS kernel version 1 Kernel Module Packaging KMP mode where the source rpm is rebuilt for each installed flavor of the kernel This mode is used for RedHat and SUSE distributions 2 Non KMP installation mode where the sources are rebuilt with the running kernel This mode is used for vanilla kernels Note If the Vanilla kernel is installed as rpm please use the disable kmp flag when installing the driver The kernel module sources are placed under usr src mellanox mlnx en 1 5 10 Run the follow ing commands to recompile the driver gt cd usr src mellanox mlnx en 1 5 10 4 scripts mlnx en patch sh d make gt make install The uninstall and performance tuning scripts are installed Note If the driver was installed without kmp support the sources would be located under usr srs mlnx en 1 5 10 4 1 3 Loading the Driver Step 1 Make sure no previous driver version is currently loaded Run gt modprobe r mlx4 en Step 2 Load the new driver version Run gt modprobe mlx4 en The resu
11. Installa tion Instructions on page 16 If you need to replace it with the short bracket that 15 included in the shipping box please follow the instructions in this section Note Due to risk of damaging the EMI gasket it is not recommended to replace the bracket more than three times To replace the bracket you will need the following parts Mellanox Technologies 15 J 1 4 Hardware Installation 3 4 1 3 4 2 3 5 The new bracket of the proper height The 2 screws saved from the removal of the bracket e The 2 fiber washers saved from the removal of the bracket Removing the Existing Bracket Figure 1 Bracket Screws Screws 1 Remove the two screws holding the bracket in place The bracket comes loose from the card YN Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal procedure above 3 Make sure that the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 Ibs in torque on the screws Card Installation Instructions 1 Before installing the card make sure that the system 15 off and the pow
12. Max Power Passive Cables 6 42W Power and Environmental Temperature Operational 0 C to 55 Non operational 0 C to 70 Humidity 9096 relative humidity Air Flow 2001 EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Regulatory Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet Mellanox Technologies 61 Rev 1 4 Specifications 7 2 MCX312C XCCT Specifications Table 11 MCX312C XCCT Specifications Table Size 5 77 in x 2 7 in 142 24 mm x 68 9 mm Physical Connector SFP 10Gb s Ethernet 0 10GBASE R and 1000BASE R Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 0W Max Power Passive Cables 6 73W Power and Environmental Temperature Operational 0 C to 55 Non operational 0 C to 70 Humidity 9096 relative humidity Air Flow 200LFMP EMC Refer to the following link www mellanox com related docs user manuals Regulato
13. Step2 Right click Mellanox Local Area Connection and left click Properties Networking Sharing Connect using Mellanox IPolB Adapter This connection uses the following items Client for Microsoft Networks QoS Packet Scheduler File and Printer Sharing for Microsoft Networks 4 Microsoft Network Adapter Mutiplexor Protocol Link Layer Topology Discovery Mapper 1 0 Driver 4 Link Layer Topology Discovery Responder Internet Protocol Version 6 TCP IPv6 Install Uninstall Properties Description Transmission Control Protocol Internet Protocol The default wide area network protocol that provides communication across diverse interconnected networks Step3 Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties 50 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Step4 Select the Use the following IP address radio button and enter the desired IP information General You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator For the appropriate IP settings Obtain an IP address automatically Use the following IP address IP address Subnet mask Default gateway Obtain DNS server address automatically Use the following DNS server addresses
14. Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure Mellanox Technologies 73 1 4 CLASS 1 LASER PRODUCT and reference to most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 20 74 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c
15. 5 L EEE 68 PL El 76 Bracket Mechanical Drawing Figure 8 Dual port Bracket Mellanox Technologies 65 Rev 1 4 Specifications Figure 9 Single port Bracket OROROZOZOROZO O O O OBO 7 7 Regulatory Statements For regulatory statements for all ConnectX 3 Pro cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf 66 Mellanox Technologies ConnectXG 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix Interface Connectors Pinout A 1 SFP Connector Pinout Figure 10 Rear View of Module With Pin Placement Module 1 1 B E 2 D id 4 F gt E _ 8 VecT TX fault 2 n Mellanox Technologies 67 1 1 Table 14 SFP Connector Pinout Pin Symbol Name Description 1 VeeT Transmitter Ground Common with Receiver Ground 2 TX Fault Transmitter Fault P 3 TX Disable Transmitter Disable Laser output disabled on high or open 4 SDA 2 wire Serial Interface Data Line 4 3 SCL 2 wire Serial Interface Clock Line 4 6 MOD ABS Module Absent Grounded within the module 7 RSO No connection required 8 RX LOS Loss of Signal indication Logic 0 indicates normal operation 9 RSI No connection required 10 VeeR Receiver Ground Common with Transmitter Groun
16. Error Description We could not Find a new version For the chosen device Please refer to the UM for Show release notes 4 2 3 2 Unattended Installation The following is an example of a MLNX WinOF win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter Step2 Install the driver Run gt MLNX WinOF 4 40 0 All win8 x64 exe S v qn Step3 Optional To configure your setup to contain the logs option please run the following com mand gt MLNX WinOF 4 40 0 All win8 x64 exe 5 v qn v l vx LogFile Step 4 Optional If you do want to upgrade your firmware version run the following command gt MLNX WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h 4 2 4 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX_WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version In Windows 2012 MLNX WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade Mellanox Technologies 47 Rev 1 4 Driver Installation 4 2 5 Installation Results Upon Instal
17. Preparing Qo REREH T T HERT T T T T THAT kerne mft HH 44114 HH HH H HH HH HHHHH HH HH H H Installing knem RPM Preparing HH H HH HH HH H 4 4 HH HH knem HH H HH HH H HH H HH HHHH HH H HH H HH HH Installing mpi selector RPM Preparing Jis H HH HH H H HH HHHH H HH HH mpi selector HH H HH HH 4 4 HH HH Mellanox Technologies 27 1 1 Driver Installation Installing user level RPMs Preparing ofed scripts Preparing HH A A HHRHH RREH RREHHRRHHHRH libibverbs THHHHHHHBHHHHHBHHHHHHHHBHHHHBHHHHBHHHHHHHHBHHHRHRHHHI Preparing libibverbs HH E RA HRR EHER RE EHRR EEHEEHE Preparing HH R M libibverbs devel HH A AE EEE AE EE EEE AE HA EH Preparing
18. gt Software gt Ethernet Drivers gt ConnectX 3 Pro EN 10GigE Linux Driver Download Follow the installation instruc tions included in the download package also available from the download page 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms CPU architectures x86 64 x86 power pc Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 1553 Operating System Linux Operating Systems RedHat EL5 8 RedHat EL5 9 RedHat EL6 2 RedHat EL6 3 OEL6 2 2 6 32 279 19 1 OEL6 3 2 6 32 279 19 1 SLESII SPI SLES11 SP2 Software Dependencies To install the driver software kernel sources must be installed on the machine MLNX EN driver cannot coexist with OFED software on the same machine Hence when installing MLNX EN all OFED packages should be removed done by the en install script 4 1 2 Installing the Driver Step 1 Download Driver Package Please download the current driver package from http www mellanox com gt Products gt Software Ethernet Driver Linux Driver Download Step 2 Install Driver 20 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Run the following commands to install the driver gt tar xzvf mlnx_en 1 5 10 tgz file H
19. Property Hardware Ids Value 15B3 amp DEV 100385UBSYS 00 PCIVEN 15B3 amp DEV 1003 amp SUBSYS 00501583 PCIVEN 15B3 amp DEV 1003 amp CC 028000 PCINVEN 15B3 amp DEV 1003 amp CC 0280 1 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string PCI VEN 15B3 amp DEV 1003 VEN is equal to Ox15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this 15 a valid Mellanox Technologies PCI Device ID If the PCI device does not have Mellanox adapter ID return to Step 2 to check another device gt The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 3 7 2 Linux Get the device location on the PCI bus by running lspci and locating lines with the string Mella nox Technologies gt Ispci grep i Mellanox 27 00 0 Network controller Mellanox Technologies MT27520 Family ConnectX 3 Pro Mellanox Technologies 19 Rev 1 4 Driver Installation 4 Driver Installation The below sections are examples using ConnectX 3 device and will be updated to d include ConnectX 3 Pro in future releases 4 1 Linux Driver For Linux download and install the latest MLNX EN driver software package available via the Mellanox web site at http www mellanox com gt Products
20. bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel k Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 75 Rev 1 4 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 76 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix E Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der YN maximal empfohlenen Temperatur von 55 131 betrieben werden Es ist ein Luft strom
21. Conventions 5 0808 sec mile RATE eme ete eee 9 Technical Support v ea dte a A ee suhu 10 Firmware and Software 10 Chapter T Introduction vtr sas E AAA Ad 11 1 14 Product Overview uc sec Su k ESAE ODE PSU DAE ERES 11 1 2 EFeatures and Benefits a RE 12 1 3 Operating Systems Distributions 13 1 4 Connectivity x A US KEMA il ba eet crepes 13 Chapter2 Interfaces asc ce DA RE mk Sol di 14 2 1 Ethernet SFP Interface 14 2 2 PCI Express Interface Z LEE La 14 2 3 I2C compatible Interface ers sad le Bee e Bt 14 274 BED Interface 3555 Sas SSS 14 Chapter 3 Hardware Installation 15 3 1 System Requirements er Berne pat pire ACE eae eas 15 3 1 1 Hardware o et Mu ida ue ete kaku an ecd 15 3 1 2 Operating Systems Distributions 15 31 3 cox PER evade eR aed el 15 3 2 Safety Precautions 4204 Lk lauren bare 15 3 3 Pre imstallation CheckliSt AO a eR 15 3 4 Bracket Installation Instructions 15 3 4 4 Removing the
22. Existing Bracket 16 3 4 2 Installing the New Bracket 16 3 5 Card Installation Instructions 16 3 6 Cables and Modules An sarah 17 3 6 1 Inserting the Optical Transceiver 17 3 6 2 Cable Installation s Sm Rer Re RED er Ee REG CR nee 18 3 7 Identify the Card in Your System 18 8 44 Windows tl dd le e lk 18 3 527 Om res ee Re PE A INO peces 19 Chapter 4 Driver 20 A oid el phu su i eed h apa 20 4 1 1 Hardware and Software Requirements 20 4 1 2 Installing the a ur ae rer pan pp 20 Mellanox Technologies 3 1 1 4 1 3 Loading the Driver de b Vere e iem obe 21 4 1 4 Unloading the Driver 21 4 1 5 Uninstalling the Driver a ar 22 4 1 6 Hardware and Software Requirements 22 4 1 7 Downloading Mellanox OFED 22 4 1 8 Installing Mellanox OFED 22 4 1 8 1 Pre installation Notes 23 4 1 8 2 Installation Script
23. HAL A HAE Preparing ar mgr HH 4444 HH HH HH Preparing ibdump HH 4444 HH HH HH HH Preparing infiniband diags Preparing HH A RAER HERRE EHER EEHEEHE qperf HH 4444 HH HH HH Preparing fca HH HHHH HH 4 HH H H HH HH INFO updating IMPORTANT NOTE The Manager and Runtime library are installed in opt mellanox fca directory The FCA Manager will not be started automatically To start FCA Manager now type etc init d fca managerd start There should be single process of FCA Manager running per fabric To start FCA Manager automatically after boot type etc init d fca managerd install service Check opt mellanox fca share doc fca README txt for quick start i
24. HH Preparing AJ HH HH HH HH HH H H HH H HH H HH HH rds tools HH A A HERAA RR EHRR REEERE HEER Preparing A HH HH H HH H HH H H H HH H HH H HH HH rds devel HERA Preparing a HH HH HH HH H H HH H HH H HH HH ibutils2 Preparing 2 HH H HH HHHH H HH H HH H HHH H H HH H HH H HH HH ibutils HH A RAER HERRE EHRE EHHRHH Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Preparing mgr HH HHHH HH 4 HH H H HH HH Preparing Ad HH HH HH HH H HH H HH H HH HH dump pr AH
25. OpenSM daemon is configured to run upon boot vma Install packages required by VMA to support both IB and Ethernet vma ib Install packages required by VMA to work over InfiniBand vma eth Install packages required by VMA to work over Ethernet v vv vvv Set verbosity level q Set quiet no messages will be printed umad dev rw Grant non root users read write permission for umad devices instead of default hugepages overcommit Set 80 of MAX MEMORY as overcommit for a huge page allocation 4 1 8 3 mlnxofedinstall Return Codes Mellanox Technologies 25 J 1 1 Driver Installation Table 7 lists the mInxofedinstall script return codes and their meanings Table 7 mlnxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start the mst driver 4 1 8 4 Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount the ISO image on your machine host1 mount o ro loop MLNX OFED LINUX lt ver gt lt 0S label gt lt CPU arch
26. PERp7 41 PRSNT2 PERNT Ada Bag GND GND 49 IZC compatible Connector Pinout Figure 12 Compatible Connector Plug and Pinout Mellanox Technologies 69 Rev 1 4 Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC 70 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix B Finding the GUID MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number and the card MAC for the Ethernet protocol Figure 13 Board Label Example gt Model CX312B ConnectX 3 EN 10 Gigabit Ethernet Made in Israel 100008081008 000010111141 P N 128 SIN MT0806X01504 Rev A2 DE MAC 00 02 C9 27 05 1C B 1 Retrieving Adapter Card GUID On ConnectX 3 Ethernet adapter cards there is a mismatch between the GUID value returned by firmware management tools and that returned by fabric driver utilities that read the GUID via device firmware e g using ibstat MIxburn flint returns Oxffff as GUID while the utilities return a value derived from the MAC address For all driver firmware software purposes the latter value should be used Please see the below example Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for
27. Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the database from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader 10 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 1 Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con nectX 3 Pro EN integrated circuit device These adapters connectivity provide the highest per forming and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics Section 1 1 Product Overview on page 11 Section 1 2 Features and Benefits on page 12 Section 1 3 Operating Systems Distributions on page 13 Section 1 4 Connectivity on page 13 1 1 Product Overview The following section provides the ordering part number port speed number of
28. an causar exposici n a niveles de radiaci n peligrosos Mellanox Technologies 79 Rev 1 4 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 80 Mellanox Technologies
29. and Hardware Requirements 22 Table 7 minxofedinstall Return Codes 26 Table 8 Supported ConnectX Port Configurations 37 Table 9 Software and Hardware Requirements 38 Table 10 MCX312B XCCT Specifications Table 61 Table 11 MCX312C XCCT Specifications Table 62 Table 12 MCX311A XCCT Specifications Table 63 Table 13 Physical and Logical Link Indication 64 Table 14 Connector Pinout 68 6 Mellanox Technologies J ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 List of Figures Figure 1 BracketScrews aaa 16 Figure 2 Module With Locking Mechanism Closed 17 Figure 3 Module With Locking Mechanism Open 17 Higur 4 Device ec an nus ein RENE SS ns Se eke eS 19 Figure 5 Support Download Assistant 57 Figure 6 Mechanical Drawing of the Dual port MCX312B XCCT 64 Figure 7 Mechanical Drawing of the Dual port MCX312C XCCT 65 Figure 8 Dua
30. and Transceivers a For both operational and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 4 Adapter LED Operation There are two I O LEDs per port See Table 13 for different LED functions Mellanox Technologies 63 1 1 Specifications Table 13 Physical and Logical Link Indication LED Function Constant on indicates a good physical link Ifneither LED is lit then the physical link has not been established Green physical link Yellow logical data A blinking yellow indicates activity data transfer activity link Stays off when there is no activity The short bracket has the same port LED footprints as the tall bracket 7 5 Board Mechanical Drawing and Dimensions All dimensions are in millimeters d the mechanical tolerances are 0 1mm Figure 6 Mechanical Drawing of the Dual port MCX312B XCCT 142 24 49 1 40 84 53 59 64 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Note The above mechanical drawing also applies to the MCX311A XCCT not including Port 2 Figure 7 Mechanical Drawing of the Dual port MCX312C XCCT 142 24 ie TEE 238 Hage T nun CR im 3 gui ml E m EH m z u E nana 2 585 B B
31. b Unzip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4103 pci Image type ConnectX 3 Pro FW Version 2 9 4100 Device ID 4103 Mellanox Technologies 57 1 1 Troubleshooting 6 6 1 58 Mellanox Technologies Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged working after installing another or deleted Reboot the server Adapters stopped Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS adapter Ensure that adapter driver s is loaded Try another port on the switch Link indicator light is Make sure the cable is securely attached off Check your are using the proper cables that do not exc
32. backported to your kernel Theraw package un backported source files are placed under usr src ofa kernel lt ver gt The script openiba is installed under etc init d This script can be used to load and unload the software stack The script connectx port config is installed under sbin This script can be used to configure the ports of ConnectX network adapter cards to Ethernet and or InfiniBand For details on this script please see Section 4 1 13 Port Type Management on Linux The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED instal lation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file 15 used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d 34 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc fIPOoIB configuration files are included ifcfg ib lt n gt files will be installed under etc sysconfig network scripts ona RedHat machine The installation process unlimits the amount of
33. dirt or foreign matter in the module or in the cage Figure 2 Module With Locking Mechanism Closed men tt MEMTTO2A SR Cleve 2 CHR 940 LM ANAN STECHHOLOSICS MPMITOLA GR EE Che mn 3 Insert module into the adapter card module 4 Close the locking Mechanism To remove the module from the cage 1 Unlock the locking mechanism by opening the handle Mellanox Technologies 17 1 4 Hardware Installation 3 6 2 3 7 3 7 1 2 Pull the module out of the cage Cable Installation 1 cables can be inserted or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated When installing cables make sure that the latches engage Always install and remove cables by pushing or pulling the cable and connector a straight line with the card 3 After inserting a cable into a port the Green LED indicator will light when the physical con nection is established that is when the unit is powered on and a cable 1s plugged into the port with the other end of the connector plugged into a functioning port See Section 7 4 Adapter LED Operation on page 63 4 After plugging in a cable lock the connector using the latching mechanism parti
34. from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the package The package 15 available at http www mellanox com gt Products gt Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Getthe Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci d Get the PSID firmware identification and programmed firmware version using the command Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for this product and are included here for illustration purposes only flint d dev mst mt4103 pci q Image
35. gt iso mnt 26 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Step 3 the installation script mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 64 root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 6418 mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX Starting MLNX OFED LINUX 2 0 2 0 0 installation Installing mlnx ofa kernel RP Preparing 27 HH H HH HH HH H HH HH mlnx ofa kernel HH H HH H HH H HH HH H HH HHHH HH HH H HH HH Installing kmod minx ofa kernel RPM Preparing 3 HH HH H HH H HHHH HH H HH HH kmod mlnx ofa kernel HH HH HH H H HH H HH HH Installing mlnx ofa_kernel devel RPM Preparing HH HH H HH HH mlnx ofa_kernel devel H HH HH H HH HH HH Installing kernel mft RPM
36. memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled 1 You run the installation script in default mode that is without the option without fw update 2 The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image If an adapter s Flash was originally programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image Incase your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file 4 1 10 Post installation Notes Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file 4 1 11 Updating Firmware After Installation In case you ran the mInxofedinstall script with the without fw update option and now you wish to manually update firmware on your a
37. performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products Software gt Windows SW Drivers Mellanox MLX4 EN Driver for VMware README Document no 3527 User Manual describing MLX4 EN driver for VMware features performance diagnostic tools content and configuration See http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers Performance Tuning Guidelines for Mellanox Network Adapters Document no 3368 Manual describes important tuning parameters and settings that can improve performance for Mellanox drivers IEEE Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega Bytes The use of Mb or Mbits small b indicates size in mega bits In this document PCIe is used to mean PCI Express Mellanox Technologies 9 Rev 1 4 Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox 1 Global
38. to protect from ESD shocks to the card server when plugging copper cables 2 1 Ethernet SFP Interface The network ports of the ConnectX 3 Pro adapter cards are compliant with the IEEE 802 3 Ethernet standards listed in Table 4 Features on page 12 The SFP port has one Tx Rx pair of SerDes Ethernet traffic is transmitted through the cards SFP connectors 22 PCI Express Interface The ConnectX 3 Pro adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists PCIe interface features PCIe Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 8 0GT s link rate x8 Auto negotiates to x8 x4 x2 or x1 Support for MSI MSI X mechanisms 23 I2C compatible Interface A three pin header on the adapter cards is provided as the I7C compatible interface See Figure 6 Mechanical Drawing of the Dual port MCX312B XCCT on page 64 for the location on the board 2 4 LED Interface There are two I O LEDs per port For LED specifications please refer to Section 7 4 Adapter LED Operation on page 63 14 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 3 Hardware Installation 31 System Requirements 3 1 4 Hardware system with a PCI Express x8 slot is required for installi
39. 64 bit machine the output will be AMD64 Step2 to MLNX WinOF for Windows Web page at http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Windows SW Drivers 38 Mellanox Technologies ConnectX 3 Pro Ethernet Single Dual SFP Port Adapter Card User Manual Rev 1 1 Step 3 Download the exe image according to the architecture of your machine see Step 1 The name of the exe is in the following format MLNX VPI WinOF lt ver sion All lt OS gt _ lt arch gt exe dows Installer 4 2 2 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open CMD console Click Start gt Run and enter CMD Step2 Enter the following command Step3 Click Next to create a server image Welcome to the InstallShield Wizard for MLNX_VPI The InstallShield R Wizard will install MLNX_YPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Ce mes Da Mellanox Technologies 39 Rev 1 4 Driver Installation Step 4 Click Change and specify the location in which the files are extracted to la Network Location Specify network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of a
40. Installation The number of uplinks claimed by MLX4 EN driver should displayed In Non Multifunction Mode port 2 is identified as a pseudo device Therefore devices 7 are not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 10 Ar 4 4 2 Installing and Running the offline bundle Driver on ESXi 5 x 1 Copy the offline bundle zip file to ESXi 5 0 machine and extract its contents 2 You can install the driver in one of the following ways a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4 en driver offline bundle package Run gt esxcli software vib install d lt path gt mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip c Reboot ESXi server The driver will be loaded automatically OR a Update the driver Run gt esxcli software vib update n net mlx4 en d lt path gt mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4 en To query network uplinks installed on your
41. Mellanox TECHNOLOGIES ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual P N MCX312B XCCT MCX311A XCCT MCX312C XCCT Rev 1 1 www mellanox com Rev 1 4 THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATI
42. ON EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2015 Mellanox Technologies Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX amp MLNX OS TestX amp PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT HPC X Mellanox CloudX Mellanox Open Ethernet Mellanox PeerDirect TM Mellanox Virtual Modular SwitchTM MetroDXTM Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 887 ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Table of Contents Table of Contents sise s assasi sia 3 List of Tables en A ana 6 List OF Figures en ER Ie RO RO e 7 Revision ouv RE asa Ei 8 About this Manual ire sanl ia ea ie ld eli li A al ekle ea ia kk 9 Intended Audiehee o E li QUE p n COSE 9 Related Documentation 9 Document
43. Preferred DNS server Alternate DNS server validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog 7 Verify the IP configuration by running ipconfig from a CMD console ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix IP Address o s o m a ete lud 12563 Subnet Mask e 6 6 8 255 25500 Default Gateway 4 229 Port Type Management on Windows After installing Mellanox WinOF VPI for Windows on your machine you can change a port s pro tocol configuration The following steps describe how to configure the port type Step 1 Display the Device Manager and expand System Devices Mellanox Technologies 51 1 4 Driver Installation File Action View Help LIN E T 4 Network adapters Lu Broadcom BCM5709C NetXtreme Il GigE NDIS Client 48 Broadcom BCM5709C NetXtreme Il GigE NDIS Client 49 IBM USB Remote NDIS Network Device EF Microsoft Kernel Debug Network Adapter Ports COM amp LPT dan Print queues ER Processors Security devices lt gt Storage controllers Ei System devices k v Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties Select the Port Protocol tab from the Properties sheet Note The Port Protocol tab is displayed only if the NIC isa
44. VPI IB and ETH If the NIC is either only IB or ETH the tab will not be shown General Port Protocol Driver Details Events Resources Current Setting Eth 2 Eth Port Type Configuration HW Defaults Port 1 C IB ETH C AUTO Port 2 C IB ETH C AUTO Port Protocol Configuration This menu displays the adapter s port type and enables you to set the network protocols for the network adapter ports The network protocol is determined according to the NIC s Hardware Defaults port type You can choose the protocol explicitly by selecting the port type to InfiniBand IB or Ethernet Eth enahle Aita Sensinn nlease chonse Al IT If the Cancel 52 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Step 3 In this step you can perform two different functions a Choose the desired port protocol for the available port s and b activate or deactivate the WSD ND and or SDP ULPs Note IB must be always the first port in Port 1 If you choose ETH as your first port in Port 1 then the second port in Port 2 can be only ETH Note WSD is not supported in Windows 7 Consequently on this OS the WSD checkbox is grayed out and cannot be selected 43 Performance Tuning For guidelines on improving network adapter performance please r
45. X_WinOF To configure your setup to contain the logs option please run the following command after opening a CMD console MLNX VPI WinOF 4 40 0 All win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run the following command MLNX VPI WinOF 4 40 0 All win8 x64 exe v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Mellanox Technologies 41 Rev 1 4 Driver Installation Click Next in Welcome screen Welcome to the InstallShield Wizard for MLNX The InstallShield R Wizard will install MLNX on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Step 2 Read then accept the license agreement and click Next License Agreement Please read the Following license agreement carefully Copyright c 2005 2013 Mellanox Technologies 11 rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer inana in farm mint wanwa Avan tha alarm I accept the terms in the license agreement I do not accept the terms in the license agreement InstallShield 42 Mellanox Tech
46. apter 7 Specifications on page 61 July 2013 1 0 First Release 8 Mellanox Technologies About this Manual ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 This User Manual describes Mellanox Technologies ConnectX 3 Pro 10 Gigabit Ethernet Sin gle and Dual SFP port PCI Express x8 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet network and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual Document 2204UG User Manual describing the set of MFT firmware management tools for a single node See http www mellanox com gt Products gt Software gt Firm ware Tools MLNX EN for Linux README Driver Kit for Mellanox Adapter Cards with 10GigE Support Document no 2950 User Manual providing information on the MLNX EN Linux driver and instructions for installing the driver on Mellanox Con nectX 3 adapter cards supporting 10GigE See http www mella nox com gt Products gt Software Ethernet Drivers Linux Driver WinOF VPI for Windows User Manual Document no 3280 User Manual describing WinOF features
47. ard s port type If a valid cable or module is connected SFP or SFP with EEPROM in the cable module Set the port type to the sensed link type IB Ethernet Otherwise Set the port type as default Ethernet During driver run time Sense link every 3 seconds if no link is sensed detected fsensed set the port type as sensed 4 2 Windows Driver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com Download this package from the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet DriversInfiniBand VPI Drivers gt Windows SW Driver gt Download Follow the installation instructions included in the download package also available from the download page Hardware and Software Requirements Table 9 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Installer Privileges The installation requires administrator privileges on the target machine 4 2 1 Downloading MLNX WinOF Follow these steps to download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click start gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURE On an x64
48. are updates to your network adapter hardware it will ask you to reboot your machine The script adds the following lines to etc security limits conf for the userspace com ponents such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a user space application If desired tune the value unlimited to a specific amount of RAM For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM refer to the OFED User Manual See Table 2 Documents List on page 9 InfiniBand only Run the hca self test ofedutility to verify whether or not the Infini Band link is up The utility also checks for and displays additional information such as HCA firmware version e Kernel architecture 32 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Driver version Number of active ports along with their states Node GUID Note For more details on hca self test ofed seethefilehca self test readme under docs hca self test ofed Performing Adapter Device Self Test Number of CAs Detected d n n 2 Device Check Puman anan asas PASS nn x86 64 EOS DAT Versi
49. capacity is 4Kb ConnectX 3 Pro effectively addresses the increasing demand for an overlay net work enabling superior performance by introducing advanced NVGRE and VXLAN hardware offload engines that enable the traditional offloads to be per formed on the encapsulated traffic With ConnectX 3 Pro data center operators can decouple the overlay network layer from the physical NIC performance thus achieving native performance in the new network architecture Leveraging Data Center Bridging capabilities RoCE provides efficient low latency RDMA services over Layer 2 Ethernet Adapter functionality enabling reduced CPU overhead allowing more available CPU Using GPUDirect RDMA adapters can directly read and write CUDA host and device memory eliminating unnecessary system memory copies and CPU over head resulting in significant performance improvements Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10 40 56GbE The hardware based stateless offload engines in ConnectX 3 Pro reduce the CPU overhead of IP packet trans port Sockets acceleration software further increases performance for latency sen sitive applications Support for port based Quality of Service enabling various application require ments for latency and SLA ConnectX 3 Pro provides dedicated adapter resourcesand guaranteed isolation and protection for virtual machines within the server ConnectX 3 Pro SR IOV technol
50. ctX 3 Pro Ethernet Single Dual SFP Port Adapter User Manual Rev 1 1 A2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Figure 11 PCle Connector Pinout 5 2U PRSNT18 1 2 124 B m 4 4 sucuk TCK A5 y B8 SMDAT A6 BZ END 100 LAZ Tus LAR 53 TRST 433v FAS 2910 LAID 11 lt 12 nsyp A12 _ 813 REFOLK 14 REFCLK AlE B1 amp A15 BiZ A12 B181 A18 19 rsvp PAL B21 PERp1 LA21 B23 AZA 825 PERp2 25 825 GND PERn2 226 821 perpa AZ B2 ema A28 829 PERpa A23 lt 831 PRSNT2 Aai __ Rsvp PAR B35 PERp4 A25 B36 GND penna A35 B3Z pETps HARZ 22 nn Azo _ B41 ASL B42 pens amp 2_ ann PERp6 43 B44 PERNE ASA _ B45 PETp7 46 7 FAME B4Z GND
51. cular to the cable vendor When a logical connection is made the Yellow LED will light When data is being transferred the yellow led will blink See Section 7 4 Adapter LED Operation on page 63 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Identify the Card in Your System On Windows 1 Open Device Manager on the server Click start Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 Pro adapter card 3 Right click the mouse on your adapter s row and select properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or Hardware Ids Windows 2008 R2 from the Properties pull down menu Note Figure 4 15 an example using ConnectX 3 device This figure will be updated in future releases 18 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Figure 4 PCI Device General Protocol Driver Details Events Resources A Mellanox Connect lt 3 MT 04094 Network Adapter
52. d VeeR Receiver Ground Common with Transmitter Ground 12 RD Receiver Inverted DATA out AC Coupled 13 Receiver Non inverted DATA out Coupled 14 VeeR Receiver Ground Common with Transmitter Ground 4 15 VccR Receiver Power Supply 16 VecT Transmitter Power Supply 17 VeeT Transmitter Ground Common with Receiver Ground 4 18 TD Transmitter Non Inverted DATA in AC Coupled 19 TD Transmitter Inverted DATA in AC Coupled 20 VeeT Transmitter Ground Common with Receiver Ground a Circuit ground is internally isolated from chassis ground b Traurr is an open collector drain output which should be pulled up with a 4 7k 10k Ohms resistor on the host board if intended for use Pull up voltage should be between 2 0V to Vcc 0 3V A high output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed ing the preset alarm thresholds A low output indicates normal operation In the low state the output is pulled to 0 8V c Laseroutput disabled on TDIS gt 2 0 or open enabled on TDIS 0 8V d Should be pulled up with 4 7kQ 10kQ on host board to a voltage between 2 0V 3 6V MOD ABS pulls line low to indicate module is plugged in e LOS is open collector output Should be pulled up with 4 7kQ 10kO on host board to a voltage between 2 0V and 3 6V Logic 0 indicates normal operation logic 1 indicates loss of signal 68 Mellanox Technologies Conne
53. dapter card s you need to perform the following steps If you need to burn an Expansion ROM image please refer to OFED User Manual See Table 2 Documents List on page 9 Mellanox Technologies 35 J 1 4 Driver Installation Step 1 Step 2 Step 3 Step 4 The following steps also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site gt Support gt Firm ware Download Start mst hostl mst start Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine host1 mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded MST devices dev mst mt25418 pciconf0 dev mst mt25418 pci dev mst mt25418 pci msix0 dev mst mt25418 pci uar0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 PCI direct access bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is A0 PCI direct access bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix pci In the example listed above this will be dev mst mt25418 pci Burn firmware 1 Burning a firmware binary image using mst flint that is already i
54. e mnt minxofedinstall OPTIONS 24 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Options c config packages config file Example of the configuration file can be found under docs n net network config file Example of the network configuration file can be found under docs p print available Printavailablepackages forthecurrentplat form and create a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update force fw update Force firmware update force Force installation without querying the user 11 Install all 1 modules libibverbs libibu mad librdmacm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI tests MPI selector perftest sdpnetstat and libsdp srptools rds tools static and dynamic libraries hpc Install all kernel modules libibverbs libibumad librdmacm mft mstflint diagnos tic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector dynamic librar ies basic Install all kernel modules libibverbs libibu mad mft mstflint dynamic libraries msm Install all kernel modules libibverbs libibumad mft mstflint diagnostic tools OpenSM ib bonding dynamic libraries NOTE With msm flag the
55. ectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la LA m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de rayos 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec LA tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr
56. eed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but Check that the latest driver is loaded with no Check that both the adapter and its link are set to the same speed communication and duplex settings established ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 6 2 Linux cat etc issue uname a cat proc cupinfo grep model uniq Environment ofed info head 1 Information ifconfig a ethtool lt interface gt ethtool 1 interface of Mellanox port num ibdev2netdev Card Detection lspci grep Mellanox Download and install http www mellanox com con tent pages php pg management tools amp menu section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d lt mst device q ibstat Ports Information devinfo Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF Mellanox Technologies 59 J Rev 1 4 Troubleshooting 6 3 Windows From the Windows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu A
57. efer to the performance tun ing guidelines for Linux Windows environment in the document http www mellanox com related docs prod software Performance Tuning Guide for Mellanox Network Adapters pdf 4 4 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ESXi Server software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt gt VMware Drivers gt Download Follow the installation instructions included in the download package also available from the download page 4 4 1 Installing and Running the Driver on ESXi 5 x 1 Log into the VMware ESXi server machine as root 2 Youcan either a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4 en driver VIB package Run gt esxcli software vib install v vib url c Reboot ESXi server The driver will be loaded automatically OR a Update the driver Run gt esxcli software vib update v vib url b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4 en gt To query network uplinks installed on your machine run gt esxcli network nic list Mellanox Technologies 53 J 1 4 Driver
58. er LED Operation 63 7 5 Board Mechanical Drawing and Dimensions 64 76 Bracket Mechanical Drawing 65 7 7 Regulatory Statements 66 Appendix A Interface Connectors Pinout 67 PCI Express x8 Connector Pinout 69 A2 I2C compatible Connector Pinout 69 Appendix Finding the GUID MAC and Serial Number on the Adapter Card 71 B 1 Retrieving Adapter Card GUID 71 Appendix Safety Warnings 73 Appendix D Avertissements de s curit d installation Warnings in French 75 Appendix E Sicherheitshinweise Warnings in German TT Appendix Advertencias de seguridad para la instalaci n Warnings in Spanish 79 Mellanox Technologies 5 Rev 1 1 List of Tables Table 1 Revision History Table 8 Tabl 2 Documents emek e ee er ne resp ua i 9 Table 3 Dual port 10 Gigabit Ethernet Adapter Cards 11 Table4 Features ss E WERE E apu ne OM ONG LAN e amg e ini 12 Table 5 Software and Hardware Requirements 20 Table 6 Software
59. er cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot 16 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 A lesser width adapter can be seated into a greater width slot x4 in 8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter edge with the PCI Express connector slot 4 Applying even pressure at both corners of the card insert the adapter card into the slot until it 15 firmly seated When the adapter 15 properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis Do not use excessive force when seating the card as this may damage the system or the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to www mellanox com gt Prod ucts Cables and Transceivers 3 6 1 Inserting the Optical Transceiver Module To insert the module into the cage 1 Open the module s locking mechanism see Figure 2 and Figure 3 2 Make sure that the male connectors on the module will align with the female connectors inside of the cage Also check that there 1s no
60. er nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel support sh script located under the docs directory Usage mlnx add kernel support sh m mlnx ofed path to MLNX OFED directory make iso make tgz gt gt make iso Create MLNX OFED ISO image gt make tgz Create MLNX OFED tarball Default t tmpdir local work dir gt kmp gt v verbose Example The following command will create a MLNX OFED LINUX ISO image for RedHat 5 6 under the tmp directory MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 docs mlnx add kernel support sh i mnt OFED LINUX 1 5 3 rhel5 6 x86 64 150 All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 150 4 1 8 2 Installation Script The usage of the installation script is described below You will use it during the installation pro cedure described in Section 4 1 8 4 Installation Procedure on page 26 1 The firmware will not be updated if you run the install script with the without fw update option Mellanox Technologies 23 1 1 Driver Installation Usag
61. gt Programs and Features gt MLNX VPI gt Uninstall NOTE This requires elevated administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start Programs gt Mellanox Technologies gt MLNX WinOF gt Uninstall MLNX WinOF 4 2 7 2 Unattended Uninstall Touninstall MLNX WinOF in unattended mode perform the following Step 1 Open CMD console Step 2 Uninstall the driver Run MLNX VPI WinOF 4 40 0 All win8 x64 exe S x v qn 4 2 8 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine To obtain the MAC address Step 1 Open a CMD console Step 2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters gt To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices 1 E Net Metwor gt v Search Network Connections 2 Organize v Device Y Ethernet Broadcom 5709 NetXtre x Ethernet 2 Ethernet 3 Ethernet 4 M U Local Area Connection IBM USB Rernote Mellanox Technologies 49 Rev 1 4 Driver Installation
62. he attached network type If no link is detected the driver retries link sensing every few seconds Table 8 lists the ConnectX port configurations supported by VPI Table 8 Supported ConnectX Port Configurations Port 1 Configuration Port 2 Configuration ib ib ib eth eth eth Note that the configuration Port eth and Port2 ib is not supported The port link type can be configured for each device in the system at run time using the sbin connectx port config script This utility will prompt for the PCI device to be modified 1f there 15 only one it will be selected automatically In the next stage the user will be prompted for the desired mode for each port The desired port configuration will then be set for the selected device This utility also has a non interactive mode sbin connectx port config d device PCI device ID gt c conf lt portl port2 gt 4 1 14 Auto Sensing Auto Sensing enables the NIC to automatically sense the link type InfiniBand or Ethernet based on the link partner and load the appropriate driver stack InfiniBand or Ethernet For example if the first port is connected to an InfiniBand switch and the second to Ethernet switch the NIC will automatically load the first switch as InfiniBand and the second as Ethernet Mellanox Technologies 37 J Rev 1 4 Driver Installation 4 1 14 1 Enabling Auto Sensing Upon driver start 1 Sense the adapter c
63. his step requires you to reboot the machine at the end of the installation process InstallShield 4 Step 6 Select a Complete or Custom installation follow Step a on page 45 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete All program features will be installed Requires the most disk space Choose which program Features you want installed and where they will be installed Recommended For advanced users InstallShield 44 Mellanox Technologies ConnectX 3 Pro Ethernet Single Dual SFP Port Adapter User Manual Rev 1 1 Select the desired feature to Install OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically Performances tools install the performance tools that are used to measure the InfiniBand performance in user environment Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment SDK contains the libraries and DLLs for developing InfiniBand application over IBAL Documentation contains the User Manual and Installation Guide ND FLIR DLLs contains the files for standalone installation of the mlx4nd provider Custom Setup Select the program features you want installed Click on an icon in the list below to change how a feature is installed Fea
64. l gt lt CPU arch gt iso You can download it from http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers Use the md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result to the value provided on the download page host1 md5sumMLNX LINUX lt ver gt lt 0S label gt iso Installing Mellanox OFED Mellanox OFED includes an installation script called minxofedinsta11 which performs the fol lowing Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX OFED LINUX binary RPMs if they are available for the current kernel 22 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Identifies the currently installed InfiniBand and Ethernet network adapters and automat ically upgrades the firmware 4 1 8 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages gt Pre existing configuration files will be saved with the extension conf rpmsave A If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy 15 to mount the ISO image on one of the clust
65. l portBracket 0 65 Figure 9 Single port Bracket 66 Figure 10 Rear View of Module With Pin Placement 67 Figure 11 PCIe Connector Pinout 69 Figure 12 Compatible Connector Plug and Pinout 69 Figure 13 Board Label Example 71 Mellanox Technologies 7 Rev 1 4 Revision History This document was printed on February 22 2015 Table 1 Revision History Table Date Rev Comments Changes February 2015 1 1 Added MCX311A XCCT and MCX312C XCCT to the UM Added note on bracket replacement See Section 3 4 Bracket Installation Instructions on page 15 Added Section 4 3 Performance Tuning on page 53 Added Performance Tuning Guidelines to Related Documenta tion on page 9 Added the following note to Chapter 5 Updating Adapter Card Firmware on page 56 Note The shown versions and or parameter values in the exam ple below may not reflect the latest or actual values for this prod uct and are included here for illustration purposes only Specified max power available through SFP ports in all specifi cation tables See Chapter 7 Specifications on page 61 Updated link to approved cables list in all specification tables See Ch
66. lation completion you can verify the successful addition of the network card s through the Device Manager To see the Mellanox network adapter device and the Ethernet or IPoIB network device depending on the used card for each port display the Device Manager and expand System devices or Network adapters File Action View Help Mellanox ConnectX 3 Ethernet Adapter 22 Microsoft Kernel Debug Network Adapter 4 Other devices jg Unknown device Y Ports COM amp LPT p Print queues gt Processors b lt Storage controllers j System devices ACPI Fixed Feature Button Composite Bus Enumerator li Direct memory access controller yi High precision event timer Intel R 5520 5500 X58 Hub Control Status and RAS Registers 3423 El Intel R 5520 5500 X58 Hub GPIO and Scratch Pad Registers 3422 Ml Intel R 5520 5500 X58 Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 Hub PCI Express Root Port 3 340A Ml Intel R 5520 5500 X58 Hub PCI Express Root Port 7 340E yili Intel R 5520 5500 X58 I O Hub System Management Registers 342E Ml Intel R 5520 5500 X58 Hub to ESI Port 3406 Intel R 82801 PCI Bridge 244E 288 Intel R ICH10 Family PCI Express Root Port 1 3440 88 Intel R ICH10 Family PCI Express Root Port 5 3448 E Intel R ICH10R LPC Interface Controller 3416 Microsoft ACPI Compliant Syste
67. lt is a new net device appearing in ifconfig a output 4 1 4 Unloading the Driver To unload the Ethernet driver run modprobe mlx4 en Mellanox Technologies 21 J Rev 1 4 Driver Installation 4 1 5 Uninstalling the Driver To uninstall the mlnx_en driver run gt sbin mlnx uninstall sh 41 6 Hardware and Software Requirements 4 1 7 Table 6 Software and Hardware Requirements Requirements Description Platforms For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for 1GB Installation Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to the Mellanox OFED Release Notes file Installer Privileges The installation requires administrator privileges on the target machine Downloading Mellanox OFED Step 3 Step 4 Step 5 Venify that the system has a Mellanox network adapter HCA NIC installed by ensuring that you can see ConnectX entries in the display The following example shows a system with an installed Mellanox HCA host1 lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27520 ConnectX 3 Pro Download the ISO image to your host The image s name has the format MLNX OFED LINUX ver OS labe
68. m Microsoft Generic Compliant Device yi Microsoft System Management BIOS Driver 188 Microsoft Virtual Drive Enumerator Numeric data processor li PCI Express Root Complex 4 2 6 OpenSM Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as it requires the GUID parameter to decide on which port to work Setting OpenSM upon setup results in it working only for the first port and not for the others gt To run OpenSM as a service assuming the package was installed in the default path use sc create OpenSM1 binPath c Program Files Mellanox MLNX VPI IB Tools opensm exe service start auto gt To start the service run sc start opensm 48 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 For further information please refer to the OpenSM Subnet Manager chapter in the User Manual 42 7 Uninstalling MLNX WinOF 4 2 7 1 Attended Uninstall Touninstall MLNX WinOF on a single node perform one of the following options 1 Click Start gt Control Panel
69. machine run gt esxcli network nic list The number of uplinks claimed by MLX4 EN driver should be displayed In Non Multifunction Mode port 2 15 identified as a pseudo device Therefore devices not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLL on page 10 of the VMware User Manual See Table 2 Documents List on page 10 44 3 Removing the VIB offline bundle Driver gt Toremove the VIB offline bundle driver package from the ESXi server machine run gt esxcli software vib remove n net mlx4 en 54 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 4 5 FlexBoot FlexBoot supports remote Boot over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot software is based on the open source iPXE project see www ipxe org For more information go to http www mellanox com gt Products Software gt Ethernet Drivers gt Download Mellanox Technologies 55 Rev 1 4 Updating Adapter Card Firmware 5 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally and the most recent firmware can be obtained
70. ng HH A A RAAHAA HERRE EHRRRHHHRHH opensm static HH Preparing HH A RAAHAA HERRE EHER REHHRHH opensm static Preparing con pat dapl HH HH 4444 HH HH H H H HH H H HH HH Preparing compat dapl Preparing HH E RAHA EER RE EHRERRHHHRHH compat dapl devel Prepar ng HH A RA HRR HERRE EHER R HHHRHH compat dapl devel Prepar ng dapl HH HH 4444 HH H HH H H H H HH H H HH HH Prepar ng HH AE AE A HAE HE AE HAE FEB HE HEHHEH 1 HH H HH HHHH H HH H H H H HH H H HH HH Prepar ng dapl deve HH HH HH H H H HH HH Prepar ng
71. ng the card 3 1 2 Operating Systems Distributions Please refer to Section 1 3 Operating Systems Distributions on page 13 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation on page 20 3 2 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make sure to use only insulated tools Verify that the system 15 powered off and is unplugged SUN It is strongly recommended to use ESD strap or other antistatic devices 3 3 Pre installation Checklist 1 Verify that your system meets the hardware and software requirements stated above 2 Shut down your system if active 3 After shutting down the system turn off power and unplug the cord 4 Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card 1f damaged 3 4 Bracket Installation Instructions The card is usually shipped with a tall bracket installed If this form factor is suitable for your requirements you can skip the remainder of this section and move to Section 3 5
72. nologies ConnectX 3 Pro Ethernet Single Dual SFP Port Adapter User Manual Rev 1 1 Step 3 Select the target folder for the installation Destination Folder Click Next to install to this Folder or click Change to install to a different Install MLNX VPI to C Program Files Mellanox MLNX_ PT InstallShield Step 4 The firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card in this case the firmware will not be updated Ifthe user has a standard Mellanox card and the firmware version 15 older than the one specified in WinOF Installation Guide 4 40 the firmware will be updated accordingly However if the user has both OEM card and Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the 5 firmware version Recommended Upgrading the Firmware version will reboot your machine Note One or more of your adapters has an old firmware version We recommend upgrading to a newer firmware version to enable improved functionality and support driver s capabilities InstallShield Mellanox Technologies 43 Rev 1 4 Driver Installation 5 Configure your system for maximum performance by checking the maximum performance box Maximum Performance Check this box to configure your system for maximum performa Y Check this box to configure your system for maximum performance Recommended Note T
73. nstalled on your machine Please refer to MSTFLINT README txt under docs 2 Burning a firmware image from a mlx file using the mixburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci fw mnt firmware fw 25408 fw 25408 rel mlx Reboot your machine after the firmware burning 15 completed 36 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 4 1 12 Uninstalling Mellanox Use the script usr sbin ofed uninstall sh to uninstall the Mellanox OFED package The script is part of the scripts RPM 4 1 13 Port Type Management on Linux ConnectX ports can be individually configured to work as InfiniBand or Ethernet ports By default both ConnectX ports are initialized as InfiniBand ports If you wish to change the port type usethe connectx port config script after the driver 15 loaded Running sbin connectx port config s will show current port configuration for all ConnectX devices Port configuration is saved in the file etc infiniband connectx conf This saved configuration is restored at driver restart only if restarting via etc init d openibd restart Possible port types are eth Ethernet jb Infiniband auto Link sensing mode Detect port type based on t
74. nstructions Preparing HH HH mxm HH HH 4444 HH HH HH HH HH Preparing HH HH HH openshmem HH HH HH HH HH HH Preparing HH HHH HH HH HH 4444 HH HH HH HH Preparing HH HH HH mvapich2 gcc HH HH HH HH HH HH Preparing HHH HH HHH HH openmpi gcc T1HBHBHUHHBWHHSHHHUUE BSHHBHHU H H AE HEH HE E AE HBHHBHE B HE Preparing HHH HH HH HH mpitests mvapich2 gcc HH HH HH HH HH HH HH Preparing HHH HH HH mpitests openmpi gcc HH HH He 41 4E 4L HH HH HH HH Preparing HH HH HH mlnxofed docs HH HH HH HH HH HH HH Mellanox Technologies 31 J 1 4 Driver In
75. ogy provides dedicated adapter resources and guaranteed isolation and protection for virtual machines VM within the server I O virtualization with ConnectX 3 Pro gives data center managers better server utilization while reducing cost power and cable complexity A consolidated compute and storage network achieves significant cost perfor mance advantages over multi fabric networks Standard block and file access protocols can leverage RDMA for high performance storage access 12 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 1 3 Operating Systems Distributions Citrix XenServer 6 1 RHEL CentOS 5 X and 6 X Novell SLES10 SP4 SLESII SPI SLES 11 SP2 OEL Fedora 14 15 17 Ubuntu 12 04 Windows Server 2008 2012 FreeBSD OpenFabrics Enterprise Distribution OFED OpenFabrics Windows Distribution WinOF VMware ESXi 4 x and 5 x 1 4 Connectivity nteroperable with 10GbE switches Passive copper cable with ESD protection Powered connectors for optical and active cable support QSFP to SFP connectivity through QSA module Passive copper cable with ESD protection Mellanox Technologies 13 J Rev 1 4 Interfaces 2 Interfaces Each adapter card includes the following interfaces Ethernet SFP Interface PCI Express Interface 2C compatible Interface LED Interface The adapter cards include special circuits
76. on RR E TTL T T TT MLNX OFED LINUX 2 0 2 0 0 OFED 2 0 2 0 0 2 6 32 279 e16 x86 64 Hoc Wier WN CEES nmana PASS Imire on CA 0 MEC v2 9 1000 Firmware Check on CA 0 NIC PASS ques o CA Hil MEC oo 3922 dil HOC Firmware Check on CA 1 NIC PASS Kost Driver Initiation oo 0000000000 PASS Numb ero EE Ge e lt A Gey e 4 Port State of Port 1 on CA 80 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 0 NIC UP 1X QDR Ethernet Port State of Port 1 on CA 1 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 1 NIC UP 1X QDR Ethernet Error Counter Check on CA 0 NIC NA Eth ports Error Counter Check on CA 1 NIC NA Eth ports Syslog Check umma PASS CA pnm 00 02 c9 03 00 07 4 8 Node GOD o CA sil Nie M 002022291108 20053500 ini DONE After the installer completes information about the OFED installation such prefix kernel version and installation parameters can be retrieved by running the com mand etc infiniband info 4 1 9 Installation Results Software The OFED and MFT packages are installed under the usr directory The kernel modules are installed under InfiniBand subsystem lib modules uname r updates kernel drivers infiniband mlx4 core driver lib modules lt ke
77. ports and PCI Express speed Each adapter comes with two bracket heights short and tall Table 3 Dual port 10 Gigabit Ethernet Adapter Cards MCX312B XCCT dual port card MCX312C XCCT dual port card MCX311A XCCT single port card Ordering Part Number OPN 10Gb s Data Transmission Rate single port SFP Number of ports dual port SFP PCI Express SerDes Speed PCIe 3 0 x8 8GT s RoHS R6 Adapter IC Part Number MT27528A0 FCCR XE 4103 for Physical Function Device ID decimal 4100 for Virtual Function Mellanox Technologies 11 1 2 Table 4 Features PCI Express PCIe 10 Gigabit Ethernet Memory Virtualized Overlay Networks RDMA over Converged Ethernet RoCE CPU offload GPUDirect RDMA Sockets Acceleration Quality of Service QoS Hardware based 1 0 virtualization SR IOV Storage Acceleration 1 1 Introduction Features and Benefits Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 4MB SPI Flash device 25 16 device by ST Microelectronics EEPROM accessible through the I C compatible interface The EEPROM
78. rnel version extra mlnx ofa kernel drivers net ethernet mellanox mlx4 mlx4 core ko Mellanox Technologies 33 J 1 4 Driver Installation mlx4 ib driver lib modules kernel version extra mlnx ofa kernel drivers infiniband hw mlx4 mlx4 ib ko mlx core driver lib modules kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx5 core mlx5 core ko mlx ib driver lib modules kernel version extra mlnx ofa kernel drivers infiniband hw mlx5 mlx5 ib ko IPoIB lib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko iSER lib modules uname r updates kernel drivers infiniband ulp iser ib iser ko elPolB lib modules uname r updates kernel drivers net eipoib eth ipoib ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko 1ib modules uname r updates kernel net rds rds tcp ko Kernel s modules location may vary depending on the kernel s configuration For example lib modules uname r extra kernel drivers net ethernet mellanox mlx4 mlx4 core The package kernel ib devel include files are placed under usr src ofa kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are
79. ry and Compliance Guide pdf Regulatory Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 62 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 7 3 MCX311A XCCT Specifications Table 12 MCX311A XCCT Specifications Table Size 5 77 in x 2 1 in 142 24 mm x 53 59 mm Physical Connector SFP 10Gb s Ethernet 0 10GBASE R and 1000BASE R Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 5 02W Max Power Passive Cables 6 17W Power and Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 Humidity 9096 relative humidity Air Flow 200LFMP EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Regulatory Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products Cables
80. ssign a file name and save Environment Information Download and install MFT http www mellanox com con tent pages php pg management tools amp menu section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d device q Ports Information vstat Download the latest firmware version using the PSID board Firmware Version ID Upgrade http www mellanox com supportdownloader flint d lt mst_device gt 1 firmware bin file b Event log viewer MST device logs mst start Collect log file mst status C Users Administrator gt flint d lt mst device dc gt dump configuration log e CAUsers Administrator mstdump mst device dc gt mstdump log 60 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 7 Specifications 7 1 MCX312B XCCT Specifications Table 10 MCX312B XCCT Specifications Table Size 5 77 in x 2 1 in 142 24 mm x 53 59 mm Physical Connector SFP 10Gb s Ethernet 0 10GBASE R and 1000BASE R Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 5 92W
81. stallation Step 4 Step 5 Step 6 Step 7 Device 05 00 0 05 00 0 Ethernet controller Mellanox Technologies MT26448 ConnectX EN 10GigE PCIe 2 0 5GT s rev b0 Link Width is not 8x PCI Link Speed 5Gb s Device 07 00 0 07 00 0 Ethernet controller Mellanox Technologies MT27500 Family ConnectX 3 Link Width 8x PCI Link Speed 5Gb s Installation finished successfully The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci 2 11 500 15 up to date Note To force firmware update use force fw update flag In case your machine has the latest firmware no firmware update will occur and the installation script will print at the end of installation a message similar to the following The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci 2 11 500 is up to date Note To force firmware update use force fw update flag In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file In case the installation script performed firmw
82. t the specified network location or click Cancel to exit the wizard Network location InstallShield Step5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location for the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX_VPI at the specified network location or click Cancel to exit the wizard Network location mee Change InstallShield 40 Mellanox Technologies ConnectX 3 Pro Ethernet Single Dual SFP Port Adapter User Manual Rev 1 1 Step 6 complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard lt Back Cancel 4 2 3 Installing MLNX WinOF This section provides instructions for two types of installation procedures Attended Installation An installation procedure that requires frequent user intervention Unattended Installation An automated installation procedure that requires no user intervention y Both Attended and Unattended installations require administrator privileges 4 2 3 1 Attended Installation The following is an example ofa MLNX WinOF win8 x64 installation session Step 1 Double click the exe and follow the GUI instructions to install MLN
83. this product and are included here for illustration purposes only Mellanox Technologies 71 Rev 1 1 72 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix C Safety Warnings 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed LA ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting Some copper cables are heavy and not flexible as such they should be carefully LA attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical A codes 8 Hazardous Radiation Exposure Caution
84. ture Description Performace Tools Subnet manager 5M Analyze Tools SDK Documentation ND FLTR DLLs This Feature requires 1164KB on your hard drive Install to C Program FilesiMellanoxVMLNX VPIAIBITools InstallShield Mellanox Technologies 45 1 4 Driver Installation Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Step 7 Click Finish to complete the installation InstallShield Wizard Completed The InstallShield Wizard has successfully installed Click Finish to exit the wizard You chose to run performance tuning The log can be found at C windows System32 LogFiles PerformanceTunin glog Cl Show release notes Cancel 46 Mellanox Technologies ConnectX 3 Pro Ethernet Single Dual SFP Port Adapter User Manual Rev 1 1 If the firmware upgrade fails following message will displayed InstallShield Wizard Completed The InstallShield Wizard has successfully installed VPI Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin g log Firmware upgrade Failed with error 1003
85. type ConnectX 3 Pro FW Version 2 9 4000 Device ID 4103 Chip Revision 0 Description Node Porti Port2 Sys image 60105 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT 1020110019 VSD PSID MT 1020110019 1 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 5 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the firmware release version and the card part number Note Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card 56 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Figure 5 Support Download Assistant AA Mellanox Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Switches Gateways 1 If a newer firmware version exists for your adapter card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above
86. von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschlie en Trennen von Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die 4 Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim 4 mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden This equipment should be installed in compliance with local and national elec LA trical codes Mellanox Technologies 77 Rev 1 1 8 Strahlenkontak 78 Mellanox Technologies ConnectX 3 Pro Ethernet Single and Dual SFP Port Adapter Card User Manual Rev 1 1 Appendix F Advertencias de seguridad para la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de con

10GbE User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents