Home

User Manual

Contents

1. Preparing a HH HH uud HH HH H HH HH H HH H HHHH HH H HH H HH HH dapl deve l static Prepar ng dapl utils 4 HH HH H H HH H H HH HH Preparing A HH HH HH HH HH H H HH H HH H HH HH perftest Preparing mstflint HH A AE EEE HAE AE A HAE HE EE HAE EH Preparing THHHHHHHBHHHHHBHHHHHHHHHHHHHHHHHBHHHHHHHHBHHHBSHBRHHI mft 4 4 HHH H H HH H H HH 4 Preparing HH A AE EEE HAE EE HAE AE HA EH srptools HH HH HH H H HH H H HH 4 Preparing 47 HH HH HH HH HH HHH H H HH H HH H HH HH rds tools Preparing A HH HHH HH H HH H HH H HHH H H HH H HH H HH 4 rds devel HH A A RA AHRR EER RE EHRRREHHRHH Preparing a HHH HH HH HH HH H H 4 H HH H HH HH ibutils2 HH AE HE EEE AE EE A HE EE EH HAE EE Preparing
2. libibmad static HE TETE EH A EE AE aE EH TETE a EH HH EEEH Preparing LA HH HH HH HH HH HH HH HH HH HH libibmad static Preparing ibsim 4 HH H H HH HH Preparing ibacm HH HH HHH HH HH Preparing librdmacm HH A AE EEE HAE AE A HAE HE EE HAE EH Preparing THHHHHHHBHHHHHBHHHHHHHHHHHHHHHHHBHHHHHHHHBHHHBSHBRHHI librdmacm HH AA EEE HAE AE A HAE HE AE HA R HHHRHH Preparing HH A AE EEE HAE EE HAE AE HA EH librdmacm utils HH AA EEE HAE EE HAE HE EE EH LEA R HHHRHH Preparing 4 THHHBHHHHBHHHHHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBSHRHHI librdmacm devel HH HH HH HH Preparing HH AE AE A HAE HE AE HA EH librdmacm devel HH HHHH HH 4 HH H H HH 4 Prepar ng HH HEH opensm libs
3. libibverbs devel Preparing THHHBHHHHHHHHHBHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHHE libibverbs devel static THHHHHHHHHHHHHHHHHHRHHBHHHHHHHHHHHHHHHHHHHRHHBHHRHRI Preparing libibverbs devel static THHHHHHHHHHHHHHHHHHRHHBHHHHHHHHHHHHHHHHHHHRHBHHRHRI Preparing HH A EEE AE EE A HAE EE EH HAE EE libibverbs utils HH Preparing libmverbs THHHHHHHBHHHHHBHHHHHHHHBHHHHHHHHBHHHHHHHHBHHHBSHRHHI Prepar 1 ng HH HH 4 4 H H HH H H 4 HH libmverbs Preparing HHH HH HHH HHH 4 libmlx4 HH HH HH HH HH HH Preparing libmlx4 HH HH 4 HH 4 HH Preparing HH A AE EEE HAE AE A HAE HE EE HAE EEHEEHE libmlx4 deve HH A RAHAA HERRE EHRRRFHHRHH Preparing HH NE libmlx4 devel HH A AE EEE AE AE A EH AE HA EH Preparing HH A R AHRR EER RE EHRERRHHHRHH libmlx5 THH
4. 17 3 5 Card Installation Instructions 18 3 6 Cables and Modules 19 3 6 1 CableInstallation ns ne ne an 19 3 7 Identify the Card in Your System 20 3 7 1 WiINdOWS ue a ae ne einen 20 3 52 SOM AAA ee 21 Chapter 4 Driver Installation 22 4 7 Linux Driver vno eis is 22 4 1 1 Hardware and Software Requirements 22 4 1 2 Downloading Mellanox 22 Mellanox Technologies 3 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 4 1 3 Installing Mellanox OFED 23 4 1 3 1 Pre installation Notes 23 415 2 Installation SCtipt uo a IE Pete um debe 24 4 1 3 3 mlnxofedinstall Return Codes 25 4 1 3 4 Installation Procedure 26 4 1 4 Installation Results 33 4 1 5 Post installation Notes 35 4 1 6 Updating Firmware After Installation 35 4 1 7 Uninstalling Mellanox 37 4 1 8
5. ib ib ib eth eth eth Note that the configuration Port1 eth and Port2 ib is not supported The port link type can be configured for each device in the system at run time using the sbin connectx port config script This utility will prompt for the PCI device to be modified if there is only one it will be selected automatically In the next stage the user will be prompted for the desired mode for each port The desired port configuration will then be set for the selected device This utility also has a non interactive mode sbin connectx port config d device PCI device ID gt c conf lt portl port2 gt 4 1 9 Auto Sensing Auto Sensing enables the NIC to automatically sense the link type InfiniBand or Ethernet based on the link partner and load the appropriate driver stack InfiniBand or Ethernet For example if the first port is connected to an InfiniBand switch and the second to Ethernet switch the NIC will automatically load the first switch as InfiniBand and the second as Ethernet Mellanox Technologies 37 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 4 1 9 1 Enabling Auto Sensing Upon driver start 1 Sense the adapter card s port type If a valid cable or module is connected QSFP SFP or SFP with EEPROM in the cable module Set the port type to the sensed link type IB Ethernet Otherwise Set the port type as def
6. Card Detection lspci grep 1 Mellanox Download and install MFT http www mellanox com con tent pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d lt mst_device gt q ibstat Ports Information Ibv_devinfo Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF Mellanox Technologies 58 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 6 3 Windows From the Windows desktop choose the Start menu run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Environment Information Download and install http www mellanox com con tent pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d lt mst_device gt q Ports Information vstat Download the latest firmware version using the PSID board Firmware Version ID Upgrade http www mellanox com supportdo
7. AN 11 wAKES OBFF B13 REFOLK B14 Als B15 AlE B16 GND 236 e B1Z A12 B18 GND A18 B13 Rsvp 18 RB21 cup PERp1 21 B22 GND P Rn1 LA22 B23 PETp2 HAM 825 25 82 GND pene 2 2 AZ B28 422 829 A2 lt 31 pasos Aal BA PETp4 Rsvp PAR B35 PERp4 35 B36 GND PERn4 A B3Z HARZ B29 nn 29 _ B41 A41 B42 PAZ _ B43 ann Ag B45 pETp7 A 46 FAME BAL PERp7 A4L PRSNT2 PERNT Ada _849 GND GND 49 A 3 12C compatible Connector Pinout Figure 13 Compatible Connector Plug and Pinout Connector Pin Number Signal Name 1 SPSDA 2 SPSCL u 3 GND 4 NC 5 NC Mellanox Technologies 70 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix Finding the GUID MAC and Serial Number the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number the card MAC the Ethernet protocol and the card GUID for th
8. ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Table 4 Features Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10 40 56GbE The hardware based stateless off Sockets Acceleration load engines in ConnectX 3 Pro reduce the CPU overhead of IP packet transport Sockets acceleration software further increases performance for latency sensitive applications Support for port based Quality of Service enabling various application require Quality of Service QoS ments for latency and SLA Hardware based I O ConnectX 3 Pro provides dedicated adapter resourcesand guaranteed isolation virtualization and protection for virtual machines within the server ConnectX 3 Pro SR IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines VM within the server virtualization with ConnectX 3 Pro gives data center managers better server utilization while reducing cost power and cable complexity SR IOV A consolidated compute and storage network achieves significant cost perfor mance advantages over multi fabric networks Standard block and file access protocols can leverage InfiniBand RDMA for high performance stor age access Storage Acceleration 13 Operating Systems Distributions Citrix XenServer 6 1 RHEL CentOS 5 X and 6 X Novell SLES10 SP4 e SLESII SPI SLES 11 SP2 OEL Fedor
9. 4 2 3 2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter Step 2 Install the driver Run gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn Step 3 Optional To configure your setup to contain the logs option please run the following com mand MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v l vx LogFile Step 4 Optional If you do not want to upgrade your firmware version run the following command gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h 4 2 4 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX_WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version In Windows 2012 MLNX WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade Mellanox Technologies 47 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 4 2 5 Installation Results Upon installation completion you can verify the successful addition of the network card s through the Device M
10. Preparing opensm libs Preparing opensm H HH HH HHH H H HH H H 4 HH Mellanox Technologies 29 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Preparing sm devel HHH HH HH HH H H H HH H H 4 HH Preparing HH HAE AA EEE HAE AE A HAE HE AE HA EH opensm devel Preparing opensm static HH A AE EEE HAE AE A HAE HE AE HA EH Preparing opensm static HH AA EEE HAE AE A HAE HE EE HAE EH Preparing
11. cards QSFP connectors For connecting to an SFP interface you can use Mella nox QSA QSFP to SFP adapter module to be ordered separately Mellanox Technologies 14 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 2 3 PCI Express Interface The ConnectX 3 Pro adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists PCIe interface features PCIe Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 or 8 0GT s link rate x8 Auto negotiates to x8 x4 x2 or Support for MSI MSI X mechanisms 24 C compatible Interface A three pin header on the adapter cards is provided as the I C compatible interface See Figure 8 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card on page 65 for the location on the board 2 5 LED Interface There are two LEDs per port For LED specifications please refer to Section 7 3 Adapter Led Operations on page 63 Mellanox Technologies 15 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 3 Hardware Installation 31 System Requirements 3 1 1 Hardware A system with a PCI Express x8 slot is required for installing the card 3 1 2 Operating Systems Distributions Please refer to Section 1 3
12. gt esxcli software vib remove n net mlx4 core Reboot the server 4 5 FlexBoot FlexBoot supports remote Boot over InfiniBand BoIB and over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot soft ware is based on the open source iPXE project see www ipxe org For more information go to http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Download Mellanox Technologies 54 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 5 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally and the most recent firmware can be obtained from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how t
13. 0S label gt lt CPU arch gt iso mnt Mellanox Technologies 26 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step3 Run the installation script mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 64 root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 641 mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX Starting MLNX OFED LINUX 2 0 2 0 0 installation Installing mlnx ofa kernel RP Preparing 27 HH H HH HH HH H HH HH mlnx ofa kernel HH H Het HH HH H HH H HH HH H HH HHHH HH HH H HH 4 Installing kmod minx ofa kernel RPM Preparing 22 HHH HH HH H HH H HHHH HH H HH HH kmod mlnx ofa_kernel HHH HH Het HHH HH 4 H H HH H HH HH Installing mlnx ofa_kernel devel RPM Preparing HHH HHH HH HHH HH HHH H HHH HHH HH HH mlnx ofa_kernel devel H HH HH H HH HHH Het HH HH Installing kernel mft
14. 188 Microsoft Virtual Drive Enumerator Numeric data processor li PCI Express Root Complex 4 2 6 OpenSM Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as it requires the GUID parameter to decide on which port to work Setting OpenSM upon setup results in it working only for the first port and not for the others gt To run OpenSM as a service assuming the package was installed in the default path use sc create OpenSM1 binPath c Program Files Mellanox MLNX VPI IB Tools opensm exe service start auto gt To start the service run sc start opensm Mellanox Technologies 48 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 For further information please refer to the OpenSM Subnet Manager chapter in the User Manual 4 2 7 Uninstalling MLNX WinOF 4 2 7 1 Attended Uninstall gt To uninstall MLNX WinOF on a single node perform one of the following options 1 Click Start gt Control Panel gt Programs and Features gt MLNX VPI gt Uninstall NOTE This requires elevat
15. Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec LA tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos Mellanox Technologies 78 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 PRODUCTO LASER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 79
16. HH HH HH Preparing HH A AE EEE AE AE A HAE HE AE EH BEA EH infiniband diags HH A RA ERRER RE EHRRREHHRHH Preparing qperf HH uud HH HHH HH HH Preparing fca HH HHHH HH 4 HH H H HH HH INFO updating IMPORTANT NOTE The FCA Manager and FCA MPI Runtime library are installed in opt mellanox fca directory The FCA Manager will not be started automatically To start FCA Manager now type etc init d fca managerd start There should be single process of FCA Manager running per fabric To start FCA Manager automatically after boot type etc init d fca managerd install service Check opt mellanox fca share doc fca README txt for quick start instructions Preparing HH HHH HHH HH mxm 4 HH LULU HH HH HH HH HH Preparing HH HHH 4 HH openshmem 4 HH HH HHH HH HH HH Preparing HHH HH HHH HHH HH HH 4 HH HHH HH HH H
17. 2 HH H HH HHHH H HH H HH H HHH H H HH H HH H HH HH ibutils Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Preparing 4 cc mgr HH HHHH HH 4 HH H H HH HH Preparing Ad HH HH HH HH HHH H HH H HH H HH HH dump pr HA HAL A HAE HAE A A LA LL Preparing THHHBHHHHBHHHBHHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBRHRHHI ar mgr HH HH HH HH Preparing ibdump HH HH HHH
18. Details Events Resources _ A Mellanox Connect lt 3 VPI MT 04099 Network Adapter en Property Hardware Ids Value PCIWEN_15B38DEV_10038SUBSYS_005015B3 amp REV_00 PCINVEN_15B3 amp DEV_1003 amp 5UBSY S_00501563 PCINVEN 15B3 amp DEV 1003 amp CC 028000 15B3 amp DEV 1003 amp CC 0280 1 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string 15B3 amp DEV 1003 VEN is equal to 0x 15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID Mellanox Technologies 20 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter User Manual Rev 1 4 y AA 3 7 2 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies Mellanox Technologies 21 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 4 Driver Installation The below sections are examples using ConnectX 3 device and will be updated to include ConnectX 3 Pro in future releases 4 1 Linux Driver For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox web site at http www mellanox com gt Products gt Soft ware gt InfiniBand VPI Dri
19. Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual Document 2204UG User Manual describing the set of MFT firmware management tools for a single node See http www mellanox com gt Products gt Software gt Firm ware Tools Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX OFED WinOF VPI for Windows User Manual Document no 3280 User Manual describing WinOF features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Windows SW Drivers Performance Tuning Guidelines for Mellanox Network Adapters Document no 3368 Manual describes important tuning parameters and settings that can improve performance for Mellanox drivers Mellanox InfiniBand OFED Driver for VMware vSphere 5 x User Manual Document no 3464 User Manual describing OFED driver for VMware features per formance InfiniBand diagnostic tools conte
20. on page 45 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete All program Features will be installed Requires the most disk space Choose which program features you want installed and where they will be installed Recommended for advanced users InstallShield Mellanox Technologies 44 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Select desired feature install OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically Performances tools install the performance tools that are used to measure the InfiniBand performance in user environment e Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment SDK contains the libraries and DLLs for developing InfiniBand application over IBAL Documentation contains the User Manual and Installation Guide ND FLTR DLLs contains the files for standalone installation of the mlx4nd provider Custom Setup Select the program features you want installed Click on an icon in the list below to change how a feature is installed Feature Description Performace Tools Subnet manager 5M Analyze Tools SDK Documentation ND FLTR DLLs This Feature requires 1164KB on your hard drive Install
21. 2015 Table 1 Revision History Table Date Rev Comments Changes October 2015 1 4 e Updated Device ID See Product Overview on page 11 2014 1 3 Added note on bracket replacement See Section 3 4 Bracket Installation Instructions on page 16 April 2014 1 2 Added Section 4 3 Performance Tuning on page 53 Added Performance Tuning Guidelines to Related Documenta tion on page 9 December 2013 1 1 Added the following note to Chapter 5 Updating Adapter Card Firmware on page 55 Note The shown versions and or parameter values in the exam ple below may not reflect the latest or actual values for this prod uct and are included here for illustration purposes only Specified max power available through QSFP ports in all speci fication tables See Chapter 7 5 on page 60 Updated link to approved cables list in all specification tables See Chapter 7 Specifications on page 60 June 2013 1 0 First Release Mellanox Technologies 8 About this Manual ConnectX 3 Pro Ethernet Dual SFP Port Adapter User Manual Rev 1 4 This User Manual describes Mellanox Technologies 9 3 Pro VPI Single and Dual QSFP port PCI Express x8 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documenta tion
22. 6 For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM refer to the OFED User Manual See Table 2 Documents List on page 9 Step 7 InfiniBand only Run the hca_self_test ofed utility to verify whether or not the Infini Band link is up The utility also checks for and displays additional information such as firmware version Kernel architecture Mellanox Technologies 32 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Driver version Number of active HCA ports along with their states Node GUID Note For more details on hca_self_test ofed see the file hca_self_test readme under docs hca self test ofed Performing Adapter Device Self Test Number of CAs Detected d n n 2 Device Check Puman anan asas PASS A ans nn 86_64 HOS CRD IV Version nn am MLNX OFED LINUX 2 0 2 0 0 OFED 2 0 2 0 0 2 6 32 279 e16 x86 64 log a Wier WN CEES nmana PASS Imire on CA 0 MEC v2 9 1000 Firmware Check on CA 0 NIC PASS ques o CA Hil MEC oo Wo dil 0 Firmware Check on CA 1 NIC PASS Kost Driver Initiation oo 0000000000 PASS Number of CA POrtS ACECIVE EE TEE rae 4 Port State of
23. Mellanox InfiniBand OFED 54 4 5 JFIEXBOOb iesus she slid we aed Mae aa a ee 54 Chapter 5 Updating Adapter Card Firmware 55 Chapter 6 Troubleshooting 57 6 1 General i e ee et ace Sn hin etek HE D De 57 6 2 Dinux ee Re rer E eL EE 58 63 Windows eR en het 59 Chapter 7 Sp cifications 60 71 MCX353A FCCT Specifications 60 7 2 MCX354A FCCT Specifications 61 7 3 Adapter Led Operations 63 7 4 Board Mechanical Drawing and Dimensions 64 7 5 Bracket Mechanical Drawing 66 7 6 Regulatory Statements 67 Appendix Interface Connectors Pinout 68 QSFP Connector Pinout 68 Mellanox Technologies 4 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 A 2 PCI Express x8 Connector Pinout 69 2C compatible Connector Pinout 70 Appendix Finding the GUID MAC and Serial Number on the Adapter Card 71 Appendix C Safety W
24. Operating Systems Distributions on page 13 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX_OFED for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation on page 22 3 2 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists 2 Make sure to use only insulated tools 3 Verify that the system is powered off and is unplugged 4 Itis strongly recommended to use an ESD strap or other antistatic devices 3 3 Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord c I Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged 3 4 Bracket Installation Instructions The card is usually shipped with a tall bracket installed If this form factor is suitable for your requirements you can skip the remainder of this section and move to Section 3 5 Card Installa tion Instructions on page 18 If you need to replace it with the short bracket t
25. THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct GPUDirect InfiniBridge InfiniHost InfiniScale Kotura logo MetroX MLNX OS PhyX ScalableHPC SwitchX TestX UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd CyPU ExtendX FabricIT FPGADirect HPC X Mellanox Care Mellanox CloudX Mellanox NEO Mellanox Open Ethernet Mellanox PeerDirect Mellanox Virtual Modular Switch MetroDX NVMeDirect StPU Switch IBTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 732 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Table of Contents Table of Contents u q ul e m S d Eist OF Tables s List OF Figures 555325 u u Quay a RR EUER ARS Revision SION Sea oer a ern Pelee eR ees Xd About this Manual xu ya y Sia asad nn nee SD eS T RD CU E
26. Technical Suppott pu an ran age ee Boge 10 Firmware and Software 10 Chapter 1 Introduction sa Hl li Product Overview sau bat se ys ada 11 1 25 Features and tak 12 1 3 Operating Systems Distributions 13 L4 Conn c vity Ebene UE WO BRS ae Hh Bee 13 Chapter 2 Interfaces x eu LA 2 1 InfiniBand Interface sinis a EUREN UR as 14 2 2 Ethernet Interface e S ARV ee a 14 2 3 PCL Express Interface eoe ee CERE CDU iUt 15 2 4 I2C compatible Interface 15 2 9 LED Interface u u aya mapasa dented SIS VS ee BA ai 15 Chapter 3 Hardware Installation 16 3 1 System Requirements 16 Sell Hardware eben bec terea aquy 16 3 1 2 Operating Systems Distributions 16 3 1 3 Software Stacks pes u u lai lisa Eger 16 3 2 Safety Precautions ia A l MN TER 16 3 3 Pre installation Checklist 16 3 4 Bracket Installation Instructions 16 3 4 1 Removing the Existing Bracket 17 3 4 2 Installing the New Bracket
27. and Compliance Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a The ConnectX 3 Pro adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mella nox InfiniBand product Mellanox Technologies 60 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 b For both operational and non operational states c Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 2 MCX354A FCCT Specifications Table 10 MCX354A FCCT Specifications Table Size 2 711 x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000BASE R 40GBASE R4 Protocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40 56Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 8 24 W Max Power Passive Cables 8 099 W Power a
28. performance please refer to the performance tun ing guidelines for a Linux Windows environment in the document_http www mellanox com related docs prod software Performance Tuning Guide for Mellanox Network Adapters pdf 4 4 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ES Xi Server software package available via the Mellanox web site at http www mellanox com Products gt Software gt InfiniBand VPI Drivers gt VMware Drivers gt Download Follow the installa tion instructions included in the download package also available from the download page 4 4 1 Installing Mellanox InfiniBand OFED Driver for VMware vSphere This chapter describes how to install and test the Mellanox InfiniBand OFED Driver for VMware vSphere package on a single host machine with Mellanox InfiniBand hardware installed The InfiniBand OFED driver installation on VMware ESXi Server 5 0 is done using VMware s VIB bundles Please uninstall any previous Mellanox driver packages prior to installing the new ver 2 sion Refer to Uninstalling Mellanox InfiniBand OFED Driver on page 54 To install the driver 1 Log into the ESXi5 0 server with root permissions 2 Install the driver gt esxcli software vib install d bundle file gt 3 Reboot the machine 4 Verify the driver was installed successfully i esxcli software vib list grep Mellanox n
29. www mellanox com gt Products gt Software gt InfiniBand VPI Drivers Mellanox Technologies 22 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Step 3 Use the md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result to the value provided on the download page host1 md5sumMLNX OFED LINUX ver OS label gt iso 4 1 3 Installing Mellanox OFED Mellanox OFED includes an installation script called m1nxofedinstall which performs the fol lowing Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX OFED LINUX binary RPMs if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automat ically upgrades the firmware 4 1 3 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Pre existing configuration files will be saved with the extension conf rpmsave Au If you need to install Mellanox OFED on an entire homogeneous cluster common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NF
30. 26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci cr0 2 11 500 is up to date Note To force firmware update use force fw update flag In case your machine has the latest firmware no firmware update will occur and the installation script will print at the end of installation a message similar to the following The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci 2 11 500 is up to date Note To force firmware update use force fw update flag In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file Step 4 In case the installation script performed firmware updates to your network adapter hardware it will ask you to reboot your machine 5 The script adds the following lines to etc security limits conf for the userspace com ponents such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a user space application If desired tune the value unlimited to a specific amount of RAM Step
31. 8 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step 3 Download the exe image according to the architecture of your machine see Step 1 The name of the exe is in the following format MLNX_VPI_WinOF lt ver sion gt _All_ lt OS gt _ lt arch gt exe ows Installer 4 2 2 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start gt Run and enter CMD Step2 Enter the following command Step 3 Click Next to create a server image Welcome to the InstallShield Wizard for MLNX VPI The InstallShield R Wizard will install MLNX VPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Mellanox Technologies 39 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter User Manual Rev 1 4 Step4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create server image of MLNX_VPI at the specified network location or click Cancel to exit the wizard Network location rr InstallShield Step 5 Click Install to extract this folder or click Change to install to a different folder Netw
32. 8 8GT s RoHS R6 Adapter IC Part Number MT27528A0 FCCR FV 4103 for Physical Function Device ID d cimal 4100 for Virtual Function Mellanox Technologies 11 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 1 2 Table 4 Features Virtual Protocol Interconnect VPI InfiniBand Architecture Specification v1 2 1 compliant PCI Express PCIe 10 40 56 Gigabit Ethernet InfiniBand FDR InfiniBand FDR10 InfiniBand QDR Memory Virtualized Overlay Networks RDMA over Converged Ethernet RoCE CPU offload GPUDirect Mellanox Technologies 12 Features and Benefits VPI enabled adapters facilitate any standard networking clustering storage and management protocol to seamlessly operate over any converged network with the same software infrastructure ConnectX 3 Pro delivers low latency high bandwidth and computing efficiency for performance driven server and storage clustering applications ConnectX 3 Pro is InfiniBand Architecture Specification v1 2 1 compliant Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ba 40 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover A pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s w
33. EE AE EE HAE HE AE HA EH libibumad devel Preparing libibumad devel HH AE HE EEE AE EE A HAE EE HAE EE Preparing libibumad static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHRHBHHRHRHI Preparing libibumad static Preparing libibmad HH AA EEE HAE AE A HAE HE EE HAE EH Preparing libibmad Preparing libibmad devel Preparing HH HAE AA EEE HAE AE A HAE HE AE HA EH libibmad devel THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing
34. FP Port Adapter Card User Manual Rev 1 4 Figure 8 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card J1 PC Connector Mellanox Technologies 65 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 7 5 Bracket Mechanical Drawing Figure 9 Dual port Bracket LEDs Port 2 Mellanox Technologies 66 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Figure 10 Single port Bracket LEDs Gasket Port 1 76 Regulatory Statements For regulatory statements for all ConnectX 3 Pro cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Mellanox Technologies 67 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix A Interface Connectors Pinout QSFP Connector Pinout Figure 11 Connector and Cage Views Top 18 33 View into Rear Le IT 0 Tid POW X129A 129A apod QN9 dexi QN9 dixi Oo 22 9 2 E SN WWW WW WM Wi 9 88 op 88 98 8 FP EB 222 339 ID ONES 26 w 6226808 SPREE RS EE SEES _ TOP 18 35 View into Front of Cage 8 50 Table 13 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4
35. GND GND 5 TXN_4 Tx4n 6 TXP_4 Tx4p 7 GND GND 8 ModSelL ModSelL 9 ResetL PortO ResetL 10 VccRx 11 SCL SCL Mellanox Technologies 68 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Table 13 Connector Pin Number Name to Signal Name Connector Pin Number Connector Pin Name Port A Signal Name 12 SDA SDA 13 GND GND 14 RXP_3 Rx3p 15 RXN_3 Rx3n 16 GND GND 17 RXP 1 Rxlp 18 RXN_1 RxIn 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP_2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 2T ModPrsl PortO Mod PrsL 28 IntL IntL 29 VecTx 30 31 LPMode_Port0O LPMode 32 GND GND 33 TXP_3 Tx3p 34 TXN_3 Tx3n 35 GND GND 36 _1 TxIp 37 TXN 1 Txin 38 GND GND 2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Mellanox Technologies 69 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Figure 12 PCle Connector Pinout 5 NA PRSNTiz 1 B2 ua 2 B m H4 As B5 sucuk TCK A5 gt SMDAT PAB BZ GND AL RR Tus LAR 2 TRST A8 B10 3
36. H Preparing HHH HH Het HH HH mvapich2 gcc HH HH HH HH HH HH Preparing HHH HH HHH HHH HHH HH openmpi gcc T1HBHBHUHHBWHHSHHHUUE BSHHBHHU H H AE HEH HE E AE HBHHBHE B HE Preparing HHH HH HHH HH HH mpitests mvapich2 gcc HH 4 HH HH HHH 4 HH HH Preparing HHH HH HHH HHH 4 mpitests openmpi gcc HH HH He 41 4E 4L HH HHH HH HH HH Preparing HHH HH Het HH HH mlnxofed docs HH HH HH HH HHH HH HH HH Mellanox Technologies 31 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Device 05 00 0 05 00 0 Ethernet controller Mellanox Technologies MT26448 ConnectX 10GigE PCIe 2 0 5GT s rev b0 Link Width is not 8x PCI Link Speed 5Gb s Device 07 00 0 07 00 0 Ethernet controller Mellanox Technologies MT27500 Family ConnectX 3 Link Width 8x PCI Link Speed 5Gb s Installation finished successfully The firmware version on dev mst mt
37. HBHBHHHHHHHBHBHHHHE libibem Preparing THHHBHHHBHHHHHBHBHHHHHHBHHHHHHHBHBHHHHHHHBHBHHHNE libibem Mellanox Technologies 28 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Preparing libibcn devel HHH HH H HH H H 4 HH Preparing HH HAE AA EEE HAE AE A HAE HE AE HA EH libibcn devel HH HH HHH HH HH Preparing libibumad HH A AE EEE HAE AE A HAE HE AE HA EH Preparing libibumad HH AA EEE HAE AE A HAE HE EE HAE EH Preparing HH A AE E
38. HH A AE EEE AE EE HAE HE AE HA EH con pat dapl HH HH HH HH H Het H H HH H H 4 HH Preparing compat dap1 HH AE HE EEE AE EE A HAE EE HAE EE Preparing compat dapl devel BEA AAA HU HU AH HE HE AA ME ME A HU E Prepar ng compat dapl devel Prepar ng 1 HH 4 HH H HH H H Het H H HH H H HH HH Prepar ng 1 HH H HH HHHH H HH H H HHH H H HH H H 4 HH Prepar ng dapl deve HHH HH HH HH H H HHH H HH HH Prepar ng HH HAE AA EEE HAE AE A HAE HE AE HA EH dapl deve L THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing dapl devel static
39. HHHHHBHHHHHBHHHHHHHHHBHBHHHHHHHBHBHHHHHHHBHBHHEHNE Preparing libmlx5 HH HH HH HH HH HH HH Preparing libmlx5 devel THHHHHHHBHHHHHBHHHHHHHHHHHHBHHHHBHHHHHHHHBHHHHRSHRHHI Preparing 00 libmlx5 devel HH A AE EEE HAE EE HAE AE HA EH Preparing HH E R AHRR HERRE EHER R HHHRHH libmverbs devel THHHHHHHBHHHHHBHBHHHHHHBHHHHHHHBHBHHHHHHHBHBHHHHE Preparing THHHBHHHHHHHHHHHBHHHHHHBHHHHHHHBHBHHHHHHHBHBHHHNE libmverbs devel THHHBHHHHBHHHHHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBSHRHHHI Preparing HH A AE EEE HAE AE A HAE HE AE HA libmge HH 4 HH HH HH 4 4 HH HH Preparing THHHHHHHHHHHHHBHBHHHHHHHHHHHHHHBHBHHHHHHHBHBHHHHE libmqe HH HH HH HH HH HH HH Preparing THHHBHHHHHHHBHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBSHRHHI libmge devel Preparing libmqe devel HH A AE EEE HE AE AE A HAE HE EE HA EH Preparing THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHH
40. Mellanox TECHNOLOGIES ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual P N MCX353A FCCT MCX354A FCCT Rev 1 4 www mellanox com Rev 1 4 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF
41. Port 1 on 0 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 0 NIC UP 1X QDR Ethernet Port State of Port 1 on CA 1 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 1 NIC UP 1X QDR Ethernet Error Counter Check on CA 0 NIC NA Eth ports Error Counter Check on CA 1 NIC NA Eth ports Syslog Check h PASS 0 00 02 c9 03 00 07 4 8 Node 9700 CA sil Nie M 00 02 c9 03 00 35 fd c0 DONE After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters can be retrieved by running the com P mand etc infiniband info 4 1 4 Installation Results Software The OFED and MFT packages are installed under the usr directory e The kernel modules are installed under InfiniBand subsystem lib modules uname r updates kernel drivers infiniband milx4 core driver lib modules lt kernel version gt extra mlnx ofa kernel drivers net ethernet mellanox mlx4 mlx4 core ko Mellanox Technologies 33 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 mlx4 ib driver lib modules lt kernel_version gt extra mlnx ofa kernel drivers infiniband hw mlx4 mlx 4 ib ko mlx5_core driver lib modules lt kernel version gt extra ml
42. Port Type Management on Linux 37 4 1 9 AMO SEO csset S ERERS AER PERSA 37 4 1 9 1 Enabling Auto Sensing 38 4 2 Windows DEVER s sete See weed ea ea e a 38 4 21 Downloading MLNX WinOF 38 4 2 2 Extracting Files Without Running Installation 39 4 2 3 Installing 41 4 2 3 1 Attended Installation 41 4 2 3 2 Unattended Installation 47 4 2 4 Upgrading MLNX_WinOF 47 4 2 5 Installation Results s ere Han as e Eo e 48 4 2 6 OpenSM Activation ue RR Re RR RR a 48 427 Uninstalling MLNX 49 4 2 7 1 Attended Uninstall 49 42 1 2 Unattended Uninstall re Re Eh ette qu bue 49 4 2 8 Assigning Port IP After Installation 49 4 2 0 Port Type Management on Windows 51 4 3 Performance Tuning i ig aaa 53 44 VMware Driver 53 4 4 1 Installing Mellanox InfiniBand OFED Driver for VMware vSphere 53 4 4 2 Uninstalling
43. RPM Preparing T T HERT T T T T THAT kerne mft HH HHHHH HH HH H HH HH HH HH H H Installing knem RPM Preparing box HH H HH HHH HH HH H 4 4 HH HH knem HH H Het HH HH H HHH HH H HH HHHH HH H HH H 4 HH Installing mpi selector RPM Preparing Jis H HH HH H HHH H HHH HH HHHH H HH HH mpi selector HH H HH HHH HHH HH HHH 4 4 HHH HH HH Mellanox Technologies 27 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Installing user level RPMs Preparing HH A AE EEE HAE AE A HAE HE EE HA EH ofed scripts Preparing HH HEHE HAE HEHE HH libibverbs Preparing HH A AE EEE HAE AE A HAE AE HA EH libibverbs HH A RAAHAA EER RE EHRRREHHRHH Preparing HH AA EEE HAE AE A HAE HE AE HAE EH libibverbs devel HH A A RAER RE HER RE EHRRRHHHRHH Preparing
44. S To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel_support sh script located under the docs directory Usage mlnx add kernel support sh m mlnx ofed path to MLNX OFED directory make iso make tgz gt gt make iso Create MLNX OFED ISO image gt make tgz Create MLNX OFED tarball Default t tmpdir local work dir kmp v verbose 1 The firmware will not be updated if you run the install script with the without fw update option Mellanox Technologies 23 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 5 6 under the tmp directory MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 docs mlnx add kernel support sh i mnt MLNX _OFED LINUX 1 5 3 rhel5 6 x86 64 180 All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 iso 4 1 3 2 Installation Script The usage of the installation script is described below You will use it during the installation pro cedure described in Section 4 1 3 4 Installation Procedure on page 26 Usage mnt
45. a 14 15 17 Ubuntu 12 04 Windows Server 2008 2012 FreeBSD OpenFabrics Enterprise Distribution OFED OpenFabrics Windows Distribution WinOF VMware ESXi 4 x and 5 x 14 Connectivity Interoperable with 10 40GbE switches Interoperable with 56GbE Mellanox switches Passive copper cable with ESD protection Powered connectors for optical and active cable support QSFP to SFP connectivity through QSA module Interoperable with 40 56 Gb s InfiniBand Passive copper cable with ESD protection Mellanox Technologies 13 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 2 Interfaces Each adapter card includes the following interfaces InfiniBand Interface Ethernet Interface PCI Express Interface 2C compatible Interface LED Interface Figure 1 MCX354A FCCT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables 2 1 InfiniBand Interface The network ports of the ConnectX 9 3 Pro adapter cards are compliant with the InfiniBand Archi tecture Specification Release 1 2 1 InfiniBand traffic is transmitted through the cards QSFP con nectors 2 2 Ethernet Interface The network ports of the ConnectX 3 Pro adapter cards are compliant with the IEEE 802 3 Ethernet standards listed in Table 4 Features on page 12 Ethernet traffic is transmitted through
46. anager To see the Mellanox network adapter device and the Ethernet or IPoIB network device depending on the used card for each port display the Device Manager and expand System devices or Network adapters File Action View Help gt As Y Mellanox ConnectX 3 Ethernet Adapter 2 Microsoft Kernel Debug Network Adapter 4 Other devices Unknown device Ports COM amp LPT p deh Print queues gt D Processors b lt gt Storage controllers 4 j System devices Fixed Feature Button 1 Composite Bus Enumerator li Direct memory access controller High precision event timer Intel R 5520 5500 X58 Hub Control Status and RAS Registers 3423 1 Intel R 5520 5500 X58 Hub GPIO and Scratch Pad Registers 3422 Intel R 5520 5500 X58 Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 Hub PCI Express Root Port 3 340A Intel R 5520 5500 X58 Hub PCI Express Root Port 7 340E Intel R 5520 5500 X58 1 0 Hub System Management Registers 342E Intel R 5520 5500 58 Hub to ESI Port 3406 Ml Intel R 82801 PCI Bridge 244E 2 Intel R ICH10 Family PCI Express Root Port 1 3440 2 Intel R ICH10 Family PCI Express Root Port 5 3448 E Intel R ICH10R LPC Interface Controller 3416 1 Microsoft ACPI Compliant System Microsoft Generic Compliant Device Microsoft System Management BIOS Driver
47. arnings 72 Appendix D Avertissements de s curit d installation Warnings in French 74 Appendix Sicherheitshinweise Warnings in German 76 Appendix Advertencias de seguridad para la instalaci n Warnings in Spanish 78 Mellanox Technologies 5 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 List of Tables Table 1 Revision History Table 8 Tabl 2 Documents List eee deren der ale Dess 9 Table 3 Single and Dual port FDR Adapter Cards 11 Tabl 4 Features 5 282328 E an bee yor die ds yee es 12 Table 5 Software and Hardware Requirements 22 Table 6 minxofedinstall Return 26 Table 7 Supported ConnectX Port Configurations 37 Table 8 Software and Hardware Requirements 38 Table 9 MCX354A FCCT Specifications Table 60 Table 10 MCX354A FCCT Specifications Table 61 Table 11 Physical and Logical Link Indications InfiniBand 63 Table 12 Physical and Logical Link Indication Ethernet Mode 63 Mellanox Technol
48. ationalen elektrischen Bes timmungen installiert werden This equipment should be installed in compliance with local and national elec LA trical codes Mellanox Technologies 76 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gefahrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 77 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix F Advertencias de seguridad para la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de Instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la LA m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de 4 rayos 4 Conexi n y desconexi n del cable Copper
49. ault Ethernet During driver run time Sense a link every 3 seconds if no link is sensed detected If sensed set the port type as sensed 4 2 Windows Driver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com Download this package from the Mellanox web site at http www mellanox com gt Products gt Software gt Infini Band VPI Drivers gt Windows SW Driver gt Download Follow the installation instructions included in the download package also available from the download page Hardware and Software Requirements Table 8 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Installer Privileges The installation requires administrator privileges on the target machine 4 21 Downloading MLNX WinOF Follow these steps to download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click start gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURE On an x64 64 bit machine the output will be AMD64 Step2 Go to the MLNX WinOF for Windows Web page at http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Windows SW Drivers Mellanox Technologies 3
50. cal Area Connection and left click Properties Networking Sharing Connect using Mellanox IPolB Adapter This connection uses the following items QoS Packet Scheduler File and Printer Sharing for Microsoft Networks Microsoft Network Adapter Multiplexor Protocol Link Layer Topology Discovery Mapper 1 0 Driver Link Layer Topology Discovery Responder A Internet Protocol Version 6 TCP IP v6 E Install Uninstall Properties Description Transmission Control Protocol Intemet Protocol The default wide area network protocol that provides communication across diverse interconnected networks Cient for Microsoft Networks v Step Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties Mellanox Technologies 50 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step4 Select the Use the following address radio button and enter the desired IP information General You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator for the appropriate IP settings Obtain an IP address automatically Use the following IP address IP address Subnet mask Default gateway Obtain DNS server address automatically Use the following DNS server addresses Preferred DNS serv
51. e Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card Mellanox Technologies 55 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Figure 6 Support Download Assistant AM Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information 1 If a newer firmware version exists for your adapter card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above b Unzip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4103 pci Image type ConnectX 3 Pro FW Version 2 9 4100 Device ID 4103 Mellanox Technologies 56 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 6 6 1 Mellanox Technologies 57 Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are com
52. e InfiniBand protocol VPI cards have botha GUID and a MAC derived from the GUID Figure 14 Card Product Label Model No CX354A ConnectX 3 FDR InfiniBand 40GigE PIN MCX354A FCAT Rev A1 III SIN MT1106X01504 GUID 00 02 C9 03 00 27 05 1C Am MAC 00 02 C9 27 05 1C Made in Israel Port 1 uses GUID or MAC described on the label To obtain the GUID or MAC for Port 2 on dual port cards add 1 to the GUID or MAC of Port 1 Mellanox Technologies 71 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix Safety Warnings 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed LA ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting Some copper cables are heavy and not flexible as such they should be carefully LA attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions 5 Equipment Installation This equipment should be installed replac
53. ed or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes k This equipment should be installed in compliance with local and national electrical codes 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure Mellanox Technologies 72 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 CLASS 1 LASER PRODUCT and reference to the most recent laser standards TEC 60 825 1 1993 1 1997 A2 2001 and EN 60825 1 1994 A 1 1996 A2 20 Mellanox Technologies 73 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des
54. ed administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start All Programs gt Mellanox Technologies gt MLNX_WinOF gt Uninstall MLNX WinOF 4 2 7 2 Unattended Uninstall gt To uninstall MLNX_WinOF in unattended mode perform the following Step 1 Open a CMD console Step 2 Uninstall the driver Run MLNX VPI WinOF 4 40 0 All win8 x64 exe S x v qn 4 2 8 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine gt To obtain the MAC address Step 1 Open a CMD console Step 2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices T E Net Networ gt Search Network Connections 2 Organize v Device Y Ethernet Broadcom BCMS5709C NetXtre x Ethernet 2 Ethernet 3 Y Ethernet 4 M U Local Area Connection IBM USB Remote Mellanox Technologies 49 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step2 Right click a Mellanox Lo
55. er Alternate DNS server _ validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from a CMD console gt ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix IP Addresse 2 12563 Subnet Mask 6 8 255 25500 Default Gateway 4 2 9 Port Type Management on Windows After installing Mellanox WinOF VPI for Windows on your machine you can change a port s pro tocol configuration The following steps describe how to configure the port type Step 1 Display the Device Manager and expand System Devices Mellanox Technologies 51 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 File Action View Help es mal al 2 9 15 4 Network adapters A Lu Broadcom BCM5709C NetXtreme Il GigE NDIS Client 48 Broadcom BCM5709C NetXtreme Il GigE NDIS Client 49 IBM USB Remote NDIS Network Device l Microsoft Kernel Debug Network Adapter Y Ports COM amp LPT p Print queues gt Processors Security devices lt gt Storage controllers System devices Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties Select the Port Protocol tab from the Properties sheet Note The Port Protocol tab is d
56. et ib core 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib ipoib 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib mad 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib sa 1 8 1 OEM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib umad 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net mlx4 core 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net mlx4 ib 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 Mellanox Technologies 53 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 After the installation process all kernel modules loaded automatically upon boot 4 4 2 Uninstalling Mellanox InfiniBand OFED Driver To uninstall the driver 1 Log into the ESXi5 0 server with root permissions 2 List the existing InfiniBand OFED driver modules i esxcli software vib list grep Mellanox net ib core 1 8 1 0EM 500 0 0 472560 Mellanox Partner Supported 2013 02 09 net ib ipoib 1 8 1 0EM 500 0 0 472560 Mellanox Partner Supported 2013 02 09 3 Remove each module using the esxcli software vib remove command gt esxcli software vib remove n net ib ipoib gt esxcli software vib remove n net mlx4 ib gt esxcli software vib remove n net ib umad gt esxcli software vib remove n net ib sa gt esxcli software vib remove n net ib mad gt esxcli software vib remove n net ib core
57. h the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408 fw 25408 rel mlx Step 4 Reboot your machine after the firmware burning is completed Mellanox Technologies 36 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 4 1 7 Uninstalling Mellanox OFED Use the script usr sbin ofed uninstall sh to uninstall the Mellanox OFED package The script is part of the ofed scripts RPM 4 1 8 Port Type Management on Linux ConnectX ports can be individually configured to work as InfiniBand or Ethernet ports By default both ConnectX ports are initialized as InfiniBand ports If you wish to change the port type use the connectx port config script after the driver is loaded Running sbin connectx port config s will show current port configuration for all ConnectX devices Port configuration is saved in the file etc infiniband connectx conf This saved configuration is restored at driver restart only if restarting via etc init d openibd restart Possible port types are eth Ethernet e ib Infiniband e auto Link sensing mode Detect port type based on the attached network type If no link is detected the driver retries link sensing every few seconds Table 7 lists the ConnectX port configurations supported by VPI Table 7 Supported ConnectX Port Configurations Port 1 Configuration Port 2 Configuration
58. hat is included in the shipping box please follow the instructions in this section Note Due to risk of damaging the EMI gasket it is not recommended to replace the bracket more than three times To replace the bracket you will need the following parts Mellanox Technologies 16 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 The new bracket of the proper height The 2 screws saved from the removal of the bracket The 2 fiber washers saved from the removal of the bracket 3 4 1 Removing the Existing Bracket Figure 2 Bracket Screws Screws 1 Remove the two screws holding the bracket in place The bracket comes loose from the card YN Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers 3 4 2 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up Please make sure to insert the bracket until the end of the EMI fingers See Figure 3 Figure 3 EMI Fingers Example TE o Mellanox Technologies 17 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal procedure above Figure 4 Placing the Bracket on the Card 3 Make sure t
59. hat the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 Ibs in torque on the screws 3 5 Card Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot A lesser width adapter can be seated into a greater width slot x4 in a x8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter connector edge with the PCI Express connector slot Mellanox Technologies 18 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 4 Applying even pressure at both of the insert adapter into the slot until it is firmly seated When the adapter is properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis Do not use excessive force when seating the card as this may damage the system or the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to www mellanox com gt Prod ucts gt Cables and Transceivers 3 6 1 Cable Installation 1 cables can be inserted
60. he following command MLNX VPI WinOF 4 40 0 All win8 x64 exe v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Mellanox Technologies 41 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Click Next in the Welcome screen Welcome to the InstallShield Wizard for MLNX The InstallShield R Wizard will install MLNX_YPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Step 2 Read then accept the license agreement and click Next License Agreement Please read the Following license agreement carefully Copyright c 2005 2013 Mellanox Technologies 11 rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer innn im odes tha lor 8 I accept the terms in the license agreement I do not accept the terms in the license agreement InstallShield Mellanox Technologies 42 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step Select the target folder for the installation Destination Folder Click Next to install to this Folder or c
61. hout fw update 2 The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image If an 5 Flash was originally programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image Incase your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file 4 1 5 Post installation Notes Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file 4 1 6 Updating Firmware After Installation In case you ran the mInxofedinstall script with the without fw update option and now you wish to manually update firmware on your adapter card s you need to perform the following steps If you need to burn Expansion image please refer to OFED User Manual See Table 2 Documents List on page 9 Mellanox Technologies 35 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 The follo
62. http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 10 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 1 Introduction This is the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX 3 Pro integrated circuit device These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics Section 1 1 Product Overview on page 11 Section 1 2 Features and Benefits on page 12 Section 1 3 Operating Systems Distributions on page 13 Section 1 4 Connectivity on page 13 1 1 Product Overview The following section provides the ordering part number port speed number of ports and PCI Express speed Each adapter comes with two bracket heights short and tall Single and Dual port FDR Adapter Cards Table 3 Single and Dual port FDR Adapter Cards MCX353A FCCT single port card ing Part a MCX354A FCCT dual port card 2 InfiniBand FDR 56 Gb s Data Transmission Rate Ethernet 40 56 Gb s single or dual port QSFP Network Connector Types PCI Express PCIe SerDes Speed PCIe 3 0 x
63. isplayed only if the NIC is a VPI IB and ETH If the NIC is either only IB or ETH the tab will not be shown General Port Protocol Driver Details Events Resources Current Setting Eth Port2 Eth HCA Port Type Configuration HW Defaults Port 1 ETH AUTO Port 2 C IB ETH C AUTO Port Protocol Configuration This menu displays the adapter s port type and enables you to set the network protocols for the netvvork adapter ports The network protocol is determined according to the NIC s Hardware Defaults port type You can choose the protocol explicitly by selecting the port type to InfiniBand IB or Ethernet Eth enahle Aita Sensing nlease chonse Al IT If the Cancel Mellanox Technologies 52 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step 3 In this step you can perform two different functions Choose the desired port protocol for the available port s and b activate or deactivate the WSD ND and or SDP ULPs Note IB must be always the first port in Port 1 If you choose ETH as your first port in Port 1 then the second port in Port 2 can be only ETH Note WSD is not supported in Windows 7 Consequently on this OS the WSD checkbox is grayed out and cannot be selected 43 Performance Tuning For guidelines on improving network adapter
64. ith a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s An InfiniBand transport and link layer running over a physical layer using the PCS encoding of 64b 66b and a bit rate of 10 3125Gb s per lane A standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10Gb s with an 8b 10b encoding PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 4MB SPI Flash device M25PX16 VMNOP device by ST Microelectronics EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb ConnectX 3 Pro effectively addresses the increasing demand for an overlay net work enabling superior performance by introducing advanced NVGRE and VXLAN hardware offload engines that enable the traditional offloads to be per formed on the encapsulated traffic With ConnectX 3 Pro data center operators can decouple the overlay network layer from the physical NIC performance thus achieving native performance in the new network architecture Leveraging Data Center Bridging capabilities RoCE provides efficient low latency RDMA services over Layer 2 Ethernet Adapter functionality enabling reduced CPU overhead allowing more available CPU Using GPUDirect RDMA adapters can directly read and write CUDA host and device memory eliminating unnecessary system memory copies and CPU over head resulting in significant performance improvements
65. lick Change to install to a different Install MLNX VPI to C Program Files Mellanox MLNX_ PT InstallShield Step 4 The firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card in this case the firmware will not be updated Ifthe user has a standard Mellanox card and the firmware version is older than the one specified in WinOF Installation Guide 4 40 the firmware will be updated accordingly However if the user has both OEM card and Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the 5 firmware version Recommended Upgrading the Firmware version will reboot your machine Note One or more of your HCA adapters has an old firmware version We recommend upgrading to a newer Firmware version to enable improved Functionality and support driver s capabilities InstallShield Mellanox Technologies 43 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step 5 Configure your system for maximum performance by checking the maximum performance box Maximum Performance Check this box to configure your system For maximum performance Y Check this box to configure your system For maximum performance Recommended Note This step requires you to reboot the machine at the end of the installation process Installshield Step6 Select a Complete or Custom installation follow Step
66. mlnxofedinstall OPTIONS Mellanox Technologies 24 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Options c config lt packages config file gt Example of the configuration file can be found under docs n net lt network config file gt Example of the network configuration file can be found under docs p print available Print availablepackagesforthecurrent plat form and create a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update force fw update Force firmware update force Force installation without querying the user all Install all kernel modules libibverbs libibu mad librdmacm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector perftest sdpnetstat and libsdp srptools rds tools static and dynamic libraries hpc Install all kernel modules libibverbs libibumad librdmacm mft mstflint diagnos tic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector dynamic librar ies basic Install all kernel modules libibverbs libibu mad mft mstflint dynamic libraries msm Install all kernel modules libibverbs libibumad mft mstflint diagnostic tools OpenSM ib bonding dynamic libraries NOTE With msm flag the OpenSM daemo
67. n is configured to run upon boot vma Install packages required by VMA to support both IB and Ethernet vma ib Install packages required by VMA to work over InfiniBand vma eth Install packages required by VMA to work over Ethernet v vv vvv Set verbosity level q Set quiet no messages will be printed umad dev rw Grant non root users read write permission for umad devices instead of default hugepages overcommit Set 80 of MAX MEMORY as overcommit for a huge page allocation 4 1 3 3 mlnxofedinstall Return Codes Mellanox Technologies 25 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Table 6 lists the mInxofedinstall script return codes and their meanings Table 6 minxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start the mst driver 4 1 3 4 Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount the ISO image on your machine host1 mount o ro loop MLNX OFED LINUX ver
68. nd Max power available through QSFP port 1 5W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manu als Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a The ConnectX 3 Pro adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mella nox InfiniBand product Mellanox Technologies 61 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 b For both operational non operational states Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet Mellanox Technologies 62 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 73 Adapter Led Operations There are two I O LEDs per port See Table 11 for different LED functions in InfiniBand mode and Table 12 for different LED functions in Ethernet mode Table 11 Physical and Logical Link Indications InfiniBand Mode LED Function Green physical link Constant on indicate
69. nt and configuratio IBTA Specification Release 1 2 1 InfiniBand Architecture Specification IEEE Std 802 5 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega Bytes The use of Mb or Mbits small b indicates size in mega bits IB is used in this document to mean InfiniBand In this document PCIe is used to mean PCI Express Mellanox Technologies 9 ConnectX 3 Pro Ethernet Dual SFP Port Adapter User Manual Rev 1 4 Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us throueh the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the database from the Mellanox Support web page
70. nx ofa kernel drivers net ethernet mellanox mlx5 core mlx5 core ko ib driver lib modules lt kernel version extra mlnx ofa kernel drivers infiniband hw mlx5 mlx 5 ib ko IPoIB lib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko iSER lib modules uname r updates kernel drivers infiniband ulp iser ib iser ko elPoIB lib modules uname r updates kernel drivers net eipoib eth ipoib ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko lib modules uname r updates kernel net rds rds tcp ko Kernel s modules location may vary depending on the kernel s configuration For example lib modules uname r extra kernel drivers net ethernet mellanox m1x4 mlx4 core The package kernel ib devel include files are placed under usr src ofa kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel Theraw package un backported source files are placed under usr src ofa kernel lt ver gt The script openibd is installed under etc init d This script can be used to load and unload the software stack The script connectx port config is installed under sbin This script can be u
71. o update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at http www mellanox com gt Products gt Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci cr0 d Get the PSID firmware identification and programmed firmware version using the command Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for this product and are included here for illustration purposes only gt flint d dev mst mt4103 pci q Image type ConnectX 3 Pro FW Version 2 9 4000 Device ID 4103 Chip Revision 0 Description Node Porti Port2 Sys image 60105 000002c900000200 000002c900000201 000002 900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT 1020110019 VSD PSID MT 1020110019 1 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 6 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the firmware release version and the card part number Not
72. ogies 6 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 List of Figures Figure 1 MCX354A FCCT Card 14 Figure2 Bracket Screws eme da da dr er ences 17 Figure 3 Fingers Example 17 Figure 4 Placing the Bracket on the Card 18 Figure POTD e vice IS tu usu eed poet vss 20 Figure Support Download Assistant 56 Figure 7 Mechanical Drawing of the Single port MCX353A FCCT Adapter Card 64 Figure 8 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card 65 Figure9 Dual port Bracket 2 2 epe re darin e Sue ua qual 66 Figure 10 Single port Bracket 67 Figure 11 Connector and Cage Views 68 Figure 12 PCIe Connector 70 Figure 13 Compatible Connector Plug and Pinout 70 Figure 14 Card Product Label 71 Mellanox Technologies 7 ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 Revision History This document was printed on October 25
73. or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated a Support the weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack b Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card c Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card d Make sure that the connector locks in place When installing cables make sure that the latches engage Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 3 After inserting a cable into a port the Green LED indicator will light when the physical con nection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See Section 7 3 Adapter Led Operations on page 63 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the Yellow LED will light When data is Mellanox Technologies 19 ConnectX 3 Pro VPI Single and Dual QSFP P
74. ork Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create server image of MLNX VPI at the specified network location or click Cancel to exit the wizard Network location Change InstallShield Mellanox Technologies 40 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Step 6 complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX_YPI Click Finish to exit the wizard lt Back Cancel 4 223 Installing MLNX_WinOF This section provides instructions for two types of installation procedures Attended Installation An installation procedure that requires frequent user intervention e Unattended Installation An automated installation procedure that requires no user intervention Both Attended and Unattended installations require administrator privileges kall 4 2 31 Attended Installation The following is an example of aMLNX_WinOF_win8 x64 installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF To configure your setup to contain the logs option please run the following command after opening a CMD console MLNX VPI WinOF 4 40 0 All win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run t
75. ort Adapter Card User Manual Rev 1 4 being transferred the yellow led will blink See Section 7 3 Adapter Led Operations page 63 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated a Pull on the latch release mechanism thereby unlatching the connector and pull the connector out of the cage b Do not apply torque to the connector when removing it from the adapter card c Remove any cable supports that were used to support the cable s weight 3 7 Identify the Card in Your System 3 7 1 On Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 Pro adapter card 3 Right click the mouse on your adapter s row and select properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or Hardware Ids Windows 2008 R2 from the Properties pull down menu Note Figure 5 is an example using ConnectX 3 device This figure will be updated in future releases Figure 5 PCI Device General Pot Protocol Driver
76. ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel k Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L elimination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 74 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 2 2001 et NE 60825 1 1994 A1 1996 A2 2001 Mellanox Technologie
77. patible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is Ensure that adapter driver s is loaded Try another port on the switch Make sure the cable is securely attached off Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but Check that the latest driver is loaded with no Check that both the adapter and its link are set to the same speed communication and duplex settings established ConnectX 3 Pro Single and Dual QSFP Port Adapter User Manual Rev 1 4 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed_info head 1 Information ifconfig ethtool lt interface gt ethtool interface of Mellanox port num ibdev2netdev
78. s 75 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Appendix Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der YN maximal empfohlenen Temperatur von 55 C 131 F betrieben werden Es ist ein Luft strom von 200 bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschliefen Trennen von Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die LA Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert aus getauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und n
79. s a good physical link e Blinking indicates a problem with the physical link If neither LED is lit then the physical link has not been established Yellow logical data constant yellow indicates a valid logical data activity link without data transfer activity link A blinking yellow indicates a valid logical link with data transfer If only the green LED is lit and the yellow LED is off then the logical link has not been established Table 12 Physical and Logical Link Indication Ethernet Mode LED Function Green physical link Constant on indicates a good physical link Ifneither LED is lit then the physical link has not been established Yellow logical data blinking yellow indicates activity data transfer activity link Stays off when there is no activity P The short bracket has the same port and LED footprints as the tall bracket Mellanox Technologies 63 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 7 4 Board Mechanical Drawing and Dimensions dimensions are in millimeters All the mechanical tolerances are 0 1mm ha Figure 7 Mechanical Drawing of the Single port MCX353A FCCT Adapter Card J1 PC Connector Flash Jumper 142 24 Fe a EE a E aa 3 E AB a TT Mellanox Technologies 64 ConnectX 3 Pro VPI Single and Dual QS
80. sed to configure the ports of ConnectX network adapter cards to Ethernet and or InfiniBand For details on this script please see Section 4 1 8 Port Type Management on Linux The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED instal lation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file is used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d Mellanox Technologies 34 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc If IPoIB configuration files are included ifcfg ib n files will be installed under etc sysconfig network scripts ona RedHat machine The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled 1 You run the installation script in default mode that is without the option wit
81. to C Program Files Mellanox MLNX_ PTIVIBY Tools InstallShield Mellanox Technologies 45 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation Tf you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Step 7 Click Finish to complete the installation InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin glog Cl Show release notes Cancel Mellanox Technologies 46 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 If the firmware upgrade fails the following message will be displayed InstallShield Wizard Completed The InstallShield Wizard has successfully installed _ Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin g log Firmware upgrade Failed with error 1003 Error Description We could not Find a new version For the chosen device Please refer to the UM for Show release notes
82. vers gt Download This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox ConnectX 3 Pro adapter hardware installed 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for 1GB Installation Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to the http www mellanox com gt Products gt Software gt Infini Band VPI Drivers gt Mellanox OFED Release Notes file Installer Privileges The installation requires administrator privileges on the target machine 4 1 2 Downloading Mellanox OFED Step 1 Verify that the system has a Mellanox network adapter HCA NIC installed by ensuring that you can see ConnectX entries in the display The following example shows a system with an installed Mellanox HCA host1 lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27520 ConnectX 3 Pro Step 2 Download the ISO image to your host The image s name has the format MLNX OFED LINUX lt ver gt lt OS label gt lt CPU arch gt iso You can download it from http
83. wing steps also appropriate case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site http www mella ae nox com gt Support gt Firmware Download Step 1 Start mst hostl mst start Step 2 Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine host1 mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre 120 module is not loaded MST devices dev mst mt25418 pciconf0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 dev mst mt25418 pci PCI direct access bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is dev mst mt25418 pci msix0 PCI direct access bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix _pci_cr0 In the example listed above this will be dev mst mt25418 pci Step3 Burn firmware 1 Burning a firmware binary image using mst flint that is already installed on your machine Please refer to MSTFLINT README txt under docs 2 Burning a firmware image from a mlx file using the mlxburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device wit
84. wnloader flint d mst device firmware bin file b Eventlog viewer e MST device logs mst start Collect log file mst status e CAUsers Administrator flint d mst device dc gt dump configura tion log e CAUsers Administrator mstdump mst device dc gt mstdump log Mellanox Technologies 59 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 4 7 Specifications 71 MCX353A FCCT Specifications Table 9 MCX354A FCCT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000BASE R 40GBASE R4 Protocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40 56Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 87W Max Power Passive Cables 7 30W Power and Max power available through QSFP port 1 5W Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manu als Regulatory

Download Pdf Manuals

Related Search

Related Contents

Valueline VLEP11040B20 power cable  BTW450  MCP XT – Programmable Control Module User`s Manual  Trépied télescopique  DECOLORANTE DECOLORANTE  Bedienungsanleitung Croozer Kid plus  Hahnel PowerStation Ultima II  MACROS GTIVE Generic Tool for Important Variable  TV 1001 Navigator - Ideal Vacuum Products, LLC    

Copyright © All rights reserved. Failed to retrieve file