Home

40GbE User Manual for OCP

1. 42 7 4 Adapter LED Operation u a en He in 43 7 5 RegulatotyStatemenis yeliz ros ropita de say ody Dee aha aa coe ar rad 43 Appendix Finding the MAC GUID and Serial Number on the Card 44 Retrieving GUID ans ae dente sm er rmm ERE 45 Appendix B Interface Connectors Pinout 47 B 1 PCI Express 8 Connector Pinout 47 2 I2C compatible Connector Pinout 48 Bid NC Sllnt rfac Piriout zv eu tn EC Ue ti al Saga wey Rey ee iS 48 Appendix C Safety 49 Appendix D Avertissements de s curit d installation Warnings in French 51 Appendix E Sicherheitshinweise Warnings in German 53 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 55 4 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 List of Figures Figure 1 12C Harness ew e eicere er BE ad ki 13 Figure 25 PCHDevicen arte RE AA 16 Figure3 oSupportDownloadAssistant 36 Figure 4 Mechanical Drawing of the Network
2. 43 Table 10 I2C compatible Connector 2 48 6 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual OSFP Port Network Interface Card User Manual for OCP Rev 1 2 Revision History This document was printed on February 4 2015 Table 1 Revision History Table Date Rev Comments Changes February 2015 1 2 Updated document to indicate card respin to comply with OCP spec 2 0 Updated Flash part number to W25Q32FVSSIG Changed description of I2C interface See Section 2 3 I2C compatible Interface on page 13 and Appendix B 2 I2C com patible Connector Pinout on page 48 Updated voltage to SVAUX and 3 3VAUX See Section 7 1 MCX345A BC P Q N Specifications on page 40 and Section 7 2 MCX346A BC P Q N Specifications on page 41 Updated LED functionality See Section 7 4 Adapter LED Operation on page 43 Added board labels See Appendix A Finding the MAC GUID and Serial Number on the Card on page 44 July 2014 1 1 Updated note on air flow See Section 7 1 MCX345A BC P QIN Specifications on page 40 and Section 7 2 MCX346A BC P Q N Specifications on page 41 Added Appendix B 3 NC SI Interface Pinout on page 48 October 2014 1 0 First Release Mellanox Technologies 7 Mellanox Technologies Confidential
3. 15 3 5 Cables and Modules io e tes e rte a ee ne he 15 3 6 Identify the Card in Your System 16 3 64 On Windows se e sakasi REO LU fog la OUR I P Pss 16 3 0 2 cm Ein hw a dal eae 17 Chapter 4 Driver 18 Als Pin DfiNet a Aa wy dads de Eo 18 4 1 1 Hardware and Software Requirements 18 4 12 Inst lline the Driver sak see ae esek al ne mii IS e d PIDE 18 413 Loading the Ditvet ea 19 4 1 Unloadmg the DEVer ein an at Ee cere ess 19 41 5 Uninstalling the Driver 22 54 eos eee serena bd EE dll by e 19 4 2 Windows Driver iel een A i an el CU E ee ip 19 4 2 1 Hardware and Software Requirements 20 422 Downloading MLNX WinOF 20 423 Extracting Files Without Running Installation 20 424 Installing MLNX WinOF m 22 4 2 4 1 o Attended Installation 23 4 2 4 2 Unattended Installation 28 4 2 5 Upgrading MLNX WinOF 7 27 29 4 26 Installation Results 2 2 eene Ie er REPAIR e rer ge 29 4 2 7 OpenSM Activation
4. Mellanox ConnectX 3 Ethernet Adapter 2 Microsoft Kernel Debug Network Adapter 4 Other devices Unknown device p Ports COM amp LPT p deh Print queues b D Processors b Storage controllers Wi System devices yi ACPI Fixed Feature Button Composite Bus Enumerator Direct memory access controller High precision event timer Intel R 5520 5500 X58 I O Hub Control Status and RAS Registers 3423 Intel R 5520 5500 X58 1 0 Hub GPIO and Scratch Pad Registers 3422 Intel R 5520 5500 X58 1 Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 I O Hub PCI Express Root Port 3 340A Intel R 5520 5500 X58 1 0 Hub PCI Express Root Port 7 340E Intel R 5520 5500 X58 I O Hub System Management Registers 342E Intel R 5520 5500 58 I O Hub to ESI Port 3406 Intel R 82801 PCI Bridge 244E Intel R ICH10 Family PCI Express Root Port 1 3440 Intel R ICH10 Family PCI Express Root Port 5 3448 Intel R ICH10R LPC Interface Controller 3416 188 Microsoft ACPI Compliant System Microsoft Generic IPMI Compliant Device 1i Microsoft System Management BIOS Driver Microsoft Virtual Drive Enumerator Numeric data processor PCI Express Root Complex 4 2 7 5 Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface
5. BOND NUVOTON EEPROM accessible through the I C compatible interface The EEPROM capacity is 4KB RDMA over Converged Leveraging Data Center Bridging capabilities RoCE provides efficient low Ethernet RoCE latency services over Layer 2 Ethernet CPU offload Adapter functionality enabling reduced CPU overhead allowing more available CPU Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10 or 40GbE The hardware based stateless offload Sockets Acceleration engines in ConnectX 3 Pro reduce the CPU overhead of IP packet transport Sockets acceleration software further increases performance for latency sensitive applications Support for port based Ouality of Service enabling various application reguire Quality of Service QoS ments for latency and SLA Hardware based I O ConnectX 3 Pro provides dedicated adapter resources and guaranteed isolation virtualization and protection for virtual machines within the server ConnectX 3 Pro effectively addresses the increasing demand for an overlay net work enabling superior performance by introducing advanced NVGRE and Virtualized Overlay VXLAN hardware offload engines that enable the traditional offloads to be per Networks formed on the encapsulated traffic With ConnectX 3 Pro data center operators can decouple the overlay network layer from the physical NIC performance thus achieving native performance in the new network archite
6. oe hee een e te es eat e 30 4 2 8 Uninstalling MLNX_WinOF 2 3l 4 2 8 1 Attended Uninstall 31 4 2 8 2 Unattended Uninstall 31 Mellanox Technologies 3 Mellanox Technologies Confidential Rev 1 2 4 2 9 Assigning Port IP After Installation 31 4 3 VMware Diver sadni sola ma da M Ras 32 43 1 Installing and Running VBI Driver on ESXi 5 x 33 4 3 2 Installing and Running the offline bundle Driver on 1 5 33 4 3 3 Removing the VIB offline bundle Driver 34 44 BlexBoot cz da n he wb dug a isn A Seu 34 Chapter 5 Updating Card 35 Chapter 6 37 6 T General a ah ge 37 6 2 el a ce tUe A ee UR 38 ES Sizi cast My ak eR Ce e n n tet ro A AERE EL 39 Chapter 7 Specifications 40 71 MCX345A BC P O N Specifications 40 7 2 MCX346A BC P Q N Specifications 41 7 3 Board Mechanical Drawing and Dimensions
7. UE V V Chapter 1 Introduction 10 120 Product Overview eme na da REA VE TREAT ES ba 10 1 2 F atures and Benefits reines seen dae Rae REGN a GR NENNEN IE ee 11 1 3 Operating Systems Distributions 11 TA est eodd Med anl et eter e Ate e Same ya aa 12 Chapter 2 13 EthemetIntertace 232228532 4 uw das daca te an dez ae ada 13 2 2 Interface nass ey rr aH er Ur Dr pg 13 2 3 DC compatiblelnterface rr 13 2 4 LED Interfaces minre joie nee Dee eges A E UR 13 2 5 Network Controller Sideband Interface NC SD 14 Chapter 3 Hardware 15 3 12 System Requirements ix finir Ar ae on EAE A e Ax 15 3151 Hardware eot e ar e dede e sce dre e 15 3 1 2 Operating Systems Distributions 15 3 3 Software Stacks ii ESAE ER EU UEM TA as Klin enr 15 3 2 Safety PreCcautions et sos E e Ate e 15 33 Pre installationChecklist 15 34 CardInstallation Instructions
8. 0x0002c90300e8eef0 System image 0x0002c90300e8eef0 Posten State Down Physical state Disabled Base 114 0 0 SM lid 0 Capability mask 0x00010000 Port GUID 0x0000000000000000 Link layer Ethernet Zee As State Down Physical state Disabled Reices JLo Base lid 0 TMEO SM lid 0 Capability mask 0x00010000 Port GUID 0x0000000000000000 Link layer Ethernet flint d dev mst mt4103_pci_cr0 qq q W Running quick query Skipping full image integrity checks Image type FW Version Device ID Description GUIDs 0002c90300e8eef3 MACs VSD PSID ConnectX 2 30 5000 4103 Node Porti Port2 Sys image 0002c90300e8eef0 0002c90300e8eef1 0002c90300e8eef2 0002c9e8eef0 0002c9e8eef1 n a MT_1200111023 46 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single Dual OSFP Port Network Interface Card User Manual for Rev 1 2 Appendix Interface Connectors Pinout B 1 Mellanox Technologies 47 PCI Express x8 Connector Pinout The cards use a standard PCI Express x8 connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification 61 T 87 UU 01 0 0 05 e 0 72 0 7 MEZZ TX DN 5 wa la io TX DN TX DP C 7 Figure 9 PCle Connector Pinout P12V 1 p12V 2 p12V 3 CND 26 CND 27 P3V3 AUX 2 CND 28 GND 2
9. OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically e Performances tools install the performance tools that are used to measure the InfiniBand performance in user environment Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment SDK contains the libraries DLLs for developing InfiniBand application over IBAL Documentation contains the User Manual and Installation Guide NDFLTR DLLs contains the files for standalone installation of the mlx4nd provider 26 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Custom Setup Select the program Features you want installed Click on an icon in the list below to change how a feature is installed Feature Description Performace Tools Subnet manager 5M Analyze Tools SDK Documentation ND FLTR DLLs This Feature requires 1164KB on your hard drive Install to C Program Files Mellanox MLNX_ PIVIB Tools InstallShield b Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation Tf you want to review or change any of your installation settings click Back Cli
10. Connectivity on page 12 1 1 Product Overview The following tables provide the ordering part number port speed number of ports and PCI Express speed Table 3 Single and Dual port 40 Gigabit Ethernet Network Interface Cards MCX345A BCPN MCX345A BCQN OMEN MCX346A BCPN MCX346A BCQN 40GigE Data Transmission Rate MCX345A BCPN single port QSFP IPMI and NC SI disabled MCX345A BCON single port QSFP with IPMI and NC SI sup Number of ports port MCX346A BCPN dual port QSFP IPMI and NC SI disabled MCX346A BCON dual port OSFP with IPMI and NC SI support PCI Express SERDES Speed PCIe 3 0 x8 8GT s RoHS R6 Adapter IC Part Number MT27524A0 FCCR BE 10 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 1 2 Features and Benefits Table 4 Features Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to PCI Express PCIe 8GT s Mellanox cards comply with the following IEEE 802 3 standards Std 802 3 2008 Ethernet 40 Gigabit Ethernet Std 802 3ae 10 Gigabit Ethernet Std 802 3ba 40 Gigabit Ethernet Std 802 3ad Link Aggregation and Failover PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 4MB SPI Flash device W25Q32FVSSIG device WIN
11. Rev 1 2 About this Manual This User Manual describes Mellanox Technologies ConnectX 3 Pro 40 Gigabit Ethernet Single and Dual QSFP Port PCI Express x8 network interface cards It provides details as to the inter faces of the board specifications required software and firmware for operating the board and rel evant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet networks and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual Document no 2204UG User Manual describing the set of MFT firmware management tools for a single node See http www mellanox com gt Products gt Software gt Firm ware Tools MLNX EN for Linux README Driver Kit for Mellanox Adapter Cards with 10GigE Support Document no 2950 This document provides information on the MLNX EN Linux driver and instructions for installing the driver on Mellanox Con nectX adapter cards supporting 10Gb s Ethernet Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX OFED WinOF VPI for Windows User Manua
12. 4 Log into the VMware ESXi server machine as root 5 You can either a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software remove net mlx4 en b Installthemlx4 en driver VIB package Run gt esxcli software vib install v lt vib url gt Reboot ESXi server The driver will be loaded automatically OR a Update the driver Run gt esxcli software vib update v lt vib url gt b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4 en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by MLX4 EN driver should be displayed In Non Multifunction Mode port 2 is identified as a pseudo device Therefore devices are not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 8 2 4 3 2 Installing and Running the offline bundle Driver ESXi 5 x 1 Copy the offline bundle zip file to ESXi 5 0 machine and extract its contents 2 You can install the driver in one of the following ways a Remove any e
13. ES F45214EF003C P0 44 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Figure 7 MCX346A BCPN Board Label Example only see Note 2009 04 22 Model No CX346A ConnectX 3 Pro 40 Gigabit Ethernet Made in Israel S N 1210 00011 PIN MCX346A BCPN 1 F45214 P1 gt EF003D lt PO F45214EF003C X F45214EF003E ME gt Figure 8 MCX346A BCQN Board Label Example only see Note 2009 04 22 Model No CX346A ConnectX 3 Pro 40 Gigabit Ethernet Made in Israel LL NL S N 1210 00011 P N MCX346A BCQN B1 F45214 P1 gt EF003D Re lt PO F45214EF003C X F45214EF003E ME gt A 1 Retrieving Card GUID On ConnectX 3 Pro Ethernet NICs there is a mismatch between the GUID value returned by firm ware management tools and that returned by fabric driver utilities that read the GUID via device Mellanox Technologies 45 J Mellanox Technologies Confidential Rev 1 2 firmware using ibstat MIxburn flint returns 0 as GUID while the utilities return a value derived from the MAC address For all driver firmware software purposes the latter value should be used Please see the below example ibstat mlx4 0 CA type MT4103 Number of ports 2 Firmware version 2 30 5000 Hardware version 0 Node
14. Refer to the User Manual for installation instructions Melanos Once installed open a CMD window and cd C Program Files Mellanox WinMFT mst start mst status flint d mst device q Ports Information vstat Download the latest firmware version using the PSID board ID on http www mellanox com supportdownloader Upgrade flint d mst device 1 firmware bin file b Event log viewer MST device logs mst start Collect log file mst status C Users Administrator gt flint d lt mst device dc gt dump_configuration log C Users Administrator gt mstdump lt mst_device gt dc gt mstdump log Mellanox Technologies 39 Mellanox Technologies Confidential Rev 1 2 Specifications 7 Specifications 7 1 MCX345A BC P Q N Specifications Table 7 MCX345A BC P Q N Specifications Table Size 2 681 x 4 3 in 68mm x 110mm Physical Connector QSFP 40Gb s Ethernet 40GBASE CRA SR4 also supports the following when used with QSA 10GBASE CR SR 1000BASE T SR Protocol Support Data Rate 1 10 40Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8lanes 2 0 and 1 1 compatible Voltage SVAUX and 3 3VAUX Typ Power Passive Cables 7 18W Max Power Passive Cables 7 73W Power and Max power available through QSFP port 1 5W Environmental Temperature Operational 0 C to
15. 35 Non operational 0 C to 70 Humidity 90 relative humidity Air Flow 300LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 Regulatory ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from the Mezz from the cooling air inlet side 40 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 7 2 MCX346A BC P Q N Specifications Table 8 MCX346A BC P Q N Specifications Table Size 2 68in x 4 3 in 68mm x 110mm Physical Connector QSFP 40Gb s Ethernet 40GBASE CRA SR4 also supports the following when used with QSA 10GBASE CR SR 1000BASE T SR Protocol Support Data Rate 1 10 40Gb s Ethernet PCI Express Gen3 SERDES 9 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage SVAUX and 3 3VAUX Typ Power Passive Cables 8 68W Max Power Passive Cables 10 66W Power and Max power available through QSFP port 1W Environmental Temperature Operational 0 C to 35 Non operational 0 C to 70 Humidity 90 relative humidity Air F
16. Ethernet standards listed in Table 4 Features on page 11 2 2 PCI Express Interface The 3 Pro network interface cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists the PCIe interface features PCle Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 or 8 0GT s link rate x8 e Auto negotiates to x8 x4 x2 or xl Support for MSI MSI X mechanisms 2 3 l C compatible Interface A three hole footprint for C harness is provided as the I2C compatible interface See Figure 1 Figure 1 2 Harness 2 4 LED Interface There are two I O LEDs per port For LED specifications please refer to Section 7 4 Adapter LED Operation on page 43 Mellanox Technologies 13 Mellanox Technologies Confidential Rev 1 2 Interfaces 2 5 Network Controller Sideband Interface NC SI The adapter supports a slave Network Controller Sideband Interface NC SI that can be con nected to a BMC The adapter s NC SI implementation supports all mandatory NC SI commands specified in the Network Controller Sideband Interface NC SI Specification Rev 1 0 0a When the ConnectX 3 Pro network controller drive is running on the main CPU messages that target the adapter s NC SI are forwarded by the adapter to the host memory the adapter then fetc
17. Installation An installation procedure that requires frequent user intervention Unattended Installation 22 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 An automated installation procedure that requires no user intervention Both Attended and Unattended installations require administrator privileges ke 4 2 4 1 Attended Installation The following is an example ofa MLNX WinOF win8 x64 installation session Step 1 opening a CMD console Double click the exe and follow the GUI instructions to install MLNX_WinOF To configure your setup to contain the logs option please run the following command after MLNX VPI WinOF 4 40 0 All win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run the following command MLNX VPI WinOF 4 40 0 All win8 x64 exe v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Step2 Click Next in the Welcome screen Lal Welcome to the InstallShield Wizard for MLNX The InstallShield R Wizard will install MLNX_YPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties lt Back Mellanox Technologies 23 Mellanox Technologies Confidential Rev 1 2 Driver Installation Step3 Read then accept
18. install the latest MLNX EN driver software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt ConnectX 3 EN 10 40GigE Linux Driver gt Download Follow the installation instructions included in the download package also available from the download page 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms CPU architectures x86 64 x86 e power pc Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux Operating Systems RedHat EL5 8 RedHat EL5 9 RedHat EL6 2 RedHat EL6 3 OEL6 2 2 6 32 279 19 1 OEL6 3 2 6 32 279 19 1 SLESII SPI SLESII SP2 Software Dependencies To install the driver software kernel sources must be installed on the machine MLNX_EN driver cannot coexist with OFED software on the same machine Hence when installing MLNX_EN all OFED packages should be removed done by the mInx en install script 4 1 2 Installing the Driver Step 1 Download Driver Package Please download the current driver package from http www mellanox com gt Products gt Software Ethernet Driver Linux Driver Download Step 2 Install Driver Run the following commands to install the driver Gael 5 10 egr iile gt cd mlnx 1 5
19. 1 Make sure no previous driver version is currently loaded Run gt modprobe r mlx4 en Step 2 Load the new driver version Run gt modprobe mlx4 en The result is a new net device appearing in ifconfig a output 4 1 4 Unloading the Driver To unload the Ethernet driver run gt modprobe mlx4 en 4 1 5 Uninstalling the Driver To uninstall the mlnx en driver run gt sbin mlnx en uninstall sh 4 2 Windows Driver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com gt Products gt Software Mellanox Technologies 19 J Mellanox Technologies Confidential Rev 1 2 Driver Installation gt Bthernet Drivers gt Windows SW Driver Download Follow the installation instructions included in the download package also available from the download page 4 2 1 Hardware and Software Requirements Table 6 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Windows Server 2008 R2 64 bit only Installer Privileges The installation requires administrator privileges on the target machine 4 2 2 Downloading MLNX WinOF Follow these steps to download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click st
20. 10 gt install sh The package consists of several source RPMs The install script rebuilds the source RPMs then installs the created binary RPMs The created kernel module binaries are placed under lib 18 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 modules lt kernel ver gt updates kernel drivers net mlx4 mlnx en installer supports 2 modes of installation The install scripts selects the mode of driver installation depending of the running OS kernel version 1 Kernel Module Packaging KMP mode where the source rpm is rebuilt for each installed flavor of the kernel This mode is used for RedHat and SUSE distributions 2 Non KMP installation mode where the sources are rebuilt with the running kernel This mode 1s used for vanilla kernels Note If the Vanilla kernel is installed as rpm please use the disable kmp flag when installing the driver The kernel module sources are placed under usr src mellanox mlnx en 1 5 10 Run the follow ing commands to recompile the driver gt cd usr src mellanox mlnx en 1 5 10 gt scripts mlnx en patch sh gt make gt make install The uninstall and performance tuning scripts are installed Note If the driver was installed without kmp support the sources would be located under usr srs mlnx en 1 5 10 4 1 3 Loading the Driver Step
21. 2 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de LA rayos 4 Instalaci n de eguipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 5 Eliminaci n de eguipos La eliminaci n definitiva de este eguipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 6 C digos el ctricos locales y nacionales A Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales Mellanox Technologies 55 J Mellanox Technologies Confidential Rev 1 2 7 Exposici n a niveles de radiaci n peligrosos 56 Mellanox Technologies Mellanox Technologies Confidential
22. 5 D3V3 5 D3V3 5 D3V3 7 p3vi 8 CND 30 LAN 3V38TB ALERT N LAN 3V3STB CLK LAN 3V3STB DAT PCIE WAKE N REVD DA 088 GND 31 REVD SATA_TX REVD SATA_TX CND 32 GND 33 CLK 100M MEZZ2 DD CLK 100M MEZZ2 DN GND 34 GND 35 MEZZ TX DP 0 TX DN 0 CND 36 CND 37 MEZZ TX DP C 1 TX DN C 1 CND 38 CND 35 MEZZ TX DP C 2 MEZZ TX DN C 2 CND 40 CND 41 MEZZ TX DP C 3 CND 42 CND 43 MEZZ TX DP C 4 MEZZ TX DN C 4 CND 44 GND 45 MEZZ TX DP C S MEZZ TX DN 5 CND 46 CND 47 MEZZ TX DP C 6 CND 48 GND 4 MEZZ TX DN C 7 GND 50 CND 51 GHD 52 MEZZ DRSNT2 N RSVD MEZZ PRSNTI N PSV AUX 1 PSV AUX 2 PSV AUX 3 GND 1 GND 2 P3V3 AUX 1 GND 3 CND 4 D3V3 1 p3vi 2 pivi 3 D3V3 4 RSVD MEZZ CPRENT1 N RSVD MEZZ CPRENT2 N REVD S8D PRENT N RST PLT MEZZ N RSVD MEZZ SMCLE RSVD MEZZ SMDATA 5 CND 6 RSVD SATA RX RSVD SATA RX GND 7 CND 8 100M MEZZ1 DP 100M MEZZ1 DN CND 5 GND 10 DP 0 DN O GND 11 GND 12 RX DP 1 RX DN 1 m 3 GND 14 MEZZ RX DP 2 MEZZ RX DN 2 GND 15 GND 16 MEZZ DD 3 MEZZ DN 3 CND 17 GND 18 MEZZ RX 4 MEZZ DN 4 GND 15 GND 20 MEZZ 5 MEZZ EX DN S 21 GND 22 RX 6 MEZZ RX DN 6 GuD 23 GND 24 MEZZ DD 7 RX DN 7 GuD 25 PCle Connector Plug Mellanox Technologies
23. Confidential 27 e Rev 1 2 B 2 l C compatible Connector Pinout A three hole footprint for 2 Harness is provided as the I2C compatible interface Figure 10 Compatible Connector Plug and Pinout Table 10 I2C compatible Connector Pinout Connector Pin Number Signal Name GND GND 2 SCL 3 SDA Lo y gt B 3 NC SI Interface Pinout Figure 11 NC SI Interface Pinout NCSI_TXD 1 0 1 0 ConnectX 3 Pro NCSI TX EN CRS DV NCSI CRS DV TXEN Ncs 70100 Management Controller 48 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Appendix C Safety Warnings For safety warnings in French see Avertissements de s curit d installation Warnings in French on page 34 For safety warnings in German see Sicherheitshinweise Warnings in German on page 36 For safety warnings in Spanish see Advertencias de seguridad para la instalaci n Warn ings in Spanish on page 38 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 55 C 131 F To guarante
24. Interface Card 42 Figure 5 MCX345A BCPN Board Label Example only see Note 44 Figure 6 45 Board Label Example only see Note 44 Figure 7 MCX346A BCPN Board Label Example only see Note 45 Figure 8 46 Board Label Example only see Note 45 Figure 9 PCle Connector 47 Figure 10 Compatible Connector Plug and 48 Figure 11 NC SiMhmterfacePinout ne nen eae 48 Mellanox Technologies 5 Mellanox Technologies Confidential Rev 1 2 List of Tables Table 1 Revision History Table 7 Table 2 7 Documents Lista ated Alar a a BRD tah sek ed ae edes 8 Table 3 Single and Dual port 40 Gigabit Ethernet Network Interface Cards 10 Table4 Features ale aa hepsi heykeline kele hibe el Ee 11 Table 5 Software and Hardware Requirements 18 Table 6 Software and Hardware Requirements 20 Table 7 MCX345A BC P Q N Specifications 40 Table 8 MCX346A BC P Q N Specifications 41 Table 9 Physical and Logical Link Indication
25. LIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS TestX PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendXTM FabricIT HPC X Mellanox Open Ethernet PeerDirect Mellanox Virtual Modular Switch MetroDX Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 3455 Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Table of Contents Table of Contents AS ORR List OF Figures sine Re B E SAS ee 3 5 Last OF Tables ss I RII ka kl iii ze Revision History 25 E ee nee ele Bk 8 About this Manuali Rey Rn
26. Mellanox TECHNOLOGIES ConnectX 3 Pro 40Gb s Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Open Compute Project P N MCX345A BCPN MCX346A BCPN MCX345A BCQN MCX346A BCON Rev 1 2 www mellanox com Mellanox Technologies Confidential 1 2 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEG
27. NX WinOF in unattended mode perform the following Step 1 Open a CMD console Step 2 Uninstall the driver Run MLNX VPI WinOF 4 40 0 All win8 x64 exe S x v qn 4 2 9 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine To obtain the MAC address Step 1 Open a CMD console Step 2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters gt To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices 1 e Net Networ gt v Search Network Connections 2 Organize v fil e Name Device Name Ethernet Xj Ethernet 2 x Ethernet 3 Ethernet 4 di 0 Local Area Connection IBM USB Remote NDIS Network Device Step2 Right click a Mellanox Local Area Connection and left click Properties Step 3 Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties Mellanox Technologies 31 Mellanox Technologies Confidential Rev 1 2 Driver Installation Step 4 Select the Use the following IP address radio button and enter the desired IP information General You can get IP settings assigned autom
28. ace Card User Manual for OCP Rev 1 2 Step3 Optional To configure your setup to contain the logs option please run the following com mand MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v l vx LogFile Step 4 Optional If you do not want to upgrade your firmware version run the following command MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h 4 2 5 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version In Windows 2012 MLNX WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade 4 2 6 Installation Results Upon installation completion you can verify the successful addition of the network card s through the Device Manager To see the Mellanox network adapter device and the Ethernet or IPolB network device depending on the used card for each port display the Device Manager and expand System devices or Network adapters Mellanox Technologies 29 Mellanox Technologies Confidential Rev 1 2 Driver Installation File Action View Help i
29. ackage from the ESXi server machine run gt esxcli software vib remove n net mlx4 en 4 4 FlexBoot FlexBoot supports remote Boot over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot software is based on the open source IPXE project see www ipxe org For more information go to http www mellanox com gt Products Software gt Ethernet Drivers gt Download 34 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 5 Updating Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally and the most recent firmware can be obtained from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your network i
30. anual for OCP Rev 1 2 Appendix E Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 55 C 131 F betrieben werden Es ist ein Luft A strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Ger teinstallation Diese Ger t sollte nur von geschultem und gualifiziertem Personal installiert ausge tauscht oder gewartet werden 5 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 6 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden Mellanox Technologies 53 Mellanox Technologies Confidential Rev 1 2 7 Strahlenkontak 54 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1
31. arlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4 en driver offline bundle package Run gt esxcli software vib install d path mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip c Reboot ESXi server The driver will be loaded automatically OR Mellanox Technologies 33 Mellanox Technologies Confidential Rev 1 2 Driver Installation a Update the driver Run gt esxcli software update net mlx4 en d lt path gt mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4 en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by MLX4 EN driver should be displayed In Non Multifunction Mode port 2 is identified as a pseudo device Therefore devices gt are not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 8 Ai 4 3 3 Removing the VIB offline bundle Driver Toremove the VIB offline bundle driver p
32. art gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURES On an x64 64 bit machine the output will be AMD64 Step2 Goto the MLNX WinOF for Windows Web page at http www mellanox com gt Products gt Software gt Ethernet Drivers gt Windows SW Drivers Step3 Download the exe image according to the architecture of your machine see step Step 1 The name of the exe is in the following format MLNX_VPI_WinOF lt ver sion All lt OS gt _ lt arch gt exe Installing the incorrect exe file is prohibited If you do so an error message will be dis E played For example if you try to install a 64 bit exe on a 32 bit machine the wizard will dis play the following or a similar error message Windows Installer X 4 This installation package is not supported by this processor type Contact your product vendor OK 4 2 3 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 CMD console Click Start gt Run and enter CMD 20 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Step 2 Enter the following command MLNX VPI WinOF version All OS lt arch gt exe a Step3 Click Next to create a server image Welcome to the InstallShield Wizard f
33. atically if your network supports this capability Otherwise you need to ask your network administrator For the appropriate IP settings Obtain an IP address automatically 8 Use the following IP address IP address Subnet mask Default gateway Obtain DNS server address automatically Use the following DNS server addresses Preferred DNS server Alternate DNS server validake settings upon exit Cancel Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from a CMD console ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix TP Address 212263 Sk a te a ass ss LANDS Default Gateway E 4 3 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ESXi Server software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers gt Download Follow the installation instructions included in the download package also available from the download page 32 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 4 3 1 Installing and Running the VBI Driver on ESXi 5 x
34. ck Cancel to exit the wizard InstallShield Mellanox Technologies 27 Mellanox Technologies Confidential Rev 1 2 Driver Installation Step8 Click Finish to complete the installation InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32LogFiles PerformanceTunin 9 109 Show release notes lt Back Cancel If the firmware upgrade fails the following message will be displayed InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPI Click Finish to exit the wizard You chose to run performance tuning The log File can be found at C windows System32 LogFiles PerformanceTunin g log Firmware upgrade Failed with error 1003 Error Description We could not Find a new version For the chosen device Please refer to the UM for Cl Show release notes lt Back 4 2 4 2 Unattended Installation The following is an example of a MLNX WinOF win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter cmd Step2 Install the driver Run gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v gn 28 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interf
35. cture ConnectX 3 Pro SR IOV technology provides dedicated adapter resources and SR IOV guaranteed isolation and protection for virtual machines VM within the server VO virtualization with ConnectX 3 Pro gives data center managers better server utilization while reducing cost power and cable complexity 1 3 Operating Systems Distributions Novell SLES Red Hat Enterprise Linux RHEL Fedora CentOS and other Linux dis tributions Mellanox Technologies 11 Mellanox Technologies Confidential Rev 1 2 Introduction Microsoft Windows Server 2008 CCS 2003 HPC Server 2008 OpenFabrics Enterprise Distribution OFED e OpenFabrics Windows Distribution WinOF VMware ESX Server 3 5 vSphere 4 0 4 1 1 4 Connectivity Interoperable with 10 40GbE switches Passive copper cable with ESD protection Powered connectors for optical and active cable support QSFP to SFP connectivity through QSA module Passive copper cable with ESD protection 12 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 2 Interfaces Each network interface card includes the following interfaces Ethernet Interface PCI Express Interface e 2C compatible Interface LED Interface 2 1 Ethernet Interface The network ports of the ConnectX 3 Pro network interface cards are compliant with the IEEE 802 3
36. e 55 C 131 F Un flux d air de 200LFM LA cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 5 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 6 Codes lectriques locaux et nationaux YN Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 51 Mellanox Technologies Confidential Rev 1 2 7 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 52 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User M
37. e proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Equipment Disposal 4 Disposal of this equipment should be in accordance to all national laws and regula tions 5 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical 4 codes 6 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure Mellanox Technologies 49 Mellanox Technologies Confidential Rev 1 2 CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 1 1996 A2 2001 50 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand d
38. firmware release version and the card part number Note Please contact Mellanox System Support if you cannot find the firmware binary for your card Mellanox Technologies 35 Mellanox Technologies Confidential Rev 1 2 Updating Card Firmware Figure 3 Support Download Assistant Mellanox TECHNOLOGIES Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR MT 1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Switches Gateways 1 If a newer firmware version exists for your network interface card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above b Unzip the firmware image c Bum the firmware image Enter gt flint d dev mst mt4099 pci i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4099 pci cr0 Image type ConnectX FW Version 2 2311010 Device ID 4099 36 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 6 6 1 Mellanox Technologies 37 Tro
39. hes the message from the host memory and forwards it to the BMC over the NC SI interface When the main CPU 1s powered down or the adapter network controller is not running messages that target the adapter s NC SI are forwarded directly by the adapter to the BMC using the NC SI interface without being copied to the host memory See Appendix B 3 NC SI Interface Pinout on page 48 14 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 3 Hardware Installation 3 1 System Requirements 3 1 1 Hardware This card requires a PCI Express connector as specified in the Open Compute Project Intel Moth erboard spec v2 0 3 1 2 Operating Systems Distributions Please refer to Section 1 3 Operating Systems Distributions on page 11 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX EN for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation 3 2 Safety Precautions The card is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make sure to use only insulated tools Verify that the system 15 powered off and is unplugged Be DD x It is strongly recomme
40. hrough the following methods URL http www mellanox com gt Support E mail support mellanox com e Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database informa tion for Mellanox products Access the data base from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Mellanox Technologies Confidential Rev 1 2 Introduction 1 Introduction This is the User Guide for Mellanox Technologies Ethernet network interface cards based on the ConnectX 3 Pro EN integrated circuit device These card s connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enter prise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics Section 1 1 Product Overview on page 10 Section 1 2 Features and Benefits on page 11 Section 1 3 Operating Systems Distributions on page 11 Section 1 4
41. l Document no 3280 User Manual describing WinOF features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products Software gt Windows SW Drivers gt Mellanox OFED for Windows WinOF Mellanox MLX4 EN Driver for VMware README Document no 3527 User Manual describing MLX4 EN driver for VMware features performance diagnostic tools content and configuration See http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechanical Specifications 8 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Online Resources Mellanox Technologies web pages http www mellanox com Mellanox Technologies Firmware download web page http www mellanox com gt Support gt Download Center Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us t
42. link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as 1t requires the GUID parameter to decide on which port to work Setting OpenSM upon setup results in it working only for the first port and not for the others Torun OpenSM as a service assuming the package was installed in the default path use Sc create OpenSM1 binPath c Program Files Mellanox MLNX VPI IB Tools opensm exe service start auto To start the service run Sc start opensm For further information please refer to the OpenSM Subnet Manager chapter in the User Manual 30 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 4 2 8 Uninstalling MLNX WinOF 4 2 8 1 Attended Uninstall Touninstall MLNX WinOF on a single node perform one of the following options 1 Click Start gt Control Panel gt Programs and Features gt MLNX VPI Uninstall NOTE This requires elevated administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start gt All Programs gt Technologies gt MLNX WinOF Uninstall MLNX WinOF 4 2 8 2 Unattended Uninstall Touninstall ML
43. low 300LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from Mezz from the cooling air inlet side Mellanox Technologies 41 Mellanox Technologies Confidential Rev 1 2 Specifications 7 3 Board Mechanical Drawing and Dimensions dimensions are in millimeters 4 All the mechanical tolerances are 0 1mm Note For single port cards Port 2 14 is not assembled Figure 4 Mechanical Drawing of the Network Interface Card 110 05 s E EEE Ig s TE 5 62 55 42 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 7 4 Adapter LED Operation There are two I O LEDs per port See Table 9 for different LED functions Table 9 Physical and Logical Link Indication Port and LED Link LED Function PO LEDO Physical link Green 40G link is present Physical link Yel Other link status is present low PO LEDI Logical Link Activity Green P1 LEDO Physical link Green 40G link i
44. nd Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the HCA s Firmware version Recommended Upgrading the Firmware version will reboot your machine Note One or more of your adapters has an old firmware version We recommend upgrading to a newer firmware version to enable improved Functionality and support driver s capabilities InstallShield Step 6 Configure your system for maximum performance by checking the maximum performance box Maximum Performance Check this box to configure your system For maximum performance Y Check this box to configure your system For maximum performance Recommended Note This step requires you to reboot the machine at the end of the installation process InstallShield Mellanox Technologies 25 Mellanox Technologies Confidential Rev 1 2 Driver Installation This step requires rebooting your machine at the end of the installation Step 7 Select a Complete or Custom installation follow Step a and on page 26 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete 3 All program features will be installed Requires the most disk space Custom Choose which program Features you want installed and where they will be installed Recommended For advanced users InstallShield lt Back Cancel a Select the desired feature to install
45. nded to use an ESD strap or other antistatic devices 3 3 Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord A Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged 3 4 Card Installation Instructions To be updated in future release 3 5 Cables and Modules To obtain the list of supported cables for your NIC go to www mellanox com gt Products gt Cables and Transceivers Mellanox Technologies 15 J Mellanox Technologies Confidential Rev 1 2 Hardware Installation 3 6 Identify the Card in Your System 3 6 1 Windows Note Figure 2 is an example using ConnectX 3 Pro device This figure will be updated in future releases 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 Pro network interface card 3 Right click the mouse on your card s row and select properties to display the network inter face card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 5 Hardware Ids Windows 2008 R2 from the Properties pull down menu Figure 2 PCI Device 1 Mella
46. nox Connect lt 3 MT 04099 Network Adapter Property Hardware Ids Value PCINVEN 15B3 amp DEV 1003 51 5 5 005015B3 amp REV 00 PCINVEN 15B3 amp DEV 1003 amp 5UBSYS 00501583 PCIVEN 15B3 amp DEV 1003 amp CC 028000 PCIVEN 15B3 amp DEV 1003400 0280 6 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string PCI VEN 15B3 amp DEV 1003 VEN is equal to Ox15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this isa valid Mellanox Technologies PCI Device ID If the PCI device does not have a Mellanox adapter ID return to Step 2 to check another device t The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 t 16 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 3 6 2 Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt Ispci grep i Mellanox 27 00 0 Network controller Mellanox Technologies MT27520 Family ConnectX 3 Pro Mellanox Technologies 17 Mellanox Technologies Confidential Rev 1 2 Driver Installation 4 Driver Installation 4 1 Linux Driver For Linux download and
47. nterface card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the package The package is available at http www mellanox com gt Products Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci cr0 d Get the PSID firmware identification and programmed firmware version using the command Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for this product and are included here for illustration purposes only flint d dev mst mt4099 pci cr0 Image type ConnectX FW Version 2 30 4260 Device ID 4099 Description Node Porti Port2 Sys image GUIDs AE TEE TE YE E IPAE LEE IE JE E JE ETE IE IE EIE ME EAE TE AE MACs 0002c9305350 0002c9305351 VSD PSID MT 1080120023 1 Compare the programmed firmware version with the latest available Go to Mellanox s web site http www mellanox com supportdownloader See Figure 3 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the
48. or MLNX VPI The InstallShield R Wizard will install MLNX VPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Step 4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX VPI at the specified network location or click Cancel to exit the wizard Network location 2 2 InstallShield Mellanox Technologies 21 Mellanox Technologies Confidential Rev 1 2 Driver Installation Step 5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX VPI at the specified network location or click Cancel to exit the wizard Network location a Change Installshield Step 6 complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPT Click Finish to exit the wizard 4 2 4 Installing MLNX WinOF This section provides instructions for two types of installation procedures Attended
49. s present Physical link Yel Other link status is present low PI LEDI Logical Link Activity Green The short bracket has the same port and LED footprints as the tall bracket 7 5 Regulatory Statements For regulatory statements for all ConnectX 3 Pro cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Mellanox Technologies 43 Mellanox Technologies Confidential 1 2 Appendix A Finding the MAC GUID and Serial Number on the Card Each Mellanox card has a label on the print side that shows the card serial number and the card MAC for Ethernet protocol Note The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card Card revision changes are communicated via Product Change Notification PCN documents that are available via card suppliers Figure 5 MCX345A BCPN Board Label Example only see Note 2014 11 13 Model No CX345A ConnectX 3 Pro 40 Gigabit Ethernet Made in Israel NET SIN MT1210X00011 P N MCX345A BCPN Rev B1 EX ME F45214EF003E 45214 003 0 gt Figure 6 MCX345A BCQN Board Label Example only see Note 2014 11 13 Model No CX345A ConnectX 3 Pro 40 Gigabit Ethernet Made in Israel TUE AN SIN MT1210X00011 P N MCX345A BCQN Rev B1 Exe 45214 003
50. shed Mellanox Technologies Confidential Rev 1 2 Troubleshooting 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed_info head 1 Information ifconfig a ethtool lt interface gt ethtool i lt interface of Mellanox port num gt ibdev2netdev Card Detection Ispci grep i Mellanox Download and install MFT http www mellanox com content pages php pg management tools amp menu section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d mst device q ibstat Ports Information ibe Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader var log messages Collect Log File dmesg gt system logF 38 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for Rev 1 2 6 3 Windows From the Windows desktop choose the Start menu and run msinfo32 Environment To export system information to a text file choose the Export option Information from the File menu Assign a file name and save Download and install MFT http www mellanox com content pages php pg management tools amp menu section 34
51. the license agreement and click Next License Agreement Please read the Following license agreement carefully Copyright 2005 2013 Mellanox Technologies 11 rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer gt Tim el mt ae 1555 4 50 in ls tha loro 8 I accept the terms in the license agreement O I do not accept the terms in the license agreement InstallShield Step 4 Select the target folder for the installation Destination Folder Click Next to install to this Folder or click Change to install to a different 1 Install MLNX VPI to C Program Files Mellanox MLNX_ PT InstallShield Step 5 firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card in this case the firmware will not be updated 24 Mellanox Technologies Mellanox Technologies Confidential ConnectX 3 Pro Ethernet Single and Dual QSFP Port Network Interface Card User Manual for OCP Rev 1 2 Ifthe user has a standard card and the firmware version is older than the one specified in WinOF Installation Guide 4 40 the firmware will be updated accordingly However if the user has both OEM card a
52. ubleshooting General GUID of ConnectX 3 Please use the GUID value returned by the fabric driver utilities Pro Ethernet adapter cards not Oxfffff See Appendix A 1 Retrieving Card GUID on page 45 Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Ensure that adapter driver s is loaded Try another port on the switch Link indicator light is Make sure the cable is securely attached off Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but Check that the latest driver is loaded with no Check that both the adapter and its link are set to the same speed communication and duplex settings establi

40GbE User Manual for OCP

Contents

Download Pdf Manuals

Related Search

Related Contents