Home

User Manual for PCIeGen3_x8

1. EEEE EEEE HEHEHEHE HEHEHE HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE 118218112111 AE AER E EEEE HEHE HEHEHEHE HE HEE HEETE ETETE HEEE HH HHHHHHHHHHHHHHHHHHHHHHHH The FCA Manager will not be started automatically To start FCA Manager now type etc init d fca managerd start There should be single process of FCA Manager running per fabric To start FCA Manager automatically after boot type etc init d fca managerd install service 26 Mellanox Technologies J Single and Dual Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Check opt mellanox fca share doc fca README txt for quick start instructions Preparing 1
2. mstflint Preparing FEFE TEHETE AE AEH EHH HH AE EH HE FE HEHEHE TE HEHE TE TE HEE TE E HEE HEHE E H mft Preparing FEFE TEHETE AE AH EHH HH AEH EE EH TE HEHE TETE HHE TE E HEE HE H H srptools srpd iso Zeem on doit Sion e imi Preparing rds tools Preparing rds devel Preparing ibutils2 Preparing ibutils
3. kernel mft mlnx kmp xen Installing knem mlnx RPM Preparing knem mlnx Installing knem mlnx kmp default 1 1 90mlnx2 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE knem mlnx kmp default THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHE Installing knem mlnx kmp trace 1 1 90mlnx2 RPM Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE knem mlnx kmp trace THHHRHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installing knem mlnx kmp xen 1 1 90mlnx2 RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE knem mlnx kmp xen THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Installing ummunotify mlnx RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE ummunotify mlnx THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Installing ummunotify mlnx kmp default 1 0 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE ummunotify mlnx kmp default Mellanox Technologies 23 Preparing mpi selector Install
4. 10 Product OVervie wn eub ceto Beine 10 1 2 Features and Benefits 11 1 3 Operating Systems and Distributions 12 1 4 Connectivity supere e eae equa gala WIN Rs a RAN als NER UR A RA ES 12 Chapter 2 Interfaces 13 2 1 InfiniBand Interface necio bete buis eer mr E E se 13 22 PCI Express Interface 13 23 LED Assignment ve eoe ee tere y AE eek ER en 14 2 4 I2C compatible Interface 14 Chapter 3 Hardware Installation 15 3 1 System Requirements 15 3 Tcl y an ert amia nt RU a I Ta EU Re 15 3 1 2 Operating Systems Distributions 15 3 13 Software Stacks ee EE ees toe ERES Ee qa 15 3 2 Safety Precautions 2 15 3 3 Pre installation Checklist 15 3 4 Bracket Installation 15 3 4 1 Removing the Existing Bracket 16 342 Installing the New Bracket 16 3 5 Card Installation Instructions
5. EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEE HHHHEHHHHEHHHHHHHHHHHHHHHH HEE AEE AEE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EHE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE EE AER E SEE HEE AEE EE aE EE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE HEEE EEH HHHHHHHHHHHHHHHHHHHHHHHH SEE EE AEE AE AE a EEEE HEHEHEHEHE HE HE HEE HEETE ETE EE HE EEEE HH HHEHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HE EEEE HH HHHHEHHHHHHHHHHHHHHHHHHHH SEE EE AEE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE a EEEE HHHHEHHHHHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHE HE HEE HEETE a HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE
6. EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEH HHHHHHHHHHHHHHHHHHHHHHHH HERE AEE AE aE EE EEEE EEEE HEHEHEHEHE HEE HEETE ETE TE E HE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH SEE AEE EEE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEE HHHHEHHHHEHHHHHHHHHHHHHHHH SEE EE AEE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE a EEEE HHHHEHHHHHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHE HE HEE HEETE a HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HE HEE HEETE a HEHEHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE EE 11 AER E EEEE HEHE HEHEHEHE HEHEHE HEETE ETETE E HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHEHE HEE 119218211121 111 IEA TENE EEEE HEHE HEHEHEHE HEHEHE 18211
7. EEEE HEHE HE HEHEHE HEETE ETE EE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE EE EEEE HEHEHEHEHE HE HE HEE HEETE ETETE HEHEHHE HHHHEHHHHHHHHHHHHHHHHHHHH Preparing libibmad static Preparing ibsim Preparing ibacm Preparing librdmacm Preparing librdmacm Preparing librdmacm utils Preparing librdmacm devel Preparing librdmacm devel Preparing opensm libs Preparing opensm libs Preparing opensm opensmd Preparing opensm devel Preparing opensm devel Preparing opensm static Preparing opensm static Preparing infiniband diags Preparing INFO updating IMPORTANT NOTE The FCA Manager and FCA MPI Runtime library are installed in opt mellanox fca directory Rev 1 2 Driver Installation EEEE HEHE HEHEHEHE HEHEHE HEETE ETE E HEEE HHHHHHHHHHHHHHHHHHHHHHHH SEE EE EE AE aE EE EEEE HEHEHEHEHE HE HE HEE HEETE 191821118111 SEE EE AEE HE aE EE EEEE EEEE HEHEHEHE HE HEE HEETE ETETE HEHE HHHHHHHHHHHHHHHHHHHHHHHH SEE EE EE EEE
8. EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HEEE EEH HHEHHHHHHHHHHHHHHHHHHHHHH SEE AE aE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE E EEEE HE HHEHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE A a IE AE EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HE EEEH HHHHHHHHHHHHHHHHHHHHHHHH HEE EE EE EE EE a EEEE HEHEHEHEHE HE HEHEHE HEETE ETE EE HEEE EHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HEHEHE EE E SEE EE EE EAE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EHE EEEE HHEHHEHHHHHHHHHHHHHHHHHHHH sur eri Asmi Segue Hower Geo EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE 192182111211 AE AE IE AER E SEE EE AEE HE AE a
9. 17 3 6 Modules sel a a a ne bone RERO I e N 17 3 6 1 Cable Installation 17 3 7 Identify the Card in Your System 18 3 7 1 18 Chapter 4 Driver 19 4 1 Edd ETT AT 19 4 1 1 Hardware and Software Requirements 19 412 Downloading Mellanox 19 413 Installing Mellanox OFED 19 Mellanox Technologies 3 Rev 1 2 4 1 3 1 Pre installation Notes 20 4 1 3 2 Installation Script 21 4 1 3 3 mlnxofedinstall Return Codes 22 4 1 3 4 Installation Procedure 22 4 1 4 Installation Results 30 4 1 5 Post installation Notes 32 4 1 6 Updating Firmware After Installation 32 Chapter 5 Firmware 34 5 1 Updating Adapter Card Firmware 34 Chapter 6 Troubleshooting 36 6 1 Generali uh lp
10. Preparing 1 Preparing Preparing Preparing dapl devel static FEFE TEHETE AE AEH EHH HHH EH EE HH TE HEHEHE TE HHE TE E HEE HHE H H Preparing FER AER AE AH EHH HH AEH EE EH HE dapl devel static Preparing FERETE HEETE A AH EHH HH AEH EH EHH TE HEHEHE TE HHE TE E HEE HHE H dapl utils Preparing perftest Preparing
11. Connector Pin Number Signal Name 1 SDA 2 SCL 3 GND 4 NC 5 NC 44 Mellanox Technologies Single and Dual QSFP Port PCI Express Gen3 8 Adapter Card User Manual Rev 1 2 Figure 9 PCle Connector Pinout PRSNT1 12V 1 12V 5 12V 2 12V 4 12V 3 GND 28 GND 29 TCK SMCLK TDI SMDAT TDO GND 11 TMS 3 3V 3 3 3V 1 TRST 3 3V 2 3 3Vaux PERST WAKES OBFF GND 22 RSVD 5 REFCLK GND 4 REFCLK 0 GND 21 0 PERpO GND 7 PERnO B17 NC PRSNT2 1 GND 19 GND 9 RSVD 4 PETp1 GND 18 PETn1 PERp1 GND 32 PERn1 GND 13 GND 15 PETp2 GND 14 PETn2 PERp2 GND 16 PERn2 GND 17 GND 24 PETp3 GND 10 PETn3 PERp3 GND 20 PERn3 RSVD 2 GND 8 B31 NC PRSNT2 1 RSVD 3 GND 23 RSVD 1 PETp4 GND 6 PETn4 PERp4 GND 26 PERn4 GND 27 GND 5 5 GND 3 PETn5 PERpS GND 30 PERnS GND 31 GND 2 PETp6 GND 1 PETn6 PERp6 GND 34 PERn6 GND 35 GND 25 PETp7 GND 12 PETn7 PERp7 GND 37 PERn7 PRSNT2 3 GND 33 GND 36 Mellanox Technologies 45 Rev 1 2 Appendix Finding the GUID Serial Number the Adapter Card Each Mellanox adapter card has a label on the print side that shows the card serial number and the card GUID for the InfiniBand protocol Figure 10 Board Labels Model No CB192A Connect IB FDR InfiniBand Made Israel If PIN MCB192A FCAT P N c S N MT0806X01504 GUID 00 02 C9 03 00 27 05 10 Ce Port 1 uses the GUID described on the la
12. umad dev rw Grant non root users read write permission for umad devices instead of default enable affinity Run mlnx affinity script upon boot disable affinity Disable mlnx affinity script Default Mellanox Technologies 21 Rev 1 2 Driver Installation enable sriov Burn SR IOV enabled firmware add kernel support Add kernel support Run mlnx add kernel support sh skip distro check Do not check MLNX OFED vs Distro matching total vfs lt 0 63 gt Maximum number of Virtual Functions in SR IOV mode Default 16 Implies enable sriov hugepages overcommit Set 80 of MAX MEMORY as overcommit for a huge page allocation Per priority bit mask uint Default 0 g Set quiet no messages will be printed with fabric collector Install fabric collector package 4 1 3 3 minxofedinstall Return Codes Table 6 lists mInxofedinstall script return codes and their meanings Table 6 minxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start mst driver 4 1 3 4
13. 24 Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 libcxgb4 devel Preparing libcxgb4 devel Preparing libnes Preparing libnes Preparing libnes devel static Preparing libnes devel static Preparing libipathverbs Preparing libipathverbs Preparing libipathverbs devel Preparing libipathverbs devel Preparing libibem Preparing libibem Preparing libibcm devel Preparing libibcm devel Preparing libibumad Preparing libibumad Preparing libibumad devel Preparing libibumad devel Preparing libibumad static Preparing libibumad static Preparing libibmad Preparing libibmad Preparing libibmad devel Preparing libibmad devel Preparing libibmad static Mellanox Technologies 25 EEEE HEHE HEHEHEHE HEHE E EE HE EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETETE HEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHEHE HEE EE AE AER E EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEH HHHHHHHHEHHHHHHHHHHHHHHHH SEE EE EE AE EE
14. Tool MFT Once installed run mst start mst status flint d lt mst device q Ibstat Ports Information iby ei Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF Mellanox Technologies 37 Rev 1 2 Specifications 7 Specifications 7 1 MCB191A FCAT Specifications Table 7 MCB191A FCAT Specifications Table Size 2 7lin x5 6 in 68 9mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X AX SDR 2 5Gb s per lane DDR SGb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s lane port Protocol Support Data Rate Up to 56Gb s FDR InfiniBand PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 9 68W Power and Max Power Passive Cables 10 67W Environmental Temperature 0 C to 55 C Air Flow 300LFM Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 Regulatory IEC 60068 2 64 29 32 RoHS RoHS R6 a Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 2 MCB192A FCAT Specifications Table 8 MCB192A FCAT Specifications Table Size 2 7lin x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP Inf
15. m a eh etant ewe petes 36 6 22 LUX Ra kmde YE eee 37 Chapter 7 Specifications 3S 7 1 MCBI91A FCAT Specifications 38 7 2 MCBI92A FCAT Specifications 38 1 3 EBD Operation er i er Posse ere ha dde sine aaa 39 7 4 Board Mechanical Drawing and Dimensions 40 7 5 Bracket Mechanical Drawings 41 4 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 List of Tables Table 1 Revision History Table 7 Table 2 Documents List 8 Table 3 Single and Dual port Connect IB Adapter Cards 10 Table 4 Features and Benefits 11 Table 5 Software and Hardware Requirements 19 Table 6 minxofedinstall Return Codes 22 Table 7 MCB191A FCAT Specifications Table 38 Table 8 192 Specifications Table 38 Table 9 LED Assignment 39 Table 10 I2C compatible Connector Pinout 44 Mellanox Technologies 5 Rev 1 2 List of Figures Figure 1 192 Card 1 ea 13 Figure 2 Place the Bracket on the Card Example 16 Figure 3 Support Download Assistant 35 Figure 4 Mechanical Drawing of t
16. 3 2 Installation Script The usage of the installation script is described below You will use it during the installation pro cedure described in Section 4 1 3 4 Installation Procedure on page 22 Usage mnt mlnxofedinstall OPTIONS Options c config packages config file Example of the configuration file can be found under docs n net network config file Example of the network configuration file can be found under docs k kernel version kernel version Use provided kernel version instead of uname r p print available Print availablepackages for the current plat form and create a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update fw update only Update firmware Skip driver installation force fw update Force firmware update force Force installation without querying the user all hpc basic msm Install all hpc basic or Mellanox Subnet manager packages correspondingly vma vma vpi Install packages required by VMA to support VPI vma ib Install packages required by VMA to work over InfiniBand with vma Set configuration for VMA use to be used with any installation parameter guest Install packages reguired by guest os hypervisor Install packages reguired by hypervisor os v vv vvv Set verbosity level
17. Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR 1020110019 PSID OPN Identifying Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Select an iten n a n 3 If a newer firmware version exists for your adapter card update the firmware as follows Download the firmware image zip file from the Download Center see Step 2a above b Unzip the firmware image c Burn the firmware image Please note that the following are examples Enter gt flint d dev mst mt4099 pci cr0 i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully flint d dev mst mt4099 pci cr0 q Image type ConnectX FW Version 2 9 4100 Device ID 4099 Mellanox Technologies 35 Rev 1 2 Troubleshooting 6 6 1 36 Mellanox Technologies Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS to reboot the server The adapter no longer works Reseat the adapter in its
18. Preparing cc_mgr FEFE TEHETE A AH EHH HE EHHE FE HEHEHE TE HH TE HEETE TEHETE E HEE HE H H Preparing dump pr FER ERE AE AEH EHH HHH AEH HEEE TE HH TE HEHE TE TE HHE TE E HE HHE H Preparing ar_mgr Preparing ibdump Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE infiniband diags compat THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Preparing THHHHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE qperf THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHRHHHHHHE mxm THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHRHHE Preparing THHHRHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Mellanox Technologies 27 Rev 1 2 Driver Installation openmpi EEEE HEHE HEHEHEHE HEHEHE HEETE ETE E HEEE HHHHHHHHHHHHHHHHHHHHHHHH Preparing EEEE HEHE HEHEHEHE HE HEE HEETE EEE HEEE HHHHHHHHHHHHHHHHH
19. and relevant documentation Intended Audience This manual is intended for the installer and user of these cards and assumes basic familiarity with InfiniBand network and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User User Manual describing the set of MFT firmware management Manual tools for a single node Document no 2204UG See http www mellanox com gt Support gt Download Firm ware Tools Mellanox OFED for Linux User Manual describing OFED features performance InfiniBand User Manual diagnostic tools content and configuration See http www mella Document no 2877 nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED IBTA Specification Release 1 2 1 InfiniBand Architecture Specification PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in Mega Bytes The use of Mb or Mbits small b indicates size in mega bits IB is used in this document to mean InfiniBand In this document PCIe is used to mean PCI Express Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us throueh the following methods URL http www mella
20. con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales Mellanox Technologies 53 Rev 1 2 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de 105 que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 54 Mellanox Technologies
21. is A0 dev mst mt25418 pci cr0 gt Bei chrace 860556 bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is 0 dev mst mt25418 pci msix0 POE higado suele bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix pci cr0 In the example listed above this will be dev mst mt25418 pci cro does not apply Step 3 Burn firmware 1 Burning a firmware binary image using mst lint that is already installed on your machine Please refer to MSTFLINT README txt under docs 2 Burning a firmware image from a mlx file using the m1xburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408 fw 25408 ryyel mlx ls mnt firmware fw 4113 10 10 1000 BUILD ID fw ConnectIB mlx MCB191A FCA Al ini MCB193A FBA Al ini MCB194A FCA Al ini syndrome list log exp rom fw strings db MCB192A FCA Al ini MCB193A FCA Al ini mlnx ofed latest txt reg l vrt 212 tmp mlxburn dev dev mst mt4113 pciconf0 fw mnt firmware fw 4113 10 10 1000 fw ConnectIB mlx I Querying device I Using auto detected configuration file mnt firmware fw 4113 10 10 1000 MCB192A FCA Al ini PSID MT 1240110019 I Generating im
22. slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Ensure that adapter driver s is loaded Try another port on the switch Make sure the cable is securely attached Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but with no communication established Check that the latest driver 15 loaded Check that both the adapter and its link are set to the same speed and duplex settings Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed_info head 1 Information ifconfig a ethtool lt interface gt ethtool i interface of Mellanox port num ibdev2netdev Card Detection Ispci grep i Mellanox Download and install MFT http www mellanox com content pages php pg management tools amp menu section 34 Refer to the User Manual for installation instructions Mellanox Firmware
23. 0 InfiniBand Error Counter Check 0 PASS ee PASS CATHO aaa 00 02 769203200 00740 DONE After the installer completes information about the OFED installation such as prefix kernel version and installation parameters can be retrieved by running the com ae mand etc infiniband info 4 1 4 Installation Results Software The OFED and MFT packages are installed under the usr directory The kernel modules are installed under InfiniBand subsystem 1ib modules uname r updates kernel drivers infiniband mixd core driver lib modules lt kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx4 mlx4 core ko mixd ib driver lib modules lt kernel version extra mlnx ofa kernel drivers infiniband hw mlx4 mlx4 ib ko mlx5 core driver lib modules lt kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx5 core mlx5 core ko mix5 ib driver lib modules lt kernel version extra mlnx ofa kernel drivers infiniband hw mlx5 mlx5 ib ko IPoIB 1ib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko 15 1ib modules uname r updates kernel drivers infiniband ulp iser ib iser ko 30 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x
24. 12111 IE AER E SEE EE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEE HHHHEHHHHHHHHHHHHHHHHHHHH SEE AEE AE AE EEEE HEHE HEHEHEHE HE HEE HEETE Ea H HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE a 11 AER E EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HE HEHEHE HEETE ETE EE EEEE HHEHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE EE E HE EEEH HHEHHEHHHHHHHHHHHHHHHHHHHH SEE AEE AE AE EEEE HEHE HE HEHEHE HEETE ETE EE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE EE
25. 8 Adapter Card User Manual Rev 1 2 elPolB lib modules uname r updates kernel drivers net eipoib eth ipoib ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko lib modules uname r updates kernel net rds rds tcp ko Kernel s modules location may vary depending on the kernel s configuration For example lib modules uname r extra kernel drivers net ethernet mellanox mlx4 ae mlx4 core The package kernel ib devel include files are placed under usr src ofa kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel Theraw package un backported source files are placed under usr src ofa kernel lt ver gt The script openibd is installed under etc init d This script can be used to load and unload the software stack The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED instal lation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file is used at driver st
26. AA Mellanox Connect Accelerate Outperform Connect IB Single and Dual Port Express Gen3 x8 Adapter Card User Manual P N MCB191A FCAT MCB192A FCAT Rev 1 2 www mellanox com Rev 1 2 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S A
27. Band Interface PCI Express Interface LED Assignment 2C compatible Interface Figure 1 MCB192A FCAT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables 2 1 InfiniBand Interface The network ports of the Connect IB adapter cards are compliant with the InfiniBand Architecture Specification Release 1 2 1 InfiniBand traffic is transmitted through the cards QSFP connectors 2 2 PCI Express Interface The Connect IB adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists the PCIe interface features PCI Express 2 0 or 3 0 compliant Auto negotiates to x8 x4 or x1 Mellanox Technologies 13 Rev 1 2 Interfaces Support for MSI X mechanisms 23 LED Assignment There are two I O LEDs per port in dual port designs and one bicolor LED located on the VO panel See Section 7 3 LED Operation on page 39 for different LED functions 2 4 12C compatible Interface A three pin header on the adapter cards is provided as the interface See Figure 4 Mechanical Drawing of the Dual port MCB192A FCAT Adapter Card on page 40 for the location on the board 14 Mellanox Technologies Connect IB Single and Dual QSFP Port
28. ETE ETETE E HE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH libvma Changing locked memory to unlimited etc security limits conf Please log out from the shell and login again in order to update this change Read more about this topic in the VMA s User Manual README txt is installed at usr share doc packages libvma 6 5 3 0 README txt Please refer to VMA journal for the latest changes usr share doc packages libvma 6 5 3 0 journal txt EEEE HEHE HEHEHEHE HE HEE HEETE HEEE HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHE E 118211182111 IE AER E EEHEHE HEHEHEHEHE HEHEHE HEETE ETETE EEEE EEEE HEHEHEHE HEHEHE a EE SEE HEE EE HE AE Preparing mlnxofed docs Preparing mpitests mvapich2 1 9 Preparing mpitests openmpi 165 Preparing mpitests openmpi 1 7 4 Device 0a 00 0 0a 00 0 InfiniBand Mellanox Technologies Device 1011 Link Width is not 8x PCI Link Speed 2 5Gb s
29. HEHEHE HE HE HEE HEETE a HEHEHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE EE 11 AER E EEEE HEHE HEHEHEHE HEHEHE HEETE ETETE E HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHEHE HEE 119218211121 111 IEA TENE EEEE HEHE HEHEHEHE HEHEHE 1821112111 IE AER E SEE EE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEEE HHHHEHHHHHHHHHHHHHHHHHHHH SEE AEE AE AE EEEE HEHE HEHEHEHE HE HEE HEETE Ea H HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE a 11 AER E EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HE HEHEHE HEETE ETE EE EEEE HHEHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE EE E HE EEEH HHEHHEHHHHHHHHHHHHHHHHHHHH SEE AEE AE AE
30. HHHHHHHHH openmpi Preparing EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EHHH HHHHHHHHHHHHHHHHHHHHHHHH bupc EEEE EEEE HEHEHEHE HE HEE HEETE ETETE HEHE HHHHHHHHHHHHHHHHHHHHHHHH Preparing SEE EE EE EEE infinipath psm Preparing infinipath psm devel EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HEEE EEH HHEHHHHHHHHHHHHHHHHHHHHHH SEE AE aE AE Preparing mvapich2 EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE E EEEE HE HHEHHEHHHHHHHHHHHHHHHHHHHH Preparing openshmem EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH Preparing EEEE HEHE HEHEHEHE HE HEE A a IE AE hcoll Preparing EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HE EEEH HHHHHHHHHHHHHHHHHHHHHHHH libibprof HERE AEE AE aE Ea EEEE HEHE HEHEHEHE HEHEHE HE
31. Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount ISO image on your machine host1 mount o ro loop MLNX OFED LINUX lt ver gt lt 0S label gt lt CPU arch gt iso mnt Step3 Run the installation script mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX root sw1014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 641 root swl014 MLNX OFED LINUX 2 0 2 0 mo0 rhel6 3 x86 641 mlnxofedinstall 22 Mellanox Technologies J Single Dual QSFP Port PCI Express Gen3 8 Adapter Card User Manual Rev 1 2 This program will install the MLNX OFED LINUX package your machine Note that all other Mellanox OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX Starting MLNX OFED LINUX 2 0 2 0 0 installation Installing mlnx ofa kernel RPM Preparing minx ofa kernel Installing mlnx ofa kernel kmp default 2 1 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHRHHHUHHHHHRHHHRHHHRHHRHH
32. Installation finished successfully The firmware version on dev mst mt4113 pciconf0 10 10 1000 is up to date Note To force firmware update use force fw update flag 28 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 In case your machine has the latest firmware firmware update will occur installation script will print at the end of installation a message similar to the following firmware version dev mst mt26448 pci cr0 2 9 1000 is up Note force firmware update use force fw update flag The firmware version on dev mst mt4099_pci_cr0 2 11 500 is up cordate Note To force firmware update use force fw update flag In case your machine has unsupported network adapter device no firmware update gt 4 will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file Step 4 case the installation script performed firmware updates to your network adapter hardware it will ask you to reboot your machine Step 5 The script adds the following lines to etc security limits cont for userspace com ponents such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a u
33. Memory PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 32Mb SPI Flash device Winbond W25Q32BVSSIG EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb Greater than 100Gb s over InfiniBand Greater than 130M messages sec lus MPI ping latency CPU offload of transport operations Application offload GPU communication acceleration End to end internal data protection End to end QoS and congestion control Hardware based virtualization RoHS R6 compliant Benefits World class cluster network and storage performance Guaranteed bandwidth and low latency services consolidation Virtualization acceleration Power efficient Scales to tens of thousands of nodes Mellanox Technologies 11 Rev 1 2 Introduction 1 3 Operating Systems Distributions RHEL other Linux distributions Microsoft Windows Server 2008 CCS 2003 HPC Server 2008 VMware ESX 5 1 OpenFabrics Enterprise Distribution OFED 1 4 Connectivity Interoperable with InfiniBand switches Passive copper cable with ESD protection Powered connectors for optical and active cable support 12 Mellanox Technologies J Single and Dual Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 2 Interfaces Each adapter card includes the following interfaces Infini
34. ND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BrideeX ConnectX Connect IB CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricITTM Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestXTM Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 4123 Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Table of Contents Table of Contents 4 2 220800 etr e e e 426 N Bist of Tables ei e au a ub ere aere ree List ob Figures na e Revision History saccos Ves RARE lie slim erek di ANY About this Manual 5545 111094 A T a ER e du Supports 12 4 24 He Bee A PA rd 8 Firmware and Software 9 Chapter I Introduction
35. PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 3 Hardware Installation 3 1 System Requirements 3 1 1 Hardware A system with a PCI Express x8 slot is required for installing the card 3 1 2 Operating Systems Distributions Please refer to Section 1 3 Operating Systems and Distributions on page 12 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX_OFED for Linux See Chapter 4 Driver Installation 322 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components 1 Remove any metallic objects from your hands and wrists 2 Make sure to use only insulated tools 3 Verify that the system is powered off and is unplugged 4 It is strongly recommended to use an ESD strap or other antistatic devices 3 3 Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord p er Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged 3 4 Bracket Installation The card is usually shipped with a tall bracket installed If this
36. RHUHRHNE mlnx ofa kernel kmp default Installing mlnx ofa kernel kmp trace 2 1 RPM Preparing mlnx ofa_kernel kmp trace Installing mlnx ofa kernel kmp xen 2 1 RPM Preparing mlnx ofa kernel kmp xen Installing mlnx ofa kernel devel RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE mlnx ofa kernel devel THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installing kernel mft mlnx kmp default any RPM Preparing THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHE kernel mft mlnx kmp default JHHHHHHHHHHHHHHHHHBHHHHBHHHHBHHHHHHHHHHHHHHHHHHHHHE Installing kernel mft mlnx kmp trace any RPM Preparing kernel mft mlnx kmp trace Installing kernel mft mlnx kmp xen RPM Preparing
37. age Current FW version on flash 10 10 1000 New FW version 10 10 1000 Note The new FW version is not newer than the current FW version on flash Do you want to continue y n n y Burning FS3 FW image without signatures OK Restoring signature OK I Image burn completed successfully Mellanox Technologies 33 Rev 1 2 Firmware 5 5 1 34 Mellanox Technologies Firmware Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com gt Support gt Download Firmware Check that the firmware on your card 15 the latest found on the Mellanox site if not update to the latest version found on the Mel lanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm ware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Support gt Down load Firmware Tools A firmware binaries table lists a binary file per adapter card The file name of each such binary is composed by combining the firmware name the firmware release version and the card part num ber Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card The following steps descri
38. apter covers the following topics 1 1 Product Overview The following tables provide the ordering part number port speed number of ports and PCI Express speed Each adapter comes with two bracket heights short and tall Table 3 Single and Dual port Connect IB Adapter Cards Ordering Part Number OPN Data Transmission Rate Network Connector Types PCI Express PCIe SerDes Speed RoHS Adapter IC Part Number Device ID decimal MCB191A FCAT single port card 192 dual port card InfiniBand FDR 56 Gb s single or dual port QSFP PCIe 3 0 x8 8GT s R6 MT27608A0 FCCR FI 4114 for Physical Function 4113 for Virtual Function 10 Mellanox Technologies Single Dual QSFP Port PCI Express Gen3 8 Adapter Card User Manual Rev 1 2 1 2 Features and Benefits Table 4 Features Benefits Features InfiniBand Architecture Specification v1 2 1 com pliant VPI enabled adapters facilitate any standard networking clustering storage and management protocol to seamlessly operate over any converged network with the same software infrastructure PCI Express 3 0 1 1 and Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an 8 edge connector up to 2 0 compatible 8GT s InfiniBand FDR A pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s
39. art restart etc init d openibd start Thefile 90 ib rules is installed under etc udev rules d If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc If IPoIB configuration files are included ifcfg ib n files will be installed under etc sysconfig network scripts ona RedHat machine The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled 1 You run the installation script in default mode that is without the option without fw update Mellanox Technologies 31 Rev 1 2 Driver Installation 2 The firmware version of adapter device is older than firmware version included with the Mellanox OFED ISO image If an adapter s Flash was originially programmed with an Expansion ROM image automatic firmware update will also burn an Expansion ROM image Ro In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message 1 Querying device E Can t auto detect fw configuration file 4 15 Post installation Notes Most
40. be how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available Note The following steps are an example and do not correspond to this device 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start Get the Mellanox mst device name using the command mst status d Getthe PSID firmware identification and programmed firmware version using the command gt flint d dev mst mt4099 pci cr0 g Image type ConnectX FW Version 2 9 4000 Device ID 4099 Chip Revision 0 Description Node Port1 Port2 Sys image GUIDs 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT 1020110019 VSD PSID MT 1020110019 2 Compare the programmed firmware version with the latest available a Goto Mellanox s web site http www mellanox com supportdownloader See Figure 3 b Enter your card PSID to display the latest firmware Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Figure 3 Support Download Assistant Mellanox TECHNOLOGIES
41. bel To obtain the GUID for Port 2 add 1 to that of Port 1 46 Mellanox Technologies Single and Dual Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Appendix Safety Warnings Below is alist of safety warnings in English For safety warnings in other languages please refer to the appendices in this user manual 1 Installation Instructions 2 Over temperature A 3 During Lightning Electrical Hazard m 4 Copper Cable Connecting Disconnecting nenn 5 Equipment Installation 6 Equipment Disposal 7 Local National Electrical Codes 4 2 as Mellanox Technologies 47 Rev 1 2 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 200 48 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temperature excessive Ce mat riel doit
42. ber and to Signal Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4 GND GND 5 TXN_4 Tx4n 6 TXP 4 4 7 GND GND 8 ModSelL PortO ModSelL 9 ResetL_PortO ResetL 42 Mellanox Technologies J Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Table 10 Connector Pin Number Name to Signal Connector Pin Number Connector Pin Name Port A Signal Name 10 VccRx 11 SCL SCL 12 SDA SDA 13 GND GND 14 RXP 3 Rx3p 15 RXN_3 Rx3n 16 GND GND 17 _1 1 18 RXN 1 RxIn 19 GND GND 20 GND GND 21 2 Rx2n 22 RXP 2 2 23 GND GND 24 RXN_4 Rx4n 25 RXP 4 Rx4p 26 GND GND 27 ModPrsl PortO PrsL 28 29 30 Vccl 31 LPMode_Port0 LPMode 32 GND GND 33 TXP_3 Tx3p 34 TXN_3 Tx3n 35 GND GND 36 _1 1 37 TXN 1 TxIn 38 GND GND Mellanox Technologies 43 Rev 1 2 A 2 PCI Express 8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8standard pinout according to the PCI Express 3 0 specification I2C compatible Connector Pinout Figure 8 Compatible Connector Plug and Pinout Table 11 I2C compatible Connector
43. ckage on a single host machine with Mel lanox Connect IB adapter hardware installed 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Reguiremenis Reguirements Description Platforms For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Reguired Disk Space for 1GB Installation Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to Mellanox OFED Release Notes file Installer Privileges The installation reguires administrator privileges on the target machine 4 1 2 Downloading Mellanox OFED Step 1 Verify that the system has a installed by ensuring that you can see Connect IB entries in the display The following example shows a system with an installed Mellanox HCA host1 lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27600 Connect IB Step2 Download the ISO image to your host The image s name has the format MLNX OFED LINUX lt ver gt lt OS label gt lt CPU arch gt iso You can download it from http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers Step 3 Use md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result t
44. d User Manual Rev 1 2 3 5 Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot lesser width adapter can seated into a greater width slot x4 in 8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter a connector edge with the PCI Express connector slot 4 Applying even pressure at both corners of the card insert the adapter card into the slot until it 15 firmly seated When the adapter is properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis Do not use excessive force when seating the card as this may damage the system or the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user manuals Mellanox approved cables pdf 3 6 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated a Support th
45. e weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack b Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card c Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card d Make sure that the connector locks in place Mellanox Technologies 17 Rev 1 2 Hardware Installation 3 After inserting cable into port Yellow LED indicator will light when physical nection is established that is when the unit is powered and a cable is plugged into the port with the other end of the connector plugged into a functioning port 4 After plugging in a cable lock the connector using latching mechanism particular to the cable vendor When a logical connection is made the Green LED will come on When data is being transferred the green led will blink 4 When installing cables make sure that the latches engage Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Cable lengths which allow for routing horizontally around to the side of
46. ed under the docs directory Usage mlnx add kernel support sh m mlnx path to MLNX OFED gt directory gt make iso make tgz gt gt make iso Create MLNX OFED ISO image gt make tgz Create MLNX OFED tarball Default t tmpdir local work dir gt kmp k kernel kernel version s kernel sources path to the kernel sources v verbose n name y ves Example The following command will create a MLNX OFED LINUX ISO image for RedHat 6 3 under the tmp directory MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 mlnx add kernel support sh m lt path gt MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 make tgz Note This program will create MLNX OFED LINUX TGZ for rhel6 3 under tmp directory 1 The firmware will not be updated if you run the install script with the without fw update option 20 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 11 Mellanox OFED or Distribution IB packages will removed Do you want to continue y N y See log file tmp mlnx ofed iso 1380 10g Building OFED RPMs Please wait Removing OFED RPMs Created tmp MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 tgz Install newly created MLNX OFED package cd tmp tar xzf MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 tgz MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 mlnxofedinstall 4 1
47. form factor is suitable for your reguirements you can skip the remainder of this section and move to Section 3 5 Card Installa Instructions on page 17 If you need to replace it with the short bracket that is included in the shipping box please follow the instructions in this section Note Due to risk of damaging EMI gasket it is not recommended to replace the bracket more than three times To replace the bracket you will need the following parts Mellanox Technologies 15 Rev 1 2 Hardware Installation The new bracket of proper height 2 screws saved from the removal of the bracket The 2 fiber washers saved from the removal of the bracket 3 4 1 Removing the Existing Bracket 1 Remove the two screws holding the bracket in place The bracket comes loose from the card Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers 3 4 2 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up See Figure 2 Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal procedure above Figure 2 Place the Bracket on the Card Example 16 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Car
48. green indicates a valid logical data activity link without data transfer Logical data activity link A blinking green indicates a valid logical link with data transfer If only the yellow LED is lit and the green LED is off then the logical link has not been established Mellanox Technologies 39 Rev 1 2 Specifications 7 4 Board Mechanical Drawing and Dimensions All dimensions are in millimeters Ax All the mechanical tolerances are 0 1mm Figure 4 Mechanical Drawing of the Dual port MCB192A FCAT Adapter 142 24 ii AO e ee L 56 15 Benmi Em m 68 9 40 Mellanox Technologies Single and Dual Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 75 Bracket Mechanical Drawings Figure 5 Dual port Bracket Port 1 Gasket LEDs Port 2 Figure 6 Single port Bracket Gasket Bicolor LED Mellanox Technologies 41 Rev 1 2 Appendix A Interface Connectors Pinout QSFP Connector Pinout Figure 7 Connector and Cage Views T 18 35 4 View into Rear Uo E Td POW X129A 129A poWwd1 QN9 GNS dixi ULXL QN9 ONS dzxu GNS ui 22292298 8 D O gt 7 2 gt E NM a UM Z 65698 6 R15 55666 p 4435 View into Front of Table 10 Connector Pin Num
49. he Dual port MCB192A FCAT Adapter Card 40 Figure 5 D al poit Bracket eoo BALLA EE Ee ni De a 41 Figure 6 Single port 41 Figure 7 Connector and Cage Views 42 Figure 8 Compatible Connector Plug and Pinout 44 Figure 9 PCleConnectorPinout niot inei 45 Figure 10 Board Labels ceste Se ser Pausen fair ser al balli de 46 6 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Revision History This document was printed on June 2 2014 Table 1 Revision History Table Date Rev Comments Changes June 2013 1 2 Added Section 3 4 Bracket Installation on page 15 December 2013 1 1 Made restructuring changes to the UM Added Chapter 4 Driver Installation on page 19 Added Chapter 5 Firmware on page 34 Added Chapter 6 Troubleshooting on page 36 January 2013 1 0 First Release Mellanox Technologies 7 1 2 About this Manual This User Manual describes Mellanox Technologies Connect IBTM Single and Dual QSFP port PCI Express 8 lanes x8 adapter cards It provides details as to the interfaces of the board speci fications required software and firmware for operating the board
50. ing user level RPMs Preparing ofed scripts Preparing libibverbs Preparing libibverbs Preparing libibverbs devel Preparing libibverbs devel Preparing libibverbs devel static Preparing libibverbs devel static Preparing libibverbs utils Preparing libmlx4 Preparing libmlx4 Preparing libmlx4 devel Preparing libmlx4 devel Preparing libmlx5 Preparing libmlx5 Preparing libmlx5 devel Preparing libmlx5 devel Preparing libcxgb3 Preparing libcxgb3 Preparing libcxgb3 devel Preparing libcxgb3 devel Preparing libcxgb4 Preparing libcxgb4 Preparing Rev 1 2 Driver Installation Installing mpi selector RPM EEEE HEHEHEHEHE HE HE HEE HEETE ETE EE HEEE HH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE EE AE E SEE EE EE AE AE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE TEE EEHEEHE HHHHEHHHHHHHHHHHHHHHHHHHH SEE AEE EE AEE EEEE HEHE AE HEHEHE HEETE ETETE E EEEE HHHHHHHHEHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHE HE HEE EE AE SEE AEE EE AE aE
51. iniBand Copper and optical 38 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Table 8 MCB192A FCAT Specifications Table InfiniBand v1 2 1 Auto Negotiation 1X 2X AX SDR 2 5Gb s per lane DDR SGb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s lane port Protocol Support Data Rate Up to 56Gb s FDR InfiniBand PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 10 73W Power and Max Power Passive Cables 12 32W Environmental Temperature 0 C to 55 C Air Flow 300LFM Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 Regulator ETAT IEC 60068 2 64 29 32 RoHS RoHS R6 a Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 73 LED Operation There are two LEDs per port in dual port designs and one bicolor LED in single port designs green and yellow located on the VO panel See Table 9 for different LED functions Table 9 LED Assignment LED Function Yellow Physical link Constant on indicates a good physical link Blinking indicates a problem with the physical link Ifneither LED is lit then the physical link has not been established When logical link is established yellow LED turns off Green constant
52. n 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden This equipment should be installed in compliance with local and national elec LA trical codes Mellanox Technologies 51 Rev 1 2 8 Strahlenkontak 52 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Appendix F Advertencias de seguridad para la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de Instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la LA m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de LA rayos 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec LA tores y su desconexi n se deben efectuar
53. nox com gt Support E mail support mellanox com Tel 1 408 016 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller 8 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the data base from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Rev 1 2 Introduction 1 Introduction This is the User Guide for Mellanox Technologies Connect IB 56Gb S InfiniBand adapter cards Connect IB provide the highest performing and most scalable interconnect solution for PCI Express Gen2 and Gen3 servers used in High Performance Computing Enterprise Data Centers and Storage environments Parallel processing transactional services clustered data bases and high performance storage applications will achieve significant performance improvements resulting in reduced completion time and lower cost per operation This ch
54. o the value provided on the download page host1 md5sumMLNX OFED LINUX lt ver gt lt 0S label gt iso 4 1 3 Installing Mellanox OFED Mellanox OFED includes an installation script called minxofedinstall which performs the fol lowing Mellanox Technologies 19 Rev 1 2 Driver Installation Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX_OFED_LINUX binary if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automat ically upgrades the firmware 4 1 3 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Pre existing configuration files will be saved with the extension conf rpmsave Au f you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh f your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel _ support sh script locat
55. of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file 4 1 6 Updating Firmware After Installation In case you ran mInxofedinstall script with the without fw update option and now you wish to manually update firmware on your adapter card s you need to perform the following steps E If you need to burn Expansion ROM image please refer to OFED User Manual See Table 2 Documents List page 8 The following steps are also appropriate case you wish to burn newer firmware that gt you have downloaded from Mellanox Technologies Web site gt Support gt Firm ae ware Download Step 1 Start mst host1 mst start Step 2 Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine host1 mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre 12C module is not loaded 32 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 MST devices dev mst mt25418 pciconf0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision
56. pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et 11 ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L elimination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux AN Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Mellanox Technologies 49 Rev 1 2 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes
57. peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 1 1996 A2 2001 50 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 Appendix E Sicherheitshinweise Warnings German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der LA maximal empfohlenen Temperatur von 55 C 1319F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gevitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschlie en Trennen von Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die A Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werde
58. ser space application If desired tune the value unlimited to a specific amount of RAM Step 6 For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM refer to the OFED User Manual See Table 2 Documents List on page 8 Step 7 InfiniBand only Run the hca self test ofed utility to verify whether or not the Infini Band link is up The utility also checks for and displays additional information such as HCA firmware version Kernel architecture Driver version Number of active HCA ports along with their states Node GUID Mellanox Technologies 29 Rev 1 2 Driver Installation Note For more details on hca self test ofed see the file self test readme under docs NumberNoRECASNPetecbedgN ee il ee ee EIS PASS A p 86_64 HOS priver Version L50 ETT T MLNX OFED LINUX 2 1 0 0 4 20131125 1547 OFED 2 1 0 0 4 20131125 1547 3 0 13 0 27 default Host Driver RPM Cheek nnm 9506 PASS elime O CN D MOM M T E v10 10 1000 Firmware Check on CA 0 HCA PASS Bose WILKE nit PASS EET T T 0 Port State of 1 0 InfiniBand port State of Port 2
59. the chassis before bending upward or downward in the rack should be used 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated a Pullon the latch release mechanism thereby unlatching the connector and pull the connector out of the cage b Do not apply torque to the connector when removing it from the adapter card c Remove any cable supports that were used to support the cable s weight Cables especially long copper cables can weigh a substantial amount Make sure that LA the weight of the cable is supported on its own and is not hanging from the adapter card 3 7 Identify the Card in Your System 3 7 4 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt lspci grep i Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27600 Connect IB 18 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x8 Adapter Card User Manual Rev 1 2 4 Driver Installation 4 1 Linux For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via web site at http www mellanox com gt Products Soft ware gt InfiniBand VPI Drivers gt Linux SW Drivers gt Download This chapter describes how to install and test the Mellanox OFED for Linux pa

User Manual for PCIeGen3_x8

Contents

Download Pdf Manuals

Related Search

Related Contents