Sun Dual Port 4x QDR IB Host Channel Adapter PCIe
Contents
1. X Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule Users Guide 7 April 2010 Declaration of Conformity To receive a copy of the latest Declaration of Conformity DoC for the product either contact your local Sun sales representative or create an online request at https www2 sun de dct forms reg us 1607 755 0 jsp xi xii Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide 7 April 2010 Preface This guide provides an overview installation instructions and specifications for the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule The instructions in this guide are designed for system administrators with experience installing network hardware and software Note The Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule is based on Mellanox Technologies MT26428 ConnectX IB adapter HCA device Related Documentation The following table lists the documentation for this product The online documentation is available at http docs sun com app docs prod qdr pcie expmod Application Title Part Number Format Location Unpacking Sun Dual Port 4x QDR IB Host Channel Adapter PCle 820 6541 Printed Shipping kit the product ExpressModule Getting Started Guide PDF Online Installation Sun Dual Port 4x QDR IB Host Channel Adapter PCIe 820 6539 PDF Online and use Ex2. on page 15 or Verify the Installation With Linux on page 20 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 Hot Swapping With the Linux OS Use the following procedure to hot swap the ExpressModule with the Linux OS W To Hot Swap the ExpressModule Using the Linux OS 1 Type the following and verify that the IB status is okay gt ibstat CA mlx4 0 CA type MT26428 Number of ports 2 Firmware version 2 6 0 Hardware version a0 Node GUID 0x0002c903000379a4 System image GUID 0x0002c903000379a7 Port 1 State Active Physical state LinkUp Rate 20 Base lid 26 LMC 0 SM lid 23 Capability mask 0x02510868 Port GUID 0x0002c903000379a5 Port 2 State Down Physical state Polling Rate 10 Base lid 28 LMC 0 SM lid 1 Capability mask 0x02510868 Port GUID 0x0002c903000379a6 2 Load the acpiphp module by typing the following Chapter 2 IB HCA ExpressModule Installation 9 Note Do not use the acpiphp command for hot plug operations in SuSE Linux SLES10SP2 Use the modprobe pciehp pciehp poll mode 1 command instead of the acpiphp command The remaining steps and commands are the same for the SuSE Linux SLES10SP2 hot plug procedure gt modprobe acpiphp 3 If IPoIB is configured type one of the following gt ifconfig ib0 down gt ifconfig ibl down Where n is the instance number of the ExpressModule
3. Rate 40 Base lid 28 LMC O SM lid 1 Capability mask 0x02510868 Port GUID 0x0002c903000379a6 Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide April 2010 CHAPTER 3 InfiniBand Software on the Solaris Operating System and Linux InfiniBand is a network architecture for the large scale interconnection of computing and I O nodes through a high speed switched fabric To operate InfiniBand on a Sun server you need an InfiniBand HCA the ExpressModule and an InfiniBand software stack This chapter provides an overview of installing and using the InfiniBand software stack for the Solaris OS and Linux operating svstem Consult the product notes for vour server for recent information about supported operating svstems firmware and software updates and other issues not covered in the main product documentation This chapter includes m InfiniBand Software on the Solaris Operating System on page 13 m InfiniBand Support Software for Linux on page 17 m Internet Protocol Over InfiniBand on Linux on page 19 m Boot Over InfiniBand on Linux on page 19 m Additional InfiniBand Software for Linux on page 21 InfiniBand Software on the Solaris Operating System InfiniBand software is bundled with the Solaris 10 OS The package containing the device driver for the Sun Dual Port 4x ODR IB Host Channel Adapter is SUNWhermon The driver name is hermon InfiniBand So
4. products or other materials that are available on or through such sites or xiv Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide 7 April 2010 resources Sun will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content goods or services that are available on or through such sites or resources Information URL PCI Express specifications http www pcisig com Mellanox Technologies HCA information http www mellanox com firmware downloads and Linux software downloads OFED information http www openfabrics org InfiniBand Specifications http infinibandta org Documentation Feedback Sun is interested in improving its documentation and welcomes your comments and suggestions Submit comments about this document by clicking the Feedback link at http docs sun com Include the title and part number of your document with your feedback Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide part number 820 6539 13 Preface xv xvi Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide 7 April 2010 CHAPTER 1 Overview This chapter includes m Adapter Hardware Overview on page 1 m Hardware and Software Requirements on page 6 Adapter Hardware Overview The main features of the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe
5. 4 Press the Hot Swap button on the ExpressModule see FIGURE 1 3 Within eight seconds the green LEDs will go out The ExpressModule is now physically ready to be removed from the chassis slot 10 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 5 Type the following gt etc init d openibd status HCA driver loaded Configured IPoIB devices Currentlv active IPoIB devices The following OFED modules are loaded rdma ucm ib sdp rdma cm ib addr ib ipoib mlx4 core mlx4 ib ib mthca ib uverbs ib umad ib ucm ib sa ib cm ib mad ib core The output displavs the driver as loaded but not attached to anv device which indicates the driver is stable 6 Insert the ExpressModule into chassis slot 7 After approximatelv one minute after all the LEDS are out press the Hot Swap button again The LEDs start to blink and both the ExpressModule and the ports come up in nine seconds Chapter 2 IB HCA ExpressModule Installation 11 12 8 Type the following to verify that the InfiniBand ports are up gt ibstat CA mlx4 0 CA type MT26428 Number of ports 2 Firmware version 2 6 0 Hardware version a0 Node GUID 0x0002c903000379a4 Svstem image GUID 0x0002c903000379a7 Port 1 State Active Phvsical state LinkUp Rate 40 Base lid 26 LMC O SM lid 23 Capabilitv mask 0x02510868 Port GUID 0x0002c903000379a5 Port 2 State Active Physical state LinkUp
6. CHAPTER 2 IB HCA ExpressModule Installation This chapter describes how to install the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule and verify that it has been installed correctly m Installing the IB HCA ExpressModule on page 7 m Install the ExpressModule on page 7 Installing the IB HCA ExpressModule The following instructions describe the basic tasks required to install the ExpressModule Refer to your system installation or service manual for detailed ExpressModule installation instructions Install the ExpressModule 1 Halt and power off your system 2 Install the ExpressModule into the ExpressModule slot pushing the module s edge connector into the connector on the chassis Ensure that the front plate on the ExpressModule card mounts flush with the chassis panel opening 3 Connect the 4x end of the InfiniBand I O cables to the ExpressModule port connectors Ensure that the connectors are properly engaged Caution Avoid putting unnecessary stress on the connection Do not bend or twist the cable near the connectors and avoid cable bends of more than 90 degrees 4 If not already connected connect the InfiniBand I O cables to the appropriate ports on the InfiniBand switch 5 Turn power back on to the system and allow the server to reboot This completes the hardware installation Proceed to the verification instructions in Verify the Installation With the Solaris 10 OS
7. Host Channel Adapter PCIe ExpressModule User s Guide April 2010
8. InfiniBand Devices on the Solaris 10 OS 17 Troubleshooting 17 InfiniBand Support Software for Linux 17 T w Download the MLNX_OFED Software and Documentation 18 Y InstalltheMLNX OFED Software on a Sun Server 18 Internet Protocol Over InfiniBand on Linux 19 Boot Over InfiniBand on Linux 19 Y Download the Boot Over IB Software and Documentation 19 Verifving the Installation With Linux 20 v Verify the Installation With Linux 20 Additional InfiniBand Software for Linux 21 A Product Specifications 23 ExpressModule Specifications 23 Index 25 iv Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule Users Guide April 2010 Safety Agency Compliance Statements Read this section before beginning any procedure The following text provides safety precautions to follow when installing a Sun Microsystems product Safety Precautions For your protection observe the following safety precautions when setting up your equipment m Follow all cautions and instructions marked on the equipment m Ensure that the voltage and frequency of your power source match the voltage and frequency inscribed on the equipment s electrical rating label m Never push objects of any kind through openings in the equipment Dangerous voltages may be present Conductive foreign objects could produce a short circuit that could cause fire electric shock or damage to your equipment Symbols The following symbols may appear in this
9. book Caution There is a risk of personal injury and equipment damage Follow the instructions Surfaces are hot and may cause personal injury if touched f Caution Hot surface Avoid contact Caution Hazardous voltages are present To reduce the risk of electric shock and danger to personal health follow the instructions Depending on the tvpe of power switch vour device has one of the following symbols may be used On Applies AC power to the system O Off Removes AC power from the system Standby The On Standby switch is in the standby position Modifications to Equipment Do not make mechanical or electrical modifications to the equipment Sun Microsystems is not responsible for regulatory compliance of a modified Sun product Placement of a Sun Product Caution Do not block or cover the openings of your Sun product Never place a Sun product near a radiator or heat register Failure to follow these guidelines can cause overheating and affect the reliability of your Sun product System Unit Cover You must remove the cover of your Sun computer system unit to add cards memory or internal storage devices Be sure to replace the cover before powering on your computer system Caution Do not operate Sun products without the cover in place Failure to take this precaution may result in personal injury and system damage Conformit aux
10. normes de s curit Veuillez lire attentivement cette section avant de commencer Ce texte traite des mesures de s curit qu il convient de prendre pour l installation d un produit Sun Microsystems Mesures de s curit Pour votre s curit nous vous recommandons de suivre scrupuleusement les mesures de s curit ci dessous lorsque vous installez votre mat riel m Suivez tous les avertissements et toutes les instructions inscrites sur le mat riel m Assurez vous que la tension et la fr quence de votre source d alimentation correspondent la tension et la fr quence indiqu es sur l tiquette de la tension lectrique nominale du mat riel m N introduisez jamais d objets quels qu ils soient dans les ouvertures de l quipement Vous pourriez vous trouver en pr sence de hautes tensions dangereuses Tout objet tranger conducteur risque de produire un court circuit pouvant pr senter un risque d incendie ou de d charge lectrique ou susceptible d endommager le mat riel Symboles Vous trouverez ci dessous la signification des diff rents symboles utilis s mat riel ou de vous blesser Veuillez suivre les instructions N Attention Vous risquez d endommager le Attention Surfaces br lantes Evitez tout contact Les surfaces sont br lantes Vous risquez de vous blesser si vous les touchez Attention Tensions dangereuses Pour r duire les risques de d charge lectrique et
11. ExpressModule are as follows m Compliant to the PCI ExpressModule Electromechanical Specification Revision 1 1 m Compliant to the InfiniBand Architecture Specification Release 1 2 m Two 40 Gbps 4x InfiniBand IB QSFP ports for connecting InfiniBand traffic 4x IB connectors m Media detect circuit supporting external InfiniBand active OSFP copper and fiber solutions m EU Restriction of Hazardous Substances RoHS compliant a IBTA InfiniBand Trade Association v1 2 compliant m PCIe 2 0 adapter m PCI Express expansion board with an x8 edge connector compatible to the PCI Express 1 0a specification m Media detect circuit which supports external InfiniBand fiber solutions m 4x IB port speed support 10 Gbps 20 Gbps or 40 Gbps QDR quad data rate ExpressModule FIGURE 1 1 ExpressModule FIGURE 1 2 ExpressModule Without Metal Case Im G 9 o 0 0 o 000 0000 Pd 9009 0000 0000 LC U o ooo mera NN BI Lee oo o oo o 9009 Pd o 000 0009 NM oooo 0000 0000 L 2 Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide April 2010 Figure Legend 1 InfiniBand Port 1 2 InfiniBand Port 2 3 PCI Express connector I O Interfaces The ExpressModule includes the following interfaces m Two 4x InfiniBand QSFP connectors m PCI ExpressModule to server or workstation connector m I O panel LEDs InfiniBand Interface The Sun Dual Port 4x ODR IB Host Channel Adapter PCIe Exp
12. Select Linux SW Drivers from the menu for InfiniBand SW Drivers Follow the instructions provided on the web page to complete the download Select and download the related documentation for MLNX_OFED D At minimum download copies of these manuals that are offered on the Boot Over IB page User s Manual Release Notes Chapter 3 InfiniBand Software on the Solaris Operating System and Linux 19 Verifying the Installation With Linux Before you can verify the installation you must install the ExpressModule in the chassis power on the server and cable the server to an operational InfiniBand switch The InfiniBand switch should automatically recognize an InfiniBand server that is connected to the fabric V Verifv the Installation With Linux The InfiniBand switch should automaticallv recognize the IB HCA ExpressModule when it is connected to the fabric if the IB Subnet Manager is running on the switch or on a host within the subnet 1 Ensure that the cables are connected to the ExpressModule and switches 2 Verify that the IB Subnet Manager is running on the IB switch or on a host within the subnet Refer to the manual for the IB Subnet Manager for more information 3 Check that the green LED is illuminated for each port that is connected to the switch If the green LED is not on check the cable connections at the ExpressModule and at the switch 4 Check that the amber LED is illuminated for each port tha
13. Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide NS amp Sun ORACLE Part No 820 6539 13 April 2010 Revision A Copyright 2009 2010 Oracle and or its affiliates All rights reserved This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws Except as expressly permitted in your license agreement or allowed by law you may not use copy reproduce translate broadcast modify license transmit distribute exhibit perform publish or display any part in any form or by any means Reverse engineering disassembly or decompilation of this software unless required by law for interoperability is prohibited The information contained herein is subject to change without notice and is not warranted to be error free If you find any errors please report them to us in writing If this is software or related software documentation that is delivered to the U S Government or anyone licensing it on behalf of the U S Government the following notice is applicable U S GOVERNMENT RIGHTS Programs software databases and related documentation and technical data delivered to U S Government customers are commercial computer software or commercial technical data pursuant to the applicable Federal Acquisition Regulation and agency specific supplemental regulations As such the use duplica
14. arks of SPARC International Inc UNIX is a registered trademark licensed through X Open Company Ltd This software or hardware and documentation may provide access to or information on content products and services from third parties Oracle Corporation and its affiliates are not res onsible for and expressly disclaim all warranties of any kind with respect to third party content products and services Oracle Corporation and its affiliates will not be responsible for any loss costs or damages incurred due to your access to or use of third party content products or services Si CA Adobe PostScript Contents Safety Agency Compliance Statements v Regulatory Compliance Statements ix Declaration of Conformity xi Preface xiii Overview 1 Adapter Hardware Overview 1 ExpressModule 2 I O Interfaces 3 InfiniBand Interface 3 PCI ExpressModule Interface 3 LED Assignment 3 Power 5 Node GUID 5 Hardware and Software Requirements 6 IB HCA ExpressModule Installation 7 Installing the IB HCA ExpressModule 7 v Install the ExpressModule 7 Hot Swapping With the Linux OS 9 Y To Hot Swap the ExpressModule Using the Linux OS 9 3 InfiniBand Software on the Solaris Operating System and Linux 13 InfiniBand Software on the Solaris Operating System 13 InfiniBand Software for Solaris 10 14 Sun Firmware Flash Update Tool for IB HCAs 14 Y Verify the Installation With the Solaris 10 OS 15 Sun Firmware Version for IB HCAs on the Solaris OS 16 Using
15. ber from the phvsical label on the IB HCA card See Node GUID on page 5 The condition column must displav ok to indicate that the driver has discovered the hardware and bound to it Refer to the cfgadm ib 1m man pages for details about the IB specific extensions Chapter 3 InfiniBand Software on the Solaris Operating System and Linux 15 b Use the cfga m 1m command to obtain port GUIDs for each port on the IB HCA cfgadm als cols ap id info hca Ap Id Information hca 2C90109763F70 VID 0x15b3 PID 0x5a44 ports 0x2 portl GUID 0x2C90109763F71 port2 GUID 0x2C90109763F72 If more than one IB HCA device is installed in the server a row is displayed for each Look for the row displaying hca GUID where GUID is the 64 bit number from the physical label on the IB HCA card See Node GUID on page 5 Use the port number and GUID displayed by this command for your IB HCA device in the following step c Use the cfga m 1m command to verify that the IB ports and partitions are configured by the Subnet Manager cfgadm als select type IB VPPA cols ap_ id Ap_Id ib 2C90109763F71 ffff ipib ib 2C90109763F72 ffff ipib The command displavs the AP ID column where each row has the format of ib Port GUID P_Key ipib Match the Port GUIDs from the previous command with these port GUIDs There must be one row corresponding to the port and P_Key setup by the Subnet Manager If an entry is missing ch
16. de danger physique observez les consignes indiqu es Selon le type d interrupteur marche arr t dont votre appareil est quip l un des symboles suivants sera utilis Marche Met le syst me sous tension alternative Arret Met le syst me hors tension alternative Veilleuse L interrupteur Marche Veille est sur la position de veille Modification du mat riel N apportez aucune modification m canique ou lectrique au mat riel Sun Microsystems d cline toute responsabilit quant la non conformit ventuelle d un produit Sun modifi Positionnement d un produit Sun Attention Evitez d obstruer ou de recouvrir les orifices de votre produit Sun N installez jamais un produit Sun pr s d un radiateur ou d une source de chaleur Si vous ne respectez pas ces consignes votre produit Sun risque de surchauffer et son fonctionnement en sera alt r Couvercle de l unit Pour ajouter des cartes de la m moire ou des p riph riques de stockage internes vous devez retirer le couvercle de votre syst me Sun Remettez le couvercle sup rieur en place avant de mettre votre syst me sous tension Sun sous tension si leur couvercle sup rieur n est pas mis en place Si vous ne prenez pas ces pr cautions vous risquez de vous blesser ou d endommager le syst me N Attention Ne mettez jamais des produits vi Sun Dual Port 4x QDR IB Host Channel Adapter PCle Expre
17. eck the Subnet Manager configuration Sun Firmware Version for IB HCAs on the Solaris OS To use this ExpressModule with the Solaris OS the minimum firmware version must be 2 7 000 Use the firmwareflash command to display the revision level of your IB HCA ExpressModule firmwareflash 1 c IB Look for the revision number that appears after the Firmware revision string If more than one HCA device is displayed look for the Node Image GUID that matches the GUID displayed on the physical GUID label of the IB HCA ExpressModule being installed See Node GUID on page 5 16 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 If the firmware version is not at 2 7 000 or higher update the firmware Only update the firmware on your IB HCA ExpressModule with files specifically approved for the Oracle product Select and download approved firmware files from http www mellanox com support firmware table Sun php After obtaining a firmware image use the firmwareflash command to install the firmware Once installed reboot the system to enable the new firmware Using InfiniBand Devices on the Solaris 10 OS For details about InfiniBand software stack configurations in a Solaris 10 Update release refer to the System Administration Guide Devices and File Systems document in the Solaris 10 System Administrator Collection available at http docs sun com Section 9 of this guide titled Usin
18. ervice Required Indicator The pair of LEDs for each port have the meanings described in TABLE 1 1 TABLE 1 1 LED Meanings LED Color LED Name LED State Meaning Green Physical Link Amber Data Activity Lit Unlit Steady light Blinking light Unlit The link bringup process has successfullv completed and the link width link speed link polaritv and link reversal have been negotiated with the neighbor port on the other end of the cable A phvsical connection has not been established Infiniband is discovered over the phvsical link but no data is being passed Data is being passed A phvsical or logical connection or both has not been established Power The ExpressModule receives power from the server or workstation slot connector All other required power is generated by ExpressModule switch mode regulators For power consumption see Appendix A Node GUID A label on the Express Module enclosure displavs the 64 bit GUID This GUID uniquelv identifies this device in the server and on the IB fabric Chapter 1 Overview 5 Hardware and Software Requirements For the latest list of supported platforms and operating systems refer to the Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule Product Notes part number 820 6540 This document is available at http docs sun com Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010
19. ftware for Solaris 10 For details about InfiniBand software supported in Solaris 10 Update releases refer to the following documents in the Solaris 10 Release and Installation Collection available at http docs sun com m Solaris 10 What s New m Solaris 10 Release Notes m Solaris 10 Package List Note The SUNWhermon package that is available in the Solaris 10 10 09 OS and subsequent Solaris Update releases must be used with this IB HCA ExpressModule The InfiniBand software stack consisting of the upper layer protocols and transport framework is included in all of the Solaris software groups described in the Solaris Installation Guide The SUNWhermon package is included in the Entire OEM Entire and Developer software groups If you are not using any of these groups you must explicitly add the SUNWhermon package during initial installation If you are not doing a software install use the pkgadd 1 utility to add the package prior to using the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule Sun Firmware Flash Update Tool for IB HCAs The Sun Firmware Flash Update tool in the Solaris 10 OS does not support the Sun Dual Port 4x ODR IB Host Channel Adapter You must download a separate package containing that tool from the Oracle Download Webpage at http www sun com download index jsp Go to the Download A Z tab and search for the Sun Firmware Flash Utility Refer to the installation instructions in t
20. g InfiniBand Devices Overview lasks describes how to set up upper layer protocols such as IPoIB and uDAPL Troubleshooting Check the Sun Dual Port 4x QDR IB Host Channel Adapter PCIe Express Module Product Notes 820 6540 for information about known issues discovered using the Solaris 10 OS with your IB HCA card When using IPolB verify that the broadcast group is configured by the Subnet Manager in the partition where the IPolB link will be used InfiniBand Support Software for Linux With most supported Linux releases vou must also install the MLNX OFED software stack This software is the Mellanox OpenFabrics Enterprise Distribution OFED for Linux Refer to vour Linux vendor for software installation recommendations and suppott Chapter 3 InfiniBand Software on the Solaris Operating System and Linux 17 V Download the MLNX OFED Software and Documentation 1 Go to the Mellanox Technologies web site http www mellanox com Select the Products tab Select InfiniBand SW Drivers from the menu for Products Select Linux SW Drivers from the menu for InfiniBand SW Drivers Select the download that corresponds to vour operating svstem Follow those instructions to complete the download Select and download the related documentation for MLNX OFED At minimum download copies of these manuals that are offered on the Linux SW Drivers page m Installation Guide u Linux User s Manual m Release Notes V I
21. he package README file To check that the correct version is installed type firmwareflash v firmwareflash version v1 9 Note This command must display version number 1 9 or higher 14 Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide April 2010 V Verify the Installation With the Solaris 10 OS Before you can verify the installation you must install the adapter in the chassis power on the server and cable the server to an operational InfiniBand switch Afterward perform the following steps 1 Ensure that the cables are connected to the adapter and switches 2 Verify that the IB Subnet Manager is running on the IB switch or on a host within the subnet Refer to the manual for the IB Subnet Manager for more information 3 Check that the green LED is illuminated for each port that is connected to the switch If the green LED is not on check the cable connections at the adapter and at the switch 4 Check that the amber LED is illuminated for each port that is connected to the switch 5 Verify that the IB HCA ports are up and the driver is attached a Use the cfgadm 1m command to obtain the state of the device installed cfgadm als cols ap id condition hca Ap Id Condition hca 2C90109763F70 ok If more than one IB HCA device is installed in the server a row is displaved for each Look for the row displaying hca GUID where GUID is the 64 bit num
22. hselstromzufuhr zum Ger t Wartezustand Der Ein Standby Netz schalter befindet sich in der Standby Position Modifikationen des Ger ts Nehmen Sie keine elektrischen oder mechanischen Ger temodifikationen vor Sun Microsystems ist fiir die Einhaltung der Sicherheitsvorschriften von modifizierten Sun Produkten nicht haftbar Geh useabdeckung Sie m ssen die Abdeckung Ihres Sun Computersystems entfernen um Karten Speicher oder interne Speicherger te hinzuzuf gen Bringen Sie vor dem Einschalten des Systems die Geh useabdeckung wieder an Achtung Nehmen Sie Sun Ger te nicht ohne Abdeckung in Betrieb Die Nichtbeachtung dieses Warnhinweises kann Verletzungen oder Ger teschaden zur Folge haben Safety Agency Compliance Statements vii Normativas de seguridad Lea esta secci n antes de realizar cualquier operaci n En ella se explican las medidas de seguridad que debe tomar al instalar un producto de Sun Microsystems Medidas de seguridad Para su protecci n tome las medidas de seguridad siguientes durante la instalaci n del equipo m Siga todos los avisos e instrucciones indicados en el equipo m Aseg rese de que el voltaje y frecuencia de la fuente de alimentaci n coincidan con el voltaje y frecuencia indicados en la etiqueta de clasificaci n el ctrica del equipo m No introduzca objetos de ning n tipo por las rejillas del equipo ya que puede quedar expuesto a voltajes peligroso
23. nstall the MLNX OFED Software on a Sun Server 1 Refer to the Mellanox OFED for Linux Installation Guide you downloaded from the Mellanox web site The instructions in that document are correct for installation on a Sun server except for a difference explained in the next step Whenever a procedure in the Mellanox guide calls for running the mlnxofedinstall script always include the without fw update option This option prevents the MLNX_OFED installation process from automatically updating the firmware on your Sun HCA Only update the firmware on that device using files specifically approved for Sun product You can select and download approved firmware files from http www mellanox com support firmware table Sun php 18 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 Internet Protocol Over InfiniBand on Linux Support for Internet Protocol Over InfiniBand IPoIB is included in the MLNX_OFED software distribution Details on using IPoIB are included in the Mellanox OFED Stack for Linux User s Manual Boot Over InfiniBand on Linux Software to enable Boot Over InfiniBand BolB on Linux is available from the Mellanox Technologies web site Download the Boot Over IB Software and Documentation 1 Go to the Mellanox Technologies web site http www mellanox com Select the Products tab Select Boot Over IB from the menu for Products Select Download
24. or the Sun Dual Port 4x ODR IB Host Channel Adapter PCIe ExpressModule TABLE A 1 Board Specifications Physical Size 198 mm X 112 mm 7 8in x 4 4in Air flow 200 LEM 55C 4x 20 Gb sec connector InfiniBand Copper current rating 0 5A max with active media adapter support Protocol Support IBTA v1 2 Auto Negotiation 40 Gbps 20 Gbps 10 Gbps QoS 8 InfiniBand virtual lanes for each port RDMA support All ports Power and Environmental Voltage 12 V 3 3 V Max power 12 W Temperature 0 to 55 Celsius 23 The auto negotiation protocol is proprietarv of Mellanox Technologies and compliant with the InfiniBand Architecture Specification Re lease 1 2 24 Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule Users Guide April 2010 Index B BolB 19 documentation 19 Boot Over IB Software BolB 19 F firmware updates 18 I I O cables 7 8 interfaces 3 panels 3 IB EM installation 7 IB Subnet Manager 20 IB HCA installation 7 installing the adapter 7 Internet Protocol Over InfiniBand IPoIB 19 IPoIB 19 L LED assignment 3 description 3 interface 3 location 4 verifv portis up 20 Linux support 13 M Mellanox Open Fabrics Enterprise Distribution MLNX OFED 17 minxofedinstall script 18 O Open Fabrics Enterprise Distribution OFED 17 OpenFabrics organization 21 S Subnet Manager 20 V Verifying the installation 20 25 26 Sun Dual Port 4x QDR IB
25. pressModule User s Guide HTML Latest Sun Dual Port 4x QDR IB Host Channel Adapter PCIe 820 6540 PDF Online Information ExpressModule Product Notes HTML Solaris Solaris 10 Operating System documentation collection PDF Online support http docs sun com app docs prod solaris 10 HTML xiii If you download software from Mellanox Technologies as described in this manual you might also need documentation that is available at http www mellanox com Details on locating specific manuals at this web site are provided in Chapter 3 Application Title Format Location Installing Linux support Mellanox OFED for Linux Installation Guide PDF Online software Use of Linux support Mellanox OFED Stack for Linux User s Manual PDF Online software and IPoIB features Latest information on Mellanox OFED for Linux Release Notes Text file Online Linux support software Use of BolB features Boot over IB BoIB User s Manual PDF Online Latest information on Mellanox Boot over IB BoIB in Mellanox OFED Text file Online BolB for Linux Release Notes Documentation Support and Training Sun Function URL Documentation http www sun com documentation Support http www sun com support Training http www sun com training Third Party Web Sites Sun is not responsible for the availability of third party web sites mentioned in this document Sun does not endorse and is not responsible or liable for any content advertising
26. ressModule is compliant with the InfiniBand Architecture Specification Release 1 2 The ExpressModule has two compliant 4x InfiniBand ports Port 1 and Port 2 each having four Tx Rx pairs of SerDes The ExpressModule provides access to these ports by means of two 4x InfiniBand connectors for external InfiniBand copper cables also compliant with the IBTA specification 1 2 Connector 1 connects to Port 1 of the device while connector 2 connects to Port 2 Furthermore the ExpressModule is embedded with a media detect circuit that supports external InfiniBand active copper and fiber solutions PCI ExpressModule Interface The ExpressModule is compatible with the PCI ExpressModule Electromechanical Specification Revision 1 1 The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations LED Assignment The ExpressModule has four LEDs located on the InfiniBand ports panel with two LEDs for each 4x port See FIGURE 1 3 Chapter 1 Overview 3 FIGURE 1 3 InfiniBand Ports and LEDs Figure Legend InfiniBand Port 1 Green LED for Port 1 Physical Link Amber LED for Port 1 Data Activity Link InfiniBand Port 2 Green LED for Port 2 Physical Link Amber LED for Port 2 Data Activity Link oa fF Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide April 2010 Figure Legend Continued 7 Power On Indicator 8 Hot Swap Button 9 S
27. s Los objetos conductores extra os pueden producir cortocircuitos y en consecuencia incendios descargas el ctricas o da os en el equipo S mbolos En este documento aparecen los siguientes s mbolos N Precauci n Existe el riesgo de que se produzcan lesiones personales y da os en el equipo Siga las instrucciones contacto Las superficies est n calientes y Precauci n Superficie caliente Evite todo pueden causar lesiones personales si se tocan Precauci n Voltaje peligroso Para reducir el riesgo de descargas el ctricas y lesiones personales siga las instrucciones En funci n del tipo de interruptor de alimentaci n del que disponga el dispositivo se utilizar uno de los simbolos siguientes Encendido Suministra alimentaci n de CA al sistema Apagado Corta la alimentaci n de CA del sistema Espera El interruptor de encendido espera est en la posici n de espera Modificaciones en el equipo No realice modificaciones de tipo mec nico ni el ctrico en el equipo Sun Microsystems no se hace responsable del cumplimiento de normativas en caso de que un producto Sun se haya modificado Colocaci n de un producto Sun Precauci n No obstruva ni tape las rejillas del producto Sun Nunca coloque un producto Sun cerca de radiadores ni fuentes de calor Si no sigue estas indicaciones el producto Sun podria sobrecalentarse v la fiabilidad de
28. s likelv to cause harmful interference in which case the user will be required to correct the interference at his own expense Modifications Anv modifications made to this device that are not approved bv Sun Microsvstems Inc mav void the authoritv granted to the user bv the FCC to operate this equipment ICES 003 Class A Notice Avis NMB 003 Classe A This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada VCCI B EIC DUT JZAAVCCI B T IFAAVCCI ORAS I YAF Y a BLOAT Y a AMI DAA ER RIETI THSOMMII TOOR H EL dT CORRI FRASER H TH mnt VCCI DHEIHE DS IF 52 ARERR CT ZOKBZRZERETH TZEERW EHI SL n HOES ZOSAKEHERETT IHREM TDLIARANZIEMBOKT BSMI Class A Notice The following statement is applicable to products shipped to Taiwan and marked as Class A on the product compliance label FR EHE E EEN KRI gt CCC Class A Notice The following statement is applicable to products shipped to China and marked with Class A on the product s compliance label Class A A gt
29. ssModule User s Guide April 2010 Einhaltung sicherheitsbeh rdlicher Vorschriften Lesen Sie vor dem Ausf hren von Arbeiten diesen Abschnitt Im folgenden Text werden Sicherheitsvor kehrungen beschrieben die Sie bei der Installation eines Sun Microsystems Produkts beachten m ssen Sicherheitsvorkehrungen Treffen Sie zu Ihrem eigenen Schutz bei der Installation des Ger ts die folgenden Sicherheitsvorkehrungen m Beachten Sie alle auf den Ger ten angebrachten Warnhinweise und Anweisungen m Stellen Sie sicher dass Spannung und Frequenz der Stromversorgung den Nennleistungen auf dem am Ger t angebrachten Etikett entsprechen m F hren Sie niemals Fremdobjekte in die ffnungen am Ger t ein Es k nnen gef hrliche Spannungen anliegen Leitf hige Fremdobjekte k nnen einen Kurzschluss verursachen der einen Brand Strom schlag oder Ger teschaden herbeif hren kann Symbole Die Symbole in diesem Handbuch haben folgende Bedeutung Achtung Gefahr von Verletzung und Ger teschaden Befolgen Sie die Anwei sungen Achtung Hei e Oberfl che Nicht ber hren da Verletzungsgefahr durch hei e Oberfl che besteht Achtung Gef hrliche Spannungen Befolgen Sie die Anweisungen um Stromschl ge und Verletzungen zu vermeiden Je nach Netzschaltertyp an Ihrem Ger t kann eines der folgenden Symbole verwendet werden Ein Versorgt das System mit Wechselstrom Aus Unterbricht die Wec
30. su funcionamiento se veria afectada viii Sun Dual Port 4x QDR IB Host Channel Adapter PCle ExpressModule User s Guide April 2010 Regulatory Compliance Statements Your Sun product is marked to indicate its compliance class Federal Communications Commission FCC USA Industry Canada Equipment Standard for Digital Equipment ICES 003 Canada Voluntary Control Council for Interference VCCI Japan e Bureau of Standards Metrology and Inspection BSMI Taiwan Please read the appropriate section that corresponds to the marking on vour Sun product before attempting to install the product FCC Class A Notice This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This device mav not cause harmful interference 2 This device must accept anv interference received including interference that mav cause undesired operation Note This equipment has been tested and found to complv with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if it is not installed and used in accordance with the instruction manual it mav cause harmful interference to radio communications Operation of this equipment in a residential area i
31. t is connected to the switch 5 Verify that the IB EM ports are up and the driver is attached by typing gt ibstat The state of each port is displayed State Comments Active The port is working Down The port is not working or the cable is not connected Initializing If the initialization does not finish the Subnet Manager might not be running for the subnet 20 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 Additional InfiniBand Software for Linux As the popularity of InfiniBand technology increases the number of Linux distributions and open source organizations producing drivers and tools will increase For up to date information check with open source organizations such as http OpenFabrics org and your current vendors The OpenFabrics organization is the Open Software solution in the InfiniBand software space The OpenFabrics Enterprise Distribution OFED is the InfiniBand suite of software produced by this organization Various vendors contribute their drivers and other software components to OFED Chapter 3 InfiniBand Software on the Solaris Operating System and Linux 21 22 Sun Dual Port 4x QDR IB Host Channel Adapter PCIe ExpressModule User s Guide April 2010 APPENDIX A Product Specifications This chapter includes m ExpressModule Specifications on page 23 ExpressModule Specifications TABLE A 1 lists the specifications f
32. tion disclosure modification and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract and to the extent applicable by the terms of the Government contract the additional rights set forth in FAR 52 227 19 Commercial Computer Software License December 2007 Oracle USA Inc 500 Oracle Parkwav Redwood Citv CA 94065 This software or hardware is developed for general use in a variety of information management applications It is not developed or intended for use in any inherently dangerous applications including applications which may create a risk of personal injury If you use this software or hardware in dangerous applications then you shall be responsible to take all appropriate fail safe backup redundancv and other measures to ensure the safe use Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications Oracle is a registered trademark of Oracle Corporation and or its affiliates Oracle and Java are registered trademarks of Oracle and or its affiliates Other names may be trademarks of their respective owners AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered tradem
Download Pdf Manuals
Related Search