Home

1 - Fujitsu manual server

Contents

1. 11 2 2 Finding the GUID and Serial Number on the Adapter Cards 12 2 3 Safety instructions lr 13 2 4 ENERGY STAR oresbia Ge aeaea ce wend 19 2 5 CE conformity ca caco kee de ee ewe eimi 19 2 6 Adapter Card Interfaces 20 26 1 VO Menaces 26 n0 3 793958 38 05 SS DRO See 3 2 20 2602 InfiniBand Interface aoaaa aa a 20 2 6 3 PCI Express Interface llle 21 2 6 4 EEDOSBEIDAITIBEIL gogo eu dioc Bh eo Bk Hoke E d 21 2 7 Adapter Card Installation 22 2 5 Hardware and Software Requirements 22 RA instalation InStUcUgNS 4 0 6 688 ee Gas oe ee RR OR 22 27 Cables a oh 22 x OO amp E dC o eae EEE ee EES d 23 2 7 3 1 Cable Installation lt 24 uoo RR 23 232 Inserting a Cable into the Adapter Card 24 2738 Removing a Cable from the AdapterCard 24 Fujitsu InfiniBand HCA FDR User Manual 3 Contents 3 Linux Installation 25 3 1 OVGIVIOW sica cca o EORR aeaa ee 25 3 2 Software Requirements 25 3 3 OFED Installation 2 se sae tok nono TR Ux rU 26 3 3 1 OFED installation on RHEL5 26 3 3 2 OFED installation on RHEL6 29 3 3 3 Post installation Notes o oo a 31 3 3 4 OFED installati n lt lt lt gt sre RR Re RRS ROO RO 35 3 3 5 OFED installation procedure when applying errata kernel 36 3 3 5 1 How to apply t
2. L 1 0 0 J 1 scsi target utils HHH HHH THHE THE HHH HHH 1HE HHH 1HE THE L 1 0 0 Lroot RX200S7 1 Packages rpm q scsi target utils scsi target utils 1 0 14 4 e16 x86 64 Unmount mnt and eject the installation disk L rootGRX200S7 1 Packages cd rootGRX200S7 1 umount mnt rootGRX200S7 1 14 Reboot the system rootGRX200S7 1 reboot rootGRX200S7 1 J After the OS boot up confirm the following two lines are included in limits conf file L rootGRX200S7 1 tail 2 etc security limits conf soft memlock unlimited hard memlock unlimited Confirm the IB drivers are loaded Lroot RX200S7 1 J etc init d openibd status HCA driver loaded Configured IPoIB devices ibO ibl Currently active IPoIB devices 40 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation The following OFED modules are loaded rdma_ucm rdma_cm ib addr ib ipoib mlx4 core mlx4 ib mlx4 en ib mthca ib uverbs ib umad ib ucm ib sa ib cm ib mad ib core iw nes 3 3 5 2 How to install OFED when the errata kernel is already applied This chapter describes how to install OFED after installing OS and applying the errata kernel This is not the initial OFED installation procedure The flow of OFED installation is as follows 1 Install OS 2 Apply the errata kernel 3 Create OFED ISO image for the errata kernel
3. InstallShield Back Next gt Cancel Figure 11 Maximum performance 52 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation 8 Click Install after the following window appears i MLNX_VPI InstallShield Wizard Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Figure 12 Installation Fujitsu InfiniBand HCA FDR User Manual 53 WinOF Installation Windows Installation 9 The following window appears after the installation process is finished Click Finish without checking any boxes i MLNX VPI InstallShield Wizard InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPI Click Finish to exit the wizard You chose not to run performance tuning Please note that it is highly recommended to run this tuning For optimum performance You can run it later by selecting the Performance Tuning tab in the Mellanox Connect 10GBit Adapter properti Enable NetworkDirect interface ND 7 Show IB release notes Show ETH release notes Figure 13 Installation completed 54 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation 10 Make sure that InfiniBand devices are recognized prope
4. There are the following two procedures e How to apply the errata kernel when OFED is already installed e How to install OFED when the errata kernel is already applied 3 3 5 1 How to apply the errata kernel when OFED is already installed This chapter explains errata kernel application procedure when OFED is already installed This is not the procedure of initial OFED installation The procedure is as follows 1 Uninstall OFED 2 Apply errata kernel 3 Create OFED ISO image for the errata kernel 4 Install OFED with the created ISO image 1 OFED Uninstallation 1 Download OFED ISO image in advance This can be downloaded from the following Fujitsu Technology Solutions web site http support ts fujitsu com download index asp levell 220966 amp Ing com amp LNID 4 2 Confirm whether the scsi target utils is installed or not If it is installed uninstall it before uninstall OFED If it is not installed go to 3 rootGRX200S7 1 1 rpm q scsi target utils scsi target utils 1 0 14 4 e16 x86 64 rootGRX200S7 1 J rpm e scsi target uti1s 1 0 14 4 e16 x86 64 L rootGRX200S7 1 14 36 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation 3 Uninstall OFED as follows with the ISO image downloaded at Enter y during the uninstallation Uninstallation of OFED can be performed also by the usr sbin ofed_uninstall sh rootGRX200S7 1 J mount ro loop MLNX OFED LINUX 1 5
5. 2004 108 EC regarding Electromagnetic Compatibility and 2006 95 EC Low Voltage Directive This is indicated by the CE marking CE Communaut Europ enne C The system complies with the requirements of the EC directives Fujitsu InfiniBand HCA FDR User Manual 19 Adapter Card Interfaces Adapter Card Hardware 2 6 Adapter Card Interfaces 2 6 1 I O Interfaces Each adapter card includes the following interfaces QSFP ports PCI Express x8 edge connector O panel LEDs For dual port cards port 1 connects to connector 1 of the device while port 2 connects to connector 2 of the device Port 1 Port 2 Figure 5 Port numbering e g IB HCA 2port FDR 2 6 2 InfiniBand Interface The ConnectX 3 device is compliant with the InfiniBand Architecture Specifi cation Release 1 3 plan Adapter cards listed in table 2 on page 11 based on this device provide access to its ports by means of QSFP connectors 20 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware Adapter Card Interfaces 2 6 3 PCI Express Interface The ConnectX2 3 adapter cards support PCI Express 3 0 1 1 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations 2 6 4 LED Assignment The board has I O LEDs located on the I O panel 2 LEDs per port The green LED when lit indicates that the driver is ru
6. co User Manual English FUJITSU PRIMERGY InfiniBand Host Channel Adapter 56Gb FDR V6 0 Linux Windows Edition February 2014 Comments Suggestions Corrections The User Documentation Department would like to know your opinion of this manual Your feedback helps us optimize our documentation to suit your individual needs Feel free to send us your comments by e mail to manuals ts fujitsu com Certified documentation according to DIN EN ISO 9001 2008 To ensure a consistently high quality standard and user friendliness this documentation was created to meet the regulations of a quality management system which complies with the requirements of the standard DIN EN ISO 9001 2008 cognitas Gesellschaft f r Technik Dokumentation mbH www cognitas de Copyright and Trademarks Copyright Fujitsu Technology Solutions GmbH 2014 All rights reserved Delivery subject to availability right of technical modifications reserved All hardware and software names used are trademarks of their respective manufacturers Contents 1 Introductloli ud d wm bu E RRAPIE Wee ree ES 7 1 1 Notational Conventions s 7 1 2 Target GOUD uuu ook hh d Ex ee eG Ru Gade Sas 8 1 3 Documentation Overview 8 1 4 OvervigW x2z223353xX 532xREkEEERx 9 1 5 Product Prerequisites 2c rr tmm 10 2 Adapter Card Hardware 11 2 1 Adapter Cards Covered in this Manual
7. lithium battery 17 log file 61 Low Voltage Directive 19 M master 45 N notational conventions 7 Fujitsu InfiniBand HCA FDR User Manual Index 88 User Manual Fujitsu InfiniBand HCA FDR
8. 29 OFED Installation Linux Installation Some RPMs may depend on the RPMs above Please uninstall them manually Uninstall the packages root localhost mntl Croot localhost mntJ rpm e nodeps allmatches rdma rdma 1 0 14 e16 noarch libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs dapl rds tools ibutils infinipath psm libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs compat opensm libs dapl rds tools infinipath psm opensm libs libipathverbs dapl libibcm libibmad libibumad ibsim ibutils ibutils libs fcoe utils compat openmpi compat openmpi psm fcoe utils 4 Execute Installer again and enter y Croot localhost mnt mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y Starting MLNX OFED LINUX 1 5 3 3 0 0 installation Installing mInx ofa kernel RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI mlnx ofa kernel THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installing kmod mlnx ofa kernel RPM Installation finished successfully Configuring etc security limits conf Please reboot your system for the changes to take effect Following RPMs com
9. 3 3 1 0 rhe16 2 x86 64 i rootGRX200S7 1 4 cd rootGRX200S7 1 mnt This program will unin machine so mnt mnt uninstall sh stall all OFED packages on your Do you want to continue Ly N1 y Running usr sbin vend or pre uninstall sh Removing OFED Software installations Running bin rpm e mpitests mvapich gcc 3 2 923 x86 64 Running bin rpm e mpitests openmpi gcc 3 2 923 x86 64 Stopping IB Subnet Man ager OK J Running tmp 7096 ofed vendor post uninstall sh root RX200S7 1 mnt 4 Unmount mnt rootGRX200S7 1 mntI cd root RX200S7 1 umount mnt rootGRX200S7 1 5 Reboot the system root RX200S7 1 reboot root RX200S7 1 JI 2 Errata kernel application 1 Apply the errata kernel by yum or rpm command 2 Reboot the system 3 OFED ISO image creation for the errata kernel 1 Check the kernel version after reboot rootGRX200S7 1 J un ame r 2 6 32 220 23 1 e16 x86 64 Fujitsu InfiniBand HCA FDR User Manual 37 OFED Installation Linux Installation 2 Create the mounted directory other than mnt for example work Because the ISO image creation command doesn t work if mnt directory is mounted rootGRX200S7 1 mkdir work rootGRX200S7 1 J Mount the ISO image used at 1 OFED Uninstallation rootGRX200S7 1 mount ro loop MLNX OFED LINUX 1 5 3 3 1 0 rhel16 2 x86 64 iso work Execute the c
10. 4 Install OFED with the created ISO image 1 OS installation Install OS 2 Applying the errata kernel Follow the step 2 Errata kernel application in 3 3 5 1 How to apply the errata kernel when OFED is already installed 3 OFED ISO image creation for the errata kernel Follow the step 3 OFED ISO image creation for the errata kernel in 3 3 5 1 How to apply the errata kernel when OFED is already installed Fujitsu InfiniBand HCA FDR User Manual 41 OFED Installation Linux Installation 4 OFED installation with the created ISO image Follow the step 4 OFED installation with the created ISO image in 3 3 5 1 How to apply the errata kernel when OFED is already installed If packages remove messages are displayed add an option nodeps Delete scsi target utils from the package currently displayed and uninstall a package Then execute installer again Depending on the version of OFED some packages are uninstalled automatically Please refer to chapter 3 3 1 OFED installation on RHEL5 or chapter 3 3 2 OFED installation on RHEL6 3 3 6 OFED installation on RHEL HPC Compute Node RHEL HPC Compute Node and RHEL Server have a difference in the package to install RHEL HPC Compute Node is less than RHEL Server Therefore installation of OFED is not complete normal Please install OFED after installing a required package in advance 1 Required packages and RHEL package set The package required for OFED installation is
11. A est conforme a la norme NMB 003 du Canada 76 User Manual Fujitsu InfiniBand HCA FDR Appendix A A 1 EMC Certification Statements 6 1 4 VCCI Statements Japan Class A Statement COR WHLBRESERMSAERMHMBRS VCC OH C3 2 72AATISEISIEAEIE C d CORE RERI CEA TS CBR BREET CEPHV ETF OBS S P389 4 XE RE BEIBKENSCEPHVETF Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions 6 1 5 KCC Certification Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and certification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL which is organized under the Ministry of Information and Communica tions KCC EMC testing includes electromagnetic emissions EMI and susceptibility EMS Certified equipment is labeled with the KCC mark and certification number 9 ZIZE BFE LE AES PSSS FH 71719 21 BHA EE AHE e BE FSSA 7 epu VF AK Sal SE TURES HVE AASLS m5 ngu Translation Class A Device This device is registered for EMC requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced wit
12. Fujitsu InfiniBand HCA FDR User Manual 61 Checking the Log File of OpenSM Linux only Troubleshooting Removed port with GUID GUID number PRIx64 LID range LID number LID number of node How to solve the problem The following changes are required Please contact the repair center 1 IB HCA You need to change the relevant IB HCA You obtain the relevant IB HCA from GUID or LID information in the error messages and the output of ibnetdiscover For example Jul 30 23 34 18 600287 4CD0D940 0x02 gt osm_drop_mgr_remove_port Removed port with GUID 0x0002c90300041306 PRIx64 LID range 2 2 of node localhost HCA 2 In this case Error Card has the GUID 0x0002c90300041306 and the LID is 2 of HCA InfiniBand Switch You need to change the relevant InfiniBand Switch You obtain the relevant Infiniband Switch from GUID or LID information in the error messages and the output of ibnetdiscover For example Jul 30 23 34 18 600424 4CDOD940 0x02 gt osm drop mgr remove port Removed port with GUID 0x0002c9020040c790 PRIx64 LID range 21 21 of node none SW 1 In this case Error InfiniBand Switch has the GUID 0x0002c9020040c790 and the LID is 21 IB cable If you have changed the above InifiniBand Switch and you still receive the messages all IB cables connected to the InfiniBand Switch need to be changed Reporting Generic Notice type 1 num 128 Link state change from LID LID number GUID GU
13. Hat Enterprise Linux instal lation or afterwards by using the OpenFabrics Enterprise Distribution OFED In the first case the correct packages must be selected during setup and should only be performed by an advanced user For simple complete installation use the OFED drivers If you purchased the products from Fujitsu Japan you can download it from http primeserver fujitsu com primergy downloads Please note that only Red Hat Enterprise Linux Version 5 7 or higher and 6 1 or higher are supported The chapter includes the following sections e OFED Installation on page 26 e Updating Firmware after Installation on page 44 e Subnet Manager on page 44 3 2 Software Requirements Required Disk Space for Linux Installation 400 MB Operating System Linux operating system x86 64 only Installer Privileges The installation requires administrator privileges on the target machine Fujitsu InfiniBand HCA FDR User Manual 25 OFED Installation Linux Installation 3 3 OFED Installation Please install OFED by the following steps after the OS installation by SVIM Server Installation Manager gt Download OFED from Fujitsu Technology Solutions Web site http support ts fujitsu com gt f you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads Warning This software is the driver package of ConnectX 3 IB HCA card 56Gbps This
14. KCC Certification Korea ln 77 S26361 F4533 L102 Specifications 78 S26361 F4533 L202 Specifications 79 Appendix B 52 25222222a5xz4 cc ska 81 Replacing a full height bracket with a low profile bracket 81 Replacing a Bracket ens 81 Removing the Existing Bracket from the Adapter Card 82 Installing the New Bracket 83 Cer tS eee Te i O A L ee eee ee ae 87 Fujitsu InfiniBand HCA FDR User Manual 5 1 Introduction This manual describes the hardware software and adapter card installation and basic use of the InfiniBand Host Channel Adapter IB HCA HCA card name Product name IB HCA 56Gb 1 port FDR 26361 F4533 E102 L102 IB HCA 56Gb 2 port FDR 26361 F4533 E202 L202 Table 1 Product names It also describes the installation of OFED OpenFabrics Enterprise Distribution for Linux and Win OpenFabrics installation for Windows 1 1 Notational Conventions The following notational conventions are used in this manual AN Caution This symbol points out hazards that can lead to personal injury loss of data or damage to equipment Li This symbol highlights important information and tips gt This symbol refers to a step that you must carry out in order to continue with the procedure italic Commands menu items names of buttons options file names and
15. Link control register aborting 28 Couldn t restore HCA reg x aborting 29 Couldn t restore HCA COMMAND aborting Table 8 Error messages of IB HCA card driver Red Hat Enterprise Linux note1 When a message occurs frequently confirm the following Fujitsu InfiniBand HCA FDR User Manual 67 IB HCA Troubleshooting No Confirmation Judgment Todo contents 1 Is the IB HCA card YES The IB HCA card must be replaced mounted surely When the message is displayed after the IB HCA card has been replaced it is necessary to replace the system server NO Place the IB HCA card again Work after IB HCA card has been exchanged Linux 1 Turn on the server 2 Linux is started sbin Ispci command is executed by the root account and it is confirmed that the IB HCA card is recognized Execution result 10 00 0 InfiniBand Mellanox Technologies MT27500 Family LConnectX 3 Subsystem Mellanox Technologies MT27500 Family LConnectX 31 30 00 0 InfiniBand Mellanox Technologies MT27500 Family L ConnectX 31 Subsystem Mellanox Technologies MT27500 Family LConnectX 31 68 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting IB HCA IB HCA card driver message Windows Server 2008 R2 SP1 Confirm the message of the IB HCA card driver by referring to the event log You refer to the event log using the Event Viewer Event ID Classification
16. Processors amp 1 7 Storage controllers H a System devices Wi System management devices Universal Serial Bus controllers Figure 7 Display of InfiniBand devices before installation of WinOF 2 Double click the InfiniBand device Click Details tab and select Hardware Ids in Property Make sure there are VEN 153B and DEV 1009 in Value which shows the device is an InfiniBand card 3 Log in to Windows as administrator and then double click the file of downloaded WinOF 48 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation 4 Click Next gt after the following window appears MLNX_VPI InstallShield Wizard Welcome to the InstallShield Wizard for MLNX VPI The InstallShield R Wizard will install MLNX VPI on your computer To continue dick Next WARNING This program is protected by copyright law and international treaties Figure 8 InstallShield Wizard Fujitsu InfiniBand HCA FDR User Manual 49 WinOF Installation Windows Installation 5 After the following window appears read License Agreement carefully If you accept tick accept the terms in the license agreement and then click Next gt i MLNX_ PI InstallShield Wizard Ea License Agreement Please read the following license agreement carefully Copyright c 2005 2011 rights reserved Mellanox Technologies All Redistribution and use in source and binar
17. To do Error Level 0x0041 xxx Failed to open Error Channel Adapter 0x0042 xxx Failed to allocate Error Protection Domain 0x0043 xxx Failed to create Error receive Completion Queue 0x0044 xxx Failed to create send Error Completion Queue 0x0045 xxx Failed to create Error Queue Pair The IB HCA card must be replaced 0x0046 xxx Failed to get Queue Please consult Error Pair number technical support ts fujitsu com 0x0047 xxx Failed to create DMA If you purchased the products from Error Memory Region Fujitsu Japan please consult the 0x0048 xxx Failed to create system engineer Error receive descriptor pool 0x0049 xxx Failed to create Error NDIS_PACKET pool to receive indications 0x004A xxx Failed to create Error NDIS_BUFFER pool to receive indications 0x004B xxx Failed to create Error NDIS_PACKET pool to send processing Table 9 source ipoib Fujitsu InfiniBand HCA FDR User Manual 69 IB HCA Troubleshooting Event ID Classification Error Level 0x004C xxx Failed to create Error NDIS_BUFFER pool to send processing 0x004D xxx Failed to allocate Error receive indication array 0x004E xxx Subnet Administrator Error query for port information timed out Make sure the SA is functioning properly Increasing the number of retries and retry timeout adapter parameters may solve the problem 0x004F xxx Subnet Administrator Error failed the query f
18. a low profile one It includes the following sections e Removing a bracket e Installing new bracket 7 2 Replacing a Bracket To replace the bracket you will need the following parts e the new bracket of the proper height e one new square gasket for each of the ports e the 2 screws saved when the bracket is removed e the 2 fiber washers saved when the bracket is removed Fujitsu InfiniBand HCA FDR User Manual 81 Remove the Existing Bracket from the Adapter Card Appendix B 7 3 Removing the Existing Bracket from the Adapter Card Screw LEDs Screw Screw Gasket Gaskets LEDs Screw Figure 21 Removing the bracket 1 Remove the two screws holding the bracket in place 2 The bracket will come loose from the card Warning Be careful not to put stress on the LEDs 3 Save the two screws and the two fiber washers 82 User Manual Fujitsu InfiniBand HCA FDR Appendix B Installing the New Bracket 7 4 Installing the New Bracket 1 Remove the paper to expose the adhesive on the gasket 2 Place the gasket for each port onto the new bracket Make sure to align the gasket correctly with the hole in the bracket 3 Ifthe old gaskets are still on the card remove them before installing the new bracket Make sure that only one gasket is used per port Figure 22 Gasket Installation 4 Place the bracket onto the card until the screw holes line up Warning Do not force the bracket onto the card Yo
19. is a problem peculiar to RHEL6 1 communication by IPoIB may not be able to be performed Fujitsu InfiniBand HCA FDR User Manual 33 OFED Installation Linux Installation Please add the following parameters PREFIX lt value gt value is corresponding to that of NETMASK Example PREFIX 24 3 Only RHEL6 3 RHEL6 3 can set the parameter of VM CONTROLLED as yes However since a different network interface may get the same IP address please add the following parameters In addition in order to check HWADDR please specify the execution result of the ib addr command NM_CONTROLLED yes TYPE InfiniBand HWADDR lt MAC Address gt Example DEVICE ibO TYPE InfiniBand IPADDR 192 168 210 100 NETMASK 255 255 255 0 PREFIX 24 BROADCAST 192 168 210 255 NM_CONTROLLED yes ONBOOT yes HWADDR 80 00 00 48 fe 80 00 00 00 00 00 00 00 02 c9 03 00 fa 11 81 Check the HWADDR is as follows MAC Address is output to the result of executing the command ip addr show ib Example ip addr show ibO 10 ib0O lt BROADCAST MULTICAST UP LOWER_UP gt mtu 65520 qdisc pfifo_fast state UP qlen 1024 link infiniband 80 00 00 48 fe 80 00 00 00 00 00 00 00 02 c9 03 00 fa 11 81 brd OO ff ff ff ff 12 40 1b ff ff 00 00 00 00 00 00 ff ff fF fF inet 192 168 210 100 24 brd 192 168 210 255 scope global ibO 34 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation The parameter
20. path names are written in italic letters in the text lt variable gt Marked variables that must be replaced by current values fixed font System output is written using a fixed font semi bold Commands to be entered through the keyboard are written ina fixed font semi bold fixed font Key symbols Keys are presented according to their representation on the keyboard If capital letters are to be entered explicitly then the Shift key is shown e g SHIFT A for A If two keys need to be pressed at the same time then this is indicated by placing a hyphen between the two key symbols Fujitsu InfiniBand HCA FDR User Manual Target Group Introduction Fujitsu Support Please contact your Fujitsu Technical Support if you require assistance http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult a system engineer 1 2 Target Group This manual is intended for users developers and system administrators responsible for setting up and maintaining switch system platforms using InfiniBand fabrics and networks The switch system platforms must have a management server or client in order for the package to work The manual assumes familiarity with the InfiniBand Architecture Specification 1 3 Documentation Overview PRIMERGY manuals are available in PDF format on the ServerView Suite DVD 2 The ServerView Suite DVD 2 is part of the Server
21. technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 9 source ipoib Fujitsu InfiniBand HCA FDR User Manual 71 IB HCA Troubleshooting Event ID Classification To do Error Level 0x0003 XXX Error 0x0004 mlx4 bus has started in Error non operational mode 0x0005 mlx4 bus has failed to start Error even in non operational mode n Look into the the previous error messages 0x0007 MAP FAcommand failed with Error error 2 n The adapter card is non functional n The IB HCA card must be Most likely a FW problem n replaced Please burn the last FW and ee jitsu com restart the mlx4 bus driver ds iii 0x0008 RUN_FW command failed If you purchased the products Error with error 2 n from Fujitsu Japan please The adapter card is non consult the system engineer functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x0009 QUERY_FW command failed Error with error 2 n The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver Table 10 source mlx4_bus 72 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting IB HCA Event ID Error Level 0x000B Error 0x000C Error Classification QUERY_DEV_CAP command failed with error 2 N The adapter card is non functio
22. the printed side of the adapter card that has the card serial number and the card GUID for InfiniBand protocol Fujitsu Technology Solutions DEUS ELLE EE ETT Ma t A3C40136899 AO FJYWWXSSSSSS QUUD UTA GUID HHHH HHHHHHHHHHHH TA su FUJ CA05954 2101 A1 Figure 3 Card Product Label e g IB HCA 1port FDR 12 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware Safety instructions 2 3 Safety instructions The following safety instructions are also provided in the manual Safety notes and other important information This device meets the relevant safety regulations for IT equipment If you have any questions about whether you can install the server in the intended environment please contact your sales outlet or our customer service team CAUTION e The actions described in this manual shall be performed by technical specialists A technical specialist is a person who is trained to install the server including hardware and software e Repairs to the device that do not relate to CSS failures shall be performed by service personnel Please note that unauthorized inter ference with the system will void the warranty and exempt the manufacturer from all liability e Any failure to observe the guidelines in this manual and any improper repairs could expose the user to risks electric shock energy hazards fire hazards or damage the equipment e Before installing removing internal options to from t
23. value of SACK net ip4 tcp_sack SACK net ipv4 tcp_sack parameter value is set to 0 by installing Mellanox OFED In addition the default configuration value on RHEL native OFED is 1 The difference in the communication control by SACK setup is as follows 1 SACK 0 OFF When one data is divided into multiple packets and it transmits even if only one packet detects an error all the packets are transmitted again 2 SACK 1 ON On the same communication condition as the above processing time is shortened because only the packet in which the error was made transmits again However if this parameter is set to 1 access of the packet by SACK control will occur at random As compared with a setup of SACK 0 a CPU usage rate is high 3 3 4 OFED uninstallation Uninstallation of OFED executes the following commands Please refer to the operation manual of OFED attachment for details rootGlocalhost usr sbin ofed uninstall sh This program will uninstall all MLNX OFED LINUX packages on your machine Do you want to continue Ly N y Reboot the system reboot When you upgrade OFED please install new OFED after uninstallation Lroot localhost Lroot localhost Fujitsu InfiniBand HCA FDR User Manual 35 OFED Installation Linux Installation 3 3 5 OFED installation procedure when applying errata kernel This chapter describes how to install OFED when applying the errata kernel
24. 4 1 2 e15 Execute Installer again and enter y Croot localhost mnt1 mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y Starting MLNX OFED LINUX 1 5 3 3 0 0 installation Installing mInx ofa kernel RPM Preparing HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE mlnx ofa kernel fHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI Installing kmod mlnx ofa kernel RPM 3 Installation finished successfully Configuring etc security limits conf Please reboot your system for the changes to take effect root localhost mntl Unmount mnt directory Croot localhost mnt cd Croot localhost 14 umount mnt root localhost 14 Restart the OS Croot localhost reboot Check the modification of limits conf file Croot localhost 14 tail 2 etc security limits conf soft memlock unlimited hard memlock unlimited root localhost Please re install from step3 when the 2 messages above were not displayed Check the version of OFED root localhost ofed info 28 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation 3 3 2 OFED installation on RHEL6 1 Warning The following is an example of RHEL6 2 Operation may change by the version of OFED Please refer to the operation manual of OF
25. ED attachment for details When especially an environment variable LANG is a Japanese locale a problem may occur Mount iso file and move to mnt Croot localhost J mount ro loop MLNX OFED LINUX 1 5 3 3 0 0 rhel16 2 x86 64 iso mnt root localhost l rootaGlocalhost J cd mnt Execute Installer and enter y rootalocalhost mnt mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y The following messages are displayed Add an option nodeps Delete scsi target utils from the package currently displayed and uninstall a package Please remove OFED RPMs coming from the Distribution Run rpm e allmatches rdma rdma 1 0 14 e16 noarch libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs dapl rds tools ibutils infinipath psm libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad ibsim librdmacm librdmacm utils opensm libs compat opensm libs dapl rds tools infinipath psm opensm libs libipathverbs dapl libibcm libibmad libibumad ibsim ibutils ibutils libs fcoe utils scsi target utils compat openmpi compat openmpi psm fcoe utils scsi target utils Fujitsu InfiniBand HCA FDR User Manual
26. F4533 L202 Specifications Physical Power and Environmental Size 2 71in x 5 6in Voltage 12V 3 3V 68 90mm x 142 25mm Air Flow 200LFM 55 C Typ Power Passive cables 7 94W Active cables 11 34W QSFP 56Gb s __ InfiniBand copper and Maximum Power Passive cables 9 35W Connector optical max power Active cables 13 35W per port 2 0W Temperature 0 C to 55 C Table 16 Specifications 1 for Dual port Card Protocol Support Regulatory InfiniBand IBTA v1 3 plan EMC FCC47CFR part 15 2006 subpart B Auto Negotiation class A 56Gb s 14Gb s ICES 003 2004 Issue 4 class A Jobi ENS 022 1990 A1 2000 A2 200 40Gb s 10Gb s ee per lane or EN 61000 3 2 2000 A2 2005 20Gb s 5Gb s EN61000 3 3 1995 A2 2005 per lane or EN 55024 1998 A1 2001 A2 2003 10Gb s 2 5Gb s standards harmonized under EMC per lane Directive 2004 108 EC Article 6 2 AS NZS 3548 QoS 8 InfiniBand Virtual Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RDMA Support Yes All Ports RoHS RoHS R6 Data Rate Up to 56 Gb s FDR InfiniBand PCI Express SERDES 8 0GT s 8 lanes Table 17 Specifications 2 for Dual port Card Fujitsu InfiniBand HCA FDR User Manual 79 7 Appendix B 7 1 Replacing a full height bracket with a low profile bracket This section provides instructions on how to remove the full height bracket of a standard IB HCA card and replace it with
27. ID number the port of InfiniBand Switch was disconnected in the subnet How to solve the problem The following changes are required Please contact the repair center 62 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting Checking the Log File of OpenSM Linux only 1 IB cable You need to change the IB cable You specify the IB cables using the following procedure 1 You specify the relevant InfiniBand Switch from the LID infor mation in the error messages For example Jul 31 22 56 46 264171 42E93940 0x02 gt osm report notice Reporting Generic Notice type 1 num 128 Link state change from LID 32 GUID fe80 2 c902 40 c790 In this case LID 32 s InfiniBand Switch is corresponding 2 The target port is the one that removed the port number from the original ibnetdiscover command output when the system was set up normally 2 InfiniBand Switch If you have changed the above IB cable and still receive the same messages you need to change the IB Switch Fujitsu InfiniBand HCA FDR User Manual 63 IB HCA Troubleshooting 5 2 IBHCA You can confirm the place and time the error occurred by referring to the following items on a system server No Confirmation item 1 IB HCA card s physical link and data activity LED 2 IB HCA card driver message IB HCA card s physical link and data activity LED You can confirm the place where the error occurred using the follo
28. M 3 5 1 Functions of OpenSM OpenSM is one of Subnet Manager distributed with RedHat This chapter describes how to configure which servers becomes the master SM Please set up the server which will be the master using the following reference Regarding the other settings of OpenSM please refer to the command help of opensmd Basically a server with the lowest GUID will be the master SM within a subnet if you intend to choose the master SM you can configure the priority which range is from 0 to 15 0 is the lowest and 15 is the highest Example for RHEL5 etc init d opensmd start p 15 44 User Manual Fujitsu InfiniBand HCA FDR Linux Installation Subnet Manager 3 5 2 Confirmation that OpenSM is Running Enter the following command to confirm that OpenSM is running etc init d opensmd status If OpenSM is running properly the result of this command displays running Enter the following command to run OpenSM if the result displays stopped etc init d opensmd start 3 5 3 Confirmation of the master SM Enter ibstat command on the system server then Base LID and SM LID are displayed The IB HCA card on which these two LIDs are the same is the master SM Fujitsu InfiniBand HCA FDR User Manual 45 Subnet Manager Linux Installation 46 User Manual Fujitsu InfiniBand HCA FDR 4 Windows Installation 4 1 Overview This chapter describes how to install a single host machine with Fujitsu Inf
29. View Suite supplied with every server If you no longer have the ServerView Suite DVDs you can obtain the relevant current version using the order number U15000 C289 the order number for Fujitsu Japan please refer to the configurator of the server http jp fujitsu com platform server primergy system The PDF files of the manuals can also be downloaded free of charge from the internet The overview page showing the online documentation available on the internet can be found using the URL for EMEA market http manuals ts fujitsu com The PRIMERGY server documentation can be accessed using the Industry standard servers navigation option If you purchased the products from Fujitsu Japan please use the URL http jp fujitsu com platform server primergy manual 8 User Manual Fujitsu InfiniBand HCA FDR Introduction Overview 1 4 Overview This document is a user manual for InfiniBand Host Channel Adapters IB HCA card in rack servers The card described in this manual has the following features Single and Dual Port 56Gb s IB HCA card Low Profile PCle form factor with exchangeable Full Height FH or Low Profile LP bracket Based on the IB controller chip Mellanox ConnectX 3 PCle interface PCle x8 bus interface PCle 3 0 8GT s Auto negotiates to x8 x4 x2 or x1 x4 x2 or x1 will decrease the performance IBTA Specification 1 3 compliant plan Hardware based congestion c
30. al 31 OFED Installation Linux Installation log_mtts_per_seg Set the minimum unit segment of the address translation table which a driver manages Default is 3 max is 7 A setup of the amount of memory which can be registered for RDMA can be changed Calculation of the amount of memory is as follows Set the twice of main memory 4KB x 2 log num mtt log num per seg The following table shows the recommended values of log num mtt and log num per seg A value changes with main memory capacity Main memory 16GB 32GB le4GB 428GB 256GB 512GB 1TB capacity log num mit 19 20 21 22 23 24 24 log mtts per seg 4 4 4 4 4 4 5 How to change QP number of HCA log num qp Set the maximum number of QP number per HCA default is 18 This parameter creates 256K QP s by the default value 18 The maximum which can set up Mellanox HCA is 24 and can create 16M QP s However because it consumes the available memory by setting a larger value please do not set the value unnecessarily large By changing parameter values into 21 from 18 the memory which can be used decreases by 2 GB The recommended value of parameter values is not decided in order to change with system configurations the number of nodes Please calculate QP number by the following formulas and if it is over the default value change a parameter setting Number of MPI processes per node 2 x number of nodes x MPI implemen tati
31. as follows It is shown that the check mark is contained in the target package set RHEL RHEL HPC Required package Package name Server Compute Node e g RHEL6 4 V V gcc gcc 4 4 7 3 el6 x86 64 v v glib glib2 2 22 5 7 el6 x86_64 v glib devel glib2 devel 2 22 5 7 el6 x86_64 v v glibc glibc 2 12 1 107 e16 x86 64 v v glibc devel glibc devel 2 12 1 107 el6 x86_64 v v glibc devel 32bit glibc devel 2 12 1 107 e16 i686 A zlib devel zlib devel 1 2 3 29 e16 x86 64 v libstdc devel libstdc devel 4 4 7 3 el6 x86_64 v v gdb gdb 7 2 60 el6 x86_64 v rpm build rpm build 4 8 0 32 el6 x86_64 v redhat rpm config redhat rpm config 9 0 3 42 el6 noarch v v libgfortran libgfortran 4 4 7 3 el6 x86_64 Table 5 Required packages 42 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation RHEL RHEL HPC Required package Package name Server Compute Node e g RHEL6 4 v v gcc g77 gcc gfortran 4 4 7 3 e16 x86 64 v v Tcl tcl 8 5 7 6 el6 x86 64 V tcl devel tcl devel 8 5 7 6 el6 x86_64 V V Tk tk 8 5 7 5 el6 x86_64 v gcc c gcc c 4 4 7 3 el6 x86_64 v v genisoimage genisoimage 1 1 9 12 el6 x86_64 Table 5 Required packages Getting insufficient packages Please check the installation situation of the above mentioned packages after installing RHEL HPC Compute Node Please get the insufficient packages from Red Hat Net
32. ated sockets of the rack s internal power supply with tested and approved power cords Ensure that the device is connected to a properly grounded power outlet close to the device 14 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware Safety instructions CAUTION Ensure that the power sockets on the device and the properly grounded power outlets are freely accessible The On Off button or the main power switch if present does not isolate the device from the mains power supply To disconnect it completely from the mains power supply unplug all network power plugs from the properly grounded power outlets Always connect the server and the attached peripherals to the same power circuit Otherwise you run the risk of losing data if for example the server is still running but a peripheral device e g memory subsystem fails during a power outage Data cables must be adequately shielded Route the cables in such a way that they do not create a potential hazard make sure no one can trip over them and that they cannot be damaged When connecting the server refer to the relevant instructions in this manual Never connect or disconnect data transmission lines during a storm risk of lightning hazard Make sure that no objects e g jewelry paperclips etc or liquids can get inside the server risk of electric shock short circuit In emergencies e g damaged casing controls or cables penetration of liq
33. ble 13 Dual port card certification status 6 1 1 FCC Statements USA Class A Statements e 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance could void the user s authority to operate the equipment Fujitsu InfiniBand HCA FDR User Manual 75 A 1 EMC Certification Statements Appendix A e 15 105 a Statement This equipment has been tested and found to comply with the limits i for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense 6 1 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures 6 1 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num rique de la classe
34. es into the trash can They must be disposed of in accordance with local regulations concerning special waste e Replace the lithium battery on the system board in accordance with the instructions in the technical manual for the system board e All batteries containing pollutants are marked with a symbol a crossed out garbage can In addition the marking is provided with the chemical symbol of the heavy metal decisive for the classification as a pollutant Cd Cadmium Hg Mercury Pb Lead Disposal Disposal of this equipment should be in accordance to all national laws and regulations Modules with Electrostatic Sensitive Devices Modules with electrostatic sensitive devices are identified by the following sticker Fujitsu InfiniBand HCA FDR User Manual 17 Safety instructions Adapter Card Hardware A Ate Figure 4 ESD label When you handle components fitted with ESDs you must always observe the following points Switch off the system and remove the power plugs from the power outlets before installing or removing components with ESDs You must always discharge static build up e g by touching a grounded object before working with such components Any devices or tools that are used must be free of electrostatic charge Wear a suitable grounding cable that connects you to the external chassis of the system unit Always hold components with ESDs at the edges or at the points marked green touc
35. etwork Direct 1 Fujitsu InfiniBand HCA FDR User Manual 57 WinOF Installation Windows Installation 17 In order to check if Network Direct is enabled properly execute the same command again The message Installing IBAL provider already installed is shown if it is enabled properly dminist rator Figure 18 Installation of Network Direct 2 4 3 2 WinOF installation on Windows Server 2012 Download OFED from Fujitsu Technology Solutions Web site http support ts fujitsu com If you purchased the products from Fujitsu Japan you download it from below http jp fujitsu com platform server primergy downloads 1 After inserting InfiniBand cards make sure that InfiniBand device is shown in the device manager 58 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation File Action View Help e9 m 5 Be E RS 4 iij n300s8 06 b 2 Batteries p jS Computer b cq Disk drives p K Display adapters b ef DVD CD ROM drives b ia Human Interface Devices b c IDE ATA ATAPI controllers p amp Keyboards b A Mice and other pointing devices b amp Monitors 4 amp Network adapters La Intel R 1350 Gigabit Network Connection La Intel R 1350 Gigabit Network Connection 2 Mellanox IPoIB Adapter Microsoft Kernel Debug Network Adapter p 9 Ports COM amp LPT b ab Print queues gt BB Processors b lt gt Storage controllers p pl System devices p gill Sy
36. h a residential use type Fujitsu InfiniBand HCA FDR User Manual 77 26361 F4533 L102 Specifications Appendix A 6 2 S26361 F4533 L102 Specifications Physical Power and Environmental Size 2 12in x 5 6in Voltage 12V 3 3V 53 95mm x 142 24mm Air Flow 200LFM 55 C Typ Power Passive cables 6 71W Active cables 8 41W QSFP 56Gb s InfiniBand copper and Maximum Power Passive cables 8 13W Connector optical max power Active cables 10 13W per port 2 0W Temperature 0 C to 55 C Table 14 Specifications 1 for Single port Card Protocol Support Regulatory InfiniBand IBTA v1 3 plan EMC FCC 47 CFR part 15 2006 subpart B Auto Negotiation class A 56Gb s 14Gb s ICES 003 2004 Issue 4 class A per anej or DE o 40Gb s 10Gb s ee pis wie Gb EN 61000 3 2 2000 A2 2005 S 5Gb s EN61000 3 3 1995 A2 2005 per lane or EN 55024 1998 A1 2001 A2 2003 10Gb s 2 5Gb s standards harmonized under EMC per lane Directive 2004 108 EC Article 6 2 AS NZS 3548 QoS 8 InfiniBand Virtual Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RDMA Support Yes All Ports RoHS RoHS R6 Data Rate Up to 56 Gb s FDR InfiniBand PCI Express SERDES Q 8 0GT s 8 lanes Table 15 Specifications 2 for Single port Card 78 User Manual Fujitsu InfiniBand HCA FDR Appendix A 26361 F4533 L202 Specifications 6 3 S26361
37. h points Do not touch any connectors or conduction paths on an ESD Place all the components on a pad which is free of electrostatic charge i For a detailed description of how to handle ESD components see the relevant European or international standards EN 61340 5 1 ANSI ESD 20 20 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware ENERGY STAR Other important information e Keep this operating manual and the other documentation such as the Technical Manual DVD close to the device All documentation must be included if the equipment is passed on to a third party 2 4 ENERGY STAR eX Products that have been certified compliant with ENERGY STAR ae and identified as such are in full compliance with the specification MZ at shipping Note that energy consumption can be affected by software that is installed or any changes that are made to the BIOS or energy options subsequently In such cases the properties guaranteed by ENERGY STAR can no longer be assured The ServerView Operations Manager user guide contains instructions for reading out measurement values including those relating to current energy consumption and air temperatures Either the Performance Monitor or the Task Manager can be used to read out CPU utilization levels A specific model of this product is recognized in United States EPA and registered The model based on this is not sold in the Japanese market 2 5 CE conformity
38. he errata kernel when OFED is already installed uos m ob E ES UR RR pak RORCROR DS x EO ue 36 3 3 5 2 How to install OFED when the errata kernel is already cp 2 565 GS he SRE RGR ER ESE DS oR A Oa 41 3 3 6 OFED installation on RHEL HPC Compute Node 42 3 4 Updating Firmware after Installation 44 3 5 Subnet Manager 2 te acento ee see how ewe x 44 3 5 1 Functions of OpenSM i usa xk Re REA wD 44 3 5 2 Confirmation that OpenSM is Running 45 35 9 Confirmation of the master SM sn 45 4 Windows Installation 47 4 1 UBFUIBU 22546645 ROO SCONTO UR ORAL RR 47 4 2 Software Requirements 47 4 3 WinOF Installation rns 48 4 3 1 WinOF installation on Windows Server 2008 R2 SP1 48 4 3 2 WinOF installation on Windows Server 2012 58 4 3 3 WinOF Upgrade Notes llle 59 5 Troubleshooting o 26 xa kx RS Rx mx ORO Bowes 61 5 1 Checking the Log File of OpenSM Linux only 61 5 2 3 7 eI be ee TUTTI 64 4 User Manual Fujitsu InfiniBand HCA FDR Contents 6 1 6 1 1 6 1 2 6 1 3 6 1 4 6 1 5 6 2 6 3 7 1 7 2 7 3 7 4 Appendix A ec eae a a Skee me SG 75 A 1 EMC Certification Statements 75 FCC Statements USA 220 75 EN Statements Europe lll lle 76 ICES Statements Canada 76 VCCI Statements Japan lll 77
39. he server turn off the server all peripheral devices and any other connected devices Also unplug all power cords from the power outlet Failure to do so can cause electric shock Before starting up CAUTION e During installation and before operating the device observe the instructions on environmental conditions for your device e Ifthe device is brought in from a cold environment condensation may form both inside and on the outside of the device Wait until the device has acclimatized to room temperature and is absolutely dry before starting it up Material damage may be caused to the device if this requirement is not observed e Transport the device only in the original packaging or in packaging that protects it from knocks and jolts Fujitsu InfiniBand HCA FDR User Manual 13 Safety instructions Adapter Card Hardware Installation and operation CAUTION AN This unit should not be operated in ambient temperatures above 35 C If the unit is integrated into an installation that draws power from an industrial power supply network with an IEC309 connector the power supply s fuse protection must comply with the requirements for non industrial power supply networks for type A connectors The unit automatically adjusts itself to a mains voltage in a range of 100 V 240 V Ensure that the local mains voltage lies within these limits This device must only be connected to properly grounded power outlets or insul
40. ic Local System This servic Manual Local System Provides pr Manual Local System emote Access Aut Creates a Manual Local System Remote Access Co Manages di Started Manual Local System C Remote Access Qu Removes v Manual Local Service QRemoteDesktop C Remote De Manual Local System v Extended A Standard Figure 15 Configuration of OpenSM 12 Change Startup type from Manual to Automatic Delay Start 13 Click the Start button to start the OpenSM service 56 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation 14 Open the Network Connections Window to check if IPoIB networks are linked up Disable the unused IPoIB adapters F network Connections mem 2 gt Ir Control Panel Network and Internet Network Connections be ae Search Network Cor Organize v A jy Private A jy Enterprise Y L Local Area Connection 4 _ Unidentified network Network Disabled 7 Intel R 82575EB Gigabit Backplane i Intel R 82575EB Gigabit Backplane 3 7 Mellanox IPoIB Adapter we Application Identifying 7 Mellanox IPoIB Adapter 2 Figure 16 Network status 15 Since no IP address is assigned to this new IPoIB adapter assign an IP address to it 16 Run the command prompt as the Administrator privilege and execute the following command to enable Network Direct ndinstall i Figure 17 Installation of N
41. ing with the Linux Distribution were removed due to dependencies please reinstall them if required scsi target utils root localhost mntl 30 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation 5 Unmount mnt directory Croot localhost mnt cd Croot localhost J umount mnt rootaGlocalhost 6 Restart the OS Croot localhost reboot 7 Check the modification of limits conf file Croot localhost 14 tail 2 etc security limits conf soft memlock unlimited hard memlock unlimited root localhost l Please re install from step3 when the 2 messages above were not displayed 8 Check the version of OFED root localhost 1 ofed info 3 3 3 Post installation Notes This section describes how to set mix4 parameters Please add the following mix4 parameters to the file After the parameter has been changed you will need to reboot In the case of the combination of RHEL5 or 6 system MellanoxOFED please add line to etc modprobe d mlx4 conf If there is no mlx4 conf please create the file In addition when mix4 conf has not been created a setup by the default value of each parameter is performed How to set mlx4 parameters options mlx4 core lt parameter gt lt value gt How to change amount of memory which can be registered for RDMA log num mtt Set the number of page table entries which HCA uses Default is 20 max is 24 Fujitsu InfiniBand HCA FDR User Manu
42. iniBand hardware installed A server can be properly installed with all required InfiniBand drivers and software during Windows Server installation or afterwards by using the Win OpenFabrics installation or OpenSM setup Please note that only Windows Server 2008 R2 and Windows Server 2012 are supported 4 2 Software Requirements Required Disk Space for Windows Installation 100 MB Operating System Windows Server 2008 R2 Windows Server 2012 Installer Privileges Installation requires administrator privileges on the target machine Fujitsu InfiniBand HCA FDR User Manual 47 WinOF Installation Windows Installation 4 3 WinOF Installation Please install WinOF by the following steps after the OS installation Download OFED from Fujitsu Technology Solutions Web site http support ts fujitsu com If you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads 4 3 4 WinOF installation on Windows Server 2008 R2 SP1 1 After inserting InfiniBand cards make sure that InfiniBand device is shown in Device Manager A Device Manager File Action View Help Coe Shall bos D E E E 8 56 m 5 fp JE Computer zs Disk drives VK Display adapters ox Human Interface Devices lt gt Keyboards n Mice and other pointing devices Ki Monitors amp Network adapters El jj Other devices Qa Infiniband Controller 1 7 Ports COM amp LPT t
43. initialize shared receive queue table aborting The IB HCA card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult your system engineer Table 8 Error messages of IB HCA card driver Red Hat Enterprise Linux 66 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting IB HCA No Classification To do 16 Failed to initialize queue pair table The IB HCA card must be replaced aborting Please consult 17 Failed to initialize multicast group technical support ts fujitsu com table aborting 18 Failed to reset HCA aborting If you purchased the products from 19 Failed to init command interface Fujitsu Japan please consult your aborting system engineer 20 Only d UAR pages need more than 128 note1 21 Profile requires Ox llx bytes won t fit in Ox llx bytes of context memory 22 Couldn t allocate memory to save HCA PCI header aborting 23 Couldn t save HCA PCI header aborting 24 Couldn t map HCA reset register The IB HCA card must be replaced aborting Please consult 25 PCI device did not come back after technical supportQ ts fujitsu com reset aborting 26 couldn restore HCA Ped Express If you purchased the products from Device Control register aborting Fujitsu Japan please consult your 27 Couldn t restore HCA PCI Express system engineer
44. k Direct Provider by the ndinstall command after removing Network Direct Provider 60 User Manual Fujitsu InfiniBand HCA FDR 5 Troubleshooting This chapter describes what to do if an InfiniBand Switch is not working properly Please contact your Fujitsu Technical Support if you require assistance http support ts fujitsu com com support index html If you purchased the products from Fujitsu Japan please consult your system engineer 5 1 Checking the Log File of OpenSM Linux only Check the log file in the server var log opensm log that started up the Subnet Manager OpenSM as Master gt Check list Are any of the following logs contained in the var log opensm log SM port is down Removed port with GUID GUID number PRIx64 LID range LID number LID number of node Reporting Generic Notice type 1 num 128 Link state change from LID LID number GUID GUID number SM port is down SubnetManager s master port was disconnected How to solve the problem The following changes are required Please contact the repair center 1 IBHCA You need to change the IB HCA that ran as the Master of the SubnetManager 2 System Server If you have changed the IB HCA and you still see the same messages you need to change the system server that installed the IB HCA 3 InfiniBand Switch If you have changed the above system server and still receive the same messages you need to change InfiniBand Switch
45. l amount Make sure that the weight of the cable is supported on its own and is not hanging from IB HCA card Fujitsu InfiniBand HCA FDR User Manual 23 Adapter Card Installation Adapter Card Hardware 2 7 3 2 Inserting a Cable into the Adapter Card 1 Support the weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack 2 Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card 3 Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card 4 Make sure that the connector locks in place Figure 6 Connector orientation 2 7 8 5 Removing a Cable from the Adapter Card 1 Pull on the latch release mechanism to unlatch the connector and pull the connector out of the cage 2 Do not apply torque to the connector when removing it from the adapter card 3 Remove any cable supports that were used to support the cable s weight 24 User Manual Fujitsu InfiniBand HCA FDR 3 Linux Installation 3 1 Overview This chapter describes how to install a single host machine with Fujitsu InfiniBand hardware installed A server can be properly installed with all required InfiniBand drivers and software during Red
46. m libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel dapl dapl devel dapl utils perftest rds tools ibutils infiniband diags qperf libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel compat dapl utils dapl dapl devel dapl utils perftest rds tools infiniband diags qperf ofed docs openib opensm libs openmpi openmpi libs ibutils ibutils libs mpitests openmpi ofed docs 1 4 1 2 e15 Some RPMs may depend on the RPMs above Please uninstall them manually Uninstall the Packages Croot localhost mnt rpm e nodeps allmatches openmpi libcxgb4 ofed docs 1 4 1 2 e15 openib 1 4 1 6 e15 libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel dapl dapl devel dapl utils perftest rds tools ibutils infiniband diags qperf libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibcm libibumad libibmad librdmacm librdmacm utils libsdp opensm libs compat dapl compat dapl devel compat dapl utils dapl dapl Fujitsu InfiniBand HCA FDR User Manual 27 OFED Installation Linux Installation devel dapl utils perftest rds tools infiniband diags qperf ofed docs openib opensm libs openmpi openmpi libs ibutils ibutils libs mpitests openmpi ofed docs 1
47. n the adapter card will not provide optimum data transfer 2 7 3 Cables The QSFP port can be connected to InfiniBand passive copper and active optical cables 2 7 3 1 Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable press the connector into the port receptacle until the connector is firmly seated The GREEN LED indicator will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the YELLOW LED will come on When data is being transferred the yellow led will flash Warning When installing cables make sure that the latches engage Warning Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card Care should be taken not to impede the air exhaust flow through the ventilation holes Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Warning Cables especially long copper cables can weigh a substantia
48. nal n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver QUERY_ADAPTER command failed with error 2 N The adapter card is non functional n Most likely a FW problem n Please burn the last FW and restart the mlx4_bus driver 0x000D Error Too few QPs were requested requested 2 reserved for FW 3 n The adapter card is non functional n Please increase the Registry LogNumQp parameter under HKLM System CurrentCon trolSet Services m x4_bus Parameters Ox001 1 Error Failed to move location string 262 status 3 0x0012 Error WdfDeviceAllocAndQue ryProperty failed status 962 Table 10 source mlx4 bus To do The IB HCA card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Fujitsu InfiniBand HCA FDR User Manual 73 IB HCA Troubleshooting Event ID Classification To do Error Level 0x0003 XXX The IB HCA card must be replaced Error Please consult technical support ts fujitsu com or your system engineer Table 11 source ibbus Work after IB HCA card has been exchanged Windows Server 2008 R2 SP1 1 Turn on the server 2 Make sure that the InfiniBand devices are recognized properly in Device Manager File Action View Help es zz B m Wa Server Manager WIN BVGS47H7FI Ez g R
49. nning and a valid physical connection between nodes exists If the green LED is flashing it indicates a problem with the physical link The yellow LED when lit indicates a valid data activity link this is the logical link The yellow LED lights up when the network is discovered over the physical link A valid data activity link without data transfer is indicated by a constant yellow LED A valid data activity link with data transfer is indicated by a flashing yellow LED If the LEDs are not active either the physical link or the logical link or both connections have not been established Port LED Name Number Port 1 Physical Link Green Constant on indicates a good PORT 1 physical link Flashing indicates a problem with the physical link Data Activity Yellow Flashing indicates data transfer Constant on indicates no data transfer Port 2 Physical Link Green Constant on indicates a good physical link Flashing indicates a problem with the physical link Data Activity Yellow Flashing indicates data transfer Constant on indicates no data transfer Table 3 LEDs port association e g IB HCA 2port FDR Fujitsu InfiniBand HCA FDR User Manual 21 Adapter Card Installation Adapter Card Hardware 2 7 Adapter Card Installation 2 7 1 Hardware and Software Requirements Before installing the IB HCA card please make sure that the system meets the hardware and software requirements lis
50. oles E aii Features El m Diagnostics Rl fa Event Viewer E Ny Performance A Device Manager Jik Configuration 83 Storage d Eg Computer Eg Disk drives fl Display adapters DVD CD ROM drives EM Human Interface Devices InfiniBand Channel Adapters E Mellanox ConnectX Virtual Channel Adapter n Mice and other pointing devices KK Monitors Network adapters Intel R 82575E8 Gigabit Backplane Connection amp Intel R 82575EB Gigabit Backplane Connection 2 amp Intel R 82575EB Gigabit Backplane Connection m 82575EB Gigabit Backplane Connection E Medie Lu Mellanox IPoIB Adapter 2 ay Mellanox IPoIB Adapter 3 Lu sel IPoIB Adapter 4 J Processors lt gt Storage controllers qm System devices Universal Serial Bus controllers DENAN H E Figure 20 Device manager 74 User Manual Fujitsu InfiniBand HCA FDR 6 Appendix A 6 1 A 1 EMC Certification Statements HCACard FCC VCCI EN ICES CE CB cTUV KCC C ccc GOST P N us TICK R IB HCA YES YES YES YES YES YES YES YES YES exemp YES 56Gb tion 1 port FDR Table 12 Single port card certification status HCACard FCC VCCI EN ICES CE CB cTUV KCC C ccc GOST P N us TICK R IB HCA YES YES YES YES YES YES YES YES YES exemp YES 56Gb tion 2 port FDR Ta
51. ommand to create the ISO image Enter y during the command execution The new ISO image is created under tmp directory We recommend moving the created ISO image to the other directory rootGRX200S7 1 J work mlnx add kernel support sh i MLNX OFED LINUX 1 5 3 3 1 0 rhe16 2 x86 64 iso Note This program will create MLNX OFED LINUX ISO for rhel6 2 under tmp directory All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue Ly N y See log file tmp mlnx ofed iso 9510 10g Building OFED RPMs Please wait Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 3 3 1 0 rhel6 2 x86 64 iso The option of add kernel support sh has been changed OFED1 5 3 1 5 3 4 0 42 or higher OFED2 0 2 0 or higher mInx add kernel support sh mlnx ofed path to MLNX OFED directory make iso Unmount work rootGRX200S7 1 1 umount work rootGRX200S7 1 14 38 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation 4 OFED installation with the created ISO image 1 Install OFED with the created ISO image Enter y during the installation rootGRX200S7 1 mount ro loop tmp MLNX OFED LINUX 1 5 3 3 1 0 rhe16 2 x86 64 iso mnt rootGRX200S7 1 I cd mnt root RX200S7 1 mnt mlnxofedinstall without 32bit without fw update This program will install the MLNX OFED LINUX package on your machine Note that all o
52. on dependent constant Number of MPI processes per node number of logical cores Number of nodes Number of nodes in the group to communicate with each other MPI implementation dependent constant FJ MPI 3 Intel MPI 2 Other MPI 2 32 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation Set value is the value of a power of 2 QP number 26 2144 52 4288 104 8576 209 7152 419 4304 838 8608 1677 7216 256K 512K 1M 2M 4M 8M 16M log num qp 18 19 20 21 22 23 24 configuration configuration file gt etc modprobe d mlx4 conf lt configuration parameters gt log_num_mtt A log_mtts_per_seg B log_num_qp C lt configuration example gt options mlx4_core log_num_mtt A log mtts per seg B log_num_qp C IPoIB adapters parameter limitation This section describes the limitation of OS RHEL6 system and IPoIB param eters Please note that the setting is different depending on the OS version Although RHELS5 system also sets up IPoIB by an ifcfg ib file there are no following limitations 1 RHEL6 0 to 6 2 If you create or edit the ifcfg ib files to configure an IPoIB the parameter NM CONTROLLED should be set to no Example etc sysconfig network scripts ifcfg ibO DEVICE ib0 IPADDR 192 168 210 100 NETMASK 255 255 255 0 BROADCAST 192 168 210 255 NM_CONTROLLED no ONBOOT yes 2 Only RHEL6 1 Since there
53. ontrol Connectivity Interoperable with InfiniBand switches QSFP connectors 1m 3m 56Gb s of passive copper cable Fujitsu InfiniBand HCA FDR User Manual 9 Product Prerequisites Introduction 1 5 Product Prerequisites System Class Subsegment Systems Max No Bracket LP FH All round Server TX300 87 88 2 FH Sub entry RX200 S7 S8 2 LP RX300 S7 S8 2 LP Entry Rack Server RX350 S7 S8 2 FH Midrange RX500 S7 2 FH High end PQ2000 24 LP FH Series CX250 81 82 1 LP Cloud Server CX270 S1 S2 1 LP LP Low Profile bracket FH Full Height bracket 10 User Manual Fujitsu InfiniBand HCA FDR 2 2 1 Adapter Card Hardware Adapter Cards Covered in this Manual The following table lists the adapter cards described in this manual HCA card name PCI Express Data Trans Bracket type RoHS SERDES Speed mission Rate port IB HCA 56Gb PCIe Gen3 8 0 GT s 56Gb s Low profile and R 6 Single port FDR Full height IB HCA 56Gb PCle Gen3 8 0 GT s 56Gb s Low profile and R 6 Dual port FDR Full height Table 2 Single and Dual Port Card a Figure 1 Single Port Card Component Side Fujitsu InfiniBand HCA FDR User Manual 11 GUID and Serial Number on Adapter Cards Adapter Card Hardware Figure 2 Dual Port Card Component Side m 2 2 Finding the GUID and Serial Number on the Adapter Cards All adapter cards have a label on
54. or port information Make sure the SA is functioning properly and compatible 0x0050 xxx Subnet Administrator Error query for port information failed 0x0055 xxx Subnet Administrator Error failed query for broadcast group information 0x0056 xxx Subnet Administrator Error failed request to joining broadcast group 0x0057 xxx The local port rate is Error too slow for the existing broadcast MC group 0x0058 xxx Incorrect value or Error non existing registry for the required IPoIB parameter 3 overriding it by default value 4 To do The IB HCA card must be replaced Please consult technical support Q ts fujitsu com If you purchased the products from Fujitsu Japan please consult the system engineer Table 9 source ipoib 70 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting IB HCA Event ID Classification Error Level 0x005B xxx Pkey index not found Error for partition change switch pkey configuration 0x005C xxx Connected Mode Error failed to initialize disabled Interface will use default UD QP transport 0x005E xxx SetDeviceRegistra Error tionAttributes failed 0x005F xxx SetAdapterRegistra Error tionAttributes failed 0x0060 xxx SetOffloadAttributes Error failed 0x0061 xxx ipoib create adapter Error failed 0x0062 xxx ipoib start adapter Error failed To do The IB HCA card must be replaced Please consult
55. ow turned off Fujitsu InfiniBand HCA FDR User Manual 65 IB HCA Troubleshooting IB HCA card driver message Linux Confirm the error message of the IB HCA card driver by referring to the val log messages og file You refer to the log file using the text editor etc No Classification To do 1 Installed FW has unsupported command interface revision d Installed FW version is d d 03d This driver version supports only revisions 96d to 96d This driver version supports only revisions 96d to d Internal error detected buf 02x 08x Failed to initialize user access region table aborting The IB HCA card must be replaced Please consult technical support ts fujitsu com If you purchased the products from Fujitsu Japan please consult your system engineer Couldn t map EQ doorbell for EQN 0x 06x Couldn t map interrupt clear register aborting Couldn t allocate FW area aborting Failed to map MCG context memory aborting note1 Failed to allocate driver access region aborting 10 Failed to initialize memory region table aborting 11 Failed to initialize event queue table aborting 12 Failed to switch to event driven firmware commands aborting 13 NOP command failed to generate interrupt IRQ d aborting 14 Failed to initialize completion queue table aborting 15 Failed to
56. package driver does not support ConnectX 2 IB HCA card 40Gbps When OFED downloaded from the above is being used please don t apply the RHEL native OFED at the time of RHEL update 3 3 4 OFED installation on RHEL5 Warning The following is an example of RHEL5 8 Operation may change by the version of OFED Please refer to the operation manual of OFED attachment for details When especially an environment variable LANG is a Japanese locale a problem may occur 1 Mount iso file and move to mnt In the following example the OFED is MLNX OFED LINUX 1 5 3 3 3 0 0 rhel5 8 x86_64 iso Croot localhost J mount ro loop MLNX OFED LINUX 1 5 3 3 0 0 rhe15 8 x86 64 iso mnt root localhost 14 cd mnt root localhost mntI 2 Execute Installer and enter y 26 User Manual Fujitsu InfiniBand HCA FDR Linux Installation OFED Installation Croot localhost J mlnxofedinstall without 32bit wi thout fw update This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue L y N1 y 3 The following messages are displayed Add an option nodeps and uninstall the package Please remove OFED RPMs coming from the Distribution Run rpm e allmatches openmpi libcxgb4 ofed docs 1 4 1 2 e15 openib 1 4 1 6 e15 libibverbs libibverbs utils libmthca libmlx4 libcxgb3 libnes libipathverbs libibc
57. removing internal options to from the server turn off the server all peripheral devices and any other connected devices Also unplug all power cords from the outlet Failure to do so can cause electric shock Do not damage or modify internal cables or devices Doing so may cause a device failure fire or electric shock Devices inside the server remain hot after shutdown Wait for a while after shutdown before installing or removing internal options The circuit boards and soldered parts of internal options are exposed and can be damaged by static electricity Before handling them first touch a metal part of the server to discharge static electricity from your body Do not touch the circuitry on boards or soldered parts Hold the metallic areas or the edges of the circuit boards 16 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware Safety instructions CAUTION e Install the screw removed during installation detaching Internal Options in former device position To use a screw of the different kind causes a breakdown of equipment e The installation indicated on this note is sometimes changed to the kind of possible options without notice Batteries CAUTION e Incorrect replacement of batteries may lead to a risk of explosion The batteries may only be replaced with identical batteries or with a type recommended by the manufacturer see the technical manual for the system board e Do not throw batteri
58. rly in Device Manager oa Device Manager File Action View Help e amp t Hs EH E X Network adapters amp Intel R 1350 Gigabit Network Connection Intel R 1350 Gigabit Network Connection 2 X Mellanox IPoIB Adapter amp Mellanox IPoIB Adapter 2 gt Other devices Ports COM amp LPT Processors lt gt Storage controllers 1l System devices qm ACPI Fixed Feature Button 1 ACPI Module Device 1 ACPI Module Device 1 Composite Bus Enumerator 1 Direct memory access controller 7 Generic Bus 1 High precision event timer JE Intel R 82801 PCI Bridge 244E qE LSI RAID Virtual Device JE Mellanox Connects Virtual Channel Adapter JE Mellanox MT27500 4099 Connectx 3 VPI Network Adapter JE Microsoft ACPI Compliant Power Meter Device 1 Microsoft ACPI Compliant System qm Microsoft Generic IPMI Compliant Device 1 Microsoft System Management BIOS Driver qm Microsoft Virtual Drive Enumerator Driver 1 Numeric data processor qm PCI bus qm PCI bus 2M PCI bus _ e Figure 14 InfiniBand devices Fujitsu InfiniBand HCA FDR User Manual 55 WinOF Installation Windows Installation 11 After WinOF installation OpenSM should be set up OpenSM is an InfiniBand compliant Subnet Manager SM At least one OpenSM is present on the InfiniBand network OpenSM has a role to initialize and configure an InfiniBand net
59. stem management devices b 8 Universal Serial Bus controllers Figure 19 InfiniBand device before WinOF installation Windows Server 2012 Copy the downloaded WinOF file to a server Log in to Windows as administrator and then double click the file of downloaded WinOF 4 3 3 WinOF Upgrade Notes 1 WinOF 4 40 Notes A difference may occur in Network Direct Provider which is installed as compared with new installation of WinOF4 40 in upgrade from the old edition WinOF In the environment where they are intermingled a problem may occur in MPI communication Network Direct Provider installed in each case is as follows The check of Network Direct Provider executes the following command after opening a command prompt window by administrator authority gt ndinstall Fujitsu InfiniBand HCA FDR User Manual 59 WinOF Installation Windows Installation e Inthe case of upgrade from the old edition WinOF OpenFabrics Network Direct OpenFabrics NDv1 Provider for Mellanox ConnectX OpenFabrics NDv2 Provider for Mellanox ConnectX e Inthe case of WinOF 4 40 new installation OpenFabrics NDv1 Provider for Mellanox ConnectX OpenFabrics NDv2 Provider for Mellanox ConnectX When Network Direct Provider has a difference remove OpenFabrics Network Direct from the server updated from the old edition WinOF Removal of Network Direct Provider executes the following command gt ndinstall r ibal Please reconfirm the situation of Networ
60. ted in table 4 Refer to chapter Linux Installation on page 25 for download and installation instructions Requirement Description Hardware Minimum 3 GB of memory PCI Express x8 or x16 slots Software For Linux the InfiniBand drivers are in the Open Operating Fabrics Enterprise Distribution OFED software Systems Distributions package which is available at http support ts fujitsu com com support downloads html Drivers amp Downloads If you purchased the products from Fujitsu Japan you can download it from http primeserver fujitsu com primergy downloads Table 4 Hardware and software requirements 2 7 2 Installation Instructions To change a full height bracket to a low profile bracket see chapter Appendix B on page 81 Read all installation instructions before connecting the equipment to the power Source The adapter cards listed in table 2 on page 11 are standard PCI Express cards each with a standard x8 edge connector Please consult the host machine documentation for instructions on how to install a PCI Express card Warning When more than one PCI slot is available make sure you use the PCI slot with the proper configuration 22 User Manual Fujitsu InfiniBand HCA FDR Adapter Card Hardware Adapter Card Installation Any PCI slot with the proper configuration is acceptable for connection If the card is installed in a PCI slot with less lanes than the card requires the
61. ther Mellanox OEM OFED or Distribution B packages will be removed Do you want to continue Ly N y Starting MLNX OFED LINUX 1 5 3 3 1 0 installation Installing kernel ib RPM Preparing 1L HHH 1L HHH HHH 1L HHH HLH HHH 1L HHH 1L HHH HLH HHH 1L HHH 1L HHH HLH HHH 1L HH T TT TT TI TTE WT WWW TT WT WW WT WT WT WWW TT TT kernel ib HH 1L HLH HH HHH HLH HHL HHH 1L HHH HLH IL 4L 4L 4L 1L 4L HHH 1LA4L IL 4L 4L 4L Hp 4L T TT TETTE TT TETTE T WT WT TTT WW WT TT TETTE WW TT WW WT Installation finished successfully Please reboot your system for the changes to take effect 2 Unmount mnt rootGRX200S7 1 mnt cd LrootGRX200S7 1 umount mnt rootGRX200S7 1 14 Fujitsu InfiniBand HCA FDR User Manual 39 OFED Installation Linux Installation 3 If you uninstalled the scsi target utils at 1 OFED Uninstallation re install the scsi target utils The scsi target utils package is in the instal lation disk of Red Hat Enterprise Linux rpm package scsi target utils 1 0 14 4 e16 x86 64 rpm rootGRX200S7 1 mount dev cdrom mnt mount block device dev srO is write protected mounting read only Lroot RX200S7 1 14 cd mnt Packages rootGRX200S7 1 Packages rpm ivh scsi target utils 1 0 14 4 e16 x86 64 rpm warning scsi target utils 1 0 14 4 e16 x86 64 rpm Header V3 RSA SHA256 Signature key ID fd431d51 NOKEY Preparing HHH HHH HHH HHH HHH
62. u may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket Fujitsu InfiniBand HCA FDR User Manual 83 Installing the New Bracket Appendix B 5 Screw on the bracket using the screws and washers saved from the procedure in Step 1 above F u Gasket a place on the bracket k ii d D Figure 23 Placing the bracket on a dual port card 84 User Manual Fujitsu InfiniBand HCA FDR Appendix B Installing the New Bracket Figure 24 Placing the bracket on a single port card 6 Make sure that the LEDs are aligned to the bracket holes 7 Screw the bracket Fujitsu InfiniBand HCA FDR User Manual 85 Installing the New Bracket Appendix B 86 User Manual Fujitsu InfiniBand HCA FDR Index A active link LED data activity 64 physical link 64 adapter cards 11 C CE marking 19 D disk space for installation 25 47 disposal 17 E electromagnetic compatibility 19 electrostatic sensitive devices ESD 17 ENERGY STAR 19 H hardware requirements 22 Oo OFED 7 installation 26 Open Fabrics Enterprise Distribution 7 OpenSM 44 45 47 61 operating system 25 R repair center 61 S safety instructions 13 software requirements 22 25 47 subnet manager Linux 44 OpenSM 61 support 61 T technical support 61 HCA adapter card 22 W Win OpenFabrics 47 I Windows 47 ibstat 45 installer privileges 25 47 L Linux 25
63. uids or foreign bodies switch off the server immediately remove all power plugs and contact your sales outlet or customer service team Fujitsu InfiniBand HCA FDR User Manual 15 Safety instructions Adapter Card Hardware CAUTION Proper operation of the system in accordance with IEC 60950 1 2 resp EN 60950 1 2 is only ensured if the casing is completely assembled and the rear covers for the installation slots have been fitted electric shock cooling fire protection interference suppression Only install system expansions that satisfy the requirements and rules governing safety and electromagnetic compatibility and those relating to telecommunication terminals If you install other expan sions they may damage the system or violate the safety regulations Information on which system expansions are approved for installation can be obtained from our customer service center or your sales outlet The components marked with a warning notice e g lightning symbol may only be opened removed or exchanged by authorized qualified personnel Exception CSS components can be replaced The warranty is void if the server is damaged during installation or replacement of system expansions Only set screen resolutions and refresh rates that are specified in the operating manual for the monitor Otherwise you may damage your monitor If you are in any doubt contact your sales outlet or customer service center Before installing
64. wing confir mation contents No Confirmation item Judgment To do 1 Is the server blade turned YES Go to No 2 on NO Turn on the server blade 2 Is the IB HCA card YES Go to No 3 mounted rigidly NO Mount the IB HCA card more rigidly 3 Is the IB cable rigidly YES Go to No 4 connected NO Connect the IB cable more rigidly Table 6 Physical link LED green turned off or flashing 64 User Manual Fujitsu InfiniBand HCA FDR Troubleshooting IB HCA No Confirmation item Judgment To do 4 Is InfiniBand Switch YES 1 Replace the IB HCA card connected working 2 If the LED is still turned off normally after replacing the IB HCA replace the system server 3 If the LED is still turned off after replacing the system server replace the IB cable 4 If the LED is still turned off please consult technical support Q ts fujitsu com If you purchased the products from Fujitsu Japan please consult your system engineer NO Replace the InfiniBand Switch Table 6 Physical link LED green turned off or flashing No Confirmation Item Judgment To do 1 Is the data activity link LED YES Confirm OpenSM running on yellow turned off the MASTER OpenSM Server NO Confirm the place where the error occurred using the above confirmation contents table 6 Physical Link LED green turned off or flashing Table 7 Data activity LED yell
65. work If HPC Cluster will consist of up to 16 nodes or less OpenSM can run on the head node computer The OpenSM service is registered to the Services Window after WinOF installation Double click OpenSM to open properties Fie Action View Help amp i as Hm mnp Services Local Services Local OpenSM Name Description Status S T On As Sl Microsoft iSCSI Initi Manages I Manual Local System iSi Microsoft Software Manages s Manual Local System Multimedia Class Sc Enables rel Manual Local System let Tcp Port Sharin Provides a Started Manual Local Service XO Netlogon Maintains a Started Automatic Local System Description Sh Network Access Pr The Netwo Manual Network S OpenSM Network Connections Manageso Started Manual Local System a Network List Service Identifies t Started Manual Local Service Sh Network Location A Collects an Started Automatic Network S Kot Network Store Inte This servic Started Automatic Local Service Fi St Automatic Local System Stop the service Pause the service Restart the service LOG ystem Enables re Manual Local Service Performan Manual Local Service Enablesac Started Automatic Local System ThePnPX Disabled Local System Enforcesg Manual Local System Manages p Started Automatic Local System Loads files Started Automat
66. work RHN If you cannot get the package from RHN copy the target package from an installation image The packages are in the Packages folder of an installation image Installation of an insufficient packages Install some packages using the rpm command Log in as root user rpm ivh PACKAGE FILE There is a dependency in a package Therefore an order to install has restrictions Please see Table 5 and install an insufficient package in an order from a top Please install OFED when you finish the installation of the insufficient packages Fujitsu InfiniBand HCA FDR User Manual 43 Updating Firmware after Installation Linux Installation 3 4 Updating Firmware after Installation If you wish to burn newer firmware you have to download it from Fujitsu i Technology Solutions Web site http support ts fujitsu com com support downloads html gt Drivers amp Downloads If you purchased the products from Fujitsu Japan you download it from http jp fujitsu com platform server primergy downloads 3 5 Subnet Manager At least one Subnet Manager SM is present on each subnet Each SM resides on a port of host channel adapter HCA or an InfiniBand Switch When there are multiple SMs on a subnet the master SM is one and the remaining SMs becomes standby SMs The master SM has a role to initialize and configure an InfiniBand subnet OpenSM is an InfiniBand compliant SM The following sections describe how to use OpenS
67. y forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer i trha cle zi Print I do not accept the terms in the license agreement InstallShield Figure 9 License Agreement 50 User Manual Fujitsu InfiniBand HCA FDR Windows Installation WinOF Installation 6 Click Next gt after the following window appears If you want to change the installation folder specify the folder you want to install on by clicking Change i MUNX_VPI InstallShield Wizard Destination Folder Click Next to install to this folder or click Change to install to a di a Install MLNX_VPI to aA C XProgram Files Mellanox MLNX_VPI InstallShield lt Back net gt cae Figure 10 Destination Folder Fujitsu InfiniBand HCA FDR User Manual 51 WinOF Installation Windows Installation 7 After the following window appears untick Check this box to configure your system for maximum 10GigE performance Recommended and then click Next gt fe MLNX_VPI InstallShield Wizard Maximum Performance Configure your system for maximum 10GigE performance J Check this box to configure your system for maximum 10GigE performance Recommended Note This step will require rebooting the machine at the end of installation

Download Pdf Manuals

Related Search

Related Contents

HP 4730s User's Manual  Image IMGE-5 User's Manual  Safety Guidelines and Operating Instructions for Working with  Comment donner du sens à la compréhension de l`oral en l`intégrant  Tricity Bendix BTD 03 User's Manual  CoolMaster Programmers Reference Manual (PRM) CoolMaster    DELL E525w    Instructions d`installation  

Copyright © All rights reserved. Failed to retrieve file