Home

Hardware Preparation Guide

1. The callouts in the figure enumerate the following 1 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 2 This port is the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 3 This port is the Ethernet connection to the Console Switch On the back of the node this port is marked with the acronym iLO Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL385 and DL385 G2 node in the hardware configuration 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the following iLO settings for each node depending on hardware model e Table 4 29 provides the iLO settings for ProLiant DL385 nodes e Table 4 30 provides the iLO settings for P
2. nO The callouts in the figure enumerate the following 106 Preparing Individual Nodes 1 NIC1 is the connection to the Administration Switch branch or root 2 Ifa Gigabit Ethernet GigE interconnect is configured this port labeled NIC2 is used for the interconnect connection Otherwise it is used for an external connection 3 The iLO2 Ethernet port is the connection to the Console Switch Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL585 and DL585 G2 node in the hardware configuration 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the following iLO settings for each node depending on hardware model e Table 4 34 provides the iLO settings for ProLiant DL585 nodes e Table 4 35 provides the iLO settings for ProLiant DL585 G2 nodes Table 4 34 iLO Settings for HP ProLiant DL585 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO us
3. O20 O20200 O8OR8 O la REA O6O5080605050 E 6G TEE Olu HPTC 0144 The callouts in the figure enumerate the following 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 89 90 1 This port is used for the connection to the Administration Switch branch or root On the rear of the node this port is marked with the number 1 NIC1 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the rear of the node this port is marked with the number 2 NIC2 3 The port labeled LO100i is used for the connection to the Console Switch Figure 4 13 shows a rear view of the HP ProLiant DL145 G3 server and the appropriate port assignments for an HP XC system Figure 4 13 HP ProLiant DL145 G3 Server Rear View The callouts in the figure enumerate the following 1 This port is used for the connection to the Administration Switch branch or root On the rear of the node this port is marked as NIC1 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the rear of the node this port is ma
4. The callouts in the figure enumerate the following 1 The iLO2 Ethernet port is the connection to the Console Switch 2 NIC1 is the connection to the Administration Switch branch or root 3 Ifa Gigabit Ethernet GigE interconnect is configured this port labeled NIC2 is used for the interconnect connection Otherwise it is used for an external connection Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL785 G5 node in the hardware configuration 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the following iLO settings for each DL785 G5 node as shown in Table 4 40 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 113 Table 4 40 iLO Settings for HP ProLiant DL785 G5 Nodes Menu Name Submenu Name Option Name User Add Set to This Value Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purpo
5. 1 Mellanox ConnectX Infiniband Cards require OFED Version 1 2 5 or later 2 The HP ProLiant DL385 G2 and DL145 G3 servers require a PCI Express card in order to use this interconnect 3 This interconnect is supported by CP6000 hardware models with PCI Express Mixing Adapters Within a given interconnect family several different adapters can be supported However HP requires that all adapters must be from the same interconnect family a mix of adapters from different interconnect families is not supported InfiniBand Double Data Rate All components in a network must be DDR to achieve DDR performance levels ConnectX InfiniBand Double Data Rate Currently ConnectX adapters cannot be mixed with other types of adapters Myrinet Adapters The Myrinet adapters can be either the single port M3F PCIXD 2 Rev D or the dual port M3F2 PCIXE 2 Rev E and Rev F mixing adapter types is not supported QsNet The QsNet high speed interconnect from Quadrics Ltd is the only version of Quadrics interconnect that is supported 1 9 Large Scale Systems 32 A typical HP XC system contains from 5 to 512 nodes To allow systems of a greater size an HP XC system can be arranged into a large scale configuration with up to 1024 compute nodes HP might consider larger systems as special cases This configuration arranges the HP XC system as a collection of hardware regions that are tied together through a ProCurve 2848 Ethernet switch The nodes o
6. CeCe CECEE EE EEE EERE EEE PEE ELE EEE ERE EEL EEEEEEE B al E 8888 A T O 2888 8 2288 ee Be eee _ VVTV es vere goo BOO H00 TE Ee orty om a for ea GEN par 10 100 1000 Base TX RJ 45 Ports Dual Personality Ports The callouts in the figure enumerate the following 1 Port 45 is used for the trunked link to the Root Administration Switch 2 Port 46 is used for the trunked link to the Root Administration Switch Allocate the ports on this switch for maximum performance as follows e Ports 1 10 13 22 25 34 and 37 44 are used for the administration ports for the individual nodes up to 38 nodes e Ports 11 12 23 24 35 36 47 and 48 are unused The ProCurve 2824 switch is shown in Figure 3 12 In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for a specific purposes described after the figure Figure 3 12 ProCurve 2824 Branch Administration Switch HAHAHAHA Bone een CETTC 2828 TEPP P ee 10 100 1000 Base TX RJ 45 Ports Dual Personality Ports The callout in the figure enumerates the following 1 Port 22 is used for the link to the Root Administration Switch Allocate the ports on this switch for maximum performance as follows e Ports 1 10 and 13 21 are used for the adm
7. 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL385 G2 node in the hardware configuration 102 Preparing Individual Nodes Make the RBSU settings accordingly The settings differ depending on the hardware model generation e Table 4 31 provides the RBSU settings for the HP ProLiant DL385 nodes e Table 4 32 provides the RBSU settings for the HP ProLiant DL385 G2 nodes Use the navigation aids shown at the bottom of the screen to move through the menus and make selections Table 4 31 RBSU Settings for HP ProLiant DL385 Nodes Menu Name Option Name Set to This Value System Options Embedded NIC Port PXE On all nodes except the head node set this value Support to Enable NIC1 PXE On the head node only set this value to Embedded NIC PXE Disabled Embedded Serial Port Disabled Virtual Serial Port COM1 IRQ4 I10 3F8h 3FFh Embedded NIC Port 1 PXE Enabled all nodes except the head node Support Embedded NIC Port 1 PXE Disabled head node only Support Power Regulator for Disabled ProLiant Standard Boot Order IPL Set the following boot order on all nodes except the head node 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed
8. Figure 4 20 shows a rear view of the HP ProLiant DL585 server and the appropriate port assignments for an HP XC system Figure 4 20 HP ProLiant DL585 Server Rear View 6 Oo ec e EIAI AJ ALALIA IAAT e ri WRR HHRH RHRRRHKRHH 0000 0000 000 00 EAS AA AASA eee 0000 eco 0000 xA HAE GEG DELEN Bie oar e 100MHz g Pale EEA 3b HAT 1 mH e oei mi ej ef e l H H Bl By E H AIL E jd de e E E H E a AH H AH del H H JE bI BI HI HL BI EI BEI B w a a a dJ U HU H e H H E a A A a dal ddl A E Ge 8 The callouts in the figure enumerate the following 1 The iLO Ethernet port is the connection to the Console Switch 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection 3 NIC1 is the connection to the Administration Switch branch or root Figure 4 21 shows a rear view of the HP ProLiant DL585 G2 server and the appropriate port assignments for an HP XC system Figure 4 21 HP ProLiant DL585 G2 Server Rear View ie QROROROSROROBOROBOROHKOHOSQ DOB OHOHOHOPOHROHOHEPEHEHL HO A a a ne OT eae cise ClSlSlelelelelelelele
9. 15 iLOs and OAs must use the same user name and password Do not use any special characters as part of the password Turn on power to the node MP CM gt pc on nc Press Ctrl B to return to the MP Main Menu Enter CO to connect to the console It takes a few minutes for the live console to display Add a boot entry and set the OS boot order Your actions for the head node differ from all other nodes Table 4 47 Adding a Boot Entry and Setting the Boot Order on HP Integrity Server Blades Head Node All Other Nodes Add a boot entry Add a boot entry 1 From the Boot Menu screen which is displayed 1 From the Boot Menu screen which is displayed during the power on of the node select the Boot during the power on of the node select the Boot Configuration Menu Configuration Menu 2 Select Add Boot Entry 2 Select Add Boot Entry 3 Select Removable Media Boot as the network boot 3 Select Load File Core LAN Port 1 as the network choice boot choice which is a Gigabit Ethernet GigE port 4 Enter the string DVD boot as the boot option 4 Enter the string Netboot as the boot option description description This entry is required and must be set 5 Press the Enter key twice forno db profileandload to the string Netboot with a capital letter N options 5 Press the Enter key twice for no db profile and load 6 If prompted save the entry to NVRAM options 6 If prompted save the entry to NVRAM Set the boot or
10. 5 Hard Drive C see Boot Controller Order On the head node set the boot order so that the CD ROM is listed before the hard disk BIOS Serial Console amp EMS BIOS Serial Console Baud Rate EMS Console BIOS Interface Mode BIOS Serial Console Port COM1 IRQ4 IO 3F8h 3FFh 115200 Disabled Command Line Press the Esc key to return to the main menu Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence Repeat this procedure for each HP ProLiant DL360 G5 node in the HP XC system 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 71 Configuring Smart Arrays On the HP ProLiant DL360 G5 nodes with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 5 5 Preparing HP Proliant DL380 G4 and G5 Nodes 72 Use the following tools to configure the appropriate settings on HP ProLiant DL380 G4 and G5 servers e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL380 G4 and G5 servers use the iLO utility thus they need certain settings
11. 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure for each HP ProLiant DL585 G5 node in the hardware configuration 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 111 1 Make the RBSU settings for each node Table 4 39 lists the RBSU setting for the DL585 G5 nodes Table 4 39 RBSU Settings for HP ProLiant DL585 G5 Nodes Menu Name System Options Option Name Embedded Serial Port Virtual Serial Port Embedded NICsI NIC 1 boot Options Embedded NICs NIC 2 boot Options Power Regulator for ProLiant Set to This Value Disabled COM1 On all nodes except the head node set this value to PXE boot Disabled head node only OS Control Mode Standard Boot Order IPL Set the following boot order on all nodes except the head node the CD ROM must be listed before the hard drive e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4PCI Embedded HP NC371i Multifunction Gigabit Adapter e IPL 5 Hard Drive C Set the following boot order on the head node e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4 Hard Drive C BIOS Serial Console and EMS Advanced BIOS Serial Console Port BIOS Serial Console Baud Rate EMS Console BIOS Interface Mode Linux x86_64 HPET Option COM1 115200 Disabled Command Line Enabled 2 Press th
12. Preparing Individual Nodes The callouts in the figure enumerate the following 1 This port is used for the connection to the Console Switch On the back of the node this port is marked with LO100i This port is used for the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 NIC1 If a Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 NIC2 Setup Procedure Perform the following procedure for each HP ProLiant DL160 G5 node in the hardware configuration Change only the values described in this procedure do not change any other factory set values unless you are instructed to do so Follow all steps in the sequence shown 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node Turn on power to the node Watch the screen carefully during the power on self test and press the F10 key when prompted to access the BIOS Setup Utility The Lights Out 100i LO 100i console management device is configured through the BIOS Setup Utility The BIOS Setup Utility displays the following information about the node BIOS ROM ID BIOS Version BIOS Build Date Record this information for future reference For each node make t
13. pdsh 15 reporting errors in 17 rrdtool 14 SLURM 14 software RAID 16 Supermon 15 syslog ng 15 SystemImager 15 TotalView 16 E EFI boot manager CP6000 systems 124 EFI firmware 59 ELANA see QsNet enclosure 84 119 136 setup guidelines 24 etc hosts file 84 119 Ethernet ports head node 61 external network creating VLANs 41 defined 41 NIC use 41 external storage 45 F feedback e mail address for documentation 17 firmware BMC 59 EFI FW 59 InfiniBand 59 IPMI 59 master list 59 MP 59 Myrinet 59 Quadrics 59 system 59 system BIOS 59 G Gigabit Ethernet interconnect 39 57 H hardware configuration supported 21 hardware models supported 19 21 hardware preparation 154 Index CP300BL 84 CP4000 87 CP4O00BL 119 CP6000 122 CP600BL 136 for all cluster platforms 61 HP xw8200 workstation 80 HP xw8400 workstation 80 HP xw8600 workstation 82 HP xw9300 workstation 116 HP xw9400 workstation 116 hardware preparation tasks HP server blades 84 119 136 HCA 58 head node in utility cabinet 45 high speed interconnects 56 host channel adapter see HCA HowTo Web site 12 HP documentation providing feedback for 17 HP Integrity BL860c Server Blade 20 21 HP Integrity rx1620 20 122 HP Integrity rx2600 20 122 HP Integrity rx2620 20 125 HP Integrity rx2660 20 127 HP Integrity rx4640 20 129 HP Integrity rx8620 20 131 HP ProLiant BL260c G5 Serve
14. Figure 4 8 HP ProLiant DL580 G5 Server Rear View The callouts in the figure enumerate the following 1 2 3 The iLO Ethernet port is used for the connection to the Console Switch NIC1 is used for the connection to the Administration Switch branch or root NIC2 is used for the connection to the external network Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL580 G5 node in the HP XC system 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the iLO settings listed in Table 4 15 for each node in the hardware configuration Table 4 15 iLO Settings for HP ProLiant DL580 G5 Nodes Submenu Name Menu Name Name Option Name Set to This Value Administration User New Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console p
15. If you are not sure about a command you need to use enter the man command with the k option to obtain a list of commands that are related to a keyword For example man k keyword HP Encourages Your Comments HP encourages comments concerning this document We are committed to providing documentation that meets your needs Send any errors found suggestions for improvement or compliments to docsfeedback hp com Include the document title manufacturing part number and any comment error found or suggestion for improvement you have concerning this document Manpages 17 18 1 Hardware and Network Overview This chapter addresses the following topics e Supported Cluster Platforms page 19 e Server Blade Enclosure Components page 22 e Server Blade Mezzanine Cards page 28 e Server Blade Interconnect Modules page 28 e Supported Console Management Devices page 29 e Administration Network Overview page 30 e Administration Network Console Branch page 31 e Interconnect Network page 31 e Large Scale Systems page 32 1 1 Supported Cluster Platforms An HP XC system is made up of interconnected servers A typical HP XC hardware configuration on systems other than Server Blade c Class servers contains from 5 to 512 nodes To allow systems of a greater size an HP XC system can be arranged into a large scale configuration with up to 1 024 compute nodes
16. O00000 0000000000000000000 EI O lt p O o Ro m oe or The callouts in the figure enumerate the following 1 NIC1 is used for the connection to the Administration Switch branch or root 2 NIC2 is used for the connection to the external network 3 The iLO Ethernet port is used for the connection to the Console Switch Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL580 G4 node in the HP XC system 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the iLO settings listed in Table 4 13 for each node in the hardware configuration 76 Preparing Individual Nodes Table 4 13 iLO Settings for HP ProLiant DL580 G4 Nodes Submenu Name Menu Name Name Option Name Set to This Value Administration User New Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purpose
17. 2 6 Cabling for the External Network Depending upon the roles you assign to nodes during the cluster configuration process some nodes might require connections to an external public network Making these connections requires one or more Ethernet ports in addition to the ports already in use The ports you use depend upon the hardware configuration and the number of available ports On non blade server nodes the appropriate port assignments for the external network are shown in Chapter 4 page 59 On a server blade the number of available Ethernet ports is influenced by the type of interconnect and the server blade height e Nodes in clusters that use an InfiniBand interconnect have only one NIC in use for the administration network e Nodes in clusters that use a Gigabit Ethernet interconnect have two NICs in use one for the administration network and one for the interconnect network e Half height server blade models have two built in NICs e Full height server blade models have four built in NICs You can use the built in NICs on a server blade if any are available If the node requires more ports you must add an Ethernet card to mezzanine bay 1 on the server blade If you add an Ethernet card to mezzanine bay 1 you must also add an Ethernet interconnect module either a switch or pass thru module to interconnect bay 3 or 4 of the c7000 enclosure On full height server blades you can avoid having to purchase an additional mezzani
18. An application that uses a distributed programming model and can run on multiple processors An HP XC MPI application is a parallel application That is all interprocessor communication within an HP XC parallel application is performed through calls to the MPI message passing library Preboot Execution Environment A standard client server interface that enables networked computers that are not yet installed with an operating system to be configured and booted remotely PXE booting is configured at the BIOS level See RGS Nodes with this role manage the allocation of resources to user applications HP Remote Graphics Software A utility that enables remote access and sharing of a graphics workstation desktop A set of services that are assigned to a node A component of the administration network The top switch in the administration network it may be a logical network switch comprised of multiple hardware switches The Root Console Switch is connected to the Root Administration Switch A node within an HP XC system that is connected directly to the Root Administration Switch Red Hat Package Manager 1 A utility that is used for software package management on a Linux operating system most notably to install and remove software packages 2 A software package that is capable of being installed or removed with the RPM software package management utility See SVA A command or user program that does not use any distributed shared
19. Connechons satntinndtea ri a teeta atiateutitiven Wate 132 Gigabit Ethernet Interconnect With Half Height Server Blades ccccccesssseeseesseeseeneeeeenes 143 InfiniBand Interconnect With Full Height Server Blades ve isc iccnatsveisecctenasiasiises seaesvelenees 144 InfiniBand Interconnect With Mixed Height Server Blades ccccsceseseeseesseesseeeeeseeseenseeses 145 List of Tables 1 1 1 2 1 3 1 4 3 1 3 2 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 4 12 4 13 4 14 4 15 4 16 4 17 4 18 4 19 4 20 4 21 4 22 4 23 4 24 4 25 4 26 4 27 4 28 4 29 4 30 4 31 4 32 4 33 4 34 4 35 4 36 4 37 4 38 4 39 4 40 4 41 4 42 4 43 4 44 4 45 4 46 8 Supported Processor Architectures and Hardware Models c cccccessseesseesecsecesseeseeeseeeeees 20 supported HP Proliant Server Blade Modelsisicccasccinininanannennaeniaineuei 21 Supported Console Management Devices iiaisvsdsimitsrsinnienssneieaninccntsaustenuincinriaminuaneianiinain 29 Supported IMerCOMME CES reisseri eoe er eat eede setae deta 32 S pport d Switch ModelS sis sserissirsiesira eri bernie pirena ESEE pear Rei P EEA EPERRA EEr A RE ERA NESER SEARE a 47 Trunking Port Use on Large Scale Systems with Multiple Regions sessssssesessseesrssesssrsesees 49 Firmware Depende di S sorensisrip ipes esnea EE aa oaa ad a aaka RE ASENTAR ESA E a 59 Ethernet Ports on th Head N d issiria anai aoa en aae anaE a
20. E a aE ASE Asa 61 BIOS Settings for HP ProLiant DL140 G2 Nodes iiccasascassnnccnscanedesnsinn tunnasisnssn sae svuaanedieassiieewrauavegy 64 BIOS Settings for HP ProLiant DL140 G3 Nodes sssessessseiseisessseissisersseessrenresresseesresnessessesees 65 BIOS Settings for HP ProLiant DL160 G5 Nod Sis sissisvesissssvassoisnvenis tasaassssnvansctisveasvssueanersaneants 67 iLO Settings for HP ProLiant DL360 G4 Nodes snsssnssssessessessissesesttssssesressestessnsentesensesrensenets 69 BIOS Settings for HP ProLiant DL360 G4 Nodes ssssssessessesenssssesressssessnssensesensenresensesrensenees 69 iLO Settings for HP ProLiant DL360 G5 Nodes ssssssessessssissrsesssrssrsesssressresresrnseesenressresresens 71 BIOS Settings for HP ProLiant DL360 G5 Nodes isicincnainnnienneennacan abn ween 71 iLO Settings for HP ProLiant DL380 G4 and G5 Nodes icsissicassncasssaneticsvseencevssnistntaneceaedannstersins 73 BIOS Settings for HP ProLiant DL380 G4 Nodes sessessssissrsessrsrssisnsssessresresernstesesressreseesens 74 BIOS Settings for HP ProLiant DL380 G5 Nodes siiscntectinenciancrataginwsradbatiorimenboeia ecbdund Miainwdte 74 iLO Settings for HP ProLiant DL580 G4 Nodes ssssessssessesseseissesesesessessensessessssestesessesnensenees 77 BIOS Settings for HP ProLiant DL580 G4 Nodes cinvsiericevcereremaccsrcncnide iometansaiedmorbeundsdennnivaimenete 78 iLO Settings for HP ProLiant DL580 G5 Nodes sssssssssessesst
21. File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL365 G5 node in the hardware configuration Preparing Individual Nodes 3 Make the RBSU settings for the HP ProLiant DL365 G5 nodes as indicated in Table 4 28 Use the navigation aids shown at the bottom of the screen to move through the menus and make selections Table 4 28 RBSU Settings for HP ProLiant DL365 G5 Nodes Menu Name Option Name Set to This Value System Options Embedded NIC Port PXE On all nodes except the head node set this value Support to Enable NIC1 PXE On the head node only set this value to Embedded NIC PXE Disabled Embedded Serial Port Disabled Virtual Serial Port COM1 IRQ4 IO 3F8h 3FFh Embedded NIC Port 1 PXE Enabled all nodes except the head node Support Embedded NIC Port 1 PXE Disabled head node only Support Power Regulator for Disabled ProLiant Standard Boot Order IPL Set the following boot order on all nodes except the head node 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk IPL1 CD ROM IPL2 Floppy Drive A IPL3 PCI Embedded HP NC7782 Gigabit Server Adapter Port 1 IPL4 Hard Drive C BIOS Serial Console and BIO
22. HP ProLiant DL785 G5 Nodes snssssnssesesesesseseesestessssestesssresnensennesensenenene 113 4 7 11 Preparing HP xw9300 and xw9400 Workstations ccccccsccecssscesscesesesseessesceseesessseeeeseees 116 4 8 Preparing the Hardware for CP4000BL Sy Stems veiesieinciecd ustacindinninemsieansiinnians 119 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems ssssesssssersessseisseserrseersesneeseesse 122 4 9 1 Setting Static IP Addresses on Integrity Servers iis citicincsccetaniecoscaranercersiiannsraeivouaencDiatee 122 4 9 2 Preparing HP Integrity rx1620 and rx2600 NOeSiniciivuc atin onncial 122 4 9 3 Preparing HP Integrity rx2620 NOU CSicecicsersmvactsanenicerimoserocimate ciewnsaueaumetannasiveimetaesentainoss 125 4 9 4 Preparing HP Integrity rx2660 NOdeS ios sisccatitueriatiornanininetitancianiedintnemad ii aidsa 127 4 9 5 Preparing HP Integrity rx4640 NOG Siisiniicicaisleidsadtnnssbanimcucauaanianincinraeahbauendasandendcbinrsienanianen 129 4 9 6 Preparing FIP Integrity rx8620 INGUES lt siisssssssansiedenaoudnsiyiadetssincinastionasintvmesnnenasaodensbhesienesanesents 131 4 Table of Contents 4 10 Preparing the Hardware tor CP6000BL Sy Stems ceecasatisntasnscpcteiintierer beeieeuebiilidbecaniunce inven 136 eB ie a oa cane ee een aan EO RHE AE REO AE aN Peau NNO em 139 hs Le WEN TOS ACCS sik esse tse cu en aceasta atin www aaec a daa aad Stee Rca A MLR TRCSaR NL 139 BL 11LO2 Devices Can Become UnresponsiV Cvsizcncnr
23. HP might consider larger systems as special cases HP Server Blade c Class servers hereafter called server blades are perfectly suited to form HP XC systems Physical characteristics make it possible to have many tightly interconnected nodes while at the same time reducing cabling requirements Typically server blades are used as compute nodes but they can also function as the head node and service nodes The hardware and network configuration on an HP XC system with HP server blades differs from that of a traditional HP XC system and those differences are described in this document You can install and configure HP XC System Software on the following platforms e HP Cluster Platform 3000 CP3000 e HP Cluster Platform 3000BL CP3000BL with HP c Class server blades e HP Cluster Platform 4000 CP4000 e HP Cluster Platform 4000BL CP4000BL with HP c Class server blades e HP Cluster Platform 6000 CP6000 e HP Cluster Platform 6000BL CP6000BL with HP c Class server blades For more information about the cluster platforms see the documentation that was shipped with the hardware 1 1 1 Supported Processor Architectures and Hardware Models Table 1 1 lists the hardware models that are supported for each HP cluster platform 1 1 Supported Cluster Platforms 19 we IMPORTANT A hardware configuration can contain a mixture of Opteron and Xeon nodes but not Itanium nodes Table 1 1 Supported Processor Architectures and Hardware
24. IP address 84 119 setting the password 62 OpenFabrics Enterprise Distribution see OFED P password iLO2 84 119 MP 126 128 130 133 136 onboard administrator 62 ProCurve switch administrator 47 PCI X 56 port connections branch administration switch 54 branch console switch 55 interconnect switch 56 root administration switch 50 root console switch 51 super root switch 49 155 processor architecture 19 AMD Opteron 19 Intel Itanium 19 Intel Xeon with EM64T 19 processor architectures 20 ProCurve 2610 24 root console switch 51 54 ProCurve 2610 48 branch console switch 55 root console switch 51 52 ProCurve 2626 47 root console switch 51 53 ProCurve 2650 47 branch console switch 55 root console switch 51 ProCurve 2824 47 branch administration switch 54 root administration switch 50 super root switch 49 ProCurve 2848 47 branch administration switch 54 root administration switch 50 super root switch 49 ProCurve switch administrator password 47 public network see external network Q OsNet 57 interconnect 31 Quadrics see QsNet R RBSU settings HP ProLiant DL365 96 HP ProLiant DL365 G5 99 HP ProLiant DL385 103 HP ProLiant DL385 G2 104 HP ProLiant DL585 109 HP ProLiant DL585 G2 110 HP ProLiant DL785 G5 115 region defined 32 reporting documentation errors feedback e mail address for 17 root administration switch 47 50 root console switch 47
25. LO 100i depending on the server model Each HP ProLiant server blade has a built in Integrated Lights Out iLO2 device that provides full remote power control and serial console access You can access the iLO2 device through the Onboard Administrator On server blades iLO2 advanced features are enabled by default and include the following e Full remote graphics console access including full keyboard video mouse KVM access through a Web browser e Support for remote virtual media which enables you to mount a local CD or diskette and serve it to the server blade over the network Each HP Integrity server blade has a built in management processor MP device that provides full remote power control and serial console access You can access the MP device by connecting a serial terminal or laptop serial port to the local IO cable that is connected to the server blade Hardware models that use iLO and iLO2 need certain settings that cannot be made until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO and iLO2 to enable telnet access Table 1 3 Supported Console Management Devices Hardware Component Firmware Dependency CP3000 HP ProLiant DL140 G2 Lights out 100i management LO 100i system BIOS HP ProLiant DL140 G3 LO 100i system BIOS HP ProLiant DL160 G5 LO 100i system BIOS HP ProLiant DL360 G4 Integrated lights out iLO system BIOS HP P
26. Message Passing Interface HP MPI is an implementation of the MPI standard that has been integrated in HP XC systems The home page and documentation is located at the following Web site http www hp com go mpi HP Serviceguard HP Serviceguard is a service availability tool supported on an HP XC system HP Serviceguard enables some system services to continue if a hardware or software failure occurs The HP Serviceguard documentation is available at the following Web site http docs hp com en ha html HP Scalable Visualization Array The HP Scalable Visualization Array SVA is a scalable visualization solution that is integrated with the HP XC System Software The SVA documentation is available at the following Web site http docs hp com en linuxhpc html HP Cluster Platform The cluster platform documentation describes site requirements shows you how to set up the servers and additional devices and provides procedures to operate and manage the hardware These documents are available at the following Web site http www docs hp com en highperfcomp html HP Integrity and HP ProLiant Servers Documentation for HP Integrity and HP ProLiant servers is available at the following web address http docs hp com en hw html For c Class Server BladeSystems see also the installation administration and user guides for the following components e HP ProLiant or Integrity C Class Server Blades e HP BladeSystem c Class Onboard
27. Models Server Type Blade Cluster Platform CP3000BL Processor Architecture Intel Xeon with EM64T Hardware Model HP ProLiant BL2x220c G5 HP ProLiant BL260c G5 HP ProLiant BL460c HP ProLiant BL480c HP ProLiant BL680c G5 CP4000BL AMD Opteron HP ProLiant BL465c HP ProLiant BL465c G5 HP ProLiant BL685c HP ProLiant BL685c G5 CP6000BL Intel Itanium HP Integrity BL860c Non Blade CP3000 Intel Xeon with EM64T HP ProLiant DL140 G2 HP ProLiant DL140 G3 HP ProLiant DL160 G5 HP ProLiant DL360 G4 HP ProLiant DL360 G4p HP ProLiant DL360 G5 HP ProLiant DL380 G4 HP ProLiant DL380 G5 HP ProLiant DL580 G4 HP ProLiant DL580 G5 HP xw8200 Workstation HP xw8400 Workstation HP xw8600 Workstation CP4000 AMD Opteron HP ProLiant DL145 HP ProLiant DL145 G2 HP ProLiant DL145 G3 HP ProLiant DL165 G5 HP ProLiant DL365 HP ProLiant DL365 G5 HP ProLiant DL385 HP ProLiant DL385 G2 HP ProLiant DL385 G5 HP ProLiant DL585 HP ProLiant DL585 G2 HP ProLiant DL585 G5 HP ProLiant DL785 G5 HP xw9300 Workstation HP xw9400 Workstation CP6000 20 Hardware and Network Overview Intel Itanium HP Integrity rx1620 HP Integrity rx2600 HP Integrity rx2620 HP Integrity rx2660 HP Integrity rx4640 HP Integrity rx8620 HP server blades offer an entirely modular computing system with separate computing and physical I O modules that are connected and shared through a common chassis called a
28. Node Interleave Disabled 32 Bit Memory Hole Enabled Serial ATA Embedded SATA Enabled SATA Mode SATA Enabled Disable Int13 Enabled support Option ROM Scan Enabled Enable Master Enabled Latency Timer 0040h Console Redirection Com Port Address On board COM A Baud Rate 115 2K Console Type ANSI Flow Control None Console connection Direct Continue C R after POST on of video pages to support 1 IPMI LAN Setting IP Address Assignment DHCP LAN Controller NIC HPET Timer Disabled 8042 Emulation Support Disabled Factory Boot Mode Disabled Boot Set the following boot order on all nodes except the head node Removable Devices CD ROM Drive MBA v9 0 6 Slot 0820 Hard Drive 5 MBA v9 0 6 Slot 0821 Set the following boot order on the head node 1 Removable Devices 2 CD ROM Drive 3 Hard Drive eu 92 Preparing Individual Nodes 4 Select Exit gt Saving Changes to exit the BIOS Setup Utility 5 Repeat this procedure for each HP ProLiant DL145 G2 and DL145 G3 node in the hardware configuration 4 7 3 Preparing HP ProLiant DL165 G5 Nodes Use the BIOS Setup utility on HP ProLiant DL165 G5 servers to configure the appropriate settings for an HP XC system For this hardware model you cannot set or modify the default console port password through the BIOS Setup Utility the way you can for other hardware models The HP XC System Software Installation Guide documents the procedure to modify the console port password You
29. O a O OY O O O O O O O OY O O O O se Co Pa 00000 BeBoB08o 2024 a Xe Xe O O O Se O Xe OY ORO Q re Q a cone 8 a oooO ACEN E Oleee The callouts in the figure enumerate the following 1 2 3 The LAN port labeled Gb 1 connects to the Administration Switch branch or root The LAN port labeled Gb 2 is used for an external connection The port labeled MP LAN is the MP connection to the ProCurve Console Switch Setup Procedure Perform the following procedure on each HP Integrity rx2660 server 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node For each node in the system ensure that the power cord is connected but that the processor is not turned on Follow this procedure to connect a personal computer to the Management Processor MP a Connect a cable to the console port located to the left of the MP LAN port on the back of the HP Integrity rx2660 server 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 127 b Connect the CONSOLE connector to a null modem cable and connect the null modem cable to the PC COM1 port c Use a terminal emulator such as HyperTerminal to open a terminal window Press
30. On such hardware models as the HP ProLiant DL380 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 5 6 Preparing HP Proliant DL580 G4 Nodes Use the following tools to configure the appropriate settings on HP ProLiant DL580 G4 servers e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL580 G4 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 7 shows a rear view of the HP ProLiant DL580 G4 server and the appropriate port assignments for an HP XC system 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 75 Figure 4 7 HP ProLiant DL580 G4 Server Rear View 0 0 0 0 0 0 0 o 000S J OPS DPSPELELHLIOLCHCPELSLGS 2EEEEEEGOSORO80808050303030505065 0825082938 Z BORO ORO OR ORO ROS oo
31. ProLiant DL585 and DL585 G2 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 7 9 Preparing HP Proliant DL585 G5 Nodes On HP ProLiant DL585 G5 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL585 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 22 shows a rear view of the HP ProLiant DL585 G5 server and the appropriate port assignments for an HP XC system 110 Preparing Individual Nodes Figure 4 22 HP ProLiant DL585 G5 Server Rear View O 00 00 CO 00 OO 00 OO OO O TE The callouts in the figure enumerate the following 1 The iLO2 Ethernet port is the connection to the Console Switch 2 NIC1 is the connection to the Administration Switch branch or root 3 Ifa Gigabit Ethernet GigE interconnect is
32. Rear View 2020202020202209 o O OBORO RS o DEREDE ISORO ORRO BRORSRORORSRORORSRORO ROE OROROORO RORO NONONONO LORRI QROROROROROROROROROROSO eters terecetecetacece QRORORORORORSRORSROROE SEO RES ERO ROR RO RESO RO IOS QEEEEEOEOS IES AKOSESISRO ROBLOLORLSLSLSLOB0LORO RH ORCRSRORSRSRORGRSRORORG ese ORORORLOKOROSO QRORORORORSRORORSROROSOE EROROROLOLOROLORORLORLOSO QRBORORORORLORORORSROROLROS OBOLOROROROLOLSLORLORARS CORSRORORSRORORSRORS OSOS EE E 0 o 0805050000000000 O mm IOE wo The callout in the figure enumerates the following 1 This port is used for the connection to the administration network Figure 4 25 shows a rear view of the xw9400 workstation and the appropriate port connections for an HP XC system Figure 4 25 xw9400 Workstation Rear View 2020202020202029 QBOKOKQEQGQGOKOKO QRORQEQBORQEOQROSQE OO000 The callouts in the figure enumerate the following 1 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection 2 This port is used for the connection to
33. SWitch cccccccccccccccccscecccccscecccscsesccsseececcsseucecssseeecesseuseesseueessseueeseseunesess 52 3 3 6 2 ProCurve 2610 48 Swit hiw cccceccccccccccccssscccccuesccsssessccsseececesseueccsseuesesscueessseuesesseueeess 52 3 3 6 3 ProCurve 2626 SWitch ccccccccccccccccsseccccccecccscecsccsscuceccsscececcsseuecesseuecesssusessseuessseseueess 53 3 3 6 4 ProCurve 2610 24 Swit hui cccccccccccccccccccssccccccscsccsseseeccsscucescsseescesseueesesscuseesseueeseseueesess 54 3 3 7 Branch Administration Switches cccccssccccccccccesseseseecccccecesueuseececeseceaseaeeesceceeeeeauaaeneseeeeeees 54 3 3 8 Branch Console Switches ccccccccccccesssseseecececececusussecccccceceaueaseececeseceaueusnescececeesauaaeeneeeeeess 55 3 4 Interconnect CONNECHONS y 0 ecesscecescscdeserovecesvevcdesseus seen evsncdebeseeberesssbesucaeiodendeoassdepnebeseteevedbevenewecoweece 56 3 4 1 OsNet Interconnect COnnections cccccccecccccecececeeeessseseeeeeeeeaeeseseseeeeeeseaseaeeeseseeseseasaaaesseseeeess 57 3 4 2 Gigabit Ethernet Interconnect Conme Chins iis ciaencrwariiaecincsnsnissnensacasd cdecimevoatar einer iesalnaineincn 57 3 4 3 Administration Network Interconnect ConnectionS sseessssesseresessssesrrrrereessesrereereesseses 57 3 4 4 Myrinet Interconnect Orie iOS vs sanncasavnccinn sxmeevoasnv danasativnasndeniniansenlasstiimiediamanciuse Manaamesinninsie 58 3 4 5 InfiniBand Interconnect CONNECTIONS ccccccccecsesesseecece
34. UC and use the menu options to remove the default MP user name and password and create your own unique user name and password HP recommends setting your own user name and password for security purposes The user name must have a minimum of 6 characters and the password must have a minimum of 8 characters You must set the same user name and password on every node The user name and password are required to access the power management device and console 8 If required set static IP addresses For more information see Setting Static IP Addresses on Integrity Servers page 122 9 Enter PC power cycle and then enter on to turn on power to the node 10 Press Ctrl B to return to the MP Main Menu 11 Enter CO to connect to the console 12 Perform this step on all nodes except the head node From the Boot Menu screen which is displayed during the power on of the node select the Boot Configuration Menu Do the following from the Boot Configuration Menu a Select Add Boot Entry b Select Load File Core LAN Gb A as the network boot choice which is a Gigabit Ethernet GigE port c Enter the string Netboot as the boot option description This entry is required and must be set to the string Netboot with a capital letter N d Press the Enter key for no db profile options e Press the Enter key for no boot options f If prompted save the entry to NVRAM For more information about how to work with these menus see the do
35. amp 6 doubl erconnect bay 7 erconnect bay 8 External Public Network I ONBOARD ADMINISTRATOR HALF HEIGHT BLADE Administration Nietiwmik Console Network Cluster Inttercommescit Nettvankk External Network B 3 InfiniBand Interconnect With Mixed Height Server Blades The configuration shown in Figure B 3 is similar to the configuration shown in Figure B 1 page 143 The only exception is that in this configuration the half height server blades require external connections as well Because half height blades have two NICs you must use NIC2 for the connection to the external network This also means that an interconnect module is required in bay 2 On the non blade server nodes the built in NICs are used for the external network connection Available ports vary based on hardware model See Chapter 3 page 45 for more information about port assignments 144 Server Blade Configuration Examples Figure B 3 InfiniBand Interconnect With Mixed Height Server Blades FULL HEIGHT BLADE Non Blade Server PCI SLOT InfiniBand PCI Cards Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT C Class Blade Enclosure NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 MEZZ 3 iLO2 NIC 1 NIC 2 HALF HEIGHT BLADE MEZZ 1 MEZZ 2 iLO2 InfiniBand Mezzanine Cards Interconnect ba
36. as follows e Ports 1 10 13 22 25 34 37 41 Starting with port 1 the ports are used for links from Branch Console Switches Trunking is not used Starting with port 41 and in descending order ports are assigned for use by individual nodes in the utility cabinet Nodes in the utility cabinet are connected directly to the Root Administration Switch A NOTE There must be at least one idle port in this set to indicate the dividing line between branch links and root node administration ports e Ports 11 12 23 24 35 36 and 43 48 are unused 3 3 6 2 ProCurve 2610 48 Switch 52 You can use a ProCurve 2610 48 switch as a Root Console Switch for the console branch of the administration network The console branch functions at a lower speed 10 100 Mbps than the rest of the administration network The ProCurve 2610 48 switch is shown in Figure 3 8 In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Making Node and Switch Connections Figure 3 8 ProCurve 2610 48 Root Console Switch Uplinks from Branches Connections to Node Console Ports Start at port 1 Ascending Start at port 41 Descending 10 100Base TX RJ 45 Ports The callouts in the figure enumerate the following 1 Port 42 must be reserved for an optional connection to the console port on the head node 2 Port 49 is re
37. based on interconnect type and server blade height The connections are color coded so consider viewing the PDF file online or printing this appendix on a color printer to take advantage of the color coding B 1 Gigabit Ethernet Interconnect With Half Height Server Blades In Figure B 1 only the half height server blades and the non blade server nodes have connections to the external network Because the two NICs of the half height server blade are already in use an Ethernet card was added to mezzanine bay 1 to allow for the external network connection on the server blade An interconnect module was added to bay 3 On the non blade server nodes PCI Ethernet NICs are used for the external network connection Available ports vary based on hardware model See the HP XC Hardware Preparation Guide for more information about port assignments Figure B 1 Gigabit Ethernet Interconnect With Half Height Server Blades Ethernet PCI Cards Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT ad 5 Admin ProCurve 2800 Series Switch Non Blade Server PCISLOT MGT NIC NIC PCI SLOT w C Class Blade Enclosure poik nterconnect bay 2 IE l H nterconnect bay 3 L nterconnect bay 5 nterconnect bay 6 nterconnect bay 7 nterconnect bay 8 Console ProCurve 2600 Series Switch A Gigabit Ethernet Interconnect Switch External Pu
38. before the hard disk IPL1 CD ROM IPL2 Floppy Drive A IPL3 PCI Embedded HP NC7782 Gigabit Server Adapter Port 1 IPL4 Hard Drive C BIOS Serial Console and BIOS Serial Console Port COM1 IRQ4 10 3F8h 3FFh EMS BIOS Serial Console Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line Advanced Options page Directory Cache Disabled PDC 1 Asmall blue dialog box near the bottom left side of the screen indicates the current setting You can make only one setting per node Table 4 32 lists the RBSU settings for the HP ProLiant DL385 G2 nodes Table 4 32 RBSU Settings for HP ProLiant DL385 G2 Nodes Menu Name Option Name Set to This Value 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 103 Table 4 32 RBSU Settings for HP ProLiant DL385 G2 Nodes continued Menu Name Option Name Set to This Value System Options Embedded Serial Port COM2 Virtual Serial Port COM1 Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 Floppy Drive A 3 USB DriveKey C 4 PCI Embedded HP NC373i Multifunction Gigabit Adapter 5 Hard Disk C see Boot Controller Order On the head node set the boot order so that the CD ROM is listed before the hard disk BIOS Serial Console and BIOS Serial Console Port COM1 EMS BIOS Serial Console
39. connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 3 This port is the Ethernet connection to the Console Switch On the back of the node this port is marked with the acronym iLO Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL365 G5 node in the hardware configuration 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the following iLO settings for ProLiant DL365 G5 nodes as shown in Table 4 27 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 97 98 Table 4 27 iLO Settings for HP ProLiant DL365 G5 Nodes Set to This Value Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Menu Name Submenu Name Option Name User Add Network DNS DHCP DHCP Enable On 4 Select
40. for maximum performance e Ports 1 10 13 22 25 34 37 42 Starting with port 1 the ports are used for links from Branch Administration Switches which includes the use of trunking Two port trunking can be used for each Branch Administration Switch A NOTE Trunking is restricted to within the same group of 10 you cannot trunk with ports 10 and 13 HP recommends that all trunking use consecutive ports within the same group 1 10 13 22 25 34 or 37 42 Starting with port 41 and in descending order ports are assigned for use by individual nodes e Ports 11 12 23 24 35 36 are unused For size limited configurations the ProCurve 2824 switch is an alternative Root Administration Switch If you are using a ProCurve 2824 switch as the switch at the center of the administration network use Figure 3 6 to make the appropriate port connections In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Making Node and Switch Connections Figure 3 6 ProCurve 2824 Root Administration Switch BHEBEEE BODES a AE The callouts in the figure enumerate the following Sy Oe Ge tal pa Uplinks from branches start at port 1 ascending 10 100 1000 Base TX RJ 45 ports Connections to node administration ports start at port 2
41. for one HP Integrity rx8620 node On each HP Integrity rx8620 node connect partition 0 to the highest numbered available open port and connect partition 1 to the next lower numbered port Repeat this step for the next HP Integrity rx8620 and so on See Table 4 46 for more information Connect the Quadrics boards in the HP Integrity rx8620 partitions to the Quadrics switch using the same pattern as the Gigabit Ethernet connections to the ProCurve 28xx switches Connect partition 0 of the first HP Integrity rx8620 server to the highest available Quadrics switch port which is followed by partition 1 to the next highest available switch port which is followed by partition 0 of the second HP Integrity rx8620 server and so on Figure 4 31 shows the HP Integrity rx8620 Core IO board and the appropriate connections to the administration network and console network 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 131 Figure 4 31 HP Integrity rx8620 Core IO Board Connections sam al lt a lil lt a YIOMJ N uld seq dA uonesiuIupy 0 uonosuuog Hod eues dN 10SS990 4 uawa pLeuen YIOMJON Josuo 0 UO I D UUOD Preparing Individual Nodes Follow this procedure for each HP Integrity rx8620 node in the hardware configuration 1 Ensure that the power cord is connected but that the processor is not turned on 2 Connect a persona
42. move the Netboot entry you just defined to the top of the boot order b If prompted save the setting to NVRAM c Enter x to return to the previous menu Perform this step on all nodes including the head node Select the Console Configuration option and do the following a Enable the Serial Acpi HWP0002 PNP0A03 700 Pci 1 1 Vt100 9600 option b Press the Esc key or enter x as many times as necessary to return to the Boot Configuration menu c When prompted save the entry to NVRAM Turn off power to the node a Press Ctrl B to exit console mode b Enter CM to display the Command Menu c Enter PC and enter off to turn off power to the node 4 9 3 Preparing HP Integrity rx2620 Nodes Figure 4 28 shows a rear view of the HP Integrity rx2620 server and the appropriate port assignments for an HP XC system Figure 4 28 HP Integrity rx2620 Server Rear View PCI X 133 PCI X 133 PCI X 133 PCI X 133 The callouts in the figure enumerate the following 1 2 3 The port labeled LAN 10 100 is the MP connection to the ProCurve Console Switch The port labeled LAN Gb A connects to the Administration Switch branch or root The port labeled LAN Gb B is used for an external connection Setup Procedure Perform the following procedure on each HP Integrity rx2620 server 1 Use the instructions in the accompanying ha
43. on power to the node 10 Press Ctrl B to return to the MP Main Menu 11 Enter CO to connect to the console 12 Perform this step on all nodes except the head node From the Boot Menu screen which is displayed during the power on of the node select the Boot Configuration Menu Do the following from the Boot Configuration Menu a Select Add Boot Entry b Select Load File Core LAN Gb A as the network boot choice which is a Gigabit Ethernet GigE port c Enter the string Netboot as the boot option description This entry is required and must be set to the string Netboot with a capital letter N d Press the Enter key for no boot options e Ifprompted save the entry to NVRAM For more information about how to work with these menus see the documentation that came with the HP Integrity server 128 Preparing Individual Nodes 13 14 15 16 17 Perform this step on all nodes except the head node From the Boot Configuration menu select the Edit OS Boot Order option Do the following a Use the navigation instructions on the screen to move the Netboot entry you just defined to the top of the boot order b If prompted press the Enter key to select the position c Enter x to return to the Boot Configuration menu Perform this step on all nodes including the head node Select the Console Configuration option and set Serial Acpi HWP0002 PNP0A03 0 Pci 1 2 Vt100 9600 as the primary console interface P
44. the Enter key to access the MP If the MP does not respond press the MP reset pin on the back of the MP and try again e Log in to the MP using the default user name and password shown on the screen The MP Main Menu appears MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 4 Enter SL to show event logs Then enter C to clear all log files and Y to confirm 5 Enter CM to display the Command Menu 6 Perform the following steps to ensure that the IPMI over LAN option is set This setting is required for Nagios monitoring a Enter SA b Verify the IPMI over LAN option is enabled c Enable this option if it is disabled d Return to the Command Menu 7 Enter UC and use the menu options to remove the default MP user name and password and create your own unique user name and password HP recommends setting your own user name and password for security purposes The user name must have a minimum of 6 characters and the password must have a minimum of 8 characters You must set the same user name and password on every node You must have this user name and password to access the power management device and console 8 If required set static IP addresses For more information see Setting Static IP Addresses on Integrity Servers page 122 9 Enter PC power cycle and then enter on to turn
45. the administration network 116 Preparing Individual Nodes Setup Procedure Use the Setup Utility to configure the appropriate settings for an HP XC system Perform the following procedure for each workstation in the hardware configuration Change only the values described in this procedure do not change any other factory set values unless you are instructed to do so 1 Dae WN ea Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node and establish a connection to the console Turn on power to the workstation When the node is powering up press the F10 key to access the Setup Utility When prompted press any key to continue Select English as the language Make the appropriate settings for the workstation depending on hardware model e Table 4 42 describes the settings for the xw9300 workstation e Table 4 43 describes the settings for the xw9400 workstation Table 4 42 Setup Utility Settings for xw9300 Workstations Menu Name Submenu Name Option Name Set to This Value Storage Boot Order Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 IDE CD ROM Drive 2 USB device 3 Network Controller 4 Hard Drive C On the head node set the boot order so that the CD ROM is listed before the hard disk Advanced Power Sleep Wake After Power Loss O
46. this utility The BIOS Setup Utility displays the following information about the node BIOS ROM ID BIOS Version BIOS Build Date Record this information for future reference 3 Make the following BIOS settings as provided in Table 4 24 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 93 94 Table 4 24 BIOS Settings for HP ProLiant DL165 G5 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Settings Configuration Bootup Num Lock Disabled Advanced I O Device Configuration Embedded Serial Port IRQ 3F8 Interrupt IRQ 4 S ATA Configuration S ATA Mode S ATA INT13 support Enabled Remote Access Base Address IRQ 3F8h 4 Configuration Serial Port Mode 115200 8 n 1 Redirection of BIOS POST Always Terminal Type ANSI IPMI Configuration LAN Configuration Share NIC Mode Disabled DHCP IP Source Enabled Boot Set the following boot order on all nodes except the head node 1 Removable Devices 2 CD ROM Drive 3 MBA v9 0 6 Slot 0820 4 Hard Drive 5 MBA v9 0 6 Slot 0821 Set the following boot order on the head node 1 Removable Devices 2 CD ROM Drive 3 Hard Drive 4 Select Exit gt Saving Changes to exit the BIOS Setup Utility 5 Repeat this procedure for each HP ProLiant DL165 G5 node in the hardware configuration 4 7 4 Preparing HP Proliant DL365 Nodes On HP ProLiant DL365 servers use the following tools to configure the appropriate settings for an HP XC system e Int
47. to structure it in parallel with the other two networks in order to make the connections easy to maintain and service 3 4 5 InfiniBand Interconnect Connections The supported InfiniBand interconnects do not have the ordering requirements of the Quadrics interconnect but it makes sense to structure it in parallel with the other two networks in order to make the connections easy to maintain and service If you use a dual ported InfiniBand host channel adapter HCA you must connect the IB cable to the lowest numbered port on the HCAs it is labeled either Port 1 or P1 This is necessary so that the OpenFabrics Enterprise Distribution OFED driver activates the IP interface called ibO instead of ib1 58 Making Node and Switch Connections 4 Preparing Individual Nodes This chapter describes how to prepare individual nodes in the HP XC hardware configuration The following topics are addressed e Firmware Requirements and Dependencies page 59 e Ethernet Port Connections on the Head Node page 61 e General Hardware Preparations for All Cluster Platforms page 61 e Setting the Onboard Administrator Password page 62 e Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems page 63 e Preparing the Hardware for CP3000BL Systems page 84 e Preparing the Hardware for CP4000 AMD Opteron Systems page 87 e Preparing the Hardware for CP4000BL Systems page 119 e Prepari
48. you delete the predefined Administrator user for security purposes If you are using the default user name Administrator set the password to be the same as the iLO2 If you create a new user name and password for the iLO2 devices you must make the same settings on all Onboard Administrators Follow this procedure to configure a common password for each active Onboard Administrator 1 Use a network cable to plug in your PC or laptop to the administration network ProCurve switch 2 Make sure the laptop or PC is set for a DHCP network 3 Gather the following information a Look at the Insight Display panel on each enclosure and record the IP address of the Onboard Administrator b Look atthe tag affixed to each enclosure and record the default Onboard Administrator password shown on the tag 4 On your PC or laptop use the information gathered in the previous step to browse to the Onboard Administrator for every enclosure and set a common user name and password for each one This password must match the administrator password you will later set on the ProCurve switches Do not use any special characters as part of the password After you set the Onboard Administrator password prepare the nodes as described in the appropriate section for all the server blade nodes in the enclosure e Preparing the Hardware for CP3000BL Systems page 84 e Preparing the Hardware for CP4000BL Systems page 119 e Preparing the Hardware for CP
49. 00 AMD Opteron Systems 107 Perform the following procedure for each HP ProLiant DL585 and DL585 G2 node in the hardware configuration 108 Preparing Individual Nodes Make the RBSU settings accordingly for each node The settings differ depending on the hardware model generation e Table 4 36 provides the RBSU settings for the HP ProLiant DL585 nodes e Table 4 37 provides the RBSU settings for the HP ProLiant DL585 G2 nodes Table 4 36 RBSU Settings for HP ProLiant DL585 Nodes Menu Name Option Name Set to This Value System Options Embedded Serial Port Disabled Virtual Serial Port COM1 IRQ4 I0 3F8h 3FFh Embedded NIC Port 1 PXE Enabled all nodes except the head Support node Embedded NIC Port 1 PXE Disabled head node only Support Power Regulator for ProLiant Disabled Standard Boot Order IPL Set the following boot order on all nodes except the head node the CD ROM must be listed before the hard drive IPL1 CD ROM IPL2 Floppy Drive A IPL3 PCI Embedded HP NC7782 Gigabit Server Adapter Port 1 IPL4 Hard Drive C On the head node set the boot order so that the CD ROM is listed before the hard disk BIOS Serial Console and EMS BIOS Serial Console Port COM1 IRQ4 I0 3F8h 3FFh BIOS Serial Console Baud Rate 115200 EMS Console Disabled BIOS Interface Mode Command Line Advanced Options page Directory Cache PDC Disabled Table 4 37 lists the RBSU setting for the DL585 G2 nodes Table 4 37 R
50. 1 descending Port 22 is used for the administration port of the head node Dual personality ports Port 24 is used as the interconnect to the Root Console Switch a ProCurve 2650 or ProCurve 2626 model switch As a result of performance considerations and given the number of ports available in the ProCurve 2824 switch the allocation order of ports is Ports 1 10 13 21 Starting with port 1 the ports are used for links from Branch Administration Switches which can include the use of trunking For example if two port trunking is used the first Branch Administration Switch uses port 1 and 2 of the Root Administration Switch A NOTE Trunking is restricted to within the same group of 10 you cannot trunk with ports 10 and 13 HP recommends that all trunking use consecutive ports within the same group 1 10 or 13 21 Starting with port 21 and descending ports are assigned for use by individual root nodes A root node is a node that is connected directly to the Root Administration Switch Ports 11 and 12 are unused Port 23 can be one of the following Console or line monitoring card for the interconnect Connection to the Interconnect Ethernet Switch IES which connects to the management port of multiple interconnect switches Port 24 is used as the interconnect to the Root Console Switch 3 3 6 Root Console Switches The following switches are supported as Root Console Switches for the console branch of t
51. 2 3 Cabling for the Administration Network For server blades the administration network is created and connected through ProCurve model 2800 series switches One switch is designated as the root administration switch and that switch can be connected to multiple branch administration switches if required NIC1 on each server blade is dedicated as the connection to the administration network NIC1 of all server blades connects to interconnect bay 1 on the enclosure The entire administration network is formed by connecting the device either a switch or a pass thru module in interconnect bay 1 of each enclosure to one of the ProCurve administration network switches Non blade server nodes must also be connected to the administration network See Chapter 3 page 45 to determine which port on the node is used for the administration network the port you use depends on your particular hardware model Figure 2 4 illustrates the connections that form the administration network 2 3 Cabling for the Administration Network 37 Figure 2 4 Administration Network Connections Non Blade Server PCI SLOT NIC Non Blade Server PCI SLOT MGT MGT NIC PCI SLOT NIC NIC PCI SLOT C Class Blade Enclosure Admin ProCurve 2800 Series Switch Interconnect bay 8 External Public Network ONBOARD ADMINISTRATOR
52. 200 EMS Console Disable BIOS Interface Mode Command Line Press the Esc key to return to the main menu 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL580 G4 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP ProLiant DL580 G4 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 5 7 Preparing HP Proliant DL580 G5 Nodes 78 Use the following tools to configure the appropriate settings on HP ProLiant DL580 G5 servers e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL580 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 8 shows a rear view of the HP ProLiant DL580 G5 server and the appropriate port assignments for an HP XC system Preparing Individual Nodes
53. 4 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F10 key when prompted to access the BIOS Setup Utility The Lights Out 100i LO 100i console management device is configured through the BIOS Setup Utility The BIOS Setup Utility displays the following information about the node BIOS ROM ID BIOS Version BIOS Build Date Record this information for future reference 3 For each node make the following BIOS settings from the Main window The settings differ depending upon the generation of hardware model e BIOS settings for HP ProLiant DL140 G2 nodes are listed in Table 4 3 e BIOS settings for HP ProLiant DL140 G3 nodes are listed in Table 4 4 Table 4 3 BIOS Settings for HP ProLiant DL140 G2 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Features Numlock Disabled Advanced PCI Device Device Enabled Configuration Ethemet on Board for Ethernet 1 2 Option ROM Scan Enabled Latency Timer 40h Advanced Hyperthreading Disabled Processor Options I O Device Serial Port BMC COM Port Configuration SIO COM Port Disabled Mouse controller Auto Detect Console Redirection Console Redirection Enabled EMS Console Enabled Baud Rate 115 2K Flow Control None Redirection After On BIOS Post IPMI LAN Setting IP Address DHCP Assignment BMC Telnet Service Enabled BMC Ping Response Enabled BMC HTTP Service Enabled Power Wake On Modem Disabled Preparing I
54. 51 S server blade boot order 84 119 defined 19 preparing HP Integrity nodes 136 preparing HP ProLiant nodes 84 119 server blade combinations 21 server blade enclosure c3000 25 c7000 22 156 Index smart array card 70 72 75 78 80 85 96 99 104 110 112 115 120 software RAID documentation 16 mdadm utility 16 storage 45 super root switch 46 49 in large scale system 48 supported cluster platforms 19 supported console management devices 29 supported HP ProLiant server blade models 21 supported interconnects 31 supported server blade combinations 21 supported server models 20 supported switch models 47 switch branch administration 47 54 branch console 47 55 choices 45 connections for workstations 49 port connections 46 47 port connections for large scale systems 48 root administration 47 50 root console 47 51 specialized use 46 super root 46 49 supported models 47 system firmware 59 T telnet enabling on iLO devices 29 telnet access 85 120 trunking 45 port use on large scale systems 49 U utility cabinet 45 V virtual local area network see VLAN VLAN creating 44 defined 41 W Web site HP XC System Software documentation 12 workstation 80 HP xw8200 80 HP xw8400 80 HP xw8600 82 HP xw9300 116 HP xw9400 116 157 x A X C H WP 3 217 C if Printed in the US
55. 6000BL Systems page 136 62 Preparing Individual Nodes 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems Follow the procedures in this section to prepare each node before installing and configuring the HP XC System Software Proceed to the following sections depending on the hardware model e Preparing HP ProLiant DL140 G2 and G3 Nodes page 63 e Preparing HP ProLiant DL160 G5 Nodes page 66 e Preparing HP ProLiant DL360 G4 Nodes page 68 e Preparing HP ProLiant DL360 G5 Nodes page 70 e Preparing HP ProLiant DL380 G4 and G5 Nodes page 72 e Preparing HP ProLiant DL580 G4 Nodes page 75 e Preparing HP ProLiant DL580 G5 Nodes page 78 e Preparing HP xw8200 and xw8400 Workstations page 80 e Preparing HP xw8600 Workstations page 82 4 5 1 Preparing HP ProLiant DL140 G2 and G3 Nodes Use the BIOS Setup Utility to configure the appropriate settings for an HP XC system on HP ProLiant DL140 G2 and DL140 G3 servers For these hardware models you cannot set or modify the default console port password through the BIOS Setup Utility as you can for other hardware models The HP XC System Software Installation Guide describes how to modify the console port password You are instructed to perform the task just after the discover command discovers the IP addresses of the console ports Figure 4 1 shows the rear view of the HP ProLiant DL140 G2 server and th
56. 626 Root Console Switch ccccccccccccssssssseececceeceaeeneseececccececeeueesececeseseaeauenseeeceeeseaeauea 53 ProCurve 2610 24 Root Console SWitch ccccccccccsesssesccccceceeeeueeececceeseaueaeeececeeeseaeaueneseeeeeeseaeauea 54 ProCurve 2848 Branch Administration Switch cccccccccccssssseseececccececceueesececcseseaeaueesseeceeeeeaeaaea 55 ProCurve 2824 Branch Administration Switch ccccccccccccsesssesesceccceceaeeuensececceeceseaueaseeeeeeeeaeaues 55 ProCurve 2650 Branch Console Switch cccccccccccsssssseececcceceeeeseseececccececeeuenesceceseeeaeuaeesseeeeeeeeaeaaea 56 ProCurve 2610 48 Branch Console SWitch cccccccessescccccceceeessesecceccceceaeeuenesceceeeseseeueneseeeeeeseneauea 56 HP ProLiant DL140 G2 and DL140 G3 Server Rear View ccccccccccseeccscceeeccsscesccsscuesesseueeess 63 HP ProLiant DL160 G5 Server Rear VieW cccccccceccsecceescecseeceusecceussscsecceececsecseuseeesesseuseceueesees 66 HP ProLiant DL360 G4 Server Rear VieW ccccccseccseccceececeecseusccceuscsceeceeecessecseusceeeesseuseceuensees 68 HP ProLiant DL360 G5 Server Rear VieW cccccccsecccsecceeececsecceusecceesssceeceecececseeseusceeeesssuseceueesees 70 HP ProLiant DL380 G4 Server Rear VieW cccccccseccseccceececeseseuccceessecueccuscessecseusceeuesseuseceuesees 72 HP ProLiant DL380 G5 Server Rear VieW c ccccccessccsecccesceceeesecceceescscueseeececsesseuseeeeesssueseueesees 73 HP ProLiant DL5
57. 7000 enclosure Front and Rear Views Front View Rear View Figure 1 2 is an illustration showing the location of the device bays power supply bays and the Insight Display at the front of the HP BladeSystem c7000 enclosure Figure 1 2 HP BladeSystem c7000 Enclosure Bay Locations Front View d teuilalo inate OION E Aled ala a aiials COO UTS Device bays 22 Hardware and Network Overview Power supply bays Insight Display For more information see Insight Display page 28 As shown in Figure 1 3 the HP BladeSystem c7000 enclosure can house a maximum of 16 half height or 8 full height server blades The c7000 enclosure can contain a maximum of 6 power supplies and 10 fans Figure 1 3 also illustrates the numbering scheme for the server bays in which server blades are inserted The numbering scheme differs for half height and full height server blades Figure 1 3 HP BladeSystem c7000 Enclosure Bay Numbering for Half Height and Full Height Server Blades Half Height Full Height F The number of fans in the enclosure influences the placement of the server blades Use the following table and the numbering scheme in Figure 1 3 to determine the placement of the server blades in the enclosure based on the number of fans Insert Half Height Server Bladesin Insert Full Height Server Blades in These Number of Fans These
58. 80 G4 Server Rear VieW ccccccccecccsecccesceccseseuceceescsceeecececccsesseueeeeesesugeceuesees 76 HP ProLiant DL580 G5 Server Rear View ccccsccccccssescececsescccecseecececunceeecaeeceeecueeeeesuneeesaueeeees 79 HP xw8200 and xw8400 Workstation Rear View cccccccccccsssseeeseececececueneseecececeseaeaueeeseeeeeeeeaeaae 81 HP xw8600 Workstation Rear View cccccccccccecessssseescecceeceaceseescecceeceaueueneececeseeeaeaaegseeeceseeeaeaaea 82 HP ProLiant DL145 Server Rear VieW cccccccccccsescccsecceesceceeeseuecceeescsceecsuececsecseuseeeeesssueseueesees 87 HP ProLiant DL145 G2 Server Rear VieW cccccccceccccsecceeececeesseuececeescecueceucececsesseusccssesesueeseueesees 89 HP ProLiant DL145 G3 Server Rear View ccccccccccssecceesceceeeceuceceuscscueccescecuecseueseueeseuseceuesees 90 HP ProLiant DL165 G5 Server Rear View ccccceccccsseccceececsseseuccceuecscuecceececsecseusceeeesseueceuesees 93 HP ProLiant DL365 Server Rear VieWw ccccccccsssscccecsesceeecsescceecsscececuecesecsenccsesuunceeeauneceeaaeneess 95 HP ProLiant DL365 G5 Server Rear VieW cccccceeeccsecceeececeeccecsccceesssceeceeececsecseuceseesssueeceunsees 97 List of Figures 4 17 4 18 4 19 4 20 4 21 4 22 4 23 4 24 4 25 4 26 4 27 4 28 4 29 4 30 4 31 B 1 B 2 HP ProLiant DL385 Server Rear View cccccsccsscccscccscccsccusccusccusceuscessceecceecesecesecesceeceesseuseenseens 100 HP ProLiant DL3
59. 85 G2 Server Rear VieW c cccccccesccceesceceeeceesccccescececeeescecuecseeceesessseeceeeesees 101 HP ProLiant DL385 G5 Server Rear View cccccccccesccceesceceeceesccceesceueceeescecuesseuseeeeessseeceeeesees 105 HP ProLiant DL585 Server Rear Viewer eair terate ee beds iad ved seed E a vada a EEEa 106 HP ProLiant DL585 G2 Server Rear VieW ccccccccceeccccesccceecceesccceescececceesceceecseesceeeescseueceeesees 106 HP ProLiant DL585 G5 Server Rear View cccccccesecceesceceeeceusccccescececeeescecuesseeseeeeesesaueceeesees 111 HP ProLiant DL785 G5 Server Rear View ccccccccceececcesceceecceuscccesscececeescecuesseusceeeesssueceuesees 113 xw9300 Workstation Rear VieW ccccccccccssssccccesssccsseussccssuusscesseescessuuscessuusscessuusscessuusecessuueseeees 116 xw9400 Workstation Rear View o c cccccccccccssscccsesssccsscsssccsseusscessescecesseuscessuuscessuuecesseussessuueseeees 116 PP integrity 1x 1620 Server Rear VieW sesiis sesini tadi ver tess ee i keria E gered rege 123 HP Integrity 2000 Server Rear VOW ss r Sieiro ieit i e seeni stress EE GS eat 123 HP Integrity rx2620 Server Redt VICW ses vsiscesssecseassevonnesh vedere ev eveosn een aa e ndopsniese tags aeara aiaia 125 HP Integrity rx2600 Servet Rear VIEW imisereiiesiroteieii aee a a a a iis 127 HP Integrity rx4640 Server Rear View ssesssssssesssssessseesserssessrsssnressstesseesseessnsessnessseenseesseesse 129 HP Integrity rx8620 Core IO Board
60. 9 1 1 Supported Cluster Platforms x tesisevedvpstdaenentestvenenbontedasscsveaseisabubapscctennphastabaniunvetepiedbibennsevenapiedieapesans 19 1 1 1 Supported Processor Architectures and Hardware Model cccccccsssccessceessesseeesseeesseees 19 1 1 2 Supported Server Blade Combatants cso sscenccagtstoscenedersunceatpevurimenstadestenvevsbepieneusesouertetesdens 21 1 2 Server Blade Enclosure CGOmiPOme nts tcasnecioxsssichetissnciesssncteaanivalveasoscantasnnace ei EEEk Nr EER rE saeara 22 1 2 1 HP BladeSystem C7000 Fmiclostit Gs sissssssinceiessescesncsavgetostondestedavectssoa vecdonbeveepinsedgannanedsinpaibendestane 22 1 2 2 HP BladeSystent C3000 Bm Ws tte cssnasiivisesocenvenctgeoieensamneasn ee a E NEE nes seen NES N 25 1 2 3 HP BladeSystem c Class Onboard Administrator s ssessessseissesessserssisessserssrsersseessenseeseense 27 A Ae TSW NE DA o i e E EEE T pninqatiacxestamnaamedeauosan atin deoatanaskeamnsemetuauenAtwne 28 1 3 Server Blade Mezzanine CardScan L EA i Ta i 28 1 4 Server Blade Interconnect Modules 0000seoe0eeeseeeeessseresssecossssresseecesssscesssecessseeesseeeeseseersssseessseeess 28 1 5 Supported Console Management Devices ssssssessessisisssestesssttsssststtstssestensssenssssenteseesesrenrenets 29 1 6 Administration Network OvervieWw cccccccccccssecccccccscccssucsccsscueeccsseuceessuueecssuuuecessuueessssuueesessuuensess 30 1 7 Administration Network Console Branch cccccccceccccccses
61. Administrator HP XC and Related HP Products Information 13 HP Server Blade c7000 Enclosure HP BladeSystem c3000 Enclosure Related Information 14 This section provides useful links to third party open source and other related software products Supplementary Software Products This section provides links to third party and open source software products that are integrated into the HP XC System Software core technology In the HP XC documentation except where necessary references to third party and open source software components are generic and the HP XC adjective is not added to any reference to a third party or open source command or product name For example the SLURM srun command is simply referred to as the srun command The location of each web address or link to a particular topic listed in this section is subject to change without notice by the site provider http www platform com Home page for Platform Computing Corporation the developer of the Load Sharing Facility LSF LSF HPC with SLURM the batch system resource manager used on an HP XC system is tightly integrated with the HP XC and SLURM software Documentation specific to LSF HPC with SLURM is provided in the HP XC documentation set Standard LSF is also available as an alternative resource management system instead of LSF HPC with SLURM for HP XC This is the version of LSF that is widely discussed on the Platform web address For your convenience the f
62. BL Systems 121 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems Follow the procedures in this section to prepare HP Integrity servers before installing and configuring the HP XC System Software The following topics including setup information for specific hardware models are discussed e Setting Static IP Addresses on Integrity Servers page 122 e Preparing HP Integrity rx1620 and rx2600 Nodes page 122 e Preparing HP Integrity rx2620 Nodes page 125 e Preparing HP Integrity rx2660 Nodes page 127 e Preparing HP Integrity rx4640 Nodes page 129 e Preparing HP Integrity rx8620 Nodes page 131 About the EFI Boot Manager User Interface Two user interfaces for the EFI Boot Manager utility are available the Enhanced and Legacy interfaces The setup instructions described here are based on the Enhanced interface To change the user interface on the system use the Set User Interface menu option 4 9 1 Setting Static IP Addresses on Integrity Servers 7 IMPORTANT Integrity nodes are configured with static addresses not DHCP On an HP XC system with one or more such nodes you must configure the Management Processor MP on all nodes for static IP addresses rather than DHCP You configure IP addresses from the Command Menu of the Management Processor MP The LC command configures the IP address subnet mask and default gateway address for the HP Integrity MP interfa
63. BSU Settings for HP ProLiant DL585 G2 Nodes Menu Name Option Name Set to This Value System Options Embedded Serial Port Disabled Virtual Serial Port COM1 Embedded NICs NIC 1 boot On all nodes except the head node set this value Options to PXE boot Embedded NICs NIC 2 boot Disabled head node only Options Power Regulator for OS Control Mode ProLiant 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 109 Table 4 37 RBSU Settings for HP ProLiant DL585 G2 Nodes continued Menu Name Option Name Set to This Value Standard Boot Order IPL Set the following boot order on all nodes except the head node the CD ROM must be listed before the hard drive e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4PCI Embedded HP NC373i Multifunction Gigabit Adapter e IPL 5 Hard Drive C Set the following boot order on the head node e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4 Hard Drive C BIOS Serial Console and BIOS Serial Console Port com1 EMS BIOS Serial Console Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line Advanced Linux x86_64 HPET Option Enabled 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each node HP ProLiant DL585 G2 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP
64. Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line Press the Esc key to return to the main menu 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL385 and DL385 G2 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP ProLiant DL385 and DL385 G2 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 7 7 Preparing HP Proliant DL385 G5 Nodes On HP ProLiant DL385 G5 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL385 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 19 shows a rear view of the HP ProLiant DL385 G5 server and the appropriate port assignments
65. Bays Bays 4 fans 1 2 9 10 1or2 6 fans 1 2 3 4 9 10 11 12 1 2 3 4 8 or 10 fans all server bays all server bays 1 Only two servers are supported in this configuration They can be inserted in any two of these bays Figure 1 4 is an illustration showing the location of the fan bays interconnect bays onboard administrator bays the power supply exhaust vent and the AC power connections at the rear of the HP BladeSystem c7000 enclosure This figure includes an inset showing the serial connector onboard administrator iLO port and the enclosure Uplink and Downlink ports 1 2 Server Blade Enclosure Components 23 Figure 1 4 HP BladeSystem c7000 Enclosure Bay Locations Rear View cee msa fl wo Fan Bays Interconnect Bay 7 Interconnect Bay 1 Interconnect Bay 8 Interconnect Bay 2 Onboard Administrator Bay 1 Interconnect Bay 3 Onboard Administrator Bay 2 Interconnect Bay 4 Power Supply Exhaust Vent Interconnect Bay 5 AC Power Connections Interconnect Bay 6 ERE EHE Onboard Administrator Integrated Lights Out port Serial connector Enclosure Downlink port Enclosure Uplink port General Configuration Guidelines The following are general guidelines for configuring HP BladeSystem c7000 enclosures e Up to four enclosures can be mounted in an HP 42U Infrastructure Rack e Ifan enclosure is not fully populated with fans and power supplies see the positioning guidelines i
66. C System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 17 shows a rear view of the HP ProLiant DL385 server and the appropriate port assignments for an HP XC system Figure 4 17 HP ProLiant DL385 Server Rear View QOKOAOWQGOAHQAOHOAAOAO re RO RORCRERORORORG SA E FERRE e SR BLORSLOLORO LER TaB T O80 HPTC 0145 The callouts in the figure enumerate the following 1 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 2 This port is the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 3 This port is the Ethernet connection to the Console Switch On the back of the node this port is marked with the acronym iLO Figure 4 18 shows a rear view of the HP ProLiant DL385 G2 server and the appropriate port assignments for an HP XC system 100 Preparing Individual Nodes Figure 4 18 HP ProLiant DL385 G2 Server Rear View O ce E anne OO
67. Controller After you make this setting make sure the Primary SATA Controller and Secondary SATA Controller settings are set to Enabled Boot Order Set the following boot order on all nodes except the head node 1 Optical Drive 2 USB device Broadcom Ethernet controller A w Hard Drive g Broadcom Ethernet controller On the head node set the boot order so that the Optical Drive is listed before the hard disk Select File gt Save Changes amp Exit to exit the Setup Utility Repeat this procedure for each workstation in the hardware configuration Turn off power to all nodes except the head node Follow the software installation instructions in the HP XC System Software Installation Guide to install the HP XC System Software 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 83 4 6 Preparing the Hardware for CP3000BL Systems Perform the following tasks on each server blade in the hardware configuration after the head node is installed and the switches are discovered 84 Set the boot order Create an iLO2 user name and password Set the power regulator Configure smart array devices Use the Onboard Administrator the iLO2 web interface and virtual media to make the appropriate settings on HP ProLiant Server Blades A NOTE The following setup procedure continues from the procedure in Setting the Onboard Administrator Password page 62 in which you used a browser to l
68. FULL HEIGHT BLADE Gigabit Ethernet Interconnect Switch m a lt rid a T 9 wi I mi E i lt T 2 4 Cabling for the Console Network The console network is part of the private administration network within an HP XC system and it is used primarily for managing and monitoring the node consoles On a small cluster the console management devices can share a single top level ProCurve 2800 root administration switch On larger hardware configurations that require more ports the console network is formed with separate ProCurve model 2600 series switches You arrange these switches in a hierarchy similar to the administration network One switch is designated as the root console switch and that switch can be connected to multiple branch console switches The top level root console switch is then connected to the root administration switch HP server blades use iLO2 as the console management device Each iLO2 in an enclosure connects to the Onboard Administrator To form the console network connect the Onboard Administrator of each enclosure to one of the ProCurve console switches Non blade server nodes must also be connected to the console network See Chapter 3 page 45 to determine which port on the node is used for the console network the port you use depends on your particular hardware model Figure 2 5 illustrates the connections that form the console network 38 Cabling S
69. Firmware Requirements and Dependencies 59 Table 4 1 Firmware Dependencies continued Hardware Component Firmware Dependency HP ProLiant BL680c G5 iLO2 system BIOS Onboard Administrator OA CP4000 HP ProLiant DL145 LO 100i system BIOS HP ProLiant DL145 G2 LO 100i system BIOS HP ProLiant DL145 G3 LO 100i system BIOS HP ProLiant DL165 G5 LO 100i system BIOS HP ProLiant DL365 iLO2 system BIOS HP ProLiant DL365 G5 iLO2 system BIOS HP ProLiant DL385 iLO2 system BIOS HP ProLiant DL385 G2 iLO2 system BIOS HP ProLiant DL385 G5 iLO2 system BIOS HP ProLiant DL585 iLO2 system BIOS HP ProLiant DL585 G2 iLO2 system BIOS HP ProLiant DL585 G5 iLO2 system BIOS HP ProLiant DL785 G5 iLO2 system BIOS CP4000BL HP ProLiant BL465c iLO2 system BIOS OA HP ProLiant BL465c G5 iLO2 system BIOS OA HP ProLiant BL685c iLO2 system BIOS OA HP ProLiant BL685c G5 iLO2 system BIOS OA CP6000 HP Integrity rx1620 Management Processor MP BMC Extensible Firmware Interface EFI HP Integrity rx2600 MP BMC EFI system HP Integrity rx2620 MP BMC EFI system HP Integrity rx2660 MP BMC EFI system HP Integrity rx4640 MP BMC EFI system HP Integrity rx8620 MP BMC EFI system CP6000BL HP Integrity BL860c Server Blade Full height MP OA Switches ProCurve 2824 switch Firmware version ProCurve 2848 switch Firmware version ProCurve 2650 switch Firmware version ProCurve 2610 switch Firmware vers
70. G2 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable on Settings CLI Serial CLI 115200 Press the F10 key to save the setting Speed bits seconds Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 105 4 7 8 Preparing HP Proliant DL585 and DL585 G2 Nodes On HP ProLiant DL585 and DL585 G2 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL585 and DL585 G2 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access
71. HP XC System Software Hardware Preparation Guide Version 3 2 1 Mp H P Part Number AXCHWP 321 Published October 2008 Copyright 2003 2004 2005 2006 2007 2008 Hewlett Packard Development Company L P The information contained herein is subject to change without notice The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services Nothing herein should be construed as constituting an additional warranty HP shall not be liable for technical or editorial errors or omissions contained herein Intel Pentium Intel Inside and the Intel Inside logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries AMD and AMD Opteron are trademarks or registered trademarks of Advanced Micro Devices Inc FLEXIm and Macrovision are trademarks or registered trademarks of Macrovision Corporation InfiniBand is a registered trademark and service mark of the InfiniBand Trade Association Intel Itanium and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries Linux is a U S registered trademark of Linus Torvalds LSF and Platform Computing are trademarks or registered trademarks of Platform Computing Corporation Lustre is a registered trademark of Cluster File Systems Inc Myrinet and Myricom are registered trademarks of M
72. Interconnect Bay Port Mapping Connections between the server blades and the interconnect bays are hard wired Each of the 8 interconnect bays in the back of the enclosure has a connection to each of the 16 server bays in the front of the enclosure The built in NIC or mezzanine card into which the interconnect blade connects depends on which interconnect bay it is plugged into Because full height blades consume two server bays they have twice as many connections to each of the interconnect bays Hardware and Network Overview See the HP BladeSystem Onboard Administrator User Guide for illustrations of interconnect bay port mapping connections on half and full height server blades 1 5 Supported Console Management Devices Table 1 3 lists the supported console management device for each hardware model within each cluster platform The console management device provides remote access to the console of each node enabling functions such as remote power management remote console logging and remote boot HP workstation models do not have console ports HP ProLiant servers provide remote management features through a baseboard management controller BMC The BMC enables functions such as remote power management and remote boot HP ProLiant BMCs comply with a specified release of the industry standard Intelligent Platform Management Interface IPMI HP XC supports two IPMI compliant BMCs integrated lights out iLO and iLO2 and Lights Out 100i
73. L785 G5 Nodes page 113 e Preparing HP xw9300 and xw9400 Workstations page 116 4 7 1 Preparing HP Proliant DL145 Nodes On an HP ProLiant DL145 server use the following tools to configure the appropriate settings for an HP XC system e BIOS Setup Utility e Intelligent Platform Management Interface IPMI Utility Figure 4 11 shows the rear view of the HP ProLiant DL145 server and the appropriate port assignments for an HP XC system Figure 4 11 HP ProLiant DL145 Server Rear View Z02000 ES s ESSE E E E E ee q PCI X 133 Box ORORORSROZOROY z 2 8 88 ge Beers 2e Ooa o 58 ENI e OW o MM IBY 5005 oS a ERORO OE Es The callouts in the figure enumerate the following 1 The console Ethernet port is the connection to the Console Switch branch or root 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection 3 NIC1 is the connection to the Administration Switch branch or root It corresponds to etho in Linux if there are no additional optional Ethernet ports installed in expansion slots On the HP ProLiant DL145 server NIC1 is the port on the right labeled with the number 1 Setup Procedure Perform the following procedure from the BIOS Setup Utility for each HP ProLiant DL145 node in the hardware configuration 1 Use the instructions in the accompa
74. LOT NIC NIC PCI SLOT NIC C Class Blade Enclosure Interconnect bay 1 Interconnect bay 2 Interconnect bay 3 Interconnect bay 4 Interconnect bay 5 Gigabit Ethernet Interconnect Switch Interconnect bay 6 Interconnect bay 7 Interconnect bay 8 External Public Network ONBOARD ADMINISTRATOR Admin ProCurve 2800 Series Switch Console ProCurve 2600 Series Switch 2 5 2 Configuring an InfiniBand Interconnect An InfiniBand interconnect requires one or more external InfiniBand switches with at least one managed switch to manage the fabric Systems using an InfiniBand interconnect require you to install an InfiniBand mezzanine card into mezzanine bay 2 of each server blade to provide a connection to the InfiniBand interconnect network The InfiniBand card in mezzanine bay 2 connects to the double wide InfiniBand switch in interconnect bays 5 and 6 on the enclosure The entire interconnect network is formed by connecting the InfiniBand switches in interconnect bays 5 and 6 of each enclosure to one of the InfiniBand interconnect switches Non blade server nodes also require InfiniBand cards and must also be connected to the interconnect network Figure 2 7 illustrates the connections for an InfiniBand interconnect Figure 2 7 InfiniBand Interconnec
75. O pol MGT NIC NIC PCI SLOT Admin ProCurve 2800 Series Switch Console ProCurve 2600 Series Switch Gigabit Ethernet Interconnect Switch External Public Network 2 6 3 Configuring the External Network Option 3 Non Gigabit Ethernet Interconnect Clusters The administration network requires only one network interface NIC1 on clusters that do not use Gigabit Ethernet as the interconnect that is they use InfiniBand or the interconnect on the administration network On these non Gigabit Ethernet interconnect clusters you have two configuration methods to configure an external network connection and the option you choose depends on whether the collection of nodes requiring external connections includes half height server blades e If only full height server blades require external connections you can use NIC3 for the external network This is similar to the way the external connection is configured in Figure 2 8 page 42 and it saves the cost of an additional interconnect device in bay 2 e If half height server blades require external connections you cannot use NIC3 because half height server blades do not have a third NIC In this case you must use NIC2 as the external connection as shown in Figure 2 8 page 42 This configuration requires an Ethernet interconnect module to be present in bay 2 Figure 2 8 page 42 also shows the use of built in NICs in the non blade server nodes for the ext
76. OR 9280n LOZORNO RRRS 2882 E02 O05 fe OROORO OOOO BOOTS 9 Ja LE y The callout in the figure enumerates the following 1 This port is used for the connection to the administration network Setup Procedure Use the Setup Utility to configure the appropriate settings for an HP XC system Perform the following procedure for each workstation in the hardware configuration Change only the values that are described in this procedure do not change any other factory set values unless you are instructed to do so ON i RGT PO Establish a connection to the console by connecting a monitor and keyboard to the node Turn on power to the workstation When the node is powering on press the F10 key to access the Setup Utility When prompted press any key to continue Select English as the language Make the following BIOS settings for each workstation in the hardware configuration BIOS settings differ depending upon the workstation model e BIOS settings for HP xw8200 workstations are listed in Table 4 17 e BIOS settings for HP xw8400 workstations are listed in Table 4 18 Table 4 17 BIOS Settings for xw8200 Workstations Menu Name Submenu Name Option Name Set to This Value Storage Boot Order Set the following boot order on all nodes except the hea
77. Root Switch The connection can be as simple as one wire and as complex as four See Trunking and Switch Choices page 45 for more information about trunking and the Super Root Switch You must configure trunks on both switches before plugging in the cables between the switches Otherwise a loop is created between the two switches Figure 3 4 illustrates a ProCurve 2848 Super Root Switch Figure 3 4 ProCurve 2848 Super Root Switch Ports 1 3 5 and 7 are the first four ports located on the top row LIT A TTT Sd 2880 L L L sey L seep Ports 2 4 6 and 8 are the first four ports 10 100 1000 Base TX RJ 45 Ports located on the bottom row Gigabit Ethernet Ports Table 3 2 shows how ports are allocated for large scale systems with multiple regions Table 3 2 Trunking Port Use on Large Scale Systems with Multiple Regions Trunking Type 4 wire Trunking Ports Used on Super Root Switch Ports Used on Root Administration Switch Region 1 1 through 4 43 through 46 Region 2 5 through 8 43 through 46 Region 3 13 through 16 43 through 46 2 wire Trunking Region 1 1 and2 45 and 46 Region 2 3 and 4 45 and 46 Region 3 5and 6 45 and 46 Region 4 7 and 8 45 and 46 Region 5 9 and 10 45 and 46 Region 6 13 and 14 45 and 46 3 3 Switches 49 3 3 5 Root Administration Switch 50 The Root Administration Sw
78. S Serial Console Port COM1 IRQ4 10 3F8h 3FFh EMS BIOS Serial Console Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line 1 Asmall blue dialog box near the bottom left side of the screen indicates the current setting You can make only one setting per node Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence Repeat this procedure for each HP ProLiant DL365 G5 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP ProLiant DL365 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 99 Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 7 6 Preparing HP Proliant DL385 and DL385 G2 Nodes On HP ProLiant DL385 and DL385 G2 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL385 and DL385 G2 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP X
79. See also improved availability availability set The collection of files and directories that represents the common files and configuration data that are applied to all nodes in an HP XC system A component of the Administration Network A switch that is uplinked to the root switch and receives physical connections from multiple nodes A set of independent computers combined into a unified system through system software and networking technologies The external cluster host name supported by LVS which enables inbound connections without having to know individual nodes names to connect and log in to the HP XC system Configuration and management database Constructed during HP XC system installation the CMDB is a MySQL database that stores information about the nodes hardware and software configuration and network connectivity This database runs on the node with the node management role A node that is assigned only with the compute role and no other Jobs are distributed to and run on nodes with the compute role no other services run on a compute node See CMDB A component of the administration network The half branch of the administration network that contains all of the console ports of the nodes of the HP XC system This branch is established as a separate branch to enable some level of partitioning of the administration network to support specific security needs Dynamic Host Control Protocol A protocol that dynamically a
80. a Gigabit Ethernet Ports 1 10 1008ase TX RJ 45 Ports The callouts in the figure enumerate the following 1 Port 22 must be reserved for an optional connection to the console port on the head node 2 Port 25 is reserved 3 Port 26 is the Gigabit Ethernet link to the Root Administration Switch Allocate the ports on this switch for consistency with the administration switches as follows e Ports 1 10 13 21 Starting with port 1 the ports are used for links from Branch Console Switches Trunking is not used Starting with port 21 and in descending order ports are assigned for use by individual nodes in the utility cabinet Nodes in the utility cabinet are connected directly to the Root Administration Switch A NOTE There must be at least one idle port in this set to indicate the dividing line between branch links and root node administration ports e Ports 11 12 23 and 24 are unused 3 3 7 Branch Administration Switches 54 The Branch Administration Switch of an HP XC system can be either a ProCurve 2848 switch or a ProCurve 2824 switch Figure 3 11 shows the ProCurve 2848 switch In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Making Node and Switch Connections Figure 3 11 ProCurve 2848 Branch Administration Switch Connections to Node Administration Ports
81. ades are not included in this example Because NIC1 and NIC3 on a full height server blade are connected to interconnect bay 1 you must use VLANs on the switch in that bay to separate the external network from the administration network Also in this example PCI Ethernet cards are used in the non blade server nodes If the hardware configuration contains non blade server nodes see Chapter 4 page 59 for information on which port to use for the external network Figure 2 8 External Network Connections Full Height Server Blades and NIC1 and NIC2 in Use Non Blade Server PCI SLOT Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT MGT NIC NIC PCI SLOT Ethernet PCI Cards C Class Blade Enclosure Admin ProCurve 2800 Series Switch erconnect_bay_1 In Nei OMINVLAN L NIC 2 EXTERNAL VLAN w Interconnect bay 2 lt NIC 3 NIC 4 Interconnect bay 3 Console ProCurve 2600 Series Switch MEzz 1 Interconnect bay 4 7 MEzz 2 pre 2 MEZz3 nterconnect bay 5 iLO2 Gigabit Ethernet Interconnect Switch nterconnect bay 6 nterconnect bay 7 nterconnect bay 8 External Public Network ONBOARD ADMINISTRATOR wm Q lt a bE o m z Ww a lt x ES 2 6 2 Configuring the External Network Option 2 42 Figure 2 9 page 43 assumes that NIC1 and NIC2 are alre
82. ady in use for the administration and interconnect networks This situation requires a third NIC for the external network but unlike Figure 2 8 page 42 this hardware configuration includes half height server blades Therefore to make another Ethernet NIC available you must add an Ethernet card to mezzanine bay 1 on each server blade that requires an external connection You must also install an Ethernet interconnect module in interconnect bay 3 for these cards In addition PCI Ethernet Cards are used in the non blade server nodes If the hardware configuration contains non blade server nodes seeChapter 4 page 59 for information on which port to use for the external network Cabling Server Blades Figure 2 9 External Network Connections Half Height Server Blades and NIC1 and NIC2 in Use Non Blade Server PCI SLOT Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT Ethernet PCI Cards C Class Blade Enclosure NIC 1 NIC 2 NIC 3 NIC 4 MEZZ 1 MEZZ 2 FULL HEIGHT BLADE MEZZ 3 iLO2 NIC 1 NIC 2 MEZZ 1 MEZZ 2 HALF HEIGHT BLADE iLO2 Ethernet Mezzanine Cards Interconnect bay 1 Interconnect bay 2 Interconnect bay 3 Interconnect bay 4 Interconnect bay 5 Interconnect bay 6 Interconnect bay 7 Interconnect bay 8 O gt is 22 a O ZS Ra 40
83. age The node from which a standard file system image is created The golden image is distributed by the image server In a standard HP XC installation the head node acts as the image server and golden client A collection of files created from the golden client file system that are distributed to one or more client systems Specific files on the golden client may be excluded from the golden image if they are not appropriate for replication The collection of directories and files that represents all of the software and configuration data of an HP XC system The software for any and all nodes of an HP XC system can be produced solely by the use of this collection of directories and files The single node that is the basis for software installation system configuration and administrative functions in an HP XC system There may be another node that can provide a failover function for the head node but HP XC system has only one head node at any one time The name given to a computer Lowercase and uppercase letters a z and A Z numbers 0 9 periods and dashes are permitted in host names Valid host names contain from 2 to 63 characters with the first character being a letter A node that has more storage available than the majority of server nodes in an HP XC system This storage is frequently externally connected storage for example SAN attached storage When configured properly an I O server node makes the additional storage availabl
84. ager LIM and master batch daemon mbat chd run on the LSF master host Each system has one master host to do all job scheduling and dispatch If the master host goes down another LSF server in the system becomes the master host Load Sharing Facility for High Performance Computing integrated with SLURM The batch system resource manager on an HP XC system that is integrated with SLURM LSF HPC with SLURM places a job in a queue and allows it to run when the necessary resources become available LSF HPC with SLURM manages just one resource the total number of processors designated for batch processing LSF HPC with SLURM can also run interactive batch jobs and interactive jobs An LSF interactive batch job allows you to interact with the application while still taking advantage of LSF HPC with SLURM scheduling policies and features An LSF HPC with SLURM interactive job is run without using LSF HPC with SLURM batch processing features but is dispatched immediately by LSF HPC with SLURM on the LSF execution host See also LSF execution host 149 LVS M Management Processor master host MCS Modular Cooling System module modulefile MP MPI MySQL NAT Network Address Translation Network Information Services NIS NIS client NIS master server NIS slave server OA 150 Glossary Linux Virtual Server Provides a centralized login capability for system users LVS handles incoming login requests and direct
85. and Changed Information in This Edition e This document was updated to include the following servers CP3000BL platform HP ProLiant BL2x220c G5 server blade Typographic Conventions This document uses the following typographical conventions or A percent sign represents the C shell system prompt A dollar sign represents the system prompt for the Korn POSIX and Bourne shells A number sign represents the superuser prompt audit 5 A manpage The manpage name is audit and it is located in Section 5 Command A command name or qualified command phrase Computer output Text displayed by the computer Intended Audience 11 Ctri x ENVIRONMENT VARIABLE ERROR NAME Key Term User input Variable WARNING CAUTION IMPORTANT NOTE A key sequence A sequence such as Ctrl x indicates that you must hold down the key labeled Ctrl while you press another key or mouse button The name of an environment variable for example PATH The name of an error usually returned in the errno variable The name of a keyboard key Return and Enter both refer to the same key The defined use of an important word or phrase Commands and other text that you type The name of a placeholder in a command function or other syntax display that you replace with an actual value The contents are optional in syntax If the contents are a list separated by you can choose one of the items The contents are requ
86. and password which must match the user name and password you set on the Onboard Administrator Do not use any special characters as part of the password You use this user name and password whenever you need to access the console port with the telnet cp nodename command Preparing Individual Nodes f The Onboard Administrator automatically creates user accounts for itself prefixed with the letters OA to provide single sign on capabilities Do not remove these accounts Enable telnet access a Inthe left frame click Access b Click the control to enable Telnet Access c Click the Apply button to save the settings Click the Virtual Devices tab and make the following settings a For every node except the head node select No to Automatically Power On Server because you do not want to automatically turn on power to the node b Click the Submit button c Inthe left frame click on the Power Regulator link d Select Enable HP Static High Performance Mode e Click the Apply button to save the settings Configure disks into the smart array from the remote graphics console All server blades have smart array cards you must add the disk or disks to the smart array before attempting to image the node To set up the smart array device click the Remote Console tab on the virtual console page of the iLO2 Web Administration Utility and then do one of the following depending on the browser type Internet Explorer If you are
87. are instructed to perform the task just after the discover command discovers the IP addresses of the console ports Figure 4 14 shows a rear view of the HP ProLiant DL165 G5 server and the appropriate port assignments for an HP XC system Figure 4 14 HP ProLiant DL165 G5 Server Rear View The callouts in the figure enumerate the following 1 This port is used for the connection to the Administration Switch branch or root On the rear of the node this port is marked as NIC1 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the rear of the node this port is marked as NIC2 3 The port labeled LO100i is used for the connection to the Console Switch Setup Procedure Perform the following procedure for each HP ProLiant DL165 G5 node in the HP XC system Change only the values that are described in this procedure do not change any factory set values unless you are instructed to do so 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F10 key when prompted to access the BIOS Setup Utility You configure the Lights Out 100i LO 100i console management device using
88. are prompted to configure the disks into the smart array Select View Logical Drives to determine if a logical drives exists If a logical drive is not present create one If you create a logical drive exit the SmartArray utility and power off the node Do not let it try to boot up Specific smart array configuration instructions are outside the scope of this document See the documentation that came with your model of HP ProLiant server for more information 120 Preparing Individual Nodes 10 11 12 Perform this step for HP ProLiant BL685c nodes proceed to the next step for all other hardware models On an HP ProLiant BL685c node watch the screen carefully during the power on self test and press the F9 key to access the ROM Based Setup Utility RBSU to enable HPET as shown Table 4 45 Table 4 45 Additional BIOS Setting for HP ProLiant BL685c Nodes Menu Name Option Name Set To This Value Advanced Linux x86_64 HPET Enabled Press the F10 key to exit the RBSU The server automatically restarts Use the virtual power functions to turn off power to the server blade Close the iLO2 utility Web page Repeat this procedure from every active Onboard Administrator and make the same settings for each server blade in each enclosure After preparing all the nodes in the enclosures return to the HP XC System Software Installation Guide to discover all the nodes and enclosures in the HP XC system 4 8 Preparing the Hardware for CP4000
89. asainrammniintednuensnia 41 2 6 2 Configuring the External Network Option 2 isssssusesceisenecsinssnnsaeiastnceneineninaynesverncnisa vias ti wnes 42 2 6 3 Configuring the External Network Option 3 Non Gigabit Ethernet Interconnect MVS i S EAEE OIE E E E sieves deneonavacnviwetenin oaveenteis gauesoadacniemesdad 43 2 6 4 Creating NV AUIS asic sistas calacen v pues nic esa E EER A E R EE E 44 S Making Node gna Switch Comme ties jas sicseusscsicsnesninsicana lina cssesnvistonnnizanorubawvessnus ia 45 Bek CADIS E EEEE E O EA EE E A OE AEO ET SE E EEE 45 3 2 Trunking and Switch Ch iceS sistini iaie aiaa aii E aaa el Ean E a E RES 45 Be SWE NE nrn E wade E E E E EE N eveles nabeddveesees 46 3 3 1 Specialized Switch USC sacscsniccsnnagrsa sce reqotanssatasincinasesnstdaseicsieaesminiehasintvmaswieni flail biosensors intiere 46 3 3 2 Administrator Passwords on ProCurve Switches ccccccccccceesccscceeesesseeusessceeeesesseuusessseuueess 47 Table of Contents 3 3 3 3 Switch Port COnnections cccccsseccccecccescecssccecsccccuscecseccecucccesssecseceeecesueccecseceuesseasecesuceeauess 47 3 3 3 1 Switch Connections and HP Workstations c ccccccccccceceseeeseececececesueseseseecceeceauenensses 49 JIA S per Root SWIC hi speed stR ESERE AAEE DEENTRAR N EE REENA SEEE EEE AEEA EEEE SEER 49 33 5 Root Administrations SWU eeren e e ede E diavvacces 50 3 3 6 Root Console SwitChes vss useen i e a e a a ae e E 51 3 3 6 1 ProCurve 2650
90. at is pertinent across multiple machines in a network The data is collected in a domain within which it is accessible and relevant The most common use of NIS is to maintain user account information across a set of networked hosts Any system that queries NIS servers for NIS database information Clients do not store and maintain copies of the NIS maps locally for their domain A system that stores the master copy of the NIS database files or maps for the domain in the var yp DOMAIN directory and propagates them at regular intervals to the slave servers Only the master maps can be modified Each domain can have only one master server A system that obtains and stores copies of the master server s NIS maps These maps are updated periodically over the network If the master server is unavailable the slave servers continue to make the NIS maps available to client systems Each domain can have multiple slave servers distributed throughout the network The enclosure management hardware software and firmware that is used to support all of the managed devices contained within the HP BladeSystem c Class enclosure onboard administrator P parallel application PXE R remote graphics software resource management role RGS role Root Administration Switch root node RPM 5 scalable visualization array serial application server blade SLURM backup controller SLURM master controller SMP See OA
91. ated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL360 G4 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 3 shows a rear view of the HP ProLiant DL360 G4 server and the appropriate port assignments for an HP XC system Figure 4 3 HP ProLiant DL360 G4 Server Rear View EZ em The callouts in the figure enumerate the following 1 The iLO Ethernet is the port used as the connection to the Console Switch 2 NIC1 is used for the connection to the Administration Switch branch or root 3 NIC2 is used for the external connection Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL360 G4 node in the HP XC system 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 For each node make the iLO settings listed in Table 4 6 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test Preparing Individual Nodes 5 Watch the screen
92. ations Rear VieW cccccccccssseseesecsseeseeesseseeensenaeenes 24 HP BladeSystem c3000 Enclosure Front and Rear Views esssesssssesssesssrssrsseesseesresresserssesees 25 HP BladeSystem c3000 Enclosure Tower Model i sccrcitoisraenosiieaussduiiside ssmmctaraatonmioriaueunbsuedates suieut 25 HP BladeSystem c3000 Enclosure Bay Locations Front VieW ccccesesscesesssesseseeseeeenseesesees 26 HP BladeSystem c3000 Enclosure Bay Namie gy wv stcsicvssuinsieriassnedencesdividenwaaekentaWaneaiiileiaibiuecn 26 HP BladeSystem c3000 Enclosure Bay Locations Rear View c cscis csccssncataetmcisceisnnseeasmcets 27 Server Blade Insight Display ssiccsncs sevcasssdnenrsvsinvaaddasevardorsvannesnevares idovsianssnvaraoiauvaaeetaneaanossnneneenguanss 28 Administration Network Console Branch Without HP Server Blades ecceeesseeeeeeeeeeeeees 31 Interconnection Diagram for a Small HP XC Cluster of Server Blades c ccscesseeeseeseeeeees 35 Interconnection Diagram for a Medium Sized HP XC Cluster of Server Blades cceeee 36 Interconnection Diagram for a Large HP XC Cluster of Server Blades c ccceesceeseeseeneees 37 Administration Network Connections cccccccccsessssseececcceceausuesecceccceceaueueneecececeseusaueaeseeceeeseneaaea 38 Console Network Conmnections ccccccccccsssesescecccececceseseseeceeccecesuesseecccceeceaueaeneeeeeeseseauaueneseeeeeess 39 Gigabit Ethernet Interconnect Girte C
93. ave the Speed setting bits seconds 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL380 node in the HP XC system 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 73 74 Make the following settings from the Main menu The BIOS settings differ depending upon the hardware model generation e BIOS settings for HP ProLiant DL380 G4 nodes are listed in Table 4 11 e BIOS settings for HP ProLiant DL380 G5 nodes are listed in Table 4 12 Table 4 11 BIOS Settings for HP ProLiant DL380 G4 Nodes Menu Name Option Name Set to This Value Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk Advanced Options Processor Hyper_threading Disable System Options Embedded Serial Port COM2 Virtual Serial Port COM1 Press the Esc key to return to the main menu BIOS Serial Console amp BIOS Serial Console Port COM1 EMS BIOS Serial Console Baud Rate 115200 EMS Console Disable BIOS Interface Mode Command Line Press the Esc key t
94. blic Network Ethernet Mezzanine FULL HEIGHT BLADE E ADMINISTRATOR HALF HEIGHT BLADE KEY Administration Network Console Network Cluster Interconnect Network External Network B 2 InfiniBand Interconnect With Full Height Server Blades In the configuration shown in Figure B 2 connections to the external network were required only on the non blade server nodes and the full height server blades On those server blades NIC3 is used for the connection to the external network A VLAN is used to separate the external network traffic from the administration network traffic on the switch in bay 1 to save the expense of an additional Ethernet interconnect module in bay 2 On the non blade server nodes the built in NICs were used for the external network connection Available ports vary based on hardware model See Chapter 3 page 45 for more information about port assignments B 1 Gigabit Ethernet Interconnect With Half Height Server Blades 143 Figure B 2 InfiniBand Interconnect With Full Height Server Blades Non Blade Server PCI SLOT Non Blade Server PCI SLOT PCI SLOT NIC NIC PCI SLOT E InfiniBand PCI Cards erconnec InfiniBand erconnec Mezzanine Cards erconnec w Q lt l a e I 9 wi T gt m erconnec
95. carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Table 4 6 iLO Settings for HP ProLiant DL360 G4 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Settings CLI Serial CLI Speed 115200 Press the F10 key to save the setting bits seconds Perform the following procedure from the RBSU for each node in the hardware configuration 1 Make the following settings from the Main menu BIOS settings for HP ProLiant DL360 G4 nodes are listed in Table 4 7 Table 4 7 BIOS Settings for HP ProLiant DL360 G4 Nodes Menu Name Option Name Set to This Value Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk Advanced Options Processor Hyper_threading Disable S
96. cccceeesccseescccsseucecesscuuecessuueeesssuuueessuuensess 31 1 8 Interconnect Network cccccccccccccccccssecccsceseccsseuceccssuucccssuusecessuusecessuuuseesseuucessuuecessuuuesessuuusessuuensess 31 1 9 Large Scale Sy SECIS sicker iepenen esasi anse bios inte O REEERE AEE PERRAS tee cipmaieeasbatbeswneaelanintiee 32 2 Cabling arar Bloda osuen 33 2 1 Blade Enclosure OVervVieW c ccccccccccsssssccccessccceusssccceuucsceseusseseseeusscesseusecessueeesessuseessssesessesesesseeeess 33 D2 Network Overview esis ive cionccivoueticiivn sea cesweudesdesuyonsncwbesendsuuevisse cdduees Sede vensadwacseddcuuusevndwevsesdedevessccdouserdcens 33 2 3 Cabling forthe Administration NetwOrthorigiaidasit antec dGies at ilartee iain 37 2 4 Cabling for the Console Netw Ot csieiicsedincnmvrimeiananmmivinnasctiaccnscys mmncsteanramivnmerautanteieniemneananien 38 2 5 Cabling for the Interconnect NGGWOt aici octets niiit ioiii eisein anni n ea pitisi 39 2 5 1 Configuring a Gigabit Ethernet Interconnect ssssssessssssesssssesestsstsressnstessssestesesseseensenees 39 2 5 2 Configuring an InfiniBand IntercOnne ch siessiasutesninsdeeseiccseenannvensiisles shesvenatesisiarsienniniantuiasnieenes 40 2 5 3 Configuring the Interconnect Network Over the Administration Network cccsee 41 2 6 Cabling for the External N tWorK citi iaincctsiiioreieiianlenmitnrd ainsi ianbel mtn eusieelnmaniaioiesras 41 2 6 1 Configuring the External Network Option siicicomieninacamiu
97. ce and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL580 G5 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP ProLiant DL580 G5 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 5 8 Preparing HP xw8200 and xw8400 Workstations 80 You can integrate HP xw8200 and xw8400 workstations into an HP XC system as a head node service node or compute node Follow the procedures in this section to prepare each workstation before installing and configuring the HP XC System Software Figure 4 9 shows a rear view of an HP xw8200 and xw8400 workstation and the appropriate port connections for an HP XC system Preparing Individual Nodes Figure 4 9 HP xw8200 and xw8400 Workstation Rear View OG 0929 2O9292E5 2525 o SBRREERBSORRREESEORS o e R OROORO OROROORO losos osos asas as Seas SeS 202000000000 900090002000000000000909 OBOROROROROROKOKOKOLOROS Cetera tere OORO O BORSREREREROROROKORORSSO QROROKOKOROREREROROSOSIG GRORORORORORORSROROROROR QROBORORSRORSRORORSROBE DROBDRORORERORORORSROR
98. ces do not configure an IP address for the slave MP You must plan the IP addresses ahead of time The following example illustrates how to plan IP addresses The information required for a 16 node cluster rx8620 server is as follows e Gateway address is 172 21 0 16 default based on 16 nodes e Subnet mask address is 255 0 0 0 e IP addresses are listed in Table 4 46 Table 4 46 Setting Static IP Addresses for MP Power Management Devices Node IP Address 26xx ProCurve Port First node after the head node is n15 17 2 24 0 15 Second node after the head node is n14 172 21 0 14 Third node after the head node is n13 172 21 0 13 First rx8620 master MP 172 21 0 x n First rx8620 slave MP N A n 1 Second rx8620 master MP 172 21 0 x 2 n 2 Second rx8620 slave MP N A n 3 4 9 2 Preparing HP Integrity rx1620 and rx2600 Nodes Figure 4 26 shows a rear view of the HP Integrity rx1620 server and the appropriate port assignments for an HP XC system 122 Preparing Individual Nodes Figure 4 26 HP Integrity rx1620 Server Rear View 00008 INSOLE REMOTE UPS oe eee C1 PCI X 133 5 ii oo A PCI X 133 asunpe Lel Io sa cary The callouts in the figure enumerate the following 1 The port labeled LAN 10 100 is the MP connection to the ProCurve Console Switch 2 The port labeled LAN Gb Aconnects to the Administration Switch branch or root 3 The por
99. configured this port labeled NIC2 is used for the interconnect connection Otherwise it is used for an external connection Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL585 G5 node in the hardware configuration 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the following iLO settings for each DL585 G5 node as shown in Table 4 38 Table 4 38 iLO Settings for HP ProLiant DL585 G5 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Settings CLI Serial CLI 115200 Press the F10 key to save the setting Speed bits seconds 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test
100. cumentation that came with the HP Integrity server 126 Preparing Individual Nodes 13 14 15 Perform this step on all nodes except the head node From the Boot Configuration menu select the Edit OS Boot Order option Do the following a Use the navigation instructions on the screen to move the Netboot entry you just defined to the top of the boot order b If prompted press the Enter key to select the position c Enter x to return to the Boot Configuration menu Perform this step on all nodes including the head node Select the Console Configuration option and do the following a Enable the Serial Acpi HWP0002 PNP0A03 700 Pci 1 1 Vt100 9600 option b Press the Esc key or enter x as many times as necessary to return to the Boot Configuration menu c When prompted save the entry to NVRAM Turn off power to the node a Press Ctrl B to exit the console mode b Enter CM to display the Command Menu c Enter PC and enter off to turn off power to the node 4 9 4 Preparing HP Integrity rx2660 Nodes Figure 4 29 shows a rear view of the HP Integrity rx2660 server and the appropriate port assignments for an HP XC system Figure 4 29 HP Integrity rx2660 Server Rear View O9 Of O O O 08 O se xe O O O O O Oy O ae GO Ko O O O O a ko xo 0 Or O O O O O O O O of ORO O86 Ea 080 Kasse OSO O80
101. cure network A workload manager for any kind of batch job Standard LSF features comprehensive workload management policies in addition to simple first come first serve scheduling fairshare preemption backfill advance reservation service level agreement and so on Standard LSF is suited for jobs that do not have complex parallel computational needs and is ideal for processing serial single process jobs Standard LSF is not integrated with SLURM HP Scalable Visualization Array A highly affordable scalable ready to run visualization solution that completes the HP Unified Cluster Portfolio s integration of computation data management and visualization in a single integrated cluster environment The HP SVA solution adds high performance HP workstations in building block configurations that combine with industry standard visualization components State of the art industry standard and open source clustering graphics and networking technology are leveraged to reduce costs and enhance flexibility The tight integration of scalable computation data management and visualization enables the following clustered parallel visualization applications with support for very large data sets display of complex high resolution images including volume visualization and real time rendering with computational steering through closely coupled visualization computation and data management See SMP Index A administration network as interco
102. d node e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4 Hard Drive C BIOS Serial Console and BIOS Serial Console Port com1 EMS BIOS Serial Console Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each node HP ProLiant DL785 G5 node in the hardware configuration Configuring Smart Arrays On HP ProLiant DL785 G5 nodes with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation For more information see the documentation that came with the HP ProLiant server 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 115 4 711 Preparing HP xw9300 and xw9400 Workstations HP xw9300 and xw9400 workstations are typically used when the HP Scalable Visual Array SVA software is installed and configured to interoperate on an HP XC system Configuring an xw9300 or xw9400 workstation as the HP XC head node is supported Figure 4 24 shows a rear view of the xw9300 workstation and the appropriate port connections for an HP XC system Figure 4 24 xw9300 Workstation
103. d node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 Network Controller 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk Advanced Processors Hyper Threading Disable Table 4 18 lists the BIOS settings for HP xw8400 workstations 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 81 Table 4 18 BIOS Settings for xw8400 Workstations Menu Name Submenu Name Option Name Set to This Value Storage Storage Options SATAEmulation Separate IDE Controller After you make this setting make sure the Primary SATA Controller and Secondary SATA Controller settings are set to Enabled Boot Order Set the following boot order on all nodes except the head node 1 Optical Drive 2 USB device 3 Broadcom Ethernet controller 4 Hard Drive 5 Intel Ethernet controller On the head node set the boot order so that the Optical Drive is listed before the hard disk 7 Select File gt Save Changes amp Exit to exit the Setup Utility 8 Repeat this procedure for each workstation in the hardware configuration 9 Turn off power to all nodes except the head node 10 Follow the software installation instructions in the HP XC System Software Installation Guide to install the HP XC System Software 4 5 9 Preparing HP xw8600 Workstations 82 You can integrate HP xw8600 workstations into an HP XC system as a h
104. der Set the boot order 1 From the Boot Configuration menu select the Edit 1 From the Boot Configuration menu select the Edit OS Boot Order option OS Boot Order option 2 Use the navigation instructions and the up arrow 2 Use the navigation instructions and the up arrow on the screen to move the DVD boot entry you just on the screen to move the Netboot entry you just defined to the top of the boot order defined to the top of the boot order 3 Press the Enter key to select the position 3 Press the Enter key to select the position 4 Press the x key to return to the Boot Configuration 4 Press the x key to return to the Boot Configuration menu menu For more information about how to work with these menus see the documentation that came with the HP Integrity server blade Perform this step on all nodes including the head node From the Boot Configuration menu select the Console Configuration option and do the following a Select Serial Acpi HWP0002 PNP0A03 0 Pci 1 2 Vt100 9600 as the primary console interface b Press the letter b key repeatedly until the baud rate is set to 115200 c Press the Esc key or press the x key as many times as necessary to return to the Boot Menu d If prompted save the entry to NVRAM e If prompted reset the system Turn off power to the node a Press Ctrl b to exit the console mode b Return to the Command Menu MP gt CM c Turn off power to the node MP CM gt p
105. e off nc 4 10 Preparing the Hardware for CP6000BL Systems 137 16 Use the RB command to reset the BMC 17 Press Ctrl b to exit the console mode and press the x key to exit After preparing all the nodes in all the enclosures return to the HP XC System Software Installation Guide to discover all the nodes and enclosures in the HP XC system 138 Preparing Individual Nodes 5 Troubleshooting This chapter describes known problems with respect to preparing hardware devices for use with the HP XC System Software and their solutions 5 1 itO2 Devices 5 1 1 iLO2 Devices Can Become Unresponsive There is a known problem with the iLO2 console management devices that causes the iLO2 device to become unresponsive to certain tools including the HP XC power daemon and the iLO2 Web interface When this happens the power daemon generates CONNECT_ERROR messages Additional symptoms include the following e Inability to use the iLO2 Web interface e Inability to control the node s boot options through the Onboard Administrator OA on HP server blade enclosures When this problem occurs the iLO2 device is not completely dead only parts of it are hung You can clear up the problem using either of these methods e Use the following procedure to restart and reboot the node 1 Completely remove power from the node by either removing the power cord or in the case of an HP server blade removing the server blade from the enclosure 2 Wait 15
106. e Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each node HP ProLiant DL585 G5 node in the hardware configuration Configuring Smart Arrays On HP ProLiant DL585 G5 nodes with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation For more information see the documentation that came with the HP ProLiant server 112 Preparing Individual Nodes 4 710 Preparing HP Proliant DL785 G5 Nodes On HP ProLiant DL785 G5 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL785 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 23 shows a rear view of the HP ProLiant DL785 G5 server and the appropriate port assignments for an HP XC system Figure 4 23 HP ProLiant DL785 G5 Server Rear View m o Z on 2 HRR HE PE OGHA go
107. e appropriate port assignments for an HP XC system Figure 4 1 HP ProLiant DL140 G2 and DL140 G3 Server Rear View amp Ol OP OFOnOnO 6858525280525 eo 5202020900000 E TD ceb OO HPTC 0144 The callouts in the figure enumerate the following 1 This port is used for the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 NIC1 2 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 NIC2 3 This port is used for the connection to the Console Switch On the back of the node this port is marked with LO100i Setup Procedure Perform the following procedure for each HP ProLiant DL140 G2 and DL140 G3 node in the hardware configuration Change only the values described in this procedure do not change any other factory set values unless you are instructed to do so Follow all steps in the sequence shown 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 63 6
108. e as global storage within the HP XC system Integrated Lights Out A self contained hardware technology available on CP3000 and CP4000 cluster platform hardware models that enables remote management of any node within a system The next generation of iLO that provides full remote graphics console access and remote virtual media See also iLO image server improved availability Integrated Lights Out interconnect interconnect module interconnect network Internet address IPMI ITRC L Linux Virtual Server load file Load Sharing Facility local storage LSF execution host LSF master host LSF HPC with SLURM A node specifically designated to hold images that will be distributed to one or more client systems In a standard HP XC installation the head node acts as the image server and golden client A service availability infrastructure that is built into the HP XC system software to enable an availability tool to fail over a subset of eligible services to nodes that have been designated as a second server of the service See also availability set availability tool See iLO A hardware component that provides high speed connectivity between the nodes in the HP XC system It is used for message passing and remote memory access capabilities for parallel applications A module in an HP BladeSystem server The interconnect module provides the Physical I O ports for the server blades and can be eit
109. ead node service node or compute node Follow the procedures in this section to prepare each workstation before installing and configuring the HP XC System Software Figure 4 10 shows a rear view of an HP xw8600 workstation and the appropriate port connections for an HP XC system Figure 4 10 HP xw8600 Workstation Rear View r 0 u The callouts in the figure enumerate the following 1 This port is used for the connection to the administration network 2 This port is used for connecting the workstation to an external network Setup Procedure Use the Setup Utility to configure the appropriate settings for an HP XC system Preparing Individual Nodes Perform the following procedure for each workstation in the hardware configuration Change only the values that are described in this procedure do not change any other factory set values unless you are instructed to do so AT PONE 10 Establish a connection to the console by connecting a monitor and keyboard to the node Turn on power to the workstation When the node is powering on press the F10 key to access the Setup Utility When prompted press any key to continue Select English as the language Make the following BIOS settings for each workstation in the hardware configuration as shown in Table 4 19 Table 4 19 BIOS Settings for xw8600 Workstations Menu Name Submenu Name Option Name Set to This Value Storage Storage Options SATAEmulation Separate IDE
110. eeceaueaseeececeeeceuueaseeececesecsauaaeeeeeeeees 58 d Preparing Individual INGE SS ssnrsanmiesssaoneda n 59 4 1 Firmware Requirements and Dependencies ssssessssrssisessssissiesrererssitnrsreessresresernsieseernessressesees 59 4 2 Ethernet Port Connections on the Head Node cccccccccsssesescececececeaueseeeecccceeseaueassesecceeessaueusgeses 61 4 3 General Hardware Preparations for All Cluster Plathormisecsacsesisssievcsssssasnssestodassanetdnins tevcinsascin 61 4 4 Setting the Onboard Administrator Password sesssssssssessssissrssrsserssirnssreersresresernsreneeseessressesees 62 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T SystemS sssssessessssisesesssersssees 63 4 5 1 Preparing HP ProLiant DL140 G2 and G3 Nodes ss ssssessseeseesessserssisrssserssrsnrsseesseenesreesse 63 4 5 2 Preparing HP ProLiant DL160 G5 NOES s ccnsentenrad oe rane 66 4 5 3 Preparing HP ProLiant DL360 G4 NOS iriciss2ivscectinnncsonnisictdn sone vetoaivbelendanectonaansieiveenevebvadnteses 68 4 5 4 Preparing HP ProLiant DL360 G5 Nodes sssssessessseesessessseessrsrssserssesnesserssrsnesrernsreneeneesse 70 4 5 5 Preparing HP ProLiant DL380 G4 and G5 NOES ies ssscnvsassessscvessnvscuviesavsaupseonsaonasavadvsnenwacaienss 72 4 5 6 Preparing HP ProLiant DL580 G4 NoOdeS cssisisiisscssiseeisessrtssvsoreriersienessrirseinsrisaveesi neetud ane 75 4 5 7 Preparing HP ProLiant DL580 G5 INGAGS cccie siscdactioncunsssasbansiiew
111. egrated Lights Out iLO Setup Utility ROM Based Setup Utility RBSU HP ProLiant DL365 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 15 shows a rear view of the HP ProLiant DL365 server and the appropriate port assignments for an HP XC system Preparing Individual Nodes Figure 4 15 HP ProLiant DL365 Server Rear View HCH The callouts in the figure enumerate the following 1 This port is the Ethernet connection to the Console Switch On the back of the node this port is marked with the acronym iLO This port is the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 If a Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL365 node in the hardware configuration 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node Turn on power to the node Watch the screen carefully during the power on self
112. er blades Use the following table and the numbering scheme in Figure 1 8 to determine the placement of the server blades in the enclosure based on the number of fans Insert Half Height Server Blades in Insert Full Height Server Blades in These Number of Fans These Bays Bays 4 fan configuration 1 2 5 6 1 2 6 fan configuration any any Figure 1 9 is an illustration showing the location of the interconnect bays fan bays onboard administrator bays the enclosure onboard administrator link module and power supplies at the rear of the HP BladeSystem c3000 enclosure This figure includes an inset showing the onboard administrator iLO port and the enclosure Uplink and downlink ports Hardware and Network Overview Figure 1 9 HP BladeSystem 3000 Enclosure Bay Locations Rear View Interconnect bay 1 Fans Interconnect bay 2 Enclosure Onboard Administrator Link Module Power Supplies Interconnect bay 3 Interconnect bay 4 Og oe A of oi Enclosure Downlink port Enclosure Uplink port Onboard Administrator Integrated Lights Out port Specific HP XC Setup Guidelines The following enclosure setup guidelines are specific to HP XC e Onevery enclosure an Ethernet interconnect module either a switch or pass through module is installed in interconnect bay 1 see callout 1 in Figure 1 9 for the administration network e Hardware configurations that use Gigabit Ethernet as the interconnect can share with the adminis
113. er name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Table 4 35 lists the iLO settings for ProLiant DL585 G2 nodes Table 4 35 iLO Settings for HP ProLiant DL585 G2 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable on Settings CLI Serial CLI 115200 Press the F10 key to save the setting Speed bits seconds 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU 4 7 Preparing the Hardware for CP40
114. erconnect based on interconnect type are discussed in the following sections e QsNet Interconnect Connections page 57 e Gigabit Ethernet Interconnect Connections page 57 e Administration Network Interconnect Connections page 57 e Myrinet Interconnect Connections page 58 e InfiniBand Interconnect Connections page 58 Making Node and Switch Connections The method for wiring the administration network and interconnect networks allows expansion of the system within the system s initial interconnect fabric without recabling of any existing nodes If additional switch chassis or ports are added to the system as part of the expansion some recabling may be necessary 3 4 1 QsNet Interconnect Connections A For the QsNet interconnect developed by Quadrics it is important that nodes are connected to the Quadrics switch ports in a specific order The order is affected by the order of the administration network and console network Because the Quadrics port numbers start numbering at 0 the highest port number on the Quadrics switch is port max node minus 1 where max node is the maximum number of nodes possible in the system This is the port on the Quadrics switch to which the head node must be connected The head node in an HP XC system is always the node connected to the highest port number of any node on the Root Administration Switch and the Root Console Switch NOTE The head node port is not
115. ernal connection but this varies by hardware model If the hardware configuration contains non blade server nodes see Chapter 4 page 59 for information about which port to use for the external network 2 6 Cabling for the External Network 43 Figure 2 10 External Network Connections Half and Full Height Server Blades and NIC1 in Use Non Blade Server PCI SLOT NIC NIC PCI SLOT Non Blade Server PCI SLOT NIC NIC PCI SLOT MGT MGT C Class Blade Enclosure erconnec nterconnect bay 2 nterconnect bay 3 nterconnect bay 4 Gigabit Ethernet Interconnect Switch nterconnect bay 7 External Public Network ONBOARD ADMINISTRATOR Admin ProCurve 2800 Series Switch Console ProCurve 2600 Series Switch FULL HEIGHT BLADE HALF HEIGHT BLADE 2 6 4 Creating VLANs Use the following procedure on GbE2c Nortel switches if you need to configure a VLAN to separate the external network from other network traffic 44 1 gi WN See the illustrations of interconnect bay port mapping connections in the HP BladeSystem Onboard Administrator User Guide to determine which ports on the switch to connect to each of the two virtual networks Remember to include at least one of the externally accessible ports in each VLAN Connect a serial dev
116. erver Blades Figure 2 5 Console Network Connections Non Blade Server PCI SLOT NIC PCI SLOT Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT MGT NIC C Class Blade Enclosure Admin ProCurve 2800 Series Switch Interconnect bay 2 Interconnect bay 3 Console ProCurve 2600 Series Switch Interconnect bay 4 Interconnect bay 5 Gigabit Ethernet Interconnect Switch Interconnect bay 6 Interconnect bay 7 Interconnect bay 8 External Public Network l ONBOARD ADMINISTRATOR 2 5 Cabling for the Interconnect Network The interconnect network is a private network within an HP XC system Typically every node in an HP XC system is connected to the interconnect The interconnect network provides a high speed communications path used primarily for user file service and for communications within applications that are distributed among nodes in the cluster Gigabit Ethernet and InfiniBand are supported as the interconnect types for HP XC hardware configurations with server blades and enclosures The procedure to configure the interconnect network depends upon the type of interconnect in use e Configuring a Gigabit Ethernet Interconnect page 39 e Configuring an InfiniBand Interconnect page 40 e Configuring the Interconnect Network Over the Ad
117. es Web address for Modules which provide for easy dynamic modification of a user s environment through modulefiles which typically instruct the module command to alter or set shell environment variables http dev mysql com Home page for MySQL AB developer of the MySQL database This web address contains a link to the MySQL documentation particularly the MySQL Reference Manual Related Software Products and Additional Publications This section provides pointers to web addresses for related software products and provides references to useful third party publications The location of each web address or link to a particular topic is subject to change without notice by the site provider Linux Web Addresses http www redhat com Home page for Red Hat distributors of Red Hat Enterprise Linux Advanced Server a Linux distribution with which the HP XC operating environment is compatible http www linux org docs index html This web address for the Linux Documentation Project LDP contains guides that describe aspects of working with Linux from creating your own Linux system from scratch to bash script writing This site also includes links to Linux HowTo documents frequently asked questions FAQs and manpages Related Information 15 16 http www linuxheadquarters com Web address providing documents and tutorials for the Linux user Documents contain instructions for installing and using applications for Linu
118. ets Cabinets are used as a packaging medium The HP XC system hardware is contained in two types of cabinets e Application cabinets The application cabinets contain the compute nodes and are optimized to meet power heat and density requirements All nodes in an application cabinet are connected to the local branch switch e Utility cabinets The utility cabinet is intended to fill a more flexible need In all configurations at a minimum the utility cabinet contains the head node Nodes with external storage and nodes that are providing services to the cluster called service nodes or utility nodes are also contained in the utility cabinet All nodes in the utility cabinet are connected to the root switches administration and console Figure 3 1 illustrates the relationship between application cabinets utility cabinets and the Root Administration Switch For more information see Root Administration Switch page 50 Figure 3 1 Application and Utility Cabinets Root Administration Switch Utility Cabinet Application Application Application Application Cabinet Cabinet Cabinet Cabinet 3 2 Trunking and Switch Choices The HP XC System Software supports the use of port trunking that is the use of multiple network ports in parallel to increase link speed faster than any one single port on the ProCurve switches 3 1 Cabinets 45 t
119. f the large scale system are divided as equally as possible between the individual HP XC systems which are known as regions The head node for a large scale HP XC system is always the head node of region 1 Hardware and Network Overview 2 Cabling Server Blades The following topics are addressed in this chapter e Blade Enclosure Overview page 33 e Network Overview page 33 e Cabling for the Administration Network page 37 e Cabling for the Console Network page 38 e Cabling for the Interconnect Network page 39 e Cabling for the External Network page 41 2 1 Blade Enclosure Overview An HP XC blade cluster is made up of one or more Blade Enclosures connected together as a cluster Each blade enclosure must contain the following e 1 to 16 blade servers e 1 Ethernet Interconnect blade in bay 1 for the Administration Network e 1 Onboard Administrator OA for managing the enclosure A NOTE Enclosures might also have a redundant Onboard Administrator e The requisite number of fans and power supplies to fill the needs of all the hardware In addition each enclosure needs an additional blade interconnect module for the cluster interconnect On a Gigabit Ethernet GigE cluster this could be either another Ethernet Switch or an Ethernet pass thru module in bay 2 On an InfiniBand IB cluster this would be one of the double wide IB Blade switches in bays 5 and 6 In certain circum
120. for an HP XC system 104 Preparing Individual Nodes Figure 4 19 HP ProLiant DL385 G5 Server Rear View mUeYy o _ of fy gt h ODA Sooo cep o 7 o 4 AD o o zi 2 24 o O OF Se A agag pean 4 hal hie Ps Seer e Repeats l mm a a A nt wee IO lt gt Jor fo a 5 o The callouts in the figure enumerate the following 1 This port is the Ethernet connection to the Console Switch On the back of the node this port is marked with the acronym iLO This port is the connection to the Administration Switch branch or root On the back of the node this port is marked with the number 1 If a Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL385 G5 node in the hardware configuration 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility Make the following iLO settings for each DL385 G5 node as shown in Table 4 33 Table 4 33 iLO Settings for HP ProLiant DL385
121. g the hardware discovery phase of the system configuration process you are prompted to supply the password for the ProCurve switch administrator and the password on every switch must match 3 3 3 Switch Port Connections Most HP XC systems have at least one Root Administration Switch and one Root Console Switch The number of Branch Administration Switches and Branch Console Switches depends upon the total number of nodes in the hardware configuration The administration network using the root and branch switches must be parallel to the console network root and branch switches In other words if a particular node uses port N on the Root Administration Switch its management console port must be connected to port N on the Root Console Switch If a particular node uses port N on the Branch Administration Switch its management console port must be connected to port N on the corresponding Branch Console Switch A graphical representation of the logical layout of the switches and nodes is shown in Figure 3 2 3 3 Switches 47 48 Figure 3 2 Node and Switch Connections on a Typical System Specialized Role Nodes Administration Console _Switches ee OEE Switches 2 Root Root Console 1 Administration Compute Nodes Figure 3 3 shows a graphical representation of the logical layout of the switches and nodes in a large scale system with a Super Root Switch The head node connects to Port 42 on the Root Admin
122. he Integrated Lights Out 2 utility in a new window You might have to turn off popup blocking for this window to open In the new window click the Administration tab In the left frame click the User Administration link e Click the New button and create a new iLO2 user name and password which must match the user name and password you set on the Onboard Administrator Do not use any special characters as part of the password an You use this user name and password whenever you need to access the console port with the telnet cp nodename command 4 8 Preparing the Hardware for CP4000BL Systems 119 f The Onboard Administrator automatically creates user accounts for itself prefixed with the letters OA to provide single sign on capabilities Do not remove these accounts 5 Enable telnet access a Inthe left frame click Access b Click the control to enable Telnet Access c Click the Apply button to save the settings 6 Click the Virtual Devices tab and make the following settings a For every node except the head node select No to Automatically Power On Server because you do not want to automatically turn on power to the node b Click the Submit button c Inthe left frame click on the Power Regulator link d Select Enable OS Control Mode e Click the Apply button to save the settings 7 Configure disks into the smart array from the remote graphics console All server blades have smart array cards you must add the d
123. he administration network ProCurve 2650 Switch page 52 ProCurve 2610 48 Switch page 52 ProCurve 2626 Switch page 53 ProCurve 2610 24 Switch page 54 3 3 Switches 51 3 3 6 1 ProCurve 2650 Switch You can use a ProCurve 2650 switch as a Root Console Switch for the console branch of the administration network The console branch functions at a lower speed 10 100 Mbps than the rest of the administration network The ProCurve 2650 switch is shown in Figure 3 7 In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Figure 3 7 ProCurve 2650 Root Console Switch Uplinks from Branches Connections to Node Console Ports Start at Port 1 Ascending Start at Port 41 Descending HEEB BEER REE REE Bee ee ee SEE Seseeee qe ge T T CELT T 8 88 888 a 880 T FETTET TE TUT FEFFE sen 10 100Base TX RJ 45 Ports Gigabit Ethernet Ports The callouts in the figure enumerate the following 1 Port 42 must be reserved for an optional connection to the console port on the head node 2 Port 49 is reserved 3 Port 50 is the Gigabit Ethernet link to the Root Administration Switch Allocate the ports on this switch for consistency with the administration switches
124. he cabinet power turned on 1 Press Ctrl B to exit out of console mode 2 Enter CM to display the Command Menu 3 Enter PE to turn off power to the partition 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 135 4 10 Preparing the Hardware for CP6000BL Systems Use the management processor MP to perform the following tasks on each server blade in the hardware configuration after the head node is installed and the switches are discovered e Clear all event logs e Enable IPMI over LAN e Create an MP login ID and password that matches all other devices e Adda boot entry for the string DVD boot on the head node and add a boot entry for the string Net boot on all other nodes e Move the DVD boot and Netboot boot entries to top of the boot order e Set the primary console e Set the console baud rate to 115200 Procedure Perform the following procedure on each HP Integrity server blade All nodes must be seated in an enclosure plugged in and power turned off 1 Connect the local IO cable also called the SUV cable to the server blade The local cable is shipped with the enclosure and connects to the server blade at one end and is divided into VGA USB and Serial ports at the other end 2 Connect a serial terminal or laptop serial port to the serial port of the local IO cable w Use a terminal emulator such as HyperTerminal to open a terminal window 4 Press the Enter key to access the MP If there is no res
125. he callouts in the figure enumerate the following 1 Port 22 must be reserved for an optional connection to the console port on the head node 2 Port 25 is reserved 3 Port 26 is the Gigabit Ethernet link to the Root Administration Switch Allocate the ports on this switch for consistency with the administration switches as follows e Ports 1 10 13 21 3 3 Switches 53 Starting with port 1 the ports are used for links from Branch Console Switches Trunking is not used Starting with port 21 and in descending order ports are assigned for use by individual nodes in the utility cabinet Nodes in the utility cabinet are connected directly to the Root Administration Switch A NOTE There must be at least one idle port in this set to indicate the dividing line between branch links and root node administration ports e Ports 11 12 23 and 24 are unused 3 3 6 4 ProCurve 2610 24 Switch You can use a ProCurve 2610 24 switch as a Root Console Switch for the console branch of the administration network The ProCurve 2610 switch is shown in Figure 3 10 In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Figure 3 10 ProCurve 2610 24 Root Console Switch Uplinks from Branches Connections to Node Console Ports Start at port 1 Ascending Start at port 21 Descending Oneone seme Oo DE ranea oa
126. he following BIOS settings from the Main window The BIOS settings for HP ProLiant DL160 G5 nodes are listed in Table 4 5 Table 4 5 BIOS Settings for HP ProLiant DL160 G5 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Features Numlock Disabled Advanced 8042 Emulation Disabled Support Remote Access Remote Access Enabled Configuration EMS Support Enabled SPCR Serial Port Mode 11520 8 n 1 Redirection after Always BIOS POST Terminal Type VT100 IPMI Configuration LAN Configuration Share NIC Mode Disabled DHCP IP Source Enabled Boot Boot Device Priority 1st Boot Device If this node is the head node set this value to Hard Drive For all other nodes set this value to Embedded NIC1 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 67 Table 4 5 BIOS Settings for HP ProLiant DL160 G5 Nodes continued Menu Name Submenu Name Option Name Set to This Value 2nd Boot Device If this node is the head node set this value to Embedded NIC1 Otherwise set this value to Hard Drive Embedded NIC1 Enabled PXE Embedded NIC2 Disabled PXE 4 From the Main window select Exit gt Save Changes and Exit to exit the utility 5 Repeat this procedure for each HP ProLiant DL160 G5 node in the HP XC system 4 5 3 Preparing HP ProLiant DL360 G4 Nodes 68 Use the following tools to configure the appropriate settings for HP ProLiant DL360 G4 including DL360 G4p servers e Integr
127. he operation of third party software is included in this document if the functionality of the third party component differs from standard behavior when used in the XC environment In this case HP XC documentation supersedes information supplied by the third party vendor For links to related third party Web sites see Supplementary Software Products page 14 Standard Linux administrative tasks or the functions provided by standard Linux tools and commands are documented in commercially available Linux reference manuals and on various Web sites For more information about obtaining documentation for standard Linux administrative tasks and associated topics see the list of Web sites and additional publications provided in Related Software Products and Additional Publications page 15 Intended Audience The information in this document is written for technicians or administrators who have the task of preparing the hardware on which the HP XC System Software will be installed Before beginning you must meet the following requirements e You are familiar with accessing BIOS and consoles with either Ethernet or serial port connections and terminal emulators e You have access to and have read the HP Cluster Platform documentation e You have access to and have read the HP server blade documentation if the hardware configuration contains HP server blade models e You have previous experience with a Linux operating system New
128. her a switch with connections to each of the server blades and some number of external ports it can be or a pass through module with individual external ports for each of the server blades See also server blade The private network within the HP XC system that is used primarily for user file access and for communications within applications A unique 32 bit number that identifies a host s connection to an Internet network An Internet address is commonly represented as a network number and a host number and takes a form similar to the following 192 0 2 0 Intelligent Platform Management Interface A self contained hardware technology available on HP ProLiant DL145 servers that enables remote management of any node within a system HP IT Resource Center The HP corporate Web page where software patches are made available The web address is http www itrc hp com To download patches from this Web page you must register as an Americas Asia Pacific or European customer See LVS A file containing the names of multiple executables that are to be launched simultaneously by a single command See LSF HPC with SLURM Storage that is available or accessible from one node in the HP XC system The node on which LSF runs A user s job is submitted to the LSF execution host Jobs are launched from the LSF execution host and are executed on one or more compute nodes The overall LSF coordinator for the system The master load information man
129. hon Swaine vncteesttntine chara veuta suing diene bettie cine shonensiesngens aaanbein 40 InfiniBand Interconnect CONNECTIONS ccccccccecessesseescecececeaaeseseececcceseaueuensececceesessaueasseseeeeseaeaaea 40 External Network Connections Full Height Server Blades and NIC1 and NIC2 in Use 42 External Network Connections Half Height Server Blades and NIC1 and NIC2 in Use 43 External Network Connections Half and Full Height Server Blades and NIC1 in Use 44 Application and Utility GAMES is cse Gancssstcast iene sinnir ironiseen desns sun luenes addade oiaetonsnadunnssoneoesinaiaens 45 Node and Switch Connections on a Typical System piscsientsbectienincias einer anaienrniecns 48 Switch Connections for a Large Scale Sy Sten escsijctarsanycichdsimsseadanatarssioriwndanmersinmishaneiedtnienions 48 ProCurve 2848 S per ROE SWilC li ciccserrstacag dete saacidenlenmtenelacbieveimnserninla duont iaa Ei E Siasia 49 ProCurve 2848 Root Administration SWitch ccccccccccccccceesssesesceccceceeeeaeeeseecceesecesueaeseeceeeseeeaaea 50 ProCurve 2824 Root Administration SWitch ccccccccccccceccesssesescececececesueseccecceeeesesueeesecceeeeeneaaea 51 ProCurve 2650 Root Console Switch cccccccccccccssssseeeececececeaeeueecceccceceaeeueaecececeseseaueaeseeceeeeeaeauea 52 ProCurve 2610 48 Root Console SWitch cccccccccccscsssesceccceceeeeseseececcceseaueaesececeseseaeaueasseeeeeeeeaeaaea 53 ProCurve 2
130. ic IP addresses For more information see Setting Static IP Addresses on Integrity Servers page 122 9 Enter PC power cycle and then enter on to turn on power to the node 10 Press Ctrl B to return to the MP Main Menu 11 Enter CO to connect to the console 12 Perform this step on all nodes except the head node From the EFI Boot Manager menu which is displayed when the node is powering on select the Boot Configuration menu Do the following from the Boot Configuration menu a Select Add Boot Entry b Select the network boot device which is a Gigabit Ethernet GigE port e On HP Integrity rx1620 servers select Load File Core LAN Gb A e On HP Integrity rx2600 servers select Acpi HWP0002 100 Pci 2 0 Mac XXXXXXXXXXXX c Enter the string Netboot as the boot option description This entry is required and must be set to the string Netboot with a capital letter N d Press the Enter key when prompted to enter a load option e If prompted save the entry to NVRAM f Enter x to return to the previous menu For more information about how to work with these menus see the documentation that came with the HP Integrity server 124 Preparing Individual Nodes 13 14 15 Perform this step on all nodes except the head node From the Boot Configuration menu select the Edit OS Boot Order option Do the following from the Edit OS Boot Order option a Use the navigation instructions shown on the screen to
131. ice to the serial console port of the GbE2c switch Press the Enter key When you are prompted for a password enter admin which is the default password Enter the following commands to access the VLAN configuration a cfg b 12 the letter 1 as in layer not the number one c vlan 2 be sure to enter a space between vlan and the vlan number Specify a name for the VLAN choose any name you want name your name Enable the VLAN ena Add each port to the VLAN one at a time If you see a message that the port is in another VLAN answer yes to move it This example adds ports 1 3 and 21 to the VLAN add 1 add 3 add 21 If you need more information about creating VLANs see the GbE2c documentation When you have completed adding ports enter apply to activate your changes and enter save to save them Cabling Server Blades 3 Making Node and Switch Connections This chapter provides information about the connections between nodes and switches that are required for an HP XC system The following topics are addressed e Cabinets page 45 e Trunking and Switch Choices page 45 e Switches page 46 e Interconnect Connections page 56 we IMPORTANT The specific node and switch port connections documented in this chapter do not apply to hardware configurations containing HP server blades and enclosures For information on cabling server blades see Chapter 2 page 33 3 1 Cabin
132. inal emulator such as HyperTerminal to open a terminal window d Press the Enter key to access the MP If there is no response press the MP reset pin on the back of the MP and try again ie 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 123 e Log into the MP using the default user name and password shown on the screen The MP Main Menu appears MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 4 Enter SL to show event logs Then enter C to clear all log files and Y to confirm Enter CM to display the Command Menu 6 Perform the following steps to ensure that the IPMI over LAN option is set This setting is required for Nagios monitoring a Enter LC b Verify the IPMI over LAN option is enabled c Enable this option if it is disabled d Return to the Command Menu g 7 Enter UC and use the menu options to remove the default MP user name and password and create your own unique user name and password HP recommends setting your own user name and password for security purposes The user name must have a minimum of 6 characters and the password must have a minimum of 8 characters You must set the same user name and password on every node The user name and password are required to access the power management device and console 8 If required set stat
133. inistration ports for the individual nodes up to 19 nodes e Ports 11 12 23 and 24 are unused 3 3 8 Branch Console Switches The Branch Console Switch of an HP XC system is a ProCurve 2650 or ProCurve 2610 48 switch The connections to the ports must parallel the connections of the corresponding Branch Administration Switch If a particular node uses port Non a Branch Administration Switch its management console port must be connected to port Non the corresponding Branch Console Switch Figure 3 13 shows the ProCurve 2650 switch and Figure 3 13 shows the ProCurve 2610 48 switch In each figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for a specific purpose described after the figures 3 3 Switches 55 Figure 3 13 ProCurve 2650 Branch Console Switch Connections to Node Console Ports HE EE ERE ERE ER TE Eee PRE RBRERE ee ee eee Eeseeaae HEA aes E BATRA AA AAA AAAA A AAAA eves Ce eee vere ve vere Gigabi 10 100Base TX RJ 45 Ports Ethernet Ports BOO Bo eon om ora forse at sot Figure 3 14 ProCurve 2610 48 Branch Console Switch Connections to Node Console Ports 10 100Base TX RJ 45 Ports Ethernet Ports The callout in these figures enumerates the following 1 Port 50 is the link to the Root Console Switch Alloca
134. ion ProCurve 2626 switch Firmware version Interconnect 60 Preparing Individual Nodes Table 4 1 Firmware Dependencies continued Hardware Component Firmware Dependency Myrinet Firmware version Myrinet interface card Interface card version QsNet Firmware version InfiniBand Firmware version 4 2 Ethernet Port Connections on the Head Node Table 4 2 lists the Ethernet port connections on the head node based on the type of interconnect in use Use this information to determine the appropriate connections for the external network connection on the head node we IMPORTANT The Ethernet port connections listed in Table 4 2 do not apply to hardware E configurations with HP server c Class blades and enclosures Table 4 2 Ethernet Ports on the Head Node Gigabit Ethernet Interconnect All Other Interconnect Types Physical onboard Port 1 is always the connection to Physical onboard Port 1 is always the connection to the administration network the administration network Physical onboard Port 2 is the connection to the e Physical onboard Port 2 is available for an external interconnect connection if needed except if the port is 10 100 then Add on NIC card 1 is available as an external it is unused connection e Add on NIC card 1 is available for an external connection if Port 2 is 10 100 4 3 General Hardware Preparations for All Cluster Platforms Make the following hardware preparations on all cluster platfo
135. ions Embedded NIC Port PXE On all nodes except the head node set this value Support to Enable NIC1 PXE On the head node only set this value to Embedded NIC PXE Disabled Embedded Serial Port Disabled Virtual Serial Port COM1 IRQ4 I0 3F8h 3FFh Embedded NIC Port 1 PXE Enabled all nodes except the head node Support Embedded NIC Port 1 PXE Disabled head node only Support Power Regulator for Disabled ProLiant Standard Boot Order IPL Set the following boot order on all nodes except the head node 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk IPL1 CD ROM IPL2 Floppy Drive A IPL3 PCI Embedded HP NC7782 Gigabit Server Adapter Port 1 IPL4 Hard Drive C BIOS Serial Console and BIOS Serial Console Port COM1 IRQ4 10 3F8h 3FFh EMS BIOS Serial Console Baud 115200 Rate EMS Console Disabled BIOS Interface Mode Command Line 1 Asmall blue dialog box near the bottom left side of the screen indicates the current setting You can make only one setting per node 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL365 node in the hardware configuration Configuring Smart Arrays On such hardware models as the HP ProLiant DL365 with smart array cards you must add the disks to the smart array before attempting to image the node T
136. ired in syntax If the contents are a list separated by you must choose one of the items The preceding element can be repeated an arbitrary number of times Separates items in a list of choices A warning calls attention to important information that if not understood or followed will result in personal injury or nonrecoverable system problems A caution calls attention to important information that if not understood or followed will result in data loss data corruption or damage to hardware or software This alert provides essential information to explain a concept or to complete a task A note contains additional information to emphasize or supplement important points of the main text HP XC and Related HP Products Information The HP XC System Software Documentation Set the Master Firmware List and HP XC HowTo documents are available at this HP Technical Documentation Web site 12 http docs hp com en linuxhpc html The HP XC System Software Documentation Set includes the following core documents HP XC System Software Release Notes HP XC Hardware Preparation Guide HP XC System Software Installation Guide HP XC System Software Administration Guide Describes important last minute information about firmware software or hardware that might affect the system This document is not shipped on the HP XC documentation CD It is available only on line Describes hardware preparation tasks specific to HP XC that a
137. isk or disks to the smart array before attempting to image the node To set up the smart array device click the Remote Console tab on the virtual console page of the iLO2 Web Administration Utility and then do one of the following depending on the browser type Internet Explorer If you are using Internet Explorer as your browser do the following a Click the Integrated Remote Console link to open a remote console window which provides access to the graphics console virtual media and power functions b Inthe remote console window click the Power button c Click the Momentary Press button d Wait a few seconds for the power up phase to begin Click the MB1 mouse button in the remote console window to put the pointer focus in this window so that your keyboard strokes are recognized e Proceed to Step 8 Mozilla Firefox If you are using Mozilla Firefox as your browser do the following a Click the Remote Console link to open a virtual console window b Go back to the iLO2 utility Web page and click the Virtual Devices tab c Click the Momentary Press button d Go back to the remote console window Wait a few seconds for the power up phase to begin Click the MB1 mouse button in this window to put the pointer focus in the remote console window so that your keyboard strokes are recognized in this window e Proceed to Step 8 8 Watch the screen carefully during the power on self test phase and press the F8 key when you
138. istration Network Console Branch Without HP Server Blades Specialized Role Nodes Head Node O Eo Root Administration Switch Branch Console Switches Compute Nodes 1 8 Interconnect Network The interconnect network is a private network within the HP XC system Typically every node in the HP XC system is connected to the interconnect The interconnect network is dedicated to communication between processors and access to data in storage areas It provides a high speed communications path used primarily for user file service and for communications within applications that are distributed among nodes of the cluster Table 1 4 lists the supported interconnect types on each cluster platform The interconnect types are displayed in the context of an interconnect family in which InfiniBand products constitute one family Quadrics QsNet constitutes another interconnect family and so on For more information about the interconnect types on individual hardware models see the cluster platform documentation 1 7 Administration Network Console Branch 31 Table 1 4 Supported Interconnects INTERCONNECT FAMILIES InfiniBand PCI Cluster Express Single InfiniBand Platform and Data Rate and ConnectX Hardware Gigabit InfiniBand Double Data Double Data Myrinet Rev A Model Ethernet PCI X Rate DDR Rate DDR D E and F QsNet CP3000 X X X X X CP3000BL X X X CP4000 x xX x xX xX x CP4000BL X X X CP6000 x x x x x
139. istration Switch in Region 1 Figure 3 3 Switch Connections for a Large Scale System ProCurve 2848 1 2 Ports 3 45 46 48 Super Root witc To Region 2 Ports 3 45 Ports 3 47 1 2 Pors 3 45 46 48 1 2 Pors 3 47 a8 50 Making Node and Switch Connections 3 3 3 1 Switch Connections and HP Workstations HP model xw workstations do not have console ports Only the Root Administration Switch supports mixing nodes without console management ports with nodes that have console management ports that is all other supported server models HP workstations connected to the Root Administration Switch must be connected to the next lower numbered contiguous set of ports immediately below the nodes that have console management ports For example if nodes with console management ports are connected to ports 42 through 36 on the Root Administration Switch the console ports are connected to ports 42 through 36 on the Console Switch Workstations must be connected starting at port 35 and lower to the Root Administration Switch the corresponding ports on the Console Switch are empty 3 3 4 Super Root Switch Figure 3 4 shows the Super Root Switch which is a ProCurve 2848 A Super Root switch configuration supports the use of trunking to expand the bandwidth of the connection between the Root Administration Switch and the Super
140. itch for the administration network of an HP XC system can be either a ProCurve 2848 switch or a ProCurve 2824 switch for small configurations If you are using a ProCurve 2848 switch as the switch at the center of the administration network use Figure 3 5 to make the appropriate port connections In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Gray colored ports are reserved for future use Figure 3 5 ProCurve 2848 Root Administration Switch Connections to Node Administration Ports IL L L L Uplinks from Branches 10 100 1000 Base TX RJ 45 Ports Gigabit Ethernet Begin at Port 1 Ascending Ports The callouts in the figure enumerate the following 1 Port 42 must be used for the administration port of the head node 2 Ports 43 through 46 are used for connecting to the Super Root Switch if you are configuring a large scale system 3 Port 47 can be one of the following e Connection or line monitoring card for the interconnect e Connection to the Interconnect Ethernet Switch IES which connects to the management port of multiple interconnect switches 4 Port 48 is used for the interconnect to the Root Console Switch ProCurve 2650 or ProCurve 2626 The ports on this switch must be allocated as follows
141. l Nodes Table 4 8 iLO Settings for HP ProLiant DL360 G5 Nodes Menu Name User Submenu Name Option Name Add Set to This Value Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Settings CLI bits seconds Serial CLI Speed 115200 Press the F10 key to save the setting Perform the following procedure from the RBSU for each node in the hardware configuration Make the following settings from the Main menu Table 4 9 lists the BIOS settings for HP 1 ProLiant DL360 G5 Nodes Table 4 9 BIOS Settings for HP ProLiant DL360 G5 Nodes Menu Name Option Name System Options Embedded Serial Port Virtual Serial Port Set to This Value COM2 IRQ3 IO 2F8h 2FFh COM1 IRQ4 IO 3F8h 3FFh Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 Floppy Drive A 3 USB DriveKey C 4 PCI Embedded HP NC373i Multifunction Gigabit Adapter
142. l computer to the Management Processor MP a Connect a null modem serial cable between the MP serial port and the PC COM1 port b Use a terminal emulator such as HyperTerminal to open a terminal window c Press the Enter key to access the MP If there is no response press the MP reset pin on the back of the MP and try again d Log in to the MP using the default user name and password shown on the screen The MP Main Menu appears MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 3 Enter SL to clear the error logs CLR 4 Enter CM to display the Command Menu 132 Preparing Individual Nodes A 10 11 NOTE Most of the MP commands of the HP Integrity rx8620 are similar to the HP Integrity rx2600 MP commands but there are some differences The two MPs for the HP Integrity rx8620 operate in a master slave relationship Only the master MP which is on Core IO board 0 is assigned an IP address Core IO board 0 is always the top Core IO board The slave MP is used only if the master MP fails Enter LC to configure the IP address subnet mask and default gateway address for the HP Integrity rx8620 master MP interfaces do not configure an IP address for the slave MP See Setting Static IP Addresses on Integrity Servers page 122 for information on static IP addresses Also
143. llocates IP addresses to computers on a local area network See DHCP Extensible Firmware Interface Defines a model for the interface between operating systems and Itanium based platform firmware The interface consists of data tables that contain platform related information plus boot and run time service calls that are available to the 147 enclosure extensible firmware interface external network node F fairshare FCFS first come first served G global storage golden client golden image golden master H head node host name I O node iLO iLO2 148 Glossary operating system and its loader Together these provide a standard environment for booting an operating system and running preboot applications The hardware and software infrastructure that houses HP BladeSystem servers See EFT A node that is connected to a network external to the HP XC system An LSF job scheduling policy that specifies how resources should be shared by competing users A fairshare policy defines the order in which LSF attempts to place jobs that are ina queue or a host partition First come first served An LSF job scheduling policy that specifies that jobs are dispatched according to their order in a queue which is determined by job priority not by order of submission to the queue See FCFS Storage within the HP XC system that is available to all of the nodes in the system Also known as local stor
144. llowing from the EFI Boot Manager screen which is displayed during the power up of the node A NOTE For more information about how to work with these menus refer to the HP Integrity rx8620 32 Server Installation Guide which was shipped with the hardware a Choose the Boot Option Maintenance Menu on the EFI Boot Manager screen i Choose Add a Boot Option ii Choose Load File Acpi HWP0002 0 Pci 1 0 Mac XXXXXXXXXXXX which is the Gigabit Ethernet GigE port on the server iii Enter the string Netboot as the description for the boot option This entry is required and must be set to the string Netboot iv Enter N for No Boot Option when prompted for the Boot Option Data Type v Choose the option to save the entry to NVRAM vi Choose Exit to quit the Add a Boot Option menu b Choose the option to Change Boot Order from the EFI Boot Manager screen i Press the u key on the keyboard to move the Netboot entry you just defined to the top of the boot order ii Save the setting to the NVRAM iii Choose Exit to quit the Change Boot Order menu 13 Enable console messages a Choose the Select Active Console Output Devices option from the Boot Option Maintenance menu to enable console messages to be displayed on the screen when you turn on the system i Set the Acpi HWP0002 0 Pci 0 1 Uart 9600 N81 VenMsg Vt100 option this is the only option that is set ii Save the setting to the NVRAM iii Choose Exit to
145. m of 6 characters and the password must have a minimum of 8 characters You must set the same user name and password on every node You must have this user name and password to access the power management device and console 8 If required set static IP addresses For more information see Setting Static IP Addresses on Integrity Servers page 122 9 Enter PC power cycle and then enter on to turn on power to the node 10 Press Ctrl B to return to the MP Main Menu 11 Enter CO to connect to the console 12 Perform this step on all nodes except the head node From the Boot Menu screen which is displayed during the power on of the node select the Boot Configuration Menu Do the following from the Boot Configuration Menu a Select Add Boot Entry b Select Load File Core LAN Gb A as the network boot choice which is a Gigabit Ethernet GigE port c Enter the string Netboot as the boot option description This entry is required and must be set to the string Netboot with a capital letter N d Press the Enter key for no db profile options 130 Preparing Individual Nodes 13 14 15 e Press the Enter key for no boot options f When prompted save the entry to NVRAM For more information about how to work with these menus see the documentation that came with the HP Integrity server Perform this step on all nodes except the head node From the Boot Configuration menu select the Edit OS Boot Order opti
146. memory form of parallelism A serial application is basically a single processor application that has no communication library calls for example MPI PVM GM or Portals An example of a serial application is a standard Linux command such as the 1s command Another example of a serial application is a program that has been built on a Linux system that is binary compatible with the HP XC environment but does not contain any of the HP XC infrastructure libraries One of the modules of an HP BladeSystem The server blade is the compute module consisting of the CPU memory I O modules and other supporting hardware Server blades do not contain their own physical I O ports power supplies or cooling The node on which the optional backup slurmct1d daemon runs On SLURM failover this node becomes the SLURM master controller The node on which the slurmct1d daemon runs Symmetric multiprocessing A system with two or more CPUs that share equal symmetric access to all of the facilities of a computer system such as the memory and I O subsystems In 151 ssh standard LSF SVA symmetric multiprocessing 152 Glossary an HP XC system the use of SMP technology increases the number of CPUs amount of computational power available per unit of space Secure Shell A shell program for logging in to and executing commands on a remote computer It can provide secure encrypted communications between two untrusted hosts over an inse
147. ministration Network page 41 2 5 1 Configuring a Gigabit Ethernet Interconnect A Gigabit Ethernet interconnect requires one or more external Ethernet switches to act as the interconnect between the enclosures that make up the HP XC system On systems using a Gigabit Ethernet interconnect one NIC on each server blade is dedicated as the connection to the interconnect network On a server blade NIC2 is used for this purpose NIC2 of all server blades connects to interconnect bay 2 on the enclosure The entire interconnect network is formed by connecting the device either a switch or a pass thru module in interconnect bay 2 of each enclosure to one of the Gigabit Ethernet interconnect switches If the device is a switch the Gigabit uplink to the higher level ProCurve switch can be a single wire or a trunked connection of 2 4 or 8 wires If the device is a pass thru module there must be one uplink connection for each server blade in the enclosure Non blade server nodes must also be connected to the interconnect network See Chapter 3 page 45 to determine which port on the node is used for the interconnect network the port you use depends on your particular hardware model Figure 2 6 illustrates the connections for a Gigabit Ethernet interconnect 2 5 Cabling for the Interconnect Network 39 Figure 2 6 Gigabit Ethernet Interconnect Connections Non Blade Server PCI SLOT MGT Non Blade Server PCI SLOT MGT NIC PCI S
148. n Table 4 43 describes the settings for the xw9400 workstation Table 4 43 Setup Utility Settings for xw9400 Workstations Menu Name Submenu Name Option Name Set to This Value Storage Boot Order Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk Optical Drive USB device nVidia Network Controller 1 Hard Drive C nVidia Network Controller 2 ar WN Set the boot order on the head node as follows 1 Optical Drive 2 USB device 3 Hard Drive C 4 nVidia Network Controller 1 5 nVidia Network Controller 2 Advanced Power Sleep Wake After Power Loss On Select File Save Changes amp Exit to exit the Setup Utility Repeat this procedure for each workstation in the hardware configuration Turn off power to all nodes except the head node 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 117 10 Follow the software installation instructions in the HP XC System Software Installation Guide to install the HP XC System Software 118 Preparing Individual Nodes 4 8 Preparing the Hardware for CP4OOOBL Systems A Perform the following tasks on each server blade in the hardware configuration after the head node is installed and the switches are discovered e Set the boot order e Create an iLO2 user name and password e Set the power regulator e Configure smart array devices Use the Onboa
149. n enclosure for more information on enclosures see Server Blade Enclosure Components page 22 Full height Opteron server blades can take up to four dual core CPUs and Xeon server blades can take up to two quad cores Table 1 2 lists the HP ProLiant hardware models supported for use in an HP XC hardware configuration Table 1 2 Supported HP ProLiant Server Blade Models Core i HP Proliant Hot Plug Mezzanine Blade Model Height Processor Number Built In NICs Drives Slots BL2x220c G5 half Intel Xeon up to two quad core or 2 1 per 0 2 1 per up to two dual core per server node server node server node BL260c G5 half Intel Xeon up to two quad core or 1 0 1 up to two dual core BL460c half Intel Xeon up to two quad core or 2 2 2 up to two dual core BL465c half AMD Opteron up to two single core or 2 2 2 up to two dual core BL465c G5 half AMD Opteron up to two single core or 2 2 2 up to two dual core BL480c full Intel Xeon up to two quad core or 4 4 3 up to two dual core BL680c G5 full Intel Xeon two quad core or four 4 2 3 quad core BL685c full AMD Opteron up to four dual core 4 2 3 BL685c G5 full AMD Opteron up to four dual core 4 2 3 BL860c full Intel Itanium up to two quad core or 4 2 3 dual core For more information on an individual server blade see the QuickSpec for your model The QuickSpecs are located at the following Web address http www hp com go clusters 1 1 2 Supported Server Blade Combinatio
150. n each node is always used for connectivity to the administration network there must be a second Gigabit Ethernet port on each node if you are using Gigabit Ethernet as the interconnect Depending upon the hardware model the port can be built in or can be an installed card Any node with an external interface must also have a third Ethernet connection of any kind to communicate with external networks 3 4 3 Administration Network Interconnect Connections In cases where an additional Gigabit Ethernet port or switch may not be available the HP XC System Software allows the interconnect to be configured on the administration network When the interconnect is configured on the administration network only a single LAN is used However be aware that configuring the system in this way may negatively impact system performance To configure the interconnect on the administration network you include the ic AdminNet option on the discover command line which is documented in the HP XC System Software Installation Guide 3 4 Interconnect Connections 57 If you do not specify the ic AdminNet option the discover command attempts to locate the highest speed interconnect on the system with the default being a Gigabit Ethernet network that is separate from the administration network 3 4 4 Myrinet Interconnect Connections The supported Myrinet interconnects do not have the ordering requirements of the Quadrics interconnect but it makes sense
151. n the HP BladeSystem c7000 enclosure documentation e Enclosures are cabled together using their uplink and downlink ports e The top uplink port in each rack is used as a service port to attach a laptop or other device for initial configuration or subsequent debugging 24 Hardware and Network Overview Specific HP XC Setup Guidelines The following enclosure setup guidelines are specific to HP XC e Onevery HP BladeSystem c7000 enclosure an Ethernet interconnect module either a switch or pass through module is installed in interconnect bay 1 see callout 2 in Figure 1 4 for the administration network e Hardware configurations that use Gigabit Ethernet as the interconnect require an additional Ethernet interconnect module either a switch or pass through module to be installed in interconnect bay 2 see callout 3 in Figure 1 4 for the interconnect network e Systems that use InfiniBand as the interconnect require a double wide InfiniBand interconnect switch module installed in interconnect bays 5 and 6 see callouts 6 and 7 in Figure 1 4 e Some systems might need an additional Ethernet interconnect module to support server blades that require external connections For more information about external connections see Cabling for the External Network page 41 1 2 2 HP BladeSystem c3000 Enclosure Figure 1 5 shows the front and rear views of the HP BladeSystem c3000 enclosure Figure 1 5 HP BladeSystem c3000 Enclosure Fron
152. nd PDF that describes how to use software RAID under a Linux operating system http www linuxdevcenter com pub a linux 2002 12 05 RAID html Provides information about how to use the mdadm RAID management utility Additional Publications For more information about standard Linux system administration or other related software topics consider using one of the following publications which must be purchased separately Linux Administration Unleashed by Thomas Schenk et al Linux Administration Handbook by Evi Nemeth Garth Snyder Trent R Hein et al Managing NFS and NIS by Hal Stern Mike Eisler and Ricardo Labiaga O Reilly MySQL by Paul Debois e MySQL Cookbook by Paul Debois e High Performance MySQL by Jeremy Zawodny and Derek J Balling O Reilly e Perl Cookbook Second Edition by Tom Christiansen and Nathan Torkington e Perlin A Nutshell A Desktop Quick Reference by Ellen Siever et al Manpages Manpages provide online reference and command information from the command line Manpages are supplied with the HP XC system for standard HP XC components Linux user commands LSF commands and other software components that are distributed with the HP XC system Manpages for third party software components might be provided as a part of the deliverables for that component Using discover 8 as an example you can use either one of the following commands to display a manpage man discover man 8 discover
153. ndered useless Trunking configurations on switches must follow these guidelines e Because of the architecture of the ProCurve switch the HP XC System Software uses only 10 ports of each 12 port segment to ensure maximum bandwidth through the switch the last two ports are not used e Trunk groups must be contiguous Thus by adhering to the trunking guidelines the following ports are used to configure a ProCurve 2848 Super Root Switch for three regions using four wire trunks e Region 1 Ports 1 2 3 4 e Region 2 Ports 5 6 7 8 e Region 3 Ports 13 14 15 16 e Ports 9 10 11 and 12 are not used 3 3 Switches The following topics are addressed in this section e Specialized Switch Use page 46 e Administrator Passwords on ProCurve Switches page 47 e Switch Port Connections page 47 e Switch Connections and HP Workstations page 49 e Super Root Switch page 49 e Root Administration Switch page 50 e Root Console Switches page 51 e Branch Administration Switches page 54 e Branch Console Switches page 55 3 3 1 Specialized Switch Use 46 The following describes the specialized uses of switches in an HP XC system Super Root Switch This switch is the top level switch in a large scale system that is an HP XC system with more than 512 nodes requiring more than one Root Administration switch Root Administration switches are connected directly t
154. ndividual Nodes Ring Table 4 3 BIOS Settings for HP ProLiant DL140 G2 Nodes continued Menu Name Submenu Name Option Name Set to This Value Wake On LAN Disabled Boot Set the following boot order on all nodes except the head node 1 CD ROM Removable Devices PXE MBA V7 7 2 Slot 0200 Hard Drive PXE MBA V7 7 2 Slot 0300 means disabled goe p N Set the following boot order on the head node 1 CD ROM 2 Removable Devices p Hard Drive PXE MBA v7 7 2 SLot 0200 PXE MBA v7 7 2 SLot 0300 oe Table 4 4 lists the BIOS settings for HP ProLiant DL140 G3 nodes Table 4 4 BIOS Settings for HP ProLiant DL140 G3 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Features Numlock Disabled Advanced 8042 Emulation Disabled Support I O Device Serial Port BMC Configuration Console Redirection Console Redirection Enabled EMS Console Enabled Baud Rate 115 2K Continue C R after Enabled POST IPMI LAN Settings IP Address DHCP Assignment BMC Telnet Service Enabled BMC Ping Response Enabled BMC HTTP Service Enabled BMC HTTPS Service Enabled Boot Set the following boot order on the head node 1 CD ROM Removable Devices Hard Drive Embedded NIC1 Embedded NIC2 eo p 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 65 Table 4 4 BIOS Settings for HP ProLiant DL140 G3 Nodes continued Menu Name Submenu Name Option Name Set to This Value Set the f
155. ne card and interconnect module by creating virtual local area networks VLANs On a full height server blade NICs 1 and 3 are both connected to interconnect bay 1 and for the c7000 enclosure NICs 2 and 4 are both connected to interconnect bay 2 If you are using one of these NICs for the connection to the external network you might have to create a VLAN on the switch in that bay to separate the external network from other network traffic For information about configuring VLANs see Creating VLANs page 44 The ports and interconnect bays used for external network connections vary depending on the hardware configuration the ports that are already being used for the other networks and the server blade height For more information about how to configure the external network in these various configurations see the illustrations in the following sections e Configuring the External Network Option 1 page 41 e Configuring the External Network Option 2 page 42 e Configuring the External Network Option 3 Non Gigabit Ethernet Interconnect Clusters page 43 2 6 1 Configuring the External Network Option 1 Figure 2 8 page 42 assumes that NIC and NIC2 are already in use for the administration and interconnect networks This situation requires a third NIC for the external network Half height 2 6 Cabling for the External Network 41 server blades do not have three NICs and therefore half height server bl
156. ng the Hardware for CP6000 Intel Itanium Systems page 122 e Preparing the Hardware for CP6000BL Systems page 136 4 1 Firmware Requirements and Dependencies Before installing the HP XC System Software verify that all hardware components are installed with the minimum firmware versions listed in the master firmware list You can find this list from the following Web page http docs hp com en linuxhpc html Look in the associated hardware documentation for instructions about how to verify or upgrade the firmware for each component Table 4 1 lists the firmware dependencies of individual hardware components in an HP XC system Table 4 1 Firmware Dependencies Hardware Component Firmware Dependency CP3000 HP ProLiant DL140 G2 Lights out 100i management LO 100i system BIOS HP ProLiant DL140 G3 LO 100i system BIOS HP ProLiant DL160 G5 LO 100i system BIOS HP ProLiant DL360 G4 Integrated lights out iLO system BIOS HP ProLiant DL360 G4p iLO system BIOS HP ProLiant DL360 G5 iLO2 system BIOS HP ProLiant DL380 G4 iLO system BIOS HP ProLiant DL380 G5 iLO2 system BIOS HP ProLiant DL580 G4 iLO2 system BIOS HP ProLiant DL580 G5 iLO2 system BIOS CP3000BL HP ProLiant BL2x220c G5 iLO2 system BIOS Onboard Administrator OA HP ProLiant BL260c G5 iLO2 system BIOS Onboard Administrator OA HP ProLiant BL460c iLO2 system BIOS Onboard Administrator OA HP ProLiant BL480c iLO2 system BIOS OA 4 1
157. nnect 41 57 console branch 31 defined 30 37 application cabinet 45 architecture see processor architecture B HP ProLiant BL260c G5 Server Blade 20 baseboard management controller see BMC HP ProLiant BL2x220c G5 Server Blade 20 BIOS settings HP ProLiant BL460c G5 Server Blade 20 HP ProLiant DL140 G2 65 HP ProLiant BL480c G5 Server Blade 20 HP ProLiant DL140 G3 66 HP ProLiant BL680c G5 Server Blade 20 HP ProLiant DL145 88 CP4000 19 HP ProLiant DL145 G2 91 hardware preparation tasks 87 HP ProLiant DL145 G3 92 HP ProLiant DL145 20 87 HP ProLiant DL160 G5 68 HP ProLiant DL145 G2 20 89 HP ProLiant DL165 G5 94 HP ProLiant DL145 G3 20 89 HP ProLiant DL360 G4 69 HP ProLiant DL165 G5 20 93 HP ProLiant DL360 G5 71 HP ProLiant DL380 G4 74 HP ProLiant DL380 G5 75 HP ProLiant DL580 G4 78 HP ProLiant DL580 G5 80 HP xw8200 workstation 81 HP xw8400 workstation 82 HP xw8600 workstation 83 BMC 29 BMC firmware 59 boot order 84 119 branch administration switch 47 54 branch console switch 47 55 HP ProLiant DL380 G4 20 72 HP ProLiant DL380 G5 20 72 HP ProLiant DL580 G4 20 75 HP ProLiant DL580 G5 20 78 HP xw8200 workstation 20 80 HP xw8400 workstation 20 80 HP xw8600 workstation 20 82 CP3000BL 19 hardware preparation tasks 84 HP ProLiant DL365 20 94 HP ProLiant DL365 G5 20 97 HP ProLiant DL385 20 100 HP ProLiant DL385 G2 20 100 HP ProLiant DL385 G5 20 104 HP ProLia
158. nnect a three way DB9 25 cable to the MP DB 9 port on the back of the HP Integrity rx4640 server This port is the first of the four DB9 ports at the bottom left of the server it is labeled MP Local b Connect the CONSOLE connector to a null modem cable and connect the null modem cable to the PC COM1 port c Use a terminal emulator such as HyperTerminal to open a terminal window Press the Enter key to access the MP If the MP does not respond press the MP reset pin on the back of the MP and try again e Log into the MP using the default user name and password shown on the screen The MP Main Menu appears MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 4 Enter SL to show event logs Then enter C to clear all log files and Y to confirm Enter CM to display the Command Menu 6 Perform the following steps to ensure that the IPMI over LAN option is set This setting is required for Nagios monitoring a Enter LC b Verify the IPMI over LAN option is enabled c Enable this option if it is disabled d Return to the Command Menu od 7 Enter UC and use the menu options to remove the default MP user name and password and create your own unique user name and password HP recommends setting your own user name and password for security purposes The user name must have a minimu
159. nodes The top portion shows a Gigabit Ethernet switch with two connections for each of the four enclosures one connection to the ProCurve managed Gigabit Ethernet switch on the enclosure and the other to the Onboard Administrator The bottom portion provides some additional detail It shows the ProCurve managed Gigabit Ethernet switch connected to the Gigabit Ethernet switch in bay 1 of each enclosure and to the Primary Onboard Administrator External Link of each enclosure Cabling Server Blades Figure 2 1 Interconnection Diagram for a Small HP XC Cluster of Server Blades ProCurve managed GigE Switch Enclosure 1 Enclosure 2 Enclosure 3 Enclosure 4 Switch 1 Switch 1 Switch 1 Switch 1 Enclosure 1 Enclosure 2 Enclosure 3 Enclosure 4 Server Blades Server Blades Server Blades Server Blades NIC NIC1 NIC NIC Enclosure 1 Enclosure 2 Enclosure 3 Enclosure 4 OA OA OA OA May be any managed GigE switch supported by CP ProCurve managed GigE Switch Enclosure 4 Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 3 Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 2 Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 1 Secondary OA Ext Link Primary OA Ext Link GigE Switch Medium Sized HP XC Cluster of Server Blades Figure 2 2 page 36 provides two illustrations of a medium sized HP XC cluster of 32 enclosures and a maximum of 512 nodes The top por
160. ns The HP XC System Software supports the following server blade hardware configurations e A hardware configuration composed entirely of HP server blades that is the head node the service nodes and all compute nodes are server blades e A hardware configuration can contain a mixture of Opteron and Xeon server blades but not Itanium server blades e A mixed hardware configuration of HP server blades and non blade servers where The head node can be either a server blade or a non blade server Service nodes can be either server blades or non blade servers All compute nodes are server blades 1 1 Supported Cluster Platforms 21 1 2 Server Blade Enclosure Components HP server blades are contained in an enclosure which is a chassis that houses and connects blade hardware components An enclosure is managed by an Onboard Administrator The HP BladeSystem c7000 and c3000 enclosures are supported under HP XC This section discusses the following topics e HP BladeSystem c7000 Enclosure page 22 e HP BladeSystem c3000 Enclosure page 25 e HP BladeSystem c Class Onboard Administrator page 27 e Insight Display page 28 For more information about enclosures and their related components see the HP Server Blade c7000 Enclosure Setup and Installation Guide 1 2 1 HP BladeSystem c7000 Enclosure Figure 1 1 shows front and rear views of the HP BladeSystem c7000 enclosure Figure 1 1 HP BladeSystem c
161. nt DL385 G5 73 HP ProLiant DL580 G4 77 HP ProLiant DL580 G5 79 HP ProLiant DL585 107 HP ProLiant DL585 G2 107 HP ProLiant DL785 G5 114 iLO2 CONNECT_ERROR 139 defined 29 features 29 setting the password 84 119 web interface 139 InfiniBand interconnect 40 insight display defined 28 Integrated Lights Out 2 see iLO2 Intelligent platform management interface see IPMI interconnect connections 56 console connection 50 Gigabit Ethernet 31 57 InfiniBand 31 58 Myrinet 31 58 network 31 on administration network 57 OsNet 31 57 interconnect bay port mapping 28 interconnect module 28 interconnect network defined 39 Gigabit Ethernet 39 InfiniBand 40 running on administration network 41 IP address for MP 122 133 IPMI 29 firmware 59 L large scale system defined 32 lights out 100 see LO 100i line monitoring card connection 50 LO 100i 29 LSF documentation 14 M management processor see MP manpages 17 mdadm utility 16 mezzanine cards 28 MP 29 accessing 125 127 129 defined 29 features 29 setting IP address 122 133 MP firmware 59 Myrinet interface cards revision 59 N network administration 30 37 administration console branch 31 console 38 external 41 interconnect 31 39 network cabling 33 network configuration 33 nodes maximum number in system 32 maximum number of 56 O OFED 58 onboard administrator defined 27
162. nt DL585 20 106 HP ProLiant DL585 G2 20 106 HP ProLiant DL585 G5 20 110 HP ProLiant DL785 G5 20 113 HP xw9300 workstation 20 116 HP xw9400 workstation 20 116 CP4000BL hardware preparation tasks 119 HP ProLiant BL465c G5 Server Blade 20 C HP ProLiant BL465c Server Blade 20 c3000 enclosure 25 HP ProLiant BL685c G5 Server Blade 20 c7000 server blade enclosure 22 HP ProLiant BL685c Server Blade 20 cabinet 45 chip architecture see processor architecture cluster platform see CP3000 see CP3000BL see CP4000 see CP6000 supported 19 CONNECT_ERROR 139 console branch network 31 console management devices 29 console network CP6000BL defined 38 hardware preparation tasks 136 core IO board 0 133 HP Integrity BL860c Server Blade 20 CP3000 19 hardware preparation tasks 63 D HP ProLiant DL140 G2 20 63 DHCP 89 HP ProLiant DL140 G3 20 63 documentation HP ProLiant DL160 G5 20 66 HP ProLiant DL360 G4 20 68 HP ProLiant DL360 G4p 20 68 HP ProLiant DL360 G5 20 70 CP6000 19 hardware preparation tasks 122 HP Integrity rx1620 20 122 HP Integrity rx2600 20 122 HP Integrity rx2620 20 125 HP Integrity rx2660 20 127 HP Integrity rx4640 20 129 HP Integrity rx8620 20 131 additional publications 16 compilers 16 FlexLM 15 HowTo 12 153 HP XC System Software 12 Linux 15 LSF 14 manpages 17 master firmware list 12 Modules 15 MPI 16 MySQL 15 Nagios 14
163. nt tick count 2433 ping delay time 280 ms ping host 0 0 0 0 Task wakeups netmain 93 nettick 4814 telnetsrv 401 c Ifthe value for ipsrc is nvmem enter dhcp at the INET gt prompt INET gt dhcp Configuring for the enabling of DHCP Note Configuration change has been made but changes will not take effect until the processor has been rebooted Do you wish to reboot the processor now may take 10 seconds y or n d Enter y to reboot the processor 7 Ifyou did not change the DHCP setting press ShifttEsc Q or enter quit at the ProLiant gt prompt to exit the Management Processor CLI and invoke the Console mode 4 7 2 Preparing HP ProLiant DL145 G2 and DL145 G3 Nodes Use the BIOS Setup utility on HP ProLiant DL145 G2 and DL145 G3 servers to configure the appropriate settings for an HP XC system For these hardware models you cannot set or modify the default console port password through the BIOS Setup Utility the way you can for other hardware models The HP XC System Software Installation Guide documents the procedure to modify the console port password You are instructed to perform the task just after the discover command discovers the IP addresses of the console ports Figure 4 12 shows a rear view of the HP ProLiant DL145 G2 server and the appropriate port assignments for an HP XC system Figure 4 12 HP ProLiant DL145 G2 Server Rear View 2626262 Je O80208 BERORSROG0
164. nying hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F10 key when prompted to access the BIOS Setup Utility 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 87 88 For each node make the BIOS settings listed in Table 4 21 Table 4 21 BIOS Settings for HP ProLiant DL145 Nodes Menu Name Submenu Name Option Name Set to This Value Boot Boot Settings Onboard NIC PXE Option Enabled for all nodes except the head Configuration for ROM node NIC1 Boot Settings Onboard NIC PXE Option Disabled for the head node Configuration for ROM NIC1 Advanced Management Set Serial Port Sharing Shared Processor Configuration BIOS Serial Console Redirection After BIOS Enabled Configuration Post Boot Boot Device Priority Maintain the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 NIC1 3 Hard Disk Set the head node to boot from CD ROM first the hard disk must be listed after CD ROM 1 TheNTIC1 interface is named Broadcom MBA and it is the second choice with this name from the Boot Screen Menu Boot Device Priority For each HP ProLiant DL145 node log in to the IPMI utility and invoke the Terminal mode 1 oe Sb Establish a connection to the server by using one
165. o this switch Making Node and Switch Connections Root Administration Switch This switch connects directly to Gigabit Ethernet ports of the head node the Root Console Switch Branch Administration Switches and other nodes in the utility cabinet Root Console Switch This switch connects to the Root Administration Switch Branch Console Switches and connects to the management console ports of nodes in the utility cabinet Branch Administration Switch This switch connects to the Gigabit Ethernet ports of compute nodes and connects to the Root Administration Switch Branch Console Switch This switch connects to the Root Console Switch and connects to the management console ports of the compute nodes we IMPORTANT Switch use is not strictly enforced on HP XC systems with HP server blades For more information about switch use with HP server blades see Chapter 2 page 33 Table 3 1 lists the switch models that are supported for each use Table 3 1 Supported Switch Models Switch Use ProCurve Switch Model Administration Switch ProCurve 2848 or 2824 Console Switch ProCurve 2650 or 2626 3 3 2 Administrator Passwords on ProCurve Switches The documentation that came with the ProCurve switch describes how to optionally set an administrator s password for the switch If you define and set a password on a ProCurve switch you must set the same password on every ProCurve switch that is a component of the HP XC system Durin
166. o create a higher bandwidth connection between the Root Administration Switches and the Branch Administration Switches For physically small hardware models such as a 1U HP ProLiant DL145 server a large number of servers more than 30 can be placed in a single cabinet and are all attached to a single branch switch The branch switch is a ProCurve Switch 2848 and two port trunking is used for the connection between the Branch Administration Switch and the Root Administration Switch For physically larger hardware models 2U and larger such the HP Integrity rx2600 and HP ProLiant DL585 servers a smaller number of servers can be placed in a single cabinet In this case the branch switch is a ProCurve Switch 2824 which is sufficient to support up to 19 nodes In this release the HP XC System Software supports the use of multiwire connections or trunks between switches in the system In a large scale system one that has regions and uses a super root switch you can use a one wire to four wire trunk between the super root switch and the root administration switch for each of the HP XC regions On a smaller scale HP XC system or within a single region one wire and two wire trunks are supported for connection between the root administration switch and the branch administration switches You must configure trunks on both switches before plugging the cables in between the switches Otherwise a loop is created between the switches and the network is re
167. o do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Preparing Individual Nodes Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 7 5 Preparing HP ProLiant DL365 G5 Nodes On HP ProLiant DL365 G5 servers use the following tools to configure the appropriate settings for an HP XC system e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL365 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 16 shows a rear view of the HP ProLiant DL365 G5 server and the appropriate port assignments for an HP XC system Figure 4 16 HP ProLiant DL365 G5 Server Rear View o S EP QOS The callouts in the figure enumerate the following 1 Ifa Gigabit Ethernet GigE interconnect is configured this port is used for the interconnect connection Otherwise it is used for an external connection On the back of the node this port is marked with the number 2 2 This port is the
168. o return to the main menu Table 4 12 lists the BIOS settings for HP ProLiant DL380 G5 BIOS nodes Table 4 12 BIOS Settings for HP ProLiant DL380 G5 Nodes Menu Name Option Name Set to This Value System Options Virtual Serial Port COM1 IRQ4 IO 3F8h 3FFh Press the Esc key to return to the main menu Embedded Serial Port COM2 IRQ3 IO 2F8h 2FFh Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 Floppy Drive A 3 USB DriveKey C 4 PCI Embedded HP NC373i Multifunction Gigabit Adapter 5 Hard Disk C see Boot Controller Order On the head node set the boot order so that the CD ROM is listed before the hard disk Preparing Individual Nodes Table 4 12 BIOS Settings for HP ProLiant DL380 G5 Nodes continued Menu Name Option Name Set to This Value Advanced Options Processor Hyper_threading Disable BIOS Serial Console amp BIOS Serial Console Port COM1 IRQ4 IO 3F8h 3FFh EMS BIOS Serial Console Baud Rate 115200 EMS Console Disabled BIOS Interface Mode Command Line Press the Esc key to return to the main menu 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL380 G4 and G5 node in the hardware configuration Configuring Smart Arrays
169. of the following methods e A serial port connection to the console port e A telnet session to the IP address of the Management NIC A NOTE For more information about how to establish these connections see Establishing a Connection Through a Serial Port page 141 or the documentation that came with the HP ProLiant server Press the Esc key and then press Shift 9 to display the IPMI setup utility Enter the administrator s user name at the login prompt the default is admin Enter the administrator s password at the password prompt the default is admin Use the C hange Password option to change the console port management device password The factory default password is admin change it to the password of your choice This password must be the same on every node in the hardware configuration ProLiant gt ChangePassword Type the current password gt admin Type the new password max 16 characters gt your password Retype the new password max 16 characters gt your password New password confirmed Preparing Individual Nodes 6 Ensure that all machines are requesting IP addresses through the Dynamic Host Control Protocol DHCP Do the following to determine if DHCP is enabled a Atthe ProLiant gt prompt enter the following ProLiant gt net b Atthe INET gt prompt enter the following INET gt state iface ipsrce IP addr subnet gateway l etl dhcp 0 0 0 0 255 0 0 0 0 0 0 0 curre
170. og in to the Onboard Administrator for the enclosure Setup Procedure Use the following procedure to prepare the CP3000BL server blades 1 In the left frame of the HP Onboard Administrator browser window click the plus sign next to Device Bays to display the list of nodes contained in the enclosure Click the link to the first hardware model in the list Wait a few seconds until the frame to the right is populated with node specific information Click the Boot Options tab a Select a boot device and use the up and down arrows on the screen to position the device so that it matches the boot order listed in Table 4 20 Table 4 20 Boot Order for HP ProLiant Server Blades Head Node All Other Nodes Set the following boot order on the head node Set the following boot order on all nodes except the 1 USB head node 2 Floppy 1 USB 3 CD 2 Floppy 4 Hard Disk 3 CD 5 PXE NIC1 4 PXE NIC 1 5 Hard Disk b Click the Apply button In the left frame do the following to create a new iLO2 user name and password on this node a Under the hardware model click iLO b Inthe body of the main window click the Web Administration link to open the Integrated Lights Out 2 utility in a new window You might have to turn off popup blocking for this window to open c Inthe new window click the Administration tab In the left frame click the User Administration link e Click the New button and create a new iLO2 user name
171. ol a round robin database tool and graphing system In the HP XC system RRDtool is used with Nagios to provide a graphical view of system status http supermon sourceforge net Home page for Supermon a high speed cluster monitoring system that emphasizes low perturbation high sampling rates and an extensible data protocol and programming interface Supermon works in conjunction with Nagios to provide HP XC system monitoring http www llnl gov linux pdsh Home page for the parallel distributed shell odsh which executes commands across HP XC client nodes in parallel http www balabit com products syslog_ng Home page for syslog ng a logging tool that replaces the traditional syslog functionality The syslog ng tool is a flexible and scalable audit trail processing tool It provides a centralized securely stored log of all devices on the network http systemimager or Home page for SystemImager which is the underlying technology that distributes the golden image to all nodes and distributes configuration changes throughout the system http linuxvirtualserver org Home page for the Linux Virtual Server LVS the load balancer running on the Linux operating system that distributes login requests on the HP XC system http www macrovision com Home page for Macrovision developer of the FLEXIm license management utility which is used for HP XC license management http sourceforge net projects modul
172. ollowing Platform Computing Corporation LSF documents are wey on the HP XC documentation CD in PDF format Administering Platform LSF Administration Primer Platform LSF Reference Quick Reference Card Running Jobs with Platform LSF LSF procedures and information supplied in the HP XC documentation particularly the documentation relating to the LSF HPC integration with SLURM supersedes the information supplied in the LSF manuals from Platform Computing Corporation The Platform Computing Corporation LSF manpages are installed by default Isf_diff 7 supplied by HP describes LSF command differences when using LSF HPC with SLURM on an HP XC system The following documents in the HP XC System Software Documentation Set provide information about administering and using LSF on an HP XC system HP XC System Software Administration Guide HP XC System Software User s Guide https computing llnl gov linux slurm documentation html Documentation for the Simple Linux Utility for Resource Management SLURM which is integrated with LSF to manage job and compute resources on an HP XC system http www nagios org Home page for Nagios a system and network monitoring application that is integrated into an HP XC system to provide monitoring capabilities Nagios watches specified hosts and services and issues alerts when problems occur and when problems are resolved http oss oetiker ch rrdtool Home page of RRDto
173. ollowing boot order on all nodes except the head node 1 CD ROM 2 Removable Devices 3 Embedded NIC1 4 Hard Drive 5 Embedded NIC2 Embedded NIC1 Enabled PXE Embedded NIC2 Disabled PXE Power Resume On Modem off Ring Wake On LAN Disabled 4 From the Main window select Exit Save Changes and Exit to exit the utility 5 If the DL140 G3 node uses SATA disks you must disable the parallel ATA option otherwise the disk might not be recognized and imaged Use the following menus to disable this option BIOS Menu Name Submenu Name Option Name Set to This Value Advanced Advanced Chipset Parallel ATA Disabled Control 6 Repeat this procedure for each HP ProLiant DL140 G2 and G3 node in the HP XC system 4 5 2 Preparing HP Proliant DL160 G5 Nodes 66 Use the BIOS Setup Utility to configure the appropriate settings for an HP XC system on HP ProLiant DL160 G5 servers For this hardware model you cannot set or modify the default console port password through the BIOS Setup Utility as you can for other hardware models The HP XC System Software Installation Guide describes how to modify the console port password You are instructed to perform the task just after the discover command discovers the IP addresses of the console ports Figure 4 2 shows the rear view of the HP ProLiant DL160 G5 server and the appropriate port assignments for an HP XC system Figure 4 2 HP ProLiant DL160 G5 Server Rear View
174. on Do the following a Use the navigation instructions on the screen to move the Net boot entry you just defined to the top of the boot order b If prompted press the Enter key to select the position c Enter x to return to the Boot Configuration menu Perform this step on all nodes including the head node Select the Console Configuration option and do the following a Enable the P Serial Acpi HWP0002 PNP 0A03 0 Pci 1 1 Vt100 9600 option on HP Integrity rx4640 servers b When prompted save the entry to NVRAM c Enter x to return to the previous menu Turn off power to the node a Press Ctrl B to exit the console mode b Enter CM to display the Command Menu c Enter PC and enter off to turn off power to the node 4 9 6 Preparing HP Integrity rx8620 Nodes Follow this procedure to prepare each HP Integrity rx8620 node 2 V4 IMPORTANT Integrity rx8620 Nodes are configured with static addresses not DHCP On an HP XC system with one or more Integrity rx8620 nodes you must configure the Management Processor MP on all nodes for static IP addresses rather than DHCP General Hardware Preparation The connection of nodes to ProCurve switch ports is important for the automatic discovery process 1 Connect the Gigabit Ethernet ports on the HP Integrity rx8620 Core IO boards into the ProCurve 28xx switch at the next available ports Use one Gigabit Ethernet port for each partition for a total of two ports
175. on Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 4 shows a rear view of the HP ProLiant DL360 G5 server and the appropriate port assignments for an HP XC system Figure 4 4 HP ProLiant DL360 G5 Server Rear View 2 l0 EENG The callouts in the figure enumerate the following 1 This port is used for the connection to the Console Switch 2 This port NIC1 is used for the connection to the Administration Switch branch or root 3 The second onboard NIC is used for the Gigabit Ethernet interconnect or for the connection to the external network Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL360 G5 node in the HP XC system 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 For each node make the iLO settings listed in Table 4 8 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Preparing Individua
176. ork Overview 30 The administration network is a private network within the HP XC system that is used primarily for administrative operations This network is treated as a flat network during run time that is communication between any two points in the network is equal in communication time between any other two points in the network However during the installation and configuration of the Hardware and Network Overview HP XC system the administrative tools probe and discover the topology of the administration network The administration network requires and uses Gigabit Ethernet The administration network has at least one Root Administration Switch and can have multiple Branch Administration Switches These switches are discussed in Switches page 46 1 7 Administration Network Console Branch The console branch is part of the private administration network within an HP XC system that is used primarily for managing and monitoring the consoles of the nodes that comprise the HP XC system This branch of the network uses 10 100 Mbps Ethernet During the installation and configuration of the HP XC system the administrative tools probe and discover the topology of the entire administration network including the console branch A nonblade HP XC system has at least one Root Console Switch with the potential for multiple Branch Console Switches Figure 1 11 shows a graphical representation of the console branch Figure 1 11 Admin
177. ort Network DNS DHCP DHCP Enable on Settings CLI Serial CLI 115200 Press the F10 key to save the Speed setting bits seconds 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 79 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL580 G5 node in the HP XC system 1 Make the following settings from the Main menu The BIOS settings for HP ProLiant DL580 G5 nodes are listed in Table 4 14 Table 4 16 BIOS Settings for HP ProLiant DL580 G5 Nodes Menu Name Option Name Set to This Value Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk System Options Embedded Serial Port COM2 Virtual Serial Port COM1 Press the Esc key to return to the main menu BIOS Serial Console amp BIOS Serial Console Port COM1 EMS BIOS Serial Console Baud Rate 115200 EMS Console Disable BIOS Interface Mode Command Line Press the Esc key to return to the main menu 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choi
178. ponse press the MP reset pin on the back of the MP and try again 5 Log in to the MP using the default administrator name and password that shown on the screen The MP Main Menu is displayed MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 6 Enter SL to show event logs Then enter C to clear all log files and y to confirm your action 7 Enter CM to display the Command Menu 8 Do the following to ensure that the IPMI over LAN option is set This setting is required for Nagios monitoring a Enter SA to display the Set Access configuration menu b Verify that the IPMI over LAN option is enabled c Enable the IPMI over LAN option if it is disabled 1 Enter the letter i to access the IPMI 2 Enter the letter e to enable the IPMI over LAN 3 Enter the letter y to confirm your action d Return to the Command Menu 9 Enter UC user configuration and use the menu options to remove the default administrator and operator accounts Then for security purposes create your own unique user login ID and password Assign all rights privileges to this new user The user login ID must have a minimum of 6 characters and the password must have exactly 8 characters You must set the same user login ID and password on every node and all MPs 136 Preparing Individual Nodes 10 11 12 13 14
179. r Blade 20 21 HP ProLiant BL2x220c G5 Server Blade 20 21 HP ProLiant BL460c Server Blade 20 21 HP ProLiant BL465c G5 Server Blade 20 21 HP ProLiant BL465c Server Blade 20 21 HP ProLiant BL480c Server Blade 20 21 HP ProLiant BL680c G5 Server Blade 20 21 HP ProLiant BL685c G5 Server Blade 20 21 HP ProLiant BL685c Server Blade 20 21 HP ProLiant DL140 G2 20 63 HP ProLiant DL140 G3 20 63 HP ProLiant DL145 20 87 HP ProLiant DL145 G2 20 89 HP ProLiant DL145 G3 20 89 HP ProLiant DL160 G5 20 66 HP ProLiant DL165 G5 20 93 HP ProLiant DL360 G4 20 68 HP ProLiant DL360 G4p 20 68 HP ProLiant DL360 G5 20 70 HP ProLiant DL365 20 94 97 HP ProLiant DL365 G5 20 HP ProLiant DL380 G4 20 72 HP ProLiant DL380 G5 20 72 HP ProLiant DL385 20 100 HP ProLiant DL385 G2 20 100 HP ProLiant DL385 G5 20 104 HP ProLiant DL580 G4 20 75 HP ProLiant DL580 G5 20 78 HP ProLiant DL585 20 106 HP ProLiant DL585 G2 20 106 HP ProLiant DL585 G5 20 110 HP ProLiant DL785 G5 20 113 HP xw8200 workstation 20 80 HP xw8400 workstation 20 80 HP xw8600 workstation 20 82 HP xw9300 workstation 20 116 HP xw9400 workstation 20 116 iLO 29 59 enabling telnet 29 Ethernet port 90 95 97 100 101 105 iLO settings HP ProLiant D360 G4 69 HP ProLiant DL360 G5 71 HP ProLiant DL365 95 HP ProLiant DL365 G5 98 HP ProLiant DL380 G4 73 HP ProLiant DL385 101 HP ProLiant DL385 G2 102 105 HP ProLia
180. ration console interconnect and external public In order for these networks to function you must connect the enclosures server blades and switches according to the guidelines provided in this chapter 2 1 Blade Enclosure Overview 33 34 Chapter 3 page 45 describes specific node and switch connections for non blade hardware configurations A hardware configuration with server blades does not have these specific cabling requirements specific switch port assignments are not required However HP recommends a logical ordering of the cables on the switches to facilitate serviceability Enclosures are discovered in port order so HP recommends that you cable them in the order you want them to be numbered Also HP recommends that you cable the enclosures in lower ports and cable the external nodes in the ports above them The configuration of an HP XC Blade System depends on its size Larger clusters require additional switches to manage the additional enclosures or regions Figure 2 1 page 35 Figure 2 2 page 36 and Figure 2 3 page 37 provide a general view of the cabling requirements for small medium and large systems Additionally Appendix B page 143 provides several network cabling illustrations based on the interconnect type and server blade height to use as a reference Small HP XC Cluster of Server Blades Figure 2 1 page 35 provides two illustrations of a small HP XC cluster of four enclosures and a maximum of 64
181. rd Administrator the iLO2 web interface and virtual media to make the appropriate settings on HP ProLiant Server Blades NOTE The following setup procedure continues from the procedure in Setting the Onboard Administrator Password page 62 in which you used a browser to log in to the Onboard Administrator for the enclosure Setup Procedure Use the following procedure to prepare the CP4000BL server blades 1 Inthe left frame of the HP Onboard Administrator browser window click the plus sign next to Device Bays to display the list of nodes contained in the enclosure 2 Click the link to the first hardware model in the list Wait a few seconds until the frame to the right is populated with node specific information 3 Click the Boot Options tab a Select a boot device and use the up and down arrows on the screen to position the device so that it matches the boot order listed in Table 4 44 Table 4 44 Boot Order for HP ProLiant Server Blades Head Node All Other Nodes Set the following boot order on the head node Set the following boot order on all nodes except the 1 USB head node 2 Floppy 1 USB 3 CD 2 Floppy 4 Hard Disk 3 CD 5 PXE NIC1 4 PXE NIC 1 5 Hard Disk b Click the Apply button 4 Inthe left frame do the following to create a new iLO2 user name and password on this node a Under the hardware model click iLO b Inthe body of the main window click the Web Administration link to open t
182. rdware documentation to connect a monitor mouse and keyboard to the node For each node in the system ensure that the power cord is connected but that the processor is not turned on Follow this procedure to connect a personal computer to the Management Processor a Connect a three way DB9 25 cable to the MP DB 25 port on the back of the HP Integrity rx2620 server b Connect the CONSOLE connector to a null modem cable and connect the null modem cable to the PC COM1 port 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 125 c Use a terminal emulator such as HyperTerminal to open a terminal window Press the Enter key to access the MP If there is no response press the MP reset pin on the back of the MP and try again e Log in to the MP using the default user name and password shown on the screen The MP Main Menu appears MP MAIN MENU CO Console VFP Virtual Front Panel CM Command Menu SMCLP Server Management Command Line Protocol CL Console Log SL Show Event Logs HE Main Help Menu X Exit Connection 4 Enter SL to show event logs Then enter C to clear all log files and Y to confirm Enter CM to display the Command Menu 6 Perform the following steps to ensure that the IPMI over LAN option is set This setting is required for Nagios monitoring a Enter LC b Verify the IPMI over LAN option is enabled c Enable this option if it is disabled d Return to the Command Menu Ji 7 Enter
183. re required to prepare each supported hardware model for installation and configuration including required node and switch connections Provides step by step instructions for installing the HP XC System Software on the head node and configuring the system Provides an overview of the HP XC system administrative environment cluster administration tasks node maintenance tasks LSF administration tasks and troubleshooting procedures HP XC System Software User s Guide Provides an overview of managing the HP XC user environment with modules managing jobs with LSF and describes how to build run debug and troubleshoot serial and parallel applications on an HP XC system QuickSpecs for HP XC System Software Provides a product overview hardware requirements software requirements software licensing information ordering information and information about commercially available software that has been qualified to interoperate with the HP XC System Software The QuickSpecs are located on line http www hp com go clusters See the following sources for information about related HP products HP XC Program Development Environment The Program Development Environment home page provide pointers to tools that have been tested in the HP XC program development environment for example TotalView and other debuggers compilers and so on http h20311 www2 hp com HPC cache 276321 0 0 0 121 html HP Message Passing Interface HP
184. ress the b key to set the baud rate to 115200 Press the Esc key or enter x as many times as necessary to return to the Boot Menu Turn off power to the node a Press Ctrl B to exit the console mode b Enter CM to display the Command Menu c Enter PC and enter off to turn off power to the node 4 9 5 Preparing HP Integrity rx4640 Nodes Figure 4 30 shows a rear view of the HP Integrity rx4640 server and the appropriate port assignments for an HP XC system Figure 4 30 HP Integrity rx4640 Server Rear View 5o Ho MN WZ HPTC 0146 The callouts in the figure enumerate the following 1 2 3 The port labeled MP LAN is the MP connection to the ProCurve Console Switch The port labeled LAN Gb connects to the Administration Switch branch or root This unlabeled port is used for an external connection Setup Procedure Perform the following procedure on each HP Integrity rx4640 server 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node For each node in the system ensure that the power cord is connected but that the processor is not turned on Follow this procedure to connect a personal computer to the Management Processor 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 129 a Co
185. return to the Boot Option Maintenance menu b Choose the Select Active Console Input Devices option from the Boot Option Maintenance Menu to enable console messages to be displayed on the screen when you turn on the system i Set the Acpi HWP0002 0 Pci 0 1 Uart 9600 N81 VenMsg Vt100 option this is the only option that is set ii Save the setting to the NVRAM iii Choose Exit to return to the Boot Option Maintenance menu c Choose the Select Active Standard Error Devices option from the Boot Option Maintenance menu to enable console messages to be displayed on the screen when you turn on the system i Set the Acpi HWP0002 0 Pci 0 1 Uart 9600 N81 VenMsg Vt100 option this is the only option that is set ii Save the setting to the NVRAM iii Choose Exit to return to the Boot Option Maintenance menu 134 Preparing Individual Nodes 14 15 16 17 From the Boot Option Maintenance menu add a boot option for the EFI Shell if one does not exist Follow the instructions in step 12a Exit the Boot Option Maintenance menu Choose the EFI Shell boot option and boot to the EFI shell Enter the following EFI shell commands EFI gt acpiconfig enable softpowerdown EFI gt acpiconfig single pci domain EFI gt reset The reset command reboots the machine You do not have to wait for this reboot to complete before continuing to the next step to turn off power to the partition Turn off power to the partition leave t
186. rked as NIC2 3 The port labeled LO100i is used for the connection to the Console Switch Setup Procedure Perform the following procedure for each HP ProLiant DL145 G2 and DL145 G3 node in the HP XC system Change only the values that are described in this procedure do not change any factory set values unless you are instructed to do so 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F10 key when prompted to access the BIOS Setup Utility You configure the Lights Out 100i LO 100i console management device using this utility The BIOS Setup Utility displays the following information about the node BIOS ROM ID BIOS Version BIOS Build Date Record this information for future reference 3 Make the following BIOS settings for each node depending on hardware model e Table 4 22 provides the BIOS settings for ProLiant DL145 G2 nodes e Table 4 23 provides the BIOS settings for ProLiant DL145 G3 nodes Table 4 22 BIOS Settings for HP ProLiant DL145 G2 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Options Numlock off Advanced MCFG Table Disabled NIC Option Dedicated NIC Preparing Individual Nodes Table 4 22 BIOS Settings for HP ProLiant DL145 G2 Nodes continued Menu Name Submenu Name Hammer Configuration PCI Config
187. rm types if you have not already done so 1 we The connection of nodes to ProCurve switch ports is important for the automatic discovery process Ensure that all nodes are connected as described in Making Node and Switch Connections page 45 When possible ensure that switches are configured to obtain IP addresses using DHCP For more information on how to do this see the documents that came with the ProCurve hardware ProCurve documents are also available at the following Web page http www hp com go hpprocurve IMPORTANT Some HP Integrity hardware models must be configured with static addresses not DHCP For HP XC systems with one or more of these hardware models you must configure all the nodes with static IP addresses rather than with DHCP The automatic discovery process requires that all nodes be configured with DHCP or with static IP addresses but not a combination of both methods Ensure that any nodes connected to a Lustre file system server are on their own Gigabit Ethernet switch Ensure that all hardware components are running the correct firmware version and that all nodes in the system are at the same firmware version See Firmware Requirements and Dependencies page 59 for more information Nagios is a component of the HP XC system that monitors sensor data and system event logs Ensure that the console port of the head node is connected to the external network so 4 2 Ethernet Port Connection
188. roLiant DL360 G4p iLO system BIOS HP ProLiant DL360 G5 iLO2 system BIOS HP ProLiant DL380 G4 iLO system BIOS HP ProLiant DL380 G5 iLO2 system BIOS HP ProLiant DL580 G4 iLO2 system BIOS HP ProLiant DL580 G5 iLO2 system BIOS CP3000BL HP ProLiant BL2x220c G5 iLO2 system BIOS Onboard Administrator OA HP ProLiant BL260c G5 iLO2 system BIOS Onboard Administrator OA 1 5 Supported Console Management Devices 29 Table 1 3 Supported Console Management Devices continued Hardware Component Firmware Dependency HP ProLiant BL460c iLO2 system BIOS Onboard Administrator OA HP ProLiant BL480c iLO2 system BIOS OA HP ProLiant BL680c G5 iLO2 system BIOS Onboard Administrator OA CP4000 HP ProLiant DL145 LO 100i HP ProLiant DL145 G2 LO 100i HP ProLiant DL145 G3 LO 100i HP ProLiant DL165 G5 LO 100i HP ProLiant DL365 iLO2 HP ProLiant DL365 G5 iLO2 HP ProLiant DL385 iLO2 HP ProLiant DL385 G2 iLO2 HP ProLiant DL385 G5 iLO2 HP ProLiant DL585 iLO2 HP ProLiant DL585 G2 iLO2 HP ProLiant DL585 G5 iLO2 HP ProLiant DL785 G5 iLO2 CP4000BL HP ProLiant BL465c iLO2 HP ProLiant BL465c G5 iLO2 HP ProLiant BL685c iLO2 HP ProLiant BL685c G5 iLO2 CP6000 HP Integrity rx1620 Management Processor MP HP Integrity rx2600 MP HP Integrity rx2620 MP HP Integrity rx2660 MP HP Integrity rx4640 MP HP Integrity rx8620 MP CP6000BL HP Integrity BL860c Server Blade Full height MP 1 6 Administration Netw
189. roLiant DL385 G2 nodes Table 4 29 iLO Settings for HP ProLiant DL385 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable on Table 4 30 lists the iLO settings for ProLiant DL385 G2 nodes 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 101 Table 4 30 iLO Settings for HP ProLiant DL385 G2 Nodes Menu Name Submenu Name Option Name User Add Set to This Value Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Settings CLI Serial CLI Speed bits seconds 115200 Press the F10 key to save the setting
190. s HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable on Settings CLI Serial CLI 115200 Press the F10 key to save the Speed setting bits seconds 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL580 G4 node in the HP XC system 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 77 1 Make the following settings from the Main menu The BIOS settings for HP ProLiant DL580 G4 nodes are listed in Table 4 14 Table 4 14 BIOS Settings for HP ProLiant DL580 G4 Nodes Menu Name Option Name Set to This Value Standard Boot Order IPL Set the following boot order on all nodes except the head node CD ROM does not have to be first in the list but it must be listed before the hard disk 1 CD ROM 2 NIC1 3 Hard Disk On the head node set the boot order so that the CD ROM is listed before the hard disk Advanced Options Processor Hyper_threading Disable System Options Embedded Serial Port COM2 Virtual Serial Port COM1 Press the Esc key to return to the main menu BIOS Serial Console amp BIOS Serial Console Port COM1 EMS BIOS Serial Console Baud Rate 115
191. s on the Head Node 61 that it is accessible to Nagios during system operation For more information on Nagios see the HP XC System Software Administration Guide 6 Review the documentation that came with the hardware and have it available if needed 7 If your hardware configuration contains server blades and enclosures proceed to Setting the Onboard Administrator Password page 62 Depending upon the type of cluster platform proceed to one of the following sections to prepare individual nodes e Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems page 63 e Preparing the Hardware for CP4000 AMD Opteron Systems page 87 e Preparing the Hardware for CP6000 Intel Itanium Systems page 122 4 4 Setting the Onboard Administrator Password If the hardware configuration contains server blades and enclosures you must define and set the user name and password for the Onboard Administrator on every enclosure in the hardware configuration we IMPORTANT You cannot set the Onboard Administrator password until the head node is installed and the switches are discovered For more information on installing the head node and discovering switches see the HP XC System Software Installation Guide The Onboard Administrator user name and password must match the user name and password you plan to use for the iLO2 console management devices The default user name is Administrator and HP recommends that
192. s them to a node with a login role See MP See LSF master host An optional integrated system that uses chilled water technology to triple the standard cooling capacity of a single rack This system helps take the heat out of high density deployments of servers and blades enabling greater densities in data centers See MCS A package that provides for the dynamic modification of a user s environment by means of modulefiles See also modulefile Contains information that alters or sets shell environment variables such as PATH and MANPATH Modulefiles enable various functions to start and operate properly Management Processor Controls the system console reset and power management functions on HP Integrity servers Message Passing Interface A library specification for message passing proposed as a standard by a broadly based committee of vendors implementors and users A relational database system developed by MySQL AB that is used in HP XC systems to store and track system configuration information Network Address Translation A mechanism that provides a mapping or transformation of addresses from one network to another This enables external access of a machine on one LAN that has the same IP address as a machine on another LAN by mapping the LAN address of the two machines to different external IP addresses See NAT See NIS Network Information Services A mechanism that enables centralization of common data th
193. seconds 3 Restore power to the node Restoring power restarts the iLO2 device and also reboots the node e Use the following procedure to restart the iLO2 device without rebooting the node EY NOTE This method can be used only if the iLO2 command line interface is not hung 1 Use the telnet or ssh command to access the hung iLO2 device Log in to the iLO2 device Use the HP XC user name and password that you defined for your console devices 3 Reboot the iLO2 device reset mapl 5 1 iLO2 Devices 139 140 A Establishing a Connection Through a Serial Port Follow this generic procedure to establish a connection to a server using a serial port connection to a console port If you need more information about how to establish these connections see the hardware documentation 1 2 3 Connect a null modem cable between the serial port on the rear panel of the server and a COM port on the host computer Launch a terminal emulation program such as Windows HyperTerminal Enter a name for the connection select an icon and click OK Select the COM port on the host computer to which the serial cable is connected and click OK Make the following port settings a Bits per second 115200 b Data bits 8 c Parity None d Stop bits 1 e Flow control None Click OK 141 142 B Server Blade Configuration Examples This appendix contains illustrations and descriptions of fully cabled HP XC systems
194. seissesesrestssenensessessssesrenessesrensenees 79 BIOS Settings for HP ProLiant DL580 G5 NOGOS vasscestcici svc snicancsssasinedeicesdivedenwanuoataritinesiniialseaniunen 80 BIOS S ttings for xw8200 WorkstationS isissisrsiisiias issis taas deaan iesaisti han aeieea at 81 BIOS Settings for xw8400 Workstations ssssessessesessessssesrtssesttsssststessssesensennesensentesensesrene nets 82 BIOS Settings for xw8600 WOrkStan Ons ossici cin ctinctebisancdesds msseadenatanstiariendanmirsiamienameiedtnninsian 83 Boot Order for HP ProLiant Server Bl d S s seiret entsteet seseer erisir dnei iaai madipaieidar 84 BIOS Settings for HP ProLiant DL145 NOG G8 iccysuswrssunctvacstnasceaanierntoumsnevisindieacteeieaayiernssnenenates 88 BIOS Settings for HP ProLiant DL145 G2 Nodes ivsicsacicdinwanacnneeana abn aie 90 BIOS Settings for HP ProLiant DL145 G3 Nodes s sessessseissisessserssisnsssessresreseenseenrenressresresees 92 BIOS Settings for HP ProLiant DL165 G5 NOdeS icici eG drennetin Ceeeed 94 iLO Settings for HP ProLiant DL365 MOGs inecctatorstiniinracheniiveidasanpeiea Weutaninnakouieistaginiseuaintaden 95 RBSU Settings for HP ProLiant DL365 NOdG6 tiacictesimitn stad caglesiiai easier iatadhd 96 iLO Settings for HP ProLiant DL365 G5 NOP Ss sinter icercesnrconauesinitenin envcarsaivimionisieundsteinniviamdscsn 98 RBSU Settings for HP ProLiant DL365 G5 NWOdeSiicinsuw sistance autre 99 iLO Settings for HP ProLiant DL385 Nodes s ssssesssess
195. served 3 Port 50 is the Gigabit Ethernet link to the Root Administration Switch Allocate the ports on this switch for consistency with the administration switches as follows e Ports 1 10 13 22 25 34 37 41 Starting with port 1 the ports are used for links from Branch Console Switches Trunking is not used Starting with port 41 and in descending order ports are assigned for use by individual nodes in the utility cabinet Nodes in the utility cabinet are connected directly to the Root Administration Switch A NOTE There must be at least one idle port in this set to indicate the dividing line between branch links and root node administration ports e Ports 11 12 23 24 35 36 and 43 48 are unused 3 3 6 3 ProCurve 2626 Switch You can use a ProCurve 2626 switch as a Root Console Switch for the console branch of the administration network The ProCurve 2626 switch is shown in Figure 3 9 In the figure white ports should not have connections black ports can have connections and ports with numbered callouts are used for specific purposes described after the figure Figure 3 9 ProCurve 2626 Root Console Switch Uplinks from Branches Connections to Node Console Ports Start at port 1 Ascending Start at port 21 Descending HAHAHAHA BWP eee ewaeeew Ahhh BAN BON TEPEE FPEFEM IC Lie wee OE Gigabit Ethernet Ports 10 100Base TX RJ 45 Ports T
196. ses HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Settings CLI Serial CLI Speed bits seconds 115200 Press the F10 key to save the setting 4 Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test 5 Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure for each HP ProLiant DL785 G5 node in the hardware configuration 114 Preparing Individual Nodes 1 Make the RBSU settings for each node Table 4 41 lists the RBSU setting for the DL785 G5 nodes Table 4 41 RBSU Settings for HP ProLiant DL785 G5 Nodes Menu Name Option Name Set to This Value System Options Embedded Serial Port Disabled Virtual Serial Port COM1 Embedded NICs NIC 1 boot On all nodes except the head node set this value Options to PXE boot Embedded NICs NIC 2 boot Disabled head node only Options Power Regulator for OS Control Mode ProLiant Standard Boot Order IPL Set the following boot order on all nodes except the head node the CD ROM must be listed before the hard drive e IPL 1 CD ROM e IPL 2 Floppy Drive A e IPL 3 USB Drive Key C e IPL 4PCI Embedded HP NC373i Multifunction Gigabit Adapter e IPL 5 Hard Drive C Set the following boot order on the hea
197. sissssseersersersresstesnrsesssernsesnreneensrenreseense 101 iLO Settings for HP ProLiant DL385 G2 Nodes esssssssssesssessssrssssressressrsersserssesnssneessresreseense 102 RBSU Settings for HP ProLiant DL385 Nodesiiwicncwainnaidaomniennceniaeeanire 103 RBSU Settings for HP ProLiant DL385 G2 Nodes siiisssstietenssanuies ctenictcsansichianeciensislerinnenndssnes 103 iLO Settings for HP ProLiant DL385 G2 Nodes ssssssssssssssssssrssssressressrsesssernsennsenressresreneense 105 iLO Settings for HP ProLiant DL585 NOS ie cusssssiuesvansotnssetasrictensanicimadintiepiainatenssvesemensenceenssnets 107 iLO Settings for HP ProLiant DL585 G2 Nodes iiciceaiaiieat amen 107 RBSU Settings for HP ProLiant DL585 NOG eS es cui ncasnencvnnevensininameruancinaninivedascnnenivenivimedanenubes 109 RBSU Settings for HP ProLiant DL585 G2 Nodes s ssnssssesseseesesressssenssssesresessesreserenssnseseeses 109 iLO Settings for HP ProLiant DL585 G5 Nodes csc casstine cckarsanshtailes aelsatnunivatibiauesdtensniecdn tine yebendleancets 111 RBSU Settings for HP ProLiant DL585 G5 Nod CSc iientnimainntinisenriameiuain ances 112 iLO Settings for HP ProLiant DL785 G5 Nodes sis ccisanvsinersacnawaciavsabecuveinanne vialesnnwesivatonieberemunttoes 114 RBSU Settings for HP ProLiant DL785 G5 Nod Siasiwaniaciiodenetaan iia diiet 115 Setup Utility Settings for xw9300 Workstations sssssssessssisissestseessesessessestesensestesessesressenees 117 Setup Utility Se
198. stances there might need to be an additional Ethernet interconect blade needed to support any required external connections This would only be needed on Gigabit Ethernet clusters with half height blades that need external connections For more information see Configuring the External Network Option 2 page 42 The various enclosures that make up a cluster are connected to each other through external ProCurve switches Every cluster needs at least one ProCurve Administrative Network switch a 2800 series and may optionally have a Console Network Switch a 2600 series It is possible to have the Console and Administrative Network combined over the single 2800 series switch on smaller configurations Gigabit Ethernet clusters require one or more external Ethernet switches to act as the cluster interconnect between the enclosures This can be set up one of two ways e Ifthe cluster uses Ethernet switches in each enclosure then you need a smaller external interconnect because you only need one connection for each enclosure in the cluster although this might be a trunked connection e Ifthe cluster uses Ethernet pass through modules in each enclosure you need a large external Ethernet switch with enough connections for each node in the cluster InfiniBand clusters require one or more external IB switches with at least one managed switch to manage the fabric 2 2 Network Overview An HP XC system consists of several networks administ
199. sures per region There is a Gigabit Ethernet switch for the HP XC system that is connected to a Gigabit Ethernet switch in each region The Gigabit Ethernet switch in a region is connected to each enclosure s Gigabit Ethernet switch in bay 1 and to a ProCurve 2650 switch The ProCurve 2650 switch is connected to the Primary Onboard Administrator External Link of each enclosure ProCurve 2650 Enclosure 32 OA Ext Link 36 Cabling Server Blades Figure 2 3 Interconnection Diagram for a Large HP XC Cluster of Server Blades GigE Switch f Region 1 l Region 8 GigE Switch GigE Switch Enclosure 512 GigE ProCurve 2650 Switch 1 Enclosure 481 Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 32 GigE Switch 1 ProCurve 2650 Enclosure 32 Enclosure 1 Secondary OA Ext Link Primary OA Ext Link cael Oooo Enclosure 2 Secondary OA Ext Link Primary OA Ext Link B Oooo Enclosure 3 Secondary OA Ext Link Primary OA Ext Link jiii OOo Enclosure 4 Secondary OA Ext Link Primary OA Ext Link iaki Oooo 3 Enclosure 512 OA Ext Link Ext Link Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 483 Secondary OA Ext Link Primary OA Ext Link GigE Switch Enclosure 484 Secondary OA Ext Link Primary OA Ext Link GigE Switch l l l l l l l l l l l Enclosure 482 l l l l l l l l l l l l
200. t Connections Non Blade Server PCI SLOT Non Blade Server PCI SLOT MGT NIC NIC PCI SLOT MGT NIC NIC PCI SLOT iniBand PCI Cards C Class Blade Enclosure Admin ProCurve 2800 Series Switch NIC 1 Interconnect bay 1 5 NOZ InfiniBand Interconnect bay 2 NIC 3 Mezzanine NIC 4 Cards Interconnect bay 3 Console ProCurve 2600 Series Switch MEZZ 1 MEZZ 2 Interconnect bay 4 MEZZ 3 Interconnect bays iLO2 5 amp 6 double wide Infiniband Interconnect Switch NIC 1 Interconnect bay 7 NIC 2 Interconnect bay 8 Double wide MEZAT InfiniBand switch MEZZ 2 module External Public Network ONBOARD iLO2 ADMINISTRATOR 40 Cabling Server Blades 2 5 3 Configuring the Interconnect Network Over the Administration Network In cases where an additional Gigabit Ethernet port or switch may not be available the HP XC System Software enables you to configure the interconnect on the administration network When the interconnect is configured on the administration network only a single LAN is used To configure the interconnect on the administration network include the ic AdminNet option on the discover command line which is documented in the HP XC System Software Installation Guide Be aware that configuring the interconnect on the administration network may negatively impact system performance
201. t and Rear Views Front View Rear View The HP BladeSystem c3000 Enclosure is available as a tower model as shown in Figure 1 6 Figure 1 6 HP BladeSystem 3000 Enclosure Tower Model Figure 1 7 is an illustration showing the location of the device bays optional DVD drive Insight display and Onboard Administrator at the front of the HP BladeSystem c3000 Enclosure 1 2 Server Blade Enclosure Components 25 26 Figure 1 7 HP BladeSystem c3000 Enclosure Bay Locations Front View f ee ac PIT He tre j cos J finm T YY ios i aC ace Device bays DVD drive optional Insight Display For more information see Insight Display page 28 Onboard Administrator OA The HP BladeSystem c3000 enclosure can house a maximum of 8 half height or 4 full height server blades Additionally the c3000 enclosure contains an integrated DVD drive which is useful for installing the HP XC System Software Figure 1 8 illustrates the numbering of the server bays of the HP BladeSystem c3000 enclosure for both half height and full height server blades BE Figure 1 8 HP BladeSystem c3000 Enclosure Bay Numbering Half Height Full Height The number of fans in the enclosure influences the placement of the serv
202. t array configuration instructions are outside the scope of this document See the documentation that came with your model of HP ProLiant server for more information Use the virtual power functions to turn off power to the server blade 4 6 Preparing the Hardware for CP3000BL Systems 85 10 Close the iLO2 utility Web page 11 Repeat this procedure from every active Onboard Administrator and make the same settings for each server blade in each enclosure After preparing all the nodes in all the enclosures return to the HP XC System Software Installation Guide to discover all the nodes and enclosures in the HP XC system 86 Preparing Individual Nodes 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems Follow the procedures in this section to prepare each node before installing and configuring the HP XC System Software See the following sections depending on the hardware model e Preparing HP ProLiant DL145 Nodes page 87 e Preparing HP ProLiant DL145 G2 and DL145 G3 Nodes page 89 e Preparing HP ProLiant DL165 G5 Nodes page 93 e Preparing HP ProLiant DL365 Nodes page 94 e Preparing HP ProLiant DL365 G5 Nodes page 97 e Preparing HP ProLiant DL385 and DL385 G2 Nodes page 100 e Preparing HP ProLiant DL385 G5 Nodes page 104 e Preparing HP ProLiant DL585 and DL585 G2 Nodes page 106 e Preparing HP ProLiant DL585 G5 Nodes page 110 e Preparing HP ProLiant D
203. t labeled LAN Gb Bis used for an external connection Figure 4 27 shows a rear view of the HP Integrity rx2600 server and the appropriate port assignments for an HP XC system The high speed interconnect card such as an InfiniBand or QsNet card must be inserted into the top PCI X slot The external connection is made on the Ethernet adapter card Figure 4 27 HP Integrity rx2600 Server Rear View PCI X 133 PCI X 133 CONSOLE PCI X 133 PCI X 133 The callouts in the figure enumerate the following 1 The top port labeled LAN 10 100 is the MP connection to the ProCurve Console Switch 2 The port labeled LAN Gb connects to the Administration Switch branch or root 3 The bottom port labeled LAN 10 100 is unused Setup Procedure Perform the following procedure on each HP Integrity server model rx1620 and rx2600 1 Use the instructions in the accompanying hardware documentation to connect a monitor mouse and keyboard to the node 2 For each node in the system ensure that the power cord is connected but that the is not turned on 3 Follow this procedure to connect a personal computer to the Management Processor a Connect a three way DB9 25 cable to the MP DB 25 port on the back of the HP Integrity rx2600 server b Connect the CONSOLE connector to a null modem cable and connect the null modem cable to the PC COM1 port Use a term
204. te the ports on this switch for maximum performance as follows e Ports 1 10 13 22 25 34 37 44 are used for the console ports of individual nodes up to 38 nodes e Ports 11 12 23 24 35 36 45 49 are unused 3 4 Interconnect Connections 56 A The high speed interconnect connects every node in the HP XC system Each node can have an interconnect card installed in the highest speed PCI slot Check the hardware documentation to determine which slot this is The interconnect switch console port or monitoring line card also connects to the Root Administration Switch either directly or indirectly as described in Root Administration Switch page 50 You must determine the absolute maximum number of nodes that could possibly be used with the interconnect hardware that you have This maximum number of ports on the interconnect switch or switches max node affects the naming of the nodes in the system The documentation that came with the interconnect hardware can help you find this number NOTE You can choose a number smaller than the absolute maximum number of interconnect ports for max node but you can not expand the system to a size larger than this number in the future without completely rediscovering the system thereby renumbering all nodes in the system This restriction does not apply to hardware configurations that contain HP server blades and enclosures Specific considerations for connections to the int
205. teeibeuibe tieetaetiomtunnMmenaleeds 78 4 5 8 Preparing HP xw8200 and xw8400 Work StaHOns weiss csecssspasnessinewiesioraseecuencianviensaersvnsanennmeases 80 4 5 9 Preparing HP xw8600 Workstations ssssssssesesssseesesssrsestessesestessssennessestenessesenssnesssnsere enes 82 4 6 Preparing the Hardware for CP3000BL Systems ssss ssnssssessessssesssesessiserseseestssennensentesenseseenesne 84 4 7 Preparing the Hardware for CP4000 AMD Opteron SystemsS cccccccceeseeeeceneeeeeneeeeeeeeeeees 87 4 7 1 Preparing HP ProLiant DIG 4S NodeS sn iiciin rana a diei iaaea 87 4 7 2 Preparing HP ProLiant DL145 G2 and DL145 G3 Nodes s ssessessseseisessseissrsessersseenesreesse 89 47 3 Preparmg HPE ProLiant DL165 G5 Nodes susisssrirsiisriarriseniei oiera sieniin a siaaa 93 4 7 4 Preparing HP ProLiant DL365 NOdeS sicauvsitencini tiinas iini a e aaan ai a aaa 94 4 7 5 Preparmg HP ProLiant DL365 G5 MOUCS icccsciventsnsncdsnagu iste reisini iseen piae 97 4 7 6 Preparing HP ProLiant DL385 and DL385 G2 Nodes ssssssesessseissssessersseersserssreneeseense 100 4 7 7 Preparing HP ProLiant DL385 G5 Nodes s sessessssissisessserssesesssrissressesernsrnnesseesseenreseenss 104 4 7 8 Preparing HP ProLiant DL585 and DL585 G2 Nodes ssssssssessssissrsessserssisnssnissresresernse 106 4 7 9 Preparing HP ProLiant DL585 G5 Nodes s sessessseessisessserssisnsssrissrssseserssinnesreenseeseesernss 110 4 7 10 Preparing
206. test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility Make the following iLO settings for ProLiant DL365 nodes as shown in Table 4 25 Table 4 25 iLO Settings for HP ProLiant DL365 Nodes Menu Name Submenu Name Option Name Set to This Value User Add Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable On Select File Exit to exit the Integrated Lights Out Setup Utility and resume the power on self test Watch the screen carefully and press the F9 key when prompted to access the ROM Based Setup Utility RBSU Perform the following procedure from the RBSU for each HP ProLiant DL365 node in the hardware configuration 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 95 96 1 Make the RBSU settings for the HP ProLiant DL365 nodes as indicated in Table 4 26 Use the navigation aids shown at the bottom of the screen to move through the menus and make selections Table 4 26 RBSU Settings for HP ProLiant DL365 Nodes Menu Name Option Name Set to This Value System Opt
207. that you cannot make until the iLO has an IP address The HP XC System Software Installation Guide provides instructions for using a browser to connect to the iLO to enable telnet access Figure 4 5 shows a rear view of the HP ProLiant DL380 G4 server and the appropriate port assignments for an HP XC system Figure 4 5 HP ProLiant DL380 G4 Server Rear View o Eec l tl f EIF R 23000002002000 5 RARA 50505080 Ho EATA IROLT I E o alaa alaaa aa ra a nmi Oo The callouts in the figure enumerate the following 1 The iLO Ethernet port is used for the connection to the Console Switch 2 NIC2 is used for the connection to the external network 3 NIC1 is used for the connection to the Administration Switch branch or root Figure 4 6 shows a rear view of the HP ProLiant DL380 G5 server and the appropriate port assignments for an HP XC system Preparing Individual Nodes Figure 4 6 HP ProLiant DL380 G5 Server Rear View O ce E anne OO The callouts in
208. the figure enumerate the following 1 2 3 This port is used for the connection to the external network This port is used for the connection to the Administration Switch branch or root The iLO Ethernet port is used for the connection to the Console Switch Setup Procedure Perform the following procedure from the iLO Setup Utility for each HP ProLiant DL380 node in the HP XC system 1 Use the instructions in the accompanying HP ProLiant hardware documentation to connect a monitor mouse and keyboard to the node 2 Turn on power to the node Watch the screen carefully during the power on self test and press the F8 key when prompted to access the Integrated Lights Out Setup Utility 3 Make the iLO settings listed in Table 4 10 for each node in the hardware configuration Table 4 10 iLO Settings for HP ProLiant DL380 G4 and G5 Nodes Submenu Name Menu Name Name Option Name Set to This Value Administration User New Create a common iLO user name and password for every node in the hardware configuration The password must have a minimum of 8 characters by default but this value is configurable The user Administrator is predefined by default but you must create your own user name and password For security purposes HP recommends that you delete the Administrator user You must use this user name and password to access the console port Network DNS DHCP DHCP Enable on Settings CLI Serial CLI 115200 Press the F10 key to s
209. the highest port number on the Root Administration Switch Other higher port numbers are used to connect to other switches If the Root Administration Switch is a ProCurve 2848 switch the head node is connected to port number 42 as discussed in Root Administration Switch page 50 If the Root Administration Switch is a ProCurve 2824 switch the head node is connected to port number 22 on that switch as discussed in Root Administration Switch page 50 The head node should however be connected to the highest port number on the interconnect switch The next node connected directly to the root switches Administration and Console should have connections to the Quadrics switch at the next highest port number on the Quadrics switch max node minus 2 All nodes connected to the Root Administration Switch will be connected to the next port in descending order Nodes attached to branch switches must be connected starting at the opposite end of the Quadrics switch The node attached to the first port of the first Branch Administration Switch should be attached to the first port on the Quadrics switch Port 0 3 4 2 Gigabit Ethernet Interconnect Connections The HP XC System Software is not concerned with the topology of the Gigabit Ethernet interconnect but it makes sense to structure it in parallel with the administration network in order to make your connections easy to maintain Because the first logical Gigabit Ethernet port o
210. tion shows a Gigabit Ethernet switch a ProCurve 2848 that is connected to the enclosure switch in bay 1 of each enclosure as well as to a ProCurve 2650 switch that connects to the Onboard Administrator external link of each enclosure The bottom portion provides some additional detail It shows the ProCurve managed Gigabit Ethernet switch connected to the Gigabit Ethernet switch of each enclosure and to a ProCurve 2650 switch which connects the Primary Onboard Administrator External Link of each enclosure 2 2 Network Overview 35 Figure 2 2 Interconnection Diagram for a Medium Sized HP XC Cluster of Server Blades GigE Switch Enclosure 2 Enclosure 32 Switch 1 Switch 1 ProCurve 2650 Enclosure 1 Switch 1 Enclosure 1 Enclosure 2 Enclosure 3 Enclosure 1 Enclosure 32 Server Blades Server Blades Server Blades OA OA NIC NICI NICI Ext Link Enclosure 2 Ext Link OA May be any Ext Link ProCurve managed GigE switch smaller systems may use ProCurve 2824 Smaller systems may use ProCurve 2626 ProCurve 2848 Enclosure 32 GigE switch Enclosure 4 Secondary OA Ext Link Primary OA Ext Link Enclosure 3 Secondary OA Ext Link Primary OA Ext Link Enclosure 2 Secondary OA Ext Link Primary OA Ext Link Enclosure 1 Secondary OA Ext Link Primary OA Ext Link Large HP XC Cluster of Server Blades Figure 2 3 page 37 illustrates a large HP XC cluster of eight regions with 32 enclo
211. tration network in interconnect bay 1 e Systems that use InfiniBand as the interconnect require a double wide InfiniBand interconnect switch module installed in interconnect bays 3 and 4 see callouts 6 and 7 in Figure 1 9 e Some systems might need an additional Ethernet interconnect module to support server blades that require external connections For more information about external connections see Cabling for the External Network page 41 1 2 3 HP BladeSystem Class Onboard Administrator The Onboard Administrator is the management device for an enclosure and at least one Onboard Administrator is installed in every enclosure 1 2 Server Blade Enclosure Components 27 You can access the Onboard Administrator through a graphical Web based user interface a command line interface or the simple object access protocol SOAP to configure and monitor the enclosure You can add a second Onboard Administrator to provide redundancy The Onboard Administrator requires a password For information on setting the Onboard Administrator Password see Setting the Onboard Administrator Password page 62 1 2 4 Insight Display The Insight Display is a small LCD panel on the front of an enclosure that provides instant access to important information about the enclosure such as the IP address and color coded status Figure 1 10 depicts the Insight Display Figure 1 10 Server Blade Insight Display Power Mode AC Redundant Po
212. ttings for xw9400 Werksta Ons sss seesnsonsessciaseatsioncinnsiyrasuesvaenanessniseretneniuacidlasints 117 Boot Order for HP ProLiant Server Glades icucaneaiicnaan eta ahem eningn 119 Additional BIOS Setting for HP ProLiant BL685c NOES isissirecuriicnstviazunstnbinnestanniinstbeiavictendsanes 121 Setting Static IP Addresses for MP Power Management DeViCES c cccccesssseeseesseeseeneeeseenes 122 List of Tables 4 47 Adding a Boot Entry and Setting the Boot Order on HP Integrity Server Blades 10 About This Document This document describes how to prepare the nodes in your HP cluster platform before installing HP XC System Software An HP XC system is integrated with several open source software components Some open source software components are being used for underlying technology and their deployment is transparent Some open source software components require user level documentation specific to HP XC systems and that kind of information is included in this document when required HP relies on the documentation provided by the open source developers to supply the information you need to use their product For links to open source software documentation for products that are integrated with the HP XC system see Supplementary Software Products page 14 Documentation for third party hardware and software components that are supported on the HP XC system is supplied by the third party vendor However information about t
213. uration Ethernet On Board for Ethernet 1 and 2 Option Name Disable Jitter bit page Directory Cache Device Option ROM Scan Latency timer Set to This Value Enabled Disabled Enabled Enabled 40h I O Device Configuration Serial Port BMC COM Port SIO COM Port Disabled PS 2 Mouse Enabled Console Redirection Console Redirection Enabled EMS Console Enabled Baud Rate 115 2K Flow Control None Redirection after BIOS On POST IPMI LAN Setting IP Address Assignment DHCP BMC Telnet Service Enabled BMC Ping Response Enabled BMC HTTP Service Enabled IPMI BIOS POST Watchdog Disabled Boot Set the following boot order on all nodes except the head node 1 CD ROM 2 Removable Devices 3 PXE MBA V7 7 2 Slot 0300 4 Hard Drive 5 PXE MBA V7 7 2 Slot 0200 means disabled Set the following boot order on the head node 1 CD ROM 2 Removable Devices 3 Hard Drive 4 PXE MBA V7 7 2 Slot 0200 5 PXE MBA V7 7 2 Slot 0300 Power Wake On Modem Ring Disabled Wake On LAN Disabled Table 4 23 provides the BIOS settings for ProLiant DL145 G3 nodes 4 7 Preparing the Hardware for CP4000 AMD Opteron Systems 91 Table 4 23 BIOS Settings for HP ProLiant DL145 G3 Nodes Menu Name Submenu Name Option Name Set to This Value Main Boot Options NumLock off Advanced l O Device Configuration Serial Port Mode BMC Serial port A Enabled Base I O address 3F8 Interrupt IRQ 4 Memory Controller Options DRAM Bank Interleave AUTO
214. using Internet Explorer as your browser do the following a Click the Integrated Remote Console link to open a remote console window which provides access to the graphics console virtual media and power functions b Inthe remote console window click the Power button c Click the Momentary Press button d Wait a few seconds for the power up phase to begin Click the MB1 mouse button in the remote console window to put the pointer focus in this window so that your keyboard strokes are recognized e Proceed to Step 8 Mozilla Firefox If you are using Mozilla Firefox as your browser do the following a Click the Remote Console link to open a virtual console window b Go back to the iLO2 utility Web page and click the Virtual Devices tab c Click the Momentary Press button d Go back to the remote console window Wait a few seconds for the power up phase to begin Click the MB1 mouse button in this window to put the pointer focus in the remote console window so that your keyboard strokes are recognized in this window e Proceed to Step 8 Watch the screen carefully during the power on self test phase and press the F8 key when you are prompted to configure the disks into the smart array Select View Logical Drives to determine if a logical drives exists If a logical drive is not present create one If you create a logical drive exit the SmartArray utility and power off the node Do not let it try to boot up Specific smar
215. verify the IPMI over LAN option is enabled Enable this option if it is disabled This setting is required for Nagios monitoring Return to the Command Menu Enter XD to apply your changes Enter R to restart the MP Enter CM to return to the Command Menu Enter SO to set the MP user name and password The user name must have a minimum of 6 characters and the password must have a minimum of 8 characters You must set the same user name and password on every node IMPORTANT The remaining steps in this procedure setting the network boot option the boot order and console options must be performed twice once for each partition on the HP Integrity rx8620 system On HP Integrity rx8620 systems some MP commands such as CO and PE prompt you to supply the partition on which to perform the action Enter PE power enable to turn on power to the cabinet if it is not already turned on Enter PE power enable to turn on power to the partition Press Ctrl B to return to the Main Menu then enter CO to connect to the console of the partition 4 9 Preparing the Hardware for CP6000 Intel Itanium Systems 133 E NOTE If the console stops accepting input from the keyboard the following message is displayed Read only use Ecf to attach to console In that situation press and hold down the Ctrl key and type the letter e Release the Ctrl key and then type the letters c and f to reconnect to the console 12 Do the fo
216. vsseoteasedersanvecessedssencceuretegied iiin i eise 139 A Establishing a Connection Through a Serial Port ccssesssseceeseseeseseseseeseeseneeee 141 B Server Blade Configuration Exciiplesc ssi ssasvcesnrssuacisiaceeateenaiebeoravasiuasciisnins 143 B 1 Gigabit Ethernet Interconnect With Half Height Server Blades ccccccccscesssesseesesseesesseeeseees 143 B 2 InfiniBand Interconnect With Full Height Server Blade sys ictccccccs cicscasiscetidtscccesesseteesessmessceswacs 143 B 3 InfiniBand Interconnect With Mixed Height Server Blade ccccccccsessseseeesteneenseeseeeeeeeeeeees 144 Ee cate a EEEE ne Ran ea ner en ean ENTE etry are vere AA rae mn re ert 147 UNC cerned N AEN AAE E AN A A E EE eeaciccs 153 Table of Contents 5 List of Figures 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 10 1 11 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 3 10 3 11 3 12 3 13 3 14 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 4 9 4 10 4 11 4 12 4 13 4 14 4 15 4 16 6 HP BladeSystem c7000 enclosure Front and Rear VieWS cccccsscessceeseesssesseeceseeeseeeeeeseees 22 HP BladeSystem c7000 Enclosure Bay Locations Front VieW c ccccscesessesesseesseesseeesenseeaeenes 22 HP BladeSystem c7000 Enclosure Bay Numbering for Half Height and Full Height Server Padesan a E E EE EE E vaveecbassdsQdansdeevuphabesedeateoeeep nevedees tiers 23 HP BladeSystem c7000 Enclosure Bay Loc
217. wer Limit Norset Dynamic Power Enabled OAI1 IP Address 192 0 2 121 OA2 IP Address Not Present Encl Name Enclosure01 7 Rack Name Racko1j DVD Drive a A Insight Display PIN Nopsetl Accept All BC STu L You can use the Insight Display panel to make some basic enclosure settings 1 3 Server Blade Mezzanine Cards The mezzanine slots on each server blade provide additional I O capability Mezzanine cards are PCI Express cards that attach inside the server blade through a special connector and have no physical I O ports on them Card types include Ethernet fibre channel or 10 Gigabit Ethernet 1 4 Server Blade Interconnect Modules 28 An interconnect module provides the physical I O for the built in NICs or the supplemental mezzanine cards on the server blades An interconnect module can be either a switch or a pass thru module A switch provides local switching and minimizes cabling Switch models that are supported as interconnect modules include but are not limited to e Nortel GbE2c Gigabit Ethernet switch e Cisco Catalyst Gigabit Ethernet switch e HP 4x DDR InfiniBand switch e Brocade SAN switch A pass thru module provides direct connections to the individual ports on each node and does not provide any local switching Bays in the back of each enclosure correspond to specific interfaces on the server blades Thus all I O devices that correspond to a specific interconnect bay must be the same type
218. x configuring hardware and a variety of other topics http www enu or Home page for the GNU Project This site provides online software and information for many programs and utilities that are commonly used on GNU Linux systems Online information include guides for using the bash shell emacs make cc gdb and more MPI Web Addresses http www mpi forum or Contains the official MPI standards documents errata and archives of the MPI Forum The MPI Forum is an open group with representatives from many organizations that define and maintain the MPI standard http www unix mcs anl gov mpi A comprehensive site containing general information such as the specification and FAQs and pointers to other resources including tutorials implementations and other MPI related sites Compiler Web Addresses http www intel com software products compilers index htm Web address for Intel compilers http support intel com support performancetools Web address for general Intel software development information http www peroup com Home page for The Portland Group supplier of the PGI compiler Debugger Web Address http www etnus com Home page for Etnus Inc maker of the TotalView parallel debugger Software RAID Web Addresses http www tldp org HOWTO Software RAID HOWTO html and http www ibiblio org pub Linux docs HOWTO other formats pdf Software RAID HOWTO pdf A document in two formats HTML a
219. y 1 Interconnect bay 2 Interconnect bay 3 Interconnect bay 4 Interconnect bays 5 amp 6 double wide Interconnect bay 7 Interconnect bay 8 ONBOARD ADMINISTRATOR KEY Administration Network Console Network Cluster Interconnect Network External Network B 3 InfiniBand Interconnect With Mixed Height Server Blades MGT NIC NIC PCI SLOT Admin ProCurve 2800 Series Switch Console ProCurve 2600 Series Switch InfiniBand Interconnect Switch External Public Network 145 146 Glossary A administration branch administration network availability set availability tool B base image branch switch Cc cluster cluster alias CMDB compute node configuration and management database console branch D DHCP Dynamic Host Control Protocol EFI The half branch of the administration network that contains all of the general purpose administration ports to the nodes of the HP XC system The private network within the HP XC system that is used for administrative operations An association of two individual nodes so that one node acts as the first server and the other node acts as the second server of a service See also improved availability availability tool A software product that enables system services to continue running if a hardware or software failure occurs by failing over the service to the other node in an availability set
220. yricom Inc Nagios is a registered trademark of Ethan Galstad The Portland Group and PGI are trademarks or registered trademarks of The Portland Group Compiler Technology STMicroelectronics Inc Quadrics and QsNet are registered trademarks of Quadrics Ltd Red Hat and RPM are registered trademarks of Red Hat Inc syslog ng is a copyright of BalaBit IT Security SystemImager is a registered trademark of Brian Finley TotalView is a registered trademark of Etnus Inc UNIX is a registered trademark of The Open Group Table of Contents About This DoCUMEent cecccsccsccecsecceccesceccescesccsccaccesccceacesccsccsscesecsecacesaccaccaeesscenceasencens 11 boende Auden esenea n a a a a E N fe2tiaaetes ss 11 New and Changed Information in This Edition sssssssessseetsesstseesesstsresssneessnsentesensesnessnnennessestees 11 Typographic Conventions meiiens apee iana ai i aanaeio ante aa ae EEE EEEa apai aee ees 11 HP XC and Related HP Products Information 0000eeseeeeeseeeersssecessssesssscossssressrecesssseesseecesssseessseee 12 Related homma Noera a a a a N meee ean deeaed 14 IY Tei te 2S xs a iaca cand usicaiinen cca am tsctinatnnhimwcuksesduimt sain dima Ea E aS NEATA E AEE ARETE AE EaR nd raea enira 17 HP Encourages Your Comments ssi ssiiisei dhise miniset ieee Sae suonadsnunencenn Ni EE SAKE EEEa S 17 1 Hardware and Network Overview ccccccccccsssecceccescessceccesssssescceseecesccaceecceneesseescaees 1
221. ystem Options Embedded Serial Port COM2 Virtual Serial Port COM1 Press the Esc key to return to the main menu BIOS Serial Console amp BIOS Serial Console Port COM1 EMS BIOS Serial Console Baud Rate 115200 EMS Console Disable BIOS Interface Mode Command Line Press the Esc key to return to the main menu 2 Press the Esc key to exit the RBSU Press the F10 key to confirm your choice and restart the boot sequence 3 Repeat this procedure for each HP ProLiant DL360 G4 node in the HP XC system 4 5 Preparing the Hardware for CP3000 Intel Xeon with EM64T Systems 69 Configuring Smart Arrays On the HP ProLiant DL360 G4 with smart array cards you must add the disks to the smart array before attempting to image the node To do so watch the screen carefully during the power on self test phase of the node and press the F8 key when prompted to configure the disks into the smart array Specific instructions are outside the scope of the HP XC documentation See the documentation that came with the HP ProLiant server for more information 4 5 4 Preparing HP Proliant DL360 G5 Nodes 70 Use the following tools to configure the appropriate settings for HP ProLiant DL360 G5 servers e Integrated Lights Out iLO Setup Utility e ROM Based Setup Utility RBSU HP ProLiant DL360 G5 servers use the iLO utility thus they need certain settings that you cannot make until the iLO has an IP address The HP XC System Software Installati

Hardware Preparation Guide

Contents

Download Pdf Manuals

Related Search

Related Contents