Home

HP HyperFabric Administrator's Guide

1. 5 Managing HyperFabric This chapter contains the following sections that give information about managing HyperFabric e Starting HyperFabric on page 109 Chapter 5 107 Managing HyperFabric e Verifying Communications within the Fabric on page 112 e Displaying Status and Statistics on page 117 e Viewing man Pages on page 127 e Stopping HyperFabric on page 128 108 Chapter 5 NOTE Chapter 5 Managing HyperFabric Starting HyperFabric Starting HyperFabric HyperFabric is started in one of these three ways e As part of the normal local node boot process HP 9000 system e By running the HyperFabric clic start command described below e By starting HyperFabric through SAM described in Using SAM on page 110 HyperFabric needs to be started in the following situations e If HyperFabric hardware and software have just been installed on the system and the clic_init command or SAM has been used to configure the HyperFabric adapters on this node e Ifthe HyperFabric configuration has been changed by using the clic_init command or using SAM In this situation you must have run clic_shutdown or used SAM to stop HyperFabric before restarting HyperFabric e Ifa new HyperFabric adapter has been added to a system online and configured using clic_init In this situation it is not necessary to run clic_shutdown before running clic_start see Online Addition and Replacement
2. What is the IP address of the first switch sw_clic0 193 0 0 20 10 What is the Ethernet hardware address of the first switch 0060b0d00257 What is the IP address of the second switch sw c1ic1 193 0 0 21 75 Configuring HyperFabric Information You Need 12 13 14 On What is the Ethernet hardware address of the second switch 0060b0d00256 What is the Multicast address for the switches to use 226 10 1 1 What is the IP address for the LAN card on the same subnet as the switches 193 0 0 10 In Figure 4 1 this is the IP address for 1an0 on node A node B 1 How many HyperFabric adapters are installed on the node Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B 11 00 11 or B 11 11 01 3 What is the IP address of the first adapter clic0 192 0 0 2 4 What is the subnet mask of the first adapter 255 255 255 0 10 11 76 If you do not specify a value for this a default mask is chosen You will most likely just accept the default However in this example we are showing a value for the subnet mask just to illustrate the correlation between the dummy information in Figure 4 1 and where that information is specified or generated during clic init and SAM What is the IP address of the second adapter c1ic1 192 0 8 4 What is the subnet mask of the second adapter 255 255 225 0 Do you want to enable switch manageme
3. Because the HyperFabric network does not currently support ServiceGuard heartbeat connections you must use an alternative type of connection for the heartbeat such as FDDI Token Ring 100BaseT or Ethernet as shown in Figure 4 2 Chapter 4 Figure 4 2 Switches Configuring HyperFabric Configuring HyperFabric with ServiceGuard An ServiceGuard Configuration with Two HyperFabric Ethernet Heartbeat LAN 1 node A HF adapter 0 HF adapter 0 HF adapter 1 Chapter 4 HF adapter 1 Ethernet Port Em Ethernet Heartbeat LAN 0 HF adapter 0 HF adapter 1 HF switch 1 Ethernet Port HF adapter 1 HF adapter 0 89 Configuring HyperFabric Configuring HyperFabric with ServiceGuard 90 How HyperFabric Handles Adapter Failures HyperFabric adapters are handled differently than other types of networking adapters such as Ethernet FDDI and Fibre Channel in the ServiceGuard environment In the non HyperFabric cases two network links are in a node and one will be active and one will be idle or in standby In the case of an active link failure ServiceGuard is notified and the network traffic is switched to the standby adapter which then becomes active However in the case of HyperFabric if two adapters are in a node both will be active If one active HyperFabric adapter fails its network traffic is switched to the other active Hype
4. To use clic_init to configure the Transparent Local Failover feature on HMP see the section Configuring HMP for Transparent Local Failover Support Using the clic_init command on page 104 If the global configuration file already exists and you are running clic_init again to change the file you have the option of retaining or modifying the existing configuration information in addition to adding new information pertaining to new hardware Also once you have completed your changes and clic_init ends its processing you must stop HyperFabric by running the clic_shutdown command or using SAM and then start HyperFabric by running the clic_start command or using SAM Otherwise your configuration information changes will not take effect See Stopping HyperFabric on page 128 and Starting HyperFabric on page 109 for more information If you include opt clic bin in your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic init You must be logged in as root to run this command The syntax is as follows clic init c where e c specifies that you want to create the global configuration file You are prompted for the information described in Information You Need on page 71 Note that if the global configuration file already exists for example when you are adding an adapter to an existing fabric
5. you need to specify additional configuration information Specifying configuration information adds or changes only the addresses and other information in the global configuration file based on the information you supply It does not perform any operations to check the relationships between that information and any physical connections within the fabric You need to create the global configuration file in the following situations e You have just installed the HyperFabric hardware and software on the system e You want to change the information in the HyperFabric global configuration file see the preceding note Creating the global configuration file also modifies the etc rc config d netconf file adding some HyperFabric related lines that end with the characters clic These lines are used by the HyperFabric software and are not comments so do not remove them from the file You can create the global configuration file by using the clic init command described in Using the clic init Command on page 79 or SAM described in Using SAM on page 81 You cannot enable switch management through SAM you must use the clic init command Chapter 4 IMPORTANT Chapter 4 Configuring HyperFabric Performing the Configuration Using the clic_init Command Run the clic_init command to create the global configuration file To view the man page for clic_init see Viewing man Pages on page 127 of this manual
6. 85 Configuring HyperFabric Configuring the HyperFabric EMS Monitor 12 Validate by pressing OK NOTE Although EMS is able to monitor each HyperFabric adapter on every node in the fabric as well as the entire HyperFabric subsystem EMS is not able to monitor HyperFabric switches For more detailed information on EMS including instructions for implementing this feature see the EMS Hardware Monitors Users Guide Part Number B6191 90028 September 2001 Edition 86 Chapter 4 NOTE Chapter 4 Configuring HyperFabric Configuring HyperFabric with ServiceGuard Configuring HyperFabric with ServiceGuard HyperFabric supports the ServiceGuard HA product If you plan to configure HyperFabric with ServiceGuard please read this section Otherwise skip this section and go on to the next section Configuring HMP for Transparent Local Failover Support on page 96 ServiceGuard lets you create HA clusters of HP 9000 server systems Within the cluster ServiceGuard allows you to group your application services individual HP UX processes into packages In the event of a single service node network or other resource failure ServiceGuard can transfer control of the package to another node in the cluster allowing services to remain available with minimal interruption ServiceGuard directly monitors cluster nodes LAN interfaces and services which are the individual processes within an application In addition specialize
7. HyperFabric Adapters on page 4 and Switches and Switch Modules on page 5 To determine the version of HyperFabric issue the following command swlist grep i hyperfabric Other Product Elements The following are the other elements of the HyperFabric product family HF2 fiber cables C7524A 2m length 7525A 16m length C7526A 50m length C7527A 200m length The HyperFabric software The software resides in ASICs and firmware on the adapter cards and includes user space components and HP UX drivers HyperFabric supports the IP network protocol stack specifically TCP IP and UDP IP HyperFabric software includes HyperMessaging Protocol HMP HMP provides higher bandwidth lower CPU overhead and lower latency the time a message takes to get from one point to another However these HMP benefits are available only when applications that are developed on top of HMP are running HMP can only be used on HP 9000 systems running HP UX 11 0 or 11i v1 provided HyperFabric A6092A or A6386A PCI 4X adapter cards are installed on those systems Chapter 1 Chapter 1 Overview of the HyperFabric Product HyperFabric Concepts HyperFabric Concepts This section briefly describes some of the basic HyperFabric concepts and terms The fabric is the physical configuration that consists of all the HyperFabric adapters the HyperFabric switches if any and the HyperFabric cables connecting them T
8. e There must be even number of HF2 adapters on any given node and all adapters installed must be configured In addition all the adapters must belong to a card pair e HMP can fail over traffic only between adapters that belong to the same card pair 96 Chapter 4 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support e HMP does not support backward compatibility in the local failover and non local failover mode However TCP UDP IP supports backward compatibility and interoperability e When HMP is configured in the local failover mode all the resources in the cluster are utilized If a resource fails in the cluster and is restored HMP does not utilize that resource until another resource fails e Before running clic_start on all the nodes in the cluster ensure that all the configured cards are connected in the cluster In other words before running the clic_start command verify that all the cables are connected to the adapters and switches e Ifaresource in the cluster fails and is restored perform the following steps to ensure full utilization of that resource Shutdown Oracle RAC Execute the clic_shutdown command Execute the clic_start command Restart Oracle RAC e After executing the clic_start command on all node in the cluster ensure that you run Oracle RAC only after one minute approximately e If all the trunks between the switches are down then execute clic_
9. switch v Ifthe associated Ethernet port is not connected to the Ethernet network the LED is off Y Ifthe connection from the associated Ethernet port to the Ethernet network is operational the LED shows as solid green v If data is flowing between the associated Ethernet port and the 148 Ethernet network the LED shows as flashing green Chapter 6 Troubleshooting HyperFabric LED Colors and Their Meanings 1 For each port on the cards switch module in the switch V Ifthe port is not operational the LED is off Some of the possible reasons for this happening are the following A cable is not attached correctly to the port or no cable is attached at all The port is connected to a non operational adapter in an HP 9000 See Adapter LEDs on page 143 or Table 6 2 on page 145 for some tips about a non operational adapter The cable is bad The port is bad Note that if a port is bad and for some reason you cannot use a different port on that card switch module you can replace the card switch module Remember though that the integrated 8 port fiber card can be removed by qualified HP personnel only The A6388A and A6389A switch modules can be removed by you or qualified HP personnel However you first should try turning the switch s power off and then back on V Ifthe connection from the port to the corresponding adapter in the HP 9000 is operational the LED on the port shows as solid green V If data is
10. Chapter 5 Chapter 5 Step Step Step Step Step Managing HyperFabric Stopping HyperFabric Using SAM To use SAM to stop HyperFabric on a local HP 9000 system running HP UX 11i v2 complete the following steps Start SAM Select the Networking and Communications area Select HyperFabric Pull down the Actions menu and select Stop HyperFabric Note that if HyperFabric is not running on the system Stop HyperFabric is grayed out and you cannot select it When HyperFabric stops a confirmation message displays Also the status HyperFabric Not running is displayed above the adapter configuration area of the screen Exit SAM 129 Managing HyperFabric Stopping HyperFabric 130 Chapter 5 6 Troubleshooting HyperFabric This chapter contains these sections that describe troubleshooting HyperFabric e Running Diagnostics on page 133 Chapter 6 131 Troubleshooting HyperFabric e Using Support Tools Manager on page 140 e Useful Files on page 141 e LED Colors and Their Meanings on page 143 e Determining Whether an Adapter or a Cable is Faulty on page 153 e Determining Whether a Switch is Faulty on page 154 e Replacing a HyperFabric Adapter on page 156 e Replacing a HyperFabric Switch on page 157 132 Chapter 6 Troubleshooting HyperFabric Running Diagnostics Running Diagnostics Before running HyperFabric diagnost
11. var adm clic ip drv trc This is one of the HyperFabric software s trace files and it is created by the clic diag D TCP IP command var adm clic ip drv trcO This is one of the HyperFabric software s trace files and it is the primary file created by the clic diag C TCP IP command var adm clic ip drv trci This is one of the HyperFabric software s trace files and it is created by the clic diag C TCP IP command when the primary trace file clic ip drv trcO0 becomes full 141 Troubleshooting HyperFabric Useful Files 142 var adm clic_fw dumpx This is the default file for a memory dump of an HyperFabric adapter created when the clic_diag d command is run without specifying an output file etc rc config d netconf This file contains IP related configuration information for all of the networking adapters installed in the HP 9000 IMPORTANT clic init and SAM modify this file adding some HyperFabric related lines that end with the characters clic These lines are used by the HyperFabric software and are not comments so do not remove them from the file etc services This is the system service name database IMPORTANT These two HyperFabric related lines must be in this file hp clic 3384 tcp clic management daemon hp clic 3384 udp clic switch management These lines are used by the HyperFabric software and are not comments so do not remove them from the file Ch
12. V The Operating Fault LED shows solid green V The Power A and Power B LEDs show solid green V The Ethernet Port Main and Ethernet Port Aux LEDs are showing solid green connected or flashing green Ethernet traffic is flowing to the switch See Figure 3 2 or Figure 3 3 below for the locations of the LEDs On the integrated 8 port fiber card in the middle slot of the switch check that for each switch port that is connected to an HF2 adapter the LED on the port shows as solid green see Figure 3 2 on page 59 or Figure 3 3 on page 60 This means the connection is operational On the switch module in the expansion slot the bottom slot of the switch check that for each switch port that is connected to a HyperFabric adapter the LED on the port shows as solid green see Figure 3 2 on page 59 or Figure 3 3 on page 60 This means the connection is operational For more information about the switch s LEDs see HF2 Switch LEDs on page 148 If you want to install another HF2 switch without using the rail kit go to step 1 If you want to install another HF2 switch using the rail kit go to With the Rail Kit on page 61 Otherwise go to Chapter 4 Configuring HyperFabric on page 67 Chapter 3 Chapter 4 Configuring HyperFabric This chapter contains the following sections that describe configuring HyperFabric e Configuration Overview on page 69 67 Configuring Hy
13. slanted type indicates document and book names daemon Courier font type indicates daemons files commands manpages and option names 16 In syntax definitions square brackets indicate items that are optional and braces indicate items that are required HP Welcomes Your Feedback HP welcomes any comments and suggestions you have on this manual You can send your comments in the following ways e Internet electronic mail netinfo feedback cup hp com e Using a feedback form located at the following URL http docs hp com assistance feedback html Please include the following information along with your comments The complete title of the manual and the part number The part number appears on the title page of printed and PDF versions of a manual e The section numbers and page numbers of the information on which you are commenting e The version of HP UX that you are using Please note that the HP UX networking communications publications group does not provide technical support for HP products 1 Overview of the HyperFabric Product This chapter contains the following sections that give general information about HyperFabric Chapter 1 Overview of the HyperFabric Product e About HyperFabric on page 3 e HyperFabric Products on page 4 e HyperFabric Concepts on page 7 2 Chapter 1 Chapter 1 Overview of the HyperFabric Product About HyperFabric About HyperFabric HyperFabric is an H
14. 2 6 is very popular for running Oracle RAC 91 Superdomes or other large servers make up the Database Tier Database Tier nodes communicate with each other using HMP Application Tier nodes communicate with each other and to the Database Tier using TCP UDP IP HMP Enterprise Database Configuration Single Connection Figure 2 6 Between Nodes perdome iperdome perdome iperdome DATABASE TIER T EE APPLICATION TIER ll a NL NL LF s SW NL NL Pe EE e P E NL LE LE NL NIL NL NL NL Chapter 2 35 Planning the Fabric Hyper Messaging Protocol HMP Enterprise Database Local Failover Supported Configuration The HMP enterprise configuration is a scalable solution For high availability and performance you can easily scale the HMP enterprise configuration with multiple connections between the HyperFabric resources Any single point of failure in the database tier of the fabric is eliminated in Figure 2 7 Figure 2 7 Local Failover Supported Enterprise Database Configuration 36 Multiple Connections between Nodes iperdome perdome perdome T el DE Le Las DATABASE TIER T D i APPLICATION TIER g Ss a Eq E a Em IL x Em P umm NIL NL NIL NL NIL NL In this configuration if a HyperFabric resource adapter cable switch or switch port fails in a cluster HMP transparently fails over traffic using another available resource For more information see Conf
15. 2 Publishing History Details Continued Edition Number Publication Date Eleventh June 2001 Twelfth September 2002 Thirteenth July 2003 Fourteenth March 2004 What Is in This Document HyperFabric Administrator s Guide is divided into several chapters each of which contains information about installing configuring or troubleshooting HyperFabric The appendixes contain supplemental information The following list describes the content in more detail Organization Chapter Description Overview of the Presents an overview of HyperFabric HyperFabric Product and lists the components that the HyperFabric product contains Planning the Fabric Describes the steps that need to be Installing HyperFabric followed while planning the fabric Describes the tasks to install the HyperFabric products on the HP UX 11i v2 operating system Configuring HyperFabric Managing HyperFabric Troubleshooting HyperFabric Describes the tasks to configure HyperFabric Describes the tasks to start stop and manage HyperFabric Describes how to troubleshoot HyperFabric xiii Xiv New and Changed Information in This Edition This edition includes information about the transparent local failover feature of Hyper Messaging Protocol HMP This feature is available with the HyperFabric version B 11 23 01 Typographic Conventions This document uses the following typographic conventions Book Title Italic
16. 4 Status A B Port an oO oo zi UX f Port Port Port Port Port Port Port Y Port Status 7 6 5 4 3 2 1 0 alda e Lia La Lle fle Lie Port Port Port Port EM eFC S C L1 L1 oO 1 A6389A HF2 4 port copper switch module in expansion slot You can install the HF2 switch in one of the following ways Using the rail kit that is shipped with the switch see the next section With the Rail Kit HP strongly recommends installing the HF2 switch this way Attaching the switch directly to the rack see Without the Rail Kit on page 65 60 Chapter 3 Chapter 3 Step 1 Step 2 Installing HyperFabric Installing HyperFabric Switches With the Rail Kit As mentioned earlier HP strongly recommends installing the HF2 switch using the rail kit When you install the HF2 switch you will be putting the front of the switch the end with the flanges wings at the back of the rack The steps for installing the HF2 switch using the rail kit are as follows Prepare the rack for rail and switch installation Install and secure the rails in the rack using two screws per rail The following figure shows the rack with the rails installed rack screws back S screws rack front WV Step 3 From the front of the rack slide the switch with the front of the switch facing the back of the rack into the rack on the rails Move it until it is snug against the back of the r
17. Chapter 2 Planning the Fabric HyperFabric Features Parameters and Supported Configurations for TCP UDP IP and HMP Applications Chapter 2 HyperFabric Features Parameters and Supported Configurations for TCP UDP IP and HMP Applications The following sections in this chapter define HyperFabric functionality for TCP UDP IP applications and Hyper Messaging Protocol HMP applications There are distinct differences in supported hardware available features and performance depending on which protocol is used by applications running on the HyperFabric 13 Planning the Fabric TCP UDP IP 14 TCP UDP IP TCP UDP IP is supported on all HF2 hardware Although some of the HyperFabric adapter cards support both HMP and TCP UDP IP applications in this section the focus is on TCP UDP IP HyperFabric applications Application Availability All applications including Oracle 9i and HP MPI that use the TCP UDP IP stack are supported Features This section discusses the following HyperFabric features on TCP UDP IP OnLine Addition and Replacement OLAR Supported The OLAR feature allows the replacement or addition of HyperFabric adapter cards while the system node is running HyperFabric supports this functionality on the SD64A rx8620 rx4640 rp54xx L class rp74xx N class rp8400 and Superdome systems running on the HP UX 11i v2 platform For more information on OLAR including instructions for implementing this f
18. HyperFabric Configuring HMP for Transparent Local Failover Support Thus if a switch fails HMP transparently fails over traffic only if at least one member of the card pair is physically reachable through the other switch Case 3 Cable Failure Between Two Switches see Figure 4 9 If a cable between two switches fails HMP traffic fails over to the other available cable between those two switches Cable Failover Between Two Switches node A HF adapter 1 HF adapter 0 HF HF adapter 1 HF Switch 0 4 y HF Switch 1 HF adapter 1 HF adapter 0 0 1ed p129 Card Pair 0 HF adapter 0 0 1ed p1e9 HF adapter 1 Card Pair 0 Chapter 4 101 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support 102 Step 1 Step 2 Step 3 Step 4 Step 5 Configuring HMP for Transparent Local Failover Support Using SAM To use SAM to configure HMP for Local Failover Support complete the following steps Start SAM Select the Networking and Communications area Select HyperFabric All HyperFabric adapters installed in the system are listed installed adapters that are not yet configured show Not Configured in the Status field Before proceeding to the next step configure all the available adapters Pull down the Actions menu and select Configure Local Failover for HMP SAM now verif
19. HyperFabric software and firmware The clic_diag command is detailed in the section Running Diagnostics on page 133 Examples of clic_probe Some examples of using clic_probe are shown below Example 1 If the local node is bent1y6 and you want to send five packets to verify that the adapter clic0 which is on bent 1y6 is able to handle traffic issue this command clic probe 1 c clicO p 5 The generated output could look like this CLIC PROBE 256 byte packets Local Loopback Source and Target Adapter ID bently6 corp 3 com clicO 256 bytes seq num 1 Packet Acknowledged 256 bytes seq num 2 Packet Acknowledged 256 bytes seq num 3 Packet Acknowledged 256 bytes Seq num 4 Packet Acknowledged 256 bytes seq num 5 Packet Acknowledged 2 RE bently6 corp3 com CLIC PROBE Statistics 5 packets transmitted 5 packets received 0 packet loss Example 2 If the local node is bent1y6 and you want to verify communications with the remote node bent1y4 issue this command clic probe bently4 Chapter 5 Chapter 5 Managing HyperFabric Verifying Communications within the Fabric CLIC_PROBE 256 byte packets Source adapter id bently6 corp3 com clicO Target adapter id bently4 corp7 com clic3 256 bytes seq_num 1 Packet Acknowledged 256 bytes seq_num 2 Packet Acknowledged 256 bytes seq_num 3 Packet Acknowledged 256 bytes seq_num 4 Packet Acknowledged 256 bytes seq_nu
20. Invalid CRC word Bad frame length Receive buffer overflow Null packets generated on link reset Chapter 5 B2 QOO OO p CO OO occdco ccd c c 121 Managing HyperFabric Displaying Status and Statistics Firmware reset notification Data corruption notification Unsupported QOS message received Invalid HMP VC ID Invalid HMP endpoint ID Invalid HMP endpoint protection key HMP message order violation Packet drops Transmit side congestion events Receive side congestion events Other Misc Statistics Mapping message send failures 0 O POOOGOQOOOQOOOQO b HMP in the local failover mode If the local node is bent1y7 and you want to display the statistics for the adapter clic1 which is on bent1y7 issue this command clic_stat c clic1 The generated output could look like this Date Node Sat Aug 5 16 08 30 2000 bently7 corp2 com 122 Adapter ID Card Pair Instance Number Adapter Type Firmware File Major Num Mgmt process driver handle Version H W Path Primary IP address Uptime time 4X HF2 PCI opt clic firmware clic fw hf232c 238 5 1 0 8 4 0 0 192 0 0 9 0 days 21 hours 1 min 3 sec Adapter State Flags Set Multiple flags may be set Configured and operational Switch mode F W Error Statistics Chapter 5 Managing HyperFabric Displaying Status and Statistics Frame buffer overflow Receive on disabled endpoint Invali
21. OLA 48 clic probe 112 to remote node 137 clic shutdown 128 clic start 110 after OLA 48 clic stat 117 viewing man pages 127 communications within fabric verifying 112 concepts 7 configuration clic init command 79 after OLA 48 global file contents of 69 creating 78 HA description of 7 information example of 74 104 needed for 71 overview of 69 typical description of 7 Index using SAM 81 102 Configuring HMP for Transparent Local Failover support 96 D deconfiguring an adapter 83 103 definitions fabric 7 HA 7 management process 7 diagnostics running 136 displaying statistics 117 status 117 F fabric communications within verifying 112 current map of 117 119 definition of 7 planning 11 files global configuration contents of 69 creating 78 software description of 52 structure of 51 H HA configuration description of 7 definition of 7 ServiceGuard and HyperFabric 87 hardware regulatory specifications 162 High Availability See HA I installation adapters 44 prerequisites for 43 software 51 steps for 55 switch HF2 59 steps for HF2 no rail kit 65 steps for HF2 rail kit 61 175 Index IP addresses relocatable See relocatable IP addresses L LEDs colors and meanings for adapters 143 145 for HF2 switches 148 152 on HF2 switches 59 60 M man pages viewing 127 management process definition of 7 o OLAR ad
22. Pe oe w ee e Ease 154 Replacing a HyperFabric Adapter 00 c cee cent eee 156 Replacing a HyperFabric Switch 0 0 2 0 0 ccc eee een nee 157 Safety Symbols i sv po ye REPRE te a A a lhe E s 161 Regulatory Statements ris rerni ero kra ee eee nee hh 162 Adapters and Switches i221 ee ok Oe dae alae inde dase be bed es 162 FCC Statement USA only osso 0 eee 162 DOC Statement Canada only 0 eee nas 163 Europe RFI Statement uale e d P n CR a ee 163 Australia and New Zealand EMI Statement 0 0 0 cece sn 163 Contents Radio Frequency Interference Japan Only 0 0 0 eens Declarations of Conformity 0 0 000 c cc eee een eens Physical Attributes 4st aE RD beh ek 8 ete wee ed cee ba ES Environmental vi Figures Figure 2 1 TCP UDP IP Point To Point Configurations 000 22 Figure 2 2 TCP UDP IP Basic Switched Configuration 000 23 Figure 2 3 TCP UDP IP High Availability Switched Configuration 24 Figure 2 4 TCP UDP IP Hybrid Configuration 0 0 0 0 cc cece eee ee 25 Figure 2 5 HMP Point To Point Configurations 0 0 00 c eee eens 34 Figure 2 6 HMP Enterprise Database Configuration Single Connection Between NODC Gis seres Queda ep Renta th chk tes tato ae boue ria ake tovto ei dn e a uteri ee cha 35 Figure 2 7 Local Failover Supported Enterprise Database Configuration Multiple Connections betw
23. The clic stat Command on page 117 You should note each switch s ID because it is used as input to other HyperFabric commands For the entire fabric you need the IP multicast address that all the switches and nodes in the fabric will register to The address must be a class D address Note that if you do not have switch management enabled you do not need this information clic init will not prompt you for it For each node in the fabric you need the IP address of the node s Ethernet LAN interface that is on the same subnet as the switches As mentioned earlier a node might have multiple LAN interfaces Note that if you do not have switch management enabled you do not need this information clic init will not prompt you for it As stated earlier HP recommends that you do not enable switch management You should also check your etc hosts file when you are using files for host name look up to ensure that the entries for all of the systems are in the correct format the official host name which is the full domain extended host name and any alias names For example Chapter 4 Configuring HyperFabric Information You Need IP_address bently6 corp3 com bently6 IP address bently4 corp7 com testi IP address bently2 corp4 com test3 Chapter 4 73 Configuring HyperFabric Information You Need Figure 4 1 74 Ethernet LAN Switch ID sw clicO IP address 193 0 0 20 HF WE IP address 193 0 0 21 Ethernet MAC
24. a dynamically loadable kernel module DLKM it can be loaded manually See the Dynamically Loadable Kernel Modules section in Configuring HP UX For Peripherals for more information e Ifthe driver is static and not configured in the kernel then the card cannot be added online The card could be physically inserted online but no driver would claim it If there is any question about the driver s presence or if it is uncertain that the replacement card is identical to the existing card ioscan can be used together with rad to investigate If more than one operational HyperFabric adapter is present when SAM requests the suspend operation for all ports on the target adapter HyperFabric will redirect the target adapter s traffic to a local backup adapter using local failover Client applications using the replaced adapter will not be interrupted in any way If the adapter being replacing is active and it is the only operational HyperFabric adapter on the HP 9000 system SAM displays the following warning message WARNING You have 1 operational HyperFabric card If you go ahead with this operation you will lose network access via HyperFabric until the on line replaced HyperFabric card becomes operational 49 Installing HyperFabric Installing HyperFabric Adapters CAUTION 50 You are asked if you want to continue If you reply Yes client applications are suspended Replace the adapter according to the pr
25. address Ethernet MAC address 00 60 b0 d0 02 57 00 60 b0 d0 02 56 IP multicast address 226 10 1 1 HF adapter 0 Adapter ID Adapter ID Adapter ID Adapter ID clicO clic1 clico clic1 IP address IP address IP address IP address 192 0 0 1 192 0 8 3 192 0 0 2 192 0 8 4 subnet mask subnet mask subnet mask subnet mask 255 255 255 0 255 255 255 0 255 255 255 0 255 255 255 0 node A node B Configuration Information Example This example uses some dummy that is not valid addresses to the components in Figure 4 1 The dummy addresses are used only to show the flow of the information provided as input to the clic_init command and SAM Do not try to use these addresses in your configuration To use clic_init to configure the Transparent Local Failover feature on HMP see the section Configuring HMP for Transparent Local Failover Support Using the clic_init command on page 104 Map for Configuration Information Example Switch ID sw_clic1 IP multicast address 226 10 1 1 HF adapter 1 HF adapter 0 HF adapter 1 IP address 193 0 0 10 IP address 193 0 0 11 Chapter 4 Configuring HyperFabric Information You Need Using the configuration information in Figure 4 1 the information you would specify when you run clic_init or SAM on each of the nodes is listed below This example is not an exact depiction of the prompts produced by cl
26. clic init prompts you with the existing configuration information As you are prompted with each piece of information you can then confirm that you want to keep it or you can change it e displays the online help for clic init 79 Configuring HyperFabric Performing the Configuration 80 If you do not specify any of the above parameters the online help for clic_init is displayed After you have entered the information for all the adapters in the node and all of the switches if any in the fabric a summary of the configuration information is displayed Once clic_init has finished you do one of the following things e If you want to configure HyperFabric with ServiceGuard complete the configuration described in Configuring HyperFabric with ServiceGuard on page 87 then run clic_start or use SAM to start HyperFabric e Ifyou have just created the global configuration file on the local node for the first time and you are not configuring ServiceGuard run clic_start or use SAM to start HyperFabric e Ifyou have just changed an existing configuration file on the node run clic_shutdown or use SAM to stop HyperFabric and then run clic_start or use SAM to start HyperFabric Until you do those two things your configuration changes will not take effect See Stopping HyperFabric on page 128 and Starting HyperFabric on page 109 for more information Examples of clic init Some examples of using the clic init
27. data is dumped into the file var adm clic ip drv trc and formatted C TCP IP operates in a way similar to D TCP IP except that the trace data in the kernel is dumped at regular intervals to the trace file var adm clic ip drv trcO and if that file gets full to var adm clic ip drv trc1 So new trace events are appended to the trace output file This stops when tracing is turned off 137 Troubleshooting HyperFabric Running Diagnostics e B specifies that you want the size in bytes of the trace buffer to be trace_buffer_size The buffer is dynamically allocated when tracing is enabled Also the buffer is circular which means that when the end of the buffer is reached the data wraps around to the beginning of the buffer and overwrites any previous data If you do not specify this parameter a default buffer is created with a size of 64k bytes e F specifies trace_file_sizeas the maximum size of the trace output file For the D TCP_IP parameter the file is var adm clic_ip_drv trc For the C TCP_IP parameter the files are var adm clic ip drv trcO0 and if needed var adm clic ip drv trcl e ddumps the memory of the adapter identified by adapter rDto the dump file filename If you do not specify filename the default file is var adm clic fw dumpx where xis the adapter instance number e displays the online help for c1ic diag If you do not specify any of the above parameters the online
28. do anything e Ifyou have changed the HyperFabric configuration with the clic_init command or SAM you must stop HyperFabric by running the clic_shutdown command or using SAM and then start HyperFabric by running the clic_start command or using SAM See either Using the clic_shutdown Command on page 128 or Using SAM on page 129 whichever is appropriate Using SAM To use SAM to start HyperFabric on an HP 9000 system running HP UX 11i v2 complete the following steps Start SAM Select the Networking and Communications area Select HyperFabric Pull down the Actions menu and select Start HyperFabric Chapter 5 Managing HyperFabric Starting HyperFabric When HyperFabric starts a confirmation message displays Also the status HyperFabric Running is displayed above the adapter configuration area of the screen Step 5 Exit SAM Chapter 5 111 Managing HyperFabric Verifying Communications within the Fabric IMPORTANT 112 Verifying Communications within the Fabric You can verify the communications within the fabric by running the clic_probe command which is described below You can also use clic_probe to verify the status of specific adapters You should also check your etc hosts file when you are using files for host name look up to ensure that the entries for all of the systems are in the correct format the official host name which is the full domain exten
29. flowing between the port and the corresponding adapter the LED shows as flashing green Note that data does not start to flow until HyperFabric initialization has occurred see Using the clic init Command on page 79 Note that a legend listing the possible port states disconnected connected and traffic and the corresponding LED colors is printed on the front of the switch for your reference See Figure 3 2 on page 59 and Figure 3 3 on page 60 for the locations of the switch LEDs and the legend Table 6 3 below summarizes the HF2 switch LED information in a table format Chapter 6 149 Troubleshooting HyperFabric LED Colors and Their Meanings Table 6 3 150 HF2 Switch LED Colors and Meanings LED Color Meaning Status None The card switch module is not operating Solid A fault is yellow occurring on the card switch module Solid green The card switch module passed the self test and is operating Power A and None Power to switch is Power B off Solid green Power to switch is on Notes Integrated Ethernet management LAN adapter card in the top slot amp integrated 8 port fiber card in the middle slot the card can be safely removed by qualified HP personnel only A6388A HF2 8 port fiber or A6389A HF2 4 port copper switch module in the expansion slot the bottom slot the switch module can be safely removed by you or qualified HP personnel One LED
30. it as part of the first prompt If you use SAM it assigns the adapter an ID and displays it in the Adapter Name column of the Configure HyperFabric Adapter screen You can also determine an adapter s ID by running the clic stat command see The clic stat Command on page 117 You should note each adapter s ID because it is used as input to other HyperFabric commands 1 For using the Transparent Local Failover feature of HMP available in the version B 11 23 01 of HyperFabric you need to define the card pairs 4 For each HyperFabric switch in the fabric if you are going to enable switch management 71 Configuring HyperFabric Information You Need IMPORTANT IMPORTANT 72 v The IP address of the switch Y The MAC address of the switch s Ethernet port If you do not already know the switch s MAC address it is printed on a label on the back of the HF switch and on the front of the HF2 switch See Figure 3 2 on page 66 for the location of the label on the HF switch and Figure 3 2 on page 59 and Figure 3 3 on page 60 for the location of the label on the HF2 switch You cannot enable switch management through SAM you must use the clic init command When clic init begins to prompt you for the information for each switch it assigns an ID for example sw_clic0 to that switch and displays it as part of the first prompt Note that you can also determine a switch s ID by running the clic stat command see
31. solid green switch is still operational Chapter 6 Table 6 3 Chapter 6 Troubleshooting HyperFabric LED Colors and Their Meanings HF2 Switch LED Colors and Meanings Continued LED Ethernet Port Main and Ethernet Port Aux Color Meaning Notes None Ethernet port is This can happen if the Ethernet disconnected port is not correctly connected to the Ethernet network or the integrated Ethernet management LAN adapter card is experiencing a fault Management is disabled when the Ethernet port is not operating however this does not prevent the fabric from operating Solid green Connection from the Ethernet port to the Ethernet network is operational Flashing Data is flowing green between the Ethernet port and the Ethernet network 151 Troubleshooting HyperFabric LED Colors and Their Meanings Table 6 3 HF2 Switch LED Colors and Meanings Continued 152 LED Port x Color None Solid green Meaning Notes Port is not operational e Acable is not attached correctly to the port or no cable is attached at all The port is connected to a non operational adapter in an HP 9000 See Adapter LEDs on page 143 or Table 6 2 on page 145 for some tips about a non operational adapter Thecable is bad The port is bad Note that if a port is bad and for some reason you cannot use a different port on that card switch modu
32. step 4 Mount the CD ROM drive by using the following command at the command prompt mount device_name where device_name is the name assigned to the CD ROM drive Run the swinstall program using the following command usr sbin swinstall This opens the Software Selection window Change the Source Host Name if necessary and then enter the mount point of the drive in the Source Depot Path field Select the OK button to return to the Software Selection window The Software Selection window now contains a list of available software to install Highlight the HyperFabric software e HP UX 11i v2 HyprFabrc 00 Choose Mark for Install from the Actions menu this chooses the highlighted software From the Actions menu select the Install menu and then choose Install This begins product installation and opens the Install Analysis window Select the OK button in the Install Analysis window when the Status field displays a Ready message Select the YES button in the Confirmation window to start software installation 55 Installing HyperFabric Installing the Software Step 11 Step 12 Step 13 Step 14 Step 15 56 swinstall loads the fileset runs the control script for the filesets and builds the kernel When the processing is finished the Status field displays a Ready message Select Done and then the Note window opens
33. switch modules 172 A6388A 0 8 inch 2 cm height 15 75 inch 40 cm width 11 5 inch 29 2 cm length 20 ounces 570 g weight A6389A 0 8 inch 2 cm height 15 75 inch 40 cm width 11 5 inch 29 2 cm length 20 ounces 570 g weight Appendix B Technical Specifications Environmental Environmental These environmental specifications are the same for the HyperFabric adapters and switches e Temperature 40 to 70 degrees C non operating storage 5 to 40 degrees C operating 20 to 30 degrees C recommended operating range e Relative Humidity 15 to 90 non operating storage 15 to 80 22 degrees C operating 15 to 80 22 degrees C recommended operating range e Altitude 15 000 feet 4 6 km non operating 10 000 feet 3 1 km operating Appendix B 173 Technical Specifications Environmental 174 Appendix B A adapters deconfiguring 83 103 faulty testing for 153 installing 44 LED colors and meanings 143 145 OLA of 48 OLR of 49 regulatory specifications 162 replacing 156 C cables testing for faulty 153 CLIC definition of 8 52 clic diag command 114 136 clic init command 79 after OLA 48 clic probe command 112 to remote node 137 clic shutdown command 128 clic start command 110 after OLA 48 clic stat command 117 CLuster InterConnect See CLIC commands clic diag 114 136 clic init 79 after
34. takes place when a resource is removed from a cluster The difference between DRU and OLAR is that OLAR applies only to the addition or replacement of adapter cards from nodes Load Balancing Supported When an HP 9000 HyperFabric cluster is running TCP UDP IP applications the HyperFabric driver balances the load across all available resources in the cluster including nodes adapter cards links and multiple links between switches Switch Management Not Supported Chapter 2 Planning the Fabric TCP UDP IP Switch Management is not supported Switch management will not operate properly if you enable it on a HyperFabric cluster Diagnostics Supported Diagnostics can be run to obtain information on many of the HyperFabric components using the clic_diag clic_probe and clic_stat commands as well as the Support Tools Manager STM For more information on HyperFabric diagnostics see Running Diagnostics on page 133 Configuration Parameters This section describes the maximum limits for TCP UDP IP HyperFabric configurations There are numerous variables that can impact the performance of any HyperFabric configuration For guidance on specific HyperFabric configurations for TCP UDP IP applications see the section TCP UDP IP Supported Configurations on page 21 Chapter 2 HyperFabric is supported only on the HP 9000 series servers and workstations TCP UDP IP is supported for all HyperFabric hardware and soft
35. 6 The generated output could look like this CLIC_PROBE 256 byte packets sent Source adapter id bently6 corp3 com clicO Target adapter id bently7 corp4 com clicl 256 bytes seq num 1 Packet Acknowledged 256 bytes seq num 2 Packet Acknowledged 256 bytes seq num 3 Packet Acknowledged 256 bytes seq num 4 Packet Acknowledged 256 bytes seq num 5 Packet Acknowledged pcc uc bently7 corp4 com CLIC PROBE Statistics 5 packets transmitted 5 packets received 0 packet loss Note that the VRID you specified 194 actually went to the adapter clici on bently7 And as explained earlier you run the clic stat d VRID command to determine the VRID and switch hopcount to specify Chapter 5 Chapter 5 Managing HyperFabric Displaying Status and Statistics Displaying Status and Statistics You can get the status of and statistics associated with many of the HyperFabric components by using the clic_stat command which is described below The clic_stat Command The following list contains some of the information that the clic_stat command provides e The current fabric map in textual format e The status of one or more HyperFabric adapters e The global configuration information for each HyperFabric adapter and switch if the fabric contains switches The information includes the firmware type 8 bit or 32 bit which is used for interoperability purposes e The card pair information if you con
36. Directive 89 336 EEC and carries the CE marking accordingly 1 The Product was tested in a typical configuration with Hewlett Packard information technology equipment Roseville CA June 12 1998 European Contact Your local Hewlett Packard Sales and Service Office or Hewlett Packard GmbH Department TRE Herrenberger StraBe 130 D 71034 B blingen FAX 49 7031 14 3143 Appendix A 165 Safety and Regulatory Information Regulatory Statements DECLARATION OF CONFORMITY According to ISO IEC Guide 22 and EN 45014 Manufacturer s Name Hewlett Packard Company Systems Interconnect Solutions Lab Manufacturer s Address 8000 Foothills Blvd Roseville CA 95747 USA declares that the product Product Name Hyperfabric PCI 4X Adapter Model Number s A6092 60001 Product No A6092A Product Options Ail conforms to the following Product Specifications Safety IEC 950 1991 A1 A2 A3 A4 EN 60950 1992 At A2 A3 A4 A11 GB 4943 1995 EMC CISPR 22 1993 EN 55022 1994 amp A2 1996 Class A CNS 13438 GB 9254 1988 CFR47 Part 15 Class A CISPR 24 1997 EN 55024 1998 IEC 61000 4 2 IEC 61000 4 3 ENV 50204 IEC 61000 4 4 IEC 61000 4 8 Supplementary Information The product herewith complies with the requirements of the EMC Directive 89 336 EEC and carries the CE marking accordingly 1 The Product was tested in a typical configuration with Hewlett Packard information tech
37. Example on page 74 to understand how to configure the fabric Decide the number of nodes that will be interconnected in the fabric Decide the type of HP 9000 system for each node for a list of supported HP 9000 systems see the HyperFabric Release Notes available at http docs hp com hpux netcom index html HyperFabric Determine the network bandwidth requirements for each node Determine the number of adapters needed for each node Determine if a High Availability ServiceGuard configuration will be needed If ServiceGuard is used each node should have at least two adapters Decide the topology of the fabric Determine how many switches will be used based on the number of nodes in the fabric The only configuration that can be supported without a switch is the node to node configuration HA or non HA HyperFabric supports meshed switches up to a depth of four switches starting with the following versions of the HyperFabric software e For HF2 switches software versions B 11 00 11 B 11 11 01 and B 11 23 00 11 Planning the Fabric Preliminary Considerations Step 11 Draw the cable connections from each node to switches if the fabric will contain switches If you use an HA configuration with switches it requires more than one switch for complete redundancy and to avoid a single point of failure For example each adapter can be connected to its own switch or two switches can be connected to four adapters 12
38. Fabric switch follow these steps If you do not have an HA environment stop HyperFabric on all nodes that are connected to the faulty switch See Stopping HyperFabric on page 128 If you have an HA environment with two switches where the backup switch will have taken over when the first switch failed disconnect all cables attached to the faulty switch Install a new switch See Installing HyperFabric Switches on page 57 If you stopped HyperFabric on all nodes connected to the switch in step 1 start it on those nodes See Starting HyperFabric on page 109 157 Troubleshooting HyperFabric Replacing a HyperFabric Switch 158 Chapter 6 A Appendix A Safety and Regulatory Information This appendix contains the following sections that contain safety information and regulatory statements for the HyperFabric hardware 159 Safety and Regulatory Information components e Safety Symbols on page 161 e Regulatory Statements on page 162 These components were tested for conformance to various national and international regulations and standards The scope of this regulatory testing includes electrical and mechanical safety electromagnetic emissions immunity acoustics and hazardous materials When required approvals are obtained from third party test agencies Approval marks appear on the product label In addition various regulatory bodies require some information under the he
39. Fabric to gather information about HyperFabric components and to diagnose hardware problems Two tools are available in STM for HyperFabric e The Information Tool provides information about the HyperFabric adapter without resetting the adapter e The Diagnostics Tool can be used to run tests on the HyperFabric adapter the tool reports any failures You can run STM in three ways e In the X Windows environment e Incommand line mode In menu mode See the Support Media User s Manual for details about using STM 140 Chapter 6 Chapter 6 Troubleshooting HyperFabric Useful Files Useful Files When you are troubleshooting HyperFabric related problems you might find it useful to look at the contents of the following files etc rc config d clic global conf This is the global configuration file Check it to confirm that the configuration information is correct var adm clic log This is a global log file that contains a history of significant HyperFabric events For example it contains a history of the fabric s startup and any errors that occurred during it var adm clic log old This is the backup copy of the log file that is created when the log file grows larger than 100 Kbytes var adm OLDclic log This is the log file from the previous time the clic start command was executed var adm syslog log This is the system log file which contains a history of events occurring on the HP 9000 system
40. HP UX 11i v2 follow these steps Start SAM Select the Networking and Communications area Select HyperFabric All HyperFabric adapters installed in the system are listed installed adapters that are not yet configured show Not Configured in the Status field Highlight the adapter you want to configure Pull down the Actions menu and select Configure Adapter In the Configure HyperFabric Adapter screen specify information for the following fields e Internet Address Required The IP address of the adapter 81 Configuring HyperFabric Performing the Configuration Subnet Mask Optional The adapter s subnet mask If you do not specify this a default mask is chosen based on the adapter s IP address Interoperability Enabled Required Whether you want the adapter to be able to interoperate with adapters that are using any HP UX 11 0 HyperFabric versions earlier than B 11 00 11 or any HP UX 11i v1 HyperFabric versions earlier than B 11 11 01 Note that if you select No the HyperFabric software on the system will not be backwards compatible with previous releases This means you must update all of the other systems in the fabric to the version that is running on the system Default No Step 7 Select OK remember you cannot enable switch management within SAM Step 8 Exit SAM Once SAM has finished you do one of the following things See If you want to config
41. HyperFabric Administrator s Guide HP UX 11i v2 Edition 14 La invent Manufacturing Part Number B6257 90043 March 2004 United States Copyright 2004 Hewlett Packard Development Company L P All rights reserved Legal Notices The information in this document is subject to change without notice Hewlett Packard makes no warranty of any kind with regard to this manual including but not limited to the implied warranties of merchantability and fitness for a particular purpose Hewlett Packard shall not be held liable for errors contained herein or direct indirect special incidental or consequential damages in connection with the furnishing performance or use of this material Warranty A copy of the specific warranty terms applicable to your Hewlett Packard product and replacement parts can be obtained from your local Sales and Service Office U S Government License Proprietary computer software Valid license from HP required for possession use or copying Consistent with FAR 12 211 and 12 212 Commercial Computer Software Computer Software Documentation and Technical Data for Commercial Items are licensed to the U S Government under vendor s standard commercial license Copyright Notice Copyright 2004 Hewlett Packard Development Company L P All rights reserved Reproduction adaptation or translation of this document without prior written permission is prohibited except as allowed under the copyri
42. P high speed packet based interconnect for node to node communications HyperFabric provides higher speed lower network latency and uses less CPU than other industry standard protocols for example Fibre Channel and Gigabit Ethernet Instead of using a traditional bus based technology HyperFabric is built around switched fabric architecture providing the bandwidth necessary for high speed data transfer This clustering solution delivers the performance scalability and high availability required by the following Parallel Database Clusters Oracle 9i Real Application Clusters RAC Oracle 8i Parallel Servers OPS Parallel Computing Clusters Client Server Architecture Interconnects for example SAP Multi Server Batch Applications for example SAS Systems Enterprise Resource Planning ERP Technical Computing Clusters HP Message Passing Interface MPI based applications OpenView Data Protector earlier known as Omniback Network Backup Data Center Network Consolidation E services Overview of the HyperFabric Product HyperFabric Products NOTE HyperFabric Products HyperFabric hardware consists of host based interface adapter cards interconnect cables and optional switches HyperFabric software resides in Application Specific Integrated Circuits ASICs and firmware on the adapter cards and includes user space components and HP UX drivers Currently fiber based HyperFabric hardware are available In ad
43. Select the OK button in the Note window to reboot The user interface disappears and the system reboots When the system comes back up log on to the system as root and view the var adm sw swagent log and var adm sw swinstall log files to view error or warning messages that may have occurred during the installation While still logged in as root view the etc services file to ensure that these two HyperFabric related lines are present e hp clic 3384 tcp clic management daemon e hp clic 3384 udp clic switch management These lines are used by the HyperFabric software and are not comments therefore do not remove them from the file Verify that all installed HyperFabric adapters have a software state of CLAIMED by running the ioscan nf C clic command A check is also done to make sure all of the HyperFabric adapters have been claimed when clic init is activated or when SAM is used to configure HyperFabric If one or more HyperFabric switches are included in the configuration go to the next section of this chapter Installing HyperFabric Switches otherwise go to Chapter 4 Configuring HyperFabric on page 67 Chapter 3 WARNING Chapter 3 Installing HyperFabric Installing HyperFabric Switches Installing HyperFabric Switches This section contains the information you need to install HyperFabric switches As stated earlier the term HyperFabric2 HF2 switch refers to the function
44. The Product was tested in a typical configuration with Hewlett Packard information technology equipment Craig Stein Hardware Engineering Manager Cupertino CA April 2001 European Contact Your local Hewlett Packard Sales and Service Office or Hewlett Packard GmbH Department HQ TRE Herrenberger Stra e 130 D 71034 B blingen FAX 49 7031 14 3143 Appendix A 167 Safety and Regulatory Information Regulatory Statements 168 Appendix A B Technical Specifications This appendix contains the following sections that contain the technical specifications for the HyperFabric hardware components e Physical Attributes on page 171 Appendix B 169 Technical Specifications e Environmental on page 173 170 Appendix B Appendix B Technical Specifications Physical Attributes Physical Attributes The physical attributes of the HyperFabric adapters and switches are as follows e HF2 adapter A6386A PCI 4X 4 2 inch 10 7 cm height linch 2 5 cm width linch 18 cm length 4 25 ounces 120 g weight 171 Technical Specifications Physical Attributes e HF2 switch chassis A6384A 3 47 inch 8 8 cm height 19 inch 48 2 cm width at flanges 17 18 inch 43 6 cm width enclosure 17 5 inch 44 5 cm length depth from back of flanges to back plate 18 5 inch 47 cm length depth including protrusion of the switch module extractors 200 ounces 5680 g weight e HF2
45. abric man pages you must first add opt clic share man to your MANPATH environment variable Then to view a man page type the following man command_name For example to view the man page for clic_stat type this man clic stat 127 Managing HyperFabric Stopping HyperFabric 128 Stopping HyperFabric You can stop HyperFabric only on a local node Stopping HyperFabric on a node makes all of the HyperFabric adapters in that node unreachable by all other nodes in the fabric It stops the HyperFabric management process which stops all interconnect operations on the node To stop HyperFabric you use 1 the clic_shutdown command described below or 2 SAM described in Using SAM on page 129 Note that if you stop HyperFabric the only ways you can restart HyperFabric on the node is to 1 run the clic_start command see Using the clic_start Command on page 110 or 2 use SAM see Using SAM on page 110 Using the clic_shutdown Command Run the clic_shutdown command to stop HyperFabric on the local node If you include opt clic bin in your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic_shutdown You must be logged in as root to run this command The syntax is as follows clic_shutdown Note that you also can issue the command clic_shutdown to display the online help for clic_shutdown
46. ack You might not have enough clearance between the switch and the rail screws this may prevent you from easily 61 Installing HyperFabric Installing HyperFabric Switches 62 Step 4 sliding the switch If so try lifting the switch over the rail screws If you cannot do this remove the rail screws slide the switch into position and put the rail screws back in Align the two holes in each flange wing on the switch s front with the holes in the rack frame Fasten each flange of the switch to the rack by putting a screw in each of the four holes in the flanges Be sure to use screws with over sized heads The following figure shows the rack with the switch in this position rack Step 5 From the front of the rack install a bracket on the outside of each rail using two screws per bracket Be sure to use the upper screw holes on each bracket Put the screws in the two square holes closest to the back of the switch so that the brackets referred to as bracket 1 and bracket 2 in these steps are snug against the switch Be sure that the brackets make contact with the beveled sloped part of the switch not Chapter 3 Installing HyperFabric Installing HyperFabric Switches the flat part Otherwise the brackets will not secure the back of the switch properly The following figure shows the rack with these two brackets installed bracket 2 bracket 1 Step 6 When you are su
47. adings noted below 160 Appendix A Safety and Regulatory Information Safety Symbols Safety Symbols The safety related symbols used in this manual are shown below WARNING A WARNING denotes a hazard that can cause personal injury CAUTION A CAUTION denotes a hazard that can damage equipment Appendix A 161 Safety and Regulatory Information Regulatory Statements 162 Regulatory Statements This section contains the regulatory statements for the HyperFabric products Adapters and Switches The following statements apply to the HyperFabric adapters and switches e FCC Part 15 Class A e CES 003 Class A e CISPR 22 Class A e EN55022 Class A e EMC Directive 89 336 EEC e AS NZS 3548 Class A e VCCI Class A applies to switch only e UL and cUL Listed applies to switch only e T V Certificated applies to switch only e EN 60950 e Low Voltage Directive 73 23 EEC e CB Certificated IEC 950 applies to switch only FCC Statement USA only This device complies with part 15 of the FCC rules Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 this device must accept any interference received including interference that may cause undesired operation This equipment has been tested and found to comply with the limits of a Class A digital device pursuant to part 15 of FCC rules These limits are designed to provide reasonable protectio
48. al switch the A6384A switch chassis with one of the switch modules installed Before Installation Before you install the HyperFabric switch you should be aware of the following 4 The A6384A HF2 switch is supported beginning with the following HyperFabric software versions HP UX 11 0 version B 11 00 11 HP UX 1li vl version B 11 11 01 HP UX 11i v2 version B 11 23 00 HyperFabric switches are not supported by software versions earlier than those mentioned above respectively To determine the version of HyperFabric you have issue the following command at the command prompt swlist grep i hyperfabric For the HF2 switch HP recommends that you use the rails shipped with the switch when you mount it in a standard 19 inch rack even though the switch can be mounted in the rack by itself without the rails To prevent overheating you must leave one rack unit 1 EIA of empty space above the HyperFabric switch After the HyperFabric switch is mounted in the rack attach the various cables to the switch To avoid damage to any of the cables follow these guidelines 57 Installing HyperFabric Installing HyperFabric Switches If your cables have dust caps over the connectors keep them in place until you are ready to connect them This prevents dirt and oils from soiling any important surfaces Be careful not to stretch puncture or crush the cable To install an HF2 switch see Insta
49. also depends on slot availability and system resources See node specific documentation for details A maximum of 8 configured IP addresses are supported by the HyperFabric subsystem per instance of the HP UX operating system Maximum Number of Switches You can interconnect mesh up to 4 switches 16 port fiber or Mixed 8 fiber ports in a single HyperFabric cluster Trunking Between Switches multiple connections Trunking between switches can be used to increase bandwidth and cluster throughput Trunking is also a way to eliminate a possible single point of failure The number of trunked cables between nodes is only limited by port availability To assess the effects of trunking on the performance of any particular HyperFabric configuration contact your HP representative Maximum Cable Lengths HF2 fiber The maximum distance is 200m 4 standard cable lengths are sold and supported 2m 16m 50m and 200m HMP supports up to 4 HF2 switches connected in series with a maximum cable length of 200m between the switches and 200m between switches and nodes HMP supports up to 4 hybrid HF2 switches connected in series with a maximum cable length of 200m between fiber ports Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP e HMP is supported on A400 A500 rp2400 rp2450 rp54xx N class rp74xx L class rp8400 and Superdome servers running 64 bit HP UX e HMP is supported on HyperFabric starting HyperFabric
50. ample 0 0 cece eee eens 74 Performing the Configuration isese seret iena eee 78 Using the clic init Command 0 e eens 79 Examples of clic init lt sce lieRe e x REP Re ge Rl tese es 80 Using SAM S s sem erected esse See s uated a ee coda eto du pu ue tva ciet a 81 Deconfiguring a HyperFabric Adapter with SAM 0 0 0 cece eee 83 Configuring the HyperFabric EMS Monitor 0 0 cece eee eee 85 Configuring HyperFabric with ServiceGuard 0 c cece eee eens 87 How HyperFabric Handles Adapter Failures 0 0 0 e eee eee ene 90 Configuring HyperFabric with the ServiceGuard Resource Monitor 94 Configuring ServiceGuard with HyperFabric Using the ASCII File 94 Configuring ServiceGuard with HyperFabric Using SAM 94 Configuring ServiceGuard for HyperFabric Relocatable IP Addresses 95 Configuring HMP for Transparent Local Failover Support 0 04 96 How Transparent Local Failover Works 000 cece ee eee ences 98 Configuring HMP for Transparent Local Failover Support Using SAM 102 Deconfiguring HMP for Local Failover support Using SAM 103 Configuring HMP for Transparent Local Failover Support Using the clic_init command sso ta yas Ra gain ge Sa es twa nade uie 104 Contents 5 Managing HyperFabric Starting Hyperbabric el eee is gk aS RP ee Re PEERS 109 U
51. ange in interface status Notification can be accomplished with a SNMP trap or by logging into a user specified log file with a choice of severity or by email to a user defined email address For more information on EMS including instructions for implementing this feature see Configuring the HyperFabric EMS Monitor on page 85 in this manual and the EMS Hardware Monitors User s Guide Part Number B6191 90028 September 2001 Edition ServiceGuard Supported Within a cluster ServiceGuard groups application services individual HP UX processes into packages In the event of a single service failure node or network EMS provides notification and ServiceGuard transfers control of the package to another node in the cluster allowing services to remain available with minimal interruption ServiceGuard using EMS directly monitors cluster nodes LAN interfaces and services the individual processes within an application ServiceGuard uses a heartbeat LAN to monitor the 27 Planning the Fabric Hyper Messaging Protocol HMP 28 nodes in a cluster ServiceGuard cannot use the HyperFabric interconnect as a heartbeat link Instead a separate LAN must be used for the heartbeat For more information on configuring ServiceGuard see Configuring HyperFabric with ServiceGuard on page 87 as well as Managing MC ServiceGuard Part Number B3936 90065 March 2002 Edition High Availability HA Supported When applications use HMP t
52. apter 6 Troubleshooting HyperFabric LED Colors and Their Meanings LED Colors and Their Meanings Listed below are the possible colors and the corresponding meaning of the LEDs on the HyperFabric adapters and switches Adapter LEDs Table 6 1 below shows the names of the LEDs on each HyperFabric adapter Note that the LEDs on the A4920A adapter are labeled but the labels might be hard to see when a cable is connected to the adapter Table 6 1 LED Names by Adapter HyperFabric LED Name Adapter Connected Traffic A4919A Link A4920A A6386A Link A4921A Connected Traffic Error All Some of the LEDs Connected Traffic Link and Link Connected Traffic are equivalent but are labeled differently depending on the adapter So their colors and meanings are the same regardless of the adapter The HyperFabric adapter LED colors and meanings are as follows v Ifthe adapter is not operational the Connected Traffic Link or Link Connected Traffic LED on the adapter is off Some of the possible reasons for this happening are the following The HP 9000 is not operational HyperFabric has not been started on the HP 9000 An adapter is installed in a slot in the HP 9000 but the cable is attached incorrectly or no cable is attached at all Chapter 6 143 Troubleshooting HyperFabric LED Colors and Their Meanings The adapt
53. apter compatibility OLA 48 adapter compatibility OLR 49 checking critical resources 47 planning and preparation for 47 terminology 46 overview of configuration 69 P planning the fabric 11 prerequisites for installation 43 R regulatory specifications 162 relocatable IP addresses 7 configuring 95 S SAM using for configuration 81 102 using for deconfiguring an adapter 83 103 using for starting HyperFabric 110 using for stopping HyperFabric 129 ServiceGuard configuring 87 handling adapter failures 90 software file structure 51 files description of 52 installing 51 steps for 55 starting HyperFabric 176 after OLA 48 using clic_start 110 using SAM 110 statistics displaying 117 status displaying 117 of one switch 119 STM tools for HyperFabric 140 stopping HyperFabric using clic_shutdown 128 using SAM 129 Support Tools Manager See STM switches installing HF2 59 steps for HF2 no rail kit 65 steps for HF2 rail kit 61 LEDs colors and meanings for HF2 148 152 location of for HF2 59 60 regulatory specifications 162 replacing 157 status of one 119 testing for faulty HF2 154 T tracing data dumping 137 formatting 137 level for drivers 137 Transparent Local Failover 28 V verifying communications within fabric 112 viewing man pages 127 VRID 71
54. are file name for each HyperFabric adapter ends in 32c All of the HyperFabric adapters in the fabric must have firmware files that end in 32c for HMP to run If this is not the case run the following commands in the order listed below clic_shutdown clic_init The answer to the interoperability question must be consistent on all nodes in the fabric clic_start All of the IP addresses in the fabric must be in the same subnet The lower 10 digits of all of the IP addresses in the fabric must be unique Every HyperFabric adapter in the fabric must be able to communicate with every other HyperFabric adapter in the fabric Every HyperFabric adapter in the fabric must be connected point to point or via a switch If one of the adapters in the fabric is not connected HMP will not be able to run Run diagnostics to make sure data can be transferred on the HyperFabric adapters Use the following command which is detailed in the next section of this chapter clic diag If you are running an Oracle application using HMP and traffic does not appear to be flowing after completing all of the steps listed above check the ORAHOME rdbms log alert log files The display should show Cluster Interconnect IPC version Oracle using HP HMP logged in the alert log files If not recompile the Oracle application to run using HMP according to the instructions provided in the Oracle Installation and Administration Guide 135 Troublesho
55. ation Availability serere ter as abunmedad eciwieeadole oruedee saa es 26 Features c ee ana lege EOS reped eee ae ad dae GES 27 Configuration Parameters corro euda ara a a EA ehh 29 HMP Supported Configurations llle 33 Point to Point Configuration 0 0 0 cece nns 33 Enterprise Database Configuration 0 cece eee eee eens 35 Technical Computing Work Stations Configuration 004 37 3 Installing HyperFabric Checking HyperFabric Installation Prerequisites lees 43 Installing HyperFabric Adapters 0 00 c ccc een eee eens 44 Contents Online Addition and Replacement 0 0 c cece 44 Planning and Preparation 0 c cece cee eee een enne 47 Critical Resources 335 654 eE A AE A O E EE EP TIRTS 47 Card Compatibility 5 ee DEED r PO EREE EREA PES 47 Installing the Softwaren ione kaa n a a E ee E A A a a 51 File Structure i2 soni a a paT PIED as D EE e Ves 51 Loading the Software 0 0 ene e ene eens 55 Installing HyperFabric Switches 00 0 0 eee eens 57 Betore Installation vs 5 eke Shes to RA ELM Reo EAE SETS 57 Installing the HF2 Switch urr eue er EEEE es 59 With the Rail Kit ocu eDreams gosta ee 61 Without the Rail Kit zs eere EAE KGa E en ee ee a ae 65 4 Configuring HyperFabric Configuration Overview carded we ba eaa Mae eR Dk OMe P RE Ea 69 Information You Need pae a e A ehh 71 Configuration Information Ex
56. bric Resource k Resource Failed Active j i j Package 5 lt HF A Packages p adapter 0 1 failover to FNS Node B HF adapter 1 HF switch 1 Ethernet Port Ethernet Port S Ethernet Heartbeat LAN 0 E S Ethernet Heartbeat LAN 1 S Chapter 4 93 Configuring HyperFabric Configuring HyperFabric with ServiceGuard NOTE 94 Configuring HyperFabric with the ServiceGuard Resource Monitor You can configure the HyperFabric Resource Monitor with ServiceGuard in either of these ways e Editing an ASCII file e Using the SAM GUI For more details please see the manual Using EMS HA Monitors You should configure HyperFabric with ServiceGuard before running the clic_start command or using SAM to start HyperFabric Configuring ServiceGuard with HyperFabric Using the ASCII File When using the ServiceGuard commands for example cmapplyconf to specify the use of the HyperFabric Resource Monitor the section of the package ASCII configuration file that has the keyword RESOURCE_NAME must be uncommented and set to the following values RESOURCE_NAME net interfaces clic status RESOURCE_POLLING_INTERVAL 10 RESOURCE_UP_VALUE UP Configuring ServiceGuard with HyperFabric Using SAM You must perform the following steps when using SAM to configure the HyperFabric Resource Monitor with ServiceGuard sam Clusters High Availability Clusters Cluster Configurat
57. command are shown below e Example 1 To create the global configuration file on the local node issue this command clic init c e Example 2 To display the online help for the clic_init command issue this command clic_init or this command clic_init Chapter 4 IMPORTANT Chapter 4 Step Step Step Step Step Step Configuring HyperFabric Performing the Configuration Using SAM This section describes how to use SAM to configure HyperFabric For information on how to use SAM to configure and deconfigure local failover feature on HMP see Configuring HMP for Transparent Local Failover Support Using SAM on page 102 and Deconfiguring HMP for Local Failover support Using SAM on page 103 If the global configuration file already exists and you are running SAM again to change the file you can keep or modify the existing configuration information in addition to adding new information pertaining to new hardware Also once you ve completed your changes and SAM ends its processing you must stop HyperFabric by running the clic_shutdown command or using SAM and then start HyperFabric by running the clic_start command or using SAM Otherwise your configuration information changes will not take effect See Stopping HyperFabric on page 128 and Starting HyperFabric on page 109 for more information To use SAM to create the global configuration file on an HP 9000 system running
58. cted For example if the SCSI controller is connected to the unmirrored root disk or swap space the system will crash when the SCSI controller is shut down During an OLAR procedure it is essential to check the targeted card for critical resources as well as the effects of existing disk mirrors and other situations where a card s functions can be taken over by another card that will not be affected As mentioned earlier SAM performs a thorough CRA automatically and presents options based on its findings If it is determined that critical resources will be affected by the OLAR procedure the card could be replaced when the system is offline If action must be taken immediately an online addition of a backup card and deletion of the target card could be attempted using the rad command Card Compatibility This section explains card compatibility considerations for doing OLAR 47 Installing HyperFabric Installing HyperFabric Adapters CAUTION 48 Online Addition OLA Multiple cards can be added at the same time When adding a card online the first issue to resolve is whether the new card is compatible with the system Each OLAR capable PCI slot provides a set amount of power The replacement card cannot require more power than there is available The card must also operate at the slot s bus frequency A PCI card must run at any frequency lower than its maximum capability but a card that could operate at only 33 MHz w
59. cted to the fabric 7 Run the following command opt clic bin clic stat dALL If a TCP UDP IP application is running Check the firmware file field to make sure the same version of firmware is downloaded on all of the HyperFabric adapters in the cluster If this is not the case run the following commands in the order listed below clic shutdown clic init The answer to the interoperability question must be consistent on all nodes in the fabric clic start Check to make sure all IP addresses have been assigned Check the subnet Every HyperFabric adapter in the fabric must be able to communicate with every other HyperFabric adapter Every HyperFabric adapter in the fabric must be connected point to point or via a switch Run diagnostics to make sure data can be transferred on the HyperFabric adapters Use the following command which is detailed in the next section of this chapter clic diag If the HyperFabric subsystem is still not usable contact your HP support representative with the diagnostics data generated using the clic diag command If an HMP application is running 134 Chapter 6 Chapter 6 Troubleshooting HyperFabric Running Diagnostics Make sure 4X HyperFabric adapters are installed on the nodes The adapter type field indicates the type of HyperFabric adapter that is installed HMP will only run on 4X HF2 PCI A6386A adapters Check the firmware file field to make sure the firmw
60. d by qualified HP personnel only If both Power LEDs are off the enclosure the power supply backplane and fan monitoring board must be replaced by qualified HP personnel only Check the switch s Ethernet ports a If the port is connected to an operational Ethernet network the ports LED Ethernet Port Main or Ethernet Port Aux shows as solid or blinking green If the port is disconnected or connected to an Ethernet network that is not operational the port s LED is off Chapter 6 Troubleshooting HyperFabric Determining Whether a Switch is Faulty Step 4 Check the ports on the cards switch module a Disconnect the cable from the card switch module port you suspect is faulty b Attach a loopback plug to the relevant port e Ifthe port is on the integrated 8 port fiber card or the A6388A HF2 8 port fiber switch module in the expansion slot use a fiber loopback plug A fiber loopback plug HP part number A6384 67004 is shipped with each HF2 switch c Determine if the port is faulty e Ifthe port is functioning correctly the Port x LED shows as solid or blinking green e Ifthe port is faulty the Port x LED is off Repeat steps a through c for each port on the cards switch module to determine if any of them are faulty d Ifthe port is faulty do not use it Instead use a known good port on that card switch module if one is available If no good ports are available
61. d endpoint ID Invalid endpoint protection key Interleaved gathered receive Interleaved multi frame bulk messages NQ overflow Send NQ overflow Invalid slot key for NQ credit update DLPI QOS receive buffer shortage Link congestion events Max send packet retry exceeded Link or switch failure events Link or switch resume events Bad route detected Bad optional data length Invalid message received CRC error Invalid CRC word Bad frame length Receive buffer overflow Null packets generated on link reset Firmware reset notification Data corruption notification Unsupported QOS message received Invalid HMP VC ID Invalid HMP endpoint ID Invalid HMP endpoint protection key HMP message order violation Packet drops Transmit side congestion events Receive side congestion events Other Misc Statistics Mapping message send failures 0 OF 0 C e Oo OC OUR O C O SO ec OGO CO OCC OO X O e Example 4 If the local node is bent1y6 and you want to display the VRIDs IP addresses switch hopcounts and physical routes for each HyperFabric adapter in bent1y6 issue this command clic stat d VRID The generated output if the nodes are connected in a Point to Point configuration could look like this Chapter 5 123 Managing HyperFabric Displaying Status and Statistics Date Sat Aug 5 16 08 12 2000 Node bently6 corp4 com Adapter clic0 VRID IP Address Switch Hopco
62. d monitors might be supplied by the developers of other components The HyperFabric monitor is supplied with the HyperFabric product and is installed with it To use the HyperFabric monitor with ServiceGuard you configure the monitor as an ServiceGuard package dependency Although HyperFabric can be used by an application within a package to communicate with other nodes it is not possible to use HyperFabric as a heartbeat LAN So in a package control script do not specify HyperFabric IPs subnets in the lines that contain the keywords IP n and SUBNET 1 Also cmqueryc1 will not discover and report HyperFabric IPs and subnets After you have configured HyperFabric as a package dependency ServiceGuard s package manager calls the Event Monitoring Service EMS to launch an external monitor for HyperFabric The package will not start unless the monitor reports that HyperFabric is available and the package will fail when HyperFabric s status is DOWN that is when all HyperFabric adapters on a node become non functional Complete instructions for configuring ServiceGuard clusters and packages are provided in the manual Managing MC ServiceGuard 87 Configuring HyperFabric Configuring HyperFabric with ServiceGuard NOTE 88 Figure 4 2 below shows a HyperFabric switch configuration with ServiceGuard This example shows a four node configuration with two HyperFabric switches and redundant heartbeat Ethernet LANs
63. d s output to find the ID of the switch you are interested in e displays the online help for clic stat If you do not specify any of the above parameters the current fabric map is displayed showing the last known status of the components Examples of clic stat Some examples of using clic stat are as follows Example 1 If the local node is bent1y7 and you want to enable DLPI driver statistics gathering issue this command clic stat p TCP The generated output could look like this Date Sat Aug 5 16 08 14 2000 Node bently7 corp2 com Performance Statistics Levels Possible perf degradation occurring DLPI performance statistics enabled e Example 2 Chapter 5 119 Managing HyperFabric Displaying Status and Statistics If the local node is bent1y7 and you want to display the management daemon clic_mgmtd configuration and statistics data issue this command clic_stat d CFG The generated output could look like this Date Sat Aug 5 16 08 12 2000 Node bently7 corp2 com CLIC Management Global Status Statistics Current Component Versions CLIC Management process version 1 0 CLIC Management API version 1 40 CLIC Driver version 10 Global Management Statistics Node failures Nodes active Command session failures Command active sessions Command total sessions Management mesh session failures Management mesh active sessions D O Or H S O O Management G
64. ded host name and any alias names For example IP address bently6 corp3 com bently6 IP address bently4 corp7 com testi IP address bently2 corp4 com test3 The clic probe Command Run the clic probe command to send 256 byte packets to verify the link out to and back from a specific destination optionally using a specific adapter for the verification The destination can be either a node or a switch if a switch is part of the fabric If you include opt clic binin your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic probe You do not have to be logged in as root to run this command The syntax is as follows clic probe node name c adapter ID c adapter ID r VRID switch hopcount 1 c adapter ID s c adapter ID p packet count Note that some of the lines in the above syntax are indented for readability purposes only When you actually type the command you do not indent anything Chapter 5 Managing HyperFabric Verifying Communications within the Fabric The command parameters are as follows e node name specifies the node you want to verify This value is conditionally required you must specify it when you are verifying traffic to a remote node unless you use the r parameter described below e c specifies that you want to use the adapter identified by adapter IDfor the verification
65. dition a hybrid switch that has 8 fiber ports is available to support HF2 clusters This section describes the various HyperFabric products For more information on HP 9000 systems that support HyperFabric products see the HyperFabric Release Notes available at http docs hp com hpux netcom index html HyperFabric This document uses the term HyperFabric HF to refer to the hardware and software that form the HyperFabric cluster interconnect product The term HyperFabric2 HF2 refers to the following fiber based hardware components e The A6386A adapter e The A6384A switch chassis e The A6388A and A6389A switch modules Although the A6389A switch module has 4 copper ports it is still considered an HF2 component because it can only be used with the A6384A HF2 switch chassis e The C7524A C7525A C7526A and C7527A cables HyperFabric Adapters The HyperFabric adapters are as follows e A6386A HF2 PCI 4X adapter with a fiber interface The A6092A HyperFabric adapter is supported beginning with the following HyperFabric software versions Chapter 1 NOTE Chapter 1 Overview of the HyperFabric Product HyperFabric Products e HP UX 11 0 HyperFabric software version B 11 00 09 e HP UX 11i v1 HyperFabric software version B 11 11 00 e HP UX 11i v2 HyperFabric software version B 11 23 00 The A6386A HyperFabric2 adapter is supported beginning with the following HyperFabric software versions e HP UX 11 0 Hyp
66. e CLIC stands for CLuster InterConnect and it is used to differentiate those HyperFabric commands or components from other commands or components For example the HyperFabric command clic_init is different from the HP UX init command 8 Chapter 1 2 Planning the Fabric This chapter contains the following sections that include general guidelines and protocol specific considerations for planning HyperFabric clusters that run TCP UDP IP or HMP applications Chapter 2 9 Planning the Fabric e Preliminary Considerations on page 11 e HyperFabric Features Parameters and Supported Configurations for TCP UDP IP and HMP Applications on page 13 e TCP UDP IP on page 14 e Hyper Messaging Protocol HMP on page 26 10 Chapter 2 Chapter 2 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Planning the Fabric Preliminary Considerations Preliminary Considerations Before assembling a fabric physically do the following to address all of the appropriate issues Read Chapter 1 Overview of the HyperFabric Product on page 1 to get a basic understanding of HyperFabric and its components Read this chapter Planning the Fabric to gain an understanding of protocol specific configuration guidelines for TCP UDP IP and HMP applications Read Configuration Overview on page 69 Information You Need on page 71 and Configuration Information
67. e r specifies that VRID switch hopcount is the routing information for the adapter To determine the VRIDand switch_hopcount to specify first run the clic stat d VRID command see The clic_stat Command on page 117 Note that if you specify this parameter r VRID switch hopcount you must also specify the c adapter_ID parameter described above e 1 specifies that you want to do local loopback testing on a particular adapter Note that if you specify this parameter 1 you must also specify the c adapter_ID parameter described above e s specifies that you want to loopback at the switch port attached to a particular adapter Note that if you specify this parameter s you must also specify the c adapter_ID parameter described above e pspecifies that you want to send packet count number of 256 byte packets packet_count can be any positive integer This parameter is useful for building scripts for debugging or for hardware verification If you do not specify this parameter one packet is sent each second until you stop the command with a CTRL C e displays the online help for clic probe If you do not specify any of the above parameters the online help for clic probe is displayed Chapter 5 113 Managing HyperFabric Verifying Communications within the Fabric NOTE 114 Also see the clic_diag command to Probe a specific remote node Dump and format trace data Set the tracing level for the
68. e of any particular HyperFabric configuration For more information on specific HyperFabric configurations for HMP applications see HMP Supported Configurations on page 33 e HyperFabric is supported on the HP 9000 series servers and workstations only e HMP is supported on the HF2 adapter A6386A only e The performance advantages that HMP offers are not completely realized unless HMP is used with A6386A HF2 fiber adapters and related fiber hardware See Table 2 2 on page 20 for details The local failover configuration of HMP is supported only on the A6386A HF2 adapters e Maximum Supported Nodes and Adapter Cards 29 Planning the Fabric Hyper Messaging Protocol HMP 30 HyperFabric clusters running HMP applications are limited to supporting a maximum of 64 adapter cards However in local failover configurations a maximum of only 52 adapters are supported In point to point configurations running HMP applications the complexity and performance limitations of having a large number of nodes in a cluster make it necessary to include switches in the fabric Typically point to point configurations consist of only 2 or 3 nodes In switched configurations running HMP applications HyperFabric supports a maximum of 64 interconnected adapter cards A maximum of 8 HyperFabric adapter cards are supported per instance of the HP UX operating system The actual number of adapter cards a particular node is able to accommodate
69. eature see Online Addition and Replacement on page 44 and Configuring HP UX for Peripherals Part Number B2355 90698 November 2000 Edition Event Monitoring Service EMS Supported In the HyperFabric version B 11 23 01 the HyperFabric EMS monitor enables the system administrator to separately monitor each HyperFabric adapter on every node in the fabric in addition to monitoring the entire HyperFabric subsystem The monitor can inform the user if the resource being monitored is UP or DOWN The administrator defines the condition to trigger a notification usually a change in interface status Notification can be accomplished with one of the following ASimple Network Management Protocol SNMP trap Chapter 2 Chapter 2 Planning the Fabric TCP UDP IP Logging into a user specified log file with a choice of severity Email to a user defined email address For more information on EMS including instructions for implementing this feature see Configuring the HyperFabric EMS Monitor on page 85 and the EMS Hardware Monitors User s Guide Part Number B6191 90028 September 2001 Edition ServiceGuard Supported Within a cluster ServiceGuard groups application services individual HP UX processes into packages In the event of a single service failure node network or other resource EMS provides notification and ServiceGuard transfers control of the package to another node in the cluster allowing services to rema
70. ed with each HF2 switch Confirm that necessary tools are available to install the HyperFabric switch mounting hardware In addition check the HP 9000 system s documentation to determine if additional tools may be required for component installation Confirm that the software media is correct Create a map of the fabric optional Confirm that HP UX super user privileges are available they are necessary to complete the HyperFabric installation The first HyperFabric installation step is installing HyperFabric adapter cards in the nodes 43 Installing HyperFabric Installing HyperFabric Adapters CAUTION 44 Installing HyperFabric Adapters This section contains information about installing HyperFabric adapters in HP 9000 systems Online Addition and Replacement OLAR information is provided in the section Online Addition and Replacement on page 44 HyperFabric adapters contain electronic components that can easily be damaged by small amount of electricity To avoid damage follow these guidelines e Store adapters in their antistatic plastic bags until installation e Work in a static free area if possible e Handle adapters by the edges only Do not touch electronic components or electrical traces e Use the disposable grounding wrist strap provided with each adapter Follow the instructions included with the grounding strap e Use a suitable ground any exposed metal surface on the computer chass
71. een Nodes 00 cece eee eee eee e eens 36 Figure 2 8 Technical Computing Configuration 0 0 0 0 c cee eens 38 Figure 2 9 Large Technical Computing Configuration 000005 39 Figure 3 1 HyperFabric File Structure 0 0 0 0 cc cece leere 51 Figure 3 2 Front of HF2 Switch A6388A Switch Module Installed 59 Figure 3 3 Front of HF2 Switch A6389A Switch Module Installed 60 Figure 4 1 Map for Configuration Information Example 74 Figure 4 2 An ServiceGuard Configuration with Two HyperFabric Switches 89 Figure 4 3 Node with Two Active HyperFabric Adapters 000005 91 Figure 4 4 Node with One Failed HyperFabric Adapter 00 92 Figure 4 5 When All HyperFabric Adapters Fail 0 0 00 cee eens 93 Figure 4 6 A Configuration supporting Local Failover 0005 98 Figure 4 7 Adapter Link or Switch Port Failover 0 0 0 0 00 cee ene 99 Figure 4 8 Switch Failover 2 ausus ex E A Ee A eS 100 Figure 4 9 Cable Failover Between Two Switches 00000 e eae 101 Figure 4 10 Configuring the Transparent Local Failover feature 104 vii Figures viii Tables Table 1 HP UX 111 Releases ccs a eco Wea eke eda kate oat REA Ro xii Table 2 Publishing History Details cc ee ee eee xii Table 3 Organization 0 0 0 eee a es xiii Table 2 1 HF2 Throughpu
72. er is bad The cable is bad The switch port is bad if the adapter is connected to a switch Note that if a switch port is bad and for some reason you cannot use a different port on the switch you must replace the switch module in the HF2 switch whichever is applicable However you first should try turning the switch s power off and then back on The adapter is connected to a non operational adapter in the remote node in a node to node configuration Y Ifthe connection from the adapter to the corresponding switch port if a switch is used or the corresponding adapter in the remote node in a node to node configuration is operational the Connected Traffic Link or Link Connected Traffic LED on the adapter shows as solid green V Ifdatais flowing between the adapter and the switch port if a switch is used or the corresponding adapter in the remote node in a node to node configuration the Connected Traffic Link or Link Connected Traffic LED shows as flashing green Note that data does not start to flow until HyperFabric initialization has occurred see Using the clic_init Command on page 79 V Ifthe adapter is in an error state that requires it to be replaced the Error LED on the adapter shows as solid yellow See Replacing a HyperFabric Adapter on page 156 if you need to replace an adapter Table 6 2 below summarizes the adapter LED information in a table forma
73. erFabric software version B 11 00 11 e HP UX 11i v1 HyperFabric software version B 11 11 01 e HP UX 11i v2 HyperFabric software version B 11 23 00 Switches and Switch Modules The HyperFabric2 switches are as follows e A6384A HF2 fiber switch chassis with one integrated Ethernet management LAN adapter card one integrated 8 port fiber card and one expansion slot For the chassis to be a functional switch install one of the following switch modules in the expansion slot The A6388A HF2 8 port fiber switch module This gives 16 fiber ports to the switch 8 from the integrated fiber card and 8 from the A6388A switch module The A6389A HF2 4 port copper switch module This gives 12 ports to the switch a mixture of 8 fiber ports from the integrated fiber card and 4 copper ports from the A6389A module The A6384A HF2 switch chassis with either module installed is supported beginning with the following HyperFabric software versions e HP UX 11 0 HyperFabric software version B 11 00 11 e HP UX 11i v1 HyperFabric software version B 11 11 01 e HP UX 11i v2 HyperFabric software version B 11 23 00 In this manual the terms HyperFabric2 switch or HF2 switch refer to the functional switch the A6384A switch chassis with one of the switch modules installed Overview of the HyperFabric Product HyperFabric Products IMPORTANT HF 2 adapters and switches are not supported by software versions earlier than those listed in
74. et as Ethernet ports of the HyperFabric switches if you are going to enable switch management A node might have multiple LAN interfaces HP recommends that you do not enable switch management You can create the global configuration file by either running the clic init command or using SAM to configure each HyperFabric adapter The clic init command and SAM also place the necessary entries into the following three files e The system etc rc config d netconf file 69 Configuring HyperFabric Configuration Overview IMPORTANT In this file clic init and SAM add some HyperFabric related lines that end with the characters clic These lines are used by the HyperFabric software and are not comments so do not remove them from the file e The system etc rc config d clic global conf file e The etc rarpd conf Reverse Address Resolution Protocol RARP support file This file is used in the management of the HyperFabric switches if you are going to enable switch management The clic init command is described in Using the clic init Command on page 79 Using SAM to configure an adapter is described in Using SAM on page 81 After you have used the c1ic init command or SAM you can configure HyperFabric with ServiceGuard if necessary For more information see Configuring HyperFabric with ServiceGuard on page 87 You can configure card pairs for the Transparent Local Failover Feature of HMP available wi
75. figure HyperFabric in the local failover mode applicable to applications using HMP The clic_stat command can also be used to enable or disable performance statistics gathering for the DLPI driver the firmware and HMP All of the statistics that can be displayed using the clic_stat command are documented in the clic_stat 1M man page on HP UX 11 0 and on later HP UX releases If you include opt clic bin in your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic_stat To use some of this command s parameters you must be logged in as root see each parameter s description below The syntax is as follows clic_stat p perf_level d display_level c adapter_ID n nodename 117 Managing HyperFabric Displaying Status and Statistics 118 Note that the second line in the above syntax is indented for readability purposes only When the command is typed there should not be any indentation The command parameters are as follows p enables disables performance statistics gathering according to the value of perf_level which is one of the following TCP Enables DLPI driver statistics when under the TCP IP stack HMP Enables Hyper Messaging Protocol HMP statistics gathering FW Enables firmware statistics gathering RST Disables resets the DLPI driver and HMP statistics Note that turn
76. following are the two families of applications that can use HMP over the HyperFabric interface e Oracle 9i Database Release 1 9 0 1 and Release 2 9 2 0 1 0 HMP has been certified on Oracle 9i Database Release 1 with HP UX 11 0 11i v1 and 11i v2 HMP has been certified on Oracle 9i Database Release 2 with HP UX 11 0 11i v1 and 11i v2 e Technical Computing Applications that use the HP Message Passing Interface HP MPI Chapter 2 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP HP MPI is a native implementation of version 1 2 of the Message Passing Interface Standard It has become the industry standard for distributed technical applications and is supported on most technical computing platforms Features The following are the HyperFabric features on HMP OnLine Addition and Replacement OLAR The OLAR feature which allows the replacement or addition of HyperFabric adapter cards while the system node is running is supported when applications use HMP to communicate Event Monitoring Service EMS Supported In the HyperFabric version B 11 23 01 the HyperFabric EMS monitor enables the system administrator to separately monitor each HyperFabric adapter on every node in the fabric in addition to monitoring the entire HyperFabric subsystem The monitor can inform the user if the resource being monitored is UP or DOWN The administrator defines the condition to trigger a notification usually a ch
77. g ServiceGuard for HyperFabric Relocatable IP Addresses on page 95 When the monitor for HyperFabric detects a failure and the backup adapter takes over the relocatable IP address is transparently migrated to the backup adapter Throughout this migration process the client application continues to execute normally When you start HyperFabric with the clic_start command through SAM or by booting the HP 9000 system you start the management process This process must be active for HyperFabric to run If the HyperFabric management process on a node stops running for some Overview of the HyperFabric Product HyperFabric Concepts reason for example if it is killed all HyperFabric related communications on that node are stopped immediately This makes the node unreachable by other components in the fabric When you start HyperFabric the fabric is verified automatically This is because each node performs a self diagnosis and verification over each adapter installed in the node In addition the management process performs automatic routing and configuring for each switch if switches are part of the fabric You can if needed run the clic_stat command to get a textual map of the fabric which can be used as another method of quick verification Notice that the commands to administer HyperFabric have a prefix of clic_ and some of the other components have CLIC as part of their name for example the CLIC firmware and the CLIC softwar
78. ght laws Trademark Notices Oracle is a registered trademark of Oracle Corporation UNIXO is a registered trademark in the United States and other countries licensed exclusively through The Open Group Contents About This Document 1 Overview of the HyperFabric Product About HyperFabtric e Deer t euge crea ce ra du e EE e 3 HyperFabtic Products 2 ce Re eR RI s Ebr e Et ule 4 HyperPFabric Adapters 522 n nates eS ee E qa ES d RE P 4 Switches and Switch Modules 0 cece ccc teen ene eens 5 Other Product Elements cure rr ERR Gv bee ketene 6 HyperFabric Concepts oenas sci eda a cae rer b aa une dak sd ps ede 7 2 Planning the Fabric Preliminary Considerations siey riera Er EE EE IE EEE ANA rae ri 11 HyperFabric Features Parameters and Supported Configurations for TCP UDP IP and HMP Applications einda e ie a E E A E a a as 13 TGP UDF IP e E a pr ite nel d e e Seea ia he ER red i et od 14 Application Availability l lslceeleeeeee eee 14 lou PM MI PP IM 14 Configuration Parameters 0 c cece ehh 17 TCP UDP IP Supported Configurations suasanae sauer 21 Point to Point Configurations 0 0 ccc cece ene enn 21 Switched Configuration 0 000 eee pa G 23 High Availability Switched Configuration 0 0c cece eee eens 24 Hybrid Configuration 0 0 0 ehh 25 Hyper Messaging Protocol HMP 0 0 ccc eee eee ene eens 26 Applic
79. he network software controls data transfer over the fabric The HyperFabric configuration contains two or more HP 9000 systems and optional HyperFabric switches Each HP 9000 acts as a node in the configuration Each node has a minimum of one and a maximum of eight HyperFabric adapters installed in it For information on the maximum number of adapters that can be installed in each system see Chapter 2 Planning the Fabric on page 9 Each HF2 switch can be configured with 12 or 16 ports HyperFabric supports a maximum of 4 HyperFabric switches You can mesh HyperFabric switches and configurations with up to four levels of meshed switches are supported You can plan a HyperFabric cluster as a High Availability HA configuration when it is necessary to ensure that each node can always participate in the fabric This is done by using ServiceGuard earlier known as MC ServiceGuard ServiceGuard OPS Edition earlier known as MC LockManager and the Event Monitoring Service EMS Configurations of up to 8 nodes are supported under ServiceGuard Beginning with HyperFabric software versions B 11 00 05 and B 11 11 00 you can use relocatable IP addresses as part of an HA configuration Relocatable IP addresses permit a client application to reroute through an adapter on a remote node allowing that application to continue processing without interruption The rerouting is transparent This function is associated with ServiceGuard see Configurin
80. help for clic diag is displayed Example of clic diag An example of the clic diag command is shown below If the local node is bent1y6 and you want to confirm that all of the adapters on bently6 are communicating with the target adapters on bently8 issue this command clic diag r bently8 The generated output could look like this CLIC PROBE 256 byte packets Source adapter id bently6 corp4 com clicO Target adapter id bently8 corp2 com clicl 256 bytes seq num 0 Packet Acknowledged 256 bytes seq num 1 Packet Acknowledged 256 bytes seq num 2 Packet Acknowledged 256 bytes seq num 3 Packet Acknowledged 256 bytes seq num 4 Packet Acknowledged artes CLIC PROBE Statistics 5 packets transmitted 5 packets received 0 packet loss CLIC PROBE 256 byte packets 138 Chapter 6 Chapter 6 Troubleshooting HyperFabric Running Diagnostics Source adapter id bently6 corp4 com clicl Target adapter id bently8 corp2 com clic3 256 bytes 256 bytes 256 bytes 256 bytes 256 bytes seq_num seq_num seq_num seq_num seq_num CLIC_PROBE 5 packets transmitted 0x Packet Acknowledged d Packet Acknowledged 2a Packet Acknowledged E Packet Acknowledged 4 Packet Acknowledged Statistics 5 packets received 0 packet loss 139 Troubleshooting HyperFabric Using Support Tools Manager Using Support Tools Manager Use Support Tools Manager STM with Hyper
81. hen you actually type the command you do not indent anything The command parameters are as follows Chapter 6 Chapter 6 Troubleshooting HyperFabric Running Diagnostics r specifies that you want to probe a specific remote node identified by remote_node_name The probe is done on all operational routes to the remote node T specifies that you want to set the tracing level for the HyperFabric software and the firmware to trace_level which can be one or more of the following you can specify either the keyword or the hexadecimal value reset or 0x0 Reset tracing to the default level for all components that is turn off all tracing send_path or 0x0002 Trace the HyperFabric software send path packet_data or 0x0004 Trace the packet data packet_hdr or 0x0008 Trace the protocol headers TCP UDP IP data_struct or 0x0010 Trace the HyperFabric software CLIC specific data structures and events rcv path or 0x0020 Trace the HyperFabric software receive path control or 0x0040 Trace the HyperFabric software control messages allor Oxffff Turn on all possible trace levels Note that this will have a severe impact on performance To find out what the current tracing level is run this parameter T without any keyword or hexadecimal value specified D specifies that you want to dump the trace buffers from the kernel to a user space file and format the data using the formatter TCP_IP The HyperFabric software trace
82. ic_init nor the fields in SAM but merely an example of the flow of information input In addition you should not try to use the dummy addresses in your actual configuration On node A 1 2 How many HyperFabric adapters are installed on the node Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B 11 00 11 or B 11 11 01 3 What is the IP address of the first adapter clic0 192 0 0 1 What is the subnet mask of the first adapter 255 255 255 0 If you do not specify a value for this a default mask is chosen You will most likely just accept the default However in this example we are showing a value for the subnet mask just to illustrate the correlation between the dummy information in Figure 4 1 and where that information is specified or generated during clic_init and SAM What is the IP address of the second adapter c1ic1 192 0 8 3 6 What is the subnet mask of the second adapter 255 255 225 0 11 Chapter 4 Do you want to enable switch management You cannot enable switch management through SAM you must use the clic init command As stated earlier HP recommends that you do not enable switch management However if you do enable it you must provide the information in items 8 through 14 If switch management has been enabled how many switches will be configured As stated earlier HP recommends that you do not enable switch management
83. ics T Chapter 6 Confirm HyperFabric adapters are installed on each node Use the following command to display a list of HyperFabric adapters installed each node ioscan funC clic Check to see if HyperFabric software is installed Use the following command to display the HyperFabric software version installed on each node swlist grep i hyperfabric Check the patch level Use the following command to display the list of patches installed on each node swlist Refer to the HyperFabric Release Notes to determine which patches need to be installed Make sure the HyperFabric subsytem is started Use the following command to determine if the HyperFabric subsystem is running on the node ps ef grep clic If the HyperFabric daemon clic mgmtd is not running start the HyperFabric subsystem by executing the following command opt clic bin clic start Make sure the correct IP addresses are assigned to the HyperFabric adapters Use the following command to display all of the network interfaces and the IP addresses assigned to them netstat in If an IP address is not assigned to a HyperFabric adapter execute the following commands in the order listed clic shutdown clic init 133 Troubleshooting HyperFabric Running Diagnostics clic start All of these commands reside in the opt clic bin directory 6 Check cabling to make sure all of the HyperFabric adapters are conne
84. ies the following e If not all the adapters are configured e Ifthe number of adapters configured are odd e If Interoperability Enabled is set to YES If any of the above conditions is true then SAM displays an appropriate error message Otherwise the Configure Local Failover for HMP window pops up In this window specify the configured adapters for each card pair and press OK On pressing OK SAM verifies the following e If HyperFabric subsystem is not running on the machine e fall card pairs are not configured e If you have chosen the same adapter for different card pairs If any of the above conditions is true SAM displays an appropriate error message Otherwise the etc rc config d clic global conf file is updated with information about the configured card pairs If Card Pair 0 comprises Chapter 4 NOTE Chapter 4 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support of adapters clic0 and clicl and if the Card Pair 1 comprises of adapters clic2 and clic3 then the following entries are added to the clic_global_conf file CARD PAIR 0 clic0O clicl CARD PATR 1 clic2 clic3 If you press Cancel you remain in the main HyperFabric Configuration screen If you press Help help text for this task appears Step 6 Exit SAM Step Step Step Step Step To view the card pair information from the etc rc config d clic global conf file se
85. igured Highlight the adapter you want to deconfigure Pull down the Actions menu and select Deconfigure Adapter In the pop up window if you want to deconfigure the adapter select OK to confirm it If you do not want to deconfigure the adapter select Cancel If you selected OK the entry for the adapter is deleted from the HyperFabric configuration files etc rc config d clic global conf and etc rc config d netconf If you selected Cancel you remain in the main HyperFabric Configuration screen Exit SAM If you have configured HMP for Transparent Local Failover support and if you select Deconfigure Adapter HyperFabric will verify if the selected adapter is configured to be part of any card pair If yes the user 83 Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM is informed and the card pair entry is removed from the etc rc config d netconf and etc rc config d clic global conf files 84 Chapter 4 Configuring HyperFabric Configuring the HyperFabric EMS Monitor Configuring the HyperFabric EMS Monitor In the HyperFabric version B 11 23 01 the HyperFabric Event Monitoring Service EMS monitor allows system administrators to separately monitor each HyperFabric adapter on every node in the fabric in addition to monitoring the entire HyperFabric subsystem The monitor can inform the user if the resource being monitored is UP or DOWN The administrator defi
86. iguring HMP for Transparent Local Failover Support on page 96 Chapter 2 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP Technical Computing Work Stations Configuration This configuration is typically used to run technical computing applications with HP MPI A large number of small nodes are interconnected to achieve high throughput see Figure 2 8 High availability is not usually a requirement in technical computing environments HMP provides the high performance low latency path necessary for these technical computing applications You can interconnect up to 56 nodes using HP 16 port switches You cannot link more than four 16 port switches in a single cluster see Figure 2 9 The HP J B and C class workstations provide excellent performance and return on investment in technical computing configurations 37 Planning the Fabric Hyper Messaging Protocol HMP Figure 2 8 Technical Computing Configuration L1 Tw 38 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP Figure 2 9 Large Technical Computing Configuration E LLL a LLL LLLI LLL LLL E E D p p 23 LE LL nn LI E mn LI m m p m SW SW m La C m oooo SW oooo SW g SEA ER g m g Edt LJ a fma ey Lm Chapter 2 39 Planning the Fabric Hyper Messaging Protocol HMP 40 Chapter 2 Chapter 3 Installing HyperFabric This chapter contains the follo
87. in an error condition Adapter is in an error condition Notes Adapter should be operating normally You must replace the adapter Chapter 6 Troubleshooting HyperFabric LED Colors and Their Meanings Chapter 6 147 Troubleshooting HyperFabric LED Colors and Their Meanings HF2 Switch LEDs The HF2 switch LED colors and meanings are explained below m For each Status LED on the switch v v v If the card switch module is not operating the LED is off For the integrated Ethernet management LAN adapter card in the top slot and the integrated 8 port fiber card in the middle slot the card can be safely removed by qualified HP personnel only For an A6388A HF2 8 port fiber or A6389A HF2 4 port copper switch module in the expansion slot the bottom slot the switch module can be safely removed by you or qualified HP personnel If a fault is occurring on the card switch module the LED shows as solid yellow If the card switch module passed the self test and is operating the LED shows as solid green m For the Power A and Power B LEDs on the switch 4 r4 If the power to the switch is off the LEDs are off If the power to the two redundant power buses is on the LEDs show as solid green Note that if only one of the Power LEDs shows as solid green the switch is still operational Q For the Ethernet Port Main and Ethernet Port Aux LEDs on the
88. in available with minimal interruption ServiceGuard via EMS directly monitors cluster nodes LAN interfaces and services the individual processes within an application ServiceGuard uses a heartbeat LAN to monitor the nodes in a cluster ServiceGuard cannot use the HyperFabric interconnect as a heartbeat LAN Instead use a separate LAN for the heartbeat For more information on configuring ServiceGuard see Configuring HyperFabric with ServiceGuard on page 87 and Managing MC ServiceGuard Part Number B3936 90065 March 2002 Edition High Availability HA Supported To create a highly available HyperFabric cluster there should not be any single point of failure Once the HP 9000 nodes and the HyperFabric hardware have been configured with no single point of failure ServiceGuard and EMS can be configured to monitor and fail over nodes and services using ServiceGuard packages If any HyperFabric resource in a cluster fails adapter card cable or switch port the HyperFabric driver transparently routes traffic over other available HyperFabric resources with no disruption of service The ability of the HyperFabric driver to transparently fail over traffic reduces the complexity of configuring highly available clusters with ServiceGuard because ServiceGuard has to take care of node and service failover only 15 Planning the Fabric TCP UDP IP 16 ServiceGuard uses a heartbeat to monitor the cluster The HyperFabric l
89. ing on performance statistics gathering will affect the performance of HyperFabric by increasing CPU usage and message latency the time it takes a message to get from one point to another It might also decrease the throughput of data HP recommends that you do not specify this parameter unless you are trying to troubleshoot HyperFabric problems You must be logged in as root to use this parameter d specifies that you want to set the level of data displayed to display_level which is one of the following NET Displays fabric component statistics CFG Displays the management daemon clic_mgmtd configuration and statistics data VRID Displays the virtual route identifier information the VRIDs IP addresses switch hopcounts and physical routes for each HyperFabric adapter in the local node ALL Displays all available status data Chapter 5 Managing HyperFabric Displaying Status and Statistics e c displays the statistics of the adapter identified by adapter ID e n displays statistics for the node identified by nodename e s specifies that you want to get the status of the switch identified by switch_ID assigned by clic init or SAM Note that this parameter is meaningful only if you enabled switch management through the clic init command To determine the switch ID runthe clic stat command without specifying any parameters This displays each switch ID known to the system Look through the comman
90. inks cannot be used for the heartbeat Instead an alternate LAN connection such as 100BaseT Ethernet Token Ring or FDDI must be made between the nodes for use as a heartbeat link End To End HA HyperFabric provides end to end HA on the entire cluster fabric at the link level If any of the available routes in the fabric fails HyperFabric transparently redirects all the traffic to a functional route and if configured notifies ServiceGuard or other enterprise management tools Active Active HA In configurations where there are multiple routes between nodes the HyperFabric software uses a hashing function to determine an adapter or a route through which it sends messages This is done on a message by message basis All of the available HyperFabric resources in the fabric are used for communication In contrast to Active Passive HA where one set of resources is not utilized until another set fails Active Active HA provides the best return on investment because all of the resources are utilized simultaneously ServiceGuard is not required for Active Active HA operation For more information on setting up HA HyperFabric clusters see Figure 2 3 TCP UDP IP High Availability Switched Configuration on page 24 Dynamic Resource Utilization DRU Supported If you add a new resource node adapter cable or switch to a cluster the HyperFabric subsystem dynamically identifies the added resource and starts using it The same process
91. ion go through all the steps to create cluster Package Configuration Create Add Package if creating new packages Chapter 4 Configuring HyperFabric Configuring HyperFabric with ServiceGuard Specify Package Name and Nodes Specify Package SUBNET Address Specify Package Services Specify Package Failover Options Specify Package Control Script Location Specify Package Control Script Information Specify Package Resources Dependencies Add Resource Name Navigate the Resource Subclass by double clicking on net until net interfaces clic status shows up in the selection box Resource Name then select it and click OK Resource Parameters Input the Resource Polling Interval for example 10 seconds Select UP from the Available Resource Values and click Add Click OK to accept the values Configuring ServiceGuard for HyperFabric Relocatable IP Addresses If you are using HyperFabric version B 11 00 05 B 11 11 00 or later and you want to use relocatable IP addresses configure the relocatable IP addresses with the IP n command in the package control script For example to configure the relocatable address 192 0 0 3 for adapter 0 and 192 0 8 5 for adapter 1 specify this IP O 192 0 0 3 IP 1 192 0 8 5 Chapter 4 95 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Configuring HMP for Transparent Local Failover Support HMP supports Local Failover in
92. is For specific instructions see system specific documentation on installing networking adapters for each type of HP 9000 system that HyperFabric adapters will be installed into When the HyperFabric adapters have been installed go to Installing the Software on page 51 Online Addition and Replacement Online Addition and Replacement OLAR allows PCI I O cards adapters or controllers to be replaced or added to HP 9000 systems without the need for completely shutting down and rebooting the system or adversely affecting other system components This feature is only Chapter 3 NOTE IMPORTANT Chapter 3 Installing HyperFabric Installing HyperFabric Adapters available on HP 9000 systems that are designed to support OLAR The system hardware uses the per slot power control combined with OS support to enable this feature OLAR is supported only on TCP UDP IP over HF2 adapters Not all add in cards have this capability but over time many cards will be gaining this capability The HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for At this time Superdome systems are not intended for access by users HP recommends that these systems only be opened by a qualified HP engineer Failure to observe this requirement can invalidate any support agreement or warranty to which the owner might otherwise be entitled There are two methods to add
93. it command and SAM Do not try to use these addresses in your configuration Configuring the Transparent Local Failover feature Switch ID sw clic1 193 0 0 20 HF IP address 193 0 0 21 HF Ethernet MAC address switch 0 Switch 1 00 60 b0 d0 02 56 IP multicast address 226 10 1 1 HF2 adapter O HF2 adapter 1 HF2 adapter 1 HF2 adapter 0 Adapter ID Adapter ID Adapter ID Adapter ID clicO clic1 clicO Clic1 IP address IP address IP address IP address 192 0 0 1 192 0 8 3 192 0 0 2 192 0 8 4 subnet mask 255 255 255 0 subnet mask 255 255 255 0 subnet mask 255 255 255 0 255 255 255 0 node B node A 104 IP address 193 0 0 10 IP address 193 0 0 11 Chapter 4 Chapter 4 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Using the configuration information in Figure 4 10 the information you would specify when you run clic_init on each of the nodes is listed below This example is not an exact depiction of the prompts produced by clic_init but merely an example of the flow of information input In addition you should not try to use the dummy addresses in your actual configuration On node A 1 How many HyperFabric adapters are installed on the node 2 Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B 11 00 11 o
94. kit are as follows Prepare the rack for switch installation Insert the HF2 switch into the rack with the front of the switch snug against the back of the rack Align the two holes in each flange on the switch s front with the holes in the rack frame Fasten each flange of the switch to the rack by putting a screw in each of the four holes in the flanges Be sure to use screws with over sized heads Tighten all of the screws so that the HF2 switch is firmly mounted in the rack For each port that will be connected to an HyperFabric adapter in an HP 9000 system attach the cable from the corresponding adapter Your connections must be copper to copper and fiber to fiber including cables Connect the switch to the Ethernet network Plug the switch s power cord into the rack s PDU if it has one Alternatively you can plug a power cord that is compatible with your country s requirements into a power strip or outlet that you want to use for the switch In this case you are responsible for obtaining a compatible power cord Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the switch There is no power switch 65 Installing HyperFabric Installing HyperFabric Switches Step 10 Step 11 Step 12 Step 13 66 Once the power is on check these LEDs on the integrated Ethernet management LAN adapter card in the top slot of the switch
95. le you can replace the card switch module The integrated 8 port fiber card in the middle slot the card can be safely removed by qualified HP personnel only A6388A HF2 8 port fiber or A6389A HF2 4 port copper switch module in the expansion slot the bottom slot the switch module can be safely removed by you or qualified HP personnel First try powering the switch off and then back on Connection from port to adapter is operational Flashing green Data does not start to flow until HyperFabric initialization has occurred Data is flowing between the port and the adapter Chapter 6 Troubleshooting HyperFabric Determining Whether an Adapter or a Cable is Faulty Determining Whether an Adapter or a Cable is Faulty If you suspect that either an HyperFabric adapter or a cable attached to it is faulty follow these steps to determine which component needs replacing Step 1 Disconnect the cable from the HyperFabric adapter Step 2 Attach a loopback plug to the adapter e For the A6386A adapter Be sure to use a fiber loopback plug one HP part number A6384 67004 is shipped with each HF2 switch Step 3 Determine if the adapter is faulty e Ifthe adapter is functioning correctly the Connected Traffic Link or Link Connected Traffic LED on the adapter shows as solid green e Ifthe adapter is faulty the Error LED on the adapter shows as solid yellow See Table 6 1
96. lect View Local Failover for HMP option in the HyperFabric Configuration screen Deconfiguring HMP for Local Failover support Using SAM To use SAM to deconfigure HMP for Local Failover support complete the following steps Start SAM Select the Networking and Communications area Select HyperFabric All HyperFabric adapter card pairs installed in the system are listed Pull down the Actions menu and select Deconfigure Local Failover for HMP to remove all the card pair entries from the etc rc config d clic global conf file In the next confirmation window that pops up select YES to confirm it If you select NO you remain in the main HyperFabric Configuration Screen Exit SAM 103 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Figure 4 10 Ethernet LAN Switch ID sw clicO IP address Ethernet MAC address 00 60 b0 d0 02 57 IP multicast address 226 10 1 1 subnet mask Configuring HMP for Transparent Local Failover Support Using the clic_init command You can configure the Transparent Local Failover feature of HMP using clic_init also Let us consider the following example where we have discussed the configuration in detail This example uses some dummy that is not valid addresses to the components in Figure 4 10 The dummy addresses are used only to show the flow of the information provided as input to the clic_in
97. lling the HF2 Switch on page 59 58 Chapter 3 Installing HyperFabric Installing HyperFabric Switches Installing the HF2 Switch This section contains information on installing an HF2 switch The front of the HF2 switch has a flange or wing on each side with two holes for attaching the switch to the rack The following figures do not show the flanges Figure 3 2 shows the front of the HF2 switch with an A6388A HF2 8 port fiber switch module installed in the switch s expansion slot Figure 3 2 Front of HF2 Switch A6388A Switch Module Installed Label showing Port LED Integrated 8 port Integrated Ethernet management Ethernet MAC colors and fiber card LAN card address legend pars Ethernet Ethernet p 1 Status s Port Port x e nee eae 4 e Port Port Port Port Port Port Port js 2 1 0 16006 6 O Q 16 O Ce Port AE on Port Port Port iu Pu eg 15 12 11 10 Oioi o Oe 16 6 Ca A6388A HF2 8 port fiber switch module in expansion slot Chapter 3 59 Installing HyperFabric Installing HyperFabric Switches Figure 3 3 shows the front of the HF2 switch with an A6389A HF2 4 port copper switch module installed in the switch s expansion slot Front of HF2 Switch A6389A Switch Module Installed Figure 3 3 Integrated 8 port Label showing eee fiber card Integrated Ethernet management Ethernet MAC meanings LAN card address legend Power Ethernet Ethernet p
98. lobal Interval Timers Fabric mapping interval ms 60000 Performance Statistics Levels Possible perf degradation occurring DLPI performance statistics enabled e Example 3 a HMP in the non local failover mode If the local node is bent1y7 and you want to display the statistics for the adapter c1ic1 which is on bent1y7 issue this command clic stat c clic1 120 Chapter 5 Managing HyperFabric Displaying Status and Statistics The generated output could look like this Date Node Sat Aug 5 16 08 30 2000 bently7 corp2 com Adapter ID clicl Instance Number 1 Adapter Type 4X HF2 PCI Firmware File opt clic firmware clic fw hf232c Major Num 238 Mgmt process driver handle 5 Version 1 0 H W Path 8 4 0 0 Primary IP address 192 0 0 9 Uptime time 0 days 21 hours 1 min 3 sec Adapter State Flags Set Multiple flags may be set Configured and operational Switch mode F W Error Statistics Frame buffer overflow Receive on disabled endpoint Invalid endpoint ID Invalid endpoint protection key Interleaved gathered receive Interleaved multi frame bulk messages NO overflow Send NQ overflow Invalid slot key for NQ credit update DLPI QOS receive buffer shortage Link congestion events Max send packet retry exceeded Link or switch failure events Link or switch resume events Bad route detected Bad optional data length Invalid message received CRC error
99. log old OLDclic log usr opt conf n d LM clic T master d S e er Ibin ibclic_dlpi_drv a lalic Aib e firmware share libha drv a clic_diag libclic_mgmt a c clic fw i CE td clic_fw_1x32c manim Z aca d clic_fw_4x8c clic ping clic fw 4x32c clic fw hf28c i eens clic fw hf232c clic fw db clic start clic stat Chapter 3 51 Installing HyperFabric Installing the Software 52 The commands and files used to administer HyperFabric typically have a prefix of c1ic CLIC stands for CLuster InterConnect and it is used to differentiate those HyperFabric commands files from other commands files For example the HyperFabric command clic init is different from the HP UX init command Each of the files shown in Figure 3 1 above is briefly described below e etc opt resmon dictionary clic 01 The HyperFabric dictionary file for the Event Monitoring Service EMS e etc rc config d clic global conf The global configuration file which contains the IP addresses for each adapter and each HyperFabric switch if any in the fabric e sbin init d clic The system boot startup script for the HyperFabric management process e var adm clic ip drv trc One of the software s trace files This file is created when the clic diag D TCP IP command is run e var adm clic ip drv trcO One of the HyperFabric software s trace files This is the primary file that is created when
100. m 5 Packet Acknowledged 256 bytes seq_num 6 Packet Acknowledged 256 bytes seq_num 7 Packet Acknowledged 256 bytes seq_num 8 Packet Acknowledged Eccc bently6 corp3 com CLIC PROBE Statistics 8 packets transmitted 8 packets received 0 packet loss Example 3 If the local node is bent1y6 and you want to send five packets to verify communications with the remote node bent1y7 using the adapter c1ic0 which is on bent1y6 issue this command clic probe bently7 c clicO p 5 CLIC PROBE 256 byte packets Source adapter id bently6 corp3 com clicO Target adapter id bently7 corp4 com clicl 256 bytes seq num 1 Packet Acknowledged 256 bytes seq num 2 Packet Acknowledged 256 bytes seq num 3 Packet Acknowledged 256 bytes seq num 4 Packet Acknowledged 256 bytes seq num 5 Packet Acknowledged musei bently7 corp4 com CLIC PROBE Statistics 5 packets transmitted 5 packets received 0 packet loss Example 4 If the local node is bent1y6 and you want to send five packets to verify communications with the remote node bent1y7 using the adapter c1ic0 which is on bent1y6 and the route identified by VRID 194 and switch hopcount 1 issue this command clic probe c clicO r 194 1 p 5 Remember because you specified the x VRID switch hopcount parameter you do not need to also specify the node name 115 Managing HyperFabric Verifying Communications within the Fabric 11
101. m Four standard cable lengths are sold and supported 2m 16m 50m and 200m TCP UDP IP supports up to four HF2 switches connected in series with a maximum cable length of 200m between the switches and 200m between switches and nodes TCP UDP IP supports up to four hybrid HF2 switches connected in series with a maximum cable length of 200m between fiber ports Chapter 2 Planning the Fabric TCP UDP IP e Throughput and Latency Table 2 1 HF2 Throughput and Latency with TCP UDP IP Applications rp7400 2 2 Gbps full duplex per link lt 42 microsec Chapter 2 19 Planning the Fabric TCP UDP IP Table 2 2 Supported HyperFabric Adapter Configurations HF Bus Supported HP UX OLAR vy 2 Adapter Type HP Systems Version Support per System A6386A PCI 4X rx2600 servers 11i v2 No 1 A6386A PCI 4X rx56XX 4 servers A6386A PCI 4X zx6000 1 workstations A6386A PCI 4X SD64A servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx7620 servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx8620 servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx4640 servers 6 20 Chapter 2 Planning the Fabric TCP UDP IP TCP UDP IP Supported Configurations Multiple TCP UDP IP HyperFabric configurations are supported to match the cost scaling and performance requirements of each installation In the previous section Configuration Parameters on page 17 the maximum limit
102. mond e opt clic firmware clic fw The 1X HSC HyperFabric 8 bit CRC firmware This file must not be modified e opt clic firmware clic fw 1x32c The 1X HSC HyperFabric 32 bit CRC firmware This file must not be modified Chapter 3 53 Installing HyperFabric Installing the Software 54 opt clic firmware clic fw 4x8c The 4X PCI HyperFabric 8 bit CRC firmware This file must not be modified opt clic firmware clic fw 4x32c The 4X HyperFabric PCI 32 bit CRC firmware This file must not be modified opt clic firmware clic fw hf28c The HyperFabric2 8 bit firmware This file must not be modified opt clic firmware clic fw hf232c The HyperFabric2 32 bit firmware This file must not be modified opt clic firmware clic fw db A binary file where adapter specific configuration information is stored The management process creates this file using default values opt clic share man manim Z The man pages for the HyperFabric commands Chapter 3 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Chapter 3 Installing HyperFabric Installing the Software Loading the Software Listed below are the steps you must follow to load the HyperFabric software using the HP UX swinstall program Log on to the system as root Insert the software media into the appropriate drive If the software is being loaded from a CD ROM go to step 3 otherwise go to
103. n the servers are highly available In addition the workstations and the servers can run the same application or different applications see Figure 2 4 TCP UDP IP Hybrid Configuration Ia E I 3E la Ia TIPP ale RE WORKSTATIONS A la 25 Planning the Fabric Hyper Messaging Protocol HMP 26 Hyper Messaging Protocol HMP Hyper Messaging Protocol HMP is an HP patented high performance cluster interconnect protocol HMP provides reliable high speed low latency low CPU overhead datagram service to applications running on the HP UX operating system HMP was jointly developed with Oracle Corp The resulting feature set was tuned to enhance the scalability of the Oracle Cache Fusion clustering technology It is implemented using Remote DMA RDMA paradigms HMP is integral to the HP UX HyperFabric driver It can be enabled or disabled at HyperFabric initialization using the clic_init command or SAM The HMP functionality is used by the applications listed in the following Application Availability section HMP significantly enhances the performance of parallel and technical computing applications HMP firmware on HyperFabric adapter cards provides a shortcut that bypasses several layers in the protocol stack boosting link performance and lowering latency By avoiding interruptions and buffer copying in the protocol stack communication task processing is optimized Application Availability The
104. n against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and Appendix A Safety and Regulatory Information Regulatory Statements if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense Any changes or modifications not expressly approved by Hewlett Packard could void the user s authority to operate this equipment Use of shielded interface cables is required to comply within the Class A limits in part 15 of the FCC rules DOC Statement Canada only This Class A digital apparatus meets all the requirements of the Canadian Interference Causing Equipment Regulations Cet appareil num rique de la classe A respecte toutes les exigences du R glement sur le mat riel brouilleur du Canada Europe RFI Statement This is a Class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures Australia and New Zealand EMI Statement This product meets the applicable requirements of the Australia and New Zealand EMC Framework Q 79 Appendix A 163 Safety and Regulatory Information Regulatory Statemen
105. nes the condition to trigger a notification usually a change in interface status Notification can be accomplished with a SNMP trap or by logging into the syslog file with a choice of severity or by email to a user defined email address To configure the HyperFabric EMS monitor it is necessary to have the EMS HA monitor product installed Product Number B7609BA This product is available on the applications CD media Use SAM to initiate monitoring of any particular HyperFabric resource following the procedure outlined below 1 10 11 Chapter 4 Start SAM Use the online help at any time for details Select Resource Management Select Event Monitoring Service 2 3 4 5 Select Action and Add Monitoring Request Select the location net interfaces clic class for HyperFabric resources Select a resource instance either all instances or a specific instance from the list Validate your choice by clicking on OK at the bottom of the screen A Monitoring Request Parameters window opens showing the resource and its status if All instances have been selected then no value is displayed Define a condition that will trigger a notification for instance When Value is equal to UP Define a polling interval default is 300 seconds Define a way of notification SNMP trap log in syslog with a choice of severity or email to a user defined email address
106. nfiguration information for an existing HyperFabric adapter or switch while you are using clic_init c to configure a new adapter When you have completed the adapter installation go to Installing the Software on page 51 Chapter 3 Chapter 3 Installing HyperFabric Installing HyperFabric Adapters Online Replacement OLR When replacing an interface card online the replacement card must be identical to the card being replaced or at least be able to operate using the same driver as the replaced card This is referred to as like for like replacement and should be adhered to because using a similar but not identical card can cause unpredictable results For example a newer version of the target card that is identical to the older card in terms of hardware might contain an updated firmware version that could potentially conflict with the current driver An A6386A adapter must be replaced with another A6386A adapter etc Also the old adapter and new adapter must have the same revision levels When a replacement card is added to an HP 9000 system the appropriate driver for that card must be configured in the kernel before beginning the replacement operation SAM ensures the correct driver is present In most cases the replacement card will be the same type as a card already in the system and this requirement will be automatically met Keep the following things in mind e Ifthe necessary driver is not present and the driver is
107. nology equipment Roseville CA March 27 2000 European Contact Your local Hewlett Packard Sales and Service Office or Hewlett Packard GmbH Department HQ TRE Herrenberger Strafe 130 D 71034 B blingen FAX 49 7031 14 3143 166 Appendix A Safety and Regulatory Information Regulatory Statements DECLARATION OF CONFORMITY According to ISO IEC Guide 22 and EN 45014 Manufacturer s Name Hewlett Packard Company Systems Interconnect Solutions Lab Manufacturer s Address 8000 Foothills Blvd Roseville CA 95747 USA declares that the product Product Name HyperFabric2 Switch Model Number s A6384 62001 Prod No A6384A HyperFabric2 Switch A6388 60001 Prod No A6388A 8 ports adapter card 46389 60001 Prod No A6389A 4 ports adapter card A6386 60001 Prod No A6386A PCI adapter card Product Options All conforms to the following Product Specifications Safety IEC 950 1991 A1 A2 A3 A4 EN 60950 1992 A1 A2 A3 A4 A11 GB 4943 1995 IEC 825 1 1993 EN60825 1 1994 A1 Class 1 EMC CISPR 22 1997 EN 55022 1998 Class A CNS 13438 GB 9254 1988 CFR47 Part 15 Class A CISPR 24 1997 EN 55024 1998 IEC 61000 4 2 IEC 61000 4 3 ENV 50204 IEC 61000 4 4 IEC 61000 4 6 IEC 61000 4 8 EN 61000 3 2 EN 61000 3 3 Supplementary Information The product herewith complies with the requirements of the Low Voitage Directive 73 23 EEC and the EMC Directive 89 336 EEC and carries the CE marking accordingly 1
108. nstalled on the node 2 Do you want this node to interoperate with nodes running any HyperFabric versions earlier than B 11 00 11 or B 11 11 01 n You must answer no if you want to run applications using HMP Local Failover or Non Local Failover for communication over HyperFabric In that case all nodes in the cluster must be running version B 11 00 11 or B 11 11 01 or later version of HyperFabric software 3 What is the IP address of the first adapter clic0 192 0 0 2 4 What is the subnet mask of the first adapter 255 255 255 0 If you do not specify a value for this a default mask is chosen You will most likely just accept the default However in this example we are showing a value for the subnet mask just to illustrate the correlation between the dummy information in Figure 4 10 and where that information is specified or generated during clic_init and SAM 5 What is the IP address of the second adapter c1ic1 192 0 8 4 6 What is the subnet mask of the second adapter 255 255 255 0 7 Do you want to configure Local Failover on this node y Enter y if you are using Oracle RAC with HMP For MPI applications enter n 8 Select any two of the following clic adapters for CARD PAIRI O clic clicl Enter the first clic adapter from above listed adapters clic0 Enter the second clic adapter from above listed adapters clic1 9 Do you want to enable switch management n 106 Chapter 4
109. nt You cannot enable switch management through SAM you must use the clic init command As stated earlier HP recommends that you do not enable switch management However if you do enable it you must provide the information in items 8 through 14 If switch management has been enabled how many switches will be configured As stated earlier HP recommends that you do not enable switch management What is the IP address of the first switch sw c1ic0 193 0 0 20 What is the Ethernet hardware address of the first switch 006050800257 What is the IP address of the second switch sw clic1 193 0 0 21 Chapter 4 Configuring HyperFabric Information You Need 12 What is the Ethernet hardware address of the second switch 0060b0d00256 13 What is the Multicast address for the switches to use 226 10 1 1 14 What is the IP address for the LAN card on the same subnet as the switches 193 0 0 11 In Figure 4 1 this is the IP address for 1an0 on node B Chapter 4 77 Configuring HyperFabric Performing the Configuration NOTE IMPORTANT 78 Performing the Configuration As explained in Configuration Overview on page 69 you must create the global configuration file etc rc config d clic_global_conf on each node in the fabric This consists mostly of specifying HyperFabric adapter related information If you are also going to enable switch management which HP does not recommend doing
110. o communicate between HP 9000 nodes in a HyperFabric cluster you can configure ServiceGuard and the EMS monitor to identify node failure and automatically failover to a functioning HP 9000 node For more information on HA when running HMP applications contact your HP representative Transparent Local Failover Supported HMP supports Transparent Local Failover in the HyperFabric version B 11 23 01 When a HyperFabric resource adapter cable switch or switch port fails in a cluster HMP transparently fails over traffic using other available resources This is accomplished using card pairs each of which is a logical entity that comprises a pair of HF2 adapters on a HP 9000 node Only Oracle applications can make use of the Local Failover feature HMP traffic can only fail over between adapters that belong to the same card pair Traffic does not fail over if both the adapters in a card pair fail However administrators do not need to configure HF2 adapters as card pairs if TCP UDP IP is run over HF2 or MPI uses HMP When HMP is configured in the local failover mode all the resources in the cluster are utilized If a resource fails in the cluster and is restored HMP does not utilize that resource until another resource fails For more information on Transparent Local Failover while running HMP applications see Configuring HMP for Transparent Local Failover Support on page 96 Dynamic Resource Utilization DRU Partially Supp
111. ocedure described in the Managing PCI Cards with OLAR chapter of the Configuring HP UX Peripherals manual When an adapter has been replaced client application activity resumes unless the TCP timers or the application timers have popped Do not use the clic_start command or the clic_shutdown command while an installed adapter is suspended Do not use SAM to start or stop HyperFabric while an installed adapter is suspended The operation will fail and an error message will be displayed After a HyperFabric adapter has been replaced SAM checks the replacement adapter to ensure that it is permitted according to the like for like rules If the adapter is permitted SAM automatically activates it If it is not permitted SAM displays an error message Chapter 3 Installing HyperFabric Installing the Software Installing the Software This section describes the HyperFabric file structure and the steps necessary to load the software The software must be installed on each instance of the HP UX operating system in the fabric File Structure The HyperFabric file structure is shown in Figure 3 1 below The structure is shown for informational purposes only The user cannot modify any of the files or move them to a different directory Figure 3 1 HyperFabric File Structure e ow d as r a 9 intt d clic_ip_drv tre resmon clic_global_conf at clic_ip_drv trcO l clic_ip_drv trc1 dictionary clic_log clic 01 clic_
112. on page 143 for a list of the LED names for each HyperFabric adapter Step 4 Ifthe adapter is faulty you must replace it see Replacing a HyperFabric Adapter on page 156 If the adapter is not faulty assume that the cable is faulty and you must replace it Chapter 6 153 Troubleshooting HyperFabric Determining Whether a Switch is Faulty 154 Step 1 Step 2 Step 3 Determining Whether a Switch is Faulty If you suspect that a HyperFabric switch is faulty follow the steps below to determine if the switch needs replacing HF2 Switch The steps for determining if an HF2 switch is faulty are as follows Check the cards switch module in the switch the integrated Ethernet management LAN adapter card integrated 8 port fiber card and switch module in the expansion slot a b c If the card switch module is functioning correctly its Operating Fault LED shows as solid green If the card switch module is experiencing a fault its Operating Fault LED shows as solid yellow If the card switch module is not operating its Operating Fault LED is off Check the switch s power a b If the power to the switch is on the Power A and Power B LEDs both show as solid green If only one of the Power LEDs shows as solid green the switch is still operational However the integrated Ethernet management LAN adapter card should be replaced soon Remember it can be replace
113. on page 44 Starting HyperFabric launches the HyperFabric CLuster InterConnect CLIC daemon clic_mgmtd This daemon process must be running for the HyperFabric product to operate correctly It is possible that other daemons will be running but it is essential that at least one CLIC daemon is running To check if a CLIC daemon is running use the following command ps ef grep clic If the CLIC daemon is not running start the HyperFabric subsystem by executing the following command opt clic bin clic start 109 Managing HyperFabric Starting HyperFabric 110 Step Step Step Step Using the clic_start Command Run the clic_start command on each node to start the HyperFabric management process on that node If you include opt clic bin in your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic_start You must be logged in as root to run this command The syntax is as follows clic_start The clic_start command can be issued to display the online help for clic_start or look at the clic_start 1m man page by issuing the man clic_start command If HyperFabric is already running you will receive an informational FYI message telling you so Your reaction to this message depends on the situation e Ifyou have simply forgotten or did not know that HyperFabric was already running you do not have to
114. ons using TCP UDP IP and the computing environment stabilizes with a requirement for higher performance migration to HMP is always an option Chapter 2 21 Planning the Fabric TCP UDP IP Figure 2 1 TCP UDP IP Point To Point Configurations Configuration A Configuration C SINGLE CONNECTION BET WEEN TWO NODES SINGLE CONNECTION BETWEEN MULTIPLE NODES Configuration B MULTIPLE CONNECTIONS BETWEEN TWO NODES 22 Chapter 2 Planning the Fabric TCP UDP IP Switched Configuration This configuration offers the same benefits as the point to point configurations illustrated in Figure 2 1 but it has the added advantage of greater connectivity see Figure 2 2 Figure 2 2 TCP UDP IP Basic Switched Configuration Chapter 2 23 Planning the Fabric TCP UDP IP High Availability Switched Configuration This configuration has no single point of failure The HyperFabric driver provides end to end HA If any HyperFabric resource in the cluster fails traffic is transparently rerouted through other available resources This configuration provides high performance and high availability see Figure 2 3 Figure 2 3 TCP UDP IP High Availability Switched Configuration LAN Link for MC ServiceGuard 24 Chapter 2 Figure 2 4 WORKSTATIONS Chapter 2 Planning the Fabric TCP UDP IP Hybrid Configuration You can interconnect servers and workstations in a single heterogeneous HyperFabric cluster In this configuratio
115. or replace OLAR compatible cards e Using the SAM utility e Issuing command line commands through olrad that refer to the HyperFabric OLAR script usr sbin olard d clicd HP recommends that SAM be used for OLAR procedures instead of the rad command This is primarily because SAM prevents the user from doing things that might have adverse effects This is not true when the rad command is used For detailed information about using either of these two procedures see Configuring HP UX For Peripherals You can order that document from HP or you can view download and print it from the following URL http www docs hp com 45 Installing HyperFabric Installing HyperFabric Adapters Table 3 1 IMPORTANT 46 Table 3 1 below explains some important OLAR related terms Important OLAR Terms Term Meaning OLAR All aspects of the OLAR feature including Online Addition OLA and Online Replacement OLR Power Domain A grouping of 1 or more interface card slots that are powered on or off as a unit Note Multi slot power domains are not currently supported The interface card which will be added or replaced using OLAR and the card slot in which it resides target card target card slot affected card affected card slot Interface cards and the card slots they reside in which are in the same power domain as the target slot In many cases other interface cards and slots within the
116. orted If you add a new HyperFabric resource node cable or switch to a cluster running an HMP application the HyperFabric subsystem will dynamically identify the added resource and start using it The Chapter 2 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP same process takes place when a resource is removed from a cluster However DRU is not supported if you add or remove an adapter from a node that is running an HMP application This is consistent with the fact that OLAR is not supported when an HMP application is running on HyperFabric e Load Balancing Supported When an HP 9000 HyperFabric cluster is running HMP applications the HyperFabric driver balances the load across all available resources in the cluster including nodes adapter cards links and multiple links between switches Switch Management Not Supported Switch Management is not supported Switch management will not operate properly if it is enabled on a HyperFabric cluster e Diagnostics Supported You can run diagnostics to obtain information on many of the HyperFabric components using the clic_diag clic_probe and clic_stat commands as well as the Support Tools Manager STM For more information on HyperFabric diagnostics see Running Diagnostics on page 149 Configuration Parameters This section discusses the maximum limits for HMP HyperFabric configurations There are numerous variables that can impact the performanc
117. oting HyperFabric Running Diagnostics 136 Diagnostics can be run on many of the HyperFabric components by using the clic_diag command If the HyperFabric subsystem is still not usable contact your HP support representative with the diagnostics data generated using the clic_diag command described below The clic_diag Command Use the clic_diag command to run the following diagnostics e Probe a specific remote node e Dump and format trace data e Set the tracing level for the HyperFabric software and the firmware Two versions of the clic_diag command are available e The command line version In this version you specify the command and the parameters you want to run which are described below e The interactive version To use this version specify only the command name clic_diag Then you are prompted for the same kinds of information you specify when you use the command line version If you include opt clic bin in your PATH statement you can run the command as it is shown below Otherwise you must include opt clic bin as part of the command name that is opt clic bin clic_diag You must be logged in as root to run this command The command line syntax is as follows clic diag r remote node name T trace level D TCP IP C TCP IP B trace buffer size F trace file size d adapter ID filename Note that some of the lines in the above syntax are indented for readability purposes only W
118. ould not work on a bus running at 66 MHz rad provides information about the bus frequency and power available at a slot as well as other slot related data If an HP 9000 system has one or more slots that support OLAR and OLA will be used to install a HyperFabric adapter in one of those slots install the adapter in the HP 9000 system according to the procedure described in the Managing PCI Cards with OLAR chapter of the Configuring HP UX Peripherals manual After adding a new HyperFabric adapter SAM tries to locate the HyperFabric software If SAM cannot locate the HyperFabric software the new adapter cannot be used until the software is installed software installation requires a system reboot If SAM locates the HyperFabric software SAM determines whether the new adapter is functional If it is not functional SAM displays an error message If the new adapter is functional SAM displays a message telling the user to configure the adapter and start HyperFabric If only one adapter is being added issue the clic_init c command or use SAM to configure the adapter and then issue the clic_start command or use SAM to start HyperFabric If multiple adapters are being added add all of the adapters first and then run clic init c and clic start or use SAM See Performing the Configuration on page 78 and Starting HyperFabric on page 109 for more information about configuring and starting HyperFabric Do not change any co
119. perFabric e Information You Need on page 71 e Performing the Configuration on page 78 e Deconfiguring a HyperFabric Adapter with SAM on page 83 e Configuring the HyperFabric EMS Monitor on page 85 e Configuring HyperFabric with ServiceGuard on page 87 e Configuring HMP for Transparent Local Failover Support on page 96 68 Chapter 4 NOTE Chapter 4 Configuring HyperFabric Configuration Overview Configuration Overview You do not need to configure the HyperFabric switch because the HyperFabric management process performs automatic routing and configuring for the switch So configuring HyperFabric consists only of creating the HyperFabric etc rc config d clic global conf global configuration file on each node in the fabric The configuration file contains the following information e The IP addresses and subnet mask of the HyperFabric adapters installed in the node e For each HyperFabric switch in the fabric the switch s IP address and the MAC address of the switch s Ethernet port This applies only if you enable switch management In addition you cannot enable switch management through SAM you must use the clic_init command e The IP multicast address that all the switches and nodes in the fabric will register to if you are going to enable switch management e The IP address of the local node s Ethernet LAN interface This LAN interface must be on the same subn
120. ports These recommended configurations offer an optimal mix of performance and availability for a variety of operating environments There are many variables that can impact HyperFabric performance If you are considering a configuration that is beyond the scope of the following HP supported configurations contact your HP representative Point to Point Configuration You can interconnect large servers like the HP Superdome to run Oracle RAC 9i and enterprise resource planning applications These applications are typically consolidated on large servers Point to point connections between servers support the performance benefits of HMP without investing in HyperFabric switches This is a good solution in small configurations where the benefits of a switched HyperFabric cluster might not be required see configuration A in Figure 2 5 If an HMP application is running over HyperFabric and another node or adapter is added to either of the nodes then it is necessary to also add a HyperFabric switch to the cluster see configuration B in Figure 2 5 Chapter 2 33 Planning the Fabric Hyper Messaging Protocol HMP Figure 2 5 HMP Point To Point Configurations Configuration A SINGLE CONNECTION BETWEEN TWO NODES Configuration B MULTIPLE CONNECTIONS BETWEEN TWO NODES 34 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP Enterprise Database Configuration The HMP enterprise configuration illustrated in Figure
121. r B 11 11 01 n You must answer no if you want to run applications using HMP Local Failover or Non Local Failover for communication over HyperFabric In that case all nodes in the cluster must be running version B 11 00 11 or B 11 11 01 or later version of HyperFabric software 3 What is the IP address of the first adapter clic0 192 0 0 1 4 What is the subnet mask of the first adapter 255 255 255 0 If you do not specify a value for this a default mask is chosen You will most likely just accept the default However in this example we are showing a value for the subnet mask just to illustrate the correlation between the dummy information in Figure 4 10 and where that information is specified or generated during clic_init and SAM 5 What is the IP address of the second adapter c1ic1 192 0 8 3 6 What is the subnet mask of the second adapter 255 255 255 0 7 Do you want to configure Local Failover on this node y Enter y if you are using Oracle RAC with HMP For MPI applications enter n 8 Select any two of the following clic adapters for CARD PAIRI O clic clicl Enter the first clic adapter from above listed adapters clic0 Enter the second clic adapter from above listed adapters clic1 9 Do you want to enable switch management n 105 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support On node B 1 How many HyperFabric adapters are i
122. rFabric adapter in the node Throughput might be slower because only one active adapter is now handling the network traffic This rearrangement is handled by the HyperFabric software and ServiceGuard is not notified However note that if all of the HyperFabric adapters fail HyperFabric does notify ServiceGuard In both cases though the events are logged to var adm clic_log and var adm syslog 1log Chapter 4 Figure 4 3 Chapter 4 Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 1 This example illustrated by Figure 4 3 below presents an HA configuration using ServiceGuard with HyperFabric Both of the HyperFabric adapters are active on node A The HyperFabric Resource Monitor reports the active status of the HyperFabric resource to the Event Monitoring Service EMS which lets ServiceGuard know that the HyperFabric resource is available to Packages A and B Node with Two Active HyperFabric Adapters HyperFabric Resource Active S Active Adapter IP address p Adapter adapter 47516 19 14 Adapter IP address 172 16 20 21 91 Configuring HyperFabric Configuring HyperFabric with ServiceGuard Figure 4 4 92 Example 2 This example illustrated by Figure 4 4 below shows the same node after the failure of one of the HyperFabric adapters The remaining adapter in node A is now handling all HyperFabric network traffic for the node Because the HyperFabric resource i
123. re the brackets are snug against the beveled part of the switch tighten all four screws in brackets 1 and 2 Step 7 For each port that will be connected to an HyperFabric adapter in an HP 9000 system attach the cable from the corresponding adapter Your connections must be copper to copper and fiber to fiber including cables Step 8 Connect the switch to the Ethernet network Chapter 3 63 Installing HyperFabric Installing HyperFabric Switches 64 Step 9 Step 10 Step 11 Step 12 Step 13 Step 14 Plug the switch s power cord into the rack s PDU if it has one Alternatively you can plug a power cord that is compatible with your country s requirements into a power strip or outlet that you want to use for the switch In this case you are responsible for obtaining a compatible power cord Power on the HF2 switch by plugging the power cord into the AC inlet on the back of the switch There is no power switch Once the power is on check these LEDs on the integrated Ethernet management LAN adapter card in the top slot of the switch V The Operating Fault LED shows solid green V The Power A and Power B LEDs show solid green Y The Ethernet Port Main and Ethernet Port Aux LEDs are showing solid green connected or flashing green Ethernet traffic is flowing to the switch See Figure 3 2 on page 59 or Figure 3 3 on page 60 for the locations of the LEDs On the integra
124. replace that card switch module Remember though that the integrated 8 port fiber card can be removed by qualified HP personnel only The A6388A and A6389A switch modules can be removed by you or a qualified HP personnel Step 5 Ifa fan is not running the fan tray must be replaced by qualified HP personnel only Chapter 6 155 Troubleshooting HyperFabric Replacing a HyperFabric Adapter 156 Step 1 Step 2 Step 3 Replacing a HyperFabric Adapter If OLAR is supported for the HP 9000 system and the HyperFabric adapter and you want to do OLR see Online Replacement OLR on page 49 The HP HyperFabric Release Notes contains information about which HP 9000 systems and HyperFabric adapters OLAR is supported for If you cannot or do not want to do OLR to replace a HyperFabric adapter follow these steps Stop HyperFabric on the node where the faulty adapter is installed see Stopping HyperFabric on page 128 Depending on the type of HP 9000 the adapter is in follow the steps in the HP 9000 s documentation for installing the adapter Remember that when you replace a HyperFabric adapter you do not have to re install the HyperFabric software Start HyperFabric on the HP 9000 see Starting HyperFabric on page 109 Chapter 6 Chapter 6 Step 1 Step 2 Step 3 Troubleshooting HyperFabric Replacing a HyperFabric Switch Replacing a HyperFabric Switch If you need to replace a faulty Hyper
125. rsion of this manual can be found online at http www docs hp com hpux netcom index html HyperFabric Intended Audience This manual is intended for system and network administrators responsible for installing configuring and managing the HyperFabric software and hardware Administrators are expected to have knowledge of operating system concepts commands and configuration It is helpful to have knowledge of Transmission Control Protocol Internet Protocol TCP IP networking concepts and network configuration This document is not a tutorial xi HP UX Release Name and Release Identifier Each HP UX 11i release has an associated release name and release identifier The uname 1 command with the r option returns the release identifier Table 1 shows the releases available for HP UX 111 Table 1 HP UX 11i Releases Supported Release Identifier Release Name Processor Architecture B 11 11 HP UX 11i v1 PA RISC B 11 20 Intel Itanium B 11 22 HP UX 11i v1 6 Intel Itanium B 11 23 HP UX 11i v2 Intel Itanium Publishing History Table 2 provides the publication date for the pertinent edition number Table 2 Publishing History Details Edition Number Publication Date First March 1998 Second June 1998 Third August 1998 Fourth October 1998 Fifth December 1998 Sixth February 1999 Seventh April 1999 Eighth March 2000 Ninth June 2000 Tenth December 2000 xii Table 3 Table
126. s for TCP UDP IP enabled HyperFabric hardware configurations were outlined In this section the TCP UDP IP enabled HyperFabric configurations that HP supports are explained These recommended configurations offer an optimal mix of performance and availability for a variety of operating environments There are many variables that can impact HyperFabric performance If you are considering a configuration that is beyond the scope of the following HP supported configurations contact your HP representative Point to Point Configurations You can interconnect large servers like the HP Superdome to run Oracle RAC 9i and enterprise resource planning applications These applications are typically consolidated on large servers Point to point connections between servers support the performance benefits of HMP without investing in HyperFabric switches This is a good solution in small configurations where the benefits of a switched HyperFabric cluster might not be required see configuration A and configuration C in Figure 2 1 If there are multiple point to point connections between two nodes traffic load is balanced over those links If one link fails the load fails over to the remaining links see configuration B in Figure 2 1 Running applications using TCP UDP IP on a HyperFabric cluster provides major performance benefits compared to other technologies such as Ethernet If a HyperFabric cluster is originally set up to run enterprise applicati
127. s still available ServiceGuard has not been notified HyperFabric handles the local HyperFabric adapter failover However the failure of adapter 1 has been logged to var adm clic_log Node with One Failed HyperFabric Adapter HyperFabric Resource Active Package A Failed Adapter Adapter IP addresses 172 16 10 11 172 16 20 21 After the failover if you issue a netstat incommand you will see that an IP address is still assigned to each adapter For example Name MTU network Address Ipkts Opkts clicl 31744 172 16 10 0 172 16 10 11 711 12 clicO 31744 172 16 20 0 172 16 20 21 1222 333 Chapter 4 Configuring HyperFabric Configuring HyperFabric with ServiceGuard Example 3 This final example illustrated by Figure 4 5 below shows a situation in which all of the HyperFabric adapters on node A fail The HyperFabric Resource Monitor reports to the Event Monitoring Service EMS The EMS then notifies the ServiceGuard cmc1d daemon that the HyperFabric resource on node A is unavailable Because HyperFabric is configured as a package dependency for Packages A and B ServiceGuard causes the packages to failover to node B In a four node configuration note that only two nodes are shown in Figure 4 5 below Packages A and B can continue to communicate through the HyperFabric network with the other active nodes in the ServiceGuard cluster Figure 4 5 When All HyperFabric Adapters Fail HyperFabric HyperFa
128. shutdown followed by clic_start on all the nodes in the cluster after replacing at least one trunk between the switches e Maintain at least one trunk between the switches when Oracle RAC is running in the cluster e If any of the nodes is shut down for administrative reasons shut down Oracle RAC before executing clic_shutdown on that node In such a case Oracle RAC continues to be operational on all the other nodes in the cluster When you bring up the node that was shut down it joins the cluster Chapter 4 97 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Figure 4 6 98 How Transparent Local Failover Works Consider a hypothetical HyperFabric configuration in a 4 node cluster with each node having two adapters see Figure 4 6 In this configuration there is no single point of failure and all adapters that are installed on any given node are configured as part of a card pair A Configuration supporting Local Failover ei E ei 1 E 0 c e a cs L Em ei S Em 0 ei 1 HF adapter 0 HF adapter 1 HF adapter 1 HF adapter 0 Card Pair 0 0 Aled pled 0 ile p129 The details of how HyperFabric handles failures of the adapters links switch ports switches and cables between switches in a cluster are discussed in the following sections Chapter 4 Configuring HyperFabric Configuring HMP for Transparen
129. sing the clic start Command 0 0 c ccc eens 110 Using SAM x ont ceva eaves er EE at ee E e bie 110 Verifying Communications within the Fabric 0 0 c cee ees 112 The clic probe Command 0 ccc eee teen ne ens 112 Examples of clic_probe 00 0 cece eee eee e nnn 114 Displaying Status and Statistics 0 ene cent eee 117 The clic stat Command 0 ccc cece hh 117 Examples of clic stat 6 ei ARE aR Re a dee eee el ce Re ra Ba 119 Viewing man Pages gg ce une has ee e Hi ev voie rre bes 127 Stopping HyperFabric rera ttre eee he 128 Using the clic shutdown Command 00 ccc eect eens 128 Using SAM uc neon obits Hebd BeOS abate eee See aed Ae ace ast 129 6 Troubleshooting HyperFabric R nning Diaenostics ve sso ence ROI Rak ee NS eRe eec rm Bo SR Rs 133 The clic diag Commands 2 65 00 12i ORE De eee ER hk eee eS 136 Example of clic diag uer hoe ee ORR a ISP es 138 Using Support Tools Manager 00 eee een een neee 140 Usef l Files feel Dice hal aah hated sah ew Mee oes pe S E 141 LED Colors and Their Meanings 000 c eee ene ee 143 Adapter LEDs kac pah oe eX eA E eee Sacr Sas ee ned s 143 HEX Switch LEDS2 2 2e wy aD ex tock ate Seater Mb eU puerum ts 148 Determining Whether an Adapter or a Cable is Faulty 0000 000s 153 Determining Whether a Switch is Faulty llle 154 HE2 Switch s cone eet xe ee eek eed e ee
130. system are dependent on the target card For example if the target card is a multiple port card suspending or deleting drivers for the target card slot also suspends individual drivers for the multiple hardware paths on that card During a card replacement operation SAM performs a Critical Resource Analysis CRA which checks all ports on the target card for critical resources that would be temporarily unavailable while the card is shut down Chapter 3 Chapter 3 Installing HyperFabric Installing HyperFabric Adapters Planning and Preparation As mentioned previously for the most part SAM prevents the user from performing OLAR procedures that would adversely affect other areas of the HP 9000 system See Configuring HP UX For Peripherals for detailed information Critical Resources The effects of shutting down a card s functions must be considered Replacing a card that is still operating can have extensive consequences Power to a slot must be turned off when a card is removed and a new card is inserted This is particularly important if there is no online failover or backup card to pick up those functions For example e Which mass storage devices will be temporarily disconnected when a card is shut down e Will a critical networking connection be lost A critical resource is one that would cause a system crash or prevent an operation from successfully completing if the resource were temporarily suspended or disconne
131. t 144 Chapter 6 Troubleshooting HyperFabric LED Colors and Their Meanings Table 6 2 HyperFabric Adapter LED Colors and Meanings LED Connected Traffic A4919A and A6092A adapters Link A4920A and A6386A adapters Link Connected Traffic A4921A adapter Chapter 6 Color None Solid green Meaning Adapter is not operational Connection from adapter to switch port or remote adapter is operational Notes HP 9000 is not operational HyperFabric is not running on HP 9000 Adapter is installed in a slot in HP 9000 but cable is attached incorrectly or no cable is attached at all Adapter is bad Cable is bad Switch port is bad if adapter is connected to a switch If switch port is bad and you cannot use a different port on the switch you must replace the entire HF2 switch module First try powering the switch off and then back on Adapter is connected to a non operational adapter in the remote node in a node to node configuration Flashing green Data is flowing between the adapter and the switch port or remote adapter Data does not start to flow until HyperFabric initialization has occurred 145 Troubleshooting HyperFabric LED Colors and Their Meanings Table 6 2 146 HyperFabric Adapter LED Colors and Meanings Continued Error LED Color None Solid yellow Meaning Adapter is not
132. t Local Failover Support Case 1 Adapter Link or Switch Port Failure see Figure 4 7 If an adapter or a link or a switch port fails HMP transparently fails over traffic through the other available link Figure 4 7 Adapter Link or Switch Port Failover node A HF adapter 1 HF HF adapter 0 adapter 1 HF Switch 0 1 HF Switch 1 EAN EAN 0 0 Aled pied Card Pair 0 HF adapter 0 HF adapter 1 0 1ed pieg Card Pair 0 HF adapter 1 HF adapter 0 Chapter 4 99 Configuring HyperFabric Configuring HMP for Transparent Local Failover Support Case 2 Switch Failure see Figure 4 8 Consider the following illustration where node A is connected to node D with traffic being routed through the HF adapter 1 on both the nodes A and D and the HF switch 1 fails HMP transparently fails over traffic through the other available switch HF switch 0 This is possible only if at least one adapter of the card pairs on both the nodes HF adapter 0 of the Card Pair 0 on nodes A and D is physically reachable through the HF switch 0 Figure 4 8 Switch Failover HF HF adapter 1 adapter 0 HF HF adapter 0 adapter 1 RRR OX eres AUN SHE SHICHI Sed cel 0 Aled plea Card Pair 0 I Wa ANS HF HF adapter 0 adapter 1 0 led plea HF HF adapter 1 adapter 0 Card Pair 0 100 Chapter 4 Figure 4 9 Configuring
133. t and Latency with TCP UDP IP Applications 19 Table 2 2 Supported HyperFabric Adapter Configurations 20 Table 2 3 HF2 Throughput and Latency with HMP Applications 31 Table 2 4 Supported HyperFabric Adapter Configurations 31 Table 3 1 Important OLAR Terms 0 es 46 Table 6 1 LED Names by Adapter 0 0 0 cc ccc ccc cece eee eens 143 Table 6 2 HyperFabric Adapter LED Colors and Meanings 145 Table 6 3 HF2 Switch LED Colors and Meanings 000 0c eee 150 Tables About This Document This document describes how to install configure and troubleshoot the HyperFabric product on the HP UX 11i v2 HP UX 11 23 operating system Before you install the HyperFabric software ensure that the operating system software and the appropriate files scripts subsets are installed The document printing date and part number indicate the document s current edition The printing date will change when a new edition is printed Minor changes may be made at reprint without changing the printing date The document part number will change when extensive changes are made Document updates may be issued between editions to correct errors or document product changes To ensure that you receive the updated or new editions you should subscribe to the appropriate product support service See your HP sales representative for details The latest ve
134. ted 8 port fiber card in the middle slot of the switch check that for each switch port that is connected to an HF2 adapter the LED on the port shows as solid green see Figure 3 2 on page 59 or Figure 3 3 on page 60 This means the connection is operational On the switch module in the expansion slot the bottom slot of the switch check that for each switch port that is connected to an HyperFabric adapter the LED on the port shows as solid green see Figure 3 2 on page 59 or Figure 3 3 on page 60 This means the connection is operational For more information about the switch s LEDs see HF2 Switch LEDs on page 148 If you want to install another HF2 switch using the rail kit go back to step 1 If you want to install another HF2 switch without using the rail kit go to the next section Without the Rail Kit Otherwise go to Chapter 4 Configuring HyperFabric on page 67 Chapter 3 Step Step Step Step Step Step Step Step Step Chapter 3 Installing HyperFabric Installing HyperFabric Switches Without the Rail Kit As mentioned earlier HP strongly recommends installing the HF2 switch using the rail kit described in the previous section With the Rail Kit on page 61 When you install the HF2 switch you will be putting the front of the switch the end with the flanges wings at the back of the rack The steps for installing the HF2 switch without using the rail
135. th the B 11 23 01 release of HyperFabric For more information see Configuring HMP for Transparent Local Failover Support on page 96 70 Chapter 4 IMPORTANT Chapter 4 Configuring HyperFabric Information You Need Information You Need When you run the clic_init command or use SAM for configuration you have to provide certain configuration information So before you run clic_init or use SAM you should have the following information For each node in the fabric determine if that node will need to interoperate with other nodes that are using any HP UX 11 0 HyperFabric versions earlier than B 11 00 11 or any HP UX 11i v1 HyperFabric versions earlier than B 11 11 01 l For each HyperFabric adapter installed in the local node V The adapter s IP address The last 10 bits of each adapter s IP address must be unique throughout the entire fabric And remember that the last part of the address cannot be 0 that is the IP address cannot be n n n 0 Also note that HyperFabric converts these 10 bits to a decimal value called the Virtual Route IDentifier VRID which is used in some HyperFabric command input and output V The subnet mask When you run clic init or use SAM if you do not specify a value for this a default subnet mask is chosen based on the adapter s IP address When clic init begins to prompt you for the information for each adapter it assigns an ID for example c1ic0 to that adapter and displays
136. the HyperFabric version B 11 23 01 If a HyperFabric resource adapter cable switch or switch port fails in a cluster HMP transparently fails over traffic Local Failover using another available resource from the card pair card pair can be defined as a logical entity comprising of a pair of HF2 adapters on a HP 9000 node For example if there are four HF2 adapters installed and configured in a node then there would be two card pairs IMPORTANT Remember the following points while configuring HMP for Local Failover support e Only Oracle applications can make use of the local failover feature Other middleware like MPI can continue using HMP without local failover support e HMP supports the local failover configuration in HyperFabric version B 11 23 01 e All nodes in the cluster need to be configured either in the local failover mode or the non local failover mode While using clic init if you answer y to the question Do you want to configure Local Failover on this node then you have configured HMP for the local failover mode Otherwise your HMP configuration is for the non local failover mode Do not mix these two modes For any incorrect configurations HP recommends that you clic shutdown and clic start the cluster e The Transparent Local Failover feature over HMP is supported only in a switch based environment If two nodes are connected through multiple point to point links local failover cannot be achieved
137. the clic diag C TCP IP command is run e var adm clic ip drv trci One of the HyperFabric software s trace files This file is created when the clic diag C TCP IP command is run and the primary trace file clic_ip_drv trc0 becomes full e var adm clic log The global log file that is updated by the HyperFabric management process e var adm clic log old The backup copy of the log file that is created when the log file grows larger than 100 Kbytes Chapter 3 Installing HyperFabric Installing the Software e var adm OLDclic log The log file from the previous time the clic start command was executed e usr conf lib libclic dlpi drv a The kernel library that contains the HyperFabric software e usr conf lib libha drv a The kernel library that contains the High Availability HA software e usr conf master d clic This file is described along with the other master files in the master man page type man master at the HP UX prompt e opt clic lib libclic mgmt a The HyperFabric management API library e opt clic bin The directory containing the HyperFabric management commands clic diag clic init clic probe clic shutdown clic start clic stat and clic dump Note that clic dump is for HP internal use only Also clic ping is replaced by clic probe This directory also contains the HyperFabric management process clic mgmtd and the HyperFabric EMS monitor process clic
138. ts Radio Frequency Interference Japan Only VCCI Class A Model A4891 62001 only CORES HWWERESERESE AMan VCCI ORE BO 22 AA REM Cd CORES Riga CHAT oC ER HEAT ROTCE MOY ET COBBLER SEUU MRE RT 45 BRENSCEMHY ET 164 Appendix A Safety and Regulatory Information Regulatory Statements Declarations of Conformity DECLARATION OF CONFORMITY according to ISO IEC Guide 22 and EN 45014 Manufacturer s Name Hewlett Packard Company Manufacturer s Address 8000 Foothills Blvd Roseville CA 95747 USA declares that the product Product Name HyperFabric high speed network link Model Number s A4891 62001 Network Switch Prod No A4891A A4919 60001 PCI Adapter Card Prod No A4919A A4919 60002 HSC Adapter Card Prod No A4920A A4919 60003 HSC Adapter Card Prod No A4921A Product Options All conforms to the following Product Specifications Safety IEC 950 1991 A1 A2 A3 A4 EN 60950 1992 A1 A2 A3 A4 EMC CISPR 22 1993 EN 55022 1994 Class A EN 50082 1 1992 Generic Immunity including IEC 801 2 1991 prEN 55024 2 1992 4 kV CD 8 kV AD IEC 801 3 1984 prEN 55024 3 1991 3 V m IEC 801 4 1988 prEN 55024 4 1993 1 kV Power Lines 0 5 kV Signal Lines IEC 1000 3 2 1995 EN 61000 3 2 1995 no limit IEC 1000 3 3 1994 EN 61000 3 3 1995 not applicable Supplementary Information The product herewith complies with the requirements of the Low Voltage Directive 73 23 EEC and the EMC
139. unt Route Id Physical route 3 192 0 0 1 0 Point to Point Connection 2 192 0 0 2 0 Point to Point Connection 12 192 0 0 12 0 Point to Point Connection 13 192 0 0 13 0 Point to Point Connection Adapter elie VRID IP Address Switch Hopcount Route Id Physical route all 192 0 0 1 0 Point to Point Connection 2 192 0 0 2 0 Point to Point Connection 12 192 0 0 12 0 Point to Point Connection 13 192 0 0 13 0 Point to Point Connection 124 Chapter 5 Managing HyperFabric Displaying Status and Statistics The generated output if the nodes are connected through a switch could look like this Date Sat Aug 5 16 08 12 2000 Node bently6 corp4 com Adapter clicO VRID IP Address Switch Hopcount Route Id Physical route 1 192 0 0 1 1 0 0x09 2 192 0 0 2 1 0 0x03 12 192 0 0 12 1 0 0x06 13 192 0 0 13 1 0 0x00 Adapter clic2 VRID IP Address Switch Hopcount Route Id Physical route 1 192 0 0 1 1 0 0x03 2 192 0 0 2 1 0 0x3d 12 192 0 0 12 1 0 0x00 13 192 0 0 13 1 0 0x3a Chapter 5 125 Managing HyperFabric Displaying Status and Statistics e Example 5 If the local node is bent1y7 and you want to disable all statistics gathering on bent1y7 issue this command clic stat p RST The generated output could look like this Date Sat Aug 5 16 08 35 2000 Node bently7 corp6 com 126 Chapter 5 Chapter 5 Managing HyperFabric Viewing man Pages Viewing man Pages If you want to be able to view the HyperF
140. ure HyperFabric with ServiceGuard complete the configuration described in Configuring HyperFabric with ServiceGuard on page 87 then run clic_start or use SAM to start HyperFabric If you have just created the global configuration file on the local node for the first time and you are not configuring ServiceGuard run clic_start or use SAM to start HyperFabric If you have just changed an existing configuration file on the node run clic_shutdown or use SAM to stop HyperFabric and then run clic_start or use SAM to start HyperFabric Until you do those two things your configuration changes will not take effect Stopping HyperFabric on page 128 and Starting HyperFabric on page 109 for more information 82 Chapter 4 Step Step Step Step Step Step Step Step NOTE Chapter 4 Configuring HyperFabric Deconfiguring a HyperFabric Adapter with SAM Deconfiguring a HyperFabric Adapter with SAM To use SAM to deconfigure a HyperFabric adapter on an HP 9000 system running HP UX 11i v2 complete the following steps Start SAM Select the Networking and Communications area Select HyperFabric All HyperFabric adapters installed in the system are listed Installed adapters that are configured show Configured in the Status field and installed adapters that are not yet configured show Not Configuredin the Status field You can deconfigure only an adapter with a status of Conf
141. versions B 11 00 11 B 11 11 01 and B 11 23 00 e HMP is not supported on the A180 or A180C server e HMP is not supported on 32 bit versions of HP UX e Throughput and Latency Table 2 3 HF2 Throughput and Latency with HMP Applications Latency rp 7400 2 2 Gbps full duplex per link 22 microsec Table 2 4 Supported HyperFabric Adapter Configurations HF Bus Supported HP UX OLAR Ans 9 Adapter Type HP Systems Version Support ber Syste A6386A PCI 4X rx2600 servers 1 A6386A PCI 4X rx56XX 4 servers A6386A PCI 4X zx6000 1 workstations A6386A PCI 4X SD64A servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx7620 servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx8620 servers 8 maximum 4 per PCI card cage A6386A PCI 4X rx4640 servers 11i v2 Yes 6 Chapter 2 31 Planning the Fabric Hyper Messaging Protocol HMP NOTE The local failover configuration on HMP is supported only on the A6386A HF 2 adapters 32 Chapter 2 Planning the Fabric Hyper Messaging Protocol HMP HMP Supported Configurations Multiple HMP HyperFabric configurations are supported to match the performance cost and scaling requirements of each installation In the section Configuration Parameters on page 29 the maximum limits for HMP enabled HyperFabric hardware configurations were outlined This section discusses the HMP enabled HyperFabric configurations that HP sup
142. ware Maximum Supported Nodes and Adapter Cards In point to point configurations the complexity and performance limitations of having a large number of nodes in a cluster make it necessary to include switching in the fabric Typically point to point configurations consist of only 2 or 3 nodes In switched configurations HyperFabric supports a maximum of 64 interconnected adapter cards A maximum of 8 HyperFabric adapter cards are supported per instance of the HP UX operating system The actual number of adapter cards a particular node is able to accommodate also depends on slot availability and system resources See node specific documentation for details HyperFabric subsystem supports a maximum of 8 configured IP addresses per instance of the HP UX operating system Maximum Number of Switches 17 Planning the Fabric TCP UDP IP 18 You can interconnect mesh up to 4 switches 16 port fiber or Mixed 8 fiber ports in a single HyperFabric cluster Trunking Between Switches multiple connections You can use trunking between switches to increase bandwidth and cluster throughput Trunking is also a way to eliminate a possible single point of failure The number of trunked cables between nodes is limited only by port availability To assess the effects of trunking on the performance of any particular HyperFabric configuration contact your HP representative Maximum Cable Lengths HF2 fiber The maximum distance is 200
143. wing sections that describe the HyperFabric installation e Checking HyperFabric Installation Prerequisites on page 43 41 Installing HyperFabric e Installing HyperFabric Adapters on page 44 e Installing the Software on page 51 e Installing HyperFabric Switches on page 57 42 Chapter 3 Chapter 3 Installing HyperFabric Checking HyperFabric Installation Prerequisites Checking HyperFabric Installation Prerequisites Before installing HyperFabric ensure that the following hardware and software prerequisites are met V V r4 4 Check the HyperFabric Release Notes for known problems required patches or other information needed for installation Confirm that the usr bin usr sbin and sbin directories are in your PATH by logging in as root and using the echo PATH command Confirm that the HP UX operating system is the correct version Use the uname a command to determine the HP UX version For more information about the required operating system versions see HyperFabric Release Notes If you are installing an HF2 switch confirm that you have four screws with over sized heads Confirm that there are cables of proper length and type fiber to make connections in the fabric adapter to adapter adapter to switch or switch to switch Confirm that there is at least one loopback plug for testing the adapters and switches a fiber loopback plug HP part number 46384 67004 is shipp

HP HyperFabric Administrator's Guide

Contents

Download Pdf Manuals

Related Search

Related Contents