Home

upSuite™ User's Guide

Contents

1. 2 150 lt INTERFACE gt subtag of the lt NODE gt tag aaa aa aaa aaa aaa aaa 150 lt PARTITION gt subtag of the lt NODE gt tag aaa aaa aaa aaa aaa aa 151 lt HEARTBEA T gt TAG iui aso eet aina a asa a a a Ei an a as sia ao aa 152 NAME attribute sioria aa oe aaa aa aaa da sa a i ies ea eis sasas 152 TYPE AMA i i ai a a i a ee a 152 TIMEOUT_MSEC Attribute nn 153 RESEND MSEC ttrib te Lasas saaa Saias a a si asi sasas asis Usa Sre os ias 153 lt NODE_REF gt subtag of lt HEARTBEAT gt ccsscccsssceeseceseeenseeseceeseceeaeceseecnseeeneeeee 153 lt LINK gt subtag of lt HEARTBEAT gt penei a a r a E on nora non aaa as 153 lt SER VICE gt TAG td 154 NAME GTI ULE Senengane ei sek a ai as ai Sea tee coach RTE Se 155 SERVICE ID attrib te iais sai sasas ao i a ka 155 TYPE GUI DU Co Asano A a i ER Res 155 STARTUPDELAY SEC attribUlC ooooococcccccccnonononananancnnononannnnncncacocncnnnananananananonannnnonos 155 PORT GUTIDUE tir ie 155 lt NODE_REF gt subtag of the lt SERVICE gt tag aaa aaa aaa as 155 lt LINK gt subtag of the lt SERVICE gt tag LL aaa aaa saaa aaa aaa 156 lt SCSIBEAT gt subtag of the lt SERVICE gt tag LL aaa aaa aaa 157 lt HANFS gt subtag of the lt SERVICE gt tag aaa aaa aaa aaa aaa aaa aaa as 157 lt SERVICE IP gt subtag of the lt SERVICE gt tag a aaa aaa 158 CONFIGURATION FILE EXAMPLE c
2. Table 10 Changing the mount options in the ipf stab file Maximum Operations Option maxops You can configure the maximum number of replay operations ipfs will accrue before flush ing operations and switching to repair mode The default is roughly one million operations 1024 1024 or 10MB of memory depending on which occurs first Memory Allocation Option maxmem klmlg Sets the maximum amount of memory to be allocated for accumulated operations Table 11 defines the maximum memory options Note that units for memory may be specified in decimal values e g 1 5g Memory units Action k Sets the specified number in kilobytes m Sets the specified number in megabytes g Sets the specified number in Gigabytes Table 11 Maximum memory allocation units Continuous Computing Corporation upSuite User s Guide 123 11 The ipfstab File upDisk uses an efficient encoding mechanism for summarizing operations while the link is down An operation op is approximately 100 bytes Creates deletes and other metadata operations allocate an op and memory to store the filename The memory allocated is propor tional to the number of operations and the names involved Attribute operations chmod chown truncate coalesce The memory allocated is pro portional to the number of files on which such operations have been performed open 2 with the O_TRUNC flag or creat 2 system calls will generate a m
3. Continuous Computing Corporation upSuite User s Guide 43 5 upBeat API Reference ubNode NAME ubNode SYNOPSIS Get the node ID that corresponds to a given node name ub_node id t ubNode upbeatp A handle upbeat_t upbeatp char nodename as returned by ubInit nodename The NAME attribute of a lt NODE gt tag in the upSuite configuration file upsui DESCRIPTION te conf Get the ID from the upSuite configuration file for a node If the value of nodename is NULL ubNode returns the ID of the current node RETURN VALUES ubNode returns the node 1 D a positive integer if the handle passed in upbeatp is not NULL and it finds the node otherwise it returns 0 ERRORS None SEE ALSO ubNodeName on page 45 44 Continuous Computing Corporation upSuite User s Guide ubNodeName ubNodeName NAME ubNodeName SYNOPSIS Get the name of a node char ubNodeName upbeat_t upbeatp ub_node id t node upbeatp A handle as returned by ubInit node Node ID of the node for which you want to get the name Use the NODE ID attribute of a lt NODE gt tag in upsuite conf or O for the current node DESCRIPTION Get the name of a node from the upSuite configuration file Also useful for translating a node ID in a node link or service callback RETURN VALUES If the handle passed in upbeatp is not NULL the node exists and the node
4. Continuous Computing Corporation 156 upSuite User s Guide lt SERVICE gt tag be used in any way the application desires For example the PREF attribute could be used to establish the order in which the service uses the specified network This attribute can be used to direct traffic to a private interface or to a faster interface while still having other interfaces public and or slower as backups in case the most desirable link is unavailable If used the link preferences on both servers must match The PREF value must be a signed integer The default is 0 Example PREF 1 lt SCSIBEAT gt subtag of the lt SSERVICE gt tag Optional Can be added to any service where you need to monitor your SCSI disks more closely than Solaris allows The lt SCSIBEAT gt subtag allows upBeat to be informed when one or more disks partitions are required for the given service to be active If a disk is failed a server cannot become active for the service if a disk fails on the active server upBeat will ini tiate a failover lt DISK gt subtag and PARTITION attribute If you add the lt SCSIBEAT gt subtag to a service you must use the PARTITION attribute of the lt DISK gt subtag to list the disk partitions from the lt PARTITION gt subtag of the lt NODE gt tag see lt NODE gt tag on page 149 For more information about SCSIbeat see SCSIbeat on page 11 Example lt SCSIBEAT gt lt DISK PARTITION
5. ce cccessseceseceseceecesecesesaeecseseeeceeeceeseeeeseneeeseeeens 89 Bringing Systems Onlin cota ok ais ves de a tee Me ots dg o o escola ete ais 89 Operational Messages cai a a ia i aa is 9 Miscellaneous Transmission EV rr cccccccccccsscseecesecccscccccscccscesesessssensssususussacacssscscssesecess 94 TROUBLESHOOTING OTHER ISSUES cssssccceeesesssececesssseeececesusceccecesseaeceeceessaeceecsensaeeeceeeneeees 94 File System Access Denied anm 94 File System Out of Disk Spacer ini a o ies 95 ipfs Unmount Unsuccessful as savas asai i a a aaa a as 95 Conflicting File Modification Times Mtime L LL Lake aaa aaa aaa aaa 96 dol a a NN 96 Split Brain Conditions peas 97 Failover di AL ono nan nn cnc nr nc nnncnc nana nananannons 99 svc_max_msg_size Causes Problems when Adding CCPUdisk Package L 99 Continuous Computing Corporation upSuite User s Guide lll 9 FAILOVER spiicate nnee A A aka 101 UPDISK FAILOVER CONSIDERATIONS ccccccccccccnococononananononnnnonononcnnnonnnnncnnnnonancnnanononononcnnnnccnnnnos 101 NORMAL FATE OVER iia 101 Double TAU ES cant LS eo ee HE SE A tasa 102 IF ULE LOCKING oa ti eds hie asi 102 MANAGED EA ER soak a Las aa aa tea as Ta Oe Rs 102 BOOT SEQUENGE voii atrio i an ai an EEEE 104 10 HIGH AVAILABILITY NFS HA NES cccccccceccccsceeeecececeeueeeeeveeeeeetetenenanaeavanens 107 How CAN UPDISK AID MY NFS SERVER ccccccccccccnconononanananoncnnonononanancncno
6. A UpSuiteConfig VERSION 2 gt lt UPBEAT STARTUPDELAY_SEC 5 gt lt NETWORK NAME Network1 DESCRIPTION 192 168 1 0 24 gt lt NETWORK NAME Network2 DESCRIPTION 192 168 2 0 24 gt lt NODE NAME left NODE_ID 1 DESCRIPTION SPARC CP1500 Solaris 2 7 gt lt INTERFAC AME hme0 NETWORK Network1 IP 192 168 1 2 gt lt INTERFACE NAME hmel NETWORK Network2 IP 192 168 2 2 gt lt NODE gt lt NODE NAME right NODE_ID 2 DESCRIPTION SPARC CP1500 Solaris 2 7 gt lt INTERFACE NAME hme0 NETWORK Network1 IP 192 168 1 3 gt lt INTERFACE NAME hmel NETWORK Network2 IP 192 168 2 3 gt lt NODE gt lt HEARTBE NAME left right YPE POINT_TO_POINI IMEOUT_MSEC 500 ESEND_MSEC 150 gt E_REF NODE_ID 1 gt E_REF NODE_ID 2 gt NK NETWORK Network1 ROUTE ROUTE gt NK WORK Network2 ROUTE ROUTE gt lt HEARTBEAT gt Ol Y Continuous Computing Corporation upSuite User s Guide 159 14 upSuite Configuration File upsuite conf lt SERVICE NAME myService ipfs SERVICE_ID 17 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NO
7. Figure4 Redundant NFS server running upDisk Continuous Computing Corporation 108 upSuite User s Guide File Handle Persistence During Failover Two HA NFS Architectures There are essentially two ways to construct an HA NFS environment both of which are illus trated in Figure 4 The architecture you construct will depend on your particular needs Note that Client 2 is connected to the system via each of the two Ethernet links and Client 1 is connected to the system via one Ethernet link In the Client 1 architecture we assume that the pair of servers are running upDisk as well as upBeat In the Client 2 architecture we assume that in addition to the pair of servers running upDisk and upBeat the client is running upBeat as well The Client 1 architecture has an inherent single point of failure its one connection to the net work The Client 2 architecture has no single point of failure In addition the NFS address may be on the local subnet or any other network upBeat modifies the routing tables on the client the cli ent is unaware of NFS address migration during failover due to upDisk and ARP However be aware that ARP will still notify any non upBeat clients of NFS address migration during failover File Handle Persistence During Failover As illustrated in Figure 5 without upSuite HA the active server associates the file handle 1234 with the foo bar file the NES client is using When the active server fails ove
8. a colon and the version number of the ubManager semicolon delimited configuration file format only version 3 is supported Your first line must look similar to the following 3 Configuration File Format 3 Don t change this line Continuous Computing Corporation upSuite User s Guide 209 18 ubManager Semicolon Delimited Configuration File Group Definition The g parameter defines a group which must have a corresponding lt SERVICE gt element in the upSuite configuration file etc upsuite upsuite conf If the group is to fol low a lead service then a lead service must also be specified which must also have a corre sponding lt SERVICE gt element in the upSuite configuration file USAGE DESCRIPTION g group_name lead service This entry defines a group The group_name and lead_service which is optional must be defined as services in the upsuite conf configuration file If lead_service is defined ubManager will control group_namess state so that it tracks that of lead_service If lead_service is not defined the group will become active on the node on which ubManager runs first Example g group1 updisk ipfs Monitor Definition The m parameter defines a monitor associated with a group Each monitor must have a corre sponding lt SERVICE gt element in the upSuite configuration file for the node it is to run on 210 Continuous Computing Corporation upSuite User s Guide Monito
9. upBeat will send it a directive to go standby and send its peer a directive to go active Handling of various service registration situations The previous section describes how service registration typically occurs However it is possi ble for clients to issue registration or acknowledgement calls that are different from those that would typically be expected For example suppose a service is given a directive to go standby but instead of acknowledg ing and accepting the directive the service acknowledges the directive with an active status In this case the acknowledgement is ignored by upBeat and the service must assume that it is standby If a standby service without being given a directive calls ubAck Svc that unexpected acknowledgement is ignored Likewise if an active service calls ubAckSvc with active status without being given a directive the acknowledgement is ignored However if an active service calls ubAckSvc with standby status without being given a directive to go standby the service will automatically be given standby status Its peer if run ning will get a directive to go active Continuous Computing Corporation upSuite User s Guide 27 4 upBeat API Programmer s Guide If a service is given a directive to go active and the service acknowledges the directive with a standby status its peer service will get a directive to go active If the peer service also acknowledges its active direc
10. Get the service ID of a named service ubSvcIPPair Get a pair of IP addresses on the same network for a pair of servers configured to provide a service ubSvcName Get the name of a service ubSvcPeer Get the node ID of the other server that is offering a service ubSvcPort Get the TCP or UDP port number for the service Table 2 Quick reference of function calls Continuous Computing Corporation upSuite User s Guide 35 5 upBeat API Reference ubAckSvc NAME ubAckSvc SYNOPSIS Respond to a UB ACTIVE or UB STANDBY directive int ubAckSvc upbeat t upbeatp ub svc id t service ub svc status t svc status upbeatp A handle as returned by ubInit service A numeric service ID corresponding to the SERVICE ID attribute for a lt SERVICE gt tag in the upSuite configuration file svc status UB ACTIVE or UB_STANDBY DESCRIPTION The application uses ubAckSvc in its directive callback to respond to a UB ACTIVE or UB_STANDBY directive from upBeat The application may agree with upBeat or it may down grade the directive from UB_ACTIVE to UB_STANDBY but it may not upgrade the directive from UB_STANDBY to UB_ACTIVE RETURN VALUES ubAckSvc returns O if it successfully sends the message to upBeat otherwise it returns 1 to indicate failure In most high availability applications failure should be treated as a catastrophic failure and the application should shut down ERRORS See Error codes o
11. and other commands to learn the state of the network wrapper then registers the service that was passed in requesting active status It then waits for upBeat to designate the service as standby or active If upBeat directs the service to be standby wrapper waits indefinitely However if upBeat directs the service to become active wrapper then runs the given command in the example later in this section iostat 10 is used and waits for it to finish After the command has finished wrapper registers the service as standby upBeat then fails the service over to an available server If there is no server available upBeat will make the ser vice active again on the same server The source code for wrapper is located in wrapper c Continuous Computing Corporation upSuite User s Guide 57 6 upBeat Sample Applications To Use wrapper Step Description Command 1 Change directory cd 2 Copy the directory src upbeat cp r opt upsuite src upbeat myupbeat 3 Gothe myupbeat directory cd myupbeat 4 Build using the supplied make file make 5 Run wrapper on both servers Press wrapper lt service gt lt command gt Ctrl C on the active side to cause a failover On the new active side press Ctr1 C to failback Note To best demonstrate upBeat failover use a command that does not return imme diately e g iostat 10 This way your program will run the command every so many seconds Note To stop wrap
12. ipfs ufs c0t0d0s7 logging 3 If necessary edit the ipfstab file to look similar vi ipfstab to the example below For complete explanation of each of the fields of the ipfstab file see ipfstab settings on page 117 ipfstab edited for sharing data ipfs ipfs localfs localfs localfs locales share set mount mount mount mount name point opts point type device opts opts ipfs ipfs ipfs ufs c0t0d0s7 logging rw 4 Start upDisk by either rebooting or manually restart Reboot ing both the active and standby systems or etc init d updisk start 3 Configure clients to run upBeat optional If you have clients on redundant networks with the upDisk servers you can increase the avail ability of your NFS partitions by running upBeat on the client machines to manage the redun dant network connections To do so follow these steps 1 2 Install upbeat on all clients Refer to the upSuite Installation Guide for instructions Update each server s upsuite conf file to include the client nodes and define heart Continuous Computing Corporation upSuite User s Guide 113 10 High Availability NFS HA NFS beats between the clients and servers To do this add a lt NODE gt tag for each client and add lt HEARTBEAT gt tags to define heartbeats between the client and server machines You do not need to define heartbeats between the clients For more information about how to define these tags see
13. on page 127 Normally upDisk mounts an entire file system It is also possible to mount only a directory However upDisk cannot run sck on a directory therefore this practice is recommended for testing purposes only This section provides explanations of the ip fstab file settings Note The ipfstab file resides in the etc upsuite directory etc upsuite ipfstab however for convenience the symbolic link etc ipfstab is created during upDisk installation This symbolic link redirects the short name to the actual file in etc upsuite You may use either the et c or the etc upsuite path to refer to and edit this file This is similar to how Solaris manages the hosts file in etc and etc inet ipfstab settings Each line in the ip fstab file includes the following settings in this order e data set name e ipfs mount point e ipfs mount options e localfs mount point localfs type e localfs device e localfs mount options e share options Continuous Computing Corporation upSuite User s Guide 117 11 The ipfstab File The settings are separated by white space a hyphen indicates a setting for which no value is specified For example mydata mydata data set name ipfs mount point ipfs mount options localfs mount point localfs type mydata ufs cntndnsn rw noatime logging This uniquely identifies the dataset that is replicated between the upDisk servers The dataset name
14. per you will need to press Ctr1 C twice The first stops the command the second stops the program leftf wrapper upstart iostat 10 Source Code ofwrapper tinc tinc tinc tinc tinc tinc tinc tinc tinc tinc tinc lude lt stdio h gt lude lt errno h gt lude lt poll h gt lude lt string h gt lude lt stdlib h gt lude lt unistd h gt lude lt sys types h gt lude lt sys errno h gt lude lt netinet in h gt lude lt arpa inet h gt lude lt libupbeat h gt 58 Continuous Computing Corporation upSuite User s Guide The wrapper Sample Application ifndef define endif ifndef define endif static static static static int main in TRUE TRUE 1 FALSE FALSE 0 void directive void ub_svc_id_t ub_svc_status_t ub_svc_id_t glob_sid ub_node_id_t glob_id glob_peerid char glob_prog t argc char argv upbeat_t ubp NULL char svcname ub_service_ip_t service_ip sip amp service_ip ub_ippair_t ippair ippp ippair These are the callbacks we will pass to ubGetState and ubAsync UpBeat passes ubp as an argument to the callbacks We do not have node link and service callbacks we do have a directive callback so we can negotiate active or standby El ub_ops_t ub_ops ubp not set yet NULL no node callback NULL no link callback NULL no service callb
15. upSuite Configuration File upsuite conf on page 145 For an example configuration file see Configuration C on page 170 3 Copy the upsuite conf file to each client machine The upsuite conf file should be identical on all servers and clients 4 Start upBeat on all machines using etc init d upbeat start 5 Start upDisk on all the servers using etc init d updisk start NFS over UDP vs TCP NES can run over UDP or TCP Most clients that are TCP enabled will try that before trying UDP By default Solaris NFS servers will serve both UDP and TCP The default behavior of upDisk HA NES is to serve UDP you can modify this by using the PROTOS attribute of the lt HANFS gt tag in upsuite conf NFS over UDP has much better failover characteristics than NES over TCP therefore UDP is recommended You may designate that the clients use UDP or you may designate that the serv ers use UDP or both In most cases it is simpler to configure the servers to use UDP On Solaris clients you can force NFS to use UDP by specifying prot o udp as a mount option in et c vfstab On Linux clients you can force NFS to use UDP by specifying udp as a mount option in etc fstab See the nfs 5 man page for details To find out how to specify UDP on MacOS X or any of the third party NFS clients for Macintosh or Win dows refer to their documentation Running the Solaris nfs daemon with CCPU HA NFS If you run the upDisk HA NFS and standard
16. upbeat_t arg switch status case UB_ACTIVE printf directive UB_ACTIVE n if ubAckSvc ubp glob_sid status 1 fprintf stderr ubAckSvc failed errno d s n errno strerror errno exit 1 break case UB_STANDBY printf directive UB_STANDBY n if ubAckSvc ubp glob_sid status 1 fprintf stderr ubAckSvc failed errno d s n errno strerror errno exit 1 if rereg Give the standby a chance to beat us in From an HA perspective this is not really necessary if we really are ready to run again but for demonstration purposes it forces a failover iv sleep 3 if ubRegSvc ubp glob_sid UB_ACTIVE 1 fprintf stderr ubRegSvc failed errno d s n errno strerror errno exit 1 rereg FALSE break Continuous Computing Corporation upSuite User s Guide 63 6 upBeat Sample Applications default printf bad directive d n status exit L break if status UB_ACTIVE system glob_prog Tell upBeat we are no longer serving if ubRegSvc ubp glob_sid UB_STANDBY 1 fprintf stderr ubRegSvc failed errno d s n errno strerror errno exit 1 Once we have acknowledged standby reregister for active rereg TRUE return The status Sample Application status demonstrates upBeat s installation verification process Its output notifies you of all available links
17. void ubp printf upbeat is running n Find our node in the upSuite configuration if node id ubNode ubp NULL 0 printf this system does not match any node in the upSuite configuration n else printf this system is node d n node_id Get the current state Any changes to this state will be reported afterward via ubAsync if ubGetState ubp opsp 1 fprintf stderr ubGetState failed errno d s n errno strerror errno exit 1 while quick struct pollfd pollfd pfdp amp pollfd int poll_timeout_msec 500 Fill in the pollfd fresh each time libupbeat will reestablish a connection if it is dropped so the fd may change ubSetupPollfd ubp pfdp We are only polling for upBeat but we could be polling for any number of file descriptors XJ Continuous Computing Corporation upSuite User s Guide 67 6 upBeat Sample Applications if poll pfdp 1 poll_timeout_msec 1 fprintf stderr poll failed d s n errno strerror errno exit 1 We could check the return value from poll and the contents of pollfd to see if we have any events but ubAsync checks again anyway plus it gets a chance to do some housekeeping if ubAsync ubp opsp 1 fprintf stderr ubAsync failed errno d s n errno strerror errno exit 1 ubFini ubp exit 0 static vo
18. 192 168 1 2 lt SERVICE_IP IP 192 168 1 3 lt SERVICE gt lt SERVICE NAME Mt SysCmd lt S SERVICE ID 51 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1801 gt lt NODE_REF NODE ID 1 gt lt NODE REF NODE ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE_IP IP 192 168 1 4 lt SERVICE_IP IP 192 168 1 5 ERVICE gt lt UpSuiteConfig gt IF hme0 20 gt IF hme0 21 gt IF hme0 22 gt IF hme0 23 gt IF hme0 24 gt Continuous Computing Corporation upSuite User s Guide 71 6 upBeat Sample Applications System Architecture The architecture is that of an HA system with two nodes configured identically with the same software components on each One node currently contains the services in the standby state while the other contains the services in the active state The application provides two services with each service on different floating IP addresses and ports A CCN control node can be used to control the nodes rebooting and bringing links up and down to start and stop the application and to view the generated log file One or more telnet client machines are used to communicate with the services on either system mt_multi_svc creates a log file in the tmp mt_multi_svc directory on each sys tem where application activity is recorded 72 C
19. Each HA NFS upDisk dataset must have its own unique IP address and interface This migratory IP address can be any IP address that is not already in use In addition it does not necessarily need to be on the same subnet as the static IP addresses in the lt NODE gt tags If all of your clients are running upBeat then the subnet is unimportant If any of your clients are not running upBeat you should pick an IP address and an interface or more accurately a sub interface that your clients can reach Note Each upDisk dataset must have its own unique PORT Perform the following steps on both systems active and standby Step Description Command 1 Log in as the superuser root 2 Go to the configuration file directory cd etc upsuite Continuous Computing Corporation upSuite User s Guide 111 10 High Availability NFS HA NFS Description Command 3 Edit the lt SERVICE gt tag of vi upsuite conf upsuite conf by adding a lt SERVICE_IP gt subtag which speci fies the IP address and an interface and an lt HANFS gt subtag Typical lt SERVICE gt tag lt SERVICE NAME updisk ipfs SERVICE ID 17 TYPE BASIC PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE gt Same entry edited for HA NES lt SERVICE NAME updisk ipfs SERVICE_ID 17 TYPE BASIC PORT 1776 gt
20. File handles without upSuite HA Figure5 Stale vs persistent file handles Building an HA NFS Server Outlined below are the basic steps necessary to build a high availability NFS server The remainder of this chapter provides more detailed instructions for how to complete each of the following steps 1 Add an IP address interface and lt HANFS gt tag to upsuite conf to reflect your HA configuration The IP address and interface are specified in the lt SERVICE_IP gt subtag of the lt SERVICE gt tag Make sure that upSuite conf contains one and only one Continuous Computing Corporation upSuite User s Guide 110 Building an HA NFS Server lt SERVICE_IP gt tag per HA NES service CAUTION If a lt SERVICE gt tag contains an lt HANFS gt subtag and does not contain exactly one lt SERVICE_IP gt subtag an error will occur and upDisk will shut down 2 Share the dataset in etc ipfstab 3 Optionally configure clients on redundant networks to run upBeat In addition you will need to decide whether you will run NES over TCP or UDP and whether you want to share replicated subdirectories or entire file systems These issues are discussed in further detail below Edit the configuration file Edit upsuite conf to reflect your HA configuration The IP address you choose will be assumed by the active upDisk server and will be configured or unconfigured as appropriate on the interface you choose Note
21. These processes are called monitors and are themselves upBeat services If any monitor for a group indicates that the resource it s monitoring is unhealthy ubManager will failover the group provided that all the monitors for the peer group on the peer node indi cate their resources are healthy Continuous Computing Corporation upSuite User s Guide 181 15 Introduction to ubManager Resource Monitoring A ubManager group can be configured to use a number of monitors to monitor the health of the resources required by the group and so determine whether a group can become or can continue to be active Monitors are upBeat services However monitors on one node are independent of monitors on other nodes they do not have upSuite peers here upBeat is only used as an event manager informing ubManager of the state of peer groups monitors Monitors may be shell scripts or application binaries they may run in a transient or permanent manner and they may have their own independent interface with upBeat or have ubManager handle that interface for them Each monitor is designed to watch a particular element of the system and report status back to ubManager either by the monitor s exit value or through upBeat If any one monitor for a group indicates that the resource it s monitoring is unhealthy then ubManager will failover a group if all of the monitors for the peer group indicate that the peer group s resources are health
22. nodes and services of which upBeat is aware However status does not offer any services The source code for status is located in status c To Use status Running status with the q flag causes st at us to print upBeat s full node link and ser vice status once then exit Running it without the q flag causes status to print upBeat s full node link and service status once then run indefinitely printing any changes to status To stop sample use Ctrl C Step Description Command 1 Change directory cd 2 Copy the directory src upbeat cp r opt upsuite src upbeat myupbeat Continuous Computing Corporation 64 upSuite User s Guide The status Sample Application Step Description Command 3 Gothemyupbeat directory cd myupbeat 4 Make a file make 5 Run status status Note To stop sample press Ctrl C left status Source Code of status include lt stdlib h gt include lt stdio h gt include lt errno h gt include lt poll h gt include lt string h gt include lt unistd h gt include lt sys types h gt include lt sys errno h gt include lt netinet in h gt include lt arpa inet h gt include lt libupbeat h gt ifndef TRUE define TRUE 1 tendif ifndef FALSE define FALSE 0 tendif static void unode void uint32 t ub status t boolean t static void ulink void uint32 t ub status t boolean t static
23. operator shutdown upBeat or upDisk decided that the link was malfunc tioning or should not be in place at which point it shut down the link As the operator you can also shut down the link manually thus generating this message as well active error There is a problem with the active system ipfs will drop the link to avoid possible damage upBeat will initiate failover standby error There is a problem with the standby system ipfs will drop the link to avoid possible damage link daemon killed The upDisk daemon was terminated unexpectedly This may have been done by the operator or it may have been done by the system as part of the normal shutdown sequence If appropriate the watchdog daemon will restart upDisk if there are persistent errors a shutdown is in progress or the operator killed the watchdog daemon upDisk will not restart Continuous Computing Corporation upSuite User s Guide 93 8 upDisk Administration Message Meaning link error ipfs attempted to use the link and discovered an error i pfs then terminates communication with the other system and notifies upBeat and upDisk of this event ipfs noticed the link had a problem before upBeat did This is atypical and may indicate the daemon is not present Use udstat for more information Table 5 Comments in console state output Miscellaneous Transmission Errors It is unlikely but possible
24. s NAME attribute has a value ubNodeName returns the name otherwise returns NULL ERRORS None SEE ALSO ubNode on page 44 Continuous Computing Corporation upSuite User s Guide 45 5 upBeat API Reference ubRegSvc NAME ubRegSvc SYNOPSIS Register to provide a service int ubRegSvc upbeat t ub_svc_id t service ub_svc_status_t svc_status upbeatp A handle as returned by ubInit service A numeric service ID corresponding to the SERVICE_ TD attribute for a lt SERVICE gt tag in the upSuite configuration file upsuite conf svc_status UB_ACTIVE or UB_STANDBY Indicates whether the application prefers the service to have active or standby status DESCRIPTION An application uses ubRegSvc to register to provide a service The application provides svc_status to indicate whether it prefers to be active or standby but there is no guarantee upBeat will honor the preference The application can call ubRegSvc again later for the same service to try to change the active standby status of a service for which it has already regis tered Later upBeat will indicate via the application s directive callback whether the application should assume an active or a standby role At that time the application should use ubAckSvc to confirm or deny the role RETURN VALUES ubRegSvc returns 0 if it successfully sends the request to upBeat
25. startup down summary previously BUSY setting DIRTY This message is the result of a system being powered off or crashing while having outstanding Operations and indicates a repair will be performed Operational Messages Messages are sent to the console during normal upDisk operations The role link and state of any given mount point are displayed with each output message In addition informational comments may appear to the right of the role link and state fields separated by a hyphen These informational comments are explained in the rest of this section Console messages appear in the following format servername mountpoint role link state status msgs Role active standby or startup The role field indicates whether this file system has assumed the active standby or startup role Message Meaning active This system is the active one Applications have read write access and changes are sent to the standby system standby This system is the standby one Applications have read only access and changes on the active system are propagated to the standby startup A role active or standby has not yet been assigned The startup role should rarely be seen Table 3 File system roles Continuous Computing Corporation upSuite User s Guide 91 8 upDisk Administration Link up or down The link field indicates whether the link between the active and standby ipfsi down the only two
26. ublnit on page 43 Continuous Computing Corporation 40 upSuite User s Guide ubGetState ubGetState NAME ubGetState SYNOPSIS Get synchronous state stored in Libupbeat int ubGetState upbeat_t upbeatp const ub_ops_t opsp upbeatp A handle as returned by ubInit opsp Pointer to an ub_ops_t structure DESCRIPTION The opsp argument points to a ub_ops_t structure which gives upBeat the client s node link service and directive callback functions if any For more information about this structure see ubAsync on page 37 An application may call ubGet State to get a snapshot of the state from 1ibupbeat The information included in this snapshot is whatever state was stored after the last call to ubInit orubAsync and is therefore not necessarily up to date with the current state of the system The snapshot includes the state of the nodes defined in the upSuite configuration file upsuite conf with which the current node shares a heartbeat the links associated with those nodes and the services associated with those nodes The node link and service callbacks will be called once for each item This is different from the behavior of ubAsync which only calls the callbacks if state has changed Typically an application should call ubGet State only during startup just after calling ubInit Thereafter the application must use ubSetupPollfd andubAsync to keep informed
27. File upsuite conf Continuous Computing Corporation 178 upSuite User s Guide Part Ill ubManager ubManager is a software layer situated above upBeat the heartbeat manager of upSuite Where upBeat detects link network and node failures ubManager extends this functionality with its ability to e Start stop and monitor non HA aware user application processes e Group services and coordinate failover among them e Add control through programs and scripts Monitor resources Group starts checks exit values for dumb monitors gt starts stops gt Registers group and dumb monitors checks lead service and smart monitor services Lead Service ubManager Monitor Services v normal upBeat interface for lead lt service and smart monitors Figure 1 ubManager and upBeat Continuous Computing Corporation upSuite User s Guide 177 Continuous Computing Corporation 178 upSuite User s Guide 15 Introduction to ubManager ubManager short for upBeat Manager and pronounced You Be Manager is designed to integrate non HA aware user applications into the upSuite HA framework This guide provides the following information related to ubManager e Description of features Command reference e ubManager Monitors We assume that as a user of ubManager you are familiar with e The Solaris operating system e TCP IP and networking e XML Notation e Continuou
28. If there is an active ser vice that depends on that disk upBeat tells the service to go standby i e a failover is initiated SCSIbeat monitors disk statistics to make sure the disk is operating SCSIbeat also sends peri odic commands to the disk this ensures there will be some activity on an otherwise idle disk and it allows SCSIbeat to discover disk errors These commands involve little overhead and provide rapid feedback about the disk s health You can adjust how often the command is sent and the timeout before SCSIbeat informs upBeat that the disk is not responding for more information see lt NODE gt tag on page 149 SCSIbeat monitors disk partitions because Solaris provides programmatic access to disks by providing programmatic access to partitions To allow for different disk partition names on each node you can specify a name for each disk partition by using the the lt PARTITION gt sub tag of the lt NODE gt tag in the upSuite configuration file You can then use the lt SCSIBEAT gt tag to configure a given service to depend on the partition by name For more information about these tags see upSuite Configuration File upsuite conf on page 145 If you use Solaris DiskSuite to mirror disks which you want to monitor with SCSIbeat monitor the DiskSuite device for example dev md rdsk d3 not the underlying disk device for example dev rdsk c0t0d0s3 You must also specify a long timeout period to avoid unnec
29. Monitoring upDisk Monitoring upDisk You can monitor the file system with the udstat command See udstat on page 132 for further explanation and usage of this command On Solaris 8 you can also get ipfs statistics using the kst at command On Solaris 7 the kstat ioctl 2 interface is supported but the kstat 1M command does not exist Logs upDisk s status can be monitored via the syslog Typically all ip fs messages are routed to var adm messages upDisk daemon messages are typically routed to var log upsuite Critical messages are automatically routed to your console The file var 10g nidb contains error information from an upDisk component a pro gram called the nidb daemon which is used for file handle translation Troubleshooting Console Messages This section discusses console messages you might see that indicate some common issues that arise when using upDisk and tells how to respond to these events Bringing Systems Online When ipfs is first provided a link for use internal ip fs consistency checks are made between the two systems If these checks do not turn up any problems or turn up problems that are not urgent ipfs will continue running Otherwise ipfs drops the link and informs upDisk of the problem In at least one case the warning message may only indicate that ipfs does not have all the information upDisk does about a given situation You may encounter one or more of the following me
30. NODE NAME server2 NODE_ID 2 gt lt INTERFACE NAME hme0 NETWORK Network1 TP 172 17 8 141 gt lt INTERFACE NAME hmel NETWORK Network2 IP 172 18 8 141 gt lt NODE gt lt HEARTBEA NAME serverl server2 YPE POINT TO POINT IMEOUT MSEC 500 RESEND_MSEC 150 gt DE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt B lt HEART Continuous Computing Corporation upSuite User s Guide 163 14 upSuite Configuration File upsuite conf lt SERVICE NAME updisk ipfs SERVICE_ID 17 YPE BASIC STARTUPDELAY_SEC 5 PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE_IP IP 172 17 8 177 lt SERVICE gt E T lt UpSuiteConfig gt The etc ipfstab File IF hme0 17 gt Below is the etc ipfstab file edited to match the system illustrated in Figure 7 data ipfs ipfs localfs localfs localfs set mount mount mount mount name point opts point type device ipfs ipfs ipfs ufs c0t0d0s7 The etc hosts File localfs share opts opts rw noatime logging rw Below is the etc host s
31. RETURN VALUES ubGetState returns 1 if the connection to the local upBeat daemon is lost and cannot be reestablished O otherwise In most high availability applications failure should be treated as a catastrophic failure and the application should shut down ERRORS See Error codes on page 55 Continuous Computing Corporation upSuite User s Guide 41 5 upBeat API Reference SEE ALSO ubAsync on page 37 Callback Functions on page 34 Continuous Computing Corporation 42 upSuite User s Guide ublnit ublnit NAME ubInit SYNOPSIS Initialize a connection to the local upBeat daemon upbeat_t ubInit DESCRIPTION The application must call ubInit before calling other upBeat API functions ubInit allocates resources and establishes a connection to the local upBeat daemon Calls to ubInit will block until the upBeat startup delay period set in the configuration file has expired When finished the application must call ubF ini to shut down the connection and free resources allocated by ubInit RETURN VALUES On success ubInit returns an upBeat handle that is to be passed to all other API functions on failure ub Init returns errno ERRORS ubInit fails if it cannot allocate resources or if it cannot establish a connection to the local upBeat daemon and sets errno See Error codes on page 55 SEE ALSO ubFini on page 40
32. SAMPLE APPLICATION cccccccnoccccnoconincnnacananananonononononononononononnononaracananana nano aaa 64 To Use Status a a a S a ai a a i aid 64 Source Code OF SUAUUS aks svei sa ais a aaa a aa aaa na a a a ai 65 THE mt multi svc SAMPLE APPLICATION ccccccccessccssssecsecssecescsscssecessesscessesseessesssessesseceecaees 69 Configuration Ple sio a api a ai a a a Ee 70 System Architecture sis Ia A da hed tee 72 Application ArchitectUte sssini iaeaea 74 LEMA e a i e S a 73 To e Tat TUM Cutral ltda d ia 76 FE INTRODUCTION TO UPDISK oceneni a a 81 WHAT IS UPDISK Po ia bets shies oa a aaa A a a E E eo iais 81 WHY SHOULD LTL USB UPDISK 5 sccccocscosdbesocscebcosvocaccoce so in Le S e ao ici caia 81 HOW DOES UPDISK WORK ccsesssccecesssseececeessaececceessaececcecssaececcecesueceseeceessceceeceessaesecesensaeees 82 8 UPDISK ADMINISTRATION ccccececeeeeeeeecteeeeeeeeteeseeeeateeeaeeesteeeatenreeeatensneneates 87 UPDISK ADMINISTRATOR CONSIDERATIONS sc ssccccesssssecececssaececcecssaececeecsessseseeceessaesecesensaeees 87 Stopping the Solaris machine LL aaa aaa aaa saaa aaa aaa aa aaa aaa non aaa aks 67 Stopping the NFS Serv r anio sa ass Oh tyson a is Ss a i k dap a is 67 Manipulating underlying file SYSteMS oononnnnnnnnnnnnnonnnnnnanonnc aaa aaa aaa kaka aaa aaa 66 Shutting down the system when a link is UD 2LL Lasa aaa aaa aaa aaa aaa aaa 66 MONITORING UPDISK cuina tas 89 TEOG T 89 TROUBLESHOOTING CONSOLE MESSAGES
33. Solaris nfs daemons on the same server the stan dard Solaris daemons might serve upDisk s shares resulting in errors during failover This issue arises if you configure UDP or TCP but not both By default the Solaris NFS daemon nfsd serves both TCP and UDP in comparison the upDisk HA NFS serves only UDP by default The Solaris NFS service is typically enabled by placing entries shares in etc dfs dfstab Solaris clients and possibly others try TCP first then fall back to UDP If the Solaris nfsd is serving protocols that the updisk HA NFS is not serving such as TCP and a client tries to use one of those protocols the client mounts the upDisk shares from the Solaris nfsd server which will cause errors during failover Continuous Computing Corporation 114 upSuite User s Guide Sharing Subdirectories vs File Systems We recommend you do not run both the HA NFS provided by Continuous Computing and the standard Solaris nfs daemon on the same server in other words we recommend against shar ing file systems via etc dfs dfstab The Continuous Computing HA NES default is UDP only and the Solaris default is both UDP amp TCP Therefore if you want to run both servers not recommended you must be careful about which protocols you serve One technique would be to change the HA NFS configuration to run both protocols set the PROTOS attribute to PROTOS UDP TCP Also note that it is critical that you use the HA N
34. aa dons ae vende a a ais 141 Sample APLCOd sz Linskis as iseina Goa a sa ss o ias k in a a a a i 141 API Overview Using upDisk s API is optional Most applications can simply use the upDisk file system with out modifications as they would any other file system However upDisk s API consists of one ioct1 function call which allows you to learn the following information e Whether you are on a valid upDisk file system e Whether you are on the active file system Function Call Guidelines All programs must include ipfs h This file is stored in the directory opt upsuite include Sample API Code Below is a sample program using upDisk s API function call include lt stdio h gt include lt errno h gt include lt fcntl h gt include lt string h gt include lt unistd h gt include lt sys types h gt includ E e y e include lt stdlib h gt e a e lt sys stat h gt e include lt sys param h gt include lt ipfs h gt Continuous Computing Corporation upSuite User s Guide 141 13 upDisk API char fs_state2string int fs state char fs_role2string int fs state int main int argc char argv char path handle MAXPATHLEN 1 int n fd status int wait This program determines whether a mount point is an IPFS upDisk mount point or whether a file or directory is in an IPFS file system if argc 2 fprintf stderr Usage
35. active if a split brain is detected The services of the other node will become standby Continuous Computing Corporation upSuite User s Guide 77 6 upBeat Sample Applications Continuous Computing Corporation 78 upSuite User s Guide Part Il upDisk upDisk is a software module that provides simple reliable file system replication over a net work By relying on redundant systems storage and networks upDisk allows system configu rations that prevent any single point of failure upDisk continually replicates data over a redundant network to a standby file system upDisk is used as part of upSuite HA and is available for Solaris 7 8 9 and 10 on SPARC Ethernet Switch Ethernet Switch SCSI Fibrechannel RAID array Figure 1 Example upDisk configuration Continuous Computing Corporation upSuite User s Guide 79 80 Continuous Computing Corporation upSuite User s Guide 7 Introduction to upDisk This chapter introduces you to upDisk a software module used to replicate file system data as part of upSuite HATM a series of software modules designed to provide transparent high availability capability for applications In this chapter Whats tpDis k L takais eats nie Reese EG aoe At ee ate a 81 Why Should I Use upDisk yi a saint saksai nis sagas Gu ii is o Ai sa p nani cree 81 How Does upDisk Work cessccssc sk csicciescasscscbens a a a a E TE al
36. admin command NAME upbeat USAGE usr sbin upbeat c d s S v h V DESCRIPTION This command should not normally be run directly it is run from the upbeat startup shutdown script as part of system startup If you want to start or stop upBeat manually use the upbeat startup shutdown script This command starts the watchdog and upBeat daemons If you weretorunps ef grep upbeat you would see two upBeats running these are the upBeat processes One pro cess acts as watchdog and will restart the main upBeat process the daemon if it stops unexpectedly Continuous Computing Corporation upSuite User s Guide 17 3 upBeat Commands OPTIONS FILES SEE ALSO Running this command without any arguments causes upBeat to run as a daemon Program messages are sent to syslog Including the c flag causes program messages to be sent to your console in addition to any other place they are being sent for example to syslog Including the d flag runs upBeat in debug mode so that upBeat runs in the foreground Messages that would normally go to syslog will output to the terminal along with additional debugging information The s flag forces program messages to be sent to syslog even if for example you have already run the command with the d flag The S flag causes program messages to not be sent to sys log even if you have already run a command which would normally route them there Th
37. ak a a 50 AA O 52 UDSVCPEEI E e Ae A ee A Sios 53 IIS CPU A A e A Ae ee EE 54 Error Codes de 55 Continuous Computing Corporation upSuite User s Guide 33 5 upBeat API Reference Callback Functions The following table lists the major upBeat daemon callback functions used by the upBeat API For additional information see ubAsync on page 37 Callback Description node Gives the state of a node This function is called as a result of a call to ubAsync orubGetState ubAsync calls this function whenever a change of state occurs to a node with which the current node shares a heartbeat When ubGet State is called this function is called for every node in the configuration except the current node The node the software is running on is always UP since the soft ware is running link Gives the state of a link This function is called as a result of a call to ubAsync orubGetState ubAsync calls this function whenever a change of state occurs to an inter face of any node with which the current node shares a heartbeat When ubGet State is called this function is called for every link in the configuration service Gives the state of a service including the node if any on which the service is active This function is called as a result of a call toubAsync orubGetState ubAsync calls this function whenever a change of state occurs to a service When ubGet State is called this function is
38. be as follows Continuous Computing Corporation upSuite User s Guide 87 8 upDisk Administration 1 The following command stops the Solaris NFS server and unshares all df stab and HANES shares etc init d nfs server stop 2 The following command restarts the Solaris NFS server etc init d nfs server start At this point HANES shares are not shared 3 The following commands stop and restart upDisk etc init d updisk stop etc init d updisk start After these commands run HANFS shares reappear Manipulating underlying file systems Do not manipulate an underlying file system or subdirectory in any way Doing so will at the very least result in all system changes going unreplicated at the very worst both servers will shut down Shutting down the system when a link is up upDisk file systems cannot be unmounted if the pair of servers has an established link Under some circumstances this can cause the system to hang during a shutdown Normally the system calls etc init d updisk stop as part of the shutdown sequence However if you run either usr ucb shutdown or init 0 to shut down the system be aware that these cause the system to initiate single user mode without going through the full shutdown sequence To avoid this problem use one of the following to shut down your system halt etc shutdown usr ucb shutdown h init s init S Continuous Computing Corporation 88 upSuite User s Guide
39. been lost often resulting in multiple active services The service that is ready to take over if the active service fails This service can read the file system but no changes can be made If the active fails the applications can be failed over to the standby using the replicated data Abbreviation for UNIX file system Abbreviation for virtual file system Continuous Computing Corporation upSuite User s Guide 217 Glossary Continuous Computing Corporation 218 upSuite User s Guide Technical Support Before contacting the Technical Support team at Continuous Computing be sure you have read this manual carefully If you continue to experience problems please contact the Technical Support team at Continu ous Computing by any of the methods listed below Please be sure to include the serial numbers for each affected module system and or part In addition we will need to know what version of Solaris you are running as well as the patch level and any other significant software packages that are installed To contact the Technical Support team at Continuous Computing do one of the following Email us at support ccpu com e Visit our support web site at http support ccpu com This site features our automatic technical support system Create a new user profile Then submit a new ticket at the Welcome to SupportWizard page This process ensures that our team delivers a timely solution to any technical prob
40. called for every service in the configuration If one of the arguments is node ID 0 the status will be down meaning that there is no active server directive Instructs a service to assume a given status either active or standby This function is called as a result of a call to ubAsync ubAsync calls this function whenever upBeat wants to initialize or change the active standby state of a service registered by this program on this node Table 1 Callback functions Continuous Computing Corporation 34 upSuite User s Guide Quick Reference of API Function Calls Quick Reference of API Function Calls Initialization and Termination ublnit Initialize a connection to the local upBeat daemon ubFini Shut down a connection to upBeat and free any resources allo cated by ubInit ubAckSvc Respond to a UB_ACTIVE or UB_STANDBY directive ubAsync Receive asynchronous status information from upBeat ubGetState Get synchronous state stored in 1 ibupbeat ubRegSvc Register to provide a service ubSetupPollfd ubNode B Set up a poll file descriptor structure so that the application can poll on libupbeat s connection to the local upBeat dae Get the node ID that corresponds to a given node name ubNodeName Get the name of a node ubServiceIP Get one of the IP addresses associated with a given service ubSvc
41. eevee ee 33 CALEBACK FUNCTIONS dd ee did dd 34 QUICK REFERENCE OF API FUNCTION CALLS 0 cccsssecesssceceesececencecesseeecasaecesaeeecsaeeesenaeeess 35 UBAGKS Va dd 36 UBAS NC dices 37 UBRINI Cacao ias 40 UBGETSTATE Dead Domi Ree dee e aie eke 41 IBN E Ades Sevens E T E a 43 UBNODE a Seda eee a iS 44 UBNODENAME ii didaa saitai sa tii i a a a aba in e aa 45 UBREGSVC Jorre na Sait A een asa aa 46 UBSERVICELP de asis aisiais o Da L on heen a Rem a Berk a a a e a 47 UBSETUPPOLLFD itakos a E a sai a ss a a aa d Tan A seth aaa eee 48 DBSVG Lu cies ii E Ia a IS aa aa Sada aa aaa aa A te el Ree ee aa 49 UBS VGIPRPAIR aiski siose dne da nan is as e a ody e sn da ano aaa 50 UBSVCNAME Mii e a sa a aaa aaa 52 UBSVCPEER O Litas aa et ce Seton Sia a i Na aaa Sa a S a a e it i E i eee 53 WBS VGPORT cc Salai sia go A a edison ba Ath oes eet 54 ERROR CODES 3 A he eet Taa a 55 EINVALS a es e id LAS EEE EE 55 LOSES ise oh a a Soca 55 ELM SG SIZE e o S BSA OE e is 55 EENOT CONN a SEE aes oa eels 55 E PROTO e EEE CE 55 EPROTONOSUPPO RT iaa 55 6 UPBEAT SAMPLE APPLICATIONS cooooccccccinconnncnnnnnnnnn conan aa aa aaa aaa aaa 57 THE wrapper SAMPLE APPLICATION csscescesseecceseessecseecsecasesaeceecsaecescseeseceeeeeeeesaeeasensenaes 57 A WTADDEL O a a i ees sai cage nie ange a a a aa i a e 58 Source Code of WLAPPeL sorted anal a a ais 58 Continuous Computing Corporation upSuite User s Guide THE status
42. for there to be a protocol error between the two upDisk servers An XOP error will appear on the console and in var adm messages to notify you of the problem Normally upDisk will reset the network link fix any problems and continue If you encounter XOP errors do the following 1 Investigate the health of the standby system because XOP errors often indicate a bad disk or other problem on the standby 2 Verify the health of the system by running udstat because if an XOP error is present upDisk will usually drop the link bring up a new link and run a repair automatically 3 Even if the above seems to solve the problem and your system appears healthy please send the XOP error to Continuous Computing s Technical Support via email at support ccpu com along with var adm messages from both the active and standby servers so the error can be investigated Troubleshooting other issues This section describes some common issues that arise when using upDisk and tells how to respond to these events File System Access Denied If your access to the file system is denied it may have been designated as standby by upDisk To find out use the udstat ipfs command at your prompt Note that the standby side 1s read only and therefore all changes must be made to the active system Continuous Computing Corporation 94 upSuite User s Guide Troubleshooting other issues File System Out of Disk Space If one of your systems runs
43. group s lead service is an upDisk dataset then it is possible to cause a standby dataset to be active thus causing the group to become active by having the cause_failover script call the udactive command with A options for the standby dataset If a lead service other than upDisk is used then a communication mechanism other than upBeat must be used between the standby and active service instances to make the standby service active via the cause_failover script Note The default cause_failover script must be modi fied to include the proper handling for each group 188 Continuous Computing Corporation upSuite User s Guide Shell Commands OPTIONS FILES SEE ALSO This option includes the force option when calling the cause_failover script Any special action taken by this option is up to the cause_failover script h hostname p port or This option changes the IP address that ubmfailover uses to communicate with ubManager IPv4 IP addresses only are sup ported The default is the local host hostname is the resolvable name of a machine not its IP address Note that no form of security is used e g authentication with this command and it can be run by root and non root users This option changes the TCP port used to com municate with ubManager The default is 2005 This option prints help for the command usage and exits cause_failover None Continuous Computing Cor
44. it is unlikely there will ever be any file system damage If there was damage upDisk attempts a repair fsck o p If after the repair there is still damage upDisk alerts the operator and shuts down upDisk mounts the local file system 104 Continuous Computing Corporation upSuite User s Guide Boot Sequence 5 upDisk mounts ipfs 6 upDisk starts its daemon for this dataset 7 If there was damage upDisk initiates a repair 8 The upDisk daemon determines the last state of the file system active standby replay repair etc If active the daemon registers with upBeat and requests active status other wise it requests standby status upBeat will assign a role asynchronously If that role matches the request or if that role is standby upDisk confirms that role with upBeat and assumes the role If upDisk requested standby and upBeat assigned active upDisk declines by confirming standby A standby upDisk server daemon will accept an assignment to failover from standby to active if it is in sync with an active upDisk server daemon In other words if the current instance of the standby daemon has ever successfully established a connection with an active upDisk server and as long as its state is normal i e is not replay repair or summary the daemon will accept an assignment to failover from standby to active Continuous Computing Corporation upSuite User s Guide 105 9 Failover Continuous Com
45. its main poll or select call a multi threaded application can dedicate a thread to ubAsync e To register a service call ubRegSvc Ifthe upBeat client application is sufficiently complex it may be beneficial to allocate a separate thread specifically for interacting with upBeat e Multi threaded application callbacks must lock data Otherwise functions may be per formed in undesired sequences thus damaging your data All programs must call ubF ini before they exit All applications must include the following header file include lt libupbeat h gt e Applications should be compiled and linked as follows gcc I opt upsuite include c o myapp o myapp c gcc L opt upsuite lib o myapp myapp o lupbeat Multithreaded applications gcc D REENTRANT I opt upsuite include c o myapp o myapp c gcc L opt upsuite lib o myapp myapp o lupbeat lthread or gcc L opt upsuite lib o myapp myapp o lupbeat lpthread libupbeat is not gcc specific All programs incorporating 1ibupbeat s function calls will compile on any Sun SPARC C compiler Continuous Computing Corporation 24 upSuite User s Guide Guidelines for Designing Applications Guidelines for Designing Applications This section contains a detailed discussion of how to use the upBeat API calls in an applica tion It covers the following topics e Initialization of upBeat Clients that Provide Services e Registerin
46. lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt HANFS gt lt SERVICE_IP IP 192 168 1 77 IF hme0 17 gt lt SERVICE gt CAUTION You must use one and only one lt SERVICE_IP gt subtag per HA NFS service Other wise an error will occur and upDisk will shut down 2 Share the dataset in Jetc ipfstab Edit etc ipfstab so that the dataset is shared You can use any of the normal sharing options e g rw marketing engineering root engineering Note If your clients have redundant network links you must specify both of them under share options separated by a colon Refer to the following man pages for more detailed information about sharing share 1m and share_nfs 1m Note Do not use etc dfs dfstab to share the dataset Sharing must happen after upDisk is running unsharing must happen before upDisk shuts down You must use etc ipfstab as described below Perform the following steps on both systems active and standby Continuous Computing Corporation 112 upSuite User s Guide Building an HA NFS Server Step Description Command 1 Go to the ipfstab file directory cd etc upsuite 2 View the end of the file tail ipfstab Typical ipfstab entry data ipfs ipfs localfs localfs localfs locals share set mount mount mount mount name point opts point type device opts opts ipfs ipfs
47. must match the service name in the upSuite HA configuration file upsuite conf for example if the dataset name is my data then the service name in upsuite conf must be ipfs mydata The dataset name must be identical on all servers that replicate this dataset All other fields may vary from server to server This field is the mount point for the ip fs file system This is the directory through which users and applications access files If this field is no user specified mount options are passed to the ipfs mount otherwise this field is passed as o ipfs mount options For information about the available options see ipfs Mount Options on page 121 This field is the mount point for the underlying file system This mount point should be obscured by the ipfs mount point typically by making it the same as the ipf s mount point The underlying local file system must not be already mounted when upDisk starts Delete or comment out entries in etc vfstab for these local file systems We strongly recommend that the local file system mount point must be obscured by the ipfs mount point typically by mak ing both mount points the same so that ipfs hides or obscures the local file system For debugging or demonstration purposes it may be desirable to have the local file system accessible but in a production environment it would be disas trous if someone modified the local file system directly while upDisk was running If
48. no such sharing is underway service_status One of the following values UB_STATUS_UP or UB_STATUS_DOWN If the callback function was called by ubAsync UB_STATUS_UP indicates that an upBeat service has become active on a node and UB_STATUS_DOWN indicates that an upBeat service has become standby on a node If the callback function was called by ubGet State UB_STATUS_UP indicates that an upBeat service is active on a node and UB_STATUS_DOWN indicates either that an upBeat service is standby on a node or does not exist on that node If service_status is UB_STATUS_DOWN and node_idis 0 the 38 Continuous Computing Corporation upSuite User s Guide ubAsync svc_status async node link service service is not active on any node Specifies what role a service should take as directed by upBeat One of the following values UB_STANDBY or UB_ACTIVE UB_STANDBY indi cates that the service must become standby UB_ACTIVE indicates that the service may become active the service can also refuse and retain its standby status by using ubAckSvc with its svc_status argument set to UB_STANDBY as described in ubAckSvc on page 36 B_TRUE if the callback function was called by ubGetState the state is synchronous B_FALSE if the callback function was called by ubAsync the state is asynchronous The use of this argument a
49. no values will be displayed Deferred operations accumulate if the standby is offline or if the server is busy completing a repair Deferred operations are moved to the replay list when the standby comes back online or as a result of repair activity During a repair it is possible to see both replay and deferred operations because the server may briefly defer operations for parts of the file system stopped stopped indicates that upDisk has stopped operations operator splitbrain operator splitbrain indicates that operator inter vention is required and that a splitbrain condition is the rea son nodaemon nodaemon indicates that upDisk is not running for this dataset Continuous Computing Corporation 134 upSuite User s Guide udstat OPTIONS cont d dataset not mounted dataset not mounted indicates that the dataset is listed in ipf stab but is not currently available offline The local upDisk daemon is running this is not the active server for this dataset and the local daemon is not in contact with the peer daemon There are several reasons of fline might be displayed upBeat has detected a problem with a disk that this ser vice depends on as configured in upsuite conf and has told this service to go offline e Ifa repair fails upDisk repeatedly waits and retries the repair During the waiting periods between failed repair attempts the standby is offline e Inasplit brain situation both ser
50. possible states of the link State normal repair replay or summary The state field indicates the state of the system s up or Message Meaning tions are being sent in real time normal The system is running normally the link is up and opera for later replay summary The link is down and the system is summarizing operations ously summarized operations replay The link has been restored and the system is replaying previ 1t repair Damage was detected and the system is currently repairing Table 4 System states Comments You may encounter comments to the right of your role link and state messages separated by a hyphen which inform you about changes to the role link or state the specific messages are listed later in this section Such messages are purely informational in nature and si intended to keep you apprised of system conditions If however an error is logged mply upDisk in most cases will solve the problem if the error requires operator intervention messages will be sent to the console and logs In addition if you run the udstat command its output will indicate that operator intervention is required Dropping the link Listed below are the circumstances under which a link is dropped e Operator shutdown upDisk or upBeat has detected that a link has failed and instructed ipfs to stop using it or as the operator you have intentionally dropped the link L
51. publication may be reproduced stored in a retrieval system or transmitted in any form or by any means photocopying recording or other wise without prior written consent of Continuous Computing Corporation No patent liability is assumed with respect to the use of the information contained herein While every precaution has been taken in the preparation of this publication Con tinuous Computing Corporation assumes no responsibility for errors or omissions This publication and features described herein are subject to change without notice Sun the Sun logo SPARCengine Solaris and OpenBoot are trademarks or registered trademarks of Sun Microsystems Inc in the United States and other countries All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc in the United States and other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems Inc CompactPCI is a registered trademark of PICMG The information contained in this document is not designed or intended for use in human life support systems on line con trol of aircraft aircraft navigation or aircraft communications or in the design construction operation or maintenance of any nuclear facility Continuous Computing Corporation disclaims any express or implied warranty of fitness for such uses Send comments about this document to documentation ccpu com Table of Conte
52. return it to service File Locking For NFS clients only ipfs does not replicate locks In the case of a failover you may encounter errors as the files unlock Managed Failover If your configuration changes you must restart upBeat and everything that interacts with upBeat The simplest way to deal with configuration changes is to perform a managed failover This example assumes left is the standby system and right is the active system at the beginning of the procedure Continuous Computing Corporation 102 upSuite User s Guide Managed Failover The recommended way to instigate a manual failover is to run this command on the active udactive S or udactive Sf It is not recommended to instigate a manual failover by running udactive Af on the standby Doing so will possibly lead to operational problems Description 1 On the standby system stop upBeat and everything left currently standby that interacts with it left etc init d updisk stop left etc init d upbeat stop 2 Edit the configuration file and or ipfstab to left currently offline reflect your changes left vi etc upsuite conf eft vi etc ipfstab 3 Restart upBeat and everything that interacts with it left currently standby left etc init d upbeat start left etc init d updisk start 4 On the active system stop upBeat and everything right currently active that int
53. returns 1 otherwise returns 0 ERRORS None SEE ALSO ubSvc on page 49 ubSvcIPPair on page 50 Continuous Computing Corporation upSuite User s Guide 47 5 upBeat API Reference ubSetupPollfd NAME ubSetupPollfd SYNOPSIS Set up a poll file descriptor structure so that the application can poll on Libupbeat s con nection to the local upBeat daemon void ubSetupPollfd upbeat_t upbeatp struct pollfd pollfdp upbeatp A handle as returned by ubInit pollfdp Pointer to a po 11 fd structure DESCRIPTION An application must call ubAsync periodically One way to do that is to poll on libupbeat s file descriptor and call ubAsync whenever there are events ubSetupPollfd fills ina poll fa structure with the file descriptor and the required pollfdp gt events An application that uses select can extract the file descriptor from poll fdp An application should call ubSetupPollfd before each call to poll in case ubAsync has reestablished a connection to upBeat RETURN VALUES None ERRORS None SEE ALSO ubAsync on page 37 Continuous Computing Corporation 48 upSuite User s Guide ubSvc ubSvc NAME ubSvc SYNOPSIS Get the service ID of a named service ub svc id t ubSvc upbeat t upbeatp char servicename upbeatp A handle as returned by ubInit servicename The NAME attribute of a lt SERVICE gt tag in the upSuite con
54. s file directory mount_point n argv 0 exit 1 Every IPFS file system has an entry at the top level which is designated by IPFS_HANDLE and which can be opened to determine status 57 n snprintf handle MAXPATHLEN s s argv 1 IPFS HANDLE if n gt MAXPATHLEN n lt 0 fprintf stderr path too long or snprintf failedin exit 1 Try the handle first in case we were handed a mount point then try the path directly in case we were handed a file or directory path handle if fd open path O RDONLY 1 if errno ENOENT errno ENOTDIR path argv 1 if fd open path O RDONLY 1 fprintf stderr open s failed errno d s n path errno strerror errno exit 1 Continuous Computing Corporation 142 upSuite User s Guide Sample API Code else fprintf stderr open s failed errno d sIn path errno strerror errno exit 1 Try to get IPFS status ENOTTY tells us this is not an IPFS file system any other error is reported as an error If wait is 0 return status immediately E wait 0 if status ioctl fd IPFS_STATUS wait 1 if errno ENOTTY printf s not an IPFS file system n path else fprintf stderr ioctl IPFS_STATUS failed errno d s n errno strerror errno exit 1 else printf s IPFS file sy
55. s links services or nodes can be monitored via the opt upsuite bin ups program You can learn whether links services or nodes are UP or DOWN the only possible states listed with the output message See upBeat Com mands on page 17 for further explanation and usage of this command Running upBeat in debug mode usr sbin upbeat d allows you to monitor upBeat operations See upbeat admin command on page 17 As an alternative you may also use the log file var log upsuite to monitor histori cal operations of upBeat Maintenance upBeat s status can be monitored via the syslog By default the upBeat daemon sends all output to the syslog facilty unless upBeat is running in debug mode in which case it sends all output to the terminal Continuous Computing Corporation upSuite User s Guide 15 2 upBeat Administration Troubleshooting This section describes some common issues that arise when using upBeat and tells how to respond to these events L nks Down If ups reports a link is down try pinging that link manually If there is no answer to the ping check your cabling and switches If you find the link fluctuating between UP and DOWN check your heartbeat TIMEOUT_MSEC setting in upsuite conf Ensure that this setting is not too short for your network Busy Disks If you encounter the message below your disk is not responding Oct 3 09 52 41 left upbeat 418 disk or partiti
56. standby ubManager registers the group with upBeat to be standby Since ubManager passively tracks the lead service any service chosen as a group lead must already be independently integrated with upBeat If a user application uses disk storage an upDisk dataset would serve as an ideal lead service If a group has no lead service then upon initialization ubManager will register the group with upBeat to be active dependent upon the status of any monitors If the group s peer is already active then upBeat will issue a standby directive for the group The ubmfailover utility can be used to failover a group whether it has a lead service or not User Applications These are application programs that are non upBeat aware When a group becomes active on a node ubManager starts the user applications for that group on that node and when a group becomes standby on a node ubManager gracefully terminates the user applications for that group on that node Failover Scripting After ubManager receives a directive from upBeat it runs either a go active script for an active directive or a go standby script for a standby directive before acknowledg ing the directive These scripts provide failover scripting and can launch or terminate pro cesses configure network interfaces or manage any other configuration for the group Monitors optional A group can be configured to use processes that monitor resources required by the group
57. stop which prevents you from running etc init d updisk start You must successfully run stop before being able torun start Table 6 lists the reasons ipfs will not unmount and instructions for solving the problem After you have solved the problem run etc init d updisk start to remount the file systems and restart upDisk Continuous Computing Corporation upSuite User s Guide 95 8 upDisk Administration Reason Solution A user is either in ipfs 1 Run etc fuser c mountpoint to find out who or using a file in it is using ipfs See the manual page for fuser for more information about this command 2 Ask the user to cd out of ipfs or kill the processes that fuser lists ipfs is shared 1 Use the share command to find out which file systems are shared via NFS 2 Unshare exported file systems ipfs is busy Wait for operations to complete before trying to run etc init d updisk stop again Table 6 ipfs unmount solutions Conflicting File Modification Times mtime File modification times under normal operations may appear up to one second off between the active and standby systems The modification times may be off only by tens of millisec onds but some system utilities will round this number up to the nearest second This is due to operations of the virtual file system Because of this modification times may not appear iden tical on each system but they will not dif
58. system gt lt DISK PARTITION one gt lt SCSIBEAT gt lt HANFS gt subtag of the lt SERVICE gt tag Optional The lt HANFS gt subtag is used only when configuring a high availability NFS sys tem It has the following optional attributes e PORT specifies the port number of the NFS server This is the port used by NFS clients to connect NSERVER is the number of kernel threads created to handle NFS requests This deter mines how many NFS requests can be handled at the same time e PROTOS specifies which networking protocols are configured for the NFS server This is specified in a space separated list The valid protocols are UDP and TCP Default is UDP Continuous Computing Corporation upSuite User s Guide 157 A 14 upSuite Configuration File upsuite conf Example lt HANFS PORT 2049 NSERVER 17 PROTOS UDP TCP gt Be sure you also add the required share settings to the ipfstab file For more information about this and other HA NES topics refer to High Availability NFS HA NES on page 107 lt SERVICE IP gt subtag of the SERVICE gt tag Optional This tag is used to set up the necessary information for IP failover Therefore this attribute can be omitted only for services that do not require IP failover Defines the IP address host name and interface name of the service A lt SERVICE gt tag can have more than one lt SERVICE_IP gt subtag unless you are configuri
59. that was forced to become active is restarted it will forget its forced active status when the two sides come back into contact it will be re flagged as split brain and you will have to start fixing the split brain condition again Technique 2 Force status on both sides Use this technique when the two systems are not in contact with each other 1 Determine which system has the most current data by determining which went down first or last Do this by inspecting the relevant files and or system logs and by using the fol lowing command udstat hh 2 Force the desired system to become active udactive Af dataset 3 Force the other system to become standby udactive Sf dataset 4 Let the systems come back into contact For detailed information about the udactive and udstat commands see upDisk Com mand Reference on page 127 98 Continuous Computing Corporation upSuite User s Guide Troubleshooting other issues Techniques to Avoid When Recovering from Split Brain There are two things to avoid when trying to fix a split brain condition By doing either of the following you will create a new split brain condition Running udactive A f on both systems while the systems are unable to commu nicate with each other or Running udactive A f on server A while server B is down offline or otherwise unable to communicate with server A and then rebooting or restarting upDisk on server A In this cas
60. the con figuration file you specify here rather than its default configuration file etc upsuite ubmgr conf client_port This option changes the TCP port that clients ubmstat ubmfailover use to commu nicate with ubManager The default is 2005 script_dir This option changes the directory that ubMan ager will look for the user definable scripts goactive gostandby cause_failover The default is usr lib ubmgr This option causes program messages that would normally be sent to the syslog to instead be sent to the console or to the terminal that started ubmgr This option prints the ubManager version num ber and exits 192 Continuous Computing Corporation upSuite User s Guide Shell Commands OPTIONS cont d FILES SEE ALSO or This option prints help for the command usage and exits ubmgr conf ubmgr init d Continuous Computing Corporation upSuite User s Guide 193 16 ubManager Command Reference NAME USAGE DESCRIPTION ubmstat usr sbin ubmstat a h hostname i interval m N p port v This command outputs various status information about ubManager groups monitors and applications The basic status output is state information For a group the states are UNREG unregistered with upBeat WAIT_MON a group is waiting for its monitors to start WAIT_ACT a group has registered for active and is waiting for an active directive WAIT
61. the reliance on unresponsive name services Note that upBeat will reject hosts with multiple IP addresses Therefore if replacing IP with the HOST attribute be sure that the host name specified is not associated with more than one IP address While the value of IP is passed to inet addr the value of HOST is passed to gethostbyname lt PARTITION gt subtag of the lt VNODE gt tag Specifies the name device and timeouts of a partition Use the lt PARTITION gt subtag only if you need to monitor SCSI disks in a Solaris environ ment this is due to Solaris SCSI disk driver which might not report an unresponsive disk in a timely manner The wait may be too long for your specific needs If you need to monitor one or more particular SCSI disks more closely in addition to adding the lt PARTITION gt subtag to the lt NODE gt tag you must add a lt SCSIBEAT gt subtag under the lt SERVICE gt tag as described in lt SERVICE gt tag on page 154 SCSIbeat initiates a failover if a SCSI disk becomes unresponsive The Solaris interface requires that you specify a partition even though you may only want to monitor a disk Therefore to monitor the disk cOt 0d0 you must watch one of its partitions cOt0d0s0 through cOt0d0s7 If your application depends on two partitions on the same disk for example cOt 0d0s0 as the system partition and cOt0d0s5 as the data par tition you only need to monitor one of the partitions NAME att
62. ubSvcPort returns the port number otherwise returns 0 ERRORS None SEE ALSO ubSvcIPPair on page 50 Continuous Computing Corporation 54 upSuite User s Guide Error codes Error codes Many of the upBeat API functions indicate an error by returning 1 or NULL When this occurs the value of errno is also set This value can be set in two ways e The upBeat API call e An underlying function or system call that was made by the upBeat API It is not possible to list all the values that might be set by underlying calls This section describes the error values that can be set by the upBeat API EINVAL The handle an upbeat t passed to the upBeat API was NULL ElO The application encountered a problem when trying to send a message to the upBeat daemon EMSGSIZE The upBeat daemon is sending a packet that is too big for the application to handle It is likely that EPROTONOSUPPORT would be seen first allowing the application to avoid EMSGSIZE ENOTCONN A connection to the local upBeat daemon is maintained by 1 ibupbeat on behalf of the application When ENOTCONN is returned it indicates that 1ibupbeat has lost this con nection to the upBeat daemon and cannot re establish it When returned by ubInit this code indicates that a connection was never made EPROTO A packet received from the upBeat daemon contained a protocol error EPROTONOSUPPORT There is a version conflict The application is att
63. upDisk upSuite s data replication module for the ability to write to an upDisk dataset If dependencies such as these exist between services you would need to implement special policy software to manage those relationships ubManager is that software ubManager allows you to create groups of services controlled by a lead service and or mon itors see below which you designate All services associated with the lead service follow its behavior in the event of a change to its status For example if an upDisk dataset is the lead ser vice for a database application and the dataset fails due to a disk failure the database applica tion also fails over and so do all other application members of the group specifying that particular upDisk dataset as the lead service Continuous Computing Corporation 180 upSuite User s Guide How Does ubManager Work A ubManager service group has the following characteristics lt SERVICE gt Entry in upSuite Configuration File Each ubManager service group is itself an upBeat service which ubManager controls so each group must have its own lt SERVICE gt tag entry in the upSuite configuration file etc upsuite upsuite conf Lead Service optional ubManager controls the group to follow any designated lead service with exceptions noted below This means that when the lead service becomes active ubManager registers the group with upBeat to be active and when the lead service goes to
64. void uservice void uint32 t uint32 t ub status t boolean t static void usage char int main int argc char argv ub_node_id_t node_id upbeat_t ubp NULL int Cc Continuous Computing Corporation upSuite User s Guide 65 6 upBeat Sample Applications int quick FALSE extern int optind These are the callbacks we will pass to ubGetState and ubAsync UpBeat passes ubp as an argument to the callbacks We have node link and service callbacks for status we do not have a directive callback since we are not offering a service and so will not be active or standby A ub ops t ub ops ubp not set yet unode ulink uservice NULL no directive callback y ub_ops_t opsp amp ub_ops Check command line arguments q tells us to run ubGetState once and not run ubAsync at all e h or print the usage line Ef while c getopt argc argv hq EOF switch c case q Check status then exit do not loop quick TRUE break default case h case usage argv 0 break if optind argc usage argv 0 Initialize libupbeat and the connection to the upbeat daemon Ka Continuous Computing Corporation 66 upSuite User s Guide The status Sample Application if ubp ubInit NULL fprintf stderr ubInit failed errno d s n errno strerror errno exit 1 ub_ops arg
65. you are mounting a directory for testing purposes this field should be If you are mounting an entire file system recommended this should be the file system type 118 Continuous Computing Corporation upSuite User s Guide ipfstab settings localfs device Note upDisk and ipfs can use any underlying file system but the startup script etc init d updisk only knows how to repair damage detected by sck on UNIX file systems If you want to use an underlying file system other than UFS you will have to modify etc init d updisk as appropriate Standard Solaris file systems and standard Disk Suite file systems are both UFS If localfs type is this field should be Otherwise this field is the device to mount on the local file system mount point Devices are presumed to follow the dsk rdsk convention For example if you specify dev dsk c0t0d0s0 there should be a corresponding dev rdsk c0t0d0s0 for fsck to use The device can be a full path or a short cut If the device begins with Indicates the device is a full path c upDisk will look in dev dsk d upDisk will look in dev md dsk If the device begins with any other character or if it was not found in dev dsk or dev md dsk upDisk will look in dev vx dsk For example Continuous Computing Corporation upSuite User s Guide 119 11 The ipfstab File If you enter upDisk will look in dev dsk cO0t0d0s0
66. you want to monitor host2 is a second IP address or host specified with the host name you want to optionally monitor Example ubpinger serverl netl t 400 192 168 1 200 192 168 1 201 Continuous Computing Corporation upSuite User s Guide 207 17 ubManager Monitors Continuous Computing Corporation 208 upSuite User s Guide Comments 18 ubManager Semicolon Delimited Configuration File ubManager supports two different formats for its configuration file an XML format and the previously supported semicolon delimited file format The XML format is preferred This chapter describes the semicolon delimited file format which is still supported for the purposes of backwards compatibility only ubManager includes a sample semicolon delimited configuration file etc upsuite examples ubmgr sc A semicolon delimited configuration file consists of the following e A version string required e Zero or more group definitions e Zero or more application definitions e Zero or more monitor definitions Each of these parameters is detailed below Comments Lines beginning with an octothorpe are considered comments and are ignored by the pro gram The exception to this rule is that the first characters of the configuration file must be as follows 3 Version The first line of ubmgr conf must contain a version string so ubManager can tell the file format The first characters of the file must be an octothorpe
67. 00 emphasized User s Manual Ctrl Keystroke press Send a break using Ctrl A Caution Failure to heed the instruc tions that follow the Cau tion symbol may result in damage to the equipment Continuous Computing Corporation upSuite User s Guide 215 Typographic Conventions Continuous Computing Corporation 216 upSuite User s Guide Glossary Active service Client Failover HA NFS ipfs Local Mean Time to Repair MTTR NFS Node Peer SCSIbeat Split Brain Standby service UFS VFS The service that can read and write to the file system An application that uses upBeat in order to monitor the net work The migration of applications or services from one machine to another in the event of failure Abbreviation for a high availability implementation of NFS see the NFS definition below A pair of standard NFS serv ers plus upSuite HA software are used to build a high avail ability NFS server The kernel module of upDisk The current system on which upBeat resides The average time it takes to complete repairs on equipment or services A computer industry standard network file system protocol developed for distributing files within a heterogeneous net work Any processor running upSuite HA The partner of the current local node SCSIbeat is a component of upBeat that monitors disk failures All network connectivity between two nodes has
68. 000 FREQ MSEC 600 gt lt PARTITIO AME two DEVICE dev rdsk c0t1d0s4 IMEOUT_MSEC 3000 FREQ MSEC 700 gt lt NODE gt Continuous Computing Corporation upSuite User s Guide 149 14 upSuite Configuration File upsuite conf lt NODE NAME right NODE_ID 2 DESCRIPTION SPARC CP1500 Solaris 2 7 gt lt INTERFACE NAME hme0 NETWORK Network1 IP 192 168 1 3 gt lt INTERFACE NAME hmel NETWORK Network2 IP 192 168 2 3 gt lt PARTITION NAME system DEVICE dev rdsk c0t0d0s0 IMEOUT_MSEC 2000 FREQ_MSEC 600 gt lt PARTITION NAME one DEVICE dev rdsk c0t1d0s4 IMEOUT_MSEC 2000 FREQ_MSEC 600 gt lt PARTITION NAME two DEVICE dev rdsk c0t1d0s5 IMEOUT_MSEC 3000 FREQ_MSEC 700 gt lt NODE gt A given system locates its own node ID number by comparing its IP addresses to those listed for each node in upsuite conf A system must have all the IP addresses that are listed for a node and it can have more IP addresses as well upBeat issues an error in three situations it cannot find a matching IP address the current system has some but not all IP addresses listed for a node or some of the current system s addresses are listed under one node and some are listed under another node NAME attribute Required This names the node
69. 1 O 132 UPDISK ADMIN COMMAND iaa canada sia a a iuo sko EE ia a a a a dia 137 UPDISK STARTUP SCRIPT coria saaa iseina kasikais apa aa aaa 139 T3 UPDISKAP Wie aaa 141 INPINOWER VIEW eners orreta dis o ea r a pce ss a 141 FUNCTION CALL GUIDELINES areae eian E aidas asa kainos a sian asa aaa sea ainas aa 141 SAMP E API CODE cite rad 141 Continuous Computing Corporation IV upSuite User s Guide 14 UPSUITE CONFIGURATION FILE UPSUITE CONF coooccccnnccnnnnconnnnnnnconnnnnnnns 145 THE CONFIGURATION FILE ccsssscecsssscessscessesnsesssceecessecsenssecssnececnsecesssessnsecessnsesenensessneees 145 Managing the Configuration File on Multiple Machines 146 Using XML in the Configuration File aaa aaa aaa aaa aaa aaa 146 Avoiding Name Conflicts in the Configuration File o n 146 EDITING UPSUITE CONB vireeseen kaikas e aki in Seek sad sneka ienas aaa 147 lt UPSUTTECONEIG gt TAG sis 148 lt UPBEAT gt TAG a ala dia niente 148 lt NETWORK gt TAG ei fate eee ee dedos 149 NAME attribute a cosets a ieee et Be Resp A A i 149 DESCRIPTION Ein Omn 2 149 lt NODE gt TAG E e e e a A a i ao e ed 149 NAME tiri bit Is kissa tas eva a ai i asa as ls sa ai as D ai si 150 NODETD GU Pi DU ncnia a EE EIT Is ui o RES 150 DESCRIPTION Ein Omn
70. 1 on failure Success does not mean that service registration is complete or that the svc_status preference has been granted ERRORS See Error codes on page 55 SEE ALSO ubAckSvc on page 36 Continuous Computing Corporation 46 upSuite User s Guide ubServicelP ubServicelP NAME ubServiceIP SYNOPSIS Get one of the IP addresses associated with a given service int ubServicelP upbeat t upbeatp ub svc id t service in addr t in addr ub service ip t service ip upbeatp A handle as returned by ubInit service Service ID of the service for which you want to get an IP address Use an ID from a service callback or one returned by ubSvc in addr IP address returned by the most recent previous call to ubServicelP or 0 service_ip Out parameter in which a set of data including the IP address found by ubServicelP is returned DESCRIPTION In the upSuite configuration file the lt SERVICE_IP gt subtags of the lt SERVICE gt tag specify which IP addresses upBeat manages for the service These are the IP addresses involved in IP failover and are the addresses at which clients expect to contact the service You can use ubServicel P iteratively to get these IP addresses from the configuration file RETURN VALUES If ubServiceIP executes successfully the handle passed in upbeatp is not NULL lt SERVICE_IP gt subtags are found and you are not at the end of the list
71. 1 or 4 1 ratio of TIMEOUT MSEC to RESEND MSEC If you are using a MAN or WAN you will need to experiment with your ratios for your particular needs lt NODE_REF gt subtag of lt HEARTBEAT gt The lt NODE_REF gt subtag of the lt HEARTBEAT gt tag refers to the nodes as you have defined them in the NODE_ID attribute of the lt NODE gt tag Each lt HEARTBEAT gt tag must contain two lt NODE_REF gt subtags NODE_ID attribute Required This is the node as it is defined in the NODE ID attribute of the lt NODE gt tag lt LINK gt subtag of lt HEARTBEAT gt Required Defines the links over which heartbeats are to be sent The lt HEARTBEAT gt tag must contain at least one lt LINK gt subtag although two or more are recommended A high availability system requires at least two networks NETWORK attribute Required Name of the network This allows the heartbeats to associate IP addresses from two different nodes This is a reference to the NETWORK attribute of the lt INTERFACE gt subtag of the lt NODE gt tag ROUTE attribute Optional Specifying ROUTE enables sockets to route This enables you to establish a heart beat between nodes that are on different subnets or that are separated by routers Packets will be routed according to your node s routing tables You must ensure there are separate network paths for each link and that packets for one link cannot be routed over a path for another link refer to lt SERVICE gt t
72. AD ubapp22 DEAD ubapp23 DEAD p port This option changes the TCP port used to com municate with ubManager The default is 2005 v This option short for verbose provides addi tional informational output often helpful for troubleshooting Continuous Computing Corporation 196 upSuite User s Guide Shell Commands OPTIONS cont For group information For all output except otherwise indicated 1 active 0 standby Gs Gp Indicates the group s state on the self local node Gs and the peer node Gp Ts Tp Indicates the group s lead or tracking service s state on the self node Ts and the peer node Tp Ms Mp Overall aggregate state of all the group s monitors on self Ms and peer Mp nodes AppOk 1 indicates that all of the applications in the group are initializing and or run ning on the self node O indicates that one or more applica tions in the group are not running and or are in the process of terminating on the self node AppDead 1 indicates that all applications in the group are dead on the self node O indicates one or more applications in the group are running on the self node Continuous Computing Corporation upSuite User s Guide 197 16 ubManager Command Reference Example ubmstat v Group State Gs Gp Ts Tp Ms Mp AppOk AppDead groupl ACT 10 1 1 0 1 1 1 1 1 1 0 group2 STBY 0 1 0 1 1 1 0 1 Continuous Computing C
73. Beat estab lishes heartbeats listens to other upBeat servers and listens to local applications this information is then shared with other upBeat servers However upBeat does not tell local applications anything about the network and upBeat does not assign any service roles during this time After this period upBeat notifies local applications about which services are up or down on which nodes servers and starts assigning roles Finally upBeat processes its services according to the following logic 1 If there is already an active service the local service is assigned standby even if it is requesting active 2 Ifthe service is requesting active and there is not already an active service it will be assigned active 3 If a service requests to be active on two nodes upBeat instructs one to be active and the other to be standby 4 If no nodes are requesting active then one of the nodes requesting standby may be assigned active Note If itis inappropriate for a node to become active it is the responsibility of the service not upBeat to decline ubManager ubManager pronounced You Be Manager is an additional package available for use with upBeat but is not part of the upBeat installation ubManager is designed to enhance the fail ure detection abilities of upBeat To that end ubManager is ideal for situations in which moni toring upBeat clients is required With ubManager you can monitor a single servic
74. C 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 3 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK NetworkA gt lt LINK NETWORK NetworkB gt lt HEARTBEAT gt Continuous Computing Corporation 172 upSuite User s Guide Sample Configurations lt HEARTBEAT NAME 4 2 TYPE POINT_TO_POINT TIMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 4 gt lt lt lt NODE_REF NODE_ID 2 gt LINK NETWORK NetworkA gt LINK NETWORK NetworkB gt lt HEARTBEAT gt lt SERVICE NAME updisk ipfs SERVICE_ID 17 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK NetworkA gt lt LINK NETWORK NetworkB gt lt SERVICE IP IP 172 17 8 177 IF hme0 17 gt lt SERVICE gt lt UpSuiteConfig gt The etc ipfstab File Below is the etc ipfstab file edited to match the system illustrated in Figure 9 data ipfs ipfs localfs localfs localfs localfs share set mount mount mount mount name point opts point type device opts opts ipfs ipfs 7 ipfs ufs c0t0d0s7 rw noatime logging rw Continuous Computing Corporation upSuite User s Guide 173 14 upSuite Configuration File upsuite conf The etc hos
75. Corporation 76 upSuite User s Guide The mt multi svc Sample Application Description 5 Run mt_multi_svc The com mt_multi_svc s method h mand s options are explained below Command usage mt_multi_svc s method h Command options Including the h flag causes the program to print its usage information and then exit Including the s flag denotes split brain handling where methodis one of the fol lowing case insensitive options e nodel default behavior if no method is specified Using this method causes the program to keep the services of the node with the lower node ID active if a split brain is detected The services of the other node will become standby The node with the lower ID does not literally have to be named nodel node2 Using this method causes the program to keep the services of the node with the higher node ID active if a split brain is detected The services of the other node will become standby The node with the higher ID does not literally have to be named node2 e firstactive Using this method causes the program to keep the services of the node whose services were active first upon program instantiation to become or remain active if a split brain is detected The services of the other node will become standby e lastactive Using this method causes the program to keep the services of the node whose services were standby first upon program instantiation to become or remain
76. DE_REF NODE_ID 2 gt lt LINK NETWORK Network1 ROUTE ROUTE gt lt LINK NETWORK Network2 ROUTE ROUTE gt lt SERVICE_IP IP 192 168 1 1 IF hme0 17 gt lt SERVICE gt lt SERVICE NAME generalservice SERVICE_ID 26 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1223 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE gt lt SERVICE NAME upstart SERVICE_ID 23 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1957 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE gt lt UpSuiteConfig gt Continuous Computing Corporation 160 upSuite User s Guide Sample Configurations Sample Configurations This section describes three example configurations A B and C including the following information for each e Diagram e upsuite conf file e etc ipfstab file e etc hosts file e etc vfstab file e Partition table These configurations are provided as samples to help you configure your systems correctly Each of these samples assumes you are running upDisk on the systems as we
77. E POINT_TO_POINT IMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt B lt HEART Continuous Computing Corporation upSuite User s Guide 167 14 upSuite Configuration File upsuite conf lt SERVICE NAME updisk ipfs SERVICE_ID 17 YPE BASIC STARTUPDELAY_SEC 5 PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE IP IP 172 17 8 177 IF hme0 17 gt lt SERVICE gt E T lt UpSuiteConfig gt The etc ipfstab File Below is the etc ipfstab file edited to match the system illustrated in Figure 8 data ipfs ipfs localfs localfs localfs localfs share set mount mount mount mount name point opts point type device opts opts ipfs ipfs ipfs ufs do rw noatime logging rw Continuous Computing Corporation 168 upSuite User s Guide Sample Configurations The etc hosts File Below is the etc hosts file edited to match the system illustrated in Figure 8 Note that listed to the right of the IP address is the actual name of the server 172415 ETZE LP LTZ hs 172 18 1724 18 0 o
78. EMICOLON DELIMITED CONFIGURATION FILE aaee 209 COMMENTS a i 209 VERSION EEE E S AE E E E a e ds 209 GROUP DERBINITION e el a as 210 MONITOR DEFINITION n 210 APPLICATION DEFINITION cd 212 SAMPLE SEMICOLON DELIMITED CONFIGURATION FILE cccccccccnnnnonanananananononnonononananonononos 212 TYPOGRAPHIC CONVENTIONS c oocmcccccccnconononcnnanonnnnnnanonannanananannrnnnannanananananose 215 GLOSSARY a 217 TECHNIECAL SUPPORT ii ua G ais a at ee 219 INDEX aa a a a i aa a a i da a o is 221 Continuous Computing Corporation V upSuite User s Guide Welcome to upSuite HA upSuite HAM is a high availability software product for Solaris 7 8 9 and 10 that instantly provides high availability for any application or server upSuite HA detects software hard ware or network failure and provides sub second failover to a standby system Real time data replication at the filesystem level enables upSuite to provide rapid failover and recovery upSuite can provide these services transparently to the application and operating system No APIs are necessary Sub second failure detection and failover Runs on TCP IP LAN Tunable heartbeats control failover Efficient application transparent failover Efficient repair and replay recovery Minimal use of CPU and bandwidth Installs in minutes Simple to implement and maintain active standby architecture About This Manual This manual describes the following so
79. ES service IP address or hostname rather than the server s normal hostname in the client side mount command Sharing Subdirectories vs File Systems upDisk can replicate an entire file system or a subdirectory Usually it is better to replicate an entire file system but for testing or demonstration purposes a subdirectory may be appropri ate Sharing an ancestor of the ipfs dataset is not recommended In other words if you are repli cating a subdirectory and sharing that upDisk dataset and if you are also sharing any of the upper directories via et c dfs dfstab your results may be unpredictable This is due to Solaris semantics for sharing a subdirectory of an already shared directory This is illus trated in Figure 6 export shared via dfs dfstab ipfs shared via ipfstab Figure6 Sharing an ancestor of the ipfs dataset not recommended Continuous Computing Corporation upSuite User s Guide 115 10 High Availability NFS HA NFS Continuous Computing Corporation 116 upSuite User s Guide 11 The ipfstab File The ipfstab file is a text file that contains a series of one line entries each of which con sists of a series of settings for a single file system The ipfstab file is used to inform upDisk which ipfs file systems to mount This file is also used by the following commands udstat udrepair and udactive For explanation of these commands see upDisk Command Reference
80. ESCRIPTION 172 18 33 0 24 gt E lt NODE NAME left NODE_ID 1 DESCRIPTION SPARC CP1500 Solaris 2 8 gt lt INTERFACE NAME hme0 NETWORK Network1 IP 172 17 33 120 gt lt INTERFACE NAME hmel NETWORK Network2 IP 172 18 33 120 gt lt NODE gt lt NODE NAME right NODE ID 2 DESCRIPTION SPARC CP1500 Solaris 2 8 gt lt INTERFACE NAME hme0 NETWORK Network1 IP 172 17 33 121 gt lt INTERFACE NAME hmel NETWORK Network2 IP 172 18 33 121 gt lt NODE gt Continuous Computing Corporation 70 upSuite User s Guide The mt_multi_svc Sample Application lt HE NAME left right YPE POINT_TO_POINT IMEOUT_MSEC 2000 RESEND_MSEC 650 gt lt NODE_REF NODE ID 1 gt lt NODE REF NODE ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt HEARTBEAT gt lt SERVICE NAME MtAppCt1 SERVICE_ID 50 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1800 gt lt NODE_REF NODE ID 1 gt lt NODE REF NODE ID 2 gt lt LINK NETWORK Network1 gt lt LINK NETWORK Network2 gt lt SERVICE_IP IP 192 168 1 1 lt SERVICE_IP IP
81. N Continuous Create Deploy Converge upSuite User s Guide Version 2 6 CC02603 00 9380 Carroll Park Drive San Diego CA 92121 2256 858 882 8800 www ccpu com O 2001 2003 Continuous Computing Corporation All rights reserved The information contained in this document is provided as is without any express representations of warranties In addi tion Continuous Computing Corporation disclaims all implied representations and warranties including any warranty of merchantability fitness for a particular purpose or non infringement of third party intellectual property rights This document contains proprietary information of Continuous Computing Corporation or under license from third parties No part of this document may be reproduced in any form or by any means or transferred to any third party without the prior written consent of Continuous Computing Corporation Continuous Computing the Continuous Computing logo upSuite upDisk upBeat upState Continuous Control Node CCN Continuous System Controller CCPUnet CCNtalk Field Replaceable Microprocessor FR and Field Replace able System are trademarks or registered trademarks of the Continuous Computing Corporation or its affiliates All other product names mentioned herein are trademarks or registered trademarks of their respective owners The products described in this document maybe protected by U S patents foreign patents or pending applications No part of this
82. ORS None SEE ALSO ubSvc on page 49 Continuous Computing Corporation 52 upSuite User s Guide ubSvcPeer ubSvcPeer NAME ubSvcPeer SYNOPSIS Get the node ID of the other server that is offering a service ub_node_id_t ubSvcPeer upbeat_t upbeatp ub_node_id_t node ub_svc_id_t service upbeatp A handle as returned by ubInit node Node ID of one server offering the service specified by the service argument For the current node use 0 service Service ID DESCRIPTION Get the node ID of the other server that is offering a service Typically the node argument is the current node as returned by ubNode upbeatp NULL RETURN VALUES ubSvcPeer returns the node ID of the other server if the handle passed in upbeatp is not NULL it finds the service and node is one of the servers otherwise it returns 0 ERRORS None SEE ALSO ubNode on page 44 ubSvc on page 49 Continuous Computing Corporation upSuite User s Guide 53 5 upBeat API Reference ubSvcPort NAME ubSvcPort SYNOPSIS Get the TCP or UDP port number for the service in_port_t ubSvcPort upbeat_t upbeatp ub_svc_id_t service upbeatp A handle as returned by ubInit service Service ID DESCRIPTION Get the TCP or UDP port number for the service from the upSuite configuration file RETURN VALUES If the handle passed in upbeatp is not NULL the service exists and it has a PORT attribute
83. RTBEAT gt tag with sockets set to route would look similar to the following lt HEARTBEAT NAME left right YPE POINT TO POINT IMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 ROUTE ROU lt LINK NETWORK Network2 ROUTE ROU lt HEARTBEAT gt NAME attribute E gt E gt Required This names the heartbeat The name must be unique that is the NAME attributes of all lt HEARTBEAT gt tags must be different TYPE attribute Required Must be POINT_TO_POINT Future releases of upBeat will support broadcast and multicast heartbeats Continuous Computing Corporation 152 upSuite User s Guide lt HEARTBEAT gt tag TIMEOUT_MSEC attribute Required Controls the time it takes the server to determine if a link to its peer is down The value is specified in milliseconds As latency increases you should increase the time We rec ommend starting with between a 3 1 or 4 1 ratio of TIMEOUT MSEC to RESEND_MSEC If you are using a MAN or WAN you will need to experiment with your ratios for your particular needs RESEND_MSEC attribute Required The time between sending out heartbeat packets The value is specified in millisec onds As latency increases you should increase the time We recommend starting with between a 3
84. The name must be unique that is the NAME attributes of all lt NODE gt tags must be different In a lt NODE gt tag the NAME attribute does not necessarily have to match the name in the host file NODE D attribute Reguired The numerical ID of the node Choose any number you like as long as it is unigue among all the lt NODE gt tags and a positive 32 bit integer greater than or equal to 1 DESCRIPTION attribute Optional This is your desired description of the given node lt INTERFACE gt subtag of the lt NODE gt tag The lt INTERFACE gt subtag specifies the name network and IP address IPv4 only of the interface NAME attribute Required Names the given interface The name must be unique that is the NAME attributes of all lt INTERFACE gt tags within a given lt NODE gt tag must be different Continuous Computing Corporation 150 upSuite User s Guide lt NODE gt tag NETWORK attribute Required Used by the lt LINK gt subtag in services and heartbeats to tie together two IP addresses from a pair of servers IP attribute Required unless the HOST attribute is used This is a host name or IP address HOST attribute You may replace the IP address attribute with the HOST attribute in which you would specify the host name of your server for example HOST ServerA Using HOST may be convenient for testing and evaluation purposes but we recommend that HA deployments rely on the IP address to reduce
85. USAGE usr sbin udactive A S f dataset a d dataset m mountpoint DESCRIPTION This command is used to request or force changes in active standby status Continuous Computing Corporation 128 upSuite User s Guide udactive OPTIONS The A flag makes the system on which the command is run the active system This is the default behavior This command causes upDisk go active as long as upBeat is not reporting another active for that dataset on the network and as long as there are no operator issues e g split brain upDisk registers or reregisters with upBeat to become active and waits to be informed it is active The primary reason for this flag is to indi cate which side should be the active after you have first installed upBeat The S flag makes the system on which the command is run the standby system The f flag forces the system on which the command is run to become either active or standby depending on the required argument you use Including this flag forces upDisk to clear any problems e g split brain and instructs upDisk to tell the other side if it is available to become the status opposite of that of the current machine for example if the current machine is active this command instructs the peer to go standby and vice versa Then the situation is identical to the A flag upDisk reg isters or reregisters with upBeat to become active and waits for instructions Use this comman
86. _STBY a group has registered for standby and is waiting for a standby directive INIT_ACT a group has received an active directive and is waiting for all of its applications to start INIT_STBY a group has received a standby directive and is waiting for its applications to terminate ACT a group is active its applications are running STBY a group is standby its applications are not running and THRASH a group has registered for standby or active a number of times but has not yet received a corresponding directive For a monitor the states are UNREG unregistered with upBeat SMART the monitor is running and interfaces to upBeat itself use the ups command to find its upBeat service state WAIT_ACT ubManager has registered the monitor to be active and is waiting for an active directive for the monitor WAIT_STBY ubManager has registered the monitor to be standby and is waiting for a standby directive for the monitor ACT the monitor is active and is therefore indicating that the resource it s monitoring is healthy STBY the monitor is standby and is therefore indicating that the resource it s monitor ing is not healthy REFUSE_ACT ubManager has started a monitor and registered that monitor to be active but before receiving an active directive the monitor has terminated and WAIT_STBY_REREG ubManager has registered a monitor to be standby and while waiting for a standby directive for the monitor the monitor has be
87. ability of the application The API can be organized into the following categories Initialization and termination activities e Normal operation including service registration and polling for application events Error handling The API library is defined in the files Libupbeat aand libupbeat h Summary of API Usage The minimum that an application must do to use the upBeat API and thus become an upBeat client is as follows the very first upBeat API call it makes must be to ubInit which returns a handle used for the rest of the API calls When the application terminates or when it is finished using upBeat the very last call must be to ubF ini to terminate the applica tion s registration with upBeat for the local node and to free the resources allocated for the application by ubInit Continuous Computing Corporation upSuite User s Guide 23 4 upBeat API Programmer s Guide The following guidelines provide a summary of the suggested minimum use of the upBeat API in a client application For a more detailed discussion see Guidelines for Designing Applica tions on page 25 All programs must call ubInit before they call any other function in the upBeat API e Programs should call ubAsync periodically to gather system information The best way to do this is is to use ubSetupPollfd together with the UNIX API call poll or select A single threaded application can add the upBeat fd to
88. ack directive y ub_ops_t opsp amp ub_ops Check command line arguments Note that argv 2 is the entire command to run and it may need to be guoted in the shell if it has arguments Example Continuous Computing Corporation upSuite User s Guide 59 6 upBeat Sample Applications wrapper lt svcname gt iostat 10 A if argc 3 fprintf stderr Usage s lt servicename gt lt command gt n argv 0 exit 1 svcname argv 1 glob_prog argv 2 if ubp ubInit NULL fprintf stderr ublnit failed errno d s n errno strerror errno exit 1 ub_ops arg void ubp printf upbeat is running n Find our node in the upSuite configuration ES if glob_id ubNode ubp NULL printf we are node d n glob_id else fprintf stderr upsuite misconfiguration this system does not match any node in the upSuite configuration n exit 1 Find our service in the upSuite configuration oy A if glob_sid ubSvc ubp svcname printf service id d service name s n glob_sid svcname else fprintf stderr no such service s n svcname exit 1 peerid may be 0 glob_peerid ubSvcPeer ubp glob_id glob_sid printf service peer is d n glob peerid Get service IP addresses the addresses upbeat manages during IP failover 60 Continuous Computing Corporatio
89. ad acts as the signal handler for mt multi svc The upBeat Interface Thread This thread is responsible for interfacing to upBeat The upBeat Interface Thread is also responsible for handling the split brain condition The Socket Acceptance Thread This thread is responsible for accepting clients for both services and then spawning other threads that actually handle the service All threads have access to the log file Ifmt multi svcisusedas a model for your own applications you can substitute your services for those of mt multi svc 74 Continuous Computing Corporation upSuite User s Guide The mt_multi_svc Sample Application Also responsible Main thread forks itself off as a daemon process before spawning child mt multi svc threads pl k eee LS notifications 1 L Thread spawns die gt notification for resolving split brain condition upBeat info local amp remote y node link and service status spawns die notification MtSysCmd Main Thread Acceptance servicehandling spawns die notification Thread 2 thread change of state ACT STBY spawns die user termination Y notification notification 4h L MtAppctl service a handling thread rites application activity with a f WriteToLog function that uses Y system Command service where mutex for exclusive access to log file user s passed in string is used as the argument to the system comman
90. ag on page 154 for information about configuring links Incorrect routing tables may create failure detection problems or a single point of failure leaving your system vulnerable to a split brain condition If you do not use the ROUTE attribute the default behavior is to not route specified via the Continuous Computing Corporation upSuite User s Guide 153 14 upSuite Configuration File upsuite conf SO_DONTROUTE socket option meaning that packets can only be sent to other nodes on the same subnet and can only be sent across hubs and switches This ensures reliable detection of link or network failures If you choose to route you must ensure that your routing tables are set up so that packets intended for one network do not get routed to a different network If this happens there could be undetected latent failures which could eventually lead to system downtime due to network outage without warning Therefore ensure that you have independent paths to each network and that routers do not route packets between the two networks lt SERVICE gt tag Configures the service Features of a service include two servers associated with the service At most one can be active that is active active is not supported Under failover circum stances upBeat will perform IP failover for this service if an IP address and interface are pro vided under the lt SERVICE gt tag A typical lt SERVICE gt tag might look similar to the f
91. ag translates the license file usually etc upsuite license into a binary file dat which the upSuite HA programs use and a text file txt which allows you to verify the number of valid licenses you have Running this command overwrites previ ously generated dat and t xt license files but not etc upsuite license This is typically what you would run when upgrading from older versions of upSuite HA Including the v verify flag causes a check to be run on the system to ensure the licenses are present in etc upsuite license Ifno arguments are specified this is the default command The f filename argument enables you to run the pro gram in a test environment without affecting the system For example you could run usr sbin uplicense f mytestfile fora system other than the one on which you are running the com mand etc upsuite license Refer to the Installation Guide for license installation instruc tions 20 Continuous Computing Corporation upSuite User s Guide ups command ups command NAME USAGE DESCRIPTION OPTIONS FILES SEE ALSO ups opt upsuite bin ups q This command gives status information for upBeat The output from this command notifies you of all available links nodes and services of which upBeat is aware Including the q flag will run this command once print out sta tus and finish Otherwise ups runs continuously alerting you to all changes in links nodes
92. al upBeat Clients PACA libupbeat status and directives TCP Socket s v queries requests acknowledgements upBeat Daemon Process lt 4 terminates A instantiates etc init d upbeat Script upBeat dl instantiates terminates Watchdog Process upsuite Log LJ File V reads and parses etc upsuite I upsuite conf Configuration File Figure2 upBeat operations sequence Failover TCP Sockets Peer Node s upBeat Daemon Heartbeats Process and coordination of services upBeat manages failover between two services If the active service unexpectedly terminates or it re registers requesting a standby role upBeat sends the standby service a directive to become active If necessary upBeat migrates the service IP address to the new active server this is known as IP failover by e Deconfiguring the service IP on the old active if it is alive e Configuring the service IP on the new active e Sending a gratuitous ARP so that systems and routers on the local subnet immediately communicate with the new active Continuous Computing Corporation upSuite User s Guide 1 Introduction to upBeat Operation Overview Upon startup upBeat spends a brief period of time discovering the state and health of the net work This period is configurable the default is five seconds During this time up
93. and services upsuite conf upbeat on page 17 upbeat on page 19 Continuous Computing Corporation upSuite User s Guide 21 3 upBeat Commands 22 Continuous Computing Corporation upSuite User s Guide 4 upBeat API Programmer s Guide This chapter contains an introduction to upBeat s Application Programming Interface API and gives information about how to use the API calls together in an application For details on the syntax and use of each API call see upBeat API Reference on page 33 For complete example programs see upBeat Sample Applications on page 57 In this chapter ABT OVETVIEW L si ii setai sia i one dubs i a ia sai a k od ness i e niais 23 Guidelines for Designing Applications ooooccconnoncnonconanonconnnanonn nono aaa aaa aaa aaa 25 API Overview By using the upBeat API you can reguest upBeat to send status information about the network to your application In addition the application can be notified whenever the state of the net work changes Notification lets the application know when any change occurs in the status of links nodes and services You can also register services with upBeat through the API Individual services may reguest active or standby status and upBeat grants or denies the reguested status according to its knowledge of the network As changes occur on the network upBeat dynamically designates components as active or standby to maintain high avail
94. andby and ipfs cannot proceed upDisk will assume a split brain condition Operator intervention is required before operations will proceed Warning messages about the underlying file system Oct 3 09 52 41 left unix DIRTY detected active SYNC standby DIRTY The message above indicates ipfs has detected that an underlying file system is dirty in need of repair and that ipfs expected to be in repair mode but is not In other words if either of the file systems are dirty and ipfs is not in repair then the above message is gener ated Note however that there is one circumstance under which you may be sent the above message unnecessarily If a link drops during replay upDisk restarts the replay again ipfs does not have enough information to know that this is the case and thus warns that the standby is dirty In each of the cases described above no action is required Oct 3 09 52 41 left unix NEW FS detected active OLD standby NEW The message above is a warning indicating that a new file system has been detected and ipfs is not in repair mode For more information see Replay and Repair on page 85 Continuous Computing Corporation 90 upSuite User s Guide Troubleshooting Console Messages Starting up and mounting ipfs At mount time when ipfs detects that the file system was busy had operations that were not acknowledged by the standby it reports the following Oct 09 18 38 30 port ipfs
95. are practices may have unexpected results of which system administrators should be apprised upDisk Administrator Considerations This section describes some situations which should be handled with care Stopping the Solaris machine There are several ways to shut down a Solaris system and not all of them ensure a graceful exit for upSuite software The following shutdown techniques run the upSuite shutdown scripts and are therefore recommended when you need to stop a Solaris system where upSuite is run ning usr sbin shutdown man shutdown sbin init man init The following shutdown methods cause a repair and should therefore be avoided if possible usr sbin reboot usr sbin halt and usr ucb shutdown To reboot the system the method we recommend to ensure proper upSuite operation is to use the following command usr sbin shutdown i 6 g O Stopping the NFS server If you run the command etc init d nfs server stop NFS turns off and noth ing is shared If you then run nfs server start NFS will turn on if there are entries in etc dfs dfstab however no upDisk directories will be shared whether or not NFS is turned on Therefore do not run nfs server stop There is no harm in running nfs server start even if NES is already running If you need to stop the Solaris NFS server manually with nfs server stop you must then stop and restart upDisk to re share the HA NES datasets The sequence of events and commands would
96. at runs on each node in the system Each upBeat instance interacts with every other upBeat configured in the configuration file upsuite conf for more infor mation see upSuite Configuration File upsuite conf on page 145 There may be any num ber of nodes running upBeat and any number of upBeat clients monitoring the network However any particular service may reside on at most two nodes on one node it will be in the standby state while on the other node it can be either in the standby or active state On any node where multiple services are running there may be some services that are active and some that are standby Thus a service not a node is referred to as being either active or standby When an application registers to provide a service the application can inform upBeat whether it would prefer to start off as the active or the standby upBeat may honor that request or not depending on the state of the rest of the system Once a service application is active it can reg ister with upBeat to become standby if it is no longer able to provide its service Alternatively a service application can refuse an active directive from upBeat if it is unable to provide the service Refer to Guidelines for Designing Applications on page 25 for more detailed infor mation about initialization and registration Continuous Computing Corporation 8 upSuite User s Guide How Does upBeat Work Local Node Loc
97. ate In this way the callback functions can perform slightly different behav ior depending on which function called them For more information about this and other argu ments to the callback functions see ubAsync on page 37 For an overview of the different types of callback functions see Callback Functions on page 34 Using the link callback function The Link callback function indicates whether a connection is available to the peer by return ing UP or DOWN To keep a mapping between the local and peer nodes interfaces use ubSvcIPPair The link callback function s ub link id t argument is the IP address of a peer node s Continuous Computing Corporation upSuite User s Guide 29 4 upBeat API Programmer s Guide interface and may be cast to an in_addr_t The following example shows an implementation of a link callback function static void link_callback void argp ub_link_id_t link ub_status_t status boolean_t async struct in_addr in in s_addr in_addr_t link printf link_callback link s status s called by s n inet_ntoa in UB_STATUS_UP status UP DOWN async ubAsync ubGetState When calling inet_ntoa be aware that it returns a pointer to a buffer that will change the next time inet_ntoa is called Therefore if you need to make multiple calls to inet_ntoa copy the contents of this buffer to your program s local buffer sto
98. ation File a group2 g2a2 usr lib ubmgr ubapp5 Group3 g group3 updisk ipfs The sample upsuite conf file etc upsuite examples upsuite 4ubmgr xml shown in the XML configuration file section also applies to etc upsuite examples ubmgr sc Continuous Computing Corporation upSuite User s Guide 213 18 ubManager Semicolon Delimited Configuration File Continuous Computing Corporation 214 upSuite User s Guide Typographic Conventions Typeface Meaning Example AaBbCc123 Courier font indicates the names of Edit your login file commands files and directories and At the ok prompt on screen computer output AaBbCc123 Bold Courier font indicates a com To turn the unit on type mand you type contrasted with on on at the ccpu gt prompt screen computer output Also used ccpu gt on for command arguments AaBbCc123 Bolditalics indicate a command line To delete a file type placeholder or token to be replaced FM filename with a real name or value AaBbCc123 Square brackets indicate an optional help argument do not type brackets dir filename a b Curly braces indicate a choice of grade a b c d f required argument do not type brackets The vertical line sepa rates the choices You must choose one and only one of the items AaBbCc123 Italics indicate book titles new e This manual is used in words or terms or words to be conjunction with the SPARCengine CP15
99. ble upSuite configuration options You can copy one of these files to et c upsuite upsuite conf and use it as the basis for a new configuration file Continuous Computing Corporation upSuite User s Guide 145 14 upSuite Configuration File upsuite conf ubManager has its own configuration file The same directory contains example files for ubManager e ubmgr xml sample 4ubmgr xml As an XML file upsuite conf contains tags and attributes Each tag or attribute corre sponds to an item you can configure The file s root tag lt UpSuiteConfig gt has several subtags that correspond to various configuration items such as networks nodes and services The rest of this chapter describes these tags in detail The upsuite conf file resides in the etc upsuite directory etc upsuite upsuite conf however for convenience the symbolic link etc upsuite conf is created during installation This symbolic link redirects the short name to the actual file in etc upsuite You may use either the et c or the etc upsuite path to refer to and edit this file This is similar to how Solaris manages the hosts file in etc and etc inet Managing the Configuration File on Multiple Machines You must configure upSuite on all active and standby systems in your high availability system The configuration file must be the same on all machines Each machine uses its IP addresses to determine which node it is If you change your con
100. bubmgr a operator Nx etc upsuite lt asi psuite 2 ubmgrcon ubmgr config dtd ubMgr Client App If ete syslog conf L configuration file has a local6 debug line usr lib ubmgr Creates default l install script ubmgr conf and scripts Figure2 ubManager components Continuous Computing Corporation 186 upSuite User s Guide 16 ubManager Command Reference This chapter explains ubManager s commands Shell Commands This section describes the usage and function of ubManager s UNIX shell commands Com mands are listed in alphabetical order Continuous Computing Corporation upSuite User s Guide 187 16 ubManager Command Reference NAME USAGE DESCRIPTION ubmfailover usr sbin ubmfailover h hostname p port group This command informs ubManager that the operator wants the specified group s to failover If a called for group has a lead ser vice ubManager calls the cause_failover script with the name of the group as an argument that is expected to cause the lead service for the group to failover If the group does not have a lead service ubManager immediately registers the group for standby if it is active The primary use ofubmfai lover is to cause an active group to go to standby If there is no lead service then running ubmfailover with the name of a group in standby will have no effect If a
101. c init d ubmgr stop etc init d updisk stop 5 Stop upBeat on the standby system etc init d upbeat stop 6 Stop upBeat on the active system etc init d upbeat stop 7 Start upBeat on all systems in your configuration etc init d upbeat start 8 Start any services of upBeat on all systems in your configuration For example etc init d updisk start etc init d ubmgr start lt UpSuiteConfig gt tag The lt UpSuiteConfig gt tag is the root tag of the configuration file Everything within this tag is parsed to extract configuration settings anything outside this tag is ignored The lt UpSuiteConfig gt tag has one required attribute VERSION which must be set to the ver sion number of the software you are using in the current release this must be set to 2 not 2 0 2 0 0 or any variation lt UpSuiteConfig VERSION 2 gt lt UpSuiteConfig gt lt UPBEAT gt tag The lt UPBEAT gt tag is optional This tag has one optional attribute STARTUPDELAY_SEC which specifies the startup period in seconds If the lt UPBEAT gt tag is not used or the attribute is not set the default is 5 seconds The following example shows the lt UPBEAT gt tag with the attribute set lt UPBEAT STARTUPDELAY_SEC 7 gt Continuous Computing Corporation 148 upSuite User s Guide lt NETWORK gt tag During the startup delay period specified in the lt UPBEAT gt tag ubInit will block lt NETWORK gt
102. ccccccsssssececesessnaeececessaaseccceceauececceceaseaecesceensaececcsensaeseceseneaeees 17 UPBEAT STARTUP SHUTDOWN SCRIPT ccccssssssssececesseasecececssuececcecesaesesescseececeeceesaeseeesensaeees 19 UPLICENSE COMMAND oia a c 20 UPS COMMAND A ie 21 Continuous Computing Corporation upSuite User s Guide 4 UPBEAT API PROGRAMMERS GUIDE 1 0 ener nines 23 API OVERVIEW ia ai aa saaa adele eee ee A ec ie ee ees 23 Summary OF API USAGE nierstene eee ere ii 23 GUIDELINES FOR DESIGNING APPLICATIONS 0 cs sseessseceeseecessececesceceeneeecesaeceseeecesaesenenaeeees 25 Initialization of upBeat Clients that Provide Services ooooonnnoniinnnnomorencrnncnncnnecnncnnos 25 Registering a Service With upBedt ccscccssccesceesscesecesncesseceaceceneeeacessecenseceaceceeeeenaeeenees 26 PONE S saaa A Ces as oti a ane aioe teak cea i i TN 28 Monitoring the state of the System cccccsessecessecsseeesceceneeeseceececeneeescesencesseceaceceeeeenseeeaees 29 Using Callback FUnctions 1 ceccesesccescesseeeeseeesceseessecsecacesecacsececeaeeeeeesseeeaeeaeeeasens 29 TEO NANA ai ane ina avi a ae k a a a a av aaa aa S a da adeir 30 TP Addresses and Radar a a ia ras 3 Detecting Split Brain Conditions cccecceeseesseceseeenseeececesecesceceneecaeesacensecesaeceeeeceneeeaaees 3 Strategies for Resolving a Split Brain ConditiON oooonccnnnnnnconnonnnnnncnnnnnnnancnnnran cnn nrnncnn 32 5 UPBEAT API REFERENGE ut a ties dag
103. co oo Internet host table LIT 140 141 140 141 The etc vfstab File NFS server serverl serverl networkl server2 server2 networkl serverl network2 server2 network2 Below is the etc vfstab file edited to match the system illustrated in Figure 8 device to mount fd proc dev dsk c0t0d0s1 dev dsk c0t0d0s0 dev dsk c0t0d0s6 dev dsk c0t0d0s3 dev dsk c0t0d0s4 swap These partitions dev dsk c0t1d0s2 dev dsk c0t2d0s2 dev dsk c0t3d0s2 This partition is dev md dsk do device mount FS fsck mount mount to fsck point type pass at boot options dev fd fd no proc proc no swap no dev rdsk c0t0d0s0 ufs 1 no logging dev rdsk c0t0d0s6 usr ufs 1 no logging dev rdsk c0t0d0s3 var ufs 1 no logging dev rdsk c0t0d0s4 opt ufs 2 yes logging tmp tmpfs yes are reserved for OLDS metadevice dev md dsk d0 dev rdsk cO0t1d0s2 dev rdsk c0t1d0s2 dev rdsk c0t1d0s2 reserved for upDisk Leave it commented out dev md rdsk d0 ipfs ufs 2 yes noatime logging Continuous Computing Corporation upSuite User s Guide 169 14 upSuite Configuration File upsuite conf Configuration C In configuration C two servers are connected via redundant networks upBeat is running between both servers and upDisk is running between the disks The upDisk service uses one migrating IP address 172 17 8 177 Two clients are connected t
104. code 65 Stopping upBeat 19 Stopping upDisk 139 syslog 15 89 138 System administration 13 87 T TCP 111 114 technical support 219 Timeout msec attribute 153 Troubleshooting conflicting file modification times 96 file system access denied 94 file system out of disk space 95 HA NES failover 99 1pfs unmount unsuccessful 95 monitoring upDisk 89 split brain conditions 97 Typographic conventions 215 U ubAckSvc 36 ubAsync 37 ubFini 40 ubManager 10 ubNode 44 ubNodeName 45 ubRegSvc 46 ubServiceIP 47 ubSetupPollfd 48 ubSvc 49 ubSvcIPPair 50 ubSvcName 52 ubSvcPeer 53 ubSvcPort 54 udactive 128 UDP 111 114 udrepair 131 udstat 132 Underlying file systems 88 118 119 Unmounting ipfs 95 upBeat benefits 6 features 6 maintenance 15 major functions 6 monitoring 15 network operation 3 7 starting 19 stopping 19 upbeat admin command 17 upbeat startup shutdown script 19 upDisk benefits 81 82 features 81 operations 82 starting 139 stopping 139 updisk admin command 137 updisk startup script 139 uplicense 20 upsuite conf 145 editing 147 example 159 ubGetState 41 ublnit 43 Continuous Computing Corporation 224 upSuite User s Guide V Virtual LANs VLANs 15 W WANs 153 171 wrapper sample API program 57 source code 58 Write operations 85 X XML 145 146 attributes 146 Continuous Computing Corporation upSuite User s Guide 225
105. come active For an application the states are DEAD the application is not running RUN the application is running and BROKEN the application has been started and has unexpectedly died a number of times 194 Continuous Computing Corporation upSuite User s Guide Shell Commands OPTIONS a This option includes the application status in the output h hostname This option changes the IP address that ubmstat uses to communicate with ubManager IPv4 IP addresses only are supported The default is the local host hostname is the resolvable name of a machine not its IP address Note that no form of security is used e g authentication with this command and it can be run by root and non root users i interval If this option is specified ubmstat will not exit until you send a break with Ctr1 C Each interval seconds ubmstat prints out the latest status m This option includes the monitor status in the output This option will cause group status to not be in the output If N is used but neither a nor m then ubmstat will behave as if all three options were used Continuous Computing Corporation upSuite User s Guide 195 16 ubManager Command Reference Example ubmstat ma Group State groupl ACT group2 STBY Monitor State glnlmi AGT ginim2 SMART gln2mi UNREG gln2m2 UNREG g2n1m1 ACT g2n1m2 ACT g2n2m1 UNREG g2n2m2 UNREG Appl State ubapp11 RUN ubapp12 RUN ubapp21 DE
106. d and D Amp standard output is redirected back to the mt multi svc lo LT Multi SVC I0g socket so the user can see the results of fik the command e g Is I Application Control service where user 1 Enters u or U to get the upBeat information for the process 2 Enters I or L to get log file dump 3 Enters k or K to kill entire process 4 Enters q or Q to have service disconnect from client Figure4 mt_multi_svc application architecture Termination You can terminate the mt multi svc application at the console by using the k 111 utility to send a SIGTERM or SIGQUIT signal to the application A user connected to the Application Control service Mt AppCt 1 can terminate the applica tion by issuing a k111 command to the service The command is sent over a socket to an MtAppCt 1 service handling thread which after a positive confirmation then issues a SIGQUIT signal to the main thread Continuous Computing Corporation upSuite User s Guide 75 6 upBeat Sample Applications To Usemt multi svc Touse mt multi svc invoke the program on both of your systems by performing the fol lowing steps Step Description Command 1 Change directory cd 2 Copy the directory src upbeat cp r opt upsuite src upbeat myupbeat 3 Change to the myupbeat directory cd myupbeat 4 Build using the supplied makefile make Continuous Computing
107. d accepts clients for services and spawns other threads to handle those services Continuous Computing Corporation upSuite User s Guide 69 6 upBeat Sample Applications Upon instantiation the application becomes a daemon and the main thread instantiates the other threads The main thread then acts as a signal handler receiving the SIGUSR1 signal from the upBeat Interface Thread upon a change of service state for example a service going from standby to active or vice versa Concurrency is achieved among the other threads via pipes condition variables and mutexes You can terminate the application either through one of the services or by sending a SIGTERM or SIGQUIT signal to the application via the kill command at the console This section explains the following about the mt multi svc application e Configuration File e System Architecture e Application Architecture e Termination ToUsemt multi svc The source code is located in opt upsuite src upbeat mt multi svc c Configuration File For your reference below is an example of a configuration file etc upsuite conf you could use on each of the two systems using mt multi svc Your configuration how ever may reguire a different setup lt xml version 1 0 gt lt UpSuiteConfig VERSION 2 gt lt UPBEAT STARTUPDELAY_SEC 5 gt lt NETWORK NAME Network1 DESCRIPTION 172 17 33 0 24 gt WORK NAME Network2 D
108. d the data on the standby is repaired by being deleted For these reasons a server may become active in only these three ways e Normal failover e Cold start of a server that was active last time it was started e Operator intervention In other words a server cannot automatically become active upon reboot i e upon restart of upBeat upDisk unless it was active when it went down This prevents an inappropriate file system e g a replaced disk from becoming active One side effect of this constraint is that when upDisk is first configured on a pair of servers you must designate one of them as active Normal Failover In normal operations the assumption is that there are no double component failures and that failed components are quickly repaired low MTTR If the standby upDisk server fails there is no disruption in service The active server summa rizes all changes to data in order to replay them to the failed standby server once it is functional again When it is the standby is resynchronized with the active server either via replay or Continuous Computing Corporation upSuite User s Guide 101 9 Failover repair See Replay and Repair on page 85 If the active system fails it quickly fails over to the former standby making the former standby the new active server The newly active server summarizes all changes to data When the for merly active is back in service it assumes the standby role The new stan
109. d with caution Note that this command will succeed even if the standby is not in sync with the active This flag is primarily for recovering from split brains or other problems that require operator intervention Without this command may refuse to do what is asked based on the current system status from upBeat The lt dataset gt argument limits the operation to the speci fied dataset The a flag specifies all datasets in ipfstab The d flag is the default The following two commands yield the same information udactive dataset and udactive d dataset The m flag allows you to specify the mount point of the dataset This is only useful if the two are different from one another Continuous Computing Corporation upSuite User s Guide 129 12 upDisk Command Reference FILES etc ipfstab etc upsuite upsuite conf SEE ALSO usr sbin udstat usr sbin udrepair Normal Failover on page 101 Continuous Computing Corporation 130 upSuite User s Guide udrepair udrepair NAME USAGE DESCRIPTION OPTIONS FILES SEE ALSO udrepair usr sbin udrepair dataset a d dataset m mountpoint This command initiates a repair as long as updisk is running The dataset argument limits the status check to the speci fied dataset only Refer to the Options of the udactive command for explana tion of the a d and m flags as their usage is identical with udrepair etc
110. dataset and mountpoint arguments specify the dataset and mount point from the ipf stab file The h elp flag prints the usage of this command The V ersion flag prints the version of upDisk FILES etc ipfstab etc upsuite conf SEE ALSO usr sbin upbeat usr sbin udrepair usr sbin udstat usr sbin udactive Continuous Computing Corporation 138 upSuite User s Guide updisk startup script updisk startup script NAME USAGE DESCRIPTION OPTIONS FILES SEE ALSO updisk startup script etc init d updisk start stop This command is run automatically upon Solaris reboot and invokes usr sbin updisk If you have installed upDisk and want to begin using it without rebooting then use the above command with the start argument start starts upDisk for all datasets in ipfstab and mounts the ipfs and underlying file systems stop stops upDisk for all datasets in ip fstab and unmounts the i pfs and underlying file systems etc ipfstab usr sbin updisk etc rc2 d K98updisk etc rc3 d S98updisk init 1M inittab 4 Continuous Computing Corporation upSuite User s Guide 139 12 upDisk Command Reference Continuous Computing Corporation 140 upSuite User s Guide 13 upDisk API This chapter explains upDisk s optional application programming interface API In this chapter APT OV VIEW Li chal sa dai ea i Eiht 141 Function Call Guid elie ks esse ias an Ka ss a boven
111. dby but will not permit a write to succeed on that machine until it is the active This means that applications can open a file for writing in order to become operational but will not be permitted to write to the file until the standby system becomes the active node Files on the active system are always avail able for writing Writes for FIFOs and devices special files are not replicated because they are local in scope As such writes to these files are allowed on both the active and standby at all times Replay and Repair In normal operation upDisk sends any file system operations over the network to the standby server If the standby system is unavailable the active server tracks changes When the standby server is once again available the active server sends over all missing changes This function is called replay If upDisk ever detects that the two servers are out of sync or that either local file system has been damaged it performs a complete checksum on both sides and makes necessary changes This function is called repair If a repair fails for any reason upDisk tries to repair again until the repair is successfully completed Continuous Computing Corporation upSuite User s Guide 85 7 Introduction to upDisk 86 Continuous Computing Corporation upSuite User s Guide 8 upDisk Administration In this chapter we detail issues specific to system administrators When applied to upDisk common softw
112. dby is resynchronized with the new active server either via replay or repair Note that if you are gracefully or otherwise failing over to the standby system and there are large amounts of data waiting to be written to its local file system there may be a pause to allow the standby system to process these operations before it becomes the active system The console will inform you that operations are pending during which time you can use udstat for more information The Standby Throttle Option setting in the ip fstab file allows you to limit the number of operations that can remain pending Double Failures Double failures occur when the active system attempts to failover to the standby system but cannot because that system is offline or dirty If this happens upDisk will be out of service for the dataset After fixing the problem that caused the failure on the active you must manu ally run udact ive on the active system to return it to service In the case of a disk failure SCSIbeat initiates a failover to the standby If that system is offline or dirty and if after the attempted failover the disk becomes operational again upDisk on the active system will not automatically resume its active role it will instead fail because its peer 1s offline even though the disk problem that originally initiated the failover has been fixed As a result both systems are out of service and you must manually run udact ive on the active system to
113. de updisk admin commana updisk admin command NAME USAGE DESCRIPTION updisk admin command updisk c d F s S v dataset mountpoint h V This command starts the watchdog process and the upDisk dae mon these are the core upDisk executables Typically this command is run by the upDisk startup script and not manually Continuous Computing Corporation upSuite User s Guide 137 12 upDisk Command Reference OPTIONS Running this command without any arguments causes upDisk to run as a daemon Program messages are sent to syslog Including the c flag causes program messages to be sent to your console in addition to any other place they are being sent for example the sys 1log if upDisk is running as a daemon Including the d flag runs upDisk in debug mode so that upDisk runs in the foreground Messages that would usually go to syslog will output to the screen along with additional debugging information The F sck flag causes a repair The s flag forces program messages to be sent to syslog even if for example you have already run the command with the d flag which routes messages to the console rather than syslog The S flag causes program messages to not be sent to syslog even if you have already run a command which routes them there for example if upDisk is running as a dae mon The v erbose flag sends more program messages with more detail The
114. de are active 5 If a service is active and its service callback function indicates that its peer service is also active the service has positively detected a split brain condition Strategies for Resolving a Split Brain Condition How a service resolves a split brain condition depends on the application You can devise any strategy best suited to your configuration needs For example e Issue a message indicating that operator intervention is required e Always have the service running on the node with the lower or higher node ID value remain active while the other goes standby e Have the service that most or least recently became active go standby e Design your code to prefer the system that is most reliable most accessible or has the most memory Continuous Computing Corporation 32 upSuite User s Guide 5 upBeat API Reference This chapter contains an alphabetical listing of the function calls in the upBeat API In this chapter Callback Functions dadas 34 Quick Reference of API Function Calls ooocnnnnoccnonccnnononcnoncnnconnnononnncconnnnonn corno aaa aaa 35 IDA A Sed A O et 36 IDAS MA a a as 37 EoI maT TE Bee ee a a a NSA RASA A a 40 bGetState J arr a oee See acne a eee Nh ae ed sia tes 41 O PAE O a ON 43 Pi TOO 44 UDNOd NAME iia E A io Ae 45 UDRESSVC O i ai 46 a ete aa a ein ed ha tel nied et Re 47 ubSetupP llista nas 48 MA aed WR MHA BA A RS 49 UDSVCIPP alt s secs ides sis da a ol
115. dev dsk c0t0d0s0 c0t0d0s0 dev dsk cO0Ot0d0s0 first and dev vx dsk c0Ot0d0s0 second do dev md dsk d0 first and dev vx dsk d0 second v0 dev vx dsk v0 disk0 v0 dev vx dsk disk0 v0 Table 8 Local file system device entries localfs mount options The letters of the local file system device entry indicate the fol lowing cn SCSI controller number tn SCSI device target number dn Disk number SCSI LUN default typically 0 sn Partition slice For example if you have one SCSI controller one disk and your partition slice is 7 your entry here would be c0t0d0s7 Refer to the following Solaris manual pages for more information sd 7D disks 1M mount 1M mkfs 1M newfs 1M Iflocalfs type is this field should be If this field is no user specified mount options are passed to the localfs mount otherwise this field is passed as o localfs mount options Note If localfs type localfs device and localfs mount options are all then localfs mount point is treated as a local subdirectory no fsck 1s done and no attempt is made to mount a local file system 120 Continuous Computing Corporation upSuite User s Guide ipfs Mount Options ipfs just uses the subdirectory share options If this field is upDisk does not share the file system Otherwise the entire i pf s mount point is shared via NFS with the specified share options and the NFS daemon and mount daemon ar
116. e v erbose flag increases the amount of detail in your out put messages regardless of where they are being sent The h elp flag prints the usage of this command The V ersion flag prints the version of upBeat upsuite conf upbeat on page 19 18 Continuous Computing Corporation upSuite User s Guide upbeat startup shutdown script upbeat startup shutdown script NAME USAGE DESCRIPTION OPTIONS FILES SEE ALSO upbeat etc init d upbeat start stop This command is run automatically upon Solaris reboot and invokes usr sbin upbeat Including the start argument allows you to begin using upBeat after you have installed it without rebooting The stop argument gracefully shuts down upbeat When Solaris gracefully shuts down it uses this command and argu ment Use this argument if you are having a problem and want to debug Then run usr sbin upbeat d When you are done debugging run etc init d upbeat start upsuite conf upbeat on page 17 Continuous Computing Corporation upSuite User s Guide 19 3 upBeat Commands uplicense command NAME USAGE DESCRIPTION OPTIONS FILES SEE ALSO uplicense usr sbin uplicense v o filename Run this command after you have installed your licenses to val idate your upSuite HA software This command is run at boot time with the o flag by etc init d uplicense Including the o overwrite fl
117. e server A forgets that the was run on it creating a new split brain when B is again operational Failover Problems If one or more of your systems is down 1 Determine which system has the most current data by determining which went down first or last Do this by inspecting the relevant files and or system logs and by using the udstat hh command 2 Runudactive f dataset on the system with the most current data to force it to become active A repair will start automatically svc_max_msg_size Causes Problems when Adding CCPUdisk Package When installing the CCPUdisk software package you might see a message like the follow ing Unable to update system parameters dynamically You must reboot before starting updisk Installation of lt CCPUdisk gt was successful IMPORTANT NOTICE This machine must now be rebooted in order to ensure sane operation Execute shutdown y i6 g0 and wait for the Console Login prompt Note particularly the first line Unable to update system parameters dynamically As directed in the messages reboot the machine Watch for the following in the boot messages sorry variable svc_max_msg_size is not defined in the rpcmod module Continuous Computing Corporation upSuite User s Guide 99 8 upDisk Administration If you will not be configuring HA NFS or will be using the UDP protocol with HA NFS you can safely ignore this situation However if you
118. e or a set of services that you have defined as a group In doing so you can monitor the health of the system and according to the conditions you have configured determine whether or not a given system or service may remain the active one ubManager relies on service monitors to learn about the health of the system essentially using upBeat as a database to determine whether a system can remain active Monitor processes may be scripts or binary executables Each service monitor is designed to watch a particular 10 Continuous Computing Corporation upSuite User s Guide SCSlbeat element of the system and report status back to ubManager via upBeat ubManager can support five types of service monitors permanent transient periodic smart or dumb Permanent transient and periodic monitors function exactly as their names imply Smart monitors are able to directly update service status in upBeat while dumb monitors rely on ubManager to perform this function based on the exit status of the given service SCSlbeat SCSIbeat a component of upBeat monitors disks and reports failures Solaris may spend a long time attempting retries on a failed disk before reporting the event to disk dependent appli cations SCSIbeat however notifies upBeat of disk failures promptly upBeat then initiates disk failover designating disk dependent applications as standby When a disks fails or becomes unresponsive SCSIbeat tells upBeat
119. e started if necessary Exporting subdirecto ries of the ip fs mount point is not supported by ipfstab and etc init d updisk upDisk shares and unshares file systems via NFS depending on the share options Delete or comment out entries in etc dfs dfstab for either the underlying local file systems or the ipfs file systems ipfs Mount Options The mount options of i pfs the filesystem kernel control various performance and integrity tradeoffs You set these options by editing the ipfstab file Options for Returning to the Application return local sent recv remote There are four opportunities to return to the application controlled by the return local sent recv remote mount option recv is the default option Table 9 defines each of the return options Continuous Computing Corporation upSuite User s Guide 121 11 The ipfstab File Return option Action Notes local Return when the local This option provides asynchronous replica operation completes tion It may be useful when applications have bursty sequences of writes This option induces a lag in replication and is not appro priate for continuously exceeding network capacity sent Return after ipfs This option provides additional reliability in sends the operation a network where packets are not lost e g across the network direct connections but allows the applica tion to return before it is known to be received by the s
120. ed The numerical ID of the service Choose any number you like as long as it is unique among all the lt SERVICE gt tags and a positive 32 bit integer greater than or equal to 1 Example SERVICE_ID 17 TYPE attribute Required The type of service Currently only one type BASIC is defined Example TYPE BASIC STARTUPDELAY_SEC attribute Required A period of time specified in seconds between the time the given service is func tional and the time when it is possible for upBeat to make the service active Example STARTUPD ELAY_S EC 7 PORT attribute Optional The port of the given service If the port is specified it is made available to the appli cation to use as it sees fit Example PORT 1776 lt NODE_REF gt subtag of the lt SERVICE gt tag Required One or two lt NODE REF gt subtags are required to specify the nodes on which the service runs Continuous Computing Corporation upSuite User s Guide 155 14 upSuite Configuration File upsuite conf NODE ID attribute This tag has a single required attribute NODE_ ID which is a node number The number must not be the same as in any other lt NODE REF gt subtag within the same lt SERVICE gt tag Example lt NODE_REF NODE_ID 1 gt lt L NK gt subtag of the lt SERVICE gt tag Optional For those services that require that the acti
121. empting unsuccessfully to interact with an incompatible upBeat daemon Continuous Computing Corporation upSuite User s Guide 55 5 upBeat API Reference Continuous Computing Corporation 56 upSuite User s Guide 6 upBeat Sample Applications This chapter explains the three sample applications included with your upSuite HA software wrapper status and mt_multi_ svc Each of these applications demonstrates an aspect of upBeat s functionality wrapper demonstrates initialization and failover status performs an ongoing system status checkup and mt_multi_svc handles multi ple threads and split brain events These programs use the upBeat APIs and are intended to provide examples you can follow when developing your own client applications The source code for each program is located in opt upsuite src upbeat In this chapter The wrapper Sample Application peesi orenen reiasa oree e SESS EE E SES e iEn 57 The status Sample Application ooooncninnnnnionnnononncononononnnnanonn nono conc a aaa aaa coca non nc on non aaa 64 The mt multi svc Sample ApplicatiON oooncnnnnnonnonnonnconononononononannnnnrnn Siesa EEES 69 The wrapper Sample Application wrapper demonstrates the major initializing and failover functions of upBeat wrapper is a proxy service the name of a service is passed to wrapper on its command line The ser vice passed in to wrapper must be configured in upsuite conf wrapper first calls ubInit
122. eracts with it This will cause a failover to occur rightt etc init d updisk stop rightt etc init d upbeat stop Continuous Computing Corporation upSuite User s Guide 103 9 Failover Step Description 5 Edit the configuration file and or ipfstab to right currently offline reflect your changes right vi etc upsuite conf right vi etc ipfstab Restart upBeat and everything that interacts with it right currently standby right etc init d upbeat start right etc init d updisk start If desired fail back to the original active left currently active left udactive S a Table 7 Performing a managed failover Boot Seguence upDisk performs the following steps for each dataset upon startup This section assumes each upDisk dataset is using an entire local file system 1 upDisk checks to see whether the local file system is mounted If so upDisk alerts the operator and shuts down This is to avoid problems with people or applications coming in the back door and modifying the local file system directly The file system should only be controlled and accessed through upDisk ip fs In particular the file system should not be mounted via etc vfstab upDisk checks to see if the local file system has damage sck m and notes this infor mation for future reference Note It is recommended that UFS file systems be mounted with the logging option in which case
123. essary failover Solaris DiskSuite takes approximately three seconds to detect a disk failure and detach the disk If the SCSIbeat timeout set in the TIMEOUT_MSEC attribute of the lt NODE gt tag is less than this SCSIbeat will fail and force a failover unneces sarily In order to avoid this and failover only if both disks fail set TIMEOUT_MSEC to 3500 msec or more depending on the specific performance of your disk drive Continuous Computing Corporation upSuite User s Guide 11 1 Introduction to upBeat 12 Continuous Computing Corporation upSuite User s Guide 2 upBeat Administration This chapter details issues specific to system administrators Guidelines to Ensure High Availability Certain system administration activities can prevent even the highest priority real time pro cesses in the system from running If you are administering a high availability system with aggressive timeouts we recommend you avoid these types of interaction with a running high availability machine whenever possible The following activities should be undertaken with care or avoided e Installing software Inserting or Removing Media e Stopping the system The rest of this section explains each of these areas of potential concern in more detail Installing software The drvconfig add_drv or pkgadd commands temporarily lock up the system and cause upBeat heartbeats to fail This results in a split brain condition Note that drvconf
124. etState ubAsync calls up to four callback functions A pointer to a data structure is passed to ubAsync that gives the functions if any to call back The callback functions can include any of the following e The node callback function is called whenever a change to a peer node s state occurs Typ ically when all of the links to a peer node are down this callback function is called to indicate that the peer node is down When a link comes back up this function is called to indicate that the node is back up e The link callback function is called whenever a change to the state of a link to a peer node changes the link has either gone down or has come up e The service callback function is called whenever a change to the state of a service occurs either the service has become active or has become standby e For an upBeat client that provides a service the directive callback function is called when ever upBeat wants to initialize or change the state of the service either the service is to go active or to go standby The service must acknowledge a directive with ubAckSvc 28 Continuous Computing Corporation upSuite User s Guide Guidelines for Designing Applications For more information see Callback Functions on page 34 Monitoring the state of the system The upBeat API provides two functions that you can use to find out the state of the system ubAsync and ubGet State e ubAsync get
125. eta ae a Le Me a ene ahs 6 Why Should Use upBeat Tocino Ip asi sn ss i is a i suns ss sa cios 6 How Does upBeat Work iii ii si Vei ik ai a e i Au S 7 Falover L ans ais ais a ee hoe kG aa i a ai a es ia i a te as a Beeb a eee ack eh 9 UD Mania POR ss asa stos e ss i si a si a a sa a EE EREE E 10 SCS Death NR e ea is 11 Continuous Computing Corporation upSuite User s Guide 5 1 Introduction to upBeat What is upBeat upBeat is a failure detection manager that monitors the activity and state of components over an IP network upBeat detects node interface link or application failures and initiates failover The three major functions of upBeat are 1 Manages and initiates failover 2 Shares information with upBeat running on other components 3 Shares data with interested local applications upBeat is designed for failure detection and failover in an IP network environment Other features of upBeat include e Heartbeats Server server monitoring allows a standby server to take over in case of failure e Server service monitoring notifies a server as to which services need to be removed to a standby server e Client server monitoring notifies clients which servers are up and which services are active on each server Immediate disk failure notification upBeat is available for Solaris 7 8 9 and 10 on SPARC Why Should Use upBeat upSuite HA provides rapid and flexible failure detection over an IP network By provid
126. eta ces 82 What is upDisk upDisk provides simple reliable file system replication over a network upDisk performs all of the functions necessary for completing simultaneous disk writes across a TCP IP network at wireline speeds Features of upDisk include e Simple administration e No disruption during failover e Rapid recovery from temporary network failures e Automatic or manual recovery from prolonged outages or equipment replacement Full integration with upBeat architecture Easy HA NES construction e No API or application changes necessary The upDisk server software and optional client API are available for Solaris SPARC 7 8 9 and 10 Why Should Use upDisk By relying on redundant systems storage and networks upDisk allows system configurations that prevent any single point of failure This fact significantly reduces the need for single point hardening and creates a new level of flexibility dramatically lowering the cost of redundant application and NFS server systems deployment Continuous Computing Corporation upSuite User s Guide 81 7 Introduction to upDisk In addition other benefits of upDisk include e Control over which files are in replicated directories and which files are in conventional directories e Freedom from worry about application data redundancy e High availability provided to all NFS clients How Does upDisk Work upDisk continually replicates data over a redundant netw
127. etadata and attribute op Write operations coalesce whenever possible into an offset and length The general case of sequential writes into a file will require one op Random writes generally coalesce over time into one op The amount of memory allocated is therefore generally proportional to the number of files to which writes have been performed Examples are provided below e Creating 10 000 files with names 0000 through 9999 would allocate 10 000 x 100 bytes for ops 5 x 10 000 for names equaling a total of 1 050 000 bytes e Deleting 10 000 files with names 0000 through 9999 would allocate the same amount of memory e Writing 1 byte to 10 000 different files would allocate 1 000 000 bytes for ops e Writing 1MB to 10 000 files would allocate 1 000 000 bytes for ops e Writing ITB to 10 files would allocate 1 000 bytes for ops Standby Throttle Option throttle low high ipfs has a multi threaded standby implementation which allows it to acknowledge opera tions from the active before they are performed e g return local sent recv In situations where the active system presents many operations faster than the standby can per form them the drain time for these operations on the standby may be significant during failover In other words it may take several seconds to drain thousands of operations To limit the failover drain time a standby throttle implemented with a high water low water mecha
128. f disks is attached to each server via a metadevice managed by Solaris Online Disk Suite upDisk is replicating the file systems on the metadevice Ethernet Switch Ethernet Switch 172 17 8 141 172 18 8 141 172 17 8 140 172 18 8 140 A AAA A ia 1 172 17 8 177 1 l i 1 i 5 L o Cc O lt a upDisk Metadevice managed by Online Disk Suite Array of disks Figure8 Configuration B block diagram Continuous Computing Corporation 166 upSuite User s Guide Sample Configurations The upsuite conf File Below is the upsuite conf file edited to match the system illustrated in Figure 8 lt xml version 1 0 gt lt UpSuiteConfig VERSION 2 gt lt UPBEAT STARTUPDELAY_SEC 5 gt lt NETWORK NA lt NETWORK NA i ll Network1 gt Network2 gt zi ll lt NODE NAME serverl1 NODE_ID 1 gt lt INTERFACE NAME hme0 NETWORK Network1 TP 172 17 8 140 gt lt INTERFACE NAME hmel NETWORK Network2 IP 172 18 8 140 gt lt NOD El V lt NODE NAME server2 NODE_ID 2 gt lt INTERFACE NAME hme0 NETWORK Network1 TP 172 17 8 141 gt lt INTERFACE NAME hmel NETWORK Network2 TP 172 18 8 141 gt lt NODE gt lt HEARTBEA NAME serverl server2 YP
129. fer by more than one second However the order of the modification times should be identical on either system Therefore utilities that check sort and compare modification times should produce equivalent results If you use the touch command in UNIX to set the modification times those remain absolute Repair If a repair fails for any reason upDisk tries to repair again immediately Each time a repair fails upDisk waits longer before the repair is attempted again until a maximum of one attempt every four minutes is reached once that maximum is achieved the attempts continue indefi nitely or until the repair is able to successfully complete You can manually initiate the repair by running udrepair on the active system If a repair has already been started by upDisk you will get a message that a repair is already in progress 96 Continuous Computing Corporation upSuite User s Guide Troubleshooting other issues Split Bra n Conditions Any HA environment is at risk for a split brain condition We ve provided examples below to help you rectify a split brain condition A split brain can occur under the following two conditions 1 All network connections have been severed 2 upBeat thinks all network connections have been severed for example because the time outs have been set too low in upsuite conf If a split brain occurs it is likely that both sides will assume the active role When the split is rectified
130. figuration you must change or copy the configuration file to every other system and you must restart upBeat and everything that interacts with upBeat Using XML in the Configuration File XML follows strict guidelines For example all characters are case sensitive There are two types of attributes required and optional The required attributes must be present for the con figuration file to function the optional attributes can be omitted Each of the attributes is explained in detail in this chapter Avoiding Name Conflicts in the Configuration File No two nodes or services can have the same name or ID In other words the NAME and NODE ID or SERVICE ID attributes of any given lt NODE gt or lt SERVICE gt tag must be set to different values from the same attributes in other tags of the same type For example a lt NODE gt tag and a lt SERVICE gt tag could both contain NAME attributes set to the same value but no two lt SERVICE gt tags should contain NAME attributes set to the same value Within a node partition names must be unique That is no two lt PARTITION gt subtags within a given lt NODE gt tag can have NAME attributes set to the same value However if the lt PARTITION gt subtags are within different lt NODE gt tags the value of NAME can be identi cal if desired Continuous Computing Corporation 146 upSuite User s Guide Editing upsuite conf Example Valid namespace use The following partition name
131. figuration file upsuite conf DESCRIPTION Get the ID from the upSuite configuration file for a service RETURN VALUES ubSvc returns the service ID a positive integer if the handle passed in upbeatp is not NULL and it finds the service otherwise it returns 0 ERRORS None SEE ALSO ubServicelP on page 47 ubSvcName on page 52 Continuous Computing Corporation upSuite User s Guide 49 5 upBeat API Reference ubSvcIPPair NAME ubSvcIPPair SYNOPSIS Get a pair of IP addresses on the same network for a pair of servers configured to provide a ser vice int ubSvcIPPair upbeat t upbeatp ub node id t node ub svc id t service in addr t in addr ub ippair t ippairp upbeatp A handle as returned by ubInit node Node ID or 0 for the current node service Service ID in_addr Previous IP address or 0 ippairp Pointer to location where ubSvcIPPair stores the IP addresses DESCRIPTION Services in the upSuite configuration file are configured for certain nodes and networks This func tion fetches a pair of IP addresses on the same network one from each node and places them in the ub_ippair_t pointed to by ippairp typedef struct ub_ippair in addr t local in addr t remote ub ippair t The address for the node specified by the node argument is placed in ippairp gt local the other address is placed in ippairp gt remote Subsequent calls return additional IP add
132. file edited to match the system illustrated in Figure 7 Note that listed to the right of the IP address is the actual name of the server Internet host table LIQ LT BaL nfs server 172 17 8 140 serverl serverl networkl 172 17 8 141 server2 server2 networkl 172 18 8 140 serverl network2 172 18 8 141 server2 network2 164 Continuous Computing Corporation upSuite User s Guide Sample Configurations The etc vfstab File Below is the etc vfstab file edited to match the system illustrated in Figure 7 device device mount FS fsck mount mount to mount to fsck point type pass at boot options fd dev fd fd no proc proc proc no dev dsk c0t0d0sl swap no dev dsk c0t0d0s0 dev rdsk c0t0d0s0 ufs ab no logging dev dsk c0t0d0s6 dev rdsk c0t0d0s6 usr ufs 1 no logging dev dsk c0t0d0s3 dev rdsk c0t0d0s3 var ufs dl no logging dev dsk c0t0d0s4 dev rdsk c0t0d0s4 opt ufs 2 yes logging swap 2 tmp tmpfs yes This partition is reserved for upDisk Leave it commented out dev dsk c0t0d0s7 dev rdsk c0t0d0s7 Continuous Computing Corporation upSuite User s Guide 165 14 upSuite Configuration File upsuite conf Configuration B Like configuration A configuration B features two servers connected via redundant networks upBeat is running between both servers upBeat enables one IP address 172 17 8 177 on the server where the service is currently active In addition an array o
133. ftware units that make up upSuite Part I upBeat Part II upDisk Part III ubManager In this manual it is assumed that you are familiar with The Solaris operating system TCP IP and networking XML notation C programming Continuous Computing Corporation upSuite User s Guide 1 Welcome to upSuite HA Continuous Computing Corporation upSuite User s Guide Part I upBeat upBeat is a failure detection manager that monitors the activity and state of components over an IP network by sending heartbeats between components upBeat can detect node inter face link or application failures If a failure is detected upBeat initiates failover in less than one second making upBeat well suited to environments in which high availability is required upBeat can be used alone or as part of upSuite HA and is available for Solaris 7 8 9 and 10 on SPARC Ethernet lt Switch Ethernet Figure 1 upBeat network operation Continuous Computing Corporation upSuite User s Guide 3 Continuous Computing Corporation upSuite User s Guide 1 Introduction to upBeat This chapter introduces you to upBeat an architecture for managing heartbeats among clients servers and services as part of upSuite HAM a series of software modules designed to pro vide transparent high availability capability for applications In this chapter What 1s ipBeat it hes reat a aa n
134. g a Service with upBeat e Polling e Monitoring the state of the system e Using Callback Functions e Error handling e IP Addresses and Failover e Detecting Split Brain Conditions e Strategies for Resolving a Split Brain Condition Initialization of upBeat Clients that Provide Services This section describes how upBeat clients that provide services use the upBeat API during ini tialization The first call the application makes to the upBeat API is to ubInit This begins the ini tialization However ubInit will block until the upBeat startup delay period set in the configuration file has expired For more information about setting the startup delay see lt UPBEAT gt tag on page 148 During initialization the client needs to know its own service name or service ID as specified inupsuite conf This information can be hard coded into the program passed as a command line argument or provided using whatever technique you prefer If a service knows only the name of the service it provides it can call ubSvc to find its corresponding service ID To find out its local node ID an upBeat client can call ubNode The client can also use ubNode to find out the node ID of any peer node if it knows the name of the peer node Alternatively the client can call ubSvcPeer with its own service ID to find the node ID where its peer service is running If the service has any lt LINK gt tags in the configuration f
135. he split brain flag from only one side then bring the two systems into contact upDisk will detect the reamining split brain flag on the other side and automatically re flag the fixed side as split brain There are two ways to effectively intervene on both systems to fix a split brain condition e Force one side to be active and let it force the other side to be standby This is the most common technique e Explicitly force one side to be active and the other to be standby Continuous Computing Corporation upSuite User s Guide 97 8 upDisk Administration The rest of this section explores each of these techniques Technique 1 Force active status on one system 1 Determine which system has the most current data by determining which went down first or last Do this by inspecting the relevant files and or system logs and by using the fol lowing command udstat hh 2 Force the desired side to become active udactive Af dataset If the active and standby systems are in contact when you force one side to become active the other side is forced to become standby immediately This is the most common way to recover from a split brain condition If the two systems are not in contact the side you just forced to become active will remember 1t was forced for as long as it is not restarted again for example by a reboot If it comes back into contact with the other side that side is forced to become standby However if the system
136. her nodes upbeat is lt only used as an event manager for Monitors causes failover Lead upBeat lt Ausra Service cause failover script called when operator runs ubmfailover or a monitor causes group to failover User Application status all services and Vv oc directives for a group usr lib ubmgr goactive script called when a group queries requests registers group spawns is to go ACT acknowledgements monitors A as ACT or STBY A kills A Lead and Monitor services with upBeat called when a group is to go STBY gt I I ubMgr Daemon usr lib ubmgr gostandby script anses lt requests failover of termination specified group s status Y kills A lt requests usr sbin Narllog instantiates A State ubmfailover T upsuite Log spawns File kills BS T lt Ietolinita clus syslogd ubmgr ubmstat yslog Watchdog soia get group monitor and user upSuite app info failover a T_ Debug Log reads and group File parses Y validates if XML file F 2 format li
137. i aa Aloe i ai Vasa Sea cee a in ia a i aaa as 205 RPCMONa A a asa ed ais 206 UDI tao a a asas 207 Monitors Overview ubManager can use monitors to learn about the health of the system essentially using upBeat as an event manager to determine whether or not user applications can run or remain running Each CCPU supplied monitor is designed to watch a particular element of the system and report status back to ubManager either directly through the monitor s exit status for RPCmon and 10adMon or indirectly through upBeat for ubP inge r In addition you can write your own monitors to suit your needs a monitor can be script or a binary executable it can be an upBeat service or not If all of the monitors for a group indicate that the resources that they are monitoring are healthy ubManager will allow the group to remain active or to become active However if any one monitor of a group indicates that the resource that it is monitoring is not healthy then ubManager will initiate a failover of the group if it is active provided that its peer group s monitors indicate that all of the resources that they re monitoring are healthy If a group is standby ubManager will not allow the group to become active until all of its monitors indicate a healthy status for their resources ubManager supports two types of monitors smart or dumb Smart A smart monitor is started by ubManager and interfaces directly with upBeat t
138. id unode void arg uint32_t id ub_status_t status boolean_t async printf node d status s s n id status UB_STATUS_UP UP DOWN async ASYNC SYNC di return static void ulink void arg uint32_t id ub_status_t status boolean t async struct in_addr inaddr inaddr s_addr id Continuous Computing Corporation 68 upSuite User s Guide The mt_multi_svc Sample Application printf Mtlink s status s s n inet_ntoa inaddr status UB_STATUS_UP UP DOWN async ASYNC SYNC di return static void uservice void arg uint32_t sid uint32_t id ub_status_t status boolean_t async printf service d node d status s s n sid id status UB_STATUS_UP UP DOWN async ASYNC SYNC di return static void usage char progname fprintf stderr Usage s q uick n progname exit 1 NOTREACHED return The mt_multi_svc Sample Application The mt multi svc application is a multi threaded program managing split brain events and multiple services via sockets Interaction with the application is accomplished via a telnet client The mt_multi_svc application has three primary threads the main thread the upBeat Interface Thread and the Socket Acceptance Thread The upBeat Interface Thread interacts with the upBeat daemon and detects and handles the split brain condition The Socket Accep tance Threa
139. ig can be run manually or it may be run as a result of add_drv or pkgadd in which the package installed contains a kernel module such as a device driver Therefore do not run drvconfig add_drv or pkgadd on your system If you need to install or upgrade software 1 Shut down upSuite on the standby system 2 Install the software on the standby system 3 Restart upSuite on the standby system 4 Perform a failover 5 Repeat steps 1 3 on the formerly active system Inserting or Removing Media Inserting or removing media especially from SCSI devices can lock up the system and cause upBeat heartbeats to fail This results in a split brain condition The Solaris SCSI disk driver Continuous Computing Corporation upSuite User s Guide 13 2 upBeat Administration can block the processor busy wait for a long time in certain cases when CDs CDRWs or MOS are inserted or ejected In a conservative high availability system it is therefore inadvisable to use removeable media If however your system availability is not as critical using very long timeouts can provide some protection against going into a split brain condition when removing media Stopping the system Stopping the system for example using break L1 A or Setup A and then continuing will cause heartbeats to fail resulting in a split brain condition The two most typical reasons for stopping a system are to use the kernel debugger which is not normally nece
140. ile the application can call ubSvcIPPair iteratively to get pairs of IP addresses for the local and peer nodes Typi cally these pairs of addresses are used for the service applications on the local and peer nodes to communicate with each other Continuous Computing Corporation upSuite User s Guide 25 4 upBeat API Programmer s Guide If the service has a TCP or UDP port number configured in upsuite conf the applica tion can call ubSvcPort to get the port number its service uses in its socket binding If the service has a floating or migrating IP address configured in upsuite conf the appli cation can call ubServiceT P to get the address The last upBeat API call made during an upBeat client s initialization activities will most likely be ubGet State This call provides state information to the client Up to three callback functions are called as a result of the call to ubGet State A pointer to a data structure is passed to ubGet State that gives the functions if any to call back If any of the following three callback functions are defined they will be called e The node callback function is called once for each peer node giving the peer node s state e The link callback function is called once for each link giving the link s state e The service callback function is called once for each service giving the node on which the service is active if any For more information see Callback F
141. ing sub second detection and response initiation upSuite HA can decrease failover time to less than a second Specific benefits of upBeat include e Creation of a heartbeat framework is unnecessary e Heartbeats can be added over additional media e Applications can register with the heartbeat manager e Failover scenarios support flexible configurations Continuous Computing Corporation 6 upSuite User s Guide How Does upBeat Work How Does upBeat Work upBeat sends heartbeats similar to ping packets over the networks between components or on private networks between processors to verify that they are active and healthy Heartbeats are used to detect node interface link or application failures in which case upBeat initiates a failover Ethernet 7 Switch Ethernet Figure 1 upBeat network operation upBeat uses two types of notification elements which help to govern failover and communica tion between components 1 Node to node heartbeats are sent back and forth between instances of upBeat running on different processors to verify the status of the processors and to share information between them 2 Client registration and notification requests about the state of the system are made by a resident application through upBeat Continuous Computing Corporation upSuite User s Guide 7 1 Introduction to upBeat Active and Standby Services An instance of upBe
142. ink error ipfs detected a link error before upDisk or upBeat informed ipi e Active error an error has occurred during normal operations on the active syst fs about it em The Continuous Computing Corporation 92 upSuite User s Guide Troubleshooting Console Messages link is down and upDisk will failover to the standby system Typically you will need to fix the problem and then perform a repair with the udrepai r command See upDisk Command Reference on page 127 e Standby error an error has occurred during normal operations on the standby system The link is down and typically you will need to fix the problem and then perform a repair with the udrepair command In addition to the above be aware that a link can be dropped as a side effect of a termination of the link daemon via the system shutting down upDisk being stopped or a direct termination of the daemon Active Standby Errors The active server will fail only if there is an operational or disk error in which case the active server will failover to the standby in addition to those errors the standby server will fail if it runs out of disk space or encounters disk errors or other operational difficulties In short if either system finds that it cannot correctly perform operations it will drop the link and repair will be required Therefore it is important to determine which system is causing the error before trying to fix it Message Meaning
143. ipfstab usr sbin udstat usr sbin udactive etc upsuite upsuite conf Continuous Computing Corporation upSuite User s Guide 131 12 upDisk Command Reference udstat NAME USAGE DESCRIPTION OPTIONS udstat usr sbin udstat i h h h dataset a d dataset m mountpoint This command provides information about the dataset Note Using udstat without any arguments is the same as udstat i The dataset argument limits the status check to the specified dataset only The i flag provides information about the dataset Refer to the Options of the udactive command for explana tion of the a d and m flags as their usage is identical Note the a option will print the dataset and mount point as fol lows lt dataset gt lt mountpoint gt ACTIVE normal UP clean 0 0 Note You will see both a dataset and mount point listed together only if they are not named identically Otherwise you will see only the dataset name 132 Continuous Computing Corporation upSuite User s Guide udstat OPTIONS cont Below are the possible messages you will see for each column of output ACTIVE STANDBY STARTUP Indicates the role of the dataset ACTIVE indicates this system is the active for the specified dataset STANDBY indicates the system is the standby for the speci fied dataset STARTUP indicates the system is starting up Under normal operati
144. irio titi ida aii 159 SAMPLE CONFIGURATIONS ooooconconcconconnonnnonnnnnnnn ccoo aaa saaa nooo nena rene nn nen ron aa aaa 161 CONTISUTORON Acacia O04 Sadi i a oot ened La as is 161 CONFLSULAMO BA ie ena a 166 Configuration CL i aiti i k a i a K a a A a 170 Partition Table rin Ad ds 175 Engineering Guidelines for Upsuite cccccccccsccessessceseesscesceseeseeeseeeeeeaeessecaecsecasenaeeaes 176 Continuous Computing Corporation upSuite User s Guide V Part III ubManager ccccccccncncnnonnnnccncnnancnnnnnnos 177 15 INTRODUCTION TO UBMANAGER ccccccceceeteeeeeeeeeeeeeeeetseveeeseeateeansteatenanens 179 WHAT IS UBMANAGER 179 HOW DOES UBMANAGER WORK ococcccccccccccocccinnocanananonononcnnoncnnnnoconoconononancnnananonononannnnncnnnnos 180 Service GFOUDS iii ii ias 180 Resource MONITOFINS ii A vane sa abv a io d as 182 Application Process Launching and Monitoring ooonoinnnnnnnnnnonnnnnnnnonncrnnonnn aaa 183 Failover SCriptne da e ied ee eed ee os 183 ubManager COMPONENts ceccescceccecceeceerserseessessecseceseeseceseeescesceseeesecseesaecssensecnscnaeeeees 185 16 UBMANAGER COMMAND REFERENCE cccecsceceeeeeseeeeeteeveeaegteeveesustsareesners 187 SHELL COMMANDS a di 187 17 UBMANAGER MONITORS narai a ai a a A A E a a 203 MONITORS OVERVIEW m 203 LOAN cs 205 IRPCMON ee os o da NE 206 UBRINGER ua saaa 207 18 UBMANAGER S
145. isks used by Online Disk Suite would have different partition tables in which only partition 2 most likely would be configured Part Tag Flag Cylinders Size Blocks 0 root wm 0 327 512 50MB 328 0 0 1049600 1 swap wu 328 655 512 50MB 328 0 0 1049600 2 backup wm 0 11198 17 09GB 11199 0 0 35836800 3 var wm 656 983 512 50MB 328 0 0 1049600 4 unassigned wm 985 1640 1 00GB 656 0 0 2099200 5 unassigned wu 0 0 0 0 0 0 6 usr wm 1641 2296 1 00GB 656 0 0 2099200 7 unassigned wm 2297 11198 13 58GB 8902 0 0 28486400 Table 12 Partition table for sample configurations A B and C Continuous Computing Corporation upSuite User s Guide 175 14 upSuite Configuration File upsuite conf Engineering Guidelines for UpSuite This section describes the configuration settings that are required for an improvement in the performance of Upsuite The sync rate of files from active server to standby server depends on the following parame ters 1 Disk Latency 2 TCP IP Network Speed 3 Physical Memory 4 etc ipfstab options 5 CPU speed The memory used by upSuite depends on the number of datasets used the number of files cre ated and their sizes and the options in etc ipfstab The number of files and the sizes of files upSuite can replicate depends on limit imposed by UFS The number of datasets upSuite supports is 262143 Ox3ffff is the limit in Solaris on min
146. ive with standby status The service will get a directive to go active If a service gets a directive to go standby it must assume that it is in standby mode at that point Similarly if a service gets a directive to go active and it can go active it can assume that 1t is active at that point If a service gets a directive to go active but is unable to do so for any reason it must acknowledge the directive with a standby status using ubAckSvc and can assume that it remains standby Typical service registration as active or standby In a typical situation two peer services registering as active upon instantiation As indicated previously the first service to register as active will get a directive to go active and its peer will get a directive to go standby Both services acknowledge the directive by calling ubAckSvc Normally a service directed to go active will acknowledge the directive with an active status and a service directed to go standby will acknowledge the directive with a standby status Alternatively peer services can initially register as standby and then later re register as active depending on events If both of the peer services are in standby mode they will remain so until one of them registers as active In this situation upBeat will not send a directive to go active unless a service requests active status by calling ubRegSvc If a service is active and then registers for standby status by calling ubRegSvc
147. ld catch this signal and when received perform any ter mination activities e g deleting temporary files and exit within 10 seconds If an application has not exited by that time ubManager will forcefully terminate it with a SIGKILL signal Failover Scripting User definable shell scripts are called by ubManager during failover transitions These scripts provide failover scripting and can launch or terminate processes configure net work interfaces or manage any other configuration for the group e goactive After ubManager registers a group with upBeat to be active and has received a directive from upBeat for the group to go active ubManager runs the goact ive script It is Continuous Computing Corporation upSuite User s Guide 183 15 Introduction to ubManager passed the group name as an argument so the same script may be used for all groups If the script exits with a value of zero ubManager starts all of the user applications of the group and acknowledges the directive If the goact ive script does not exist or if it exits with a non zero value then ubManager does not start the user applications and acknowledges the directive with a standby status for the group thus preventing the group from going active gostandby After ubManager registers a group with upBeat to be standby and has received a directive from upBeat for the group to go standby ubManager acknowledges the directive then shuts down the user applications a
148. lem you have e Call us at 858 882 8911 9 00 a m 5 00 p m PST Note If you have a Gold or Platinum service contract follow the contact instructions provided with your contract Continuous Computing Corporation upSuite User s Guide 219 Technical Support Continuous Computing Corporation 220 upSuite User s Guide Index Directories Files etc dfs dfstab 121 etc ipfstab 117 161 etc upsuite 117 146 etc upsuite conf 146 var log nidb 89 var log upsuite 15 Configuration Tags lt LINK gt 156 lt HANES gt 157 lt HEARTBEAT gt 152 lt NETWORK gt 149 lt NODE gt 149 lt NODE_REF gt 155 lt SCSIBEAT gt 157 lt SERVICE gt 154 lt UPBEAT gt 148 lt UpSuiteConfig gt 148 A Active definition 217 API 23 33 function calls 35 overview 23 quick reference 35 Application write operations 85 Applications local 10 multi threaded 24 B benefits 177 179 C Changing mount options 123 Client definition 217 command line 211 Commands 17 ubManager CLI 187 udactive 128 udrepair 131 udstat 132 upbeat admin 17 updisk admin command 137 updisk startup script 139 ups 21 Configuration file 145 changes 102 editing 145 147 example 159 ubmgr conf application definition 212 comments 209 group definition 210 monitor 210 version 209 Configurations sample A 161 sample B 166 sample C 170 sample partition table 175 Configuring services 154 Console messages 89 acti
149. ll as upBeat Configuration A In configuration A two servers are connected via redundant networks upBeat is running between both servers and upDisk is running between the disks upBeat enables the service s migrating IP address 172 17 8 177 on the server where the service is active and disables it on the server where the service is standby Continuous Computing Corporation upSuite User s Guide 161 14 upSuite Configuration File upsuite conf Ethernet Switch Ethernet Switch 172 17 8 141 172 18 8 141 172 17 8 140 172 18 8 140 a grange pein See seem TSS EA 172 17 8 177 both servers I gt 0 S 2 Cc S lt 49 upDisk Figure7 Configuration A block diagram Continuous Computing Corporation 162 upSuite User s Guide Sample Configurations The upsuite conf File Below is the upsuite conf file edited to match the system illustrated in Figure 7 lt xml version 1 0 gt lt UpSuiteConfig VERSION 2 gt lt UPBEAT STARTUPDELAY_SEC 5 gt lt NETWORK NA lt NETWORK NA i ll Network1 gt Network2 gt i ll lt NODE NAME serverl1 NODE_ID 1 gt lt INTERFACE NAME hme0 NETWORK Network1 TP 172 17 8 140 gt lt INTERFACE NAME hmel NE TP 172 18 8 140 gt lt NODE gt T WORK Network2 lt
150. llback function that its peer is active then a dual active condition has been detected The rest of this section examines the sequence of events in detail Consider the following scenario a system has two nodes all of the links to one node go down later a link comes back up 1 For each upBeat client on each node the link callback function is called for each link that has gone down 2 After the final link callback is called the node callback function is called indicating that the peer node is down 3 upBeat sends the standby services a directive to go active 4 Ifa service is active and its standby peer service goes active when directed a split brain condition occurs Each side loses communication with the other and each service can accept clients on its portion of the now isolated network If a link comes up the following occurs Continuous Computing Corporation upSuite User s Guide 31 4 upBeat API Programmer s Guide 1 Each upBeat client s link callback function is called indicating the link has come up 2 If the service is active and is implemented in such a way that it will always accept the active role when directed to the service can assume a split brain condition 3 After the link callback functions are called the node callback functions are called indicat ing that the peer node has come back up 4 All of the service callback functions are called indicating which services on the peer no
151. llows the same callbacks to be used by both ubAsync and ubGetState A pointer to the node callback function For more information see Call back Functions on page 34 A pointer to the link callback function For more information see Callback Functions on page 34 A pointer to the service callback function For more information see Call back Functions on page 34 directive A pointer to the directive callback function For more information see RETURN VALUES Callback Functions on page 34 ubAsync returns 1 if the connection to the local upBeat daemon is lost and cannot be rees tablished Returns O otherwise In most high availability applications failure should be treated as a catastrophic failure and the application should shut down ERRORS See Error codes on page 55 SEE ALSO ubGetState on page 41 ubSetupPollfd on page 48 Callback Functions on page 34 Continuous Computing Corporation upSuite User s Guide 39 5 upBeat API Reference ubFini NAME ubFini SYNOPSIS Shut down a connection to upBeat and free any resources allocated by ubInit void ubFini upbeat_t upbeatp A handle as returned by ubInit DESCRIPTION ubInit allocates resources and establishes a connection to the local upBeat daemon ubF ini shuts down the connection and frees the resources RETURN VALUES None ERRORS None SEE ALSO
152. monitor which monitors CPU load bin sh load mon is invoked with a 15 minute average load limit and exits with 0 success if the current 15 minute average load is below the passed in limit or 1 error if the current 15 minute average load is above the limit Continuous Computing Corporation 182 upSuite User s Guide How Does ubManager Work if ne 1 then echo usage 0 lt limit eg 0 80 gt exit 1 Ed Extract the load average over the past 15 minutes from uptime Multiply by 100 to make the value an integer UPTIME uptime awk e print int substr 10 1 100 3 Multiply the user s passed in load limit by 100 to allow us to compare it with the current load LIMIT echo 1 awk e print int 1 100 echo ut SUPTIME ul LIMIT if SUPTIME lt SLIMIT then The load is below the limit return success 0 exit 0 else The load is above the limit return error 1 exit 1 fi Application Process Launching and Monitoring A user application process is started when the group it belongs to becomes active and grace fully terminated when the group becomes standby In addition if an application process termi nates while the group is active a failover will be triggered User application processes can be scripts or compiled programs ubManager terminates user applications by sending them a SIGTERM signal In order to shut down gracefully user applications shou
153. n upSuite User s Guide The wrapper Sample Application Note this application does not use these addresses this code is include for the purposes of illustration 7 sip gt addr 0 while ubServicelP ubp glob sid sip gt addr sip printf service ip s s n sip gt interface sip gt ip Get pairs of IP addresses the active and standby can use to communicate Note this application does not use these addresses this code is include for the purposes of illustration If an application did use these addresses it would also want to register a link callback ippp gt local 0 while ubSvcIPPair ubp glob id glob sid ippp gt local ippp struct sockaddr in sockaddr in sockaddr in sin addr s addr ippp gt local printf local s inet ntoa sockaddr in sin addr sockaddr in sin addr s addr ippp gt remote printf remote s n inet ntoa sockaddr in sin addr Give libupbeat a chance to do housekeeping e if ubGetState ubp opsp 1 fprintf stderr ubGetState failed errno d s n errno strerror errno exit 1 Register to provide our service We register as willing to be active but there is no guarantee we will be told to be active Ey Continuous Computing Corporation upSuite User s Guide 61 6 upBeat Sample Applications if ubRegSvc ubp glob_sid UB_ACTIVE 1 fprintf stderr ubRegSvc failed err
154. n page 55 SEE ALSO ubRegSvc on page 46 Continuous Computing Corporation 36 upSuite User s Guide ubAsync ubAsync NAME ubAsync SYNOPSIS Receive asynchronous status information from upBeat int ubAsync upbeat_t upbeatp const ub_ops_t opsp upbeatp A handle as returned by ubInit opsp Pointer to a ub_ops_t structure DESCRIPTION The application is required to call ubAsync periodically ubAsync can run in its own thread or it can be called from an application thread If 1 ibupbeat s connection to the local upBeat daemon is ever lost which is very unlikely ubAsync will attempt to reestablish the connection The opsp argument points to a ub_ops_t structure which gives upBeat the client s node link service and directive callback functions if any The structure also contains the argument passed to the callback functions The ub ops t structure is defined as follows typedef struct ub_ops void arg void node void arg ub node id t node id ub status t node status boolean t async void link void arg ub link id t link id ub status t link status boolean t async void service void arg ub svc id t service id ub node id t node id ub status t service status boolean t async void directive void arg ub svc id t service id ub svc status t svc status ub ops t Continuous Computing Corporation upSuite User s Guide 37 5 upBeat API Refere
155. nce The arguments are as follows arg node_id link_id service_id Any pointer Can be NULL Passed as the first argument to each of the call back functions Corresponds to the NODE_ID attribute of the lt NODE gt tag in the upSuite configuration file upsuite conf An IP address Can be cast to an in addr t Corresponds to the SERVICE_ID attribute of the lt SERVICE gt tag in the upSuite configuration file upsuite conf node_statusOne of the following values UB_STATUS_UP or UB_STATUS_DOWN Tf the callback function was called by ubAsync UB_STATUS_UP indicates that a peer node s upBeat has started sharing heartbeats with the local node and UB_STATUS_DOWN indicates that such sharing has stopped If the callback function was called by ubGet State UB_STATUS_UP indicates that a peer node s upBeat is sharing heartbeats with the local node and UB_STATUS_DOWN indicates that no such shar ing is underway link_statusOne of the following values UB_STATUS_UP or UB_STATUS_DOWN If the callback function was called by ubAsync UB_STATUS_UP indicates that a link between a peer node and the local node has started shar ing heartbeats and UB_STATUS_DOWN indicates that such sharing has stopped If the callback function was called by ubGet State UB_STATUS_UP indicates that a link between a peer node and the local node is sharing heartbeats and UB_STATUS_DOWN indicates that
156. nce of ipfs on a second machine This instance of ipfs passes the write down through the standby processor s local file system to its disk array This implementation has several advantages As a file system i pfs is transparent upwards and downwards The application makes the same system calls which are intercepted at the VFS layer and propagated over the network link Below ipfs the local file system and disk drivers are not affected so any advantages in redundancy or performance inherent in the local file system or disk subsystem are preserved Continuous Computing Corporation 84 upSuite User s Guide How Does upDisk Work write operations offer an interesting example of updisk ipfs settings Below is the typical sequence of an application write 1 An application calls a write 2 Thewrite system call is vectored to ipfs r 3 ipfs does the local FS write 4 ipfs sends the operation over the network to the standby ipfs after the local FS oper ation completes 5 TCP acknowledges the transmission of the operation and ipfs sends a receive acknowl edgement 6 The standby ipfs does the standby local FS write 7 The standby ipfs sends an acknowledgement over the network to the active ipfs after the standby local FS operation completes In non upDisk systems file system permissions and open flags control write access to a file i pfs however will allow a file to be open for writing on the stan
157. nd runs the gostandby script It is passed the group name as an argument so the same script may be used for all groups The standby directive cannot be denied like the active directive can so the exit value of the gostandby script is ignored r cause failover This script is called when a group with a lead service is to failover because ubmfailover has been run by the operator a monitor within the group indicates an unhealthy resource or an application within the group unexpectedly exited It is passed the group name as an argument and is expected to run whatever commands are needed to initiate a failover of the lead service 184 Continuous Computing Corporation upSuite User s Guide How Does ubManager Work ubManager Components Figure 2 on the next page illustrates ubManager s components Continuous Computing Corporation upSuite User s Guide 185 15 Introduction to ubManager Monitors are spawned by ubMgr A Monitor may optionally be a non upBeat process that ubMgr spawns periodically or as needed and which ubMgr registers the Monitor with upBeat on behalf of the Monitor The Monitor s exit value in this case informs ubMgr of the status of whatever the Monitor was monitoring Unique name and ID must be in etc upsuite upsuite conf for group s Each Monitor on each node must have a unique service name and service ID in upsuite conf Monitor services do not have peers on ot
158. ng for HA NFS When configuring for HA NFS this subtag is required CAUTION You must use one and only one lt SERVICE_IP gt subtag when configuring an HA NFS service If any lt SERVICE gt tag contains an lt HANFS gt subtag and also contains more than one lt SERVICE_IP gt subtag or does not contain a lt SERVICE_IP gt subtag an error will occur and upDisk will stop running IP attribute Optional The IP address of the given service If you use the HOST attribute you can not use the IP attribute Specify a different IP address for each service The value of IP is passed to inet addr Example IPS T9OZ D681 HOST attribute Optional The host name Can be used instead of IP but you can not use both Using HOST may be convenient for testing and evaluation purposes but we recommend that HA deploy ments rely on the IP attribute to reduce the reliance on unresponsive name services Example HOST cpul IF attribute Required The network interface on which the given service can be accessed Specify a differ ent interface in each lt SERVICE_IP gt tag The syntax must be as follows hme0 x x can only be a number no white spaces allowed Continuous Computing Corporation 158 upSuite User s Guide Configuration File Example Example IF hme0 17 Configuration File Example Below is an example of the upSuite HA configuration file in its entirety for your reference lt xml version 1 0 gt
159. nism enables you to configure the number of outstanding operations on the standby The defaults are 75 for the low water mark and 100 for the high water mark e g throttle 75 100 Set this option using throttle low high The values must be set in integer decimal values with high being greater than low Larger values may increase performance by allowing more operations to be queued on the standby the tradeoff being a longer drain time during Continuous Computing Corporation 124 upSuite User s Guide ipfs Mount Options failover To disable the throttle use throttle 0 0 Synchronous Asynchronous Option sync operations are forced synchronous sync operations are forced non synchronous You can use sync and sync to override application control of synchronous operations to the underlying file system The absence of either of these options leaves the application in con trol In particular NFS servers will still be synchronous sync forces all directory and IO operations to be synchronous sync forces all directory and IO operations to be cached non synchronous and can be used for database and NFS server applications to greatly increase performance Since there is an up to date replicated copy of the data on another system no loss of data will occur in the event of a system or component failure Standby Modification Time Option mtime Controls the setting of modification time on the standby The default behavior
160. no d s n errno strerror errno exit 1 while 1 struct pollfd pollfd pfdp amp pollfd int poll_timeout_msec 500 Fill in the pollfd fresh each time upBeat will reestablish a connection if it is dropped so the fd may change A ubSetupPollfd ubp pfdp We are only polling for upBeat but we could be polling for any number of file descriptors if poll pfdp 1 poll timeout msec 1 fprintf stderr poll failed 3d s n errno strerror errno exit 1 We could check the return value from poll and the contents of pollfd to see if we have any events but ubAsync checks again anyway plus it gets a chance to do some housekeeping Ef if ubAsync ubp opsp 1 fprintf stderr ubAsync failed errno d s n errno strerror errno exit 1 NOTREACHED exit 0 This application blocks when we call system glob prog until glob prog completes which means we will not call ubAsync or anything else for the duration Continuous Computing Corporation 62 upSuite User s Guide The wrapper Sample Application A multithreaded application or an application that forked execed and managed the child process could continue to poll and could respond to UB_STANDBY message by killing the child process el static void directive void arg ub_svc_id_t service ub_svc_status_t status static int rereg FALSE upbeat_t ubp
161. nononocanananaananononenons 107 Two HA NES ArchiteCtures ccccccccccccccccccsssssssssscescsccccccccccscsssescscesceccesscassscssesscsseceseseees 109 FILE HANDLE PERSISTENCE DURING FAILOVER occococononononononnnnononnnnncononococnonanocanananananancnnanon 109 BUILDING AN HA NES SERVER occccccoccnccccccccnonoconananananononnnnonononcnnncnnncocnnnonannnananononononannnnncnonoos 110 Edit the configuration file 111 Share the dataset in etc ipfstab 112 Configure clients to run upBeat optional 113 NES VER UDP VS TCP iii ai 114 SHARING SUBDIRECTORIES VS FILE SYSTEMS ccscsscccessececsscecesneceseaeceseeeessseeceeaeeeseeeens 115 11 THETPRESTAB EE 117 IPFSTAB SETTINGS sic dde iii 117 IPRS MOUNT OPTIONS ce aa ata a aes aed idea aa asis 121 Options for Returning to the Application ooononinnonnnnnnnonnocnnanonncnnnonnc ran aaa aaa 121 Maximum Operations Option ooooconicnoccnnnnnnnnnnnannnnnnnnonncnn conan ono a aa aaa as 123 Memory Allocation Option ooonccicnnonconnnnncnncononnnonnnnanonnn aaa aaa aaa aaa aaa as 123 Standby Throttle OpllON cin e tds 124 Synchronous Asynchronous Option sccecsccseeecssecencecsseeesecseccenseeesceceeeeeseeeaceeeeeeeeeeaees 125 Standby Modification Time Option sccccceccesccescesscesseseesceseeesecseeesecsensecaecnseenscneeeaees 125 12 UPDISK COMMAND REFERENCE cccececeeeete cent eee ee tenet neta tans neat aaa 127 UDACTIVE cuota id dile a i ia iii diles 128 UDREPATR io 13
162. nts WELCOME TO UPSUITE HA ororen arron iinr aa Raa EAEE 1 ABOUT THIS MANUAL rererere ieoi did ibi 1 o SALA a asas i S 3 1 INTRODUCTION TO UPBEAT Ldakak aaa aaa aa saaa aaa aa aa aaa a aaa aa aa aa aa aaa aa aa aaa 5 WHATS UPBEAT A asi siai ia Ius e ss se dec sai nd aaa Sios aaa ai 6 WHY SHOULD LUSE UPBEAT ino aE eae eaea ee E nE EE a is 6 HOW DOES UPBEAT WORK dei 7 Active and StandbyS rvices Li aaa adis A vu i sus su k k d i i da i si a a 8 ILAA AES RAAR EAER EEE E EAA S ass saaa aa aka aaa as reno aa kaka aaa saaa 9 OPERATION OVERVIEW sis ns ae 10 UBMANAGER a o cal Le eco S Cs Pi A 10 SCSIBEAT L asi a ia ad SS aa aa a i Ina asa Na a aa a a es 11 2 UPBEAT ADMINISTRATION iais aa aa da 13 GUIDELINES TO ENSURE HIGH AVAILABILITY aaa aa aaa aa aaa aaa asas aaa aa aaa aaa aaa aaa 13 UR ESA A OR e o beat as o sa sea i 13 Inserting or Removing Media oooonocinocinncnnnncnnnnnonnnnnncnnnnnonnn cnn aaa aaa conocen aaa aaa 13 Stopping THEY MEA A A Ge vos Ei S Siu Wace antes A ia e 14 NETWORK GUIDELINES 45 44 asai a 14 Routine saika sekas ce sie ien dos Ia ae eth ig tose abe o Ba e ai e odis andes iai as beatae o 14 MAC Addir SS Sis sasas kasa a so a Va oia lla ANS 15 MONITORING UPBEAT di 15 IMLAINTENANGE reg oo as Sessa ad 15 TROUBLESHOOTING R 16 TANKS DOWN decrees ee iros 16 BUSY DISKS i aaa 16 Operation Problems di i 16 3 UPBEATCOMMANDS 00 17 UPBEAT ADMIN COMMAND cssc
163. o the network via redundant links and are running upBeat 4 3 172 18 8 151 172 17 8 151 172 18 8 150 172 17 8 150 Switch Ethernet 172 17 8 141 172 18 8 140 172 17 8 140 172 18 8 141 172 17 8 177 active server Active Server 1 j upDisk Figure9 Configuration C block diagram Continuous Computing Corporation 170 upSuite User s Guide Sample Configurations The upsuite conf File Below is the upsuite conf file edited to match the system illustrated in Figure 9 Note If your clients are separated from your servers by routers e g LANs MANs or WANS you will need to enable the routing feature in the configuration file The default is to not route which is how the sample file below is configured notice that ROUTE does not appear beneath the lt HEARTBEAT gt and lt SERVICE gt tags For information about how to enable routing see lt HEARTBEAT gt tag on page 152 and lt SERVICE gt tag on page 154 lt xml version 1 0 25 lt UpSuiteConfig VERSION 2 gt lt UPBEAT STA lt NETWORK NA lt NETWORK NA RTUPDELAY_SEC 5 gt E NetworkA gt E NetworkB gt lt NODE NAME serverl NODE_ID 1 gt lt INTERFACE NAME hme0 NETWORK NetworkA IP 172 17 8 140 gt lt INTERFACE NAME hmel NETWORK NetworkB IP 172 18 8 140 gt lt NODE gt lt NODE NAME serve
164. o update its state ubManager tracks the monitor s active standby state through ubManager s upBeat service callback All smart monitors are considered permanent if they should ter minate ubManager fails over its associated group if it is active and then restarts the monitor Continuous Computing Corporation upSuite User s Guide 203 17 ubManager Monitors Dumb A dumb monitor relies on ubManager to interface with upBeat on the monitor s behalf A dumb monitor can be permanent or periodic Permanent These monitors start once and remain running If a permanent monitor process exits while its corresponding group is active ubManager will failover the group and restart the moni tor If a permanent monitor process exits while its group is standby ubManager will restart the monitor and refuse any request to become active while the monitor is dead Periodic These monitors run periodically returning a status code each time they exit If a periodic monitor exits with a non zero value ubManager considers the node unhealthy for the monitor s associated group and if a periodic monitor exits with a value of zero ubMan ager considers the resource the monitor was monitoring as healthy Provided with the ubManager package are three monitors loadMon Monitors a CPU s load average RPCmon Monitors RPC server programs such as NFSd ubPinger Monitors third party IP addresses 204 Continuous Computing C
165. of upDisk is not to use the mt ime option Without this option files on the standby may have a later modifica tion time than those on the active due to replication delay However each file on the standby will have the correct modification time relative to other files on the standby With this option turned on the modification time on the standby is made exactly the same as on the active This requires upDisk to perform an extra attribute operation which can affect performance Continuous Computing Corporation upSuite User s Guide 125 11 The ipfstab File Continuous Computing Corporation 126 upSuite User s Guide 12 upDisk Command Reference This chapter explains upDisk s commands The commands are organized alphabetically and include usage a description options relevant files and other related commands You must be logged in as superuser root to use these commands In all commands dataset and mountpoint are as you have defined them in the ipfstab file In this chapter UdaCHVes Aina teeth e et chs oes Bie A A Be a cane Meh ee 128 Ure pais soe sess eee eee aa S ei ana as i 131 UWStat visi ina as i a a a At 132 updisk admin COMMANA eeecesseceeeeceecessecescecsecesceceeeeseceeceesseceseeceeeesueceseeceeecsaeeeaeeeeees 137 Updisk Startup Script iii sia e i a ai o In Ua 139 Continuous Computing Corporation upSuite User s Guide 127 12 upDisk Command Reference udactive NAME udactive
166. ollowing lt SERVICE NAME myService ipfs SERVICE_ID 17 TYPE BASIC STARTUPDELAY_SEC 5 PORT 1776 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK Network1 ROUTE ROUTE gt lt LINK NETWORK Network2 ROUTE ROUTE gt lt SCSIBEAT gt lt DISK PARTITION system gt lt DISK PARTITION one gt lt SCSIBEAT gt lt HANES gt lt SERVICE_IP IP 192 168 1 1 IF hme0 17 gt lt SERVICE gt upBeat optionally associates IP addresses with general services These IP addresses migrate to the active server and should be different from the fixed IP addresses that the operating system configures upon booting In the example above the active server would configure the IP address 192 168 1 1 on hme0 17 The standby system deconfigures hme0 17 if neces sary In the event of a failover this process occurs in reverse Continuous Computing Corporation 154 upSuite User s Guide lt SERVICE gt tag NAME attribute Required A name for the given service must be different from the NAME attribute setting in any of the other lt SERVICE gt tags The NAME attribute should be used as a convenient file handle for other configuration files like ip fstab or for system messages Example NAME myService ipfs SERVICE_ D attribute Requir
167. on dev rdsk cO0t0d0s0 is not responding If investigation reveals no problems with your disk or SCSI bus you may have your SCSIbeat settings too low in upsuite conf Increase the values of the TIMEOUT_MSEC and FREQ MSEC attributes beneath the lt PARTITION gt subtag of the lt NODE gt tag See lt NODE gt tag on page 149 for more detailed information about these settings Operation Problems If you find error messages in var log upsuite ensure that your system configuration as you have defined it in the configuration file upsuite conf matches your actual con figuration You can do this by using the ifconfig a command at the Solaris prompt Match the ifconfig a output information against that in your configuration file 16 Continuous Computing Corporation upSuite User s Guide 3 upBeat Commands This chapter describes upBeat s commands and their uses The commands are organized alpha betically and include usage a description options relevant files and other related commands You must be logged in as superuser root to use these commands In this chapter upbeat admin COMMANA eeeeeeeceseceececseeeesceeeeeessecescecsseecsaeseneeeseeeeaeceeecueseneeceneecseeseeeens 17 upbeat startup shutdown SCT Pt oooconccnoccnocconconoconononononononoconnonncon non E aaa aaa 19 Uphicense COMMANd AAA i i Sou i as Uni o ia i asa Ei ai a 20 UPS COMMAnd AAA i E ERE a as a aso an aa i A e a as 21 upbeat
168. ons this is a transitory state Ifa STARTUP message persists it usually means the daemons are not running this will be accompanied by the nodaemon output described later in this section normal summary replay repair Indicates the state of the system normal indicates the system is running normally repair indicates that a repair is currently taking place replay indicates that the active system is currently send ing material to update the recently failed standby system summary indicates that the active is not currently replicat ing to the standby The active is creating a log of all operations that will have to be performed on the standby when it is avail able again This message typically indicates the standby is offline Continuous Computing Corporation upSuite User s Guide 133 12 upDisk Command Reference OPTIONS cont UP DOWN Indicates whether the link between the active and the standby ipfs is up or down clean dirty busy clean indicates that each system is in sync with the other dirty indicates that one side or the other or both believe a repair is necessary busy indicates the system is working but may be behind the other by a few transactions 0 0 These numbers indicate the number of deferred operations and the number of replay operations respectively If there are deferred or replay operations outstanding both of these num bers will be displayed If there are no outstanding operations
169. ontinuous Computing Corporation upSuite User s Guide The mt_multi_svc Sample Application The following illustration shows the deployment of mt_multi_svc The Sample Application mt multi svc registers to upBeat EA to provide two services MtApp tl at port 1800 and IP addresses 192 168 1 1 192 168 1 2 and 192 168 1 3 and MtSysCmd at port 1801 and IP addresses 192 168 1 4 and 192 168 1 5 CP1500 Node 2 Heartbeats over each link TCPAP upBeat info local amp Ethernet 2 links upBeat info local amp remote node link y wW remote node link and service status and service status Y Application activity Ampl Amp mt_mutti_swc log mt_multi_swc log fik fik Client s using Client s usin Mt amp ppctl service A e 2 MtSysCmd service telnet client for telnet client for L MtAppCtl MtSysCmd L service service A Client Machine s User running controlling and monitoring application Figure3 mt_multi_svc system architecture Continuous Computing Corporation upSuite User s Guide 73 6 upBeat Sample Applications Application Architecture The following illustration is a component diagram illustrating mt_multi_svc s architec ture Notice that mt multi svc first becomes a daemon process It is composed of multiple threads specifically e The Main Thread This thread is responsible for spawning the upBeat Interface Thread and the Socket Acceptance Thread The main thre
170. or number for device drivers The below specified options can be used to improve the performance of upSuite ipfstab File The following options can be set in etc ipfstab file for an improvement in the performance of upSuite on Solaris U3 and US based on the number of data sets used and the size of replicated data 1 maxmem 2 throttle 3 maxops For example a system which has 20 datasets and has more than 150GB of data for replication can have the following entry in the etc ipfstab file with maxmem and throttle set data ipfs ipfs localfs localfs localfs localfs share set mount mount mount mount name point opts point type device opts opts ipfs ipfs maxmem 3g throttle 95 100ipfs ufs cO0t0d0s7 rw noatime loggingrw Continuous Computing Corporation 176 upSuite User s Guide Engineering Guidelines for UpSuite TCPAP Configuration The TCP maximum buffer size the congestion window size the transmit buffer size and the the receive buffer size can be set for improving the perfomance of upSuite on Solaris 10 U3 and US as per Solaris TCP IP tuning guidelines For example for a 1Gbps link the following commands can be used to set the above specified parameters ndd set dev tcp tcp_max_buf 16777216 ndd set dev tcp tcp_cwnd_max 8388608 ndd set dev tcp tep xmit hiwat 1048576 ndd set dev tcp tep recv hiwat 1048576 Continuous Computing Corporation upSuite User s Guide 177 14 upSuite Configuration
171. ork to a standby file system as illus trated in Figure 1 upDisk replicates file system operations so that those occurring on the active happen simultaneously on the standby Only operations that change the file system are sent across the network For example reads happen on the active only but creates and writes are sent to the standby as well Note that while the file systems are referred to as Active and Standby activity is constantly occurring on both machines Ethernet Switch Ethernet Switch SCSI Fibrechannel RAID array Figure 1 Example upDisk configuration As you can see in the example above the active processor writes information to its local disk while that information is simultaneously written to the disk of a standby processor via the net work The specific disk media be it standard SCSI Fibre channel or RAID arrays is irrele Continuous Computing Corporation 82 upSuite User s Guide How Does upDisk Work vant to upDisk Note that the standby file systems are read only for processes on the standby side Because upDisk replicates file systems the replicated file systems may be physically located on two separate servers Therefore server A may be active on file system 1 and standby on file system 2 server B may be standby on file system 1 and active on file system 2 This con figuration is illustrated in Figure 2 Ethernet Switch Ethe
172. orporation 198 upSuite User s Guide Shell Commands OPTIONS cont For monitor information Ms Mp State of self Ms and peer Mp moni tor 1 active 2 non existent PSt Process state of monitor DEAD RUN or BROKEN For a dumb periodic monitor it is possible that the monitor state is active but that the process itself is DEAD until ubManager restarts it PID Last process ID of monitor Exit Last exit value of monitor Num Times Forked The number of times that the monitor has been instantiated Num Times Died The number of times that the monitor has unexpectedly terminated Num Times AWOL The number of times that the monitor has been sent a termination signal with out an indication that the monitor has actually terminated Last Start The date and time that the monitor was last instantiated Continuous Computing Corporation upSuite User s Guide 199 16 ubManager Command Reference 2007 c007 oz OT cT c0 PI PT v N V N og das uon oe das uon V N V N oe das uonN og das uon OOO Oa SS Qo Oo Oo 2 0 Oo GO a at OS tot G E Qi 11 11 E e zuzuz6 Tuzuzb ZuTuz TuTuzb ZUZuT TuZuT ZUTUuUT TuTuT 21839 JSPT pextod SaWTI wWunN uus I0 TUON 323suqn 9 duwbxq Continuous Computing Corporation upSuite User s Guide 200 Shell Commands OPTIONS cont For application information PSt Proce
173. orporation upSuite User s Guide loadMon loadMon loadMon enables the monitoring of the CPU s load average loadMon is a dumb periodic mon itor implemented as a script NAME load mon USAGE usr lib ubmgr load mon limit DESCRIPTION This utility is designed to monitor the current CPU s load aver age EXIT VALUES Returns O if the usage is below the limit success and 1 if it is equal to or above failure ARGUMENTS limit is the CPU load average with which to compare the actual CPU load It is expressed as a percentage in decimal form Example load mon 0 80 Continuous Computing Corporation upSuite User s Guide 205 17 ubManager Monitors RPCmon This utility monitors the status of RPC remote procedure call server programs such as the network file system daemon NFSd RPCmon is a dumb permanent monitor implemented as a script NAME rpcmon USAGE usr lib ubmgr rpcmon type IP address program version interval DESCRIPTION This utility is designed to monitor an RPC process at a specified interval EXIT VALUES Exits with a value of 1 if the RPC process does not respond otherwise the script does not normally exit ARGUMENTS type is the protocol type and must be either udp or tep IP_addressis the IP address of the RPC server programis the name of the RPC program you want to moni tor version is the version number of program interval is the number of seconds between monitor calls to the RPC serve
174. out of disk space you must remove files on your active system to free space and then perform a repair even if 1t is the standby disk that has run out of space Note that if it is the standby system s disk that runs out of space the active system will continue to provide full service but there will be no replication of files on the standby from the active If the active system s disk is out of space you can still read delete and overwrite data but you cannot create files or directories If one of your systems runs out of disk space a message similar to those below will appear on your console Sep 28 17 16 37 port ipfs ID 941318 kern notice i The standby is out of space and must be fixed Sep 30 03 16 03 left unix WARNING ipfs File system full Sep 27 10 59 45 left unix NOTICE alloc ipfs file system full Again if you get one of the messages above free space on your active system and perform a repair Under some circumstances which depend on the order in which files were created and deleted you will not be able to complete the repair If you are unable to complete the repair 1 Take your standby system offline 2 Remove files from your standby system to free space on its disk 3 Perform the repair again pfs Unmount Unsuccessful ipfs will not unmount under certain circumstances Therefore you will not be able to suc cessfully run etc init d updisk
175. pSuite HA configuration file upsuite conf The configura tion file which is in XML format contains settings that govern various aspects of the behavior of upSuite You must edit upsuite conf to reflect the settings of your system To help you through this process this chapter includes descriptions of each setting in the configuration file In addition three sample configurations are included to be used as guides for creating your own upSuite architectures In this chapter The Cont P Ur atl OM Piles aeee esaeo iea atra aii alitas 145 Editing UpSuite COM aer EE 147 lt UpSuiteConfis gt taB iii asi a e iii si ai i a Ia lesiona dida 148 lt UPBEAT gt TAB RAI 148 SNETWORK gt gt TAS td 149 NODES Tae Leik niai bei na I ii id 149 lt HEARTBEATS tag ec apa ii i pr 152 lt SERVICE gt AB iseis iia ias 154 Configuration File Examples nitre sceessevhanng sbschestey rean ir E TE 159 Sample Comfi guration scesero iee si e eoa sa a EEs E e A ai as a sa E si 161 The Configuration File The upSuite HA configuration file upsuite conf is an XML file that contains settings you can modify to customize the behavior of upSuite software Several example configuration files are provided with the upSuite software in the etc upsuite examples direc tory e upbeat xml contains example settings relevant to upBeat updisk xml contains example settings relevant to upDisk e upsuite xml contains example settings for all possi
176. poration upSuite User s Guide 189 16 ubManager Command Reference NAME ubmgr init d USAGE etc init d ubmgr start stop restart DESCRIPTION ubManager is normally started by an rc script etc rc3 d S99ubmgr during system startup and ter minated by another rc script etc rc2 d K99ubmgr during system shutdown The etc init d ubmgr command allows you to stop start or restart ubManager manually OPTIONS start Starts ubManager stop Stops ubManager restart Restarts ubManager FILES ubmgr conf SEE ALSO ubmgr admin command Continuous Computing Corporation 190 upSuite User s Guide Shell Commands NAME USAGE DESCRIPTION ubmgr admin command usr sbin ubmgr c control port d config file p client port s script dir S v This command runs ubManager To run ubManager with non default options the init script may be edited or ubManager may be started manually from the shell prompt Continuous Computing Corporation upSuite User s Guide 191 16 ubManager Command Reference OPTIONS c control_port ubManager uses a TCP port to determine whether or not it is the only ubManager instance running This option allows this control port to be changed The default is 2006 This option runs ubManager in the foreground not as a daemon All output is sent to stdout instead of syslog Useful for debugging config file This option causes ubManager to read
177. puting Corporation 106 upSuite User s Guide 10 High Availability NFS HA NFS This chapter explains how to use upSuite HA to build a high availability NFS server from a pair of servers In this chapter How Can upDisk Aid My NES Server oi aaa aa kaikas aaa aaa aaa kaka aaa 107 File Handle Persistence During Failover oooonconnnioccnonconccnnconnonnnanonn ccoo aaa aaa aaa 109 Building an HA NES Server oseiro peues a is ko ina is a i sn dene sa 110 NES over UDP ys TCP Lasse ai 114 Sharing Subdirectories vs File Systems aaa aaa aaa aaa kasa aa aaa aaa 115 How Can upDisk Aid My NFS Server Yet another capability of the ipfs layer in upDisk is demonstrated in the case of a redundant NES server running upDisk The architecture for such a system is shown in Figure 4 where a machine running as an NES client is connected via a network link s to a set of redundant pro cessors and disks one of which is an active NES server in active operation and both of which are running upDisk and upBeat Continuous Computing Corporation upSuite User s Guide 107 10 High Availability NFS HA NFS 10143 10 123 Ethernet Switch a Ethernet lt 1 Switch L 192 168 1 1 10442 101221 10111 10121 nes Sve NFS svr i i 192 168 1 77 5 E i 2 o a lt File handles maintained from active processor in case of failover
178. r Example rpcmon udp localhost nfs 3 5 Continuous Computing Corporation 206 upSuite User s Guide ubPinger ubPinger This utility monitors third party IP addresses ubPinger is a smart monitor designed to monitor network connectivity and to cause a failover if an interface has failed ubPinger uses third party addresses to determine whether a server has good connectivity to the network These third party addresses are machines out on the network that the server can use to determine if a path exists from the server to the network Careful choice of the third party addresses is crucial If an address is chosen and then goes off the network it may trigger an improper failover ubPinger allows you to specify a number of addresses to check to reduce the chance of improper failover If any one of the addresses can be reached the interface is presumed functional NAME ubpinger USAGE usr lib ubmgr ubpinger t ping time service name hostl host2 DESCRIPTION This utility is designed to monitor network connectivity and cause a failover if an interface has failed OPTIONS ping timeis the time between pings in milliseconds The default value is 500 ms ARGUMENTS service nameis the name of the service that ubPinger becomes to report success or failure This is defined under a lt SERVICE gt tag of the upSuite configuration file upsuite conf host 1 is the IP address or host specified with the hostname
179. r Definition USAGE DESCRIPTION m group name node_id monitor_service cmdline options This entry defines a monitor process A monitor is associated with a particular group as specified by group_name and with a particular node as specified by node_id Monitors require a unique service definition in upsuite conf for each node This is so that ubManager can concurrently keep track of the groups monitors on both nodes The unique service name for the node is supplied in the node_id and monitor service parameters A cmdline is also required so ubManager can start and stop the monitor process at the appropriate times The available options include interval n This would cause a restart of a dumb monitor every n sec onds This defines the monitor as a dump periodic monitor smart This would cause ubManager to monitor the monitor s state through upBeat If there is no interval option and no smart option then the monitor is considered to be a dumb permanent monitor Note The command line arguments should not be separated by semicolons only white spaces Example m group1 1 serverlm2 monitors rpcmon udp localhost nfs 3 5 Continuous Computing Corporation upSuite User s Guide 211 18 ubManager Semicolon Delimited Configuration File Application Definition The a parameter defines an application associated with a group USAGE a group_name appl_name cmdline DESCRIPTION This entry defines a
180. r to the standby system it associates the file handle 5678 with the foo bar file With upSuite HA foo bar is associated with the file handle 1234 Upon failover to the standby system the file continues to be associated with the same file handle Therefore when the standby server awakens upon the active s failure the client sees absolutely no difference between it and the active server that was previously providing service Without file handle persistence the failover process would involve closing all files which likely means shutting down applications unmounting the file systems which may require rebooting remounting the file systems and reopening all files Depending on the complexity of the situation this process could take anywhere from thirty seconds to five minutes and can require a great deal of effort With upSuite HA NFS this procedure is avoided Note Refer to Normal Failover on page 101 for information regarding file locking for NFS clients Continuous Computing Corporation upSuite User s Guide 109 10 High Availability NFS HA NFS Active upDisk upDisk Standby Standby File foo bar associated File foo bar associated Filehandle 1234 File foo bar associated with filehandle 5678 with filehandle 1234 maintained for foo bar with filehandle 1234 File handles with upSuite HA
181. r2 NODE ID 2 5 gt lt INTERFACE NAME hme0 NETWORK NetworkA IP 172 17 8 141 gt lt INTERFACE NAME hmel NETWORK NetworkB IP 172 18 8 141 gt lt NODE gt lt NODE NAME client1 NODE_ID 3 gt lt INTERFACE NAME hme0 NETWORK NetworkA IP 172 17 8 150 gt lt INTERFACE NAME hmel NETWORK NetworkB IP 172 18 8 150 gt lt NODE gt lt NODE NAME client2 NODE_ID 4 gt lt INTERFACE NAME hme0 NETWORK NetworkA IP 172 17 8 151 gt lt INTERFACE NAME hmel NETWORK NetworkB IP 172 18 8 151 gt lt NODE gt Continuous Computing Corporation upSuite User s Guide 171 14 upSuite Configuration File upsuite conf lt HEARTBEAT NAME serverl server2 TYPE POINT_TO_POINT TIMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 1 gt lt NODE_REF NODE_ID 2 gt lt LINK NETWORK NetworkA gt lt LINK NETWORK NetworkB gt lt HEARTBEAT gt lt HEARTBEAT NAME 3 1 TYPE POINT_TO_POINT TIMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 3 gt lt NODE_REF NODE_ID 1 gt lt LINK NETWORK NetworkA gt lt LINK NETWORK NetworkB gt lt HEARTBEAT gt lt HEARTBEAT NAME 4 1 TYPE POINT_TO_POINT TIMEOUT_MSEC 500 RESEND_MSEC 150 gt lt NODE_REF NODE_ID 4 gt lt NODE_REF NODE_ID 1 gt lt LINK NETWORK NetworkA gt lt LINK NETWORK NetworkB gt lt HEARTBEAT gt lt HEARTBEAT NAME 3 2 TYPE POINT_TO_POINT TIMEOUT_MSE
182. rage before each subsequent call Using the service callback function When called by ubGet State the service callback function s ub_node_id_t argu ment will be zero if the specified service is not yet active on any node The service callback function s ub_status_t argument indicates the specified service s active standby state If the argument is UB_STATUS_DOWN the specified service either does not exist or is in standby on the specified node If the argument is UB_STATUS_UP the specified service is active on the specified node Note that a service callback function will be called indicating the state of all services not just for the service that an upBeat client may have registered for Error handling If an upBeat API call fails the application should call ubF ini and cease high availability operations at least until a successful call to ubInit is made Many of the upBeat API functions indicate an error by returning 1 or NULL When this occurs the value of errno is also set This value can be set in two ways e The upBeat API call e An underlying function or system call that was made by the upBeat API If an underlying function or system call failed the value set by that call is preserved Other wise the upBeat API call sets errno itself There is no way to detect whether an upBeat API or an underlying call set the errno value You can use the strerror function to get an 30 Continuous Computing Corpo
183. ration upSuite User s Guide Guidelines for Designing Applications error string for an error It is not possible to list all the values that might be set by underlying calls The error values that can be set by the upBeat API are detailed in Error codes on page 55 P Addresses and Failover When a service is active any floating or migrating IP address that is configured for the ser vice in upsuite conf is brought up on the node for which the service is active The cor responding floating IP address is brought down on the node on which the service becomes standby When a failover occurs any TCP connections to the floating IP address are lost and clients receive errors when they try to send data on those connections In this situation clients should close and reestablish their connections Detecting Split Brain Conditions If two servers lose heartbeat connectivity but both are still running there is a network partition referred to throughout upSuite documentation as a split brain condition If the same service is running on both of the servers both services will become active each service can no longer detect the heartbeat from the other server and so must assume that the other service has failed Once heartbeat connectivity is restored it is possible to detect and correct the dual active con dition If an application is active in response to the directive callback function and later hears through the service ca
184. ress pairs for the same service on other networks These pairs of addresses are typically used between two servers providing a service The first time ubSvcIPPair is called in_addr should be 0 on subsequent calls in_addr should be the ippairp gt local from the previous call This way an application can retrieve all the pairs of IP addresses for a service Calling ubSvcIPPair iteratively Continuous Computing Corporation 50 upSuite User s Guide ubSvcIPPair returns the IP addresses from the lt NODE gt tags as indexed by NETWORK RETURN VALUES ubSvcl or you are at the end of the list otherwise it returns non zero ERRORS None SEE ALSO ubServiceIP on page 47 PPair returns O if the handle passed in upbeatp is NULL there are no IP pairs Continuous Computing Corporation upSuite User s Guide 51 5 upBeat API Reference ubSvcName NAME ubSvcName SYNOPSIS Get the name of a service char ubSvcName upbeat_t upbeatp ub_svc_id_t service upbeatp A handle as returned by ubInit service Service ID of the service for which you want to get the name Use an ID from a service callback or one returned by ubSvc DESCRIPTION Get the name of a service from the upSuite configuration file RETURN VALUES If the handle passed in upbeatp is not NULL the service exists and its NAME attribute has a value ubSvcName returns the name otherwise returns NULL ERR
185. ribute Required Gives a name to the partition The name must be unique within the node that is the NAME attributes of all lt PARTITION gt subtags of a given lt NODE gt tag must be set to differ ent values The systems in your HA configuration do not all have to use the same physical par tition as illustrated by the example earlier in this section Continuous Computing Corporation upSuite User s Guide 151 14 upSuite Configuration File upsuite conf DEVICE attribute Required This is the device name of the given partition It must be unique TIMEOUT_MSEC attribute Optional This is the amount of time SCSIbeat will wait to hear from the disk before issuing a disk failure The default is 2000 ms FREQ MSEC attribute Optional This is the frequency with which a SCSIbeat inquiry is sent to the disk The default is 950 ms Note that because SCSIbeat causes disk activity the disk LEDs may flicker or remain lit depending on the disk you are using For more information about SCSTbeat see SCSIbeat on page 11 lt HEARTBEAT gt tag A lt HEARTBEAT gt tag instructs upBeat to send heartbeats between two systems nodes over specified links Each system uses these heartbeats to determine the health of the network con nections When all links to another node are down the node is declared down If a server with an active service is declared down the standby will be told to become active for that service A lt HEA
186. rnet Switch File System 1 active File System 1 standby File System 2 File System 2 standby active SCSI Fibrechannel RAID array Figure2 Multiple file system replication operation Tracing the route that a disk write takes through the system helps to illuminate how upDisk executes its tasks as illustrated in Figure 3 The typical system call a write in this case is sent from the application to the kernel The call then passes to the VFS virtual file system before going to the local file system for example UFS UNIX file system At this point the call becomes a system specific call to a device driver for the storage medium in question Continuous Computing Corporation upSuite User s Guide 83 7 Introduction to upDisk A Application write y orany system call y TCP IP IPFS on Standby processor Y Through file system device driver and storage medium on Standby machine y ce 1 Without upSuite HA With upSuite HA Figure3 Write execution upDisk accomplishes simultaneous writes by inserting an extra layer in this process Where upDisk has been installed a virtual file system called ip fs exists between the VFS and the local FS file system layer The system call is passed from the VFS to ipfs ipfs then passes the call both to the local FS and also laterally over the network to another insta
187. s while identical are not in conflict because they occur in differ ent lt NODE gt tags lt NODE NAME nodel NODE_ID 1 gt A PARTITION NAME data DEVICE dev rdsk c0t1d0s2 gt lt NODE gt lt NODE NAME node2 NODE_ID 2 gt lt PARTITION NAME data DEVICE dev rdsk c0t1d0s2 gt lt NODE gt Example Invalid namespace use The following example shows partition names that are invalid because the two identical names occur within one lt NODE gt tag lt NODE NAME nodel NODE_ID 1 gt lt PARTITION NAME data DEVICE dev rdsk cOt1d0s2 gt lt PARTITION NAME data DEVICE dev rdsk c0t2d0s2 gt lt NODE gt Editing upsuite conf If your configuration changes you must edit the upsuite conf file and restart upBeat and all of its services To do so follow these steps 1 Save the existing upsuite conf to a backup file For example cp upsuite conf upsuite sav 2 Edit upsuite conf to reflect your configuration changes For example vi upsuite conf 3 Stop all clients of upBeat on the standby system For example if ubManager and upDisk are clients of upBeat etc init d ubmgr stop etc init d updisk stop Continuous Computing Corporation upSuite User s Guide 147 14 upSuite Configuration File upsuite conf 4 Stop all clients of upBeat on the active system For example if ubManager and upDisk are clients of upBeat et
188. s 10 Local file system device 119 mount options 120 mount point 118 type 118 Locking files 102 M MAC addresses 15 Managed failover 102 MANSs 153 171 Metadevice 166 Monitoring SCSI disks 157 Monitoring upBeat 15 Monitoring upDisk 89 monitors dumb 204 periodic 204 permanent 204 smart 203 Mount options 121 changing 123 maximum operations 123 memory allocation 124 Multi threaded applications 24 N Nodes 217 O O_SYNC 122 Online Disk Suite 166 175 Operation overview 10 Operations 82 overview 82 write 85 overview ubManager 180 P Packages adding 13 Partition table 175 Partitions 149 151 SCSI disks 149 Partitions See also SCSIbeat Peer definition 217 R Registering services API 23 Repair 85 failure 96 Replay 85 RESEND_MSEC attribute 153 Restarting 102 147 ROUTE attribute 153 Routing 14 152 153 154 171 S sample sample API program 64 Sample API programs multi threaded 69 status checker 64 source code 65 wrapper 57 source code 58 Sample configurations 161 SCSI disks 149 151 SCSIbeat 11 16 151 217 sample subtag 154 Service tag of upsuite conf 112 Continuous Computing Corporation upSuite User s Guide 223 Services configuring 154 registering 23 Share options 121 Sharing datasets in ipfstab 112 Sockets 152 153 Split brain 13 14 97 101 153 definition 217 recovering from 97 Standby 217 Starting upBeat 19 Starting upDisk 139 status API program source
189. s Computing Corporation s upBeat failure detection management software e Continuous Computing Corporation s upDisk file system replication software In this chapter What 1s UDManager sssini preesensin earte ir EE duty EEE E TE E RE 179 What is ubManager ubManager is a software layer situated above upBeat the heartbeat manager of upSuite Where upBeat detects link network and node failures ubManager extends this functionality with its ability to e Start stop and monitor non HA aware user application processes e Group services and coordinate failover among them e Add control through programs and scripts Monitor resources Continuous Computing Corporation upSuite User s Guide 179 15 Introduction to ubManager How Does ubManager Work Figure 1 illustrates the relationship between ubManager a ubManager service group and upBeat Group starts checks exit values for dumb monitors starts stops Registers group and dumb monitors checks lead service and smart monitor services Lead Service ubManager Monitor Services v normal upBeat interface for lead lt service and smart monitors Figure 1 ubManager and upBeat Service Groups By default each upBeat service is independent of all others However in a typical implementa tion certain dependencies exist among upBeat services and your applications For example your database application might depend on
190. s asynchronous status information from upBeat An application must call ubAsync periodically to keep 1ibupbeat up to date using pol 1 is the recommended way e ubGetState gets synchronous status information stored in 1ibupbeat An application may call ubGet State anytime but this is generally unnecessary the state information will be as recent as the most recent call to ubInit or ubAsync Using Callback Functions Callback functions are functions defined within your application They run when when the upBeat daemon has notified your application of a change of status and your application calls ubGetState orubAsync The callback functions come in four types The node link and service callback functions are used to provide status information about nodes links and services The directive callback function is used to tell a service to assume a given status active or standby Your application can define some or all of these types of callback functions An upBeat client that acts only as a network monitor defines one or more of the node 1ink and service types of callback functions An upBeat client that provides a service must have the direc tive type of callback function and may optionally define any of the others The node link and service callback functions are passed an argument that indicates whether the callback function is being called as a result of a call to ubAsync oras a result of a call to ubGet St
191. ss state of application PID Process ID of application Exit Last exit value of application Num Times Forked The number of times that the applica tion has been instantiated Num Times Died The number of times that the applica tion has unexpectedly terminated Num Times AWOL The number of times that the applica tion has been sent a termination signal without an indication that the applica tion has actually terminated Last Start The date and time that the application was last instantiated Continuous Computing Corporation upSuite User s Guide 201 16 ubManager Command Reference Example ubmstat van Num Num Num Times Times Times Appl PSt PID Exit Forked Died AWOL Last Start ubapp11 RUN 276 0 2 0 0 Mon Sep 30 4 03 42 2002 ubapp12 RUN 218 0 3 0 0 Mon Sep 30 14 03 42 2002 ubapp21 DEAD 88 53248 1 0 0 Mon Sep 30 4 03 12 2002 ubapp22 DEAD 90 53248 1 0 0 Mon Sep 30 4 03 12 2002 ubapp23 DEAD 98 53248 1 0 0 Mon Sep 30 4 03 12 2002 or This option prints the command usage FILES None SEE ALSO None Continuous Computing Corporation 202 upSuite User s Guide 17 ubManager Monitors This chapter contains information about the use of monitors with ubManager An alphabeti cally ordered list of the monitors provided with ubManager and their functions is also pro vided In this chapter Monitors Overview 2 c2 cence E E ee ee Le ete esi een Ale 203 TO Ad MOMs L skai a
192. ssages upon bringing your systems up Problems with startup exchange Oct 3 09 52 41 left unix ipfs GDAY failed losing link The message above indicates that a network error occurred during the initial exchange upDisk will attempt to restore the connection immediately The network problems should be addressed but upDisk will continue to provide links while able Oct 3 09 52 41 left unix ipfs GDAY wrong losing link The message above indicates that the initial exchange was invalid or corrupt upDisk will Continuous Computing Corporation upSuite User s Guide 89 8 upDisk Administration attempt to restart services If the problem continues it generally indicates that another applica tion is using the remote port and upDisk is not running on that system Operator intervention required Oct 3 09 52 41 left unix ipfs role link state dropping link operator intervention for split required The message above indicates that a condition requiring operator intervention has previously been detected and ipfs cannot proceed with operations until the problem has been fixed The link is dropped and upDisk notes the reason required for operator intervention in status displays Active Standby Mismatches Oct 3 09 52 41 left unix ROLE CLASH local ACTIVE remote ACTIVE Oct 3 09 52 41 left unix ROLE CLASH local STANDBY remote STANDBY The messages above indicate that both systems are either active or st
193. ssary in a healthy production system and to prepare to reboot a hung system Stopping the system should not cause a problem but stopping and then continuing will do so Network Guidelines In this chapter we detail issues specific to those upBeat users designing complex network architectures Routing By default upBeat only communicates with other nodes on the same subnet by using the SO_DONTROUTE socket option This means you are limited to systems that are connected by hubs and switches This behavior guarantees network paths thereby allowing upBeat to reli ably detect network failures You can change this default behavior and establish a heartbeat between nodes that are on dif ferent subnets or that are separated by routers Packets will be routed according to your node s routing tables You must ensure there are separate network paths for each link and that packets for one link cannot be routed over a path for another link Incorrect routing tables may create failure detection problems or a single point of failure leaving your system vulnerable to a split brain condition If you choose to route you must ensure that your routing tables are set up so that packets intended for one network do not get routed to a different network If this happens there could be undetected latent failures which could eventually lead to system downtime due to network outage without warning Therefore ensure that you have independent paths to each net
194. stem n path printf t s n fs state2string IPFS S STATE status printf t s n fs role2string IPFS S ROLE status printf t s n status IPFS S ACTIVE active not active printf t s n status IPFS_S_LINK link up link down printf t s n status IPFS S SYNC clean dirty printf t s n status IPFS S STOP stopped not stopped void close fd exit 0 char fs_state2string int fs_state char state_string switch fs_state Continuous Computing Corporation upSuite User s Guide 143 13 upDisk API case S NORMAL state string normal break case S SUMMARY state string summary break case S REPLAY state string replay break case S_REPAIR state string repair break default state string unknown state break return state string char fs_role2string int fs role char role_string switch fs_role case S_STARTUP role_string startup break case S_STANDBY role_string standby break case S_ACTIVE active role_string break default role_string unknown role break return role_string Continuous Computing Corporation 144 upSuite User s Guide 14 upSuite Configuration File upsuite conf This chapter describes the u
195. tag The lt NETWORK gt tag defines your networks There is no minimum or maximum number of networks A typical set of lt NETWORK gt tags looks similar to the following lt NETWORK NAME Network1 DESCRIPTION 192 168 1 0 24 gt lt NETWORK NAME Network2 DESCRIPTION 192 168 2 0 24 gt NAME attribute Required Provide a name for the network The value of the NAME attribute for each network must be different from the NAME attribute in any other lt NETWORK gt tag DESCRIPTION attribute The DESCRIPTION attribute is optional Use it to describe your network in any way you choose In this example the IP address is used lt NODE gt tag The lt NODE gt tag defines your nodes and their interfaces In addition the lt NODE gt tag may specify partitions which you can use to monitor your SCSI disks A lt NODE gt tag with parti tions specified would look similar to the following lt NODE NAME left NODE_ID 1 DESCRIPTION SPARC CP1500 Solaris 2 7 gt lt INTERFACE NAME hme0 NETWORK Network1 IP 192 168 1 2 gt lt INTERFACE NAME hmel NETWORK Network2 IP 192 168 2 2 gt lt PARTITION NAME system DEVICE dev rdsk c0t0d0s0 IMEOUT_MSEC 2000 FREQ MSEC 600 gt lt PARTITIO AME one DEVICE dev rdsk c0t1d0s5 IMEOUT_MSEC 2
196. tandby system recv Return when ipfs This option is the best tradeoff between per acknowledges the receipt formance and reliability because local and of the operation remote operations are overlapped and the operation is known to be on the standby This is called net synchronous and is the default behavior remote Return when the remote This option provides the highest reliability operation completes in combination with synchronous opera tions at the cost of some performance Table 9 Return mount options Note that the behavior of the local operation and the remote standby operation depends on whether synchronous writes are being performed For example net synchronous may provide similar reliability as does sync but with better performance For more information about fsync and O_SYNC see the following manual pages open 2 Fcnt1 2 Continuous Computing Corporation 122 upSuite User s Guide ipfs Mount Options Table 10 illustrates how to configure return behavior Changing the ipfstab file s mount options Typical ipfstab entry data ipfs ipfs localfs localfs localfs localfs share set mount mount mount mount name point opts point type device opts opts ipts ipfs ipts ufs c0t0d0s7 logging Same entry changed to document the default return behaviors ipfs ipfs return recv maxops 1000000 maxmem 10m throttle 75 100 ipfs ufs c0t0d0s7 logging
197. tive with standby status the first service will again get a directive to go active This ping ponging effect will continue for a total of three times before upBeat stops giving the services an active directive That is if a service acknowledges an active direc tive with standby status three times in a row upBeat will stop giving the service an active directive and the service remains in standby If both of the peer services are still in standby mode at this point one of them must register for active status before an active directive will again be sent to it Polling After an upBeat client finishes its initialization activities and perhaps registers a service it typ ically enters an event handling loop The UNIX API call poll or select is used to block until data is sent to the application to be processed To receive information from upBeat asynchronously the client should call ubSetup Pollfd each time before calling poll or select Calling ubSetup Pollfd will properly set up a pol1 d data structure in case the client s connection to the upBeat daemon is lost and then reestablished The po11 d data structure is used directly by poll and its file descriptor can be extracted for use by select After ubSetupPollfa is called and then poll is called and returns the client should call ubAsync regardless of the returned events for the associated pol 1fd data structure Similar to ubG
198. ts File Below is the etc host s file edited to match the system illustrated in Figure 9 Note that listed to the right of the IP address is the actual name of the server Internet host table 172 17 8 177 NFS server 172 17 8 140 serverl serverl networkl 172 17 8 141 server2 server2 networkl 172 18 8 140 serverl network2 172 18 8 141 server2 network2 clientl LIZ ELT LSO clientla 172 18 8 150 clientlb client2 172 17 8 151 client2a 172 1 8 8 4151 client2b The etc vfstab File Below is the etc vfstab file edited to match the system illustrated in Figure 9 device device mount FS fsck mount mount to mount to fsck point type pass at boot options fd dev fd fd no proc proc proc no dev dsk c0t0d0sl swap no dev dsk c0t0d0s0 dev rdsk c0t0d0s0 ufs 1 no logging dev dsk c0t0d0s6 dev rdsk c0t0d0s6 usr ufs i no logging dev dsk c0t0d0s3 dev rdsk c0t0d0s3 var ufs i no logging dev dsk c0t0d0s4 dev rdsk c0t0d0s4 opt ufs 2 yes logging swap tmp tmpfs yes This partition is reserved for upDisk Leave it commented out dev dsk cO0t0d0s7 dev rdsk c0t0d0s7 Continuous Computing Corporation 174 upSuite User s Guide Sample Configurations Partition Table Table 12 defines the partition table from the systems illustrated in Figure 7 Figure 8 and Fig ure 9 Note For sample configuration B Figure 8 the system disk s partitions are illustrated in Table 12 The d
199. unctions on page 34 Registering a Service with upBeat If an upBeat client is to provide a service it must call ubRegSvc to register the service with upBeat A service may request active or standby status upon registering and upBeat grants or refuses this request according to its knowledge of the network As changes occur on the network upBeat dynamically designates a service as active or standby accordingly Typically upBeat registers as active the first service that requests active status Any peer ser vice that later attempts to register as active is designated as standby The service will know whether it is currently active or standby once its directive callback function is called upBeat client applications can monitor the states of other services by using their service callback func tions For more information see Callback Functions on page 34 Directives from upBeat upBeat will send a directive to a service using the directive callback function in two situa tions e A service registers for active or standby status A service registering for standby status will get a directive to go standby A service regis tering for active might get a directive to go either active or standby depending on the sys tem conditions 26 Continuous Computing Corporation upSuite User s Guide Guidelines for Designing Applications The peer service is active and registers for standby or acknowledges an active direc t
200. upBeat or upDisk will notice that both sides are active If this happens upDisk shuts down the service and then signals a split brain condition both to the operator and to the peer Under normal operating circumstances this can only happen after a double failure has occurred if you are using only two links multiple communications failures if you are using more than two links and then been rectified After one of the conditions described earlier has been detected you will see a message similar to the following on your console Aug 8 08 03 24 yoursys ipfs standby down repair operator intervention now required to fix split Confirm that the split brain event still exists by running the udstat command The output from this command will let you know whether the condition has been rectified or requires your intervention upSuite HA does not take automatic corrective action once recovery from a split brain has been detected As the operator or application developer you are the only person who can determine what dataset s is most accurate Recovering from Split Brain When an upDisk server pair detects a dual active condition both sides are flagged as split brain and taken offline As a result there is no active service Tf either side is flagged as split brain upDisk flags the other side as split brain as soon as they come into contact Therefore to fix a split brain condition you must intervene on both sides If you fix and remove t
201. user application process that should always be running when group_name is active A unique name for this process must be supplied in appl_name Finally a cmdline must be given so that ubManager can start and stop the application process at the appropriate times If the user application process exits ubManager will initiate group recovery failover Note The command line arguments should not be separated by semicolons only white spaces Example a group1 ubapp usr lib ubmgr ubapp Sample Semicolon Delimited Configuration File Following is the a sample configuration file in semicolon delimited format this is in the etc upsuite examples ubmgr sc file 3 Configuration File Format 3 Don t change this line Groupl g groupl m group1 1 g1n1m1 usr lib ubmgr load mon 0 80 interval 5 m group1 2 g91n2m1 usr 1lib ubmgr load mon 0 80 interval 5 a groupl glal usr lib ubmgr ubappl argl arg2 a groupl gla2 usr lib ubmgr ubapp2 gar arg a group1 gla3 usr lib ubmgr ubapp3 Group2 g group2 updisk mydata m group2 1 g2n1ml1 usr lib ubmgr rpcmon udp machinel rpcapp 3 5 m group2 2 g2n2m1 usr lib ubmgr rpcmon udp machine2 rpcapp 3 5 m group2 1 g2n1m2 usr lib ubmgr ubpinger t 1000 g2n1m2 machinel smart m group2 2 g2n2m2 usr lib ubmgr ubpinger t 1000 g2n2m2 machine2 smart a group2 g2al usr lib ubmgr ubapp4 Continuous Computing Corporation 212 upSuite User s Guide Sample Semicolon Delimited Configur
202. ve and standby applications communicate one or more lt LINK gt subtags are used to define a pair of IP addresses that the application can use to communicate The information in the lt LINK gt tags is available to the application through the ubSvcIPPair API the application can use this information however it sees fit Many services do not require that the active and standby applications communicate and for these services the lt LINK gt subtag can be omitted Also in services with only a single node the lt LINK gt subtag is not needed If you define links using this subtag in a lt SERVICE gt tag define the same links in a lt HEART BEAT gt tag so that the application can receive link callbacks for those links Example lt LINK NETWORK Network1 ROUTE ROUTE gt NETWORK attribute Required This allows the services to associate IP addresses from two different nodes This is a reference to the NETWORK attribute of the lt INTERFACE gt subtag of the lt NODE gt tag Example NETWORK Network1 ROUTE attribute Optional The value of this attribute is made available to the application through the ubSvcIPPair API the application may use the information however it wishes If the attribute is not present the default behavior is to not route Example ROUTE ROUTE PREF attribute Optional The information specified in the PREF attribute is passed to the user application to
203. ve standby errors 93 bringing systems online 89 comments 92 link status 92 miscellaneous transmission errors 94 mismatches 90 Continuous Computing Corporation upSuite User s Guide 221 operational messages 91 ublnit 43 operator intervention 90 ubNode 44 problems with startup exchange 89 ubNodeName 45 role status 91 ubRegSvc 46 starting up and mounting ipfs 91 ubServiceIP 47 state status 92 ubSetupPollfd 48 warning messages underlying file system 90 ubSvc 49 Ctrl C 58 ubSvcIPPair 50 ubSvcName 52 D ubSvcPeer 53 ubSvcPort 54 Datasets 104 in ipfstab 118 H sharing in ipfstab 112 Debug mode 15 18 19 138 HA NFS 107 Definitions of Terms 217 subtag 157 Disk space 95 HA NES definition 217 E l Editing the configuration file 102 111 145 147 Interface 111 lt SERVICE gt tag 112 ioctl function call 141 IP addresses 111 146 150 154 158 F ipfs 84 217 mount options 118 Failover 7 102 154 mount point 118 Failover definition 217 mounting 91 features 81 177 179 operations 85 File locking 102 starting up 91 unmounting 95 File modification times 96 ipfstab 117 File systems 95 access denied 94 underlying 118 119 K FSYNC 122 Function calls 35 kstat 89 guidelines 23 ubAckSvc 36 L ubAsync 37 ubFini 40 LANs 15 171 ubGetState 41 libupbeat 23 Continuous Computing Corporation 222 upSuite User s Guide Local definition 217 Local application
204. vers go active when the split brain is repaired upDisk notices the dual active con dition and takes both sides offline To find out why offline is displayed consult the log files Examples ipfs STANDBY summary DOWN dirty offline ipfs STANDBY summary DOWN clean of fl ipfs STANDBY repair DOWN dirty offline ipfs STARTUP repair DOWN dirty operator splitbrain offline The h flag returns history information The history entries are updated every time there is a change in the state among startup active and standby If a given component is without a role and becomes standby the history is changed and vice versa Other wise the current entry is updated Continuous Computing Corporation upSuite User s Guide 135 12 upDisk Command Reference OPTIONS cont d FILES SEE ALSO The h flag returns the most current change Output will look similar to the following left udstat h ipfs 2001 10 01 14 25 10 4710 45 00000448bcda1494 00000000 ACTIVE replay replay repair busy dirty fsck odirty new onew gday session epoch def rep 0 501 The hh flag returns a predefined number currently eight of recent changes made The hhh flag returns the same information as hh and includes the times the changes were made etc ipfstab usr sbin udrepair usr sbin udactive 136 Continuous Computing Corporation upSuite User s Gui
205. want to use the TCP protocol in an HA NFS system this message is of con cern HA NFS may fail immediately after the first failover The failure will be indicated by the following message which appears on the new active server s console and in var adm messages NOTICE KRPC record fragment from client of size 10216 exceeds maximum 9000 Fragment header was 0x8000807c Disconnecting The client size and maximum size in the actual message might be different The server is not affected by this Errors if any will be on the NFS client Solaris clients seem to recover well but Linux clients hang and must be rebooted We recommend that you call Continuous Computing Corporation s technical support if you encounter this problem 100 Continuous Computing Corporation upSuite User s Guide 9 Failover This chapter explains automatic and managed failover In this chapter upDisk Failover Considerati0MS ooonnonincnocncononnconononono aaa aaa aaa aaa aaa aaa aaa 101 Normal Falo ver ui io 101 Managed FallOvercoio cosillas i 102 Boot Sequence moi ia ie i E 104 upDisk Failover Considerations The split brain issue is more important for upDisk than for many applications If a standby upDisk server has different data from its active server the standby is repaired to match the active In the worst case scenario a disk is replaced the pair of servers is rebooted the server with the empty disk is selected as active an
206. work and that routers do not route packets between the two networks For more information about how to enable routing see lt HEARTBEAT gt tag on page 152 14 Continuous Computing Corporation upSuite User s Guide Monitoring upBeat MAC Addresses Solaris by default instructs all Ethernet interfaces in the same system to use the same MAC address If each of the interfaces is connected to a different network this works well However if you connect several LANs to the same hub or switch or cascade of hubs and switches net work slowdowns or outages may result In particular if the hubs and switches are the kind that learn on which port a MAC address is located more problems may occur If you segment a switch into virtual LANs VLANs be aware that some switches incorrectly route among VLANs based on the MAC address As mentioned before these switches have the ability to learn albeit incorrectly in some cases MAC addresses Listed below are possible solutions to this problem e Set up Solaris so that it does not instruct all Ethernet hardware to use the same address by modifying the local mac address and OpenBoot variable e Use only switches that do not have leaky VLANs e Avoid the use of VLANs Continuous Computing Corporation recommends setting the local mac address variable to true on all Sun systems running upSuite HA Monitoring upBeat Current status and changes to upBeat
207. y if the peer group cannot become active then the local group will remain active ubManager can support two types of monitor smart or dumb Smart monitors are able to directly update their service status with upBeat while dumb monitors rely on ubManager to do this for them When a smart monitor is active that informs ubManager that the resource that the monitor is monitoring is healthy Likewise when a smart monitor is standby that informs ubManager that the resource that the monitor is monitoring is unhealthy Dumb monitors may be permanent or periodic When a dumb permanent monitor exits no matter its exit value that informs ubManager that the resource that the monitor is monitoring is unhealthy and ubManager registers the monitor s upBeat service as standby if it is not already and restarts the monitor A dumb periodic monitor is expected to exit quickly and ubManager restarts it at a specified interval The exit value of a dumb periodic monitor indi cates if the resource that it s monitoring is healthy exit value of zero or unhealthy non zero exit value Monitor processes may be scripts or binary executables Scripting can provide a simple yet powerful way to implement a monitor where disk usage processor utilization process status and the like can be monitored ubManager Monitors on page 203 contains information about the monitors shipped with ubManager As an example below is loadMon a dumb periodic

Download Pdf Manuals

Related Search

Related Contents

User manual - Humantechnik  Product Specification  Digital Projector User Manual ES500/EX501  THB CC 9055  Manual de Usuario - Instituto Politécnico Nacional  Bedienungsanleitung  IT Support Specialist - Abramson Family Cancer Research Institute  Makita 1002BA User's Manual  DFI NS35-SL Motherboard  Protection économique du consommateur  

Copyright © All rights reserved. Failed to retrieve file