Home

Veritas Cluster Server User's Guide

1. a Sse Sis Clients Redirected Cluster B Application Failover Separate Separate Storage Storage In a global cluster if an application or a system fails the application is migrated to another system within the same cluster If the entire cluster fails the application is migrated to a system in another cluster Clustering on a global level also requires replicating shared data to the remote site See How VCS global clusters work on page 462 Chapter 3 VCS configuration concepts About the VCS configuration language About the main cf file The types cf file About VCS attributes About VCS keywords and reserved words VCS environment variables VCS configuration concepts About configuring VCS About configuring VCS Configuring VCS means conveying to the VCS engine the definitions of the cluster service groups resources and resource dependencies VCS uses two configuration files in a default configuration m main cf Defines the cluster including services groups and resources m types cf Defines the resource types By default both files reside in the directory etc VRTSvcs conf config Additional files similar to types cf may be present if agents have been added such as Oracletypes cf In a VCS cluster the first system to be brought online reads the configuration file and creates an
2. Start system use only Indicates whether a resource was started the process of bringing it online was initiated on a system m Type and dimension integer scalar m Default 0 State system use only Resource state displays the state of the resource and the flags associated with the resource Flags are also captured by the Flags attribute This attribute and Flags present a comprehensive view of the resource s current state Values ONLINE OFFLINE FAULTED ONLINE STATE UNKNOWN ONLINE MONITOR TIMEDOUT ONLINE UNABLE TO OFFLINE OFFLINE STATE UNKNOWN FAULTED RESTARTING A FAULTED resource is physically offline though unintentionally m Type and dimension integer scalar m Default 0 TriggerEvent system use only A flag that turns Events on or off m Type and dimension boolean scalar m Default 0 620 VCS attributes Resource type attributes Resource type attributes Table D 2 lists the resource type attributes You can override some static attributes for resource types See Overriding resource type static attributes on page 208 For more information on any attribute listed below see the chapter on setting agent parameters in the Veritas Cluster Server Agent Developer s Guide Table D 2 ActionTimeout user defined Resource type attributes Timeout value for the Action entry point m Type and dimension integer scalar m Default 30 seconds AgentCl
3. hagrp autoenable service_group sys system Service group is waiting for the resource to be brought online taken offline Recommended Action Review the State attribute of all resources in the service group to locate which resource is waiting to go online or which is waiting to be taken offline Use the hastatus command to help identify the resource See the engine and agent logs in var VRTSvcs log for information on why the resource is unable to be brought online or be taken offline To clear this state make sure all resources waiting to go online offline do not bring themselves online offline Use the command hagrp flush to clear the internal state of VCS You can now bring the service group online or take it offline on another system Service group is waiting for a dependency to be met Recommended Action To see which dependencies have not been met type hagrp dep service_group to view service group dependencies or hares dep resource to view resource dependencies Service group not fully probed This occurs if the agent processes have not monitored each resource in the service group When the VCS engine HAD starts it immediately probes to find the initial state of all of resources It cannot probe if the agent is not returning a value A service group must be probed on all systems included in the SystemList attribute before VCS attempts to bring the group online as part of AutoStart This ensures that even if th
4. Database IP Address HS Cm Disk Group Resource dependencies determine the order in which resources are brought online or taken offline For example a disk group must be imported before volumes in the disk group start and volumes must start before file systems are mounted Conversely file systems must be unmounted before volumes stop and volumes must stop before disk groups are deported A parent is brought online after each child is brought online and so on up the tree until finally the application is started Conversely to take a managed application offline you stop resources beginning at the top of the hierarchy In this example the application is stopped first followed by the database application Next the IP address and file systems can be stopped concurrently 22 Introducing Veritas Cluster Server Logical components of VCS since they do not have any resource dependency between them and so on down the tree Child resources must be online before parent resources can be brought online Parent resources must be taken offline before child resources can be taken offline If resources do not have parent child interdependencies they can be brought online or taken offline concurrently Categories of resources Different types of resources require different levels of control In VCS there are three categories of resources m On Off VCS starts and stops On Off resources as required For example VCS
5. See Running a virtual fire drill from the Java Console on page 168 See Running a virtual fire drill from the command line on page 275 Chapter Predicting VCS behavior using VCS Simulator About VCS Simulator Installing VCS Simulator Administering VCS Simulator from the Java Console Administering VCS Simulator from the command line 9 318 Predicting VCS behavior using VCS Simulator About VCS Simulator About VCS Simulator VCS Simulator enables you to simulate and test cluster configurations Use VCS Simulator to view and modify service group and resource configurations and test failover behavior VCS Simulator can be run on a standalone system and does not require any additional hardware VCS Simulator runs an identical version of the VCS High Availability Daemon HAD as in a cluster ensuring that failover decisions are identical to those in an actual cluster You can test configurations from different operating systems using VCS Simulator For example you can run VCS Simulator on a Windows system and test VCS configurations for Windows Linux and Solaris clusters VCS Simulator also enables creating and testing global clusters You can administer VCS Simulator from the Java Console or from the command line Installing VCS Simulator To install VCS Simulator on UNIX systems 1 Insert the VCS installation disc into a drive 2 Navigate to the following directory and locate the package VRTScss
6. hagrp display service group attribute attribute sys system Note that system names are case sensitive Administering the cluster from the command line 255 Querying VCS Querying resources To display a resource s dependencies Type the following command hares dep resource To display information about a resource Type the following command hares display resource If resource is not specified information regarding all resources is displayed To confirm an attribute s values are the same on all systems Type the following command hares global resource attribute value key key value To display resources of a service group Type the following command hares display group service group To display resources of a resource type Type the following command hares display type resource type To display attributes of a system Type the following command hares display sys system 256 Administering the cluster from the command line Querying VCS Querying resource types To display all resource types Type the following command hatype list To display resources of a particular resource type Type the following command hatype resources resource type To display information about a resource type Type the following command Type the following command hatype display resource type If resource_type is not specified information regardin
7. 530 Administering global clusters from the command line Administering resources Administering resources To take action on a resource across clusters hares action resource token actionargs argl sys system clus cluster localclus The option clus implies resources on the cluster If the designated system is not part of the local cluster an error is displayed If the sys option is not used it implies resources on the local node To invoke the Info entry point across clusters hares refreshinfo resource sys system clus cluster localclus Causes the Info entry point to update the value of the ResourceInfo resource level attribute for the specified resource if the resource is online If no system or remote cluster is specified the Info entry point runs on local system s where the resource is online To display usage for the resource command To display usage for the command hares and its various options hares help modify list Administering global clusters from the command line 531 Administering clusters Administering clusters To add a remote cluster object haclus add cluster ip The variable cluster represents the cluster This command does not apply to the local cluster To delete a remote cluster object haclus delete cluster The variable cluster represents the cluster To modify an attribute of a local or remote cluster object haclus modify attribute value
8. 8 Bring the ClusterService service group online Sample configuration group ClusterService SystemList vcssol5 vcssol6 AutoStartList vcssol5 vcssol6 OnlineRetryLimit 3 IP webip Address 162 39 9 85 NetMask 255 255 255 0 Device qfe0 VRTSWebApp VCSweb AppName cmc InstallDir opt VRTSweb VERITAS TimeForOnline 5 Critical 0 vCSweb requires webip webip requires csgnic 74 Administering the cluster from the Cluster Management Console Logging in to the Cluster Management Console Logging in to the Cluster Management Console The Cluster Management Console is a web application that is installed on a VCS node You must log in to the Cluster Management Console to manage your VCS cluster To run the console the local system the system you use to run the console must also be running a supported web browser To log in to the console 1 In your Internet browser enter the URL address configured for the Cluster Management Console during installation You can enter either http System 8181 cmc or https System 8443 cmc where Systemis either a system name or the virtual IP address configured for the webip resource in the ClusterService service group On Windows you can click Start gt Programs gt VERITAS Cluster Management Console to view the log in screen Under Please enter a web admin username and password to continue select a language from the dr
9. Advanced m Enter the credentials of a native user You can use nis or nis accounts or accounts set up on the local system If you do not enter the name of the domain VCS assumes the domain is the local system If the user does not have root privileges on the system VCS assigns guest privileges to the user To override these privileges add the domain user to the VCS administrators list See Administering user profiles on page 174 172 Administering the cluster from Cluster Manager Java console Administering Cluster Monitor The Java Console connects to the cluster using the authentication broker and the domain type provided by the engine To change the authentication broker or the domain type click Advanced See About security services on page 28 T vcslinux145 14141 Eg User name froot Password ii Domain a Broker 0 212 102 32 2821 2 Domain Type funx o Cancel Collapse Select a new broker and domain type as required Click OK The Server Credentials dialog box displays the credentials of the cluster service to which the console is connected fT Server Credentials xi Following Server Credentials were received User Name _HA_VCS_vesw2kpm8 Domain HA_SERVICES vcsw2kpm8 vcsnet com Domain Type vx Issued By CN broker OU root vcsw2kpms vcsnet com O Do you wish to continue connecting to the server T Do not show during startup OK Cancel To d
10. Note You cannot delete the heartbeat if it is the only remaining heartbeat between a global and remote cluster To delete a global heartbeat 1 Inthe Cluster Heartbeats view in the Heartbeats Listing table click the X button in the line item corresponding to the heartbeat that you want to delete 2 Click OK to confirm that you want to delete the heartbeat Administering global clusters from the Cluster Management Console 501 Administering global heartbeats Modifying a global heartbeat Change the clusters that participate in a heartbeat and the heartbeat parameters at any time 1 In the Cluster Heartbeats view in the Heartbeats Listing table click the edit button in the line item corresponding to the heartbeat that you want to modify In the Edit Heartbeat dialog box specify the following details for the heartbeat and then click OK m The heartbeat name Enter a name for the heartbeat that is indicative of the generating it m The global clusters to add to the cluster list for the heartbeat In the Cluster List for Heartbeat table check the box preceding the line item for each cluster that you want to participate in the heartbeat Clear the check box to exclude clusters from the heartbeat list m The Arguments attribute value interval and timeout options Click the edit button in the Settings column to specify the value for the Arguments attribute the timeout options and the interval options Click
11. ceeesccsesessesesseseseeeeceseeesesesecseeeeseeeeeeaeees 18 Physical components Of VCS o ccesssssesssssesesesesesessessssceseseseseseseesseseeeeseseseseseees 20 Logical components Of VCS uu cccecesessssesssssssesesesesessssssssseseesesesesesesesseeseneeeseeseaes 21 Putting the pieces together ci sex esscdssiteeerieess es hesst exci nui a RRE 30 About cluster topologies Basic failover configurations cccsesesssssesesesesesessssesssecesesesesesesstsseeeeseesecseaes 34 Advanced failover configurations cccssssssssesssesssesesessssssessceseseseseseeetseseeees 38 Cluster topologies and storage configurations ccccessssseseseseseseseseseseseeees 41 VCS configuration concepts About configuring VCS o ccccccsesssssesesesesessssssssscesesescsesessssssssceseeeseseseeesseeeeees About the VCS configuration language About the main cf file 2 cecececeeceseceeseseseseseseescseseseseceseeesesescseseseseseeteneeeseaeaees Thetypescf file pnanicnnennrn n ann n E E E tater eweeee About VCS attributes occ ecsessssesecesseeseseseseseceeeeeeeseseaeeeseeeeeeeeetaeseaeeeeeeees About VCS keywords and reserved words ccccccssssesesseseseeceseeeeeeseeeeeeseeeeaees 55 VCS environment variables 0 eeseseseseseeteeeesececeseseseseseeseeeeeeeseaeaeaeeeeeeeeeees 56 Administration Putting VCS to work About the VCS user privilege model About VCS user privileges and roles ccceccseeseseseeseseseeseseeceseeceeesesee
12. m From the Select Application list select the application to be the parent m From the Available Applications box click on the application to be the child Note Make sure that there is no circular dependency among the applications m Click the button with the right arrow icon to move the selected application to the Child Applications box To remove an application dependency select the application in the Child Applications box and click the button with the left arrow icon m Click Next Configuring applications and resources in VCS 309 Configuring NFS service groups 14 Review your configuration and change resource names if desired Service Group Summary Review the configuration and make any necessary changes to the resource names f ApplicationGroup aes Zone_zone pplication Group appl Saent vessun51 app2 AutoStartList vcssun51 DiskRes_dev_dsk_c1t0d0 Mount_zone_izone Attributes More information lt Back Finish Cancel The left pane lists the configured resources Click on a resource to view its attributes and their configured values in the Attributes box To edit a resource name select the resource name and click on it Press Enter after editing each name Note that when modifying service groups you can change names of newly created resources only which appear in black Click Finish The wizard starts running commands to create or modify the service g
13. 240 Administering the cluster from the command line Logging on to VCS Logging on to VCS VCS prompts user name and password information when non root users execute haxxx commands You can use the halogin command to save the authentication information so that you do not have to enter your credentials every time you run a VCS command Note that you might need specific privileges to run VCS commands When you run halogin VCS stores encrypted authentication information in the user s home directory In secure clusters the command also sets up a trust relationship and retrieves a certificate from an authentication broker If you run the command for different hosts VCS stores authentication information for each host After you run the command VCS stores the info until you end the session Root users need not run halogin when running VCS commands from the local host To log on to a cluster running in secure mode 1 Set the following environment variables m VCS_DOMAIN Name of the Security domain to which the user belongs m VCS_DOMAINTYPE Type of VxSS domain unixpwd nt nis nisplus or VX 2 Define the node on which the VCS commands will be run Set the VCS_HOST environment variable to the name of the node To run commands in a remote cluster you set the variable to the virtual IP address configured in the ClusterService group 3 Logon to VCS halogin vcsusername password To log on to a cluster not running in
14. A Add Service Group Link Service Groups Unlink Service Groups Edit Attribute The task pane contains a narrow button on the border that separates it from the status pane on the right This button toggles the task pane between maximum and minimum size In the startup view click this button to minimize the task pane Minimizing the task pane hides the Configuration task panel and enlarges the status pane enabling you to view information that might have been partially off screen Click the narrow button again to maximize the task pane Even with the task pane maximized you can still display off screen information in the status pane using the scroll bar located along the bottom border of any view About task panels The task panels in the task pane list specific operations that you can perform in each view of the Cluster Management Console For views that list specific objects clusters service groups systems and so on the task panels list all the possible tasks that you can perform on those objects Tasks related to object 79 80 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console management are usually divided between two task panels named Operations and Configuration The name of a task panel indicates the type of tasks listed in that panel Underlined tasks indicate a link that performs an action directly or which starts a task wizard sequence of dialog boxes
15. E Remote cluster information Cluster Name vcslinux145_146_QA22 Host NamejIP address vcslinux145 Port 14141 Use connected clusters credentials C Enter new credentials User Name Password Domain Cancel m Enter the host name of a cluster system an IP address of a cluster system or the IP address of the cluster that will join the global environment m Verify the port number m Choose to connect to the remote cluster with the credentials used for the current cluster connection or enter new credentials including the user name password and the domain m Click OK 7 Click Next 8 Click Finish After running the wizard the configurations on all the relevant clusters are opened and changed the wizard does not close the configurations To add a remote cluster to a global cluster environment in Command Center Note Command Center enables you to perform operations on the local cluster this does not affect the overall global cluster configuration 1 Click Commands gt Configuration gt Cluster Objects gt Add Remote Cluster 2 Enter the name of the cluster 3 Enter the IP address of the cluster 4 Click Apply Administering global clusters from Cluster Manager Java console 509 Deleting a remote cluster Deleting a remote cluster The Remote Cluster Configuration Wizard enables you to delete a remote cluster This operation involves the following tasks Taking the wac resource i
16. In this step you update the global cluster list of available global clusters To perform the update you must first verify or supply the authentication credentials for each remaining global cluster The Cluster Management Console can then connect to those clusters and update the list In the Remote Cluster Configuration dialog box verify the required information for the remaining remote clusters and then click Next To change authentication information click a cluster name under Existing Clusters and then enter the authentication information in the fields to the right The requisite information in this dialog box varies depending upon whether or not the cluster is secure uses an authentication broker Click Next Click Finish Administering global service groups 494 Administering global clusters from the Cluster Management Console Administering global service groups After connecting clusters in a global cluster environment use the Global Group Configuration wizard to convert a local service group that is common to the global clusters to a global group This wizard also enables you to convert global groups into local groups Administering global groups requires the following conditions A group that is intended to serve as the global group must have the same name across all applicable clusters You must know the user name and password for the administrator for each cluster in the configuration Use the Cluster summary Cl
17. 639 Service group attributes Table D 3 Service group attributes TriggerResFault user defined Defines whether VCS invokes the resfault trigger when a resource faults The value 0 indicates that VCS does not invoke the trigger See resfault event trigger on page 454 m Type and dimension integer scalar m Default 1 TriggerResStateChan ge user defined Determines whether or not to invoke the resstatechange trigger if resource state changes See resstatechange event trigger on page 456 m Type and dimension boolean scalar Default 0 disabled TypeDependencies user defined Creates a dependency via an ordered list between resource types specified in the service group list and all instances of the respective resource type m Type and dimension string keylist m Default 640 VCS attributes Service group attributes Table D 3 Service group attributes UserIntGlobal user defined Use this attribute for any purpose It is not used by VCS m Type and dimension integer scalar m Default 0 UserStrGlobal user defined VCS uses this attribute in the ClusterService group Do not modify this attribute in the ClusterService group Use the attribute for any purpose in other service groups m Type and dimension string scalar m Default 0 UserIntLocal user defined Use this attribute for any purpose It is not used by VCS m Type and di
18. Examples of system state transitions If VCS is started on a system and if that system is the only one in the cluster with a valid configuration the system transitions to the RUNNING state INITING gt CURRENT_DISCOVER_WAIT gt LOCAL_BUILD gt RUNNING If VCS is started on a system with a valid configuration file and if at least one other system is already in the RUNNING state the new system transitions to the RUNNING state INITING gt CURRENT_DISCOVER_WAIT gt REMOTE_BUILD gt RUNNING If VCS is started on a system with an invalid configuration file and if at least one other system is already in the RUNNING state the new system transitions to the RUNNING state INITING gt STALE_DISCOVER_WAIT gt REMOTE_BUILD gt RUNNING If VCS is started on a system with an invalid configuration file and if all other systems are in STALE_ADMIN_WAIT state the system transitions to the STALE_ADMIN_WAIT state as shown below A system stays in this state until another system with a valid configuration file is started INITING gt STALE_DISCOVER_WAIT gt STALE_ADMIN_WAIT If VCS is started on a system with a valid configuration file and if other systems are in the ADMIN_WAIT state the new system transitions to the ADMIN_WAIT state INITING gt CURRENT_DISCOVER_WAIT gt ADMIN_WAIT If VCS is started on a system with an invalid configuration file and if other systems are in the ADMIN_WaIT state the new system transitions to
19. This command initializes a service group that is ready to contain various resources To employ the group properly you must populate its SystemList attribute to define the systems on which the group may be brought online and taken offline A system list is an association of names and integers that represent priority values To delete a service group Type the following command hagrp delete service group Note that you cannot delete a service group until all of its resources are deleted Modifying service group attributes To modify a service group attribute Type the following command hagrp modify service group attribute value sys system The variable value represents system_name1 priority system_name2 priority2 If the attribute being modified has local scope you must specify the system on which to modify the attribute except when modifying the attribute on the system from which you run the command For example to populate the system list of service group groupx with Systems A and B type hagrp modify groupx SystemList add SystemA 1 SystemB 2 Similarly to populate the AutoStartList attribute of a service group type hagrp modify groupx AutoStartList SystemA SystemB 262 Administering the cluster from the command line Administering service groups You may also define a service group as parallel To set the Parallel attribute to 1 type the following command Note that the default for this
20. VCS Application Configuration Wizard Application Options Create a new application or modifyidelete an existing one Create Application Application Name app1 More information lt Back Next gt cancel m To create an application choose the Create Application option and enter the name of the application To modify an application choose the Modify Application option and select the application To delete an application click Delete Application m Click Next Note Choose the Configure Application Dependency option only after you have finished with adding modifying or deleting applications 304 Configuring applications and resources in VCS Configuring application service groups 8 Specify information about the executables used to start stop and clean the application VCS Application Configuration Wizard Application Details Enter the application details Start Program zoneflzonefrootvoptApplicationTestapp_test_start Start Program Parameters Stop Program zoneflzonefroovoptApplicationTestapp_test_stop Stop Program Parameters Clean Program lzonesizonefroatioptiApplicationTestiapp_test_clean Clean Program Parameters User x Discover Users E More information lt Back Next gt Cancel m Specify the locations of the Start Stop and Clean optional programs along with their parameters You must specify values for the Start and Stop progra
21. VCS Simulator copies the sample configurations to the path sim_dir clustername and creates a system named clustername_sys1 For example to add cluster clus_a using ports 15555 and 15575 run the following command sim dir hasim setupclus clus_a simport 15555 wacport 15575 328 Predicting VCS behavior using VCS Simulator Administering VCS Simulator from the command line Similarly add the second cluster sim dir hasim setupclus clus _b simport 15556 wacport 15576 To create multiple clusters without simulating a global cluster environment specify 1 for the wacport Start the simulated clusters sim _dir hasim start clustername _sysl1 clus clustername Set the following environment variables to access VCS Simulator from the command line m VCS_SIM PORT port_number m VCS_SIM WAC PORT wacport Note that you must set these variables for each simulated cluster otherwise Simulator always connects default_clus the default cluster You can use the Java Console to link the clusters and to configure global service groups See Administering the cluster from Cluster Manager Java console on page 135 You can also edit the configuration file main cf manually to create the global cluster configuration Predicting VCS behavior using VCS Simulator 329 Administering VCS Simulator from the command line Administering simulated clusters from the command line The functionality of VCS Simulator commands mimic tha
22. Veritas Cluster Server User s Guide Solaris 5 0 N18535F Ss 3 antec 7 Veritas Cluster Server User s Guide Copyright 2006 Symantec Corporation All rights reserved Veritas Cluster Server 5 0 Symantec the Symantec logo Veritas are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U S and other countries Other names may be trademarks of their respective owners The product described in this document is distributed under licenses restricting its use copying distribution and decompilation reverse engineering No part of this document may be reproduced in any form by any means without prior written authorization of Symantec Corporation and its licensors if any THIS DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS REPRESENTATIONS AND WARRANTIES INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR NON INFRINGEMENT ARE DISCLAIMED EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID SYMANTEC CORPORATION SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING PERFORMANCE OR USE OF THIS DOCUMENTATION THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE The Licensed Software and Documentation are deemed to be commercial computer software and commercial computer software documentation as defined in FAR Sections 12 212 and DF
23. root Password Should belong to the group root on vcslabweb m Email Email address of the new recipient m Severity Threshold for receiving Web server events Select one of the following values INFO WARN ERROR SEVERE Locale The locale in which notification is to be sent m Username User having superuser administrative privileges on the Web server system m Password Password for the superuser Click OK 671 672 Administering Symantec Web Server Configuring SMTP notification for VRTSweb Deleting an SMTP recipient Delete an SMTP recipient to prevent VRTSweb from sending notifications to the recipient To delete an SMTP recipient from the command line Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui smtp delrcpt email The variable email represents the email address of the recipient to be deleted For example opt VRTSweb bin webgui smtp delrcpt admin company com To delete an SMTP recipient from the Web console 1 Access the Web server on a configured port For example http hostname 8181 Click the Configuration tab Click Delete SMTP Recipient on the left side of the Configuration page In the Delete SMTP Recipient dialog box m Enter the email address of the recipient to be deleted m Enter the name and password for a user having superuser administrative privileges on the Web server system Click OK Administering Syma
24. 1 Inthe Service Groups tab of the Cluster Explorer configuration tree click a service group to which the resource will be added 2 On the Edit menu click Add and click Resource or Click Add Resource in the Cluster Explorer toolbar 3 Enter the details of the resource m Enter the name of the resource GA Add Resource Group name TestGrp Resource name fTestRes Resource Type Fei onopp Dimension 3 IV Critical Enabled El Show Command m Click the resource type m Fdit resource attributes according to your configuration The Java Console also enables you to edit attributes after adding the resource m Select the Critical and Enabled check boxes if applicable The Critical option is selected by default A critical resource indicates the service group is faulted when the resource or any resource it depends on faults An enabled resource indicates agents monitor the resource you must specify the values of mandatory attributes before enabling a resource If a resource is 200 Administering the cluster from Cluster Manager Java console Administering resources created dynamically while VCS is running you must enable the resource before VCS monitors it VCS will not bring a disabled resource nor its children online even if the children are enabled m Click Show Command in the bottom left corner to view the command associated with the resource Click Hide Command to clo
25. 290 Configuring applications and resources in VCS About VCS bundled agents Network agents Network agents make your IP addresses and computer names highly available Agent m NIC m IP m MultiNICA C IPMultiNIC m MultiNICB AIX HP UX Solaris m IPMultiNICB AIX HP UX Solaris m DNS Fileshare agents Description Monitors a NIC Network Interface Card Monitors an IP address Monitors multiple network interfaces Manages the virtual IP address configured as an alias on an interface of a MultiNICA resource Monitors multiple network interfaces Configures and manages virtual IP addresses IP aliases on an active network device specified by the MultiNICB resource Updates and monitors the canonical name CNAME mapping in the domain name server when failing over applications across subnets performing a wide area failover File Service agents make shared directories and subdirectories highly available Agent m NFS m NFSRestart m Share m Samba Linux Description Monitors a shared directory Use in conjunction with the NFSRestart agent Recovers NFS record locks after sudden reboots or crashes on clients and servers This avoids file corruption and provides the high availability of NFS record locks Use with the NFS agent Configure as the top most resource in the service group dependency Shares unshares and monitors a directory Suite of three agents that work together to pr
26. By default VCS monitors each resource every 60 seconds You can change this by modifying the MonitorInterval attribute for the resource type You may consider reducing monitor frequency for non critical or resources with expensive monitor operations Note that reducing monitor frequency also means that VCS may take longer to detect a resource fault By default VCS also monitors offline resources This ensures that if someone brings the resource online outside of VCS control VCS detects it and flags a 548 VCS performance considerations How cluster components affect performance concurrency violation for failover groups To reduce the monitoring frequency of offline resources modify the OfflineMonitorInterval attribute for the resource type The VCS agent framework uses multithreading to allow multiple resource operations to run in parallel for the same type of resources For example a single Mount agent handles all mount resources The number of agent threads for most resource types is 10 by default To change the default modify the NumThreads attribute for the resource type The maximum value of the NumThreads attribute is 30 Continuing with this example the Mount agent schedules the monitor entry point for all mount resources based on the MonitorInterval or OfflineMonitorInterval attributes If the number of mount resources is more than NumThreads the monitor operation for some mount resources may be required to wait to execute t
27. Configuring notification Configuring notification involves creating a resource for the Notifier Manager NotifierMgr agent in the ClusterService group See the Veritas Cluster Server Bundled Agents Reference Guide for more information about the agent VCS provides several methods for configuring notification m Manually editing the main cf file m Using the Notifier wizard See Setting up VCS event notification using the Notifier wizard on page 225 Chapter 1 4 VCS event triggers m About VCS event triggers m Using event triggers m List of event triggers 446 VCS event triggers About VCS event triggers About VCS event triggers Triggers let you invoke user defined scripts for specified events in a cluster VCS determines if the event is enabled and invokes the hatrigger script The script is located at VCS_HOME bin hatrigger VCS also passes the name of the event trigger and associated parameters For example when a service group comes online on a system VCS invokes the following command hatrigger postonline system service_group VCS does not wait for the trigger to complete execution VCS calls the trigger and continues normal operation VCS invokes event triggers on the system where the event occurred with the following exceptions m VCS invokes the sysoffline and nofailover event triggers on the lowest numbered system in the RUNNING state m VCS invokes the violation event trigger on all
28. Global Scalar o Global Scalar 2 Global Scalar o Global Scalar Read Write Global Scalar 5 al Global Scalar a ao Ba In the Edit Attribute dialog box enter the changes to the attribute values To edit a scalar value Enter or click the value To edit a non scalar value Use the button to add an element Use the button to delete an element To change the attribute s scope Click the Global or Per System option To change the system for a local attribute Administering the cluster from Cluster Manager Java console 223 Editing attributes Click the system from the menu 5 Click OK 224 Administering the cluster from Cluster Manager Java console Querying the cluster configuration Querying the cluster configuration 1 From Cluster Explorer click Query on the Tools menu or On the Cluster Explorer toolbar click Query 2 Enter the details of the query T Custer Query Query Select state X Offline GAD COR fautostarttist gt feontains vcshp71 Query Result At Feb 21 2006 12 51 11 PM Service Group State AutoStartlist CMC veshpi0 Offline veshp71 Online veshp10 veshp71 ClusterService veshp10 Offline veshp71 Online veshp10 veshp71 newgroup2 veshp10 Offline veshp71 Online veshp10 veshp71 m Click the VCS object to search m Depending on the selected object click the specific entity to search m Click the
29. In the Resources Listing table click the linked name of a resource In the Resource Summary view on the secondary tab bar click Dependency This view presents a graphical representation of the selected resource and all currently configured resources that have a dependency relationship with the resource Click a resource icon to get information about the resource that it represents Unlinking resources Unlink two resources to remove a dependency You must have the role of cluster administrator or service group administrator to unlink resources To unlink two resources 1 In the Resource Summary view in the Configuration task panel click Link Unlink Resource In the Link Unlink Resource dialog box in the Currently Linked table click the X button on the line item that corresponds to the link that you want to remove Click OK Modifying a resource attribute Edit the attributes of a resource You must have the role of cluster administrator or service group administrator to edit resource attributes To edit a resource attribute using the attribute table 1 2 In the Resource Summary view on the secondary tab bar click Attributes In the All attributes for resource table locate the line item for the attribute you want to edit Administering the cluster from the Cluster Management Console 123 Administering resources In the Edit column click the edit button In the Edit Attribute dialog box specify
30. RIRIS Take offline Modify View state Display View dependencies KIARISIS List Value KIRAIA 808 RIRIS Probe Override Remove overrides Run an action Refresh info KISINAN ANAA AURAA AIS SSN SEN SYS QS NUS QS NUS Flush info QE SESPSE SESS SSE SESE SENRENSES ESESESYS VCS user privileges administration matrices 589 Administration matrices System operations hasys Table A 8 lists system operations and required privileges Table A 8 User privileges for system operations Add gt Delete Freeze and unfreeze S QS S788 Freeze and unfreeze E persistent Freeze and evacuate ae Display Y Y Y Y Start forcibly Modify View state NUS NUS NUS QUES List QS SNS N 8 8 878 Update license 590 VCS user privileges administration matrices Administration matrices Resource type operations hatype Table A 9 lists resource type operations and required privileges Table A 9 User privileges for resource type operations Add r 2 z Delete Display View resources Modify List SN SN SN S User operations hauser Table A 10 lists attribute operations and required privileges Table A 10 User privileges for user operations KIAINA OS Add Y Delete Vv Update Vv Vv Vv Y Displ
31. Temporary or Persistent from the menu The persistent option maintains the frozen state after a reboot if the user saves this change to the configuration To freeze a system from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Freeze System Click the system If necessary select the persistent and evacuate check boxes The evacuate option moves all service groups to a different system before the freeze operation takes place The persistent option maintains the frozen state after a reboot if the user saves this change to the configuration Click Apply Unfreezing a system Unfreeze a frozen system to enable service groups to come online on the system To unfreeze a system from Cluster Explorer 1 2 Click the Systems tab of the configuration tree In the configuration tree right click the system and click Unfreeze To unfreeze a system from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Unfreeze System Click the system Click Apply 220 Administering the cluster from Cluster Manager Java console Administering clusters Administering clusters Use the Java Console to specify the clusters you want to view from the console and to modify the VCS configuration The configuration details the parameters of the entire cluster Use Cluster Explorer or Command Center to open save a
32. The Java Console enables you to customize the log display of messages generated by the engine In the Logs dialog box you can set filter criteria to search and view messages and monitor and resolve alert messages To browse the logs for detailed views of each log message double click the event s description Use the arrows in the VCS Log details pop up window to navigate backward and forward through the message list T VCS Log details x Date Feb 21 2006 Error Time 3 55 18 AM Description V 16 10021 6036 veshp10 Mount vNFS_Mount1 monitor Input Argument validation Failed Customizing the log display From the Logs dialog box use the Edit Filters feature to customize the display of log messages To customize the display for VCS logs 1 Inthe VCS Logs tab click Edit Filters 2 Enter the filter criteria m Click the types of logs to appear on the message display Logtypes V Critical I Error IV Warning Notice IV Information Logs of All v Named All bd Logs From last hour s Search String IT Whole String m From the Logs of list select the category of log messages to display m From the Named menu select the name of the selected object or component To view all the messages for the selected category click All Administering the cluster from Cluster Manager Java console 229 Administering logs m Inthe Logs from last field enter the numerical value and select the time un
33. The count for the file _vrtsweb log The size of the file_command log which contains the logs related to administrative commands The count for the file_command log The size of the file _binary log which contains the binary representation of other log files The count for the file _binary log The size of the file _jvm log which contains JVM related measurements The file records the memory that is consumed by the JVM at various times The count for the file _jvm log Parameter protocol_client_size protocol_client_count protocol_server_size protocol_server_count out_size out_count err_size err_count webapps_ size webapps_count Administering Symantec Web Server 677 Configuring logging for VRTSweb Description The size of the file _protocol_client log which contains the communication sent and received between various utilities and the server The count for the file _protocol_client log The size of the file _protocol_server log which contains the communication sent and received by the server to various utilities The count for the file _protocol_server log The size of the file _out log which contains messages that are logged to the standard output stream of the JVM The count for the file _out log The size of the file _err log which contains messages that are logged to the standard error stream of the JVM including any stack traces The count for the file _err log The
34. To assign privileges to an administrator or operator Type the following command hauser addpriv user Adminstrator Operator group service groups To remove privileges from an administrator or operator Type the following command hauser delpriv user Adminstrator Operator group service groups To assign privileges to an OS user group Type the following command hauser addpriv usergroup AdminstratorsGroup OperatorsGroup group service groups To remove privileges from an OS user group Type the following command hauser delpriv usergroup AdminstratorsGroup OperatorsGroup group service groups Modifying a user Users in the category Cluster Guest cannot modify users To modify a user 1 Set the configuration to read write mode haconf makerw 2 Modify the user hauser update user 3 Enter anew password when prompted 4 Reset the configuration to read only haconf dump makero Administering the cluster from the command line 253 Managing VCS users from the command line Deleting a user You can delete a user from the VCS configuration To delete a user 1 Set the configuration to read write mode haconf makerw 2 For users with Administrator and Operator access remove their privileges hauser delpriv user Adminstrator Operator group service_groups 3 Delete the user from the list of registered users hauser delete user 4 Reset the config
35. When a system has only one interconnect link remaining to the cluster GAB can no longer reliably discriminate between loss of a system and loss of the network The reliability of the system s membership is considered at risk A special membership category takes effect in this situation called a jeopardy membership This provides the best possible split brain protection without membership arbitration and SCSI 3 capable devices When a system is placed in jeopardy membership status two actions occur m Service groups running on the system are placed in autodisabled state A service group in autodisabled state may failover on a resource or group fault but can not fail over on a system fault until the autodisabled flag is manually cleared by the administrator m VCS operates the system as a single system cluster Other systems in the cluster are partitioned off in a separate cluster membership 352 About communications membership and data protection in the cluster About cluster membership and data protection without I O fencing Daemon Down Node Alive DDNA Daemon Down Node Alive DDNA is a condition in which the VCS high availability daemon HAD on a node fails but the node is running When HAD fails the hashadow process tries to bring HAD up again If the hashadow process succeeds in bringing HAD up the system leaves the DDNA membership and joins the regular membership In a DDNA condition VCS does not have information about t
36. When the fencing driver is started it reads the physical disk names from the etc vxfentab file Using these physical disk names it determines the serial numbers of the coordinator disks and builds an in memory list of the drives The fencing driver verifies that any other systems in the cluster that are already up and running see the same coordinator disks The fencing driver examines GAB port B for membership information If no other systems are up and running it is the first system up and is considered the correct coordinator disk configuration When a new member joins it requests a coordinator disks configuration The system with the lowest LLT ID will respond with a list of the coordinator disk serial numbers If there is a match the new member joins the cluster If there is not a match vxfen enters an error state and the new member is not allowed to join This process ensures all systems communicate with the same coordinator disks The fencing driver determines if a possible preexisting split brain condition exists This is done by verifying that any system that has keys on the coordinator disks can also be seen in the current GAB membership If this verification fails the fencing driver prints a warning to the console and system log and does not start About communications membership and data protection in the cluster 343 About membership arbitration m If all verifications pass the fencing driver on each system registers ke
37. _ 16 6 15004 veshp10 hatriager Falled to send trigger for nfs_restart script doesn t exist H e Feb 20 2006 4 14 23 AM V 16 1 10447 Group CMC is online on system veshp10 Feb 20 2006 4 14 23M _ 16 1 10298 Resource CMC_ClusterConnector Owner unknown Group CMC is online on veshp10 VCS in Feb 20 2006 4 14 23AM V 16 10021 9003 veshp10 Process CMC_ClusterConnector monitor Setting cookie for proc bin sh fopt V fi Feb 20 2006 4 14 21 AM 16 2 13001 vcshp10 Resource CMC_ClusterConnector Output of the completed operation online ttyt Feb 20 2006 4 14 19 AM V 16 1 10301 Initiating Online of Resource CMC_ClusterConnector Owner unknown Group CMC on Syste Feb20 20064 14 19AM_ V 16 1 10298 Resource CMC_ClusterConfig Owner unknown Group CMC is online on veshp10 VCS initiat e Feb 20 2006 4 14 16AM_ V 16 1 10301 Initiating Online of Resource CMC_ClusterConfig Owner unknown Group CMC on System v Feb 20 2006 4 14 16 AM _ 16 1 10233 Clearing Restart attribute for group CMC on all nodes Feb 20 2006 4 14 16 AM _ 16 1 10493 Evaluating veshp71 as potential target node for group CMC Feb 20 2006 4 14 16 AM V 16 1 10493 Evaluating veshp10 as potential target node for group CMC Feb20 2006 4 13 58AM V 16 6 15004 veshp10 hatrigger Falled to send trigger For postoffline script doesn t exist Feb 20 2006 4 13 58 AM _ 16 6 15004 veshp10 hatrigger Failed to send trigger for
38. another system Child is online on another system If Child fails over to the system on which Parent was online Parent switches to another system If Child fails over to another system Parent restarts on original system If Child cannot fail over VCS takes the parent offline Parent fails over to a system where Child is not online If the only system available is where Child is online Parent is not brought online If no system is available Child remains online offline local Failover Child offline on the same system Child is offline on the same system If Child fails over to the system on which parent in not running parent continues running If child fails over to system on which parent is running parent switches to another system if available If no system is available for Child to fail over to Parent continues running Parent fails over to system on which Child is not online If no system is available Child remains online 418 The role of service group dependencies Service group dependency configurations Failover parent Parallel child With a failover parent and parallel child no hard dependencies are supported online local Instance of Instance of Parent fails over to Parent fails over to soft parallel Child Child is other system and other system and group on online on depends on Child depends on Child same system same instance there instance the
39. click Finish now to exit the wizard Administering the cluster from the Cluster Management Console 103 Administering service groups The parent resource Use the Parent Resource drop down menu to specify which resource must be brought online after the child resource The child resource Use Child Resource list box to select which resource should be brought online first and then click Add Link Modifying the system list of a service group Edit the list of systems that can host a service group You must have the role of cluster administrator or service group administrator to modify the system list of a service group To modify the system list of a service group 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to modify 2 Inthe Group Summary view in the Configuration task panel click Modify System List 3 Inthe System List Configuration dialog box specify the following details for the target systems and then click Finish The target systems are the systems on that can host the service group The target system names Find the system you want in the list under Select and Add Systems To quickly find a specific system click Filter type a few letters of the system name and then click Apply To select a system click the system name under Select and Add Systems and then click the right arrow button to move it under Selected Systems You can select one or more
40. clus cluster The variable cluster represents the cluster To declare the state of a cluster after a disaster haclus declare disconnet outage disaster replica clus cluster failover The variable cluster represents the remote cluster 532 Administering global clusters from the command line Administering clusters Changing the cluster name This section describes how to change the ClusterName in a global cluster configuration The instructions describe how to rename VCSPriCluster to VCSPriCluster2 in a two cluster configuration comprising clusters VCSPriCluster and VCSSecCluster configured with the global group AppGroup Before changing the cluster name make sure the cluster is not part of any ClusterList in the wide area Heartbeat agent and in global service groups To change the name of a cluster 1 2 3 Run the following commands from cluster VCSPriCluster hagrp offline ClusterService any hagrp modify AppGroup ClusterList delete VCSPriCluster haclus modify ClusterName vCSPriCluster2 hagrp modify AppGroup ClusterList add vCSPriCluster2 0 Run the following commands from cluster VCSSecCluster hagrp offline ClusterService any hagrp modify appgrp ClusterList delete vCSPriCluster hahb modify Icmp ClusterList delete vCSPriCluster haclus delete VCSPriCluster haclus add vCSPriCluster2 your ip address hahb modify Icmp ClusterList add VCSPriCluster2 hahb modify Icmp Arguments your ip
41. comprising nodes located at the secondary site and attached to the secondary storage Oracle is installed and configured on all nodes in the cluster Oracle data is located on shared disks within each RDC zone and is replicated across RDC zones to ensure data concurrency The Oracle service group is online on a system in the current primary zone and is configured to fail over in the cluster Clients Redirected Zone 1 Public Network Replicated Data Separate eparate Storage Storage In the event of a system or application failure VCS attempts to fail over the Oracle service group to another system within the same RDC zone However in the event that VCS fails to find a failover target node within the primary RDC zone VCS switches the service group to a node in the current secondary RDC zone zone 1 VCS also redirects clients once the application is online on the new location 538 Setting up replicated data clusters Setting up a replicated data cluster configuration Setting up a replicated data cluster configuration This section describes the steps for planning configuring testing and using the VCS RDC configuration to provide a robust and easy to manage disaster recovery protection for your applications It describes an example of converting a single instance Oracle database configured for local high availability in a VCS cluster
42. expanded view shows all cluster panels The collapsed view shows one cluster panel at a time as the panels scroll upward Operations enabled for the expanded view of cluster panels such as viewing menus are also enabled on the collapsed view after the panels stop scrolling To collapse the Cluster Monitor view On the View menu click Collapse or Click Collapse on the Cluster Monitor toolbar To expand the Cluster Monitor view On the View menu click Expand or Click Expand on the Cluster Monitor toolbar To pause a scrolling cluster panel Click the cluster panel or Click Stop on the Cluster Monitor toolbar 146 Administering the cluster from Cluster Manager Java console About Cluster Monitor Customizing the Cluster Manager display Customize the Cluster Manager to display objects according to your preference To customize the Cluster Manager display 1 From Cluster Monitor click Preferences on the File menu If you are using a Windows system proceed to step 2 Otherwise proceed to step 3 2 Inthe Look amp Feel tab for Windows systems T Preferences Look amp Feel Appearance saand File Edit Help 2I oO S cluster Bs group Native Windows look amp feel Java Metal look amp feel ee Cancel i Ar m Click Native Windows or Motif look amp feel or Java Metal look amp feel m Click Apply 3 Inthe Appearance tab Preferences BEES Look
43. fine tune various configuration parameters This tool is especially useful when evaluating complex multi node configurations It is convenient in that you can design a specific configuration without test clusters or changes to existing configurations You can also fine tune values for attributes governing the rules of failover such as Load and Capacity in a simulated environment VCS Simulator enables you to simulate various configurations and provides the information you need to make the right choices It also enables simulating global clusters See Predicting VCS Behavior Using VCS Simulator on page 539 286 Administering the cluster from the command line Administering simulated clusters from the command line Chapter 8 Configuring applications and resources in VCS About configuring resources and applications About VCS bundled agents Which agents should I use Configuring application service groups Configuring NFS service groups Testing resource failover using virtual fire drills 288 Configuring applications and resources in VCS About configuring resources and applications About configuring resources and applications Configuring resources and applications in VCS involves the following tasks Create a service group comprising all resources required for the application VCS provides configuration wizards to configure commonly used resources You can also use Cluster Manager Java Console the web based Clust
44. hares link IP1 NIC1 To unlink resources Type the following command hares unlink parent resource child resource Bringing resources online To bring a resource online Type the following command hares online resource sys system Taking resources offline To take a resource offline Type the following command hares offline ignoreparent resource sys system The option ignoreparent enables a resource to be taken offline even if its parent resources in the service group are online This option does not work if taking the resources offline violates the group dependency 273 274 Administering the cluster from the command line Administering resources To take a resource offline and propagate the command to its children Type the following command hares offprop ignoreparent resource sys system As in the above command the option ignoreparent enables a resource to be taken offline even if its parent resources in the service group are online This option does not work if taking the resources offline violates the group dependency Probing a resource To prompt an agent to monitor a resource on a system Type the following command hares probe resource sys system Though the command may return immediately the monitoring process may not be completed by the time the command returns Clearing a resource To clear a resource Type the following command Initiate a state
45. m Remote cluster states m System states 606 Cluster and system states Remote cluster states Remote cluster states In global clusters the health of the remote clusters is monitored and maintained by the wide area connector process The connector process uses heartbeats such as ICMP to monitor the state of remote clusters The state is then communicated to HAD which then uses the information to take appropriate action when required For example when a cluster is shut down gracefully the connector transitions its local cluster state to EXITING and notifies the remote clusters of the new state When the cluster exits and the remote connectors lose their TCP IP connection to it each remote connector transitions their view of the cluster to EXITED To enable wide area network heartbeats the wide area connector process must be up and running For wide area connectors to connect to remote clusters at least one heartbeat to the specified cluster must report the state as ALIVE There are three hearbeat states for remote clusters HBUNKNOWN HBALIVE and HBDEAD Cluster and system states 607 Remote cluster states The following table provides a list of VCS remote cluster states and their descriptions See Examples of system state transitions on page 611 Table C 1 VCS state definitions INIT The initial state of the cluster This is the default state BUILD The local cluster
46. system clus cluster localclus The option clus displays the attribute value on the cluster designated by the variable cluster the option localclus specifies the local cluster If the attribute has local scope you must specify the system name except when querying the attribute on the system from which you run the command To display the state of a resource across clusters hares state resource sys system clus cluster localclus The option clus displays the state of all resources on the specified cluster the option localclus specifies the local cluster Specifying a system displays resource state on a particular system To display resource information across clusters hares display resources attribute attributes group service groups type types sys systems clus cluster localclus The option clus lists all service groups on the cluster designated by the variable cluster the option localclus specifies the local cluster For a list of resources across clusters hares list conditionals clus cluster localclus The option clus lists all resources that meet the specified conditions in global service groups on a cluster as designated by the variable cluster To display usage for the resource command hares help modify list 524 Administering global clusters from the command line Global querying Querying systems To display system attribute values across clusters
47. systems at the SQL or database levels Host based replication products such as Veritas Volume Replicator maintain consistent storage at the logical volume level Storage or array based replication maintains consistent copies of data at the disk or RAID LUN level The following illustration shows a hybrid shared storage replicated data cluster in which different failover priorities are assigned to nodes according to particular service groups Figure 2 10 Shared storage replicated data cluster a Replication J Replicated data clusters can also be configured without the ability to fail over locally but this configuration is not recommended See Setting up a replicated data cluster configuration on page 538 44 About cluster topologies Cluster topologies and storage configurations Global cluster A global cluster links clusters at separate locations and enables wide area failover and disaster recovery Local clustering provides local failover for each site or building Campus and replicated cluster configurations offer protection against disasters affecting limited geographic regions Large scale disasters such as major floods hurricanes and earthquakes can cause outages for an entire city or region In such situations you can ensure data availability by migrating applications to sites located considerable distances apart Figure 2 11 Global cluster Public Cluster A Network
48. the production service group will fail over to the secondary site in the event of an actual failure disaster at the primary site If the fire drill service group does not come online review the VCS engine log to troubleshoot the issues so that corrective action can be taken as necessary in the production service group You can also view the fire drill log located at tmp fd servicegroup pid Caution Remember to take the fire drill offline once its functioning has been validated Failing to take the fire drill offline could cause failures in your environment For example if the application service group were to fail over to the node hosting the fire drill service group there would be resource conflicts resulting in both service groups faulting Scheduling a fire drill Schedule the fire drill for the service group by adding the file opt VRTSvcs bin fdsched to your crontab You can make fire drills highly available by adding the file to every node in the cluster The scheduler runs the command hagrp online firedrill_group any at periodic intervals 486 Connecting clusters Creating global clusters Setting up a fire drill Chapter 1 6 Administering global clusters from the Cluster Management Console About creating global service groups Administering global clusters Administering global service groups Administering global heartbeats 488 Administering global clusters from the Cluster Management Console Abo
49. to verify that the resource remains in its correct state Under normal circumstances the monitor is run every 60 seconds when a resource is online and every 300 seconds when a resource is expected to be offline m Clean Cleans up after a resource fails to come online fails to go offline or fails while in an ONLINE state The clean entry point is designed to clean up after an application The entry point ensures that the host system is returned to a valid state For example the clean function may remove shared memory segments or IPC resources that are left behind by a database 26 Introducing Veritas Cluster Server Logical components of VCS m Action Performs actions that can be completed in a short time and which are outside the scope of traditional activities such as online and offline Some agents have predefined action scripts that you can run by invoking the action entry point m Info Retrieves specific information for an online resource The retrieved information is stored in the resource attribute ResourceInfo This entry point is invoked periodically by the agent framework when the resource type attribute InfoInterval is set to a non zero value The InfoInterval attribute indicates the period after which the info entry point must be invoked For example the Mount agent may use this entry point to indicate the space available on the file system Agent classifications Bundled agents Bundled agents are packaged with V
50. 1 to identify a resource as a member of a path in the dependency tree to be taken offline on a specific system after a resource faults m Type and dimension boolean scalar m Default 0 Probed system use only Indicates whether the state of the resource has been determined by the agent by running the monitor entry point m Type and dimension boolean scalar m Default 0 Resourcelnfo system use only This attribute has three predefined keys State values are Valid Invalid or Stale Msg output of the info entry point captured on stdout by the agent framework TS timestamp indicating when the ResourcelInfo attribute was updated by the agent framework m Type and dimension string association m Default State Valid Msg Ts Table D 1 VCS attributes 619 Resource attributes Resource attributes ResourceOwner user defined Used for VCS email notification and logging VCS sends email notification to the person designated in this attribute when an event occurs related to the resource VCS also logs the owner name when an event occurs m Type and dimension string scalar m Default If ResourceOwner is not specified in main cf the default value is unknown Signaled system use only Indicates whether a resource has been traversed Used when bringing a service group online or taking it offline m Type and dimension integer association m Default Not applicable
51. 3 are in the regular membership and System2 in a jeopardy membership Service groups on System2 are autodisabled All normal cluster operations continue including normal failover of service groups due to resource fault Cluster interconnect link failure followed by system failure In this example the link to System2 fails and System2 is put in the jeopardy membership Subsequently System2 fails due to a power fault Public Network System0O System1 System3 System2 Regular membership 0 1 3 with known previous jeopardy membership for System2 Systems 0 1 and 3 recognize that System2 has faulted The cluster is reformed Systems 0 1 and 3 are ina regular membership When System2 went into jeopardy membership service groups running on System2 were autodisabled Even though the system is now completely failed no other system can assume ownership of these service groups unless the system administrator manually clears the AutoDisabled flag on the service groups that were running on System2 However after the flag is cleared these service groups can be manually brought online on other systems in the cluster About communications membership and data protection in the cluster 355 Examples of VCS operation without I O fencing All high priority cluster interconnect links fail In this example all high priority links to System2 fail This can occur two ways Public Network Sy
52. Agent attributes for non global zone support The following attributes define the zone where the application runs m ContainerType A resource type static attribute which indicates that the application runs in a Solaris zone For applications running in zones set this attribute to Zone m ContainerName A resource attribute which you set to the name of the zone When you set the ContainerName VCS runs the entry points for the resource in the specified zone About the Zone agent Configuring VCS in non global zones 593 Configuring VCS in zones The Zone agent monitors zones brings zones online and takes them offline For more information about the agent see the Veritas Cluster Server Bundled Agents Reference Guide The agent creates a user account with group administrative privileges to enable inter zone communication if the user account doesn t exist In secure clusters it also renews the authentication certificate before the certificate expires Configuring VCS in zones Configuring VCS in zones involves the following tasks First Review the prerequisites See Prerequisites for configuring VCS in zones on page 594 Second Decide on the location of the zone root See Setting the zone root path on page 595 A zone root is the topmost directory in a section of the file system hierarchy in which the non global zone is configured The zone root may be on shared storage or on local storage If zon
53. Agents Reference Guide m Make sure that the applications are not configured in any other service group m Verify the directories on which the applications depend reside on shared disks and are mounted m Ifthe application runs in a zone make sure the zone is started Run the wizard on a system where the zone is up m Verify the mount points on which the applications depend are not configured in any other service group m Verify the virtual IP addresses on which applications depend are up Verify the IP addresses are not configured in any other service group m Make sure the executable files required to start stop monitor and clean optional the application reside on all nodes participating in the service group If the application runs in a non global zone the executable files must reside in the non global zone m StartProgram The executable created locally on each node that starts the application m StopProgram The executable created locally on each node that stops the application m CleanProgram The executable created locally on each node that forcibly stops the application m You can monitor the application in the following ways m Specify the program that will monitor the application m Specify a list of processes to be monitored and cleaned m Specify a list of pid files that contain the process ID of the processes to be monitored and cleaned These files are application generated files Each PID file contains
54. Cluster Management Console 1 Create a service group called ClusterService 2 Add a resource of type NIC to the service group Name the resource csgnic Set the value of the Device attribute to the name of the NIC Configure other attributes if desired 3 Add a resource of type IP to the service group Name the resource webip Configure the following attributes for the resource m Address A virtual IP address assigned to the Cluster Management Console The Cluster Management Console is accessed using this IP address m Device The name of the public network card on the system from which the Cluster Management Console is intended to run The device is defined as a local attribute for each system in the cluster m NetMask The subnet to which the virtual IP address belongs m Critical Set this attribute to 1 indicating True to make webip a critical resource 4 Link the NIC and IP resources such that the IP resource depends on the NIC resource 5 Add a resource of type VRTSWebApp to the service group Name the resource VCSweb Configure the following attributes for the resource m Appname Set to cmc m InstallDir Set to opt VRTSweb VERITAS Administering the cluster from the Cluster Management Console 73 Configuring the Cluster Management Console manually m TimeForOnline Set to 5 m Critical Set to 0 indicating False 6 Link the VCSweb and webip resources making VCSweb the parent resource 7 Enable both resources
55. For example if you delete an IP address after starting the wizard and click Discover IP the wizard displays the deleted IP addresses in the Virtual IP Address list 314 Configuring applications and resources in VCS Configuring NFS service groups m For each system specify the associated ethernet If the ethernet card for a system does not appear in the list click Discover NIC to discover NICs m Click Next 9 Review your configuration and change resource names if desired VCS NFS Configuration Wizard Service Group Summary Review the configuration and make any necessary changes to the resource names ki NFSGroup Attributes NFSRestart_NFSGroup INFS Group NFS_NFSGroup SystemList vcssun84 vessun85 Share_nts autoStartList vessun84 vessun85 IP_10_212_88_29 NIC_bgeo Mount_nfs E More information lt Back Finish Cancel The left pane lists the configured resources Click on a resource to view its attributes and their configured values in the Attributes box To edit a resource name select the resource name and click on it Press Enter after editing each name Note that when modifying service groups you can change names of newly created resources only which appear in black Click Finish The wizard starts running commands to create or modify the service group 10 Onthe Completing the NFS Configuration Wizard dialog box select the check box to bri
56. G2 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 40 group G3 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 30 396 Controlling VCS behavior Sample configurations depicting workload management group G4 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 10 group G5 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 50 group G6 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 30 group G7 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 20 group G8 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 40 Controlling VCS behavior 397 Sample configurations depicting workload management AutoStart operation In this config
57. I O fencing ensures data integrity Controlling VCS behavior 391 Service group workload management Service group workload management Workload management is a load balancing mechanism that determines which system hosts an application during startup or after an application or server fault Service Group Workload Management provides tools for making intelligent decisions about startup and failover locations based on system capacity and resource availability Enabling service group workload management The service group attribute FailOverPolicy governs how VCS calculates the target system for failover Set FailOverPolicy to Load to enable service group workload management See Controlling VCS behavior at the resource level on page 373 About system capacity and service group load The Load and Capacity construct allows the administrator to define a fixed amount of resources a server provides Capacity and a fixed amount of resources a specific service group is expected to utilize Load The system attribute Capacity sets a fixed load handling capacity for servers Define this attribute based on system requirements The service group attribute Load sets a fixed demand for service groups Define this attribute based on application requirements When a service group is brought online its load is subtracted from the system s capacity to determine available capacity VCS maintains this info in the attribute AvailableCapac
58. If the ClusterService group is configured but the Notifier resource is not configured the Notifier resource will be created and added to the ClusterService group To set up event notification using the Notifier wizard 1 From Cluster Explorer click Notifier Wizard on the Tools menu or On the Cluster Explorer toolbar click Launch Notifier Resource Configuration Wizard 2 Click Next 3 Inthe dialog box Notifier Resource Configuration Wizard Service Group Configuration for Notifier The wizard will create a Notifier resource and add it to the ClusterService group Resource Name ntfr System List for ClusterService group Available Systems Systems For Service Group System name Startup Priority lt Back cancel Hep Enter the name of the resource For example ntfr m Click the target systems in the Available Systems box m Click the right arrow to move the systems to the Systems for Service Group table To remove a system from the table click the system and click the left arrow m Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every time HAD is started m The priority number starting with 0 is assigned to indicate the order of systems on which the service group will start in case of a failover If 226 Administering the cluster
59. Load Load 30 group G4 SystemList LargeServerl1 MedServer2 SystemZones LargeServerl1 0 MedServer2 1 AutoStartPolicy Load AutoStartList FailOverPolicy Load Load 20 LargeServer2 0 MedServerl1 MedServerl 1 Semaphores 5 LargeServer2 0 LargeServer2 Processors 6 MedServerl1 MedServerl 1 Semaphores 5 LargeServer2 0 LargeServer2 LargeServer2 0 LargeServer2 Processors 6 MedServerl1 MedServerl 1 MedServerl MedServer2 MedServerl1 MedServerl 1 MedServerl MedServer2 Controlling VCS behavior 401 Sample configurations depicting workload management AutoStart operation In this configuration the AutoStart sequence resembles G1 LargeServer1 G2 LargeServer2 G3 MedServer1 G4 MedServer2 All groups begin a probe sequence when the cluster starts Groups G1 and G2 have an AutoStartList of LargeServer1 and LargeServer2 When these groups probe they are queued to go online on one of these servers based on highest AvailableCapacity value If G1 probes first it chooses LargeServer1 because LargeServer1 and LargeServer2 both have an AvailableCapacity of 200 but LargeServerl is lexically first Groups G3 and G4 use the same algorithm to determine their servers Normal operation The configuration resembles Server AvailableCapacity CurrentLimits Online Groups LargeServer1 100 ShrMemSeg 10 G1 Semapho
60. LoadWarningLevel 80 sets the warning level to 80 percent The value of this attribute can be set from 1 to 100 If set to 1 system load must equal 1 percent of system capacity to begin incrementing the LoadTimeCounter If set to 100 system load must equal system capacity to increment the LoadTimeCounter m Type and dimension integer scalar m Default 80 NoAutoDisable system use only When set to 0 this attribute autodisables service groups when the VCS engine is taken down Groups remain autodisabled until the engine is brought up regular membership Setting this attribute to 1 bypasses the autodisable feature m Type and dimension boolean scalar m Default 0 Nodeld system use only System node identification specified in etc Ilttab m Type and dimension integer scalar m Default Not applicable OnGrpCnt system use only Number of groups that are online or about to go online ona system m Type and dimension integer scalar m Default Not applicable 646 VCS attributes System attributes Table D 4 System attributes ShutdownTimeout Determines whether to treat system reboot as a fault for service user defined groups running on the system On many systems when a reboot occurs the processes are stopped first then the system goes down When the VCS engine is stopped service groups that include the failed system in their SystemList attributes are autodisabled However if the
61. Management Console 81 Overview of the Cluster Management Console Figure 5 7 Filter panel showing criteria selections By Consolidated State M g Online M amp offline M EA Partial M G Faulted M gh Unknovm M Global groups Apply Reset About the Alerts panel The Alerts panel is a special panel in the task pane that provides current counts of active alerts categorized by severity The Alerts panel is in the task pane of every view in the Cluster Management Console Table 5 1 Alert severity icons ga Critical S Eror A Warming i Information 82 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console Figure 5 8 Alerts panel fax Alerts gt oO 2 0 ke The number below each alert indicates the current number of alerts for that particular alert severity This number is also a hyperlink to a filtered list of those alerts only For example to view critical alerts click the number below the critical alert icon This link takes you to the Notifications Logs view which contains a table that lists the active informational alerts Collapsing and expanding panels in the task pane Clicking the up arrow in the center top of each task or filter panel collapses the panel to show only the heading When a panel is collapsed the up arrow changes to a down arrow When you click the down arrow the panel expands to show the panel contents About the status
62. O 11 12 13 14 Create a file system on shared storage for the zone root The file system that is to contain the zone root may be in the disk group as the file system containing the shared data On the system where the file system is mounted create the zone with the zonecfg command Set the zonepath parameter to specify a location for the zone root Create a loop back file system in the zone Set a virtual IP address of the system to be the IP address of the zone and define the device name for the NIC associated with the IP address Exit the zonecfg configuration Create the actual zone root directory Set permissions for the zone root directory Repeat the previous zone creation steps on each other system in the service group s SystemList Mount the file system containing shared storage on one of the systems that share the storage Install the non global zone using zoneadm List the zones including the global zone On the same system mount the file system containing the application s data on the shared storage Use the mount point used in the global zone Boot the zone 598 Configuring VCS in non global zones Configuring VCS in zones 15 Make sure the zone created from the first system is in the installed state on all other systems in the service group s System List To create zone root on shared disks with data using direct mount file system 1 10 11 12 13 14 15 Create a file sy
63. Operations task panel click Switch 3 Inthe Switch Service Group dialog box specify the following details for the task and then click OK m The target cluster Select the cluster to which you want to switch the remote service group from the Select the cluster you want to switch this group to drop down menu The cluster choices are populated using the cluster list for the remote service group m The target system Administering global clusters from the Cluster Management Console 499 Administering global heartbeats Select the system to which you want to switch the remote service group from the Select the system you want to online this group on drop down menu The system choices are populated using the system list for the remote service group The Anywhere option causes this task to try every other system in the list until the service group is successfully brought online Administering global heartbeats Use cluster heartbeats to monitor the health of clusters configured into a global cluster environment A heartbeat is a list of clusters to which the Cluster Management Console listens The console listens for repetitive signals that confirm that communications with each cluster are valid and that each cluster is in an online state Global clustering requires a minimum of one heartbeat between clusters you can add additional heartbeats as a precautionary measure Use the Cluster Heartbeats view to manage heartbeats The Cluster Heartbe
64. Recent Critical Error Logs No Logs are available For global clusters this view displays the state of the remote clusters For global groups this view shows the status of the groups on both local and remote clusters To access the Status view 1 From Cluster Explorer click an object in the configuration tree 2 Inthe view panel click the Status tab Properties view Administering the cluster from Cluster Manager Java console About Cluster Explorer The Properties View displays the attributes of VCS objects These attributes describe the scope and parameters of a cluster and its components 2 VCSHP10_71_0214a W Properties iew resource1_ng2 G Key Attributes Beles Help Attribute Yalue Edit Name Z4 MonitorTimeStats yeshp7i Avg0 Zi veshp10 en TS Last Online veshp10 ComputeStats false Z4 ResourceInfo State Valid z4 Msg 1S Critical i true Z4 Type Specific Attributes Attribute value Edit Help PathName jtmp saa txt Zi 2 Overridden Attributes No Attributes are Overridden 5 ot ES z To view information on an attribute click the attribute name or the icon in the Help column of the table See VCS attributes on page 613 By default this view displays key attributes of the object selected in the configuration tree The Properties View for a resource displays key attributes of the resource and attributes spec
65. Save to exit these options and save your isolations 502 Administering global clusters from the Cluster Management Console Administering global heartbeats Chapter 1 Administering global clusters from Cluster Manager Java console About global clusters Adding a remote cluster Deleting a remote cluster Administering global service groups Administering global heartbeats 504 Administering global clusters from Cluster Manager Java console About global clusters About global clusters The process of creating a global cluster environment involves creating a common service group for specified clusters making sure all the service groups are capable of being brought online in the specified clusters connecting the standalone clusters and converting the service group that is common to all the clusters to a global service group Use the console to add and delete remote clusters create global service groups and manage cluster heartbeats Creating a global cluster environment requires the following conditions m All service groups are properly configured and able to come online m The service group that will serve as the global group has the same unique name across all applicable clusters m The clusters must use the same version of VCS m The clusters must use the same operating system m The clusters are standalone and do not already belong to a global cluster environment Through the Java Console you can simulate t
66. Serializing the failover choice allows complete load based control and adds less than one second to the total failover time Following the first failure the configuration now resembles Server AvailableCapacity Online Groups Server1 40 G1 G6 and G4 Server2 20 G2 and G8 Server3 0 G3 G7 and G5 In this configuration Server3 fires the loadwarning trigger to notify that the server is overloaded An administrator can then switch group G7 to Server1 to balance the load across groups G1 and G3 When Server4 is repaired it rejoins the cluster with an AvailableCapacity value of 100 making it the most eligible target for a failover group Cascading failure scenario If Server3 fails before Server4 can be repaired group G3 chooses Server1 group G5 chooses Server2 and group G7 chooses Server1 This results in the following configuration Server AvailableCapacity Online Groups Server1 10 G1 G6 G4 G3 and G7 Server2 30 G2 G8 and G5 Server1 fires the loadwarning trigger to notify that it is overloaded Controlling VCS behavior 399 Sample configurations depicting workload management Sample configuration Complex four node cluster The cluster in this example has two large enterprise servers LargeServer1 and LargeServer2 and two medium sized servers MedServer1 and MedServer2 It has four service groups G1 through G4 with various loads and prerequisites Groups G1 and G2 are database applications with sp
67. Services manually See Enabling and disabling Security Services on page 280 Components for administering VCS VCS provides the following components to administer clusters Cluster Management Console A Web based graphical user interface for monitoring and administering the cluster m Install the Cluster Management Console on cluster nodes to manage a single cluster See the Veritas Cluster Server Centralized Management Guide fore more information m Install the Cluster Management Console on a management server outside the cluster to manage multiple clusters See Administering the cluster from the Cluster Management Console on page 69 Cluster Manager Java console A cross platform Java based graphical user interface that provides complete administration capabilities for your cluster The console runs on any system inside or outside the cluster on any operating system that supports Java See Administering the cluster from Cluster Manager Java console on page 135 VCS command line interface CLI The VCS command line interface provides a comprehensive set of commands for managing and administering the cluster See Administering the cluster from the command line on page 233 30 Introducing Veritas Cluster Server Putting the pieces together Putting the pieces together In this example a two node cluster exports an NFS file system to clients Both nodes are connected to shared storage wh
68. The attributes ClusterList AutoFailOver and Parallel are mismatched for the same global service group on different clusters VCS event notification 439 VCS events and traps SNMP specific files VCS includes two SNMP specific files vcs mib and vcs_trapd which are created in etc VRTSvcs snmp The file vcs mib is the textual MIB for built in traps that are supported by VCS Load this MIB into your SNMP console to add it to the list of recognized traps The file vcs_trapd is specific to the HP OpenView Network Node Manager NNM SNMP console The file includes sample events configured for the built in SNMP traps supported by VCS To merge these events with those configured for SNMP traps xnmevents merge vcs_trapd When you merge events the SNMP traps sent by VCS by way of notifier are displayed in the HP OpenView NNM SNMP console Note For more information on xnmevents see the HP OpenView documentation 440 VCS event notification VCS events and traps Trap variables in VCS MIB Traps sent by VCS are reversible to SNMPv2 after an SNMPv2 gt SNMPv1 conversion For reversible translations between SNMPv1 and SNMPv2 trap PDUs the second last ID of the SNMP trap OID must be zero This ensures that once you make a forward translation SNMPv2 trap gt SNMPv1 RFC 2576 Section 3 2 the reverse translation SNMPv1 trap gt SNMPv2 trap RFC 2576 Section 3 1 is accurate The VCS notifier follows this guideline
69. These variables are set for VCS when the hastart command is run To set a variable use the syntax appropriate for the shell in which VCS starts For example if you use the bash shell define variables as export VCS_GAB TIMEOUT 18000 export umask 011 Note By default files generated by VCS inherit the system s umask settings To override the system s umask settings for files generated by VCS define a umask value in the vcesnv file 58 VCS configuration concepts VCS environment variables Section Administration Putting VCS to work m Chapter 4 About the VCS user privilege model on page 61 m Chapter 5 Administering the cluster from the Cluster Management Console on page 69 m Chapter 6 Administering the cluster from Cluster Manager Java console on page 135 m Chapter 7 Administering the cluster from the command line on page 233 m Chapter 8 Configuring applications and resources in VCS on page 287 m Chapter 9 Predicting VCS behavior using VCS Simulator on page 317 60 Administration Putting VCS to work Chapter About the VCS user privilege model m About VCS user privileges and roles How administrators assign roles to users m User privileges for OS user groups in secure clusters m About VCS privileges for users with multiple roles 62 About the VCS user privilege model About VCS user privileges and roles About VCS user privileges and role
70. This could occur if a system is down for maintenance when the cluster comes up Caution It is not recommended to seed the cluster manually unless the administrator is aware of the risks and implications of the command Before manually seeding the cluster check that systems that will join the cluster are able to send and receive heartbeats to each other Confirm there is no possibility of a network partition condition in the cluster To manually seed the cluster type the following command sbin gabconfig c x Note there is no declaration of the number of systems in the cluster with a manual seed This command will seed all systems in communication with the system where the command is run See Seeding and I O Fencing on page 566 340 About communications membership and data protection in the cluster About cluster membership Ongoing cluster membership Once the cluster is up and running a system remains an active member of the cluster as long as peer systems receive a heartbeat signal from that system over the cluster interconnect A change in cluster membership is determined as follows When LLT on a system no longer receives heartbeat messages from a system on any of the configured LLT interfaces for a predefined time LLT informs GAB of the heartbeat loss from that specific system This predefined time is 16 seconds by default but can be configured It is set with the set timer peerinact command as described
71. This section describes how to enable and disable Security Services Do not edit the VCS configuration file main cf to enable or disable VxSS You must set up a root broker before enabling security services on a cluster See the Veritas Cluster Server Installation Guide for instructions on setting up a root broker To enable Symantec Product Authentication Services on a Veritas Cluster Server cluster 1 Verify you have a root broker configured See the Veritas Cluster Server Installation Guide for instructions Start the installvcs program with the security option installvcs security Review the output as the installer displays the directory where the logs are created Enter 1 to enable the Authentication Service on the cluster 1 Enable Veritas Security Services on a VCS Cluster 2 Disable Veritas Security Services on a VCS Cluster 3 Install Veritas Security Services Root Broker Select the Security option you would like to perform 1 3 q 1 If Veritas Cluster Server is not configured in the system from where you started the installvcs program enter the name of a node in the cluster that you want to enable the Authentication Service Enter the name of one system in the VCS Cluster that you would like to enable Veritas Security Services north The installer proceeds to verify communication with the node in the cluster Review the output as the installer verifies whether Veritas Cluster Server configuration files exi
72. Trigger Resource Offline Admin_Wait resadminwait Trigger Controlling VCS behavior 381 Controlling VCS behavior at the resource level VCS behavior after a resource is declared faulted After a resource is declared faulted VCS fires the resfault trigger and examines the value of the FaultPropagation attribute If FaultPropagation is set to 0 VCS does not take other resources offline and changes the group state to OFFLINE FAULTED or PARTIAL FAULTED The service group does not fail over If FaultPropagation is set to 1 VCS takes all resources in the dependent path of the faulted resource offline up to the top of the tree VCS then examines if any resource in the dependent path is critical If no resources are critical the service group is left in its OFFLINE FAULTED or PARTIAL FAULTED state If a resource in the path is critical VCS takes the all resources in the service group offline in preparation of a failover If the AutoFailOver attribute is set to 0 the service group is not failed over it remains in a faulted state If AutoFailOver is set to 1 VCS examines if any systems in the service group s SystemList are possible candidates for failover If no suitable systems exist the group remains faulted and VCS calls the nofailover trigger If eligible systems are available VCS examines the FailOverPolicy to determine the most suitable system to which to fail over the service group 382 Controlling VCS behavior Cont
73. Type and dimension integer scalar m Default Not applicable Failover system use only Indicates service group is in the process of failing over m Type and dimension boolean scalar m Default Not applicable FailOverPolicy user defined Sets the policy VCS uses to determine which system a group fails over to if multiple systems exist This attribute can take the following values Priority The system defined as the lowest priority in the SystemList attribute is chosen Load The system defined with the least value in the system s Load attribute is chosen RoundRobin Systems are chosen according to how many active service groups they are hosting The system with the least number of active service groups is chosen first m Type and dimension string scalar m Default Priority 632 VCS attributes Service group attributes Table D 3 Service group attributes FaultPropagation user defined Specifies if VCS should propagate the fault up to parent resources and take the entire service group offline when a resource faults The value 1 indicates that when a resource faults VCS fails over the service group if the group s AutoFailOver attribute is set to 1 If The value 0 indicates that when a resource faults VCS does not take other resources offline regardless of the value of the Critical attribute The service group does not fail over on resource fault m Type and dimension boolean
74. View and Properties View m Click a cluster in the configuration tree to access the Service Group view System Connectivity view and Remote Cluster Status View for global clusters only m Click a service group in the configuration tree to access the Resource view To create a tear off view On the View menu click Tear Off and click the appropriate view from the menu or Right click the object in the configuration tree click View and click the appropriate view from the menu 152 Administering the cluster from Cluster Manager Java console About Cluster Explorer Status view The Status View summarizes the state of the object selected in the configuration tree Use this view to monitor the overall status of a cluster system service group resource type and resource For example if a service group is selected in the configuration tree the Status View displays the state of the service group and its resources on member systems It also displays the last five critical or error logs Point to an icon in the status table to open a ScreenTip about the relevant VCS object T VCSHP10_71_0214a of xi Status View ClusterService 6 Group Status on Member Systems Member Systems State AutoDisabled Disabled veshp10 Online veshp71 Offline G Resource Status Resource Name Resource Type Status vcSweb VRTSWebApp Online on veshp10 amp csgnic NIC Online on veshp10 webip IP Online on veshp10
75. You can conduct fire drills only on regular VxVM volumes volume sets vset are not supported Configuring the fire drill service group Use the RVG Secondary Fire Drill Wizard to set up the fire drill configuration The wizard performs the following specific tasks m Prepares all data volumes with FMR 4 0 technology which enables space optimized snapshots m Creates a Cache object to store changed blocks during the fire drill which minimizes disk space and disk spindles required to perform the fire drill m Configures a VCS service group that resembles the real application group m Schedules the fire drill and the notification of results The fire drill wizard sets up the first RVG in an application If the application has more than one RVG you must create space optimized snapshots and configure VCS manually using the first RVG as reference 484 Connecting clusters Creating global clusters Setting up a fire drill To run the wizard 1 Start the RVG Secondary Fire Drill wizard on the VVR secondary site where the service group is not online opt VRTSvcs bin fdsetup Read the information on the Welcome screen and press the Enter key The wizard identifies the global service groups Enter the name of the service group for the fire drill Review the list of volumes in disk group that could be used for a space optimized snapshot Enter the volumes to be selected for the snapshot Typically all volumes used by
76. address clus vcsPriCluster2 hagrp modify AppGroup ClusterList add vCSPriCluster2 0 hagrp online ClusterService any Run the following command from the cluster renamed to VCSPriCluster2 hagrp online ClusterService any Administering global clusters from the command line 533 Administering heartbeats Administering heartbeats To create a heartbeat hahb add heartbeat For example type the following command to add a new IcmpS heartbeat This represents a heartbeat sent from the local cluster and immediately forks off the specified agent process on the local cluster hahb add IcmpS To modify a heartbeat hahb modify heartbeat attribute value clus cluster If the attribute is local that is it has a separate value for each remote cluster in the ClusterList attribute the option clus cluster must be specified Use delete keys to clear the value of any list attributes For example type the following command to modify the ClusterList attribute and specify targets phoenix and houston for the newly created heartbeat hahb modify ICMP ClusterList phoenix houston To modify the Arguments attribute for target phoenix hahb modify ICMP Arguments phoenix veritas com clus phoenix To delete a heartbeat hahb delete heartbeat To change the scope of an attribute to cluster specific hahb local heartbeat attribute For example type the following command to change the scope of the attribute AYAInterval f
77. also known as the IP alias should not be confused with the base IP address which is the IP address that corresponds to the host name of a system A accessibility assistive technology support 681 overview 679 ActionTimeout attribute 620 ActiveCount attribute 628 AdministratorGroups attribute for clusters 648 for service groups 628 Administrators attribute for clusters 648 for service groups 628 agent log format 564 location 564 AgentClass attribute 620 AgentDirectory attribute 620 AgentFailedOn attribute 620 AgentFile attribute 620 AgentPriority attribute 621 AgentReplyTimeout attribute 621 agents classifications of 26 DNS 466 entry points 25 framework 26 Heartbeat 465 impact on performance 547 RVG 467 RVGPrimary 467 RVGSnapshot 467 starting from command line 267 stopping from command line 267 Wide Area Heartbeat 465 AgentStartTimeout attribute 621 AgentState attribute 655 AgentStopped attribute 641 aggregate notifications monitoring 443 alerts deleting from Java Console 230 monitoring from Java Console 230 ArgList attribute 621 ArgListValues attribute 614 assistive technology support 681 association attribute dimension 52 asymmetric configuration 34 AttrChangedTimeout attribute 621 attribute dimensions association 52 keylist 52 scalar 52 vector 52 attribute types boolean 51 integer 51 string 51 attributes about 51 editing from Cluster Management Console 130 editing from J
78. also redirects clients once the application is online on the new location Connecting clusters Creating global clusters 463 VCS global clusters The building blocks VCS global clusters The building blocks VCS extends clustering concepts to wide area high availability and disaster recovery with the following m Visualization of remote cluster objects m Global service groups m Global cluster management m Serialization The authority attribute m Resiliency and Right of way m VCS framework m The steward process Split brain in two cluster global clusters Visualization of remote cluster objects VCS enables you to visualize remote cluster objects using the VCS command line the Java Console and the Web Console You can define remote clusters in your configuration file main cf The Remote Cluster Configuration wizard provides an easy interface to do so The wizard updates the main cf files of all connected clusters with the required configuration changes See Adding a remote cluster on page 505 Global service groups A global service group is a regular VCS group with additional properties to enable wide area failover The global service group attribute ClusterList defines the list of clusters to which the group can fail over The service group must be configured on all participating clusters and must have the same name on each cluster The Global Group Configuration wizard provides an easy interface to configure g
79. amp Feel Appearance Sound Fie Edit Help S Uo cluster E ga group Icon Small X JZ Show Tooltips I Remove Cluster Manager colors ex Cancel i Click the color applies to Java Metal look amp feel Click an icon size Select the Show Tooltips check box to enable ToolTips Select the Remove Cluster Manager colors check box to alter the standard color scheme m Click Apply Administering the cluster from Cluster Manager Java console 147 About Cluster Monitor In the Sound tab T Preferences BEE Look amp Feel Appearance Sound J Enable Sound J Delete Group Add System Delete System fa Group Add Resource Delete Resource Online i offline Faulted Note This tab requires a properly configured sound card m Select the Enable Sound check box to associate sound with specific events Click an event from the Events configuration tree Click a sound from the Sounds list box To test the selected sound click Play Click Apply Repeat these steps to enable sound for other events After you have made your final selection click OK 148 Administering the cluster from Cluster Manager Java console About Cluster Explorer About Cluster Explorer Cluster Explorer is the main window for cluster administration From this window you can view the status of VCS objects and perform various operations VCSHP10_71_0214a Cluster Explor
80. and click the button with the left arrow icon m To change a system s priority in the service group s system list select the system in the Systems in Priority Order box and click the buttons with the up and down arrow icons The system at the top of the list has the highest priority while the system at the bottom of the list has the lowest priority m Click Next 302 Configuring applications and resources in VCS Configuring application service groups If the cluster does not have zones configured proceed to step 7 on page 296 If the cluster has zones configured proceed to the next step 5 Configure the zone in which the application runs VCS Application Configuration Wizard Zone Configuration Configure Zone resource if required by applications Solaris Zones koe Jo _Discover Zone C Are applications accessible over network System Name Ethernet Name vessundt erid Iz Discover NIC E More information lt Back Next gt cancer m Select the zone in which the application runs If the zone does not appear in the list click Discover Zone m Ifthe application depends on an IP address select the Are applications accessible over the network check box Select an ethernet card for the IP address associated with the application on each system Click Discover NIC if required m Click Next If you chose a global zone for the application proceed to step 7 on page 296 If you c
81. and tasks for administering a single resource To navigate to the Group Summary view 1 2 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Groups 116 Administering the cluster from the Cluster Management Console Administering resources In the Cluster Groups view in the Service Groups Listing table click a linked service group name In the Group Summary view the Configuration task panel contains several resource related tasks To navigate to the Group Resources view 1 2 3 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Groups In the Cluster Groups view in the Service Groups Listing table click a linked service group name In the Group Summary view on the secondary tab bar click Resources The Group Resources view contains a graphical representation of the resources in the service group and any dependencies that exist among them Adding a resource to a service group Add and configure a required resource as a member of a service group You must have the role of cluster administrator or service group administrator to adda resource to a service group To add a resource to a service group 1 In the Group Summary view in the Configuration task panel click Add Modify Resource In the Resource Creation dialog box specify the following details about the resource and the service group and then click OK m The
82. appropriate phrase or symbol between the search item and value m Click the appropriate value for the specified query Certain queries allow the user to enter specific filter information Click System click Online Group Count click lt and type the required value in the blank field or Click Resource click provide attribute name and type in the name of an attribute click or contains and type the appropriate value of the attribute in the blank field For example click Resource click provide attribute name and type in pathname click contains and type c temp in the blank field m To use additional queries click as many times as necessary to select the appropriate options Click to reduce the number of queries m Click AND or OR for each filter selection m Click Search The results appear in tabular format at the bottom of the dialog box To search a new item click Reset to reset the dialog box to its original blank state Administering the cluster from Cluster Manager Java console 225 Setting up VCS event notification using the Notifier wizard Setting up VCS event notification using the Notifier wizard The information presented here assumes that you need to create both the ClusterService group and the Notifier resource If the ClusterService group exists but the Notifier resource is configured under another group you can modify the attributes of the existing Notifier resource and system list for that group
83. are active online or waiting to go online When the number drops to zero the service group is considered offline m Type and dimension integer scalar m Default Not applicable AdministratorGroups user defined List of operating system user account groups that have administrative privileges on the service group This attribute is valid only in secure clusters m Type and dimension string keylist m Default Administrators user defined List of VCS users with privileges to administer the group Note A Group Administrator can perform all operations related to a specific service group but cannot perform generic cluster operations See About the VCS user privilege model on page 61 m Type and dimension string keylist m Default Authority user defined Indicates whether or not the local cluster is allowed to bring the service group online If set to 0 it is not if set to 1 it is Only one cluster can have this attribute set to 1 for a specific global group See Serialization The authority attribute on page 465 m Type and dimension integer scalar m Default 0 AutoDisabled system use only Indicates that VCS does not know the status of a service group or specified system for parallel service groups This could occur because the group is not probed on specified system for parallel groups in the SystemList attribute Or the VCS engine is not running on a node de
84. are the same on both sides the Mount resources will mount the same block devices and the same Oracle instance will start at the secondary in case of a failover Connecting clusters Creating global clusters 477 Setting up a global cluster Linking clusters Once the VCS and VVR infrastructure has been set up at both sites you must link the two clusters The Remote Cluster Configuration wizard provides an easy interface to link clusters Before linking clusters verify the virtual IP address for the ClusterAddress attribute for each cluster is set Use the same IP address as the one assigned to the IP resource in the ClusterService group If you are adding a stand alone cluster to an existing global cluster environment run the wizard from a cluster in the global cluster environment Otherwise run the wizard from any cluster From Cluster Explorer click Edit gt Add Delete Remote Cluster See Adding a remote cluster on page 505 To configure an additional heartbeat between the clusters optional 1 On Cluster Explorer s Edit menu click Configure Heartbeats 2 Inthe Heartbeat configuration dialog box enter the name of the heartbeat and select the check box next to the name of the cluster 3 Click the icon in the Configure column to open the Heartbeat Settings dialog box 4 Specify the value of the Arguments attribute and various timeout and interval fields Click to add an argument value click to delete it If you s
85. as a shared hub or ethernet card Low priority LLT links in clusters with or without I O fencing is recommended In clusters without I O fencing this is critical About communications membership and data protection in the cluster 359 Summary of best practices for cluster communications Note An exception to this is if the cluster uses fencing along with Cluster File Systems CFS or Oracle Real Application Clusters RAC The reason for this is that low priority links are usually shared public network links In the case where the main cluster interconnects fail and the low priority link was the only remaining link large amounts of data would be moved to the low priority link This would potentially slow down the public network to unacceptable performance Without a low priority link configured membership arbitration would go into effect in this case and some systems may be taken down but the remaining systems would continue to run without impact to the public network It is not recommended to have a cluster with CFS or RAC without I O fencing configured Disable the console abort sequence Most UNIX systems provide a console abort sequence that enables the administrator to halt and continue the processor Continuing operations after the processor has stopped may corrupt data and is therefore unsupported by VCS When a system is halted with the abort sequence it stops producing heartbeats The other systems in the cluster c
86. attribute is 0 which designates the service group as a failover group hagrp modify groupx Parallel 1 This attribute cannot be modified if resources have already been added to the service group You can modify the attributes SystemList AutoStartList and Parallel only by using the command hagrp modify You cannot modify attributes created by the system such as the state of the service group About modifying the SystemList attribute When using the hagrp modify command ot change a service group s existing system list you can use the options modify add update delete or delete keys For example suppose you originally defined the SystemList of service group groupx as SystemA and SystemB Then after the cluster was brought up you added a new system to the list hagrp modify groupx SystemList add SystemC 3 You must take the service group offline on the system being modified When you add a system to a service group s system list the system must have been previously added to the cluster When using the command line you can use the hasys add command When you delete a system from a service group s system list the service group must not be online on the system to be deleted If you attempt to change a service group s existing system list using hagrp modi fy without other options such as add or update the command fails Administering the cluster from the command line 263 Administering service grou
87. button at the end of the line item corresponding to the user that you want to delete Administering the cluster from the Cluster Management Console 95 Administering users 2 Inthe Delete User dialog box click OK to confirm that you want to delete the selected user Note You cannot delete the user account that is currently logged in 96 Administering the cluster from the Cluster Management Console Administering a cluster Administering a cluster The Cluster Management Console enables you to initially set and then modify the parameters of the VCS cluster configuration After opening the configuration you can save it to disk Use the Cluster Summary view to administer the cluster To navigate to the Cluster Summary view 1 2 On the navigation bar click Cluster On the secondary tab bar click Summary In the Cluster Summary view you can choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster Opening the configuration You can modify a read only configuration file to a read write file by opening the configuration This task is available only when the cluster configuration has been saved in read only mode To open a configuration 1 In the Cluster Summary view in the Configuration task panel click Open Configuration In the Open Configuration dialog box click OK to confirm tha
88. click the Service Groups tab of the configuration tree 2 Right click an online resource click Fault Resource and click the system name Simulating cluster faults in global clusters Use VCS Simulator to imitate the process of generating and clearing cluster faults See Monitoring alerts on page 230 To generate a cluster fault 1 From Cluster Explorer click the cluster in the configuration tree 2 Right click the cluster click Fault Cluster and click the cluster name To clear a cluster fault 1 From Cluster Explorer click the cluster in the configuration tree 2 Right click the cluster click Clear Cluster Fault and click the cluster name Administering VCS Simulator from the command line 326 Predicting VCS behavior using VCS Simulator Administering VCS Simulator from the command line The functionality of the VCS Simulator command line interface mimics that of standard ha commands Start VCS Simulator before creating or administering simulated clusters Note VCS Simulator treats clusters created from the command line and the Java Console separately So clusters created from the command line are not visible in the graphical interface Also If you delete a cluster from the command line you may see the cluster in the Java Console Starting VCS Simulator from the command line To start VCS Simulator from the command line UNIX 1 To simulate a cluster running a particular operating system copy t
89. cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Offline and click Remote offline In the Offline global group dialog box oe Offline global group Custer 0 st i SOSCi d Systemifvesinux139 O Cancel m Click the remote cluster to take the group offline m Click the specific system or click All Systems to take the group offline m Click OK In the Question dialog box click Yes 516 Administering global clusters from Cluster Manager Java console Administering global service groups Switching a service group to a remote cluster 1 In the Service Groups tab of the Cluster Explorer configuration tree of a local cluster right click the service group or Click a local cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Switch To and click Remote switch In the Switch global group dialog box oes Switch global group Cluster ee System vcslinux139 X Cancel m Click the cluster to switch the group m Click the specific system or click Any System to take the group offline m Click OK In the Question dialog box click Yes Administering global clusters from Cluster Manager Java console 517 Administering global heartbeats Administering global heartbeats Use Cluster Explorer to add modify a
90. consoles on the following types of alerts m CFAULT When the alert is presented clicking Take Action guides you through the process of failing over the global groups that were online in the cluster before the cluster faulted m GNOFAILA When the alert is presented clicking Take Action guides you through the process of failing over the global group to a remote cluster on which the group is configured to run m GNOFAIL There are no associated actions provided by the consoles for this alert Negating events VCS deletes a CFAULT alert when the faulted cluster goes back to the running state VCS deletes the GNOFAILA and GNOFAIL alerts in response to the following events m The faulted group s state changes from FAULTED to ONLINE 578 Troubleshooting and recovery for VCS Troubleshooting licensing The group s fault is cleared The group is deleted from the cluster where alert was generated Concurrency violation at startup VCS may report a concurrency violation when you add a cluster to the ClusterList of the service group A concurrency violation means that the service group is online on two nodes simultaneously Recommended Action Verify the state of the service group in each cluster before making the service group global Troubleshooting licensing This section cites problems you may encounter with VCS licensing It provides instructions on how to validate license keys and lists the error messages associated wi
91. designate is 30 for example the entry point is invoked every 30 seconds for all ONLINE resources of the particular resource type m Type and dimension integer scalar m Default 0 InfoTimeout user defined Timeout value for info entry point If entry point does not complete by the designated time the agent framework cancels the entry point s thread m Type and dimension integer scalar m Default 30 seconds VCS attributes 623 Resource type attributes Table D 2 Resource type attributes LogDbg user defined Indicates the debug severities enabled for the resource type or agent framework Debug severities used by the agent entry points are in the range of DBG_1 DBG_21 The debug messages from the agent framework are logged with the severities DBG_AGINFO DBG_AGDEBUG and DBG_AGTRACE representing the least to most verbose m Type and dimension string keylist m Default none LogFileSize user defined Specifies the size in bytes of the agent log file Minimum value is 65536 bytes Maximum value is 134217728 bytes 128MB m Type and dimension integer scalar m Default 33554432 32MB MonitorInterval user defined Note This attribute can be overridden Duration in seconds between two consecutive monitor calls for an ONLINE or transitioning resource A low value may impact performance if many resources of the same type exist A high value may delay detection of a f
92. disk preventing the case where three or more systems each win the race for one coordinator disk If the cluster splits such that one of the subclusters has at least 51 of the members of the previous stable membership that subcluster is given priority to win the race The system in the smaller subcluster s delay a short period before beginning the race This ensures that as many systems as possible will remain running in the cluster If the vxfen module discovers on startup that the system that has control of the coordinator disks is not in the current GAB membership an error message indicating a possible split brain condition is printed to the console The administrator must clear this condition manually with the vxfenclearpre utility About communications membership and data protection in the cluster 347 Examples of VCS operation with 1 0 fencing Two system cluster where one system fails In this example System1 fails and System0 carries out the I O fencing operation as follows The GAB module on System0 determines System1 has failed due to loss of heartbeat signal reported from LLT GAB passes the membership change to the fencing module on each system in the cluster The only system that is still running is SystemO System0 gains control of the coordinator disks by ejecting the key registered by System1 from each coordinator disk The ejection takes place one by one in the order of the coordinator disk s serial numb
93. each of the four systems also passes the membership change to HAD HAD waits for the result of the membership arbitration from the fencing module before taking any further action Assume System0 wins the race for the coordinator disks and ejects the registration keys of System2 and System3 off the disks The result is as follows m System0 wins the race for the coordinator disk The fencing module on System0 communicates race success to all other fencing modules in the current cluster in this case System0 and System1 The fencing module on each system in turn communicates success to HAD System0 and System1 remain valid and current members of the cluster m System2 loses the race for control of the coordinator disks The fencing module on System2 calls a kernel panic and the system restarts m System3 sees another membership change from the kernel panic of System2 Because that was the system that was racing for control of the coordinator disks in this subcluster System3 also panics HAD carries out any associated policy or recovery actions based on the membership change System2 and System3 are blocked access to the shared storage if the shared storage was part of a service group that is now taken over by System0 or System 1 350 About communications membership and data protection in the cluster Examples of VCS operation with I O fencing m To rejoin System2 and System3 to the cluster the administrator must do the following
94. execution For example an agent is commanded to bring a resource online The agent responds back with the success or failure of the operation Once the resource is online the agent communicates with the engine only if this status changes About communications membership and data protection in the cluster 335 About cluster communications About inter system cluster communications VCS uses the cluster interconnect for network communications between cluster systems Each system runs as an independent unit and shares information at the cluster level On each system the VCS High Availability Daemon HAD which is the decision logic for the cluster maintains a view of the cluster configuration This daemon operates as a replicated state machine which means all systems in the cluster have a synchronized state of the cluster configuration This is accomplished by the following m All systems run an identical copy of HAD m HAD on each system maintains the state of its own resources and sends all cluster information about the local system to all other machines in the cluster m HAD on each system receives information from the other cluster systems to update its own view of the cluster m Each system follows the same code path for actions on the cluster The replicated state machine communicates over a purpose built communications package consisting of two components Group Membership Services Atomic Broadcast GAB and Low Latency Tra
95. five seconds the system is sent an iofence message with reason set to quick reopen When the system receives the message it tries to kill the client process VCS performance considerations 555 How cluster operations affect performance When a Service group switches over The time it takes to switch a service group equals the time to offline a service group on the source system plus the time to bring the service group online on the target system When a service group fails over The time it takes to fail over a service group when a resource faults equals the time it takes to detect the resource fault the time it takes to offline the service group on source system the time it takes for the VCS policy module to select target system the time it takes to bring the service group online on target system The time it takes to fail over a service group when a system faults equals the time it takes to detect system fault the time it takes to offline the service group on source system the time it takes for the VCS policy module to select target system the time it takes to bring the service group online on target system The time it takes the VCS policy module to determine the target system is negligible in comparison to the other factors If you have a firm group dependency and the child group faults VCS offlines all immediate and non immediate parent groups before bringing the child group online on the target system Therefore the
96. for example cluster A and cluster B with the clusters of another global environment for example cluster C and cluster D separate cluster C and cluster D into standalone clusters and add them one by one to the environment containing cluster A and cluster B To add a remote cluster to a global environment 1 Inthe Cluster Summary view in the Configuration task panel click Add Delete Remote Cluster 2 Inthe Remote Cluster Configuration wizard read the introductory information and then click Next 3 Inthe Configuration Options dialog box click Add Cluster and then click Next 4 Doone of the following w For nonsecure clusters 5 Administering global clusters from the Cluster Management Console 491 Administering global clusters In the Connection Details dialog box specify the following details for the connection to the remote cluster and then click Next m Aname or address Enter the IP address of the cluster the IP address of a cluster system or the name of a cluster system m The port Verify the port number The default is 14141 m An administrator user name and password Enter an administrator level user name and password that is valid on the remote cluster m For secure clusters In the Connection Details dialog box specify the following details for the connection to the remote cluster and then click Next m A name or address Enter the IP address of the cluster the IP address of a cluster system or the na
97. for the state of a resource FAULTED returns all resources whose state begins with FAULTED Multiple conditional statements can be used and imply AND logic You can only query attribute value pairs displayed in the output of the command hagrp display See Querying service groups on page 254 To display the list of service groups whose values match a conditional statement Type the following command hagrp list conditional statement If no conditional statement is specified all service groups in the cluster are listed To display a list of resources whose values match a conditional statement Type the following command hares list conditional statement If no conditional statement is specified all resources in the cluster are listed To display a list of agents whose values match a conditional statement Type the following command haagent list conditional_statement If no conditional statement is specified all agents in the cluster are listed Administering the cluster from the command line 261 Administering service groups Administering service groups This section describes how to add delete and modify service groups It also describes how to perform service group operations from the command line Adding and deleting service groups To add a service group to your cluster hagrp add service group The variable service_group must be unique among all service groups defined in the cluster
98. from Cluster Manager Java console Setting up VCS event notification using the Notifier wizard necessary double click the entry in the Priority column to enter a new value m Click Next 4 Choose the mode of notification which needs to be configured Select the check boxes to configure SNMP and or SMTP if applicable 5 Inthe SNMP Configuration dialog box if applicable notification SNMP Consoles T Notifier Resource T SNMP Configuration Specify the SNMP consoles the SNMP trap port and the severity level for SNMP Trap Port Consoles Severity Level fez 4 cancel Hep m Click to create the appropriate number of fields for the SNMP consoles and severity levels Click to remove a field m Enter the console and click the severity level from the menu For example snmpserv and Information m Enter the SNMP trap port For example 162 is the default value m Click Next 6 Inthe SMTP Configuration dialog box if applicable SMTP Server SMTP Recipients F Notifier Resource Configuration Wizard SMTP Configuration Specify the SMTP server the notification recipients Fully qualified wherever required and notification severity levels fmepserv cancel Hep m Enter the name of the SMTP server m Click to create the appropriate number of fields for recipients of the notification and severity levels Clic
99. frozen at the time of the fault the parent remains in its original state If the child cannot fail over to another system the parent remains offline m Ifthe parent group faults the child group may remain online m The child group cannot be taken offline if the parent group is online The parent group can be taken offline while the child is online m When the link is created the parent group must be offline However if both groups are online their online state must not conflict with the type of link Hard dependency Dependency limitations The role of service group dependencies 413 About service group dependencies Imposes the maximum constraints when VCS brings the parent of child service groups online or takes them offline For example If a child group faults the parent is taken offline before the child group is taken offline If the child group fails over the parent fails over to another system or the same system for a local dependency If the child group cannot fail over the parent group remains offline If the parent faults the child is taken offline If the child fails over the parent fails over If the child group cannot fail over the parent group remains offline Note When the child faults if the parent group is frozen the parent remains online The faulted child does not fail over The following restrictions apply when configuring a hard dependency m Only online local hard dependencies are support
100. groups in a cluster You add or modify service groups using the Service Group Configuration wizard See Adding a service group to a cluster on page 98 See Modifying a service group on page 101 Faulting a service group Manually simulate a fault on a service group to test failover behavior This operation is available only when using the simulator To fault a single service group 1 4 In the Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to fault In the Group Summary view in the Simulation task panel click Fault Service Group In the Fault Service Group dialog box select the systems on which you want to fault the service group from the drop down menu This drop down menu is populated using the system list of the service group The system list of a service group contains a list of all systems on which the service group is configured Click OK To fault one or more service groups 1 In the Cluster Groups view in the Groups Listing table check the check box preceding the line item for each service group that you want to fault Select all service groups by checking the check box at the top In the Simulation task panel click Fault Service Groups In the Fault Service Group s dialog box use the appropriate drop down menu to select the systems on which you want to fault each service group Each drop down menu is populated using the system list
101. hastop is issued with the force option Configure one of the following values for the attribute depending on the desired functionality for the hastop command EngineShutdown Description Value Enable Process all hastop commands This is the default behavior Disable Reject all hastop commands DisableClusStop Do not process the hastop all command process all other hastop commands PromptClusStop Prompt for user confirmation before running the hastop all command process all other hastop commands PromptLocal Prompt for user confirmation before running the hastop local command process all other hastop commands PromptAlways Prompt for user confirmation before running any hastop command Administering the cluster from the command line 239 Stopping VCS Additional considerations for stopping VCS If using the command reboot behavior is controlled by the ShutdownTimeOut parameter After HAD exits if GAB exits within the time designated in the ShutdownTimeout attribute the remaining systems recognize this as a reboot and fail over service groups from the departed system For systems running several applications consider increasing the value in the ShutdownTimeout attribute Stopping VCS on a system autodisables each service group that includes the system in their SystemList attribute This does not apply to systems that are powered off If you use the evacuate option evacuation occurs before VCS is brought down
102. highest severity level Information the lowest Note that these severity levels are case sensitive System A System B VCS event notification 431 About VCS event notification SNMP traps are forwarded to the SNMP console Typically traps are predefined for events such as service group or resource faults You can use the hanotify utility to send additional traps Event messages and severity levels When the VCS engine starts up it queues all messages as Information However when notifier connects it communicates one of the following severity levels to HAD depending on which is the lowest m lowest severity for SNMP options m lowest severity for SMTP options If notifier is started from the command line without specifying a severity level for the SNMP console or SMTP recipients notifier communicates the default severity level Warning to HAD If notifier is configured under VCS control severity must be specified See the description of the NotifierMngr agent in the Veritas Cluster Server Bundled Agents Reference Guide For example if the following severities are specified for notifier m Warning for email recipient 1 m Error for email recipient 2 m SevereError for SNMP console Notifier communicates the minimum severity Warning to HAD which then queues all messages labelled severity level Warning and greater Notifier ensures recipients gets only the messages they are designated to receive acco
103. imports a disk group when required and deports it when it is no longer needed m On Only VCS starts On Only resources but does not stop them For example VCS requires NFS daemons to be running to export a file system VCS starts the daemons if required but does not stop them if the associated service group is taken offline m Persistent These resources cannot be brought online or taken offline For example a network interface card cannot be started or stopped but it is required to configure an IP address A Persistent resource has an operation value of None VCS monitors Persistent resources to ensure their status and operation Failure of a Persistent resource triggers a service group failover Resource types VCS defines a resource type for each resource it manages For example the NIC resource type can be configured to manage network interface cards Similarly all IP addresses can be configured using the IP resource type VCS includes a set of predefined resources types For each resource type VCS has a corresponding agent which provides the logic to control resources See About agents in VCS on page 25 Introducing Veritas Cluster Server 23 Logical components of VCS Service groups A service group is a virtual container that contains all the hardware and software resources that are required to run the managed application Service groups allow VCS to control all the hardware and software resources of the mana
104. in the list under Select and Add Systems To quickly find a specific system click Filter type a few letters of the system name and then click Apply To select a system click the system name under Select and Add Systems and then click the right arrow button to move it under Selected Systems You can select one or more systems To reject a system click the system name under Selected Systems and then click the left arrrow button to move it back under Select and Add Systems m The startup system Under Selected Systems check the Startup option next to a system name if you want the service group to startup automatically on that system 102 Administering the cluster from the Cluster Management Console Administering service groups m The system order of preference Use the up and down arrow buttons to the right of the Selected Systems table to set the order of systems in the list To change the position of a system in the list click the system name in the Selected Systems table and then click the up or down arrow button to move it up or down in the list VCS attempts to bring the service group online on the first system in the system list unless you have selected a different system as the startup system using the Startup option If the service group cannot be brought online on the first system or on the designated startup system VCS does one of the following m Ifthe first system is also the startup system VCS attempts to bring
105. is receiving the initial snapshot from the remote cluster RUNNING Indicates the remote cluster is running and connected to the local cluster LOST_HB The connector process on the local cluster is not receiving heartbeats from the remote cluster LOST_CONN The connector process on the local cluster has lost the TCP IP connection to the remote cluster UNKNOWN The connector process on the local cluster determines the remote cluster is down but another remote cluster sends a response indicating otherwise FAULTED The remote cluster is down EXITING The remote cluster is exiting gracefully EXITED The remote cluster exited gracefully INQUIRY The connector process on the local cluster is querying other clusters on which heartbeats were lost TRANSITIONING The connector process on the remote cluster is failing over to another node in the cluster 608 Cluster and system states System states Examples of cluster state transitions m Ifa remote cluster joins the global cluster configuration the other clusters in the configuration transition their view of the remote cluster to the RUNNING State INIT gt BUILD gt RUNNING m Ifa cluster loses all heartbeats to a remote cluster in the RUNNING state inquiries are sent If all inquiry responses indicate the remote cluster is actually down the cluster transitions the remote cluster state to FAULTED RUNNING gt LOST_HB gt INQUIRY gt FAULTED m If at least one r
106. launched For example if using Korn shell type the following command to display on the system myws export DISPLAY myws 0 Using Java Console with secure shell You can use Java Console with secure shell SSH using X11 forwarding or Port forwarding Make sure that SSH is correctly configured on the client and the host systems To use x11 forwarding 1 2 In the ssh configuration file set ForwardX11 to yes ForwardXx11 yes Log on to the remote system and start an X clock program that you can use to test the forward connection Administering the cluster from Cluster Manager Java console 139 Getting started xclock amp Do not set the DISPLAY variable on the client X connections forwarded through a secure shell use a special local display setting To use port forwarding In this mode the console connects to a specified port on the client system This port is forwarded to port 14141 on the VCS server node 1 Inthe ssh configuration file set GatewayPorts to yes GatewayPorts yes 2 From the client system forward a port client_port to port 14141 on the VCS server ssh L client_port server_host 14141 server host You may not be able set GatewayPorts in the configuration file if you use openSSH In this case use the g option in the command ssh g L client _port server_host 14141 server host 3 Open another window on the client system and start the Java Console opt VRTSvcs bin hagui 4 Adda clust
107. m Shut down System2 and System3 m Fix the cluster interconnect links m Restart System2 and System3 About communications membership and data protection in the cluster 351 About cluster membership and data protection without I O fencing About cluster membership and data protection without I 0 fencing Jeopardy Proper seeding of the cluster and the use of low priority heartbeat cluster interconnect links are best practices with or without the use of I O fencing Best practice also recommends multiple cluster interconnect links between systems in the cluster This allows GAB to differentiate between A loss of all heartbeat links simultaneously which is interpreted as a system failure In this case depending on failover configuration HAD may attempt to restart the services that were running on that system on another system m A loss of all heartbeat links over time which is interpreted as an interconnect failure In this case the assumption is made that there is a high probability that the system is not down and HAD does not attempt to restart the services on another system In order for this differentiation to have meaning it is important to ensure the cluster interconnect links do not have a single point of failure such as a network hub or ethernet card In all cases when LLT on a system no longer receives heartbeat messages from another system on any of the configured LLT interfaces GAB reports a change in membership
108. membership arbitration About data protection Examples of VCS operation with I O fencing About cluster membership and data protection without I O fencing Examples of VCS operation without I O fencing Summary of best practices for cluster communications 334 About communications membership and data protection in the cluster About cluster communications About cluster communications VCS uses local communications on a system and system to system communications About intra system communications Within a system the VCS engine HAD uses a VCS specific communication protocol known as Inter Process Messaging IPM to communicate with the GUI the command line and the agents The following illustration shows basic communication on a single VCS system Note that agents only communicate with HAD and never communicate with each other AGENT COMMAND LINE UTILITIES e VCS HIGH AVAILABILITY DAEMON HAD The following illustration depicts communication from a single agent to HAD AGENT SPECIFIC CODE AGENT FRAMEWORK Status Control HAD The agent uses the agent framework which is compiled into the agent itself For each resource type configured in a cluster an agent runs on each cluster system The agent handles all resources of that type The engine passes commands to the agent and the agent returns the status of command
109. nfs_postoffline script doesn t exist Feb 20 2006 4 13 55 AM V 16 1 10446 Group CMC is offline on system veshp10 Feb 20 2006 4 13 55 AM _ 16 1 10305 Resource CMC_ClusterConfig Owner unknown Group CMC is offline on veshp10 VCS initia Click the Agent Logs tab to display logs according to system resource type and resource filter criteria Use this tab to view the log type time and details of an agent event Click the Command Logs tab to view the status success or failure time command ID and details of a command The Command Log only displays commands issued in the current session Click the Alerts tab to view situations that may require administrative action Alerts are generated when a local group cannot fail over to any system in the local cluster a global group cannot fail over or a cluster fault takes place A current alert will also appear as a pop up window when you log on to a cluster through the console Administering the cluster from Cluster Manager Java console 167 Accessing additional features of the Java Console To access the Logs dialog box From Cluster Explorer click Logs on the View menu or On the Cluster Explorer toolbar click Show the Logs Server and user credentials If VCS is running in secure mode you can view server and user credentials used to connect to the cluster from Cluster Explorer To view user credentials From Cluster Explorer click User Creden
110. of the Cluster Management Console Cluster Management Console layout Logging in to the management server starts the Cluster Management Console The startup view is the Cluster Summary view Figure 5 1 Cluster Management Console default view Manage sol50_510a_sect 4 Configuration Cluster Summary a Bahai facia chins i Summary Groups systems Attributes Resources Group Dependency Resource Types or alerts cw iiaiai ecic oei Cluster Name sol50_510a_sec1 State Online Cluster Type VCS Version 5 0 00 0 Faulted Objects Fa Groups 4 4 o o o e Ofline Faulted Partial Critical and Error logs in last 8 hours Log Messages Time SSL handshake failed 131 May 16 2006 10 16 SSL handshake failed 131 May 16 2006 10 16 SSL handshake failed 131 May 16 2006 10 16 SSL handshake failed 131 May 2006 10 16 SSL handshake failed 131 May 16 2006 10 16 All views in the Cluster Management Console are divided into areas called bars panes panels and tables About the title bar The title bar contains the console name Veritas Cluster Management Console and three links About Logout and Help Figure 5 2 Title bar Click About to see information about the Cluster Management Console click Logout to end a console session click Help to access online help for using the console 78 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console
111. on each machine passes to GAB that it has noticed a membership change Specifically m LLT on System0 passes to GAB that it no longer sees System2 and System3 m LLT on System passes to GAB that it no longer sees System2 and System3 m LLT on System2 passes to GAB that it no longer sees System0 and System1 m LLT on System3 passes to GAB that it no longer sees System0 and System1 m After LLT informs GAB of a heartbeat loss the systems that are remaining do a GAB Stable Timeout 5 seconds In this example About communications membership and data protection in the cluster 349 Examples of VCS operation with I O fencing m System0 and System agree that both of them do not see System2 and System3 m System2 and System3 agree that both of them do not see System0 and System1 GAB marks the system as DOWN and excludes the system from the cluster membership In this example m GAB on System0 and System1 mark System2 and System3 as DOWN and excludes them from cluster membership m GAB on System2 and System3 mark System0 and System1 as DOWN and excludes them from cluster membership GAB on each of the four systems passes the membership change to the vxfen driver for membership arbitration Each subcluster races for control of the coordinator disks In this example m System0 has the lower LLT ID and races on behalf of itself and System1 m System2 has the lower LLT ID and races on behalf of itself and System3 GAB on
112. on resource faults VCS considers a resource faulted in the following situations m When the resource state changes unexpectedly For example an online resource going offline m When arequired state change does not occur For example a resource failing to go online or offline when commanded to do so In many situations VCS agents take predefined actions to correct the issue before reporting resource failure to the engine For example the agent may try to bring a resource online several times before declaring a fault When a resource faults VCS takes automated actions to clean up the faulted resource The Clean function makes sure the resource is completely shut down before bringing it online on another node This prevents concurrency violations When a resource faults VCS takes all resources dependent on the faulted resource offline The fault is thus propagated in the service group About critical and non critical resources The Critical attribute for a resource defines whether a service group fails over when the resource faults If a resource is configured as non critical by setting the Critical attribute to 0 and no resources depending on the failed resource are critical the service group will not fail over VCS takes the failed resource offline and updates the group status to ONLINE PARTIAL The attribute also determines whether a service group tries to come online on another node if during the group s online process a re
113. on the system where VRTSweb is installed VRTSWEB_ HOME bin webgui smtp getserver The command displays the SMTP server address or hostname if it is configured To retrieve the name of the SMTP server from the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 The SMTP Recipients table on the right side of the page displays the configured SMTP server 668 Administering Symantec Web Server Configuring SMTP notification for VRTSweb Setting the SMTP server Configure an SMTP server for VRTSweb To set the SMTP server from the command line Run any of the following commands on the system where VRTSweb is installed VRTSWEB_HOME bin webgui smtp setserver server ip hostname VRTSWEB_HOME bin webgui smtp delserver The setserver command sets the SMTP server to the specified hostname IP address The delserver command deletes the SMTP server setting and disables SMTP notification For example opt VRTSweb bin webgui smtp setserver smtphost company com opt VRTSweb bin webgui smtp setserver 101 1 2 3 opt VRTSweb bin webgui smtp delserver To set the SMTP server from the Web Console 1 Access the Web server on a configured port For example http hostname 8181 Click the Configuration tab Click Configure SMTP Server on the left side of the Configuration page In the Configure SMTP Server dialog box m Enter the IP address or ho
114. operations are allowed on the service group m Type and dimension boolean scalar m Default 1 enabled VCS attributes Service group attributes Table D 3 Service group attributes MigrateQ system use only Indicates the system from which the service group is migrating This attribute is specified when group failover is an indirect consequence in situations such as a system shutdown or another group faults and is linked to this group m Type and dimension string association m Default Not applicable NumRetries system use only Indicates the number of attempts made to bring a service group online This attribute is used only if the attribute OnlineRetryLimit is set for the service group m Type and dimension integer scalar m Default Not applicable OnlineRetryInterval user defined Indicates the interval in seconds during which a service group that has successfully restarted on the same system and faults again should be failed over even if the attribute OnlineRetryLimit is non zero This prevents a group from continuously faulting and restarting on the same system m Type and dimension integer scalar m Default 0 OnlineRetryLimit user defined If non zero specifies the number of times the VCS engine tries to restart a faulted service group on the same system on which the group faulted before it gives up and tries to fail over the group to another system m Type and dimensio
115. panels 82 controlling page updates 89 deleting users 94 expanding panels 82 filter panels 80 icon conventions 87 logging out of 75 managing cluster configuration 96 sorting data tables 88 supported browsers 72 Cluster Manager Java Console See Java Console Cluster Monitor about 142 adding clusters 169 administering 169 behavior during failover 144 collapsing displays 145 configuring existing panels 170 configuring new panels 169 expanding displays 145 icon colors 144 logging off of acluster 173 logging on toacluster 171 menus 143 monitoring cluster connection 144 monitoring cluster objects 144 panels 144 pausing scrolling panels 145 toolbar 143 cluster name changing in global configuration 532 Cluster Operator about 63 adding user as 251 Cluster Query in Java Console 165 ClusterAddress attribute 648 ClusterFailOverPolicy attribute 630 clustering criteria for data storage 19 criteria for monitor procedure 18 criteria for start procedure 18 criteria for stop procedure 18 license and host name issues 19 Index 689 ClusterList attribute 630 ClusterLocation attribute 649 ClusterName attribute 649 ClusterOwner attribute 649 clusters administering from Java Console 220 connecting to Cluster Monitor 169 ClusterTime attribute 649 ClusterUUID attribute 649 Command Center accessing 163 adding resources 200 adding service groups 179 adding systems 218 autoenabling service groups 190 bringing resources online 204 bringin
116. parameter is meaningful only if the clean operation is implemented m Type and dimension integer scalar m Default 0 OnlineTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the online entry point must complete or else be terminated m Type and dimension integer scalar m Default 300 seconds 626 VCS attributes Resource type attributes Table D 2 Resource type attributes OnlineWaitLimit Number of monitor intervals to wait for the resource to come online after completing the online procedure Increase the value of this attribute if the resource is likely to take a longer Note This attribute can time to come online be overridden user defined m Type and dimension integer scalar m Default 2 OpenTimeout Maximum time in seconds within which the open entry point user defined must complete or else be terminated m Type and dimension integer scalar Note This attribute can m Default 60 seconds be overridden Operations Indicates valid operations for resources of the resource type Values are OnOnly can online only OnOff can online and user defined offline None cannot online or offline m Type and dimension string scalar m Default OnOff RestartLimit Number of times to retry bringing a resource online when it is user defined taken offline unexpectedly and before VCS declares it FAULTED m Type and dime
117. receives a message acknowledgement from notifier when notifier has delivered the message to at least one designated recipient Example two SNMP consoles and two email recipients are designated Notifier sends an acknowledgement to VCS even if the message reached only one of the four recipients Error messages are also printed to the log files when delivery errors occur VCS event notification 433 Components of VCS event notification Components of VCS event notification This section describes the notifier process and the hanotify utility The notifier process The notifier process configures how messages are received from VCS and how they are delivered to SNMP consoles and SMTP servers Using notifier you can specify notification based on the severity level of the events generating the messages You can also specify the size of the VCS message queue which is 30 by default You can change this value by modifying the MessageQueue attribute See the VCS Bundled Agents Reference Guide for more information about this attribute When notifier is started from the command line VCS does not control the notifier process For best results use the NotifierMngr agent that is bundled with VCS Configure notifier as part of a highly available service group which can then be monitored brought online and taken offline For information about the agent see the Veritas Cluster Server Bundled Agents Reference Guide Note that notifier must be confi
118. regarding resource_type is stored in the file config resource_type cf and an include line for resource_type cf is added to the main cf file Make sure that the path to the SourceFile exists on all nodes before you run this command To set the value of static resource type attributes hatype modify Overriding resource type static attributes You can override some resource type static attributes and assign them resource specific values When a static attribute is overriden and the configuration is saved the main cf file includes a line in the resource definition for the static attribute and its overriden value To override a type s static attribute hares override resource static_attribute To restore default settings to a type s static attribute hares undo_ override resource static_attribute Initializing resource type scheduling and priority attributes The following configuration shows how to initialize resource type scheduling and priority attributes through configuration files The example shows attributes of a FileOnOff resource Administering the cluster from the command line 283 Administering resource types type FileOnoff static str AgentClass RT static str AgentPriority 10 static str ScriptClass RT static str ScriptPriority 40 static str ArgList PathName str PathName Setting Scheduling and Priority attributes To update the AgentClass hatype modify resource type AgentClass val
119. required by the application Select the mounts required by the application Specify the options for the Fsck operations For wes file systems select SnapUmount if desired Select Mount Point_ Block Device File System Mount Options Fsck Options _ SnapUmount m jfzonefizone fdevidskic1t0d ufs rw intrlargefile y na m _ Discover Mounts Z Create mount points on all systems if they do not exist 2 More Information m Select the check boxes next to the mount points to be configured in the Application service group Click Discover Mounts to discover mounts created after the wizard was started m Specify the Mount and Fsck options if applicable The agent uses these options when bringing the resource online m If using the vxfs file system you can select the SnapUmount check box to take the MountPoint snapshot offline when the resource is taken offline m Select the Create mount points on all systems if they do not exist check box if desired m Click Next Configuring applications and resources in VCS 307 Configuring application service groups 11 Configure the IP and NIC resources for the application Network Configuration VCS Application Configuration Wizard Configure IP and NIC resources if required by the application X Does application require virtual IP Virtual IP Address 10 212 88 174 Discover IP Subnet Mask 255 255 254 0 System
120. resource improved while it was online This trap is generated when statistical analysis for the time taken by the monitor entry point of an agent is enabled for the agent See VCS agent statistics on page 560 This trap is generated when the agent framework detects a sudden change in the time taken to run the monitor entry point for a resource The trap information contains details of m The change in time required to run the monitor entry point m The actual times that were compared to deduce this change Event Resource is in ADMIN_WAIT state Error Events and traps for systems Event VCS is being restarted by Warning hashadow VCS is in jeopardy Warning VCS is up on the first nodein Information the cluster VCS has faulted SevereError Anode running VCS has joined Information cluster VCS has exited manually Information CPU usage exceeded threshold Warning on the system VCS event notification 437 VCS events and traps Severity Level Description The resource is in the admin_wait state See Controlling Clean behavior on resource faults on page 368 Severity Level Description Self explanatory One node running VCS is in jeopardy Self explanatory Self explanatory Self explanatory VCS has exited gracefully from one node on which it was previously running The system s CPU usage continuously exceeded the value that is set in the Notify threshold for a duration
121. resource name Enter the name of the resource that you want to add to the service group into the Resource Name text box Names must consist of only one string and must not contain special characters like period or _ underscore m The resource type Select the type of the new resource from the Resource Type drop down menu m The resource startup state If you want the resource to start in an enabled state after the service group comes online check Enable Resource Administering the cluster from the Cluster Management Console 117 Administering resources m The resource list for the service group Click Add Resource to add the resource to the member resource list for the service group The resource list is used to populate the Resource List table In the Resource List table you can edit resource attributes or delete a resource using the buttons in the Edit and Delete columns Overriding resource attributes Change the default value of one or more resource attributes You must have the role of cluster administrator or group administrator to override resource attribute values To override resource attributes 1 In the Cluster Resources view in the Resources Listing table click a linked resource name In the Resource Summary view in the Configuration task panel click Override Attributes In the Override Attributes dialog box select the attribute you want to override from the drop down menu and then click OK In the Ed
122. resynchronization will not occur 9 Set resource dependencies such that the Mount resource depends on the RVGPrimary resource Connecting clusters Creating global clusters 475 Setting up a global cluster The appgroup now looks like T vcspri O xj Resource View appgroup on all systems EP TR RI Auto arrange A amp I J appgroup thori65S thori4 10 If your setup uses BIND DNS add a resource of type DNS to the appgroup service group and configure its attributes Domain Domain name For example veritas com Alias Alias to the canonical name For example www Hostname Canonical name of a system or its IP address For example mtv veritas com TTL Time To Live in seconds for the DNS entries in the zone being updated Default value 86400 StealthMasters List of primary master name servers in the domain This attribute is optional if the primary master name server is listed in the zone s NS record If the primary master name server is a stealth server the attribute must be defined Note that a stealth server is a name server that is authoritative for a zone but is not listed in the zone s NS records 476 Connecting clusters Creating global clusters Setting up a global cluster Linking the application and replication service groups Set an online local hard group dependency from appgroup to appgroup_rep to ensure that the service groups fail over and switch together To lin
123. same host name after an outage Custom scripting to modify a system host name on failover is not recommended 20 Introducing Veritas Cluster Server Physical components of VCS Symantec recommends you configure applications and licensing to run properly on all hosts Physical components of VCS Nodes A VCS cluster comprises of systems that are connected with a dedicated communications infrastructure VCS refers to a system that is part of a cluster as a node Each cluster has a unique cluster ID Systems in a cluster are connected by redundant cluster communication links VCS nodes host the service groups managed applications Each system is connected to networking hardware and usually also to storage hardware The systems contain components to provide resilient management of the applications and start and stop agents Nodes can be individual systems or they can be created with domains or partitions on enterprise class systems Individual cluster nodes each run their own operating system and possess their own boot device Each node must run the same operating system within a single VCS cluster Clusters can have from 1 to 32 nodes Applications can be configured to run on specific nodes within the cluster Shared storage Networking Storage is a key resource of most applications services and therefore most service groups A managed application can only be started on a system that has access to its associated data files
124. sdiff f filename s snapid sys hostname s Identifies the snapshot ID of the comparison snapshot sys Indicates the host on which the snapshot is to be compared If you do not specify any options the command uses the latest snapshot to compare the files on each node in the cluster Examples The following command displays the differences between the current configuration and the snapshot vcs 20030101 22232 hasnap sdiff s vcs 20030101 22232 The following command displays the difference between the configuration on system host1 and the snaphot stored in the file tmp backup 2 2 2003 hasnap sdiff f tmp backup 2 2 2003 sys host1 Comparing a file with its snapshot copy Use the hasnap fdiff to displays differences between a file on the cluster and its copy stored in a previously created snapshot To compare a file with its snapshot copy Run the following command hasnap fdiff f filename s snapid sys hostname file s Specifies the ID of the snapshot sys Specifies the host on which the snapshot is to be compared file The file to compare If you do not specify any options the command uses the latest snapshot to compare the file on each node in the cluster Examples The following command displays the differences between the files etc VRTSves conf config main cf on host1 and its version in the last snapshot hasnap fdiff sys host1 etc VRTSvcs conf config main cf The following command disp
125. secure mode 1 Define the node on which the VCS commands will be run Set the VCS_HOST environment variable to the name of the on which to run commands You can set the variable to the virtual IP address configured in the ClusterService group 2 Logon to VCS halogin vesusername password To end a session for a host Run the following command halogin endsession hostname Administering the cluster from the command line 241 Logging on to VCS To end all sessions Run the following command halogin endallsessions VCS prompts you for credentials every time you run a VCS command 242 Administering the cluster from the command line Managing VCS configuration files Managing VCS configuration files This section describes how to verify back up and restore VCS configuration files See About the main cf file on page 47 See The types cf file on page 49 About the hacf utility The hacf utility translates the VCS configuration language into a syntax that can be read by the VCS engine Specifically hacf translates the contents of the main configuration file main cf into commands for the VCS server The hacf utility verifies the configuration before loading it into VCS The configuration is not loaded under the following conditions m If main cf or include files are missing m If syntax errors appear in the cf files m If the configuration file is invalid See Setting the Configuration
126. service group type 10 11 Administering the cluster from Cluster Manager Java console 197 Administering service groups m Click Next Click Next again to configure the service group with a template and proceed to step 7 Click Finish to add an empty service group to the selected cluster systems and configure it at a later time Click the template on which to base the new service group The Templates box lists the templates available on the system to which Cluster Manager is connected The resource dependency graph of the templates the number of resources and the resource types are also displayed Click Next If a window notifies you that the name of the service group or resource within the service group is already in use proceed to step 9 Otherwise proceed to step 10 Click Next to apply all of the new names listed in the table to resolve the name clash or Modify the clashing names by entering text in the field next to the Apply button clicking the location of the text for each name from the Correction drop down list box clicking Apply and clicking Next Click Next to create the service group A progress indicator displays the status After the service group is successfully created click Next to edit attributes using the wizard Click Finish to edit attributes at a later time using Cluster Explorer Review the attributes associated with the resources of the service group If necessary proceed to step 11 to
127. sorted by the values in that column in ascending order A triangle pointing up displays in that column heading 3 If desired click the column heading a second time The list is sorted by the values in that column in descending order A triangle pointing down displays in that column heading Viewing multiple pages of a data table Tables in Cluster Management Console views may contain multiple pages The Cluster Management Console displays twenty rows per page You can view subsequent pages of a table using the go to page bar To access pages ina table Inatable with multiple pages locate the go to page bar just below the table Figure 5 13 Go to page bar 1 2 3 4 5 gt amp Click the controls on the go to page bar to move forward and backward through the table pages The following are the controls on the go to page bar and their meaning Numerals 1 2 3 Go to that page number Single right arrow Go to next page Single left arrow Go to previous page Double right arrow Go to last page Double left arrow Go to first page Administering the cluster from the Cluster Management Console 89 Overview of the Cluster Management Console Controlling view updates As you use the Cluster Management Console the status of the cluster systems and applications will probably change Devices go online and offline alerts are generated and application data fluctuates You can control when and how the information in your co
128. specified by the ToleranceLimit before the ToleranceCount is reset TC TL the resource is considered failed After the agent determines the resource is not online VCS checks the Frozen attribute for the service group If the service group is frozen VCS declares the resource faulted and calls the resfault trigger No further action is taken If the service group is not frozen VCS checks the ManageFaults attribute If ManageFaults NONE VCS marks the resource state as ONLINE ADMIN_WAIT and calls the resadminwait trigger If ManageFaults ALL VCS calls the Clean entry point with the CleanReason set to Unexpected Offline If the Clean entry point fails exit code 1 the resource remains online with the state UNABLE TO OFFLINE VCS fires the resnotoff trigger and monitors the resource again The resource enters a cycle of alternating Monitor and Clean entry points until the Clean entry point succeeds or a user intervenes If the Clean entry point is successful VCS examines the value of the RestartLimit RL attribute If the attribute is set to a non zero value VCS increments the RestartCount RC attribute and invokes the Online entry point This continues till the value of the RestartLimit equals that of the RestartCount At this point VCS attempts to monitor the resource If the Monitor returns an online status VCS considers the resource online and resumes periodic monitoring If the monitor returns an offline status 376 Contro
129. system goes down within the number of seconds designated in ShutdownTimeout service groups previously online on the failed system are treated as faulted and failed over If you do not want to treat the system reboot as a fault set the value for this attribute to 0 m Type and dimension integer scalar m Default 120 seconds SourceFile File from which the configuration was read Always set to user defined main cf Make sure the path exists on all nodes before configuring this attribute m Type and dimension string scalar m Default main cf SysInfo Provides platform specific information including the name system use only version and release of the operating system the name of the system on which it is running and the hardware type m Type and dimension string scalar m Default Not applicable SysName Indicates the system name system use only m Type and dimension string scalar m Default Not applicable SysState Indicates system states such as RUNNING FAULTED EXITED etc system use only m Type and dimension integer scalar m Default Not applicable SystemLocation Indicates the location of the system user defined m Type and dimension string scalar m Default Not applicable Table D 4 VCS attributes System attributes 647 System attributes SystemOwner user defined This attribute is used for VCS email notification and logging VCS sends email notification to t
130. system states Remote cluster states ccccccccccecssccsscsscssccsscsssssecsscssessscssscssesessssseesessssseseeeeees 606 System states VCS attributes About attributes and their definitions 20 00 cecccsccseescseseeseseseeeceneesens 614 Resource attributes srne aa E A A N Resource type attributes Service group attributes System attributes 0 Cluster attributes 00 Heartbeat attributes enina AA NAER Administering Symantec Web Server About Symantec Web Server o ccccccccessssssesessesesseseseeeeseseeeeseseeceseeeeseseeeeseeeeaees Getting Started oc eeeseeseeeeeees Configuring ports for VRTSWED o cecceccessssssssesesesseesesesescsesseeesseeseseseseseaeeees Managing VRTSweb SSL certificates Configuring SMTP notification for VRTSweb Configuring logging for VRTSWeD u ccceseseseseseeeeseeseeeseeeeeees Modifying the maximum heap size for VRTSWeD o cccseseseseseeeseteeteteees 678 Accessibility and VCS About accessibility in VCS 0 Navigation and keyboard shortcuts Support for accessibility settings 0 0 ceseseseseseseseesesetssesseeescseseseeeseeeeees Support for assistive technologies c c ccesesssssesesesesesesesesseseenscsescsesesesesesees 683 687 11 12 Contents Clusterin g concepts and terminology m Chapter 1 Introducing Veritas Cluster Server on page 15 m Chapter 2 About cluster topologies on page 33 m Chapter 3 V
131. systems To reject a system click the system name under Selected Systems and then click the left arrrow button to move it back under Select and Add Systems The startup system Under Selected Systems check the Startup option next to a system name if you want the service group to startup automatically on that system The system order of preference Use the up and down arrow buttons to the right of the Selected Systems table to set the order of systems in the list To change the position of a system in the list click the system name in the Selected Systems table and then click the up or down arrow button to move it up or down in the list 104 Administering the cluster from the Cluster Management Console Administering service groups VCS attempts to bring the service group online on the first system in the system list unless you have selected a different system as the startup system using the Startup option If the service group cannot be brought online on the first system or on the designated startup system VCS does one of the following m Ifthe first system is also the startup system VCS attempts to bring the service group online on each remaining system in the order listed m If the first system and the startup system are different VCS attempts to bring the service group online on the first system in the list If the first system fails VCS attempts to bring the service group online on each remaining system in the order lis
132. systems on which the service group was brought partially or fully online Using event triggers VCS provides a sample Perl script in VCS_HOME bin sample_triggers for each event trigger Customize the scripts according to your requirements you may choose to write your own Perl scripts To use an event trigger 1 Use the sample scripts in VCS_HOME bin sample_triggers to write your own custom actions for the trigger 2 Move the modified trigger script to VCS_HOME bin triggers on each node 3 Configure other attributes that may be required to enable the trigger See the usage information for the trigger for more information See List of event triggers on page 447 VCS event triggers 447 List of event triggers List of event triggers The information in the following sections describes the various event triggers including their usage parameters and location Cpuusage event trigger Description The cpuusage event trigger is invoked when the system s CPU usage exceeds the ActionThreshold value of the system s CPUUsageMonitoring attribute for a duration longer than the ActionTimeLimit value The trigger is not invoked if it was invoked on the system within the last five minutes Usage To enable the trigger To disable the trigger See When a resource comes online on page 550 This event trigger is configurable cpuusage triggertype system cpu_usage triggertype represents whether trigger is custo
133. tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Switch To and click the appropriate system from the menu To switch a service group from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Controls gt Switch Service Group Click the service group For global groups select the cluster in which to switch the service group Click the system on which to bring the group online or select the Any System check box Click Apply 186 Administering the cluster from Cluster Manager Java console Administering service groups Freezing a service group Freeze a service group to prevent it from failing over to another system The freezing process stops all online and offline procedures on the service group To freeze a service group from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Freeze and click Temporary or Persistent from the menu The persistent option maintains the frozen state after a reboot if you save this change to the configuration To freeze a service group from Command Center 1 In the Command Center confi
134. that help you to perform a task Figure 5 6 Task panel links ie Configuration a A Add Delete Remote Cluster Save Configuration For views that show details for a specific object some tasks may not be linked underlined Unlinked tasks can indicate one or both of the following m The tasks cannot be performed on the object or objects that you are viewing m Your current user role does not authorize you to perform those tasks About filter panels Many views contain a filter panel in the task pane A filter panel enables you to view only those objects that meet the criteria you specify You specify the filter criteria using controls such as check boxes or drop down menus If you want to perform a task on only certain objects you can first filter the view for those objects and then select the task The following are the possible filter names and settings m By Consolidated State shows objects with the following values for the ConsolidatedState attribute m Online shows responsive functioning objects m Offline shows unresponsive nonfunctioning objects m Partial shows objects clusters or service groups in which one or more but not all member objects have experienced a fault m Faulted shows objects that have experienced a fault m Unknown shows objects in an undetermined consolidated state m Global Groups shows service groups configured using VCS GCO features Administering the cluster from the Cluster
135. the ADMIN_WAIT state INITING gt STALE_DISCOVER_WAIT gt ADMIN_WAIT When a system in RUNNING state is stopped with the hastop command it transitions to the EXITED state as shown below During the LEAVING state any online system resources are taken offline When all of the system s resources are taken offline and the agents are stopped the system transitions to the EXITING State then EXITED RUNNING gt LEAVING gt EXITING gt EXITED 612 Cluster and system states System states Appendix VCS attributes About attributes and their definitions Resource attributes Resource type attributes Service group attributes System attributes Cluster attributes Heartbeat attributes for global clusters About attributes and their definitions 614 VCS attributes About attributes and their definitions In addition to the attributes listed in this appendix see the Veritas Cluster Server Agent Developer s Guide You can modify the values of attributes labelled user defined from the command line or graphical user interface or by manually modifying the main cf configuration file You can change the default values to better suit your environment and enhance performance When changing the values of attributes be aware that VCS attributes interact with each other After changing the value of an attribute observe the cluster systems to confirm that unexpected behavior does not impair performance The values of at
136. the application whether replicated or not should be prepared otherwise a snapshot might not succeed Press the Enter key when prompted Enter the cache size to store writes when the snapshot exists The size of the cache must be large enough to store the expected number of changed blocks during the fire drill However the cache is configured to grow automatically if it fills up Enter disks on which to create the cache Press the Enter key when prompted The wizard starts running commands to create the fire drill setup Press the Enter key when prompted The wizard creates the application group with its associated resources It also creates a fire drill group with resources for the application Oracle for example the Mount and the RVGSnapshot types The application resources in both service groups define the same application the same database in this example The wizard sets the FireDrill attribute for the application resource to 1 to prevent the agent from reporting a concurrency violation when the actual application instance and the fire drill service group are online at the same time Connecting clusters Creating global clusters 485 Setting up a fire drill Verifying a successful fire drill Bring the fire drill service group online on a node that does not have the application running Verify that the fire drill service group comes online This action validates that your disaster recovery solution is configured correctly and
137. the original reason for the membership change has cleared during the restart m If the systems that restart can not exchange heartbeat with the number of cluster systems declared in etc gabtab they will not automatically seed and HAD will not start This is a possible split brain condition and requires administrative intervention Note Forcing a manual seed at this point will allow the cluster to seed However when the fencing module checks the GAB membership against the systems that have keys on the coordinator disks a mismatch will occur vxfen will detect a possible split brain condition print a warning and will not start In turn HAD will not start Administrative intervention is required About communications membership and data protection in the cluster 345 About data protection About data protection Membership arbitration by itself is inadequate for complete data protection because it assumes that all systems will either participate in the arbitration or are already down Rare situations can arise which must also be protected against Some examples are m A system hang causes the kernel to stop processing for a period of time m The system resources were so busy that the heartbeat signal was not sent m A break and resume function is supported by the hardware and executed Dropping the system to a system controller level with a break command can result in the heartbeat signal timeout In these types of sit
138. the service group vill be added m Click the right arrow to move the selected systems to the Systems for Service Group box The priority number starting with 0 is automatically assigned to indicate the order of systems on which the service group will start in case of a failover If necessary double click the entry in the Priority column to enter a new value Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every time HAD is started Administering the cluster from Cluster Manager Java console 179 Administering service groups m Click the appropriate service group type A failover service group runs on only one system at a time a parallel service group runs concurrently on multiple systems m To add anew service group based on a template click Templates Otherwise proceed to step 2g Alternative method to add a new service group based on a template From Cluster Explorer click Templates on the Tools menu Right click the Template View panel and click Add as Service Group from the menu m Click the appropriate template name then click OK Click Show Command in the bottom left corner if you want to view the command associated with the service group Click Hide Command to close the view of the command m Click OK To add a service group from Command Center 1 Inthe Command Center configuration tree expand
139. then enter the domain name in the box provided 5 Inthe Create RemoteGroup Resource dialog box specify the following monitoring details and then click Finish m The remote service group Select the remote service group from the Remote Service Group drop down menu This menu is populated using the service groups configured in the remote cluster that you specified m The type of monitoring you want to perform The following options are available on the Type of Monitoring drop down menu 114 Administering the cluster from the Cluster Management Console Administering service groups OnlineOnly The remote group resource only brings the remote service group online The RemoteGroup resource cannot take the remote service group offline MonitorOnly The remote group resource only monitors the state of the remote service group The remote group resource cannot online or offline the remote service group m OnOff The remote group resource onlines or offlines the remote service group m Monitoring details The System Based Monitoring Details setting has two options Any and One to One Mapping See the Veritas Cluster Server Bundled Agents Reference Guide for information on this setting After you configure the remote group resource you can view the service group dependency Cluster Group Dependency view See Viewing service group dependencies on page 109 Administering the cluster from the Cluster Management Console 115 Admini
140. time a resource faults To display the status of service groups and resources on specific systems Type the following command hastatus sound sys system name sys system name 258 Administering the cluster from the command line Querying VCS To display the status of specific resources Type the following command hastatus sound resource resource name resource resource name To display the status of cluster faults including faulted service groups resources systems links and agents Type the following command hastatus summary Note Unless executed with the summary options the hastatus command continues to produce output of online state transitions until you interrupt it with the command CTRL C Querying log data files LDFs Log data files LDFs contain data regarding messages written to a corresponding English language file Typically for each English file there is a corresponding LDF To display the hamsg usage list Type the following command hamsg help To display the list of LDFs available on the current system Type the following command hamsg list To display general LDF data Type the following command hamsg info path path name LDF The option path specifies where hamsg looks for the specified LDF If not specified hamsg looks for files in the default directory var VRTSves ldf To display specific LDF data Type the following com
141. to Read Write on page 67 About multiple versions of cf files When hacf creates a cf file it does not overwrite existing cf files A copy of the file remains in the directory and its name includes a suffix of the date and time it was created such as main cf 03Dec2001 175904 In addition the previous version of any cf file is saved with the suffix previous for example main cf previous Verifying a configuration Use hacf to verify check syntax of the main cf and the type definition file types cf VCS does not execute if hacf detects errors in the configuration To verify a configuration Run the following command hacf verify config directory The variable config_directory refers to directories containing a main cf file and any cf files included in main cf No error message and a return value of zero indicates that the syntax is legal Administering the cluster from the command line 243 Managing VCS configuration files Scheduling automatic backups for VCS configuration files Configure the BackupInterval attribute to instruct VCS to create a back up of the configuration periodically VCS backs up the main cf and types cf files as main cf autobackup and types cf autobackup respectively To start periodic backups of VCS configuration files Set the cluster level attribute BackupInterval to a non zero value For example to back up the configuration every 5 minutes set BackupInterval to 5 Saving a configurati
142. to a disaster protected RDC infrastructure The solution uses Veritas Volume Replicator to replicate changed data Typical replicated data cluster configuration The following illustration depicts a typical RDC configuration Application Group Application Lanman i online local hard dependency Replication Group NIC In this example a single instance Oracle database is configured as a VCS service group oragroup on a four node cluster with two nodes in the primary RDC system zone and two in the secondary RDC system zone In the event of a failure on the primary node VCS fails over Oracle to the second node in the primary zone The process involves the following steps m Setting Up Replication m Configuring the Service Groups m Configuring the Service Group Dependencies Setting up replicated data clusters 539 Setting up a replicated data cluster configuration Setting up replication Veritas Volume Replicator VVR technology is a license enabled feature of Veritas Volume Manager VxVM so you can convert VxVM managed volumes into replicated volumes managed using VVR In this example the process involves grouping the Oracle data volumes into a Replicated Volume Group RVG and creating the VVR Secondary on hosts in another VCS cluster located in your DR site When setting up VVR it is a best practice to use the same DiskGroup and RVG name on both sites If the volume names are
143. to retrieve this information 674 Administering Symantec Web Server Configuring logging for VRTSweb Modifying log levels for VRTSweb Customize the log levels for VRTSweb To modify log levels from the command line Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui log server level webapps level other level You can specify any of the following values for the variable Zevel for each Web server component FINE FINER FINEST CONFIG INFO WARNING SEVERE Set the level to a lower value to generate more logs FINEST is the lowest level while SEVERE is the highest level For example opt VRTSweb bin webgui log server FINEST webapps INFO other ERROR opt VRTSweb bin webgui log server INFO Administering Symantec Web Server 675 Configuring logging for VRTSweb To modify log levels from rom the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 Click Configure Logging on the left side of the Configuration page 4 Inthe Configure Logging dialog box Configuration Configure Logging This page lets you configure the logging level for various components of the web server Select a lower level of threshold to generate more logs Web Server Web Applications Other Username Password Should belong to the group root on vcslabweb m Select the logging levels for th
144. trigger processes are determined by the attributes ProcessClass default TS and ProcessPriority default Both attributes can be modified according to the class and priority at which the trigger processes run Table 20 2 Default scheduling classes and priorities Engine RT 52 2 Min 0 57 Strongest Strongest Max 99 Strongest 2 2 2 Process TS 0 N A 0 60 created by Strongest engine Agent TS 0 N A 0 0 Script TS 0 N A 0 0 Note For standard configurations Symantec recommends using the default values for scheduling unless specific configuration requirements dictate otherwise Note that the default priority value is platform specific When priority is set to empty string VCS converts the priority to a value specific to the platform on which the system is running For TS the default priority equals the strongest priority supported by the TimeSharing class For RT the default priority equals two less than the strongest priority supported by the RealTime class So if the strongest priority supported by the RealTime class is 59 the default priority for the RT class is 57 For SHR on Solaris only the default priority is the strongest priority support by the SHR class 558 VCS performance considerations CPU binding of HAD CPU binding of HAD In certain situations the Solaris operating systems may assign the CPU to high priority interrupt thread to the CPU on which HAD i
145. user defined Indicates if service groups can be brought online on the system Groups cannot be brought online if the attribute value is 1 m Type and dimension boolean scalar m Default False EngineVersion Specifies the major minor maintenance patch and point patch version of VCS The value of EngineVersion attribute is in hexa decimal format To retrieve version information Major Version EngineVersion gt gt 24 amp Oxff Minor Version EngineVersion gt gt 16 amp Oxff Maint Patch EngineVersion gt gt 8 amp Oxff Point Patch EngineVersion amp Oxff m Type and dimension integer scalar m Default Not applicable GUIIPAddr Determines the local IP address that VCS uses to accept user defined connections Incoming connections over other IP addresses are dropped If GUIIPAddr is not set the default behavior is to accept external connections over all configured local IP addresses See User privileges for CLI commands on page 64 m Type and dimension string scalar m Default 644 VCS attributes System attributes Table D 4 System attributes LicenseType Indicates the license type of the base VCS key used by the system Possible values are 0 DEMO 1 PERMANENT 2 PERMANENT_NODE_LOCK 3 DEMO_NODE_LOCK 4 NER 5 DEMO_EXTENSION 6 NFR_NODE_LOCK 7 DEMO_EXTENSION_NODE_LOCK m Type and dimension integer scalar m Default Not applicable system use only Lim
146. uses the VCS Command Server Daemon running on the remote nodes to establish connection The connection fails if the Daemon is not running on the remote node Recommended Action Verify the VCS Command Server Daemon is running on all cluster nodes Start it by running the following command opt VRTSvcs bin CmdServer m The remote node might be currently down or unreachable Recommended Action Run the hasnap command again after the bringing the remote node online Troubleshooting and recovery for VCS 575 Troubleshooting and recovery for global clusters Troubleshooting and recovery for global clusters This section describes the concept of disaster declaration and provides troubleshooting tips for configurations using global clusters Disaster declaration When a cluster in a global cluster transitions to the FAULTED state because it can no longer be contacted failover executions depend on whether the cause was due to a split brain temporary outage or a permanent disaster at the remote cluster If you choose to take action on the failure of a cluster in a global cluster VCS prompts you to declare the type of failure m Disaster implying permanent loss of the primary data center m Outage implying the primary may return to its current form in some time m Disconnect implying a split brain condition both clusters are up but the link between them is broken m Replica implying that data on the takeover target has been ma
147. vestinux146 Port 14141 User Name admin Password po ooa lt Back cancel Hep m Enter the host name of a cluster system an IP address of a cluster system or the IP address of the cluster that will join the global environment m Verify the port number m Enter the user name and the password m Click Next If the cluster is running in secure mode Remote Cluster Configuration Wizard New Cluster Details Enter the details of the new cluster to be added The IP address can be the cluster address or the host IP of a node in the new cluster Host Name IP address vesinu Port haan oo C Use connected clusters credentials Enter new credentials User Name foot S Password Domain cancel Hep m Enter the host name of a cluster system an IP address of a cluster system or the IP address of the cluster that will join the global environment Verify the port number m Choose to connect to the remote cluster with the credentials used for the current cluster connection or enter new credentials including the user name password and the domain If you have connected to the remote cluster using the wizard earlier you can use the credentials from the previous connection Administering global clusters from Cluster Manager Java console Click Next Adding a remote cluster 5 Enter the details of the existing remote clusters this information on administrator
148. view in the Groups Listing table click the linked name of the offline service group that you want to bring online 2 Inthe Group Summary view in the Remote Operations task panel click Online 3 Inthe Online Service Group dialog box specify the following details for the task and then click OK m The target cluster Select the cluster on which you want to bring the remote service group online from the Select the cluster you want to online this group on drop down menu The cluster choices are populated using the cluster list for the remote service group m The target system Select the system on which you want to bring the remote service group online from the Select the system you want to online this group on drop down menu The system choices are populated using the system list for the remote service group The Anywhere option causes this task to try each system in the list until the service group is successfully brought online Taking a remote service group offline Manually put a remote service group into an unresponsive nonfunctioning state You must have the role of cluster administrator or service group administrator to take a service group offline You can take a remote service group offline on a specific system or you can take the service group offline anywhere If you select the Anywhere option the service group is taken offline on the first system in the remote cluster on which it is found 498 Administering globa
149. view service group dependencies and get status information 1 On the navigation bar click Cluster 2 On the secondary tab bar click Summary and then click Group Dependency This view presents a graphical representation of all currently configured service groups and the dependencies that exist among them 3 Click a service group icon to get information about the service group that it represents Unlinking service groups Remove dependencies among service groups To unlink two service groups that share a parent child dependency 1 Inthe Cluster Groups view in the Configuration task panel click Unlink Service Groups 2 Inthe Unlink Service Groups dialog box select the parent group and the child group from the corresponding drop down menus and then click OK Adding systems to a Service group Add systems to the system list of a service group You can add systems to the system list of a service group when you add or modify the service groups ina cluster You add or modify service groups using the Service Group Configuration wizard See Adding a service group to a cluster on page 98 See Modifying a service group on page 101 110 Administering the cluster from the Cluster Management Console Administering service groups Deleting systems from a service group Delete systems from the system list of a service group You can delete systems from the system list of a service group when you add or modify the service
150. when it is online it attempts to go offline Typically this is the result of issuing an offline command while resource was waiting to go online WAITING TO GO OFFLINE reverse propagate Same as above but resource propagates the offline operation m Type and dimension integer scalar m Default 1 NOT WAITING LastOnline system use only Indicates the system name on which the resource was last online This attribute is set by VCS m Type and dimension string scalar m Default Not applicable MonitorOnly system use only Indicates if the resource can be brought online or taken offline If set to 0 resource can be brought online or taken offline If set to 1 resource can only be monitored Note This attribute can only be modified by the command hagrp freeze m Type and dimension boolean scalar m Default 0 618 VCS attributes Resource attributes Table D 1 Resource attributes MonitorTimeStats system use only Valid keys are Average and TS Average is the average time taken by the monitor entry point over the last Frequency number of monitor cycles TS is the timestamp indicating when the engine updated the resource s Average value m Type and dimension string association m Default Average 0 TS Name system use only Contains the actual name of the resource m Type and dimension string scalar m Default Not applicable Path system use only Set to
151. you are installing an application supported by a VCS enterprise agent see the installation and configuration guide for the agent Install the agent Agent packages are installed in the global zone as well as the currently existing non global zones The operating system installs the agents in future non global zones when they are created In the zone configure all mount points used by the application If you use a direct mount set the mount points used by the application relative to the zone s root For example if a zone z ora needs to use oracle mount the drive at export home z ora root oracle 600 Configuring VCS in non global zones Configuring VCS in zones Configuring the service group for the application The following diagrams show the dependencies among the resources In one case the zone root is set up on local storage In the other zone root is set up on shared storage Resource dependency diagrams zone root on local disks Figure B 1 Zone root on local disks with loop back file system Application DiskGroup Figure B 2 Zone root on local disks direct mount file system Application DiskGroup Configuring VCS in non global zones 601 Configuring VCS in zones Resource dependency diagrams zone root on shared disks Figure B 3 zone root on shared disks with loop back file system Application e Zone ne Mount Mount DiskGroup DiskGroup Fig
152. 2232 in the non interactive mode hasnap restore n s ves 20030101 22232 The following command restores the snapshot stored in the file tmp backup 2 2 2003 hasnap restore f tmp backup 2 2 2003 Viewing snapshots of configuration files Usethehasnap display command to view details of previously created snapshots To view snapshots of configuration files hasnap display f filename list s snapid m 1 t list Displays the list of snapshots in the repository s Specifies the snapshot ID m Displays snapshot description 1 Lists files in the snapshot t Displays the snapshot timestamp If no options are specified the command displays all information about the latest snapshot Examples The following command lists all snapshots hasnap display list The following command displays the description and the time of creation of the specified snapshot hasnap display s vcs 20030101 2232 m t The following command displays the description the timestamp and the list of all files in the snapshot file tmp backup 2 2 2003 hasnap display f tmp backup 2 2 2003 Viewing files changed after a snapshot Use the hasnap sdiff command to display files that were changed on the local system after a specific snapshot was created To view files that changed after a snapshot Run the following command Administering the cluster from the command line 247 Managing VCS configuration files hasnap
153. 3 HPUX 11 23 IA and PA RISC RHEL 4 Update 3 and SLES9 SP3 Solaris SPARC 2 8 2 9 and 2 10 Windows 2000 Windows XP and Windows 2003 Note Make sure you are using the operating system version that supports JRE 1 5 138 Administering the cluster from Cluster Manager Java console Getting started Verify the configuration has a user account A user account is established during VCS installation that provides immediate access to Cluster Manager If a user account does not exist you must create one See Adding a user on page 174 On UNIX systems you must set the display for Cluster Manager See Setting the display on UNIX systems on page 138 Start Cluster Manager See Starting Cluster Manager Java console on page 139 Add a cluster panel See Configuring a new cluster panel on page 169 Log on to a cluster See Logging on to and off of a cluster on page 171 Make sure you have adequate privileges to perform cluster operations See About the VCS user privilege model on page 61 Setting the display on UNIX systems The UNIX version of the Cluster Manager Java Console requires an X Windows desktop Setting the display is not required on Windows workstations To set the display 1 Type the following command to grant the system permission to display on the desktop xhost Configure the shell environment variable DISPLAY on the system where Cluster Manager will be
154. 56 Restart attribute 637 RestartLimit attribute about 373 definition 626 root broker 28 RVG agent 467 RVGPrimary agent 467 RVGSnapshot agent 467 S saving a configuration 243 scalar attribute dimension 52 scheduling classes 556 defaults 557 priority ranges 557 ScriptClass attribute 626 scripting VCS commands 284 ScriptPriority attribute 626 SCSI III Persistent Reservations 345 search conducting 132 secure VCS See VERITAS Security Services server credentials viewing 167 service group attributes 628 service group dependencies about 409 autorestart 365 benefits of 410 categories of 411 creating 424 creating from Java Console 192 limitations of 413 manual switch 425 removing from Java Console 194 service group workload management Capacity and Load attributes 391 load policy 366 load based autostart 367 overload warning 392 sample configurations 394 SystemZones attribute 366 service groups adding from Cluster Management Console 98 101 adding from command line 261 adding from Java Console 178 administering from Cluster Management Console 98 administering from command line 261 administering from Java Console 178 Index 697 service groups continued autoenabling from Java Console 190 bringing online from Cluster Management Console 104 bringing online from command line 263 bringing online from Java Console 182 clearing from Cluster Management Console 111 creating using configuration wizard 196 deleting from
155. 66 LoadTimeCounter attribute 644 LoadTimeThreshold attribute 645 loadwarning event trigger 449 LoadWarningLevel attribute 645 local attributes 54 LockMemory attribute 651 LogDbg attribute 623 LogFileSize attribute 623 Logging 69 logging agent log 564 engine log 564 message tags 564 VRTSweb 673 logs customizing display in Java Console 228 for VRTSweb 673 searching from Java Console 228 viewing from Java Console 166 LogSize attribute 652 main cf about 46 cluster definition 47 group dependency clause 48 include clauses 47 resource definition 47 resource dependency clause 47 sample configuration 48 service group definition 47 system definition 47 ManageFaults attribute about 368 definition 633 ManualOps attribute 633 message tags about 564 MigrateQ attribute 634 MonitorInterval attribute 623 MonitorOnly attribute 617 MonitorStartParam attribute 624 MonitorTimeout attribute 624 MonitorTimeStats attribute 618 multinicb event trigger 450 N 1 configuration 38 Name attribute 618 navigation online help 91 network failure 158 network links detecting failure 553 networks detecting failure 554 NFS service groups configuring 309 NoAutoDisable attribute 645 Nodeld attribute 645 nofailover event trigger 450 notification about 430 deleting messages 432 error messages 431 error severity levels 431 event triggers 446 hanotify utility 434 message queue 432 notifier process 433 setting using wizard 225 SN
156. A green link indicates a hard dependency typically used with VVR in disaster recovery configurations To access the Service Group view 1 From Cluster Explorer click a cluster in the configuration tree 2 Inthe view panel click the Service Groups tab Administering the cluster from Cluster Manager Java console 155 About Cluster Explorer Resource view The Resource view displays the resources in a service group Use the graph and ScreenTips in this view to monitor the dependencies between resources and the status of the service group on all or individual systems in a cluster 2 VCSHP10_71_0214a Resource View ClusterService on all systems we ce ClusterSer veshp71 veshpi0 In the graph the line between two resources represents a dependency or parent child relationship Resource dependencies specify the order in which resources are brought online and taken offline During a failover process the resources closest to the top of the graph must be taken offline before the resources linked to them are taken offline Similarly the resources that appear closest to the bottom of the graph must be brought online before the resources linked to them can come online m A resource that depends on other resources is a parent resource The graph links a parent resource icon to a child resource icon below it Root resources resources without parents are displayed in the top row m A resource on which the other res
157. AB tries to kill HAD by sending a SIGABRT signal If it does not succeed GAB sends a SIGKILL and closes the port By default GAB tries to kill HAD five times before closing the port The number of times GAB tries to kill HAD is a kernel tunable parameter gab_kill_ntries and is configurable The minimum value for this tunable is 3 and the maximum is 10 This is an indication to other nodes that HAD on this node has been killed Should HAD recover from its stuck state it first processes pending signals Here it will receive the SIGKILL first and get killed After sending a SIGKILL GAB waits for a specific amount of time for HAD to get killed If HAD survives beyond this time limit GAB panics the system This time limit is a kernel tunable parameter gab_isolate_time and is configurable The minimum value for this timer is 16 seconds and maximum is 4 minutes Network failure If anetwork partition occurs a cluster can split into two or more separate sub clusters When two clusters join as one VCS designates that one system be ejected GAB prints diagnostic messages and sends iofence messages to the system being ejected The system receiving the iofence messages tries to kill the client process The k option applied here If the j option is used in gabconfig the system is halted when the iofence message is received Quick reopen If a system leaves cluster and tries to join the cluster before the new cluster is configured default is
158. ARS Section 227 7202 Symantec Corporation 20330 Stevens Creek Blvd Cupertino CA 95014 www symantec com Third party legal notices Third party software may be recommended distributed embedded or bundled with this Symantec Corporation product Such third party software is licensed separately by its copyright holder All third party copyrights associated with this product are listed in the accompanying release notes AIX is a registered trademark of IBM Corporation HP UX is a registered trademark of Hewlett Packard Development Company L P Linux is a registered trademark of Linus Torvalds Solaris is a trademark of Sun Microsystems Inc Windows is a registered trademark of Microsoft Corporation Licensing and registration Veritas Cluster Server is a licensed product See the Veritas Cluster Server Installation Guide for license installation instructions Technical support For technical assistance visit http support veritas com and select phone or email support Use the Knowledge Base search feature to access resources such as TechNotes product alerts software downloads hardware compatibility lists and our customer email notification service Contents Section Chapter 1 Chapter 2 Chapter 3 Section II Chapter 4 Clustering concepts and terminology Introducing Veritas Cluster Server What isa VCS cluster isscectaitia i acindiciis tenis ieric weevil cniad 16 Can my application be clustered
159. About membership arbitration 00 0 0 cccccescssessesceseseeseescesesecsecseeseseceensees 341 About data protecion esson E S 345 Examples of VCS operation with I O fencing cceeseseeeeeseeseetseeeeeees 346 About cluster membership and data protection without I O fencing 351 Examples of VCS operation without I O fencing c cccceeesesesssssseeeeeeees 353 Summary of best practices for cluster communications ceeeeeees 358 Chapter 11 Controlling VCS behavior About VCS behavior on resource faults oe ceeeeececeseseseeeseseceseseseeseseseneneeeees 362 Controlling VCS behavior at the service group level cecseseeeeeeeeee 365 Controlling VCS behavior at the resource level ccceseesesesesseseteeeeseteeeees 373 Changing agent file paths and binaries 0 0 0 0 ccccesesesetseseceeeeeseseseseeeeeeees 387 VCS behavior on loss of storage connectivity 0 eeeeesesseseseseeeseseseseseeees 388 Service group workload management ccececesesseseseeeeseseseeseeeeceseeeeseseeeees 391 Sample configurations depicting workload management cceeeee 394 Chapter 12 The role of service group dependencies About service group dependencies ccececseesesseseseeseseseeceseeseseseeseseseeeeseeees Service group dependency configurations cccssesesecececeseseeseseeseseeeees Group Dependency FAQS ccsecsssssssssseseseesesesesceseceseseeseseeeeseseeseseeeeseeeseseeaees Linking Servic
160. About the main tab bar The main tab bar contains a tab representing a set of views for each category of operations available in the Cluster Management Console The tabs are Manage management for cluster object configuration and manipulation and Search for conducting and saving character based searches on management server and cluster objects Figure 5 3 Main tab bar anane EST Clicking a tab takes you to the initial view for that tab The main tab bar always contains the same tabs About the search bar The search bar contains a text edit box next to a Search button Figure 5 4 Search bar Po Search The search bar enables you to find any management server object or cluster object in the management server database About the task pane The left pane of the console is the task pane The task pane contains one or more task panels depending upon the view selected In the startup view the task pane contains the Alerts and Configuration task panels The task pane can contain as many as four task panels Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console Figure 5 5 Task panels in the task pane Y Groups Filter a is By Consolidated State M amp Online M amp Offline M amp Partial M OG Faulted M Fal Unknown M Global groups Apply Reset F Operations ma Online Anyvhere Offline Anyvhere Clear Fault Freeze Unfreeze ie Configuration
161. Administering the cluster from the Cluster Management Console Administering resources To fault one or more resources 1 Inthe Cluster Resources view in the Resources Listing table check the check box preceding the line item for each resource that you want to fault Select all resources by checking the check box at the top 2 Inthe Simulation task panel click Fault Resources In the Fault Resource s dialog box use the appropriate drop down menu to select the systems on which you want to fault each resource Each drop down menu is populated using the system list of each resource The system list of a resource contains a list of all systems on which the resource is configured 3 Click OK Clearing a faulted resource Clear a resource to remove a fault and make the resource available to go online You must have the role of cluster operator or service group operator to clear a faulted resource A resource fault can result from occurences such as a power failure or a faulty configuration To clear a single faulted resource 1 Inthe Cluster Resources view in the Resources Listing table click the linked name of the faulted resource that you want to clear 2 Inthe Resource Summary view in the Operations task panel click Clear Fault To clear one or more faulted resources 1 Inthe Cluster Resources view in the Resources Listing table check the check box preceding the line item for each resource that you want to clear Select all re
162. CS They include agents for Disk Mount IP and various other resource types See the Veritas Cluster Server Bundled Agents Reference Guide for a complete list Enterprise agents Enterprise agents control third party applications and are licensed separately These include agents for Oracle Sybase and DB2 Contact your sales representative for more information Custom agents Custom agents can be developed by you or by Symantec consultants Typically agents are developed because the user requires control of an application that the current bundled or enterprise agents do not support See the Veritas Cluster Server Agent Developer s Guide for information on developing a custom agent About the VCS agent framework The VCS agent framework is a set of common predefined functions that are compiled into each agent These functions include the ability to connect to the VCS engine HAD and to understand common configuration attributes The agent framework frees the developer from developing support functions that are required by the cluster and instead focus on controlling a specific resource type For more information on developing agents see the Veritas Cluster Server Agent Developer s Guide Introducing Veritas Cluster Server 27 Logical components of VCS About cluster control communications and membership Cluster communications ensure that VCS is continuously aware of the status of each system s service groups and resources The
163. CS configuration concepts on page 45 14 Clustering concepts and terminology Chapter Introducing Veritas Cluster Server What is a VCS cluster Can my application be clustered Physical components of VCS Logical components of VCS Putting the pieces together 1 16 Introducing Veritas Cluster Server What is a VCS cluster What is a VCS cluster Veritas Cluster Server VCS from Symantec connects multiple independent systems into a management framework for increased availability Each system or node runs its own operating system and cooperates at the software level to form a cluster VCS links commodity hardware with intelligent software to provide application failover and control When a node or a monitored application fails other nodes can take predefined actions to take over and bring up services elsewhere in the cluster How VCS detects failure VCS detects failure of an application by issuing specific commands tests or scripts to monitor the overall health of an application VCS also determines the health of underlying resources supporting the application such as file systems and network interfaces VCS uses a redundant network heartbeat to discriminate between the loss of a system and the loss of communication between systems VCS also uses SCSI3 based membership coordination and data protection for detecting failure on a node and on fencing See About cluster control communications and memb
164. CS performance considerations 553 How cluster operations affect performance When a system fails When a system crashes or is powered off it stops sending heartbeats to other systems in the cluster By default other systems in the cluster wait 21 seconds before declaring it dead The time of 21 seconds derives from 16 seconds default timeout value for LLT peer inactive timeout plus 5 seconds default value for GAB stable timeout The default peer inactive timeout is 16 seconds and can be modified in the etc Ilttab file For example to specify 12 seconds set timer peerinact 1200 Note After modifying the peer inactive timeout you must unconfigure then restart LLT before the change is implemented To unconfigure LLT type litconfig u To restart LLT type lltconfig c GAB stable timeout can be changed by specifying gabconfig t timeout_value_milliseconds Though this can be done we do not recommend changing the values of the LLT peer inactive timeout and GAB stable timeout If a system reboots it becomes unavailable until the reboot is complete The reboot process kills all processes including HAD When the VCS process is killed other systems in the cluster mark all service groups that can go online on the rebooted system as autodisabled The AutoDisabled flag is cleared when the system goes offline As long as the system goes offline within the interval specified in the ShutdownTimeout value VCS treats this as a system reb
165. Clean Timeoutf300 Stop Timeoutf300 8 Cancel m Click OK m Click OK on the Heartbeat configuration dialog box To add a cluster heartbeat from Command Center 1 Click Commands gt Configuration gt Cluster Objects gt Add Heartbeat 2 Enter the name of the heartbeat 3 Click Apply Administering global clusters from Cluster Manager Java console 519 Administering global heartbeats Modifying a global heartbeat 1 From Cluster Explorer click Configure Heartbeats on the Edit menu 2 Inthe Heartbeat Configuration dialog box T Heartbeat configuration x ma Heartbeat Configuration C New Heartbeat Existing Heartbeat Remote Cluster m Click Existing Heartbeat Click the name of the existing heartbeat from the menu m Select or clear the check box next to the name of a cluster to add or remove it from the cluster list for the heartbeat m If necessary click the icon in the Configure column to open the Heartbeat Settings dialog box Otherwise proceed to the last step m Change the values of the Arguments attribute and various timeout and interval fields Click to add an argument value click to delete it S Heartbeat Settings Heartbeat type clus_hb Cluster Name C2 Arguments ft0 212 88 189 Are You Alive settings Interval 60 Timeout 300 Retry Limit 3 Timeout settings Init Timeout 300 Clean Timeut 300 Start Timeoutf3o0
166. Cluster Management Console 100 deleting from command line 261 deleting from Java Console 181 disabling from Cluster Management Console 108 disabling from Java Console 189 displaying dependencies from command line 254 enabling from Cluster Management Console 107 enabling from Java Console 188 flushing from command line 266 flushing from Java Console 191 freezing from command line 264 freezing from Java Console 186 linking from Java Console 192 querying from command line 254 switching from command line 263 switching from Java Console 185 taking offline from Java Console 184 taking remote groups offline 515 troubleshooting 568 unfreezing from command line 264 unfreezing from Java Console 187 unlinking from Java Console 194 shared nothing configuration 42 shared storage replicated data configuration 43 ShutdownTimeout attribute 646 Signaled attribute 619 Simulator See VCS Simulator SMTP notification configuring for VRTSweb 667 SMTP server retrieving name of 667 SNMP 430 files for notification 439 HP OpenView 439 merging events with HP OpenView NNM 439 SNMP supported consoles 430 698 Index SourceFile attribute for clusters 653 for resource types 626 for service groups 637 for systems 646 split brain in global clusters 468 ssh configuration for Java Console 138 Start attribute 619 Starting 233 State attribute for resources 619 for service groups 637 steward process about 468 configuring 478 Stewards attribute 653 st
167. Commands gt Configuration gt Cluster Objects gt Add Service Group or Click Add service group in the Command Center toolbar 2 Enter the name of the service group VCSHP10_71_0214a Command Center admin Cluster Administrator File View Help RBM BEA AF AGO BF ea Add Service Group Service Group name TestGrp Available Systems Systems for Service Group lt a vy Q Delete haa ce GM Delete System gt E Attributes 5 Dependencies al E Operations Service Group Type Failover C Paralel C Hybrid Selected Template None Templates _ arp add TestGrp sarp modify TestGrp Systemlist veshp10 0 sarp modify TestGrp AutoStartList veshp10 aarp modify TestGrp Parallel O L Lle 3 Inthe Available Systems box click the systems on which the service group will be added 4 Click the right arrow to move the selected systems to the Systems for Service Group box The priority number starting with 0 is automatically assigned to indicate the order of systems on which the service group will start in case of a failover If necessary double click the entry in the Priority column to enter a new value 180 Administering the cluster from Cluster Manager Java console Administering service groups Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every tim
168. DNS DNS server File shares m NFS shares Share NFS NFSRestart 294 Configuring applications and resources in VCS Which agents should use Table 8 1 VCS bundled agents m Samba shares Samba suite SambaServer Linux SambaShare NetBIOS m Apache Web server Apache m Application Application m Process Process Or ProcessOnOnly m Solaris 10 zone Zone VCS infrastructure and support m VCS notifier NotifierMgr m Application run using Veritas Web Server VRTSWebApp m VCS resource in another service group Proxy m Service group in another cluster RemoteGroup Configuring applications and resources in VCS 295 Which agents should use 296 Configuring applications and resources in VCS Configuring the RemoteGroup agent Configuring the RemoteGroup agent The RemoteGroup agent monitors and manages service groups in a remote cluster Use the RemoteGroup agent to establishes dependencies between applications that are configured on different VCS clusters For example you configure an Apache resource in a local cluster and an Oracle resource in a remote cluster In this example the Apache resource depends on the Oracle resource You can use the RemoteGroup agent to establish this dependency between these two resources See the Veritas Cluster Server Bundled Agents Reference Guide for more information about the agent and its attributes See Adding a RemoteGroup resource from the Java Con
169. Enabled from the menu To enable all resources in a service group from Cluster Explorer 1 From Cluster Explorer click the Service Groups tab in the configuration tree 2 Right click the service group 3 Click Enable Resources To enable all resources in a service group from Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Availability gt Enable Resources for Service Group 2 Click the service group 3 Click Apply 210 Administering the cluster from Cluster Manager Java console Administering resources Disabling resources in a service group Disable resources in a service group to prevent them from coming online This disabling process is useful when you want VCS to temporarily ignore resources rather than delete them while the service group is still online To disable an individual resource in a service group 1 From Cluster Explorer click the Service Groups tab in the Cluster Explorer configuration tree 2 Right click a resource in the configuration tree An enabled resource will display a check mark next to the Enabled option that appears in the menu 3 Click Enabled from the menu to clear this option To disable all resources in a service group from Cluster Explorer 1 From Cluster Explorer click the Service Groups tab in the configuration tree 2 Right click the service group and click Disable Resources To disable all resources in a service group fro
170. File attribute For example to use specify the For example if the MyApplication agent is installed at opt VRTSvcs bin CustomAgents MyApplication specify this path as the attribute value If the agent file is not named MyApplicationAgent configure the AgentFile attribute If you do not set these attributes and the agent is not available at its default location VCS looks for the agent at the opt VRTSagents ha bin AgentName AgentNameAgent To change the path of an agent Before configuring a resource for the agent add AgentFile and AgentDirectory as static attributes to the agent s resource type haattr add static resource type AgentFile binary name haattr add static resource type AgentDirectory complete path to agent binary 388 Controlling VCS behavior VCS behavior on loss of storage connectivity VCS behavior on loss of storage connectivity When a node loses connectivity to shared storage input output operations I O to volumes return errors and the disk group gets disabled In this situation VCS must fail the service groups over to another node This failover is to ensure that applications have access to shared storage The failover involves deporting disk groups from one node and importing them to another node However pending I Os must complete before the disabled disk group can be deported Pending I Os cannot complete without storage connectivity VCS assumes data is being read from or
171. Highlights the string occurrences in the search result tables Combine Exact Match and Match All Terms to minimize the number of candidate occurrences 5 Visit each search view to see the hits in each object category Your first results are displayed in the same view in which you started the search Click the other secondary tabs Clusters Groups and so on to see any hits the other search views Note After you perform a search changing the Exact Match Match All Terms and Highlight option settings automatically updates the results 134 Administering the cluster from the Cluster Management Console Conducting a search Chapter Administering the cluster from Cluster Manager Java console m About the Cluster Manager Java Console m Getting started m Reviewing components of the Java Console m Icons in the Java Console m About Cluster Monitor m About Cluster Explorer m Accessing additional features of the Java Console m Administering Cluster Monitor m Administering user profiles m Administering service groups m Administering resources m Importing resource types m Administering systems m Administering clusters m Executing commands m Editing attributes m Querying the cluster configuration m Setting up VCS event notification using the Notifier wizard 136 Administering the cluster from Cluster Manager Java console m Administering logs m Administering VCS Simulator Administering the cluster f
172. Limit OWL attribute This attribute defines how many monitor cycles can return an offline status before the agent framework declares the resource faulted Each successive Monitor cycle increments the OnlineWaitCount OWC attribute When OWL OWC or if OWL 0 VCS determines the resource has faulted VCS then examines the value of the ManageFaults attribute If the ManageFaults is set to NONE the resource state changes to OFFLINE ADMIN_WAIT If the ManageFaults is set to ALL VCS calls the Clean entry point with the CleanReason set to Online Ineffective If the Clean entry point is not successful exit code 1 the agent monitors the resource It determines the resource is offline and calls the Clean entry point with the Clean Reason set to Online Ineffective This cycle continues till the Clean entry point is successful after which VCS resets the OnlineWaitCount value If the OnlineRetryLimit ORL is set to a non zero value VCS increments the OnlineRetryCount ORC and invokes the Online entry point This starts the Controlling VCS behavior 379 Controlling VCS behavior at the resource level cycle all over again If ORL ORC or if ORL 0 VCS assumes that the Online operation has failed and declares the resource as faulted 380 Controlling VCS behavior Controlling VCS behavior at the resource level Resource Offline Online Resource Waiting to Online Resource Online Resource Offline Admin_Wait resadminwait
173. MP files 439 troubleshooting 573 Notifier attribute 652 notifier process 433 Notifier Resource Configuration wizard 225 N to 1 configuration 36 N to N configuration 40 NumRetries attribute 634 NumThreads attribute definition 625 modifying for performance 548 0 OfflineMonitorInterval attribute 625 OfflineTimeout attribute 625 OnGrpCnt attribute 645 online help accessing 91 locating topics 91 navigating 91 OnlineRetryInterval attribute 634 OnlineRetryLimit attribute for resource types 625 for service groups 634 OnlineTimeout attribute 625 OnlineWaitLimit attribute 626 On Off resource 22 On Only resource 22 OpenTimeout attribute 626 Operations attribute 626 OperatorGroups attribute for clusters 652 for service groups 634 Operators attribute for clusters 652 for service groups 634 overload warning for SGWM 392 Index 695 P PanicOnNoMem attribute 652 Parallel attribute 635 passwords changing from Java Console 175 Path attribute 618 PathCount attribute 635 performance agents 547 GAB 546 HAD 547 impact of VCS 546 Java Console 549 modifying entry points 548 modifying NumThreads attribute 548 monitoring CPU usage 559 when a cluster is booted 550 when a resource fails 551 when a resource is brought online 550 when a resource is taken offline 551 when a service group fails over 555 when a service group is brought online 551 when a system fails 553 Persistent resource 22 postoffline event trigger 451 poston
174. Mount resource for the application is set to export home oradata m Usea direct mount file system In this condition all file system mount points used by the application running in a zone must be set relative to the zone s root For example if the Oracle application uses oradata and you create the zone with the zonepath as z_ora then the mount must be z ora root oradata The MountPoint attribute of the Mount resource must be set to this path Using custom agents in non global zones If you use custom agents to monitor applications running in zones make sure the agents use script based entry points VCS does not support running C entry points inside a non global zone If the custom agent monitors an application running in a zone add the resource type to the APP_TYPES environment variable If the custom agent monitors an application running in the global zone add the resource type to the SYS_TYPES environment variable Configuring VCS in non global zones 595 Configuring VCS in zones Setting the zone root path Each non global zone has its own section of the file system hierarchy in the zone root directory Processes running in the zone can access files only within the zone root You can set the zone root in two ways m Zone root on local storage In this configuration you must create a zone on each node in the cluster m Zone root on shared storage In this configuration create a zone in shared storage from one s
175. Name Ethernet Name vessunst oreo Discover NiC More information lt Back Next gt cancer Select the Does application require virtual IP check box if required From the Virtual IP Address list select the virtual IP for the service group Click Discover IP to discover IP addresses configured after wizard was started Note that the wizard discovers all IP addresses that existed when you started the wizard For example if you delete an IP address after starting the wizard and click Discover IP the wizard displays the deleted IP addresses in the Virtual IP Address list For each system specify the associated ethernet Click Discover NIC if required Click Next 308 Configuring applications and resources in VCS Configuring application service groups 12 Specify whether you want to configure more applications in the service group If you want to add more applications to the service group select the Configure more applications check box Click Next Note If you choose to configure more applications the wizard displays the Application Options dialog box See step 7 on page 303 13 Configure application dependencies if you chose to do so VCS Application Configuration Wizard Application Dependency Configure application dependency Select Application app x Available Applications Child Applications 6 app 2 More Information lt Back Next gt Cancel
176. Propagation 0 In the following example the FaultPropagation attribute is set to 0 When resource R2 faults the Clean entry point is called and the resource is marked as faulted The fault is not propagated up the tree and the group is not taken offline Controlling VCS behavior 373 Controlling VCS behavior at the resource level Controlling VCS behavior at the resource level This section describes how you can control VCS behavior at the resource level Note that a resource is not considered faulted until the agent framework declares the fault to the VCS engine About resource type attributes that control resource behavior The following attributes affect how the VCS agent framework reacts to problems with individual resources before informing the fault to the VCS engine About the RestartLimit attribute The RestartLimit attribute defines whether VCS attempts to restart a failed resource before informing the engine of the fault If the RestartLimit attribute is set to a non zero value the agent attempts to restart the resource before declaring the resource as faulted When restarting a failed resource the agent framework calls the Clean entry point before calling the Online entry point However setting the ManageFaults attribute to NONE prevents the Clean entry point from being called and prevents the Online entry point from being retried About the OnlineRetryLimit attribute The OnlineRetryLimit attribute
177. RTSvcs conf VVRTypes cf m Click Import Configure the RVG resource See the VVR documentation for more information about the resource Note that the RVG resource starts stops and monitors the RVG in its current state and does not promote or demote VVR when you want to change the direction of replication That task is managed by the RVGPrimary agent Set dependencies as per the following information m RVG resource depends on the IP resource m RVG resource depends on the DiskGroup resource m IP resource depends on the NIC resource 474 Connecting clusters Creating global clusters Setting up a global cluster The service group now looks like T vcspri o xj EA Resource Yiew appgroup_rep on all systems PUR Si Auto arrange P 22 appgroup thori65 thori4 7 Delete the DiskGroup resource from the appgroup service group 8 Inthe appgroup service group add a resource of type RVGPrimary and configure its attributes m RVGResourceName The name of the RVG resource that this agent will promote m AutoTakeover A flag that indicates whether the agent should perform a takeover in promoting a Secondary RVG if the original Primary is down Default is 1 meaning a takeover will be performed m AutoResync A flag that indicates whether the agent should configure the RVG to perform an automatic resynchronization after a takeover and once the original Primary is restored Default is 0 meaning automatic
178. RTSvesplatformagent_name bmc gcoconfig messages hawizard messages Heartbeat framework messages VCS trigger messages Wide area connector process messages Fencing messages GAB command line interface messages hagetcf messages LLT command line interface messages Agent framework messages VCS alert messages VCS API messages Common modules messages VCS engine HAD messages VCS bundled agent messages VCS enterprise agent messages 566 Troubleshooting and recovery for VCS Troubleshooting the VCS engine Troubleshooting the VCS engine HAD diagnostics When the VCS engine HAD dumps core the core is written to the directory VCS_DIAG diag had The default value for variable VCS_DIAG is var VRTSves When HAD core dumps review the contents of the vcs_DIAG diag had directory See the following logs for more information m Operating system console log m Engine log m hashadow log VCS runs the script opt VRTSvcs bin vcs_diag to collect diagnostic information when HAD and GAB encounter heartbeat problems The diagnostic information is stored in the vcs_DIAG diag had directory When HAD starts it renames the directory to had times tamp where timestamp represents the time at which the directory was renamed DNS configuration issues cause GAB to kill HAD If HAD is periodically killed by GAB for no apparent reason review the HAD core files for any DNS resolver functions res_send res_query res_search
179. RTSweb certificate for another application or Web server Visit http java sun com for more information about keystores To clone the VRTSweb SSL certificate Run the following command VRTSWEB HOME bin webgui cert clone keystore storepass alias keypass If a clone keystore exists the command renames it to keystore old If the VRTSweb SSL certificate does not exist the command prompts you to create one For example opt VRTSweb bin webgui webgui cert clone myapp myserv keystore mystorepass myalias mykeypass Administering Symantec Web Server 667 Configuring SMTP notification for VRTSweb Configuring SMTP notification for VRTSweb You can configure VRTSweb to send out email notifications about events that are associated with the Web server For example m The Web server is starting stopping severity INFORMATION m The Web console is starting stopping severity INFORMATION m The Web server s allocated heap size very close to the maximum allowed severity SEVERE To send an email notification VRTSweb needs to know the IP address or hostname of a configured SMTP server The SMTP server address applies to all Web consoles running on the Web server So you do not need to configure the SMTP server at multiple places Retrieving the name of the configured SMTP server Retrieve the name of the SMTP server configured for VRTSweb To retrieve the name of the SMTP server from the command line Run the following command
180. Share2 G vNFS_Mount1 vNFS_NFS vNFS_Mount2 vNFS_Volumet vNFS_Volume2 vNFS_DiskGroup_ To access the template view From Cluster Explorer click Templates on the Tools menu Administering the cluster from Cluster Manager Java console 161 Accessing additional features of the Java Console System Manager Use System Manager to add and remove systems in a service group s system list A priority number starting with 0 is assigned to indicate the order of systems on which the service group will start in case of a failover If necessary double click the entry in the Priority column to enter a new value Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every time HAD is started T System Manager x we Service Group newgroup2 Available Systems Systems for Service Group vcshp71 Systemname startup Priority 0 vcshp10 gt pio fv i 4 a To access system Manager From Cluster Explorer click the service group in the configuration tree and click System Manager on the Tools menu or In the Service Groups tab of the Cluster Explorer configuration tree click a service group and click Manage systems for a Service Group on the toolbar 162 Administering the cluster from Cluster Manager Java console Accessing additional features of the Java Console User Manag
181. Stop Timeoutf300 m Click OK m Click OK on the Heartbeat Configuration dialog box 520 Administering global clusters from Cluster Manager Java console Administering global heartbeats Deleting a global heartbeat Note You cannot delete the last heartbeat between global clusters To delete a cluster heartbeat from Command Center 1 Click Commands gt Configuration gt Cluster Objects gt Delete Heartbeat 2 Click the heartbeat to delete 3 Click Apply Chapter 1 8 Administering global clusters from the command line Global querying Administering global service groups Administering resources Administering clusters Administering heartbeats For remote cluster operations you must configure a VCS user with the same name and privileges in each cluster See User privileges in global clusters on page 64 522 Administering global clusters from the command line Global querying Global querying VCS enables you to query global cluster objects including service groups resources systems resource types agents and clusters You may enter query commands from any system in the cluster Commands to display information on the global cluster configuration or system states can be executed by all users you do not need root privileges Only global service groups may be queried Querying global cluster service groups To display service group attribute values across clusters hagrp value service group attribu
182. Therefore a service group can only run on all systems in the cluster if the storage is shared across all systems In many configurations a storage area network SAN provides this requirement 1 0 fencing technology can be used for data protection to block access to shared storage from any system that is not a current and verified member of the cluster See About cluster topologies on page 33 Networking in the cluster is used for the following purposes m Communications between the cluster nodes and the customer systems m Communications between the cluster nodes Introducing Veritas Cluster Server 21 Logical components of VCS Logical components of VCS VCS is comprised of several components that provide the infrastructure to cluster an application Resources and resource dependencies Resources are hardware or software entities that make up the application Resources include disk groups and file systems network interface cards NIC IP addresses and applications Resource dependencies indicate resources that depend on each other because of application or operating system requirements Resource dependencies are graphically depicted in a hierarchy also called a tree where the resources higher up parent depend on the resources lower down child Figure 1 1 shows the hierarchy for a database application Figure 1 1 Sample resource dependency graph Application requires database and IP address Application
183. VCS_HOST VCS node on which ha commands will be run VCS_GAB_PORT GAB port to which VCS connects Default h VCS_GAB_TIMEOUT Timeout in milliseconds for HAD to send heartbeats to GAB Default 15000 Note If the specified timeout is exceeded GAB kills HAD and all active service groups on system are disabled VCS_HAD_RESTART_TIMEO Set this variable to designate the amount of time the UT hashadow process waits sleep time before restarting HAD Default 0 VCS_LOG Root directory for log files and temporary files Default var VRTSvcs Note If this variable is added or modified you must reboot the system to apply the changes VCS configuration concepts 57 VCS environment variables Table 3 1 VCS environment variables VCS_SERVICE Name of configured VCS service Default vcs Note The specified service should be configured before starting the VCS engine HAD If a service is not specified the VCS engine starts with port 14141 VCS_TEMP_DIR Directory in which temporary information required by or generated by hacf is stored Default var VRTSvcs This directory is created in tmp under the following conditions m The variable is not set m The variable is set but the directory to which it is set does not exist m The utility hacf cannot find the default location Defining VCS environment variables Define VCS environment variables in the file vcsenv which is located at the path opt VRTSvcs bin
184. VERE gt locale lt en any other installed _locale gt m The variable email represents the email address of the new recipient m The optional attribute severity represents the threshold for receiving Web server events It can assume one of the following values INFO WARN ERROR SEVERE If no value is specified for this attribute it takes the default ERROR level m The optional attribute locale specifies the locale in which the notification is to be sent If no value is specified for this attribute it takes the default locale of the system For example opt VRTSweb bin webgui smtp addrcpt admin company com severity INFO locale ja_JP opt VRTSweb bin webgui smtp addrcpt admin company com severity ERROR opt VRTSweb bin webgui smtp addrcpt admin company com To add an SMTP recipient from the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 Click Add SMTP Recipient on the left side of the Configuration page 4 Administering Symantec Web Server Configuring SMTP notification for VRTSweb In the Add SMTP Recipient dialog box Configuration Add SMTP Recipient This page lets you configure an email recipient who will receive notifications about events occuring in the web server A valid SMTP Server needs to be configured before notifications can be sent out Email Severity eror M Locale English x Username
185. abled m Type and dimension integer scalar m Default Not applicable 648 VCS attributes Cluster attributes Cluster attributes Table D 5 lists the cluster attributes Table D 5 Cluster attributes AdministratorGroups List of operating system user account groups that have user defined administrative privileges on the cluster This attribute is valid only in secure clusters m Type and dimension string keylist m Default Administrators user defined Contains list of users with Administrator privileges m Type and dimension string keylist m Default AutoStartTimeout user defined If the local cluster cannot communicate with one or more remote clusters this attribute specifies the number of seconds the VCS engine waits before initiating the AutoStart process for an AutoStart global service group m Type and dimension integer scalar m Default 150 seconds BackupInterval user defined Time period in minutes after which VCS backs up the configuration files The value 5 indicates VCS backs up configuration files every 5 minutes You must set the configuration to read write to enable backups The value 0 indicates VCS does not back up configuration files Set this attribute to at least 3 See Scheduling automatic backups for VCS configuration files on page 243 m Type and dimension integer scalar m Default 0 ClusState system use only Indicates the cur
186. acknowledged by the server and all others are contained in the redo logs If an application cannot recover gracefully after a server crashes it cannot run in a cluster environment The takeover server cannot start up because of data corruption and other problems External data storage The application must be capable of storing all required data and configuration information on shared disks The exception to this rule is a true shared nothing cluster See Shared nothing cluster on page 42 To meet this requirement you may need specific setup options or soft links For example a product may only install in usr local This limitation would require linking usr local to a file system mounted from the shared storage device or mounting file system from the shared device on usr local The application must also store data to disk rather than maintaining it in memory The takeover system must be capable of accessing all required information This requirement precludes the use of anything inside a single system inaccessible by the peer such as NVRAM accelerator boards and other disk caching mechanisms that are contained in a local host Licensing and host name issues The application must be capable of running on all servers that are designated as potential hosts This means strict adherence to licensing requirements and host name dependencies Changing host names can lead to significant management issues when multiple systems have the
187. add type definitions in their own files include types cf Cluster definition Defines the attributes of the cluster including the cluster name and the names of the cluster users cluster demo UserNames admin cDRpdxPmHzpsS n See Cluster attributes on page 648 System definition Lists the systems designated as part of the cluster The system names must match the name returned by the command uname a Each service group can be configured to run on a subset of systems defined in this section system Server1 system Server2 Service group definition Service group definitions in main cf comprise the attributes of a particular service group group NFS_group1 SystemList Serverl Server2 AutoStartList Serverl See Service group attributes on page 628 See About the SystemList attribute on page 48 Resource definition Defines each resource used in a particular service group Resources can be added in any order and the utility hacf arranges the resources alphabetically the first time the configuration file is run DiskGroup DG_sharedl DiskGroup shared1 Resource dependency clause Defines a relationship between resources A dependency is indicated by the keyword requires between two resource names IP_nfsl requires Share_home See Resources and resource dependencies on page 21 48 VCS configuration concepts About the main cf file m Service group dependency
188. al user defined Indicates whether a fault of this resource should trigger a failover of the entire group or not If Critical is 0 the resource fault will not cause group failover m Type and dimension boolean scalar m Default 1 Enabled user defined Indicates agents monitor the resource If a resource is created dynamically while VCS is running you must enable the resource before VCS monitors it For more information on how to add or enable resources see the chapters on administering VCS from the command line and graphical user interfaces When Enabled is set to 0 it implies a disabled resource See Troubleshooting VCS startup on page 567 m Type and dimension boolean scalar Default If you specify the resource in main cf prior to starting VCS the default value for this attribute is 1 otherwise it is 0 616 VCS attributes Resource attributes Table D 1 Resource attributes Flags system use only Provides additional information for the state of a resource Primarily this attribute raises flags pertaining to the resource Values NORMAL Standard working order RESTARTING The resource faulted and that the agent is attempting to restart the resource on the same system STATE UNKNOWN The latest monitor call by the agent could not determine if the resource was online or offline MONITOR TIMEDOUT The latest monitor call by the agent was terminated because it exceeded th
189. al cluster Creating the global service group Configure the Oracle service group appgroup as a global group by running the Global Group Configuration wizard To create the global service group 1 9 In the service group tree of Cluster Explorer right click the application service group appgroup Select Configure and Global from the menu Enter the details of the service group to modify appgroup From the Available Clusters box click the clusters on which the group can come online The local cluster is not listed as it is implicitly defined to be part of the ClusterList Click the right arrow to move the cluster name to the ClusterList box Select the policy for cluster failover m Manual prevents a group from automatically failing over to another cluster m Auto enables a group to automatically fail over to another cluster if it is unable to fail over within the cluster or if the entire cluster faults m Connected enables a group to automatically fail over to another cluster if it is unable to fail over within the cluster Click Next Enter or review the connection details for each cluster Click the Configure icon to review the remote cluster information for each cluster Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system Enter the user name and the password for the remote cluster and click OK 10 Click Next 11 Click Finish 12 Save the co
190. all simulated cluster m Global View Lists global clusters Through the Java Console VCS Simulator enables you to configure a simulated cluster panel bring a system in an unknown state into an online state simulate power loss for running systems simulate resource faults and save the configuration while VCS is offline For global clusters you can simulate the process of generating and clearing cluster faults You can run multiple simulated clusters on a system by using different port numbers for each cluster The Java Console provides the same views and features that are available for online configurations See Administering the cluster from Cluster Manager Java console on page 135 Predicting VCS behavior using VCS Simulator 321 Administering VCS Simulator from the Java Console Starting VCS Simulator from the Java Console To start VCS Simulator from the Java Console UNIX 1 Type the following command to grant the system permission to display on the desktop xhost 2 Configure the shell environment variable DISPLAY on the system where Cluster Manager will be launched For example if using Korn shell type the following command to display on the system myws export DISPLAY myws 0 3 Run the following command opt VRTSvcs bin hasimgui To start VCS Simulator from the Java Console Windows Click Start gt Programs gt Veritas gt VCS Simulator Java Console 322 Predicting VCS behavior usi
191. ames are shown in the following format Cluster gt gt Service Group gt gt Resource These objects are also the parent objects of the resource you selected The navigation trail ends with the selected resource name This tells you that you are at the resource level of views Figure 5 11 Navigation bar with a resource selected sol50 510a seci CMC CMC _ClusterConfig The path on the navigation bar is also called a navigation trail Object names higher up listed nearer the beginning of the trail are underlined indicating that they are a navigation link to the level of views for those objects You can move quickly up and down a view hierarchy by clicking linked object names About the view name The view name appears immediately below the navigation trail The view name is a unique name for the current view that also indicates m Your current level in the view hierarchy as indicated by the navigation trail Your current level is the level that the current view occupies in the view hierarchy such as cluster group or resource m Your current position within the current level as indicated by the secondary tab bar Your current position is the position that the current view occupies within the current level The position is the name of whichever tab is selected on the secondary tab bar such as Summary Attributes or Logs For example m According to the view hierarchy table the view name Cluster Resources indica
192. an be accessed at https hostname portno m Enter an IP address to bind the new port to a specific IP address instead of each IP address on the system The IP address must available on the system before starting the Web server Use this attribute to restrict Web server access to specific administrative subnets m Enter the name and password for a user having superuser administrative privileges on the Web server system 5 Click OK Administering Symantec Web Server 663 Configuring ports for VRTSweb Deleting ports Delete unused ports from VRTSweb To delete a port from the command line Run the following command on the system where VRTSweb is installed VRTSWEB HOME bin webgui delport lt portno gt bind ip address The variable portno represents the port number to be deleted If the port was bound to a particular IP address use the bind_ip_address option You must ensure that at least one port remains configured for the Web server For example opt VRTSweb bin webgui delport 443 101 1 1 2 opt VRTSweb bin webgui delport 80 To delete a port from the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 Click Delete Port on the left side of the Configuration page 4 Inthe Delete Port pane m Enter the port number to be deleted You cannot delete the port being used to access the Web page m Ifthe port was bound to a particular IP addres
193. an infrastructure check for the Mount resource verifies the existence of the mount directory defined in the MountPoint attribute for the resource You can run an infrastructure check only when the service group is online The check verifies that the specified node is a viable failover target capable of hosting the service group The virtual fire drill provides an option to fix specific errors detected during the infrastructure check 316 Configuring applications and resources in VCS Testing resource failover using virtual fire drills Infrastructure checks and fixes for supported agents Application m Isthe specified Program available m Does the specified Program have execute permissions m Does specified user exists on host m Does the same binary exist on all nodes DiskGroup m Is Veritas Volume Manager licensed m Are all disks in the diskgroup visible from the host IP m Does a route exist to the IP from the specified NIC Mount m Does mount directory exist m Create the mount m Issome other filesystem directory mounted at the specified mount directory NIC m Does the device exist on the host Process m Does the specified Program exist and does it have execute permissions m Isthe specified Program a binary executable m Does the same binary exist on all nodes Running a virtual fire drill You can run a virtual fire drill from the command line or from Cluster Manager Java Console
194. and is updated after every 10 monitor cycles The default value for this attribute is 0 A flag that specifies whether VCS keeps track of the monitor times for the resource bool ComputeStats 0 The value 0 indicates that VCS will not keep track of the time taken by the monitor routine for the resource The value 1 indicates that VCS keeps track of the monitor time for the resource The default value for this attribute is 0 Chapter Troubleshooting and recovery for VCS m Logging m Troubleshooting the VCS engine m Troubleshooting VCS startup m Troubleshooting service groups m Troubleshooting resources m Troubleshooting notification m Troubleshooting VCS configuration backup and restore m Troubleshooting and recovery for global clusters m Troubleshooting licensing 564 Troubleshooting and recovery for VCS Logging Logging VCS generates two error message logs the engine log and the agent log Log file names are appended by letters Letter A indicates the first log file B the second C the third and so on The engine log is located at var VRTSvcs log engine_A log The format of engine log messages is Timestamp Year MM DD Mnemonic Severity UMI Message Text m Timestamp the date and time the message was generated m Mnemonic the string ID that represents the product for example VCS m Severity levels include CRITICAL ERROR WARNING NOTICE and INFO most to least severe respectivel
195. ands changes the attribute values stored in the VCS configuration file By default all attributes are global meaning that the value of the attribute is the same for all systems Note VCS must be in read write mode before you can change the configuration For instructions see Setting the configuration to read w rite on page 243 Encrypting passwords Use the vcsencrypt utility to encrypt passwords when editing the VCS configuration file main cf to add VCS users or when configuring agents that require user passwords Administering the cluster from the command line 235 Installing a VCS license Note Do not use the vcsencrypt utility when entering passwords from a configuration wizard or from the Java and Web consoles To encrypt a password 1 Run the utility from the command line To encrypt a password for an agent configuration vesencrypt agent To encrypt a VCS user password vesencrypt vcs 2 Theutility prompts you to enter the password twice Enter the password and press Return Enter New Password Enter Again 3 The utility encrypts the password and displays the encrypted password Use the displayed password to edit the VCS configuration file main cf Installing a VCS license The utility vxlicinst installs a new permanent license or updates a license You must have root privileges to use this utility This utility must be run on each system in the cluster the utility cannot install o
196. ard No Can child group be taken offline when parent group is online Soft Yes Firm No Can parent group be switched while child group is running Soft Yes Firm Yes Hard Yes Can child group be switched while parent group is running Soft Yes Firm No Can child group be taken offline when parent group is online Firm No Soft Yes Can parent group be switched while child group is running Firm Yes but not to system on which child is running Soft Yes but not to system on which child is running Can child group be switched while parent group is running Firm No Soft Yes but not to system on which parent is running Can parent group be brought online when child group is offline Yes Can child group be taken offline when parent group is online Yes Can parent group be switched while the child group is running Yes but not to system on which child is running Can child group be switched while the parent group is running Yes but not to system on which parent is running Linking service groups 424 The role of service group dependencies Linking service groups You can link service groups from the command line or from the Java and Web consoles Note that a configuration may require that a certain service group be running before another service group can be brought online For example a group containing resources of a database service must be running before the database application is brought online See also Lin
197. ass user defined Indicates the scheduling class for the VCS agent process m Type and dimension string scalar m Default TS AgentDirectory user defined Complete path of the directory in which the agent binary and scripts are located Agents look for binaries and scripts in the following directories m Directory specified by the AgentDirectory attribute m opt VRTSvcs bin type m opt VRTSagents ha bin type If none of the above directories exist the agent does not start Use this attribute in conjunction with the AgentFile attribute to specify a different location or different binary for the agent m Type and dimension string scalar m Default AgentFailedOn system use only A list of systems on which the agent for the resource type has failed m Type and dimension string keylist m Default Not applicable AgentFile user defined Complete name and path of the binary for an agent If you do not specify a value for this attribute VCS uses the agent binary at the path defined by the AgentDirectory attribute m Type and dimension string scalar m Default Table D 2 VCS attributes Resource type attributes 621 Resource type attributes AgentPriority user defined Indicates the priority in which the agent process runs m Type and dimension string scalar m Default 0 AgentReplyTimeout user defined The number of seconds the engine waits to receiv
198. assign different numbers to each system in SystemList Initial configuration When VCS is installed a basic main cf configuration file is created with the cluster name systems in the cluster and a Cluster Manager user named admin with the password password The following is an example of the main cf for cluster demo and systems SystemA and SystemB include types cf cluster demo UserNames admin cDRpdxPmHzpS system SystemA system SystemB VCS configuration concepts 49 The types cf file The types cf file The types cf file describes standard resource types to the VCS engine specifically the data required to control a specific resource The following example illustrates a DiskGroup resource type definition for Solaris type DiskGroup static keylist SupportedActions license vfd disk vfd numdisks tatic int OnlineRetryLimit 1 tatic str ArgList DiskGroup StartVolumes StopVolumes MonitorOnly MonitorReservation tempUseFence PanicSystemOnDGLoss DiskGroupType tr DiskGroup tr StartVolumes 1 tr StopVolumes 1 tatic int NumThreads 1 boolean MonitorReservation 0 temp str tempUseFence INVALID boolean PanicSystemOnDGLoss 1 str DiskGroupType private nun s S s S The types definition performs two important functions Defines the type of values that may be set for each attribute In the DiskGroup example the NumThreads and OnlineRetryLimit att
199. ation tree 151 deleting resources 204 deleting service groups 181 deleting users 175 disabling resources 210 disabling service groups 189 Cluster Explorer continued editing attributes 221 enabling resources 209 enabling service groups 188 flushing service groups 191 freezing service groups 186 freezing systems 219 importing resource types 217 linking resources 212 linking service groups 192 logs 228 modifying system lists for service groups 161 monitoring group dependencies 154 monitoring resource dependencies 155 Notifier Wizard 164 opening configuration files 220 probing resources 207 Properties view 153 refreshing ResourceInfo attribute 217 Remote Cluster Status View 159 Resource View 155 running virtual fire drill 168 saving configuration files 220 service group configuration wizard 196 Service Group View 154 Status View 152 switching service groups 185 System Connectivity View 158 System Manager 161 taking resources offline 205 taking resources offline and propagating 206 taking service groups offline 184 tear off view 151 Template View 160 toolbar 149 unfreezing service groups 187 unfreezing systems 219 unlinking resources 214 unlinking service groups 194 User Manager 162 view panel 151 Cluster Guest about 63 adding user as 251 Cluster Management Console about 76 adding users 94 administering resources 127 administering service groups 98 adminstering resources 115 collapsing
200. ats tab is available only if global service groups are configured To navigate to the Cluster Heartbeats view 1 On the main tab bar click Management 2 On the secondary tab bar click Cluster Heartbeats 500 Administering global clusters from the Cluster Management Console Administering global heartbeats Adding a global heartbeat Add a heartbeat to monitor the health of a remote cluster that is configured in a global cluster environment To add a global heartbeat 1 Inthe Cluster Heartbeats view in the Configuration task panel click Add Heartbeat 2 Inthe Add Heartbeat dialog box specify the following details for the heartbeat and then click OK m The heartbeat name Enter a name for the heartbeat that is indicative of the generating it m The global clusters to add to the cluster list for the heartbeat In the Cluster List for Heartbeat table check the box preceding the line item for each cluster that you want to participate in the heartbeat Clear the check box to exclude clusters from the heartbeat list m The Arguments attribute value interval and timeout options Click the edit button in the Settings column to specify the value for the Arguments attribute the timeout options and the interval options Click Save to exit these options and save your selections Deleting a global heartbeat Delete a heartbeat that is monitoring the health of a remote cluster that is configured in a global cluster environment
201. attribute cannot be modified at runtime A two phase commit process prevents timing issues If two administrators simultaneously try to bring a service group online in a two cluster global group one command is honored and the other is rejected The attribute prevents bringing a service group online in a cluster that does not have the authority to do so If the cluster holding authority is down you can enforce a takeover by using the command hagrp online force service_group This command enables you to fail over an application to another cluster when a disaster occurs Note A cluster assuming authority for a group does not guarantee the group will be brought online on the cluster The attribute merely specifies the right to attempt bringing the service group online in the cluster The presence of Authority does not override group settings like frozen autodisabled non probed and so on that prevent service groups from going online You must seed authority if it is not held on any cluster Offline operations on global groups can originate from any cluster and do not require a change of authority to do so because taking a group offline does not necessarily indicate an intention to perform a cross cluster failover 466 Connecting clusters Creating global clusters VCS global clusters The building blocks Resiliency and Authority and AutoStart The attributes Authority and AutoStart work together to avoid potential concu
202. attribute for heartbeat ICMP hahb value Icmp ClusterList Note that ClusterList is a global attribute To display usage for the command hahb hahb help modify If the modify option is specified the usage for the hahb modi fy option is displayed 528 Administering global clusters from the command line Administering global service groups Administering global service groups Operations for the VCS global clusters option are enabled or restricted depending on the permissions with which you log on The privileges associated with each user role are enforced for cross cluster service group operations See User privileges in global clusters on page 64 To bring a service group online across clusters for the first time hagrp online force To bring a service group online across clusters hagrp online service group sys system clus cluster localclus The option clus brings the service group online on the system designated in the cluster If a system is not specified the service group is brought online on any node within the cluster The option localclus brings the service group online in the local cluster Administering global clusters from the command line 529 Administering global service groups To bring a service group online on any node hagrp online force service group any clus cluster localclus The option any specifies that HAD brings a failover group online on the optimal syste
203. aulted resource m Type and dimension integer scalar m Default 60 seconds 624 VCS attributes Resource type attributes Table D 2 Resource type attributes MonitorStatsParam user defined Stores the required parameter values for calculating monitor time statistics static str MonitorStatsParam Frequency 10 ExpectedValue 3000 ValueThreshold 100 AvgThreshold 40 Frequency The number of monitor cycles after which the average monitor cycle time should be computed and sent to the engine If configured the value for this attribute must be between 1 and 30 The value 0 indicates that the monitor cycle ti me should not be computed Default 0 ExpectedValue The expected monitor time in milliseconds for all resources of this type Default 100 ValueThreshold The acceptable percentage difference between the expected monitor cycle time ExpectedValue and the actual monitor cycle time Default 100 AvgThreshold The acceptable percentage difference between the benchmark average and the moving average of monitor cycle times Default 40 m Type and dimension integer association m Default Different value for each parameter MonitorTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the monitor entry point must complete or else be terminated m Type and dimension integer scalar m Default 60 seconds Table D 2 VCS attributes Resour
204. ava Console 221 for clusters 648 for heartbeats 655 for resource types 620 for resources 614 for service groups 628 for systems 641 local and global 54 overriding from command line 282 overriding from Java Console 208 authentication broker 28 Authority attribute about 465 definition 628 AutoDisabled attribute 628 AutoFailover attribute about 365 definition 629 AutoRestart attribute 629 AutoStart attribute for resources 614 for service groups 629 AutoStartIfPartial attribute 629 AutoStartList attribute 629 688 Index AutoStartPolicy attribute 630 AutoStartTimeout attribute 648 AvailableCapacity attribute 641 BackupInterval attribute 648 binary message catalogs about 565 location of 565 boolean attribute type 51 bundled agents 26 c Capacity attribute 641 Changing 361 Chapter 13 59 331 427 CleanTimeout attribute 621 client process detecting failure 554 CloseTimeout attribute 621 ClusState attribute 648 Cluster Administrator about 62 adding user as 251 cluster attributes 648 Cluster Explorer about 148 accessing 148 adding resources 199 adding service groups 178 adding systems 218 adding users 174 autoenabling service groups 190 bringing resources online 204 bringing service groups online 182 changing user passwords 175 changing user privileges 176 clearing resource faults 211 clearing ResourceInfo attribute 217 closing configuration files 221 Cluster Query 165 Command Center 163 configur
205. ay Y Y Y Y Y List Y Y Y Y Y Modify privileges Y Y Appendix B Configuring VCS in non global zones About VCS support for non global zones Configuring VCS in zones Verifying the zone configuration Maintenance tasks Troubleshooting information 592 Configuring VCS in non global zones About VCS support for non global zones About VCS support for non global zones Solaris zones provide an isolated and secure environment for running applications VCS provides high availability to applications running in zones by extending the failover capability to zones The VCS engine VCS agents and engine components run in the global zone For applications running within zones agents run entry points inside the zones If a zone faults VCS fails over the entire service group containing the zone to another node While applications may run in zones system resources including CPUs I O and adapters can be managed only at the global zone level VCS supports non global zones in both secure and non secure cluster environments If VCS is running in a secure environment using Symantec Product Authentication Service communication from non global zones to global zones is secure VCS agents that manage resources in non global zones The following VCS agents are zone aware and support applications running in non global zones m Process m ProcessOnOnly m Application m IP m IPMultiNIC m IPMultiNICB m Oracle m DB2 m Sybase
206. b Review the information required for the Global Group Configuration Wizard and click Next Enter the details of the service group to modify T Global Group Configuration Wizard Cluster List Details Select the group to modify B Global_Grp z Available Clusters Clusters for Service Group Select cluster fail over policy Manual x lt Back cancel Hep m Click the name of the service group that will be converted from a local group to a global group or vice versa m From the Available Clusters box click the clusters on which the group can come online Click the right arrow to move the cluster name to the Clusters for Service Group box for global to local cluster conversion Administering global clusters from Cluster Manager Java console 513 Administering global service groups click the left arrow to move the cluster name back to the Available Clusters box A priority num ber starting with 0 indicates the cluster in which the group will attempt to come online If necessary double click the entry in the Priority column to enter a new value m Select the policy for cluster failover m Manual prevents a group from automatically failing over to another cluster m Auto enables a group to automatically fail over to another cluster if it is unable to fail over within the cluster or if the entire cluster faults m Connected enables a group to auto
207. base atomic commit While the communications function in GAB is known as Atomic Broadcast no actual network broadcast traffic is generated An Atomic Broadcast message is a series of point to point unicast messages from the sending system to each receiving system with a corresponding acknowledgement from each receiving system Low Latency Transport LLT The Low Latency Transport protocol is used for all cluster communications as a high performance low latency replacement for the IP stack LLT has two major functions Traffic distribution LLT provides the communications backbone for GAB LLT distributes load balances inter system communication across all configured network links This distribution ensures all cluster communications are evenly distributed across all network links for performance and fault resilience If a link fails traffic is redirected to the remaining links A maximum of eight network links are supported Heartbeat LLT is responsible for sending and receiving heartbeat traffic over each configured network link LLT heartbeat is an Ethernet broadcast packet This broadcast heartbeat method allows a single packet to notify all other cluster members the sender is functional as well as provide necessary address information for the receiver to send unicast traffic back to the About communications membership and data protection in the cluster About cluster communications sender The heartbeat is the only broadcast t
208. bject attribute name add value sys system m Update list with user supplied values modify object attribute name update entry value sys system m Delete all values in list you cannot delete an individual element of a vector modify object attribute name delete keys sys system keylist m Replace list of keys duplicate keys not allowed modify object attribute name value sys system m Add keys to list duplicate keys not allowed modify object attribute name add value sys system m Delete user supplied keys from list modify object attribute name delete key sys system m Delete all keys from list modify object attribute name delete keys sys system 272 Administering the cluster from the command line Administering resources Table 7 2 Making VCS attributes local association m Replace list of key value pairs duplicate keys not allowed modify object attribute name value sys system m Add user supplied list of key value pairs to existing list duplicate keys not allowed modify object attribute name add value sys system m Replace value of each key with user supplied value modify object attribute name update key value sys system m Delete a key value pair identified by user supplied key modify object attribute name delete key sys system m Delete all key value pairs from association modify object attribute name del
209. ble The redundant server is on stand by with full performance capability If other applications are running they present no compatibility issues About cluster topologies 35 Basic failover configurations Symmetric or Active Active configuration In asymmetric configuration each server is configured to run a specific application or service and provide redundancy for its peer In this example each server runs one application service group When a failure occurs the surviving server hosts both application groups Before Failover After Failover Application1 Application2 Figure 2 2 Symmetric failover Symmetric configurations appear more efficient in terms of hardware utilization In the asymmetric example the redundant server requires only as much processor power as its peer On failover performance remains the same In the symmetric example the redundant server requires adequate processor power to run the existing application and the new application it takes over Further issues can arise in symmetric configurations when multiple applications running on the same system do not co exist properly Some applications work well with multiple copies started on the same system but others fail Issues can also arise when two applications with different I O and memory requirements run on the same system 36 About cluster topologies Basic failover configurations N to 1 configuration An N to 1 fa
210. bles you to create a subset of systems to use in an initial failover decision This feature allows fine tuning of application failover decisions and yet retains the flexibility to fail over anywhere in the cluster If the attribute is configured a service group tries to stay within its zone before VCS selects the system with the lowest priority as the failover target The Priority failover policy is ideal for simple two node clusters or small clusters with few service groups Priority is set in the SystemList attribute implicitly via ordering such as SystemList SystemA SystemB or explicitly such as SystemList SystemA 0 SystemB 1 Priority is the default behavior VCS selects the system running the fewest service groups as the failover target This policy is ideal for large clusters running many service groups with similar server load characteristics for example similar databases or applications The Load failover policy comprises the following components System capacity and service group load represented by the attributes Capacity and Load respectively System limits and service group prerequisites represented by the attributes Limits and Prerequisites respectively choosing a host in another zone For example in a three tier application infrastructure with Web application and database servers you could create two system zones one each for the application and the database In the event of a failover a service
211. bring the service group online For failover groups VCS sets this attribute to 0 when the group is taken offline For parallel groups it is set to 0 for the system when the group is taken offline or when the group faults and can fail over to another system VCS sets this attribute to 2 for failover groups if VCS attempts to autostart a service group for example attempting to bring a service group online on a system from AutoStartList m Type and dimension integer scalar m Default Not applicable LastSuccess system use only Indicates the time when service group was last brought online m Type and dimension integer scalar m Default Not applicable Load user defined Integer value expressing total system load this group will put on a system For example the administrator may assign a value of 100 to a large production SQL and 15 to a Web server m Type and dimension integer scalar m Default 0 ManageFaults user defined Specifies if VCS manages resource failures within the service group by calling the Clean entry point for the resources This attribute can take the following values NONE VCS does not call the Clean entry point for any resource in the group User intervention is required to handle resource faults See Controlling Clean behavior on resource faults on page 368 m Type and dimension string scalar m Default ALL ManualOps user defined Indicates if manual
212. by using OIDs with second last ID as zero enabling reversible translations severityld This variable indicates the severity of the trap being sent It can take the following values Severity Level and Description Value in Trap PDU Information 0 Important events exhibiting normal behavior Warning 1 Deviation from normal behavior Error 2 A fault Severe Error 3 Critical error that can lead to data loss or corruption VCS event notification 441 VCS events and traps entityType and entitySubType These variables specify additional information about the entity Entity Type Entity Sub type Resource String For example disk Group The type of the group m Failover m Parallel System String For example Solaris 2 8 Heartbeat The type of the heartbeat VCS String GCO String Agent name Agent name entityState This variable describes the state of the entity Entity States VCS states m User has logged into VCS m Cluster has faulted m Cluster is in RUNNING state Agent states m Agent is restarting m Agent has faulted Resources states Resource state is unknown Resource monitoring has timed out Resource is not going offline Resource is being restarted by agent Resource has faulted L L Gd L m Resource went online by itself L m Resource is in admin wait state L Resource monitor time has changed 442 VCS event notification VCS events and traps Entity Service group states System
213. canei Enter the reason for deleting the alert Click OK 232 Administering the cluster from Cluster Manager Java console Administering VCS Simulator Administering VCS Simulator VCS Simulator enables you to view state transitions experiment with configuration parameters and predict how service groups will behave during cluster or system faults Use this tool to create and save configurations in an OFFLINE state Through the Java Console VCS Simulator enables you to configure a simulated cluster panel bring a system in an unknown state into an online state simulate power loss for running systems simulate resource faults and save the configuration while VCS is offline For global clusters you can simulate the process of generating and clearing cluster faults You can run multiple simulated clusters on a system by using different port numbers for each cluster The Java Console provides the same views and features that are available for online configurations See Predicting VCS behavior using VCS Simulator on page 317 Administering the cluster from the command line About administering VCS from the command line Installing a VCS license Starting VCS Stopping VCS Logging on to VCS Managing VCS configuration files Managing VCS users from the command line Querying VCS Administering service groups Administering agents Administering resource types Administering resources Administering systems Admini
214. ce 2 Click the resource 3 Click the system on which to take the resource and the child resources offline 4 Click Apply To take child resources offline from Command Center while ignoring the state of the parent resource 1 Inthe Command Center configuration tree expand Commands gt Operations gt Controls gt OffProp Resource 2 Click the resource 3 Click the system on which to take the resource and the child resources offline 4 Select the ignoreparent check box 5 Click Apply Administering the cluster from Cluster Manager Java console 207 Administering resources Probing a resource Probe a resource to check that it is configured and ready to bring online To probe a resource from Cluster Explorer 1 2 In the Service Groups tab of the configuration tree right click the resource Click Probe and click the appropriate system from the menu To probe a resource from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Controls gt Probe Resource Click the resource Click the system on which to probe the resource Click Apply 208 Administering the cluster from Cluster Manager Java console Administering resources Overriding resource type static attributes You can override some resource type static attributes and assign them resource specific values When a static attribute is overridden and the configuration is saved the main cf fi
215. ce group The system list of a service group contains a list of all systems on which the service group is configured 3 Click OK Modifying service group attributes Edit the attributes of a service group You must have the role of cluster administrator or service group administrator to edit attributes To edit a service group attribute using the attribute table 1 Inthe Group Summary view click Attributes 2 Inthe All attributes for group table locate the line item for the attribute you want to edit 3 Inthe Edit column click the edit button 4 Inthe Edit Attribute dialog box specify the following attribute details and then click OK 111 112 Administering the cluster from the Cluster Management Console Administering service groups m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value You can enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values m The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List To edit a service group attribute using the task pane 1 In
216. ce in the Service Group Resources box After selecting the parent resource the corresponding child resources are displayed in the Child Resources box File View Help VCSHP10_71_0214a Command Center admin Cluster Administrator e amp 4 Se ASO FE ome on QA Unlink Resources E E Configuration File Custer Objects newgroup E E Attributes Dependencies Service Group Resources Link Resources Bs Link Service Groups fe unlink Resources Bs Unlink Service Groups 2 Operations Child Resources j unlink resourcel_ng2 TestRes Click the child resource Click Apply 216 Administering the cluster from Cluster Manager Java console Administering resources Invoking a resource action Cluster Explorer enables you to initiate a predefined action script Some examples of predefined resource actions are splitting and joining disk groups To invoke a resource action 1 Inthe Service Groups tab of the configuration tree right click the resource 2 Click Actions 3 Specify the details of the action T Fire Action xi CMC_ClusterConnector Action Jprogram vFd 7 System vcshp71 v m Click the predefined action to execute m Click the system on which to execute the action m To add an argument click the Add icon and enter the argument Click the Delete icon to remove an argument m Click OK Administering the cluster from Cluster Manager Java cons
217. ce type attributes 625 Resource type attributes NumThreads user defined Number of threads used within the agent process for managing resources This number does not include threads used for other internal purposes If the number of resources being managed by the agent is less than or equal to the NumThreads value only that many number of threads are created in the agent Addition of more resources does not create more service threads Similarly deletion of resources causes service threads to exit Thus setting NumThreads to 1 forces the agent to just use 1 service thread no matter what the resource count is The agent framework limits the value of this attribute to 30 m Type and dimension integer scalar m Default 10 OfflineMonitorInterval user defined Note This attribute can be overridden Duration in seconds between two consecutive monitor calls for an OFFLINE resource If set to 0 OFFLINE resources are not monitored m Type and dimension integer scalar m Default 300 seconds OfflineTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the offline entry point must complete or else be terminated m Type and dimension integer scalar m Default 300 seconds OnlineRetryLimit user defined Note This attribute can be overridden Number of times to retry the online operation if the attempt to online a resource is unsuccessful This
218. change from RESOURCE_FAULTED to RESOURCE_OFFLINE hares clear resource sys system Clearing a resource initiates the online process previously blocked while waiting for the resource to become clear If system is not specified the fault is cleared on each system in the service group s SystemList attribute See To clear faulted non persistent resources in a service group on page 265 This command also clears the resource s parents Persistent resources whose static attribute Operations is defined as None cannot be cleared with this command and must be physically attended to such as replacing a raw disk The agent then updates the status automatically Administering the cluster from the command line 275 Administering resources Running a virtual fire drill from the command line The service group must be online when you run the virtual fire drill See Testing resource failover using virtual fire drills on page 315 To run a virtual fire drill for a specific resource Type the following command hares action lt resname gt lt vfdaction gt vfd sys lt sysname gt The command runs the infrastructure check verifies whether the system lt sysname gt has the required infrastructure to host the resource lt resname gt should a failover require the resource to come online on the system For the variable lt sysname gt specify the name of a system on which the resource is offline The variable lt vfdact
219. change that affects the zone configuration you must run the hazonesetup command to reconfigure the zones in VCS See Configuring the service group for the application on page 600 Make sure that the zone configuration files are consistent on all the nodes at all times The file is located at etc zones zone_name xml When you add a patch or upgrade the operating system on one node make sure to upgrade the software on all nodes Make sure that the application configuration is identical on all nodes If you update the application configuration on one node apply the same updates to all nodes 604 Configuring VCS in non global zones Troubleshooting information Troubleshooting information VCS HA commands do not work Recommended Action Verify the VCS packages are installed Run the hazonesetup script to set up the zone configuration Run the hazoneverify command to verify the configuration Run the halogin command from the zone See Logging on to VCS on page 240 Verify your VCS credentials Make sure the password is not changed Verify the VxSS certificate is not expired Resource does not come online in the zone Recommended Action Verify VCS and the agent packages are installed correctly Verify the application is installed in the zone Verify the configuration definition of the agent Make sure the ContainerType and ContainerName attributes are defined Appendix C Cluster and system states
220. cies Service group dependency configurations Group Dependency FAQs Linking service groups VCS behavior with service group dependencies 410 The role of service group dependencies About service group dependencies About service group dependencies Service groups can be dependent on each other The dependent group is the parent and the other group is the child For example a finance application parent may require that the database application child is online before it comes online While service group dependencies offer more features to manage application service groups they create more complex failover configurations A service group may function both as a parent and a child Veritas Cluster Server supports five levels of service group dependencies The role of service group dependencies 411 About service group dependencies Dependency links The dependency relationship between a parent and a child is called a link The link is characterized by the dependency category the location of the service groups and the rigidity of dependency m A dependency may be online or offline m A dependency may be local global or remote m A dependency may be soft firm or hard with respect to the rigidity of the constraints between parent and child service group You can customize the behavior of service groups by choosing the right combination of the dependency category location and rigidity Dependency categories online or offl
221. cify a password for the user Select the check box if you wish to specify advance options to connect to a secure cluster Otherwise proceed to the last step m Specify the domain of which the node is a part m Select a domain type m Specify the authentication broker and port Click Next Administering the cluster from Cluster Manager Java console 203 Administering resources In the Remote Group Resource Details dialog box T Remote Group Resource Wizard Remote Group Resource Details Provide the Following information related to Remote Group resource configuration Remote Group Name cmc Resource Dependency View CMC Mode of Monitor MonitorOnly z Scope of Monitoring Any System CMC_ClusterConnector Specific System Local Systems Remote Systems Resource snapshot at May 9 2006 3 13 15 PM lt Back IJ cee Hep Select a group you wish to monitor m Select the mode of monitoring m Choose the MonitorOnly option to monitor the remote service group You will not be able to perform online or offline operations on the remote group m Choose the OnlineOnly option to monitor the remote service group and bring the remote group online from the local cluster m Choose the OnOff option to monitor the remote service group bring the remote group online and take it offline from the local cluster m Specify whether the RemoteGroup resource should monitor the state of the remote group on a specific sy
222. clause To configure a service group dependency place the keyword requires in the service group declaration of the main cf file Position the dependency clause before the resource dependency specifications and after the resource declarations group_x requires group_y See About service group dependencies on page 410 About the SystemList attribute The SystemList attribute designates all systems on which a service group can come online By default the order of systems in the list defines the priority of systems used in a failover For example the following definition configures SystemA to be the first choice on failover followed by SystemB and then SystemC SystemList SystemA SystemB SystemC System priority may also be assigned explicitly in the SystemList attribute by assigning numeric values to each system name For example SystemList SystemA 0 SystemB 1 SystemC 2 If you do not assign numeric priority values VCS assigns a priority to the system without a number by adding 1 to the priority of the preceding system For example if the SystemList is defined as follows VCS assigns the values SystemA 0 SystemB 2 SystemC 3 SystemList SystemA SystemB 2 SystemC Note that a duplicate numeric priority value may be assigned in some situations SystemList SystemA SystemB 0 SystemC The numeric values assigned are SystemA 0 SystemB 0 SystemC 1 To avoid this situation do not assign any numbers or
223. cluster and connects with peers in remote clusters It receives and transmits information about the status of the cluster service groups and systems This communication enables VCS to create a consolidated view of the status of all the clusters configured as part of the global cluster The process also manages wide area heartbeating to determine the health of remote clusters The process also transmits commands between clusters and returns the result to the originating cluster Connecting clusters Creating global clusters 465 VCS global clusters The building blocks Wide area heartbeats The wide area Heartbeat agent manages the inter cluster heartbeat Heartbeats are used to monitor the health of remote clusters See Heartbeat attributes on page 655 You can change the default values of the heartbeat agents using the command hahb modify Sample configuration Heartbeat Icmp Clusterlist C1 C2 AYAInterval Cl 20 AYAInterval Cl1 30 Arguments cl 192 168 10 10 Arguments c2 64 203 10 12 Serialization The authority attribute VCS ensures that multi cluster service group operations are conducted serially to avoid timing problems and to ensure smooth performance The Authority attribute prevents a service group from coming online in multiple clusters at the same time Authority is a persistent service group attribute and it designates which cluster has the right to bring a global service group online The
224. clusters one instance of the Cluster Management Console is installed outside all clusters on a standalone server The console enables users to visually and intuitively input commands to the multi cluster management engine the management server The management server initiates monitoring and management actions based upon those commands The management server uses a database to store cluster configurations cluster status events event policies report jobs report outputs and more If the managment server and cluster nodes are separated by a firewall a component called cluster connector is installed on each cluster node Cluster connector enables communication between the management server and clusters through firewalls Cluster connector also provides buffering for cluster data If the management server goes offline and then comes back online it can retrieve data collected during the offline period from the cluster connector buffer The console offers additional capability for administering users reports events and notification If the cluster environment includes licensed VCS global clusters disaster recovery DR capability is also available Administering the cluster from the Cluster Management Console 71 About Veritas Cluster Management Console The configurational differences between the modes mean that you cannot switch a single Cluster Management Console installation from one mode to the other However the modes can co e
225. cons to communicate information about cluster objects and their states See Cluster and system states on page 605 Table 6 1 Icons in Cluster Manager Java Console Cluster System Service Group Resource Type Resource OFFLINE e el el e amp Table 6 1 Administering the cluster from Cluster Manager Java console Icons in the Java Console 141 Icons in Cluster Manager Java Console Faulted in UP BUT NOT IN CLUSTER MEMBERSHIP state Faulted in EXITED state PARTIAL Link Heartbeats in UP and DOWN states Disk Heartbeats in UP and DOWN states UP AND IN JEOPARDY FROZEN AUTODISABLED UNKNOWN ADMIN_WAIT Global Service Group requires the VCS Global Cluster Option Remote Cluster in RUNNING state requires the VCS Global Cluster Option el A e el lel e e Remote Cluster in EXITING EXITED INIT INQUIRY LOST_CONN LOST_HB TRANSITIONING Or UNKNOWN state 142 Administering the cluster from Cluster Manager Java console About Cluster Monitor About Cluster Monitor After starting Cluster Manager the first window that appears is Cluster Monitor This window includes one or more panels displaying general information about actual or simulated clusters Use Cluster Monitor to log on to and off of a cluster view summary information on various VCS objects customize the display use VCS Simulator and exit Clus
226. copy stored in a snapshot Exports a snapshot from the local predefined directory to the specified file Configures the list of files or directories to be included in new snapshots in addition to those included automatically by the backup command Configures the list of files or directories to be excluded from new snapshots when backing up the configuration using the backup command Deletes snapshots from the predefined local directory on each node Note With the exception of the include exclude and the delete options all options can be combined with the f option This option indicates that all files be backed up to or restored from the specified single file instead of a local predefined directory on each node This option is useful when you want to store the configuration data to an alternate location that is periodically backed up using backup software like Veritas Net Backup Administering the cluster from the command line 245 Managing VCS configuration files Backing up configuration files The hasnap backup command backs up files in a snapshot format A snapshot is a collection of VCS configuration files backed up at a particular point in time typically before making changes to the existing configuration A snapshot also contains information such as the snapshot name description creation time and file permissions The command backs up a predefined list of VCS configuration files as well as a user defined
227. ctories Examples The following command displays the exclude file list on each node in the cluster hasnap exclude list The following command adds the file etc VRTSvcs conf config temp cf to the exclude file list on host1 which results in this file being excluded from the snapshot the next time the hasnap backup command is run hasnap exclude add sys hostl etc VRTSvcs conf config temp cf The following command removes the file etc VRTSvcs conf config temp cf from the exclude list on host1 hasnap exclude del sys hostl etc VRTSvcs conf config temp cf 250 Administering the cluster from the command line Managing VCS configuration files Deleting snapshots Use the hasnap delete command to delete snapshots from the predefined local directory on each node To delete a snapshot Run the following command hasnap delete s snapid s Specifies the ID of the snapshot to be deleted If you do not specify the snapshot ID the command lists the snapshots that can be deleted Example The following command deletes snapshot vcs 20030101 22232 from the cluster hasnap delete s vcs 20030101 22232 Administering the cluster from the command line 251 Managing VCS users from the command line Managing VCS users from the command line Adding a user You can add modify and delete users on any system in the cluster provided you have the privileges to do so If VCS is running in secure mode specify fully qual
228. d 50 MedServer3 AutoStart operation Based on the preceding main cf example the AutoStart sequence resembles Controlling VCS behavior Sample configurations depicting workload management Databasel LargeServer1 Database2 LargeServer2 Database3 LargeServer3 Application1 MedServer1 Application2 MedServer2 Application3 MedServer3 Application4 MedServer4 Application5 MedServer5 Normal operation The configuration resembles Server AvailableCapacity CurrentLimits Online Groups LargeServer1 100 ShrMemSeg 10 Databasel Semaphores 20 Processors 12 LargeServer2 100 ShrMemSeg 10 Database2 Semaphores 20 Processors 12 LargeServer3 100 ShrMemSeg 10 Database3 Semaphores 20 Processors 12 MedServer1 50 ShrMemSeg 5 Application1 Semaphores 10 Processors 6 MedServer2 50 ShrMemSeg 5 Application2 Semaphores 10 Processors 6 MedServer3 50 ShrMemSeg 5 Application3 Semaphores 10 Processors 6 407 408 Controlling VCS behavior Sample configurations depicting workload management Server AvailableCapacity CurrentLimits Online Groups MedServer4 50 ShrMemSeg 5 Application4 Semaphores 10 Processors 6 MedServer5 50 ShrMemSeg 5 Application5 Semaphores 10 Processors 6 Failure scenario In the following example LargeServer3 fails VCS scans all available systems in the SystemList for the Database3 group for
229. d configure ports SMTP recipients SMTP servers and logging click Configuration Administering Symantec Web Server 659 Getting Started 3 Review the information in the Configuration tab Symantec Web Server Configuration Connected to host vcslabweb Select Language Go Configuration Configuration e Add Port Delete Port S r a Configured Ports Configure SMTP Server san EE Port Number Protocol IP Address e Add SMTP Recipient 8181 http e Delete SMTP Recipi 8443 https e Configure Logging SMTP Recipients Email Severity Locale SMTP Server localhost Logging Category Level Web Server fine Web Applications fine Other info Log Directory var VRTSweb lot A symantec Log y fvar 3 109 Configured Ports Lists information about the configured ports m SMTP Recipients Displays information about configured SMTP recipients and the SMTP server m Logging Lists the log levels for various Web server components 660 Administering Symantec Web Server Configuring ports for VRTSweb Configuring ports for VRTSweb You can view add edit or remove ports for VRTSweb About VRTSweb ports By default VRTSweb serves HTML content on the following ports 8181 HTTP Non secure port used for backward compatibility VRTSweb redirects users to the secure port 8443 HTTPS Secure SSL port VRTSweb presents a self signed SSL certificate issued by Symantec to the browser You m
230. d only if it is fully or partially online The switch option is not supported for switching parallel service groups and for switching hybrid service groups across system zones 264 Administering the cluster from the command line Administering service groups Freezing and unfreezing service groups Freeze a service group to prevent it from failing over to another system This freezing process stops all online and offline procedures on the service group Unfreeze a frozen service group to perform online or offline operations on the service group To freeze a service group disable online offline and failover operations Type the following command hagrp freeze service group persistent The option persistent enables the freeze to be remembered when the cluster is rebooted To unfreeze a service group reenable online offline and failover operations Type the following command hagrp unfreeze service group persistent Enabling and disabling service groups Enable a service group before bringing it online A service group that was manually disabled during a maintenance procedure on a system may need to be brought online after the procedure is completed Disable a service group to prevent it from coming online This process temporarily stops VCS from monitoring a service group on a system undergoing maintenance operations To enable a service group Type the following command hagrp enable service group
231. d updates VCS with the resource status The action to bring a resource online or take it offline differs significantly for each resource type For example bringing a disk group online requires importing the disk group But bringing a database online requires starting the database manager process and issuing the appropriate startup commands VCS monitors resources when they are online and offline to ensure they are not started on systems on which they are not supposed to run For this reason VCS starts the agent for any resource that is configured to run on a system when the cluster is started If no resources of a particular type are configured the agent is not started For example if there are no Oracle resources in your configuration the Oracle agent is not started on the system Agent operations Agents carry out specific operations on resources The functions an agent performs are called entry points For details on any of the following entry points see the Veritas Cluster Server Agent Developer s Guide m Online Brings a specific resource ONLINE from an OFFLINE state m Offline Takes a resource from an ONLINE state to an OFFLINE state m Monitor Tests the status of a resource to determine if the resource is online or offline The entry point runs at the following times During initial node startup to probe and determine status of all resources on the system m After every online and offline operation Periodically
232. de consistent from a backup source and that the RVGPrimary can initiate a takeover when the service group is brought online This option applies to VVR environments only You can select the groups to be failed over to the local cluster in which case VCS brings the selected groups online on a node based on the group s FailOverPolicy attribute It also marks the groups as being offline in the other cluster If you do not select any service groups to fail over VCS takes no action except implicitly marking the service groups as offline on the downed cluster Lost heartbeats and the inquiry mechanism The loss of internal and all external heartbeats between any two clusters indicates that the remote cluster is faulted or that all communication links between the two clusters are broken a wide area split brain VCS queries clusters to confirm the remote cluster to which heartbeats have been lost is truly down This mechanism is referred to as inquiry If in a two cluster configuration a connector loses all heartbeats to the other connector it must consider the remote cluster faulted If there are more than two clusters and a connector loses all heartbeats to a second cluster it queries the remaining connectors before declaring the cluster faulted If the other connectors view the cluster as running the querying connector transitions the cluster to the UNKNOWN State a process that minimizes false cluster faults If all connectors 576 Troub
233. default size for log files of all Web applications running VRTSweb Individual Web applications can override this default value The count for log files of all Web applications running VRTSweb Individual Web applications can override this default value 678 Administering Symantec Web Server Modifying the maximum heap size for VRTSweb Modifying the maximum heap size for VRTSweb The default maximum allowed heap size for the VRTSWeb Java Virtual Machine JVM is 256MB You may need to modify this limit for large configurations or for when a large number of consoles share the same VRTSweb instance You can modify the maximum heap size only from the command line To modify the maximum heap size 1 Type the following command VRTSWEB_HOME bin webgui maxheap new size in MB For example opt VRTSweb bin webgui maxheap 512 2 Restart the Web server after specifying a new limit VRTSWEB HOME bin webgui restart 3 View the current limit VRTSWEB HOME bin webgui maxheap Appendix Accessibility and VCS About accessibility in VCS Navigation and keyboard shortcuts Support for accessibility settings Support for assistive technologies 680 Accessibility and VCS About accessibility in VCS About accessibility in VCS Symantec products meet federal accessibility requirements for software as defined in Section 508 of the Rehabilitation Act http www access board gov 508 htm Keyboard shortcuts are available
234. down the items of a list m Alt in combination with the underlined mnemonic letter for a field or command button shifts the focus to that field or button m Either Enter or the Spacebar activates your selection For example after pressing Tab to select Next in a wizard panel press the Spacebar to display the next screen Support for accessibility settings Symantec software responds to operating system accessibility settings On UNIX systems you can change the accessibility settings using desktop preferences or desktop controls Support for assistive technologies m Cluster Manager Java Console is compatible with JAWS 4 5 m Though graphics in the documentation can be read by screen readers setting your screen reader to ignore graphics may improve performance m Symantec has not tested screen readers for languages other than English 682 Accessibility and VCS Support for assistive technologies Glossary Agent A process that starts stops and monitors all configured resources of a type and reports their status to VCS Active Active Configuration A failover configuration where each systems runs a service group If either fails the other one takes over and runs both service groups Also known as a symmetric configuration Active Passive Configuration A failover configuration consisting of one service group on a primary system and one dedicated backup system Also known as an asymmetric configuration Authenticati
235. e oe AYS QI SNES NESS Freeze unfreeze persistent KIARA SSS SENSE SENSE NESE SNSYS Enable Disable Modify Display View dependencies View resources List S S 878 XS 8 XS SN 8 8 Enable resources Disable resources Flush Autoenable Ignore QS NUS QPS SESE SSS SESS SES QS NUS NESSES SSS SNES ESNESES VCS user privileges administration matrices 587 Administration matrices Heartbeat operations hahb Table A 5 lists heartbeat operations and required privileges Table A 5 User privileges for heartbeat operations Add z z Delete Make local Make global Display View state KIAIA SIAS S RIAIS QS SNS KINIS KINIS List Log operations halog Table A 6 lists attribute operations and required privileges Table A 6 User privileges for log operations Enable debug tags Y Delete debug tags Y Add messages to Y a log file Display Y Y Y Y Y Display log file info WY Vv Y Y Vv 588 VCS user privileges administration matrices Administration matrices Resource operations hares Table A 7 lists resource operations and required privileges Table A 7 User privileges for resource operations Add Delete Make local Make global Link and unlink Clear Bring online KIRIS
236. e click the linked name of the service group that you want to delete Administering the cluster from the Cluster Management Console 101 Administering service groups In the Group Summary view in the Configuration task panel click Delete Service Group In the Delete Service Group dialog box click OK to confirm that you want to delete the specified service group Modifying a service group Edit a service group configuration You cannot edit the service group name or type using this task To change the name or type you must first remove the service group and then add it back using the desired name and type You must have the role of cluster administrator or service group administrator to edit a service group To edit a service group configuration 1 In the Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to modify In the Group Summary view in the Configuration task panel click Modify Service Group In the Service Group Configuration wizard read the introductory text and then click Next In the Service Group Configuration dialog box the service group name and type display but cannot be modified using this task Click Next In the System List Configuration dialog box specify the following details for the target systems and then click Next The target systems are the systems on that can host the service group m The target system names Find the system you want
237. e offline state of a parent resource to its child resources This link is disabled if any of the following conditions exist You do not have the role of either cluster administrator or cluster operator The resource does not depend on any other resource An online resource depends on this resource The resource is not online To take a parent resource and all of its child resources offline 1 In the Cluster Resources view in the Resources Listing table click the linked name of the resource you want to take offline In the Resource Summary view in the Operations task panel click Offline Propagate In the Offline Propagate for Resource dialog box select the system on which to bring the resource online from the drop down menu Click OK Faulting a resource Manually simulate a fault on a resource to test failover behavior This operation is available only when using the simulator To fault a single service group 1 In the Cluster Resources view in the Resources Listing table click the linked name of the resource that you want to fault In the Resource Summary view in the Simulation task panel click Fault Resource In the Fault Resource dialog box on the drop down menu select the systems on which you want to fault the resource This drop down menu is populated using the system list of the resource The system list of a resource contains a list of all systems on which the resource is configured Click OK 120
238. e 140 include clauses about 47 InfoInterval attribute 622 InfoTimeout attribute 622 injeopardy event trigger 448 integer attribute type 51 IntentOnline attribute 633 Istate attribute 617 J Java Console about 29 administering clusters 137 administering logs 228 administering resources 199 administering service groups 178 administering systems 218 administering user profiles 174 administering VCS Simulator 320 arranging icons 156 Cluster Explorer 148 Cluster Manager 140 Cluster Monitor 142 Cluster Query 165 components of 140 customizing display 146 disability compliance 137 icons 140 impact on performance 549 logging off of acluster 173 logging on to a cluster 171 overview 137 running commands from 221 running virtual fire drill 168 setting initial display 138 starting 139 user profiles 174 694 Index Java Console continued using with ssh 138 viewing server credentials 167 viewing user credentials 167 Java Console views Properties 153 Remote Cluster Status 159 Resource 155 Service Group 154 Status 152 System Connectivity 158 tear off option 151 K keylist attribute dimension 52 keywords 55 keywords list of 55 L LastOnline attribute 617 LastSuccess attribute 633 license keys about 235 installing 235 troubleshooting 578 LicenseType attribute 644 licensing issues 19 Limits attribute 644 LinkHbStatus attribute 644 LLT about 336 LLTNodeld attribute 644 Load attribute 633 Load policy for SGWM 3
239. e HAD is started 5 Click the appropriate service group type A failover service group runs on only one system at a time a parallel service group runs concurrently on multiple systems 6 To add a new service group based on a template click Templates Otherwise proceed to step 9 7 Click the appropriate template name 8 Click OK 9 Click Apply To add a service group from the template view 1 From Cluster Explorer click Templates on the Tools menu 2 Right click the Template View panel and click Add as Service Group from the pop up menu This adds the service group template to the cluster configuration file without associating it to a particular system Template View as on System vcshp71 _ ol x Template View YolumeNFSGroup Templates vNFS_NIC vNFS_Share1 wNFS_Share2 G vNFS_Mount1 vNFS_NFS vNFS_Mount2 vNFS_Volume1 vNFS_Volume2 vNFS_DiskGroup 3 Use System Manager to add the service group to systems in the cluster See System Manager on page 161 Administering the cluster from Cluster Manager Java console 181 Administering service groups Deleting a service group Delete a service group from Cluster Explorer or Command Center Note You cannot delete service groups with dependencies To delete a linked service group you must first delete the link To delete a service group from Cluster Explorer 1 Inthe Service Groups tab of the configuration tree right click the s
240. e HTTPS port can be accessed at https hostname portno m bind ip address Optional variable Specifies that the new port be bound to a particular IP address instead of each IP address on the system Use this option to restrict Web server access to specific administrative subnets The IP address must be available on the system before the Web server starts For example opt VRTSweb bin webgui addport 443 HTTPS 101 1 1 2 opt VRTSweb bin webgui addport 80 HTTP To add a port from the Web console 1 Access the Web server using the configured port number for example http hostname 8181 Click the Configuration tab Click Add Port on the left side of the Configuration page 662 Administering Symantec Web Server Configuring ports for VRTSweb 4 Inthe Add Port pane Configuration Add Port This page lets you add a new port to the web server To add a secure port select the https protocol IP Address is optional If not specified the specified port will be bound to all IP addresses on the machine Port Number Protocol http O https IP Address optional Username root Password Should belong to the group root on veslabweb m Enter the port number to be added m Choose the HTTP option to add a normal port choose the HTTPS option to add a secure SSL port Web servers using the HTTP port can be accessed at http hostname portno Web servers using the HTTPS port c
241. e Web server Web applications and for other components m Enter the name and password for a user having superuser privileges on the Web server system 5 Click OK 676 Administering Symantec Web Server Configuring logging for VRTSweb Modifying size limit and rollover count for VRTSweb logs You can modify the maximum size limit and rollover count for logs maintained by VRTSweb only from the command line To modify the size limit and rollover count for logs Run the following command on the system where VRTSweb is installed VRTSWEB HOME bin webgui log vrtsweb size size vrtsweb count count command size size command_count count binary size size binary count count jvm_size size jvm_count count protocol client size size protocol client _count count protocol server size size protocol _server_count count out_size size out_count count err _size size err _count count webapps size size webapps _count count For example opt VRTSweb bin webgui log vrtsweb_size 100000 vrtsweb count 4 opt VRTSweb bin webgui log err _size 200000 opt VRTSweb bin webgui log webapps _count 4 The following table describes the command parameters Parameter vrtsweb_size vrtsweb_count command _size command_count binary_size binary_count jvm_size jvm_count Description The size of the file _vrtsweb log which contains the Web server logs and the tomcat container related logs
242. e a heartbeat from the agent before restarting the agent m Type and dimension integer scalar m Default 130 seconds AgentStartTimeout user defined The number of seconds after starting the agent that the engine waits for the initial agent handshake before restarting the agent m Type and dimension integer scalar m Default 60 seconds ArgList user defined An ordered list of attributes whose values are passed to the open close online offline monitor clean info and action entry points m Type and dimension string vector m Default Not applicable AttrChangedTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the attr_changed entry point must complete or be terminated m Type and dimension integer scalar m Default 60 seconds CleanTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the clean entry point must complete or else be terminated m Type and dimension integer scalar m Default 60 seconds CloseTimeout user defined Note This attribute can be overridden Maximum time in seconds within which the close entry point must complete or else be terminated m Type and dimension integer scalar m Default 60 seconds 622 VCS attributes Resource type attributes Table D 2 Resource type attributes ConfInterval user defined Note This a
243. e a new certificate 3 Restart the server for the new certificate to take effect VRTSWEB HOME bin webgui restart Administering Symantec Web Server 665 Managing VRTSweb SSL certificates Exporting the SSL certificate to a file You can export the public key that is associated with an SSL certificate to a file You can then import the key to other applications to establish trust with the VRTSweb instance To export the SSL certificate to a file Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui cert export cert file rfc If the VRTSweb SSL certificate does not exist the command prompts you to create one If you specify the RFC option the key output is encoded in a printable format which is defined by the Internet RFC 1421 standard For example opt VRTSweb bin webgui cert export myapp vrtsweb cer rfc Configuring a CA signed SSL certificate By default VRTSweb presents a self signed SSL certificate every time you access VRTSweb over the SSL port You can install a certificate signed by a Certificate Authority CA like Verisign com or Thawte com To configure a CA signed SSL certificate 1 If you do not have a self signed certificate with information that the CA can verify create a certificate VRTSWEB_HOME bin webgui cert create See Creating a self signed SSL certificate on page 664 Generate a Certificate Signing Request CSR for the certificate Run
244. e action is taken after the first interval of ActionTimeLimit As CPU usage continues to exceed the threshold value action is taken after five minutes If the values of ActionThreshold or ActionTimeLimit are set to 0 no action is taken Actions can have one of the following values 560 VCS performance considerations VCS agent statistics NONE No action will be taken and the message is logged in the VCS engine log REBOOT System is rebooted CUSTOM The cpuusage trigger is invoked VCS agent statistics You can configure VCS to track the time taken for monitoring resources You can use these statistics to configure the MonitorTimeout attribute You can also detect potential problems with resources and systems on which resources are online by analyzing the trends in the time taken by the resource s monitor cycle Note that VCS keeps track of monitor cycle times for online resources only VCS calculates the time taken for a monitor cycle to complete and computes an average of monitor times after a specific number of monitor cycles and stores the average in a resource level attribute VCS also tracks increasing trends in the monitor cycle times and sends notifications about sudden and gradual increases in monitor times VCS uses the following parameters to compute the average monitor time and to detect increasing trends in monitor cycle times m Frequency The number of monitor cycles after which the monitor time average is comput
245. e and dimension string scalar m Default main cf State system use only Group state on each system OFFLINE All non persistent resources are offline ONLINE All resources whose AutoStart attribute is equal to 1 are online FAULTED At least one critical resource in the group is faulted or is affected by a fault PARTIAL At least one but not all resources with Operations OnOff is online and not all AutoStart resources are online STARTING Group is attempting to go online STOPPING Group is attempting to go offline A group state may be a combination of the multiple states described above For example OFFLINE FAULTED OFFLINE STARTED PARTIAL FAULTED PARTIAL STARTING PARTIAL STOPPING ONLINE STOPPING m Type and dimension integer scalar m Default Not applicable SystemList user defined List of systems on which the service group is configured to run and their priorities Lower numbers indicate a preference for the system as a failover target Note You must define this attribute prior to setting the AutoStartList attribute m Type and dimension integer association m Default none 638 VCS attributes Service group attributes Table D 3 Service group attributes SystemZones user defined Indicates the virtual sublists within the SystemList attribute that grant priority in failing over Values are string integer pairs The string key is the name
246. e brought online and taken offline If the online or offline operation hangs on a particular resource flush the service group to halt the operation on the resources waiting to go online or offline Flushing a service group typically leaves the cluster in a partial state After completing this process resolve the issue with the particular resource if necessary and proceed with starting or stopping the service group To flush a service group from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Flush and click the appropriate system from the menu To flush a service group from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Flush Service Group Click the service group Click the system on which to flush the service group Click Apply 192 Administering the cluster from Cluster Manager Java console Administering service groups Linking service groups To link a service group from Cluster Explorer 1 2 Click a cluster in the configuration tree In the view panel click the Service Groups tab This opens the service group dependency graph To link a parent group with a child group Click Link Click the parent group Move the mouse toward the child
247. e child resources of parent resource are offline Recommended Action Bring the child resources online first Waiting for parent resources One or more parent resources are online Recommended Action Take the parent resources offline first Waiting for resource to respond The resource is waiting to come online or go offline as indicated VCS directed the agent to run an online entry point for the resource Recommended Action Verify the resource s State attribute See the engine and agent logs in var VRTSvcs engine_A log and var VRTSvcs agent_A log for information on why the resource cannot be brought online 572 Troubleshooting and recovery for VCS Troubleshooting resources Agent not running The resource s agent process is not running Recommended Action Use hastatus summary to see if the agent is listed as faulted Restart the agent haagent start resource type sys system Invalid agent argument list The scripts are receiving incorrect arguments Recommended Action Verify that the arguments to the scripts are correct Use the output of hares display resource to see the value of the ArgListValues attribute If the ArgList attribute was dynamically changed stop the agent and restart it To stop the agent haagent stop resource type sys system To restart the agent haagent start resource type sys system The Monitor entry point of the disk group agent returns ONLINE even if the disk group i
248. e cluster first highest level If you are at the Cluster Groups view the Groups tab is active According to the table you are at the second position at the cluster third level As you drill down into lower and lower view levels using tabs and other links the number and variety of the objects and tasks available to you generally becomes fewer and fewer About the navigation bar The navigation bar is located along the top of the status pane and indicates two things m Your current level in the current view heirarchy m The lineage of the object detailed in your current view For example assume that you are at the startup view the Cluster Summary view The navigation bar shows Cluster where Cluster is the name of the local managed cluster the cluster on which VCS is installed Cluster alone with no additional links in the trail indicates that you are at the top level of views Now assume that you perform the following actions in order if you are currently using the console follow along m Clicka cluster name in the table m Click the Groups tab Administering the cluster from the Cluster Management Console 85 Overview of the Cluster Management Console m Click a service group name in the table m Click the Resources tab m Click a resource name in the table You arrive at a Resource Summary view The navigation bar shows a list of the objects that you selected to navigate to this view The object n
249. e cluster from the Cluster Management Console 91 About online help About online help The Cluster Management Console has a cross platform browser based online help system You can access the help system by clicking Help in the upper right corner of the console window You can view the Cluster Management Console online help system using a supported web browser Online help navigation Navigation tools appear at the top of all Cluster Management Console help pages Roll the mouse pointer over a tool to see the name of the tool m The Show Navigation tool displays the Table of Contents This tool also displays the Show in Contents tool m The Show in Contents tool shows the location of the current help page in the Table of Contents m The Next and Previous tools right and left arrows enable you to navigate help topics from the current help page These tools take you to the next or previous topic in the order that it appears in the Table of Contents m The Print tool enables you to print the current help topic m The Bookmark tool enables you to add the current help topic to your Favorite s list in your web browser Locating topics Cluster Management Console online help offers several options for locating topics When you click the Help button in the upper right corner of the screen the help system opens in context to the current view Clicking the Show in Contents tool shows the Table of Contents location of the current h
250. e clusters you can also add a role to an operating system user group See User privileges for OS user groups in secure clusters on page 66 For example an administrator may assign a user the Group Administrator role for specific service groups Now the user has privileges to perform operations on the specific service groups You can manage users and their privileges from the command line or from the graphical user interface See Managing VCS users from the command line on page 251 See Administering user profiles on page 174 User privileges for OS user groups in secure clusters In secure clusters you can assign privileges to native users individually or at an operating system OS user group level For example you may decide that all users that are part of the OS Administrators group get administrative privileges to the cluster or to a specific service group Assigning a VCS role to a user group assigns the same VCS privileges to all members of the user group unless you specifically exclude individual users from those privileges When you add a user to an OS user group the user inherits VCS privileges assigned to the user group Assigning VCS privileges to an OS user group involves adding the user group in one or more of the following attributes m AdministratorGroups for a cluster or for a service group m OperatorGroups for a cluster or for a service group m Guests for a cluster or for a service g
251. e for the group contains the list of clusters over which the group spans 684 Glossary hashadow Process A process that monitors and when required restarts HAD High Availability Daemon HAD The core VCS process that runs on each system The HAD process maintains and communicates information about the resources running on the local system and receives information about resources running on other systems in the cluster Jeopardy Anode is in jeopardy when it is missing one of the two required heartbeat connections When a node is running with one heartbeat only in jeopardy VCS does not restart the applications on a new node This action of disabling failover is a safety mechanism that prevents data corruption LLT Low Latency Transport LLT is a communication mechanism of the VCS engine that provides kernel to kernel communications and monitors network communications main cf The file in which the cluster configuration is stored Network Partition If all network connections between any two groups of systems fail simultaneously a network partition occurs When this happens systems on both sides of the partition can restart applications from the other side resulting in duplicate services or split brain A split brain occurs when two independent systems configured in a cluster assume they have exclusive access to a given resource usually a file system or volume The most serious problem caused by a network partition is t
252. e group dependency configurations The role of service group dependencies 415 Service group dependency configurations In the following tables the term instance applies to parallel groups only If a parallel group is online on three systems for example an instance of the group is online on each system For failover groups only one instance of a group is online at any time The default dependency type is Firm Failover parent Failover child online local Failover Child Child is Parent stays online Child stays online soft online on online on If Child fails over to same system same another system system Parent migrates to the same system If Child cannot fail over Parent remains online online local Failover Child Child is Parent taken offline Child stays online firm online on online on If Child fails over to same system same another system system Parent migrates to the same system If Child cannot fail over Parent remains offline online local Failover Child Child is Parents taken offline Child taken offline hard online on online on before Child is taken If Child fails over same system same offline Parent migrates to system If Child fails over to the same system another system Parent migrates to another system If Child cannot fail Over Parent remains offline If Child cannot fail over Parent remains offline 416 The role of service group dependencies Service group dependency config
253. e groups ccccscesesesseseseeceseseteeseseeeesees VCS behavior with service group dependencies Section IV Administration Beyond the basics Chapter 13 VCS event notification About VCS event notification 0 ceccsesesesesessssssesecesesesesessesesssseeseseseseseeees 430 Components of VCS event notification oo cccccscsessctscsesessesesesecsssecsees 433 VCS events anid traps dcasenalcdsedsiieinciohe sot eae eee ones 435 Monitoring aggregate events cee eceecesesseseeseeseseeseeseeaeseesecaeeneeseseeseeaeeaeeneets 443 Detecting complementary events oo ee cceccescessseseeseseeeceseeeeceseeceseseseeseseeaees 444 Configuring notification ccesesssssesesesesessessseesessesesesesessseeeesesseseseseseeess 444 Chapter 14 Section V Chapter 15 Chapter 16 Chapter 17 Chapter 18 Contents VCS event triggers About VCS event triggers cccccceccssesssssseseeseseseeseseeeeseseseeseessesenseseeeeseeeeeesees 446 Using event trig gers oorner erna ean r EE EAER oA TE EO SEERE SES 446 List of event triggers oo eeccseessesessssesesscesesesesesessssssssssesesesesesesssssesssessseeseses 447 Multi cluster configurations Connecting clusters Creating global clusters How VCS global clusters work c ccccccesesssesesssseseseeceseseeceseeeeceseeeeseeeeseseteeseees VCS global clusters The building blocks 2 0 ececeseseeseseteeceseeeeseeeteesees Prerequisites for global clusters ccecseesesesesssseses
254. e hadebug command m Type and dimension integer scalar m Default 0 ConnectorState system use only Indicates the state of the wide area connector wac If 0 wac is not running If 1 wac is running and communicating with the VCS engine m Type and dimension integer scalar m Default Not applicable 650 VCS attributes Cluster attributes Table D 5 Cluster attributes CounterInterval Intervals counted by the attribute GlobalCounter indicating approximately how often a broadcast occurs that will cause the user defined GlobalCounter attribute to increase The default value of the GlobalCounter increment can be modified by changing CounterInterval If you increase this attribute to exceed five seconds consider increasing the default value of the ShutdownTimeout attribute m Type and dimension integer scalar m Default 5 DumpingMembership Indicates that the engine is writing to disk system use only m Type and dimension integer scalar m Default Not applicable EngineClass The scheduling class for the VCS engine HAD user defined m Type and dimension string scalar m Default RT EnginePriority The priority in which HAD runs user defined m Type and dimension string scalar m Default Not applicable EngineShutdown Defines the options for the hastop command The attribute can user defined assume the following values Enable Process all hastop commands This is the de
255. e maximum time specified by the static attribute MonitorTimeout UNABLE TO OFFLINE The agent attempted to offline the resource but the resource did not go offline This flag is also set when a resource faults and the clean entry point completes successfully but the subsequent monitor hangs or is unable to determine resource status m Type and dimension integer scalar m Default Not applicable Group system use only String name of the service group to which the resource belongs m Type and dimension string scalar m Default Not applicable Table D 1 VCS attributes 617 Resource attributes Resource attributes IState system use only The internal state of a resource In addition to the State attribute this attribute shows to which state the resource is transitioning Values NOT WAITING Resource is not in transition WAITING TO GO ONLINE Agent notified to bring the resource online but procedure not yet complete WAITING FOR CHILDREN ONLINE Resource to be brought online but resource depends on at least one offline resource Resource transitions to WAITING TO GO ONLINE when all children are online WAITING TO GO OFFLINE Agent notified to take the resource offline but procedure not yet complete WAITING TO GO OFFLINE propagate Same as above but when completed the resource s children will also be offline WAITING TO GO ONLINE reverse Resource waiting to be brought online but
256. e resources 202 Administering the cluster from Cluster Manager Java console Administering resources Adding a RemoteGroup resource from the Java Console A RemoteGroup resource is typically useful in scenarios where resources configured in a local service group are dependant on the state of a remote service group For example a web server application running in a local cluster could be dependant on a database application running in a remote cluster A RemoteGroup resource monitors the state of a remote service group in a local cluster Once you have added the RemoteGroup resource to a local service group you can link the resource to the existing resources of the service group You must have administrative privileges to configure RemoteGroup resources See Configuring the RemoteGroup agent on page 296 To add a RemoteGroup resource 1 On the Tools menu click Add Remote Group Resource or Click Configure Remote Group Resource Wizard in the Cluster Explorer toolbar Read the information on the Welcome dialog box and click Next In the Remote Group Resource Name dialog box specify the name of the resource and the service group to which the resource will be added Click Next In the Remote Cluster Information dialog box Specify the name or IP address of a node in the remote cluster Specify the port on the remote node on which the resource will communicate Specify a username for the remote cluster Spe
257. e root is on shared storage it may exist in the same disk group that is to contain the application s data Third Install the application in the zone See Installing the application on page 599 Fourth Create the application service group and configure its resources See Configuring the service group for the application on page 600 Fifth Configure zone specific resources and set up inter zone communication See Configuring the service group for the application on page 600 594 Configuring VCS in non global zones Configuring VCS in zones Prerequisites for configuring VCS in zones All nodes hosting applications configured in zones must be running the same version of the operating system VCS supports UFS and VxFS mounts for the zone root VCS does not support CFS mounts for the zone root The autoboot property for the zone must be set to false Mounts must meet one of the following two conditions m Usea loop back file system In this case all mounts used by the application must be part of the zone configuration and must be configured in the service group For example you can create a zone z ora and define the file system containing the application s data to have the mount point as oradata When you create the zone you can define a path in the global zone for example export home oradata that the mount directory in the non global zone maps to The MountPoint attribute of the
258. e service group was online prior to VCS being brought up VCS will not inadvertently bring the service group online on another system Recommended Action Use the output of hagrp display service_group to see the value of the ProbesPending attribute for the system s service group It should be zero To determine which resources are not probed verify the local Probed attribute for each resource on the specified system Zero means waiting for probe result 1 means probed and 2 means VCS not booted See the engine and agent logs for information Troubleshooting and recovery for VCS 571 Troubleshooting resources Troubleshooting resources This section cites the most common problems associated with bringing resources online and taking them offline Bold text provides a description of the problem Recommended action is also included where applicable Service group brought online due to failover VCS attempts to bring resources online that were already online on the failed system or were in the process of going online Each parent resource must wait for its child resources to be brought online before starting Recommended Action Verify that the child resources are online Waiting for service group states The state of the service group prevents VCS from bringing the resource online Recommended Action Review the state of the service group See Cluster and system states on page 605 Waiting for child resources One or mor
259. e system s CPU usage using various factors The values for ActionTimeLimit and NotifiyTimeLimt represent the time in seconds The values for ActionThreshold and NotifyThreshold represent the threshold in terms of CPU percentage utilization See Monitoring CPU usage on page 559 m Type and dimension string association m Default Enabled 0 NotifyThreshold 0 NotifyTimeLimit 0 ActionThreshold 0 ActionTimeLimit 0 Action NONE CurrentLimits System maintained calculation of current value of Limits system use only CurrentLimits Limits additive value of all service group Prerequisites m Type and dimension integer association m Default Not applicable DiskHbStatus Deprecated attribute Indicates status of communication disks system use only on any system m Type and dimension string association m Default Not applicable Table D 4 VCS attributes System attributes 643 System attributes DynamicLoad user defined System maintained value of current dynamic load The value is set external to VCS with the hasys 1load command m Type and dimension integer scalar m Default 0 EngineRestarted system use only Indicates whether the VCS engine HAD was restarted by the hashadow process on a node in the cluster The value 1 indicates that the engine was restarted 0 indicates it was not restarted m Type and dimension boolean scalar m Default 0 Frozen
260. eOnlining system use only Indicates that VCS engine invoked the preonline script however the script has not yet returned with group online m Type and dimension integer scalar m Default Not applicable PreonlineTimeout user defined Defines the maximum amount of time in seconds the preonline script takes to run the command hagrp online nopre for the group Note that HAD uses this timeout during evacuation only For example when a user runs the command hastop local evacuate and the Preonline trigger is invoked on the system on which the service groups are being evacuated m Type and dimension integer scalar m Default 300 636 VCS attributes Service group attributes Table D 3 Service group attributes Prerequisites user defined An unordered set of name value pairs denoting specific resources required by a service group If prerequisites are not met the group cannot go online The format for Prerequisites is Prerequisites Name Value name2 value2 Names used in setting Prerequisites are arbitrary and not obtained from the system Coordinate name value pairs listed in Prerequisites with the same name value pairs in Limits See About system limits and service group prerequisites on page 392 m Type and dimension integer association PrintTree user defined Indicates whether or not the resource dependency tree is written to the configuration file The value 1
261. ecific shared memory and semaphore requirements Groups G3 and G4 are middle tier applications with no specific memory or semaphore requirements include types cf cluster SGWM demo system LargeServerl Capacity 200 Limits ShrMemSeg 20 Semaphores 10 Processors 12 LoadWarningLevel 90 LoadTimeThreshold 600 system LargeServer2 Capacity 200 Limits ShrMemSeg 20 Semaphores 10 Processors 12 LoadWarningLevel 70 LoadTimeThreshold 300 system MedServerl Capacity 100 Limits ShrMemSeg 10 Semaphores 5 Processors 6 system MedServer2 Capacity 100 Limits ShrMemSeg 10 Semaphores 5 Processors 6 400 Controlling VCS behavior Sample configurations depicting workload management group G1 SystemList LargeServerl LargeServer2 MedServer2 SystemZones LargeServerl1 0 MedServer2 1 AutoStartPolicy Load AutoStartList LargeServer1 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 10 group G2 SystemList LargeServerl LargeServer2 MedServer2 SystemZones LargeServerl1 0 MedServer2 1 AutoStartPolicy Load AutoStartList LargeServer1 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 10 group G3 SystemList LargeServerl LargeServer2 MedServer2 SystemZones LargeServerl1 0 MedServer2 1 AutoStartPolicy Load AutoStartList FailOverPolicy
262. econdary tab bar click Systems In the Cluster Systems view in the Systems Listing table click a linked system name In the Systems Summary view you can choose a task from the task pane or select another system level view using the tabs on the secondary tab bar Each view contains information and tasks for administering a single system Freezing a System Freeze a system to prevent the service groups that it hosts from failing over to another system Use this procedure before performing a system upgrade To freeze a system 1 In the Cluster Systems view in the Systems Listing table click the linked name of the system that you want to freeze In the System Summary view in the Operations task panel click Freeze In the Freeze System dialog box select one or both of the following options m Select Persistent if you want the system to retain a frozen state when the cluster is rebooted m Select Evacuate if you want the system s active service groups to fail over to another system in the cluster before the freezing operation takes place Click OK 128 Administering the cluster from the Cluster Management Console Administering systems You can freeze one or more systems at a time directly from the Cluster Systems view by selecting the check boxes for systems in the Systems Listing table and then clicking Freeze in the Operations task panel Unfreezing a system Unfreeze a frozen system to perform online or offl
263. ecting to other systems the system icon changes its appearance to indicate the link or disk heartbeat is down In this situation a jeopardy warning may appear in the ScreenTip for this system To access the System Connectivity view 1 From Cluster Explorer click a cluster in the configuration tree 2 Inthe view panel click the System Connectivity tab Remote Cluster Status view Administering the cluster from Cluster Manager Java console 159 About Cluster Explorer Note This view requires the VCS Global Cluster Option The Remote Cluster Status View provides an overview of the clusters and global groups in a global cluster environment Use this view to view the name address and status of a cluster and the type Icmp or IcmpS and state of a heartbeat ea All Clusters Status amp Remote Cluster Status View LoL Cluster Name Cluster Address Status amp Ci Local Cluster 10 212 88 28 Running Heartbeat Status SB 10 212 88 29 Running Temp ALIVE All Global Groups Status Clusters gt 3 Global Groups a sd eeo Offline Offline This view enables you to declare a remote cluster fault as a disaster disconnect or outage Point to a table cell to view information about the VCS object To access the Remote Cluster Status view 1 From Cluster Explorer click a cluster in the configuration tree 2 Inthe view panel click the Remote Cluster Sta
264. ects against all elements in the IT environment that might be trying to write illegally to storage not only VCS related elements Membership arbitration combined with data protection is termed I O Fencing Examples of VCS operation with I O fencing 346 About communications membership and data protection in the cluster Examples of VCS operation with I O fencing This topic describes the general logic employed by the I O fencing module along with some specific example scenarios About the I O fencing algorithm To ensure the most appropriate behavior is followed in both common and rare corner case events the fencing algorithm works as follows The fencing module is designed to never have systems in more than one subcluster remain current and valid members of the cluster In all cases either one subcluster will survive or in very rare cases no systems will The system with the lowest LLT ID in any subcluster of the original cluster races for control of the coordinator disks on behalf of the other systems in that subcluster If a system wins the race for the first coordinator disk that system is given priority to win the race for the other coordinator disks Any system that loses a race will delay a short period of time before racing for the next disk Under normal circumstances the winner of the race to the first coordinator disk will win all disks This ensures a clear winner when multiple systems race for the coordinator
265. ecurity option you would like to perform 1 3 q 2 If Veritas Cluster Server is not configured in the system from where you started the installvcs program enter the name of a node in the cluster that you want to disable the Authentication Service Enter the name of one system in the VCS Cluster that you would like to disable Veritas Security Services north Review the output as the installer proceeds with a basic verification Press Enter at the prompt to confirm that you want to disable the Authentication Service Would you like to disable Veritas Security Services on this cluster y n q y Y Review the output as the installer modifies the Veritas Cluster Server configuration files to disable the Authentication Service and starts Veritas Cluster Server in a non secure mode 282 Administering the cluster from the command line Administering resource types Administering resource types Adding deleting and modifying resource types After creating a resource type use the command haattr to add its attributes By default resource type information is stored in the types cf configuration file To add a resource type hatype add resource type To delete a resource type hatype delete resource type You must delete all resources of the type before deleting the resource type To add or modify resource types in main cf without shutting down VCS hatype modify resource type SourceFile resource type cf The information
266. ed m Only a single level parent child relationship can be configured as a hard dependency m Only one parent and one child group can be configured in a hard dependency m Bringing the child group online does not automatically bring the parent online m Taking the parent group offline does not automatically take the child offline m Bringing the parent online is prohibited if the child is offline Multiple parent service groups may depend on a child service group although a parent group may depend on only one child group m A group dependency tree may be at most five levels deep E groupy B am groupx gt Loto global firm S Pad onne remote firm C groupz gt C groupw gt groupx gt T online local La firm lt groupy gt C groupU D online global ien T GA online global firm C groupz Dd m You cannot link two service groups whose current states violate the relationship 414 The role of service group dependencies About service group dependencies For example all link requests are accepted if all instances of parent group are offline All link requests are rejected if parent group is online and child group is offline except in offline dependencies All online global online remote link requests to link two parallel groups are rejected All online local link requests to link a parallel parent group to a failover child group are rejected Servic
267. ed and sent to the VCS engine For example if Frequency is set to 10 VCS computes the average monitor time after every 10 monitor cycles m ExpectedValue The expected monitor time in milliseconds for a resource VCS sends a notification if the actual monitor time exceeds the expected monitor time by the ValueThreshold So if you set this attribute to 5000 for a FileOnOff resource and if ValueThreshold is set to 40 VCS will send a notification only when the monitor cycle for the FileOnOff resource exceeds the expected time by over 40 that is 7000 milliseconds m ValueThreshold The maximum permissible deviation in percent from the expected monitor time When the time for a monitor cycle exceeds this limit VCS sends a notification about the sudden increase or decrease in monitor time For example a value of 100 means that VCS sends a notification if the actual monitor time deviates from the expected time by over 100 VCS sends these notifications conservatively If 12 consecutive monitor cycles exceed the threshold limit VCS sends a notification for the first spike and then a collective notification for the next 10 consecutive spikes m AvgThreshold The threshold value in percent for increase in the average monitor cycle time for a resource VCS performance considerations 561 VCS agent statistics VCS maintains a running average of the time taken by the monitor cycles of a resource The first such computed running ave
268. ed if a string begins with a letter and contains only letters numbers dashes and underscores _ For example a string defining a network interface such as hme0 or ethO does not require quotes as it contains only letters and numbers However a string defining an IP address contains periods and requires quotes such as 192 168 100 1 Signed integer constants are a sequence of digits from 0 to 9 They may be preceded by a dash and are interpreted in base 10 Integers cannot exceed the value of a 32 bit signed integer 21471183247 A boolean is an integer the possible values of which are 0 false and 1 true 52 VCS configuration concepts About VCS attributes Attribute dimensions VCS attributes have the following dimensions Scalar Vector Keylist Association A scalar has only one value This is the default dimension A vector is an ordered list of values Each value is indexed using a positive integer beginning with zero Use a comma or a semi colon to separate values A set of brackets after the attribute name denotes that the dimension is a vector For example an agent s ArgList is defined as static str ArgList RVG DiskGroup Primary SRL Links A keylist is an unordered list of strings and each string is unique within the list Use a comma or a semi colon to separate values For example to designate the list of systems on which a service group will be started with VCS
269. edServer3 MedServer4 MedServer5 FailOverPolicy Load Load 50 406 Controlling VCS behavior Sample configurations depicting workload management group Application3 SystemList LargeServerl LargeServer2 MedServerl MedServer2 MedServer3 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 MedServer4 MedServer5 FailOverPolicy Load Load 50 group Application4 SystemList LargeServerl LargeServer2 LargeServer3 MedServer4 MedServer3 1 MedServer3 LargeServer3 MedServerl MedServer2 MedServer3 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 MedServer4 MedServer5 FailOverPolicy Load Load 50 group Application5 SystemList LargeServerl LargeServer2 MedServerl MedServer2 MedServer3 MedServer4 MedServer3 1 MedServer3 LargeServer3 MedServer4 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 MedServer4 MedServer5 FailOverPolicy Load Loa
270. edundancy to any other server Figure 2 6 N 1 failover 40 About cluster topologies Advanced failover configurations N to N configuration An N to N configuration refers to multiple service groups running on multiple servers with each service group capable of being failed over to different servers For example consider a four node cluster with each node supporting three critical database instances Figure 2 7 N to N configuration i If any node fails each instance is started on a different node ensuring no single node becomes overloaded This configuration is a logical evolution of N 1 it provides cluster standby capacity instead of a standby server SG Service Group N to N configurations require careful testing to ensure all applications are compatible Applications must also have complete control of where service groups fail when an event occurs About cluster topologies 41 Cluster topologies and storage configurations Cluster topologies and storage configurations This section describes commonly used cluster topologies along with the storage configuration used to support the topologies Basic shared storage cluster In this configuration a single cluster shares access to a storage device typically over a SAN An application can only be started on a node with access to the required storage For example in a m
271. ee right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Enable and click the appropriate system from the menu Click All Systems to enable the group on all systems To enable a service group from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Enable Service Group Click the service group Select the Per System check box to enable the group on a specific system instead of all systems Click Apply Administering the cluster from Cluster Manager Java console 189 Administering service groups Disabling a service group Disable a service group to prevent it from coming online This process temporarily stops VCS from monitoring a service group on a system undergoing maintenance operations To disable a service group from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Disable and click the appropriate system in the menu Click All Systems to disable the group on all systems To disable a service group from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Disable Serv
272. eeeseeeeseseeeeseseeaees Using the wait option in scripts Running a virtual fire drill Administering simulated clusters from the command line 000 285 Configuring applications and resources in VCS About configuring resources and applications ccccceeseseseeseseseeseseeeeees 288 About VCS bundled agents cccccsesesssseseseeceseeseseeeeeseeeeseseeseseeeeseeeseesees 289 Which agents Should I use eccceeesessesesseseseseeceseeeeseseeseseeceseseeseseneeseeeseesees 293 Configuring the RemoteGroup agent ccceeseseseseseeesetssseeeecsesesesesesesssees 296 Configuring application Service groups c ceecesesesesesesesesstssseceseseseseseseseseeees 300 Configuring NFS service groups Testing resource failover using virtual fire drills cccceeeeeseeeteees 315 Predicting VCS behavior using VCS Simulator AbOUtVESSimulator a chal a oe a a a a a aaie AREE 318 Installing VOS Sim laitor serenon E E E E 318 Administering VCS Simulator from the Java Console cccceseseseseeeeees 320 Administering VCS Simulator from the command line 0 0 0 eee 326 7 8 Contents Section III VCS communication and operations Chapter 10 About communications membership and data protection in the cluster About cluster communications 0 eeesseseeeeeceseeceseececeseeceseseseeseeeseeeeees 334 About cluster membership 0 c cc cccccscsscscsecssescescsscssesecscescesesecsecseeeesecseenseas 338
273. eeesesceesseeesseseaes 221 Editing attributes 01 ccc cics cage noin E aE ages nied sieie E 221 Querying the cluster Configuration 0 cc eecsesssesesesesssseseeeeesesesesesesseeeees 224 Setting up VCS event notification using the Notifier wizard 0 Administering logs 00 Administering VCS Simulator Chapter 7 Chapter 8 Chapter 9 Contents Administering the cluster from the command line About administering VCS from the command line oe eseeseeeeeeeeees 234 Installing a VCS license Starting VCS woe Stopping VES n ea caeaniana aan cman vent Logging ONO VCS siszictees sessstecsciesc E E eiae ER Managing VCS configuration files oo cesssssesssesesesesesesesseseesseeeseseseseeeees 242 Managing VCS users from the command line ceeeesesessseeseseseeeseseeeseees 251 Querying VOS e costes chs Sie A re ERS 254 Administering Service groups cccescscesesseseseseeceseseeeeseeeeseseeceseeeeseseeeeseseeeees 261 Administering agents esonora aere aE oahavissins 267 Administering resources sseseessssseseessseserersrstsestsnestststenestssenesesenerstsnsreetsene 268 Administering systems aiena a eased R R N O ER 276 Administering clusters s eneseseseesesoseseeseseseresstsesesrsntststsnesesessenesesenerseseseeeesese 276 Enabling and disabling Security Services ccccecesesesseseteeeeseseeeeseeeseeseeees 280 Administering resource types ccccccescscesesseseseseeseseseeseseeeesese
274. eeseeeeaees 62 How administrators assign roles to users scecesessssesesseceseeeeseseeeeceeeeeseseens 66 User privileges for OS user groups in secure clusters 0 ceeeeseeseeseteeees 66 About VCS privileges for users with multiple roles 0 0 cesssseseeeeeeeeesees 68 6 Contents Chapter 5 Chapter 6 Administering the cluster from the Cluster Management Console About Veritas Cluster Management Console ccceseessesesesseseseeseeeseeeeseees 70 Verifying installation and browser requirements ccccseseseseseeeseeeseeees 72 Configuring the Cluster Management Console manually ccceee 72 Logging in to the Cluster Management Console cececesesseeseeseseseeeeseeees 74 Logging out of the Cluster Management Console cccssssesesesecesesesesesees 75 Overview of the Cluster Management Console cccesssseseseeeseseseetetsteeeeeees 76 Aboutionline Help E eae ns tees doce sees 91 Administering users ec eeceecesesssseseeeseseseeseseeceseeseceseeeseseeseseeeesesesaeseeeseeeeetaes 93 Administering a clister ee a acne eater neers 96 Administering service groups ccecseessssesesseseseeseseseeceseseeseseseeseseeceseeeeseeeeeeaes 98 Administering resources eonen EN EE EEE AN EEEE 115 Administering resource types ccceccccssssesesssceseseeseseseeseseeceseeeeeseeeeseseeeeseeees 125 Administering systems aar n E EEE 127 Administering attributes 0 0 ceccceesessssescscesese
275. elp page You can also access the Index and Search features to find a topic To use the Index feature Inthe help viewer window click the Index tab The index opens in the task pane enabling you to browse topics If you select index entries with more than one reference a list of options for that term displays Select a reference that seems most relevant to navigate to the corresponding topic page 92 Administering the cluster from the Cluster Management Console About online help To use the Search feature 1 2 In the help viewer window click the Search tab Enter the term to be searched in the field provided The Cluster Management Console help system displays all matches for this term Select an item to display the help topic for that item Administering the cluster from the Cluster Management Console 93 Administering users Administering users Users can be created with the roles of cluster administrator cluster operator or cluster guest They can also be created with the roles of service group administrator service group operator or service group guest for specific service groups See About the VCS user privilege model on page 61 You can view a list of users in the Cluster Users view To navigate to the list in the Cluster Users view 1 On the navigation bar click Cluster 2 On the secondary tab bar click Users In the Cluster Users view you can choose a task from the task pane or select ano
276. em represents the name of the system m available_capacity represents the system s AvailableCapacity attribute AvailableCapacity Capacity sum of Load for system s online groups 450 VCS event triggers List of event triggers multinicb event trigger Description Usage Invoked when a network device that is configured as a MultiNICB resource changes its state The trigger is also always called in the first monitor cycle VCS provides a sample trigger script for your reference You can customize the sample script according to your requirements multinicb_postchange triggertype resource name device name previous state current state monitor_heartbeat nofailover event trigger triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 resource name represents the MultiNICB resource that invoked this trigger device name represents the network interface device for which the trigger is called previous state represents the state of the device before the change The value 1 indicates that the device is up 0 indicates it is down current state represents the state of the device after the change monitor heartbeat an integer count which is incremented in every monitor cycle The value 0 indicates that the monitor routine is called for first time Description Called from the lowest numbered system in RUNNING state when a service group cannot
277. ensures that VVR volumes are made writable and the DNS agent ensures that name services are resolved to the DR site The application can be started at the DR site and run there until the problem with the local storage is corrected If the storage problem is corrected you can switch the application to the primary site using VCS Switching the service group Before switching the application back to the original primary RDC zone you must resynchronize any changed data from the active DR site since the failover This can be done manually through VVR or by running a VCS action from the RVGPrimary resource To switch the service group 1 Inthe Service Groups tab of the configuration tree right click the resource 542 Setting up replicated data clusters Setting up a fire drill 2 Click Actions 3 Specify the details of the action E Fire Action RYGPrimary oradg Action fastfailback w System thor122 X Action Arguments m From the Action list choose fast failback m Click the system on which to execute the action m Click OK This begins a fast failback of the replicated data set You can monitor the value of the ResourcelInfo attribute for the RVG resource to determine when the resynchronization has completed 4 Once the resynchronization completes switch the service group to the primary cluster In the Service Groups tab of the of the Cluster Explorer configuration t
278. enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values Administering the cluster from the Cluster Management Console 129 Administering systems m The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List To edit a system attribute using the task pane 1 In the Cluster Summary view in the Clusters Listing table check the check box preceding the line item for each system with the attribute that you want to edit Select all systems by checking the check box at the top In the Configuration task panel click Edit Attribute Specify the attribute details See To edit a system attribute using the attribute table in Modifying system attributes on page 128 130 Administering the cluster from the Cluster Management Console Administering attributes Administering attributes The Cluster Management Console enables you to edit the attributes of cluster objects clusters service groups systems resources and resource types Ensure that the cluster configuration is open in read write mode before editing attributes Attributes for an object are available in any object summary view such as the Cluster Summary view or the R
279. epresents the name of the service group that went online 452 VCS event triggers List of event triggers preonline event trigger Description Usage To enable the trigger To disable the trigger Indicates that when the HAD should call a user defined script before bringing a service group online in response tothe hagrp online command or a fault If the trigger does not exist VCS continues to bring the group online If the script returns 0 without an exit code VCS runs the hagrp online nopre command with the checkpartial option if appropriate If you do want to bring the group online define the trigger to take no action This event trigger is configurable preonline triggertype system service_group whyonlining system_where_group_faulted m triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 m system represents the name of the system m service_group represents the name of the service group on which the hagrp command was issued or the fault occurred m whyonlining represents two values FAULT Indicates that the group was brought online in response to a group failover or switch MANUAL Indicates that group was brought online manually on the system that is represented by the variable system m system_where_group_faulted represents the name of the system on which the group has faulted or switched This variable is opti
280. er When the fencing module on System0 successfully controls the coordinator disks HAD carries out any associated policy connected with the membership change System1 is blocked access to the shared storage if this shared storage was configured in a service group that was now taken over by System0 and imported Figure 10 4 I O Fencing example with system failure system0 system1 a FT Coordinator Disks 348 About communications membership and data protection in the cluster Examples of VCS operation with I O fencing Four system cluster where cluster interconnect fails In this example the cluster interconnect fails in such a way as to split the cluster from one four system cluster to two system clusters The cluster performs membership arbitration to ensure that only one subcluster remains Due to loss of heartbeat System0 and System1 both believe System2 and System3 are down System2 and System3 both believe System0 and System1 are down The progression of I O fencing operations are as follows Figure 10 5 Four system cluster where cluster interconnect fails System0O System1 System2 System 3 SAN Connection LLT Links PT Coordinator Disks m LLT on each of the four systems no longer receives heartbeat messages from the systems on the other side of the interconnect failure on any of the configured LLT interfaces for the peer inactive timeout configured time m LLT
281. er User Manager enables you to add and delete user profiles and to change user privileges If VCS is not running in secure mode User Manager enables you to change user passwords You must be logged in as Cluster Administrator to access User Manager E User Manager New User _ change Password E ra Privileges __Removeuser User To access user Manager From Cluster Explorer click User Manager on the File menu Command Center Administering the cluster from Cluster Manager Java console 163 Accessing additional features of the Java Console Command Center enables you to build and execute VCS commands most commands that are executed from the command line can also be executed through this window The left pane of the window displays a Commands tree of all VCS operations The right pane displays a view panel that describes the selected command The bottom pane displays the commands being executed The commands tree is organized into Configuration and Operations folders Click the icon to the left of the Configuration or Operations folder to view its subfolders and command information in the right pane Point to an entry in the commands tree to display information about the selected command Ele View Help Configuration 1 2 Configuration File E E Cluster objects E E Attributes 2 Controls m f Availabilty VCSHP10_71_0214a Command Center admin Cluster Administrator b am
282. er admin Cluster Administrator Fle Edt View Tools Help hha 264442 8090788240 amp 3 9 E Status amp Service Groups FY System Connectivity Properties Fa i gm CMC Status View YCSHP10_71_0214a 2 Sry G Status all Groups status on systems CSHP10_71_0214a amp sem_scz Systems gt 3 E ga newgroup2 E Fe newgroup ATS veshp10 veshp71 ere O Online Offline GF ClusterService Online Offline Bscm_sct 5cm_sc2 Be newgroup2 S Online Offline WCSHP10_71_0214a The display is divided into three panes The top pane includes a toolbar that enables you to quickly perform frequently used operations The left pane contains a configuration tree with three tabs Service Groups Systems and Resource Types The right pane contains a panel that displays various views relevant to the object selected in the configuration tree To access Cluster Explorer 1 Logon to the cluster 2 Click anywhere in the active Cluster Monitor panel or Right click the selected Cluster Monitor panel and click Explorer View from the menu Administering the cluster from Cluster Manager Java console 149 About Cluster Explorer Cluster Explorer toolbar The Cluster Explorer toolbar contains 18 buttons Available operations are described below Note Some buttons may be disabled depending on the type of cluster local or global and the priv
283. er Administrator Tom Group Operator Cluster Administrator Situation Roles at an individual and OS user group level secure clusters only Tom Group Operator Group Operator OSUserGroup1 Rule VCS gives precedence to the role Cluster Administrator granted at the individual level Situation Different roles for different OSUserGroup1 Cluster Administrator OS user groups secure clusters only Cluster Rule VCS grants the highest privilege Administrators or a union of all privileges of all user OSUserGroup2 groups to the user Cluster Operators Situation Roles at an individual and OS OSUserGroup1 Group Operator user group level secure clusters only Cluster Rule VCS gives precedence to the role Administrators granted at the individual level OSUserGroup2 You can use this behavior to exclude specific users from inheriting VCS privileges assigned to their OS user groups Cluster Operators Tom Group Operator Chapter Administering the cluster from the Cluster Management Console m About Veritas Cluster Management Console m Verifying installation and browser requirements m Configuring the Cluster Management Console manually m Logging in to the Cluster Management Console Logging out of the Cluster Management Console m Overview of the Cluster Management Console m About online help m Administering users m Administering a cluster m Administering service groups m Administering re
284. er Java Console to set up your global cluster Other heartbeats must be configured manually ClusterService group The ClusterService group must be configured with the wac NIC and IP resources It is configured automatically when VCS is installed or upgraded or by the GCO configuration wizard The service group may contain additional resources for Cluster Management Console and notification if these components are configured If you entered a license that includes VCS global cluster support during the VCS install or upgrade the ClusterService group including the wide area connector process is automatically configured If you add the license after VCS is operational you must run the GCO Configuration wizard See Running the GCO configuration wizard on page 472 Replication setup VCS global clusters are also used in case of disaster recovery so you must set up real time data replication between clusters You can use VCS agents for supported replication solutions to manage the replication If your configuration uses Veritas Volume Replicator you must add the VTRSvcsvr package to all systems Connecting clusters Creating global clusters 471 Setting up a global cluster Setting up a global cluster This section describes the steps for planning configuring and testing a global cluster It describes an example of converting a single instance Oracle database configured for local high availability in a VCS cluster to a h
285. er Management Console or the command line to configure resources Add required resources to the service group and configure them For example to configure a database in VCS you must configure resources for the database and for the underlying shared storage and network resources Use appropriate agents to configure resources See About VCS bundled agents on page 289 Configuring a resource involves defining values for its attributes See the Veritas Cluster Server Bundled Agents Reference Guide for a description of the agents provided by VCS The resources must be logically grouped in a service group When a resource faults the entire service group fails over to another node Assign dependencies between resources For example an IP resource depends on a NIC resource Bring the service group online to make the resources available VCS provides configuration wizards to configure commonly used resources Application Configuration wizard Creates and modifies Application service groups which provide high availability for applications in a VCS cluster The wizard creates service groups for applications running in global and non global zones NFS Configuration wizard Creates and modifies NFS service groups which provide high availability for fileshares in a VCS cluster RemoteGroup Configuration wizard Configures a RemoteGroup resource to monitor and manage the state of a service group on another cluster See Remote Gro
286. er dialog box indicates privileges for each user Administrator and Operator privileges are separated into the cluster and group levels See About VCS user privileges and roles on page 62 Adding a user 1 From Cluster Explorer click User Manager on the File menu 2 Inthe User Manager dialog box click New User 3 Inthe Add User dialog box UserName Jismith Password rr Confirm Password Cluster Administrator I Cluster Operator I Group Administrator Select Groups MICE ee E Select Groups x Available Groups Selected Groups cmc ISCM_SG1 ClusterService ISCM_SG2 inewgroup2 m Enter the name of the user If the cluster is not running in secure mode enter a password for the user and confirm it m Select the appropriate check boxes to grant privileges to the user To grant Group Administrator or Group Operator privileges proceed to step the next step Otherwise proceed to the last step m Click Select Groups m Click the groups for which you want to grant privileges to the user and click the right arrow to move the groups to the Selected Groups box m Click OK to exit the Select Group dialog box then click OK again to exit the Add User dialog box 4 Click Close Deleting a user oOo A U N e Administering the cluster from Cluster Manager Java console 175 Administering user profiles From Cluster Explorer click User Manager on the File menu In the Use
287. er panel in the Cluster Monitor When prompted enter the name of client system as the host and the client_port as the port Do not enter localhost Starting Cluster Manager Java console You can run the Java Console on Windows or UNIX systems To start the Java Console on Windows systems Double click the Veritas Cluster Manager Java Console icon on the desktop To start the Java Console on UNIX systems After establishing a user account and setting the display type the following command to start Cluster Manager opt VRTSvcs bin hagui The command hagui will not work across firewalls unless all outgoing server ports are open 140 Administering the cluster from Cluster Manager Java console Reviewing components of the Java Console Reviewing components of the Java Console Cluster Manager Java Console offers two windows Cluster Monitor and Cluster Explorer from which most tasks are performed Use Cluster Manager to manage configure and administer the cluster while VCS is running online The Java Console also enables you to use VCS Simulator Use this tool to simulate operations and generate new configuration files main cf and types cf while VCS is offline VCS Simulator enables you to design configurations that imitate real life scenarios without test clusters or changes to existing configurations See Administering VCS Simulator on page 232 Icons in the Java Console The Java Console uses the following i
288. er with the credentials used for the current cluster connection or enter new credentials including the user name password and the domain If you have connected to the remote cluster using the wizard earlier you can use the credentials from the previous connection m Click OK Repeat these steps for each cluster in the global environment In the Remote cluster information dialog box click Next Click Finish Bringing a service group online in a remote cluster 1 In the Service Groups tab of the Cluster Explorer configuration tree of a local cluster right click the service group or Click a local cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Online and click Remote online In the Online global group dialog box T Online global group xi ae Online global group amp usterce O O C System fvcslinux139 Cancel m Click the remote cluster to bring the group online m Click the specific system or click Any System to bring the group online m Click OK In the Question dialog box click Yes Administering global clusters from Cluster Manager Java console 515 Administering global service groups Taking a service group offline in a remote cluster 1 In the Service Groups tab of the Cluster Explorer configuration tree of a local cluster right click the service group or Click a local
289. erService newgroup To access the Query dialog box From Cluster Explorer click Query on the Tools menu or In the Cluster Explorer toolbar click Query 166 Administering the cluster from Cluster Manager Java console Accessing additional features of the Java Console Logs The Logs dialog box displays the log messages generated by the VCS engine VCS agents and commands issued from Cluster Manager to the cluster Use this dialog box to monitor and take actions on alerts on faulted global clusters and failed service group failover attempts Note To ensure the time stamps for engine log messages are accurate make sure to set the time zone of the system running the Java Console to the same time zone as the system running the VCS engine Click the VCS Logs tab to view the log type time and details of an event Each message presents an icon in the first column of the table to indicate the message type Use this window to customize the display of messages by setting filter criteria T VCSHP10_71_0214a Logs admin Cluster Administrator _ ol x Ele View Help VCS Logs agent Logs Command Logs Alerts Edt Fiters Log type description Critical Error Warning Notice Information Log Type Time Description Feb20 2006 4 14 25 AM V 16 6 15004 veshp10 hatrigger Failed to send trigger For postonline script doesn t exist Feb20 2006 4 14 25 AM
290. ernal triggertype 1 For this trigger triggertype 0 system represents the system on which the resource is not going offline resource represents the name of the resource 456 VCS event triggers List of event triggers resstatechange event trigger Description This trigger is invoked under the following conditions Usage To enable the trigger Resource goes from OFFLINE to ONLINE Resource goes from ONLINE to OFFLINE Resource goes from ONLINE to FAULTED Resource goes from FAULTED to OFFLINE When fault is cleared on non persistent resource Resource goes from FAULTED to ONLINE When faulted persistent resource goes online or faulted non persistent resource is brought online outside VCS control Resource is restarted by an agent because resource faulted and RestartLimit was greater than 0 This event trigger is configurable resstatechange triggertype system resource previous_state new_state triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 system represents the name of the system resource represents the name of the resource previous_state represents the resource s previous state new_state represents the resource s new state This event trigger is not enabled by default You must enable resstatechange by setting the attribute TriggerResStateChange to 1 in the main cf file or by issuing the command ha
291. ership on page 27 How VCS ensures application availability When VCS detects an application or node failure VCS brings application services up on a different node in a cluster VCS virtualizes IP addresses and system names so client systems continue to access the application and are unaware of which server they use IP Address Application Storage Storage Introducing Veritas Cluster Server 17 What is a VCS cluster For example in a 2 node cluster consisting of db serverl and db server2 a virtual address may be called db server Clients access db server and are unaware of which physical server hosts the db server About switchover and failover Switchover and failover are the processes of bringing up application services on a different node in a cluster Switchover A switchover is an orderly shutdown of an application and its supporting resources on one server and a controlled startup on another server Failover A failover is similar to a switchover except the ordered shutdown of applications on the original node may not be possible so the services are started on another node 18 Introducing Veritas Cluster Server Can my application be clustered Can my application be clustered Most applications can be placed under cluster control provided the following guidelines are met m Defined start stop and monitor procedures m Ability to restart in a known state m Ability to st
292. ersistent resource In some cases when a system boots and VCS starts VCS probes all resources on the system When VCS probes the NIC resource the resource may not be online because the networking is not up and fully operational In such situations VCS marks the NIC resource as faulted and does not bring the service group online However when the NIC resource becomes online and if AutoRestart is enabled the service group is brought online Controlling failover on service group or system faults The AutoFailOver attribute configures service group behavior in response to service group and system faults m If the AutoFailOver attribute is set to 1 the service group fails over when a system or a service group faults provided a suitable system exists for failover m Ifthe AutoFailOver attribute is set to 0 the service group does not fail over when a system or service group faults If a fault occurs in a service group the group is taken offline depending on whether any of its resources are configured as critical If a system faults the service group is not failed over to another system 366 Controlling VCS behavior Controlling VCS behavior at the service group level Defining failover policies The service group attribute FailOverPolicy governs how VCS calculates the target system for failover There are three possible values for FailOverPolicy Priority RoundRobin Load About system zones The SystemZones attribute ena
293. erver Agent Developer s Guide for more information DNS agent The DNS agent updates the canonical name mapping in the domain name server after a wide area failover See the Veritas Cluster Server Bundled Agents Reference Guide for more information about the agent Connecting clusters Creating global clusters 467 VCS global clusters The building blocks RVG agent The RVG agent manages the Replicated Volume Group RVG Specifically it brings the RVG online monitors read write access to the RVG and takes the RVG offline Use this agent when using VVR for replication RVGPrimary agent The RVGPrimary agent attempts to migrate or take over a Secondary toa Primary following an application failover The agent has no actions associated with the offline and monitor routines RVGSnapshot agent The RVGSnapshot agent used in fire drill service groups takes space optimized snapshots so that applications can be mounted at secondary sites during a fire drill operation Note See the Veritas Cluster Server Agents for Veritas Volume Replicator Configuration Guide for more information about the RVG RVGPrimary and RVGSnapshot agents 468 Connecting clusters Creating global clusters VCS global clusters The building blocks The steward process Split brain in two cluster global clusters Failure of all heartbeats between any two clusters in a global cluster indicates one of the following m The remote cluster is faulted m A
294. erver meets the required Limits and has AvailableCapacity value Group G2 chooses MedServer1 because it is the only remaining system that meets the required Limits Controlling VCS behavior 403 Sample configurations depicting workload management Sample configuration Server consolidation The following configuration has a complex eight node cluster running multiple applications and large databases The database servers LargeServer1 LargeServer2 and LargeServer3 are enterprise systems The middle tier servers running multiple applications are MedServer1 MedServer2 MedServer3 MedServer4 and MedServer5 In this configuration the database zone system zone 0 can handle a maximum of two failures Each server has Limits to support a maximum of three database service groups The application zone has excess capacity built into each server The servers running the application groups specify Limits to support one database even though the application groups do not run prerequisites This allows a database to fail over across system zones and run on the least loaded server in the application zone include types cf cluster SGWM demo system LargeServerl Capacity 200 Limits ShrMemSeg 15 Semaphores 30 Processors 18 LoadWarningLevel 80 LoadTimeThreshold 900 system LargeServer2 Capacity 200 Limits ShrMemSeg 15 Semaphores 30 Processors 18 LoadWarningLevel 80 LoadTimeThreshold 900 system LargeSer
295. ervice group or Click a cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel 2 Click Delete from the menu 3 Click Yes To delete a service group from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Cluster Objects gt Delete Service Group 2 Click the service group 3 Click Apply 182 Administering the cluster from Cluster Manager Java console Administering service groups Bringing a service group online To bring a service group online from the Cluster Explorer configuration tree 1 Inthe Service Groups tab of the configuration tree right click the service group or Click a cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel 2 Click Online and click the appropriate system from the menu Click Any System if you do not need to specify a system To bring a service group online from the Cluster Explorer toolbar 1 Click Online Service Group on the Cluster Explorer toolbar 2 Specify the details for the service group ie Bring service group online Service Groups Select Cluster Local fu global_orp I Any System I No Preonline El Show Command m Click the service group For global groups select the cluster in which to bring the group online m Click the system on which to bri
296. es not display a past date Make sure the Mode attribute displays the correct value If you have purchased a license key for the Global Cluster Option make sure its status is Enabled 4 Start VCS If HAD rejects a license key see the licensing error message at the end of the engine_A log file Licensing error messages This section lists the error messages associated with licensing These messages are logged to the file var VRTSvcs log engine_A log Licensing Insufficient memory to perform operation The system does not have adequate resources to perform licensing operations Licensing No valid VCS license keys were found No valid VCS keys were found on the system Licensing Unable to find a valid base VCS license key No valid base VCS key was found on the system Licensing License key can not be used on this OS platform This message indicates that the license key was meant for a different platform For example a license key meant for Windows is used on a Solaris platform Licensing VCS evaluation period has expired The VCS base demo key has expired Licensing License key can not be used on this system Indicates that you have installed a key that was meant for a different system i e node locked keys Licensing Unable to initialize the licensing framework This is a VCS internal message Call Veritas Technical Support Licensing QuickStart is not supported in this release VCS QuickStart is not supported in this
297. es online after the RemoteGroup resource is online Thus you have established an application level dependency across two different VCS clusters The Apache resource does not go online unless the RemoteGroup goes online The RemoteGroup resource does not go online unless the database service group goes online Configuring applications and resources in VCS 299 Configuring the RemoteGroup agent Unexpected offline of the database service group The RemoteGroup resource detects that the database group has gone OFFLINE or has FAULTED The RemoteGroup resource goes into a FAULTED state All the resources in the Apache service group are taken offline on the node The Apache group fails over to another node As part of the fail over the Oracle service group goes online on another node in cluster2 Taking the Apache service group offline All the resources dependant on the RemoteGroup resource are taken offline The RemoteGroup agent tries to take the Oracle service group offline Once the Oracle service group goes offline the RemoteGroup goes offline Thus the Web server is taken offline before the database goes offline 300 Configuring applications and resources in VCS Configuring application service groups Configuring application service groups Prerequisites Before running the wizard review the resource types and the attribute descriptions of the Application Mount NIC and IP agents in the Veritas Cluster Server Bundled
298. eseceseseseseseesesssseseeeceesenes Setting up a global cluster oo eececesssseseseeeeseseeceseeeseseeeeseesseseneeseeeeseaeeees When a cluster faults oo ccceceseceseceeeeeseeeseeeseeeseseseeeeeeeeeeees Switching the service group back to the primary Setting up a fire drill occ ecscscsssesesesesesesssessseesceesesesesesssseeneeeeeeseseseseaseees Administering global clusters from the Cluster Management Console About creating global Service groups ccceceesesesseseseeeeseeeseseeeeseseeeeseeeeeesees 488 Administering global clusters ccccccccccssesseseseeceseseeceseeeeseseeseseneeseseeeeseseeees 489 Administering global service groups ccccssssesesesseseseeseseseeeeseeeeeeseeeeseseeeees 494 Administering global heartbeats 20 0 0 ccccessesesesseceseeeeseeeeeseeseeeseeeeseseneees 499 Administering global clusters from Cluster Manager Java console About global clusters 2 2c lt iec cxeseeaceetnadindeseieistcesnen sein deadeatee Adding a remote cluster eor neoinean niece E REE Deleting a remote cluster 0 0 cee Administering global service groups Administering global heartbeats n n snssosesesessesessssesesssnssesesesresesssnesesesereesese Administering global clusters from the command line Global querying s s ssseessssssssesessesesesssseseseseesesesersssrsestsstststssssenesesseseseseneessseserstne 522 Administering global service groups s s ssssssessssssesesrsrsresssssreses
299. eseseeesseseeeeseeeeeseeeeseseeeeaeeess 130 VAC WIN SO SS eae a R Ea E hs A A E EE 130 Conducting a search s sssessssssesessseseesestststssrststsssstsesesetsesesetsesnserersrsesesssnenesesne 132 Administering the cluster from Cluster Manager Java console About the Cluster Manager Java Console ccccccecsseseeseseseeseseeeeseeeeeseeeeees 137 Getting Started soere er EE AEE Maal EENE REE E anes 137 Reviewing components of the Java Console ccceesesessseceeesesesesetsesseeeees 140 Icons in the Java Console sricnrneidanen a a E n 140 About Cluster Monitor sisistesccieieacovsssecaiceetarcvecnnaienndieeie EEE 142 About Cluster Explorer rinena a E NAERA 148 Accessing additional features of the Java Console ccccesessseeseeseseeees 160 Administering Cluster Monitor se ssssssesesnesesoseseesesestsssrststsnsststssssesesesereesese 169 Administering user profiles ccccccscsssesesesesessssesesesececeseseseseseseeseeeeecseseeess 174 Administering service groups s sesssssessssessssssesesssrststesesestnsstsesesseseseseretsest 178 Administering reSOULCeS inenen enn ia 199 Importing resource CYPES oe nnn E E E EREE ERRE 217 Administering SYStOMS 0 0 cccccsseseseesesesseseseseesesesceseseseeseeesseseeeeseseeeeseeeeaeeees 218 Administering CIUStCrS cccccecceceseseeseseeseseseeeeseseeeeseeeseseeeeseeeseeseeeeseseeeeaeeees 220 Executing COMMAMAS cceceessesssesssceseseeseseeeseseseeseseeceseseecese
300. esource Summary view After navigating to an object summary view click the Attributes tab on the secondary tab bar Changes to certain attributes such as a webip attribute may involve taking the service group offline modifying the configuration file and bringing the group back online To edit an attribute 1 Navigate to an Object Summary view for the object with the attributes you want to edit See Overview of the Cluster Management Console on page 76 for information about navigating among views in the view hierarchy 2 Inthe Object Summary view on the secondary tab bar click Attributes 3 Inthe All attributes table click the edit button for the attribute you want to edit 4 Inthe Edit Attribute dialog box specify the following attribute details and then click OK m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value Some Edit Attribute dialog boxes have just a text field for the value You can enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values m The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables y
301. esponse does not indicate the cluster is down the cluster transitions the remote cluster state to UNKNOWN RUNNING gt LOST_HB gt INQUIRY gt UNKNOWN m When the ClusterService service group which maintains the connector process as highly available fails over to another system in the cluster the remote clusters transition their view of that cluster to TRANSITIONING then back to RUNNING after the failover is successful RUNNING gt TRANSITIONING gt BUILD gt RUNNING m When a remote cluster in a RUNNING state is stopped by taking the ClusterService service group offline the remote cluster transitions to EXITED RUNNING gt EXITING gt EXITED System states Whenever the VCS engine is running on a system it is in one of the states described in the table below States indicate a system s current mode of operation When the engine is started on a new system it identifies the other systems available in the cluster and their states of operation If a cluster system is in the state of RUNNING the new system retrieves the configuration information from that system Changes made to the configuration while it is being retrieved are applied to the new system before it enters the RUNNING state If no other systems are up and in the state of RUNNING or ADMIN_WAIT and the new system has a configuration that is not invalid the engine transitions to the state LOCAL_BUILD and builds the configuration from disk If the configura
302. essesesesereesese 528 Administering reSOULCeS o ceccccecescsseseseeseseeseseseecesesesseseseeseseeceseeeeseseneeaeeeeaees 530 Administering CIUStCrS ccecccccsessssesesseseseeceseseeceseeeeseseseeseseeeeseeeeseseeeeseeesees 531 Administering heartbeats ccccccccescscesesesseseseeceseseeceseeeeseseeeeseeeseeseeeeseseeees 533 9 10 Contents Chapter 19 Section VI Chapter 20 Chapter 21 Section VII Appendix A Appendix B Setting up replicated data clusters About replicated data clusters oo cee cccccccescsseseescesessessescescsscsecsecseeeesecseeaeees How VCS replicated data clusters work seek Setting up a replicated data cluster configuration Migrating a service group senasa on r e E ERRE E R Setting up a fire drill oc ccescsssssssssesesesecssseseeecsesesesessesseneeecseseseseseeeeeees Troubleshooting and performance VCS performance considerations How cluster components affect performance 0 ccccccesceseesesesetseseneeeees 546 How cluster operations affect performance cccccecssssesesseseseteeecseneesees 550 Scheduling class and priority configuration cccccsseseseseseeeseseteteeeeeees 556 CPU binding Of HAD aienea AN E EE AEEA Ee 558 Monitoring CPU Usd E rae eea are aer oae AET EEE TEE aT ATES 559 VCS agent Statistics sia E E AEREE R eess 560 Troubleshooting and recovery for VCS TOS BU 8 matee e A A N OEA 564 Troubleshooting the VCS engine s s s
303. etc in the stack trace The presence of DNS resolver functions may indicate DNS configuration issues The VCS High Availability Daemon HAD uses the gethostbyname function On UNIX platforms if the file etc nsswitch conf has DNS in the hosts entry a call to the gethostbyname function may lead to calls to DNS resolver methods If the name servers specified in the etc resolve conf are not reachable or if there are any DNS configuration issues the DNS resolver methods called may block HAD leading to HAD not sending heartbeats to GAB in a timely manner Seeding and I O Fencing When I O fencing starts up a check is done to make sure the systems that have keys on the coordinator disks are also in the GAB membership If the gabconfig command in etc gabtab allows the cluster to seed with less than the full number of systems in the cluster or the cluster is forced to seed with the gabconfig c x command it is likely that this check will not match In this case the fencing module will detect a possible split brain condition print an error and HAD will not start Troubleshooting and recovery for VCS 567 Troubleshooting VCS startup It is recommended to let the cluster automatically seed when all members of the cluster can exchange heartbeat signals to each other In this case all systems perform the coordinator disk key placement after they are already in the GAB membership Preonline IP check You can enable a preonline check
304. ete keys sys system Note If multiple values are specified and if one is invalid VCS returns an error for the invalid value but continues to process the others In the following example if sysb is part of the attribute SystemList but sysa is not sysb is deleted and an error message is sent to the log regarding sysa hagrp modify groupl SystemList delete sysa sysb sys system Administering the cluster from the command line Administering resources Linking and unlinking resources Link resources to specify a dependency between them A resource can have an unlimited number of parents and children When linking resources the parent cannot be a resource whose Operations attribute is equal to None or OnOnly Specifically these are resources that cannot be brought online or taken offline by an agent None or can only be brought online by an agent OnOnly Loop cycles are automatically prohibited by the VCS engine You cannot specify a resource link between resources of different service groups To link resources Type the following command hares link parent resource child resource The variable parent_resource depends on child_resource being online before going online itself Conversely parent_resource go offline before child_resource goes offline For example a NIC resource must be available before an IP resource can go online so for resources IP1 of type IP and NIC1 of type NIC specify the dependency as
305. fail over Usage This event trigger is non configurable nofailover triggertype system service_group triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 system represents the name of the last system on which an attempt was made to online the service group service_group represents the name of the service group postoffline event trigger VCS event triggers 451 List of event triggers Description This event trigger is invoked on the system where the group went offline from a partial or fully online state This trigger is invoked when the group faults or is taken offline manually Usage This event trigger is non configurable postoffline triggertype system service_group postonline event trigger triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 system represents the name of the system service_group represents the name of the service group that went offline Description This event trigger is invoked on the system where the group went online from an offline state Usage This event trigger is non configurable postonline triggertype system service_group triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 system represents the name of the system service_group r
306. fault behavior Disable Reject all hastop commands DisableClusStop Do not process the hastop all command process all other hastop commands PromptClusStop Prompt for user confirmation before running the hastop all command process all other hastop commands PromptLocal Prompt for user confirmation before running the hastop local command reject all other hastop commands PromptAlways Prompt for user confirmation before running any hastop command m Type and dimension string scalar m Default Enable Table D 5 VCS attributes Cluster attributes 651 Cluster attributes GlobalCounter system use only This counter increases incrementally by one for each counter interval It increases when the broadcast is received VCS uses the GlobalCounter attribute to measure the time it takes to shut down a system By default the GlobalCounter attribute is updated every five seconds This default value combined with the 120 second default value of ShutdownTimeout means if system goes down within twelve increments of GlobalCounter it is treated as a fault The default value of GlobalCounter increment can be modified by changing the CounterInterval attribute m Type and dimension integer scalar m Default Not applicable Guests user defined List of operating system user account groups that have Guest privileges on the cluster This attribute is valid only in secure clusters m Type and dimens
307. faulted Note that when a resource is faulted resources within the upward path of the faulted resource are also brought down This event trigger is configurable To configure this trigger you must define the following TriggerResFault Set the attribute to 1 to invoke the trigger when a resource faults resfault triggertype system resource previous_state m triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 m system represents the name of the system m resource represents the name of the faulted resource m previous_state represents the resource s previous state To invoke the trigger when a resource faults set the TriggerResFault attribute to 1 Invoked on the system if a resource in a service group does not go offline even after issuing the offline command to the resource This event trigger is configurable To configure this trigger you must define the following Resource Name Define resources for which to invoke this trigger by entering their names in the following line in the script resources resourcel resource2 If any of these resources do not go offline the trigger is invoked with that resource name and system name as arguments to the script VCS event triggers 455 List of event triggers Usage resnotoff triggertype system resource triggertype represents whether trigger is custom triggertype 0 or int
308. for all major graphical user interface GUI operations and menu items Symantec products are compatible with operating system accessibility settings as well as a variety of assistive technologies All manuals also are provided as accessible PDF files and the online help is provided as HTML displayed in a compliant viewer Navigation and keyboard shortcuts VCS uses standard operating system navigation keys and keyboard shortcuts For its unique functions VCS uses its own navigation keys and keyboard shortcuts which are documented below Navigation in the Java Console The following table lists keyboard navigation rules and shortcuts used in Cluster Manager Java Console in addition to those provided by the operating system Shift F10 Opens a context sensitive pop up menu Spacebar Selects an item Ctrl Tab Navigates outside a table F2 Enables editing a cell Navigation in the Web Console Cluster Management Console supports standard browser based navigation and shortcut keys for supported browsers All Symantec GUIs use the following keyboard navigation standards m Tab moves the focus to the next active area field or control following a preset sequence Shift Tab moves the focus in the reverse direction through the sequence m Ctrl Tab exits any Console area that you internally navigate with Tab Accessibility and VCS 681 Support for accessibility settings m Up and Down arrow keys move focus up and
309. from agents and update messages from the global counter VCS may exchange additional messages when an event occurs but typically overhead is nominal even during events Note that this depends on the type of event for example a resource fault may involve taking the group offline on one system and bringing it online on another system A system fault invokes failing over all online service groups on the faulted system To continuously monitor VCS status use the VCS graphical user interfaces or the command hastatus Both methods maintain connection to VCS and register for events and are more efficient compared to running commands like hastatus summary or hasys ina loop The number of clients connected to VCS can affect performance if several events occur simultaneously For example if five GUI processes are connected to VCS VCS sends state updates to all five Maintaining fewer client connections to VCS reduces this overhead The impact of agents The VCS agent processes have the most impact on system performance Each agent process has two components the agent framework and the agent entry points The agent framework provides common functionality such as communication with the HAD multithreading for multiple resources scheduling threads and invoking entry points Agent entry points implement agent specific functionality Follow the performance guidelines below when configuring agents Monitoring resource type and agent configuration
310. g See Adding deleting and modifying resource attributes on page 269 VCS configuration concepts 55 About VCS keywords and reserved words About VCS keywords and reserved words The following list includes the current keywords reserved for the VCS configuration language Note they are case sensitive action false after firm ArgListValues global before group boolean Group cluster hard Cluster heartbeat condition int ConfidenceLevel State event keylist local offline online MonitorOnly Name NameRule Path Probed remote remotecluster requires resource set Signaled soft start Start state State static stop str system System temp type Type 56 VCS configuration concepts VCS environment variables VCS environment variables Table 3 1 lists VCS environment variables Table 3 1 VCS environment variables VCS_CONF Root directory for VCS configuration files Default etc VRTSvcs Note If this variable is added or modified you must reboot the system to apply the changes VCS_DOMAIN The Security domain in which users are configured VCS_DOMAINTYPE Type of domain unixpwd nt nis nisplus or vx VCS_DIAG Directory where VCS dumps HAD cores VCS_ENABLE_LDF Designates whether or not log data files LDFs are generated If set to 1 LDFs are generated If set to 0 they are not VCS_HOME Root directory for VCS executables Default opt VRTSvcs
311. g all types is displayed Querying agents To display the run time status of an agent Type the following command haagent display agent If agent is not specified information regarding all agents is displayed Run Time Status Definition Faults Indicates the number of agent faults and the time the faults began Messages Displays various messages regarding agent status Running Indicates the agent is operating Started Indicates the file is executed by the VCS engine HAD Administering the cluster from the command line 257 Querying VCS Querying systems To display a list of systems in the cluster Type the following command hasys list To display information about each system Type the following command hasys display system Querying clusters To display the value of a specific cluster attribute Type the following command haclus value attribute To display information about the cluster Type the following command haclus display Querying status To display the status of all service groups in the cluster including resources Type the following command hastatus To display the status of a particular service group including its resources Type the following command hastatus sound group service group group service group If you do not specify a service group the status of all service groups is displayed The sound option enables a bell to ring each
312. g information m RVGresource depends on the IP resource m RVG resource depends on the DiskGroup resource m IP resource depends on the NIC resource Set the SystemZones attribute of the child group oragrp_rep such that all nodes in the primary RDC zone are in system zone 0 and all nodes in the secondary RDC zone are in system zone 1 To configure the application service group 1 In the original Oracle service group oragroup delete the DiskGroup resource Add an RVGPrimary resource and configure its attributes Set the value of the RvgResourceName attribute to the name of the RVG type resource that will be promoted and demoted by the RVGPrimary agent Set the AutoTakeover and AutoResync attributes from their defaults as desired See RVGPrimary agent on page 467 Set resource dependencies such that all Mount resources depend on the RVGPrimary resource If there are a lot of Mount resources you can set the TypeDependencies attribute for the group to denote that the Mount resource type depends on the RVGPRimary resource type Set the SystemZones attribute of the Oracle service group such that all nodes in the primary RDC zone are in system zone 0 and all nodes in the secondary RDC zone are in zone 1 The SystemZones attribute of both the parent and the child group must be identical If your setup uses BIND DNS add a resource of type DNS to the oragroup service group Set the Hostname attribute to the canonical name of the
313. g service groups online 183 clearing resource faults 211 closing configuration files 221 deleting resources 204 deleting service groups 181 deleting systems 218 disabling resources 210 disabling service groups 189 editing attributes 222 enabling resources 209 enabling service groups 188 executing commands 221 flushing service groups 191 freezing service groups 186 freezing systems 219 ignoreparent option 206 linking resources 212 linking service groups 193 opening configuration files 220 probing resources 207 saving configuration files 220 switching service groups 185 taking resources offline 205 taking resources offline and propagating 206 taking service groups offline 185 unfreezing service groups 187 unfreezing systems 219 unlinking resources 215 unlinking service groups 194 commands scripting 284 CompareRSM attribute 649 ComputeStats attribute 615 conditional statements 260 690 Index ConfidenceLevel attribute 615 ConfigBlockCount attribute 641 ConfigCheckSum attribute 641 ConfigDiskState attribute 641 ConfigFile attribute 641 ConfigInfoCnt attribute 642 ConfigModDate attribute 642 configuration closing from Cluster Management Console 96 closing from Java Console 221 dumping 243 opening from Cluster Management Console 96 opening from Java Console 220 saving 243 saving from Cluster Management Console 96 saving from Java Console 220 saving in VCS Simulator 325 setting to read write 243 setting to read only 243 taki
314. ged application as a single unit When a failover occurs resources do not fail over individually the entire service group fails over If there is more than one service group on a system a group may fail over without affecting the others Figure 1 2 Typical database service group Application IP Address A single node may host any number of service groups each providing a discrete service to networked clients If the server crashes all service groups on that node must be failed over elsewhere Service groups can be dependent on each other For example a finance application may be dependent on a database application Because the managed application consists of all components that are required to provide the service service group dependencies create more complex managed applications When using service group dependencies the managed application is the entire dependency tree See About service group dependencies on page 410 24 Introducing Veritas Cluster Server Logical components of VCS Types of service groups VCS service groups fall in three main categories failover parallel and hybrid Failover service groups A failover service group runs on one system in the cluster at a time Failover groups are used for most applications that do not support multiple systems to simultaneously access the application s data Parallel service groups A parallel service group runs simultaneously on more than one sys
315. ggerResFault attribute 639 TriggerResStateChange attribute 639 triggers See event triggers troubleshooting back up and restore files 574 license keys 578 logging 564 notification 573 resources 571 service groups 568 VCS startup 567 TRSE attribute 647 TypeDependencies attribute 639 TypeLimit attribute 653 types cf 46 U umask setting for VCS files 57 unable_to_restart_had trigger 457 458 UpDownState attribute 647 UseFence attribute 653 user credentials viewing 167 user privileges about 62 assigning from command line 252 changing from Java Console 176 Cluster Administrator 62 Cluster Guest 63 Cluster Operator 63 for specific commands 584 Group Administrator 63 Group Operator 63 in global clusters 64 removing from command line 252 253 UserInt attribute 647 UserIntGlobal attribute 640 UserIntLocal attribute 640 UserNames attribute 653 users adding from Cluster Management Console 94 adding from Java Console 174 deleting from command line 253 deleting from Java Console 175 displaying from command line 253 UserStrGlobal attribute 640 UserStrLocal attribute 640 utilities hacf 242 hanotify 434 vxlicinst 235 VCS accessibility 679 additional considerations for stopping 239 assistive technology support 681 event triggers 446 logging 564 logging off of 240 logging on to 240 notification 430 querying from command line 254 SNMP and SMTP 430 starting as time sharing process 236 Index 699 VCS continued sta
316. greater than the Notify time limit See When a resource comes online on page 550 438 VCS event notification VCS events and traps Events and traps for service groups Event Service group has faulted Service group concurrency violation Service group has faulted and cannot be failed over anywhere Service group is online Service group is offline Service group is autodisabled Service group is restarting Service group is being switched Service group restarting in response to persistent resource going online The global service group is online partial on multiple clusters Global Cluster Option Attributes for global service groups are mismatched Global Cluster Option Severity Level Description Error SevereError SevereError Information Information Information Information Information Information SevereError Error Self explanatory A failover service group has become online on more than one node in the cluster Specified service group faulted on all nodes where group could be brought online There are no nodes to which the group can fail over Self explanatory Self explanatory VCS has autodisabled the specified group because one node exited the cluster Self explanatory VCS is taking the service group offline on one node and bringing it online on another Self explanatory Aconcurrency violation occurred for the global service group
317. group The yellow line snaps to the child group If necessary press Esc on the keyboard to delete the line between the parent and the pointer before it snaps to the child Click the child group In the Link Service Groups dialog box click the group relationship and dependency type See About service group dependencies on page 410 ES Link Service Groups Parent Service Group CMC Child Service Group newgroup2 Relationship online local offline local C online remote online global Dependency Type firm C soft hard Cancel Click OK Administering the cluster from Cluster Manager Java console 193 Administering service groups You can also link the service groups by performing steps 1 and 2 right clicking the parent group and clicking Link from the menu In the dialog box click the child group relationship dependency type and click OK T Link Service Groups x er Parent Group CMC Child Group SCM_SG1 SCM_SG2 BP newgroup Relationship online local offline local online remote online global Dependency Type firm C soft C hard cn To link a service group from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Dependencies gt Link Service Groups 2 Click the parent resource group in the Service Groups box After selecting the parent group the potential groups that can serve as chi
318. group in the application zone will try to fail over to another node within the zone If no nodes are available in the application zone the group will fail over to the database zone based on the configured load and limits In this configuration excess capacity and limits on the database backend are kept in reserve to handle the larger load of a database failover The application servers handle the load of service groups in the application zone During a cascading failure the excess capacity in the cluster is available to all service groups Controlling VCS behavior 367 Controlling VCS behavior at the service group level About load based autostart VCS provides a method to determine where a service group comes online when the cluster starts Setting the AutoStartPolicy to Load instructs the VCS engine HAD to determine the best system on which to start the groups VCS places service groups in an AutoStart queue for load based startup as soon as the groups probe all running systems VCS creates a subset of systems that meet all prerequisites and then chooses the system with the highest AvailableCapacity Set AutoStartPolicy Load and configure the SystemZones attribute to establish a list of preferred systems on which to initially run a group Freezing service groups Freezing a service group prevents VCS from taking any action when the service group or a system faults Freezing a service group prevents dependent resources from g
319. group offline from the Cluster Explorer toolbar 1 Click Offline Service Group in the Cluster Explorer toolbar 2 Enter the details of the service group ee Take service group offline Service Groups Select Cluster ClusterService Local ud Tall Systems El Show Command m Click the service group m For global groups select the cluster in which to take the group offline Click the system on which to take the group offline or click All Systems m Click Show Command in the bottom left corner if you want to view the command associated with the service group Click Hide Command to close the view of the command m Click OK Administering the cluster from Cluster Manager Java console Administering service groups To take a service group offline from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Controls gt Offline Service Group or Click Take service group offline in the Command Center toolbar Click the service group For global groups select the cluster in which to take the group offline Click the system on which to take the group offline or click the All Systems check box Click Apply Switching a service group The process of switching a service group involves taking it offline on its current system and bringing it online on another system To switch a service group from Cluster Explorer 1 In the Service Groups
320. grp modify service group TriggerResStateChange 1 Note Use the resstatechange trigger carefully For example enabling this trigger for a service group with 100 resources means 100 hatrigger processes and 100 resstatechange processes are fired each time the group is brought online or taken offline Also this is not a wait mode trigger Specifically VCS invokes the trigger and does not wait for trigger to return to continue operation sysoffline event trigger VCS event triggers 457 List of event triggers Description Called from the lowest numbered system in RUNNING state when a system Usage leaves the cluster This event trigger is non configurable sysoffline system system_state m system represents the name of the system m system_state represents the value of the State attribute See System states on page 608 unable_to_restart_agent event trigger Description This trigger is invoked when an agent faults more than a predetermined Usage To disable number of times with in an hour When this occurs VCS gives up trying to restart the agent VCS invokes this trigger on the node where the agent faults You can use this trigger to notify the administrators that an agent has faulted and that VCS is unable to restart the agent The administrator can then take corrective action unable_to_restart_agent system resource_type m system represents the name of the system m resource_type represe
321. guration tree expand Commands gt Operations gt Availability gt Freeze Service Group Click the service group Select the persistent check box if necessary The persistent option maintains the frozen state after a reboot if you save this change to the configuration Click Apply Administering the cluster from Cluster Manager Java console 187 Administering service groups Unfreezing a service group Unfreeze a frozen service group to perform online or offline operations on the service group To unfreeze a service group from Cluster Explorer 1 Inthe Service Groups tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel 2 Click Unfreeze To unfreeze a service group from Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Availability gt Unfreeze Service Group 2 Click the service group 3 Click Apply 188 Administering the cluster from Cluster Manager Java console Administering service groups Enabling a service group Enable a service group before bringing it online A service group that was manually disabled during a maintenance procedure on a system may need to be brought online after the procedure is completed To enable a service group from Cluster Explorer 1 In the Service Groups tab of the configuration tr
322. gured in a failover group not parallel because only one instance of notifier runs in the entire cluster Also note that notifier does not respond to SNMP get or set requests notifier is a trap generator only Notifier enables you to specify configurations for the SNMP manager and SMTP server including machine names ports community IDs and recipients email addresses You can specify more than one manager or server and the severity level of messages that are sent to each Example of notifier command notifier s m north s m south p 2000 1 Error c your company t m north e abc your company com 1 SevereError In this example notifier m Sends all level SNMP traps to north at the default SNMP port and community value public m Sends Warning traps to north Sends Error and SevereError traps to south at port 2000 and community value your_company m Sends SevereError email messages to north as SMTP server at default port and to email recipient abc your_company com 434 VCS event notification Components of VCS event notification The hanotify utility The hanotify utility enables you to construct user defined messages The utility forwards messages to HAD which stores them in its internal message queue Along with other messages user defined messages are also forwarded to the notifier process for delivery to email recipients SNMP consoles or both System A System B Example of hanotify command
323. h the following commands m haclus haclus wait attribute value clus cluster time timeout Use the clus option in a global cluster environment m hagrp hagrp wait group attribute value clus cluster sys system time timeout Use the sys option when the scope of the attribute is local Use the clus option in a global cluster environment m hares hares wait resource attribute value clus cluster sys system time timeout Use the sys option when the scope of the attribute is local Use the clus option in a global cluster environment m hasys hasys wait system attribute value clus cluster time timeout Use the clus option in a global cluster environment See the man pages associated with these commands for more information Running a virtual fire drill The service group must be online when you run the virtual fire drill See Testing resource failover using virtual fire drills on page 315 To run a virtual fire drill for a specific resource Type the following command hares action lt resname gt lt vfdaction gt vfd sys lt sysname gt The command runs the infrastructure check verifies whether the system lt sysname gt has the required infrastructure to host the resource lt resname gt should a failover require the resource to come online on the system For the variable lt sysname gt specify the name of a system on which the resource is offline The variable lt vfdaction g
324. h to take the service group offline or click Anywhere 4 Click OK to confirm that you want to take the service group offline To take one or more service groups offline anywhere 1 Inthe Cluster Groups view in the Clusters Listing table check the check box preceding the line item for each online service group that you want to take offline Select all service groups by checking the check box at the top 2 Inthe Operations task panel click Offline Anywhere 3 Inthe Offline Service Group s dialog box click OK to confirm that you want to take the selected service groups offline anywhere on any system in the cluster 106 Administering the cluster from the Cluster Management Console Administering service groups Switching a service group Manually switch a service group to another system in the cluster To switch a service group 1 In the Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to switch In the Group Summary view in the Operations task panel click Switch In the Switch Service Group dialog box select the destination system for the service group from the drop down menu Select Anywhere to switch the service group to any other system in the cluster The system choices in this step are populated using the system list for the service group The Anywhere option causes this task to try every other system in the list until the service group is successfully bro
325. hagrp display service_group to verify the value of the State attribute Use the command hagrp offline to offline the group on another system A critical resource faulted Output of the command hagrp display service_group indicates that the service group has faulted Recommended Action Use the command hares clear to clear the fault Service group autodisabled When VCS does not know the status of a service group on a particular system it autodisables the service group on that system Autodisabling occurs under the following conditions m When the VCS engine HAD is not running on the system m When all resources within the service group are not probed on the system Under these conditions all service groups that include the system in their SystemList attribute are autodisabled This does not apply to systems that are powered off 570 Troubleshooting and recovery for VCS Troubleshooting service groups Recommended Action Use the output of the command hagrp display service_group to verify the value of the AutoDisabled attribute Caution To bring a group online manually after VCS has autodisabled the group make sure that the group is not fully or partially active on any system that has the AutoDisabled attribute set to 1 by VCS Specifically verify that all resources that may be corrupted by being active on multiple systems are brought down on the designated systems Then clear the AutoDisabled attribute for each system
326. hall be considered valid See Authentication Broker Seeding Seeding is used to protect a cluster from a pre existing network partition By default when a system comes up it is not seeded Systems can be seeded automatically or manually Only systems that have been seeded can run VCS Systems are seeded automatically only when an unseeded system communicates with a seeded system or all systems in the cluster are unseeded and able to communicate with each other See Network Partition Service Group A service group is a collection of resources working together to provide application services to clients It typically includes multiple resources hardware and software based working together to provide a single service Service Group Dependency A mechanism by which two service groups can be linked by a dependency rule 686 Glossary Shared Storage Storage devices that are connected to and used by two or more systems SNMP Notification Simple Network Management Protocol SNMP developed to manage nodes on an IP network State The current activity status of a resource group or system types cf The types cf file describes standard resource types to the VCS engine specifically the data required to control a specific resource Virtual IP Address A unique IP address that associated with the cluster It may be brought up on any system in the cluster along with the other resources of the service group This address
327. hanotify i 1 3 6 1 4 1 1302 3 8 10 2 8 0 10 1 Warning n agentres T 7 t custom agent o 4 S sysl1 L mv p sys2 P mv c MyAgent C 7 O johndoe m Custom message In this example the number 1 3 6 1 4 1 1302 3 8 10 2 8 0 10 is the OID for the message being sent Because it is a user defined message VCS has no way of knowing the OID associated with the SNMP trap corresponding to this message Users must provide the OID The message severity level is set to Warning The affected systems are sys1 and sys2 Running this command sends a custom message for the resource agentres from the agent MyAgent VCS events and traps VCS event notification 435 VCS events and traps This section lists the events generate traps email notification or both Note that SevereError indicates the highest severity level Information the lowest Traps specific to global clusters are ranked from Critical the highest severity to Normal the lowest Events and traps for clusters Event Cluster has faulted Heartbeat is down Global Cluster Option Remote cluster is in RUNNING state Global Cluster Option Heartbeat is alive Global Cluster Option User has logged on to VCS Events and traps for agents Event Agent is faulted Agent is restarting Severity Level Description Error Error Information Information Information Self explanatory The connector on the local cluster lost its heartbeat con
328. hasys value system attribute clus cluster localclus The option clus displays the values of a system attribute in the cluster as designated by the variable cluster the option localclus specifies the local cluster To display the state of a system across clusters hasys state system clus cluster localclus Displays the current state of the specified system The option clus displays the state in a cluster designated by the variable cluster the option localclus specifies the local cluster If you do not specify a system the command displays the states of all systems For information about each system across clusters hasys display systems attribute attributes clus cluster localclus The option clus displays the attribute values on systems if specified in a cluster designated by the variable cluster the option localclus specifies the local cluster For a list of systems across clusters hasys list conditionals clus cluster localclus Displays a list of systems whose values match the given conditional statements The option clus displays the systems in a cluster designated by the variable cluster the option localclus specifies the local cluster Administering global clusters from the command line 525 Global querying Querying clusters For the value of a specific cluster attribute on a specific cluster haclus value attribute cluster localclus The attribute must be specified in t
329. hat it affects the data on shared disks See Jeopardy See Seeding Node The physical host or system on which applications and service groups reside When systems are linked by VCS they becomes nodes in a cluster N to 1 An N to 1 configuration is based on the concept that multiple simultaneous server failures are unlikely therefore a single backup server can protect multiple active servers When a server fails its applications move to the backup server For example in a 4 to 1 configuration one server can protect four servers which reduces redundancy cost at the server level from 100 percent to 25 percent N to N N to N refers to multiple service groups running on multiple servers with each service group capable of being failed over to different servers in the cluster For example consider a four node cluster with each node supporting three critical database instances If any node fails each instance is started on a different node ensuring no single node becomes overloaded N to M Glossary 685 N to M or Any to Any refers to multiple service groups running on multiple servers with each service group capable of being failed over to different servers in the same cluster and also to different servers in a linked cluster For example consider a four node cluster with each node supporting three critical database instances and a linked two node back up cluster If all nodes in the four node cluster fail each in
330. havior on loss of storage connectivity How VCS attributes control behavior on loss of storage connectivity If you use I O fencing you can configure VCS attributes to ensure that a node panics on losing connectivity to shared storage The panic causes service groups to fail over to another node A system reboot or shutdown could leave the system in a hung state because the operating system cannot dump the buffer cache to the disk The panic operation ensures that VCS does not wait for I Os to complete before triggering the failover mechanism thereby ensuring application availability However you might have to perform a file system check when you restart the node The following attributes define VCS behavior on loss of storage connectivity PanicSystemOnDGLoss FaultOnMonitorTimeouts Applies to the DiskGroup resource and defines whether the agent panics the system when storage connectivity is lost The system panics only if the cluster uses I O fencing that is if you set the UseFence attribute to SCSI3 The number of consecutive monitor timeouts after which VCS calls the Clean entry point to mark the resource as FAULTED or restarts the resource If you set the attribute to 0 VCS does not treat a Monitor timeout as a resource fault By default the attribute is set to 4 This means that the Monitor entry point must time out four times in a row before VCS marks the resource faulted When the Monitor entry point for the Dis
331. he types cf file for the operating system from the types directory to opt VRTSsim default_clus conf config For example if the cluster to be simulated runs on the AIX platform copy the file types cf aix Add custom type definitions to the file if required and rename the file to types cf If you have a main cf file to run in the simulated cluster copy it to opt VRTSsim default_clus conf config Start VCS Simulator sim _dir hasim start system name The variable system_name represents a system name as defined in the configuration file main cf This command starts Simulator on port 14153 For example to start the default cluster sim _dir hasim start sysl Note that the default configuration includes system sys1 Add systems to the configuration if desired sim _dir hasim sys add system name sim _dir hasim up system name Verify the states of each node in the cluster sim _dir hasim sys state Use the command line or the Java Console to manage the simulated cluster See To simulate global clusters from the command line on page 327 Predicting VCS behavior using VCS Simulator 327 Administering VCS Simulator from the command line To start VCS Simulator from the command line Windows VCS Simulator installs platform specific types cf files at the path VCS_SIMULATOR_HOME types The variable VCS_SIMULATOR_HOME represents the Simulator installation directory typically C Program Files Veri
332. he csgnic gcoip and wac resources must be able to come online The service group intended to serve as the global group has the same unique name on all clusters that participate in the global environment The clusters all have different unique names The clusters must use the same version of VCS The clusters must use the same operating system The clusters are standalone and do not already belong to a global cluster environment All clusters must either be configured in secure mode or not configured in secure mode Mixing of modes is not supported For remote cluster operations you must configure a VCS user with the same name and privileges in each cluster See User privileges in global clusters on page 64 Use the Cluster summary Cluster Resources or Group Summary views to administer a global cluster To navigate to the Cluster Summary view 1 2 3 On the main tab bar click Manage On the secondary tab bar click Summary In the Cluster Summary view choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster To navigate to the Cluster Resources view 1 2 3 On the main tab bar click Manage On the secondary tab bar click Resources In the Cluster Resources view choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view c
333. he details to connect to the cluster Host name esio Port haa o Failover retries hze o Cancel Enter the host name or IP address of any system in the cluster m Enter the port number and the number of failover retries VCS sets the default port number to 14141 and failover retries number to 12 VCS Simulator uses a default port number of 14153 For simulated panels click the platform for the configuration m Click OK Administering the cluster from Cluster Manager Java console 171 Administering Cluster Monitor Logging on to and off of a cluster After you add or configure a cluster panel in Cluster Monitor log on to a cluster to access Cluster Explorer Use Cluster Monitor to log off a cluster when you have completed administering the cluster Logging on to a cluster 1 If Cluster Monitor is in the default expanded state proceed to step 2 If Cluster Monitor is in the collapsed state On the View menu click Expand or On the View menu click Stop when an active panel appears as the view panel Click the panel that represents the cluster you want to log on to or If the appropriate panel is highlighted click Login on the File menu Enter the information for the user If the cluster is not running in secure mode m Enter the VCS user name and password m Click OK If the cluster is running in secure mode E veslinux145 14141 xi User name root Password mes Domain i Cancel
334. he latest values for that attribute To refresh the Resourcelnfo attribute 1 In the Cluster Resources view in the Resources Listing table click the linked name of a resource with the ResourcelInfo attribute 124 Administering the cluster from the Cluster Management Console Administering resources In the Resource Summary view in the Operations task panel click Refresh Resource Info In the Refresh Resource Info dialog box select the system on which you want to refresh the ResourcelInfo attribute Click OK Clearing the Resourcelnfo attribute Clear the Resourcelnfo attribute to reset all the parameters in this attribute To clear the parameters of the Resourcelnfo attribute 1 In the Cluster Resources view in the Resources Listing table click the linked name of a resource with the ResourcelInfo attribute In the Resource Summary view in the Operations task panel click Clear Resource Info In the Clear Resource Info dialog box select the system on which you want to clear the ResourceInfo attribute Administering the cluster from the Cluster Management Console 125 Administering resource types Administering resource types The Cluster Management Console enables you to add and configure resource types according to the requirements of the resources that they represent Use the following views to administer a resource type Cluster Resource Types This is a cluster level view of all resource types in the cl
335. he monitor entry point until the thread becomes free Additional considerations for modifying the NumThreads attribute include m Ifyou have only one or two resources of a given type you can set NumThreads to a lower value m If you have many resources of a given type evaluate the time it takes for the monitor entry point to execute and the available CPU power for monitoring For example if you have 50 mount points you may want to increase NumThreads to get the ideal performance for the Mount agent without affecting overall system performance You can also adjust how often VCS monitors various entry points by modifying their associated attributes The attributes MonitorTimeout OnlineTimeOut and OfflineTimeout indicate the maximum time in seconds within which the monitor online and offline entry points must complete or else be terminated The default for the MonitorTimeout attribute is 60 seconds The defaults for the OnlineTimeout and OfflineTimeout attributes is 300 seconds For best results Symantec recommends measuring the time it takes to bring a resource online take it offline and monitor before modifying the defaults Issue an online or offline command to measure the time it takes for each action To measure how long it takes to monitor a resource fault the resource and issue a probe or bring the resource online outside of VCS control and issue a probe Agents typically run with normal priority When you develop agents cons
336. he person designated in this attribute when an event occurs related to the system VCS also logs the owner name when an event occurs m Type and dimension string scalar m Default unknown TFrozen user defined Indicates if a group can be brought online or taken offline m Type and dimension boolean scalar m Default 0 TRSE system use only Indicates in seconds the time to Regular State Exit Time is calculated as the duration between the events of VCS losing port h membership and of VCS losing port a membership of GAB m Type and dimension integer scalar m Default Not applicable UpDownState system use only This attribute has four values 0 down System is powered off or GAB and LLT are not running on the system 1 Up but not in cluster membership GAB and LLT are running but the VCS engine is not 2 up and in jeopardy The system is up and part of cluster membership but only one network link LLT remains 3 up The system is up and part of cluster membership and has at least two links to the cluster m Type and dimension integer scalar m Default Not applicable UserInt user defined Stores a system s integer value m Type and dimension integer scalar m Default 0 VCSFeatures system use only Indicates which VCS features are enabled Possible values are 0 No features enabled VCS Simulator 1 L3 is enabled 2 Global Cluster Option is en
337. he process of generating and clearing global cluster faults in an OFFLINE state Use VCS Simulator to complete these operations Predicting VCS behavior using VCS Simulator on page 317 For remote cluster operations you must configure a VCS user with the same name and privileges in each cluster See User privileges in global clusters on page 64 Note Cluster Manager Java Console provides disabled individuals access to and use of information and data that is comparable to the access and use provided to non disabled individuals See Accessibility and VCS on page 679 Administering global clusters from Cluster Manager Java console 505 Adding a remote cluster Adding a remote cluster Cluster Explorer provides a wizard to create global clusters by linking standalone clusters Command Center only enables you to perform remote cluster operations on the local cluster m Ifyou are creating a global cluster environment for the first time with two standalone clusters run the wizard from either of the clusters m Ifyou are adding a standalone cluster to an existing global cluster environment run the wizard from a cluster already in the global cluster environment The following information is required for the Remote Cluster Configuration Wizard in Cluster Explorer m The active host name or IP address of each cluster in the global configuration and of the cluster being added to the configuration m The user
338. he same system as fail over only to PAME the Child system where Child system If Child cannot fail instance is running over Parent remains and other instance of online Parent is not running online local Parallel Child Parallel Parent taken offline Child stays online firm instance Child If Child fails over to Parent instance can online on Instance IS another system VCS fail over only to same system online on brings an instanceof system where Child neue the Parent online on instance is running system the same systemas and other instance of Child Parent is not If Child cannot fail TU27N8 over Parent remains offline 422 The role of service group dependencies Service group dependency configurations offline local Parallel Child offline on same system No instance of Child is online on same system Parent remains Child remains online if Child fails online over to another system The role of service group dependencies 423 Group Dependency FAQs Group Dependency FAQs This section lists some commonly asked questions about group dependencies Dependency Online local Online global Online remote Offline local Frequently asked questions Can child group be taken offline when parent group is online Soft Yes Firm No Hard No Can parent group be switched while child group is online Soft No Firm No Hard No Can child group be switched while parent group is online Soft No Firm No H
339. he state of service groups on the node So VCS places all service groups that were online on the affected node in the autodisabled state The service groups that were online on the node cannot fail over Manual intervention is required to enable failover of autodisabled service groups The administrator must release the resources running on the affected node clear resource faults and bring the service groups online on another node About communications membership and data protection in the cluster 353 Examples of VCS operation without I O fencing Examples of VCS operation without I O fencing The following scenarios describe events and how VCS responds in a cluster without I O fencing Four system cluster without a low priority link Consider a four system cluster that has two private cluster interconnect heartbeat links The cluster does not have any low priority link Public Network Regular membership 0 1 2 3 Cluster interconnect link failure In this example a link to System2 fails leaving System2 with only one cluster interconnect link remaining Public Network System2 System3 ai m m se Ea Ea at Regular membership 0 1 2 3 Jeopardy membership 2 354 About communications membership and data protection in the cluster Examples of VCS operation without I O fencing The cluster is reformed Systems 0 1 2 and
340. his command If you do not specify the cluster name the command displays the attribute value on the local cluster To display the state of a local or remote cluster haclus state cluster localclus The variable cluster represents the cluster If a cluster is not specified the state of the local cluster and the state of all remote cluster objects as seen by the local cluster are displayed For information on the state of a local or remote cluster haclus display cluster localclus If a cluster is not specified information on the local cluster is displayed For a list of local and remote clusters haclus list conditionals Lists the clusters that meet the specified conditions beginning with the local cluster To display usage for the cluster command haclus help modify To display the status of a faulted cluster haclus status cluster Displays the status on the specified faulted cluster If no cluster is specified the command displays the status on all faulted clusters It lists the service groups that were not in the OFFLINE or the FAULTED state before the fault occurred It also suggests corrective action for the listed clusters and service groups Querying status For the status of local and remote clusters hastatus 526 Administering global clusters from the command line Global querying Querying heartbeats The hahb command is used to manage WAN heartbeats that emanate from the local clu
341. hose a non global zone for the application proceed to the next step 6 Ifyou chose to configure a resource for a non global zone configure the Mount resources for the zone and the application VCS Application Configuration Wizard Mount Configuration Configure the Mount resources required by the zone Select the mounts required by the zone Specify the options for the Fsck operations For vefs file systems select SnapUmount if desired Select Mount Point_ BlockDevice File System Mount Options Fsck Options SnapUmount m zonefizone JdewdskicttOd jufs tw intr largefile y X Discover Mounts Z Create mount points on all systems if they do not exist More information lt Back Next gt cancel Configuring applications and resources in VCS 303 Configuring application service groups m Select the check boxes next to the mount points required for the application and for the non global zones Click Discover Mounts to discover mounts if required m Specify the Mount and Fsck options if applicable The agent uses these options when bringing the resource online m If using the vxfs file system you can select the SnapUmount check box to take the MountPoint snapshot offline when the resource is taken offline m Select the Create mount points on all systems if they do not exist check box if desired m Click Next Select to create or modify applications
342. host or virtual IP address that the application uses on that cluster This ensures DNS updates to the site when the group is brought online A DNS resource would be necessary only if the nodes in the primary and the secondary RDC zones are in different IP subnets Setting up replicated data clusters 541 Migrating a service group Configuring the service group dependencies Set an online local hard group dependency from application service group to the replication service group to ensure that the service groups fail over and switch together 1 Inthe Cluster Explorer configuration tree select the cluster name 2 Inthe view panel click the Service Groups tab This opens the service group dependency graph 3 Click Link 4 Click the parent group oragroup and move the mouse toward the child group oragroup_rep 5 Click the child group oragroup_rep 6 On the Link Service Groups dialog box click the online local relationship and the hard dependency type and click OK Migrating a service group In the RDC set up for the Oracle database consider a case where the primary RDC zone suffers a total failure of the shared storage In this situation none of the nodes in the primary zone see any device The Oracle service group cannot fail over locally within the primary RDC zone because the shared volumes cannot be mounted on any node So the service group must fail over to a node in the current secondary RDC zone The RVGPrimary agent
343. ice Group Click the service group Select the Per System check box to disable the group on a specific system instead of all systems Click Apply 190 Administering the cluster from Cluster Manager Java console Administering service groups Autoenabling a service group A service group is autodisabled until VCS probes all resources and checks that they are ready to come online Autoenable a service group in situations where the VCS engine is not running on one of the systems in the cluster and you must override the disabled state of the service group to enable the group on another system in the cluster To autoenable a service group from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the service group or Click the cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel Click Autoenable and click the appropriate system from the menu To autoenable a service group from Command Center 1 In the Command Center configuration tree expand Commands gt Operations gt Availability gt Autoenable Service Group Click the service group Click the system on which to autoenable the group Click Apply Administering the cluster from Cluster Manager Java console 191 Administering service groups Flushing a service group As a service group is brought online or taken offline the resources within the group ar
344. ice group G1 has three resources R1 R2 and R3 VCS first offlines R1 When R1 is offline VCS offlines R2 When R 2 is offline VCS offlines R3 The time it takes to offline G1 equals the time it takes for all resources to go offline When a resource fails The time it takes to detect a resource fault or failure depends on the MonitorInterval attribute for the resource type When a resource faults the next monitor detects it The agent may not declare the resource as faulted if the ToleranceLimit attribute is set to non zero If the monitor entry point reports offline more often than the number set in ToleranceLimit the resource is 552 VCS performance considerations How cluster operations affect performance declared faulted However if the resource remains online for the interval designated in the ConfInterval attribute previous reports of offline are not counted against ToleranceLimit When the agent determines that the resource is faulted it calls the clean entry point if implemented to verify that the resource is completely offline The monitor following clean verifies the offline The agent then tries to restart the resource according to the number set in the RestartLimit attribute if the value of the attribute is non zero before it gives up and informs HAD that the resource is faulted However if the resource remains online for the interval designated in ConfInterval earlier attempts to restart are not counted against Resta
345. ich enables them to access the directories being shared A single service group NFS_Group fails over between System A and System B as necessary The VCS engine HAD reads the configuration file determines what agents are required to control the resources in the service group and starts the agents HAD determines the order in which to bring the resources online based on the resource dependencies VCS issues online commands to the corresponding agents in the correct order The following figure shows the dependency graph for the service group NFS_Group Figure 1 3 Dependency graph for the sample NFS group NFSRestart H Co DiskGroup VCS starts the agents for disk group mount share NFS NIC and IP on all systems that are configured to run NFS_Group The resource dependencies are configured as m The home file system configured as a Mount resource requires the disk group configured as a DiskGroup resource to be online before mounting m The NFS export of the home file system Share requires the file system to be mounted and the NFS daemons NFS be running m The high availability IP address nfs_IP requires the file system Share to be shared and the network interface NIC to be up m The NFSRestart resource requires the IP address to be up Introducing Veritas Cluster Server 31 Putting the pieces together m The NFS daemons and the disk group have no child dependencies so they can start in
346. ider the following m If you write a custom agent write the monitor entry point using C or C If you write a script based monitor VCS must invoke a new process each time with the monitor This can be costly if you have many resources of that type VCS performance considerations 549 How cluster components affect performance If monitoring the resources is proving costly you can divide it into cursory or shallow monitoring and the more extensive deep or in depth monitoring Whether to use shallow or deep monitoring depends on your configuration requirements Additional considerations for agents Properly configure the attribute SystemList for your service group For example if you know that a service group can go online on sysa and sysb only do not include other systems in the SystemList This saves additional agent processes and monitoring overhead The VCS graphical user interfaces The VCS graphical user interfaces Cluster Manager Java Console and Cluster Management Console maintain a persistent connection to HAD from which they receive regular updates regarding cluster status For best results run the Java and Web Consoles on a system outside the cluster to avoid impact on node performance 550 VCS performance considerations How cluster operations affect performance How cluster operations affect performance This section describes how operations on systems resources and service groups in the cluster affect
347. ific to the resource types It also displays attributes whose values have been overridden See Overriding resource type static attributes on page 208 To view all attributes associated with the selected VCS object click Show all attributes To access the properties view 1 2 From Cluster Explorer click a VCS object in the configuration tree In the view panel click the Properties tab 153 154 Administering the cluster from Cluster Manager Java console About Cluster Explorer Service Group view The Service Group view displays the service groups and their dependencies in a cluster Use the graph and ScreenTips in this view to monitor create and disconnect dependencies To view the ScreenTips point to a group icon for information on the type and state of the group on the cluster systems and the type of dependency between the service groups VCSHP10_71_0214a Lolk Ha Service Group Yiew YCSHP10_71_0214a g Link 8 Auto Arrange P D ClusterService The line between two service groups represents a dependency or parent child relationship In VCS parent service groups depend on child service groups A service group can function as a parent and a child See About service group dependencies on page 410 The color of the link between service groups indicates different types of dependencies m A blue link indicates a soft dependency m Ared link indicates a firm dependency m
348. ified user names in the format username domain You cannot assign or change passwords for users when VCS is running in secure mode The commands to add modify and delete a user must be executed only as root and only if the VCS configuration is in read write mode See Setting the configuration to read w rite on page 243 Note You must add users to the VCS configuration to monitor and administer VCS from the graphical user interface Cluster Manager Users in the category Cluster Guest cannot add users To add a user 1 Set the configuration to read write mode haconf makerw 2 Add the user hauser add user priv lt Administrator Operator gt group service_groups 3 Enter a password when prompted 4 Reset the configuration to read only haconf dump makero To add a user with cluster administrator access Type the following command hauser add user priv Administrator To add a user with cluster operator access Type the following command hauser add user priv Operator To add a user with group administrator access Type the following command hauser add user priv Administrator group service groups 252 Administering the cluster from the command line Managing VCS users from the command line To add a user with group operator access Type the following command hauser add user priv Operator group service groups Assigning and removing user privileges
349. ighly available disaster protected infrastructure using a second cluster The solution uses Veritas Volume Replicator to replicate changed data real time In this example a single instance Oracle database is configured as a VCS service group appgroup on a two node cluster The service group configuration looks like Resource View appgroup on all systems RP Tk R Auto arrange P EA J appgroup thori65 thori4 Note Before beginning the process review the prerequisites listed in the section Prerequisites for global clusters on page 469 and make sure your configuration is ready for a global cluster application The process involves the following steps m Preparing the application for the global environment m Running the GCO configuration wizard m Configuring replication m Linking the application and replication service groups m Configuring the second cluster m Linking clusters m Configuring the Steward process m Creating the global service group 472 Connecting clusters Creating global clusters Setting up a global cluster Preparing the application for the global environment Install the application Oracle in this example in the second cluster Make sure the installation is identical with the one in the first cluster Set up replication between the shared disk groups in both clusters If your configuration uses VVR the process involves grouping the shared data volumes in the first c
350. iguring NFS service groups 5 Select the shares to be configured in the service group and click Next Share Paths Select the shares to be configured Select the path names to be configured in the service group Select Path Name VCS NFS Configuration Wizard Options i me Ww 2 More Information The wizard displays the shared paths whose mount points do not appear in the file etc vfstab If the path to be configured does not appear in the list make sure the path is shared and click Discover Shares Discover Shares lt Back Next gt cancer Configure Mount resources for the shares VCS NFS Configuration Wizard Mount Configuration Configure the Mount resources required by the shares Specify the options for the Mount and Fsck operations For ws file systems select SnapUmount if desired Mount Paint Block Device File System Mount Options Fsck Options SnapUmount nfs Jdewdskic1tid0s0 ufs inw intr largefiles l f n Z Create mount points on all systems if they do not exist F More Information m Specify the Mount and Fsck options if applicable The agent uses these lt Back Next gt J cancer options when bringing the resource online m If using the vxfs file system you can select the SnapUmount check box to take the MountPoint snapshot offline when the resource is taken offline m Select the Create mount points on all
351. ilable monitor timeout count is exhausted and VCS must now take corrective action m Ifthe ManageFaults attribute is set to NONE VCS marks the resource as ONLINE ADMIN_WaAIT and fires the resadminwait trigger If the ManageFaults attribute is set to ALL the resource enters a GOING OFFLINE WAIT state VCS invokes the Clean entry point with the reason Monitor Hung Controlling VCS behavior 375 Controlling VCS behavior at the resource level If the Clean entry point is successful that is Clean exit code 0 VCS examines the value of the RestartLimit attribute If Clean fails exit code 1 the resource remains online with the state UNABLE TO OFFLINE VCS fires the resnotoff trigger and monitors the resource again If the Monitor routine does not time out it returns the status of the resource as being online or offline If the ToleranceLimit TL attribute is set to a non zero value the Monitor cycle returns offline exit code 100 for a number of times specified by the ToleranceLimit and increments the ToleranceCount TC When the ToleranceCount equals the ToleranceLimit TC TL the agent declares the resource as faulted If the Monitor routine returns online exit code 110 during a monitor cycle the agent takes no further action The ToleranceCount attribute is reset to 0 when the resource is online for a period of time specified by the ConfInterval attribute If the resource is detected as being offline a number of times
352. ild service group Use the corresponding drop down menus for this specification m The relationship type In a Local dependency an instance of the parent service group depends on an instance of the child service group being online or offline on the same system depending on the category of group dependency Ina Global dependency an instance of the parent service group depends on one or more instances of the child service group being online on any system m The dependency category In a Soft dependency VCS imposes minimal constraints while bringing the parent and child service groups online and taking them offline In a Firm dependency VCS takes the child service group offline before taking the parent service group offline when the child service group Administering the cluster from the Cluster Management Console 109 Administering service groups faults In a Hard dependency VCS takes the parent service group offline before taking the child service group offline when the child service group faults Hard dependencies are designed for use with Veritas Volume Replicator in disaster recovery configurations In these configurations the application is in the parent service group and the replication resources are in the child service group Viewing service group dependencies View a graphical representation of dependencies that exist among service groups View a graphical representation of dependencies that exist among service groups To
353. ileges with which you logged on to the cluster ohm 2SBhe BA HOOF EGR4EH BOD From left to right Open Configuration Modifies a read only configuration to a read write file This enables you to modify the configuration Save Configuration Writes the configuration to disk Save and Close Configuration Writes the configuration to disk as a read only file Add Service Group Displays the Add Service Group dialog box Add Resource Displays the Add Resource dialog box Add System Displays the Add System dialog box Manage systems for a Service Group Displays the System Manager dialog box Online Service Group Displays the Online Service Group dialog box Offline Service Group Displays the Offline Service Group dialog box Show Command Center Enables you to perform many of the same VCS operations available from the command line Show Shell Command Window Enables you to launch a non interactive shell command on cluster systems and to view the results on a per system basis COB e e e e a e e 150 Administering the cluster from Cluster Manager Java console About Cluster Explorer Show the Logs Displays alerts and messages received from the VCS engine VCS agents and commands issued from the console Launch Configuration Wizard Enables you to create VCS service groups Launch Notifier Resource Configuration Wizard Enables you to set up VCS event notification Remote Group Resource Co
354. ilover configuration reduces the cost of hardware redundancy and still provides a potential dedicated spare In an asymmetric configuration there is no performance penalty and there are no issues with multiple applications running on the same system however the drawback is the 100 percent redundancy cost at the server level Redundant Server Application Figure 2 3 N to 1 configuration An N to 1 configuration is based on the concept that multiple simultaneous server failures are unlikely therefore a single redundant server can protect multiple active servers When a server fails its applications move to the redundant server For example in a 4 to 1 configuration one server can protect four servers which reduces redundancy cost at the server level from 100 percent to 25 percent In this configuration a dedicated redundant server is cabled to all storage and acts as a spare when a failure occurs About cluster topologies 37 Basic failover configurations The problem with this design is the issue of failback When the failed server is repaired all services hosted on the server must be failed back to the server The failback action frees the spare server and restores redundancy to the cluster Redundant Server a h E v N k z S S o Figure 2 4 N to 1 failover requiring failback Most shortcomings of early N to 1 cluster configurations were caused by the lim
355. im m Solaris pkgs 3 Install the VRTScssim package To use Cluster Manager with Simulator you must also install the VRTScscm package To install VCS Simulator on Windows systems 1 Insert the VCS installation disc into a drive 2 From Windows Explorer navigate to the path of the Simulator installer file located at windows VCSWindowslInstallers Simulator 3 Double click the installer file 4 Read the information in the Welcome screen and click Next 5 Inthe Destination Folders dialog box click Next to accepted the suggested installation path or click Change to choose a different location 6 Inthe Ready to Install the Program dialog box click Back to make changes to your selections or click Install to proceed with the installation 7 Inthe Installshield Wizard Completed dialog box click Finish Predicting VCS behavior using VCS Simulator 319 Installing VCS Simulator Reviewing the installation VCS Simulator installs Cluster Manager Java Console and Simulator binaries on the system The Simulator installation creates the following directories Directory Contents attrpool Information about attributes associated with VCS objects bin VCS Simulator binaries default_clus Files for the default cluster configuration sample_clus A sample cluster configuration which serves as a template for each new cluster configuration templates Various templates used by the Java Console types The types cf files for all suppo
356. in the cluster the group is brought online when the system is started VCS uses the AutoStartPolicy attribute to determine the system on which to bring the service group online Note For the service group to start AutoStart must be enabled and Frozen must be 0 Also beginning with 1 3 0 you must define the SystemList attribute prior to setting this attribute m Type and dimension string keylist m Default none 630 VCS attributes Service group attributes Table D 3 Service group attributes AutoStartPolicy user defined Sets the policy VCS uses to determine on which system to bring a service group online if multiple systems are available This attribute has three options Order default Systems are chosen in the order in which they are defined in the AutoStartList attribute Load Systems are chosen in the order of their capacity as designated in the AvailableCapacity system attribute System with the highest capacity is chosen first Priority Systems are chosen in the order of their priority in the SystemList attribute Systems with the lowest priority is chosen first m Type and dimension string scalar m Default Order ClusterFailOverPolicy user defined Determines how a global service group behaves when a cluster faults The attribute can take the following values Manual The group does not fail over to another cluster automatically Auto The group fails over to another cl
357. in the llttab manual page When LLT informs GAB of a heartbeat loss the systems that are remaining in the cluster coordinate to agree which systems are still actively participating in the cluster and which are not This happens during a time period known as GAB Stable Timeout 5 seconds VCS has specific error handling that takes effect in the case where the systems do not agree GAB marks the system as DOWN excludes the system from the cluster membership and delivers the membership change to the fencing module The fencing module performs membership arbitration to ensure that there is not a split brain situation and only one functional cohesive cluster continues to run The fencing module is turned on by default See About cluster membership and data protection without I O fencing for actions that occur if the fencing module has been deactivated About communications membership and data protection in the cluster 341 About membership arbitration About membership arbitration Membership arbitration is necessary on a perceived membership change because systems may falsely appear to be down When LLT on a system no longer receives heartbeat messages from another system on any configured LLT interface GAB marks the system as DOWN However if the cluster interconnect network failed a system can appear to be failed when it actually is not In most environments when this happens it is caused by an insufficient cluster intercon
358. in your configuration the wizard prompts you for the netmask associated with the virtual IP The wizard detects the netmask you can accept the suggested value or enter another value 6 The wizard starts running commands to create or update the ClusterService group Various messages indicate the status of these commands After running these commands the wizard brings the ClusterService group online Connecting clusters Creating global clusters 473 Setting up a global cluster Configuring replication VCS supports several replication solutions for global clustering Please contact your Symantec sales representative for the solutions supported by VCS This section describes how to set up replication using Veritas Volume Replicator VVR Adding the RVG resources 1 2 3 Create a new service group say appgroup_rep Copy the DiskGroup resource from the appgroup to the new group Configure new resources of type IP and NIC in the appgroup_rep service group The IP resource monitors the virtual IP that VVR uses for replication Configure a new resource of type RVG in the new appgroup_rep service group The RVG agent ships with the VVR software If the RVG resource type is not defined in your configuration import it as instructed below m On the File menu click Import Types m Inthe Import Types dialog box click the file from which to import the resource type By default the RVG resource type is located at the path etc V
359. indicates the tress is written m Type and dimension boolean scalar m Default 1 Priority user defined Enables users to designate and prioritize the service group VCS does not interpret the value rather this attribute enables the user to configure the priority of a service group and the sequence of actions required in response to a particular event m Type and dimension integer scalar m Default 0 Probed system use only Indicates whether all enabled resources in the group have been detected by their respective agents m Type and dimension boolean scalar m Default Not applicable ProbesPending system use only The number of resources that remain to be detected by the agent on each system m Type and dimension integer scalar m Default Not applicable Responding system use only Indicates VCS engine is responding to a failover event and is in the process of bringing the service group online or failing over the node m Type and dimension integer scalar m Default Not applicable Table D 3 VCS attributes 637 Service group attributes Service group attributes Restart system use only For internal use only m Type and dimension integer scalar m Default Not applicable SourceFile system use only File from which the configuration was read Always set to main cf Make sure the path exists on all nodes before configuring this attribute m Typ
360. ine dependencies Dependency categories determine the relationship of the parent group with the state of the child group Online group The parent group must wait for the child group to be brought online dependency before it can start For example to configure a database application and a database service as two separate groups specify the database application as the parent and the database service as the child Offline group The parent group can be started only if the child group is offline and dependency vice versa This behavior prevents conflicting applications from running on the same system For example configure a test application on one system as the parent and the production application on another system as the child Dependency location local global or remote dependencies The relative location of the parent and child service groups determines whether the dependency between them is a local global or remote Local dependency The parent group depends on the child group being online or offline on the same system Global dependency An instance of the parent group depends on one or more instances of the child group being online on any system Remote dependency An instance of parent group depends on one or more instances of the child group being online on any system other than the system on which the parent is online 412 The role of service group dependencies About service group dependencies Dependency rigidity s
361. ine operations on the system To unfreeze a system 1 In the Cluster Systems view in the Systems Listing table click the linked name of the system that you want to unfreeze In the System Summary view in the Operations task panel click Unfreeze In the Unfreeze System dialog box click OK to confirm that you want to unfreeze the system You can unfreeze one or more systems at a time directly from the Cluster Systems view by selecting the check boxes for systems in the Systems Listing table and then clicking Unfreeze in the Operations task panel Modifying system attributes Edit the attributes of a system You must have the role of cluster administrator or service group administrator to edit system attributes Edit the attributes of a cluster You must have the role of cluster administrator to edit cluster attributes To edit a cluster attribute using the attribute table 1 2 In the System Summary view on the secondary tab bar click Attributes In the All attributes for system table locate the line item for the attribute you want to edit Not all attributes can be edited In the Edit column click the edit button In the Edit Attribute dialog box specify the following attribute details and then click OK m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value You can
362. internal in memory representation of the configuration Systems brought online after the first system derive their information from systems running in the cluster You must stop the cluster while you are modifying the files from the command line Changes made by editing the configuration files take effect when the cluster is restarted The node on which the changes were made should be the first node to be brought back online About the VCS configuration language The VCS configuration language specifies the makeup of service groups and their associated entities such as resource types resources and attributes These specifications are expressed in configuration files whose names contain the suffix cf There are several ways to generate configuration files m Use the web based Cluster Management Console m Use Cluster Manager Java Console m Use the command line interface m If VCS is not running use a text editor to create and modify the files VCS configuration concepts 47 About the main cf file About the main cf file The format of the main cf file comprises include clauses and definitions for the cluster systems service groups and resources The main cf file also includes service group and resource dependency clauses Include clauses Include clauses incorporate additional configuration files into main cf These additional files typically contain type definitions including the types cf file Typically custom agents
363. ion string keylist m Default GroupLimit user defined Maximum number of service groups m Type and dimension integer scalar m Default 200 HacliUserLevel user defined This attribute has two case sensitive values NONE hacli is disabled for all users regardless of role COMMANDROOT hacli is enabled for root only Note The command haclus modify HacliUserLevel can be executed by root only m Type and dimension string scalar m Default NONE LockMemory user defined Controls the locking of VCS engine pages in memory This attribute has the following values Values are case sensitive ALL Locks all current and future pages CURRENT Locks current pages NONE Does not lock any pages m Type and dimension string scalar m Default ALL 652 VCS attributes Cluster attributes Table D 5 Cluster attributes LogSize Size of engine log file user defined Minimum value 64KB Maximum value 128MB m Type and dimension integer scalar m Default 33554432 Notifier Indicates the status of the notifier in the cluster specifically system use only State Current state of notifier such as whether or not it is connected to VCS Host The host on which notifier is currently running or was last running Default None Severity The severity level of messages queued by VCS for notifier Values include Information Warning Error and SevereError Default Warning Queue The si
364. ion Guide To add a system to a cluster 1 6 Make sure the system meets the hardware and software requirements for VCS See the Veritas Cluster Server Installation Guide for details Set up the private communication links from the new system Install VCS and require patches on the new system Add the VCS license key See Installing a VCS license on page 235 Configure LLT and GAB to include the new system in the cluster membership Add the new system using the hasys add command To remove a node from a cluster 1 2 Make a backup copy of the current configuration file main cf Switch or remove any VCS service groups from the node The node cannot be removed as long as it runs service groups on which other service groups depend Stop VCS on the node hastop sys systemname Delete the system from the SystemList of all service groups hagrp modify groupname SystemList delete systemname Delete the node from the cluster hasys delete systemname Remove the entries for the node from the following files on each remaining node m etc gabtab a etc Ilthosts Unconfigure GAB and LLT on the node leaving the cluster Remove VCS and other packages from the node 278 Administering the cluster from the command line Administering clusters 9 Remove GAB and LLT configuration files from the node To modify a cluster attribute haclus help modify Setting cluster attributes from the command li
365. ion gt specifies the Action defined for the agent The virtual fire drill checks for a resource type can are defined in the SupportedActions attribute and can be identified with the vfd suffix To run a virtual fire drill for a service group Type the following command havfd grp lt grpname gt sys lt sysname gt The command runs the infrastructure check verifies whether the system lt sysname gt has the required infrastructure to host resources in the service group lt grpname gt should a failover require the service group to come online on the system For the variable lt sysname gt specify the name of a system on which the resource is offline To fix detected errors Type the following command hares action lt resname gt lt vfdaction gt vfd fix sys lt sysname gt The variable lt vfdaction gt represents the check that reported errors for the system specified by the variable lt sysname gt The virtual fire drill checks for a resource type can are defined in the SupportedActions attribute and can be identified with the vfd suffix 276 Administering the cluster from the command line Administering systems Administering systems To modify a system s attributes Type the following command hasys modify modify options Some attributes are internal to VCS and cannot be modified For details on system attributes see About the modify option on page 234 To display the value of a system s n
366. is set to 0 resource faults are contained at the resource level VCS does not take the dependency tree offline thus preventing failover If the resources in the service group remain online the service group remains in the PARTIAL FAULTED state If all resources are offline or faulted the service group remains in the OFFLINE FAULTED state When a resource faults VCS fires the resfault trigger and sends an SNMP trap The trigger is called on the system where the resource faulted and includes the name of the faulted resource See resfault event trigger on page 454 Controlling VCS behavior 371 Controlling VCS behavior at the service group level Customized behavior diagrams The illustrations in this section depict how the ManageFaults and FaultPropagation attributes change VCS behavior when handling resource faults The following illustration depicts the legends used in the section Resource Color Code sinc Cy Online Admin_Wait Scenario Resource with a critical parent and ManageFaults NONE The service group in the following example has five resources The ManageFaults attribute is set to NONE for resource R2 If resource R2 fails the resource is marked as ONLINE ADMIN_WAIT The Clean entry point is not called for the resource VCS does not take any other resource offline 372 Controlling VCS behavior Controlling VCS behavior at the service group level Scenario Resource with a critical parent and Fault
367. is the system on which you configure the service group The target system name Find the system you want in the list under Select and Add Systems To quickly find a specific system click Filter type a few letters of the system name and then click Apply To select a system click the system name under Select and Add Systems and then click the right arrow button to move it under Selected Systems You can select multiple systems To reject a system click the system name under Selected Systems and then click the left arrrow button This action moves the system back under Select and Add Systems The startup system Under Select and Add Systems check the Startup option next toa system name if you want the service group to startup automatically on that system 5 Inthe Resource Creation dialog box specify the following details about the resources and the service group and then click Next The resource name Enter the name of the resource that you want to add to the service group into the Resource Name text box Resource names must 99 100 Administering the cluster from the Cluster Management Console Administering service groups Consist of only one string m Not contain special characters like period or amp ampersand m Begin with an alphabetic character The remaining characters in the name can be letters digits hyphens or underscores m The resource type Select the type of the new resource from the Resource Type d
368. isable this dialog box from being displayed every time you connect to the cluster select the Do not show during startup check box Click OK to connect to the cluster The animated display shows various objects such as service groups and resources being transferred from the server to the console Cluster Explorer is launched automatically upon initial logon and the icons in the cluster panel change color to indicate an active panel Administering the cluster from Cluster Manager Java console 173 Administering Cluster Monitor Logging off of a cluster 1 If Cluster Monitor is in the default expanded state proceed to step 2 If Cluster Monitor is in the collapsed state On the View menu click Expand or On the View menu click Stop when an active panel appears as the view panel Right click the active panel and click Logout or If the appropriate panel is highlighted click Logout on the File menu Cluster Explorer closes and the Cluster Monitor panel becomes inactive You may be prompted to save the configuration if any commands were executed on the cluster To log off from Cluster Explorer Click Log Out on the File menu 174 Administering the cluster from Cluster Manager Java console Administering user profiles Administering user profiles The Java Console enables a user with Cluster Administrator privileges to add modify and delete user profiles The icon next to each user name in the User Manag
369. istent disabled resources whose AutoStart attributes are set to 1 the group state is PARTIAL even though enabled resources with Autostart 1 are online This is because the disabled resource is considered for the group state To have the group in the ONLINE state when enabled resources with AutoStart set to 1 are in ONLINE state set the AutoStart attribute to 0 for the disabled non persistent resources Controlling VCS behavior 387 Changing agent file paths and binaries Changing agent file paths and binaries By default VCS runs agent binaries from the path VCS_HOME bin AgentName AgentNameAgent For example opt VRTSvcs bin FileOnOff FileOnOffAgent You can instruct VCS to run a different set of agent binaries or scripts by specifying values for the AgentFile and AgentDirectory attributes AgentFile Specify a value for this attribute if the name of the agent binary is not the same as that of the resource type For example if the resource type is MyApplication and the agent binary is called MyApp set the AgentFile attribute to MyApp For a script base agent you could configure AgentFile as opt VRTSvcs bin ScriptAgent AgentDirectory Specify a value for this attribute if the agent is not installed at the default location When you specify the agent directory VCS looks for the agent file AgentNameAgent in the agent directory If the agent file name does not conform to the AgentNameAgent convention configure the Agent
370. it m To search log messages enter the search string Select the Whole String check box if required m Click OK To customize the display for agent logs 1 Inthe Agent Logs tab enter the filter criteria VCSHP10_71_0214a Logs admin Cluster Administrator file view Help VCS Logs Agent Logs Command Logs Alerts System Name fvcshp10 Last Logs 50 Resource Type Apache T Resource Named fal z Log Type Time Description Click the name of the system Enter the number of logs to view Click the resource type Click the name of the resource To view messages for all resources click All m Click Get Logs Resetting the log display Use the Reset Filters feature to set the default settings for the log view For example if you customized the log view to only show critical and error messages using the Edit Filters feature the Reset Filters feature will set the view to show all log messages To reset the default settings for the log display In the VCS Logs tab click Reset Filters 230 Administering the cluster from Cluster Manager Java console Administering logs Monitoring alerts The Java Console sends automatic alerts that require administrative action and are displayed on the Alerts tab of the Logs dialog box Use this tab to take action on the alert or delete the alert To take action on an alert 1 2 3 4 In the Alert tab or dialog box c
371. it Attribute dialog box specify the following attribute details and then click OK m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value You can enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values m The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List To remove resource overrides 1 In the Cluster Resources view in the Resources Listing table click a linked resource name In the Resource Summary view in the Configuration task panel click Remove Overrides 118 Administering the cluster from the Cluster Management Console Administering resources 3 Click OK to confirm that you want to remove previously configured override values Enabling and disabling resources Prepare a resource to be brought online or prevent a resource from being brought online A resource must be enabled before it can be brought online A disabled resource cannot be brought online You must have the role of cluster operator or service group operator to enable or disable resources To enable or di
372. itations of storage architecture Typically it was impossible to connect more than two hosts to a storage array without complex cabling schemes and their inherent reliability problems or resorting to expensive arrays with multiple controller ports 38 About cluster topologies Advanced failover configurations Advanced failover configurations The advent of SANs and second generation high availability HA products such as VCS has enabled several new and useful failover configurations described in the following sections N 1 configuration With the capabilities introduced by storage area networks SANs you can not only create larger clusters but more importantly can connect multiple servers to the same storage Service Group Se Figure 2 5 N 1 configuration A dedicated redundant server is no longer required in the configuration Instead of N to 1 configurations you can use an N 1 configuration In advanced N 1 configurations an extra server in the cluster is spare capacity only About cluster topologies 39 Advanced failover configurations When a server fails the application service group restarts on the spare After the server is repaired it becomes the spare This configuration eliminates the need for a second application failure to fail back the service group to the primary system Any server can provide r
373. itoring of VCS Adding deleting and modifying resource attributes Resource names must be unique throughout the cluster and you cannot modify resource attributes defined by the system such as the resource state To modify a new resource Type the following command hares modify resource attribute value hares modify lt resource gt lt attr gt lt value gt sys lt system gt wait time lt waittime gt The variable value depends on the type of attribute being created To set a new resource s enabled attribute to 1 Type the following command hares modify resourceA Enabled 1 The agent managing the resource is started on a system when its Enabled attribute is set to 1 on that system Specifically the VCS engine begins to monitor the resource for faults Agent monitoring is disabled if the Enabled attribute is reset to 0 To add a resource attribute haattr add resource type attribute value dimension default The variable value is a string default integer or boolean The variable dimension is scalar default keylist assoc or vector The variable default is the default value of the attribute and must be compatible with the value and dimension Note that this may include more than one item as indicated by ellipses 270 Administering the cluster from the command line Administering resources To delete a resource attribute haattr delete resource type attribu
374. its An unordered set of name value pairs denoting specific resources available on a system Names are arbitrary and are set by the administrator for any value Names are not obtained from the system user defined The format for Limits is Limits Name Value Name2 Value2 m Type and dimension integer association m Default LinkHbStatus Indicates status of private network links on any system system use only m Type and dimension string association m Default Not applicable LLTNodeld Displays the node ID defined in the file etc IIttab system use only m Type and dimension integer scalar m Default Not applicable LoadTimeCounter System maintained internal counter of how many seconds the system load has been above LoadWarningLevel This value resets to zero anytime system load drops below the value in LoadWarningLevel system use only m Type and dimension integer scalar m Default Not applicable VCS attributes System attributes 645 Table D 4 System attributes LoadTimeThreshold user defined How long the system load must remain at or above LoadWarningLevel before the LoadWarning trigger is fired If set to 0 overload calculations are disabled m Type and dimension integer scalar m Default 600 LoadWarningLevel user defined A percentage of total capacity where load has reached a critical limit If set to 0 overload calculations are disabled For example setting
375. ity When a failover occurs VCS determines which system has the highest available capacity and starts the service group on that system During a failover involving multiple service groups VCS makes failover decisions serially to facilitate a proper load based choice System capacity is a soft restriction in some situations value of the Capacity attribute could be less than zero During some operations including cascading failures the value of the AvailableCapacity attribute could be negative Static load versus dynamic load Dynamic load is an integral component of the Service Group Workload Management framework Typically HAD sets remaining capacity with the function 392 Controlling VCS behavior Service group workload management AvailableCapacity Capacity sum of Load values of all online service groups If the DynamicLoad attribute is defined its value overrides the calculated Load values with the function AvailableCapacity Capacity DynamicLoad This enables better control of system loading values than estimated service group loading static load However this requires setting up and maintaining a load estimation package outside VCS It also requires modifying the configuration file main cf manually Note that the DynamicLoad specified with hasys load is subtracted from the Capacity as an integer and not a percentage value For example if a system s capacity is 200 and the load estimation package determine
376. ize of the view To view a segment of the graph point to the box to the right of the magnifying glass icon Use the red outline in this box to encompass the appropriate segment of the graph Click the newly outlined area to view the segment To return to the original view click the magnifying glass icon labeled 1 158 Administering the cluster from Cluster Manager Java console About Cluster Explorer System Connectivity view The System Connectivity view displays the status of system connections in a cluster Use this view to monitor the system links and disk group heartbeats VCSHP10_71_0214a olki a System Connectivity View CSHP10_71_0214a g 3 ae pot yveshp71 veshp10 VCS monitors systems and their services over a private network The systems communicate via heartbeats over an additional private network which enables them to recognize which systems are active members of the cluster which are joining or leaving the cluster and which have failed VCS protects against network failure by requiring that all systems be connected by two or more communication channels When a system is down to a single heartbeat connection VCS can no longer discriminate between the loss of a system and the loss of a network connection This situation is referred to as jeopardy Point to a system icon to display a ScreenTip on the links and disk group heartbeats If a system in the cluster is experiencing a problem conn
377. k to remove a field Administering the cluster from Cluster Manager Java console 227 Setting up VCS event notification using the Notifier wizard m Enter the recipient and click the severity level in the drop down list box For example admin example com and Information m Click Next 7 Inthe NIC Resource Configuration dialog box Notifier Resource Configuration Wizard NIC Resource Configuration We recommend a Notifier resource dependency on a NIC resource I7 Configure NIC Resource Resource Name esanic Node vvesinuct41 vesinuxl42 cancel Hep Notifier Resource Configuration Wizard NIC Resource Configuration We recommend a Notifier resource dependency on a NIC resource I7 Configure NIC Resource Resource Name esgnic Node Device Discover Network Hosts vveshp10 il 10 212 101 80 veshp71 Fl 3 cancel Hep m Click Configure NIC Resource recommended by Symantec and proceed to the next step Otherwise click Next m If necessary enter the name of the resource m Click the icon in the Discover column of the table to find the MACAddress for each system m Click OK on the Discover dialog box m Click Next 8 Click the Bring the Notifier Resource Online check box if desired 9 Click Next 10 Click Finish 228 Administering the cluster from Cluster Manager Java console Administering logs Administering logs
378. k New Simulator on the File menu or Click New Cluster on the Cluster Monitor toolbar Enter the details to connect to the cluster Host name fo Port haa ooo Failover retries hz o o px Enter the host name or IP address of a system in the cluster m If necessary change the default port number of 14141 VCS Simulator uses a default port number of 14153 Note that you must use a different port to connect to each Simulator instance even if these instances are running on the same system m Enter the number of failover retries VCS sets the default failover retries number to 12 m For simulated clusters click the platform for the configuration m Click OK An inactive panel appears in Cluster Monitor 170 Administering the cluster from Cluster Manager Java console Administering Cluster Monitor Modifying a cluster panel configuration Modify a cluster panel to point to another cluster to change the port number or the number of failover retries 1 uo F amp F WwW N If Cluster Monitor is in the default expanded state proceed to step 2 If Cluster Monitor is in the collapsed state On the View menu click Expand or On the View menu click Stop when an active panel appears as the view panel Right click the cluster panel If the panel is inactive proceed to step 4 On the menu click Logout The cluster panel becomes inactive Right click the inactive panel and click Configure Edit t
379. k and the low priority link remaining Public Network System0 System1 System2 System3 Regular membership 0 1 3 No jeopardy Other systems send all cluster status traffic to System2 over the remaining private link and use both private links for traffic between themselves The low About communications membership and data protection in the cluster 357 Examples of VCS operation without I O fencing priority link continues carrying the heartbeat signal only No jeopardy condition is in effect because two links remain to determine system failure Cluster interconnect link failure followed by system failure In this example the link to System2 fails Because there is a low priority heartbeat link System2 is not put in the jeopardy membership Subsequently System2 fails due to a power fault Public Network System0 System1 System2 System3 Regular membership 0 1 3 Systems 0 1 and 3 recognize that System2 has faulted The cluster is reformed Systems 0 1 and 3 are in a regular membership The service groups on System2 that are configured for failover on system fault are attempted to be brought online on another target system if one exists All high priority cluster interconnect links fail In this example both high priority cluster interconnect links to System2 fail leaving System2 with only the low priority link remaining Cluster status communication is now
380. k the service groups 1 2 In the Cluster Explorer configuration tree click the cluster name In the view panel click the Service Groups tab This opens the service group dependency graph Click Link Click the parent group appgroup and move the mouse toward the child group appgroup_rep Click the child group appgroup_rep In the Link Service Groups dialog box click the online local relationship and the firm dependency type and click OK Configuring the second cluster 1 Run the GCO Configuration wizard in the second cluster See Running the GCO configuration wizard on page 472 Create a configuration that is similar to the one in the first cluster You can do this by either using Cluster Manager Java Console to copy and paste resources from the primary cluster or by copying the configuration of the appgroup and appgroup_rep groups from the main cf file in the primary cluster to the secondary cluster To assign remote administration privileges to users configure users with the same name and privileges on both clusters See User privileges in global clusters on page 64 Make appropriate changes to the configuration For example you must modify the SystemList attribute to reflect the systems in the secondary cluster Make sure that the name of the service group appgroup is identical in both clusters It is a VVR best practice to use the same disk group and RVG name on both sites If the volume names
381. kGroup agent times out case 3 in the table above the FaultOnMonitorTimeouts attribute defines when VCS interprets the resource as faulted and invokes the Clean entry point If the CleanReason is monitor hung the system panics 389 390 Controlling VCS behavior VCS behavior on loss of storage connectivity About VCS behavior when a disk group is disabled Figure 11 6 describes VCS behavior for a disabled diskgroup Figure 11 6 VCS behavior when a disk group is disabled DG state is disabled Monitor 1 0 fencing No failover Yes Clean Entry Point PanicSyst No __ Log message Attempts Trigger emOnDG gt and return gt deportofDG7 _ failover Loss yes Panic system dump crash and halt processes Recommendations to ensure application availability Symantec makes the following recommendations to ensure application availability and data integrity when a node loses connectivity to shared storage m Do not set the failover attribute for the FC driver to 0 However if you do set the failover attribute to 0 set the FaultOnMonitorTimeouts value for the DiskGroup resource type to a finite value m If you use I O fencing set the PanicSystemOnDGLoss attribute for the DiskGroup resource to 1 This ensures that the system panics when it loses connectivity to shared storage and causes applications to fail over to another node The failover ensures application availability while
382. kes no action except implicitly marking the service groups as offline in the failed cluster Switching the service group back to the primary You can switch the service group back to the primary after resolving the fault at the primary site Before switching the application to the primary site you must resynchronize any changed data from the active Secondary site since the failover This can be done manually through VVR or by running a VCS action from the RVGPrimary resource To switch the service group when the primary site has failed and the secondary did a takeover 1 2 3 In the Service Groups tab of the configuration tree right click the resource Click Actions Specify the details of the action a From the Action list choose fbsync b Click the system on which to execute the action c Click OK This begins a fast failback of the replicated data set You can monitor the value of the ResourceInfo attribute for the RVG resource to determine when the resynchronization has completed Once the resynchronization completes switch the service group to the primary cluster a Inthe Service Groups tab of the Cluster Explorer configuration tree right click the service group b Click Switch To and click Remote switch 482 Connecting clusters Creating global clusters Switching the service group back to the primary c Inthe Switch global group dialog box click the cluster to switch the group Click the specific system or c
383. king service groups on page 192 To link service groups from the command line Type the following command hagrp link parent group child group gd category gd_location gd type parent_group Name of the parent group child_group Name of the child group gd_category category of group dependency online offline gd_location the scope of dependency local global remote gd_type type of group dependency soft firm hard Default is firm VCS behavior with service group dependencies VCS enables or restricts service group operations to honor service group dependencies VCS rejects operations if the operation violates a group dependency Online operations in group dependencies Typically bringing a child group online manually is never rejected except under the following circumstances For online local dependencies if parent is online a child group online is rejected for any system other than the system where parent is online For online remote dependencies if parent is online a child group online is rejected for the system where parent is online For offline local dependencies if parent is online a child group online is rejected for the system where parent is online The role of service group dependencies 425 VCS behavior with service group dependencies The following examples describe situations where bringing a parallel child group online is accepted m Fora parallel child group linked online local with failover
384. l click Enable 3 Inthe Enable Service Group dialog box use the drop down menu to select a system on which to enable the service group Select all to enable the service group on all systems in the cluster 4 Click OK 107 108 Administering the cluster from the Cluster Management Console Administering service groups Disabling a service group Disable a service group to prevent it from coming online Use disablement to temporarily prevent agents from monitoring a service group on a system undergoing maintenance operations To disable a service group 1 In the Cluster Groups view in the Groups Listing table click the linked name of the enabled service group that you want to disable In the Group Summary view in the Operations task panel click Disable In the Disable Service Group dialog box use the drop down menu to select a system on which to disable the service group Select all to disable the service group on all systems in the cluster Click OK Linking service groups Create dependencies among service groups See About service group dependencies on page 410 To link two service groups with a parent child dependency 1 In the Cluster Groups view in the Configuration task panel click Link Service Groups In the Link Service Groups dialog box specify the following details of the dependency and then click OK m The service groups that you intend to serve as the parent service group and the ch
385. l clusters from the Cluster Management Console Administering global service groups To take a remote service group offline 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the online service group that you want to take offline 2 Inthe Group Summary view in the Remote Operations task panel click Offline 3 Inthe Offline Service Group dialog box specify the following details for the task and then click OK m The target cluster Select the cluster on which you want to take the remote service group offline from the Select the cluster you want to offline this group on drop down menu The cluster choices are populated using the cluster list for the remote service group m The target system Select the system on which you want to take the remote service group offline from the Select the system you want to offline this group on drop down menu The system choices are populated using the system list for the remote service group The Anywhere option causes this task to take the service group offline on the first system it is found to be running Switching a remote service group Take a service group offline on one system in a cluster and bring it online on another system using a single task To switch a service group to another system 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the remote service group that you want to switch 2 Inthe Group Summary view in the Remote
386. lays the differences between the files var Ilttab on each node in the cluster and the version stored in the snapshot contained in the file var backup 2 2 2003 hasnap fdiff f tmp backup 2 2 2003 etc llttab 248 Administering the cluster from the command line Managing VCS configuration files Exporting snapshots Use the hasnap export command to export a snapshot from the local predefined directory on each node in the cluster to a specified file This option is useful when you want to store a previously created snapshot to an alternate location that is periodically backed up using backup software like Veritas NetBackup To export a snapshot Run the following command hasnap export f filename s snapid s Specifies the ID of the snapshot f Specifies the file If you do not specify a snapshot ID the command exports the latest snapshot to the specified file Example The following command exports data from snapshot vces 20030101 22232 from each node in the cluster to the file tmp backup 2 2 2003 on the current node hasnap export f tmp backup 2 2 2003 s vces 20030101 22232 Adding and removing files for snapshots Use the hasnap include command to configures the list of files or directories to be included in new snapshots This list is in addition to the files included by the backup command See Backing up configuration files on page 245 To add or remove files for a snapshots Run the fo
387. ld group If Child fails over to maintaining group are online the system on which dependence on the remaining on other Parent is online Child instances online on systems Parent fails over to other another system systems online All instances Allinstances Parent is taken Parent fails over to remote firm parallel Child of Child offline another system group group are If Child fails over to maintaining remaining online on the system on which dependence on all online on other Parent is online Child instances other systems Parent fails over to systems another system If Child fails over to another system Parent is brought online on its original system offline local Parallel Child No instance Parent remains Child remains offline on of Child is online if Child fails online same system online on over to another same system system If Child fails over to the system on which Parent is online Parent fails over 420 The role of service group dependencies Service group dependency configurations Parallel parent Failover child online Failover Child Failover Parent remains Child remains online global soft group online Child is online somewhere in online the cluster somewhere in the cluster online Failover Child Failover Allinstances of Child stays online global firm group Child is Parent taken offline somewherein online After Child fails over the cluster somewhere Parent instances ate in the brought fai
388. ld groups are displayed in the Child Service Groups box 3 Click a child service group 4 Click the group relationship and dependency type See About service group dependencies on page 410 5 Click Apply 194 Administering the cluster from Cluster Manager Java console Administering service groups Unlinking service groups To delete a service group dependency from Cluster Explorer 1 2 3 4 5 Click a cluster in the configuration tree In the view panel click the Service Groups tab In the Service Group view right click the link between the service groups Click Unlink from the menu T VCSHP10_71_0214a _ Ol x amp Link f Auto arrange P Gs Service Group View VCSHP10_71_0214a ClusterService SCM_SG1 Dependency Click Yes To delete a service group dependency from Command Center 1 In the Command Center configuration tree expand Commands gt Configuration gt Dependencies gt Unlink Service Groups Click the parent resource group in the Service Groups box After selecting the parent group the corresponding child groups are displayed in the Child Service Groups box Click the child service group Click Apply Administering the cluster from Cluster Manager Java console 195 Administering service groups Managing systems for a service group From Cluster Explorer use System Manager to add and remove systems in a service group s system list To add a
389. le includes a line in the resource definition for the static attribute and its overridden value To override resource type static attribute 1 Right click the resource in the Service Groups tab of the configuration tree or in the Resources tab of the view panel Click Override Attributes Select the attributes to override Click OK The selected attributes appear in the Overridden Attributes table in the Properties view for the resource To modify the default value of an overridden attribute click the icon in the Edit column of the attribute To restore default settings to a type s static attribute 1 Right click the resource in the Service Groups tab of the configuration tree or in the Resources tab of the view panel Click Remove Attribute Overrides Select the overridden attributes to be restored to their default settings Click OK Administering the cluster from Cluster Manager Java console 209 Administering resources Enabling resources in a Service group Enable resources in a service group to bring the disabled resources online A resource may have been manually disabled to temporarily stop VCS from monitoring the resource You must specify the values of mandatory attributes before enabling a resource To enable an individual resource in a service group 1 From Cluster Explorer click the Service Groups tab of the configuration tree 2 Right click a disabled resource in the configuration tree and click
390. learing from Java Console 217 definition 618 refreshing from Java Console 217 ResourceLimit attribute 653 ResourceOwner attribute 619 resources about 21 adding from Cluster Management Console 116 adding from command line 268 adding from Java Console 199 administering from Cluster Management Console 115 127 administering from Java Console 199 bringing online from Cluster Management Console 118 bringing online from command line 273 bringing online from Java Console 204 categories of 22 clearing faults from Cluster Management Console 120 clearing faults from Java Console 211 creating faults in VCS Simulator 325 deleting from Cluster Management Console 121 deleting from command line 269 deleting from Java Console 204 disabling from command line 384 disabling from Java Console 210 enabling from command line 264 enabling from Java Console 209 how disabling affects states 386 invoking actions 216 limitations of disabling 384 linking from command line 273 linking from Java Console 212 On Off 22 On Only 22 Persistent 22 probing from Cluster Management Console 120 probing from Java Console 207 querying from command line 255 taking offline from command line 273 taking offline from Java Console 205 troubleshooting 571 unlinking 122 unlinking from Cluster Management Console 122 unlinking from command line 273 unlinking from Java Console 214 Responding attribute 636 resstatechange event trigger 4
391. led over cluster or restarted on the same systems online Failover Child Failover If Child fails over to Child remains remote soft group on Child is system on which online Parent tries another online on Parent is online to fail over to system another Parent fails overto another system system other systems where child is not If Child fails over to online another system Parent remains online online Failover Child Failover Allinstances of Child remains remote firm group on Child is Parent taken offline online Parent tries another online on If Child fails overto t fail over to system another system on which another system system where child is not Parent was online Parent fails over to other systems If Child fails over to another system Parent brought online on same systems online The role of service group dependencies 421 Service group dependency configurations offline local Failover Child Failover offline on same system Child is not online on same system Parent remains online if Child fails over to another system Child remains online Parallel parent Parallel child Global dependencies between parallel parent groups and parallel child groups are not supported online local Parallel Child Parallel If Child fails over to Child instance stays soft instance Child another system online online on instanceis Parent migrates to Payentdnstance can same system online on t
392. leshooting and recovery for VCS Troubleshooting and recovery for global clusters VCS alerts report that the cluster is faulted the querying connector also considers it faulted and transitions the remote cluster state to FAULTED VCS alerts are identified by the alert ID which is comprised of the following elements m alert_type The type of the alert See Types of alerts m cluster The cluster on which the alert was generated m system The system on which this alert was generated m object The name of the VCS object for which this alert was generated This could be a cluster or a service group Alerts are generated in the following format alert_type cluster system object For example GNOFAILA Clusterl oracle_grp This is an alert of type GNOFAILA generated on cluster Cluster1 for the service group oracle_grp Types of alerts VCS generates the following types of alerts m CFAULT Indicates that a cluster has faulted m GNOFAILA Indicates that a global group is unable to fail over within the cluster where it was online This alert is displayed if the ClusterFailOverPolicy attribute is set to Manual and the wide area connector wac is properly configured and running at the time of the fault m GNOFAIL Indicates that a global group is unable to fail over to any system within the cluster or in a remote cluster Some reasons why a global group may not be able to fail over to a remote cluster m The ClusterFailO
393. lete command 261 hagrp dep command 254 hagrp disable command 264 hagrp disableresources command 265 hagrp display command for global clusters 522 for local clusters 254 hagrp enable command 264 hagrp enableresources command 264 hagrp freeze command 264 hagrp list command for global clusters 522 for local clusters 260 hagrp modify command 261 hagrp offline command for global clusters 529 for local clusters 263 hagrp online command for global clusters 528 for local clusters 263 hagrp resources command 254 hagrp state command for global clusters 522 for local clusters 254 hagrp switch command for global clusters 529 for local clusters 263 hagrp unfreeze command 264 hagrp unlink command 267 hagrp value command 522 hagrp wait command 284 hahb add command 533 hahb display command 526 hahb list command 526 halogin command 240 hamsg info command 258 hamsg list command 258 hanotify utility 434 hares action command 530 hares add command 268 hares clear command 274 hares delete command 269 hares dep command 255 hares display command for global clusters 523 for local clusters 255 hares global command 255 hares info command 530 hares link command 273 hares list command for global clusters 523 for local clusters 260 hares local command 271 hares modify command 269 hares offline command 273 hares offprop command 274 hares online command 273 hares override command 282 hares p
394. lick Any System and click OK Connecting clusters Creating global clusters 483 Setting up a fire drill Setting up a fire drill The Disaster Recovery Fire Drill procedure tests the fault readiness of a configuration by mimicking a failover from the primary site to the secondary site This procedure is done without stopping the application at the primary site and disrupting user access interrupting the flow of replicated data or causing the secondary to need resynchronization The initial steps to create a fire drill service group on the secondary site that closely follows the configuration of the original application service group and contains a point in time copy of the production data in the Replicated Volume Group RVG Bringing the fire drill service group online on the secondary site demonstrates the ability of the application service group to fail over and come online at the secondary site should the need arise Fire drill service groups do not interact with outside clients or with other instances of resources so they can safely come online even when the application service group is online You must conduct a fire drill only at the Secondary site do not bring the fire drill service group online on the node hosting the original application VCS also supports virtual fire drills to verify a resource can fail over to another node in the cluster See Testing resource failover using virtual fire drills on page 315 Note
395. lick the alert to take action on E VCSSUNS4_85_Feb14 Alert Jolx Alerts Administrative action is required For the Following alerts Please take appropriate action for the alerts or delete them Alert Name Alert Description Cluster Fault Fire Dril za DRILL ALERT CFAULT_FIRE_DRILL VCSSU J7 Always pop up a dialog box when an alert is generated Click Take Action Enter the required information to resolve the alert If the alert warns that a local group cannot fail over to any system in the local cluster you cannot take any action If the alert warns that a global group cannot fail over the action involves bringing the group online on another system in the global cluster environment If the alert warns that a global cluster is faulted the action involves declaring the cluster as a disaster disconnect or outage and determining the service groups to fail over to another cluster E Declare Cluster x S Declare Cluster clusi46 Declare Disconnect Select groups to Failover and appropriate clusters V sgi FD clusi45 T sot_fd clus145 Click OK To delete an alert 1 In the Alert tab or dialog box click the alert to delete Administering the cluster from Cluster Manager Java console 231 Administering logs 2 Click Delete Alert 3 Provide the details for this operation T Alert delete Ea Please provide the description for deleting the Alert ok
396. line Select all service groups by checking the check box at the top In the Operations task panel click Online Anywhere Administering the cluster from the Cluster Management Console 105 Administering service groups 3 Inthe Online Service Group s dialog box click OK to confirm that you want to bring the selected service groups online on any system in the cluster The candidate systems are members of the system list for each service group The Online Anywhere task tries each system in a system list until every service group is successfully brought online Taking a service group offline Manually put a service group into an unresponsive nonfunctioning state You must have the role of cluster operator or service group operator to take a service group offline You can take a service group offline on a specific system or you can take the service group offline anywhere If you select the Anywhere option the service group is taken offline on the first system in the cluster on which it is found To take a service group offline with the opportunity to specify one system 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the online service group that you want to take offline 2 Inthe Group Summary view in the Operations task panel click Offline 3 This step is necessary if only if the service group is online on more than one system In the Offline Service Group dialog box select the system on whic
397. line event trigger 451 PreOnline attribute 635 preonline event trigger 452 PreOnlineTimeout attribute 635 PreOnlining attribute 635 Prerequisites attribute 636 pretty printing 242 PrintMsg attribute 652 PrintTree attribute 636 priorities defaults 557 ranges 557 scheduling 556 specifying 557 Priority attribute 636 priority ranges for sched classes 556 privileges See user privileges Probed attribute for resources 618 for service groups 636 ProbesPending attribute 636 ProcessClass attribute 652 ProcessPriority attribute 653 696 Index Q quick reopen 554 R ReadOnly attribute 653 Recovering After a Disaster 541 Remote Cluster Configuration wizard 505 Remote Cluster States 606 remote clusters monitoring from Java Console 159 states of 607 replicated data clusters about 43 setting up 538 replicated data configuration 43 resadminwait event trigger 452 reserved words 55 reserved words list of 55 resfault event trigger 454 resnotoff event trigger 454 resource attributes 614 resource dependencies creating from Cluster Management Console 121 creating from command line 273 creating from Java Console 212 displaying from command line 255 removing from Cluster Management Console 122 removing from command line 273 removing from Java Console 214 resource faults clearing from Java Console 211 simulating 325 resource type attributes 620 resource types importing 217 querying from command line 256 Resourcelnfo attribute c
398. list The predefined list includes all the cf files custom agents LLT and GAB configuration files triggers custom heartbeats and action scripts See the include and exclude commands to construct a user defined list To back up VCS configuration files Run the following command hasnap backup f filename n m description Use the n option to run the command in the non interactive mode Use the m option to specifies a description of the snapshot Examples The following command creates a backup of the configuration in the non interactive mode and adds Test Backup as the backup description hasnap backup n m Test Backup The following command creates a backup of the configuration files and saves it as tmp backup 2 2 2003 on the node where the command was run hasnap backup f tmp backup 2 2 2003 Restoring VCS configuration files The hasnap restore command restores configuration files from a previously created snapshot To restore VCS configuration files Run the following command hasnap restore f filename n s snapid n Run the command in the non interactive mode s option to specifies the ID of the snapshot to be restored If you do not specify a snapshot ID the command lists the snapshots that are available for restoration 246 Administering the cluster from the command line Managing VCS configuration files Examples The following command restores the snapshot vcs 20030101 2
399. ll communication links between the two clusters are broken In global clusters with more than three clusters VCS queries the connected clusters to confirm that the remote cluster is truly down This mechanism is called inquiry In a two cluster setup VCS uses the Steward process to minimize chances of a wide area split brain The process runs as a standalone binary on a system outside of the global cluster configuration Cluster A i ae pen se Cluster B Steward When all communication links between any two clusters are lost each cluster contacts the Steward with an inquiry message The Steward sends an ICMP ping to the cluster in question and responds with a negative inquiry if the cluster is running or with positive inquiry if the cluster is down The Steward can also be used configurations with more than two clusters See Configuring the Steward process on page 478 A Steward is effective only if there are independent paths from each cluster to the host running the Steward If there is only one path between the two clusters you must prevent split brain by confirming manually via telephone or some messaging system with administrators at the remote site if a failure has occurred By default VCS global clusters fail over an application across cluster boundaries with administrator confirmation You can configure automatic failover by setting the ClusterFailOverPolicy attribute to Auto If you start a service group on a remote cl
400. lling VCS behavior Controlling VCS behavior at the resource level the resource is faulted and VCS takes actions based on the service group configuration Resource Online Resource online TC no cleared Resource Unable to Offline resnotofi Trigger Clean Unexpected Offline Resource Online Admin_Wait resadminwaitTrigger YES Controlling VCS behavior 377 Controlling VCS behavior at the resource level Resource Online Admin_Wait resadminwait Trigger Resource online CMT Resource not cleared Going Offline Wait Clean Monitor Hung Resource Unable to Offline resnotoff Trigger 378 Controlling VCS behavior Controlling VCS behavior at the resource level VCS behavior when a resource fails to come online In the following example the agent framework invokes the Online entry point for an offline resource The resource state changes to WAITING TO ONLINE If the Online entry point times out VCS examines the value of the ManageFaults attribute If ManageFaults is set to NONE the resource state changes to OFFLINE ADMIN_WAIT If ManageFaults is set to ALL VCS calls the Clean entry point with the CleanReason set to Online Hung If the Online entry point does not time out VCS invokes the Monitor entry point The Monitor routine returns an exit code of 110 if the resource is online Otherwise the Monitor routine returns an exit code of 100 VCS examines the value of the OnlineWait
401. llowing command hasnap include add del list sys hostname files directories add Adds the specified files or directories to the include file list del Removes the specified files or directories from the include file list files Files or directories to be added or removed directories Administering the cluster from the command line 249 Managing VCS configuration files Examples The following command displays the list of files or directories to be included in new snapshots on each node of the cluster hasnap include list The following command adds the file opt VRTSweb conf vrtsweb xml to the include list on host1 which results in this file being included in the snapshot the next time the hasnap backup command is run hasnap include add opt VRTSweb conf vrtsweb xml The following command removes the file opt VRTSweb conf vrtsweb xml from the include list on host1 hasnap include del sys host1 opt VRTSweb conf vrtsweb xml Excluding files from snapshots Use the hasnap exclude command to configure the list of files or directories that should not be included in new snapshots To exclude files from snapshots Run the following command hasnap exclude add del list sys hostname files directories add Adds the specified files or directories to the exclude file list del Removes the specified files or directories from the exclude file list files Files or directories to be added or removed dire
402. llowing events occur the resource enters the admin_wait state Event Resource state The offline entry point did not complete within the ONLINE ADMIN_WAIT expected time The offline entry point was ineffective ONLINE ADMIN_WAIT The online entry point did not complete within the OFFLINE ADMIN_WAIT expected time The online entry point was ineffective OFFLINE ADMIN_WAIT The resource was taken offline unexpectedly OFFLINE ADMIN_WAIT For the online resource the monitor entry point ONLINE MONITOR_TIM consistently failed to complete within the expected EDOUT ADMIN_WAIT time Controlling VCS behavior 369 Controlling VCS behavior at the service group level Clearing resources in the ADMIN_WAIT state When VCS sets a resource in the ADMIN_WAIT state it invokes the resadminwait trigger according to the reason the resource entered the state See resadminwait event trigger on page 452 To clear a resource 1 Take the necessary actions outside VCS to bring all resources into the required state Verify that resources are in the required state by issuing the command hagrp clearadminwait group sys system This command clears the ADMIN_WaAIT state for all resources If VCS continues to detect resources that are not in the required state it resets the resources to the ADMIN_WAIT state If resources continue in the ADMIN_WaIT state repeat step 1 and step 2 or issue the following command to stop VCS from setting the re
403. lobal groups See Administering global service groups on page 512 Replication during cross cluster failover is managed by VCS agents You can configure a resource of type DNS to perform a canonical name update if cross cluster failover spans subnets See DNS agent on page 466 464 Connecting clusters Creating global clusters VCS global clusters The building blocks Global cluster management VCS enables you to perform operations online offline switch on global service groups from any system in any cluster using the VCS command line interface the Java Console or the Web Console You must log on with adequate privileges for cluster operations See User privileges in global clusters on page 64 You can bring service groups online or switch them to any system in any cluster If you do not specify a target system VCS uses the FailOverPolicy to determine the system See Defining failover policies on page 366 Management of remote cluster objects is aided by inter cluster communication enabled by the wide area connector wac process Wide area connector process The wide area connector wac is a failover Application resource that ensures communication between clusters App App App App App App Group Group Group Group Group Group ae a a gt D Lene Cluster 1 Cluster 2 The wac process runs on one system in each
404. luster in the list The Cluster Management Console can then connect to those clusters and update the lists In the Remote Cluster Configuration dialog box verify the required information for the remaining remote clusters and then click Next To change authentication information click a cluster name under Existing Clusters and then enter the authentication information in the fields to the right The requisite information in this dialog box varies depending upon whether or not the cluster is secure uses an authentication broker Click No if you want the operation to be completed only if the wizard can connect to all selected clusters Click Next 496 Administering global clusters from the Cluster Management Console Administering global service groups 9 Click Finish Administering global clusters from the Cluster Management Console 497 Administering global service groups Bringing a remote service group online Manually put a remote service group into a responsive functioning state You must have the role of cluster administrator or service group administrator to bring a remote service group online You can bring a remote service group online on a specific system or you can bring the service group online anywhere in the remote cluster If you select the Anywhere option the service group is brought online on the first available system in the remote cluster To bring a remote service group online 1 Inthe Cluster Groups
405. luster into a Replicated Volume Group RVG and creating the VVR Secondary on hosts in the new cluster located in your remote site Running the GCO configuration wizard If you are upgrading from a single cluster setup to a multi cluster setup run the GCO Configuration wizard to create or update the ClusterService group The wizard verifies your configuration and validates it for a global cluster setup You must have adequate licenses installed on all nodes in the cluster See Installing a VCS license on page 235 If you just added a license make sure to update the license on all nodes by running the following commands hasys updatelic all haclus updatelic If you use Cluster Manager Java Console you may need to restart the console for the licensing changes to take effect To run the GCO configuration wizard 1 Start the GCO Configuration wizard opt VRTSvcs bin gcoconfig 2 The wizard discovers the NIC devices on the local system and prompts you to enter the device to be used for the global cluster Specify the name of the device and press Enter 3 If you do not have NIC resources in your configuration the wizard asks you whether the specified NIC will be the public NIC used by all systems Enter y if it is the public NIC otherwise enter n If you entered n the wizard prompts you to enter the names of NICs on all systems 4 Enter the virtual IP to be used for the global cluster 5 If you do not have IP resources
406. luster name and click Add Delete Remote Clusters Review the required information for the Remote Cluster Configuration Wizard and click Next In the Wizard Options dialog box m Click Delete Cluster m Click Next In the Delete Cluster dialog box m Click the name of the remote cluster to delete m Click Next Administering global clusters from Cluster Manager Java console Deleting a remote cluster 5 Review the connection details for each cluster Click the Configure icon to review the remote cluster information for each cluster T Remote Cluster Configuration Wizard x Remote cluster information Enter the connection details for each cluster Click the Configure button and enter the Following details For each cluster The IP address orhostname of a node in the cluster User name and password for an adminstrator to the cluster Existing cluster configuration details Remote Cluster Host Name IP Address Username Configure ed NCSHP1O_71 Wweshp10 root WWCSHP11_72 i i lt Back Cancel Hep If the cluster is not running in secure mode Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system Verify the port number Enter the user name Enter the password Click OK If the cluster is running in secure mode Enter the IP address of the remote cluster the IP address of a cluster sy
407. ly one 344 About communications membership and data protection in the cluster About membership arbitration winner The first system to issue a preempt and abort command will win and eject the key of the other system When the second system issues a preempt and abort command it can not perform the key eject because it is no longer a registered system with a valid key m Ifthe preempt and abort command returns success that system has won the race for that coordinator disk Each system will repeat this race to all the coordinator disks The race is won by and control is attained by the system that ejects the other system s registration keys from a majority of the coordinator disks m On the system that wins the race the vxfen module informs all the systems that it was racing on behalf of that it won the race and that subcluster is still valid This information is passed back to GAB m On the system s that do not win the race the vxfen module will trigger a system panic The other systems in this subcluster will note the panic determine they lost control of the coordinator disks and also panic and restart m Upon restart the systems will attempt to seed into the cluster m Ifthe systems that restart can exchange heartbeat with the number of cluster systems declared in etc gabtab they will automatically seed and continue to join the cluster Their keys will be replaced on the coordinator disks This case will only happen if
408. m Type dependent Attributes that apply to a particular resource type These attributes appear in the type definition file types cf for the agent Examples The MountPath attribute applies only to the Mount resource type The Address attribute applies only to the IP resource type Attributes defined in the file types cf apply to all resources of a particular resource type Defining these attributes in the main cf file overrides the values in the types cf file for a specific resource For example setting StartVolumes 1 for the DiskGroup types cf defaults StartVolumes to True for all DiskGroup resources Setting the value in main cf overrides the value on a per resource basis m Static These attributes apply for every resource of a particular type These attributes are prefixed with the term static and are not included in the resource s argument list You can override some static attributes and assign them resource specific values See Overriding resource type static attributes on page 282 Attributes that define a specific resource Some of these attributes are type independent For example you can configure the Critical attribute for any resource Some resource attributes are type dependent For example the Address attribute defines the IP address associated with the IP resource These attributes are defined in the main cf file 54 VCS configuration concepts About VCS attributes Attribute scope across systems gl
409. m based on the requirements of service group workload management and existing group dependencies If bringing a parallel group online HAD brings the group online on each system designated in the SystemList attribute To take a service group offline across clusters hagrp offline force ifprobed service group sys system clus cluster localclus The option clus takes offline the service group on the system designated in the cluster To take a service group offline anywhere hagrp offline ifprobed service group any clus cluster localclus The option any specifies that HAD takes a failover group offline on the system on which it is online For a parallel group HAD takes the group offline on each system on which the group is online HAD adheres to the existing group dependencies when taking groups offline To switch a service group across clusters hagrp switch service group to system clus cluster localclus The option clus identifies the cluster to which the service group will be switched The service group is brought online on the system specified by the to system argument If a system is not specified the service group may be switched to any node within the specified cluster To switch a service group anywhere hagrp switch service group clus cluster The option clus identifies the cluster to which the service group will be switched HAD then selects the target system on which to switch the service group
410. m triggertype 0 or internal triggertype 1 If 0 the trigger is invoked from opt VRTSvcs bin triggers cpuusage If 1 the system reboots by invoking the trigger from opt VRTSvcs bin internal_triggers cpuusage system represents the name of the system cpu_usage represents the percentage of CPU utilization on the system Set following values in the system s CPUUsageMonitoring attribute Enabled 1 ActionTimeLimit Non zero value representing time in seconds ActionThreshold Non zero value representing CPU percentage utilization Action CUSTOM or REBOOT CUSTOM Invokes trigger from opt VRTSvcs bin triggers cpuusage REBOOT invokes trigger from opt VRTSvcs bin internal_triggers cpuusage and the system reboots Set one of the following values in CPUUsageMonitoring system attribute to 0 for the system ActionTimeLimit 0 ActionThreshold 0 448 VCS event triggers List of event triggers injeopardy event trigger Description Invoked when a system is in jeopardy Specifically this trigger is invoked Usage when a system has only one remaining link to the cluster and that link is a network link LLT This event is a considered critical because if the system loses the remaining network link VCS does not fail over the service groups that were online on the system Use this trigger to notify the administrator of the critical event The administrator can then take appropriate action to ensure that the syste
411. m Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Availability gt Disable Resources for Service Group 2 Click the service group 3 Click Apply Administering the cluster from Cluster Manager Java console 211 Administering resources Clearing a resource Clear a resource to remove a fault and make the resource available to go online A resource fault can occur in a variety of situations such as a power failure or a faulty configuration To clear a resource from Cluster Explorer 1 Inthe Service Groups tab of the configuration tree right click the resource 2 Click Clear Fault and click the system from the menu Click Auto instead of a specific system to clear the fault on all systems where the fault occurred To clear a resource from Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Availability gt Clear Resource 2 Click the resource To clear the fault on all systems listed in the Systems box proceed to step 5 To clear the fault on a specific system proceed to step 3 3 Select the Per System check box 4 Click the system on which to clear the resource 5 Click Apply 212 Administering the cluster from Cluster Manager Java console Administering resources Linking resources Use Cluster Explorer or Command Center to link resources in a service group To link resources from Cluster Explorer 1 Inthe configura
412. m has at least two links to the cluster This event trigger is non configurable injeopardy triggertype system system_state m triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 m system represents the name of the system m system_state represents the value of the State attribute VCS event triggers 449 List of event triggers loadwarning event trigger Description Invoked when a system becomes overloaded because the load of the system s Usage online groups exceeds the system s LoadWarningLevel attribute for an interval exceeding the LoadTimeThreshold attribute For example say the Capacity is 150 the LoadWarningLevel is 80 and the LoadTimeThreshold is 300 Also the sum of the Load attribute for all online groups on the system is 135 Because the LoadWarningLevel is 80 safe load is 0 80 150 120 Actual system load is 135 If system load stays above 120 for more than 300 seconds the LoadWarningLevel trigger is invoked Use this trigger to notify the administrator of the critical event The administrator can then switch some service groups to another system ensuring that no one system is overloaded This event trigger is non configurable loadwarning triggertype system available_capacity triggertype represents whether trigger is custom triggertype 0 or internal triggertype 1 For this trigger triggertype 0 m syst
413. m that caused the concurrency violation Specifically it takes the service group offline on the system where the trigger was invoked Note that this trigger applies to failover groups only The default trigger takes the service group offline on the system that caused the concurrency violation This event trigger is non configurable violation system service_group m system represents the name of the system m service group represents the name of the service group that was fully or partially online Section Multi cluster configurations m Chapter 15 Connecting clusters Creating global clusters on page 461 m Chapter 16 Administering global clusters from the Cluster Management Console on page 487 m Chapter 17 Administering global clusters from Cluster Manager Java console on page 503 m Chapter 18 Administering global clusters from the command line on page 521 m Chapter 19 Setting up replicated data clusters on page 535 460 Multi cluster configurations Chapter 1 D Connecting clusters Creating global clusters How VCS global clusters work VCS global clusters The building blocks Prerequisites for global clusters Setting up a global cluster When a cluster faults Setting up a fire drill 462 Connecting clusters Creating global clusters How VCS global clusters work How VCS global clusters work Local clustering provides local failover for each site or buildi
414. mand hamsg any sev C E W N I otype VCS RES GRP SYS AGT oname object name msgid message ID path path_name lang language LDF any sev otype oname msgid path lang Administering the cluster from the command line 259 Querying VCS Specifies hamsg return messages matching any of the specified query options Specifies hamsg return messages matching the specified message severity Critical Error Warning Notice or Information Specifies hamsg return messages matching the specified object type m VCS general VCS messages m RES resource m GRP service group m SYS system m AGT agent Specifies hamsg return messages matching the specified object name Specifies hamsg return messages matching the specified message ID Specifies where hamsg looks for the specified LDF If not specified hamsg looks for files in the default directory var VRTSvcs ldf Specifies the language in which to display messages For example the value en specifies English and ja specifies Japanese 260 Administering the cluster from the command line Querying VCS Using conditional statements to query VCS objects Some query commands include an option for conditional statements Conditional statements take three forms Attribute Value the attribute equals the value Attribute Value the attribute does not equal the value Attribute Value the value is the prefix of the attribute for example a query
415. matically fail over to another cluster if it is unable to fail over within the cluster m Click Next 4 Enter or review the connection details for each cluster Global Group Configuration Wizard Remote cluster information Enter the connection details for each cluster Click the Configure button and enter the folowing details For each cluster The cluster address of the cluster or the IP address hastname of a node in the cluster User name and password for an adminstrator to the cluster 3 Existing cluster configuration details te Remote Cluster Host Name IP Address Username Configure i WCSHPL1_72 vcshp11 lt Back cancel Hep Click the Configure icon to review the remote cluster information for each cluster If the cluster is not running in secure mode m Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system m Verify the port number m Enter the user name and pas m Click OK sword Repeat these steps for each cluster in the global environment If the cluster is running in secure mode m Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system m Verify the port number 514 Administering global clusters from Cluster Manager Java console Administering global service groups m Choose to connect to the remote clust
416. me of a cluster system m The port Verify the port number The default is 14141 m Authentication credentials Choose to connect to the remote cluster with the credentials used for the current cluster connection or enter new credentials You must specify the user name password domain and domain type If you have connected to the remote cluster before using this wizard you can use the credentials from the previous connection Click Finish The cluster icon changes to indicate that the cluster is a global cluster Deleting a remote cluster The Cluster Management Console enables you to delete a remote cluster This operation involves the following tasks Taking the wac resource in the ClusterService group offline in the cluster to be removed from the global environment For example to delete cluster C2 from a global environment containing C1 and C2 log in to C2 and take the wac resource offline Removing the name of the specified cluster C2 from the cluster lists of all global groups The Cluster Management Console updates the cluster lists for heartbeats Log in to the local cluster C1 to complete this task using the Global Groups Wizard 492 Administering global clusters from the Cluster Management Console Administering global clusters Removing the cluster C2 from the local cluster C1 using the Cluster Summary view on the local cluster C1 Note You cannot delete a remote cluster if the cluster is part of a clu
417. mension integer scalar m Default 0 UserStrLocal user defined Use this attribute for any purpose It is not used by VCS m Type and dimension string scalar m Default System attributes VCS attributes 641 System attributes Table D 4 lists the system attributes Table D 4 System attributes AgentsStopped system use only This attribute is set to 1 on a system when all agents running on the system are stopped m Type and dimension integer scalar m Default Not applicable AvailableCapacity system use only Indicates system s available capacity when trigger is fired If this value is negative the argument contains the prefix percentage sign for example 4 m Type and dimension integer scalar m Default Not applicable Capacity user defined Value expressing total system load capacity This value is relative to other systems in the cluster and does not reflect any real value associated with a particular system For example the administrator may assign a value of 200 to a 16 processor machine and 100 to an 8 processor machine m Type and dimension integer scalar m Default 100 ConfigBlockCount system use only Number of 512 byte blocks in configuration when the system joined the cluster m Type and dimension integer scalar m Default Not applicable ConfigCheckSum system use only Sixteen bit checksum of configuration identifying when
418. modify the default values of the attributes Otherwise proceed to step 12 to accept the default values and complete the configuration Modify the values of the attributes if necessary Configuration Wizard DiskGroup Name of disk group configured with VERITAS Volume Manager vNFS_Mount_1 vNFS_Mount2_1 vNFS_NFS_1 vNFS_NIC_1 HR vNFS_sharel_1 Pontoronly vNFS_Share2_1 G vNFS_Volume1_1 vNFS_Volume2_1 onitorReservat PanicSystemOnD tartVolumes BtopVolumes fempUseFence Show All Attributes m Click the resource 198 Administering the cluster from Cluster Manager Java console Administering service groups Click the attribute to be modified Click the Edit icon at the end of the table row In the Edit Attribute dialog box enter the attribute values Click OK m Repeat the procedure for each resource and attribute 12 Click Finish Administering the cluster from Cluster Manager Java console 199 Administering resources Administering resources Use the Java Console to administer resources in the cluster Use the console to add and delete bring online and take offline probe enable and disable clear and link and unlink resources You can also import resource types to the configuration Adding a resource The Java Console provides several ways to add a resource to a service group Use Cluster Explorer or Command Center to perform this task To add a resource from Cluster Explorer
419. monitor immediately after the offline entry point finishes so the first monitor detects the resource as offline When a Service group comes online The time it takes to bring a service group online depends on the number of resources in the service group the service group dependency structure and the time to bring the group s resources online For example if service group G1 has three resources R1 R2 and R3 where R1 depends on R2 and R2 depends on R3 VCS first onlines R3 When R3 is online VCS onlines R2 When R2 is online VCS onlines R1 The time it takes to online G1 equals the time it takes to bring all resources online However if R1 depends on both R2 and R3 but there was no dependency between them the online operation of R2 and R3 is started in parallel When both are online R1 is brought online The time it takes to online the group is Max the time to online R2 and R3 plus the time to online R1 Typically broader service group trees allow more parallel operations and can be brought online faster More complex service group trees do not allow much parallelism and serializes the group online operation When a Service group goes offline Taking service groups offline works from the top down as opposed to the online operation which works from the bottom up The time it takes to offline a service group depends on the number of resources in the service group and the time to offline the group s resources For example if serv
420. mote cluster The RVGPrimary agent ensures that VVR volumes are made writable and the DNS agent ensures that name services are resolved to the remote site The application can be started at the remote site Declaring the type of failure If a disaster disables all processing power in your primary data center heartbeats from the failover site to the primary data center fail VCS sends an alert signalling cluster failure If you choose to take action on this failure VCS prompts you to declare the type of failure You can choose one of the following options to declare the failure Connecting clusters Creating global clusters 481 Switching the service group back to the primary Disaster implying permanent loss of the primary data center Outage implying the primary may return to its current form in some time Disconnect implying a split brain condition both clusters are up but the link between them is broken Replica implying that data on the takeover target has been made consistent from a backup source and that the RVGPrimary can initiate a takeover when the service group is brought online This option applies to VVR environments only You can select the groups to be failed over to the local cluster in which case VCS brings the selected groups online on a node based on the group s FailOverPolicy attribute It also marks the groups as being OFFLINE in the other cluster If you do not select any service groups to fail over VCS ta
421. mp BBO Commands eee Q Link Resources E Configuration File C Custer Objects newgroup E F Attributes 5 amp Dependencies Child Resources Pe Link Service Groups Unlink Resources amp Unlink Service Groups 2 Operations j ink resourcel_ng2 TestRes 4 Clicka child resource 5 Click Apply 214 Administering the cluster from Cluster Manager Java console Administering resources Unlinking resources Use Cluster Explorer or Command Center to unlink resources in a service group To unlink resources from Cluster Explorer 6 From the configuration tree click the Service Groups tab Click the service group to which the resources belong In the view panel click the Resources tab In the Resources View right click the link between the resources Click Unlink from the menu E VCSHP10_71_0214a L Jorx a Link R Auto arrange P Ja Resource View newgroup2 on all systems ae Dependency D 1 J newgro veshp71 veshp10 In the Question dialog box click Yes to delete the link To unlink resources from Command Center 1 4 5 Administering the cluster from Cluster Manager Java console 215 Administering resources In the Command Center configuration tree expand Commands gt Configuration gt Dependencies gt Unlink Resources Click the service group that contains the linked resources Click the parent resour
422. ms m Select the user in whose context the programs will run Click Discover Users if some users were added after starting the wizard m Click Next Configuring applications and resources in VCS 305 Configuring application service groups Specify information about how the application will be monitored VCS Application Configuration Wizard Monitor Options Specify at least the Monitor Program the Pid Files or the Monitor Processes to monitor the application Monitor Program onestzone rootiopApplicationTestiapp_test_monitor Monitor Program Parameters Pid Files Monitor Processes 2 More Information lt Back Next gt Cancel Specify at least one of the MonitorProgram Pid Files or MonitorProcesses attributes You can specify some or all of these m Specify the complete path of the monitor program with parameters if any You can browse to locate files m Click the corresponding or buttons to add or remove Pid files or monitor processes m Click the corresponding button to modify a selected file or process m Click Next If you are configuring an application running in a non global zone proceed to step 12 on page 301 Otherwise proceed to the next step 306 Configuring applications and resources in VCS Configuring application service groups 10 Configure the Mount resources for the applications Mount Configuration Configure the Mount resources
423. must also specify a password for the new user and confirm it In the Assign Privileges dialog box specify the following options and then click OK The role The available roles for a cluster user are Cluster Administrator Cluster Operator Group Administrator or Group Operator You can select more than one but be aware that some option combinations are redundant The objects to which the role grants authority To begin this procedure you selected a single cluster from a Clusters Listing table If you select a role of Cluster Administrator or Cluster Operator you have already specified the object to which the user s role applies You are not required to specify anything more so the service group selection boxes are unavailable If you select Group Administrator or Group Operator you must select at least one service group To select a service group click a service group name under Available Groups and then click the right arrow button To reject a service group click a name under Selected Groups and then click the left arrow button to return it to Available Groups Note This procedure does not validate the user account For a cluster that is configured in secure mode the account is assumed to exist on the domain For a cluster that is not configured in secure mode the account is assumed to exist on the cluster To delete a cluster user 1 In the Cluster Users view in the Delete column of the Users table click the X
424. must configure a VCS user with the same name and privileges in each cluster See User privileges in global clusters on page 64 Configured applications Applications to be configured as global groups must be configured to represent each other in their respective clusters The multiple application groups of a global group must have the same name in each cluster The individual resources of the groups can be different For example one group might have a MultiNIC resource or more Mount type resources Clients redirected to the remote cluster in case of a wide area failover must be presented with the same application they saw in the primary cluster However the resources that make up a global group must represent the same application from the point of the client as its peer global group in the other cluster Clients redirected to a remote cluster should not be aware that a cross cluster failover occurred except for some downtime while the administrator initiates or confirms the failover 470 Connecting clusters Creating global clusters Prerequisites for global clusters Wide area heartbeats There must be at least one wide area heartbeat going from each cluster to every other cluster VCS starts communicating with a cluster only after the heartbeat reports that the cluster is alive VCS uses the ICMP ping by default the infrastructure for which is bundled with the product VCS configures the ICMP heartbeat if you use Cluster Manag
425. n integer scalar m Default 0 OperatorGroups user defined List of operating system user groups that have Operator privileges on the service group This attribute is valid only in secure clusters m Type and dimension string keylist m Default Operators user defined List of VCS users with privileges to operate the group A Group Operator can only perform online offline and temporary freeze unfreeze operations pertaining to a specific group See About the VCS user privilege model on page 61 m Type and dimension string keylist m Default Table D 3 VCS attributes 635 Service group attributes Service group attributes Parallel user defined Indicates if service group is failover 0 parallel 1 or hybrid 2 m Type and dimension integer scalar m Default 0 PathCount system use only Number of resources in path not yet taken offline When this number drops to zero the engine may take the entire service group offline if critical fault has occurred m Type and dimension integer scalar m Default Not applicable PreOnline user defined Indicates that the VCS engine should not online a service group in response to a manual group online group autostart or group failover The engine should instead call a user defined script that checks for external conditions before bringing the group online m Type and dimension boolean scalar m Default 0 Pr
426. n the ClusterService group offline on the cluster that will be removed from the global environment For example to delete cluster C2 from a global environment containing C1 and C2 log on to C2 and take the wac resource offline Removing the name of the specified cluster C2 from the cluster lists of the other global groups using the Global Group Configuration Wizard Note that the Remote Cluster Configuration Wizard in Cluster Explorer updates the cluster lists for heartbeats Log on to the local cluster C1 to complete this task before using the Global Group Configuration Wizard Deleting the cluster C2 from the local cluster C1 through the Remote Cluster Configuration Wizard Note You cannot delete a remote cluster if the cluster is part of a cluster list for global service groups or global heartbeats or if the cluster is in the RUNNING BUILD INQUIRY EXITING Or TRANSITIONING states To take the wac resource offline 1 3 From Cluster Monitor log on to the cluster that will be deleted from the global cluster environment In the Service Groups tab of the Cluster Explorer configuration tree right click the wac resource under the Application type in the ClusterService group or Click the ClusterService group in the configuration tree click the Resources tab and right click the wac resource in the view panel Click Offline and click the appropriate system from the menu To remove a cluster from a cluster lis
427. n the Tools menu or On the Cluster Explorer toolbar click Launch Notifier Resource Configuration Wizard Remote Group Resource Configuration Wizard A RemoteGroup resource enables you to manage or monitor remote service groups from a local cluster For each service group running in a remote cluster you can create a corresponding RemoteGroup resource in the local cluster See Adding a RemoteGroup resource from the Java Console on page 202 To access Remote Group Resource Configuration Wizard From Cluster Explorer click Remote Group Resource Wizard on the Tools menu or On the Cluster Explorer toolbar click Configure Remote Group Resource Wizard Cluster query Administering the cluster from Cluster Manager Java console 165 Accessing additional features of the Java Console Use Cluster Query to run SQL like queries from Cluster Explorer VCS objects that can be queried include service groups systems resources and resource types Some queries can be customized including searching for the system s online group count and specific resource attributes See Querying the cluster configuration on page 224 Guster Query EL Query Select Service Group where state a f a Onine xl Query Result At Feb 20 2006 3 05 42 PM Service Group State vcshp10 Online veshp71 Offline veshp10 Online veshp71 Offline veshp10 Online veshp71 Offline cmc Clust
428. name and password of the administrator for each cluster in the configuration m The user name and password of the administrator for the cluster being added to the configuration Note Symantec does not support adding a cluster that is already part of a global cluster environment To merge the clusters of one global cluster environment for example cluster A and cluster B with the clusters of another global environment for example cluster C and cluster D separate cluster C and cluster D into standalone clusters and add them one by one to the environment containing cluster A and cluster B To add a remote cluster to a global cluster environment in Cluster Explorer 1 From Cluster Explorer click Add Delete Remote Cluster on the Edit menu or From the Cluster Explorer configuration tree right click the cluster name and click Add Delete Remote Clusters 2 Review the required information for the Remote Cluster Configuration Wizard and click Next 3 Inthe Wizard Options dialog box m Click Add Cluster m Click Next 4 Enter the details of the new cluster 506 Administering global clusters from Cluster Manager Java console Adding a remote cluster If the cluster is not running in secure mode T Remote Cluster Configuration Wizard x New Cluster Details Enter the details of the new cluster to be added The IP address can be the cluster address or the host IP of a node in the new cluster Host Name IP address
429. nd save and close a configuration VCS Simulator enables you to administer the configuration on the local system while VCS is offline Opening a cluster configuration Use Cluster Explorer or Command Center to open or make changes to the VCS configuration To open a configuration from Cluster Explorer On the File menu click Open Configuration or Click Open Configuration on the Cluster Explorer toolbar To open a configuration from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Configuration File gt Open Configuration 2 Click Apply Saving a cluster configuration After updating the VCS configuration use Cluster Explorer or Command Center to save the latest configuration to disk while maintaining the configuration state in read write mode To save a configuration from Cluster Explorer On the File menu click Save Configuration or Click Save Configuration on the Cluster Explorer toolbar To save a configuration from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Configuration File gt Save Configuration 2 Click Apply Administering the cluster from Cluster Manager Java console 221 Executing commands Saving and closing a cluster configuration After updating the VCS configuration use Cluster Explorer or Command Center to save the latest configuration to disk and close or change the configuration s
430. nd delete heartbeats in a global cluster environment Icmp heartbeats send Icmp packets simultaneously to all IP addresses IcmpS heartbeats send individual Icmp packets to IP addresses in serial order Global clustering requires a minimum of one heartbeat between clusters the Icmp heartbeat is added when the cluster is added to the environment You can add additional heartbeats as a precautionary measure Adding a global heartbeat To add a cluster heartbeat from Cluster Explorer 1 Click Configure Heartbeats on the Edit menu 2 Inthe Heartbeat Configuration dialog box af Heartbeat Configuration New Heartbeat flush C Existing Heartbeat im C ds Remote Cluster Arguments Configure m Enter the name of the heartbeat m Select the check box next to the name of the cluster to add it to the cluster list for the heartbeat m Click the icon in the Configure column to open the Heartbeat Settings dialog box 518 Administering global clusters from Cluster Manager Java console Administering global heartbeats m Specify the value of the Arguments attribute and various timeout and interval fields Click to add an argument value click to delete it LB Heartbeat Settings Heartbeat type clus_hb Cluster Name C2 Arguments 5 10 212 88 189 Are You Alive settings Intervalf6o Timeout 300 Retry Limit 3 Timeout settings Init Timeout 30 Start Timeout 30 8
431. ne To update the EngineClass haclus modify EngineClass value For example to set the EngineClass attribute to RealTime haclus modify EngineClass RT To update the EnginePriority haclus modify EnginePriority value For example to set the EnginePriority to 20 haclus modify EnginePriority 20 To update the ProcessClass haclus modify ProcessClass value For example to set the ProcessClass to TimeSharing haclus modify ProcessClass TS To update the ProcessPriority haclus modify ProcessPriority value For example to set the ProcessPriority to 40 haclus modify ProcessPriority 40 Note For the attributes EngineClass and EnginePriority changes are effective immediately For ProcessClass and ProcessPriority changes become effective only for processes fired after the execution of the haclus command Administering the cluster from the command line 279 Administering clusters Initializing cluster attributes in the configuration file You may assign values for cluster attributes while configuring the cluster See Cluster Attributes on page 645 for a description of each attribute cited below Review the following sample configuration cluster vcs india EngineClass RT EnginePriority 20 ProcessClass TS ProcessPriority 40 280 Administering the cluster from the command line Enabling and disabling Security Services Enabling and disabling Security Services
432. ne This command is useful if you have simulated a fault of a persistent resource and want to simulate the fix Simulates a cluster fault Clears a simulated cluster fault 330 Predicting VCS behavior using VCS Simulator Administering VCS Simulator from the command line Command hasim getsimconfig cluster_name hasim hb hasim disablel10n hasim clus hasim sys hasim grp hasim res hasim type hasim conf hasim attr Description Retrieves information about VCS Simulator ports Equivalent to standard hahb command Disables localized inputs for attribute values Use this option when simulating UNIX configurations on Windows systems Equivalent to standard haclus command Equivalent to standard hasys command Equivalent to standard hagrp command Equivalent to standard hares command Equivalent to standard hat ype command Equivalent to standard haconf command Equivalent to standard haat tr command VCS communication and operations m Chapter 10 About communications membership and data protection in the cluster on page 333 m Chapter 11 Controlling VCS behavior on page 361 m Chapter 12 The role of service group dependencies on page 409 332 VCS communication and operations Chapter 1 0 About communications membership and data protection in the cluster About cluster communications About cluster membership About
433. nect network infrastructure usually one that routes all communication links through a single point of failure If all the cluster interconnect links fail it is possible for one cluster to separate into two subclusters each of which does not know about the other subcluster The two subclusters could each carry out recovery actions for the departed systems This is termed split brain In a split brain condition two systems could try to import the same storage and cause data corruption have an IP address up in two places or mistakenly run an application in two places at once Membership arbitration guarantees against such split brain conditions Components of membership arbitration The components of membership arbitration are the fencing module and the coordinator disks Fencing module Each system in the cluster runs a kernel module called vxfen or the fencing module This module is responsible for ensuring valid and current cluster membership on a membership change through the process of membership arbitration vxfen performs the following actions m Registers with the coordinator disks during normal operation m Races for control of the coordinator disks during membership changes Coordinator disks Coordinator disks are a number of special purpose disks that act together as a global lock device Racing for control of these disks is used to determine cluster membership Control is won by the system that gains control of a majori
434. nection to the remote cluster Local cluster has complete snapshot of the remote cluster indicating the remote cluster is in the RUNNING state Self explanatory A user log on has been recognized because a user logged on by Cluster Manager or because a haxxx command was invoked Severity Level Description Warning Information The agent has faulted on one node in the cluster VCS is restarting the agent 436 VCS event notification VCS events and traps Events and traps for resources Event Resource state is unknown Resource monitoring has timed out Resource is not going offline Health of cluster resource declined Resource went online by itself Resource has faulted Resource is being restarted by agent The health of cluster resource improved Resource monitor time has changed Severity Level Description Warning Warning Warning Warning Warning not for first probe Error Information Information Warning VCS cannot identify the state of the resource Monitoring mechanism for the resource has timed out VCS cannot take the resource offline Used by agents to give additional information on the state of a resource Health of the resource declined while it was online The resource was brought online on its own Self explanatory The agent is restarting the resource Used by agents to give extra information about state of resource Health of the
435. nfiguration 480 Connecting clusters Creating global clusters When a cluster faults The appgroup service group is now a global group and can be failed over between clusters vcspri Cluster Explorer admin Cluster Administrator se ioj xj File Edt View Tools Help 1 ha 2946 dA HOOTRBHAR a o fl Status Pk Remote Cluster Status Service Groups hy System Connectivity E Properties Remote Cluster Status Vie Bray Clusterservice Remote A Eg apparoup Hg appgroup_rep G All Clusters Status Cluster Name Cluster Address Status Heartbeat Status amp vespri Local Cluster 10 182 2 137 Running vessec 10 182 2 138 Running Temp ALIVE 4 All Global Groups Status Clusters gt vespri wessec Online Offline Global Groups v G apporoup vespri For remote cluster operations you must configure a VCS user with the same name and privileges in each cluster See User privileges in global clusters on page 64 When a cluster faults In the global cluster setup consider a case where the primary cluster suffers a failure The Oracle service group cannot fail over in the local cluster and must fail over globally to a node in another cluster In this situation VCS sends an alert indicating that the cluster is down An administrator can bring the group online in the re
436. nfiguration Wizard Enables you to configure resources to monitor a service group in a remote cluster Add Delete Remote Clusters Enables you to add and remove global clusters Configure Global Groups Enables you to convert a local service group to a global group and vice versa Query Enables you to search the cluster configuration according to filter criteria Virtual Fire Drill Enables you to check the DR readiness of your cluster by running a virtual fire drill Show Cluster Explorer Help Enables you to access online help Administering the cluster from Cluster Manager Java console 151 About Cluster Explorer Cluster Explorer configuration tree The Cluster Explorer configuration tree is a tabbed display of VCS objects m The Service Groups tab lists the service groups in the cluster Expand each service group to view the group s resource types and resources m The Systems tab lists the systems in the cluster m The Types tab lists the resource types in the cluster Cluster Explorer view panel The right pane of the Cluster Explorer includes a view panel that provides detailed information about the object selected in the configuration tree The information is presented in tabular or graphical format Use the tabs in the view panel to access a particular view The console enables you to tear off each view to appear in a separate window m Click any object in the configuration tree to access the Status
437. nfirm that you want to freeze the specified service groups Unfreezing a service group Return a frozen service group to normal failover behavior Unfreezing a service group restarts online and offline operations on the service group To unfreeze a service group 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the frozen service group that you want to unfreeze 2 Inthe Group Summary view in the Operations task panel click Unfreeze 3 Inthe Unfreeze Service Group dialog box click OK to confirm that you want to unfreeze the specified service group To unfreeze one or more service groups 1 Inthe Cluster Groups view in the Groups Listing table check the check box preceding the line item for each frozen service group that you want to unfreeze Select all service groups by checking the check box at the top 2 Inthe Operations task panel click Unfreeze 3 In the Unfreeze Service Group s dialog box click OK to confirm that you want to unfreeze the specified service groups Enabling a service group Enable a disabled service group so that it can be brought online A service group that is manually disabled during a maintenance procedure must first be enabled before it can be brought online To enable a service group 1 Inthe Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to enable 2 Inthe Group Summary view in the Operations task pane
438. ng But these configurations do not provide protection against large scale disasters such as major floods hurricanes and earthquakes that cause outages for an entire city or region The entire cluster could be affected by an outage In such situations VCS global clusters ensure data availability can by migrating applications to remote clusters located considerable distances apart Let us take the example of an Oracle database configured in a VCS global cluster Oracle is installed and configured in both clusters Oracle data is located on shared disks within each cluster and is replicated across clusters to ensure data concurrency The Oracle service group is online on a system in cluster A and is configured to fail over globally on clusters A and B Figure 15 1 Sample global cluster setup OG i cli Cluster A Netwe Wa Redirected Cluster B Application Failover Replicated Data Separate Storage Separate Storage VCS continuously monitors and communicates events between clusters Inter cluster communication ensures that the global cluster is aware of the state of global service group at all times In the event of a system or application failure VCS fails over the Oracle service group to another system in the same cluster If the entire cluster fails VCS fails over the service group to the remote cluster which is part of the global cluster VCS
439. ng VCS Simulator Administering VCS Simulator from the Java Console Creating a simulated cluster You can start a sample cluster configuration or create a new simulated cluster See Creating a simulated cluster on page 322 To create a simulated cluster 1 Inthe Simulator console click Add Cluster 2 Inthe Add Cluster dialog box f T Add Configuration xi a Add Cluster Cluster Name NEWCluster System Name NEWCluster_syst Port 145514 Platform W2K z J7 Enable Global Cluster Option Wac Port 1 OK Cancel Enter a name for the new cluster Accept the suggested system name or enter a new name for a system in the cluster Enter a unique port number for the simulated cluster Select the platform for the cluster nodes If the cluster will be part of a a global cluster configuration select the Enable Global Cluster Option check box and enter a unique port number for the wide area connector WAC process Click OK VCS creates a simulated one node cluster and creates a new directory for the cluster s configuration files VCS also creates a user called admin with Cluster Administrator privileges You can start the simulated cluster and administer it by launching the Java Console Deleting a cluster Predicting VCS behavior using VCS Simulator 323 Administering VCS Simulator from the Java Console Deleting a simulated cluster removes all files and directories associated with the cluster Bef
440. ng command hagrp link parent group child group gd category gd_location gd type parent_group Name of the parent group child_group Name of the child group gd_category category of group dependency online offline gd_location the scope of dependency local global remote gd_type type of group dependency soft firm hard Default is firm To unlink service groups Type the following command hagrp unlink parent group child group Administering agents Under normal conditions VCS agents are started and stopped automatically To start an agent Run the following command haagent start agent sys system To stop an agent Run the following command haagent stop agent force sys system The force option stops the agent even if the resources for the agent are online Use the force option when you want to upgrade an agent without taking its resources offline 268 Administering the cluster from the command line Administering resources Administering resources Adding resources Add resource to a service group or remove resources from a service group To add a resource Type the following command hares add resource resource type service group The resource name must be unique throughout the cluster The resource type must be defined in the configuration language The resource belongs to the group service_group About adding resources When you add a resource all non static attributes of the re
441. ng snapshots of 243 verifying 242 configuration files generating 46 main cf 46 restoring from snaphots 243 taking snapshots of 243 types cf 46 configuration language local and global attributes 54 configurations asymmetric 34 global cluster 44 N 1 38 N to 1 36 N to N 40 replicated data 43 shared nothing 42 shared storage replicated data 43 symmetric 35 ConfInterval attribute about 373 definition 622 ConnectorState attribute 649 coordinator disks 343 CounterInterval attribute 650 CPU binding of HAD 558 CPU usage how VCS monitors 559 CPUUsage attribute 642 CPUUsageMonitoring attribute 642 Critical attribute 615 CurrentCount attribute 630 CurrentLimits attribute 642 custom agents in zones 594 custom agents about 26 D Daemon Down Node Alive 352 DDNA 352 DeferAutoStart attribute 630 dependencies between resources 122 for resources 22 for service groups disability compliance in Java Console 137 DiskHbStatus attribute 642 DNS agent 466 dumping aconfiguration 243 DumpingMembership attribute 650 DynamicLoad attribute 643 E Enabled attribute for resources 615 for service groups 631 engine log format 564 location 564 EnginePriority attribute 650 enterprise agents about 26 entry points about 25 modifying for performance 548 environment variables 56 error messages agent log 564 at startup 567 engine log 564 message catalogs 565 Evacuate attribute 631 Evacuating attribute 631 event trigger
442. ng the group online or select the Any System check box m Select the No Preonline check box to bring the service group online without invoking the preonline trigger Click Show Command in the bottom left corner to view the command associated with the service group Click Hide Command to close the view of the command m Click OK Administering the cluster from Cluster Manager Java console 183 Administering service groups To bring a service group online from Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Controls gt Online Service Group or Click Bring service group online in the Command Center toolbar 2 Click the service group 3 For global groups select the cluster in which to bring the group online 4 Click the system on which to bring the group online or select the Any System check box 5 Click Apply 184 Administering the cluster from Cluster Manager Java console Administering service groups Taking a service group offline To take a service group offline from Cluster Explorer configuration tree 1 Inthe Service Groups tab of the configuration tree right click the service group or Click a cluster in the configuration tree click the Service Groups tab and right click the service group icon in the view panel 2 Click Offline and click the appropriate system from the menu Click All Systems to take the group offline on all systems To take a service
443. ng the service group online on the local system Click Close Configuring applications and resources in VCS 315 Testing resource failover using virtual fire drills Testing resource failover using virtual fire drills Configuring high availability for a database or an application requires several infrastructure and configuration settings on multiple systems However cluster environments are subject to change after the initial setup Administrators add disks create new diskgroups and volumes add new cluster nodes or new NICs to upgrade and maintain the infrastructure Keeping the cluster configuration updated with the changing infrastructure is critical Virtual fire drills detect discrepancies between the VCS configuration and the underlying infrastructure on a node discrepancies that might prevent a service group from going online on a specific node The virtual fire drill feature is supported with the following agents m Application m Diskgroup m Mount a IP m Process m NIC About virtual fire drills The virtual fire drill feature uses the Action entry point associated with the agent The Action entry points of the supported agents are updated to support the virtual fire drill functionality running infrastructure checks and fixing specific errors The infrastructure check verifies the resources defined in the VCS configuration file main cf have the required infrastructure to fail over on another node For example
444. nistering the cluster from the Cluster Management Console 75 Logging out of the Cluster Management Console Logging out of the Cluster Management Console Log out to end a Cluster Management Console session To log out of the console Onthe title bar of the Cluster Management Console click Logout 76 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console Overview of the Cluster Management Console The Cluster Management Console consists of several views A view is an HTML page that displays m Objects and information about objects in your high availability environment m Familiar mouse or keyboard accessible controls such as buttons menu items check boxes and text boxes that enable you to issue commands or specify options m Links to other views or task wizards Before you start your first Cluster Management Console session use this material to familiarize yourself with the default view navigation methods and icon representations used in the console Note If you are a user with a privilege level of guest or operator certain tabs or task links in the console may be unavailable An unavailable tab or task link indicates that the corresponding view the view to which the tab navigates or task link is restricted Users with the privilege level of administrator have access to all views Administering the cluster from the Cluster Management Console 77 Overview
445. nistratorGroups OperatorGroups or Guests attributes 4 Inthe Edit Attribute dialog box m Use the button to add an element m Click the newly added element and enter the name of the user group in the format group domain m Click OK 178 Administering the cluster from Cluster Manager Java console Administering service groups Administering service groups Use the Java Console to administer service groups in the cluster Use the console to add and delete bring online and take offline freeze and unfreeze link and unlink enable and disable autoenable switch and flush service groups You can also modify the system list for a service group Adding a service group The Java Console provides several ways to add a service group to the systems in a cluster Use Cluster Explorer Command Center or the Template View to perform this task To add a service group from Cluster Explorer 1 Onthe Edit menu click Add and click Service Group or Click Add Service Group in the Cluster Explorer toolbar 2 Enter the details of the service group T Add Service Group x a Add Service Group Service Group name Available Systems Systems for Service Group veshp10 System name Startup Priority vcshp71 Service Group Type Failover C Parallel C Hybrid Selected Template None El Show Command Enter the name of the service group m Inthe Available Systems box click the systems on which
446. note the command and option can be executed A dash indicates they cannot Agent Operations haagent Table A 1 lists agent operations and required privileges Table A 1 User privileges for agent operations Start agent Stop agent Display info Y Y Y List agents Y Y Y QS 8 KIRUNAS Attribute Operations haattr Table A 2 lists attribute operations and required privileges Table A 2 User privileges for attribute operations Add z 7 Y Change m z Vv default value Delete A Display Y Y Y Y Y VCS user privileges administration matrices 585 Administration matrices Cluster Operations haclus haconf Table A 3 lists cluster operations and required privileges Table A 3 User privileges for cluster operations Display Y Y Y Y Y Modify Y Add 7 Y Delete Y Declare Y Y View state or status Y Y Y Y Y Update license A Make configuration read Y Y write Save configuration Y Y Make configuration read Y z Y only 586 VCS user privileges administration matrices Administration matrices Table A 4 Add and delete Service group operations hagrp Table A 4 lists service group operations and required privileges User privileges for service group operations Link and unlink Clear Bring online Take offline View state Switch Freeze unfreez
447. nsion integer scalar Note This attribute can m Default 0 be overridden ScriptClass Indicates the scheduling class of the script processes for user defined example online created by the agent m Type and dimension string scalar m Default TS ScriptPriority Indicates the priority of the script processes created by the user defined agent m Type and dimension string scalar m Default 0 SourceFile File from which the configuration is read Always set to types cf Make sure the path exists on all nodes before user defined configuring this attribute m Type and dimension string scalar m Default types cf SupportedActions Valid action tokens for the resource type user defined m Type and dimension string vector m Default VCS attributes 627 Resource type attributes Table D 2 Resource type attributes ToleranceLimit After a resource goes online the number of times the monitor user defined entry point should return OFFLINE before declaring the resource FAULTED Note This attribute can be overridden A large value could delay detection of a genuinely faulted resource m Type and dimension integer scalar m Default 0 628 VCS attributes Service group attributes Service group attributes Table D 3 lists the service group attributes Table D 3 Service group attributes ActiveCount system use only Number of resources in a service group that
448. nsole Editing attributes In the Properties or Attributes View click the icon in the Edit column of the Key Attributes or Type Specific Attributes table In the Attributes View click the icon in the Edit column of the attribute In the Edit Attribute dialog box enter the changes to the attribute values To edit a scalar value Enter or click the value To edit a non scalar value Use the button to add an element Use the button to delete an element To change the attribute s scope Click the Global or Per System option To change the system for a local attribute Click the system from the menu Click OK To edit an attribute from Command Center 1 In the Command Center configuration tree expand Commands gt Configuration gt Attributes gt Modify vcs_object Attributes Click the VCS object from the menu In the attribute table click the icon in the Edit column of the attribute f VCSHP10_71_0214a Command Center admin Cluster Administrator Ele View Help nB amp S6S4 4S BBO FR commands S Be raii Modify Cluster Attributes 1 2 Configuration File AdministratorGroups C Custer Objects AdministratorGroups ENED Attributes Mc Modify Resource Attributes Modify System Attributes Global al Modify Resource Type Attri Global Scalar 2 i612 Dependencies Global Scalar VCSHP10_71_02 f Operations Global Scalar 10 212 98 128 2 Global Scalar Global Scalar
449. nsole display updates refreshes to reflect the latest data Update refresh mode settings and icons An update icon near the top right hand corner of the Cluster Management Console indicates the current update mode and the state of the information displayed in the console Clicking the update icon enables you to set the update mode The appearance of the update icon changes according to the following update modes m Disabled Provides no indication of available updates disables update notification Use the refresh function in your web browser to update the console information Figure 5 14 Disabled setting icon m Notify Only Indicates when data updates are available Icon changes to the Refresh Notification icon when data updates are available Figure 5 15 Notify Only setting icon Notify Only For the Notify Only setting when data is no longer current the Page is Stale icon appears This is the default setting Click this icon to update the console information Figure 5 16 Page is Stale icon Notify Only setting m Auto Refresh Automatically updates the console information when it is no longer current Figure 5 17 Auto Refresh setting icon PIG Auto Refresh 90 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console To change the refresh mode Click the update icon to cycle among the refresh modes You can select any mode except Page is Stale Administering th
450. nsport LLT Figure 10 1 illustrates the overall communications paths between two systems of the replicated state machine model Figure 10 1 Cluster communications with replicated state machine AGENT AGENT AGENT AGENT User HAD User Space HAD Space Kernel GAB GAB Kerne Space Space LLT LLT 336 About communications membership and data protection in the cluster About cluster communications Group Membership Services Atomic Broadcast GAB The Group Membership Services Atomic Broadcast protocol GAB is responsible for cluster membership and reliable cluster communications GAB has two major functions Cluster membership GAB maintains cluster membership by receiving input on the status of the heartbeat from each system via LLT When a system no longer receives heartbeats from a cluster peer LLT passes the heartbeat loss to GAB GAB marks the peer as DOWN and excludes it from the cluster In most configurations membership arbitration is used to prevent network partitions Cluster communications GAB s second function is reliable cluster communications GAB provides guaranteed delivery of messages to all cluster systems The Atomic Broadcast functionality is used by HAD to ensure that all systems within the cluster receive all configuration change messages or are rolled back to the previous state much like a data
451. ntec Web Server 673 Configuring logging for VRTSweb Configuring logging for VRTSweb You can configure the amount of logs that individual VRTSWeb components generate VRTSweb comprises the following components m Web server m Web applications m Other components You can set the logging threshold for each component separately The lower the threshold the more are the logs generated Symantec recommends setting log levels to lower values only for debugging Most of the logs are located at m var VRTSweb log for UNIX m VRTSWEB HOME log for Windows Individual Symantec Web consoles choose their own locations for their logs See the documentation for the specific Web console for more information Retrieving log levels Display the current settings for VRTSweb logging To retrieve log levels from the command line Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui log This returns the logging thresholds for various components and the limit and rollover count of various log files for VRTSweb To retrieve log levels from the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 The Logging table on the right side of the Configuration page lists the log levels for various components of the Web server The table does not display the limit and rollover count of various log files you must use the command line
452. nts the resource type associated with the agent Remove the files associated with the trigger from the VCS_HOME bin the trigger triggers directory 458 VCS event triggers List of event triggers unable_to_restart_had event trigger Description Usage violation event trigger Description Usage This event trigger is invoked by hashadow when hashadow cannot restart HAD on a system If HAD fails to restart after six attempts hashadow invokes the trigger on the system The default behavior of the trigger is to reboot the system However service groups previously running on the system are autodisabled when hashadow fails to restart HAD Before these service groups can be brought online elsewhere in the cluster you must autoenable them on the system To do so customize the unable_to_restart_had trigger to remotely execute the following command from any node in the cluster where VCS is running hagrp autoenable service_group sys system For example if hashadow fails to restart HAD on system1 and if group1 and group2 were online on that system a trigger customized in this manner would autoenable group1 and group2 on system1 before rebooting Autoenabling group1 and group2 on system1 enables these two service groups to come online on another system when the trigger reboots system1 This event trigger is non configurable unable_to_restart_had This trigger has no arguments This trigger is invoked only on the syste
453. obal and local attributes An attribute whose value applies to all systems is global in scope An attribute whose value applies on a per system basis is local in scope The at operator indicates the system to which a local value applies An example of local attributes can be found in the MultiNICA resource type where IP addresses and routing options are assigned per machine MultiNICA definition for Solaris MultiNICA mnic Device sysa leO 166 98 16 103 qfe3 Device sysb leO 166 98 16 104 qfe3 NetMask 255 255 255 0 ArpDelay 5 Options trailers RouteOptions sysa default 166 98 16 1 1 RouteOptions sysb default 166 98 16 1 1 166 98 16 103 166 98 16 104 Wool Attribute life temporary attributes You can define temporary attributes in the types cf file The values of temporary attributes remain in memory as long as the VCS engine HAD is running Values of temporary attributes are not available when HAD is restarted These attribute values are not stored in the main cf file Temporary attributes cannot be converted to permanent and vice versa When you save a configuration VCS saves temporary attributes and their default values in the file types cf The scope of these attributes can be local to a node or global across all nodes in the cluster Local attributes can be defined even when the node is not part of a cluster You can define and modify these attributes only while VCS is runnin
454. ode ID as defined in the file etc Ittab Type the following command hasys nodeid node ID To freeze a system prevent groups from being brought online or switched on the system Type the following command hasys freeze persistent evacuate system persistent Enables the freeze to be remembered when the cluster is rebooted Note that the cluster configuration must be in read write mode and must be saved to disk dumped to enable the freeze to be remembered evacuate Fails over the system s active service groups to another system in the cluster before the freeze is enabled To unfreeze a frozen system reenable online and switch of service groups Type the following command hasys unfreeze persistent system Administering clusters Retrieving version information Retrieve information about the version of VCS running on the system To retrieve information about the VCS version on the system Runone of the following commands had version The command retrieves information about the engine version the join version the build date and the PSTAMP Administering the cluster from the command line 277 Administering clusters had v The command retrieves information about the engine version Adding and removing systems This section provides an overview of tasks involved in adding and removing systems from a cluster For detailed instructions see the Veritas Cluster Server Installat
455. of a failover IP address to protect against network partitioning The check pings a service group s configured IP address to verify it is not already in use If it is the service group is not brought online A second check verifies the system is connected to its public and private networks If the system receives no response from a broadcast ping to the public network and a check of the private networks it determines the system is isolated and does not bring the service group online To enable the preonline IP check Move the preonline trigger script from the sample triggers directory into the triggers directory cp opt VRTSvcs bin sample triggers preonline ipc opt VRTSvcs bin triggers preonline Change the file permissions to make it executable Troubleshooting VCS startup This section includes error messages associated with starting VCS shown in bold text and provides descriptions of each error and the recommended action VCS 10622 local configuration missing VCS 10623 local configuration invalid The local configuration is invalid Recommended Action Start the VCS engine HAD on another system that has a valid configuration file The system with the configuration error pulls the valid configuration from the other system Another method is to correct the configuration file on the local system and force VCS to reread the configuration file If the file appears valid verify that is not an earlier versi
456. of a system in the SystemList attribute and the integer is the number of the zone Systems with the same zone number are members of the same zone If a service group faults on one system in a zone it is granted priority to fail over to another system within the same zone despite the policy granted by the FailOverPolicy attribute m Type and dimension integer association m Default none Tag user defined Identifies special purpose service groups created for specific VCS products m Type and dimension string scalar m Default Not applicable TargetCount system use only Indicates the number of target systems on which the service group should be brought online m Type and dimension integer scalar m Default Not applicable TFrozen user defined Indicates if service groups can be brought online on the system Groups cannot be brought online if the attribute value is 1 m Type and dimension boolean scalar m Default 0 not frozen ToQ system use only Indicates the node name to which the service is failing over This attribute is specified when service group failover is a direct consequence of the group event such as a resource fault within the group or a group switch m Type and dimension string association m Default Not applicable TriggerEvent system use only For internal use only m Type and dimension boolean scalar m Default Not applicable VCS attributes
457. of each service group The system list of a service group contains a list of all systems on which the service group is configured Click OK Administering the cluster from the Cluster Management Console Administering service groups Clearing a faulted service group Clear a service group to remove the resource faults within the group You must have the role of cluster administrator or service group administrator to clear a faulted service group This task makes the service group available to be brought online A resource fault in a service group may occur in several situations such as a power failure or faulty configuration To clear a faulted service group 1 Inthe Group Summary view in the Operations task panel click Clear Fault 2 Inthe Clear Faulted Group dialog box use the drop down menu to select the system on which to clear the service group To clear the group on all systems click All Systems 3 Click OK To clear one or more service groups 1 Inthe Cluster Groups view in the Groups Listing table check the check box preceding the line item for each service group that you want to clear Select all service groups by checking the check box at the top 2 Inthe Operations task panel click Clear Fault In the Clear Service Group s Faults dialog box use the appropriate drop down menu to select the systems on which you want to clear each service group Each drop down menu is populated using the system list of each servi
458. of the Cluster Management Console The Cluster Management Console uses the following icons to represent cluster objects Table 5 3 Object icons cay Cluster Be Group System Resource w Resource type General status indicators are applied to the basic object icons to create status icons Status icons indicate both the type and the state of the object The following are the status indicators Color icons indicate objects that are in a healthy running condition Gray icons indicate objects that are offline Icons with orange lines indicate objects that are partially online with only some member contained objects online Icons with red lines indicate faulted objects Icons with a question mark indicate objects whose status is unknown Icons with a red lock indicate objects in a frozen state Icons with a red circle and white cross bar indicate objects in an autodisabled state The online help appendixes contain a comprehensive list of the status icons that are used in the Cluster Management Console 87 88 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console Sorting the contents of a data table In some Cluster Management Console tables you can sort the rows by the data in acolumn Some columns are not sortable To sort rows in a table by data in a column 1 Inthe table locate a column you want to use to sort the data 2 Click the column heading The list is
459. oft firm or hard dependencies The type of dependency defines the rigidity of the link between parent and child groups A soft dependency means minimum constraints whereas a hard dependency means maximum constraints Soft dependency Firm dependency Specifies the minimum constraints while bringing parent and child groups online The only constraint is that the child group must be online before the parent group is brought online For example in an online local soft dependency an instance of the child group must be online on the same system before the parent group can come online Soft dependency provides the following flexibility m Ifthe child group faults VCS does not immediately take the parent offline If the child group cannot fail over the parent remains online m When both groups are online either group child or parent may be taken offline while the other remains online m Ifthe parent group faults the child group may remain online m When the link is created the child group need not be online if the parent is online However when both groups are online their online state must not conflict with the type of link Imposes more constraints when VCS brings the parent or child groups online or takes them offline In addition to the constraint that the child group must be online before the parent group is brought online the constraints include m Ifthe child group faults the parent is taken offline If the parent is
460. oing offline when a resource faults It also prevents the Clean function from being called on a resource fault You can freeze a service group when performing operations on its resources from outside VCS control This prevents VCS from taking actions on resources while your operations are on For example freeze a database group when using database controls to stop and start a database 368 Controlling VCS behavior Controlling VCS behavior at the service group level Controlling Clean behavior on resource faults The ManageFaults attribute specifies whether VCS calls the Clean entry point when a resource faults ManageFaults is a service group attribute you can configure each service group to operate as desired m Ifthe ManageFaults attribute is set to ALL VCS calls the Clean entry point when a resource faults m Ifthe ManageFaults attribute is set to NONE VCS takes no action ona resource fault it hangs the service group until administrative action can be taken VCS marks the resource state as ADMIN_WAIT and does not fail over the service group until the resource fault is removed and the ADMIN_WAIT state is cleared VCS calls the resadminwait trigger when a resource enters the ADMIN_WAIT state due to a resource fault if the ManageFaults attribute is set to NONE You can customize this trigger to provide notification about the fault See resadminwait event trigger on page 452 When ManageFaults is set to none and one of the fo
461. ole 217 Importing resource types Refreshing the Resourcelnfo attribute Refresh the ResourcelInfo attribute to view the latest values for that attribute To refresh the Resourcelnfo attribute 1 Inthe Service Groups tab of the configuration tree right click the resource 2 Click Refresh Resourcelnfo and click the system on which to refresh the attribute value Clearing the Resourcelnfo attribute Clear the Resourcelnfo attribute to reset all the parameters in this attribute To clear the parameters of the Resourcelnfo attribute 1 Inthe Service Groups tab of the configuration tree right click the resource 2 Click Clear Resourcelnfo and click the system on which to reset the attribute value Importing resource types The Java Console enables you to import resource types into your configuration main cf For example use this procedure to import the types cf for enterprise agents to your configuration You cannot import resource types that already exist in your configuration To import a resource type from Cluster Explorer 1 Onthe File menu click Import Types 2 Inthe Import Types dialog box m Click the file from which to import the resource type The dialog box displays the files on the system that Cluster Manager is connected to m Click Import Administering systems 218 Administering the cluster from Cluster Manager Java console Administering systems Use the Java Console to administer systems in the cl
462. ollowing command from any system in the cluster to force VCS to use the configuration file from the system specified by the variable system hasys force system To start VCS ona single node Type the following command to start an instance of VCS that does not require the GAB and LLT packages Do not use this command on a multisystem cluster hastart onenode To start VCS as a time sharing process Run the following command hastart ts Administering the cluster from the command line 237 Stopping VCS Stopping VCS The hastop command stops HAD and related processes You can customize the behavior of the hastop command by configuring the EngineShutdown attribute for the cluster See Controlling the hastop behavior using the EngineShutdown attribute on page 238 The hastop command includes the following options hastop all force hastop help hastop local force evacuate noautodisable hastop sys system force evacuate noautodisable Option Description all Stops HAD on all systems in the cluster and takes all service groups offline help Displays command usage local Stops HAD on the system on which you typed the command force Allows HAD to be stopped without taking service groups offline on the system The value of the EngineShutdown attribute does not influence the behavior of the force option evacuate When combined with local or sys migrates the system s acti
463. on Type the following commands to verify the configuration cd etc VRTSvcs conf config hacf verify 568 Troubleshooting and recovery for VCS Troubleshooting service groups VCS 11032 registration failed Exiting GAB was not registered or has become unregistered Recommended Action GAB is registered by the gabconfig command in the file etc gabtab Verify that the file exists and that it contains the command gabconfig c GAB can become unregistered if LLT is set up incorrectly Verify that the configuration is correct in etc llttab If the LLT configuration is incorrect make the appropriate changes and reboot Waiting for cluster membership This indicates that GAB may not be seeded If this is the case the command gabconfig a does not show any members and the following messages may appear on the console or in the event log GAB Port a registration waiting for seed port membership GAB Port h registration waiting for seed port membership Troubleshooting service groups This section cites the most common problems associated with bringing service groups online and taking them offline Bold text provides a description of the problem Recommended action is also included where applicable VCS does not automatically start service group VCS does not automatically start a failover service group if the VCS engine HAD in the cluster was restarted by the hashadow process This behavior prevents service g
464. on When you save a configuration VCS renames the file main cf autobackup to main cf VCS also save your running configuration to the file main cf autobackup If have not configured the BackupInterval attribute VCS saves the running configuration See Scheduling automatic backups for VCS configuration files on page 243 To save a configuration Run the following command haconf dump makero The option makero sets the configuration to read only Setting the configuration to read write To set the mode to read write Type the following command haconf makerw 244 Administering the cluster from the command line Managing VCS configuration files Taking snapshots of VCS configuration files Use the hasnap command to take snapshots of VCS configuration files on each node in the cluster You can also restore the configuration from a snapshot The command includes the following options each option is described in detail in the following sections nasnap nasnap hasnap nasnap hasnap hasnap nasnap hasnap hasnap backup restore display sdiff fdiff export include exclude delete Backs up files in a snapshot format Restores a previously created snapshot Displays details of previously created snapshots Displays files that were changed on the local system after a specific snapshot was created Displays the differences between a file in the cluster and its
465. on Broker The Veritas Security Services component that serves one level beneath the root broker as an intermediate registration authority and a certification authority The authentication broker can authenticate clients such as users or services and grant them a certificate that will become part of the Veritas credential An authentication broker cannot however authenticate other brokers That task must be performed by the root broker See Root Broker Cluster One or more computers linked together for the purpose of multiprocessing and high availability The term is used synonymously with VCS cluster meaning one or more computers that are part of the same GAB membership Daemon Down Node Alive DDNA A situation where the VCS high availability daemon has failed on a system and has not been restarted by the hashadow process Disaster Recovery A solution that supports fail over to a cluster in a remote location in the event that the local cluster becomes unavailable Disaster recovery global clustering heartbeating and replication Failover A failover occurs when a service group faults and is migrated to another system GAB Group Atomic Broadcast GAB is a communication mechanism of the VCS engine that manages cluster membership monitors heartbeat communication and distributes information throughout the cluster Global Service Group A VCS service group that spans across two or more clusters The ClusterList attribut
466. on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List 98 Administering the cluster from the Cluster Management Console Administering service groups Administering service groups The Cluster Management Console enables you to add and configure a service group according to the requirements of the application that the service group supports Use the following views to administer a service group Cluster Groups This is a cluster level view of all service groups in the cluster Group Summary This is the first view at the service group level in the Cluster Management Console To navigate to the Cluster Groups view 1 2 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Groups In the Cluster Groups view the Service Groups Listing table lists all service groups in the selected cluster The task panels contain several tasks that you can perform in a one to many fashion on the listed service groups To navigate to the Group Summary view 1 2 3 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Groups In the Cluster Groups view in the Groups Listing table click a linked service group name In the G
467. onal and set when the engine invokes the trigger during a failover or switch Set the PreOnline attribute in the service group definition to 1 Set the PreOnline attribute in the service group definition to 0 resadminwait event trigger Description Invoked when a resource enters ADMIN_WAIT state When VCS sets a resource in the ADMIN_WAIT state it invokes the resadminwait trigger according to the reason the resource entered the state See Clearing resources in the ADMIN_WAIT state on page 369 This event trigger is non configurable Usage VCS event triggers 453 List of event triggers resadminwait system resource adminwait_reason m system represents the name of the system resource represents the name of the faulted resource m adminwait_reason represents the reason the resource entered the ADMIN_WAIT state Values range from 0 5 0 The offline entry point did not complete within the expected time 1 The offline entry point was ineffective 2 The online entry point did not complete within the expected time 3 The online entry point was ineffective 4 The resource was taken offline unexpectedly 5 The monitor entry point consistently failed to complete within the expected time 454 VCS event triggers List of event triggers resfault event trigger Description Usage To enable the trigger resnotoff event trigger Description Invoked on the system where a resource has
468. onality Agent m ElifNone Agent m FileNone Agent m FileOnOff Agent m FileOnOnly Agent Description Monitors a file Checks for the file s absence Monitors a file Creates a file monitors it and deletes it Creates and monitors a file Configuring applications and resources in VCS 293 Which agents should I use This information in the table enables you to decide which agent to use depending on the resource you want to make highly available Table 8 1 VCS bundled agents Shared storage Which agents should use m Disk groups and volumes DiskGroup Volume m Disk groups mount points and volumes DiskGroup Mount Volume m SCSI disks and volumes on Linux DiskReservation Volume Or DiskReservation Mount Volume m Volume groups managed using LVM on AIX LVMVG m Volume groups and logical volumes LVMVolumeGroup and managed using LVM on Linux LVMLogicalVolume m Volume groups and logical volumes LVMVolumeGroup and managed using LVM on HP UX LVMLogicalVolume Or LVMCombo m Disk groups and volumes in a SAN DiskGroup and SANVolume managed using Storage Foundation Volume Server Solaris and Linux Network m IP address on a single adapter IP and NIC m IP addresses on multiple adapter systems IPMultiNIC and MultiNICA AIX HP UX Linux Solaris m IP addresses on multiple adapter systems IPMultiNICB and MultiNICB AIX HP UX Solaris m Canonical name CNAME mapping on a
469. one PID which will be monitored m All or some of the above Configuring applications and resources in VCS 301 Configuring application service groups Running the wizard 1 Start the Application wizard from a node in the cluster hawizard application Read the information on the Welcome screen and click Next On the Wizard Options dialog box select to create a new service group or modify an existing group If you chose to modify an existing service group select the service group In the Modify Application Service Group mode you can add modify or delete applications in the service group You can also modify the configuration of the Mount IP and NIC resources if the service group is offline Click Next Specify the service group name and the system list VCS Application Configuration Wizard Service Group Configuration Specify the service group name and the failover priority for each system Service Group Name ApplicationGroup Group System List Available Cluster Systems Systems in Priority Order vessunst More information lt Back Next gt cancel m Enter a name for the service group In the Available Cluster Systems box select the systems on which to configure the service group and click the right arrow icon to move the systems to the service group s system list To remove a system from the service group s system list select the system in the Systems in Priority Order box
470. onsider the system failed and take over its services If the system is later enabled with another console sequence it continues writing to shared storage as before even though its applications have been restarted on other systems Symantec recommends disabling the console abort sequence or creating an alias to force the go command to perform a restart on systems not running 1 0 fencing Select the smallest possible LUNs for use as coordinator disks No more than three coordinator disks are needed in any configuration Do not reconnect the cluster interconnect after a network partition without shutting down one side of the split cluster Acommon example of this happens during testing where the administrator may disconnect the cluster interconnect and create a network partition Depending on when the interconnect cables are reconnected unexpected behavior can occur 360 About communications membership and data protection in the cluster Summary of best practices for cluster communications Controlling VCS behavior m About VCS behavior on resource faults m Controlling VCS behavior at the service group level m Controlling VCS behavior at the resource level m Changing agent file paths and binaries m VCS behavior on loss of storage connectivity m Service group workload management m Sample configurations depicting workload management 362 Controlling VCS behavior About VCS behavior on resource faults About VCS behavior
471. ontains information and tasks for administering the cluster 490 Administering global clusters from the Cluster Management Console Administering global clusters To navigate to the Group Summary view 1 On the main tab bar click Manage 2 On the secondary tab bar click Groups 3 Inthe Cluster Groups view in the Groups Listing table click a linked service group name 4 Inthe Group Summary view choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster Adding a remote cluster Use this procedure to create global clusters by linking standalone clusters m If you are creating a global cluster environment for the first time with two standalone clusters run the operation from either of the clusters m Ifyou are adding a standalone cluster to an existing global cluster environment run the operation from a cluster already in the global cluster environment The following information is required for this procedure m The IP address of the cluster the IP address of a system in the cluster or the name of a system in the cluster being added to the configuration m The user name and password of the administrator for the cluster being added to the configuration Veritas does not support adding a cluster that is already part of a global cluster environment To merge the clusters of one global cluster environment
472. oot The ShutdownTimeout default value of 120 can be changed by modifying the attribute See System attributes on page 641 When a network link fails If a system loses a network link to the cluster other systems stop receiving heartbeats over the links from that system As mentioned above LLT detects this and waits for 16 seconds before declaring the system lost a link When a system panics There are several instances in which GAB will intentionally panic a system including if it detects an internal protocol error or discovers an LLT node ID conflict Three other instances are described below 554 VCS performance considerations How cluster operations affect performance Client process failure If a client process fails to heartbeat to GAB the process is killed If the process hangs in the kernel and cannot be killed GAB halts the system If the k option is used in the gabconfig command GAB tries to kill the client process until successful which may have an impact on the entire cluster If the b option is used in gabconfig GAB does not try to kill the client process Instead it panics the system when the client process fails to heartbeat This option cannot be turned off once set HAD heartbeats with GAB at regular intervals The heartbeat timeout is specified by HAD when it registers with GAB the default is 15 seconds If HAD gets stuck within the kernel and cannot heartbeat with GAB within the specified timeout G
473. op down menu Enter a valid user name and password for the cluster On UNIX the default user account is the local root account On Windows the default user account is the account that was used to perform the VCS installation Enter the domain name associated with the account and then click Login If the user account you are using to connect is not a domain account enter the fully qualified system name for the local host system The local host is the cluster node running the Cluster Management Console If you are logging in to a secure cluster you can set Domain Type to unixpwd to automatically set Domain to the local host name You can then log in without being required to know the name of the node that is running the Cluster Management Console This scenario is for the typical case in which log in authentication takes place on the local host where the Cluster Management Console is running You can also log in by authenticating on a cluster node other than the one where the Cluster Management Console is running To authenticate on another node alternate host set Domain Type to unixpwd and then enter the alternate host name into the Domain box In this scenario a trust relationship must exist between the authentication broker installed on the local host and the one installed on the alternate host See your Veritas Cluster Server 5 0 Installation Guide for information on setting up trust relationships between authentication brokers Admi
474. ore deleting a cluster make sure the cluster is not configured as a global cluster You can delete global clusters from the Global View To delete a simulated cluster 1 2 From Simulator Explorer select the cluster and click Delete Cluster In the confirmation dialog box click Yes Starting a simulated cluster Start the cluster to begin administering it To start a simulated cluster 1 2 3 4 In the Simulator console select the cluster Click Start Cluster After the cluster starts click Launch Console to administer the cluster Enter a valid user name and password to log on to the cluster VCS Simulator does not validate passwords you can log on to a simulated cluster by just entering a valid VCS user name If you use the default configuration enter admin for the user name Cluster Explorer is launched upon initial logon and the icons in the cluster panel change color to indicate an active panel Verifying a simulated cluster configuration Verify the configuration is valid To verify the simulated cluster configuration 1 2 In the Simulator console select the cluster Click Verify Configuration 324 Predicting VCS behavior using VCS Simulator Administering VCS Simulator from the Java Console Simulating a global cluster configuration Simulate a global cluster environment to test your global cluster configuration See How VCS global clusters work on page 462 To simulate a global clus
475. ore required data on shared disks m Adherence to license requirements and host name dependencies Defined start stop and monitor procedures The application to be clustered must have defined procedures for starting stopping and monitoring Start procedure Stop procedure Monitor procedure The application must have a command to start it and all resources it may require VCS brings up the required resources in a specific order then brings up the application using the defined start procedure For example to start an Oracle database VCS must know which Oracle utility to call such as sqlplus VCS must also know the Oracle user instance ID Oracle home directory and the pfile An individual instance of the application must be capable of being stopped without affecting other instances For example killing all HTTPd processes on a Web server is unacceptable because it would also stop other Web servers If VCS cannot stop an application cleanly it may call for a more forceful method like a kill signal After a forced stop a clean up procedure may be required for various process and application specific items that may be left behind such as shared memory segments or semaphores The application must have a monitor procedure that determines if the specified application instance is healthy The application must allow individual monitoring of unique instances For example the monitor procedure for a Web server connects to
476. ou to select nodes from the System List Viewing logs In the Manage tab view hierarchy the following levels of views each have a logs tab on the secondary tab bar Administering the cluster from the Cluster Management Console 131 Viewing logs Cluster Group System Resource See The Manage tab secondary tabs and the view hierarchy on page 83 The Logs tab takes you to a view of log entries about the individual object that you select To navigate to the logs view for an object 1 2 On the navigation bar click Home Navigate to one of the following views for a particular object m Cluster Summary Group Summary m System Summary m Resource Summary In the Object Summary view on the secondary tab bar click Logs Object is the type of the object that you selected Each logs tab contains a search box and Search button so that you can quickly find the log entries that you want See Overview of the Cluster Management Console on page 76 for information about navigating among views See Conducting a search on page 132 Note Logs for clusters and service groups contain entries about only the object that you select Entries for descendant objects are not included 132 Administering the cluster from the Cluster Management Console Conducting a search Conducting a search The search feature uses text based comparison You can perform a search using either the search bar or the Sea
477. oup Administrator Ect Group J7 Group Operator Cancel m Select the appropriate check boxes to grant privileges to the user To grant Group Administrator or Group Operator privileges proceed to the next step Otherwise proceed to the last step Click Select Groups m Click the groups for which you want to grant privileges to the user then click the right arrow to move the groups to the Selected Groups box m Click OK in the Change Privileges dialog box then click Close in the User Manager dialog box Administering the cluster from Cluster Manager Java console 177 Administering user profiles Assigning privileges for OS user groups in secure clusters In secure clusters you can assign privileges to native users at an operating system OS user group level Assigning VCS privileges to an OS user group involves adding the user group in one or more of the following attributes m AdministratorGroups for a cluster or for a service group m OperatorGroups for a cluster or for a service group m Guests See User privileges for OS user groups in secure clusters on page 66 To assign privileges to an OS user group 1 From Cluster Explorer configuration tree select the cluster to assign privileges for the cluster or a service group to assign privileges for specific service groups 2 From the view panel click the Properties tab 3 From the list of key attributes click the edit icon against Admi
478. oup Operator privileges for specific service groups Note By default newly created users are assigned Cluster Guest permissions Group Guest Group Guests have read only access to the service group meaning they can view the configuration but cannot change it The Group Guest role is available in secure clusters only 63 64 About the VCS user privilege model About VCS user privileges and roles About the hierarchy in VCS roles The following illustration shows the roles and how they overlap with one another Cluster Administrator r y includes privileges for Cluster Operator P A A E O E AS Yy Includes privileges for Cluster Guest includes privileges for gt Group Administrator includes privileges for Group ane A E ES v includes privileges for Group Guest For example Cluster Administrator includes privileges for Group Administrator which includes privileges for Group Operator User privileges for CLI commands The following concepts apply to users executing commands from the command line m Users logged on as root or administrator are granted privileges that exceed those of Cluster Administrator such as the ability to start and stop a cluster m Ifyou do not have root privileges VCS prompts for your VCS user name and password when you execute haxxx commands You can use the halogin command to save the authentication information so that you do not have to enter your creden
479. oup to be taken offline and brought online Taking the RemoteGroup resource offline does not take the remote service group offline If you are choosing one to one mapping between the local nodes and remote nodes then the value of the AutoFailover attribute of the remote service group must be 0 Note When you set the value of ControlMode to OnlineOnly or to MonitorOnly the recommend value of the VCSSysName attribute of the RemoteGroup resource is ANY If you want one to one mapping between the local nodes and the remote nodes then a switch or fail over of local service group is impossible It is important to note that in both these configurations the RemoteGroup agent does not take the remote service group offline Example Configuring a RemoteGroup resource In this example m VCS cluster cluster1 provides high availability for Web services Configure a VCS service group ApacheGroup with an agent to monitor the Web server for example Apache to monitor the Web services m VCS cluster cluster2 provides high availability for the database required by the Web services Configure a VCS service group OracleGroup with a database agent for example Oracle to monitor the database The database resource must come online before the Web server comes online You create this dependency using the RemoteGroup agent To configure the RemoteGroup agent in the example 1 Adda RemoteGroup resource in the ApacheGroup service group in cl
480. ources depend is a child resource The graph links a child resource icon to a parent resource icon above it m A resource can function as a parent and a child Point to a resource icon to display ScreenTips about the type state and key attributes of the resource The state of the resource reflects the state ona specified system local In the bottom pane of the Resource view point to the system and service group icons to display ScreenTips about the service group status on all or individual systems in a cluster Click a system icon to view the resource graph of the service group on the system Click the service group icon to view the resource graph on all systems in the cluster 156 Administering the cluster from Cluster Manager Java console About Cluster Explorer To access the Resource view 1 From Cluster Explorer click the service groups tab in the configuration tree 2 Click a service group in the configuration tree 3 Inthe view panel click the Resources tab Moving and linking icons in Service Group and Resource views The Link and Auto Arrange buttons are available in the top right corner of the Service Group or Resource view Link 83 Auto Arrange P Click Link to set or disable the link mode for the Service Group and Resource views Note There are alternative ways to set up dependency links without using the Link button The link mode enables you to create a dependency link by clicking on
481. ovide high availability to Samba shares Include SambaServer SambaShare and NetBIOS agents Configuring applications and resources in VCS 291 About VCS bundled agents Services and Applications agents Services and application agents make web sites applications and processes highly available Agent m Apache m Application m Process m ProcessOnOnly m Zone agent Solaris 10 Description Makes an Apache Web server highly available Brings applications online takes them offline and monitors their status Starts stops and monitors a user specified process Starts and monitors a user specified process Brings Solaris 10 zones online takes them offline and monitors their status VCS infrastructure and support agents The VCS infrastructure and support agents provide high availability for VCS related operations Agent m NotifierMngr E Description Monitors the VCS notifier process VRTSWebApp agent Brings Web applications configured using Veritas Web Proxy Agent Phantom Agent Server VRTSWeb online takes them offline and monitors their status Monitors the state of a resource on a local or remote system Determines the state of service groups having resources of type None only RemoteGroup agent Monitors the status of a remote service group 292 Configuring applications and resources in VCS About VCS bundled agents Testing agents Use the following agents to test VCS functi
482. p 2S4 Ha BBO Command Center The Command Center helps you build and execute VCS commands The user interface helps build a command or a set of commands The text field at the bottom of the screen displays the commands to be executed Close To access Command Center From Cluster Explorer click Command Center on the Tools menu or On the Cluster Explorer toolbar click Show Command Center 164 Administering the cluster from Cluster Manager Java console Accessing additional features of the Java Console Configuration wizard Use Configuration Wizard to create and assign service groups to systems in a cluster See Creating service groups with the configuration wizard on page 196 To access Configuration Wizard From Cluster Explorer click Configuration Wizard on the Tools menu or On the Cluster Explorer toolbar click Launch Configuration Wizard Notifier Resource Configuration wizard VCS provides a method for notifying an administrator of important events such as aresource or system fault VCS includes a notifier component which consists of the notifier daemon and the hanotify utility This wizard enables you to configure the notifier component as a resource of type NotifierMngr as part of the ClusterService group See Setting up VCS event notification using the Notifier wizard on page 225 To access Notifier Resource Configuration Wizard From Cluster Explorer click Notifier Wizard o
483. p online or to bring it online Note Disabling a resource is not an option when the entire service group requires disabling In that case set the service group attribute Enabled to 0 To disable a resource To disable the resource when VCS is running hares modify resource name Enabled 0 To have the resource disabled initially when VCS is started set the resource s Enabled attribute to 0 in main cf Limitations of disabling resources When VCS is running there are certain prerequisites to be met before the resource is disabled successfully m An online non persistent resource cannot be disabled It must be in a clean OFFLINE state The state must be OFFLINE and State must be NOT WAITING m fitis a persistent resource and the state is ONLINE on some of the systems all dependent resources parents must be in clean OFFLINE state The state must be OFFLINE and State must be NOT WAITING Therefore before disabling the resource you may be required to take it offline if it is non persistent and take other resources offline in the service group Controlling VCS behavior 385 Controlling VCS behavior at the resource level Additional considerations for disabling resources m When a group containing disabled resources is brought online the online transaction is not propagated to the disabled resources Children of the disabled resource are brought online by VCS only if they are required by another enabled resource m Yo
484. pane The status pane can contain many different panels and tables depending upon the view selected These panels and tables contain status information about the cluster objects detailed by the selected view In the startup view the status pane contains panels labelled Faulted Groups Groups and Systems and a table labelled Critical and Error logs in last eight hours Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console Figure 5 9 Status pane in the startup Cluster Summary view sol50_510a_sec1 Cluster Summary Summary Groups Systems Attributes Resources Group Dependency Resource Types DR Alerts Clu Cluster Name sol50_510a_sec1 State Online Cluster Type VCS Version 5 0 00 0 Faulted Objects Ja Groups 4 AJ Systems 2 The Manage tab secondary tabs and the view hierarchy The Management tab on the main tab bar represents the main category of tasks and information in the Cluster Management Console It also marks the beginning of a multi level navigation path or view hierarchy The management tab branches down into a set of levels and each level contains a set of views Under the Manage tab the first level down is called cluster The cluster level under the Management tab indicates that you are viewing your cluster environment at the highest and broadest level possible From the cluster level you can navigate to views that contain info
485. parallel m The NIC resource is a persistent resource and does not require starting The service group NFS_Group can be configured to start automatically on either node in the preceding example It can then move or fail over to the second node on command or automatically if the first node fails Upon failover or relocation VCS takes the resources offline beginning at the top of the graph VCS then starts resources on the second node beginning at the bottom 32 Introducing Veritas Cluster Server Putting the pieces together Chapter 2 About cluster topologies m Basic failover configurations m Advanced failover configurations m Cluster topologies and storage configurations 34 About cluster topologies Basic failover configurations Basic failover configurations This section describes basic failover configurations including asymmetric symmetric and N to 1 Asymmetric or Active Passive configuration In an asymmetric configuration an application runs on a primary or master server A dedicated redundant server is present to take over on any failure The redundant server is not configured to perform any other functions In the following illustration a database application is moved or failed over from the master to the redundant server Before Failover After Failover Application Application 1 A Figure 2 1 Asymmetric failover This configuration is the simplest and most relia
486. parallel parent multiple instances of child group online are acceptable m Fora parallel child group linked online remote with failover parent multiple instances of child group online are acceptable as long as child group does not go online on the system where parent is online m Fora parallel child group linked offline local with failover parallel parent multiple instances of child group online are acceptable as long as child group does not go online on the system where parent is online Offline operations in group dependencies VCS rejects offline operations if the procedure violates existing group dependencies Typically firm dependencies are more restrictive to taking child group offline while parent group is online Rules for manual offline include m Parent group offline is never rejected For all soft dependencies child group can go offline regardless of the state of parent group m For all firm dependencies if parent group is online child group offline is rejected m For the online local hard dependency if parent group is online child group offline is rejected Switch operations in group dependencies Switching a service group implies manually taking a service group offline on one system and manually bringing it back online on another system VCS rejects manual switch if the group does not comply with the rules for offline or online operations 426 The role of service group dependencies VCS behavior with se
487. pecify IP addresses in the Arguments attribute make sure the IP addresses have DNS entries 5 Click OK 6 Click OK in the Heartbeat configuration dialog box Now you can monitor the state of both clusters from the Java Console 478 Connecting clusters Creating global clusters Setting up a global cluster Configuring the Steward process In case of a two cluster GCO you can configure a Steward to prevent potential split brain conditions provided the proper network infrastructure exists See The steward process Split brain in two cluster global clusters on page 468 To configure the Steward process optional 1 Identify a system that will host the Steward process Make sure both clusters can connect to the system through a ping command Copy the file steward from a node in the cluster to the Steward system The file resides at the path opt VRTSvcs bin In both clusters set the Stewards attribute to the IP address of the system running the Steward process For example cluster cluster1938 UserNames admin gNOgNInKOjOOmWOiNL ClusterAddress 10 182 147 19 Administrators admin CredRenewFrequency 0 CounterInterval 5 Stewards 10 212 100 165 On the system designated to host the Steward start the Steward process steward start To stop the Steward process use the following command steward stop Connecting clusters Creating global clusters 479 Setting up a glob
488. peration are used for heartbeat and link state maintenance only and the frequency of heartbeats is reduced to 50 of normal to reduce network overhead 338 About communications membership and data protection in the cluster About cluster membership If there is a failure of all configured high priority links LLT will switch all cluster communications traffic to the first available low priority link Communication traffic will revert back to the high priority links as soon as they become available While not required best practice recommends to configure at least one low priority link and to configure two high priority links on dedicated cluster interconnects to provide redundancy in the communications path Low priority links are typically configured on the public or administrative network About cluster membership The current members of the cluster are the systems that are actively participating in the cluster It is critical for HAD to accurately determine current cluster membership in order to take corrective action on system failure and maintain overall cluster topology A change in cluster membership is one of the starting points of the logic to determine if HAD needs to perform any fault handling in the cluster There are two aspects to cluster membership initial joining of the cluster and how membership is determined once the cluster is up and running Initial joining of systems to cluster membership When the cluster ini
489. performance Booting a cluster system When a cluster system boots the kernel drivers and VCS process start ina particular order If it is the first system in the cluster VCS reads the cluster configuration file main cf and builds an in memory configuration database This is the LOCAL_BUILD state After building the configuration database the system transitions into the RUNNING mode If another system joins the cluster while the first system is in the LOCAL_BUILD state it must wait until the first system transitions into RUNNING mode The time it takes to build the configuration depends on the number of service groups in the configuration and their dependencies and the number of resources per group and resource dependencies VCS creates an object for each system service group type and resource Typically the number of systems service groups and types are few so the number of resources and resource dependencies determine how long it takes to build the configuration database and get VCS into RUNNING mode If a system joins a cluster in which at least one system is in RUNNING mode it builds the configuration from the lowest numbered system in that mode Note Bringing service groups online as part of AutoStart occurs after VCS transitions to RUNNING mode When a resource comes online The online entry point of an agent brings the resource online This entry point may return before the resource is fully online The subsequent moni
490. pically overhead of VCS kernel components is minimal Kernel components provide heartbeat and atomic information exchange among cluster systems By default each system in the cluster sends two small heartbeat packets per second to other systems in the cluster Heartbeat packets are sent over all network links configured in the etc llttab configuration file System to system communication is load balanced across all private network links If a link fails VCS continues to use all remaining links Typically network links are private and do not increase traffic on the public network or LAN You can configure a public network LAN link as low priority which by default generates a small approximately 64 byte broadcast packet per second from each system and which will carry data only when all private network links have failed VCS performance considerations 547 How cluster components affect performance The VCS engine HAD The VCS engine HAD runs as a daemon process By default it runs as a high priority process which ensures it sends heartbeats to kernel components and responds quickly to failures HAD runs logging activities in a separate thread to reduce the performance impact on the engine due to logging VCS waits in a loop waiting for messages from agents ha commands the graphical user interfaces and the other systems Under normal conditions the number of messages processed by HAD is few They mainly include heartbeat messages
491. ps Bringing service groups online To bring a service group online Type the following command hagrp online service group sys system To start a service group on a system and bring online only the resources already online on another system Type the following command hagrp online service group sys system checkpartial other system If the service group does not have resources online on the other system the service group is brought online on the original system and the checkpartial option is ignored Note that the checkpartial option is used by the Preonline trigger during failover When a service group configured with Preonline 1 fails over to another system system 2 the only resources brought online on system 2 are those that were previously online on system 1 prior to failover Taking service groups offline To take a service group offline Type the following command hagrp offline service group sys system To take a service group offline only if all resources are probed on the system Type the following command hagrp offline ifprobed service group sys system Switching service groups The process of switching a service group involves taking it offline on its current system and bringing it online on another system To switch a service group from one system to another Type the following command hagrp switch service group to system A service group can be switche
492. r Sample configurations depicting workload management Sample configurations depicting workload management This section lists some sample configurations that use the concepts described in this chapter System and Service group definitions The main cf in this example shows various Service Group Workload Management attributes in a system definition and a service group definition See VCS attributes on page 613 include types cf cluster SGWM demo system LargeServerl Capacity 200 Limits ShrMemSeg 20 Semaphores 10 Processors 12 LoadWarningLevel 90 LoadTimeThreshold 600 group G1 SystemList LargeServerl LargeServer2 MedServerl1 MedServer2 SystemZones LargeServerl 0 LargeServer2 0 MedServerl 1 MedServer2 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 10 Semaphores 5 Processors 6 Controlling VCS behavior 395 Sample configurations depicting workload management Sample configuration Basic four node cluster include types cf cluster SGWM demo system Serverl Capacity 100 system Server2 Capacity 100 system Server3 Capacity 100 system Server4 Capacity 100 group G1 SystemList Serverl Server2 Server3 Server4 AutoStartPolicy Load AutoStartList Serverl Server2 Server3 Server4 FailOverPolicy Load Load 20 group
493. r This is the initial state for all systems LEAVING The system is leaving the cluster gracefully When the agents have been stopped and when the current configuration is written to disk the system transitions to EXITING LOCAL_BUILD The system is building the running configuration from the disk configuration REMOTE_BUILD The system is building a running configuration that it obtained from a peer in a RUNNING state 610 Cluster and system states System states Table C 2 VCS system states RUNNING The system is an active member of the cluster STALE_ADMIN_WAIT The system has an invalid configuration and there is no other system in the state of RUNNING from which to retrieve a configuration If a system with a valid configuration is started that system enters the LOCAL_BUILD state Systems in STALE _ADMIN_WAIT transition to STALE_PEER_WAIT STALE_DISCOVER_WAIT The system has joined the cluster with an invalid configuration file It is waiting for information from any of its peers before determining how to transition to another state STALE_PEER_WAIT The system has an invalid configuration file and another system is doing a build from disk LOCAL_BUILD When its peer finishes the build this system transitions to the state REMOTE_BUILD UNKNOWN The system has not joined the cluster because it does not have a system entry in the configuration Cluster and system states 611 System states
494. r the disk groups containing the Replicated Volume Group RVG must be imported at the primary and secondary sites The replication service group must be online at both sites simultaneously and must be configured as a hybrid VCS service group The application service group is configured as a failover service group The application service group must be configured with an online local hard dependency on the replication service group Note VVR supports multiple replication secondary targets for any given primary However RDC for VCS supports only one replication secondary for a primary An RDC configuration is appropriate in situations where dual dedicated LLT links are available between the primary site and the disaster recovery secondary site but lacks shared storage or SAN interconnect between the primary and secondary data centers In an RDC data replication technology is employed to provide node access to data in a remote site Note You must use dual dedicated LLT links between the replicated nodes Setting up replicated data clusters 537 How VCS replicated data clusters work How VCS replicated data clusters work To understand how a replicated data cluster configuration works let us take the example of an Oracle database configured in a VCS RDC The configuration has two system zones m Primary zone zone 0 comprising nodes located at the primary site and attached to the primary storage m Secondary zone zone 1
495. r ActionThreshold and NotifyThreshold represent the threshold in terms of CPU percentage utilization If Enabled is set to 1 HAD monitors the usage and updates CPUUsage attribute If Enabled is set to 0 default HAD does not monitor the usage If the system s CPU usage continuously exceeds the value set in NotifyThreshold for a duration greater than the value set in NotifyTimeLimit HAD sends notification via an SNMP trap or SMTP message If the CPU usage continuously exceeds the value set in NotifyThreshold for a duration greater than the value set in NotifyTimeLimit subsequent notifications are sent after five minutes to avoid sending notifications too frequently if the NotifyTimeLimit value is set to a value less than five minutes In this case notification is sent after the first interval of NotifyTimeLimit As CPU usage continues to exceed the threshold value notifications are sent after five minutes If the values of NotifyThreshold or NotifyTimeLimit are set to 0 no notification is sent If system s CPU usage exceeds the value set in ActionThreshold continuously for a duration greater than the value set in ActionTimeLimit the specified action is taken If the CPU usage continuously exceeds the ActionThreshold for a duration greater than the value set in ActionTimeLimit subsequent action is taken after five minutes to avoid taking action too frequently if the ActionTimeLimit value is set to less than five minutes In this cas
496. r Manager dialog box click the user name Click Remove User Click Yes Click Close Changing a user password A user with Administrator Operator or Guest privileges can change his or her own password You must be logged on as Cluster Administrator to access User Manager Before changing the password make sure the configuration is in the read write mode Cluster administrators can change the configuration to the read write mode Note This module is not available if the cluster is running in secure mode To change a password as an administrator 1 2 3 4 From Cluster Explorer click User Manager on the File menu Click the user name Click Change Password In the Change Password dialog box m Enter the new password m Re enter the password in the Confirm Password field m Click OK Click Close 176 Administering the cluster from Cluster Manager Java console Administering user profiles To change a password as an operator or guest 1 From Cluster Explorer click Change Password on the File menu 2 Inthe Change Password dialog box m Enter the new password m Reenter the password in the Confirm Password field m Click OK 3 Click Close Changing a user privilege 1 From Cluster Explorer click User Manager on the File menu 2 Click the user name 3 Click Change Privileges and enter the details for user privileges T Change Privileges admin xi IV Cluster Administrator I Cluster Operator J Gr
497. r update a license on remote nodes To install a new license Run the following command on each node in the cluster cd opt VRTS bin vxlicinst k XXXX XXXX XXXX XXXX XXXX XXX To update licensing information in a running cluster 1 Install the new license on each node in the cluster using the vxlicinst utility 2 Update system level licensing information on all nodes in the cluster hasys updatelic all You must update licensing information on all nodes before proceeding to the next step 3 Update cluster level licensing information haclus updatelic 236 Administering the cluster from the command line Starting VCS Starting VCS The command to start VCS is invoked from the following file etc rc3 d S99ves or sbin rc3 d S99vcs When VCS Starts Starting VCS When VCS is started it checks the state of its local configuration file and registers with GAB for cluster membership If the local configuration is valid and if no other system is running VCS it builds its state from the local configuration file and enters the RUNNING state If the configuration on all nodes is invalid the VCS engine waits for manual intervention or for VCS to be started on a system that has a valid configuration See Cluster and system states on page 605 Use the hastart command to start VCS To start VCS Run the following command hastart To start VCS when all systems are in the ADMIN_WAIT state Run the f
498. raffic generated by VCS Each system sends 2 heartbeat packets per second per interface All other cluster communications including all status and configuration traffic is point to point unicast This heartbeat is used by the Group Membership Services to determine cluster membership The heartbeat signal is defined as follows m LLT on each system in the cluster sends heartbeat packets out on all configured LLT interfaces every half second m LLT on each system tracks the heartbeat status from each peer on each configured LLT interface m LLT on each system forwards the heartbeat status of each system in the cluster to the local Group Membership Services function of GAB m GAB receives the status of heartbeat from all cluster systems from LLT and makes membership determination based on this information Figure 10 2 Heartbeat in the cluster AGENT AGENT AGENT AGENT HAD HAD Broadcast heartbeat on every interface every GAB 4 GAB 0 5 second LLT ur UER Each LLT module tracks heartbea LLT forwards heartbeat status from each peer on each status of each system to configured interface GAB LLT can be configured to designate specific cluster interconnect links as either high priority or low priority High priority links are used for cluster communications to GAB as well as heartbeat signals Low priority links during normal o
499. rage is used as a benchmark average If the current running average for a resource differs from the benchmark average by more than this threshold value VCS regards this as a sign of gradual increase or decrease in monitor cycle times and sends a notification about it for the resource Whenever such an event occurs VCS resets the internally maintained benchmark average to this new average VCS sends notifications regardless of whether the deviation is an increase or decrease in the monitor cycle time For example a value of 25 means that if the actual average monitor time is 25 more than the benchmark monitor time average VCS sends a notification Tracking monitor cycle times VCS marks sudden changes in monitor times by comparing the time taken for each monitor cycle with the ExpectedValue If this difference exceeds the ValueThreshold VCS sends a notification about the sudden change in monitor time Note that VCS sends this notification only if monitor time increases VCS marks gradual changes in monitor times by comparing the benchmark average and the moving average of monitor cycle times VCS computes the benchmark average after a certain number of monitor cycles and computes the moving average after every monitor cycle If the current moving average exceeds the benchmark average by more than the AvgThreshold VCS sends a notification about this gradual change in the monitor cycle time 562 VCS performance considerations VCS agent sta
500. rce types Table 4 1 About the VCS user privilege model About VCS user privileges and roles User roles in VCS Cluster Operator Cluster Operators can perform all cluster group and resource level operations including modifying the user s own password and bringing service groups online Note Cluster Operators can change their own passwords only if configuration is in read write mode Cluster Administrators can change the configuration to the read write mode Users with this role can be assigned Group Administrator privileges for specific service groups Group Administrator Group Administrators can perform all service group operations on specific groups such as bringing groups and resources online taking them offline and creating or deleting resources Additionally users can establish resource dependencies and freeze or unfreeze service groups Note that Group Administrators cannot create or delete service groups Group Operator Group Operators can bring service groups and resources online and take them offline Users can also temporarily freeze or unfreeze service groups Cluster Guest Cluster Guests have read only access to the cluster meaning they can view the configuration but cannot change it They can modify their own passwords only if the configuration is in read write mode They cannot add or update users Additionally users with this privilege can be assigned Group Administrator or Gr
501. rch tab The search bar is located immediately below the main tab bar and contains a text entry box and a Search button You may need to scroll right to see these controls on some systems The search bar is available for all views except those under the Search tab The Search tab is located on the main tab bar Views under the Search tab contain a text entry box and a Search button and offer some filtering options that the search bar does not The search feature displays results in one or more of the search views which you navigate to using secondary tabs under Search These tabs are labeled Clusters Groups Systems Resources Resource Types and Logs Each tab contains a table that lists the search results or hits in the corresponding object category By default the console takes you to the search views after each search The destination view is whichever view was most recently visited Searches are limited to text strings Separate two or more strings using a space character The search feature supports a form of Boolean AND and OR operations as well as wildcard searches with the Exact Match and Match All Terms options The search feature does not currently support m Direct textual entry of Boolean operators m Direct textual entry of wildcards m Query expressions To perform a search using the search bar 1 Inthe text entry box type one or more text strings that you want to find Use a space character to separate text st
502. rding to the specified severity level However until notifier communicates the specifications to HAD HAD stores all messages because it does not know the severity the user has specified This behavior prevents messages from being lost between the time HAD stores them and notifier communicates the specifications to HAD 432 VCS event notification About VCS event notification Persistent and replicated message queue VCS includes a sophisticated mechanism for maintaining event messages which ensures that messages are not lost On each node VCS queues messages to be sent to the notifier process This queue is persistent as long as VCS is running and the contents of this queue remain the same on each node If the notifier service group fails notifier is failed over to another node in the cluster Because the message queue is consistent across nodes notifier can resume message delivery from where it left off even after failover How HAD deletes messages The VCS engine HAD stores messages to be sent to notifier HAD deletes messages under the following conditions The message has been in the queue for one hour and notifier is unable to deliver the message to the recipient This behavior means that until notifier connects to HAD messages are stored permanently in the queue until one of the following conditions are met or The message queue is full and to make room for the latest message the earliest message is deleted or VCS
503. re system Child Instance remains online where the Parent faulted online local Instance of Instance of Parent is taken Parent fails over to firm parallel Child Child is offline Parent fails other system and group on online on over to other system depends on Child same system same and depends on instance there system Child instance there Child Instance remains online where Parent faulted online Allinstances One or more Parent remains Parent fails over to global soft of parallel instances of online if Child faults another system Child group Child group on any system maintaining remaining is online If faulted Child fails dependence on all online somewhere over toanothier Child instances in the system Parent is cluster brought online on any system If Child cannot fail over to another system Parent remains offline The role of service group dependencies Service group dependency configurations 419 online All instances Allinstances Parent is taken Parent fails over to global firm of parallel of Child offline After Child another system Child group group are fails over Parent maintaining remaining online fails over to another dependence on all online somewhere system Child instances in the If Child cannot fail cluster over Parent remains offline online Oneormore Oneormore Parent remains Parent fails over to remote soft instances instances of online another system parallel Child Chi
504. ree right click the service group 5 Click Switch To and select the system in the primary RDC zone to switch to and click OK Setting up a fire drill You can use fire drills to test the configuration s fault readiness by mimicking a failover without stopping the application in the primary data center See Setting up a fire drill on page 483 Section Troubleshooting and performance m Chapter 20 VCS performance considerations on page 545 m Chapter 21 Troubleshooting and recovery for VCS on page 563 544 Troubleshooting and performance Chapter 20 VCS performance considerations How cluster components affect performance How cluster operations affect performance Scheduling class and priority configuration CPU binding of HAD Monitoring CPU usage VCS agent statistics 546 VCS performance considerations How cluster components affect performance How cluster components affect performance VCS and its agents run on the same systems as the applications Therefore VCS attempts to minimize its impact on overall system performance The three main components of clustering that have an impact on performance include the kernel specifically GAB and LLT the VCS engine HAD and the VCS agents For details on attributes or commands mentioned in the following sections see the chapter on administering VCS from the command line and the appendix on VCS attributes Kernel components GAB and LLT Ty
505. rent state of the cluster m Type and dimension integer scalar m Default Not applicable ClusterAddress system use only Specifies the cluster s virtual IP address used by a remote cluster when connecting to the local cluster m Type and dimension string scalar m Default Not applicable Table D 5 VCS attributes Cluster attributes 649 Cluster attributes ClusterLocation user defined Specifies the location of the cluster m Type and dimension string scalar m Default ClusterName user defined The name of cluster m Type and dimension string scalar m Default ClusterOwner user defined This attribute is used for VCS notification specifically VCS sends notifications to persons designated in this attribute when an event occurs related to the cluster See About VCS event notification on page 430 m Type and dimension string scalar m Default unknown ClusterTime system use only The number of seconds since January 1 1970 This is defined by the lowest node in running state m Type and dimension string scalar m Default Not applicable ClusterUUID system use only Unique ID assigned to the cluster by Availability Manager m Type and dimension string scalar m Default Not applicable CompareRSM user defined Indicates if VCS engine is to verify that replicated state machine is consistent This can be set by running th
506. res 5 Processors 6 LargeServer2 100 ShrMemSeg 10 G2 Semaphores 5 Processors 6 MedServer1 70 ShrMemSeg 10 G3 Semaphores 5 Processors 6 MedServer2 80 ShrMemSeg 10 G4 Semaphores 5 Processors 6 Failure scenario In this scenario if LargeServer2 fails VCS scans all available systems in group G2 s SystemList that are in the same SystemZone and creates a subset of systems that meet the group s prerequisites In this case LargeServer1 meets all 402 Controlling VCS behavior Sample configurations depicting workload management required Limits Group G2 is brought online on LargeServer1 This results in the following configuration Server AvailableCapacity CurrentLimits Online Groups LargeServer1 0 ShrMemSeg 0 G1 G2 Semaphores 0 Processors 0 MedServer1 70 ShrMemSeg 10 G3 Semaphores 5 Processors 6 MedServer2 80 ShrMemSeg 10 G4 Semaphores 5 Processors 6 After 10 minutes LoadTimeThreshold 600 VCS fires the loadwarning trigger on LargeServer1 because the LoadWarningLevel exceeds 90 percent Cascading failure scenario In this scenario another system failure can be tolerated because each system has sufficient Limits to accommodate the service group running on its peer If MedServerl fails its groups can fail over to MedServer2 If LargeServer1 fails the failover of the two groups running on it is serialized The first group lexically G1 chooses MedServer2 because the s
507. ributes are both classified as int or integer The DiskGroup StartVolumes and StopVolumes attributes are defined as str or strings See Attribute data types on page 51 Defines the parameters passed to the VCS engine through the ArgList attribute The line static str ArgList xxx yyy zzz defines the order in which parameters are passed to the agents for starting stopping and monitoring resources 50 VCS configuration concepts The types cf file For another example review the following main cf and types cf representing an IP resource main cf for Solaris IP nfs_ipl Device hme0 Address 192 168 1 201 types cf for Solaris type IP static keylist SupportedActions device vfd route vfd static str ArgList Device Address NetMask Options ArpDelay IfconfigTwice ContainerName tr Device tr Address tr NetMask tr Options int ArpDelay 1 int IfconfigTwice tr ContainerName S S S S S The high availability address is configured on the interface defined by the Device attribute The IP address is enclosed in double quotes because the string contains periods See Attribute data types on page 51 The VCS engine passes the identical arguments to the IP agent for online offline clean and monitor It is up to the agent to use the arguments it requires All resource names must be unique in a VCS cluster About the ClusterService group The ClusterService group i
508. rights enables the wizard to connect to all the clusters and make changes to the configuration Remote Cluster Configuration Wizard x Remote cluster information Enter the connection details For each cluster amp Host Name IP Address Username Configure z4 Click the Configure button and enter the following details for each cluster The cluster address of the cluster or the IP a jame of a node in the cluster User name and password for an adminstrator to the cluster Existing cluster configuration details Remote Cluster WCSHP11_72 cancel Hep 6 Click the Configure icon The Remote cluster information dialog box is displayed If the cluster is not running in secure mode T Remote cluster information xi Remote cluster information Cluster Name vcslinux147_148_QA22 Host Name IP address vesinu Pot haa o sts S User Name admin Password reek Cancel Enter the host name of a cluster system an IP address of a cluster system or the IP address of the cluster that will join the global environment Verify the port number Enter the user name Enter the password Click OK Repeat these steps for each cluster in the global environment 507 508 Administering global clusters from Cluster Manager Java console Adding a remote cluster If the cluster is running in secure mode T Remote cluster information xi
509. ring VCS objects with Cluster Monitor Cluster Monitor summarizes the state of various objects in a cluster and provides access to in depth information about these objects in Cluster Explorer The right pane of a Cluster Monitor panel displays the connection status online offline up or down of service groups systems and heartbeats The left pane of a Cluster Monitor panel displays three icons representing service groups systems and heartbeats The colors of the icons indicate the state of the cluster for example m A flashing red slash indicates Cluster Manager failed to connect to the cluster and will attempt to connect to another system in the cluster m A flashing yellow slash indicates Cluster Manager is experiencing problems with the connection to the cluster Pointing to an icon accesses the icon s ScreenTip which provides additional information on the specific VCS object Administering the cluster from Cluster Manager Java console 145 About Cluster Monitor To review detailed information about VCS objects in Cluster Explorer Logs and Command Center right click a panel to access a menu Menus are enabled when the Cluster Monitor display appears in the default expanded view If you activate a menu on a collapsed scrolling view of Cluster Monitor the scrolling stops while accessing the menu Expanding and collapsing the Cluster Monitor display Cluster Monitor supports two views expanded default and collapsed The
510. ring attribute type 51 SupportedActions attribute 626 Symantec Product Authentication Service about 28 authentication broker 28 disabling 280 enabling 280 root broker 28 viewing credentials 167 symmetric configuration 35 SysInfo attribute 646 SysName attribute 646 sysoffline event trigger 457 SysState attribute 646 System Attributes 641 system attributes 641 system states 608 SystemList attribute about 48 262 definition 637 modifying 262 SystemLocation attribute 646 SystemOwner attribute 647 systems adding from command line 276 277 adding from Java Console 218 administering from command line 276 administering from Java Console 218 bringing online in VCS Simulator 324 client process failure 554 deleting from Java Console 218 detecting failure 553 systems continued displaying node ID from command line 276 freezing from Cluster Management Console 127 freezing from Java Console 219 panic 553 quick reopen 554 starting from command line 236 states 608 unfreezing from Cluster Management Console 128 unfreezing from Java Console 219 systems and nodes 20 SystemZones attribute 638 T Table 62 590 Tag attribute 638 TargetCount attribute 638 templates accessing Template View 160 adding resources from 201 adding service groups from 180 TFrozen attribute for service groups 638 for systems 647 The 463 ToleranceLimit attribute 627 ToQ attribute 638 TriggerEvent attribute for resources 619 for service groups 638 Tri
511. rings 2 Click Search 3 Visit each search view to see the hits in each object category Your particular destination view depends on which search view was most recently visited Click the other secondary tabs Clusters Groups and so on to visit the other search views To perform a search using the Search tab 1 On the main tab bar click Search 2 On the secondary tab bar click the object category Clusters Groups and so on in which you want to begin your search 3 Inthe text entry box type one or more text strings that you want to find Administering the cluster from the Cluster Management Console 133 Conducting a search Use a space character to separate text strings 4 Check one or more of the following options and then click Search Exact Match Requires each search result to match at least one whole search string This option excludes occurrences in which a search string is part of a longer string Match All Terms Requires each search result to match all search terms strings The behavior of this option depends on the setting of the Exact Match option according to the following example Consider object1 with attribute path tmp f3 and object2 with attribute path tmp f4 Search String Exact Match Match All Terms Search results Option Setting Option Setting path f3 checked checked none path f3 checked cleared none path f3 cleared checked object1 path f3 cleared cleared object1 object2 Highlight
512. rmation about any object configured in the cluster Each level in the view hierarchy contains a set of views You can navigate to these views using the secondary tab bar The secondary tab bar is located along the top of the status pane and contains a row of one or more tabs Each tab represents a view at a level in the view hierarchy Figure 5 10 Secondary tab bar on the Cluster Summary view Summary Groups Systems Attributes Resources Group Dependency Resource Types 83 84 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console On the Manage tab the view hierarchy levels are cluster service group system resource and resource type Each level contains views represented by the tabs on the secondary tab bar Table 5 2 Levels and views in the Manage tab view hierarchy Cluster Summary Groups Systems Attributes Resources Service Group Dependency Resource Types DR Alerts Cluster Heartbeats Users Logs Service group Summary Attributes Resources Logs System Summary Attributes Logs Resource Summary Resource Dependency Attributes Logs Resource type Attributes On the secondary tab bar one tab is always active selected that tab represents your position at current level in the view hierarchy For example if you are at the startup view Cluster Summary the Summary tab is active According to the table you are in the first position at th
513. robe command 274 hares state command 523 hares undo_override command 282 hares unlink command 273 hares value command 523 hares wait command 284 hashadow process 27 hasnap backup command 245 hasnap delete command 250 hasnap exclude command 249 hastart command 236 hastart onenode command 236 hastart ts command 236 hastatus command for global clusters 525 for local clusters 257 hastatus group command 257 258 hastatus summary command 258 hastop command 237 hasys display command for global clusters 524 for local clusters 257 hasys freeze command 276 hasys list command for global clusters 524 for local clusters 257 hasys modify command 276 hasys nodeid command 276 hasys state command 524 hasys unfreeze command 276 277 hasys value command for global clusters 524 hasys wait command 284 hatype add command 282 hatype delete command 282 hatype display command 256 hatype list command 256 hatype modify command 282 hatype resources command 256 hauser add command 252 hauser addpriv command 252 hauser delete command 253 hauser delpriv command 252 253 hauser display command 253 hauser list command 253 heap size for VRTSweb 678 Heartbeat 613 Heartbeat agent 465 heartbeat attributes 655 Index 693 heartbeats modifying for global clusters 533 help accessing online 91 locating topics 91 navigating online 91 host name issues 19 T O fencing about 27 icons colors of 144 in Java Consol
514. rolling VCS behavior at the resource level m If FailOverPolicy is set to Load a NoFailover situation may occur because of restrictions placed on service groups and systems by Service Group Workload Management Resource faults resfault Trigger Offline all resources in dependent path Critical resources YES Offline entire tree Y Auto Failover A Failover based on FailOverPolicy Controlling VCS behavior 383 Controlling VCS behavior at the resource level No other resources affected No group failover No other resources affected No group failover Service group offline in Faulted state Service group offline in Faulted state nofailover trigger 384 Controlling VCS behavior Controlling VCS behavior at the resource level Disabling resources Disabling a resource means that the resource is no longer monitored by a VCS agent and that the resource cannot be brought online or taken offline The agent starts monitoring the resource after the resource is enabled The resource attribute Enabled determines whether a resource is enabled or disabled A persistent resource can be disabled when all its parents are offline A non persistent resource can be disabled when the resource is in an OFFLINE state When to disable a resource Typically resources are disabled when one or more resources in the service group encounter problems and disabling the resource is required to keep the service grou
515. rom Cluster Manager Java console 137 About the Cluster Manager Java Console About the Cluster Manager Java Console Cluster Manager Java Console offers complete administration capabilities for your cluster Use the different views in the Java Console to monitor clusters and VCS objects including service groups systems resources and resource types Many of the operations supported by the Java Console are also supported by the command line interface and Cluster Management Console See the Veritas Cluster Server Installation Guide for instructions on how to install Cluster Manager Java Console Disability compliance Cluster Manager Java Console for VCS provides disabled individuals access to and use of information and data that is comparable to the access and use provided to non disabled individuals including Alternate keyboard sequences for specific operations See Accessibility and VCS on page 679 High contrast display settings Support of third party accessibility tools Note that Symantec has not tested screen readers for languages other than English Text only display of frequently viewed windows Getting started Make sure you have the current version of Cluster Manager Java Console installed If you have a previous version installed upgrade to the latest version Cluster Manager Java Console is compatible with earlier versions of VCS Cluster Manager Java Console is supported on m AIX5 2and5
516. rom global to cluster specific hahb local ICMP AYAInterval To change the scope of an attribute to global hahb global heartbeat attribute value key key value For example type the following command to change the scope of the attribute AYAInterval from cluster specific to cluster generic hahb global ICMP AYAInterval 60 534 Administering global clusters from the command line Administering heartbeats Chapter 1 9 Setting up replicated data clusters About replicated data clusters How VCS replicated data clusters work Setting up a replicated data cluster configuration Migrating a service group Setting up a fire drill 536 Setting up replicated data clusters About replicated data clusters About replicated data clusters The Replicated Data Cluster RDC configuration provides both local high availability and disaster recovery functionality in a single VCS cluster This chapter describes how to setup RDC in a VCS environment using Veritas Volume Replicator VVR A Replicated Data Cluster RDC uses data replication to assure data access to nodes An RDC exists within a single VCS cluster In an RDC configuration if an application or a system fails the application is failed over to another system within the current primary site If the entire primary site fails the application is migrated to a system in the remote secondary site which then becomes the new primary For VVR replication to occu
517. rop down menu m The resource startup state If you want the resource to start in an enabled state after the service group comes online check Enable Resource m The resource list for the service group Click Add Resource to add the resource to the member resource list for the service group The resource list is used to populate the Resource List table In the Resource List table you can edit resource attributes or delete a resource using the buttons in the Edit and Delete columns 6 Ifyou need to link one or more resources provide the following relationship details in the Link Unlink Resource dialog box and then click Finish Otherwise click Finish now to exit the wizard and create the new service group m The parent resource Use the Parent Resource drop down menu to specify which resource must be brought online after the child resource m The child resource Use Child Resource list box to select which resource should be brought online first and then click Add Link Deleting a service group from a cluster Manually remove a service group from cluster membership You must have the role of cluster administrator to remove a service group from a cluster You cannot delete a service group if it is a child in a dependency You must first remove the dependency and then delete the service group See Unlinking service groups on page 109 To delete a service group from a cluster 1 Inthe Cluster Groups view in the Groups Listing tabl
518. roup 15 On the Completing the Application Configuration Wizard dialog box select the check box to bring the service group online on the local system Click Close Configuring NFS service groups This NFS Configuration wizard enables you to create an NFS service group which provides high availability for fileshares Before running the wizard review the resource type and the attribute descriptions of the NFS Share Mount NIC and IP agents in the Veritas Cluster Server Bundled Agents Reference Guide The wizard supports the following configurations m Multiple Share Paths m Single Virtual IP 310 Configuring applications and resources in VCS Configuring NFS service groups Prerequisites Verify the paths to be shared are exported Verify the paths to be shared are mounted and are not configured in any other service group Verify the virtual IP to be configured is up and is not configured in any other service group Disable the Service Management Facility SMF for NFS daemons for the NFS agent to work on Solaris 10 You must disable SMF for nsd and mountd before bringing the NFS agent online To do so svccfg delete f svc network nfs server default Configuring applications and resources in VCS 311 Configuring NFS service groups Running the wizard 1 Start the wizard from a node in the cluster using the following command hawizard nfs Read the information on the Welcome page and click Next On the Wi
519. roup About the VCS user privilege model 67 User privileges for OS user groups in secure clusters For example user Tom belongs to an OS user group OSUserGroup1 You can assign VCS privileges to user Tom in the following ways Cluster cluster Administrators cluster AdministratorGroups Administrator tom domain OSUserGroup1 domain Cluster cluster Operators cluster OperatorGroups Operator tom domain OSUserGroup1 domain Cluster Guest Cluster group_name Guests tom domain cluster group_name Guests OSUserGroup1 domain Group Administrator group group_name Administrators tom domain group group_name AdministratorGroups OSUserGroup1 domain Group Operator group group_name Operators tom domain group group_name OperatorGroups OSUserGroup1 domain Group Guest group group_name Guests OSUserGroup1 domain 68 About the VCS user privilege model About VCS privileges for users with multiple roles About VCS privileges for users with multiple roles Table 4 2 describes how VCS assigns privileges to users with multiple roles The scenarios describe user Tom who is part of two OS user groups OSUserGroup1 and OSUserGroup2 Table 4 2 VCS privileges for users with multiple roles Situation Multiple roles at an individual level Rule VCS grants highest privileges or a union of all the privileges to the user Tom Clust
520. roup Summary view you can choose a task from the task pane or select another service group level view using the tabs on the secondary tab bar Each view contains information and tasks for administering a single service group Adding a service group to a cluster Add and configure a service group as a member of the cluster You must have the role of cluster administrator to add a service group to a cluster To add a service group to a cluster 1 In the Cluster Groups view in the Configuration task panel select Add Service Group In the Service Group Configuration wizard read the introductory text and then click Next Administering the cluster from the Cluster Management Console Administering service groups 3 Inthe Service Group Configuration dialog box specify the following details of the new service group and then click Next The service group name Enter the name into the Service Group Name text box Service group names must Consist of only one string m Not contain special characters like period or amp ampersand m Begin with an alphabetic character The remaining characters in the name can be letters digits hyphens or underscores The service group type Click either Failover or Parallel to specify the service group type See Types of service groups on page 24 4 Inthe System List Configuration dialog box specify the following details for the target system and then click Next The target system
521. roups from coming online automatically due to events such as GAB killing HAD because to high load or HAD committing suicide to rectify unexpected error conditions System is not in RUNNING state Recommended Action Type hasys display system to verify the system is running See System states on page 608 Troubleshooting and recovery for VCS 569 Troubleshooting service groups Service group not configured to run on the system The SystemList attribute of the group may not contain the name of the system Recommended Action Use the output of the command hagrp display service_group to verify the system name Service group not configured to autostart If the service group is not starting automatically on the system the group may not be configured to AutoStart or may not be configured to AutoStart on that particular system Recommended Action Use the output of the command hagrp display service_group to verify the values of the AutoStart and AutoStartList attributes Service group is frozen Recommended Action Use the output of the command hagrp display service_group to verify the value of the Frozen and TFrozen attributes Use the command hagrp unfreeze to unfreeze the group Note that VCS will not take a frozen service group offline Failover service group is online on another system The group is a failover group and is online or partially online on another system Recommended Action Use the output of the command
522. routed over the low priority link to System2 System2 is placed in a jeopardy membership The service groups on 358 About communications membership and data protection in the cluster Summary of best practices for cluster communications System2 are autodisabled and the service group attribute AutoFailOver is set to 0 meaning the service group will not fail over on a system fault Public Network System0 System1 System2 System3 Regular membership 0 1 3 Jeopardy membership 2 When a cluster interconnect link is re established all cluster status communications revert back to the cluster interconnect and the low priority link returns to sending heartbeat signal only At this point System2 is placed back in regular cluster membership Summary of best practices for cluster communications The following are the recommended best practices for cluster communications to best support proper cluster membership and data protection Properly seed the cluster by requiring all systems and not just a subset of systems to be present in the GAB membership before the cluster will automatically seed If every system is not present manual intervention by the administrator must eliminate the possibility of a split brain condition before manually seeding the cluster Configure multiple independent communication network links between cluster systems Networks should not have a single point of failure such
523. rrency violations in multi cluster configurations If the AutoStartList attribute is set and if a group s Authority attribute is set to 1 HAD waits for the wac process to connect to the peer If the connection fails it means the peer is down and the AutoStart process proceeds If the connection succeeds HAD waits for the remote snapshot If the peer is holding the authority for the group and the remote group is online because of takeover the local cluster does not bring the group online and relinquishes authority If the Authority attribute is set to 0 AutoStart is not invoked Right of way VCS global clusters maintain resiliency using the wide area connector process and the ClusterService group The wide area connector process runs as long as there is at least one surviving node in a cluster The wide area connector its alias and notifier are components of the ClusterService group described in The ClusterService Group on page 13 VCS framework VCS agents now manage external objects that are part of wide area failover These objects include replication DNS updates and so on These agents provide a robust framework for specifying attributes and restarts and can be brought online upon fail over Action and info entry points The action and info entry points allow for detailed management of cluster and replication related objects See the Veritas Cluster Server Bundled Agents Reference Guide and the Veritas Cluster S
524. rtLimit In most cases ToleranceLimit is 0 The time it takes to detect a resource failure is the time it takes the agent monitor to detect failure plus the time to clean up the resource if the clean entry point is implemented Therefore the time it takes to detect failure depends on the MonitorInterval the efficiency of the monitor and clean if implemented entry points and the ToleranceLimit if set In some cases the failed resource may hang and may also cause the monitor to hang For example if the database server is hung and the monitor tries to query the monitor will also hang If the monitor entry point is hung the agent eventually kills the thread running the entry point By default the agent times out the monitor entry point after 60 seconds This can be adjusted by changing the MonitorTimeout attribute The agent retries monitor after the MonitorInterval If the monitor entry point times out consecutively for the number of times designated in the attribute FaultOnMonitorTimeouts the agent treats the resource as faulted The agent calls clean if implemented The default value of FaultOnMonitorTimeouts is 4 and can be changed according to the type A high value of this parameter delays detection of a fault if the resource is hung If the resource is hung and causes the monitor entry point to hang the time to detect it depends on MonitorTimeout FaultOnMonitorTimeouts and the efficiency of monitor and clean if implemented V
525. rted platforms Additionally VCS Simulator installs directories for various cluster configurations VCS Simulator creates a directory for every new simulated cluster and copies the contents of the sample_clus directory Simulator also creates a logs directory within each cluster directory for logs associated with the cluster Simulator ports VCS Simulator uses the following ports m Ports 15550 through 15558 to connect to the various cluster configurations m Ports 15560 through 15563 for the wide area connector WAC process Set the WAC port to 1 to disable WAC simulation 320 Predicting VCS behavior using VCS Simulator Administering VCS Simulator from the Java Console Administering VCS Simulator from the Java Console The Simulator Console enables you to start stop and manage simulated clusters 2 Symantec Veritas Cluster Server Simulator Of x Veritas Cluster Server miso eee Add Cluster g Make Clusters Global Global View Help Cluster Name status Platform THE amp HP_NFS fm Verify Configuration amp LN_NFS Amx E Start Cluster SOL_ORACLE Solaris aoe aunch Console GM SOL_ORA_SRDF_C1 Solaris S sOi ORA SRDE C A solaris Z C Delete Cluster Cluster Details Cluster Name AIX_NFS Port Configuration Simulator port is 15557 and Wide Area Connector port is 1 Global Option enabled No The console provides two views m Cluster View Lists
526. rting from command line 236 starting on single node 236 stopping from command line 237 stopping with other options 238 stopping without force 237 troubleshooting resources 571 troubleshooting service groups 568 VCS agent statistics 560 VCS attributes 51 VCS Simulator administering from Java Console 320 bringing systems online 324 clearing cluster faults from Java Console 325 creating power outages 325 description of 285 faulting resources 325 installing 318 saving offline configurations 325 simulating cluster faults from command line 329 simulating cluster faults from Java Console 323 starting from command line 320 VCSFeatures attribute for clusters 654 for systems 647 VCSi3Info attribute 654 VCSMode attribute 654 vector attribute dimension 52 version information retrieving 276 violation event trigger 458 virtual fire drill about 315 running 167 supported agents 316 VRTSweb adding ports 661 adding SMTP recipients 670 deleting ports 663 deleting SMTP recipients 672 logging 673 modifying log levels 674 notification for 667 ports for 660 retrieving log levels 673 retrieving ports 660 setting heap size 678 setting SMTP server 668 700 Index vxfen See fencing module vxlicinst utility 235 Ww wac 464 WACPort attribute 654 wide area connector 464 wide area failover 44 Wide Area Heartbeat agent 465 wizards GCO Configuration 472 NFS 309 Notifier Resource Configuration 225 Remote Cluster Configuration 505 Z
527. rvice group dependencies Section Administration Beyond the basics m Chapter 13 VCS event notification on page 429 m Chapter 14 VCS event triggers on page 445 428 Administration Beyond the basics Chapter VCS event notification m About VCS event notification m Components of VCS event notification m VCS events and traps m Monitoring aggregate events m Configuring notification 430 VCS event notification About VCS event notification About VCS event notification VCS provides a method for notifying important events such as resource or system faults to administrators or designated recipients VCS includes a notifier component which consists of the notifier process and the hanotify utility VCS support SNMP consoles that can use an SNMP V2 MIB The notifier process performs the following tasks m Receives notifications from HAD m Formats the notification m Generates an SNMP V2 trap or sends an email to the designated recipient or does both If you have configured owners for resources groups or for the cluster VCS also notifies owners of events that affect their resources A resource owner is notified of resource related events a group owner of group related events and so on See VCS Attributes for descriptions of the attributes that define owners for cluster objects There are four severity levels SevereError Error Warning and Information SevereError indicates the
528. s on page 291 for more information about these wizards To create a service group using the configuration wizard 1 Open the Configuration Wizard From Cluster Explorer click Configuration Wizard on the Tools menu Read the information on the Welcome dialog box and click Next Specify the name and target systems for the service group T Configuration Wizard x Specify the service group and the system list Enter the name of the service group and the service group s system related information Service Group name TestGrp Service Group Type Failover C Parallel C Hybrid Available Systems Systems for Service Group System name Startup Priority D veslinux147 lt Back Cancel Hep Enter the name of the group Click the target systems in the Available Systems box Click the right arrow to move the systems to the Systems for Service Group table To remove a system from the table click the system and click the left arrow Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every time HAD is started The priority number starting with 0 is automatically assigned to indicate the order of systems on which the service group will start in case of a failover If necessary double click the entry in the Priority column to enter a new value Click the
529. s Cluster operations are enabled or restricted depending on the privileges with which you log on VCS has three privilege levels Administrator Operator and Guest VCS provides some predefined user roles each role has specific privilege levels For example the role Guest has the fewest privileges Cluster Administrator the most See VCS user privileges administration matrices on page 583 About VCS privilege levels VCS privilege levels include m Administrators Can perform all operations including configuration options on the cluster service groups systems resources and users m Operators Can perform specific operations on a cluster or a service group m Guests Can view specified objects About user roles in VCS Table 4 1 lists the predefined VCS user roles with a summary of their associated privileges Table 4 1 User roles in VCS Cluster Cluster Administrators are assigned full privileges including making Administrator configuration read write creating and deleting groups setting group dependencies adding and deleting systems and adding modifying and deleting users All group and resource operations are allowed Users with Cluster Administrator privileges can also change other users privileges and passwords Note Cluster Administrators can change their own and other users passwords only after changing the configuration to read write mode Cluster Administrators can create and delete resou
530. s on page 387 266 Administering the cluster from the command line Administering service groups Flushing service groups As a service group is brought online or taken offline the resources within the group are brought online and taken offline If the online or offline operation hangs on a particular resource flush the service group to halt the operation on the resources waiting to go online or offline Flushing a service group typically leaves the cluster in a partial state After completing this process resolve the issue with the particular resource if necessary and proceed with starting or stopping the service group To flush a service group on a system Type the following command hagrp flush group sys system clus cluster localclus To flush all service groups on a system 1 Save the following script as haflush at the location opt VRTSvcs bin bin ksh PATH opt VRTSvcs bin SPATH export PATH if ne 1 then echo usage 0 lt system name gt exit 1 fi hagrp list while read grp sys junk do locsys S sys case Slocsys in 1 hagrp flush grp sys locsys esac done 2 Runthe script haflush systemname Administering the cluster from the command line 267 Administering agents Linking and unlinking service groups Link service groups to create a dependency between them See About service group dependencies on page 410 To link service groups Type the followi
531. s about 446 injeopardy 448 loadwarning 449 location of 446 multinicb 450 nofailover 450 postoffline 451 postonline 451 preonline 452 resadminwait 452 resfault 454 resnotoff 454 resstatechange 456 sysoffline 457 unable_to_restart_had 457 458 using 446 violation 458 F failback about 37 Failover attribute 631 FailOverPolicy attribute 631 FaultOnMonitorTimeouts attribute 622 fire drills about 483 for global clusters 483 for replicated data clusters 542 FireDrill attribute 622 Flags attribute 616 FromQ attribute 632 Frozen attribute for service groups 632 for systems 643 G GAB about 336 impact on performance 546 when a system panics 553 gab_isolate time timer 554 GCO Configuration wizard 472 global attributes 54 global cluster configuration 44 Index 691 global clusters adding from Cluster Management Console 490 adding from Java Console 505 bringing remote groups online 514 deleting from Cluster Management Console 491 deleting from Java Console 509 operation 462 prerequisites for 469 setting up 471 switching remote groups 516 upgrading to 472 user privileges 64 global heartbeats administering from Cluster Management Console 499 administering from command line 533 administering from Java Console 517 deleting from Cluster Management Console 500 deleting from Java Console 520 modifying from Cluster Management Console 501 modifying from Java Console 519 global service groups administering from Clus
532. s enter the IP address m Enter the name and password for a user having superuser administrative privileges on Web server system 5 Click OK Changing the administrative port You can change the administrative port for VRTSweb only from the command line To change the administrative port 1 Stop the Web server VRTSWEB_HOME bin webgui stop force 2 Set the administrative port to a new value VRTSWEB_ HOME bin webgui adminport new port_no 3 Restart the Web server VRTSWEB HOME bin webgui restart 664 Administering Symantec Web Server Managing VRTSweb SSL certificates Managing VRTSweb SSL certificates VRTSweb presents a self signed SSL certificate issued by Symantec when VRTSweb serves content over the secure port Note Certificate management commands are available only by the command line interface Commands that modify the certificate require a server restart Viewing SSL certificate information Display information about the configured SSL certificate To view information about the SSL certificate Run the following command on the system where VRTSweb is installed VRTSWEB_ HOME bin webgui cert display Creating a self signed SSL certificate Create a customized self signed SSL certificate for VRTSweb To create a self signed SSL certificate 1 Run the following interactive command on the system where VRTSweb is installed VRTSWEB HOME bin webgui cert create 2 Follow the prompts to creat
533. s a special purpose service group which contains resources that are required by VCS components The group contains resources for m Cluster Management Console Notification m wide area connector WAC process which is used in global clusters The ClusterService group can fail over to any node despite restrictions such as frozen The ClusterService group the first service group to come online and cannot be autodisabled The group comes online on the first node that goes in VCS configuration concepts 51 About VCS attributes the running state The VCS engine discourages taking the group offline manually About VCS attributes VCS components are configured using attributes Attributes contain data about the cluster systems service groups resources resource types agent and heartbeats if using global clusters For example the value of a service group s SystemList attribute specifies on which systems the group is configured and the priority of each system within the group Each attribute has a definition and a value Attributes also have default values assigned when a value is not specified Attribute data types VCS supports the following data types for attributes String Integer Boolean A string is a sequence of characters enclosed by double quotes A string may also contain double quotes but the quotes must be immediately preceded by a backslash A backslash is represented in a string as Quotes are not requir
534. s before the agent reports that heartbeat to the cluster is down m Type and dimension integer scalar m Default 3 AYATimeout user defined The maximum time in seconds that the agent will wait for a heartbeat AYA entry point to return ALIVE or DOWN before being canceled m Type and dimension integer scalar m Default 300 CleanTimeOut user defined Number of seconds within which the Clean entry point must complete or be canceled m Type and dimension integer scalar m Default 300 seconds ClusterList user defined List of remote clusters m Type and dimension string keylist m Default 656 VCS attributes Heartbeat attributes Table D 6 Heartbeat attributes InitTimeout user defined Number of seconds within which the Initialize entry point must complete or be canceled m Type and dimension integer scalar m Default 300 seconds LogDbg user defined The log level for the heartbeat m Type and dimension string keylist m Default State The state of the heartbeat m Type and dimension integer scalar m Default Not applicable StartTimeout user defined Number of seconds within which the Start entry point must complete or be canceled m Type and dimension integer scalar m Default 300 seconds StopTimeout user defined Number of seconds within which the Stop entry point must complete or be canceled without stopping
535. s disabled This is expected agent behavior VCS assumes that data is being read from or written to the volumes and does not declare the resource as offline This prevents potential data corruption that could be caused by the disk group being imported on two hosts You can deport a disabled disk group when all I O operations are completed or when all volumes are closed You can then reimport the disk group to the same system Reimporting a disabled disk group may require a system reboot Note A disk group is disabled if data including the kernel log configuration copies or headers in the private region of a significant number of disks is invalid or inaccessible Volumes can perform read write operations if no changes are required to the private regions of the disks Troubleshooting and recovery for VCS 573 Troubleshooting notification Troubleshooting notification Occasionally you may encounter problems when using VCS notification This section cites the most common problems and the recommended actions Bold text provides a description of the problem Notifier is configured but traps are not seen on SNMP console Recommended Action Verify the version of SNMP traps supported by the console VCS notifier sends SNMP v2 0 traps If you are using HP OpenView Network Node Manager as the SNMP verify events for VCS are configured using xnmevents You may also try restarting the OpenView daemon ovw if after merging VCS events in vcs_
536. s running thereby interrupting HAD In this scenario HAD cannot function until the interrupt handler completes its operation To overcome this issue VCS provides the option of running HAD on a specific processor VCS masks off all interrupts on that processor set Configure HAD to run on a specific processor by setting the CPUBinding attribute See System attributes on page 641 To modify the CPUBinding attribute Type the following command hasys modify sysl CPUBinding BindTo CPUNUM NONE ANY CPUNum number m The value NONE indicates HAD does not use CPU binding m The value ANY indicates that HAD binds to any available CPU m The value CPUNUM indicates that HAD binds to CPU specified in the CPUNum attribute m The variable number specifies the number of the CPU Note that you cannot use the add update or delete keys options for the hasys modify command to modify the CPUBinding attribute VCS performance considerations 559 Monitoring CPU usage Monitoring CPU usage VCS includes a system attribute CPUUsageMonitoring which monitors CPU usage on a specific system and notifies the administrator when usage has been exceeded The default values for the CPUUsageMonitoring attribute are m Enabled 0 NotifyThreshold 0 NotifyTimeLimit 0 m ActionThreshold 0 m ActionTimeLimit 0 m Action NONE The values for ActionTimeLimit and NotifyTimeLimit represent the time in seconds The values fo
537. s the server is 80 percent loaded it must inform VCS that the DynamicLoad value is 160 not 80 About overload warning Overload warning provides the notification component of the Load policy When a server sustains the preset load level set by the attribute LoadWarningLevel for a preset time set by the attribute LoadTimeThreshold VCS invokes the loadwarning trigger See Using event triggers on page 446 See System attributes on page 641 The loadwarning trigger is a user defined script or application designed to carry out specific actions It is invoked once when system load exceeds the LoadWarningLevel for the LoadTimeThreshold It is not invoked again until the LoadTimeCounter which determines how many seconds system load has been above LoadWarningLevel is reset About system limits and service group prerequisites Limits is a system attribute and designates which resources are available on a system including shared memory segments and semaphores Prerequisites is a service group attribute and helps manage application requirements For example a database may require three shared memory segments and 10 semaphores VCS Load policy determines which systems meet the application criteria and then selects the least loaded system If the prerequisites defined for a service group are not met on a system the service group cannot be brought online on the system When configuring these attributes define the service gro
538. sable selected resources 1 In the Cluster Resources view in the Resources Listing table click a linked resource name In the Operations task panel click Enable or Disable In either the Enable Resource or Disable Resource dialog box click OK Bringing a resource online Manually bring a resource online You must have the role of cluster operator or group operator to bring a resource online 1 In the Cluster Resources view in the Resources Listing table click the linked name of the resource you want to bring online In the Resource Summary view in the Operations task panel click Online In the Online Resource dialog box select the system on which to bring the resource online from the drop down menu Click OK Taking a resource offline Manually take a resource offline You must have the role of cluster operator or group operator to take a resource online 1 In the Cluster Resources view in the Resources Listing table click the linked name of the resource you want to bring online In the Resource Summary view in the Operations task panel click Online In the Online Resource dialog box select the system on which to bring the resource online from the drop down menu Click OK Administering the cluster from the Cluster Management Console 119 Administering resources Taking a resource offline and propagating the command Take a resource and all of its dependents offline Use this task to propagate th
539. scalar m Default 1 FromQ system use only Indicates the system name from which the service group is failing over This attribute is specified when service group failover is a direct consequence of the group event such as a resource fault within the group or a group switch m Type and dimension string association m Default Not applicable Frozen user defined Disables all actions including autostart online and offline and failover except for monitor actions performed by agents This convention is observed by all agents supplied with VCS m Type and dimension boolean scalar m Default 0 not frozen GroupOwner user defined VCS sends email notification to the person designated in this attribute when an event occurs related to the service group VCS also logs the owner name when an event occurs m Type and dimension string scalar m Default If GroupOwner is not specified in main cf the default value is unknown Guests user defined List of operating system user account groups that have Guest privileges on the service group This attribute is valid only in secure clusters m Type and dimension string keylist m Default Table D 3 VCS attributes 633 Service group attributes Service group attributes IntentOnline system use only Indicates whether to keep service groups online or offline VCS sets this attribute to 1 if an attempt has been made to
540. se the view of the command m Click OK To add a resource from Command Center 1 ou A WwW N File View Help In the Command Center configuration tree expand Commands gt Configuration gt Cluster Objects gt Add Resource or Click Add resource in the Command Center toolbar VCSHP10_71_0214a Command Center admin Cluster Administrator BRE GSU SAG BESO fp Commands e B Configuration ws Add Resource H E Configuration File B Cluster Objects Group name TestGrp Ga Add Ser 3 ce Resource name ffestres Ga Delete Service Group 7 Delete Resource Resource Type Ficonoff z 9 Delete System 2 Attributes 2 Dependencies Attribute name Type Dimension f Operations PathName String calar I Critical Enabled ares add TestRes FileOnOff TestGrp ares modify TestRes Critical 1 ares modify TestRes PathName ares modify TestRes Enabled 0 Select the service group to contain the resource Enter the name of the resource Click the resource type Edit resource attributes according to your configuration The Java Console also enables you to edit attributes after adding the resource Select the Critical and Enabled check boxes if applicable The Critical option is selected by default A critical resource indicates the service group is faulted when the resource or any resource it depends on faults An enabled resource indicates agents monitor the resource you m
541. signated in the SystemList attribute but the node is visible m Type and dimension boolean scalar m Default 0 Table D 3 VCS attributes 629 Service group attributes Service group attributes AutoFailOver user defined Indicates whether VCS initiates an automatic failover if the service group faults m Type and dimension boolean scalar m Default 1 enabled AutoRestart user defined Restarts a service group after a faulted persistent resource becomes online See About service group dependencies on page 410 Note This attribute applies only to service groups containing persistent resources m Type and dimension boolean scalar m Default 1 enabled AutoStart user defined Designates whether a service group is automatically started when VCS is started m Type and dimension boolean scalar m Default 1 enabled AutoStartIfPartial user defined Indicates whether to initiate bringing a service group online if the group is probed and discovered to be in a PARTIAL state when VCS is started m Type and dimension boolean scalar m Default 1 enabled AutoStartList user defined List of systems on which under specific conditions the service group will be started with VCS usually at system boot For example if a system is a member of a failover service group s AutoStartList attribute and if the service group is not already running on another system
542. sole on page 202 See Adding a RemoteGroup resource from Cluster Management Console on page 113 About the ControlMode attribute In the ControlMode attribute you can use these values depending on your needs OnOff MonitorOnly and OnlineOnly OnOff Select the OnOff value of this attribute when you want the RemoteGroup resource to manage the remote service group completely In case of one to one mapping set the value of the AutoFailover attribute of the remote service group to 0 This avoids unnecessary onlining or offlining of the remote service group MonitorOnly Select the MonitorOnly value of this attribute when you want to monitor the state of the remote service group When you choose the MonitorOnly attribute the RemoteGroup agent does not have control over the remote service group and cannot bring it online nor take it offline The remote service group should be in an ONLINE state before you bring the RemoteGroup resource online Symantec recommends that the AutoFailover attribute of the remote service group be set to 1 OnlineOnly Select the OnlineOnly value of this attribute when the remote service group takes a long time to come online or to go offline When you use OnlineOnly for Configuring applications and resources in VCS 297 Configuring the RemoteGroup agent the ControlMode attribute a switch or fail over of the local service group with VCSSysName set to ANY does not cause the remote service gr
543. source Link resources to create a dependency You must have the role of cluster administrator or service group administrator to link resources To link two resources 1 In the Resource Summary view in the Configuration task panel click Link Unlink Resource In the Link Unlink Resource dialog box specify the following details for the resource link and then click OK m The parent resource Select the parent resource from the drop down menu This resource must be brought online before the child resource The Parent Resource menu is populated using the resource list of the service group in which the resources are configured m The child resource Select the child resource from the scroll box This resource is brought online after the parent resource The Child Resource scroll box is populated using the resource list of the service group in which the resources are configured This list is filtered depending upon the 122 Administering the cluster from the Cluster Management Console Administering resources selection you make in Parent Resource Click Add Link after you select a child resource to add the link to the Currently Linked table Viewing resource dependencies View a graphical representation of the dependencies that exist for a resource To view service group dependencies and get status information 1 2 3 4 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Resources
544. source s type plus their default values are copied to the new resource Three attributes are also created by the system and added to the resource m Critical default 1 If the resource or any of its children faults while online the entire service group is marked faulted and failover occurs AutoStart default 1 If the resource is set to AutoStart it is brought online in response to a service group command All resources designated as AutoStart 1 must be online for the service group to be considered online This attribute is unrelated to AutoStart attributes for service groups m Enabled If the resource is set to Enabled the agent for the resource s type manages the resource The default is 1 for resources defined in the configuration file main cf 0 for resources added on the command line Note Adding resources on the command line requires several steps and the agent must be prevented from managing the resource until the steps are completed For resources defined in the configuration file the steps are completed before the agent is started Administering the cluster from the command line 269 Administering resources Deleting resources Delete resources from a service group To delete a resource Type the following command hares delete resource Note that deleting a resource won t take offline the object being monitored by the resource The object remains online outside the control and mon
545. source fails to come online VCS behavior diagrams This section describes the default functionality of VCS when resources fault The illustration displays the symbols used in this section Resource Configuration Actions Resource Color Code Ce Critical Resource Online Cy Resource Going Offline Offline t Resource Coming Online Faulted Controlling VCS behavior 363 About VCS behavior on resource faults Scenario Resource with critical parent faults The service group in the following example has five resources of which resource R1 is configured as a critical resource 4 When resource R2 faults the fault is propagated up the dependency tree to resource R1 When the critical resource R1 goes offline VCS must fault the service group and fail it over elsewhere in the cluster VCS takes other resources in the service group offline in the order of their dependencies After taking resources R3 R4 and R5 offline VCS fails over the service group to another node Scenario Resource with non critical parent faults The service group in the following example does not have any critical resources When resource R2 faults the engine propagates the failure up the dependency tree Neither resource R1 nor resource R2 are critical so the fault does not result in the tree going offline or in service group failover 364 Controlling VCS behavior About VCS behavior on resource faults Scenario Resource wi
546. source to the ADMIN_WAIT state hagrp clearadminwait fault group sys system This command has the following results m If the resadminwait trigger was called for reasons 0 or 1 the resource state is set aS ONLINE UNABLE_TO_OFFLINE m If the resadminwait trigger was called for reasons 2 3 or 4 the resource state is set as FAULTED Please note that when resources are set as FAULTED for these reasons the clean entry point is not called Verify that resources in ADMIN WAIT are in clean OFFLINE state prior to invoking this command When a service group has a resource in the ADMIN_WAIT state the following service group operations cannot be performed on the resource online offline switch and flush Also you cannot use the hastop command when resources are in the ADMIN_WAIT state When this occurs you must issue the hastop command with force option only 370 Controlling VCS behavior Controlling VCS behavior at the service group level Controlling fault propagation The FaultPropagation attribute defines whether a resource fault is propagated up the resource dependency tree It also defines whether a resource fault causes a service group failover m If the FaultPropagation attribute is set to 1 default a resource fault is propagated up the dependency tree If a resource in the path is critical the service group is taken offline and failed over provided the AutoFailOver attribute is set to 1 m Ifthe FaultPropagation
547. sources m Administering resource types m Administering systems m Administering attributes m Viewing logs m Conducting a search 70 About Veritas Cluster Management Console Administering the cluster from the Cluster Management Console About Veritas Cluster Management Console The Veritas Cluster Management Console is a high availability cluster administration and management solution It can be configured to locally manage a single cluster or to centrally manage multiple clusters These two modes of configuration and operation are called single cluster and multi cluster The differences between the modes are significant Multi cluster mode offers greater capability on a larger scale Both modes require different installation and configuration procedures The following table summarizes some of the differences between single cluster and multi cluster mode Operational mode Local management of one cluster single cluster mode Centralized comprehensive enterprise wide administration of multiple clusters multi cluster mode Configurational description The Cluster Management Console is installed along with VCS on each node in the cluster and is configured for failover It is integrated with VCS as part of the ClusterService service group The Cluster Management Console offers robust cluster management capability and can be run from any supported Web browser on any system When configured to centrally administer multiple
548. sources by checking the check box at the top 2 Inthe Operations task panel click Clear Fault Probing a resource Probe a resource to confirm that it is properly configured and ready to bring online To probe a resource 1 Inthe Cluster Resources view in the Resources Listing table click the linked name of the resource that you want to probe 2 Inthe Resource Summary view in the Operations task panel click Probe Administering the cluster from the Cluster Management Console 121 Administering resources In the Probe Resource dialog box select the system on which you want to probe the specified resource from the drop down menu and then click OK The dialog box displays a message indicating success or failure Success indicates that the resource is properly configured and if not already online is ready to be brought online Deleting a resource from a service group Delete a resource from membership in a service group You must have the role of cluster administrator or service group administrator to delete a resource from a service group To delete a resource from a service group 1 Linking resources In either the Cluster Resources or Group Summary view in the Resource Listing table click the linked name of the resource that you want to delete In the Resource Summary view in the Configuration task panel click Delete Resource In the Delete Resource dialog box click OK to confirm that you want to delete the re
549. specifies the number of times the Online entry point is retried if the initial attempt to bring a resource online is unsuccessful When the OnlineRetryLimit set to a non zero value the agent framework calls the Clean entry point before rerunning the Online entry point Setting the ManageFaults attribute to NONE prevents the Clean entry point from being called and also prevents the Online operation from being retried About the Conflnterval attribute The ConfInterval attribute defines how long a resource must remain online without encountering problems before previous problem counters are cleared The attribute controls when VCS clears the RestartCount ToleranceCount and CurrentMonitorTimeoutCount values About the ToleranceLimit attribute The ToleranceLimit attribute defines the number of times the Monitor routine should return an offline status before declaring a resource offline This attribute is typically used when a resource is busy and appears to be offline Setting the attribute to a non zero value instructs VCS to allow multiple failing monitor 374 Controlling VCS behavior Controlling VCS behavior at the resource level cycles with the expectation that the resource will eventually respond Setting a non zero ToleranceLimit also extends the time required to respond to an actual fault About the FaultOnMonitorTimeouts attribute The FaultOnMonitorTimeouts attribute defines whether VCS interprets a Monitor entry point timeou
550. sssessssssessseseesesesereeseserersssrsesesneseseseesesesee 566 Troubleshooting VCS startup ccccecssssessesesesssseseeseseseeseseseeseseeeeseeeeseseneees 567 Troubleshooting service groups e ssssessssssesesssseseseserseseserersrseessrsrsesssnssesesse 568 Troubleshooting resources ceeesseseseeseseteeeeseeeeeees Troubleshooting notification Troubleshooting VCS configuration backup and restore ou 574 Troubleshooting and recovery for global clusters Troubleshooting licensing ceecessessseesesseceseeeeseceseseeecsesceseseseeseesaesenees Appendixes VCS user privileges administration matrices About administration matrices ceeseeseseseseeeseececececeseseseeeeeseeeeeeeseseneeees 583 Administration matrices cscsessssssecscecesesesescseeseseceseseeesesescseeceeeeeeeeesesesens 584 Configuring VCS in non global zones About VCS support for non global zones ccceessseseseseessesesesseeseseseseseeees 592 Configuring VCS in ZONES esesesesesesesesesessesessseeeessesesesessesseeenseseseseseseeeseees 593 Verifying the zone Configuration ccccceescssssesececesesesesesesssesesseeseseseseseeees 603 Maintenance tasks ceccccscesssessssesesceseseeceseseeseseseesesesceseseseeseeeeesesasseessenesaees 603 Troubleshooting information ccceesessssesessceceseseseseetsessceesscseseseetseeseees 604 Appendix C Appendix D Appendix E Appendix F Glossary Index Contents Cluster and
551. st The installer also verifies that Veritas Cluster Server is running on all systems in the cluster Press Enter to confirm that you want to enable the Authentication Service Would you like to enable Veritas Security Services on this cluster y n q y y Proceed with the configuration tasks Enter credentials used when you set up the root broker Review the output as the installer modifies the Veritas Cluster Server configuration files to enable the Authentication Service and starts Veritas Cluster Server in a secure mode Administering the cluster from the command line 281 Enabling and disabling Security Services The installer creates the Security service group creates Authentication Server credentials on each node in the cluster and Web credentials for Veritas Cluster Server users and sets up trust with the root broker To disable Symantec Product Authentication Services on a Veritas Cluster Server cluster 1 Verify you have a root broker configured See the Veritas Cluster Server Installation Guide for instructions Start the installvcs program with the security option installvcs security Review the output as the installer displays the directory where the logs are created Enter 2 to disable the Authentication Service on the cluster 1 Enable Veritas Security Services on a VCS Cluster 2 Disable Veritas Security Services on a VCS Cluster 3 Install Veritas Security Services Root Broker Select the S
552. stance is started on a node in the linked back up cluster Replication Replication is the synchronization of data between systems where shared storage is not feasible The systems that are copied may be in local backup clusters or remote failover sites The major advantage of replication when compared to traditional backup methods is that current data is continuously available Resources Individual components that work together to provide application services to the public network A resource may be a physical component such as a disk or network interface card a software component such as Oracle8i or a Web server or a configuration component such as an IP address or mounted file system Resource Dependency A dependency between resources is indicated by the keyword requires between two resource names This indicates the second resource the child must be online before the first resource the parent can be brought online Conversely the parent must be offline before the child can be taken offline Also faults of the children are propagated to the parent Resource Types Each resource in a cluster is identified by a unique name and classified according to its type VCS includes a set of predefined resource types for storage networking and application services Root Broker The first authentication broker which has a self signed certificate The root broker has a single private domain that holds only the names of brokers that s
553. states GCO heartbeat states States Service group is online Service group is offline Service group is auto disabled Service group has faulted Service group has faulted and cannot be failed over anywhere Service group is restarting Service group is being switched Service group concurrency violation Service group is restarting in response to persistent resource going online Service group attribute value does not match corresponding remote group attribute value Global group concurrency violation VCS is up on the first node in the Cluster VCS is being restarted by hashadow VCS is in jeopardy VCS has faulted Anode running VCS has joined cluster VCS has exited manually CPU Usage exceeded the threshold on the system Cluster has lost heartbeat with remote cluster Heartbeat with remote cluster is alive VCS event notification 443 Monitoring aggregate events Monitoring aggregate events This section describes how you can detect aggregate events by monitoring individual notifications How to detect service group failover VCS does not send any explicit traps when a failover occurs in response to a service group fault When a service group faults VCS generates the following notifications if the AutoFailOver attribute for the service group is set to 1 m Service Group Fault for the node on which the service group was online and faulted m Service Group Offline for the node on which the service group faulted m Ser
554. stem or the host name of a cluster system Verify the port number Choose to connect to the remote cluster with the credentials used for the current cluster connection or enter new credentials including the user name password and the domain If you have connected to the remote cluster using the wizard earlier you can use the credentials from the previous connection Click OK 6 Click Finish 511 512 Administering global clusters from Cluster Manager Java console Administering global service groups Administering global service groups After connecting clusters in a global cluster environment use the Global Group Configuration Wizard to convert a local service group that is common to the global clusters to a global group This wizard also enables you to convert global groups into local groups Administering global groups requires the following conditions A group that will serve as the global group must have the same name across all applicable clusters You must know the user name and password for the administrator for each cluster in the configuration Use Cluster Explorer to bring a global group online and take a global group offline on a remote cluster Converting local and global groups 1 From Cluster Explorer click Configure Global Groups on the Edit menu or From the Cluster Explorer configuration tree right click the service group click Configure As Global or Make Local and proceed to step 3
555. stem on shared storage for the zone root The file system that is to contain the zone root may be in the disk group as the file system containing the shared data On the system where the file system is mounted create the zone with the zonecfg command Set the zonepath parameter to specify a location for the zone root Set a virtual IP address of the system to be the IP address of the zone and define the device name for the NIC associated with the IP address Exit the zonecfg configuration Use the zonecfg command to display information about the zone s properties and their values Create the actual zone root directory Set permissions for the zone root directory Repeat the previous zone creation steps on each other system in the service group s SystemList Mount the file system containing on shared storage on one of the systems that share the storage Install the non global zone using zoneadm List the zones including the global zone Boot the zone On one system mount the file system containing the application s data on shared storage Make sure the zone created from the first system is in the installed state on all other systems in the service group s System List Configuring VCS in non global zones 599 Configuring VCS in zones Installing the application Install the application to be clustered in the zone If you have created zones on each node install the application identically on each node If
556. stem or any system in the remote cluster m Choose the Any System option to enable the RemoteGroup resource to monitor the state of the remote service group irrespective of the system on which it is online m Choose the Specific System option to enable the RemoteGroup resource to monitor the state of the remote group on a specific system in the remote cluster Both service groups must be configured on the same number of systems This option provides one to one mapping between the local and remote systems The Local Systems list displays the systems on which the RemoteGroup resource is configured Click the fields under the Remote Systems list and select the systems from drop down list If the remote group fails over to another system in 204 Administering the cluster from Cluster Manager Java console Administering resources the remote cluster the RemoteGroup resource will also fail over to the corresponding system in the local cluster m Click Next Review the text in the dialog box and click Finish to add the RemoteGroup resource to the specified service group in the local cluster You must now create dependencies between the RemoteGroup resource and the existing resources of the service group See Linking resources on page 212 Deleting a resource To delete a resource from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the resource or Click a service group in the configura
557. stem2 System0O System3 Regular membership 0 1 3 Cluster 1 Both links to System2 fail at the same time System2 was never in jeopardy membership Without a low priority link the cluster splits into two subclusters where SystemO 1 and 3 are in one subcluster and System2 is in another This is a split brain scenario Both links to System2 fail at different times System2 was in a jeopardy membership when the second link failed and therefore the service groups that were online on System2 were autodisabled No other system can online these service groups without administrator intervention Systems 0 1 and 3 form a mini cluster System2 forms another single system mini cluster All service groups that were present on systems 0 1 and 3 are autodisabled on System2 356 About communications membership and data protection in the cluster Examples of VCS operation without I O fencing Four system cluster with low priority link Consider a four system cluster that has two private cluster interconnect heartbeat links and one public low priority link Public Network SystemO System1 System2 System3 a l i a e Regular membership 0 1 2 3 Cluster status on private networks Heartbeat only on public network Cluster interconnect link failure In this example a link to System2 fails leaving System2 with one cluster interconnect lin
558. ster Administrators can monitor the health of the remote cluster via heartbeat commands and mechanisms such as Internet satellites or storage replication technologies Heartbeat commands are applicable only on the cluster from which they are issued Note You must have Cluster Administrator privileges to add delete and modify heartbeats The following commands are issued from the command line For a list of heartbeats configured on the local cluster hahb list conditionals The variable conditionals represents the conditions that must be met for the heartbeat to be listed To display information on heartbeats configured in the local cluster hahb display heartbeat If heartbeat is not specified information regarding all heartbeats configured on the local cluster is displayed To display the state of the heartbeats in remote clusters hahb state heartbeat clus cluster For example to get the state of heartbeat ICMP from the local cluster to the remote cluster phoenix hahb state ICMP clus phoenix Administering global clusters from the command line 527 Global querying To display an attribute value of a configured heartbeat hahb value heartbeat attribute clus cluster The value option provides the value of a single attribute for a specific heartbeat The cluster name must be specified for cluster specific attribute values but not for global For example to display the value of the ClusterList
559. ster Manager Java console About Cluster Monitor Cluster monitor panels To administer a cluster add a cluster panel or reconfigure an existing cluster panel in Cluster Monitor Each panel summarizes the status of the connection and components of a cluster Monitoring the cluster connection with Cluster Monitor The right pane of a panel in Cluster Monitor displays the status of the connection to a cluster An inactive panel will appear grey until the user logs on and connects to the cluster To alter the connection to a cluster right click a panel to access a menu m The menu on an active panel enables you to log off a cluster m The menu on an inactive panel enables you to log on to a cluster configure the cluster and delete the cluster from Cluster Monitor Menus are enabled when the Cluster Monitor display appears in the default expanded view If you activate a menu on a collapsed scrolling view of Cluster Monitor the scrolling stops while accessing the menu If the system to which the console is connected goes down a message notifies you that the connection to the cluster is lost Cluster Monitor tries to connect to another system in the cluster according to the number of Failover retries set in the Connectivity Configuration dialog box The panels flash until Cluster Monitor is successfully connected to a different system If the failover is unsuccessful a message notifies you of the failure and the panels turn grey Monito
560. ster list for global service groups or global heartbeats or if the cluster is in the RUNNING BUILD INQUIRY EXITING Or TRANSITIONING states To take the wac resource offline 1 4 In the Cluster Resources view in the Resources Listing table click the linked name of the wac resource In the Resource Summary view in the Operations task panel click Offline In the Offline Resource dialog box select the system on which you want to take the wac resource offline from the drop down menu Click OK To remove a cluster from the cluster list for a global group 1 In the Group Summary view in the Configuration task panel click Configure Global Group In the Global Group Configuration wizard read the introductory information and then click Next In the Cluster List Configuration dialog box under Selected Clusters click the name of the cluster that you want to remove Click the lt left arrow button This action removes the cluster from the cluster list for the selected global service group and places it back under Available Clusters To remove all clusters click the lt lt double left arrow button Select the failover policy for the global service group and then click Next m Manual prevents the service group from automatically failing over to another cluster m Auto enables the service group to automatically fail over to another cluster if it is unable to fail over within the cluster or if the entire clus
561. stering clusters Enabling and disabling Security Services Administering simulated clusters from the command line 234 Administering the cluster from the command line About administering VCS from the command line About administering VCS from the command line This chapter describes commonly used VCS commands For more information about specific commands or their options see their usage information or the man pages associated with the commands Most commands can be entered from any system in the cluster when VCS is running The command to start VCS is typically invoked at system startup How VCS identifies the local system VCS checks the file VCS_CONF conf sysname If this file does not exist the local system is identified by its node name To view the system s node name type uname n The entries in this file must correspond to those in the files etc 11lthosts and etc 11ttab About specifying values preceded by a dash When specifying values in a command line syntax you must prefix values beginning with a dash with a percentage sign If a value begins with a percentage sign you must prefix it with another percentage sign The initial percentage sign is stripped by HAD and does not appear in the configuration file About the modify option Most configuration changes are made using the modi fy options of the commands haclus hagrp hares hasys and hat ype Specifically the modi fy option of these comm
562. stering resources Administering resources The Cluster Management Console enables you to add and configure resources according to the requirements of the service groups that the resource supports Use the following views to administer a resource Cluster Resources This is a cluster level view of all resources in the cluster Resource Summary This is the first view at the resource level in the Cluster Management Console Group Summary This is the first view at the service group level in the Cluster Management Console Group Resources This is a service group level view of all resources in a service group and any dependencies that exist among them To navigate to the Cluster Resources view 1 2 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Resources In the Cluster Resources view the Resources Listing table lists all resources in the selected cluster The task panels contain tasks that you can perform in a one to many fashion on the listed resources To navigate to the Resource Summary view 1 2 3 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Resources In the Cluster Resources view in the Resources Listing table click a linked resource name In the Resource Summary view you can choose a task from the task pane or select another resource level view using the tabs on the secondary tab bar Each view contains information
563. stering service groups Adding a RemoteGroup resource from Cluster Management Console A RemoteGroup enables you to monitor manage a service group on a remote cluster You must have the role of cluster administrator or service group administrator to configure a remote group resource Use the RemoteGroup agent to create a dependency between a local and remote service group Typically the remote group resource depends on an IP resource in the local service group See the Veritas Cluster Server Bundled Agents Reference Guide for more information on this agent See also Configuring the RemoteGroup agent on page 296 Note Although this topic refers to a remote service group the Add Remote Group task is not part of VCS global cluster configuration This task is intended to provide a simple way to monitor and manage a service group on another cluster To configure a remote group resource on a local cluster 1 Inthe Group Summary view in the Configuration task panel click Add Remote Group Resource 2 Inthe Remote Group Resource wizard read the introductory information and then click Next 3 Inthe RemoteGroup Resource dialog box enter a name in the RemoteGroup Resource Name box and then click Next 4 Inthe Remote Cluster Authentication dialog box enter the host IP address and all of the required user authentication information If you are logging in to a secure cluster check Select to enter domain name for Secure Cluster and
564. stname of the SMTP server to be used for notification An empty string disables notification m Enter the name and password for a user having superuser administrative privileges on the Web server system m Click OK Retrieving SMTP settings Retrieve configuration information for VRTSweb SMTP notification To retrieve SMTP recipients from the command line Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui smtp listrcpt This command retrieves the email addresses of the configured recipients the notification severity level and the notification locale Administering Symantec Web Server 669 Configuring SMTP notification for VRTSweb To retrieve SMTP recipients from the Web console 1 Access the Web server on a configured port For example http hostname 8181 2 Click the Configuration tab 3 The SMTP Recipients table on the right side of the Configuration page lists the configured SMTP recipients To retrieve the list of installed locales Run the following command VRTSWEB_HOME bin webgui smtp listlocales 670 Administering Symantec Web Server Configuring SMTP notification for VRTSweb Adding an SMTP recipient Add a user to receive SMTP notifications from VRTSweb To add an SMTP recipient from the command line Run the following command on the system where VRTSweb is installed VRTSWEB HOME bin webgui smtp addrcpt email severity lt INFO WARN ERROR SE
565. sure only one cluster survives a split of the private network I O fencing also provides the ability to perform SCSI 3 persistent reservations on failover The shared disk groups offer complete protection against data corruption by nodes assumed to be excluded from cluster membership See About the I O fencing algorithm on page 346 28 Introducing Veritas Cluster Server Logical components of VCS About security services VCS uses the Symantec Product Authentication Service to provide secure communication between cluster nodes and clients including the Java and the Web consoles VCS uses digital certificates for authentication and uses SSL to encrypt communication over the public network In secure mode m VCS uses platform based authentication m VCS does not store user passwords m All VCS users are system and domain users and are configured using fully qualified user names For example administrator vcsdomain VCS provides a single sign on mechanism so authenticated users need not sign on each time to connect to a cluster VCS requires a system in your enterprise to be configured as a root broker Additionally all nodes in the cluster must be configured as authentication brokers m A root broker serves as the main registration and certification authority it has a self signed certificate and can authenticate other brokers The root broker may be a system in the cluster but the recommended practice is to have a single roo
566. sys system A group can be brought online only if it is enabled To disable a service group Type the following command hagrp disable service group sys system A group cannot be brought online or switched if it is disabled To enable all resources in a service group Type the following command hagrp enableresources service group Administering the cluster from the command line 265 Administering service groups To disable all resources in a service group Type the following command hagrp disableresources service group Agents do not monitor group resources if resources are disabled Clearing faulted resources in a Service group Clear a resource to remove a fault and make the resource available to go online To clear faulted non persistent resources in a service group Type the following command hagrp clear service group sys system Clearing a resource initiates the online process previously blocked while waiting for the resource to become clear m If system is specified all faulted non persistent resources are cleared from that system only m If system is not specified the service group is cleared on all systems in the group s SystemList in which at least one non persistent resource has faulted To clear resources in ADMIN_WAIT state in a service group Type the following command hagrp clearadminwait fault service group sys system See Changing agent file paths and binarie
567. system to the service group s system list 1 Inthe System Manager dialog box click the system in the Available Systems box T System Manager x er Service Group newgroup2 Available Systems Systems for Service Group iweshp71 System name Startup gt 4 2 Click the right arrow to move the available system to the Systems for Service Group table 3 Select the Startup check box to add the systems to the service groups AutoStartList attribute This enables the service group to automatically come online on a system every time HAD is started 4 The priority number starting with 0 is assigned to indicate the order of systems on which the service group will start in case of a failover If necessary double click the entry in the Priority column to enter a new value 5 Click OK To remove a system from the service group s system list 1 Inthe System Manager dialog box click the system in the Systems for Service Group table 2 Click the left arrow to move the system to the Available Systems box 3 Click OK 196 Administering the cluster from Cluster Manager Java console Administering service groups Creating service groups with the configuration wizard This section describes how to create service groups using the configuration wizard Note VCS also provides wizards to create service groups for applications and NFS shares See the chapter Configuring Application and NFS Service Group
568. systems if they do not exist check box if desired m Click Next Configuring applications and resources in VCS 313 Configuring NFS service groups Configure the NFS and NFSRestart resources VCS NFS Configuration Wizard NFSRestart Configuration Configure NFS and NFSRestart resource Specify the options for NFS and NFS Restart NFS Lock Failover Z Enable Lock recovery Mount Point Nar NFS Server Number of Threads 16 Discover Mounts More information lt Back Next gt cancer m Specify whether the NFS locks should be recovered after a failover m Specify the path name of the directory to store the NFS locks for all the shared file systems m Specify the number of concurrent NFS requests the server can handle m Click Next Configure the IP and NIC resources for the shares VCS NFS Configuration Wizard Network Configuration Configure the IP and NIC resources for the service group Virtual IP Address 10 212 88 29 discover ip Subnet Mask 255 255 254 0 System Name Ethernet Name vessune4 vessun85 Discover NIC E More information lt Back Next gt cancer m From Virtual IP Address list select the virtual IP for a mount If the virtual IP address for a share does not appear in the list click Discover IP to discover virtual IPs Note that the wizard discovers all IP addresses that existed when you started the wizard
569. systems in the same SystemZone and identifies systems that meet the group s prerequisites In this case LargeServer1 and LargeServer2 meet the required Limits Database3 is brought online on LargeServerl1 This results in the following configuration Server AvailableCapacity CurrentLimits Online Groups LargeServer1 0 ShrMemSeg 5 Databasel Database3 Semaphores 10 Processors 6 LargeServer2 100 ShrMemSeg 10 Database2 Semaphores 20 Processors 12 In this scenario further failure of either system can be tolerated because each has sufficient Limits available to accommodate the additional service group Cascading failure scenario If the performance of a database is unacceptable with two database groups running on a single server the SystemZones policy can help expedite performance Failing over a database group into the application zone has the effect of resetting the group s preferred zone For example in the above scenario Database3 was moved to LargeServer1 The administrator could reconfigure the application zone to move two application groups to a single system The database application can then be switched to the empty application server MedServer1 MedServer5 which would put Database3 in Zonel application zone If a failure occurs in Database3 the group selects the least loaded server in the application zone for failover Chapter 1 2 The role of service group dependencies About service group dependen
570. t LargeServerl LargeServer2 LargeServer3 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 5 Semaphores 10 Processors 6 Controlling VCS behavior 405 Sample configurations depicting workload management group Database3 SystemList LargeServerl LargeServer2 LargeServer3 MedServerl MedServer2 MedServer3 MedServer4 MedServer5 SystemZones LargeServer 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList LargeServerl LargeServer2 LargeServer3 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 5 Semaphores 10 Processors 6 group Applicationl SystemList LargeServerl LargeServer2 LargeServer3 MedServerl MedServer2 MedServer3 MedServer4 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 MedServer3 MedServer4 MedServer5 FailOverPolicy Load Load 50 group Application2 SystemList LargeServerl LargeServer2 LargeServer3 MedServerl MedServer2 MedServer3 MedServer4 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList MedServerl MedServer2 M
571. t specifies the Action defined for the agent The Administering the cluster from the command line 285 Administering simulated clusters from the command line virtual fire drill checks for a resource type can are defined in the SupportedActions attribute and can be identified with the vfd suffix To run a virtual fire drill for a service group Type the following command havfd grp lt grpname gt sys lt sysname gt The command runs the infrastructure check and verifies whether the system lt sysname gt has the required infrastructure to host resources in the service group lt grpname gt should a failover require the service group to come online on the system For the variable lt sysname gt specify the name of a system on which the resource is offline To fix detected errors Type the following command hares action lt resname gt lt vfdaction gt vfd fix sys lt sysname gt The variable lt vfdaction gt represents the check that reported errors for the system specified by the variable lt sysname gt The virtual fire drill checks for a resource type can are defined in the SupportedActions attribute and can be identified with the vfd suffix Administering simulated clusters from the command line VCS Simulator is a tool to assist you in building and simulating cluster configurations With VCS Simulator you can predict service group behavior during cluster or system faults view state transitions and designate and
572. t as a resource fault If the attribute is set to 0 VCS does not treat Monitor timeouts as a resource faults If the attribute is set to 1 VCS interprets the timeout as a resource fault and the agent calls the Clean entry point to shut the resource down By default the FaultOnMonitorTimeouts attribute is set to 4 This means that the Monitor entry point must time out four times in a row before the resource is marked faulted How VCS handles resource faults This section describes the process VCS uses to determine the course of action when a resource faults VCS behavior when an online resource faults In the following example a resource in an online state is reported as being offline without being commanded by the agent to go offline m VCS first verifies the Monitor routine completes successfully in the required time If it does VCS examines the exit code returned by the Monitor routine If the Monitor routine does not complete in the required time VCS looks at the FaultOnMonitorTimeouts FOMT attribute m If FOMT O the resource will not fault when the Monitor routine times out VCS considers the resource online and monitors the resource periodically depending on the monitor interval If FOMT 1 or more VCS compares the CurrentMonitorTimeoutCount CMTC with the FOMT value If the monitor timeout count is not used up CMTC is incremented and VCS monitors the resource in the next cycle m If FOMT CMTC this means that the ava
573. t broker per domain typically a data center acting as root broker for all products using Symantec Product Authentication Services The root broker is only used during initial creation of an authentication broker m Authentication brokers serve as intermediate registration and certification authorities An authentication broker authenticates users such as a login to the cluster or services such as daemons running on application nodes but cannot authenticate other brokers Authentication brokers have certificates that are signed by the root Each node in VCS serves as an authentication broker Security credentials for the authentication broker must be obtained from the root broker For secure communication VCS components acquire credentials from the authentication broker that is configured on the local system The acquired certificate is used during authentication and is presented to clients for the SSL handshake VCS and its components specify the account name and the domain in the following format m HAD Account name _HA VCS_ systemname domain HA_SERVICES fully_qualified_system_name m CmdServer Introducing Veritas Cluster Server 29 Logical components of VCS name _CMDSERVER_VCS_ systemname domain HA_SERVICES fully_qualified_system_name For instructions on how to set up Security Services while setting up the cluster see the Veritas Cluster Server Installation Guide You can also enable and disable Security
574. t for a global group 1 2 3 From Cluster Explorer click Configure Global Groups on the Edit menu Click Next Enter the details of the service group to modify m Click the name of the service group m For global to local cluster conversion click the left arrow to move the cluster name from the cluster list back to the Available Clusters box 510 Administering global clusters from Cluster Manager Java console Deleting a remote cluster 5 6 m Click Next Enter or review the connection details for each cluster Click the Configure icon to review the remote cluster information for each cluster If the cluster is not running in secure mode m Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system Verify the port number Enter the user name Enter the password Click OK If the cluster is running in secure mode m Enter the IP address of the remote cluster the IP address of a cluster system or the host name of a cluster system m Verify the port number m Choose to connect to the remote cluster using the connected cluster s credentials or enter new credentials including the user name password and the domain m Click OK Click Next Click Finish To delete a remote cluster from the local cluster 1 From Cluster Explorer click Add Delete Remote Cluster on the Edit menu or From the Cluster Explorer configuration tree right click the c
575. t of standard ha commands Command hasim start system_name hasim setupclus clustername simport port_no wacport port_no sys systemname hasim deleteclus lt clus gt hasim start clustername_sys1 clus clustername disablel10n hasim stop hasim poweroff system_name hasim up system_name hasim fault system_name resource_name hasim online system_name resource_name hasim faultcluster clustername hasim clearcluster clustername Description Starts VCS Simulator The variable system_name represents the system that will transition from the LOCAL_BUILD state to RUNNING Creates a simulated cluster and associates the specified ports with the cluster Deletes specified cluster Deleting the cluster removes all files and directories associated with the cluster Before deleting a cluster make sure the cluster is not configured as a global cluster Starts VCS Simulator on the cluster specified by clustername If you start VCS Simulator with the disable110n option the simulated cluster does not accept localized values for attributes Use this option when simulating a UNIX configuration on a Windows system to prevent potential corruption when importing the simulated configuration to a UNIX cluster Stops the simulation process Gracefully shuts down the system Brings the system up Faults the specified resource on the specified system Brings specified resource onli
576. t you want to open the configuration in read write mode Saving the configuration After updating the VCS cluster configuration save it to disk in either read only or read write mode To save the configuration 1 In the Cluster Summary view in the Configuration task panel click Save Configuration In the Save Configuration dialog box check the Make Configuration Read Only option to save the configuration in read only mode or leave it clear to save the configuration in read write mode Click OK Administering the cluster from the Cluster Management Console 97 Administering a cluster Modifying cluster attributes Edit the attributes of a cluster You must have the role of cluster administrator to edit cluster attributes To edit a cluster attribute using the attribute table 1 Inthe Cluster Summary view on the secondary tab bar click Attributes 2 Inthe All attributes for cluster table locate the line item for the attribute you want to edit 3 Inthe Edit column click the edit button 4 Inthe Edit Attribute dialog box specify the following attribute details and then click OK m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value You can enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values m The systems
577. tas VCS Simulator 1 To simulate a cluster running a particular operating system copy the types cf file for the operating system from the types directory to VCS_SIMULATOR_HOME default_clus conf config For example if the cluster to be simulated runs on the AIX platform copy the file types cf aix Add custom type definitions to the file if required and rename the file to types cf If you have a main cf file to run in the simulated cluster copy it to VCS_SIMULATOR_HOME default_clus conf config Start VCS Simulator VCS_SIMULATOR_HOME bin gt hasim start system name The variable system_name represents a system name as defined in the configuration file main cf This command starts Simulator on port 14153 Add systems to the configuration if desired VCS_SIMULATOR_HOME bin gt hasim sys add system name VCS_SIMULATOR_HOME 3 bin gt hasim up system name Verify the state of each node in the cluster VCS_SIMULATOR_HOME bin gt hasim sys state See To simulate global clusters from the command line on page 327 To simulate global clusters from the command line 1 Install VCS Simulator in a directory sim_dir on your system See Installing VCS Simulator on page 318 Set up the clusters on your system Run the following command to add a cluster sim dir hasim setupclus clustername simport port_no wacport port no Do not use default_clus as the cluster name when simulating a global cluster
578. tate to read only mode To save and close a configuration from Cluster Explorer On the File menu click Close Configuration or Click Save and Close Configuration on the Cluster Explorer toolbar To save and close a configuration from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Configuration File gt Close Configuration 2 Click Apply Executing commands Use Command Center to execute commands on a cluster Command Center enables you to run commands organized as Configuration and Operation To execute a command from Command Center 1 From Command Center click the command from the command tree If necessary expand the tree to view the command 2 Inthe corresponding command interface click the VCS objects and appropriate options if necessary 3 Click Apply Editing attributes Use the Java Console to edit attributes of VCS objects By default the Java Console displays key attributes and type specific attributes To view all attributes associated with an object click Show all attributes To edit an attribute from Cluster Explorer 1 From the Cluster Explorer configuration tree click the object whose attributes you want to edit 2 Inthe view panel click the Properties tab If the attribute does not appear in the Properties View click Show all attributes This opens the Attributes View 222 Administering the cluster from Cluster Manager Java co
579. te To add a static resource attribute haattr add static resource type static attribute value dimension default To delete a static resource attribute haattr delete static resource type static attribute To add a temporary resource attribute haattr add temp resource type attribute value dimension default To delete a temporary resource attribute haattr delete temp resource type attribute To modify the default value of a resource attribute haattr default resource type attribute new value The variable new_value refers to the attribute s new default value Administering the cluster from the command line 271 Administering resources Defining attributes as local Localizing an attribute means that the attribute has a per system value for each system listed in the group s SystemList These attributes are localized on a per resource basis For example to localize the attribute attribute_name for resource only type hares local resource attribute name Note that global attributes cannot be modified with the hares local command The following table lists the commands to be used to localize attributes depending on their dimension Table 7 2 Making VCS attributes local scalar Replace a value modify object attribute name value sys system vector m Replace list of values modify object attribute name value sys system m Add list of values to existing list modify o
580. te system clus cluster localclus The option clus displays the attribute value on the cluster designated by the variable cluster the option localclus specifies the local cluster If the attribute has local scope you must specify the system name except when querying the attribute on the system from which you run the command To display the state of a service group across clusters hagrp state service groups sys systems clus cluster localclus The option clus displays the state of all service groups on a cluster designated by the variable cluster the option localclus specifies the local cluster To display service group information across clusters hagrp display service groups attribute attributes sys systems clus cluster localclus The option clus applies to global groups only If the group is local the cluster name must be the local cluster name otherwise no information is displayed To display service groups in a cluster hagrp list conditionals clus cluster localclus The option c1us lists all service groups on the cluster designated by the variable cluster the option localclus specifies the local cluster To display usage for the service group command hagrp help modify link list Administering global clusters from the command line 523 Global querying Querying resources To display resource attribute values across clusters hares value resource attribute
581. ted Bringing a service group online Manually put a service group into a responsive functioning state You must have the role of cluster operator or service group operator to bring a service group online You can bring a service group online on a specific system or you can bring the service group online anywhere If you select the Anywhere option the service group is brought online on the first available system in the cluster To bring a service group online with the opportunity to specify one system 1 4 In the Cluster Groups view in the Groups Listing table click the linked name of the offline service group that you want to bring online In the Group Summary view in the Operations task panel click Online This step is necessary only if the system list for the service group has more than one member system In the Online Service Group dialog box select the system on which to bring the service group online or click Anywhere The system choices in this step are populated using the system list for the service group The Anywhere option causes this task to try each system in the list until the service group is successfully brought online Click OK to confirm that you want to bring the service group online To bring one or more service groups online anywhere 1 In the Cluster Groups view in the Groups Listing table check the check box preceding the line item for the offline service group that you want to bring on
582. tem Resources E myMount mountpoint vfd Mountpoint oradata exists mounted vfd Mountpoint oradata is not mounted vxfslic vfd Not a VxF5 filesystem Storage Resources No Virtual Fire Drill actions For resources of type Storage Application Resources amp No Virtual Fire Drill actions For resources of type Application E 03 Replication Resources amp No Virtual Fire Drill actions For resources of type Replication Close Completed m Select the type of check to run m Select a service group for which to run the infrastructure checks Make sure you select a service group that is online m Select a system to run the checks on m Click Perform checks m View the result of the check If the virtual fire drill reports any errors right click the resource and select Fix it Click Close Administering the cluster from Cluster Manager Java console 169 Administering Cluster Monitor Administering Cluster Monitor Use the Java Console to administer a cluster or simulated cluster by adding or reconfiguring a cluster panel in Cluster Monitor To activate the connection of the procedures log on to the cluster after completing the final step Configuring a new cluster panel You must add a cluster panel for each cluster that you wish to connect to using the Java GUI To configure a new cluster panel 1 From Cluster Monitor click New Cluster on the File menu For simulated clusters clic
583. tem in the cluster A parallel service group is more complex than a failover group Parallel service groups are appropriate for applications that manage multiple application instances running simultaneously without data corruption Hybrid service groups A hybrid service group is for replicated data clusters and is a combination of the failover and parallel service groups It behaves as a failover group within a system zone and a parallel group across system zones See About system zones on page 366 A hybrid service group cannot fail over across system zones VCS allows a switch operation on a hybrid group only if both systems are within the same system zone If there are no systems within a zone to which a hybrid group can fail over VCS invokes the nofailover trigger on the lowest numbered node Hybrid service groups adhere to the same rules governing group dependencies as do parallel groups See About service group dependencies on page 410 See nofailover event trigger on page 450 Introducing Veritas Cluster Server 25 Logical components of VCS About agents in VCS Agents are multi threaded processes that provide the logic to manage resources VCS has one agent per resource type The agent monitors all resources of that type for example a single IP agent manages all IP resources When the agent is started it obtains the necessary configuration information from VCS It then periodically monitors the resources an
584. tem in the service group s SystemList On one system mount the file system containing the application s data on shared storage Use the same mount point as in the global zone Boot the zone To create zone root on local disks with data using direct mount file system 1 2 N OO wo A Create the zone with the zonecfg command Set the zonepath parameter to specify a location for the zone root Set a virtual IP address of the system to be the IP address of the zone and define the device name for the NIC associated with the IP address For example Exit the zonecfg configuration Create the actual zone root directory Set permissions for the zone root directory Install the non global zone using zoneadm 10 Configuring VCS in non global zones 597 Configuring VCS in zones Repeat the previous zone creation steps on each other system in the service group s SystemList Boot the zone On one system mount the file system containing the application s data on shared storage Use the same mount point as in the global zone Creating zone root on the shared disks Use the following steps to create a disk group on shared storage mount the shared file system and create the zone The file system typically for data that is on a shared device is either the loop back type or direct mount Choose one of the procedures that follow To create zone root on shared disks with data using loop back file system 1 ol O ON
585. ter 1 In the Command Center configuration tree expand Commands gt Operations gt Controls gt Offline Resource Click a resource Click a system on which to take the resource offline If necessary select the ignoreparent check box to take a selected child resource offline regardless of the state of the parent resource This option is only available through Command Center Click Apply 205 206 Administering the cluster from Cluster Manager Java console Administering resources Taking a resource offline and propagating the command Use the Offline Propagate OffProp feature to propagate the offline state of a parent resource This command signals that resources dependent on the parent resource should also be taken offline Use the Offline Propagate OffProp ignoreparent feature to take a selected resource offline regardless of the state of the parent resource This command propagates the offline state of the selected resource to the child resources The ignoreparent option is only available in Command Center To take a resource and its child resources offline from Cluster Explorer 1 Inthe Resources tab of the configuration tree right click the resource 2 Click Offline Prop and click the appropriate system from the menu To take a resource and its child resources offline from Command Center 1 Inthe Command Center configuration tree expand Commands gt Operations gt Controls gt OffProp Resour
586. ter Management Console 494 administering from command line 528 administering from Java Console 512 querying from command line 522 GlobalCounter attribute 651 Group Administrator about 63 adding user as 251 Group attribute 616 group dependencies See service group dependencies Group Operator about 63 adding user as 252 GroupLimit attribute 651 GroupOwner attribute 632 Guests attribute for clusters 651 for service groups 632 GUI See Java Console or Cluster Management Console GUIIPAddr attribute 643 692 Index H haagent display command 256 haagent list command 260 haattr add command 269 haattr default command 270 haattr delete command 270 hacf utility about 242 creating multiple cf files 242 loading a configuration 242 pretty printing 242 hacf verify command 242 HacliUserLevel attribute about 62 definition 651 haclus add command 531 haclus declare command 531 haclus delete command 531 haclus display command for global clusters 525 for local clusters 257 haclus list command 525 haclus modify command 531 haclus state command 525 haclus status command 525 haclus value command for global clusters 525 for local clusters 257 haclus wait command 284 haconf dump makero command 243 haconf makerw command 243 HAD about 27 binding to CPU 558 impact on performance 547 HAD diagnostics 566 had v command 276 had version command 276 hagrp add command 261 hagrp clear command 265 hagrp de
587. ter Manager T Cluster Monitor Lolk File view Help bamsmaan o Veritas Cluster Server Cluster Manager Group Status No groups configured i System Status Online 2 Link Status Up Cluster Name veslinux145_146_14b Connected to veslinux146 Group Status Online 1 Offline 1 pp System status Online 2 Link Status Up Connect to veslinux145 A Click here to log in Administering the cluster from Cluster Manager Java console 143 About Cluster Monitor Cluster monitor toolbar The Cluster Monitor toolbar contains the following buttons Available operations are described below T Cluster Monitor oj x File view Help Sai ms mmaoi o From left to right New Cluster Adds a new cluster panel to Cluster Monitor l Delete Cluster Removes a cluster panel from Cluster Monitor Expand Expands the Cluster Monitor view a Collapse Collapses the Cluster Monitor view aA in Stop Pauses cluster panel scrolling Start Resumes scrolling Login Log on to the cluster shown in the cluster panel gt Show Explorer Launches an additional window of Cluster Explorer after logging on to that cluster Move Cluster Panel Up Moves the selected cluster panel up Move Cluster Panel Down Moves the selected cluster panel down v Help Access online help 144 Administering the cluster from Clu
588. ter configuration 1 Create the simulated clusters for the global configuration See Creating a simulated cluster on page 322 Select the Enable Global Cluster Option check box and enter a unique port number for the wide area connector WAC process In the Simulator console click Make Global In the Make Global Configuration dialog box T Make Clusters Global x amp Make clusters global Name lobalCluster zl Available clusters Configured clusters iclus a Select an existing global cluster or enter the name for a new global cluster b From the Available Clusters list select the clusters to add to the global cluster and click the right arrow The clusters move to the Configured Clusters list c Click OK Bringing a system up Bring a system up to simulate a running system To bring a system up 1 2 From Cluster Explorer click the Systems tab of the configuration tree Right click the system in an unknown state and click Up Predicting VCS behavior using VCS Simulator 325 Administering VCS Simulator from the Java Console Powering off a system 1 From Cluster Explorer click the Systems tab of the configuration tree 2 Right click the online system and click Power Off Saving the offline configuration 1 From Cluster Explorer click Save Configuration As from the File menu 2 Enter the path location 3 Click OK Simulating a resource fault 1 From Cluster Explorer
589. ter faults m Connected enables the service group to automatically fail over to another cluster if it is unable to fail over within the cluster In this step you update the cluster list of remaining instances of the selected global service group To perform the update you must first verify or supply the authentication credentials for each remaining global cluster in Administering global clusters from the Cluster Management Console 493 Administering global clusters the list The Cluster Management Console can then connect to those clusters and update the lists In the Remote Cluster Configuration dialog box verify the required information for the remaining remote clusters and then click Next To change authentication information click a cluster name under Existing Clusters and then enter the authentication information in the fields to the right The requisite information in this dialog box varies depending upon whether or not the cluster is secure uses an authentication broker Click Next Click Finish You must repeat this procedure for all global service groups that are configured on the global cluster to be removed To remove a remote cluster from the local cluster 1 In the Cluster Summary view in the Configuration task panel click Add Delete Remote Cluster In the Remote Cluster Configuration wizard select the cluster to delete and then click Next The cluster that you want to delete cannot be in a running state
590. tes that this view is the fifth view at the cluster level It contains cluster level information about resources and enables you to view the status of resources that are members of the cluster See the table Levels and views in the Manage tab view hierarchy on page 84 86 Administering the cluster from the Cluster Management Console Overview of the Cluster Management Console m The view name Group Summary indicates that this view is the first view at the service group level It contains summary information about a service group and enables you to view a composite status of the service group About quick navigation links In many views the status panel contains underlined object names For example the name of the cluster in the Cluster summary view links to a detail view for the objects that are members of that cluster Likewise clicking other underlined objects in the status panel takes you to a detail view for those objects Figure 5 12 Quick navigation links in the status panel links are underlined sol50 510a sect ClusterService Group Summary Summary Attributes Resources Logs Name ClusterService State Online Frozen No State information on member Systems System Name 4 State go thor311 Online Be thor312 Offline Resources Listing Resource Name 4 Type wac Application Icon conventions Administering the cluster from the Cluster Management Console Overview
591. th critical parent fails to come online In the following example when a command is issued to bring the service group online resource R2 fails to come online EE A co eereeeee VCS calls the Clean function for resource R2 and propagates the fault up the dependency tree Resource R1 is set to critical so the service group is taken offline and failed over to another node in the cluster Controlling VCS behavior 365 Controlling VCS behavior at the service group level Controlling VCS behavior at the service group level This section describes how you can configure service group attributes to modify VCS behavior in response to resource faults About the AutoRestart attribute If a persistent resource on a service group GROUP_1 faults VCS fails the service group over to another system if the following conditions are met m The AutoFailOver attribute is set m Another system in the cluster exists to which GROUP_1 can fail over If neither of these conditions is met GROUP_1 remains offline and faulted even after the faulted resource becomes online Setting the AutoRestart attribute enables a service group to be brought back online without manual intervention Or if GROUP_1 could no failover targets are available setting the AutoRestart attribute enables VCS to bring the group back online on the first available system after the group s faulted resource came online on that system For example NIC is a p
592. th licensing Validating license keys The installvecs script handles most license key validations However if you install a VCS key outside of installvcs using vxlicinst for example you can validate the key using the procedure described below 1 The vxlicinst command handles some of the basic validations node lock Ensures that you are installing a node locked key on the correct system demo hard end date Ensures that you are not installing an expired demo key Run the vxlicrep command to make sure a VCS key is installed on the system The output of the command resembles Veritas License Manager vxlicrep utility version 3 02 003 Copyright C Veritas Software Corp 2002 All Rights reserved Creating a report on all Veritas products installed on this system License Key XXXX XXXX XXXX XXXX XXXX XXXK Product Name Veritas Cluster Server License Type PERMANENT OEM ID 4095 Features Platform Solaris Version 5 0 Tier Unused Reserved 0 Mode vcs Global Cluster Option Enabled Troubleshooting and recovery for VCS 579 Troubleshooting licensing 3 Look for the following in the command output Make sure the Product Name lists the name of your purchased component for example Veritas Cluster Server If the command output does not return the product name you do not have a VCS key installed If the output shows the License Type for a VCS key as DEMO ensure that the Demo End Date do
593. the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui cert certreq certreq file The variable certreq_file specifies the file to which the CSR is written The file is written using the Public Key Cryptography Standard PKCS 10 For example opt VRTSweb bin webgui cert certreq myapp vrtsweb csr Submit the CSR to a certification authority who issues a CA signed certificate Import the CA issued certificate to VRTSweb Run the following command on the system where VRTSweb is installed VRTSWEB_HOME bin webgui import ca cert file The variable cert_file represents the certificate that is issued by the certification authority For example opt VRTSweb bin webgui cert import myapp vrtsweb cer 666 Administering Symantec Web Server Managing VRTSweb SSL certificates Note that the import command fails if the CA root certificate is not a part of the trust store that is associated with VRTSweb If the command fails add the CA root certificate to the VRTSweb trust store VRTSWEB_HOME bin webgui cert trust ca root cert file For example opt VRTSweb bin webgui cert trust myapp caroot cer Once the certificate used to sign the CSR is added to VRTSweb trust store you can import the CA assigned certificate into VRTSweb 5 Restart VRTSweb S VRTSWEB HOME bin webgui restart Cloning the VRTSweb SSL certificate You can clone the VRTSweb SSL keypair into a keystore and use the cloned V
594. the specified server and verifies that it is serves Web pages In a database environment the monitoring application can connect to the database server and perform SQL commands to verify read and write access to the database The closer a test comes to matching what a user does the better the test is in discovering problems You should balance the level of monitoring between ensuring the application is up and minimizing monitor overhead Introducing Veritas Cluster Server 19 Can my application be clustered Ability to restart the application in a known state When the application is taken offline it must close out all tasks store data properly on shared disk and exit Stateful servers must not keep that state of clients in memory States should be written to shared storage to ensure proper failover Commercial databases such as Oracle Sybase or SQL Server are good examples of well written crash tolerant applications On any client SQL request the client is responsible for holding the request until it receives acknowledgement from the server When the server receives a request it is placed in a special redo log file The data is confirmed as being saved before acknowledging the client After a server crashes the database recovers to the last known committed state by mounting the data tables and applying the redo logs This returns the database to the time of the crash The client resubmits any outstanding client requests that are un
595. the system joined the cluster m Type and dimension integer scalar m Default Not applicable ConfigDiskState system use only State of configuration on the disk when the system joined the cluster m Type and dimension integer scalar m Default Not applicable ConfigFile user defined Directory containing the configuration files m Type and dimension string scalar m Default 642 VCS attributes System attributes Table D 4 System attributes ConfigInfoCnt The count of outstanding CONFIG_INFO messages the local node expects from a new membership message This attribute is non zero for the brief period during which new membership is processed When the value returns to 0 the state of all nodes in the cluster is determined system use only m Type and dimension integer scalar m Default Not applicable ConfigModDate Last modification date of configuration when the system joined system use only the cluster m Type and dimension integer scalar m Default Not applicable CPUUsage Indicates the system s CPU usage by CPU percentage utilization This attribute s value is valid if the Enabled value in the CPUUsageMonitoring attribute below equals 1 The value of this attribute is updated when there is a change of five percent since the last indicated value system use only m Type and dimension integer scalar m Default Not applicable CPUUsageMonitoring Monitors th
596. the Cluster Summary view in the Groups Listing table check the check box preceding the line item for each service group with the attribute that you want to edit Select all service groups by checking the check box at the top In the Configuration task panel click Edit Attribute Specify the attribute details See To edit a service group attribute using the attribute table in Modifying service group attributes on page 111 Flushing a service group Postpone the resolution of resource related problems that occur while a service group is being brought online or taken offline You must have the role of cluster administrator or service group administrator or service group operator to flush a service group Flushing a service group halts the current operation on the resources yet to be brought online or taken offline Flushing a service group typically leaves the cluster in a partial state After flushing a service group you can resolve the issue with the problematic resource To flush a service group 1 2 In the Group Summary view in the Operations task panel click Flush In the Flush Service Group dialog box select the system on which you want to flush the service group from the drop down menu The menu is populated with system from the service group system list which contains the systems on which the service group is configured Click OK Administering the cluster from the Cluster Management Console 113 Admini
597. the following attribute details and then click OK m The attribute value In the Values field either click a value to change the it or click the plus button to add a value Click a value and then click the minus button to remove a value You can enter more than one value for some attributes Use the up and down arrow buttons to scroll among multiple values m The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List Invoking a resource action Run a predefined action script on a resource Examples of predefined resource actions include splitting and joining disk groups To invoke a resource action 1 In the Cluster Resources view in the Resources Listing table click the linked name of the resource on which you want to invoke an action In the Resource Summary view in the Operations task panel click Invoke Action In the Invoke Action dialog box select the script that you want to run and the system on which you want to run it Specify arguments for the action Click the plus button to add arguments Click an argument name and then click the minus button to remove it Click OK Refreshing the Resourcelnfo attribute Refresh the ResourcelInfo attribute to view t
598. the heartbeat m Type and dimension integer scalar m Default 300 seconds Appendix Administering Symantec Web Server m About Symantec Web Server m Getting Started m Configuring ports for VRTSweb m Managing VRTSweb SSL certificates m Configuring SMTP notification for VRTSweb m Configuring logging for VRTSweb m Modifying the maximum heap size for VRTSweb 658 Administering Symantec Web Server About Symantec Web Server About Symantec Web Server Symantec Web Server VRTSweb is a Web Server component shared by various Symantec Web consoles including Veritas Cluster Server and Veritas Volume Replicator This document describes how to administer VRTSweb and provides instructions for common configuration tasks Note that changes to the VRTSweb configuration apply to all Web consoles sharing the Web server The Web server is installed at the following path m UNIX opt VRTSweb m Windows C Program Files Veritas VRTSweb Getting Started Connect to the Web server to start administering it To connect to Symantec Web Server 1 Access the Web server using the configured port number for example http hostname 8181 2 Accept the certificate issued by Symantec Symantec Web Server Home Configuration Connected to host vcslabweb Select Language Go VERITAS Cluster Server Web Console m To view and select the available Web consoles click the Home m To view an
599. the parent icon dragging the yellow line to the icon that will serve as the child and then clicking the child icon Use the Esc key to delete the yellow dependency line connecting the parent and child during the process of linking the two icons If the Link mode is not activated click and drag an icon along a horizontal plane to move the icon Click Auto Arrange to reset the appearance of the graph The view resets the arrangement of icons after the addition or deletion of a resource service group or dependency link Changes in the Resource and Service Group views will be maintained after the user logs off and logs on to the Java Console at a later time Administering the cluster from Cluster Manager Java console 157 About Cluster Explorer Zooming in on Service Group and Resource views The Resource view and Service Group view include a navigator tool to zoom in or out of their graphs Click the magnifying glass icon in the top right corner to open the zoom panel Ee Service Group Yiew C3 Ed Link R auto arrange P Ohl eka Slt 2 To move the view to the left or right click a distance in pixels from the drop down list box between the hand icons Click the lt or gt hand icon to move the view in the desired direction To shrink or enlarge the view click a size factor from the drop down list box between the magnifying glass icons Click the or magnifying glass icon to modify the s
600. the same on both zones the Mount resources will mount the same block devices and the same Oracle instance will start on the secondary in case of a failover Configuring the service groups To configure the replication group 1 Create a hybrid service group oragrp_rep for replication You can use the VvrRvgGroup template to create the service group See Types of service groups on page 24 2 Copy the DiskGroup resource from the application to the new group Configure the resource to point to the disk group that contains the RVG 3 Configure new resources of type IP and NIC 4 Configure a new resource of type RVG in the service group The RVG agent ships with the VVR software If the RVG resource type is not defined in your configuration import it as instructed below a On the File menu click Import Types b Inthe Import Types dialog box Click the file from which to import the resource type By default the RVG resource type is located at the path etc VRTSvcs conf VVRTypes cf c Click Import 5 Configure the RVG resource See the VVR documentation for more information Note that the RVG resource starts stops and monitors the RVG in its current state and does not promote or demote VVR when you want to change the direction of replication The RVGPrimary agent manages that task 540 Setting up replicated data clusters Setting up a replicated data cluster configuration Set resource dependencies as per the followin
601. the service group online on each remaining system in the order listed m If the first system and the startup system are different VCS attempts to bring the service group online on the first system in the list If the first system fails VCS attempts to bring the service group online on each remaining system in the order listed 6 Inthe Resource Creation dialog box specify the following details about the resources that belong to the service group and then click Next m The member resource list Enter the name of any new resources that you want to add to the service group into the Resource Name text box Resource names must Consist of only one string Not contain special characters like period or amp ampersand m Begin with an alphabetic character The remaining characters in the name can be letters digits hyphens or underscores m The resource startup state If you want the resource to start in an enabled state after the service group comes online check Enable Resource Click Add Resource to add the resource to the Resource List m The resource attribute values or resource removal In the Resource List table you can edit resource attributes or delete a resource using the buttons in the Edit and Delete columns See Adding a resource to a service group on page 116 7 Ifyou need to link one or more resources provide the following relationship details in the Link Unlink Resource dialog box and then click Finish Otherwise
602. ther cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster About user listing tables The following is a list of the columns and data in tables that list user information m Name the user name string m Privileges the roles assigned to the user m Assign Privileges a button that takes you to the role assignment part of the Add User wizard See Adding and deleting a user on page 94 m Delete a button that prompts you to delete the user Sorting the list You can sort a user listing table according to your needs To sort users 1 Inthe user listing table locate the column by which you want to sort the list 2 Click the column heading The list is sorted by the values in the selected column in ascending order A triangle pointing up displays in that column heading Click the column heading again to reverse the sort this action also reverses the triangle 94 Administering the cluster from the Cluster Management Console Administering users Adding and deleting a user Users must have a role assigned on the cluster before they can perform any management tasks To add a cluster user 1 2 In the Cluster Users view in the Configuration task panel click Add a user In the New User wizard specify the following new user details and then click Next The new user name If the target cluster is not configured in secure mode you
603. tially boots LLT determines which systems are sending heartbeat signals and passes that information to GAB GAB uses this information in the process of seeding the cluster membership Seeding a new cluster Seeding insures a new cluster will start with an accurate membership count of the number of systems in the cluster This prevents the possibility of one cluster splitting into multiple subclusters upon initial startup A new cluster can be automatically seeded as follows m When the cluster initially boots all systems in the cluster are unseeded m GAB checks the number of systems that have been declared to be members of the cluster in the etc gabtab file The number of systems declared in the cluster is denoted as follows sbin gabconfig c n where the variable is replaced with the number of systems in the cluster Note This number should represent 100 of the systems in the cluster About communications membership and data protection in the cluster 339 About cluster membership m When GAB on each system detects that the correct number of systems are running based on the number declared in etc gabtab and input from LLT it will seed m HAD will start on each seeded system HAD will only run on a system that has seeded Manual seeding of a cluster Seeding the cluster manually is appropriate when the number of cluster systems declared in etc gabtab is more than the number of systems that will join the cluster
604. tials every time you run a VCS command See Logging on to VCS on page 240 User privileges in global clusters VCS permits a cross cluster online or offline operation only if the user initiating the operation has one of the following privileges Group Administrator or Group Operator privileges for the group on the remote cluster Cluster Administrator or Cluster Operator privileges on the remote cluster VCS permits a cross cluster switch operation only if the user initiating the operation has the following privileges Group Administrator or Group Operator privileges for the group on both clusters About the VCS user privilege model 65 About VCS user privileges and roles m Cluster Administrator or Cluster Operator privileges on both clusters User privileges in secure clusters In secure mode VCS assigns Guest privileges to all native users When assigning privileges in secure clusters you must specify fully qualified user names in the format username domain You cannot assign or change passwords for users using VCS when VCS is running in secure mode 66 About the VCS user privilege model How administrators assign roles to users How administrators assign roles to users To assign a role to a user an administrator performs the following tasks m Add a user to the cluster if the cluster is not running in secure mode m Assign a role to the user m Assign the user a set of objects appropriate for the role In secur
605. tials on the View menu T User Credentials El Credentials of the connected user User Name root Domain vcslinux145 vxindia veritas com Domain Type unixpwd Issued By CN broker OU root veslinux145 vxindia veritas com O vx To view server credentials From Cluster Explorer click Server Credentials on the View menu T Server Credentials xi Credentials of the connected server User Name _HA_ CS_veslinux145 Domain HA_SERYICES vcslinux145 vxindia veritas com Domain Type VX Issued By CN broker OU root veslinux145 vxindia veritas com O vx IV Do not show during startup 168 Administering the cluster from Cluster Manager Java console Accessing additional features of the Java Console Running a virtual fire drill from the Java Console Use the Cluster Manager to run virtual fire drills in a local cluster To run a virtual fire drill 1 On the Cluster Explorer toolbar click Virtual Fire Drill or From Cluster Explorer click Virtual Fire Drill on the Tools menu Specify details to run a virtual fire drill T Virtual Fire Drill Infrastructure checks Select type of check All Infrastructure Checks Select Group myGrp X Select System veslinux146 X Perform checks Results EGA Network Resources E mylP device vfd Device eth0 exists fd Route e 10 212 peat device vid NIC eth0 exists Route exists to 10 212 102 248 via eth0 E 003 File Sys
606. time it takes a parent group to be brought online also depends on the time it takes the child group to be brought online 556 VCS performance considerations Scheduling class and priority configuration Scheduling class and priority configuration VCS allows you to specify priorities and scheduling classes for VCS processes VCS supports the following scheduling classes m RealTime specified as RT in the configuration file m TimeSharing specified as TS in the configuration file On Solaris 9 the following classes are supported m FairShare specified as FSS in the configuration file m FairPriority specified as FX in the configuration file Priority ranges The following table displays the platform specific priority range for RealTime TimeSharing and SRM scheduling SHR processes Table 20 1 Priority ranges AIX RT 126 50 126 50 TS 60 Priority varies with CPU consumption Note On AIX use ps ael HP UX RT 127 0 127 0 TS N A N A Note On HP UX use ps ael Linux RT 1 99 L high priority task TS N high priority task Note On Linux use ps ael Solaris RT 0 59 100 159 TS 60 60 N A SHR 60 60 N A Note On Solaris use ps ae o pri args VCS performance considerations 557 Scheduling class and priority configuration Default scheduling classes and priorities The following table lists the default class and priority values used by VCS The class and priority of
607. tion is invalid the system transitions to the state of STALE_ADMIN_WAIT The following table provides a list of VCS system states and their descriptions Cluster and system states 609 System states See Examples of system state transitions on page 611 Table C 2 VCS system states ADMIN_WAIT The running configuration was lost A system transitions into this state for the following reasons m The last system in the RUNNING configuration leaves the cluster before another system takes a snapshot of its configuration and transitions to the RUNNING state m A system in LOCAL_BUILD state tries to build the configuration from disk and receives an unexpected error from hacf indicating the configuration is invalid CURRENT_DISCOVER_WAIT The system has joined the cluster and its configuration file is valid The system is waiting for information from other systems before it determines how to transition to another state CURRENT_PEER_WAIT The system has a valid configuration file and another system is doing a build from disk LOCAL_BUILD When its peer finishes the build this system transitions to the state REMOTE_BUILD EXITING The system is leaving the cluster EXITED The system has left the cluster EXITING_FORCIBLY An hastop force command has forced the system to leave the cluster FAULTED The system has left the cluster unexpectedly INITING The system has joined the cluste
608. tion tree click the Resources tab and right click the resource icon in the view panel Click Delete from the menu Click Yes To delete a resource from Command Center 1 In the Command Center configuration tree expand Commands gt Configuration gt Cluster Objects gt Delete Resource Click the resource Click Apply Bringing a resource online To bring a resource online from Cluster Explorer 1 In the Service Groups tab of the configuration tree right click the resource or Click a service group in the configuration tree click the Resources tab and right click the resource icon in the view panel Click Online and click the appropriate system from the menu To bring a resource online from Command Center 1 2 In the Command Center configuration tree expand Commands gt Operations gt Controls gt Online Resource Click a resource 3 4 Administering the cluster from Cluster Manager Java console Administering resources Click a system on which to bring the resource online Click Apply Taking a resource offline To take a resource offline from Cluster Explorer 1 2 In the Service Groups tab of the configuration tree right click the resource or Click a service group in the configuration tree click the Resources tab and right click the resource icon in the view panel Click Offline and click the appropriate system from the menu To take a resource offline from Command Cen
609. tion tree click the Service Groups tab 2 Click the service group to which the resources belong 3 Inthe view panel click the Resources tab This opens the resource dependency graph To link a parent resource with a child resource Click Link Click the parent resource Move the mouse towards the child resource The yellow line snaps to the child resource If necessary press Esc to delete the line between the parent and the pointer before it snaps to the child Click the child resource In the Confirmation dialog box click Yes or Right click the parent resource and click Link from the menu In the Link Resources dialog box click the resource that will serve as the child Click OK f Link Resources Eg Parent Resource resourcel_ng2 Group newgroup2 Child Resource Cancel Click OK To link resources from Command Center 1 Inthe Command Center configuration tree expand Commands gt Configuration gt Dependencies gt Link Resources 2 Click the service group to contain the linked resources Administering the cluster from Cluster Manager Java console 213 Administering resources 3 Click the parent resource in the Service Group Resources box After selecting the parent resource the potential resources that can serve as child resources are displayed in the Child Resources box T VCSHP10_71_0214a Command Center admin Cluster Administrator oL File yew Help B 624 a
610. tistics VCS attributes enabling agent statistics This section describes the attributes that enable VCS agent statistics MonitorStatsParam A resource type level attribute which stores the required MonitorTimeStats ComputeStats parameter values for calculating monitor time statistics static str MonitorStatsParam Frequency 10 ExpectedValue 3000 ValueThreshold 100 AvgThreshold 40 m Frequency Defines the number of monitor cycles after which the average monitor cycle time should be computed and sent to the engine If configured the value for this attribute must be between 1 and 30 It is set to 0 by default m ExpectedValue The expected monitor time in milliseconds for all resources of this type Default 3000 m ValueThreshold The acceptable percentage difference between the expected monitor cycle time ExpectedValue and the actual monitor cycle time Default 100 m AvgThreshold The acceptable percentage difference between the benchmark average and the moving average of monitor cycle times Default 40 Stores the average time taken by a number of monitor cycles specified by the Frequency attribute along with a timestamp value of when the average was computed str MonitorTimeStats Avg 0 TS This attribute is updated periodically after a number of monitor cycles specified by the Frequency attribute If Frequency is set to 10 the attribute stores the average of 10 monitor cycle times
611. to edit Not all attributes can be edited In the Edit column click the edit button 126 Administering the cluster from the Cluster Management Console Administering resource types 4 Inthe Edit Attribute dialog box specify the following attribute details and then click OK The attribute value In the Values field specify an appropriate string value for the attribute The systems on which to change the attribute Specify a setting for the Apply value to option Apply to all nodes applies the changes to all systems listed the System List which lists all systems on which the attribute is configured Apply to selected nodes enables you to select nodes from the System List Administering the cluster from the Cluster Management Console 127 Administering systems Administering systems The Cluster Management Console enables you to configure and manage the systems that host the cluster service groups and resources Use the Cluster Systems view and the System Summary view to administer systems To navigate to the Cluster Systems view 1 2 On the navigation bar click Cluster On the secondary tab bar click Systems In the Cluster Systems view the Systems Listing table lists all systems in the selected cluster The task panels contain several tasks that you can perform in a one to many fashion on the listed systems To navigate to the System Summary view 1 2 3 On the navigation bar click Cluster On the s
612. tor determines if the resource is online then reports that information to VCS The time it takes to bring a resource online equals the time for the resource to go online plus the time for the subsequent monitor to execute and report to VCS Most resources are online when the online entry point finishes The agent schedules the monitor immediately after the entry point finishes so the first monitor detects the resource as online However for some resources such as a database server recovery can take longer In this case the time it takes to bring a resource online depends on the amount of data to recover It may take multiple monitor intervals before a database server is reported online When this occurs it is important to have the correct values configured for the OnlineTimeout and OnlineWaitLimit attributes of the database server resource type VCS performance considerations 551 How cluster operations affect performance When a resource goes offline Similar to the online entry point the offline entry point takes the resource offline and may return before the resource is actually offline Subsequent monitoring confirms whether the resource is offline The time it takes to offline a resource equals the time it takes for the resource to go offline plus the duration of subsequent monitoring and reporting to VCS that the resource is offline Most resources are typically offline when the offline entry point finishes The agent schedules the
613. trapd the events are not listed in the OpenView Network Node Manager Event configuration By default notifier assumes the community string is public If your SNMP console was configured with a different community reconfigure it according to the notifier configuration See the Veritas Cluster Server Bundled Agents Reference Guide for more information on NotifierMngr 574 Troubleshooting and recovery for VCS Troubleshooting VCS configuration backup and restore Troubleshooting VCS configuration backup and restore This section cites the problem you may encounter when using the hasnap command to backup and restore VCS configuration files Error connecting to remote nodes in the cluster The hasnap command is a distributed command in the sense that it tries to backup and restore files from all cluster nodes in a single session It needs to establish connection with all cluster nodes from the node where the command is executed The connection may fail for one of the following reasons m The hasnap command retrieves the list of cluster nodes from the lithosts configuration file However the node names in this file may not always be DNS resolvable in which case the command cannot establish connection with the remote nodes Recommended Action For each node in the cluster map the VCS node names to the actual DNS resolvable names using the hasnap configuration file opt VRTSvcs cutil conf vcsmappings properties m The hasnap command
614. tributes labelled system use only are set by VCS and are read only They contain important information about the state of the cluster The values labeled agent defined are set by the corresponding agent and are also read only See About VCS attributes on page 51 Resource attributes Table D 1 lists the resource attributes Table D 1 Resource attributes ArgListValues List of arguments passed to the resource s agent on each agent defined system This attribute is resource and system specific meaning that the list of values passed to the agent depend on which system and resource they are intended m Type and dimension string vector m Default non applicable AutoStart Indicates the resource is brought online when the service group is user defined brought online m Type and dimension boolean scalar m Default 1 Table D 1 VCS attributes 615 Resource attributes Resource attributes ComputeStats user defined Indicates to agent framework whether or not to calculate the resource s monitor statistics m Type and dimension boolean scalar m Default 0 ConfidenceLevel agent defined Indicates the level of confidence in an online resource Values range from 0 100 Note that some VCS agents may not take advantage of this attribute and may always set it to 0 Set the level to 100 if the attribute is not used m Type and dimension integer scalar m Default 0 Critic
615. ttribute can be overridden When a resource has remained online for the specified time in seconds previous faults and restart attempts are ignored by the agent See ToleranceLimit and RestartLimit attributes for details m Type and dimension integer scalar m Default 600 seconds FaultOnMonitorTimeou ts user defined Note This attribute can be overridden When a monitor times out as many times as the value specified the corresponding resource is brought down by calling the clean entry point The resource is then marked FAULTED or it is restarted depending on the value set in the RestartLimit attribute When FaultOnMonitorTimeouts is set to 0 monitor failures are not considered indicative of a resource fault A low value may lead to spurious resource faults especially on heavily loaded systems m Type and dimension integer scalar m Default 4 FireDrill user defined Specifies whether or not fire drill is enabled for resource type If set to 1 fire drill is enabled If set to 0 it is disabled m Type and dimension boolean scalar m Default 0 InfoInterval user defined Duration in seconds after which the info entry point is invoked by the agent framework for ONLINE resources of the particular resource type If set to 0 the agent framework does not periodically invoke the info entry point To manually invoke the info entry point use the command hares refreshinfo If the value you
616. tus tab 160 Administering the cluster from Cluster Manager Java console Accessing additional features of the Java Console Accessing additional features of the Java Console Use Cluster Manager to access the Template View System Manager User Manager Command Center Configuration Wizard Notifier Resource Configuration Wizard Remote Group Resource Configuration Wizard Query Module and Logs You can also use the Cluster Manager to run virtual fire drills to check for any configurational discrepancies that might prevent a service group from coming online on a specific node Template view The Template View displays the service group templates available in VCS Templates are predefined service groups that define the resources resource attributes and dependencies within the service group Use this view to add service groups to the cluster configuration and copy the resources within a service group template to existing service groups In this window the left pane displays the templates available on the system to which Cluster Manager is connected The right pane displays the selected template s resource dependency graph Template files conform to the VCS configuration language and contain the extension tf These files reside in the VCS configuration directory T Template View as on System vcshp71 of x Template View YolumeNFSGroup Templates VolumeNFSGroup ae Mi VNFS_IP ke egas i vNFS_NIC wNFS_Share1 vNFS_
617. ty of the coordinator disks so there must always be an odd number of disks with three disks recommended Coordinator disks cannot be used for any other purpose in the cluster configuration such as data storage or inclusion in a disk group for user data 342 About communications membership and data protection in the cluster About membership arbitration Any disks that support SCSI 3 Persistent Reservation can be coordinator disks Best practice is to select the smallest possible LUNs for use as coordinator disks You can configure coordinator disks to use Veritas Volume Manager Dynamic Multipathing DMP feature For more information on using DMP see the Veritas Volume Manager Administrator s Guide How the fencing module starts up The fencing module starts up as follows The coordinator disks are placed in a disk group This allows the fencing start up script to use Veritas Volume Manager VxVM commands to easily determine which disks are coordinator disks and what paths exist to those disks This disk group is never imported and is not used for any other purpose The fencing start up script on each system uses VxVM commands to populate the file etc vxfentab with the paths available to the coordinator disks For example if the user has configured 3 coordinator disks with 2 paths to each disk the etc vxfentab file will contain 6 individual lines representing the path name to each disk such as dev rdsk c1t1d1s2
618. u can bring children of disabled resources online if necessary m When a group containing disabled resources is taken offline the offline transaction is propagated to the disabled resources The following figures show how a service group containing disabled resources is brought online Rua Re eI N a esource_ 9 p Re Sei a 6 L p v aa IN a Resource 2 Res pe S a ka Sf a Resource_3 is disabled Resource_4 is offline Resource_5 is offline In the figure above Resource_3 is disabled When the service group is brought online the only resources brought online by VCS are Resource_1 and Resource_2 Resource_2 is brought online first because VCS recognizes Resource_3 is disabled In accordance with online logic the transaction is not propagated to the disabled resource 386 Controlling VCS behavior Controlling VCS behavior at the resource level In the figure below Resource_2 is disabled When the service group is brought online resources 1 3 4 are also brought online Resource_4 is brought online first Note Resource_3 the child of the disabled resource is brought online because Resource_1 is enabled and is dependent on it a Resource D C Resource_2 7 x o P Resource _2 is disabled SU g Resource_ 3 4 an fo Si oe S a gt a Dop C Resource A How disabled resources affect group states When a service group is brought online containing non pers
619. uations the systems are not actually down and may return to the cluster after cluster membership has been recalculated This could result in data corruption as a system could potentially write to disk before it determines it should no longer be in the cluster Combining membership arbitration with data protection of the shared storage eliminates all of the above possibilities for data corruption Data protection fences off removes access to the shared data storage from any system that is not a current and verified member of the cluster Access is blocked by the use of SCSI 3 persistent reservations SCSI 3 Persistent Reservation SCSI 3 Persistent Reservation SCSI 3 PR supports device access from multiple systems or from multiple paths from a single system At the same time it blocks access to the device from other systems or other paths VCS logic determines when to online a service group on a particular system If the service group contains a disk group the disk group is imported as part of the service group being brought online When using SCSI 3 PR importing the disk group puts registration and reservation on the data disks Only the system that has imported the storage with SCSI 3 reservation can write to the shared storage This prevents a system that did not participate in membership arbitration from corrupting the shared storage SCSI 3 PR ensures persistent reservations across SCSI bus resets Note Use of SCSI 3 PR prot
620. ue For example to set the AgentClass attribute of the FileOnOff resource to RealTime type hatype modify FileOnOff AgentClass RT To update the AgentPriority hatype modify resource type AgentPriority value For example to set the AgentPriority attribute of the FileOnOff resource to 10 type hatype modify FileOnOff AgentPriority 10 To update the ScriptClass hatype modify resource type ScriptClass value For example to set the ScriptClass of the FileOnOff resource to RealTime type hatype modify FileOnOff ScriptClass RT To update the ScriptPriority hatype modify resource type ScriptPriority value For example to set the ScriptClass of the FileOnOff resource to RealTime type hatype modify FileOnOff ScriptPriority 40 Note For attributes AgentClass and AgentPriority changes are effective immediately For ScriptClass and ScriptPriority changes become effective for scripts fired after the execution of the hat ype command 284 Administering the cluster from the command line Using the wait option in scripts Using the wait option in scripts The wait option is for use in scripts using VCS commands to change attribute values The option blocks the VCS command until the value of the specified attribute is changed or until the timeout if specified expires Specify the timeout in seconds The option can be used only with changes to scalar attributes The wait option is supported wit
621. ue 1 indicated the cluster runs in secure mode This attribute cannot be modified when VCS is running m Type and dimension boolean scalar m Default 0 SourceFile user defined File from which the configuration was read Always set to main cf Make sure the path exists on all nodes before configuring this attribute m Type and dimension string scalar m Default Not applicable Stewards user defined The IP address and hostname of systems running the steward process m Type and dimension string keylist m Default Not applicable TypeLimit user defined Maximum number of resource types m Type and dimension integer scalar m Default 100 UseFence user defined Indicates whether the cluster uses SCSI 3 I O fencing The value SCSI3 indicates that the cluster uses I O fencing the value NONE indicates it does not m Type and dimension string scalar m Default NONE UserNames user defined List of VCS users Note Default user name is admin m Type and dimension string association m Default 654 VCS attributes Cluster attributes Table D 5 Cluster attributes VCSi3Info Enables VCS service groups to be mapped to Veritas i3 applications This attribute is managed by the i3 product and system use only should not be set or modified by the user Contact your local Veritas sales representative for more information on the benefits of integrating VCS a
622. ught online If insufficient room exists on the target system you can check the Evacuate lower priority service groups option This option makes room for the service group being switched by taking lower priority service groups offline Click OK Freezing a service group Prevent a service group from failing over to another system Freezing a service group stops all online and offline operations on the service group To freeze a service group 1 In the Cluster Groups view in the Groups Listing table click the linked name of the service group that you want to freeze In the Group Summary view in the Operations task panel click Freeze In the Freeze Service Group dialog box click OK to confirm that you want to freeze the specified service group If you want the service group to remain frozen until explicitly unfrozen check the Freeze Persistently option Otherwise the service group unfreezes when either the management server or the Cluster Management Console is restarted To freeze one or more service groups 1 In the Cluster Groups view in the Groups Listing table check the check box preceding the line item for each unfrozen service group that you want to freeze Select all service groups by checking the check box at the top In the Operations task panel click Freeze Administering the cluster from the Cluster Management Console Administering service groups 3 Inthe Freeze Service Group s dialog box click OK to co
623. ulti node cluster any node designated to run a specific database instance must have access to the storage where the database s tablespaces redo logs and control files are stored Shared disk architecture is also the easiest to implement and maintain When a node or application fails all data required to start on another node is stored on the shared disk GEELI o Figure 2 8 Shared disk architecture for basic cluster 42 About cluster topologies Cluster topologies and storage configurations Shared nothing cluster Systems in shared nothing clusters do not share access to disks they maintain separate copies of data VCS shared nothing clusters typically have read only data stored locally on both systems For example a pair of systems in a cluster that includes a critical Web server which provides access to a backend database The Web server runs on local disks and does not require data sharing at the Web server level Figure 2 9 Shared nothing cluster About cluster topologies 43 Cluster topologies and storage configurations Replicated data cluster In a replicated data cluster there is no shared disk Instead a data replication product synchronizes copies of data between nodes Replication can take place at the application host and storage levels Application level replication products such as Oracle DataGuard maintain consistent copies of data between
624. up s prerequisites first then the corresponding system limits Each system can have a different limit Controlling VCS behavior 393 Service group workload management and there is no cap on the number of group prerequisites and system limits Service group prerequisites and system limits can appear in any order You can also use these attributes to configure the cluster as N to 1 or N to N For example to ensure that only one service group can be online on a system at a time add the following entries to the definition of each group and system Prerequisites GroupWeight 1 Limits GroupWeight 1 System limits and group prerequisites work independently of FailOverPolicy Prerequisites determine the eligible systems on which a service group can be started When a list of systems is created HAD then follows the configured FailOverPolicy Using capacity and limits When selecting a node as a failover target VCS selects the system that meets the service group s prerequisites and has the highest available capacity If multiple systems meet the prerequisites and have the same available capacity VCS selects the system appearing lexically first in the SystemList Systems having an available capacity of less than the percentage set by the LoadWarningLevel attribute and those remaining at that load for longer than the time specified by the LoadTimeThreshold attribute invoke the loadwarning trigger 394 Controlling VCS behavio
625. up Resource Configuration Wizard on page 164 Notifer Resource Configuration wizard Configures the VCS notifier See Notifier Resource Configuration wizard on page 164 Configuring applications and resources in VCS 289 About VCS bundled agents About VCS bundled agents Bundled agents are categorized according to the type of resources they make available Storage agents Storage agents make your shared disks diskgroups volumes and mounts highly available Agent Description m DiskGroup Brings online takes offline and monitors a Veritas Volume Manager VxVM disk group m DiskReservation Reserves disks to guarantee safe and exclusive access to Linux only shared disks m Volume agent Brings online takes offline and monitors a Veritas Volume Manager VxVM volume m LVMLogicalVolume Brings online takes offline and monitors Logical Volume HP UX and Linux Manager LVM logical volumes m LVMVG Activates deactivates and monitors a Logical Volume AIX Manager LVM volume group m LVMVolumeGroup Activates deactivates and monitors a Logical Volume HP UX and Linux Manager LVM volume group m LVMCombo Activates and deactivates logical volumes and volume HP UX only groups m Mount Brings online takes offline and monitors a file system or NFS client mount point m SANVolume Monitors and controls access to a SAN volume managed Linux and Solaris using Storage Foundation Volume Server
626. uration assume that groups probe in the same order they are described G1 through G8 Group G1 chooses the system with the highest AvailableCapacity value All systems have the same available capacity so G1 starts on Server1 because this server is lexically first Groups G2 through G4 follow on Server2 through Server4 With the startup decisions made for the initial four groups the cluster configuration resembles Server AvailableCapacity Online Groups Server1 80 G1 Server2 60 G2 Server3 70 G3 Server4 90 G4 As the next groups come online group G5 starts on Server4 because this server has the highest AvailableCapacity value Group G6 then starts on Server1 with AvailableCapacity of 80 Group G7 comes online on Server3 with AvailableCapacity of 70 and G8 comes online on Server2 with AvailableCapacity of 60 The cluster configuration now resembles Server AvailableCapacity Online Groups Server1 50 G1 and G6 Server2 20 G2 and G8 Server3 50 G3 and G7 Server4 40 G4 and G5 In this configuration Server2 fires the loadwarning trigger after 600 seconds because it is at the default LoadWarningLevel of 80 percent 398 Controlling VCS behavior Sample configurations depicting workload management Failure scenario In the first failure scenario Server4 fails Group G4 chooses Server1 because Server1 and Server3 have AvailableCapacity of 50 and Server is lexically first Group G5 then comes online on Server3
627. uration to read only haconf dump makero Displaying a user Display a list of users and their privileges To display a list of users Type the following command hauser list To display the privileges of all users Type the following command hauser display To display the privileges of a specific user Type the following command hauser display user 254 Administering the cluster from the command line Querying VCS Querying VCS VCS enables you to query various cluster objects including resources service groups systems resource types agents and clusters You may enter query commands from any system in the cluster Commands to display information on the VCS configuration or system states can be executed by all users you do not need root privileges Querying service groups To display the state of a service group on a system Type the following command hagrp state service group sys system To display the resources for a service group Type the following command hagrp resources service group To display a list of a service group s dependencies Type the following command hagrp dep service group To display a service group on a system Type the following command hagrp display service group sys system If service_group is not specified information regarding all service groups is displayed To display attributes of a system Type the following command
628. urations online Failover Child Child is Parent stays online Child stays online global soft online online If Child fails over to Parent fails over to porns aa a another system any available system e cluster in the A cluster ee TEMAS If no system is onime available Parent If Child cannot fail remains offline over Parent remains online online Failover Child Child is Parent taken offline Child stays online global firm online online after Child is Parent fails over to somewhere in somewhere offlined any available system the cluster in the i lister If Child fails over to If no system is another system available Parent Parent is brought remains offline online on any system If Child cannot fail over Parent remains offline online Failover Child Child is If Child fails over to Child stays online remote soft online on online on the system on which Parent fails ov r to another another Parent was online system where Child system system Parent migrates to another system If Child fails over to another system Parent continues to run on original system If Child cannot fail over Parent remains online is not online If the only system available is where Child is online Parent is not brought online If no system is available Child remains online The role of service group dependencies 417 Service group dependency configurations online Failover Child remote firm online on
629. ure B 4 zone root on shared disks direct mounted file system Application Mount DiskGroup As you create the service group note the following points DiskGroup m Set the MountPoint attribute of the Mount resource to the mount path m If the application requires an IP address configure the IP resource in the service group Make sure the ContainerName attribute for the resource is set to the name of the non global zone m Where the zone root file system is on shared storage you can configure separate mounts for the zone and the application as shown in the illustration but you may configure the same disk group for both 602 Configuring VCS in non global zones Configuring VCS in zones To modify the service group configuration 1 Run the hazonesetup p1 script to set up the zone configuration hazonesetup servicegroup name zoneres name zone name password systems servicegroup_name Name of the application service group zoneres_name Name of the resource configured to monitor the zone zone_name Name of the zone password Password to be assigned to VCS or Security Symantec Product Authentication Service user created by the command systems List of systems on which the service group will be configured Use this option only when creating the service group The script adds a resource of type Zone to the application service group It also creates a user account with group administrati
630. ust accept the certificate before accessing the secure Web consoles The SSL protocol prevents malicious users from sniffing Web console data from the network 14300 Administrative port If you use these ports for another application on the system configure VRTSweb to use different ports Retrieving the list of VRTSweb ports Retrieve the list of ports that are configured for VRTSweb To retrieve the list of ports from the command line Run the following command on the system where VRTSweb is installed VRTSWEB HOME bin webgui listports The output displays the list of configured ports and their protocols To retrieve the list of ports from the Web Console 1 Access the Web server using the configured port number for example http hostname 8181 Click the Configuration tab The Configured Ports table on the right side of the Configuration page lists the ports Administering Symantec Web Server 661 Configuring ports for VRTSweb Adding ports for VRTSweb Add ports for use by VRTSweb To add a port from the command line Run the following command on the system where VRTSweb is installed S VRTSWEB HOME bin webgui addport portno protocol bind ip address m portno The port number to be added m protocol The protocol for the port HTTP specifies a normal HTTP port HTTPS specifies a secure SSL port Web servers using the HTTP port can be accessed at http hostname portno Web servers using th
631. ust specify the values of mandatory attributes Administering the cluster from Cluster Manager Java console 201 Administering resources before enabling a resource If a resource is created dynamically while VCS is running you must enable the resource before VCS monitors it VCS will not bring a disabled resource nor its children online even if the children are enabled Click Apply To add a resource from the Template view 1 2 From Cluster Explorer click Templates on the Tools menu In the left pane of the Template View click the template from which to add resources to your configuration In the resource graph right click the resource to be added to your configuration T Template View as on System vcshp71 ol x Template View YolumeNFSGroup Templates reas seee ps3 vNFS_NIC vNFS_Share1 vNFS_Share2 Self Self and Child Nodes vNFS_Volumet vNFS_Volume2 vNFS_DiskGroup Click Copy and click Self from the menu to copy the resource Click Copy and click Self and Child Nodes from the menu to copy the resource with its dependent resources In the Service Groups tab of the Cluster Explorer configuration tree click the service group to which to add the resources In the Cluster Explorer view panel click the Resources tab Right click the Resource view panel and click Paste from the menu After the resources are added to the service group edit the attributes to configure th
632. uster Resource Type Attributes This is the first and only view at the resource type level in the Cluster Management Console To navigate to the Cluster Resource Types view 1 2 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Resource Types In the Cluster Resource Types view the Resource Types Listing table lists all resource types in the selected cluster The task panels contain tasks that you can perform in a one to many fashion on the listed resource types To navigate to the Resource Type Attributes view 1 2 3 On the navigation bar click Cluster On the secondary tab bar click Summary and then click Resource Types In the Cluster Resource Types view in the Resource Types Listing table click a linked resource type name In the Resource Type Attributes view you can choose a task from the task pane or click the button in the Edit column of the table to edit the attribute value Modifying resource type attributes Edit the attributes of a resource type You must have the role of cluster administrator or service group administrator to edit resource type attributes To edit resource type attributes using the attribute table 1 In the Cluster Resource Types view in the Resource Types Listing table click a linked resource name In the Resource Type Attributes view in the All attributes for resource type table locate the line item for the attribute that you want
633. uster 1 2 Link the resources such that the Web server resource depends on the RemoteGroup resource 3 Configure the RemoteGroup resource to monitor or manage the service group containing the database resource m IpAddress Set to the IP address or DNS name of a node in cluster2 You can also set this to a virtual IP address m GroupName Set to OracleGroup 298 Configuring applications and resources in VCS Configuring the RemoteGroup agent m ControlMode Set to OnOff m Username Set to the name of a user having administrative privileges for OracleGroup m Password Encrypted password for defined in Username Encrypt the password using the vesencrypt agent command m VCSSysName Set to local per node values m VCSSysName locali Set this value to remotel m VCSSysName local2 Set this value to remote2 Note If the remote cluster runs in secure mode you must set the value for DomainType or BrokerIp attributes Set the value of the AutoFailover attribute of the OracleGroup to 1 Service group behavior with the RemoteGroup agent Consider the following potential actions to better understand this solution Bringing the Apache service group online The Apache resource depends on the RemoteGroup resource The RemoteGroup agent communicates to the remote cluster and authenticates the specified user The RemoteGroup agent brings the database service group online in cluster2 The Apache resource com
634. uster Use the console to add delete freeze and unfreeze systems Adding a system Cluster Explorer and Command Center enable you to add a system to the cluster A system must have an entry in the LLTTab configuration file before it can be added to the cluster To add a system from Cluster Explorer 1 4 On the Edit menu click Add and click System or Click Add System on the Cluster Explorer toolbar Enter the name of the system Click Show Command in the bottom left corner to view the command associated with the system Click Hide Command to close the view of the command Click OK To add a system from Command Center 1 Deleting a system Click Add System in the Command Center toolbar or In the Command Center configuration tree expand Commands gt Configuration gt Cluster Objects gt Add System Enter the name of the system Click Apply To delete a system from Command Center 1 2 3 In the Command Center configuration tree expand Commands gt Configuration gt Cluster Objects gt Delete System Click the system Click Apply Freezing a system Administering the cluster from Cluster Manager Java console 219 Administering systems Freeze a system to prevent service groups from coming online on the system To freeze a system from Cluster Explorer 1 2 Click the Systems tab of the configuration tree In the configuration tree right click the system click Freeze and click
635. uster Resources or Group Summary views to administer a global service group To navigate to the Cluster Summary view 1 2 3 On the main tab bar click Manage On the secondary tab bar click Summary In the Cluster Summary view choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster To navigate to the Cluster Resources view 1 2 3 On the main tab bar click Manage On the secondary tab bar click Summary and then click Resources In the Cluster Resources view choose a task from the task pane or select another cluster level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the cluster To navigate to the Group Summary view 1 2 3 On the main tab bar click Manage On the secondary tab bar click Groups In the Cluster Groups view in the Groups Listing table click a linked service group name In the Group Summary view choose a task from the task pane or select another service group level view using the tabs on the secondary tab bar Each view contains information and tasks for administering the service group Administering global clusters from the Cluster Management Console 495 Administering global service groups Converting local service groups to global service groups Use the Global Group Configuration wizard to configure a ser
636. uster automatically if it is unable to fail over within the local cluster or if the entire cluster faults Connected The group fails over automatically to another cluster only if it is unable to fail over within the local cluster m Type and dimension string scalar m Default Manual ClusterList user defined Specifies the list of clusters on which the service group is configured to run m Type and dimension integer association m Default Not applicable CurrentCount system use only Number of systems on which the service group is active m Type and dimension integer scalar m Default Not applicable DeferAutoStart system use only Indicates whether HAD defers the auto start of a local group in case the global cluster is not fully connected m Type and dimension boolean scalar m Default Not applicable Table D 3 VCS attributes 631 Service group attributes Service group attributes Enabled user defined Indicates if a group can be failed over or brought online If any of the local values are disabled the group is disabled m Type and dimension boolean scalar m Default 1 enabled Evacuate user defined Indicates if VCS initiates an automatic failover when user issues hastop local evacuate m Type and dimension boolean scalar m Default 1 Evacuating system use only Indicates the node ID from which the service group is being evacuated m
637. uster while the service group is running on the primary cluster data corruption does not occur because the clusters use replicated data Instead divergent data sets result which must be merged manually once the split brain is resolved VCS does not automatically take a service group offline after an inter cluster split brain is reconnected Connecting clusters Creating global clusters 469 Prerequisites for global clusters Prerequisites for global clusters This section describes the prerequisites for configuring global clusters Cluster setup You must have at least two clusters to set up a global cluster Every cluster must have the required licenses A cluster can be part of one global cluster VCS supports a maximum of four clusters participating in a global cluster Clusters must be running on the same platform the operating system versions can be different Clusters must be using the same VCS version Cluster names must be unique within each global cluster system and resource names need not be unique across clusters Service group names need not be unique across clusters however global service groups must have identical names Every cluster must have a valid virtual IP address which is tied to the cluster Define this IP address in the cluster s ClusterAddress attribute This address is normally configured as part of the initial VCS installation The IP address must have a DNS entry For remote cluster operations you
638. usually at system boot AutoStartList SystemA SystemB SystemC An association is an unordered list of name value pairs Use a comma or a semi colon to separate values A set of braces after the attribute name denotes that an attribute is an association For example to designate the list of systems on which the service group is configured to run and the system s priorities SystemList SystemA 1 SystemB 2 SystemC 3 VCS configuration concepts 53 About VCS attributes Attributes and cluster objects VCS has the following types of attributes depending on the cluster object the attribute applies to Cluster attributes Service group attributes System attributes Resource type attributes Resource attributes Attributes that define the cluster For example ClusterName and ClusterAddress Attributes that define a service group in the cluster For example Administrators and ClusterList Attributes that define the system in the cluster For example Capacity and Limits Attributes that define the resource types in VCS These can be further classified as m Type independent Attributes that all agents or resource types understand Examples RestartLimit and MonitorInterval these can be set for any resource type Typically these attributes are set for all resources of a specific type For example setting MonitorInterval for the IP resource type affects all IP resources
639. ut creating global service groups About creating global service groups The process of creating a global cluster environment involves m Creating a common service group on specified local clusters and on clusters at a remote location m Making sure the common service group is capable of being brought online on the local and remote clusters m Designating the remote clusters as failover targets for the common service group adding remote clusters m Converting the service group that is common to all the clusters to a global service group creating the global service group After you set up the global cluster environment you can configure global cluster heartbeats to monitor the health of the failover target clusters Use the Cluster Management Console to create global service groups to add and delete remote clusters and to create and monitor cluster heartbeats How VCS global clusters work on page 462 Note The Cluster Management Console provides disabled individuals access to and use of information and data that is comparable to the access and use provided to non disabled individuals See Accessibility and VCS on page 679 Administering global clusters from the Cluster Management Console 489 Administering global clusters Administering global clusters Creating a global cluster environment requires the following conditions The ClusterService group for all clusters must be properly configured for GCO operations T
640. vailability management with i3 m Type and dimension string association m Default VCSFeatures Indicates which VCS features are enabled Possible values are system use only O No features are enabled VCS Simulator 1 L3 is enabled 2 Global Cluster Option is enabled m Type and dimension integer scalar m Default Not applicable VCSMode Denotes the mode for which VCS is licensed system use only m Type and dimension integer scalar m Default Not applicable WACPort The TCP port on which the wac Wide Area Connector process on the local cluster listens for connection from remote clusters Type and dimension integer scalar m Default 14155 user defined VCS attributes Heartbeat attributes 655 Heartbeat attributes Table D 6 lists the heartbeat attributes These attributes apply to global system use only clusters Table D 6 Heartbeat attributes AgentState The state of the heartbeat agent m Type and dimension integer scalar m Default INIT Arguments user defined List of arguments to be passed to the agent entry points For the Icmp agent this attribute can be the IP address of the remote cluster m Type and dimension string vector m Default AYAInterval user defined The interval in seconds between two heartbeats m Type and dimension integer scalar m Default 60 seconds AYARetryLimit user defined The maximum number of lost heartbeat
641. ve privileges to enable inter zone communication If the application service group does not exist the script creates a service group with a resource of type Zone Set the ContainerName attribute for the IP resource to the name of the non global zone Modify the resource dependencies to reflect your zone configuration See the resource dependency diagrams for more information Verify the ContainerType attribute for the application resource type is set to Zone Set the ContainerName attribute for the application resource to the name of the zone in which the application runs Save the service group configuration and bring the service group online Configuring VCS in non global zones 603 Verifying the zone configuration Verifying the zone configuration Verify the zone configuration by running the hazoneverify command The command verifies the following requirements The systems hosting the service group have the required operating system to run zones The service group does not have more than one resource of type Zone The dependencies of the Zone resource are correct To verify the zone configuration 1 If you use custom agents make sure the resource type is added to the APP_TYPES or SYS_TYPES environment variable See Using custom agents in non global zones on page 594 Run the hazoneverify command to verify the zone configuration hazoneverify servicegroup name Maintenance tasks Whenever you make a
642. ve service groups to another system in the cluster before the system is stopped noautodisable Ensures that service groups that can run on the node where the hastop command was issued are not autodisabled This option can be used with evacuate but not with force sys Stops HAD on the specified system Stopping VCS without force option When VCS is stopped on a system without using the force option it enters the LEAVING state and waits for all groups to go offline on the system Use the output of the command hasys display system to verify that the values of the SysState and the OnGrpCnt attributes are non zero VCS continues to wait for the service groups to go offline before it shuts down See Troubleshooting resources on page 571 238 Administering the cluster from the command line Stopping VCS Stopping VCS with options other than force When VCS is stopped by options other than force on a system with online service groups the groups running on the system are taken offline and remain offline This is indicated by VCS setting the attribute IntentOnline to 0 Using the option force enables service groups to continue running while HAD is brought down and restarted IntentOnline remains unchanged Controlling the hastop behavior using the EngineShutdown attribute Use the EngineShutdown attribute to define VCS behavior when a user runs the hastop command Note VCS does not consider this attribute when the
643. ver3 Capacity 200 Limits ShrMemSeg 15 Semaphores 30 Processors 18 LoadWarningLevel 80 LoadTimeThreshold 900 system MedServerl Capacity 100 Limits ShrMemSeg 5 Semaphores 10 Processors 6 404 Controlling VCS behavior Sample configurations depicting workload management system MedServer2 Capacity 100 Limits ShrMemSeg 5 Semaphores 10 Processors 6 system MedServer3 Capacity 100 Limits ShrMemSeg 5 Semaphores 10 Processors 6 system MedServer4 Capacity 100 Limits ShrMemSeg 5 Semaphores 10 Processors 6 system MedServer5 Capacity 100 Limits ShrMemSeg 5 Semaphores 10 Processors 6 group Databasel SystemList LargeServerl LargeServer2 LargeServer3 MedServerl MedServer2 MedServer3 MedServer4 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartList LargeServerl LargeServer2 LargeServer3 FailOverPolicy Load Load 100 Prerequisites ShrMemSeg 5 Semaphores 10 Processors 6 group Database2 SystemList LargeServerl LargeServer2 LargeServer3 MedServerl MedServer2 MedServer3 MedServer4 MedServer5 SystemZones LargeServerl 0 LargeServer2 0 LargeServer3 0 MedServerl 1 MedServer2 1 MedServer3 1 MedServer4 1 MedServer5 1 AutoStartPolicy Load AutoStartLis
644. verPolicy is set to either Auto or Connected and VCS is unable to determine a valid remote cluster to which to automatically fail the group over m The ClusterFailOverPolicy attribute is set to Connected and the cluster in which the group has faulted cannot communicate with one ore more remote clusters in the group s ClusterList Troubleshooting and recovery for VCS 577 Troubleshooting and recovery for global clusters m The wide area connector wac is not online or is incorrectly configured in the cluster in which the group has faulted Managing alerts Alerts require user intervention You can respond to an alert in the following ways m Ifthe reason for the alert can be ignored use the Alerts dialog box in the Java or Web consoles or the haalert command to delete the alert You must provide a comment as to why you are deleting the alert VCS logs the comment to engine log m Take an action on administrative alerts that have actions associated with them You can do so using the Java or Web consoles m VCS deletes or negates some alerts when a negating event for the alert occurs An administrative alert will continue to live if none of the above actions are performed and the VCS engine HAD is running on at least one node in the cluster If HAD is not running on any node in the cluster the administrative alert is lost Actions associated with alerts This section describes the actions you can perform from the Java and the Web
645. version of VCS 580 Troubleshooting and recovery for VCS Troubleshooting licensing Licensing Your evaluation period for the feature has expired This feature will not be enabled the next time VCS starts The evaluation period for the specified VCS feature has expired Section Appendixes m Appendix A VCS user privileges administration matrices on page 583 m Appendix C Cluster and system states on page 605 m Appendix D VCS attributes on page 613 m Appendix E Administering Symantec Web Server on page 657 m Appendix F Accessibility and VCS on page 679 582 Appendixes Appendix VCS user privileges administration matrices m About administration matrices m Administration matrices About administration matrices In general users with Guest privileges can execute the following command options display state and value Users with privileges for Group Operator and Cluster Operator can execute the following options online offline and switch Users with Group Administrator and Cluster Administrator privileges can execute the following options add delete and modify See About the VCS user privilege model on page 61 584 VCS user privileges administration matrices Administration matrices Administration matrices Review the matrices in the following section to determine which command options can be executed within a specific user role Checkmarks de
646. vice Group Online for the node to which the service group failed over How to detect service group switch When a service group is switched VCS sends notification to indicate the following events m Service group is being switched m Service Group Offline for the node from which the service group is switched m Service Group Online for the node to which the service group was switched This notification is sent after VCS completes the service group switch operation Note You must configure appropriate severity for the notifier to receive these notifications To receive VCS notifications the minimum acceptable severity level is Information 444 VCS event notification Detecting complementary events Detecting complementary events Table 13 1 lists some events that complement each other or cancel each other out Table 13 1 Complementary events in VCS Remote cluster has faulted Global Cluster Option Remote cluster is in RUNNING state Heartbeat is down Heartbeat is alive Agent is faulted Agent is restarting Resource state is unknown Resource went online by itself Health of cluster resource declined Health of cluster resource improved VCS has faulted Anode running VCS has joined cluster Service group has faulted Service group is online Service group is offline Service group is online Service group is being switched Service group is online
647. vice group in a local cluster as a global service group To convert a service group on a local cluster to a global service group 1 In the Cluster Summary view in the Groups Listing table click the linked name of the service group that you want to convert This service group should already have been commonly configured on at least one local and one remote cluster In the Group Summary view in the Configuration task panel click Configure Global Group In the Global Group Configuration wizard read the introductory information and click Next In the Cluster List Configuration dialog box under Available Clusters select the clusters on which the global service group can come online To select a cluster click the right arrow button to move the cluster name under Selected Clusters Select the policy for service group failover and then click Next m Manual prevents a service group from automatically failing over to another cluster m Auto enables a service group to automatically fail over to another cluster if it is unable to fail over within the cluster or if the entire cluster faults m Connected enables a service group to automatically fail over to another cluster if it is unable to fail over within the cluster In this step you update the cluster list of remaining instances of the selected global service group To perform the update you must first verify or supply the authentication credentials for each remaining global c
648. written to disks and does not declare the DiskGroup resource as offline This behavior prevents potential data corruption that may be caused by the disk group being imported on two hosts However this also means that service groups remain online on a node that does not have storage connectivity and the service groups cannot be failed over unless an administrator intervenes This affects application availability Some Fibre Channel FC drivers have a configurable parameter called failover which defines the number of seconds for which the driver retries I O commands before returning an error If you set the failover parameter to 0 the FC driver retries I O infinitely and does not return an error even when storage connectivity is lost This also causes the Monitor entry point for the DiskGroup to time out and prevents failover of the service group unless an administrator intervenes About disk group configuration and VCS behavior Table 11 2 describes how the disk group state and the failover attribute define VCS behavior when a node loses connectivity to shared storage Table 11 2 Disk group state and VCS behavior 1 Enabled Nseconds Fails over service groups to another node 2 Disabled Nseconds DiskGroup resource remains online No failover 3 Enabled 0 DiskGroup resource remains in monitor timed out state No failover 4 Disabled 0 DiskGroup resource remains online No failover Controlling VCS behavior VCS be
649. xist in the same multi cluster environment with single cluster mode installations on VCS cluster nodes and multi cluster mode installations on management server hosts Such a deployment can be desirable if different IT administrators in your enterprise have different scopes of responsibility See the Veritas Cluster Server Installation Guide for information about installing Cluster Management Console See the Veritas Cluster Server Centralized Management Guide for information about managing multiple clusters using Cluster Management Console This chapter describes how to use the Cluster Management Console in single cluster mode 72 Administering the cluster from the Cluster Management Console Verifying installation and browser requirements Verifying installation and browser requirements For information about browser support browser settings how to verify console installation and other basic prerequisite conditions see the Veritas Cluster Server Installation Guide Configuring the Cluster Management Console manually The resources required for the Cluster Management Console are configured in the ClusterService group This configuration is performed by the VCS installation program if the Cluster Management Console option is selected during installation If the Cluster Management Console option was not selected during VCS installation you must create and configure the ClusterService service group manually To configure the
650. y m UMI a unique message ID m Message Text the actual message generated by VCS A typical engine log resembles 2003 02 10 16 08 09 VCS INFO V 16 1 10077 received new cluster membership The agent log is located at var VRTSvcs log agent_A log The format of agent log messages resembles Timestamp Year MM DD Mnemonic Severity UMI Agent Type Resource Name Entry Point Message Text A typical agent log resembles 2003 02 23 10 38 23 VCS WARNING V 16 2 23331 Oracle VRT monitor Open for ora_lgwr failed setting cookie to null Note that the logs on all nodes may not be identical because m VCS logs local events on the local nodes m All nodes may not be running when an event occurs Message catalogs Troubleshooting and recovery for VCS 565 Logging VCS includes multilingual support for message catalogs These binary message catalogs BMCs are stored in the following default locations The variable language represents a two letter abbreviation opt VRTS messages language module_name The VCS command line interface displays error and success messages in VCS supported languages The hamsg command displays the VCS engine logs in VCS supported languages The BMCs are gcoconfig bmc hawizard bmc VRTSvcsHbfw bmc VRTSvesTriggers bmc VRTSvcsWac bmc vxfen bmc gab bmc hagetcf bmc llt bmc VRTSvcsAgfw bmc VRTSvesAlerts bmc VRTSvcsApi bmc VRTSvcsCommon bmc VRTSvcsHad bmc VRTSvecsplatformAgent bmc V
651. y also enable VCS to recognize which systems are active members of the cluster which have joined or left the cluster and which have failed About the high availability daemon HAD The VCS high availability daemon HAD runs on each system Also known as the VCS engine HAD is responsible for m building the running cluster configuration from the configuration files m distributing the information when new nodes join the cluster m responding to operator input m taking corrective action when something fails The engine uses agents to monitor and manage resources It collects information about resource states from the agents on the local system and forwards it to all cluster members The local engine also receives information from the other cluster members to update its view of the cluster HAD operates as a replicated state machine RSM The engine running on each node has a completely synchronized view of the resource status on each node Each instance of HAD follows the same code path for corrective action as required The RSM is maintained through the use of a purpose built communications package consisting of the protocols Low Latency Transport LLT and Group Membership Services Atomic Broadcast GAB See About inter system cluster communications on page 335 The hashadow process monitors HAD and restarts it when required About the I O fencing module The I O fencing module implements a quorum type functionality to en
652. ys with each coordinator disk Figure 10 3 Topology of coordinator disks in the cluster SystemO System1 System2 System 3 SAN Connection CE LLT Links Coordinator Disks How membership arbitration works Upon startup of the cluster all systems register a unique key on the coordinator disks The key is based on the LLT system ID for example LLT ID 0 A When there is a perceived change in membership membership arbitration works as follows m GAB marks the system as DOWN excludes the system from the cluster membership and delivers the membership change the list of departed systems to the fencing module m The system with the lowest LLT system ID in the cluster races for control of the coordinator disks m Inthe most common case where departed systems are truly down or faulted this race has only one contestant m Ina split brain scenario where two or more subclusters have formed the race for the coordinator disks is performed by the system with the lowest LLT system ID of that subcluster This system races on behalf of all the other systems in its subcluster m The race consists of executing a preempt and abort command for each key of each system that appears to no longer be in the GAB membership The preempt and abort command allows only a registered system with a valid key to eject the key of another system This ensures that even when multiple systems attempt to eject other each race will have on
653. ystem and duplicate the configuration on each node in the cluster Setting the zone root on shared storage means you need install the non global zone on shared storage from one system only The non global zone and the zone root can fail over to the other systems To do this the system software including the patches must be identical on each system during the existence of the zone 596 Configuring VCS in non global zones Configuring VCS in zones Creating a zone root on local disk Use the following steps to create zone root on the local disk of each system The file system typically for data that is on a shared device is of the loop back type or direct mount Choose one of the procedures that follow To create zone root on local disks with data using loop back file system 1 o oo N DD O 10 11 Create the zone with the zonecfg command zonecfg z newzone zonecfg newzone gt create Set the zonepath parameter to specify a location for the zone root zonecfg newzone gt set zonepath export home newzone Create a loop back file system in the zone Set a virtual IP address of the system to be the IP address of the zone and define the device name for the NIC associated with the IP address For example Exit the zonecfg configuration Create the actual zone root directory Set permissions for the zone root directory Install the non global zone using zoneadm Repeat the previous zone creation steps on each other sys
654. zard Options dialog box select to create a new service group or modify an existing group The wizard allows only one NFS service group in the configuration If you choose to modify a service group you can add and remove shares from the service group You can also modify the configuration of the IP and NIC resources Click Next Specify the service group name and the system list VCS NFS Configuration Wizard Service Group Configuration Specify the service group name and the failover priority for each system Service Group Name INFSGroup Group System List Available Cluster Systems Syst Priority Order 84 5 More information lt Back Next gt cancer m Enter a name for the service group m In the Available Cluster Systems box select the systems on which to configure the service group and click the right arrow icon to move the systems to the service group s system list To remove a system from the service group s system list select the system in the Systems in Priority Order box and click the button with the left arrow icon m To change a system s priority in the service group s system list select the system in the Systems in Priority Order box and click the buttons with the up and down arrow icons The system at the top of the list has the highest priority while the system at the bottom of the list has the lowest priority m Click Next 312 Configuring applications and resources in VCS Conf
655. ze of queue for messages queued by VCS for notifier m Type and dimension string association m Default Different values for each parameter OperatorGroups List of operating system user groups that have Operator user defined privileges on the cluster This attribute is valid only in secure clusters m Type and dimension string keylist m Default Operators List of users with Cluster Operator privileges user defined m Type and dimension string keylist m Default PanicOnNoMem For internal use only system use only m Type and dimension boolean scalar m Default Not applicable PrintMsg Enables logging TagM messages in engine log if set to 1 user defined m Type and dimension boolean scalar m Default 0 ProcessClass The scheduling class for HAD processes for example triggers user defined m Type and dimension string scalar m Default TS Table D 5 VCS attributes Cluster attributes 653 Cluster attributes ProcessPriority user defined The priority of HAD processes for example triggers m Type and dimension string scalar m Default ReadOnly user defined Indicates that cluster is in read only mode m Type and dimension integer scalar m Default 1 ResourceLimit user defined Maximum number of resources m Type and dimension integer scalar m Default 5000 SecureClus Indicates whether the cluster runs in secure mode The val
656. zone root 595 zones attributes 592 creating a root on local disk 596 creating a root on shared disk 597 prerequisites 594 setting zone root 595 supported agents 592 using custom agents 594

Veritas Cluster Server User's Guide

Contents

Download Pdf Manuals

Related Search

Related Contents