Home

Chapter 1 TrueCopy Agent for VERITAS Cluster Server

1. lt 9 t 68 5 4 6 MONITOR Entry Point Timeout Gu nnn 68 5 5 System Recovery Procedures eee ee ht 69 5 5 1 Recovery Procedure for a Split Brain Situation 69 5 5 2 Recovery Procedure for All TrueCopy Links Failed with Fence Level DATA 71 5 5 3 Falling Back after Failover Process is Complete 73 5 6 Calling the Hitachi Data Systems Support Center or VERITAS Technical Support 75 Chapter 6 Messages 6 1 Message Foal ac a 9d Pa ane RED V EOS Ud PU ed 77 o Message ll a ue E ee AG OTI 77 6 2 1 Message ID 3000001 3000099 FC GG uu uu 77 6 2 2 Message ID 3000100 3000279 F FF ee eens 78 6 2 3 Message ID 3000400 3000499 llle 78 6 3 MessadeS coe ge ac o D foro re YC ov au ette ro RADI UR oe WD 79 Acronyms and AbbreviationS L Y FF FF GG I uu iu 85 viii Contents List of Figures Figure 1 1 Figure 2 1 Figure 3 1 Figure 3 2 Figure 3 3 Figure 3 4 Figure 3 5 Figure 3 6 Figure 3 7 Figure 3 8 Figure 3 9 Figure 3 10 Figure 3 11 Figure 3 12 Figure 3 13 Figure 3 14 Figure 3 15 Figure 3 16 Figure 3 17 Figure 4 1 Figure 4 2 Figure 4 3 Figure 4 4 Figure 4 5 Figure 4 6 Figure 4 7 Figure 4 8 Figure 4 9 Figure 5 1 Figure 5 2 Figure 5 3 Figure 5 4 Figure 5 5 Figure 5 6 Figure 6 1 Overview of TrueCopy Agent VERITAS Cluster Server Hitachi TrueCopy 1 Resource Type Definition o
2. M fe Attributes View Truecopy E Properties eh Md ON LL P ee e a A anoaiiaeaii GM f carm BOTTO 8 c UNS com rh s oorr Gw e MEL evel mem chem om com v D LE JPEIULOYY CAM GLU IWS 7 vcscluster2 anne LogLevel LogTags Monitorinterval MonitorTimeout NameRule Num Threads Dd OfflineMonitorinte IY t Lh REC Dw Wigs Scu WINE REL YLIT IL OnlineTimeout OnlinevvaitLimit OpenTimeout Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide T REIS Glabal Global Global Global Glabal Glnbal GM Usual Global Global Global AST Scalar Scalar Scalar Scalar Scalar LL Statar oSLalar Scalar Scalar Scalar it OnlineTimeout Time after which the agent should stop onlining the resource and consider error ZH Name Scope Dimension Value Edit AgentClass Global Scalar TS AgentF ailedOn Global Keylist AgentPriority Global Scalar lAgentRepyTimeout Global Sealar 1 130 CleanTimeout Global Scalar 60 TT CloseTimeout Global Scalar 60 tn i A error E ABCDEFGH wal E 60 EM 60 EHI TrueCopy amp resou 10 THE a EEE Iv SDI ae Oa SS Em a uE 420 sad 2 E ou ae Changing the OnlineTimeout Value for the TrueCopy Type Resource 31 3 5 2 Setting the TrueCopy Resource Online Timeout Value Using the CLl To set the OnlineTimeout value of the TrueCopy resource using the GU
3. Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 49 4 4 3 Server Failure at the Primary Site In case the heartbeat communication between nodes in the primary site and secondary site is stopped by a failure such as a server failure VCS in the secondary site detects the failure of primary site and directs the TrueCopy Aegnt to call the ONLINE entry point The ONLINE entry point executes the takeover process to bring the TrueCopy resource Online 50 Chapter 4 TrueCopy Agent Operations 4 44 Failure at the Primary Site of the Lightning 9900 Subsystem In the unlikely event that the Lightning 9900V or 9900 at the primary site is down the server would receive I O errors from the Lightning 9900 9900V and the service group would fail VCS at the secondary site detects the failure of the primary site and directs the TrueCopy Agent to call the ONLINE entry point The ONLINE entry point calls the takeover function for CCI to bring the resource Online Note Refer to the section 5 5 3 regarding the fallback to the primary site LAN Failover Process 1 VCS directs TrueCopy Agent to call ONLINE entry point ONLINE entry point issues CCI command to switch S VOL to P VOL CCI performs takeover TrueCopy Agent returns the status to VCS VCS starts up the application Figure 4 4 Failure at the Primary Site of the Lightning 9900 Subsystem Hitachi TrueCopy Agent for VERITAS Cluster Server Ins
4. Lightning 9900V 9900 Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 73 74 Fallback The recovery procedure to fall back the service from the secondary site to the primary site is as follows 1 Recover the server TrueCopy link and Lightning 9900 subsystem at primary site and then start VCS At this point the service has been continued at the secondary site Execute the following command at the secondary site before switch back the service pairresync g lt DevGrp gt swaps lt DevGrp gt Device Group Name for TrueCopy Note Refer to CCI User s Guide for information on the pairresync command Switch back the service to primary site VERITAS Cluster Server Command hagrp switch lt SGname gt to lt Sysname gt lt SGname gt Service Group Name lt Sysname gt System Name Note Refer to the VERITAS Cluster Server user s guide for SWITCH BACK procedure for more detail The service is switched back to Primary site and the service continues Chapter 5 Troubleshooting 5 6 Calling the Hitachi Data Systems Support Center or VERITAS Technical Support If you need to call the Hitachi Data Systems Support Center make sure to provide as much information about the problem as possible including m The circumstances surrounding the error or failure m The exact content of any error messages displayed and or logged on the host system s m The reference codes and severity levels of the mos
5. Note The term 9900 refers to the entire Hitachi Lightning 9900 subsystem family e g 9960 9910 unless otherwise noted Note The use of Hitachi TrueCopy the TrueCopy Agent and all other Hitachi Data Systems products is governed by the terms of your license agreement s with Hitachi Data Systems Software Version This document revision applies to TrueCopy Agent version 1 0 and higher COMMENTS Please send us your comments on this document doc commentsehds com Make sure to include the document title number and revision Please refer to specific page s and paragraph s whenever possible All comments become the property of Hitachi Data Systems Corporation Thank you Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide V vi Preface Contents Chapter 1 Chapter 2 Chapter 3 Chapter 4 TrueCopy Agent for VERITAS Cluster Server 1 1 1 2 TrueCopy Agent for VERITAS Cluster Server uu Important Terms and Concepts Y Y Y hn Requirements for TrueCopy Agent Operations 2 1 2 2 2 3 2 4 2 5 System Requirements s a Ve OR ER DI ce oC GU EACUS USER AC UNE E TrueCopy Pair Type ReguirementsS eee hn tmn Notice on Data Consistency in the Secondary Volumes CCI Device Group Requirements m9 nr VERITAS Cluster Server Setup Reguirements cece cee cece cece Installing
6. Server C CCI 6 TrueCopy Agent calls Takeover function to CCI and then the volume status change to P Vol 7 CCl executes the Nop takeover that the object volume is PVOL 8 TrueCopy Agent returns the online status to VCS which is called by MONITOR entry point 9 VCS will be the online for the other resource 10 VCS starts up the application Figure 4 8 Switching the Service Group from the Primary to Secondary Site Manually Chapter4 TrueCopy Agent Operations 4 4 9 Moving the Application from Primary Server to Secondary Site Manually This example assumes that the movement of the application from the primary site to the secondary site is for load balancing between the nodes or movement of the data center Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 57 58 Chapter 4 TrueCopy Agent Operations Chapter5 Troubleshooting 5 1 General Troubleshooting Please perform troubleshooting procedures to analyze logs when you detect a failure or problem e g checking log periodically or notification of VCS and determine that some causes of the problem might exist in Truecopy Agent for VCS Make sure that all TrueCopy Agent requirements are met see Chapter 2 Check all input values and parameters to make sure that you entered the correct information Table 5 1 lists the troubleshooting cases for TrueCopy Agent operations and the section s to which you should refer For further informat
7. root CounterInterval 5 Factor runque 5 memory 1 disk 10 cpu 25 network 5 MaxFactor rungue 100 memory 10 disk 100 cpu 100 network 100 cDRpdxPmHpzS system beta system alpha snmp vcs TrapList 1 A new system has joined the VCS Cluster An existing system has changed its state A service group has changed its state One or more heartbeat links has gone down An HA service has done a manual restart An HA service has been manually idled An HA service has been successfully started group TestSG SystemList alpha beta Mount Test MntO MountPoint test0 BlockDevice alpha dev dsk c2t0d9s0 BlockDevice beta dev dsk c2t0d23s0 FSType vxfs TrueCopy Test TrueCopy Critical O GroupName TestGr Instance Figure 3 17 Example of Main cf File continues on next page Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 41 Process Test Proc PathName test0 Test sh Arguments alpha alpha Arguments beta beta Test_MntO requires Test TrueCopy Test Proc requires Test MntO0 resource dependency tree group TestsG Process Test Proc Mount Test MntO0 TrueCopy Test TrueCopy Figure 3 17 Example of Main cf File continued 42 Chapter 3 Installing the TrueCopy Agent Chapter 4 TrueCopy Agent Operations 4 1 Entry Points Only VCS can che
8. testO in this example as follows amp bin ksh while ture do echo 1 dev console echo is alive dev console sleep 5 done Save the shell script and change its mode chmod 744 test0 Test sh Unmount the file system tcd umount testO Create the configuration definition file for the CCI instance a Use the inqraid command to find the command device In this example device c2t0d15 is the command device CM is appended to the product ID ls dev rdsk HORCM usr bin ingraid CLI DEVICE FILE PORT SERIAL LDEV CIG H M 12 SSID R Group PRODUCT ID c0t0d0s2 ST34371W SUN4 2G c0t1d0s2 ST34371W SUN4 2G c0t6d0s2 c2t0d5s2 c2t0d6s2 c2t0d7s2 G2t0d952 G2t0d9s2 c2t0d10s2 s s ss s s ss s s ss s s ss s s ss s s ss OPEN 9 OPEN 9 OPEN 9 OPEN 9 OPEN 9 OPEN 9 SUN SUN SUN SUN SUN SUN c2t0d15s2 OPEN 9 CM b Make a copy of the sample configuration file horcm conf in the etc directory Copy this file as horcm0 conf cp etc horcm conf etc horcm0 conf Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 15 c Edit the horcm0 conf file to add the HORCM MON and HORCM CMD information tt KKK KKK CK KK ek KK KK For HORCM MON AK IKK I Ck I KK I k Ck I k ck I KK I KK KK I KK KK KK f HORCM MON fip address Service poll 10ms timeout 10ms alpha horcmO 1000 3000 it ft KKK KKK KK KK IKK IK KK KK k F
9. horctakeover command when VERITAS Cluster Server calls online to the Agent The TrueCopy Agent decides whether to execute the horctakeover command based on the TrueCopy volume resource status Note TrueCopy issues the horctakeover command but CCI decides what type of horctakeover should be executed swap takeover PVOL_Takeover S VOL SSUS takeover No operation etc Therefore the TrueCopy Agent does not directly control TrueCopy failover When a pair status is S VOL in SSUS the TrueCopy Agent does not issue the horctakeover command and loes the message into the VERITAS Cluster Server log Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 43 4 1 2 MONITOR Entry Point Behavior of TrueCopy Agent 44 In general when the MONITOR entry point is called the TrueCopy Agent checks the TrueCopy volume pair status and then sends the resource status to VERITAS Cluster Server But in the following case the TrueCopy Agent issues the horctakeover command Monitoring TrueCopy resource TrueCopy Agent checks the TrueCopy volume pair status and responds the resource status to VCS based on the following criteria TrueCopy Agent judges the status of TrueCopy resource from the following factors Working environment of TrueCopy Agent is initialized Status of TrueCopy volume pair Regarding the TrueCopy volume pair status if the status of the TrueCopy volume pair is S VOL in any status except
10. 0 CL1 A 30039 OPEN 9 0 CL1 A 30039 OPEN 9 dev rdsk c2t0d9s2 16 Chapter 3 Installing the TrueCopy Agent h Edit the horcmO conf file to add the HORCM DEV and HORCM INST information note that the CCI device group name in this example is TestGr df KKK KKK KC CK kk ke ek Ie ke e For HORCM MON AK IKK I IK kc AK kc k k kc IK kck kckck ckckck kckck kk k ke ke e v x HORCM MON fip address Service poll 10ms timeout 10ms alpha horcmO0 1000 3000 it ft KKK KKK KK KK KK IK Ie ke KK KK For HORCM CMD ACkCkCk kCk k Ck IK IK I IK I KK IK IK I KK IK I KK KK HORCM CMD fdev_ name dev name dev name dev rdsk c2t0d15s2 it ft KKK KKK KK KK KK IK ke ke KK KK For HORCM DEV XACkCKCk Ck I k Ck Ik IK kc k Ck k OK I KK I IK KK IK I KK KK HORCM DEV fdev group dev name porti TargetID LU MU TestGr test00 CL1 A 0 9 0 it ft KKK KK KKK KK I KK IK e ke KKK For HORCM INST KK I KK I KK kc k k I IK I KK I IK KK IK I KK KK HORCM ENST fdev group ip addess service TestGr beta horcmO 9 Stop and restart the CCI instance horcmshutdown sh O horcemstart sh O 10 Use the raidqry command to make sure that the CCI instance is running You will see the following output if the CCI instance is running raidgry 1 No Group Hostname HORCM ver Uid Serial Micro ver Cache MB alpha NN NN NN NN O NNNNN NN NN NN NN NNN Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 17 To configure the C
11. Agent Figure 3 13 Entering the CCI Device Group Name Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 37 Figure 3 14 Enabling the New Resource 38 Chapter 3 Installing the TrueCopy Agent vcsclustera Cluster Explorer root Cluster Administrator 3 B amp OF ssl El Resouces Properties File Edit view Tools Help DB Truecopy Figure 3 15 Successful Installation of the TrueCopy Resource Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 39 Figure 3 16 Example of TrueCopy Resource Icon in a Complex Service Group 40 Chapter 3 Installing the TrueCopy Agent 3 6 2 Editing the main cf File to Add the TrueCopy Resource If you manually installed pkgadd VERITAS Cluster Server then you need to manually edit and define the main cf file to add the TrueCopy resource Figure 3 17 shows a main cf example which details the use of the TrueCopy volume resource in a system definition and a service group definition Hitachi provides the HTCTypes cf file that defines TrueCopy volume resource Note The sample main cf file in Figure 3 17 corresponds to the sample system configuration described in section 3 2 refer to Figure 3 2 Note Please refer to the VERITAS Cluster Server user documentation for further information on adding a resource by editing the main cf file include types cf include HTCTypes cf cluster vcs UserNames
12. I KK IK KK KK KK For HORCM DEV 1K IK kK I k Ck k IK I KK I IK I KK I IK I KK IK IK KK KK HORCM_DEV dev_group dev_name port Target ID LU MU it tt KKK KKK KK KK I KK KK KK KKK For HORCM INST KK k k Ck kck k I IK kck kckck kckck IK I KK IK I KK KK HORCM_INST fdev group ip addess service d Edit the etc services file to set up the service port name and number add the following line You must use a unigue number for the horcm service port number You must also perform the same update to each server s services file horcm0 NNNN udp horcm0 service port number NNNN horcm0 12345 udp Rhorcm0 services port number 12345 e Start the CCI instance This creates horcmO conf so the instance number is 0 horemstart sh O starting HORCM inst O HORCM inst O starts successfully 18 Chapter 3 Installing the TrueCopy Agent f Set the environment variable HORCMINST as instance number 0 export HORCMINST 0 g Use the raidscan command to find the port TID and LUN of the TrueCopy device ls dev rdsk raidscan find DEVOICh BL LE UID S F PORT SERIAL DEV PRODUCOL LD dev rdsk c2t0d23s2 0 F CLl A 63039 23 OPEN 9 dev rdsk c2t0d24s2 0 F CL1 A 63039 24 OPEN 9 CM dev rdsk c2t0d3s2 O0 F CL1 A 3 63039 3 OPEN 9 dev rdsk c2t0d4s2 0 F CL1 A 63039 4 OPEN 9 h Edit the horcmO conf file to add the HORCM DEV and HORCM INST information tt KKK KK KKK KK CK kk e ke e For HORCM MON HK IKK I Ck k k Ck I k
13. Log Periodically TrueCopy Agent logs messages to VERITAS Cluster Server Log Desk For troubleshooting refer to messages in the VERITAS Cluster Server Log Desk and the details of the TrueCopy Agent log messages in section 6 3 Note For information on accessing VERITAS Cluster Server Log Desk refer to the VERITAS Cluster Server user s guide The user and or maintenance personal must check the message of TrueCopy Agent in the VERITAS Cluster Server Log Desk periodically TrueCopy Agent logs the following messages Before or After execution of TrueCopy takeover process m Start and End of each entry point except of MONITOR entry point 64 Chapter 5 Troubleshooting 5 4 Solving the Entry Point Timeout Conditions Table 5 3 shows the message ID and section for each entry point timeout condition Table5 3 X Entry Point Timeout Message IDs and Sections 13006 CLEAN entry point timeout 13007 CLOSE entry point timeout 5 4 1 CLEAN Entry Point Timeout Possible Condition Start message of CLEAN entry point ID 3000306 is logged But end message of CLEAN entry point ID 3000311 is not logged in VCS log Possible Cause TrueCopy Agent called CLEAN entry point and it started TrueCopy Agent cannot receive the response from CLEAN entry point within the CleanTimeout value set to TrueCopy type resource CleanTimeout value may be short against to the actual response time for CLEAN entry point Action Set the appropriat
14. SSWS the TrueCopy Agent determines the resource is in offline status Otherwise TrueCopy Agent determines the resource is in online status Issuing horctakeover command In the following case the TrueCopy Agent issues the horctakeover command based on the following criteria in order to make the resource accessible Working environment of TrueCopy Agent is initialized TrueCopy volume pair is P VOL in PSUE or PDUB status TrueCopy fence level is DATA synchronous mode only The reason that the TrueCopy Agent issues the horctakeover command in this case is When a TrueCopy link failure occurs in this configuration the status of the TrueCopy P VOL becomes PSUE or PDUB If the fence level of the pair is DATA read and write to the P VOL are prohibited Therefore the TrueCopy Agent makes the volume accessible by issuing the horctakeover command for the volume Chapter 4 TrueCopy Agent Operations 4 2 Log Information of TrueCopy Agent The TrueCopy Agent logs messages into the VERITAS Cluster Server Log Desk The TrueCopy Agent does not have its own log file Please refer to sections 5 3 1 and 6 3 Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 45 4 3 System Configuration 46 Figure 4 1 shows example outline of TrueCopy and VERITAS Cluster Server configuration The VERITAS Cluster Server VCS cluster management software can manage up to 32 nodes in a single cluster system
15. TrueCopy SYNC Mirroring consistency is ensured for pairs whose fence level is DATA since a write error is returned if Fence Level DATA mirror consistency with the remote secondary volume is lost The secondary volume can continue operation regardless of the status Note A primary volume write that discovers a link down situation will return an error to the host and will likely be recorded on only the primary volume side TrueCopy ASYNC TrueCopy Asynchronous uses asynchronous transfers to ensure the sequence of write data between the primary volume and secondary volume Writing to the primary volume is enabled regardless of whether the secondary volume status is updated or not Thus the mirror consistency of the secondary volume is not ensured 8 Chapter 2 Requirements for TrueCopy Agent Operations 2 3 Notice on Data Consistency in the Secondary Volumes The conditions for data consistency in the secondary volumes are see Table 2 2 TrueCopy Asynchronous TrueCopy Asynchronous update copy mode guarantees data consistency for the secondary volumes in a device group TrueCopy Synchronous If you define two or more TrueCopy Synchronous volume pairs in a TrueCopy device group data consistency in the secondary volumes may not be guaranteed in the following cases Migration switch of VCS service group Sudden application failure Split brain following network partition If a TrueCopy device group has two or more T
16. TrueCopy pair type SYNC and ASYNC in a device group TrueCopy Agent does not support an intermix of TrueCopy Synchronous and TrueCopy Asynchronous pairs in the same device group If this happens the TrueCopy resource is placed in an error state when the monitor or online entry point is executed and the TrueCopy Agent loes an error message to the VERITAS Cluster Server log An intermix of SYNC and ASYNC pairs in the same device group can occur if a volume pair of the opposite type is added to a device group by mistake e g SYNC pair added to device group of ASYNC pairs Simplex volume status in a device group TrueCopy Agent does not support TrueCopy volumes with the simplex SMPL status in a device group If this happens the TrueCopy resource is placed in an error state when the monitor or online entry point is executed and the TrueCopy Agent logs an error message to the VERITAS Cluster Server log A simplex volume in a device group can occur if an error is made when adding a volume pair to a device group e g simplex volume specified instead of P VOL by mistake Secondary volume SSUS status in a device group When a secondary volume in a device group has the SSUS suspended status the TrueCopy Agent will not issue the horctakeover command to CCI and failover will not occur until the secondary volume status changes from SSUS to PAIR or COPY PSUE PDUB TrueCopy status in a device group If the TrueCopy Agent detects a volu
17. pairs can use the resources efficiently Chapter 4 TrueCopy Agent Operations Zone A Zone B Primary zone for Service A1 An Secondary zone for Service A1 An Secondary zone for Service B1 Bn Primary zone for Service B1 Bn 1 A1 me OX ea Bi re oe T VOLs for Service An S TrueCopy LINK Be o for Service B1 S P VOLs for Service B1 sau Mdd Lightning 9900 9900V Lightning 9900 9900V S VOLs for Service An Figure 4 1 VERITAS Cluster Server System Configuration Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 47 4 4 System Operation when TrueCopy Agent is Applied 4 4 1 48 The following cases are examples of typical operations Note Failover Process is used to explain the abbreviation in each diagram 1 VCS VERITAS Cluster Server 2 Upper case notation indicates an entry point name OPEN ONLINE MONITOR OFFLINE 3 Lower case and bold notation indicates the status of a resource Online Offline Startup of TrueCopy Agent 1 VERITAS Cluster Server VCS on each node start automatically TrueCopy agent process according to the resource definition in the cluster configuration if TrueCopy type resource is defined in the configuration When the resource is enabled VCS informs the TrueCopy Agent process on all nodes defined in the configuration for the service group containing TrueC
18. the TrueCopy Agent 3 1 3 2 3 3 3 4 3 5 3 6 Preparing tor installationer e dre durar RE RECESSU Sete ES Installing and Configuring CCl and TrueCopy Fu uu 3 2 4 Configuring the CCI Instance Y hn 3 2 2 Creating the TrueCopy Volume Pair llle Installing the TrueCopy Agent for VERITAS Cluster Server 3 3 1 Installation Directory and Files ce ee ee ee eee 3 3 2 Deinstalling the TrueCopy Agent GG Y ee ees Importing the TrueCopy Resource Type Using Cluster Explorer Setting OnlineTimeout Value based on TrueCopy takeover Time 3 5 1 Setting the TrueCopy Resource Online Timeout Value Using the GUI 3 5 2 Setting the TrueCopy Resource Online Timeout Value Using the CLI Adding the TrueCopy Resource to a Service Group eee 3 6 1 Using Cluster Explorer to Add the TrueCopy Resource 3 6 2 Editing the main cf File to Add the TrueCopy Resource TrueCopy Agent Operations 4 1 4 2 4 3 4 4 ENV POIDS Loue us wen hae aan Oe Poe episodes de e RU toa aee ie ai 4 1 1 ONLINE Entry Point Behavior of TrueCopy Agent 4 1 2 MONITOR Entry Point Behavior of TrueCopy Agent Log Information of TrueCopy Agent FF Fu uu System Connigul aLIOnes d aca em te rl a ew ha do Mec ees i doo e eios oio Seco eae System Operation when TrueC
19. 32 nodes are supported by VERITAS Cluster Server version 2 0 A set of one or more node could be defined as a zone in VCS The example configuration shows a set of An nodes connected to P VOL is defined as the primary zone and a set of nodes connected to S VOL is the secondary zone If a failure occurs on a certain node the service group running on the node fails over to the other node in the same zone at first refer to the VERITAS Cluster Server documentation for details The service group can fail over to the node in the other zone if there is no node in the same zone which it can fail over to Then TrueCopy Agent calls ONLINE entry point to perform takeover function and it enables the S VOL to be accessible eventually it becomes new P VOL TrueCopy resource could be used for failover group The TrueCopy volume in S VOL state could not be an active resource therefore the nodes in the secondary zone are the standby nodes and also the resource is the standby As indicated in Figure 4 1 TrueCopy volume pairs are set up between two Lightning 9900 9900V storage systems One storage system is for each zone Zone A is the primary zone for Service A1 to An And Zone B is primary zone for Service B1 to Bn If Zone B is configured only for Service A1 to An as secondary zone Standby zone the resources in zone B are wasted So different services B1 to Bn can be configured in zone B Different services which use the different TrueCopy volume
20. CI instance on server BETA 1 Select the mirror device i e the secondary volume S VOL of the TrueCopy pair For this example dev rdsk c2t0d23s0 on server BETA is selected refer to Figure 3 2 2 Create a mount point for the device mkdir test0 3 Create the configuration definition file for the CCI instance a Use the inqraid command to find the command device In this example device c2t0d24s2 is the command device CM is appended to the product ID ls dev rdsk HORCM usr bin inqraid CLI DEVICE FILE PORT SERIAL LDEV CTG H M 12 SSID R Group PRODUCT ID GOtOd0Os2 ST34371W SUN4 2G coc LdDO S2 ST34371W SUN4 2G eOtod0s2 c2t0d3s2 s s ss 6001 OPEN 9 SUN c2t0d4s2 s s ss 6001 OPEN 9 SUN G2t0dZ3s2 23 s s ss 6001 OPEN 9 SUN c2t0d24s2 24 B B OPEN 9 CM b Make a copy of the sample configuration file horcm conf in the etc directory Copy this file as horcm0 conf cp etc horcm conf etc horcm0 conf c Edit the horcm0 conf file to add the HORCM MON and HORCM CMD information df KK KKK CK CK eK ke ek IG KKK For HORCM MON KKK Ck k Ck k k ck kc k Ck k k Ck kck ckckck ckckck kckck kk k ke e v x HORCM MON fip address Service poll 10ms timeout 10ms beta horcmO 1000 3000 it ft KKK KKK KK KK IK KK KK KK k For HORCM CMD XACkCkCk ck IK IK I k Ck I IK I KK IK ckckck kckck I KK KK HORCM CMD fdev_ name dev name dev name dev rdsk c2t0d24s2 it ft KKK KKK KK KK
21. Ck I k Ck kck kckck ckckck kckck KK KK KK HORCM MON fip address Service poll 10ms timeout 10ms beta horcmO 1000 3000 it ft KKK KKK KK KK I KK ek ke ke KK KK For HORCM CMD KKK I k Ck I k k I k Ck I IK KK I IK KK IK IK KK KK HORCM CMD fdev name dev name dev name dev rdsk c2t0d24s2 it ft KKK KKK KK K CK kk Ie ke ek For HORCM DEV KI kK I kk kck k kc k Ck kck kckck kckck ckckck kckckckck ck kc kk HORCM DEV fdev group dev name porti TargetID LU MU TestGr test00 CL1 A 0 23 0 it tt KKK KKK KK KK KK KK KK KK For HORCM INST KI kK kc k Ck Ik k k kc KK I IK I KK I IK KK IK I KK KK HORCM_INST dev_group ip_addess service TestGr beta horcmO 4 Stop and restart the CCI instance horcmshutdown sh O thorcmstartsh 0 5 Use the raidqry command to make sure that the CCI instance is running You will see the following output if the CCI instance is running raidqry 1 No Group Hostname HORCM ver Uid Serial Micro ver Cache MB beta NN NN NN NN O NNNNN NN NN NN NN NNN Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 19 3 2 2 Creating the TrueCopy Volume Pair After you have configured the CCI instance on each server you are ready to create the TrueCopy pair s This section provides instructions for using CCI to create the TrueCopy pair for the sample system configuration shown in Figure 3 2 Note This example uses the korn shell and you must log in as root user To cr
22. ERITAS Cluster Server 3 Install Configure CCI Configure TrueCopy Set up CCI and TrueCopy Import the TrueCopy resource type to your configuration Install configure TrueCopy Agent for VERITAS Cluster Server Add the TrueCopy resource to a service group Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 3 2 3 2 1 14 Installing and Configuring CCI and TrueCopy Before the TrueCopy Agent can be installed CCl and TrueCopy need to be installed and configured if not already done m Installing CCl For instructions on installing the CCI software please refer to the Hitachi Command Control Interface CCI User and Reference Guide MK 90RDO11 a Defining the command device For instructions on defining the CCl command device please refer to the Hitachi LUN Manager User s Guide for the Hitachi subsystem m Installing TrueCopy For instructions on installing the TrueCopy remote console software refer to the Hitachi TrueCopy User and Reference Guide for the subsystem After the CCI and TrueCopy software are installed and the CCl command device is defined you are ready to configure the CCI instance see section 3 2 1 and create the TrueCopy pairs see section 3 2 2 Note on Configuration Services If desired Hitachi Data Systems will install and configure CCI and or TrueCopy for you For information on CCI and TrueCopy configuration services please contact your Hitachi Da
23. HITACHI DATA SYSTEMS Hitachi Freedom Storage M Lightning 9900 V Series and Lightning 9900 Hitachi TrueCopy Agent for VERITAS Cluster Server Administration and Support Guide 2003 Hitachi Data Systems Corporation ALL RIGHTS RESERVED Notice No part of this publication may be reproduced or transmitted in any form or by any means electronic or mechanical including photocopying and recording or stored in a database or retrieval system for any purpose without the express written permission of Hitachi Data Systems Corporation Hitachi Data Systems reserves the right to make changes to this document at any time without notice and assumes no responsibility for its use Hitachi Data Systems products and services can only be ordered under the terms and conditions of Hitachi Data Systems applicable agreements All of the features described in this document may not be currently available Refer to the most recent product announcement or contact your local Hitachi Data Systems sales office for information on feature and product availability This document contains the most current information available at the time of publication When new and or revised information becomes available this entire document will be updated and distributed to all registered users Trademarks Hitachi Data Systems is a registered trademark and service mark of Hitachi Ltd and the Hitachi Data Systems design mark is a trademark and service mark of H
24. I 1 Change the attribute of the VERITAS Cluster Server configuration file to write enable 4 haconf makerw 2 Modify the OnlineTimeOut value for TrueCopy Agent hatype modify TrueCopy OnlineTimeout NN NN desired OnlineTimeOut value 3 Change the attribute of the VERITAS Cluster Server configuration file back to read only haconf dump makero Please refer to the VERITAS Cluster Server User s Guide for further information 32 Chapter 3 Installing the TrueCopy Agent 3 6 Adding the TrueCopy Resource to a Service Group VERITAS Cluster Server provides several ways to add a resource to a service group This section shows two ways to add the TrueCopy resource to a service group Please refer to the VERITAS Cluster Server user documentation for further information on adding resources Section 3 6 1 shows how to add the TrueCopy resource to a service group using the Cluster Explorer GUI Section 3 6 2 shows how to add the TrueCopy resource to a service group by editing the main cf file directly The main cf file in section 3 6 2 corresponds to the sample system configuration used in section 3 2 1 refer to Figure 3 2 3 61 Using Cluster Explorer to Add the TrueCopy Resource To add the TrueCopy resource using Cluster Explorer 1 Open the Add Resource panel for the desired service group see Figure 3 10 a oO O Q Select the Service Group tab Tf of the configuration tree Se
25. K 90RDO11 Lightning 9900 documents m Hitachi Lightning 9900 User and Reference Guide MK 90RDOs and Lightn iv Preface Preface This document provides instructions for installing and using the Hitachi TrueCopy Agent for the Lightning 9900 V Series 9900V and 9900 subsystems operating in a VERITAS Cluster Server environment Please read this document carefully to understand how to use the product and maintain a copy that is accessible from your TrueCopy Agent for reference purposes This document assumes that m the user has a background in data processing and understands direct access storage device subsystems and their basic functions m the user is familiar with the Hitachi Lightning 9900V and or 9900 array subsystems please refer to the User and Reference Guide for the subsystem m the user is familiar with the Hitachi TrueCopy feature please refer to the TrueCopy User and Reference Guide for the subsystem m the user is familiar with the Hitachi Command Control Interface CCI software please refer to the Hitachi Lightning 9900 V Series and Lightning 9900 Command Control Interface CCI User and Reference Guide MK 90RD011 and m the user is familiar with the VERITAS Cluster Server product please refer to the user documentation for the VERITAS Cluster Server product Note The term 9900V refers to the entire Lightning 9900 V Series subsystem family e g 9980V 9970V unless otherwise noted
26. al volume With file system Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 9 2 4 CCI Device Group Requirements T CCI device group name The device group name in the CCI configuration definition file horcm conf must be 31 characters or less Multiple CCI device groups in a single CCI instance Changing the configuration of TrueCopy and CCI requires the reboot of CCI instance Example Two or more device groups are defined in a single CCI instance e g erp1 and erp2 and a TrueCopy resource is created for each device group If you need to change or update device group grp1 you need to suspend the CCI instance so that the TrueCopy resource for device group erp2 can be suspended Mixing primary and secondary volumes in a device group TrueCopy Agent does not support an intermix of primary and secondary volumes TrueCopy P VOLs and S VOLs in a single device group TrueCopy Agent checks the volume types in a device group at the primary site when the monitor entry is executed If P VOL s and S VOL s are detected in a device group the TrueCopy Agent logs an error to the VERITAS Cluster Server log An intermix of P VOL s and S VOL s in the same device group can occur if a volume pair of the opposite configuration is added to a device group by mistake e g S VOL added to a group of P VOLs or if the horctakeover command is performed manually on only one device pair in a group Mixing
27. am Hse Gan miri User Manager FileMNane Filenoff File nanly IP IPM ultiMIc Mount hd ITEM OA NFS NIC Oracle Phantom Process Proxy ServiceGraupHB Share Sqinet VRTSWebApp m Volume vcscluster2 E Figure 3 4 Specifying the Import Type Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 25 Figure 3 5 Specifying the Location of HTCTypes cf 26 Chapter 3 Installing the TrueCopy Agent vescluster2 Cluster Explorer root Cluster Administrator File Edit View Tools Help DECIR d Q Oh E en eo 5e Ie 7 a 3 vescluster n Status View Truecopy Application Disk mb Status of Resources of Type Truecopy DiskReservatian Elif one FileNone FileQnortt File Dnon y IP IP LIIEiMT G M aunt ht UIT CA NFS MIC Oracle Phantom Process Proxy SGerviceGraupHB ohare Sqinet WRTSwenaApn m Volume ia Truecopy Truecopy dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg dg Figure 3 6 Successful Import of TrueCopy Agent Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 27 3 5 Setting OnlineTimeout Value based on TrueCopy takeover Time The VERITAS Cluster Server online timeout value may need to be configured depending on the actual TrueCopy failover time the default online timeout is 300 sec The failover time varies depending on multiple factors so you should set the online
28. ar log HTC log Q Restart the syslog daemon ps e grep syslogd 6988 2 0 01 syslogd F kill 688 syslogd debug 1 Change this timeout syslog 1 After the user approves this can be changed Hitachi Truecopy Agent for VERITAS Cluster Server Version All Rights reserved Copyright C 2002 Hitachi Ltd oF O N gt In line 4 stands for the VCS version number e g Version 1 1 Figure 5 1 Setup for TrueCopy Agent Log Step 4 62 Chapter 5 Troubleshooting 5 3 Flow of Troubleshooting Activities Figure 5 2 shows the flow of troubleshooting activities for TrueCopy Agent operations Startup TrueCopy Agent Check VCS Log regularly Refer to 5 3 1 Entry Point Timeout Message ID for TrueCopy type resource logged 13006 13007 13011 13012 13014 13027 Yes Refer to 5 4 g No VCS Log Message ID with Tag name TAG_B 3000001 to 300499 is logged Refer to 6 Message ID 3000400 3000499 Refer to 6 2 3 Message ID 3000100 3000279 Refer to 6 2 2 What number is the message ID with Tag name TAG B essage ID 3000001 3000099 Refer to 6 2 1 Refer to 6 3 and 6 2 1 Refer to 6 3 and 6 2 2 No Y Call Support Center Refer to 6 3 and 6 2 3 Problem solved Yes Figure 5 2 Flow of Troubleshooting Activities Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 63 5 3 1 Check VCS
29. btain these files for each cluster optVRTSvcs bin TrueCopy directory Note 1 node on which TrueCopy Agent is running EC O Log File engine_X log Note 2 Need to obtain this info for each cluster node on which TrueCopy Agent is running fw a command Note 4 Note 5 node on which TrueCopy Agent is running Eu Resse RN pion node on which TrueCopy Agent is running E eee 3 mem mme ON EE 60 Chapter 5 Troubleshooting Note 1 The file is a hidden attribute file Confirm the existence of the files by using Ls a as root Note 2 X A B or C Collect all log files when an error occurs The log message may be overwritten by a new log message if you do not get the log file at the time the problem occurred Note 3 Execute the following command and confirm the device group name and instance number s etc horcminstancenumber conf Note 4 Confirm the volume pair status using the following command pairdisplay g devicegroupname fc CLI Note 5 X is the instance number Refer to the CCI User s Guide for more details Note 6 When TrueCopy Agent is configured to log into syslog Note 7 The Hitachi Data Systems representative should refer to the SVP section of the Maintenance Manual section 2 8 DUMP LOG FD copy for instructions on obtaining the DKC dump data Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 61 5 2 1 Placing the TrueCopy Agent in Debug Mode TrueCopy Ag
30. ch is online to the host s at the primary site The S VOL is the secondary volume at the remote site which is the mirror of the P VOL TrueCopy Pair Status The TrueCopy pair statuses are m SMPL This volume is not currently assigned to a TrueCopy volume pair m COPY The initial copy operation for this pair is in progress This pair is not yet synchronized m PAIR This pair is synchronized Updates to the P VOL are duplicated on the S VOL m PSUS This pair is not synchronized the user has split the pair or deleted the pair from the RCU m PSUE This pair is not synchronized the pair has been suspended due to an error condition m PDUB This LUSE pair is not synchronized one or more individual LDEV pairs within this LUSE pair has been suspended due to some error condition Horctakeover The horctakeover CCI command checks the specified volume s or device group s attributes determines the takeover function based on the attributes executes the chosen takeover function and returns the result TrueCopy takeover functions designed for HA software operation are takeover switch swap takeover PVOL takeover and SVOL takeover Chapter 1 Overview of the TrueCopy Agent for VERITAS Cluster Server VERITAS Cluster Server VERITAS Cluster Server enables you to monitor systems and application services and to restart services on a different system when hardware or software fails The primary components of a VCS system include cl
31. ck which entry points should be issued after certain entry point is called The entry points themselves do not check sequential order of entry points issued by VCS VCS guarantees that entry points perform correctly under normal situation of the system If VCS cannot call an entry point correctly in an abnormal situation the entry point may not be able to maintain its normal activity In this case Truecopy Agent cannot guarantee normal behavior of the entry point because Truecopy Agent cannot detect abnormality or conflict of entry point s activity The TrueCopy Agent supports the entry points listed in Table 4 1 The TrueCopy Agent brings the TrueCopy volumes online and monitors the TrueCopy volume status In the event of disaster recovery or server failure VERITAS Cluster Server issues the online command and the TrueCopy Agent then notifies CCI to execute the horctakeover command Note The TrueCopy Agent does not initiate failover action of its own accord in the system The TrueCopy Agent performs action only when VERITAS Cluster Server calls one of the entry points of the TrueCopy Agent The VERITAS Cluster Server system initiates and controls failover activities Table 4 1 X Entry Points for the TrueCopy Agent TrueCopy Agent initializes the work environment of TrueCopy Agent TrueCopy Agent removes the work environment of TrueCopy Agent 4 1 1 ONLINE Entry Point Behavior of TrueCopy Agent The TrueCopy Agent issues the
32. condary node The VERITAS Cluster Server system will then detect the failure of the resource on the primary node due to the sudden TrueCopy resource failover and will take the service group offline If the Volume resource is also being used in this service group the Volume Agent will not be able to take the Volume resource offline and the Volume Agent will hang Table2 3 TrueCopy Agent Resource Type Type and Description Dimension GroupName string scalar Device group name of TrueCopy pairs device group name specified in CCI horcm conf file Note The device group name must be 31 characters or less Instance string scalar CCl instance number common instance number on each cluster node in a single cluster Note For further information on the CCI instance refer to the CCI User and Reference Guide type TrueCopy Static str ArgList GroupName Instance NameRule resource GroupName _ TrueCopy str GroupName Str Instance Figure 2 1 Resource Type Definition of the TrueCopy Agent Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 11 12 Chapter 2 Requirements for TrueCopy Agent Operations Chapter3 Installing the TrueCopy Agent 3 1 Preparing for Installation Figure 3 1 shows the flow of installation activities for TrueCopy Agent Before beginning make sure that VERITAS Cluster Server is installed on each cluster node and that manual failover is functioning Install Configure V
33. e CleanTimeout value to TrueCopy type resource Otherwise CCI commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLI and bring the resource online again if necessary Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 65 5 4 2 CLOSE Entry Point Timeout Possible Condition Start message of CLOSE entry point ID 3000303 is logged But end message of CLOSE entry point ID 3000308 is not logged in VCS log Possible Cause TrueCopy Agent called CLOSE entry point and it started TrueCopy Agent cannot receive the response from CLOSE entry point within the CloseTimeout value set to TrueCopy type resource CloseTimeout value may be short against to the actual response time for CLOSE entry point Action Set the appropriate CloseTimeout value to TrueCopy type resource Otherwise CCI commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLl 5 4 3 OFFLINE Entry Point Timeout 66 Possible Condition Start message of OFFLINE entry point ID 3000305 is logged But end message of OFFLINE entry point ID 3000310 is not logged in VCS log Possible Cause TrueCopy Agent called OFFLINE entry point and it started TrueCopy Agent cannot receive the response from OFFLINE entry point within the OfflineTimeout value set
34. e failover to the other node occurred then it must done successfully Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 71 72 Recovery Procedure After recovering the remote link of TrueCopy user has to issue the pairresync command to resynchronize the suspended pair Consistency of data on S VOL has been is maintained on S VOL after the pair is suspended since the fence level is DATA By the command the pair is resynchronized and the pair status becomes from PSUE to PAIR Note By setting the value of ToleranceLimit and RestartLimit of the resource type that has dependency to TrueCopy resource properly like Mount Resource that manage TrueCopy volumes VCS attempts to restart the resource according to the number set in RestartLimit before it gives up and fails over so you can avoid that VCS to invoke the failing over the service group to another system the just for the TrueCopy link failure Chapter 5 Troubleshooting 5 5 3 Falling Back after Failover Process is Complete Failover In the following cases the system and Lightning 9900 subsystem fail over to the secondary site as shown in Figure 5 6 and continue the services m Failure of the server at the primary site m Failure of the entire primary site m Failure disconnection of the TrueCopy link or power down of the Lightning 9900V 9900 subsystem at the primary site LAN 8 Primary Site Secondary Site Server B TRUECOPY Agent
35. e resource type also determines how a resource is monitored Continuing with the example above a file system resource tests as online if mounted and an IP address tests as online if configured Service group A service group is a set of resources working together to provide application services to clients VCS performs administrative operations on resources including starting stopping restarting and monitoring at the service group level Service group operations initiate administrative operations for all resources within the group For example when a service group is brought online all the resources within the group are brought online When a failover occurs in VCS resources never fail over individually the entire service group that the resource is a member of is the unit of failover If more than one group is defined on a server one group may fail over without affecting the other group s on the server Note If a service group is to run on a particular server all of the resources it reguires must be available to the server In addition the resources comprising a service group may have interdependencies that is some resources e g volumes must be operational before other resources e g the file system can be made operational Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 3 Agent An agent is an installed program designed to control a particular resource type VCS includes a set of predefined resourc
36. e the local volume status of TrueCopy group has SVOL SSUS status and the local volume status of the group had COPY status at the last time when MONITOR entry point was called me Re synchronize TrueCopy pair You should recover the primary site with consistent backup and then you me recreate the TrueCopy pair if the primary site totally failed Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 79 Table 6 2 Error Messages continued Message ID Message Text 3000008 TrueCopy group contained S VOL in SSUS state S VOL was in PAIR state at the last time when MONITOR entry point was called Manual intervention required CONDITION Agent does not execute takeover because the local volume status of TrueCopy group has SVOL SSUS status and the local volume status of the group had PAIR or SMPL status at the last time when MONITOR entry point was called recreate or resynchronize the TrueCopy pair if the primary site already failed 3000009 Agent detected non supported fence level Manual intervention required CONDITION Agent detected non supported fence level ACTION Re synchronize the TrueCopy volume pair You should recover the primary site first and then you should ACTION Refer to User s manual Chapter Requirement for TrueCopy Operations You should recreate the TrueCopy pair with the proper fence level 3000010 Agent detected two or more fence levels Manual intervention re
37. e types and each has a corresponding agent which is designed to control the resource Agents control resources according to information hardcoded into the agent itself or by running scripts Agents act as the intermediary between a resource and VCS The agent recognizes the resource requirements and communicates them to VCS For example for VCS to bring an Oracle resource online it does not need to understand Oracle it simply passes the online command to the Oracle agent which calls the server manager and issues the appropriate startup command Agents can be proactive they can restart a failed resource prior to declaring it as faulted A resource cannot be brought online or taken offline without an agent and the actions required to do either differ significantly from resource to resource For example bringing a disk eroup online requires importing the Disk Group but bringing an Oracle database online requires starting the database manager process and issuing the appropriate startup command VCS agents are multithreaded a single VCS agent monitors multiple resources of the same resource type on one host For example the Disk agent manages all disk resources VCS agents are located in the opt VRTSvcs bin directory For example the Disk agent and its online offline and monitor scripts are located in the directory opt VRTSvcs bin Disk Entry point A VCS agent is implemented via entry points An entry point is a user defined plug in that i
38. eate the TrueCopy pair using CCI for the example in Figure 3 2 1 From server ALPHA primary site unmount the file system for the volume which will be the TrueCopy P VOL tcd fumount testO 2 Make sure that the CCI instance is running If not start the CCI instance by executing the horcmstart sh command horcmstart sh 0 3 Set the HORCMINST environment variable to the instance number 0 in this example export HORCMINST 0 4 Display the TrueCopy status for the device group TestGr in this example pairdisplay g TestGr fc CLI Group PairVol L R Port TID LU Seq LDEV P S Status Fence P LDEV M TestGr test00 L CLl A On 9 60039 9 SMPL a dm TestGr test00 R CL1 A 0 23 63039 23 SMPL 5 Create the TrueCopy pair as shown below You need to decide which volume is the primary volume and select the fence level data or async In this example ALPHA is the primary side and the fence level is data TrueCopy Synchronous paircreate g TestGr vl f data Parameters vl local volume becomes the P VOL f data sets fence level to data 6 Display the TrueCopy pair status to confirm that the create pair operation completes successfully When the create pair operation starts the pair status changes from SMPL to COPY When the create pair operation completes the pair status changes to PAIR fbairdisplay g TestGr fc CLI Group PairVol L R Portf TID LU Seg LDEV P S Status Fence Tes
39. eed to be set as shown in Table 4 2 The Oracle resource has to wait for the start of the Mount resource so it is set up as ToleranceLimit 2 Table 4 2 ToleranceLimit and RestartLimit Values Entry Point RestartLimit ToleranceLimit Chapter 4 TrueCopy Agent Operations 4 4 7 Channel Path Failure at the Primary Site If a channel path of the node to the Lightning 9900V 9900 at the primary site fails the service group fails over to the other node at the same primary site in the case shown in Figure 4 7 The actual TrueCopy takeover process does not occur in this case The TrueCopy Agent just makes sure the pair status is P VOL after the ONLINE entry point was called Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 55 4 4 8 Switching the Service Group from Primary to Secondary Site Manually 96 The actual TrueCopy takeover process does not occur when switching the service group within the same site The TrueCopy Agent just makes sure the volume pair status is P VOL after the ONLINE entry point was called LAN Primary Site Heart Beat Secondary Site 1 User operates VCS to take the service group in the server A that is operated at the primary site offline 2 VCS calls the OFFLINE to TrueCopy Agent 3 TrueCopy resource changes to the offline status User operates VCS to bring the service group in the server B online 5 VCS calls the ONLINE entry point to TrueCopy Agent Server B
40. ent has a debug mode to allow you to obtain the detail trace log for further investigation and troubleshooting Note on restarting the syslog daemon To set the debug mode you may need to restart the syslog daemon Obtain prior approval from the user to do this If the user does not approve restarting the syslog daemon perform step 4 carefully and do not perform step 5 1 Move to the directory of TrueCopy Agent cd opt VRTSvcs bin TrueCopy 2 Confirm the TrueCopy Agent environment file HTCVersion This file is a hidden file To see the environment file log in as root or equivalent and use the command Is al 3 Open the environment file using a text editor e g vi editor 4 vi HTCVersion 4 Edit the contents of the environment file as follows see Figure 5 1 a Change the first line debug 0 to debug 1 b If the user approved to use syslog change the third line syslog 0 to syslog 1 Caution Do not input an Enter key when you change the content If you do the debug mode might not work correctly 5 Set up syslog for output Caution Perform this step only if the user approves restarting the syslog daemon a Specify a file for syslog output b Open the file etc syslog conf using a text editor and then add the following line ser inrosusert err Fear HOG HIC LOG N semi colon N tab N file name for output Note Use the tab key for the blank space Do not use spaces C Create a file touch v
41. f the TrueCopy Agent 11 Overview of TrueCopy Agent Installation Activities 13 Sample System Configuration for CCl and TrueCopy 14 Installation Directory and Files FF Fu ee teens 23 Specifying the Import Type FF ee ee eee eee R9 hn 25 Specifying the Location of HTCTypes cf nh 26 Successful Import of TrueCopy Agent ec ee ee ee ee eee 27 Status View of TrueCopy Type ResourCE uu 29 Properties View of TrueCopy Type ResourCE 30 Changing the OnlineTimeout Value for the TrueCopy Type Resource 31 Opening the Add Resource Panel uu uu 34 Entering the TrueCopy Resource Name and Resource Typ 35 Entering the CCI Instance Number Eu nnn 36 Entering the CCI Device Group Name FE uu 37 Enabling the New Resource Y ht 38 Successful Installation of the TrueCopy ResourC 39 Example of TrueCopy Resource Icon in a Complex Service Group 40 Example of Main cf File ccce 9 mn 41 42 VERITAS Cluster Server System Configuration 47 Failure of the Entire Primary Site o 49 Server Failure at the Primary Site llle 50 Failure at the Primary Site of the Lightning 9900 Subsystem 51 Heartbeat Link Fall re u a tone oS od teat tena aaa 52 Remote Copy Co
42. ide for the Hitachi disk subsystem For further information on VERITAS Cluster Server operations please refer to the VERITAS Cluster Server user documentation Hitachi TrueCopy Hitachi TrueCopy provides a storage based hardware solution for disaster recovery which enables fast and accurate system recovery TrueCopy enables you to create and maintain remote copies of all user data stored on the Hitachi Lightning 9900V and 9900 subsystems for data duplication backup and disaster recovery purposes Hitachi TrueCopy provides synchronous and asynchronous copy modes to accommodate a wide variety of user requirements and data copy movement scenarios TrueCopy operations can be performed using the 9900V 9900 TrueCopy remote console software or the Hitachi Command Control Interface CCl software on the host server Hitachi CCI The Hitachi Command Control Interface CCl software product enables you to perform Hitachi TrueCopy and Shadowlmage operations on the Hitachi Lightning 9900V and 9900 subsystems by issuing commands from the host server to the subsystem The CCI software interfaces with the system software and high availability HA software on the host as well as the Hitachi TrueCopy and Shadowlmage software on the RAID subsystem CCI provides failover and operation commands which support mutual hot standby in conjunction with industry standard failover products TrueCopy P VOLs and S VOLs The TrueCopy P VOL is the primary volume whi
43. ion on troubleshooting TrueCopy Agent operations please contact your Hitachi Data Systems representative If you need to call the Hitachi Data Systems Support Center or VERITAS Technical Support please see section 5 6 for instructions Table5 1 Troubleshooting Cases epn Sut Logging messages 5 9 1 9 4 Recovery procedure for a split brain situation Recovery procedure for all TrueCopy links failed with fence level DATA Falling back after the failover process has completed For information on troubleshooting VERITAS Cluster Server operations please refer to the user documentation for the VERITAS Cluster Server product For information on troubleshooting TrueCopy operations please refer to the TrueCopy User and Reference Guide for the 9900V or 9900 subsystem MK 92RD108 MK 91RDO051 For information on troubleshooting CCI operations please refer to the Hitachi Lightning 9900 V Series and Lightning 9900 Command Control Interface CCI User and Reference Guide MK 90RDO11 Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 59 9 2 Required Data for Troubleshooting Table 5 2 lists the data that you need to collect at each cluster node See section 5 2 1 for instructions on placing the TrueCopy agent in debug mode Table 5 2 Required Data for Troubleshooting TrueCopy Agent Version Execute the VersionDisp command from the command line interface It is started from for all files in Need to o
44. itachi Ltd Hitachi Freedom Storage Hitachi TrueCopy and Lightning 9900 are trademarks of Hitachi Data Systems Corporation Oracle is a registered trademark of Oracle Corporation Sun and Solaris are registered trademarks or trademarks of Sun Microsystems Inc VERITAS VERITAS Cluster Server and VERITAS Volume Manager are registered trademarks or trademarks of VERITAS Software Corp All other brand or product names are or may be trademarks or service marks of and are used to identify products or services of their respective owners Notice of Export Controls Export of technical data contained in this document may require an export license from the United States government and or the government of Japan Please contact the Hitachi Data Systems Legal Department for any export compliance questions Document Revision Level Revision Date Description MK 92RD143 0 January 2003 Initial Release Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide iii Source Documents for this Revision m VCS Agent Administration and Support GuideO1 00 draft z2 30 Oct 2002 Hitachi SSD document Referenced Documents Lightning 9900 V Series documents m Hitachi Lightning 9900 V Series User and Reference Guide MK 92RD100 m Hitachi Lightning 9900 V Series TrueCopy User and Reference Guide MK 92RD108 m Hitachi Lightning 9900 V Series and Lightning 9900 Command Control Interface CCI User and Reference Guide M
45. lect the desired service group Select the Resources tab for the selected service group Right click in the Resource View area and select Add Resource Set the parameters for the TrueCopy resource on the Add Resource panel a Enter the name of the resource in the Resource name field select the Resource Type drop down list and select TrueCopy see Figure 3 11 Select Edit for the Instance attribute and enter the CCI instance number see Figure 3 12 Note The CCI instance number was set in step 8d on page 16 Select Edit for the GroupName attribute and enter the CCI device group name see Figure 3 13 Note The CCI device group name was set in step 8h on page 17 Select the Enabled check box and then select OK see Figure 3 14 Note Please refer to the VERITAS Cluster Server documentation for information on the Critical setting The TrueCopy resource icon is now displayed on the Cluster Explorer see Figure 3 15 and Figure 3 16 Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 33 ss Figure 3 10 Opening the Add Resource Panel 34 Chapter 3 Installing the TrueCopy Agent r gt ate RE Eg oti Burm Carm Eie Eut View Tools Help Figure 3 11 Entering the TrueCopy Resource Name and Resource Type Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 35 Figure 3 12 Entering the CCI Instance Number 36 Chapter 3 Installing the TrueCopy
46. list and describe the messages associated with TrueCopy Agent operations Table 6 2 Error Messages continues on the next page E ACTION You should wait for TrueCopy pair volume status become PAIR status If you set Retry Limit for oe Type resource VCS will retry to bring the resource online in a certain time 3000004 The volume status does not support online state But Agent expects the status is online Manual intervention required CONDITION The volume pair status does not support online state when MONITOR entry point expects the volume status could be online ACTION You should clear the resource fault with VCS CLI or Cluster Administrator GUI And you should change the TrueCopy volume pair status into proper status with CCI command 3000005 TrueCopy group contained both P VOLs and S VOLs CONDITION PVOLs and SVOLs coexist in the local volume status of the same TrueCopy pair volume group ACTION Change the TrueCopy volume pair status into proper status with CCI command i 3000006 Warning Takeover was performed in Asynchronous mode CONDITION Agent has successfully done takeover process in Asynchronous mode ACTION You should make sure the data consistency in current P VOL that was formerly S VOL mes TrueCopy group contained S VOL in SSUS state S VOL was in COPY state at the last time when me a entry point was called Manual intervention required CONDITION Agent does not execute takeover becaus
47. m Hitachi TrueCopy feature must be installed and enabled m TrueCopy Agent The TrueCopy Agent is supplied on CD ROM The TrueCopy Agent takes up 2 MB of space and requires 800 kB of memory m Command Control Interface CCI software Please contact your Hitachi Data Systems representative for information on CCI software version requirements m Host server s Sun Solaris OS Please contact your Hitachi Data Systems representative for information on supported OS versions Note Root access to each cluster node is required m VERITAS Cluster Server software Please contact your Hitachi Data Systems representative for information on supported software versions Note Patch level required for system construction If VERITAS VxFS 3 4 is used patch 02 of VERITAS VxFS 3 4 is necessary Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 2 2 TrueCopy Pair Type Requirements Table 2 1 lists the TrueCopy pair types supported by the TrueCopy Agent The TrueCopy Agent supports all TrueCopy Asynchronous volume pairs and TrueCopy Synchronous pairs with the fence level of DATA The TrueCopy Agent does not support TrueCopy Synchronous volume pairs with the fence level of STATUS or NEVER Note For further information on the TrueCopy fence level setting please refer to the TrueCopy User and Reference Guide for the 9900V or 9900 subsystem Table2 1 TrueCopy Pair Types Supported by TrueCopy Agent
48. me with the status PSUE or PDUB when the monitor entry point is executed the TrueCopy Agent issues the horctakeover command to CCI to enable write access from the servers enabling the primary site to continue to be online The TrueCopy Agent repeats the horctakeover command until the volume status changes from PSUE PDUB to PAIR or COPY You need to remove the error condition on the volume pair and resync the pair as soon as possible 10 Chapter 2 Requirements for TrueCopy Agent Operations 2 5 VERITAS Cluster Server Setup Requirements 1 TrueCopy Agent resource name The TrueCopy Agent resource name must be 63 characters or less 2 TrueCopy Agent resource type The TrueCopy Agent provides the HTCTypes cf file that defines the TrueCopy resource type for VERITAS Cluster Server This file is automatically copied to your system during TrueCopy Agent installation Table 2 3 shows the resource type of the TrueCopy Agent and attribute Figure 2 1 shows the resource type definition of the TrueCopy Agent 3 VERITAS Volume Manager Volume Agent and TrueCopy Agent in a single service group Hitachi strongly recommends that you NOT use VERITAS Volume Manager Volume Agent and TrueCopy Agent in a single service group If TrueCopy Agent and VERITAS Volume Manager Volume Agent are used in the same service group and a heartbeat link failure occurs then if a split brain situation should occur the TrueCopy resource failover will start on the se
49. mode according to the section 5 2 1 Reproduce the problem and collect the VCS log file and syslog file 6 2 3 Message ID 3000400 3000499 Condition These messages are logged when the error is returned for the CCl command that TrueCopy Agent issued The error message format is shown Figure 6 1 Example the incorrect instance number is set TAG B Jan 11 2002 10 20 30 PM Oracle TrueCopy 3000400 Can t start CCI instance CMD raidqry l Return XXX starting HORCM inst 1 O HORCM inst 1 O has failed to start Date January 11 2002 PM LO 20 30 Resource Name Oracle TrueCopy Message ID 3000400 Message Can t start CCI instance CMD Command Name raidgry 1 Return XXX XXX is returned a command value which is used Message from CCI for the issued command Figure 6 1 CCl Command Error Action Collect the information described in Table 5 2 and send it to the Support Center For support person Investigate the problem according to the CCI User s Guide using the following information in the TrueCopy Agent error message CMD the used command name Return XXX Message from CCI for the issued command If the problem persists then TrueCopy Agent might have a problem If the problem can be reproduced turn on the debug mode according to section 5 2 1 Reproduce the problem collect the VCS log file and syslog file and send it to the Support Center 76 Chapter 6 Messages 6 3 Messages Table 6 2 Table 6 6
50. nnecting Failure with Fence Level DATA 53 Channel Path Failure at the Primary Site elles 55 Switching the Service Group from the Primary to Secondary Site Manually 56 Moving the Application from the Primary Server to Secondary Site Manually 57 Setup for TrueCopy Agent Log Step4 ee ee ee ee ees 62 Flow of Troubleshooting Activities ee eee ee ene 63 Horctakeover Command Timeout Log ONLINE Entry Point 67 Heartbeat LINK Falle prescris teed ee ht ev ae EXPE ERO QE RET 69 TrueCopy EID Ea vacui t 0 19 re RU TD RENTA ATA CERIS 71 Failover PEOCOSS Sae v peto die oi Ss RODA ee Ea NER 73 CCLCOMMand ENOL 25 3 7 ses RR GDG GA S es 78 Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide ix List of Tables Table 2 1 Table 2 2 Table 2 3 Table 3 1 Table 4 1 Table 4 2 Table 5 1 Table 5 2 Table 5 3 Table 6 1 Table 6 2 Table 6 3 Table 6 4 Table 6 5 Table 6 6 Contents TrueCopy Pair Types Supported by TrueCopy Agent 8 Guarantee of Data Consistency on the Secondary Volume in Various Cases 9 TrueCopy Agent Resource Type Fu uu 11 Installation Directory and File List for TrueCopy Agent 23 Entry Points for the TrueCopy Agent cee eee ee ee ee uu 43 ToleranceLimit and RestartLimit Values eee eee 54 Troubleshnooting Cases 2553 3 AG bd OEC bebe a ERR CRGA DEN 59 Required Da
51. ntify whether PVOL of the PSUE status is accessible or not so the system repeats step 2 to 5 6 The other resource of the primary site is WriteDisable thus the OFFLINE returns to VCS The service becomes FAULT so that VCS is setup the OFFLINE for the service group 7 The service starts up at the secondary site so that VCS calls ONLINE to TrueCopy Agent 8 TrueCopy Agent calls Takeover function for CCI and the pair volume change to the SSWS status 9 CCI performs the swap takeover 10 TrueCopy Agent returns the ONLINE status to VCS for the response of MONITOR 11 VCS starts up the application Truec opy Figure 4 6 Remote Copy Connecting Failure with Fence Level DATA Phenomenon When all TrueCopy links fail and the fence level of the TrueCopy pair is DATA I Os to the P VOL are disabled and the application at the primary site will fail Status After all TrueCopy links failed if the TrueCopy Agent detected write disable I O error earlier than the Agents for the other resources the failover process will not be executed and the loop from step 2 to 5 in the above figure will be repeated However if the Agent for another resource e g Mount Agent detects the failure earlier than the TrueCopy Agent then the failover process will be executed and the service will continue at the secondary site How to recover from the loop of Step 2 5 Repair the TrueCopy link connection so it is exactly the same as befo
52. ocedure when all TrueCopy link failed with fence level DATA configuration Figure 5 5 shows TrueCopy Link failure Primary Site Secondary Site LAN VERITAS Cluster Server Appli TrueCopy cation Agent VERITAS Cluster Server Heartbeat Link TrueCopy Agent HE TrueCopy Link Figure 5 5 TrueCopy Link Failure Phenomena When the fence level of TrueCopy pair volume is configured as DATA and all TrueCopy links failed I O will be an error at P VOL and the application will be hang up Theory 1 VERITAS Cluster Server calls MONITOR entry point TrueCopy Agent notices that P VOL is online However the pair status is PSUE so Application cannot access to the P VOL 2 TrueCopy Agent issues horctakeover and make the P VOL enabled to access from application However the P VOL remains PSUE status even after horctakeover is executed 3 Every time when VERITAS Cluster Server calls MONITOR entry point TrueCopy Agent executes horctakeover command since the Agent cannot judge whether the P VOL with PSUE status is enabled to access or not Note The system status will vary depending on whether the MONITOR entry point was called after the remote link failure occurred For example application and the other resource e g Mount might become offline or the service group may fail over to other node If the service group is offline status then user has to bring the service group online manually If th
53. opy Agent is Applied 4 4 1 Startup of TrueCopy Agent lt lt ee ce eee eee eens 4 4 2 Failure of the Entire Primary Site llle 4 4 3 Server Failure at the Primary Sit e clle 4 4 4 Failure at the Primary Site of the Lightning 9900 Subsystem 445 Heart Beat Link Failure 5 x0 SUR sean RR an E b Rid 4 4 6 Remote Copy Connection Failure with Fence Level DATA 4 4 7 Channel Path Failure at the Primary Site 4 4 8 Switching the Service Group from Primary to Secondary Site Manually 4 4 9 Moving the Application from Primary Server to Secondary Site Manually Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide Vii Chapter5 Troubleshooting 5 1 General Troubleshooting ee ee eee ht 59 5 2 Required Data for Troubleshooting uu 60 5 2 1 Placing the TrueCopy Agent in Debug Mod lt 62 5 3 Flow of Troubleshooting Activities uu 63 5 3 1 Check VCS Log Periodically Fu 64 5 4 Solving the Entry Point Timeout Conditions ee ee ee eee 65 9 4 1 CLEAN Entry Point Timeollt o lt 3 ER a n RR eR x ee 65 5 4 2 CLOSE Entry Point TimeoUul ac v eher UC RC ARE tex aras 66 5 4 3 OFFLINE Entry Point Timeout eee R n 66 5 4 4 ONLINE Entry Point Timeout cee ee ee hn 67 5 4 5 OPEN Entry Point Timeout
54. opy type resource Then Agent process receives the parameters for the resource from VCS and initiate OPEN entry point for initial setup of the resource management OPEN entry point will start the CCI instance if the CCI instance for the TrueCopy resource is not running Then MONITOR entry point is called for determining the initial state of the resource VCS directs TrueCopy Agent to call ONLINE entry point on a primary node on which the application service should be running ONLINE entry point executes takeover process VCS directs TrueCopy Agent to call MONITOR entry point to make sure the resource is Online status after ONLINE entry point was called If the TrueCopy volume status is already P VOL then TrueCopy Agent returns Online status to VCS immediately VCS directs TrueCopy Agent to call MONITOR entry point on each node at a certain periods MONITOR entry point returns the online status of the TrueCopy resource to VCS if the TrueCopy volume pair status is P VOL Also MONITOR entry point returns Offline status to VCS when TrueCopy volume pair status is S VOL Chapter4 TrueCopy Agent Operations 4 4 2 Failure of the Entire Primary Site If the primary site is down due to a natural disaster e g earthquake VCS at the secondary site detects a failure of the primary site and directs the TrueCopy Agent to call the ONLINE entry point The ONLINE entry point executes the takeover process to bring the TrueCopy resource Online see Figure 4 2
55. or HORCM CMD XACkCkCk ck IK IK I k Ck I IK I KK IK ckckck kckck I KK KK HORCM CMD fdev_ name dev name dev name dev rdsk c2t0d15s2 it ft KKK KK KKK KK IK KK KK KK KK For HORCM DEV KK k k Ck I IK I KK I IK IK IK I IK KK IK IK KK KK HORCM_DEV dev_group dev_name port Target ID LU MU it ft KK KK KK KK KK KK KK KK KK KKK For HORCM INST XACkCKkCk Ck k k Ck I k k kc k Ck kck kckck kckck IK I KK IK I KK KK HORCM INST fdev group ip addess service d Edit the etc services file to set up the service port name add the following line You can use any unigue number for horcm service port and you should also set the same port number among the servers horcm0 NNNN udp horcm0 service port number NNNN horcm0 12345 udp Rhorcm0 services port number 12345 e Start the CCI instance This creates horcmO conf so that the CCI instance number is ED Pfhorcmstart sh O starting HORCM inst O HORCM inst O starts successfully f Set the environment variable HORCMINST as instance number 0 export HORCMINST 0 g Use the raidscan command to find the port TID and LUN of the TrueCopy device fls dev rdsk raidscan find DEVICE FILE UID S F PORT SERIAL DEV BRODUCI ID dev rdsk c2t0d10s2 O0 F CL1 A 30039 0 OPEN 9 dev rdsk c2t0d15s2 0 CL1 A 30039 15 OPEN 9 CM dev rdsk c2t0d5s2 0 CL1 A 30039 OPEN 9 dev rdsk c2t0d6s2 0 CL1 A 30039 OPEN 9 dev rdsk c2t0d7s2 0 CL1 A 30039 OPEN 9 dev rdsk c2t0d8s2
56. quired CONDITION Agent detected two or more fence levels in the same TrueCopy group ACTION Refer to User s manual Chapter Requirement for TrueCopy Operations You should recreate the TrueCopy pair with the proper fence level 3000011 Agent detected SMPL status Manual intervention reguired CONDITION Agent detected SMPL status ACTION You should create the TrueCopy pair with the proper fence level And then start up TrueCopy agent Table 6 3 Internal Error Messages continues on the next page MessagelD Message Text CONDITION MONITOR entry point cannot create copyfile ACTION Call Support Center i 3000104 Can t create Status Old file i 3000103 Can t create copyfile CONDITION MONITOR entry point cannot create old statusfile ACTION Call Support Center 80 Chapter6 Messages Table 6 3 Internal Error Messages continued Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 81 82 Table 6 4 Debug Mode Messages 3000280 3000299 SS 3 48g t Chapter 6 Messages Table 6 5 Information Messages i s a 1S a a MS NE NEM O O a 7 TrueCopy pair status from the last TrueCopy pair status a Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 83 64 Table 6 6 Errors in CCI Command Execution Message ID Message Text 3000400 Can t start up CCI in
57. r t 1999999 g Oradg 2 gt amp 1 i Jan LTT 2002 1022032330 PM Oracle Truecopyt 300053045 Online Bantry Porno start horctakeover startup log Figure 5 3 Horctakeover Command Timeout Log ONLINE Entry Point Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 67 5 4 5 OPEN Entry Point Timeout Possible Condition Start message of OPEN entry point ID 3000302 is logged But end message of OPEN entry point ID 3000307 is not logged in VCS log Possible Cause TrueCopy Agent called OPEN entry point and it started TrueCopy Agent cannot receive the response from OPEN entry point within the OpenTimeout value set to TrueCopy type resource OpenTimeout value may be short against to the actual response time for OPEN entry point Action Set the appropriate OpenTimeout value to TrueCopy type resource Otherwise CCI commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLl and make the resource enable again if necessary 5 4 6 MONITOR Entry Point Timeout 68 Possible Condition No log message of Monitor Entry point is logged VERITAS Cluster Server Log in default Once set up the debug mode refer to 5 2 1 then start message of MONITOR ID 3000282 could be logged But end message of MONITOR ID 3000283 entry point could not be logged in VCS log Possible Cause TrueCopy Agent called MONITOR en
58. re and resynchronize the volume pair using CCl commands The pair status of the P VOL changes from PSUE to PAIR How to prevent the failover VCS attempts to restart the resource according to the number set in RestartLimit before it gives up and fails over with the appropriate value of ToleranceLimit The RestartLimit of the resource type such as the Mount resource is dependent on the TrueCopy resource You can avoid the situation when VCS invokes the failover of the service group to another system when all TrueCopy links fail Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 53 94 RestartLimit Affects how the agent responds to a resource fault A non zero RestartLimit causes VCS to invoke the online entry point instead of failing over the service group to another system VCS attempts to restart the resource according to the number set in RestartLimit before it gives up and fails over ToleranceLimit A non zero ToleranceLimit allows the monitor entry point to return OFFLINE several times before the ONLINE resource is declared FAULTED If the monitor entry point reports OFFLINE more times than the number set in ToleranceLimit the resource is declared FAULTED However if the resource remains online for the interval designated in Conflnterval any earlier reports of OFFLINE are not counted against ToleranceLimit Example Oracle gt Mount gt DiskGroup gt TrueCopy In the above dependence case the values n
59. rueCopy Synchronous volume pairs and write requests for the primary volumes continue during the takeover process some data may be mirrored on the secondary volumes and some data may not be mirrored Therefore data consistency in the secondary volumes cannot be guaranteed To avoid data inconsistency in the secondary volumes m Use VERITAS Volume Manager VxVM logical volumes or file system When a TrueCopy device group contains two or more TrueCopy Synchronous pairs you must use a disk group resource and VxVM logical volumes or a file system to stop I O requests to the primary volume before performing the takeover process from the primary to secondary site m Prevent Split Brain from occurring When a TrueCopy device group contains two or more TrueCopy Synchronous pairs there is no way to avoid loss of data consistency in the secondary volumes if a split brain condition occurs Note Hitachi strongly recommends that you refer to the section Network Partition and Split brain in the VCS User s Guide and read How VCS avoids Split brain Table 2 2 X Guarantee of Data Consistency on the Secondary Volume in Various Cases Split Brain Split Brain Migration Sudden following Network following Network Switch of Application Partition except Partition include TrueCopy LVM and FS Configuration Service Group Failure TrueCopy Links TrueCopy Links happenin oe Ld Synchronous No VxVM logical volume Mode No file system With VxVM logic
60. ry node due to the sudden TrueCopy resource failover and will take the service group offline Therefore if the volume resource is also being used in this service group the Volume Agent will not be able to take the Volume resource offline and the Volume Agent will hang Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 69 70 Action In such case please follow the following procedure and recover the system Note Please refer to the user documentation for VERITAS Volume Manager VERITAS Cluster Server and CCI and confirm the details of commands before using the following commands 1 Recovery HeartBeat link of VCS 2 Take the service group offline on all cluster nodes hagrp offline service group sys system Note If you do not take the service group offline on all cluster nodes and follow the procedure then same problem will be happen on the secondary site 3 Make the P VOL at primary site enabled to write with horctakeover command horctakeover g disk group t 1999999 4 Deport the disk group vxdg deport lt diskgroup gt 5 Make VCS recognize that Volume resource and Disk group resource is offline hares probe lt resource 6 Start up the service group and bring the resource online hagrp online service group sys system Chapter 5 Troubleshooting 5 5 2 Recovery Procedure for All TrueCopy Links Failed with Fence Level DATA This section explains the recovery pr
61. s called when an event occurs within the VCS agent An entry point can be a C function or a script The VCS agent framework supports the entry points listed below With the exception of VCSAgStartup and monitor all entry points are optional m VCSAgStartup m Monitor supported by TrueCopy Agent mw Online supported by TrueCopy Agent m Offline supported by TrueCopy Agent m Clean supported by TrueCopy Agent m Attr m Changed m Open supported by TrueCopy Agent m Close supported by TrueCopy Agent m Shutdown Network partition Under normal conditions when a VCS system ceases heartbeat communication with its peers due to an event such as power loss or a system crash the peers assume that the system has failed and issue a new regular membership excluding the departed system A designated system in the cluster then takes over the service groups running on the departed system ensuring that the application remains highly available However heartbeats can also fail due to network failures If all network connections between any two groups of systems fail simultaneously a network partition occurs When this happens systems on both sides of the partition can restart applications from the other side resulting in duplicate services or split brain Chapter 1 Overview of the TrueCopy Agent for VERITAS Cluster Server Split brain Split brain occurs when two independent systems configured in a cluster assume they have exclu
62. s or comments Table6 1 TrueCopy Agent Message IDs 3000100 3000279 TAG B Internal error message 3000280 3000299 TAG E For debugging 3000300 3000399 TAG E Agent internal state messages or comments 3000400 3000499 TAG B Error message related to CCI 6 2 1 Message ID 3000001 3000099 Condition These messages are logged when TrueCopy type resources are configured incorrectly or TrueCopy Agent entry point does not work properly because of the inappropriate status of TrueCopy volume pair Action Check the configuration of TrueCopy type resource If it is inappropriate correct the configuration of the TrueCopy type resource Check the TrueCopy volume pair status If it is inappropriate correct the TrueCopy volume pair status If the problem persists collect the information described in Table 5 2 and send it to the Support Center For support person If the problem can be reproduced turn on the debug mode according to section 5 2 1 Reproduce the problem and collect the VCS log file and syslog file Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 77 6 2 Message ID 3000100 3000279 Condition These messages are logged when the internal errors of TrueCopy Agent occur or there could be something wrong around the server environment Action Collect the information described in Table 5 2 and send it to the Support Center For support person If the problem can be reproduced turn on the debug
63. sive access to a given resource usually a file system or volume All failover management software uses a predefined method to determine if its peer is alive If the peer is alive the system recognizes it cannot safely take over resources Split brain occurs when the method of determining peer failure is compromised A true split brain means multiple systems are online and have accessed an exclusive resource simultaneously Note Splitting communications between cluster nodes does not constitute a split brain A split brain means cluster membership was affected in such a way that multiple systems use the same exclusive resources usually resulting in data corruption Heartbeat All systems within a cluster are connected by redundant private networks over which they communicate by heartbeats which are signals sent periodically from one system to another The heartbeat signals are used to determine the health of each node within a cluster Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide Chapter 1 Overview of the TrueCopy Agent for VERITAS Cluster Server Chapter 2 Requirements for TrueCopy Agent Operations 2 1 System Requirements The TrueCopy Agent system involves the Lightning 9900V and or 9900 subsystem s TrueCopy Agent CCI software host server s and VERITAS Cluster Server software The system requirements for TrueCopy Agent operations are a Lightning 9900 V Series 9900V and or 9900 subsyste
64. stance lt CCI command gt return code NNN message text from CCI gt CONDITION OPEN ONLINE MONITOR entry point tried to start up CCI instance But it failed to start up uli Check the number of CCI instance that is specified in TrueCopy resource attributes is correct And uui configuration file of CCI instance is correct CONDITION CLEAN entry point cannot shutdown CCI instance ACTION Check the number of CCI instance that is specified in TrueCopy resource attributes is correct i 3000403 Can t shutdown CCI instance lt CCl command return code NNN message text from CCI gt Chapter 6 Messages CCI DMP FS GAB GB HA kB KB LLT LUSE LVM MB MCU msec PDUB PSUE P VOL RCU SIM SMPL SSUS S VOL VxVM Acronyms and Abbreviations Command Control Interface Dynamic Multipathing file system group membership atomic broadcast gigabyte high availability kilobyte low latency transfer LUN Expansion logical volume manager megabyte main control unit millisecond pair duplex bind LUSE pair with one or more suspended LDEV pairs pair suspended error primary volume remote control unit service information message simplex secondary suspended secondary volume VERITAS Volume Manager Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 85 86 Acronyms and Abbreviations
65. t recent service information messages SIMs Logged on the 9900V 9900 Remote Console PC The worldwide Hitachi Data Systems Support Centers are Hitachi Data Systems North America Latin America San Diego California USA 1 800 348 4357 Hitachi Data Systems Europe Contact Hitachi Data Systems Local Support Hitachi Data Systems Asia Pacific North Ryde Australia 011 61 2 9325 3300 For technical assistance or information regarding VERITAS service packages contact VERITAS Technical Support as follows m Customers in the U S and Canada 1 800 342 0652 m Customers in the rest of the world with the exception of Japan visit the technical support website at http www support veritas com or e mail supportGveritas com Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 75 76 Chapter 5 Troubleshooting Chapter 6 Messages 6 1 Message Format The message format is DATE TIME node resource name gt lt Message ID message text gt 6 2 Message IDs The messages of TrueCopy Agent are classified as shown in Table 6 1 The message ID number of TrueCopy Agent is assigned in the range 3 000 000 to 3 000 500 VCS generates several types of messages Tags A Z but TrueCopy Agent generates the following two only m TAG B indicates failure of a cluster component unanticipated state change or termination or unsuccessful completion of a VCS action m TAG E informs the user of various state message
66. tGr test00 L CL1 A 0 9 60039 9 PVOL COPY DATA TestGr test00 R CLl A O 23 63039 23 SVOL COPY DATA Note The TrueCopy initial copy operation may take a while pairdisplay g TestGr fc CLI Group PairVol L R Port TID LU Seq LDEV P S Status Fence P LDEV TestGr test00 L CL1 A 0 9 60039 9 PVOL PAIR DATA 100 23 TestGr test00 R CLl A Or 23 63039 23 SVOL PAIR DATA 9 20 Chapter 3 Installing the TrueCopy Agent Confirm that TrueCopy manual failover is functioning using the horctakeover CCI command Use the horctakeover command on server BETA horctakeover g TestGr Use the pairdisplay command on server ALPHA to verify that the pairs are swapped pairdisplay g TestGr fc CLI Group PairVol L R Port TID LU Segq LDEV P S Status Fence P LDEV M TestGr test00 L CL1 A 0 9 60039 9 SVOL PAIR DATA 100 TestGr testOO R CL1 A 023 63039 23 PVOL PAIR DATA o 9 Use the horctakeover command on server ALPHA horctakeover g TestGr Use the pairdisplay command on server ALPHA to verify that the pairs are swapped back fbairdisplay g TestGr fc CLI Group PairVol L R Port TID LU Segq LDEV P S Status Fence P LDEV M TestGr testOO L CL1 A 0 9 60039 9 PVOL PAIR DATA 100 TestGr test00 R CL1 A 0 23 63039 23 SVOL PAIR DATA 9 em Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 21 3 3 Installing the TrueCopy Agent for VERITAS Cluster Server If the TrueCopy Agent for VCS is alread
67. ta Systems account team Configuring the CCI Instance The CCI instance number and CCI device group name must be known when configuring the TrueCopy Resource This section provides an example set of instructions for configuring the CCI instance and group name using the system configuration shown in Figure 3 2 The TrueCopy P VOL and S VOL volumes will be used to store the application data mirrored between the primary site and secondary site Note This example uses the korn shell and you must log in as root user Note Section 3 6 2 shows the main cf file which corresponds to this sample system configuration see Figure 3 17 Server ALPHA Server BETA at Primary Site at Secondary Site Hl WWW NGA NUL Command device Truecopy LINK Lightning 9900V 9900 Lightning 9900V 9900 e TrueCopy MCU e TrueCopy RCU e S N 260039 e S N 263039 Figure 3 2 Sample System Configuration for CCI and TrueCopy Chapter 3 Installing the TrueCopy Agent To configure the CCI instance on server ALPHA 1 A UU N Select the device which will be the primary volume P VOL of the TrueCopy pair For this example dev rdsk c2t0d9s0 on server ALPHA is selected refer to Figure 3 2 Create a file system on the device newfs dev rdsk c2t0d9s0 Create a mount point for the file system mkdir testO Mount the file system on the mount point mount dev dsk c2t0d9s0 testO Create the shell script Test sh in this directory
68. ta for Troubleshooting uu uu 60 Entry Point Timeout Message IDs and SectionS 65 TrueCopy Agent Message IDs FF GG hh nn 77 EFO MeSssade5 aa uo Sots Wee Ste la eared YY GAN GOD rats YR 79 80 Internal Error Messages ec ee ee hh hat 80 81 Debug Mode Messages 3000280 3000299 e 82 Information Messages clle hh 83 Errors in CCI Command Execution Fu uu 84 Chapter 1 TrueCopy Agent for VERITAS Cluster Server 1 4 TrueCopy Agent for VERITAS Cluster Server The Hitachi TrueCopy Agent for the Lightning 9900 subsystems enables you to integrate the VERITAS Cluster Server product with TrueCopy remote copy operations to build a disaster recovery system When the TrueCopy Agent is applied to VERITAS Cluster Server operations the VERITAS Cluster Server system will periodically monitor the TrueCopy resource volume as one of its system resources In the event of a system failure or disaster at the primary main site failover of a cluster node from the primary site to the secondary site can be performed in conjunction with the TrueCopy resource VERITAS Cluster Server provides the agent framework to manage vendor or user unigue resources of particular types within a high availability HA cluster environment The VERITAS Cluster Server agent framework supports entry points that enable VERITAS Cluster Server to monitor resources on a host Fig
69. tallation Guide 51 4 4 5 Heart Beat Link Failure 52 If the heartbeat communication of VCS is terminated between the primary site and the secondary site by the disconnection of all heartbeat links at the same time this might cause a split brain condition VCS at the secondary site detects the failure at the primary site and directs the TrueCopy Agent to call the ONLINE entry point The TrueCopy Agent then performs the takeover process On the other hand the volume at the primary site now becomes an S VOL due to the takeover process performed by the TrueCopy Agent at the secondary site which makes it WRITE access disable The application running on the server at the primary site fails Therefore VCS at the primary site intends to take the service group offline Primary Site Chapter 4 TrueCopy Agent Operations 4 4 6 Remote Copy Connection Failure with Fence Level DATA LAN Failover Process 1 When TrueCopy link is terminated I O to PVOL becomes an error This is set up the fence mode in the pair volume at the DATA only it can not be started up the Async 2 VCS calls MONITOR to TrueCopy Agent However if the other resource ex Mount Resource finds WriteDisable I O error go to step to 6 3 To prevent Write Disable on MONITOR process TrueCopy Agent issues horctakeover command to CCl 4 CCI performs PVOL PSUE takeover 5 After PVOL takeover is done the pair status remains the PSUE TrueCopy Agent cannot ide
70. timeout value based on test results for your system Recommendation Configure the timeout value based on the estimate that you calculated from the number of volume pairs Please use the following formula to estimate the VERITAS Cluster Server online timeout value 1 sec x number of TrueCopy pairs in device group Please measure the failover time of VERITAS Cluster Server 9900V 9900 TrueCopy disaster recovery system and set the appropriate timeout value 3 5 1 Setting the TrueCopy Resource Online Timeout Value Using the GUI To set the OnlineTimeout value of the TrueCopy resource using the GUI 1 Open the Status view of the TrueCopy resource see Figure 3 7 2 Select the Properties tab for the TrueCopy resource see Figure 3 8 and then select the Show all attributes button top right of screen 3 Set the desired value for Online Timeout attribute see Figure 3 9 28 Chapter 3 Installing the TrueCopy Agent vcscluster2 Cluster Explorer ront Cluster Administrator File Edit view Tools Help ii Y gia oo oh d AA MAL escis a DiskGroup There are no resources of this type DiskReseryation hh AA Y Elif one FileMone FileOnotr ens EE ATT i RE LE RT E T nemi Ti LC ee eR ee E ea rere aa M aunt hd Uti IN EA MFS NIC Oracle Phantom Process Proxy ServiceGroupHe Share Sqinet VRTSWebApp Volume Truecapy AA AAA Truecopy Figure 3 7 Status Vie
71. to TrueCopy type resource OfflineTimeout value may be short against to the actual response time for OFFLINE entry point Action Set the appropriate OfflineTimeout value to TrueCopy type resource Otherwise CCI commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLl Chapter 5 Troubleshooting 5 4 4 ONLINE Entry Point Timeout Possible Condition After horctakeover is issued start message of takeover process ID 3000300 from ONLINE entry point is logged But end message of takeover process ID 3000301 from ONLINE entry point is not logged in VCS log Possible Cause TrueCopy Agent issued horctakeover command but TrueCopy Agent cannot receive the response for the command from CCI within the OnlineTimeout value set to TrueCopy type resource OnlineTimeout value may be short against to the actual elapsed time for CCI takeover process Action Set the appropriate OnlineTimeout value to TrueCopy type resource Refer to 3 5 OnlineTimeout value based on takeover time Otherwise CCl commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLI and bring the resource online again VCS GUI Log T Log is not output for the entry point Jan 11 2002 10 20 30 PM Oracle TrueCopy 3000300 TrueCopy Takeover Perform horctakeove
72. try point and it started But TrueCopy Agent cannot receive the response from MONITOR entry point within the MonitorTimeout value set to TrueCopy type resource MonitorTimeout value may be short against to the actual response time for MONITOR entry point Action Set the appropriate MonitorTimeout value to TrueCopy type resource Otherwise CCI commands response time is too long Check the TrueCopy configuration and configure it appropriately Clear the fault of the resource with VCS GUI or CLI and bring the resource online again if necessary Chapter 5 Troubleshooting 5 5 System Recovery Procedures 5 5 1 Recovery Procedure for a Split Brain Situation Condition This situation will occur when the TrueCopy type resource attempts online to more than one node when all Heartbeat links failed at the same time Figure 5 4 shows Heartbeat link failure Secondary Site Primary Site VERITAS Cluster Server TrueCopy Agent TrueCopy Agent cation Heartbeat Link TrueCopy Link LI W Figure 5 4 Heartbeat Link Failure If TrueCopy Agent and VERITAS Volume Manager Volume Agent are used in the same service group and all heartbeat link fail at the same time then a split brain situation should occur And the TrueCopy resource failover will start on the secondary node The VERITAS Cluster Server system will then detect the failure of the resource on the prima
73. ure 1 1 shows the relation of the TrueCopy Agent VERITAS Cluster Server system and TrueCopy volume pairs The Command Control Interface CCl software for the Lightning 9900 subsystem provides the interface between the TrueCopy Agent and the TrueCopy volume pairs on the Lightning 9900 subsystems The basic functions of the TrueCopy Agent are m _ To check whether or not a TrueCopy volume resource is configured and return online or offline accordingly To issue the horctakeover command to TrueCopy to execute failover when TrueCopy Agent receives the online command from VERITAS Cluster Server ES i failover VERITAS VERITA Cluster Server UU Se 1M m Monitor the status of volumes Appli TrueCopy TrueCopy Agent checks the nm True opy cation TrueCopy volume pair status cation Agent and returns the status Figure 1 1 Overview of TrueCopy Agent VERITAS Cluster Server and Hitachi TrueCopy Note The use of Hitachi TrueCopy the TrueCopy Agent and all other Hitachi Data Systems products is governed by the terms of your license agreement s with Hitachi Data Systems Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 1 1 2 Important Terms and Concepts This section provides brief descriptions of the key terms and concepts for TrueCopy Agent operations For further information on Hitachi TrueCopy operations please refer to the TrueCopy User and Reference Gu
74. usters resources and resource types service groups agents and communications GAB and LLT Cluster A VCS cluster consists of multiple systems connected in various combinations to shared storage devices All systems within a cluster have the same cluster ID and are connected by redundant private networks over which they communicate by heartbeats signals sent periodically from one system to another Applications can be configured to run on specific nodes within the cluster Storage is configured to provide access to shared data for nodes hosting the application so storage connectivity determines where applications are run All nodes sharing access to storage are eligible to run an application Nodes that do not share storage cannot fail over an application that stores its data on disk Resource Resources are hardware or software entities e g disks network interface cards NICs IP addresses applications databases that are brought online taken offline or monitored by VCS Each resource is identified by a unique name Resources with similar characteristics are known collectively as a resource type for example two disk resources are both classified as type Disk The resource type determines how a resource is started and stopped For example a file system resource is started when mounted and an IP resource is started by configuring the IP address on a NIC VCS monitors a resource by testing it to determine if it is online or offline Th
75. ver Installation Guide 23 3 3 2 Deinstalling the TrueCopy Agent 24 To deinstall the TrueCopy Agent 1 2 3 5 Log in to the VERITAS Cluster Server system as root Delete the TrueCopy resource from the service group Stop the VCS process by executing the following command hastop local Caution Before executing this command confirm that all VCS Service Groups are OFFLINE Remove the TrueCopy Agent by executing the following command pkgrm HTCAgent Repeat steps 1 4 for each cluster node Note Deinstallation of the TrueCopy Agent does not deinstall CCl For information on deinstalling CCI please refer to the CCI User and Reference Guide Chapter 3 Installing the TrueCopy Agent 3 4 Importing the TrueCopy Resource Type Using Cluster Explorer To import a resource type using Cluster Explorer 1 Select Import Types from the File drop down menu see Figure 3 4 2 Inthe Import Type box specify the location of the HTCTypes cf file and import the TrueCopy Resource Type see Figure 3 5 You should have copied this file to the following location during TrueCopy Agent installation see step 5 in section 3 3 l etcoc VRISvOS COnt contig 3 When the import completes the icon for the TrueCopy resource is displayed in the list of resources see Figure 3 6 vescluster2 Cluster Explorer root Cluster Administrator File Edit View Tools Help Open Configuration c save Gonfiguratl
76. w of TrueCopy Type Resource Hitachi TrueCopy Agent for VERITAS Cluster Server Installation Guide 29 30 on vcscluster2 Cluster Explorer root Cluster Administrator owl File Edit View Tools Help in sin db amp Ob de uoo FOOT a oan T ww rroperues view IT ecopy Application H2 Show all attributes dg Disk saril Mg DIskGroup 5 Key Attributes DiskReservation dg mn Attribute Value Help Edit Mj ElitNone FileNone gt m ox Name Rule TrueGopy _ resource GroupName 12 a tp FileonOnly ArgList gt GroupName 2 Ei SH s n ip instance m qui a gt wx Log Level error Ez 2 oo i de Mj MultiNICA Monitorinterval gp 2 a Se nrs Operations ce t uc p OnOff e fe iiec AC NAN N SIE A N E E Eee les cai TE Rost e Russel E E E eod Process a dy Proxy tp serviceGroupHB s i oot dg share P a dg Sglnet ood 3 Truecopy 1 ex CE vRTSWebApp 1 Ad voume Figure 3 8 Properties View of TrueCopy Type Resource Chapter 3 Installing the TrueCopy Agent XEM e db MP amp enchaeto 2 i HEN OU pnm y dy Disk My DiskGroup y DiskReservation M ElitNone My FileNone dj Fieonof d Fileononly dy iP dy IPMultiNIC dg Mount af ES cl Rr Oracle P ud dy Proxy dg ServiceGroupHB dy share dg Sglnet M Truecopy fin ve TS wehano 4 e Truecopy Figure 3 9
77. y installed deinstall the existing TrueCopy Agent refer to section 3 3 2 before installing the new TrueCopy Agent To install the TrueCopy Agent 1 Log in to the VERITAS Cluster Server system as root 2 Place the TrueCopy Agent CD ROM in the CD ROM drive 3 If you are updating your existing TrueCopy Agent make a backup copy of the existing TrueCopy Agent resource type definition file ted PERC VRIOVGS7Cont Contig x cp HTCTVDesoct HTCTVpesscf orig 4 Execute the following commands to start installation A d cc Om cd em pkgadd d HTCAgent pkg 5 Copy the TrueCopy Agent resource type definition file to the VERITAS Cluster Server conf config directory cp etc VRTSvcs conf sample TrueCopy HTCTypes cf leto lVRISvOGS cont confrg HTCTyVDpes ct 6 If the VCS process stops during installation execute the following command to restart the VCS process hastart 7 Repeat steps 1 6 for each cluster node 22 Chapter 3 Installing the TrueCopy Agent 3 3 1 Installation Directory and Files Figure 3 3 shows the TrueCopy Agent installation directory and files Table 3 1 lists and describes the TrueCopy Agent files opt VRTSvcs bin TrueCopy open close online offline monitor clean VersionDisp TrueCopyAgent etc VRTSvcs conf config HTCtypes cf Figure 3 3 Installation Directory and Files Table3 1 Installation Directory and File List for TrueCopy Agent Hitachi TrueCopy Agent for VERITAS Cluster Ser

Chapter 1 TrueCopy Agent for VERITAS Cluster Server

Contents

Download Pdf Manuals

Related Search

Related Contents