Home

Volume Manager Troubleshooting Guide

1. V 5 1 5475 VxXVM vxconfigd found in bootdg ERROR V 5 1 5475 Volume volume for mount point usr not disk group Description The system is configured to boot with usr mounted on a volume but the volume associated with usr is not listed in the configuration of the boot disk group There are two possible causes of this error Case 1 The etc vfstab file was erroneously updated to indicate the device for the usr file system is a volume but the volume named is not in the boot disk group This should happen only as a result of direct manipulation by the administrator Case 2 The system somehow has a duplicate boot disk group one of which contains the usr file system volume and one of which does not or uses a different volume name and vxconfigd somehow chose the wrong boot disk group Since vxconfigd chooses the more recently accessed version of the boot disk group this error can happen if the system clock was updated incorrectly at some point causing the apparent access order of the two disk groups to be reversed This can also happen if some disk group was deported and assigned the same name as the boot disk group with locks given to this host Action In case 1 boot the system on a CD ROM or networking mounted root file system If the root file system is defined on a volume then start and mount the root volume If the root file system is not defined on a volume mount the root file system directly Edit the e
2. To enable syslog logging of console output specify the option x syslog to vxconfigd as shown here vxconfigd x 1 9 x syslog Messages with a priority higher than Debug are written to var adm syslog syslog 1og and all other messages are written to var vxvm vxconfigd 1log If you do not specify a debug level only Error Fatal Error Warning and Notice messages are logged Debug messages are not logged Configuring Logging in the Startup Script To enable log file or syslog logging you can edit the following portion of the lib svc method vxvm sysboot script that starts the VxVM configuration daemon vxconfigd comment out or uncomment any of the following lines to enable or disable the corresponding feature in vxconfigd opts Sopts x syslog use syslog for console messages opts Sopts x log messages to vxconfigd log opts Sopts x logfile foo bar specify an alternate log file opts Sopts x timestamp timestamp console messages To turn on debugging console output uncomment the following line The debug level can be set higher for more output The highest debug level is 9 debug 1 enable debugging console output Comment or uncomment the lines corresponding to the features that you want to be disabled or enabled at startup For example the opts Sopts x syslog string is usually uncommented so that vxconfigd uses syslog logging by default Inserting a character at the beginni
3. Displaying Volume and Plex States To display detailed information about the configuration of a volume including its state and the states of its plexes use the following form of the vxprint command vxprint g diskgroup hvt volume The following example shows a disabled volume vol which has two clean plexes vol 01 and vol 02 each with a single subdisk vxprint g mydg hvt vol Disk group mydg V NAME RVG VSET COKSTATE STATE LENGTH READPOL PREF PLEXUTY PE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WIDMODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYRLENGTH COL OFF AM NM MODE SC NAME PLEX CACHE DISKOFFSLENGTH COL OFF DEVICE MODE DC NAME PARENTVOL KOGVOL SP NAME SNAPVOL DCO v vol DISABLED ACTIVE 212880 SELECT oe fsgen pl vol 01 vol DISABLED CLEAN 212880 CONCAT z RW sd mydg11 01 vol 01 mydg11 0 212880 0 c1t0d0 ENA pl vol 02 vol DISABLED CLEAN 212880 CONCAT RW sd mydg12 01vol 02 mydg12 0 212880 0 c1t1d0 ENA See the Creating and Administering Plexes and Administering Volumes chapters in the VERITAS Volume Manager Administrator s Guide for a description of the possible plex and volume states 2 VERITAS Volume Manager Troubleshooting Guide Understanding the Plex State Cycle Understanding the Plex State Cycle Changing plex st
4. This usually happens if a controller is disabled for maintenance Action None V 5 0 111 VxVM vxdmp NOTICE V 5 0 111 disabled dmpnode dmpnode_device_number Description A DMP node has been marked disabled in the DMP database It will no longer be accessible for further IO requests This occurs when all paths controlled by a DMP node are in the disabled state and therefore inaccessible Action Check hardware or enable the appropriate controllers to enable at least one path under this DMP node V 5 0 112 VxVM vxdmp NOTICE V 5 0 112 disabled path path_device_number belonging to dmpnode dmpnode_device_number Description A path has been marked disabled in the DMP database This path is controlled by the DMP node indicated by the specified device number This may be due to a hardware failure Action Check the underlying hardware if you want to recover the desired path Chapter 5 Error Messages 47 r Understanding Messages V 5 0 144 VxVM vxio WARNING V 5 0 144 Double failure condition detected on RAID 5 volume Description I O errors have been received in more than one column of a RAID 5 volume This could be caused by e a controller failure making more than a single drive unavailable the loss of a second drive while running in degraded mode two separate disk drives failing simultaneously unlikely Action Correct the hardware failures if possible Then recover the volume using th
5. configuration database and kernel update log Disk groups usually have enough copies of this configuration information to make such import failures unlikely A more serious failure is indicated by errors such as Configuration records are inconsistent Disk group has no valid configuration copies Duplicate record in configuration Format error in configuration copy Invalid block number Invalid magic number These errors indicate that all configuration copies have become corrupt due to disk failures writing on the disk by an application or the administrator or bugs in VxVM VERITAS Volume Manager Troubleshooting Guide Understanding Messages Some correctable errors may be indicated by other error messages that appear in conjunction with the auto import failure message Look up those other errors for more information on their cause Failure of an auto import implies that the volumes in that disk group will not be available for use If there are file systems on those volumes then the system may yield further errors resulting from inability to access the volume when mounting the file system Action If the error is clearly caused by excessive disk failures then you may have to recreate the disk group configuration by using the procedures given in Restoring a Disk Group Configuration on page 37 and restore contents of any volumes from a backup There may be other error messages that appear which provide further inform
6. Volumes When this occurs the vxvol start command returns the following error message VxVM vxvol ERROR V 5 1 1236 Volume r5vol is not startable RAID 5 plex does not map entire volume length At this point the contents of the RAID 5 volume are unusable Another possible way that a RAID 5 volume can become unstartable is if the parity is stale and a subdisk becomes detached or stale This occurs because within the stripes that contain the failed subdisk the parity stripe unit is invalid because the parity is stale and the stripe unit on the bad subdisk is also invalid The figure Invalid RAID 5 Volume on page 16 illustrates a RAID 5 volume that has become invalid due to stale parity and a failed subdisk Invalid RAID 5 Volume O CFO fo disk00 00 disk01 00 disk02 00 Data Data Parity Cay A SS X Data Parit Data X eo ee e y lt gt ee ee eee y Parity Data Data y We Se EE y disk03 00 disk04 00 disk05 00 z Data Data Parity z RAID 5 Plex This example shows four stripes in the RAID 5 array All parity is stale and subdisk disk05 00 has failed This makes stripes X and Y unusable because two failures have occurred within those stripes This qualifies as two failures within a stripe and prevents the use of the volume In this case the output display from the vxvol start command is as follows VxVM vxvol ERROR V 5 1 1237 Volume r5vol is not startab
7. any failure of a disk within the volume causes its data to be lost Besides the vulnerability to failure the resynchronization process can tax the system resources and slow down system operation RAID 5 logs reduce the damage that can be caused by system failures because they maintain a copy of the data being written at the time of the failure The process of resynchronization consists of reading that data and parity from the logs and writing it to the appropriate areas of the RAID 5 volume This greatly reduces the amount of time needed for a resynchronization of data and parity It also means that the volume never becomes truly stale The data and parity for all stripes in the volume are known at all times so the failure of a single disk cannot result in the loss of the data within the volume Chapter 1 Recovery from Hardware Failure 9 Failures on RAID 5 Volumes Disk Failures An uncorrectable I O error occurs when disk failure cabling or other problems cause the data on a disk to become unavailable For a RAID 5 volume this means that a subdisk becomes unavailable The subdisk cannot be used to hold data and is considered stale and detached If the underlying disk becomes available or is replaced the subdisk is still considered stale and is not used If an attempt is made to read data contained on a stale subdisk the data is reconstructed from data on all other stripe units in the stripe This operation is called a reco
8. any unmirrored volumes on a disk when it is detached those volumes are also disabled Note Apparent disk failure may not be due to a fault in the physical disk media or the disk controller but may instead be caused by a fault in an intermediate or ancillary component such as a cable host bus adapter or power supply The hot relocation feature in VxVM automatically detects disk failures and notifies the system administrator and other nominated users of the failures by electronic mail Hot relocation also attempts to use spare disks and free disk space to restore redundancy and to preserve access to mirrored and RAID 5 volumes For more information see the Administering Hot Relocation chapter in the VERITAS Volume Manager Administrator s Guide Recovery from failures of the boot root disk requires the use of the procedures described in Recovery from Boot Disk Failure on page 19 Listing Unstartable Volumes Listing Unstartable Volumes An unstartable volume can be incorrectly configured or have other errors or conditions that prevent it from being started To display unstartable volumes use the vxinfo command This displays information about the accessibility and usability of volumes vxinfo g diskgroup volume The following example output shows one volume mkt ing as being unstartable home fsgen S tarted mkting fsgen Unstartable src fsgen S tarted rootvol root Started swapvol swap Started
9. automatically Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 527 VxVM vxconfigd NOTICE V 5 1 527 Detached subdisk subdisk in volume volume Description The specified subdisk was disabled as a result of a disk failure or as a result of the administrator removing a disk with vxdg k rmdisk A failing disk is indicated by a Detached disk message Action If hot relocation is enabled VERITAS Volume Manager objects affected by the disk failure are taken care of automatically Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take V 5 1 528 VxVM vxconfigd NOTICE V 5 1 528 Detached volume volume Description The specified volume was detached as a result of a disk failure or as a result of the administrator removing a disk with vxdg k rmdisk A failing disk is indicated by a Detached disk message Unless the disk error is transient and can be fixed with a reboot the contents of the volume should be considered lost Action Contact VERITAS Technical Support V 5 1 543 VxXVM vxconfigd ERROR V 5 1 543 Differing version of vxconfigd installed Description A vxconfigd daemon was started after stopping an earlier vxconfigd with a non matching version number This can happen for example
10. because its record identifier has changed Action Use the following command to perform the snapback vxplex g diskgroup snapback volume plex V 5 1 4551 VxVM vxconfigd ERROR V 5 1 4551 dg_move_recover can t locate disk s giving up Description Disks involved in a disk group move operation cannot be found and one of the specified disk groups cannot be imported Action Manual use of the vxdg recover command may be required to clean the disk group to be imported See Recovering from Incomplete Disk Group Moves on page 18 for more information V 5 1 5448 VxVM vxconfigd ERROR V 5 1 5448 Mount point path volume not in boot disk group Description The volume device listed in the etc vfstab file for the given mount point directory normally usr is listed as in a disk group other than the boot disk group This error should not occur if the standard VERITAS Volume Manager procedures are used for encapsulating the disk containing the usr file system Action Boot VxVM from a network or CD ROM mounted root file system Then start up VxVM using fixmountroot on a valid mirror disk of the root file system After starting VxVM mount the root file system volume and edit the etc vfstab file Change the file to use a direct partition for the file system There should be a comment in the etc vfstab file that indicates which partition to use VERITAS Volume Manager Troubleshooting Guide Understanding Messages
11. controller that is not available Description This message is returned by the vxdmpadm utility when an attempt is made to enable a controller that is not working or is not physically present Action Check hardware and see if the controller is present and whether I O can be performed through it Chapter 5 Error Messages 67 Understanding Messages V 5 1 2353 VxVM vxconfigd ERROR V 5 1 2353 Disk group group Cannot recover temp database reason Consider use of vxconfigd x cleartempdir see vxconfigd 1M Description This can happen if you kill and restart vxconfigd or if you disable and enable it with vxdct1 disable and vxdctl enable This error indicates a failure related to reading the file var vxvm tempdb group This is a temporary file used to store information that is used when recovering the state of an earlier vxconfigd The file is recreated on a reboot so this error should never survive a reboot Action If you can reboot do so If you do not want to reboot then do the following a Ensure that no vxvol vxplex or vxsd processes are running Use ps e to search for such processes and use kil 1 to kill any that you find You may have to run kil1 twice to make these processes go away Killing utilities in this way may make it difficult to make administrative changes to some volumes until the system is rebooted b Recreate the temporary database files for all imported disk groups using the following co
12. from the latest binary copy of the configuration that was saved for the disk group If any of the disk headers are reinstalled a saved copy of the disks attributes is used to recreate their private and public regions These disks are also assigned new disk IDs The VxVM objects within the disk group are then recreated using the backup configuration records for the disk group This process also has the effect of creating new configuration copies in the disk group Volumes are synchronized in the background For large volume configurations it may take some time to perform the synchronization You can use the vxtask 1 list command to monitor the progress of this operation Note Disks that are in use or whose layout has been changed are excluded from the restoration process For more information see the vxconfigrestore 1M manual page Resolving Conflicting Backups for a Disk Group In some circumstances where disks have been replaced on a system there may exist several conflicting backups for a disk group In this case you see a message similar to the following from the vxconfigrestore command VxVM vxconfigrestore ERROR V 5 1 6012 There are two backups that have the same diskgroup name with different diskgroup id 1047336696 19 xxx veritas com 1049135264 31 xxx veritas com The solution is to specify the disk group by its ID rather than by its name to perform the restoration The backup file etc vx cbr bk diskgroup dgid
13. hardware failure and how to understand and deal with VERITAS Volume Manager VxVM error messages during normal operation It includes guidelines for recovering from the failure of disks and other hardware upon which virtual software objects such as subdisks plexes and volumes are constructed in VxVM Information is also included on how to configure command and transaction logging and to back up and restore disk group configurations This guide assumes that you have a Basic knowledge of the Solaris operating system OS Basic understanding of Solaris system administration Basic understanding of storage management in VxVM Note Most VERITAS Volume Manager commands require superuser or other appropriate privileges How This Guide Is Organized This guide is organized as follows Recovery from Hardware Failure Recovery from Boot Disk Failure Logging Commands and Transactions Backing Up and Restoring Disk Group Configurations Error Messages Refer to the Release Notes for information about the other documentation that is provided with this product Conventions Conventions Convention Usage Example monospace Used for path names Read tunables from the commands output directory etc vx tunefstab file and file names functions and see the 1s 1 manual page for more parameters information monospace Indicates user input 1s pubs bold C gt dir pubs
14. if you upgrade VxVM and then run vxconfigd without first rebooting Action Reboot the system V 5 1 544 VxVM vxconfigd WARNING vV 5 1 544 Disk disk in group group flagged as shared Disk skipped Description The given disk is listed as shared but the running version of VxVM does not support shared disk groups Action This message can usually be ignored If you want to use the disk on this system use vxdiskadd to add the disk Do not do this if the disk really is shared with other systems Chapter 5 Error Messages 55 Understanding Messages V 5 1 545 VxVM vxconfigd WARNING V 5 1 545 Disk disk in group group locked by host hostid Disk skipped Description The given disk is listed as locked by the host with the VERITAS Volume Manager host ID usually the same as the system host name Action This message can usually be ignored If you want to use the disk on this system use vxdiskadd to add the disk Do not do this if the disk really is shared with other systems V 5 1 546 VxVM vxconfigd WARNING vV 5 1 546 Disk disk in group group Disk device not found Description No physical disk can be found that matches the named disk in the given disk group This is equivalent to failure of that disk Physical disks are located by matching the disk IDs in the disk group configuration records against the disk IDs stored in the VERITAS Volume Manager header on the physical disks This error messag
15. it does not contain any valid plexes This can happen for example if disk failures have caused all plexes to be unusable It can also happen as a result of actions that caused all plexes to become unusable for example forcing the dissociation of subdisks or detaching dissociation or offlining of plexes Action It is possible that this error results from a drive that failed to spin up If so rebooting may fix the problem If that does not fix the problem then the only recourse is to repair the disks involved with the plexes and restore the file system from a backup V 5 1 7909 VxVM vxconfigd NOTICE V 5 1 7909 Detached disk disk Description The named disk appears to have become unusable and was detached from its disk group Additional messages may appear to indicate other records detached as a result of the disk detach Action If hot relocation is enabled VERITAS Volume Manager objects affected by the disk failure are taken care of automatically Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 7910 VxVM vxconfigd WARNING V 5 1 7910 Detaching plex plex from volume volume Description This error only happens for volumes that are started automatically by vxconfigd at system startup that is for the root and usr file system volumes The plex is being detached as
16. italic Identifies book titles new See the User s Guide for details terms emphasized text and The variable system_name indicates the variables replaced with a system on which to enter the command name or value bold Depicts GUI objects such as Enter your password in the Password fields list boxes menu field selections etc Also depicts Press Return GUI commands blue text Indicates hypertext links See Getting Help on page vii Unix superuser prompt all cp pubs 4 0 user_book shells release_mgnt 4 0 archive C gt Windows user prompt C gt copy pubs 4 0 user_book c release_mgnt 4 0 archive 4 wri VERITAS Volume Manager Troubleshooting Guide Getting Help Getting Help For technical assistance visit http support veritas com and select phone or email support This site also provides access to resources such as TechNotes product alerts software downloads hardware compatibility lists and the VERITAS customer email notification service Use the Knowledge Base Search feature to access additional product information including current and past releases of product documentation Diagnostic tools are also available to assist in troubleshooting problems associated with the product These tools are available on disc or can be downloaded from the VERITAS FTP site See the README VRTSspt file in the support directory for details For license information software updates and sales co
17. longer valid and must be removed 4 Because v01 01 was the only plex of the volume the volume contents are irrecoverable except by restoring the volume from a backup The volume must also be removed If a backup copy of the volume exists you can restore the volume later Keep a record of the volume name and its length as you will need it for the backup procedure Remove irrecoverable volumes such as v01 using the following command vxedit r rm v01 Chapter 2 Recovery from Boot Disk Failure 25 m Recovery by Reinstallation 5 It is possible that only part of a plex is located on the failed disk If the volume has a striped plex associated with it the volume is divided between several disks For example the volume named v02 has one striped plex striped across three disks one of which is the reinstalled disk disk01 The vxprint th v02 command produces the following output V NAME RVG VSET COKSTATESTATE LENGTH READPOL PREF PLEXUTY PE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WIDMODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE v v02 DISABLEDACTIVE 30720 SELECT v02 01 fsgen pl v02 01 v02 DISABLEDNODEVICE30720 STRIPE 3 128 RW sd disk02 02v02 01 disk01 424144 10240 0 0 c1t5d2 ENA sd disk01 05v02 01 disk01 620544 10240 1 0 c1t5d3 DIS sd disk03 01v02 01 disk03 620544 10240 2 0 c1lt5d4 ENA The display shows three disks across which the plex v02 01 is
18. messages Cannot create device 76 Cannot exec bin rm to remove directory 76 Cannot exec usr bin rm to remove directory 76 Cannot find device number 46 Cannot fork to remove directory 77 cannot log commit record for Diskgroup bootdg 46 Cannot open log file 78 Detaching plex from volume 79 detaching RAID 5 46 Disk device not found 56 Disk group is disabled 58 Disk group log may be too small 60 Disk in group flagged as shared 55 Disk in group locked by host 56 Disk names group but group ID differs 57 Disk skipped 55 disks not updated with new host ID 58 Double failure condition detected on RAID 5 48 Duplicate virtual device number s 62 error 28 46 Failed to log the detach of the DRL volume 49 Failure in RAID 5 logging operation 49 Illegal vminor encountered 50 Kernel log full 50 Kernel log update failed 50 library and vxconfigd disagree on existence of client 74 log object detached from RAID 5 volume 50 Log size should be at least 60 mod_install returned errno 51 object detached from RAID 5 volume 51 object plex detached from volume 45 Overlapping mirror plex detached from volume 51 RAID 5 volume entering degraded mode operation 51 read error on mirror plex of volume 52 Received spurious close 47 Root volumes are not supported on your PROM version 53 subdisk failed in plex 53 Uncorrectable read error 45 Uncorrectable write error 45 volume is detached 48 Volume remapped 62 VERITAS Volume Manager Troub
19. number V 5 1 3141 V indicates that this is a VERITAS product error message the first numeric field 5 encodes the product in this case VxVM the second field 1 represents information about the product component and the third field 3141 is the message index The text of the error message follows the message number 44 VERITAS Volume Manager Troubleshooting Guide Understanding Messages Messages This section contains a list of messages that you may encounter during the operation of VERITAS Volume Manager However the list is not exhaustive and the second field may contain the name of different command driver or module from that shown here If you encounter a product error message record the unique message number preceding the text of the message When contacting VERITAS Technical Support either by telephone or by visiting the VERITAS Technical Support website be sure to provide the relevant message number VERITAS Technical Support will use this message number to quickly determine if there are TechNotes or other information available for you V 5 0 2 VxVM vxio WARNING V 5 0 2 object_type object_name block offset Uncorrectable read error VxVM vxio WARNING V 5 0 2 object type object name block offset Uncorrectable write error Description A read or write operation from or to the specifiedVERITAS Volume Manager object failed An error is returned to the application Action These errors may represen
20. that a RAID 5 log has failed Action To restore RAID 5 logging to a RAID 5 volume create a new log plex and attach it to the volume Chapter 5 Error Messages 49 Or Understanding Messages V 5 0 181 VxVM vxio WARNING V 5 0 181 Illegal vminor encountered Description An attempt was made to open a volume device other than the root volume device before vxconfigd loaded the volume configuration Action None under normal startup conditions this message should not occur If necessary start VxVM and re attempt the operation V 5 0 194 VXVM vxio WARNING V 5 0 194 Kernel log full volume detached Description A plex detach failed because the kernel log was full As a result the mirrored volume will become detached Action It is unlikely that this condition ever occurs The only corrective action is to reboot the system V 5 0 196 VXVM vxio WARNING V 5 0 196 Kernel log update failed volume detached Description Detaching a plex failed because the kernel log could not be flushed to disk As a result the mirrored volume became detached This may be caused by all the disks containing a kernel log going bad Action Repair or replace the failed disks so that kernel logging can once again function V 5 0 207 VxVM vxio WARNING V 5 0 207 log object object_name detached from RAID 5 volume Description Indicates that a RAID 5 log has failed Action To restore RAID 5 logging to a RAID 5 volume create a ne
21. the name of the disk group and dgid is the disk group ID If a disk group is to be recreated on another system copy these files to that system Caution Take care that you do not overwrite any files on the target system that are used by a disk group on that system VERITAS Volume Manager Troubleshooting Guide Restoring a Disk Group Configuration To back up a disk group manually use this command etc vx bin vxconfigbackup diskgroup To back up all disk groups use this version of the command etc vx bin vxconfigbackup For more information see the vxconfigbackup 1M manual page Restoring a Disk Group Configuration You can use the vxconfigrestore utility to restore or recreate a disk group from its configuration backup The restoration process has two stages precommit and commit In the precommit stage you can examine the configuration of the disk group that would be restored from the backup The actual disk group configuration is not permanently restored until you choose to commit the changes Note None of the disks or VxVM objects in the disk group may be open or in use by any application while the restoration is being performed You can choose whether or not any corrupted disk headers are to be reinstalled at this stage If any of the disks private region headers are invalid restoration may not be possible without reinstalling the headers for the affected disks The following command performs a preco
22. the wrong disk as this could lead to data corruption Arrays with mirroring capability in hardware are particularly susceptible to such data corruption Action User intervention is required in the following cases Case 1 When DMP is disabled to an array that has multiple paths then each path to the array is claimed as a unique disk If DMP is suppressed VxVM does not know which path to select as the true path You must choose which path to use Decide which path to exclude and then either edit the file etc vx vxvm exclude or if vxconfigd is running select item 1 suppress all paths through a controller from VxVM s view or item 2 suppress a path from VxVM s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view The following example shows a vxvm exclude file with paths c6t0d0s2 c6t0d1s2 and c6t0d2s2 excluded from VxVM exclude_all 0 paths c6t0d0s2 pci 1f 4000 SUNW ifp 2 ssd w50060e8003275705 0 c6t0d1ls2 pci 1f 4000 SUNW ifp 2 ssd w50060e8003275705 1 c6t0d2s2 pci 1lf 4000 SUNW ifp 2 ssd w50060e8003275705 2 controllers VERITAS Volume Manager Troubleshooting Guide Understanding Messages product pathgroups Case 2 Some arrays such as EMC and HDS provide mirroring in hardware When a LUN pair is split depending on how the process is performed this may result in two disks with the same disk ID Check with your array vendor to make sure that
23. 03 01 does not use any space on the invalid disk so it can still be used The second plex v03 02 uses space on invalid disk disk01 and has a state of NODEVICE Plex v03 02 must VERITAS Volume Manager Troubleshooting Guide Recovery by Reinstallation be removed However the volume still has one valid plex containing valid data If the volume needs to be mirrored another plex can be added later Note the name of the volume to create another plex later To remove an invalid plex use the vxplex command to dissociate and then remove the plex from the volume For example to dissociate and remove the plex v03 02 use the following command vxplex o rm dis v03 02 7 Once all the volumes have been cleaned up clean up the disk configuration as described in the section Clean up Disk Configuration on page 27 Clean up Disk Configuration Once all invalid volumes and plexes have been removed the disk configuration can be cleaned up Each disk that was removed reinstalled or replaced as determined from the output of the vxdisk list command must be removed from the configuration To remove the disk use the vxdg command To remove the failed disk disk01 use the following command vxdg rmdisk disk01 If the vxdg command returns an error message some invalid mirrors exist Repeat the processes described in Clean up Volumes on page 24 until all invalid volumes and mirrors are removed Final Volume Reconfigurat
24. 19 30 57 2003 usr sbin vxdisk list Disk_1 Each entry usually contains a client ID that identifies the command connection to the vxconfigd daemon the process ID of the command that is running a time stamp and the command line including any arguments If the client ID is 0 as in the third entry shown here this means that the command did not open a connection to vxconfigd Note The client ID is the same as that recorded for the corresponding transactions in the transactions log See Logging Transactions on page 31 and Associating Command and Transaction Logs on page 33 for more information Most command scripts are not logged but the command binaries that they call are logged Exceptions are the vxdisksetup vxinstall and vxdiskunsetup scripts which are logged If there is an error reading from the settings file command logging switches to its built in default settings This may mean for example that logging remains enabled after being disabled using vxcmdlog m off command If this happens use the vxcmdlog utility to recreate the settings file or restore the file from a backup See the vxcmdlog 1M manual page for more information about the vxcmd1log utility wr 30 VERITAS Volume Manager Troubleshooting Guide Logging Transactions Logging Transactions The vxtranslog command allows you to log VxVM transactions to a file The following table demonstrates the usage of vxtranslog Comm
25. 212 71 DETACHED 11 V 5 1 3243 71 ENABLED 12 V 5 1 3362 71 volume states V 5 1 3689 72 ACTIVE 12 V 5 1 4551 72 DEGRADED 10 V 5 1 480 53 NEEDSYNC 13 V 5 1 484 54 REPLAY 11 V 5 1 525 54 SYNC 11 13 V 5 1 526 54 volumes V 5 1 527 55 cleaning up 24 V 5 1 528 55 displaying states of 2 V 5 1 543 55 listing unstartable 2 V 5 1 544 55 RAID 5 data loss 9 V 5 1 5448 72 reconfiguring 27 V 5 1 545 56 recovering mirrors 5 V 5 1 546 56 recovering RAID 5 12 V 5 1 5475 73 restarting disabled 7 V 5 1 554 57 stale subdisks starting 17 V 5 1 557 57 vxcmdlog V 5 1 5591 74 controlling command logging 29 V 5 1 568 58 vxconfigbackup V 5 1 569 35 58 backing up disk group configuration 37 V 5 1 571 59 vxconfigd V 5 1 577 60 log file 41 V 5 1 579 60 66 vxconfigd log file 41 V 5 1 583 61 vxconfigrestore V 5 1 587 61 restoring a disk group configuration 37 V 5 1 6012 38 vxdg V 5 1 663 62 recovering from failed disk group move V 5 1 7265 74 split or join 18 V 5 1 7604 76 vxedit V 5 1 768 63 clearing a disk failing flag 7 V 5 1 7809 76 vxinfo command 2 V 5 1 7829 76 vxmend command 5 V 5 1 7834 77 vxplex command 14 V 5 1 7839 77 vxprint 4 Index 87 ww displaying volume and plex states 2 vxreattach reattaching failed disks 8 vxtranslog controlling transaction logging 31 VxVM RAID 5 recovery process 11 recovering configuration of 22 reinstalling 22 vxvol recover command 14 vxvol resync command 13 vxvol start command 5 WARNING messages 44 warning
26. 29 and Associating Command and Transaction Logs on page 33 for more information If there is an error reading from the settings file transaction logging switches to its built in default settings This may mean for example that logging remains enabled after being disabled using vxtranslog m off command If this happens use the vxtranslog utility to recreate the settings file or restore the file from a backup 32 VERITAS Volume Manager Troubleshooting Guide Associating Command and Transaction Logs Associating Command and Transaction Logs The Client and process IDs that are recorded for every request and command assist you in correlating entries in the command and transaction logs To find out which command issued a particular request in transaction log use a command such as the following to search for the process ID and the client ID in the command log egrep n PID cmdlog egrep Clid In this example the following request was recorded in the transaction log Wed Feb 12 21 19 36 2003 Clid 8309 PID 2778 Part 0 Status 0 Abort Reason 0 DG_IMPORT foodg DG_IMPORT foodg DISCONNECT lt no request data gt To locate the utility that issued this request the command would be egrep n 2778 cmdlog egrep 8309 7310 8309 2778 Wed Feb 12 21 19 36 2003 The output from the example shows a match at line 7310 in the command log Examining lines 7310 and 7311 in the command log indicates that t
27. 5 volume that is in degraded mode 10 VERITAS Volume Manager Troubleshooting Guide Failures on RAID 5 Volumes A disk containing a RAID 5 log plex can also fail The failure of a single RAID 5 log plex has no direct effect on the operation of a volume provided that the RAID 5 log is mirrored However loss of all RAID 5 log plexes in a volume makes it vulnerable to a complete failure In the output of the vxprint ht command failure within a RAID 5 log plex is indicated by the plex state being shown as BADLOG rather than LOG This is shown in the following display where the RAID 5 log plex r5vo1 02 has failed V NAME RVG VSET COKSTATESTATE LENGTH READPOL PREFPLEX UTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WID MODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYRLENGTH COL OFF AM NM MODE vV r5vol ENABLED ACTIVE 204800 RAID raid5 pl r5vol 01 r5vol ENABLED ACTIVE 204800 RAID 3 16 RW sd disk01 01 r5vol 01disk01 0 102400 0 0 c2t9d0 ENA sd disk02 01 r5vol 01ldisk02 0 102400 1 0 c2t10d0 ENA sd disk03 01 r5vol 01disk03 0 102400 2 0 c2t11d0 ENA pl r5vol 02 r5vol DISABLEDBADLOG 1440 CONCAT RW sd disk04 01 r5vol 02disk04 0 1440 0 c2t12d0 ENA pl r5vol 03 r5vol ENABLED LOG 1440 CONCAT RW sd disk05 01 r5vol 12disk05 0 1440 0 c2t14d0 ENA Default Startup Recovery Process for RAID 5 VxVM may need to perform sever
28. VERITAS VERITAS Volume Manager 4 1 Troubleshooting Guide Solaris x64 Platform Edition N13107F November 2005 Disclaimer The information contained in this publication is subject to change without notice VERITAS Software Corporation makes no warranty of any kind with regard to this manual including but not limited to the implied warranties of merchantability and fitness for a particular purpose VERITAS Software Corporation shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing performance or use of this manual VERITAS Legal Notice Copyright 2005 VERITAS Software Corporation All rights reserved VERITAS and the VERITAS Logo are trademarks or registered trademarks of VERITAS Software Corporation or its affiliates in the U S and other countries Other names may be trademarks of their respective owners VERITAS Software Corporation 350 Ellis Street Mountain View CA 94043 USA Phone 650 527 8000 Fax 650 527 2908 www veritas com Third Party Legal Notices Data Encryption Standard DES Copyright Copyright 1990 Dennis Ferguson All rights reserved Commercial use is permitted only if products that are derived from or include this software are made available for purchase and or use in Canada Otherwise redistribution and use in source and binary forms are permitted Copyright 1985 1986 1987 1988 1990 by the Massachusetts Institute
29. WID MODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYRLENGTH COL OFF AM NM MODE vV r5vol ENABLED NEEDSYNC204800 RAID raid5 pl r5vol 01 r5vol ENABLED ACTIVE 204800 RAID 3 16 RW sd disk01 01 r5vol Oldisk01 0 102400 0 0 c2t9d0 ENA sd disk02 01 r5vol Oldisk02 0 102400 1 0 c2t10d0 ds sd disk03 01 r5vol Oldisk03 0 102400 2 0 c2t11d0 ENA This output lists the volume state as NEEDSYNC indicating that the parity needs to be resynchronized The state could also have been SYNC indicating that a synchronization was attempted at start time and that a synchronization process should be doing the synchronization If no such process exists or if the volume is in the NEEDSYNC state a synchronization can be manually started by using the resync keyword for the vxvol command For example to resynchronize the RAID 5 volume in the figure Invalid RAID 5 Volume on page 16 use the following command vxvol g mydg resync r5vol Chapter 1 Recovery from Hardware Failure 13 r Failures on RAID 5 Volumes Parity is regenerated by issuing VOL_R5_RESYNC ioct1s to the RAID 5 volume The resynchronization process starts at the beginning of the RAID 5 volume and resynchronizes a region equal to the number of sectors specified by the o josize option If the o josize option is not specified the default maximum I O size is used The resync operation then moves onto the next region until the entire length of t
30. a result of I O failure disk failure during startup or prior to the last system shutdown or crash or disk removal prior to the last system shutdown or crash Action To ensure that the root or usr file system retains the same number of active mirrors remove the given plex and add a new mirror using the vxassist mirror operation Also consider replacing any bad disks before running this command V 5 1 7916 VxVM vxconfigd ERROR V 5 1 7916 DG move can t import diskgroup giving up Description The specified disk group cannot be imported during a disk group move operation The disk group ID is obtained from the disk group that could be imported Action The disk group may have been moved to another host One option is to locate it and use the vxdg recover command on both the source and target disk groups Specify the o clean option with one disk group and the o remove option with the other disk group See Recovering from Incomplete Disk Group Moves on page 18 for more information V 5 1 7934 VxVM vxconfigd ERROR V 5 1 7934 Disk group group Disabled by errors Description This message indicates that some error condition has made it impossible for VxVM to continue to manage changes to a disk group The major reason for this is that too many disks have failed making it impossible for vxconfigd to continue to update configuration copies There should be a preceding error message that indicates the specific error
31. a volume it may be impossible to recover the volume using vxmend In this case remove the volume recreate it on hardware that is functioning correctly and restore the contents of the volume from a backup or from a snapshot image Chapter 1 Recovery from Hardware Failure 5 Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State A plex is shown in the RECOVER state if its contents are out of date with respect to the volume This can happen if a disk containing one or more of the plex s subdisks has been replaced or reattached If a plex is shown as being in this state it can be recovered as follows 1 Use the following command to force the plex into the OFFLINE state vxmend g diskgroup o force off plex Place the plex into the STALE state using this command vxmend g diskgroup on plex If there are other ACTIVE or CLEAN plexes in the volume use the following command to reattach the plex to the volume vxplex g diskgroup att plex volume If the volume is already enabled resynchronization of the plex is started immediately If there are no other clean plexes in the volume use this command to make the plex DISABLED and CLEAN vxmend g diskgroup fix clean plex If the volume is not already enabled use the following command to start it and preform any resynchronization of the plexes in the backgrou
32. al operations to restore fully the contents of a RAID 5 volume and make it usable Whenever a volume is started any RAID 5 log plexes are zeroed before the volume is started This prevents random data from being interpreted as a log entry and corrupting the volume contents Also some subdisks may need to be recovered or the parity may need to be resynchronized if RAID 5 logs have failed VxVM takes the following steps when a RAID 5 volume is started 1 Ifthe RAID 5 volume was not cleanly shut down it is checked for valid RAID 5 log plexes Ifvalid log plexes exist they are replayed This is done by placing the volume in the DETACHED volume kernel state and setting the volume state to REPLAY and enabling the RAID 5 log plexes If the logs can be successfully read and the replay is successful go to step 2 Ifno valid logs exist the parity must be resynchronized Resynchronization is done by placing the volume in the DETACHED volume kernel state and setting the volume state to SYNC Any log plexes are left in the DISABLED plex kernel state Chapter 1 Recovery from Hardware Failure 1 Failures on RAID 5 Volumes The volume is not made available while the parity is resynchronized because any subdisk failures during this period makes the volume unusable This can be overridden by using the o unsafe start option with the vxvol command If any stale subdisks exist the RAID 5 volume is unusable Caut
33. and Description vxtranslog 1 List current settings for transaction logging vxtranslog m on Turn on transaction logging vxtranslog s 512k Set the maximum transaction log file size to 512KB vxtranslog n 10 Set the maximum number of historic transaction log files to 10 vxtranslog n no_limit Remove any limit on the number of historic transaction log files vxtranslog q on Turn on query logging vxtranslog q off Turn off query logging vxtranslog m off Turn off transaction logging Transactions are logged to the file translog in the directory etc vx 1log This path name is a symbolic link to a directory whose location depends on the operating system If required you can redefine the directory which is linked If you want to preserve the settings of the vxtranslog utility you must also copy the settings file translog to the new directory Note The translog file is a binary and should not be edited The size of the transaction log is checked after an entry has been written so the actual size may be slightly larger than that specified When the log reaches a maximum size the current transaction log file translog is renamed as the next available historic log file translog number where number is an integer from 1 up to the maximum number of historic log files that is currently defined and a new current log file is created A limited number of historic log files is preserved to avoid filling up the file syst
34. and the target driver claims the boot disk the message may be ignored if the device path corresponds to the boot disk V 5 0 64 VxVM vxio WARNING V 5 0 64 cannot log commit record for Diskgroup bootdg error 28 Description Usually means that multipathing is misconfigured Action See V 5 1 7265 on page 74 for possible causes and solutions V 5 0 106 VXVM vxio WARNING V 5 0 106 detaching RAID 5 volume Description Either a double failure condition in the RAID 5 volume has been detected in the kernel or some other fatal error is preventing further use of the array Action If two or more disks have been lost due to a controller or power failure use the vxrecover utility to recover them once they have been re attached to the system Check for other console error messages that may provide additional information about the failure VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 0 108 VxVM vxio WARNING V 5 0 108 Device major minor Received spurious close Description A close was received for an object that was not open This can only happen if the operating system is not correctly tracking opens and closes Action No action is necessary the system will continue V 5 0 110 VxVM vxdmp NOTICE V 5 0 110 disabled controller controller_name connected to disk array disk_array_serial_number Description All paths through the controller connected to the disk array are disabled
35. ara at tan ra aah a alt fa awa ara tat Drea a 8 Sau ae ab Pa aida v How This Guide Is Organized unaua eee ee v Co onye tons os cece dees pened ees a a tag aa baad a esas Ea NE as vi Gein SLE Dette ASIN Ad SIN R I Ese el Se E e e BR Ges eae vii Documentation Feedback 00 ccc ccc tenet ene e nee nee vii Chapter 1 Recovery from Hardware Failure 00 cee eee e eee eee 1 Listing Unstartable Volumes 66 66 EE EEEE nee een ees 2 Displaying Volume and Plex States 0 eee eee 2 Understanding the Plex State Cycle 0 2 6 cen eens 3 Recovering an Unstartable Mirrored Volume 0 cece eee eens 5 Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State 6 Forcibly Restarting a Disabled Volume 000s 7 Clearing the Failing Flag fora Disk 0 0 00000 e eee 7 Reattaching Failed Disks tsiti nrinig e a eee eens 8 Failures on RAID 5 Volumes 0 cee cece cece ence ence ence nnes 9 SyStem Pail ares iia gcd irion Saat bce aE et a Rede angel Beebe EE RE 9 Disk Faures seas iit Sik oe ie SEY Re Gee bie CHS le PAE ee WH eS 10 Default Startup Recovery Process for RAID 5 1 eee 11 Recovering a RAID 5 Volume 6 eee eee 12 Recovery After Moving RAID 5 Subdisks 0 00 cence ees 15 Starting RATD 5 Volimes vaerei ein esra i i eae e ia eens a a g 15 Recovering from Incomplete Disk Group Moves 0 00 e cece eee eee 18
36. ate 10 detached RAID 5 log plexes 14 DETACHED volume kernel state 11 DISABLED plex kernel state 3 11 disk group errors new host ID 57 disk groups backing up configuration of 35 36 configuration backup files 36 recovering from failed move split or join 18 resolving conflicting backups 38 restoring configuration of 35 37 disk IDs fixing duplicate 74 disks causes of failure 1 cleaning up configuration 27 failing flag 7 failures 10 fixing duplicated IDs 74 reattaching 8 reattaching failed 8 DMP fixing duplicated disk IDs 74 EMPTY plex state 3 ENABLED plex kernel state 3 ENABLED volume kernel state 12 error message enable failed 69 ERROR messages 44 error messages A virtual disk device is open 53 All transactions are disabled 79 Attempt to disable controller failed 71 Attempt to enable a controller that is not available 67 83 wy can t import diskgroup 79 Can t locate disk s 72 Cannot auto import group 35 58 Cannot kill existing daemon 67 cannot open dev vx config 77 Cannot recover temp database 68 Cannot remove last disk group configuration copy 70 Cannot reset VxVM kernel 53 Cannot start volume 54 78 Configuration records are inconsistent 58 Configuration too large for configuration copies 70 default log file 41 Device is already open 77 Differing version of vxconfigd installed 55 Disabled by errors 36 79 Disk for disk group not found 62 Disk group does not exist 18 70 Disk group errors mult
37. ates are part of normal operations and do not necessarily indicate abnormalities that must be corrected A clear understanding of the various plex states and their interrelationship is necessary if you want to be able to perform the recovery procedures described in this chapter The figure Main Plex State Cycle shows the main transitions that take place between plex states in VxVM For more information about plex states see the chapter Creating and Administering Plexes in the VERITAS Volume Manager Administrator s Guide Main Plex State Cycle Start up vxvol start PS ACTIVE PKS ENABLED PS CLEAN PKS DISABLED Shut down vxvol stop PS Plex State PKS Plex Kernel State At system startup volumes are started automatically and the vxvol start task makes all CLEAN plexes ACTIVE At shutdown the vxvol stop task marks all ACTIVE plexes CLEAN If all plexes are initially CLEAN at startup this indicates that a controlled shutdown occurred and optimizes the time taken to start up the volumes The figure Additional Plex State Transitions on page 4 shows additional transitions that are possible between plex states as a result of hardware problems abnormal system shutdown and intervention by the system administrator When first created a plex has state EMPTY until the volume to which it is attached is initialized Its state is then set to CLEAN Its plex kernel state remains set to DISABLED an
38. ation See those other error messages for more information on how to proceed If those errors do not make it clear how to proceed contact VERITAS Technical Support V 5 1 571 VxVM vxconfigd ERROR V 5 1 571 Disk group group Disk disk Skip disk group with duplicate name Description Two disk groups with the same name are tagged for auto importing by the same host Disk groups are identified both by a simple name and by a long unique identifier disk group ID assigned when the disk group is created Thus this error indicates that two disks indicate the same disk group name but a different disk group ID VxVM does not allow you to create a disk group or import a disk group from another machine if that would cause a collision with a disk group that is already imported Therefore this error is unlikely to occur under normal use However this error can occur in the following two cases A disk group cannot be auto imported due to some temporary failure If you create a new disk group with the same name as the failed disk group and reboot the new disk group is imported first The auto import of the older disk group fails because more recently modified disk groups have precedence over older disk groups A disk group is deported from one host using the h option to cause the disk group to be auto imported on reboot from another host If the second host was already auto importing a disk group with the same name then reboot of tha
39. case of a failing RAID 5 log plex relocation occurs only if the log plex is mirrored the vxrelocd daemon then initiates a mirror resynchronization to recreate the RAID 5 log plex If hot relocation is disabled at the time of a failure the system administrator may need to initiate a resynchronization or recovery VERITAS Volume Manager Troubleshooting Guide Failures on RAID 5 Volumes Note Following severe hardware failure of several disks or other related subsystems underlying a RAID 5 plex it may be impossible to recover the volume using the methods described in this chapter In this case remove the volume recreate it on hardware that is functioning correctly and restore the contents of the volume from a backup Parity Resynchronization In most cases a RAID 5 array does not have stale parity Stale parity only occurs after all RAID 5 log plexes for the RAID 5 volume have failed and then only if there is a system failure Even if a RAID 5 volume has stale parity it is usually repaired as part of the volume start process If a volume without valid RAID 5 logs is started and the process is killed before the volume is resynchronized the result is an active volume with stale parity For an example of the output of the vxprint ht command see the following example for a stale RAID 5 volume V NAME RVG VSET COKSTATESTATE LENGTH READPOL PREFPLEX UTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL
40. d is not set to ENABLED until the volume is started Chapter 1 Recovery from Hardware Failure 3 Understanding the Plex State Cycle Additional Plex State Transitions Create plex PS EMPTY PKS DISABLED PS ACTIVE PKS DISABLED After crash and reboot vxvol start Recover data vxvol resync Initialize plex vxvol init clean Start up vxvol start Take plex offline vxmend off PS CLEAN PKS DISABLED PS ACTIVE PKS ENABLED PS OFFLINE PKS DISABLED esync data vxplex att Shut down vxvol stop Put plex online vxmend on Uncorrectable I O failure PS IOFAIL PKS DETACHED PS STALE PS Plex State PKS DETACHED PKS Plex Kernel State After a system crash and reboot all plexes of a volume are ACTIVE but marked with plex kernel state DISABLED until their data is recovered by the vxvol resync task A plex may be taken offline with the vxmend of f command made available again using vxmend on and its data resynchronized with the other plexes when it is reattached using vxplex att A failed resynchronization or uncorrectable I O failure places the plex in the IOFAIL state Recovering an Unstartable Mirrored Volume on page 5 and subsequent sections describe the actions that you can take if a system crash or I O error leaves no plexes of a mirrored volume in a CLEAN or ACTIVE state For information on the r
41. dgid dginfo contains a timestamp that records when the backup was taken ww 38 VERITAS Volume Manager Troubleshooting Guide Restoring a Disk Group Configuration The following is a sample extract from such a backup file that shows the timestamp and disk group ID information TIMESTAMP Tue Apr 15 23 27 01 PDT 2003 DISK_GROUP_CONFIGURATION Group mydg dgid 1047336696 19 xxx veritas com Use the timestamp information to decide which backup contains the relevant information and use the vxconfigrestore command to restore the configuration by specifying the disk group ID instead of the disk group name Chapter 4 Backing Up and Restoring Disk Group Configurations 39 r Restoring a Disk Group Configuration 40 VERITAS Volume Manager Troubleshooting Guide Error Messages 5 This chapter provides information on error messages associated with the VERITAS Volume Manager VxVM configuration daemon vxconfigd the kernel and other utilities It covers most informational failure and error messages displayed on the console by vxconfigd and by the VERITAS Volume Manager kernel driver vxio These include some errors that are infrequently encountered and difficult to troubleshoot Note Some error messages described here may not apply to your system Clarifications are included to elaborate on the situation or problem that generated a particular message Wherever possible a recovery procedure Action is pr
42. e vxrecover command V 5 0 145 VXVM vxio WARNING vV 5 0 145 DRL volume volume is detached Description A Dirty Region Logging volume became detached because a DRL log entry could not be written If this is due to a media failure other errors may have been logged to the console Action The volume containing the DRL log continues in operation If the system fails before the DRL has been repaired a full recovery of the volume s contents may be necessary and will be performed automatically when the system is restarted To recover from this error use the vxassist addlog command to add a new DRL log to the volume V 5 0 146 VxVM vxdmp NOTICE V 5 0 146 enabled controller controller_name connected to disk array disk_array_serial_number Description All paths through the controller connected to the disk array are enabled This usually happens if a controller is enabled after maintenance Action None V 5 0 147 VxVM vxdmp NOTICE V 5 0 147 enabled dmpnode dmpnode_device_number Description A DMP node has been marked enabled in the DMP database This happens when at least one path controlled by the DMP node has been enabled Action None VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 0 148 VxVM vxdmp NOTICE V 5 0 148 enabled path path_device_number belonging to dmpnode dmpnode_device_number Description A path has been marked enabled in the DMP database This path i
43. e for use during the subdisk recovery Otherwise the volume kernel state is set to DETACHED and it is not available during subdisk recovery This is done because if the system were to crash or the volume was ungracefully stopped while it was active the parity becomes stale making the volume unusable If this is undesirable the volume can be started with the o unsafe start option Caution The o unsafe start option is considered dangerous as it can make the contents of the volume unusable It is therefore not recommended The volume state is set to RECOVER and stale subdisks are restored As the data on each subdisk becomes valid the subdisk is marked as no longer stale If any subdisk recovery fails and there are no valid logs the volume start is aborted because the subdisk remains stale and a system crash makes the RAID 5 volume unusable This can also be overridden by using the o unsafe start option Caution The o unsafe start option is considered dangerous as it can make the contents of the volume unusable It is therefore not recommended If the volume has valid logs subdisk recovery failures are noted but they do not stop the start procedure When all subdisks have been recovered the volume is placed in the ENABLED kernel state and marked as ACTIVE It is now started Chapter 1 Recovery from Hardware Failure 17 Recovering from Incomplete Disk Group Moves Recove
44. e is displayed for any disk IDs in the configuration that are not located in the disk header of any physical disk This may result from a transient failure such as a poorly attached cable or from a disk that fails to spin up fast enough Alternately this may happen as a result of a disk being physically removed from the system or from a disk that has become unusable due to a head crash or electronics failure Any RAID 5 plexes DRL log plexes RAID 5 subdisks or mirrored plexes containing subdisks on this disk are unusable Such disk failures particularly on multiple disks may cause one or more volumes to become unusable Action If hot relocation is enabled VERITAS Volume Manager objects affected by the disk failure are taken care of automatically Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 554 VxVM vxconfigd WARNING V 5 1 554 Disk disk names group group but group ID differs Description As part of a disk group import a disk was discovered that had a mismatched disk group name and disk group ID This disk is not imported This can only happen if two disk groups have the same name but have different disk group ID values In such a case one group is imported along with all its disks and the other group is not This message appears for disks in the un selected group Acti
45. e volume unusable Action If hot relocation is enabled VERITAS Volume Manager objects affected by the disk failure are taken care of automatically Mail is sent to root indicating what actions were taken by VxVM and what further actions the administrator should take V 5 1 1589 VxVM vxconfigd ERROR V 5 1 1589 enable failed aborting Description Regular startup of vxconfigd failed This error can also result from the command vxdct1 enable Action The failure was fatal and vxconfigd was forced to exit The most likely cause is that the operating system is unable to create interprocess communication channels to other utilities Chapter 5 Error Messages 65 v Understanding Messages VxVM vxconfigd ERROR V 5 1 1589 enable failed Error check group configuration copies Database file not found Description Regular startup of vxconfigd failed This error can also result from the command vxdct1 enable The directory var vxvm tempdb is inaccessible This may be because of root file system corruption if the root file system is full or if var is a separate file system because it has become corrupted or has not been mounted Action If the root file system is full increase its size or remove files to make space for the tempdb file If var is a separate file system make sure that it has an entry in etc vfstab Otherwise look for I O error messages during the boot process that indicate either a hardware proble
46. ecovery of RAID 5 volumes see Failures on RAID 5 Volumes on page 9 and subsequent sections wr 4 VERITAS Volume Manager Troubleshooting Guide Recovering an Unstartable Mirrored Volume Recovering an Unstartable Mirrored Volume A system crash or an I O error can corrupt one or more plexes of a mirrored volume and leave no plex CLEAN or ACTIVE You can mark one of the plexes CLEAN and instruct the system to use that plex as the source for reviving the others as follows 1 Place the desired plex in the CLEAN state using the following command vxmend g diskgroup fix clean plex For example to place the plex vol01 02 in the CLEAN state vxmend g mydg fix clean vol01 02 2 To recover the other plexes in a volume from the CLEAN plex the volume must be disabled and the other plexes must be STALE If necessary make any other CLEAN or ACTIVE plexes STALE by running the following command on each of these plexes in turn vxmend g diskgroup fix stale plex 3 To enable the CLEAN plex and to recover the STALE plexes from it use the following command vxvol g diskgroup start volume For example to recover volume vol 01 vxvol g mydg start vol01 For more information about the vxmend and vxvol command see the vxmend 1M and vxvol 1M manual pages Note Following severe hardware failure of several disks or other related subsystems underlying all the mirrored plexes of
47. ed all plexes to become unusable for example forcing the dissociation of subdisks or detaching dissociation or offlining of plexes Action It is possible that this error results from a drive that failed to spin up If so rebooting may fix the problem If that does not fix the problem then the only recourse is to repair the disks involved with the plexes and restore the file system from a backup Restoring the root or usr file system requires that you have a valid backup V 5 1 525 VxVM vxconfigd NOTICE V 5 1 525 Detached log for volume volume Description The DRL or RAID 5 log for the named volume was detached as a result of a disk failure or as a result of the administrator removing a disk with vxdg k rmdisk A failing disk is indicated by a Detached disk message Action If the log is mirrored hot relocation tries to relocate the failed log automatically Use either vxplex dis or vxsd dis to remove the failing logs Then use vxassist addlog see the vxassist 1M manual page to add a new log to the volume V 5 1 526 VxVM vxconfigd NOTICE V 5 1 526 Detached plex plex in volume volume Description The specified plex was disabled as a result of a disk failure or as a result of the administrator removing a disk with vxdg k rmdisk A failing disk is indicated by a Detached disk message Action If hot relocation is enabled VERITAS Volume Manager objects affected by the disk failure are taken care of
48. em If the maximum number of historic log files has been reached the oldest historic log file is removed and the current log file is renamed as that file Chapter 3 Logging Commands and Transactions 31 Logging Transactions Each log file contains a header that records the host name host ID and the date and time that the log was created The following are sample entries from a transaction log file Thu Feb 13 19 30 57 2003 Clid 26924 PID 3001 DA_GET SENAO_1 DISK_GET_ATTRS SENAO_1 DISK_DISK_OP SENAO_1 8 DEVNO_GET SENAOQ_1 DANAME_GET 0x1d801d8 0x1d801a0 GET_ARRAYNAME SENA 50800200000e78b8 Part 0 Status 0 Abort Reason 0 CTLR_PTOLNAME pci 1f 4000 pci 5 SUNW qlc 4 fpe 0 0 GET_ARRAYNAME SENA 50800200000e78b8 CTLR_PTOLNAME pci 1f 4000 pci 5 SUNW qlc 5 fpe 0 0 DISCONNECT lt no request data gt The first line of each log entry is the time stamp of the transaction The Clid field corresponds to the client ID for the connection that the command opened to vxconfigd The PID field shows the process ID of the utility that is requesting the operation The Status and Abort Reason fields contain error codes if the transaction does not complete normally The remainder of the record shows the data that was used in processing the transaction Note The client ID is the same as that recorded for the corresponding command line in the command log See Logging Commands on page
49. eparate disk leave that disk connected Several of the automatic options for installation access disks other than the root disk without requiring confirmation from the administrator Disconnect all other disks containing volumes or other data that should be preserved prior to reinstalling the operating system For example if you originally installed the operating system with the home file system on a separate disk disconnect that disk to ensure that the home file system remains intact Chapter 2 Recovery from Boot Disk Failure 21 Recovery by Reinstallation Reinstall the Operating System Once any failed or failing disks have been replaced and disks not involved with the reinstallation have been detached reinstall the operating system as described in your operating system documentation Install the operating system prior to installing VxVM Ensure that no disks other than the root disk are accessed in any way while the operating system installation is in progress If anything is written on a disk other than the root disk the VERITAS Volume Manager configuration on that disk may be destroyed Note During reinstallation you can change the system s host name or host ID It is recommended that you keep the existing host name as this is assumed by the procedures in the following sections Reinstall Vx VM To reinstall VERITAS Volume Manager from CD ROM follow the instructions in the Installation Guide Cautio
50. ered stale the RAID 5 volume is active and has no valid log areas Only the third case can be overridden by using the o force option Subdisks of RAID 5 volumes can also be split and joined by using the vxsd split command and the vxsd join command These operations work the same way as those for mirrored volumes Note RAID 5 subdisk moves are performed in the same way as subdisk moves for other volume types but without the penalty of degraded redundancy Starting RAID 5 Volumes When a RAID 5 volume is started it can be in one of many states After a normal system shutdown the volume should be clean and require no recovery However if the volume was not closed or was not unmounted before a crash it can require recovery when it is started before it can be made available This section describes actions that can be taken under certain conditions Under normal conditions volumes are started automatically after a reboot and any recovery takes place automatically or is done through the vxrecover command Unstartable RAID 5 Volumes A RAID 5 volume is unusable if some part of the RAID 5 plex does not map the volume length the RAID 5 plex cannot be sparse in relation to the RAID 5 volume length the RAID 5 plex does not map a region where two subdisks have failed within a stripe either because they are stale or because they are built on a failed disk Chapter 1 Recovery from Hardware Failure 15 Failures on RAID 5
51. etached subdisk in volume 55 Detached volume 55 disabled controller connected to disk array 47 disabled dmpnode 47 disabled path belonging to dmpnode 47 enabled controller connected to disk array 48 enabled dmpnode 48 enabled path belonging to dmpnode 49 Offlining config copy 63 read error on object 52 removed disk array 52 Rootdisk has just one enabled path 46 Unable to resolve duplicate diskid 74 Volume entering degraded mode 65 PANIC messages 43 parity regeneration checkpointing 14 resynchronization for RAID 5 13 stale 9 plex kernel states DISABLED 3 11 ENABLED 3 plex states ACTIVE 3 BADLOG 11 CLEAN 3 EMPTY 3 IOFAIL 4 LOG 11 STALE 5 plexes defined 3 displaying states of 2 in RECOVER state 6 mapping problems 15 recovering mirrored volumes 5 process ID in command logging file 30 in transaction logging file 32 Index 85 wy R RAID 5 detached subdisks 10 failures 9 hot relocation 12 importance of log plexes 9 parity resynchronization 13 recovering log plexes 14 recovering stale subdisk 14 recovering volumes 12 recovery process 11 stale parity 9 starting forcibly 17 starting volumes 15 startup recovery process 11 subdisk move recovery 15 unstartable volumes 15 reattaching disks 8 reconstructing read mode stale subdisks 10 RECOVER state 6 recovery disk 8 reinstalling entire system 20 REPLAY volume state 11 restarting disabled volumes 7 resynchronization RAID 5 parity 13 root disks reco
52. f a disk with a failed status Be sure to enclose the disk name in quotes in the command Otherwise the command returns an error message The vxprint command returns a list of volumes that have mirrors on the failed disk Repeat this command for every disk with a failed status The following is sample output from running this command vxprint g mydg sF vname e sd disk disk01 v01 3 Check the status of each volume and print volume information using the following command vxprint th volume where volume is the name of the volume to be examined The vxprint command displays the status of the volume its plexes and the portions of disks that make up those plexes For example a volume named v01 with only one plex resides on the reinstalled disk named disk01 The vxprint th v01 command produces the following output V NAME RVG VSET COKSTATESTATE LENGTH READPOL PREF PLEXUTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WIDMODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE v v01 DISABLEDACTIVE 24000 SELECT fsgen pl v01 01 v01 DISABLEDNODEVICE2 4000 CONCAT RW sd disk01 06 v01 01d disk01 245759 24000 0 c1t5d1 ENA The only plex of the volume is shown in the line beginning with p1 The STATE field for the plex named v01 01 is NODEVICE The plex has space on a disk that has been replaced removed or reinstalled The plex is no
53. form these steps 1 Establish which VM disks have been removed or reinstalled using the following command vxdisk list This displays a list of system disk devices and the status of these devices For example for a reinstalled system with three disks and a reinstalled root disk the output of the vxdisk list command is similar to this DEVICE TYPE DISK GROUP STATUS c0t0d0s2 sliced error c0t1d0s2 sliced disk02 mydg online c0t2d0s2 sliced disk03 mydg online disk01 mydg failed was c0t0d0s2 The display shows that the reinstalled root device c0t0d0s2 is not associated with a VM disk and is marked with a status of error The disks disk02 and disk03 were not involved in the reinstallation and are recognized by VxVM and associated with their devices cOt1d0s2 and c0t2d0s2 The former disk01 which was the VM disk associated with the replaced disk device is no longer associated with the device cOt0d0s2 If other disks with volumes or mirrors on them had been removed or replaced during reinstallation those disks would also have a disk device listed in error state and a VM disk listed as not associated with a device VERITAS Volume Manager Troubleshooting Guide Recovery by Reinstallation 2 Once you know which disks have been removed or replaced locate all the mirrors on failed disks using the following command vxprint g diskgroup sF vname e sd_disk disk where disk is the access name o
54. g configuration daemon process could not be killed A configuration daemon process for purposes of this discussion is any process that opens the dev vx config device only one process can open that device at a time If there is a configuration daemon process already running then the k option causes a SIGKILL signal to be sent to that process If within a certain period of time there is still a running configuration daemon process the above error message is displayed Action This error can result from a kernel error that has made the configuration daemon process unkillable from some other kind of kernel error or from some other user starting another configuration daemon process after the SIGKILL signal This last condition can be tested for by running vxconfigd k again If the error message reappears contact VERITAS Technical Support V 5 1 2274 VxVM vxconfigd ERROR V 5 1 2274 volume vxconfigd cannot boot start RAID 5 volumes Description A volume that vxconfigd should start immediately upon booting the system that is the volume for the usr file system has a RAID 5 layout The usr file system should never be defined on a RAID 5 volume Action It is likely that the only recovery for this is to boot VxVM from a network mounted root file system or from a CD ROM and reconfigure the usr file system to be defined on a regular non RAID 5 volume V 5 1 2290 VxVM vxdmpadm ERROR V 5 1 2290 Attempt to enable a
55. g flagged as STALE However the reattach must occur before any volumes on the disk are started The vxreattach command is called as part of disk recovery from the vxdiskadm menus and during the boot process If possible vxreattach reattaches the failed disk media record to the disk with the same device name Reattachment places a disk in the same disk group that it was located in before and retains its original disk media name To reattach a failed disk 1 Use the vxdisk list command to see which disks have failed as shown in the following example vxdisk list DEVICE TYPE DISK GROUP STATUS c1t1d0s2 auto sliced mydg01 mydg online c1t2d0s2 auto sliced mydg02 mydg online mydg03 mydg failed was c1t3d0s2 mydg04 mydg failed was c1t4d0s2 2 Once the fault has been corrected the disks can be reattached by using the following command to rescan the device list usr sbin vxdctl enable 3 Use the vxreattach command with no options to reattach the disks etc vx bin vxreattach After reattachment takes place recovery may not be necessary unless a disk was faulty and had to be replaced Reattachment can fail if the original or another cause for the disk failure still exists 8 VERITAS Volume Manager Troubleshooting Guide Failures on RAID 5 Volumes You can use the command vxreattach c to check whether reattachment is possible without performing the operation Instead it displays the disk group and d
56. he RAID 5 volume has been resynchronized For larger volumes parity regeneration can take a long time It is possible that the system could be shut down or crash before the operation is completed In case of a system shutdown the progress of parity regeneration must be kept across reboots Otherwise the process has to start all over again To avoid the restart process parity regeneration is checkpointed This means that the offset up to which the parity has been regenerated is saved in the configuration database The o checkpt size option controls how often the checkpoint is saved If the option is not specified the default checkpoint size is used Because saving the checkpoint offset requires a transaction making the checkpoint size too small can extend the time required to regenerate parity After a system reboot a RAID 5 volume that has a checkpoint offset smaller than the volume length starts a parity resynchronization at the checkpoint offset Log Plex Recovery RAID 5 log plexes can become detached due to disk failures These RAID 5 logs can be reattached by using the att keyword for the vxplex command To reattach the failed RAID 5 log plex use the following command vxplex g mydg att r5vol r5vol 11 Stale Subdisk Recovery Stale subdisk recovery is usually done at volume start time However the process doing the recovery can crash or the volume may be started with an option such as o delayrecover that prevents subdi
57. he vxdg import command was run on the foodg disk group sed e 7310 7311 d cmdlog 8309 2778 Wed Feb 12 21 19 36 2003 7311 usr sbin vxdg m import foodg Note If there are multiple matches for the combination of the client and process ID you can determine the correct match by examining the time stamp If a utility opens a conditional connection to vxconfigd its client ID is shown as zero in the command log and as a non zero value in the transaction log You can use the process ID and time stamp to relate the log entries in such cases Chapter 3 Logging Commands and Transactions 33 OO Associating Command and Transaction Logs wr 34 VERITAS Volume Manager Troubleshooting Guide Backing Up and Restoring Disk Group Configurations 4 Disk group configuration backup and restoration allows you to backup and restore all configuration data for VERITAS Volume Manager VxVM disk groups and for VxVM objects such as volumes that are configured within the disk groups Using this feature you can recover from corruption of a disk group s configuration that is stored as metadata in the private region of a VM disk After the disk group configuration has been restored and the volume enabled the user data in the public region is available again without the need to restore this from backup media Caution The backup and restore utilities act only on VxVM configuration data They do not back up or restore any user or app
58. her location if desired V 5 0 252 VxVM vxio NOTICE V 5 0 252 read error on object subdisk of mirror plex in volume volume start offset length length corrected Description A read error occurred which caused a read of an alternate mirror and a writeback to the failing region This writeback was successful and the data was corrected on disk Action None the problem was corrected automatically Note the location of the failure for future reference If the same region of the subdisk fails again this may indicate a more insidious failure and the disk should be reformatted at the next reasonable opportunity V 5 0 258 VxVM vxdmp NOTICE V 5 0 258 removed disk array disk_array_serial_number Description A disk array has been disconnected from the host or some hardware failure has resulted in the disk array becoming inaccessible to the host Action Replace disk array hardware if this has failed VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 0 281 VxVM vxio WARNING V 5 0 281 Root volumes are not supported on your PROM version Description If your system s PROMs are not a recent OpenBoot PROM type root volumes are unusable Action If you have set up a root volume undo the configuration by running vxunroot or removing the rootdev line from etc system as soon as possible Contact your hardware vendor for an upgrade to your PROM level V 5 0 386 VxXVM vxio WARNING V 5 0 386
59. ii w Chapter 2 Recovery from Boot Disk Failure 0 2000 cece neces 19 Recovery by Reinstallation 00 eens 20 General Reinstallation Information 60 0 cee cece eee 20 Reinstalling the System and Recovering VXVM 66 66 c cece eens 21 Chapter 3 Logging Commands and Transactions 002e00eeees 29 Logging Commands sss sise she eile eee ees eed ee Peed E 29 Logging Transactions issir eae hie Beanie EE Glass wigs 4 ase Randy GE goa Ne 31 Associating Command and Transaction Logs 6 60 c cece ees 33 Chapter 4 Backing Up and Restoring Disk Group Configurations sc is 022 ease ce Sea eae aot eee in oe aA 35 Backing Up a Disk Group Configuration 000 e eee eee eee eens 36 Restoring a Disk Group Configuration 0 6 eee ees 37 Resolving Conflicting Backups for a Disk Group 0000 000005 38 Chapter 5 Error Messages 20c eee e eee teens 41 Logging Error Messages 6 cece aA i A AE AE EAE nee 41 Configuring Logging in the Startup Script 0 eee eee eee 42 Understanding Messages 66 6 c matkani aia i kavea kea iria a 43 Messages ut Aara t a s A a eaaa stk Soak se EE EE E a E ESG 45 DAEX oiea naa aeaa a e A aa a wa ae Oe ee alee See a ari tog D a a 83 VERITAS Volume Manager Troubleshooting Guide Preface The VERITAS Volume Manager Troubleshooting Guide provides information about how to recover from
60. inor to major minor Description The configuration of the named disk group includes conflicting device numbers A disk group configuration lists the recommended device number to use for each volume in the disk group If two volumes in two disk groups happen to list the same device number then one of the volumes must use an alternate device number This is called device number remapping Remapping is a temporary change to a volume If the other disk group is deported and the system is rebooted then the VERITAS Volume Manager Troubleshooting Guide Understanding Messages volume that was remapped may no longer be remapped Also volumes that are remapped once are not guaranteed to be remapped to the same device number in further reboots Action Use the vxdg reminor command to renumber all volumes in the offending disk group permanently See the vxdg 1M manual page for more information V 5 1 768 VxVM vxconfigd NOTICE V 5 1 768 Offlining config copy number on disk disk Reason reason Description An I O error caused the indicated configuration copy to be disabled This is a notice only and does not normally imply serious problems unless this is the last active configuration copy in the disk group Action Consider replacing the indicated disk since this error implies that the disk has deteriorated to the point where write errors cannot be repaired automatically The error can also result from transient problems
61. ion Any other disks that were replaced should be added using the vxdiskadm command If the disks were reinstalled during the operating system reinstallation they should be encapsulated otherwise they can be added Once all the disks have been added to the system any volumes that were completely removed as part of the configuration cleanup can be recreated and their contents restored from backup The volume recreation can be done by using the vxassist command or the graphical user interface For example to recreate the volumes v01 and v02 use the following command vxassist make v01 24000 vxassist make v02 30720 layout stripe nstripe 3 Once the volumes are created they can be restored from backup using normal backup restore procedures Chapter 2 Recovery from Boot Disk Failure 2 Recovery by Reinstallation Recreate any plexes for volumes that had plexes removed as part of the volume cleanup To replace the plex removed from volume v03 use the following command vxassist mirror v03 Once you have restored the volumes and plexes lost during reinstallation recovery is complete and your system is configured as it was prior to the failure The final step is to start up hot relocation if this is required Start up Hot Relocation To start up the hot relocation service either reboot the system or manually start the relocation watch daemon vxrelocd this also starts the vxnotify process Note Hot relocat
62. ion The o unsafe start option is considered dangerous as it can make the contents of the volume unusable Using it is not recommended 2 Any existing log plexes are zeroed and enabled If all logs fail during this process the start process is aborted 3 Ifno stale subdisks exist or those that exist are recoverable the volume is put in the ENABLED volume kernel state and the volume state is set to ACTIVE The volume is now started Recovering a RAID 5 Volume The types of recovery that may typically be required for RAID 5 volumes are the following Parity Resynchronization Log Plex Recovery Stale Subdisk Recovery Parity resynchronization and stale subdisk recovery are typically performed when the RAID 5 volume is started or shortly after the system boots They can also be performed by running the vxrecover command For more information on starting RAID 5 volumes see Starting RAID 5 Volumes on page 15 If hot relocation is enabled at the time of a disk failure system administrator intervention is not required unless no suitable disk space is available for relocation Hot relocation is triggered by the failure and the system administrator is notified of the failure by electronic mail Hot relocation automatically attempts to relocate the subdisks of a failing RAID 5 plex After any relocation takes place the hot relocation daemon vxrelocd also initiates a parity resynchronization In the
63. ion should only be started when you are sure that it will not interfere with other reconfiguration procedures See Modifying the Behavior of Hot Relocation in the Administering Hot Relocation chapter of the VERITAS Volume Manager Administrator s Guide for more information about running vxrelocd and about modifying its behavior To determine if hot relocation has been started use the following command to search for its entry in the process table ps ef grep vxrelocd VERITAS Volume Manager Troubleshooting Guide Logging Commands and Transactions 3 This chapter provides information on how to administer logging of commands and transactions in VERITAS Volume Manager VxVM For information on how to administer error logging see Error Messages on page 41 Logging Commands The vxcmdlog command allows you to log the invocation of other VxVM commands to a file The following table demonstrates the usage of vxcmdlog Command Description vxcmdlog 1 List current settings for command logging vxcmdlog m on Turn on command logging vxcmdlog s 512k Set the maximum command log file size to 512KB vxcmdlog n 10 Set the maximum number of historic command log files to 10 vxcmdlog n no_limit Remove any limit on the number of historic command log files vxcmdlog m off Turn off command logging Command lines are logged to the file cmd1log in the directory etc vx 1log This path name is a s
64. iple disk failures 79 Disk group has no valid configuration copies 58 61 62 Disk group version doesn t support feature 68 Disk is in use by another host 61 Disk is not usable 70 Disk not moving but subdisks on it are 71 Disk write failure 57 Duplicate record in configuration 58 enable failed 65 66 Error in disk group configuration copies 69 Errors in some configuration copies 58 60 66 Format error in configuration copy 58 import failed 61 Insufficient DRL log size logging is disabled 71 Invalid block number 58 Invalid magic number 58 logging 41 Memory allocation failure 80 Name conflicts with imported diskgroup 70 No more space in disk group F configuration 81 No such device or address 77 No such file or directory 77 no valid complete plexes 54 No valid disk found containing disk group 61 no valid plexes 78 not updated with new host ID 57 RAID 5 plex does not map entire volume length 16 Record already exists in disk group 69 Record is associated 69 Record volume is in disk group diskgroup1 plex is in group diskgroup2 63 Reimport of disk group failed 61 Skip disk group with duplicate name 59 some subdisks are unusable and the parity is stale 16 startup script 42 System boot disk does not have a valid rootvol plex 64 System startup failure 81 The VxVM restore daemon is already running 71 There are two backups that have the same diskgroup name with different diskgroup id 38 There is no volume co
65. ir the situation by rebooting Otherwise the disk group may have to be recreated and restored from a backup V 5 1 8063 VxVM vxconfigd ERROR V 5 1 8063 Memory allocation failure Description This implies that there is insufficient memory to start VxVM and to get the volumes for the root and usr file systems running Action This error should not normally occur unless your system has very small amounts of memory Adding swap space will probably not help because this error is most likely to occur early in the boot sequence before swap areas have been added V 5 1 8064 VxVM vxconfigd FATAL ERROR V 5 1 8064 Memory allocation failure during startup Description This implies that there is insufficient memory to start up VxVM and to get the volumes for the root and usr file systems running Action This error should not normally occur unless your system has very small amounts of memory Adding swap space probably will not help because this error is most likely to occur early in the boot sequence before swap areas have been added VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 8259 VxVM vxconfigd ERROR V 5 1 8259 System startup failed Description Either the root or the usr file system volume could not be started rendering the system unusable The error that resulted in this condition should appear prior to this error message Action Look up other error messages appearing on the con
66. isk media name where the disk can be reattached See the vxreattach 1M manual page for more information on the vxreattach command Failures on RAID 5 Volumes Failures are seen in two varieties system failures and disk failures A system failure means that the system has abruptly ceased to operate due to an operating system panic or power failure Disk failures imply that the data on some number of disks has become unavailable due to a system failure such as a head crash electronics failure on disk or disk controller failure System Failures RAID 5 volumes are designed to remain available with a minimum of disk space overhead if there are disk failures However many forms of RAID 5 can have data loss after a system failure Data loss occurs because a system failure causes the data and parity in the RAID 5 volume to become unsynchronized Loss of synchronization occurs because the status of writes that were outstanding at the time of the failure cannot be determined If a loss of sync occurs while a RAID 5 volume is being accessed the volume is described as having stale parity The parity must then be reconstructed by reading all the non parity columns within each stripe recalculating the parity and writing out the parity stripe unit in the stripe This must be done for every stripe in the volume so it can take a long time to complete Caution While the resynchronization of a RAID 5 volume without log plexes is being performed
67. isk to detach At this point not all data disks exist to provide the data upon request Instead parity regions are used to regenerate the data for each stripe in the array Consequently access takes longer and involves reading from all drives in the stripe Action Check for other console error messages that indicate the cause of the failure Replace any failed disks as soon as possible Chapter 5 Error Messages ol r Understanding Messages V 5 0 251 VXVM vxio WARNING V 5 0 251 read error on object object of mirror plex in volume volume start offset length length Description An error was detected while reading from a mirror This error may lead to further action shown by later error messages Action If the volume is mirrored no further action is necessary since the alternate mirror s contents will be written to the failing mirror this is often sufficient to correct media failures If this error occurs often but never leads to a plex detach there may be a marginally defective region on the disk at the position indicated It may eventually be necessary to remove data from this disk see the vxevac 1M manual page and then to reformat the drive If the volume is not mirrored this message indicates that some data could not be read The file system or other application reading the data may report an additional error but in either event data has been lost The volume can be partially salvaged and moved to anot
68. k group has become disabled Earlier error messages should indicate the cause This message indicates that disks in that disk group were not updated with a new VERITAS Volume Manager host ID This warning message should result only from a vxdct1 hostid operation Action Typically unless a disk group was disabled due to transient errors there is no way to repair a disabled disk group The disk group may have to be reconstructed from scratch If the disk group was disabled due to a transient error such as a cabling problem then a future reboot may not automatically import the named disk group due to the change in the system s VERITAS Volume Manager host ID In such a case import the disk group directly using vxdg import with the C option V 5 1 569 VxVM vxconfigd ERROR V 5 1 569 Disk group group Disk disk Cannot auto import group reason Description On system startup vxconfigd failed to import the disk group associated with the named disk A message related to the specific failure is given in reason Additional error messages may be displayed that give more information on the specific error In particular this is often followed by VxVM vxconfigd ERROR V 5 1 579 Disk group group Errors in some configuration copies Disk device copy number Block bno error The most common reason for auto import failures is excessive numbers of disk failures making it impossible for VxVM to find correct copies of the disk group
69. le some subdisks are unusable and the parity is stale This situation can be avoided by always using two or more RAID 5 log plexes in RAID 5 volumes RAID 5 log plexes prevent the parity within the volume from becoming stale which prevents this situation see System Failures on page 9 for details VERITAS Volume Manager Troubleshooting Guide Failures on RAID 5 Volumes Forcibly Starting RAID 5 Volumes You can start a volume even if subdisks are marked as stale for example if a stopped volume has stale parity and no RAID 5 logs and a disk becomes detached and then reattached The subdisk is considered stale even though the data is not out of date because the volume was in use when the subdisk was unavailable and the RAID 5 volume is considered invalid To prevent this case always have multiple valid RAID 5 logs associated with the array whenever possible To start a RAID 5 volume with stale subdisks you can use the f option with the vxvol start command This causes all stale subdisks to be marked as non stale Marking takes place before the start operation evaluates the validity of the RAID 5 volume and what is needed to start it Also you can mark individual subdisks as non stale by using the following command vxmend g diskgroup fix unstale subdisk Ifsome subdisks are stale and need recovery and if valid logs exist the volume is enabled by placing it in the ENABLED kernel state and the volume is availabl
70. leshooting Guide
71. lication data that is contained within volumes or other VxVM objects If you use vxdiskunsetup and vxdisksetup on a disk and specify attributes that differ from those in the configuration backup this may corrupt the public region and any user data therein The vxconfigbackupd daemon monitors changes to the VxVM configuration and automatically records any configuration changes that occur Two utilities vxconfigbackup and vxconfigrestore are provided for backing up and restoring a VxVM configuration for a disk group When importing a disk group any of the following errors indicate that the disk group configuration and or disk private region headers have become corrupted VxVM vxconfigd ERROR V 5 1 569 Disk group group Disk disk Cannot auto import group reason The reason for the error is usually one of the following Configuration records are inconsistent Disk group has no valid configuration copies Duplicate record in configuration Errors in some configuration copies Format error in configuration copy Invalid block number Invalid magic number 35 wy Backing Up a Disk Group Configuration If VxVM cannot update a disk group s configuration because of disk errors it disables the disk group and displays the following error VxVM vxconfigd ERROR V 5 1 123 Disk group group Disabled by errors If such errors occur you can restore the disk group configuration from a backup after you have corrected any unde
72. m or misconfiguration of any logical volume management software being used for the var file system Also verify that the encapsulation if configured of your boot disk is complete and correct VxXVM vxconfigd ERROR V 5 1 1589 enable failed transactions are disabled Description Regular startup of vxconfigd failed This error can also result from the command vxdctl enable vxconfigd is continuing to run but no configuration updates are possible until the error condition is repaired Additionally this may be followed with VxVM vxconfigd ERROR V 5 1 579 Disk group group Errors in some configuration copies Disk device copy number Block bno error Other error messages may be displayed that further indicate the underlying problem Action Evaluate the error messages to determine the root cause of the problem Make changes suggested by the errors and then try rerunning the command If the Errors in some configuration copies error occurs again that may indicate the real problem lies with the configuration copies in the disk group You can recreate a disk group configuration by using the procedures given in Restoring a Disk Group Configuration on page 37 VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 2031 VxXVM vxconfigd ERROR V 5 1 2031 Cannot kill existing daemon pid process_ID Description The k kill existing vxconfigd process option was specified but a runnin
73. mmand vxconfigd x cleartempdir 2 gt dev console The vxvol vxplex and vxsd commands make use of these tempdb files to communicate locking information If the file is cleared then locking information can be lost Without this locking information two utilities can end up making incompatible changes to the configuration of a volume V 5 1 2829 VxVM vxdg ERROR V 5 1 2829 diskgroup Disk group version doesn t support feature see the vxdg upgrade command Description The version of the specified disk group does not support disk group move split or join operations Action Use the vxdg upgrade diskgroup command to update the disk group version VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 2841 VxVM vxconfigd ERROR V 5 1 2841 enable failed Error in disk group configuration copies Unexpected kernel error in configuration update transactions are disabled Description Usually means that multipathing is misconfigured Action See V 5 1 7265 on page 74 for possible causes and solutions V 5 1 2860 VxVM vxdg ERROR V 5 1 2860 Transaction already in progress Description One of the disk groups specified in a disk group move split or join operation is currently involved in another unrelated disk group move split or join operation possibly as the result of recovery from a system failure Action Use the vxprint command to display the status of the disk groups i
74. mmit analysis of the state of the disk group configuration and reinstalls the disk headers where these have become corrupted etc vx bin vxconfigrestore p 1 directory diskgroup dgid The disk group can be specified either by name or by ID The 1 option allows you to specify a directory for the location of the backup configuration files other than the default location etc vx cbr bk See Backing Up a Disk Group Configuration on page 36 for details To specify that the disk headers are not to be reinstalled use this form of the command etc vx bin vxconfigrestore n 1 directory diskgroup dgid At the precommit stage you can use the vxprint command to examine the configuration that the restored disk group will have You can choose to proceed to commit the changes and restore the disk group configuration Alternatively you can cancel the restoration before any permanent changes have been made To abandon restoration at the precommit stage use this command etc vx bin vxconfigrestore d 1 directory diskgroup dgid Chapter 4 Backing Up and Restoring Disk Group Configurations 37 r Restoring a Disk Group Configuration To commit the changes that are required to restore the disk group configuration use the following command etc vx bin vxconfigrestore c 1 directory diskgroup dgid If no disk headers are reinstalled the configuration copies in the disks private regions are updated
75. n Guide FATAL ERROR A fatal error message from a configuration daemon such as vxconfigd indicates a severe problem with the operation of VxVM that prevents it from running The following is an example of such a message VxVM vxconfigd FATAL ERROR V 5 0 662 Group bootdg Cannot update kernel ERROR An error message from a command indicates that the requested operation cannot be performed correctly The following is an example of such a message VxVM vxassist ERROR V 5 1 5150 Insufficient number of active snapshot mirrors in snapshot_volume WARNING A warning message from the kernel indicates that a non critical operation has failed possibly because some resource is not available or the operation is not possible The following is an example of such a message VxVM vxio WARNING V 5 0 55 Cannot find device number for boot_path NOTICE A notice message indicates that an error has occurred that should be monitored Shutting down the system is unnecessary although you may need to take action to remedy the fault at a later date The following is an example of such a message VxVM vxio NOTICE V 5 0 252 read error on object Subdisk of mirror plex in volume volume start offset length length corrected INFO An informational message does not indicate an error and requires no action The unique message number consists of an alpha numeric string that begins with the letter V For example in the message
76. n To reconstruct the Volume Manager configuration that remains on the non root disks do not use vxinstal11 to initialize VxVM after loading the software from CD ROM Recover the VERITAS Volume Manager Configuration Once the VERITAS Volume Manager packages have been loaded and you have installed the license for VxVM recover the VERITAS Volume Manager configuration using the following procedure 1 Touch etc vx reconfig d state d install db 2 Shut down the system 3 Reattach the disks that were removed from the system 4 Reboot the system 5 When the system comes up bring the system to single user mode using the following command exec init S 6 When prompted enter the password and press Return to continue VERITAS Volume Manager Troubleshooting Guide Recovery by Reinstallation 7 Remove files involved with installation that were created when you loaded VxVM but which are no longer needed using the following command rm rf etc vx reconfig d state d install db 8 Start some VERITAS Volume Manager I O daemons using the following command vxiod set 10 9 Start the VERITAS Volume Manager configuration daemon vxconfigd in disabled mode using the following command vxconfigd m disable 10 Initialize the vxconfigd daemon using the following command vxdctl init 11 Initialize the DMP subsystem using the following command vxdctl initdmp 12 Enable vxconfigd using the following c
77. nd vxvol g diskgroup o bg start volume Note If the data in the plex was corrupted and the volume has no ACTIVE or CLEAN redundant plexes from which its contents can be resynchronized it must be restored from a backup or from a snapshot image VERITAS Volume Manager Troubleshooting Guide Forcibly Restarting a Disabled Volume Forcibly Restarting a Disabled Volume If a disk failure caused a volume to be disabled and the volume does not contain any valid redundant plexes you must restore the volume from a backup after replacing the failed disk Any volumes that are listed as Unstartable must be restarted using the vxvol command before restoring their contents from a backup vxvol g diskgroup o bg f start volume The f option forcibly restarts the volume and the o bg option resynchronizes its plexes as a background task For example to restart the volume myvo1 so that it can be restored from backup use the following command vxvol g mydg o bg f start myvol Clearing the Failing Flag for a Disk If I O errors are intermittent rather than persistent VERITAS Volume Manager sets the failing flag on a disk rather than detaching the disk Such errors can occur due to the temporary removal of a cable controller faults a partially faulty LUN in a disk array or a disk with a few bad sectors or tracks If the hardware fault is not with the disk itself for example it is caused by problems with the cont
78. nd on the other disk group to remove the objects that have TUTILO fields marked as MOVE vxdg o remove recover diskgroup2 If only one disk group is available to be imported use the following command to reset the MOVE flags on this disk group vxdg o clean recover diskgroup VERITAS Volume Manager Troubleshooting Guide Recovery from Boot Disk Failure 2 VERITAS Volume Manager VxVM protects systems from disk and other hardware failures and helps you to recover from such events This chapter describes recovery procedures and information to help you prevent loss of data or system access due to the failure of the boot root disk It also includes procedures for repairing the root and usr file systems For information about recovering volumes and their data on non boot disks see Recovery from Hardware Failure on page 1 For more information about protecting your system see the VERITAS Volume Manager Installation Guide Note The examples in this chapter assume that the boot root disk is configured on the device cOt0d0s2 Your system may be configured to use a different device Encapsulation and mirroring of the root disk is not supported in this release Recovery by Reinstallation Recovery by Reinstallation Reinstallation is necessary if all copies of your boot root disk are damaged or if certain critical files are lost due to file system damage If these types of failures occur attempt t
79. nfigured for the root device 64 Transaction already in progress 69 transactions are disabled 69 Unexpected kernel error in configuration update 69 Unrecognized operating mode 76 update failed 80 Version number of kernel does not match vxconfigd 65 Volume for mount point usr not found in rootdg disk group 73 Volume is not startable 16 volume not in rootdg disk group 72 Volume record id is not found in the configuration 72 vxconfigd cannot boot start RAID 5 volumes 67 failing flag 84 VERITAS Volume Manager Troubleshooting Guide clearing 7 failures disk 10 system 9 FATAL ERROR messages 44 fatal error messages bootdg cannot be imported during boot 63 Memory allocation failure during startup 80 files disk group configuration backup 36 hardware failure recovery from 1 hot relocation defined 1 RAID 5 12 starting up 28 INFO messages 44 install db file 23 IOFAIL plex state 4 listing unstartable volumes 2 log file default 41 syslog error messages 42 vxconfigd 41 LOG plex state 11 log plexes importance for RAID 5 9 recovering RAID 5 14 logging associating commands and transactions 33 commands and transactions 29 directory 29 31 mirrored volumes recovering 5 MOVE flag set in TUTILO field 18 NEEDSYNC volume state 13 NOTICE messages 44 notice messages added disk array 45 Attempt to disable controller failed 46 Detached disk 78 Detached log for volume 54 Detached plex in volume 54 D
80. ng of the line turns off syslog logging for vxconfigd After making changes to the way that vxconfigd is invoked in the startup file run the following command on a Solaris 10 system to notify that the service configuration has been changed svcadm refresh vxvm vxvm sysboot VERITAS Volume Manager Troubleshooting Guide Understanding Messages Note By default vxconfigd is started at boot time with the x syslog option This redirects vxconfigd console messages to syslog If you want to retain this behavior when restarting vxconfigd from the command line include the x syslog argument as restarting vxconfigd does not preserve the option settings with which it was previously running Similarly any VERITAS Volume Manager operations that require vxconfigd to be restarted may not retain the behavior that was previously specified by option settings For more information on logging options for vxconfigd refer to the vxconfigd 1M manual page Understanding Messages VxVM is fault tolerant and resolves most problems without system administrator intervention If the configuration daemon vxconfigd recognizes the actions that are necessary it queues up the transactions that are required VxVM provides atomic changes of system configurations either a transaction completes fully or the system is left in the same state as though the transaction was never attempted If vxconfigd is unable to recognize and fix system problems the sys
81. nstructing read This is a more expensive operation than simply reading the data and can result in degraded read performance When a RAID 5 volume has stale subdisks it is considered to be in degraded mode A RAID 5 volume in degraded mode can be recognized from the output of the vxprint ht command as shown in the following display V NAME RVG VSET COKSTATESTATE KENGTH READPOL PREFPLEX UTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WID MODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFF DEVICE MODE SV NAME PLEX VOLNAME NVOLLAYRLENGTH COL OFF AM NM MODE vV r5vol ENABLED DEGRADED204800 RAID raid5 pl r5vol 01 r5vol ENABLED ACTIVE 204800 RAID 3 16 RW sd disk01 01 r5vol 01ldisk01 0 102400 0 0 c2t9d0 ENA sd disk02 01 r5vol 01disk02 0 102400 1 0 c2t10d0 ds sd disk03 01 r5vol 01disk03 0 102400 2 0 c2t11d0 ENA pl r5vol 02 r5vol ENABLED LOG 1440 CONCAT RW sd disk04 01 r5vol 02disk04 0 1440 0 c2t12d0 ENA pl r5vol 03 r5vol ENABLED LOG 1440 CONCAT RW sd disk05 01 r5vol 03disk05 0 1440 0 c2t14d0 ENA The volume r5vol is in degraded mode as shown by the volume state which is listed as DEGRADED The failed subdisk is disk02 01 as shown by the MODE flags d indicates that the subdisk is detached and S indicates that the subdisk s contents are stale Note Do not run the vxr5check command ona RAID
82. ntacts visit https my veritas com productcenter ContactVeritas jsp For information on purchasing product documentation visit http webstore veritas com Documentation Feedback Your feedback on product documentation is important to us Send suggestions for improvements and reports on errors or omissions to foundation_docs veritas com Include the title and part number of the document located in the lower left corner of the title page and chapter and section titles of the text on which you are reporting Our goal is to ensure customer satisfaction by providing effective quality documentation For assistance with topics other than documentation visit http support veritas com Preface vii y Documentation Feedback r viii VERITAS Volume Manager Troubleshooting Guide Recovery from Hardware Failure 1 VERITAS Volume Manager VxVM protects systems from disk and other hardware failures and helps you to recover from such events This chapter describes recovery procedures and information to help you prevent loss of data or system access due to disk and other hardware failures If a volume has a disk I O failure for example because the disk has an uncorrectable error VxVM can detach the plex involved in the failure I O stops on that plex but continues on the remaining plexes of the volume If a disk fails completely VxVM can detach the disk from its disk group All plexes on the disk are disabled If there are
83. nvolved If vxprint shows that the TUTILO field for a disk group is set to MOVE and you are certain that no disk group move split or join should be in progress use the vxdg command to clear the field as described in Recovering from Incomplete Disk Group Moves on page 18 Otherwise retry the operation V 5 1 2866 VxVM vxdg ERROR V 5 1 2866 object Record already exists in disk group Description A disk group join operation failed because the name of an object in one disk group is the same as the name of an object in the other disk group Such name clashes are most likely to occur for snap objects and snapshot plexes Action Use the following command to change the object name in either one of the disk groups vxedit g diskgroup rename old_name new_name For more information about using the vxedit command see the vxedit 1M manual page V 5 1 2879 VxVM vxdg ERROR V 5 1 2879 Subdisk Record is associated Description The named subdisk is not a top level object Action Objects specified for a disk group move split or join must be either disks or top level volumes Chapter 5 Error Messages 69 Wr Understanding Messages V 5 1 2907 VxVM vxdg ERROR V 5 1 2907 diskgroup Disk group does not exist Description The disk group does not exist or is not imported Action Use the correct name or import the disk group and try again V 5 1 2911 VxVM vxdg ERROR V 5 1 2911 diskname Disk is not usable Descrip
84. o preserve as much of the original VxVM configuration as possible Any volumes that are not directly involved in the failure do not need to be reconfigured You do not have to reconfigure any volumes that are preserved General Reinstallation Information This section describes procedures used to reinstall VxVM and preserve as much of the original configuration as possible after a failure Note System reinstallation destroys the contents of any disks that are used for reinstallation All VxVM related information is removed during reinstallation Data removed includes data in private areas on removed disks that contain the disk identifier and copies of the VxVM configuration The removal of this information makes the disk unusable as a VM disk The system root disk is always involved in reinstallation Any other disks that are involved in the reinstallation or that are removed and replaced can lose VxVM configuration data including volumes and mirrors If a disk including the root disk is not under VxVM control prior to the failure no VxVM configuration data is lost at reinstallation For information on replacing disks see Removing and Replacing Disks in the Administering Disks chapter of the VERITAS Volume Manager Administrator s Guide When reinstallation is necessary the only volumes saved are those that reside on or have copies on disks that are not directly involved with the failure and reinstallation Any v
85. o valid configuration copies This indicates that the disk group configuration copies have become corrupt due to disk failures writing on the disk by an application or the administrator or bugs in VxVM You can recreate a disk group configuration by using the procedures given in Restoring a Disk Group Configuration on page 37 V 5 1 587 VxVM vxdg ERROR V 5 1 587 disk group groupname import failed reason Description The import of a disk group failed for the specified reason e Action The action to be taken depends on the reason given in the error message Disk is in use by another host No valid disk found containing disk group The first message indicates that disks have been moved from a system that has crashed or that failed to detect the group before the disk was moved The locks stored on the disks must be cleared The second message indicates that the disk group does not contain any valid disks not that it does not contain any disks The disks may be considered invalid due to a mismatch between the host ID in their configuration copies and that stored in the etc vx volboot file To clear locks on a specific set of devices use the following command vxdisk clearimport devicename To clear the locks during import use the following command vxdg C import diskgroup Chapter 5 Error Messages 61 Sr Understanding Messages Caution Be careful when using the vxdisk clearimport or vxdg C impor
86. of Technology All rights reserved Export of this software from the United States of America may require a specific license from the United States Government It is the responsibility of any person or organization contemplating export to obtain such a license before exporting WITHIN THAT CONSTRAINT permission to use copy modify and distribute this software and its documentation for any purpose and without fee is hereby granted provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation and that the name of M LT not be used in advertising or publicity pertaining to distribution of the software without specific written prior permission M I T makes no representations about the suitability of this software for any purpose It is provided as is without express or implied warranty Sun Microsystems Trademarks Sun Solaris SunOS Java Sun Java System Cluster Sun StorEdge Solstice DiskSuite Sun Fire Sun Enterprise Online Backup and Netra are trademarks or registered trademarks of Sun Microsystems Inc in the United States and other countries All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International SPARC trademarks are based upon an architecture developed by Sun Microsystems Inc ji VERITAS Volume Manager Troubleshooting Guide Contents Preface iia aia fa ceangl iit tan k
87. olumes on the root disk and other disks involved with the failure or reinstallation are lost during reinstallation If backup copies of these volumes are available the volumes can be restored after reinstallation r 20 VERITAS Volume Manager Troubleshooting Guide Recovery by Reinstallation Reinstalling the System and Recovering VxVM To reinstall the system and recover the VERITAS Volume Manager configuration use the following procedure These steps are described in detail in the sections that follow 1 Prepare the System for Reinstallation on page 21 Replace any failed disks or other hardware and detach any disks not involved in the reinstallation 2 Reinstall the Operating System on page 22 Reinstall the base system and any other unrelated Volume Manager packages 3 Reinstall VxVM on page 22 Add the Volume Manager package but do not execute the vxinstal1 command 4 Recover the VERITAS Volume Manager Configuration on page 22 5 Clean up the System Configuration on page 24 Restore any information in volumes affected by the failure or reinstallation 6 Start up Hot Relocation on page 28 Prepare the System for Reinstallation To prevent the loss of data on disks not involved in the reinstallation involve only the root disk and any other disks that contain portions of the operating system in the reinstallation procedure For example if the usr file system is configured on a s
88. ommand vxdctl enable The reconfiguration is complete at this point For information on replacing disks see Removing and Replacing Disks in the Administering Disks chapter of the VERITAS Volume Manager Administrator s Guide There are several methods available to replace a disk choose the method that you prefer If the root disk or another disk was involved with the reinstallation any volumes or mirrors on that disk or other disks no longer attached to the system are now inaccessible If a volume had only one plex contained on a disk that was reinstalled removed or replaced then the data in that volume is lost and must be restored from backup Chapter 2 Recovery from Boot Disk Failure 23 Recovery by Reinstallation Clean up the System Configuration To clean up the configuration of your system after reinstallation of VxVM you must address the following issues Clean up Volumes Clean up Disk Configuration Final Volume Reconfiguration Clean up Volumes Determine which volumes need to be restored from backup The volumes to be restored include those with all mirrors all copies of the volume residing on disks that have been reinstalled or removed These volumes are invalid and must be removed recreated and restored from backup If only some mirrors of a volume exist on reinstalled or removed disks these mirrors must be removed The mirrors can be re added later To restore the volumes per
89. on If the disks should be imported into the group this must be done by adding the disk to the group at a later stage during which all configuration information for the disk is lost V 5 1 557 VxVM vxconfigd ERROR V 5 1 557 Disk disk group group device device not updated with new host ID Error reason Description This can result from using vxdct1 hostid to change the VERITAS Volume Manager host ID for the system The error indicates that one of the disks in a disk group could not be updated with the new host ID This usually indicates that the disk has become inaccessible or has failed in some other way Action Try running the following command to determine whether the disk is still operational vxdisk check device If the disk is no longer operational vxdisk should print a message such as device Error Disk write failure This will result in the disk being taken out of active use in its disk group if it has not already been taken out of use If the disk is still operational which should not be the case vxdisk prints device Okay If the disk is listed as Okay try running vxdct1 hostid again If it still results in an error contact VERITAS Technical Support Chapter 5 Error Messages 57 Understanding Messages V 5 1 568 VxVM vxconfigd WARNING V 5 1 568 Disk group group is disabled disks not updated with new host ID Description As a result of failures the named dis
90. ovided to help you to locate and correct the problem You may find it useful to consult the VxVM command and transaction logs to understand the context in which an error occurred See Logging Commands and Transactions on page 29 for more information Logging Error Messages VxVM provides the option of logging console output to a file This logging is useful in that any messages output just before a system crash will be available in the log file presuming that the crash does not result in file system corruption vxconfigd controls whether such logging is turned on or off If enabled the default log file is var vxvm vxconfigd log vxconfigd also supports the use of syslog to log all of its regular console messages When this is enabled all console output is directed through the syslog interface syslog and log file logging can be used together to provide reliable logging to a private log file along with distributed logging through syslogd Note syslog logging is enabled by default Log file logging is disabled by default 41 wy Configuring Logging in the Startup Script To enable logging of console output to the file var vxvm vxconfigd 1og edit the startup script for vxconfigd as described in Configuring Logging in the Startup Script or invoke vxconfigd under the C locale as shown here vxconfigd x 1 9 x log There are 9 possible levels of debug logging 1 provides the least detail and 9 the most
91. ption A volume with an insufficient DRL log size was started successfully but DRL logging is disabled and a full recovery is performed Action Create anew DRL of sufficient size V 5 1 3243 VxXVM vxdmpadm ERROR V 5 1 3243 The VxVM restore daemon is already running You can stop and restart the restore daemon with desired arguments for changing any of its parameters Description The vxdmpadm start restore command has been executed while the restore daemon is already running Action Stop the restore daemon and restart it with the required set of parameters as shown in the vxdmpadm 1M manual page V 5 1 3362 VxVM vxdmpadm ERROR V 5 1 3362 Attempt to disable controller failed One or more devices can be accessed only through this controller Use the f option if you still want to disable this controller Description Disabling the controller could lead to some devices becoming inaccessible Action To disable the only path connected to a disk use the f option Chapter 5 Error Messages 71 Understanding Messages V 5 1 3689 VxVM vxassist ERROR V 5 1 3689 Volume record id rid is not found in the configuration Description An error was detected while reattaching a snapshot volume using snapback This happens if a volume s record identifier rid changes as a result of a disk group split that moved the original volume to a new disk group The snapshot volume is unable to recognize the original volume
92. ring from Incomplete Disk Group Moves If the system crashes or a subsystem fails while a disk group move split or join operation is being performed VxVM attempts either to reverse or to complete the operation when the system is restarted or the subsystem is repaired Whether the operation is reversed or completed depends on how far it had progressed Automatic recovery depends on being able to import both the source and target disk groups If this is not possible for example if one of the disk groups has been imported on another host perform the following steps to recover the disk group 1 Use the vxprint command to examine the configuration of both disk groups Objects in disk groups whose move is incomplete have their TUTILO fields set to MOVE Enter the following command to attempt completion of the move vxdg recover sourcedg This operation fails if one of the disk groups cannot be imported because it has been imported on another host or because it does not exist VxVM vxdg ERROR V 5 1 2907 diskgroup Disk group does not exist If the recovery fails perform one of the following steps as appropriate If the disk group has been imported on another host export it from that host and import it on the current host If all the required objects already exist in either the source or target disk group use the following command to reset the MOVE flags in that disk group vxdg o clean recover diskgroup1 Use the following comma
93. rlying problem such as failed or disconnected hardware Configuration data from a backup allows you to reinstall the private region headers of VxVM disks in a disk group whose headers have become damaged to recreate a corrupted disk group configuration or to recreate a disk group and the VxVM objects within it You can also use the configuration data to recreate a disk group on another system if the original system is not available Note Restoration of a disk group configuration requires that the same physical disks are used as were configured in the disk group when the backup was taken The following sections describe how to back up and restore disk group configurations Backing Up a Disk Group Configuration VxVM uses the disk group configuration daemon to monitor the configuration of disk groups and to back up the configuration whenever it is changed By default the five most recent backups are preserved If required you can also back up a disk group configuration by running the vxconfigbackup command The following table describes the files that record disk group configuration information Backup Files Description etc vx cbr bk diskgroup dgid dgid dginfo Disk group information etc vx cbr bk diskgroup dgid dgid diskinfo Disk attributes etc vx cbr bk diskgroup dgid dgid binconfig Binary configuration copy etc vx cbr bk diskgroup dgid dgid cf grec Configuration records in vxprint m format Here diskgroup is
94. roller or the cable path to the disk you can use the vxedit command to unset the failing flag after correcting the source of the I O error Caution Do not unset the failing flag if the reason for the I O errors is unknown If the disk hardware truly is failing and the flag is cleared there is a risk of data loss To clear the failing flag ona disk 1 Use the vxdisk list command to find out which disks are failing vxdisk list DEVICE TYPE DISK GROUP STATUS c1t1d0s2 auto sliced mydg01 mydg online c1t2d0s2 auto sliced mydg02 mydg online failing c1t3d0s2 auto sliced mydg03 mydg online 2 Use the vxedit set command to clear the flag for each disk that is marked as failing in this example mydg02 vxedit set failing off mydg02 Chapter 1 Recovery from Hardware Failure 7 Reattaching Failed Disks 3 Use the vxdisk list command to verify that the failing flag has been cleared vxdisk list DEVICE TYPE DISK GROUP STATUS c1t1d0s2 auto sliced mydg01 mydg online c1t2d0s2 auto sliced mydg02 mydg online c1t3d0s2 auto sliced mydg03 mydg online Reattaching Failed Disks You can perform a reattach operation if a disk could not be found at system startup or if VxVM is started with some disk drivers unloaded and unloadable causing disks to enter the failed state If the underlying problem has been fixed such as a cable or controller fault use the vxreattach command to reattach the disks without plexes bein
95. root mirror on that disk which is not stale or offline Action Try to boot from one of the named disks using the associated boot command that is listed in the message V 5 1 1063 VxVM vxconfigd ERROR V 5 1 1063 There is no volume configured for the root device Description The system is configured to boot from a root file system defined on a volume but there is no root volume listed in the configuration of the boot disk group There are two possible causes of this error Case 1 The etc system file was erroneously updated to indicate that the root device is pseudo vxio 0 0 This can happen only as a result of direct manipulation by the administrator Case 2 The system somehow has a duplicate boot disk group one of which contains a root file system volume and one of which does not and vxconfigd somehow chose the wrong one Since vxconfigd chooses the more recently accessed version of the boot disk group this error can happen if the system clock was updated incorrectly at some point reversing the apparent access order of the two disk groups This can also happen if some disk group was deported and assigned the same name as the boot disk group with locks given to this host Action In case 1 boot the system on a CD ROM or networking mounted root file system directly mount the disk partition of the root file system and remove the following lines from etc system rootdev pseudo vxio 0 0 set vxio vol_rootdev_is_vol
96. rt some of the configuration copies in the named disk group were found to have format or other types of errors which make those copies unusable This message lists all configuration copies that have uncorrected errors including any appropriate logical block number If no other reasons are displayed then this may be the cause of the disk group import failure Action If some of the copies failed due to transient errors such as cable failures then a reboot or re import may succeed in importing the disk group Otherwise the disk group configuration may have to be restored You can recreate a disk group configuration by using the procedures given in Restoring a Disk Group Configuration on page 37 VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 583 VxVM vxconfigd ERROR V 5 1 583 Disk group group Reimport of disk group failed reason Description After vxconfigd was stopped and restarted or disabled and then enabled VxVM failed to recreate the import of the indicated disk group The reason for failure is specified Additional error messages may be displayed that give further information describing the problem Action A major cause for this kind of failure is disk failures that were not addressed before vxconfigd was stopped or disabled If the problem is a transient disk failure then rebooting may take care of the condition The error may be accompanied by messages such as Disk group has n
97. s Device is already open This indicates that some process most likely vxconfigd already has dev vx config open Less likely reasons are No such file or directory or No such device or address For either of these reasons likely causes are The VERITAS Volume Manager package installation did not complete correctly The device node was removed by the administrator or by an errant shell script Action If the reason is Device is already open stop or kill the old vxconfigd by running the command vxdctl k stop For other failure reasons consider re adding the base VERITAS Volume Manager package This will reconfigure the device node and re install the VERITAS Volume Manager kernel device drivers See the Installation Guide for information on how to add the package If you cannot re add the package contact VERITAS Technical Support for more information Chapter 5 Error Messages 77 Understanding Messages V 5 1 7842 VxVM vxconfigd WARNING V 5 1 7842 Cannot open log file log_ filename reason Description The vxconfigd console output log file could not be opened for the given reason Action Create any needed directories or use a different log file path name as described in Logging Error Messages on page 41 V 5 1 7857 VxVM vxconfigd ERROR V 5 1 7857 Cannot start volume volume no valid plexes Description This error indicates that the volume cannot be started because
98. s controlled by the DMP node indicated by the specified device number This happens if a previously disabled path has been repaired the user has reconfigured the DMP database using the vxdct1 1M command or the DMP database has been reconfigured automatically Action None V 5 0 166 VxVM vxio WARNING V 5 0 166 Failed to log the detach of the DRL volume volume Description An attempt failed to write a kernel log entry indicating the loss of a DRL volume The attempted write to the log failed either because the kernel log is full or because of a write error to the drive The volume becomes detached Action Messages about log failures are usually fatal unless the problem is transient However the kernel log is sufficiently redundant that such errors are unlikely to occur If the problem is not transient that is the drive cannot be fixed and brought back online without data loss recreate the disk group from scratch and restore all of its volumes from backups Even if the problem is transient reboot the system after correcting the problem If error messages are seen from the disk driver it is likely that the last copy of the log failed due to a disk error Replace the failed drive in the disk group The log re initializes on the new drive Finally force the failed volume into an active state and recover the data V 5 0 168 VxVM vxio WARNING V 5 0 168 Failure in RAID 5 logging operation Description Indicates
99. s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view If more than one array is connected to the controller suppress the path from DMP using item 5 Prevent multipathing of all disks on a controller by VxvVM and then item 2 Suppress a path from VxVM s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view Chapter 5 Error Messages Z5 Er Understanding Messages V 5 1 7604 VxVM vxconfigd ERROR V 5 1 7604 mode Unrecognized operating mode Description An invalid string was specified as an argument to the m option Valid strings are enable disable and boot Action Supply a correct option argument V 5 1 7809 VxVM vxconfigd WARNING V 5 1 7809 Cannot create device device_path reason Description vxconfigd cannot create a device node either under dev vx dsk or under dev vx rdsk This should happen only if the root file system has run out of inodes Action Remove some unwanted files from the root file system Then regenerate the device node using the command vxdctl enable V 5 1 7829 VxXVM vxconfigd WARNING V 5 1 7829 Cannot exec usr bin rm to remove directory reason Description The given directory could not be removed because the usr bin rm utility could not be executed by vxconfigd This is not a serious error The only side effect of a directory not being removed is that the directory and its contents continue to u
100. se space in the root file system However this does imply that the usr file system is not mounted or on some systems that the rm utility is missing or is not in its usual location This may be a serious problem for the general running of your system Action If the usr file system is not mounted you need to determine how to get it mounted If the rm utility is missing or is not in the usr bin directory restore it 7 VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 7834 VxVM vxconfigd WARNING V 5 1 7834 Cannot fork to remove directory directory reason Description The given directory could not be removed because vxconfigd could not fork in order to run the rm utility This is not a serious error The only side effect of a directory not being removed is that the directory and its contents will continue to use space in the root file system The most likely cause for this error is that your system does not have enough memory or paging space to allow vxconfigd to fork Action If your system is this low on memory or paging space your overall system performance is probably substantially degraded Consider adding more memory or paging space V 5 1 7839 VxVM vxconfigd ERROR V 5 1 7839 cannot open dev vx config reason Description The dev vx config device could not be opened vxconfigd uses this device to communicate with the VERITAS Volume Manager kernel drivers The most likely reason i
101. sk recovery In addition the disk on which the subdisk resides can be replaced without recovery operations being performed In such cases you can perform subdisk recovery using the vxvol recover command For example to recover the stale subdisk in the RAID 5 volume shown in the figure Invalid RAID 5 Volume on page 16 use the following command vxvol g mydg recover r5vol disk05 00 A RAID 5 volume that has multiple stale subdisks can be recovered in one operation To recover multiple stale subdisks use the vxvol recover command on the volume as follows vxvol g mydg recover r5vol VERITAS Volume Manager Troubleshooting Guide Failures on RAID 5 Volumes Recovery After Moving RAID 5 Subdisks When RAID 5 subdisks are moved and replaced the new subdisks are marked as STALE in anticipation of recovery If the volume is active the vxsd command may be used to recover the volume If the volume is not active it is recovered when it is next started The RAID 5 volume is degraded for the duration of the recovery operation Any failure in the stripes involved in the move makes the volume unusable The RAID 5 volume can also become invalid if its parity becomes stale To avoid this occurring vxsd does not allow a subdisk move in the following situations a stale subdisk occupies any of the same stripes as the subdisk being moved the RAID 5 volume is stopped but was not shut down cleanly that is the parity is consid
102. sole and take the actions suggested in the descriptions of those messages V 5 1 10128 VxVM vxassist ERROR V 5 1 10128 No more space in disk group configuration Description There is no more space in the disk group s configuration database for VxVM object records Action Copy the contents of several volumes to another disk group and then delete the volumes from this disk group To avoid the problem in the future do not create more than a few hundred volumes in a disk group or specify a larger size for the private region when adding disks to a new disk group Chapter 5 Error Messages 81 Understanding Messages wr 82 VERITAS Volume Manager Troubleshooting Guide Symbols cmdlog file 29 translog file 31 etc vx cbr bk diskgroup dgid dgid binconfig file 36 dgid cfgrec file 36 dgid diskinfo file 36 dgid dginfo file 36 etc vx log logging directory 29 31 lib svc method vxvm sysboot file 42 var adm configd log file 41 var adm syslog syslog log file 42 ACTIVE plex state 3 ACTIVE volume state 12 BADLOG plex state 11 boot disks recovery 19 CLEAN plex state 3 client ID in command logging file 30 in transaction logging file 32 cmdlog file 29 commands associating with transactions 33 logging 29 configuration backing up for disk groups 35 36 backup files 36 resolving conflicting backups 38 restoring for disk groups 35 37 data loss RAID 5 9 degraded mode RAID 5 10 DEGRADED volume st
103. striped the lines starting with sd represent the stripes One of the stripe areas is located on a failed disk This disk is no longer valid so the plex named v02 01 has a state of NODEVICE Since this is the only plex of the volume the volume is invalid and must be removed If a copy of v02 exists on the backup media it can be restored later Keep a record of the volume name and length of any volume you intend to restore from backup Remove invalid volumes such as v02 using the following command vxedit r rm v02 A volume that has one mirror on a failed disk can also have other mirrors on disks that are still valid In this case the volume does not need to be restored from backup since the data is still valid on the valid disks The output of the vxprint th command for a volume with one plex on a failed disk disk01 and another plex on a valid disk disk02 is similar to the following V NAME RVG VSET COKSTATESTATE 1ENGTH READPOL PREFPLEX UTYPE PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL WIDMODE SD NAME PLEX DISK DISKOFFSLENGTH COL OFFDEVICE MODE v v03 DISABLEDACTIVE 0720 SELECT fsgen pl v03 01 v03 DISABLEDACTIVE 30720 CONCAT RW sd disk02 01 v03 01 disk01 620544 30720 0 c1lt5d5 ENA pl v03 02 v03 DISABLEDNODEVICE30720 CONCAT RW sd disk01 04 v03 02 disk03 262144 30720 0 c1lt5d6 DIS This volume has two plexes v03 01 and v03 02 The first plex v
104. subdisk subdisk failed in plex plex in volume volume Description The kernel has detected a subdisk failure which may mean that the underlying disk is failing Action Check for obvious problems with the disk such as a disconnected cable If hot relocation is enabled and the disk is failing recovery from subdisk failure is handled automatically V 5 1 480 VxXVM vxconfigd ERROR V 5 1 480 Cannot reset VxVM kernel reason Description The r reset option was specified to vxconfigd but the VxVM kernel drivers could not be reset The most common reason is A virtual disk device is open This implies that a VxVM tracing or volume device is open Action If you want to reset the kernel devices track down and kill all processes that have a volume or VERITAS Volume Manager tracing device open Also if any volumes are mounted as file systems unmount those file systems Any reason other than A virtual disk device is open does not normally occur unless there is a bug in the operating system or in VxVM Chapter 5 Error Messages 53 v Understanding Messages V 5 1 484 VxVM vxconfigd ERROR V 5 1 484 Cannot start volume volume no valid complete plexes Description These errors indicate that the volume cannot be started because the volume contains no valid complete plexes This can happen for example if disk failures have caused all plexes to be unusable It can also happen as a result of actions that caus
105. t command on systems that have dual ported disks Clearing the locks allows those disks to be accessed at the same time from multiple hosts and can result in corrupted data An import operation fails if some disks for the disk group cannot be found among the disk drives attached to the system Disk for disk group not found Disk group has no valid configuration copies The first message indicates a recoverable error The second message indicates a fatal error that requires hardware repair or the creation of a new disk group and recovery of the disk group configuration and data If some of the disks in the disk group have failed you can force the disk group to be imported with this command vxdg f import diskgroup Caution Be careful when using the f option It can cause the same disk group to be imported twice from different sets of disks causing the disk group to become inconsistent These operations can also be performed using the vxdiskadn utility To deport a disk group using vxdiskadnm select menu item 9 Remove access to deport a disk group To importa disk group select item 8 Enable access to import a disk group The vxdiskadm import operation checks for host import locks and prompts to see if you want to clear any that are found It also starts volumes in the disk group V 5 1 663 VxVM vxconfigd WARNING V 5 1 663 Group group Duplicate virtual device number s Volume volume remapped from major m
106. t host will yield this error Action If you want to import both disk groups then rename the second disk group on import See the vxdg 1M manual page for information on how to use the import operation to rename a disk group Chapter 5 Error Messages 59 Understanding Messages V 5 1 577 VxVM vxconfigd WARNING V 5 1 577 Disk group group Disk group log may be too small Log size should be at least number blocks Description The log areas for the disk group have become too small for the size of configuration currently in the group This message only occurs during disk group import it can only occur if the disk was inaccessible while new database objects were added to the configuration and the disk was then made accessible and the system restarted This should not normally happen without first displaying a message about the database area size Action Reinitialize the disks in the group with larger log areas Note that this requires that you restore data on the disks from backups See the vxdisk 1M manual page To reinitialize all of the disks detach them from the group with which they are associated reinitialize and re add them Then deport and re import the disk group to effect the changes to the log areas for the group V 5 1 579 VxVM vxconfigd ERROR V 5 1 579 Disk group group Errors in some configuration copies Disk disk copy number Block number reason Description During a failed disk group impo
107. t lost data Data may need to be restored and failed media may need to be repaired or replaced Depending on the type of object failing and on the type of recovery suggested for the object type an appropriate recovery operation may be necessary V 5 0 4 VXVM vxio WARNING V 5 0 4 Plex plex detached from volume volume Description An uncorrectable error was detected by the mirroring code and a mirror copy was detached Action To restore redundancy it may be necessary to add another mirror The disk on which the failure occurred should be reformatted or replaced V 5 0 34 VxVM vxdmp NOTICE V 5 0 34 added disk array disk_array_serial_number Description A new disk array has been added to the host Action None Chapter 5 Error Messages 45 or Understanding Messages V 5 0 35 VxVM vxdmp NOTICE V 5 0 35 Attempt to disable controller controller_name failed Rootdisk has just one enabled path Description An attempt is being made to disable the one remaining active path to the root disk controller e Action The path cannot be disabled V 5 0 55 VxVM vxio WARNING V 5 0 55 Cannot find device number for boot path vxvm vxdmp WARNING V 5 0 55 Cannot find device number for boot_path Description The boot path retrieved from the system PROMs cannot be converted to a valid device number Action Check your PROM settings for the correct boot string If a target driver such as an ATF coexists with VxVM
108. tc vfstab file to correct the entry for the usr file system In case 2 either boot with all drives in the offending version of the boot disk group turned off or import and rename see vxdg 1M the offending boot disk group from another host If you turn off drives run the following command after booting vxdg flush bootdg This updates time stamps on the imported version of the boot disk group bootdg which should make the correct version appear to be the more recently accessed If this does not correct the problem contact VERITAS Technical Support Chapter 5 Error Messages 73 Understanding Messages V 5 1 5591 VxVM vxconfigd WARNING V 5 1 5591 library and vxconfigd disagree on existence of client number Description This warning may safely be ignored Action None required V 5 1 7265 VxVM vxconfigd NOTICE V 5 1 7265 Unable to resolve duplicate diskid Description When VxVM detects disks with duplicate disk IDs unique internal identifiers VxVM attempts to select the appropriate disk using logic that is specific to an array If a disk can not be selected VxVM does not import any of the duplicated disks into a disk group In the rare case when VxVM cannot make the choice you must choose which duplicate disk to use Note In releases prior to 3 5 VxVM selected the first disk that it found if the selection process failed FromVxVM 3 5 onward the default behavior is to avoid the selection of
109. tem administrator needs to handle the task of problem solving using the diagnostic messages that are returned from the software The following sections describe error message numbers and the types of error message that may be seen and provide a list of the more common errors a detailed description of the likely cause of the problem together with suggestions for any actions that can be taken Messages have the following generic format product component severity message_number message_text For VERITAS Volume Manager the product is set to VxVM The component can be the name of a kernel module or driver such as vxdmp a configuration daemon such as vxconfigd ora command such as vxassist Messages are divided into the following types of severity in decreasing order of impact on the system PANIC A panic is a severe event as it halts a system during its normal operation A panic message from the kernel module or from a device driver indicates a hardware problem or software inconsistency so severe that the system cannot continue The operating system may also provide a dump of the CPU register contents and a stack trace to aid in identifying the cause of the panic The following is an example of such a message VxVM vxio PANIC V 5 0 239 Object association depth overflow Chapter 5 Error Messages 43 Understanding Messages Note For full information about saving system crash information see the Solaris System Administatio
110. that was encountered If the disk group that was disabled is the boot disk group the following additional error is displayed VxVM vxconfigd ERROR V 5 1 7749 All transactions are disabled This additional message indicates that vxconfigd has entered the disabled state which makes it impossible to change the configuration of any disk group not just the boot disk group Chapter 5 Error Messages 79 Or Understanding Messages Action If the underlying error resulted from a transient failure such as a disk cabling error then you may be able to repair the situation by rebooting Otherwise the disk group configuration may have to be recreated by using the procedures given in Restoring a Disk Group Configuration on page 37 and the contents of any volumes restored from a backup Failure of the boot disk group may require reinstallation of the system if your system uses a root or usr file system that is defined on a volume V 5 1 7935 VxVM vxconfigd ERROR V 5 1 7935 Disk group group update failed reason Description I O failures have prevented vxconfigd from updating any active copies of the disk group configuration This usually implies a large number of disk failures This error will usually be followed by the error VxVM vxconfigd ERROR V 5 1 7934 Disk group group Disabled by errors Action If the underlying error resulted from a transient failure such as a disk cabling error then you may be able to repa
111. tion The specified disk has become unusable Action Do not include the disk in any disk group move split or join operation until it has been replaced or repaired V 5 1 2928 VxVM vxdg ERROR V 5 1 2928 diskgroup Configuration too large for configuration copies Description The disk group s configuration database is too small to hold the expanded configuration after a disk group move or join operation Action None V 5 1 2933 VxVM vxdg ERROR V 5 1 2933 diskgroup Cannot remove last disk group configuration copy Description The requested disk group move split or join operation would leave the disk group without any configuration copies Action None The operation is not supported V 5 1 3008 VxVM vxdg ERROR V 5 1 3008 object Name conflicts with imported diskgroup Description The target disk group of a split operation already exists as an imported disk group Action Choose a different name for the target disk group VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 1 3091 VxVM vxdg ERROR V 5 1 3091 diskname Disk not moving but subdisks on it are Description Some volumes have subdisks that are not on the disks implied by the supplied list of objects Action Use the o expand option to vxdg 1istmove to produce a self contained list of objects V 5 1 3212 VxXVM vxconfigd ERROR V 5 1 3212 Insufficient DRL log size logging is disabled Descri
112. ume 1 VERITAS Volume Manager Troubleshooting Guide Understanding Messages In case 2 either boot with all drives in the offending version of the boot disk group turned off or import and rename see vxdg 1M the offending boot disk group from another host If you turn off the drives run the following command after booting vxdg flush bootdg This updates time stamps on the imported version of the specified boot disk group bootdg which should make the correct version appear to be the more recently accessed If this does not correct the problem contact VERITAS Technical Support V 5 1 1171 VxXVM vxconfigd ERROR V 5 1 1171 Version number of kernel does not match vxconfigd Description The release of veconfigd does not match the release of the VERITAS Volume Manager kernel drivers This should happen only as a result of upgrading VxVM and then running vxconfigd without a reboot Action Reboot the system If that does not cure the problem re add the VxVM packages V 5 1 1185 VxVM vxconfigd NOTICE V 5 1 1185 Volume volume entering degraded mode Description Detaching a subdisk in the named RAID 5 volume has caused the volume to enter degraded mode While in degraded mode performance of the RAID 5 volume is substantially reduced More importantly failure of another subdisk may leave the RAID 5 volume unusable Also if the RAID 5 volume does not have an active log then failure of the system may leave th
113. very 19 root file system damaged 20 stale parity 9 states displaying for volumes and plexes 2 subdisks marking as non stale 17 recovering after moving for RAID 5 15 recovering stale RAID 5 14 stale starting volume 17 SYNC volume state 11 13 syslog error log file 42 system reinstalling 20 system failures 9 Technical assistance vii transactions associating with commands 33 logging 31 translog file 31 TUTILO field clearing MOVE flag 18 V 5 0 106 46 V 5 0 108 47 V 5 0 110 47 V 5 0 111 47 V 5 0 112 47 V 5 0 144 48 V 5 0 145 48 V 5 0 146 48 V 5 0 147 48 V 5 0 148 49 V 5 0 166 49 V 5 0 168 49 V 5 0 181 50 V 5 0 194 50 V 5 0 196 50 V 5 0 2 45 V 5 0 207 50 V 5 0 216 51 V 5 0 237 51 V 5 0 243 51 V 5 0 249 51 V 5 0 251 52 V 5 0 252 52 V 5 0 258 52 V 5 0 281 53 V 5 0 34 45 V 5 0 35 46 V 5 0 386 53 V 5 0 4 45 V 5 0 55 46 V 5 0 64 46 V 5 1 1049 64 V 5 1 1063 64 V 5 1 1171 65 V 5 1 1185 65 V 5 1 123 36 V 5 1 1236 16 V 5 1 1237 16 V 5 1 1589 65 V 5 1 2031 67 V 5 1 2274 67 VERITAS Volume Manager Troubleshooting Guide V 5 1 2290 67 V 5 1 7842 78 V 5 1 2353 68 V 5 1 7857 78 V 5 1 2829 68 V 5 1 7909 78 V 5 1 2841 69 V 5 1 7910 79 V 5 1 2860 69 V 5 1 7916 79 V 5 1 2866 69 V 5 1 7934 79 V 5 1 2879 69 V 5 1 7935 80 V 5 1 2907 18 70 V 5 1 8063 80 V 5 1 2911 70 V 5 1 8064 80 V 5 1 2928 70 V 5 1 8259 81 V 5 1 2933 70 V 5 1 923 63 V 5 1 3008 70 V 5 1 946 63 V 5 1 3091 71 volume kernel states V 5 1 3
114. w log plex and attach it to the volume wr 50 VERITAS Volume Manager Troubleshooting Guide Understanding Messages V 5 0 216 VxVM vxio WARNING V 5 0 216 mod_install returned errno Description A call made to the operating system mod_instal11 function to load the vxio driver failed Action Check for additional console messages that may explain why the load failed Also check the console messages log file for any additional messages that were logged but not displayed on the console V 5 0 237 VXVM vxio WARNING V 5 0 237 object subdisk detached from RAID 5 volume at column column offset offset Description A subdisk was detached from a RAID 5 volume because of the failure of a disk or an uncorrectable error occurring on that disk Action Check for other console error messages indicating the cause of the failure Replace a failed disk as soon as possible V 5 0 243 VxVM vxio WARNING V 5 0 243 Overlapping mirror plex detached from volume volume Description An error has occurred on the last complete plex in a mirrored volume Any sparse mirrors that map the failing region are detached so that they cannot be accessed to satisfy that failed region inconsistently Action The message indicates that some data in the failing region may no longer be stored redundantly V 5 0 249 VXVM vxio WARNING V 5 0 249 RAID 5 volume entering degraded mode operation Description An uncorrectable error has forced a subd
115. with cabling or power V 5 1 809 VxXVM vxplex ERROR V 5 1 809 Plex plex in volume volume is locked by another utility Description The vxplex command fails because a previous operation to attach a plex did not complete The vxprint command should show that one or both of the temporary and persistent utility fields TUTILO and PUTILO of the volume and one of its plexes are set Action If the vxtask list command does not show a task running for the volume use the vxmend command to clear the TUTILO and PUTILO fields for the volume and all its components for which these fields are set vxmend g diskgroup clear all volume plex V 5 1 946 VxVM vxconfigd FATAL ERROR V 5 1 946 bootdg cannot be imported during boot Description Usually means that multipathing is misconfigured Action See V 5 1 7265 on page 74 for possible causes and solutions Chapter 5 Error Messages 63 Fr Understanding Messages V 5 1 1049 VxVM vxconfigd ERROR V 5 1 1049 System boot disk does not have a valid rootvol plex Please boot from one of the following disks DISK MEDIA DEVICE BOOT COMMAND diskname device boot vx diskname Description The system is configured to use a volume for the root file system but was not booted on a disk containing a valid mirror of the root volume Disks containing valid root mirrors are listed as part of the error message A disk is usable as a boot disk if there is a
116. ymbolic link to a directory whose location depends on the operating system If required you can redefine the directory which is linked If you want to preserve the settings of the vxcmdlog utility you must also copy the settings file cmdlog to the new directory 29 wy Logging Commands Note The cmdlog file is a binary and should not be edited The size of the command log is checked after an entry has been written so the actual size may be slightly larger than that specified When the log reaches a maximum size the current command log file cmdlog is renamed as the next available historic log file cmdlog number where number is an integer from 1 up to the maximum number of historic log files that is currently defined and a new current log file is created A limited number of historic log files is preserved to avoid filling up the file system If the maximum number of historic log files has been reached the oldest historic log file is removed and the current log file is renamed as that file Each log file contains a header that records the host name host ID and the date and time that the log was created The following are sample entries from a command log file 0 2329 Wed Feb 12 21 19 31 2003 usr sbin vxdctl mode 17051 2635 Wed Feb 12 21 19 33 2003 usr sbin vxdisk q o alldgs list 0 2722 Wed Feb 12 21 19 34 2003 etc vx diag d vxprivutil dumpconfig dev vx rdmp Disk_4s2 26924 3001 Thu Feb 13
117. you are using the correct split procedure If you know which LUNs you want to use choose which path to exclude and then either edit the file etc vx vxvm exclude or if vxconfigd is running select item 1 Suppress all paths through a controller from VxVM s view oritem2 Suppress a path from VxVM s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view Case 3 If disks have become duplicated using the dd command or any other disk copying utility choose which set of duplicate disks you want to exclude and then either edit the file etc vx vxvm exclude or if vxconfigd is running select item 1 Suppress all paths through a controller from VxVM s view oritem2 Suppress a path from VxVM s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view Case 4 Solaris 9 only After suppressing one path to a multipathed array from DMP you then have suppressed that path from VxVM using item 1 suppress all paths through a controller from VxVM s view from vxdiskadm option 17 Prevent multipathing Suppress devices from VxVM s view If all bootdg disks are from that array vxconfigd does not start You may also see errors V 5 0 64 V 5 1 946 and V 5 1 2841 If only one array is connected to the controller suppress the path from DMP using item 5 Prevent multipathing of all disks on a controller by VxvVM and then item 1 Suppress all paths through a controller from VxVM

Volume Manager Troubleshooting Guide

Contents

Download Pdf Manuals

Related Search

Related Contents