Home

IBM Tivoli Monitoring: Troubleshooting Guide

1. UNIX and Linux based Component Windows systems Tivoli Enterprise Monitoring cms exe cms Server kdsmain exe kdsmain Tivoli Enterprise Portal KfwServices exe KfwServices Server UNIX agent N A kuxagent Linux agent N A klzagent Windows agent kntcma exe N A Universal agent kuma610 exe kuma610 Log Alert agent N A kulagent Warehouse proxy agent khdxprto exe khdxprtj Summarization and Pruning ksy610 exe ksy610 Agent Eclipse help server kkfhelpsvr exe kkfstart sh for the java process search for kf in the process name Copyright IBM Corp 2005 2010 233 234 IBM Tivoli Monitoring Troubleshooting Guide Appendix B Environment variables This appendix lists IBM Tivoli Monitoring environment variables that you can customize Common environment variables The following table lists the environment variables that are common to all components Table 52 Common environment variables Variable Value type Useful for KDC_DEBUG Yor N Default is N Diagnosing communications or connectivity problems between the Tivoli Enterprise Portal Server to the Tivoli Enterprise Monitoring Server KDH_DEBUG YorN Default is N Diagnosing connectivity problem with integrated web server KDH_SERVICEPOINT Determining the service point KBB_RAS1 Trace specification string See each specific environment variable section below KBB_RASILOG Determining
2. Restoring the original database contents The migrate clean bat utility cleans the contents of the database Use the migrate clean bat file with caution You must backup the database before running the migrate clean bat file or you lose all customization of the database When you restart the Tivoli Enterprise Monitoring Server the database is restored to its original state after installation This is a quick way to reset the database back to its original state After running this bat file and restarting the Tivoli Enterprise Monitoring Server server the original content provided by IBM Tivoli Monitoring is restored to the database If you modify your password or if it expires The database requires the following user IDs Copyright IBM Corp 2005 2010 201 db2admin This ID is added when you install the database and is required by the product installer when you configure the Tivoli Enterprise Portal Server data source TEPS This ID is added during installation for creating the Tivoli Enterprise Portal Server data source If the Windows Local Security Settings are enabled for long or complex passwords ensure your password meets those syntax requirements for these IDs If your Windows environment requires you to change passwords regularly do the following to change the Tivoli Enterprise Portal Server database user account password Note The following instructions do not apply in UNIX based systems i On the comp
3. Ports 3661 and 1920 must be available Run the test sh command on UNIX based systems or the test cmd on Windows to determine if there is a communication problem between IBM Tivoli Enterprise Console server and Tivoli Enterprise Monitoring Server Refer to Chapter 6 Connectivity troubleshooting 5 I page 107 for additional information about solving connectivity problems 220 IBM Tivoli Monitoring Troubleshooting Guide Table 47 Situation event was not acknowledged or closed continued Problem Resolution There a firewall between Tivoli Enterprise Monitoring Server and Tivoli Enterprise Console that is blocking communication over certain ports You might need to include the IP addresses of other IBM Tivoli Monitoring components in the access control for the firewall Refer to Chapter 6 Connectivin aisi np lia additional information about solving connectivity problems TEC_ITM_OM Situation_Sync_Error events occur TEC_ITM_OM_Situation_Sync_Error events occur if there is a Tivoli Enterprise Console server communication problem or if you need to increase the rules cache Table 48 Reasons for the TEC_ITM_OM_Situation_Sync_Error event Problem Resolution IOException is generated There is no connectivity with SOAP Logon Validation Failure is returned by the SOAP server SOAP server Fatal Error is returned by the The connection to the SOAP server r
4. 137 Corba user exception is included in the Tivoli Enterprise Portal Server log when creating situations L22137 Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 139 Troubleshooting Tivoli Enterprise Monitoring Server problems on distributed systems 139 Exposure of passwords in the clear 139 Receive a seeding failed message 139 High monitoring server CPU after restarting with Warehouse Proxy Agents configured 139 Upgrade inconsistency between the iii and Object dialogs 140 Attribute groups started fax collection on ihe managed systems should not be available on the monitoring server list 140 To decrypt a password KDS_VALIDATE_EXT Y is required 141 Remote Tivoli Enterprise Monitoring Server consumes high CPU when large number of agents connect 141 Unable to start the Tivoli Enterprise Monitoring Server after the kdsmain process is terminated abnormally 141 THRESHOLDS XML aiid Tivoli Enterprise Monitoring Server table not cleaned when managed system override is removed 142 Situations fail to trigger for attributes by applying group function 142 Tivoli Enterprise Monitoring Server application support will complete all seeding functions but may crash as the program is exiting 142 tacmd login fails when monitoring server is configured with LDAP authentication 143 Some agents are displa
5. Note Create a backup of the registry before editing it Expand the key HKEY_LOCAL MACHINE registry key 10 11 Expand the SOFTWARE registry key Expand the Candle registry key and record any sub keys that are present If the Candle key does not exist proceed to step Expand the OMEGAMON registry key under the Candle key and record the content of the OMEGAMON key values Delete the Candle registry key and all sub keys On Windows XP you can right click the Candle registry key and click Delete Close the Registry Editor Open the Windows Explorer and find the IBM Tivoli Monitoring installation location on your system The default value is C IBM ITM Delete this directory and all subdirectories Remove the IBM Tivoli Monitoring bookmark from the Start menu a Click Start from the Windows desktop to display the Start menu items b Click Programs c Right click IBM Tivoli Monitoring to display the bookmark menu options d Click Delete to remove the IBM Tivoli Monitoring bookmark from the Windows desktop start menu You can now install IBM Tivoli Monitoring IBM Tivoli Monitoring Troubleshooting Guide Removing a failed upgrade To remove a failed upgrade first ensure that there is no entry in the Add and Remove Programs tool for the new component you are attempting to install If there is an entry use that entry to uninstall the product If there is no entry use the following steps to remove the failed upg
6. kmspdcol help FUNCTION THIS PROGRAM IS INTENDED TO COLLECT DIAGNOSTIC INFORMATION FROM THE COMPUTER ON WHICH IT IS EXECUTED COLLECTED DATA INCLUDES SYSTEM CONFIGURATION NETWORK INFORMATION OUTPUT FROM THE SPECIFIED JOB SYNTAX sae KMSPDCOL ccs HELP prora paS 33 JOBOUT DSN s isise o JOBNAME JNAME 3 e JOBNO JNUM rons gt gt i RHILEV X RVHILEV X SYS X OPERANDS HELP REQUESTS DISPLAY OF THIS HELP INFORMATION JOBOUT DSN IS THE NAME OF THE DATASET CONTAINING THE COMPLETE OUTPUT FROM THE ITM JOB JOBNAME JNAME IS THE NAME OF THE ITM JOB IN THE JES SPOOL JOBNO JNUM IS THE JES JOB NUMBER OF THE ITM JOB REQUIRED IF THERE ARE MULTIPLE JOBS WITH THE SAME NAME NOTE THAT SDSF IS REQUIRED TO SUPPORT THE USE OF JOBNAME JOBNO IF YOU DO NOT HAVE SDSF YOU MUST USE JOBOUT RHILEV SPECIFIES THE HLQS OF THE NON VSAM DATASETS IN THE ITM RTE RVHILEV SPECIFIES THE HLQS OF THE VSAM DATASETS IN THE ITM RTE SYS SPECIFIES THE NAME OF THE ITM RTE RHILEV RVHILEV AND SYS WILL NORMALLY BE EXTRACTED FROM THE JOB OUTPUT BUT MAY BE SPECIFIED IF THE PROGRAM IS UNABLE TO FIND THEM OR IF YOU WISH TO OVERRIDE THE EXTRACTED VALUES Note The output data is placed in the dataset PDCOLLCT DAT under the high level qualifier designated by the PREFIX option of the TSO PROFILE command See the PROFILE command description in the TSO E Command Reference for details To ma
7. 2 Identify the SQL query that caused the high CPU usage issues from the collector logs You can identify the SQL query that caused the high CPU usage issue from ITM Oracle Agent logs or the Oracle tools Use the following procedure to identify the problematic cursors from ITM Oracle Agent logs Open the collector logs and find CFE1645 messages The messages show the return time of each cursors For example CFE1645T 165929 Time 2008 06 06 16 59 29 collected records in 6 seconds The default timeout value of ITM Oracle Agent is 45 seconds If it takes more than 45 seconds it might cause a timeout problem and Open Probe pipe error will be reported in the collector log CFE1645T 170246 Time 2008 06 06 17 02 46 collected records in 203 seconds When a timeout happens review the previous cursor that executed before this message For example PDR3000T 170002 Deleting 1 rows for cursor DB6 RPF0300T 170002 Doing prep_1_fet for cursor DB6 ORU0085I 170002 ORU0090I 170002 Starting new SQL query ORU0095I 170002 lt SELECT RULE COUNT EXTENTS FROM SYS DBA_EXTENTS gt ORU0085I 170002 CAT1610I 170213 Dump of row 1 UPXO100T 000 20202020 20202020 20202032 34313135 24115 The previous cursor DB6 took about 2 minutes and 11 seconds to return data causing the performance problem Were you able to ide
8. Data loss prevention This section provides information about utilities you can use to back up and restore Tivoli Enterprise Portal Server databases Backing up the database for recovery purposes You can use the following utilities to back up the database Table 44 Utilities for backing up the database migrate The migrate export bat utility backs up the entire database by writing its export bat contents as insert statements to a flat file called saveexport sq located in install _dir cnps sqllib It can also be used to move the contents of the database from one database instance to another You might use this utility to move the contents of the database from one windows server to another migrate This utility is used to read the contents of the saveexport sql file created import bat in the migrate export process and insert them back into the database This utility can be used to recover the database It reads the contents in the install_dir cnps sqllib saveexport sql and rebuilds the database tables and contents You can also use the migrate import bat to move the contents of the database to another windows server running the database 1 Run migrate export bat 2 Copy the saveexport sql file from the old Tivoli Enterprise Portal Server to the new Tivoli Enterprise Portal Server into the install_dir cnps sqllib directory 3 Run migrate import bat to read and build the database tables and contents on the new machine
9. entry_point Represents the name of the entry point An IBM Software Support representative can tell you what value to specify UNIT Indicates that the trace is to look for a match between the compilation unit dispatched and the fully or partially qualified compilation unit specified on the RAS1 statement A match results in a trace entry unit_name Represents the name of the compilation unit In most instances this name defines the component that is being traced The value is likely to be the three character component identifier for the monitoring agent KHL for OMEGAMON z OS Management Console IBM Tivoli Monitoring Troubleshooting Guide class One of the same values specified for global_class but because of its position inside the parentheses narrowed in scope to apply only to the unit_name specified Note The default setting for monitoring agents on z OS is KBB_RAS1 ERROR meaning that only error tracing is enabled You can specify any combination of UNIT COMP and ENTRY keywords No keyword is required However the RAS1 value you set with the global class applies to all components For more information on setting RAS1 tracing on z OS system see your individual monitoring agent s user s guide Dynamically modify trace settings for an IBM Tivoli Monitoring component You can access the Tivoli Enterprise Monitoring Server Tivoli Enterprise Portal Server almost all of the agents and other IBM Tivoli Monitoring compone
10. ip6addr n To remove interfaces from KDEB_INTERFACELIST ip4addr 1 ip4addr n consideration KDEB_INTERFACELIST_IPV6 ip6addr 1 ip6addr n To add interfaces for consideration KDEB_INTERFACELIST ip4addr 1 ip4addr n KDEB_INTERFACELIST_IPV6 ip6addr 1 ip6addr n where ip4addr Specifies either ip6addr Specifies either a symbolic network name or a raw form dotted decimal network address a symbolic network name or a raw form colon separated hex digit network address Note The plus sign must stand alone 116 IBM Tivoli Monitoring Troubleshooting Guide Chapter 7 Tivoli Enterprise Portal troubleshooting This chapter describes problem you might experience with the Tivoli Enterprise Portal Client allows you to save a situation with an invalid character It is possible to create and save a situation with a wildcard character such as an asterix However a situation created with this character is not valid Using an administrator name with non latin1 characters cannot log onto the Tivoli Enterprise Portal If you log onto a Windows system with an administrator name with non latin1 characters you cannot log onto the portal server by either the Tivoli Enterprise Portal desktop client or the Tivoli Enterprise Portal browser client Set the cnp browser installdir Tivoli Enterprise Portal parameter to a path that does not contain any non latin characters by completing t
11. Chapter 14 Event synchronization troubleshooting 213 Table 45 Troubleshooting when situations are not forwarded continued Problem Resolution Events are cached because the 1 Check the cache files to determine if events are cached on the Tivoli target IBM Tivoli Enterprise Enterprise Monitoring Server The cache files are located in the TECLIB Console server is not currently directory on Tivoli Enterprise Monitoring Server ae e Windows IBM ITM cms TECLIB e UNIX based systems opt IBM ITM tables hubname TECLIB where hubname Specifies the name of the hub monitoring server 2 Run the wstartesvr command to restart the IBM Tivoli Enterprise Console server from a Tivoli sourced environment 3 Run the wstatesvr S server command on the IBM Tivoli Enterprise Console server to check the status of an event server where S server Specifies the name of an event server in name registry format The default server is the local event server To indicate a remote server specify EventServer tmr where tmr is the name of a Tivoli region Refer to the IBM Tivoli Enterprise Console documentation for more information about checking the status of the IBM Tivoli Enterprise Console server Note On z OS and 64 bit native platforms a physical event cache is not available from the TECLIB directory The event cache is in a memory cache See Trace logging on page 35 Enterprise Console forwarding for mor
12. Chapter 15 Tivoli Common Reporting troubleshooting 231 The generated report displays the message The requested data is not available In this scenario you complete the report parameters dialog to generate a report and click Submit After clicking Submit you see a blank report page with no graphed or tabular information and the error message The requested data is not available is displayed in the message area When you check version levels you find that Tivoli Data Warehouse is your data source but none of the requested columns in the Tivoli Enterprise Portal attribute table contain data This indicates that no data is being collected in Tivoli Data Warehouse Verify this by querying the same table in the database or by requesting data from the matching workspace for this time period in the Tivoli Enterprise Portal If for example your installation stopped collecting historical data for this report 8 days ago and you query the last 7 day no data will be returned If the managed resource for example a TN3270 Server or a CICS Region were taken offline then no data can be collected To address the time period issue expand the time period for your query To correct the resource availability issue ensure that the managed resource is online and your agent is collecting data for it The solution might require changes to the persistent datastore the z OS repository for short term historical data Those changes should be mad
13. Classes EVERYT EVERYE EVERYU 4558D8CC 0034 1114 kpxreqds cpp 661 determineTargetsFromNodelist Active RAS1 Classes EVERYT EVERYE EVERYU 4558D8CC 0035 1114 kpxreqds cpp 661 determineTargetsFromNodelist Entry 4558D8CC 0036 1114 kpxreqds cpp 669 determineTargetsFromNodelist Exit 4558D8CC 0037 1114 kpxreqds cpp 821 determineTargetsFromAccessList Active RAS1 Classes EVERYT EVERYE EVERYU 4558D8CC 0038 1114 kpxreqds cpp 821 determineTargetsFromAccessList Entry 4558D8CC 0039 1114 kpxreqds cpp 837 determineTargetsFromAccessList Calling KFA GetAccessListNodes for NT_Paging_File Critical 5140 4558D8CC 003A 1114 kpxreqds cpp 852 determineTargetsFromAccessList Node 0 Primary KATE NT 4558D8CC 003B 1114 kpxreqds cpp 891 determineTargetsFromAccessList Deleting NodeRecEntry 0 node entries 0x1B63B90 next OxNULL ptr_next OxNULL 4558D8CC 003C 1114 kpxreqds cpp 898 determineTargetsFromAccessList Exit 3 Did the agent return data On the monitoring server set this trace level UNIT kpxrpcrq ERROR STATE to show the number of rows returned by each agent 3A933B00 24A827C0 154 kpxrpcrq cpp 547 IRA_NCS_ Sample Rcvd 1 rows sz 448 tbl NTLOGINFO req NT_Log_Space Low lt 4294706777 761 gt node lt Primary NODE1 NT Yes See step 3 on page 25 No Is the situation authored correctly At the agent trace UNIT kdsfilt all Yes The problem might be related to the monitoring agent See the Troubleshooting appendi
14. Complete the following steps to address this issue 1 Stop the hub monitoring server in your environment 2 Make a configuration change that would alter the total number of Warehouse Proxy Agent Global Location Broker entries as described in items listed in the problem explanation This might include the following steps a Stopping one of the running Warehouse Proxy Agents b Configuring and activating an additional Warehouse Proxy Agent c Adding or removing one or more network protocols for example IP PIPE IP from a Warehouse Proxy Agent configuration 3 Restart the hub monitoring server 4 Restart the modified Warehouse Proxy Agents Upgrade inconsistency between the History and Object dialogs The historical collections created in previous versions of IBM Tivoli Monitoring are not eligible to be members of a group in a later version of the software Attribute groups started for collection on the managed systems should not be available on the monitoring server list Attribute groups can be started for collection on either the managed systems or the monitoring server but not for both from the history collection configuration window When a collection setting that is started for collection on the managed system is grouped in a historical group and then the historical group is distributed to a monitoring server the collection is collected against the managed system and not the monitoring server IBM Tivoli Monitoring Trou
15. Ensure that the system sending SNMP traps to IPv6 destinations can resolve its own IPv6 address The system where the trap emitting agent is running must have a valid IPv6 address associated with its own local hostname If the DNS configuration has not been updated for IPv6 then it is necessary to modify the etc hosts file or etc resolv conf file on the agent system in order to associate the local hostname with a valid local IPv6 address i e not a loopback or link local address Situation overrides cannot be used to disable situations on specific systems at specific times Dynamic thresholding has a restriction on the override of situations with calendar entries Situation overrides can be used to change the conditions under which situations run on specific systems at specific times However they cannot be used to disable a situation on specific systems at specific times To override the time a specific situation runs the situation must include a time attribute for example Local_Time Day_Of_Week and then you can use an override to change the value of the time attribute that causes the situation to run Situation or calendar name in thresholds xml file appears incorrect If situation and calendar overrides are created with names that do not conform to the naming convention the names will be a randomly generated name and be displayed in the xml file as such The naming convention includes the following conditions Chapter 10 Mo
16. IBM Tivoli Monitoring Version 6 2 2 Fix Pack 2 Revised May 2010 Troubleshooting Guide a x uz w m IBM Tivoli Monitoring Version 6 2 2 Fix Pack 2 Revised May 2010 Troubleshooting Guide all yet Note Before using this information and the product it supports read the information in Notices on page 255 This edition applies to version 6 2 2 Fix Pack 2 of IBM Tivoli Monitoring product number 5724 C04 and to all subsequent releases and modifications until otherwise indicated in new editions Copyright IBM Corporation 2005 2010 US Government Users Restricted Rights Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp Contents Chapter 1 Introduction to troubleshooting Sources of troubleshooting information Problem classification Finding release notes Viewing Technotes Creating an e mail subscription to Technotes Chapter 2 Collecting and DIRE troubleshooting data Appropriate IBM Tivoli Monitoring RASI trace output P Be ee ee E OE Snapcore files Core files drWatson dumps aud logs KxxCMA RAS files Other important information NNNEH_ E di N NNOO KfwSQLClient utility Clearing the JAR cache Using the UAGENT ee DPLOG ACTION workspace pdcollect tool rasllog tool Log and Trace Analyzer Tool Backspace Check utility Build TEPS Database utility IBM Tivoli Monitoring Operations Lo
17. KFW_CMW_EVENT_SLEEP 5 Desktop client performs poorly after installing Language Packs for IBM Tivoli Monitoring V6 1 Some configuration settings for the IBM Tivoli Monitoring Warehouse database can cause performance degradation Perform the following steps from a db2cmd prompt to correct the DB2 settings 1 Check the Health Monitor configuration db2 get dbm config find I health 2 Turn the Health Monitor off db2 update dbm config using HEALTH_MON OFF 3 Determine what databases IBM Tivoli Monitoring connects to db2 list application 4 Check system buffer pool settings on the Tivoli Enterprise Portal Server and PROXYWH databases a db2 connect to TEPS db2 select from SYSIBM SYSBUFFERPOOLS b db2 connect reset C db2 connect to PROXYWH d db2 select from SYSIBM SYSBUFFERPOOLS Must be a value of 250 5 Adjust buffer pool settings that are too high Chapter 7 Tivoli Enterprise Portal troubleshooting 131 Note You must be connected to that database db2 alter bufferpool ibmdefaultbp size 250 6 Disconnect all applications db2 force application all 7 Stop DB2 db2stop 8 Start DB2 db2start 9 Recycle Tivoli Enterprise Portal Server service Existing OMEGAMON product imagery displays after upgrading to IBM Tivoli Monitoring V6 1 Existing OMEGAMON product imagery displays after upgrading because there it was cached on the client browser Use the following steps to correct the problem 1 Close all browse
18. NODE will usually return an appropriate value If the value is being provided through a link it can often be picked from the Server Name attribute of the link source row When in doubt examining a display of the unfiltered table can help determine what is expected In summary it should be emphasized that the product provided workspaces based on these queries should be used whenever possible A message appears after accepting the license UNIX only During installation of the Monitoring Agent immediately after accepting the license agreement a message similar to the 1slpp message shown below may be displayed Press Enter to continue viewing the license agreement or enter 1 to accept the agreement 2 to decline it 3 to print it 4 to read non IBM terms or 99 to go back to the previous screen 1 Islpp Fileset gsksa rte not installed Chapter 10 Monitoring agent troubleshooting 179 This message is harmless and is associated with verifying the versions of the gskit component installed by IBM Tivoli Monitoring The message is displayed at a point where the UNIX install normally pauses for a number of seconds causing a tendency to think the install has failed although this is not the case Do not interrupt or cancel the installation process at this point wait for the installation prompts to continue after the message is displayed and proceed to finish the installation as you normally would Adding agent help
19. On Windows installation _dir CMS KBBENV On UNIX based systems installation_dir config host_name_ms_Tivoli_Enterprise_Monitoring_Server_ID config KDY1009E Increase the OS agent timeout to 600 seconds TIMEOUT 600 The KDY1017E default is 300 seconds KDY1018E KDY1022E On Windows installation_dir CMS KNTENV KDY1025E na a KDY1030E On UNIX based systems installation _dir ux ini On Linux set it in KDY1033E installation dir lz ini The value must be set in seconds Chapter 10 Monitoring agent troubleshooting 169 Table 39 Resolutions for agent deploy operations that TIMEOUT continued Problem Resolution A system error occurs when running a tacmd command Increase the timeout for the tacmd setting environment variable to 3600 seconds On Windows Enter the following command set timeout 3600 Note Be aware that this command does not affect the TACMD_TIMEOUT in the KUIRAS1 109 but it does indeed change the timeout period Another solution is to change the TACMD_TIMEOUT environment variable in the itm_home bin tacmd file on GNU Linux and UNIX systems or the itm_home bin KUIENV file on Windows systems The TACMD_TIMEOUT in these files must be in minutes You can also change the environment variable in the kui env file on Windows systems and the tacmd shell script on non Windows systems Both of these files can be found in the CANDLEHOME logs directory A failure occurs whe
20. e Go to the Agent Management Services workspace for the agent in question and run the AMS Start Agent Take Action against the Proxy Agent Services agent with a resetRestartCount of 0 Chapter 5 Installation and configuration troubleshooting 71 72 Watchdog utility requires Windows Script Host 5 6 The OS Agent watchdog utility calls scripts that require Windows Script Host 5 6 at a minimum If these scripts are run on a system running an earlier version of Windows Script Host for example 5 1 then the script continues to run and over time results in multiple cscript processes running on the system Upgrade Windows Script Host to version 5 6 or later Unable to deploy monitoring agents from the Tivoli Enterprise Receive an error when attempting to deploy an monitoring agent from a previous version of IBM Tivoli Monitoring through the Tivoli Enterprise Portal KFWITM291E An agent configuration schema was not found error popup The application support for the version being deployed must be installed to the portal server or the agent configuration xml file for example r2_dd_062100000 xm1 must be manually copied to the same location in the portal server classes candle kr2 resources config where the current level configuration xml file for example r2_dd_062200000 xm1 resides Installing application support with a silent installation response file fails Running the Application Support Installer with a silent installati
21. e IBM Tivoli Monitoring Service Console sa 54 Displaying tasks in the command prompt soa a a D9 Copyright IBM Corp 2005 2010 General installation problems baa resolutions General installation frequently asked questions Windows installation frequently asked questions UNIX based systems installation frequently asked questions x Upgrade SQL file not found when installing application support on the standby hub Many files in the First Failure Data Capture log directory or multi instance agents are installed Incorrect behavior after an uninstallation and re installation Where Remote Deployment of agents is not supported Application Support lnstaller hangs An agent bundle is not visible from the Tivoli Enterprise Portal Agent Management Services fails afler deployment on Linux Itanium and xLinux with kernel 2 4 systems Watchdog utility requires Windows Seript H st 5 6 Unable to deploy monitoring agents ironi the Tivoli Enterprise Portal z Installing application support with a silent installation response file fails Unable to run gsk7ikm exe _cq_ log files appear SPD Installing a bundle on the wrong operating system architecture or kernel Installing a Software Package Block SPB on top of an existing running IBM Tivoli Monitoring agent Problems with the SPB file aon 4 Installation was halted and now receive message about active install 55 56 56 56 DZ
22. the client session to ensure all components are synchronized with your user permissions Changes to user permissions include navigator view assignment differences since the last logon If you want to apply new permissions for other users immediately make all necessary changes and recycle the Tivoli Enterprise Monitoring Server When the Tivoli Enterprise Monitoring Server recycle is complete each user is reconnected and their user ID validated If there were changes to their profiles users must restart the Tivoli Enterprise Portal client session The Tivoli Enterprise Portal client session doe snot to be restarted if no changes were made to their profiles Tivoli Enterprise Portal Server cannot connect to the Tivoli Enterprise Monitoring Server private interface If the Tivoli Enterprise Monitoring Server is installed on an AIX server with a public and a private interface the Tivoli Enterprise Portal Server cannot connect to the Tivoli Enterprise Monitoring Server There are two environment variables you can set to control which interfaces to publish For IPV4 use KDEB_INTERFACELIST for IPV6 use KDEB_INTERFACELIST_IPV6 In either address family you can set those variables to set restrict or add to the interfaces in use Table 38 Control interface publishing Interface control Environment variable To set specific interfaces for consideration KDEB_INTERFACELIST ip4addr 1 ip4addr n KDEB_INTERFACELIST_IPV6 ip6addr 1
23. 206 Incorrect data is collected in the warehouse for filtering if using a wildcard 206 Too much historical data is collected 207 Warehouse Proxy agent failed to export data 207 There are ORACLE or DB2 errors in the khdrasl log file 207 SQL0552N ITMUSER do noi have th privilege to perform operation CREATE BUFFERPOOL SQLSTATE 42502 2 2 ww 208 Chapter 14 Event AR troubleshooting 209 Unable to send events from user bacini situations to OMNIBUS 209 Preliminary IBM Tivoli Enterprise Console event synchronization troubleshooting 209 T EC ITM Synchronization Tasks 209 Changing the TCP IP timeout setting on your event server 210 Unexpected exception occurred when importing kqr baroc kqt baroc and kqv baroc 210 Editing the default destination server information from the command line does not work 210 Situation continues to send events even after it is stopped or sends events for undefined processes 210 IBM Tivoli Enterprise Console task timeout expires while in process 211 Situations with assigned severity riot taking the user assigned severity from the Tivoli Enterprise Portal a 211 Situations are not forwarded fom Tivoli Enterprise Monitoring Server to the Tivoli Enterprise Console server 211 Verifying that vents are being peed by Tivoli Enterprise Console 214 Enabling t
24. 57 60 60 63 63 63 63 64 65 67 67 67 67 68 69 70 Monitoring agents fail to start after agent support 70 SZ sat a ZL 71 71 72 72 fe 3 72 4 73 73 73 s73 74 iii Receive an install sh error when installing two components or agents in the same installation directory P When attempting i install IBM java 1 5 0 on Windows 64 bit system nothing happens During a remote Tivoli Enterprise Monitoring Server upgrade you receive a The Backup procedure for TEMS database files has failed pop up On Solaris systems GSKit installation fails to install Remote configuration of deployed Monitoring Agent for DB2 agent fails Monitoring Server cannot find your deployment depot The agent installati n log on the endpoint indicates that error AMXUT7502E occurred Failure occurs when sharing directories for the agent deploy depot You receive a KFWITM290E error when using deploy commands with a z OS monitoring server Running deployment i ina hotstandby environment i Difficulty with default port namber Selecting Security Validation User displaye a blank popup When installing a monitoring agent on top of the Systems Monitor Agent you receive an error OS Agent does not install and a message indicates it was already installed An IBM Tivoli Monitoring 6 1 upgraded agent is not configured to point to the same Tivoli Enterprise Monitoring S
25. Console link under the Service Point system balayne_ms When you select one of the links you will be prompted for a user ID and password to access the system This is any valid user that has access to the system Typing displays a list of the supported commands Chapter 4 Tools 51 52 The command for modifying the trace settings is ras1 If you type ras1 in the field at the bottom of the screen you will then see the help for this command The set option ras1 set turns on the tracing but does not affect existing tracing An example would be ras1 set UNIT xxx ALL UNIT yyy Detail This command will enable full tracing for the xxx class of the component and low level detailed tracing on the yyy class of the component The ras1 list command lists what tracing is set as default It is best to do an initial list in order to track what changes you have made to the tracing settings The following list describes the options of tracing available ALL Provides all trace levels Shown as ALL when using the ras1 list command Flow Provides control flow data describing function entry and exit Shown as FI when using the ras1 list command ERROR Logs internal error conditions Shown as ER when using the ras1 list command The output also shows as EVERYE EVERYU ER Other settings which provide component specific information are Detail Shown as Det when using the ras1 list command INPUT Shown as IN when using the ra
26. Installation and configuration troubleshooting 93 94 Repairing errors in Local Location Broker service initialization Bind failures due to insufficient authorization are reported with Errno 2 EACCESS o the Local Location Broker as the name LOCAL implies is done with a local socket address The bind fails for the following reasons e Insufficient authorization e The address is unavailable 32645848 E8E45647 kdebnws c 64 KDEB_NewSocket Status 1DE00000 KDE1 STC_CANTBIND 3265B3F0 E8E45647 kdebnws c 84 KDEB_NewSocket lt 0x176A97D4 0x10 gt BSD bind details Family 2 Socket 0 Status 1DE00000 Errno 2 00000000 00022EE1 00000000 00000000 00000000 326B1EA8 E8E45647 kdcsuse c 98 KDCS UseFamily status 1c010005 cant bind socket ncs KDC1_STC_CANT_ BIND 2001 252 04 42 41 KDC00008 Unable to create location server status 1C010005 Bind failures due to address in use 3ACDB600 DEB3B73F kdebnws c 62 KDEB NewSocket Status 1DE00030 KDE1_STC_ ENDPOINTINUSE 3ACF5028 DEB3B73F kdcsuse c 99 KDCS_UseFamily status 1c010005 cant bind socket ncs KDC1_STC_CANT_BIND Bind failure due to address in use but not fatal 1CF7B1F8 E6D9D743 kdcsuse c 99 KDCS UseFamily status 1c010005 cant bind socket ncs KDC1_STC_CANT_BIND KDSNC007 Local Location Broker is active If the bind failure is due to EADDRINUSE but the Broker service is started the error might not be fatal Determine whether or not the bind o
27. Support information If you have a problem with your IBM software you want to resolve it quickly IBM provides the following ways for you to obtain the support you need Online Go to the IBM Software Support site at http www ibm com software support probsub html and follow the instructions IBM Support Assistant The IBM Support Assistant ISA is a free local software serviceability workbench that helps you resolve questions and problems with IBM software products The ISA provides quick access to support related information and serviceability tools for problem determination To install the ISA software go to http www ibm com software support isa Searching knowledge bases You can search the available knowledge bases to determine whether your problem was already encountered and is already documented Searching the information center IBM provides extensive documentation that can be installed on your local computer or on an intranet server You can use the search function of this information center to query conceptual information instructions for completing tasks and reference information Searching the Internet If you cannot find an answer to your question in the information center search the Internet for the latest most complete information that might help you resolve your problem To search multiple Internet resources for your product use the Web search topic in your information center In the navigation fram
28. The link to an OS agent workspace is pointing to a superseded version of the workspace SNMP trap Sendto fails Situation overrides cannot be used to disable situations on specific systems at specific times Situation or calendar name in thresholds xml file appears incorrect BAROC file is missing for IBM Tivoli Monitoring 5 x Endpoint situations 7 A The target hostname platform anas version information is not displayed for the deploy status in the CLI or the workspace A workspace view is showing an error Agent upgrade and restart using non root After installing and configuring a monitoring agent it fails to start p situation_fullname slot missing for delete events Logs are using the situation ID string instead of the display name If a managed system list is removed ar a situation the situation stops Descriptions are not displayed for default situations Agent configuration failed on remote deploy while using single quotes for configuration properties New attributes missing Receive errors when modifying the JAVA HEAP SIZE for the Summarization and Pruning Agent When associating situations they fire but cannot be viewed PD The Summarization and Pruning agent fails when processing an index created in a previous release of the product Summarization and Pruning agent schedule not affected by daylight saving time Attribute names must be ne under 28 dharacien long Agent deploy operation
29. Tivoli Platform and Database Support Matrix The Tivoli Platform and Database Support Matrix provides platform and database support information for most Tivoli products The matrix can be found at http www ibm com software sysmgmt products support Tivoli_Supported_Platforms html Chapter 3 Solving common problems 11 Workspaces are missing or views are empty 12 The following problems might occur The workspaces return no data There_are no child Navigator items under the agent node in the Navigator view See Table 1 on page 13 e The Navigator items are labeled with internal names such as Knt KNT1076 instead of the correct names such as Disk See Table 1 on page 13 EE Windows OS Fi Fy Windows 05 i ER Knt KNT1076 Disk oR KnbEENTIO8S i E Enterprise Services colt Kn ANT 1126 Eh Memory EH Knt KNT1142 EH Network EH Kn KNT1166 EH Printer po EH Bint KNT1176 EH Process HER KntENT1194 e E Processor i EH KntKNT1200 EH System You receive message KFWITM217E Request error SQL1_CreateRequest failed rc 209 See Table 1 on page 13 You receive message KFWITM220E Request failed during execution See Table 3 Note For workspaces related to historical data refer to Historical data is missing Preliminary diagnostics i 5 Refresh the Navigator View gt Refresh 2 Verify that the monitoring agent has been started Restart if needed 3 4 If your data is missing in an Oracle
30. Unable to install InstallShield Scripting Runtime This is an InstallShield limitation You cannot install the product from the specified network drive Try installing from another network drive Install the product from a local drive if you continue to receive the error Installation on a Windows 2003 server fails with Error Number 0x80040707 The following error can occur while installing IBM Tivoli Monitoring V6 1 on a Windows 2003 Server This error can occur for the following reasons 1 Windows Service Pack 1 or higher is not installed 2 Windows Installer 3 1 KB893803 is not installed KB893803 is included in SP1 You must install Windows Service Pack 1 or higher or the KB893803 individual update You can download the update from www windowsupdate com IBM Tivoli Monitoring Troubleshooting Guide Extracting the nls_replace script causes remote deployment to fail The tool used to extract the file might have an option to automatically convert CRLF for text files If users extract the UNIX or Linux installation media tar files on Windows and this option is enabled the files are modified and become incompatible on some platforms The CR LF conversion must be disabled or another tool used to extract the files that does not convert the text files Upgrade tool deploys agent to the wrong directory The baseline XML file used for Tivoli Distributed Monitoring upgrade contains attributes that specify an installation direct
31. Universal Agent trapcnfg file in Universal Agent work directory When a trap is received the SNMP data provider looks up the trap enterprise OID value generic trap number and specific trap number in the trapcnfg The trap definition values received by the SNMP data provider must match the definitions in the trapcnfg file If the SNMP data provider does not find a match for the values in the trapcnfg it sets the status of the trap as Unavailable in the SNMP MANAGER Trap table attributes Although the traps are received they display the Unavailable status in the SNMP MANAGER Trap workspace The following example shows an intelliWatch trap definition Chapter 12 Universal Agent troubleshooting 197 intelliWatch Monitor 1 3 6 1 4 1 1983 1 1 Type 2 criticalAlarm 1 3 6 1 4 1 1983 1 1 6 1A 10 Status Events Type 3 where 1983 Specifies the Enterprise OID 6 Specifies generic trap number 1 Specifies the specific trap number Define the correct traps values in the trapcnfg file in the Universal Agent work directory The TRAP workspace does not display expected traps For the SNMP to receive traps from SNMP agents the trap destination list of the monitoring agent must include the host name or IP address of the host on which the Data Provider resides Configuration of SNMP agents varies from monitoring agent to monitoring agent Refer to your SNMP monitoring agent documentation If you are not receiving traps ensure that
32. You can find the browser log in the following directory ensure that your folder options are set to display hidden files e C Documents and Settings Administrator Application Data IBM Java Deployment log gt plugin150 trace for IBM java plugin e C Documents and Settings Administrator Application Data Sun Java Deployment log gt plugin150_xx trace for sun jre Tivoli Enterprise Portal has high memory usage and poor response time The amount of Java memory used by the Tivoli Enterprise Portal increases with the size of the monitored environment If the maximum Java heap size setting is too low the amount of time spent by the JVM performing garbage collection can become excessive resulting in high CPU utilization and poor response time Verbose garbage collection GC data can be used to determine if excessive garbage collection occurs If the percentage of time spent performing garbage collection is greater than 5 increase the maximum Java heap size the Xmx parameter to provide more memory and reduce garbage collection activity For more information about collecting verbose GC data and Java heap tuning parameter changes refer to the IBM Developer Kit and Runtime Environment Java 2 Technology Edition Version 1 4 2 Diagnostics Guide SC34 6358 01 which is available from www ibm com developerworks java jdk diagnosis Use the following steps to increase the maximum Java heap size Tivoli Enterprise Portal browser client
33. continued Variable Value type Useful for KFW_REPORT_FIND_ YorN Default is Y When set to Y the portal server queries the WAREHOUSE_AT_ data warehouse during startup If set to N the STARTUP warehouse will be initialized at the first deliberate query instead of pre initialized KFW_REPORT_TERM_ YorN Default is N When set to Y specifies the point where a BREAK_POINT historical request selects from short term or long term history data KFW_SQL_VERBOSE YorN Default is N When set to Y provides RAS1 trace of each SQL statement issued by the Tivoli Enterprise Portal Server to the Tivoli Enterprise Monitoring Server or to the DB2 ODBC data source The use of this variable is deprecated because the same capability is available using the standard RASI trace setting KBB_RAS1 ERROR UNIT ctsql IN ER MSG_MODE kms or MSG2 Specify kms if you want to use IBM Tivoli Monitoring Operations Logging Specify MSG2 to use MSG2 logging Tivoli Enterprise Monitoring Server environment variables The following table lists Tivoli Enterprise Monitoring Server environment variables Table 54 Tivoli Enterprise Monitoring Server environment variables Variable Purpose ATTRLIB Specifies the Tivoli Enterprise Monitoring Server attribute ATR files directory CANDLE_HOME The directory where the product was installed CMS_BINPATH Specifies the Tivoli Enterprise Monitoring Server binary files dire
34. e For information on reading RAS1 logs see Reading RASI logs on page 41 e For information on the rasllog tool see rasllog tool on page 60 Snapcore files The snapcore command gathers a core file program and libraries used by the program and compresses the information into a pax file The file can then be downloaded to disk or tape or transmitted to a remote system The information gathered with the snapcore command allows you to identify and resolve problems with an application To collect all the information you might need to debug and analyze the problem you can use the snapcore command as shown in the following steps 1 Change to the directory where the core dump file is located 1s 1 total 84176 rw r r 1 root system 2704 Feb 21 09 52 core 18048 01084144 2 Run the snapcore command to collect all needed files snapcore d tmp myDir core 18048 01084144 The snapcore command gathers all information and creates a new compressed pax archive in the tmp myDir directory If you do not specify a special directory using the d flag the archive will be stored in tmp snapcore directory The new archive file will be named as snapcore_ pid pax Z 1s 1 tmp myDir total 5504 rw r r 1 root system 2815081 Feb 21 09 56 snapcore_20576 pax Z 3 To check the content of the pax archive use the following command uncompress c snapcore_20576 pax Z pax core 18048 01084144 README lslpp out errpt
35. or both UNIX is a registered trademark of The Open Group in the United States and other countries Other company product and service names may be trademarks or service marks of others 258 IBM Tivoli Monitoring Troubleshooting Guide Index Special characters Tivoli Management Framework Reference Manual Numerics 32 bit 149 64 bit 149 A abends core files capturing 68 accessibility 248 addBundles command 184 addSystem command 189 agent displays offline 131 Agent deploy Timeout 169 agent not uninstalling 106 agents display offline 146 not available 177 not displaying in the navigator 176 AMXUT7502E 75 AMXUT7512E error 106 Application Support Installer 79 arithmetic overflow error 230 attributes missing 168 Backspace Check utility 63 books see publications 248 browser mode clear JAR cache 121 help not displaying in 121 uninstall procedure 56 browser problems 68 Build TEPS Database 63 BuildPresentation bat 63 C can bind a LONG value only for insert 177 CandleManage 176 184 clear JAR cache 121 client logon error messages 107 set trace options 42 cmwrasl log 42 command wrules_check 222 command prompt display tasks in the 55 connectivity 111 Connectivity Tivoli Enterprise Portal on XP 113 Copyright IBM Corp 2005 2010 Connectivity continued Tivoli Enterprise Portal Server lost contact with the Tivoli Enterprise Monitoring Server 113 connectivity problems 107 conso
36. or the monitoring server are listed in the Manage Tivoli Enterprise Monitoring Services window right click each and click Advanced gt Unconfigure Repeat this step for all components that are listed Close the Manage Tivoli Enterprise Monitoring Services utility Open the Windows Control Panel Double click Administrative Tools and then double click Services Verify that all related Candle OMEGAMON and IBM Tivoli Monitoring services have been removed These services match those listed in the Manage Tivoli Enterprise Monitoring Services window Open the Registry Editor by clicking Start gt Run and typing regedt32 Click OK Note Create a backup of the registry before editing it Expand the key HKEY_LOCAL_MACHINE registry key Expand the SOFTWARE registry key Expand the Candle registry key and record any sub keys that are present If the Candle key does not exist proceed to step Expand the OMEGAMON registry key under the Candle key and record the content of the OMEGAMON key values Delete the Candle registry key and all sub keys On Windows XP you can right click the Candle registry key and click Delete Close the Registry Editor Open the Windows Explorer and find the Candle OMEGAMON and IBM Tivoli Monitoring installation directories The default value for Candle OMEGAMON is C Candle the default value for IBM Tivoli Monitoring is C IBM ITM Delete this directory and all subdirectories Use the steps in Veri
37. returns NPTL 2 3 4 or something like it This enables me to run the install sh script without causing the memory fault OR Changing the JAVA_COMPILER variable to NONE before upgrading allows me to continue without hitting the core dump General installation problems and resolutions This section describes general installation problems and resolutions Upgrade SQL file not found when installing application support on the standby hub When adding application support to the hubs in a hot standby setup after the first hub has been seeded you might receive an error message similar to the following about the productcode_upg sql file not being found while seeding the second hub Seeding support for Monitoring Agent for Microsoft SharePoint Server 8 of 10 KCIIN1602E ERROR file not found boadata IBM ITM tables cicatrsq SQLLIB kqp_upg sql Option f install upgrade can be used with the itmcmd support command to force using the pristine installation or upgrade support file for the product s application support Seeding failed Seeding support for Monitoring Agent for Microsoft Virtual Server 9 of 10 KCIIN1602E ERROR file not found boadata IBM ITM tables cicatrsq SQLLIB kqr_upg sql Option f install upgrade can be used with the itmcmd support command to force usingthe pristine installation or upgrade support file for the product s application support Seeding failed This error is not necessarily a fatal error
38. s managed system name Unsupported for z OS based agents CTIRA_SIT_MGR Specify whether or not to use the Tivoli Enterprise Monitoring Agent persistent situation file when registering with the Tivoli Enterprise Monitoring Server Using this file improves Tivoli Enterprise Monitoring Server performance Set to N to disable usage For z OS Tivoli Enterprise Monitoring Agent the value must be N since this feature not implemented for z OS For all other platforms default is Y CTIRA_SIT_PATH Required variable specifying directory where the Tivoli Enterprise Monitoring Agent based agent persistent situation file is stored This is the Tivoli Enterprise Monitoring Agent s only file and it contains a copy of the Tivoli Enterprise Monitoring Server monitoring situations for Tivoli Enterprise Monitoring Agent use during Tivoli Enterprise Monitoring Server registration The file is named psit_msn str where msn is the Managed System Name of the agent process Unsupported for z OS based agents CTIRA_STANDALONE OBSOLETE Should never be set If it is specified it MUST only be set to N or n Using any other value including an empty string will cause the agent to fail to connect to a Tivoli Enterprise Monitoring Server CTIRA_SUBSYSTEM_ID Optional variable to override the subsystem ID qualifier subsystem hostname nodetype of the agent managed system name MSN Describes a monitored resource instance to help make t
39. this utility would also populate the database with tables Now it is necessary to also run the BuildPresentation utility to build the tables in the database To build and populate a database complete the following steps 1 From the Manage Tivoli Enterprise Monitoring Services window right click TEPS 2 Select Advanced gt Utilities gt Build TEPS Database 3 Run the BuildPresentation bat file found in install_dir CNPS IBM Tivoli Monitoring Operations Logging You can use this logging facility to determine the cause of IBM Tivoli Monitoring problems IBM Tivoli Monitoring Operations Logging replaces MSG2 logging With MSG2 logging physical space problems can occur due to MSG2 logs that grow without bound until a process is stopped IBM Tivoli Monitoring Operations Logging enables you to configure log file management to avoid these problems Note This functionality is not supported for agents It is only supported for the log messages produced through the MSG2 facility as used by the Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal Server Windows and UNIX systems The new optional logs replace the Tivoli Enterprise Monitoring Server log files install_dir cms kdsmain msg on Windows systems and install_dir logs hostname_ms_timestamp log on UNIX based systems For the Tivoli Enterprise Chapter 4 Tools 63 64 Portal Server they replace the install_dir logs kfwservices msg file on Windows systems and
40. 1 is ip pipe 9 42 46 26 21343 GLB entry 2 is ip 9 42 46 26 21343 GLB entry 3 is sna USCACOO1 VWCTHLB CANCTDCS SNASOCKETS 135 GLB entry 4 is ip pipe 9 42 46 26 21343 GLB entry 5 is ip 9 42 46 26 21343 GLB entry 6 is sna USCACOO1 VWCTHLB CANCTDCS SNASOCKETS 135 IBM Tivoli Monitoring Troubleshooting Guide Connectivity between the Omegamon XE and Tivoli Enterprise Monitoring Server address space fails if this list is incorrect The GLB entries display in the order in which they are configured in the global site text file Additionally the address of the local platform is appended to this list This is an RPC architecture requirement It allows the local platform to be queried when the GLB list has been exhausted and no hub is found Confirming the Global Location Broker service initialized The bind messages in the RKLVLOG indicate the success or failure of the GLB service initialization A message ID prefixes the GLB status messages and indicates how the GLB service was started KDSNC008 is the message prefix issued on successful GLB process initialization when the LLB is started internally by the Tivoli Enterprise Monitoring Server Bind of global location broker complete ip pipe 9 42 46 26 21343 Bind of global location broker complete ip 9 42 46 26 21343 Bind of global location broker complete sna USCACOO1 VWCTHLB CANCTDCS SNASOCKETS 135 Global Location Broker is active Repairing errors in Global Location
41. 27 1 4 ACF1 kbbacli c 400 May 29 2007 12 54 28 1 11 ACF1 kdhsfcn c 400 May 29 2007 13 00 11 1 1 KDH kdhserq c 400 May 29 2007 12 59 53 1 1 KDH kdhblpr c 400 May 29 2007 12 59 39 1 1 KDH kdhsgnh c 400 May 29 2007 12 59 49 1 1 KDH kdhOuts c 400 May 29 2007 12 59 23 1 1 KDH kdhsrsp c 400 May 29 2007 13 00 13 1 2 KDH kdhslrp c 400 May 29 2007 13 00 12 1 1 KDH kdhscsv c 400 May 29 2007 12 59 58 1 9 KDH kdebbac c 400 May 29 2007 12 56 50 1 10 KDE The UNIT value matches any unit that starts with the specified value For example UNIT kra FLOW prints the FLOW traces for all files which match kra How to turn tracing back off The option for turning the tracing off is ANY For example you would use the following command to turn off tracing for the kbbcrcd class of the Windows OS agent rasl set UNIT kbbcrcd ANY Using the IBM Tivoli Monitoring Service Console The IBM Tivoli Monitoring Service Console enables you to read logs and turn on traces for remote product diagnostics and configuration The IBM Tivoli Monitoring Service Console is uniquely identified by its service point name All IBM Tivoli Monitoring Service Consoles for a host are linked and presented on the IBM Tivoli Monitoring Service Index for that host Point a browser to the HTTP port 1920 on a specific host for example http goby 1920 to launch the IBM Tivoli Monitoring Service Index You can also launch the Service cons
42. ALL UNIT s_ ALL The UNIT s_ ALL trace level includes tracing of system calls during SNMP processing Setting the trace option for the IBM Tivoli Universal Agent Use the IBM Tivoli Universal Agent trace facility to diagnose problems with the Universal Agent The Universal Agent uses RASI tracing By default Universal Agent trace data is written to a file in the logs subdirectory The default RAS1 trace level is ERROR for all Universal Agent components and modules On Windows Tivoli Monitoring Services overwrites the kumras1 10g each time the Universal IBM Tivoli Monitoring Troubleshooting Guide Agent starts and there is no method for archiving previous RAS1 log files Therefore you must obtain the RAS1 log that matches the problem occurrence before contacting IBM Software Support You can set tracing options for individual Universal Agent components and modules in the KUMENV file on Windows or the um ini file on UNIX based systems RASI supports pattern matching For example UNIT kums options traces all SNMP data provider modules because they all begin with kums Detailed RAS1 tracing can degrade Universal Agent performance due to high CPU usage and I O overhead Therefore set the Universal Agent RAS1 tracing to KBB_RAS1 ERROR after problem diagnosis If a module produces excessive error messages and fills the RASI log set UNIT modulename None to suppress the module until you resolve the errors If you discover an old Windows R
43. Agent 198 IBM Tivoli Monitoring Troubleshooting Guide Universal Agent emitted traps are not received by third party SNMP Manager The policies are configured to emit SNMP traps but the traps do not display in the receiving system console Use the following table to determine the source of this problem Table 43 Universal Agent emitted traps are not received by third party SNMP Manager Symptom Explanation There are error messages in the Universal Agent RAS1 log in the kumaeagt cpp module Error messages indicate why the policy is not being driven for example kumaeagt cpp 154 kum_kumact_agent TakeSample Error could not find terminator in lt Local_Time Day_Of_Week gt 02 AND Local_Time Day_Of Week lt 06 AND Local _Time Time gt 053000 AND Local _Time Hours lt 21 AND Queue_Statistics Current_Depth gt 12800 AND STRSCAN QMQUEUES QNAME ACK 0 gt tok1 NULL tok2 NULL tok3 NULL attrValTemp lt gt The kumaeagt cpp module implements Universal Agent s Alert Emitter probe Any error messages in this module might indicate that the policy request never got to the Universal Agent module that sends the traps There are no error messages in the Universal Agent RAS1 log but the traps are not received If the Universal Agent RAS1 log has no error messages check whether there are error messages in the Tivoli Enterprise Monitoring Server log If there are errors that
44. Connection JDBC URL KSY_WAREHOUSE_USER The Warehouse Database Connection User encrypted KSY_WEEKENDS_AS_VACATIONS Whether or not weekends are vacation with Y or N KSZ_JAVA_ARGS Java arguments Tivoli Enterprise Monitoring Agent deploy environment variables The following Tivoli Enterprise Monitoring Agent parameters can be changed in the KBBENV file Table 59 Tivoli Enterprise Monitoring Agent deploy environment variables Variable Purpose AGENTDEPOT An OS agent parameter Determines the location on the agent host where agent bundles will be transferred CMS_MSGBASE Applies to is 5 platform Tivoli Enterprise Monitoring Agent only Specify the MSG2 message file for Tivoli Enterprise Monitoring Agent framework messages CT_CMSDIRECT OBSOLETE Has been replaced by IBM Tivoli Monitoring v6 X firewall communications services Specify the full NAT firewall address of the Tivoli Enterprise Monitoring Server to be connected to including protocol address port CT_CMSLIST Required variable specifying the primary and or secondary Tivoli Enterprise Monitoring Server for the agent to connect with Takes a semicolon delimited list of Tivoli Enterprise Monitoring Server by network protocol hostname or protocol address 242 IBM Tivoli Monitoring Troubleshooting Guide Table 59 Tivoli Enterprise Monitoring Agent deploy environment variables continued Variable
45. Console The Tivoli Monitoring Services integral Web server is installed automatically with the Tivoli Enterprise Portal Server and enables users to access the IBM Tivoli Monitoring Technology Service Console You can prevent users from accessing the IBM Tivoli Monitoring Technology Service Console that is available through the integral Web server http computer_name 1920 To block access to the IBM Tivoli Monitoring Technology Service Console disable the integral Web server However if you disable the integral Web server you must install a third party Web server on the Tivoli Enterprise Portal Server computer to access the images and style sheets for the graphic view and edit the application parameters at every desktop client 1 From the Windows desktop select Start gt Run gt 2 Type regedit 3 Open the Tivoli Enterprise Portal Server Environment folder HKEY_LOCAL_computer SOFTWARE Tivoli Monitoring Services KFW Tivoli Enterprise Portal Server KFWSRV Environment 4 Locate the KDC_FAMILIES in the right frame and add a space and type the following at the end of the line http_server n Example IP PORT 1918 SNA use n IP PIPE use n http_server n 5 Install a third party Web server on each computer where you installed the Tivoli Enterprise Portal desktop client IBM Tivoli Monitoring Troubleshooting Guide a From the Windows desktop select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring
46. Edit the Java heap parameters using the Java Plug in Control Panel Before you change the values for the Java Plug in consider that any changes affect all Java applications that use the Java Plug in See Preparing your Tivoli Enterprise Portal environment in the IBM Tivoli Monitoring Administrator s Guide for more information 1 Open the Windows Control Panel 2 Double click Java Plug in for IBM Java V1 4 2 or right click and select Open If you have multiple Java Plug ins check that you have the properties for the right plug in opened The About tab shows V1 4 2 and the Cache or Certificates tab shows IBM in the path or in the signed certificate 3 In the Advanced tab select the IBM JRE 1 4 2 from the Java Runtime Environment list 4 In the Java Runtime Parameters field enter Xms128m Xmx256m 5 Click Apply Tivoli Enterprise Portal desktop client Edit the cnp bat file in install _dir CNP to change the Java heap parameters in the set command to Xms128m Xmx256m Tivoli Enterprise Portal has high memory usage 126 The amount of Java memory used by the Tivoli Enterprise Portal depends on the size of the monitored environment The largest component of the memory usage is for the Java heap which contains both short term and long term data When a memory request cannot be satisfied from free Java heap storage garbage IBM Tivoli Monitoring Troubleshooting Guide collection is performed to reclaim free memory Th
47. IBM Tivoli Monitoring V6 1 universal message when a situation is true following these steps on Tivoli Enterprise Portal client Open the situation editor Select a situation Select the Action tab Check Universal Message button Move the cursor to Message or Category text field Enable Input Method IM for DBCS Type a key to input DBCS SN ot PO INS However at step 7 nothing is set into the text field because the text field does not accept double byte characters DBCS Disable the Input Method and input only single byte characters SBCS An error occurs when remotely removing an instance on Windows The following log shows an error occurs when remotely removing an instance on Windows 4300DFC2 0000 828 kbbssge c 52 BSS1 GetEnv KBB_RAS1 ERROR gt C IBM ITM tmaitm6 logs KNTRAS1 LOG 4300DFC2 0001 828 kbbssge c 52 BSS1 GetEnv CANDLE HOME C IBM ITM 4300DFCE 0000 828 kdytasks cpp 1063 runCommand Bad return code 3221225477 from command C IBM ITM TMAITM6 kdy_xa exe r DB2 FROBERTS UD pc ud 4300DFCE 0001 828 kdytasks cpp 1066 runCommand STDOUT 0 characters is 4300DFCE 0002 828 kdytasks cpp 1069 runCommand STDERR 76 characters is Error line 391 Unable to delete file C IBM ITM tmaitm6 KUDCMA_DB2 exe 4300DFCE 0003 828 kdytasks cpp 3291 setConfig KDY1008E Received bad return code 3221225477 from two way translator command C IBM ITM TMAITM6 kdy_xa exe r DB2 FROBERTS UD
48. Monitoring Server main menu 5 Submit the job and check for good condition codes 6 Recycle the Tivoli Enterprise Monitoring Server DATA SET NOT FOUND JCL error occurs when submitting the DS 3xxxx Tivoli Enterprise Monitoring Server Create runtime members job Ensure the following e The pp 1xxxx RTE Build job successfully ran for this RTE To perform the RTE Build job 1 Place the B option next to the RTE on the KCIPRTE RTE main menu to instruct Configuration tool generates the pp 1xxxx mRTE Build job 2 Ensure that the RTE Build job contains allocations for the amp rvhilev amp rte RKDS and amp rvhilev amp rte RK runtime libraries 3 Submit the RTE Build job e The Tivoli Enterprise Monitoring Server in this RTE field is set to Y on the RTE Update panel if the RTE Build job does not contain amp rvhilev amp rte RKDS libraries If you must edit the field regenerate the RTE Build job Refer to the IBM Tivoli Monitoring Installation and Setup Guide manual for more information about configuring a z OS Tivoli Enterprise Monitoring Server and the RTE Build job The error CMSLIST NOT ALLOWED occurs on the Specify Advanced Configuration Values panel when Tivoli Enterprise Monitoring Server type equals hub The F10 CMSLIST key from the Specify Advanced Configuration Values panel is only applicable to a remote Tivoli Enterprise Monitoring Server This PF Key allows the remote Tivoli Enterprise Monitoring Server to select
49. Monitoring Administrator s Guide for information about changing the omegamon r1s file Chapter 14 Event synchronization troubleshooting 219 Table 47 Situation event was not acknowledged or closed continued Problem Resolution The events are not cached in the expected location The location of the cache file is specified during installation The default locations are as follows e On Windows C tmp TME TEC OM_TEC persistence e On UNIX based systems var TME TEC OM_TEC persistence The value of fileLocation is defined in the active configuration file as indicated by the situpdate properties file in the following locations On Windows C Program Files TME TEC OM_TEC etc e On UNIX based systems etc TME TEC OM_TEC You can edit the location of the cache file by running the sitconfig sh script Refer to the IBM Tivoli Monitoring Administrator s Guide for sitconfig sh script usage The IBM Tivoli Enterprise Console Situation Update Forwarder is not currently running To use event forwarding you must start the Situation Update Forwarder This process is started automatically when the event server starts To start the process manually change to the BINDIR TME TEC OM_TEC bin directory and run the following command On Windows startSUF cmd On UNIX based systems startSUF sh To stop the process run the stopSUF sh command stopSUF cmd on Windows in the same directory Note If the Situation Updat
50. NAME KfwServices STACK EXECUTION DISABLED drWatson dumps and logs If the systems stops on Windows collect the drwtsn32 10g and user dmp files if available The drwtsn32 10g and user dmp files are located in Documents and Settings All Users Documents DrWatson 1 oO gt o Enter the following command at the command prompt to enable Dr Watson as default debugger drwtsn32 i Enter the following command at the command prompt to open the Dr Watson configuration dialog drwtsn32 Set the Crash dump Type to FULL Remove the check from the Dump Symbol Table box Place a check in the Dump all Thread Contexts box Place a check in the Create Crash Dump File box KxxCMA RAS files KxxCMA RAS files are created on Windows systems usually under the C windows system32 directory to gather information about IBM Tivoli Monitoring process crashes For example for the Monitoring Agent for Windows OS this file would be called KNTCMA RAS These files contain symptom dump information similar to the drWatson 10g but are generated by the IBM Tivoli Monitoring infrastructure Other important information Other important information includes the following Monitored application file as specified on the SOURCE FILE statement if applicable Description of the operation scenario that led to the problem Incorrect output such as Tivoli Enterprise Portal screencaptures or a description of what you observed if a
51. No See Table 8 on page 19 4 Collect the portal server log or the operations log and look for the following text strings e KFWDBVER version not found see Table 6 on page 18 e TEPS database not found see Table 6 on page 18 e user ID or password invalid see Table 7 on page 18 e DB2 instance not started see Table 8 on page 19 5 Run the TEPS Analysis ITMSUPER tool The following tables include possible causes and corresponding solutions for the problems described in this section Chapter 3 Solving common problems 17 Table 6 Correcting database problems 1 of 3 e Portal server database is not found or is missing e Missing table e Version of portal server does not match version record in database Diagnostic and corrective actions 1 Reconfigure the portal server On Windows Open Manage Tivoli Monitoring Services right click the portal server and select Reconfigure If the problem still persists go on to the next step On Linux or UNIX e GUI interface Open Manage Tivoli Monitoring Services right click the portal server and select Reconfigure e Command line interface Run this command itmemd config A cq 2 On Windows Run one of the following commands and set the correct password in the window that is displayed e For an SQL database cnpsdatasource exe e For an DB2 database db2datasource exe Then run the buildpresentation bat script Table 7 Correcting database problems 2 of 3
52. TEC OM_TEC bin Change the debugVerbose expr 0 to debugVerbose expr 1 The files are then tmp situpdate log and tmp situpdate trace The tracing can be increased on the Situation Update Forwarder by editing the sitconfig sh script to change the logLevel in the configuration file to low med or verbose The logfiles will be tmp itmsynch logs synch_trace log and synch_msg log Verify that the situpdate log log file for the short running script indicates events are being written to the cache file such as Sat Sep 2 15 36 03 GMT 2006 51600 1 events from tmp SUF1 tmp written to cache file space Tivoli_4141 var TME TEC OM_TEC persistence situpdate_1157135641 Verify that the Situation Update Forwarder is running Verify from the SUF log synch_trace 1og file that it is reading from the cache files such as 2006 09 02 17 36 05 876 02 00 com tivoli candlenet SituationUpdateForwarder readCacheLine IBM Tivoli Monitoring TEC Synchronization Verify that the SUF log synch_trace 1log file contains successful send event messages 2006 09 02 17 36 06 311 02 00 com tivoli candlenet SituationUpdateForwarder pollCacheFile IBM Tivoli Monitoring TEC Synchronization Machine _Name IP Successfully sent event If there are no successful event sent messages check the synch_trace 1og for exceptions If there is no indication that the Situation Update Forwarder attempted to send the event ensure the Situation Update Forwarder
53. Table 37 Cannot connect to Tivoli Enterprise Portal Server continued Problem Resolution If you are running the Tivoli Do the following on the computer where the Tivoli Enterprise Portal Server is Enterprise Portal in browser installed mode and reaching the Tivoli 1 In Manage Tivoli Enterprise Monitoring Services right click the Tivoli eee ig ioe nee across Enterprise Portal Browser service and select Reconfigure from the menu Dee vee een 2 In the Launch URL field change host name in http hostname 1920 cnp might not be able to resolve the host name client to the IP address of the Tivoli Enterprise Portal Server to specify the numerical address for example http 10 21 2 166 1920 cnp client 3 Click OK 4 Start Tivoli Enterprise Portal browser mode using the IP address instead of the host name 5 If you are still unable to connect contact IBM Software Support See Collecting and reviewing troubleshooting data on page 5 for information on what types of data to collect before contacting Support Cannot launch the Tivoli Enterprise Portal on an XP computer after installation with the message KFWITM215E The message KFWITM215E Unable to process logon request occurs A firewall setting on the client computer prevents the Tivoli Enterprise Portal client from connecting to the Tivoli Enterprise Monitoring Server Set the IBM JVM Java launcher as a trusted program to allow the Tivoli Enterpri
54. Tivoli Enterprise Monitoring Server main menu select the Specify communication protocols option 3 On the Specify communication protocols panel specify the IP protocols of choice and ensure Specify SNA PIPE as one of the protocols that the remote Tivoli Enterprise Monitoring Server uses for connection 4 Navigate forward to specify the communication protocols values for the selected protocols 5 From the Configure the Tivoli Enterprise Monitoring Server main menu select the Create runtime members option to generate the DS 3xxxx Create runtime members job 6 Submit the job and check for good condition codes 7 From the Configure the Tivoli Enterprise Monitoring Server main menu select the Complete the configuration option 8 Review the remaining tasks to finish the configuration of the product before starting the Tivoli Enterprise Monitoring Server The pp Mxxxx Tivoli Enterprise Monitoring Server migration job generated by the Configuration tool ended with a condition code 08 The JCL used to upgrade an existing RTE to IBM Tivoli Monitoring v6 1 level must perform several steps to ensure the existing data is not lost nor corrupted The batch job stops processing data in the event of an error such as insufficient DASD The batch job is not submitted a second time but three batch job are created that can be used to restore the data to its original state Therefore the upgrade JCL can be submitted to run and start from the begi
55. Tivoli Monitoring Installation and Setup Guide or OMEGAMON XE and Tivoli Management Services on z OS Common Planning and Configuration Guide KFWITMO10I Tivoli Enterprise Portal Server not ready OR KFWITM402E Communication with the Tivoli Enterprise Server could not be established e Wait for the portal server to establish connection To determine whether or not the portal server is ready for portal client logon search the portal server trace log for this text string Waiting for requests If that string is not found the portal server has not completed initialization Portal server initialization can take as long as 20 minutes To view the trace log open Manage Tivoli Monitoring Services right click the portal server and select Advanced gt View trace log e Recycle the portal server Tivoli Enterprise Portal Server does not start or For more information see If the status bar displays the Validating User Credentials message continuously the monitoring server stopped OR The message TEP has lost communication with TEMS displays continuously OR KFWITMQO8W The Tivoli Enterprise Portal Server has lost contact with the Tivoli Enterprise Monitoring Server If you are an administrator restart the monitoring server Otherwise notify an administrator and wait for the monitoring server to be restarted 108 IBM Tivoli Monitoring Troubleshooting Guide Table 36 Cannot
56. UAGENT which includes the DPLOG and ACTION workspaces DPLOG The DPLOG is a pure event table in that it maintains only the most recent 100 rows unless overridden by the KUMA_MAX_EVENT_ENTRIES environment variable The DPLOG contains informational and error messages about the status of a data provider that indicate e If a metafile was validated successfully e If a metafile failed validation which means the application will not come online e If a data source was available at startup e Which console ports and socket listening ports were used or unavailable e When monitoring started and stopped for a data source When monitoring switched from one file to another When an API or socket client program connected and disconnected The DPLOG also records other actions including metafile refreshes The two most common Universal Agent problem symptoms are e One or more managed systems do not come online e The managed systems are online but the workspaces are empty IBM Tivoli Monitoring Troubleshooting Guide Use the UAGENT application workspaces as one of the first tools to diagnose a Universal Agent problem You might find the solutions for both problems in the appropriate DPLOG The ODBC data provider also includes a DPLOG message indicating when monitoring started for every attribute group listed in every ODBC metafile ACTION workspace Whenever a Take Action command is issued or a Reflex Action fires an entry is
57. Xms128m Xmx256m 4 Click OK 5 From the General tab complete the following steps to clear the browser cache a Click Delete Files b Check the box labeled Downloaded Applets c Click OK 6 Click OK in the Java TM Control Panel Java errors occur with the IBM v1 4 2 JRE If you experience any problems using the IBM 1 4 2 JRE use the version shipped with IBM Tivoli Monitoring v6 1 IBM Tivoli Monitoring v6 2 ships with IBM 1 5 0 JRE Use this version if you have installed IBM Tivoli Monitoring v6 2 Web Portal Client does not work with Sun JRE Update one file associated with the Tivoli Enterprise Portal browser client in order to ensure that the Sun JRE is used The file name is jrelevel js and it is located in the IBM ITM CNB directory on Windows systems Here is the default value currently seeded in the file var jreLevel 1 5 0 The following value picks up the latest 1 5 x_xx version var jreLevel 5 0 family JRE identifier The following value use the default JRE which should be the Sun JRE assuming it was the last one installed var jreLevel x Clear the browser cache and restart your browser client To clear the browser cache Open Java Control Panel start Control Panel Java and complete the following steps 1 Select the General tab 2 Click Delete Files Chapter 7 Tivoli Enterprise Portal troubleshooting 125 3 Check the box labeled Downloaded Applets 4 Click OK
58. a less than optimal system event occur You can use the Log and Trace Analyzer to create resource sets Resource sets are sets of definitions that contain the path locations of the logs that you need to examine and the levels of information that they contain You can keep customized definitions to reuse The definitions provide the same set of instructions about where to find a log and what kind of information to gather from the log saving time during subsequent log imports The Log and Trace Analyzer also makes it possible for you to download and store symptom database catalogs to your local system These catalogs provide detailed diagnostic solutions to a variety of scenarios which can give direction to your troubleshooting tasks To download the Log and Trace Analyzer complete the following steps 1 If you do not have the IBM Support Assistant ISA installed go to the ISA Web site at www ibm com software support isa to download the software Instructions for downloading and installing the ISA are on the ISA Web site 2 Using the ISA built in Updater component download and install the ISA plug in for IBM Tivoli Monitoring from the IBM Web site at www ibm com software support isa Instructions for downloading and installing the ISA plug in are on the ISA Web site 3 Using the ISA built in Updater component download and install the plug in for Log and Trace Analyzer from the IBM Web site at www ibm com software 60 IBM Tivoli M
59. a single server AIX system the situations were not seeded properly After performing an initial install and running the itmemd support command there were no situations displayed in the Tivoli Enterprise Portal Reinstalling the Tivoli Enterprise Monitoring Server and the Tivoli Enterprise Portal Server Support and re running the itmcmd support command only sees partial seeding of the situations When reviewing the log itmaix8_ci_39128 1og there are many errors similar to the one below BRK R KEK K ERIK KE AK EKA K EKA K KEK KEKE AK KER ARERR ERK xxxxxxxeee Create Request Was Done x KERR e KARR EKA K EAR KER AKER AK ERK K EEA k kK k kk k kk RE UPDATE O4SRV TOBJACCL SET LSTRELEASE LSTUSRPRF IBM WHERE NODEL NT_SYSTEM AND OBJCLASS 5140 AND OBJNAME NT_Available Bytes Critical KEK RK RRR KEKE ARK A KKK KK KAKA KK EKA KER AKER AKER xxxxxxxxee Create Request Was Done x RRR KKK KARR EKA K EAR KER TTT TTT TTT TTT k kk k kk SQL1_OpenRequest status 79 If you have situations before you seed this can cause the installer to run the upgrade sql rather than the sql for a fresh seeding Incomplete event data from a situation is forwarded to IBM Tivoli Enterprise Console The typical situation for monitoring agents in IBM Tivoli Monitoring targets a specific resource and generates a limited amount of monitoring data In some cases a Tivoli OMEGAMON XE for Storage on z OS situation can monit
60. account To check the DB2 user ID and password for the database and data source Go to Control Panel gt Administrative Tools gt Data Sources ODBC At the System DSN tab select TEPS2 and click Configure Enter your user ID and password For example db2admin for database and CNPS for data source Click on Connect to test the connection to UDB database On the Advanced Settings tab verify that the DATABASE name is correct Table 8 Correcting database problems 3 of 3 DB2 instance is not started Diagnostic and corrective actions 1 Check the status of the instance in the DB2 Control Panel 2 Recycle the portal server and resolve any issues reported 3 Ensure the user ID and password are correct See Table 7 on page 18 for more information Chapter 3 Solving common problems 19 Tivoli Enterprise Portal logon fails 20 The following problems might occur e You cannot log on to the portal client e You receive one or more of the following error messages KFWITM392E Internal error occurred during logon KFWITMO09I The Tivoli Enterprise Portal Server is still being initialized and is not ready for communications KFWITMO10I Tivoli Enterprise Portal Server not ready KFWITM395E User ID has been locked or disabled KFWITM396E User ID has been locked or disabled by Tivoli Enterprise Portal Server Preliminary diagnostics e Refer to the IBM Tivoli Monitoring Messages reference
61. actions e Check connectivity between monitoring agent and monitoring server e Use the ping command to verify whether communication exists between the monitoring server and agents Ping from the monitoring agent system to the monitoring server and then from the monitoring server system to the monitoring agent Use the IP address of the host name specified during agent configuration If the communication is broken and you see high CPU proceed to the corrective actions e Turn on the RASI trace log to verify whether the monitoring agent has made a connection to the monitoring server See Setting traces on page 42 for more information Corrective actions If you still have high CPU usage issues even after ensuring proper connectivity across firewalls open a problem report with IBM Software Support For more information refer to the IBM Tivoli Monitoring Installation and Setup Guide Chapter 3 Solving common problems 31 Table 21 Correcting Oracle DB Agent problems Oracle DB Agent cursor performance problem e Oracle Agent does not return any data to the portal client e Oracle Agent consumes high CPU on the system Diagnostic actions 1 Collect the detail traces of collector and RAS1 log Refer to Enabling detailed tracing in the collector trace log and Setting RAS trace parameters in Appendix E Problem determination in the IBM Tivoli Monitoring for Databases Oracle Agent User s Guide
62. added to the ACTION workspace The Action table is keyed and ActionID is the Key attribute The Action table rows have a time to live value of 30 minutes Unlike the DPLOG which is data provider specific the ACTION table is shared by all data providers If you run multiple data providers the ACTION workspace under every UAGENT application contains the same rows The Action_Result can indicate what happened to a particular Take Action command For example if Universal Agent reflex actions fire faster than one per second the ACTION workspace temporarily stops recording the results Recording resumes after several minutes if the action rate slows down pdcollect tool This tool allows you to collect the most commonly used information from a system to allow for IBM service to investigate a problem It gathers log files configuration information version information etc This tool also provides the ability to manage the size of trace data repositories This tool is available for Windows UNIX Linux and z OS systems It is located in the ITM_Install bin directory on Windows UNIX and Linux systems It is supplied as the KMSPDCOL member of the RKANSAM dataset on z OS systems It is also available from the IBM Support Assistant ISA Web site at www ibm com software support isa This tool is included in the basic image install for IBM Tivoli Monitoring Invocation is simple and is documented in the command There are a number of option
63. after it is stopped or sends events for undefined processes 210 You create a situation and set a Take Action command to send email when the situation occurs The situation checks for specified missing processes usr sshd usr syslogd usr ntpd usr xinetd usr crond The situation continues to send email even after the situation has been stopped the Take Action command has been removed and applied and the situation has been deleted You try to recycle the agent recreate the situation and delete but you still receive the emails IBM Tivoli Monitoring Troubleshooting Guide Also the situation event console only reports on the defined situations as expected but the Take Action commands send events for several processes rather than reporting only those that are defined The primary remote Tivoli Enterprise Monitoring Server probably has a required but missing C Runtime IBM Tivoli Enterprise Console task timeout expires while in process Some tasks like Close_All_Reset_Events can require up to an hour to complete depending on the number of events processed The task continues processing in the background if the timeout expires however the task output does not display You do not need to run the task again You can increase the task timeout value on task panel to prevent the task from timing out Situations with assigned severity not taking the user assigned severity from the Tivoli Enterprise Portal T
64. agent command Specify the path with the d option using either or from a UNIX or when you deploy a Windows OS agent from a UNIX Linux Linux host when host The following examples display the correct usage to install the using the d option Windows OS agent in the path C ITM61 WIN 0SAgent and as the path Windows separator C ITM61 Win OSAgent UNIX or Linux c ITM61 Win 0SAgent tacmd suggestbaseline minimum maximum and average function values are ignored One or more function values entered for either the minimum maximum or average parameter is not valid so these incorrect values are ignored tacmd suggestbaseline command receives an error When using this command you receive this message The specified managed system is not overrideable because it does not have the appropriate affinity feature bit enabled The tacmd suggestbaseline command does not support pre IBM Tivoli Monitoring v6 2 1 agents When using the listSystems command the last two digits for the version appear as XX Extended version information for every agent might not always be available When this happens the last two digits of the version displayed are represented as XX This occurs for subnode agents or when agents are not enabled for Agent Deploy support The command tacmd restartAgent fails if the agent is already stopped If the tacmd restartAgent command is issued against an agent
65. also terminated A CMS EXE left running in response to the earlier failure is likely to cause a subsequent start of Tivoli Enterprise Monitoring Server to fail Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 141 142 THRESHOLDS XML and Tivoli Enterprise Monitoring Server table not cleaned when managed system override is removed Removing an existing managed system override definition by removing the managed system from a situation distribution list does not result in the override being removed from the Tivoli Enterprise Monitoring Server table and the THRESHOLDS xm1 file on the agent You must first check to see if any override is associated with that particular managed system before removing it from the situation If an override is found remove it before the situation s distribution is modified Situations fail to trigger for attributes by applying group function If you create a Situation with attributes for example Elapsed_Time and Virtual Bytes for NT_PROCESS and then apply the condition MAX Elapsed Time Seconds TRUE AND Virtual Bytes 5 the created situation should be triggered and forwarded to Tivoli Enterprise Console server However the situation is not triggered The Tivoli Enterprise Portal expects that the Tivoli Enterprise Monitoring Server should dynamically find the MAX row and then apply further conditions This is not how column functions work According to standard grouping functions can
66. an agent or monitoring server on z OS Error 8 trying to set up table lt table name gt KRAIRA000 Starting UADVISOR_Kpp_table name where pp is the two character component or product code and table name is the application table name Verify that the files are not being used by another task Verify that the files are initialized correctly and that the KppPDICT is inserted into the persistent data store files Verify that the maintenance procedure is correctly processing the persistent data store files Chapter 3 Solving common problems 25 A situation does not raise when expected You are working in the portal client or Tivoli Enterprise Console and know that conditions exist that should have raised a situation but the situation has not been raised Preliminary diagnostics 1 Verify that the monitoring agent has started 2 Verify whether the situation is associated with a node in the Tivoli Enterprise Portal Navigator 3 Confirm the situation is raised in the event console 4 Verify that maintenance has not been run against situations One possible tacmd command that could have been run is the tacmd maintAgent If maintenance has been run wait for the situation to restart 5 Click on any workspace where the data should be and verify that data is arriving 6 Running the following ITMSUPER tools might also provide more information e Situation Test tool e Exceptions Analysis tool e Distributions Analysis tool T
67. attribute is being used It allows adding characters to the command without spaces such as a forward slash or backward slash If the data in the action command must be transformed from the raw data to some other format that must be accomplished using shell commands In Windows or Linux and UNIX that could be a shell command like Perl or REXX or ksh and in zOS that would be a REXX command which is STARTed The shell command would transform the raw data into a display format and issue the command actually desired If a shell command is already being used it could simply take the needed formatting into account in its logic Events can also be sent through a Workflow Policy activity SNMP emitter or action command That data will also be unscaled Large difference in attribute values between the Tivoli Enterprise Portal and Perfmon On Windows 2008 systems when viewing attributes there is a very large difference in their values between the Tivoli Enterprise Portal and Perfmon The Agent Service Interface is not globalized This window is only displayed in English There is not a workaround for this issue OS agent installation does not detect System Monitor agents Any agent released with IBM Tivoli Monitoring v6 2 2 or before other than agents built with the latest Agent Builder tool should not be installed on top of the IBM Tivoli Monitoring v6 2 2 System Monitor agents Some attribute group names are unintelligible from th
68. bind db2cli lst blocking all grant public c SQLLIB bnd gt DB2 bind db2schema bnd blocking all grant public Some rows do not display in an upgraded table You might not see all rows after upgrading the Warehouse Proxy agent to IBM Tivoli Monitoring V6 1 because some tables might be corrupted Do the following to find the errors that occurred during the upgrade 1 Edit the KHDRAS1 Mig Detail log file 2 Search for the word EXCEPTION The KHD_MAX_ROWS_SKIPPED_PER_TABLE environment variable in the KHDENV_MIG file allows you to skip bad data Use KHD_MAX_ROWS_SKIPPED_PER_TABLE to specify the number of rows per table to skip to migrate if the data that needs to be inserted is incorrect When this number is reached migration of the table is aborted Using DB2 v8 1 Warehouse Proxy Agent crashes DB2 8 1 FP16 or higher is required Using DB2 V9 1 for z OS Warehouse Proxy agent encounters a large number of disconnections When using DB2 for z OS 9 1 for the warehouse database the Warehouse Proxy agent can encounter repeated disconnections from the database The default idle thread timeout value DSN6FAC IDTHTOIN in DSNZPxxx is 120 seconds The Warehouse Proxy agent uses a pool of database connections to process export requests from monitoring agents The warehousing interval used by agents can be set to values ranging from 15 minutes up to 24 hours The database connections are idle between export requests and if the idle thread tim
69. book that does not contain the fully qualified hostname Edit the resulting xml file and change the shortname to the fully qualified hostname Tivoli Enterprise Portal Server performance is slow If you want to increase the performance of your portal server and you are not concerned about security you can disable Secure Socket Layer data encryption on the portal server If you do not want to use Secure Socket Layer communication between IBM Tivoli Monitoring components and the Tivoli Enterprise Portal Server use the following steps to disable it 1 In Manage Tivoli Enterprise Monitoring Services right click Tivoli Enterprise Portal Server 2 Click Advanced gt Edit ENV file 3 Find the following line kfw_interface_cnps_ssl Y 4 Change the Y to N 5 Save the file and exit 6 Click Yes when you are asked if you want to recycle the service Cannot create a Tivoli Enterprise Portal Server database When using DB2 8 1 or 8 2 you must install the correct fix pack versions of DB2 in order to create a Tivoli Enterprise Portal Server database These fix pack versions are DB2 V8 1 with Fix Pack 10 or higher fix packs e DB2 V8 2 with Fix Pack 3 or higher fix packs Also on AIX systems a failure occurs when you attempt to install a Tivoli Enterprise Portal Server with a DB2 database Using the db2 installation User ID default is db2inst1 do the following 1 Stop the DB2 server if not already stopped using the follow
70. continuously When Tivoli Enterprise Portal detects a connection error it can repair the error and your client session can resume unaffected Use the information in this section to diagnose and recover from connectivity problems If you are running the Tivoli Enterprise Monitoring Server on z OS see Troubleshooting z OS based installations on page 88 for information about configuration problems affect connectivity A monitoring process fails to startup after changing a profile for root IBM Tivoli Monitoring processes such as the monitoring server portal server warehouse proxy agent summarization and pruning agent and other agents are all started while you are logged on as a user ID on Linux and UNIX systems For many shell environments the user ID has a profile file that is run during the initial processing to ensure a consistent working environment The profile must satisfy these requirements e During startup do not start any user interaction when there is not a connected console e Ensure that a korn shell ksh is available In general any shell can be used for profile except csh which has problems with output redirection e Eliminate any logic that can create an error associated with undefined variable evaluation or use korn file controls to suppress the errors e Set the PATH statements to what is needed for the environment Ensure that the profile completes and does not loop If any of these requirement
71. cpp 5547 IBInterface_refreshIB Hub is not there 42AEAZE5 0019 6 ko4crtsq cpp 5547 IBInterface_refreshIB Hub is not there 42AEA2E5 001A 6 ko4crtsq cpp 5547 IBInterface_refreshIB Hub is not there Tue Jun 14 04 27 01 2005 K041034 Monitoring for situation UADVISOR_OMUNX_SP20S ended 42AEA2E5 001B 6 ko4crtsq cpp 5547 IBInterface_refreshIB Hub is not there Tue Jun 14 04 27 01 2005 K041036 Monitoring for situation UADVISOR_OMUNX_SP20S started 42AEA3C3 0000 6 kdssqrun c 2995 Fetch QueryRowset error status 302 Tue Jun 14 04 30 43 2005 K041039 Error in request Notify Status 1105 Reason 302 42AEA3C3 0001 6 ko4async cpp 4744 IBInterface completeRequest Close failed request lt 55BE90 gt 42AEA53C 0000 6 ko4ibstr cpp 1090 IBStream insertDef Ret code 155 indicates hub connection lost Attempting to switch hubs o r reconnect 42AEA53C 0001 6 kdcgbin c 118 KDCG Bind Using GLB at ip 9 48 157 26 1918 42AEA53D 0000 6 ko4crtsq cpp 6456 IBInterface restartAllObjects No access list records changed 42AEA53D 0001 6 ko4mxque cpp 97 MutexQueue MutexQueue Reply store lt Fc0798B8 gt still associated with request lt 503D98 gt info re ply lt FC0798B8 gt info oType lt 5546 gt info oName lt INSERTO4SRV TNODESTS gt info sitName lt noname gt info regState lt 1 gt info physicall0 lt 1 gt info logIt lt 0 gt info reqGen lt 412 gt Tue Jun 14 04 37 01 2005 K041034 Monitoring for situation UADVISOR_OMUN
72. crash It is very possible that there is something unique about the system where this crash has been seen The crash has only been observed on one internal test system which was a Linux for AMD Opteron system tacmd login fails when monitoring server is configured with LDAP authentication Set the monitoring server tracing and LDAP client side tracing on the hub monitoring server KBB_RAS1 ERROR UNIT kds1g ALL UNIT kdsv ALL UNIT kg11g ALL UNIT kgl1d ALL UNIX or Linux systems Run the following commands on the system hosting the UNIX or Linux hub monitoring server 1 export LDAP_DEBUG 65535 2 export LDAP_DEBUG_FILE opt IBM ITM 1ogs ldaptrace txt or whatever path file you want 3 rm opt IBM ITM 1ogs ldaptrace txt to remove the file before restarting the Hub Restart the hub monitoring server Note that these LDAP related trace settings remain active until the monitoring server is restarted from a shell session where the variables are not exported If the monitoring server is restarted from the same session where these variables are still exported then the settings will be active again after restart Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 143 144 Reproduce the problem As you reproduce it any LDAP requests add trace content to that Idaptrace txt file Retrieve that 1daptrace txt file and the hub monitoring server log files The hub monitoring server logs will show any possible failures th
73. document should verify the applicable data for their specific environment Information concerning non IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non IBM products Questions on the capabilities of non IBM products should be addressed to the suppliers of those products All statements regarding IBM s future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only All IBM prices shown are IBM s suggested retail prices are current and are subject to change without notice Dealer prices may vary This information is for planning purposes only The information herein is subject to change before the products described become available This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental COPYRIGHT LICENSE This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and dis
74. e Graphic View Edit Navigator View Navigator view toolbar 1 On the computer where the Tivoli Enterprise Portal Server is installed select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services 2 Right click the Tivoli Enterprise Portal Server service point to Licensing and select Tivoli Enterprise Portal Server from the menu 3 In the Tivoli Enterprise Portal Service License window enter the new license key if one was issued 4 Check Enable Tivoli Enterprise Monitoring Server 5 Click OK When you click OK the server stops and a message in Tivoli Enterprise Portal says the server is unavailable If you do not close the client work session after the server has been started again next step another message says your Tivoli Enterprise Portal permissions have changed and prompts you to restart 6 Restart the Tivoli Enterprise Portal Server and after startup is complete restart the Tivoli Enterprise Portal client Situations are not firing Do the following to determine why situations are not firing in the Tivoli Enterprise Portal e Confirm the situation is firing in the event console e Ensure that the situation was distributed e Verify whether the situation is associated with a node in the Tivoli Enterprise Portal Navigator e Ensure that the situation condition is true e Check the operations log of the agent Historical UADVISOR situations are started on t
75. e Stop Script e Configuration Script e Operating System Version e Operating System Name e Dependencies There is a situation distribution discrepancy if there is a hub monitoring server outage when one or more remote monitoring servers remain active Run the itmsuper tool following the hub monitoring server recycle and take note of any distribution discrepancy between any of the remote monitoring servers and the hub monitoring servers that the tool reports on For the situations involved in the distribution discrepancy an undistribute followed by a redistribute of situations rectifies the problem by causing proactive notifications to occur where the agents are currently reporting Installing v6 2 2 agent application support on a monitoring server for a prior release causes agents to fail Do not install agent application support for the current release on a Tivoli Enterprise Monitoring Server for a prior release for example do not install V6 2 2 agent application support on a V6 2 monitoring server Doing so could cause agents to fail If you see the 208 return code at the Tivoli Enterprise Portal Server console you have installed application support for v6 2 2 on a back level Tivoli Enterprise Monitoring Server 162 IBM Tivoli Monitoring Troubleshooting Guide Installing backlevel Windows OS agent on existing environment causes Tivoli Enterprise Monitoring Server not to start Due to packaging optimizations in v6 2 1 insta
76. e histstopcollection e histunconfiguregroups e histviewattributegroup e histviewcollection e exportcalendarentries e importcalendarentries This problem has to do with your locale and system configuration and can be fixed by performing the following procedure 1 Open the command prompt 2 Change the command prompt locale according to the table below 3 Country Code Page Default Setting New Code Page Value Latin 1 Brazilian Portuguese 850 1252 French German Italian Spanish Latin 2 Czech Hungarian Polish 852 1250 Russian 866 1251 a To change the locale in the command prompt type chcp in the command prompt where is the new value for your code page and press enter For example if your system locale is Latin 2 you would type chcp 1250 in the command prompt To check the results of this change type chcp and press enter The command prompt will display the following message Active code page If the value displayed after Active code page is the same as the value you just entered then you have successfully changed the settings For example if your system locale is Latin 2 the command prompt should display the message Active code page 1250 4 Change the font displayed within the command prompt a You can do this by right clicking the title bar and clicking Properties from the drop down menu Click the Font tab and select Lucida Console from the list of
77. errors in the ODBC driver code e Independent Software Vendor ISV API errors The Universal Agent makes API calls to the ISV ODBC driver to implement the connections and SQL select statements The following RAS1 log excerpt lists ODBC status messages using default tracing KUMP_ProcessStartUpConfig Loading metafile lt f candle cma metafiles TIVOLI_DATA_WAREHOUSE md1 gt from startup config file f cand1e CMA WORK KUMPCNFG_INST1 DCHserver dp_register Application TIVOLI_DATA_WAREHOUSE successfully registered KUMP_ProcessStartUpConfig 1 application metafile s processed from startup config file f cand le CMA WORK KUMPCNFG_INST1 KUMP_StartDataProvider Starting ODBC Data Provider KUMP_Wait0DBCsourceReadyForMonitor Reusing connection handle for ODBC source TIVOLI_DATA_WAREHOUSE table lt syscharsets gt KUMP_ODBCserver Successfully connected to ODBC source TIVOLI_DATA_WAREHOUSE table lt syscharsets gt KUMP_WaitODBCsourceReadyForMonitor Reusing connection handle for ODBC source TIVOLI_DATA_WAREHOUSE table lt syscomments gt The Reusing connection handle messages indicates the ODBC provider is reusing resource to conserve memory The ODBC data provider allocates a connection for each metafile with multiple attribute groups that connect to the same data source using the same user ID and password combination Each SQL Select statement that is run for the various attribute groups shares the same connection hand
78. example 127 0 0 1 localhost localdomain localhost 192 168 0 9 tepshost Chapter 6 Connectivity troubleshooting 111 Table 37 Cannot connect to Tivoli Enterprise Portal Server continued Problem Resolution Tivoli Enterprise Portal client 1 cannot find Tivoli Enterprise Portal Server Select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services 2 Check that the Tivoli Enterprise Portal Server service is running and if not restart it 3 If the Tivoli Enterprise Portal Server is started see the KFWRAS1 LOG for any errors reported by the Tivoli Enterprise Portal Server application When the server starts an event ID 1 KFW0001 Tivoli Enterprise Portal Server Tivoli Enterprise Portal Server startup complete information entry is logged If you see an error entry double click the item to see the description Cannot connect to the Tivoli Do the following to determine if the Tivoli Enterprise Portal Server stopped and Enterprise Portal Server because restart it it stopped 1 On the computer where the Tivoli Enterprise Portal Server is installed select Start gt Programs gt IBM Tivoli Monitoring Services gt Manage IBM Services 2 Optional Right click the Tivoli Enterprise Portal Server entry and select Change Startup from the menu 3 In the window that opens select System Account and place a check in the Allow Service to Interact
79. files requires a restart of the Eclipse Help Server and the Tivoli Enterprise Portal Server When an agent s online help files are added to the eclipse server they are not available until the eclipse help server is restarted This also requires a restart of the Tivoli Enterprise Portal Server Unable to create historical collection directory for ud db2inst1 If you receive the following message check if the IBM Tivoli Monitoring environment is in SecureMode db2inst1 amsnt148d opt IBM ITM bin gt itmcmd agent o db2inst1 start ud Sourcing db2profile for user db2instl Starting Monitoring Agent for DB2 KCIINO174E Unable to create historical collection directory for ud db2inst1 There are two possible fixes for this problem Either manually set the file permission to history dir or add the db2 instance user to the root group See the IBM Tivoli Monitoring Installation and Setup Guide for instructions on how to have monitoring agents work properly with SecureMain in place Receive a large amount of data back from the warehouse for a baseline command 180 The variable TACMD_JVM_MAX_MEMORY must be more than 256 and less than 2048 The command defaults to 256 megabytes of heap space for the JVM IBM Tivoli Monitoring Troubleshooting Guide Chapter 11 Command troubleshooting This chapter describes problems you might experience with commands Commands with embedded single quotation marks fail The commands executeact
80. fonts in the window Click OK A window will appear allowing you to select the windows to apply the font change to Select Modify shortcut that started this window and click OK 5 You should no longer see corrupted characters in the CLI IBM Tivoli Monitoring Troubleshooting Guide The listSystems command consumes high CPU in enterprise environments In enterprise environments with many managed systems high CPU consumption is expected behavior for the listSytems command Improving tacmd command response time when using VMWare If you experience slow response time when invoking the tacmd command while running on a VMWare ESX system consider disabling virtualization of the Time Stamp Counter TSC To make this change add the following setting in the vmx configuration file for the virtual machine where the tacmd command is being invoked monitor_control virtual_rdtsc false This parameter is described in the Timekeeping in VMWare Virtual Machines paper on the VMWare Web site Pe cry Nenana arene PTY Measurement experience has shown that this setting can significantly improve the tacmd command response time on VMWare ESX systems The addSystem command fails with error message KUICCRO99E The KUICCRO99E error occurs when at least one incorrect parameter was specified When adding managed systems with the addSystem command ensure that you e Specify the correct product code e Specify a correct node that i
81. in the following example IBM Tivoli Monitoring Troubleshooting Guide INSTALL_ENCRYPTION_KEY IBMTi vol iOMEGAMONEncrytionKey61 The error Unexpected Signal 4 occurred at PC 0xFEC3FDE4 occurs during installation A Java VM heap dump occurs during installation which uses the JRE Use the following steps to resolve the problem 1 Ina terminal window run the following command to display the Java command java version 2 Determine where the Java executable is located by running which java 3 Rename or uninstall the Java executable Note This effects any other applications which depend on Java Be sure that it is safe to do so If unsure just rename the Java executable 4 Run the following command again to ensure that the java executable is not found in the path which java 5 Install the product Installing IBM Tivoli Monitoring on Red Hat 5 and see the following error KCI1235E terminating problem with starting Java Virtual Machine If you try to install IBM Tivoli Monitoring on Red Hat 5 with SELinux set to permissive or disabled mode enforced mode is not supported by IBM Tivoli Monitoring directly after rebooting the system and you see the following error KCI1235E terminating problem with starting Java Virtual Machine at the beginning of the installation before the license is displayed try executing the prelink a command to resolve the issue Installation of the IBM Tivoli Monitoring on the L
82. interface SYSL is found and source hostname indicates that the hostname SYSL was successfully resolved to an IP address Repairing the Name Resolution failures The following messages illustrate a Name Resolution failure IBM Tivoli Monitoring Troubleshooting Guide kdebpre c 661 interface_discovery IPV4 interface list WINMVS2C 9 20 138 199 source GE1 seq 0 flags 0441 kdebprc c 214 register string Unable to resolve interface address WINMVS2C In the messages above the absence of source hostname indicates an interface was discovered but the name is not resolvable to an address Typically this error results when the z OS image does not contain a TCP IP resolver setup file that provides either GLOBAL or DEFAULT configuration data Consequently native z OS address spaces are not enabled for name resolution by default By adding a DD statement for SYSTCPD to the started task JCL of the IBM Tivoli Monitoring task pointing to a usable file in USER PARMLIB TCPDATA resolver support can be enabled The following messages illustrate a variant of name resolution failure kdebprc c 661 interface discovery IPV6 interface list NULL KDE1I_OpenTransportProvider Status 1DE00048 KDE1 STC_NOINTERFACESREGISTERED The messages above indicate that no IPV6 interface is registered This can also result in TCP IP service initialization failure for the IBM Tivoli Monitoring address space The absence of an interface can only be fixed
83. it to another directory before starting Tivoli Enterprise Portal again The kcj 10g file contains errors generated by the Sun Java libraries used in the Tivoli Enterprise Portal client To set the trace options 1 Always backup the files before altering them 2 From the Tivoli Enterprise Portal menu select File gt Trace Options 3 Select a trace class from the list or as instructed by IBM Software Support such as UNIT Workspace ALL ALL provides data for all classes Use the setting temporarily because it generates large amounts of data ERROR logs internal error conditions This setting provides the minimum level of tracing with little resource overhead and ensures that program failures will be caught and detailed NONE turns off the error log so no data is collected 4 Click OK to close the window and turn on logging To set the trace log on UNIX systems Setting the trace option for the Tivoli Enterprise Portal Server trace You must set the trace options for the Tivoli Enterprise Portal Server through Manage Tivoli Enterprise Monitoring Services Before you set the trace options for the Tivoli Enterprise Portal Server determine the trace string The trace string specifies the trace setting Set trace options for the Tivoli Enterprise Portal Server when you start the Tivoli Enterprise Portal Server The log file continues to grow until you either turn off the trace or recycle the Tivoli Enterprise Portal Server Always b
84. log in to the Tivoli Enterprise Portal continued Problem Corrective action and solution Portal client cannot connect to the portal server because of firewall configuration OR KFWITM392E Internal error occurred during logon By default the portal client connects to the portal server on port 1920 or 15001 Open the blocked port or reassign ports accordingly For environments with multiple interfaces reconfigure the portal server to specify a specific interface by following the instruction below On Windows Use ipconfig all to verify the current network interface configuration Start the Manage Tivoli Monitoring Services and right click the TEPS entry and choose Advanced gt Set network interface Enter the correct IP address here On UNIX or Linux Use ifconfig a to verify the current network interface configuration Edit the agent ini file and add KDEB_INTERFACELIST IP_address where IP_address is the correct address For more information see Configuring port number assignments for the portal server in the IBM Tivoli Monitoring Installation and Setup Guide The portal server cannot initialize because of a DB2 shutdown OR KFWITMO09I The Tivoli Enterprise Portal Server is still being initialized and is not ready for communications Start DB2 or wait for DB2 to finish initializing If you receive message KFWITM009I you can look at the most recent trace log to verify that the port
85. minute timeout the following message might display when you try to add a IBM Tivoli Enterprise Console to a Tivoli Enterprise Portal client that is behind a router The RDBMS cannot be reached During the timeout period the Tivoli Enterprise Portal client seems hung To avoid this problem the Tivoli Management Framework Administrator must enable communications based on the host name rather than the IP address Use the following steps to prevent this problem 1 On the IBM Tivoli Enterprise Console Server on which you are logged in set the single_port_bdt and set_iom_by_name Tivoli Management Framework settings to TRUE by running the following commands on the IBM Tivoli Enterprise Console Server odadmin single_port_bdt TRUE all odadmin set_iom_by_name TRUE all 2 Run the following command to recycle the object dispatchers odadmin reexec all Refer to the Tivoli Management Framework Reference Manual for more information about the odadmin command The error ECO2029E Failed to connect to http servername port_number cgi bin tec_help pl occurs after clicking the Information button on the IBM Tivoli Enterprise Console Error ECO2029E occurs because the spider Web Server provided with the IBM Tivoli Enterprise Console Server does not automatically start To start the spider Web Server run the wstarthttpd command on the Tivoli Management Region server of the IBM Tivoli Enterprise Console that you are logged in to After the wstarth
86. not find the resolution to a problem you experience with a monitoring agent refer to the agent specific user guide listed in IBM Tivoli Monitoring library on page 251 section of this book Linux OS agent fails to start and remain started On some Linux systems SUSE 10 Linux Itanium there is a problem with OS Agent operation after deployment The current workaround is to use the o KDYRXA AUTOCLEAN NO option when executing the tacmd createnode command to deploy the OS agent to a remote node This option places the transferred installation image in the system temporary directory on the remote node Configured non root user agent starts up as root After configuring the agent to run as non root whenever it is restarted the agent restarts as root This is a present issue with the Solaris operating system Unable to access History Collection Configuration for any agent You are unable to access History Collection Configuration from the Tivoli Enterprise Portal for any agent and you receive the following message Cannot Load Product Configuration Data If you find that the Tivoli Data Warehouse database does not contain a WAREHOUSEPRUNE table and if in the portal server database the KFWWHISTDATA table has been renamed KFWWHISTDATABAK rename the KFWHISTDATABAK table to KFWWHISTDA in the portal server and then reconfigure the warehouse The WAREHOUSEPRUNE table will then be in the Tivoli Data Warehouse database and the iss
87. not update the screen The TEMP ITM_AppSupport_Install log Windows or tmp ITM AppSupport_Instal l 1og UNIX and Linux also fails to be updated after this point even after waiting for hours Change to the directory where setup jar exists and then use java jar setup jarto run the installer An agent bundle is not visible from the Tivoli Enterprise Portal The bundle has been added to the depot and is viewable from there but it is missing from the list of agents available for deployment from the Tivoli Enterprise Portal for a given node You cannot deploy an agent from the Tivoli Enterprise Portal if the xml version in the depot is later than the installed version because the newer xml might contain configuration properties that the back level agent does not support This issue was noticed for the DB2 agent Agent Management Services fails after deployment on Linux Itanium and xLinux with kernel 2 4 systems Agent Management Services fails after deployment on Linux Itanium and xLinux with kernel 2 4 systems when the o KDYRXA AUTOCLEAN YES option is used The Proxy Agent Services agent will not start when the deployment process completes if the option that removes the temporary directory used by remote deployment was used To start the OS agent when this problem occurs do one of the following actions e On the agent system manually restart the OS agent e On the agent system run CANDLEHOME bin itmcmd execute c 1z startWatchdog sh
88. numbers for the shift KSY_SHIFT2_HOURS A comma separated list of hour numbers for the shift KSY_SHIFTS_ENABLED Shift periods Only two shifts are allowed If shifts are enabled each hour 0 23 must be specified once KSY_START_OF_WEEK_ DAY Start of the week day for example 0 Sunday 1 Monday KSY_TIMEZONE_IND Time zone indicator AGENT specifies use the time zone offset of the agent WAREHOUSE specifies use the time zone offset of the warehouse KSY_VACATION_DAYS Vacation days in a comma separated list of days in YYYYMMDD format KSY_VACATIONS_ENABLED Whether or not vacation are enabled with Y or N KSY_WAREHOUSEAGGREGLOG_PRUNE Use to specify the pruning for the WAREHOUSEAGGREGLOG table The format of the value should be number unit where number is the number of units to retain and unit specifies the retention unit which can be one of day month or year KSY_WAREHOUSELOG_PRUNE Use to specify the pruning for the WAREHOUSELOG table The format of the value should be number unit where number is the number of units to retain and unit specifies the retention unit which can be one of day month or year KSY_WAREHOUSE_DRIVER The Warehouse Database Connection JDBC Driver KSY_WAREHOUSE_PASSWORD The Warehouse Database Connection Password encrypted KSY_WAREHOUSE_SCHEMA The Warehouse Database Connection Schema KSY_WAREHOUSE_URL The Warehouse Database
89. only through Tivoli Enterprise Monitoring Server you do not need SNMP However if you are sending traps to the emitter through the Tivoli CA uni center or HP Open view SNMP is required 68 IBM Tivoli Monitoring Troubleshooting Guide Table 28 Frequently asked questions for UNIX based systems installation continued Problem Solution Pressing the backspace key characters such as A and AH appear on the screen The backspace key appears to be working correctly when entering text but you later find characters such as A and H in configuration files and your software malfunctions When running the install sh script on a Linux system I get a Memory fault core dump at different random stages of the install regardless of what selections I make If you receive one of these symptoms when using the backspace on UNIX computers you have incorrectly configured the backspace key Configure your terminal and stty erase to use the same key code for backspace Consider using as the key code Verify your configuration with the IBM Tivoli Monitoring distributed utility Install BackspaceCheckUtility When I run the command getconf GNU_LIBPTHREAD_VERSION on my system the response I receive is linuxthreads 0 10 or something similar This is caused by the etc profile entry of LD_ASSUME_KERNEL 2 4 If I unset this variable or change the value of etc profile to 2 6 the getconf command
90. or aix533 3 Reconfigure the portal server and disable LDAP authentication using the itmemd config A cq command invoked from the installation directory 4 Start the portal server with the itmcmd agent start cq command invoked from installation directory The portal server authentication through the monitoring server is now enabled 5 If the monitoring server was also configured to use LDAP and the reason for this procedure being applied is LDAP being out of service ensure you also change the monitoring server configuration to not authenticate through LDAP following steps from the monitoring server configuration help 1 For Windows systems stop the portal server service using the Manage Tivoli Enterprise Monitoring Services application 2 Run the disableLDAPRepository bat script from candle_home CNPSJ scripts 3 Reconfigure the portal server using the Manage Tivoli Enterprise Monitoring Services application and disable the Validate User with LDAP option 4 Start the portal server service using the Manage Tivoli Enterprise Monitoring Services application The portal server authentication through the monitoring server is now enabled 5 If the monitoring server was also configured to use LDAP and the reason for this procedure being applied is LDAP being out of service ensure you also change the monitoring server configuration to not authenticate through LDAP following the steps from the monitoring server configurat
91. password credentials Change the password of Idapuser from the Active Directory system Reconfigure the Tivoli Enterprise Portal Server again for a new password Restart the Tivoli Enterprise Monitoring Server and the Tivoli Enterprise Portal Server By default both the old and the new passwords continue to work for approximately one hour after the password change After one hour the old password stops working Windows 2003 Service Pack 1 introduced this behavior into Active Directory Please refer to Microsoft KB article 906305 for information on what occurs and for instructions on disabling the behavior if necessary 122 IBM Tivoli Monitoring Troubleshooting Guide There is a memory leak in the Tivoli Enterprise Portal browser client when the number of workspace switches increases When using the Tivoli Enterprise Portal browser client on a Windows system with Internet Explorer 6 x 7 x or Mozilla Firefox 2 x a memory leak might be observed that continues to grow as the number of workspace switches increases This problem is due to a handle leak introduced by the IBM Java 1 5 plug in component The same problem is present in the Sun Java 1 5 plug in component Restart the Tivoli Enterprise Portal browser client whenever the memory utilization of the browser being used Internet Explorer or Firefox becomes excessive Typical symptoms of this problem include poor response time of the Tivoli Enterprise Portal client as well as out of memo
92. portal server Run the portal server trace collect logs and call IBM Software Support The following tables include possible causes and corresponding solutions for the problems described in this section Table 4 Correcting monitoring agent problems Monitoring agent has been started but the Navigator has not been refreshed in the portal client Diagnostic and corrective actions 1 Open the Managed System Status workspace and click Refresh 2 Make sure the monitoring agent is connected to the correct monitoring server 3 Check the status of the monitoring server the monitoring agent is connected to Proceed to Table 5 on page 16 Chapter 3 Solving common problems 15 Table 5 Correcting monitoring server problems e Remote monitoring server has shut down e Loss of connectivity between the monitoring agent and the remote monitoring server to which it reports or between that remote monitoring server and the hub monitoring server e You receive the following message in the monitoring server log KDS9151E The heartbeat from remote TEMS variable was not received at its scheduled time and the remote TEMS has been marked offline Diagnostic and corrective actions 1 Check the Managed System Status workspace in the portal client 2 If the monitoring agent is connected through a remote monitoring server confirm that the remote monitoring server is connected to the hub monitoring server 3 If
93. possible causes and corresponding solutions for the problems described in this section Table 9 Correcting storage or memory problems Specific problem that leads to lack of storage or memory Diagnostic and corrective actions e Reconfigure the Java Control Panel See Tivoli Enterprise Portal has hig Table 10 Correcting configuration problems Incorrect configuration Diagnostic and corrective actions e Disable DirectDraw See DirectDraw thread loops infinitely causing poor Tivoli Enterprise Portal performance Chapter 3 Solving common problems 21 Historical data is missing or incorrect The following problems might occur e Workspace is missing historical data e Workspace graphs or tables contain short term but not long term historical data By default long term historical data is older than 24 hours Summarized historical data is not displayed e You suspect that the values returned for historical data are incorrect Preliminary diagnostics 1 Verify component connectivity through the Self Monitoring Topology workspace To open this workspace right click the Enterprise Navigator item and then select Workspace gt Self Monitoring Topology You can also use the workspaces assigned to the Warehouse Proxy and Summarization and Pruning Agent in the physical Navigator view 2 Verify the configuration of historical data by using the portal client or by issuing the f
94. properties were not provided when using the p property properties command line option When you have provided the required configuration properties using the p property properties command line option the agent instance is properly deployed Uninstallation is not available for Application Support on Windows systems Currently there is not a workaround Problems installing directly from the zip file Instead of installing directly from the zip file extract the files from the zip and then install from the setup exe file Installation hangs or loops after presenting initial splash screen When installing IBM Tivoli Monitoring or IBM Tivoli Monitoring agents on Windows systems IBM Tivoli Monitoring installation should present a pop up Chapter 5 Installation and configuration troubleshooting 83 requesting reboot if files are locked However there are times when the IBM Tivoli Monitoring installation does not inform you of locked files This can cause the IBM Tivoli Monitoring installation to loop or hang If you experience a delay longer than fifteen minutes during an IBM Tivoli Monitoring Windows installation cancel the IBM Tivoli Monitoring install processes and reboot the system UNIX based system installation problems and resolutions 84 This chapter describes known problems related to installation and uninstallation that might occur with IBM Tivoli Monitoring Message is received about the Korn She
95. redirect the output to a file Review the file afterwards to show which variables are being used 4 Collect the monitoring agent operations log which shows whether reflex automation occurred A monitoring server message log also confirms what error occurred 5 Correct the identified problem Chapter 3 Solving common problems 29 High CPU usage on a distributed system The following problems might occur e Performance degrades or availability is lost because of high processing in an application or a computer e No data is returned in the portal client and the collector log contains the text string Open Probe pipe error See Table 21 on page 32 e Situations alert you frequently about a managed system cycling between online and offline See Table 20 on page 31 Preliminary diagnostics 1 Determine whether an IBM Tivoli Monitoring component is the root cause Another application or process running on the system might be causing high CPU usage On Windows Use the tools and data provided by Task Manager to identify the process causing high CPU usage In the Processes tab you can reorder the processes by CPU usage An example of a process name is kntcma exe for the Windows OS agent On UNIX and Linux Use the top command to display processes using high CPU For UNIX you can also use the ps auxww command 2 Verify the following e Is historical data collection enabled e Is the database undergoing a backup 3 I
96. remote Tivoli Enterprise Monitoring Server See the following example Server list for a HUB CMS LLB entry 1 is ip 10 248 16 1 1918 LLB entry 2 is sna ATOOENO1 K10DSLB CANCTDCS SNASOCKETS 135 GLB entry 1 is ip 10 248 16 1 1918 GLB entry 2 is sna ATOOENQ1 K10DSLB CANCTDCS SNASOCKETS 135 GLB entry 3 is ip 10 248 16 1 1918 GLB entry 4 is sna ATOOENQ1 K10DSLB CANCTDCS SNASOCKETS 135 Server list for a REMOTE CMS LLB entry 1 is ip 10 248 17 2 1918 LLB entry 2 is sna ATOOENO1 K20DSLB CANCTDCS SNASOCKETS 135 GLB entry 1 is ip 10 248 16 1 1918 GLB entry 2 is sna ATOOENO1 K10DSDS CANCTDCS SNASOCKETS 135 GLB entry 3 is ip 10 248 17 2 1918 GLB entry 4 is sna ATOOENO1 K20DSLB CANCTDCS SNASOCKETS 135 IBM Tivoli Monitoring Troubleshooting Guide Confirming the Server list is correct In general the first half of the GLB server list always points to the Tivoli Enterprise Monitoring Server hub The first half of the GLB entries in the Server list are taken from member KDCSSITE of RKANPAR If the LLB entries are derived implicitly and the GLB entries are explicitly configured in the KDCSSITE member you can diagnose and repair the errors in KDCSSITE e The number of LLB entries must be half the number of GLB entries If this is not the case then there might be a mismatch between the number of transports services configured to this Tivoli Enterprise Monitoring Server the KDC_FAMILIES environment variable versus the n
97. report a duplicate CAT and ATR files there might be a name conflict with other installed IBM Tivoli Monitoring emitters Search for errors showing policies ending with a cmd failed reason code In an MVS based Tivoli Enterprise Portal Server a cmd failed reason code indicates the KUMATR and KUMCAT files are missing from RKANDATV If the Universal Agent RASI log contains SNMP trap emitted to destinations s messages but the traps do not display on the 3rd party SNMP Manager console A firewall might prevent communication between the machine where Universal Agent is running and the machine where the SNMP Manager product is running Security rules might filter block or divert SNMP traps Configuration steps might be required on the SNMP Manager product so that it can receive and display the Universal Agent traps The SNMP Manager product might have a log file that indicates whether it receives or rejects the Universal Agent traps If the host name of the Universal Agent SNMP Data Provider is added as a trap destination the Universal Agent Emitter traps do not display in the SNMP MANAGER Trap workspace The NETWORK workspace is empty Ensure the following e The host where the SNMP Data Provider is running is TCP IP configured e You can ping the default router gateway from the host of the SNMP Data Provider e You have initiated network discovery using Take Action gt Man
98. rule set file with a file extension of rls encoding encoding Specifies the character encoding of the class file If this option is not specified a class file being imported into a rule base is opened in UTF 8 character encoding force Adds the rule set to the rule base even if a rule references a nonexistent event class 2 wrb imptgtrule rule set rule pack before after rule_set rule _pack target rule_base where rule_base The name of the rule base that contains the rule base target rule_pack The name of the rule pack to import The rule pack must have been imported into the rule base first rule_set The name of the rule set to import The rule set must have been imported into the rule base first target The name of the rule base target to receive the imported rule set or rule pack You must have first created the target with the crttarget command option after rule_set rule_pack Specifies the name of the rule set or rule pack after which the imported rule set or rule pack should be located before rule_set rule_pack Specifies the name of the rule set or rule pack before which the imported rule set or rule pack should be located For any changes you make to the rule base to take affect you must complete the following steps e Recompile the rule base Load the rule base Stop the Tivoli Enterprise Console server e Restart the Tivoli Enterprise Console server Refer to the IBM Tivoli
99. running Example KUMP_API_VERBOSE Y dpapi log The KUMP_API_VERBOSE option is valuable when debugging an API program that communicates with the Universal Agent API data provider KUMP_HTTP_DEBUG HTTP Data Provider tracing KUMP_ODBC_DEBUG ODBC Data Provider tracing KUMP_SCRIPT_DEBUG Script Data Provider tracing KUMP_SNMP_DEBUG_TRAP KUMP_SNMP_DEBUG_DISCOVERY_ROUTE KUMP_SNMP_DEBUG_DISCOVERY_NETWORK KUMP_SNMP_DEBUG_MIB_MANAGER KUMP_SNMP_DEBUG_MIB_IO SNMP Data Provider tracing All of the debug environment variables listed default to No As an example if you use the SNMP Data Provider and have problems collecting MIB data you set these two environment variables KUMP_SNMP_DEBUG_MIB_MANAGER Y KUMP_SNMP_DEBUG_MIB_IO Y For a complete list of the Universal Agent environment variables see the IBM Tivoli Universal Agent User s Guide Appendix B Environment variables 239 Tivoli Data Warehouse environment variables The following Tivoli Data Warehouse parameters can be changed in the KHDENV_MIG file to customize tracing for the warehouse upgrade tool Table 56 Tivoli Data Warehouse environment variables Variable KHD_MAX_ROWS_SKIPPED_PER_ TABLE Specifies the maximum number of rows per table that can be Purpose skipped during an upgrade Rows with incorrect data are skipped until the maximum specified number is reached When this number of rows are skipped the m
100. set to Y 244 IBM Tivoli Monitoring Troubleshooting Guide Table 59 Tivoli Enterprise Monitoring Agent deploy environment variables continued Variable Purpose ITM_BINARCH Set by Tivoli Enterprise Monitoring Agent installer to supply the platform architecture code Used by Tivoli Enterprise Monitoring Agent to read the agent installation version files and retrieve the agent version information KBB_RAS1 e View state of main Tivoli Enterprise Monitoring Agent functions like situation and report processing ERR UNIT KRA ST e Detailed debug messages for Tivoli Enterprise Monitoring Agent functions ERR UNIT KRA ALL e View state of short term history data uploads to Tivoli Data Warehouse ERR UNIT KHDX ST e Detailed debug messages of short term history data uploads to Tivoli Data Warehouse ERR UNIT KHD ALL KHD_HISTRETENTION Specifies the default retention period in hours for the short term history files default is 24 hours This can be used to reduce the amount of data kept on disk after a successful upload to the warehouse is performed TIMEOUT Specifies the time in seconds that Agent Deployment tool has to complete a task If the tool does not complete in the task in the time specified by the TIMEOUT value the task is terminated The default value is 600 seconds Appendix B Environment variables 245 246 IBM Tivoli Monitoring Troubleshooting Guide Appendix C
101. sets in which the data is stored have filled up so that the data cannot be added situations definitions may not be installed or the definitions may be incomplete If application support has been installed check the NonResSeedkpp 1og files in install_dir cnps logs for errors where pp is the two letter product code of a monitoring product for which you installed support Any SQL1_OpenRequest status 81 errors may indicate that you have a VSAM I O error Workaround If you see this error check data sets whose names end in RKDS to determine if they are out of space or have run out of extents For example amp rhilev amp rte amp vsamfsv RKSSSITF where amp rhilev is the VSAM runtime high level qualifier amp rte is the RTE name and amp vsamvsf is the monitoring server EIB VSAM low level qualifier Refer to the TEMS started task to see a complete list of VSAM EIB files If the data sets are out of space 1 Use IDCAMS to copy the data to a flat file 2 Delete the existing file 3 Modify the ICAT PP 1xxxx job to increase the size where PP is the two letter product code for the product DS for a standalone Tivoli Enterprise Monitoring Server and xxxx is the RTE JCL suffix as follows a Invoke the Configuration Tool by executing this TSO command EX amp shilev INSTLIB where amp shilev is the installation high level qualifier b On the Configuration Tool MAIN MENU enter 3 Configure Products and select the product you are w
102. socket with router well known port The error 13 indicates a permission failure The Universal Agent was not started with a root ID so the trap receiving and router network discovery processes did not acquire the low numbered ports needed The kuma610 continues to run without those two processes Note The kumstrap and kumsnetd processes are required only if you configured the SNMP data provider on a UNIX based system with the following environment variables specified e KUMP_SNMP_MONITOR_TRAP Y e KUMP_SNMP_NET_DISCOVERY Y You can correct the authorization problem either by doing one of the following e Start the Universal Agent with a root ID OR IBM Tivoli Monitoring Troubleshooting Guide Use the SetPerm script in the bin directory to grant root access to the kumstrap and kumsnetd binaries If you use the SetPerm script to grant root access to kumstrap and kumsnetd you do not need to start Universal Agent with a root ID for the two processes to start successfully Because the kumsnetd process has the necessary root authority to acquire port 520 you can ignore the following error messages in the Universal Agent RASI log file kumOsock c 134 KUMO_OpenLocalSocket bind failed for local address UDP socket 67 port 520 errno 13 kums lrtu c 148 KUMS ListenRouterUpdates unable to open socket with router well known port The Universal Agent stops functioning if the system IP address changes while it is running I
103. specify a directory containing the BAROC event class definitions files baroc to be used as input If not provided the TEC_CLASSES subdirectory for the actively loaded rule base is used by default For more information about the Rules Check Utility refer to the Tivoli Enterprise Portal Administrator s Guide 222 IBM Tivoli Monitoring Troubleshooting Guide The NetView menus are unavailable Ensure the NetView console or Web console are installed on the Tivoli Enterprise Portal machine and that the NVWC_HOME or NV_DRIVE variables are set correctly Errors occur during installation of IBM Tivoli Enterprise Console event synchronization When installation is complete the results are written to the itm_tec_event_sync_insta 1og file located in the following directories e Windows The itm_tec_event_sync_instal1 log file is created in the directory defined by the TEMP environment variable To determine where this directory is defined for the current command line window run the following command echo TEMP e UNIX based systems The itm_tec_event_sync_instal1 1log file is always created in the tmp directory The following error is harmless and there is currently no resolution One or more errors occured during the replacement of files tecSyncAl1Filel with files tecSyncAllFilel Refer to install log for more details One or more errors occured during the replacement of files tecSyncA11File2 with fil
104. the above script run the SetPerm command which is located under install_dir bin directory This command sets root permission for certain UNIX based systems agent files How can I set the trace option to capture any abends core files Add the following in the agent ini file For an example if it is KUX agent add the following line in install_dir config ux ini file KBB_SIG1 trace dumpoff In an environment of 50 servers with at least one agent per server a new agent vt was installed outside the firewall The new agent must be configured on Tivoli Enterprise Monitoring Is it necessary to change all the other UNIX based systems agents for IP PIPE Does SNMP need to be turned on to monitor UNIX based systems host The monitoring server is running WINNT4 0 and monitoring agent is running on HPUX Server for IP PIPE communication Is it not necessary to change all the other UNIX based systems agents for IP PIPE You have to configure only the agent which connects to the Tivoli Enterprise Monitoring Server through a firewall However you must configure the Tivoli Enterprise Monitoring Server for IP PIPE communication While configuring the agent which communicate through the firewall you get the following options Does the agent connect through a firewall YES or NO Default is NO IP PIPE Port Number Default is 1918 e Enter name of KDC_PARTITION Default is null If you are communicating
105. the install_dir logs kfwservices msg file on UNIX based systems To use the new logging facility for the Tivoli Enterprise Monitoring Server modify the install_dir cms KBBENV file on Windows systems or the install_dir config hostname_ms_TEMS ID config file and install_dir config kbbenv ini file on UNIX based systems Add the following line to the file MSG_MODE kms To disable the new logging facility and return to original logging either remove this line in the file or change it to MSG_MODE MSG2 To use the new logging facility for the Tivoli Enterprise Portal Server modify the install_dir cnps kfwenv file on Windows systems or the install_dir config cq ini file on UNIX based systems Add the following line to the file MSG_MODE kcq To disable the new logging facility and return to original logging either remove this line from the file or change it to MSG_MODE MSG2 When you have enabled the new logging facility the Tivoli Enterprise Monitoring Server writes a new log file install_dir itmLogs itmc_hostname_kms 1log The Tivoli Enterprise Portal Server also writes to a new file install_dir itmLogs jtmc_hostname_kcq 1og The properties file install_dir itmLogs itmc_kms properties for the Tivoli Enterprise Monitoring Server and the install_dir itmLogs itmc_kcq properties file for the Tivoli Enterprise Portal Server determines the maximum size and the number of rolling log files The default is 100000 bytes per file and
106. the remote monitoring server is not running and you are an administrator restart it Otherwise notify an administrator and wait for the remote monitoring server to be restarted 4 Running the following ITMSUPER tools might also provide more information e Topology tool e Connectivity tool e Agent Response Time tool e TEMS Server Analysis tool 5 Correct the connectivity failures identified 16 IBM Tivoli Monitoring Troubleshooting Guide Tivoli Enterprise Portal Server does not start or stops responding The following problems might occur e Portal client logon fails See Tivoli Enterprise Portal logon fails on page 20 e The portal server stops responding during normal operation of the portal client e You receive message KFWITMO91E View not available at this time e You receive message KFWITMO10I Tivoli Enterprise Portal Server not ready e You receive message KFWITM402E Communication with the Tivoli Enterprise Porat Server could not be established e You find a similar text string to KFWDBVER version not found when trying to start the portal server See Table 6 on page 18 Preliminary diagnostics 1 Refer to the IBM Tivoli Monitoring Messages reference guide for more information about any messages received Operator responses and general information are provided for each message 2 Allow the portal client enough time to establish a connection with the portal server 3 Is DB2 running Yes See step
107. the target operating system This occurs when using a tacmd command related to agents like tacmd getdeploystatus or tacmd addsystem 186 IBM Tivoli Monitoring Troubleshooting Guide Ensure that you are using the right format for the product code It must be a 2 digit product code as in um for the Universal Agent and not kum You receive a message when trying to use the tacmd maintagent command The tacmd maintAgent command is disabled for the IBM Tivoli Monitoring v6 2 2 release The command stops and starts situations on individual agents without notifying the Tivoli Enterprise Portal Server or Tivoli Enterprise Monitoring Server therefore the Tivoli Enterprise Portal Server or the Tivoli Enterprise Monitoring Server can potentially lose track of the state of the situation on the agent After running the maintAgent command the only way to check if the situation is running is to look at the agent s startup log Endpoint fails to connect to monitoring server when running createnode from a monitoring server in a different domain When running the tacmd createnode command from a hub or remote monitoring server to an endpoint that is in a different domain than the connecting monitoring server the endpoint might fail to connect back to the monitoring server If the failure occurs it could be due to the fact that the endpoint cannot resolve the provided hostname to the fully qualified hostname for example the hostname is itmserv
108. this temporary file is left in the specified directory location On Windows systems the temporary file name is prefixed with cxm followed by random characters and has a tmp filetype for example cxm1C tmp On UNIX systems the temporary file name is prefixed with cxp followed by alphanumeric characters for example cxp5pYmUa Over time this presence of potentially large temporary files could present problems that ultimately result in future getfile command or putfile command failures due to lack of space As a result any temporary files should be deleted periodically Receive an OutOfMemory exception when using the import or export commands Edit the environment variable TACMD_JVM_MAX_MEMORY and override the default maximum JVM memory value size of 256 MB You can edit the value to be between 256 and 2048 inclusive Incorrect values or values out of range are disregarded with an error written to the kuiras1 trace log The suggestbaseline or acceptbaseline commands fail You receive the following type of error KUICACO14E The command failed because the Tivoli Data Warehouse did not contain any historical data for the managed system s for attribute NT_Process_64 Processor Time for the specified time period The warehouse did not contain any historical data from the managed system for the specified situation attribute during the specified time period The time period is established by the start time and end time an
109. to the IBM Tivoli Monitoring product install directory default is c IBM ITM e Navigate to the appropriate subdirectory right click one of the executable files listed below and select Properties itm_home CMS cms exe itm_home CMS kdsmain exe itm_home CMS kdstsns exe itm_home Instal1ITM kinconfg exe e When the Properties window appears select the Compatibility tab e In the Privilege Level section check the box next to Run this program as an administrator e Click OK e Repeat this procedure for each of the files listed After an upgrade the Tivoli Enterprise Portal Server is in the stop pending state and cannot be manually started After the upgrade the Tivoli Enterprise Portal Server was in the stop pending state Attempts to manually start the Tivoli Enterprise Portal Server failed End the kfwservices process from the Windows task manager and then attempt the manual start Your Tivoli Enterprise Portal Server can then be started and stopped This behavior can happen if a program that locks files is running in the background such as Norton Antivirus Chapter 5 Installation and configuration troubleshooting 81 82 When running the setup exe an unknown publisher error message displays If you run the setup exe from a network drive on Windows a window displays with the following message File Download Security Warning The Publisher could not be verified Are you sure you want to run this s
110. to use the substring or scan for string functions instead of the default value and equals The equals operator only works with full matches IBM Tivoli Monitoring Troubleshooting Guide Too much historical data is collected The Summarization and Pruning agent is responsible for generating and storing summarized data and pruning the data based on information that is stored in the Tivoli data warehouse The data in the Tivoli data warehouse is a historical record of activity and conditions in your enterprise The size of summarization data that is collected depends on the following criteria e The number of agents collecting data e The number of table collected per agent e The size of the table number and size of columns e The collection interval for example 5 10 15 or minutes Pruning data is deleting old data automatically rather than manually To reduce the data that is collected limit the size of your database tables by regularly pruning old data from the Tivoli data warehouse If you installed the Summarization and Pruning agent your configuration settings are set to default values You can view the current values in the History Collection Configuration window Refer to Changing configuration settings using the History Collection Configuration window in the Tivoli Enterprise Portal in the IBM Tivoli Monitoring Administrators Guide for instructions If you need to install the Summarization and Pruning agent see the IBM Tiv
111. use the following 01 00 01 59 17 00 17 59 KSY_CNP_SERVER_HOST Tivoli Enterprise Portal Server connection default host KSY_CNP_SERVER_PORT Tivoli Enterprise Portal Server connection default port KSY_DAY_AGE_UNITS The minimum age of data for daily data in days before aggregation is done KSY_EVERY_N_DAYS How often to run the schedule in days number KSY_EVERY_N_MINS Indicates the frequency between Summarization and Pruning agent runs when using the flexible scheduling Must be a multiple of 15 minutes with a minimum of 15 and a maximum of 1440 one day KSY_FIXED_SCHEDULE Indicates whether the Summarization and Pruning agent is configured for fixed schedule if set to Y or flexible schedule when set to N KSY_HOUR_AGE_UNITS The minimum age of data for hourly data in hours before aggregation is done KSY_HOUR_AM_PM Whether to run in the AM or PM AM PM KSY_HOUR_TO_RUN The hour of the day to run number KSY_MAX_ROWS_PER_ TRANSACTION Maximum rows per database transaction KSY_MAX_WORKER_THREADS Maximum number of simultaneous worker threads Default is 1 Recommended value is the number of processors on your server minus Appendix B Environment variables 241 Table 58 Summarization and Pruning agent environment variables continued Variable Purpose KSY_MINUTE_TO_RUN The minute of the day to run number KSY_SHIFT1_HOURS A comma separated list of hour
112. what is displayed in the Tivoli Enterprise Portal 160 Large difference in attribute Values between fhe Tivoli Enterprise Portal and Perfmon 160 The Agent Service Interface is not globalized 160 OS agent installation does not detect System Monitor agents 160 Some attribute group names are fe unintelligible toi the History Collection Configuration window 160 History collection fails to dee the most recent 24 hours of data 161 Situations with attributes from more tia 1 group not supported with autonomous agent 161 Failure when importing situation xml file edited with WordPad 161 Printer details of another system 2 are displayed o on the Tivoli Enterprise Portal 161 CTIRA_MAX_RECONNECT_TRIES environment variable is now obsolete 161 Agent goes offline after removing history path 162 Override button is not present for a situation 162 Agent s Management Definition View columns are not showing data lt 162 There is a situation distribution discrepancy if there is a hub monitoring server outage when one or more remote monitoring servers remain active 162 Installing v6 2 2 agent application support on a monitoring server for a prior release causes agents to fall 162 Installing backlevel Windows OS agent on existing environment causes Tivoli Enterprise Monitoring Server not to start 163 Contents vii
113. 0 from TKANMODL that interfere with operation of new version of Tivoli Enterprise Monitoring Server on z OS Chapter 5 Installation and configuration troubleshooting 99 Table 31 TKANMODL obsolete members KLXABIND KLXACLOS KLXAHBAD KLXAHBNM KLXAINIT KLXAINTE KLXAOPEN KLXAPEER KLXARECV KLXASELF KLXASEND KLXASLIH KLXAWAIT KLXAXDAT KLXAXTPL KLXI INT KLXI IUC KLXIDESP Where Remote Deployment of agents is not supported Remote Deployment is not supported for OMEGAMON agents It is also not supported in environments with a z OS Tivoli Enterprise Monitoring Server Uninstallation problems and workarounds 100 This section provides information about how to troubleshoot an uninstallation Unable to uninstall multi instance agent from a managed system on windows 64bit When uninstalling a multi instance agent from a managed system that also has an IBM Tivoli Monitoring v622 Fix Pack 2 agent installed instances are not uninstalled leaving an entry in the MTEMS This only occurs on the Windows 64 Bit platform Other IBM Tivoli Monitoring components running on the system continue to operate normally Remove the agent instances first and then uninstall the agent If the agent was already uninstalled you can re install the agent again remove the instances and then uninstall the agent again Prompted for msi file during uninstallation pro
114. 000000100000000000000F84CD256887CAE56E8F 400570000000048010000EE5DB656B88B9655A4F70057220000002200000000000000D8F900572C F800578C81C056ACEF0057C04DD256484DD2563EB4B656C04DD2560000000000F5005701 000000000000002000000002000000879D8B558C81C056A8EF0057 The expected seeding completion messages will follow the crash message The normal seeding messages are checked by the IBM Tivoli Monitoring configuration IBM Tivoli Monitoring Troubleshooting Guide tool for successful completion of all seeding functions The crash message always appears in the Tivoli Enterprise Monitoring Server seeding output even if a core or dump file is not produced Tivoli Enterprise Monitoring Server seeding output files are stored in different files on UNIX and Windows systems UNIX examples ITM_HOME 1ogs Node_ci_query_Process_ID log ITM_HOME logs Node_ci_query_Process_ID log where Node The system host name Process_ID The program process ID Windows example C IBM ITM CNPS logs seedApp log where App The 3 character product code such as knt for the Monitoring Agent for Windows OS The exact cause of the Tivoli Enterprise Monitoring Server seeding program crash has not been determined The program has finished all seeding functions and is exiting The crash has only occurred when only a few seeding changes are required Seeding functions making many updates to the Tivoli Enterprise Monitoring Server tables have never resulted in this type of program
115. 010 139 140 e The number of network interface cards resident on any of the Warehouse Proxy Agent systems e The total number of Warehouse Proxy Agent that have been configured and connected to the hub monitoring server The following example of Global Location Broker entries for the Warehouse Proxy Agent is from a remote monitoring server RAS1 log when the remote monitoring server has the configuration parameters KDC_DEBUG Y and KDE_DEBUG Y set in its environment 4A8F0367 007D object 85f536a00000 02 0a 09 fe 31 00 00 00 4A8F0367 007D type 85f532330000 02 0a 09 fe 31 00 00 00 4A8F0367 007D interface 865fc14a0000 02 0a 09 fe 31 00 00 00 4A8F0367 007D annotation Candle Warehouse Proxy 4A8F0367 007D flags 0x2 addr len 16 4A8F0367 007D saddr ip 9 77 148 246 205 In some cases the annotation value of Candle_Warehouse_Proxy is not present but the values for the object type and interface match those shown above As an alternative to viewing the Global Location Broker entries using the RAS1 log you can use the Manage Tivoli Enterprise Monitoring Servers workspace to view this information After navigating to the Manage Tivoli Enterprise Monitoring Servers workspace select the Protocols link for the hub monitoring server to view the global location broker entries For further information about the Manage Tivoli Enterprise Monitoring Servers workspace see the IBM Tivoli Monitoring Tivoli Enterprise Portal User s Guide
116. 11 dbisk candleUa410 s01276 um work KUMSNETS open failed ErrorText lt No such file or directory gt Load network configuration bypassed 18 kbbssge c 52 BSS1 GetEnv KUMP_SNMP_NET_DISCOVER_ENTERPRISE N Network configuration discovery tasks started 17 kumsrcfg c 267 KUMS ReadRouterConfigFromExt Router configuration file control1 dbisk candleUa410 s01276 um work KUMSROUT open failed ErrorText lt No such file or directory gt Load router configuration bypassed 17 kumsrcfg c 294 KUMS_ReadRouterConfigFromExt 0 router record s loaded from router configuration file contro11 dbisk candleUa410 s01276 um work KUMSROUT 19 kbbssge c 52 BSS1 GetEnv KUMP_SNMP_MANAGE_LOCAL_NETWORK N 16 kumsp lst c 167 KUMS PrepareListenSNMPtrap Unable to open SNMP trap listen port 16 kumsplst c 193 KUMS_PrepareListenSNMPtrap Note Determine if another SNMP Manager process is active on this system Monitoring of SNMP traps disabled No trap receiving port or API available Insufficient process authority for executing ICMP procedures 17 kum0sock c 132 KUMO OpenLocalSocket bind failed for local address UDP socket 71 port 520 error 13 17 kums lrtu c 148 KUMS ListenRouterUpdates unable to open socket with router well known port 17 kum0sock c 132 KUMO_ OpenLocalSocket bind failed for local address UDP socket 73 port 520 error 13 17 kums rtu c 148 KUMS ListenRouterUpdates unable to open
117. 2 Java errors e RedHat 5 0 ships with SE Linux turned on by default It has to be set to permissive in order for the install to run Edit the etc selinux config file to specify SELINUX permissive e A Java library is missing classes Install the following to get libXp so 6 libXp 1 0 0 8 s390x rpm libXp 1 0 0 8 s390 rpm e The just in time compiler JIT needs to be turned off Issue the export JAVA_COMPILER NONE command before issuing install sh On HP UX systems where the hostname does not equal the nodename the upgrade installation fails to stop running processes Running IBM Tivoli Monitoring processes were not shutdown by the installer during the upgrade of the Tivoli Enterprise Monitoring Server or agents when the nodename does not equal the hostname on HP UX systems On HP UX if the system has a hostname longer than 8 characters then the nodename should be set to 8 characters or less If the nodename and the hostname are not in sync then the you must shutdown all IBM Tivoli Monitoring processes before starting an addition product installation or upgrade EIF Slot Customization does not work on upgraded zlinux systems When you open the EIF Slot Customization Editor from the Situation Editor slot names under the Base Slots panel do not display for the class Clicking Select Event Class does not cause default event classes to display When typing a new event class name slot names under the Base Slots panel do not displa
118. 3 files You can modify these values by changing these parameters in the properties file fh maxFileSize 1000000 fh maxFiles 3 When the log file exceeds the maxFileSize the file moves to a new name for example itmc_hostname_kms1 10g and new messages are then written to the original file name for example itmc_hostname_kms 10g The process continues for the number of maxFiles z OS systems To use the new logging facility for the Tivoli Enterprise Monitoring Server modify the RKANPARU member Add the following line to the file MSG_MODE kms To disable the new logging facility and return to original logging either remove this line in the file or change it to MSG_MODE MSG2 The number of log datasets is determined by how many datasets are defined in the Tivoli Enterprise Monitoring Server JCL procedure The amount of data that can be written to each dataset depends on the amount of space allocated at the time each dataset is created IBM Tivoli Monitoring Troubleshooting Guide To use IBM Tivoli Monitoring Operations Logging on z OS systems complete the following procedure 1 Create the log datasets You must create at least one to use the new logging facility but it is best to create three The only limit to the number of datasets that you create is the limit imposed by the system typically about 70 Each dataset must reside on a disk device The DCB attributes are DSORG PS RECFM VB LRECL 256 The BLKSIZE specification
119. 6 4968 In other countries contact your software account representative to order Tivoli publications To locate the telephone number of your local representative perform the following steps 1 Go tofhttp www elink ibmlink ibm com publications servlet 2 Select your country from the list and click Go 3 Click About this site in the main panel to see an information page that includes the telephone number of your local representative Accessibility Accessibility features help users with a physical disability such as restricted mobility or limited vision to use software products successfully With this product you can use assistive technologies to hear and navigate the interface You can also use the keyboard instead of the mouse to operate all features of the graphical user interface Visit the IBM Accessibility Center at http www ibm com alphaworks topics for more information about IBM s commitment to accessibility Tivoli technical training For Tivoli technical training information refer to the following IBM Tivoli Education Web site at http www ibm com software tivoli education 248 IBM Tivoli Monitoring Troubleshooting Guide Tivoli user groups Tivoli user groups are independent user run membership organizations that provide Tivoli users with information to assist them in the implementation of Tivoli Software solutions Through these groups members can share information and learn from the know
120. 74 On distributed systems the data is written to the local file system When the Warehouse Proxy Agent receives the data it copies the data to the Tivoli Data Warehouse If the Tivoli Data Warehouse is not available the collected data could take up a significant amount of memory On z OS systems the data is written to the persistent datastore Maintenance procedures must be installed and workable to handle cases other then simply saving copies of the data These procedures are used to recover corrupted datasets and to handle migration issues When the persistent datastore is started it looks at the status of a dataset and determines if it has corrupted data If there is corruption it launches maintenance with the options to export the data reallocate and initialize the dataset and then restore the exported data Also when the persistent datastore is started it compares the information in the dataset against the current configuration to see if any table structures have changed When it detects a change it goes through the same process that effectively does a database REORG If you do not have the maintenance procedures installed and usable the datasets might become unusable and therefore there might be a loss of data If maintenance is set up so that the data is rolled over the data that would have been copied to the Tivoli Data Warehouse is copied over after a set period of time You can set maintenance to roll off the data For more info
121. A reports depending on how the OSA adapter is configured Confirm that the values for these attributes are being displayed in Tivoli Enterprise Portal If no data is being collected for a key attribute because of configuration issues consult the product manual to see what configuration change would provide data for the missing attributes To determine the cause of this problem you need to enable detailed logging in Tivoli Common Reporting as described in the IBM Tivoli Common Reporting User s Guide and you need to know which agent table or attribute group contains the attribute that is not being displayed Go to the Tivoli Enterprise Portal console and validate that this column contains data and if the issue is with summarized data that summarization is enabled for this attribute Another reason might be that you have not configured historical collection when you installed and configured the monitoring agent or that not enough time has passed for Tivoli Data Warehouse to collect the data for your requested report time period before you tried to generate an historical report with Tivoli Common Reporting If you do not see the historical collection icon a clock in the workspace in Tivoli Enterprise Portal then data has not been collected in Tivoli Data Warehouse The solution might require changes to the persistent datastore the z OS repository for short term historical data Those changes should be made using the Configuration Tool
122. ARO20E 83 KUICCRO99E 189 KUICLRO99E 148 kumsnetd 196 kumstrap 196 L language packs 131 Icfd log 76 LDAP configuration 146 libraries IBM Tivoli Monitoring 251 Linux installation error 87 LINUX MONITORING AGENT V610Rnnn unable to install agent 87 listSit 176 184 listsystems command 170 listSystems command 189 log turn on Tivoli Enterprise Portal Server trace 42 Log agent UNIX based system 173 Log and Trace Analyzer tool 60 logging built in features 1 logon error messages 107 logs 35 227 agent 37 Agent Deploy tool trace 44 dynamically modifying 51 IBM Tivoli Enterprise Console 38 logs continued IBM Tivoli Enterprise Console Situation Update Forwarder 48 IBM Tivoli Enterprise Console trace 47 IBM Tivoli Monitoring upgrade toolkit 47 IBM Tivoli Universal Agent trace 44 incorrect Tivoli Enterprise Monitoring Server 148 installation 39 RAS1 41 reading 41 reading RAS1 41 Summarization and Pruning Agent 45 tacmd commands 46 Tivoli Enterprise Monitoring Server 35 36 38 Tivoli Enterprise Monitoring Server trace 43 Tivoli Enterprise Portal browser client 35 Tivoli Enterprise Portal desktop client 36 Tivoli Enterprise Portal Server 35 turn on client trace 42 UNIX based system 35 Upgrading from Tivoli Distributed Monitoring 40 Warehouse Proxy agent 37 Warehouse Proxy trace 45 where to find 35 Windows 35 M Managed system 195 manuals see publications 248 memory usage high 126 memory use Tivoli Enterprise Mon
123. AROC files that must be imported to support the IBM Tivoli Monitoring V6 1 events The BAROC files are located in the TECLIB directories on Tivoli Enterprise Monitoring Server e Windows IBM ITM cms TECLIB e UNIX based systems opt IBM ITM tables hubname TECLIB where hubname Specifies the name of the hub server You can run the following command from the IBM Tivoli Enterprise Console event server to import a BAROC file a file of event class specifications into a rule base appending it to the end of the class specifications unless otherwise specified with arguments wrb imprbclass class_file encoding encoding before class_file after class_file force rule base where class_file Specifies the name of the class file to import This must be a path to the BAROC file rule_base The name of the rule base to receive the imported class set after class_file Specifies the name of the class file after which the imported class file is to be placed before class_file Specifies the name of the class file before which the imported class file is to be placed encoding encoding Specifies the character encoding of the class file If this option is not specified a class file being imported into a rule base is opened in UTF 8 character encoding force Imports the class file even if it can cause rule base inconsistency An error message is displayed if the class set file contains syntax errors referenc
124. ASI log file the KBB_RAS1 environment was erased or commented out in the KUMENV file add KBB_RAS1 ERROR to the install_dir logs hostname_um_timestamp log to reactivate Universal Agent RAS1 tracing Set the IBM Tivoli Universal Agent trace from Manage Tivoli Enterprise Monitoring Services Right click the Universal Agent Select Advanced gt Edit Trace Parms Select the RASI filters The default setting is ERROR Accept the defaults for the rest of the fields Click OK to set the new trace options PO NI Click Yes to recycle the service Setting the trace option for the Warehouse Proxy agent 1 On the computer where the Tivoli Enterprise Monitoring Server is installed select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services Right click Warehouse Proxy Select Advanced gt Edit Trace Parms Select the RASI filters The default setting is ERROR Accept the defaults for the rest of the fields Click OK to set the new trace options D aa N 7 Click Yes to recycle the service Setting the trace option for the IBM Tivoli Summarization and Pruning Agent Use the IBM Tivoli Universal Agent trace facility to diagnose problems with the zi i The Summarization and Pruning Agent uses C based RASI tracing Java based RAS1 tracing and Java based internal tracing By default Summarization and Pruning Agent trace data is written to a file in the logs subdirectory The de
125. Agent workspace see Table 21 on page 32 Verify that the monitoring agent configuration is correct Similar problems might exist for other monitoring agents Check that application support has been added See Table 1 on page 13 The following tables include possible causes and corresponding solutions for the problems described in this section IBM Tivoli Monitoring Troubleshooting Guide Table 1 Correcting application support problems Lack of application support or application support level mismatch among components monitoring server portal server desktop and Java Web Start clients and monitoring agents Diagnostic and corrective actions 1 Check application support on the monitoring server portal server and portal client e On Windows Run the kincinfo exe i command in the CANDLE_HOME InstallITM directory to show what is installed e On Linux or UNIX Run the cinfo i command in the CANDLEHOME bin directory to show what is installed e For a monitoring server on z OS Look in the amp rhilev amp rte RKANDATV data set where amp rhilev is the high level qualifier and amp rte is the mid level qualifier of the libraries for the runtime environment where the monitoring server is configured for files named KppCAT and KppATR where pp is the two character product or component code 2 You can also run the TEMS Servers Analysis tool provided by ITMSUPER against the hub monitoring server to ensure th
126. BUFCNT Specifies Tivoli Enterprise Monitoring Server internal table buffer count KGL_CBTBUFSZ Specifies Tivoli Enterprise Monitoring Server internal table buffer size KGL_KEYRING_FILE Specifies LDAP authentication SSL GSKit keyring file KGL_KEYRING_LABEL Specifies LDAP authentication SSL GSKit keyring label KGL_KEYRING_PASSWORD Specifies LDAP authentication SSL GSKit keyring password KGL_KEYRING_STASH Specifies LDAP authentication SSL GSKit keyring password stash file KGL_KGLMSGBASE Specifies the Tivoli Enterprise Monitoring Server KGL message file directory KGL_LDAP_BASE LDAP authentication LDAP search base KGL_LDAP_BIND_ID KGL_LDAP_BIND_PASSWORD LDAP authentication LDAP server bind Distinguished Name DN LDAP authentication LDAP server bind password KGL_LDAP_HOST_NAME KGL_LDAP_PORT LDAP authentication LDAP server host name LDAP authentication LDAP server port KGL_LDAP_SSL_ENABLED Whether to use LDAP SSL communications KGL_LDAP_USER_FILTER LDAP authentication user filter KGL_LDAP_VALIDATE Whether to use LDAP authentication KGL_MSG2_EVENTLOG Whether to format Event Log KGL_MSG2_UNIVERSAL Whether to enable Universal Messages KGL_TRC1 Whether to enable the error log KHD_HISTRETENTION Specifies the default retention period in hours for the short term history files default is 24 hours This can be used to r
127. Broker service initialization GLB service failures occur because there are errors in member KDCSSITE of RKANPAR Each socket address in KDCSSITE is assumed to be the socket address of the Tivoli Enterprise Monitoring Server hub If none of the entries in the KDCSSITE file are the correct socket address of the Tivoli Enterprise Monitoring Server hub this process initialization fails Tivoli Enterprise Monitoring Server hub availability The following message indicates the Tivoli Enterprise Monitoring Server hub is available ko4locbr cpp 731 Mgr locateEverbody 1bLookupHub returned error lt 0 gt ip lt ip 9 42 46 26 21343 gt sna lt gt pipe lt ip pipe 9 42 46 26 21343 gt Do the following if the Tivoli Enterprise Monitoring Server hub is not available e Review the RAS1 log for the Tivoli Enterprise Monitoring Server to ensure it is connected e Review network topology to ensure Firewall policy does not prohibit connection initiation from the Tivoli Enterprise Monitoring Server hub e Review Transport opened on the Tivoli Enterprise Monitoring Server to ensure at least one transport service is common between it and this Tivoli Enterprise Monitoring address space z OS based installations problems and resolutions This section describes problems that might occur with a z OS based installation How do you customize the following Tivoli Enterprise Monitoring Server configuration values using the Configuration tool You can i
128. Check Utility refer to the IBM Tivoli Monitoring Administrator s Guide Expected impacted rules are not displayed when entering multiple event classes with class attributes Ensure you input ran the Rules Check Utility command with the correct options wrules_check h v rd rules_directory cd baroc_classes_directory class attribute attribute classN attributeN attributeN f class_file of output_file For more information about the Rules Check Utility refer to the Tivoli Enterprise Portal Administrator s Guide The wrules_ check command fails When you run the wrules_check command without specifying the cd and rd options you must have permission to run the wrb subcommands Run the wrules_check command with the cd and rd options or refer to the IBM Tivoli Enterprise ConsoleIBM Tivoli Enterprise Console documentation for more information about running the wrb subcommands The following lists the syntax for the Rule Check Utility wrules_check h v rd rules_directory cd baroc_classes_directory class attribute attribute classN attributeN attributeN f class_file of output_file Table 50 Rule Base directory and BAROC event class definition directory Option Description rd Used to specify a directory containing the rulesets rls to be checked If not provided the TEC_RULES subdirectory for the actively loaded rule base is used by default cd Used to
129. E variables Then click OK to save the changes 4 Start the monitoring server Some agents are displayed in the Service Console list that are not accessible from that user interface For instance if you click IBM Tivoli M5 Agent Service Interface a popup is displayed for the User Name and Password There is no valid user name and password combination that will work and you must click Cancel to get out of this page Other agents do not have this problem tacmd login fails after hub monitoring server is recycled The tacmd login process uses SOAP to interface with the hub monitoring server validation process SOAP runs with the monitoring server process and also with the IBM Tivoli Monitoring internal web server All usually run in the same process during a normal startup The IBM Tivoli Monitoring internal web sever process runs on the first IBM Tivoli Monitoring process started up If that first process stops the web server swaps to another IBM Tivoli Monitoring process The web server supports the service console and port forwarding logic as well as SOAP IBM Tivoli Monitoring Troubleshooting Guide tacmd When the monitoring server is stopped the internal web server swaps to another IBM Tivoli Monitoring process such as an OS Agent When the monitoring server starts again the monitoring server and SOAP are running but the internal web server is not in the same process You can determine which process is running the internal web ser
130. ECT _TRIES Number of consecutive times without success the agent will attempt to connect to a Tivoli Enterprise Monitoring Server before giving up and exiting Default is 720 This environment variable is now obsolete If you decide to use this variable and the number of connection attempts to the monitoring server exceeds CTIRA_MAX_RECONNECT_TRIES the agent attempts to shutdown If the Agent Management Services Watchdog is running it immediately restarts the agent If you want the agent to shutdown when CTIRA_MAX_RECONNECT_TRIES is exceeded this Watchdog process must be disabled Use the AMS Stop Management action to disable this watchdog process CTIRA_NCSLISTEN Number of RPC listen server threads to create for the agent Default is 10 CTIRA_NODETYPE Supply the agent node type qualifier subsystem hostname nodetype of the agent managed system name MSN Provide the agent product indicator in the MSN Value can also be set by the Tivoli Enterprise Monitoring Agent API CTIRA_OS_INFO Override default value for agent entries in the Tivoli Enterprise Monitoring Server ManagedSystem Host_Info column Used to build Tivoli Enterprise Portal Server navigation tree Value must match existing entry in the CNPS osnames file Not applicable to subnode type records in the ManagedSystem table CTIRA_PRODUCT_SEP Supply alternate qualifier for the agent managed system name MSN Default is a colon character CTIRA_
131. ESPACE_NAME Enabled Set FREEBYTES to zero Navigation Tree Databases gt Database Summary Workspace Oracle Database Database Summary gt Database Summary Bar Chart View Oracle Database Database Summary gt Database Summary Table View Column DB Percent Free Space 0 System TS Percent Free 0 Navigation Tree Databases gt Enterprise Database Summary Workspace Oracle Statistics _Enterprise Databases Global gt Database Summary Bar Chart View Oracle Statistics Enterprise Databases Global gt Database Summary Table View Column System TS Percent Free 0 Situation Oracle_DB_PctFree Space_Low always true Oracle SystemTS_PctFree_ Critica always true Oracle SystemTS_PctFree_ Warning always false For more information on the cursors refer to the Oracle Agent 6 2 0 TIV ITM_ORA LA0001 README or a higher version of the README 34 IBM Tivoli Monitoring Troubleshooting Guide Chapter 4 Tools IBM Tivoli Monitoring provides several tools some include functionality for diagnosing problems The primary diagnostic tool of IBM Tivoli Monitoring is logging Logging refers to the text messages and trace data generated by the software Messages and trace data are sent to an output destination such as a console screen or a file Trace logging Trace logs capture information about the operating environment when component software fails to operate as intended The principal log type is the reliability availability
132. Eclipse Help Server and the Tivoli Enterprise Portal Server e Open a DOS Prompt then navigate to the HOME CNB c asses cand e directory e Run the helpmerg bat file e Start the Eclipse Help Server e Start the Tivoli Enterprise Portal Server Note The Tivoli Enterprise Portal Server might not have to be stopped and started for this procedure Uninstallation of an agent occurring more than once stops the OS agent If uninstallation an agent is performed more than once it stops the Monitoring Agent for Windows OS agent as well as other IBM Tivoli Monitoring components Also an error message does not appear in the console After uninstallation Tivoli Enterprise Monitoring Server folder is not deleted There is no negative impact from these files being left on the system Removing a failed installation on Windows The following sections describe the process for removing a release that is partially installed and cannot be removed by using the Add and Remove Programs tool The following topics are discussed Table 32 Removing a failed installation on Windows Where to find information Remove a failed installation from computer Removing a failed first time installation that has never had IBM Tivoli Monitoring or Candle OMEGAMON installed Chapter 5 Installation and configuration troubleshooting 101 102 Table 32 Removing a failed installation on Windows continued Goal Where to find information R
133. Edit Trace Parms to display the Trace Parameters window Select the RASI filters RASI is the unit trace for the Tivoli Enterprise Monitoring Server The default setting is ERROR Note There must be a space between each UNIT trace setting For example ERROR UNIT kdy all UNIT kfaprpst all Accept the defaults for the rest of the fields Click OK to set the new trace options Click Yes to recycle the service On UNIX systems i 2 Always backup the files before altering them Set the following variable in the ms ini file in the lt CANDLEHOME config directory KBB_RAS1 ERROR UNIT filter trace level where filter is the component you want to trace and trace_level is the level of tracing you want The following example traces everything in the Deploy component KBB_RAS1 ERROR UNIT KDY ALL Note There must be a space between each UNIT trace setting For example KBB_RAS1 ERROR UNIT KDY ALL UNIT KFAPRPST ALL Chapter 4 Tools 43 44 3 Set the following variable in CANDLEHOME bin tacmd to trace the command line interface of the Tivoli Enterprise Monitoring Server KBB_RAS1 ERROR UNIT filter trace_level 4 Regenerate the host_name_ms_TEMS_NAME config file by running the itmemd config S h install_dir a arch t tems_name command 5 Recycle the Tivoli Enterprise Monitoring Server by restarting or stop and then start The command syntax for starting and stopping the monitoring server is itm
134. F111E Monitoring server is too busy For the executecommand command KUIPTF111E Monitoring server is too busy Reconfiguring an agent and then getting the deploy status yields a port number message When reconfiguring an agent by running the tacmd configuresystem command a subsequent tacmd GetDeployStatus command yields a message like the following KDY0030E lookupUAPort failed Operation to determine the port number used by Universal Agent agent was not successful If installing from images on an NFS mount the NFS mounts need world execute permissions to be accessible by the process doing the distribution tacmd getfile or putfile command is failing On UNIX and Linux systems either of these commands fail if the requested file s size is larger than the user limit file size defined by the ulimit command When using the tacmd getfile command ensure that the local system s ulimit f setting can accommodate the requested file s size When using the tacmd putfile command ensure that the remote system s ulimit f setting can accommodate the specified file s size 182 IBM Tivoli Monitoring Troubleshooting Guide Temporary files remain when tacmd getfile or putfile is interrupted When issuing a tacmd getfile command or tacmd putfile command the file is copied as a temporary file to the directory location as specified by the d destination option If the command is interrupted for example if the session is closed
135. I filters in the process If you set this global class by itself it is global in scope and the trace cannot filter on any of the other keywords Separate combined classes with a space The following values are possible Valid abbreviations are in parentheses ERROR ER returns severe error messages only this is the default for most applications STATE ST records the condition or current setting of flags and variables in the process If state tracing is enabled you can see the current state of particular variables or flags as the process is running FLOW EL causes a message to be generated at an entry or exit point of a function DETAIL DE produces a detailed level of tracing INPUT IN records data created by a particular API function or process ALL causes all available messages to be recorded This setting combines all the other forms of tracing COMP Indicates that the trace includes a component type The COMP keyword is used to trace groups of routines related by function or component Use this keyword only at the explicit request of an IBM Software Support representative component_type Identifies a component type An IBM Software Support representative can tell you what value to specify ENTRY Narrows a filtering routine to specify a specific ENTRY POINT Since multiple entry points for a single routine are rare use this keyword only at the explicit request of an IBM Software Support representative
136. ISPEXEC TBOPEN KCITVARS WRITE SHARE ISPEXEC TBSORT KCITVARS FIELDS amp SORTLIST ISPEXEC TBCLOSE KCITVARS PAD 30 EXIT CODE 0 3 Invoke ICAT 4 On the ICAT main menu select Configure products gt Services and utilities gt Execute a CLIST in the TKANCUS library option 5 On the next panel specify XSORT as the Name Ensure that the panel displays a Completion Code 0 message after this CLIST is invoked 6 Navigate to the RTE main menu and regenerate the batch parameter member for RTE SYP1 Ensure that all the applications are now generated Cannot encrypt text A call to CSNBSYE failed Cannot encrypt contents of keyfile On the Specify configuration values option on the Configure the Tivoli Enterprise Monitoring Server main menu you can provide the Integrated Cryptographic Service Facility ICSF related values for password encryption These values generate the KAES256 step in the Tivoli Enterprise Monitoring Server Create runtime members job To create the encryption key file KAES256 in the amp rte RKANPAR library If within this step the error occurs this message indicates that ICSF is not configured correctly in your system Consult with your system administrator Refer to the Configuring Tivoli Enterprise Monitoring Server on z OS manual for more information about configuring a z OS Tivoli Enterprise Monitoring Server and the security related information The error KLVST005 MVS JOBSTEP AUTHORIZATION REQUIRED KLVST001 CA
137. It simply means the application did not provide an upgrade seeding file There are generally two types of seeding files install and upgrade The installer determines which one to apply by checking to see if there are already situations belonging to the application on the target hub If no situation is found then the installation seeding file is chosen otherwise the upgrade seeding file is used if provided In a hot standby setup as soon as one hub is seeded the other hub can copy the situations immediately So when seeding is applied to the second hub the installer detects existing situations and looks for the upgrade seeding file instead Even though some applications do not provide 69 Chapter 5 Installation and configuration troubleshooting 70 upgrade seeding files because the hubs automatically synchronize seeded data it is generally not a serious issue Seeding can still be forced on the second hub by using the f option Many files in the First Failure Data Capture log directory On Windows systems there are eWAS logs in the following location of the IBM Tivoli Monitoring home directory CANDLE_HOME CNPSJ profiles ITMProfile logs ffdc And on UNIX systems they are found in the following directory CANDLE_HOME arch iw profiles ITMProfile logs ffdc These log files might contain the following exceptions org omg CORBA BAD_OPERATION CORBA TRANSIENT ClassNotFound on MQJMS These exceptions can be ignored and have n
138. L Specifies the Tivoli Enterprise Monitoring Server catalogue CAT files directory 238 IBM Tivoli Monitoring Troubleshooting Guide Table 54 Tivoli Enterprise Monitoring Server environment variables continued Variable Purpose SOLLIB directory Specifies the Tivoli Enterprise Monitoring Server seeding and query SQL files Universal Agent environment variables The following table lists environment variables you can set to customize tracing for the Universal Agent Table 55 Universal Agent environment variables Variable Purpose KBB_RAS1 ERROR UNIT kumamain ALL Problems involving managed system online offline processing KBB_RAS1 ERROR UNIT kumpdpda Error Output UNIT kumpmd2a Error Detail Incorrect report data KBB_RAS1 ERROR UNIT kumpfile Error State Detail Flow Metrics UNIT kumpdcmf ALL KBB_RAS1 ERROR UNIT kumpsosr ALL UNIT kumpspst ALL UNIT kumpscku ALL UNIT kumpstcp ALL UNIT kumplpba ALL KBB_RAS1 ERROR METRICS Detailed File Data Provider tracing Detailed API or Socket Data Provider tracing Problems involving Universal Agent memory usage KUMA_DCH_TRAPEMIT SNMP Emitter tracing Use to display emitted traps in the UAGENT Action report KUMA_VERBOSE Tracing a Universal Agent API client program This variable must be set on the machine where the API client program is executing not where Universal Agent is
139. Monitoring processes 233 Appendix B Environment variables 235 Common environment variables 235 Tivoli Enterprise Portal Server environment variables 236 Tivoli Enterprise Monitoring Server envitonineni variables E Ee Universal Agent environment variables sos a 239 Tivoli Data Warehouse environment variables 240 Tivoli Enterprise Monitoring Agent deploy environment variables a 242 Appendix C Support information Searching knowledge bases Searching the information center x Searching the Internet Obtaining fixes x Accessing terminology online Accessing publications online Ordering publications Accessibility Tivoli technical training Tivoli user groups 247 247 247 247 247 248 248 248 248 248 249 Appendix D Documentation library 251 IBM Tivoli Monitoring library 251 Documentation for the base agents 4252 Related publications 253 Other sources of documentation 253 Notices 255 Trademarks 257 Index 259 Contents X xii IBM Tivoli Monitoring Troubleshooting Guide Chapter 1 Introduction to troubleshooting This guide helps you to decide where to begin looking for causes when you have a problem with IBM Tivoli Monitoring Usually you start with a symptom or set of symptoms and trace them back to their cause This process is called troubleshooting Troubleshooting is not the sa
140. Monitoring agent troubleshooting 171 2 If you cannot find the CTIRA_HOSTNAME environment variable you must add it to the configuration file of the monitoring agent On Windows Use the Advanced gt Edit Variables option On UNIX and Linux Add the variable to the config product_code ini and to config product_code config files e On z OS Add the variable to the RKANPAR library member Kproduct_codeENV e On i5 OS Add the variable to the QAUTOTMP KMSPARM library in member KBBENV Some monitoring agents for example the monitoring agent for MQ Series do not reference the CTIRA_HOSTNAME environment variable to generate component names Check the documentation for the monitoring agent that you are using for information on name generation If necessary contact IBM Software Support Other symptoms that can be observed when multiple instances of a monitoring agent with the same managed system name attempt to connect to a Tivoli Enterprise Monitoring Server include the following e A managed system name s status toggles ON and OFF line constantly as one agent heartbeat overlays the other s information High CPU usage is observed that is caused by a constant thrashing of the Tivoli Enterprise Portal Server or Tivoli Enterprise Monitoring Server Situation distribution Tivoli Enterprise Monitoring Server table relationship updates Tivoli Enterprise Portal Server topology view updates all could be initiated as each agent heartb
141. NDLE ENGINE INITIALIZATION ERROR S ABEND U0012 CSV019I Required module KLVSTWTO not accessed it is not APF Authorized RKANMODL CSV028I ABEND 306 0C occurs in the z OS Tivoli Enterprise Monitoring Server RKLVLOG during startup Ensure that the load libraries including RKANMOD and RKANMODL are correctly APF authorized Additionally in the Configuration tool Complete the configuration step specify that any runtime libraries concatenated in the STEPLIB IBM Tivoli Monitoring Troubleshooting Guide DDNAME and in the RKANMODL DDNAME of the Tivoli Enterprise Monitoring Server started task must be APF authorized The error KLVSQ000 carved mode in effect for extended storage occurred in the RKLVLOG during startup You can increase the MINIMUM storage settings in the amp rhilev amp rte RKANPAR KDSSYNSIN member if IBM Support personnel instructs you to do so The default value for the MINIMUM parameter is MINIMUM 150000 X Use the following steps to increase this value or any other storage related parameters E 5 On the Configure the Tivoli Enterprise Monitoring Server main menu select the Specify configuration values option On the next panel navigate to the F5 Advanced key Do the following on the Specify Advanced Configuration Values e Edit the Minimum extended storage field to 300000 K e Edit the Maximum storage request size fields to 16 Primary and 23 Extended On the Configure the Tivoli Enterprise Moni
142. OY Group Window tacmd creategroup g Window t DEPLOY d Windows Deploy Group echo Adding ACHAN1 to Window DEPLOY group tacmd addgroupmember g Window t DEPLOY m achanl raleigh ibm com p KDYRXA RXAusername achan KDYRXA RXApassword xxxxx KDYRXA instal 1Dir C data echo Issuing group DEPLOY of Windows OS agent tacmd createNode g Window Transaction ID 1222782542297000000015724 Command CREATENODE Status SUCCESS Retries 0 TEMS Name HUB_ACHAN2 Target Hostname Platform Product ITM Version i Error Message KDY0028I Request completed successfully Deployment request was processed successfully and is now completed Transaction ID 1222782542297000000015724 Command CREATENODE Status SUCCESS Retries 0 TEMS Name HUB_ACHAN2 164 IBM Tivoli Monitoring Troubleshooting Guide Target Hostname achanl raleigh ibm com Platform WINNT Product NT Version 062100000 Error Message KDY2065I The node creation operation was a success Old Component deploy New Component itm_scn_ID A workspace view is showing an error It is possible that the workspace definition was saved incorrectly An example workspace would be the IBM Tivoli Monitoring for Databases Oracle Agent s SQL Text workspace To solve this problem replace the view and save the workspace Agent upgrade and restart using non root The monitoring agent can run using a non root user ID on UNIX and Linux systems This
143. PI failures include e The name specified in KLXINTCP is wrong TCP IP_USERID is selected based on the specification for TCPIPJOBNAME found in the file pointed to by SYSTCPD in the TCP IP started task JCL The default if no TCPIPJOBNAME exists is TCPIP There exists field documentation on the RACF procedure These two items should be checked first e The started task name does not have RACF authority for the OMVS segment All address spaces must be given RACF or ACF2 permission for the OMVS segment to use TCP IP services Failure to grant this permission which is granted to the started task name can result in INITAPI failures e MAXPROCUSER has been exceeded For MAXPROCUSER problems you can use console operator command SETOMVS MAXPROCUSER xxx to increase the current MAXPROCUSER value as seen by DOMVS O Name Resolution IBM Tivoli Monitoring V6 1 depends on IBM s HPNS EZASMI getaddrinfo and EZASMI getnameinfo calls for resolver services These calls are used to find the symbolic name and dotted decimal IP address of the default network interface for the z OS image A failure in either EZASMI call results in failure to initialize the TCP IP service for the z OS address space Confirming that the Name Resolution calls are successful The following message indicates Name Resolution was successful kdebprc c 661 interface discovery IPV4 interface list SYSL 9 42 46 26 source hostname 0 seq 0 flags 0441 In this example the
144. Portal server database user password is out of sync e User ID does not match operating system s logon user ID e Password does not match operating system s password e Registry does not have the correct password Diagnostic and corrective actions Reconfigure the portal server On Windows Run the tacmd configureportalserver command If the problem persists see the steps in the following row On UNIX and Linux Reconfigure the portal server through the command line or user interface You might also need to check the following to make sure that the user ID and password are correct and in sync On Windows 1 Make sure your portal client user ID is identical with the logon user ID of your system Right click My Computer and select Manage Select Local Users and Groups Select Users Right click your user ID and select Properties to change your password settings For db2admin set the password to never expire 2 Use the correct capitalization for your user ID and password 18 IBM Tivoli Monitoring Troubleshooting Guide Table 7 Correcting database problems 2 of 3 continued 3 Check the DB2 UDB database and make sure the db2admin user ID and password match those of the db2admin local account Go to Control Panel gt Administrative Tools gt Services Right click DB2 DB2 and select Properties Select the Log On tab and make sure the db2admin user ID and password match the db2admin UDB
145. Purpose CTIRA_CELL_NAME OBSOLETE Agents only Replaced by agent configuration variable CT_CMSLIST CTIRA_HEARTBEAT The interval in minutes of the agent to Tivoli Enterprise Monitoring Server heartbeat data sample Default is 10 minutes Using shorter heartbeat intervals will increase network traffic between the agent and Tivoli Enterprise Monitoring Server CTIRA_HIST_DIR Required variable specifying the directory where Tivoli Enterprise Monitoring Agent based short term history data files will be stored Does not apply to Tivoli Enterprise Monitoring Server short term history data files CTIRA_HOSTNAME Used by many but not all agents to provide an alternate hostname qualifier subsystem hostname nodetype for the published agent managed system name Used to remove a long network domain name for example acme us com from the default Tivoli Enterprise Monitoring Agent hostname Not honored by all agents For some agents might cause unpredictable Tivoli Enterprise Portal navigation tree results CTIRA_IP_PORT Applies to z OS agents only DO NOT MODIFY Is set to 0 so the OS can provide the agent rpc listen port Avoids a port conflict for some z OS configurations CTIRA_LOG_PATH Required variable specifying the directory where the Tivoli Enterprise Monitoring Agent Operations Log file is stored The file names use suffix of LGO and LG1 Does not apply to z OS based agents CTIRA_MAX_RECONN
146. RECONNECT_WAIT Time interval in seconds for the agent to wait between attempts to register with a Tivoli Enterprise Monitoring Server Default is 600 seconds CTIRA_REFLEX_ATOMIC Default is Y For subnode targets only Evaluate situation state by any existing specified display item column name when deciding which reflex situation automation command the Tivoli Enterprise Monitoring Agent should execute Not applicable to reflex situation commands executed or evaluated by the Tivoli Enterprise Monitoring Server Disable by setting to N Appendix B Environment variables 243 Table 59 Tivoli Enterprise Monitoring Agent deploy environment variables continued Variable Purpose CTIRA_REFLEX_TARGET Default is Y For subnode targets only Evaluate situation state by subnode name value in the ORIGINNODE column when deciding which reflex situation automation command the Tivoli Enterprise Monitoring Agent should execute Not applicable to reflex situation commands executed or evaluated by the Tivoli Enterprise Monitoring Server Disable by setting to N CTIRA_SIT_CLEAN Number of seconds between garbage clean up of stale entries in the agent persistent situation file Default is 900 seconds CTIRA_SIT_FILE Specify an alternate name for the default Tivoli Enterprise Monitoring Agent based persistent situation file This should only be done in exceptional conditions since the file name reflects the agent
147. RT_NODE_LIMIT xxx where xxx is an integer that is equal to or greater than one of the following e The number of managed systems defined in a managed system list OR e Explicitly assigned to a query over 200 in a Tivoli Enterprise Portal workspace view You must add the KFW_REPORT_NODE_LIMIT environment variable or remove the comment marker in the following Tivoli Enterprise Portal Server environment files and restart the portal server e Windows systems ibm itm cnps kfwenv Linux or AIX systems opt IBM config cq ini After you change the KFW_REPORT_NODE LIMIT variable you might receive the following error KFWITM217E Request error SQL1 CreateAccessPlan failed rc 1 Typically this problem is caused when too many explicitly defined managed systems are assigned to a query for a workspace view The best practice for resolving this problem is as follows Chapter 8 Tivoli Enterprise Portal Server troubleshooting 135 1 Create a managed system list that specifies the explicitly defined managed systems 2 Remove the explicit assignments from the query 3 Assign the managed system list to the query Alternatively you can reduce the number of managed systems that you explicitly define in the query Non hub situations are not associated at the Tivoli Enterprise Portal Server level Only pure hub situations should be associated to the Tivoli Enterprise Portal Server However if you want non hub situations to be as
148. RegExp A Za z0 9 _ amp Required Y gt with the following text lt Type argl t arg2 type Type String ValidationRegExp A Za z0 9 _ Required Y gt Take Action command names do not accept non English characters There is not a workaround at the present time addBundles command times out 184 When using the addBundlescommand to add bundles to a depot the command might time out The default timeout is 600 seconds 10 minutes The following message is returned after successful execution KUICAB022I The following bundles were successfully added to the C IBM ITM CMS depot depot If the addBundles command times out this message is not returned Set the TIMEOUT environment variable to more than 600 before running the addBundles command For example set TIMEOUT 1800 You can also reset the TIMEOUT after the command times out Then run the addBundles command You can also change the LAN linkspeed and duplex setting from auto detect to 100Mbps Full Duplex Then re start the addbundle process IBM Tivoli Monitoring Troubleshooting Guide createNode command fails Table 40 on page 185 provides information to help you recover when the createNode command fails Table 40 createNode command fails Symptom Resolution The createNode Because tacmd on UNIX and Linux is a wrapper script for the command fails ona createNode command the character is removed from the Windows OS
149. Services b Right click Tivoli Enterprise Portal desktop and select Reconfigure from the menu c In the list of parameters that opens double click cnp http url DataBus to open the Edit Tivoli Enterprise Portal Parm window d Type the URL to the external Web server and to the cnps ior file in the candle cnb directory For example if the Web server name is myWeb hostname com and its document root was configured to be candle cnb the value to type is http myWeb hostname com cnps ior e Check In Use and click OK Displaying tasks in the command prompt The Tivoli Enterprise Portal Server has an option to display the tasks at the command prompt This is used primarily with IBM Software Support for gathering diagnostic information 1 From your Windows desktop select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services 2 Right click Tivoli Enterprise Monitoring Server then select Change Startup from the menu 3 Check Allow Service to Interact with Desktop KfwSQLClient utility This utility provides an optional cleanup step if any of the portal server generated workspace queries must be deleted A sample scenario where this might be necessary is if you initially create a metafile application called DISKMONITOR for the Tivoli Universal Agent that has five attribute groups in it Assume that you subsequently remove two of the attribute groups which results in a new ap
150. T kui all In order to debug KT1 as well edit the line to be like the following example KBB_RAS1 ERROR UNIT ksh all UNIT kui all UNIT kt1 all Setting the trace option for the IBM Tivoli Monitoring upgrade toolkit Table 25 Setting the trace option for the IBM Tivoli Monitoring upgrade toolkit Trace option Instructions Endpoint tracing Run the following command to setting log_threshold 3 or higher on an endpoint and enable endpoint tracing wep ep set_config log _threshold 3 Traces are written to 1cfd 10g on the endpoint in LCF_DATDIR Tracing in a test environment A Boolean value of TRUE or FALSE default The default is FALSE Run the following command from a Tivoli Management Environment command prompt to enable tracing idlcall oid _set_debug TRUE where oid Specifies the object ID of the Upgrade Manager object Run the wlookup Framework command to locate the Upgrade Manager object ID in the Tivoli Management Environment wlookup a grep Upgrade Note Setting the trace value to TRUE sets all Upgrade Toolkit tools to TRUE affecting all users running Upgrade Toolkit tools A trace file named trace_tool_timestamp 1og is created in the DBDIR AMX trace directory in XML format with tool being witmscantmr witmassess and witmupgrade and timestamp a time stamp that includes data and time of execution Each record in this log contains a time stamp and message Additionally these tools in
151. TBEAT heartbeat requests when UNIT ctdata IN ER request tracing is used KFW_DATABUS_QUERY_ Yor N Default is N Tivoli Enterprise Portal Server client side VERBOSE response time and request life cycle tracing KFW_MIGRATE_CMS Yor N Default is N When set to Y causes initial migration of managed objects and user IDs from the monitoring server the first time the Tivoli Enterprise Portal Server starts after installation KFW_MIGRATE_FORCE Yor N Default is N When set to Y SQL seed files are processed even if the date time or size of the files have not changed since prior seeding This occurs when running the buildpresentation bat file Windows and when running the InstallPresentation sh file or reconfiguring the Tivoli Enterprise Portal Server UNIX Linux To be effective this value must be set in kfwalone Windows or in cq ini or Inxenvnocms UNIX Linux KFW_MIGRATE_VERBOSE Yor N Default is N When set to Y provides greater detail about operations performed in migrate log when seeding the Tivoli Enterprise Portal Server database This occurs when running the buildpresentation bat file Windows and when running the InstallPresentation sh file or reconfiguring the Tivoli Enterprise Portal Server UNIX Linux To be effective this value must be set in kfwalone Windows or in cq ini or Inxenvnocms UNIX Linux 236 IBM Tivoli Monitoring Troubleshooting Guide Table 53 Tivoli Enterprise Portal Server environment variables
152. Troubleshooting Guide displayed The Security Validation is working despite a blank popup with this label that has a yellow triangle and exclamation point TEMS User Authentication actions are needed When installing a monitoring agent on top of the Systems Monitor Agent you receive an error If you try to install a monitoring agent that is not one of the agents built with IBM Tivoli Monitoring v6 2 2 Agent Builder on top of the Systems Monitor Agent you receive an error install sh failure KCI1163E cannot read file opt IBM ITM registry imdprof tbl Monitoring agents that have been configured to connect to a monitoring server cannot be installed on the same system as those that have been configured for autonomous operation Also monitoring agents that have been configured for autonomous operation cannot be installed on the same system as those that are connected to a monitoring server OS Agent does not install and a message indicates it was already installed The file status properties located in DBDIR AMX data is not deleted when you uninstall the Upgrade Toolkit The Upgrade Toolkit refers to the old status properties file that contains information indicating there the OS Agent was installed You might experience this problem if you do the following in the order listed 1 Upgrade an endpoint Uninstall the Upgrade Toolkit Clean the endpoint manually Reinstall the Upgrade Toolkit Attempt to upgrade the endpoin
153. USE Reduces network flow when the Tivoli Data Warehouse database is not installed on the same machine where the warehouse proxy agent resides this variable must be set to Y This functionality is available on both Windows and UNIX systems This is done automatically when the Use Batch check box on the UNIX warehouse proxy configuration panel is checked KHD_CNX_POOL_SIZE The total number of preinitialized ODBC connection objects available to the KHD_CNX_WAIT The time in minutes to wait before trying to reconnect Default is 10 minutes work queue export threads The default value is 3 KHD_CNX_WAIT_ENABLE A time to wait before a retry Default is Y Changing this variable to N does not wait before retries Setting this variable to N can generate a large log file if the tests to the database fail at each retry KHD_EXPORT_THREADS The number of worker threads exporting data to the database The default value is 10 240 IBM Tivoli Monitoring Troubleshooting Guide Table 57 Warehouse proxy environment variables continued Variable Purpose KHD_QUEUE_LENGTH The length of the KHD work queue This is an integer that identifies the maximum number of export work requests that can be placed on the work queue before the queue starts rejecting requests The default value of KHD_QUEUE_LENGTH is 1000 Setting this value to 0 means the queue length has no limit KHD_SRV_STATUSTIMEOUT The time in seconds set by default
154. Universal Agent troubleshooting Correct operation of the IBM Tivoli Monitoring requires correct application data definitions environment variables and Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal configuration Many problems are caused by incorrectly formatted data or data interpretation Therefore begin Universal Agent troubleshooting with the data provider Review the following list of fundamental Universal Agent troubleshooting tasks Validate the metafiles using the console command VALIDATE Review the validation messages and report Resolve all identified errors and warnings Because the IBM Tivoli Monitoring calls the same validation subroutines when it loads a metafile it encounters the same problems as the KUMPCON VALIDATE program Verify that the first three characters of the application name defined in the APPL statement of the metafile are unique throughout the enterprise Verify that the sequence of data fields on the data record matches the listed sequence of attributes in the metafile In addition ensure that the attribute type and the maximum data value size correspond to the actual application data values Verify that the actual data fields are delimited exactly as specified in the delimiter specification of the ATTRIBUTES statement If the delimiter is specified as NONE ensure that defined attribute value sizes exactly match the data values on the application data record For FILE Data Providers ver
155. X_SP20S ended Tue Jun 14 04 37 02 2005 K041036 Monitoring for situation UADVISOR_OMUNX_SP20S started 42AEA61B 0000 6 kdssqrun c 2995 Fetch QueryRowset error status 302 Tue Jun 14 04 40 43 2005 K041039 Error in request Notify Status 1105 Reason 302 42AEA61B 0001 6 ko4async cpp 4744 IBInterface completeRequest Close failed request lt 61D5E0 gt The Platform view in the Manage Tivoli Enterprise Monitoring Services panel shows the Tivoli Enterprise Monitoring Server as running as a 32 bit application but my agents are shown as running as 64 bit applications The Tivoli Enterprise Monitoring Server is a 32 bit application that runs on both 32 and 64 bit operating systems Tivoli Enterprise Monitoring Server does not release memory after running a large SQL query Running a query for data beyond a 24 hour period consumes high CPU and memory because the data is not stored on the server and must be retrieved from the endpoints All users might experience low system performance while a large amount of data is retrieved from endpoints Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 149 SQL queries with more than 200 OR predicates do not complete If an SQL query to the hub monitoring server contains more than 200 OR predicates a limit is reached and the query does not complete An example of this is if the tacmd listSystems command is run specifying a remote monitoring server that contains universal agents that
156. _INVALIDSITNAME 1145 Invalid sitname supplied define ERR_RULESYNTAX 1203 Generic rule syntax error 5 Did SITMON receive the data Monitoring server trace UNIT ko4async ERROR STATE FLOW UNIT ko4tobje ALL UNIT ko4sitma ALL Yes Does the situation apply to the Enterprise For example ENTERPRISE in the MSG2 message of the monitoring server message log when the situation was created 11 08 06 16 18 49 K046256 Situation definition CheckIfSituationCreated created by ENTERPRISE Only Enterprise situations show up in the portal client user interface A non Enterprise situation does not show up in the portal client user interface even if the situation is raised The distinction between Enterprise and non Enterprise situations is seen in the following monitoring server log examples Enterprise situation K041046 Monitoring for enterprise situation MS Offline started Non Enterprise situation K041036 Monitoring for situation Weekday started Yes See step 6 No Reconfigure the situation to include the Enterprise flag setting No Use the Monitoring server trace UNIT kdsruc1l ERROR STATE UNIT kfaadloc all to see where the data is getting filtered out This trace generates a large amount of data Turn the trace off as soon as you finish troubleshooting 6 Is there a MSG2 message indicating the situation raised Yes Contact IBM Software Support See Chapter 2 Collecting and reviewing 5 for information on what types of data
157. a TMO Applet Failed error There is a known problem with Firefox 3 5 and possibily higher that results in firefox a Loading Java TMO Applet Failed error when using the Java web browser client There is a problem with way Firefox s JSObject is used to communicate between java and javascript For more information see the following website http groups google com browse_thread thread cea83988856cffb6 tacmd createUser output indicates that the path to the Java home directory was not found 120 This problem occurs when issuing tacmd createUser on Windows 2003 SP1 The following error message is displayed KUICCU097E The path to the Java home directory was not found The directory name returned by the CandleGetJavaHome script was not found Call IBM Software Support To resolve this problem launch the CandleGetJavaHome bat script located in InstallITM subdirectory of ITM home directory Ensure that valid Java Home directory is returned This directory should exist and contain an installation of Java 1 5 If the message CScript Error Loading your settings failed Access is denied is returned instead ensure that the Windows Script Host WHS works properly in your environment To verify that Windows Script Host is working on your machine execute the following steps 1 Start gt Run 2 Type cmd and hit Enter 3 Run the cscript exe file If usage options are displayed Windows Script Host is
158. a hub Tivoli 96 IBM Tivoli Monitoring Troubleshooting Guide Enterprise Monitoring Server to which it can connect The F10 CMSLIST function key is unavailable to a hub Tivoli Enterprise Monitoring Server Go to the previous Specify Configuration Values panel to verify what type of Tivoli Enterprise Monitoring Server you are configuring The Enter required field error occurs for the Global location broker applid of Hub or the Network ID of Hub VTAM related fields On the Specify Configuration Hub Values for Remote Tivoli Enterprise Monitoring Server panel the message Enter required field occurs although the remote z OS Tivoli Enterprise Monitoring Server connects to the Hub Tivoli Enterprise Monitoring Server through IP protocols If you are configuring a remote z OS based Tivoli Enterprise Monitoring Server that connects to a non z OS Hub Tivoli Enterprise Monitoring Server via the IP or IPPIPE communication protocol use the following steps as a resolution 1 On the Specify Configuration Hub Values for Remote Tivoli Enterprise Monitoring Server panel enter any value in the following fields e Global location broker applid of Hub For example enter default CTDDSLB if this VTAM APPLID is not used e Network ID of Hub For example enter the NETID value from SYS1 VTAMLST ATCSTRnn Note Neither of these values adversely affect the connection for the remote Tivoli Enterprise Monitoring Server 2 From the Configure the
159. ackup the files before altering them On Windows systems 1 Always backup the files before altering them IBM Tivoli Monitoring Troubleshooting Guide ONDAY On the computer where the Tivoli Enterprise Portal Server is installed click Start gt Programs gt IBM Tivoli Monitoring gt Manage Tivoli Enterprise Monitoring Services Right click the Tivoli Enterprise Portal Server service Select Advanced gt Edit Trace Parms to display the Trace Parameters window Select the RAS1 filters The default setting is ERROR Accept the defaults for the rest of the fields Click OK to set the new trace options Click Yes to recycle the service On UNIX systems 1 2 Always backup the files before altering them Set the following variable in the CANDLEHOME config cq ini KBB_RAS1 ERROR UNIT filter trace_level where filter is the component you want to trace and trace_level is the level of tracing you want Recycle the Tivoli Enterprise Portal Server by restarting or stop and then start Setting the trace option for the Tivoli Enterprise Monitoring Server trace On Windows systems LE 2 6 7 8 Always backup the files before altering them On the computer where the Tivoli Enterprise Monitoring Server is installed select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services Right click the Tivoli Enterprise Monitoring Server service Select Advanced gt
160. ad return code is returned to install sh The problem can be manually recreated by running the JAVA command used to launch LAP or simply by running Java version from the jre under CANDLEHOME Indications show that the system might be missing an OS patch required for the level of Java or possibly an incorrect version of Java has been packaged and installed on the system Command line interface program of the Application Support Installer is not currently available The command line interface program of the Application Support Installer is not currently available thus you cannot run the install in command line mode However you can run the install in silent mode instead If your UNIX or Linux computer does not have X Windows set up you must use the silent installation method Silent installation on UNIX based systems of Tivoli Enterprise Monitoring Server returns an encryption key setting error The following errors occurs if you attempt a silent install on UNIX based systems or UNIX based systems and the encryption key is not exactly 32 characters Exception in thread main candle kjr util CryptoFailedException CRYERR_INVALID_KEY_ LENGTH at candle kjr util CRYPTO validateKeyLength CRYPTO java 911 at candle kjr util CRYPTO setKey CRYPTO java 452 at ITMinstall gskSetkey lt init gt gskSetkey java 179 at ITMinstall gskSetkey main gskSetkey java 26 Set the encryption key parameter in the silent install file to exactly 32 characters as
161. ademark information at www ibm com legal copytrade shtml Adobe Acrobat PostScript and all Adobe based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States other countries or both Cell Broadband Engine and Cell B E are trademarks of Sony Computer Entertainment Inc in the United States other countries or both and is used under license therefrom Intel Intel logo Intel Inside Intel Inside logo Intel Centrino Intel Centrino logo Celeron Intel Xeon Intel SpeedStep Itanium and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce ITIL is a registered trademark and a registered community trademark of the Office of Government Commerce and is registered in the U S Patent and Trademark Office 4 Java and all Java based trademarks and logos are trademarks or registered trademarks of Sun Microsystems Inc in the United States lava other countries or both COMPATIBLE Linux is a trademark of Linus Torvalds in the United States other countries or both Notices 257 Microsoft Windows Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States other countries
162. after upgrade e Remote update of a Windows system fails because of a pending rename that requires a reboot Note In this case a RC_INFO Pending rename operations found must reboot before install can continue message is found in the Abort IBM Tivoli Monitoring for Databases 1og file IBM Tivoli Monitoring Troubleshooting Guide When installing and configuring the Tivoli Enterprise Monitoring Server on Windows Server 2008 a number of popups and errors occur To install and configure the Tivoli Enterprise Monitoring Server on Windows Server 2008 there are a number of manual steps and workarounds that must be performed 1 Disable the Windows firewall by following these steps a Login to Windows Server 2008 and start the Server Manager b In the Security Information section click Go to Windows Firewall c In the Overview section click Windows Firewall Properties This displays the Windows Firewall with Advanced Security window d To completely disable the firewall it must be turned off in 3 places in the window e Domain Profile e Private Profile e Public Profile Each of these tabs must be individually selected and the Firewall state must be changed to Off Each time you change the state to Off click Apply e After you have changed the Firewall state in all of the profiles click OK 2 Configure the monitoring server to work correctly with Windows User Account Control UAC e Using Windows Explorer navigate
163. again If it fails again contact IBM Software Support for assistance See Chapter 2 Collecting and reviewing troubleshooting data on page 5 for information on what types of data to collect before contacting Support Both products are available in the Add and Remove Programs tool Use the following steps if both the Candle OMEGAMON or IBM Tivoli Monitoring entries exist in the Add and Remove Programs tool 1 Uninstall IBM Tivoli Monitoring through the Add and Remove Programs tool 2 Uninstall Candle OMEGAMON through the Add and Remove Programs tool 3 Proceed to Verifying the uninstallation Verifying the uninstallation Use the following steps to verify that you successfully removed the failed installation 1 Verify that the installation home directory and all contents have been removed 2 Open the Registry Editor by clicking Start gt Run and typing regedt32 Click OK 3 Expand the key HKEY_LOCAL_ MACHINE registry key Expand the SOFTWARE registry key 5 Verify that the Candle registry key and all sub keys have been removed from HKEY_LOCAL_MACHINE SOFTWARE B You are now ready to install IBM Tivoli Monitoring Incorrect behavior after an uninstallation and reinstallation You might experience incorrect behavior if you uninstall then reinstall the product without restarting the system For example you might experience the following problems e Inability to create trace logs e Agents
164. age Start Chapter 12 Universal Agent troubleshooting 199 200 A MIB workspace is not showing expected data For MIB workspaces to display data you must initiate data collection using the Monitor Start option of the Take Action feature If you are still not receiving data e Ensure that the correct metafile was loaded e Confirm that the correct target monitoring agent nodes were specified in the Monitor Start window separated by commas e Confirm that the node and correct community name were entered in brackets The following example displays correct formatting if the SNMP community name of the target is not the default public nor specified by KUMP_SNMP_NET_COMMUNITY and there is no entry in the KUMSCOMM file athens IBM Take Action gt Manage Start or Take Action gt Manage Stop is not working Ensure the following for the Manage Start and Manage Stop Take Action to work e The address is correct if you entered the network address manually rather than selecting it in the NETSUMMARY workspace e The destination managed system you specified when distributing the action is host name SNMP MANAGERNN The SNMP Data Provider does not start with IBM Tivoli Universal Agent Ensure you included SNMP as a value for environment variable KUMA_STARTUP_DP You do not see the TakeAction options Seed the Tivoli Enterprise Monitoring Server with KUM SQL and restart the Tivoli Enterprise Monitoring Server and Tivoli Enterprise P
165. al Server is unavailable x 115 Tivoli Enterprise Portal Server is not reconnecting 115 Cannot reconnect to the Tivoli Enterprise Monitoring Server 115 Tivoli Enterprise Portal Server cannot connect to the Tivoli Enterprise Monitoring Server private interface 3 116 Chapter 7 Tivoli Enterprise Portal troubleshooting do 117 Client allows you to save a situation wath an invalid character z a IZ Using an administrator name with non Jatini characters cannot log onto the Tivoli Enterprise Portal gt lt 117 The Tivoli Enterprise Portal desktop does not Work when exporting DISPLAY 117 Tivoli Enterprise Portal browser client yields 1 Eitor 2721 and will not run 118 Some attribute groups showing a different name in the Tivoli Enterprise Portal 118 Monitoring agents show in an unexpected position in the navigation tree 118 Tivoli Enterprise Portal Desktop called through Java Web Start does not work properly after adding agents support for Tivoli mae Portal Server 2 120 Receive a Loading Tava TMO Applet Failed error tacmd createUser output indicates that the path to the Java home directory was not found Cannot launch the Tivoli Enterprise Portal help On an Active Directory Server sysadmin cannot logon to the Tivoli Enterprise Portal client Several enterprise workspaces are returning an error KFWITM217E Request Error SQL1_CreateRequest Failed re 350 Si
166. al server is initialized by searching for the text string Waiting for requests Startup completed 109 Chapter 6 Connectivity troubleshooting Table 36 Cannot log in to the Tivoli Enterprise Portal continued Problem Corrective action and solution If the Tivoli Enterprise Portal Server connection to LDAP is lost When the portal server is configured to authenticate against the LDAP server with optionally enabled Single Sign On capability if you lose the portal server to LDAP connection this will cause any log in attempt to fail with error code KFWITM393E User ID or password is invalid This authentication failure will be reported for any user including the default administrative user sysadmin and not only for users defined in the LDAP repository Re establish the connection to LDAP As soon as the portal server to LDAP connection is re established you can log in to the Tivoli Enterprise Portal If there is still a problem connecting with LDAP de configure LDAP authentication If the LDAP connection is broken and the normal procedure to switch off LDAP based authentication does not work the following steps need to be performed 1 For AIX and Linux systems stop the portal server with the itmemd agent stop cq command invoked from the installation directory 2 Run the disableLDAPRepository sh script from candle_home arch iw scripts where arch is the machine architecture for example li6263
167. all_dir config product_code ini For example the file name for the Monitoring Agent for UNIX OS is ux ini 2 Find the line the begins with CTIRA_HOSTNAME 3 Type anew name for host name that is a unique shorter name for the host computer The final concatenated name including the subsystem name new host name and AGENT_CODE cannot be longer than 32 characters Note You must ensure that the resulting name is unique with respect to any existing monitoring component that was previously registered with the Tivoli Enterprise Monitoring Server 4 Save the file 5 Restart the agent Issues with starting and stopping an agent as a non Administrator user You might experience issues with starting and stopping an agent as a non Administrator user This issue is caused because of improper permissions set for the hostname_pc run file That file is created or modified every time an instance is started or stopped All instances must use the same user ID UNIX based systems Log agent was deployed configured and started but returns the KFWITM290E error The Tivoli Enterprise Monitoring Server is timing out waiting for deployment action to complete The default timeout is 600 seconds You can change the timeout setting to KFW_SQL1_ASYNC_NOTIFY_MAX_WAIT in kfwenv KFW_SQL1_ASYNC_NOTIFY_MAX_WAIT 1000 KDY1024E error displays when configuring the run as user name for an agent The error message KDY1024E displays when configu
168. and OR logic and using values from two different attribute groups the traps do not list the predicates in an expected way If you have defined a situation that attempts to combine attributes across more than 1 attribute group this is not currently supported by any autonomous agent processing mode or private situations The monitoring server performs the evaluation of situations having combined attribute groups This can be either from embedded situations or two or more attribute groups in the same predicate for example WHERE USER abc AND LOCALTIME today Failure when importing situation xml file edited with WordPad If you edit an xml file for a situation using WordPad and then import the situation tacmd createsit i xml the command fails If you edit the xml file using Notepad it works correctly Use Notepad to edit situation xml files Printer details of another system are displayed on the Tivoli Enterprise Portal When connecting to a system remotely printers on that local system can be seen from the Tivoli Enterprise Portal on the remote system You should be aware that by using a remote desktop printer information might be displayed in the Tivoli Enterprise Portal and shared with others CTIRA_MAX_RECONNECT_TRIES environment variable is now obsolete The agent now attempts communication with a monitoring server until a connection is successfully established There is no longer a limit on the number of connection attemp
169. and serviceability RAS trace log RAS logs are in the English language only The RAS trace log mechanism is available on the Tivoli Enterprise Monitoring Server the Tivoli Enterprise Portal Server and the monitoring agent By default the logs are stored in the installation path for IBM Tivoli Monitoring IBM Software Support uses the information captured by trace logs to trace a problem to its source or to determine why an error occurred The default configuration for tracing such as whether tracing is enabled or disabled and trace level depends on the source of the tracing You can choose how many files to keep when the log rolls If you cannot find the log files you need restart the system and try again This section includes instructions for configuring trace logging Table 22 lists the location of the log files directories Table 22 Location of log files on the Tivoli Enterprise Monitoring Server Component Windows UNIX based systems Portal Server Tivoli Enterprise install_dir logs install_dir logs hostname_PC_timestamp log where install_dir Specifies the directory where Tivoli Enterprise Portal Server was installed hostname Specifies the name of the system hosting the product PE Specifies the product code cq for the Tivoli Enterprise Portal Server timestamp A decimal representation of the time at which the process was started Tivoli Enterprise C Documents and Settings None Portal bro
170. ands Reconfiguring an agent ind hen getting the deploy status yields a port number message tacmd getfile or putfile command is failing Temporary files remain when tacmd getfile or putfile is interrupted Receive an OutOfMemory exception when using the import or export commands The suggestbaseline or acceptbaseline domande fail 176 176 176 176 176 176 177 177 177 177 ATT 177 178 178 179 180 180 180 181 181 181 181 181 182 182 182 182 183 183 183 Problems with Take Action commands and curly Universal Agent does not start 194 brackets 183 The agent fails but events are still seen as Active 194 When configuring the Monitoring Agent for Sybase Previously defined situations display in the Tivoli and the Warehouse Proxy Agent receive message Enterprise Portal with a Problem or Error status 194 to use CandleManage 184 All Universal E traps ei with the listSit command with the type option fails with z a same severity 194 KUIC02001E message on Japanese Systems 184 The Managed system version suffix i increases 195 Take Action command names do not accept Tivoli Enterprise Portal column headings are non English characters 184 missing for a Universal Agent application 195 addBundles command times out 184 Tivoli Enterprise Portal charts and
171. ant to configure ITM Tivoli Monitoring Services or an OMEGAMON XE monitoring agent on the PRODUCT SELECTION MENU c On the RUNTIME ENVIRONMENTS RTES menu type B for Build libraries next to the runtime environment in which the monitoring server is configured and press Enter The PP 1xxxx job that allocates the runtime libraries is displayed Chapter 7 Tivoli Enterprise Portal troubleshooting 129 d Edit the CYL parameter in the job to increase the VSAM allocation to whatever value your DASD can accommodate 4 Submit the PP 1xxxx job Use IDCAMS to copy data from the flat file to the new VSAM 6 Reinstall the application support for the product or products whose situations are missing or not starting correctly a For instructions on installing application support for a monitoring agent installed on z OS refer to the configuration guide for your monitoring agent For instructions on installing application support for monitoring agents installed on a distributed system Windows UNIX Linux see the IBM Tivoli Monitoring Installation and Setup Guide kshsoap client fails because of missing libraries on UNIX based systems On UNIX based systems the CandleSoapClient shell script calls kshsoap binary from inside while sourcing the environment You do not need to run the kshsoap binary as required on a Windows platform Category and Message field of the universal message does not accept DBCS To record a DBCS
172. at application support is installed consistently throughout your environment 3 If application support is missing add the appropriate application support to the portal server and monitoring server for the monitoring agents If the desktop or Java Web Start client is being used application support must also be installed on the portal client For more information and instructions on installing application support see Configuring application support for non base monitoring agents in the IBM Tivoli Monitoring Installation and Setup Guide For instructions on installing application support on a monitoring server on z OS see Configuring the Tivoli Enterprise Monitoring Server on z OS Table 2 Correcting monitoring server problems e Monitoring server is not started e Connectivity has been lost between the portal server and monitoring server or between the monitoring server and monitoring agent Diagnostic and corrective actions 1 If you are an administrator restart the monitoring server Otherwise notify an administrator and wait for the monitoring server to be restarted 2 Running the following ITMSUPER tools might also provide more information e Topology tool e Connectivity tool e TEMS Server Analysis tool e TEPS Analysis tool 3 Check the portal server logs for messages indicating communication failures to the monitoring server Check the monitoring server logs for messages indicating communication failures to th
173. at occur leading up to the calls to the SOAP server The Idaptrace txt file will show any activity and possible failures occurring when it actually binds to the SOAP server and attempts to lookup users Windows Systems Windows systems use the same KBB_RASI trace settings as UNIX and Linux systems but enabling the additional LDAP trace requires a different procedure From the Manage Tivoli Enterprise Monitoring Services MTEMS GUI complete the following steps 1 Stop the monitoring server 2 Right click the entry for the monitoring server and select Advanced gt Edit Variables 3 In the Override Local Variable Settings window that appears click Add Enter LDAP_DEBUG for the Variable and 65535 for the value and click OK 4 Click Add again and enter LDAP_DEBUG_FILEfor the Variable and any path or file that you wish for example C temp ldaptrace txt Then click OK 5 Click OK to save the changes 6 Start the monitoring server New login and LDAP related monitoring server activity is now logged in the LDAP_DEBUG_FILE When you are finished reproducing the problem and want to stop tracing go back to the Manage Tivoli Enterprise Monitoring Services MTEMS GUI and complete the following steps 1 Stop the monitoring server 2 Right click the entry for the monitoring server and choose Advanced gt Edit Variables 3 Highlight the variables and click Delete to delete both the LDAP_DEBUG and LDAP_DEBUG_FIL
174. ata is once again usable Error code 3205 The last 3 digits represent the error and the beginning digits represent the persistent data store function that was being called The 205 indicates the error RowExceedsFileFormat This error is generated if the row you attempt to insert is larger than what will fit in a block allocated to the persistent data store data set The actual maximum length is about 100 bytes smaller than the block size Therefore if you allocate a block size of 1000 Window 1 and attempt to write a row greater then 900 you receive this message The persistent data store cannot span a data row across multiple blocks One other possibility is that either the API calls to the persistent data store to do the insert are specifying an invalid row length or the lengths of all the columns put together for the insert exceed the buffer length Error code 35404 This code has many causes One possibility is that a PARMA parameter intended for the agent processing is mistakenly set to the monitoring server and interpreted as a column name This might be due to obsolete SQL saved in the monitoring server database In most cases you can ignore this error Set monitoring server traces to UNIT kdssqprs input error The UNIT kdssqprs input error trace returns large amounts of data Turn the trace off as soon as you finish troubleshooting KFAPERR error code 14209 Persistent data store Filename is Not Available messages in the RKLVLOG of
175. ate and reuse as many log sets as you need For example when importing log files from multiple servers you need more than one log set Using the Log and Trace Analyzer you can correlate multiple IBM Tivoli Monitoring log files into a single view The IBM Tivoli Monitoring log files can be combined in a single view ordered by time stamp to correlate the operation of the IBM Tivoli Monitoring components There are two ways to correlate log files e Simple To correlate all imported log file complete the following steps 1 In the Log and Trace Analyzer navigation tree view right click Logs 2 Click View All Logs e Advanced To correlate a set of log files by creating a custom correlation complete the following steps In the Log and Trace Analyzer navigation tree view right click Correlations Click New gt Log Correlation In the window that is displayed type the name for the correlation Click Finish 1 2 3 4 Add the log files that you want to include for the correlation 5 6 Refresh the navigation tree view 7 In the navigation tree view right click the correlation name you typed and click Open With gt Log View After you create a view of the logs you can organize the log data to isolate problems The following list identifies some of the ways that you can organize the data e Sort log records For example you can sort by time component and server name e Highlight log records For example
176. ated CAT ATR and ODI files were not uploaded to the Tivoli Enterprise Portal Server and Tivoli Enterprise Portal Server directories The most current and accurate copies of these definition files are in Universal Agent s work directory If necessary you can copy or FTP these files from the work directory to the appropriate Tivoli Enterprise Portal Server and Tivoli Enterprise Portal Server directories A variation of this problem is that some of the column headings display but not all of them or the column headings are incorrect This could indicate old or out of synch CAT ATR and ODI files in the Tivoli Enterprise Portal Server and Tivoli Enterprise Portal Server directories Again you can always synchronize these files by copying them from the Universal Agent work directory and recycling the Tivoli Enterprise Portal Server and Tivoli Enterprise Portal Server Another possible explanation is that Universal Agent is connected to a remote Tivoli Enterprise Portal Server and the Universal Agent generated CAT and ATR files for the application didn t get propagated to the hub Tivoli Enterprise Portal Server Tivoli Enterprise Portal Server only queries the hub Tivoli Enterprise Portal Server so it s unable to retrieve the data rows Copy or FTP the Universal Agent generated CAT and ATR files from the remote Tivoli Enterprise Portal Server to the hub Tivoli Enterprise Portal Server Tivoli Enterprise Portal charts and graphs do not work f
177. ation bat 63 IBM Tivoli Monitoring Operations Logging 63 ITMSuper 65 Log and Trace Analyzer tool 60 pdcollect tool 57 rasllog 60 UAGENT application ACTION workspace 56 DPLOG 56 used to diagnose problems 35 trace 35 Agent Deploy tool trace 44 dynamically modifying 51 IBM Tivoli Enterprise Console Situation Update Forwarder 48 IBM Tivoli Enterprise Console trace 47 IBM Tivoli Monitoring upgrade toolkit 47 IBM Tivoli Universal Agent trace 44 setting 42 262 IBM Tivoli Monitoring Troubleshooting Guide trace continued setting a client 42 Summarization and Pruning Agent 45 tacmd commands 46 Tivoli Enterprise Monitoring Server trace 43 Warehouse Proxy trace 45 trace logs 35 training Tivoli technical 248 TRAP workspace 198 troubleshooting 227 arithmetic overflow error 230 browser mode 121 built in features 1 columns in the table are blank 231 connectivity 107 IBM Tivoli Monitoring Technology Service Console for 53 installation issues 227 JDBC issues 227 locations of log files 227 logon error messages 107 missing drivers 227 no data is plotted 231 out of memory error 229 requested data not available 232 setting a trace 42 SQL error 230 SQL query not valid 228 SQL statement messages 229 Tivoli Data Warehouse issues 232 Troubleshooting Event synchronization problems 209 monitoring agent deploy 155 SNMP data provider 197 Tivoli Enterprise Monitoring Server 139 Tivoli Enterprise Monitoring Server on distributed 139 Tivoli En
178. ava 575 at com tivoli twh ksy agg RawTable aggregateData RawTable java 2356 at com tivoli twh ksy agg Originnode aggregateDataForNode Originnode java 180 at com tivoli twh ksy agg RawTable createAggregatesAndPrune RawTable java 3286 at com tivoli twh ksy agg Worker run Worker java 98 Use a smaller number of rows per database transaction Chapter 13 Database troubleshooting 203 Data did not migrate from the Microsoft SQL Server database to the DB2 Universal Database You can run the movefrom m2i bat located in the install_dir cnps directory to recover the contents of the Universal Database The movefrom m2i bat utility creates a flat file from the SQL server database contents and imports into the Universal Database The movefrom m2i bat utility runs during the installation of the Tivoli Enterprise Portal when the option to migrate from Microsoft SQL Server to the Universal Database is selected but can also be used after installation or routine Tivoli Enterprise Portal usage This migration utility can fail if the Tivoli Enterprise Portal userid and password do not have the correct authority to connect to the Tivoli Enterprise Monitoring Server Universal Database The movefrom m2i bat requires that the Microsoft SQL server database is on the same Windows platform as the Tivoli Enterprise Monitoring Server and the new Universal Database installation Use this utility only after migration problems and before customizing Tivoli Enterpr
179. avigate to them directly while in ADMIN MODE result in the observed error because required context is not available Also when navigating to any workspace that is the target of a link that target workspace does not appear on the Workspaces menu Situation editor cannot display advanced advice help files In double byte languages when the font is set to italic it cannot display the font in italics format After acknowledging a situation event and selecting the link for that situation you might receive a message You hover over a navigator item and then right click gt Quick Ack on any situation event If you then select the link for the same situation you might receive an error message If you then wait without clicking OK the Tivoli Enterprise Portal client exits If you encounter this problem use any recent Sun JRE 1 5 0_14 or above This problem is limited to IBM s 1 5 java Note A Sun bug report bug_id 6578895 which describes the issue was fixed in Sun s 1 5 0_14 Password problem using the LDAP Security option on Active Directory system Create Idapuser on the Active Directory system Configure the Tivoli Enterprise Portal Server with the LDAP Security option Logon to the Tivoli Enterprise Portal as sysadmin and add Idapuser from the Administer Users option In User Information Distinguished Name should be cn ldapuser o ITMSSOEntry Logon to the Tivoli Enterprise Portal as Idapuser with the Idapuser
180. before the install The behavior is harmless and there is no workaround currently Errors occur during installation of Event IBM Tivoli Monitoring Event Forwarding tool The product functions normally in spite of the error Check the installation log for more details One or more errors occured during the replacement of files tecSyncAl1Filel with files tecSyncAllFilel Refer to install log for more details One or more errors occured during the replacement of files tecSyncAl1File2 with files tecSyncAllFile 1 Refer to install log for more details One or more errors occured during the replacement of files tecSyncAl1File3 with files tecSyncAllFilel Chapter 5 Installation and configuration troubleshooting 79 Refer to install log for more details Windows installation problems and resolutions 80 On Windows systems the installation fails randomly when installing different features On Windows systems the installation fails randomly when installing different features An example of the error looks like 7 9 2008 10 49 35 OnMiaMoving Processing Feature KIWWICNS 7 9 2008 10 49 35 OnMiaMoving Error log C IBM ITM InstallITM MiaError log created 7 9 2008 10 49 35 OnMiaMoving Feature KIWWICNS Tivoli Enterprise Portal Server Extensions will be installed 7 9 2008 10 49 35 OnMiaMoving CMD C temp tmv621 d8185a 200807040554 base_windows WINDOWS KIWWICNS exe Parameters backup n force y silent y i
181. bleshooting Guide Distribution to a monitoring server on the object group editor is not equivalent to a monitoring server distribution in the historical collection configuration window To decrypt a password KDS_VALIDATE_EXT Y is required KDS_VALIDATE_EXT Y is required on a SLES 10 64 bit zLinux monitoring server to successfully decrypt a password sent by the portal server for validation This operating system uses Pluggable Authentication Modules PAM and this monitoring server parameter for this purpose For all other purposes PAM is not supported by adding the parameter KDS_VALIDATE_EXT Y to a monitoring server configuration Remote Tivoli Enterprise Monitoring Server consumes high CPU when large number of agents connect In enterprise environments a large number of agents can connect to a remote Tivoli Enterprise Monitoring Server in a short period of time Examples of when this might occur are during startup of the Tivoli Enterprise Monitoring Server or when agents failover from a primary to secondary Tivoli Enterprise Monitoring Server In these cases the amount of CPU processing is directly proportional to the total number of situations that have been distributed to agents connected to the remote Tivoli Enterprise Monitoring Server For example if there are 1000 agents connecting to the remote Tivoli Enterprise Monitoring Server and each agent has an average of 20 situations distributed to it the total number of situations dis
182. by the z OS TCP IP administrator The First SEND This section provides information about confirming whether or not First SEND was successful as well as how to repair failures in the First SEND Confirming that he First SEND was successful The sequence of the following communication messages indicate the first SEND operation an Ib__lookup RPC request and the first RECEIVE operation KDCRO_ Send request FFFF 0 0 200 ip pipe 9 42 46 26 1918 KDCRO_InboundPacket response FFFE 0 0 320 ip pipe 9 42 46 26 1918 KDCL_GetBinding Using LLB at ip pipe 9 42 46 26 1918 When the first network I O is successful the response indicates link and transport connectivity with the hub computer Repairing the failures in the First SEND There are two consideration specific to OS 390 and z OS platforms e RACF permission to the started task for the OMVS segment e Presence of the well known port on the TCP IP Port List The RACF permission problem might not be detected until the failure of the first network I O The KDCRO_Send request fails with Errno 2 EACCESS This failure can occur with the first use of the started task name A similar problem results in EACCESS the well known port is defined on the TCP IP port list ISPF Option 6 netstat portlist confirms the presence of the well known port in the TCP IP reserved port list The well known port should not be on this list SNA service initialization IBM Tivoli Monitoring A
183. cally by Agent Management Services in several seconds When the OS agent is started refresh workspace you find only the Watchdog s management status workspace as managed all of the other agents workspaces status display Not managed On a heavily loaded system increase the command time out parameter KCA_CMD_TIMEOUT in the OS agent s ini file from its default value to something larger up to 120 seconds Also you may need to increase the checkFrequency value in the OS agent s availability checking policy file to a larger value See the IBM Tivoli Monitoring Administration Guide s chapter on Tivoli Agent Management Services for information on how to change the parameters of this file Calendar entries overlap After an upgrade the calendar entries for PrimeShift and NonPrimeShift might have overlaps for hours 8 and 17 The default PrimeShift and NonPrimeShift calendar entries should look as follows Name NonPrimeShi ft Type CRON Data 0 1 2 3 4 5 6 7 17 18 19 20 21 22 23 1 5 Name PrimeShift Type CRON Data 8 16 1 5 To correct the problem use the tacmd editcalendarentry command to correct the calendar overlaps so that the calendar entries look as shown above Receive an error when deploying an System Service Monitor agent When deploying a System Service Monitor agent using a hub monitoring server outside a firewall through a remote monitoring server inside the firewall to a client inside the firewall t
184. can be done by running the itmcmd agent start command while logged in as a non root user and this can be done remotely by deploying the agent using the Run As option on the GUI or using the _UNIX_STARTUP_ Username option on the tacmd addSystem command line If the agent is running using a non root user ID and then the agent is upgraded restarted remotely restarted as a result of a system reboot or the itmcmd agent start is run using the root user ID then the monitoring agent subsequently runs as the root user To confirm the user ID that the monitoring agent is using run the following command jtm_install bin cinfo r If the installation is not permissioned properly then you might be unable to restart the agent as a non root user ID after it has been run as the root user ID To prevent this problem ensure that the secureMain lock command with the g option has been previously run See the Securing your IBM Tivoli Monitoring installation on Linux or UNIX appendix in the IBM Tivoli Monitoring Installation and Setup Guide for further details If the agent is running as root and that is not the desired user ID then use the following steps to restart the agent 1 Log in as root 2 Run the itmcmd agent stop command 3 Log in or su to the user ID that you want the agent to run as 4 Run the itmcmd agent start command If the agent was running as root because of a system reboot then complete the following steps so that the appro
185. ception message sp_cursoropen sp_cursorprepare The statement parameter can only be a single select or a single stored procedure This was fixed by not adding by default the selectMethod cursor string in a Microsoft SQL Server URL but you also must remove the string selectMethod cursor that comes by default when choosing the Microsoft SQL Server database in the Warehouse Proxy Agent configuration panel on UNIX systems Chapter 10 Monitoring agent troubleshooting 177 Custom defined workspace views do not handle symbol substitution as expected Symbol references in the header and footer regions of custom defined workspace views do not resolve as expected in the Tivoli Enterprise Portal Symbol references in Header and Footer expressions set through the view Properties dialog will only be substituted if the workspace is reached as the target of a link Verify that the custom workspace for which the expression is being specified is being reached through a link rather than being selected directly from a navigator node or the Workspaces menu Otherwise the symbols will evaluate as an empty string For best results also ensure that the expression is assigned to the Header or Footer target workspace property through the Link Wizard rather than by editing the Title text field for the Header or Footer region on the Style panel of the Properties Dialog Whatever expression is assigned to the target workspace property through the Link Wizard wil
186. cess started from Add Remove Programs on systems with v6 2 2 installed Press cancel and enter the path to the msi file on the original installation image from which the feature was installed You can also cancel the entire uninstallation process and start the installer from the original image and continue the uninstallation process as you were doing from Add Remove Programs IBM Tivoli Monitoring Troubleshooting Guide If the original installation image is not available 1 Cancel the current uninstallation process 2 Move the temporary INSTALL ver and INSTALLX ver files from the CANDLE_HOME InstallITM ver directory to a safe location and start the uninstallation process again 3 When the feature is successfully removed restore the INSTALL ver and INSTALLX ver files back to the CANDLE_HOME Insta lITM ver directory Uninstallation is blocked by another process that is using the IBM Tivoli Monitoring eclipse help server Kill the javaw tasks associated with the IBM Tivoli Monitoring eclipse help server on the system so that the uninstall can continue Uninstallation of an agent produces help errors After uninstalling an IBM Tivoli Monitoring v6 1 1 agent and its application support files the IBM Tivoli Monitoring Online F1 Help produces an error when selected The browser displays The page cannot be displayed with a URL of http host 9999 help index jsp To solve this problem complete the following steps e Stop the
187. cled they detect situations with no corresponding definition files These situations are flagged as a problem until the application definition files are regenerated and uploaded again to the Tivoli Enterprise Portal Server and Tivoli Enterprise Portal Server All Universal Agent emitted traps display with the same severity Many different policy generated traps are sent by Universal Agent to a 3rd party SNMP Manager product but all the traps show up with the same severity and it s not the severity specified in the policy definition Turn on the Universal Agent environment variable KUMP_TRAP_USE_POLICY_SEVERITY 194 IBM Tivoli Monitoring Troubleshooting Guide KUMP_TRAP_USE_POLICY_SEVERITY Y The Managed system version suffix increases Each time the Universal Agent is restarted the managed system version suffix for an application is incremented even though the metafile has not changed There are two metafiles activated that have the same three character application prefix If the metafiles are activated one after the other the Universal Agent determines that the three character application was modified and registers a new version suffix and upload new CAT ATR and ODI files Tivoli Enterprise Portal column headings are missing for a Universal Agent application A Universal Agent application is online to Tivoli Enterprise Portal but all of its column headings are missing in the application workspace The Universal Agent gener
188. cmd server h install_dir 1 n start stop tems_name Setting the trace option for the Agent Deploy tool On Windows systems 1 On the computer where the Tivoli Enterprise Monitoring Server is installed select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services Right click the Tivoli Enterprise Monitoring Server service Select Advanced gt Edit Trace Parms to display the Trace Parameters window Type UNIT kdy all in the Enter RAS1 Filters field Accept the defaults for the rest of the fields Click OK to set the new trace options NOAA ON Click Yes to recycle the service On Linux systems set the following variable in CANDLEHOME config 1z ini KBB_RAS1 ERROR UNIT kdy ALL UNIT kdd ALL On UNIX systems other than Linux 1 Set the following variable in CANDLEHOME config ux ini KBB_RASI ERROR UNIT kdy ALL UNIT kdd ALL 2 Recycle the OS Agent on that endpoint Setting any monitoring agent s trace option for SNMP alerts When troubleshooting SNMP Alerts for any agent set the following trace ERROR UNIT KRA ALL If the agent is configured to use SNMPv3 Encryption when emitting the SNMP alerts set COMP SNMP ALL so that the trace setting would be the following ERROR UNIT KRA ALL COMP SNMP ALL Use COMP SNMP ALL when you are focusing on SNMP traps If you are focusing on an agent communication error or crash then use KBB_RAS1 UNIT KRA
189. configured and working properly If instead the following error is displayed IBM Tivoli Monitoring Troubleshooting Guide CScript Error Loading your settings failed Access is denied Follow the instructions on the Microsoft Support site to resolve this error in your environment before proceeding Cannot launch the Tivoli Enterprise Portal help The Tivoli Enterprise Portal help might not display for the following reasons Popup blocker If the browser toolbar has a popup blocker running the help does not open whether you select Contents gt Index from the Tivoli Enterprise Portal Help menu or click Help in a window Turn off the popup blocker Internet Explorer shortcut keys the same for Tivoli Enterprise Portal Some Tivoli Enterprise Portal shortcut keys are also used by Internet Explorer If you are using the browser client and press F1 to open the Tivoli Enterprise Portal help help for Internet Explorer displays instead Select Contents and Index from the Tivoli Enterprise Portal Help menu JavaScript not enabled in Microsoft Internet Explorer JavaScript might not be enabled in Microsoft Internet Explorer See the Microsoft Knowledge Base Article 236846 for troubleshooting information http support microsoft com default aspx scid kb en us 236846 On an Active Directory Server sysadmin cannot logon to the Tivoli Enterprise Portal client You receive the error message invalid id password You must set the loca
190. contain more than 200 subnodes the query will OR together all of the subnodes Tivoli Enterprise Monitoring Server aborts unexpectedly when exiting the telnet session used to start it A UNIX based systems Tivoli Enterprise Monitoring Server aborts unexpectedly when exiting the telnet session used to start it either from the client or the command line If you start the Tivoli Enterprise Monitoring Server from a Bourne shell the Tivoli Enterprise Monitoring Server session terminates when you exit the telnet session Do the following so you can exit the telnet session without shutting down the Tivoli Enterprise Monitoring Server 1 Enter the Korn shell ksh 2 Start Tivoli Enterprise Monitoring Server KCIINOO84E Timeout appears while waiting for Tivoli Enterprise Monitoring Server to start on AIX 5 3 After installation the Tivoli Enterprise Monitoring Server and Remote Tivoli Enterprise Monitoring Server performance is very slow Confirm that the prerequisite software has been installed The C libraries are critical for the Tivoli Enterprise Monitoring Server performance at start and stop times and are important for communication between Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal Server The installation should check the prerequisites and show information in one of the logs such as the candle installation log or the Tivoli Enterprise Monitoring Server log If one of the prerequisites is missing the installation wi
191. ctory CMS_EXTERNALBROKERS Whether there are internal brokers CMS_FTO Whether to use Tivoli Enterprise Monitoring Server Hot Standby CMS_MSGBASE Specifies the Tivoli Enterprise Monitoring Server message file directory CMS_NODEID The Tivoli Enterprise Monitoring Server IBM Tivoli Monitoring Node ID CMS_NODE_VALIDATION Enables the rejection of incorrect managed system names KDC_GLBSITES Specifies the Tivoli Enterprise Monitoring Server global sites network file with Tivoli Enterprise Monitoring Server Hub host names KDH_SERVICEPOINT The service point KDS_CATLGLIB Specifies the catalog library KDS_VALIDATE KDS_HUB Specifies that this Tivoli Enterprise Monitoring Server is a Hub Tivoli Enterprise Monitoring Server LOCAL or Remote Tivoli Enterprise Monitoring Server REMOTE KDS_NCS Whether to use IBM Tivoli Monitoring network lookup services KDS_RULELIB Specifies the rule library KDS_RUN Specifies Tivoli Enterprise Monitoring Server components or probes to run at Tivoli Enterprise Monitoring Server startup KDS_START Specifies Tivoli Enterprise Monitoring Server KDS component startup Whether to use Tivoli Enterprise Monitoring Server authentication KGLCB_FSYNC_ENABLED Whether to enable UNIX or Linux fsync calls Appendix B Environment variables 237 Table 54 Tivoli Enterprise Monitoring Server environment variables continued Variable Purpose KGL_CBT
192. d and Task Reference for more information and the Configuring Tivoli Enterprise Console integration section in the IBM Tivoli Monitoring Administrator s Guide for more information Receiving error message when running the Rule Check Utility with the cd and rd options Ensure the class definitions in the TEC_CLASSES directory that you specified with the cd option include the supporting classes for all rule sets defined in the rule base directory specified by the by rd option Table 49 Rule Base directory and BAROC event class definition directory Option Description rd Used to specify a directory containing the rulesets rls to be checked If not provided the TEC_RULES subdirectory for the actively loaded rule base is used by default Chapter 14 Event synchronization troubleshooting 221 Table 49 Rule Base directory and BAROC event class definition directory continued Option Description cd Used to specify a directory containing the BAROC event class definitions files baroc to be used as input If not provided the TEC_CLASSES subdirectory for the actively loaded rule base is used by default The following lists the syntax for the Rule Check Utility wrules_check h v rd rules_directory cd baroc_classes_directory class attribute attribute classN attributeN attributeN f class_file of output_file For more information about the Rules
193. d is further constrained by the calendar entries you specified In addition historical data collection must be configured and enabled for the attribute group and the warehouse proxy agent must be configured and running on the same host as the managed system Verify that historical data collection is enabled for the appropriate attribute group and that the warehouse proxy agent is installed and running on the same host s as the specified managed system or managed systems Problems with Take Action commands and curly brackets Take Action commands that are created with curly brackets are not recorded and cannot be selected from the Tivoli Enterprise Portal This happens only for and not for and only when the backward curly brace is embedded with single quotes There has been a change in syntax where attributes that are enclosed in curly brackets no longer are required to have quotes See the following example amp grpl atrl amp grp2 atr2 Chapter 11 Command troubleshooting 183 When configuring the Monitoring Agent for Sybase and the Warehouse Proxy Agent receive message to use CandleManage The CandleManage command has been deprecated The message should reference the itmemd manage command listSit command with the type option fails with a KUIC02001E message on Japanese Systems Edit the kuilistsitV1d xml file to replace the following text lt Type argl t arg2 type Type String Validation
194. ddress Spaces can configure be configured to use SNA exclusively or in conjunction with TCP IP as a transport service This configuration is done in the environment member xxxENV of RKANPAR If SNA services are viewed as optional then removal of KDCFC_ALIAS KDCFC_MODE and KDCFC_TPNAME from the xxxENV member of RKANPAR will effectively disable use of SNA Chapter 5 Installation and configuration troubleshooting 91 92 Initializing the SNA The following messages are printed in the RAS1 RKLVLOG when the local SNA configuration is processed from the XXXENV member of RKANPAR kbbssge c 52 BSS1 GetEnv KDCFP_ALTAS KDCFC_ALIAS KLXBS_ALIAS ASTAGLB kbbssge c 52 BSS1 GetEnv KDCFP_TPNAME KDCFC_TPNAME KLXBS_TPNAME SNASOCKETS kbbssge c 52 BSS1 GetEnv KDCFP_MODE KDCFC_MODE KLXBS_MODE CANCTDCS kdeslrp c 140 getEnv AF_SNA configuration Alias ASIAGLB Mode CANCTDCS TpName SNASOCKETS KDCFC_ALIAS identifies the APPL definition of the Independent Logical Unit to be used in this process KDCFC_MODE identifies the LOGMODE name the same name found in the LOGMODE specification of the KDCFC_Alias APPL definition KDCFC_TPNAME is the Transaction Processing Name The message which indicates the LOCALLU is operational the configuration is good is the transport opened message kdelotp c 118 KDE1I_OpenTransportProvider Transport opened coml sna pipe Repairing SNA initialization failures There following lists the reasons for SNA in
195. do not start e Agents data is corrupt Restart the system to resolve the problems Tivoli Data Warehouse database does not uninstall When you uninstall IBM Tivoli Monitoring the Tivoli Data Warehouse database is not removed and the ODBC data source exists You must remove the Tivoli Data Warehouse database and the ODBC manually Chapter 5 Installation and configuration troubleshooting 105 The agent installation log on the endpoint indicates that error AMXUT7512E occurred The error AMXUT7512E might occur when running the Distributed Monitoring Upgrade Toolkit The agent was not uninstalled for one of the following reasons e There is another uninstallation in progress that cannot complete until the computer is restarted OR e The uninstallation requires stopping a process that is currently in use by a another component Refer to the 1cfd 10g on the endpoint and agent installation log as listed in Table 33 to determine the exact cause of the problem Table 33 Installation logs Windows UNIX based systems install _Dir Install Abort IBM Tivoli install_Dir logs candle_installation log Monitoring timeStamp log You can manually uninstall the operating system agent by running the command for your platform as listed in Table 34 Table 34 Uninstall OS command Windows UNIX based systems LCF_BINDIR TME ITMUpgrade LCF_BINDIR TME ITMUpgrade ITMUpgradeManager setup ITMUpgradeManager unin
196. e click Troubleshooting and support Searching knowledge bases and select Web search From this topic you can search a variety of resources including the following e IBM technotes e IBM downloads e IBM Redbooks e IBM developerWorks e Forums and newsgroups e Google Obtaining fixes A product fix might be available to resolve your problem Obtain these fixes from the following site http www14 software ibm com webapp download home jsp Copyright IBM Corp 2005 2010 247 Accessing terminology online The IBM Terminology Web site consolidates the terminology from IBM product libraries in one convenient location You can access the Terminology Web site at the following Web address http www ibm com software globalization terminology Accessing publications online IBM posts publications for this and all other Tivoli products as they become available and whenever they are updated to the Tivoli Documentation Central Web site at http www ibm com tivoli documentation Note If you print PDF documents on other than letter sized paper set the option in the File gt Print window that allows Adobe Reader to print letter sized pages on your local paper Ordering publications You can order many Tivoli publications online at www elink ibmlink ibm com publications servlet pbi You can also order by telephone by calling one of these numbers e In the United States 800 879 2755 e In Canada 800 42
197. e Console event synchronization functionality See the IBM Tivoli Monitoring Installation and Setup Guide for instructions The expected rule base was not loaded Actively load the expected rule base You can run the following command from the IBM Tivoli Enterprise Console event server to load a rule base wrb loadrb rule_base where rule_base The name of the rule base to load This makes rule_base the currently active rule base on the event server The rule base must already be defined at the server and any event class specification files and rule files in the directory must be valid The loaded rule base replaces all event class specifications and rules currently defined at the server Only one rule base can be active at a time Loading another rule base overwrites the currently active rule base Note Notes 1 If you do not compile the rule base before attempting to load it you get an error message 2 You must stop and restart the TEC server for changes to the rule base to be active Use the following commands e wstopesvr S server e wstartesvr S server See the IBM Tivoli Enterprise Console documentation for more information about loading rule bases including the IBM Tivoli Enterprise Console Command and Task Reference 216 IBM Tivoli Monitoring Troubleshooting Guide Table 46 Forwarded events display with the PARSING_FAILED status continued Problem Resolution There are missing B
198. e Forwarder process stops abnormally run the stopSUF sh stopSUF cmd on Windows command before restarting the process The correct Tivoli Enterprise Monitoring Server information is not defined for the updated events in the cms_hostname attribute 1 Ensure the event cms_hostname attribute includes a fully qualified host name for the computer where the monitoring server is running The monitoring server hostname must match the hostname recorded in events coming from this monitoring server 2 Ensure user validation is enabled on Tivoli Enterprise Monitoring Server and the correct userid and password information is specified for the Tivoli Enterprise Monitoring Server 3 Run the sitconfsvruser sh script to update the configuration for the Tivoli Enterprise Monitoring Server Refer to the IBM Tivoli Monitoring Administrator s Guide for more information There is a communication problem between IBM Tivoli Enterprise Console server and Tivoli Enterprise Monitoring Server Run the test sh command on UNIX based systems or the test cmd on Windows to determine if there is a communication problem between IBM Tivoli Enterprise Console server and Tivoli Enterprise Monitoring Server Refer to Chapter 6 Connectivity troubleshooting DI information about solving connectivity problems on page 107 for additional The expected server ports 3661 and 1920 are not currently open on the Tivoli Enterprise Monitoring Server
199. e History Collection Configuration window If you have configured historical collection for the monitoring agents and upgraded to IBM Tivoli Monitoring v6 2 2 you might notice that the names listed for attribute groups are now unintelligible 160 IBM Tivoli Monitoring Troubleshooting Guide In some cases these names are not friendly and hovering over the item on the navigation tree shows the attribute group to which it belongs You can alter the name of these collections by editing the historical collection and modifying only the name History collection fails to display the most recent 24 hours of data When requesting a display of the historical availability for beyond 24 hours the most recent 24 hours of data was not displayed only the 48 hours previous to the current 24 was displayed This seems to be a case where data is retrieved correctly from the long term store in the warehouse but is not retrieving data from the local short term history There can be a variety of factors that lead to a failed history collection e The size of the history data file affects the amount of time it takes the agent to read the file e The amount of data the agent is trying to send to the monitoring server e The bandwidth of the communications for example a slow data rate Situations with attributes from more than 1 group not supported with autonomous agent When using autonomous agent in Manage Connected Mode creating situations with AND
200. e Portal client Do the following to ensure that data can return to the Tivoli Enterprise Portal client e Ensure that the monitoring agent is online e Verify that all the application related files were installed with the Tivoli Enterprise Portal Server e Check the kfwras1 1log for errors e Set the following trace option in the KFWENV file UNIT ctsql INPUT DirectDraw thread loops infinitely causing poor Tivoli Enterprise Portal performance Java uses DirectDraw by default but VMWare does not support DirectDraw Perform the following steps for both desktop and browser clients from Manage Tivoli Enterprise Monitoring Services to set Java Runtime not to use DirectDraw for graphic rendering 1 Right click Tivoli Enterprise Portal 2 Select Advanced gt Edit Variables 3 Double click the line with sun java2d noddraw to open the Edit Tivoli Enterprise Portal Parm window Chapter 7 Tivoli Enterprise Portal troubleshooting 127 Ensure the value is true Click the In Use box so to place a check mark in the box Click OK to close the window Click OK to close the configuration window vo oO Workflow Editor is disabled and the following tools do not display Event Console Graphic View Edit Navigator View Navigator view toolbar If you did not enable Manage Tivoli Enterprise Monitoring Services during installation the Workflow Editor is disabled and the following tools do not display e Event Console
201. e Proxy is installed Chapter 13 Database troubleshooting 207 2 Restart the Windows computer so that the Warehouse Proxy windows service recognizes the change SQL0552N ITMUSER does not have the privilege to perform operation CREATE BUFFERPOOL SQLSTATE 42502 208 If the Warehouse database user does not have the correct permission the following error can occur 42ED71FA 0000 E4C khdxbase cpp 250 setError Error 20 3 552 FFFFFDD8 0 executing SQLExecute 42ED71FA 0001 E4C khdxbase cpp 266 setError Error IBM CLI Driver DB2 NT SQL 552N ITMUSER does not have the privilege to perform operation CREATE BUFFERPOOL SQLSTATE 42502 When you configure a DB2 Warehouse Proxy connection from the Manage Tivoli Enterprise Monitoring Services utility using the Configure DB2 Datasource for Warehouse window the user ID the Warehouse Proxy uses to connect to the warehouse database must have SYSADM permission SYSADM permission is required to create an 8K Tablespace and Bufferpool Windows If the database is on Windows the user must be a member of the local Administrators group UNIX based system If the database is on Linux or UNIX user must belong to the SYSADM group 1 Log in as the DB2 instance owner usually su db2inst1 2 Run the following command to determine the group that the UNIX based system user must belong db2 get dbm cfg grep SYSADM IBM Tivoli Monitoring Troubleshooting Guide Chapte
202. e SNMP is in the list of data providers being started e The host of the SNMP Data Provider is configured to receive traps e Agents are configured to send traps to the host of the SNMP data provider e That the trap listener started successfully by checking the UAGENT DPLOG workspace for the SNMP Data Provider e KUMP_SNMP_MONITOR_TRAP is not set to N e You add the trap description data to the file trapcnfg if a trap field shows UNAVAILABLE SNMP metafiles are imported but MIB data collection is not working The MIB application workspaces are empty after initiating MIB data collection Take Action gt Monitor Start Table 42 MIB data collection is not working Problem Resolution The community name was not specified correctly for the Use an MIB browser or freeware utility such as snmputil SNMP agent to retrieve MIB variables from this SNMP agent to determine the source of the problem The SNMP agent is not responding to any SNMP Get requests The MIB OIDs on the SNMP agent host do not match the OIDs specified in the Universal Agent metafile The agent does not respond to basic SNMP Get requests All SNMP agents support the MIB2 OIDs such as from Universal Agent SysName and SysLocation Use the rfc1213 metafile to establish a baseline of functionality Use the Universal Agent RFC1213_mib 2 mdl to collect MIB 2 data from the agent to verify that the SNMP agent responds to basic SNMP Get requests from Universal
203. e error KLVSQ000 carved mode in effect for extended storage occurred in the RKLVLOG during startup 147 147 148 148 148 148 149 149 150 150 150 150 150 151 152 152 152 152 153 Error message KDSMA013 OPEN VTAM for VDMIAPPL failed with status 8 occurs in the Tivoli Enterprise Monitoring Server start task CANSDSST default 153 Chapter 10 Monitoring agent troubleshooting 155 Linux OS agent fails to start and remain started 155 Configured non root user agent starts up as root 155 Unable to access iii Collection Configuration for any agent 155 Agent names and icons are displayed incorrectly 155 64 bit monitoring agents are not started 155 Errors in the configuration xml file 156 Subnode Limitations for autonomous function 157 Binary Path attribute of the Windows OS A does not show a value i 157 Installing pre v6 2 1 Monitoring Agent for Windows OS onto a v6 2 1 or later monitoring server inadvertently unconfigures the monitoring server 158 The tacmd Section commend fails for certain take actions 158 OS agent restarted unexpectedly c on heavily loaded systems P 159 Calendar entries overlap oe 159 Receive an error when deploying an System Service Monitor agent i 159 Attributes differ between the sit ation action commands and
204. e information about tracing for IBM Tivoli Verifying that events are being received by Tivoli Enterprise Console On the Tivoli Enterprise Monitoring Server set this UNIT trace level ERROR UNIT kfaot ALL This will trace the communication between the Tivoli Enterprise Monitoring Server and Tivoli Enterprise Console The _ms_ logs from the Tivoli Enterprise Monitoring Server show this traffic You can then verify that events are being received by Tivoli Enterprise Console by looking at the wtdumprl output If there are no PARSE FAILED messages and the Tivoli Enterprise Console is working correctly the events should be processed and show in the Tivoli Enterprise Console Enabling tracing of events from the Tivoli Enterprise Console to the Tivoli Enterprise Monitoring Server Complete the following steps to enable tracing of events 1 Rule tracing can be set up for the omegamon rls In the OMEGAMON rule trace file verify that the exec_program predicate is called in the rules trace to indicate that the rule set invoked situpdate sh A line like this will be seen in the log 214 IBM Tivoli Monitoring Troubleshooting Guide 357 call exec_program 0x209372e8 OM_TEC bin situpdate sh size s num s path s s 50000 10 space Tivoli_ 4141 var TME TEC OM_TEC persistence tmp SUFO tmp N0 Tracing can be set up within the situpdate sh running by editing the script the situpdate sh script is located at BINDIR TME
205. e remote monitoring servers or to monitoring agents 4 Correct the communication failures indicated in the logs Chapter 3 Solving common problems 13 Table 3 Correcting monitoring agent problems e The monitoring agent is running but no data is being returned e The monitoring agent log contains an error message similar to Endpoint unresponsive Diagnostic and corrective actions Note For monitoring agents on z OS see each product s Problem Determination Guide or the IBM Tivoli Monitoring Troubleshooting Guide 1 Verify that the agent is connected Check the monitoring server log for messages similar to Remote node lt SYS MQIRA gt is ON LINE 2 If the agent is online check to see whether subnodes are online in the agent log For example KMQMI171I Node JSG1 SYS MQESA is online 3 If subnodes are online are workspaces showing correct titles No Verify that application support has been installed correctly and buildpresentation bat ran correctly Yes Go to the next step 4 If workspaces contains titles is there a column heading No Verify that application support has been installed correctly and buildpresentation bat ran correctly Yes Go to the next step 5 If there is only a column heading with no data turn on UNIT KRA ALL in the agent and verify that rows are being returned when the workspaces are displayed 14 IBM Tivoli Monitoring Troubleshooting Guide Status of a monitoring agent is mi
206. e using the Configuration Tool You receive the message serverName is unknown host This error message is displayed as red text at the bottom of a report Although this error can occur for several reasons the most common problem is incorrectly entering the Tivoli Data Warehouse database URL into the data source for the report If some reports are working and others are not the URL was entered incorrectly If no reports are working contact those within your organization who installed the product for the correct URL You receive the message Empty Data Set The Empty Data Set message is not an error The message is displayed to convey that no data was returned when it ran the report The message is usually displayed for one of three reasons e The report parameters have excluded all the data e You recently started using IBM Tivoli Monitoring for Energy Management and have not put data into the Tivoli Data Warehouse e The computers you are monitoring do not have the correct level of firmware If you think you have received this message incorrectly run the Exception Report to find if there is another reason why no data is being returned 232 IBM Tivoli Monitoring Troubleshooting Guide Appendix A IBM Tivoli Monitoring processes The following table lists the processes according to operating system platform and IBM Tivoli Monitoring component Table 51 IBM Tivoli Monitoring processes by operating system
207. eat registers its changing properties Other solutions besides ensuring that each managed system name is unique are the following e Detect and stop the agent process that is running improperly This can be done by checking the Tivoli Enterprise Portal Server Managed System Status network address of the managed system name that seems to toggle ON and OFF line Go to the machine indicated in the network address and check for multiple running monitoring agents If the agents running on the same system are the same product stop or kill the unintended agent process Delete the faulty agent managed system name from the enterprise managed system status so that the new managed system name can register properly with the Tivoli Enterprise Monitoring Server You might need to stop the correct agent process so that it is OFF line One monitoring agent s workspaces are listed under another agent node on the Tivoli Enterprise Portal 172 This problem has been seen in ira based agents that exist on the same system Set the CTIRA_HOSTNAME environment variable configuration file for the monitoring agent as follows Open the configuration file for the monitoring agent which is located in the following path e On Windows install_dir tmaitm6 Kproduct_codeCMA INI For example the product code for the Monitoring Agent for Windows OS is NT file name for is KNTCMA INI IBM Tivoli Monitoring Troubleshooting Guide On UNIX and Linux inst
208. ecific error codes s TCP IP SOCKETS DB2ConnectionCorrelator G92A17E8 C3D2 071018074633 at com ibm db2 jcc b a a a java 373 at com ibm db2 jcc b a y a java 346 Chapter 10 Monitoring agent troubleshooting 175 at com ibm db2 jcc b a 1 a java 298 at com ibm db2 jcc c j c j java 234 at com ibm db2 jcc c uf 1b uf java 1934 at com ibm db2 jcc c uf addBatch uf java 1348 at com tivoli twh khd khdxjdbc addBatch khdxjdbc java 1290 Check the ethernet adapter settings on both the client and server There are problems if the adapter is set to Auto and the switch is set to 100 Full Duplex When configuring the Monitoring Agent for Sybase and the Warehouse Proxy Agent receive message to use CandleManage The CandleManage command has been deprecated The message should reference the itmemd manage command listSit command with the type option fails with a KUIC02001E message on Japanese Systems Edit the kuilistsitV d xml file to replace the following text lt Type argl t arg2 type Type String ValidationRegExp A Za z0 9 _ amp Required Y gt with the following text lt Type argl t arg2 type Type String ValidationRegExp A Za z0 9 _ Required Y gt Take Action command names do not accept non English characters There is not a workaround at the present time Creating a situation from a group member does not copy the distribution list The indirect assignments coming fr
209. ect specified for connection to the _ information for connection to the Tivoli Enterprise Console server Tivoli Enterprise Console server in the om_tec config file ServerLocatiot Specifies the host name or IP address of IBM Tivoli Enterprise Console server ServerPort Specifies the IBM Tivoli Enterprise Console server listening port Specify 0 if the IBM Tivoli Enterprise Console server uses port mapper Ona Windows system 1 Click Start gt Programs gt IBM Tivoli Monitoring gt Manage Tivoli Enterprise Monitoring Services 2 Right click Tivoli Enterprise Monitoring Server gt Advanced 3 Select Edit EIF configuration to open the om_tec config file in Notepad 4 Specify the correct parameters and save the file e On a UNIX based systems system 1 cd opt IBM ITM tables hubname TECLIB where hubname Specifies the name of the hub monitoring server 2 Open the om_tec config file in a text editor 3 Specify the correct parameters and save the file where hubname Specifies the name of the hub monitoring server 212 IBM Tivoli Monitoring Troubleshooting Guide Table 45 Troubleshooting when situations are not forwarded continued Problem Resolution Events are filtered out based on the parameter values in the om_tec config file Edit the om_tec config file in the following locations On a windows system c ibm itm cms TECLIB e On UNIX based systems opt IBM ITM tableshost
210. ect on log file names Setting CTIRA_HOSTNAME with the virtual hostname shows the agent on the workspace as one entity no matter the node on which it is running However the setting has no effect on the log file names These names still use the local nodename instead of the virtual hostname The Summarization and Pruning Agent and the Warehouse Proxy Agent do not work with DB2 9 1 Fix Pack 2 Do not try to use these agents with this version of DB2 An error of can bind a LONG value only for insert appears The following message appears in the Warehouse Proxy Agent ORA 01461 can bind a LONG value only for insert into a LONG column Upgrade to Oracle 10 1 0 5 or later Errors in either the Warehouse Proxy Agent or Summarization and Pruning Agent logs You receive the following error in either the Warehouse Proxy Agent or Summarization and Pruning Agent logs DB2 SQL error SQLCODE 964 SQLSTATE 57011 SQLERRMC null The solution is to increate the DB2 logging for the warehouse database See the IBM Tivoli Monitoring for Databases DB2 Agent User s Guide for more information Receive a message saying that the statement parameter can only be a single select or a single stored procedure You receive the following message when connecting to a Microsoft SQL Server 2000 database on a Windows 2000 system from a Warehouse Proxy Agent on a Linux system using Microsoft SQL Server 2005 JDBC driver 1 2 April 2007 SSQL error Ex
211. ections e IBM Tivoli Monitoring z OS initialization checklist on page 88 e z OS based installations problems and resolutions on page 95 IBM Tivoli Monitoring z OS initialization checklist Use the IBM Tivoli Monitoring z OS initialization checklist to troubleshoot problems with your IBM Tivoli Monitoring installation on z OS The IBM Tivoli Monitoring z OS initialization checklist includes the following sections e The Server list on page 9 i Tivoli Monitoring Services Engine initialization Tivoli Monitoring Services Engine is a collection of basic Operating System and Communication service routines built specifically for the OS 390 and z OS Operating environments All IBM Tivoli Monitoring address spaces load and employ the services of Tivoli Monitoring Services Engine Initializing the Tivoli Monitoring Services Engine service Tivoli Monitoring Services Engine successful initialization is noted by message KLVIN408 IBM OMEGAMON PLATFORM ENGINE VERSION 400 READY There are two classes of Tivoli Monitoring Services Engine initialization failures e Failures that result from unsupported Tivoli Monitoring Services Engine startup parameters For example User abend U0012 e Failures that result from protocol initialization failures For example User abend U0200 Repairing Tivoli Monitoring Services Engine initialization failures For U0012 Abends incorrect Engine STARTUP parameters examine and correct the param
212. ector of Licensing IBM Corporation North Castle Drive Armonk NY 10504 1785 U S A For license inquiries regarding double byte DBCS information contact the IBM Intellectual Property Department in your country or send inquiries in writing to Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd 1623 14 Shimotsuruma Yamato shi Kanagawa 242 8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement might not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements and or changes in the product s and or the program s described in this publication at any time without notice Any references in this information to non IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at t
213. ectory logs candle_installation 1og file and look for references to runGSkit For example output similar to the following might be present runGSkit Running command opt IBM ITM 1s3263 gs bin private_verifyinstal opt IBM ITM 1s3263 gs bin gsk7ver error while loading shared libraries libstdc so 5 cannot open shared object file No such file or directory Error Verify Failed Expected Details of gskit in opt IBM ITM 1s3263 gs Got runGSkit return code from command is 99 runGSkit End of running command runGSkit error Return error code 99 runGSkit error GSKit check failure script opt IBM ITM 1s3263 gs bin private_ verifyinstall runGSkit error 183263 GSK check error verifyInstall test failed IBM Tivoli Monitoring Troubleshooting Guide In the example above the libstdc s0 5 file is not present This comes from the package compat libstdc 33 3 2 3 61 s390 rpm When this package is installed the problem no longer occurs Receive KUIC02101W error The error states that the Java Runtime Environment shared library could not be loaded Java will execute using unsecured credentials On some Solaris Linux and HP platforms IBM Tivoli Monitoring has files with different bit sizes executable files are 64 bit while the provided JRE is 32 bit This mismatch does not allow the JNI to work properly so the current method cannot be used and the previous method will be used Receive JVMDG080 or JVMXM01
214. ed applications in the Log and Trace Analyzer online help IBM Tivoli Monitoring Troubleshooting Guide e You can use symptom catalogs to quickly recognize known problems The Log and Trace Analyzer provides a log analysis capability that allows it to recognize known problems that are defined in a knowledge database called the symptom catalog IBM provides a symptom catalog for known problems with several products including IBM Tivoli Monitoring It also provides a way for you to capture and define your own symptom information For more information refer to the topic titled Synchronizing time of log records for distributed applications in the Log and Trace Analyzer online help Backspace Check utility On UNIX systems if you have incorrectly configured the backspace key you will see the following e When you press the backspace key characters such as A and AH are displayed on the screen e The backspace key seems to be working correctly when entering text but you later find characters such as and AH in configuration files and your software malfunctions Configure your terminal and stty erase to use the same key code for backspace Consider using as the key code Verify your configuration with the IBM Tivoli Monitoring distributed utility Install BackspaceCheckUtility Build TEPS Database utility You can use this utility to build a blank database Prior to the IBM Tivoli Monitoring v6 1 release
215. ed for the portal server the sysadmin ID should exist in the monitoring server s local OS user registry but should not be defined in the LDAP server otherwise the sysadmin ID will not be able to login On AIX systems newly created users with auto expire passwords cause installation failures When installing on AIX systems security policies for newly created users auto expire the password after the first use and require you to set a new or same password as a permanent password The Tivoli Enterprise Portal Server configuration interface allows you to create a new user ID for the portal server and warehouse database but using the interface always fails because the user password is not set and is expired You must ssh telnet into the same server using the target user ID and set the password appropriately Copyright IBM Corp 2005 2010 133 Linux portal server unable to FTP catalog attribute files Linux portal server unable to FTP catalog attribute files with Manage Tivoli Enterprise Monitoring Server A monitoring server is needed along with the portal server on the same system to be able to seed a z OS monitoring server Upgrading the Tivoli Enterprise Portal Server takes a long time Performing a Tivoli Enterprise Portal Server upgrade depending on the efficiency of the hardware platform can take anywhere from 30 minutes to over an hour Running the Tivoli Management Services Discovery Library Adapter results in a
216. educe the amount of data kept on disk after a successful upload to the warehouse is performed KIB_MAXCOLS Tivoli Enterprise Monitoring Server internal dictionary column maximum KMS_DISABLE_TEC_EMITTER TEC Emitter to be disabled KMS_OMTEC_ GLOBALIZATION_LOC TEC Integration Globalization locale KMS_OMTEC_INTEGRATION TEC Integration enabled KPX_WAREHOUSE_LOCATION Allows a fixed warehouse route for the agents connected to that Tivoli Enterprise Monitoring Server when the usage of the Global Location Broker default algorithm is not supported It is a list of fully qualified semicolon delimited network names such as KPX_WAREHOUSE_LOCATION family_protocol network_address port_number KPX_WAREHOUSE_REGCHK Number of minutes to wait between re checking the Global Location Broker for any warehouse proxy agent registration changes The default value is set to 60 minutes KSH_DIRECTORY Specifies the Tivoli Enterprise Monitoring Server SOAP Server HTML files directory KT1_TEMS_SECURE MSG_MODE YES is required for the tacmd commands putfile getfile and executecommand Also required when executeaction is issued with advanced options Not required when issuing executeaction to agent without advanced options Specify kms if you want to use IBM Tivoli Monitoring Operations Logging Specify MSG2 to use MSG2 logging NLS1_LOCALEDIR The directory of the locale file RKDSCAT
217. elete it after the situation s associated event is displayed in Tivoli Enterprise Console the situation_fullname slot is missing The expected result would be that for the deleted event the situation status is set to D Events do not reflect the long names if the definition is deleted and the Tivoli Enterprise Monitoring Server logs also do not reflect the long name Logs are using the situation ID string instead of the display name Situations have both a display name up to 256 bytes in length UTF8 enabled and an ID string up to 32 bytes in length For example a situation display name can be don t let the pigeon drive the bus or 100 characters of Japanese text The associated ID string will be of the form 000000000000008D723DE7DFED450F If the Tivoli Enterprise Portal Server Universal Message Console and the Tivoli Enterprise Monitoring Server RKLVLOG cannot display the situation display name the ID string is displayed instead To ensure that your situations are displayed using the display name instead of the display ID make sure that your situation names do not exceed 31 characters in length and that they do not contain any special characters If a managed system list is removed for a situation the situation stops If a managed system list is removed for a situation the situation stops At this point you see a message in the Message Log that the situation has been stopped and deleted for the Remote Tivoli Enterpri
218. emove a failed installation from a computer that you were trying to upgrade from Candle OMEGAMON Removing a failed first time installation Use the following steps to remove a partially installed IBM Tivoli Monitoring installation 1 12 13 14 15 16 17 Ensure that there is no entry in the Add and Remove Programs tool for the component that you attempted to install If there is an entry use that entry to uninstall the product If there is no entry proceed to the next step Open the Windows Explorer and navigate to the IBM Tivoli Monitoring installation directory C IBM ITM by default Launch the Manage Tivoli Enterprise Monitoring Services utility by double clicking the KinConfg exe file located in either the Install or InstallITM subdirectory If any agents the portal server or the monitoring server are listed in the Manage Tivoli Enterprise Monitoring Services window right click each and click Advanced gt Unconfigure Repeat this step for all components that are listed Close the Manage Tivoli Enterprise Monitoring Services utility Open the Windows Control Panel Double click Administrative Tools and then double click Services Verify that all related IBM Tivoli Monitoring services have been removed These services match those listed in the Manage Tivoli Enterprise Monitoring Services window Open the Registry Editor by clicking Start gt Run and typing regedt32 Click OK
219. encountered error message KLVST044 LOADLIST MEMBER NOT FOUND IN RKANPAR DATASET KDSLLIST KppLLIST KLVST001 CANDLE ENGINE INITIALIZATION ERROR S ABEND U0012 in the RKLVLOG at startup Ensure the following conditions e The pp 2xxxx RTE Load job ran successfully for this RTE 1 To perform the RTE Load place the L option next to the RTE on the KCIPRTE RTE main menu The Configuration tool generates the pp 2xxxx RTE Load job 2 Submit the RTE Load job The RTE Load job populates the amp rhilev amp RTE RK runtime libraries by copying required elements from the SMP E target high level qualifiers amp thilev TK e If the RTE Load job was performed then the job references the amp thilev TK SMP E target datasets from where the members are copied Ensure the datasets are correct SMP E target datasets where amp svhilev CSI is installed e If the RTE Load job does not reference any amp thilev TK SMP E target datasets or generates an incomplete RTE Load job then ensure that the Configuration tool references the correct SMP E target high level qualifiers The Configuration tool only generates appropriate IEBCOPY TK gt RK steps for SMP E target datasets that it can access Use the following steps to ensure that the Configuration tool references the correct SMP E target high level qualifiers 1 From the Configuration tool main menu select Install products or maintenance gt Set up product or maintenance environment gt Specify en
220. ent User s Guide SC23 5239 HMC Base Agent User s Guide SA23 2239 VIOS Premium Agent User s Guide SA23 2238 e Other base agents Monitoring Agent for IBM Tivoli Monitoring 5 x Endpoint User s Guide S C32 9490 Related publications You can find useful information about related products in the IBM Tivoli Monitoring and OMEGAMON XE Information Center at publib boulder ibm com infocenter tivihelp v15r1 Other sources of documentation You can also obtain technical documentation about IBM Tivoli Monitoring and related products from the following sources e IBM Tivoli Open Process Automation Library OPAL http www ibm com software tivoli opal OPAL is an online catalog that contains integration documentation and other downloadable product extensions e Redbooks http www redbooks ibm com IBM Redbooks and Redpapers include information about products from platform and solution perspectives e Technotes Technotes provide the latest information about known product limitations and workarounds You can find Technotes through the IBM Software Support Web Tivoli wikis on the IBM developerWorks Web site Tivoli Wiki Central at http www ibm com developerworks wikis display tivoli Home is the home for interactive wikis that offer best practices and scenarios for using Tivoli products The wikis contain white papers contributed by IBM employees and content created by customers and business partner
221. ent is running in its own address space check its RKPDLOG 2008 07 28 08 44 18 Response amp philev RMQSGRP3 1700 83 14 5000 Active Write 2008 07 28 08 44 18 Response amp philev RMQSGRP2 1700 25 0 5000 Empty Read Access 2008 07 28 08 44 18 Response amp philev RMQSGRP1 1700 25 0 5000 Empty Read Access 2008 07 28 08 44 18 Response amp philev RKMQPDS3 23327 31 0 4000 Empty Read Access 2008 07 28 08 44 18 Response amp philev RKMQPDS2 23327 6598 143 4000 Partial Read Access 2008 07 28 08 44 18 Response amp philev RKMQPDS1 23327 3523 105 4000 Active Write See for various error codes to look for in the RKPDLOG 24 IBM Tivoli Monitoring Troubleshooting Guide Table 15 Correcting persistent data store for z OS problems continued 4 5 6 Zi For further information about the persistent data store see OMEGAMON XE and Tivoli Management Services on z OS Common Planning and Configuration Guide The following are examples of persistent data store error codes found in the RKPDLOG Error code 25804 Indicates that an attempt was made to read slot 0 of the GENHIST dataset This is a protected record and the persistent data store will not allow the slot to be read One possible cause is a problem with DELETE processing The warehouse code which is the only code that attempts to use the delete logic might be generating bad condition Run the RECOVERY command which will save the data and rebuild the indexes so that the d
222. eout value is less than the warehousing interval the database connections may timeout This results in numerous error messages written to the Warehouse Proxy agent log The Warehouse Proxy agent Statistics workspace will also show a large number of Disconnections in the Failure Disconnections view To avoid repeated disconnections consider increasing the DB2 idle thread timeout value to a value higher than the warehousing interval Specifying a value of 0 disables time out processing If time out processing is disabled idle server threads remain in the system and continue to hold their resources if any Chapter 13 Database troubleshooting 205 For more information on the DB2 IDLE THREAD TIMEOUT field IDTHTOIN subsystem parameter refer to the DB2 Version 9 1 for z OS Installation Guide Historical data is not warehoused Check the following Warehouse Proxy agent logs for errors that indicate why historical data is not warehoused e Windows Event Log all critical errors e WHProxy Agent RAS1 Log e Operations Log The Warehouse Proxy agent contains an audit trail for each export written to the warehouse database You can also check the database table called WAREHOUSELOG as it contains the same information as the logs Historical data for logs is incorrect If there are duplicate or missing rows in a table incorrect historical data is collected for logs such as managed system or situation status Correct the incorrec
223. epeatedly times out because of broken communication network outage or the SOAP server is down The event message indicates reset resurface or acknowledge action failed The original event was not in the Tivoli Enterprise Console rules cache Verify that the network connections are up and running the SOAP server is running and receiving requests and the ports 3661 and 1920 on the Tivoli Enterprise Monitoring Server machine are open Use the following steps to determine why the error occurred 1 Ensure that the User ID and password configured for the Tivoli Enterprise Monitoring Server Server ID on the event server during installation is correct Run the sitconfsvruser sh script to update the configuration for the Tivoli Enterprise Monitoring Server Refer to the IBM Tivoli Monitoring Administrator s Guide for more information 2 Run the test sh command on UNIX based systems or the test cemd on Windows to determine if there is a communication problem between IBM Tivoli Enterprise Console server and Tivoli Enterprise Monitoring Server 3 If the SOAP Server is running and receiving requests initiate a CT_GET request for the SOAP server from the Web client See the Tivoli Enterprise Monitoring Web services section in the IBM Tivoli Monitoring Administrator s Guide for more information Increase the size of the rules cache using the Tivoli desktop or the wsetesvrcfg command Refer to the IBM Tivoli Enterprise Console Comman
224. eps 1 Open the configuration file for the monitoring agent which is located in the following path e On Windows amp install_dir tmaitm6 Kproduct_codeCMA INI For example the product code for the Monitoring Agent for Windows OS is NT and the file name is KNTCMA INI On UNIX and Linux install_dir config product_code ini and product_code config For example the file names for the Monitoring Agent for UNIX OS are ux ini and ux config 2 Find the line the begins with CTIRA_HOSTNAME 3 Type anew name for host name that is a unique shorter name for the host computer The final concatenated name including the subsystem name new host name and AGENT_CODE cannot be longer than 32 characters Note You must ensure that the resulting name is unique with respect to any existing monitoring component that was previously registered with the Tivoli Enterprise Monitoring Server 4 Save the file Restart the agent o 6 If you do not find the files mentioned in Step 1 perform the workarounds listed in the next paragraph If you do not find the files mentioned in the preceding steps perform the following workarounds 1 Change CTIRA_HOSTNAME environment variable in the configuration file of the monitoring agent e Find the KAGENT_CODEENV file in the same path mentioned in the preceding row e For z OS agents find the RKANPAR library e For i5 OS agents find the QAUTOTMP KMSPARM library in member KBBENV Chapter 10
225. er 2 Confirm that this line can be found An OutOfMemory error happened while running the report If you find this line do the following to correct this problem 1 Stop Tivoli Common Reporting 2 Edit the startServer bat file usually found in this location On Windows Systems C Program Files IBM tivoli tip bin 3 Locate this instruction about half way through the bat file JAVA_HOME bin java Dcmd properties file TMPJAVAPROPFILE amp WAS_TRACE WAS_DEBUG CONSOLE ENCODING CLIENTSAS CLIENTSSL SUSER_INSTALL_PROP Dwas install root WAS_HOME com ibm ws bootstrap WSLauncher com ibm ws management tools WsServerLauncher CONFIG_ROOT ZWAS_CELL WAS NODE WORKSPACE_ROOT_PROP 4 Increase the JVM heap size by adding the xms and xmx options as in the example shown below SJAVA_HOME bin java Xms128m Xmx512m Dcmd properties fi le lt 4TMPJAVAPROPFILE WAS TRACE WAS DEBUG CONSOLE_ENCODING CLIENTSAS SCLIENTSSL USER_INSTALL_PROP Dwas install root WAS HOME com ibm ws bootstrap WSLauncher com ibm ws management tools WsServerLauncher SCONFIG_ROOT WAS CELL WAS _NODE WORKSPACE_ROOT_PROP 5 Re start the Tivoli Common Reporting server and try generating the report again Chapter 15 Tivoli Common Reporting troubleshooting 229 The generated report displays the message SQL Error In this scenario you complete the report parameters dialog to generate a report and clic
226. er and the fully qualified hostname is itmserver raleigh ibm com Either update the systems host tables and correctly setup the DNS domain search so that the link between itmserver and itmserver raleigh ibm com can be made or supply the monitoring server fully qualified hostname during the createnode deployment using the SERVER itmserver raleigh ibm com property Take Action commands do not work if unrequired values are left blank The predefined Take Action commands work if None is provided in the keyword and the value fields after at least one database manager configuration parameter config keyword value that you wish to update has been provided Take Action commands do not display messages when run from a Navigator Item or from a workspace view Take Action commands do not display messages when run from a Navigator Item or from a workspace view for return codes 0 1 or 4 The destination and return code is displayed but not the message for the return code Receive corrupted tacmd hist or calendarentries responses in Command Line Interface With the default code page setting some systems might display certain tacmd commands with corrupted characters Below you can find a list of these commands e histconfiguregroups e histcreatecollection e histdeletecollection e histeditcollection e histlistattributegroups Chapter 11 Command troubleshooting 187 e histlistcollections e histlistproduct e histstartcollection
227. er to Internet Explorer or complete the following steps 1 If the help is open close the browser window 2 On the computer where the Tivoli Enterprise Portal Server is installed locate the contents htm file Windows lt install_dir gt cnb classes candle fw resources help UNIX based systems Linux lt install_dir gt cnb classes candle fw resources help Chapter 7 Tivoli Enterprise Portal troubleshooting 123 3 Rename contents htm to contents bak 4 Rename contents _dhtml htm to contents htm Tivoli Enterprise Portal Navigator items are listed in an unexpected order When expanding items in the Tivoli Enterprise Portal Navigator parent items might list child items in an unexpected order Navigator items are ordered internally on the Tivoli Enterprise Portal Server and therefore might not reflect a logical ordering in any particular language Use the More indicator at the end of the branch of child items to append them until you locate a the item you need Clicking on the Timespan icon for one view brings up the data for another view The timespan panel provides this check box at the bottom of the panel Apply to all views associated with this view s query If both views in question share the same query and this check box is selected a change in one view s time span also affects the other as expected However when the check box is unselected this behavior is still exhibited If you want one of the views to
228. ere are many Java heap tuning parameters for IBM Java run time environments that can be used to influence garbage collection and memory management Notable among these are the minimum free percentage Xminf and maximum free percentage Xmaxf parameters IBM Java documentation provides the following descriptions Xminf lt number gt A floating point number 0 through 1 that specifies the minimum free heap size percentage The heap grows if the free space is below the specified amount The default is 3 that is 30 Xmaxf lt number gt A floating point number between 0 and 1 which specifies the maximum percentage of free space in the heap The default is 0 6 or 60 When this value is set to 0 heap contraction is a constant activity With a value of 1 the heap never contracts You can lower the amount of free space maintained in the Java heap at the expense of higher CPU utilization and longer response time by setting the minimum free and maximum free percentages to lower values Default values Xminf0 30 Xmaxf0 60 Consider the following values Xminf0 15 Xmaxf0 30 The IBM Java documentation warns that setting these values too low can cause poor Java performance For more information on Java heap tuning parameters refer to the IBM Developer Kit and Runtime Environment Java 2 Technology Edition Version 1 4 2 Diagnostics Guide SC34 6358 01 which i is available from Data is not returned to the Tivoli Enterpris
229. erent domain 187 The NETWORK workspace is empty 199 Take Action commands do not woik if unrequired A MIB workspace is not showing expected data 200 values are left blank 187 Take Action gt Manage Start or Take Action gt Take Action commands do not display messages Manage Stop is not working 200 when run from a Navigator Item or from a The SNMP Data Provider does not aie with workspace view 187 IBM Tivoli Universal Agent 200 Receive corrupted fend hist or calendarentries You do not see the TakeAction options 200 responses in Command Line Interface 187 A managed system for a MIB application does The listSystems command consumes high CPU i in not come online as expected 200 enterprise environments 189 The NETSUMMARY workspace does not show Improving tacmd command response dime when expected data 200 using VMWare 189 A MIB workspace displays empty when you The addSystem command fails with error message expect to see data 200 KUICCR099E 189 The tacmd getdeploystatus command is not Chapter 13 Database troubleshooting 201 returning status return codes 190 Data loss prevention 201 tacmd createSit does not send errors if you mistype Backing up the database for recovery purposes 201 the name of an attribute 190 Restoring the original database contents 201 wsadmin commands SATS indicates the wrong If you modify you
230. erprise Monitoring Server from Cand eHome cms TECLIB and compile it into the current rule base The IBM Tivoli Enterprise Console is disabled The Tivoli Enterprise Console relies on the Tivoli Management Framework for common services including communications If the console loses contact with Tivoli Management Framework no new events are added to the view Do the following to enable event reporting LS Reestablish contact between Tivoli Enterprise Console and IBM Tivoli Monitoring Log off the Tivoli Enterprise Portal Note If you use Tivoli Enterprise Portal browser client you must also close and restart the Web browser Log in to the Tivoli Enterprise Portal again to enable the Tivoli Enterprise Console view to resume event reporting 218 IBM Tivoli Monitoring Troubleshooting Guide The IBM Tivoli Enterprise Console does not indicate that a situation event was acknowledged or closed Table 47 Situation event was not acknowledged or closed Problem Resolution The omegamon r1s rule set was not imported in the currently loaded rule base Run the following commands from the IBM Tivoli Enterprise Console event server to import a rule set into a rule base 1 wrb imprbrule rule file encoding encoding force rule_base where rule_base The name of the rule base in which the rule pack lives rule_file The file name of the rule set to import to the rule base This must be a path to a
231. erver and agent startup time It is always best to calculate space for 24 hours as best as possible As far as the potential of losing historical data if the warehouse is down over 24 hours that is a potential problem assuming the persistent datastore s backup processing is functioning and therefore the agent does not stop writing to the persistent datastores Since you have the choice of collecting history data at the Tivoli Enterprise Monitoring Server or the Tivoli Enterprise Monitoring Agent the persistent datastore should be defined in both places If you are 100 sure that you will always collect at the Tivoli Enterprise Monitoring Agent or always collect at the Tivoli Enterprise Monitoring Server then you can optionally define the persistent datastore in only one location Note that many configuration issues occur because the person installing the product selects one location for the persistent datastore and sometime later someone else enables history collection for the other location Error list appears in warehouse logs The following error list appears in the warehouse logs 25 t Thread 1 com ibm db2 jcc c DisconnectException A communication error has been detected Communication protocol being used T4Agent sendRequest Communication API being used OutputStream flush Location where the error was detected There is no process to read data written to a pipe Communication function detecting the error Protocol sp
232. erver as it was before upgrading the agent Installation of OS agent on a Mict s ft Windows Server 2003 fails with this error Unable to establish BSS1 environment can t continue After upgrading to IBM Tivoli Monitoring V6 1 the link to the Queue Statistics workspace in the navigation tree is incorrect Some rows do not display in an upgtaded table Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal Server automatically start after running Application Support Installer Errors occur during installation of Event IBM Tivoli Monitoring Event Forwarding tool Windows installation problems and resolutions iv On Windows systems the installation fails randomly when installing different features Problems that are cleared by rebooting the Windows system When installing and configuring the Tivoli Enterprise Monitoring Server on Windows Server 2008 a number of popups and errors occur After an upgrade the Tivoli Enterprise Portal Server is in the stop pending state and cannot be manually started bce When running the setup exe an unknown publisher error message displays IBM Tivoli Monitoring Troubleshooting Guide 74 74 74 70 75 75 75 76 76 76 76 76 AL 77 78 78 79 79 79 79 80 80 80 81 81 82 The error Could not open DNS registry ii occurs Agent not connecting io Tivoli Enterprise Monitoring Ser
233. es tecSyncAl1File 1 Refer to install log for more details One or more errors occured during the replacement of files tecSyncAl1File3 with files tecSyncAllFilel Refer to install log for more details When using a Tivoli Enterprise Console view in a workspace receive an error message instead of event information When you use a Tivoli Enterprise Console view in a Tivoli Enterprise Portal workspace you can right click an event and select Information in the context menu to see details regarding an event However you might see the following message instead of event information C02032E Failed to receive event information from Install and configure the optional Sample Event Information component of Tivoli Enterprise Console This feature enables you to create a help file to record information about an event Use of the Sample Event Information feature is optional However if you do not provide event information for an event you experience the limitation that is described here The IBM Tivoli Enterprise Console does not display in a workspace after logging off and logging on the Tivoli Enterprise Portal Recycle the Tivoli Enterprise Portal browser so the Tivoli Enterprise Portal can display the IBM Tivoli Enterprise Console in a workspace Chapter 14 Event synchronization troubleshooting 223 The Tivoli Enterprise Portal stops when attempting to launch the IBM Tivoli Enterprise Console behind a router After a two
234. es to nonexistent event classes or enumerations or if duplicate event classes or enumeration are defined If the classes in a class file are derived from classes in another file import the class files into the rule base in an order that preserves the inheritance structure For example if class file B baroc contains classes that are derived from class file A baroc you must import class file A baroc first For any changes you make to the rule base to take affect you must complete the following steps e Recompile the rule base Load the rule base Stop the Tivoli Enterprise Console server e Restart the Tivoli Enterprise Console server Chapter 14 Event synchronization troubleshooting 217 Table 46 Forwarded events display with the PARSING_FAILED status continued Problem Resolution There are missing BAROC files that must be imported to support the IBM Tivoli Monitoring V6 1 events Continued Refer to the IBM Tivoli Monitoring Administrator s Guide for information about changing the omegamon r1s file See the IBM Tivoli Enterprise Console Command and Task Reference V3 9 0 for more information about importing class files into rule bases Parsing errors reported in Tivoli Enterprise Console server for ITM_ManagedSystem events The ITM_ManagedSystem event class is defined in the kib baroc file The kib baroc file is not included in the active rule base Copy the kib baroc file from the Tivoli Ent
235. etafiles 198 SNMP MANAGER Trap workspace 197 Software Support 247 SQL errors 230 SQL queries 228 SQL statement 229 SQL0443N 204 SQL0552N 208 statement parameter 177 Summarization and Pruning Agent 206 support assistant 247 SYSIBM CLI 805 204 Index 261 T tacmd commands 46 Take Action 183 Manage Start 200 Manage Stop 200 Take Action command 57 TakeAction 200 task IBM Tivoli Enterprise Console 211 TCP IP timeout setting 210 TEC_ITM_OM_Situation_Sync_Error 221 Technotes 2 253 subscribe to 2 viewing 2 telnet session 150 Tivoli Data Warehouse 174 uninstallation 105 Tivoli Data Warehouse issues troubleshooting 232 Tivoli Enterprise Monitoring Server not releasing memory 149 Tivoli Enterprise Monitoring Server on distributed problems 139 Tivoli Enterprise Monitoring Server on z OS problems 150 Tivoli Enterprise Monitoring Server private interface 116 Tivoli Enterprise Portal charts and graphs 195 Data is not returned to the 127 erratic sorting behavior 131 Tivoli Enterprise Portal help cannot search 130 problem 130 Tivoli Enterprise Portal performance 126 Tivoli Enterprise Portal Server failed connection 111 slow performance of the 134 turn on trace 42 Tivoli Enterprise Portal Server does not start 107 Tivoli Information Center 248 Tivoli Monitoring Services enable 128 Tivoli technical training 248 Tivoli user groups 249 tmcmd server stop start 148 tools Backspace Check utility 63 Build TEPS Database 63 BuildPresent
236. eters pointed to by the RKLVIN DD statement of the started task JCL Most often U0012 Abend failures can be resolved by backing out the last change made to the Tivoli Monitoring Services Engine startup parameters For U0200 Abends IBM Tivoli Monitoring Troubleshooting Guide the root cause of the protocol failures must be remedied These failures are covered RAS1 service initialization The Reliability Availability and Servicability RAS1 service refers to the RAS1 building block Basic Services component used for diagnostic tracing Nearly all diagnostic information for IBM Tivoli Monitoring is delivered via the RAS1 trace component This component is configured in member KBBENV of RKANPAR using the KBB_RAS1 environment variable Often customers redirect the initialization member via CT Engine INITLIST processing INITLIST processing is always echoed to the RKLVLOG with the KLVIN411 message The following shows an example of a typical KBBENV override to KDSENV KLVIN410 INITLIST MEMBER KDSINIT BEING PROCESSED KLVIN411 KLVINNAM KDSINNAM KLVIN411 KLVINTB KDSINTB KLVIN411 KLVINVLG KDSINVLG KLVIN411 KLVINNAF KDSINNAF KLVIN411 KLVINVPO KDSINVPO KLVIN411 KLVINSTG KDSINSTG KLVIN411 KLVINVAM KDSINVAM KLVIN411 KBBENV KDSENV In this instance configuration of KBB_RAS1 must display in member KDSENV of RKANPAR TCP IP service initialization TCP IP service is Transmission Control Protocol TCP IP provides end to end connectivit
237. example instead of just the hostname myhost1 being used the resulting hostname might be myhost1 acme north prod com Inclusion of the network domain name causes the agent name in the example above to expand to IBM Tivoli Monitoring Troubleshooting Guide SERVER1 myhost1l acme north prod com KXX This resulting name is 39 characters long It is truncated to 32 characters resulting in the name SERVER1 myhost1 acme north prod The agent name truncation is only a problem if there is more than one monitoring agent on the same system In this case the agent name truncation can result in collisions between agent products attempting to register using the same truncated name value When truncated agent names collide on the same system this can lead to Tivoli Enterprise Monitoring Server problems with corrupted EIB tables The agent name collision in the Tivoli Enterprise Monitoring Server might cause a registered name to be associated with the wrong product In general create names that are short but meaningful within your environment Use the following guidelines e Each name must be unique One name cannot match another monitoring agent name exactly e Each name must begin with an alpha character e Do not use blanks or special characters including and e Each name must be between 2 and 32 characters in length e Monitoring agent naming is case sensitive on all operating systems Create the names by completing the following st
238. f a network connection failure occurs and the UA system is assigned a different IP address after the reconnection UA will stop processing new input data because of socket bind failures Recycle the Universal Agent so that it uses the new IP address Unable to get attribute name for tablename column error occurs in the Tivoli Enterprise Monitoring Server log after creating a situation After creating a situation for Universal Data Provider to fire event the following error occurs in the Tivoli Enterprise Monitoring Server log file 4320916A 0049 F60 kfaottev c 1572 Translate ResultBuffer Unable to get attribute name for tablename column lt UAG524400 UA4 gt Ignored The Tivoli Enterprise Monitoring Server log messages are harmless for Universal Agent applications because of the time lag between the issuing of the messages and when the Universal Agent uploaded attribute files are stored at the Tivoli Enterprise Monitoring Server However if these messages occur for other agent applications they can indicate that the agent attribute files were not installed on the Tivoli Enterprise Monitoring Server SNMP data provider problems This section describes symptoms and tips for diagnosing SNMP data provider problems SNMP MANAGER Trap workspace is unavailable in many columns Every Independent Software Vendor ISV product that emits a trap to the SNMP DP must have a Type 2 and Type 3 record with a unique enterprise OID defined in
239. f this IBM Tivoli Monitoring address space was to the LLB in that address space In some instances an address space can bind to the LLB of another address space This can only occur in the same system image If the bind failure is fatal then another process on this system image has the well known port Bind failures due to insufficient authorization are fixed by granting RACF permission for the OMVS segment to the Omegamon Platform started task name Global Location Broker service initialization The Global Location Broker GLB differs from the Local Location Brokers in one important respect other than the name there is only one GLB for the domain or enterprise By definition there will be only one Local Location Broker which points to the Global Location Broker for the domain The RPC server LLB which points to the GLB and there will be only one of these in an enterprise is by definition the hub The important thing to remember from all this discussion of local and global brokers is this For a process to locate the Tivoli Enterprise Monitoring Server hub the process must query issue Ib__lookup RPC requests to the list of candidate Global Location Brokers in order as specified in the global site text file glb_site txt on distributed platforms and the KDCSSITE member of RKANPAR for OS 390 and z OS platforms Below are the product communication messages which enumerate the candidate GLB list GLB entry 1 GLB entry 2 etc GLB entry
240. face program of the Application ua Installer is not currently available Silent installation on UNIX hased systems of Tivoli Enterprise Monitoring Server returns an encryption key setting error The error Unexpected Signal 4 canti at PC 0xFEC3FDE4 occurs during installation Installing IBM Tivoli Monitoring on Red Hat 5 and see the following error KCI1235E terminating problem with starting Java Virtual Machine Installation of the IBM Tivoli Monitoring on the Linux 390 R2 6 64 bit operating system fails with the message LINUX MONITORING AGENT V610Rnnn unable to install agent where nnn is the release number SE x 82 82 82 82 83 83 83 83 83 84 84 84 84 85 85 85 85 86 86 86 86 86 87 87 87 The UNIX OS agent ux version 350 does not start on AIX 5 3 after upgrading other agents to version 6 1 0 88 Troubleshooting z OS paced install ti ns 88 IBM Tivoli Monitoring z OS initialization checklist 88 z OS based installations problems and resolutions 95 Uninstallation problems aid eerkatoands 100 Unable to uninstall multi instance agent from a managed system on windows 64bit 100 Prompted for msi file during uninstallation process started from Add Remove Programs on systems with v6 2 2 installed 100 Uninstallation is blocked by another process that is using the IBM Tivoli Mon
241. failed in all the managed systems This failure occurs when the following ExecuteAction options are specified e e stderr e o stdout e r returncode e l layout e p path Certain action commands must be handled by specialized agent command handler functions These include all action commands that are prefixed by a combination of the associated agent s two character product code pc and a colon for example UX These take actions must never be run with any of the listed ExecuteAction options otherwise the tacmd ExecuteAction command fails To understand if an action has a prefix run the tacmd viewaction command and view the action s Command details For example the UNIX AMS Stop Agent take action command has a UX prefix tacmd viewaction n AMS Stop Agent t ux Action Name AMS Stop Agent Action Type UNIX OS Description Kux KUX6065 Command UX AMS_Stop_Agent amp KCA_UX_Agent_Active_Runtime_Status PAS_Agent_Name amp KCA_UX_Agent_Active_Runtime_Status Process_ Name amp KCA_UX_Agent_Active_Runtime_Status Process_ IDKey KUX_1212721981813 158 IBM Tivoli Monitoring Troubleshooting Guide OS agent restarted unexpectedly on heavily loaded systems When Agent Management Services has a high workload the OS agent is restarted automatically At this point the OS agent is stopped and its workspace on the Tivoli Enterprise Portal is grayed out then the monitoring agent starts automati
242. fault RAS1 trace level is ERROR for all Summarization and Pruning Agent components and modules The following trace options are available for the IBM Tivoli Summarization and Pruning Agent KBB_RAS1 ERROR Trace general errors KBB_RAS1 ERROR Affects the content of the C based RASI tracing hostname_sy_HEXtimestamp nn 10g Chapter 4 Tools 45 46 KBB_RAS1 ERROR UNIT ksz ALL Trace agent startup Affects the content of the C based RAS1 tracing hostname_sy_HEXtimestamp nn 10g KBB_RAS1 ERROR COMP com tivoli twh ksy ALL Minimum level trace for summarization Affects the content of the Java based RAS1 tracing hostname_sy_rasljava_timestamp nn 109 KBB_RAS1 ERROR UNIT ksy1 ALL Medium level trace for summarization Affects the content of the Java based internal tracing hostname_sy_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksy2 ALL Connection level trace for summarization Affects the content of the Java based internal tracing hostname_sy_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksy3 ALL Statement level trace for summarization Affects the content of the Java based internal tracing hostname_sy_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksy4 ALL ResultSet level trace for summarization Affects the content of the Java based internal tracing hostname_sy_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksy5 ALL Column value level trace for summarization Affects the content of the Java based internal tracing hostname_s
243. for the Summarization and Pruning Agent On 32 bit Windows systems with 4 GB of RAM there is a maximum heap size limit of approximately 1 6 GB You might need to add the 3GB switch to the boot ini file Determine the max heap size in MBs by testing when setting KSZ_JAVA_ARGS Xms256M XmxSIZEM in the KSYENV file If the Xmx value is too large the agent fails with an error When associating situations they fire but cannot be viewed When a situation association is done at a specific node level on the navigator tree the situation fires when true and the event is then associated to the navigator item The situation is not associated with that navigator item if the navigator item or node is not part of its distribution The Summarization and Pruning agent fails when processing an index created in a previous release of the product If a database insert fails while running the Summarization and Pruning Agent and the message indicates the insert has failed because the maximum index size has been exceeded complete the following steps to correct this issue 1 Stop the Summarization and Pruning Agent 2 Drop the index causing the failure 3 Run the Schema Publication Tool in configured mode KSY_PRODUCT_SELECT configured 4 Open the tdw schema_index sql file and find the index that was deleted in step 1 5 Edit out all statements except the index you want to recreate D Run the create index statement 7 Start the Summar
244. future Technotes e mailed to you Alternatively you can watch demos of these procedures at the following Web site http www 306 ibm com software support sitetours html Viewing Technotes Perform the following actions to access Technotes for this product 1 Launch the IBM Software Support Web site http www ibm com software support 2 In the Products A Z field select the product name to open the product specific support site For this product select I gt IBM Tivoli Monitoring 3 In the Self help field click Technotes 4 Scroll through the Technotes or you can optionally type a search term to refine the displayed data For tips on refining your search click Search tips Creating an e mail subscription to Technotes You can subscribe to e mail notification about product tips and newly published fixes through My support a personalized portal that enables you to e Specify the products for which you want to receive notifications e Choose from flashes downloads and Technotes e Receive an e mail update in your inbox Perform the following actions to subscribe to My support e mails 1 Launch an IBM support Web site such as the following site http www ibm com support us 2 Click My support in the upper right corner of the page 3 If you have not yet registered click register in the upper right corner of the support page to create your user ID and password 4 Sign in to My support 5 On the My suppo
245. fying the uninstallation on page 105 to verify that you successfully removed the failed upgrade Remove the IBM Tivoli Monitoring bookmark from the Start menu a Click Start from the Windows desktop to display the Start menu items b Click Programs c Right click IBM Tivoli Monitoring to display the bookmark menu options d Click Delete to remove the IBM Tivoli Monitoring bookmark from the Windows desktop start menu One product is available in the Add and Remove Programs tool Use the following steps if an entry for either Candle OMEGAMON or IBM Tivoli Monitoring exists in the Add and Remove Programs tool 1 Attempt to uninstall both releases from the Add and Remove Programs entry 2 If this is successful proceed to Verifying the uninstallation on page 105 104 IBM Tivoli Monitoring Troubleshooting Guide 3 If this is not successful and the entry has been removed from the Add and Remove Prosranie tool sce 4 If the entry is still present in the Add and Remove Programs tool copy the KINWIINS VER file from the WINDOWS VERFILES KINWIINS VER directory on the installation CD to the lt itm_installdir gt Instal1 Ver directory where lt itm_installdir gt is the IBM Tivoli Monitoring installation directory Delete the KINWIINSMSTR VER file from this directory if it exists Note You might need to create the Install Ver subdirectory if it is does not exist 5 Attempt to uninstall the release
246. g Servers obtain their agent support from the hub Tivoli Enterprise Monitoring Server In order to apply agent support to a remote Tivoli Enterprise Monitoring Server the hub Tivoli Enterprise Monitoring Server must be running and reachable by the remote Tivoli Enterprise Monitoring Server During installation of a remote Tivoli Enterprise Monitoring Server on a Windows system the agent support is typically applied This fails if the hub Tivoli Enterprise Monitoring Server is unreachable During remote Tivoli Enterprise Monitoring Server installation on the Windows platform ensure the hub Tivoli Enterprise Monitoring Server is running Using a Deploy Group with addSystem or updateAgent commands When using a deploy group with addSystem or updateAgent commands remote deploy might fail to locate the existing Managed System Name for some hosts Message received KDYO012E The target target_hostname is incorrect or is offline The command did not complete because the value for the target is incorrect or the target is offline This message normally indicates that the OS agent is not online If the agent is in fact online cancel current operations to this node tacmd cleardeploystatus h hostname Then issue the operation directly by using the Managed System Name parameter instead of the deploy group tacmd updateAgent t product_code n managed_05 Tivoli Enterprise Monitoring Server requires restart if you issue itmcmd serve
247. gging Windows and UNIX systems z OS systems ad a ITMSuper Chapter 5 Installation and configuration troubleshooting Frequently asked questions Chapter 3 Solving common problems About the tools ITMSUPER Tools Log Analyzer tool pdcollect tool IBM Support Assistant A I am trying to find out what fwar i is supported 11 Le o vv v vo O Workspaces are missing or views are empty lt 12 Status of a monitoring agent is mismatched Between the portal client and tacmd command 15 Tivoli Enterprise Portal Server does not start or stops responding i a IZ Tivoli Enterprise Portal logon fails no oa eo ZO Tivoli Enterprise Portal does not respond 21 Historical data is missing or incorrect 22 A situation does not raise when expected 26 A reflex automation script does not run when it should US ae 29 High CPU usage on a distributed system 4050 Chapter 4 Tools 35 Trace logging do Sn i are Gt ee TO Installation log files eta gt goa 299 UNIX installer and configuration logs x od T weg Upgrading from Tivoli Distributed kaor log file Doa Ge Gro gona 40 Reading RAS1 logs ee g oe amp 2 amp AI Setting traces 42 Dynamically modify trade settings fot an IBM Tivoli Monitoring component DI Using the IBM Tivoli Monitoring Service Console 53 Starting the IBM Tivoli Monitoring Service Console 53 Blocking access io
248. graphs do not createNode command fails 185 work for numeric data 195 tacmd suggestbaseline minimum maximum id kumstrap and kumsnetd processes do hot start 196 average function values are ignored 185 The Universal Agent stops functioning if the tacmd suggestbaseline command receives an error 185 system IP address changes while it is running 197 When using the listSystems command the last two Unable to get attribute name for digits for the version appear as XX 185 tablename column error occurs in the Tivoli The command tacmd restartAgent fails if the agent Enterprise Monitoring Server log after creating a is already stopped 185 situation be goa a 197 Using the kinconfig command wid remotely SNMP data provider problems LE i so A a 197 starting stopping or recycling oe fails on SNMP MANAGER Trap workspace i is Windows 2000 systems 186 unavailable in many columns 197 You receive a message when using a facnid The TRAP ms does not display expected command related to agents 186 traps 198 You receive a message when trying to use the SNMP metafiles are imported but MIB data tacmd maintagent command s Le 187 collection is not working 198 Endpoint fails to connect to monitoring server Universal Agent emitted traps are nol received when running createnode from a monitoring server by third party SNMP Manager 199 in a diff
249. guide for operator responses e Look in the portal server or portal client logs for more information concerning the message See Cannot log in to the Tivoli Enterprise Portal on page 108 for more symptoms and corrective actions IBM Tivoli Monitoring Troubleshooting Guide Tivoli Enterprise Portal does not respond During normal operation of the portal client the client stops running or stops responding Preliminary diagnostics 1 Verify that the monitoring server is started 2 If you have selected a workspace that is retrieving large amounts of data wait for the data to be returned If the workspace returns empty see Workspaces are missing or views are empty on page 12 3 On Windows e Check the Windows Task Manager e Run the kincinfo exe command in the CANDLE_HOME Instal11TM directory kincinfo exe r to show running processes kincinfo exe i to show what is installed 4 On Linux or UNIX run the cinfo command in CANDLEHOME bin directory e cinfo r to show running processes e Jcinfo i to show what is installed 5 If your portal client stops responding while in an Oracle Agent workspace see Your problem might be related to a high CPU usage problem Similar problems might exist for other monitoring agents 6 Running the following ITMSUPER tools might also provide more information e Stressed Resources tool e Connectivity tool e Topology tool The following tables include
250. gured in the address space of the monitoring server then historical data can be collected only at the monitoring server If at the monitoring server see 2 If at the agent see 2 Check the RKPDLOG output of the monitoring server to verify that the persistent data store is configured correctly For example 2008 07 28 08 45 41 KPDIFIL Status of files assigned to group GENHIST 2008 07 28 08 45 41 2008 07 28 08 45 41 amp philev RGENHIS3 Status Active 2008 07 28 08 45 41 amp philev RGENHIS2 Status Offline 2008 07 28 08 45 41 amp philev RGENHIS1 Status Offline 2008 07 28 08 45 41 2008 07 28 08 45 41 KPDIFIL End of group GENHIST status See for various error codes to look for in the RKPDLOG 3 Check the RKPDLOG of the agent to verify that the persistent data store is configured correctly at the agent For example If KM5AGENT this agent runs in the monitoring server check the RKPDLOG of the monitoring server 2008 07 28 08 48 27 KPDIFIL Status of files assigned to group PLEXDATA 2008 07 28 08 48 27 2008 07 28 08 48 27 amp philev RKM5PLX3 Status Active 2008 07 28 08 48 27 amp philev RKM5PLX2 Status Empty 2008 07 28 08 48 27 amp philev RKM5PLX1 Status Partially Full 2008 07 28 08 48 27 If the MQ ag
251. has another IBM Tivoli Monitoring agent running the second agent is not successfully installed due to overlapping libraries and ports configuration To prevent this problem stop the running agent and use Tivoli Configuration Manager TCM or Tivoli Provisioning Manager TPM to install the second agent Problems with the SPB file If an Software Package Definition SPD file created with the tacmd exportBundles command is moved to a different machine to create an SPB the files copied by the tacmd exportBundles command need to be moved with the SPD file as well and the SOURCE_DIR in the default_variable section of the SPD file needs to be updated to reflect the new directory where the agent files are located Chapter 5 Installation and configuration troubleshooting 73 74 Installation was halted and now receive message about active install If for any reason the installation was halted either by invoking Ctrl C or by a power outage if you then run uninstallation you receive the following message An install may currently be running in data itmfp6_preUPGR from the following machine s Continue with this uninstallation 1 yes 2 no 2 is default Recovery from a hard kill of the installer is currently not a supported scenario since the current installer does not have built in rollback capability Executing a hard stop of the installer will leave some or all IBM Tivoli Monitoring functions including uninstall in an unpredic
252. he install _dir installITM DB2DataSource exe file 4 Read the error messages after running the program 5 If the error is security related or mentions incorrect user IDs or passwords log in to the windows server with the db2admin user account and run the db2datasource program If the password for db2admin changes the logon information for the services must also change otherwise the Tivoli Enterprise Portal Server database does not start because the DB2 processes cannot logon Use the following steps to resolve this problem 1 From a Windows desktop select Start gt Control Panel gt Administrative Tools gt Services 2 Scan the column on the right for the value db2admin 3 Do the following for each db2admin value a Open the Properties window b Select the Log On tab c Type the new password for the user Oracle problem with JDBC drivers prior to 11 1 0 7 You receive an error like the following on the Summarization and Pruning java log when using Oracle 509 t work1 java lang ArrayIndexOutOfBoundsException at oracle jdbc driver OraclePreparedStatement setupBindBuffers OraclePreparedStatement java 2673 at oracle jdbc driver OraclePreparedStatement executeBatch OraclePreparedStatement java 10689 at com tivoli twh ksy agg BatchManager executeBatch BatchManager java 381 at com tivoli twh ksy agg BatchManager commit BatchManager java 488 at com tivoli twh ksy agg BatchManager checkCommit BatchManager j
253. he MSN unique Value can also be set by the Tivoli Enterprise Monitoring Agent API CTIRA_SYSTEM_NAME Set alternate system name value for the agent entries in the Tivoli Enterprise Monitoring Server ManagedSystem Host Address column within the lt NM gt mysystem lt NM gt tags Used to build the Tivoli Enterprise Portal Server navigation tree Not applicable to subnode type records in the Tivoli Enterprise Monitoring Server ManagedSystem table CTIRA_USE_HBTIMER This variable should never be used It will degrade agent heartbeat performance by NOT creating a separate heartbeat data thread DATACHUNKSIZE Specifies the size of the data chunk that will be passed to RPC for transfer DEPLOYQUEUESIZE Specifies the maximum number of requests that the request queue on the agent deployment controller DEPLOYTHREADPOOLSIZE Specifies the number of threads that are available to the deployment controller DEPOTHOME Specifies the root directory of the agent depot on the file system IRA_DUMP_DATA Used by Tivoli Enterprise Monitoring Agent and Tivoli Enterprise Monitoring Server for debugging Set to Y to do a hex dump of certain detailed data row contents into the RAS1 log Default is N Can produce voluminous RAS1 message output if enabled IRA_EIF_ ENABLE _LOG Enables logging of EIF events sent for private situations directly from the Tivoli Enterprise Monitoring Agent to the agent s LGO operational log if
254. he agent if historical collection is configured to collect data Anytime you configure an IBM Tivoli Monitoring historical collection for any agent UA or otherwise the name of the history situation is always called UADVISOR_xxxxx If you see these UADVISOR_xxxxx entries in the list of defined situations even though they were never explicitly defined these history situations were automatically defined by an IBM Tivoli Monitoring component 128 IBM Tivoli Monitoring Troubleshooting Guide At the bottom of each view you see a historical workspace KFWITM217E error At the bottom of each view you see the following historical workspace KFWITM217E error Request failed during execution and a red icon Ensure that you configure all groups that supply data to the view In the Historical Configuration view ensure that data collection is started for all groups that supply data to the view Views containing multi row attributes show this message if no row data are collected Installation of situation data fails due to I O on VSAM data sets After installation of application support product provided situations do not appear in the Tivoli Enterprise Portal Situation editor or do not auto start This problem occurs only with a z OS hub monitoring server Explanation The definitions of product provided situations are installed on the hub Tivoli Enterprise Monitoring Server when application support for a product is installed If the VSAM data
255. he following steps 1 Click Start gt Programs gt IBM Tivoli Monitoring gt Manage Tivoli Monitoring Services 2 In the Manage Tivoli Monitoring Services window right click Tivoli Enterprise Portal gt Browser or Tivoli Enterprise Portal gt Desktop_InstanceName and click Reconfigure 3 In the client configuration window that opens double click the cnp browser installdir parameter 4 In the edit parameter window that opens enter the path to where the browser view files should be installed on the client computer If this is the browser client that you are configuring use a path that is available at any computer from which users log on such as c temp and choose a path that does not contain any non latin1 chars 5 Select the In Use check box and click OK 6 Click OK to save your changes The Tivoli Enterprise Portal desktop does not work when exporting DISPLAY The Tivoli Enterprise Portal desktop does not work when exporting the DISPLAY from a Linux system to a Windows system running cygwin The Tivoli Enterprise Portal desktop client log on the Linux system install_dir logs kcjras1 1og contains the following error EXCEPTION Attempting to load home workspace java lang IllegalArgumentException Width 0 and height 0 Edit the cnp sh startup file by doing one of the following e Add the java system property Dawt toolkit sun awt motif MToolkit by doing the following 1 Locate the install _di
256. he following tables include possible causes and corresponding solutions for the problems described in this section Table 16 Correcting situation specific problems e The situation is distributed to an incorrect managed system e The situation is not associated with a node in the Navigator e The situation is formulated incorrectly Diagnostic and corrective actions 1 Verify that the situation was started by checking one of the following log files for text strings based on the specific situation Agent operations log For example 1061110125731000KRAIRA000 Starting FireOnWednesday lt 7340776 3145895 gt for KPX LOCALTIME Monitoring server log For example 11 13 06 15 07 21 K041046 Monitoring for enterprise situation FireOnWednesday started Did the situation start Yes See step amp on page 2 No See step 26 IBM Tivoli Monitoring Troubleshooting Guide Table 16 Correcting situation specific problems continued 2 Is the situation distributed to the agent and is the agent online Look for a text string similar to this one in the monitoring server log K041047 Situation CheckIfSituationCreated distribution Primary KATE NT added Yes See step 4 on page 28 No Use UNIT kpxreqds all to trace the distribution at the monitoring server for a situation The following example is taken from a monitoring server log 4558D8CC 0033 1114 kpxreqds cpp 621 DetermineTargets Active RAS1
257. he name is already used and the user must select a new name If the request fails then the name is not already used The failure to find the situation name is reflected in the log as the CORBA exception The CORBA user exception indicates that the name is unique Chapter 8 Tivoli Enterprise Portal Server troubleshooting 137 138 IBM Tivoli Monitoring Troubleshooting Guide Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting This section describes problems that might occur with the Tivoli Enterprise Monitoring Server and provides resolutions to recover from those problems It includes the following sections problems on distributed g Tivoli Enterprise Monitoring systems on page 139 g Tivoli Enterprise Monitoring systems on page 150 Server problems on z OS Troubleshooting Tivoli Enterprise Monitoring Server problems on distributed systems The problems described in this section might occur on distributed systems For information about configuring the Tivoli Enterprise Monitoring Server refer to the IBM Tivoli Monitoring Installation and Setup Guide Exposure of passwords in the clear Currently all versions of IBM Tivoli Monitoring incorporating the IBM Tivoli Directory Server Client at the monitoring server are exposed to an unaudited security risk of exposure of passwords in the clear To avoid displaying passwords in the clear when troubleshooting LDAP problems use the foll
258. hen a component or application fails to operate as designed IBM Software Support personnel use the captured trace information to determine the source of an error or unexpected condition See the Chapter 4 Tools on page 35 for more information about tracing Problem classification The first task in troubleshooting is to determine the origin of the problem or which component or function is experiencing a problem To assist you in determining the origin of the problem collect documentation at the time of the error You might experience problems with IBM Tivoli Monitoring in the following areas e Installation e Upgrading e Configuration e Connectivity e Tivoli Enterprise Portal e Tivoli Enterprise Portal Server e Tivoli Enterprise Monitoring Server Copyright IBM Corp 2005 2010 1 e Tivoli Enterprise Monitoring Agent deployment e Databases e Tivoli Data Warehouse e Universal Agent IBM Tivoli Enterprise Console Finding release notes 2 You can find release note information online by viewing IBM Technotes Technotes replace the Release Notes manual for this product Technotes are short documents that cover a single topic You can search the Technotes collection for common problems and solutions as well as known limitations and workarounds Technotes are continuously updated to provide current product information The following two procedures describe how to view Technotes and subscribe to have
259. herit Framework FFTC mechanisms such as wtrace and odstat for transaction and method stack traces See the Tivoli management Framework documentation for more information about the commands OS Agent tracing OS Agent tracing is enabled at a minimum level by default Agent tracing levels can be adjusted with agent specific settings Logs are stored in install_dir instal 1ITM on Windows agents or install_dir logs on UNIX based systems agents These logs follow the RASI log format Setting the trace option for the IBM Tivoli Monitoring event forwarding If your monitoring environment is configured for the IBM Tivoli Monitoring event forwarding you can forward situation events to the IBM Tivoli Enterprise Console and view events on the event server through the Tivoli Enterprise Portal If you want to forward situation events to and view updates from IBM Tivoli Enterprise Console in the Tivoli Enterprise Portal you can set the trace for the event forwarder on the Tivoli Enterprise Monitoring Server Use the IBM Tivoli Monitoring event forwarding trace facility to diagnose problems with the IBM Tivoli Monitoring event forwarding The IBM Tivoli Monitoring event forwarding trace facility uses RAS1 tracing The IBM Tivoli Monitoring event forwarding is set during installation The acceptable values include Chapter 4 Tools 47 48 STATE e DETAIL e ALL The default trace value is STATE If you change the trace level you mus
260. his error occurs KDY3010E The SNMP command instal1SSM timed out with an SNMP return code of 0 The SNMP command timed out because there was a network error or the agent was stopped or the specified SNMP community user does not have write and create privileges This happens because the SNMPPORT is not available to use The default SNMPPORT is 161 You should try to specify a different SNMPPORT to use when deploying the agent Here is an example command tacmd createNode h smb target_endpoint_hostname p snmpport 4567 server RTEM hostname u user_id w password t ssm Chapter 10 Monitoring agent troubleshooting 159 Attributes differ between the situation action commands and what is displayed in the Tivoli Enterprise Portal The raw data delivered by an monitoring agent is a string or number The Tivoli Enterprise Portal has format information to control the display When an attribute is used in a system command in the Situation Action tab the raw data from the agent is substituted For example if a situation had an action command to send an email it could look like this in the situation editor some_command Warning too many processes amp System Load Average_1 Min options If the average was actually 6 99 as displayed in the Portal client the command would be executed as some_command Warning too many processes 699 options amp xx yy is the best way to specify an attribute because there is no uncertainty about what
261. his issue please consult the VMWare website www vmware com How To Tell If You Have This Problem A simple way for determining whether or not your Linux guest has a clock problem is to benchmark it against a real world clock Here is an example of a procedure that you can use 1 From a Linux shell prompt type date to get the current system date and time While you are pressing Enter look at a real clock wall clock watch etc to get the real world time in minutes and seconds Record the time from both your Linux guest and the real clock Example Real Clock 10 30 00 Linux Clock 10 20 35 2 After 10 real time minutes have expired type the date command again you should type the date command ahead of time so you only have to press Enter when 10 minutes have elapsed Record the new times from both your Linux guest and real clock Example Real Clock 10 40 00 Linux Clock 10 26 35 3 Compute the elapsed time for both your Linux guest and real clock If the elapsed times are not the same your Linux guest has a clock problem Since we waited exactly 10 minutes using the real clock we would expect that the elapsed time for the Linux clock would also be 10 minutes Using the above figures we can see that the elapsed time for the Linux guest is 6 minutes 10 26 35 10 20 35 Since this is less than the real world time this means that the Linux guest clock is running slow This causes the IBM Tivoli Monit
262. his issue only occurs in situations that were defined in fix packs prior to IBM Tivoli Monitoring Fix Pack 5 You set the Tivoli Enterprise Portal severity but there was a SEV parm in the SITINFO that took precedence over the assign tec severity by situation name suffix rule in the code Complete one of the following options to correct this problem e Code an entry in the tecserver txt file to explicitly assign the correct Tivoli Enterprise Console severity for the situation Manually update the situation definition to either remove the SEV parm from the SITINFO column or remove the SEV parm Situations are not forwarded from Tivoli Enterprise Monitoring Server to the Tivoli Enterprise Console server Table 45 on page 211 provides troubleshooting information for when Tivoli Enterprise Monitoring Server does not forward situations to the Tivoli Enterprise Console server Table 45 Troubleshooting when situations are not forwarded Problem Resolution The IBM Tivoli Enterprise See Configuring IBM Tivoli Enterprise Console integration in the IBM Tivoli Console Event Integration Monitoring Administrator s Guide Facility is not enabled or configured correctly Chapter 14 Event synchronization troubleshooting 211 Table 45 Troubleshooting when situations are not forwarded continued Problem Resolution The correct values are not Edit the following parameters in the om_tec config to specify the corr
263. hose Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk Copyright IBM Corp 2005 2010 255 256 IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Licensees of this program who wish to have information about it for the purpose of enabling i the exchange of information between independently created programs and other programs including this one and ii the mutual use of the information which has been exchanged should contact IBM Corporation 2Z4A 101 11400 Burnet Road Austin TX 78758 U S A Such information may be available subject to appropriate terms and conditions including in some cases payment of a fee The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement IBM International Program License Agreement or any equivalent agreement between us Any performance data contained herein was determined in a controlled environment Therefore the results obtained in other operating environments may vary significantly Some measurements may have been made on development level systems and there is no guarantee that these measurements will be the same on generally available systems Furthermore some measurement may have been estimated through extrapolation Actual results may vary Users of this
264. i Instance Agent or IBM Tivoli Monitoring V6 2 Agent Support is installed any one of the following corrective actions can be taken e Run kinconfig exe G on the managed system e OR e Reconfigure any of the IBM Tivoli Monitoring V6 2 1 or later monitoring agents on the managed system by running kinconfig exe rKproductcode e OR Install another IBM Tivoli Monitoring V6 2 1 monitoring agent or later IBM Tivoli Monitoring Troubleshooting Guide Incorrect behavior after an uninstallation and re installation You might experience incorrect behavior if you uninstall then reinstall the product without rebooting For example you might experience the following problems e Inability to create trace logs e Agents do not start e Agents data is corrupt Reboot the system to resolve the problems Where Remote Deployment of agents is not supported Remote Deployment is not supported for OMEGAMON agents It is also not supported in environments with a z OS Tivoli Enterprise Monitoring Server Remote Deployment is not supported when the Tivoli Enterprise Monitoring Server Tivoli Enterprise Portal Server or the Tivoli Enterprise Portal are on the same machine as the agent Application Support Installer hangs The Application Support Installer ASI gets to the screen indicating Please select which applications you would like to add support for but hangs there After selecting the Next button the installation hangs there and does
265. i Monitoring Troubleshooting Guide Message SQL Statement does not return a ResultSet object displayed In this scenario the report generation process fails to produce a report and this error message will be displayed SQL Statement does not return a ResultSet object This message is followed by some SQL codes and the name of the table or view that is absent This message is displayed when the tables and views from which data is to be extracted do not exist in the database To view the SQL query that generated this problem and determine what historical data is not being collected review the log files See Locations of log files on page 227 for the locations of log files Your report fails to generate with unexpected error messages displayed In this scenario you have defined report parameters for an OMEGAMON XE or NetView for z OS report and clicked Run After a long wait these messages are displayed Processing has ended because of an unexpected error See the Tivoli Common Reporting log files for more information These error messages are generated by Tivoli Common Reporting and might indicate that you need to increase the default heap size for the JVM on the java command to start the Tivoli Common Reporting server To confirm that this is the problem do the following 1 Open the System0ut 1log file found in this location On Windows Systems lt tcr_install_dir gt tcr eWas61 profiles tcrProfile logs tcrServ
266. ibution list A changed situation name does not show up New agents do not display in the Tivoli Enterprise Portal navigator An agent displays unavailable i in the Tivoli Enterprise Portal CTIRA_HOSTNAME has no y effect on n log file names The Summ tization and Praning Agent aad dhe Warehouse Proxy e do not work with DB2 9 1 Fix Pack 2 An error of can bind a LONG Wale only for insert appears Errors in either the Warehouse Proxy Agent o or Summarization and Pruning Agent logs Receive a message saying that the statement parameter can only be a single select or a single stored procedure Custom defined workspace views aie Aot handle symbol substitution as expected Unresolved variables in custom queries A message appears after accepting the license Adding agent help files requires a restart of the Eclipse Help Server and the Tivoli Enterprise Portal Server Unable to create historical collection directory for ud db2inst1 7 Receive a large amount f data baek from the warehouse for a baseline command Chapter 11 Command troubleshooting a Commands with embedded single quotation iu fail tacmd exportnavigator O hoe behaving correctly TACMD xxxxAction commands fail on Japanese systems Overrides set against an agent canine be deleted from the command line tacmd listsit m UX Managed System gives no result Receive a busy monitoring server message when using the getfile putfile or executecommand comm
267. ich the issue can appear e DB2 Agent e Oracle Agent e Microsoft SQL Server Agent e Sybase Server Agent e Microsoft Exchange Server Agent e Lotus Domino Agent e VMWare VI Agent e Microsoft BizTalk Server Agent e Microsoft Cluster Server Agent e Microsoft Exchange Server Agent e Microsoft SQL Server Agent e mySAP Agent e Siebel Agent Errors in the configuration xml file If a tag error occurs in an XML definition file it is displayed in the LGO log for example misspelling the tag in the pc_eventdest xml file In the following example Srv should be Server lt EventDest gt lt Destination id 0 type T default y gt lt Srv location xx xx xx xx port 5529 gt lt Destination gt lt EventDest gt You will see the following in the LGO log file 1090918140248857KRAX002I XML begin tag Srv unrecognized XML Parser 1090918140323448KRAX003I XML end tag Srv unrecognized XML Parser If you have a problem with the value in the xml file you need to check the agent log file For example if you misspell the value in the pc_eventmap map file as follows lt itmEventMapping agent xmlns itmEventMapping http www ibm com tivoli itm agentEventMapping xmlns xsi http www w3 org 2001 XMLSchema instance xsi schemaLocation http www ibm com tivoli itm agentEventMapping agentEventMap xsd gt lt idUD id gt lt version gt 6 2 0 lt version gt lt event_mapping gt lt situation name UDB_Buff_Used_Pct_War
268. iew 132 Cannot load a ws pres CSS file i in order to select a language other than English 132 Chapter 8 Tivoli Enterprise Portal Server troubleshooting 133 tacmd exportWorkspaces or importWorkspaces receives an out of memory error 133 The portal server and Warehouse Proxy Agent fail to connect to the database on a 64 bit Windows system amp 133 Failed to login a as sysadmin with portal server LDAP enabled sca 133 On AIX systems newly created users with auto expire passwords cause installation failures 133 Linux portal server unable to FTP catalog attribute files 134 Upgrading the Tivoli Enterprise Portal Reis lakes a long time 134 Running the Tivoli Management Services Discovery Library Adapter results in a book that does not contain the fully qualified hostname 134 Tivoli Enterprise Portal Server performance is slow 134 Cannot create a Tivoli cee Portal Server database 134 You receive a KFW error hen a query is cant to more than 200 managed systems 135 vi IBM Tivoli Monitoring Troubleshooting Guide Non hub situations are not associated at the Tivoli Enterprise Portal Server level 136 Starting and stopping the Eclipse Help Server 136 Non root stopping or starting agents causes problems 136 Root password is not accepted during non roct Tivoli Enterprise Portal Server configuration
269. if the file exists under the user provided path Crash on Linux remote Tivoli Enterprise Monitoring Server during hub failover to Hot Standby The monitoring server can use a large number of file descriptors especially in a large environment On UNIX and Linux systems the maximum number of file descriptors available to a process is controlled by user limit parameters To display the user limits run the following command ulimit a The nofiles parameter is the number of file descriptors available to a process For the monitoring server process kdsmain the nofiles parameter should be set larger than the maximum number of agents that will be connecting to the monitoring server If the monitoring server is unable to get file descriptors when needed unexpected behavior can occur including program failures Consider increasing the value to 1000 file descriptors or more There are other user limit parameters that control how much data stack and memory are available to a process For large environments consider increasing these memory related user limit parameters for the monitoring server kdsmain process Configuring the user limit parameters usually requires root access and involves changing system startup files which are operating system specific Consult the operating system manuals for information on how to configure the user limit parameters HUB Tivoli Enterprise Monitoring Server quiesce prevents the display of the data co
270. ify that only one file source SOURCE FILE statement is specified for each attribute group NAME statement or that you used ManagedSystemName to distinguish the sources For SOCK Data Providers verify that you have the correct socket source host name SOURCE SOCK specified for the application Examine the UAGENT DPLOG report in Tivoli Enterprise Portal It might include messages that provide clear indications concerning data provider operation and the data source disposition Copyright IBM Corp 2005 2010 193 Universal Agent does not start The Universal Agent did not allocate its DCH port 1919 for one of the following reasons Table 41 Universal Agent does not start Problem Another Universal Agent process is running on the same system and the other Universal Agent was not shut down There is a non Universal Agent process running on the same system and the Universal Agent was not shut down Explanation The following are the last messages that display in the Universal Agent RASI log kumOsock c 110 KUMO_OpenLocalSocket bind failed for local address UDP socket 512 port 1919 error 10048 kum0sock c 110 KUMO_OpenLocalSocket bind failed for local address TCP socket 512 port 1919 error 10048 kumdsock cpp 964 ipcSock allocateDCHport Error Could not open TCP UDP sockets bound to universal agent DCH port 1919 kumdsock cpp 965 ipcSock allocateDCHport Determine if another copy of Unive
271. igration of the table is aborted KHD_MAX_ROWS_PER_ TRANSACTIONS Specifies how many rows are committed at each insert of data KHD_CNP_SERVER_HOST Specifies the name of the CNP server KHD_CNP_SERVER_PORT Specifies the port used by the CNP server KHD_SOURCE_DATABASE_ SCHEMA Specifies the owner or schema of the Candle tables in the source database KHD_SOURCE_JDBC_DRIVER Specifies the JDBC driver used to connect to the source database KHD_SOURCE_URL Specifies the URL to connect to the source database KHD_SOURCE_DATABASE_USER Specifies the user to connect to the source database KHD_SOURCE_DATABASE_PASSWORD Specifies the password to connect to the source database KHD_TARGET_JDBC_DRIVER Specifies the JDBC driver used to connect to the target database KHD_TARGET _URL Specifies the URL to connect to the target database KHD_TARGET _DATABASE SCHEMA Specifies the owner or schema of the tables in the target database KHD_TARGET _DATABASE USER Specifies the user to connect to the target database KHD_TARGET _DATABASE PASSWORD Specifies the password to connect to the target database The following Warehouse Proxy agent parameters can be changed in the KHDENV file on Windows systems and the hd ini file in non Windows systems to customize tracing for the Warehouse Proxy agent Table 57 Warehouse proxy environment variables Variable Purpose CTIRA_NCSLISTEN The number of RPC threads KHD_BATCH_
272. ing command cd db2inst1 sqllib adm db2stop 2 Issue the following configuration changes export EXTSHM ON db2set DB2ENVLIST EXTSHM db2set all 134 IBM Tivoli Monitoring Troubleshooting Guide 3 Using your preferred editor add the following lines to the db2inst1 sqllib db2profi le file EXTSHM ON export EXTSHM 4 Restart the DB2 server using the following command cd db2inst1 sqllib adm db2start 5 Restart the Tivoli Enterprise Portal Server using the following command cd opt IBM ITM bin itmemd agent start cq You receive a KFW error when a query is sent to more than 200 managed systems You receive the following error when a query is sent to more than 200 managed systems KFWITM217E Request error Request to xxx nodes exceeds the limit of 200 Please specify a smaller distribution or increase the maximum There is a default limit of 200 nodes for any single query for a workspace view If the following conditions exist in the query for a workspace view you must increase the KFW_REPORT_NODE_LIMIT environment variable for the Tivoli Enterprise Portal server environment variable as described below e The query is assigned to a managed system list that contains more than 200 managed systems OR e More than 200 managed systems are explicitly assigned to a query in any workspace view Under these conditions you must increase the following Tivoli Enterprise Portal server environment variable KFW_REPO
273. ing monitoring agents and their installation packages and for adding functions to existing agents Documentation for the base agents If you purchased IBM Tivoli Monitoring as a product you received a set of base monitoring agents as part of the product If you purchased a monitoring agent product for example an OMEGAMON XE product that includes the commonly shared components of Tivoli Management Services you did not receive the base agents The following publications provide information about using the base agents e Operating system agents Windows OS Agent User s Guide SC32 9445 UNIX OS Agent User s Guide SC32 9446 Linux OS Agent User s Guide SC32 9447 i5 OS Agent User s Guide SC32 9448 UNIX Log Agent User s Guide SC32 9471 IBM Tivoli Monitoring Troubleshooting Guide e Agentless operating system monitors Agentless Monitoring for Windows Operating Systems User s Guide SC23 9765 Agentless Monitoring for AIX Operating Systems User s Guide SC23 9761 Agentless Monitoring for HP UX Operating Systems User s Guide SC23 9763 Agentless Monitoring for Solaris Operating Systems User s Guide SC23 9764 Agentless Monitoring for Linux Operating Systems User s Guide SC23 9762 e Warehouse agents Warehouse Summarization and Pruning Agent User s Guide SC23 9767 Warehouse Proxy Agent User s Guide SC23 9766 e System P agents AIX Premium Agent User s Guide SA23 2237 CEC Base Ag
274. int to the same Tivoli Enterprise Monitoring Server as it was before upgrading the agent During upgrade of agents from OMEGAMON 350 to IBM Tivoli Monitoring 6 1 the OMEGAMON 350 agent specific Tivoli Enterprise Monitoring Server configuration data does not persist to the upgraded IBM Tivoli Monitoring 6 1 agent The upgraded agents are configured to use the agent default settings Do the following to change the default settings configured for OMEGAMON 350 1 Right click an agent in the Manage Tivoli Enterprise Monitoring 2 Select Set defaults for all agents Installation of OS agent on a Microsoft Windows Server 2003 fails with this error Unable to establish BSS1 environment can t continue This error is caused by the deletion of the gskit directory whether intentionally or by accident without clearing the registry information If gskit was previously IBM Tivoli Monitoring Troubleshooting Guide installed by another product and has a dependency on it for example DB2 9 1 then let that product reinstall it or if there are no other products that depend on the version of that gskit then you can clear the GSK7 entry in the registry that can be found under My Computer HKEY_LOCAL_MACHINE SOFTWARE IBM GSK7 Then rerun the IBM Tivoli Monitoring installation to allow the gskit to be reinstalled Note Create a backup of the registry before editing it After upgrading to IBM Tivoli Monitoring V6 1 the link to the Queue Statistics
275. inux S390 R2 6 64 bit operating system fails with the message LINUX MONITORING AGENT V610Rnnn unable to install agent where nnn is the release number Perform the following steps to resolve this problem before running the installation again 1 Run the following command before running any installation or configuration command for the agent export JAVA_COMPILER NONE 2 Install the s390x rpm RPM Red Hat Package Manager files in addition to the s90 rpm files located in the CD ISO images for Red Hat As 4 0 s390x e compat libstdc 295 2 s390x rpm e compat libstdc 33 3 s390x rpm It requires the two s390x rpm files in addition to the s390 rpm files You can obtain the required RPM files from the CD for Red Hat As 4 0 s390x Chapter 5 Installation and configuration troubleshooting 87 The UNIX OS agent ux version 350 does not start on AIX 5 3 after upgrading other agents to version 6 1 0 Do one of the following to enable UNIX OS agent ux version 350 e Upgrade the UNIX OS agent to version 6 1 0 OR e Add the following line to the uxaix513 ver file in the CANDLEHOME registry runArch aix533 Note Create a backup of the registry before editing it Troubleshooting z OS based installations 88 This section describes some problems you might experience with z OS based installations including problems you can resolve with the Installation and Configuration Assistance Tool ICAT It includes the following s
276. ion and executecommand fail if they contain single quotation marks Also commands that contain embedded single quotation marks and a right curly brace also fail There is no workaround at this time tacmd exportnavigator o not behaving correctly This option should export only the custom navigator view will be exported Workspaces queries and situation associations referenced within the custom navigator view should not be exported However if you reimport the navigator view using the xml generated using the o option the customizations showed up on the custom navigator view in the Tivoli Enterprise Portal This behavior occurs because this option does not delete the customizations from the portal server database TACMD xxxxAction commands fail on Japanese systems The following commands fail on Japanese systems e tacmd createAction e tacmd viewAction e tacmd editAction e tacmd executeAction Run the command with LANG C or LANG Ja_JP UTF 8 Edit the kuixxxxactionVld xml removing the dash For example change the following lt ActionName argl n arg2 name Type String ValidationRegExp A Za z0 9 _ Required Y gt to lt ActionName argl n arg2 name Type String ValidationRegExp A Za z0 9 _ Required Y gt Overrides set against an agent cannot be deleted from the command line It appears as though the override does not exist when deleting however it allows you to list
277. ion help 110 IBM Tivoli Monitoring Troubleshooting Guide Cannot connect to Tivoli Enterprise Portal Server Before performing any of the following troubleshooting steps verify that the connection problems are not the result of firewall settings The following table provides resolutions for problems logging in to the Tivoli Enterprise Portal Server Table 37 Cannot connect to Tivoli Enterprise Portal Server Problem Resolution KFWITMOO1W Unable to 1 connect to Tivoli Enterprise Portal Server Check the kfwlras 1og for details if an attempt to log in fails with message KFWITM001W The kfwlras 10g can list any of the following messages that indicate a reason for the failure e SQL1224N A database agent could not be started to service a request or was terminated as a result of a database system shutdown or a force command e SQL1226N The maximum number of client connections are already KFWITM215E Unable to started SQLSTATE 57030 process logon request re Both messages SQL1224N and SQL1226N occur when the Tivoli Enterprise Portal Server attempts to validate the user ID entered in the browser 2 Restart the database 3 Attempt the login again A remote Tivoli Enterprise A remote Tivoli Enterprise Portal client login dialog does not connect to a Tivoli Portal client does not connect to Enterprise Portal Server hosted on a UNIX based system but the following are a UNIX based system Tivoli true Enterpr
278. ir install_dir Specifies the directory where Tivoli Enterprise Specifies the directory Portal Server was installed where monitoring agent hostname was installed Specifies the name of the system hosting the PC Specifies the product product codes for example um for pc Specifies the product code cq for the Tivoli Universal Agent or nt for Enterprise Portal Server Windows timestamp HEXtimestamp A decimal representation of the time at which the A hexadecimal process was started representation of the time at which the process was started nn Represents the circular sequence in which logs are rotated Ranges from 1 5 by default though the first is always retained since it includes configuration parameters IBM Tivoli install_dir logs hostname_PC_ Not supported Warehouse Proxy agent IBM Tivoli Summarization and Pruning agent timestamp log where PC Specifies the product code hd is the product code for the IBM Tivoli Warehouse Proxy agent The Summarization and Pruning Agent uses C based RASI tracing Java based RAS1 tracing and Java based internal tracing By default Summarization and Pruning Agent trace data is written to a file in the logs subdirectory install_dir logs hostname_PC_ HEXtimestamp nn log install _dir logs hos tname_PC_ rasljava_HEXtimestamp nn log install _dir logs hostname_PC_ java_HEXtimestamp nn log install_dir logs hostname_PC_ HEXtimestamp nn log install_dir logs h
279. is running and verify that the Tivoli Enterprise Monitoring Server of the event indicated by sv TEMS server port in the cache file or the cms_hostname attribute of the TEC event matches any of the Tivoli Enterprise Monitoring Servers defined in the situser conf configuration file Situation Events have an UNKNOWN severity in the Tivoli Enterprise Console If a situation was not mapped to a specific severity in the tecserver txt file in the TECLIB directory then IBM Tivoli Enterprise Console event synchronization forwards these situation events to Tivoli Enterprise Console with an UNKNOWN severity You can determine the event severity by checking the following e The situation SITINFO field e The severity keyword if the situation name was defined in tecserver txt See the section Defining event severities from forwarded IBM Tivoli Monitoring situations in the IBM Tivoli Monitoring Administrator s Guide Chapter 14 Event synchronization troubleshooting 215 Forwarded IBM Tivoli Enterprise Console events display with the PARSING_FAILED status in the IBM Tivoli Enterprise Console reception log The following table lists possible causes and resolutions for the PARSING_FAILED status Table 46 Forwarded events display with the PARSING_FAILED status Problem Resolution The IBM Tivoli Enterprise Console event synchronization was not installed or was not installed correctly Reinstall the IBM Tivoli Enterpris
280. is usually in the presence of another process such as the IBM Tivoli Monitoring Universal Agent or the Monitoring Agent for Unix OS When the problem occurs both instances of the Hub Tivoli Enterprise Monitoring Server are listed on the IBM Tivoli Monitoring Service Index web page that is produced on port 1920 The tacmd and SOAP interfaces use the service index as part of their communications and the extra Hub Tivoli Enterprise Monitoring Server entry disrupts them from communicating at all When the problem occurs make sure the Hub Tivoli Enterprise Monitoring Server started under the non root login is not running Then recycle whichever process shows up first on the Service Index page The system crashes when attempting a bulk import or export command You might be attempting to import from a 0 byte XML file Ensure that the XML file has content Also this can also occur if the file contents are corrupted if some expected XML elements are missing The Tivoli Enterprise Monitoring Server fails to start but then does after a reboot When your Tivoli Enterprise Monitoring Server does not start up properly and you see the following messages in the Tivoli Enterprise Monitoring Server logs you need to check if anything is using the location server broker default is port 1918 Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 145 146 4703AF9A 002B 4 kdcsuse c 99 KDCS UseFamily status 1c010005 cant bind socket
281. ise Monitoring Server The movefrom m2i bat is only used to migrate the contents of Microsoft SQL Server to Universal Database Database contents are incorrect after installation You can run the movefrom m2i bat located in the install _dir cnps directory to recover the contents of the database The movefrom m2i bat utility creates a flat file from the SQL server database contents and imports into the Universal Database The movefrom m2i bat utility runs during the installation of the Tivoli Enterprise Portal when the option to migrate from Microsoft SQL Server to the Universal Database is selected but can also be used after installation or routine Tivoli Enterprise Portal usage This migration utility can fail if the Tivoli Enterprise Portal userid and password do not have the correct authority to connect to the Tivoli Enterprise Monitoring Server Universal Database The movefrom m2i bat requires that the Microsoft SQL server database is on the same Windows platform as the Tivoli Enterprise Monitoring Server server and the new Universal Database installation Use this utility only after migration problems and before customizing Tivoli Enterprise Monitoring Server The movefrom m2i bat is only used to migrate the contents of Microsoft SQL Server to Universal Database The error SQL0443N with SYSIBM CLI 805 occurs after upgrading to DB2 UDB Version 8 1 Fix Pack 10 204 An error occurs with the SQLTables if the database was created before a
282. ise Portal Server with the error message KFWITMOO1W Unable to connect to Tivoli Enterprise Portal Server e A local Tivoli Enterprise Portal client connects to the Tivoli Enterprise Portal Server You can ping the Tivoli Enterprise Portal Server computer from the remote computer e A Web browser can remotely connect to http host_name 1920 to get to the service links assuming the default service port of 1920 was used during installation The hostname might not resolve to the correct IP address on the local Tivoli Enterprise Portal Server host To confirm that the hostname resolves to the correct IP address run the following command from the host on which the Tivoli Enterprise Portal Server was installed ping hostname OR ping s hostname The ping command displays the IP address of the computer it pings Ensure that the IP address is the same as the one to which the remote Tivoli Enterprise Portal client is attempting to connect For example if your Tivoli Enterprise Portal Server is located on a host with the hostname tepshost and the host s etc hosts file includes an entry similar to the following 127 0 0 1 localhost localdomain localhost tepshost Running ping tepshost displays an IP address of 127 0 0 1 which is the address of the local loopback network interface and the reason a remote computer cannot connect to it In this case you must update the etc hosts file to give tepshost its own entry as in the following
283. ither explicitly or implicitly assigns values to the symbols It can also be done through special controls like the Timespan dialog but these must be built into the product Custom query processing differs from standard query processing in that for custom queries a value must be provided for every symbol reference while for standard queries the system discards segments of the WHERE clause involving symbols for which a non null value cannot be found In order to support historical queries against summarized data various agent groups distribute product provided custom queries that reference symbols that are meant to be provided by the Timespan dialog when the Use summarized data option is selected These queries are used with the product provided Historical Summarized workspaces available from the Workspaces menu of the Operating System summary workspaces like Windows Systems In order for the queries to return data the Warehouse Proxy and Summarization and Pruning agents must be configured and have been running long enough to 178 IBM Tivoli Monitoring Troubleshooting Guide collect and summarize the data being viewed The queries reference a number of symbols Following are some of the commonly referenced symbols and example values KFW_USER Name of the summarized database schema owner Default is ITMUSER but it can be any value assigned by the customer during installation SHIFTPERIOD Indicator of shifts to inc
284. itialization failures e The ILU configured for use is not available to the application The ACBNAME or APPLNAME is properly defined in SYS1 VTAMLST but not in the connectable state CONCT The ACB must be varied ACTIVE to NET prior to Omegamon Platform Address Space startup The MVS command to verify the state of the ACB is D NET ID acbname E e The ILU is available but not a valid LU6 2 definition In this case the is a KBBCM001 message with an SNA sense code found in the RAS1 RKLVLOG Diagnose the 8 byte SNA sense code typically 087Dnnnn using the SNA Formats and Protocol manual e The LOGMODE is not a valid LU6 2 LOGMODE or the LOGMODE and MODETAB specification associated with the ILU definition are not the same by name on both systems hosting the endpoints This is most likely the case for SNA session establishment hangs The message in the RKLVLOG is Receive XID pending NULL and it is followed by another RPC timeout message The Server list IBM Tivoli Monitoring processes build and query a list of possible Tivoli Enterprise Monitoring Server hub addresses called the Server list This server list contains local LLB and global GLB entries The LLB entries of the Server list are derived The GLB entries of the Server list are built from the content of the KDCSSITE member of RKANPAR Shown below are two server lists The first Server List is for a Tivoli Enterprise Monitoring Server hub the second Server List is for a
285. itoring Server 149 message 112 Tivoli Enterprise Portal Server has lost contact 113 Tivoli Enterprise Portal Server is still being initialized 113 Unable to process log on 108 User authorization has failed 108 messages built in features 1 MIB data collection 198 MIB workspace 200 movefrom m2i bat 204 MVS Tivoli Enterprise Monitoring Server 152 N NETSUMMARY workspace 200 NetView menus unavailable 223 network interface cards 82 NETWORK workspace 199 nls_replace script 83 non hub situations 136 non root 136 137 173 O odadmin command 224 online publications accessing 248 OPAL documentation 253 ordering publications 248 out of memory error 229 P pdcollect tool 57 performance problems 126 popup blocker 121 private interface 116 problems types of 1 process send commands to a IBM Tivoli Monitoring Technology 53 publications accessing online 248 developerWorks Web site 253 OPAL 253 ordering 248 Redbooks 253 related 253 Technotes 253 types 251 wikis 253 Q Questions 67 Queue Statistics workspace 79 R rasllog tool 60 Redbooks 253 refreshTECinfo 225 release notes 2 Rule Check Utility error 221 rules not display 222 Rules Check Utility 222 S security Windows local 202 service console 53 see IBM Tivoli Monitoring Service Console 54 Silent install error 86 SITINFO 211 Situation Update Forwarder 48 situations 128 defined 194 non hub 136 not forwarded 211 UADVISOR 128 SNMP m
286. itoring vi help server 101 Uninstallation of an agent produces help errors 101 Uninstallation of an agent occurring more than once stops the OS agent 101 After uninstallation Tivoli Enterprise Monitoring Server folder is not deleted 101 Removing a failed installation on Windows 101 Incorrect behavior after an uninstallation and reinstallation 105 Tivoli Data Warehouse database dges ol uninstall 105 The agent installation log on nthe endpoint indicates that error AMXUT7512E occurred 106 Prompted to uninstall a database that was not running during uninstallation 106 Chapter 6 Connectivity troubleshooting a 107 A monitoring process fails to startup after changing a profile for root z 107 The Tivoli Enterprise Portal Server does not start after installation 107 Cannot log in to the Tivoli Enterprise Portal 108 Cannot connect to Tivoli Enterprise Portal Server 111 Cannot launch the Tivoli Enterprise Portal on an XP computer after installation with the message KFWITM215E 113 Tivoli Enterprise Portal Serveri is initializing and is not ready for communications 113 Tivoli Enterprise Portal Server lost contact with the Tivoli Enterprise Monitoring Server and is attempting to reconnect A 113 DB2 errors during a logon attempt to the Tivoli Enterprise Portal and portal server s3 Heartbeat issues when running IBM Tivoli Monitoring v6 x on a Linux guest using VMWare 114 Tivoli Enterprise Port
287. ituation Update Forwarder is started you must restart the Situation Update Forwarder for the change to take effect There are two trace files synch_trace log is always created synch_msg log is created if an error occurs while running the Situation Update Forwarder Run the following command to set the trace levels BINDIR TME TEC OM_TEC bin sitconfig sh update fileName configuration_file_ name logLevel trace_level where configuration_file_name The file name of the actively loaded configuration file as indicated by the situpdate properties file trace_level Specifies the level of trace as low med or verbose Setting up RAS1 tracing on z OS systems This syntax is used to specify a RAS1 trace in the KppENV file where pp is the product code HL for the OMEGAMON z OS Management Console or DS for the Tivoli Enterprise Monitoring Server After you add this command to the KppENV file you must stop and restart the address space for the command to take effect After that it remains in effect for the life of the address space To end the trace you must edit the KppENV file again to reset the trace level and stop and restart the address space The basic syntax of the RAS1 trace command is Chapter 4 Tools 49 50 KBB_RAS1 global_class COMP component_type ENTRY entry_point UNIT unit_name class where global_class Indicates the level of tracing that you want This is a global setting that applies to all RAS
288. ivoli Common Reporting User s Guide Locations of log files There are several logs that might contain information to help you debug problems with Tivoli Common Reporting and reports By default only errors are logged in the WebSphere Application Server SystemOut log file e If you enable logging and tracing using the information in Logging and tracing section of the IBM Tivoli Common Reporting User s Guide the Log and trace files are located in the profiles TIPProfile logs serverName subdirectory of the Tivoli Common Reporting installation directory Standard informational log messages are written to the System0ut 1og file detailed trace messages are written to the trace log file e If the connection to Tivoli Data Warehouse cannot be established look in the WebSphere Application Server SystemOut 1log file or SystemErr 1og file for more information To address this error ensure the drivers are placed in the correct directory Tivoli Common Reporter uses logger scripts to log during report generation If you see Javascript errors in the reports that you create look for Caused by in the stack trace This phrase points out the line number of the script in the report design at which the error occurred To see the SQL query that was generated by this error look at the log file You are missing drivers after the Tivoli Common Reporting installation In this scenario you have installed Tivoli Common Repo
289. ization and Pruning Agent 168 IBM Tivoli Monitoring Troubleshooting Guide Summarization and Pruning agent schedule not affected by daylight saving time If after starting the Summarization and Pruning agent you allow daylight savings time to be reached the agent start time is now 1 hour later than the time before the time change The agent does not seem to be aware of the time change The agent is respecting flexible scheduling in this case That it appears to be running one hour later is an artifact of the time change The number of minutes for the run remains the same Attribute names must be kept under 28 characters long Attribute names must be kept under 28 characters long due to the Summarization and Pruning Agent adding suffixes to the attribute names There are also limits in how long the column name can be for some of the DBMS that are supported by Tivoli Data Warehouse Agent deploy operations are not completing before the TIMEOUT expires When running agent deploy operations TIMEOUTs can occur because of slow network connections or slow hardware The agent deploy operations can complete if you increase the TIMEOUT value Some operations can complete even after a timeout is returned Table 39 Resolutions for agent deploy operations that TIMEOUT Problem Resolution KDY0014E Increase the Tivoli Enterprise Monitoring Server timeout value to 1200 message seconds TIMEOUT 1200 The default is 600 seconds
290. k Submit After clicking Submit you see a new report page that shows the error message SQL error displayed at the bottom of the report Here are some possible reasons for this error e The Tivoli Data Warehouse does not contain all of the attributes requested by the report This happens when you are running the report against an unsupported product version such as a version 3 1 OMEGAMON XE monitoring agent or a version of NetView for z OS prior to 5 3 e You did not allocate and configure historical data sets when you configured the monitoring agent using the Configuration Tool or did not enable historical collection from Tivoli Enterprise Portal or not enough time has passed for Tivoli Data Warehouse to collect the data for your requested report time period before you tried to generate an historical report with Tivoli Common Reporting e You might have incorrect or missing agent atr files installed at the Warehouse Proxy Agent e You might have attempted to use reports with a non supported database manager All reports work with DB2 UDB and some reports work with other database managers supported by Tivoli Data Warehouse DW You could circumvent this problem by collecting Tivoli Data Warehouse data in DB2 and then switching the TCR Data Source for the affected report package to that DB2 database Verify at the Tivoli Enterprise Portal console that you see the Historical Collection icon for the subject workspace and can draw a report for
291. kflow Editor is disabled and the following tools do not display Event Console Graphic View Edit Navigator View Navigator view ee 128 Situations are not firing 128 Historical UADVISOR sttvationd are started on the agent if historical collection is i to collect data 128 At the bottom of each view you see a historical workspace KFWITM217E error 129 Installation of situation data fails due fo I Ox on VSAM datasets 129 kshsoap client fails because of missing libraries on UNIX based systems 130 Category and Message field of ihe niversal message does not accept DBCS 130 An error occurs when remotely removing an instance on Windows gt 130 Agents display offline in the Tivoli Enterprise Portal but still fire situations and the agent logs are report that they are running 131 The Tivoli Enterprise Portal displays date sorting behavior when removing more than one sua system simultaneously x doll Multiple events that occur at the same time are loaded too slowly sai Desktop client performs poorly after installing Language Packs for IBM Tivoli Monitoring V6 1 131 Existing OMEGAMON product imagery displays after upgrading to IBM Tivoli Monitoring V6 1 132 The Warehouse Proxy Agent started but does not appear in the Managed System Status list on the Tivoli Enterprise Portal 132 Cannot start or i from the Navigator v
292. l override the Title text entered in the Style panel when the workspace is reached through the link This problem was reported against Tivoli Management Portal v1 9 6 but also applies to the Tivoli Enterprise Portal v6 1 0 v6 2 0 v6 1 1 and v6 2 2 Unresolved variables in custom queries Whenever you assign a custom query containing delimited symbol references to a view through the Properties dialog a popup appears asking for values for all symbol references for which a value cannot be found The values provided through this dialog are used to parameterize the query issued to fill the Preview pane of the Properties dialog The values provided will NOT be saved in the workspace state although they will remain defined from when the Properties dialog is dismissed until the workspace is refreshed The reason for the prompt for values is to allow the author of the query workspace to test whether or not the query returns the expected result set for the provided values The values are optional and need not be provided in order to complete the assignment of the query to the view The only impact of not providing values is that the query triggers a syntax error when executed by the Properties dialog Preview pane and the view is empty In order for the query to work correctly in the workspace values must be provided for all the referenced symbols through the execution environment Typically this is done by reaching the workspace through a link that e
293. l security policy on an Active Directory server If you installed IBM Tivoli Monitoring on a system where Microsoft Windows Active Directory is installed you must update the local security policy on that system to include the sysadmin user so that you can log on to the Tivoli Enterprise Portal This configuration task is necessary only on Active Directory systems and must be performed whether or not you plan to use the Monitoring Agent for Active Directory Follow these steps to configure the local security policy e Enter secpol msc at a command prompt The Local Security Settings window is displayed e In the navigation pane on the left select Local Policies gt User Rights Assignments e In the list of policies right click Log on locally and select Security e Click Add The Select Users or Groups window is displayed e Select sysadmin from the list of user names Click Add e Click OK e On the Local Security Policy Setting window verify that the check box in the Effective Policy Setting column is selected for the sysadmin user e Click OK to save the new setting and exit Chapter 7 Tivoli Enterprise Portal troubleshooting 121 Several enterprise workspaces are returning an error KFWITM217E Request Error SQL1_CreateRequest Failed rc 350 The following workspaces are Link Targets and should not be navigated to directly e Deploy Status By Product e Deploy Status By Deploy Group e Installation Logs Attempts to n
294. le The following is an excerpt from later in the same log Chapter 4 Tools 41 42 userDataList calculateChecksum Initial creation of catalog attribute tables for app Name lt Tivoli_Data_Warehouse gt KUMP_ODBCserver ODBC source lt Tivoli_Data_Warehouse gt table lt syscharsets gt is now online to the data provider KUMP_ODBCserver ODBC source lt Tivoli_Data_Warehouse gt table lt syscacheobjects gt is now online to the data provider KUMP_ODBCserver ODBC source lt Tivoli_Data_Warehouse gt table lt syscomments gt is now online to the data provider Setting traces When you encounter an error with IBM Tivoli Monitoring that requires contacting IBM Software Support you might be asked to submit a copy of the error log The error log is part of the trace diagnostic tool in Tivoli Enterprise Portal It is set to log errors and you can set other parameters for collecting specific details Always backup the files before altering them Setting the trace option for the Tivoli Enterprise Portal client trace A log file is created automatically the first time you start Tivoli Enterprise Portal and is named install _dir cnp logs kcjrasl 1og This log file contains all of the RASI tracing for the Tivoli Enterprise Portal client Whenever you start a new work session the log file is purged and rewritten for the current work session If you want to preserve the log file from the last work session you must rename it or copy
295. le mode Tivoli Enterprise Portal Server 55 Corba user exception 137 CPU consumption high 189 createNode command 185 createSit 190 CT_CMSLIST 82 CTIRA_HOSTNAME 177 D data prevent losing 201 data helpful types of 5 database backing up the 201 recovering the 201 restoring the 201 DB2 UDB user ID 201 DBCS not accepted 130 developerWorks Web site 253 diagnostic tool ACTION workspace 57 diagnostic tools Backspace Check utility 63 DPLOG 56 Log and Trace Analyzer tool 60 pdcollect tool 57 rasllog 60 UAGENT application ACTION workspace 56 DPLOG 56 DirectDraw thread loops 127 DNS registry key 82 documentation See publications E Eclipse Help Server 136 ECO2029E 224 education see Tivoli technical training 248 enable Tivoli Monitoring Services 128 encryption key setting error 86 error ECO2029E 224 Error 0x80040707 82 Could not open DNS registry key 82 error message Failed connection to Tivoli Enterprise Portal Server 111 Reconnect Authorization Failed 115 259 error message continued Tivoli Enterprise Portal client cannot find Tivoli Enterprise Portal Server 112 Tivoli Enterprise Portal Server is not reconnecting 115 errors Rule Check Utility 221 event Omegamon_Generic 215 event forwarder 47 Event IBM Tivoli Monitoring Event Forwarding tool error 79 events 131 PARSING_FAILED status 216 TEC_ITM_OM_Situation_Sync_Error 221 F Failed to attach to the DB2 instance db2inst1 86 FAQ see Frequently asked que
296. ledge and experience of other Tivoli users Tivoli user groups include the following members and groups e 23 000 members e 144 groups Access the link for the Tivoli Users Group at www tivoli ug org Appendix C Support information 249 250 IBM Tivoli Monitoring Troubleshooting Guide Appendix D Documentation library This appendix contains information about the publications related to IBM Tivoli Monitoring and to the commonly shared components of Tivoli Management Services These publications are listed in the following categories e IBM Tivoli Monitoring library e Related publications See IBM Tivoli Monitoring and OMEGAMON XE Products Documentation Guide SC23 8816 for information about accessing and using the publications You can find the Documentation Guide in the IBM Tivoli Monitoring and OMEGAMON XE Information Center at http publib boulder ibm com infocenter tivihelp v15r1 To open the Documentation Guide in the information center select Using the publications in the Contents pane To find a list of new and changed publications click What s new on the Welcome page of the IBM Tivoli Monitoring and OMEGAMON XE Information Center To find publications from the previous version of a product click Previous versions under the name of the product in the Contents pane IBM Tivoli Monitoring library The following publications provide information about IBM Tivoli Monitoring and about the commonly shared compone
297. leshooting on page 107 tacmd exportWorkspaces or importWorkspaces receives an out of memory error If you get an OutOfMemory Error when running this command you can increase the maximum Java heap size for the tacmd java JVM by using the TACMD_JVM_MAX_MEMORY environment variable This variable specifies the maximum java heap size in megabytes for the tacmd Java virtual machine Memory for tacmd is freed when the tacmd invocation finishes Valid values are 256 through to 2048 inclusive For IBM Tivoli Monitoring v6 2 2 Fix Pack 2 or higher you can set it in the command environment using SET export or you can set it in the environment files KUIENV on Windows systems or the CANDLEHOME bin tacmd shell script on UNIX and Linux systems The portal server and Warehouse Proxy Agent fail to connect to the database on a 64 bit Windows system The ODBC control panel available in the Windows Start menu is for 64 bit ODBC configuration Since the portal server and the Warehouse Proxy Agent are 32 bit applications the 32 bit ODBC control panel must be used Launch the 32 bit ODBC dialog from C Windows SysWOW64 odbccp32 cpl and manually create the ITM Warehouse TEPS ODBC source by going to the System DSN tab and clicking Add Failed to login as sysadmin with portal server LDAP enabled When LDAP authentication is enabled for the monitoring server the sysadmin ID must be defined in the LDAP server However when LDAP is enabl
298. listOverride and even modify setOverride the override If you use the setOverride command using a predicate of 99 for example the agent applies this value as 99 0 If you then try the deleteOverride command using 99 rather than 99 0 the command does not find a matching override and it fails If you read the listOverrides command output and use the reported values to run the deleteOverride command the override is deleted Copyright IBM Corp 2005 2010 181 tacmd listsit m UX Managed System gives no result The tacmd listsit command on AIX systems gives no results as shown below tacmd listsit m UX Managed System gives no result Return code 255 This behavior might be caused by an OUT OF MEMORY condition on the AIX system where the command is issued due to specific AIX memory management This problem can be solved by setting the environment variable LDR_CNTRL MAXDATA 0x80000000 to be exported in the shell from which the tacmd command is issued export LDR_CNTRL MAXDATA 0x80000000 Receive a busy monitoring server message when using the getfile putfile or executecommand commands If you receive a busy monitoring server message there are too many file transfers in progress Wait until some of the transfers complete and then retry your command The following error messages are received for the respective commands For the getfile command KUIGTF111E Monitoring server is too busy For the putfile command KUIPT
299. lkit The agent was not installed for one of the following reasons e There is another installation in progress that cannot complete until the computer is restarted OR e You are attempting to install a component that is already installed Chapter 5 Installation and configuration troubleshooting 75 Refer to the 1cfd 10g on the endpoint and the agent installation log as listed in Table 29 to determine the exact cause of the problem Table 29 Icfd log file Windows install _dir Install Abort IBM Tivoli Monitoring install_dir logs candle_installation log timeStamp log UNIX based systems Contact IBM Software Support if you cannot install the agent See Collecting and reviewing troubleshooting e 5 for information on what types of data to collect betore contacting Support Failure occurs when sharing directories for the agent deploy depot Although it is more efficient to use a network shared directory for the agent deploy depot directory there are weaknesses that might negatively impact deployment in large enterprises e If an NFS is used to contain the depot and there is a problem with the NFS then the deployment activity is suspended for all deployments in progress e For UNIX environments the directories that are mentioned on the shared directory must have the names of each of the Tivoli Enterprise Monitoring Server servers Administrator privileges need to be assigned based on a domain User ID This i
300. ll after running the install sh file You receive the following message after running the install sh file This system is using a version of the Korn Shell ksh which will allow the installation of specific older releases of ITM Monitoring application agents to regress the installation configuration and execution tools for this installation Refer to the Flash note titled Newer ksh shell may allow regression of ITM installation for more information including steps to take to avoid damage to your IBM Tivoli Monitoring installation This technote includes detailed descriptions of the root cause and also information on how to deal with this issue http g01zciwas003 ahe pok ibm com support dcf preview wss host g01zcidbs003 ahe pok ibm com amp db support swg tivtech nsf amp unid FACF542BAA62D17B8525765C006BC192 amp tax0C SSCQLMD amp MD 2009 11 05 2012 11 58 amp sid AIX stat_daemon memory leak There is a memory leak in the process stat_daemon that is fixed by an AIX APAR on AIX 5 3 and AIX 6 1 To fix the memory leak install APAR IZ62080 for AIX 5 3 and APAR IZ58432 for AIX 6 1 UNIX and Linux install sh command fails with error code 99 and error code 4 Installation on UNIX and Linux systems uses install sh command running this command fails with a runGSkit failure Return error code 99 and a install sh failure Return error code 4 Because it failed within runGSkit when it called verifyInstall review the InstallDir
301. ll not continue automatically Troubleshooting Tivoli Enterprise Monitoring Server problems on z OS systems This section includes describes problems you might experience with Tivoli Enterprise Monitoring Server on z OS and provides resolutions It covers problems that occur during run time that you can resolve with the Installation and Configuration Assistance Tool ICAT For more information about configuring the Tivoli Enterprise Monitoring Server on z OS refer to the Configuring Tivoli Enterprise Monitoring Server on z OS manual Receive Program KDFCINIT and Program FAXCMON messages Program KDFCINIT with task id 8 ended and Program FAXCMON with task id 7 ended messages are generated in the IBM Tivoli Monitoring z OS Tivoli Enterprise Monitoring Server RKLVLOG if either or both IBM Tivoli OMEGAMON XE on z OS and OMEGAMON XE for Storage on z OS are running in the 150 IBM Tivoli Monitoring Troubleshooting Guide monitoring server address space These are informational messages that report on internal task terminations which in turn help identify non terminating internal tasks These messages are benign The Tivoli Enterprise Monitoring Server start task CANSDSST default encountered error message KLVST044 LOADLIST MEMBER NOT FOUND IN RKANPAR DATASET KDSLLIST KppLLIST KLVST001 CANDLE ENGINE INITIALIZATION ERROR S ABEND U0012 in the RKLVLOG at startup The Tivoli Enterprise Monitoring Server start task CANSDSST default
302. llected by the attached Tivoli Enterprise Monitoring Agents A HUB Tivoli Enterprise Monitoring Server has been running A shutdown of the Tivoli Enterprise Monitoring Server and Tivoli Enterprise Monitoring Agents on the remote systems is in process but the shutdown takes awhile due to abends in the remote Tivoli Enterprise Monitoring Server About 8 or 9 minutes go by before the HUB Tivoli Enterprise Monitoring Server quiesces There are a lot of remote request communication messages in the HUB s RKLVLOG prior to the QUIESCE but no other signs of errors until after the abend You cannot restart the remote environments following the quiesce until after the HUB environment is recycled The value of the MINIMUM parameter within the KDSSYSIN member of the RKANPARU library might need to be increased if the STGDEBUG X or STGDEBUG Y parameter is also supplied within KDSSYSIN If the address space controlled by this KDSSYSIN member enters a storage quiesce state indicated by a KLVxxxxx message stating that there is a storage shortage or quiesce in effect you should increase the value of the MINIMUM parameter and restart the address space Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 147 148 During installation of a remote Tivoli Enterprise Monitoring Server on a Windows system the agent support is applied but fails In a multiple level Tivoli Enterprise Monitoring Server environment the remote Tivoli Enterprise Monitorin
303. lling a backlevel Windows OS Agent into an existing 6 2 1 environment is not supported and the Tivoli Enterprise Monitoring Server can not start If this is the desired deployment the backlevel Windows OS Agent should be installed first The link to an OS agent workspace is pointing to a superseded version of the workspace Links point to specific object names and are not automatically updated to return the latest version of a workspace A link from an external agent pointing into the OS agent must be updated to point to the current release If the current version of the agent is not truly versioned using the VRF appendage to the object name but is a different object name to facilitate 64 bit data workspace exposure the external anchor should have two links associated with it one pointing to the existing version of the workspace and a second pointing to the new 64 bit enabled workspace Non OS agents should not expect a specific version of the workspace to be available unless their respective agent requires that base version In this case it might not be feasible to update agents until the next required release SNMP trap Sendto fails An agent running on an IPv4 or dual stack IPv4 IPv6 system tries to emit an SNMP trap to an IPv6 destination address defined in the trapcnfg XML file but the trap is not received on the destination system In the agent log file there is an error message stating that the SNMP trap Sendto failed with rc 1
304. log files created during installations Table 23 Installation log files Windows e ITM_HOME Instal1ITM Abort lt Product_name gt lt date_timestamp gt 1og This log is created if an abort occurs for either a first time install or a modification of previous installation of IBM Tivoli Monitoring e ITM HOME Instal lITM lt Product_name gt lt timestamp gt log This log is created during a normal clean installation e ITM HOME Instal lITM MOD_ lt Product_name gt timestamp log This log is created if you modify an existing product specified with the PC or when adding or deleting components where Product_name Specifies the product name IBM Tivoli Monitoring 20050923 1815 10g is the log file name for the IBM Tivoli Monitoring installation CD timestamp A decimal representation of the time at which the process was started UNIX based systems CANDLEHOME 1ogs candle_ installation log You can find a log for uninstallation on Windows in the root directory where the product was installed Uninstall lt PC gt lt date_timestamp gt log UNIX installer and configuration logs For tracing and logging java code that is run on UNIX systems this mechanism enables problem debugging Two sets of information are created logs and traces Logs log are globalized and traces trc are in English They contain entry and exit parameters of method and stack traces for exceptions The amount of information traced depend
305. lt timeout_value gt where lt timeout_value gt is the length of the timeout period in half seconds To configure a timeout of 30 seconds set the lt timeout_value gt value to 60 On Solaris and HP UX run the following command ndd set dev tcp tcp_ip_abort_cinterval lt timeout_value gt where lt timeout_value gt is the length of the timeout period in milliseconds To configure a timeout of 30 seconds set the lt timeout_value gt value to 30000 Unexpected exception occurred when importing kqr baroc kqt baroc and kqv baroc First import both the Sentry baroc and om_tec baroc files to the rulebase on the Tivoli Enterprise Console server Then import the kqr baroc kqv baroc or kvm baroc files Editing the default destination server information from the command line does not work If the command tacmd editEventdest is run specifying a default destination server these changes are not persistent in the om_tec config file after running the tacmd refreshTECinfo command Here is an example of this command tacmd editEventdest i 0 p host2 nuke tivlab austin ibm com The new information also does not appear in the output of the command tacmd viewEventDest This occurs since the default destination server information cannot be edited from the command line but only manually in the om_tec config file Manually edit the om_tec config file to specify a default destination server Situation continues to send events even
306. lude 1 AllShifts 1 OffPeak 2 Peak VACATIONPERIOD Indicator of vacation days to include 1 AllDays 0 WorkDays 1 VacationDays TIMESPAN Set of values captured by the Timespan dialog It is not practical to construct this directly SUMMARY_DAY Day to select A string in the format YYYY MM DD The following is an example link wizard expression that can build such a string value from the a TEP 16 character timestamp attribute value assuming years in the 20yy range dt knt Processor ATTRIBUTE NTPROCSSR_H WRITETIMES yyyy 20 STR dt 2 2 mm STR dt 4 2 dd STR dt 6 2 yyyy mm dd SUMMARY_WEEK First day of week to select A string in the format YYYY MM DD Because of the open ended nature of the custom queries and their provision by multiple agent groups it is possible that other symbols might also be used One way to understand what values are required is to save a copy of the query and edit it to remove clauses involving the symbols When the modified query is assigned to a view assuming any syntax errors are overcome it should return an unfiltered view of the table The values in the table in conjunction with close examination of the query text can be used as a guide to what to provide as filter values Another symbol that has been specifically asked about is Server_Name This is the name of the server of interest If the workspace is below the agent level in the navigator tree
307. me as problem solving although during the process of troubleshooting you can obtain enough information to solve a problem Examples of situations where this can happen include e End user errors e Application programming errors e System programming errors such as in resource definitions However you might not always be able to solve a problem yourself after determining its cause For example a performance problem might be caused by a limitation of your hardware If you are unable to solve a problem on your own contact IBM Software Support for a solution See Chapter 2 Collecting and reviewing troubleshooting data on page 5 for information on what types of data to collect before contacting Support Sources of troubleshooting information The primary troubleshooting feature is logging Logging refers to the text messages and trace data generated by the software Messages and trace data are sent to an output destination such as a console screen or a file Typically text messages relay information about the state and performance of a system or application Messages also alert the system administrator to exceptional conditions when they occur Consult the explanation and operator response associated with the displayed messages to determine the cause of the failure See the document IBM Tivoli Monitoring Messages for message information Trace data capture transient information about the current operating environment w
308. me must match the name of the DB2 connection port service that you placed in etc services 6 Start and stop db2 db2stop db2start o o After following these instructions you should be able to open the Tivoli Enterprise Portal with workspaces Heartbeat issues when running IBM Tivoli Monitoring v6 x on a Linux guest using VMWare 114 When the Linux operating system is run as a guest using VMWare it is possible for the clock of the Linux guest to run either faster or slower than real world time If any IBM Tivoli Monitoring v6 x products are installed on Linux guests whose clocks are not running correctly the result can be erratic system behavior For example if the Monitoring Agent for Linux OS is installed on a Linux operating system guest whose clock is running too slow heartbeats from the agent are not produced on time This results in the agent continuously going OFFLINE and ONLINE at the Tivoli Enterprise Monitoring Server as the heartbeats arrive after the time interval has expired VMWare is aware of this issue and has written several articles that address this problem The following articles were current at the time this guide was published e Clock in a Linux Guest Runs More Slowly or Quickly Than Real Time http kb vmware com selfservice microsites search do language en_US amp cmd displayKC amp externalID 1420 IBM Tivoli Monitoring Troubleshooting Guide For the most up to date information on t
309. modified to use an older less robust encoding algorithm Edit the kfwenf file in install_dir CNPS where install_dir is C IBM ITM by default in a text editor 2 Ina line by itself type the text USE_EGG1_FLAG 1 3 Save the document and exit 4 Stop the Tivoli Enterprise Portal Server if it is running then start it On a system with one or more existing OMEGAMON monitoring environments the global CSI is corrupt and cannot be used for installing a new version Use the following steps manually merge existing datasets with IBM Tivoli Monitoring V6 1 datasets only after IBM Software Support instructs you to do so 1 Backup the following datasets e Runtime e Target NSTDATA NSTDATW STJOBS STLIB STLIBW STOLCK NSTSTAT Z ZZ e DSRS RF Note You must backup the specified datasets 2 After installing the product on a new CSI refer to the Program Directory document to verify that the datasets contain enough allocated space for the new libraries 3 From the ISPF panel 3 3 or a JCL batch job copy the contents of the new modified Target libraries in Table 30 to their counterparts in the old Target libraries ensuring that like named members are replaced Table 30 Modified target libraries TKANCMD TKANCUS TKANDATV TKANHENU TKANMAC TKANMOD TKANMODL TKANPAR TKANPENU TKANSAM TKCIINST TKNSLOCL 4 Manually delete obsolete datasets listed in Table 31 on page 10
310. mple is an excerpt from an Upgrade Toolkit tool log file lt Message Id AMXUT2504I Severity INF0 gt lt Time Millis 1121977824199 gt 2005 07 21 15 30 24 199 CST lt Time gt lt Server Format IP gt YFELDMA1 austin ibm com lt Server gt IBM Tivoli Monitoring Troubleshooting Guide lt Product Id gt AMXAMX lt Product Id gt lt Component gt ScanTMR lt Component gt lt Component gt 1 lt ProductInstance gt lt LogText gt lt CDATA AMXUT2504I The software is creating a new baseline file C PROGRA 1 Tivoli db YFELDMA1 db AMX shared analyze scans 1889259234 xml1 lt LogText gt lt TranslationInfo Type JAVA Catalog com ibm opmt utils messages MigrationManager_ msgs MsgKey AMXUT25041 gt lt Param gt lt CDATA C PROGRA 1 Tivoli db YFELDMA1 db AMX shared analyze scans 1889259234 xm1 lt Parm gt lt TranslationInfo gt lt Principal gt lt Principal gt lt Message gt Reading RAS1 logs This section provides an example of the Universal Agent RASI trace logs By default the Universal Agent RASI trace log lists the following details about the health of an ODBC data provider application e Whether the ODBC tables come online during startup e Whether the ODBC table data is collected e Errors with the ODBC related status messages including informational messages about when each ODBC connection completes e Errors that occur during ODBC data provider data retrieval including
311. must be at least 260 but you can allow it to default to the system determined value for best performance The amount of space allocated to the datasets is not critical Allocating five 3390 cylinders allows space for about 50000 log records The number varies depending on the lengths of the messages Note Do not specify secondary allocations Any secondary allocations are ignored 2 Edit your Tivoli Enterprise Monitoring Server JCL procedure typically named CANSDSST but can be named otherwise For each of your datasets add a DD statement with the DDNAME RKMSLGnn that points to the dataset with DISP SHR The first dataset should use RKMSLG00 as the DDNAME with nn incrementing by one for each additional dataset Note DO NOT SKIP VALUES of nn If any values are skipped subsequent OpsLog DD statements are ignored You can examine the contents of the log datasets using ISPF Browse or any equivalent tool Note that the dataset that is currently receiving log data might appear to be empty You can force a switch to the next dataset which in turn will flush any buffered log data to the current dataset using the MODIFY OPSLOGSW console command The syntax of this command is F procname OPSLOGSW KMS KMS indicates that the log associated with Tivoli Enterprise Monitoring Server is to be processed ITMSuper The ITMSUPER tool performs audits of the IBM Tivoli Monitoring environment topology connectivity applicati
312. n attribute The tacmd createSit command enables you to create situations without using the Tivoli Enterprise Portal However if you mistype the name of an attribute when using this command you do not receive an error The situation is created skipping the attribute that you meant to type in the command If the created situation had for example 6 attributes to monitor the new created situation has only 5 if you made a mistake in typing 1 of the attribute names in the command If you are using the IBM Tivoli Monitoring command line tacmd createSit function for situation creation you can use the Situation editor in the Tivoli Enterprise Portal to validate your specified attributes wsadmin commands output indicates the wrong server name When the wsadmin commands are run the output indicates to restart server which is the default name for WAS server However the eWAS server for IBM Tivoli Monitoring is called ITMServer C ibm ITM CNPSJ profiles ITMProfile bin gt wsadmin connType NONE WASX7357I By request this scripting client is not connected to any server process C TEPSEWASBundle loaded WASX7029I For help enter Help help wsadmin gt securityoff LOCAL 0S security is off now but you need to restart serverl to make it affected Restart the server by its correct name whether that is the default name or not When using the viewuser command you receive a message that an option is repeating The v p a and o opti
313. n TCP IP service request The INITAPI establishes a communication pipe between the TCP IP and the OMEGAMON Platform address Chapter 5 Installation and configuration troubleshooting 89 90 space The INITAPI identifies the TCP IP stack to be used by name The TCP IP stack name used in the INITAPI is configured in the KLXINTCP member of RKANPAR This step must complete successfully An INITAPI failure is fatal no TCP IP services are available to the address space Confirming that the INITAPI call was successful The following messages indicate a successful INITAPI KLXINOO1 HPNS INTERFACE AVAILABLE KLXINOO1 SOCKET INTERFACE TO TCPIPL AVAILABLE Repairing the INITAPI call failures Most of INITAPI failures are the result of the wrong name specified in KLXINTCP The following is a classic example of an INITAPI failure KLVIN405 STARTUP MODULE KLXINTCP SEQUENCE 1 USING RKANPAR MEMBER KLXINTCP KLXINOO1 TCP IP CONFIGURATION TCP IP_USERID TCPIPG KLXINO 3 TCP IP INTERFACE INITIALIZED KLXINOO9 SOCKET INTERFACE TO TCPIPG UNAVAILABLE RC FFFFFFFF ERRNO 000003F3 KLXINOO4 TCP IP INTERFACE NOT OPENED RC 4 Notice that the INITAPI failure is characterized by a return code of 1 and an ERRNO value in this case X 3F3 or decimal 1011 ERRNO s have names These names are found in TCPERRNO H and decimal 1011 is EIBMBADTCPNAME The most common INTIAPI ERRNOs are EMVSINITIAL 156 EIBMBADTCPNAME 1011 and no name 10218 Reasons for INITA
314. n deploying an agent from the Tivoli Enterprise Portal The Tivoli Enterprise Portal Server times out waiting for deployment action to complete The default timeout is 600 seconds You can change the timeout setting to KFW_SQL1_ASYNC_NOTIFY_MAX_WAIT in kfwenv KFW_SQL1_ASYNC_NOTIFY_MAX_WAIT 1000 KUICCN068E error when running tacmd createnode Increase the timeout value in seconds by adding o TIMEOUT 3600 to the createnode command Deploy cannot tell if the install failed When installing an OS Agent on an endpoint that already has an OS Agent the installation program does not write out a C IBM ITM Insta lITM Abort IBM Tivoli Monitoring 20070924 1319 1og in the createNode specified directory It writes messages to the existing C data itm61_oqv_ga2_koy Insta l ITM IBM Tivoli Monitoring 20070924 1319 1og log file and reports the error in that log file An agent does not display in the Tivoli Enterprise Portal or in the output from the listSystems command 170 If you have multiple instances of a monitoring agent you must decide how to name the monitoring agents This name is intended to uniquely identify that monitoring agent The agent s default name is composed of three qualifiers e Optional instance name e Machine network hostname e Agent product node type An agent name truncation problem can occur when the network domain name is included in the network hostname portion of the agent name For
315. n page 57 Copyright IBM Corp 2005 2010 9 IBM Support Assistant The IBM Support Assistant is a free stand alone application that you can install on any workstation You can then enhance the application by installing product specific plug in modules for the IBM products you use For more information see Appendix C Support information on page 247 10 IBM Tivoli Monitoring Troubleshooting Guide I am trying to find out what software is supported The following information lists the resources available to determine what software is supported IBM Tivoli Monitoring Installation and Setup Guide The IBM Tivoli Monitoring Installation and Setup Guide provides specific information about the supported software for IBM Tivoli Monitoring Use the Hardware and software requirements section in Chapter 3 of the Installation and Setup Guide to find information on supported software To find the Hardware and software requirements section of the Installation and Setup Guide go to the Tivoli Monitoring and OMEGAMON XE Information Center at http publib boulder ibm com infocenter tivihelp v15r1 index jsp In the left navigation panel select IBM Tivoli Monitoring expand the version of the software you are using for example Version 6 2 2 then expand Installing and Configuring Then navigate through the following levels Planning your installation gt Preparing for installation gt Hardware and software requirements
316. n with zkum To confirm that portal server generated Tivoli Universal Agent queries have been deleted or to see which queries are currently defined run the following select command against the KFWQUERY table kfwsqlclient d TEPS2 e select id name from kfwquery where id like zkum Clearing the JAR cache If you encounter problems IBM Software Support might instruct you to uninstall and to clear the JAR cache 1 If Tivoli Enterprise Portal is running exit by closing the browser window 2 Start the Java Plug in Additional Information You can find the Java Plug in in Start gt Settings gt Control Panel To start it double click the Java Plug in icon Your Windows desktop might have a shortcut to the Java Plug in 1 4 2 3 In the Java Plug in Control Panel window select the Cache tab and click Clear JAR Cache 4 When a message indicates the JAR cache is cleared click OK If you want to start Tivoli Enterprise Portal browser mode again restart Internet Explorer and type the URL for Tivoli Enterprise Portal The Java Extension Installation progress bar shows as each Java archive file is downloaded Upon completion the logon window opens and prompt you to enter a user ID Using the UAGENT application 56 The UAGENT application is a diagnostic tool to help solve problems you might experience with the IBM Tivoli Universal Agent Every Universal Agent data provider automatically activates an application called
317. n_pr gt lt class name ITM_KUDINF000 gt 156 IBM Tivoli Monitoring Troubleshooting Guide lt slot slotNamel msg gt lt mappedAttribute name Incorrect slotName gt lt slot gt lt event_mapping gt You will see the following errors in the agent log 4AB91AF5 0005 2 kraaeevx cpp 686 process Slot_Tag lt slotName gt attribute not specified for element lt slot gt 4AB91AF5 0006 2 kraaeevx cpp 192 IRA_EIF_endXMLElement Null Emb lt 110A05EB0 gt or Esb lt NULL gt processing slot end element 4AB91AF5 0007 2 kraaeevm cpp 788 processMapFile Event map parsing error Map entries from file lt data achan private_situation ud_eventmap map gt not added Subnode Limitations for autonomous function The Service Interface has the following limitations when working with subnode agents e The Queries Link has been removed and is not supported for subnode agents e The Situations Link only shows situations from the agent instance level While situations can be distributed to run on specific subnodes the Situations page does not filter by subnode so all situations for all subnodes defined in the agent instance are shown e The History views show metrics for all subnodes combined in a single table Currently the service interface does not allow filtering by subnode SNMP alerts sent from the monitoring agent have some limitations when working with subnodes Currently the monitoring agent does not support situa
318. nage the size of trace data repositories complete the following steps 1 Run pdcollect on the remote system 2 Use FTP to retrieve the collected file 3 Unpack the collected file Chapter 4 Tools 59 ras1log tool This is a tool that converts the time stamps contained in trace logs into readable values This tool can be found in the itm_install bin directory on both Windows and UNIX systems The following lists how the help appears usage rasllog 1 u logfile 1 for local time u for UTC time logfile can be either a file name or for stdin default You can either pass the tool a file name or you can filter a file through it to obtain a readable log You do not need to specify any arguments The following examples work on Windows systems rasllog lt balayne_ms_46c071a6 01 1og rasllog lt balayne_ms_46c071a6 01 1log grep GetEnv rasllog lt balayne ms_46c071a6 01 1og gt tems_log The first example sends the result to the screen the second sends the result to grep to find all of the lines with the text GetEnv in them which are then printed on the screen and the third sends the result to a file named tems_1og By default this tool converts the timestamps to UTC time When using the l option it writes local time instead Log and Trace Analyzer Tool With the Log and Trace Analyzer you can gather system and performance data from local and remote systems The data can be used for troubleshooting should
319. name TECLIB where opt IBM ITM is the default location where the Tivoli Enterprise Monitoring Server is installed and the host name is the value supplied during the Tivoli Enterprise Monitoring Server configuration e Ona Windows system 1 Click Start gt Programs gt IBM Tivoli Monitoring gt Manage Tivoli Enterprise Monitoring Services Right click Tivoli Enterprise Monitoring Server gt Advanced Select Edit EIF configuration to open the om_tec config file in Notepad Examine the FilterMode and Filter statements a PON Specify the appropriate filter statements See the IBM Tivoli Monitoring Administrator s Guide for the supported EIF configuration parameters Also see the Tivoli Event Integration Facility Version 3 9 reference manual SC32 1241 for more details e Ona UNIX based systems system 1 cd opt IBM ITM tables hubname TECLIB where hubname Specifies the name of the hub monitoring server 2 Open the om_tec config file in a text editor 9 Examine the FilterMode and Filter statements 4 Specify the appropriate filter statements See the IBM Tivoli Monitoring Administrator s Guide for the supported EIF configuration parameters Also see the Tivoli Event Integration Facility Version 3 9 reference manual SC32 1241 for more details Note On z OS and 64 bit native platforms a physical event cache is not available from the TECLIB directory The event cache is in a memory cache
320. ncrease the following storage related parameters if IBM Software Support personnel instructs you to do so e Web Services SOAP Server e startup console messages e communications trace e storage detail logging and associated intervals minimum extended storage Chapter 5 Installation and configuration troubleshooting 95 e primary and extended maximum storage request size e language locale e persistent datastore parameters Use the following steps to increase the storage related parameters 1 From the Configure the Tivoli Enterprise Monitoring Server main menu select the Specify configuration values option 2 On the next panel press F5 Advanced to open the Specify Advanced Configuration Values panel The next panel includes the following options e Enable OMEGAMON SOAP Server applicable to a Hub Tivoli Enterprise Monitoring Server only Enable startup console messages Enable communications trace e Enable storage detail logging and associated intervals e Specify the Virtual IP Address VIPA type e Specify the minimum extended storage e Specify the primary and extended maximum storage request size e Specify the language locale for globalization e Specify the persistent datastore parameters 3 Customize the fields with the preferred values in the Specify Advanced Configuration Values 4 Select the Create runtime members option to regenerate the DS 3xxxx Create runtime members job from the Configure the Tivoli Enterprise
321. ncrease the maximum setting in increments of 64 MB until the symptoms disappear e Make sure the client workstation has enough memory to handle the maximum heap size To determine if the client workstation has sufficient memory observe the available physical memory as shown on the Windows Task Manager Performance tab when the workstation is not running the Tivoli Enterprise Portal client but is running any other applications that need to run concurrently with the portal client Verify that the client workstation has enough available 124 IBM Tivoli Monitoring Troubleshooting Guide physical memory to hold the entire maximum Java heap size for the Tivoli Enterprise Portal plus another 150 MB The additional 150 MB provides an allowance for non Java heap storage for the Tivoli Enterprise Portal and extra available memory for use by the operating system e Set the Java Plug in cache to an unlimited size to avoid performance problems due to insufficient cache space for Tivoli Enterprise Portal JAR files e If you have just upgraded to a new release or fix pack clear the plug in cache to remove old versions of the Tivoli Enterprise Portal JAR files Complete the following steps to adjust the Java Plug in settings 1 Open the Windows Control Panel 2 Double click IBM Control Panel for Java TM to display the Java TM Control Panel 3 From the Java TM tab a Click View b Double click on the Java Runtime Parameters field and enter
322. ncs KDC1_STC_CANT_BIND_SOCK 4703AF9A 002C 4 kdebpap c 125 KDEBP_AssignPort ip pipe bound to port 14206 base 1918 limit 1918 4703B06C 0000 4 kdcclsr c 562 rpc__sar Endpoint unresponsive ip pipe 9 42 22 26 1918 1C010001 1DE0000F 210 5 2 FFFF 1 1 1 1 9 d7273a 4703B06F 0000 4 kdc10c1 c 129 KDCLO ClientLookup status 1c020006 location server unavailable ncs KDC1_STC_SERVER_UNAVAILABLE 4703B08F 0000 4 kdcclsr c 562 rpc__sar Endpoint unresponsive ip 9 42 22 26 1918 1C010001 1DEO0000F 32 5 2 FFFF 2 1 1 1 9 d7273a 4703B092 0000 4 kdc10c1 c 129 KDCLO ClientLookup status 1c020006 location server unavailable ncs KDC1_STC_SERVER_UNAVAILABLE You can find out what is using that port stop the process and then configure your system to use another port You can also reboot the system to clean up any stale IBM Tivoli Monitoring processes that might be bound to this port Remote Tivoli Enterprise Monitoring Server lost connection to the hub Tivoli Enterprise Monitoring Server and all agents display offline Check the log for error messages in the RASI trace log that indicate timestamp problems such as Ignoring invalid lastTimeStamp This error occurs because you synchronized the time on the hub and remote Tivoli Enterprise Monitoring Server with another time server Restart the Tivoli Enterprise Monitoring Server experiencing the errors because timers and POSIX timed waits for example depend on reliable sy
323. nd also removes the new situations and managed system list Follow these steps to roll back the upgrade 1 Change to the DBDIR AMX shared analyze directory cd DBDIR AMX shared analyze 2 Type the following command to roll back the upgrade witmupgrade x profilemanagers DM_TEST PM xml r f scans baseline xml where x profilemanagers DM_TEST_PM xml Specifies the name and location of the output file that resulted from the assessment of the DM_TEST_PM profile manager r Indicates that the purpose of this command is to perform a rollback f scans baseline xml Specifies the name and location of the baseline file to use as input for this command 3 Restart the Windows endpoint The rollback option can also be used to roll back an endpoint upgrade or a profile upgrade independently By rolling back the profile manager upgrade you roll back all upgrades profile manager profile and endpoint in one step SQL1_OpenRequest status 79 return code occurs during when upgrading an agent The return code SQL1_OpenRequest status 79 occurs in the agent log when the application support is added during an upgrade This return code results from an attempt to delete a table entry that does not exist in the table When you add application support for a V6 1 agent the return code is expected behavior because the agent application support data does not exist in the table An IBM Tivoli Monitoring 6 1 upgraded agent is not configured to po
324. ne of the following 1 Install products to the local host 2 Install products to depot for remote deployment requires TEMS 3 Install TEMS support for remote seeding 4 Exit install Please enter a valid number 1 Initializing Do you agree to the terms of the license in file LICENSE TXT on the CD l yes 2 no 3 to read the agreement now choosing 2 exits the installation process 1 2 3 1 You are not entitled to install from this media Setup will not proceed root sdogoff ud_062000000_116263 Many interactive elements have been removed from the agent bundle output of the tacmd exportBundle o LOCAL command in order to optimize it for remote transmission and silent execution using software distribution technologies In order to install the exported agent bundle the silentInstall sh or silentInstall bat script available in the destination directory should be run instead The addbundles command fails You receive a message that says that an error occurred attempting to add the specified bundle to the depot You should upgrade the monitoring server before upgrading agents The exportBundles command does not work for patches This command should not be used to install patches Chapter 11 Command troubleshooting 191 192 IBM Tivoli Monitoring Troubleshooting Guide Chapter 12 Universal Agent troubleshooting This chapter describes problems that might occur with the Universal Agent Preliminary
325. ng agent troubleshooting 157 Installing pre v6 2 1 Monitoring Agent for Windows OS onto a v6 2 1 or later monitoring server inadvertently unconfigures the monitoring server When a pre v6 2 1 Monitoring Agent for Windows OS is installed onto a system that has a v6 2 1 or later monitoring server the agent installation inadvertently unconfigures the monitoring server This results in the monitoring server being left in a stopped state Settings for the monitoring server remain preserved in the monitoring server ini and env files Attempts to manually start the monitoring server fail with the message Unable to start tems service Check the event log Monitoring Agents for Windows OS that are v6 2 1 and higher can be installed onto a monitoring server with no adverse side effects IBM Tivoli Monitoring Application Agents regardless of version can also be installed onto a monitoring server with no adverse side effects If this issue is encountered the monitoring server should be re configured and restarted from Manage Tivoli Enterprise Monitoring Server or through the CLI Settings for the monitoring server from before the installation preserved in the monitoring server ini and env files will automatically be used The tacmd executeaction command fails for certain take actions For certain action commands the tacmd ExecuteAction command fails with the following error message KUICXA029E The execution of the take action ActionName
326. nitoring agent troubleshooting 163 e Must start with an alphabetical character a z A Z e Can contain 1 30 additional alphanumeric characters a z A Z 0 9 e Can also contain the following special character _ e Must end with an alphanumeric character BAROC file is missing for IBM Tivoli Monitoring 5 x Endpoint situations A BAROC file does not exist for the IBM Tivoli Monitoring 5 x Integration Agent that defines the class definitions for the following situations e KTM_Health_Indication e KTM_Missing_Metadata e KTM_Resource_Model_Statu As a result when any of these situations are triggered the forwarded situation events are not successfully processed by the EIF receiver The BAROC file for the Integration Agent is no longer supplied with IBM Tivoli Monitoring Generate the BAROC files as needed by using the BAROC file generator tool found here http catalog lotus com topal NavCode 1TW10TM43 The target hostname platform and version information is not displayed for the deploy status in the CLI or the workspace Target hostname platform and version information is not displayed for the deploy status in the CLI or the workspace For group commands the target hostname platform and version information are not displayed However the transaction ID for the grouping command can be used to query all the transactions started by this group command Here is an example of a group with one member echo Creating DEPL
327. nitoring log set a Click File gt Import Log File b Create a new log set c Type the name for the log set For example you could type the following text IBM Tivoli Monitoring Log files for server xxxx d Click Add e In the Name Filter window to limit the list of log files to the IBM Tivoli Monitoring log files type Discovery f Select the type of log file you are adding to the log set g Type the name of the log file on your local system Ensure the type of log file matches the log file you specified h Enter the correct version of the IBM Tivoli Monitoring product that corresponds to the log file Refer to the Log and Trace Analyzer online help for additional options i To add the log file to the log set click OK For every log file you want to include in the log set repeat step e Best practice The first time you create the log set you save time later by including every log file you want to include in the log set e Reuse an existing IBM Tivoli Monitoring log set a Select File gt Import Log File b Select an existing Log Set Definition from the drop down list of defined log sets c If necessary change the contents of the log set definition You can add edit or remove from the list of log files in the log set d To indicate the file should be imported to the log set select the checkbox next to the log file e To import the log files click Finish Chapter 4 Tools 61 62 You can cre
328. nning to allow for conditions that initially caused the problem to be corrected Based on what step was running when the failure occurred you might need to run one or more of the following jobs in RKANSAM e KDSRSTRO Chapter 5 Installation and configuration troubleshooting 97 98 KDSRSTRN KDSRSTRV Use the following VSAM migration steps La Run RNMEVSAM ALLCVSAM step to e Rename the previous version s VSAM file to a backup VSAM name e Allocate the new V610 VSAM file based on the new format Note This step applies to VSAM files containing data that do not need to be converted OR Edit and submit KDSRSTRN to restore to files to the original state Run KDSRCONV step to e Allocate the new V610 VSAM libraries using temporary XKDS VSAM names e REPRO the previous version VSAM file data RKDS to flat file e Convert the fields to ASCII and resize columns accordingly REPRO the converted flat file to the new XKDS VSAM libraries e Write out the output from the conversion OR Edit and submit KDSRSTRV to restore a file to original state Run RNMEEIBO step to rename the previous version s VSAM files RKDS to backup VSAM names BKDS if the KDSRCONV step is successful OR Edit and submit KDSRSTRO to restore file to original state Run RNMEEIBN step to rename the new V610 XKDS VSAM libraries to the runtime RKDS VSAM names if the RNMEEIBO step is successful OR Edit and submit KDSRSTRN to restore to original
329. not have the time span you must now go back in and change it manually HEAPDUMPs and JAVACore files are placed on the desktops when running Tivoli Enterprise Portal in browser mode The Tivoli Enterprise Portal client uses the IBM Java Plug in which is automatically installed on your computer with the Tivoli Enterprise Portal Adjust the IBM Java Plug in properties if performance is slow or your workstation receives HEAPDUMPs and JAVACore files an out of memory condition when you are logged on Make the following adjustments to correct this problem e Increase the Java heap size settings Set the minimum heap size to 128 MB Set the maximum heap size to 256 MB If you continue to experience problems increase the maximum setting in increments of 64 MB until the symptoms disappear e When memory requests by the Tivoli Enterprise Portal cannot be satisfied from Java heap storage the JVM performs garbage collection to reclaim free memory If the Java heap size settings are too small the amount of time it takes to perform garbage collection becomes excessive resulting in high CPU utilization and poor response time In some cases Java HEAPDUMPS and JAVACore files are placed on user desktops indicating an out of memory condition Increasing the Java heap size parameters over the default values helps you avoid these problems e If you observe symptoms of heap memory exhaustion after changing the heap size settings to the suggested values i
330. ns to save the data then the dataset is initialized and marked as empty If the BACKUP or EXPORT was specified but the maintenance jobs fail to do their job the recording would stop in the persistent datastore In this case datasets are taken off line until there are no more available datasets for reading or writing Note If you allocate persistent datastores that fit more than 24 hours worth The agent initializes and writes as much data as it can fit in the persistent datastores 24 hours or more The Tivoli Enterprise Portal for short term history display only pulls up 24 hours worth of data The warehouse can archive ALL the data in the persistent datastores regardless if it has been 24 hours or more worth of data If you create a Tivoli Enterprise Portal query that is over 24 hours then the warehouse fulfills that request regardless if the data is in the online persistent datastores Also because the agent or Tivoli Enterprise Monitoring Server reads the entire persistent datastores at initialization time you should not allocate very large persistent datastores to potentially store more than 24 hours That increases the Tivoli Enterprise Monitoring Server and agent startup time As mentioned above the agent writes to it but the Tivoli Enterprise Portal only displays 24 hours from it The warehouse processing reads all the data in the persistent datastores 24 hours or more but there is a trade off in Tivoli Enterprise Monitoring S
331. nstalldir C IBM ITM backupdir C IBM ITM Backup MIA 1C IBM ITM Insta lITM MiaInstall log issued 7 9 2008 10 51 23 OnMiaMoving Install for Feature KIWWICNS Tivoli Enterprise Portal Server Extensions successful RC 1073741819 7 9 2008 10 51 32 OnMiaMoving Install for Feature KIWWICNS Tivoli Enterprise Portal Server Extensions may have failed please check Checking the CANDLE_HOME MiaInstal 1og file you find errors similar to 2008 07 09 10 48 35 D Installing file CNPS classes cnp jar gt C IBM ITM CNPS classes cnp jar 2008 07 09 10 48 35 C EXTRACTFILE src CNPS classes cnp jar dest C IBM ITM CNPS classes cnp jar 2008 07 09 10 48 37 E ERROR File extraction failed CNPS classes cnp jar gt C IBM ITM CNPS classes cnp jar 3 2008 07 09 10 48 37 S Internal Error trying rollback 2008 07 09 10 48 37 S Attempting to stop child process 2008 07 09 10 48 51 S Rollback successful The return code 3 indicates the software could not create a file due to either an issue with file permissions or a corruption on the hard drive First check the file permissions of the destination file Running the Windows chkdsk and defragmentation programs might resolve issues with corruption on the physical hard drive Problems that are cleared by rebooting the Windows system There are a set of problems that are cleared by rebooting the Windows system that has been installed or upgraded e Situations do not fire
332. nterprise Monitoring Services panel shows the Tivoli Enterprise Monitoring Server as running as a 32 bit application but my agents are shown as running as 64 bit applications Tivoli Enterprise Monitoring Server does not release memory after running a large SQL query SQL queries with more than 200 OR predicates do not complete Tivoli Enterprise Monitoring Servet aborts unexpectedly when si the telnet session used to start it KCIIN0084E Timeout appears while waiting for Tivoli Enterprise Monitoring Server to start on AIX 5 3 Troubleshooting Tivoli Enterprise Monitoring Server problems on z OS systems Receive Program KDFCINIT and Program FAXCMON messages The Tivoli Enterprise Monitoring Server start task CANSDSST default encountered error message KLVST044 LOADLIST MEMBER NOT FOUND IN RKANPAR DATASET KDSLLIST KppLLIST KLVST001 CANDLE ENGINE INITIALIZATION ERROR S ABEND U0012 in the RKLVLOG at startup If the zOS CT Engine stops the Tivoli Enterprise Monitoring Server must be recycled KDS Parameters not generated from the batch parm deck Cannot encrypt text A call to CSNBSYE failed Cannot encrypt contents of keyfile The error KLVST005 MVS JOBSTEP AUTHORIZATION REQUIRED KLVST001 CANDLE ENGINE INITIALIZATION ERROR S ABEND U0012 CSV019I Required module KLVSTWTO not accessed it is not APF Authorized RKANMODL CSV028I ABEND 306 0C occurs in the z OS Tivoli Enterprise Monitoring Server RKLVLOG during startup Th
333. ntify and SQL query Yes See Corrective action 1 Corrective actions Note The following corrective actions are available only for products that include the 6 2 0 TIV ITM_ORA LA0001 limited availability fix and higher 1 Disable the problematic cursors by setting an environment variable On Windows COLL_DISABLE_CURSORS On UNIX and Linux db_extparms 32 IBM Tivoli Monitoring Troubleshooting Guide Table 21 Correcting Oracle DB Agent problems continued Set the environment variables as follows On Windows Launch Manage Tivoli Enterprise Monitoring Services Right click the row that contains the name of the monitoring agent whose environment variables you want to set Select Advanced gt Edit Variables in the pop up menu If the agent is running accept the prompt to stop the agent The list dialog box is displayed When only the default settings are in effect no variables are displayed The variables are listed after you override them Override the variable settings as follows a Click Add b Select the COLL_DISABLE_CURSORS in the Variable pull down menu If it is not there you can add it c Type a value in the Value field d Select OK twice e Restart the agent On UNIX and Linux Use a text editor to enter a new value for the db_extparms variables in the hostname_or_instance_name cfg file in the install_dir config directory The cursors that are listed below take longer
334. nts from this utility This method of modifying trace settings on an IBM Tivoli Monitoring component is the most efficient method since it allows you to do so without restarting the component Settings take effect immediately Modifications made this way are not persistent Note When the component is restarted the trace settings are read again from the env file Dynamically modifying these settings does not change the settings in the env files In order to modify these trace settings permanently modify them in the env files How to turn tracing on In order to use this utility you need to know a local log on credential for the machine This method uses the IBM Tivoli Monitoring Service Console The Service Console is accessed using a web browser Access the utility by using the following link http hostname 1920 where hostname is the hostname or IP address of the system where the IBM Tivoli Monitoring component is running The utility then appears with information about the components that are currently running on this machine For example the component Tivoli Enterprise Portal Server shows as cnp the Monitoring Agent for Windows OS shows as nt and the Tivoli Enterprise Monitoring Server shows as ms Select the link below the component for which you want to modify the trace settings In the previous view if you want to modify tracing for the Tivoli Enterprise Monitoring Server you select the IBM Tivoli Monitoring Service
335. nts of Tivoli Management Services e Quick Start Guide GI11 8058 Introduces the components of IBM Tivoli Monitoring e Installation and Setup Guide GC32 9407 Provides instructions for installing and configuring IBM Tivoli Monitoring components on Windows Linux and UNIX systems e Program Directory for IBM Tivoli Management Services on z OS GI11 4105 Gives instructions for the SMP E installation of the Tivoli Management Services components on z OS e Configuring the Tivoli Enterprise Monitoring Server on z OS SC32 9463 Gives detailed instructions for using the Configuration Tool to configure Tivoli Enterprise Monitoring Server on z OS systems Includes scenarios for using batch mode to replicate monitoring environments across the z OS enterprise Also provides instructions for setting up security and for adding application support to a Tivoli Enterprise Monitoring Server on z OS e Administrator s Guide SC32 9408 Describes the support tasks and functions required for the Tivoli Enterprise Portal Server and clients including Tivoli Enterprise Portal user administration Copyright IBM Corp 2005 2010 251 252 e High Availability Guide for Distributed Systems SC23 9768 Gives instructions for several methods of ensuring the availability of the IBM Tivoli Monitoring components e Tivoli Enterprise Portal online help Provides context sensitive reference information about all features and customization options of the Tivoli Enterp
336. o a csv file and then work with that to extract the hex data The affinity is the first 32 characters Columns 33 34 are the product version code Columns 35 43 are version flags 6 Leave this workspace up but when you switch away from this workspace be sure not to save it since it was only a temporary change 7 Open the installdir cnps affinity properties Windows or the installdir arch cq data affinity properties Linux and UNIX file Confirm that 00080000000000000000000000000000 corresponds to the Monitoring Agent for Windows OS 8 Next search for tags AFF_ Now you can see that Affinity 00080000000000000000000000000000 represents the AFF_NT_SYSTEM value Using the above process you can determine the correct settings for the agents where you need the KFW_TOPOLOGY_CLUSTER_LIST configuration 9 You should recycle the portal server at this point and look at the display In many cases after you have completed the steps above you have finished this task However there are a few cases where additional configuration is necessary The host address is normally used for machine identification however with KFW_TOPOLOGY_CLUSTER_LIST set the first part of the Origin Node is used Primary is ignored That first part defaults to the TCP IP hostname but it is replaced if CTIRA_HOSTNAME is set This gives you control over deciding where an agent is positioned Configuring an agent to have a specific CTIRA_HOSTNAME value manages ca
337. o impact on either eWAS or IBM Tivoli Monitoring functionality Monitoring agents fail to start after agent support or multi instance agents are installed Monitoring agents on an IBM Tivoli Monitoring V6 2 1 or later managed system that have an unsupported system GSKit version installed might fail to start after an IBM Tivoli Monitoring V6 2 Multi Instance Agent or IBM Tivoli Monitoring V6 2 Agent Support is locally installed The installer used by both IBM Tivoli Monitoring V6 2 Multi Instance Agents including fix packs and IBM Tivoli Monitoring V6 2 Application Support causes monitoring agents on an IBM Tivoli Monitoring V6 2 1 managed system or later to revert back to using the system GSKit instead of the IBM Tivoli Monitoring embedded GSKit This issue occurs on local installations only Remote installation remote deploy does not have this issue If a system GSKit is installed on the managed system at a level supported by IBM Tivoli Monitoring the monitoring agents continue to operate normally Monitoring agents might fail to start however if all of the following conditions are met e The managed system does not have a system GSKit installed or the system GSKit is at a version not supported by IBM Tivoli Monitoring V6 2 1 or later e The agent is configured to use secure communications IP SPIPE rather than normal communication IP PIPE If agents on a managed system fail to start after an IBM Tivoli Monitoring V6 2 Mult
338. oftware Selecting Cancel closes the window and the installation cannot complete To install the software without this problem map the network drive and run the setup exe file from a DOS prompt The error Could not open DNS registry key occurs This message is informational an no action is required The Windows agent reports the fact that it could not find a registry entry for the DNS Server Event Log indicating that the DNS Server Event Log is not installed You can stop all situations and recycle the Tivoli Enterprise Monitoring Server with no situations in ACTIVATE AT STARTUP to prevent this message being written to the trace log Agent not connecting to Tivoli Enterprise Monitoring Server If you find a message similar to Unable to find running TEMS on CT_CMSLIST in the Log file the Agent is not connecting to the Tivoli Enterprise Monitoring Server Confirm the following to ensure connectivity e Multiple network interface cards NICs exist on the system e If multiple NICs exist on the system determine which one is configured for the monitoring server Ensure that you specify the correct hostname and port for communication in the IBM Tivoli Monitoring environment InstallShield displays the error 1607 Unable to install InstallShield Scripting Runtime during installation on Windows from a network mounted drive When running setup exe on Windows from a network mounted drive the following error occurs InstallShield 1607
339. ole can also be launched with the https protocol by connecting via the https protocol and port 3661 You can perform operations on a specific IBM Tivoli Monitoring process by selecting the IBM Tivoli Monitoring Service Console associated with a service point name The IBM Tivoli Monitoring Service Index has links to service consoles for the components installed on the computers Now when you go to the Service Index you will also see links to the Agent Service Interface Use the Agent Service Interface to get reports for an installed agent whether it is a Tivoli Enterprise Monitoring Agent or Tivoli System Monitoring Agent After logging into the local operating system you can choose reports of agent information private situations private history and attribute descriptions and current values You can also make a service interface request using provided XML elements Starting the IBM Tivoli Monitoring Service Console Use the following procedure to start the IBM Tivoli Monitoring Service Console 1 Start Internet Explorer V5 or higher 2 In the Address field type the URL for the Tivoli Enterprise Portal browser client installed on your Web server The URL for the Tivoli Monitoring Services Web server is http hostname 1920 hostname Specifies the computer where the Tivoli Enterprise Portal Server was installed If the IBM Tivoli Monitoring Service Console is not displayed Chapter 4 Tools 53 54 on page 54 3 Click the IBM Ti
340. oli Monitoring Installation and Setup Guide There you can find information for environment wide capacity planning You can find agent specific capacity planning information in the user guide for the specific monitoring agent Warehouse Proxy agent failed to export data The ODBC connection enables the Warehouse Proxy agent to export data to the warehouse database The WAREHOUSELOG table lets you know how many exports succeeded and how many failed because of an ODBC error or a TIMEOUT issue See the IBM Tivoli Monitoring Installation and Setup Guide for more information about the WAREHOUSELOG table and configuring the Warehouse Proxy agent There are ORACLE or DB2 errors in the khdras1 log file The following errors can occur in the khdras1 10g if the globalization system environment variable is not set correctly ORACLE error Oracle ODBC OraJORA 01461 can bind a LONG value only for insert into a LONG column 1 Set the environment variable NLS_LANG AMERICAN_AMERICA AL32UTF8 as a system environment on the Windows computer on which the Warehouse Proxy is installed 2 Restart the Windows computer so that the Warehouse Proxy windows service recognizes the change DB2 error SQL0302N The value of a host variable in the EXECUTE or OPEN statement is too large for its corresponding use SOLSTATE 22003 sqlstate 22003 1 Set the environment variable DB2CODEPAGE 1208 as a system environment on the Windows computer where the Warehous
341. ollowing tacmd commands tacmd histlistproduct tacmd histlistattributegroups tacmd histviewattributegroup tacmd histConfigureGroups tacmd histViewAttributeGroup tacmd histUnconfigureGroups tacmd histStartCollection tacmd histStopCollection The following tables include possible causes and corresponding solutions for the problems described in this section Table 11 Correcting Warehouse Proxy connection problems There is a connection problem to the Warehouse Proxy agent Possible causes Short term historical data is stored at the location of the monitoring agent see left column below or monitoring server see right column below and that component cannot connect to the Warehouse Proxy agent Diagnostic actions e Make sure the Warehouse Proxy agent is running e Look for export failures to the Warehouse Proxy agent in either the monitoring agent RAS1 log or monitoring server RAS1 log Depending on where the error is found see the coordinating column below Monitoring agent Monitoring server 1 Verify that the correct socket connection is being 1 Verify that the connection between the monitoring used server and Warehouse Proxy agent is not being 2 Verify that the monitoring agent is connected to the stopped by a firewall Warehouse Proxy agent 2 Verify that the correct port is being used for each component 22 IBM Tivoli Monitoring Troubleshooting Guide Table 11 Correcting Warehouse P
342. om the group are due to the original situation s membership within that group When you create another situation from one of these member situations that operation does not allow for the new situation being part of that same group Copy only the distributions that are directly assigned A changed situation name does not show up If you change the name of a situation the Tivoli Enterprise Portal and the listsit command does not show the name change Once the situation is created it has to be referenced by its original name New agents do not display in the Tivoli Enterprise Portal navigator The Tivoli Enterprise Portal navigator does not update automatically when an agent is installed or uninstalled from to the managed system You must refresh the navigator to display changes Do the following to refresh the navigator from the Tivoli Enterprise Portal menu e Press the F5 key on your keyboard OR e Click View gt Refresh 176 IBM Tivoli Monitoring Troubleshooting Guide An agent displays unavailable in the Tivoli Enterprise Portal The agent is not online Do the following to ensure the agent is online e Check the agent log for data communication errors e Check the managed system status in the Tivoli Enterprise Portal e Ensure that the agent process started e Check the Tivoli Enterprise Monitoring Server kfwras1 1og for errors e Check the Tivoli Enterprise Monitoring Server kfwras1 1og CTIRA_HOSTNAME has no eff
343. on response file to apply application support on the monitoring server the portal server or the Tivoli Enterprise Portal fails and displays a failure message Error java lang ArrayIndexOutOfBoundsException 0 Additionally the resulting application support files contained in the support package are not installed Using the Application Support Installer with the Silent Installation Response file option is not supported The recommended mechanism for the installation is using the GUI interface Unable to run gsk7ikm exe Unable to run c IBM ITM GSK7 bin gsk7ikm exe as it fails with the following error Failed to parse JAVA _HOME setting On UNIX and Linux systems complete the following steps 1 Open console 2 Get the IBM Java location by running the following script CANDLEHOME bin CandleGetJavaHome 3 Export variable JAVA_HOME to point to the IBM Java path For 64bit gsk7ikm has to be 64bit java 4 Check the path for a local gskit This path is CANDLEHOME config gsKit config GskitInstallDir points to a 32bit GSKit and GskitInstallDir_64 points to a 64bit GSKit IBM Tivoli Monitoring Troubleshooting Guide 5 Run GSKit key manager by running the following depending on your system setup GskitInstallDir bin gsk7ikm_32 32bit on HP GskitInstallDir bin gsk7ikm 32bit on Linux Aix or Solaris GskitInstallDir _64 bin gsk7ikm_64 64bit On Windows systems 1 Run cmd 2 Get the IBM Java location by running the foll
344. on support consistency checks situations distribution warehouse analysis etc A Windows environment is required This tool can be run in stand alone mode by pointing to the Tivoli Enterprise monitoring server on any platform You may run the ITMSUPER tool from a Windows client without having other IBM Tivoli Monitoring software installed Chapter 4 Tools 65 66 IBM Tivoli Monitoring Troubleshooting Guide Chapter 5 Installation and configuration troubleshooting This chapter contains the following sections which provide information about problems that might occur during installation upgrading from previous versions and uninstallation of the product and product components e Frequently asked questions on page 67 e General installation problems and resolutions on page 69 e Windows installation problems and resolutions on page 80 e UNIX based system installation problems and resolutions on page 84 Troubleshooting z OS based installations on page 88 Uninstallation problems and workarounds on page 100 Frequently asked questions General installation frequently asked questions The following table lists general installation frequently asked questions Table 26 General frequently asked questions Question Answer Are fix packs required if a user migrates Candle monitoring agent to IBM Tivoli Monitoring Fix packs for CNP196 are delivered as each monitoring agent is migra
345. onitoring Troubleshooting Guide NO oS support isa The Log and Trace Analyzer plug in is included in the list of plug ins for Common Component Tools After installation of the Log and Trace Analyzer is complete start the ISA From the list of tasks click Tools From the list of products click IBM Tivoli Monitoring From the list of tools for IBM Tivoli Monitoring click Log and Trace Analyzer The Log and Trace Analyzer should start working To import the IBM Tivoli Monitoring log files to the Log and Trace Analyzer complete the following steps le Copy the relevant log files from the Monitoring servers to the system where you installed the IBM Support Assistant workbench Put the log files for each server in a unique directory For example c ITM logs serverXXX Import the IBM Tivoli Monitoring log files The Log and Trace Analyzer organizes related log files into log sets Log sets can be used to import and analyze a set of related log files This facility is used to organize and import your IBM Tivoli Monitoring log files Log set definitions provide information to the Log and Trace Analyzer specifying where log and trace data reside and what kind of data to gather from local and remote systems The Log and Trace Analyzer allows you to import pre defined log sets that contain the necessary path information required for retrieving log files on demand Use one of the following procedures e Create the initial IBM Tivoli Mo
346. only return the grouping function results and any columns used in the grouping This predicate is looking for a single row but a grouping function is an aggregate of the grouped rows Tivoli Enterprise Monitoring Server application support will complete all seeding functions but may crash as the program is exiting The Tivoli Enterprise Monitoring Server seeding program that adds Tivoli Enterprise Monitoring Server application support completes all seeding functions but may crash as the program is exiting This crash has only been observed rarely during product testing The IBM Tivoli Monitoring configuration tool checks the output produced by the seeding functions and it reports that the Tivoli Enterprise Monitoring Server application support was added successfully Since all seeding functions were completed the Tivoli Enterprise Monitoring Server tables with application support are correct and not corrupted A core or dump file might be created during the program crash Creation of a core or dump file usually depends on if the system has been configured to save crash information However even if the system is configured to save crash data this particular crash might not produce a core or dump file The Tivoli Enterprise Monitoring Server seeding output files contains information about the crash An operating system message indicates the condition that caused the crash A sample crash message would be signal 11 SIG1_SIGSEGV 29 0B000000000
347. ons for this command are mutually exclusive If you enter more than one you will receive a message that the second option entered is repeating For example C IBM ITM bin gt tacmd viewuser u sysadmin w mypassword a v KUICO2022E The command did not complete because v option is repeating Commands fail when a situation name consists of characters When using the character while executing commands you must use the escape character This is a general command line restriction For example root vger ksh tacmd createsit s abc 123 b Linux_Process High Cpu 190 IBM Tivoli Monitoring Troubleshooting Guide tacmd addSystem fails if agent already exists When using the command tacmd addSystem to install an existing instance of the agent The expected result would be a message saying that the agent is already installed The actual results are that a message does not appear and the installation does not overwrite the existing agent Installing an exported agent bundle using install sh causes an error Attempting to execute the interactive install sh script found within the output directory of the tacmd exportBundle o LOCAL command results in the following error message root sdogoff ud_062000000_116263 install sh INSTALL Enter the name of the IBM Tivoli Monitoring directory default opt IBM ITM ITM home directory opt IBM ITM already exists OK to use it 1 yes 2 no 1 is default Select o
348. onsole event synchronization installation e Use the IBM Tivoli Enterprise Console Java Console to make any configuration changes to consoles and associated operator and event groups e Connect to a different IBM Tivoli Enterprise Console server using the embedded viewer From the desktop client 1 Log off the Tivoli Enterprise Portal 2 Log on the Tivoli Enterprise Portal 3 Log into a different IBM Tivoli Enterprise Console server From the browser client Recycle the browser e For a listing of product messages refer to the document IBM Tivoli Monitoring Messages T EC ITM Synchronization Tasks By default all situation events that originate from a recycled Tivoli Enterprise Monitoring Server are CLOSED As a result you do not need to run any of the three tasks that reside in the T EC ITM Synchronization Task Library Copyright IBM Corp 2005 2010 209 Changing the TCP IP timeout setting on your event server If the Situation Update Forwarder cannot reach a monitoring server to send an update depending on the TCP IP settings for the computer where your event server is running it could be up to 15 minutes before the Situation Update Forwarder tries to connect to the monitoring server again This might occur if your event server is running on an AIX Solaris or HP UX computer Use the following steps to change the TCP IP timeout for your computer On AIX run the following command no o tcp_keepinit
349. or multiple sub objects on a managed system For example one situation might monitor multiple channels As a result relatively large amounts of monitoring data might be generated every time that the thresholds of the situation are triggered When TEC integration is enabled the IBM Tivoli Enterprise Console receives a buffer of monitoring data that cannot exceed 4K bytes in size In most cases this buffer size is sufficient However some situations that monitor multiple sub objects in Tivoli OMEGAMON XE for Storage on z OS can generate more than 4K bytes of data When the data generated for the situation exceeds this limit the excess data is truncated and is not presented at the IBM Tivoli Enterprise Console For situations that monitor multiple sub objects be aware that IBM Tivoli Enterprise Console is retrieving only the first 4K bytes of data Chapter 14 Event synchronization troubleshooting 225 226 IBM Tivoli Monitoring Troubleshooting Guide Chapter 15 Tivoli Common Reporting troubleshooting If you cannot display reports consider the issues and use the messages discussed in this section Notes 1 When a report fails to generate or generates incorrectly click View the report with errors to make diagnosis of the underlying problem easier 2 When you troubleshoot problems with Tivoli Common Reporting you should enable detailed logging using the instructions found in the Logging and tracing section of the IBM T
350. or numeric data Tivoli Enterprise Portal charts and graphs for a Universal Agent application are empty even though the attributes are defined as N in the Universal Agent metafile The Universal Agent N attribute type means DisplayNumeric It is a string representation of a numeric attribute that can include non numeric characters such as a floating point number or a large number with comma separators A DisplayNumeric attribute is a string in the ODI file and is not eligible for Tivoli Enterprise Portal charting and graphing If you want to use a Universal Agent attribute in a chart or graph define the attribute in the metafile as a true numeric attribute for example e C for Counter e G for Gauge Chapter 12 Universal Agent troubleshooting 195 kumstrap and kumsnetd processes do not start 196 After running itmcmd agent start um on UNIX based systems the following error messages display elsun02 itmcmd agent start um itmcmd agent installer level 350 552 itmcmd agent running sol25 jre Starting agent sd Note Universal agent process kumsnetd did not start NOte UNiversal agent process kumstrap did not start Agent Started When you check the Universal Agent RASI log file the following messages display at the bottom 16 kumOsock c 132 KUMO OpenLocalSocket bind failed for local address UDP socket 71 port 162 error 13 F kumsrcfg c 73 KUMS ReadNetConfigFromExt Network configuration file contro
351. oring product to behave erratically if the clock is not fixed Tivoli Enterprise Portal Server is unavailable When a message indicates the server is unavailable suspend further interactions until a message indicates the server is available unless you decide to end the current Tivoli Enterprise Portal session When the Tivoli Enterprise Portal Server is available again your client session is automatically reconnected and you can resume normal interactions If the server is available and the client has not reconnected after three to five minutes exit the browser restart the browser and restart a Tivoli Enterprise Portal client session Tivoli Enterprise Portal Server is not reconnecting If the Tivoli Enterprise Portal Server does not reconnect recycle the Tivoli Enterprise Monitoring Server and restart the Tivoli Enterprise Portal Server Cannot reconnect to the Tivoli Enterprise Monitoring Server This message displays after the Tivoli Enterprise Monitoring Server goes down and attempts to reconnect The Tivoli Enterprise Portal client attempts to log on once again to the Tivoli Enterprise Portal Server session After a successful logon the Tivoli Enterprise Portal client compares the Tivoli Enterprise Portal user authorities that were in effect when the original logon occurred with the current Tivoli Enterprise Portal user authorities If any permission is different you must restart Chapter 6 Connectivity troubleshooting 115
352. ortal sessions A managed system for a MIB application does not come online as expected Ensure that you imported the application metafile either through the console command IMPORT or updating your KUMPCNFG file Check your DPLOG workspace in the application UAGENT for error messages regarding the importing of the metafile The NETSUMMARY workspace does not show expected data Ensure that you can ping the default gateway or router from the host running the SNMP Data Provider A MIB workspace displays empty when you expect to see data A MIB workspace displays empty if the community name used for the SNMP query was not the correct one for a particular SNMP monitoring agent To determine that a MIB workspace is empty as a result of an incorrect community name monitor for the Authentication failure trap from the SNMP monitoring agent Assuming the SNMP monitoring agent is configured to generate traps to the host running the SNMP DP you can browse your SNMP MANAGER TRAP workspace to look for Authentication failure traps Generic trap number 4 IBM Tivoli Monitoring Troubleshooting Guide Chapter 13 Database troubleshooting This chapter provides information you need to solve problems you might experience with the database It covers data loss prevention information and descriptions of common problems and appropriate resolutions It also addresses issues you might experience with the Tivoli data warehouse and Warehouse Proxy agent
353. ory for OS monitoring agents on Tivoli endpoints When an endpoint is upgraded the Upgrade tool normally deploys the OS monitoring agent to the specified directory on the endpoint However if the CANDLE_HOME environment variable is already set on a Windows endpoint the Upgrade tool deploys the monitoring agent to the directory specified by the environment variable even if a different directory is specified in the baseline file The CANDLE_HOME environment variable is already set if any IBM Tivoli Monitoring V6 1 component is already installed on the endpoint Deploying an agent instance without providing any configuration properties causes KUICARO20E error The problem occurs after you have already successfully deployed a multi instance agent such as a database agent to a node and then tried to deploy another instance without providing any configuration properties which is an error You see this error message KUICARO20E The addSystem command did not complete because a deployment error occurred Refer to the following error returned from the server The monitoring server encountered an error while deploying the managed system The kuirasl log file may provide more information about this error If you require further assistance resolving the error contact IBM Software Support The agent received incorrect SQL The CONTEXT column was not specified and is a required parameter A correct message would tell you that required configuration
354. ostname_PC_rasljava_ HEXtimestamp nn log install _dir logs hostname_PC_java_ HEXtimestamp nn log where install_dir Specifies the directory where monitoring agent was installed PC agent HEXtimestamp Specifies the product codes for example sy for IBM Tivoli Summarization and Pruning A hexadecimal representation of the time at which the process was started nn Represents the circular sequence in which logs are rotated Ranges from 1 5 by default though the first is always retained since it includes configuration parameters Chapter 4 Tools 37 Table 22 Location of log files on the Tivoli Enterprise Monitoring Server continued Component Windows UNIX based systems IBM Tivoli install_dir logs hostname_PC install_dir logs hostname_PC_timestamp log Enterprise HEXtimestamp nn log install _dir logs hostname_PC_HEXtimestamp nn log Console Event Sa Forwarder where where install_dir install_dir Specifies the directory Specifies the directory where Tivoli Enterprise where Tivoli Enterprise Portal Server was installed Monitoring Server was hostname ia Specifies the name of the system hosting the PC Specifies the product code product ms for Tivoli Enterprise PC Specifies the product code cq for the Tivoli Monitoring Server Enterprise Portal Server HEXtimestam P A Popiel ce i i i A decimal representation of the time at which the representation of the time process was started at
355. out Copyright IBM Corp 2005 2010 vi usr lib libc a usr lib libcrypt a Jusr lib libcurses a usr 1ib n1s loc en_US usr lib libi18n a usr lib libiconv Core files If the system stops on UNIX based systems collect the core file from the directory where the binary file that the process belongs to is stored For example if the failing process is the portal server process KfwServices the core is created in the opt IBM ITM_61 archtype cq bin directory To retrieve information about where the core file is created use the errpt a command This command shows a summary information about the most recent crashes and also the location of the core file LABEL CORE_DUMP IDENTIFIER A63BEB70 Date Time Tue Jun 30 15 38 47 DFT 2009 Sequence Number 1229 Machine Id 0056536D4C00 Node Id nc114062 Class S Type PERM Resource Name SYSPROC Description SOFTWARE PROGRAM ABNORMALLY TERMINATED Probable Causes SOFTWARE PROGRAM User Causes USER GENERATED SIGNAL Recommended Actions CORRECT THEN RETRY Failure Causes SOFTWARE PROGRAM Recommended Actions RERUN THE APPLICATION PROGRAM IF PROBLEM PERSISTS THEN DO THE FOLLOWING CONTACT APPROPRIATE SERVICE REPRESENTATIVE Detail Data SIGNAL NUMBER 11 USER S PROCESS ID 32248 FILE SYSTEM SERIAL NUMBER 10 INODE NUMBER 655367 PROCESSOR ID 0 CORE FILE NAME opt IBM ITM_61 aix533 cq bin core 6 IBM Tivoli Monitoring Troubleshooting Guide PROGRAM
356. owing option LDAP_DEBUG 65519 Receive a seeding failed message Before you seed a remote monitoring server you must ensure that the hub monitoring server is running However if you receive this message start the hub monitoring server and then manually seed the support using the itmcmd support command High monitoring server CPU after restarting with Warehouse Proxy Agents configured A remote monitoring server process incurs significant CPU utilization or percentage increases when any IBM Tivoli Monitoring Warehouse Proxy Agents have been started in the hub monitoring server environment Due to an issue in the monitoring server KRANDREG module the remote monitoring server can go into a loop making continuous calls to the IBM Tivoli Monitoring Global Location Broker facility This loop can happen whenever the hub monitoring server Global Location Broker contains EXACTLY 50 entries relating to the IBM Tivoli Monitoring Warehouse Proxy Agent The Global Location Broker is the hub monitoring server facility that shows which remote monitoring server or Warehouse Proxy Agents have been registered in this environment The total number of entries registered in the hub monitoring server Global Location Broker from the Warehouse Proxy Agent are a combination of the following amounts e The number of IBM Tivoli Monitoring network protocols configured for the Warehouse Proxy Agent for example IP PIPE IP SPIPE Copyright IBM Corp 2005 2
357. owing script CANDLEHOME Insta ITM GetJavaHome bat 3 Set the JAVA_HOME variable that points to the IBM Java location 4 Get the GSKit location by running the following script CANDLEHOME Insta ITM GetGSKitHome bat 5 Change the directory to GSKit path bin 6 Run the gsk7ikm exe file _cq_ log files appear Some of the _cq_ 10g files are from seeding operations So exception messages are expected by design SPD Installing a bundle on the wrong operating system architecture or kernel When you attempt to install a bundle on a system that does not correspond to the correct binaries for example installing a 32 bit bundle on a 64 bit system or installing a 2 4 kernel level bundle on a 2 6 kernel level system Look at the logs Software Package Block SPB logs are located in the temporary directory of the system tmp for UNIX or temp for Windows These will show that GSKit could not be installed To identify the right bundle for a particular system the generated Software Package Definition SPD file uses the naming convention product_code interp spd The interp tells you in which operating system architecture or kernel the bundle can be installed Installing a Software Package Block SPB on top of an existing running IBM Tivoli Monitoring agent When you attempt to install another IBM Tivoli Monitoring agent bundle using Tivoli Configuration Manager TCM or the Tivoli Provisioning Manager TPM on a system that
358. pc ud 130 IBM Tivoli Monitoring Troubleshooting Guide 4300E020 0000 864 knt10agt cpp 243 getBinaryPath Error retrieving binary path for process ID 8 4300E021 0000 864 knt10agt cpp 243 getBinaryPath Error retrieving binary path for process ID 8 This error can occur when the Tivoli Enterprise Monitoring Server is restarted and the agent displays offline even though its running because the agent did not register a heartbeat yet Agents display offline in the Tivoli Enterprise Portal but still fire situations and the agent logs are report that they are running This error can occur if the agent names in the group identified contained embedded spaces Agent names cannot contain embedded spaces Edit the agent names to remove the spaces The CTIRA_HOSTNAME environment variable on the agents must include a specific definition for correct agent host names The Tivoli Enterprise Portal displays erratic sorting behavior when removing more than one managed system simultaneously The Tivoli Enterprise Portal display erratic sorting behavior when you remove multiple managed systems at the same time after clicking Remove Offline Systems menu item from the Managed System Status view Allow the display to finish updating before attempting another operation Multiple events that occur at the same time are loaded too slowly Manually set the variable KFW_CMW_EVENT_SLEEP in cq ini on Linux or kfwenv on Windows to less than 10 seconds
359. plication version suffix You then decide to run um_cleanup to reset the DISKMONITOR version back to 00 After completing the cleanup process the Navigator tree still shows workspaces for each of the five original attribute groups even though the metafile contains only three attribute groups This mismatch is caused by the fact that the portal server saves workspace queries in the KFWQUERY table of the portal server database which is not updated by the um_cleanup script Therefore the original 00 version of the queries which knows about the five original attribute groups is still being used when you view the DISKMONITORO0 application If you determine that you need to delete one or more portal server generated queries for your Tivoli Universal Agent applications there is a Tivoli Universal Agent provided script called um_cnpsCleanup bat which is installed on Windows computers that demonstrates how to perform the delete The script is very short and uses only the following command kfwsqlclient d TEPS2 e delete from kfwquery where id like zkum For a Windows based portal server this command is entered from the IBM ITM CNPS directory The command assumes that the portal server database is using the default data source name of TEPS2 but you can change it if you have configured a different data source name Chapter 4 Tools 55 Note that this command deletes all portal server generated Universal Agent queries which always begi
360. port_number cgi bin tec_help pl occurs after clicking the Information button on the IBM Tivoli Enterprise Console 224 Situations return to acknowledged 224 tacmd refresh TECinfo t all shows no results on console 225 Pre existing situations do not display i in the Tivoli Enterprise Portal after performing an initial installation of IBM Tivoli Monitoring v6 2 225 Incomplete event data from a situation is forwarded to IBM Tivoli Enterprise Console 225 Chapter 15 Tivoli Common Reporting troubleshooting 227 Locations of log files 3227 You are missing drivers after the Tivoli Conimen Reporting installation e ee ae de os LBZ You receive message CTGTRW1000E a xt is 228 The report fails to generate because the SQL query was not valid 228 Message SQL Statement dues jetait a ResultSet object displayed 229 Your report fails to generate with unexpected error messages displayed 229 The generated report eee the message SOL Error 230 The report fails with a SOLSTATE 22003 arithine d overflow error 230 No data is plotted in graph or some columns i in the table are blank 231 The generated report displays the message The requested data is not available s a 232 You receive the message serverName is unlnown host i a a DOL You receive the message Empty Data Set 292 Appendix A IBM Tivoli
361. pplicable Log files collect these from failing systems You can collect all logs or logs of a certain type such as RAS trace logs or message logs Messages and other information displayed on the screen Application information this would be for the application you are monitoring such as DB2 or SAP This information would include the version number and patch level and a sample application data file if you are monitoring a file Operating system version number and patch level Version number of the following members of the monitoring environment IBM Tivoli Monitoring Also provide the patch level if available Monitoring Agent version number Tivoli Enterprise Portal Select About Tivoli Enterprise Portal from the Help menu Chapter 2 Collecting and reviewing troubleshooting data 7 Note The version number of the Tivoli Enterprise Portal and the Tivoli Enterprise Portal server must always be synchronized 8 IBM Tivoli Monitoring Troubleshooting Guide Chapter 3 Solving common problems The following sections give troubleshooting instructions for the most common problems encountered by customers using the IBM Tivoli Monitoring product or the components of Tivoli Management Services e I am trying to find out what software is supported on page 11 e Workspaces are missing or views are empty on page 12 Status of a monitoring ag tacmd command on page 15 e Tivoli Enterp
362. pplying IBM DB2 V8 1 Fix Pack 10 also known as Version 8 2 Fix Pack 3 you encounter an SQL0443N error if you run a DB2 Call Level Interface CLI catalog function such as SQLTables SQLColumns or SQLStatistics The following is an example of the error in a log 430F82BD 0000 3C4 khdxodbc cpp 319 initializeDatabase Connection with Datasource ITM Warehouse successful 430F82BD 0001 3C4 khdxbase cpp 250 setError Error 20 3 443 FFFFFE45 0 executing SQLTables 430F82BD 0002 3C4 khdxbase cpp 266 setError Error IBM CLI Driver DB2 NT SQLO443N Routine SYSIBM SQLTABLES specific name TABLES has returned an error SQLSTATE with diagnostic text SYSIBM CLI 805 SQLSTATE 38553 430F82BD 0002 Bind the db2schema bnd file against each database to resolve this error Run the following command from a DB2 command prompt 1 db2 connect to dbname Warehouse Database IBM Tivoli Monitoring Troubleshooting Guide where dbname Specifies the name of a database to which you want to bind the utilities or the Warehouse Database name 2 DB2 bind path_name where path_name Specifies the full path name of the directory where the bind files are located usually sqllib bnd The following examples shows the rebinding commands from the c SQLLIB bnd gt DB2 directory c SQLLIB bnd gt DB2 connect to dbname Warehouse Database c SQLLIB bnd gt DB2 bind db2ubind 1st blocking all grant public c SQLLIB bnd gt DB2
363. priate user ID is used the next time the system is rebooted Editing the startup file is no longer supported Instead you must modify the config kcirunas cfg file and then run bin UpdateAutoRun sh 1 Edit install _dir config kcirunas cfg 2 Adda section after the agent line to specify the agent or agent instance that you want to start as a specific user ID To specify the user ID to start a non instance agent or to start all instances of an agent use the following syntax lt productCode gt product_code lt productCode gt lt default gt lt user gt user_name lt user gt lt default gt Chapter 10 Monitoring agent troubleshooting 165 166 IBM Tivoli Monitoring To specify different user IDs to start different instances of an agent use the following syntax lt productCode gt product_code lt productCode gt lt instance gt lt name gt instance_namel lt name gt lt user gt user_name lt user gt lt instance gt lt instance gt lt name gt instance_name2 lt name gt lt user gt user_name lt user gt lt instance gt Where product_code 2 character product code for the agent for example ux for the Unix OS monitoring agent user_name Name of the user to run the agent as instance_namel1 Name of an instance instance_name2 Name of another instance Examples For the Unix OS monitoring agent to run as itmuser lt productCode gt ux lt productCode gt lt default gt lt user gt i tmuser lt
364. pse Help Server first stop the Tivoli Enterprise Portal Server The UNIX command option f itmemd agent f stop kf allows you to stop the Eclipse Help Server unconditionally On Windows systems reconfiguration restarts all components To reconfigure the Eclipse Help Server while the Tivoli Enterprise Portal Server is running restart the Tivoli Enterprise Portal Server along with the Eclipse Help Server Non root stopping or starting agents causes problems You might experience issues while starting or stopping agents on servers when using a non root user ID The following message might be received KCIIN1191E Cannot execute product_code stop script To avoid this situation use the root account or an account with granted required permissions itmuser group 136 IBM Tivoli Monitoring Troubleshooting Guide Root password is not accepted during non root Tivoli Enterprise Portal Server configuration While configuring the Tivoli Enterprise Portal Server when using non root user the provided root password is not validated correctly You should use root account or an account with granted required permissions itmuser group Corba user exception is included in the Tivoli Enterprise Portal Server log when creating situations When a user creates a new situation the situation name must be unique To verify that the new name is unique the software attempts to access a situation by the new name If the situation is found then t
365. r 14 Event synchronization troubleshooting This section provides descriptions of and resolutions for problems you might experience with event synchronization including the forwarding situations and the Rules Check Utility Unable to send events from user defined situations to OMNIBUS Using the tacmd editeventdest command to change the type is not updating the bufevtmaxsize The event forwarder only updates the server location and the port instead of doing a replacement if it finds an existing config file for the destination This is done in order to preserve any user specified EIF parameters Complete the following steps to solve this issue 1 Locate the data set pointed to by the RKDSEVT DDNAME 2 Edit all the evtdsnnn members and change BufEvtMaxSize 8192 to BufEvtMaxSize 4096 3 Recycle the hub monitoring server Preliminary IBM Tivoli Enterprise Console event synchronization troubleshooting Review the following list of fundamental IBM Tivoli Enterprise Console event synchronization troubleshooting tasks e Before you install the IBM Tivoli Enterprise Console event synchronization on Windows and import event forwarding functionality into an existing rule base with an absolute path you must copy setupwin32 exe to the local drive on which the rule base resides to import the IBM Tivoli Enterprise Console event synchronization functionality into that rule base Launch the copied setupwin32 exe to start IBM Tivoli Enterprise C
366. r architecture cj bin cnp sh file on the Linux system Copyright IBM Corp 2005 2010 117 2 Change this line TEP_JAVA_HOME bin java Xms64m Xmx256m showversion noverify classpath CPATH Dkjr trace mode LOCAL to include the system property TEP_JAVA_HOME bin java Xms64m Xmx256m showversion noverify classpath CPATH Dawt toolkit sun awt motif MToolkit Dkjr trace mode LOCAL e Use a 1 6 JRE to run the Tivoli Enterprise Portal desktop by doing the following 1 Locate the install _dir architecture cj bin cnp sh file on the Linux system 2 Export TEP_JAVA_HOME opt ibm java i386 60 3 Export KCJ_HOME Tivoli Enterprise Portal browser client yields Error 2721 and will not run This error occurs on Windows systems and is due to registry issues with Java 1 5 0 The error occurs when you attempt to bring up the Tivoli Enterprise Portal from IBM Tivoli Monitoring v6 2 2 and then there is an automatic attempt to install Java 1 5 0 Use the Windows Add Remove Programs utility and uninstall Java 1 5 0 Note you can leave Java 1 4 2 installed It does not affect this issue When you have uninstalled Java 1 5 0 reboot your system Now access the Tivoli Enterprise Portal following any prompts to install Java 1 5 0 and the Tivoli Enterprise Portal should then launch and run on your Windows system Some attribute groups showing a different name in the Tivoli Enterprise Portal The Tivoli Enterprise Po
367. r instances 2 Clear the browser cache 3 Reload the Tivoli Enterprise Portal browser client 4 Verify the correct images display The Warehouse Proxy Agent started but does not appear in the Managed System Status list on the Tivoli Enterprise Portal The database was either not created in UTF 8 or the DB2CODEPAGE was not setup in the environment as required for proper operation You need to set the DB2CODEPAGE 1208 no spaces after the 1208 on the system environment After you have done this shutdown the Warehouse Proxy Agent drop the UTF8TEST table from the database and then restart the agent Cannot start or stop agents from the Navigator view This feature is not supported in environments with a z OS Tivoli Enterprise Monitoring Server Cannot load a ws_pres css file in order to select a language other than English The user language parameter allows you to specify the language code The portal client uses cascading style sheets to render the application text If no localized version of a style sheet such as ws_press css is available the English version will be used 132 IBM Tivoli Monitoring Troubleshooting Guide Chapter 8 Tivoli Enterprise Portal Server troubleshooting This chapter describes problems that might occur with the Tivoli Enterprise Portal Server If you do find the resolution to a problem you experience with the Tivoli Enterprise Portal Server in this chapter see Chapter 6 Connectivit troub
368. r navigator positioning instead of on the host address A sample setting looks like this KFW_TOPOLOGY_CLUSTER_LIST AFF_xxx AFF_yyy IBM Tivoli Monitoring Troubleshooting Guide Where the setting lists the agent affinities to which this hostname logic should apply Here are some affinities for agents distributed with IBM Tivoli Monitoring version 6 1 AFF_NT_SYSTEM Windows OS AFF_ALL_UNIX UNIX OS AFF_0S400_0M 5 05 AFF_LINUX_SYSTEM Linux OS AFF_UNIX_LOG ALERT UNIX Logs So if you needed to use the hostname for navigator positioning for your Linux OS UNIX OS and Unix Log monitoring agents it would look like this KFW_TOPOLOGY_CLUSTER_LIST AFF_ALL_UNIX AFF_UNIX LOG ALERT AFF_LINUX SYSTEM There are many agents so listing the affinity for each would be cumbersome Here is a general method of figuring out the correct affinity name 1 If you have an agent connected and showing as online click on the navigator tree top node then right click and select Managed System Status 2 Make a temporary change to this workspace to show the agent affinity by right clicking on any row and selecting Properties 3 Click on the Filters tab Slide the scroll bar to the right and click on all the unset columns 5 Click OK and slide the Managed System Status window to the right For example you should now see that for the Monitoring Agent for Windows OS the affinity is 00080000000000000000000000000000 You can export this information t
369. r password or if it expires 201 server name E eo 190 Receive First Steps error at the end of a DB2 When using the viewuser command you receive a installation 202 message that an option is repeating 190 Tivoli Enterprise Portal Server cannot connect to Commands fail when a situation name donc of the database 202 characters 190 Oracle problem with JDBC der prior to 11 1 0 7 203 tacmd addSystem fails if agent already exists 191 Data did not migrate from the Microsoft SQL Installing an exported agent bundle using install sh Server database to the DB2 Universal Database 204 causes an error ee ce Database contents are incorrect after installation 204 The addbundles command fails 191 The error SQL0443N with SYSIBM CLI 805 occurs The exportBundles command does not work for after upgrading to DB2 UDB Version 8 1 Fix Pack patches os gt ba a e e Be e loa 10 204 Some rows do nol display i in an n upgraded table 205 Chapter 12 Universal Agent Using DB2 v8 1 Warehouse Proxy Agent crashes 205 troubleshooting o ae ci ew 193 Using DB2 V9 1 for z OS Warehouse Proxy agent Preliminary Universal Agent tonbleshooiine 193 encounters a large number of disconnections 205 Contents ix Historical data is not warehoused 206 Historical data for logs is incorrect 206 Warehouse Proxy Agent or Summarization and Pruning Agent fails due to DB2 transaction log full
370. r stop start commands when you are already logged on When you are logged in but restart the Tivoli Enterprise Monitoring Server using itmcmd server stop or start commands you receive the message KUICLRO99E The command did not complete because of a system error Refer to the log for details and contact the system administrator A new login solves the problem and enables the tacmd listsystems command Log indicates Hub Tivoli Enterprise Monitoring Servers are down when they are up The statusPut process fails periodically resulting in an incorrect hub status This condition is harmless and does not cause any operational change by the software The following is an example of the log Tue Jun 14 04 27 01 2005 K041039 Error in request sqlRequest Status 1103 Reason 1103 42AEA2E5 0011 6 ko4sndrq cpp 855 IBInterface_sqlRequest Distributed request failed IBM Tivoli Monitoring Troubleshooting Guide 42AEA2E5 0012 6 ko4state cpp 3519 IBInterface_sendInsert send insert has no request handle error 42AEA2Z2E5 0013 6 ko4ibput cpp 1407 IBInterface insertProcessing General error lt 1103 gt 42AEAZE5 0014 6 ko4ibput cpp 1657 IBInterface put_sList table put error lt 1103 gt 42AEA2E5 0015 6 ko4ibstr cpp 1139 IBStream insertDef IB Err 1103 42AEA2E5 0016 6 ko4crtsq cpp 5547 IBInterface_refreshIB Hub is not there 42AEA2E5 0017 6 ko4crtsq cpp 5547 IBInterface_refreshIB Hub is not there 42AEA2E5 0018 6 ko4crtsq
371. racing of events from ihe Tivoli Enterprise Console to the Tivoli Enterprise Monitoring Server 214 Situation Events have an UNKNOWN sevetity i in the Tivoli Enterprise Console 215 Forwarded IBM Tivoli Enterprise Console events display with the PARSING_FAILED status in the IBM Tivoli Enterprise Console reception log 216 The IBM Tivoli Enterprise Console is disabled 218 The IBM Tivoli Enterprise Console does not indicate that a situation event was i or closed 219 TEC_ITM_OM Situation Syne Eitor ayant occur 221 Receiving error message when running the Rule Check Utility with the cd and rd options 221 Expected impacted rules are not displayed when entering multiple event classes with class attributes 222 The wrules_check command fails 222 The NetView menus are unavailable 223 Errors occur during installation of IBM Tivoli Enterprise Console event synchronization 223 X IBM Tivoli Monitoring Troubleshooting Guide When using a Tivoli Enterprise Console view in a workspace receive an error message instead of event information su e eo The IBM Tivoli Enterprise Console does not display in a workspace after logging off and logging on the Tivoli Enterprise Portal 223 The Tivoli Enterprise Portal stops when attempting to launch the IBM Tivoli Enterprise Console behind arouter 224 The error ECO20298 Failed to connect o http servername
372. rade The first step to removing a failed upgrade is to determine where the install failed either before the files were copied or after the files were copied For GUI installations the files are copied after you click Next on the Start Copying Files window If you performed a silent installation look for a pair of entries separated by a blank line FirstUIBefore exiting to file copy FirstUIAfter entry after file copy If neither exist then the installation failed before the files were copied See Tnstallation failed before files were copied If both entries exist the installation failed after the files were copied See Installation failed after files were copied Installation failed before files were copied Check to see if the entry for your previous installation exists in the Add and Remove Programs tool If it does not exist follow the instructions in Your previous installation is too corrupt to use and must be completely removed You must either completely reinstall the previous release and then upgrade to IBM Tivoli Monitoring or just install IBM Tivoli Monitoring without attempting to upgrade If the entry exists in the Add and Remove Programs tool you can still use your existing Candle OMEGAMON installation Launch Manage Candle Services to start all components Installation failed after files were copied If your installation failed after the files were copied your current installation has been corrupted b
373. rent time stamps in the logs directory Setting the trace option for the IBM Tivoli Enterprise Console Situation Update Forwarder If your monitoring environment is configured for the IBM Tivoli Enterprise Console you can forward situation events to the Tivoli Enterprise Console event IBM Tivoli Monitoring Troubleshooting Guide server You can also view events on the event server through the Tivoli Enterprise Portal If you want to forward situation events to and view updates from IBM Tivoli Enterprise Console in the Tivoli Enterprise Portal you can set the trace for the Situation Update Forwarder on the IBM Tivoli Enterprise Console event server The default trace setting is low You can edit the trace setting using the sitconfig command BINDIR TME TEC OM_TEC bin sitconfig sh update fileName configuration_file name logLevel trace_level where configuration_file_name The file name of the actively loaded configuration file as indicated by the situpdate properties file trace_level Specifies the level of trace as low med or verbose Use the IBM Tivoli Enterprise Console Situation Update Forwarder trace facility to diagnose problems with the IBM Tivoli Enterprise Console Situation Update Forwarder The trace for the IBM Tivoli Enterprise Console Situation Update Forwarder is set during installation The acceptable values include e low e med e verbose The default trace value is low If you change the trace level after the S
374. ring the run as user name for an agent when the UNIX based systems monitoring agent was installed as a non root user For UNIX based systems you can only configure the run as user name if the UNIX based systems UNIX based systems monitoring agent was installed using the root user In this case leave the entry for the run as user blank or set the run as user to the same user ID used to install the UNIX based systems monitoring agent Interface unknown messages in ras1 logs Interface unknown messages appear in the ras1 log For example 46CB65C1 0001 F kdcsdrq c 466 do request Interface unknown 684152a852f9 02 c6 d2 2d fd 00 00 00 activity c638270e4738 22 02 09 2a 15 06 28 a2 7509 0 These are issued to alert you that some set components are driving RPC requests to a server that is not setup to handle that request Often this occurs when the Chapter 10 Monitoring agent troubleshooting 173 Warehouse Proxy Agent is not setup on a fixed port number For information on how to setup the Warehouse Proxy Agent see the IBM Tivoli Monitoring Installation and Setup Guide When upgrading a System Service Monitors agent from 3 2 1 to 4 0 receive KDY3008E message The previous agent is still running and using the port that is needed for the upgraded agent Stop the agent before the upgrade Once the agent is stopped the upgrade is successful The Tivoli Data Warehouse fails and you either lose data or have memory problems 1
375. rise Portal Also gives instructions for using and administering the Tivoli Enterprise Portal e Tivoli Enterprise Portal User s Guide SC32 9409 Complements the Tivoli Enterprise Portal online help The guide provides hands on lessons and detailed instructions for all Tivoli Enterprise Portal features e Command Reference SC32 6045 Provides detailed syntax and parameter information as well as examples for the commands you can use in IBM Tivoli Monitoring e Troubleshooting Guide GC32 9458 Provides information to help you troubleshoot problems with the software e Messages SC23 7969 Lists and explains messages generated by all IBM Tivoli Monitoring components and by z OS based Tivoli Management Services components such as Tivoli Enterprise Monitoring Server on z OS and TMS Engine e IBM Tivoli Universal Agent User s Guide SC32 9459 Introduces you to the IBM Tivoli Universal Agent an agent of IBM Tivoli Monitoring The IBM Tivoli Universal Agent enables you to use the monitoring and automation capabilities of IBM Tivoli Monitoring to monitor any type of data you collect e IBM Tivoli Universal Agent API and Command Programming Reference Guide 5SC32 9461 Explains the procedures for implementing the IBM Tivoli Universal Agent APIs and provides descriptions syntax and return status codes for the API calls and command line interface commands e Agent Builder User s Guide SC32 1921 Explains how to use the Agent Builder for creat
376. rise Portal logon fails on page 20 e Historical data is missing or incorrect on page 22 i i i Note Use the trace settings indicated in these troubleshooting instructions only while you are trying to diagnose a specific problem To avoid generating excessive trace data go back to the default trace settings as soon as the problem is solved About the tools This section describes several tools that are available to help you with troubleshooting ITMSUPER Tools The ITMSUPER Tools give you information about the health of your managed systems situations and environment configuration You can find the tools by searching for ITMSUPER on the Tivoli Open Process Automation Library OPAL Web site at http www ibm com software tivoli lopal Log Analyzer tool The Log Analyzer tool helps you perform detailed problem diagnosis on situations and events For more information see Log and Trace Analyzer Tool on page 60 A detailed explanation of where to find the monitoring server logs portal server logs and any other logs mentioned in this chapter can be found in Trace logging lon page 35 Table 22 on page 35 pdcollect tool The pdcollect tool collects the most commonly used information from a system It gathers log files configuration information version information and other data You can also use this tool to manage the size of trace data repositories For more information see pdcollect tool o
377. rmation on rolling of this data so that it is backed up see the IBM Tivoli Monitoring Configuring Tivoli Enterprise Monitoring Server on z OS Guide If maintenance is not performed then the agent stops writing to the Tivoli Data Warehouse until initialization is performed Because the agent has stopped writing the data is there forever until you re initialize and start again writing from the top of the first dataset If persistent datastore maintenance is allowed to proceed automatically as it is intended then the agent starts writing from the top of the first persistent datastore therefore wrapping occurs The persistent datastore is set up to allow for 24 by 7 data collection even if maintenance is not specified when configuring the product in ICAT There are typically 3 datasets though having more is allowed The minimum of 3 datasets allows for continuous collection the normal case is that one dataset is empty one or more are full and one is active When an active dataset becomes full the empty dataset is activated for continued writing When the persistent datastore detects that there are no empty datasets left it finds the IBM Tivoli Monitoring Troubleshooting Guide one with the oldest data and maintains it If the BACKUP or EXPORT options are not specified maintenance is done within the persistent datastore to initialize the dataset so that its status changes from full to empty If BACKUP or EXPORT are specified a job ru
378. roxy connection problems continued Corrective actions Store the collected data at the location of the monitoring Consider using a high port number for the monitoring server to ensure a stable connection server See the IBM Tivoli Monitoring Deployment Guide for more information on the COUNT and SKIP options for port number allocation Table 12 Correcting Warehouse Proxy agent problems 1 of 2 Monitoring agent or monitoring server is connected to the Warehouse Proxy agent but cannot send data Diagnostic actions Review the current CTIRA_NCSLISTEN and KHD_QUEUE_LENGTH settings in the Warehouse configuration file Corrective actions Set CTIRA_NCSLISTEN equal to at least 20 times the value of KHD_EXPORT_THREADS Increase KHD_QUEUE_LENGTH equal to a value greater than the number of agents being handled by that Warehouse Proxy agent Table 13 Correcting Warehouse Proxy agent problems 2 of 2 e Warehouse Proxy agent is unable to send data to the Tivoli Data Warehouse database e Expired or incorrect database password e The Warehouse Proxy agent has lost connectivity to the warehouse database Diagnostic actions 1 Verify component connectivity through the Self Monitoring Topology workspace To open this workspace right click the Enterprise Navigator item and then select Workspace gt Self Monitoring Topology 2 Verify that the warehouse database password and user ID are correc
379. rsal Agent is already active on this system Exiting Note In the RASI log above the Winsock error code 10048 indicates the Address is in use On UNIX based systems platforms the equivalent Address in use error code is usually 125 Run netstat a for more information about port usage on the local system Stop the other process if it is not required If the other process is required set KUMA_DCH_PORT nnnnn in the KUMENV or um ini file The environment variable causes the Universal Agent to allocate a different DCH port during startup The agent fails but events are still seen as active It is possible for a situation to remain open for almost four times the sample period before the event closes For example if a situation had a sample period of 15 minutes the following could occur The first sample true 00 00 sample true no change in status 15 00 agent crashes 15 01 proxy sends last sample 30 01 proxy sends last sample 45 01 proxy ends return no sample 60 00 Previously defined situations display in the Tivoli Enterprise Portal with a Problem or Error status Situations with a problem or error status are most likely the result of running um_cleanup which deletes the Tivoli Enterprise Portal Server catalog and attribute files and the Tivoli Enterprise Portal Server ODI files for all Universal Agent applications When the Tivoli Enterprise Portal Server and Tivoli Enterprise Portal server are recy
380. rt page click Edit profile 6 Select a product family and continue setting your preferences to specify the information you want in your e mails IBM Tivoli Monitoring Troubleshooting Guide 7 Click Submit Chapter 1 Introduction to troubleshooting 3 4 IBM Tivoli Monitoring Troubleshooting Guide Chapter 2 Collecting and reviewing troubleshooting data If you have a problem that you are unable to solve using the information in this guide gather the following information that relates to the problem and contact IBM Software Support for further assistance Appropriate IBM Tivoli Monitoring RAS1 trace output The reliability availability and serviceability RAS trace logs are available on the Tivoli Enterprise Monitoring Server the Tivoli Enterprise Portal Server and the monitoring agent By default the logs are stored in the installation path for IBM Tivoli Monitoring IBM Software Support uses the information captured by trace logs to trace a problem to its source or to determine why an error occurred The following links to sections in this document supply more information on these files e For information on where they are stored see Trace logging on page 35 e For information on setting the trace option for an IBM Tivoli Monitoring component see Setting traces on page 42 For information on dynamically setting the trace settings see modify trace settings for an IBM Tivoli Monitoring component on page 51
381. rtal pulls the attribute group names out of the resource bundle jar files for each product At this time these resource bundle jars are not guaranteed to be available to the CLI client For example KLZ CPU and KLZ Disk are listed in the Tivoli Enterprise Portal as Linux CPU and Linux Disk IO Monitoring agents show in an unexpected position in the navigation tree 118 In the Tivoli Enterprise Portal the navigation tree represents all the agents in the environment with a top level of Enterprise The default presentation assumes that each agent of a particular type comes from a different IP address and hostname When that is not true the agent will display at an unexpected or random location One example is when multiple agents are installed on a single server Another example is when a high availability option like Microsoft Windows Clustering is used and an IP address is shared Changes to the agent environment variable CTIRA_HOSTNAME affect the display name in the workspace but not within the navigator tree Here is an explanation of how to configure the portal server for desired navigator item display The portal server has a configuration environment variable that changes the navigation tree to depend on the host name instead of the IP address KFW_TOPOLOGY_CLUSTER_LIST This variable is added to installdir cnps kfwenv file Windows or installdir config cq ini UNIX and Linux Its purpose is to force dependence on the hostname fo
382. rting and imported a report package but when you try to run a report you see this message from Tivoli Common Reporting CTGTRVO14E The report can not be successfully formatted because it completed with errors reference ID REPORTID_3_OBJECTID_7ec67ec6 Click on the following link to view the report with the errors CTGTRVO11E See the Tivoli Common Reporting log files for more information https localhost 30343 TCR Reports view Copyright IBM Corp 2005 2010 227 This problem is occurring because you are missing drivers required by Tivoli Common Reporting You did not copy the required driver files or copied incorrect driver files Refer to the Configuring a JDBC data source topic in the IBM Tivoli Common Reporting User s Guide for additional information about this requirement For example if your Tivoli Data Warehouse is running on DB2 and you selected View the report with errors you would see error message No Data Returned Warning No data is available for this parameter set And this information is displayed The following items have errors ReportDesign id 1 Cannot open the connection for the driver org eclipse birt report data oda jdbc Cannot load JDBC Driver class com ibm db2 jcc DB2Driver To address this problem for a DB2 database you would need to copy the following files e db2jcc jar e db2jcc_license cu jar Typically found in this default DB2 installation path or in the java directory of wha
383. ry exceptions being written to the Tivoli Enterprise Portal client log files Help index and search text entry fields are disabled For some reason your browser is unable to load the Java applets necessary to enable the Index and Search fields and is rendering the help in Pure HTML If you view Tivoli Enterprise Portal help with Internet Explorer use the following steps to resolve this problem 1 From the Internet Explorer browser toolbar go to Tools gt Internet Options gt Advanced gt Java IBM 2 Clear the Use Java version 1 4 2 for lt applet gt requires restart check box If the help index and search text entry fields are still disabled after clearing the Use Java version 1 4 2 for lt applet gt requires restart check box see or Index tabs on page 123 Java exception logging onto the Tivoli Enterprise Portal from a browser If you encounter an exception that has the following text java lang UnsatisfiedLinkError com webrenderer server NativeMozillaLibrary setMozPath delete the WebRenderer directory under the home path For Linux systems this path is HOME webrendererswing and for Windows systems this path is HOMEPATH webrendererswing On Linux IBM Tivoli Enterprise Monitoring Agent topics do not display in the Help Contents or Index tabs You can see the Tivoli Enterprise Portal help topics in the Contents tab but none for your installed monitoring agents Change your default brows
384. s Appendix D Documentation library 253 254 Two of these wikis are of particular relevance to IBM Tivoli Monitoring Tivoli Distributed Monitoring and Application Management Wiki at distributed products including IBM Tivoli Composite Application Management products provides information about the OMEGAMON XE products NetView for z OS Tivoli Monitoring Agent for z TPF and other System z monitoring and application management products IBM Tivoli Monitoring Troubleshooting Guide Notices This information was developed for products and services offered in the U S A IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or imply that only that IBM product program or service may be used Any functionally equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the user s responsibility to evaluate and verify the operation of any non IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to IBM Dir
385. s impractical and is contrary to the desired effect of sharing You receive a KFWITM290E error when using deploy commands with a z OS monitoring server Remote Deployment is not supported in environments with a z OS Tivoli Enterprise Monitoring Server Running deployment in a hot standby environment The IBM Tivoli Monitoring hot standby capability allows your monitoring environment to continue operating in the event of environmental or operational issues with the primary hub monitoring server for detailed information about Tivoli Monitoring s hot standby feature see the IBM Tivoli Monitoring High Availability Guide for Distributed Systems You should refrain from deploying or updating agents when IBM Tivoli Monitoring is converting to a mirror monitoring server No agent deployments or remote deployment operations should be executed from a hot standby mirror hub as this might cause your deployment transactions to get stuck in a queued state and you might not be able to clear them Difficulty with default port numbers You can use Telnet to test if the port is open in the firewall Use the following command for this test telnet hostname 15001 where 15001 is the port number in question Selecting Security Validation User displays a blank popup While configuring the Tivoli Enterprise Monitoring Server you have an option to select the Security Validation User When selecting this option a blank popup is 76 IBM Tivoli Monitoring
386. s are not completing before the TIMEOUT expires Deploy cannot tell if the install failed An agent does not display in the Tivoli Enterprise Portal or in the output from the i a command e One monitoring agent s workspaces are listed under another agent node on the Tivoli di Portal Issues with starting and stopping an agent asa non Administrator user x UNIX based systems Log agent was deployed configured and started but returns the KFWITM290E error KDY1024E error displays when configuring the run as user name for an agent a Interface unknown messages in ras1 logs When upgrading a System Service Monitors agent from 3 2 1 to 4 0 receive KDY3008E message The Tivoli Data Warehouse fails and you either lose data or have memory problems Error list appears in warehouse logs viii IBM Tivoli Monitoring Troubleshooting Guide 163 163 163 163 164 164 165 165 167 167 167 167 167 168 168 168 168 168 169 169 169 170 170 172 173 173 173 173 174 174 175 When configuring the Monitoring Agent for Sybase and the Warehouse Proxy Agent receive message to use CandleManage listSit command with the type option fails withe a KUIC02001E message on Japanese Systems Take Action command names do not accept non English characters 5 Creating a situation from a group member does not copy the distr
387. s are violated then the results can be failure to start or even failure for normal server processes to start The profile should be simple and clear This might require creating a special user ID for this purpose to avoid impacting other users The Tivoli Enterprise Portal Server does not start after installation Check the following log files for messages that indicate why the Tivoli Enterprise Portal Server did not start Table 35 The Tivoli Enterprise Portal Server does not start after installation Log file For messages that indicate kfwrasl log A failure during upgrade The Tivoli Enterprise Monitoring Server hub did not connect install_dir cnps sql1lib migrate log Any error Copyright IBM Corp 2005 2010 107 Cannot log in to the Tivoli Enterprise Portal The following table provides resolutions for problems logging in to the Tivoli Enterprise Portal Table 36 Cannot log in to the Tivoli Enterprise Portal Problem User authorization failed OR KFWITM215E Unable to process logon request Corrective action and solution e Ensure the user ID and password are correct e Verify that the monitoring server has started e Define the user in the portal server e Configure the TEPS or TEPS2 data sources e If security validation is active on the hub monitoring server make sure the user ID is defined to the security system For more information on security validation see the IBM
388. s below verify with your database administrator that the following bullets below are not the cause of the problem e The database manager has not been started on the database server e The database manager was stopped e The database agent was forced off by the system administrator e The database manager has already allocated the maximum number of agents e The database agent was terminated due to an abnormal termination of a key database manager process If the problem is not due to any of the above it is most likely that the application is using multiple contexts with local protocol In this case the number of connections is limited by the number of shared memory segments to which a single process can be attached For example on AIX the limit is ten shared memory segments per process Do the following to resolve the problem 1 On the machine with the database that you want to connect to configure the database manager to use TCP IP on AIX 2 On the server machine login as the DB2 instance owner 3 Set DB2COMM to TPC IP for example db2set DB2COMM tcpip 4 Edit the etc services file to include both a DB2 connection service port and a DB2 interrupt connection if they do not already exist such as db2cDB2 50000 tcp DB2 connection service port db2iDB2 50001 tcp DB2 interrupt connection service port 5 Update the database manager configuration such as db2 update dbm cfg using svcename db2cDB2 The argument after svcena
389. s for pdcollect that control the data collection process You should request assistance from IBM Service to determine which if any of the options should be specified for the particular problem which you have encountered You must execute pdcollect while running under the same user ID that you use to run IBM Tivoli Monitoring to avoid problems with file access authorization On Windows UNIX and Linux systems you are prompted to examine the data that has been collected You can add additional files pertaining to the problem by copying them into the same directory You can also edit or remove data which you do not want to expose to IBM for confidentiality reasons but doing so may impact diagnosing the problem Again consult with IBM Service if you have questions The output produced by pdcollect is a single compressed file that should be transmitted in binary form to IBM for analysis The name of the output file is displayed in a message at the end of pdcollect processing Running the following command on Windows systems results in the output that follows Chapter 4 Tools 57 58 pdcollect This program is intended to collect diagnostic information from the computer on which it is executed Collected data includes system configuration environment disk space information network state errors and warnings from the event log configuration and log files Command line syntax is pdcollect cmd options work directory
390. s kdyproc_rasl_ timestamp log and hostname _ ux_kuxagent_ timestamp XX log is located in the CANDLEHOME logs directory On Linux systems the logs kdyproc_rasl_ timestamp log and hostname _ 1z_klzagent_ timestamp XX log is located in the CANDLEHOME logs directory Refer to IBM Tivoli Monitoring v 6 2 Problem Determination Guide for more information The user can verify the agent is stopped by running tacmd listSystems opt IBM ITM bin tacmd listsystems Managed System Name Product Code Version Status zpmaix13 KUL UL 06 20 00 00 N zpmaix13 08 08 06 20 00 00 Y amshpl6 tivlab raleigh ibm com K UX 06 20 00 00 Y TEMS_zpmaix13 EM 06 20 00 00 Y To start the agent user can issue tacmd startAgent opt IBM ITM bin tacmd startagent n zpmaix13 KUX t ul Using the kinconfig command and remotely starting stopping or recycling agents fails on Windows 2000 systems If the endpoint is a Windows 2000 systems you must reboot the system after the Monitoring Agent for Windows is installed to allow environment variables that have been set by the OS Agent s installation to take effect for other processes to use these variables You receive a message when using a tacmd command related to agents You receive the following message KDYOO10E The agent bundle product_code was not found in the agent bundle depot on TivoliEnterpiseMonitoringServer_Name The agent bundle specified for deployment is not installed on the agent bundle depot for
391. s of this failure in the IBM provided reports were eliminated by casting DB2 Integer values to DECIMAL column_name 31 0 This problem should not occur with IBM provided reports but could be an issue in customer generated reports No data is plotted in graph or some columns in the table are blank In this scenario you complete the report parameters dialog to generate a report After clicking Submit you see a report where some columns have data and some do not When you check version levels you find that the version of database manager you are using for Tivoli Data Warehouse is correct and that it is defined as your data source and that all the requested columns in the attribute table are available However even though data is displayed in the other columns of this table in Tivoli Enterprise Portal no data is available in the column needed to draw the graph Historical data sets have been allocated and historical collection has been configured and enabled Some data has been collected Tivoli Common Reporting is configured and is drawing a partial report If the report does not meet your needs you may be able to obtain the needed data by one of these methods e By choosing a different attribute on the Report Parameters dialog e By ensuring that data for the selected attribute is available for this workspace configuration For example the OMEGAMON XE for Mainframe Networks monitoring agent cannot collect data for some attributes in OS
392. s on the level of tracing set Level name What is logged or traced LOG_ERR Only exceptions and errors are logged and traced LOG_INFO Also log messages are logged and traced DEFAULT DEBUG_MIN Also most important method entries exits and trace messages are traced DEBUG_MID Most of the method entries exits and trace messages are traced DEBUG_MAX All of the method entries exits and trace messages are traced The level can be set in configuration files or by exporting an environment variable called TRACE_LEVEL with one of the values mentioned above Configuration of RAS settings is stored in the following files e CH config ITMInstallRAS properties for installation e CH config ITMConfigRAS properties for configuration Chapter 4 Tools 39 40 Callpoints are the only component that is handled differently their logs and traces always go to the directory CH InstallITM plugin executionEvents The default location for installation is CH logs itm_install log trc and for configuration it is CH logs itm_config log trc To gather all the needed logs and environment information in case of an error use the pdcollect tool described in pdcollect tool on page 57 Component Location File name Install logs traces CH logs candle_installation log itm_install log trc Config logs traces CH logs itm_config log trc Logs for component startup CH logs pc env lists env variables pas
393. s online You can run the listsSystems command to verify that the node is online e Specify correct properties tacmd addSystem t type TYPE n node MANAGED 0S d dir directory NODEDIR i imagePath IMAGEPATH p property properties NAME VALUE where t type Specifies the type of managed system to add to the monitoring system You can specify a managed system type name or its associated two character code Use viewDepot to display a list of correct managed system types n node Specifies the node to start A node is identified by the managed operating system that it contains MANAGED OS Specifies a correct managed operating systems d dir directory NODEDIR Specified the correct name of the directory that contains the node components including the OS agent This syntax is only correct when the node is on the local machine il imagePath Specified the correct directory that contains agent install images p property properties Specifies one or more NAME VALUE pairs that identify configuration Chapter 11 Command troubleshooting 189 properties of the new system and their values Run the describeSystemType command to determine correct values for the properties The tacmd getdeploystatus command is not returning status return codes At this time there is not a workaround for this issue tacmd createSit does not send errors if you mistype the name of a
394. s the situation writing a lot of Windows event logs Yes Disable all event log monitoring situations 4 Select each of the workspaces in turn to see which one is consuming high CPU 5 Running the following ITMSUPER tools might also provide more information e Stressed Resources tool e Connectivity tool e Situations tool 6 When the computer where the monitoring agent is running has multiple Network Interface Cards NICs the agent might not be bound to the Primary NIC and therefore might not be able to establish connectivity with the monitoring server High CPU usage can result from the agent s frequent attempts to connect To correct this you might need to set the environment variable KDEB_INTERFACELIST or KDEB_INTERFACELIST IP_address where IP_address is the address of the NIC Make the changes in the associated agent ENV configuration file for Windows or the ini configuration file for UNIX or Linux The following tables include possible causes and corresponding solutions for the problems described in this section 30 IBM Tivoli Monitoring Troubleshooting Guide Table 19 Correcting situation problems A situation is causing the high CPU usage problem Diagnostic actions Run the Situation Test ITMSUPER tool 2 Find out which situations have been deployed to the monitoring agent On Windows Open the agent specific 190 file found in the install_dir TMAITM6 logs directory to view a list of situations s
395. s1 list command Metrics Shown as ME when using the ras1 list command OUTPUT Shown as OUT when using the ras1 list command State Shown as ST when using the ras1 list command Setting trace to ALL includes every trace point defined for the component This might result in a large amount of trace If you have been given a more specific setting use it ALL can sometimes be necessary when isolating a problem It is the equivalent of setting Error Detail Flow State Input Output Metrics The ras1 units command is used to determine the list of UNITs and COMPs available in an IBM Tivoli Monitoring component The first column is the list of available UNIT values the last column lists the corresponding COMP values Turning on COMP KDH ALL will turn ALL level tracing on for all of the files where KDH is listed in the right hand column highlighted below The following is a subset of the results for the Monitoring for Windows agent kbbcrel c 400 May 29 2007 12 54 43 1 1 kbbcrnl c 400 May 29 2007 12 54 42 1 1 kdhblde c 400 May 29 2007 12 59 34 1 1 KDH kdhOmed c 400 May 29 2007 12 59 24 1 1 KDH kdhsrej c 400 May 29 2007 13 00 06 1 5 KDH kdhb1fh c 400 May 29 2007 12 59 33 1 1 KDH kdhbloe c 400 May 29 2007 12 59 38 1 2 KDH kdhsins c 400 May 29 2007 13 00 08 1 3 KDH IBM Tivoli Monitoring Troubleshooting Guide kbbacdl c 400 May 29 2007 12 54 27 1 2 ACF1 kbbaclc c 400 May 29 2007 12 54
396. se Monitoring Server and events do not appear for any remaining managed system list that is in the distribution list for the situation Manually restart the situation after the managed system list has been removed Descriptions are not displayed for default situations If you use the command to view any default situation for example when inputting tacmd viewsit s UNIX_User_CPU_Critical the following is displayed Name UNIX _User_CPU_ Critica Full Name Description Kux KUX3736 Type UNIX OS This is a limitation of the current product Chapter 10 Monitoring agent troubleshooting 167 Agent configuration failed on remote deploy while using single quotes for configuration properties You cannot provide a deployment or configuration property with single quote characters embedded within the properties For paths with spaces in them wrap the entire property in single or double quotes The following examples are valid DBSETTINGS SYBASE data sybase DBSETTINGS SYBASE data sybase DBSETTINGS SYBASE data sybase Whereas this example is NOT valid DBSETTINGS SYBASE data sybase New attributes missing When viewing a workspace not all of the attribute group s attributes are displayed in a table view To see the new attributes in a table you must create a new query to retrieve the new attributes and you must create a new workspace to use the new query Receive errors when modifying the JAVA HEAP SIZE
397. se Portal client to connect to the Tivoli Enterprise Monitoring Server You might need to include the IBM Java program in the programs section of your firewall software and include the IP addresses of other Tivoli Monitoring Services components in the access control for the firewall Tivoli Enterprise Portal Server is initializing and is not ready for communications If you attempt to start Tivoli Enterprise Portal just after starting the Tivoli Enterprise Portal Server the Tivoli Enterprise Portal Server is not ready for requests from the client until initialization and kfwservices exe is fully started The Tivoli Enterprise Portal Server is not ready for requests from the client until its process kfwservices exe is fully started Keep the Logon window open and click OK after waiting a moment or two Tivoli Enterprise Portal Server lost contact with the Tivoli Enterprise Monitoring Server and is attempting to reconnect This message displays when the Tivoli Enterprise Portal Server lost its connection to the Tivoli Enterprise Monitoring Server usually because the Tivoli Enterprise Monitoring Server stopped or is recycling See also KFW_CMW_RECYCLE_DELAY DB2 errors during a logon attempt to the Tivoli Enterprise Portal and portal server You are able to log in but receive an error message that you cannot open a workspace in the Tivoli Enterprise Portal Chapter 6 Connectivity troubleshooting 113 Before completing the step
398. sed to the agent hostname_pc_ID log Callpoint logs traces CH InstallITM plugin callpoint tre log executionEvents stderr logs timestamp stdout install config plugin_type pc Upgrading from Tivoli Distributed Monitoring log file All upgrade actions performed by the IBM Tivoli Monitoring Upgrade Toolkit are recorded in a central log with an associated user ID and a time stamp Upgrade actions taken outside of the Upgrade Toolkit are not recorded in the log The detailed results from running the Upgrade Toolkit are recorded in a log file Table 24 Upgrading from Tivoli Distributed Monitoring log file Windows UNIX based systems DBDIR AMX logs log_tool_ DBDIR AMX logs log_tool_ timestamp log timestamp log where DBDIR The Tivoli Management Environment Framework environment variable that specifies the directory where the Object Repository odb bdb is located tool Specifies the IBM Tivoli Monitoring Upgrade Toolkit tool witmscantmr witmassess or witmupgrade timestamp Specifies a time stamp that includes data and time of execution For example 1og_witmscantmr_20050721_15_30_15 1og The log file name displays when the Upgrade Toolkit tool completes the upgrade operation Each time a Upgrade Toolkit tool runs its generates a new log file that is never reused by any tool The contents of the log file conform to the Tivoli Message Standard XML logging format The following exa
399. ser does not have authority to create a data source and register with windows or the authority to create a windows user account Continue installing the software using the following steps to resolve the error 1 When the install completes log off the current Windows user session and log in with using the db2admin userid 2 Run install _dir instal ITM DB2DataSource exe from Windows Explorer or a command prompt You can run this program again even if one or more of the tasks completed the first time it ran 3 Start the Tivoli Enterprise Monitoring Server after the software indicates the Tivoli Enterprise Portal Server configuration was successful IBM Tivoli Monitoring Troubleshooting Guide If the Tivoli Enterprise Portal installation is complete but it does not start the data source might not be defined Go to the install dir cnps kfwras1l 1og file If error messages similar to the following are present in the log the data source was not defined IBM ODBC Driver Manager Data source name not found and no default driver specified Use the follow steps verify whether or not the data source was created and resolve the problem 1 Open the ODBC datasource window Start gt Settings gt Control Panel gt Administrative tools and double click the Data Sources ODBC 2 Verify that the Tivoli Enterprise Monitoring Server IBM DB2 ODBC DRIVER datasource is defined 3 If the IBM DB2 ODBC DRIVER data source is not present run t
400. service in a location broker RPC clients callers of rpc__sar use the location broker to obtain the address of a server prior to making a call to that server Use of the location broker is well defined by the Ib__lookup Remote Procedure Call It is also appropriate to mention that there are two types of location brokers the local location broker LLB and the global location broker GLB There is one local location broker for every RPC server The Tivoli Enterprise Monitoring Server has it s own LLB The monitoring agent the Warehouse Proxy agent and the Tivoli Enterprise Portal Server all have their own instance of a LLB RPC servers by definition publish the service offered and address of this service in their local location broker Confirming the Local Location Broker service initialized The bind messages in the RKLVLOG indicate the success or failure of the LLB service initialization One of two message IDs prefix the LLB status messages depending on how the LLB service was started KDSNC007 is the message prefix issued on successful LLB process initialization when the LLB is started internally by the Tivoli Enterprise Monitoring Server KDSNC004 Bind of local location broker complete ip pipe 9 42 46 26 21343 KDSNC004 Bind of local location broker complete ip 9 42 46 26 21343 KDSNC004 Bind of local location broker complete sna USCACOO1 VWCTHLB CANCTDCS SNASOCKETS 135 KDSNC007 Local Location Broker is active Chapter 5
401. ses where the IP address changes and then KFW_TOPOLOGY_CLUSTER_LIST forces the navigation display to use that apparent hostname instead of the TCP IP hostname Some agents require a different mechanism to change the apparent host name In the case of the MQ Agent for example AFF_MVS_MQM update the mq cfg file and add Chapter 7 Tivoli Enterprise Portal troubleshooting 119 SET AGENT NAME hostname Long hostnames can create confusion such as when a hostname is fully qualified The default portal server processing only uses the first part of the fully qualified name so two names like abc xyz large com and abc def large com would both appear at the node labeled abc That can be controlled by adding the following to the portal server environment file KFWENV cq ini KFW_TOPOLOGY_KEEP_DOT_IN_NODE_NAMES Y Remember the managed system names are limited to 32 characters so the hostname product can be truncated and cause accidental duplications That would be another case where you would need to set the CTIRA_HOSTNAME Tivoli Enterprise Portal Desktop called through Java Web Start does not work properly after adding agents support for Tivoli Enterprise Portal Server Launch the Manage Tivoli Enterprise Monitoring Services utility on the system where Tivoli Enterprise Portal Server is installed and execute the TEPS reconfigure action Restart the Tivoli Enterprise Portal Desktop through the Java Web Start Receive a Loading Jav
402. smatched between the portal client and tacmd command This problem involves a discrepancy or mismatch between a monitoring agent s status as shown in the portal client and as returned by a tacmd command For example A monitoring agent shows as offline in the portal client and online in the results of a tacmd command A monitoring agent shows as online in the portal client and offline in the results of a tacmd command Preliminary diagnostics 1 2 Verify the state of the monitoring agent in Manage Tivoli Monitoring Services Compare the status of the node in the physical Navigator view with the status reported in the Managed System Status workspace If the status in the physical Navigator view agrees with the status shown in the Managed System Status workspace then the problem is at the monitoring agent See Fable 4 To determine whether the problem is in the portal server or monitoring server set the following trace in the portal server ERROR UNIT ctcmw IN ER Then examine the portal server log for the following statement Node Status event received managed system name If the trace shows that the last node status record received for the managed system matches the status shown in the portal client then the problem is located in the monitoring server See Table 5 on page 16 If the trace shows that the last node status record received for the managed system indicated the correct status then the problem is located in the
403. sociated at the Tivoli Enterprise Portal Server level set the Tivoli Enterprise Portal Server environment variable KFW_CMW_SPECIAL_HUB_ENTERPRISE N When non hub situations are associated at the Tivoli Enterprise Portal Server they turn TRUE meaning they are visible in the situation event console Through the situation editor if you assign all the agents and managed systems lists from a situation that situation event continues to appear in the situation event console Starting and stopping the Eclipse Help Server The Tivoli Enterprise Portal Server depends on the Eclipse Help Server to display standalone help The Eclipse Help Server is treated in the same way as Tivoli Enterprise Portal Server as a server component as opposed to IBM Tivoli Monitoring v6 1 where the Eclipse Help Server was treated as an agent This change necessitates that the Eclipse Help Server is started whenever the Tivoli Enterprise Portal Server is started Eclipse Help Server start failure does not affect Tivoli Enterprise Portal Server startup The UNIX command itmemd agent stop or start all does not stop or start the Eclipse Help Server now since it does not also stop or start Tivoli Enterprise Portal Server On Windows systems starting or stopping of all components using kinconfg exe still applies to all components without exceptions The Eclipse Help Server by default cannot be stopped if the Tivoli Enterprise Portal Server is running To stop the Ecli
404. stall sh Contact IBM Software Support if you cannot uninstall the agent See Chapter 2 and reviewing troubleshooting e 5 for information on what types ot data to collect before contacting Support Prompted to uninstall a database that was not running during uninstallation During uninstallation when prompted for the DB2 username and password in order to remove the Tivoli Enterprise Portal Server from the DB2 database you were prompted with the following question would you like to delete the Tivoli Enetrprise Portal MSSQL MSDE Database The Microsoft MS SQL Server database was not running My Tivoli Enterprise Portal Server is installed and configured with a DB2 database and not a Microsoft MS SQL Server database It is likely that the system you are using at one time had a Microsoft MS SQL Server database installed that was not properly uninstalled It does not matter whether the database is running or not if the data source exists you will be asked the question and if you answer yes there will be an attempt to remove the database 106 IBM Tivoli Monitoring Troubleshooting Guide Chapter 6 Connectivity troubleshooting This section provides descriptions of and resolutions for problems you might experience with connectivity including logging in passwords and communication among IBM Tivoli Monitoring components Tivoli Enterprise Portal monitors the connections between server and client and server and host data sources
405. state Note By default the restore jobs runs against all of the processed datasets The comments section of these jobs detail that some of the datasets can be commented out depending on the time and nature of the failure experienced For instance only 4 of 7 datasets were renamed during RNEVSAM so only those 4 must be restored to their original names If you are not certain which job to run or what datasets to comment out in these jobs contact IBM Software Support The Tivoli Enterprise Monitoring Server starts normally in a system without the Integrated Cryptographic Service Facility but does not connect to the Tivoli Enterprise Portal Server Although Integrated Cryptographic Service Facility ICSF provides robust password encryption you are not required to use it as it can affect compatibility with the z OS OMEGAMON monitoring products Perform the following steps so that the Tivoli Enterprise Portal Server can connect to the Tivoli Enterprise Monitoring Server IBM Tivoli Monitoring During configuration 1 Select Configure the Tivoli Enterprise Monitoring Server gt Specify configuration values gt Integrated Cryptographic Service Facility ICSF installed 2 Specify N in the on the Integrated Cryptographic Service Facility ICSF installed panel After the Tivoli Enterprise Monitoring Server configuration is complete and it is running Troubleshooting Guide 1 The Tivoli Enterprise Portal Server configuration must be
406. stem time After the set timeout the Tivoli Enterprise Monitoring Server is still pending When you attempt to stop or start the Tivoli Enterprise Monitoring Server service after the set timeout the Tivoli Enterprise Monitoring Server is still pending the start or stop You will receive the following error TEMS service is still pending start stop Check ITM documentation for more details The default time for starting and stopping a Tivoli Enterprise Monitoring Server service is ten minutes In the following two situations this time can be ten times as long 1 If you have a large or complicated infrastructure connected to the Tivoli Enterprise Monitoring Server 2 If you have a remote Tivoli Enterprise Monitoring Server but the hub Tivoli Enterprise Monitoring Server is offline In any other situation call IBM Software Support Providing the wrong path to configuration files during LDAP configuration causes the Tivoli Enterprise Portal login window to hang During Security and LDAP configuration at Tivoli Enterprise Monitoring Server if you provide an incorrect path to the key kdb and key sth files the Tivoli Enterprise Portal login window goes into an indefinite loop This occurs after restarting the Tivoli Enterprise Monitoring Server and launching the Tivoli IBM Tivoli Monitoring Troubleshooting Guide Enterprise Portal client Ensure that the provided paths are correct during configuration The installer does not check
407. stions 67 firewall 199 Fixpack CNP196 67 Frequently asked questions 67 UNIX based systems installation 68 Windows installation 67 G GSKit 75 H HEAPDUMPs 124 help not displaying 121 historical data incorrect 206 not warehoused 206 too much 207 Hot Standby 147 IBM Support Assistant 247 IBM Tivoli Enterprise Console events not acknowledged 219 is disabled 218 IBM Tivoli Enterprise Console event forwarder 47 IBM Tivoli Monitoring Operations Logging 63 IBM Tivoli Monitoring Service Console 53 blocking access to the 54 IBM Tivoli Monitoring Technology Service Console 53 IBM Tivoli Universal Agent 44 IBM V1 4 2 JRE errors 125 installation problems 84 removing a failed Windows 100 Installation problems 67 troubleshooting problems 67 InstallShield error 82 instance removing remotely 130 Integrated Cryptographic Service Facility 152 interface unknown 173 IP PIPE 68 ISA 247 260 IBM Tivoli Monitoring Troubleshooting Guide ITMSuper 65 J JAR cache clearing the 56 Java heap size 229 Java VM heap dump 87 JAVACore 124 JDBC 227 JRE 87 124 125 K KDSMA013 153 KDY1024E 173 KFW_CMW_RECYCLE_DELAY 113 KFW_CMW_SPECIAL _HUB_ENTERPRISE N 136 KFWENV 127 KFWITM215E 113 KFWITM290E 173 kfwrasl log 42 KHD_MAX_ROWS_SKIPPED_PER_ TABLE 79 KHDRAS1_Mig_Detail log 205 khdras1 log ORACLE or DB2 errors in the 207 kinconfig 186 KLVSQ000 153 KLVST005 152 KLVST044 151 kshsoap client fails 130 KUIC02001E 176 184 KUIC
408. t rows to ensure reliable logs Warehouse Proxy Agent or Summarization and Pruning Agent fails due to DB2 transaction log full If the DB2 transaction log is not large enough and fills operations performed by the Warehouse Proxy Agent or Summarization and Pruning Agent will fail If this happens you will see a message like the following in the Warehouse Proxy Agent log file hostname_hd_java_nnnnnnnnnn 1og or the Summarization and Pruning Agent log file hostname_sy_java_nnnnnnnnnn 10g com ibm db2 jcc a SqlException DB2 SQL error SQLCODE 964 SQLSTATE 57011 SQLERRMC null Increase the DB2 transaction log size See the DB2 manuals for altering the LOGFILSIZ LOGPRIMARY and LOGSECOND parameters within DB2 Incorrect data is collected in the warehouse for filtering if using a wildcard 206 This behavior could be caused by either of these cases e There are multiple historical collections distributed to your agent for the tablespace attribute group All of the collections will write to the same short term history files and to the same database tables e You already had data in the short term history file for the tablespace attribute group before you created and distributed the new historical collection that has the filter The older data would have been exported to the warehouse proxy and shown up in the Tivoli Data Warehouse database Wild card matching is not supported The only way to mimic that functionality would be
409. t and have not expired 3 Look at the Warehouse Proxy agent RAS1 logs for export resource availability timeout The Warehouse Proxy agent might be unable to export because it is too busy Corrective actions Update the configuration parameters See Tivoli Data Warehouse environment variables on page 240 for a list of warehouse proxy environment variables Table 14 Correcting Summarization and Pruning Agent problems e Summarization and Pruning Agent yields unexpected values e Unanticipated attribute behavior leads to unexpected data Diagnostic actions 1 Look at a workspace view that shows real time data and compare it with the unexpected data 2 The Tivoli Management Services Warehouse and Reporting IBM Redbooks publication discusses aggregation methods used by the Summarization and Pruning Agent This Redbook might help you understand how data is aggregated for various data types Corrective actions Review documentation for the monitoring agents that are generating the unexpected values to clarify the expected types of values for the attributes in question Chapter 3 Solving common problems 23 Table 15 Correcting persistent data store for z OS problems Incorrect configuration of the persistent data store for z OS agents Diagnostic and corrective actions 1 Is historical data configured to be collected at the agent or the monitoring server If the agent is confi
410. t restart the Tivoli Enterprise Monitoring Server for the change to take effect Use the following instructions to set the trace levels On Windows 1 5 6 From Manage Tivoli Enterprise Monitoring Services right click the Tivoli Enterprise Monitoring Server Click Advanced gt Edit trace parms Under Enter RAS1 Filter add UNIT kfaot trc_class where trc_class Specifies STATE DETAIL or ALL which produces increasingly more trace information The default log file location is C IBM ITM CMS 1ogs KMSRAS1 LOG change if necessary Click OK to set the trace Recycle the Tivoli Enterprise Monitoring Server for the trace to take effect On UNIX based systems 1 P Edit install_dir config hostname_ms_Tivoli_Enterprise_Monitoring_Server_ID config where install _dir Specifies the install directory of Tivoli Enterprise Monitoring Server hostname Specifies the host name value supplied during install Add UNIT kfaot trc_class to the line KBB_RAS1 ERROR where trc_class Specifies one of the following levels of trace detail e STATE minimum detail e DETAIL medium detail e ALL maximum detail For example KBB_RAS1 ERROR UNIT kfaot STATE Save the file Recycle Tivoli Enterprise Monitoring Server for the trace to take effect The Tivoli Enterprise Monitoring Server log can be found in install_dir logs hostname_ms_nnnnnnn 10g where is a time stamp There might be multiple files with diffe
411. t you previously upgrade infi of ON use the following steps to verify that information in the status properties file is causing this problem 1 Open the status properties 2 Look for an entry like the following example Copyright IBM Corporation 2005 Wed Sep 14 15 54 43 CDT 2005 Endpoint east EndpointClass TMF_Endpoint Endpoint COMPLETE Monitor Coast 120401 Threshold critical COMPLETE In this example the status of the endpoint east is COMPLETE which indicates that it was upgraded successfully The witmupgrade command does not upgrade any item with the COMPLETE status and reports that it was already upgraded To upgrade the endpoint the status for the endpoint east must be the INCOMPLETE such as in the following example Endpoint east EndpointClass TMF_Endpoint Endpoint INCOMPLETE The only way to change the endpoint status in the status properties file to INCOMPLETE is to perform a rollback on the upgrade of the item See the IBM Tivoli Monitoring Upgrading from Tivoli Distributed Monitoring Chapter 5 Installation and configuration troubleshooting 77 78 Rolling back the upgrade You can use the rollback option r option of the witmupgrade command to remove the new IBM Tivoli Monitoring resources that you created This is a necessary step if you want to repeat the test scenario Rolling back the upgrade for the test scenario removes the Windows OS monitoring agent from the Windows endpoint a
412. table or disabled state However you should be able to continue with the uninstallation after ensuring that there is indeed no installation being run on the system Receive an install sh error when installing two components or agents in the same installation directory Installing two components or agents in the same CANDLEHOME or installation directory is supported as long as the User ID used to run the installation is always the same Installing two components or agents in the same CANDLEHOME or installation directory using different User IDs is not supported When attempting to install IBM Java 1 5 0 on Windows 64 bit system nothing happens Only 32 bit browsers are supported on the AMD 64 Windows environment due to the lack of a native 64 bit Web Start or Java Plug in supports During a remote Tivoli Enterprise Monitoring Server upgrade you receive a The Backup procedure for TEMS database files has failed pop up During a remote Tivoli Enterprise Monitoring Server upgrade you receive the following pop up message The Backup procedure for TEMS database files has failed If you continue with the installation your customized tables could be lost Would you like to abort the installation If you click YES there is a risk of losing your customized tables To ensure that you do not lose data complete the following steps 1 Click NO and exit the upgrade installation 2 Restart the remote Tivoli Enterprise Monitoring Server s
413. tarted for that agent On UNIX and Linux Open the agent specific 190 file found in install_dir logs directory 3 Examine the situation definition and formula 4 Does the situation contain any wildcard characters against UTF8 columns Yes See corrective actions below 5 Switch between situations to see which one is causing the high CPU Corrective actions e Change the formulation of any situations that are causing excessive processing e For situations with wildcard characters Rewrite the situation using the SCAN strescan function instead of the character by character pattern matching function LIKE For example situations with this simple LIKE process pattern can be rewritten as SCAN process OR Rewrite the situation using non UTF8 columns For example IF LIKE NT_System User_Name_U EQ group can be rewritten as IF LIKE NT_System User_Name EQ group where User_Name is a non UTF8 column and User_Name_U is the corresponding UTF8 column OR Rewrite the situation combining predicates with an OR For example IF LIKE NT_System User_Name_U EQ group can be rewritten as IF VALUE NT System User_Name_U EQ groupA OR VALUE NT_System User_Name_U EQ groupB OR VALUE NT System User_Name_U EQ groupC Table 20 Correcting firewall problems e A firewall interference problem e Communication layer cannot establish communication with the monitoring server Diagnostic
414. ted to IBM Tivoli Monitoring Note The IBM Tivoli Monitoring download image or CD provides application fixpacks for the monitoring agents that are installed from that CD for example the agents for operating systems such as Windows Linux UNIX and i5 OS The migration software for other agents is located on the download image or CDs for that specific monitoring agent such as the agents for database applications If you do not migrate the monitoring agent to IBM Tivoli Monitoring the agent continues to work However you must migrate to have all the functionality that IBM Tivoli Monitoring offers Do presentation files and customized OMEGAMON DE screens for Candle monitoring agents need to be migrated to a new zLinux system The migration from version 350 to IBM Tivoli Monitoring handles export of the presentation files and the customized OMEGAMON DE screens Windows installation frequently asked questions Table 27 Windows installation frequently asked questions Question Answer How can I determine if Windows Security logging is on If the sysadmin account that you use to log on to Tivoli Enterprise Portal is not a Windows Administrator you do not see the security log Windows security logging is not turned on by default Normally data is not collected in the security log unless the Windows administrator turns it on The Record Count 0 in the Windows monitored logs report confirm that securit
415. terprise Monitoring Server on z OS 150 Tivoli Enterprise Portal 117 Tivoli Enterprise Portal Server problems 133 Universal Agent problems 193 Windows 80 troubleshooting features 1 U UAGENT application 56 Unable to create TEPS return code 3 86 Unexpected Signal 4 occurred at PC 0xFEC3FDE4 87 Unhandled Exception 82 uninstallation problems 100 Universal Agent does not start 194 traps 194 upgrade toolkit 47 user groups Tivoli 249 user ID failed user authorization 108 V VMWare 127 W Warehouse database user permission 208 Warehouse Proxy rows not present after upgrade 79 Warehouse Proxy agent 45 205 Warehouse Proxy Agent 132 174 206 failed to export data 207 wikis 253 window Java TM Plug in Control Panel 56 Logon 107 Trace Options 42 Windows Frequently asked questions 67 Local Security Settings 202 Security logging 67 user ID 201 Workflows enable Tivoli Monitoring Services 128 wrules_check 222 Z zOS Tivoli Enterprise Monitoring Server 152 Index 263 264 IBM Tivoli Monitoring Troubleshooting Guide Printed in USA
416. tever alternate path you specified for DB2 installation into this directory Make these corrections and then try running the report again You receive message CTGTRW1000E This error message is often displayed when first entering the Integrated Services Console ISC and clicking on Tivoli Common Reporting on the left side of the page Ignore this error message Click OK on the error message to remove it and the ISC and Tivoli Common Reporting reports will remain operational The report fails to generate because the SQL query was not valid 228 If you have made changes to the reports or to the workspace against which the query is being run you could see SQL query errors Typically this error looks similar to this The following items have errors ReportDesign id 1 Cannot get the result set metadata SQL statement does not return a ResultSet object SQL error 1 DB2 SQL error SQLCODE 206 SQLSTATE 42703 SQLERRMC ASDF SQL error 2 DB2 SQL error SQLCODE 727 SQLSTATE 54098 SQLERRMC 2 200 42703 ASDF SQL error 3 DB2 SQL error SQLCODE 727 SQLSTATE 50098 SQLERRMC 2 200 42703 ASDF Additional information about the error might be found in the SQLERRMC file If you experience SQL errors try generating the report with a different set of parameters If all executions of the report generate SQL errors it is likely that for some reason the running report is not compatible with your database IBM Tivol
417. that is stopped it will generate an error message opt IBM ITM bin tacmd restartagent n zpmaix13 KUX t ul KUICRA006I Are you sure you want to restart the UL agent s that manage zpmaix13 KUL Enter Y for yes or N for no Y KUICRA007I Restarting UL agent s KUICRA009E A problem occurred while restarting UL refer to the following error Chapter 11 Command troubleshooting 185 returned from the server The monitoring server encountered an error while restarting the managed system If the error information returned from the server is not sufficient to help you resolve the error contact IBM Software Support The command opt IBM ITM bin CandleAgent h opt IBM ITM c stop ul did not start or stop agent The command returned a return code Enable Deployment trace logging on the agent machine Contact Customer Service for details on this procedure Collect the following log files On Windows the log kdsmain msg log is located in the CANDLEHOME CMS directory and hostname _ms_ timestamp XX log files are located in CANDLEHOME logs directory On Unix Based systems the logs hostname _ timestamp log and hostname _ms_ timestamp XX log is located in the CANDLEHOME logs directory On the target Managed System Node machine collect the following log files On Windows the logs kdyproc_rasl_ timestamp log and hostname _nt_kntcma_ timestamp XX log are located in the CANDLEHOME tmaitm6 logs directory On Unix systems the log
418. the GSKit product by executing the InsGSKit exe program in the target directory C Program Files ibm directory e Assign the System Environment variable named ICCRTE_DIR to the directory path of the currently installed GSKit product for example C ibm gsk7 e When the error is reported manually configure the Monitoring Agent for DB2 Service Startup Parameters to use the correct user name and password to interact with the DB2 9 1 product Ensure that the InteractsWithDesktop Service is not enabled for this DB2 Agent Service Monitoring Server cannot find your deployment depot If you create a shared deployment repository named depot on the server hosting the deployment depot and you create this repository in a subdirectory of the depot directory the monitoring server will not be able to find your deployment depot and you will receive this message KDY2077E The specified agent bundle depot hubtems depot is not a directory Either the agent bundle depot directory does not exist or it is not a directory The agent bundle depot directory does not exist because no bundles have been added Create the repository at the C IBM ITM CMS level of the directory structure not at the C IBM ITM CMS depot level Then set DEPOTHOME to DEPOTHOME hubtems centralrepository depot The agent installation log on the endpoint indicates that error AMXUT7502E occurred The error AMXUT7512E might occur when running the Distributed Monitoring Upgrade Too
419. the count limit and the max files settings TACMD _ JVM_MAX _MEMORY 256 through to 2048 Increase the java JVM memory size Copyright IBM Corp 2005 2010 235 Tivoli Enterprise Portal Server environment variables The following table lists Tivoli Enterprise Portal Server environment variables Table 53 Tivoli Enterprise Portal Server environment variables Variable Value type Useful for DSUSER1 Yor N Part of a set of 9 variables DSUSER1 DSUSER9 that you can set using the tacmd configureportalserver command For more information see the IBM Tivoli Monitoring Administrator s Guide KBB_RAS1 Trace specification Default tracing level string KBB_RAS1 ERROR Diagnosing client request problems KBB_RAS1 ERROR UNIT ctsql IN ER UNIT ctdata IN ER Diagnosing client or Tivoli Enterprise Monitoring Server interaction problems KBB_RAS1 ERROR UNIT ctsql IN ER UNIT ctdata IN ER UNIT ctemw IN ER UNIT kv4 IN ER Diagnosing SQL generation problems KBB_RAS1 ERROR UNIT ctsql IN ER UNIT ctdata IN ER UNIT ctreport ALL Diagnosing login problems KBB_RAS1 ERROR UNIT ctsql IN ER UNIT ctdata IN ER UNIT ctauth ALL KFW_CMW_SPECIAL_HUB_ Yor N Default is N When set to Y associates situations to the ENTERPRISE N Tivoli Enterprise Portal Server KFW_DATABUS_INPUT_ Yor N Default is N Reduces trace volume by skipping client TRACE _IGNORE_HEAR
420. the subject time period If you cannot you do not have historical collection enabled Start it and try the task again The report fails with a SQLSTATE 22003 arithmetic overflow error In this scenario you have defined report parameters for an OMEGAMON XE or NetView report and clicked Run Report processing ends rather quickly and you see an error messages similar to these CTGTRVO14E The report can not be successfully formatted because it completed with errors reference ID REPORTID 59 OBJECTID 6bee6bee Click on the following link to view the report with the errors Or CTGTRVO11E See the Tivoli Common Reporting log files for more information https localhost 30343 TCR Reports view After seeing this message if you click to the link View the report with the errors you see this message displayed at the bottom ReportDesign id 1 Cannot execute the statement SQL statement does not return a ResultSet object SQL error 1 DB2 SQL error SQLCODE 802 SQLSTATE 22003 SQLERRMC null Finding the statement SQLSTATE 22003 indicates that you are experiencing a SQL arithmetic overflow at the Tivoli Data Warehouse database 230 IBM Tivoli Monitoring Troubleshooting Guide To remedy the problem choose a smaller time period the recommended action or possibly a different metric This change causes the calculations being performed for the report to process less data reducing the chance of overflow Note All occurrence
421. tion attribute atomization This means that for a situation that returns multiple rows from a table an SNMP alert is only sent for the first row returned For example a situation called Low_Disk_Space will trigger if available_disk_space LE 20 might be true for more than 1 disk at a time An enterprise situation would define the disk name as the display item so a separate situation alert would be displayed for each disk with less than 20 free space available The monitoring agent does not support the displayname atomization so the SNMP alert emitter will only emit an SNMP alert for the first row in the disk table where the situation is true This limitation applies to subnodes as well An instance of a subnode agent collects metrics for all subnodes in one table These metrics are filtered by subnode when displayed in the Tivoli Enterprise Portal but situations running against multiple subnodes in an instance are running against a single table If a situation becomes True for 1 subnode an SNMP alert defined for that situation will be emitted but no SNMP alerts will be emitted for any other subnodes for that situation since no further rows are processed in the table Binary Path attribute of the Windows OS agent does not show a value On 64 bit systems with an installed 32 bit Windows OS agent the value of the Binary Path attribute of the NT_Process attribute group is null if the process is running as 64 bit native Chapter 10 Monitori
422. to 600s 10 minutes Set KHD_SRV_STATUSTIMEOUT less than KHD_STATUSTIMEOUT by at least 60 seconds KHD_STATUSTIMEOUT The time in seconds set by default to 900s 15 minutes An export request on the application agent is resent if a status is not received from the Warehouse Proxy agent before the timeout expires KHD_WAREHOUSE_TEMS_LIST A space or coma delimited separated list of monitoring server A warehouse proxy agent serves all the agents reporting to those monitoring servers listed The same monitoring server name must not appear more than once in all the warehouse proxy Tivoli Enterprise Monitoring Server lists in the entire enterprise environment Table 58 lists the Summarization and Pruning agent parameters that you can edit in the KSYENV file on Windows systems and the SY ini file on non Windows systems to customize tracing for the Summarization and Pruning agent Table 58 Summarization and Pruning agent environment variables Variable Purpose KSY_BLACKOUT Specifies a comma separated list of exception times where the Summarization and Pruning agent should not start when using the flexible scheduling The values in the list should be in the format HH MM HH MM where HH must be between 00 and 23 and MM must be between 00 and 59 The starting time must be smaller than the end time of the exception period For example to block the Summarization and Pruning agent from starting between 1 and 2 AM and 5 and 6 PM
423. to collect before contacting Support troubleshooting data on Table 17 Correcting application support problems Lack of application support or application support level mismatch among components monitoring server portal server and monitoring agents Diagnostic and corrective actions See Table 1 on page 13 for instructions on updating application support 28 IBM Tivoli Monitoring Troubleshooting Guide A reflex automation script does not run when it should An action that should have occurred after a situation raised did not occur Preliminary diagnostics If the situation did not raise see A situation does not raise when expected on page 26 26 Table 18 Correcting format and variable problems e The situation or script is not formulated correctly e Authentication or environment variable problems Begin at step 3 Diagnostic and corrective actions 1 Does the system command run correctly from a command line Yes Go on to the next step No Verify that the command typed in is correct 2 Is the length of the command within the limit for your operating system Yes Go on to the next step No The command cannot be executed on this platform You might be able to write a wrapper script to issue the command 3 Are the required user type and environment variables correct Yes Go on to the next step No Include the set command in the shell script or batch script and
424. to return data and consume excessive system resources in some customer environments DB3 DB6 KF1 KF4 STATLTRN TS1 TS3 TS5 and TS6 Each comma delimited no white space value represents a change to the SQL cursor that is executed during data gathering operations within the agent The values are the SQL cursor name For example setting the Extended Parameters field to DB3 TS1 means that the DB3 and TS1 SQL cursor is enabled for Set FREEBYTES to zero Set TSNEXTS to zero and Set MAXEXTTS to zero The SQL cursor name is not case sensitive 2 Recycle the Monitoring Agent for Oracle to recognize these changes to the Extended Parameters value Using the name of the SQL cursor you can look in the korcoll ctl file for the SQL modification that is done when the SQL cursor is enabled The korcoll ctl file is located in the following locations On Windows CANDLE_HOME TMAITM6 On UNIX and Linux CANDLEHOME misc When these cursors are enabled the Monitoring Agent for Oracle displays default attribute values of these cursors in the Tivoli Enterprise Portal meaning the Monitoring Agent for Oracle no longer monitors the attributes of the enabled cursors Chapter 3 Solving common problems 33 Table 21 Correcting Oracle DB Agent problems continued An example of an SQL cursor is displayed below SQL cursor DB3 ARCHIVE LOG DISPLAY SQL SELECT TABLESPACE_NAME UTSNAME SUM BYTES FREEBYTES FROM SYS DBA_FREE_SPACE GROUP BY TABL
425. toring Server main menu select the Create runtime members option to generate the DS 3xxxx Create runtime members job Submit the job and verify good condition codes Recycle the Tivoli Enterprise Monitoring Server Error message KDSMA013 OPEN VTAM for VDM1APPL failed with status 8 occurs in the Tivoli Enterprise Monitoring Server start task CANSDSST default Error message KDSMA013 and the task ends in ABEND S000 U0200 REASON 00000000 KDSMA003 Tivoli Enterprise Monitoring Server data collection server ended successfully Ensure the following he The KDS_VTAMID parameter exists in the amp rhilev rte RKANPAR KDSENV member If it does not exist then ensure that the Tivoli Enterprise Monitoring Server is configured correctly Refer to the Configuring Tivoli Enterprise Monitoring Server on z OS manual for more information about configuring a z OS Tivoli Enterprise Monitoring Server If the KDS_VTAMID VTAM APPLID exists ensure that the Tivoli Enterprise Monitoring Server VTAM major node is activated correctly For more information review the Complete the configuration option on the Configure the Tivoli Enterprise Monitoring Server main menu Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 153 154 IBM Tivoli Monitoring Troubleshooting Guide Chapter 10 Monitoring agent troubleshooting This chapter describes problems you might experience with the monitoring agent deploy tool and monitoring agents If you do
426. tribute these sample programs in any form without payment to IBM Tivoli Monitoring Troubleshooting Guide IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs You may copy modify and distribute these sample programs in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to IBM s application programming interfaces Each copy or any portion of these sample programs or any derivative work must include a copyright notice as follows your company name year Portions of this code are derived from IBM Corp Sample Programs Copyright IBM Corp _enter the year or years_ All rights reserved If you are viewing this information in softcopy form the photographs and color illustrations might not be displayed Trademarks IBM the IBM logo and ibm com are trademarks or registered trademarks of International Business Machines Corp registered in many jurisdictions worldwide Other product and service names might be trademarks of IBM or other companies A current list of IBM trademarks is available on the Web at Copyright and tr
427. tributed to agents connected to the remote Tivoli Enterprise Monitoring Server would be 20 thousand To minimize the amount of CPU processing when a large number of agents connect consider reducing the total number of situations distributed by avoiding distribution of situations that are not being used Some situations including predefined situations have the default distribution set as a managed system list These situations are distributed to all managed systems in the managed system list even if the situation is not being used Limiting the distribution to only managed systems where the situation will be used minimizes the total number of situations distributed from the remote Tivoli Enterprise Monitoring Server and minimizes the CPU processing when a large number of agents connect The distribution specification for a situation can be changed using the Situation Editor or the tacmd editsit command Unable to start the Tivoli Enterprise Monitoring Server after the kdsmain process is terminated abnormally When the kdsmain process is terminated abnormally a stale cms process is left behind This stale cms process prevents the proper startup of the Tivoli Enterprise Monitoring Server The cms process should be killed first and then a startup of the Tivoli Enterprise Monitoring Server should be retried for a successful startup A restart of the Tivoli Enterprise Monitoring Server should be attempted only after verifying the CMS EXE process is
428. ts If the CTIRA_MAX_RECONNECT_TRIES environment variable is specified it is accepted and results in the agent reverting to the Chapter 10 Monitoring agent troubleshooting 161 previous behavior of there being a limit on connection attempts A trace message is also produced indicating that this variable is obsolete If you specify this variable and the number of connection attempts to the monitoring server exceeds CTIRA_MAX_RECONNECT_TRIES the agent attempts to shutdown If the Agent Management Services Watchdog is running it immediately restarts the agent If you want the agent to shutdown when CTIRA_MAX_RECONNECT_TRIES is exceeded this Watchdog process must be disabled Use the AMS Stop Management action to disable this watchdog process Agent goes offline after removing history path Deleting the directory where historical collection occurs is not supported If this is done the directory must be manually recreated Override button is not present for a situation A situation cannot be overridden if it contains repeated combinations of attributes or operators Agent s Management Definition View columns are not showing data There will be no data shown in any columns for which an element in the corresponding CAP file is empty An element is left empty when it is not needed to manage a particular agent by Agent Management Services For monitoring agents for example the following columns are left empty e Startup Script
429. ttpd command completes the information button works Refer to the IBM Tivoli Enterprise Console 3 9 Installation Guide for more information about the wstarthttpd command Situations return to acknowledged 224 The default action of the omegamon r1s rule set is to reject acknowledgement expirations that are received from the Tivoli Enterprise Monitoring Server and to then acknowledge the situation again on the Tivoli Enterprise Monitoring Server This action of the omegamon r1s rule set might be changed to accept acknowledgement expirations from the Tivoli Enterprise Monitoring Server and therefore leave the situation open in the Tivoli Enterprise Monitoring Server Refer to the IBM Tivoli Monitoring Administrator s Guide for information on changing parameters of the omegamon r1s rule set IBM Tivoli Monitoring Troubleshooting Guide tacmd refreshTECinfo t all shows no results on console The Tivoli Enterprise Monitoring Server facility used DS START command to trigger the refresh of the EIF info does not give a return code So it is not possible to give feedback as to the success or failure of the operation back to the CLI For a result of the refresh look in the Tivoli Enterprise Monitoring Server log or the Universal Message Console on the Tivoli Enterprise Portal Pre existing situations do not display in the Tivoli Enterprise Portal after performing an initial installation of IBM Tivoli Monitoring v6 2 When installing
430. tuation editor cannot display advanced advice help files After acknowledging a situation event nd selecting the link for that situation you might receive a message Password problem using the LDAP Security option 122 on Active Directory system There is a memory leak in the Tivoli Enterprise Portal browser client when the number of workspace switches increases Help index and search text entry fields are disabled Java exception logging onto the Tivoli Enterprise Portal from a browser On Linux IBM Tivoli Enterprise Monitoring Agent topics do not display in the Help Contents or Index tabs Tivoli Enterprise Portal Navigator items are listed in an unexpected order Clicking on the Timespan icon for one view w brings up the data for another view HEAPDUMPs and JAVACore files are e placed on the desktops when running Tivoli SE Portal 124 125 125 in browser mode Java errors occur with the IBM vi 4 2 JRE Web Portal Client does not work with Sun JRE Tivoli Enterprise Portal has high memory usage and poor response time 12 Tivoli Enterprise Portal has high memory usage Data is not returned to the Tivoli ag sia Portal client DirectDraw thread loops infinitely causing poor Tivoli Enterprise Portal performance Contents 120 120 121 a 121 122 122 122 123 123 123 123 124 124 126 126 127 127 Vv Wor
431. ue will be resolved Agent names and icons are displayed incorrectly When installing portal server support from images older than v6 2 2 the support installation might fail with no symptom leading to names and icons related to the agent being displayed incorrectly in the Tivoli Enterprise Portal If this occurs reconfigure the portal server in the Manage Tivoli Enterprise Monitoring Services window by right clicking the portal server entry and then clicking Reconfigure 64 bit monitoring agents are not started If a monitoring agent is installed as an Agent Template an instance needs to be created before the agent can be started For instance you install and start a v6 2 2 Fix Pack 1 64 bit monitoring agent and perform a local installation of a pre v6 2 2 Fix Pack 1 agent template for example the v6 2 Agentless Monitoring for Windows Operating Systems After the installation ends the 64 bit monitoring agent is not running There is no trace of the monitoring agent attempting to start in the main installation log Copyright IBM Corp 2005 2010 155 The following conditions will bring about this issue e installations are performed locally e v6 2 2 Fix Pack 1 monitoring agents are 64 bit e pre v6 2 2 Fix Pack 1 monitoring agents are a template To ensure that these monitoring agents are started manually start them from Manage Tivoli Monitoring Services Examples of monitoring agents from before v6 2 2 Fix Pack 1 for wh
432. umber of transports configured for the hub Tivoli Enterprise Monitoring Server KDCSSITE e For a hub Tivoli Enterprise Monitoring Server each LLB entry must be identical to the corresponding GLB entry in the Server list As in the Server list for a hub Tivoli Enterprise Monitoring Server example LLB entry 1 is the same as GLB entry 1 and LLB entry 2 is the same as GLB entry 2 e For a remote Tivoli Enterprise Monitoring Server the opposite is true each LLB entry must be different than the corresponding GLB entry in the Server list As in the example Server list for a REMOTE Tivoli Enterprise Monitoring Server LLB entry 1 is different than GLB entry 1 LLB entry 2 is different than GLB entry 2 Repairing errors in the Server list Errors in SNA initialization might be name mismatches Examine the LLB entries and the GLB entries for the Omegamon Platform address space for typographical errors The VTAM network ID is victim of frequent error Typically the VTAM network ID the first component of the SNA socket address ATOOENO1 in the example above is the same for ALL entries While it CAN differ typically it does not A difference between the LLB VTAM net ID and the GLB VTAM net ID is often an error in member KDCSSITE of RKANPAR Local Location Broker service initialization An intrinsic part of Remote Procedure Call architecture is the location broker RPC servers callers of rpc__listen publish their service and the address of this
433. user gt lt default gt For the DB2 monitoring agent instances to run as the instance owner IDs lt productCode gt ud lt productCode gt lt instance gt lt name gt db2inst1 lt name gt lt user gt db2instl lt user gt lt instance gt lt instance gt lt name gt db2inst2 lt name gt lt user gt db2inst2 lt user gt lt instance gt For the Websphere MQ monitoring agent instances to all run as the mqm user ID and for the default instance to not be started lt productCode gt mq lt productCode gt lt default gt lt user gt mqm lt user gt lt default gt lt instance gt lt name gt None lt name gt lt autoStart gt no lt autoStart gt lt instance gt Repeat step 2 for each agent or agent instance that you want to start as a specific user ID Save the file Run install _dir bin UpdateAutoRun sh as root user Troubleshooting Guide After installing and configuring a monitoring agent it fails to start If the SecureMode file is in the registry directory SecureMain was run in environment This does not allow the monitoring agent to start if you try to start it without root privileges See the IBM Tivoli Monitoring Installation and Setup Guide for instructions on how to have monitoring agents work properly with SecureMain in place situation_fullname slot missing for delete events If you create a situation that has a long name longer than 32 characters distribute the situation to an agent and then d
434. uter where the Tivoli Enterprise Portal Server is installed be sure you are logged on to Windows with an ID having administrator authority From your Windows desktop select Start gt Programs gt Tivoli Monitoring Services gt Manage Tivoli Enterprise Monitoring Services Right click the Tivoli Enterprise Portal Server and select Advanced gt Utilities gt Build Tivoli Enterprise Portal Server Database from the menu Click DB2 to open the Tivoli Enterprise Portal Server Data Source Config Parameters window Enter the Admin Password Enter the new Database Password for the Tivoli Enterprise Portal Server Database User ID Receive First Steps error at the end of a DB2 installation At the end of my DB2 install I received an error that First Steps could not initialize because a supported browser was not present on my system Here is the link to the full publib boulder ibm com infocenter db2luw v9 topic com ibm db2 udb doc c0025020 htm resultof 22 66 69 72 73 74 22 20 22 73 74 65 70 procedure on how to resolve this problem 73 22 20 22 73 74 65 70 22 20 22 62 72 6f 77 73 65 72 22 20 Tivoli Enterprise Portal Server cannot connect to the database 202 If an error message displays indicating the connection failed for security reasons the end user is logged in to the server with a userid with administrator authority but is logged into the local domain instead of locally on the machine The u
435. ver InstallShield displays the error 1607 Ufable to install InstallShield Scripting Runtime during installation on Windows from a network mounted drive Installation on a Windows 2003 server r fails with Error Number 0x80040707 Extracting the nls_replace script causes remote deployment to fail sa Upgrade tool Eee agent to the wrong directory Deploying an agent instance without providing any configuration properties causes KUICAR020E error Uninstallation i is hot available foi Application Support on Windows systems Problems installing directly from the zip file r Installation hangs or loops after presenting initial splash screen UNIX based system installation problems andl resolutions Message is received about the Korn Shell after running the install sh file AIX stat_daemon memory leak 3 43 UNIX and Linux install sh command fails with error code 99 and error code 4 Receive KUIC02101W error Receive JVMDG080 or JVMXM012 Javale errors On HP UX systems where the hostname does not equal the nodename the upgrade installation fails to stop running processes EIF Slot Customization does not work on upgraded zlinux systems 5 Running of the KfwSQLClient binary Falso on Linux and AIX systems Failed to attach to the DB2 instance db2inst1 ERROR Unable to create TEPS return code 3 Installation on SLES9 terminates with install sh failure KCI1008E terminating license declined Command line inter
436. ver by starting a browser session to the service console http server 1920 Ensure the browser View Status is checked and then move the cursor over each link The port involved will be seen in the status line like this Service Point cnp gt IBM Tivoli Monitoring Service Console gt IBM Tivoli Enterprise Portal Web Client Service Point nmp180_hd gt IBM Tivoli Monitoring Service Console In this case the cursor was on the service console link under Service Point nmp180_hd and that was the process running the internal web server If a firewall rule is in place between the tacmd login process and the server running the hub monitoring server and SOAP the tacmd login command might fail The tacmd logic attempts to use the base port connected with the IBM Tivoli Monitoring Web Services link If that is unavailable it uses the 1920 port That will fail if the 1920 process is not the same as the monitoring server process If this condition occurs it can be resolved by stopping all IBM Tivoli Monitoring processes on that server starting the hub monitoring server and then starting up the rest of the IBM Tivoli Monitoring processes When things are running again the tacmd login begins to operate as expected and SOAP are not able to connect If two instances of the Hub Tivoli Enterprise Monitoring Server are started under different user IDs root plus one other tacmd and SOAP are not able to connect When the problem occurs it
437. vironment information 2 Verify that the values for the high level qualifiers are correct If the high level qualifiers values are not the correct use the following steps to unlock the SMP E high level qualifiers a Run amp shilev INSTLIB b On the Installation tool main menu select Services and utilities gt Unlock SMP E high level qualifiers c Specify Y on the Unlock SMP E high level qualifiers field d Edit the high level qualifier modifications on the Specify installation environment information panel 3 Go to the RTE main menu and regenerate the RTE Load job Chapter 9 Tivoli Enterprise Monitoring Server troubleshooting 151 152 If the zOS CT Engine stops the Tivoli Enterprise Monitoring Server must be recycled If anything causes the zOS CT Engine to stop then the Tivoli Enterprise Monitoring Server must be recycled The problem that caused the stoppage also needs to be addressed KDS Parameters not generated from the batch parm deck If you try to clone an RTE through Create Batch Mode parameters processing KDS variables for example KDS_CMS_TYP are not generated in the output A possible workaround is to manually add the missing parameters The problem is that the KCITVARS ISPF table is out of order The procedure to sort the table is 1 Back up INSTDATA 2 In TKANCUS create a CLIST called XSORT The contents will be as follows PROC 0 SET SORTLIST amp STR CIGPRF C A CIGPRD C A CIGVAR C A
438. voli Monitoring Service Console link associated with the desired process service point name 4 When the log in window opens click OK In secure environments you need a valid user ID and password to proceed Upon successful login the IBM Tivoli Monitoring Service Console opens with three areas e Header e Command Results e Command Field You can now issue IBM Tivoli Monitoring Service Console commands in the command input area For a list available commands type a question mark and click Submit The IBM Tivoli Monitoring Service Console performs user authentication using the native OS security facility If you use the IBM Tivoli Monitoring Service Console on z OS systems your user ID and password are checked by the z OS security facility RACF SAF If you use the IBM Tivoli Monitoring Service Console on Windows systems then you must pass the Windows workstation user ID and password prompt This is the rule except for instances of a NULL or blank password The IBM Tivoli Monitoring Service Console never accepts a NULL or BLANK password A password is always required to access the service console Blank passwords even if correct cannot access to the service console Even if a user ID is allowed to login to the operating system without a password access to the service console is denied Create a password for the user ID that is being used to login to the service console Blocking access to the IBM Tivoli Monitoring Service
439. where work directory is the directory to be used to collect the data Do not specify the root directory of a disk partition as the work directory If not specified TEMP is used options are any of the following in any order or case noapp specifies that application event log data is NOT to be collected nosec specifies that security event log data is NOT to be collected nosys specifies that system event log data is NOT to be collected noevent specifies that NO event log data is to be collected equivalent to specifying noapp nosec and nosys nohist specifies that history files are NOT to be collected nologs specifies that ITM log trace files are NOT to be collected noprompt specifies that all interaction with the user is suppressed options may also be prefixed with instead of Note On UNIX platforms the combined file name and path name cannot be larger than a total of 255 characters On AIX systems the file name must be less than 100 characters and path names must be less than 155 characters The z OS version of pdcollect is a TSO clist that is installed in RKANSAM KMSPDCOL You can copy this clist into a clist library of your choice renaming it if you desire Once the clist resides in a dataset accessible through SYSPROC in a TSO logon procedure then it can be invoked by name In the following example the KMSPDCOL name was retained IBM Tivoli Monitoring Troubleshooting Guide
440. which the process was started nn Represents the circular sequence in which logs are rotated Ranges from 1 5 by default though the first is always retained since it includes configuration parameters IBM Tivoli c tmp itmsynch logs tmp itmsynch logs synch_ Enterprise synch_trace log trace log Console Situation c tmp itmsynch logs tmp itmsynch logs synch_ Update synch_msg log msg log Forwarder eer Note IBM Tivoli Enterprise Console Situation Update Forwarder logs are created on the IBM Tivoli Enterprise Console server The log files are managed as follows e On Windows the log file name includes a time stamp in hexadecimal format By default the logs are stored in the installation path for IBM Tivoli Monitoring The following is an example of a log file name that includes the time stamp in hexadecimal format ibm kpmn803v01_cq_472649ef 01 10g e On UNIX based systems the log file name includes a time stamp The UNIX based systems RAS1 log files are stored in the logs directory The following is an example of a log files name that includes the time stamp f50pa2b_ux_1112097194 1og e On Linux systems using the Firefox browser candle_home java deployment log Note When you communicate with IBM Software Support you must capture and send the RAS1 log that matches any problem occurrence that you report 38 IBM Tivoli Monitoring Troubleshooting Guide Installation log files The following table lists and describes the
441. with Desktop box 4 Click OK to open a command prompt window when the Tivoli Enterprise Portal Server is started Internal Tivoli Enterprise Portal Server commands display in the command prompt window 5 Verify that the Tivoli Enterprise Portal Server service is started The Tivoli Enterprise Portal Server is started when the following messages display KfwServices lt timestamp gt KFW1002I Starting Service Configuration v1 0 KfwServices lt timestamp gt KFW1003I Started Service Configuration v1 0 KfwServices lt timestamp gt KFW1002I Starting Service Situation v1 0 KfwServices lt timestamp gt KFW1003I Started Service Situation v1 0 KfwServices lt timestamp gt KFW1002I Starting Service Automation v1 0 KfwServices lt timestamp gt KFW1003I Started Service Automation v1 0 KfwServices lt timestamp gt KFW1002I Starting Service CEV v1 0 KfwServices lt timestamp gt KFW1003I Started Service CEV v1 0 KfwServices lt timestamp gt KFW1002I Starting Service Startup Complete v1 0 KfwServices lt timestamp gt KFW1003I Started Service Startup Complete v1 0 KfwServices lt timestamp gt KFW1020I x x x Waiting for requests Startup complete x x 6 Do one of the following e If it is stopped start the Tivoli Enterprise Portal Server Ifit is started recycle the Tivoli Enterprise Portal Server 112 IBM Tivoli Monitoring Troubleshooting Guide
442. workspace in the navigation tree is incorrect The link to the Queue Statistics workspace in the navigation tree opens the Open Queue Handles workspace instead of the Queue Statistics workspace Use the following steps to correct the link 1 Open the Queue Statistics workspace from the navigation tree 2 Ensure that Queue Statistics workspace opened instead of the Open Queue Handles workspace 3 From the Tivoli Enterprise Portal select Edit gt Properties 4 Place a check in the Assign as default for this Navigator Item box Some rows do not display in an upgraded table You might not see all tables after upgrading the Warehouse Proxy to IBM Tivoli Monitoring V6 1 because some tables might be corrupted Do the following to find the errors that occurred during the upgrade 1 Edit the KHDRAS1 Mig Detail log file 2 Search for the word EXCEPTION The KHD_MAX_ROWS_SKIPPED_PER_TABLE environment variable allows you to skip bad data KHD_MAX_ROWS_SKIPPED_PER_TABLE indicates the number of rows per table to skip to migrate if the data that needs to be inserted is incorrect When this number is reached migration of the table is aborted Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal Server automatically start after running Application Support Installer After running the Application Support Installer the Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal automatically start even if they were not running
443. wser Administrator client Application Data Java Dep oyment log plugin142 trace Copyright IBM Corp 2005 2010 35 Table 22 Location of log files on the Tivoli Enterprise Monitoring Server continued Component Windows UNIX based systems Tivoli Enterprise Portal desktop client install_dir CNP kcjerror log install_dir CNP kcjrasl log When launched via Java Web Start SUSERPROFILE amp Application Data IBM Java Deployment log javawsnnnnn trace where nnnnn is a unique randomly generated numeric suffix to support generational logs i e the last generated log will not be overlayed by the most current execution of Tivoli Enterprise Portal using Java Web Start This is in contrast to the Tivoli Enterprise Portal Browser client which has a fixed name and is overlayed with each execution cycle install_dir logs hostname_PC_timestamp log where install_dir Specifies the directory where Tivoli Enterprise Portal Server was installed hostname Specifies the name of the system hosting the product PC Specifies the product code cq for the Tivoli Enterprise Portal Server timestamp A decimal representation of the time at which the process was started When launched via Java Web Start user home java deployment log javawsnnnnn trace where nnnnn is a unique randomly generated numeric suffix to support generational logs i e the last generated log will not be overlayed b
444. x of the distributed f ent s User s Guide or the Troubleshooting Guide of the z OS monitoring agent No See step Chapter 3 Solving common problems 27 Table 16 Correcting situation specific problems continued 4 Look in the log of the monitoring server to which the agent is attached Search for the situation name and look for any errors logged Catalog errors message return codes 202 and 209 Ensure the application support is installed at the monitoring server Message K041046 is missing situation failed to lodge message K041039 Error in request MCS_Sit Status 1133 Reason 1133 K041039 Error in request MCS_Sit Status 1131 Reason 1131 4558E8EF 0079 11A4 ko4sitma cpp 782 IBInterface lodge error Lodge lt 1131 gt 4558E8EF 007A 11A4 ko4ibstr cpp 659 IBStream op_1s_req IB Err 1131 4558E8EF 007B 11A4 ko4sit cpp 658 Situation slice Sit MCS Sit Unable to lodge giving up K048156 Not able to start monitoring for situation MCS_ Sit SITMON IB Lodge errors Attribute file is incorrect wrong version or missing and the RULE could not be created A value of 1133 or 1203 leads to a value of 1131 A value of 1145 generally means that the referenced situation either has been deleted or has not been distributed correctly define ERR_LODGEERROR 1131 Bad lodge request define ERR_NOATTRIBUTE 1133 No attribute found define ERR_DUPLICATEINSERT 1144 Duplicate record exists define ERR
445. y The following setting should be added to the CANDLEHOME platform iw profiles ITMProfile config cells ITMCel1 nodes ITMNode servers ITMServer server xml file The section genericJvmArguments DKFW_DATA products e6 itm 1s3263 cq data gt should be changed to genericJvmArguments Djava compiler NONE DKFW_DATA products e6 itm 1s3263 cq data gt The portal server must be restarted to have this change go into effect Chapter 5 Installation and configuration troubleshooting 85 86 Running of the KfwSQLClient binary fails on Linux and AIX systems Depending on the system the following environment variable should be set before running the command e Linux LD_LIBRARY_PATH e Aix LIBPATH The variable value can be taken from the CANDLEHOME config cq config file Failed to attach to the DB2 instance db2inst1 ERROR Unable to create TEPS return code 3 While installing a Tivoli Enterprise Portal Server on a UNIX based system and using a DB2 database the following error message is displayed where db2inst1 is the supplied name of the DB2 instance Failed to attach to the DB2 instance db2instl ERROR Unable to create TEPS return code 3 Ensure that the DB2 instance is started by running the db2start command as the instance user su db2instl db2start Installation on SLES9 terminates with install sh failure KCI1008E terminating license declined On systems when LAP cannot run and Java does not function a b
446. y logging is not turned on Copyright IBM Corp 2005 2010 67 Table 27 Windows installation frequently asked questions continued Question Answer How can I diagnose problems with product browse settings 1 Click on Start gt Programs gt Tivoli Monitoring gt Manage Tivoli Enterprise Monitoring Services to display the Manage Tivoli Enterprise Monitoring Services window 2 Right click the Windows agent and select Browse Settings A text window displays 3 Click Save As and save the information in the text file If requested you can forward this file to IBM Software Support for analysis UNIX based systems installation frequently asked questions Table 28 Frequently asked questions for UNIX based systems installation Problem Solution The product was installed as root Without re installing the product how can I change from root to another ID If you installed and started the agent as root the files do not have correct permissions so the result is unpredictable For this reason do not use root ID either to install or start the UNIX based systems agents Create a user ID with all the authority and permissions to install run or use any other ID other than root As root run the command UnSetRoot which is located under install_dir bin directory This script resets all the files under the install_dir directory owned by root UnSetRoot h CANDLEHOME userID After executing
447. y for application layer codes such as telnet FTP and the IBM Tivoli Monitoring Server Tivoli Enterprise Portal Server and the Tivoli Enterprise Monitoring agents Initializing the TCP IP service TCP IP services for this address space are available if any of the following messages are present KDE1I_OpenTransportProvider Transport opened socket ip tcp KDE1I_OpenTransportProvider Transport opened socket ip pipe KDE1I_OpenTransportProvider Transport opened socket ip udp These messages are only displayed when KDC_DEBUG Y is active in the environment KDC_DEBUG Y must be added to member KDSENV of RKANPAR or the appropriate initialization member to obtain the level of tracing required to have these messages echoed to the RAS1 log IF KDC_DEBUG Y is set and if none of these messages are in the log then initialization of the TCP IP service failed Repairing TCP IP service initialization failures Use the following steps to ensure the TCP IP transport service is available Note Failure at any of the following prevents the TCP IP service from initializing in the address space 1 Ensure the INITAPI service is successful See The INITAPI call on page 89 2 Ensure the Name Resolution is successful See Name Resolution on page 90 3 Ensure the first SEND ran without error See The First SEND on page 91 The INITAPI call IBM s implementation of TCP IP requires that an address space perform an INITAPI before issuing a
448. y the most current execution of Tivoli Enterprise Portal using Java Web Start This is in contrast to the Tivoli Enterprise Portal Browser client which has a fixed name and is overlayed with each execution cycle Tivoli Enterprise install _dir logs hostname_PC_ install _dir logs hostname_PC_timestamp log Monitoring HEXtimestamp nn log Server where where install_dir Specifies the directory where Tivoli Enterprise install_dir g ve Portal Server was installed Specifies the directory where Tivoli Enterprise hostname Monitoring Server was Specifies the name of the system hosting the installed product PC Specifies the product code PC Specifies the product code cq for the Tivoli ms for Tivoli Enterprise Enterprise Portal Server Monitoring Server timestamp HEXtimestamp A decimal representation of the time at which the A hexadecimal process was started representation of the time at which the process was started nn Represents the circular sequence in which logs are rotated Ranges from 1 5 by default though the first is always retained since it includes configuration parameters 36 IBM Tivoli Monitoring Troubleshooting Guide Table 22 Location of log files on the Tivoli Enterprise Monitoring Server continued Component Windows UNIX based systems Monitoring install_dir tmaitm6 logs install_dir logs hostname_PC_timestamp log agents hostname_PC_HEXtimestamp nn log where where install_d
449. y the partial installation of IBM Tivoli Monitoring You must either completely reinstall the previous release and then upgrade to IBM Tivoli Monitoring or just install IBM Tivoli Monitoring without attempting to upgrade Check the Add and Remove Programs tool to see if either your previously installed Candle OMEGAMON or IBM Tivoli Monitoring is available If neither are available see Neither products are available in the Add and Remove If one is available see One product is available in the Add and Remove Programs tool on page 104 If both are available Both products are available in the Add and Remove Programs tool on page 105 Neither products are available in the Add and Remove Programs tool Use the following steps if neither Candle OMEGAMON or IBM Tivoli Monitoring exists in the Add and Remove Programs tool Chapter 5 Installation and configuration troubleshooting 103 a Ti 12 13 14 15 16 17 Open the Windows Explorer and navigate to the IBM Tivoli Monitoring installation directory By default the installation location is C IBM ITM Install C IBM ITM Instal 111M C Candle Install or C Candle Instal11TM Launch the Manage Tivoli Enterprise Monitoring Services utility by double clicking the KinConfg exe file located in either the Install or InstallITM subdirectory Launch the KinConfg exe from the InstallITM directory if possible If any agents the portal server
450. y_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksysql ALL Traces every SQL statement being executed Affects the content of the Java based internal tracing hostname_sy_java_timestamp n 109 KBB_RAS1 ERROR UNIT ksysql1 ALL Same as UNIT ksysql ALL but also includes all the parameter values used in the parameterized statements Notes 1 The following settings UNIT ksy3 ALL or UNIT ksy4 ALL or UNIT ksy5 ALL produce a high volume of trace output 2 By default the Java based internal trace hostname_sy_java_timestamp n 10g wraps at 5 files and each file contains 300000 lines To change the defaults use the following settings in the KSYENV Windows or sy ini UNIX files KSZ_JAVA_ARGS Dibm tdw maxNumberDetai 1 TraceFi les lt A gt Dibm tdw maxLinesForDetai 1 TraceFi 1e lt B gt where lt A gt Specifies the maximum number of Java based internal trace files that can exist at any one time for a single launch lt B gt Specifies the maximum number of lines per Java based internal trace file Setting the trace option for the tacmd commands For Windows systems manually edit the KUIENV file in the CANDLEHOME directory with the standard KBB_RAS1 statement to include the following KBB_RASI ERROR UNIT ksh all UNIT kui all IBM Tivoli Monitoring Troubleshooting Guide On UNIX systems manually edit the CANDLEHOME bin tacmd shell script to add a line like the following example KBB_RAS1 ERROR UNIT ksh all UNI
451. yed in the Service Console list that are not accessible from that user interface 144 tacmd login fails after hub monitoring s server is recycled 144 tacmd and SOAP are not able t coni a 145 The system crashes when attempting a bulk import or export command 145 The Tivoli Enterprise Monitoring Server fails to start but then does after a reboot 145 Remote Tivoli Enterprise Monitoring Server lost connection to the hub Tivoli Enterprise Monitoring Server and all agents display offline 146 After the set timeout the Tivoli Enterprise Monitoring Server is still pending 146 Providing the wrong path to configuration files during LDAP configuration causes the Tivoli Enterprise Portal login window to hang 146 Crash on Linux remote Tivoli Enterprise Monitoring Server during hub failover to Hot Standby 3 HUB Tivoli Enterprise Monitoring Serve quiesce prevents the display of the data collected by the attached Tivoli Enterprise Monitoring Agents During installation of a remote Tivoli Enterprise Monitoring Server on a Windows system the agent support is applied but fails Using a Deploy Group with a or updateAgent commands a Tivoli Enterprise Monitoring Servet requires restart if you issue itmcmd server stop start commands when you are already logged on Log indicates Hub Tivoli Enterprise Monitoring Servers are down when they are up The Platform view in the Manage Tivoli E
452. you can highlight all error events in red or show all events from a specific component in blue Highlighting is similar to filtering but instead of eliminating data from a view you can highlight the relevant information within the full list of events e Filtering log records You can narrow the scope of a problem and the data shown based on filter criteria Examples of filter criteria include time stamps severity component and server e Finding log records You can search for specific information in a log file For example you can search to see events related to interaction with a specific server or user For more information about how to organize the data in the Log and Trace Analyzer online help search for the Analyzing log files topic Filtering Sorting Finding and Highlighting is a subheading in this topic In addition there are some other topics in the online help that you might find useful e When trying to correlate log files from multiple servers the time clocks on those servers can be out of sync This synchronization problem could be something simple like different time zones or more subtle such as a clock being a few milliseconds off from another server s clock The Log and Trace Analyzer imbeds a function to synchronize the time between multiple log files by allowing you to adjust the time stamps in a log file For more information refer to the topic titled Synchronizing time of log records for distribut
453. ystem 3 Stop all the IBM Tivoli Monitoring components from the Manage Tivoli Enterprise Monitoring Services window 4 Rerun the upgrade installation now with the remote Tivoli Enterprise Monitoring Server in the stopped state The upgrade installation is complete IBM Tivoli Monitoring Troubleshooting Guide On Solaris systems GSKit installation fails to install If you have previously installed GSKit and then manually removed it the installation might fail Perform the following actions to remove the two gskit installations and then continue the installation as normal pkgrm gsk7bas pkgrm gsk7bas64 Remote configuration of deployed Monitoring Agent for DB2 agent fails The following message is returned when running the tacmd addsystem command The agent action SETCONFIG failed with a return code of 1073741819 for product code ud Remote configuration and installation of a database agent requires that IBM Global Security Kit GSKit be installed in directory C Program Files ibm gsk7 or that the GSKit directory be defined in the Windows System environment variable ICCRTE_DIR DB2 9 1 installs the GSKit package in C ibm gsk7 and the ICCRTE_DIR environment variable is not exported as a System environment variable Therefore tacmd addsystem remote configuration processing cannot execute and results in the failure message reported to the user Choose one of the following resolutions that best fits your environment e Install

IBM Tivoli Monitoring: Troubleshooting Guide

Contents

Download Pdf Manuals

Related Search

Related Contents