Home

Cisco Systems 78-16845-01 Network Router User Manual

1. 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems When any add or modify connection request is issued an SNMP VarBind is set on the switch via cmgrd The switch can reject this VarBind with this error The reason for the error is that cmgrd sent an invalid value for one of the MIB objects The CTM DE should do the investigation for this error as it indicates that there is a problem in the data that cmgrd either received from the clients or internally mapped from the other side of the connection Try the same connection from both the Connection Proxy and CM GUI interfaces and note the behavior from both of the clients If the provisioning result from both clients is the same it tends to indicate that this is either a cmgrd issue or a sdbroker vpi vci issue Defect Information The cmgrd log ConnProxy cmsvr log and sdbroker logs should be collected Possible alternative workaround If you are doing an add and you are using default values then there is no workaround However if you are doing a modify you can change the value of the object being modified and retry the connection Also if this is a sdbroker vpi vci issue retry the connection because sdbroker does not reuse the previous vpi vci combination J 10 8 4 18 Cmgrd aAddition Errors Provisioning a Connection Results in Object Exists on the Agent or Specified VPI VCI not availa
2. w__ Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting Step 4 J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems W c Verify whether or not the alarms are aggregate port alarms If so this is expected behavior and there is no defect If there are alarms that have greater severity then what is shown on the platform and they are not aggregate port alarms continue to the next step to see whether the alarm is in the database Verify whether the database has the correct alarm state Refer to the Cisco Transport Manager Release 6 0 Database Schema for information on what table to look up for this particular entity type Defect Information Collect the following information for further analysis e NMsServer log e nmControl dump e CMSCclient log on client machine Possible alternative workaround Open a new GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report J 10 5 5 5 Alarm List Shows Alarm that Does Not Exist on Platform Step 1 Step 2 There are several alarms that are correlated by NMServer and are not handled by the switch This is the case because the NMS keeps track of alarm conditions that may not be relevant to the switch alone but are relevant to the switch and the NMS One example of such an alarm is the node sync up state The switch itself is not interested in the fact that the NMS may not b
3. 2 Use nmControl and perform Resync Node specifying the node ID J 10 5 4 4 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows a Card Line or Port that Is Not Present in a Node Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 An extra card line or port element is shown incorrectly on the Configuration Center Chassis View Diagnostics Center or Statistics Report Check whether the node is synced up This can be done by using the command selnd on the CTM CLI If the node has not synced up then wait for it to do so Check whether the card line or port is populated in the database Refer to the Cisco Transport Manager Release 6 0 Database Schema If the element is present in the database but does not match what is shown in the GUI see J 10 4 Equipment Management Problems page J 40 If element is not in the database check the NMServer cache This can be done by using the command nmControl on the CTM CLI Check the nmControl lt pid gt dump Use nmClient and verify whether getChildren for FDN returns the element in the nmClient lt pid gt dump file If element is not present in the NMServer cache open a new GUI and verify whether the extra element is shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Possible alternative workarounds 1 Open anew GUI a
4. Cisco Transport Manager Release 6 0 User Guide P32 i 78 16845 01 AppendixJ Troubleshooting J 10 8 Connection Management Problems W e opt svplus log configserver log Possible alternative workaround None J 10 7 2 8 Config Server Reports Error Messages This section describes the various kinds of errors that could happen on the config server Step 1 Open the opt svplus log configserver log file Step2 If data type mismatch occurs an error message is displayed in the GUI and the corresponding error message seen in configserver log file is as follows 24560 9 07 49 14 CRIT SnmpCommExcept ion 2 NOT_INTEGER_VALUE SnmpVar getInteverValue incorrectly called for ASN type 4 In this case the data type of the MIB object defined in the XML file is not the same as the data type of the MIB object definition Contact CTM engineers with defect information Step3 If you enter an invalid value an error message is displayed in the GUI with the parameter name appended The corresponding error message seen in configserver log file is as follows 17252 6 08 31 20 ERR SnmpCommException 3 ERR_BADVALUE Snmp Error 3 Bad Value object index 2 Step4 If you try to up add more than one element an error message is thrown in the GUI and the corresponding error message thrown in the opt svplus log configserver log file is as follows 23868 6 14 01 09 ERR SnmpCommException 3 ERR_GENERR Snmp Error 5 Gener
5. Verify whether the entity node card line or port that caused this error is supported in the CTM version being used If this error is seen for a supported node card go to step 2 Open the opt svplus log DCServer log file and look for the message information from when this error occurred A typical message in the log looks like this ERR Fatal Error at file line 0 char 0 Message An exception occurred Type RuntimeException Message The primary document entity could not be opened Id opt svplus xml diagcenter XXX XXX XXX xml lt someNumber gt lt x gt lt someTime Stamp gt ERR InternalError XML Parsing Error If the xml filename mentioned previously has two consecutive hyphens for example ABC XYZ xml or if it has a preceding hyphen for example ABC xml or terminates with a hyphen before the file extension for example ABC xml proceed to step 3 where investigating incorrectly formatted XML filenames is discussed See if the xml file mentioned in the log message as shown in the previous substep exists under the opt svplus xml diagcenter directory If it does not exist contact the CTM engineers with the defect information Perform the following substeps to investigate incorrectly formed XML filename strings The format of XML filenames is lt Platform gt lt Card gt lt InterfaceType gt lt SomeEntityName gt xml This format is generic with a few exceptions Also note that Platform and Interf
6. echo select from the node where node_id lt NodeId gt dbaccess c Get the card table information for the card using the following command then save it in a file echo select from the card where node_id lt NodeId gt and slot lt Slot gt dbaccess Cisco Transport Manager Release 6 0 User Guide P70 78 16845 01 Appendix J Troubleshooting J 10 6 Chassis View Problems W d Save the log as CMSCclient log to your local drive e Take a copy of the chassisview jar from the opt svplus java jars cwm directory You need to check that gif files used to draw the lines are available in the jar Possible alternative workaround Select the cards from the tree view to launch other applications J 10 6 4 DCA DCB Status Not Displayed on PXM Cards DCA DCB status is always grayed out in Chassis View DCA DCB status information is not received from the switch so the status is not displayed or updated J 10 6 5 Ethernet Status Not Updated on PXM Cards Ethernet status is received only when the Chassis View is launched Dynamic changes in states of the Ethernet status are not updated in Chassis View J 10 6 6 HIST CBRX and CBTX Status Not Updated in MGX Nodes For MGX nodes only CPUOK LED status is updated in Chassis View Chassis View does not manage the LEDs for HIST CBRX or CBTX J 10 6 7 RPM Card Status Not Updated Dynamic event updates are not generated for RPM cards on MGX PXM1 based nodes so the
7. Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting Step 2 Step 3 J 9 6 XR 12000 BGP and CDP Neighbor Discovery Issues W Enter the following commands to verify that the router s running configuration contains the correct PM configuration performance mgmt statistics bgp template default sample size 1 sample interval 15 I performance mgmt statistics mpls ldp template default sample size 1 sample interval 15 I performance mgmt statistics node cpu template default sample size 1 sample interval 15 I After dumping the performance data the XR 12000 sends a notification indicating the location of the file This notification is logged in the Alarm Log every 15 minutes and triggers the CTM server connecting to the TFTP server Check for notifications in the Alarm Log that look similar to the following pm_collector 301 PM_COLLECTOR 6 TFTP_SEND Successfully dumped file tftp lt IP_address gt lt TFTP_directory gt lt NE_name gt _node_process_1100886887 212000 J 9 6 XR 12000 BGP and CDP Neighbor Discovery Issues CTM discovers all the CDP and BGP neighbor routers and adds them to the tree view if the CDP or BGP is enabled on the router CDP and BGP neighbor discovery occurs after the NE is synchronized If the neighbors are not discovered enter the following CLI commands to check whether CDP or BGP is enabled on the router and interfaces show bgp neighbors show cd
8. Retrieve the IMA group ATM cell layer ingress egress counters Retrieve the IMA group level real time statistics IMA Link Diagnostics Retrieve the IMA link level attributes such as IMA link status Start stop IMA link test Modify IMA link test pattern Retrieve the IMA link level real time statistics Connection Diagnostics Retrieve the connection level attributes such as connection status Retrieve the local and remote endpoint attributes such as A bit AIS and OAM Retrieve the local and remote endpoint real time statistics The Diagnostics Server interfaces with the CM Server to support the following connection related diagnostics operations Connection loopback Up connection Down connection Connection trace Test connection Test delay Test connection segment Test ping OAM J 10 9 1 Diagnostics Center Framework This section includes the following information J 10 9 1 1 Cannot Launch Diagnostics Center page J 109 J 10 9 1 2 Cannot Launch Other Applications from the Diagnostics Center page J 109 Cisco Transport Manager Release 6 0 User Guide 3108 78 16845 01 AppendixJ Troubleshooting J 10 9 Diagnostics Center Problems W e J 10 9 1 3 An Exception Is Raised when the Diagnostics Center Is Launched page J 110 e J 10 9 1 4 Double Click Operation In Diagnostics Center Tree View Does Not Launch an Internal Frame page J 110 e J 10 9 1 5 Drag and Drop Functionality Errors Within the Diagn
9. Screen snapshots in particular error information messages dialog boxes Cmsvr log file from the opt svplus log directory Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 2 Cannot Launch Other Applications from the Configuration Center The Configuration Center does not launch other applications using one of the following methods 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 7 Configuration Center Management e Choose an application from the Tools menu item e Right click on the selected object from the hierarchical tree and choose the target application Step 1 Check the target application jar file Make sure the target application jar file is located in the opt svplus java jars cwm directory of the target machine Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 3 An Exception Is Raised when the Configuration Center Is Launched When the Configuration Center is launched using one of the following methods an exception is rai
10. cat6509 enable copy tftp flash IP address or name of remotehost lt TFTP_server_IP_address gt Name of file to copy from lt Full_path_name_and_filename_of_the_image_location gt For example tf tpboot cat6k 7 2 lt image_name gt Step6 Enter the following command to verify that the downloaded image exists on the device cat6509 enable dir bootflash Step7 Enter the following command to verify that the download was successful cat6509 enable verify lt image_name gt Step8 Enter the following command to clear the flash system cat6509 enable clear boot system flash bootflash lt previous_image_name gt Step9 Enter the following command to set the boot system flash with the new image name cat6509 enable set boot system flash bootflash lt new_image_name gt prepend Step 10 Enter the following command to reset the device cat6509 enable reset w__ Cisco Transport Manager Release 6 0 User Guide 78 16845 01 AppendixJ Troubleshooting J9 XR12000Problems W J 9 XR 12000 Problems This section describes troubleshooting procedures for problems related to the XR 12000 J 9 1 Reporting XR 12000 Issues When reporting an XR 12000 problem include detailed steps to reproduce the problem and the software version s of the NE s to which the problem applies Also enter the following router CLI commands on the XR 12000 and collect the data show version show platfo
11. xml agent corba ssl Router config commit amp Note SSL is optional and enables SSL CORBA services If SSL CORBA services are running enter the following commands to verify that HTTP over SSL is running Router config http server ssl Router config commit The CORBA agent on the router accepts connectivity only when the connection is done through hostname Verify that the device IP address has an entry in DNS If not add an entry to the etc hosts file Verify that CTM supports the software image that is running on the router In the Domain Explorer choose Administration gt Supported NE Table and verify that the software version is supported When the CRS 1 is added to CTM the NE type is retrieved from the NE to verify that the NE is a CRS 1 If the device does not report CRS 1 as the NE type CTM reports a loss of communication LOC to the NE and raises an NE type mismatch alarm Verify that the NE hostname specified in DNS or in the CTM server s etc hosts file matches the hostname configured on the NE If the hostnames do not match CTM reports an LOC to the NE J 8 2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service Step 1 Step 2 Verify that the IP address entered in CTM is configured on the Catalyst 6509 Enter the following command to configure the IP address cat6509 enable set interface lt logical_interface_name_like_sc0 sc1 gt lt IP_address_of
12. How widespread is the problem Is it getting worse Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EN Appendix J Troubleshooting ME J2 Server Problems S Note This chapter assumes that the server is installed under the default opt CiscoTransportManagerServer directory and the client is installed under the default opt CiscoTransportManagerClient Solaris or C Cisco TransportManagerClient directory Windows If a directory other than the default installation directory is specified replace the default path with the installed path J 2 Server Problems amp This section describes troubleshooting procedures for CTM server related problems Note Log in as the root user on the Solaris workstation where the CTM server is installed to perform any operations on the Solaris workstation J 2 1 Conditions that Affect CTM Server Performance The System Requirements chapter in the Cisco Transport Manager Installation Guide lists server CPU CPU speed disk space and RAM requirements for different types of CTM installations These requirements are derived from guidelines based on scalability simulation testing Actual CTM server performance varies for each customer and is affected by e The total number of NEs actively managed by CTM e The number of NE related services including northbound services running on the server e The rate of circuit provisioning and the methods used for provisionin
13. If the CTC progress screen idles for a long time before the login dialog box appears there might be problems with network connectivity to the device a From a DOS command window or a Sun Solaris terminal ping the device b If no response is received set up routes so that the device is available then restart CTC If CTC does not start the workstation might have resource constraints c Close some open applications and try again d Close some instances of CTC that are managing another group of CTC based NEs and try again If the GNE for the selected NE was not found there might be a mismatch between the GNE specified for the device and the available GNEs This means that the data in the database is corrupt Contact Cisco technical support for assistance J 4 4 Added a New Software Version to the Wrong NE Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Step 8 In the Domain Explorer window choose Administration gt Supported NE Table In the Supported NE Table select the incorrect entry and choose Edit gt Delete Click OK in the confirmation dialog box Select the correct NE row from the Supported NE Table Add the correct software version In the Domain Explorer gt Network Element Properties pane set the operational state of all NEs that are behaving erroneously to Out of Service Click Save In the Network Element Properties pane set the operational state of all the NEs back to In Service Cisco Trans
14. Node name s not found in table node NO_SUCH_CARD_IN_DB Card Node d Slot d not found in table card NO_SUCH_LOG_PORT_IN_DB Card Node d Slot d LogPort d d in database not found in table s NO_SUCH_PHY_PORT_IN_DB Card Node d Slot d Line d PhyPort d d in database not found in table s NO_SUCH_LINE_IN_DB Card Node d Slot d Line d not found in table s NO_SUCH_CONNECTION_IN_DB Connection s not found in database UP_CONN_FAILED Upping Connection Failed Test Result from switch s DOWN_CONN_FATLI ED NO_ONE_SEGMENT_HYBRID INCORRECT_LINE_SUBTYPE ad not d INVALID_LINE_NUMBER func NO_CONTROLLER_IN_PORT node s INVALID_FDN INVALID_PORT XLMI trunk ports s Downing Connection Failed Test Result from switch Cannot create one segment hybrid connection Incorrect Line SubType The given FDN format or value is invalid s Line Node d slot d ifindex d is s Invalid Line Number s slot d port d does not have controller of type s 3s Provisioning operations are not allowed on feeder trunk routing trunk and Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting Step 1 Step 2 J 10 8 Connection Management Problems W Verify that the CM Server and sdbroker are up and running by entering psg sdb
15. a 78 16845 01 Appendix J Troubleshooting Step 6 Step7 J 10 4 Equipment Management Problems W It should not take long to declare node resync failure because of the SNMP request failure To verify that there is an SNMP failure check the ooemc and snmpcomm log files Using ooemc10 6568 log as an example of an ooemc log filename 10 is the child ID the 6568 is the ooemc process ID The child ID is calculated from the following formula Remainder NEDBACCSSID number of ooemc child 1 To retrieve the process ID use the CLI command psg em Grep RESYNC from the log file The grep result should include the node ID and node mode Review the log messages in the log file to determine if an SNMP failure has occurred Do the following Check the log file for trap 60903 If you find it contact the platform team to investigate the bulk file creation failure on the switch Very often the node mode 2 problem is caused by a bulk file creation abort process on the switch If there are no SNMP request failures the switch should send traps 60901 and 60902 to ooemc Trap 60901 indicates that the switch will start bulk file creation while trap 60902 indicates bulk file creation done When the ooemc receives a trap 60902 it will start to FTP the bulk file which is a shelf generic configuration file This is also the first file to be uploaded in every resync mode coldstart warmstart or periodic resync The commonly seen m
16. and hybrid J 10 8 4 2 Cmgrd Sdbroker Errors During an addition request cmgrd could return errors that it has received from sdbroker The following are some of the most common errors seen when a provisioning request is issued J 10 8 4 3 Cmgrd Sdbroker Addition Modification Deletion Errors Adding a Connection Results in Fail to Communicate with sDatabroker Error Related key index entries cmgrd sdbroker When any Connection Add Mod Del request is made cmgrd checks its ILOG connection with the sdbroker This error will result when cmgrd could not verify that sdbroker is available for servicing requests In other words the ILOG connection between them is broken The problem must be researched on the sdbroker side Step 1 Verify that the sdbroker is up and running Execute psg sdbroker on the command line Step2 View the sdbroker log for ILOG errors There could be multiple sdbrokers running on the machine To check which sdbroker cmgrd talked to for that particular request do the following a vithe cmgrd log file and go to the end of the file Then do a backward search for the string sdbroker That string should indicate the number of the sdbroker that cmgrd could not talk to b Execute psg sdbroker and then check the corresponding log file of this sdbroker Defect Information Collect the following e Stack trace of the sdbroker process pstack lt pid of sdbroker gt e Sdbroker and cmgrd logs Possible alternative wo
17. 4 NE Connection State Is Listed as Unavailable Step 1 Step 2 Step 3 Step 4 Step 5 If the connection state of an NE is listed as Unavailable in the Domain Explorer window a connectivity or configuration problem exists Wait 5 to 10 minutes after adding the NE to the CTM domain then complete the following steps To see the NE IP address select the NE in the Domain Explorer window The Address tab of the Network Element Properties pane lists the IP address of the selected NE From the CTM server enter the following command to verify connectivity between the CTM server and the NE ping lt IP_address gt If the ping fails a physical or configuration problem exists in the data communications network DCN a If connectivity existed earlier e If there were no configuration changes made to any routers in the DCN or to the CTM server the problem is a physical problem e Ifthe configuration was changed the change might have introduced problems Verify the changed configuration b If connectivity was never established e Verify that there are no problems on the physical level e Verify the DCN configuration If the ping succeeds verify that the NE software version is listed in the Supported NE Table See 4 3 7 Adding a New NE Software Version to the CTM Domain page 4 37 For ONS 15501 ONS 15530 and ONS 15540 NEs SNMP read and write community strings must be set up properly on each NE The running configuration on
18. Addition Errors Provisioning a Connection Results in Vpi vci ranges retrieval from svc_operation failed Error page J 102 e J 10 8 4 15 Cmgrd Addition Modification Deletion Errors Provisioning a Connection Results in Can t get segment info from Data base Error page J 103 e J 10 8 4 16 Cmgrd Switch Errors page J 103 e J 10 8 4 17 Cmgrd Addition and Modification Errors Provisioning a Connection Results in Agent reported bad wrong value for one of the variables in the request Switch Error page J 103 e J 10 8 4 18 Cmgrd Addition Errors Provisioning a Connection Results in Object Exists on the Agent or Specified VPI VCI not available Switch Error page J 104 e J 10 8 4 19 Cmgrd Addition Modification Errors Provisioning a Connection Results in Requested cell rate Iscr Ipcr or rpcr rscr is too high Switch Error page J 104 e J 10 8 4 20 Cmgrd Addition Errors Provisioning a Connection Results in SPVC is not allowed on this interface Switch Error page J 105 e J 10 8 4 21 Cmgrd Addition Errors Provisioning a Connection Results in Local Channels not enough Switch Error page J 105 e J 10 8 4 22 Cmgrd Addition and Modification Errors Provisioning a Connection Results in Error in Traffic Parameters Switch Error page J 105 e J 10 8 4 23 Cmgrd Addition Modification and Deletion Errors Provisioning a Connection Results in Network Busy try later Switch Error page J 106 e J 10 8 4 24 Cmgrd M
19. Chassis View does not get event updates on hot insertion or removal of RPM cards The card will be identified when coldstart is done J 10 6 8 RPM Secondary Card Status Is Blue For RPM cards the standby state shows the card status in blue because the card has only one LED CPUOK to show the status of the card unlike the other cards For other types of cards the standby state is indicated by a yellow LED J 10 6 9 Lines Not Displayed on Secondary Card If two cards are in a redundancy relationship the primary card for example the logical slot is used to display the children and for all provisioning and troubleshooting activities even if the primary slot becomes a standby The secondary slot does not show any children under it even if it becomes active Hierarchical views in all applications behave in this manner Similarly provisioning is allowed only on the working line of an APS pair regardless of whether or not that line is currently active However monitoring is done on both working and protection lines Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EN AppendixJ Troubleshooting W sJ 10 7 Configuration Center Management J 10 6 10 Lines Not Selectable Sometimes lines become unselectable For example when trying to select line 3 line 1 may get selected or vice versa Lines must be spaced sufficiently apart If they are not they may overlap causing them to become unselectable Defect Informati
20. Complete the following substeps a Enter the following command on the switch CLI of the node that cannot be discovered dspsnmp Cisco Transport Manager Release 6 0 User Guide P34 78 16845 01 Appendix J Troubleshooting J 10 2 Discovery Issues at Startup W b Compare this community string with the community string in the node_info table The community string in the node_info table is encrypted You will need to decrypt this string and verify it Step3 For MGX nodes if persistent topology is enabled on the switch gateway and some nodes in a peer group are not discovered complete the following substeps a Enter the following CLI command to verify that persistent topology is enabled on the gateway in the peer group dsptopogw b If the gateway flag is enabled enter the following CLI command to verify that the node is part of the list of nodes on the gateway dsptopondlist Defect information Collect the following information for further analysis e Topod log and ILMITopoc log e Data in the node_info table e Output of dspndparms dspsnmp and dspipif on the MGX node J 10 2 2 Node Name Is Incorrect in the Database If the node name of the standalone MGX node is incorrect complete the following steps Step 1 If the node table contains the correct name complete the following substeps a Enter the following command to dump the cache of topod and ILMITopoc kill USR1 lt process gt b From the data colle
21. Get the process ID of the ooemc process that manages the node Step3 Save the config upload files and ooemce log files Defect information Collect the ooemc and config upload files from the opt svplus log directory Possible alternative workaround Manually resync the node If the problem persists perform a coldstart If the problem is still not resolved collect the log files and report the problem J 10 4 7 CTM CM GUI Shows Incomplete Connections After Successful Node Resync and Dbroker Sync Up CTM shows that some of connections are incomplete after the node resync process and dbroker synchronization process Step 1 Determine which ooemc process manages the node The ooemc will manage the connection segment in the CTM database which terminates on the MGX Find the child ID of the ooemc process that manages your node The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection The debugging process for this issue mainly focuses on log file inspection Step2 After you have identified the ooemc log files grep NotifyDataBroker from the files to see some log message examples before you modify your grep format This will help you more efficiently grep the pair of log messages that correspond to the local and remote ends of one endpoint primary endpoint or secondary endpoint of a connection segment In other words each end either primary end or secondary end of a connection segment manag
22. Gets the detailed information about an object in the tree Subscribe for all events Subscribes to all updates on the server Unsubscribe for all event Unsubscribes from all updates on the server The GUI client s log files CMSCclient log are saved under log directories for example in Windows D Documents and Settings lt username gt log or in Unix opt svplus log J 10 5 2 Basic Issues This section includes the following information e J 10 5 2 1 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show Any Nodes page J 55 e J 10 5 2 2 Configuration Center Chassis View Diagnostics Center or Statistics Report Is Unable to Connect to Server page J 56 e J 10 5 2 3 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Newly Added Node page J 56 J 10 5 2 1 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show Any Nodes Step 1 Check whether the nodes are discovered See J 10 2 1 No Nodes Are Discovered page J 34 Step2 Check whether NMServer has the nodes and trunks in its cache This can be done by using the command nmControl in the CTM CLI Step3 If the nodes are in the NMServer cache open a new GUI and verify whether the nodes are shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log NMServer log e nmControl dump e CMSCclient log Cis
23. This table generally corresponds to the Line or Port table of the card for which this error occurred d Ifthe Entity Name part of the XML filename is incorrect contact CTM engineers with the defect information required Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e opt svplus log configserver log Cisco Transport Manager Release 6 0 User Guide P73 E 78 16845 01 Appendix J Troubleshooting J 10 7 Configuration Center Management W e Dspcd command output from the switch for the controller card and the service module where this error is detected e Ddspln or dspport command output if this error seen for a line or port Possible alternative workaround None J 10 7 2 2 SNMP No Data Error Step 1 Step 2 An SNMP NO DATA error is seen while launching the Configuration Center either by the drag and drop method or the right click method for a network element See if the network element for which this error occurred is active and that the node is synced up To do this use the selnd command on the CTM machine Check whether the correct community strings are used If this error is noticed for a properly synced up node on an active element perform the following Use an SNMP Manager tool to determine whether the switch is responding to SNMP queries The SNMP MIB ob
24. and retrieving a list of existing connections Connection tab The Configuration Center uses the CTM framework and workflow mechanism to launch applications and to create and display connection information e Connections can be selected in CTM applications and the Configuration Center Connection Tab can be launched to view modify the selected connection e Connections can be dragged and dropped from other application to the Configuration Center s Connection tab for modification The following sections describe guidelines for troubleshooting issues related to creating modifying and viewing network connection using the Configuration Center J 10 8 1 1 Connections Tab Double Click Operation on Tree View Does Not Launch Connection s Internal Frame Step 1 The double click operation on a network element in the tree view on the Connection tab a fails to open an internal frame b fails to recycle the contents of an existing internal frame to display the double clicked connections attributes or c results in the Operation Not Supported message dialog box Determine if the double click operation is supported for the selected object The Connection tab supports connection management for the Node Card Line and Port objects The drag and drop functionality of Folder IMA and IMA link objects is not supported Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and
25. bin CTMServer root 3876 0 5 0 6129464 8648 R 16 58 12 SnmpTrapService root 3798 26 8 4 115732060968 S 16 57 37 0 29 SMService Manually stop the server if you see fewer than four processes running Enter the following command ctms stop If you changed the server IP address verify that the configuration files shown in Table 4 13 on page 4 44 have been updated See 4 4 2 Updating the Configuration Files After Changing the CTM Server IP Address page 4 44 Verify that the Oracle database is accepting connections a Enter the following command to log in as the Oracle user su oracle b Enter the following command to open an SQL Plus session omu u60 3 sqlplus ctmanager ctm123 Reboot the server if you receive another error message and the SQL prompt does not appear Wait enough time for the server to boot up and try to run the client The SQL prompt indicates that the Oracle database is running and accepting connections Enter the following commands to manually restart CTM SQL gt exit omu u60 3 3 exit omu u60 3 logout ctms start If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot enter the following command Cisco Transport Manager Release 6 0 User Guide Ps g 78 16845 01 Appendix J Troubleshooting Step 11 J 2 4 NE Connection State Is Listed as Unavailable W sudo ctms start Wait for 5 minutes and run the client J 2
26. data to the NM server grep ComposeNMSMssg from the ooemce log files The key words in log messages that correspond to the supported database tables in step 2 are the following EMC_DBProperty_Node_c ComposeNMSMsg 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting E J 10 4 Equipment Management Problems Step 4 EMC_DBProperty_Card_c ComposeNMSMsg EMC_DBProperty_Line_c ComposeNMSMsg e EMC_DBProperty_AusmPort_c ComposeNMSMsg EMC_DBProperty_CesmPort_c ComposeNMSMsg e EMC_DBProperty_FrPort_c ComposeNMSMsg e EMC_DBProperty_RpmPort_c ComposeNMSMsg e EMC_DBProperty_SvcPort_c ComposeNMSMsg e EMC_DBProperty_VtPort_c ComposeNMSMsg e EMC_DBProperty_Aps_c ComposeNMSMsg e EMC_DBProperty_ImaGroup_c ComposeNMSMsg e EMC_DBProperty_ImaLink_c ComposeNMSMsg e EMC_DBProperty_LineDist_c ComposeNMSMsg e EMC_DBProperty_AU4TUG3_c ComposeNMSMsg e EMC_DBProperty_Ctrlr_c ComposeNMSMsg e EMC_DBProperty_LicenseInUse_c ComposeNMSMsg e EMC_DBProperty_MfrBundle_c ComposeNMSMsg e EMC_DBProperty_MfrLink_c ComposeNMSMsg e EMC_DBProperty_Peripheral_c ComposeNMSMsg e EMC_DBProperty_RedCard_c ComposeNMSMsg e EMC_DBProperty_Sensor_c ComposeNMSMsg Note Database data will not be sent to NMServer during the initial node resync It is after the initial resync that ooemc starts sending updated or newly provisioned database data to NMServer Each ooemc log message from grep contains d
27. e VsiCtrlr Trap e DS3Line Trap e AtmPhy Trap e Peripheral Trap e CoreSwth Trap e TrapLost Trap e AtmAddr Trap e Restart Trap e Node Trap e SonetAps Trap e LMIPort Trap e Pnni IF Trap e Bulkfile Trap e Vism Trap e Vism Ann Trap e Sublf Trap e NBSMCnfg Trap e NBSMChan Trap e NBSMLine Trap e AusmLine Trap e AusmPort Trap e AusmChan Trap e AusmIma Trap e FrChan Trap e FrPort Trap e CesmPort Trap e CesmChan Trap e HsFrPort Trap e HsFrChan Trap e LineDist Trap e DS3Path Trap e Chan Upload e Party Trap e PrefRoute Trap e CardIma Trap e DSI Line Trap Cisco Transport Manager Release 6 0 User Guide EEN 78 16845 01 Appendix J Troubleshooting SctPort Trap SvcDerouteGroomTrap Channel Trap TUG3Path Trap Cug Trap AddrCug Trap RSC Upload APS Upload FrPort State Upload MPSM Upload Vism ToneDetect Trap License Trap PortAtmIf Trap VxsmPvcRed Trap ChanProt Trap VxsmGwDsp Trap VxsmGwIp Trap VxsmGw Trap VxsmSysRes Trap VxsmGw1 Trap VxsmMgc Trap VxsmMgclIp trap VxsmMegcGrpParam Trap VxsmMgcGrpMgc Trap VxsmMgcGrpProt Trap VxsmAal2Prof Trap VxsmCodec Trap VxsmSve Trap VxsmAal2CrossConn Trap VxsmAal25DataProfileTrap VxsmSensor Trap VxsmSensorThrhd Trap VxsmModule Trap DSOGrp Trap VxsmAnnounce Trap VxsmAudioFile Trap VxsmDs0OXConn Trap VxsmMegaco Trap J 10 4 Equipment Management Problems W 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Tr
28. f lt ooemc_log_file gt Collect the ooemc nts snmpcomm and cwmftpd log files from the opt svplus log directory Possible alternative workaround None J 10 4 5 Node Resync Mode Is 5 Node mode 5 indicates that node resync has failed This could be caused by the config upload or parsing failure of the shelf generic file CARD_01_CC CF 13 for example or pnni file PNNI_01_CC CF 13 for example In general mode 5 signifies that a problem has happened in one or more stages of the entire resync process The resync process is composed of the following stages and each stage alone can lead to mode 5 problems 1 The ooemc triggers node resync 2 The ooemc sends an SNMP request to the switch for bulk file creation Cisco Transport Manager Release 6 0 User Guide Ps 78 16845 01 Appendix J Troubleshooting Step 1 Step 2 Step 3 Step 4 J 10 4 Equipment Management Problems W 3 The ooemc receives bulk file creation related traps 60901 60902 and 60903 4 The ooemc FTPs the config upload files from the switch after it has received 60901 and 60902 from the switch 5 The ooemc parses the config upload files 6 The ooemc declares sync up done Determine if the problem is caused by the SNMP request a Grep RESYNC from the ooemce log files The ooemc will change node mode to 2 when it starts the node resync process You should see something like the following from the log NOTICE N17 lt EMC_Node_c InSy
29. file is missing it has been deleted Reinstall the CTM client amp Note The client is bundled primarily with shape files shp and only a few map background GIF files J 4 12 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation Step 1 Step 2 Step 3 To run the client on a Sun Ultra 5 workstation the color map must be changed from 8 bit to 24 bit If the colors do not look right change the color map from 8 bit the default to 24 bit as follows To see the current color settings enter the following command usr sbin m64config prconf Command output looks similar to the following example Hardware Configuration for dev fbs m640 ASIC version 0x7c004750 DAC version 0x0 PROM version 104 Card possible resolutions 720x400x88 640x480x60 640x480x72 640x480x75 800x600x56 800x600x60 800x600x72 800x600x75 1024x768x87 1024x768x60 1024x768x70 1024x768x75 1280x1024x75 1280x1024x60 1152x900x66 1152x900x76 1280x1024x67 1280x800x76 1280x1024x85 1280x1024x76 1152x864x75 1024x768x77 1024x800x84 vga svga 1152 1280 800x600 1024x768 1280x1024 1152x900 Monitor possible resolutions 720x400x70 720x400x88 640x480x60 640x480x67 640x480x72 640x480x75 800x600x56 800x600x60 800x600x72 800x600x75 832x624x75 1024x768x60 1024x768x70 1024x768x75 1280x1024x75 1152x870x75 1152x900x66 1152x900x76 1280x1024x67 1280x1024x76 vga svga 1152 1280 800x6
30. file under the opt svplus log directory Possible alternative workaround None J 10 8 1 8 Cross Application Connection Tab s Content Pane as Drop Target The drag and drop functionality of a connection object from another application to the Configuration Center s connection tab content pane a fails to open an internal frame b fails to recycle the contents of an existing frame to display the newly dropped connection object or c results in the Operation Not Supported message dialog box Step 1 Determine if the source and the target application belong to the same instance of the Java virtual machine VM CTM framework does not support drag and drop functionality across applications belonging to different Java VMs Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information in particular any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 9 Configuration Center s Connection Tab Does Not Respond The Configuration Center s Connection tab does not respond and the GUI is disabled Step 1 Try to reproduce the problem in order to collect additional Java related data using the Java DOS win
31. found then proceed as follows a Use cwmver to get the CTM version b Get the node table information for the node using the following command then save it in a file echo select from the node where node_id lt NodeId gt dbaccess c Get the card table information for the card using the following command then save it in a file echo select from the card where node_id lt NodeId gt and slot lt Slot gt dbaccess d Save the line table information for the line using the following command echo select from the line where node_id lt NodeId gt and slot lt Slot gt dbaccess e Save the log as CMSCclient log to your local drive f Take a copy of the chassisview jar from the opt svplus java jars cwm directory You need to check that gif files used to draw the lines are available in the jar Possible alternative workaround Select the lines from the tree view to launch other applications J 10 6 3 Card Not Displayed in the Chassis View Step 1 Check that entries for the card are available in the card table Step2 Check whether the lines are defined in the XML file Chassis View xml Step 3 Make sure that the node is in sync mode is 3 If the details are not found in the database this probably is an EM issue If the details are found then proceed as follows a Use cwmver to get the CTM version b Get the node table information for the node using the following command then save it in a file
32. hosts file Verify that CTM supports the software version that is running on the router The CTM release notes list the CTM supported NE software versions Go to http www cisco com univercd cc td doc product rtrmgmt optnet ctm ctmreln index htm When an NE is added to CTM the NE type is retrieved to verify whether the NE is an XR 12000 If the device is not of NE type XR 12000 the NE is moved to an LOC state in CTM and an NE type mismatch alarm is raised 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HI J 9 3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer Step 6 The NE hostname specified in DNS or in the CTM server etc hosts file must match the hostname configured on the NE If the hostnames do not match the NE is moved to an LOC state in CTM J 9 3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer If the LAN cable is removed there is a delay before the status of the XR 12000 is updated in the Domain Explorer It might take several minutes or more to reflect the status as unreachable if the router is disconnected If you want the status to be updated sooner you must modify the jacorb connection client pending_reply_timeout parameter in the opt CiscoTransportManagerServer openfusion classes jacorb properties file The default timeout is 5 minutes 300000 ms Updates take approximately the health poll interval plus two times the jacor
33. information e J 10 13 1 NTS page J 127 Cisco Transport Manager Release 6 0 User Guide P3126 78 16845 01 Appendix J Troubleshooting J 10 13 1 NTS J 10 13 Miscellaneous Problems W e J 10 13 2 Data Inconsistency page J 129 e J 10 13 3 CTM FTP Daemon page J 134 This section includes the following information e J 10 13 1 1 Nodes Stay in Mode 1 After ColdStart page J 127 e J 10 13 1 2 Config Change or Provisioning Activity Not Reflected on CTM page J 127 e J 10 13 1 3 How To Interpret NTS Log page J 128 J 10 13 1 1 Nodes Stay in Mode 1 After ColdStart Step 1 Step 2 Step 3 Step 4 The nts cannot manage a particular switch successfully In other words nts declared the switch to be in the Unreachable or Unregistered state As a result nts notified its clients with a LINK_DOWN message The nts declares a node to be in OK state if it is in the Trap Manager List of the node and can successfully perform SNMP operations on the node Check the PING reachability Run ntsControl and enter ni at the prompt Then enter q to exit ntsControl This shows the nts node information A node has to be in OK state so that CTM can start syncing it up If the node is not in OK state first thing to check is PING Run ping lt ip_address gt If the PING succeeded go to step 2 Otherwise fix the network reachability between the switch and the CTM station Check the CTM IP address configuration This only applies to CTM workstati
34. kill USR1 signal lt process gt b Check the data collected in the usr user svplus log directory and verify whether the node name is correct in the cache dumps If the node name is correct open a new instance of the GUI and check whether the node name is correct in that GUI Step2 If the node table does not have an entry for the node with the correct name complete the following substeps a Check the ILMItopoc log and verify whether it received a 60006 and 70202 trap b If you see the trap in the log verify that ILMITopoc updates its cache based on the trap If it fails to update the node name ILMITopoc log will contain the message ILMITopoc 3 updateFailed lt Node_c updateName gt Failed to update node name c If you do not see the traps in the ILMITopoc log see J 10 13 1 NTS page J 127 Defect information Collect the following information for further analysis e ILMITopoc log topod log NMServer log and nts log e Dump outputs of ILMITopoc topod and NMServer Enter the following command kill USR1 signal lt process gt e Collect the output of the switch CLI selnd and dbnds J 10 3 3 Node IP Address Change Is Not Updated in the Configuration Center Chassis View Diagnostics Center or Statistics Report If the node IP address is not updated dynamically in the Configuration Center Chassis View Diagnostics Center or Statistics Report complete the following steps Step 1 If the node table has an entr
35. nonzero values in the database Choosing Optimized requires less space than saving all values including zero values If you choose Normal in the PM Data Storage field the database saves all PM values including zero values amp Note The Optimized PM data storage feature is available only for CTC based and ONS 155xx NEs Choose Administration gt Error Log to see if there are any critical major or minor errors against the PM service 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting E J210 CTC Based NE Is Not Discovered Step6 Verify that the NE is reachable during the duration that PM data is collected If the NE is not reachable there is an entry in the Audit Log table Administration gt Audit Log J 2 10 CTC Based NE Is Not Discovered Step 1 Verify that the NE is up and running Step2 Verify that the NE IP address and default route are configured correctly Step3 To verify that the NE is running the correct version of system software open a CTC session to the NE and check the NE software version Step4 Verify that the NE that acts as the gateway NE GNE through which this node is discovered is added to CTM and is available Step5 Open CTC from the GNE and see if CTC can discover the NE J 2 11 CTC Based NE Is Not Reachable Step 1 Verify that the NE is up and running Step2 Verify that the NE IP address and default route are correctly configured Step3 W
36. not synchronized Step 1 Step 2 Step 3 This can happen when the time stamp between the CTM and the collector is not synchronized or the time stamp between the collector and the node is not synchronized Check the sync_info table by executing the following command echo select offset from sync_info where sync_node_id lt node_id gt dbaccess Check the time on the CTM Collector and on the node and make sure the offset from the previous query is the difference of time in seconds For feeders check the time difference between CTM and the routing node of the feeder Defect Information Collect the following information for further analysis e Pmcollector log e Related key index entries e Pmcollector Possible alternative workaround Set the time on the CTM Collector and the node to the same time stamp J 10 10 1 5 PM Collection Fails for the Node Log File Shows Snmp failed Step 1 Step 2 This can happen due to SNMP failure caused by an incorrect community string or a timeout Check get_str and set_str in the node_info table and make sure they are the community strings on the node Check the SNMP community strings on the node using the command dspsnmp To check the log files for more details on the error grep snmp scm If the error is a timeout error change the timeout setting and retry the setting in opt svplus config pmcollector conf Defect Information Collect the following information for further analysi
37. on has no more resources available to accept the traffic parameters that the current SNMP SET is requesting At this point the switch will reject the request and return this error For the user endpoint check the user port on the CLI using the command dsppnportrsrc This will indicate the resources available per port If the failure is not happening on a user endpoint port but instead on an intermediate port for example for multisegment connections find out the local and remote endpoint feeder trunk ports This can be done by using the Network Topology GUI Cisco Transport Manager Release 6 0 User Guide ea 78 16845 01 Appendix J Troubleshooting Step 3 J 10 8 Connection Management Problems W On the CLI execute the dsppnportrsre command to check for the feeder trunk ports Defect information If you continue to receive this error after you have verified that the ports have enough bandwidth to add these connections save the cmgrd log Possible alternative workaround Freeing up resources on the port which needs the bandwidth to add the connection or lowering the traffic parameters value in the connection request J 10 8 4 20 Cmgrd Addition Errors Provisioning a Connection Results in SPVC is not allowed on this interface Switch Error Step 1 Step 2 Related key index entries cmgrd interface During an add connection request it is possible that the user endpoint port that the connection is being set o
38. opt svplus log directory Possible alternative workaround Force the system to declare sync up by sending the DMD a forced sync signal This will not place the nodes into mode 3 any faster but will allow you to attempt provisioning some nodes while other nodes are in the process of syncing up Execute command selnd from the CWM command line Note the nodes in mode 3 Connections provisioned after forced sync up can only go through nodes with mode 3 Send the sync up signal to the DMD by entering command kill ALRM dmd View the DBKR_SYNCUP_MESSAGE file for the sync up completion J 10 8 4 6 Cmgrd Sdbroker Addition Errors Adding a Connection Results in No more vpi vci available for local trunk end Error Related key index entries cmgrd sdbroker vpi vci Cisco Transport Manager Release 6 0 User Guide yos W 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W Provisioning fails and the following error messages are displayed e No more vpi vci available for local trunk end and Remote trunk e Local end of the connection already exists Remote end and Both end e Vpcon already exists for Local Vpi Remote Vpi and Both ends Each of these error messages identifies the reason for the failure of provisioning They sometimes can be displayed in error The reason is a mismatch between the switch and the DMD s cache To find out if a mismatch has occurred dump the DMD cache using
39. opt svplus logs directory Copy all the cmgrd logs from opt svplus logs directory Capture the user_connection table query output from the CTM database for the given endpoint if the request is not an addition request Switch CLI output for the trace diagnostics on the same connection Possible alternative workaround a b wy Check the status of the connection and clear any alarms due to line port issues adding loopback correcting the physical cable issues by verifying the physical port connections and so on After the connection comes up and fine in the Clear state rerun the trace diagnostics Verify the connection status If the connection status is Down trace cannot be executed on the connection If the connection status is failed the trace result will probability be an empty string Note The connection trace cannot be executed on an incomplete or DAX PNNI segment connections This section includes the following information J 10 8 4 1 High Level Connection Management Subsystem Architecture page J 96 J 10 8 4 2 Cmgrd Sdbroker Errors page J 97 J 10 8 4 3 Cmgrd Sdbroker Addition Modification Deletion Errors Adding a Connection Results in Fail to Communicate with sDatabroker Error page J 97 J 10 8 4 4 Cmgrd Sdbroker Addition Modification and Deletion Errors Provisioning a Connection Results in sDatabroker process busy Please retry Error page J 97 J 10 8 4 5 Cmgrd Sdbroker Addition Errors A
40. session limit reached key ftperr_499 Cisco Transport Manager Release 6 0 User Guide Pye i 78 16845 01 Appendix J Troubleshooting J 10 13 Miscellaneous Problems W J 10 13 3 9 Failed to Acquire a Session with a Switch For MGX 8850 Release 4 0 or later CTM waits in a predefined port to acquire the session if it receives the 499 Session limit reached error This wait time is 1 minute by default and is configurable If CTM does not get a response from the switch within this wait time it times out and throws the following exception and then again retries to open the session 9583 6 08 37 29 ERR CwmFtpDaemonException 3 SESSION_TIMEOUT Session Id 28 ClientSeq 6521 Host 172 25 69 218 User superuser Priority 3 get stat 1 04 Con 052920030730 opt svplus spool MGX98101 1 04 Con 052920030730 2 timeout in 3 minutes and needs to be retried It retries 5 times by default this number is configurable If all retries fails cwmftpd throws the following exception 9583 172473 08 37 58 ERR CwmFtpDaemon 3 SESSION_FAIL Thread Host 172 2 5 69 218 User superuser NoRequest failed to acquire session after 5 retries Session wait time 5 Related key index entries switch session FTP J 10 13 3 10 Failed to Acquire Session After All Retries Step 1 The cwmftpd command failed to acquire a session with a switch on all retries Check whether all four sessions have been opened by opening a manual FTP s
41. support this functionality Retrieve card level real time statistics if any Retrieve information about all lines and paths that are in loopback on the card Retrieve information about currently running BERT tests on the card Retrieve information about currently running IMA link tests e Port Diagnostics Retrieve the port level attributes such as port status Perform loopback Perform start stop modify BERT Retrieve the results and or current status of the BERT Monitor the scheduled grooming results Configure the on demand grooming Retrieve the port level real time statistics e Line Diagnostics Retrieve the line level attributes such as line status and loopback status Perform loopback Perform start stop modify BERT Retrieve the results and or current status of the BERT 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 9 Diagnostics Center Problems Retrieve the line level real time statistics Path Diagnostics Retrieve the path level attributes such as path status Perform loopback Retrieve the path level real time statistics Trunk Diagnostics Retrieve the trunk local and remote end real time statistics IMA Group Diagnostics Retrieve the IMA group level attributes such as IMA group status Retrieve the accumulated delay on all the IMA links under the IMA group Clear the accumulated delay on all the IMA links under the IMA group Restart the IMA group
42. svplus log directory Possible alternative workaround None J 10 9 1 4 Double Click Operation In Diagnostics Center Tree View Does Not Launch an Internal Frame Double clicking on an object in the Diagnostics Center tree view does not launch an internal frame in the content window to open the object in the context pane internal frame Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 1 5 Drag and Drop Functionality Errors Within the Diagnostics Center The drag and drop functionality of a network element from the Diagnostics Center tree view to the content pane a fails to open an internal frame or recycle the contents of an existing frame to display the object s attributes or b results in the Operation Not Supported message dialog box Step 1 Determine if the drag and drop functionality is supported for the dropped object The following objects can be dropped from the tree view to the Diagnostics Center content pane e For the Element tab the Network Node Card Line Port IMA and IMA links objects are supported and Folder objects are not supported
43. syncing due to regular node resync or due to 2 trap e Card is syncing and the trap is related to the card The traps will be buffered in the queue until card resync completes e Summary alarm trap is processed Summary alarm is being uploaded or parsed You can grep the log files for information related to summary alarm traps To verify that the trap has been put into the queue you can do the following grep EMC_TrapQueue_c append ooemc_log grep trap_num You should see something like the following INFO lt EMC_TrapQueue_c append gt entering gt append trap 60303 to trap queue getHdlrLevel 5 getTrapLevel 5 174 Defect information Collect ooemc nts and snmpcomm log files from the opt svplus log directory Possible alternative workaround Manually resync the node If the problem persists perform a coldstart If the problem is still not resolved collect the log files and report the problem Cisco Transport Manager Release 6 0 User Guide Ps E 78 16845 01 AppendixJ Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems This section includes the following information e J 10 5 1 CTM Configuration Center Chassis View Diagnostics Center or Statistics Report Basics page J 53 e J 10 5 2 Basic Issues page J 55 e J 10 5 3 Topology Di
44. utility in the EMS server bin directory returns the provisioned operational state NE_Info_Table NEState and the read only modifiers to the operational state NE_Info_Table NEDiscoveryState The get nestate command returns the following possible values e Preprovisioned e In Service e Out of Service e Under Maintenance e In Service Initializing e In Service Sync Configuration In addition the show nelist command lists all of the NEs with the following information e NE database ID e NE IP address e Model type e SID e Network partition ID e Operational state J 3 Client Connectivity Problems The CTM client might not be able to connect to the CTM server for various reasons Complete the following procedures in the order listed until the problem is resolved J 3 1 Database Is Not Available If the CTM client cannot connect to the CTM server verify that the database is available Step 1 Log into the CTM server as the Oracle user Step2 Enter the following command to connect to the database sqlplus ctmanager ctm123 Step3 If the error message maximum processes exceeded is received the maximum number of database connections has been reached Close several clients or ask the database administrator to increase the maximum number of processes for the database 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting W J 3 2 Database Timeout Occurs J 3 2 Data
45. 00 1024x768 1280x1024 1152x900 Possible depths 8 24 Current resolution setting 1280x1024x76 Current depth 8 To change the color setting to 24 bit log in as the root user and enter the following commands su Password lt password gt usr sbin m64config depth 24 res 1152x900x76 Reboot the workstation for the new color settings to take effect 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting E J413 How Dol Collect Thread Dumps Note The resolution in 24 bit depth is a little lower than is possible with 8 bit depth 1152 x 900 x 76 versus 1280 x 1024 x 76 but the difference is hardly noticeable J 4 13 How Do I Collect Thread Dumps Thread dumps are helpful references when debugging the CTM process To collect thread dumps Step 1 Log into the server workstation as the root user Step 2 On the command line enter the following command thread_dumper lt Group Service gt where e Group is the group name for which to collect thread dumps It can be SM SNMPTRAP NE PM GW e Service is the service name for which the thread dump is required It can be SMService SNMPTrapService CORBAGWService NS 15216NEService NS 15302NEService NS 15305NEService NS 153 10NEService NS 15327NEService NS 15454NEService INS 15454SDHNEService INS 15501NEService NS 15530NEService NS 15540NEService NS 15600NEService NS 15600SDHNEService NS 1580
46. 01 AppendixJ Troubleshooting J 10 2 Discovery Issues at Startup W J 10 2 5 Reachable Node Is Shown as Unreachable If a node that is reachable from CTM is shown as unreachable in the Configuration Center Chassis View Diagnostics Center or Statistics Report complete the following steps Step 1 If the node table reports the alarm state of this node as minor major critical or clear complete the following substeps a Enter the following command to dump the cache of NMServer topod and ILMITopoc kill USR1 signal lt process gt b Check the data collected in the opt svplus log directory and verify whether the node alarm status is correct If the alarm status is correct in the cache dump open a new instance of the GUI and check whether the node shows the correct alarm status in that window Step2 Ifthe node table shows the alarm state of an MGX node as unreachable complete the following substeps a Verify that the node is IP reachable from CTM b Ping the active IP address of the node The active IP address is the IP address of the node that is populated in the node table c Enter the following CLI command to retrieve the community string of the node dspsnmp d Verify that the community string in the node_info table is the same as the community string on the node e Check the nts logs to see if nts has declared the node as reachable See J 10 13 1 NTS page J 127 If nts has declared the node as reachable che
47. 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting W J415 How Do Replace the Alarm Interface Panel J 4 15 How Do Replace the Alarm Interface Panel The Alarm Interface Panel AIP provides surge protection for CTC based NEs This panel has a nonvolatile memory chip that stores the unique node address known as the MAC address The MAC address identifies the nodes that support circuits It allows CTM to determine circuit sources destinations and spans If an AIP fails an alarm will be generated and the LCD display on the fan try assemblies of the NEs will go blank To perform an in service replacement of the AIP you must contact Cisco technical support For contact information go to the technical support website at http www cisco com tac You can replace the AIP on an in service system without affecting traffic by using the circuit repair feature See 7 2 10 Repairing a Circuit page 7 66 J 5 Client Debug Messages Step 1 Step 2 Step 3 Step 4 On Windows you can start the CTM client with a console window that displays exceptions and client debug messages You can use these messages to help troubleshoot any client problems Depending on your network size enter one of the following commands to start the client with a console window bin ctm small debug exe bin ctm medium debug exe bin ctm large debug exe bin ctm highend debug exe When the CTM client exits the console window
48. 0NEService O O O O O O O O O O O O O ONS15801NEService Cisco Transport Manager Release 6 0 User Guide rss D 78 16845 01 Appendix J Troubleshooting J 4 14 How Do l Enable or Disable the Automatic Refresh Data Action W ONS 15808NEService UnmanagedNEService ONS15216PMService ONS15302PMService ONS15305PMService ONS15310PMService ONS15327PMService ONS15454PMService ONS15454SDHPMService ONS15501PMService ONS15530PMService ONS15540PMService ONS15600PMService ONS15600SDHPMService ONS15800PMService ONS15801PMService ONS15808PMService J 4 14 How Do I Enable or Disable the Automatic Refresh Data Action Step 1 Step 2 Step 3 Step 4 The automatic Refresh Data feature automatically refreshes all data being displayed by CTM When automatic refresh is enabled you receive the following prompt Refresh Data action suggested This action will result in closing all windows and might take some time Do you want to continue Yes No In an unstable environment where NEs are synchronizing or NEs change their operational state frequently you might receive the preceding prompt continuously To disable the prompt In the Domain Explorer window choose Edit gt User Preferences The User Preferences dialog box opens Click the Miscellaneous tab To disable the prompt uncheck the Enable Refresh Data Timer check box To enable the prompt leave the check box checked Click OK 78 16845
49. 1 slot 14 line 1 port 0 logPort 0 sCh1 32 sCh2 234 type 1 subType 0 nni 0 netprefix Items of interest in the above example are 21 49 43 The time at which the event occurred Local node11 slot 15 sCh1 1 sCh2 37 The local endpoint Cisco Transport Manager Release 6 0 User Guide Pv130 78 16845 01 Appendix J Troubleshooting J 10 13 Miscellaneous Problems W Remote node 11 The remote endpoint MsgType 100 The message type The mapping is as follows FEEDER_ADDUPD 100 MASTER_SPVC_ADDUPD 101 SLAVE_SPVC_ADDUPD 102 SINGLEENDSPVC_ADDUPD 103 PTOMPSPVC_ADDUPD 104 MASTER_PVC_ADDUPD 105 SLAVE_PVC_ADDUPD 106 MASTER_LOCALDAX_ADDUPD 107 SLAVE_LOCALDAX_ADDUPD 108 FEEDER_DEL 109 MASTER_SPVC_DEL 110 SLAVE_SPVC_DEL 111 SINGLEENDSPVC_DEL 112 SLAVE_SPVC_DEL 111 SINGLEENDSPVC_DEL 112 PTOMPSPVC_DEL 113 MASTER_PVC_DEL 114 SLAVE_PVC_DEL 115 MASTER_LOCALDAX_DEL 116 SLAVE_LOCALDAX_DEL 117 WILDCARD_DEL 121 grep conObjld xxxxxx dmd lt dmd_id gt Msg grep lt NotifyDataBroker gt node x slot y sub1 v sub2 w ooemc lt for more EM queries see EM gt If the DMD received the message but it is not reflected in the database identify which databroker module caused the error First check the DMD to see if it forwarded the message If it did not check for the reason The format may be wrong or there could be other similar problems Try to search grep node slot port vp
50. 2 config cdp Router 2 config commit J 8 11 Cannot Delete an Out of Service NE from CTM If you cannot delete an out of service NE from CTM follow the procedure in 3 5 7 3 Using the prune_ne sh Script to Remove an Out of Service NE from the Database page 3 27 J 8 12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty If you cannot see the software version in the Domain Explorer gt Identification tab and NE Software table is empty complete the following steps Step 1 Verify that the NE is in service and the communication state is Available Step2 Verify that the device is configured with either Telnet or SSH J 8 13 Cannot Use Configuration gt Telnet SSH from the CRS 1 NE Explorer If you cannot use Configuration gt Telnet SSH from the CRS 1 NE Explorer try pinging the NE from the CTM client You might be able to reach the NE from the CTM server but not from the client if they are in different subnets J 8 14 How Do Configure Telnet on the CRS 1 To configure Telnet on the CRS 1 log into the router through the console and enter the following commands Router conf t Router config telnet ipv4 server Router config commit J 8 15 How Do Configure SSH on the CRS 1 Step 1 Enter the following command to verify that the security PIE k9 is installed and activate on the router The PIE k9 is required for encryption decryption IPSec SSH SSL and PKI Router sh insta
51. A transient alarm rule can have only one new alarm associated with it Note There is no ClearAlarmCondID associated with a transient rule Transient alarms can be manually cleared by the user cleared by the same transient alarm occurring twice or purged by NMServer when a high threshold is hit J 10 5 5 3 Alarm Severity and Object Severity Every alarm in an alarm rules XML file is assigned two severities object and alarm The two severities are called Object and Alarm severity These severities do match for most alarms There are some cases however where these severities are different The following is an example of an XML alarm rule where object and alarm severity do not match lt CorrelatedRule State 1 1 gt lt ClearAlarmCond1ID gt 10302 lt ClearAlarmCondID gt lt ClearAlarmCondID gt 10303 lt ClearAlarmCondID gt lt ClearAlarmCond1ID gt 10304 lt ClearAlarmCondID gt lt NewAlarmCondID gt 10301 lt NewAlarmCondID gt lt NewAlarmServAf fect gt 0 lt NewAlarmServAf fect gt lt NewAlarmTransient gt 0 lt NewAlarmTransient gt lt NewAlarmDbPersistent gt 0 lt NewAlarmDbPersistent gt 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems amp lt NewAlarmSev gt 6 lt NewAlarmSev gt lt NewAlarmObj Sev gt 7 lt NewAlarmObjSev gt lt NewAlarmDescrSuffix gt Sync Up h
52. APPENDIX Troubleshooting This chapter offers troubleshooting steps to help solve high level problems while operating CTM or CTM GateWay Follow the troubleshooting procedures described in this chapter before contacting Cisco technical support This chapter includes the following troubleshooting information e J 1 Overview page J 1 e J 2 Server Problems page J 2 e J 3 Client Connectivity Problems page J 11 e J 4 Client Operational Problems page J 13 e J 5 Client Debug Messages page J 20 e J 6 CTM GateWay TLI1 Problems page J 21 e J 7 CTM GateWay CORBA Problems page J 21 e J 8 Cisco CRS 1 and Catalyst 6509 Problems page J 22 e J 9 XR 12000 Problems page J 31 e J 10 Problems with MGX Voice Gateway Devices page J 34 amp Note For information about troubleshooting CiscoView problems see I 9 Troubleshooting page I 6 in Appendix I Using CiscoView to Configure and Monitor ONS 15501 ONS 15530 and ONS 15540 NEs J 1 Overview Troubleshooting involves 1 Identifying the source of the problem Which devices links interfaces hosts or applications have the problem 2 Locating the problem on the network On what VLAN subnet or segment is the problem occurring 3 Comparing current network performance against an established baseline Is the performance better or worse 4 Finding out when the problem started When did you first see the problem Is it recurring 5 Determining the extent of the problem
53. Alarm Components Alarm Client Alarm Service Client Alarm Observer Network Monitor Server Alarm Components Alarm Service Client Handler Alarm Repository lt gt Object Tree ee eee 120750 J 10 5 5 2 XML Schema for Alarm Rules Alarm rules are defined in XML specifically in a file named HOME xml ruledata xml These alarm rules are read once when NMServer starts When events are processed the rules are queried from memory to determine what action with regard to alarms should be taken on the event There are three types of alarm rules defined in the XML schema Correlated Correlated Bitmap and Transient Specifics on each of these three types are as follows 1 Correlated Alarm Rule Type The correlated alarm rule type is the most common in ruledata xml This rule is used when a network element can be in only one of many states at any one time If the entity were to change states then the previous alarm state would be cleared Most network elements managed by CTM fall into this category A simple example is the database sync up status of a node The sync up status can be one of Partial Sync Up Sync Up Failed or In Sync Any one of these states correlates out any other See Figure J 3 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HE J 10 5 Configuration
54. CMSCclient log file under the D Documents and Settings lt username gt log directory Screen snapshots in particular error information messages dialog boxes Cmsvr log file under the opt svplus log directory DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 2 Diagnostics Center Server Specific Issues This section includes the following information J 10 9 2 1 XML Parser Error page J 113 J 10 9 2 2 SNMPCOMM_TIMEOUT Error While Polling MIB Objects page J 113 J 10 9 2 3 Node Sync Up Status Are Not Shown in the Diagnostics Center page J 114 J 10 9 2 4 Node Resync Fails page J 114 J 10 9 2 5 Polling of Real Time Counters Fails page J 114 J 10 9 2 6 Some Real Time Counters Are Not Shown page J 115 J 10 9 2 7 Unable to Start Stop or Modify BERT Operations page J 115 J 10 9 2 8 Unable to Start Stop IMA Link Test Patterns page J 116 J 10 9 2 9 Cannot Perform Connection Diagnostics page J 116 J 10 9 2 10 Miscellaneous Issues or Problems in Diagnostics Center page J 117 Cisco Transport Manager Release 6 0 User Guide ea 78 16845 01 Appendix J Troubleshooting J 10 9 Diagnostics Center Problems W J 10 9 2 1 XML Parser Error Step 1 Step 2 Step 3 The XML parsing error is seen while launching the Diagnostics Center either by the drag and drop method or the right click method for a network element The popup window says Internal Error XML Parsing Error
55. Center Chassis View Diagnostics Center and Statistics Report Problems Figure J 3 Correlated Alarm Rule Diagram Gan 1 nmserver CorrelatedRuleListType nmserver CorrelatedRuleType r ee E FAlwaysClear L DBSyneUpHodeRules EH H Corretatedrule E 5 NewAlarmCondiD 1 0 1 0 NewAlarmServAffect FHewAlarmDbPersistent HewAlarmSev HewAlarmObjSev HewAlarmDescrSuffix 120747 A CorrelatedRule can have any of the following e 1 AlwaysClear Used when a given element never has an alarm such as in a top level network e 0 or more ClearAlarmCondlIds e 1 new alarm which consists of NewAlarmConditionID NewAlarmServAffect and so on 2 Correlated Bitmap Alarm Rule Type The correlated bitmap alarm rule type is different from the correlated alarm rule type because the bitmap rule type can represent the many states an entity can be in at any one particular time The correlated bitmap rule type is used primarily to represent line alarms Lines can have many different alarms associated with them at any given time such as Loss of Signal and Loss of Frame See Figure J 4 Figure J 4 Correlated Bitmap Alarm Rule Diagram nmserver CorrelatedBmRuleListType nmserver CorrelatedBmRule Type e EH clearalarmCondiD 0 00 HewAlarmCondiD HewAlarmServAffect HewAlarmTransient HewAlarmDbPersistent 1 0 1 0 DS1LineRules CorrelatedBmRule E i HewA
56. Center BERT window error information messages dialog boxes Possible alternative workaround Use the CLI DiagProxy for BERT operations J 10 9 2 8 Unable to Start Stop IMA Link Test Patterns Step 1 Step 2 Step 3 Step 4 Step 5 Unable to start stop modify IMA Link Test pattern from the Diagnostics Center Check whether the node is reachable and the community strings are set correctly Check whether the card is in the Active state Check whether the correct VarBinds have been sent for snmpset Check whether the IMA Link Test pattern can be performed from the CLI Use any SNMP tool and try to set the objects shown in the DCServer log file Defect Information Collect the following information for further analysis e DCServer log file from opt svplus log directory e Screen snapshots in particular error information messages dialog boxes Possible alternative workaround Use the CLI for IMA Link test patterns operations J 10 9 2 9 Cannot Perform Connection Diagnostics Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 The connection diagnostics are performed through the CM Server application The Diagnostic Server forwards all the connection diagnostics requests to the CM Server If there are any connection diagnostics errors they are more likely to be in the CM Server application Check whether the sync up is complete Check whether the node is reachable and the community strings are set correctly Check whether the card is
57. Center Launches Another Application page J 74 J 10 7 1 5 Element Tab Internal Frame Does Not Launch page J 75 J 10 7 1 6 Element Tab Drag and Drop Functionality Does Not Launch An Internal Frame page J 75 J 10 7 1 7 Element Tab Create Details Modify and Refresh Button Issues page J 75 J 10 7 1 8 Element Management Drag and Drop Within Configuration Center page J 75 J 10 7 1 9 Cross Application Configuration Center as Drag Source page J 76 J 10 7 1 10 Cross Application Configuration Center s Element Tab as Drop Target page J 76 J 10 7 1 11 Element Tab Internal Frame Displays Incorrect Object or Object Data page J 77 J 10 7 1 12 Configuration Center s Element Tab Does Not Respond GUI Is Grayed Out page J 77 J 10 7 1 1 Cannot Launch the Configuration Center The Configuration Center cannot be launched using one of the following methods Click the Configuration Center icon from the Launch Center or from any application Choose Tools gt Configuration Center from any application Right click the selected node from the hierarchical tree and choose Configuration Center Step 1 Check the configcenter jar file Make sure that the configcenter jar file is located in the opt svplus java jars cwm directory of the target machine Defect Information Collect the following information for further analysis CMSCclient log file from the D Documents and Settings lt username gt log directory Java Console information
58. CiscoTransportManagerServer cfg CTMServer cfg file is not corrupted The file should contain the db config mode auto parameter in the database section If the entry is missing the CTM server configuration file is corrupt Reinstall the CTM server 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J23 Cannot Connect to the CTM Server Step 2 Verify that the first entry in the var opt oracle oratab file looks similar to CTM6_0 oraclesw9i product 9 2 Y If this entry is missing the Oracle database might not be installed The Oracle database is a prerequisite for installing the CTM server J 2 3 Cannot Connect to the CTM Server Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Step 8 Step 9 Step 10 Ping the server s IP address from the client PC or workstation If the ping fails resolve the IP connectivity problem and try again Telnet to the server and log in as the root user Enter the following command to verify that CTM is running showctm If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot enter the following command sudo showctm The server should have at least the following four processes running root 3778 0 1 0 1567 9592 S 16 57 36 0 00 opt CiscoTransportManagerServer bin CTMServer root 3771 0 1 0 4 6208 pts 1 S 16 57 34 0 00 opt CiscoTransportManagerServer
59. Connection Proxy timed out on An ideal case for this would be when a switch timeout is received for a multisegment case and cmegrd is still busy backing off other segments and waiting for the timeout response of additional segments Just to be sure that cmgrd is still running issue the following commands Step 1 Issue the command psg cmgrd to ensure that the process is indeed running Step2 Check for cmgrd coredumps in the corefilesdir directory If there indeed was a coredump collect the corefile Step3 Execute tail f cmgrd log in the logs directory to see if cmgrd is processing requests Reissue the request after waiting for a few minutes Defect Information If you get the same timeout error after reissuing the request then collect the cmegrd log Possible alternative workaround Reissue the request after some time this interval could enable cmgrd to clear its pending request J 10 8 4 13 Cmgrd Addition Modification Deletion Errors Provisioning a Connection Results in Fatal error IOSMGR is not running may need to be manually started Error Related key index entries cmgrd iosmgr When any add modify or delete connection request is made with a Popeye RPM Card cmgrd issues a request to the switch but the request is actually first issued to the process iosmgr which in turn sends the request to the node via an expect script session In this scenario this error can be returned when cmgrd cannot talk to iosmgr via the ILO
60. Connection diagnostics testdelay testcon up down reroute fail with one of the following error messages e TestDelay failed with Switch error e TestConn failed with Switch error Query the connection information from the user_connection table for that connection If possible execute the testdelay testcon diagnostics for the same connection from ConnProxy Service Agent and verify the results Query the connection information from the switch CLI and verify the connection status dspcon or dspchan Execute the testdelay testcon diagnostics from the switch CLI and verify that the results match what was reported in CTM tstdelay tstconn Defect Information Do the following e Capture selnd output e Copy all the cmsvr logs from the opt svplus logs directory e User_connection table query output from the CTM database for the given endpoint if the request is not an addition request e Switch CLI output for the testdelay testconn diagnostics on the same connection Possible alternative workaround Check the status of the connection and clear any alarms due to line port issues adding loopback correcting the physical cable issues by verifying the physical port connections and so on After the connection comes up and is in the Clear state rerun the testdelay testconn diagnostics The testcon is not applicable for single end SPVC connections and on AXSM card types The connection status should be Clear or Fail for the testd
61. EMPLATE_NOT_PRESENT The Connection Template s is not present in the conn_template table e CONN_TEMPLATE_TABLE_INCONSISTENT The Connection Template table contains more than one entry for the template s e CONN_TEMPLATE_PARAMS_NOT_PRESENT The Connection Template s is not present in the conn_templ_param table e DATABASE_ERROR Database error s e UNKNOWN_EXCEPTION Program Error Unknown Exception is caught in s e SUBSYSTEM_EXCEPTION Program Error Subsystem exception s caught in s e INTERNAL_ERROR Program or System Error Function s reports s Verify that the CM Server sdbroker and xdbroker are up and running by entering psg sdbroker psg xdbroker and psg cmserver on the command line Verify if the CM Server restarted while the operation was being done by checking the cmsvr log and searching for the string START OF THE PROCESS Verify that the time stamp of the server process was restarted while the operation was being performed Also check for any cmsvr coredumps under the opt svplus corefilesdir directory Defect Information Collect the following information for further analysis 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HZ J 10 8 Connection Management Problems CMSCclient log file under the D Documents and Settings lt username gt log directory Java Console information in particular any Java raised exceptio
62. For example issue the following command on a node with node_id 9 ls ltr 9 If files have been uploaded for this node then the node resync process is ongoing Make sure that the time stamps for these files refer to the current time amp Note Refer to step 6 for a list of configuration upload files Step4 If there are no files uploaded for a node it is possible that the trap IP address on the switch does not match the IP address used for node discovery It is also possible that an SNMP request failure occurred To check the IP addresses perform the following a Use the CLI command dbnds to display the IP address used for node discovery b Use the CLI command dsptrapip to display the trap IP address c If there is a mismatch use the CLI command cnftrapip to reconfigure the trap IP address Use the IP address used by CTM for node discovery See J 10 4 2 Network Setup and Configuration Required for OOEMC Sync Up Process page J 41 for more information Step5 If the trap IP address is not the issue check whether there is an SNMP request failure Normally after the node resync process is triggered ooemc will send SNMP requests to the switch to start the configuration upload file creation process If the request fails then ooemc will continue resending the SNMP requests until it exceeds the maximum number of retries and declares node resync failure with node mode equal to 5 Cisco Transport Manager Release 6 0 User Guide
63. G connection The reason for the error is that cmgrd could not establish an ILOG connection with iosmgr Step 1 Issue the command psg iosmgr to ensure that the process is indeed running Step2 Check for iosmgr coredumps in the corefilesdir directory If you determine that the iosmgr process is not running then you must do a warmstart to ensure that the process comes up When the process starts the ILOG connection with cmgrd is established Defect Information TIf the iosmgr process does not run after a warmstart then collect the runiosmgr log file Possible alternative workaround None Iosmegr has to be running to be able to provision on Pop RPM cards J 10 8 4 14 Cmgrd Addition Errors Provisioning a Connection Results in Vpi vci ranges retrieval from svc_operation failed Error Related key index entries cmgrd svc_operation This error is only present for a hybrid connection add request cmgrd queries the svc_operation table of the feeder trunk ports for the vpi vci ranges These ranges in turn are given to the sdbroker which in turn returns an available vpi vci that cmgrd uses to set on this feeder trunk port Cisco Transport Manager Release 6 0 User Guide La 78 16845 01 Appendix J Troubleshooting Step 1 Step 2 Step 3 Step 4 J 10 8 Connection Management Problems W The reason for the error is that when cmgrd queries the svc_operation table for the feeder trunk port s vpi and vci ranges the entry
64. I J 10 5 3 Topology Discovery Issues This section includes the following information e J 10 5 3 1 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Node page J 57 e J 10 5 3 2 Node Information Incorrect on the Configuration Center Chassis View Diagnostics Center or Statistics Report page J 57 e J 10 5 3 3 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows a Node that Is Not in the Network page J 57 e J 10 5 3 4 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows Incorrect SyncState for a Node page J 58 e J 10 5 3 5 Duplicate Nodes Are Displayed on the Configuration Center Chassis View Diagnostics Center or Statistics Report page J 58 Cisco Transport Manager Release 6 0 User Guide P56 78 16845 01 AppendixJ Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il J 10 5 3 1 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Node Step 1 Check whether the nodes are discovered See J 10 2 1 No Nodes Are Discovered page J 34 for more information Step2 Check whether NMServer has the nodes in its cache This can be done by using the command nmControl on the CTM CLI Step3 If the nodes or trunks are in the NMServer cache open a new GUI and verify whether the nodes are shown in it Defect Information Col
65. MGRD First use the dbemap command opt svplus tools dbcmap to identify the DMD that is used to process the node Then dump that DMD s cache Verify that the DMD does not contain the endpoints in question in its cache b Determine why the DMD doesn t have the information on endpoints that exist on a switch You first need to determine if the EM sent the add message to the DMD See J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 If the add message is not found and the logs do not go back to coldstart then you will need to check to see if the EM has done any processing of this endpoint Check for the endpoint in the xxx_Connection table If the endpoint is not there check the EM to determine why it did not process the endpoint If the endpoint is in the xxx_connection table you know that the EM processed it but you do not know if it forwarded it to the DMD Defect Information e Ifthe problem is an intermediate endpoint and it is on the switch but the EM didn t process it or didn t send the message to DMD check the EM log and nts log from opt svplus log e Ifthe problem is within the DMD you will need the DMD logs DMD message logs and the DMD cache dump Possible alternative workaround None 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems J 10 8 4 8 Cmgrd Sdbroker Modification Deletion Errors Modify
66. ND Successfully dumped file tftp lt IP_address gt lt TFTP_directory gt lt NE_name gt _node_process_1100886887 212000 If events do not appear after 15 minutes verify that configuration change notifications are enabled See J 8 3 CTM Does Not Receive Configuration Changed Notifications from the CRS 1 page J 23 If events are present verify that the TFTP server is reachable from the CTM server To do this ping the IP address that is present in the event notification If the IP address is not reachable configure the routes so that it becomes reachable Alternately include a mapping file in the opt CiscoTransportManagerServer cfg hfr IPAddressMapping cfg directory that includes the following mapping lt IP_address_in_event_notification gt lt reachable_IP_address_of_the_same_TFTP_server gt For example 172 19 64 51 223 255 254 254 J 8 7 CRS 1 PM Tables Collect Data at the Wrong Collection Interval You might experience a problem where the PM data collection interval is set to 15 minutes but the CRS 1 PM tables are collecting data more frequently or less frequently The CRS 1 collects and dumps data to a binary file on a TFTP server at user configured intervals based on the IOS XR CLI After dumping the PM data the CRS 1 notifies CTM with the location of the file CTM then collects PM data from the CRS 1 Therefore you should configure the CRS 1 to collect data every 15 minutes J 8 8 Memory Backup or Restore Fai
67. SNMP configuration on the Catalyst 6509 enter the following command cat6509 enable show snmp J 8 19 How Do Load a Software Image on the Catalyst 6509 Complete the following steps Step 1 Enter the following command to view all the software images on the Catalyst 6509 cat6509 enable dir bootflash Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN Appendix J Troubleshooting HZ J 8 19 How Do Load a Software Image on the Catalyst 6509 The output is similar to the following example length date time name 2 5849256 Aug 09 2004 14 41 25 cat6000 sup2 6 3 7 bin 3 8801412 Aug 09 2004 15 30 57 cat6000 sup2k8 8 3 2 bin 4 5713736 Aug 11 2004 13 08 02 cat6000 sup2 6 3 2a bin 5 8182352 Aug 12 2004 09 15 00 cat6000 sup2k8 8 2 1 bin 17 1786 Aug 23 2004 14 16 30 6509svt 171 180 19 1688 Sep 24 2004 15 26 40 6509svt 171 180 cfg 3346628 bytes available 28634940 bytes used Step2 Enter the following command to retrieve the image name that is currently on the device cat6509 enable whichboot The output is similar to the following example Boot image name is bootflash cat6000 sup2 6 3 7 bin Step3 Enter the following command to delete the existing image cat6509 enable del lt image_name gt Step4 Enter the following command to remove the deleted image cat6509 enable squeeze bootflash Step5 Enter the following commands to copy the image to flash memory
68. Settings lt username gt log directory e Java Console information in particular any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 2 Connections Tab Drag and Drop Does Not Launch Connection s Internal Frame The drag and drop functionality of a network element from the Configuration Center tree view to the Connections tab s content pane a fails to open an internal frame b fails to recycle the contents of an existing frame to display the dropped object s attributes or c results in the Operation Not Supported message dialog box Cisco Transport Manager Release 6 0 User Guide 34 E 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W Step 1 Determine if the drag and drop functionality is supported for the dropped object The Connection tab supports the drag and drop functionality for the Node Card Line and Port objects The drag and drop functionality of Folder IMA and IMA link objects is not supported Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information in particular any java raised exceptions e Screen snapshots in particular
69. Troubleshooting W J 10 7 Configuration Center Management Step 4 If the node or card is supported open the opt svplus log CMSCclient log file and check whether the FDN of the object being launched matches the information from Step 1 Contact CTM engineers with appropriate defect information A typical example of the log follows DD Mon YYYY HH MM SS DEBUG appsConfigCenter CcElementInternalFrame modifyActionPer formed DD Mon YYYY HH MM SS DEBUG appsConfigCenter CcTabularView modify DD Mon YYYY HH MM SS INFO appsConfigCenter Selected fdn NW X RND XX CD X LN X PT X Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt og directory e Java Console information e Contents of opt svplus log configserver log file Possible alternative workaround None J 10 7 2 7 Element Data Inconsistent with Switch Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Information displayed in the GUI and the switch is not the same Make sure that the window has been launched for the correct element in the GUI If not contact CTM engineers with the defect information SSH or Telnet to the switch change a few parameters and see if the change is reflected in the GUI If not contact CTM engineers Open the opt svplus log configserver log file and find the MIB OID of the element that displays the inc
70. Type cannot be PVC for connections with a PNNI segment The problem must be fixed on the CM GUI while adding the connection Step 1 Verify that the endpoints that are being used to add the connection are not on PNNI nodes or feeders to PNNI nodes Step2 Change the end to end type in the CM GUI to either SPVC or Hybrid Defect Information Collect the cmsvr logs from the opt svplus log directory Possible alternative workaround None Cisco Transport Manager Release 6 0 User Guide P30 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W J 10 8 3 2 Unsupported Service Type for the Foresight Connections for PNNI SPVC Endpoints Step 1 Step 2 During connection addition the error message could state CM Server does not support Connection type Service type and Endtoend type between Card type and The parentheses are filled with the appropriate values Determine if the service type is applicable to the endpoints Determine the cards on which the connection is being added and the service type given in the GUI This error is thrown when the service type is either abr fs atfst for a PVC SPVC or Hybrid end to end type and the card is either a pxm axsm rpm or frsm12 card Defect Information Collect the cmsvr logs from the opt svplus log directory Possible alternative workaround Change either the endpoints or the service type to add the connection J 10 8 3 3 CE CE Conn
71. User Guide jg AppendixJ Troubleshooting WE J10 4 Equipment Management Problems Step 8 Step 9 9 SM_CARD_01_RPM CF 10 PNNI_01_CC CF Files 1 and 2 are uploaded for each NBSM Files 4 to 7 are uploaded for each AXSM File 9 is uploaded for all RPM RPM PR cards on the switch The following files are uploaded from the MGX NE It includes VISM AXSM VXSM SRM RPM RPM PR and RPM XF cards 1 CARD_01_CC CF SM_1_slot CF SM_1_slot CS SM_CARD_01_slot CF SM_CONN_01_slot CF SM_ALARM_01_slot CF SM_CON_UPDATE_01_slot CF SM_SC_slot _transactionID_date CF SM_IC_slot _transactionID_date CF SM_CARD_01_SRM CF SM_CARD_01_RPM CF 12 PNNI_01_CC CF The difference between the items in these lists is in the files uploaded for AXSM cards and RPM XF cards For the switch software release 4 0 or later the static file SM_SC_slot _transactionID_date CF and incremental file SM_IC_slot _transactionID_date CF are uploaded For the switch software release 3 0 and earlier conn alarm and conn update files are uploaded for each AXSM Files 4 to 7 in the second list are uploaded for each RPM XF card and file 11 is uploaded for all RPM RPM PR cards on the switch o o N SF FT FPF YS DN _ In another scenario in which the node mode stays in 2 for a very long time the parsing of one particular config upload file has taken a very long time and has not completed To view the parsing process enter the following command tail
72. _the_Catalyst_6509 gt lt mask gt Enter the following command to verify that the device is configured correctly with the default route cat6509 enable set ip route default lt gateway_IP_address gt Cisco Transport Manager Release 6 0 User Guide P22 E 78 16845 01 Appendix J Troubleshooting Step 3 Step 4 J 8 3 CTM Does Not Receive Configuration Changed Notifications fromthe CRS 1 W Verify that CTM supports the software image that is running on the Catalyst 6509 In the Domain Explorer choose Administration gt Supported NE Table and verify that the software version is supported amp Note For the Catalyst 6509 the supported software version is Release 6 3 7 or later Verify that the community string shown in the Domain Explorer gt Address tab is same as the community string on the device If there is a mismatch enter the following command to change the SNMP community string on the device cat6509 enable set snmp community read only lt community_string gt For example cat6509 enable set snmp community read only public J 8 3 CTM Does Not Receive Configuration Changed Notifications from the CRS 1 Step 1 Step 2 Enter the following commands to verify that the CTM server hostname and logging events level are configured in the running configuration Router sh run include domain ipv4 host Router sh run include logging events level informational If the CTM server ho
73. ace data rates template default sample size 1 sample interval 15 l performance mgmt statistics mpls TE link template default sample size 1 sample interval 15 performance mgmt statistics node memory template default sample size 1 sample interval 15 f performance mgmt statistics node process template default sample size 1 sample interval 15 performance mgmt statistics mpls TE tunnel template default sample size 1 sample interval 15 performance mgmt statistics mpls interface template default sample size 1 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting W J 8 7 CRS 1PM Tables Collect Data at the Wrong Collection Interval Step 3 Step 4 Step 5 Step 6 sample interval 15 If the files are not being dumped to the target directory on the TFTP server verify that the TFTP server is using the correct tftpd tftp daemon binary The tftpd should be a version that allows the CRS 1 to create new files in the specified directory The default tftpd only allows pre existing TFTP files to be written The default tftpd does not create new TFTP files If the files appear on the TFTP server but do not appear in the PM table verify that the CTM server is being notified of PM TFTP file dumping events To do this choose Fault gt Alarm Log and check for events that arrive every 15 minutes and resemble the following output pm_collector 301 PM_COLLECTOR 6 TFTP_SE
74. aceType are optional and will not be seen in many files for example ABC Card xml is a valid XML filename If the Platform part of the XML filename is incorrect missing check the Node table to see if it is correctly populated If the Card part of the XML filename is incorrect missing check the Card table to verify if it is correctly populated If the InterfaceType part of the XML filename is incorrect missing check the appropriate table to verify if itis correctly populated This table generally corresponds with the line port table of the card for which this error is noticed If the EntityName part of the XML filename is incorrect contact the CTM engineers with the defect information required Defect Information Collect the following information for further analysis Complete screen snapshots for the investigative commands used for debugging this issue DCServer log file from the log directory Possible alternative workaround None J 10 9 2 2 SNMPCOMM_TIMEOUT Error While Polling MIB Objects Step 1 The Diagnostics Center shows an SNMP timeout error when polling real time counters Check the status of the node 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 9 Diagnostics Center Problems Step2 If the node is reachable then check whether the community strings are set properly Step3 Check the DCServer log file to determine whether the community string u
75. ager Release 6 0 User Guide Py120 78 16845 01 AppendixJ Troubleshooting J 10 11 Statistics Report Problems W J 10 10 2 PM Parsing Issues This section includes the following information e J 10 10 2 1 StatsParser Generic Troubleshooting page J 121 e J 10 10 2 2 StatsParser Files Are Parsed But the Files Are Listed in the BadFileList page J 121 J 10 10 2 1 StatsParser Generic Troubleshooting StatsParser reads the data from the stat files and inserts the data into the database When performing generic troubleshooting always do the following Step 1 Verify that the statsparser process is running by executing the psg statsparser command and checking for a result Step2 Check the log level for the statsparser You can increase the log level by editing the config file config statsparser conf and changing the value in the LOG_LEVEL field to 7 J 10 10 2 2 StatsParser Files Are Parsed But the Files Are Listed in the BadFileList While parsing the stats files and inserting the data into the database if the StatsParser encounters a problem with the parsing of data in the stats file or a problem with the format of the file the file is put in the BadFileList The node entry is not available in the scmnode table Perform the following query and verify whether e All the nodes that are in the Collection state are listed in the output Also determine whether Node entries are duplicated The query is echo SELECT fro
76. agnostics Center or Statistics Report J 10 5 5 6 Transient Event Has Disappeared Unexpectedly Step 1 Step 2 Step 3 Transient alarms behave somewhat differently depending on whether the entity is managed by the network element or the NMS itself If the transient alarm is on a managed network element for example an alarm generated by an FTP transfer failure then the alarm is self clearing This means if the same transient alarm happens more than twice the previous active alarm is cleared by the latest If the transient alarm is not a managed network element but rather the NMS itself then the alarm is not self clearing and there will be multiple occurrences of the same alarm conditions Since these NMS alarms or events that pertain to the NMS are not cleared by the NMS NMS alarms are not correlated They must be cleared manually by the operator If these events are not manually cleared the list will grow only to the value of MAX_ACTIVE_NMS_EVENTS that is specified in the NMServer conf file Once the list of NMS alarms reaches MAX_ACTIVE_NMS_EVENTS NMServer will automatically clear the oldest alarms in the list to make room for the new events The number of events that will be cleared each time the maximum is reached is also specified in NMServer conf That configuration parameter is called EVENTS_TO_CLEAR_ WHEN_MAX_ REACHED If there was a transient event that has unexpectedly disappeared it is most likely because NMServer purged t
77. ait for five poll cycles while CTM reestablishes connectivity with the NE Step4 To test IP connectivity to the NE from the CTM server enter the following command from the Solaris workstation running CTM ping lt NE_IP_address gt Step5 To verify that the NE is running the version of system software supported by CTM open a CTC session to the NE and check the NE software version Step6 Verify that the username and password that CTM uses to reach the NE exist on the NE J 2 12 ONS 15501 ONS 15540 or ONS 15530 Is Not Reachable Step 1 Enter the following command on the server to verify IP connectivity between the server and the NE ping lt NE_IP_address gt Step2 Ifthe NE is reachable through IP verify that the correct read community string is configured on the server e On the server choose Administration gt ONS 155XX gt ONS 155XX SNMP Settings Table to view the community strings set in CTM e On the NE enter the following command to view the read community string set on the NE show run begin SNMP Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting Step 3 J 2 13 Circuits Are Not Displayed W e In the running configuration the community strings are shown in entries similar to the following snmp server community public RO snmp server community private RW In this example the read string is public and the read write string is private If IP connectivity and co
78. al Error object index 0 Defect Information Collect the following information for further analysis e Collect all complete screen snapshots for the investigative commands used for debugging this issue e Contents of opt svplus log CMSCclient log file Possible alternative workaround None J 10 8 Connection Management Problems This section includes the following information e J 10 8 1 Configuration Center GUI Framework Issues page J 83 e J 10 8 2 Configuration Center GUI CM Server Reported Errors page J 87 e J 10 8 3 Cmsvr Errors page J 90 e J 10 8 4 CMGRD page J 95 J 10 8 1 Configuration Center GUI Framework Issues The Configuration Center management functions are divided into the following categories e Connections Manages the connections Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EN AppendixJ Troubleshooting HZ J 10 8 Connection Management Problems wy Note For connection management the Configuration Center communicates with the Connection Management Server process e NE Manages the nodes and their components S Note For network element management the Configuration Center communicates with the Config Server process The Configuration Center s Connection tab is the main window for creating modifying and viewing network connections The Connection tab content pane provides additional tabs for creating connections Advanced Mode and Quick Mode tabs
79. alled Enter the following command to view the status of the CTM server processes showctm If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot enter the following command sudo showctm If there is a line containing CTMServer the CTM server is running If there is no line containing CTMServer the CTM server is not running Proceed to Step 3 wy Note You can also check the ctmop log file in opt CiscoTransportManagerServer log to determine if the server was stopped by another user or if it stopped abnormally If it stopped abnormally proceed to Step 3 Run the getinfo sh CTM server tool and send the data to Cisco technical support for analysis If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot enter the following command sudo getinfo sh Start the CTM server by using the ctms start script in the opt CiscoTransportManagerServer bin directory a Log in as the root user b Change the directory to opt CiscoTransportManagerServer bin and enter the following command ctms start If you do not have root user privileges but you belong to the UNIX group that can use sudo functionality to run commands as nonroot enter the following command sudo ctms start If the preceding procedure does not solve the problem complete the following steps Verify that the opt
80. ap 6 50017 15668881 to EMC 5 24596 The above line says nts delivered Node 42 Trap 50017 Sequence Number 15668881 to EMC child 5 PID 24610 In normal case you see the following cluster for each trap in nts log 21359 46 21359 46 01 46 49 WARNING N15 Trap 6 60303 25278 01 46 49 WARNING N15 Trap 25278 buffered 21359 48 01 46 49 WARNING N15 Trap 6 60303 25278 to CSA 21359 50 01 46 49 WARNING N15 Trap 6 60303 25278 to RtmProxy 21359 60 01 46 49 WARNING N15 Trap 6 60303 25278 to ooemc 24610 21359 58 01 46 49 WARNING N15 Trap 6 60303 25278 to EMD 21359 49 01 46 49 WARNING N15 Trap 6 60303 25278 to HPOV where e Line 1 means Trap 6 60303 25278 is received by nts e Line 2 means it is successfully buffered by nts e Lines 3 to 7 mean it is delivered to the clients who registered to receive it in their XML filter setting Step2 If you cannot find the particular trap verify if the switch is sending traps with a wrong trap IP address If the switch trap IP is wrong traps from it is declared as coming from unmanaged node and dropped The Trap IP address has to be the primary IP address of the switch Otherwise nts cannot correlate the trap with the CTM node information and has to discard that trap See J 10 13 1 2 Config Change or Provisioning Activity Not Reflected on CTM page J 127 for more information The nts prints a different message when a trap comes from a node i
81. are also incorrect go to step 3 Step3 In some cases a more detailed review of the message flow between EM DMD and sdbroker is necessary to determine the source of the error a Verify that DMD received the message from EM Search the message log for the connection This search is on the DMD first followed by the sdbroker and xdbroker xpvc only The search is either by connection object ID or node slot port vpi vci The DMD logs are checked and if no messages are found then the ooemc or other EM processes are checked If the message is found or the logs are incomplete then you need to look deeper into the databroker processes If the DMD or the EM logs are complete and the message was not sent to DMD then it is most likely an EM issue Find the DMD by using the command opt svplus dbcmap d lt node_id gt The output will provide you with the DMD whose logs need to be queried grep node lt node_id gt slot lt slot gt port lt logical_port gt sChl lt vpi or dlci gt scCh2 lt vci gt dmd lt dmd_id gt Msg Each output line is similar to this 7 21 49 43 1058910582 MsgType 100 connObjId 65665 connStatus 1 secStatus 1 upperLevelAlrm0 oamStatus 0 channelNo 4272512 termination 1 masterFlag 0 wildCardFlag 0 controllerType 0 subType 55 1PercUtil 100 rPercUtil 100 persistentSlave 1 prefRoutelId 0 directRouteFlag 0 Local node 11 slot 15 line 1 port 0 logPort 0 sCh1 1 sCh2 37 type 1 subType 0 nni 1 netprefix Remote node 1
82. are disabled when a corresponding operation is not supported Open the opt svplus log configserver log file for the MIB If the Create button is disabled use any SNMP tool to create the object using the SNMP SET operation e Ifthe SNMP SET operation passes contact CTM engineers 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 7 Configuration Center Management Step 3 Step 4 Step 5 e Ifthe SNMP SET operation fails SSH or Telnet to the CLI and try to add the element If the operation succeeds contact CTM engineers If the Delete button is disabled use any SNMP tool to delete the object using the SNMP SET operation e Ifthe SNMP SET operation passes contact CTM engineers e Ifthe SNMP SET operation fails SSH or Telnet to CLI and try to delete the element If the operation succeeds contact CTM engineers The only reason why a Details button is disabled is that all the information related to the element is already displayed in the tabular view Check the configserver log file to determine if any MIB entry appropriate to the element is available If a MIB entry is available perform an SNMP walk on the MIB entry using any SNMP tool If there are is information in addition to what is displayed in the tabular view and if it is important to be displayed in the frame contact CTM engineers The only reason why a Refresh button is disabled is that all the information displa
83. arms to the dump file Client Manager Cache Dumps the information about all the clients to the dump file Error Statistics Dumps the error information associated with the sync up to the dump file Sync Up Status Dumps the sync up state to the dump file Cisco Transport Manager Release 6 0 User Guide Ps 78 16845 01 Appendix J Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il Configuration Dumps the configuration data to the dump file e nmClient This utility is used to isolate an issue between client and server It is used to query the NMServer the same way as the Configuration Center Chassis View Diagnostics Center or Statistics Report GUI queries the server Output is redirected to opt svplus log nmClient log Cache dumps are redirected to opt svplus log nmClient dump You must perform the Register Client operation before performing any other operations When you are finished using the client perform the Unregister Client operation Register Client Registers with the server Update Filter Updates the subscription filter with the server passing the FDN Unregister Client Unregisters with the server Get Topology Gets the topology information of networks and nodes Get Children Gets the list of children for a particular object in the tree Get CwmInfo Gets the CTM sync up state information Get ManagedObject
84. as not started yet lt NewAlarmDescrSuffix gt lt CorrelatedRule gt This alarm will occur if the southbound processes EMs send a node message with EM sync up status as or 1 If that does occur then the node should be unreachable severity value 7 in the tree view Note that there is no unreachable severity in the alarm list This is why there are two severities for each alarm Unreachable alarms have Critical value 6 severity in the alarm list Another instance when alarm and object severity do not match is for aggregate port alarms Aggregate port alarms are alarms that summarize the condition of the connections on the port Since these alarms should not affect the severity of the port the object severity of these is alarms is Clear value 3 The following is the XML for an aggregate port alarm lt CorrelatedBmRule StateBm 1 gt lt NewAlarmCondID gt 40801 lt NewAlarmCondID gt lt NewAlarmServAf fect gt 0 lt NewAlarmServAf fect gt lt NewAlarmTransient gt 0 lt NewAlarmTransient gt lt NewAlarmDbPersistent gt 1 lt NewAlarmDbPersistent gt lt NewAlarmSev gt 5 lt NewAlarmSev gt lt NewAlarmObj Sev gt 3 lt NewAlarmObjSev gt lt NewAlarmDescrSuffix gt Aggregate Port alarm One or more connections on this port are in primary failure lt NewAlarmDescrSuffix gt lt CorrelatedBmRule gt Note The alarm severity NewAlarmSev tag is Major value 5 J 10 5 5 4 Severity Shown on Tree View Does Not Match Severi
85. ay CORBA option is installed later the Control Panel does not refresh automatically after the CORBA installation to show that CTM GateWay CORBA is installed This problem occurs because the CTM GateWay CORBA installation is handled by a script that notifies the database that the component is installed but does not notify the CTM server To work around this problem click the Refresh Data button in the Control Panel It might take some time for the Control Panel to show that CTM GateWay CORBA is installed You might need to close and reopen the Control Panel to see the change 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J Cisco CRS 1 and Catalyst 6509 Problems J 8 Cisco CRS 1 and Catalyst 6509 Problems This section describes troubleshooting procedures for problems related to the Cisco CRS 1 and the Catalyst 6509 J 8 1 CRS 1 Communication State Remains Unavailable After Marking the Device as In Service Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 Verify that the CRS 1 is configured with hfrems as the username and hfrems as the password with root system privileges By default CTM expects hfrems as both the username and password You can change the username and password in the Control Panel gt Security Properties gt CRS 1 tab Enter the following commands to verify that the XML CORBA agent service is running Router config
86. b connection client pending_reply_timeout value J 9 4 XR 12000 Alarm and Event Reporting Issues Step 1 Step 2 If CTM does not receive XR 12000 alarm and event notifications complete the following steps Enter the following commands to verify that the CTM server hostname and logging events level are configured in the running configuration Router sh run include domain ipv4 host Router sh run include logging events level informational If the CTM server hostname and logging events level are not configured in the running configuration enter the following commands Router conf t Router config domain ipv4 host lt CTM_server_hostname gt lt IP_address gt Router config logging events level informational Router config commit J 9 5 XR 12000 PM Collection Issues Step 1 The XR 12000 collects and dumps data to a binary file on a TFTP server at intervals configured by the user through the CLI After dumping the performance data the XR 12000 sends a notification indicating the location of the file Upon receiving this notification CTM collects performance data from the XR 12000 Therefore the network operator should configure the XR 12000 to collect data every 15 minutes Enter the following command to verify whether the TFTP server and directory are configured in the running configuration performance mgmt resources tftp server lt server_IP_address gt directory lt directory_on_TFTP_server gt
87. base Timeout Occurs Step 1 Step 2 Step 3 Step 4 Reduce the scope of the query by selecting a group or an NE and not the entire domain before opening tables such as the Alarm Browser Alarm Log Audit Log and PM tables Increase the client database query timeout period by editing the ems client cfg file in the C Cisco TransportManagerClient config or opt CiscoTransportManagerClient config directory Add hardware resources to the Oracle database server Ping the Oracle database server from the client to verify the response time Increase the bandwidth if the round trip response time is inadequate J 3 3 Are the CTM Client and the CTM Server Connected Step 1 Step 2 Step 3 If the database is available check connectivity between the client and the server To see the CTM server IP address enter the following command on the Solaris workstation that is running the CTM server ifconfig a The command output looks similar to the following example hme0 flags 863 lt UP BROADCAST NOTRAILERS RUNNING MULTICAST gt mtu 1500 inet 192 168 120 93 netmask ffffff00 broadcast 192 168 120 255 The IP address is the address following the inet field To verify that the physical connection between the CTM client and the CTM server does not have problems enter the following command from the CTM client ping lt IP_address gt where the IP address belongs to the Solaris workstation that runs the CTM server If the ping comman
88. ble Switch Error Step 1 Step 2 Related key index entries cmgrd object exists When any add connection request is issued an SNMP VarBind is set on the switch via cmgrd The switch can reject this VarBind and the subsequent cmgrd query to the switch error table can return these errors The reason for the error is that the connection that CTM is trying to add already exists on the switch This could be a user endpoint or an intermediate endpoint Retry the same connection If this second attempt succeeds it indicates that the intermediate endpoint s vpi vci was already present on the switch If this is the case look at the cmgrd sdbroker error EndPoint Exists Local Remote Both end of the connection already exists If the switch returns the same error on the second attempt this could indicate an inconsistency between the CTM database and the switch and it must be debugged by a CTM DE Defect information Save the cmgrd log ConnProxy log cmsvr log and sdbroker log Possible alternative workaround Retry the connection addition more than once If it still fails a CTM DE will have to look into the issue J 10 8 4 19 Cmgrd Addition Modification Errors Provisioning a Connection Results in Requested cell rate Iscr lper or rpcr rscr is too high Switch Error Step 1 Step 2 Related key index entries cmgrd cell rate During an add or modify connection request it is possible that the port the connection is being set
89. ceived when logging into the CTM client the CTM server might be initializing Wait for five minutes while the CTM server finishes initializing then try to log in again Alternately check your username and password and enter them again The username and password are case sensitive J 4 Client Operational Problems This section describes troubleshooting procedures for CTM client operational problems J 4 1 Model Index Is Unknown Step 1 Step 2 Step 3 Step 4 Step 5 If a new NE is added to the CTM domain and marked as In Service but the model index is unknown complete the following steps Verify that the IP address and other parameters are entered correctly The CTM server might not be able to establish connectivity to the GNE for the NE Verify that the GNE for the NE is marked as In Service and that the GNE is correctly populated in the CTM domain Verify that the GNE is DCC connected to the NE The CTM server might not recognize the NE s software version In the Domain Explorer tree select the NE and choose Fault gt Test NE Connectivity If the NE is available contact Cisco Systems and verify that the version of the software on the NE is compatible with the CTM server being used Use the Supported NE Table to provide the new NE software version that the CTM server will recognize The CTM server might be configured to attempt communication with new NEs after a certain delay To change the health poll frequency a In
90. cess for which files are not getting transferred by cwmftpd Related key index entries login failed J 10 13 3 5 cwmftpd File Not Available On Switch A file is not transferred because it is not available Step 1 Check whether the file is available in the target switch Step2 Check whether cwmftpd log contains an exception like this FtpException 3 FTPERR_550 Requested action not taken File unavailable e g file not found no access 550 File stat 1 05 Con 062020031215 not found or permission problem Defect Information TIf the file is available but was not FTP d and an FTPERR_550 exception is thrown then collect cwmftpd log cwmftpd request_log and logs of the process for which the file was not transferred by cwmftpd and report the problem Possible alternative workaround None Related key index entries file unavailable ftperr_550 requested action not taken J 10 13 3 6 FTP Sessions in Switch A switch allows only four FTP sessions When cwmftpd makes a request to open a session if the switch cannot service the request because it has reached its session limit or if the file is locked then the switch will respond with the following error message 499 Session limit reached queuing lt IP address gt key lt Nodename XXXXXXXX gt Key is a unique string for the node which is used later when the session is available CTM waits on a TCP Port 5120 for the switch to connect to When a session is available the swit
91. cessful Node Resync when Node Mode Is 3 MGX nodes are managed by the ooemc component in CTM Equipment management EM uses coldstart warmstart and periodic resync to synchronize network data Each of these resync methods involves configuration file upload parsing and database population The node mode in the node table indicates the state of the synchronization process The possible node modes are 1 2 3 and 5 The expected mode for a successful node resync is 3 If the node mode stays in other modes for a long time it indicates a resync problem Any synchronization problem could be caused by the switch SNMP FTP or ooemc You must investigate different areas to identify the root cause of any synchronization problem The following is the flow of events in the ooemce coldstart synchronization process 1 The topo process discovers a node in the network The process changes the node mode in the node database table to 1 It then notifies the ooemc component for the discovery The ooemc changes the node mode from 1 to 1 after the component begins managing the node 2 When the nts component is able to register with the node it sends a link up message to the ooemc that manages the node The ooemc component changes the node mode from to 2 to signify the start of the node synchronization 3 The ooemc component sends an SNMP bulk file creation request to the switch If the request is allowed the switch sends a trap 60901 to CTM to sign
92. ch connects to the predefined port and sends the key as data to CTM CTM uses the key to identify which switch it needs to open the session with The session should now be opened without encountering a failure and cwmftpd can proceed with the FTP request Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN Appendix J Troubleshooting Hi J 10 13 Miscellaneous Problems The switch will reserve the session for the CTM station for 30 seconds before it services other stations CTMftpd will retrieve multiple files within a single FTP session A maximum limit per session will be imposed This feature is supported in CTM with MGX 8850 Release 4 0 or later For Switch Software SWSW earlier than MGX 8850 Release 4 0 the switch responds with 421 Session limit reached closing control connection The number of FTP sessions currently opened by the switch can be determined by executing the switch command dsptasks For each FTP session dsptasks output will have an entry like following FtpdServl 0x9a00al 0x82d8a8a0 If there are two sessions there will be two entries Related key index entries switch FTP session J 10 13 3 7 421 Session Limit Reached Step 1 For SWSW earlier than MGX8850 Release 4 0 the switch responds with 421 Session limit reached closing control connection when it reaches the limit of four FTP sessions Check whether all four sessions have been opened by opening a manual FTP session to the swi
93. ck the topod log for nonRoutingNodeMsg for this node Verify that the alarm status in this message is the correct alarm status Defect information Collect the following information for further analysis e ILMITopoc log topod log NMServer log nts log and ooemc log e Dump outputs of ILMITopoc topod and NMServer Enter the following command kill USR1 signal lt process gt e Output of the switch CLI selnd and dbnds J 10 2 6 CTM State Is Incorrect If the management state of the node in the node table mgmt_state column shows as DOWN 2 or UNKNOWN 0 even when the node is reachable by IP complete the following steps Step 1 Check the trap manager registration of the CTM station on the node Log into the node and enter the following command to check whether the CTM station has registered to the node for traps This is required for the node to be declared as manageable mgmt_state UP dsptrapmgr Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN AppendixJ Troubleshooting W J 10 3 Discovery Issues at Runtime Step2 Enter the community strings SNMP GET and SNMP SET of the node through the Domain Explorer gt Network Element Properties pane gt NE Authentication tab The SNMP strings might be incorrect in the node_info table The strings can be verified if the decrypt tool for the encrypted strings is available in the database Step3 Check the nts log to see if trap registration succe
94. client problems Depending on your network size enter one of the following commands to start the client with a console window ctmcdebug start small ctmcdebug start medium ctmcdebug start large ctmcdebug start highend J 6 CTM GateWay TL1 Problems Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step7 If the OSS cannot connect to CTM GateWay TLI or if the OSS does not receive a response check the cables address bindings and configuration If the connection times out check the cables and connections Verify that the address bindings between the OSS and the Solaris workstation running the CTM GateWay TLI service are correct Using the ping command verify connectivity between the OSS and the Solaris workstation running the CTM GateWay TLI service Verify that the TCP IP socket connection is still open for the OSS to CTM GateWay TLI port Choose Administration gt Control Panel Click GateWay TL1 Service and confirm that the service status is Active Verify that the username and password under Control Panel gt Security Properties match the user configuration of the NEs that CTM GateWay TLI will manage Verify that the username and password in the Domain Explorer gt Network Element Properties gt NE Authentication tab match the user configuration of the NEs that CTM GateWay TLI will manage J 7 CTM GateWay CORBA Problems If the CTM server is installed without the CTM GateWay CORBA option and the CTM GateW
95. closes If you want to capture and save the messages when the client exits you must redirect the output to a file In this case the console will not be displayed To save exceptions to a file open one of the following files that corresponds to your network size e bin ctm small debug lax e bin ctm medium debug lax e bin ctm large debug lax e bin ctm highend debug lax amp Note Itis strongly recommended that you save a copy of the original lax file before modifying it Using a text editor open the appropriate lax file and change the following lines by replacing console with a filename lax stderr redirect console lax stdout redirect console When specifying filenames make sure to use escaped backslashes such as c myfolder client log For example to save messages to the C myfolder client log file change the lines to read lax stderr redirect c myfolder client log lax stdout redirect c myfolder client log After modifying the lax file enter one of the following commands to launch the CTM client bin ctm small debug exe bin ctm medium debug exe Cisco Transport Manager Release 6 0 User Guide 20 78 16845 01 AppendixJ Troubleshooting J 6 CTMGateWay TL1 Problems W bin ctm large debug exe bin ctm highend debug exe On Solaris you can start the CTM client with a console window that displays exceptions and client debug messages You can use these messages to help troubleshoot any
96. cmgrd log directory e The output of the query select from node_info where node_id X J 10 8 4 11 Cmgrd aAddition Modification and Deletion Errors Provisioning a Connection Results in Agent not responding request timed out or Fatal error Snmpcomm is not running Error Step 1 Step 2 Related key index entries cmgrd snmpcomm When any add modify or delete connection request is made cmgrd issues a request to the switch but the request is actually first issued to the process snmpcomm which in turn sends the SNMP SET GET to the node In this scenario these errors can be returned by the process snmpcomm or the error could be a time out error from the snmpcomm process or an ILOG communication error The reason that snmpcomm can send a timeout error is because it is too busy processing requests and cannot accept more requests at the moment The reason for the SnmpComm not Running error is that cmgrd could not establish an ILOG connection with it Issue the command psg snmpcomm to ensure that the process is indeed running Check for snmpcomm coredumps in the corefilesdir directory Reissuing the request after waiting for few minutes will take care of the timeout error If you confirm that the snmpcomm process is not running then you must do a warmstart to ensure that the process comes up When the process starts the ILOG connection with cmgrd is established Defect Information e If you get the same timeout error aft
97. co Transport Manager Release 6 0 User Guide 78 16845 01 ECN AppendixJ Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems Possible alternative workaround Open a new GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report GUI J 10 5 2 2 Configuration Center Chassis View Diagnostics Center or Statistics Report Is Unable to Connect to Server Step 1 Step 2 Check whether the client registers in NMServer log Enable orbix logs by creating the orbix directory in opt svplus log directory Defect Information Collect the following information for further analysis e CMSCclient log NMServer log e Orbix logs Possible alternative workaround None J 10 5 2 3 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Newly Added Node Step 1 Step 2 Step 3 Check whether the nodes have been discovered See J 10 2 1 No Nodes Are Discovered page J 34 for more information Check whether NMServer has the nodes in its cache This can be done by using the command nmControl in the CTM CLI If the nodes are in the NMServer cache open a new GUI and verify whether the nodes are shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Possible alternative workaround Open a new GU
98. connections in question For example if the connection has a local endpoint of node n slot s logical port p vpi dlci s1 vci s2 then the query would look like this SELECT from user_connection where 1_node_id n 1_slot s 1_logical_port p 1_subchnl1_1 s1 1_subchnl_2 s2 If you are unsure which side is local and which is remote try a query with both _node_id n and r_node_id n and slot port and so on If this row in the database is inconsistent with the same information on the switch more investigation is required Verify that the databroker has received all traps from the EMs To do this view the columns inseg_tbl_1 inseg_tbl_2 and inseg_tbl_3 If the flag is set to 1 then all traps for the given segment have been received A flag tells you if the databroker received an add message for the given segment If it is set to 0 it means the segment trap has not been processed by the databroker If the flag is set to 2 or 3 that means only one end of the segment has been processed amp Note If the connection does not have three segments the unused segments default to 1 You should now check the EM connection tables for the connections in question If you see that the segment tables are correct and the user_connection table is not or if the inseg_tbl_x flags are not all 1 you should continue testing see J 10 13 2 2 Inconsistent Connection Status page J 132 If you see that the segment tables
99. ct Determine whether the source and the target application belong to the same instance of the Java VM CTM framework does not support drag and drop functionality across applications belonging to different Java VMs Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 1 7 Drop Target Displays Incorrect Object or Object Data The Diagnostics Center s internal frame opens successfully but either it displays either information related to another object or the dropped object attribute values are not correct Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 1 8 Diagnostics Center Does Not Respond GUI Is Grayed Out Step 1 Determine whether a Java DOS window has already been enabled a Launch the WebStart application b Click on File gt Preferences c Select the Java tab 78 16845 01 Cisco Transport Manager R
100. cted verify which process contains the incorrect information If the information is correct in all processes open a new instance of the GUI and verify whether the problem still exists Step2 If the node table contains the incorrect name check the ILMITopoc log file to see from which node it collected the wrong information Complete the following substeps a Enter the following command snmpget c lt community gt lt IP_address gt 1 3 6 1 2 1 1 5 b Verify that the name received as a response is correct If the name received is incorrect verify the name on the switch Defect information Collect the following information for further analysis e ILMITopoc log and topod log e Dump outputs of ILMITopoc and topod Enter the following command kill USR1 signal lt process gt e Output of the switch CLI selnd and dbnds Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN Appendix J Troubleshooting HI J 10 2 Discovery Issues at Startup J 10 2 3 Node IP Is Incorrect in the Database Step 1 Step 2 If the node table contains the correct IP address complete the following substeps a Enter the following command to dump the cache of topod and ILMITopoc kill USR1 lt process gt b From the data collected verify which process has the incorrect information If the information is correct in all processes open a new instance of the GUI and verify whether the problem still exists If the node table do
101. d Object Repository Alarm Repository Interface Layer Event Handler Network Topology Fault Equipment Manager and eae Management Management Sync up g y Multiservice WAN Network 120752 NMServer provides inventory and alarm information for the Configuration Center Chassis View Diagnostics Center or Statistics Report The node information is pushed by topod to NMServer The initial inventory of card lines ports and so on is read from the database The dynamic updates are pushed to NMServer from topod ooemc and sdbroker There are two utilities under the opt svplus util directory that are shipped with CTM to debug issues in NMServer e nmControl This utility provides the means to check the NMServer cache and its state Output is redirected to opt svplus log nmControl log Cache dumps are redirected to opt svplus log nmControl dump The utility allows the following operations all of which are nondestructive and perform the tasks in a passive mode amp Note Do not use the All Cache operation to dump the cache for a large network that has more than 1000 nodes Resync Node Resyncs the node specified by the node_id All Cache Dumps all the data in its cache to the dump file Topology Cache Dumps topology node data to the dump file Node Cache Dumps a node s data specified by node_id to the dump file CTM Alarm Cache Dumps the CTM specific al
102. d is not successful fix the physical connectivity then log into the CTM client J 3 4 Cannot Log In as Provisioner or Operator Step 1 Step 2 Step 3 Use the default username and password to log into the CTM client Username SysAdmin Password Ctm123 If the SysAdmin login is successful but the provisioner or operator login fails verify that the provisioner or operator exists and is not disabled In the Domain Explorer window choose Administration gt CTM Users to view a table of all configured CTM users If the provisioner or operator is not in the CTM Users table the user is not configured Configure the provisioner or operator then log in as that user Cisco Transport Manager Release 6 0 User Guide M2 E 78 16845 01 Appendix J Troubleshooting Step 4 Step 5 J 3 5 Cannot Authenticate User Message Appears W If the provisioner or operator is in the CTM Users table select the row corresponding to that user and click the Modify User Properties tool to bring up the Modify CTM User Properties wizard Verify that the Login State field is set to Enabled If the login state is disabled enable it and log in as the provisioner or operator The user might have been disabled after attempting to log in with an incorrect password If the password is not correct set a new password for the user and log in again J 3 5 Cannot Authenticate User Message Appears If the Cannot authenticate user error message is re
103. d to the Configuration Center are presented J 10 8 2 1 Connection Creation Modification Deletion and Retrieval Errors When performing any connection creation modification deletion and retrieval the CM Server could encounter an error and notify the Configuration Center by throwing an application exception error that contains information describing the error As a result the Configuration Center displays the error message The following lists some CM Server related exceptions DATABASE_ERROR Database error s UNKNOWN_EXCEPTION Program Error Unknown Exception is caught in s SUBSYSTEM_EXCEPTION Program Error Subsystem exception s caught in s INTERNAL _ERROR Program or System Error Function s reports s CONNECTION_LOST Connection from host s application s lost during request UNKNOWN_PARAM VALUE_TYPE Unknown parameter value type d is processed in s PARAM _TYPE_VALUE_MISMATCH Parameter value s is does not match type d VALUE_OUT_OF_RANGE Value d is outside of the valid d d range NO_SUCH_CONNECTION_TYPE_ENUM No such connection type enum d NO_SUCH_SERVICE_TYPE_ENUM No such service type enum d NO_SUCH_ENDPOINT_TYPE_ENUM No such endpoint type enum d NO_PORT_TABLE_INFO_FOR_CARD No por
104. dding a Connection Results in CTM sync up in progress Error page J 98 J 10 8 4 6 Cmgrd Sdbroker Addition Errors Adding a Connection Results in No more vpi vci available for local trunk end Error page J 98 J 10 8 4 7 Cmgrd Sdbroker Addition Errors Adding a Connection Results in EndPoint Exists Local Remote Both end of the connection already exists Error page J 99 J 10 8 4 8 Cmgrd Sdbroker Modification Deletion Errors Modifying or Deleting a Connection Results in sDatabroker Could not lock connection entry Error page J 100 J 10 8 4 9 Cmgrd Errors page J 100 J 10 8 4 10 Cmgrd Addition Modification and Deletion Errors Provisioning a Connection Results in Fail due to Switch Timeout Error page J 100 J 10 8 4 11 Cmgrd Addition Modification and Deletion Errors Provisioning a Connection Results in Agent not responding request timed out or Fatal error Sampcomm is not running Error page J 101 J 10 8 4 12 Cmgrd Addition Modification and Deletion Errors Provisioning a Connection Results in Outstanding request exists for same object Error page J 101 J 10 8 4 13 Cmgrd Addition Modification Deletion Errors Provisioning a Connection Results in Fatal error IOSMGR is not running may need to be manually started Error page J 102 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HZ J 10 8 Connection Management Problems e J 10 8 4 14 Cmgrd
105. des the physical view of WAN devices For a specific node it displays node cards and lines Chassis View does not display ports It can handle the following events e Node status changes e Card status changes e Line status changes The alarms handled by Chassis View are e Card alarm e Line alarm Chassis View shows an empty slot in the chassis when displaying unsupported cards For a reserved card it shows the lines but they are disabled Chassis View does not show lines under a card when the card is in another state other than active or standby 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting WE J 10 6 Chassis View Problems Chassis View displays the chassis by combining static data from the XML file and dynamic data from the database The first step in debugging is to determine if data is populated properly in the database The next step is to check that the card or line is defined in the XML file J 10 6 2 Lines Not Displayed in the Chassis View Step 1 Determine if data about the lines are populated for the corresponding lines in the line table in the database Step2 Check whether the lines are defined in the XML file Chassis View xml Step3 Make sure that the line numbers in the database start from 1 and not from 2 or above Step 4 Make sure that the node is in sync mode is 3 If the details are not found in the database this probably is an EM issue If the details are
106. directory Possible alternative workaround is to manually resync the node If the problem persists perform a coldstart If the problem is still not resolved collect the log files and report the problem J 10 4 8 CTM GUI Shows Mismatched Information Between GUI Views and Database Data Step 1 Step 2 Step 3 After an initial ooemc node resync such as a coldstart process GUI views such as Network Monitor tree view Inspector View and Chassis View display information that does not match newly provisioned or updated database data The problem persists even after each subsequent manual or periodic resync Determine which ooemc process manages the node and find the child ID of the ooemc process that manages your node The purpose of finding the child ID of the ooemc process is to find the correct ooemc log files for inspection The debugging process for this issue mainly focuses on log file inspection Initiate an initial node resync such as coldstart Ooemc starts sending newly provisioned or updated database data to the network management NM server The current supported database tables for NM server message forwarding include the following node card line ausm_port cesm_port frp rpm_port svc_port virtual_port aps ima_group ima_link linedistribution au4tug3 controller license_in_use mfr_bundle mfr_link peripheral redundantcard sensor To determine whether or not ooemc sent newly provisioned or updated database
107. dow Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 2 Configuration Center GUI CM Server Reported Errors For connection management the Configuration Center communicates with the Connection Management CM Server process to a create modify delete or retrieve connections and b create modify delete or retrieve a connection template The CM Server throws application exceptions if it cannot successfully complete the requested operation The CM Server exception contains an error message that describes the error condition The Configuration Center displays the reported error message for further analysis In Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEE AppendixJ Troubleshooting HZ J 10 8 Connection Management Problems such cases the problem must be researched on the CM Server side The CM Server troubleshooting sections describe the steps required to further analyze the CM Server related issues See J 10 8 Connection Management Problems page J 83 In this section the required validation and data collection procedures relate
108. e For the Connection tab the Node Card Line Port objects are supported and Folder IMA and IMA link objects are not supported Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Screen snapshots in particular error information messages dialog boxes Cisco Transport Manager Release 6 0 User Guide cea 78 16845 01 Appendix J Troubleshooting J 10 9 Diagnostics Center Problems W e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 1 6 Drag and Drop from to Diagnostics Center to from Other Applications Step 1 Step 2 Step 3 As a drag source the drag and drop functionality of an object from the Diagnostics Center to another CTM application fails to display the selected object in the target application As a drop target the drag and drop functionality from another application to the Diagnostics Center s content pane a fails to open an internal frame or recycle the contents of an existing frame to display the dropped object s attributes or b results in the Operation Not Supported message dialog box Determine whether the drag and drop functionality is supported for the selected object Determine whether the target application supports the drag and drop functionality for the dropped obje
109. e J 10 13 2 1 Connection Inconsistency Between the Switch and GUL page J 129 Related key index entries inconsistency connection J 10 13 2 3 Inconsistent Connection Secondary Status The GUI display of Secondary Status is from the Secondary_status field of the user_connection table The field represents the worst condition of any of the segments in the connection This field is a bit map with each secondary status using 2 bits The values of each status entry are e 0 unknown e l o0 k _ Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting Step 1 J 10 13 Miscellaneous Problems W e 2 fail The bit pattern is e Bits 1 2 Local abit e Bit 2 4 Local AIS e Bit 5 6 Local OAM e Bit 7 8 Local Conditioned e Bit 9 10O Remote abit e Bit 11 12 Remote AIS e Bit 13 14 Remote OAM e Bit 15 16 Remote Conditioned If the value is not what is expected check the connection level databases for each segment to see if they are correct See J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 The last message on the segment in question is the important one Defect Information See J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 Possible alternative workaround See J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 Related key index entries inconsistency connection J 10 13 2 4 Inco
110. e of transfer with the number of bytes transferred and the time 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting W J 10 13 Miscellaneous Problems e TRANSFER RETRY Specifies that the transfer request needs to be retried e TRANSFER _FAILED Specifies that cwmftpd was unable to transfer the file e CONTROL_CONN_TIMEOUT This exception will be thrown where there is no transfer of information for X seconds once the FTP session is opened Related key index entries FTP log Cisco Transport Manager Release 6 0 User Guide 3 138 78 16845 01
111. e synced up with it but the NMS is interested in this information Therefore if the node is still syncing up with the NMS an alarm will be displayed for this node in the alarm list but there will not be such an alarm on the platform The following is a summary of alarms that may be seen in the alarm list but not on the platform If there is an alarm in the alarm list and not on the platform see if the suspect alarm is included in the following list e Node sync up status alarms e Node database sync up status alarms e Node Management State status alarms e Node Aggregate alarm status e Link0O Link1 Node alarms e Card sync up status alarms e Aggregate Port Connection alarms If the suspect alarm is not included in any of the alarm list mentioned in step 1 it may be a defect Perform the following a Verify whether the database has the correct alarm state See the Cisco Transport Manager Release 6 0 Database Schema for information on what table to look up for this particular entity type b Collect the following information topod log linktopoc log ILMITopoc log NMServer log and fileTopoc log nmControl dump CMSCclient log 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems Possible alternative workaround Open a new GUI and a new Configuration Center Chassis View Di
112. e where node_name lt node name gt dbaccess e Related key index entries e pmcollector Possible alternative workaround TIf the node is not mode 3 look up troubleshooting for the EM module J 10 10 1 3 PM Collection Fails for the Node Log File Shows Card not discovered This can happen when the card is not in the Active state Step 1 Check the node_info table by executing the following command echo select card_not_discovered from node_info where node_id lt node_id gt and slot lt slot gt dbaccess If card_not_discovered is set to 1 it means that the card has not been discovered Step2 Check the card table by executing the following command echo Select fc_state fc_type slot from card where node_id lt node id gt and slot lt slot num gt dbaccess The fc_state of the card must be 3 Defect Information Collect the following information for further analysis e Pmcollector log output of Cisco Transport Manager Release 6 0 User Guide cea 78 16845 01 Appendix J Troubleshooting J 10 10 Performance Management Collection and Parsing Problems Mi echo select fc_state fc_type slot from card where node_id lt node id gt and slot lt slot num gt dbaccess e Related key index entries e Pmcollector Possible alternative workaround TIf the card is not in fc_state 3 look up troubleshooting for the EM module J 10 10 1 4 PM Collection Fails for the Node Log File Shows Time
113. each NE should contain entries similar to the following example snmp server community public RO snmp server community private RW The community strings configured on the server should match the strings configured on the NE For information on configuring community strings in CTM for ONS 15501 ONS 15530 and ONS 15540 NEs see 3 5 1 4 Prerequisites for Adding ONS 15501 ONS 15530 and ONS 15540 NEs page 3 10 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 25 Launching Tables Results in Database Errors J 2 5 Launching Tables Results in Database Errors Step 1 Step 2 Step 3 Step 4 If the Oracle database or Oracle listener that CTM is using is down launching tables will generate database errors To troubleshoot table launching errors Log in as the Oracle user and enter the following command sqlplus ctmanager ctm123 If the login is successful an SQL gt prompt appears which indicates that the Oracle database has been installed and the database server is up and running If login fails either the Oracle database has not been installed or the database server is not running To start the Oracle database log into the Solaris workstation as the Oracle software owner user Enter the following command at the shell prompt to start the Oracle database dbstart Enter the following command at the shell prompt to start the Oracle listener lsnrctl start If there are
114. ection Port Speed Should Be the Same on Both Endpoints Step 1 Step 2 While adding a CE CE connection this error message might be thrown For Ce Ce Connection Port Speed should be same on both sides Port Type should be same except for Structured 8T1 and E1 Make sure that the port speed is the same for both the local and remote endpoints Make sure that the port types are the same for both the end types except for structured 8T1 and 8E1 Defect Information Collect the cmsvr logs in the directory opt svplus log Possible alternative workaround Change the endpoints accordingly to add the connection J 10 8 3 4 Cannot Create One Segment Hybrid Step 1 While adding a one segment connection a check is made to make sure that it is not a hybrid connection If it is the following error message appears Cannot create one segment hybrid connection Make sure the end to end type is not Hybrid for a single segment connection Defect Information Collect the cmsvr logs in the directory opt svplus log Possible alternative workaround Change the end to end type to something other than Hybrid J 10 8 3 5 Cannot Add Connection Between byte Port Header and byte Port Header for chan type NIW and NIW replace During a connection addition between two FRSM_12 ports the cmsvr checks to see if the port headers are the same 4 byte or 2 byte for both the endpoints An error is thrown if they are not Cannot add Connection betw
115. ed by ooemc should be logged with one pair of messages one for local end and one for remote end This pair of messages corresponds to one atm_connection segment entry in the database Verify the information that ooemc sends to dbroker Dbroker will refer to these messages from ooemce and construct a user_connection database entry CTM Connection Management CM GUI displays connection information based on the user_connection database entry Whether or not the connection is displayed as complete or incomplete depends on the information in the user_connection database entry It is important to verify that ooemc sent the correct information to dbroker Also verify that the segment database entries are populated correctly Step3 If the correct number of messages have been forwarded to dbroker and the data in the messages is correct but the user_connection database entry still contains invalid data see J 10 13 2 Data Inconsistency page J 129 for more debugging information or contact the dbroker development engineer DE for further investigation If the problem is due to wrong data populated in segment database entries then the EM DE should look at the problem Defect information e Collect ooeme and dbroker log files from the opt svplus log directory Cisco Transport Manager Release 6 0 User Guide P46 78 16845 01 Appendix J Troubleshooting J 10 4 Equipment Management Problems W Collect config upload files from the opt svplus tmp
116. eded for the node See J 10 13 1 NTS page J 127 Step4 Save ILMITopoc log topod log and ooemc log for analysis J 10 3 Discovery Issues at Runtime This section includes the following information e J 10 3 1 Node Is Not Shown in the Configuration Center Chassis View Diagnostics Center or Statistics Report page J 38 e J 10 3 2 Node Name Change Is Not Updated in the Configuration Center Chassis View Diagnostics Center or Statistics Report page J 39 e J 10 3 3 Node IP Address Change Is Not Updated in the Configuration Center Chassis View Diagnostics Center or Statistics Report page J 39 J 10 3 1 Node Is Not Shown in the Configuration Center Chassis View Diagnostics Center or Statistics Report If a dynamically added routing node is not shown in the Configuration Center Chassis View Diagnostics Center or Statistics Report complete the following steps Step 1 If the node table has an entry for the node complete the following substeps a Enter the following command to dump the cache of NMServer topod and ILMITopoc kill USR1 signal lt process gt b Check the data collected in the usr user svplus log directory and verify whether the node is present in the cache dumps If the node is present open a new instance of the GUI and check whether the node is visible in that GUI Step2 If the node table does not have an entry for the node complete the following substeps a Check the ILMItopoc log and verify whethe
117. een 4 byte port header to 2 byte port header for chan type NIW or NIW replace 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems Step 1 Determine what port header was used while creating the port in question Then use ports with matching port headers to create the connection Defect Information Collect the cmsvr logs from the opt svplus log directory Possible alternative workaround None J 10 8 3 6 Unsupported Service Type General node slot port version During a connection addition the cmsvr checks the node slot and port versions to ensure that the service type is supported on that version of the node card and port Step 1 Make sure that the version of the node slot or port supports that service type from CTM Step2 Check the ConnGroupService mib as an additional check for the versions and the supported parameters Defect Information Collect the cmsvr logs from the opt svplus log directory Possible alternative workaround None J 10 8 3 7 Endpoint ls Not Present in One of the Connection Tables During a Modify Operation When a connection modification is attempted an error message is thrown Connection not found in database Step 1 Verify that a row entry exists for the connection in the user_connection Perform a query gt echo select from user_connection where 1_node_id A and 1_slot B and l_
118. elay testcon diagnostics to be executed otherwise the test will be blocked from cmsvr The testconn testdelay diagnostics cannot be done on RP to RPM connections The testdelay diagnostic is not applicable for voice and data connections 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems Note For any connection if the local end of the connection does not support that particular diagnostic cmsvr will verify if the diagnostic can be executed from the remote end If the diagnostic is not supported by both endpoints of a connection an appropriate error message will be given J 10 8 3 10 Connection Up Down Reroute Failed Connection up down reroute diagnostic failed with a switch error Step 1 Query the connection information from the user_connection table for that connection Step2 Query the connection information from the switch CLI and verify the connection status dspcon or dspchan Step3 Execute the up down reroute diagnostic from the switch CLI and verify that the results match what was reported in CTM upcon dncon rrtcon Defect Information Do the following e Capture selnd output e Copy all the cmsvr logs from the opt svplus logs directory e Capture the user_connection table query output from the CTM database for the given endpoint if the request is not an addition request e Switch CLI output fo
119. elease 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 9 Diagnostics Center Problems For the selected Java entry check the Command column entry If needed expand the column to display all information to determine if it is set to javaw exe or java exe If the Command column setting is set to java exe then the Java DOS window is already enabled and a DOS window task should be running on the machine Otherwise the Java DOS window is not enabled Command column entry is set to javaw exe Step 2 If the Java DOS window is already enabled collect defect information Step3 Ifthe Java DOS window is not enabled then the current log information might not be sufficient to determine the root cause a b c Collect currently available log information Enable the Java DOS window by changing the javaw exe to java exe Verify that the WebStart Java DOS is enabled Use the PC client to launch a GUI application and verify that a DOS box has been launched Try to reproduce the problem in order to collect additional Java related data using the WebStart Java DOS window Defect Information Collect the following information for further analysis Java DOS window data Select the Java DOS box Issue a Ctrl and Break command Hold the Ctrl key down and click on the Break Pause key This action should result in the DOS window showing Java thread related information Copy the data to a log file for further analysis
120. emory restore or download failure See Appendix H Error Messages for the correct action to take in response to the error J 2 18 Memory Autobackup Software Commit or Software Revert Fails Step 1 Step 2 Step 3 In the Domain Explorer window choose Administration gt Audit Log The Audit Log table shows the status of the operation Return to the Domain Explorer window and choose Administration gt Error Log The Error Log table shows the reason for the failure See H 2 CTM Server Error Messages page H 94 for the correct action to take in response to the error J 2 19 The getinfo sh Script Fails You might experience a problem where the getinfo sh script fails with the following error Error permission denied while running rsh Check the environment and make sure the root is permitted to run rsh on host lt JP_address gt This problem occurs because the getinfo sh script does not run unless an entry is made in the rhosts file This problem occurs even on a local server and database configuration This is a known issue that has been tracked as DDTS number CSCin66495 The solution is to add an entry for the server IP address to the rhosts file Cisco Transport Manager Release 6 0 User Guide Pio E 78 16845 01 Appendix J Troubleshooting J 2 20 Using the get nestate Command J 2 20 Using the get nestate Command W The get nestate command line interface CLI command available in the ctm
121. eout When any add modify or delete connection request is made cmgrd issues a request to the switch with an SNMP community string either a SET or GET string These community strings are overwritten by the process snmpcomm This process uses the values of the strings which are present in the node_info table Thus when a provisioning request times out from the switch one possible problem is that the node s community string in the node_info table does not match the string on the node itself The problem can be researched by checking the community strings on the node and the node_info table Check the community strings on the nodes by issuing the command dsnmp Check the community string in the node_info table based on node_id The strings are encrypted so use the decrypt binary which displays the string in the Clear format Once the strings are cross referenced and you have determined that a discrepancy is indeed present then you can change the strings in either of the following ways e Issue the command cnfsnmp on the node and change the community strings to match that of the CTM Cisco Transport Manager Release 6 0 User Guide Py 100 I 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W e Use the RunConfigurator process to change the string in the CTM to match that of the switch Defect Information If these methods do not work collect the following e Cmerd log located in the opt svplus
122. er Guide jg AppendixJ Troubleshooting HM J 10 13 Miscellaneous Problems J 10 13 3 CTM FTP Daemon This section includes the following information e J 10 13 3 1 FTP Daemon Overview page J 134 e J 10 13 3 2 Generic Troubleshooting page J 134 e J 10 13 3 3 FTP Username and Password page J 134 e J 10 13 3 4 cwmftpd Files Are Not Transferred Due to Wrong Username Password page J 135 e J 10 13 3 5 cwmftpd File Not Available On Switch page J 135 e J 10 13 3 6 FTP Sessions in Switch page J 135 e J 10 13 3 7 421 Session Limit Reached page J 136 e J 10 13 3 8 499 Session Limit Reached page J 136 e J 10 13 3 9 Failed to Acquire a Session with a Switch page J 137 e J 10 13 3 10 Failed to Acquire Session After All Retries page J 137 e J 10 13 3 11 General Log Information page J 137 J 10 13 3 1 FTP Daemon Overview The cwmftpd process is an ILOG server that allows CTM modules to request an FTP put get or directory listing service to and from network nodes or other CTMs This server acts as an FTP client from an FTP communication standpoint and performs services similar to the interactive ftp program The following processes use cwmftpd to FTP the files e ooemc e PM collector Related key index entries FTP J 10 13 3 2 Generic Troubleshooting When performing generic troubleshooting always do the following e Check whether the cwmftpd process is up and running psg cwmftpd e Check the free disk space df k opt svpl
123. er log file from opt svplus log directory e Screen snapshots in particular error information messages dialog boxes e CLI output of dspcd lt slot No gt of the card e List of missing counters for the particular node card Possible alternative workaround Use the CLI to get the real time counter values J 10 9 2 7 Unable to Start Stop or Modify BERT Operations Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Unable to start stop or modify BERT operations from the Diagnostics Center Check whether the node is reachable and the community strings are set correctly Check whether the card is in the Active state Check whether the correct VarBinds have been sent for snmpset Check whether the options set for start stop modify BERT are applicable Check whether the BERT operations are successful in the CLI with the options chosen from the Diagnostics Center The Diagnostics Center sends only the mandatory and modified values to the Diagnostics Center server for snmpset Use any SNMP tool with these values alone and perform a BERT operation If it succeeds then there is some problem with the Diagnostics Center 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 9 Diagnostics Center Problems Defect Information Collect the following information for further analysis e DCServer log file from opt svplus log directory e Screen snapshots in particular the Diagnostics
124. er log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 7 Element Tab Create Details Modify and Refresh Button Issues When the Element tab is selected and an object is displayed in an internal frame the Create Details Modify and Refresh buttons either do not launch other internal frames for further provisioning or they do not update the user interface with data for the selected operation Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 8 Element Management Drag and Drop Within Configuration Center The drag and drop functionality of a network element from the Configuration Center tree view to the Element tab s content pane does one of the following e Fails to open an internal frame e Fails to recycle the contents of an existing frame to display the object s attributes Cisco Transport Manager Release 6 0 User Guide 78 16845 01 a 75 Appendix J Troubleshooting W J 10 7 Configuration Center Management e Results in the Operation Not Supported message box Step 1 Determine if the drag and drop funct
125. er reissuing the request then collect the cmgrd log and snmpcomm log e Ifthe snmpcomm process does not run after a warmstart then collect the watchdog log file Possible alternative workaround Reissue the request after waiting a while This interval could enable snmpcomm to clear its pending requests Do a warmstart to ensure that the snmpcomm process starts up J 10 8 4 12 Cmgrd aAddition Modification and Deletion Errors Provisioning a Connection Results in Outstanding request exists for same object Error Related key index entries cmgrd outstanding request When any add modify or delete connection request is made cmgrd processes these requests in a multithreaded manner Sometimes a request can take longer than 3 minutes to process because of the different scenarios that the switch responds with for example timeouts If the request takes longer than 3 minutes from the CM GUI then the GUI will timeout and indicate this to the user From the Connection Proxy path the timeout value is 2 minutes after which it will indicate a timeout error If you reissue this request immediately cmgrd could return the error Outstanding requests exists for the same object 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems The reason this error is returned is because cmgrd is still processing the initial request which the clients CM GUI and
126. ernative workaround None J 10 7 2 6 Object Not Found In Tree View Error This error occurs when a network element is launched from the tabular view of the Configuration Center Step 1 Collect the information of the object being launched This can be done in two ways e In the tabular view there will be some minimal information such as node name slot number line number trunk number port number and so on e Some minimal information regarding the object being launched will be displayed in the title of the window in one of the following formats For anode the window title will have lt NodeName gt Fora slot the window title will have lt NodeName gt lt SlotNumber gt Fora line the window title will have lt NodeName gt lt SlotNumber gt lt LineNumber gt Fora port the window title will have lt NodeName gt lt SlotNumber gt lt PortNumber gt Step2 With the information obtained from Step 1 see if that object appears in the tree view e If it does appear go to step 3 e If it doesn t appear refresh the tabular view two or three times and see if the object still exists in the tabular view and if it continues giving the same error contact CTM engineers with the appropriate defect information Step3 Ifthe object being launched is a node or card verify whether that particular node or card is supported in the CTM version being used Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN Appendix J
127. error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 3 Connection List Tab Cannot Launch Internal Frame to Modify an Existing Connection The Details button fails to launch an internal frame to display the connection details for a retrieved connection Step 1 Check that the connection status is not in the Incomplete state Only connection in the OK and Fail states can be launched using the Details button on the Connection tab Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information in particular any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 4 Advanced Mode Tab Cannot Launch the Connection Details Dialog Box Using the Details Button The Details button on the Advanced Mode tab fails to launch the Connection Details dialog box Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information in particula
128. es not have the IP address set correctly verify whether the node is managed by the LAN IP address In CTM since the node does not have any PNNI trunks and the persistent topology feature is not enabled the node is managed by the LAN IP address by default Defect information Collect the following information for further analysis e ILMITopoc log and topod log e Dump outputs of ILMITopoc and topod Enter the following command kill USR1 signal lt process gt e Collect the output of the switch CLI selnd and dbnds J 10 2 4 Node Alarm Is Shown Incorrectly in the Database Step 1 Step 2 Step 3 Enter the following command to dump the cache of topod and ILMITopoc kill USR1 lt process gt From the data collected verify which process contains the incorrect information If the node table does not have the node alarm status set correctly check the ILMITopoc log file and enter the following command snmpget c lt community string gt lt IP_address gt 1 3 6 1 4 1 351 110 1 1 14 The snmpget command returns the alarm state of the node where Clear 2 Minor 3 Major and 4 Critical Defect information Collect the following information for further analysis e ILMITopoc log and topod log e Dump outputs of ILMITopoc and topod Enter the following command kill USR1 signal lt process gt e Collect the output of the switch CLI selnd and dbnds Cisco Transport Manager Release 6 0 User Guide P36 78 16845
129. ession to the switch Defect Information If manually opening the FTP session succeeds but cwmftpd still throws a SESSION_FAIL exception then report the problem with cwmftpd log cwmftpd request_log and logs of the process for which files are not getting transferred by cwmftpd Possible alternative workaround None Related key index entries session_timeout session_fail J 10 13 3 11 General Log Information The cwmftpd request_log file will have one entry for each request It will have information about whether the request succeeded or failed and how much time it took It has information in a short and simple format and since there is not much data per request the file has more data in terms of amount of time Cwmftpd maintains information based on the IP address of the node To track a request for a particular node grep for the IP address as the keyword in cwmftp log and cwmftpd request_log When ooemce sends FTP requests it also sends the node ID along with the request as part of the destination file This node ID is used to retrieve the FTP user password from the node_info table This can be used for user password validation The following keywords are from the cwmftpd log file e TRANSFER STARTED Specifies that cwmftpd started transferring the file e TRANSFER COMPLETED Specifies that the file has been transferred completely and also gives information about local and remote file size e TRANSFER _RATE Specifies the rat
130. etailed information on the identity of the NE object and the required fields from the corresponding database table The following is an example of NMServer message for line tables ooemc10 24370 log old 5 24370 4 23 15 17 INFO N0O C7 B2 L2 lt EMC_DBProperty_Line_c ComposeNMSMsg gt MODIFY LINE aNMSEvent node 0 type 4 subType 1 lpbkType 1 alarmState 1 adminState 1 sectStatus 6 pathState 2 In this example the identity of the NE object is NO C7 B2 L2 which is the line with node ID 0 slot 7 bay 2 and line 2 The database operation is update The rest of the message shows the values of the required fields from the line table If grep returns log messages that indicate matched data then you need to continue investigation on the NM server See J 10 5 1 CTM Configuration Center Chassis View Diagnostics Center or Statistics Report Basics page J 53 for information on nmController Defect information Collect ooemc and NMServer log files from the opt svplus log directory Possible alternative workaround Manually resync the node If the problem persists perform a coldstart If the problem is still not resolved collect the log files and report the problem Cisco Transport Manager Release 6 0 User Guide yas E 78 16845 01 AppendixJ Troubleshooting J 10 4 Equipment Management Problems W J 10 4 9 CTM Database and Switch Data Inconsistency Issues After Node Provisioning for MGX This section includes the followin
131. fig int GigabitEthernet0 2 0 0 Router 1 config if ipv4 address lt IP_address gt lt mask gt Router 1 config if no shutdown config if commit Enter the following commands on the second router Router 2 config int GigabitEthernet0 1 0 0 Router 2 config if ipv4 address lt IP_address gt lt mask gt Router 2 config if no shutdown Router 2 config if commit e If the link is a POS link enter the following commands on the first router Router 1 config int POS 0 2 0 0 Router 1 config if ipv4 address lt IP_address gt lt mask gt Router 1 config if no shutdown Router 1 config if commit Enter the following commands on the second router Router 2 config int POS 0 1 0 0 Router 2 config if ipv4 address lt IP_address gt lt mask gt Router 2 config if no shutdown Router 2 config if commit Verify that the SONET controllers are up For example enter the following commands Router 1 config controller SONET 0 2 0 0 Router 1 config sonet no shutdown Router 1 config sonet commit Router 2 config controller SONET 0 1 0 0 Router 2 config sonet no shutdown Router 2 config sonet commit Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EZE AppendixJ Troubleshooting Hs J 8 11 Cannot Delete an Out of Service NE from CTM Step3 Enter the following commands to enable CDP on both NEs Router 1 config cdp Router 1 config commit Router
132. g e The rate of network growth for example adding 50 NEs to CTM at once e The rate of configuration change updates received from the NEs e The number of circuits provisioned with no signals that generate threshold crossing alerts TCAs e The number of unacknowledged alarms in the CTM database e The rate of alarm bursts received from the NEs e The actual number of circuits provisioned in the CTM database The following conditions might indicate that your CTM installation is near or at capacity e Your daily CPU utilization is consistently above 80 e You experience loss of connectivity LOC to 1 to 5 of nodes at random e Your RAM usage is 100 or your system swaps frequently e You receive many configuration updates from the NEs e Your system experiences high Java garbage collection time amp Note The Java garbage collection time is determined by engineering after analyzing your CPU utilization data Cisco Transport Manager Release 6 0 User Guide m2 E 78 16845 01 Appendix J Troubleshooting J 2 2 CTM Server Does Not Respond W If you are experiencing any of these conditions contact your account representative Your account representative can engage Cisco s Advanced Services team to audit your CTM server and recommend a higher capacity server and resources J 2 2 CTM Server Does Not Respond Step 1 Step 2 Step 3 Step 4 Step 1 Log in as the root user on the Solaris workstation where the CTM server is inst
133. g information e J 10 4 9 1 Database Table Population Through Traps and SNMP Upload page J 49 e J 10 4 9 2 Switch Data Does Not Match CTM Database Table After Node Provisioning page J 49 J 10 4 9 1 Database Table Population Through Traps and SNMP Upload Except for node and or card resync another mechanism that CTM ooemc uses to populate the database table entries is through trap processing followed by SNMP upload if necessary You can provision the switch through switch CLI CTM GUI or CTM service agents All three cases will involve trap processing and SNMP upload on ooemc Only connections can be provisioned by the CTM CM GUI Other provisioning such as line and port have to go to CTM service agents The MGX switch CLI can do all J 10 4 9 2 Switch Data Does Not Match CTM Database Table After Node Provisioning After switch provisioning through the switch CLI CTM GUI or CTM service agents the switch data does not match the CTM database Step 1 If the issue is caused by a trap for example if you provision a connection on a switch or through CTM GUI or service agent and CTM does not populate the database entry or the information in the database entry does not match the switch data it is possible that CTM did not receive the trap or that it received the trap but the trap is buffered in the ooemc trap queue and trap processing has been delayed On the other hand if the data in the CTM database is not correct it is also possible that
134. gistered clients by NMServer when alarms occur Object severity for any and all network elements is determined by the XML alarm rule Administrative states of network element states are not displayed in the alarm list only alarm states Alarm List displays mostly active alarms however some events are also displayed For details on events versus alarms see J 10 5 5 2 XML Schema for Alarm Rules page J 63 Some alarms result in the network element having a different alarm severity then the actual alarm For details on object severity versus alarm severity see J 10 5 5 3 Alarm Severity and Object Severity page J 65 Alarms for various network elements are stored in memory cache in the NMServer Object Tree GUI clients register for alarms from NMServer by providing the FDN of the network element for which they would like alarms All alarms for network elements under the FDN registered will be sent if the parent FDN is registered GUI clients may also update alarms for example by flagging an alarm as acknowledged or manually clearing an alarm These updates are done through the Alarm Repository component of NMServer Figure J 2 illustrates the client server architecture for alarm components in NMServer Cisco Transport Manager Release 6 0 User Guide Pe E 78 16845 01 Appendix J Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il Figure J 2 Client Server Architecture for
135. gs lt username gt log directory e Java Console information specifically any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 10 Cross Application Configuration Center s Element Tab as Drop Target The drag and drop functionality of a network element from another CTM application to the Configuration Center s Element tab content pane does one of the following e Fails to open an internal frame e Fails to recycle the contents of an existing frame to display the dropped object s attributes e Results in the Operation Not Supported message dialog box Cisco Transport Manager Release 6 0 User Guide P76 E 78 16845 01 Appendix J Troubleshooting Step 1 J 10 7 Configuration Center Management W Determine if the drag and drop functionality is supported for the dropped object The following objects can be dropped from the tree view to the content pane For the Element tab the Network Node Card Line Port IMA and IMA link objects are supported and Folder objects are not supported For the Connection tab the Node Card Line and Port objects are supported and Folder IMA and IMA link objects are not supported Defect Information Collect the following information for further analysis CMSCclient log file from t
136. guration Center communicates with the Connection Management CM Server process This section describes troubleshooting guidelines for the element management Configuration Center Elements Tab section The Connection Management Configuration Center Connections Tab section describes troubleshooting guidelines related to Connection Management This section includes the following information e J 10 7 1 Configuration Center Framework page J 72 e J 10 7 2 Configuration Center Element Management page J 78 J 10 7 1 Configuration Center Framework The Configuration Center uses the CTM framework and workflow mechanism to launch applications and drag and drop objects across application Cisco Transport Manager Release 6 0 User Guide 372 E 78 16845 01 Appendix J Troubleshooting J 10 7 Configuration Center Management W Network elements can be selected in the tree view and the Configuration Center Elements Tab can be launched to view or modify the selected object Network elements can be dragged and dropped from other application to the Configuration Center s Elements tab for modification This section includes the following information J 10 7 1 1 Cannot Launch the Configuration Center page J 73 J 10 7 1 2 Cannot Launch Other Applications from the Configuration Center page J 73 J 10 7 1 3 An Exception Is Raised when the Configuration Center Is Launched page J 74 J 10 7 1 4 An Exception Is Raised when the Configuration
137. he D Documents and Settings lt username gt log directory Java Console information specifically any Java raised exceptions Screen snapshots in particular error information messages dialog boxes Cmsvr log file from the opt svplus log directory Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 11 Element Tab Internal Frame Displays Incorrect Object or Object Data The Element tab successfully creates the internal frame but either it displays information related to another object or the object s attribute values are not valid Defect Information Collect the following information for further analysis CMSCclient log file from the D Documents and Settings lt username gt log directory Java Console information specifically any Java raised exceptions Screen snapshots in particular error information messages dialog boxes Cmsvr log file from the opt svplus log directory Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 12 Configuration Center s Element Tab Does Not Respond GUI Is Grayed Out The Configuration Center s Element tab does not respond and the GUI is grayed out Defect Information Collect the following information for further analysis CMSCclient log file from the D Documents and Settings lt username gt log directory Java Console information Screen snapshots in particular error info
138. he event Filter the alarm list on CTM alarms Check the alarm count and compare it with the MAX_ACTIVE_NMS_EVENTS value in NMserver conf If the alarm count is close to the MAX_ACTIVE_NMS_EVENTS then this explains why the transient event has been purged If the alarm count for NMS alarms has not reached the MAX_ACTIVE_NMS_EVENTS the missing event may have been manually cleared The NMServer log will indicate whether the event was manually cleared Defect Information Collect the following information for further analysis e CMSCclient log NMServer log and CMSCclient log e Capture NMServer dump use nmControl Possible alternative workaround Restart the Alarm List GUI J 10 5 5 7 Port in Tree View Displays Aggregate Alarm However No Children Exist Under Port Step 1 The tree view in CTM client GUIs displays network elements from the top level physical view down to the port Alarm severities are aggregated from children up to parents Since the port is at the lowest level of the tree the question that often arises is how a port can have an aggregate alarm if it has no children The answer to this question is that connections are virtual entities under ports in the tree view Virtually you will not see connections in the tree But connection alarms are aggregate up to the port in the tree view Compare the connection alarms with the aggregate port alarms a In the Configuration Center click the Connections tab b Find the p
139. i vci This is output every time a cache query is done grep DROP MSG validation error dmd lt dmd_id gt lt pid gt prints out dropped messages If the message made it to DMD you need to see if the sdbroker received the message grep node lt node_id gt slot lt slot gt port lt logical port gt sCh1 lt vpi dlci gt sCh2 lt vci gt sdbroker Msg If it looks like the message was received by the sdbroker but the database does not reflect the change then you should verify if there is an inconsistency between the databroker cache and the database To dump the cache enter psg sdbroker The process ID of the sdbroker will be returned kill USR1 lt process ID returned from the previous command gt 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 13 Miscellaneous Problems The cache dump will be written to a file in opt svplus log sdbkrCache dump Verify if the connection in question is correct in the cache Defect Information You need the logs of the processes on both sides of the interface on which the message was dropped A dump of the specific user_connection table entry that is incorrect is also useful as well as the segment tables for this connection The specific node slot logical_port vpi and vci of both ends of the connection in question is also useful If a cache dump was done include that also Possible alternative workarounds TIf the
140. ication Defect Information Collect the following information for further analysis CMSCclient log file under the D Documents and Settings lt username gt log directory Java Console information in particular any Java raised exceptions Screen snapshots in particular error information messages dialog boxes Cmsvr log file under the opt svplus log directory Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 7 Cross Application Connection List as Drag Source As drag source the drag and drop functionality of a connection from the Configuration Center s connection list to another application fails to display the selected connection in the target application Step 1 Determine if the target application supports connection object drop operations Step2 Determine if the target application supports the drag and drop functionality for connection objects Defect Information Collect the following information for further analysis CMSCclient log file under the D Documents and Settings lt username gt log directory Java Console information in particular any Java raised exceptions Screen snapshots in particular error information messages dialog boxes Cmsvr log file under the opt svplus log directory Cisco Transport Manager Release 6 0 User Guide ise W 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W e Configserver log
141. ify the start of bulk file creation When bulk file creation is complete on the switch the switch sends a trap 60902 to CTM After CTM receives the trap it begins to FTP the config upload files 4 Ifthe upload and parsing of all config files completes without errors the ooemc changes the node mode from 2 to 3 If the upload or parsing encounters errors for any service module on the switch the node synchronizes partially and the ooemc changes the node mode from 2 to 4 at the end of the node synchronization process If the error occurs on the shelf generic card file CARD_01_CC CF or pnni file PNNI_01_CC_CF the ooemc changes the node mode from 2 to 5 to signify node sync up failure Cisco Transport Manager Release 6 0 User Guide P40 78 16845 01 AppendixJ Troubleshooting J 10 4 Equipment Management Problems W J 10 4 2 Network Setup and Configuration Required for OOEMC Sync Up Process The EM requires network setup and CTM configuration prerequisites before it can start the sync up process On the switch perform the following e Use the dsptrapmgr CLI command to verify that the CTM IP address has been added to the trap manager If your CTM IP address is not registered with the switch add the CTM IP address to the trap manager by entering the addtrapmgr lt cwm_ip gt 2500 CLI command where 2500 is the port number and cwm_ip is the IP address of your CTM machine You might need to delete other IP addresses from the trap manage
142. in the Active state Check whether the correct VarBinds have been sent for snmpset Check for any specific exceptions in the DCServer log file If there are no exceptions in the DCServer log file then the problem could be originating in the Connection Manager Defect Information Collect the following information for further analysis e DCServer log and cmsvr log files from opt svplus log directory e Screen snapshots in particular error information messages dialog boxes Possible alternative workaround Use the CLI or SNMP conn proxy for connection diagnostics Cisco Transport Manager Release 6 0 User Guide cea 78 16845 01 AppendixJ Troubleshooting J 10 10 Performance Management Collection and Parsing Problems W J 10 9 2 10 Miscellaneous Issues or Problems in Diagnostics Center If you encounter any other issues or problems contact the CTM engineers with the defect information Step 1 Check whether a similar operation can be performed successfully through the CLI Defect Information Collect the following information for further analysis e Describe the operation during which the problem issue was encountered e DCServer log and cmsvr log files from opt svplus log directory e Screen snapshots e CLI dump of this operation if applicable Possible alternative workaround None J 10 10 Performance Management Collection and Parsing Problems PM collection is done on the entire node and statistics files are up
143. ing Convention There is one audit trail log file per day The filename convention is defined as AuditTrail lt mmddyyyy gt log Example audit trail log files AuditTrail 10102001 log for October 10 2001 J 10 12 2 3 When You Open a Dialog Box in a CTM GUI There Is No Record in the Audit Trail Log File Step 1 Open the CTM Security Manager GUI and check the Audit Read permission for that CTM GUI in the security profile associated with the user If the Audit Read permission is not turned on then no record is the expected behavior Otherwise go to step 2 Step2 Verify that the Audit Logger server is running Use psg AuditLogger on the CTM machine If no AuditLogger server is running then this is the problem Go to defect information and follow the steps to collect all the information needed Step3 Check to see if any CORBA related exceptions have been thrown on the console If so collect those error messages and or call stacks Defect Information Collect the following information for further analysis e Complete screen snapshots of the CTM GUI e Errors exceptions thrown on the console if any e log opt svplus log AuditLogger log opt svplus log LogServer log AuditTrail log file location is set in opt svplus config AuditLogger conf opt svplus log watchdog log e Core dump files in opt svplus corefilesdir especially for those AuditLogger core dump files J 10 13 Miscellaneous Problems This section includes the following
144. ing or Deleting a Connection Results in sDatabroker Could not lock connection entry Error Step 1 Related key index entries cmgrd sdbroker lock connection During a modify or delete connection request this error could result if sdbroker cannot find the connection in its cache All scenarios of modify delete connection failures display the same error To determine exactly what happened you must view the opt svplus log sdbrokerX XXXX_ log file Search for the error lt sDbkr_CmgrdModConn_c Process gt INTERFACE ERROR near the end of the file A detailed reason for the error will follow on the same line amp Note For delete connection requests search for sDbkr_CmgrdDelConn_c Process instead of sDbkr_CmgrdModConn_c Process Defect Information e Ifthe problem is an intermediate endpoint and it is on the switch but the EM didn t process it or didn t send the message to DMD check the EM logs and nts logs from opt svplus log e Ifthe problem is within the DMD you will need the DMD logs DMD message logs and the DMD cache dump Possible alternative workaround None J 10 8 4 9 Cmgrd Errors During an addition request cmgrd could return errors which cmgrd has received from snmpcomm or the switch itself J 10 8 4 10 Cmgrd Addition Modification and Deletion Errors Provisioning a Connection Results in Fail due to Switch Timeout Error Step 1 Step 2 Step 3 Related key index entries cmgrd tim
145. ionality is supported for the dropped object The following objects can be dropped from the tree view to the content pane e For the Element tab the Network Node Card Line Port IMA and IMA link objects are supported and Folder objects are not supported e For the Connection tab the Node Card Line and Port objects are supported and Folder IMA and IMA link objects are not supported Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information specifically any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 9 Cross Application Configuration Center as Drag Source As drag source the drag and drop functionality of an object from the Configuration Center s tree view or Element tab to another CTM application fails to display the selected object in the target application Step 1 Determine if the drag and drop functionality is supported for the selected object Step2 Determine if the target application supports the drag and drop functionality for the dropped object Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settin
146. is not found in the table Reissue the hybrid connection add request but tail the cmgrd log In this log a query in the form of SELECT MIN_SVCC_VPI MAX_SVCC_VPI MIN_SVCC_VCI MAX_SVCC_VCI FROM SVC_OPERATION WHE RE NODE_ID X AND SLOT X AND PORT 1 X will be seen This is the actual query that is failing Run the query from the command line or dbaccess stratacom prompt to figure out which port s information is not present in the svc_operation table The reason could be that for that port the resource partition is not configured on the CLI Go to the CLI and issue the command addpart for that port After issuing the command addpart on the CLI check the CTM table svc_operation database for the existence of the row If the row is present the connection request can be reissued If the row still is not present in the svc_operation then it is a probably an EM issue Defect Information If after configuring the resource partition on the CLI and the svc_operation table is still not populated then see J 10 4 Equipment Management Problems page J 40 and collect the appropriate ooemc logs Possible alternative workaround None J 10 8 4 15 Cmgrd aAddition Modification Deletion Errors Provisioning a Connection Results in Can t get segment info from Data base Error Related key index entries cmgrd segment info This error is returned by cmgrd when it cannot get a row from one of the CTM database tables for the current connection
147. istency by using a database query 3 Isolate the connection inconsistency by using a message flow Check the node s state after startup If the node is in an abnormal state it can result in incorrect counts or the presence of incomplete connections a Determine if sync up has been completed b If the system has declared sync up use the CTM CLI command selnd to verify that the nodes on which you see the inconsistency are in mode 3 The second to last column identifies the mode for a given node Check the user_connection table and the segment tables for problems The information displayed on the Connection Manager GUI is taken directly from the user_connection table The Connection Manager is rarely the problem This procedure uses database access queries to help isolate the problem If the problem is found in the user_connection table and the segment tables are incorrect the problem is usually in the EM If the user_connection table is wrong but the segment tables are correct you will need to continue with the investigation before you can determine a root cause 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 13 Miscellaneous Problems wy Note All ports in the database and the messages are 0 based and all ports on the switch CLI are 1 based Therefore if the switch shows port 3 then you should query for port 2 Query the user_connection table for one of the
148. ive workaround None Cisco Transport Manager Release 6 0 User Guide EzE 78 16845 01 Appendix J Troubleshooting J 10 7 Configuration Center Management W J 10 7 1 5 Element Tab Internal Frame Does Not Launch When the Element tab is selected and you double click on a supported network element an internal frame is not created or the content of an existing internal frame is not recycled Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 6 Element Tab Drag and Drop Functionality Does Not Launch An Internal Frame When the Element tab is selected and you drag and drop supported network elements to the Element tab s content pane an internal frame is not created or the content of an existing internal frame is not recycled Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserv
149. jects to be queried for a particular dialog box can be obtained from the svplus log configserver log file Refer to the entries in this log file when this error occurs and perform the following a Using HP OV SNMP operations perform a walk on system MIB table for this example using public community string as opt OV bin snmpwalk c public nodeName system b If an error is seen during the SNMP operations from the SNMP manager tool check the switch to verify if this information should really be present If yes proceed to step 2 Collect the information from the svplus log configserver log file when this error occurs Perform the following a From the configserver log file identify the MIB objects that caused the error b Identify the MIB to which these objects belong c Verify whether these MIB objects are supported in this release If the objects are supported but still the SNMP agent is not responding for SNMP queries contact CTM engineers Defect Information Collect the following information for further analysis e Complete screen snapshots for the investigative commands used for debugging this issue e Configserver log e Related information on the switch Possible alternative workaround None J 10 7 2 3 Details Create Delete and Refresh Buttons Are Not Enabled Step 1 Step 2 The Details Create Delete and Refresh buttons are enabled or disabled based on the requirement in the tabular view These buttons
150. larmSev HewAlarmObjSev F NewAlarmDescrSuffix 120749 Cisco Transport Manager Release 6 0 User Guide Po E 78 16845 01 Appendix J Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il A CorrelatedBmRule can have any of the following e 0 or more ClearAlarmCondIds e 1 new alarm which consists of NewAlarmConditionID NewAlarmServAffect and so on 3 Transient Alarm Rule Type The transient alarm rule type is used to distinguish events from alarms The alarm list shows some events Events are normally associated with the NMS itself Examples of events are Process Restarted or Primary Gateway Disconnected These are NMS events that are not correlated but are displayed in the alarm list Another example of a transient event that is not related to the NMS is a card switchover If automatic protection switching APS is enabled on a card and there is an APS switchover an event will be displayed in the alarm list The primary distinction between transient events and correlated alarms is that transient events are not cleared by the network whereas correlated alarms are See Figure J 5 Figure J 5 Transient Alarm Rule Diagram l nmserver TransientRuleListType nmserver TransientRuleType HewAlarmCondiD HewAlarmServAffect HewAlarmTransient HMSProcessRules GH j TransientRule HewAlarmSev 120748
151. lect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Possible alternative workaround Open a new GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report J 10 5 3 2 Node Information Incorrect on the Configuration Center Chassis View Diagnostics Center or Statistics Report Step 1 Check the node information in the database using selnd command Step2 Check the node information in the NMServer s cache This can be done by using the command nmControl on the CTM CLI Step3 Check the CMSCclient log for the node information Step4 From nmClient use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI Step5 Ifthe node information is correct in the NMServer cache open a new GUI and verify whether the node is shown correctly in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e selnd o p e nmControl dump e CMSCclient log Possible alternative workaround Open a new GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report J 10 5 3 3 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows a Node that Is Not in the Network Step 1 Check the node information in the database by querying the node packet_line table Verify whether the n
152. ll active Cisco Transport Manager Release 6 0 User Guide 23 By 78 16845 01 AppendixJ Troubleshooting J 8 16 How Do I Create and Grant Certificates for SSL Configured CRS 1NEs W Step2 Enter the following command to verify that DSA keys have been generated on the router Router crypto key generate dsa Step3 Enter the following command to verify that RSA keys have been generated on the router Router crypto key generate rsa Step4 Enter the following command to verify that SSH is enabled on the router Router crypto key generate rsa Step5 Enter the following command to enable the SSH server on the router Router config ssh server enable J 8 16 How Do Create and Grant Certificates for SSL Configured CRS 1 NEs See 5 4 25 Configuring Secure Socket Layer for the CRS 1 and XR 12000 page 5 188 J 8 17 How Do Trigger a linkDown or linkUp Trap from the Catalyst 6509 Complete one of the following options e To trigger a linkDown trap from the Catalyst 6509 enter the following command cat6509 enable set port disable lt module_number port_number gt For example cat6509 enable set port disable 5 1 e To trigger a linkUp trap from the Catalyst 6509 enter the following command cat6509 enable set port enable lt module_number port_number gt For example cat6509 enable set port enable 5 1 J 8 18 How Do View the SNMP Configuration on the Catalyst 6509 To view the
153. llhome messages Facility Default Severity Current Session Severity acl 5 5 callhome 2 2 cdp 4 4 cops 3 3 w__ Cisco Transport Manager Release 6 0 User Guide 78 16845 01 Appendix J Troubleshooting J 8 6 CTM Does Not Collect Historical PM Data forthe CRS 1 W sys 5 5 vmps 2 2 vtp 2 2 0 emergencies 1 alerts 2 critical 3 errors 4 warnings 5 notifications 6 information 7 debugging J 8 6 CTM Does Not Collect Historical PM Data for the CRS 1 Step 1 Step 2 Verify that PM collection is enabled for the CRS 1 To do this go to the Network Element Properties gt Status tab and in the PM Collection area check the 15 Min check box Click Save Also verify that the CRS 1 PM Service Status is Active in the Control Panel window Enter the following commands to verify that the correct PM configuration is in the running configuration on the CRS 1 Router config show run perf performance mgmt resources tftp server lt IP_address_of_server gt directory lt directory_on_TFTP_server gt performance mgmt statistics bgp template default sample size 1 sample interval 15 performance mgmt statistics mpls ldp template default sample size 1 sample interval 15 l performance mgmt statistics node cpu template default sample size 1 sample interval 15 performance mgmt statistics interface generic counters template default sample size 1 sample interval 15 performance mgmt statistics interf
154. loaded from all applicable cards on that node The Stats Parser parses these statistics files and stores the stats data in the database J 10 10 1 PM Collection Issues This section includes the following information e J 10 10 1 1 pmcollector Generic Troubleshooting page J 117 e J 10 10 1 2 PM Collection Fails for the Node Log File Shows Node not discovered page J 118 e J 10 10 1 3 PM Collection Fails for the Node Log File Shows Card not discovered page J 118 e J 10 10 1 4 PM Collection Fails for the Node Log File Shows Time not synchronized page J 119 e J 10 10 1 5 PM Collection Fails for the Node Log File Shows Snmp failed page J 119 e J 10 10 1 6 PM Collection Fails for the Node Log File Shows Ftp failed page J 120 e J 10 10 1 7 PM Collection Fails for the Node Log File Shows Polling failed and wait for major retry or Major retry failed and wait for critical retry or Critical retry failed and wait for history retry or History retry failed and wait for next retry or No more retry page J 120 J 10 10 1 1 pmcollector Generic Troubleshooting When performing generic troubleshooting always do the following e Verify that the pmcollector process is running by executing the psg pmcollector command and checking for a result e Make sure the stats files are in the opt svplus purge directory of CTM Check for all stats files there should not be any missing file and there should be no files acc
155. ls for the CRS 1 Step 1 Step 2 Step 3 Verify that the NE is in service and the communication state is Available Verify that you can ping the CTM server from the NE The NE is used as a TFTP client for downloading the configuration file In the etc inetd conf file verify that the TFTP server is enabled on the CTM server Also verify that the specified tftpboot directory has the correct permissions Cisco Transport Manager Release 6 0 User Guide P26 E 78 16845 01 Appendix J Troubleshooting J 8 9 How Do I Back Up the CRS 1 Configuration File Automatically Whenever the Configuration Is Changed onthe W J 8 9 How Do I Back Up the CRS 1 Configuration File Automatically Whenever the Configuration Is Changed on the Router Step 1 In the Domain Explorer choose Administration gt Control Panel Step2 Click NE Service Step3 In the NE AutoBackup tab choose Cisco CRS 1 as the NE model Alternately choose All NE Models Step4 Check the Enable Auto Backup check box Step5 Specify the number of backup copies and the backup time Step6 Click Save J 8 10 For the CRS 1 CTM Does Not Automatically Discover Neighboring NEs Step 1 Verify that there is a physical link between the two NEs Step2 Complete one of the following options depending on your configuration e Ifthe link is an Ethernet link bring up the interfaces on both NEs For example enter the following commands on the first router Router 1 Router 1 con
156. m scmnode dbaccess The file is corrupted Read the file manually using the statsreader statsreader lt stats filename gt amp Note If the output doesn t look normal or the file looks corrupted then the problem is most probably on the switch side Report the problem to the CTM team and the switch team e There is space left on the machine to insert any more If not report the problem immediately Defect Information Collect the following information for further analysis e Statsparser log output of df k and mount e Related key index entries e BadfileList J 10 11 Statistics Report Problems This section includes the following information e J 10 11 1 Statistics Report page J 122 e J 10 11 2 You Collected Data but You See No Data Available page J 122 Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN AppendixJ Troubleshooting WE J 10 11 Statistics Report Problems e J 10 11 3 You Generated a Report but Do Not See Data for a Long Time page J 122 e J 10 11 4 You See FDNs for Other Entities when You Generate a Report page J 122 e J 10 11 5 You See the Wrong FDN for a Raw Report page J 122 e J 10 11 6 Utilization Report Value Is Greater than 100 page J 123 J 10 11 1 Statistics Report You can use the Statistics Report to view reports of statistics data that are collected from the switch The SRT GUI provides the ability to generate and view historical statistics reports Reports s
157. message in the log looks like the following ERR Fatal Error at file line 0 char 0 Message An exception occurred Type RuntimeException Message The primary document entity could not be opened Id opt svplus xml configcenter XXX XXX XXX xml lt someNumber gt lt x gt lt someTime Stamp gt ERR InternalError XML Parsing Error If the xml filename mentioned above has two consecutive hyphens for example ABC X YZ xml or if it has a preceding hyphen for example ABC xml or terminates with hyphen before the file extension for example ABC xml proceed to step 4 See if the xml file mentioned in the log message exists If it does not exist contact CTM engineers Perform the following substeps to investigate incorrectly formed XML filename strings The format of XML filenames is lt Platform gt lt Card gt lt InterfaceType gt lt SomeEntityName gt xml This format is generic with a few exceptions Also note that Platform and InterfaceType are optional and will not be seen in many files For example ABC Card xml is a valid XML filename a Ifthe Platform part of the XML filename is incorrect missing check the Node table to see if it is correctly populated b If the Card part of the XML filename is incorrect missing check the Card table to verify if it is correctly populated c Ifthe Interface Type part of the XML filename is incorrect missing check the appropriate table to verify if it is correctly populated
158. mmand nmControl on the CTM CLI Check the CMSCclient log for the node Verify whether the correct message was received for the node From nmClient use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI Defect Information Collect the following information for further analysis e ILMITopoc log topod log and NMServer log e Dump outputs of ILMITopoc topod and NMServer The dump can be captured by issuing a kill USR1 signal to the process e Output of the switch CLI selnd and dbnds J 10 5 3 5 Duplicate Nodes Are Displayed on the Configuration Center Chassis View Diagnostics Center or Statistics Report Step 1 Step 2 Step 3 Step 4 Step 5 Check whether the node table has duplicate node IDs The node table can be queried using the following command tballraker18 echo select from node where node_id 2 and slot 1 dbaccess stratacom Check whether both nodes in the database have active 1 If both nodes are active then collect the logs If duplicate nodes do not exist in the database check the cache of NMServer using nmControl Check the CMSCclient log for the node From nmClient use the getTopology option to retrieve the node information and verify whether it matches the database and the GUI Cisco Transport Manager Release 6 0 User Guide Piss W 78 16845 01 Appendix J Troubleshooting J 10 5 Configuration Center Chassis View Diagnostics Ce
159. mmunity strings are verified and the device is still unavailable check the error log for any errors reported by ONS 155xx NE Service J 2 13 Circuits Are Not Displayed Step 1 Step 2 Step 3 Step 4 If CTM fails to display some circuits or if the information displayed by CTC and CTM differs complete the following steps Wait for two minutes while the CTM server synchronizes with the NE If the Circuit table is not in autorefresh mode click the Refresh Data tool when the tool flashes Verify that the Circuit table is opened from either the source or destination NE of the circuit to be viewed Verify that the NEs that are part of the circuit source destination or drop are available in CTM Open another CTC session and compare the circuit information in the newly opened CTC session with the circuit information displayed by CTM J 2 14 Circuit State for Monitor Circuits Reads Duplicate ID CTM generates a unique number that is appended to the circuit name to make the monitor circuit name unique On rare occasions CTM might create two or more monitor circuits with the same name The circuit state for these circuits reads Duplicate ID This is a known issue that has been tracked as DDTS number CSCdz87566 When the circuit state reads Duplicate ID you cannot see the actual circuit state In this case you must change the duplicate name to a unique name so that you can see the correct circuit state Use the Modify Circ
160. mplete Connections Step 1 Step 2 A common issue is having too many connections on a given card This problem may be a result of incomplete connections For example if a feeder segment of a three segment connection has not been processed by the databroker the incomplete connection will have an endpoint of a routing node not the missing feeder There will appear to be too many connections on the routing node A connection can be identified as incomplete if the state is equal to four in the user_connection table Check the user_connection table for the number of segments and the inseg flags for the connection in question Select num_segs status inseg_tbl_1 inseg_tbl_2 inseg_tbl_3 from user_connection where _node_id x and _slot y and l_port z and _subchnl_1 v and _subchnl_2 w Defect Information You need the logs of the processes on both sides of the interface on which the message was dropped A dump of the specific user_connection table entry that is incorrect is also useful The specific node slot logical_port vpi vci of both ends of the connection in question is also useful Possible alternative workarounds TIf the problem is between the sdbroker cache and the database a cache resync can be done To do this execute the command ustr usrs svplus tools CacheResync t lt time in seconds gt Related key index entries inconsistency incomplete connection 78 16845 01 Cisco Transport Manager Release 6 0 Us
161. n has an incorrect signaling configured on it This error is indicative of this condition For nontrunking ports the signaling on the port cannot be PNNI For the user endpoint do the following a Use the dsppnport command to verify that the interface is a PNNI port or a controller port b Change the port to a UNI port Before doing this make sure that no calls are going through this interface dnpnport lt portID gt cnfpnportsig lt portID gt univer uni30 After changing the configuration retry the connection request Defect Information Not applicable as this error has to be a switch configuration issue J 10 8 4 21 Cmgrd Addition Errors Provisioning a Connection Results in Local Channels not enough Switch Error Step 1 Step 2 Step 3 Related key index entries cmgrd channels During an add connection request it is possible that the user endpoint port or card that the connection is being set on has already reached its limit of allowable connections Check the port or card of the user endpoints on the CLI Check the port resources of the endpoint using the dsppnportrsrc If the number of LCNs is zero delete some connections from this interface After changing the configuration retry the connection request Defect Information Not applicable as this error has to be a switch configuration issue J 10 8 4 22 Cmgrd Addition and Modification Errors Provisioning a Connection Results in Error in Traffic Parame
162. nc gt SENDING RESYNC STATUS 2 FOR NODE 17 TO GUI Node is synchronizing This message tells you that ooemc will start the node resync process for nodes with ID equal to 17 N17 If you look further in the log you should see log messages related to SNMP requests and SNMP responses If the SNMP request is successful the switch will respond to the request The ooemce will then process the SNMP response by invoking response function which may do nothing A EMC_SnmpFunc_c ProcFunc_GenNodeBulkFile_1 gt entering b If the log messages indicate SNMP error refer to the snmpcomm log files for more failure information Determine if the problem is caused by bulk creation traps Grep 60901 60902 and 60903 from the log files See J 10 4 3 Node Mode Remains in 1 page J 41 for more information You should see something like the following INFO lt EMC_TrapClientImpl onIncomingTrap gt NTS NodeId 17 genericTrap 6 specificTrap 60901 INFO lt EMC_TrapClientImpl onIncomingTrap gt NTS NodeId 17 genericTrap 6 specificTrap 60902 Determine if the problem is caused by the config file FTP Check the FTP or the FTP plus config upload filename You should see something like the following NOTICE N17 lt EMC_NodeFsmHandler_c LoadShelf gt to ftp opt svplus tmp CARD_01_CC CF 17 INFO lt ParseFile_c CheckFile gt OOEMC9 CHECKSUM OK FOR FTP FILE opt svplus tmp CARD_01_CC CF 17 These messages indicate that FTP is succe
163. nd a new Configuration Center Chassis View Diagnostics Center or Statistics Report GUI 2 Use nmControl and perform Resynch node specifying the node ID 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems J 10 5 5 Configuration Center Chassis View Diagnostics Center or Statistics Report Alarm Issues This section includes the following information J 10 5 5 1 Alarm Processing Basics page J 62 J 10 5 5 2 XML Schema for Alarm Rules page J 63 J 10 5 5 3 Alarm Severity and Object Severity page J 65 J 10 5 5 4 Severity Shown on Tree View Does Not Match Severity Shown on Platform page J 66 J 10 5 5 5 Alarm List Shows Alarm that Does Not Exist on Platform page J 67 J 10 5 5 6 Transient Event Has Disappeared Unexpectedly page J 68 J 10 5 5 7 Port in Tree View Displays Aggregate Alarm However No Children Exist Under Port page J 68 J 10 5 5 1 Alarm Processing Basics Alarm processing is done within the NMServer processes The following are some vital points with regard to alarm processing Alarm rules are defined in an XML file opt svplus xml ruledata xml For details on XML schema alarm rules see Figure J 2 GUI clients register for alarms by providing the parent FDN GUI clients pull initial alarms from NMServer After initial pull of alarms all alarms are pushed to re
164. nformation of an element shows incorrectly on the Configuration Center Chassis View Diagnostics Center or Statistics Report Check whether the node is synced up This can be done by using the command selnd on the CTM CLI If the node has not synced up then wait for it to do so Check whether the card line or port is populated in the database Refer to the Cisco Transport Manager Release 6 0 Database Schema If the entry in the database matches the GUI see J 10 4 Equipment Management Problems page J 40 If the entry doesn t match the database check the NMServer cache This can be done by using the command nmControl on the CTM CLI Check the nmControl lt pid gt dump Cisco Transport Manager Release 6 0 User Guide Po 78 16845 01 AppendixJ Troubleshooting Step 6 Step7 J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems il Use nmClient and verify whether getChildren for FDN returns the correct information for the element in the nmClient lt pid gt dump file If the element is present in the NMServer cache open a new GUI and verify whether the missing elements are shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Possible alternative workarounds 1 Open a new GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report
165. ns Screen snapshots in particular error information messages dialog boxes Cmsvr log file under the opt svplus log directory Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 3 Cmsvr Errors The following are some of the most common errors seen when a provisioning request is issued J 10 8 3 1 End to End Type Cannot Be PVC for Connections with PNNI Segment page J 90 J 10 8 3 2 Unsupported Service Type for the Foresight Connections for PNNI SPVC Endpoints page J 91 J 10 8 3 3 CE CE Connection Port Speed Should Be the Same on Both Endpoints page J 91 J 10 8 3 4 Cannot Create One Segment Hybrid page J 91 J 10 8 3 5 Cannot Add Connection Between byte Port Header and byte Port Header for chan type NIW and NIW replace page J 91 J 10 8 3 6 Unsupported Service Type General node slot port version page J 92 J 10 8 3 7 Endpoint Is Not Present in One of the Connection Tables During a Modify Operation page J 92 J 10 8 3 8 Cmsvr Connection Diagnostic Issues page J 93 J 10 8 3 9 TestConn or TestDelay Failed with Switch Error page J 93 J 10 8 3 10 Connection Up Down Reroute Failed page J 94 J 10 8 3 11 Connection Trace Failures page J 94 J 10 8 3 1 End to End Type Cannot Be PVC for Connections with PNNI Segment When an add connection is attempted cmsvr checks its the endpoints and the end to end type to make sure it is supported The error message states EndtoEnd
166. nter and Statistics Report Problems il Step6 If duplicate nodes are not present in the NMServer cache open a new GUI and verify whether the node is shown in it Defect Information Collect the following information for further analysis e CMSCclient log file under opt svplus log For PC Client collect CMSCclient log file under D Documents and Settings lt username gt log e Node data from node table for that specific node e ILMITopoc log and topod log if there are duplicate nodes in the database Possible alternative workaround None J 10 5 4 Network Element Discovery Issues This section includes the following information e J 10 5 4 1 All the Cards Under a Node Are Missing page J 59 e J 10 5 4 2 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Card Line or Port for a Node page J 60 e J 10 5 4 3 Card Line or Port Information Is Incorrect on Configuration Center Chassis View Diagnostics Center or Statistics Report page J 60 e J 10 5 4 4 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows a Card Line or Port that Is Not Present in a Node page J 61 J 10 5 4 1 All the Cards Under a Node Are Missing No cards are shown under a node in the Configuration Center Chassis View Diagnostics Center or Statistics Report Step 1 Check whether the node is synced up This can be done by using the command selnd on the CTM CLI Step2 Ifthe node ha
167. ode 2 problem is that the switch will abort the bulk file creation and send trap 60903 to CTM CTM will reschedule the next SNMP request unless it exceeds maximum retries Check the log file for trap 60902 CARD_01_CC CF 10 This file is a shelf generic file and is the first file to be uploaded after CTM receives trap 60902 from the switch The 10 is the node ID If you find trap 60902 in the log file but no config file has been uploaded after a reasonable amount of time it is possible that the FTP failed To find out if FTP failure has occurred do the following Grep to ftp file and the config filename or just FTP from the log files Trace the log messages in the log file For a more detailed information on SNMP request failure and FTP failure check the snmpcomm and cwmftpd log files and the cwmftp request_log Search for the filename at the time the error happened in both files The cwmftp request_log gives the summary or final result of the FTP operation and any error is reported The cwmftp log shows the FTP operation details CTM will upload and parse a set of config files from the switch The following files are uploaded from the MGX NE which contains VISM AXSM VXSM SRM and RPM RPM PR cards 1 o Nn oa FF VN CARD_01_CC CF SM_1_slot CF SM_1_slot CS SM_CARD_01_slot CF SM_CONN_01_slot CF SM_ALARM_01_slot CF SM_CON_UPDATE_01_slot CF SM_CARD_01_SRM CF 78 16845 01 Cisco Transport Manager Release 6 0
168. ode is active Step 2 If the node is active see J 10 2 1 No Nodes Are Discovered page J 34 Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EGN Appendix J Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems Step 3 Step 4 Step 5 Step 6 Check the node in the NMServer s cache This can be done by using the command nmControl in the CTM CLI Check the CMSCclient log for the node trunk Verify whether a delete message was received for the node From nmClient use the getTopology option to retrieve the node information and verify whether it matches the database and GUI If the node is not present in the NMServer cache open a new GUI and verify whether the node is shown in it Defect Information Collect the following information for further analysis e ILMITopoc log topod log and NMServer log e Dump outputs of ILMITopoc topod and NMServer The dump can be captured by issuing a kill USR1 signal to the process e Output of the switch CLI selnd and dbnds J 10 5 3 4 Configuration Center Chassis View Diagnostics Center or Statistics Report Shows Incorrect SyncState for a Node Step 1 Step 2 Step 3 Step 4 Check the node information in the database by querying the node table This can be done by using the command selnd on the CTM CLI Check the status Check the node in the NMServer s cache This can be done by using the co
169. odification Errors Modifying a Connection Results in Connection does not exist in CproDb Agent returned no such name Switch Error page J 106 J 10 8 4 1 High Level Connection Management Subsystem Architecture Figure J 6 gives a high level view of the Connection Management Subsystem Figure J 6 Connection Management Subsystem CM Clients ee CM GUI Server ConnProxy f Log interface Log interface SDatabroker lt _ gt Cnmgrd x IOSMgr Log interface DB SnmpComm lt gt a SNMP R SET GET a Related key index entries connproxy cmserver xcmgrd cmgrd The Connection Management Subsystem comprises the following modules Cisco Transport Manager Release 6 0 User Guide Pwo 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W e ConnProxy The SNMP Agent Interface for Connection Management e CmServer Provides the backend functionality to the Connection Management GUI e Cmgrd tThe backend module which accepts requests from ConnProxy CmsServer Clients and issues the SNMP VarBinds to the switch for addition modification and deletion Subsequently cmgrd informs sdbroker after an addition modification and deletion operation of the status of the particular request upon which sdbroker updates the CTM database The following errors are applicable to all types of connections PVC SPVC XPVC
170. olling of Real Time Counters Fails Step 1 Check whether the node is reachable and the community strings are set correctly Step2 Check whether the real time counters are applicable Check any appropriate commands from the CLI Cisco Transport Manager Release 6 0 User Guide ca 78 16845 01 Appendix J Troubleshooting Step 3 Step 4 J 10 9 Diagnostics Center Problems W Use any SNMP tool and check whether the MIB objects are retrievable Narrow down the problem by selecting one counter at a time and determining for which counters the polling fails Defect Information Collect the following information for further analysis e DCServer log file from opt svplus log directory e Screen snapshots in particular error information messages dialog boxes e CLI output of dspcd lt slot No gt for that card Output of any appropriate CLI commands used e If step 4 was performed provide the list of counters for which polling failed Possible alternative workaround Use the CLI to monitor the real time counters J 10 9 2 6 Some Real Time Counters Are Not Shown Step 1 Step 2 Step 3 Some real time counters are missing from the Diagnostics Center Check whether the counters are applicable to the card version mode if applicable Check any appropriate commands from the CLI Use any SNMP tool and check whether the MIB objects are retrievable Defect Information Collect the following information for further analysis e DCServ
171. on Collect the following information for further analysis e Get the copy of the Chassis View xml file used e Take a copy of the chassisview jar from the opt svplus java jars cwm directory Verify that gif files were used to draw the lines Possible alternative workaround Try selecting from the tree view J 10 6 11 Wrong Tooltip Is Displayed When you move the cursor over a line the wrong tooltip is displayed For example when moving the cursor below the last line in the card the tooltip of the line gets displayed instead of the tooltip of the card Lines must be spaced sufficiently apart If they are not they may overlap causing the wrong tooltip to be displayed Defect Information Collect the following information for further analysis e Get the copy of the XML file used Chassis View xml for MGX nodes and BPXIGX xml for BPX IGX nodes e Take a copy of the chassisview jar from the opt svplus java jars cwm directory Verify that gif files were used to draw the lines Possible alternative workaround Try selecting from the tree view J 10 7 Configuration Center Management The Configuration Center management functions are divided into the following categories e Network elements Manages the nodes and their components For network element management the Configuration Center communicates with the Config Server process e Connections Manages the connections between the nodes For connection management the Confi
172. on with multiple IP interfaces The NMS_IP_ADDRESS setting in opt svplus config svplus conf has to match the interface users chose for CTM To check the IP interfaces run ifconfig a CTM stations with only one IP interface does not need to have the NMS_IP_ADDRESS setting Check the Trap Manager List on the switch SSH or Telnet to the switch Run dsptrapmgr command to show the Trap Manager List Check whether the CTM is in the table If so go to step 4 Otherwise if the table is already full remove unwanted entries with deltrapmgr command The nts should be able to register the CTM into the table within a few minutes If not go to step 4 Check the SNMP community strings In case the SNMP community string on the switch has been changed from the default setting the SNMP community string on the CTM have to match the new ones on the switch The SNMP community string on the switch can be viewed with dspsnmp command The setting on the CTM can be viewed in Domain Explorer Defect Information Collect the following information for further analysis e Selnd and dbnds command output e Nts log and EM log of the node in question Possible alternative workaround If community strings do not match run the CTM Configurator GUI to correct them opt svplus java bin runConfigurator Related key index entries nts traps J 10 13 1 2 Config Change or Provisioning Activity Not Reflected on CTM The nts does not receive any traps from a particular swi
173. onsistent data See if the MIB OID is pointing to the information that you are looking for If not contact CTM engineers Use any SNMP tool to perform an SNMP GET operation on the MIB object which shows inconsistent If the value retrieved using the SNMP GET operation and the value shown in the GUI are not the same contact CTM engineers The following shows the syntax of the SNMP GET operation in a HP OV snmpget Cf options lt hostname gt lt community gt lt objectID gt Check the data type of the MIB object that shows inconsistent data If the data type of the MIB object in the configserver log is not the same as the data type defined in the MIB object definition available at opt svplus mibs directory contact CTM engineers with defect information Check the data type of the MIB object a If the data type of the MIB object is Bitmap then convert the decimal value of the MIB object to a binary value and compare the bits SET RESET with the values defined in the MIB object definition and displayed in the GUI Contact CTM engineers with defect information b If the data type of the MIB object is an IpAddress verify whether the value is in the appropriate format for example lt aaa bbb ccc ddd gt If the value is not in proper format contact CTM engineers Defect Information Collect the following information for further analysis e Screen snapshots for the investigative commands used for debugging this issue
174. ooemc has been sent then collect the log files and report the problem Step4 Ifthe rtm link up message cannot be found then search for link down message Grep RTM_LINK_DOWN from the emd log files If the rtm link down message is found then the node is not reachable by the nts process Check with the network administrator and see J 10 13 1 NTS page J 127 if RTM_LINK_UP and RTM_LINK_DOWN messages are not found Defect information Collect the following information for further analysis e Ooemce and nts log files from the opt svplus log directory Possible alternative workaround None J 10 4 4 Node Mode Remains in 2 After EMD receives the rtm link up message and the node active message has been sent to ooemc ooemc will change the node mode from to 2 and trigger the node resync process The node resync time varies depending on the switch configuration and network activities If the node mode stays in mode 2 for longer than the normal time for node resynchronization there can be a problem with the node resync process Step 1 The debugging process for this problem is mainly focused on log file inspection Make sure that log level of all related CTM processes is set at the right level See J 10 4 2 Network Setup and Configuration Required for OOEMC Sync Up Process page J 41 for more information Step2 Go to the opt svplus tmp directory to check for configuration upload files Step3 Determine which files have been uploaded for a node
175. ook for errors related to software download memory backup or memory restore Cisco Transport Manager Release 6 0 User Guide 78 16845 01 m Appendix J Troubleshooting HZ J48 ONS 15800 ONS 15801 or ONS 15808 NE Generates Too Many Alarms J 4 8 ONS 15800 ONS 15801 or ONS 15808 NE Generates Too Many Alarms Occasionally ONS 15800 ONS 15801 and ONS 15808 NEs generate so many alarms that the CTM client cannot process all of them Set the operational state of the NE to Under Maintenance while debugging the cause of the alarms Step 1 In the CTM Domain Explorer window right click the ONS 15800 ONS 15801 or ONS 15808 that is generating the large number of alarms and choose Mark Under Maintenance Step 2 Debug the ONS 15800 ONS 15801 or ONS 15808 NE to see which card is in error Find the card that is generating the alarms and mark it as Under Maintenance Step3 While the card is under maintenance for debugging set the ONS 15800 ONS 15801 or ONS 15808 NE back to In Service J 4 9 Bandwidth Intensive Operations Are Blocked Bottlenecks in the DCN affect bandwidth intensive operations such as software download Tune the timeout and retry values to match DCN performance Log into the NE and enter the following commands e ip tftp min timeout Sets the minimum wait time in seconds before retrying the read write request e ip tftp max timeout Sets the maximum wait time in seconds before retrying the read write re
176. or critical retry or Critical retry failed and wait for history retry or History retry failed and wait for next retry or No more retry This could happen for various reasons but it is mainly due to ftp tftp failures Step 1 Verify that the node is reachable pingable with nw_ip_address or lan_ip_address e If only the nw_ip_address is reachable and the lan_ip_address is unreachable then make sure that IP Routing for stats collection is using In Band e If only the lan_ip_address is reachable and the nw_ip_address is not reachable then make sure that IP Routing for stats collection is using Out of Band e Use the dspifip command to check the lan_ip_address and nw_ip_address Step2 Check the pmcollector log to verify whether or not you are using the correct ip_address for stats collection Step3 Check for the existence of connections on the slot for which you are collecting the stats Also check for the stats flag Use the command sumDBShow or sfmDBShow on the node Step 4 Verify that the ftp_user_name and ftp_user_passwd in the coll_info table and the ftp_user_name and ftp_user_passwd in the node_info table of the stratacom database are matching and are valid Defect information Collect the following information for further analysis e Pmcollector log dump of node_info and coll_info e Related key index entries e Pmcollector Possible alternative workaround Use the atm or LAN IP that is reachable Cisco Transport Man
177. ort in the tree view and drag and drop it to the right window pane The Connection view window appears Cisco Transport Manager Release 6 0 User Guide Pe W 78 16845 01 Appendix J Troubleshooting Step 2 J 10 6 Chassis View Problems W c Click the Get button The connections should be listed and the alarm status should match the alarm status in the alarm list If the connection alarms do not match the aggregate port alarm s in Alarm List GUI there may be a defect Collect the following defect information e CMSCclient log NMServer log sdbroker log xdbroker log e Capture NMServer dump use nmControl Possible alternative workaround None J 10 6 Chassis View Problems This section includes the following information e J 10 6 1 Chassis View Basics page J 69 e J 10 6 2 Lines Not Displayed in the Chassis View page J 70 e J 10 6 3 Card Not Displayed in the Chassis View page J 70 e J 10 6 4 DCA DCB Status Not Displayed on PXM Cards page J 71 e J 10 6 5 Ethernet Status Not Updated on PXM Cards page J 71 e J 10 6 6 HIST CBRX and CBTX Status Not Updated in MGX Nodes page J 71 e J 10 6 7 RPM Card Status Not Updated page J 71 e J 10 6 8 RPM Secondary Card Status Is Blue page J 71 e J 10 6 9 Lines Not Displayed on Secondary Card page J 71 e J 10 6 10 Lines Not Selectable page J 72 e J 10 6 11 Wrong Tooltip Is Displayed page J 72 J 10 6 1 Chassis View Basics Chassis View is a read only application that provi
178. ostics Center page J 110 e J 10 9 1 6 Drag and Drop from to Diagnostics Center to from Other Applications page J 111 e J 10 9 1 7 Drop Target Displays Incorrect Object or Object Data page J 111 e J 10 9 1 8 Diagnostics Center Does Not Respond GUI Is Grayed Out page J 111 J 10 9 1 1 Cannot Launch Diagnostics Center Step 1 The Diagnostics Center cannot be launched main window will not appear using one of the following methods e Click the Diagnostics Center icon from the Launch Center or from any application e Choose Tools gt Diagnostics Center from any application e Right click the selected node from the hierarchical tree and choose Diagnostics Center Check diagcenter jar file under opt svplus java jars cwm directory Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e DCSserver log file under the opt svplus log directory Possible alternative workaround None J 10 9 1 2 Cannot Launch Other Applications from the Diagnostics Center Step 1 The Diagnostics Center cannot launch other applications using one of the following methods e Choose the target application under the Tools menu item e Right click on the selected object from the hierarchical tree and choose the targe
179. oubleshooting WE J10 4 Equipment Management Problems e VxsmCrr Trap e VxsmTone Trap e VxsmAs Trap e VxsmAsp Trap e VxsmAs Trap e VxsmLapd Trap e VismABCDBitTemplate Trap Review the log messages using the traplist command and decide how you can grep the messages according to your needs In the above example there is another key word PROCESSED that indicates that the trap has been processed This gives you the starting point in the log file so that you can study the log messages If the database is not populated correctly then the subsequent log messages should provide you information on the database inconsistency problems Some trap processing may invoke SNMP data upload From the log messages you can verify whether the uploaded data is correct or not Other possible reasons for database inconsistency are e SNMP upload failure and maximum retries is exceeded SNMP timeout or throttle error You can verify the problem in the snmpcomm log files e Database operation error You can verify the problem in the ooemc log files If you do not know exactly what trap sequence is sent by the switch to CTM you should try a working scenario and study the log for the trap sequence or you can use HPOV to determine the trap sequence Step3 To verify whether or not a trap has been received from the switch and forwarded to ooemc refer to the nts log If the trap has been buffered in the trap queue the possible reasons are e Node is
180. p 5 Step 6 Step7 Check whether the node is synced up This can be done by using the command selnd on the CTM CLI If the node has not synced up wait for it to do so If the node has synced up partially check whether the card line or port is populated in the database Refer to the Cisco Transport Manager Release 6 0 Database Schema If the entry is not populated in the database see J 10 4 Equipment Management Problems page J 40 If the entry is present in the database check the NMServer cache This can be done by using the command nmControl on the CTM CLI Check the nmControl lt pid gt dump If the element is not present in the NMServer cache use nmClient and verify whether getChildren for FDN returns the entities in the nmClient lt pid gt dump file If the element is present in the NMServer cache open a new GUI and verify whether the missing elements are shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Possible alternative workarounds 1 Open anew GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report 2 Use nmControl and perform Resync Node specifying the node ID J 10 5 4 3 Card Line or Port Information Is Incorrect on Configuration Center Chassis View Diagnostics Center or Statistics Report Step 1 Step 2 Step 3 Step 4 Step 5 The entity i
181. p neighbors J 9 7 XR 12000 NE Explorer Applications Are Disabled The CTM NE Explorer enables only those applications for which the XML version matches on CTM and the router If an NE Explorer application is disabled there might be a version mismatch Do the following e Collect all the XML files from the lt CTM_home gt cfg metadata directory e Run the lt CTM_home gt bin vwdata sh script to collect the hfr_version_info database table entries On Windows if an NE Explorer or GUI application returns an error repeat the same operation after invoking the GUI by entering one of the following commands depending on your network size lt CTM_home gt bin ctm small debug exe lt CTM_home gt bin ctm medium debug exe lt CTM_home gt bin ctm large debug exe This opens a console that logs all the exceptions Capture the console log for debugging 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HI J10 Problems with MGX Voice Gateway Devices On UNIX enter the lt CTM_home gt bin ctmcdebug start command to start the client and repeat the same operation This generates the error and exception messages on the terminal Capture the terminal messages for debugging J 10 Problems with MGX Voice Gateway Devices This section describes troubleshooting procedures for problems related to the MGX Voice Gateway devices J 10 1 Discovery Mechanism CTM manages the Private Network Network In
182. port C and 1_subchn1_1 D and 1_subchn1_2 E dbaccess stratacom Step2 Ifit does not exist look in the corresponding connection table to see if the connection exists Connection entries are as follows e connection for FR e atm_connection for atm segments e cesm_connection for cesm segments e rpm_connection for rpm segments e voice_conn for vism segments Step3 If it does not exist look in the dbkr_temp table for an entry for that connection Defect Information The following log files need to be collected e Cmsvr sdbroker and corresponding em logs in the directory opt svplus log e Outputs of the database entries for the endpoints in the segment tables and user_connection table Possible alternative workaround Resync the node or wait for sync up to complete Cisco Transport Manager Release 6 0 User Guide Plo 78 16845 01 Appendix J Troubleshooting J 10 8 Connection Management Problems W J 10 8 3 8 Cmsvr Connection Diagnostic Issues The cmsvr processes validates and services the connections provisioning requests from the CM GUI During an add modify or delete connection request the request is forwarded to the cmgrd process through an ILOG IPC request If the request is for connection diagnostics such as testdelay or testconn except conntrace cmsvr sends it to the snmpcomm process for forwarding to the switch J 10 8 3 9 TestConn or TestDelay Failed with Switch Error Step 1 Step 2 Step 3
183. port Manager Release 6 0 User Guide rss E 78 16845 01 Appendix J Troubleshooting J 4 5 Cannot Delete a Subnetwork Hl Step9 Click Save J 4 5 Cannot Delete a Subnetwork If a subnetwork cannot be deleted from the Subnetwork Explorer complete the following steps Step 1 Verify that the user logged in as a SuperUser not as a Provisioner or an Operator Provisioners and Operators cannot delete subnetworks If the user logged in as a Provisioner or Operator restart the CTM client session and log in as a SuperUser Step2 Verify that the target subnetwork does not contain NEs Move all NEs to another subnetwork before deleting the empty target subnetwork J 4 6 Cannot Move an NE Between Subnetworks When automatic grouping of NEs in subnetworks is enabled you cannot move NEs between subnetworks To move NEs from one subnetwork to another Step1 In the Domain Explorer window choose Administration gt Control Panel Step2 In the Control Panel click UI Properties Step3 Under Subnetwork Grouping uncheck the Automatically Group NEs in Subnetworks check box Step4 Click Save J 4 7 Cannot Schedule Jobs The CTM client can schedule three types of administrative tasks e Software download e Memory backup e Memory restore CTM maintains an Error Log and Audit Log to track potential problems To view the Error Log or Audit Log Step1 In the Domain Explorer window choose Administration gt Audit Log or Error Log Step2 L
184. problem is between the sdbroker cache and the database a cache resync can be done To do this execute the command usr usrs svplus tools CacheResync t lt time in seconds gt If the problem still exists collect the defect information Related key index entries inconsistency connection databroker J 10 13 2 2 Inconsistent Connection Status Step 1 Step 2 The GUI display of Status is from the state field of the user_connection table The field represents the worst condition of any of the segments in the connection This field values are e 1 Clear e 2 Fail e 3 Down e 4 Incomplete e 5 Error If the value is not what is expected check the connection level databases for each segment to see if they are correct The last message on the segment in question is the important one If the status is incomplete it means that one of the segments of the connection is not in the user_connection table check the in_seg flags in the user_connection table entry for this connection Search the database and messages for the segment that has flag 0 If two primary endpoints each have the same secondary endpoint then you have an errored connection The first connection established will have a status of Error and the second primary endpoint connected to the secondary will have a status of Incomplete Defect Information See J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 Possible alternative workaround Se
185. quest e ip tftp backoff delay Sets the number of seconds to extend wait time if the read write request times out e ip tftp write retries Sets the maximum number of times to retry the TFTP read write request before declaring a failure J 4 10 NE Displays Incorrect Configuration Management Data Step1 In the Domain Explorer window choose Administration gt Service Monitor and verify that the NE Service is running If it is not running start the NE Service Step2 Test NE connectivity a Inthe Domain Explorer tree select the NE and choose Fault gt Test NE Connectivity The message lt NE_name gt lt IP_address gt is available should be generated b Ifthe NE is unavailable establish connectivity to the NE before retrieving configuration data Cisco Transport Manager Release 6 0 User Guide o 78 16845 01 Appendix J Troubleshooting J 4 11 Cannot Customize the Network Map W J 4 11 Cannot Customize the Network Map Step 1 Step 2 Step 3 If an image file is not displayed while changing the Network Map background or while changing a node icon complete the following steps Choose another image file The file might be corrupt Check the size of the image file The image file might be larger than 100 KB which is too big to load If the file is too big use a smaller image file Verify that the image file exists in the images mapbkgnds shapefiles directory or the images mapicons directory If the
186. r any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN Appendix J Troubleshooting HZ J 10 8 Connection Management Problems J 10 8 1 5 Advanced Mode Tab Cannot Launch the Template Details Dialog Box Using the Template Details Button The Template Details button on the Advanced Mode tab fails to launch the Template Configuration dialog box Defect Information Collect the following information for further analysis CMSCclient log file under the D Documents and Settings lt username gt log directory Java Console information in particular any Java raised exceptions Screen snapshots in particular error information messages dialog boxes Cmsvr log file under the opt svplus log directory Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 1 6 Cross Application Cannot Launch Other Applications from the Connection List For a retrieved connection on the Connection tab the right click popup menu does not launch the selected target application for the connection or it results in the Operation Not Supported message dialog box Step 1 Determine if the operation is supported by the target appl
187. r it received a 70005 and 70201 trap if the persistent topology feature is enabled b If you see the trap in the log verify that ILMITopoc s SNMP requests issued after receiving the trap are successful If an SNMP request does not go through verify whether CTM has IP and SNMP connectivity to the newly added node c Ifyou do not see the traps in the ILMITopoc log see J 10 13 1 NTS page J 127 Step3 Complete the following substeps to collect defect information for analysis a Save ILMITopoc log topod log NMServer log and nts log b Enter the following command to collect the dump outputs of ILMITopoc topod and NMServer kill USR1 signal lt process gt c Collect the output of the switch CLI selnd and dbnds Cisco Transport Manager Release 6 0 User Guide P33 78 16845 01 AppendixJ Troubleshooting J 10 3 Discovery Issues at Runtime Mil Step4 Ifthe preceding steps do not solve the problem delete the node from the network and add it again J 10 3 2 Node Name Change Is Not Updated in the Configuration Center Chassis View Diagnostics Center or Statistics Report If changes to the node name are not reflected on the Configuration Center Chassis View Diagnostics Center or Statistics Report complete the following steps Step 1 If the node table has an entry for the node with the correct name complete the following substeps a Enter the following command to dump the cache of NMServer topod and ILMITopoc
188. r list if the list is full If the list is not full the registration takes place automatically if CTM is configured to manage the node e The trap IP should have been configured on the switch properly Whenever you configure the switch to use any IP atm0 IP or InPci0 IP of the switch as primary IP the topology component of CTM should use that IP for node discovery You can display the primary and secondary IP information of the switch by typing the CLI command dspndparms To configure the primary and secondary IP addresses you can enter enfndparms and the CLI will prompt you for options After you have set up the primary and secondary IP addresses you should also configure the primary IP as trap IP on the switch by entering the CLI command enftrapip configured_IP where configured_IP is the IP that you have chosen to use as the primary IP in the CLI command enfndparms e On CTM the following is a list of configurable parameters in the file opt svplus config emd conf of your CTM machine that you should set up for proper problem logging OODebug For debugging purposes this parameter should be set to level 6 or above for example the log level statement in emd conf should be OODebug Level 6 OOKeep For debugging purposes this parameter should be set to a value depending on how many log files you want to keep for example the statement in emd conf should be OOKeep 100 ooemc log files per oochild Note Sometimes a different ne
189. r the up down reroute diagnostics on the same connection Possible alternative workaround a Check the status of the connection and clear any alarms due to line port issues adding loopback correcting the physical cable issues by verifying the physical port connections and so on After the connection comes up and is in the Clear state rerun the up down reroute diagnostics b Verify the connection status to find out if the connection is already down when trying to do a down operation or the connection is already up when trying to do an up operation SX Note The connection reroute might take the connection to the Fail state for a while J 10 8 3 11 Connection Trace Failures Step 1 Make sure the connection is not in the Fail state Connection trace does not work on failed connections in addition CTM does not block this request Step 2 Query the connection information from the user_connection table for that connection Step3 Query the connection information from the switch CLI and verify the connection status dspcon or dspchan Step4 Execute the connection trace diagnostics from switch CLI and verify that the results match what was reported in CTM conntrc dsptrcbuffer Defect Information Do the following Cisco Transport Manager Release 6 0 User Guide Pro 78 16845 01 Appendix J Troubleshooting J 10 8 4 CMGRD J 10 8 Connection Management Problems W Capture selnd output Copy all the cmsvr logs from
190. request If the request is an add then the query for the intermediate endpoints from either the node or the bis_object tables may have failed If it is a Mod Del then in addition to the failure of the queries into the node and bis_object tables the problem could be that the connection is not found in the user_connection table In this error scenario there could be multiple reasons of failure that differ between add and modify delete requests But most of the time this error is returned if one of the queries to the node bis_object and user_connection tables fails for the current request Some basic checks would be to see if the nodes in question and in the intermediate nodes are really synced up with CTM You should also try the same request from both the CM GUI and Connection Proxy paths Because of the nature of this error it is best that a CTM DE investigate Defect Information Collect cmgrd log cmsvr log ConnProxy log and sdbroker log Possible alternative workaround None J 10 8 4 16 Cmgrd Switch Errors During an addition modification or deletion request cmgrd sends SNMP requests to the switch When the switch rejects an SNMP request cmgrd queries the Error MIB table and gets the error string to display J 10 8 4 17 Cmgrd Addition and Modification Errors Provisioning a Connection Results in Agent reported bad wrong value for one of the variables in the request Switch Error Related key index entries cmgrd bad value
191. rkaround None J 10 8 4 4 Cmgrd Sdbroker Addition Modification and Deletion Errors Provisioning a Connection Results in sDatabroker process busy Please retry Error Related key index entries cmgrd sdbroker The error message states sDatabroker process busy Please retry This indicates that the cmgrd command was able to process the request and sent it to the sdbroker but the sdbroker did not respond prior to the cmgrd s request timeout However between the cmgrd and sdbroker the request appear to be working Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EN Appendix J Troubleshooting HZ J 10 8 Connection Management Problems Defect Information e If the sdbroker is processing many traps more than 20 per second during the time of the provisioning request then there is no sdbroker defect Investigate the switch NTS or EM if you feel that there should not be so many requests occurring at the given time e Ifthe sdbroker appears to be dormant you need to capture the current status of the sdbroker Execute the pstack lt pid gt command twice and save the output Possible alternative workaround Repeated attempts at provisioning should be made If there is a switch that is sending out many alarms fixing the problem on the switch is the best solution If the sdbroker has been sleeping for a long period of time then it should be killed and restarted J 10 8 4 5 Cmgrd Sdbroker Addition Erro
192. rm show inventory show ipv4 interface brief show running config show events show log events buffer all in buffer Turn on trace level debugging in the Control Panel for the service in question wait until the problem occurs and collect the GSRIOXNEService logs from opt CiscoTransportManagerServer log J 9 2 CTM Cannot Discover an XR 12000 Node Step 1 Step 2 Step 3 Step 4 Step 5 Complete the following steps if CTM cannot discover an XR 12000 node Verify that the NE is configured with hfrems as the username and hfrems as the password with root system privileges By default CTM expects the username and password to be hfrems hfrems You can change the security properties from the Control Panel gt Security Properties pane gt XR 12000 tab CTM will use the new username and password to authenticate all XR 12000 devices in the domain Check whether the XML CORBA agent service is running To configure the XML CORBA agent enter Router config xml agent corba ssl Router config commit Note ssl is optional and enables SSL CORBA services If SSL CORBA services are activated enter the following commands to check whether HTTP over SSL is running Router config http server ssl Router config commit The CORBA agent on the router accepts connections only when using the hostname Check to see if the device IP address has an entry in DNS If DNS is not used add an entry in the etc
193. rmation messages dialog boxes Cmsvr log file from the opt svplus log directory Configserver log file from the opt svplus log directory Possible alternative workaround None 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 7 Configuration Center Management J 10 7 2 Configuration Center Element Management The Configuration Center can be used to configure different network element objects The Configuration Center s Element tab is the main window for creating modifying and viewing network elements The Element tab uses the CTM Internal Frame mechanism to display the attributes groups and categories associated with a particular network element The following sections describe guidelines for troubleshooting issues related to creating modifying and viewing network elements J 10 7 2 1 XML Parsing Error Step 1 Step 2 Step 3 Step 4 XML parsing error is seen while launching the Configuration Center either by the drag and drop method or the right click method for a network element The popup window says Internal Error XML Parsing Error Verify whether the network element on which you want to launch the Configuration Center is supported in the CTM version being used If this error is seen for a supported node card go to step 2 Open the opt svplus log configserver log file and look for the message information showing when this error occurred A typical
194. rns this error it indicates that there is some CPU intensive activity happening on the controller card The cmgrd request will be backed off the other segments and this error will be returned However the operation is not backed off for a delete operation because this is a destructive command There is no workaround for this error You will have to wait and retry the request later If you continue to receive the error after waiting and retrying multiple times then the problem must be checked from the switch side Defect Information Not applicable Possible alternative workaround Wait and try the connection at a later time J 10 8 4 24 Cmgrd Modification Errors Modifying a Connection Results in Connection does not exist in CproDb Agent returned no such name Switch Error Related key index entries cmgrd cprodb During a modify connection request the switch could return this error when the connection is truly not present on the switch Check the connection in the user_connection table From this entry verify that all of the segments do exist on the switch CLI If all segments are truly present in the switch CLI then the issue must be checked from the switch side However if any of the segments is not present on the switch then this could be a sdbroker or EM issue Defect Information TIf it is deemed a sdbroker or EM issue collect the appropriate logs and cmgrd log If all of the segments of the connection are present on the
195. roker and psg cmserver on the command line Verify if the CmServer restarted while the operation was being done by checking the cmsvr log and searching for the string START OF THE PROCESS Verify that the time stamp of the server process was restarted while the operation was being performed Also check for any cmsvr coredumps under the opt svplus corefilesdir directory Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Java Console information in particular any Java raised exceptions e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e Configserver log file under the opt svplus log directory Possible alternative workaround None J 10 8 2 2 Connection Template Creation Modification Deletion and Retrieval Errors Step 1 Step 2 When performing any connection template creation modification deletion and retrieval the CM Server could encounter an error and notify the Configuration Center by throwing an application exception error that contains information describing the error As a result the Configuration Center displays the error message The following lists some CM Server related connection template exceptions e CONN_TEMPLATE_ALREADY_PRESENT The Connection Template s is already present in the conn_template table e CONN_T
196. rs Adding a Connection Results in CTM sync up in progress Error Step 1 Step 2 Step 3 Step 1 Step 2 Step 3 Related key index entries cmgrd sdbroker sync up While the system is in the process of syncing up you cannot provision connections You receive the error CTM sync up in progress You need to determine if the system is really in sync up or not If the system is still syncing up then this is proper operation View the sync log located in opt svplus log DBKR_SYNCHUP_MESSAGE note the period before the filename Search for the last xxx Start Begin at line where xxx is either Warm or Cold WarmStart will take a short time compared to coldstart which could take several hours If there is a line at the end of the file stating Declare Sync Up Complete or Warmstart Cache Rebuild Complete then sync up is done and there is a problem with DMD If sync up is not complete view the start time of sync up Compare this to the maximum sync time identified in the usr user svplus config dmd conf file next to the keyword SYNCHUP_ALL_NODES_SYNC_WAIT_LIMIT If the time taken to sync is longer than the time identified in the dmd conf file then DMD has a problem If the system is truly still syncing up then the system behavior is correct Defect Information If there is a DMD problem save the DMD cmgrd sdbroker logs and message logs along with the DBKR_SYNCHUP_MESSAGE and DMD_SYNCHUP_MESSAGE files located in the
197. s e Pmcollector log e Related key index entries e Pmcollector manual SNMP results Possible alternative workaround If the community strings are different on CTM and the node look up troubleshooting for the Topology module 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 10 Performance Management Collection and Parsing Problems J 10 10 1 6 PM Collection Fails for the Node Log File Shows Ftp failed This could happen for various reasons generally it is because of an incorrect username password or a timeout Step 1 Check the node_id node_name and IP address in the node_info table and make sure they are the correct ones Step2 Check the type of IP routing used while starting collection Verify whether the IP address used in the IP routing is reachable or valid Step 3 Verify that the ftp_user_name and ftp_user_passwd in the coll_info table and the ftp_user_name and ftp_user_passwd in the node_info table of the stratacom database are matching and are valid Defect Information Collect the following information for further analysis e Pmcollector log dump of coll_info table dump of node_info table of stratacom e Related key index entries e Pmcollector Possible alternative workaround Use the atm or LAN IP that is reachable J 10 10 1 7 PM Collection Fails for the Node Log File Shows Polling failed and wait for major retry or Major retry failed and wait f
198. s not synched up wait for it to do so Step3 Ifthe node has synced up check whether cards are populated in the card table If the card table is empty collect the EM logs See J 10 4 Equipment Management Problems page J 40 for more information Step4 Check whether NMServer has cards in its cache This can be done by using the command nmControl on the CTM CLI Step5 If cards are not present in the NMServer cache collect the logs Step6 If cards are present in the NMServer cache use nmClient and verify whether getChildren for node s FDN returns the cards in the nmClient lt pid gt dump file Step7 If cards are present in the NMServer cache open a new GUI and verify whether the cards are shown in it Defect Information Collect the following information for further analysis e topod log ILMITopoc log and NMServer log e nmControl dump e CMSCclient log Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN AppendixJ Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems Possible alternative workarounds 1 Open anew GUI and a new Configuration Center Chassis View Diagnostics Center or Statistics Report 2 Use nmControl and perform Resync Node specifying the node ID J 10 5 4 2 Configuration Center Chassis View Diagnostics Center or Statistics Report Does Not Show a Card Line or Port fora Node Step 1 Step 2 Step 3 Step 4 Ste
199. scovery Issues page J 56 e J 10 5 4 Network Element Discovery Issues page J 59 e J 10 5 5 Configuration Center Chassis View Diagnostics Center or Statistics Report Alarm Issues page J 62 J 10 5 1 CTM Configuration Center Chassis View Diagnostics Center or Statistics Report Basics CTM manages MGX NEs The topology module discovers the nodes and trunks Once the nodes are discovered the EM module syncs up with the nodes to discover the card line port and so on All the discovered nodes trunks and node elements are published by NMServer and are displayed in the Configuration Center Chassis View Diagnostics Center or Statistics Report The Configuration Center Chassis View Diagnostics Center or Statistics Report have the tree view and Inspector View The tree view and Inspector View feed off the data published by NMserver CTM is notified of all subsequent changes in the network through traps Figure J 1 shows the end to end architecture Cisco Transport Manager Release 6 0 User Guide 78 16845 01 ECN AppendixJ Troubleshooting HI J 10 5 Configuration Center Chassis View Diagnostics Center and Statistics Report Problems Figure J 1 CTM End to End Architecture Chassis View Tree View Topology View Alarm List View Presentation Layer Inventory Service Alarm Service Data an
200. sed and the Java Console shows the exception trace information e Choose an application from the Tools menu item e Right click on the selected object from the hierarchical tree and choose the target application Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternative workaround None J 10 7 1 4 An Exception Is Raised when the Configuration Center Launches Another Application When the Configuration Center launches another application using one of the following methods an exception is raised and the Java Console shows the exception trace information e Choose an application from the Tools menu item e Right click on the selected object from the hierarchical tree and choose the target application Defect Information Collect the following information for further analysis e CMSCclient log file from the D Documents and Settings lt username gt log directory e Java Console information e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file from the opt svplus log directory e Configserver log file from the opt svplus log directory Possible alternat
201. sed for querying the objects is set correctly Step4 Check whether the query has been sent to the switch Defect Information Collect the following information for further analysis e CMSClient log file under the D Documents and Settings lt username gt log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None J 10 9 2 3 Node Sync Up Status Are Not Shown in the Diagnostics Center When the Diagnostics Center is launched for a network element it does not show the node sync up status for all the nodes in the network Step 1 Run selnd as svplus and check the status of all the nodes If they are not shown properly then there is probably a problem with the EM process Check the EM logs Step2 Check the node table entries for all the nodes Defect Information Collect the following information for further analysis e Node table entry for the corresponding node e DCServer log file from the log directory Possible alternative workaround None J 10 9 2 4 Node Resync Fails Node resync fails from the Diagnostics Center Step 1 Check whether the node is reachable Step2 Check for any exceptions in the DCServer log file Defect Information Collect the following information for further analysis e Node table entry for the corresponding node e DCServer log file from the log directory e Emd log from the log directory Possible alternative workaround None J 10 9 2 5 P
202. ssful and there is no checksum error in the file From the log file you should able to find out if there is an FTP problem Refer to the cwmftpd log files for more failure information Determine if the problem is caused by shelf generic file or pnni file parsing After you have located where to do a checksum check on the shelf generic file CARD_01_CC CF 17 for example or pnni file PNNI_01_CC CF 17 for example continue to trace the log messages to see whether or not parsing error has occurred If there are no errors you should see the following INFO N17 lt EMC_NodeFsmHandler_c FinishShelf gt Parse opt svplus tmp CARD_01_CC CF 17 successfully Defect information e Collect the ooemc nts sampcomm and cwmftpd log files from the opt svplus log directory e Collect the config upload files from the opt svplus tmp directory Possible alternative workaround None 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting E J 10 4 Equipment Management Problems J 10 4 6 CTM Database Is Inconsistent with Switch Data After Successful Coldstart CTM Server Stop Start or Periodic Resync After a successful node resynchronization triggered by the periodic resync the CTM database is found to be inconsistent with the switch data Step 1 Collect the ooemc log files and all config upload files for the node The ooemc implements a node based cache You can dump and save the cache Step2
203. still problems with starting the Oracle database or the Oracle listener refer to the Oracle documentation or contact Oracle support J 2 6 SNMP Traps Are Not Forwarded from NEs SNMP traps might not be forwarded either because the trap port is already in use or because the NE is not properly configured J 2 7 Trap Port Is Unavailable Step 1 Step 2 The CTM server requires exclusive access to the SNMP trap port to receive SNMP traps from the NE To verify that the standard SNMP trap port number 162 is not being used by another application running on the same Solaris workstation enter the following command netstat a grep 162 If the following line is seen the SNMP trap port is being used by another application 162 Idle If the trap port is being used by another application stop the other application Cisco Transport Manager Release 6 0 User Guide ae E 78 16845 01 Appendix J Troubleshooting J 2 8 Problems with ONS 15530 and ONS 15540 Configuration W J 2 8 Problems with ONS 15530 and ONS 15540 Configuration Step 1 Step 2 ONS 15530 and ONS 15540 NEs must be properly configured to send traps to the server The running configuration on the NE should contain entries similar to the following example snmp server enable traps snmp authentication warmstart snmp server enable traps bgp snmp server enable traps oscp snmp server enable traps config snmp server enable traps entity snmp server enable
204. stname and logging events level are not configured in the running configuration enter the following commands Router conf t Router config domain ipv4 host lt CTM_server_hostname gt lt IP_address gt Router config logging events level informational Router config commit J 8 4 CTM Does Not Receive Traps from the Catalyst 6509 Step 1 Step 2 Step 3 Enter the following command to configure the CTM server as the trap receiver on the Catalyst 6509 cat6509 enable set snmp trap lt CTM_server_IP_address gt lt community_string gt Enter the following command to configure the Catalyst 6509 to enable only specific traps cat6509 enable set snmp trap enable lt specific_trap gt where lt specific_trap gt is auth bridge chassis config entity ippermit module stpx syslog systemp vmps or vtp Enter the following command to configure the Catalyst 6509 to enable all traps cat6509 enable set snmp trap enable all 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HI J 85 CTM Does Not Receive syslog Events from the Catalyst 6509 wy Note The set snmp trap enable all command enables traps only on physical ports For logical ports such as scO or scl you must enable traps separately Go to Step 4 Step4 Enter the following command to enable traps on logical ports cat6509 enable set interface trap lt logical_port_name gt enable For e
205. switch and the switch still returns this error contact the switch DEs Possible alternative workaround None Cisco Transport Manager Release 6 0 User Guide Py106 i 78 16845 01 Appendix J Troubleshooting J 10 9 Diagnostics Center Problems W J 10 9 Diagnostics Center Problems The Diagnostics Center provides different diagnostics operations at the following different network elements levels e Network Diagnostics Retrieve the current sync status and alarm status of all the nodes in the network Support the network node health statistics and manageability checks At the network level only the node manageability check results are supported e Node Diagnostics Retrieve current sync status node alarm status IP address and several other node attributes Retrieve the sync status of all the cards in the node Request a node resync Request VSI partition data and also perform VSI consistency checks on the node Support node health statistics such as SNMP success counts and FTP TFTP success failure counts Perform node manageability checks such as IP reachability and SNMP community string checks e Card Diagnostics Retrieve card level information such as sync status card status and firmware revision Retrieve card CPU usage on cards that support this functionality Retrieve the memory pool usage information on cards that support this functionality Retrieve the memory buffer pool usage Information on cards that
206. t application Check the target application jar file Make sure the target application jar file exists under the opt svplus java jars cwm directory on the target machine Defect Information Collect the following information for further analysis e CMSCclient log file under the D Documents and Settings lt username gt log directory e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt svplus log directory Possible alternative workaround None 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting W J 10 9 Diagnostics Center Problems J 10 9 1 3 An Exception Is Raised when the Diagnostics Center Is Launched When the Diagnostics Center is launched using one of the following methods an exception is raised and the Java Console shows the exception trace information e Choose the target application under the Tools menu item e Right click on the selected object from the hierarchical tree and choose the target application Defect Information Collect the following information for further analysis e Java Console information e CMSCclient log file under the D Documents and Settings lt username gt log directory e Screen snapshots in particular error information messages dialog boxes e Cmsvr log file under the opt svplus log directory e DCServer log file under the opt
207. t does not yet manage unknown node_id for example nts 7275 log old 2 7275 45 03 32 39 WARNING Trap unmanaged 172 28 140 16 Cisco Transport Manager Release 6 0 User Guide P3128 78 16845 01 Appendix J Troubleshooting J 10 13 Miscellaneous Problems W In that case it prints the IP address only and throw it away Defect Information None Possible alternative workarounds None Related key index entries nts log J 10 13 2 Data Inconsistency This section includes the following information e J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 e J 10 13 2 2 Inconsistent Connection Status page J 132 e J 10 13 2 3 Inconsistent Connection Secondary Status page J 132 e J 10 13 2 4 Incomplete Connections page J 133 J 10 13 2 1 Connection Inconsistency Between the Switch and GUI Step 1 Step 2 This section discusses strategies for debugging inconsistencies between the switch and the CTM GUI The inconsistencies discussed in this section are not directly related to provisioning The problems may be caused by coldstart warmstart or provisioning but they are noticed sometime after the fact for example you might look at the system and suddenly realize there is an inconsistency This section gives a list of steps to try to determine more precisely the cause of the inconsistency These steps should be performed in order 1 Verify the node s sync states 2 Isolate the connection incons
208. t on which OSS Manager are listening for In the above example the OSS Manager 120753 Manager is sent traps on port 162 The registration deregistration and other such scripts can be found at opt svplus scripts proxyscripts rtmproxy on a CTM machine These scripts can be used for reference for registration and deregistration with RtmProxy These scripts are meant only for internal use Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN AppendixJ Troubleshooting E J 10 12 Service Agent Problems J 10 12 1 1 Registration with RtmProxy Failed Step 1 Step 2 Step 3 Step 4 Step 5 Manager s SNMP request to register with RtmProxy returned an error Verify that the CTM core is up and running by executing ps ef grep RtmProxy on the CTM machine Verify that the snmpset request was sent to port 8161 Verify that the community string for the snmpset is set to private Verify that the MIB objects that are being set are correct The MIB objects that should be set for the Manager s registration are e stratacom rtm trapsConfig trapConfigTable trapConfigEntry managerRowStatus e mstratacom rtm trapsConfig trapConfigTable trapConfigEntry trapFilterRegisterCategory e stratacom rtm trapsConfig trapConfigTable trapConfigEntry managerPortNumber Verify that the IP address to which you are sending the snmpset request is correct Defect Information Collect the following information for further analysis e R
209. t table information for card family d NO_LINE_TABLE_INFO_FOR_CARD No line table information for card family d FILTER_REQUIRED Filter s is required UNEXPECTED_PARAMSET CM Server does not expect parameter set d UNSUPPORTED_ENDPOINT CM Server does not support end point with card type s vs s interface type s node platform s controller type s for connection type s service type s endtoend type s UNSUPPORTED_PARAMETER_FOR_VISM Unsupported parameter setting on the s end CM Server does not support s s PVC s for VISM cards with firmware revision less than s and card mode set to s PROTECTED CONNECTION s end of the connection is protected Protected connections cannot be deleted UNSUPPORTED_NODE_PLATFORM CM Server does not support node platform d UNSUPPORTED_CONN_TYPE CM Server does not support Connection of type d UNSUPPORTED_SERV_TYPE CM Server does not support Connection type s Service type ss and Endtoend type s between Card type s and s UNSUPPORTED_SERV_TYPE_ONE_END CM Server does not support Connection type s Service type s and Endtoend type s on Card type s UNSUPPORTED_CARD_PATR Connection between card type s and card type s is not supported NO_SUCH_NODE_ID_IN_DB Node id d not found in table node NO_SUCH_NODE_NAME_IN_DB
210. tch or by executing the switch command dsptasks Defect Information If manually opening the FTP session succeeds or the dsptasks command shows less than four entries for FtpdServ but cwmftpd still throws FTPERR_421 then report the problem with cwmftpd log cwmftpd request_log and logs of the process for which files are not getting transferred by cwmftpd Possible alternative workaround None Related key index entries service not available closing control connection 421 session limit reached ftperr_421 J 10 13 3 8 499 Session Limit Reached Step 1 For SWSW MGX8850 Release 4 0 or later the switch responds with 499 Session limit reached queuing lt IP address gt key lt Nodename KXXXXXXX gt when it reached the limit of four FTP sessions The cwmftpd command will throw the exception FTPERR_499 when it tries to connect to a switch which responded with the error 499 Session limit reached queuing lt IP address gt key lt Nodename XXXXXXXX gt Check whether all four sessions have been opened by opening a manual FTP session to the switch or by executing the switch command dsptasks Defect Information If manually opening the FTP session succeeds or the dsptasks command shows less than four entries for FtpdServ but cwmftpd still throws FTPERR_499 then report the problem with cwmftpd log cwmftpd request_log and logs of the process for which files are not getting transferred by cwmftpd Related key index entries 499
211. tch when it actually had generated traps 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 13 Miscellaneous Problems Step 1 See if the node is in OK state If the ntsControl node information says the particular node is not in the OK state see J 10 13 2 1 Connection Inconsistency Between the Switch and GUI page J 129 If the node is indeed in OK state go to step 2 Step2 Check the Trap IP address setting on the switch The Trap IP address has to be the primary IP address of the switch Otherwise nts cannot correlate the trap with the CTM node information and has to discard that trap SSH or Telnet to the switch The primary IP address can be found with dspndparms and then dspifip commands Enter dsptrapip to see the current Trap IP address setting Use enftrapip to correct See J 10 13 1 3 How To Interpret NTS Log page J 128 for more information Defect Information None Possible alternative workaround None Related key index entries nts traps J 10 13 1 3 How To Interpret NTS Log Locating a specific trap from a particular node in the nts log Step 1 Verify trap handling The nts log has information on the traps delivered to a specific client The nts by default keeps 20 old logs in addition to the current one You can form your grep command with the key fields such as node ID trap number client name and pid For example 21359 63 19 24 40 WARNING N42 Tr
212. terface PNNI The ILMITopoc process uses SNMP protocol to discover the physical PNNI network All the discovered nodes are displayed in the MGX NE GUIs Configuration Center Diagnostics Center Statistics Report and Chassis View CTM is notified of all subsequent changes in the network through traps for MGX nodes Once the ILMITopoc process discovers all the nodes it sends all the nodes to the topod process Topod then sends these nodes and trunks to other processes in CTM such as ooemc NMServer nts and so on J 10 2 Discovery Issues at Startup This section includes the following information e J 10 2 1 No Nodes Are Discovered page J 34 e J 10 2 2 Node Name Is Incorrect in the Database page J 35 e J 10 2 3 Node IP Is Incorrect in the Database page J 36 e J 10 2 4 Node Alarm Is Shown Incorrectly in the Database page J 36 e J 10 2 5 Reachable Node Is Shown as Unreachable page J 37 e J 10 2 6 CTM State Is Incorrect page J 37 J 10 2 1 No Nodes Are Discovered If no nodes are in the CTM database complete the following steps Step 1 Verify that the nodes are I P reachable Complete the following substeps a Enter the following CLI commands to retrieve the primary IP address of the node dspndparms dspipif b Enter the following command to check whether CTM has IP connectivity to the node ping lt node_IP_address gt Step2 If CTM has IP connectivity to the nodes verify that the community strings of the gateway nodes are correct
213. ternative workaround Reregister with RtmProxy Cisco Transport Manager Release 6 0 User Guide Pyi24 78 16845 01 Appendix J Troubleshooting J 10 12 Service Agent Problems W J 10 12 1 3 Manager Gets Deregistered Step 1 Step 2 The Manager is deregistered from RtmProxy after running for a while Verify that the keepalive script is running The Manager will automatically get deregistered if no SNMP is done on any of the tables in RtmProxy To keep the Manager registered with RtmProxy run the keepalive script in the background as follows bin sh Usage 0 lt Agent Ip Address gt lt Manager IP address gt if lt 2 then echo Usage Usage exit 1 fi managerRowStatus 1 3 6 1 4 1 351 120 1 1 1 3 managerPortNumber 1 3 6 1 4 1 351 120 1 1 1 2 lastseq 1 3 6 1 4 1 351 120 1 3 0 while true do sleep 60 CMD opt OV bin snmpget cpublic p8161 t3000 r0 1 managerRowStatus 2 lastseq echo CMD CMD done exit 0 Verify that IP reachablity the CTM station is not lost of from the station where the Manager is running Defect Information Collect the following information for further analysis e RtmProxy log and snmpd log Possible alternative workaround Reregister the Manager with RtmProxy J 10 12 1 4 Nodal Community Strings Do Not Work Step 1 Step 2 Snmpwalk using the nodal community strings returns an error Verify that the SNMP request is being issued to port 8161 Verif
214. ters Switch Error Related key index entries cmgrd traffic parameters 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg Appendix J Troubleshooting HZ J 10 8 Connection Management Problems During an add or modify connection request if the local endpoint s remote parameters do not match the remote endpoint s local parameters then this error is indicated by the switch The particular parameter depends on the service type of the connection Note Step 1 Step 2 This error cannot be easily debugged I just has to be checked by a Connection Management DE because the error indicates that the mapping of local to remote and or remote to local has a problem in the value that is being set on the switch Retry the connection from both the CM GUI and Connection Proxy interfaces If the result from both the interfaces is the same this could be a cmgrd bug If the request succeeds from one of the interfaces then this is more likely a client issue Defect Information Collect cmgrd log ConnProxy log and cmsvr log Possible alternative workaround Change some of the traffic parameter values in the connection request and retry the connection J 10 8 4 23 Cmgrd aAddition Modification and Deletion Errors Provisioning a Connection Results in Network Busy try later Switch Error Related key index entries cmgrd network busy During an add or modify connection request if the switch retu
215. the Domain Explorer window choose Administration gt Control Panel and click NE Service b In the NE Poller tab look at the NE Health Poll Interval field The default is 240 seconds If the value is too large there might be a communication delay between CTM and the NEs Under normal conditions the CTM server attempts communication with new NEs for no more than twice the number of seconds specified in the NE Health Poll Interval field 78 16845 01 Cisco Transport Manager Release 6 0 User Guide gy Appendix J Troubleshooting WE J42 Cannot Delete an NE J 4 2 Cannot Delete an NE If an NE cannot be deleted verify that the user logged in as SuperUser or NetworkAdmin not as a Provisioner or Operator Provisioners and Operators cannot delete NEs If the user logged in as Provisioner or Operator restart the CTM client session and log in as SuperUser J 4 3 CTC Fails to Start Step 1 Step 2 Step 3 If CTC cannot be started for a CTC based NE complete the following steps If an error occurs while opening the NE Explorer yet the CTC progress screen and login dialog boxes appear the CTC username or password might be incorrect a Choose Administration gt CTM Users b In the CTM Users table choose Edit gt Modify The Modify CTM User Properties wizard opens c Click Next to move to the CTC Craft User Properties panel and change the CTC username and password to match those configured on the device Click Finish
216. the SNMP upload failed to upload correct data Study the log files to understand the root cause of the inconsistency problem Step2 To determine whether or not a trap has been received and processed there are key words in the ooemc log files that you can grep For example to find out the channel traps from node_id 4 slot 6 vpi 1 and vci 326 you can grep TRAPLIST as shown in the following cwmult60 grep TRAPLIST N4 ooemc grep Channel Trap grep C6 grep vpi 1 vci 326 ooemc10 5760 log 01d 55 5760 10 23 21 12 INFO TRAPLIST N4 Channel Trap 60310 from C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 67 ooemc10 5760 log 01d 55 5760 10 23 21 12 INFO TRAPLIST N4 Channel Trap 60310 lt PROCESSED gt from C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 67 ooemc10 5760 log 0o1d 73 5760 10 23 23 48 INFO TRAPLIST N4 Channel Trap 60310 from C6 B2 L1 P20 Ch299 ifIndex 17176597 vpi 1 vci 326 upCntr 0 vpcFlag 2 operS 1 alarm 66 For other kinds of traps you can use the following key words to supplement TRAPLIST in your grep statement e Port Trap e RscPart Trap e Sve Trap e SonetLn Trap e SctCard Trap e SonetPath Trap e FunMod Trap e LineMod Trap Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EN AppendixJ Troubleshooting E J10 4 Equipment Management Problems e RedCard Trap e TrapMiss Trap
217. the command pkill USR1 dmd and compare the values in the cache with those on the switch for endpoints associated with the error Defect Information If the switch and DMD cache are not in sync you will need the DMD logs message logs DMD cache dump and the EM logs for the node in question You will also need the switch CLI screen shot of the connection in question Possible alternative workaround TIf there is a cache inconsistency the only workaround is a complete cache resync opt svplus tools CacheResync If there is no DMD cache inconsistency it is a normal operational scenario Different connection add parameters should be chosen J 10 8 4 7 Cmgrd Sdbroker Addition Errors Adding a Connection Results in EndPoint Exists Local Remote Both end of the connection already exists Error Step 1 Related key index entries cmgrd sdbroker endpoint exists During an add connection request the cmgrd will request intermediate endpoint vpi vci from sdbroker If the sdbroker incorrectly chooses endpoints that are already in use by the switch the cmgrd will try to provision these endpoints and return the error endpoint exists to the user Determine which of the endpoints already exist on the switch If it is the intermediate endpoints then it is an sdbroker problem a When provisioning hybrid connections the CMGRD requests intermediate endpoints from the sdbroker The sdbroker then requests them from DMD and then sends them back to C
218. tmProxy log and snmpd log Possible alternative workaround None J 10 12 1 2 Not Receiving Any Traps Step 1 Step 2 Step 3 Traps are not being received on the port with which the Manager registered Verify that the registration with RtmProxy went through successfully If the registration did not go through see J 10 12 1 1 Registration with RtmProxy Failed page J 124 Use the following to verify that the port number that RtmProxy is sending traps to is the same as the port number on which the trap receive utility or Manager is listening tballraker29 opt OV bin snmpget d v1 p8161 t 3000 r 0 cpublic 172 28 131 137 stratacom rtm trapsConfig trapConfigTable trapConfigEntry managerPortNumber 172 28 131 158 where 172 28 131 137 is the CTM IP address and 172 28 131 158 is the Manager IP address or IP address of the station where the trap receive utility is running Use the following to verify that the Manager is still registered with RtmProxy tballraker29 opt OV bin snmpget d v1 p8161 t 3000 r 0 cpublic 172 28 131 137 stratacom rtm trapsConfig trapConfigTable trapConfigEntry managerRowStatus 172 28 131 158 where 172 28 131 137 is the CTM IP address and 172 28 131 158 is the Manager IP address or IP address of the station where the trap receive utility is running The SNMP should return 1 Defect Information Collect the following information for further analysis e RtmProxy log and snmpd log Possible al
219. traps fru ctrl snmp server enable traps topology throttle interval 60 snmp server enable traps optical monitor min severity not alarmed snmp server enable traps rf snmp server enable traps aps snmp server enable traps patch snmp server enable traps cdl all snmp server enable traps alarms snmp server enable traps threshold min severity degrade snmp server host 172 20 126 214 version 2c public snmp bgp oscp config entity fru ctrl topology optical rf aps patch cdl alarms threshold In this example the host 172 20 126 214 will receive the specified traps from the NE You can also run the following commands on the server and device a As the root user enter the following command on the server snoop udp port 162 b On the device enter the following command to generate a configuration change wr mem If the device is configured correctly the snoop command reports a configuration change trap J 2 9 Performance Monitoring Data Is Not Displayed Step 1 Step 2 Step 3 Step 4 Step 5 Use the Domain Explorer NE Properties pane to see if PM collection is disabled for the selected NE Choose Administration gt Control Panel and expand PM Service Select the NE type to see if the service status is Active or Not Active Choose Administration gt Audit Log to see if PM was ignored for the selected NE Choose Administration gt Control Panel and click PM Service Look at the PM Data Storage field Choose Optimized to save only
220. twork environment requires other parameters in the same files to be tuned for correct functioning of CTM J 10 4 3 Node Mode Remains in 1 Step 1 Step 2 The node has been discovered by the network topology process in CTM its node mode has been changed from 1 to 1 in node table entry and it stays in mode 1 for a long period of time If CTM stays in mode 1 for a long time after CTM core has been started you need to check the trap manager on the switch See J 10 4 2 Network Setup and Configuration Required for OOEMC Sync Up Process page J 41 for trap manager and trap IP setup If step 1 is not an issue then check for rtm link up message that nts sends to EMD The ooemce will start node resync once EMD notifies the ooemc that the node is active Therefore the second step for debugging this mode 1 problem is to see whether rtm link up message has been received by EMD and whether it has been forwarded to the ooemc For example to find the rtm link up messages received by EMD for node with ID 9 issue the following command grep RTM_LINK_UP emd grep Node id 9 Once the location of the rtm link up message is found in the log view the log file for more information on notification to ooemc 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting E J 10 4 Equipment Management Problems Step3 Ifthe rtm link up message for the node is found and the log indicates that notification to
221. ty Shown on Platform Step 1 Step 2 Step 3 Severity of a network element in tree view does not match the severity the switch is showing for the same network element Verify whether the customer is comparing the correct severity icon a There are two severities associated with every object in the tree view the aggregate severity and the self severity The aggregate severity is on the left the self severity is on the right Since the switch does aggregation of many faults the customer should compare the aggregate severity of the object in the tree view with the severity of the switch b If you are looking at the correct severity icon and there is still a discrepancy between the severities continue to step 2 Verify whether CTM has synced up with the node a Log in to the CTM server and type selnd b Verify that the node mode in question is 3 If the mode is 3 verify whether the discrepancy is caused by aggregated connection alarms NMServer does more aggregation of alarms than the platform For instance NMServer aggregates connection alarms up the port and the switch does not Therefore if the tree view displays a higher severity then the switch it may be caused by one or more aggregated port alarms a Right click the NE in the tree view and choose show alarms All of the alarms for this NE and its children should appear in the alarm list b Filter on the alarms that have greater severity then what the platform is showing
222. uch as Raw Statistics and Top Utilization are shown in table format and performance data reports are shown in graph and table formats J 10 11 2 You Collected Data but You See No Data Available Step 1 Verify that the data has been collected and parsed Step2 Verify that the corresponding table has data for the specified time period Defect Information Collect the following information for further analysis e opt svplus log srtserver log Possible alternative workaround None J 10 11 3 You Generated a Report but Do Not See Data for a Long Time If you are trying to generate reports and you don t see any data for a long time and you only see the message Retrieving data check srtserver log for any error messages Defect Information Collect the following information for further analysis e opt svplus log srtserver log Possible alternative workaround None J 10 11 4 You See FDNs for Other Entities when You Generate a Report When you generate a raw report for trunk data on one card you might also see trunk data from a different card This is because some of the tables do not have slot information and hence you get data only with node information This is not an error You will see extra data Defect Information Collect the following information for further analysis e opt svplus log srtserver log Possible alternative workaround None J 10 11 5 You See the Wrong FDN for a Raw Report The FDN reported is contains either
223. uit wizard to enter a unique name for each monitor circuit J 2 15 NE Model Type Appears as Unknown If an in service NE is added to the Domain Explorer but the model type appears as unknown the software version of the NE might not be prepopulated in the database In other words CTM cannot match the NE with a recognizable version Use the NE Software Table to add the NE software version string to the CTM database See 4 3 5 Viewing Software Versions and Restarting the NE with a New Software Image page 4 35 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting W J 2 16 Cannot Copy CTC Binary to the CTM Server J 2 16 Cannot Copy CTC Binary to the CTM Server If a CTC binary fails to be copied to the CTM server the lt CTM_install_directory gt cms directory might be missing or write protected If the directory is missing create a cms directory with write access permissions Otherwise change the permissions of the existing cms directory to allow write access J 2 17 Memory Backup Memory Restore or Software Download Fails Step 1 Step 2 Step 3 In the Domain Explorer window choose Administration gt Job Monitor The Job Monitor table shows the status of the operation The reason for the failure is shown in the Additional Information column Return to the Domain Explorer window and choose Administration gt Error Log The Error Log table shows information about the backup m
224. umulated in the opt svplus spool directory e Check the file_err_log table for any file collection failure Cisco Transport Manager Release 6 0 User Guide 78 16845 01 E Appendix J Troubleshooting HZ J 10 10 Performance Management Collection and Parsing Problems e Check the collsvr_err_log table for any failure during start stop collection e Check the coll_err_log table for entire collection server errors e Check opt svplus cache scm scmcollout log for a completed stats file request e Check opt svplus cache scm scmcollin log to verify whether or not stats file has been completed e Check opt svplus cache scm scmcollout log for stats file collection errors J 10 10 1 2 PM Collection Fails for the Node Log File Shows Node not discovered This can happen when the node is not in mode 3 Step 1 Check the node_info table by executing the following command echo select node_not_discovered from node_info where node_id lt node_id gt and slot lt slot gt dbaccess If node_not_discovered is set to 1 it means that the node has not been discovered Step2 Check the state of the node by executing the following command echo select from node where node_name lt node name gt dbaccess Step3 Check the mode of the node in the output the mode must be 3 Defect Information Collect the following information for further analysis e pmcollector log output of selnd or output of echo select from nod
225. us e Check whether target switch CTM is reachable e Check for debug level If logs are not giving detailed information debug level can be increased by editing svplus config cwmftpd conf e Set the config parameter LOG_LEVEL to 7 to get detailed logs Related key index entries FTP J 10 13 3 3 FTP Username and Password The command cwmftpd uses the node_info table of the stratacom database to retrieve the FTP username and password For SCM requests scmcollsvr sends the password along with the request Related key index entries FTP username password Cisco Transport Manager Release 6 0 User Guide Ea 78 16845 01 Appendix J Troubleshooting J 10 13 Miscellaneous Problems W J 10 13 3 4 cwmftpd Files Are Not Transferred Due to Wrong Username Password Files are not transferred between CTM and the switch or between CTMs due to a wrong username password The files are not FTP d between the switch and CTM or between CTMs due to a wrong FTP username password Step 1 Check whether the FTP username and password are correct Step2 Check whether cwmftpd log is having an exception like the following where the FTP username and password are wrong 22589 204248 10 12 27 ERR FtpException 3 LOGIN_FAILED Login failed login host 172 23 30 111 user superuser Defect Information If username password are correct but LOGIN_FAILED is still shown in logs collect cwmftpd log cwmftpd request_log and logs of the pro
226. wrong values or junk values Cisco Transport Manager Release 6 0 User Guide La 78 16845 01 Appendix J Troubleshooting This is an error on the server while forming FDN Defect Information Collect the following information for further analysis e opt svplus log srtserver log Possible alternative workaround None J 10 11 6 Utilization Report Value Is Greater than 100 The Utilization report value is greater than 100 This is an error Defect Information Collect the following information for further analysis e opt svplus log srtserver log Possible alternative workaround None J 10 12 Service Agent Problems J 10 12 Service Agent Problems RtmProxy is the only component in the Service Agent that is used in CTM The Service Agent processes your requests through SNMP convey them to the switch agent and finally returns the result to you J 10 12 1 RtmProxy RtmProxy is CTM s northbound SNMP interface providing traps from all switches to customer applications It is an interface with which the customer application registers for traps The customer application uses SNMP to register with RtmProxy Figure J 7 shows the flow of information Figure J 7 RTMProxy Event Port 9863 Master Port 8161 RtmProx NTS y Port 162 Databroker The OSS Manager send an SNMP request to port 8161 of the CTM machine In this request the OSS Manager specify the por
227. xample to enable traps on the logical interface sc0 enter cat6509 enable set interface trap sc0 enable J 8 5 CTM Does Not Receive syslog Events from the Catalyst 6509 The Catalyst 6509 sends syslog traps only if the trap severity is both e Less than or equal to facility severity which is set with the set logging level command e Less than or equal to history severity which is set with the set logging history severity command It is recommended that you set the history severity to debugging 7 The show logging command shows both the history severity and the severity for all facilities For example in the following output the history severity is 7 and facility severity is 5 Syslog events of severity X will be triggered from the device if X is less than or equal to 5 and less than or equal to 7 Cat6509 enable show logging Trace logging disabled Logging buffer size 500 timestamp option enabled Logging history size ul severity debugging 7 Logging console enabled Logging telnet enabled Logging server enabled 10 77 56 151 172 19 75 86 server facility SYSLOG server severity debugging 7 Current Logging Session enabled Callhome Functionality disabled Callhome Severity LOG_CRIT 2 SMTP servers are not configured for callhome messages Destination addresses are not configured for callhome messages From Address is not configured for callhome messages Reply To Address is not configured for ca
228. xml configcenter CCC directory Cisco Transport Manager Release 6 0 User Guide yso 78 16845 01 Appendix J Troubleshooting J 10 7 Configuration Center Management W Step3 Ifthe file is not available contact CTM engineers Step4 Open the document you got in step 2 and do the following e Check the Access field of the MIB object in the XML file e Check the Access field in the opt svplus mibs directory If the access field from XML file and MIB definition are not same contact CTM engineers Step5 Check the Range field of the MIB object in the XML file and MIB definition If they are not the same contact CTM engineers Step6 Perform an SNMP SET operation on the MIB object with the same value shown in the GUI and appropriate community string as displayed in opt svplus log configserver log using HP OV A syntax of the SNMP SET operation is shown as follows opt OV bin snmpset c private nodeName MIBVariable MIBDataType MIBValue Step7 If the SNMP SET operation succeeds contact CTM engineers with appropriate defect information Step8 SSH or Telnet to the switch and try to configure the same parameter with the value given in the GUI and SNMP tool If the CLI does not report errors contact CTM engineers with defect information Defect Information Collect the following information for further analysis e Complete screen snapshots for the investigative commands used for debugging this issue e Configserver log Possible alt
229. y for the node with the correct IP address complete the following substeps a Enter the following command to dump the cache of NMServer topod and ILMITopoc kill USR1 signal lt process gt b Check the data collected in the opt svplus log directory and verify whether the IP address is correct in the cache dumps If the IP address is correct open a new instance of the GUI and check whether the IP address is correct in that GUI Cisco Transport Manager Release 6 0 User Guide 78 16845 01 EEN AppendixJ Troubleshooting E J 10 4 Equipment Management Problems Step2 If the node table does not have an entry for the node with the correct IP address complete the following substeps a Check the ILMItopoc log and verify whether it received a 60007 and 70202 trap b If you see the trap in the log verify that ILMITopoc updates its cache based on the trap c Ifyou do not see the traps in the ILMITopoc log see J 10 13 1 NTS page J 127 Defect information Collect the following information for further analysis e ILMITopoc log topod log NMServer log and nts log e Dump outputs of ILMITopoc topod and NMServer Enter the following command kill USR1 signal lt process gt e Collect the output of the switch CLI selnd and dbnds J 10 4 Equipment Management Problems This section describes troubleshooting procedures for equipment management problems on the Cisco MGX devices J 10 4 1 Equipment Management Declares Suc
230. y that the community string is correct Defect Information Collect the following information for further analysis e RtmProxy log and snmpd log Possible alternative workaround None J 10 12 2 Audit Trail Logging Mechanism Problems Audit trail logging provides CTM with the ability to record user activities across different modules in a persistent file All CTM Java front GUI applications send information about user activities to the Audit Logger server through the CORBA interface The Audit Logger server then writes the data into the persistent log file 78 16845 01 Cisco Transport Manager Release 6 0 User Guide jg AppendixJ Troubleshooting HM J 10 13 Miscellaneous Problems This section includes the following information e J 10 12 2 1 AuditLogger conf Usage page J 126 e J 10 12 2 2 Audit Trail Log File Naming Convention page J 126 e J 10 12 2 3 When You Open a Dialog Box in a CTM GUI There Is No Record in the Audit Trail Log File page J 126 J 10 12 2 1 AuditLogger conf Usage The following can be configured in the configuration file e Location of the log file The default is opt svplus log AL e Number of days By defining the number of days for the audit trail log files to be kept all other obsoleted audit trail logs will be deleted automatically e The name of the user group whose member can read the audit trail log files lIt is a standard UNIX user group J 10 12 2 2 Audit Trail Log File Nam
231. yed in the frame is read only If there are any editable fields and the Refresh button is disabled contact CTM engineers Defect Information Collect the following information for further analysis e Complete screen snapshots for the investigative commands used for debugging this issue e Configserver log Possible alternative workaround For creating or deleting elements use either the SNMP SET operation on the CTM machine or CLI commands on the switch J 10 7 2 4 SNMP Timeout Error Step 1 This error occurs when you try to set or get values to or from SNMPCOMM and the values look like the following Check the sync up linkO linkl status of the Node in the inspectorview If the status is Unmanaged Unreachable Failed in Sync or Partially Synced check the connectivity of the node using the ping command Telnet or any other utility Defect Information Collect the following information for further analysis e Complete screen snapshots for the investigative commands used for debugging this issue e Configserver log Possible alternative workaround None J 10 7 2 5 SNMP Set Error Step 1 Step 2 This error occurs when the SNMP SET operation is done on an uneditable MIB object or when you use invalid values Open the opt svplus log configserver log file Check for the last part of the document name For example if the document name is AAA BBB CCC xml then the document AAA BBB CCC xml can be found in the opt svplus

Cisco Systems 78-16845-01 Network Router User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents