HP rp7410 Troubleshooting
Contents
1. psb oselogs parstatus parstatus c CELL V vparstatus vparstatus v vparstatus e output of ps ef and top urrent and last boot logfile etc re log RST event log file var opt resmon log rst log rst log sas txt sautil txt service_guard txt sfinconfig txt sfim_event log sfimlog txt shutdownlog spmgr out sw_inventory xml swlist txt sasdiag like information about SAS disks and cards connected to LSI SAS HBAs improved own script nformation about Smartarray raid controllers mcelview v output if Service Guard cmcld is running utput of sfmconfig w q system fault manager config var opt sfm log event log ystem fault manager logs similar to EMS event log etc shutdownlog VA diskarray infos sed by mission critical HP support for patch analysis utput of swlist l bundle swlist l product and swiist l fileset a state grep inst syslog log OLDsyslog log last 2048 MB each use f option to capture the complete files s information from stand krs system krs ontains f e SAS dynamic hardware path assignments only collected if SAS is used fsan Output of tddiag if fibre channel cards exist for this driver Directory with tombstone files HPMC or MCA Also crashinfo outputs and EFI binary MCA files Directory with binary CPEs and CMCs from ogExtract comes with SysFltMgr gzipped output of nm stand vmunix Needed for software support for extended analysis of MCAs hen no creash dump i2. C7000 OA via ssh without oamp only show all is captured To capture only MP logs data from blades in bays 1 and 2 via the OA getsysinfo sh mp mpopt ssh oamp 12 List of all HPMPL options for use with mpopt getsysinfo sh mp mpopt lt list of options gt HPMPL Options max number of MCA logs to capture for SD2 5 oamp bays collect MP logs via Onboard Administrator OA P ibay bays is by default 12345678 all blade bays oavc user pass collect VC data via OA login req collect VC connect support dump and save it to the FTP server specified by the FTPURL No subdir or filename can be specified on the FTP server zat V print debug info v v logs all incoming input How to capture additional data from C Class or Superdome 2 OA With mp option see previous chapter on page 5 ksh getsysinfo all sh mp With built in oa option no hpmpl jar or sd2collect pl needed Getsysinfo sh can capture the show all output from Onboard Administrators of C7000 C3000 and Superdome 2 Onboard Administrators by connecting with the HP UX telnet to the OA You see the whole output from the telnet session In case of problems press CTRL C The output will go into a file show_all hostname txt in the sysinfo tgz With sd2collect pl With the sd2 option getsysinfo sh automatically calls the perl script sd2collect pl which have to be in the same directory as getsysinfo sh unless you use the
3. SNAM 4 J Fl D XBACKDATE 180 for some logfiles get information less than 180 days old UNCRASHINFO 0 l run crashinfo on last crash 2 run crashinfo for all crashes SAN 0 do not capture fcddiag by default NET 0 do not capture nettl and netstat logs by default diskjn 0 do not query all disks with diskinfo or scsimgr by default I P MPJAVA 0 do not run jar file by default only with mp option GETOA 0 get OA info via telnet SD2 0 do not run be default SGINFO 0 do not run sginfo by default MAXFILESIZE 1024 Maximum size of some logfile in kbytes GETFIRSTLINES 2000 if logfiles are truncated also the first lines are saved M R NITDATA 0 do not capture initdata stuff by default ANI 0 do not use print manifest by default SASOPTION 1 0 run sasmgr in any case XPINFO 0 1 run XPINFO NOHIST 0 vweb b history by default set to 1 if it makes problems NMVMUNIX 0 do not collect symbols from vmunix by default ESCSIDIAG 0 do not capture escsi diag output SWINFO 0 higher values capture more data for SW suport OSELOGS 1 capturing of var opt psb oselogs T better do not c
4. ashinfo ksh getsysinfo sh c By default only the last dump is analyzed To analyze all dumps younger than 6 months you can use ksh getsysinfo sh c all Attention Analyzing dumps may be a time and storage var adm crash consuming task depending on the size of the dump Other Examples are ksh getsysinfo sh sw sw for software collections also collects dump ksh getsysinfo sh sw c 0 disable dump collection when sw is set ksh getsysinfo sh c 2 collect the last two dumps yunger than 6 months ksh getsysinfo sh c all h _ collect really all dumps also older 180 days How can modify the temporary directory Simply set the environment variable TMPDIR which is used by mktemp to specify the location of temporary files and directories For example TMPDIR var tmp ksh getsysinfo sh How to run collect additional commands files with the x option Sometimes other information may be needed than collected by getsysinfo sh Instead of modifying the script see next chapter the x option can be used to collect additional files or to run additional commands Of course you should use this option with caution x lt file s gt to capture additional file s By specifying one or more files full path needed behind the x option you can force getsysinfo sh to capture them as well output filename will be addfile lt pathname gt all are replaced by _ Examples ksh getsysinfo sh x var a
5. dm syslog mail log gt addfile__ var_adm_syslog_mail log ksh getsysinfo sh x var adm sw sw log X exec command to run additional commands By putting an exec before the x argument getsysinfo sh will run the command and save the output to addemds log ATTENTION Make sure you only use safe commands for that and it is highly recommended putting the command into single quotation marks to avoid any shell replacements with the command you intend Better do not use wild cards it is risky Examples getsysinfo sh x exec ls 1R var adm sw getsysinfo sh x exec opt fcms bin fcdutil x exec usr sbin sar d 5 5 Can I modify the script It is safe to modify the parameters at the beginning of the script usr bin ksh Revision 9 235 Sby S Stechemesser HPS Date 10 22 20145 the below construct is needed no line in getsysinfo sh is longer than 80 chars for cut paste READMELINK https h20565 www2 hp com portal site hpsc template PAGE public READMELINK SREADMELINK kb docDisplay docId emr_na c03800758 e G xport LANG C ETSYSINFOVERSION 9 23 CKSUM 3194520570 use the TMPDIR environment variable if needed s man mktemp TMPDIR S TMPDIR tmp set TMPDIR to tmp if not set NAMEBASE sysinfo default output name prefix add hostname timestamp by default NAME NAMEBASE hostname date Y m d H M DIR STMPDIR
6. embedded versions getsysinfo sd2 sh or getsysinfo ms sh The output will go into a subdirectory sd2collect in the sysinfo tgz archive getsysinfo sh sd2 sd2collect pl will ask for login data to OA getsysinfo sh sd2 sd2opt s get less data to save time getsysinfo sh sd2 sd2opt a get all data output may be huge getsysinfo sh sd2 sd2opt d MM DD YY get only data from after that date getsysinfo sh sd2 sd2opt hostname user password Note by default sd2collect only captures data from the last 30 days unless getsysinfo sh option h or a was given How to capture Service Guard related information By giving either the sw or sginfo option the script sginfo is run see https h20628 www2 hp com km ext content webapp document docId emr_na c03661098 It sginfo was given the output filename defaults to sysinfo sginfo_host_date tgz Without that option getsysinfo sh still collects by default the cmclview v information into the file service_guard txt How to automatically run crashinfo ksh getsysinfo sh c runs opt sfm tools crashinfo on the latest dump if present Since version 9 18 getsysinfo sh can call the crashinfo tool which comes with System Fault Manager and usually is located in opt sfm tools You can also specify another crashinfo location with the CRASHINFO environment variable for example CRASHINFO tmp cr
7. g Includes h d net sw san xp p The diag option should only be used on special requests when software problems with STM EMS or System Fault Manager have to be troubleshot The output may be huge How long does a script run take Of course this depends on the size of the system Normally it should only take some minutes But in some cases an ioscan on a full size Superdome can already take 20 minutes oe You can stop the script at any time by pressing CTRL C and then restart it with the s option which should run much faster but collects less information How big is the output Usually between 200 kB and 1 MB depending on the size of the system If the f or diag options are set the output may be extremely huge depending on the size of the log files on your system Can the script crash the server The script is optimized not to harm the system even in case of hardware problems Some intelligence is built in to avoid known problems with commands like cstm sasmgr or femsutil It only happened in very rare cases that one of the used commands triggered a crash on systems with a severe hardware or software problem How can give feedback to improve the script or report problems Send a mail to getsysinfo groups hp com or give feedback to the HP support engineer who sent you this script It would be nice to attach the getsysinfo log file obtain by ksh getsysinfo sh 2 gt amp 1 tee getsysinfo l
8. getsysinfo sh current version 9 23 21 10 2014 Latest revision of this document is also available on HP Support Center getsysinfo sh is a data collection shell ksh script for HP UX gt 10 20 used by Hewlett Packard hardware support for Business Critical Server BCS It collects diagnostic data for hardware troubleshooting from a running HP 9000 or Integrity Server The script is optimized for small output and reliability even on systems with bad hardware It not only collects various log files but also runs some common commands to obtain additional troubleshooting information The result is a gzip ed tar archive that can be viewed with WinZIP tar gzip or other tools The default output filename can be changed with the n option is tmp sysinfo_hostname_timestamp tgz Some of the raw binary log files in the sysinfo tgz archive need special viewers and can only be analyzed by HP support New since Version 9 0 is the built in data collection from Onboard Administrators option oa via telnet In addition to this getsysinfo sh can now call the additional data collectors hpmpl 286 jar java tool to collect data from iLO Management Processors MP option mp sd2collect pl perl script for data collection from Superdome 2 OA option sd2 sginfo shell script to obtain Service Guard related information Note that these additional tools have to be present in the local directory unless you have one of the embedded getsysinfo
9. ginii e a E a a diese 2 How lo g does as ript run tak Tisnisorecncnreciris sei pan iaaieo riu arestio rira ake ant 3 PLOW 1S Ele AIT aoe oie ces cea E A E E E E 3 Ca the SCript crasi the SECVer 7 ssccvncca escyvazetalaysadinenahsanst wees dnd gnastanvassdauunehayativer sd nowy T aA e iaei 3 How can I give feedback to improve the script or report problems ceceeseeeseeeteeeeeneeeeeee 3 What is NOT collected by the script 2 as sucescssacechccnvenczss hdapuasnceach leant ssaeiseataataadeeueeenaaavnanediealacs 3 Whatis collected misarer en pote Bleed TE e EA E A eis ois cr uit ch 4 How to capture additional data from an MP or OA with hpmpl jar esesessesseeeseesssesssseeseseese 5 How to capture additional data from C Class or Superdome 2 OA eeceeeeseeeeteeeeteeesneeenes 6 How to capture Service Guard related information cceecccessceeseceseeeeeeeceeeeeeeesneeceneeeeenens 7 How to automatically run crashinfo 7 cic csaesgcnasxdashedaeson lave cohdcisers cadetszauchacs teaulacstouetandaneaddehcies 7 How can I modify the temporary directory 25 ccietes sapesmdics acradenceh wits aiicameeute aisiudenieetiestet 7 How to run collect additional commands files with the x option ssssssessssssseseesssrssssesssesesse 7 Cart DaT TAi DA TEA ET D1 AE AE EA EA EE AAE E EEE A 8 How to check if I have the official getsysinfo sh script or a modified ssesssssesssessssseseesseese 9 How to transfer data via ASCII termina
10. hange parameters below here unless you know what you are doing Integrity check Note when the script is modified and the CKSUM line is not adjusted properly a warning about the modification will be printed and the script will pause for 15 seconds when started How to check if have the official getsysinfo sh script or a modified Simply run cksum getsysinfo sh and verify with the below values The cksum outputs of getsysinfo sh version 9 23 official revisions are 2176148563 54116 getsysinfo sh 3089166024 230078 getsysinfo all sh 164757447 88606 getsysinfo sd2 sh 4145209390 170673 getsysinfo mp sh 351488249 205162 getsysinfo ms sh How to transfer data via ASCII terminal applications with cut amp paste getsysinfo sh can be transferred via cut amp paste to the target server via a terminal application like putty No line is longer than 80 chars cat gt tmp getsysinfo sh then paste the script into your terminal application CTRL D You can use the u option to transfer the result back to you PC ksh tmp getsysinfo sh u now start logging to a file on your terminal application cat tmp sysinfo uu stop logging You can then unpack the file on a Unix Server by deleting everything before begin with uudecode logfile and you have a gzipped tar archive that can be unpacked with WinZIP or gunzip and tar Can I run the script regularly from cron This is not recommended Although t
11. he script is not known to cause problems it was not written for collecting configuration data but to troubleshoot hardware problems A lot of commands are used which directly interact with internal and external hardware LVM and other system resources which may cause strange logs in syslog log or device logs if running too often
12. ion Several diagnostic related information 1l R var stm var psb var opt sfm var opt resmon output of diskinfo v of all disks only with d var opt resmon log event log files younger than 90 days san Output of felpdiag or feddiag or fcocdiag if fibre channel cards exist for this driver ice grep output from femsutil Fibre Channel Infos tdlist fedlist hassis Error Log Files on newer systems from var stm logs os younger than 90 days og of STDOUT and STDERR of the getsysinfo sh run onsole log of virtual machines captured on the VSP not if s is set nformation about HP virtual machines and vpars gt 6 0 output of icod_stat icod_stat s ver gt 6 or icod_stat u lt 6 ioscan fn s ioscan fnk or HPUX 11 31 ioscan fN and also ioscan m dsf ioscan m lun ioscan m hwpath or Superdome 2 ioscan m resourcepath only for HPUX 11 31 output of ioscan fnk while ioscan txt contains output of ioscan fN output of irdiag v for Raid4Si cards lanscan nwmgr lanadmin netstat outputs and netconf ith net additional files for hp_apa config urrent raw logfile from the online diagnostics has to be decoded with logtool stm utput of machinfo IA64 system information emory error log file output of logtool commands vd and vda 320 SCSI interfaces output of mptutil and mptconfig output of netfmt f var adm nettl LOGO000 only with net option utput of rad q or olrad q aw 11 31 diagnostic logfiles from var opt
13. l applications with cut amp paste sssssseseeseeeesseeeeseeees 9 Can I run the script regularly from cron seessesseessessesseesressresetssessesetssresresstssresressessresresseessse 9 Why should I use the script It simply saves time if the support engineer gets all information from the OS needed for troubleshooting in one step And running a script is much more easy than collecting everything manually What options should I use The script should be run as root with ksh without any option by default It then collects all data necessary to troubleshoot recent hardware problems ksh getsysinfo sh Note The script is only tested with Korn Shell usr bin ksh and also works with the Posix default shell for the root user sbin sh Other shells like csh or bash may cause problems To also capture MP or OA data C3000 C7000 Superdome 2 via network ksh getsysinfo all sh mp java gt 1 5 needed only for getsysinfo mp sh getsysinfo all sh or if hpmpl jar is in the same dir If one of the commands in the script hangs usually ioscan you can abort the current script run CTRL C and run ksh getsysinfo sh s Only if you want to collect more data than usual you can add the options h also logfiles files older than 180 days are collected f do not truncate large files to 1 MB sw capture also software related information crashdumps sginfo etc a capture all information except dia
14. og What is NOT collected by the script etc passwd or any other password or secret containing files are NOT collected MP iLO logs not collected unless you use the mp option and the external hpmpl jar user or application data is not collected What is collected The tmp sysinfo hostname _date shar tgz uu file contains the following Files bdf_swapinfo txt bootconf txt ccerrlog config lvm crash txt crashinfo crash X html cstm info cprop txt event log fed fclp fcoc diag_date txt gz femsutil out fpl log getsysinfo log hpvmconsole vmname log hpvmstatus txt icod txt icod log ioscan txt ioscan_fnk txt irdiag out lanconfig nettl txt olar txt oselogs tgz rc log old Output of bdf and swapinfo tam vinboot v lifls of bootdisks AUTO file s setboot 1l stand vmunix SP Chassis codes from var stm logs os binary file only PA Risc Servers general information uname model etc ernel configuration LVM relevant info lvmtab fstab vgdisplay amp Ivlnboot v not with s nformation about crash config and var adm crash directory iith option c if opt sfm tools crashinfo is present and a dump is in var adm crash the dump is analyzed with crashinfo H c TM information logout from all devices output of System Management Homepage related information cprop details a nformation needed for diag software troubleshooting only with diag opt
15. s written system krs ltddiag_date txt gz itombstones tombstones sfm_sal vmunix nm gz vxvm txt nformation about Veritas Volumemanager VXVM config xpinfo out utput of xpinfo for XP diskarrays How to capture additional data from an MP or OA with hpmpl jar At least java 1 5 is needed for this feature If you do not have getsysinfo mp sh or getsysinfo all sh which have the hpmpl tool embedded the hpmpl 286 jar file needs to be in the same directory as getsysinfo sh Note getsysinfo sh is trying to guess the MP IP or OA address with the cprop utility If successful it will connect automatically to this IP getsysinfo sh mp getsysinfo will ask for login data to MP or OA getsysinfo sh mp mpopt ssh access MP via ssh default for i2 i4 server getsysinfo sh mp mpopt h hostname u user p password IMPORTANT hpmpl will ask for login and password to MP or OA during the getsysinfo sh run Note hpmpl also supports capturing logs via the Onboard Administrator OA of C Class Blade enclosures and Superdome 2 Servers It will automatically if no mpopt option is set detect if an enclosure is present and by default connect via ssh to the OA instead of the iLO For 12 and i4 blades it will then try capturing the MP logs from the Onboard Administrator Enhanced example give OA IP and login when hpmpl asks getsysinfo sh mp mpopt ssh oamp capture OA ilO MP data from blades and
16. versions getsysinfo sd2 sh getsysinfo mp sh getsysinfo all sh which have one or all tools embedded as a shar archive Syntax run as root ksh getsysinfo sh options default output will be written to tmp sysinfo_hostname_timestamp tgz Options no option is needed unless HP support asks to use a specific option a get all information except diag h f san xp p net d S secure use only non hanging script hangs commands h get all files from var tombstones directory instead only those younger than 180 days Collect SFM history and SAL records f do not truncate some of the logfiles to 1024 kB f regexp filter syslog files with regexp Example inetd ftpd sh make shell archive tmp sysinfo_ hostname_timestamp shar ascii u uuencoded tgz file gt tmp sysinfo hostname timestamp uu ascii n FN explicitly set output filename to FN d bdt diskinfo from all disks scsimgr get info on 11 31 san capture fcddiag or td fclp fcocdiag net capture netfmt var adm nett1 LOG000 p include information from print manifest diag capture additional data for diagnostics SW troubleshooting diag diag capture more diag data Huge Output may be the result sas force running of sasmgr sw capture additional information needed for software support c all N run crashinfo on last N or all crashdumps lt 180 da
17. ys xp get information about XP diskarrays with xpinfo may take long oa capture show all etc from onboard administrator OA via telnet x file capture file as additional information x exec command log the output of command to file addcmds log Only if external script hpmpl jar is in same dir or with getsysinfo mp sh or getsysinfo all sh mp run external mp collection java program hpmpl 286 jar mpopt options set additional options for hpmpl 286 jar Only if external script sd2collect pl is in same dir or with getsysinfo sd2 sh or getsysinfo all sh sd2 capture Superdome 2 or C 37 000 OA logs with external sd2collect pl sd2Zopt options set additional options for sd2collect pl Only if external script sd2collect pl is in same dir or with getsysinfo sd2 sh or getsysinfo all sh sginfo capture Service Guard logs with external script sginfo The option s prevents hanging commands like ioscan f diskinfo vgdisplay etc hanging means the command may hang due to a hardware problem NOT the server Frequently Asked Questions getsysinfo sh current version 9 23 211 0 2014 wicca casero ess ti sivas aacdaies lt uae een Ae 1 SDAA UU exe cis K010 DO RAAE AE E A E E AA A EEEE EE 1 Frequently Asked Questi Ons sneer iiien cian E dies yuly cured iad congas wane seen saree 2 Why should us the script V4 cceveevocrctoviocarcsnierasiesvtaresevnnat a a EE E aa a ia 2 What optionsshould Tuse Pa
Download Pdf Manuals
Related Search