Home

Windows Internals and Advanced Troubleshooting

1. Include Exclude Cancel Highlight Log Reads Log Other Log Writes Log Successes Log Errors iv M Iv M Regmon Lab 1 Run Regmon Highlight Notepad exe Run Notepad Change font to Times New Roman Exit Go back to Regmon Stop logging Find line showing storing of font name in registry Hint search for times OTOL oN Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Using Regmon Identify missing Registry keys Search for status NOTFOUND Troubleshoot permission problems Search for status ACCESS DENIED Find incorrect or corrupt data Examine values read and or written in Other column Example Problem Internet Explorer failed to start Internet Connection Wizard x Cannot load the Internet Configuration library ICFGNT DLL The following error occurred The specified module could not be found Solution Looked backwards from end of Regmon log Last queries were to HKCU Software Microsoft Internet Connection Wizard Looked here and found a single value Completed set to 0 Compared to other users theirs was 1 Set this manually to 1 and problem went awa Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Regmon Applications If you suspect registry data is causing
2. Role Of Subsystem Components API DLLs Export the APIs defined by the subsystem Implement them by calling Windows native services or by asking the subsystem process to do the work Subsystem process Maintains global state of subsystem Implements a few APIs that require subsystem wide state changes Processes and threads created under a subsystem Drive letters Window management for apps with no window code of their own character mode apps Handle and object tables for subsystem specific objects win32k Sys Implements Win32 User and GDI functions calls routines in GDI drivers Also used by Posix and OS 2 subsystems to access the display Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Symmetric Multiprocessing SMP No master processor All the processors share just one memory space Interrupts can be serviced on any processor F Any processor can cause another processor to reschedule what it s running Current implementation supports up to 32 CPUs 64 bit edition is 64 internally Not an architectural limit just implementation Maximum of CPUs stored in registry HKLM System CurrentControlSet Control Session Manager LicensedProcessors Memory SMP SMP Scalability Scalability is a function of parallelization and resource contention Can t make a general statement
3. But node local memory is faster Scheduling algorithms take this into account Hyperthreading support CPU fools OS into thinking there are multiple CPUs Example dual Xeon with hyperthreading can support 2 logical processors Windows Server 2003 is hyperthreading aware Logical processors don t count against physical processor limits Scheduling algorithms take into account logical vs physical processors Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Many Packages 1 Windows XP Home Edition 1 CPU 4GB RAM 2 Windows 2000 amp XP Professional Desktop version but also is a fully functional server system 2 CPUs 4GB RAM Windows Server 2003 Web Edition new Reduced functionality Standard Server no domain controller 2 CPUs 2GB RAM Windows 2000 Server Windows Server 2003 Standard Edition Adds server and networking features active directory based domains host based mirroring and RAID 5 NetWare gateway DHCP server WINS DNS Also is a fully capable desktop system 4 CPUs 2 in Server 2003 4GB RAM 5 enews 2000 Advanced Server Windows Server 2003 Enterprise ition 3GB per process address space option Clusters 8 nodes 8 CPUs 8GB RAM 32GB in Server 2003 32 bit 64GB on 64 bit Windows 2000 Server 2003 Datacenter Edition Process Control Manager Licensed for 32 CPUs 64GB RAM 128GB on 64 bit edition
4. Servicing an interrupt raises processor IRQL to that interrupt s IRQL This masks subsequent interrupts at equal and lower IRQLs High User mode is limited to IRQL 0 Powerfail_ No waits or page faults at Interprocessor Interrupt IRQL gt DISPATCH_LEVEL Clock Device n Hardware interrupts Device 1 Dispatch DPC Deferrable software interrupts i _ normal thread execution Passive Deferred Procedure Calls DPCs Used to defer processing from higher device interrupt level to a lower dispatch level Driver usually ISR queues request One queue per CPU DPCs are normally queued to the current processor but can be targetted to other CPUs Executes specified procedure at dispatch IRQL or dispatch level also DPC level when all higher IRQL work interrupts completed Used heavily for driver after interrupt functions Also used for quantum end and timer expiration queue head DPC object DPC object DPC object Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Interrupt Time Accounting Time servicing interrupts are NOT charged to interrupted thread Time spent at IRQL 2 appears as DPC time Time spent at IRQL gt 2 appears as interrupt time Hence no process appears to be running What if system is not idle but no process appears to be run
5. device drivers for the computer Go to Control Panel gt System Hardware tab Click on Device Manager Click on Computer Right click Properties on as driver for PC BUMS m32 hal dl AWINNT System32 ntkmipa exe CAWINNT System32 ntoskmlexe Driver File Details F 2lxi m d m Advanced Configuration and Power Interface ACPI pe Device Manager Action view S mela E pe oj Batteries Provider Microsoft Corporation Computer E Advanced Configuration and Power Interface ACPI PC File version 5 00 2121 1 Copyright Copyright C Microsoft Corp 1981 1999 Screen snapshot from Control Panel System Hardware Device Manager Computer properties Driver Details 1 34 Debug Version Checked Build Special debug version of system called Checked Build Provided with MSDN Primarily for driver testing but can be useful for catching timing bugs in multithreaded applications Built from same source files as free build a k a retail build DBC compile time symbol defined which enables Error tests for can t happen conditions in kernel mode ASSERTs Validity checks on arguments passed from one kernel mode routine to another ifdef DBG if something that should never happen has happened KeBugCheckEx endif Multiprocessor kernel of course runs on UP systems Since no checked Server CD
6. Application Processes NTDLL DLL Executive Win32 Device Drivers Kernel User GDI Hardware Abstraction Layer HAL Environment Subsystems Windows NT 4 0 shipped with three environment subsystems Win32 32 bit Windows API OS 2 1 x character mode apps only Removed in Windows 2000 Posix only Posix 1003 1 bare minimum Unix services no networking windowing threads etc Removed in Windows XP Server 2003 enhanced version ships with Services For Unix 3 0 Of the three Win32 provides access to the majority of the native functions Of the three Win32 is required to be running System crashes if Win32 subsystem process exits POSIX and OS 2 subsystems are Win32 programs POSIX and OS 2 start on demand first time an app is run Stay running until system shutdown Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Subsystem Components API DLLs a For Win32 Kernel32 DLL Gdi32 DLL User32 DLL etc Subsystem process For Win32 CSRSS EXE Client Server Runtime SubSystem For Win32 only kernel mode GDI code Win32K SYS this code was formerly part of CSRSS Environment Subsystems Td User Application os 2 z POSI ami DLL Win32 Executive Win32 Device Drivers kemel User GDI Hardware Abstraction Layer HAL
7. Different for different applications e g file server versus SQL versus Exchange Windows kernel provides a scalable foundation Multiple threads of execution within a single process each of which can execute simultaneously on different processors Ability to run operating system code on any available processor and on multiple processors at the same time Fine grained synchronization within the kernel as well as within device drivers allows more components to run concurrently on multiple processors Multiple programming mechanisms that facilitate scalable server applications e g I O completion ports Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich SMP Scalability More efficient locking mechanism pushlocks Minimized lock contention for hot locks E g PFN Page Frame Database lock Some locks completely eliminated Charging nonpaged paged pool quotas allocating and mapping system page table entries charging f commitment of pages allocating mapping physical memory through AWE functions Even better in Server 2003 is reduction of use of spinlocks amp length they are e pispatcnet scheduling database locking now per New MP Configurations NUMA non uniform memory architecture systems Groups of physical processors called nodes that have local memory Still an SMP system e g any processor can access all of memory
8. NTOSKRNL EXE Uniprocessor NTKRNLMP EXE Multiprocessor gt 4GB new as of Windows 2000 NTKRNLPA EXE Uniprocessor w extended addressing support NTKRPAMP EXE Multiprocessor w extended addressing support Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich HAL Hardware Abstraction Layer Responsible for a small part of hardware abstraction Components on the motherboard not handled by drivers System timers Cache coherency and flushing SMP support Hardware interrupt priorities Subroutine library for the kernel and device drivers Isolates OS amp drivers from platform specific details a Presents uniform model of I O hardware interface to drivers Reduced role in Windows 2000 Bus support moved to bus drivers Majority of HALs are vendor independent NTOSKRNL And HAL Selection Selected at installation time See windows repair setup log to find out which one Can select manually at boot time with HAL in boot ini NT distribution CD ROM i386 Boot Partition Windows System32 NTOSKRNL EXE NTKRNLPA EXE NTKRNLMP EXE NTKRPAMP EXE NTOSKRNL EXE HAL D NTKRNLPA EXE HALACPI DLL HAL DLL etc see windowsl repair setup log Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich NTOSKRNL And HAL Selection NTOSKRNL amp HAL considered to be the
9. Stop code and parameters A guess at offending driver The analysis is the result of the automated execution of the analyze debugger command Debugger Commands Two types of commands Dot commands are built in Are commands are provided with extension S Extension DLLs allow Microsoft and third parties to dynamically add commands The main extension DLL is the kernel debugger extension DLL kdexts dll Each OS has a subdirectory with its own kdexts dll version as well as other development area specific extension DLLs e g Rpcexts dll ndiskd dll Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Deeper Analysis Always execute analyze with the v option to get more information Text description of stop code Meaning if any of parameters Stack dump Analyze uses heuristics to walk up the stack and determine what driver is the likely cause of the crash Followup is taken from optional triage ini file Useful Commands When you load a dump into the debugger it executes lanalyze Sometimes identifies the cause of a crash Always execute analyze v to see more The next steps Look at the current process process List all processes process 0 0 Look at a thread thread lt thread address or ID gt List loaded drivers Im kv Look at an I O request packet lirp lt irp address gt Disassemble code u
10. Word background print can continue to edit during print Take advantage of multiple processors On an MP system with n CPUs n threads can literally run at the same time Questions Given a single threaded application will adding a second processor make it run faster Will a multithreaded application run faster on an MP system Depends if application internal synchronization permits this Having too many runnable threads causes excess context switching 1 13 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich 00000000 Unique per process accessible in user or kernel mode 7FEFFFFF 80000000 Per process accessible only in kernel mode c000000 System wide accessible only in kernel mode FFFFFFFF 0 6FC00000000 1FFFFF0000000000 2000000000000000 3FFFFF0000000000 E000000000000000 E000060000000000 FFFFFF0000000000 User Address Space System PTE Space System Cache Paged pool Non paged pool Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich 32 Bit Virtual Address Space Code EXE DLLs Data EXE DLL static storage per thread user mode stacks process heaps etc x86 2 GB per process Address space of one process is not directly reachable from other processes 2 GB system wide Code NTOSKRNL HAL drivers Data kernel st
11. lt address or function name gt Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Hung Systems You can tackle a hung system but only if you ve prepared Boot in debug mode or Set the keystroke crash Registry value For debug mode you need a second system the debugger host connected to the target via serial cable Run Windbg Kd on the host Edit the target s boot ini file debugport comX baudrate XXX When the system hangs connect with the debugger and hit Ctrl C Hung Systems To configure keystroke crash Set HKEY_LOCAL_MACHINE System CurrentControlSet Services i804 2prt Parameters CrashOnCtrlScrl to 1 Enter right ctrl scroll lock scroll lock to crash the system Use thread to see what s running Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Microsoft On line Crash Analysis OCA Have Microsoft process dumps at oca microsoft com XP asks you if you want to submit after a crash You can visit OCA and manually submit a dump OCA accepts Win2K and XP dumps but is focused on XP Currently requires a Passport account to check crash analysis status if it doesn t know right away What Does OCA Do Server farm uses analyze but uses Microsoft s Triage ini file and database that includes information about known problems Sev
12. Accounting System Threads Process based code Summary Processes And Threads Per process i dd What is a process address space Represents an instance of a running program You create a process to run a program Starting an application creates a process Process defined by Address space Resources e g open handles Security profile token What is a thread An execution context within a process a of scheduling threads run processes don t run All threads in a process share the same per process address space Services provided so that threads can synchronize access to shared resources critical sections mutexes events semaphores All threads in the system are scheduled as peers to all others without regard to their parent process System wide Address Space Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Scheduling Priorites Realtime Time Critical Realtime Realtime sane Levels 16 31 Realtime Idle Above Normal Dynamic Levels 1 15 Dynamic Idle System Idle Processes And Threads Every process starts with one thread First thread executes the program s main function Can create other threads in the same process Can create additional processes Why divide an application into multiple threads Perceived user responsiveness parallel oackground execution Examples
13. But one OS Through Windows 2000 core operating system executables are identical NTOSKRNL EXE HAL DLL xxxDRIVER SYS etc XP amp Server 2003 have different kernel versions but not substantially different Registry indicates system type set at install time HKEY LOCAL _MACHINE System CurrentControlSet Control ProductOptions ProductType WinNT Workstation ServerNT Server not a domain controller LanManNT Server that is a Domain Controller ProductSuite indicates type of Server Advanced Datacenter or for Windows NT 4 0 Enterprise Edition Terminal Server Code in the operating system tests these values and behaves slightly differently in a few places Licensing limits number of processors number of inbound network connections etc Boot time calculations mostly in the memory manager Default length of time slice Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich NTOSKRNL EXE Core operating system image Contains Executive and Kernel Kernel versions Windows NT 4 0 is 4 0 client and server Windows 2000 is 5 0 client and server Windows XP is 5 1 client only Windows Server 2003 is 5 2 server only Kernel evolution NT4 gt Windows 2000 significant change Windows 2000 gt Windows XP modest change Windows XP gt Server 2003 minimal change NTOSKRNL Variants Four variations 4GB or less
14. Exe Control panel can start and stop services and change startup parameters Service Processes Control Panel a Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Mapping Services to Service Processes Service properties displayed through Control Panel services msc show name of EXE But not which process started services are in Tlist S or Tasklist svc new as of XP list internal name of services inside service processes Process Explorer shows both internal and external name Services Infrastructure Improvements Two new less privileged accounts for built in services LOCAL SERVICE NETWORK SERVICE Less rights than LocalSystem Reduces possibility of damage if system compromised More services run in generic service host process svchost exe Reduces number of processes Four instances at least SYSTEM SYSTEM 2nd instance for RPC LOCAL SERVICE NETWORK SERVICE Later we ll see how to understand WHICH service is consuming CPU time when a multi service process is running Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Logon Process 1 Winlogon sends username password to Lsass Either on local system for local logon or to Netlogon service on a domain Windows XP enhancement Winlogon doesn t wait for Workstation ser
15. Prompt robocopy z edimgsrv1 Calendar Microsoft Outlook E Command Prompt Klinbox Microsoft Outlook D Tasks Microsoft Outlook Z 1 Reminder dep353 ppt gt Endtask W switchto New Task Commit Charge 197M 11 Processes 42 CPU Usage 2 Running means waiting for window messages Dealing with a CPU Hog Option 1 Try and figure out what it s doing using monitoring tools explained later in this talk Option 2 Lower the priority Option 3 Suspend the process with PsSuspend Another use you ve started a long running job but want to pause it to do something else Lowering the priority still leaves it running Option 4 Kill the process Identify The Image Once you ve found the process of interest what is it Sometimes name of EXE identifies clearly e g Winword exe Often it doesn t since Task Manager doesn t show the full path of the image We need more information Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Explorer Sysinternals Super Task Manager Shows full image path command line environment variables parent process security access token open handles loaded DLLs amp mapped files Process Explorer Sysinternals www sysinternals com File View Process Handle Options Search Help Bans gx Process CPU Description Owner Session Ha O Syst
16. Properties General Performance Hardware Profiles System Startup Environment User Profiles Show list for 1 seconds Recovery Startup windows NT Workstation Version 4 00 H I Write an event to the system log I Send an administrative alert T Write debugging information to When a STOP error occurs do the following Enabling Dumps In Windows 2000 XP General Network Identification Hardware User Profiles Advanced m Performance Performance options control how applications use memory which affects the speed of your computer Performance Options m Environment Variables E Environment variables tell your computer where to find certain types of information Environment Variables m Startup and Recovery Startup and recovery options tell your computer how to sl and what to do if an error causes your computer to stop Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Startup and Recovery r System startup Default operating system I Display list of operating systems for 5 a seconds r System Failure IV Write an event to the system log IV Send an administrative alert IV Automatically reboot Write Debugging Information Kernel Memory Dump bal Dump File SystemRoot MEMORY DMP I overwrite any existing file At The Crash A component calls KeBugChe
17. ROM 5 Windows XP Media Center Edition 6 Windows XP Tablet PC Edition Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows Server 2003 Replacement for Windows 2000 Server family Name changes for flavors Windows Server 2003 Web Edition new package Windows Server 2003 Standard Edition was Server Windows Server 2003 Enterprise Edition was Advanced Server Windows Server 2003 Datacenter Edition no change New features More scalable 64 processor systems 8 node clusters larger memory maximums IIS 6 0 HTTP in the kernel Connection failover Active Directory enhancements Many new group policies Remote Installation Support RIS Bundles NET Framework Level Of Kernel Change Windows NET Server 2003 amp Windows XP are modest upgrades as compared to the changes from Windows NT 4 0 to Windows 2000 Kernel architecture is basically unchanged No new subsystems No new API sets Internal version numbers confirm this Windows 2000 is 5 0 Windows XP is 5 1 not 6 0 Windows NET Server is 5 2 Not the same kernel as XP a superset But nonetheless still lots of interesting kernel changes Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time
18. not for Win95 Got proper version for Win95 problem went away Example Problem Access Hangs Problem Access 2000 would hang when trying to import an Excel file Worked fine on other users workstations User had Access 97 and Access 2000 installed Compared a Filemon log from the working and failing system Failing system was loading an old Access DLL from windows system32 due to having installed Access 97 previously Removed DLL and problem went away Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich DII Version Mismatch Lab With Word XP installed in the default folder 1 Go to folder Program Files Microsoft Office Office 1033 Rename MSO9INTL DLL to MSO9INTL DLL1 Copy OUTLLIBR DLL to MSOYINTL DLL Try and start Word Send error report to Microsoft Use FileMon to confirm which DLL is likely causing the problem Configuration Problems Missing corrupted or overly secure Registry settings often lead to application crashes and errors Some applications don t completely remove registry data at uninstall Regmon may yield the answer Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Regmon Output Request OpenKey CreateKey SetValue QueryValue CloseKey Path HKCU HKEY_CURRENT_USER per user settings HKLM HKEY_LOCAL_MACHINE system wide sett
19. not maximum To display maximums use vm kernel debugger command Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Debugging Pool Leaks Two options Poolmon In the Support Tools and the Device Driver Kit DDK Requires that you turn on Pool Tagging with Gflags on Windows NT and Windows 2000 Driver Verifier Select all drivers Turn on pool tracking Troubleshooting with Poolmon Poolmon exe Support Tools Shows paged and nonpaged pool consumption by data structure tag Must first turn on pool tagging with Resource Kit gflags tool amp reboot On by default in Windows Server 2003 command Prompt poolmon Memory 130484K Avail 63296K PageFlts e InRam Krnl 2816K P 12908K Commit 56740K Limit 322000K Peak 57028K Pool N 2464K P 15072K Tag Type Allocs Frees Diff Bytes Per Alloc Key Paged 33275 o 33013 J 262 16800 CMkb Paged 33275 33155 120 23104 ObSq Paged 31597 31597 o o Paged 9968 j 9056 j 912 129984 Paged 7050 6519 531 9335104 Nonp 5477 3932 1545 296640 Paged 5039 5011 28 1792 Paged 3572 3368 204 264320 Paged 3498 3477 21 4256 Paged 2862 2596 266 34048 Paged 2839 2651 188 33536 Nonp 2660 1629 1031 65984 Nonp 2517 1515 1002 96160 Nonn 27305 aT 71927 aT 113 14RRO Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Troubleshoot
20. point to the problem Have to first massage log file to remove data that differs run to run Delete first 3 columns they are always different line time process id Easy to do with Excel by deleting columns Then compare with FC built in tool or Windiff Resource Kit Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Filemon operation number Process image name process id Request internal I O request code Result return code from I O operation Other flags passed on I O request amp File Monitor Sysinternals www sysinternals com File Edit Options Drives Help GW ABE O oF AN Time Process Request Path Result Other 222 6 57 20 AM_notepad exe 2592 IRP_MJ_CREATE CA SUCCESS Attributes Any Options Open D 223 6 57 20 AM_notepad exe 2592 IRP_MJ_DIRECTORY CA NO SUCH FILE FileBothDirectorylnformation te 224 6 57 20 AM_notepad exe 2592 IRP_MJ_CLEANUP CA SUCCESS 225 6 57 20 AM__notepad exe 2592 _IRP_MJ_CLOSE CA SUCCESS notepad exe 2592 IRP_MJ_ CREATE CiAtest bt FILE NOT FOUND 227 6 57 20 AM winlogon exe 484_ FASTIO_LQUERY_OPEN C WINNT m SUCCESS 228 6 57 20AM winlogon exe 484 IRP_MJ_CREATE CAWINNT m SUCCESS Attributes N Options Open 229 6 57 20 AM _ winlogon exe 484 FASTIO QUERY STA CAWINNT m SUCCESS Size 42576 Controlling Filemon Start stop logging Control E Clear display Control X Open Explorer wind
21. problems rename the key and re run the application Most applications re create user settings when run In this way the data won t be seen by the application Can always rename the key back Regmon Lab 2 Run Notepad Change Font and point size Enable Word wrap Run Regmon amp filter to Notepad exe Exit Notepad In Regmon log find location of user specific Notepad settings Double click on a line to jump to Regedit Delete top level Notepad user settings key Re run Notepad and confirm font and word wrap reset to default setting Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Example Problem Internet Explorer hung when started Default internet connection was set but wasn t being dialed Dialing the connection first manually and then running IE worked Background information User had previously installed the AT amp T Dialer program but had uninstalled it and created dial up connection manually Solution Ran Regmon Looked backwards from end at the point IE was hung Found references to ATT under a PhoneBook key Renamed ATT key and problem went away Conclusion registry junk was left from uninstall Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Example Problem User somehow disabled all toolbars and menus in Word No way to open files change settin
22. this problem We have created an error report that you can send to us We will treat this report as confidential and anonymous To see what data this error report contains click here Send Error Report Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows Error Reporting On XP amp Server 2003 when an unhandled exception occurs System first runs DWWIN EXE DWWIN creates a process microdump and XML file and offers the option to send the error report Then runs debugger Drwtsn32 exe accvio EXE has encountered a problem and needs to close We are sorry for the inconvenience If you were in the middle of something the information you were working on might be lost Please tell Microsoft about this problem We have created an error report that you can send to us We will treat this report as confidential and anonymous To see what data this error report contains click here Send Error Report Windows Error Reporting Configurable with System Properties You can choose to have software emors gt Advanced gt Error Be al Moved nb rove hive Reporting HKLM SOFTWARE Microsoft PCHealth 3 3 ErrorReporting Windows operating system Configurable with group Programs policies HKLM SOFTWARE Lee Policies Microsoft PCHealth Error Reporting O Disable eror reporting Windows Internals and Advanced Trou
23. vendor s support Note that the Verifier means fewer crashes on oe XP than Windows 2000 than Windows Getting Past a Crash Last Known Good Boots with driver kernel configuration last used during a successful boot Safe Mode Boots the system with core set of drivers and services Network and non network The Recovery Console Manually disable offending service replace corrupt images update files ERD Commander 2002 Registry Editor Explorer Driver Service Manager password changer Event Log viewer Notepad Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich The Bluescreen Screen Saver Scare your enemies and fool your friends with the Sysinternals Bluescreen Screen Saver Be careful your job may be on the line More Information Inside Windows 2000 3 edition section on System Crashes in chapter 4 Debugging Tools help file Knowledge Base Articles http Awww microsoft com ddk debugqging Other books http Awww microsoft com ddk newbooks asp The debugger team wants your feedback and bug reports Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich End of Tutorial Thank you for coming Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich
24. 08 Screen snapshot from kernel debugger memusage command 1 20 ol Oy OrFNAIOW OW O Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Page Files Page Files What gets sent to the paging file Not code only modified data code can be re read from image file anytime When do pages get paged out Only when necessary Page file space is only reserved at the time pages are written out Once a page is written to the paging file the space is occupied until the memory is deleted e g at process exit even if the page is read back from disk Can run with no paging file Windows NT4 Windows 2000 Zero pagefile size actually created a 20MB temporary page file Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Do Need More Memory If heavy paging activity Monitor Memory gt Page Reads sec Not Page Faults sec which includes soft faults Should not stay high for sustained period Some hard page faults unavoidable Process startup Normal file I O done via paging To eliminate normal file I O subtract System gt File Read Operations sec Or use Filemon to determine what file s are having paging I O asterisk next to I O function Sizing The Page File Given understanding of page file usage how big should the total paging file space be Windows supports multiple pagin
25. 3 exe lt T h read S Process ID m Select thread of interest TE Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich E Pid 908 WinDbg 6 0 0007 0 Call Stacks EEEREN tlee E Ala HPE al Fala 333 es Al 32 ED f not obvious from function names note Ags Funcinfo source aders Headings Nonvolatie regs Frame nums name of DLL and look m s at description in Process Explorer aaa ata re ca sree anes ntd1l NtWaitForSingleObject 0xc Ru n Stri n gs alrsvc SvchostPushServiceGlobals 0x4c lalrsve ServiceMaint 0xz15 Sysi nte rnals on D LL ADVAPI32 CryptVerifySignatureAt Oxa2 lt TA Calls 000 38c C WINNT System32 svchost exe 000 390 001 398 002 3a8 003 4 4 004 4 8 005 4fc Process _ Descriptio T svchost exe ic Hq Z spoolsv exe Aee Base Size n 0 Col Sys 0 lt Local gt Proc000 38 Thrd 006 50c 0x63000000 0x94000 Wi PEC O Te x Di 0x70F80000 0x7000 Alerter Service DLL 8 23 2001 7 00 AM _c WINNT system32 alrsvc dil 0x71950000 OxE4000 e perience Coniro G8 00 AM CAWINNTWin 86 Micros 0x71A50000 0x3B000 Microsoft Windows Socket 5 8 23 2001 7 00 AM C WINNT system32 mswsock Mezi AS0000 0x8000 Windows Sockets Helper 5 8 23 2001 7 00 AM CAWINN TisustamaZiwshicnin hd svchost exe pid 908 Refresh Rate Paused Examining Syste
26. ARE Microsoft Windows CurrentVersion Run FE CAWINDOWS System32 00THotkey exe Z 000StTHK exe Sh C Program Files TOSHIBA TME3STMESRV31 EXE Logon Z C Program Files TOSHIBASTME3STMER2CILEXE Service FA C Program Files TOSHIBA TME 3 T MESBS32 EXE Client CAProgram Files CA eT rust Antivirus tealmon exe C systemroat system32 dumprep 0 k f C Program Files NightCreation webPublisher for Outlook WPO EXE Minimize Sg HKCU Software Microsoft Windows CurrentVersion Run C Program Files Messenger msmsas exe background C Program Files Microsoft ActiveS ync WCESCOMM EXE HKLM SOFTWARE Microsoft Windows CurrentVersion RunServices HKLM SOFTWARE Microsoft Windows CurrentVersion RunServicesOnce HKCU Software Microsoft windows CurentVersion RunServices Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Startup Item pinger TPWRTRAY TosHKCW fest qttask dumprep 0 x vdtask Prog ent C Program evntsve C Program K y ws m Firewall Client C PROGRA Common Startup Z HotSync Man C PROGRA Startup v Disable All Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time Accounting System Threads Process based code Summary Kernel Architecture System Processes Services Service Control Mgr SHE Explorer WinMgt Exe M WinLogon SpoolSv Exe User i App
27. America Online Error Your MAIN IDX file is damaged If this problem persists you may need to reinstall the America Online software For more information go to Keyword help and refer to MAIN IDX in the Error Messages section Filemon showed this waol exe OPEN C PROGRA 1 AMERIC 1 0 IDB main ind ACCESS DENIED User did not have admin rights to AOL directory Example Access Denied For example an application failed with this error x A Application defined or object defined error Message ID Connect LoadGloballSw Variables LN EN 287 Ran Filemon and found it was getting Access Denied OUTLOOK EXE FASTIO_CHECK_I C WINDOWS System32 MSCOMCTLOCX SUCCESS QUTLOOK EXE FASTIO_LREAD C AWINDOWS System32 MSCOMCTL OCK SUCCESS OUTLOOK EXE FASTIO_QUERY_ C WINDOWS System32 MSCOMCTL OCK SUCCESS OUTLOOK EXE FASTIO QUERY C WINDOWS System32 MSCOMCTL OCX SUCCESS a OUTLOOK EXE IRP MJ CREATE wSrv1 Dept DeptApp Someone had misread a request to remove EDIT rights and removed all rights Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Hot File Analysis Understand disk activity system wide Run Filemon for a period of time Save output in a log file Import into Excel and make a pie chart by file name or operation type Example used Filemon on a server to determine which file s were being accessed mo
28. G i ri Total 204868 Total 42764 paging file s N Bs 1136712 Paged 34676 does not reflect true maximum Peak 205340 Nonpaged 3088 page file sizes expansion when total reaches limit further Processes 41 CPU Usage 4 Commit Charge 200M 1110M VirtualAlloc attempts by any process will fail Commit Charge K Kernel Memory K Screen snapshot from Task Manager Performance tab Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Why Page File Usage on Systems with Ample Free Memory Because memory manager doesn t let process working sets grow arbitrarily Processes are not allowed to expand to fill available memory previously described Bias is to keep free pages for new or expanding processes This will cause page file usage early in the system life even with ample memory free We talked about the standby list but there is another list of modified pages recently removed from working sets Modified private pages are held in memory in case the process asks for it back When the list of modified pages reaches a certain threshold the memory manager writes them to the paging file or mapped file Pages are moved to the standby list since they are still valid and could be requested again Memory Leaks Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Mem
29. Hardware Abstraction Layer HAL hardware interfaces buses I O devices interrupts interval timers DMA memory cache control etc etc Original copyright by Microsoft Corporation Used by permissibr Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Tools used to dig in Many tools available to dig into Windows 2000 XP internals Helps to see internals behavior in action We ll use these tools to explore the internals Many of these tools are also used in the labs that you can do after each module Several sources of tools Support Tools Resource Kit Tools Debugging Tools Sysinternals com Inside Windows 2000 3 edition book CD Additional tool packages with internals information Platform Software Development Kit SDK Device Driver Development Kit DDK Windows XP Six variants Windows XP Professional replaces Windows 2000 Professional Windows XP Home Edition new First consumer focused release of NT Replaces Windows ME Millenium Edition Has slightly less features than Windows XP Professional Windows XP Professional 64 bit Edition new First 64 bit version of NT 64 bit pointers much larger address space Runs on Intel Itanium amp Itanium 2 later AMD Opteron Windows XP Embedded Same kernel as regular 32 bit XP Configurable to remove unnecessary components Boot and execute from ROM OS runs from RAM apps from
30. Windows Internals and Advanced Troubleshooting Part 1 Kernel Architecture Mark Russinovich Winternals Software David Solomon David Solomon Expert Seminars Purpose of Tutorial Give IT Professionals a foundation understanding of the Windows OS kernel architecture Note this is a small but important part of Windows The plumbing in the boiler room Condensed from a 5 day internals class Benefits Able to troubleshoot problems more effectively Understand system performance issues Applies to NT4 Windows 2000 Windows XP and Windows Server 2003 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Outline 1 Kernel Architecture 2 Troubleshooting Processes and Threads 3 Troubleshooting Memory Problems 4 Crash Dump Analysis Kernel Architecture System Processes Services Applications Service Control Mgr SvcHost Exe rea WinMgt Exe WinLogon SpoolSv Exe Uisar i Session Services Exe Application Manager Subsystem DLLs Environment Subsystems System NTDLL DLL Threads System Service Dispatcher kernel mode callable interfaces jen Device amp File Sys Drivers JOUN aoualajay Ayunoasg Kiowa speaiyL 8 saSsa00ld A s1691 JBW uo einByuog eo np 201d e007 Win32 USER GDI Graphics Drivers Kernel
31. a8 80454faf 00000000 00000000 00000000 nt KeBalanceSetManager 0x7e f5c67ddc 80468ec2 804634e0 00000000 00000000 nt PspSystemThreadStartup 0x69 00000000 00000000 00000000 00000000 00000000 nt KiThreadStartup 0x16 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Agenda Introduction amp Data Structures Identifying the Process Analyzing Process Thread Activity Application Failures Troubleshooting Application Failures Most applications do a poor job of reporting file related or registry related errors E g permissions problems Missing files Missing or corrupt registry data Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Troubleshooting Application Failures When in doubt run Filemon and Regmon Filemon monitors File I O Regmon monitors registry I O Ideal for troubleshooting a wide variety of application failures Also useful for to understand and tune file system access E g understanding hard drive activity Work on all Windows OSs Used extensively within Microsoft Using Regmon Filemon Two basic techniques Go to end of log and look backwards to where problem eee or is evident and focused on the last things one Compare a good log with a bad log Often comparing the I O and Registry activity of a failing process with one that works may
32. acks Process page tables Non paged pool Paged pool The operating system is loaded here and appears in every process s address space The operating system is not a process though there are processes that do things for the OS more or less in background 3 GB user space and Address Windowing Extensions AWE t b d User Mode User Space 64 Bit Virtual Address Space Kernel Mode User Space User Page Tables Itanium Session Space Session Space Page Tables System Space Session Space Page Tables 64 bit Windows 7152 GB 128 GB 1TB 128 GB 128 GB 32 bit Windows 2or3 GB 2 GB 960 MB 650 MB 256 MB Memory Protection Model No user process can touch another user process address space Without first opening the process means passing through NT security All kernel components share a single address space This is how driver bugs can cause blue screens Most other commercial OSs Unix Linix VMS etc have the same design Memory Protection Model Controlled by using two hardware access modes user and kernel X86 Ring 0 Ring 3 Itanium Privilege Level 0 amp 3 Each memory page is tagged to show the required mode for access Associated with threads Threads can change from user to kernel mode and back via a secure interface Part of saved context along with registers etc Does not affect scheduling Windo
33. an reference without incurring a page fault A process always starts with an empty working set Pages itself into existence XP prefetches pages to speed up application startup Many page faults may be resolved from memory newer pages older pages PerfMon gt Process WorkingSet Process Memory Information Task Manager as Processes tab Ce CA TL Mem Uesgel va Size E dle Pr i System 36K Mem Usage physical smes exe 164K memory used by process O NE ee working set size not ES EXE 1124K eee E 948 K working set limit i 2008 K gt Note Shared pages are Ae 188K counted in each ceee aer process RPCSS EXE 820K VM Size private not TE ee 664 K shared committed virtual whisve exe rly a EXPLORER 1768K space in processes PSTORES EXE 1812K potential pagefile usage RASMAN EXE 1080 K wixmod32 exe 1496 A Mem Usage in status bar f is not total of Mem Usage End Process column see later slide Processes 38 CPU Usage 3 Mem Usage 68312K 274772K A Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Shared Memory Like most modern OSs Windows provides a way for processes to share memory High speed IPC used by LPC which is used by RPC Process 1 Threads share address space but Address applications ma
34. are What limits total process virtual memory Page file size most of physical memory Called Commit limit What limits physical size of a process Physical memory Memory Manager policies Based on memory demands and paging rates 00000000 Unique per process accessible in user or kernel mode 7FFFFFFF 80000000 Per process accessible only in kernel mode c000000 System wide accessible only in kernel mode FFFFFFFF Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Code EXE DLLs Data EXE DLL static storage per thread user mode stacks process heaps etc Code NTOSKRNL HAL drivers Data kernel stacks Process page tables hyperspace Non paged pool Paged pool 32 Bit Virtual Address Space x86 2 GB per process Address space of one process is not directly reachable from other processes 2 GB system wide The operating system is loaded here and appears in every process s address space The operating system is not a process though there are processes that do things for the OS more or less in background 3 GB user space and Address Windowing Extensions AWE t b d m 3GB Process Space Option Unique per 3GB option in BOOT INI 7 sae EXE code Provides up to 3 GB per process user or kernel Globals address space mode Per thread user Win
35. bleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Dr Watson for Windows Dr Watson Log File Path nee Crash Dump C Documents and Settings All Browse g User message box Number of Instructions 10 doesn t mention most Naba doe To Sae 0 important thin g Crash Dump Type Full Mini NT4 compatible Full Options A dump file was created 2 oma T MV Dump All Thread Contexts Can customize by Pincay T running 7 eae DRWTSN32 EXE Note servers default to no visual notification To set Dr Watson as default debugger Drwtsn32 i Application Emors Cancel Help Dumping a Running Processes Instead of killing a hung process leaving no debug info run Dr Watson on it Dr Watson creates a crash dump file and then kills process drwtsn32 p processid Autodump Debugging Tools will snapshot a process without killing it E g a server process that is having problems on a production system Snapshot the process and debug offline Determine if the process needs to be restarted or not Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich End of Troubleshooting Processes amp Threads Next Troubleshooting Memory Problems Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows I
36. ckEx which takes five arguments Stop code 4 stop code defined parameters KeBugCheckEx Turns off interrupts Tells other CPUs to stop Paints the blue screen Notifies registered drivers of the crash If a dump is configured Verifies checksums Calls dump I O functions Common Stop Codes There are about 150 defined stop codes IRQL_ NOT LESS OR _EQUAL 0x0A Usually an invalid memory access INVALID KERNEL _MODE_TRAP 0x7F and KMODE_EXCEPTION NOT HANDLED Ox1E Generated by executing garbage instructions Its usually caused when a stack is trashed Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich At The Reboot WinLogon Session Manager G SaveDump NtCreatePagingFil 2 reatePagingFile Paging User mode Kernel mode At The Reboot Session Manager process winnt system32 smss exe initializes paging file NtCreatePagingFile NtCreatePagingFile determines if the dump has a crash header Protects the dump from use WinLogon calls NtQuerySystemInformation to tell if there s a dump Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich At The Reboot f there s a dump Winlogon executes SaveDump winnt system32 savedump exe Writes an event to the System event log SaveDump writes contents to appr
37. d the dump to You Can Do It Many systems administrators ignore Windows NT Windows 2000 s crash dump options don t know what to do with one Its too hard It won t tell me anything anyway Basic crash dump analysis is actually pretty straightforward Even if only 1 out of 5 or 10 dumps tells you what s wrong isn t it worth spending a few minutes Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich What Causes Crashes System crashes when a fatal error prevents further execution Any kernel mode component can crash the system Drivers and the OS share the same memory space Therefore any driver or OS component can due to a bug corrupt system memory Note This is for performance reasons and is the same on Linux most Unix s VMS etc Dump Options Complete memory dump Windows NT 4 Windows 2000 Windows XP Full contents of memory written to lt systemroot gt memory dmp Kernel memory dump Windows 2000 Windows XP System memory written to lt systemroot gt memory dmp Small memory dump Windows 2000 Windows XP Also called a minidump or triage dump 64KB of summary written to lt systemroot gt minidump MiniMMDDYY NN dmp Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Enabling Dumps In Windows NT 4 System
38. dows NET Server supports mode stacks variations from 2GB to 3GB DLL code USERVA Per process Process heaps Restrictions to use accessible Only available on Windows 2000 only in kernel Advanced Server amp NET Server mode Enterprise Edition Limits memory to 16 GB EXE must have large address System wide space aware flag in image accessible ___ _YP Pers header or they re limited to 2 only in kernel GB specify at link time or with mode Exec kernel HAL imagecfg exe in Resource Kit drivers etc FFFFFFFF Better solution address windowing extensions 00000000 BFFFFFFF c0000000 Process page tables hyperspace 64 Bit Virtual Address Space 0 User Mode User Space 6FC00000000 Kernel Mode User Space 1FFFFF0000000000 User Page Tables Itan l um 2000000000000000 Session Space 3FFFFF0000000000 Session Space Page Tables E000000000000000 E000060000000000 System Space FFFFFF0000000000 Session Space Page Tables 64 bit Windows 32 bit Windows User Address Space 7152 GB 2or3 GB System PTE Space 128 GB 2 GB System Cache 1TB 960 MB Paged pool 128 GB 650 MB Non paged pool 128 GB 256 MB Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Memory Usage Working Set Working set All the physical pages owned by a process Essentially all the pages the process c
39. e 1808 Flumnat ava ARAD CPU Description tmesbs3 2 TMESRV3 EXE 1692 Toshiba MobileExtension Service VMnet DHCP Service Identify The Image Sometimes description is not meaningful Process Explorer Sysinternals www sysinternals com File View Process Handle Options Search Help H anarsa Process msmsgs exe 2ISATRAY EXE Ed vdtask exe netsrv exe Check full path of EXE with Process Explorer Often pinpoints which produc Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Description Messenger A D oF TASKBARW MFC Application vdtask exe Properties Image Performance Security Environment Image File 2e TASKBARW MFC Application Version 1 00 0000 0001 Path C Program Files FarStone WirtualDrive vdtask exe Identify The Image Often applications are installed in Windows System32 sie Or in folders with eg TFneky unrecognizable names E Check company name or copyright Process Explorer doubl click on process File version 0 2 0 4 Explorer gt right cli Desoipion TFncKy properties on EXE by Corrir Other version information TFncKy exe Properties Image Performance Security Environment General Version Compatibility Security Summary ltem name Value Toshiba Corporation Identify The Image qttask exe Properties Image Performance Securty Environmen
40. em Idle Process lt access denied gt 0 0 O System NT AUTHORITY 0 455 El smss exe 396 Windows NT Session Manager NT AUTHORITY 0 21 Flesrss exe 452 Client Server Runtime Process NT AUTHORITY 0 510 Elwinlogonexe 475 Windows NT Logon Application NTAUTHORITY 0 568 Y explorer exe 312 Windows Explorer DSOLOMON ds 679 OUTLOOK EXE 1312 435 0 0 0 0 Microsoft Outlook DSOLOMON ds 2 hh exe 1316 _ R procexp exe 2932 Microsoft HTML Help Executable DSOLOMON ds Sysinternals Process Explorer DSOLOMON ds 0 0 0 0 E cmd exe 1980 0 Windows Command Processor DSOLOMON ds 0 0 0 0 0 Handle Type Access Name Desktop 0x000F01FF Default Desktop 0x000F01 FF Winlogon Desktop 0x000F01 FF Disconnect Desktop 0x000F01 FF Default Directory 0x000F000F Windows Directory 0x0002000F BaseNamedObjects Directory 0x00000003 KnownDils Event 0x00100000 BaseNamedObjects winSta0_DesktopSwitch Event 0x001F0003 BaseNamedObjects ThemesStartEvent Event 0x001F0003 BaseNamedObjects WFP_IDLE_TRIGGER Refresh Rate Paused Process Explorer Process tree If left justified parent has exited Disappears if you sort by any column Bring back with View gt Show Process Tree Additional details in process list Icon and description from EXE User Name shows domain name Highlight Own Services Processes Differences highighting Green new Red gone View gt Update speed gt Paused Windows Internals and Adva
41. eral ways to get OCA results Via e mail At the OCA site Sometimes OCA will point you at KB articles that describe the problem KB articles may tell you to use Windows Update to get newer drivers a hotfix or install a Service Pack Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Driver Verifier This tool was introduced in Windows 2000 and can be useful to validate a suspicion about a driver The Verifier performs the following checks IRQL rule adherence O request consistency Proper memory usage Special Pool Special pool is a kernel buffer area where buffers are sandwiched with invalid pagen 2 valid pages an y Higher Conditions for a driver Addresses allocating from special FORRIN aranais pool Driver Verifier is verifying driver Invalid Special pool is enabled Allocation is slightly less than one page 4 KB on x86 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Driver Verifier If the Verifier detects a violation it crashes the system and identifies the driver If you find a driver in a crash dump that looks like it might be the cause of the crash turn on verification for it Use Last Known Good if the verifier detects a bug during the boot If a bug is detected in a third party product check for updates and or contact the
42. ernals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich UN TOT i I Kept in memory on standby or modified page list Managing Physical Memory System keeps unowned physical pages on one of several lists Free page list Modified page list Standby page list Zero page list Bad page list pages that failed memory test at system startup Standby And Modified Page Lists Modified pages go to modified dirty list Avoids writing pages back to disk too soon Unmodified pages go to standby clean list They form a system wide cache of pages likely to be needed again Pages can be faulted back into a process from the standby and modified page list These are counted as page faults but not page reads Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Free And Zero Page Lists Free Page List Used for page reads Private modified pages go here on process exit Pages contain junk in them e g not zeroed On most busy systems this is empty Zero Page List Used to satisfy demand zero page faults References to private pages that have not been created yet When free page list has 8 or more pages a priority zero thread is awoken to zero them On most busy systems this is empty too Paging Dynamics demand zero page read from page faults disk or kernel a
43. g files Size should depend on total private virtual memory used by applications and drivers Therefore not related to RAM size except for taking a full memory dump Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Sizing The Page File Worst case Windows has to page all private data out to make room for code pages To handle minimum size should be the maximum of VM usage Commit Charge Peak Hard disk space is cheap so why not double this Normally make maximum size same as minimum But max size could be much larger if there will be infrequent demands for large amounts of page file space Performance problem Page file extension will likely be very fragmented Extension is deleted on reboot thus returning to a contiguous page file Memory Management Information Task Manager SPESEN Performance tab m ome Applications Processes Performance Networking CPU Usage CPU Usage History Total committed private virtual memory total of VM Size in process tab Kernel sag Page File Usage History Memory Paged not all of this space has actually been used in the paging files it is 200 MB how much would be used if it was TE Physical Memory K all paged out SAER Handles 93008 Total 768944 Commit charge limit sum of men Be ete 100580 physical memory available for cokes nma processes current total size of
44. gs etc With Regmon captured startup of Word Found location of user specific settings for Word Deleted this Registry key Re ran Word menus and toolbars were back Word re created user settings from scratch Filemon Regmon as a Service Sometimes need to capture I O or registry activity during the logon or logoff process E g errors occuring during logon logoff Solution Run Filemon Regmon with AT command Install and run Filemon Regmon as a service Use Srvany tool from Resource Kit In either case but tools remain running after logoff Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Analyzing Process Crashes If you still can t determine why a process is crashing next step is to get a process dump to the developer But until XP few knew there was a process dump Process Crashes Dr Watson for Windows NT An application error has occurred and an application error log is being generated accvio exe Exception access violation 0xcO000005 Address 0400401032 Cancel Help accvio EXE Program Error accyvio EXE has encountered a problem and needs to accvio exe has generated el close We are sory for the inconvenience You will need to restart the p A i If you were in the middle of something the information you were working on An error log is being created might be lost Please tell Microsoft about
45. hread s are running and what code they are executing With Performance Monitor monitor Processor Time for threads inside a process Find thread s consuming CPU time Analyzing Thread Activity Then try and determine what code they are executing by finding which code module the thread started in 1 Get thread start address with Tlist 2 With Process Explorer DLL view sort by base address and find in which module the address lies Can also do this with Tlist Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Analyzing Thread Activity Start address may not be enough May need to look at call stack Can attach with Windbg or Ntsd and issue k command Caution pre XP exiting debugger kills debugee if real debugger attachment Attach noninvasive Freezes threads while connected Allows viewing information in process but not changing data Analyzing Call Stacks Attach to Process With Windbg click on z File gt Attach to Process ah eE 564 services exe H 576 lsas i l 740 host Then View gt sep tae OE Oe gs ee 764 svchost exe Services AudioSrv BITS CryptSvc Dhep a ac 3 e52 svchost exe Services Dnscache Command Line C N 908 svchost exe h Vi Services Alerter LmHosts RemoteRegis Then View 1056 epcolev uce 1320 Fast exe 1392 THOTKEY EXE gt Processes and 1440 Tnesbs
46. ing o x File Options view Help Applications Processes Performance CPU Usage CPU Usage History El Rn MEM Usage p Memory Us Handles 6801 Total 65076 Threads 423 Available 4104 Totals Physical Memory K Processes 36 E Cache 23752 Total 132268 Total 50624 Paged 44800 Nonpaged S824 Limit 149344 Peak 191068 r Commit Charge K 2 Memory K Processes 36 CPU Usage 6 Mem Usage 132268K 149344K 7 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich End of Troubleshooting Memory Problems Next Crash Dump Analysis Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows Internals and Advanced Troubleshooting Part 4 Crash Dump Analysis Outline What causes crashes Crash dump options Analysis with WinDbg Kd Debugging hung systems Microsoft On line Crash Analysis Using Driver Verifier Live kernel debugging Getting past a crash Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Why Analyze Dumps The debuggers and Microsoft Online Crash Analysis OCA often solve crashes Sometimes however they do not so your analysis might tell you What driver to disable update or replace with different hardware What OEM to sen
47. ing with Poolmon Once you find pool tag that is leaking Look up in Windows Debugging Tools subfolder triage pooltag txt May not be there if 3 party driver Run Strings from Sysinternals on all drivers strings windows system32 drivers sys findstr Xyzz Troubleshooting with Driver Verifier Use Driver Verifier to enable pool tracking for all drivers or ones of interest System tracks pool usage by driver Poolmon looks at pool usage by structure tag Verification type I Special pool FF Force IROL checking ow lesouce wulation Verification Status Disabled Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Looking for Leaks Reboot and look at the pool usage of each driver A leaker exhibits the following Current allocations is always close to or equal to the peak The peak grows over time If the leak is significant the peak allocations or bytes will be large f Driver erifier Manager Driver Status Global Counters Pool Tracking Settings Volatile r Individual counters Paged pool Current allocations fees Peak allocations fees Current bytes psa Peak bytes fiis72i20 Not tracked allocations fo Causing a Pool Leak Run NotMyFault and select Leak Pool Allocates paged pool buffers and doesn t free them Stops leaking when you select Stop Leak
48. ings Result return code from Registry operation Other extended information or results File Edit Options Help APO VF we Time Process Request Path Result 04 notepad exe 2704 CreateKey HKCU Software Microsoft iNotepad SUCCESS 04 notepad exe 2704 SetValue HKCU Software Microsoft NotepadilfEscap SUCCESS 04 notepad exe 2704 SetValue HKCU Software Microsoft Notepad lfOrient SUCCESS 5 04 notepad exe 2704 SetValue HKCU Software Microsoft NotepadiilfWeight SUCCESS 04 notepad exe 2704 SetValue HKCU Software Microsoft Notepadilfitalic SUCCESS UCI CafiararalMiernenfiNatansdiiiindarl Chicceed F MA natanad ava 27NA Catala Controlling Regmon Start stop logging Control E Clear display Control X Regedit jump opens Registry Editor and jumps directly to key Double clicking on a line does this Filtering Highlighting Find Save to log file Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Regmon Filtering Normally registry activity should be only at application system startup and exit But sadly lots of processes perform needless registry querying Filtering options Process name or registry path or partial name Success failure read write Regmon Filter Enter multiple filter match strings separated by the character Appl txts P Apply is a wildcard Reset
49. l files Symbol files contain global function and variable names At the minimum get the symbol file s for ntoskrnl exe ntkrnimp exe ntkrnipa exe ntkrpamp exe Symbols are service pack specific and have an installer default directory is winnt symbols Windows NT 4 dbg Windows 2000 dbg pdb Windows XP pdb Note SP symbols only include updates Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Microsoft Symbol Server WinDbg and Kd can download symbols automatically from Microsoft Pick a directory to install symbols and add the following to the debugger s symbol path SRV directory http msdl microsoft com download symbols The debugger automatically detects the OS version of a dump and downloads the symbols on demand Installing the Symbol Files OnCDs Windows NT 4 on Windows NT 4 Setup CD under support debug Windows 2000 SPO Windows XP SPO on Customer Support Diagnostics CD Windows 2000 SP1 on SP1 CD Online Windows NT 4 All US service packs are at ftp microsoft com bussys winnt winnt public fixes usa nt40 Windows 2000 XP http www microsoft com ddk debugging symbols asp Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Automated Analysis When you open a crash dump with Windbg or Kd you get a basic crash analysis
50. lications Environment Subsystems Session Services Exe Application Manager Subsystem DLLs NTDLL DLL System Threads System Service Dispatcher kernel mode callable interfaces Win32 USER GDI Device amp ea T TE z Graphics File Sys Drivers Drivers Kernel JO WUOW aoualajay Ayunoasg Kiowa Ien spe s yL 8 saSsa00ld A s1691 JBW uo einByuog leo np d01d e007 Hardware Abstraction Layer HAL hardware interfaces buses I O devices interrupts interval timers DMA memory cache control etc etc Original copyright by Microsoft Corporation Used by permissidi Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Four Contexts For Executing Code Full process and thread context User applications Win32 Services Environment subsystem processes System startup processes Have thread context but no real process Threads in System process Routines called by other threads processes Subsystem DLLs Executive system services NtReadFile etc GDI32 and User32 APIs implemented in Win32K Sys and graphics drivers No process or thread context arbitrary thread context Interrupt dispatching Device drivers Core Kernel System Files Kernel32 DIl Gdi32 DIl User32 DIl Export Win32 entry points NtDII DII P
51. llocations modified page writer t Modified Page working set List replacement Private pages at process exit Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Memory Management Information Task Manager Performance tab E windows Task Manager File Options View Help Applications Processes iP r CPU Usage CPU Usage History Available sum of free EEE standby and zero page 49 i lists physical Majority are likely standby pages System Cache size of ar See standby list size of Handles 5729 Total 261616 Threads 285 Available 7558 system working set file Processes 34 System Cache 119424 Commit Charge K J Kernel Memory K cache paged pool Total 162400 Tot 45016 Limit 632408 Paged 40192 pageable OS driver code Peak 209732 Nonpaged 5624 amp data MEM Usage Memory Usage History Processes 34 CPU Usage 49 Mem Usage 182400K 632408K 4 Screen snapshot from Task Manager Performance tab Viewing the Paging Lists Only way to get actual size of physical memory lists is to use memusage in Kernel Debugger lkd gt memusage loading PFN database 0 12 392992 2252 Zeroed 0 Free Standby Modified ModifiedNowrite Active Valid Transition Unknown TOTAL 192252 373748 4 0 7690
52. m Threads f System threads are consuming CPU time cannot use WinDbg to attach to process and examine user stack System threads always run in kernel mode No user stack Need to find out what code is running since it could be any one of a variety of components Memory manager modified page writer Swapper File server worker threads Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Examining System Threads With user mode tools 1 PerfMon monitor Processor time for each thread in System process 2 Determine which thread s are running 3 From this get Start address address of thread function in Pviewer 4 Run pstat to find which driver thread start address falls in gt Look for what driver starts near the thread start address Examining System Threads With Kernel Debugger In List Near lt startaddress gt will give name of driver and function Use process or thread to see kernel stack Ikd gt In 8061adb8 8061adb8 nt MiModifiedPageWriter 8061af38 Ikd gt process 4 THREAD 816113e0 Cid 8 50 WAIT Executive KernelMode Non Alertable f5c67d70 NotificationTimer 80482540 SynchronizationEvent 8 68000 Limit f5c65000 Call 0 ChildEBP RetAddr Args to Child f5c67cd8 8042d5a3 ffffffff ff676980 00000000 nt KiSwapThread 0xc5 f5c67d0c 8046355e 00000002 f5c67d98 00000001 nt KeWaitForMultipleObjects 0x266 f5c67d
53. nced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Properties Image tab Description company name version from EXE Full image path Command line used to start process van ae Current directory Microsoft Corporation Parent process Version 10 00 4205 0000 Path User name C Program Files Microsoft Office Office ONPOWERPNT EXE Start time Performance tab Basic process CPU memory usage Security tab Access token groups list privilege list PEE AE Environment tab environment Stated 55605PM_ 1 8 2003 variables Services tab only for service OK processes List of services hosted by process POWERPNT EXE Properties Image Performance Securty Environment Image File Command line C Program Files Microsoft Office Office1 O POWERPNT EXE Current directory CAsysint Kill Process Process Tree Process Explorer Sysinternals www sysinternals com File View Process Handle Options Search Help Banaras amp Process System Idle Process System Z smss exe lesrss exe a winlogon exe services exe svchost exe svchost exe ITMEDevRm exe wuauclt exe D svchost exe T svchost exe T spoolsv exe DFast exe THotkey exe 2 tmesbs3 exe System keeps track of parent child relationship What if parent exits System only keeps track of parent PID If parent exits no way to find its ancestors without a on N g trace of pr
54. ndows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Which Process Is Running Determine which process threads are consuming the most CPU time Quick method Open Task Manager gt Processes Sort processes by CPU usage column Other tools Qslice exe Resource Kit Performance Monitor monitor Processor Time counter in process object for all processes Task Manager Applications vs Processes Applications tab List of top level visible windows Windows are owned b threads Right click on a widow and select Go 6 process Processes tab List of processes Can configure with View gt Select columns Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich E Windows Task Manager File Options View Help Applicationgg Processes Performance Networking Image Name PID taskmgr exe Acrobat exe POWERPNT EXE notepad exe calc exe cmd exe OUTLOOK EXE planner exe TEXPLORE EXE hh exe Netint exe TFNFS exe pinger exe vmnat exe VMnetDHCP exe vmware authd exe 2 Windows Task Manager m Eg File Options View Windows Help Applications Processes Performance Networking Status Running Running Running Running Running Running Running Running Running crosoft PowerPoint dep353 ppt MindManager Troubleshooting Process amp M G9 Command
55. ning Must be due to interrupt related activity Performance counters Processor object Interrupt time time spent processing hardware interrupts DPC time software generated interrupts Can also look at Interrupts sec amp DPCs Queued sec Time Accounting Quirks Looking at total CPU time for each process may not reveal where system has spent its time CPU time accounting is driven by programmable interrupt timer Normally 10 msec 15 msec on some MP Pentiums Thread execution and context switches between clock intervals NOT accounted E g one or more threads run and enter a wait state before clock fires Thus threads may run but never get charged Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time Accounting System Threads Process based code Summary System Threads Functions in OS and some drivers that need to run as real threads E g need to run concurrently with other system activity wait on timers perform background housekeeping work Always run in kernel mode Not non preemptible unless they raise IRQL to 2 or above For details see DDK documentation on PsCreateSystemThread What process do they appear in System process Windows NT 4 0 PID 2 Windows 2000 PID 8 Windows XP PID 4 In Windo
56. nternals and Advanced Troubleshooting Part 3 Troubleshooting Memory Problems Troubleshooting Memory Problems System and process memory usage may degrade performance Or eventually cause process failures How do you determine memory leaks Process vs system How do you know if you need more memory How do you size your page file What do system and process memory counters really mean Understanding process and system memory information can help answer these questions Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows Memory Management Demand paged virtual memory Unit of protection and usage is one page x86 4 KB Itanium 8 KB Pages are read in on demand and written out when necessary to make room for other memory needs Provides illusion of flat virtual address space to each process 32 bit 4 GB 64 bit 16 Exabytes theoretical Supports up to 64 GB 32 bit systems or 512 GB 64 bit systems physical memory Intelligent automatic sharing of memory Process Memory Usage Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Memory Usage Process virtual size By default 2 GB on 32 bit Windows 64 bit Windows 7152 GB Up to 3 GB with Windows NET Enterprise Server USERVA or 3GB Application must be marked large address space aw
57. ocess creations Process Explorer shows orphans left justified TMESRV3 EXE ymware authd exe F VMNetDHCP exe Flyvmnat exe Flisass exe 3 explorer exe Hotsync exe Devntsvc exe 3msmsgs exe ecoocoooooccocooooccoooooeoco H0 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Handle and DLL Views Lower half of display shows either Open handles Loaded DLLs amp mapped files Handle View Sort by handle Objects of type File and Key are most interesting for general troubleshooting DLL View Shows loaded DLLs EXE and any memory mapped files Process Explorer Lab Run Process Explorer Sort on first column Process and note tree view disappears Click on View gt Show Process Tree to bring it back Change update speed to paused Run Notepad In ProcExp hit F5 and notice new process Find value of PATH environment variable in Notepad Exit Notepad In ProcExp hit F5 and notice Notepad in red Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Identify The Image Continued Check Description column in Process Explorer Taken from EXE header Process Explorer Sysinternals www sysinternals com File View Process Handle Options Search Help Basa es amp Process PID tmesbs3 exe 1440 Symware authd exe 1728 FIVMNetDHCP ex
58. ocesse J Applications Proces i CPU Usage l CPU Usage History Memory Usage History p Physical Memory K 6970 Total 65076 431 Available 14724 Processes 39 System Cache 10532 6970 Total 65076 Threads 431 Available 14724 Processes 3 System Cache 10532 Commit Charge K p Kernel M K Commit Charge K Kernel Memory K ommit Charge K p Kernel Memor y K 139076 40 ie Sy i 149344 Paged 45916 2a 5824 Limit Peak 141920 Nonpaged Processes 39 CPU Usage 9 Mem Usage 139076K 149344K_ Spek Mem Usage 139076K 149344K 7 Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Kernel Memory Pools Two system memory pools Nonpaged Pool and Paged Pool Used for systemwide persistent data visible from any process context Pool sizes are a function of memory size amp Server vs Workstation Can be overidden in Registry HKLM System CurrentControlSet Control Session Manager Memory Management Kernel Memory Pools Nonpaged pool Has initial size and upper limit can be grown dynamically up to the max 32 bit upper limit 256 MB on x86 NT4 128MB 64 bit limit 128 GB Paged pool 32 bit upper limit 650MB Windows Server 2003 470MB Windows 2000 192MB Windows NT 4 0 64 bit limit 128 GB Pool size performance counters display current size
59. ols apimon system call and page fault monitoring GUI oh display open handles character cell pviewer processes and threads and security details GUI ptree display process tree and kill remote processes GUI pulist lists processes and usernames character cell pstat process threads and driver addresses character cell qslice can show process relative thread activity GUI Tools from www sysinternals com Process Explorer super Task Manager shows open files loaded DLLs security info etc Pslist list processes on local or remote systems Ntpmon shows process thread create deletes and context switches on MP systems only Listdlls displays full path of EXE amp DLLs loaded in each process Tools We ll Look At Task Manager see what s using CPU Process Explorer Procexp view process details Filemon monitors file I O Regmon monitors registry I O Pssuspend suspends a proces Strings dumps printable strings in files Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Agenda Introduction amp Data Structures Identifying the Process Analyzing Process Thread Activity Application Failures System is busy may be slow What is running A user or system process Interrupt activity What s it doing File I O Network I O Registry calls Application code Wi
60. on Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 3 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 3 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 User looked up what LEX file was Related to Word proofing tools Uninstalled and reinstalled proofing tools amp problem went away Access Denied Many applications don t report access denied errors well Example try to save a file with Notepad to a folder you don t have access to Use Filemon to verify access denied errors are not occurring on file opens Check Result column Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Example Access Denied AOL reported this error
61. opriate file Crash dump portion of paging file is in use during copy so virtual memory can run low Why Crash Dumps Fail Most common reasons Paging file on boot volume is too small Not enough free space for extracted dump Less common The crash corrupted components involved in the dump process Miniport driver doesn t implement dump I O functions Windows 2000 and Windows XP storage drivers must implement dump 1 0 to get a Microsoft signature Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Generating A Test Dump Get BSOD from Sysinternals www sysinternals com ntw2k freeware bluesave shtml t crashes the system by Allocating kernel memory Freeing the memory Raising the IRQL Touching the freed memory Analyzing a Crash Dump There are two kernel level debuggers WinDbg Windows program Kd command line program Same functionality Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Debugging Tools Get the latest from www microsoft com ddk debugging Supports Windows NT 4 Windows 2000 Windows XP Server 2003 Check for updates frequently Don t use older version on install media Install to c Debuggers Easy access from command prompt Symbol Files Before you can use any crash analysis tool you need symbo
62. ory Leaks System says running low on virtual memory Before increasing size of page file look for a process or system memory leak Look for who is consuming pagefile space Process memory leak Check Task Manager Processes tab VM Size column Or Perfmon private bytes same counter Leakyapp Test Program Leakyapp exe is in the Resource Kit Continuously allocates private nonshareable virtual memory When there is no more it just keeps trying Run several copies to fill pagefile more quickly amp Windows Task Manager ma x Eile Options View Help ications Processes Performance Networking Users 13 016 K 1 424K 2 384 K 2 696 K 1 552K Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Handle Leaks Processes that open resources but don t close them can exhaust system memory Check total handle count in Task Manager Performance tab To find offending process on Process tab add Handle Count and sort by that column Using Process Explorer handle view with differences highlighting you can even find which handle s are not being closed Kernel Memory Leaks A driver leaking nonpaged Or a rowing Memory pool shows up as large and Usage and Paged growing Nonpaged pool pool usage usage E windows Task Manager E windows Task Manager File Options view Help File Options View Help Applications Pr
63. ow to folder containing file Double click on a line does this Find finds text within window Save to log file History depth Advanced mode Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Limiting Filemon Output Can set filters for including excluding and highlighting output Filemon Filter Enter multiple filter match strings separated by the character ay is a wildcard Apply Include Exclude Highlight Log Reads Filemon Lab 1 Run Filemon Set filter to only include Notepad exe Run Notepad Type some text Save file as test txt Go back to Filemon Stop logging Set highlight to test txt Find line representing creation of new file Hint look for create operation SOO ee Or ON ee 0 Ne Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Filemon Example While typing in the document Word XP closes without any prompts Filemon log showed this Time Process Request Path Result Other 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 31 48 PM WINWORD EX READ C Program Files Common Files Microsoft Shared Proof MSSP3ES LEX END OF FILE Offset 1251810 Length 457 2 2 31 48 PM WINWORD EX READ C Program Files Comm
64. provided can copy checked NTOSKRNL HAL to a normal Server system Select debug kernel and HAL with Boot ini KERNEL HAL switches See Knowledge base article 314743 HOWTO Enable Verbose Debug Tracing in Various Drivers and Subsystems Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time Accounting System Threads Process based code Summary Interrupt Dispatching user or kernel mode kernel mode code Note no thread or process context switch Interrupt dispatch routine Disable interrupts Interrupt service routine Record machine state trap frame to allow resume Tell the device to stop interrupting Mask equal and lower IRQL Interrogate device state interrupts start next operation on device etc Find and call appropriate Request a DPC ISR Return to caller Dismiss interrupt Restore machine state including mode and enabled interrupts Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich IRQL Interrupt Request Level The precedence of the interrupt Interrupt Precedence Via IRQLs IRQL is also a state of the processor with respect to other interrupts Different interrupt sources have different IRQLs Not the same as IRQ
65. rocesses is not an official Microsoft defined name Win32 Services Let s examine the system process tree Use Tlist T or Process Explorer Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Based NT Code System Startup Processes First two processes aren t real processes Not running a user mode EXE No user mode address space Different utilities report them with different names Data structures for these processes and their initial threads are pre created in NtosKrnl Exe and loaded along with the code Idle Process id 0 Part of the loaded system image Home for idle thread s not a real process nor real threads Called System Process in many displays System Process id 2 8 in Windows 2000 4 in XP Part of the loaded system image Home for kernel defined threads not a real process Thread 0 routine name Phase Initialization launches the first real process running smss exe and then becomes the zero page thread Process Based NT Code System Startup Processes Software Microsoft Windows NT WinLogon Userinit services exe Service Controller also home for many NT supplied services Starts processes for services not part of services exe driven by Registry Machine System CurrentControlSet Services Isass exe Local Security Authentication Server userinit exe Started after logon starts Explorer exe see Sof
66. rovides user mode access to system space routines Also contains heap manager image loader thread startup routine NtosKrnl Exe or NtkrniMp Exe Executive and kernel Includes most routines that run as threads in system process Win32K Sys The loadable module that includes the now kernel mode Win32 code formerly in csrss exe Hal DIl Hardware Abstraction Library drivername Sys Loadable kernel drivers Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich End of Kernel Architecture Next Process amp Thread Troubleshooting Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Windows Internals and Advanced Troubleshooting Part 2 Troubleshooting Processes amp Threads Agenda Introduction to Tools Identifying the Process Analyzing Process Thread Activity Application Failures Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Tools for Obtaining Process amp Thread Information Many overlapping tools most show one item the others do not Built in tools in Windows 2000 XP Task Manager Performance Tool Tasklist new in XP Support Tools pviewer process and thread details GUI pmon process list character cell tlist shows process tree and thread details character cell Resource Kit to
67. ss with that of a failing one use File gt Save As DLL Problems But sometimes it s the order of DLL loads that clues you in so use Filemon Missing DLLs often not reported correctly Look for NOTFOUND or ACCESS DENIED May be opening wrong versions due to files in PATH Look at the last DLL opened before the application died Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Example Problem Word Dies Word97 starts and a few seconds later gets a Dr Watson access violation Customer tried re installing Office still failed Solution Ran Filemon looked at last DLL loaded before Dr Watson It was a printer DLL Uninstalled printer problem went away Example Problem Help Fails The Help command in an application failed on Win95 but worked fine on Win98 ME NT4 Win2000 WinXP Failed with meaningless error message Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Solution Ran Filemon on failing system and working system Reduced log to file opens Compared logs At the point they diverged looked backwards to last common thing done An OLE system DLL was loaded Noticed this OLE DLL was loaded from a directory in the user s PATH on Win95 but from Windows System on other versions Conclusion DLL loaded on Win95 system was
68. st frequently Moved these files to a different disk on a different controller Locked Files Attempting to open or delete a file that is in use simply reports file locked With Process Explorer search in handle view you can determine what process is holding a file or directory open Can even close open files be careful Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Process Explorer Lab Locked File 1 Run ProcExp Click on View gt Update speed change to Paused Run Microsoft Word Create a file called test doc and save it but don t close it From a command prompt try and delete test doc should get file locked In ProcExp hit F5 and then use Search to find open handle to test doc Access Denied on Mapped Files Attempting to delete a DLL or EXE that is in use gets access denied not file locked Can be misleading In Process Explorer DLL View search for file Example try and delete Notepad exe while you re running it Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich DLL Problems DLL version mismatches can cause strange application failures Most applications do a poor job of reporting DLL version problems Process Explorer can help detect DLL versioning problems Compare the output from a working proce
69. t What if image properties say nothing m Version n a Path Exa m j n e O p e n h a n d e S CAWINNT system32 qttask exe Command line y CAWINNT System32 qgttask exe Open files or registry keys May cren seso give a clue Parent explorer exe 1496 Owner DSOLOMON Sdsolomon Process Explorer Sysinternals www sysinternals con f Stated 11 40 38 AM 4 1 2002 Kinnas File View Process Hande Options Search Help B a aa x amp Process TosHKCW exe Pqttask exe 5 2 Handle Type Access Name Ox1C Key Ox000F HKLM 0x54 Key Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Identifying Processes If you still don t Know what the EXE is run Strings on it Dumps printable strings in binary Need to run twice No switches dumps Unicode strings a switch dumps ANSI strings Printable strings may yield clues Registry keys Help error message text Agenda Introduction amp Data Structures Identifying the Process Analyzing Process Thread Activity Application Failures Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Multi service Processes Some processes host multiple services E g Svchost exe Inetinfo exe IIS If still not clear what process is doing need to peer inside process and examine which t
70. tware Microsoft Windows NT CurrentVersion WinLogon Shell and exits hence Explorer appears to be an orphan explorer exe and its children are the creators of all interactive apps smss exe Session Manager The first created process Takes parameters from HKEY_LOCAL_MACHINE System CurrentControlSet Control Session Manager Launches required subsystems csrss and then winlogon csrss exe Win32 subsystem winlogon exe Logon process Launches services exe amp ISass exe presents first login prompt When someone logs in launches apps in Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Win32 Services An overloaded generic term A process created and managed by the Service Control Manager Services exe E g Solitaire can be configured as a service but is killed shortly after starting Similar in concept to Unix daemon processes Typically configured to start at boot time if started while logged on survive logoff Typically do not interact with the desktop Note Prior to Windows 2000 this is one way to start a process on a remote machine now you can do it with WMI Life Of A Service Install time Setup application tells Service Controller about the service Setup CreateService Application System boot initialization SCM reads registry starts Service services as directed Controller Manager Management maintenance Services
71. vice to start if Account doesn t depend on a roaming profile Domain policy that affects logon hasn t changed since last logon Controller for a network logon Creates a process to run HKLM Software Microsoft Windows NT CurrentVersion WinLogon Userinit By default Userinit exe Runs logon script restores drive letter mappings starts shell Userinit creates a process to run HKLM Software Microsoft Windows NT CurrentVersion WinLogon Shell By default Explorer exe There are other places in the Registry that control programs that start at logon Processes Started at Logon Displays order of processes configured to start at log on time Also can use new XP built in tool called System Configuration Utility Torun click on Start gt Help then Use Tools then System Configuration Utility Only shows what s defined to start vs Autoruns which shows all places things CAN be defined to start 3 Aut Svsint l Msconfig utoruns ysinterna s in Windows pchealth helpctr binaries Autoruns Sysinternals www sysinternals com az System Configuration Utility File View Help General SYSTEM INI WIN INI BOOT INI Services Startup Definitions of programs automatically run at user login HKLM SOFTWARE Microsoft Windows NT CurrentVersion Winlogon Userinit E CAWINDOWS system32 userinit exe HKLM SOFTWARE Microsoft Windows CurentVersion RunOnce HKLM SOFTWARE Microsoft Windows CurentVersion RunOnceEx HKLM SOFTW
72. ws 2000 and XP windowing system threads from Win32k sys appear in csrss exe Win32 subsystem process Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Examples Of System Threads Memory Manager Modified Page Writer for mapped files Modified Page Writer for paging files Balance Set Manager Swapper kernel stack working sets Zero page thread thread 0 priority 0 Security Reference Monitor Command Server Thread Network Redirector and Server Worker Threads Threads created by drivers for their exclusive use Examples Floppy driver parallel port driver Pool of Executive Worker Threads Used by drivers file systems Accessed via ExQueueWorkltem Understanding System Threads Later we ll see how to understand what system thread is running when the System process is consuming CPU time Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time Accounting System Threads Process based code Summary Process Based Code OS components that run in separate executables exes in their own processes Started by system Not tied to a user logon Three types Environment subsystems already described System startup processes Note system startup p
73. ws Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Accounting for Kernel Mode Time Processor Time File Edt View Options Help total busy time of Baoe H ali al processor equal to jc elapsed real time 30 Computer ORION OOO idle time a f Processor Time oo 70 Object Processor Instance User Time o Counter DPC Time Privileged Time 50 Interrupt Time Privileged Time tpn 7 40 Processor Time Privileged Time User Time i 30 APC Bypasses sec time spent in kernel mode 20 Color mm z Scale Default z width z Style aE ae 10 Privileged Time 0 includes Last 0 000 Average 0 000 Min 0 000 Max 0 000 Graph Time e Interrupt Time Color Scale Counter Instance Parent Object Con SSS SS SSS Add to Chart DPC Time Data Current Activity Screen snapshot from Programs Explained later Administrative Tools Performance Monitor click on button or select Edit Add to chart 8 Kernel Architecture Process Execution Environment Architecture Overview Interrupt Handling amp Time Accounting System Threads Process based code Summary Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Multiple OS Personality Design Environment Subsystems System User amp Service
74. y be divided into Space code multiple processes for stability reasons Processes can also create shared Physical memory sections Memory Called page file backed file mapping objects Full Windows security It does this automatically for Process 2 shareable pages Address E g code pages in an EXE or DLL Space Viewing the Working Set Working set size counts shared pages in each working set Vadump Resource Kit can dump the breakdown of private shareable and shared pages C gt Vadump o p 3968 Module Working Set Contributions in pages Total Private Shareable Shared Module 14 3 11 0 NOTEPAD EXE 46 0 43 ntdll dl1l 36 35 kerne132 d11 7 5 comdlg32 d1l1 17 15 SHLWAPI d11 44 40 msvert dl1l Windows Internals and Advanced Troubleshooting Copyright 2002 2003 by David A Solomon and Mark E Russinovich Working Set Replacement When working set count working set size must give up pages to make room for new pages This is called a local page replacement policy versus a global replacement policy common on Unix Means that a single process cannot take over all of physical memory unless other processes aren t using it Page replacement algorithm is least recently accessed Windows 2000 only on uniprocessor Windows XP and NET Server All systems Paging Lists Windows Int

Windows Internals and Advanced Troubleshooting

Contents

Download Pdf Manuals

Related Search

Related Contents