Tools that must knowDebug/Trace/Profile
Contents
1. d c o x info m genhtml o report x info m See app report index html for report Profiling Tools event based e valgrind callgrind o Instruction level profiler with cool GUI frontend kcachegrind o Cache branch prediction profiling and annotated source supported m Add g compiler flag if annotated source is wanted o Usecase m gcc g X C OX m valgrind tool callgrind dump instr yes cache sim yes branch sim yes x m kcachegrind callerind a Profiling Tools sampling based e gprof o Timer based IP sampling call event count m Use setitimer ITIMER PROF on Linux m Sampling freqency depends on kernel s HZ setting o Flat report call graph report and annotated source supported o Compiling amp Linking with flag pg m Add g if annoted source is wanted o Usecase m gcc pg g X C 0 X m x gmon out gen d m gprof x see flat call graph report m gprof A x see annotated source Profiling Tools sampling based Behind the scene of gprof e gprof is supposed to use profil syscall for IP sampling but that syscall is not implemented by Linux kernel so it falls back to mimic the syscall with setitimer e pg makes compiler injecting codes calling mcount at the entry of each function which collects call stack info O gcc S x c 0x1 S O gcc S pg x c 0X2 S O vimdiff s e This options also makes linker linking with gcrt o instead of normal crt o which provides sta2. ts Profiling Tools sampling based e Intel VTune Amplifier XE Commercial o PMC timer based sampling support GUI CLI o System wide profiling supported has locks amp waits analysis o Use Pin for instrumentation o CLI works well on Linux GUI not stable m amplxe cl collect hotspots x m amplxe cl report hotspots r rxxxxhs e AMD CodeAnalyst Commercial o oprofile based GUI only o System wide profiling supported o Provide much more events on AMD CPUs o Works not well on Linux Profiling Guideline e Determine target program performance throttle before actual profiling time helps o sys time user time wall clock time m Sys time gt gt user time reduce syscalls user kernel space profiling m User time gt gt sys time user space profiling o sys time user time lt lt wall clock time m Don t use general profiling tool consider user space tracing e Analysis profiling result hierarchically starting from outter scope first don t dive into details too early e Spot performance throttle one by one First deal with the biggest known throttle then profiling again and find the next throttle Trac Ing tools for C Tn 12 LA L L nd 1 25 di i E se ile il z ni a se ae Tracing Tools Implementation e Decouple event recording and exporting ring buffer e User space tracing o Intrusive m Call
3. TIP1 Linux Dev Tools Tips for Preface e Vhat does our world look like There is no remembrance of former things neither shall there be any remembrance of things that are to come with those that shall come after Ecclesiastes 1 11 Ve want need have to change this TIP Technology Inheritance Program Concepts e Debugging Find the cause of unexpected program behavior and fix it e Profiling Analyze program runtime behavior provide Statistical conclusions on key measurements speed resource e Tracing Temporally record program runtime behavior provide data for further debugging profiling All debugging profiling tracing tools depend on some kind of instrumentation mechanism either statical or amica 20 NE pu RE mii cs ma gging tools for C C vim Debugging Tools Implementation e Breakpoint support o Hardware breakpoint m DRO 7 regs on Intel CPU o Software breakpoint m INT3 instruction on x86 x86 64 m raise SIGTRAP signal for portable breakpoint o Virtual Machine Interpreter m Interpret instructions instead of execute it directly e Linux user space debug infrastructure o ptrace syscall Debugging Tools e gdb General purpose debugger o ptrace based o Both hw sw breakpoints supported o Reverse executing feature in 7 x version m Save reg mem op before each instr executed heavy but very handy o Use
4. an insert Statical tracepoints in your code and it does not need patching your kernel m ARA DE STIA iL i ma R Pre ctical ips ha i y ag a i DI A dao ir pa l g l 2 i Ma n es i bar A 1 pid o L s Za Ni i de m Other useful technics e gcc finstrument functions o https github com agentzh dodo tree master utils dodo hook e LD_PRELOAD crash signal handler o https github com chaoslawful phoenix nginx module tree master misc dbg_ jit e Add signal handler to normally output gprof gcov result for a interrupted program See examples at https github com chaoslawful TIP lm References e Overview o Linux Instrumentation o http Iwn net Kernel lndex e racing o 4145 utrace o utrace documentation file o Introducing utrace o Playing with ptrace Part _ o Playing with ptrace Part Il Sami System Tap DTrace LTTng pert Comparison al ite ft race SIT T o Solaris yna iC to DTrace for tama n s Observinc and d Optimizing our Ac plication with DTrace References e Tracing o SystemTap Beginner s Guide o SystemTap Language Reference o SystemTap Tapset Reference o LT Tng recommended bundles o LT Tng Ubuntu daily PPA o An introduction to KProbes o 8 FAKProbesiWain AK Tracing no shortage of options O Uprobes 11th time is the charm o Ptrace Utrace ak sa Lightweigh
5. cases m Standalone debug m gdb args lt exec gt lt argl gt lt gt m Analyze core m gdb lt exec gt lt core gt om Attach to existing process m gdb lt exec gt lt pid gt o Many resources search and learn Debugging Tools e Valgrind family o valgrind is an instruction interpreter vm framework o Impossible to attach to a running process o Useful plugin m memcheck m Memory error detector m massif m Heap usage profiler m helgrind Thread error detector DRD m another Thread error detector m ptrcheck SGCheck m Stack global array overrun detector Debugging Tools e memcheck usecases o Check memory error for all process in hierarchy m valgrind tool memcheck leak check full leak resolution high track origins yes trace children yes log file result log lt exec gt o See flags specified to memchek plugin m valgrind tool memcheck help Debugging Tools e massif usecases o Stats heap and stack usage during a program s life m valgrind tool massif stacks yes lt exec gt mms print massif o In the output of ms_print m means normal snapshot m means detail snapshot m means peak snapshot in all Debugging Tools e helgrind usecase o Check POSIX thread API misuse inconsistent lock order data races m valgrind tool helgrind lt exec gt e DRD usecase o Check POSIX thread API misuse data races lock contention and tracing all mutex activ
6. ion counter and call stack when trigger d o Generate statistically result based on record data NOTE General profiling tools can NOT reveal sleeping interruptible blocking lock wait etc or I O blocking non Interruptible blocking costs But these are usually the main throttle to the intuitive runtime performance Profiling Tools event based e gcov o A coverage testing tool but can also be used as a line count profiling tool user space only o Need statistically instrument target program compiling with one of the following gcc flags m coverage m fprofile arcs ftest coverage o When program exits normally gcda geno file will be generated o Usecase m gcc coverage X C OX m gCOV X C gen X C gCOV m less x c gcov _ Profiling Tools event based Behind the scene of gcov e ftest coverage makes compiler generating gcno files which contains infos to reconstruct basic block graph and assign source codes to blocks used by gcov e fprofile arcs makes compiler injecting codes adding counters associated with each source code line and codes that dump out gcda files when the program exits e See O gcc S x c 0 X1 s O gcc S coverage x C 0 x2 s o vimdiff s Profiling Tools event based e Icov o Graphical gcov front end o Generate beautiful coverage report in HTML format o Usecase m Assuming the source is placed in app x c m cd app m gcc coverage X C OX RO m Icov
7. ities m valgrind tool drd trace mutex yes lt exec gt e ptrcheck usecase o Check stack global array overrun m valgrind tool exp ptrcheck lt exec gt Debugging Tools e Intel Inspect XE Commercial o Cross platform proprietary debugging tools o Both GUI CLI usage supported o Memory thread error detector o Free for non commercial use o Included in Intel Parallel Studio suite standalone download available o Catch up very slow on new hardwares e g i7 o Works not well on Linux platform other platform not tested Debugging Guideline e Generally speaking all programs should pass Valgrind memcheck ptrcheck checking to eliminate most of the memory errors e Multithread programs should pass Valgrind helgrind drd checking to eliminate common racing errors e Valgrind massif can be used to track down the origin of unexpected heap allocation e gdb can be used to manually track down logical bugs in the code e Multiprocess thread programs don t fit gdb well most of the time tracing the program is much easier faster to find the source of a bug than manually gdb debugging ETA suma 4 i Se P ofi iling tools or yr 1C Res ice ma CC rama o Dr adu se hat Profiling Tools Implementation e Event based profiling o Add hook for specified event count event occuring times e Sampling based profiling o Make a repeating trigger for sampling o Record instruct
8. l return printin thread indent 4 lt probefunc c x Tracing Tools customized e LTTng 2 0 o Rewrite of LT Tng 0 9 x no need to patch kernel anymore lighter weight compare to SystemTap o User space tracing is done by inserting statical tracepoint into user program not compatible with System lap D Trace probes yet o Usage m sudo Ittng create sess1 m sudo Ittng enable event a k m sudo Ittng enable event a u m sudo Ittng start m x m sudo Ittng stop m babeltrace Ittng sess1 Tracing Tools customized e DTrace o Origins from Sun Solaris adopted by MacOS FreeBSD Oracle Unbreakable Linux o Scriptable framework light weight tracing overhead o Capable of kernel and user space joint tracing user Space tracing needs inserting statical tracepoints o Handy tracing multiple languages apps m Java Sun PHP Zend Javascript Firefox CPython CRuby MySQL PostgreSQL Erlang D Trace fork o Usecase see http dtracehol com Tracing Guideline e Be warned Tracing needs invovled efforts and solid background on Linux kernel Learn more and deeper about how the system working first e Use SystemTap for kernel user space tracing on Redhat family distros RHEL CentOS Fedora or utrace patched Kernels e Use DTrace for kernel user space tracing on MacOS FreeBSD e User space only tracing can be partially done by strace ltrace _ e LT Tng 2 0 can do kernel user space tracing if you c
9. rtup routine to init sampling timers and resources O gcc v x c grep crt O gcc V pg x c grep crt Profiling Tools sampling based e google perftools CPU profiler o Timer based call stack sampling m Use setitimer ITIMER PROF on Linux m Set sampling fregency through env var PROFILEFREQUENCY o Linked in usage NOTE profiler symbols must be referenced in your code otherwise the dependency of profiler shared library will be eliminated m CCC g X C ox Iprofiler m CPUPROFILE tmp xxx x o Preload usage m LD PRELOAD usr local lib libprofiler so CPUPROFILE tmp xxx x o Show report pprof text x tmp xxx Profiling Tools sampling based e oprofile o Support timer interrupt PMC tracepoint based sampling m PMC PerforMance Counter o Capable of doing system wide profiling o Deprecated in prefer of perf on kernel gt 2 6 26 7 o Usecase m sudo opcontrol init load oprofile module m sudo opcontrol s m x m sudo opcontrol h m sudo opreport show report m sudo opannotate s show annotated src Profiling Tools sampling based e perf o Available on kernel gt 2 6 26 7 o PMC frontend released along with kernel itself o Support PMC tracepoint based sampling o Capable of doing system wide profiling sampling events trace can also be output o Usecase m sudo perf record a g x m sudo perf report show prof report m sudo perf annotate show annotated src an RSS
10. t Dynamic Tracing il for UserApps e racing Book References e Profiling amp Debugging o google perftools Profiling heap usage o google perftools CPU Profiler o Valgrind User Manual o OProfile Manual o Debugging with GDB o GDB Internals Manual o Implementation of GProf o Gcov Data Files Postscript The important thing is not to stop questioning never lose a holy curiosity Albert Einstein
11. tracing API manually need recompiling code o Non intrusive m ptrace syscall m GNU dynamic linker LD AUDIT m utrace patched kernel e Kernel space tracing __o Dynamical mechanism m kprobes jprobes kretprobes trap short jmp instr o Statical mechanism gt m tracepoints manually inserted conditional jump m ftrace kernel gt 2 6 26 gcc mcount utilization Tracing Tools ptrace based e strace o Trace user program s syscalls o Support existing process tracing m Vatch out ptrace protection patch for nonroot proc sys kernel yama ptrace_scope o Works well with multithread programs o Usecase m strace f 1 tt T v s 1024 iC o trace out x m See man strace for detail description Tracing Tools ptrace based e Itrace o Trace user program s dynamic library calls o Can also trace syscalls but can t parse their args as strace did o Neither library gt library nor dlopen d library call trace supported o Can NOT work with multithread programs o Usecase m ltrace C f 1 n4 s1024 S tt T x m See man ltrace for detail description Tracing Tools ptrace based e Ptrace based tracing shortcoming o Heavy overhead at least 2 ctx sw 2 syscall plus signal transit overneads per tracepoint very slow on large tracepoint set o init 1 can not be traced o Processes can not be ptraced by multiple tracers o Ptrace affects the semantics of traced processes m Original parent will not be notified
12. when its child was ptraced and stopped see notes in man 2 ptrace m The overhead of ptrace will lower the num of concurrent running threads Race conditions sensitive to timings may disappear due to this resulting a Heisenberg problem Tracing Tools LD AUDIT based e latrace o Trace user program s dynamic library calls o Can NOT trace existing process o Use callback function running in target process instead of ptrace signals much lower overhead o Works well with multithread programs o Usecase m latrace SAD o trace out x m See man latrace for detail description Tracing Tools ftrace based e trace cmd o Available on kernel gt 2 6 26 o CLI frontend for ftrace framework o System wide kernel tracer no user space event available except for events like context switching scheduling etc but no call site info o Usecase m sudo trace cmd record e all p function graph F x m trace cmd report Tracing Tools ftrace based e kernelshark o GUI viewer for trace cmd result o Usecase m sudo trace cmd record e all p function_graph F x m kernelshark Tracing Tools customized e SystemTap o Linux community s reply to Solaris DTrace o Scriptable framework to utilize kprobes tracepoints o User space tracing needs utrace patched kernel Redhat distros RHEL CentOS Fedora all comes with such Kernels o Usecase m stap e probe syscall printin thread indent 4 gt probefunc probe syscal
Download Pdf Manuals
Related Search