Parallel Debugging Techniques - Great Lakes Consortium for
Contents
1. Compare across Bia value Processes in current group All 64 procs 1 3 Ux00000003 O Threads in current process Process 33 with 3 threads ie ie i nextbutferd 4 1 0000000071 xpression nextbuffer x 8 2 0x00000002 Array mode evaluate for i from al to 5 3 0x00000003 _ Limit comparison to 1 4 sf 6 4 0x00000004 Ei 5 000000005 Compare pare Statistics Visualize i8 es 3 7 0x00000007 0 35 37 43 48 53 58 63 0 1 D B 000000006 44 45 51 55 61 62 2 46 60 4 11 3 000000009 47 49 50 56 57 59 6 D 0000000001 54 8 expression cannot be evalual Use as MPI Rank Create Groups A Scaling to Petascale Virtual Summer School Viewing Editing Data TotalView Viewing data in Stack frame Expression list Variable window dive on a variable by double clicking on its name Editing data by clicking on the value in Stack frame Variable window LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Viewing Editing Data DDT Viewing data in Variable window in the main window Evaluation window Editing data Right click on the variable name in the evaluation window Then choose Edit value or Edit type LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School V2. Ranger Some common types of bugs in parallel programming Programs and instructions can be found at http www cct Ilsu edu lyan1 summerschoolO9 cs I LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Problem DBBEDBBBOBHISEIH e A 1 D periodic array with N elements e Initial value C cell x x 10 Fortran cell x mod x 1 10 e n each iteration all elements are updated with the value of two adjacent elements cell x i cell x 1 i cell x 1 i 10 A Execute Nie iterations The final outputs are the global maximum and average LSU CENTER FOR COMPUTATION http www hpcbugbase org index php Main Page amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Sequential Program Use an integer array to hold current values Use another integer array to hold the calculated values e Swap the pointers at the end of each iteration e The result is used to check the correctness of the parallel programs Chances are that we will not have such a luxury for large jobs p LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School MPI Program DBBEDBBBOBHISEIH BEEBE NEHUMHH LHDDOMEN AAA Process 1 Process 2 Process n e Divide the array among n processes e Each process works on its local array e Exchange boundary data with neighbor processes at th
3. Virtual Summer School Setting Action Points TotalView Breakpoints Right click on a source line Set breakpoint Click on the line number Watch points Right click on a variable Create watchpoint Barrier points Right click on a source line Set barrier Edit action point property Right click on a action point in the Action Points tab Properties LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Setting Action Points DDT e Breakpoints Double click on a source code line Right click in the Breakpoints tab gt Add breakpoint e Watch points Right click on a variable gt Add to Watches Right click in the Watches tab gt Add Watch LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Viewing Editing Data e View values and types of variables Atone process thread Across all processes threads e Edit variable value and type e Array Data LSU 8 3 2009 Slicing Filtering Visualization Statistics CENTER FOR COMPUTATION amp TECHNOLOGY File Edit View Tools Window 6 1 a ES Expression buffer Address 0 005teseD Sparse alice E Filter Actual Type INTEGER 4 painter l2 Type INTEGER 4 pointer DDT Cross Process Comparison View on login3 range
4. local gt B cell_mpid active threads K In kernel 20K Hemd Gl 6 3 xlacal B cell mpi active threads M Mixed H T B local gt B cell mpi 5 2 active threads El 8 fAvocal gt B cell mpi 7 2 active threads R Running E 3 B qb amp a3n B cell mpi active threads Er 10 3 Ghat B cell mpi 2 active threads T Stopped H 1 10 qhBs0 B cell mpi 1U 2 active threads Er 1r 11 qhBs0 B cell mpi 11 e active threads W At watch point H 1B 12 qhBs0 B cell mpi 12 e active threads Er 1f 13 gbat B cell mpi 13 re active threads E Ya 14 qhBs0 H cell mpi 14 rz active threads a Er JE 15 qhBs0 B cell mpi 15 re active threads TotalView ID MPI Rank CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School TotalView Process Window LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 cell_mpl 0 on qbo3L Edr View Group recess Threac Artina Foirt Tools Window Help Grup Curirur 4 HH La HP 3 vj 3 Go Halt Kil Restat Next Step Out Thread 1 182887423 ace raa Stck Trace Stack Frame cell mpi FP Tfbfffe2hl mair rp Tfbfffezdl ho argurents libe start mein FP Tfbfffel Local variables start FP Tfbtffe 3490 requestt 6d 000000040 request O 0x000000007 state INTEGER 4 id etatl INTEGER 4 147 2 0500000002 1 000000001 AUC AS allocate butler inloca lt ellucale rea Lules friltural 2 y ellocate tmp nlocal 2 cE se nlocal myrans
5. on qb631 qe W m P sa 6 Go Hak Hil Restat Mex step Ont sun To Rank 0 cell mpi Stcppec Rian CHER IE 182837423040 feto ppad Trace m A Ree FP TFhfff amp hH L5 mnrEinn rell wpi Rank D FP 7fhfffe2dl Ho arguments Process Wnrkers d iii Local vo el tt 00000040 FP e request x Proceset a y rejuestl 0 0x00000000 Thread 1 1 state INTEGER 4 i43 statl INTEBER 4 id tag2 2 Ox00000002 tajl 1 0x00000001 7 rank right Uljoa MW rank left 61 0x0000003d Group Contro A Group Correo Group Share Function cell mpi in cell m3i bug f3t allocate buffer mlocalt 2 5 allocate fnextbuffer nlocal 2 allocate tmp nlocal 2 4 offset nLocel myrank do x 1 nlaczl 42 futterc xi mod x ottset lt 1U enddo rank left myrank 1 rank right myrank i do n 1 niter da x 4 nlocal 1 ncxtbuvffor x mod bufzor x Li ibuffer i 1 10 enddo tmp gt buffer buffer nextbuffer nextbhuffer gt tmp cell mpi bug tJ Uz3 call mpi lixs Ad 2 cell mpi bug 90834 cell mpi bx320 cell mpi bug f90 52 call mpi DxTzz cell mpi bug 90 57 cell mpis x8 b vell mpi Lwuj f90x63 call mpic zs3zI O bytes Ox005fdaD sum Scope of commands and action points Group control e All processes and threads Group workers All threads that are executing user code Rank X Current process and its threads Process workers User threads in the current
6. process Thread X Y Current thread User defined group e Group Custom Groups or Create in call graph CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Process Thread Groups DDT X Allinea Distributed Debugging Tool v2 3 1 on login3 ranger tacc utexas edu Session Cortrol Search Wiew Help e i O71 Current Group All Focus o1 curen amp Group O Process o Thread f Step Threads Together P Lm Oy Tye NS a e 116 ILS 9 A o aa 18 14 aa 17 t8 te eo fet 22 eine py 46 47 as ao sop At a a a a a 57 a a 60 at a a e Create custom groups Ctri click on all desired processes Right click on the process window then A create group LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Action Points e Breakpoints stop the execution of the processes and threads that reach it Unconditional Conditional stop only if the condition is satisfied Evaluation stop and execute a code fragment when reached Useful when testing small patches Process barrier points synchronize a set of processes or threads TotalView only e Watchpoints monitor a location in memory and stop execution when its value changes A W LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale
7. COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Strategy at Large Scale e Again downscale if possible Reduce the number of processes to which the debugger is attached Reduces overhead Reduces the required number of license seats as well Focuson one or a small number of processes threads Analyze call path and message queues to find problematic processes Control the execution of as few processes threads as possible while keeping others running Provides the context where the error occurs A N LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Trends in Debugging Technology Lightweight trace analysis tools Help to identify processes threads that have similar behavior and reduce the search space Complementary to full feature debuggers Example Stack Trace Analysis Tool STAT Replay Reverse execution ReplayEngine now available from TotalView Checkpointing supported in DDT 2 4 Post mortem statistical analysis Detect anomalies by analyzing profile dissimilarity of multiple runs n N LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Hands on Exercise Debug MPI and OpenMP programs that solve a simple problem to get familiar with Basic functionalities of parallel debuggers TotalView Pople Bluefire and Athena e DDT
8. NCSA Cyberinfrastructure Tutor Debugsing Serial and Parallel Codes course e HPCBugBase http hpcbugbase org index php Main Page A u LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School
9. PUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School How Parallel Debuggers Work e Frontend User processes GUI Debugger engine Debugger Agents Control application processes Send data back Ea to the debugger P engine to analyze LSU CENTER FOR COMPUTATION amp TECHNOLOGY Compute nodes Interactive node GUI 8 3 2009 Scaling to Petascale Virtual Summer School At Very Large Scale The debugger itself becomes a large parallel application Bottlenecks Debugger framework startup cost Communication between frontend and agents Access to shared resources e g file system LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Validation Is Crucial e Have a solid validation procedure to check the correctness e Test smaller components before putting them together a LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School General Parallel Debugging Strategy Incremental debugging Downscale if possible e Participating processes problem size and or number of iterations e Example run with one single thread to detect scope errors in OpenMP programs Add more instances to reveal other issues Example run MPI programs on more than one node to detect problems introduced by network delays A LSU CENTER FOR
10. Parallel Debugging Techniques Le Yan Louisiana Optical Network Initiative ii CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Outline Overview of parallel debugging Challenges Tools Strategies Get familiar with TotalView DDT through hands on exercises a LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Bugs in Parallel Programming e Parallel programs are prone to the usual bugs found in sequential programs Improper pointer usage Stepping over array bounds nfinite loops e Plus LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Common Types of Bugs in Parallel Programming Erroneous use of language features Mismatched parameters missing mandatory calls etc Defective space decomposition Incorrect improper synchronization Hidden serialization LSU CENTER FOR COMPUTATION http www hpcbugbase org index php Main Page amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Debugging Essentials Reproducibility Find the scenario where the error is reproducible Reduction Reduce the problem to its essence Deduction Form hypotheses on what the problem might be e Experimentation Filter out invalid hypotheses Lag LI LSU Terrence Parr Learn The Essen
11. cage cell eeq Ndim gt lt Niter gt n meten I176 v AA mug ee ee MASIA Source code window amp clona i Elea Haee DO cm linen EEN LSU Parallel stack view and output window Evaluation window CENTER FOR COMPUT amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Controlling Execution e The process window TotalView or main window DDT always focuses on one process thread Switch between processes threads TotalView p p t t double click in root window process thread tab DDT click on process rank in process window Need to set the appropriate scope when Giving control commands Setting action points E T 1 i CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Control Commands TotalView Description Pp NW P 33 3 Gn X Halt Kill Restart Next Step Out Fu Go Play Continue Start resume execution Halt Pause Stop execution Kill Terminate the job Restart Restarts a running program Step over Run to the next source line without stepping into another function Step Step into Run to next source line Out Step out Run to the completion of current function Run to Run to line Run to the indicated location A ll LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Process Thread Groups TotalView LSU bit cell_mpi 0
12. co x l nlocal 4z Lucifer fA mud fatal se l 2 10 enddo rank left my asnk 1 rank ightemyrank l co n l nitar do x 2 nlo0ca1l 1 nexthu fer Gc smod buffer x 1 buffer 2414 11 enddo tmp butter bulfer gt naxthubfar nexthuffe gt tmn cell mpi bug 290430 cell mi 0x334 2 cell mpi bug 790434 cell moi Uzx3all cell mpi bug 29052 cell mi OxTaz ccll mpi bug 200457 ccll moi 0x8 7b cel mpi hn TANAN rell mai Nx43F B bytes amp Ox00SfdaTO sum Stack trace pane Call stack of routines Stack frame pane Local variables registers and function parameters Source pane Source code Action points processes threads pane Manage action points processes and threads Scaling to Petascale Virtual Summer School DDT Main Window Allinea Distributed Debugging Tool v2 3 1 on loging ranger tacc utexas edu n x em Curent a E i Variable window Sem Tree t Header Files T a Source Files r nt mainiint argc char arcv int nlocal offset i Ame 1 inl x n mex Q o PR 0 F i double sun 0 local sum 0 numprocs 4205656 1 inl rankmud 13 int numprocs myrank 14 int rank tort rine right 1 1 1 1 1 int ta g 1 tagz 2 tap buffer int nextbuf fer MPI Status Status MPT Request request request MPT_Tnil GShiry Surge MPI Comm runk MP1 COMM WORLD Smyrunk Exit if a of arguments lt z i 3 0 tt U
13. e end of each iteration Ring topology LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School OpenMP Program 01213 la sl6l7l81 la ls Thread O Thread 1 Thread n e Each thread works on its own part of the global array e All threads have access to the entire array so no data exchange is necessary LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Three Ways to Start TotalView DDT BOB x DDT Welcome Welcome to DDT What would you like to do cf Bun and Debug a Program memory debugging H Halt on memory errors Attach toa a Running Program Open a Core File F E ZOALI TO HEY HEIDHTII Cancel e Start with core dumps Start by attaching to one or more running processes Start the executable within TotalView DDT D LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School TotalView Root Window TotalView 8 3 0 1 on qb631 PENTR Status Code Description Blank Exited B At breakpoint Description 1 0 local gt E cell_mpi 0 2 active threads E lt local gt B cell mpi active threads E E B J cell mpi z active threads rror Ea x local H1 in cell mpi H Held EN lt local gt T in read nocancel El 4 3 xlacal B cell mpi 2 active threads H S 4
14. iewing Dynamic Arrays in C C e TotalView e DDT Edit type in the Drag a pointer variable variable window into the evaluation Tell TotalView how to window interpret the memory Right click on the from a starting location variable gt View as e To view an array of 100 Integers EN nt int 100 I LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School MPI dl Queues Message Queue Graph ECCLE n AC qbo31 Em Aptions e Detect Deadlocks Load balancing issues e TotalView Options on qb631 Filter Layout Cyce Detection Dave AS Tools gt Message Queue MM Pending Sends Group Control f G ra h W Pending Receives p E Unexpected h sqs Filter on specified message tags i i MPI_COMM_WO sLD Ranks More options available TET e DDT View gt Message Queues J Show Options amp t Graah Open LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School TotalView Call Graph mn lala Call Graph a out Control Group on tezpur333 nx Group Control y Auto Arrange Tools Call graph Quick view of program state Nodes functions Edges calls e Look for outliers Gi LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School DDT Parallel Stack View e Allow u
15. re dangerously attractive but Need to edit recompile and rerun when additional information is desired May change program behavior Only capable of displaying a subset of the program s state Output size grows rapidly with increasing core count and harder to comprehend Not scalable not recommended LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Compilers Can Help e Most compilers can at runtime Check array bounds Trap floating operation errors Provide traceback information e Relatively scalable but Overhead added Limited capability Non interactive LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School Parallel Debuggers Capable of what serials debuggers can do Control program execution Set action points View change values of variables More importantly Control program execution at various levels Group process thread View MPI message queues at LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School An Ideal Parallel Debugger e Should allow easy process thread control and navigation e Should support multiple high performance computing platforms e Should not limit the number of processes being debugged and should allow it to vary at runtime LSU CENTER FOR COM
16. sers to see the position of each process thread in the source code in same window e Hover over any function to see a list of processes that are currently at gthat location isu CENTER FOR COMPUTATION amp TECHNOLOGY Project Navigator Exil cell mpide Main loop 4 for n 0 n lt Niter n 6 for x21 x nlocal4 x nextbuffer x buffer x 1 buffer x 1 10 8 Header Files Source Files 9 tmp buffer buffer nextbuffer nextbuffer tmp O Exchange boundary elements with neighbors 21 MPI IsendC amp buffer 1 1 MPI INT rank left tag2 MPI COMM WORLD amp 72 nPi AecyCabutfer 0 T MINI rafk ert tani MBI COMA WORLD 75 MPI_WaitC amp requesti1 amp status 76 MPI_WaitC amp request2 amp status Y 3 6 6 6 6 6 6 6 Swap the pointers 6 Find the maximum and average 8 for x21 x lt nlocal 1 x 81 local sum buffer x 82 if buffer x gt 10ca1_max 83 local_max buffer x main cell mpi c 42 main cell mpi c 54 main cell_mpi c 74 32 MS main cell_mpi c 73 Evaluate 8 3 2009 Scaling to Petascale Virtual Summer School References e TotalView user manual http www totalviewtech com support documentation totalview ind ex html e DDT user manual http www allinea com downloads userguide pdf e LLNL TotalView tutorial https computing lInl gov tutorials totalview e
17. tials of Debugging P MUROS http www ibm com developerworks web library wa debug html ca dgr Inxw03Dbug 8 3 2009 Scaling to Petascale Virtual Summer School Challenges in Parallel Debugging Reproducibility Many problems cannot be easily reproduced Reduction Smallest scale might still be too large and complex to handle Deduction Need to consider concurrent and interdependent program instances Experimentation Cyclic debugging might be very expensive l LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School A Nasty Little Bug e What is the integer 4 i ista iend integer 4 chunksize 1024 1024 pote ntia D ro b em P call MPI_Comm_Rank MPI_COMM_WORLD myrank error ista myrank chunksize 1 iend myrank 1 chunksize do i ista iend LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School A Nasty Little Bug A bug that shows integer 4 chunksize 1024 1024 up only when cdi MPI Comm Rank MPI COMM WORLD amp running With more myrank error than 4096 cores integer 4 i ista iend ista myrank chunksize 1 iend myrank 1 chunksize do i ista iend Integer overflow if myrank gt 4096 enddo n LSU CENTER FOR COMPUTATION amp TECHNOLOGY 8 3 2009 Scaling to Petascale Virtual Summer School printf write Debugging e Extremely easy to use therefo
Download Pdf Manuals
Related Search