Exploratory Testing
Contents
1. agrees with that an important theme in oracles Consistency heuristics rely on the quality of your models of the product and its context Coverage Isn t Just Code Coverage aus Test coverage is the amount of the system space that has been tested There are as many kinds of coverage as there are ways to model the product Product Elements Structure Functional Usability Data Security Platform Scalability ality Criteria Operations Q e Time Capability Reliability Performance Maintainability Installability Portability Compatibility ocalizability Supportability Testability 17 Cost as a Simplifying Factor Try quick tests as well as careful tests Te A quick test is a cheap test that has some value but requires little preparation knowledge or time to perform e Happy Path Interruptions e Tour the Product Undermining Sample Data Adjustments Variables Dog Piling Pues Continuous Use Complexity Menus amp Windows Feature Interactions Keyboard amp Mouse Click on Help Happy Path Use the product in the most simple expected straightforward way just as the most optimistic programmer might imagine users to behave Perform a task from start to finish that an end user might be expected to do Look for anything that might confuse delay or irritate a reasonable person Documentation Tour Look in the online help or user manual and find some instructions about how to perform s2. more states existing concurrently Nested dialog boxes and non modal dialogs provide opportunities to do this Continuous Use While testing do not reset the system Leave windows and files open Let disk and memory 18 usage mount You re hoping that the system ties itself in knots over time Cost as a Simplifying Factor Try quick tests as well as careful tests Te A quick test is a cheap test that has some value but requires little preparation knowledge or time to perform Inout Constraint Attack Resource Starvation Click Frenzy Multiple Instances Shoe Test Crazy Configs Blink Test Cheap Tools Error Message Hangover Input Constraint Attack Discover sources of input and attempt to violate constraints on that input For instance use a geometrically expanding string in a field Keep doubling its length until the product crashes Use special characters Inject noise of oy kind into a system and see what happens Use Satisfice s PerlClip utility to create Strings of arbitrary length and content use PerlClip s counterstring feature to create a string that tells you its own length so that you can see where an application cuts off input Click Frenzy Ever notice how a cat or a kid can crash a system with ease Testing is more than banging on the keyboard but that phrase wasn t coined for nothing Try banging on the keyboard Try clicking everywhere dn into a touchscreen system once by poking every square centimeter of ev
3. Exploratory Testing Michael Bolton DevelopSense http www developsense com september 2009 What IS Exploratory Testing OEG e Simultaneous test design test execution and learning e James Bach 1995 But maybe it would be a good idea to underscore why that s important What IS Exploratory Testing OEG e Qeimullancous test design test execution and learning with an emphasis on learning e Com Xaner 2005 But maybe it would be a good idea to be really explicit about what goes on What IS Exploratory Testing k Tur_________________________I e follow and to some degree contributed to Kaner s definition which was refined over several peer conferences through 2007 So maybe it would be a good idea to keep it brief most of the time See Kaner Exploratory Testing After 23 Years www kaner com pdfs ETat23 pdf Testing Isn t Just Checking Se E e Checking is a process of confirming and verifying existing beliefs e Checking can and argue largely should be done mechanically Im very fast but I m slow e It is a non sapient process See http www developsense com 2009 08 testing vs checking html What S Checking e A check has three attributes e It requires an observation e The observation is linked to a decision rule e The observation and the rule can be applied without sapiente Oh no What Is Sapience SO _____________________________
4. Reference Project Specific Status Dashboard A gt Detailed procedural documentation is expensive and largely unnecessary Tutorial documentation is also usually unnecessary but if you do it then keep it separate from the working documents 20 Accountability for Exploratory Testing Session Based Test Management kd e Charter e A clear concise mission for a test session e Time Box e 90 minutes 45 e Reviewable Results asession sheet a test report whose raw data can be scanned parsed and compiled by a tool K e Debriefing a conversation between tester and manager or test lead For more info see http www satisfice com sbtm 21 Charter A Clear Mission for the Session Ca e From one to three sentences May suggest what should be tested how it should be tested and what problems to look for May refer to other documents or information sources A charter is not meant to be a detailed plan General charters may be necessary at first e Analyze the Insert Picture function Create a test coverage outline and risk list to guide future sessions Specific charters provide better focus but take more effort to design e Test clip art insertion Focus on stress and flow techniques and make sure to insert into a variety of documents We re concerned about resource leaks or anything else that might degrade performance over time 22 How To Measure ET Effectivene
5. __m l e A sapient activity is one that requires a thinking human to perform Anon sapient activity can be performed by a machine quickly and precisely e or by a human that has decided NOT to think slowly and fallibly e looks like machines win there right e BUT our job is not merely to test for repeatability but also for adaptability and value Testing IS Exploring Se e Testing as see it is all about exploration discovery investigation and learning e Testing can be assisted by machines but cant be done by machines alone Iti Enor I can t do that s a sapient process but can help you act on your ideas See http www developsense com 2009 08 testing vs checking html Automation Can t recognize qi ri investigate peewee predict em AY T Auge project a contextualize 4 rale beco P strategize question teach 3s reframe work invent scious decisions refine resource Deh troubleshoot collaborate Exploratory Testing The way we practice and teach it exploratory testing IS NOT random testing or IS ad hoc in the dictionary sloppy or slapdash testing sense to the purpose IS NOT unstructured testing e IS structured and rigorous IS NOT procedurally structured e IS cognitively structured IS NOT unteachable e IS highly teachable IS NOT unmanageable IS highly manageable IS NOT scripted e IS chartered IS NOT a technique IS an app
6. ery screen until found a secret utton Shoe Test This is any test consistent with placing a shoe on the keyboard Basically it means using auto repeat on the keyboard for a very cheap stress test Look for dialog boxes so constructed that pressing a key leads to say another dialog box lio S an error message that also has a button connected to the same key that returns to the first dialog box That way you can place a shoe or Coke can as often do but sweeping off a cowboy boot has a certain drama to it on the keyboard and walk away Let the test run for an hour If there s a resource or memory leak this kind of test will expose it Blink Test Find some aspect of the product that produces huge amounts of data or does some operation very quickly For instance look a long log file or browse database records very quickly Let the data go by too quickly to see in detail but notice trends in length or look or shape of the data Some bugs are easy to see this way that are hard to see with detailed analysis Use Excel s conditional formatting feature to highlight interesting distinctions between cells of data Error Message Hangover Make error messages happen and test hard after they are dismissed Often developers handle errors poorly Resource Starvation Progressively lower memory disk space display resolution and other resources until the product collapses or gracefully we hope degrades Multiple Instances Run a lot of instance
7. ht contain or the ways In which their absence might undermine the application Menus and Windows Tour Tour a product looking for all the menus main and context menus menu items windows toolbars icons and other controls Keyboard and Mouse Tour Tour a product looking for all the things you can do with a keyboard and mouse Run through all of the keys on the keyboard Hit all the F keys Hit Enter Tab Escape Backspace Run through the alphabet in order Combine each key with Shift Ctrl and Alt Also click on everything Interruptions Start activities and stop them in the middle Stop them at awkward times Perform stoppages using cancel buttons O S level interrupts ctrl alt delete or task manager arrange for other programs to interrupt such as screensavers or virus checkers Also try suspending an activity and returning later Undermining Start using a function when the system is in an appropriate state then change the state part way through for instance delete a file while it is being edited eject a disk pull net cables or power cords to an inappropriate state This is similar to interruption except you are expecting the function to interrupt itself by i that it no longer can proceed safely Pee ustments Set some parameter to a certain value then at any later time reset that value to something else without resetting or recreating the containing document or data structure Dog Piling Get more processes going at once
8. ks to Chad Wathington for his collaboration on this talk 26 Readings a E Perfect Software and Other Illusions About Testing Quality Software Management Vol 1 Systems Thinking Quality Software Management Vol 2 First Order Measurement Exploring Requirements Quality Before Design e Gerald M Weinberg Lessons Learned in Software Testing e Kaner Bach and Pettichord DevelopSense Web Site and blog hiip www developsense com Michael Bolton Satisfice Web Site and blog htip www satisfice com James Bach Collaborative Software Testing http www kohl ca Jonathan Kohl Quality Tree Software htip www gualitytree com e Elisabeth Hendrickson 2
9. ome interesting activity Do those actions Improvise from them If your product has a tutorial follow it You may expose a problem in the product or in the documentation either way you ve found something useful Even if you don t expose a problem you ll still be learning about the product Sample Data Tour Employ any sample data you can and all that you can the more complex or extreme the better Use zeroes where large numbers are expected use negative numbers where positive numbers are expected use huge numbers where modestly sized ones are expected and use letters in every place that s supposed to handle numbers Change the units or formats in which data can be entered Challenge the assumption that the programmers have thought to reject inappropriate data Variables Tour Tour a product looking for anything that is variable and vary it Vary it as far as possible in every dimension possible Identifying and exploring variations is part of the basic structure of my testing when first encounter a product Complexity Tour Tour a product looking for the most complex features and using challenging data sets Look for nooks and crowds where bugs can hide File Tour Have a look at the folder where the program s EXE file is found Check out the directory structure including subs Look for READMEs help files log files installation scripts cfg ini rc files Look at the names of DLLs and extrapolate on the functions that they mig
10. roach What you do next is governed by what you re learning 10 Contrasting Approaches I aisha Testing Exploratory Testing Is directed from elsewhere e Is directed from within Is determined in advance e Is determined in the moment Is about confirmation e ls about investigation Is about controlling tests e Is about improving test design Emphasizes predictability e Emphasizes adaptability Emphasizes decidability Emphasizes learning Like making a speech e Like having a conversation Like playing from a score e Like playing in a jam session The tester s mind is in control not the script 11 To Learn Excellent Exploratory Testing We Must Learn To Test S Learning how to test in an exploratory way can be challenging because Nobody ever taught us how to test e WHEREAS e Almost nobody enjoys reviewing written test procedures Almost nobody knows how to evaluate the quality of written test procedures e Almost every manager seems to think that written tests are Good Things e THEREFORE e Writing awful test procedures won t get us fired Some companies will even reward us for the poor quality of our test procedures e and That means there is little pressure on us to become excellent testers 12 Exploratory Testing IS Structured asa e Exploratory testing as we teach it is a structured process conducted by a skilled tester or by lesser skilled testers or users working under super
11. s of the app at the same time Open the same files Manipulate them from different windows Crazy Configs Modify the operating system s configuration in non standard or non default ways either before or after installing the product Turn on high contrast accessibility mode or change the localization defaults Change the letter of the system hard drive Consider that the product has configuration options too change them or corrupt them in a way that should trigger an error message or an appropriate default behavior Cheap Tools Learn how to use InCtrl5 Filemon Regmon AppVerifier Perfmon and Process Explorer and Task Manager all of which are free Have these tools on a thumb drive and carry it around Also carry a digital camera now carry a tiny 3 megapixel camera and a tiny video camera Both fit into my coat pockets use them to record screen shots and product behaviors While it s not cheap you can usually find Excel on most Windows systems use it to create test matrices tables of test data charts that display performance results and so on Use the World Wide Web Consortium s HTML Validator at http validator w3c org Pay special attention to tools that hackers use these tools can be used for good as well as for evil Netcat Burp Proxy wget and fuzzer are but a few examples 19 What Does Rapid ET Look Like Concise Documentation Minimizes Waste Testing Heuristics Risk Catalog General Test Strategy
12. ss oauw gt s Track rough percentage of time E oon on e Test design and execution Bug investigation and reporting e Setup ASK why time was spent on each Lots on T might indicate great code but might indicate poor bug finding skill e Lots on B might mean code quality problems but might suggest inefficiency in reporting e Lots on S might mean testability or configuration problems for customers or it might mean early days of testing 23 How To Measure Test Coverage dr T__ _ 6 _ r1rrr_aJ e Testcoverage isn t merely code coverage e Identify quality criteria and identify session time focused on each criterion e Consider product elements structure function data platform operations and time break them down into coverage areas Assess test coverage in terms of e Level 1 Smoke and sanity e Level 2 Common core critical aspects e Level 3 Complex challenging harsh extreme exceptional 24 How To Manage Exploratory Testing Achieve excellent test design by exploring different test designs while actually testing and interacting with the i pr a f E o essi an testers with personal supervision and Product concise documentation of test ideas Meanwhile train them so that they can guide themselves and OF SPEC be accountable for increasingly challenging work 25 Acknowledgements e James Bach hiip satisfice com Cem Kaner hitp www kaner com e Than
13. vision e The structure of ET comes from many sources Test design heuristics Chartering Not procedurally structured but Time boxing cognitively structured Perceived product risks The nature of specific tests The structure of the product being tested _ n other words The process of learning the product Ne Wes TOE x Di but systematic Development activities Constraints and resources afforded by the project The skills talents and Interests of the tester The overall mission of testing 13 Oracles CEE An oracle is a heuristic usually works might fail principle or mechanism by which someone might recognize but not decide conclusively a problem Bug n Something that bugs someone who matters All Test Oracles Are Heuristic Oracles and heuristics are fallible and context dependent Oracles can be contradicted by other oracles Multiple oracles may increase our confidence but even combinations of oracles are fallible There is no single oracle that can tell us whether a program or feature is working correctly at all times and in all circumstances Recognizing a different problem usually requires a different oracle A tester doesn t need to be aware of an oracle in advance of the observation unless the test is designed to be run by rote that is unless it s a check Any time you see a problem you must be using an oracle so what is it 15 Consistency this
Download Pdf Manuals
Related Search