Home

slides - Testing Education

Contents

1. Appropriately complex As the program gets more stable you can hit it with more complex tests and more closely simulate use by experienced users Accountable You can explain justify and prove you ran it Cost his includes time and effort as well as direct costs Opportunity Cost Developing and performing this test prevents you from doing other work Developing Exploratory Testing Skills Copyright Kaner 2006 154 Differences in the emphasis on the goodness of test attributes are the key differences between test techniques Domain testing Focused on non redundancy validity power and variables coverage Tests are typically highly repeatable simple and should be easy to maintain Not focused on representativeness credibility or motivational effect Scenario testing Focused on validity complexity credibility and motivational effect Not focused on power maintainability or coverage Not focused doesn t mean never is It means that this is a factor that we don t treat as critical in developing or evaluating this type of test Developing Exploratory Testing Skills Copyright Kaner 2006 155 How to choose a test technique This is a multi dimensional challenge Your information objectives e Attributes of the potential tests he development context Product elements Quality criteria Risks Project factors constraints and opportunities Developing Exploratory
2. Developing Exploratory Testing Skills Copyright Kaner 2006 70 Spec testing issues What 1s the specification What does the specification say Critiquing the specification what it says How it says what it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal 1ssues Developing Exploratory Testing Skills Copyright Kaner 2006 How it says what it says Ambiguity e Are multiple interpretations possible Likely Adequacy Does it provide enough information for programming documentation and testing Completeness o what extent does it cover the Feature set Use cases Usage scenarios Test relevant information such as boundaries error handling etc Developing Exploratory Testing Skills Copyright Kaner 2006 Ambiguity analysis Many sources of ambiguity in software design amp development In wording or interpretation of specifications or standards n expected response of the program to invalid or unusual input e In behavior of undocumented features In conduct and standards of regulators auditors In customers interpretation of their needs and the needs of the users they represent In definitions of compatibility among 3rd party products Whenever there is ambiguity there is a strong opportunity for a defect Richard Bender teaches this we
3. related learning test design and test execution as mutually supportive activities that run in parallel throughout the project My goal in this tutorial is to help you identify and develop some of the skills involved in effective exploratory testing Developing Exploratory Testing Skills Copyright Kaner 2006 Today s Agenda What do you think exploratory testing is What do you see as the advantages of exploratory testing What do you see as disadvantages of exploratory testing What do you see as the risks of exploratory testing What skills do you think are important for exploratory testing My intent is to focus today s tutorial on the issues you raise rather than following a preset patter e have a context setting introduction and then we ll revisit the questions above really want to do some work on concept mapping as a support for modeling Beyond that the slides that follow are a support structure for the agenda we choose not the agenda we will follow Developing Exploratory Testing Skills Copyright Kaner 2006 Overview REQUIRED MATERIAL THE PREPARED SEQUENCE e A crisis in software testing From information to test Using Updating the practice of quicktests software testing e The challenge of relevance Overview of Exploratory An overview of test techniques testing Scenario testing Using stories as e Analyzing a sample a vehicle for achieving relevance requ
4. is and none other What milestones can best mark your progress How will you know when you are successful How conclusive and specific is your answer Developing Exploratory Testing Skills Copyright Kaner 2006 Context Free Questions Context free process questions e Who is the client e What is a successful solution worth to this client e What is the real underlying reason for wanting to solve this problem e Who can help solve the problem How much time is available to solve the problem Context free product questions e What problems could this product create e What kind of precision is required desired for this product Metaquestions when interviewing someone for info e Am asking too many questions Do my questions seem relevant Are you the right person to answer these questions e Is there anyone else who can provide additional information Is there anything else should be asking Is there anything you want to ask me May return to you with more questions later Developing Exploratory Testing Skills Copyright Kaner 2006 A sample of additional questions based on Gause amp Weinberg s Exploring Requirements p 59 64 An active reading example To find and organize the claims use an active reading approach based on the Heuristic Test Strategy Model Project Factors Quality Product Criteria Elements We ll do this in our next section of the tutorial
5. Exploratory Testing Skills Copyright Kaner 2006 Enhancing the test case from the story We start with Joe and his failure We generate hypotheses for situations that might lead to a failure like that Wild pointer Stack overflow Unusual timing condition Unusual collection of things in the fridge Now the trick is to refine the hypotheses into harsher and harsher tests Until we are satisfied that if the program passes this series of tests the hypothesis under test is probably the wrong one Developing Exploratory Testing Skills Copyright Kaner 2006 16 Enhancing the test case from the story To achieve this we might Look for a potentially promising technique Work up a starting example of this type of test that appears relevant to the failure under consideration ry out the test If you get the failure this simply you can stop Otherwise polish the test gt Consider the strengths of this class of test gt Stretch the test on the attributes not normally emphasized by this technique Developing Exploratory Testing Skills Copyright Kaner 2006 Test Design Some Readings Kaner Bach amp Pettichord Testing EI Techniques in Lessons Learned in A Practitioners Guide fo Software Testing Software Test Design Kaner C 2003 What is a good test case http www testingeducation org a testcase pdf Whittaker What is testing And why is it so
6. Explorers designs can be reusable e Execution Execution can be automated or manual Interpretation VVhat do we learn from program as it performs under our test about the product and about how we are testing the product Developing Exploratory Testing Skills Copyright Kaner 2006 Exploratory testing Learning Anything that can guide us in what to test how to test or how to recognize a problem such as the project context e g development objectives resources and constraints stakeholders with influence market forces that drive the product competitors desired and customary benefits users hardware and software platforms and development history of prior versions and related products risks failure history support record of this and related products and how this product currently behaves and fails Developing Exploratory Testing Skills Copyright Kaner 2006 Examples of learning activities e Study competitive products how they work what they do what expectations they create Research the history of this related products design failures support Inspect the product under test and its data create function lists variable lists data relationship charts file structures user tasks product benefits FMEA Question Identify missing info imagine potential sources and potentially revealing questions interview developers users and other stakeholders fill i
7. hard http www computer org software so2000 pdf s 070 pdf Snnt T TT J012100110000102100 7 02111100200001001 0 1110012B8DAIDUDAU isle em Whittaker amp Atkin Software Engineering is not Enough http www sisecure com pdf jwsasofteng pdf Lee Copeland i aag a r i a P Developing Exploratory Testing Skills Copyright Kaner 2006 163 Scenario testing Developing stories as a vehicle for achieving relevance Developing Exploratory Testing Skills Copyright Kaner 2006 164 Scenario Testing Some Readings Berger Bernie 2001 The dangers of use cases employed as test cases STAR West conference San Jose CA www testassured com docs Dangers htm accessed March 30 2003 Buwalda Hans 20002 The three holy grails of test development presented at EuroSTAR conference Buwalda Hans 2000b Soap Opera Testing presented at International Software Quality Week Europe conference Brussels Collard R 1999 July Developing test cases from use cases Software Testing amp Quality Engineering available at www stickyminds com Kaner C 2003 An introduction to scenario testing http www testingeducation org articles scenario intro ver4 pdf Developing Exploratory Testing Skills Copyright Kaner 2006 Scenario testing The ideal scenario has several characteristics he test is based on a story about how the program is used including information about the mot
8. in that order Causes without effects The case X is greater than Y will trigger special processing Effects without causes f X occurs during processing then Effects with underspecified causes General protection fault Developing Exploratory Testing Skills Copyright Kaner 2006 Common ambiguities Missing facts 2 Unspecified error handling The program will accept up to 3 names Unspecified variables he program will set a flag if this happens What flag Boundaries unspecified or underspecified e Is 0 a positive number If 0 x 100 is valid how big is the maximum value that you will allow to be copied into X for evaluation Whittaker s testing approach rests on programmers being blind to a wide range of unspecified system or program constraints Unspecified quantities he program will compare the value input for X to the maximum allowed Mentioned but undefined cases e The page format dialog will display 3 column width fields at a time The user may not specify more than 10 columns Developing Exploratory Testing Skills Copyright Kaner 2006 7 Ambiguity analysis Break statements into elements Gause amp Weinberg e Mary hada little lamb read the statement several times emphasizing a different word each time and asking what the statement means read that way e Mary conned the trader for each word in the statement substitute a wide range of synonyms a
9. quality of the product Different Help managers make release decisions objectives Block premature product releases require different Help predict and control product support costs testing Check interoperability with other products strategies and Find safe scenarios for use of the product will yield Assess conformance to specifications different tests Certify the product meets a particular standard different loc Ensure the testing process meets accountability standards documentation Minimize the risk of safety related lawsuits and different Help clients improve product quality amp testability test results Help clients improve their processes Evaluate the product for a third party Developing Exploratory Testing Skills Copyright Kaner 2006 22 Defining Testing Empirical technical investigation that we conduct to provide stakeholders with information about the quality Quality is value to some person VVeinberg s definition Note that this is inherently subjective The quality of an item differs from person to person Anything that reduces the value of the product to a stakeholder is a quality related issue Testers look for different things for different stakeholders Developing Exploratory Testing Skills Copyright Kaner 2006 Defining Testing Empirical technical investigation of the product under test that we conduct to provide stakeholders with information about the qu
10. rules hold is vulnerable Do follow up testing to discover serious side effects of the mismatch Developing Exploratory Testing Skills Copyright Kaner 2006 127 Even More QuickTests from Bach s Rapid Testing Course Quick tours of the program Variability Tour Tour a product looking for anything that 1s variable and vary it Vary it as far as possible in every dimension possible Exploring variations is part of the basic structure of Bach s testing when he first encounters a product Complexity Tour Tour a product looking for the most complex features and data Create complex files Sample Data Tour Employ any sample data you can and all that you can The more complex the better Developing Exploratory Testing Skills Copyright Kaner 2006 Even More QuickTests from Bach s Rapid Testing Course Continuous Use While testing do not reset the system Leave windows and files open Let disk and memory usage mount You re hoping the system ties itself 1n knots over time Adjustments Set some parameter to a certain value then at any later tme reset that value to something else without resetting or recreating the containing document or data structure Dog Piling Get more processes going at once more states existing concurrently Nested dialog boxes and non modal dialogs provide opportunities to do this Undermining Start using a function when the system is in an appropriate state then change the state pa
11. still be within your scope Developing Exploratory Testing Skills Copyright Kaner 2006 17 Software testing is an empirical technical Investigation conducted to provide stakeholders with information about the quality of the product or service under test Developing Exploratory Testing Skills Copyright Kaner 2006 It s kind of like CSI CBS com MANY tools procedures um x sources of evidence CRIME SCENE INVESTIGATION THURSDAYS SPM ET PT EN s EST e Tools and procedures don t define an HAND B og K EVIDENCE TOOLS PROCEDURES aris cec investigation or its Agar P T eig NW Ww goals ALS Alternate light source S b b Rande A I e There is too much Analytical balance n SE Em E Anaya scalo a evidence to test tools ETE are often expensive so Ballistic gelatin H n Biohazard bag investigators must Bloody print enhancer vu Pena ER exercise judgment Bromoform The investigator must pick what to study and CS HANDBOOK PREVIOUSLY ON CSI CSI VIDEO Learn more about Room Service E Behind the Scenes pols evidence and m i A Julian Harper 23 touted as the next MOM c51 celebrates 100 episodes how in order to reveal a SS Brad Pitt is taking his bright lights Sls moment to the max Sex drugs and h d d EPISODES a High Roller Suite are put at his t e most nee e a e disposal as he enters the more f 2 Aie a intTormation Catch up
12. supporting oracles Data capture notes Screen input capture tool Log files Ongoing automated assessment of test results Charter Decide what you will work on and how you will work Developing Exploratory Testing Skills Copyright Kaner 2006 39 Examples of execution activities Configure the product under test Branch backtrack Let yourself be productively distracted from one course of action in order to produce an unanticipated new idea Alternate among different activities or perspectives to create or relieve productive tension Pair testing work and think with another person on the same problem Vary activities and foci of attention Create and debug an automated series of tests e Run and monitor the execution of an automated series of tests Developing Exploratory Testing Skills Copyright Kaner 2006 Interpretation activities e Part of interpreting the behavior exposed by a test is determining whether the program passed or failed the test A mechanism for determining whether a program passed or failed a test is called an oracle We discuss oracles in detail on video and in slides at http www testingeducation org BBS T BBSTIntrol html Oracles are heuristic they are incomplete and they are fallible One of the key interpretation activities is determining which oracle is useful for a given test or test result Developing Exploratory Testing Skills Copyright Kaner 2006 I
13. that the seller have a specific intention to undertake an obligation but an affirmation merely of the value of the goods or a statement purporting to be merely the seller s opinion or commendation of the goods does not create an obligation Developing Exploratory Testing Skills Copyright Kaner 2006 93 Using the Satisfice Heuristic Test Strategy Model to guide analysis Developing Exploratory Testing Skills Copyright Kaner 2006 94 Reviewing d document with the Heuristic Test Strategy Model he last section has many slides on active reading In the last exercise we reviewed the requirements document on its own terms We see what is there and come to understand it better Active readers often operate from a different organizational structure fitting the information from the document under review into the structure they are trying to fill rather than being bound by the structure of the document We demonstrate what active reading is about in this exercise by using an independently created structure the Heuristic Test Strategy Model as the base document and reviewing the specification in terms of how well we can map its information onto the information structure of HSTM Developing Exploratory Testing Skills Copyright Kaner 2006 Heuristic Test Strategy Model Authored by James Bach e O years of critical peer review by colleagues e Several of us have found this a very useful tool for Gu
14. the product should work Her task is to expose credible concerns to the stakeholders he tester doesn t have to make the product design tradeoffs She exposes the consequences of those tradeoffs especially unanticipated or more serious consequences than expected The tester doesn t have to respect prior agreements Caution testers who belabor the wrong issues lose credibility he scenario tester s work need not be exhaustive just useful Developing Exploratory Testing Skills Copyright Kaner 2006 168 Risks of scenario testing Other approaches are better for testing early unstable code e A scenario is complex involving many features If the first feature is broken the rest of the test can t be run Once that feature is fixed the next broken feature blocks the test est each feature in isolation before testing scenarios to efficiently expose problems as soon as they appear Scenario tests are not designed for coverage of the program t takes exceptional care to cover all features or requirements in a set of scenario tests Statement coverage simply isn t achieved this way Reusing scenarios may lack power and be inefficient e Documenting and reusing scenarios seems efficient because it takes work to create a good scenario Scenarios often expose design errors but we soon learn what a test teaches about the design Scenarios expose coding errors because they combine many features and much data To cov
15. word or statement whereas e g means exampli gratia for example Ambiguous quantities e Within between up to almost on the order of Impossible promises e The program will be fully tested Performance will be instantaneous Developing Exploratory Testing Skills Copyright Kaner 2006 Common ambiguities Logical conditions Incomplete set of logical conditions If A and B then C If A and not B then D What about B and not A Logical operators ambiguously grouped If A and Bor C then D s this A and B or C Is it A and B or C Just because precedence orders are defined by convention doesn t mean that the spec author the spec reviewers and the programmers know them Negation without explicit specification of scope If not A and B then D Is this Not A and B Is it Not A and B Is it Not A and Not B There are plenty more of these Look at any logic text Developing Exploratory Testing Skills Copyright Kaner 2006 Common ambiguities Missing facts 1 Unspecified decision maker e f X is unacceptable then Unacceptable according to who Assumes facts not specified e Spec assumes the reader is familiar with the specifics of regulations environmental constraints etc These might change or differ across countries platforms etc Ambiguity in time Does X have to precede Y In the statement Do A if X happens and Y happens and Z happens does it matter if they happen
16. 6 Additional QuickTests Explore data relationships Field Entry Source Variable 1f 47y way you can change values in V1 After VI amp V2 are brought to incompatible values what are all the Ways to display them Any way you can change values in V2 Variable 2 Related Variable Relationship After VI amp V2 are brought to incompatible values what are all the ways to display or use them Variable 2 Constraint to a range Variable 1 Constraint to a range We discuss this table more in our lectures on combination testing Developing Exploratory Testing Skills Copyright Kaner 2006 126 Additional QuickTests Explore data relationships continued Many possible relationships For example e VI lt V2 K VI is constrained by V2 K e VI f V2 where f is any function e VI is an enumerated variable but the set of choices for VI is determined by the value of V2 Relations are often reciprocal so if V2 constrains VI then VI might constrain V2 try to change V2 after setting VI Given the relationship e Try to enter relationship breaking values everywhere that you can enter VI and V2 Pay attention to unusual entry options such as editing in a display field import revision using a different component or program Once you achieve a mismatch between VI and V2 the program s data no longer obey rules the programmer expected would be obeyed so anything that assumes the
17. 6 130 Even More QuickTests from Bach s Rapid Testing Course Cheap Tools Learn how to use InCtrl5 Filemon Regmon App Verifier Perfmon Task Manager all of which are free Have these tools on a thumb drive and carry it around Also carry a digital camera Bach carries a tiny 3 megapixel camera and a tiny video camera in his coat pockets He uses them to record screen shots and product behaviors Elisabeth Hendrickson suggests several additional tools at http www bughunting com bugtools html Resource Starvation Progressively lower memory and other resources until the product gracefully degrades or ungracefully collapses Play Writer Sez Look in the online help or user manual and find instructions about how to perform some interesting activity Do those actions Then improvise from them Often writers are hurried as they write down steps or the software changes after they write the manual Developing Exploratory Testing Skills Copyright Kaner 2006 13 Even More QuickTests from Bach s Rapid Testing Course Crazy Configs Modify O S configuration in non standard or non default ways either before or after installing the product Turn on high contrast accessibility mode or change the localization defaults Change the letter of the system hard drive Grokking Find some aspect of the product that produces huge amounts of data or does some operation very quickly For instance look a long log file or browse datab
18. Developing Skills as an Exploratory Tester Quality Assurance Institute November 2006 Cem Kaner J D Ph D Professor of Software Engineering Florida Institute of Technology Copyright c Cem Kaner 2006 This work is licensed under the Creative Commons Attribution ShareAlike License To view a copy of this license visit http creativecommons org licenses by sa 2 0 or send a letter to Creative Commons 559 Nathan Abbott Way Stanford California 94305 USA These notes are partially based on research that was supported by NSF Grant EIA 01 13539 ITR SY PE Improving the Education of Software Testers Any opinions findings and conclusions or recommendations expressed in this material are those of the author s and do not necessarily reflect the views of the National Science Foundation Much of the material in these slides was provided or inspired by James Bach Michael Bolton Jonathan Bach and Mike Kelly Andy Tinkham contributed significantly to the planning and design of this tutorial Developing Exploratory Testing Skills Copyright Kaner 2006 Session Blurb Software testing is an empirical technical investigation that we conduct to provide stakeholders with information about the quality of the product or service under test Exploratory testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test
19. T Several of the tests we listed at LAWST 7 Los Altos Workshop on Software Testing 1999 are equivalent to the attacks later published by Whittaker He develops the attacks well and we recommend his descriptions LAWST generated several other quicktests including some that aren t directly tied to a simple fault model Many of the ideas in these notes were reviewed and extended by the other LAWST 7 attendees Brian Lawrence Ill Jack Falk Drew Pritsker Jim Bampos Bob Johnson Doug Hoffman Chris Agruss Dave Gelperin Melora Svoboda Jeff Payne James Tierney Hung Nguyen Harry Robinson Elisabeth Hendrickson Noel Nyman Bret Pettichord amp Rodney Wilson We appreciate their contributions Developing Exploratory Testing Skills Copyright Kaner 2006 116 Additional QuickTests Interference testing We look at asynchronous events here One task is underway and we do something to interfere with it In many cases the critical event is extremely time sensitive For example An event reaches a process just as just before or just after it is timing out or just as before during after another process that communicates with it will time out listening to this process for a response Just as if special code is executed in order to accomplish the handling of the timeout just as means during execution of that code An event reaches a process just as just before or just after it is servicing some
20. Testing Skills Copyright Kaner 2006 lest design A simplifying model for classifying and generating test techniques Developing Exploratory Testing Skills Copyright Kaner 2006 13 lesting combines techniques that focus on Testers who does the testing Coverage what gets tested Potential problems why you re testing what risk you re testing for Activities how you test Evaluation how to tell whether the test passed or failed Artefact What you will report All testing involves all five dimensions Individual techniques focus on I or 2 dimensions leaving the others float free Developing Exploratory Testing Skills Copyright Kaner 2006 158 Example of technique emphasis What is the difference between User testing Usability testing User interface testing Developing Exploratory Testing Skills Copyright Kaner 2006 Getting back to relevance If you don t have a technique at hand you will often have to invent your own Or at least polish a test idea into a good test This is especially true with stories that give an initial approach to a risk but need work Example Joe bought a smart refrigerator that tracks items stored in the fridge and prints out grocery shopping lists One day Joe asked for a shopping list for his usual meals in their usual quantities and the fridge crashed with an unintelligible error message How would you test for this bug Developing
21. age Trace tests BACK TO the specification with traceability matrices Developing Exploratory Testing Skills Copyright Kaner 2006 Traceability matrix ae r Totals Developing Exploratory Testing Skills Copyright Kaner 2006 Traceability matrix The columns involve different test items A test item might be a function a variable an assertion in a specification or requirements document a device that must be tested any item that must be shown to have been tested The rows are test cases The cells show which test case tests which items If a feature changes you can quickly see which tests must be reanalyzed probably rewritten In general you can trace back from a given item of interest to the tests that cover it This doesn t specify the tests it merely maps their coverage Traceability tool risk test case management tools can drive you into wasteful over documentation and unmaintainable repetition Developing Exploratory Testing Skills Copyright Kaner 2006 90 Spec testing issues What 1s the specification What does the specification say Critiquing the specification what it says How it says what it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal issues Developing Exploratory Testing Skills Copyright Kaner 2006 Legal issues Warranties ba
22. ality of the product or service under test e The product includes the data the documentation the hardware whatever the customer gets If it doesn t all work together it doesn t work he product is a solution space to a set of problems If it doesn t solve the problem s it doesn t work Developing Exploratory Testing Skills Copyright Kaner 2006 Testing is always done within a context e Testing is done in the face of harsh constraints Complete testing is impossible The project schedule and budget are finite The skills of the testing group are limited e Testing is done on behalf of stakeholders Project manager marketing manager customer programmer competitor attorney Which stakeholder s this time gt What information are they interested in gt What risks are they trying to mitigate Testing might be done before during or after a release e Improvement of product or process might or might not be an objective of testing Developing Exploratory Testing Skills Copyright Kaner 2006 Example of context A thought experiment Suppose you were testing a program that does calculations like a spreadsheet Consider 4 development contexts Computer game that uses the spreadsheet for occasional tasks like bargaining with another player Early development of a commercial product at the request of the project manager to help her identify product risks and help her progr
23. ammers understand the reliability implications of their work Late development of a commercial product to help the project manager decide whether the product is finished Control the operation of medical equipment or collect and store the results of research on the operational safety of the equipment Developing Exploratory Testing Skills Copyright Kaner 2006 A thought experiment slide 2 For each context What is your mission How could you organize testing to help you achieve the mission How aggressively should you hunt for bugs Why Which bugs are less important than others Why How important are issues of performance speed of operation Polish of the user interface Precision of the calculations Prevention and detection of tampering with the data How extensively will you document your work Why What other information would you expect to provide to the project if any Why Developing Exploratory Testing Skills Copyright Kaner 2006 2 Fxamples of important context factors e Who are the stakeholders with How to decide what result influence e What are the goals and qualit l criteria for i p 1 yo How to decide what other result variables to pay attention to variables to pay attention to in the event of intermittent failure e What skills and resources are available to the project What is in the product How it could fail How to troubleshoot and simplif
24. ase management bureaucracy to slow us down further e Look at the National Defense Industrial Association s Systems Engineering Workshop on the Top 5 Software Issues VVashington D C August 2006 https acc dau mil GetAttachment aspx id I9000 amp pname file amp aid 24945 Developing Exploratory Testing Skills Copyright Kaner 2006 Another Traditional Focus of Testing Sind Software Errors But an error May or may not be a coding error e May or may not be a functional error The tester who looks only for coding errors misses all of the other ways in which the program is of lower quality than it should be This was accepted by most good testing practitioners that knew as far back as 1983 Developing Exploratory Testing Skills Copyright Kaner 2006 12 A slightly less traditional view Developing Exploratory Testing Skills Copyright Kaner 2006 I3 So what does the future look like Five trends seem obvious to me Context awareness in test planning no best practices Test driven programming or other extensive unit testing Test design readily derived from state models or other automatable models High volume test automation Exploratory testing e think these are important but insufficient These are reactions extensions to 980 s style testing he next generation reactions extensions to modern practice instead of 980 s practice will develop
25. ase records very quickly Let the data go by too quickly to see in detail but notice trends in length or look or shape of the patterns as you see them Developing Exploratory Testing Skills Copyright Kaner 2006 Parlour Tricks are not Risk Free These tricks can generate lots of flash in a hurry e The DOS disk I O example The Amiga clicky click click click example As political weapons they are double edged If people realize what you re doing you lose credibility Anyone you humiliate becomes a potential enemy Some people incorrectly characterize exploratory testing as if it were a collection of quicktests As test design tools they are like good candy e Yummy Everyone likes them Not necessarily nutritious You may never get to the deeper issues of the program Developing Exploratory Testing Skills Copyright Kaner 2006 133 The challenge of relevance Developing Exploratory Testing Skills Copyright Kaner 2006 134 The relevance problem as a test design problem We often go from technique to test Find all variables domain test each Find all spec paragraphs make a relevant test for each Find all lines of code make a set of tests that collectively includes each It is much harder to go from a risk to a test The program will crash The program will have a wild pointer The program will have a memory leak The program will be hard to use The program will
26. at is the specification What is a specification For our purposes we include any document that describes the product and drives development sale support or the purchasing of the product What is the scope of this specification e Some specs cover the entire product others describe only part of it such as error handling e Some specs address the product from multiple points of view others only from one point of view Do we have the right specification Right version e Source control Do we verify version File compares Developing Exploratory Testing Skills Copyright Kaner 2006 What is the specification Is this a stable specification e Is it under change control Should it be Supplementary information assumed by the specification writer e Some aspects of the product are unspecified because they are defined among the staff perhaps in some other uncirculated document Implicit specifications e Some aspects of the product are unspecified because there are controlling cultural or technical norms These are particularly important Rather than making an unsupported statement that It s bad e g users won t like it you can justify your assertions Developing Exploratory Testing Skills Copyright Kaner 2006 5 Implicit specifications e Whatever specs exist e Software change memos that come with each new internal version of the program User manual draft and pre
27. ation Exploring stored data e Attack Apply inputs using a variety of initial conditions e Attack 12 Force a data structure to store too many or too few values e Attack 13 Investigate alternate ways to modify internal data constraints From Whittaker How to Break Software Developing Exploratory Testing Skills Copyright Kaner 2006 Attacks to expose common coding errors Testing from the user interface Data and computation Exploring computation and feature interaction e Attack 14 Experiment with invalid operand and operator combinations e Attack 15 Force a function to call itself recursively e Attack l6 Force computation results to be too large or too small e Attack 7 Find features that share data or interact poorly From Whittaker How to Break Software Developing Exploratory Testing Skills Copyright Kaner 2006 Attacks to expose common coding errors System interface attacks Testing from the file system interface Medta based attacks e Attack I Fill the file system to its capacity e Attack 2 Force the media to be busy or unavailable e Attack 3 Damage the media Testing from the file system interface File based attacks e Attack 4 Assign an invalid file name e Attack 5 Vary file access permissions e Attack 6 Vary or corrupt file contents From Whittaker How to Break Software Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests from LAWS
28. author apply it Analysis Can you does the author break down an argument or concept into smaller pieces Synthesis Does the author or can you bring together several facts ideas concepts into a coherent larger concept or a pattern More along these lines come from Bloom s taxonomy Developing Exploratory Testing Skills Copyright Kaner 2006 The classic context free questions Traditional news reporters questions Who What When Where How Why For example Who will use this feature What does this user want to do with it Who else will use it Why Who will choose not to use it What do they lose What else does this user want to do in conjunction with this feature Who is not allowed to use this product or feature why and what security is in place to prevent them We use these in conjunction with questions that come out of the testing model see below The model gives us a starting place We expand it by asking each of these questions as a follow up to the initial question Developing Exploratory Testing Skills Copyright Kaner 2006 Using context free questions to define a problem What is the unknown Developing Exploratory Testing Skills Why is it necessary to solve the problem What benefits will you receive by solving the problem What is it that you don t yet understand What is the information that you have What is the source of this problem Specs Field e
29. codes These 2 are common general purpose means I have a question about this means new or interesting idea Spot patterns and make connections Create information maps Relate new knowledge to old knowledge Plan for your retention of the material SQ3R survey question read recite review al no Developing Exploratory Testing Skills Copyright Kaner 2006 Acive Reading Cubing Cubing involves attacking a problem from 6 perspectives Originally developed as a writing strategy it s often suggested for active reading as well For the feature or concept that you are trying to understand 2 3 Describe it describe its physical attributes size shape etc and its functional attributes Compare it What s it similar to Why do you think so Associate it What other ideas products etc does it bring to mind Analyze it Break it down into its components How are they related How do they work together Apply it What can you or the user do with it Evaluate it Take a stand List reasons that it is good good feature good implementation good design good idea etc or bad If you want to be neutral make two lists one of all che ways that it s good the other of all the ways that it s bad As you develop your cube work through the specification and any other documents you have to collect the information you need to do these tasks Developing Exploratory Testing Skills C
30. corrupt its database Developing Exploratory Testing Skills Copyright Kaner 2006 The relevance problem as a test design problem he challenge of exploratory testing is often to take a test idea especially potential problem maybe learned from study of competitor s product or support history or failure of other products on this operating system or written in this programming language e And turn the test idea into one or more tests How do we map from a test idea to a test Developing Exploratory Testing Skills Copyright Kaner 2006 136 How do we map from a test idea to a test don t have a general answer Cross mapping of knowledge is one of perhaps the most difficult cognitive tasks Ability to do this is often discussed in terms of G general intelligence the hypothetical dominant factor that underlies IQ scores Developing Exploratory Testing Skills Copyright Kaner 2006 How do we map from a test idea to a test When it is not clear how to work backwards to the relevant test four tactics sometimes help Ask someone for help Ask google for help Look for discussions of the type of failure look for discussions of different faults and see what types of failures they yield Review your toolkit of techniques searching for a test type with relevant characteristics Turn the failure into a story and gradually evolve the story into something you can test fr
31. create How would you create these reports objects whatever in your application Look for sequences People or the p typically do task X in an order What are the most common orders sequences of subtasks in achieving X Developing Exploratory Testing Skills Copyright Kaner 2006 170
32. ee https www uwsp edu education Iwileon curric newtaxonomy htm for an overview and links ancepts Taxenanmy mrmap 6 13 20D6 From information to test Using quicRtests Developing Exploratory Testing Skills Copyright Kaner 2006 105 QuickTests A quicktest is a cheap test that has some value but requires little preparation knowledge or time to perform Participants at the 7th Los Altos Workshop on Software Testing Exploratory Testing 1999 pulled together a collection of these James Whittaker published another collection in How to Break Software Elisabeth Hendrickson teaches courses on bug hunting techniques and tools many of which are quicktests or tools that support them Developing Exploratory Testing Skills Copyright Kaner 2006 A Classic QuickTest The Shoe Test Find an input field move the cursor to it put your shoe on the keyboard and go to lunch Basically you re using the auto repeat on the keyboard for a cheap stress test e Tests like this often overflow input buffers In Bach s favorite variant he finds a dialog box so constructed that pressing a key leads to say another dialog box perhaps an error message that also has a button connected to the same key that returns to the first dialog box e This will expose some types of long sequence errors stack overflows memory leaks etc Developing Exploratory Testing Skills Copyright Kaner 2006 Another Clas
33. er more combinations we need new tests Do regression testing with single feature tests or unit tests not scenarios Developing Exploratory Testing Skills Copyright Kaner 2006 169 Sixteen ways to create good scenarios Write life histories for objects in the system How was the object created what happens to it how is it used or modified what does it interact with when is it destroyed or discarded List possible users analyze their interests and objectives Consider disfavored users how do they want to abuse your system List system events How does the system handle them List special events What accommodations does the system make for these List benefits and create end to end tasks to check them Look at the specific transactions that people try to complete such as opening a bank account or sending a message VVhat are all the steps data items outputs displays etc What forms do the users work with Work with them read write modify etc Interview users about famous challenges and failures of the old system Work alongside users to see how they work and what they do Read about what systems like this are supposed to do Play with competing systems Study complaints about the predecessor to this system or its competitors Create a mock business Treat it as real and process its data Try converting real life data from a competing or predecessor application Look at the output that competing applications can
34. esting implement execute and interpret a large series of tests Function testing Domain testing Risk based testing You set the wheel in motion supply the Regression testing oracle s and of results e Scenario testing User testing State model based testing e HIGH VOLUME AUTOMATED TESTING Developing Exploratory Testing Skills Copyright Kaner 2006 To different degrees good tests have these attributes Power When a problem exists the test will reveal it Valid When the test reveals a problem it is a genuine problem Value It reveals things your clients want to know about the product or project Credible Your client will believe that people will do the things that are done in this test Representative of events most likely to be encountered by the user xref Musa s Software Reliability Engineering Non redundant his test represents a larger group that address the same risk Motivating Your client will want to fix the problem exposed by this test Performable It can be performed as designed Maintainable Fasy to revise in the face of product changes Repeatable It is easy and inexpensive to reuse the test Pop short for Karl Popper It reveal things about our basic or critical assumptions Coverage lt exercises the product in a way that isn t already taken care of by other tests Easy to evaluate Supports troubleshooting Provides useful information for the debugging programmer
35. eview as for initial planning Developing Exploratory Testing Skills Copyright Kaner 2006 Using and developing models in software testing Developing Exploratory Testing Skills Copyright Kaner 2006 100 Models A model is a simplified formal representation of a relationship process or system The simplification makes some aspects of the thing modeled clearer more visible and easier to work with All tests are based on models e Many of our models are implicit e When the behavior of a program feels wrong it is clashing with your internal model of the program and how it should behave Developing Exploratory Testing Skills Copyright Kaner 2006 Madels mmap 11 13 2D085 Characteristics of a model What will we use this model for How can we evaluate the model Other aspects of the model Frequently used software models What heuristics help us construct the model A taxonomy model instructional design assessment COGNITIVE PROCESSES ew LL 9S 1 1 Cwe 9 1 1 we O 0 7 1 Developing Exploratory Testing Skills Copyright Kaner 2006 103 Facis Concepts Procedures Cognitive strategies 1 Models Skills Attitudes Metacognition This is an early version of a learning taxonomy that Cem Kaner and James Bach use in their instructional design The taxonomy is based on the Anderson Krathwohl update to Bloom s taxonomy S
36. ht Kaner 2006 Basics of active reading Adler M J and van Doren C 1972 How to Read a Book http radicalacademy com adlermarkabook htm http www justreadnow com strategies active htm http www somers k 2 ny us intranet reading PLAN html http www mindtools com pages article newlSS_04 htm http www clt astate edu bdoyle T extbookRdng ppt http titan iwu edu writcent Active_Reading htm M R J ADLER http istudy psu edu FirstYearModules Reading Materials html http www itrc ucf edu forpd strategies stratCubing html http www ncrel org sdrs areas issues students learning Ir2befor htm Developing Exploratory Testing Skills Copyright Kaner 2006 Active reading Prioritize what you read by Surveying read table of contents headings abstracts e Skimming read quickly for overall sense of the material e Scanning seek specific words or phrases Search for information in the material you read by Asking information gathering questions and search for their answers Creating categories for information and read to fill in the categories Questioning challenging probing what you re reading Organize it Read with a pen in your hand If you underline or highlight don t do so until AFTER you ve read the section Make notes as you go Key points Action items Questions Themes Organizing principles Use concise codes in your notes especially on the book or article Make up 4 or 5 of your own
37. iding exploration see Bach s and Bolton s courses Structuring a failure mode and effects analysis gt See Giri Vijayaraghavan amp Cem Kaner Bug taxonomies Use them to generate better tests at http www kaner com pdfs Bug I axonomies pdf and Giri s thesis A Taxonomy of E Commerce Risks and Failures at http www testingeducation org a tecrf pdf gt Another thesis on mobile wireless apps coming soon by Ajay Jha Specification analysis my primary use of the model Developing Exploratory Testing Skills Copyright Kaner 2006 An active reading example To find and organize the claims use an active reading approach based on the Heuristic Test Strategy Model Project Factors As you read the spec e Start from the assumption that every sentence in the spec is meant to convey information Quality Product Criteria Elements Take four writing pads mark them Project Product Quality and To Do On the appropriate pad note briefly what the spec tells you about the project and how it is structured funded or timed or the product what it is and how it works or the quality criteria you should evaluate the product against or things you need to do that you learned from the spec Developing Exploratory Testing Skills Copyright Kaner 2006 97 An active reading example As you note what you have discovered make additional notes in a different pen color such as I
38. in interpreting this Clarification What does this mean Is it restated elsewhere in a clearer way Comparison How is this similar to that and Contrast How is this different from that Implications f X is true does that mean that Y must also be true Affective How does the author or you feel about that Relational How does this concept theme or idea relate to that one Problem solving How does this solve that problem or help you solve it Developing Exploratory Testing Skills Copyright Kaner 2006 6 More questions Relevance Why is this here What place does it have in the message or package of information the author is trying to convey If it is not obviously relevant is it a distractor Author s comprehension Does the author understand this Is the author writing in a way that suggests s he is inventing a concept without having researched it Author credibility What basis do you have for believing the author knows what s he is talking about Author perspective bias What point of view is the author writing from What benefit could the author gain from persuading you that X is true or desirable or false etc The Michigan Educational Assessment Association has some useful material at http www meap org html T Question ypes htm Developing Exploratory Testing Skills Copyright Kaner 2006 More questions Application How can you apply what the author is saying How does the
39. ing e SPECIFICATION BASED TESTING Domain testing Check every claim made in the e Scenario testing reference document such as a contract specification Risk based testing Regression testing e Stress testing Test to the extent that you have proved e State model based testing the claim true or User testing High volume automated testing false Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing Specification based testing Focus on variables such as inputs e DOMAIN TESTING outputs configuration or internal e g file Risk based testing handling variables Scenario testing For every variable or combination of variables consider the Stress testing space of possible values Simplify it by partitioning into e State model based testing subsets Pick a few representatives of each subset Regression testing User testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing e Specification based testing Domain testing e RISK BASED TESTING e Scenario testing e Regression testing e Stress testing e User testing State model based testing e High volume automated testing A program is a collection of opportunities for things to go wrong For each way that you can imagine the program failing desig
40. irements specification Using the Satisfice Heuristic Test Strategy Model to guide analysis THERE S PLENTY OF ADDITIONAL MATERIAL ON Using and developing models THE DISK in software testing Developing Exploratory Testing Skills Copyright Kaner 2006 The current crisis in software testing Developing Exploratory Testing Skills Copyright Kaner 2006 A crisis in software testing e NIST The Economic Impacts of Inadequate Infrastructure for Software Testing www nist gov director prog ofc report02 3 pdf Hmmm Software defects cost the economy a lot of money Let s blame that on bad testing gt To use these numbers for testing we ignore the other non testing factors e g organizational that lead to product release with known defects or non surprising defects suppose we could use that as a basis for declaring a testing crisis and selling lots of test consulting services That s not the crisis I m talking about today Developing Exploratory Testing Skills Copyright Kaner 2006 Here s the crisis on my radar We have gotten much better at testing documenting the testing reporting status of the testing of small programs But as the size of programs grows geometrically and the efficiency of testers grows maybe linearly gt we impact less of the program every year System level testing will become irrelevant because we will impact so little of the
41. iscussed When to apply it What software errors make the attack successful How to determine if the attack exposed a failure How to conduct the attack and An example of the attack We ll list How to Break Software s attacks here but recommend the book s full discussion Developing Exploratory Testing Skills Copyright Kaner 2006 Attacks to expose common coding errors User interface attacks Exploring the input domain e Attack Apply inputs that force all the error messages to occur e Attack 2 Apply inputs that force the software to establish default values e Attack 3 Explore allowable character sets and data types e Attack 4 Overflow input buffers e Attack 5 Find inputs that may interact and test combinations of their values e Attack 6 Repeat the same input or series of inputs numerous times From Whittaker How to Break Software Developing Exploratory Testing Skills Copyright Kaner 2006 Attacks to expose common coding errors User interface attacks Exploring outputs Attack 7 Force different outputs to be generated for each input e Attack 8 Force invalid outputs to be generated e Attack 9 Force properties of an output to change e Attack lO Force the screen to refresh From Whittaker How to Break Software Developing Exploratory Testing Skills Copyright Kaner 2006 Attacks to expose common coding errors Testing from the user interface Data and comput
42. ivations of the people involved he story is motivating A stakeholder with influence would push to fix a program that failed this test he story is credible It not only could happen in the real world stakeholders would believe that something like it probably will happen he story involves a complex use of the program or a complex environment or a complex set of data he test results are easy to evaluate This is valuable for all tests but is especially important for scenarios because they are complex Developing Exploratory Testing Skills Copyright Kaner 2006 166 Why use scenario tests Learn the product Connect testing to documented requirements Expose failures to deliver desired benefits Explore expert use of the program e Make a bug report more motivating Bring requirements related issues to the surface which might involve reopening old requirements discussions with new data or surfacing not yet identified requirements Developing Exploratory Testing Skills Copyright Kaner 2006 Scenarios Designing scenario tests is much like doing a requirements analysis but is not requirements analysis They rely on similar information but use it differently he requirements analyst tries to foster agreement about the system to be built The tester exploits disagreements to predict problems with the system he tester doesn t have to reach conclusions or make recommendations about how
43. lain your product common bugs in their design See listservs your niche or on your platform Websites etc and for discussions of how Exact comparisons with products some features are supposed you emulate by some to work Content reference materials e g an atlas to check your on line geography program Developing Exploratory Testing Skills Copyright Kaner 2006 Spec testing issues What is the specification What does the specification say Critiquing the specification what it says How it says what it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal issues Developing Exploratory Testing Skills Copyright Kaner 2006 What does the spec say Much of what is written about specification analysis has to do with the specification in the small interpreting the fine details in one or two pages of text e These are useful skills But specifications are often one or two thousand pages or more spread across multiple documents which incorporate several other documents by reference using undefined inconsistently defined or idiosynchratically defined vocabulary Specification readers often suffer severe information overload Active reading skills and strategies are essential for effective specification analysis Developing Exploratory Testing Skills Copyrig
44. ll in his courses on Requirements Based Testing His course has some great labs and he coaches well recommend it If you can t take his course you can find notes based on his work in Rodney Wilson s Software RX Secrets of Engineering Quality Software e An interesting workbook Cecile Spector Saying One Thing Meaning Another She discusses and provides examples and exercises with many additional ambiguities in common English than can cover here Developing Exploratory Testing Skills Copyright Kaner 2006 13 Common ambiguities in use of the language Undefined words e The user may authenticate incoming documents by processing their security attributes Incorrectly used words ypeface refers to a set of characters having the same design or to the design Font refers to a specific size and style of a typeface See google define typeface and define font A version of OpenOffice labeled a list of typefaces as fonts and a list of styles italics bold etc as typefaces How would you interpret help documentation that referred to typefaces Contradictorily defined words Use valid to mean sometimes a value considered valid by a user and other times a value that meets input criteria constraints in a program Vague words Etc will display a message process upgrade performance user friendly Commonly misunderstood words i e means id est that is and calls for a restatement or redefinition of a previous
45. maintenance Sales or marketing communication gt What are the corporate consequences if it IS inaccurate Regulatory compliance Developing Exploratory Testing Skills Copyright Kaner 2006 48 Why are you reviewing the spec or testing the product against the specification Context factors e To what extent is a test against the Spec necessary sufficient or useful gt To what extent can you change the product or process via spec review critique Will people invest in your developing an ability to understand the Spec Developing Exploratory Testing Skills Contract related risk management Regulatory related risk management Development group wants to use the spec as an internal authoritative standard Learn about the product Prevent problems before they are coded in Identify testing issues before you get code Help company assess product drift It s a source of information test tool to help you find potential bugs in product or spec Copyright Kaner 2006 40 Spec testing issues What is the specification What does the specification say Critiquing the specification what it says How it says what it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal issues Developing Exploratory Testing Skills Copyright Kaner 2006 Wh
46. n or supplement answers from reference materials Review written sources specifications and other authoritative documents culturally authoritative sources persuasive sources Try out potentially useful tools Developing Exploratory Testing Skills Copyright Kaner 2006 3 Examples of learning activities continued Create and apply models state usage data data flow other relationships dependencies among data task workflows user expectations physical systems or business systems being automated or simulated Hardware software platform Design and run experiments to establish lab procedures or polish lab techniques Research the compatibility space of the hardware software see e g Kaner Falk Nguyen s Testing Computer Software chapter on Printer Testing T eam research brainstorming or other group activities to combine and extend knowledge Paired testing mutual mentoring foster diversity in models and approaches Developing Exploratory Testing Skills Copyright Kaner 2006 Examples of design activities Design to create fashion execute or construct according to plan to conceive and plan out in the mind Websters e Map test ideas to FMEA or other lists of variables functions risks benefits tasks etc e Map test techniques to test ideas e Map tools to test techniques e Map staff skills to tools techniques develop training as necessary Develop supporting test data Develop
47. n tests to determine whether the program actually will fail in that way Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing e Specification based testing Tests are complex stories that capture Risk based testing how the program will e SCENARIO TESTING be used in real life situations Domain testing Regression testing These are e Stress testing combination tests whose combinations are credible e State model based testing reflections of real User testing High volume automated testing use Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing e Specification based testing Hepeat the same test after some change to Risk based testing the program Scenario testing You can use any test e REGRESSION TESTING 858regression test but if you do a lot of e Stress testing regression testing you Will or should learn to design cases State model based testing for efficient reuse Domain testing User testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing e Specification based testing Many detinitions of stress Domain testing pino When say stress testing mean tests intended to e Scenario testing overwhelm the product to Regression testing
48. nd review the statement s resulting meaning Slice amp dice Thinkertoys Make read a statement about the program Work through the statement one word at a time asking what each word means or implies These approaches can help you ferret out ambiguity in the product definition By seeing how different people can interpret a key statement in the spec you can imagine new tests to check which meaning is operative in the program Developing Exploratory Testing Skills Copyright Kaner 2006 Break statements into elements Quality is value to some person Quality Who is this person How are you the agent for this person How are you going to find out what this person wants How will you report results back to this person How will you take action if this person is mentally absent Person Developing Exploratory Testing Skills Copyright Kaner 2006 What it says about the product Correctness Does it accurately describe the program Controversy e Which parts are controversial Who are the stakeholders who disagree and why do they disagree Adequacy Does it provide enough information for programming documentation and testing Completeness Does it cover the feature set Design e Can you tell whether it specifies design errors e Is it understandable usable trainable consistent appropriate for the market Does it set up the program programmer for common errors es
49. ndout that provide here provides an outline for what should be a 3 4 day course It s a stunningly rich set of skills hope to get a chance to take a Bach Kelly course on ET Dynamics in the near future In this abbreviated form the lists are particularly useful for audit and mentoring purposes to help you spot gaps in your test activities or those of someone whose work you are evaluating Developing Exploratory Testing Skills Copyright Kaner 2006 Analyzing a sample requirements specification Developing Exploratory Testing Skills Copyright Kaner 2006 44 The specification The Disaster Missing Person Tracker Website Anonymized and slightly revised student project Developed in a requirements course by a team of grad students with significant development experience Developing Exploratory Testing Skills Copyright Kaner 2006 The opening exercise with this specification Please review the specification working in groups of 2 to 4 Please imagine that this is a genuine document that it has gone through its approval process and that you are now analyzing the document from the point of view of how you will test the product rather than how you want someone else to revise the specification As you sample the document please consider What tests clusters of tests should be run for a given requirement How much more or what instead is needed compared to the tests pro
50. new paradigms Our task is to update their foundations education to help them get past where we are stuck Developing Exploratory Testing Skills Copyright Kaner 2006 14 Basic Background Quality and errors Quality is value to some person Jerry Weinberg Under this view Quality is inherently subjective The value differs from person to person and therefore so does the quality Developing Exploratory Testing Skills Copyright Kaner 2006 15 Software error An attribute of a software product e that reduces its value to a favored stakeholder e or increases its value to a disfavored stakeholder without a sufficiently large countervailing benefit In other words A bug is something that bugs somebody James Bach Developing Exploratory Testing Skills Copyright Kaner 2006 Reject the Not My Job definition of testing Testing is not only about finding errors in code Testing is not only about doing tasks that some programmer can imagine for you or meeting the objectives that some programmer wishes on you unless that programmer is your primary staReholder e Anything that threatens value of a product to a stakeholder with influence threatens quality in a way important to the project You might be called on to investigate any of those possible threats including security performance usability suitability for intended purpose etc Tasks beyond your personal skill set can
51. nterpretation activities Examples of oracles Consistent within Product Behavior consistent with behavior of comparable functions or functional patterns within the product Consistent with Comparable Products Behavior consistent with behavior of similar functions 1n comparable products Consistent with a Model s Predictions Behavior consistent with expectations derived from a model Consistent with History Present behavior consistent with past behavior Consistent with our Image Behavior consistent with an image that the organization wants to project Consistent with Claims Behavior consistent with documentation or ads Consistent with Specifications or Regulations Behavior consistent with claims that must be met Consistent with User s Expectations Behavior consistent with what we think users want Consistent with Purpose Behavior consistent with apparent purpose Developing Exploratory Testing Skills Copyright Kaner 2006 42 A different structuring of key activities Jonathan Bach Mike Kelly and James Bach are working on a broad listing tutorial of ET activities which hope to see in book form See Bachs presentation on Exploratory Testing Dynamics at http www quardev com whitepapers html We reviewed preliminary drafts of Exploratory Testing Dynamics at the Exploratory Testing Research Summit spring 2006 and Consultants Camp 2006 August looking specifically at teaching issues The four page ha
52. ntext free questions to evaluate a plan Will this solve the whole problem Part of the problem What would you like the resolution to be Can you picture it How much of the unknown can you determine What reference data are you using if any What product output will you evaluate How will you do the evaluation Can you derive something useful from the information you have Have you used all the information Have you taken into account all essential notions in the problem Can you separate the steps in the problem solving process Can you determine the correctness of each step What creative thinking techniques can you use to generate ideas How many different techniques Can you see the result How many different kinds of results can you see How many different ways have you tried to solve the problem Based on The CIA s Phoenix Checklists Thinkertoys p 1 40 and Bach s Evaluation Strategies Rapid Testing Course notes Developing Exploratory Testing Skills Copyright Kaner 2006 67 Using context free questions to evaluate a plan What have others done Can you intuit the solution Can you check the results What should be done How should it be done Where should it be done When should it be done Who should do it What do you need to do at this time Who will be responsible for what Can you use this problem to solve some other problem What is the unique set of qualities that makes this problem what it
53. om There are no guarantees in this but you get better at it as you practice and as you build a broader inventory of techniques Developing Exploratory Testing Skills Copyright Kaner 2006 An overview of test techniques Developing Exploratory Testing Skills Copyright Kaner 2006 139 How do we use test techniques to create tests Analyze the situation Model the test space A test technique Is a recipe for performing these tasks in order to reveal something worth reporting select what to cover Determine test oracles Configure the test system Operate the test system Observe the test system Evaluate the test results Developing Exploratory Testing Skills Copyright Kaner 2006 140 Designing test cases Focus on procedure e A set of test inputs execution conditions and expected results developed for a particular objective such as to exercise a particular program path or to verify compliance with a specific requirement IEEE Focus on the test idea e A test idea is a brief statement of something that should be tested For example if you re testing a square root function one idea for a test would be test a number less than zero The idea is to check if the code handles an error case Marick Developing Exploratory Testing Skills Copyright Kaner 2006 lest cases In my view A test case is a Be D you ask the program The point of running the te
54. on previous stories CSI NEWSLETTER Be among the first to know Subscribe now for email updates now Developing Exploratory Testing Skills Copyright Kaner 2006 19 Defining Testing Empirical derived from experiment and observation rather than theory technical We use technical means including experimentation logic mathematics models tools testing support programs and tools measuring instruments event generators etc Investigation An organized and thorough search for information his is an active process of inquiry VVe ask hard questions aka run hard test cases and look carefully at the results Developing Exploratory Testing Skills Copyright Kaner 2006 Defining Testing Empirical technical investigation that we conduct to provide stakeholders e Someone who has a vested interest in the success of the product e Someone who has a vested interest in the success of the testing effort A stakeholder with influence is someone who has authority to influence the design or marketing of the product with information e The information of interest is often about the presence or absence of bugs Quality revealing information apart from specific bugs may be more vital to a particular stakeholder at a particular time Developing Exploratory Testing Skills Copyright Kaner 2006 2 Examples of Information Objectives Find important bugs to get them fixed Assess the
55. ood specification driven test e Same as what is a good test But tests come from specs Might be that a test that covers several spec items is preferred to a single item test Might be that tests that resolve or expose and show implications of specification ambiguities are particularly important Developing Exploratory Testing Skills Copyright Kaner 2006 86 Driving tests from the specification Coverage Key issue is coverage of the specification Cover items individual statements gt But how many tests per statement do you need gt Many groups require only one per spec assertion Cover specified relationships gt To test A amp amp B gt You probably want to test at leastA true and B true gt E uq uh usc gt Ede ee te ee Brian Marick s multi tool is useful for this Students at Florida Tech are now publishing a Release 2 0 of multi see www testingeducation org in December Developing Exploratory Testing Skills Copyright Kaner 2006 Driving tests from the spec Coverage Important to understand the level of generality called for when testing a spec item For example imagine a field X We could test a single use of X Or we could partition possible values of X and test boundary values e Or we could test X in various scenarios Which is the right one his partially depends on whether specification driven testing is your exclusive style of testing How do we track cover
56. opyright Kaner 2006 50 a a a Asking questions Here are some key contrasts Hypothetical what would happen if vs behavioral what have you done what has happened in the past in response to Factual factual answers can be proved true or false vs opinion what is the author s or your interpretation of these facts Historical what happened already vs predictive what the author or you expect to happen in the future under these conditions Open calls for an explanatory or descriptive answer doesn t reveal the answer in the question vs closed calls for a specific true answer often answerable yes or no Context dependent the question is based on the specific details of the current situation vs context free the question is usable in a wide range of situations it asks about the situation but was written independently of it Developing Exploratory Testing Skills Copyright Kaner 2006 EXPLORING REQUIREMENTS c Weinberg 1s a superb source for context free questions 60 More questions Causal Why did this happen Why is the author saying that Ask for evidence What proof is provided Why should you believe this Evidentiary sufficiency s this conclusion adequately justified by these data Trustworthiness of the data Were the data collection and analysis methods valid and reliable Critical awareness What are the author s assumptions What are your assumptions
57. other event An event reaches a process just as just before or just after a resource needed to accomplish servicing the event becomes available or unavailable Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Interference testing Generate interrupts from a device related to the task e g pull out a paper tray perhaps one that isn t in use while the printer is printing from a device unrelated to the task e g move the mouse and click while the printer is printing from a software event e g set another programs or this program s time reminder to go off during the task under test Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Interference Change something this task depends on swap out a floppy change the contents of a file that this program is reading change the printer that the program will print to without signaling a new driver change the video resolution Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Interference testing Cancel Cancel the task at different points during its completion Cancel some other task while this task is running a task that is in communication with this task the core task being studied a task that will eventually have to complete as a prerequisite to completion of this task a task that is totally unrelated to
58. product Developing Exploratory Testing Skills Copyright Kaner 2006 The traditional approach Define system level software testing as e functional e focusing on verification of the program s features e preterablp against an authoritative specification Developing Exploratory Testing Skills Copyright Kaner 2006 The traditional approach is Easy to understand Easy to translate into low skill work and routine automation NOT what you want as your career path AND e Maybe not so effective e Maybe outdated Developing Exploratory Testing Skills Copyright Kaner 2006 An underlying crisis Most of today s software testing techniques were developed in the 1970 s Back then long programs were 10 000 statements Code was often readable COBOL An enterprising tester could read the entire program identify all the variables and most of the relevant combinations Developing Exploratory Testing Skills Copyright Kaner 2006 An underlying crisis We have not experienced revolutions in testing practice VVe are not much more productive today than we were three decades ago Regression test automation offers small incremental improvements in productivity High volume test automation is still rarely done and is poorly understood by the general e g academic testing community est case documentation is as overblown as ever with a new generation of semi automated test c
59. rt way through for instance delete a file while 1t 1s being edited eject a disk pull net cables or power cords to an inappropriate state This 1s similar to 1nterruption except you are expecting the function to interrupt itself by detecting that it no longer can proceed safely Developing Exploratory Testing Skills Copyright Kaner 2006 129 Even More QuickTests from Bach s Rapid Testing Course Error Message Hangover Make error messages happen Test hard after they are dismissed Developers often handle errors poorly Bach once broke into a public kiosk by right clicking rapidly after an error occurred It turned out the security code left a 1 5 second window of opportunity for me to access a special menu and take over the system Click Frenzy Testing 1s more than banging on the keyboard but that phrase wasn t coined for nothing Try banging on the keyboard Try clicking everywhere Bach broke into a touchscreen system once by poking every square centimeter of every screen until he found a secret button Multiple Instances Run a lot of instances of the application at the same time Open the same files Feature Interactions Discover where individual functions interact or share data Look for any interdependencies Tour them Stress them Bach once crashed an app by loading up all the fields 1n a form to their maximums and then traversing to the report generator Developing Exploratory Testing Skills Copyright Kaner 200
60. s Interference testing Compete Examples Compete for a device such as a printer put device in use then try to use it from software under test e start using device then use it from other software e f there is a priority system for device access use software that has higher same and lower priority access to the device before and during attempted use by software under test Compete for processor attention some other process generates an interrupt e g ring into the modem or a time alarm in your contact manager e try to do something during heavy disk access by another process Send this process another job while one is underway Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Follow up recent changes Code changes cause side effects Test the modified feature change itself est features that interact with this one Test data that are related to this feature or data set est scenarios that use this feature in complex ways Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Explore data relationships Pick a data item race its flow through the system What other data items does it interact with e What functions use it e Look for inconvenient values for other data items or for the functions look for ways to interfere with the function using this data item Developing Exploratory Testing Skills Copyright Kaner 200
61. sed on claims to the public Article Liability for defective documentation http www kaner com pdfs liability sigdoc pdf Warranties based on claims to custom product customer Claims of compatibility with other products Article Liability for product incompatibility http www kaner com pdfs liability sigdoc pdf Errors in your product documents that are not about your products Article Liability for defective content http www kaner com pdfs sigdocContent pdf Developing Exploratory Testing Skills Copyright Kaner 2006 Testing claims against the product Uniform Commercial Code Article 2 2003 revision SECTION 2 313A 2 If a seller in a record packaged with or accompanying the goods makes an affirmation of fact or promise that relates to the goods provides a description that relates to the goods or makes a remedial promise and the seller reasonably expects the record to be and the record is furnished to the remote purchaser the seller has an obligation to the remote purchaser that a the goods will conform to the affirmation of fact promise or description unless a reasonable person in the position of the remote purchaser would not believe that the affirmation of fact promise or description created an obligation and b the seller will perform the remedial promise 3 It is not necessary to the creation of an obligation under this section that the seller use formal words such as warrant or guarantee or
62. sic Example of a QuickTest Traditional boundary testing All you need is the variable and its possible values You need very little information about the meaning of the variable why people assign values to it what it interacts with You test at boundaries because miscoding of boundaries is a common error Note the foundation of this test There is a programming error so common that it s worth building a test technique optimized to find errors of that type Developing Exploratory Testing Skills Copyright Kaner 2006 108 Mtacks to expose common coding errors ann ee eaa a aan Jorgensen amp Whittaker pulled is book to you noe od pisi inte eese together a collection of common coding errors many of them involving How to insufficiently or incorrectly Br eak Software constrained variables oes They created or identified common 0010101011011 101 attacks to test for these An attack is a stereotyped class of tests optimized around a specific type of error James A Whittaker Think back to boundary testing Boundary testing for numeric input fields is an example of an attack The error is mis specification or mis typing of the upper or lower bound of the numeric input field Developing Exploratory Testing Skills Copyright Kaner 2006 109 Attacks to expose common coding errors In his book How to Break Software Professor Whittaker expanded the list and for each attack d
63. st is to gain information for example whether the program will pass or fail the test he test must be CAPABLE of revealing valuable information e The SCOPE of a test changes over time because the information value of tests changes as the program matures e The METRICS that count test cases are essentially meaningless because test cases merge or are abandoned as their information value diminishes Developing Exploratory Testing Skills Copyright Kaner 2006 142 Ten dominating techniques Function testing e Specification based testing Domain testing Risk based testing These are 10 common Examples e Scenario testing Regression testing e Stress testing There are many Others User testing State model based testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 143 Ten dominating techniques e FUNCTION TESTING EL Test each feature or e Specification based testing function on its own e D in testi omain testing Scan through the Risk based testing product covering Scenario testing every feature or function with at least enough testing to Stress testing determine what it User testing does and whether it is working Regression testing State model based testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function test
64. subject it to so much input so little memory e STRESS TESTING such odd combinations that expect it to fail and l am exploring its behavior e State model based testing as and after it fails Risk based testing User testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Function testing Specification based testing Give the program to a user see what he does with it and how Risk based testing it responds Scenario testing User tests can be Regression testing tightly structured or very loosely defined The essence is room e USER TESTING for action and response by users Domain testing e Stress testing State model based testing High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Model the program as a state machine e Specification based testing that runs from state Domain testing to state in response to events such as new inputs Function testing Risk based testing e Scenario testing In each state does it respond correctly to e Stress testing each event Regression testing User testing e STA TE MODEL BASED TESTING e High volume automated testing Developing Exploratory Testing Skills Copyright Kaner 2006 Ten dominating techniques Program the computer to design e Specification based t
65. t it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal 1ssues Developing Exploratory Testing Skills Copyright Kaner 2006 ritiquing specs Process notes Review meetings Test groups often train to facilitate technical reviews Detailed comments on the specification e Same guidelines as for critiquing other tech pubs See Testing Computer Software HANDBOOK OF Walkthroughs Inspections and Technical Reviews THIRD EDITION DH Developing Exploratory Testing Skills Copyright Kaner 2006 84 Spec testing issues What 1s the specification What does the specification say Critiquing the specification what it says How it says what it says What it says about the product What it says about the testing of the product Critiquing the specification doing the critique Driving tests from the specification Legal 1ssues Developing Exploratory Testing Skills Copyright Kaner 2006 Driving tests from the specification Who are the stakeholders here are stakeholders for all services Who are yours Regulators Marketing End customer Journalists Attorney Court Expert witness Client company you re the outsource test lab These stakeholders would have different test result test documentation expectations from the typical project team What is a g
66. tems that haven t yet been specified that you think are relevant References to later parts of the specification or to other documents that you ll need to understand the spec Questions that come to mind about how the product works how the project will be run or what quality criteria are in play Your disagreements or concerns with the product project as specified Beware of getting too detailed in this If the spec provides a piece of information you don t need to rewrite it Just write down a pointer and a spec page number Your list is a quick summary that you build as you read to help you read not a rewriting of the document As you read further some of your earlier questions will be answered Others won t Ask the programmers or spec writers about them Developing Exploratory Testing Skills Copyright Kaner 2006 90 Heuristic test strategy model The HSTM is another example of a tool that is especially useful for auditing mentoring purposes It provides you a support structure for discovering what is missing or buried in someone else s work We have seen this already in the EI Dynamics handout My bug appendix in Testing Computer Software was widely used for that and HSTM has been the root of comparable but more recent documents e g Vijayaraghavan s thesis The Phoenix questions in the previous section provide another strong example of a question set that is at least as useful for post creation r
67. this task Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Interference testing Pause Find some way to create a temporary interruption in the task Pause the task for a short time for a long time long enough for a timeout if one will arise For example Put the printer on local Put a database under use by a competing program lock a record so that it can t be accessed yet Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTests Interference testing Swap out of memory Swap the process out of memory while it s running e g change focus to another application keep loading or adding applications until the application under test is paged to disk Leave it swapped out for 10 minutes or whatever the timeout period is Does it come back What is its state What is the state of processes that are supposed to interact with it Leave it swapped out much longer than the timeout period Can you get it to the point where it is supposed to time out then send a message that is supposed to be received by the swapped out process then time out on the time allocated for the message What are the resulting state of this process and the one s that tried to communicate with it Swap a related process out of memory while the process under test is running Developing Exploratory Testing Skills Copyright Kaner 2006 Additional QuickTest
68. tin Developing Exploratory g Skills Copyright Kaner 2006 What it says about testing Early in the project you can review the spec s implications for testing and change them or prepare for them Implications for test design What test techniques will be most appropriate for this project Will you need additional training or tools for them Are there ways to simplify or otherwise change to product in ways that would call for simpler or cheaper or more easily structured techniques How much exploring will this project require gt Does your staff have the knowledge skills and connections Test schedule and resource commitments implications When will you receive deliverables from others When are you to deliver your work What do you need to get this done Are any of your commitments unreasonable Testability support Developing Exploratory Testing Skills Copyright Kaner 2006 Design reviews Testability Controllability Observability Availability Scriptable Interface Simplicity Stability Information Separation of functional components Availability of oracles Testing is far more rapit when the product is far more testable Developing Exploratory Testing Skills Copyright Kaner 2006 02 lesting the program against the spec What 1s the specification What does the specification say Critiquing the specification what it says How it says wha
69. vided If you had the code in front of you would tests of the code NOW help clarify the specification What key information is missing and how would you get it Developing Exploratory Testing Skills Copyright Kaner 2006 46 Notes on spec based testing from Kaner amp Bach s BBST course We ve seen at least three different meanings of specification based testing A style of testing collection of test related activities and techniques focused on discovering what claims are being made in the specifications and on testing them against the product This is what we mean by spec based testing A style of testing focused on proving that the statements in a specification and the code that matches the statements are logically correct A set of test techniques focused on logical relationships among variables that are often detailed in specifications Developing Exploratory Testing Skills Copyright Kaner 2006 Context factors Is this intended Why did they write the specification as an authoritative e Enforceable contract for custom software document Who is its champion Facilitate and record agreement among stakeholders About specific issues or Who cares if it s about the whole thing kept up to date u and correct e Vision document Who doesn t e Support material for testing tech support e Who is technical writers accountable for l its accuracy and Marketing support
70. vious versions manual e Product literature e Published style guide and UI standards e Published standards such as C language e 3rd party product compatibility test suites e Published regulations e Internal memos e g project mgr to engineers describing the feature definitions Developing Exploratory Testing Skills Marketing presentations selling the concept of the product to management Bug reports responses to them Reverse engineer the program e Interview people such as e development lead etech writer e customer service e subject matter experts project manager Look at header files source code database table definitions e Specs and bug lists for all 3rd party tools that you use Prototypes and lab notes on the prototypes Copyright Kaner 2006 53 Implicit specifications e Get lists of compatible equipment Interview development staff and environments from Marketing from the last version in theory at least Look at customer call records oca ization guide probably from the previous version What published for localizing products on bugs were found in the field your platform Usability test results Look at compatible products to Beta test results find their failures then look for 3 party tech support these in your product how they databases magazines and web designed features that you don t sites with reports of bugs in understand and how they exp
71. would be Developing Exploratory Testing Skills Copyright Kaner 2006 What we need for design Is a constantly evolving set of tests hat exercise the software in new ways new combinations of features and data e So that we get our choice of broader coverage of the infinite space of possibilities gt adapting as we recognize new classes of possibilities and sharper focus gt on risks or issues that we decide are of critical interest today For that we do exploratory testing Developing Exploratory Testing Skills Copyright Kaner 2006 Exploratory testing Developing Exploratory Testing Skills Copyright Kaner 2006 33 Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test related learning test design test execution and test result interpretation as mutually supportive activities e that run in parallel throughout the project Developing Exploratory Testing Skills Copyright Kaner 2006 Exploratory testing Learning Anything that can guide us in what to test how to test or how to recognize a problem e Design to create fashion execute or construct according to plan to conceive and plan out in the mind Websters Designing is not scripting The representation of a plan is not the plan
72. xperience An individual stakeholder s preference Who are the stakeholders How does it relate to which stakeholders What isn t the problem Is the information sufficient Or is it insufficient Or redundant Or contradictory Should you draw a diagram of the problem A figure Based on The CIA s Phoenix Checklists Thinkertoys p 140 and Bach s Evaluation Strategies Rapid Testing Course notes Copyright Kaner 2006 65 Using context free questions to define a problem Where are the boundaries of the problem What product elements does it apply to How does this problem relate to the quality criteria Can you separate the various parts of the problem Can you write them down What are the relationships of the parts of the problem What are the constants things that can t be changed of the problem What are your critical assumptions about this problem Have you seen this problem before Have you seen this problem in a slightly different form Do you know a related problem Think of a familiar problem having the same or a similar unknown Suppose you find a problem related to yours that has already been solved Can you use it Can you use its method Can you restate your problem How many different ways can you restate it More general More specific Can the rules be changed What are the best worst and most probable cases you can imagine Developing Exploratory Testing Skills Copyright Kaner 2006 66 Using co
73. y a failure so as to better e motivate a stakeholder who Potential consequences of might advocate for a fix potential failures enable a fixer to identify and Who might care about which stomp the bug more quickly consequence of what failure How to expose and who to expose to undelivered benefits unsatisfied implications traps and missed opportunities How to trigger a fault that generates the failure we re seeking How to recognize failure Developing Exploratory Testing Skills Copyright Kaner 2006 The Analogy to Manufacturing QC Fixed design e Well understood risks he same set of errors appear on a statistically understood basis est for the same things on each instance of the product e Scripting makes a lot of sense Developing Exploratory Testing Skills Copyright Kaner 2006 The Analogy to Design QC he design is rich and not yet trusted A fault affects every copy of the product he challenge is to find new design errors not to look over and over and over again for the same design error e Scripting is probably an industry worst practice for design QC Software testing is assessment of a design not of the quality of manufacture of the copy Developing Exploratory Testing Skills Copyright Kaner 2006 30 Imagine Imagine crime scene investigators real investigators of real crime scenes following a script How effective do you think they

Download Pdf Manuals

Related Search

Related Contents

Le supermarché de l`accessoire ..  Casio 2532 MO0404-EC User's Manual  be.ez LA robe MacBook Pro 17 B&W  G3 Ferrari Girandolo 40  international standard norme internationale  Vital Force MKIII user guide.  Samsung Monitor LED Curvo 29" S29E790C Manual de Usuario  演算式接地抵抗計 - 双興電機製作所  こちら  TENDER NO  

Copyright © All rights reserved. Failed to retrieve file