Home

Developing the Right Test Documentation

Contents

1. 70 ed d James Bach All rights reserv Copyright 2001 Cem Kaner an Test Documentation Examples of integer input tests OOOO e Nothing e Valid value e At LB of value e At UB of value e AtLB of value 1 e At UB of value 1 e Outside of LB of value e Outside of UB of value e 0 e Negative e At LB number of digits or chars e At UB number of digits or chars e Empty field clear the default value Test Documentation Outside of UB number of digits or chars Non digits Wrong data type e g decimal into integer Expressions Space Non printing char e g Ctrl char DOS filename reserved chars EO Var Upper ASCII 128 254 Upper case chars Lower case chars Modifiers e g Ctrl Alt Shift Ctrl etc Function key F2 F3 F4 etc Copyright 2001 Cem Kaner and James Bach All rights reserved 7 1 Error Handling when Writing a File COo e full local disk e almost full local disk e write protected local disk e damaged I O error local disk e unformatted local disk e remove local disk from drive after opening file e timeout waiting for local disk to come back online e keyboard and mouse 1 O during save to local disk e other interrupt during save to local drive e power out during save to local drive Test Documentation full network disk almost full network disk write protected network disk damaged I O error network disk remove n
2. The notes also reflect our view that testing is an exercise in critical thinking and careful questioning A test case is a question that you ask of the program Are you broken in this way The point of a test case is to reduce uncertainty associated with the product A test is good if it will reduce uncertainty whether it finds a bug or not A test plan is a structure for asking questions of the project and the product These notes suggest strategies for asking better questions and they provide useful clusters of questions The notes also provide samples of some common test planning documents such as tables and matrices These will probably be among the building blocks of any testing program that you set up Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 3 Overview Eo e Problems with the allegedly standard approach e Defining your documentation requirements e A model for testing and test documentation e Test documentation elements Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach Loo O IEEE Standard 829 for Software Test Documentation Test plan Test design specification Test case specification e Test case specification identifier We ofi ten see e Test items one or more e Input specifications pages per e Output specifications Environmental needs e Special procedu
3. Anything that drives or constrains design e Stakeholders Favored disfavored and neutral stakeholders e Stakeholders interests Favored disfavored and neutral interests e Actions Actions support or interfere with interests e Objects Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 1 Exercise E i sVX H 1 List the Stakeholders Favored Disfavored Neutral stakeholders 2 For each Stakeholder list her Interests Favored Disfavored Neutral interests 3 For each Interest list Actions Actions support an interest Actions interfere with an interest Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 12 Exercise E Objects The Stuff You Create Such as features data of the product For each object what is its relationship to a stakeholder a stakeholder s interest or in the actions the stakeholder wants to take or will have taken on her Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 13 Testers Questions Does Your Car Work OO HOW CAN YOU TELL THAT SOMETHING WORKS How do you know your car works Are there situations in which your car would stop working Who else uses your car Do they use it differently than you so that it might work for you but fail for them What facts would cause you to believe that your car
4. Is testing approach oriented toward proving conformance to specs or nonconformance with customer expectations e Does your testing style rely more on already defined tests or on exploration e Should test docs focus on what to test objectives or on how to test for it procedures e Should the docs ever control the testing project Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 21 Test Docs Requirements Questions CO If the docs control parts of the testing project should that control come early or late in the project Who are the primary readers of these test documents and how important are they How much traceability do you need What docs are you tracing back to and who controls them To what extent should test docs support tracking and reporting of project status and testing progress How well should docs support delegation of work to new testers What are your assumptions about the skills and knowledge of new testers Is test doc set a process model a product model or a defect finder Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 22 Test Docs Requirements Questions O A test suite should provide prevention detection and prediction Which is the most important for this project e How maintainable are the test docs and their test cases And how well do they ensure that test changes will follow code changes e Will t
5. Look at Marick Craft of Software Testing for discussion of catalogs of tests for data relationships Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved SO Data Relationship Table a e Looking at the Word options you see the real value of the data relationships table Many of these options have a lot of repercussions e You might analyze all of the details of all of the relationships later but for now it is challenging just to find out what all the relationships ARE e The table guides exploration and will surface a lot of bugs e PROBLEM e Works great for this release Next release what is your support for more exploration Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 8 1 Configuration Planning Table OO EE Wan Yarz vas Var Var Config 5 Config 6 This table defines 6 standard configurations for testing In later tests the lab will set up a Config 1 system a Config 2 system etc and will do its compatibility testing on these systems The variables might be software or hardware choices For example Var 1 could be the operating system V1 1 is Win 2000 V1 2 is Win ME etc Var 2 could be how much RAM on the computer under test V2 1 is 128 meg V2 2 is 32 meg etc Var 3 could be the type of printer Var 4 the machine s language setting French German etc Config planning tables are often filled in using the All Pairs algorithm
6. Test Strategy OO u u u Makes use of test techniques May be expressed by test procedures and cases Not to be confused with test ogistics which involve the details of bringing resources to bear on the test strategy at the right time and place You don t have to know the entire strategy in advance The strategy can change as you learn more about the product and its problems Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 5 7 Test Cases Procedures Loo O Test cases and procedures should manifest the test strategy If your strategy is to execute the test suite got from Joe Third Rrty how does that answer the prime strategic questions How will you cover the product and assess quality How is that practical and justified with respect to the specifics of this project and product If you don t know then your real strategy is that you re trusting things to work out Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 5 8 Diverse Half Measures OOO e There is no single technique that finds all bugs e We cant do any technique perfectly e We cant do all conceivable Use diverse half measures lots of different points of view approaches techniques even if no one strategy is performed completely vopyright 2001 Cem Kaner and James Bach All rights reserved 59 Test Plan Components OOO e The foll
7. Installability Supportability and i uninstallability a Localizability Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 36 Risk Doo Hazard A dangerous condition something that could trigger an accident Risk Possibility of suffering loss or harm Accident A hazard is encountered resulting in loss or harm e Useful material available free at http seir sei cmu edu http www coyotevalley com Brian Lawrence e Good paper by Stale Amland Risk Based Testing and Metrics 16th International Conference on Testing Computer Software Test peste rettion Copyright 2001 Cem Kaner and James Bach All rights reserved 37 Risk Doo e Project risk management involves Identification of the different risks to the project issues that might cause the project to fail or to fall behind schedule or to cost too much or to dissatisfy customers or other stakeholders Analysis of the potential costs associated with each risk Development of plans and actions to reduce the likelihood of the risk or the magnitude of the harm Continuous assessment or monitoring of the risks or the actions taken to manage them Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 38 Risk Based Testing O u u e Two key dimensions Find errors risk based approach to technical tasks of testing Manage the process of finding errors risk b
8. Myrvold Hung Quoc Nguyen Noel Nyman Neal Reizer Amit Singh and Melora Svoboda Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 2 Abstract Do This workshop has grown out of our dissatisfaction with paper intensive approaches that attempt to provide a seemingly reproducible somewhat mechanical process for planning and managing testing and test documentation Over the past 17 years we have criticized IEEE standard 829 on software test documentation and related approaches as being often inappropriate Colleagues have asked what we would put in IEEE 829 s place To date our responses have been piecemeal This seminar s notes are a draft of our attempt to write a more comprehensive response They start from the premise that the best approach to test documentation depends on the project context For example creating detailed test documentation can be useful for some projects but can get in the way of the development of a high volume automated testing strategy What are the relevant differences between these projects Before adopting an implementation guideline like IEEE 829 we should analyze our requirements There is no point spending a fortune on creating a deliverable here the test documentation set that will not be used or that will interfere with the efficient running of the project Instead we should build a documentation set that will actually satisfy the real needs of the project
9. e n a random sequence of standalone tests we might want to qualify each test T1 T2 etc as able to run on its own Then when we test a sequence of these tests we know that errors are due to interactions among them rather than merely to cumulative effects of repetition of a single test e Therefore for each Ti we run the test on its own many times in one long series randomly switching as many other environmental or systematic variables during this random sequence as our tools allow e We call this the sandbox series Ti is forced to play in its own sandbox until it proves that it can behave properly on its own This is an 80 20 rule operation We do want to avoid creating a big random test series that crashes only because one test doesn t like being run or that fails after a few runs under low memory We want to weed out these simple causes of failure But we don t want to spend a fortune trying to control this risk Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 15 1 Random Testing Sandboxing the Regression Tests rr Suppose that you create a random sequence of standalone tests that were not sandbox tested and these tests generate a hard to reproduce failure You can run a sandbox on each of the tests in the series to determine whether the failure is merely due to repeated use of one of them Test Documentation Copyright 2001 Cem Kaner and James Bach All rights re
10. g2 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Configuration Test Matrix Oe __ Config 1_ Config 2 Config Config 4 Config 5 Config 6_ Pass Pass Pass Pass_ Pass Test2 Fail __ Pass Pass Pass Test3 Pass Pass _ Pass Pass _ Pass Test4 Pass Fail Fall Pas Test5 Fail __ Pass_ Fail_ Pass Pass This matrix records the results of a series of tests against the 6 standard configurations that we defined in the Configuration Planning Table In this table Config 1 has passed 3 tests failed 1 and hasn t yet been tested with Test 2 Config 6 is still untested Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 83 Testing Variables in Combination Interesting papers Cohen Dalal Parelius Patton The Combinatorial Design Approach to Automatic Test Generation IEEE Software Sept 96 http computer org 80 software so1996 sStoc htm Cohen Dalal Fredman Patton The AETG System An Approach to Testing Based on Combinatorial Design EEE Trans on SW Eng Vol 23 7 July 97 http computer org 80 tse ts 1997 e7toc htm OnLine requires IEEE membership for free access See http www computer org epub Several other papers on AETG are available at https aetgweb tipandring com A boutAETGweb html Also interesting http www stsc hill af mil CrossTalk 1997 oct planning html Jor
11. AKA partitioning equivalence analysis boundary analysis e Fundamental question or goal This confronts the problem that there are too many test cases for anyone to run This is a stratified sampling strategy that provides a rationale for selecting a few test cases from a huge population e General approach Divide the set of possible values of a field into subsets pick values to represent each subset Typical values will be at boundaries More generally the goal is to find a best representative for each subset and to run tests with these representatives Advanced approach combine tests of several best representatives Several approaches to choosing optimal small set of combinations e Paradigmatic case s Equivalence analysis of a simple numeric field Printer compatibility testing multidimensional variable doesn t map to a simple numeric field but stratified sampling is essential 11 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 7 Domain Testing Loo o e In classical domain testing Two values single points or n tuples are equivalent if the program would take the same path in response to each e The classical domain strategies all assume that the predicate interpretations are simple linear inequalities the input space is continuous and coincidental correctness 1s disallowed e It is possible to move away from these assumpti
12. Exploring Context free product questions Requirements What problems could this product create p 59 64 What kind of precision is required desired for this product Metaquestions when interviewing someone for info Am I asking too many questions Do my questions seem relevant Are you the right person to answer these questions Is there anyone else who can provide additional information Is there anything else I should be asking Is there anything you want to ask me May I return to you with more questions later Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 19 What is your group s mission a e Find important problems e Advise about QA e Assess quality e Advise about testing e Certify to standard e Advise about quality e Fulfill process mandates e Maximize efficiency e Satisfy stakeholders e Minimize time e Assure accountability e Minimize cost The quality of testing depends on which of these possible missions matter and how they relate Many debates about the goodness of testing are really debates over missions and givens Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 20 Test Docs Requirements Questions O e Is test documentation a product or tool Is software quality driven by legal issues or by market forces How quickly is the design changing How quickly does the specification change to reflect design change
13. Techniques O u u e Function e Regression e Domain driven e Stress driven e Specification driven e Risk driven e Scenario use case transaction flow e User testing e Exploratory e Random statistical All of these have been used as the dominant technique in some companies How can approaches so different yield good overall results We think that the answer is that each of these fixes only one of the dimensions for testing techniques For example function testing speaks to coverage but not to testers risks activities or evaluation You can vary all four of these and still be doing function testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 54 General Test Techniques Loo e We provide an appendix that describes the 10 general test techniques that we listed on the previous slide e We arent going to work through that appendix or not in much detail in this workshop but these notes may be helpful for self dudy to fill in some of the details that we re skipping here Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 55 Test Strategy OO u u u e How we plan to cover the product so as to develop an adequate assessment of quality e A good test strategy is Diversified Specific Practical Defensible Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 56
14. columns involve different test items A test item might be a function a variable an assertion in a specification or requirements document a device that must be tested any item that must be shown to have been tested e The rows are test cases e The cells show which test case tests which items e Ifa feature changes you can quickly see which tests must be reanalyzed probably rewritten e In general you can trace back from a given item of interest to the tests that cover it e This doesnt specify the tests it merely maps their coverage Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 63 Myers Boundary Table Variable Valid Case Invalid Case Boundaries Equivalence Equivalence and Special Classes Classes 99 to 99 gt 99 lt 99 99 non number expressions Second same as first same as first same 198 to 198 Are there other sources of data for this variable Ways to feed it bad data Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 64 Revised Boundary Analysis Table Equivalence Alternate Boundaries Class Equivalence and Special Class 99 to 99 gt 99 lt 99 99 100 digits non digits 0 9 leading spaces or 0s expressions Null entry Second same as first Same as first same number 198 to 198 Are there other 127 to 127 198 to 128 127 128 127 epee o 128 to 198 128 IS varia e a
15. defined requirements they say are generally incomplete and natural language is inherently ambiguous Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 100 Heuristics for Test Plan Evaluation COO 8 The test project should promote Other teams and stakeholders often collaboration with all other functions have information about product of the project especially problems or potential problems that developers technical support and can be of use to the test team Their technical writing Whenever perspective may help the testers possible testers should also make a better analysis of risk collaborate with actual customers Testers may also have information and users in order to better that is of use to them understand their requirements 9 The test project should consult The likelihood that a test strategy will with development to help them serve its purpose is profoundly build a more testable product affected by the testability of the product Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 101 Heuristics for Test Plan Evaluation COO 10 A test plan should highlight the Virtually every software project worth non routine project specific doing involves special technical aspects of the test strategy and test challenges that a good test effort project must take into account A completely generic test plan usually indicates a weak test plannin
16. related variable Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 78 Tabular Format for Data Relationships O Many relationships among data Independence e Varying one has no effect on the value or permissible values of the other Causal determination e By changing the value of one we determine the value of the other For example in MS Word the extent of shading of an area depends on the object selected The shading differs depending on Table vs Paragraph Constrained to a range e For example the width of a line has to be less than the width of the page e In a date field the permissible dates are determined by the month and the year if February Selection of rules e Example hyphenation rules depend on the language you choose Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 79 Tabular Format for Data Relationships O Many relationships among data Logical selection from a list e processes the value you entered and then figures out what value to use for the next variable Example timeouts in phone dialing 0 on complete call 555 1212 but 95551212 10 on ambiguous completion 955 5121 30 seconds incomplete 555 121 Logical selection of a list e For example in printer setup choose OfficeJet get Graphics Quality Paper Type and Color Options LaserJet 4 get Economode Resolution and Half toning
17. s Beta testing In house experiments using a stratified sample of target market Usability testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 35 User Testing Loo oO Strengths Design issues are more credibly exposed Can demonstrate that some aspects of product are incomprehensible or lead to high error rates in use In house tests can be monitored with flight recorders capture replay video debuggers other tools In house tests can focus on areas tasks that you think are or should be controversial Blind spots Coverage is not assured serious misses from beta test other user tests Test cases can be poorly designed trivial unlikely to detect subtle errors Beta testing is not free beta testers are not skilled the technical results are mixed Distinguish marketing betas from technical betas Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 36 Exploratory Testing Loo o Simultaneously Learn about the product Learn about the market Learn about the ways the product could fail Learn about the weaknesses of the product Learn about how to test the product Test the product Report the problems Advocate for repairs Develop new tests based on what you have learned so far Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 137 Exploratory Testin
18. test Use each function in a mainstream way positive testing Push it in as many ways as possible as hard as possible Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 1 1 Regression lesting _ OO e Tag line Repeat testing after changes e Fundamental question or goal Manage the risks that a a bug fix didn t fix the bug or b the fix or other change had a side effect e Paradigmatic case s Bug regression Show that a bug was not fixed Old fix regression Show that an old bug fix was broken General functional regression Show that a change caused a working area to break Automated GUI regression suites Strengths Reassuring confidence building regulator friendly Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 12 Regression Testing Loo o e Blind spots weaknesses Anything not covered in the regression series Repeating the same tests means not looking for the bugs that can be found by other tests Pesticide paradox Low yield from automated regression tests Maintenance of this standard list can be costly and distracting from the search for defects Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 13 Automating Regression Testing Loo e This is the most commonly discussed automation approach create a test case run it and inspect t
19. the Problem OOOO Where are the boundaries of the problem What product elements does it apply to How does this problem relate to the quality criteria Can you separate the various parts of the problem Can you write them down What are the relationships of the parts of the problem What are the constants things that can t be changed of the problem What are your critical assumptions about this problem Have you seen this problem before Have you seen this problem in a slightly different form Do you know a related problem Try to think of a familiar problem having the same or a similar unknown Suppose you find a problem related to yours that has already been solved Can you use it Can you use its method Can you restate your problem How many different ways can you restate it More general More specific Can the rules be changed What are the best worst and most probable cases you can imagine Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 18 Context Free Questions Loo UU Context free process questions Who is the client A sample of i i ale itional What is a successful solution worth to this client additiona What is the real underlying reason for wanting to solve this resi ple problem Gause amp Who can help solve the problem Weinberg s How much time is available to solve the problem
20. understand them Probes or diagnostics to help observe the product under test Matrices checklists other testing documentation Information As needed for testing about the project or product Specifications requirements documents other reference materials to help us determine pass fail or to credibly challenge odd behaviour e What is the availability of these documents e What is the volatility of these documents Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 28 Project Factors OT e items Under Test Anything that will be tested For each product element e Is it available or when will it be e Is it volatile and what is the change process e Is it testable e Logistics Facilities and support needed for organizing and conducting the testing Do we have the supplies physical space power light security systems if needed procedures for getting more e Budget Money and other resources for testing Can we afford the staff space training tools supplies etc e Deliverables The observable products of the test project Such as bug reports summary reports test documentation master disk e What are you supposed to create and can you do it Will we archive the items under test and other products of testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 29 Product Elements A product is Sn An expe
21. 001 Cem Kaner and James Bach All rights reserved 43 Risks Where to look for errors Loo o e Weak testing tools if tools dont exist to help identify isolate a class of error e g wild pointers the error is more likely to survive to testing and beyond e Unfixability risk of not being able to fix a bug e Language typical errors such as wild pointers in C See Bruce Webster Pitfalls of Object Oriented Development Michael Daconta et al Java Pitfalls Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 44 Risks Where to look for errors o ee Criticality severity of failure of very important features Popularity likelihood or consequence if much used features fail Market severity of failure of key differentiating features Bad publicity a bug may appear in PC Week Liability being sued Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 45 Bug Patterns as a Source of Risk ee e Testing Computer Software lays out a set of 480 common defects You can use these or develop your own list Find a defect in the list Ask whether the software under test could have this defect fit is theoretically possible that the program could have the defect ask how you could find the bug if it was there Ask how plausible it is that this bug could be in the program and how serious the failure would be if it was there If appro
22. 74 Complex Data Relationships View General Edit Prink Save Spelling amp Grammar Track Changes User Information _ompatibility File Locations Compatibility options For Documenti Font Substitution Recommended options Far Microsoft Word a7 Miptions Wj combine table borders like word 5S For the Macintosh O Do Full justification like WordPerfect 6 x For Windows Don t add automatic tab stop For hanging indent Don t add extra space For raisedslowered characters Don t add leading extra space between rows of kext Don t add space For underlines Don t balance columns For Continuous section starts Don t balance SECS characters and DBCS characters Don t blank the area behind metafile pictures O Don t center exact line height lines _ Don t convert backslash characters into yen signs Default AOU Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Tabular Format for Data Relationships Field Entry Display Print Related Relationship source variable Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 76 Tabular Format for Data Relationships Loo oO Once you identify two variables that are related test them together using boundary values of each or pairs of values that will trigger some other boundary This is not the most powerful process for looking at relationships An approach like Cause Eff
23. Developing the Right Test Documentation Loo o Cem Kaner J D Ph D Department of Computer Sciences Florida Institute of Technology James Bach Satisfice Inc October 2001 Pacific Northwest Software Quality Conference Acknowledgments OT e These notes outline the test planning chapters in prep for Testing Computer Software 3rd Ed by Cem Kaner James Bach Hung Quoc Nguyen Jack Falk Brian Lawrence amp Bob Johnson They incorporate and adapt materials by these authors The notes are also based on materials developed for Lessons Learned in Software Testing a book just completed by Cem Kaner James Bach and Bret Pettichord e Many of the ideas in these notes were reviewed and refined at the Third Los Altos Workshop on Software Testing LAWST February 7 8 1998 and at the Eleventh LAWST October 28 29 2000 The participants at LAWST 3 were Chris Agruss James Bach Karla Fisher David Gelperin Kenneth Groder Elisabeth Hendrickson Doug Hoffman III recorder Bob Johnson Cem Kaner host Brian Lawrence facilitator Brian Marick Thanga Meenakshi Noel Nyman Jeffery E Payne Bret Pettichord Johanna Rothman Jane Stepak Melora Svoboda Jeremy White and Rodney Wilson The participants at LAWST 11 were Chris Agruss James Bach Hans Buwalda Marge Farrell Sam Guckenheimer Elisabeth Hendrickson Doug Hoffman III recorder Bob Johnson Karen Johnson Cem Kaner host Brian Lawrence facilitator Alan
24. Papers CO e Thomas Ostrand amp Mark Balcer The Category partition Method For Specifying And Generating Functional Tests Communications of the ACM Vol 31 No 6 1988 e Debra Richardson et al A Close Look at Domain Testing IEEE Transactions On Software Engineering Vol SE 8 NO 4 July 1982 e Michael Deck and James Whittaker Lessons learned from fifteen years of cleanroom testing STAR 97 Proceedings in this paper the authors adopt boundary testing as an adjunct to random sampling e Richard Hamlet amp Ross Taylor Partition Testing Does Not Inspire Confidence Proceedings of the Second Workshop on software Testing Verification and Analysis IEEE Computer Society Press 206 215 July 1988 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 121 Stress Testing OOo u e Tag line Overwhelm the product e Fundamental question or goal Learn about the capabilities and weaknesses of the product by driving it through failure and beyond What does failure at extremes tell us about changes needed in the program s handling of normal cases e Paradigmatic case s Buffer overflow bugs High volumes of data device connections long transaction chains Low memory conditions device failures viruses other crises e Strengths Expose weaknesses that will arise in the field Expose security risks e Blind spots Weaknesses that are not made more visible by s
25. Telephone example asserts Embedded software example diagnostics Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 148 Random Testing Stochastic Regression Based Loo O e Fundamental question or goal High volume random testing using random sequence of pre defined tests that can self check for pass fail e Paradigmatic case s Life testing Search for specific types of long sequence defects Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 149 Random Testing Stochastic Regression Based Loo O e Notes Create a series of regression tests Design them so that they don t reinitialize the system or force it to a standard starting state that would erase history The tests are designed so that the automation can identify failures Run the tests in random order over a long sequence This is a low mental overhead alternative to model based testing You get pass fail info for every test but without having to achieve the same depth of understanding of the software Of course you probably have worse coverage less awareness of your actual coverage and less opportunity to stumble over bugs Unless this is very carefully managed there is a serious risk of non reproduceability of failures 150 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Random Testing Sandboxing the Regression Tests oO
26. ach All rights reserved 26 Project Factors OOO e Stakeholders Anyone who is a client of the main project Anyone who is a client of the testing project e Includes customers purchasers end users tech support programmers project mgr doc group etc e Processes The tasks and events that comprise the main project e How the overall project is run The tasks and events that comprise the test project e How the testing project is run e Staff Everyone who helps develop the product e Sources of information and assistance Everyone who will perform or support testing e Special talents or experiences of team members e Size of the group e Extent to which they are focused or are multi tasking e Organization collaboration amp coordination of the staff e Is there an independent test lab Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 27 Project Factors OOO e Schedules The sequence duration and synchronization of events When will testing start and how long is it expected to take When will specific product elements be available to test When will devices or tools be available to support testing e Equipment Hardware required for testing What devices do we need to test the product with Do we have them e Tools amp Test Materials Software required or desired for testing Automation Are such tools available Do we want to use them Do we have them Do we
27. and James Bach All rights reserved 140 Random Statistical Testing Loo oO e Tag line High volume testing with new cases all the time e Fundamental question or goal Have the computer create execute and evaluate huge numbers of tests e The individual tests are not all that powerful nor all that compelling e The power of the approach lies in the large number of tests e These broaden the sample and they may test the program over a long period of time giving us insight into longer term issues Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 141 Random Statistical Testing OT e Paradigmatic case s Some of us are still wrapping our heads around the richness of work in this field This is a tentative classification e NON STOCHASTIC RANDOM TESTS e STATISTICAL RELIABILITY ESTIMATION e STOCHASTIC TESTS NO MODEL e STOCHASTIC TESTS USING ON A MODEL OF THE SOFTWARE UNDER TEST e STOCHASTIC TESTS USING OTHER ATTRIBUTES OF SOFTWARE UNDER TEST Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 142 Random Statistical Testing Non Stochastic Loo OO e Fundamental question or goal The computer runs a large set of essentially independent tests The focus is on the results of each test Tests are often designed to minimize sequential interaction among tests e Paradigmatic case s Function equivalence testing Compare t
28. ased test management e Our focus today is on methods for finding errors efficiently Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 39 Risks Where to look for errors OOO e Qualities Failure to conform to a quality criterion risk of unreliability risk of unmaintainability etc e New things newer features may fail e New technology new concepts lead to new mistakes New markets A different customer base will see and use the product differently e Learning Curve mistakes due to ignorance e Changed things changes may break old code e Late changes rushed decisions rushed or demoralized staff lead to mistakes e Rushed work some tasks or projects are chronically underfunded and all aspects of work quality suffer Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 40 Risks Where to look for errors o e Poor design or unmaintainable implementation Some internal design decisions make the code so hard to maintain that fixes consistently cause new problems e Tired programmers long overtime over several weeks or months yields inefficiencies and errors e Other staff issues alcoholic mother died two programmers who won t talk to each other neither will their code e Just slipping it in pet feature not on plan may interact badly with other code e N I H external components can cause problems e N I B not i
29. combinations of a system s test parameters These are the parameters that determine the system s test scenarios Examples are system configuration parameters user inputs and other external events We implemented this new method in the AETG i Sl S ea s2 z Copyright c 1997 1999 Cem Kaner All Rights Reserved 85 Combinations Exercise Illustration Find e Find whet flowereasel O O oOo Direction Cancel O Match case e Lp Down e Here is a simple Find dialog It takes three inputs Find what a text string Match case yes orno Direction up or down Simplify this by considering only three values for the text string lowercase and Mixed Cases and CAPITALS e Note To do a better job we d also choose input documents that would yield a find and a don t find for each case The input document would be another variable or really the intended result Find Don t would be the variable We ll think about that again after the exercise Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 86 Combinations Exercise __wwryr _ _ 1 How many combinations of these three variables are possible 2 List ALL the combinations of these three variables 3 Now create combination tests that cover all possible pairs of values but don t try to cover all possible triplets List one such set 4 How many test cases are in this se
30. d time operating system version i _ __ transitions between algorithms variations within a group of compatible optimizations different ways to printers sound cards modems etc compute a function m equivalent event times when something e most recent event first event happens timing how long between event A and event B and in which order races input or output intensity voltage e speed extent of voltage transition e g from very soft to very loud sound Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 68 Using Test Matrices for Routine Issues Loo OO e After testing a simple numeric input field a few times you may prefer a test matrix to present the same tests more concisely e Use a test matrix to show track a series of test cases that are fundamentally similar For example for most input fields you ll do a series of the same tests checking how the field handles boundaries unexpected characters function keys etc As another example for most files you ll run essentially the same tests on file handling e The matrix is a concise way of showing the repeating tests Put the objects that you re testing on the rows Show the tests on the columns Check off the tests that you actually completed in the cells Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 69 Reusable Test Matrix
31. doesn t work In what ways could your car not work yet seem to you that it does In what ways could your car work yet seem to you that it doesn t Do you know enough about cars to answer these questions Have you observed your car enough today to answer them Under what circumstances would these questions matter Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 14 Questioning OOOO e Requirements analysis requires information gathering Read books on consulting Gause amp Weinberg Exploring Requirements is an essential source on context free questioning e There are many types of questions Open vs closed Hypothetical vs behavioral Opinion vs factual Historical vs predictive Context dependent and context free Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 15 The classic context free questions Ee e The traditional newspaper reporters questions are Who What When Where How Why e For example Who will use this feature What does this user want to do with it Who else will use it Why Who will choose not to use it What do they lose What else does this user want to do in conjunction with this feature Who is not allowed to use this product or feature why and what security is in place to prevent them e We use these in conjunction with questions that come out of the testing model se
32. e below The model gives us a starting place We expand it by asking each of these questions as a follow up to the initial question Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 16 Context Free Questions Defining the Problem OOO Based on The CIA s Phoenix Checklists Thinkertoys p What problems are 140 and Bach s Evaluation Strategies Rapid Testing we trying to define Course notes What test plan an should te Why is it necessary to solve the problem Ael anh What benefits will you receive by solving the limited time problem resources and What is the unknown information What is it that you don t yet understand an arts ti What is the information that you have evecare Sana What is mat y l for a particular part What is the source of this problem Specs Field of the system experience An individual stakeholder s preference How have the Who are the stakeholders programmers How does it relate to which stakeholders AOT a difficult Cae 9 echnical issue if What isn t the problem oe E you can understand Is the information sufficient Or is it insufficient Or their approach you redundant Or contradictory can understand how Should you draw a diagram of the problem A figure to test it Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 17 Context Free Questions Defining
33. ect Graphing is more powerful if you have or can build a complete specification I started using this chart as an exploratory tool for simplifying my look at relationships in overwhelmingly complex programs There doesn t have to be a lot of complexity to be overwhelming Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved TI Tabular Format for Data Relationships Loo O e THE TABLE S FIELDS Field Create a row for each field Consultant End Date and Start Date are examples of fields Entry Source What dialog boxes can you use to enter data into this field Can you import data into this field Can data be calculated into this field List every way to fill the field every screen etc Display List every dialog box error message window etc that can display the value of this field When you re enter a value into this field will the new entry show up in each screen that displays the field Not always sometimes the program makes local copies of variables and fails to update them Print List all the reports that print the value of this field and any other functions that print the value Related to List every variable that is related to this variable What if you enter a legal value into this variable then change the value of a constraining variable to something that is incompatible with this variable s value Relationship Identify the relationship to the
34. etwork disk after opening file timeout waiting for network disk keyboard mouse I O during save to network disk other interrupt during save to network drive local power out during save to network network power during save to network Copyright 2001 Cem Kaner and James Bach All rights reserved 72 Routine Case Matrices _ NWW_ You can often re use a matrix like this across products and projects You can create matrices like this for a wide range of problems Whenever you can specify multiple tests to be done on one class of object and you expect to test several such objects you can put the multiple tests on the matrix Mark a cell green if you ran the test and the program passed it Mark the cell red if the program failed Write the bug number of the bug report for this bug Write in the cell the automation number or identifier or file name if the test case has been automated Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 73 Routine Case Matrices NN e Problems What if your thinking gets out of date What if this program poses new issues not covered by the standard tests Do you need to execute every test every time or ever What if the automation ID number changes We still have a maintenance problem but it is not as obscure Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved
35. g O e Tag line Simultaneous learning planning and testing e Fundamental question or goal Software comes to tester under documented and or late Tester must simultaneously learn about the product and about the test cases strategies that will reveal the product and its defects e Paradigmatic case s Skilled exploratory testing of the full product Rapid testing Emergency testing including thrown over the wall test it today testing Third party components Troubleshooting follow up testing of defects Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 3 8 Exploratory Testing Loo SS Strengths Customer focused risk focused Takes advantage of each tester s strengths Responsive to changing circumstances Well managed it avoids duplicative analysis and testing High bug find rates e Blind spots The less we know the more we risk missing Limited by each tester s weaknesses can mitigate this with careful management This is skilled work juniors aren t very good at it Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 39 Exploratory Testing Interesting Papers __ O e Chris Agruss amp Bob Johnson Ad Hoc Software Testing Exploring the Controversy of Unstructured Testing e Whittaker amp Jorgenson How to Break Software Test Documentation Copyright 2001 Cem Kaner
36. g process It could also indicate that the test plan is nothing but unchanged boilerplate Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 102 Heuristics for Test Plan Evaluation COO 11 The test project should use Many test projects suffer under the humans for what humans do well false belief that human testers are and use automation for what effective when they use exactingly automation does well Manual specified test scripts or that test testing should allow for automation duplicates the value of improvisation and on the spot human cognition in the test execution critical thinking while automated process Manual and automated testing should be used for tests that testing are not two forms of the same require high repeatability high thing They are two entirely different speed and no judgment classes of test technique Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 103 Heuristics for Test Plan Evaluation 12 The test schedule should be A monolithic test schedule in a test represented and justified in such a plan often indicates the false belief way as to highlight any that testing is an independent activity dependencies on the progress of The test schedule can stand alone development the testability of the only to the extent that the product the product time required to report highly testable development is problems and the project team s com
37. genson Software Testing A Craftsman s Approach Brian Marick Multi Generating test ideas from expressions with booleans and relational operators http www testing com tools multi README html Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 84 H The AETG System An Approach to Testing Based on Combinatorial Design Netscape File Edit Wiew Go Communicator Help BLS lt a Bookmarks inaner Netsite htps actgwebtipandring com papers AETGieeeS7 html A N The AETG System An Approach to Testing Based on Combinatorial Design Appeared in July 1997 issue of IEEE Transactions On Software Engineering Vol 23 No 7 By D M Cohen IDA CCS Work done while at Bellcore S R Dalal Bellcore M L Fredman Rutgers University G C Patton Bellcore TABLE OF CONTENTS Abstract 1 Introduction 2 The Basic Combinatorial Design Paradigm 3 Logarithmic Growth for n Way Interaction Testing 4 S A Heuristic Algorithm AETG Input Language 5 1 Constraints 5 2 Hierarchy and hierarchical testing 6 Experiments 7 Overview of Applications 7 1 High Level Test Planning 7 2 Test Case Generation 8S Related Methods 9 Summary Acknowledgements References Abstract This paper describes a new approach to testing that uses combinatorial designs to generate tests that cower the pair wise triple or n way
38. hat detect and recover from errors including error messages Testability functions provided to help test the product such as diagnostics log files asserts test menus etc e Temporal relationships How the program functions over time Sequential operation state to state transitions Data changes in variables over time System interactions such as synchronization or ordering of events in distributed systems Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 32 Product Elements Loo UU e Data Everything that the product processes Input data that is processed by the product Output data that results from processing by the product Preset data supplied as part of the product or otherwise built into it such as prefab databases default values etc Persistent data stored internally and expected to persist over multiple operations This includes modes or states of the product such as options settings view modes contents of documents etc Temporal data based on time such as date stamps or number of events recorded in a unit of time Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 33 Product Elements Do e Platform Everything on which the product depends External Hardware components and configurations that are not part of the shipping product but are required or optional in order for the product to work Includes CPU s memory ke
39. he output if the program fails report a bug and try again later if the program passes the test save the resulting outputs in future tests run the program and compare the output to the saved results Report an exception whenever the current output and the saved output don t match Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 14 Potential Regression Advantages OO u u e Dominant paradigm for automated testing e Straightforward e Same approach for all tests e Relatively fast implementation e Variations may be easy e Repeatable tests Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 15 GUI Regression Interesting Papers OOO e Chris Agruss Automating Software Installation Testing e James Bach Test Automation Snake Oil e Hans Buwalda Testing Using Action Words e Hans Buwalda Automated testing with Action Words Abandoning Record amp Playback e Elisabeth Hendrickson The Difference between Test Automation Failure and Success e Cem Kaner Avoiding Shelfware A Manager s View of Automated GUI Testing e John Kent Advanced Automated Testing Architectures e Bret Pettichord Success with Test Automation e Bret Pettichord Seven Steps to Test Automation Success e Keith Zambelich Totally Data Driven Automated Testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 16 Domain Testing OOOO e
40. he test docs help us identify and revise restructure in face of a permanent shift in the risk profile of the program e Are should docs be automatically created as a byproduct of the test automation code Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 23 Ultimately write a mission statement NT e Try to describe your core documentation requirements in one sentence that doesn t have more than three components e Examples The test documentation set will primarily support our efforts to find bugs in this version to delegate work and to track status The test documentation set will support ongoing product and test maintenance over at least 10 years will provide training material for new group members and will create archives suitable for regulatory or litigation use Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 24 A Model of Software Testing OO Environment Criteria Test Techniques Elements Test Results Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 25 Project Environment Factors _ s s OO Stakeholders Processes Staff Schedules Equipment These aspects of the Tools amp Test Materials environment constrain and Information enable the testing project Items Under Test Logistics Budget Deliverables Test Documentation Copyright 2001 Cem Kaner and James B
41. iques can you use to generate ideas How many different techniques Can you see the result How many different kinds of results can you see How many different ways have you tried to solve the problem Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 107 Evaluating Your Plan Context Free Questions Loo What have others done Can you intuit the solution Can you check the results What should be done How should it be done Where should it be done When should it be done Who should do it What do you need to do at this time Who will be responsible for what Can you use this problem to solve some other problem What is the unique set of qualities that makes this problem what it is and none other What milestones can best mark your progress How will you know when you are successful How conclusive and specific is your answer Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 108 Appendix on General Test Techniques Loo OO e The following slides review 10 general test techniques In previous talks we ve called these paradigms because many companies have organized their entire testing effort and testing thinking around one or two of them e We wont discuss many of these slides in the workshop but we hope that these will be helpful reference materials to add some detail to comments
42. itions and detect invalid states This work Whittaker 1997b developed the concept of an operational mode that functionally decomposes abstracts states Operational modes provide a mechanism to encapsulate and describe state complexity By expressing states as the cross product of operational modes and eliminating impossible states the number of distinct states can be reduced alleviating the state explosion problem Operational modes are not a new feature of software but rather a different way to view the decomposition of states All software has operational modes but the implementation of these modes has historically been left to chance When used for testing operational modes have been extracted by reverse engineering Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 155 Random Testing Thoughts Toward an Architecture OOOO e We have a population of tests which may have been sandboxed and which may carry self deck info A test series involves a sample of these tests e We have a population of diagnostics probably too many to run every time we run a test In a given test series we will run a subset of these e We have a population of possible configurations some of which can be set by the software In a given test series we initialize by setting the system to a known configuration We may reset the system to new configurations during the series e g every 5th test Test Doc
43. les of these channels are uncovered inspections field testing or informal testing by people outside of the test team 16 All documentation related to the test Tunnel vision is the great occupational strategy including test cases and hazard of testing Review not only procedures should be undergo review helps to reveal blind spots in test by someone other than the person who design but it can also help promote wrote them The review process used dialog and peer education about test should be commensurate with the practices criticality of the document Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 106 Evaluating Your Plan Context Free Questions Loo oO Based on The CIA s Phoenix Checklists Thinkertoys p 140 and Bach s Evaluation Strategies Rapid Testing Course notes Can you solve the whole problem Part of the problem What would you like the resolution to be Can you picture it How much of the unknown can you determine What reference data are you using if any What product output will you evaluate How will you do the evaluation Can you derive something useful from the information you have Have you used all the information Have you taken into account all essential notions in the problem Can you separate the steps in the problem solving process Can you determine the correctness of each step What creative thinking techn
44. leteness OOO Reading a spec linearly is not a particularly effective way to read the document It s too easy to overlook key missing SSUES eWe may not have time to walk through this method in this class but the general approach that use is based on James Bach s Satisfice Heuristic Test Strategy Model at http www satisfice com tools satisfice tsm 4p pdf You can assume not always correctly but usually that every sentence in the spec is meant to convey information The information will probably be about e the project and how it is structured funded or timed or e about the product what it is and how it works or e about the quality criteria that you should evaluate the product against 126 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Scenario Testing i OO Tag lines Do something useful and interesting Do one thing after another Fundamental question or goal Challenging cases that reflect real use Paradigmatic case s Appraise product against business rules customer data competitors output Life history testing Hans Buwalda s soap opera testing Use cases are a simpler form often derived from product capabilities and user model rather than from naturalistic observation of systems of this kind Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 127 Scena
45. ll rights reserved Problems with the allegedly standard approach OOO It is essential to understand your requirements for test documentation Unless following a standard helps you meet your requirements it is empty at best anti productive at worst Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 8 Requirements OT There are many different notions of what a good set of test documentation would include Before spending a substantial amount of time and resources it s worth asking what documentation should be developed and why Test documentation is expensive and it takes a long time to produce If you figure out some of your main requirements first you might be able to do your work in a way that achieves them Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 9 Defining documentation requirements __ OO e Stakeholders interests actions objects Who would use or be affected by test documentation What interests of theirs does documentation serve or disserve What will they do with the documentation What types of documents are of high or low value e Asking questions e Context free questions e Context free questions specific to test planning e Evaluating a plan Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 10 Discovering Requirements OOO e Requirements
46. n budget Unbudgeted tasks may be done shoddily Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 41 Risks Where to look for errors o e Ambiguity ambiguous descriptions in specs or other docs can lead to incorrect or conflicting implementations e Conflicting requirements ambiguity often hides conflict result is loss of value for some person e Unknown requirements requirements surface throughout development Failure to meet a legitimate requirement is a failure of quality for that stakeholder e Evolving requirements people realize what they want as the product develops Adhering to a start dhe project requirements list may meet contract but fail product check out http www agilealliance org Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 42 Risks Where to look for errors OOO Complexity complex code may be buggy Bugginess features with many known bugs may also have many unknown bugs Dependencies failures may trigger other failures Untestability risk of slow inefficient testing Little unit testing programmers find and fix most of their own bugs Shortcutting here is a risk Little system testing so far untested software may fail Previous reliance on narrow testing strategies e g regression function tests can yield a backlog of errors surviving across versions Test Documentation Copyright 2
47. ner and James Bach All rights reserved 48 Test Case Design COO e If the purpose of testing is to gain information about the product then a test case s function is to elicit information quickly and efficiently e In information theory we define information in terms of reduction of uncertainty If there is little uncertainty there is little information to be gained e A test case that promises no information is poorly designed A good test case will provide information of value whether the program passes the test or fails it Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 49 Thinking About Test Techniques NNN o e Analyze the situation e Model the test space e Select what to cover e Determine test oracles e Configure the test system e Operate the test system e Observe the test system e Evaluate the test results A test technique is a recipe for performing these tasks that will reveal something worth reporting Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 5 0 Thinking About Test Techniques OT e What is the difference between User testing Usability testing User interface testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 5I Thinking About Test Techniques O u u e Testing combines techniques that focus on Testers who does the testing Co
48. odel C Basis for the Heuristic The later in the project that a problem is found the greater the risk that it will not be safely fixed in time to ship The sooner a problem is found after it is created the lesser 1 Testing should be optimized to find important problems fast rather than attempting to find all problems with equal urgency 2 Test strategy should focus most effort on areas of potential technical risk while still putting some effort into low risk areas just in case the risk analysis is wrong 3 Test strategy should address test platform configuration how the product will be operated how the product will be observed and how observations will be used to evaluate Test Documentation Complete testing is impossible and we can never know if our perception of technical risk is completely accurate Sloppiness or neglect within any of these four basic testing activities will increase the likelihood that important problems will go undetected Copyright 2001 Cem Kaner and James Bach All rights reserved J 8 Heuristics for Test Plan Evaluation EEE 4 Test strategy should be diversified in terms of test techniques and perspectives Methods of evaluating test coverage should take into account multiple dimensions of coverage including structural functional data platform operations and requirements 5 The test strategy should specify how test data will be designed and generated No single
49. of tests Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 32 Evaluating Risk OO e Many of us who think about testing in terms of risk analogize testing of software to the testing of theories Karl Popper in his famous essay Conjectures and Refutations lays out the proposition that a scientific theory gains credibility by being subjected to and passing harsh tests that are intended to refute the theory We can gain confidence in a program by testing it harshly if it passes the tests Subjecting it to easy tests doesn t tell us much about what will happen to the program in the field Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 33 Risk Based Testing Interesting Papers OO e Stale Amland Risk Based Testing e James Bach Reframing Requirements Analysis e James Bach Risk and Requirements Based Testing e James Bach James Bach on Risk Based Testing e Stale Amland amp Hans Schaefer Risk based testing a response e Carl Popper Conjectures amp Refutations Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 134 User Testing OO e Tag line Strive for realism Let s try this with real humans for a change e Fundamental question or goal Identify failures that will arise in the hands of a person 1 e breakdowns in the overall human machine software system e Paradigmatic case
50. ons but the cost can be high and the emphasis on paths is troublesome because of the high number of possible paths through the program e Clarke Hassell amp Richardson p 388 118 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Equivalence and Risk EE Our working definition of equivalence Two test cases are equivalent if you expect the same result from each This is fundamentally subjective It depends on what you expect And what you expect depends on what errors you can anticipate Two test cases can only be equivalent by reference to a specifiable risk Two different testers will have different theories about how programs can fail and therefore they will come up with different classes A boundary case in this system is a best representative A best representative of an equivalence class is a test that is at least as likely to expose a fault as every other member of the class Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 1 19 Domain Testing OOOO e Strengths Find highest probability errors with a relatively small set of tests Intuitively clear approach generalizes well e Blind spots Errors that are not at boundaries or in obvious special cases Also the actual domains are often unknowable Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 120 Domain Testing Interesting
51. out of memory how many names in a mailing list records in a database variables in a spreadsheet bookmarks abbreviations size of the sum of variables or of some other computed value think binary and think digits Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved size of a number that you enter number of digits or size of a character string size of a concatenated string size of a path specification size of a file name size in characters of a document size of a file note special values such as exactly 64K exactly 512 bytes etc size of the document on the page compared to page margins across different page margins page sizes 67 Variables Well Suited to Equivalence Class Analysis _ OOO size of a document on a page in terms length of time after a timeout from of the memory requirements for the JUST before to way after what page This might just be in terms of events are important resolution x page size but it may be speed of data entry time between more complex if we have compression keystrokes menus etc equivalent output events such as e speed of input handling of concurrent printing documents events amount of available memory gt 128 e number of devices connected active meg gt 640K etc system resources consumed available visual resolution size of screen number also handles stack space etc of colors date an
52. owing slides give examples of several charts tables etc e You probably won t have enough time to create all the documentation that would be useful Treat these materials as optional e Use the components that you find most useful to Clarify your own thinking Communicate your thinking to others Track your work or the work of someone else Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 60 Basic Test Documentation Components _ O Lists Such as lists of fields error messages DLLs Outlines An outline organizes information into a hierarchy of lists and sublists Such as the testing objectives list later in the course notes Tables A table organizes information in two dimensions showing relationships between variables Such as boundary tables decision tables combination test tables Matrices A matrix is a special type of table used for data collection Such as the numeric input field matrix configuration matrices Refer to Testing Computer Software pages 217 241 For more examples see page Testing Computer Software page 218 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 61 Traceability Matrix iar Yar vers ior es rave a A reel ill ll a or i oe a O Lic CN A 2 per i id Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 62 Traceability Matrix Loo O e The
53. plete and the test process is not assessment of risk interrupted by the frequent need to report problems 13 The test process should be kept This is important in order to deflect off of the critical path to the extent pressure to truncate the testing possible This can be done by testing process in parallel with development work and finding problems worth fixing faster than the developers fix them Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 104 Heuristics for Test Plan Evaluation OOO 14 The feedback loop between This is important in order to maximize testers and developers should be the efficiency and speed of quality as tight as possible Test cycles improvement It also helps keep testing should be designed to provide rapid off of the critical path feedback to developers about recent additions and changes they have made before a full regression test is commenced Whenever possible testers and developers should work physically near each other Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 105 Heuristics for Test Plan Evaluation ee 15 The test project should employ By examining product quality channels of information about quality information gathered through various other than formal testing in order to help means beyond the test team blind evaluate and adjust the test project spots in the formal test strategy can be Examp
54. priate design a test or series of tests for bugs of this type Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 46 Build Your Own Model of Bug Patterns OOO Too many people start and end with the TCS bug list It is outdated It was outdated the day it was published And it doesnt cover the issues in your system Building a bug list is an ongoing process that constantly pays for itself Here s an example from Hung Nguyen This problem came up in a client server system The system sends the client a list of names to allow verification that a name the client enters is not new Client 1 and 2 both want to enter a name and client 1 and 2 both use the same new name Both instances of the name are new relative to their local compare list and therefore they are accepted and we now have two instances of the same name As we see these we develop a library of issues The discovery method is exploratory requires sophistication with the underlying technology Capture winning themes for testing in charts or in scripts on their way to being automated Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 47 Building Bug Patterns OOO There are plenty of sources to check for common failures in the common platforms www bugnet com Www cnet com links from www winfiles com various mailing lists Test Documentation Copyright 2001 Cem Ka
55. pt on column 5 We achieve all pairs of GH with columns 1 2 and 3 but miss it for column 4 The most recent arbitrary choice was HG in the 2nd section Once that was determined we picked HG for the third in order to pair H with a 1 in the third column So we will erase the last choice and try again Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 92 Combination Testing e We flipped the last arbitrary choice column 5 section 2 to GH from HG and erased section 3 We then fill in section 3 by checking for missing pairs GH GH gives us two XG XG pairs so we flip to HP for the third section and have a column 2 X with a column 5 H and a column 2 Y witha column 5 G as needed to obtain all pairs Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 93 Combination Testing D u u e But when we add the next column we see that we just can t achieve all pairs with 6 values The first one works up to column 4 but then fails to get pair EJ or Fl The next fails on Gu HI Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 94 Combination Testing e When all else fails add rows We need one for GJ and one for HI so add two rows In general we would need as many rows as the last column has values e The other values in the two rows are arbitrary leave them blank and fill them in as needed when you add new column
56. ral requirements e Intercase dependencies Test procedure specification test case Test item transmittal report Test log Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach OOO e What is the documentation cost per test case e What is the maintenance cost of the documentation per test case e f software design changes create documentation maintenance costs how much inertia do we build into our system How much does extensive test documentation add to the cost of late improvement of the software How much should we add e What inertia is created in favor of invariant regression testing e ls this incompatible with exploratory testing Do we always want to discourage exploration Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Problems with the allegedly standard approach OOO e What is the impact on high dume test automation e How often do project teams start to follow 829 but then give it up mid poect What does this do to the net quality of the test documentation and test planning effort e WHAT REQUIREMENTS DOES A STANDARD LIKE THIS FULFILL e WHICH STAKEHOLDERS GAIN A NET BENEFIT FROM IEEE STANDARD DOCUMENTATION e WHAT BENEFITS DO THEY GAIN AND WHY ARE THOSE BENEFITS IMPORTANT TO THEM Test Documentation Copyright 2001 Cem Kaner and James Bach A
57. rience or solution provided to a customer Everything that comes in the box plus the box Functions and data executed on a platform that serve a purpose for a user 1 A software product is much more than code 2 It involves a purpose platform and user 3 It consists of many interdependent elements Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 30 Product Elements Loo O e Structures Everything that comprises the physical product Code the code structures that comprise the product from executables to individual routines Interfaces points of connection and communication between subsystems Hardware hardware components integral to the product Non executable files any files other than programs such as text files sample data help files etc Alternate Media anything beyond software and hardware such as paper documents web links and content packaging license agreements etc Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 3 1 Product Elements Loo o Functions Everything that the product does User Interface functions that mediate the exchange of data with the user System Interface functions that exchange data with something other than the user such as with other programs hard disk network printer etc Application functions that define or distinguish the product or fulfill core requirements Error Handling functions t
58. rio Testing C O u u u e he ideal scenario has several characteristics tis realistic e g it comes from actual customer or competitor situations There is no ambiguity about whether a test passed or failed The test is complex that is it uses several features and functions There is a stakeholder who will make a fuss if the program doesn t pass this scenario e Strengths Complex realistic events Can handle help with situations that are too complex to model Exposes failures that occur develop over time e Blind spots Single function failures can make this test inefficient Must think carefully to achieve good coverage Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 128 Scenario Testing Interesting Papers O e Hans Buwalda on Soap Operas in the conference proceedings of STAR East 2000 e Kaner A pattern for scenario testing at www testing com e Lots of literature on use cases Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 129 Risk Based Testing O u u e Tag line Find big bugs first e Fundamental question or goal Define and refine tests in terms of the kind of problem or risk that you are trying to manage OR prioritize the testing effort in terms of the relative risk of different areas or issues we could test for e Paradigmatic case s Failure Mode and Effects Analysi
59. s At the very end fill the remaining blank ones with arbitrary values Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 95 Combination Testing a elf a variable is continuous but maps to a number line partition and use boundaries as the distinct values under test If all variables are continuous we end up with all pairs of all boundary tests of all variables We don t achieve all triples all quadruples etc elf some combinations are of independent interest add them to the list of n tuples to test With the six columns of the example we reduced 96 tests to 8 Give a few back make it 12 or 15 tests and you still get enormous reduction Examples of independent interest are known from tech support high risk cases cases that jointly stress memory configuration combinations Var 1 is operating systems Var 2 is printers etc that are prevalent in the market etc Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 96 Charts References O u u You can find plenty of example charts in Bill Perry s books such as Effective Methods for Software Testing 2nd Ed Wiley Several of these will probably be useful though like the charts in these notes you ll have to adapt them to your circumstances Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 97 Heuristics from James Bach s Test Plan Evaluation M
60. s FMEA Equivalence class analysis reformulated Test in order of frequency of use Musa Stress tests error handling tests security tests tests looking for predicted or feared errors sample from predicted bugs list Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 130 Risk Based Testing OOO Strengths Optimal prioritization assuming we correctly identify and prioritize the risks High power tests e Blind spots Risks that were not identified or that are surprisingly more likely Some risk driven testers seem to operate too subjectively How will I know what level of coverage that I ve reached How do I know that I haven t missed something critical Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 13 1 Evaluating Risk D u u e Several approaches that call themselves risk based testing ask which tests we should run and which we should skip if we run out of time e We think this is only half of the risk story The other half is focuses on test design It seems to us that a key purpose of testing is to find defects So a key strategy for testing should be defect based Every test should be questioned e How will this test find a defect e What kind of defect do you have in mind e What power does this test have against that kind of defect Is there a more powerful test A more powerful suite
61. s possible values are A B C and V2 is 2 then column 1 would contain A A blank row B B blank row C C blank row Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 89 Combination Testing TT Building an alk fairs combination table In the second column list all the values of the variable skip the line list the values etc For example if variable 2 s possible values are X Y then the table looks like this so far Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 90 Combination Testing Building an alk fairs combination table Each section of the third column think of AA as defining a section BB as defining another etc will have to contain every value of variable 3 Order the values such that the variables also make all pairs with variable 2 Suppose variable 2 can be 1 0 The third section can be filled in either way and you might highlight it so that you can reverse it later The decision say 1 0 is arbitrary Now that we ve solved the 3 column exercise let s try adding more variables Each of them will have two values Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 91 Combination Testing e The 4th column went in easily note that we started by making sure we hit all pairs of values of column 4 and column 2 then all pairs of column 4 and column 3 Watch this first attem
62. served 152 Random Testing Model based Stochastic Tests ____ _ OO e Fundamental Question or Goal Build a state model of the software The analysis will reveal several defects in itself Generate random events inputs to the program The program responds by moving to a new state Test whether the program has reached the expected State e Paradigmatic case s I haven t done this kind of work Here s what I understand e Works poorly for a complex product like Word e Likely to work well for embedded software and simple menus think of the brakes of your car or walking a control panel on a printer e In general well suited to a limited functionality client that will not be powered down or rebooted very often e Maintenance is a critical issue because design changes add or subtract nodes forcing a regeneration of the model Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 153 Random Testing Model based Stochastic Tests _ _ O Alan Jorgensen Software Design Based on Operational Modes Ph D thesis Florida Institute of Technology The applicability of state machine modeling to mechanical computation dates back to the work of Mealy Mealy 1955 and Moore Moore 1956 and persists to modern software analysis techniques Mills et al 1990 Rumbaugh et al 1999 Introducing state design into software development process began in earnest in the late 1980 s with the adven
63. ss amenable to automation Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 15 8 Random Statistical Testing O u u e Blind spots Testers might spend much more time analyzing the code and too little time analyzing the customer and her uses of the software Potential to create an inappropriate prestige hierarchy devaluating the skills of subject matter experts who understand the product and its defects much better than the automators Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 159 Random Testing Interesting Papers CO e Larry Apfelbaum Model Based Testing Proceedings of Software Quality Week 1997 not included in the course notes e Michael Deck and James Whittaker Lessons learned from fifteen years of cleanroom testing STAR 97 Proceedings e Doug Hoffman Mutating Automated Tests e Alan Jorgensen An API Testing Method e Noel Nyman GUI Application Testing with Dumb Monkeys e Harry Robinson Finite State Model Based Testing on a Shoestring e Harry Robinson Graph Theory Techniques in Model Based Testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 160
64. t Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 87 Combination Testing OT e Imagine a program with 3 variables V1 has 3 possible values V2 has 2 possible values and V3 has 2 possible values e lf V1 and V2 and V3 are independent the number of possible combinations is 12 3 x 2 x 2 e Building a simple combination table Label the columns with the variable names listing variables in descending order of number of possible values Each column before the last will have repetition Suppose that A B and C are in column K of N columns To determine how many times rows in which to repeat A before creating a row for B multiply the number of variable values in columns K 1 K 2 N Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 88 Combination Testing TT Building an alk fairs combination table Label the columns with the variable names listing variables in descending order of number of possible values If the variable in column 1 has V1 possible values and the variable in column 2 has V2 possible values then there will be at least V1 x V2 rows draw the table this way but leave a blank row or two between repetition groups in column 1 Fill in the table one column at a time The first column repeats each of its elements V2 times skips a line and then starts the repetition of the next element For example if variable 1
65. t of the cleanroom software engineering methodology Mills et al 1987 and the introduction of the State Transition Diagram by Yourdon Yourdon 1989 A deterministic finite automata DFA is a state machine that may be used to model many characteristics of a software program Mathematically a DFA is the quintuple M Q S d q0 F where M is the machine Q is a finite set of states S is a finite set of inputs commonly called the alphabet d is the transition function that maps Q x S to Q q0 is one particular element of Q identified as the initial or stating state and F c Q is the set of final or terminating states Sudkamp 1988 The DFA can be viewed as a directed graph where the nodes are the states and the labeled edges are the transitions corresponding to inputs Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 154 Random Testing Model based Stochastic Tests Loo OO Alan Jorgensen Software Design Based on Operational Modes Ph D thesis Florida Institute of Technology When taking this state model view of software a different definition of software failure suggests itself The machine makes a transition to an unspecified state From this definition of software failure a software defect may be defined as Code that for some input causes an unspecified state transition or fails to reach a required state Recent developments in software system testing exercise state trans
66. test The testing software might be able to detect failures based on crash performance lags diagnostics or improper interaction with other better understood parts of the system but it cannot detect a failure simply based on the question Is the program doing what it is supposed to or not Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 146 Stochastic Tests No Model Dumb Monkeys O O e Paradigmatic case s Executive monkeys Know nothing about the system Push buttons randomly until the system crashes Clever monkeys More careful rules of conduct more knowledge about the system or the environment See Freddy O S compatibility testing No model of the software under test but diagnostics might be available based on the environment the NT example Early qualification testing Life testing Load testing e Notes Can be done at the API or command line just as well as via UI Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Stochastic assert or diagnostics based random tests Loo e Fundamental question or goal High volume random testing using random sequence of fresh or pre defined tests that may or may not self check for pass fail The primary method for detecting pass fail uses assertions diagnostics built into the program or other e g system diagnostics e Paradigmatic case s
67. test technique can reveal all important problems in a linear fashion We can never know for sure if we have found all the problems that matter Diversification minimizes the risk that the test strategy will be blind to certain kinds of problems Use diverse half measures to go after low hanging fruit It is common for the test strategy to be organized around functionality or code leaving it to the testers to concoct test data on the fly Often that indicates that the strategy is too focused on validating capability and not focused enough on reliability Heuristics for Test Plan Evaluation C 6 Not all testing should be pre A rigid test strategy may make it more specified in detail The test strategy likely that a particular subset of problems should incorporate reasonable variation will be uncovered but in a complex and make use of the testers ability to system it reduces the likelihood that all use situational reasoning to focuse on important problems will be uncovered important but unanticipated problems Reasonable variability in testing such as that which results from interactive exploratory testing increases incidental test coverage without substantially sacrificing essential coverage 7 It is important to test against implied Testing only against explicit written requirements the full extent of what requirements will not reveal all important the requirements mean not just what problems since
68. that we make about these techniques in class e There is nothing magical about these techniques They overlap They don t collectively cover everything that would be good to do e Imagine that you are one of the people who has adopted one of these techniques as your primary approach your paradigm What makes for an excellent test What is your approach best for What are some weaknesses in your approach Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 109 Function Testing ss OO e Tag line Black box unit testing e Fundamental question or goal Test each function thoroughly one at a time e Paradigmatic case s Spreadsheet test each item in isolation Database test each report in isolation e Strengths Thorough analysis of each item tested e Blind spots Misses interactions misses exploration of the benefits offered by the program 110 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved Some Function Testing Tasks Loo o Identify the program s features commands From specifications or the draft user manual From walking through the user interface From trying commands at the command line From searching the program or resource files for command names Identify variables used by the functions and test their boundaries Identify environmental variables that may constrain the function under
69. tress Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 122 Stress Testing Interesting Papers CO e Astroman66 Finding and Exploiting Bugs 2600 e Bruce Schneier Crypto Gram May 15 2000 e James A Whittaker and Alan Jorgensen Why Software Fails e Whittaker amp Jorgenson How to Break Software Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 123 Specification Driven Testing OOOO e Tag line Verify every claim e Fundamental question or goal Check the product s conformance with every statement in every spec requirements document etc e Paradigmatic case s Traceability matrix tracks test cases associated with each specification item User documentation testing Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 124 Specification Driven Testing OOOO Strengths Critical defense against warranty claims fraud charges loss of credibility with customers Effective for managing scope expectations of regulatory driven testing Reduces support costs customer complaints by ensuring that no false or misleading representations are made to customers e Blind spots Any issues not in the specs or treated badly in the specs documentation Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 125 Reviewing a Specification for Comp
70. umentation Copyright 2001 Cem Kaner and James Bach All rights reserved 156 Random Testing Thoughts Toward an Architecture OOOO e We have an execution tool that takes as input a list of tests or an algorithm for creating a list a list of diagnostics initial diagnostics at start of testing diagnostics at start of each test diagnostics on detected error and diagnostics at end of session an initial configuration and alist of configuration changes on specified events e The tool runs the tests in random order and outputs results to a standard format log file that defines its own structure so that multiple different analysis tools can interpret the same data Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 15 7 Random Statistical Testing Loo DD e Strengths Regression doesn t depend on same old test every time Partial oracles can find errors in young code quickly and cheaply Less likely to miss internal optimizations that are invisible from outside Can detect failures arising out of long complex chains that would be hard to create as planned tests e Blind spots Need to be able to distinguish pass from failure Too many people think Not crash not fail Executive expectations must be carefully managed These methods will often cover many types of risks but will obscure the need for other tests le
71. verage what gets tested Potential problems why you re testing what risk you re testing for Activities how you test Evaluation how to tell whether the test passed or failed e All testing involves all five dimensions e A technique focuses your attention on one or a few dimensions leaving the others open to your judgment You can combine a technique focused on one dimension with techniques focused on the other dimensions to achieve the result you want Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 52 Thinking About Test Techniques Loo o e Examples Testers e User testing Beta testing Subject matter experts Coverage e Function testing Domain testing State based testing Path testing Statement coverage Configuration coverage Potential problems e Input output computation storage constraints Risk based testing Activities e Exploratory testing Scenario testing Load testing Performance testing Evaluation e Oracle based testing Comparison with saved results e These examples are not definitive how you classify a testing approach depends on what you think is most central to it For example is load testing problem oriented denial of service or activity oriented e The important thing is to conscious manage the 5 dimensions Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 53 General Test
72. wo functions e g math functions using the second as an oracle for the first Attempt to demonstrate that they are not equivalent 1 e that the achieve different results from the same set of inputs Other test using fully deterministic oracles see discussion of oracles below Other tests using heuristic oracles see discussion of oracles below Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 143 Statistical Reliability Estimation OO e Fundamental question or goal Use random testing possibly stochastic possibly oracle based to estimate the stability or reliability of the software Testing is being used primarily to qualify the software rather than to find defects e Paradigmatic case s Clean room based approaches Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 144 The Need for Stochastic Testing An Example ae Caller h Tou Qa Connected ung up t On Hold iHi Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 145 Stochastic Tests No Model Dumb Monkeys o EE e Fundamental question or goal High volume testing involving a long sequence of tests A typical objective is to evaluate program performance over time The distinguishing characteristic of this approach is that the testing software does not have a detailed model of the software under
73. yboards peripheral boards etc External Software software components and configurations that are not a part of the shipping product but are required or optional in order for the product to work Includes operating systems concurrently executing applications drivers fonts etc e Operations How the product will be used Usage Profile the pattern of usage over time including patterns of data that the product will typically process in the field This varies by user and type of user Environment the physical environment in which the product will be operated including such elements as light noise and distractions Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 34 Product Elements Coveras2 C Product coverage is the proportion of the product that has been tested y KINdS of coverage as there are ways to i10del the product Structural Functional See Software Negligence Temporal amp Testing Coverage at Data www kaner com for 101 Platform examples of coverage Operations measures Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 35 Quality Criteria O u u u Accessibility Maintainability Quality is value Capability e Performance to some person Compatibility Portability Jerry Concurrency Recoverability Weinberg Conformance Reliability to Standards Scalability Efficiency Security
74. ys to feed it bad data Note that we ve dropped the issue of valid and invalid This lets us generalize to partitioning strategies that don t have the concept of valid for example printer equivalence classes Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 65 Equivalence Classes A Broad Concept OOO The notion of equivalence class is much broader than numeric ranges Here are some examples Membership in a common group e such as employees vs non employees Note that not all classes have shared boundaries Equivalent hardware e such as compatible modems Equivalent event times e such as before timeout and after Equivalent output events e perhaps any report will do to answer a simple the question Will the program print reports Equivalent operating environments e such as French amp English versions of Windows 3 1 Test Documentation Copyright 2001 Cem Kaner and James Bach All rights reserved 66 Variables Well Suited to Equivalence Class Analysis Loo OOO Many types of variables including input output internal hardware and system software configurations and equipment states can be subject to equivalence class analysis Here are some examples ranges of numbers character codes how many times something is done e g shareware limit on number of uses of a product e g how many times you can do it before you run

Download Pdf Manuals

Related Search

Related Contents

Guia do Usuário  Epson EB-1751  Manuale tecnico pompe di calore ad alta efficienza Pompe di calore  Puchacz SD50 G-CHEP  高水準GR スタンダード。  SERVICE MANUAL  

Copyright © All rights reserved. Failed to retrieve file