Home

slides

1. Effective for managing scope expectations of regulatory driven testing Reduces support costs customer complaints by ensuring that no false or misleading representations are made to customers e Blind spots Any issues not in the specs or treated badly in the specs documentation Teaching Testing Copyright 2001 Cem Kaner All rights reserved 99 Specification OO e Tasks review specifications for e Ambiguity e Adequacy it covers the issues e Correctness it describes the program e Content not a source of design errors e Testability support Create traceability matrices Document management spec versions file comparison utilities for comparing two spec versions etc Participate in review meetings Teaching Testing Copyright 2001 Cem Kaner All rights reserved 100 Specification OO e Skills Understand the level of generality called for when testing a spec item For example imagine a field X e We could test a single use of X e Or we could partition possible values of X and test boundary values e Or we could test X in various scenarios e Which is the right one Ambiguity analysis e Richard Bender teaches this well If you can t take his course you can find notes based on his work in Rodney Wilson s Software RX Secrets of Engineering Quality Software Teaching Testing Copyright 2001 Cem Kaner All rights reserved 101 Specification TT e Skills Ambigu
2. The program will print the sum Press Enter after each number To start the program type ADDER Before you start testing do you have any questions about the spec Teaching Testing Copyright 2001 Cem Kaner All rights reserved 27 Let s Brainstorm Some Tests EE ee e Brainstorming Rules The goal is to get lots of ideas You are brainstorming together to discover categories of possible tests There are more great ideas out there than you think Don t criticize others contributions Jokes are OK and are often valuable Work later alone or in a much smaller group to eliminate redundancy cut bad ideas and refine and optimize the specific tests Facilitator and recorder keep their opinions to themselves Teaching Testing Copyright 2001 Cem Kaner All rights reserved 28 Let s Brainstorm Some Tests Doo KB ae What Why Teaching Testing Copyright 2001 Cem Kaner All rights reserved 29 Traditional Presentation Myers e One input or output field The valid values for the field fit within one 1 easily specified range Valid values 99 to 99 Invalid values value lt value gt 99 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 30 Myers Boundary Table Variable Valid Case Invalid Case Boundaries Equivalence Equivalence and Special Classes Classes 99 to 99 gt 99 lt 99 Second
3. Empathetic Empirical frame of managers reference Empowering Fast abstraction skills Financially aware and Finds bugs intuitive Flexible sophisticated tester Goal setting Glue promotes group Humility cohesiveness Integrity honest keeps Interpersonally Interviewer commitments perceptive Investigative reader Leadership Long term thinker Meeting manager Multi tasking Organizer and planner Politically perceptive Policy and procedure Pragmatic Programmer developer Teaching Testing Copyright 2001 Cem Kaner All rights reserved 122 Some characteristics of great testers OO Protective stands behind Punctual Scholarly collects his staff information can back up or evaluate arguments Spoken communication Strength of character Subject matter expert Substance abuser not Team builder Tolerant of ambiguity Tolerant of different UI design development approaches Versatile many abilities Warm makes the human Written communication environment more pleasant Zealot Rarely desirable in Catalyst large quantities Teaching Testing Copyright 2001 Cem Kaner All rights reserved 123 Closing Notes Loo oO e Given a group of motivated bright people who have appropriate background knowledge and belong to your testing group or your testing class Only some testers will learn well in lectures Only some testers will learn well by being thrown into a project and being told to figure it ou
4. Teaching Testing Copyright 2001 Cem Kaner All rights reserved 145 Long Answers OOOO 18 Imagine that you were testing the feature Enters and or edits matrices Describe four examples of each of the following types of attacks that you could make on this feature and for each one explain why your example is a good attack of that kind Input constraint attacks Output constraint attacks Storage constraint attacks Computation constraint attacks Notes for you while you study Refer to Jorgensen Whittaker s paper on how to break software Don t give me two examples of what is essentially the same attack In the exam will not ask for all 16 examples but might ask for 4 examples of one type or two examples of two types etc Teaching Testing Copyright 2001 Cem Kaner All rights reserved 146 Long Answers OOO 19 Imagine that you were testing the feature Stat calculation tool Describe a scenario test that you would use to test this feature Explain why this is a particularly good scenario test Teaching Testing Copyright 2001 Cem Kaner All rights reserved 147 Long Answers OOO 20 Imagine that you were testing the feature Stat calculation tool Explain how you would develop a set of soap operas to test this feature Describe one test that might qualify as a soap opera Explain why this is a good soap opera test Teaching Testing Copyri
5. you select an area by moving the mouse The real area selected is not a perfect rectangle The rectangle just shows us where the selected area is ERNST ETERR CT gt ee Dm For Help click Help Topics on the Help Menu Teaching Testing Copyright 2001 Cem Kaner All rights reserved 157 Bug Question Continued Now we cut the selection To do this press Ctr X The jagged border shows exactly the area that was selected n BURCRRSNCRRENE CCM OT ee Dm For Help click Help Topics on the Help Menu iW Teaching Testing Copyright 2001 Cem Kaner All rights reserved 15 8 Bug Question Continued p untitled Paint si x Next exit the program File Edit View Image Options Help restart it color the background grey draw the circle select the area around the circle and drag it up and to the right This works n BORCRRSNRCRUENE PT OT ae RM For Help click Help Topics on the Help Menu iW Teaching Testing Copyright 2001 Cem Kaner All rights reserved 159 Bug Question Continued Loo o This time we ll try the Rectangular Selection tool With this one if you move the mouse to select an area the area that is actually selected is the smallest rectangle that encloses the path that your mouse drew So exit the program start it up color the background draw a circle click the Rectangular Selection tool select t
6. All rights reserved 56 Homework File Name Matrix ee You could do this for almost any type of variable For example imagine listing all of the hardware including connection power etc error conditions that could cause failure of a file save operation At this point though it is interesting to look at variables that have values that can be treated as equivalent simpler examples are string fields numeric rational numbers percentages date fields time fields Teaching Testing Copyright 2001 Cem Kaner All rights reserved 5 7 Matrix Construction Brainstorm Loo O Take up the assignment with a brainstorming session e Expect to spend 1 15hours if the students are prepared and2 hours if they are not e Brainstorming Rules Don t criticize others contributions Jokes are OK and are often valuable Goal is to get lots of ideas filter later Recorder and facilitator keep their opinions to themselves Teaching Testing Copyright 2001 Cem Kaner All rights reserved 5 8 Matrix Construction Brainstorm E i i a e Facilitating and Recording Rules Exercise patience Goal is to get lots of ideas Encourage non speakers to speak Use multiple colors when recording Echo the speaker s words Record the speaker s words The rule of three 10 s Silence is OK Switch levels of analysis Some references e S Kaner Lind Toldi Fisk a
7. these at two levels First prompt with questions to get the student to puzzle further through the problem Second provide answers that describe the ambiguities commonly spotted by students Teaching Testing Copyright 2001 Cem Kaner All rights reserved 4 Equivalence Classes A Broad Concept Loo OO The notion of equivalence class is much broader than numeric ranges Here are some examples Membership in a common group e such as employees vs non employees Note that not all classes have shared boundaries Equivalent hardware e such as compatible modems Equivalent event times e such as before timeout and after Equivalent output events e perhaps any report will do to answer a simple the question Will the program print reports Equivalent operating environments e such as French amp English versions of Windows 3 1 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 42 Variables Suited to Equivalence Class Analysis OO There are many types of variables such as m input variables m output variables m internal variables m hardware and system software configurations and m equipment states Any of these can be subject to equivalence class analysis Here are some examples Teaching Testing Copyright 2001 Cem Kaner All rights reserved 43 Variables Suited to Equivalence Class Analysis OO Eee ranges of numbers character codes how many times something
8. to test for it procedures e Should the docs ever control the testing project Teaching Testing Copyright 2001 Cem Kaner All rights reserved 104 Test Docs Requirements Questions CO If the docs control parts of the testing project should that control come early or late in the project Who are the primary readers of these test documents and how important are they e How much traceability do you need What docs are you tracing back to and who controls them e To what extent should test docs support tracking and reporting of project status and testing progress e How well should docs support delegation of work to new testers e What are your assumptions about the skills and knowledge of new testers e Is test doc set a process model a product model or a defect finder Teaching Testing Copyright 2001 Cem Kaner All rights reserved 105 Test Docs Requirements Questions Loo DD e A test suite should provide prevention detection and prediction Which is the most important for this project e How maintainable are the test docs and their test cases And how well do they ensure that test changes will follow code changes e Will the test docs help us identify and revise restructure in face of a permanent shift in the risk profile of the program e Are should docs be automatically created as a byproduct of the test automation code Teaching Testing Copyright 2001 Cem Kaner All rights re
9. Help Menu Teaching Testing Copyright 2001 Cem Kaner All rights reserved 166 Bug Question Continued Fy untitled Paint MCE Eile Edit View Image Options Help Exit the program start again color the background draw the circle Zoom to 200 and this time grow the window so you can see the whole drawing area CI aC i a eC TST ae 7 5 Fort Help click Help Topics o onthe Help Menu j ji g Mh Teaching Testing Copyright 2001 Cem Kaner All rights reserved Fie Now select the circle That seems to work Teaching Testing Bug Question Continued fa untitled Paint Copyright 2001 Cem Kaner All rights reserved Bug Question Continued F untitled Paint ioj x File Edit View Image Options Help But when you press Ctrl X to cut the circle the 3 program cuts the wrong area CRIS ISO IN IT APT OT lel Help click Help Topics on the Help Menu FE Teaching Testing Copyright 2001 Cem Kaner All rights reserved Bug Question Continued a Now write a bug report want two sections The Problem summary or title The Problem Description how to reproduce the problem Additionally please describe three follow wptests that you would run with this bug Teaching Testing Copyright 2001 Cem Kaner All rights reserved 170
10. factors and associated tests ask for 2 tests per factor Break Class brainstorm onto flipcharts list risk factors Pairs project fill out the charts e Students work in pairs to complete their chart e Each chart has one risk factor as a title e List tests derived from risk factor on body of page e When a pair is done with its page they move around the room adding notes to other pairs pages Paired testing one chart per pair test SUT per that risk factor Teaching Testing Copyright 2001 Cem Kaner All rights reserved 90 Risk Based Testing Exercises e Collect or create some test cases for the software under test Make a variety of tests Mainstream tests that use the program in safe ways Boundary tests Scenario tests Wandering walks through the program Etc If possible use tests the students have suggested previously e For each test ask How will this test find a defect What kind of defect did the test author probably have in mind What power does this test have against that kind of defect Is there a more powerful test A more powerful suite of tests Teaching Testing Copyright 2001 Cem Kaner All rights reserved 91 Risk Based Test Management O U U u u e Project risk management involves Identification of the different risks to the project issues that might cause the project to fail or to fall behind schedule or to cost too much or to di
11. illustrate an abstract concept or problem solving algorithm use at least one numerical example sensing to supplement the usual algebraic example intuitive For example when presenting Euler s method for numerical integration instead of simply giving the formulas for successive steps use the algorithm to integrate a simple function like y x and work out the first few steps on the chalkboard with a hand calculator e Use physical analogies and demonstrations to illustrate the magnitudes of calculated quantities sensing global For example tell your students to think of 100 microns is about the thickness of a sheet of paper and to think of a mole as a very large dozen molecules Have them pick up a 100 ml bottle of water and a 100 ml bottle of mercury before talking about density Teaching Testing Copyright 2001 Cem Kaner All rights reserved 9 Examples from Felder s Matter of Style OO e Occasionally give some experimental observations before presenting the general principle and have the students preferably working in groups see how far they can get toward inferring the latter inductive For example rather than giving the students Ohm s or Kirchoff s Law up front and asking them to solve for an unknown give them experimental voltage current resistance data for several circuits and let them try to figure out the laws for themselves e Provide class time for students to think about the material being presented reflecti
12. needed when you add new columns At the very end fill the remaining blank ones with arbitrary values Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 17 Combination Testing e Ifa variable is continuous but maps to a number line partition and use boundaries as the distinct values under test If all variables are continuous we end up with all pairs of all boundary tests of all variables We don t achieve all triples all quadruples etc e If some combinations are of independent interest add them to the list of n tuples to test With the six columns of the example we reduced 96 tests to 8 Give a few back make it 12 or 15 tests and you still get enormous reduction Examples of independent interest are known from tech support high risk cases cases that jointly stress memory configuration combinations Var 1 is operating systems Var 2 is printers etc that are prevalent in the market etc Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 1 8 Many Additional Key Testing Tasks ee Here are just a few more to stimulate your thinking Questioning Creating models Keeping lab notes Estimating tasks Designing tests that make failures Facilitating and recording meetings obvious such as technical reviews Status reporting Regression automation Bug advocacy Editing Usability testing Systematic exploration Predicting errors Training staff Packaging info for tech su
13. same as first 1 Same as first same 198 to 198 Are there other sources of data for this variable Ways to feed it bad data Teaching Testing Copyright 2001 Cem Kaner All rights reserved 3 1 Notes on the Traditional Presentation Sr e Inthe traditional approach we analyzed the code from the point of view of the programmer and the specification e The specification provides us with only one of several classes of risk We have to test against a broader set e Experts in different subject matters will see different opportunities for failure and different classes of cases that might reveal these failures e Thinking outside of the box posed by the explicit design is at the essence of black box testing Teaching Testing Copyright 2001 Cem Kaner All rights reserved 32 Expanding the Notion of Equivalence OO Consider these cases Are these paired tests equivalent 55 56 56 57 57 58 58 59 59 60 60 61 61 62 62 63 63 64 64 65 65 66 66 67 67 68 68 69 69 70 70 71 Nr Nr S Teaching Testing Copyright 2001 Cem Kaner All rights reserved 33 Equivalence Classes and Partitions are Subjective Eee Example The Hockey Game My working definition of equivalence Two test cases are equivalent if you expect the same result from each This is fundamentally subjective It depends on what you expect And what you expect depends on what errors you can anticipate Two test cases can only
14. Domain testing e Function testing e Risk based testing e Exploratory testing e Stress testing e Specification based testing e User testing e Scenario testing e Regression testing e Random testing e Configuration testing Teaching Testing Copyright 2001 Cem Kaner All rights reserved 17 So Let s Start the Analysis with Domain Testing Loo oO Domain Testing ne e AKA partitioning equivalence analysis boundary analysis e Fundamental question or goal This confronts the problem that there are too many test cases for anyone to run This is a stratified sampling strategy that provides a rationale for selecting a few test cases from a huge population e General approach Divide the set of possible values of a field into subsets pick values to represent each subset Typical values will be at boundaries More generally the goal is to find a best representative for each subset and to run tests with these representatives Advanced approach combine tests of several best representatives Several approaches to choosing optimal small set of combinations e Paradigmatic case s Equivalence analysis of a simple numeric field Printer compatibility testing multidimensional variable doesn t map to a simple numeric field but stratified sampling is essential Teaching Testing Copyright 2001 Cem Kaner All rights reserved 19 Domain Testing EE e Some of the Key Tasks Parti
15. Knowledge of the subject matter of the program under test will enable you to create test cases that are not directly suggested by the specification If you lack that knowledge you will miss key tests This knowledge is sometimes called domain knowledge not to be confused with domain testing Few students checked non numeric values bad delimiters or non integers The only boundaries tested were at MaxInt or 0 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 25 Notes on the Exercise Loo oO e James Bach uses a variation on this simple example to teach several lessons He s probably willing to share his demo program Contact him at james satisfice com e In Bach s example You actually enter three numbers a test case on a client computer All the clients send back student tests to the teacher s machine so James can look at trends He can ask for specific tests like try to find the largest number the program can accept Remarkably students don t test numbers anywhere near as big as they could For example if the input field window is 17 characters wide some students will type an input 17 characters long but few will go wider Teaching Testing Copyright 2001 Cem Kaner All rights reserved 26 Another Example e Here is the programs specification This program is designed to add two numbers which you will enter 2 Each number should be one or two digits
16. Teaching Testing A Skills Based Approach Loo o Cem Kaner J D Ph D Department of Computer Sciences Florida Institute of Technology May 2001 Software Quality Week Abstract Doo e Training software testers involves teaching culture vocabulary concepts and skills think that many of the commercial seminars and certification review courses teach vocabulary and many concepts quite well Some of them address cultural issues Fewer address skills but skills development is essential for new testers e I ve been training testers for 18 years sometimes personally coaching them sometimes teaching a pretty successful commercial seminar and now teaching undergraduate and graduate students Over the past year I ve rethought my approach to teaching e When studied mathematics we learned a lot of conceptual material but we also did a lot of drill exercise upon exercise from the course text and from exercise books like the Schaum s Outlines These exercises forced the student to learn how to work with the concepts and how to apply them under a wide range of circumstances e Working primarily with James Bach Helene Astier and Alan Jorgensen I ve been trying to develop a list of specific skills that testers use in the normal course of their work and then develop exercises that will practice them in those skills have several such exercises now and am developing more as go e This session will go through practice exer
17. ach field Name the type of the field integer rational string etc List the range of entries that are valid for the field Partition the field and identify boundary conditions List the entries that are almost too extreme and too extreme for the field List a small number of test cases for the field and explain why the values you have chosen are best representatives of the sets of values that they were selected from Identify any constraints imposed on this field by other fields Teaching Testing Copyright 2001 Cem Kaner All rights reserved 3d Additional Basic Exercises EE e Basic exercises like these can be presented with solutions lm working on developing a series of these Imagine giving a student 20 examples of questions like these with 20 solutions comments in the back of the book e There might not be one true solution but think we can give three one being a typical weak solution one being an average acceptable solution and one being excellent e n some cases there are alternative good solutions e In any case think the answers should include commentary Teaching Testing Copyright 2001 Cem Kaner All rights reserved 38 Specifications and Boundaries DLO OOO e In general word puzzles are important exercises Math students can often solve equations but have real trouble identifying the equations when they are embedded in simple sentences Teaching Testing Copyright 2001 Cem Ka
18. are unlikely to discover if you focus your testing on line and branch coverage 4 Give three different definitions of software error Which do you prefer Why 5 Ostrand amp Balcer described the category partition method for designing tests Their first three steps are a Analyze b Partition and c Determine constraints Describe and explain these steps Teaching Testing Copyright 2001 Cem Kaner All rights reserved 129 Long Answers OOOO 1 Imagine testing a date field The field is of the form MM DD YYYY two digit month two digit day 4 digit year Do an equivalence class analysis and identify the boundary tests that you would run in order to test the field 2 I J and K are integers The program calculates K I J For this question consider only cases in which you enter integer values into and J Do an equivalence class analysis from the point of view of the effects of and J jointly on the variable K Identify the boundary tests that you would run the values you would enter into and J if I J K are unsigned integers I J K are signed integers Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 30 Long Answers a 3 The following statement about TI is not true but pretend it is for exam purposes TI has just announced that they will include database support in Release 2 0 of the Interactive product which they will ship in November 2001 They announce that the firs
19. be equivalent by reference to a specifiable risk Two different testers will have different theories about how programs can fail and therefore they will come up with different classes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 34 Another Example SE Character ASCII Code 47 lower bound 0 1 2 3 4 5 53 6 7 8 upper bound 9 A 65 a Teaching Testing Copyright 2001 Cem Kaner All rights reserved Revised Boundary Analysis Table Variable Equivalence Alternate Boundaries Class Equivalence and Special Class 99 to 99 gt 99 lt 99 99 100 digits non digits 0 9 leading spaces or 0s ressions null entry Second same as first same as first same number 198 to 198 277 277 Are there other 127 to 127 198 to 128 127 128 127 AE u Hi 128 to 198 128 this varia e Ways to feed it bad data Note that we ve dropped the issue of valid and invalid This lets us generalize to partitioning strategies that don t have the concept of valid for example printer equivalence classes For discussion of device compatibility testing see Kaner et al Chapter 8 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 36 Additional Basic Exercises Loo SS e Send students to common dialog boxes such as these in MS Word Print dialog Page setup dialog Font format dialog e For each dialog Identify each field and for e
20. cises in bug reporting domain testing combination testing all pairs specification analysis for ambiguity specification analysis for holes and possibly some other areas The more time we have the more techniques we look at In this session I ll share course notes quizzes exam questions and explain how use them Teaching Testing Copyright 2001 Cem Kaner All rights reserved Scope e This workshop is focused on classroom style teaching though much of it will apply to programs involving self paced self study e I ve been involved in classroom style teaching As a senior tester conducting classes for my staff in house Tester College As an instructor on the road teaching 3 day seminars in hotels Asa University of California Extension instructor teaching courses to working professionals on nights and weekends covering my 3 day seminar in some courses and test automation architecture in others As atraveling test teacher delivering courses onsite at several companies As a university professor teaching senior undergrads and various graduate students Teaching Testing Copyright 2001 Cem Kaner All rights reserved Work In Progress Loo O e These slides are far from complete It feels to me as though I am at the beginning of skills analysis for software testing tasks and techniques not the end These slides reflect that early stage They are inconsistent in depth The exe
21. cs course First describe problems associated with the design of buildings and bridges and artificial limbs and perhaps give the students some of those problems and see how far they can go before they get all the tools for solving them e Balance conceptual information intuitive with concrete information sensing Intuitors favor conceptual information theories mathematical models and material that emphasizes fundamental understanding Sensors prefer concrete information such as descriptions of physical phenomena results from real and simulated experiments demonstrations and problem solving algorithms For example when covering concepts of vapor liquid equilibria explain Raoult s and Henry s Law calculations and nonideal solution behavior but also explain how these concepts relate to barometric pressure and the manufacture of carbonated beverages Teaching Testing Copyright 2001 Cem Kaner All rights reserved 8 Examples from Felder s Matter of Style OO e Make extensive use of sketches plots schematics vector diagrams computer graphics and physical demonstrations visual in addition to oral and written explanations and derivations verbal in lectures and readings For example show flow charts of the reaction and transport processes that occur in particle accelerators test tubes and biological cells before presenting the relevant theories and sketch or demonstrate the experiments used to validate the theories e To
22. der test e Use each function in a mainstream way positive testing and push it in as many ways as possible as hard as ossible j 73 Teaching Testing Copyright 2001 Cem Kaner All rights reserved Risk Based Testing and Risk Based Test Management Loo oO Risk Based Testing O u u e Karl Popper in his famous essay Conjectures and Refutations lays out the proposition that a scientific theory gains credibility by being subjected to and passing harsh tests that are intended to refute the theory e We can gain confidence in a program by testing it harshly if it passes the tests e Subjecting a program to easy tests doesn t tell us much about what will happen to the program in the field In risk based testing we create harsh tests for vulnerable areas of the program Teaching Testing Copyright 2001 Cem Kaner All rights reserved 75 Risk Based Testing OT e Two key dimensions Find errors risk based approach to the technical tasks of testing Manage the process of finding errors risk based test management e Let s start with risk kased testing and proceed later to risk kased test management Teaching Testing Copyright 2001 Cem Kaner All rights reserved 76 Risk Based Testing Definitions Hazard dangerous condition something that could trigger an accident Risk Possibility of suffering loss or harm probability of an accident caused by a given hazard Accident
23. e ask which values of K are interesting partition number 1 and then which values of and J would produce those K values and do a partitioning on the sets of I J pairs to find best representatives of those e As practitioners we do this type of analysis often but many of us probably don t think very formally about it Teaching Testing Copyright 2001 Cem Kaner All rights reserved 49 Fuzzy Boundaries e In theory the key to partitioning is dividing the space into mutually exclusive ranges Subsets Each subset is an equivalence class This is nice in theory but let s look at printers e Problem There are about 2000 Windows compatible printers plus multiple drivers for each We can t test them all e But maybe we can form equivalence classes e An example from two programs desktop publishing and an address book developed in 1991 92 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 50 Fuzzy Boundaries e Primary groups of printers at that time HP Original HP LJII PostScript Levell PostScript Level II Epson 9 pin etc e LaserJet Il compatible printers huge class maybe 300 printers depending on how we define it 1 Should the class include LJII LJII and LIIP LJIID compatible subclasses 2 What is the best representative of the class Teaching Testing Copyright 2001 Cem Kaner All rights reserved 5I Fuzzy Boundaries Loo oO e Exam
24. ect Teaching Testing Copyright 2001 Cem Kaner All rights reserved 142 Long Answers 15 Why is it important to design maintainability into automated regression tests Describe some design of the test code choices that will usually make automated regression tests more maintainable Teaching Testing Copyright 2001 Cem Kaner All rights reserved 143 Long Answers OOOO 16 Suppose that you find a reproducible failure that doesn t look very serious Describe three tactics for testing whether the defect is more serious than it first appeared Asa particular example suppose that the display got a little corrupted stray dots on the screen an unexpected font change that kind of stuff in the TI program when you entered data into a matrix Describe some follow up tests that you would run Teaching Testing Copyright 2001 Cem Kaner All rights reserved 144 Long Answers OOOO 17 The oracle problem is the problem of finding a method that lets you determine whether a program passed or failed a test Suppose that you were doing automated testing of the TI Interactive function Graphs functions and plots statistical data Describe three different oracles that you could use or create to determine whether this feature was working For each of these oracles identify a bug that would be easy to detect using the oracle and identify another bug that your oracle would be more likely to miss
25. ed 22 An Introductory Exercise O u u The program reads three integer values from a card The three values are interpreted as representing the lengths of the sides of a triangle The program prints a message that states whether the triangle is scalene isosceles or equilateral e From Glen Myers The Art of Software Testing 1 Write a set of test cases that would adequately test this program 2 Imagine testing this program completely What proportion of complete testing does your set of test case represent 3 Hand it in when you are done Teaching Testing Copyright 2001 Cem Kaner All rights reserved 23 Myers Answer Loo o Test case for a valid scalene triangle Test case for a valid equilateral triangle Three test cases for valid isosceles triangles a b b c a C One two or three sides has zero value 5 cases One side has a negative Sum of two numbers equals the third e g 1 2 3 is invalid b c not a triangle tried with 3 permutations a b c a c b b c a Sum of two numbers is less than the third e g 1 2 4 3 permutations Non integer Wrong number of values too many too few Teaching Testing Copyright 2001 Cem Kaner All rights reserved 24 Notes on the Exercise C_CeuN e Several classes of issues were missed by most students For example Few students checked whether they were producing valid triangles 1 2 3 and 1 2 4 cannot be the lengths of any triangle e
26. enario test and describe the characteristics of a good scenario test Imagine developing a set of scenario tests for the TI interactive product that were focused on configuration related issues compatibility of hardware software with the product What research would you do in order to develop a series of scenario tests Describe two scenario tests that you would use and Explain why each is a good example of a scenario test Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 36 Long Answers OOOO 9 Define a scenario test and describe the characteristics of a good scenario test Imagine developing a set of scenario tests for a C compiler What research would you do in order to develop a series of scenario tests NOTE I am not asking for tests of the user interface to the compiler I m asking for tests of what we typically think of as compiler functionality Describe two scenario tests that you would use and explain why each is a good test Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 37 Long Answers OO 10 What is regression testing What are some benefits and some risks associated with regression testing Under what circumstances would you use regression tests Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 3 8 TI InterActive toolbar Teaching Testing Long Answers a 11 Consider the tools in TI interactive For each tool identif
27. er the program will satisfy a given quality criterion Write down these issues These are primary material for asking the programmer or product manager about the spec Teaching Testing Copyright 2001 Cem Kaner All rights reserved 108 Combination Testing Loo o These notes are reminders see demonstration in class Combination Testing rr e Imagine a program with 3 variables V1 has 3 possible values V2 has 2 possible values and V3 has 3 possible values e If V1 and V2 and V3 are independent the number of possible combinations is 12 3 x 2 x 2 e Building a simple combination table Label the columns with the variable names listing variables in descending order of number of possible values Each column before the last will have repetition Suppose that A B and C are in column K of N columns To determine how many times rows in which to repeat A before creating a row for B multiply the number of variable values in columns K 1 K 2 N Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 10 Combination Testing OT e Building an alt airs combination table Label the columns with the variable names listing variables in descending order of number of possible values If the variable in column 1 has V1 possible values and the variable in column 2 has V2 possible values then there will be at least VI x V2 rows draw the table this way but leave a blank row or two betwee
28. er to find using a diagnostics based strategy than by using a lower volume strategy such as exploratory testing spec based testing or domain testing Teaching Testing Copyright 2001 Cem Kaner All rights reserved 140 Long Answers oo 13 You are using a high ume random testing strategy for the TI Interactive calculator and will evaluate results by using an oracle For the functional area Generates a table of values for defined functions how would you create an oracle or group of oracles What would the oracle s do For the functional area Graphs functions and plots statistical data how would you create an oracle or group of oracles What would the oracle s do Which oracle would be more challenging to create or use and why Teaching Testing Copyright 2001 Cem Kaner All rights reserved 141 Long Answers Loo 14 Consider testing the TI Interactive function Transfers data to or from a connected calculator How would you develop a list of risks for this capability If you are talking to people who would you ask and what would you ask them If you are consulting books or records or databases what are you consulting and what information are you looking for in it Why is this a good approach for building a list of risks List 10 risks associated with this function For each risk briefly very briefly describe a test that could determine whether there was an actual def
29. from Windows Paint 95 Please don t spend your time replicating the steps or the bug You re welcome to do so if you are curious but I will design my marking scheme to not give extra credit for that extra work Treat the steps that follow as fully reproducible If you go back to ANY step you can reproduce it For those of you who aren t familiar with paint programs the essential idea is that you lay down dots For example when you draw a circle the result is a set of dots not an object If you were using a draw program you could draw the circle and then later select the circle move it cut it etc In a paint program you cannot select the circle once you ve drawn it You can select an area that includes the dots that make up the circle but that area is simply a bitmap and none of the dots in it have any relationship to any of the others Teaching Testing Copyright 2001 Cem Kaner All rights reserved 153 Bug Question Continued O u u Here s the opening screen The F untitled Paint _ lol x background iS Bee View Image Options Help white The first zat faces thing that we ll do is select the Paint Can Je We ll use this to lay down a layer of grey paint on top of the background Then when we cut Ol MONE a Ate a 7 SESSREee we ll see the white an non nant backg round behind Fills an area with the current drawing color 4 what was moved Teaching Testing Copyright 2001 Cem Kaner All
30. ght 2001 Cem Kaner All rights reserved 148 Long Answers OOOO 21 Imagine that you were testing the feature Graphs functions and plots statistical data Describe four examples of each of the following types of attacks that you could make on this feature and for each one explain why your example is a good attack of that kind Input constraint attacks Output constraint attacks Storage constraint attacks Computation constraint attacks Notes for you while you study Refer to Jorgensen Whittaker s paper on how to break software Don t give me two examples of what is essentially the same attack In the exam will not ask for all 16 examples but might ask for 4 examples of one type or two examples of two types etc Teaching Testing Copyright 2001 Cem Kaner All rights reserved 149 Long Answers OOOO 22 Imagine that you were testing the feature Transfers data to or from a connected calculator Describe four examples of each of the following types of attacks that you could make on this feature and for each one explain why your example is a good attack of that kind Input constraint attacks Output constraint attacks Storage constraint attacks Computation constraint attacks Notes for you while you study Refer to Jorgensen Whittaker s paper on how to break software Don t give me two examples of what is essentially the same attack In the exam will not as
31. hazard is encountered resulting in loss or harm Teaching Testing Copyright 2001 Cem Kaner All rights reserved 17 Risks Where to look for errors Quality Categories Capability Reliability Each quality category is a Usability risk category as in Performance the risk of unreliability Installability Compatibility Efficiency Supportability Localizability Testability Portability Maintainability Derived from James Bach s Satisfice Model Teaching Testing Copyright 2001 Cem Kaner All rights reserved 78 Risks Where to look for errors e New things newer features may fail e New technology new concepts lead to new mistakes e Learning Curve mistakes due to ignorance e Changed things changes may break old code e Late change rushed decisions rushed or demoralized staff lead to mistakes e Rushed work some tasks or projects are chronically underfunded and all aspects of work quality suffer e Tired programmers long overtime over several weeks or months yields inefficiencies and errors e Adapted from James Bach s lecture notes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 79 Risks Where to look for errors e Other staff issues alcoholic mother died two programmers who won t talk to each other neither will their code e Just slipping it in pet feature not on plan may interact badly with other code N I H external componen
32. he area around the circle and move it up It works et CT CT ee Dm Selects a rectangular part of the picture to move copy iW Teaching Testing Copyright 2001 Cem Kaner All rights reserved 160 Bug Question Continued D u u u Well this was just too boring because everything is working When you don t find a bug while testing a feature one tactic is to keep testing the feature but combine it with some other test In this case we ll try Zooming the image When you zoom 200 the picture itself doesn t change size but the display doubles in size Every dot is displayed as twice as tall and twice as wide So we exit the program start it up color the background grey draw the circle and then Teaching Testing Copyright 2001 Cem Kaner All rights reserved 161 Bug Question Continued Bring up the Custom oom Px Custom Zoom Current zoom 100 0a 0g and select Zoom to cancel 200 Vo ZOOM click Cc 100 400 C 800 OK C piis Teaching Testing Copyright 2001 Cem Kaner All rights reserved 162 Bug Question Continued lt worked The paint area is displayed twice as tall Fle Eat SERRE bE a and twice as wide We re looking at the bottom right corner To see the rest we could move the scroll bars up or left H ECS ee ele to TOM OI MC MR For Help click Help Topics on the Help Menu Teaching Testing Copyright 2001 Cem Kaner All righ
33. hich order races e length of time after a timeout from JUST before to way after what events are important e speed of data entry time between keystrokes menus etc e speed of input handling of concurrent events e number of devices connected active Teaching Testing Copyright 2001 Cem Kaner All rights reserved 45 Variables Suited to Equivalence Class Analysis system resources consumed available also handles stack space etc date and time transitions between algorithms optimizations different ways to compute a function most recent event first event input or output intensity voltage speed extent of voltage transition e g from very soft to very loud sound Teaching Testing Copyright 2001 Cem Kaner All rights reserved 46 Constraints Interactions Among Variables EE Rather than thinking about a single variable with a single range of values a variable might have different ranges such as the day ofthe month in a date 1 28 1 29 1 30 1 31 We analyze the range of dates by partitioning the month field for the date into different sets February April June September November Jan March May July August October December For testing you want to pick one of each There might or might not be a boundary on months The boundaries on the days are sometimes 1 28 sometimes 1 29 etc This is nicely analyzed by Jorgensen Software Testing A Craftsman s Ap
34. how much they can learn with personal involvement and how little they get from traditional lectures Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 3 Where are you training O u u e Different constraints for Undergraduates Working professionals in a hotel room Working professionals in a short course onsite Working professionals at a many week extension course Working professionals in a long term course onsite On line training One on One coaching Teaching Testing Copyright 2001 Cem Kaner All rights reserved 14 Example of a Task Breakdown e Bug Reporting Reproduce the bug Simplify the bug Follow up testing Describe the screen Describe the sequence Determine customer impact Determine technical impact What sub tasks and skills are involved in each of these Teaching Testing Copyright 2001 Cem Kaner All rights reserved 15 Bug Reporting Practice Example e You will be given a copy of a computer screen Please write a description all words no pictures of that screen When asked please pass your description to your partner You will receive a description of a different screen from your partner Please draw the screen that your partner is describing Teaching Testing Copyright 2001 Cem Kaner All rights reserved 16 Basic Testing Styles What tasks and skills are involved in each of these e
35. ices Graphs functions and plots Generates a table of values Copyright 2001 Cem Kaner All rights reserved Performs typical spreadsheet operations Performs statistics regression calculations on lists of data Transfers data to from a connected calculator Captures the screen of a connected calculator TI 83 or TI 83 Plus Browses the Web and extracts data directly from Web pages Sends e mail attachments of your current document 127 Definitions Loo O 1 Domain testing as defined by Clarke et al 2 Equivalence class 3 Boundary condition 4 Best representative 5 Fault vs failure vs defect 6 Function testing 7 Regression testing 8 Specification based testing 9 Finite state machine 10 Stochastic testing 11 Oracle 12 Risk factors as described by Amland Teaching Testing 13 14 15 16 17 18 19 20 21 22 23 Copyright 2001 Cem Kaner All rights reserved Exploratory testing Dumb monkey State Line or statement coverage Boundary chart Software quality Black box testing Glass box white box testing Risk based testing All pairs combination testing Combination testing 128 Short Answers Doo 1 Describe the characteristics of a good scenario test 2 Describe two difficulties and two advantages of state machine model based testing 3 Give two examples of defects you are likely to discover and five examples of defects that you
36. ing Tasks Teaching Testing Copyright 2001 Cem Kaner All rights reserved 63 Function Testing Tasks Teaching Testing Copyright 2001 Cem Kaner All rights reserved 64 Function Testing Tasks Teaching Testing Copyright 2001 Cem Kaner All rights reserved 65 Function Testing Skills Teaching Testing Copyright 2001 Cem Kaner All rights reserved 66 Function Testing Skills Teaching Testing Copyright 2001 Cem Kaner All rights reserved 67 Function Testing Skills Teaching Testing Copyright 2001 Cem Kaner All rights reserved 68 Function Testing Exercises OOOO Teaching Testing Copyright 2001 Cem Kaner All rights reserved 69 Function Testing Exercises OOOO Teaching Testing Copyright 2001 Cem Kaner All rights reserved 70 Function Testing Exercises OOOO Teaching Testing Copyright 2001 Cem Kaner All rights reserved 71 Function Testing Exercises OOOO Teaching Testing Copyright 2001 Cem Kaner All rights reserved 12 Some Function Testing Tasks Loo o e Identify the programs features commands From specifications or the draft user manual From walking through the user interface From trying commands at the command line From searching the program or resource files for command names e Identify variables used by the functions and test their boundaries e Identify environmental variables that may constrain the function un
37. is done e g shareware limit on number of uses of a product e g how many times you can do it before you run out of memory how many names in a mailing list records in a database variables in a spreadsheet bookmarks abbreviations size of the sum of variables or of some other computed value think binary and think digits Teaching Testing size of a number that you enter number of digits or size of a character string size of a concatenated string size of a path specification size of a file name size in characters of a document size of a file note special values such as exactly 64K exactly 512 bytes etc size of the document on the page compared to page margins across different page margins page sizes Copyright 2001 Cem Kaner All rights reserved 44 Variables Suited to Equivalence Class Analysis ____ OOO size of a document on a page in terms of memory requirements for the page This may be in terms of resolution x page size but is more complex if we have compression equivalent output events such as printing documents amount of available memory gt 128 meg gt 640K etc visual resolution size of screen number of colors operating system version variations within a group of compatible printers sound cards modems etc equivalent event times when something happens timing how long between event A and event B and in w
38. ity analysis e Another book provides an excellent introduction to the ways in which statements can be ambiguous and provides lots of sample exercises Cecile Cyrul Spector Saying One Thing Meaning Another Activities for Clarifying Ambiguous Language Teaching Testing Copyright 2001 Cem Kaner All rights reserved 102 Example Requirements Questions O e Important to trace from requirements to implications e Exercise Give a list of questions e Examples are the test documentation requirements questions next slides and the automation maintainability questions at www kaner com pdfs shelfwar padf For each question e Ask the students to name at least two decisions that they would make on the basis of the answer to the question Make this a small group exercise splitting up the questions groups fill flipcharts then bring back to full class Teaching Testing Copyright 2001 Cem Kaner All rights reserved 103 Test Docs Requirements Questions O e Is test documentation a product or tool Is software quality driven by legal issues or by market forces How quickly is the design changing How quickly does the specification change to reflect design change Is testing approach oriented toward proving conformance to specs or nonconformance with customer expectations e Does your testing style rely more on already defined tests or on exploration e Should test docs focus on what to test objectives or on how
39. k Workshop 1998 TeraQuest Risk Based Test Management A e Tasks List all areas of the program that could require testing On a scale of 1 5 assign a probability of failure estimate to each On a scale of 1 5 assign a severity of failure estimate to each For each area identify the specific ways that the program might fail and assign probability of failure and severity of failure estimates for those Prioritize based on estimated risk Develop a stop loss strategy for testing untested or lightly tested areas to check whether there is easy to find evidence that the areas estimated as low risk are not actually low risk Teaching Testing Copyright 2001 Cem Kaner All rights reserved 95 Risk Based Test Management LO OMM e Skills e Exercises Teaching Testing Copyright 2001 Cem Kaner All rights reserved 96 Specification Driven Testing Loo oO Specification Driven Testing OOOO e Tag line Verify every claim e Fundamental question or goal Check the product s conformance with every statement in every spec requirements document etc e Paradigmatic case s Traceability matrix tracks test cases associated with each specification item User documentation testing Teaching Testing Copyright 2001 Cem Kaner All rights reserved 98 Specification Driven Testing ne Strengths Critical defense against warranty claims fraud charges loss of credibility with customers
40. k for all 16 examples but might ask for 4 examples of one type or two examples of two types etc Teaching Testing Copyright 2001 Cem Kaner All rights reserved 15 0 Long Answers Loo 23 We are going to do some configuration testing on the TI Interactive product We want to test it on Windows 95 98 and 2000 the latest service pack level of each Printing to an HP inkjet a LexMark inkjet and a Xerox laser printer Connected to the web with a dial up modem 28k a DSL modem and a cable modem With a 640x480 display and a 1024x768 display e How many combinations are there of these variables e Explain what an all pairs combinations table is e Create an all pairs combinations table e Explain why you think this table is correct Teaching Testing Copyright 2001 Cem Kaner All rights reserved 15 1 Long Answers OOOO 24 Imagine that you are an external test lab and the TI company comes to you with TI interactive They want you to test the product How will you decide what test documentation to give them Suppose that when you ask them what test documentation they want they say that they want something appropriate but they are relying on your expertise What questions would you ask up to 7 questions and how would the answers to those questions guide you in deciding what to do Teaching Testing Copyright 2001 Cem Kaner All rights reserved 152 Long Answers OOOO 25 The following group of slides are
41. lence classes boundary cases and expected results for a long distance telephone service that bills calls at a flat rate of 0 05 per minute Assume that the chargeable time of a call begins when the called party answers and ends when the calling party disconnects Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 33 Long Answers Loo o 6 Describe a traceability matrix How would you build a traceability matrix for the TI Interactive product What is the traceability matrix used for What are the advantages and risks associated with driving your testing using a traceability matrix Give examples of advantages and risks that you would expect to deal with if you used a traceability matrix for the TI Interactive product Answer this in terms of two of the main features of the TI Interactive product You can choose which two features Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 34 Long Answers a 7 Define a scenario test and describe the characteristics of a good scenario test Imagine developing a set of scenario tests for the TI interactive product that involved matrix handling What research would you do in order to develop a series of scenario tests Describe two scenario tests that you would use and Explain why each is a good example of a scenario test 135 Teaching Testing Copyright 2001 Cem Kaner All rights reserved Long Answers a 8 Define a sc
42. mp Berger Facilitator s Guide to Participatory Decision Making e Freedman amp Weinberg Handbook of Walkthroughs Inspections amp Technical Reviews e Doyle amp Straus How to Make Meetings Work Teaching Testing Copyright 2001 Cem Kaner All rights reserved 59 Useful Readings The Impossibility of Complete Testing at www kaner com imposs htm Clarke Hassell amp Richardson A Close Look at Domain Testing IEEE Transactions on Software Engineering Vol SE 8 No 4 July 1982 A clear presentation of the formal assumptions Ostrand amp Balcer The category partition method for specifying and generating functional tests Communications of the ACM Vol 31 6 pages 676 686 June 1988 An excellent description of the method laying it out for the reader step by step Teaching Testing Copyright 2001 Cem Kaner All rights reserved 60 Analysis of Function Testing Loo oO Let s work on this one together Function Testing ne e Tag line Black box unit testing e Fundamental question or goal Test each function thoroughly one at a time e Paradigmatic case s Spreadsheet test each item in isolation Database test each report in isolation Strengths Thorough analysis of each item tested e Blind spots Misses interactions misses exploration of the benefits offered by the program Teaching Testing Copyright 2001 Cem Kaner All rights reserved 62 Function Test
43. n 2 then all pairs of column 4 and column 3 e Watch this first attempt on column 5 We achieve all pairs of GH with columns 1 2 and 3 but miss it for column 4 e The most recent arbitrary choice was HG in the 2nd section Once that was determined we picked HG for the third in order to pair H with a 1 in the third column e So we will erase the last choice and try again Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 14 Combination Testing e We flipped the last arbitrary choice column 5 section 2 to GH from HG and erased section 3 We then fill in section 3 by checking for missing pairs GH GH gives us two XG XG pairs so we flip to HP for the third section and have acolumn 2 X with acolumn 5H anda column 2 Y with a column 5 G as needed to obtain all pairs Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 15 Combination Testing But when we add the next column we see that we just can t achieve all pairs with 6 values The first one works up to column 4 but then fails to get pair EJ or Fl The next fails on GJ HI Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 16 Combination Testing e When all else fails add rows We need one AUX i He teh for GJ and one for HI so add two rows In general we would need as many rows as the last column has values e The other values in the two rows are arbitrary leave them blank and fill them in as
44. n repetition groups in column 1 Fill in the table one column at a time The first column repeats each of its elements V2 times skips a line and then starts the repetition of the next element For example if variable 1 s possible values are A B C and V2 is 2 then column 1 would contain A A blank row B B blank row C C blank row Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 1 1 Combination Testing OT e Building an alt airs combination table In the second column list all the values of the variable skip the line list the values etc For example if variable 2 s possible values are X Y then the table looks like this so far Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 12 Combination Testing e Building an alt airs combination table Each section of the third column think of AA as defining a section BB as defining another etc will have to contain every value of variable 3 Order the values such that the variables also make all pairs with variable 2 Suppose variable 2 can be 1 0 The third section can be filled in either way and you might highlight it so that you can reverse it later The decision say 1 0 is arbitrary Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 13 Combination Testing e The 4th column went in easily note that we started by making sure we hit all pairs of values of column 4 and colum
45. n specify multiple tests to be done on one class of object and you expect to test several such objects you can put the multiple tests on the matrix e Mark a cell green if you ran the test and the program passed it e Mark the cell red if the program failed and write the bug number of the bug report for this bug e Write in the cell the automation number or identifier or file name if the test case has been automated Teaching Testing Copyright 2001 Cem Kaner All rights reserved 54 Reusable Test Matrix Numeric Integer Input Field number of digits or Empty field clear the default value digits or chars Outside of UB e im Q z c aa mD LB of value 1 LB number of digits or chars gt O aa Negative LT sonate gt i m i gt en eu NN i S This includes only the first few columns of a matrix that I ve used commercially but it gets across the idea Teaching Testing Copyright 2001 Cem Kaner All rights reserved 55 Homework File Name Matrix DLO OO e Thursday we ll illustrate the process of creating test matrices by brainstorming a specific one File name field Windows 95 98 NT 2000 Treat the path as a separate issue e Please spend 15 minutes at home writing a list of file name tests Bring your notes with you on Thursday Teaching Testing Copyright 2001 Cem Kaner
46. ner All rights reserved 39 Specifications and Boundaries For each of the following list The variable s of interest The valid and invalid classes The boundary value test cases 1 FoodVan delivers groceries to customers who order food over the Net To decide whether to buy more more vans FV tracks the number of customers who call for a van A clerk enters the number of calls into a database each day Based on previous experience the database is set to challenge ask Are you sure any number greater than 400 calls 2 FoodVan schedules drivers one day in advance To be eligible for an assignment a driver must have special permission or she must have driven within 30 days of the shift she will be assigned to Teaching Testing Copyright 2001 Cem Kaner All rights reserved 40 Ambiguity in Specifications O u u e Most people have trouble with Question 2 on the previous slide For example what does within mean If today is May 29 what is the longest ago date that would qualify them to drive What if someone has permission to drive for their first time tomorrow morning can they be scheduled for a second shift tomorrow evening e For more on ambiguity analysis see the discussion of spec based testing later e Self study exercises can work with these but the problem is discovering what ambiguities are troublesome to the students and thinking about how to prompt them think that would deal with
47. og of errors surviving across versions Weak testing tools if tools don t exist to help identify isolate a class of error e g wild pointers the error is more likely to survive to testing and beyond e Adapted from James Bach s lecture notes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 82 Risks Where to look for errors e Unfixability risk of not being able to fix a bug e Language typical errors such as wild pointers in C See Bruce Webster Pitfalls of Object Oriented Development Michael Daconta et al Java Pitfalls e Criticality severity of failure of very important features e Popularity likelihood or consequence if much used features fail e Market severity of failure of key differentiating features e Bad publicity a bug may appear in PC Week e Liability being sued e Adapted from James Bach s lecture notes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 83 Bug Patterns as a Source of Risk ee Testing Computer Software lays out a set of 480 common defects You can use these or develop your own list Find a defect in the list Ask whether the software under test could have this defect fit is theoretically possible that the program could have the defect ask how you could find the bug if it was there Ask how plausible it is that this bug could be in the program and how serious the failure would be if it was there If approp
48. omobiles Discuss where the energy comes from and where it goes in each of these processes and how cell metabolism differs Then consider the photosynthetic origins of the energy stored in C H bonds and the conditions under which the earth s supply of usable energy might run out Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 1 Blooms Taxonomy Slides e Bloom s Taxonomy created in 1956 by Dr Benjamin Bloom of the University of Chicago and his group of educational psychologists is a categorization of verbs describing cognitive Skills verbs into six classes knowledge comprehension application analysis synthesis and evaluation The classes are ranked from least complex knowledge to most complex evaluation in terms of the level of thinking required for students to achieve these objectives In general critical thinking skills encompass only the three most complex categories analysis synthesis and evaluation e From Michael Bowen s web page htip 207 233 105 16 curriculum bloomtax htm to be distributed in class Teaching Testing Copyright 2001 Cem Kaner All rights reserved 12 Testing Training ns e _ tis easy to teach to the first few levels tests would have students recall what was taught define terms make simple applications e It is much tougher to teach the underlying skills of testing e Putting students through tests exercises and exams has been eye ening in terms of
49. ple graphic complexity error handling HP II original was the weak case e Example special forms HP I original was strong in paper handling We worked with printers that were weaker in paper handling e Examples of additional queries for almost equivalent printers Same margins offsets on new printer as on HP II original Same printable area Same handling of hairlines Postscript printers differ What exercises can we do to support development of this type of analysis Teaching Testing Copyright 2001 Cem Kaner All rights reserved 52 Test Matrices and Domain Testing ee e Use a test matrix to show track a series of test cases that are essentially the same For example for most input fields you Il do a series of the same tests checking how the field handles boundaries unexpected characters function keys etc As another example for most files you ll run essentially the same tests on file handling e The matrix is a concise way of showing the repeating tests Put the objects that you re testing on the rows Show the tests on the columns Check off the tests that you actually completed in the cells Teaching Testing Copyright 2001 Cem Kaner All rights reserved 53 Typical Uses of Test Matrices Loo o e You can often re use a matrix like this across products and projects e You can create matrices like this for a wide range of problems Whenever you ca
50. pport Software architecture just try doing a big test automation without doing good architecture then try again Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 19 Many Additional Key Skills or Attributes OO e Here are some examples of skills or attributes that look for when recruiting testers obviously can t have all of them in any one person e We can improve some of these e In other cases we might not know how to train for improvement but we might discover that a specific tester has other strengths rather than this one Teaching Testing Copyright 2001 Cem Kaner All rights reserved 120 Some characteristics of great testers Attentive to detail Analytical problem solver Architect Arrogance usually less is Artistic knowledgeably better critique esthetic issues Commitment keep Commitment to a task do Commitment to quality promises stick around what it takes Copes with difficult Courageous Creative circumstances Credible Curious inquisitive Customer focused Decision maker good Decisive Not very defensive able judgment to take criticism Diplomatic Editor criticize improve Effective with junior printed materials testers Effective with senior Effective with test Effective with testers managers programmers Teaching Testing Copyright 2001 Cem Kaner All rights reserved 121 Some characteristics of great testers OOO Effective with non testing
51. proach It works well as a puzzle for students alone or in groups Teaching Testing Copyright 2001 Cem Kaner All rights reserved 47 Another example of interaction e Interaction thinking is important when we think of an output variable whose value is based on some input variables Here s an example that gives my students headaches on tests J and K are integers The program calculates K I J For this question consider only cases in which you enter integer values into and J Do an equivalence class analysis from the point of view of the effects of and J jointly on the variable K Identify the boundary tests that you would run the values you would enter into and J if I J K are unsigned integers I J K are signed integers Teaching Testing Copyright 2001 Cem Kaner All rights reserved 48 Interaction example Loo OO e K can run from Minint smallest integer to Maxlnt e For any value of K there is a set of values of and J that will yield K The set is the null set for impossible values of K The set might include only two pairs such as K Maxlnt when Maxlnt is prime 7 could be a MaxInt e IJ 1 Maxint Maxint 1 The set might include a huge number of pairs such as when K is 0 e I J 0 0 1 0 2 0 Maxint 0 0 1 0 MaxInt 1 e A set of pairs of values of and J can be thought of as an equivalence set they all yield the same value of K and so w
52. rcises are of varying quality The range of exercises per topic varies widely e expect to work on this for several years Your comments criticisms and suggestions would be very much appreciated Please write me at kaner kaner com Teaching Testing Copyright 2001 Cem Kaner All rights reserved Learning Styles e People learn in different ways There are several models of learning styles Different presentation methods work best for different styles of learners e m not up to date on this literature Back when was a grad student in psychology we talked about Visual vs auditory learners Visual learners have more success in lectures if they have a copy of every slide Active vs passive learners Some people learn better in lecture or by reading memorizing Others learn better through exercises Individual vs group learners Some people learn better on their own others on project teams Concrete vs conceptual learners Some people learn better from detailed stories others from the formal presentation of the underlying principles Teaching Testing Copyright 2001 Cem Kaner All rights reserved Learning Styles Loo oO e m still thinking about how to develop a course that works well with the different styles I m especially interested in this for building on line courses that are worth teaching taking e For now my approach involves Lecture of key material Hard copies of every
53. riate design a test or series of tests for bugs of this type Teaching Testing Copyright 2001 Cem Kaner All rights reserved 84 Build Your Own Model of Bug Patterns EE Too many people start and end with the TCS bug list It is outdated lt was outdated the day it was published And it doesn t cover the issues in your system Building a bug list is an ongoing process that constantly pays for itself Here s an example from Hung Nguyen This problem came up in a client server system The system sends the client a list of names to allow verification that a name the client enters is not new Client 1 and 2 both want to enter a name and client 1 and 2 both use the same new name Both instances of the name are new relative to their local compare list and therefore they are accepted and we now have two instances of the same name As we see these we develop a library of issues The discovery method is exploratory requires sophistication with the underlying technology Capture winning themes for testing in charts or in scripts on their way to being automated Teaching Testing Copyright 2001 Cem Kaner All rights reserved 85 Building Bug Patterns OOO e There are plenty of sources to check for common failures in the common platforms www bugnet com www cnet com links from www winfiles com various mailing lists Teaching Testing Copyright 2001 Cem Kaner All rights reserved 86 Risk Ba
54. rights reserved 154 Bug Question Continued O U u Here s the screen again but the F3 untitled Paint NR background has been File Edit View Image Options Help painted gray z The star in the upper left corner is a freehand selection tool After you click on it you can trace around any part of the picture The tracing selects that Pa pers me OT Then you can cut it OP OT MR CT MR copy It move It etc Selects a free form part of the picture to move copy or iW Teaching Testing Copyright 2001 Cem Kaner All rights reserved 155 Bug Question Continued This shows an area selected with the freehand Av wal OR EEE AE selection tool The bottom El a right corner is selected The dashed line surrounds the selected area NOTE The actual area selected might not be perfectly rectangular The freehand tool shows a rectangle that is just big enough to enclose the pe eee selected area For our et ele lei el purposes this is not a DC mon naine D bug This IS a design For Help click Help Topics on the Help Menu decision by Microsoft Teaching Testing Copyright 2001 Cem Kaner All rights reserved 156 Bug Question Continued Next we ll draw a circle F untitled Paint SO you can see what s selected then use the freehand select tool to select the area around it File Edit View Image Options Help When you use the freehand selection tool
55. sed Testing Loo e Tasks Identify risk factors hazards ways in which the program could go wrong For each risk factor create tests that have power against it Assess coverage of the testing effort program given a set of risk based tests Find holes in the testing effort Build lists of bug histories configuration problems tech support requests and obvious customer confusions Evaluate a series of tests to determine what risk they are testing for and whether more powerful variants can be created Teaching Testing Copyright 2001 Cem Kaner All rights reserved 87 Risk Based Testing O u u e Skills Teaching Testing Copyright 2001 Cem Kaner All rights reserved 88 Risk Based Testing e Exercises Given a list of ways that the program could fail for each case e Describe two ways to test for that possible failure e Explain how to make your tests more powerful against that type of possible failure e Explain why your test is powerful against that hazard Given a list of test cases e Identify a hazard that the test case might have power against e Explain why this test is powerful against that hazard Teaching Testing Copyright 2001 Cem Kaner All rights reserved 89 Risk Based Testing Exercises O e Take a function in the software under test Make a hypothetical change to the product Do a risk analysis For N 10 to 30 minutes the students privately list risk
56. served 106 Reviewing a Specification for Completeness Loo o e Reading a spec linearly is not a particularly effective way to read the document It s too easy to overlook key missing issues e We don t have time to walk through this method in this class but the general approach that use is based on James Bach s Satisfice Heuristic Test Strategy Model at http www satisfice com tools satisfice tsm 4p pdf Youcan assume not always correctly but usually that every sentence in the spec is meant to convey information The information will probably be about e the project and how it is structured funded or timed or e about the product what it is and how it works or e about the quality criteria that you should evaluate the product against 107 Teaching Testing Copyright 2001 Cem Kaner All rights reserved Reviewing a Specification for Completeness EE e Spec Review using the Satisfice Model continued The Satisfice Model lists several examples of project factors product elements and quality criteria For a given sentence in the spec ask whether it is telling you project product or quality related information Then ask whether you are getting the full story As you do the review you ll discover that project factors are missing such as deadline dates location of key files etc or that you don t understand recognize certain product elements or that you don t know how to tell wheth
57. slide Many stories with rich detail to support key points Several assignments which can be done by individuals or jointly Sample questions that can be studied by individuals or jointly e And will soon involve Self study drills A library of examples that students can read on their own Teaching Testing Copyright 2001 Cem Kaner All rights reserved Learning Styles e There s nothing magical about the division that learned and several others are more popular in the literature e For a useful introduction and discussion of several style classifications see R M Felder Matters of Style http www2 ncsu edu unity lockers users f felder public Papers LS Prism htm e Felder makes the point that the basic lessons for instructional design end up being pretty similar across the different classification schemes The next slides taken verbatim from his paper show his advice to chemistry teachers Teaching Testing Copyright 2001 Cem Kaner All rights reserved Examples from Felder s Matter of Style Loo O Here are some strategies to ensure that your courses present information that appeals to a range of learning styles These suggestions are based on the Felder Silverman model e Teach theoretical material by first presenting phenomena and problems that relate to the theory sensing inductive global For example don t jump directly into free body diagrams and force balances on the first day of a stati
58. ssatisfy customers or other stakeholders Analysis of the potential costs associated with each risk Development of plans and actions to reduce the likelihood of the risk or the magnitude of the harm Continuous assessment or monitoring of the risks or the actions taken to manage them e Useful material available free at http seir sei cmu edu e http www coyotevalley com Brian Lawrence e Good paper by Stale Amland Risk Based Testing and Metrics 16th International Conference on Testing Computer Software 1999 Teaching Testing Copyright 2001 Cem Kaner All rights reserved 92 Risk Driven Testing Cycle Perform Appropriate Tests Analyze amp Prioritize Hazards Improve Risk Analysis Process Report amp Resolve Problems Teaching Testing Copyright 2001 Cem Kaner All rights reserved 93 le Edit View Tools Window Help aa alal i DOS al Categories of Risk Sources Mission and goals Decision drivers Organization management Customer end user Budget cost Schedule Project characteristics Development process Development environment Personnel Operational environment New technology Project Consequences Cost overruns Schedule slips Inadequate functionality Canceled projects Sudden personnel changes Customer dissatisfaction Loss of company image Demoralized staff Poor product performance Legal proceedings HE TeraQuest SEPG Ris
59. t Only some testers will learn well from books Only some testers will learn well or work well on their own and some will only learn well or work well on their own e Build training strategies to achieve the learning you want to convey accommodate multiple learning styles and push for mastery not memory Teaching Testing Copyright 2001 Cem Kaner All rights reserved 124 Appendix Sample Exam Questions Loo oo From Florida Tech Testing Class Spring 2001 Sample Exam Questions e The following are examples of the review questions that my first sanester testing students work from They get the questions during the term and work together to create answers e The tests and exams are closed kook and present students with a subset of these questions e teach the course using a reference product Last year it was Tl s interactive calculator We work many of our examples and tests around that one example product Teaching Testing Copyright 2001 Cem Kaner All rights reserved 126 Reference Information TI InterActive toolbar Teaching Testing The buttons on the TI InterActive toolbar give you quick access to the program s main features Performs calculations and defines variables and functions Specifies the mode settings for each object B oO ja statistical data E e for defined functions Enters and or edits lists of data HE LUE Enters and or edits matr
60. t application that uses the database will be a feature that allows teachers to store the students grades calculate percentages final grade percentages class averages etc and print associated reports List and briefly explain 5 risk factors that you would expect to find associated with the database part of the 2 0 project Refer to Amland s paper for discussion of risk factors For each risk factor predict 2 defects that could arise in the database part of the 2 0 project By predict I mean name and describe the potential defect and explain why that particular risk factor makes this defect more likely Teaching Testing Copyright 2001 Cem Kaner All rights reserved 13 1 Long Answers a 4 Ostrand amp Balcer described the category artition method for designing tests Their first three steps are 1 Analyze 2 Partition and 3 Determine constraints Apply their method as follows I J and K are integers For each function F K Fd J The program is a simple calculator and handles the usual basic arithmetic functions Use Ostrand amp Balcer s category partition method to design a series of tests for four of the basic arithmetic functions Teaching Testing Copyright 2001 Cem Kaner All rights reserved 1 32 Long Answers OOOO 5 The Spring and Fall changes between Standard and Daylight Savings time creates an interesting problem for telephone bills Create a table that shows risks equiva
61. tioning into equivalence classes Discovering best representatives of the sub classes Combining tests of several fields Create boundary charts Find fields variables environmental conditions Identify constraints non independence in the relationships among variables Teaching Testing Copyright 2001 Cem Kaner All rights reserved 20 Domain Testing OOOO e Some Relevant Skills Identify ambiguities in specifications or descriptions of fields Find biggest smallest values of a field Discover common and distinguishing characteristics of multi dimensional fields that would justify classifying some values as equivalent to each other and different from other groups of values Standard variable combination methods such as all pairs or the approaches in Jorgensen and Beizer s books Teaching Testing Copyright 2001 Cem Kaner All rights reserved 21 Domain Testing COO e Ideas for Exercises Find the biggest smallest accepted value in a field Find the biggest smallest value that fits in a field Partition fields Read specifications to determine the actual boundaries Create boundary charts for several variables Create standard domain testing charts for different types of variables For finding variables see notes on function testing Here are a few of the exercises I actually use Teaching Testing Copyright 2001 Cem Kaner All rights reserv
62. ts can cause problems e N I B not in budget Unbudgeted tasks may be done shoddily e Ambiguity ambiguous descriptions in specs or other docs can lead to incorrect or conflicting implementations e Adapted from James Bach s lecture notes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 80 Risks Where to look for errors e Conflicting requirements ambiguity often hides conflict result is loss of value for some person e Unknown requirements requirements surface throughout development Failure to meet a legitimate requirement is a failure of quality for that stakeholder e Evolving requirements people realize what they want as the product develops Adhering to a start of the project requirements list may meet contract but fail product check out http www agilealliance org e Complexity complex code may be buggy e Bugginess features with many known bugs may also have many unknown bugs e Adapted from James Bach s lecture notes Teaching Testing Copyright 2001 Cem Kaner All rights reserved 8 1 Risks Where to look for errors Dependencies failures may trigger other failures Untestability risk of slow inefficient testing Little unit testing programmers find and fix most of their own bugs Shortcutting here is a risk Little system testing so far untested software may fail Previous reliance on narrow testing strategies e g regression function tests can yield a backl
63. ts reserved 163 Bug Question Continued So we select part of the AMEE Ex circle using the freehand sew Image por Hele selection tool We ll try fe the move and cut features ee ee Cutting fails When we try to cut the selection the dashed ine disappears but P sesreraseetses nothing goes away Selects a free form part of the picture to move copy or i Uh Teaching Testing Copyright 2001 Cem Kaner All rights reserved 164 Bug Question Continued OE Exit the program start FETE CEA again color the File Edit View Image Options Help background draw the an circle zoom to 200 select the area Drag the area up and to the right It works mm mm mm mm mm el miel le im 1 Se RER CRE Selects a free form part of the picture to move copy ot WY Teaching Testing Copyright 2001 Cem Kaner All rights reserved 165 Bug Question Continued Exit the program start again color the background draw the circle zoom to 200 select the area Now try this Select the area and move it a bit THEN press Ctrl C to cut This time cutting works F untitled Paint oO x F untitled Paint oO x File Edit View Image Options Help File Edit View Image Options Help G SE EN el ele me Slee DT CT RM CICR Pee RM For Help click Help Topics on the Help Menu p For Help click Help Topics on the
64. ve and for active student participation active Occasionally pause during a lecture to allow time for thinking and formulating questions Assign one minute papers near the end of a lecture period having students write on index cards the lecture s most important point and the single most pressing unanswered question Assign brief group problem solving exercises in class that require students to work in groups of three or four e Encourage or mandate cooperation on homework every style category Hundreds of research studies show that students who participate in cooperative learning experiences tend to earn better grades display more enthusiasm for their chosen field and improve their chances for graduation in that field relative to their counterparts in more traditional competitive class settings Teaching Testing Copyright 2001 Cem Kaner All rights reserved 10 Examples from Felder s Matter of Style OO e Demonstrate the logical flow of individual course topics sequential but also point out connections between the current material and other relevant material in the same course in other courses in the same discipline in other disciplines and in everyday experience global For example before discussing cell metabolism chemistry in detail describe energy release by glucose oxidation and relate it to energy release by nuclear fission electron orbit decay waterfalls and combustion in fireplaces power plant boilers and aut
65. y the two testing strategies that you would consider most useful and explain why Across the different tools please describe a variety of different testing strategies The buttons on the TI InterActive toolbar give you quick access to the program s main features Performs calculations and defines variables and functions Specifies the mode settings for each object a oo Lies Graphs functions and plots Ki statistical data Generates a table of values for defined functions EI ULEI Enters and or edits lists of data ma QE Enters and or edits matrices Copyright 2001 Cem Kaner All rights reserved Performs typical spreadsheet operations Performs statistics regression calculations on lists of data Transfers data to from a connected calculator Captures the screen of a connected calculator TI 83 or TI 83 Plus Browses the Web and extracts data directly from Web pages Sends e mail attachments of your current document 139 Long Answers jE 12 Suppose that you had access to the TI source code and the time opportunity to revise it Suppose that you decided to do a diagnostics based high volume automated test strategy to test TI Interactive s treatment of lists What diagnostics would you add to the code and why Describe 3 potential defects defects that you could reasonably imagine would be in the list handling software that would be easi

slides

Contents

Download Pdf Manuals

Related Search

Related Contents