Software Testing - UCSB Computer Science
Contents
1. Every module is unit tested in isolation After all of the modules are tested they are all integrated together at once and tested No driver or stub is needed However in this approach it may be hard to isolate the bugs System and Acceptance testing follows the integration phase testing the system as a whole Test cases can be constructed based on the the requirements specifications main purpose is to assure that the system meets its requirements Manual testing Somebody uses the software on a bunch of scenarios and records the results Use cases and use case scenarios in the requirements specification would be very helpful here manual testing is sometimes unavoidable usability testing system Testing Acceptance Testing UC Santa Barbara e Alpha testing is performed within the development organization e Beta testing is performed by a select group of friendly customers e Stress testing push system to extreme situations and see if it fails large number of data high input rate low input rate etc You should preserve all the test cases for a program During the maintenance phase when a change is made to the program the test cases that have been saved are used to do regression testing figuring out if a change made to the program introduced any faults Regression testing is crucial during maintenance Itis a good idea to automate regression testing so that all test cases a2. 0 print x is positive else print x is negative if y gt 0 print y is positive else print y is negative Test set T x 12 y 5 x 1 y 35 gives both branch and statement coverage but it does not give path coverage BO false B2 print x is n true B1 print x is p true false B4 B5 print y is p print y is n B6 return Set of all execution paths B0 B1 B3 B4 B6 BO B1 B3 B5 B6 BO B2 B3 B4 B6 BO B2 B3 B5 B6 Test set T executes only paths BO B1 B3 B5 B6 and BO B2 B3 B4 B6 areTheyPositive int x int y if x gt 0 print x is positive else print x is negative if y gt 0 print y is positive else print y is negative Test set T x 12 y 5 x 1 Y 35 x 115 y 13 x 91 y 2 gives both branch statement and path coverage BO true B1 print x is p true B4 print y is p B6 return false B2 print x is n false B5 print y is n Number of paths is exponential in the number of conditional branches testing cost may be expensive Note that every path in the control flow graphs may not be executable Itis possible that there are paths which will never be executed due to dependencies between branch conditions In the presence of cycles in the control flow graph for example loops w
3. Big bang Sandwich We assume that the uses hierarchy is a directed acyclic graph If there are cycles merge them to a single module Only terminal modules i e the modules that do not call other modules are tested in isolation Modules at lower levels are tested using the previously tested higher level modules Non terminal modules are not tested in isolation Requires a module driver for each module to feed the test case input to the interface of the module being tested However stubs are not needed since we are starting with the terminal modules and use already tested modules when testing modules in the lower levels Bottom up Integration UC Santa Barbara e Only modules tested in isolation are the modules which are at the highest level e After a module is tested the modules directly called by that module are merged with the already tested module and the combination is tested e Requires stub modules to simulate the functions of the missing modules that may be called However drivers are not needed since we are starting with the modules which is not used by any other module and use already tested modules when testing modules in the higher levels Top down Integration UC Santa Barbara e Sandwich Integration Compromise between bottom up and top down testing Simultaneously begin bottom up and top down testing and meet at a predetermined point in the middle e Big Bang Integration
4. that is for the quality of the information in the written report Walk through A manual static analysis technique in which a designer or programmer leads members of the development team and other interested parties through a segment of documentation or code and the participants ask questions and make comments about possible errors violation of development standards and other problems Walk throughs are a form of manual simulation Two variations led by a reader or presenter who could be the person responsible for the product led by a moderator independent of the person responsible for the product In a code walkthrough you go over the code statement by statement explaining what each statement does Objectives detect identify and describe software element defects examine alternatives and stylistic issues provide a mechanism that enables the authors to collect valuable feedback on their work yet allows them to retain the decision making authority for any changes Planning identify the walkthrough team select a place and schedule a meeting distribute all necessary input materials to the participants allowing for adequate preparation time Participants review the input material during the preparation phase e During the meeting author makes an overview presentation of the software element the author walks through the specific software element so that member of the walkthrough team
5. it satisfies the requirements specifications Involves testing a single isolated module Note that unit testing allows us to isolate the errors to a single module we know that if we find an error during unit testing it is in the module we are testing Modules in a program are not isolated they interact with each other Possible interactions calling procedures in other modules receiving procedure calls from other modules sharing variables For unit testing we need to isolate the module we want to test we do this using two things drivers and stubs Driver A program that calls the interface procedures of the module being tested and reports the results A driver simulates a module that calls the module currently being tested Stub A program that has the same interface as a module that is being used by the module being tested but is simpler A stub simulates a module called by the module currently being tested Module procedure Under Test procedure Driver call call access to global variables e Driver and Stub should have the same interface as the modules they replace e Driver and Stub should be simpler than the modules they replace Integration testing Integrated collection of modules tested as a group or partial system Integration plan specifies the order in which to combine modules into partial systems Different approaches to integration testing Bottom up Top down
6. 1 y 35 x 115 y 13 x 91 y 2 There are smaller test cases which will give us statement coverage too T x 12 y 5 x 1 y 35 There is a difference between these two test sets though e Nodes in the control flow graph are basic blocks A basic block is a sequence of statements always entered at the beginning of the block and exited at the end e Edges in the control flow graph represent the control flow e Each block has a sequence of statements e No jump from or to the middle of the block e Once a block starts executing it will execute till the end Statement vs Branch Coverage UC Santa Barbara assignAbsolute int x Consider this program segment the test set T x 1 will give statement coverage if x lt 0 however not branch coverage X 17 TX Test set x 1 does not execute this edge hence it does not give branch coverage Mi e Construct the control flow graph e Select a test set T such that by executing program P for each test case din T each edge of P s control flow graph is traversed at least once Test set x 1 does not execute this edge hence it does not give branch coverage Test set x 1 x 2 gives both statement and branch coverage e Select a test set T such that by executing program P for each test case din T all paths leading from the initial to the final node of P s control flow graph are traversed areTheyPositive int x int y if x gt
7. Planning moderator makes arrangements the materials to be inspected the checklists to be used selecting a place and scheduling the meeting The moderator controls the meeting by walking through the code A checklist is used to identify the defects e A typical inspection checklist for code inspections may include wrongful use of data unintialized variables array index out of bounds dangling pointers faults in declarations use of undeclared variables declaration of the same name in nested blocks faults in computations division by zero overflow wrong use of variables of different types in one and the same expression faults caused by an erroneous conception of operator priorities faults in relational expressions using an incorrect operator an erroneous conception of priorities of Boolean operators faults in control flow infinite loops a loop that gets executed n 1 or n 1 times rather than n faults in interfaces incorrect number of parameters parameters of the wrong types inconsistent use of global variables Audit An independent examination of a work product or a set of work products to assess compliance with specifications standards contractual agreements or other criteria Similar to inspections but More interactive than inspections Less structured than inspections You can consider your project demo an audit Software Requirements Analysis Requirements Walkthroughs Requirements Sp
8. UC Santa Barbara CS189A Capstone Christopher Kruegel Department of Computer Science UC Santa Barbara http www cs ucsb edu chris Verification Demonstration of consistency completeness and correctness of the software artifacts at each stage of and between each stage of the software life cycle Different types of verification manual inspection testing formal methods Verification answers the question Am building the product right Validation The process of evaluating software at the end of the software development to ensure compliance with respect to the customer needs and requirements Validation can be accomplished by verifying the artifacts produced at each stage of the software development life cycle Validation answers the question Am building the right product Verification Validation Testing UC Santa Barbara e Testing Examination of the behavior of a program by executing the program on sample data sets Testing is a verification technique used at the implementation stage Every phase of software life cycle requires verification techniques to find errors violating correctness omissions violating completeness contradictions violating consistency Requirements analysis and specification e use cases scenarios of expected system use help in establishing completeness and can be used to generate test cases later on in the implementation stage e formal requirement
9. e need to clarify what we mean by path coverage Given a cycle in the control flow graph we can go over the cycle arbitrary number of times which will create an infinite set of paths Redefine path coverage as each cycle must be executed 0 1 k times where k is a constant k could be 1 or 2 In branch coverage we make sure that we execute every branch at least once For conditional branches this means that we execute the TRUE branch at least once and the FALSE branch at least once Conditions for conditional branches can be compound boolean expressions A compound boolean expression consists of a combination of boolean terms combined with logical connectives AND OR and NOT Condition coverage Select atest set T such that by executing program P for each test case d in T 1 each edge of P s control flow graph is traversed at least once and 2 each boolean term that appears in a branch condition takes the value TRUE at least once and the value FALSE at least once Condition coverage is a refinement of branch coverage part 1 is same as the branch coverage something int x T x 1 y 1 x 1 y 1 will achieve statement branch and path coverage however T will if x lt 0 y lt x not achieve condition coverage because the boolean term y lt x never evaluates to true This test set y y satisfies part 1 but does not satisfy part 2 X t x BO SAART T x 1 y 1 x1 y 0 wi
10. e used to address this question Coverage metrics Statement coverage all statements in the programs should be executed at least once Branch coverage all branches in the program should be executed at least once Path coverage all execution paths in the program should be executed at lest once The best case would be to execute all paths through the code but there are some problems with this the number of paths increases fast with the number of branches in the program the number of executions of a loop may depend on the input variables and hence may not be possible to determine most of the paths can be infeasible bool isEgual int x int y e Choose a test set T such that by executing program P for each test case in T each basic pe aE z false statement of P is executed at else least once Z7 45 false e Executing a statement once return z and observing that it behaves correctly is not a guarantee for correctness but it is an heuristic int max int x int y this goes for all testing efforts since in general checking Le Ce Sy correctness is undecidable return x else return x P O rt o areTheyPositive int x int y 1f x gt Q print x 1s else Print x 15 if y gt 0 print y 1s else print y as positive negative positive negative Following test set will give us statement coverage T4 x 12 y 5 x
11. eated as a black box black box testing need an oracle oracle states precisely what the outcome of a program execution will be for a particular test case This may not always be possible oracle may give a range of plausible values e A systematic approach to functional testing requirements based testing driving test cases automatically from a formal specification of the functional requirements Partition the input domain to equivalence classes For some requirements specifications it is possible to define equivalence classes in the input domain Here is an example A factorial function specification lf the input value n is less than O then an appropriate error message must be printed If Os ns 20 then the exact value n must be printed If 20 lt n lt 200 then an approximate value of n must be printed in floating point format using some approximate numerical method The admissible error is 0 1 of the exact value Finally if n gt 200 the input can be rejected by printing an appropriate error message Possible equivalence classes D n lt 0 D 0 lt n lt 20 D 20 lt n lt 200 D n gt 200 Choose one test case per equivalence class to test If the equivalence classes are disjoint then they define a partition of the input domain If the equivalence classes are not disjoint then we can try to minimize the number of test cases while choosing representatives from different equivalence classes Exam
12. ecification Inspection Software Requirements Review Preliminary Design Design Walkthroughs Preliminary Design Review Detailed Design Design Specification Inspection Critical Design Review Coding and Unit Test System Test and Integration Code User Manual Walkthroughs Inspection Code Inspections Functional and Physical Configuration Audits Correctness software should match its specifications software should meet its functional requirements Testing is necessary because we cannot guarantee correctness in the software development process Testing techniques of checking software correctness by executing the software on some data sets Goal of testing finding faults in the software demonstrating that there are no faults in the software It is not possible to prove that there are no faults in the software using testing Testing should help locate errors not just detect their presence a yes no answer to the question is the program correct is not very helpful Testing should be repeatable could be difficult for distributed or concurrent software effect of the environment uninitialized variables If you are testing a bridge s ability to sustain weight and you test it with 1000 tons you can infer that it will sustain weight lt 1000 tons This kind of reasoning does not work for software systems software systems are not linear nor co
13. have the following three phases 1 The planning phase stating the purpose of the review arranging the participants ensuring that review materials are provided for their inspection well prior to the conduct of the review making arrangements for the location and support required preparing an agenda The meeting conduct phase follow the agenda in a disciplined manner identify problems and assign action for their resolution not try to fix them during the review amoderator or review leader maintains control recorder has to be assigned to transcribe the proceedings for preparing a record of the meeting and post meeting action list The post meeting phase flexible depending on the actions required actions are followed to completion by management and reported in the next review Review A process or meeting during which a work product or a set of work products is presented to project personnel managers users customers or other interested parties for comment or approval Types include code review design review and requirements review Characteristics for a review Review should generate a written report on status of the product reviewed a report that is available to everyone involved in the project including management Review requires active and open participation of everyone in the review group Review requires full responsibility of all participants for the quality of the review
14. ll not achieve condition coverage either false This test set satisfies part 2 but does not satisfy part 1 It does not achieve branch coverage since both test cases take the true branch and hence it does not achieve condition coverage by definition Control Flow Graph T x 1 y 2 661 y 1 achieves condition coverage e Multiple Condition Coverage requires that all possible combination of truth assignments for the Boolean terms in each branch condition should happen at least once e For example for the previous example we had x lt 0 amp amp yx Xx NT NT term1 term2 e Test set x 1 y 2 x21 y 1 achieves condition coverage test case x 1 y 2 makes term1 true term2 true and the whole expression evaluates to true i e we take the true branch test case x 1 y 1 makes term1 false term2 false and the whole expression evaluates to false i e we take the false branch e However test set x 1 y 2 x 1 y 1 does not achieve multiple condition coverage since we did not observe the following truth assignments term1 true term2 false term1 false term2 true Unit Module testing testing of a single module in an isolated environment Integration testing testing parts of the system by combining the modules System testing testing of the system as a whole after the integration phase Acceptance testing testing the system as a whole to find out if
15. may ask questions or raise issues about the software element and or make notes documenting concerns the recorder writes down comments and decision of inclusion in the walk through report e Output the walk through report contains identification of the walkthrough team identification of the software elements examined the statement of objectives that were to be handled during the walkthrough meeting A list of noted deficiencies omissions contradictions and suggestions for improvement Inspection A static analysis technique that relies on visual examination of development products to detect errors violations of developing standards and other problems Types include code inspection design inspections Inspections are used to manually check for common errors A method of rapidly evaluating material by confining attention to a few selected aspects one at a time In an inspection the inspector uses a rigid set of guidelines or a checklist to assess the degree of compliance with the checklist or guidelines In a code inspection you have a checklist that looks for errors such as uninitialized variables division by zero etc and check each item in the checklist one by one Objective is to detect defects in the product by comparison with a checklist that typifies the types of defects that are common to the type of product being inspected There is a moderator and inspectors the developer of the product and a recorder
16. ntinuous Exhaustively testing all possible input output combinations is too expensive the number of test cases increase exponentially with the number of input output variables Let P be a program and let D denote its input domain A test case d is an element of input domain d D atest case gives a valuation for all the input variables of the program A test set 7 is a finite set of test cases i e a subset of D TC D The basic difficulty in testing is finding a test set that will uncover the faults in the program Exhaustive testing corresponds to setting 7 D e Number of possible test cases assuming 32 bit integers int max int x int y _ 932 y 932 964 if x gt y e Do bigger test sets help return x Test set else x 3 y 2 x 2 y 3 return x will detect the error Testset x 3 y 2 x 4 y 3 x 5 y 1 will not detect the error although it has more test cases e Itis not the number of test cases e But if T C T then T will detect every fault detected by T Assume that the input for the max procedure was an integer array of size n Number of test cases 23 Assume that the size of the input array is not bounded Number of test cases o The point is exhaustive testing is pretty hopeless Use a random number generator to generate test cases Derive estimates for the reliability of the software using some probabilistic analysis Coverage is a problem bool iskg
17. ple D x is even D x is odd D x lt 0 D x gt 0 Test set x 48 x 23 covers all the equivalence classes On one extreme we can make each equivalence class have only one element which turns into exhaustive testing The other extreme is choosing the whole input domain D as an equivalence class which would mean that we will use only one test case For each range R R listed in either the input or output specifications choose five cases Values less than R R R Values equal to R Values greater than R but less than R Values equal to R Values greater than R For unordered sets select two values 1 in 2 notin For equality select 2 values 1 equal 2 not equal For sets lists select two cases 1 empty 2 not empty Testing Boundary Conditions UC Santa Barbara e For the factorial example ranges for variable n are O 0 20 20 200 200 Apossible test set e n 5 n 0 n 11 n 20 n 25 n 200 n 3000 lf we know the maximum and minimum values that n can take we can also add those n MIN n MAX to the test set e Structural Testing the test data is derived from the structure of the software white box testing the internal structure of the software is taken into account to derive the test cases e One of the basic questions in testing when should we stop adding new test cases to our test set Coverage metrics ar
18. re run after each modification to the software When you find a bug in your program you should write a test case that exhibits the bug Then using regression testing you can make sure that the old bugs do not reappear e Testing is a complicated task itis a good idea to have a test plan e A test plan should specify Unit tests Integration plan System tests Regression tests
19. s specifications Statecharts OCL can be checked for properties such as consistency and completeness automatically e As mentioned earlier late discovery of requirements errors is very costly Verification Through Software Life Cycle UC Santa Barbara Design e Pre post conditions class invariants can be used for verification at the detailed design stage e Design walk throughs design inspections and design review Implementation e Program testing is one of the main verification tools at this stage e Code walk throughs code inspections code review audits e Dynamic analysis tools such as dynamic monitoring of assertions and dynamic design by contract monitoring software system test software requirements testing planning preliminary integration test integration design planning testing l delivery detailed unit test x ni production design planning testing deployment maintenance system When we have an executable program we can use testing methods for verification Can we find a way to check requirements specifications design specifications and source code Manual verification techniques help in these situations Walkthroughs Inspections Reviews Audits Sometimes all of these techniques together are called Reviews These tasks are done typically in meetings manually They are useful when no automated technique is available Each Review Walk through Inspection and Audit should
20. ual Int x 1f x y z false else z false return z int y If we pick test cases randomly it is unlikely that we will pick a case where x and y have the same value If x and y can take 2 2 different values there are 2 possible test cases In 292 of them x and y are equal probability of picking a case where x is equal to y is 2 3 It is not a good idea to pick the test cases randomly with uniform distribution in this case e Testing can be categorized in different ways Functional vs Structural testing e Functional testing Generating test cases based on the functionality of the software e Structural testing Generating test cases based on the structure of the program Black box vs White box testing e Black box testing is same as functional testing Program is treated as a black box its internal structure is hidden from the testing process e White box testing is same as structural testing In white box testing internal structure of the program is taken into account Module vs Integration testing e Module testing Testing the modules of a program in isolation e Integration testing Testing an integrated set of modules e Functional testing identify the the functions which software is expected to perform create test data which will check whether these functions are performed by the software no consideration is given how the program performs these functions program is tr
Download Pdf Manuals
Related Search