Home

bacterio mutation test system user manual

1. 14 2 5 1 Mutant generation sss nan nnnenea annan 18 2 5 2 Mutant combination for producing mutant versions esses 19 2 5 3 Testcase 20 9 AFIHST EXECUTIODU es es eee 21 3 1 INITIAL EXECUTION AND 2 1 21 3 2 CONFIGURATION OF FOLDER 23 3 3 MUTANT GENERATION STEP 2 24 3 3 1 Selecting the elements to 0000000000000000000005 25 3 3 2 Selection of mutation operatofs 0 00000000000000000000000 essen 26 3 3 3 Selecting options to generate 28 3 3 4 Generation of mutant classes and mutant 8 0 00 000000000000000000000000 30 3 4 TEST GA E EXECUTIO m uP v o ai 32 3 5 RESULT a e mmm mmm Mmmm rm 36 Random les CASES TEE 39 3 5 2 Enriching the test suite with more test 41 3 5 8 Detection of equivalent versions 0 66000000000000000000 10000 nnns hannes 45 2 Paseo d
2. 22 2 1 2 2 nana 63 MUTATION PROCESS ADAPTED BACTERIO 64 7 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 1 WHAT IS MUTATION TESTING 1 2 Introduction Mutation is a testing technique which relies in the capability of test cases to find faults on the system under test Mutation has been developed by researchers and is also mainly applied by researchers After 30 years of research we consider that mutation is mature enough to be transferred to industry Being the system under test a mutant of S is a copy of S which contains a change Mutation is applied with mutation tools From the system S a mutation tool generates a set M M Ms of mutants of S Each contains one or more syntactic changes most of which will be faults and should be discovered by a good test suite Therefore the original idea of mutation is to build a test suite that finds all the faults injected on the mutants syntactic changes are seeded by mutation operators Suppose we have a program with a sum function to add two integer numbers e Figure 1 a shows the source code of the original program and of some mutants e Figure 1 b presents the results obtained from executing some test cases on the different program versions The tes
3. 9 mie big 2 MUUTA MON SO RE 9 FIGURE A POSSIBLE MUTATION TESTING 5 10 FIGURE 4 A PIECE OF CODE AND SOME TEST 5 5 11 FIGURE 5 THE TESTEQUILATERAL TEST CASE IN MUJAVA FORMAT 12 FIGURE 6 KILLING MATRIX FOR A SUPPOSED PROGRAM AND A SUPPOSED TEST SUITE 13 FIGURE 7 KILLING MATRIX FIBEEDSINABY ROW S eset quao 14 FIGURE 8 KILLING MATRIX FILLED IN BY 14 FIGURE ASNEPROSJECT EIGEINSE rmt dte e els MUI DUUM 16 FIGURE 10 JODE LICENSE 17 FIGURE 11 MUTANTS GENERATED BY BACTERIO FOR A SUPPOSED 18 FIGURE 12 SOME MUTANTS OF DIFFERENT ORDERS 18 FIGURE 13 GENERATION OF CLASS MUTANTS IN 19 FIGURE 14 AN SCHEMA OF HOW BACTERIO PRODUCES MUTANT VERSIONS 20 LES IT CASE EXECUTION 21 FIGURE 16 FOLDER STRUCTURE OF THE SUDOKUSOLVER 22 FIGURE 17 MAIN SCREEN OF 22 FIGURE 18 CONFIGURATION DIALOG BEFORE OUR SETUP 23 FIGURE 19 CON
4. Reachability the test case must reach the mutated statement Infection once the statement has been reached the test case must cause an erroneous state on the mutant e Propagation the erroneous state must be propagated to the output In this way the erroneous state will be observable from the test case s oracle If the erroneous state is not propagated until the output then the oracle will not distinguish it and will consider the mutant as alive Actually the detection of faults with JUnit style oracles is a special type of strong mutation called Functional Qualification in this case the tester uses the JUnit oracle assert and fail statements for detecting the killed mutants However also the RIP conditions are required for this Weak Mutation is another type of mutation which only requires the first two conditions in some place between the mutated statement and the program output the mutation execution tool compares the states of both original program and of the mutant If they differ the mutant is killed otherwise is alive There are several places where the tool can check the states differences and thus there are several types of weak mutation BB Weak 1 Execution is stopped just after execution of the mutated statement and the state of the mutated system is compared with the state of the original system BB Weak n Execution is stopped just after the execution of the mutated statement if it is not inside a lo
5. Empresa de Base Tecnol gica Universidad de Castilla La Mancha then the mutants of P can be generated Test cases on 7 are executed against mutants note that when a mutant is killed this is when its fault is discovered no more test cases should be executed against it for cost savings ought to this the square in the figure is labeled as Run T on each alive mutant There will also be some ineffective test cases this is test cases which do not discover any faults or which discovers the same faults than others If the Mutation Score threshold is reached then the testing can stop otherwise there are undiscovered faults which should be discovered by new test cases which must be added to the test suite These new test cases must be executed against P the program under test since maybe this one contains faults in unexplored areas of its code Note moreover an additional task named Remove equivalent mutants Actually the detection of equivalent mutants is made by hand and may require a huge time Thus maybe the Mutation Score threshold one of the stop conditions for the testing process the other one is the non discovering of faults on P could be decreased down to 80 Fur O i enerate A tesl cases Input test program Are there vali mutants Ps c Pul yf Create mutants Run T on each alive mutant Add new tes cases ta T Define threshold am a a T
6. TriTyp t new TriTyp t setI 4 E setJ 5 t setK 9 t getType assertTrue t triLyp TriTyp NOT A TRIANGLE Figure 4 A piece of code and some test cases Suppose we change gt by on the line marked up with and we leave it as follows if trityp 3 which is the only branch in the function to decide about equilateral triangles Therefore the testEOUILATERAL test case will evidence a different behaviour on the original program and on the mutant so killing it In this case the test case has found the error in the mutant In other mutation tools such as MuJava test cases are slightly different The same testEOUILATERAL test case would be as in Figure 5 11 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Public int testEQUILATERAL TriTyp t new TriTyp t setI 5 t setJ 5 t setK 5 return t getType Figure 5 The testEQUILATERAL test case in MuJava format When the MuJava test case is executed on the original program it returns 3 the value of the TriTyp EQUILATERAL contant when executed against the mutant it returns 4 NOT A TRIANGLE Thus MuJava detects the fault and considers the mutant as killed This way of detecting the faults is called Strong mutation Strong mutation requires the three R P conditions to consider a test case kills a mutant
7. nnnm nnn nnne nennen 33 FIGURE 34 BACTERIO LOADS AND SHOWS THE ALIVE VERSIONS CURRENTLY NONE IS A BEIDE i 34 FIGURE 35 SELECTION OF THE TEST CASES TO 35 FIGURE 36 BACTERIO TRYING TO EXECUTE AND KILL MUTANTS 35 FIGURE 37 RESULTS OF THE EXECUTION OF THE TEST CASES eee 36 FIGURE 38 TOTAL NUMBER OF KILLED necs Video 36 FIGURE 39 PERCENTAGE OF KILLED tenidas 37 FIGURE IM TRIX 37 FIGURE 41 EXECUTION TIME VA aeta 38 FIGURE 42 SUMMA OF RESULT nta 38 FIGURE 43 THE EXECUTION WINDOW UPDATED WITH NEW 38 FIGURE 44 SCORE MUTTION MATRIX WITH GLOBAL 39 FIGURE 45 EXECUTING A RANDOM 39 FIGURE 46 RANDOMNESS FOUND 40 FIGURE 47 SOURCE CODE OF THE ORIGINAL LEFT AND OF THE MUTANT RIGHT 42 FIGURE 48 ALIVE MUTANTS 8 43 FIGURE 49 NEW TEST CASE 43 FIGURE 50 RESULTS OP THE NEW TEST GASE dl da o daa 44 FIGURES TOTAL AB SUL
8. 0 b 0 throw new NANException ABS int div int a int b throws int div int a int b throws if Math abs a 0 amp amp b 0 throw if a 0 amp amp Math abs b 0 throw if b 0 throw new ByZeroException x return a b if 6 0 throw new ByZeroException return a b int div int a int b throws int div int a int b throws if a 0 amp amp b 0 throw if a 0 amp amp b 0 throw if b 0 throw if b 0 throw return a b return a b int div int a int b throws NaNException ByZeroException if a 0 b 0 throw new NANException if b 0 throw new ByZeroException return a b Table 1 Source code representation of some changes introduced in a small piece of bytecode 27 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter For this example we ll select all of them excepting LCR Figure 24 Interface mutation operators vi Swap parameters A Throw exceptions INC M DEC NUL Traditional mutation operators AOR M ROR LCR Figure 24 Selection of mutation operators 3 3 3 Selecting options to generate versions Suppose we have the system in Figure 25 It is composed by four classes A B C D which respectively have produced 5 4 6 and 3 mutants Original system A C Dj Mutants A
9. 1 Figure 53 The Equivalent versions list updated 3 5 4 Exploratory test cases Some applications are mainly based in graphical user interfaces thus the designing of test cases could be hard and it is more useful to explore the system by hand This exploratory testing usually is done by a tester who runs the system and interacts with the graphical user interfaces to execute functionalities In order to apply mutation analysis to this GUI based systems Bacterio can store the interactions of a user with the system and reproduce them with the mutant versions Exploratory test cases cannot be analyzed with Functional Qualification mutation kind so first we must to configurate the Bacterio properly Figure 54 shows the configuration windows properly configured In this case we have selected Flexible Weak Mutation kind but we could select any other kind different from Functional Qualification The remaining properties can be left as usual 46 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Configuration Y ByteCode modification in runtime Comparison Method 9 toString Fields inspection toString if defined otherwise fields inspection Hashcode fields inspection Q toString if defined otherwise hashcode fields inspection Depth in field inspection Servers configuration Local execution Loca
10. Iversion102 Alive versions Unselect all y Select all Jversionl n version10 T fversion100 fversion 10 1 H E live versions 550 Figure 34 Bacterio loads and shows the alive versions currently none is killed Now we select the versions we are interested in testing which will be all of them Then we press the Select all button and then the Execute mutants button Bacterio shows now a small dialog Figure 35 to select the test cases contained in the test suites saved on the Tests cases folder we specified in the configuration window of Figure 19 34 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha ADE Test Cases selection All the tests M tests Testl M testGoForward M testGoBack O Full Tests Execute first with JUnit 9 Version oriented Cancel MM Test Case oriented OK Figure 35 Selection of the test cases to execute We leave the options as presented and press the Ok button to execute the test cases against the mutants Figure 36 1 In the first time Bacterio executes twice the test cases against the original system In this way Bacterio is capable of detecting areas in the SUT whose behavior may depend on chance which are marked and will not be taken into account during mutant execution 2 Then Bacterio launch
11. Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha coverage code coverage input data partition When the tests are made in the state 2 the tester has to test the system and fix any bug found by the test cases Then tester goes to the state 3 Here the tester uses Bacterio In this state the tester has to create mutant versions of the system with the correct configuration specified by the inputs of the process When the versions are prepared in the state 4 the tester has to run all the test cases T with all alive versions in this case all versions Then the tester checks the mutation score reached if it is equal or higher than the mutation score desired the process finish Otherwise the tester must check is there are uncovered mutant versions in other words mutants versions that its mutant part has no be reached by the test cases In that case in state 5 the tester has to select one decompile it and inspect it to know how is the mutation done and how to kill that version If there are not uncovered versions in state 6 the tester has to select a covered but no killed versions and inspect that in the same way If the inspected mutant version is equivalent in state 7 the tester has to mark the version as equivalent and check again the mutation score reached If the version is not equivalent in state 8 the tester has to design and implement a new test case in
12. 3 6 The Mutation Process With Baterio Reducing the effort of testers The mutation process presented in Figure 84 is a generic process that could be used 63 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha with any tool However in order to improve the process and reduce human effort the process has been adapted to Bacterio tool taking advantage of the features of the tool The Bacterio s mutation process is shown in Figure 84 This is the process that a tester should follow to test a system using mutation and he Bacterio tool At the end of the process the user will have a set of mutate adequate test cases and all the bugs found by the tests fixed _ 8 Incremental test suite T Does the System Change ls equivalent Are there uncovered mutants J Score achieved Figure 84 Mutation process adapted to Bacterio tool The inputs of the process are the system under test the desired mutation score to reach and the mutation techniques desired mutation operators high order mutation etc that are supported by Bacterio In the state 1 the tester has to design test cases for the system Any method can be used to design tests thus the tester can use the most comfortable for him her functionalities 64 Paseo de la Universidad 4 13071 Ciudad Real
13. 55 M M A lib 5 4 X jar Y jar 5 images Y etc II A version 2 version Mutated versions Figure 14 An schema of how Bacterio produces mutant versions 2 5 3 Test case execution Bacterio may execute different types of test cases and in several ways It takes the test suite and launches its test cases to the original application and to the mutant versions Depending on the mutation type selected strong weak flexible weak or functional qualification the results may have differences Moreover Bacterio also allows the execution of functional tests interacting with the frames composing the original application the events launched by the tester to the SUT can be saved and later reproduced on each mutant to find faults Figure 15 illustrates this double possibility 20 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Actual user Test cases execution versions version 1 Step 3 Version execution version 2 version K Execution results Mutated versions Figure 15 Test case execution 3 A FIRST EXECUTION 3 1 Initial execution and configuration We have executed Bacterio with no problems on Windows Linux and Mac
14. An class Bm class Zp class aa Mutant suite Y jar images etc Original application Figure 13 Generation of class mutants in Bacterio 2 5 2 Mutant combination for producing mutant versions In a previous version Bacterio built so many copies of the SUT as mutants there were Due to the high space requirements of this approach we modified it and now it loads at runtime the corresponding mutants of each version That primitive way of mounting the mutant versions is however valid to exemplify the results consider that it works as in Figure 14 where 2 mutant versions are mounted version 1 contains a class mutant from A and another one from Z whereas B C etc are copies of the original ones version 2 would contain any other two mutant classes etc 19 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha versions version 1 A1 class B1 class Z1 class E 4 e ec A2 class B2 class 22 488 zz Step 2 2 4 7 SIC Version generation m i A classes An class Bm class Zp class yO ve Mutant suite
15. OS systems The examples in this chapter are executed on a MacBook Pro with Mac OS X For this section we will use the SudokuSolver system which in our computer is saved on a SudokuSolver folder on the desktop The application has the folder structure shown on Figure 16 Note the source code is under the src folder whereas the compiled code is under bin The remaining subfolders preserve the package structure of the application Note there is also a tests package containing a Test class which contains JUnit test cases 21 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha A SudokuSolver gll bin Y sudokusolver B3 dominio Casilla class Ristras class 2 Shower class 2 Sudoku class Utilidades class gt 3 gui TI tests E Testl class src Y sudokusolver dominio Casilla java Ristras java Shower java Sudoku java Utilidades java gt B gui Y B tests Testl java Figure 16 Folder structure of the SudokuSolver project Bacterio is executed just launching the Bacterio jar file The main window of the tool appears Figure 17 m a A Mutation functional testing Configuration Mutation Help Exploratory Testing Mutation Test System Figure 17 Main screen of Bacterio 22 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 92
16. Q 25 FK Mount versions Figure 26 Algorithms and options for class mutants combination into mutated versions Depending on the algorithm the number and properties of the mutated versions will be different In all cases the mutated versions will have four classes but e With One fault per version each mutated version will contain one mutated class and three original classes Versions in this category are for example A B C D As B C D A 0 B Cs D etc e With Each Choice each mutated version will contain n mutants being n the selected order Each mutant will be introduced in at least a mutated version The selection of the mutants to build the versions is sequential starting at the beginning of the list of mutants e With Each Choice first last each mutated version will contain n mutants being n the selected order Each mutant will be introduced in at least a mutated version The selection of the mutants to build the versions uses two sequences from the beginning and from the end of the lists of mutants selecting each time a mutant from a different sequence e With Random Each Choice each mutated version will contain n mutants being n the selected order Each mutant will be introduced in at least one mutated version The selection of the mutants to build the versions is random e With Pair wise each pair of mutant classes proceeding from different classes will be introduce
17. los meros que faltan 5 1 9 2 Si la resoluciUn no avanza entonces tendr s que ayudarle introduciendo un mer L lt Run version 2 1 p esto puedes ayudarte de los botones Avanzar y Retroceder 4 Ten en cuenta que el programa no te informa de la correcci n o incorrecciUn de lo 5 7 H N 3 2 6 PETERET Ee E EN Load A LE Retroceder Avanzar Salir Te 6 7 5 A LE A 4 5 1 9 2 A 213 7 9 x Ls LI 4117 3111 Sudoku de muestra Figure 58 Sudoku application running Executing a version original Java Options App Entrada Test case name ATT ExploratoryTestCasel gt Figure 59 Saving a exploratory test case Now Bacterio can re execute the captured test case with the mutant versions that we select In the execution window we select Captured Test radio button Figure 60 and click on Execute Mutants button At this point Bacterio give us a warning Figure 61 because during the re execution of captured tests the user must not use the keyboard and mouse because Bacterio uses them to reexecute the tests in fact Bacterio move the mouse and create key and click events If we accept the window of Figure 62 appears where we can select the captured tests to be run with the selected mutants Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 49 Em
18. oR DA TED URS UR aaa 44 FIGURE 52 DECOMPIEED OF VERSIONS Tis 45 FIGURE 53 THE EQUIVALENT VERSIONS LIST UPDATED 46 FIGURE 54 CONFIGURATION FOR EXPLORATORY 47 FIGURE 55 GO TO CAPTURE EXPLORATORY 5 47 FIGURE 56 CAPTURE EXPLORATORY TESTS WINDOW 48 FIGURE 97 EXECUTION SEI aaa aaa aaa aa 48 FIGURE 58 SUDOKU APPLICATION 49 FIGURE 59 SAVING EXPLORATORY TEST 6 49 FIGURE 60 CAPTURE TEST BUTTON 9 50 FIGURE 61 WARNING MESSAGE BEFORE TO EXECUTE A EXPLORATORY 50 FIGURE 62 EXPLORATORY TEST 50 FIGURE 63 PARALLEL EXECUTION 51 FIGURE 64 START A REMOTE EXECUTOR NODE 52 FIGURE 65 START A FACADE NODE otitis 52 FIGURE 66 FACADE NODE nnns 54 FIGURE 67 LOCAL NODE CONFIGURED TO EXECUTE IN DO FIGURE 68 BASIC CONFIGURATION 55 FIGURE 69 ADVANCED CONFIG
19. order to kill the mutant version inspected in states 5 or 6 and add the new test case to the test suite In state 9 the tester has to test the system with the new test case and fix any bug found by the new test case If there were bugs and the system had to be change to fix they the tester goes to the state 3 because it is necessary to create new versions of the system During the creation of this new mutant versions Bacterio analyze and compare them with the information stored about equivalent mutants detected until now If a new mutant version has mutations previously defined as equivalent the new mutant version is mark equivalent automatically in order to do not lose previous work done this process takes into account the modification of the system thus the equivalent mutations done in modified parts are removed from the base of knowledge of Bacterio If there were no bugs the tester goes to state 10 Here the tester has to execute all alive versions uncovered and covered but not killed with the new test It has not sense re execute the complete test suite because the old test cases would not kill any alive version Then the tester has to check the mutation score reached again This process finish when the tester checks the mutation scores reached and are equal or higher than the mutation scores desired 65 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354
20. paramet paramet paramet paramet paramet paramet paramet paramet Versions Version VCI SIUII2 1 version542 version543 version544 version545 version546 version547 version548 version549 version5 50 Mutantl PUL SUUUKRUSUIVE UULU 2 1_1_ 117 ror sudokusolver dominio Casilla 54 4 420 ror sudokusolver dominio Casilla 58 3 424 ror sudokusolver dominio Casilla 7 2 373 uoi sudokusolver dominio Casilla 11 0 1792 uoi sudokusolver dominio Casilla 12 0 1793 uoi sudokusolver dominio Casilla 19 0 1800 uoi sudokusolver dominio Casilla 3 0 1784 uoi sudokusolver dominio Casilla 4 0 1785 uoi sudokusolver dominio Casilla 9 0 1790 mm n 0 n Figure 31 Location of the mutant versions Fecha de modificaci n Tamano Ww Ww W YYYY 2011 1 3 20 50 30 mutation session txt 2011 1 3 20 50 43 generation session txt 27 equivalentMutants 3 original 3 version1 version2 LJ version3 version4 version5 C version6 LJ version7 version8 version9 version10 version11 C version12 3 version13 version14 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 Hoy 20 50 3 4 Test cases execution 295 KB 147 KB 0 KB Clase Texto norm
21. part of code with a 5 in the input for example which will produce a different behavior of the system This new test case can be designed as a new test method in the same class that the other tests or in a new test class but the new c ass files created must be in the same classpath defined in the configuration window Figure 19 When a new test case has been added to the test suite in the example testFivelnput test case it can be executed with Bacterio In order to execute this new test case only the 42 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha lived versions must be selected Figure 48 because there is not sense to execute again the dead versions Died versions Alive versions Alive Covered versions l l S p E H l 3 l I 9 UT Select all Unselectall Select All Unselect All Selectall Unselectall a al a jversionl ah 115 versionlO fversion166 version92 jversion100 fversion189 version93 101 jversion190 versionlO 2 1 jversionl91 rr ue Total covered versions 300 Total died versions 208 Figure 48 Alive mutants selected Then we execute the selected versions normally Figure 49 shows that now the new test case can be sel
22. the functionalities described in this document create mutants execute mutants and view results The mutant creation and the result calculation are done in the local node only the execution of the test cases is done in parallel The remote executor nodes are in charge of running tests cases These nodes receive tests and versions execute that tests with the versions and send back the tests results The facade node is in charge of receiving from the local node all the selected versions and tests and of their distribution between the remote executor nodes based on a distribution algorithm To configure the system to run in parallel first we have to execute each remote executor node in a different computer using the command java jar remoteExecutor jar port tempFolder numberOfProcesses where ip is the ip of the computer port an available port of the system tempFolder is the path of a empty folder to store temporal data test cases versions log files and numberOfProcesses is the number of cores of the processor of the computer this parameter cat be set as1 by default The Figure 64 shows an example of how to run a remote executor node 51 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Terminal java 105 9 A O O java jar remoteExecutor jar 127 0 0 1 12345 temp 1 B main Figur
23. useful when the computer has more than one processor It is not used when the parallel execution is done remotely 3 5 6 2 Mutant generation window Classpath 3 Apply mutation on Figure 70 Generation mutants window Classpaths Classpath This box is used to include the classpath of the system under test classes folders and jar files These elements can be added and removed with the buttons and Apply mutation on This box is used to specify the folders that contain the classes to be mutated This folders can be added or removed with the buttons gt gt and lt lt T ww mutation operators Swap parameters Throw exceptions INC DEC NUL Traditional mutation operators AOR ROR JABS UOI LCR Get versions 5 Figure 71 Genertaion mutants window mutation operators e Interface Mutation operators Mutations operators for interfaces that can be selected Swap TEX INC DEC NUL e Traditional mutation operators Traditional mutation operators that can be selected AOR ROR ABS UOI LCR e Get versions Button used to apply the mutation operators 58 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha E 9 Clear previous mutants Ma Figure 72 Generation mutants window Clear previous mutants butto
24. 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Execution options Selected versions Test case file C Captured test Execute original Execute Mutants View Results Figure 79 Execution mutants window Execution options Execution options This radio buttons are used to define which kind of execution will be done There are three options o Selected versions This option executes each versions directly thus the user can interact directly with the mutated versions of the system under test o Test case file This option executes test cases in JUnit or UlSpect4J format o Captured test This option executes exploratory tests previously captured with Bacterio e Execute original This button is used to execute only the original version of the system under tests This option is useful when we make changes in a test case that can be executed previously e Execute mutants this button is used to execute all the selected versions and the original version The original version is only executed if it was not executed before e View results This button is used to view the results of the executions Main class sudokusolver dominio Casilla class Figure 80 Execution mutants window Main class class Combobox used to select the main class of the system Used only when the options Selected versions is selec
25. 6 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 3 2 Configuration of folders Once we are in this window the first to do is to configure the tool properly For this select Configuration in the menu and the window in Figure 18 appears AMOO Configuration Version folder c dominio Select Test cases folder c dominio original Select Timeout 20000 B Avanced Options mcs x on Figure 18 Configuration dialog before our setup We ll establish the Version folder to a version folder we must create in any place for example on the desktop The Test case folder must point to the root folder containing the compiled test cases In this example test cases are in bin tests Test1 class Figure 16 the complete name of the Test1 class class is tests Test1 class Thus the Tests cases folder will be bin and the configuration remains as in Figure 19 ADO Configuration Version folder Users Maco Desktop versions Select Test cases folder Users Maco Desktop SudokuSolver bin Timeout 20000 B Avanced Options AS Cancel 1 J 1 J Figure 19 Configuration for this example As you see there is a Timeout field in the configuration dialog Sometimes the fault introduced in a mutant may lead the execution into an infinite loop Figure 20 To consider a mutant as killed when it spends too much time under execution you can configure the value of
26. Aa As B Bz B4 C Cz C3 Cy Cs D D D3 Figure 25 A system composed by four classes and its mutants As Figure 26 shows these class mutants can be combined into mutated versions applying two mutation techniques High Order Mutation and Mutant Sampling The High Order Mutation technique combines more than one mutant in each version This technique can be applied with several algorithms One fault version three Each choice and Pair wise and restrictions do not joint mutants in the same class method or with the same mutation operator to combine errors and with different orders which can be determine to indicate how many mutants combine in each version The Mutant sampling bar appearing beneath allows to mount and to test only a percentage of the total number of versions which are randomly selected Some studies have shown that the Mutation Score got on a small random percentage of mutants is almost equivalent to the Mutation Score on the total number of mutants 28 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha High order mutation Do not join mutants of the same operator Do not join mutants in the same class Do not join mutants in the same method 9 One fault version Each choice Pair wise Each choice first last Random each choice Order Mutant sampling Size
27. BACTERIO MUTATION TEST SYSTEM a Mutation Tool for Multi Class Java Systems USER MANUAL 13071 Ciudad Real Telf 926 295 480 Fax 926 295 354 www alarcosqualitycenter com Empresa de Base Tecnol gica Universidad de Castilla La Mancha User Manual November 2011 HISTORICAL EVOLUTION Comments 09 11 2009 Alarcos Quality Center 10 11 2011 Pedro Reales Mateo MEE Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha INDEX OF DE CONTENT INDEX OF Ms zies qp gc E 2 INDEX TABLE PC 4 INDEX OF FIGURE S c 5 1 WHAT IS MUTATION TESTING a a publi bio 8 1 2 INTRODUCTION M 8 1 3 MUTATION TYPE T 10 1 4 KIEEING MA ucbicqdmc 13 2 AN OVERVIEW OF BACTERIO conim tis ostate iut uai Guida unici iiri Cni 15 2 2 BAC TERIO INA saa a 15 2 3 15 2 4 THIRD PARTY SOP DW 15 2 41 e nee A 15 2 4 8 JODE Java Optimize and Decompile Environment sessi 16 222 ADOMI 17 2 5 THE TESTING PROCESS WITH BACTERIO AN
28. FIGURATION FOR THIS EXAMPLE 23 FIGURE 20 THE MUTANT PRODUCES AN INFINITE LOOP 24 FIGURE 21 ADVANGED OFTIONS Rusa S daB Ed PI uide 24 FIGURE 22 WINDOW FOR MUTANT 8 25 FIGURE 23 SELECTING THE SUT LOCATION LEFT AND SELECTION OF THE LOCATION OF THE GLASS FILES TOBE MUTATED RIGHT 25 FIGURE 24 SELECTION OF MUTATION 5 28 FIGURE 25 A SYSTEM COMPOSED BY FOUR CLASSES AND ITS 5 2 2 222 28 FIGURE 26 ALGORITHMS AND OPTIONS FOR CLASS MUTANTS COMBINATION INTO MUTATED VEROON an ee eer 29 FIGURE 27 POSSIBLE COMBINATIONS WITH PAIR WISE AND ORDERudA esee 30 FIGURE 28 A FINE SELECTION OF THE CLASSES TO 30 FIGURE 29 THE GENERATION OF CLASS MUTANTS HAS FINISHED 31 FIGURE 30 LOCATION OF THE 31 FIGURE 31 LOCATION OF THE MUTANT 5 32 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha FIGURE 32 BACTERIO CREATES A FOLDER FOR SAVING THE RESULTS OF EACH VERSION 32 FIGURE 33 MUTANT EXECUTION
29. Figure 43 The execution window updated with new information Also the information with the score achieved with all generated versions is updated Figure 44 38 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Total versions results versions mutants ror inc uoi dec swa aor nuf abs 37 81 37 81 36 86 66 66 69 81 80 0 72 72 32 14 30 0 1 83 Figure 44 Score muttion matrix with global results 3 5 1 Random test cases Sometimes depending of the domain of the system under test the test cases and the Bacterio configuration Bacterio detects chance in the outputs of the original system For example when some values are randomly calculated or generated in the system and they are considered part of the output or take effect in the output like dice results Chance has a dramatic effect in the mutants results because all the mutants would be killed by a test case that produce a random output because the output will be almost always different thus all the mutants will produce a different output than the output of the original system and they will be killed Bacterio detects this randomness executing the original version twice an applies a special treatment to the random outputs discarding the random parts of the outputs However the random test cases suppose a risk and can produce deviations in the mutatio
30. In particular we ll put our focus on those versions whose mutations have not been traversed by test cases such as version91 version92 version93 and version94 in Figure 43 In order to write a test case to kill version91 and likely other versions we can decompile version91 to inspect its source code and see the mutation done Bacterio makes this thanks to the JODE Java decompiler 2 section 2 4 2 The resulting source code is not exactly the same written by the programmer but can help the tester to define a test that could kill the mutant Then we locate version91 in the left side list of Figure 43 and make double click on it Bacterio opens a window Figure 47 which show the source code of the original version Figure 47 left and the mutated version Figure 47 right and highlights the mutated statement the mutation operator has changed gt by lt in the validar function of the JTFCasilla class 41 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Source comparison Original source code Mutant source code package sudokusolver gui import java awt Rectangle import java awt event KeyAdapter import java awt event KeyEvent package sudokusolver qui import java awt Rectangle import java awt event KeyAdapter import java awt event KeyEvent import javax swing JTextField import javax swing
31. JTextField public class JTFCasilla extends JTextField public class JTFCasilla extends JTextField public JTFCasilla public JTFCasilla 1 initialize initialize private void initialize private void initialize 1 setBounds new Rectangle 0 0 24 257 addKeyListener new KeyAdapter 1 final JTFCasilla this 0 JTFCasilla this superi public void keyReleased KeyEvent keyevent 1 this O validar keyevent getKeyChar public void validar char c setText if getTextO length gt 1 setText setBounds new Rectangle 0 0 24 25 addKeyListener new KeyAdapter 1 final JTFCasilla this 0 JTFCasilla this super public void keyReleased KeyEvent keyevent 1 this O validar keyevent getKeyChar n public void validarichar c setText if getTextO length gt 1 setText public String toString 1 return getText public String toString 1 return getText Mark as equivalent Figure 47 Source code of the original left and of the mutant right The original conditional statement checks whether the value in a cell of the sudoku is between 1 and 9 in order to validate an input to the system what means that the user can only insert a number between 1 and 9 However the mutated version has a mutated condition which do not allows numbers lower 10 Thus the tester should design a test case that could execute that
32. Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha ADE Pararell Execution System Distributor Server Address Distributor Server Address Distributor Server Address Port 192 168 0 2 12346 Algorithms PEDRO Algorithm 0 Distribute versions 1 Distribution Algorithms Distribute test cases 2 Give versions on demand 3 Give test cases on demand 4 Parallel Nodes Unused addresses Used addresses gt 127 0 0 1 12345 nnns A e Remove 4 EEE Remove Selected Addresses Remove Selected Addresses IP Address 127 0 0 1 Add Address Port Address 12345 Start Distributor Server Figure 66 Facade node configured When the remote executor nodes and the face node are running we can configurate the local node In the advanced configuration window we de select local execution check button and fill the fields Local IP Local Port Server IP and Server Port Figure 67 Server IP and Server Port are the address of the facade node Now all the executions are done in parallel 54 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha ByteCode modification runtime Comparison Method 9 toString O Fields inspection toString if defined otherwise fields
33. URATION 56 FIGURE 70 GENERATION MUTANTS WINDOW 5 58 6 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 FIGURE 71 FIGURE 72 FIGURE 73 FIGURE 74 FIGURE 75 FIGURE 76 FIGURE 77 FIGURE 78 FIGURE 79 FIGURE 80 FIGURE 81 FIGURE 82 FIGURE 83 FIGURE 84 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter GENERTAION MUTANTS WINDOW MUTATION OPERATORS eene 58 GENERATION MUTANTS WINDOW CLEAR PREVIOUS MUTANTS BUTTON 59 GENERATION MUTANTS WINDOW VERSIONS 59 GENERATION MUTANTS WINDOW SHOW RESULTS 60 EXECUTION MUTANTS WINDOW LOAD 60 EXECUTION MUTANTS WINDOW AVAILABLE VERSIONS 60 EXECUTION MUTANTS WINDOW TOTAL RESULTS 61 EXECUTION MUTANTS WINDOW CONSOLE OUTPUT HH 61 EXECUTION MUTANTS WINDOW EXECUTION OPTIONS 62 EXECUTION MUTANTS WINDOW MAIN 5 62 EXECUTION MUTANTS WINDOW RESET TEST CASES 63 EXECUTION MUTANTS WINDOW VIEW PREVIOUS EXECUTION RESULTS BUTTON 63 TEST CASES SELECTION
34. able versions Four lists that show the mutant versions generated o Alive versions Versions that the mutated parts have not been reached by the executed test If any test has not been executed only this list contains versions 60 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter o Alive Covered versions Versions that the mutated parts have been reached by at list one of the executed tests but the output of the version is equals to the output of the original version o Died versions Versions that have been killed o Equivalent versions Versions marked as equivalent Total versions results versions mutants ror inc uoi dec swa aor nuf abs 38 9796 38 9796 38 1396 66 66 70 75 80 0 72 7296 32 1496 40 096 2 7796 Figure 77 Execution mutants window Total results e Total versions results Table that shows the mutation scores relative to all the generated versions and mutants console output Console activate Figure 78 Execution mutants window Console output e Console output Box that shows the standard output of the test cases lt has one option o Console activated This option can be selected to capture the standard output of the test and show it in the box If this check button is not selected the standard output is not captured 61 Paseo de la Universidad 4
35. ain class sudokusolver gui JFSudoku class Load Versions on execution Start Capturing Figure 56 Capture exploratory tests window When the Start Capturing Box is clicked the execution console appears Figure 57 Here we can insert arguments to the system and to the Java Virtual Machine For the sudoku application no arguments are needed thus we just need to click on Run version button Then the Sudoku application starts and all the interactions with it are store Figure 58 When we close the application and the console window Bacterio asks for the name of the test case in the example ExploratoryTestCase1 see Figure 59 and after clicking the Ok button the test case 1 stored with the Bacterio s format in the path configured 806 Executing a version original Java Options Application Arguments Run version o AA Figure 57 Execution console 48 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Executing a version original Mutation functional testing Java Options Application Arguments A Solucionador de sudokus por pasos c Macario Polo Usaola 2007 3 5 48 Pasos 1 Escribe los n meros que conozcas en las casillas que correspondan 1 A 2 Pulsa el botUn Avanzar que te dE algunos de
36. al Texto normal Documento Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Carpeta Figure 32 Bacterio creates a folder for saving the results of each version At this moment we have all we require to execute test cases against the original and the mutants We have mutant versions of the system We have test cases in tests Test1 class Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 32 enter Empresa de Base Tecnol gica Universidad de Castilla La Mancha In order to execute the test cases saved in Test1 class we select Mutation Execution on the menu bar of the main screen Figure 17 The window in Figure 33 appears eoo Configuration Mutation Help Exploratory Testing Mutation functional testing Main class Available versions Alive versions Alive Covered versions Died versions RIA nsE AA SIA RA Selectall Unselectall Select All Unselect All Selectall Unselectall Total versions results Load T HM E Reset Test Cases N Equivalent versions Execution options Selected versions e Test case file Captured test Execute original Execute Mutants View Results Console activate View Previous Execution Results Fi
37. am says that a triangle instance with sides of lengths 5 5 and 5 is in fact an equilateral triangle import junit framework TestCase return 1 if scalene 2 if isosceles 3 if equilateral 4 if not a triangle er publicvoid getType publicclass TriTypTest extends TestCase publicvoid testEQUILATERAL if i 3 TriTyp t new TriTyp LEriryp trityp 1 t setI 5 t setJ 5 if i k Lrityp trityp 27 if j E 1 bribvp Lrityp 4 3 if 1 lt 0 1 lt 0 k lt 0 trityp TriTyp NOT A TRIANGLE return t setK 5 t getType assertTrue t trityp TriTyp EQUILATERAL publicvoid testISOSCELES TriTyp t new TriTyp set1 5 t setJ 5 t setK 8 t getType if trityp 0 4 if i p k j lt i i k lt j trityp TriTyp NOT A TRIANGLE return else assertTrue t trityp TriTyp ISOSCELES publicvoid testSCALENE 1 TriTyp t new TriTyp t setI 4 trityb t setJ 5 return t setK 8 t getType assertTrue t trityp TriTyp SCALENE LE trityp gt 3 4 trityp TriTyp EQUILATERAL elseif trityp 1 amp amp i 1 gt trityp TriTyp ISOSCELES elseif trityp 2 amp amp i k gt j Lrityp TriTyp ISOSCELES elseif trityp 3 amp amp j gt 1 1 trityp TriTyp ISOSCELES else trityp TriTyp NOT A TRIANGLE Tri Typ SCALENE publicvoid testNOTRIANGLE
38. ble risk M All the acceptable risk tests OK Figure 46 Randomness found window After Bacterio executes the original system the window of Figure 46 appears In this window there are four kinds of risk 1 Low risk Less than 5 of the output of the system is random so this is a very low risk and thanks to the special treatment of Bacterio to execute the test is safety enough 2 Acceptable risk Between 596 and 1096 of the output of the system is random so this risk is acceptable and thanks to the special treatment of Bacterio the mutation scores will not be affected 3 High risk Between 1096 and 5096 of the output of the system is random so this risk is high and even with the special treatment of Bacterio the mutation scores can be affected 4 No acceptable risk More than 50 of the output of the system is random so this risk is not acceptable and even with the special treatment of Bacterio the mutation scores will be affected Thus after to study the possible risks the tester can deselect those tests desired and they will not executed by Bacterio 40 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 3 5 2 Enriching the test suite with more test cases oince we are only killing 37 8196 of mutants we need to add new test cases to the test suite in order to reach a higher mutation score
39. cation of the mutations Now we press the Mount versions button in order to leave Bacterio to combine the class mutants into mutant versions Since the selected algorithm was One fault version Bacterio produces so many versions as mutants which are saved in the Versions folder specified in the configuration dialog Figure 19 Desktop versions in this case For this example there are 2 199 mutants As we selected 2596 of Mutant sampling the tool generates 550 mutant versions randomly selected Figure 31 As the One fault mutant algorithm was selected each version holds a mutant class Also Bacterio creates an empty folder for each version Figure 32 Later when test cases are executed Bacterio will leave on them some data of the execution 31 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Total mutants inc uoi dec swa 47 418 47 32 216 Mutants Mutant Na dec sudo dec sudo dec sudo dec sudo dec sudo dec sudo dec sudo dec sudo dec sudo Class sudokuso sudokuso sudokuso sudokuso sudokuso sudokuso sudokuso sudokuso sudokuso Method columnas columnas columnas marcarPr hayCerte marcarPr marcarPr marcarPr marcarPr Operator dec dec dec dec dec dec dec dec dec Extra Info
40. cel Figure 21 Advanced options At this point the tool is configured thus let s press the OK button to save the configuration if the Cancel button is clicked all the configuration changes would be undone 3 3 Mutant generation step To generate mutants from the system under test we select Mutation Generation in the menu bar of Figure 17 so the window in Figure 22 appears 24 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 800 Mutation functional testing Configuration Mutation Help Exploratory Testing uA Classpath Apply mutation on a gt gt lt lt High order mutation Do not join mutants of the same operator Do not join mutants the same class C Do not join mutants in the same method 9 One fault version Each choice Pair wise Interface mutation operators Each choice first last Swap parameters Throw exceptions INC DEC NUL Traditional mutation operators Random each choice AOR ROR C ABS C UOI LCR Order MM ing Get versions Mutant sampling Size SSS 100 A lt Mount versions _ Clear previous mutants Show results A Tu Figure 22 Window for mutant generation 3 3 1 Selecting the elements to mutate Now we must select the loca
41. contain one fault each 2 order two faults etc Original system fA B C D 1 A C D M1 A1 C D 1 1 D M1 fA4 B C D1 M2 As B C D M2 A1 Bo D 2 Bj Co D M2 A B1 Do C D Mj A Ba Cp D Figure 12 Some mutants of different orders At first glance dealing with high order mutants decreases the costs of test case execution since there are less mutants but it is relatively easy to discover just one of the n faults inserted in the system The testing process with Bacterio is summarized in the following subsections 2 5 1 Mutant generation In this step Bacterio generates faulty copies of the classes composing the application under test Figure 13 represents a system composed of Z classes After the first step Bacterio gets a set of mutants for all of them 18 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha application SIC classes A class B class A1 class B1 class Z1 class S Step 1 A2 class B2 class Z2 class Mutant generation 77444 Z class lib
42. d in at least a mutated version consider there are 6 pair tables A A B B D and C D Table B D for example contains 12 pairs Di B D Bi Dz Be Di Be Do Bo Bs D Do 05 Ba D1 D2 Ba D3 The idea behind Pair wise is to generate versions visiting all the pairs in all the pair tables Data 29 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha on Figure 27 represent 30 possible combinations of the mutant classes which correspond to 30 possible mutant versions In this case they have been generated with Orderz4 see Figure 26 what means that each mutant version may have up to 4 mutant classes con cos EI Figure 27 Possible combinations with Pair wise and Order 4 25 A5 B1 C3 D1 A5 B1 C4 D1 27 AS BILCSDI A5 B2 C6 D2 3 3 4 Generation of mutant classes and mutant versions For this example we ll leave the options shown in Figure 26 note the Mutant sampling was reduced to 25 With all the mutation operators already selected Figure 24 we can press the Get versions button Bacterio allows us now to do a finest selection of the classes and methods to mutate Figure 28 we ll leave out of the mutation the tests package since it has no business code but only test cases that must not b
43. e 64 Start a remote executor node When all the remote executor nodes are running we must execute the facade node This node is contained in the jar file facadeNode jar When we run this jar file the window of the Figure 65 appears Here we can configurate the facade node AO Pararell Execution System Distributor Server Address Distributor Server Address Distributor Server Address Port fo Algorithms PEDRO Algorithm 0 Distribute versions 1 Distribution Algorithms Distribute test cases 2 Give versions on demand 3 Give test cases on demand 4 Parallel Nodes Unused addresses Used addresses Add gt M lt Remove Selected Addresses Remove Selected Addresses IP Address Add Address Port Address Start Distributor Server Figure 65 Start a facade node 52 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha The first two parameters to configurate are the ip address and port of the facade node Then we must select one of the 5 distribution algorithms implemented in Bacterio 1 PEDRO Algortihm Parallel Execution with Dynamic Ranking and Ordering Algorithm this algorithm distributes workloads dynamically giving in each moment a different amount of load depending in the execution times or the previously workloads ex
44. e la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter 3 5 4 Exploratory lest Cases MAMMA UU 46 vrl MEE VT Ricci ER 50 3 5 6 Exploring the remaining options of Bacterio eene nnns 55 3 5 6 1 Configuration 6 2 00000000000000000000 55 3 5 6 2 Mutant generation 224400000000000000005050000000000 nna nnns 58 3 5 6 8 Mutant Execution windOw eese sese 60 3 5 6 4 Test cases selection 63 3 6 THE MUTATION PROCESS WITH BATERIO REDUCING THE EFFORT OF TESTERS 63 3 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha INDEX OF TABLES TABLE 1 SOURCE CODE REPRESENTATION OF SOME CHANGES INTRODUCED IN A SMALL 27 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha INDEX OF FIGURES FIGURE 1 CODE OF SOME MUTANTS AND THEIR RESULTS WITH SOME TEST DATA
45. e mutated Classes selection to mutated M AIl packages v M sudokusolver dominio gt M Casilla v M Ristras M init OV M unSudoku Ljava lang String gt M Shower gt M Sudoku gt M Utilidades gt M sudokusolver gui gt LIES OK Cancel Figure 28 A fine selection of the classes to mutate We press Ok and after some moments Bacterio says that it has finished of applying the last mutation operator Figure 29 30 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Finished LCR Get versions Figure 29 The generation of class mutants has finished Now we can press the Show results button to see the locations of the changes introduced by Bacterio and what operator has been applied Figure 30 Total mutants Total Mutants ror inc uoi dec swa aor nuf tex abs 2199 997 47 418 47 32 216 23 1 418 Mutants Mutant Name Class Method Operator Extra Info dec sudokusolver dominio Sudoku 26 0 112 sudokusolver dominio Sudoku columnasCubiertast llll Z dec parameter 2 A dec sudokusolver dominio Sudoku 27 0 113 sudokusolver dominio Sudoku columnasCubiertas llll Z dec parameter 3 dec sudokusolver dominio Sudoku 28 0 114 sudokusolver dominio Sudoku columnasCubiertast lllI Z dec parameter 4 dec sudokusolver dominio Sudoku 29 0 115 sudokusolver dominio sudoku ma
46. ected Only this test is selected because the other two test cases have been already executed The Figure 50 shows the results of the execution of the new test case These results are relative to the executed versions and shows that the new test case have kill 5 versions that were alive Test Cases selection M All the tests Y M tests Testl testGoForward _ testGoBack E testFivelnput C Full Tests Execute first with JUnit 9 Version oriented A re A A 1 Test Case oriented OK Cancel Figure 49 New test case selected 43 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 BO uai 1 32 mutants 5 342 versions 5 342 ror 3 149 uai 3 1256 mutants 1 46 versions 1 4556 ror 2 01 Version version95 version94 version9 3 version9 2 version9 1 tests Testl testFivelnput Version tests Testl testFivelnput version9s 32 version94 73 version93 21 version92 24 version91 20 Versions killed by time out Versions killed normally and by time out Versions covered but not killed Versions not covered and no killed Next page 4 334 Goto page Empresa de Base Tecnol gica Universidad de Castilla La Mancha Total killed mutants inc dec 0 4 0 2 abs 0 107 nuf 1 7 aor 0 38 swa 0 3 Percentage inc aor 0 0 0 0 abs 0 056 nuf 14 28 Killing
47. ecuted by the remote executor nodes 2 Distribute versions this algorithm splits the number of selected versions by the number of remote executor nodes and gives to each node all the test cases with a part of the versions 3 Distribute test cases this algorithm splits the number of selected test cases by the number of remote executor nodes and gives to each node all the selected versions with a part of the selected test cases 4 Give versions on demand this algorithm gives to each remote executor node all the selected test cases and one version When a remote executor finishes the facade node gives to that remote executor node another version until all selected versions have been executed 5 Give test cases on demand this algorithm gives to each remote executor node all the selected versions and one test case When a remote executor finishes the facade node gives to that remote executor node another test case until all selected test cases have been executed To configure the facade node we select PEDRO Algorithm The last necessary configuration information is the ip port addresses of the remote executor nodes This addresses can be added with the p address and Port address fields and the add Address button Then with the buttons add gt and Hemove we can select the address that we want to connect Finally to start the face node we click on Start Distributor Server Figure 66 o3 Paseo de la Universidad 4 13071 Ciudad
48. es the test cases against the mutant versions Executing test cases Killing mutants adii Executing mutant versions E 16 Cancel Figure 36 Bacterio trying to execute and kill mutants 35 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 3 5 Result analysis When all test cases have been executed Bacterio shows the results in a window like that in Figure 37 This windows shows relative results of the versions executed for example if we would generate 100 versions but we only execute 50 the results would be calculated taking into account only the 50 executed versions thus if the test would kill the 50 versions this windows would show 10096 of mutation score but for the complete mutation process we would obtain 5096 which is shown in the execution window Figure 33 after this window is closed Total killed mutants versions mutants ror uoi inc dec swa 208 550 208 550 87 236 74 106 8 10 8 11 Percentage versions mutants ror uoi inc dec swa aor 37 81 37 81 36 86 69 81 66 66 80 0 72 72 32 14 6 E Killing matrix Version tests Testl testGoForward tests Testl testGoBack version249 X version248 X version247 O version246 X M Time execution matrix milliseconds Version tests Testl testGoForward tests Testl testGoBack ver
49. estl testGoBack version28 2990 4327 wersionl68 4808 5375 wersion200 1229 version27 7495 9651 167 11349 18318 wersionzb 7 166 1 wersion25 2153 274 wersionl65 223 244 version24 112 Figure 41 Execution time matrix 5 Finally Figure 42 it shows a summary of the results Extra infomation Versions killed by time out 3 Total kills 205 Versions killed normally and by time out 0 Total Time outs 3 Versions covered but not killed 300 Versions not covered and no killed 43 Next page Coto page 55 Figure 42 Summary of results Additionally when we close the results dialog and return to the Execution window Figure 33 Bacterio updates its data now it reports Figure 43 about the completely alive versions those whose mutated statement have not been traversed by test cases the alive but covered versions the killed versions the equivalent versions marked by the tester Available versions Alive versions Alive Covered versions Died versions Equivalent versions Select all Unselectall Select All Unselect Selectall Unselect all version481 E versionl A version115 version91 version10 D version166 0 version92 version1 00 version1 89 version93 version101 version1 90 EE 2 rn live versions 42 Total covered versions 300 Total died versions 208 Total equivalent versions 0
50. eter decrement Nullify and Throw exceptions proceed from or are inspired by the work of Ghosh and Mathur who define a set of interface mutation operators for testing components from a black box point of view o Swap interchanges in an operation the values of two parameters of compatible types by means of a modification of the operation signature thus if a class has the operation foo int a int b int c mutants will be produced interchanging a b and c for example b a c or c a b o Parameter increment INC and decrement DEC proceed from the Twiadle operator They modify the value of numeric parameters respectively incrementing or decrementing the value in one when the method is entered o Nullifier sets the value of an object parameter to null o Throw exceptions is inspired by the Exception coverage criterion defined in the referenced work A test set is adequate with respect to exception coverage if it leads to 100 exception coverage If a method throws n exceptions this operator produces n mutants each one replacing the original code of the method by a single statement which throws each of the n possible exceptions The remaining five operators introduce small changes in the structure of the actual method body AOR arithmetic operator replacement ROR relational operator replacement ABS absolute value insertion LCR logical connector operator and UOI unary operator insertion are classic mutation operator
51. eturn numero public int valor return numero public candidatos Vector vector new Vector for int i 0 i lt prohibidos length i if prohibidos i public candidatos Vector vector new Vector for int i 0 i lt prohibidos length i if Iprohibidosli vector add Integer valueOf i 1 B vector add Integer valueOf i 1 v Mark as equivalent Figure 52 Decompiled code of version517 In this case this version must be mark as equivalent which can be done simply making click in Mark as equivalent check button in the bottom side of the window showed in Figure 52 Then the mutant is mark as equivalent and the execution window is updated Figure 53 45 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Available versions Alive versions Alive Covered versions Died versions Equivalent versions a Unselectall _ SSS _ n N A N EN 9 P Selectall Unselectall Select Unselect Selectall T y AA ESA AP las _ jversion509 A jversion517 5 1 version5 16 5 18 versionS 19 4 reali versions 38 Total covered versions 298 Total died versions 213 Total equivalent versions
52. gure 33 Mutant execution window The first thing to do now is to load the mutant versions For this we press the Load button on the right side of the window and the Alive versions list is filled in Figure 34 This list is loaded because in the configuration window Figure 19 we specified that the versions would be saved on Desktop versions 33 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha OOO Configuration Mutation Help Exploratory Testing functional 1g Main class sudokusolver dominio Casilla class Available versions Alive versions Alive Covered versions Selectall Unselectall Select All Unselect All Selectall Unselectall Jversion100 version101 beta live versions 550 Total covered versions 0 Total versions results ELA Reset Test Cases Died versions Equivalent versions Execution options Selected versions e Test case file Captured test 1f Execute original Total died versions 0 Total equivalent versions 0 dec 0 096 uoi 0 096 inc 0 0 mutants 0 0 ror 0 0 versions 0 0 abs 0 096 tex 0 0 nuf 0 096 aor 0 0 swa 0 0 21 Execute Mutants View Results 3f C Console activate View Previous Execution Results
53. he acquisition of a license Please contact http www alarcosqualitycenter com index php productos bacterio to get a license 2 4 Third party software Bacterio uses the following third party software 2 4 1 ASM ASM is an all purpose Java bytecode manipulation and analysis framework It can be used to modify existing classes or dynamically generate classes directly in binary form Provided common transformations and analysis algorithms allow to easily assemble custom 15 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha complex transformations and code analysis tools ASM offer similar functionality as other bytecode frameworks but it is focused on simplicity of use and performance Because it was designed and implemented to be as small and as fast as possible it makes it very attractive for using in dynamic systems tc 2000 2005 INRIA France Telecom All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met 1 Redistributions of source code must retain the above copyrignt notice this list of conditions and the following disclaimer 2 Redistributions in binary form must reproduce the above Cp ELE notice this list of conditions and the following disclaimer in the documentation and or o
54. hreshold reached Remove equivalent mutants Eliminate ineffective Figure 3 A possible mutation testing process 1 5 Mutation types Test cases can find the faults inserted in the mutants in several ways First of all let s remind that in general the structure of a test case consists in 1 the execution of a set of steps to put the SUT in the required initial state 2 execution of services on the SUT to put it in the expected state 3 comparison of the actual obtained state with the expected one If the expected and the obtained states differ then the test case has found an error on the SUT Suppose the code in the left side of Figure 4 it is the getType function included in a 10 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha TriTyp class and determines the type of a triangle depending on the lengths of its sides which is tested by the test cases in the right side These ones are JUnit test cases Each one builds an instance of the class under test TriTyp executes a set of services on this instance setl setJ seik try to lead the instance into a given state call to get ype and finally includes an oracle to determine whether the instance of the CUT has or has not the expected behaviour for example the testEOUILATERAL test case checks whether the TriTyp progr
55. inspection Hashcode fields inspection toString if defined otherwise hashcode fields inspection Depth in field inspection 3 Servers configuration Local execution Local IP 192 168 0 3 Local Port 12347 Server IP 192 168 0 2 Server Port 12346 Captured Tests Folder Users Pedro Documents WSBacterioForWin CapturedTests Kind of mutation 9 Flexible Weak Mutation BB Weak 1 Mutation BB Weak N Mutation Strong Mutation O Functional Qualification Execution Cost Instrument mutants to get number of source code lines ex Parallel execution nf proc 1 8 7 Stop the execution between test cases Basic options Cancel Figure 67 Local node configured to execute in parallel 3 5 6 Exploring the remaining options of Bacterio This section briefly describes all the options in Bacterio 3 5 6 1 Configuration windows Version folder Pedro Documents WSBacterioForWin sudokuManual2 Select Test cases folder uments workspaceForExperiments SudokuManual bin Timeout 20000 2 Avanced Options Figure 68 Basic configuration window Version folder Path of a folder to store temporal files e Test cases folder Path where the class files of the test cases 55 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Timeout Maximum time that a m
56. ion166 c Alive mutants that furthermore have not been traversed by test cases have no mark version28 for example d With respect to other mutation tools Bacterio includes a useful novelty covered mutants remaining alive this is mutants whose mutated statement has been traversed by the test case but with no discovering the change are marked with This is the case of version167 Killing matrix Version tests Testl testGoForward tests Testl testGoBack version2 8 versionl68 O version2 00 X version 7 versionl6b7 versionz6 X versionl66 X T version25 O versionlb5 0 O versionz4 X Figure 40 Killing matrix In the previous figure version200 was killed by testGoForward and there is no mark under testGoBack with the execution algorithm we left selected as default Versions oriented Figure 35 if a test case kills a version then no more test cases are executed against it remind the Killing matrix section in jError No se encuentra el origen de la referencia Therefore the Versions oriented test case execution algorithm fills in the killing matrix by rows 37 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 4 n Figure 41 the times of execution of each test case against each version Time execution matrix milliseconds Version tests Testl testGoForward tests T
57. is An important difference of Bacterio with respect to other mutation tools is that Bacterio generates in just a single step all the mutants for all the classes in the SUT Thus if a system is composed by four classes A B C and D Bacterio builds mutants for all of them Suppose it builds P mutants from A Q from B R from C and S from D Left side of Figure 11 shows these sets of mutants grouped by the corresponding original class right side shows the actual set of mutants that Bacterio considers this is all the class mutants together 17 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Original system A C Ma fA Az Ap Mp B1 Bz x Boj M A1 Ap B1 Bz on Ba 1 C2 np CR Mc Ci Cs CR Do Ds Figure 11 Mutants generated by Bacterio for a supposed application As Bacterio is focused on the testing of complete systems another difference is also in the way of composing the mutated versions Other tools put their focus on the class under test whilst Bacterio puts it on the complete system under test Thus what Bacterio makes is to build mutant versions of the complete system each mutated version contains at least a mutant class although may have more The order of a mutant references the number of faults it contains Figure 12 illustrates this idea 1 order mutants
58. ith the original versions There are five options o Flexible Weak Mutation The mutants are compared with the original version at the beginning and at the end the each method and the execution is not stopped until a difference is found o BB Weak 1 Mutation The mutants are compared with the original version after the execution of the basic block of code with the mutation and the execution is stopped in that point o BB Weak N Mutation The mutants are compared with the original version after the execution of the basic block of code with the mutation and the execution is stopped in that point In the case that the basic block in into a loop the execution is not stopped until the end of the loop o String mutation The mutants are compared with the original version at the end of the execution 97 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter o Functional Qualification The mutants are compared with the original version at the end of the execution only taking into account the test verdicts pass or fail Execution costs This option can be selected to instrument the system under test in order to obtain the number of execute lines by the test cases e Parallel execution n processes This option is used to determine how many process are launched in parallel in the same computer to run test cases This is
59. iversidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter o toString Field inspection Using the toString declared methods and for the objects that do not have a declared toString inspecting the fields of the object o Hash code fields inspection Calculating a hash code with the values of the fields of the objects o toString Hash code Using the toString declared methods and for the objects that do not have a declared toString calculating a hash code with the values of the fields of the object Depth in field inspection Number to references to a complex object used when field inspection or hash code field inspections is used This parameter solves the problem due to loops in the references for example Object1 field Object2 Oject2 field Object3 and Object3 field Object1 Local execution Indicates if the system is configured to execute in remote parallel or in local Local IP of the local computer Used in remote parallel execution Local Port Port in the local machine Used in remote parallel execution Server IP Ip of the server executor Used in remote parallel execution Server Port Port in the server executor Used in remote parallel execution Capture tests folder Path where the captured test files are stored Kind of mutation Mutation technique used to determine when a mutant version must be compared w
60. l IP 161 67 106 60 Local Port 33345 Server IP 161 67 106 6C Server Port 33344 n Captured Tests Folder Users Pedro Documents WSBacterioForWin CapturedTests Select Kind of mutation 9 Flexible Weak Mutation O BB Weak 1 Mutation BB Weak N Mutation O Strong Mutation Functional Qualification Execution Cost Instrument mutants to get number of source code lines ex Parallel execution nf proc 1 A Stop the execution between test cases Basic options Cancel Figure 54 Configuration for exploratory execution When Bacterio is properly configured we can create an exploratory test case selecting Exploratory Testing gt Capture in the main windows Figure 55 Then the capture window appears _ Configuration Mutation Help i Capture Main class sudokusolver d lass Figure 55 Go to Capture exploratory tests window To create an exploratory test case we must click on Load button to load all the classes of the system in the combobox Then we select the main class in the combobox JFSudoku class in our case and click on Start Capturing button Figure 56 47 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Mutation functional testing Configuration Mutation Help Exploratory Testing M
61. matrix Extra infomation Total kills 4 Total Time outs 1 0 to 34 Figure 50 Results of the new test case When the results window is closed the execution window is updated with the new results The killed versions are moved to the killed versions list and the covered and not killed versions are moved to the covered versions list Also the total mutation scores are updated adding the new killed versions Figure 51 Alive versions Alive Covered versions Selectall Unselectall Select All 22 5 Jversion4 56 Jversion95 version467 version96 version468 version97 version469 version98 version99 Died versions Equivalent versions Unselect All Select all 3 Unselect all y E jwecriomiiu 5 version80 version9 version91 version92 version94 d version481 Total versions results versions mutants ror inc uoi dec swa aor nuf abs 38 72 38 72 38 13 66 66 70 75 80 0 72 72 32 14 40 0 1 83 z zz Total covered versions 299 Total died versions 213 Total equivalent versions 0 Figure 51 Total results updated 44 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 3 5 3 Detection of equivalent versions In the process to enrich the test suite with more test cases it is possible tha
62. n e Clear previous mutants Button used to remove all the temporal files mutants versions log files excepting results files High order mutation C Do not join mutants of the same operator Do not join mutants in the same class Do not join mutants in the same method 9 One fault version Each choice Pair wise Each choice first last Random each choice Order 2 Mutant sampling Size Se 100 _ M Mount versions _ Figure 73 Generation mutants window Versions generation e High order mutation Technique implemented in Bacterio to create mutant versions with more than one fault This technique has several options Do not join mutants of the same operator Restriction applied during the combination of mutants that implies that two mutants created with the same mutation operator cannot be joint Donot join mutants in the same Restriction applied during the combination of mutants that implies that two mutants which are in the same class cannot be joint Do not join mutants of the same operator Restriction applied during the combination of mutants that implies that two mutants which are in the same method cannot be joint Each choice Option to select the each choice algorithm to combine mutants Each choice first last Option to select the each choice first last algorithm to combine mutants Each choice random Option to select the each choice random algorithm to combi
63. n scores Thus in addition to apply the special treatment Bacterio calculates the risk of the random test cases detected and before the execution of the tests Bacterio asks to the user if the test should be run or not with a window like Figure 46 Suppose that we create a random test case in the example testRandom and we execute it Figure 45 e nm Test Cases selection All the tests Y M tests Testl testGoForward testGoBack testFivelnput V testRandom Full Tests Execute first with JUnit Version oriented oy A O Test Case oriented OK Cancel Figure 45 Executing a random test 39 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha OOO Attention Randomness found in the selected test cases The next test cases have been found with randomness Please deselect the test cases that you don not want to execute The randomness in tests can produce variation in the mutation score so two exections of the same mutants with the same tests can produce two different mutation scores Also randomless can produce that a test kills all the mutants lt 5 randomless Low risk 5 10 randomless Acceptable risk M All the acceptable risk tests 10 50 randomness High risk M All the acceptable risk tests gt 50 randomness No accepta
64. ne mutants Pairwise Option to select the pairwise algorithm to combine mutants 59 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter o One fault version Option to create versions with only one fault This option must be selected if we want not use the high order mutation technique o Order Number of mutants that will be insert into each version Mutant sampling Technique implemented in Bacterio to create versions with a randomly selected sample of the total set of mutants This technique has one option o Size Size of the sample Show results _ an Figure 74 Generation mutants window Show results button Show results Button used to show the generated mutants and versions 3 5 6 3 Mutant Execution window Figure 75 Execution mutants window Load button Load This button is used to load all the versions generated and the executions results obtained previously Available versions Alive versions Alive Covered versions Died versions Equivalent versions 4 E Unselect all Select All A _ Unselect All e l s a MEUSE gt Select all Selectall Unselectall T A H O ad Figure 76 Execution mutants window Available versions Avail
65. op If it is execution is stopped after the execution of the loop At the point where the execution is stopped the states of 12 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha the mutant and of the original system are compared Recently we have proposed Flexible Weak Mutation as a new technique for applying mutation at system level With this mutation type an execution engines almost continuously looks for differences between the original and the mutant systems Bacterio is the first tool supporting Flexible Weak Mutation although it also supports otrong Mutation Weak Mutation and Functional Qualification 1 4 Killing matrixes Supposing a tool that executes all test cases against all the mutants the final result could be seen as a killing matrix Let be P a program for which a mutation tool generates 7 mutants Figure 6 shows the possible Killing Matrix cells containing X denote that the corresponding test case has found the mutant in the row tc6 for example has found the faults inserted on m3 to m7 whereas tc4 has not found any fault and could be removed from the test suite Note moreover that tc3 kills a subset of mutants killed by tc6 and that tc2 kills the same mutants that tc1 and tc5 Therefore the original test suite composed by 6 test cases could be reduced to a new test suite containing only 2
66. presa de Base Tecnol gica Universidad de Castilla La Mancha Execution options 2 Selected versions J Test case file Captured test Figure 60 Capture test button selected BOM Do you want to continue Y During the execution of captured tests you must not move the mouse or press any key of the keyboard because the process can fail and can produce damage in your system Cancelar Figure 61 Warning message before to execute a exploratory test ADO Test Cases selection M All the tests Captured Test Cases M ExploratoryTestCasel A Full Tests Execute first with JUnit 9 Version oriented 7 A E B _ Test Case oriented OK Cancel Figure 62 Exploratory test selected 3 5 5 Parallel Execution Bacterio can run tests and mutants in parallel The architecture used by Bacterio is in Figure 63 It is composed by three kinds of nodes 1 Local node 2 Remote executor node and 3 Parallel facade node 90 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Local Node Facade node Remote executor nodes cabs Figure 63 Parallel execution architecture Each node of the architecture can be running in a different computer In the local node the user interacts with Bacterio to execute
67. presa de Base Tecnol gica Universidad de Castilla La Mancha The decompiler reads in class files and produces something similar to the original java file Of course this can t be perfect There is no way to produce the comments or the names of local variables except when compiled with debugging JODE is Copyright O 1998 2000 by Jochen Hoenicke This program is free software you can redistribute it and or modify it under the terms of the GNU General Public License as published by the Free Software Foundation either version 2 of the License or at your option any later version You can redistribute some of the packages under the terms of the of the GNU Lesser General Public License as published by the Free Software Foundation See the copyright headers in the Source code This program is distributed in the hope that it will be useful but without any warranty without even the implied warranty of merchantability or fitness for a particular purpose See the GNU General Public License for more details Figure 10 JODE license 2 4 3 A DOOM image In order to animate the test case execution window Bacterio makes use of an animated gif from famous the DOOM2 computer game which is under GNU GPL license 2 5 Thetesting process with Bacterio an overview Basically the testing process with Bacterio is guite similar to that described on Chapter 1 in fact it consists of a step of mutant generation test case execution and result analys
68. rcarProhibidosPorFila lDV parameter 1 dec sudokusolver dominio Sudoku 2 0 88 sudokusolver dominio Sudoku hayCerteza lll Z dec parameter 2 dec sudoakusolver dominio Sudoku 30 0 116 sudnkusnlver dominio Sudoku marcarProhihidosPorFila IhWV dec narameter 2 M Versions Version Mutantl VEISIUTTD 1 PUL SUUURUSUIVE UULU 31 1 is version5 42 ror sudokusolver dominio Casilla 54 4 420 version5 43 ror sudokusolver dominio Casilla 58 3 424 version544 ror sudokusolver dominio Casilla 7 2 373 version545 uoi sudokusolver dominio Casilla 11 0 1792 version546 uoi sudokusolver dominio Casilla 12 0 1793 Total mutants Total Mutants ror inc uoi dec swa aor nuf tex abs 2199 997 47 418 47 32 216 23 1 418 Mutants Mutant Name Class Method Operator Extra Info dec sudokusolver dominio Sudoku 26 0 112 sudokusolver dominio Sudoku columnasCubiertas llll Z dec parameter 2 dec sudokusolver dominio Sudoku 27 0 113 sudokusolver dominio Sudoku columnasCubiertas IIII Z dec parameter 3 dec sudokusolver dominio Sudoku 28 0 114 sudokusolver dominio Sudoku columnasCubiertas llll Z dec parameter 4 dec sudokusolver dominio Sudoku 29 0 115 sudokusolver dominio sudoku marcarProhibidosPorFila lV parameter 1 dec sudokusolver dominio Sudoku 2 0 88 sudokusolver dominio Sudoku hayCerteza lll Z dec parameter 2 der sudokusolver dominio Sudoku 30 0 116 sudnkusnlver dominin Sudoku marcarProhihidosPorFila INV dec narameter 2 M Figure 30 Lo
69. s and are among the most significant and valuable operators according to several studies on selective mutation Table 1 shows the changes introduced by these operators in a small fragment of Java Code In order to make the example more illustrative the table shows the Java source code translation corresponding to the actual changes introduced in the bytecode mutants 26 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha Original code int div int a int b throws NaNException ByZeroException if 0 amp amp b 0 throw new NANException if b 0 throw new ByZeroException return a b Changes introduced int div int b int a int div int a int b throws int div int a int b throws DEC i return a b Nullifier int div int a int b throws int div int a int b throws null null Not applicable to int type if It is just an example if 4 return a b return a b j Throw int div int a int b throws nt div int a int b throws exceptions throw new NaNException throw new ByZeroException j int div int int b int div int a int b throws f int div int a int b throws throws if A if return a b return a b return a b int div int a int b throws if a
70. sion249 831 version248 168 version247 231 242 version246 67 Extra infomation Versions killed by time out 3 Total kills 205 Versions killed normally and by time out 0 Total Time outs 3 Versions covered but not killed 300 Versions not covered and no killed 43 _ TS TS N gt Previous Next Goto page 55 Figure 37 Results of the execution of the test cases That window reports on 1 In Figure 38 the total number of killed versions with respect to the number of versions executed as well as the number of mutants proceeding from each mutation operator Total killed mutants versions mutants ror uoi inc dec SWA aor 208 550 208 550 87 236 74 106 8 12 8 10 8 11 18 56 Figure 38 Total number of killed 36 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 2 In Figure 39 the percentages of killed versions Percentage versions mutants ror uoi inc dec Siva aor mut abs 37 81 37 81 36 86 69 81 66 66 80 0 72 7296 32 14 30 0 1 83 Figure 39 Percentage of killed 3 In Figure 40 the killing matrix this is what test cases have killed what versions a Killed mutants are marked with X for example version200 is killed by the testGoForward test case b Killed mutants by Timeout Figure 18 are marked with X T such as vers
71. t case corresponding to the test data 7 7 produces different outputs on the original program whose output is correct and on Mutant 1 thus this test case has found the fault introduced in the mutant and it is said that the mutant is killed moreover this test case is good On the other side since all test cases offer the same output on the original program and on Mutant 4 it is said that Mutant 4 is a ive Moreover this mutant will never be killed by any test case since variable b is incremented after returning the result Mutants like this one are called functionally equivalent mutants and may be considered as noise in the result analysis step since they represent obstacles in knowing the quality of the test suite they have a fault actually a syntactic change with respect to the original source code that cannot be found though Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha int sum int a int b Original return a 4 b Test data a b mi 2 o 47 Wai 0 0 Muente 1 9 0 Wu 2 0 3 sum int a int b Mutant 4 return a b b Results with some test data sum int a int b Mutant 1 return a b T sum int a int b Mutant 2 return a b versions sum int a int b Mutant 3 return a b a Code some mutants Figure 1 Code of some m
72. t some versions cannot be killed because they are equivalent mutants For example the Figure 52 shows the decompiled code of the version517 In the mutated code the absolute value function has been applied to the variable of the method prohibidos of Casilla class before use this variable in a mathematic operation However as the code shows this variable will never have a negative value thus the absolute value function will not cause a different behavior of the system in any case thus this version is equivalent to the original system eoe Source comparison Original source code Mutant source code public void addProhibidotint i if i 0 amp amp numero 0 prohibidos i 1 true public void addProhibidotint i if i 0 amp amp numero 0 prohibidos i 1 true public int numeroDeProhibidos public int numeroDeProhibidos 1 int i 0 int i 0 for inti 0 0 10 lt prohibidos length i 0 for inti 0 0 10 lt prohibidos length i 0 i i prohibidosli 0 1 0 i prohibidosli 0 1 0 return i return i public prohibidos public prohibidos 1 int is new int prohibidos length int is new int prohibidos length for int i 0 i lt prohibidos length i for int i 0 i lt prohibidos length i if prohibidos i if prohibidos i E Too else else isi 0 isi 0 return is return is public int valor 1 r
73. tants Test cases Figure 7 Killing matrix filled in by rows If the matrix is filled in by columns then each test case is executed against all the mutants remaining alive tc against m1 which is removed since it is killed tc against m2 etc Figure 8 but the final result is the same Test cases Figure 8 Killing matrix filled in by columns 14 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 2 AN OVERVIEW OF BACTERIO 2 2 Bacterio in a nutshell Bacterio is a tool for testing Java applications based on mutation Mutation is applied at bytecode level Bacterio supports Strong mutation BB Weak 1 BB Weak N Functional qualification and Flexible weak mutation This document explains how to test a system by means of Bacterio To illustrate it we will use a small Sudoku Solver system which runs on a single local machine You can download it from here 2 3 Licensing The tool has been developed by Pedro Reales and Macario Polo Alarcos Research Group University of Castilla La Mancha Spain and is commercialized and distributed by Alarcos Quality Center http www alarcosqualitycenter com Research and education institutions can ask for a free license of Bacterio in this case the tool is strictly licensed only for research and educational purposes Any other kind of use requires t
74. ted 62 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha i _ 8 Reset Test Cases Figure 81 Execution mutants window Reset Test Cases button Reset Test Cases Button used to reset the execution resutls of a set of tests cases When the buttons is clicked a windows apperas where we can select the test cases that we want to reset x s View Previous Execution Results A Figure 82 Execution mutants window View Previous Execution Results button View Previous Execution Results Button used to see previous executions that are store in the temporal forder configured 3 5 6 4 Test cases selection window Test Cases selection All the tests tests Testl M testGoForward M testGoBack M testFivelnput Full Tests Execute first with JUnit 9 Version oriented A rh 2 Test Case oriented OK Cancel Figure 83 Test cases selection window Full tests This option is used to fill the killing matrix completely Version oriented This option is used to fill the killing matrix by rows Test case oriented This option is used to fill the killing matrix by columns Execute first with JUnit This option is used to execute first the test cases and the original version with the JUnit framework in order to see if the test find errors
75. test cases Test cases Figure 6 Killing matrix for a supposed program and a supposed test suite Obviously the construction of the complete killing matrix requires the execution of all the test cases against all the mutants what may require more execution time that the execution only on those mutants remaining alive which is the proposal depicted in Figure 3 However some times the complete execution can be interesting in order to do further reductions of the test suite such as in this example when the system evolves in the future only 2 7 14 executions will be required for complete regression test 13 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha enter When killed mutants are no longer considered by the next test cases the killing matrix can be filled in in two ways more at least by rows or by columns If we fill in the matrix by rows then tc7 is executed against m1 As this mutant is killed then m1 is removed from the suite and the process continues executing tc on m2 which is also killed Then tc1 is executed against m3 which remains alive and forces the execution of tc2 against m3 With this process the killing matrix remains as in Figure 7 The test suite will consist of tc1 tc2 and tc6 Coloured cells represent executions that are not performed since correspond to previously killed mu
76. that field 23 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha publicvoid testl publicvoid testl for int 1 0 1 lt 100 1 for int 1 0 1 lt 100 1 System out println i System osut printlin l Figure 20 The mutant produces an infinite loop The test cases in the Test1 class file are JUnit test cases and are suitable to be executed with Functional Qualification Then let s press the Advanced options button and let s select Functional Qualification as our Kind of mutation Figure 21 Configuration F BvteCode modification in runtime Comparison Method e toString Fields inspection toString if defined otherwise fields inspection Hashcode fields inspection toString if defined otherwise hashcode fields inspection Depth in field inspection 2 Servers configuration M Local execution Local IP 127 0 0 1 Local Port Server IP 127 0 0 1 Server Port 651 Captured Tests Folder c capturedTests Select Kind of mutation Flexible Weak Mutation BB Weak 1 Mutation BB Weak N Mutation Q Strong Mutation Q Functional Qualification Execution Cost Instrument mutants to get number of source code lines ex Parallel execution nf proc 1 Stop the execution between test cases Basic options A A Can
77. ther materials provided with the distribution 3 Neither the name of the copyright holders nor the names of ILS contributors may be used to endorse or promote products derived from this software without specific prior written permission THIS SOFTWARE I9 PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR PARTICULAR PURPOSE ARE DISCLAIMED IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT INDIRECT INCIDENTAL SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES INCLUDING NOT LIMITED TO PROCUREMENT OF SUBSTITUIE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE EVEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Figure 9 ASM Project License 2 4 2 JODE Java Optimize and Decompile Environment JODE is a java package containing a decompiler and an optimizer for java This package is freely available under the GNU GPL New The bytecode package and the core decompiler is now under GNU Lesser General Public License so you can integrate it in your project 16 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Em
78. tion of the system under test In our case all the required files are on Users Maco Desktop SudokuSolver bin We can add this path by pressing the button labelled left side of Figure 23 If the system is dispersed on several folders requires libraries etc they should be added also pressing the button Then the tester must select the folder containing the class files which will be mutated For this s he selects the location in the list of the left side and passes it to the right side pressing the gt gt button right side of Figure 23 eoo Mutation functic Configuration Mutation Help Exploratory Testing Apply mutation on lt Classpath quu JUsers Maco Desktop SudokuSolver bin Users Maco Desktop SudokuSolver bin 6 lt lt AC Figure 23 Selecting the SUT location left and selection of the location of the c ass files to be mutated right Note that the left list Figure 23 left should include the elements that compose the classpath necessary to run the system correctly jar files and folders which contain class 25 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354 Empresa de Base Tecnol gica Universidad de Castilla La Mancha 3 3 2 Selection of mutation operators Now the tester needs to select the mutation operators to be applied Bacterio implements 10 mutation operators e Five Swap Parameter increment Param
79. utant version can be executing After this time the version is considered dead This parameter is used to stop infinite loop produced by the mutations Configuration Y ByteCode modification in runtime Comparison Method 9 toString C Fields inspection toString if defined otherwise fields inspection 2 Hashcode fields inspection C toString if defined otherwise hashcode fields inspection Depth in field inspection Servers configuration Local execution Local 161 67 106 60 Local Port 33345 Server IP 161 67 106 60 Server Port 33344 Captured Tests Folder Users Pedro Documents WsSBacterioForWin CapturedTests Select Kind of mutation Flexible Weak Mutation C BB Weak 1 Mutation O BB Weak N Mutation Strong Mutation Functional Qualification Execution Cost Instrument mutants to get number of source code lines ex Parallel execution nf proc 1 4 gt Stop the execution between test cases a Basic options a ok Cancel Figure 69 Advanced configuration window Comparison Method Method used to determine the outputs of the system and for detecting the killed mutants There are four comparison methods They may have influence on the execution time The quickest comparison method is Hash code fields inspection o toString Using the toString declared methods o Field inspection Inspecting the fields of the objects 96 Paseo de la Un
80. utants and their results with some test data In general the process of mutation testing has three stages 1 Mutant generation By means of a tool the tester generates a set of mutants Usually each mutant contains a simple change wich is normally a fault Depending on the mutation operators used the number of equivalent mutants may vary but can be around 20 of the total 2 Test case execution Also with a tool the tester executes the test cases against the mutants and the original program 3 Result analysis The tester evaluates the Mutation Score that the test suite reaches on the program under test The Mutation Score gives a measure of the test suite quality Its value depends on the number of mutants killed the total number of mutants and on the number of equivalent mutants Figure 2 A test suite is mutation adequate for a program when it reaches 10096 as mutation Score P program under test K T test suite where number of killed mutants AM Y M number of generated mutants E number of equivalent mutants Figure 2 Mutation Score Figure 3 summarizes a possible testing process based on mutation starting from a program P and a test suite 7 the tester executes 7 on order to find faults If 7 finds faults on P then P must be fixed until no errors are found When 7 does not find any fault 9 Paseo de la Universidad 4 13071 Ciudad Real Telf 34 926 295 300 Fax 34 926 295 354

bacterio mutation test system user manual

Contents

Download Pdf Manuals

Related Search

Related Contents