user manual
Contents
1. Te st TESTAR Quick Manual Sebastian Bauersfeld and Tanja E J Vos UNIVERSITAT POLITECNICA DE VALENCIA Centro de Investigacion en M todos de Produccion de Software Copyright 2014 FITTEST Consortium 0 REQUIREMENTS 1 INSTALLING TESTAR 2 STARTING TESTAR 3 SETTING UP YOUR TESTS 3 1 GENERAL SETTINGS TAB 3 2 FILTER TAB 3 3 SPECIFYING SIMPLE ORACLES 3 4 TIME SETTINGS TAB 3 5 MISCELLANEOUS TAB 3 6 SPECIFYING ADVANCED ORACLES 4 RUNNING THE TOOL 4 1 ACTION VISUALIZATION IN SPY MODE 4 2 TEST SEQUENCE GENERATION AND EXECUTION 5 VIEWING AND REPLAYING THE RESULTS APPENDIX Copyright 2014 FITTEST Consortium SEVENTH FRAMEWORK PROGRAMME 0 Requirements The current version of TESTAR runs on Windows 7 64 bit To ensure that TESTAR runs on your system you need to install the Java Development Kit JDK in version 1 6 or above 1 Installing TESTAR TESTAR comes in a file called testar zip which contains all files necessary for its execution Just unzip this file into a directory with write access There is no setup routine 2 Starting TESTAR Within the main directory you will find several files which are crucial for the execution of TESTAR The most important one is testar bat Double click on it and TESTAR will start and display the main screen TESTAR pro pm ME taa Spy M nerate About General Settings Oracle Filters T2. These might popup in the foreground and block the SUTs GUI They might also consume excessive memory etc TESTAR will kill any process whose name matches the given regular expression Copyright 2014 FITTEST Consortium 7 36 A 3 3 Specifying simple Oracles In order to detect faults you need to tell TESTAR what to look for In the Oracle tab you can specify a simple oracle which analyzes each state of the GUI and reports errors The screenshot below shows TESTAR s oracle settings TESTAR Spy A enerate About General Settings Oracle Filters Time Settings Misc Suspicious Titles p C NullPointer Pplroblem Oo ccurred pP roblem lEejrror Freeze Time 1 seconds 1 Suspicious Titles In this text box you can enter a regular expression that describes those messages that you consider to be related to possible errors TESTAR will apply this expression to each title of each widget on the screen If it matches any widget s title TESTAR will report an error and save the sequence for later inspection For example Imagine you are looking for a critical message box with the title A NullPointerException Exception has been thrown You Copyright 2014 FITTEST Consortium A AAA _ SEVENTH FRAMEWORK PROGRAMME could simply add the expression NullPointerException which will match any title that contains the word NullPointerException the
3. are placeholders for arbitrary characters To learn more about Regular Expressions see http en wikipedia org wiki Regular Expression Freeze Time TESTAR is able to detect crashes automatically because 1t realizes when the SUT is not running anymore However if the SUT does not really crash but just freezes is unresponsive for a long time then TESTAR does not know whether it is just carrying out heavy computations or hangs If the SUT 1s unresponsive for more than the given amount of seconds TESTAR will consider it to be crashed and mark the current sequence as erroneous 3 4 Time Settings Tab The time Settings tabs allows the user to configure the following About General Settings Oracle Filters Time Settings Misc Action Duration 0 5 seconds Action Wait Time 0 5 seconds SUT Startup Time 6 5 seconds Max Test Time 31 536 000 seconds Use Recorded Action Timing during Replay L Copyright 2014 FITTEST Consortium A AAA _ SEVENTH FRAMEWORK PROGRAMME 1 Action Duration The higher this value the longer the execution of actions will take Mouse movements and typing become slower so that it 1s easier to follow what TESTAR is doing This can be useful during Replay Mode in order to replay a recorded sequence with less speed to better understand a fault 2 Time to wait after execution of an action This is the time that TESTAR pauses after havi
4. etc 6 It can be difficult to verify whether TESTAR will do what you told him to and thus you have to test your settings Therefore you may use the View mode in order to inspect the set of actions that TESTAR will execute later on hit Shift 1 to see the generated actions or you can switch to the Generate Debug mode using Shift Left Arrow Right Arrow Copyright 2014 FITTEST Consortium 7 36 A Generally 1t will take some time until you have everything set up for a full test Make sure that you experiment with your settings in the View Mode and look for potential problems that might arise Take a look at the appendix which lists helpful keyboard shortcuts e g stopping test generation etc that you can use once a test is running 5 Viewing and replaying the results During test generation in the Generate Mode TESTAR will save erroneous suspicious sequences into the output directory Once TESTAR has finished a test you might want to inspect those generated sequences to better understand faults or problems There are basically two ways to do that 1 Replay the sequence Of course this is the preferred way since it shows you directly what TESTAR has generated When you start TESTAR you can click on the Replay Mode Button and TESTAR will ask you for the file to replay Just select a sequence file of your choice and hit ok and TESTAR will try to replay it You can even slow down the sequence in case it was recorded at
5. a high speed just lower the value for action duration However certain sequences might not be reproducible because the SUT 1s not in the correct starting state have you considered deleting restoring settings files or the sequence was recorded to fast decrease the value of action duration and action wait time during sequence generation In that case you can 2 View Screenshots of the Generated Sequence Just hit the button for the View Mode in the RU s start screen Again TESTAR will ask you for a file of a recorded sequence Browse to the location of the file and hit ok Now you will enter the RU s View Mode which will show you screenshots of each state that the SUT has been in during sequence generation Just keep hitting Next to proceed to the next step The View Mode will always work and it allows you to see what happened on the screen during sequence generation which can be very helpful in case you are unable to replay a previously recorded sequence Copyright 2014 FITTEST Consortium 7 36 A APPENDIX Keyboard Shortcuts Within the various modes TESTAR accepts several shortcuts Shortcut Effect Modes Shift Arrow Down Quit Monkey Generate Spy Shift Arrow Up save snapshot of current Generate Spy Shift Arrow Left Switch Mode Generate Spy E onartt 2 Toggle visualization of Spy PTT midget mager cursor o Toggle widget information Eq RE Directories temp Temporary files such
6. the GUI enough time to react your generated sequences will still find faults However these faults might be more exotic a human user might not be able to trigger them at all and are usually very hard to reproduce since the timing aspect plays an important factor See section 3 4 to learn how to set these settings 3 Make sure that the SUT always starts in the same initial state This 1s very important to guarantee reproducibility The large majority of SUTs remembers specific settings or saves the position of its windows as they have been during the last session If you do not restore the SUT s settings to their defaults a previously recorded sequence might not be replayed properly simply because the SUT starts in a different states during sequence generation and sequence replay e g starts already with the last edited document opened You can use the settings in the misc tab to delete or restore the SUT s settings files 4 Define your oracle TESTAR automatically detects certain faults such as crashes However you might want to look for critical error messages or low responsiveness Sections 3 3 and 3 6 describe how to set up oracles that help you find certain types of faults 5 Stopping criteria Depending on how long you want TESTAR to run you have to adjust your stopping criteria You might want to runt he RU for 5 hours then you can use the Maximum Time setting in the General Settings tab or have it generate 1000 sequences
7. as the currently recorded output TESTAR outputs log files state snapshots and generated sequences into this directory resources If you have files for your SUT that need to be restored through copying you can put them in here Copyright 2014 FITTEST Consortium SEVENTH FRAMEWORK PROGRAMME
8. de When in Spy Mode TESTAR displays the detected actions Shift 1 to toggle visualization Each action type has a specific appearance as described in the following table Left click Yellow circle Right click Left double click Drag amp Drop Operation Blue Text Click into text field and type the displayed text Copyright 2014 FITTEST Consortium _ SEVENTH FRAMEWORK OGRAMME 4 2 Test sequence generation and execution Before running a complete test in the Generate Mode you have to pay attention to a few things 1 Define the set of actions that you want TESTAR to execute on your SUT Although you want to setup a thorough test that fully stresses your SUT you might want to spare out certain actions such as printing documents terminating the SUT which might be detected as a crash by the oracle or minimizing it etc You might also want to only test a specific subset of all actions because you suspect that faults in specific dialogs of the GUI are more likely to be triggered You can use the ClickFilter in the Filters tab to exclude actions on particular widgets see section 3 2 for further details 2 Define startup time action duration or wait time after actions Those time settings might be important for your test since they influence the reproducibility of sequences If you start sequence generation to early before the SUT has been fully loaded or execute actions too fast do not give
9. heir default so that the SUT always starts in the same state Therefore you can define pairs of paths copy from to TESTAR will copy each specified file from the given source location to the given destination Simply click the text area and a file dialog will pop up 4 Files to delete before SUT start Certain SUTs generate configuration files temporary files and files that save the system s state This might be problematic during sequence replay when you want a system to always start in the same state Therefore you can specify these files to be deleted before O Copyright 2014 FITTEST Consortium 7 36 A the SUT gets started If you click the text area a file dialog will pop up which allows selecting files and directories to be deleted Output Directory output Temp Directory temp Copy Files on SUT Startup Source File Directory Destination Delete Files on SUT Startup 3 6 Specifying Advanced Oracles Copyright 2014 FITTEST Consortium 7 36 A a TESTAR ole fs r gt AMI yen n h E a i protected Set lt Action gt deriveActions SUT system State state throws ActionBuildException PA About General Settings Oracle Filters Time Settings Misc Y ts Assert notNull state enerate Set lt Action gt actions Util newHashSet create an action compiler which helps us create actions such as clicks drag drop typin Path to SUT C Windo
10. ime Settings Misc TESTAR v1 0 PEA UNIVERSITAT apos Test DES DE VALENCIA As shown in the picture above the main screen contains four buttons which start TESTAR into its four main modes Copyright 2014 FITTEST Consortium AAA _ SEVENTH FRAMEWORK PROGRAMME 36 A 1 Start in Spy Mode This mode does not execute any actions It will start the System under Test SUT and allows you to inspect the GUI Simply use the mouse cursor to point on a widget and TESTAR will display everything it knows about it The Spy Mode will also visualize the set of actions that TESTAR recognizes so that you can see which ones will be executed during a test 2 Start in Generation Mode This mode will start the SUT and execute a full test on the SUT 3 Start in Replay Mode This mode replays a previously recorded sequence TESTAR will ask you for the sequence to replay 4 Start in View Mode The View Mode allows you to inspect all steps of a previously recorded sequence Contrary to the Replay Mode it will not execute any actions but only show you the screenshots that were recorded during sequence generation This is ideal if a sequence turns out not to be reproducible 3 Setting up your tests 3 1 General Settings Tab The Screenshot below shows the tab with the most important settings for TESTAR 1 Path to the SUT Pick the executable of the SUT or insert a custom command line 2 Number of sequences to gene
11. ng executed an action in Generation Mode Usually this value is set to 0 However sometimes it can make sense to give the GUI of the SUT more time to react before executing the next action If this value is set to a value gt 0 it can greatly enhance reproducibility of sequences at the expense of longer testing times 3 SUT startup time This is the time that TESTAR waits for the SUT to load Large and complex SUTs might need more time than small ones Only after this time has expired TESTAR will start sequence generation 4 Maximum test time seconds TESTAR will cease to generate any sequences after this time has elapsed This is useful for specifying a test time 99 66 99 66 out e g 1 hour one day one week 5 Use Recorded Action Timing during Replay This option only affects Replay Mode If checked TESTAR will use the action duration and action walt time that was used during sequence generation If you uncheck the option you can specify your own values 3 5 Miscellaneous Tab This tab contains some settings that are less frequently used 1 Output directory This determines the directory where TESTAR outputs recorded sequences log files and state snapshots 2 Temporary Directory TESTAR will use this directory to store temporary files during the execution of sequences 3 Files to copy before SUT start When you start the SUT sometimes it can be useful to restore certain configuration files to t
12. rate 3 Sequence length After having executed the given amount of actions TESTAR will stop the SUT and proceed with the next sequence 4 Force the SUT to the foreground During test generation the SUT s windows might get minimized or other processes might block its GUI If you check this option TESTAR will force the SUT to the foreground 5 Stop sequence generation on fault If TESTAR detects an error it will immediately stop sequence generation 6 Logging verbosity The higher the value the more information will be written to the RU s log file The log files of each run can be found in the RU s output directory and contain information about the actions that were executed the faults that were found and potential problems that occurred during the test 7 Edit the TESTAR Protocol By clicking this button you will open the Protocol Editor for the RU As we will see later on in this Manual this editor allows you to override and extend the RU s basic functionality in order to implement complex action sets and sophisticated oracles Copyright 2014 FITTEST Consortium 7 36 A 8 Load settings file If you have a specific setup that you saved into a file you can load it here This is ideal for switching between different settings for the same SUT or between the settings of different SUTs 9 Save the current settings to a file Remark TESTAR automatically saves all setting changes you make Thus you do not explicitly ha
13. ve to save your settings to a file every time you make an adjustment Path to SUT C Windows System32 calc exe Number of Sequences 50 5 Sequence Length 1001 Hs actions Force SUT to Foreground Y Stop Generation on Fault Y Logging Verbosity va u Edit Protocol E B Copyright 2014 FITTEST Consortium 7 36 A 3 2 Filter Tab In this tab you will be able to do the following two things 1 Tell the tool which actions not to click because they might be dangerous or undesirable the click filter 2 Tell the tool which processes to kill during test generation the process filter TESTAR r Spy enerate About General Settings Oracle Filters Time Settings Misc Click Filter Exit Import Close Print Open Save Cerrar Bro wse Help Contents Help New Pp roject Export pPlroject Add Fflile Process Filter helppan e exe CL VIEW EXE lfirefox exe AcroRd32 exel 1 Click filter Certain actions that TESTAR wants to execute might be dangerous or undesirable such as printing out documents creating moving or deleting files TESTAR will not execute clicks on any widget whose title matches the given regular expression To see whether or not your expression works simply start TESTAR in Spy Mode which will visualize the detected actions 2 Processes to kill Some SUTS start other processes during test sequence generation
14. ws System32 calc exe StdActionCompiler ac new AnnotatingActionCompiler Number of Sequences 50 if there is an unvanted process running kill it Strin rocessRE settings get ProcessesToKillDuringTest Sequence Length 100 actions zii cet gees gt for Pair lt Long String process state get RunningProcesses Collections Pair Long String emj Force SUT to Foreground Y if process right null amp amp process right matches processRE Stop Generation on Fault M actions add ac killProcessByName process right 2 Logging Verbosity 4 3 return actions Edit Protocol Save and Compile All the settings presented so far help you to setup tests for small and less complicated SUTSs However at some point you might feel the need to implement more complex actions or setup a more sophisticated oracle TESTAR allows you to edit its protocol i e the source code that determines its behavior The above screenshot shows the Protocol Editor which you can enter by navigating to the General Settings tab and clicking the Edit Protocol button It shows 1 The Source Code of the Protocol 2 The Save and Compile Button which compiles the protocol and saves it to be used during the next test 3 The Error Console which informs you about potential errors during compilation The source code used in the protocol 1s plain Java 4 Running the tool 4 1 Action Visualization in Spy Mo
Download Pdf Manuals
Related Search