a pre-print - Computer Science & Engineering
Contents
1. amp externalProtocol Listing 4 Different types of API preference name parameters Finally we show examples of hidden preferences In the String example in Listing 4 the preference autoadmin append_emailaddr appears in the source code but it does not exist in any preference files unless added by the user Preferences shown in Listing 5 are some other examples of hidden preferences from Firefox source code Our config uration space analysis RQ1 misses these preferences We do not know how many exist in Firefox pref browser homepage disable_button bookmark page pref browser homepage disable_button current_page pref browser homepage disable_button restore_default Listing 5 Hidden preferences We also investigate how configuration values are read in the code in ABB First there is a configuration manager class written in C that reads the values at different levels it may read values of a single preference it may read a single instance that contains a couple of preferences or it may read all instances that under the same configuration category Just like in Firefox all these values are passed by reference Second the name of the preferences can be in various forms such as string variable and macro Finally there are several configuration options that are accessed in the code but not in the document hidden preferences and there are also some configuration options that are in the document but a2. 1 a Model of the possible configuration space In order to sample the configuration space for testing or debug ging the configuration model needs to be known 2 to Know the Mapping of the configuration space to programmatic elements This is required in order to understand the impact a configurable item can have and to automate the modification of configurations for testing and bug reproduction and 3 an Accurate Configuration Snapshot to provide the full state of the application when a bug is encoun tered configuration aware eet i techniques a N onfi ion and tools O configuration N environment i N Test I D YD e IN Engineer j yi me End User wa i i O bug report le Maintenance customer support Engineer Figure 1 Configuration aware testing and debug ging expected use case We examine each of these requirements in relation to the existing work Configuration aware testing techniques 5 10 21 propose various methods to sample and prioritize the configuration space for testing but all of this work assumes that the configuration model is known or is somehow ex tracted from the code Based on our informal examination of systems like Firefox we do not believe that this can be easily achieved First we have discovered that the config uration control is not found within a single location of the code or in specific external files In fact most o
3. Engineering pages 193 202 2011 S K Sahoo J Criswell and V Adve An empirical study of reported bugs in server software with implications for automated bug diagnosis In International Conference on Software Engineering ICSE pages 485 494 2010 C Song A Porter and J S Foster iTree efficiently discovering high coverage configurations using interaction trees In The International Conference on Software Engineering ICSE pages 903 913 2012 The Document Foundation http blog documentfoundation org 2011 09 28 2011 C Yilmaz M B Cohen and A Porter Covering arrays for efficient fault characterization in complex configuration spaces IEEE Transactions on Software Engineering 31 1 20 34 Jan 2006 Z Yin X Ma J Zheng Y Zhou L N Bairavasundaram and S Pasupathy An empirical study on configuration errors in commercial and open source systems In Symposium on Operating Systems Principles SOSP pages 159 172 2011 S Zhang and M D Ernst Automated diagnosis of software configuration errors In International Conference on Software Engineering ICSE pages 312 321 2013 T Zimmermann N Nagappan P J Guo and B Murphy Characterizing and predicting which bugs get reopened In International Conference on Software Engineering ICSE pages 1074 1083 2012
4. the configurations in the code In Firefox the return value is almost always passed by reference For example the signa ture of a boolean preference access functions from the source file prefapi h under modules libpref src is shown in Listing 2 As we can see the configuration option value return_val is passed as a pointer in the formal parameter list The func tion returning value i e nresult is just an binary indicator of whether the actions defined in this function succeed or fail This prevents us from using the techniques developed by Rabkin et al 23 24 because the preference type cannot be inferred by tracking return value types T main xul lt preference id browser startup page name browser startup page type int gt lt menulist id browserStartupPage preference browser startup page gt lt menupopup gt lt menuitem label amp startupHomePage label value 1 id browserStartupHomePage gt lt menuitem label amp startupBlankPage label value 0 id browserStartupBlank gt lt menuitem label amp startupLastSession label value 3 id browserStartupLastSession gt lt menupopup gt lt menulist gt main js let startupPref document getElementById browser startup page startupPref updateElements Listing 1 Setting Firefox preferences using XUL nsresult PREF_GetBoolPref const char pref bool ret
5. tion manipulation in modern software systems 2 A case study that quantifies the complexity of three modern highly configurable software systems 3 A set of lessons learned that will help practitioners to better understand and control configuration instances for software engineering tasks such as testing and de bugging The rest of this paper is structured as follows In the next section we present some motivating examples motivating our study In Section 3 we present our case study Results are then given in Section 4 followed by lessons learned in Section 5 We highlight related work in Section 6 and conclude and present future directions in Section 7 2 MOTIVATION A configurable system is a software system with a core set of functionality and a set of variable features which are defined by a set of configuration options or preferences Changes to the value of a preference changes the program s behavior in some way For instance Firefox a popular web browser is a highly configurable system and one that we use to motivate some of the problems that we have encountered In Firefox an example configuration option that can be set via the option menu is called Warn me when closing multiple tabs This is a Boolean configuration with two values true false Its default value is set to true which means that if you try to close a window when multiple tabs are open you will lIn the remainder of this paper we use configuratio
6. Methodology 16 4 Sept 2007 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 J I Maletic and M L Collard TQL A query language to support traceability In ICSE Workshop on Traceability in Emerging Forms of Software Engineering TEFSE pages 16 20 2009 A Marcus X Xie and D Poshyvanyk When and how to visualize traceability links In CSE Workshop on Traceability in Emerging Forms of Software Engineering TEFSE pages 56 61 2005 Firefox http www mozilla org en US firefox 2013 X Qu M Acharya and B Robinson Configuration selection using code change impact analysis for regression testing International Conference on Software Maintenance ICSM 0 129 138 2012 X Qu M B Cohen and G Rothermel Configuration aware regression testing An empirical study of sampling and prioritization In International Symposium on Software Testing and Analysis ISSTA pages 75 85 July 2008 X Qu M B Cohen and K M Woolf Combinatorial interaction regression testing A study of test case generation and prioritization In International Conference on Software Maintenance ICSM pages 255 264 Oct 2007 A Rabkin and R Katz Static extraction of program configuration options In International Conference on Software Engineering ICSE pages 131 140 2011 A Rabkin and R H Katz Precomputing possible configuration error diagnoses In Automated Software
7. Prefs Pz Default Shut down normal start l start Il start Ill Start up Running normal start start Il start Ill Figure 6 ABB lifecycle diagram 1 Configuration Modeling Should Merge Multiple Layers We return to our first question of how one can model the full configuration space when performing testing and debug ging Although the application code is the ground truth the maintenance engineers may not always have access to code If instead we use the user manuals documentation we most certainly miss out on some configurations Moreover in the applications studied the menu on the user interface contained only a small subset of the configuration options While these might contain the most widely used preferences they do not provide a true indication of the real configura bility of a system Finally we can use the persistent configu ration preference files but we must first understand how in what order and when these are activated in the dynamic system Two issues that have arisen during our analysis are those of hidden preferences and dead preferences These constitute a small part of the configuration space model but one should be aware of their potential existence Given the results of our study we believe that to obtain an accu rate model of the configuration space one should consider and merge multiple artifacts which includes preference files menus and documentation Additionally since documenta
8. The user can modify these on disk directly if they under stand the format The next time the application opens these files will be read assuming that they have not been overwritten in the meantime see RQ3 for a discussion of that mechanism and the preferences will be activated The user can also open Firefox and use the about config web page to control or look at the preferences If a user modifies a preference in the about config it will be written to the user preference file and be set via the preference modules in the code Additionally the user can go through the options menu This contains only a subset of the full set of possi ble options only 126 out of the 1957 calculated in Table 1 We do not quantify or discuss the Add on configura tion options in this paper but these are also manipulated through a menu Finally there is an SQLite database which contains page specific option settings for the browser e g if a user zooms in on a particular website this information will be stored for the next time they open that site The preference modules are accessible through a set of preference APIs The APIs are used to interface with a dynamic hash table which contains all active configurations when an application is running There is a 1 to 1 mapping of the preference files to the hash table but an N to 1 mapping of the menu items These are used as variables in the code and several names may map to the same individual option
9. and Supporting Debugging of Field Failures In International Conference on Software Engineering ICSE pages 261 270 Minneapolis Minnesota May 2007 4 7 8 9 10 11 12 13 14 15 16 J Cleland Huang J H Hayes and J M Domel Model based traceability In ICSE Workshop on Traceability in Emerging Forms of Software Engineering TEFSE pages 6 10 2009 M B Cohen M B Dwyer and J Shi Constructing interaction test suites for highly configurable systems in the presence of constraints A greedy approach IEEE Transactions on Software Engineering 34 5 633 650 2008 M B Cohen J Snyder and G Rothermel Testing across configurations implications for combinatorial testing SIGSOFT Software Engineering Notes 31 6 1 9 2006 T Dasgupta M Grechanik E Moritz B Dit and D Poshyvanyk Enhancing software traceability by automatically expanding corpora with relevant documentation In International Conference on Software Maintenance ICSM pages 22 28 Sep 2013 J M Davril E Delfosse N Hariri M Acher J Cleland Huang and P Heymans Feature model extraction from large collections of informal product descriptions In The Joint Meeting on Foundations of Software Engineering ESEC FSE pages 290 300 2013 LibreOffice http libreoffice org 2013 E Dumlu C Yilmaz M B Cohen and A Porter Feedback driven adaptive combinatorial testing In International Symp
10. configurable system will be have differently while running under the same set of test cases 6 21 28 For instance in the work of Qu et al 21 as many as 80 of the faults had the potential to go un detected if tested under certain configurations Therefore configuration aware testing techniques have been proposed to systematically explore the configuration space 21 28 But all of this work assumes that the configuration model is known or is easily extracted During debugging this is also important Knowing the exact configuration instance that a user was in when the failure occurred can help with reproducibility Bettenburg et al 1 found that there is a strong mismatch in bug reports between what developers need to reproduce and fix a bug and that which is provided by users Other studies have also shown that bug reports lack information needed for bug reproduction 2 There has research aimed at reproducing field failures 3 14 but this line of work does not explicitly consider the configuration at the time of failure Given the complexity of today s software systems deter mining the configuration space may not be a trivial task For instance in the industrial system studied in Qu et al 20 they reported that there are more than 500 configuration op tions that their users can modify Firefox the open source web browser has over 1 900 configuration options available to a user The space of possible unique configurat
11. provided to pro grammatically manipulate internal data structures that hold configuration information For ABB we have a user man ual that is written for those who will modify and change preference files In addition we have asked questions of de velopers to confirm our assumptions In Firefox we utilize the source code examine the internal dynamic data struc tures via an API call when the application is running as well as study the about config page a utility for modify ing configurations We also study the Options menu the SQLite database that holds page specific preferences and online documentation For LibreOffice with the help of on line documentation we study the preference files and use an API to connect to the dynamic data structures when the program is running To answer RQ1 we calculate the ABB configuration space based on the user manual and we calculate the configuration space for Firefox and LibreOf fice through a program we wrote to query the dynamic data structures at runtime When we collect the configuration information we make some assumptions First constraints between options are ignored We realize that this might over approximate the configuration space slightly but extracting the exact con figurations options may not be feasible without in depth knowledge of each system Second the plug ins add ons are not included in our calculations In Firefox and Libre Office we build clean versions of the sys
12. will take place on the next startup Therefore the running configuration and the one reflected in the persistent memory after the ap plication closes may be inconsistent Faced with the complexity that we have described infor mally so far we want to quantify how often we see these problems with the aim of developing a generic model of how modern highly configurable software is structured and manipulated We also want to know if there is a ground truth for the configuration model and dynamic configuration states in modern configurable systems We next present a case study that we have developed for this purpose 3 CASE STUDY Our study has two main objectives First we want to quantify the complexity of the configuration space and what mechanisms are used to define and manipulate this space Second we want to understand what are the challenges that we will face as we develop configuration aware testing and debugging techniques To address these issues we will center our study around answering the following research questions RQ1 What is the complexity of the configuration space in modern configurable software systems RQ2 How are configuration options structured changed and accessed by the user in these systems RQ3 Are the selected configuration options synchronized between the different parts of the system and throughout the lifecycle of program execution 3 1 Software Subjects Studied We have selected three different softwar
13. 43182 93940 330805 944333318 9 in Firefox resulting in 846 boolean options and 1 111 options of either integer or string each with three values The total configuration space is equal to 284 x 31111 ABB has a variety of cardinalities for its configuration options We have a more exact model due to better documentation Our total configuration space for this application is 6 46 x 10 Finally we look at Table 3 which shows the configuration options in LibreOffice broken down by individual modules within the suite of tools This is based on the hierarchical path used to display the configuration option name For instance all of the preferences under Writer have the prefix org openoffice Office Writer We do not believe that all 36 322 would be used together in any test or debug model Instead one would test an application such as Writer indi vidually Although we can identify which preferences be long to specific applications such as Writer or Calc there are some categories such as UI which may be shared among applications These all fall into the Others category The complete categorizations are contained on our website ABB has preference files that contain additional infor mation not found in the open source applications This is because it is an embedded system with configuration op tions that can be customized for different drivers or ports The number of devices and ports is open ended The two additional pieces of informati
14. Configurations Everywhere Implications for Testing and Debugging in Practice Dongpu Jin Xiao Qu Dept of Comp Sci amp Eng Industrial Software Systems Dept of Comp Sci amp Eng ABB Corporate Research Raleigh NC 27606 USA xiao qu us abb com Univ of Nebraska Lincoln Lincoln NE 68588 USA djin cse unl edu ABSTRACT Many industrial systems are highly configurable complicat ing the testing and debugging process While researchers have developed techniques to statically extract quantify and manipulate the valid system configurations we conjecture that many of these techniques will fail in practice In this paper we analyze a highly configurable industrial applica tion and two open source applications in order to quantify the true challenges that configurability creates for software testing and debugging We find that 1 all three appli cations consist of multiple programming languages hence static analyses need to cross programming language barriers to work 2 there are many access points and methods to modify configurations implying that practitioners need con figuration traceability and should gather and merge meta data from more than one source and 3 the configuration state of an application on failure cannot be reliably deter mined by reading persistent data a runtime memory dump or other heuristics must be used for accurate debugging We conclude with a roadmap and lessons learned to help prac titioners
15. better handle configurability now and that may lead to new configuration aware testing and debugging tech niques in the future Categories and Subject Descriptors D 2 5 Software Engineering Testing and Debugging Testing tools General Terms Verification Keywords Configurable Systems Testing Debugging 1 INTRODUCTION Many software systems are highly configurable allowing the user to customize an individual instance of the program while retaining a core set of functionality This customiz ability provides benefit to the end user however it also in Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page To copy otherwise to republish to post on servers or to redistribute to lists requires prior specific permission and or a fee ICSE 14 May 31 June 7 2014 Hyderabad India Copyright 2014 ACM 978 1 4503 2768 8 14 05 15 00 Brian Robinson ABB Inc Raleigh NC 27606 USA brian p robinson us abb com Myra B Cohen Univ of Nebraska Lincoln Lincoln NE 68588 USA myra cse unl edu troduces many challenges during testing and or debugging because configurability complicates the process of finding and or reproducing the failure Research has shown that different instances of a highly
16. bility Re search has shown that failures tend to have feature local ity 11 so it is possible we can leverage some of those ideas for this work 6 RELATED WORK We provide a short overview of several areas of research that are closely related to this work The role of software users and essential information in bug fixing has been em phasized in several studies 1 2 25 31 Bettenburg et al 1 found that there is usually a strong mismatch in bug reports between what developers need to reproduce and fix a bug and what is provided by users Herbold et al 13 developed a tool to capture usage logs for replaying bugs Other work tries to reproduce field failures 3 14 however the focus is on using the call graph None of this work tries to capture the software configuration used during the failure Several researchers have been focusing on extracting con figuration options from code Rabkin et al 23 24 propose a method to statically detect system configurations but as already mentioned this analysis works on a single language Java and assumes that all configurations are contained in a single class Yin et al 29 conducted empirical studies to understand the configuration errors in commercial and open source systems Zhang et al 30 have proposed a technique to diagnose crashing and non crashing errors related to soft ware misconfigurations Again their tool only works on a single language Java and the configurations they st
17. d Firefox As we can see we have only three types Table 1 Quantifying number of preference files and preferences of ABB Firefox and LibreOffice ABB Firefox LibreOffice Operating System Embedded System Ubuntu 12 04 Ubuntu 12 04 Version Mozilla Firefox 27 0al LibreOffice 4 0 LOC M 10 0 9 6 6 8 C 41 CQ1 Primary C 3 7 JavaScript 16 Java 3 1 C 82 Languages C 29 6 C 8 Python 2 7 Assembly 1 2 Java 6 Shell script 1 Total Pref Files 6 11 193 Total Prefs 524 1957 36322 Table 2 Categorization of configuration space for ABB and Firefox The total number of preferences are shown as c where c is the cardinality of the preference number of values and n is the number of times we have this cardinality We have combined like cardinalities together therefore the total boolean values for example may include some from the others category Types ABB Firefox Boolean 2 92 846 Integer 3 271 517 String 3 27 594 String with condition 4 110 E Others 24 Total eae Te oe le eee Table 3 Categorization of the configuration space for LibreOffice broken down by module Types Writer Calc Impress Draw Math Database Others Total Boolean 2 201 58 69 44 77 44 3940 4433 Integer 3 157 43 26 22 110 15 5087 5460 Others 298 70 32 3 141 167 25718 26429 Total 9201 3455 9583113 269358 944325 2773251 94
18. e systems to study The first is a large real time embedded software system de veloped at ABB called ABB hereafter ABB has ap proximately 10 Million lines of code is highly configurable and has more than 58 modules each module defines a sub system that implements a different set of functionality of the system The second subject Firefox 19 is an open source web browser which works on multiple operating systems and has over 300 Million users worldwide and over 9 6 Million lines of code The third subject is LibreOffice It is an open source office productivity suite consisting of a word proces sor spreadsheet application presentation tool drawing ap plication math formula tool and database 9 LibreOffice has 6 8 Million lines of code and 25 Million users worldwide estimated by The Document Foundation in 2011 27 We believe that these subjects are different from each other in functionality and in development style yet they are all large highly configurable and used in a production environment by a large pool of users 3 2 Study Design To answer our research questions we collect configura tion information from both a static and dynamic perspec tive on each system We manually study all artifacts that are publicly available to users including documents e g user manuals and online help pages software option menus on the user interface preference files and source code We also utilize tools or APIs that have been
19. ed by categories and the number of categories for each prefer ence file In this paper when we compute the configuration space shown in Table 2 we made a conservative assump tion that all options will appear a single time regardless of instances to make it in consistant with other systems 2The ABB user manual states that if the option is as signed the default value then it will not be listed in the configuration file and this is why the third instance only has one option explicitly written 1 2 CATEGORY A 3 Name x Count 2 4 Name y Count 5 5 Name z 6 7 CATEGORY B 8 Unit X Length 10 status ON Figure 3 Example of ABB preference file Table 4 Number of options grouped by categories in ABB Preference Number of Number of Files Categories Options File 1 3 26 File 2 11 50 File 3 10 78 File 4 7 22 File 5 39 348 Total 70 524 4 2 RQ2 Configuration Access We begin answering RQ2 by examining the structure of one of our open source systems Firefox Figure 4 shows this schematically In this figure there are a number of pref erence files both user and default that contain values for specific preferences During the application startup the de fault configuration options are read there are 1932 of them and after that the user preferences are read there are 50 of them initially These are read by the preference modules
20. f the systems we have studied have a multi tiered layout of how configu rations are defined and accessed and this can be done both offline and at run time Figure 2 shows a schema that rep resents all of the systems we have studied First there is a static view of the system labeled 1 This includes any existing user manuals web pages etc that contain docu mentation on the possible configuration options and their values This often is incomplete or out of date see our dis cussion of hidden or dead preferences in Section 5 The second static element is the source code itself This contains the ground truth but source code may not be available to everyone who wants and needs to understand the configura tion model Moreover as we shall see using this to extract the full configuration space is non trivial When controlling what configurations are set there are usually external mechanisms 2 in Figure 2 such as pref erence files or databases These can often be accessed in dependently of the program even while it is running and therefore may or may not contain the current state of the configurations We have also seen that these may not con tain the ground truth of the configuration space Finally as is shown in 3 there are usually some run time access mechanisms that connect to the internal data structures or database For instance most programs have a menu system that allows the user to set preferences but in the sys
21. in memory Finally the code itself program modules contain the ground truth for the configuration space We have dis covered several options in the code that are hidden These are options without default values that can be set if a user 4 gt Workflow user defau about config gt Page S Preferences Modules Pref APIs eS gt Mapping 4 Hidden ay fae ae Prefs lt 2 ees San At J Program Modules Page specific prefs e g zoom in out image loading Figure 4 Firefox configuration structural diagram knows about them but which do not appear in our results for RQ1 since they are not in the hash table or preference files unless explicitly set by the user We have analyzed the user interface UI source code of the Firefox option menu and retrieved 126 preferences that are bound to the option menu UI elements Listing 1 shows an example of binding the preference browser startup page which specifies the start up page Firefox opens to a drop down menu list in the option menu Thus only 6 4 of the total preferences exist in the option menu in Firefox We note that both the ABB and LibreOffice systems have similar structures therefore we do not show them all here but an extraction of the general structure is illustrated in Figure 2 and introduced in Section 2 We next investigate how configuration values are read in the code First we take a look at the APIs used to access
22. ions grows exponentially with the number of configuration options or preferences therefore we can only evaluate a representative sample of all possible configurations As we work more and more with highly configurable sys tems in practice we have discovered common issues that arise which make available configuration aware techniques insufficient For instance in our industrial systems there is usually no single document that describes the complete set of possible configuration options We can examine ex ternal preference files but we find that there may be mul tiple files and they still tell only a partial story because there are hidden but valid preferences found only in the source code We can try to use an analysis technique such as those proposed by Rabkin et al 23 24 to reverse engineer a complete mapping of our configuration space but many applications are written in multiple programming languages such as C Java and JavaScript and often use aliasing to refer to preference names neither of which are supported by existing techniques Finally if we assume that we can somehow obtain the ground truth model of the configura tion space then in order to manipulate the configurations for testing and debugging we need mechanisms to automate this process as well as ways to capture which configuration was active during a failure Again we have learned that the complexity of real software makes this difficult configura tio
23. les before the Firefox Prefs Lifecycle 1 Permanent 1 write If User Prefs i 1 1 Use Memory prefs j 1 1 1 LibreOffice Prefs Lifecycle i User wre conne i I Admin Memory Y if coms J I I chema conne J u i Shut down Start up Running Figure 5 Firefox and LibreOffice lifecycle diagram application closes In Firefox the user preference file is over written but the default ones are not This means that if a user modified the user preference files during runtime those changes will never be seen not even on the next startup However if they modified other preference files they will appear on next startup ABB has a more complicated restart behavior described next When the system is restarted normally denoted as start I the current system will be stopped All system pref erences will be saved Restarting this way will activate any configuration changes A second option is to restart and select another configuration start II In this case the cur rent system will be stopped All system preferences will be saved so that the system state can be resumed later The last restart is to restart and return to default settings start II After restart the system state will be resumed but any changes done to system preferences will be lost Instead sys tem preferences are read from the originally installed system on delivery Furthermore in AB Be there are three sets of preferences acti
24. n abstraction of this mechanism We also see that there is no single easily available ground truth to determine the full possible configuration space To this end we recommend merging multiple sources developing cross language anal ysis tools and providing traceability between the different configuration layers We have also seen that the dynamic behavior can be difficult to understand therefore we need to be cognizant of the lifecycle of the application to under stand our exact configuration state during debugging In future work we plan to implement some configuration merg ing techniques and traceability links between the various layers We also plan to examine a larger variety of highly configurable systems to understand if the same model holds 8 ACKNOWLEDGMENTS This research was supported in part by the National Sci ence Foundation awards CCF 1161767 CNS 1205472 and the Air Force Office of Scientific Research award FA9550 10 1 0406 9 REFERENCES 1 N Bettenburg S Just A Schr ter C Weiss R Premraj and T Zimmermann What makes a good bug report In International Symposium on Foundations of Software Engineering FSE pages 308 318 2008 2 S Breu R Premraj J Sillito and T Zimmermann Information needs in bug reports improving cooperation between developers and users In Conference on Computer Supported Cooperative Work CSCW pages 301 310 2010 3 J Clause and A Orso A Technique for Enabling
25. n handle aliasing and that use additional heuristics to identify the actual getter and setter code 4 Configuration State Capture or Approximation Techniques are Needed As we argued at the start of this paper we need a way to capture the active configuration when the system fails so that we can reproduce and debug the failing test case Each of the three systems we studied allows the user or mainte nance engineer to modify the configurations both externally or internally during runtime While our open source ap plications update the memory and files immediately in our industrial application the configuration is not activated un til possibly startup with the exact behavior dependent on the type of reboot selected Even if we understand how the configuration manipulation works there is the possibility of race conditions in all of the applications depending on the exact timing of the configuration modification and failure It is also possible to make changes to external files for mod ifications at startup yet these may be overwritten during a normal shutdown In order to extract the ground truth of the configuration at failure monitors are needed that cap ture this information But these may incur overhead and cause concerns for privacy Alternatively we know that the persistent memory contains a large portion of the correct configuration space so algorithms that work from this point and search close by may be useful for reproduci
26. n methods start with get or set As shall see these assumptions do not hold for any of the applications we studied For instance there are instances in Firefox where the preference code includes JavaScript and other languages such as the markup language XUL We see instances where the Javascript API is able to query and update a preference however it uses the XUL code as a reference to the given preference name binding it to a user interface element We also see preference code that is not using the name value pair mechanism but instead uses references macros or member fields to refer to the prefer ence name Another issue that we have encountered is that the API method names of Firefox do not always start with get or set We need more different techniques if we plan to extract all of these configuration options from the code Finally if we are concerned with knowing the current state of the configuration space at some point in time we need a technique that captures an accurate configuration snap shot at runtime Indeed it may not be straightforward to get this information from the system In some of the ap plications we have studied Firefox and LibreOffice when the user modifies a preference value dynamically through the option menu the change is reflected immediately in the dynamic memory and preference files However in our in dustrial application the change made by the user will be stored temporarily and the new preference
27. n option and user preference interchangeably get a warning asking if you want to close all of the tabs If you uncheck this on the menu set it to false it will prevent a warning from being produced and immediately close the window The actual preference name for this found in the preference file is called browser tabs warnOnClose There is another closely related preference in the preference files called browser tabs warnOnCloseOtherTabs which is set to true by default but has no menu counterpart When test ing the system or when a failure occurs we need to have information about the values that were selected for each of these configuration options something that may not be ob vious by examining just the menu alone We assume an idealized use case for testing and debugging as shown in Figure 1 In this scenario we have three entities that interact with the configurable system The end user can modify configurations and will send bug reports to and possibly read reports from customer support As can be seen in the figure he or she may use the menu or they can directly write to configuration files A set of configuration aware techniques and tools sit between the application and the tester and maintenance engineer which feed informa tion about configurations back to the bug reporting cus tomer support system The challenge is to enable these configuration aware techniques We have identified three important requirements We need
28. ning appli cation to understand when and where its configurations are synchronized between its layers We model three distinct phases startup runtime and shutdown Figure 5 shows the behaviors of Firefox and LibreOffice and Figure 6 shows the behaviors of ABB The numbers on the leftmost side specifies the number of preference files in different groups of files Solid arrows represent direct connections while dashed arrows indicate the need for a mapping traceability In all three systems at startup the configurations are read from persistent storage configuration files and loaded into memory There is a specific order in which these are loaded If the same configuration options are repeated set to dif ferent values the last one read will be the one which holds While the applications are running a user can modify the configuration files directly This is not immediately reflected in the dynamic memory If a failure occurs at this point the persistent memory is out of sync with the dynamic In all three systems the user can also dynamically modify the con figurations while the application is running In Firefox and LibreOffice these will take effect immediately and be written back to the preference files In ABB the dynamic memory is not updated The changed configurations are held in tem porary memory and take effect at the next startup On shutdown in Firefox and LibreOffice the dynamic memory overwrites the current preference fi
29. ns can be modified and viewed from multiple locations and are found in both dynamic and static structures Fi nally we have discovered that it is possible for the static structures to be out of synchronization with the dynamic ones at the time of failure In this paper we attempt to uncover and quantify the ex tent to which these problems exist on an industrial scale We empirically examine several large highly configurable appli cations to understand the implications for testing and de bugging in practice We study one industrial application and two widely used open source applications We quan tify the size of the configuration space and evaluate where and if the ground truth for the configuration model exists We also examine how a user tester or maintenance engi neer can manipulate the configuration options Finally we examine the runtime factors involved in capturing the cur rent configuration space Our study shows somewhat sur prisingly that both the industrial and open source appli cations have elements of configurability in common which leads to a set of lessons learned and a roadmap for devel oping configuration aware testing and debugging tools We see this study as a way to share with practitioners the issues configurability brings and a springboard to accurate and us able configuration aware testing and debugging techniques The contributions of this work are 1 An abstraction of the general structure of configura
30. on in these preference files are category and instance Certain preferences are grouped into a category and for each category we have one or more in stances that consist of the same set of preferences Each category may contain multiple instances therefore one pref erence can appear multiple times To understand this better we can consider a situation where each instance is associated with a specific hardware or virtual device Some devices are in the same category thus have the same set of preferences however the device that is being controlled differs An example of a snippet of the ABB preference file is il lustrated in Figure 3 the names are changed for proprietary purposes There are five options in this figure bold fonts Name string Count integer Unit string Length integer and Status boolean Name and Count are grouped under CATEGORY A while Unit Length and Sta tus are grouped under CATEGORY B There are three instances in CATEGORY A in the first instance line 3 the Name is assigned with value x and Count is assigned with 2 in the second instance line 4 the Name is assigned with y and Count is assigned with 5 in the third instance line 5 the Name is z and Count is the default value Similarly there is one instance in CATEGORY B line 8 the option Unit is assigned with X the Length is assigned with 10 and the Status is assigned with ON Table 4 shows the number of configuration options group
31. osium on Software Testing and Analysis ISSTA pages 243 253 2011 B Garvin M B Cohen and M B Dwyer Failure avoidance in configurable systems through feature locality In J Camara R Lemos C Ghezzi and A Lopes editors Assurances for Self Adaptive Systems volume 7740 of Lecture Notes in Computer Science pages 266 296 Springer Berlin Heidelberg 2013 O Gotel J Cleland Huang J H Hayes A Zisman A Egyed P Grunbacher and G Antoniol The quest for ubiquity A roadmap for software and systems traceability research International Requirements Engineering Conference RE 0 71 80 2012 S Herbold J Grabowski S Waack and U Biinting Improved bug reporting and reproduction through non intrusive GUI usage monitoring and automated replaying In International Conference on Software Testing Verification and Validation Workshops ICSTW pages 232 241 2011 W Jin and A Orso BugRedux reproducing field failures for in house debugging In International Conference on Software Engineering ICSE pages 474 484 2012 L C Lamb W Jirapanthong and A Zisman Formalizing traceability relations for product lines In ICSE Workshop on Traceability in Emerging Forms of Software Engineering TEFSE pages 42 45 2011 A D Lucia F Fasano R Oliveto and G Tortora Recovering traceability links in software artifact management systems using information retrieval methods ACM Transactions on Software Engineering and
32. re never read in the code dead preferences Table 5 Number of configurations accessible at dif ferent layers System Static View Extern Control Tab 1 Code Manual Files Menu ABB 524 428 166 524 lt 524 lt 524 Firefox 1957 gt 1957 NA gt 1957 126 Table 5 summarizes the number of configuration options that are accessible at different layers defined in Figure 2 The first column Table 1 shows the values we obtained for RQ1 The last column Menu is used to represent con figuration control via menu in Firefox and via ABB and ABB in ABB For ABB there are 428 166 594 op tions accessed in code 428 options are also described in the manual but 166 options only appear in code hidden pref erences and 524 428 96 options only appear in the document dead preferences This shows that the document is not updated accordingly as the code is changed although the document is a very important artifact that tightly con nects the system with customers We do not have accurate numbers of the preferences accessible by external control el ements but quote the manual which says if the option is assigned the default value then it will not be listed in the preference file there are also preferences not in ABB or ABB given that some configurations have to be changed in preference files 4 3 RQ3 Configuration Synchronization To answer RQ3 we map the lifecycle of a run
33. source applica tions and our tools for gathering the data can be found on our associated website see http cse unl edu myra artifacts SEIP 2014 4 1 RQ1 Configuration Complexity To answer RQ1 we turn to Tables 1 2 and 3 Table 1 pro vides the basic statistics for our applications It first shows the operating system and versions of the two open source applications We then list the primary languages that are used in each application We show all languages that make up at least 1 of the code We leave out markup languages such as XML or XUL All three applications consist of at least two languages Firefox has the most with C C JavaScript Python Assembly and some shell script Libre Office has both C and Java ABB has a mixture of three languages C C and C We also list the number of preference files that are used to store the current set of preferences and that are read at startup As we see this ranges from 6 files in ABB to 193 in LibreOffice there are six preference files in ABB but we were unable to access one of them so all of the computation that follows uses only five files Finally we list the total numbers of unique pref erences that we counted in each of these applications This ranges from 524 in ABB to 36 322 in LibreOffice We next look at Tables 2 and 3 We show a breakdown of the configuration options by the data types and number of values associated with each type Table 2 has data for ABB an
34. tem from source code for study Any default plug ins that come with those will have their configuration options included however no additional plug ins are enabled To calculate the number of values associated with an option we have detailed infor mation for many of the configuration options in the ABB manual However when they are not available and for Fire fox and LibreOffice we use a set of rules to come up with a small set of categories For Boolean configuration options we use true or false For integers we use a default value a non default legal value and an illegal value resulting in 3 values For strings we use no string an empty string and a legal string again resulting in 3 values In ABB we have some strings with constraints For these we use 4 values by adding an illegal string This partitioning may under estimate the true configuration space it is a conservative model but it is consistent with prior work 5 For RQ2 and RQ3 we analyze the systems further and experiment with the various ways that one can modify con figurations when the system is not running We also analyze what happens if configurations are modified while the pro grams are running as well as what occurs with the changed configuration options during startup and shutdown We ex amine some of the preference setter code and also look for hidden preferences that may not have been exposed earlier We look at bo
35. tems we have studied this accounts for only a subset of the full set of configurations Other specialty tools exist such as the about config mechanism of Firefox that allows one to pull up a web page where configurations can be mod ified dynamically Again these may not show the complete set of configuration options that are available There may also be an API to allow programmatic access to an internal memory structure such as the hash table in Firefox This should be the ground truth of what preferences are set at any point in time but it will not contain the hidden preferences External Control Database G Runtime Access Database or Memory Specialty Tools Figure 2 General view of configuration layers Q Static View Preece Files User Manual Source Code Preference Menu Suppose instead of using the menus or preference files we want to extract the preferences from the code itself which also helps to build a mapping between the configuration space and code Rabkin et al 23 24 presented techniques to statically analyze Java programs with JChord Upon study ing their work in more detail we find that it does not directly apply to a system like Firefox First it assumes a single pro gramming language Java second they assume all of the preference manipulation code exists as name value pairs and is found in a single class and finally they assume that configuration manipulatio
36. th menu access as well as file access We also use the specialized tools such as the about config to inter face with Firefox and the ABB tools denoted as ABB and ABB to interface with AB Be 3 3 Threats to Validity As with any study there are threats to validity which we document here First we have only studied three software systems While we believe they are different enough one is an industry application while two are open source applica tions with different sets of developers we can not be sure that our results will generalize to all configurable applica tions Our second main threat is that we are not developers of these systems so we have relied on the documentation and code to extract the information that we need With ABB we were able to confirm our questions with developers In the Firefox and LibreOffice environment we do not have this as a source of validation But we used third party APIs that are commonly used to interact with the configuration envi ronments and made an effort to validate our result internally We have made the tools we used to query Firefox and Libre Office available online as well as the artifacts that we have obtained to reduce this threat Finally we could have mea sured different elements for this study but feel that the set of metrics we collected supports our research questions 4 RESULTS We now present our results for each of the three research questions Supplemental data for the open
37. tion is the primary artifact a user would read it should be updated as the design and code changes in a timely man ner particularly when it comes to system testing or other configuration related tasks 2 Configuration Traceability is a Necessity Given the variety of places that configurations are ac cessed and mentioned it seems that the task of simply set ting a configuration option requires deep knowledge of the application If we return to our example Firefox one needs to know the mapping of menu names to preference variable names to modify them automatically Furthermore we have seen both in Firefox and ABB a many to one mapping of variables in the code and preference files and dynamic memory Providing traceability mapping between elements of the configuration manipulation mechanism are essential to making configuration aware techniques work 3 Analysis Tools Need to Cross the Programming Language Barrier As we have seen the current state of research in anal ysis for extracting configurations from code expects a sin gle programming language and single class files where the configuration information code such as setting and getting configuration lies Yet this is not realistic for the large scale subjects that we have studied Our configuration options are manipulated and referenced across programming language barriers and in multiple modules We need therefore new analysis techniques that cross these boundaries ca
38. udy are simple We look at more complex configuration spaces with multiple languages and multiple preference layers etc From a traceability perspective there has been a large body of research 4 7 12 16 18 but most focuses on the traceability of requirements architecture and quality at tributes Recent research has looked at extracting trace ability for feature models a type of configuration model space 8 15 but this has been achieved only through docu mentation rather than by examining the multiple layers of the software preference space We believe some of this work can be leveraged for configurability Finally there has been a large body of work in the testing community that demonstrates the need for configuration aware testing techniques 20 22 28 and proposes methods to sample and prioritize the configuration space 5 10 26 28 There has also been recent work that uses configurability as a way to avoid failures through self adaptation 11 But all of this work assumes that the configuration model is known or is somehow extracted 7 CONCLUSIONS AND FUTURE WORK In this paper we have presented a case study to evaluate the complexity that configurability adds for developers and testers To do so we have studied three highly configurable software systems We have shown that our open source and industrial applications all have similar mechanisms for main taining and modifying configuration options and presented a
39. urn_val bool get_default Listing 2 Return value is passed by reference Second the preferences are accessed via multiple program ming languages The Listing 1 and 3 show two examples of the Firefox source code interfacing with the preference sys tem via XUL and JavaScript respectively The JavaScript performs most of the manipulation but the XUL code in terfaces and dereferences the preference name nsBrowserContentHandler js var choice prefb getIntPref browser startup page nsBrowserGlue js Services prefs setIntPref browser startup page 3 Listing 3 JavasScript Setting Firefox preferences using Third the preference name can be in various forms when passing to preference APIs The name of the preference is usually passed as the first parameter to the preference APIs Listing 4 shows a few examples of passing the preference name as a string a variable an object macro a function macro or a class member String rv mPrefBranch gt GetBoolPref autoadmin append_emailaddr amp appendMail Variable prefBranch gt GetIntPref kCookiesLifetimeBehavior amp lifetimeBehavior Object macro rv branch gt GetIntPref DISK_CACHE_CAPACITY_PREF amp capacity Function macro rv prefs gt GetIntPref HTTP_PREF connection retry timeout amp val Class member rv prefBranch gt GetBoolPref externalProtocolPref get
40. ve loaded by default backup and default During startup instead of loading different sets of preferences in order as happens in Firefox and LibreOffice the system only loads one set of preferences into memory based on the type of restart During normal start and start I the ac tive preferences are loaded during start II a selected set of previous backup preferences are loaded and during start III the factory default preferences are loaded During run time the users can make configuration changes in preference files directly or through ABB or AB B but changes will not take effect until a restart The changes will be stored temporarily in a memory different from the active prefer ences Users can also save the currently active preferences as a backup Finally all changes made at the runtime will be written back into the active preference files when the sys tem is normally shutdown or restarted in I or II Note that if the users select a start III all changes will be lost 5 LESSONS LEARNED In this section we summarize the implications and lessons learned from our study The first two lessons learned are geared towards practitioners since they reflect the state of the art The last two provide a roadmap for researchers who plan to develop new tools and techniques for configuration aware testing and debugging ABB_c Prefs Lifecycle NOT for Start IIl 4 N 4 gt Ny Sal aetveUser H 6 ActiveUser
Download Pdf Manuals
Related Search