Home

FastFix Platform User Manual

1. ccn 19 3 10 Ticket browser configuration element 0084 21 TOT 27 DU E RSGSMESSSRPSEAM RS TS S 23 DICERE REOR 24 3 14 Starting registered sensors via the FastFix client UL 25 225 9 3 9 953 295 P2495 mJ m 30 Cc 91 pee ee see 49 952 959 599 59925 55 9 9 ae se 32 Pet eet ete teneeneneews ss 92 LS EUER ES UEM EIS ERE RP ES 33 poet 9 tae etek eee ee eas eee eee S 34 arias adas EE EE 39 pee eee ee See ee Bee ee oe 39 cierras a 36 3 24 Viewing an error Teportl e Of ee ee ee ee ee ee eee 38 asa 39 Page 4 of 1 Introduction This document describes the final version of the FastFix platform It is a supplement to the FastFix source code which can be accessed in the FastFix source code repository on SourceForgd The document first gives a general conceptual overview of the platform as a whole and summarizes current source code metrics Then it describes how FastFix can be used from the point of view of two actors First the FastFix users 1 e maintenance engineers who want to remotely maintain their software with FastFix and need to deploy and setup the FastFix platform Second developers who want to develop for and extend FastFix for instance by writing additional sensors adding rules or adding possibilities to configure the platform For more details on specific aspects and functions of the FastFix platform we refer the reader to the corresponding deliverables as summar
2. This traces can for example be generate when the application is under test and therefore represent a set of good behaviors of the system corresponding to traces collected from passing tests These traces can later be used for patch validation ADO Preferences type filter text Runtime Self Healing Dro rw General Agent Controller Self Healing Runtime settings Ant M Activate Controller dataStore s NUN errorReportGeneration upervisor Butter 5ize 10000 FastFix Server Request Period in ms 60000 Help Install Update ivi Testing Mode for Trace Collection Java Maven Memory Analyzer Plug in Development Profiling and Logging Run Debug Runtime Self Healina Self Healing serverConnection Tasks Team Test Usage Data Collector Validation XML Restore Defaults Apply Figure 3 13 FastFixSH configuration within FastFix Client 3 1 2 FastFix Client Setup The FastFix client is implemented as an Eclipse RCP Java application It can also be run in console mode for systems where no graphical user interface is available The latest binary version of the FastFix client can be accessed in the FastFix repository at SourceForge To run the FastFix client the following steps have to be accomplished Page 24 of FastFix Platform User Manual 1 Download the latest build of the FastF ix client and unzip into a folder on the client machine 2 Configure the FastFix client see Section 3 1 1 3 Start
3. e sequenceTimeWindow long It refers to the value in milliseconds of the time window of a sequence e itemSetTimeWindow long It refers to the value in milliseconds of the time win dow of a itemSet A sequence is composed by one or more itemSets so the value of sequenceTimeWindow must be higher than itemSetTimeWindow As a summary depending on the application temporal properties the developer can specify what should be considered as frequent support what number of events will be considered as a sequence what is the time window of each sequence as well as the period of time to consider events as part of a different item set 4 2 4 Error Reporting Extending the development of FastFix in terms of error reporting can proceed along two main lines adding additional sensors that trigger the generation of error reports and supporting additional issue trackers Different type of applications and error types may require the design of new sen sors Sensor design is described in Section 4 2 1 The connection of sensors to error reporting is done by the fact that an error reporting sensor sends events of type ER ROR REPORT EVENT e g eu fastfiz targetapplication sensor javaapplication imter nal JavaExceptionSensor the sensor class for unhandled exceptions in Java AWT applica tions Sensors who issue the same type of event will trigger the emission of an error report These events are processed by the FastFix s client f
4. FastFix Platform User Manual Dennis Pagano Tobias Roehm Emitz Guzm n Authors Sergio Zamarripa Joao Garcia Benoit Gaudin Javier Cano Christophe Joubert Walid Maalej Project co founded by the European Commission under the Seventh Framework Programme SEVENTH FRAMEWORK C S2 TUM LERO INESC ID TXT PRODEVELOP Abstract This document describes the final version of the FastFix platform It is a supplement to the source code which can be accessed on the FastFix open source project repository at SourceForge It gives a conceptual overview of the platform as a whole and describes how the FastFix platform is typically deployed set up and used by end users and developers This document has been produced in the context of the FastFix Project The FastFix project is part of the European Community s Seventh Framework Program for research and development and is as such funded by the European Commission All information in this document is provided as is and no guarantee or warranty is given that the information is fit for any particular purpose The user thereof uses the information at its sole risk and liability For the avoidance of all doubts the European Commission has no liability is respect of this document which is merely representing the authors view Contents 5 7 2 eu fastfx client a 2 aw ee eww xo HS 4 Xo wow ee xw omo 4 Y 2 2 __eu fastfix common IEEE 8 2 3 eu fastfix targetapplication 2 a
5. e to allow maintenance engineers to replay errors f to create patches and send these patches to clients to self heal them and g to provide a user interface to allow maintenance engineers to access the FastFix functionality e g error replay and patch generation Figure 2 3 shows a screenshot of the FastFix server UI which mainly consists of a log and specific FastFix menus The server namespaces currently contains 415 classes with 2 826 methods in 115 pack ages 21 bundles with a total of 40 164 lines of code in 410 files FastFix Server Help Pattern Management Moi ADO Fastfix Serv Model Generation Configurations Pattern Generation Patch Generation TICtp z7WWW TUStTIXprUJECLC Eu UITtOLUBgLEeS MUOTITLCUTF LTTQUT Patch Mergence http www fastfixproject u ontologies Monitoring n Patch Deployment http www fastfixproject eu ontologies Maintenancel_ E http www fastfFixproject eu ontologies Moni torino oni texrernatcomponentearr http www fastfixproject eu ontologies MonitoringOntology owl AttributeAccess 0 F v to om m rm rp m http wwwW fastfixproject eu ontologies Maintenance ntology owl SQLDropTable TastTixproject eu ontologies Monttoring ntology owl HttpRequestOnClientSide 2d k mi ee oe mmm m m Mor dm mom m mil lum d m vu UAE maa E dan zc T 4 k Figure 2 3 FastFix server platform UI 2 5 eu fastfix dependencies The dependencies namespa
6. I I U I jJ x p I c s US I I gt 1 1 gt o Y I 1 D 1 Se gt e qu On I I Figure 2 1 Conceptual overview of the FastFix platform As described in 11 FastFix components are structured into namespaces according to their responsibility in the platform The namespace of each code bundle can be identified from the bundle name Currently the integrated platform contains 79 bundles belonging to 5 different namespaces In the following we briefly explain all five namespaces and give additional details in terms of code metrics Details on the source code in terms of interfaces and packages can be found in as well as in the source code documentation JavaDoc 2 1 eu fastfix client The client namespace contains bundles constituting the FastFix client component The FastFix client runs in the environment of the target application i e in the application Including bundles needed to group other bundles parent bundles The project source code is available at SourceForge https svn code sf net p fastfixrsm code trunk Ph Page 7 of FastFix Platform User Manual usage environment Its main purposes are a to collect context information monitored by sensors b to perform data pre processing tasks before sending information to the maintenance site for security and performance c to access supervisor mechanisms in the target application for self healing and d to provide a user interface to allow
7. 33 of FastFix Platform User Manual Fastfix Server File Help Self Healing Run Search MAA Launch learning process Event Correlation gt Launch mining process New mined error patterns E Console 23 X Ex 58 A Evra v B OSGi Console r Figure 3 20 Pattern Mining menu Learn application behavior patterns Not many steps are required for launching the learning process just select the menu option Pattern Management Pattern Mining Launch learning process The purpose of this process is to detect sequences of events that represents the normal behavior of the target application This sequences or patterns are stored in a file which is specified in configuration xml file see section 3 1 1 2 If no normal behavior patterns are detected after launching the process with default configuration values the maintenance team can tune some parameter values like the time window of the sequence and the support The smaller is the time window of a sequence or support parameter value the big the number of sequences that will be detected However the number of false positives could be also higher Mine new error patterns Launch mining process is easy just select the menu option Pattern Management Pattern Mining Launch mining process This procedure will analyze a stream of events and will compare them against the normal behavior patterns obtained in the last process The mainte
8. Roehm D Pagano and W Maalej FastFix Deliverable D4 5 1st iteration prototype of the pattern mining module Technical report 2012 Page 47 of
9. Type Cause Name Info Figure 3 18 Cause Tab 3 2 2 2 View and edit pattern of error Using this feature each member of the maintenance team can view all the existing patterns and edit them The steps to view and edit a pattern are the following 1 Access to the edition pattern interface selecting the menu option Pattern Manage ment Event Correlation View error pattern Figure 3 15 2 Select one error pattern from the list and press Edit Figure 3 19 The resulting interface is the same as the last use case with the information of the selected pattern 3 Edit general information the name of the pattern is not editable Page 32 of FastFix Platform User Manual 4 Edit symptoms add and delete operations are also supported 5 Edit cause information 6 Save changes If any mandatory field is empty a message is shown noticing the engineer that must review the tabs and complete all the required information Once a pattern is edited changes are automatically applied in the FastFix correlation system i e restarting the FastFix server is not needed 290 Maintenance Patterns Enter pattern name or subject Name Subject pAttemptedSQLinjectionUpdate FastFix Issue Report A failed attempt of SQL Injection has been detected pEffectiveSQLinyectionDrop FastFix Issue Report An effective SQL Injection has been detected pEffectiveSQLinjectionDump FastFix Issue Report A
10. client and server In the current state of the system the FastFix common namespace includes bundles from nine different areas of concern including context observation persistency error reporting fault replication and communication between client and server More details can be found in 11 The common namespaces currently contains 340 classes with 2 113 methods in 56 pack ages 12 bundles with a total of 17 894 lines of code in 343 files 2 3 eu fastfix targetapplication The targetapplication namespace contains bundles that are supposed to run in the runtime environment of the target application Typically such components are sensors or actua tors Bundles in this namespace communicate with the FastFix client via the interfaces of the application bridge cf 11 The targetapplication namespaces currently contains 117 classes with 824 methods in 32 packages 11 bundles with a total of 9 267 lines of code in 114 files Ph Page 8 of FastFix Platform User Manual 2 4 eu fastfix server The server namespace contains bundles constituting the FastF ix server component The FastFix server runs in the maintenance environment or in the application engineering environment Its main purposes are a to collect information sent by FastFix client b to investigate this information and detect performance degradation trends errors and possible causes c to create and update user profiles d to provide access to issue trackers
11. directory and subdirectories and files More information can be found in the project wiki under http fastfixproject eu wiki Configuration file structure Page 12 of 47 FastFix Platform User Manual Sensor Behavior The behavior of sensors can be configured using the following param eters An example configuration is shown in Figure e heartbeatInterval int The time interval in milliseconds in which sensors send heartbeat commands to the FastFix client to indicate that they are still alive and to receiveinstructions Used by sensors that are implemented in own threads and not as listeners e registerSensorsOnAppStart boolean Determines whether sensors should be registered automatically on application start or by explicit trigger of the user lt Parameters used by the sensors lt Sensors heartbeatIntervalz 6BBH registerSensors n ppStart z truge gt Figure 3 3 Sensor Configuration Datastore Monitored events are stored in the datastore i e MySQL database holding event data The following parameters can be used to configure the database access for FastFix client and FastFix server An example configuration is shown in Figure e driver String The database driver that should be loaded and used in Java e url String The url under which the database can be accessed e db String The name of the database in the MySQL server instance e user String password String Credentials
12. for storage of these files is provided The TargetApplications element configures needed at the client about the applica tions using FastF ix fault replication Its parameters are e AwtInstrumentedClassesSubdir The folder containing the instrumented versions of AWT applications used with fault replication at this client e ModelSubdir The folder containing the GUI model description of AWT applica tions used with fault replication at this client e InstructionsFile The instructions file e EnableAWTAnonymizer Indicates whether GUI anonimization should be used If set to true the error reports of AWT applications are reduced to a minimum Each sub element of TargetApplications represents an application using FastFix in the sample file in the image Moskitt myJPass and Robot This part of the configuration is used by the server Each application has a version and a type either SWT or AWT If it s SWT the location parameter indicates where at the FastFix server the code for that particular version of that application is located If it is AWT the mainClassPath contains the path to the main class of the application and the mainClassName contains the name of the application s main class name The SwtRecorder element is used to contain the SwtRecordedEventsPath which points to the location at the client where events logged by FastFix enabled SW T applications can be logged The AwtRecorder element is used to contain the AwtRecordedEv
13. or using a task scheduler This configuration element is used for this pur pose e automatic boolean If the value of this parameter is true both procedures will be performed according to the cron expression specified in learningSchedule and miningSchedule If the value is false both process should be launched using the FastFix server interface as it is detailed in section e learningSchedule String This field contains the cron expression used to launch the learning process in automatic mode e niningSchedule String This field contains the cron expression used to launch the mining process in automatic mode Page 16 of 47 FastFix Platform User Manual e Results e normalBehaviorPatterns String It refers to the path of the file where the results of the learning process are stored i e the sequences of events patterns which represents the normal behavior of the target application e minedPatterns String It refers to the path of the file where the results of the mining process are stored i e the patterns of error identified Maintenance team can view them using FastF ix interface as it s detail in section lt The PatternMining element has parameters to configure the pattern mining components of FastFix gt PatternMining id fastfix pattern mining configuration lt EventStream eventTimeWindow 864800000 mac pluggedServerDataStore true gt Algorithm chosenAlgorithm prefixpan su
14. taken where no expertise on the application is used This corresponds to the selection of the Automatic Entry Point option in Figure Alternatively some expertise on the application can be taken into account through the use of the expert file Section 3 1 1 4 This allows for improved selection of the methods to be instrumented Therefore this improves on the runtime overhead and the relevance of the extracted models and collected traces Models and collected traces play themselves an important role in the relevance of the generated patches With the use of an expert file for a new application to be self healed using the FastFix platform users must import the source code in Eclipse and define an expert file such as the one in Figure 3 12 This file contains a description of sets of methods of the application that are relevant for monitoring and control i e prevention at runtime Methods whose calls is triggered by user interactions or any other application entry points correspond to such method For instance typical sets of methods are the ones that handle graphical events e g actionPerformed methods of the SWING library etc Ph Page 44 of 5 Summary This document gives a conceptual overview of the FastFix platform describes how to deploy set up and use the FastFix platform for remote maintenance and explains how to extend it for further applicability The document is a supplement to the FastF ix source code which can be accessed
15. AWT mainClassPath pathToRobot mainClassName MainClassName gt lt TargetApplications gt lt SwtRecorder SwtRecordedEventsPath Users fastFIX SvtRecorded guievents AwtRecorder AwtRecordedEventsPath Users fastFIX AvtRecorded FilePrefix ListenersSequence gt Reap reap dir vorkspace fastFIX eu fastfix targetapplication sensor reap workspace dir vorkspace fastFIX LinkProperties target project Robot properties file reap properties lt Reap gt lt FaultReplication gt Figure 3 10 Ticket browser configuration element 3 1 1 4 Patch Generation and Self Healing Several configuration parameters can help tune the Patch Generation and Self Healing component These parameters are illustrated in Figure for the server side as they appear in the FastFix maintenance environment extension for Eclipse The Parameters and Attributes option decides on whether models should take method parameters and attributes into account When not selected the supervisor is less accurate but more efficient at runtime This option can be useful whenever the runtime overhead is expected to be high Automatic Entry Point and ExpertFile Entry Point both relate to the selection of the methods to be instrumented and for which models are extracted The automated option computes entry points automatically performing static analysis of the source code The ExpertFile option allows developers to describe relevant methods to be instrumented Expert
16. Tracker Running FastFix Sensors Figure 3 1 Deployment of FastFix Platform Figure illustrates how FastFix is deployed in typical remote maintenance scenario components machines and connections needed for FastFix are shown in blue The Client MachineT refers to any machine hosting parts of the target application In the case of a simple desktop application this might just be a standard desktop client In the case of a three tier application it might refer to the presentation tier desktop clients logic tier server hosting business logic and data tier server hosting database machines respectively For FastFix a Client Machine refers to any machine where data about the application runtime environment or user can and shall be monitored To this end the FastFix client and sensors are deployed onto these Client Machines The Server Machine is a logical additional machine added to the application deployment scenario which hosts the FastFix server application This machine and the FastFix server is connected to the clients via internet The Tracker Machine denotes the machine where the issue tracker is running typically a stand alone server The FastFix server connects at runtime to this server in order to access the issue tracker The following sections describe on a high level how the FastFix client server and sensors have to be installed Depending on the specific platform and target application the det
17. a a a a a 8 24 eu fastfix server a a a 9 2 0 eu fastfix dependencies a a a 9 20 SUMMARY zo xcw 6 6x9 cosmos diarad oS eG 10 3 User Manual 11 3 1 Deployment and Setup a 0 200 000 2 11 3 1 1 FastFix Configuration 12 3 1 2 FastFix Client Setup 0 0 0 0 0 000 020002 24 3 1 9 FastFix Server Setup 2 a a 25 3 1 4 FastFix Sensors Setup 2 26 narran 29 3 2 1 Context Observation 2 20 0 0 a 29 ee ee E 30 14 99995 99995 E 33 3 224 Error Reporting xo ok REORUM hue obo eee he PE SP S RS 36 eee 993 eee ee eae eeee nee eee ee 37 3 2 6 Patch Generation and Selt Healing 38 4 Developer Manual 40 po eee e epee nt eee se ee ee ee eae es 40 4 2 Feature Extensions 40 4 2 1 Context Observation teca a rra X 409 AAA 40 TRE 42 4 2 3 Pattern Mining e e conos oe hee dS 9 RUE Se ee Oe 4e ox 42 4 2 4 Error Reporting 2222s 43 ra ee eee 44 4 2 6 Patch Generation and Self Healing 44 5 Summary 45 Bibliography 46 E Page 3 of List of Figures rado 7 2 2 FastFix client platform UL 8 2 3 FastFix server platform UL 9 crasas Bae ee baw ee SES 11 AA 12 AEREA 13 TOP 13 A 15 TA 17 3 7 Ontology configuration element 0 000000 08 18 3 8 Communication configuration element 004 18 3 9 Issue tracker configuration element
18. ails of these steps may vary and additional steps may be required We put the term Client Machine in quotation marks to indicate that these machines are not necessarily clients in terms of the target application language but in terms of FastFix Note that the FastFix Server application might in theory also be installed on the Tracker machine or the Client machine Ph Page 11 of 47 FastFix Platform User Manual In Section 3 1 1 we show how to configure the FastFix platform components In the following two sections we describe how to obtain and run the FastFix client Section 3 1 2 and server Section 3 1 3 Finally in Section we illustrate how to set up and connect FastFix sensors 3 1 1 FastFix Configuration Both FastFix server and client should be configured with a configuration file in the Fast Fix home directory The configuration file allows maintenance engineers to specify the behavior of the following components 3 1 1 1 Context Observation Communication between sensors and FastFix client The application bridge is the component of the FastFix client to which the sensors communicate The configuration parameters shown in Figure are used both by the FastFix client and the sensors to communicate In the following we briefly describe each configuration parameter e sensorsAutoStart boolean Determines if sensors are started automatically when they are registered or if this has to be done manual
19. ains all utilities for GUI anonymization If you run java jar GUIAnon jar you ll be provided with all available options of GUIAnon see Figure 3 2 5 1 3 below Usage Help java jar GUIAnon jar h Launch app java jar GUIAnon jar launch MainClass Instrument java jar GUIAnon jar instrument MainClass Rip Graphical Interface java jar GUIAnon jar rip MainClass Record Events java jar GUIAnon jar recevents MainClass Record Listeners java jar GUIAnon jar reclisteners MainClass Convert Tracefile java jar GUIAnon jar convertl2e guistructure tracefile Generate anonymized java jar GUIAnon jar anonymize guistructure tracefile MainClass Generate anonymized java jar GUIAnon jar smartanonymize guistructure tracefile MainClass Figure 3 25 GUIAnon usage instructions To instrument the application assuming a main in MainClass we must run java jar GUIAnon jar instrument MainClass which generates an instrumented version of the ap plication in a folder placed in the same parent folder as the main class and which is called instrument result Then we must extract the graphical model of the AWT application by running java jar GUIAnon jar rip MainClass Select all options available in the appli cation s GUI click every button open every menu etc Then close the application which generates a file called model txt Just as in the case of console applications in order to deploy the application
20. alue of this field represents the minimum number of occurrences in percent of a sequence of context events to be considered as frequent e minimumSequenceSize int It refers to the minimum number of context events needed to become a sequence e sequenceTimeWindow long It refers to the value in milliseconds of the time window of a sequence e itemSetTimeWindow long It refers to the value in milliseconds of the time window of a itemSet A sequence consists of one or more item sets so the value of sequence Time Window must be higher than ztemSetTime Window e converter utput String The stream of events provided by the server data store must be serialized using a concrete format in order to be processed later by learning process This field contains the path of the file used for this purpose e session0utput String Once the events have been formated and serialized the context events are grouped into sequences The result of this process is stored in a file which will be read later by the chosen algorithm This field contains the path of this file e preprocessed Output String This field means the same than converterOut put but for the mining process e machineLearning String It refers to the path where the executable jar for learning process is located This jar contains PrefisSpan and CSPADE algo rithms e LaunchProcess Learning and mining process can be launched in two modes on demand
21. anually Ph Page 27 of 47 FastFix Platform User Manual 3 1 4 3 Log Sensor Technology and Framework Dependencies The FastFix Log sensor monitors log files and new log entries appended to them Hence it can be used for all applications writing log files It can be directly used for applications writing log entries using the same pattern as PostgreSQL or MySQL error messages and has to be adapted for other log entry patterns In the following we use Espigon as an exemplary application writing information about errors to a log file Requirements The following is required to deploy the FastFix Log sensor on a machine e Java 1 5 Runtime Environment installed Sensor Installation The following steps are necessary to install the FastFix Log sensor The the FastFix Log sensor is installed by copying the binary and running it from console 1 Download the latest build of the Log sensor 2 Move the downloaded jar into the espigon tomcat logs folder of the Espigon ap plication hosted within a Tomcat server 3 Configure the application bridge and the general sensor settings see Section 3 1 1 1 The Log sensor is now installed Perform the following steps to run it 1 Open the folder espigon tomcat logs in a console window 2 Run the Log sensor by entering the command java jar logsensor jar f catalina out p PostgreSQL from this console 3 Do not close the console window The Log sensor is now run
22. ault replication component See the handleContext of the eu fastfix client faultReplication internal ContextListenerImpl class for which of the event s fields are used in the creation of an error report T he only mandatory fields is the summary If additional auxiliary files need to be sent to the server their absolute paths at the client should be included as event fields and the processing of the fields at eu fastfix chient faultReplication internal ContextListenerlmpl correspondingly extended Extending the number of supported issue trackers can be done by implementing the ser vices described in the eu fastfix server error reporting abstractions service classes as done in the eu fastfix server error reporting trac and eu fastfix server error reporting eventum he Page 43 of FastFix Platform User Manual examples The only restriction is that the insertion of a ticket in the issue tracker should return an unique integer identifier because this identifier is used to name the folder on the server where any auxiliary logs are stored As long as this is ensured FastFix will behave correctly with any issue tracker 4 2 5 Fault Replication There are basically two approaches to additional development of FastFix fault replication extending current record replay components or creating new ones The extension of current record replay depends on the the component being extended In the case of the current console and AWT sensors appl
23. ce contains third party libraries which we have wrapped in OSGi bundles This process is sometimes necessary to be able to use specific libraries in an OSGi context We created OSGi wrappers for the following 9 third party components e eu fastfix dependencies apache http wrapping org apache http e eu fastfix dependencies axis wrapping org apache axis e eu fastfix dependencies drools wrapping org drools e eu fastfix dependencies flexjson wrapping net sf flexjson e eu fastfix dependencies javassist wrapping javassist e eu fastfix dependencies jena2 wrapping com hp hpl jena e eu fastfix dependencies jpf wrapping Java PathFinder e eu fastfix dependencies mysql wrapping mysql jdbc connector e eu fastfix dependencies soot wrapping soot polyglot and jasmin Page 9 of FastFix Platform User Manual 2 6 Summary Table summarizes the code metrics of the FastFix open source remote maintenance platform Table 2 1 Code Metrics of Integrated FastFix Platform A bles FOC eufastfixdependencies o Page 10 of 47 3 User Manual In this section we provide a user manual for FastF ix i e we describe how the FastFix platform is typically deployed which steps are necessary to set the platform up and how it can be used for remote maintenance 3 1 Deployment and Setup Client Machine Running Target Tracker Machine Application Running FastFix Client Running Issue
24. ch field is explained and some tips to correctly configure them are explained in section 3 2 3 e EventStream Pattern mining procedures use context event streams as input data This configuration element concerns the parameters of the set of context events to be analyzed by pattern mining module e eventTimeWindow long It refers to the value in milliseconds of the time window of the event stream i e if maintenance engineers team want analyze the context events occurred in last 4 hours the value of this field would be 14400000 milliseconds e mac String The MAC address of the user whose events would be analized by the module If no MAC address is specified all the events are taken into account e pluggedServerDataStore boolean This field is used for development pur poses If its value is true the pattern mining module will be fed with real Page 15 of 47 FastFix Platform User Manual context events provided by the server data store component However some times simulating the events is useful specially for testing new pattern mining features in this case false is the appropiate value The correct value for pro duction environments is true e Algorithm This configuration element refers to the parameters used by the learning and mining procedures e chosenAlgorithm String It refers to the algorithm used in the learning phase Possible values are PrefixSpan and CSPADE e support long The v
25. e Self Healing and Patch Generation Component FastFixSH Technical report 2011 IS B Gaudin R Yates and M Hinchey FastFix Deliverable D6 6 4th Prototype of the Self Healing and Patch Generation Component Technical report 2013 9 E Guzm n A Mahmuzic J Garcia and W Maalej FastFix Deliverable D3 3 1st Prototype of the Context Observer Technical report 2011 10 D Pagano E Guzman J Cano N Narayan A Mahmuzic A Waldmann S Za marripa L pez D De los Reyes J Garcia and W Maalej FastFix Deliverable D2 2e Integration Plan and Technical Project Guidelines extended version Tech nical report 2012 111 D Pagano T Roehm E Guzm n S Zamarripa L pez J Garcia B Gaudin J Cano and W Maalej FastFix Deliverable D2 4 Architecture Changes and Change Rationales Technical report 2012 12 T Roehm J Garcia and D Pagano FastFix Deliverable D3 6 Refined and Inte grated Version of Context Observer User Profiler and Error Reporting Technical report 2012 Ph Page 46 of FastFix Platform User Manual 113 T Roehm D Pagano S Zamarripa L pez and M del Carmen Calle FastFix Deliverable D3 4 1st Prototype of the User Profiler Technical report 2012 14 S Zamarripa L pez and M del Carmen Calle FastFix Deliverable D4 6 Second refined prototype of the event correlation component Technical report 2012 15 S Zamarripa L pez M del Carmen Calle E Guzm n T
26. e metrics In Section 3 we describe how the FastFix platform can be deployed set up and used for remote maintenance In Section 4 we describe how developers can config ure extend and change the FastFix functionality in order to be able to provide remote maintenance services tailored to their customers requirements Section 5 summarizes and concludes this document Page 6 of 2 Platform Overview FastFix is a remote software maintenance platform As Figure 2 1 illustrates its compo nents are distributed among two environments First the environment where the target application is running and second the maintenance environment where development maintenance and testing tools are installed Consequently the FastFix system consists of two main components the FastFix client and the FastFix server Both server and chent communicate and exchange data with each other Runtime Maintenance Platform I 1 I 1 ES 1 2 i o C 1 I E B 2 1 oD E 1 c c I I 1 un 1 6 e l i pa NS 1 O I I LL gt I E 1 12 o I N L o I y 1 1 Lu I 1 I I LL i l e o 1 a 1 Uv 1 a QM EE I lt I I I ud 1 o I 1 I 5 1 di 1 1 BE au EO I dE Y I 1 O I U I q 1 i N 2 J E I I C d c I I I I O 1 I 1 1 c I I gt l e i I I E L o 1 I j 1 LLI 1 Q 1 I ie I 4 I E 1 I I G I O I I I aD I V I 1 gt x lt i o I 1 AE e 1 OL 4 1 VU I 1 a 1 e 42 E I I O I 1 E
27. entsPath which points to the location at the client where events logged by FastFix enabled SWT applications can be logged The FilePrefix element designates the prefix usaed to name these logs Finally the Reap configuration element configures the anonymization of user input text reap dir points to the location at the client of the anonymization bundle eu fastfix tar getapplication sensor reap and workspace dir should point to a folder on the client that can be used by the anonymizer to store temporary files Page 20 of FastFix Platform User Manual The FaultReplication element has parameters for the fault replication component of FastFix gt lt FaultReplication id fastfix fault replication configuration lt The LogStore element has parameters to copy the logs to a remote server gt lt LogStore host test domain com port 22 user fastfix replication password IHaveNone location fault replication logs lt The TargetApplications section provides information about Applications which are used to replay faults gt TargetApplications AwtInstrumentedClassesSubdir InstrumentResult ModelSubdir structure Model txt InstructionsFile Users fastFIX targetApp txt EnableAwtAnonymizer true gt lt Moskitt version version type SWT location pathToMoskitt gt lt myJpass version 1 0 type AWT mainClassPath pathToMyJpass mainClassName MainClassName gt Robot version 1 0 type
28. ers to change such parameters using a mechanism called unified configuration see also Section 3 1 1 All components can be configured in a single place namely a configuration file in XML format The FastFix project wiki describes in detail how this configuration file is structured To guarantee the extensibility of FastFix the configuration parameters for all compo nents can be extended In addition new elements can be created for newly developed FastFix components Details on the steps required to extend the configuration mechanism are described in the FastFix project wikj 4 2 Feature Extensions The following sections show how to extend the existing FastFix functionality to work with additional applications and technologies for instance to satisfy additional monitoring needs and detect additional errors 4 2 1 Context Observation 4 2 1 1 Implementation and Integration of New Sensors Every application can be instrumented and integrated into the FastFix framework The instrumentation can be done on source code or byte code level On source code level additional statements are added in the programming language used that log the occur rence of certain events as specified by the developer of the application In a scenario where a virtual machine is used to execute a software application monitoring instructions can be added on byte code level without having access to source code of the application Instrumentation can be done man
29. es and click Next and Finish accepting the license agreement 6 Close MOSKitt 7 Configure the application bridge and the general sensor settings see Section 3 1 1 1 The RCP sensor is now installed If sensors are configured to start automatically upon application startup it will start when MOSKitt is started Otherwise the sensor has to be started manually via the FastFix menu added to the MOSKitt main menu Formerly called MOSKitt sensor Ph Page 26 of FastFix Platform User Manual 3 1 4 2 Struts Sensor Setup Technology and Framework Dependencies The FastFix Struts sensor monitors HTTP requests and corresponding actions defined by Struts web application framework Hence it can be used for all applications hosted in Tomcat that are using Struts In the following we use Espigon as an exemplary web application based on Struts and hosted in Tomcat Requirements The following is required to deploy the FastFix Tomcat sensor on a machine e Local Tomcat server instance with Espigon web application installed e PostgreSQL Server 8 1 or higher installed e Java 1 5 installed Sensor Installation The following steps are necessary to install the FastFix Struts sen sor The Struts sensor is installed by copying downloaded libraries into certain lib folders and modifying configuration files of a Tomcat web application 1 Download the latest build of the Struts sensor 2 Unzip the downloaded file into a directory espig
30. f the FastFix Self Healing component Figure 3 26 Search Project Run SUAFA Pattern Management Window Help Model Generation Configurations MSN M Pattern Generation gt L EG ea av Patch Generation J Patch Mergence 4 Patch Deployment L Duro aman Figure 3 26 FastFix Self Healing plugin menu This menu matches the different steps describe above except for Point 2 that is per formed on the running application and is therefore controlled by parameters in the FastFix client Testing Mode for Trace Collection in this case It is worthwhile noting Point 2 and 3 are actually optional When Point 2 is performed it allows for more relevant patch validation When Point 3 is also performed it allows for an alternative patching strategy Page 39 of 4 Developer Manual In this section we describe how developers can configure extend and change the FastF ix functionality in order to be able to provide remote maintenance services tailored to their customers requirements 4 1 Configuration Extensions The FastFix platform consists of several components running on two distributed systems These components provide specialized services to other components which can be config ured in order to set the components up for a specific runtime environment For instance the FastFix client listens for sensor calls on a specific port However since the default port might be reserved on some machines FastF ix allows us
31. files are xml files as presented in Figure 3 12 Page 21 of 47 FastFix Platform User Manual Preferences type filter text Self Healing roo General z Agent Controller Self Healing settings Ant vf Parameters and Attributes dataStore E pm eR errorReportGeneration LJ ExpertFile Entry Point FastFix Server Iv Automatic Entry Point Help Install Update Java Maven Memory Analyzer Plug in Development Profiling and Logging Run Debug Runtime Self Healing k Self Healing serverConnection Tasks Team Test Usage Data Collector Validation XML ih a O O r Cane y rf Cancel OK Figure 3 11 FastFixSH configuration attributes Expert files describe sets of method declarations Method name declaring class name declaring parent class name and declaring package name is the information that can be used in the file Each set of methods is described within the lt method gt lt method gt tag lt methodName gt lt methodName gt is a sub tag that can contain a regular expression that relevant methods name must fulfill lt class gt lt class gt tag contains information about the declaring class of the relevant methods It is possible to enter a regular expression in the lt className gt lt className gt tag in order to filter the declaring classes according to their names Page 22 of FastFix Platform User Manual lt method gt lt Specify the name of the Method methodA gt lt methodName
32. for MySQL server e cache int The size of the event cache for database access i e the number of events that are cashed by FastFix client and FastFix server and written to database together xl The Datastore element has parameters to configure the datastore component of FastFix xDatastore id fastfix datastore configuration xl The Connection element has parameters to connect to a database both in client and server Connection driverz com mvsql jdbc Driver urlz jdbc mvsql lacalhost FastFixServer dbz FastFixServer user DbUsername password DbPassword cache 1 7s z Datastore Figure 3 4 Datastore Configuration Page 13 of 47 FastFix Platform User Manual 3 1 1 2 Event Correlation and Pattern Mining Event correlation component FastFix event correlation system will automatically de tect an error pattern during the application s operation generating automatic tickets and gathering all the information that can be useful for further analysis In order to prop erly configure the event correlation components we need to define the file system folders where the required files are located This configuration is defined by specifying several file paths which are located under the FastFix home directory Event correlation con figuration allow the developer to disconnect the module from both the context and the report system e DRL String Both items path and url must point to the same resource mainte na
33. gt actionPerformed lt methodName gt l Specify the information of the Class initiated the methodA gt class lt c LassName gt lt parentClassName gt ActLonListener lt parentt LassName gt lt packageName gt lt class gt lt method gt lt The method that you want to instrument as EntryPoints before extracting FastFix models gt lt method gt l Specify the name of the Method methodA gt lt methodName gt run lt methodName gt lt controllable gt true lt controllable gt l Specify the information of the Class initiated the methodA gt class className lt parentC LassName gt Thread lt parentC LassName gt lt packageName gt lt class gt lt method gt lt method gt l Specify the name of the Method methodA gt lt methodName gt newF i Lter lt methodName gt lt controllable gt false lt controllablep l Specify the information of the Class initiated the methodA gt class className ControlWindow className lt parentCLassName gt lt packageName gt gu1 lt packageName gt lt class gt method expertFile Figure 3 12 Sample Expert File It is also possible to use the lt parentClassName gt lt parentClassName gt to enter a regular expression that one of the parent classes or interfaces name must fulfill Finally the lt packageName gt lt packageName gt tag contains a regular expression that the package in which the method is declared m
34. hase it s recommended change the value to false in order to not flood the tracking system The correct value for production environments is true e post boolean It refers to the granularity of the information shown in reports ie if its value is true the module includes extra information about some context events e CorrelationData e path String It refers to the folder where the collected results are going to be stored in other words the location of the file containing information about the error being detected as well as the client name and other configurable fields described in the configurable fields described in generalConf and rdfConf Ph Page 14 of FastFix Platform User Manual e generalConf String It is the folder of the properties file describing the general fields to be retrieved from the events associated to the current detected error The general fields consist of a group of default fields OntologyId ClientName UID and a list of configurable fields defined inside the properties file The properties file is a list of name value pairs where the name represents the column in the generated csv file while the value represents the event data property to be retrieved from the FastFix events e rdfConf String It refers to the folder of the properties file describing the rdf fields to be retrieved from the server datastore corresponding to previous events that might be associated with the curre
35. ications are being monitored by using bytecode instrumentation that resorts to the SOOT tool http www sable mcgill ca soot For SWT applications are monitored using system graphical events The logs generated by the record procedure are transferred with the error reports to the FastFix server If any changes are made to the recorder the replayer must be updated to reflect the changes in recording instrumentation and to inject into the replayed application the additional information being recorded Creating new record replay components involves basically two steps develloping a sensor see Section 4 2 1 that detects the events that are relevant to the record and replay you want to perform and which sends the FastFix client an ERROR REPORT EVENT event and update the replay mechanism in the ticket browser see makeActions method in the eu fastfix server maintenance ticketbrowser views FastFizTicketBrowser class so that the result of the record procedure can be replayed The ticket browser uses tags embedded in the error report summary to decide on the replay method 4 2 6 Patch Generation and Self Healing The FastFix patch generation and self healing component tackles any Java application In order to be applied the source code of the application must be available and imported into Eclipse The main features of the self healing component are indeed accessed through an Eclipse plugin see Figure 3 26 A fully automated approach can be
36. il in Section The interface consist of three tabs General Information Figure 3 16 Symptoms Figure and Cause Figure 3 18 Via these tabs users can enter the required informa tion to create a pattern of error The symptom tab is the most important one since the symptoms contains the conditions to be matched by context events to infer the error emergence The more different types of events are detected by sensors the greater the coverage of errors since the system can use a bigger variety of symptoms to conform a greater variety of patterns Thus to extend the error type coverage of FastFix when the system is applied to monitor new applications developers must take into account also how to implement and extend sensors which it is explained in section 4 2 1 4 2 3 Pattern Mining From a developer s perspective pattern mining is an extension mechanism since it dis covers new patterns of unexpected behavior by analyzing and processing the executed event sequences associated to the application environment Because of its connection with the event correlation component it might be consid ered as the main method to extend FastFix to support error detection for additional applications technologies and especially error types Nevertheless there are some considerations that should be taken into account in order to conveniently mine new patterns depending on application specific properties i e when it comes to average
37. in the FastFix source code repository on SourceForgd and complements the detailed Javadoc documentation of the FastFix source code as well as the project Wikd https svn code sf net p fastfixrsm code trun gt n iki Main Page Page 45 of Bibliography 1 M del Carmen Calle and S Zamarripa L pez FastFix Deliverable D4 4 First iteration prototype of the Event Processor Technical report 2012 2 J Garcia P Romano L Rodrigues N Coracao N Machado and J Matos FastFix Deliverable D3 5 Ist Prototype of the Error Reporting Technical report 2011 3 J Garcia P Romano L Rodrigues N Coracao N Machado and J a Matos Fast Fix Deliverable D5 3 1st prototype of the execution recorder replayer tool Technical report 2011 4 J a Garcia P Romano L Rodrigues J a Matos J a Nuno Silva J a Barreto and T R hm FastFix Deliverable D5 4 2nd prototype of the execution recorder replayer tool Technical report 2012 5 B Gaudin Z Cui P Monjallon and M Hinchey 3rd Prototype of the Self Healing and Patch Generation Component Technical report 2012 16 B Gaudin M Hinchey P Nixon and N Al Haider FastFix Deliverable D6 4 2nd Prototype of the Self Healing and Patch Generation Component FastFixSH Technical report 2012 7 B Gaudin M Hinchey P Nixon R Ali and N Al Haider FastFix Deliverable D6 3 Ist Prototype of th
38. it must be connected to the sensor that will monitor potential application crashes unhandled exceptions The shortcut to start the application should point to the sensor s main class eu fastfix targetapplication sensor javaapplication Start JavaApplicationSensor 3 2 6 Patch Generation and Self Healing In its current state the FastFix Self Healing component applies to applications written in Java and for which the source code is available A typical use case for self healing and patch generation follows the steps below Ph Page 38 of FastFix Platform User Manual 1 Instrumenting source code and generate application models 2 Collecting a set of traces representing desired behaviors This can be done by executing passing tests again on the instrumented application The Testing Model for Trace Collection option in Figure should be enabled for this 3 Extracting patterns This is achieved using the traces collected in Point 2 This results in more application models that will be loaded at runtime with the ones extracted in Point 1 Point 1 2 and 3 are performed just before the application is deployed 4 Performing patch generation This is done once the application is deployed and new files are received on the server side 5 Patch merging and deployment Once a new patch has been generated it can be merged with the currently applied one and the outcome can be deployed These different steps are also illustrated in menu o
39. ized in the following list e First Prototype of Context Observer 9 e First Prototype of the User Profiler e First Prototype of the Error Reporting System e Refined and Integrated Version of Context Observer User Profiler and Error Re porting e First Prototype of the Event Processor e First Prototype of the Pattern Mining Module e Second refined prototype of the event correlation component e First and Second Prototype of the Execution Recorder and Replayer e First Second Third and Fourth Prototype of the Self Healing and Patch Genera tion Component 7 6 This document does not describe in detail how to set up a development environment in order to build the FastFix platform from source code FastFix is a relatively big system and the project utilizes specialized technologies for the build process Therefore in this document we concentrate on giving additional conceptual information on the source code and illustrate deployment and usage scenarios For information on how to start developing for FastFix we refer the reader to and to the project s Wiki documentation iff In Section 2 we give a conceptual overview of the final FastFix platform We illustrate how and why bundles are distributed among different namespaces and give details using https svn code sf net p fastfixrsm code trunk http fastfixproject eu wiki Howto Set up development infrastructure Page 5 of FastFix Platform User Manual cod
40. ly by the user via the FastFix client e RMIBridge active boolean Determines whether the RMI bridge i e the component of the FastFix client used for communication via RMI is available or not e RMIBridge host Hostname port Port and name String Setup for the RMI connection This is used by the FastFix client to provide services via RMI and the sensors to connect to FastFix client and consume the services e HTTPBridge active boolean Determines whether the HT TP bridge i e the component of the FastFix client used for communication via HTTP is available or not e HTTPBridge host Hostname and port Port Setup for the HTTP connection lt The ApplicationBridge element has configuration about the RMIBridge and the HTTPBridge x ApplicationBridge id z fFastfix applicationbridge configuration zensors utoStart z true 1 RMI bridge in use or not with location host and port and name to locate it gt xRHMIBridge activez true host localhost port 1699 namez RMIBridge r zxl HTTP bridge in use or not with location host and port gt lt HTTPBridge active true host localhost portz 118B gt x ApplicationBridge Figure 3 2 Configuration of Application Bridge The FastFix home directory is a the folder specified by the Java environment property fastfix home or b the folder fastfix in the user s home directory FastFix expects read and write permissions in all the FastFix home
41. m there they can be controlled via the user interface as described in 3 1 2 During the FastFix project several sensors were developed The general configura tion applicable to all sensors is described above in Section The setup of sensors developed during FastFix is described in detail in this section 3 1 4 1 Eclipse RCP Sensor Setup Technology and Framework Dependencies The FastFix RCP senso monitors user actions and exceptions of RCP applications Hence it can be used for all applications based on Eclipse and RCP In the following we use MOSkitt as an exemplary Eclipse RCP application Requirements The following is required to deploy the FastFix RCP sensor on a machine e Java 1 5 installed e MOSKitt installed Sensor Installation The following steps are necessary to install the RCP sensor Overall the RCP sensor is installed using the RCP Eclipse update mechanism 1 Download the update site of the latest build of the RCP sensor 2 Unzip the downloaded file into a directory moskittsensors 3 Start MOSKitt 4 Navigate to Help gt Install new Software in the main menu Make the the down loaded update site known by clicking on Add then clicking on Local and selecting the moskittsensors folder Specify FastFix MOSKitt Sensor as name of the update site and acknowledge with Ok 5 Select the new entry FastFix Moskitt Sensor in the list of potential software up dat
42. n effective SQL Injection has been detected pTextinputValidation FastFix Error Report Input validation issue detected pEffectiveSQLinjectionUpdate i FastFix Issue Report An effective SQL Injection has been detected pAttemptedSQLinjectionDrop FastFix Issue Report A failed attempt of SQL Injection has been detected pAttemptedSQLInjection i FastFix Issue Report A failed attempt of SQL Injection has been detected pUnderAtacck FastFix Issue Report Application under SQL injection attack Figure 3 19 List of existing patterns of error 3 2 3 Pattern Mining Apart from the predefined error patterns and ones added by the maintenance team Fast Fix correlation system provides a pattern mining module to suggest unknown patterns The process to mine them consists of sub processes the learning and mining procedures and can be launched in two modes on demand or automatically i e using a cron expres sion To use this feature automatically see section 3 1 1 2 First two use cases are related with these sub processes in case on demand launching The third one is validate the discovered pattern i e once a pattern is discovered a member of the maintenance team must validate it in order to add it to the correlation module By the other hand if engineers consider the new pattern as a false positive they can discard it As others FastFix correlation system use cases maintenance team can access them through the FastFix Server menu Page
43. nance team can view the mined patterns using the menu option Pattern Manage ment Pattern Mining New mined error patterns A dialog will present the list of new patterns Figure 3 22 and engineers can look at the details of the pattern An example is illustrated in Figure 3 21 Page 34 of FastFix Platform User Manual Pattern Mining New Mined Patterns Pattern Detail p024502362028 IE xLinp i Sac Wi oat A th P EG 3 PressButton Fri Dec 15 03 05 23 CE Listado de transeuntes HttpRequestOnServers Fri Dec 15 03 05 23 CE espigonvalencia war perfil do DbStatementEvent Fri Dec 15 03 05 23 CE Select from cod pue where s ApplicationEvent Sat Mar 14 19 58 43 Cl espigonvalencia war perfil do Figure 3 21 Detail of a mined pattern Discard error patterns Maintenance team users can validate or discard mined patterns Both features are acces sible in the interface where the mined patterns are listed via buttons In case that the maintenance team member considers that the pattern is a false positive or a duplicate they can discard the pattern using the button Discard Pattern Once a pattern is dis carded it is automatically deleted in the FastFix correlation system so it is not necessary to restart the FastFix server 20 Q Pattern Mining New Mined Patterns aterm p3636473612 Mon Dec 10 12 49 09 CET 2012 New Figure 3 22 Dialog showing a new mined pattern Validate error patterns If the e
44. nce rules drl This file contains the rules which are used to detect patterns of error and it s an internal i e it is not recommended to change the default value since it must be in the classpath of FastFix It is used by different classes which use different methods to load the file So it is necessary to use two different parameters e Changeset String The item path points to the folder to be scanned in order update the rule engine with changes in the rules It s also an internal file so it is not recommended to change the value e Context5ystem e plugged boolean This field is used for development purposes If its value is true the event correlation module will be fed with real context events How ever sometimes simulating the events is useful specially for testing new event correlation features in this case false is the appropiate value The correct value for production environments is true e nonAnonymousEvents String It refers to the context event types that contain information about the user who create it For example http www fastfixpro ject eu ontologies MonitoringOntology owl77 TextInput http www fastfixpro ject eu ontologies MonitoringOntology owl7 HttpRequestOnServerSide e UserReportingsystem e plugged boolean This field is used for development purposes If its value is true the event correlation module inserts a ticket in the issue tracker if a fault is detected However in development p
45. ng Run Search Pattern Mining gt View error patterns Add new error pattern E Console 23 X Ee bE r EBvrjv B OSGi Console H Figure 3 15 Event Correlation Menu 3 2 2 1 Create a pattern of error FastFix includes several error patterns by default but the maintenance team can add more based on their expertise in the target application The steps to create a pattern of error are the following 1 Access to the creation pattern interface selecting the menu option Pattern Manage ment Event Correlation Add new error pattern This interface consists of three tabs General Information Symptoms and Cause 2 Fill general information fields All of them are mandatory Figure 3 16 e Pattern Name Represents the name of the pattern It will be used to identify the pattern e Subject This field represents the subject of the report associated with this new pattern This report will be inserted in issue tracker Jira Eventum or Trac when current pattern of error occurs e Summary In the same way that Subject this field must content the summary of the report associated with this new pattern 3 Add symptoms This step is the most important since the symptoms contains the conditions to be matched to detect a pattern For each symptom some fields must be inserted The following are the most important but all of them are explained in detail in D4 6 Figure 3 17 e Event Type This field represents the context e
46. ngineer consider the mined pattern is not a false positive he can validate it The process to validate a pattern is the same than creating a new one see section 3 2 2 1 he Page 35 of FastFix Platform User Manual The difference is that some information about the symptoms are pre inserted Once a mined pattern is validated it is automatically inserted in the FastFix correlation system so it is not necessary to restart the FastF ix server gt 6 Pattern Mining Create New Maintenance Pattern General Information Symptoms Cause Event Type Symptom Operator Value Info Reaction Delay Reuse Sympton ApplicationEvent s02 i Jespigonvalencia war perfi i i i DbStatementEvent 528 Select from cod_pue whi HttpRequestOnServerSide s20 Jespigonvalencia war perfj PressButton 36 Listado de transeuntes Textinput s45 3 Possible anomaly If you choose an operator you have to fill the field Value since the operator is used to evaluate it In the same way if you fill the field Value you have to choose an operator If you fill the field Reaction you have to fill the field Delay since reaction event should occur at most as many seconds as specified in Delay field In the same way if you fill the field Delay you have to fill Reaction field Criterion KA Delete Symptom Add Symptom Lud Sake fatten Figure 3 23 Validating error patterns 3 2 4 Error Reporting Error repo
47. ning 3 1 4 4 WCF Sensor Setup The WCF sensoi monitors the content of WCF messages Hence it can be used for all applications developed in NET that are using WCF In the following we use U AT Execute as an example of a NET application whose WCF messages are monitored Requirements The following is required to deploy the WCF sensor on a machine e Windows operating system l Formerly called TXT Execute sensor Ph Page 28 of FastFix Platform User Manual Sensor Installation The following steps are necessary to install the WCF sensor 1 Download the latest microsoft installation file msi extension of the WCF sensor 2 Run the installer by double clicking on the file and choose the desired location for the installation of the sensor 3 Open the IIS Manager and select the ExecuteService site Then on the right side double click Modules 4 Click on Add Managed Module on the upper right and insert the following into the resulting dialog box a Name FastFixHttpModule b Type FastFix FastFixHttpModule FastFixHttpModule Version 1 0 0 0 Culture neutral PublicKeyToken 6fead1b590d37761 Sensor Configuration In order for the sensor to run with the start of the TXT client application The Web config file from the TXT service needs too be modified To do so the following steps are necessary 1 Open the execute service folder where the ExecuteService is deployed and add the f
48. nt situation of error Again this properties file represents several name value pairs where the first element describes the name of the column in the resulting csv file while the second is the event data property to be retrieved The following figure represents an example of the event correlation configuration element of the FastFix configuration file The EventCorrelator element has parameters to configure the event correlator components of FastFix gt EventCorrelator ids fastfix event correlator configuration DRL pathe classpath drl maintenance rules drl urle file drl maintenance rules drl gt Changeset pathe gt ContextSystem plugged true nonAnonymousEventss http www fastfixproject eu ontologies MonitoringO ntology owl TextInput UserReportingSystem pluggede true poste false gt lt CorrelationData pathe eventCorrelation collectedFields csv generalConfe eventCorrelation generalFields properties rdfConfes eventCorrelation dataStoreFields properties gt EventCorrelator Figure 3 5 Event correlation configuration element Pattern mining component Apart from the predefined patterns of error of FastFix correlation system provides a pattern mining module to suggest unknown patterns The process to mine them is made by two sub processes the learning and mining procedures Pattern mining configuration is key for having good results In this section the meaning of ea
49. of the FastFixed application includes the following FastFix OSGi bundles eu fastfix client faultReplication guiRecorder eu fastfix client fault Replication sensing eu fastfix chent faultReplication and eu fastfix common faultReplica tion Gut 3 2 5 1 2 Console Applications In the case of console applications the compiled ap plication needs to be instrumented in order to support FastFix record replay Assuming Ph Page 37 of FastFix Platform User Manual that the main class of the console application is called MainClass We must run java cp path to FastFir server plugins eu fastfiz targetapplication sensor reap edu hkust leap transformer LEA P Transform Xmz2g MainClass This creates a version of the application for running at the client in a subfolder called instrumentResult Both the original and the instrumented Java class files of the application need to be copied by the deploy ment process to the client using them In order to deploy the application it must be connected to the sensor that will monitor potential application crashes unhandled ex ceptions The shortcut to start the application should point to the sensor s main class eu fastfix targetapplication sensor reap StartSensor 3 2 5 1 3 AWT Applications In the case of AWT the machine where applications are instrumented must have the GUIAnon jar which comes from generating an executable jar of eu fastfix targetapplication sensor javaapplication and cont
50. ollowing under the lt configuration gt tag of the Web config file lt appsettings gt add key sensorBridgeURL value http localhost 9999 gt lt add key callTimeoutMS value 3000 gt lt add key filterURL value service gt lt add key logEnabled value true gt lt add key logfile value fastfixsensor txt lt appSettings gt 2 Save and close the file the Web config file 3 Restart the ExecuteService Web Site in the IIS Manager 3 2 Platform Usage The following sections describe typical use cases of the FastFix platform features in detail specifying current requirements and configuration possibilities 3 2 1 Context Observation The main use case of FastFix sensors is context monitoring 1 e collecting data that is used by other components such as event correlation fault replication and self healing However manual interaction by the maintenance engineer is required only for installing and configuring sensors How to setup FastFix sensors is described in Section 3 1 4 Ph Page 29 of FastFix Platform User Manual 3 2 2 Event Correlation The pattern of an error is the key piece of the FastFix event correlation system So the use cases for maintenance engineers are related with its management create and edit patterns of error Each member of the maintenance team can access both features through the FastFix Server menu 200 Fastfix Server File Help Pattern Management Self Heali
51. onsensors 3 Stop Tomcat 4 Copy the following jars from espigonsensors in the webapps EspigonValencia WEB INF lib folder of the Espigon web application eu fastfix targetapplication sensor tomcat struts jar eu fastfix common applicationbrige jar eu fastfix common configuration jar eu fastfix common logging jar flexjson 2 1 jar 5 Modify the webapps EspigonValencia WEB INF struts config valencia xml file In a clean Espigon install it contains the value controller processorClass org apache struts tiles TilesRequest Processor gt That value has to be changed to lt l FastFix Struts sensor hook controller processorClass eu fastfix targe tapplication sensor tomcat struts processor FastFixTilesProcessor gt 6 Modify the webapps EspigonValencia WEB INF web xml file Add the following tag within the lt web app gt lt web app gt root XML element lt l FastFix StartStop Listener Definition lt listener gt lt listener class gt eu fastfix targetapplication sensor tomcat struts processor ApplicationStartStopListener lt listener class lt listener gt 7 Configure the application bridge and the general sensor settings see Section 3 1 1 1 The Struts sensor is now installed It will be activated the next time Tomcat server is started and if sensors are configure to start automatically upon application startup Please note that there is no way to start the Struts sensor m
52. pport 8 2 minimumSeguenceSize 2 sequenceTimeWindow 3000 itemSetTimeWindow 1546 converter utput patternMining converter Output txt session utput patternMining session utput txt preprocessed utput patternMining preprocessed utput txt machineLearning patternMining machineLlearning learningProcess v1302191653 jar lt LaunchProcess automatic true learningSchedule 0 2 miningSchedule 6 8 5 s Results normalBehaviorPatterns patternMining normalBehaviorPatterns normalPatternBehavior txt minedPatterns patternMining minedPatterns minedPatterns owl i gt Figure 3 6 Pattern mining configuration element e Known ontologies Focusing on the ontology configuration FastFix currently supports two ontologies one representing the monitoring events and other representing all the concepts associated to software errors the maintenance ontology In order to properly configure the ontologies location FastFix unified configuration uses the Ontologies element to describe the location and main properties of these ontologies Page 17 of 47 FastFix Platform User Manual lt The Ontologies element has parameters and locations for the ontologies component in FastFix gt lt Ontologies id fastfix ontologies configuration lt An Ontology is identified by its nsUri and is located in a path relative to the FastFix home folder gt lt Onto Logy nsUri http www fa
53. rting in FastF ix is triggered by FastFix sensors that trigger the error reporting mechanism Use of FastFix error reporting is mainly an issue of connecting the correct sensors to the FastFix enabled application as described above in section 3 1 4 If the installed sensors detect a fault they will request an error report which is sent to the FastF ix server and inserted into an issue tracker Once the error report is inserted in the issue tracker it can be viewed by the mainte nance engineers The mode of presentation of a particular error report depends on the specific issue tracker being used More commonly the T RAC issue tracker has been used in FastFix An example of the display of an error can be seen in Figure If fault replication is being used the error report number can be used for fault replay see next section Page 36 of FastFix Platform User Manual me Jaks t Rdblty https traclabs fastfixproject eu ticket 1254 E v Google J g fastfix logged in as automatic reporting Logout Preferences Help Guide About Trac Previous Ticket Back to Query Next Ticket Ticket 1254 new defect Modify FastFix Issue Report An exception has occurred while trying to select an Opened 2 months ago element of a dialog Reported by automatic reporting Ovmed by szamarripa Priority minor Milestone Component componenti Version Keywords Cc Description A NullPointerException has been detected while trying
54. rver Setup The FastFix server is implemented as an Eclipse RCP Java application similar to the FastFix client It can also be run in console mode for systems where no graphical user interface is available The latest binary version of the FastFix server can be accessed in the FastFix repository at SourceForge To run the FastFix server the following steps have to be accomplished 1 Download the latest build of the FastFix server and unzip into a folder on the server machine 2 Configure the FastFix server see Section 3 1 1 3 Start the server usually by double clicking After the last step the FastFix server is ready and waits to receive information from FastFix clients SFor details on the sensor lifecycle we refer the reader to TFor more details on current FastFix sensors we refer the reader to Page 25 of FastFix Platform User Manual 3 1 4 FastFix Sensors Setup FastF ix is an open source project and as such provides a generic extensible maintenance platform The extensibility of the context observation system is a main enabler for the applicability of FastFix in different scenarios To this end the FastFix platform is designed to work with arbitrary sensors as long as these sensors implement the FastFix sensor protocol called sensor lifecycle 12 The setup procedures for sensors depend on the particular sensor at hand In general sensors are first installed and then register at the FastFix client Fro
55. ssueTracker id fastfix issuetracker configuration 1 The Ticket element has parameters to configure several aspects of the tickets for the target issue tracker gt Ticket attachmentMaxSize 2048 f gt lt TracConnector trac server main uri https traclabs fastfixproject eu trac login uri https traclabs fastfixproject eu login new ticket uri https traclabs fastfixproject eu nevwticket usernamec password qd trac get login true useFmbeddedTruststoreFile true truststore resources tracConnector truststore trac authorization domain traclabs fastfixproject enu trac truststore password FastFix gt lt TicketBrowser TicketURLPrefix https traclabs fastfixproject enu EbEicket lt fIssueTracker gt Figure 3 9 Issue tracker configuration element Fault Replication The last configuration in terms of fault replication is the configura tion of the fault replication component contained in the FastFix configuration file as well and shown below in Figure 3 10 Page 19 of 47 FastFix Platform User Manual The LogStore element configures for the FastFix client the connection to the machine where any auxiliary logs of the error reports are to be stored This is the machine where error reports are to be replayed normally the FastFix Server This connection is defined by the host and port of the machine In order to store the logs a username user password and location
56. st Envelope Body LoginRequest request gt Ph Page 41 of 47 FastFix Platform User Manual add key WCF child list value SessionSequence ClientLastView WCFException ClientProgRequest UserCode lt appSettings gt In this example there are two XPath rules that denote the location of the new fields that we want the sensor to monitor They are under the XPaths Envelope Body EntitiesRequest request and Envelope Body LoginRequest request the names of the WCF fields that we want to monitor are SessionSequence ClientLastView WCFException ClientProgRequest and UserCode 4 2 2 Event Correlation Event correlation component is a key point for extension since it is the FastFix compo nent where complex faults are detected Each fault or error is represented by a pattern Patterns of error are stored in the maintenance ontology Hence this ontology must be modified to add or update error patterns Hence it will extend the error type coverage of FastFix when the system is applied to monitor new applications Developers can edit ontologies by using ontology edition tools such as Protege How ever because this is a very common use case of FastFix we developed a user interface to facilitate the corresponding task Using this approach the extension of error patterns can be performed by engineers without any knowledge about ontologies such as the main tenance team The corresponding use case is explained in deta
57. stfixproject eu ontologies Monitoring ntology owl path ontologies Monitoring ntology owl useReasoner true gt lt Onto Logy nsUri z http www fastfixproject eu ontologies Maintenance ntology owl path ontologies Maintenance ntology owl useReasoner false modelSpec 0WL DL MEM gt ntologies 1P Figure 3 7 Ontology configuration element 3 1 1 3 Error Reporting and Fault Replication Communication Configuring the FastFix communication subsystem is critical to a cor rect operation of FastFix This component manages all connections between FastFix clients and servers Figure highlights the element in the FastFix XML configuration which control the communication system The server s host and port must be the same on the client FastFix configuration and on the server s The port choice depends only on the server s administrative restrictions The client host and port are only used on the FastFix client and are freely chosen barring any administrative restrictions Finally SSL can be used for securing the communication channel between client and server To use SSL the active property must be set to true keypath and keyvalue are unused The remaining properties configure the usual SSL parameters For the FastFix server the trustStore and trustStorePassword configure the location and password to access the list of the trusted clients And for the client the keystore and keyStorePassword configure the location and password
58. the client usually by double clicking The FastFix client is now ready and waits for sensors to register ff As soon as sensors register at the FastF ix client they show up in the client UI Sensors are started automat ically once registered unless the FastFix client is not configured to a manual start mode In the latter case sensors can be started from the client UI as illustrated in Figure FastFix Client Sensor ID Q http www fastfixproject eu ontologies MonitoringOntology owl TomcatTextlnputSensor 40815f98 4430 4fbd b7c0 04664b53cc96 Q http www fastfixproject eu ontologies MonitoringOntology owl ApplicationStartStopSensor e84e4ad2 dced 47c7 a316 7feffS55bec5 O http www fastfixproject eu ontologies MonitoringOntology owl4TomcatHttpRequestSensor_04a26d0e d443 414a af36 5e667b7c29e0 E Console 5 Ew BH r El r3 GB Console eu fastfix client selfHealing internal ServerDatalistener Registering for messages ISA AA AA A 21111103 Leu fastfix client communication internal ClientCommManager Instance eu fastfix client communication internal ClientCommManage Leu fastfix client communication internal ClientCommManager no messages in client for ClientSelfHealing bundle ClientWorker 1 client ContextBus Added event Type http www fastfixproject eu ontologies MonitoringOntology owl HttpRequestOnServerSide Ar lt gt Figure 3 14 Starting registered sensors via the FastFix client UI 3 1 3 FastFix Se
59. time delays between events event generation rates also associated with the sensors and which events are monitored as well as minimum support in order to consider certain event types as frequent Hence there is a group of parameters that can be tuned or modified in order to achieve a better effectiveness while mining new patterns This group of parameters are located Ph Page 42 of FastFix Platform User Manual in the unified configuration file configuration xml which are represented on figure Among the main parameters that can be tuned the main one is the pattern mining algorithm that will be executed in order to look for normal behavior patterns in our case we have selected the PrefixSpan algorithm but any developer can develop and select the algorithm that would better fit for mining normal behavior and then in the mining phase it will detect any event sequence that is different from the normal sequences Additionally other parameters can be tuned depending on the temporal properties and differences that any application can be characterized by Thus the developer can configure the following parameters as explained in section e support long The value of this field represents the minimum number of occur rences in percent of a sequence of context events to be considered as frequent e minimumSequenceSize int It refers to the minimum number of context events needed to become a sequence
60. tional monitoring code recompiling the sensor and redeploying it For example the RCP sensor can be extended to monitor SWT events currently not being sensed by registering it as a listener to these SW T events and creating events representing the occurrence of those events by calling the event creation routines Similarly the expert file within the FastFix self healing fea ture can be extended by additional methods to be sensed and then re instrumenting the application code with the extended expert file 4 2 1 3 Extension of Existing Sensors by Configuration The TXT sensor is implemented so that the addition of new sensed fields in the WCF message can be added easily In order to add a new field to sense from the WCF message the following needs to be added in the appSettings tag of the Web config file of the ExecuteService 1 A key value pair that with a key named XPath_WCF_Rule and a value that that denotes the position in the WCF message where the new field is located in the form of an XPath rule 2 A key value pair with a key named WCF_child_list and a value that denotes the name of the field to sense 3 Save and close the file the Web config file 4 Restart the ExecuteService Web Site in the IIS Manager The new sensed fields are then added as data to the corresponding event An example of steps 1 and 2 can be seen below lt appsettings gt lt add key XPath_WCF _Rule value Envelope Body EntitiesRequest reque
61. to access the file with the client s own SSL certificate Obviously each FastFix client must have its certificate added to the keystore file and the FastFix server must add each client s certificate to its truststore file lt The Communication element deals with general communication channels for communication between client and server Communication id fastfix communication configuration lt Server location by host and port gt Server host localhost port 5555 f gt lt Client location by host and port gt Client host localhost port 5556 gt Secure connection active or not with parameters to configure the secure channel in java lt 55L active false keyPath communication ssl keys keyValue 4Baseb4EncodedStringAsTheKey trustStore communication sslS5Store keyStore communication sslStore trustStorePassword 123456 keyStorePassword 123456 gt lt Communication gt Figure 3 8 Communication configuration element Issue Tracker FastFix uses an issue tracker to store any error reports that it generates Therefore the server uses the FastFix configuration to configure the connection to the Ph Page 18 of 47 FastFix Platform User Manual issue tracker Figure highlights the element in the FastFix XML configuration which configures the issues tracker connection on the FastFix server The Ticket element has the attachmentMaxSize parameter to limit the si
62. to select an element of a dialog This error Reply occurs d the same dialog is opened several times and the element is selected from the item list of the first opened dialog Symptoms PRESS BUTTON open User jdoe on client IP Address 172 18 232 49 clicked on a button with label open PRESS BUTTON open User jdoe on client IP Address 172 18 232 49 WIDGET SELECTED User jdoe selected a widget PRESS BUTTON ok User jdoe on client IP Address 172 18 232 49 EXCEPTION NullPointerException Possible cause Loss of object s reference The reference to an object has been lost due to multiple creation of other instances v Attachments Attach file Add a comment Hent BIJ A 1 B You may use WikiFormatting here Figure 3 24 Viewing an error report 3 2 5 Fault Replication Fault replication the ability to see the replay of a client fault at a maintenance server is available for Java console AWT and SWT applications Configuring fault replication requires preparing the application instrumentation in most cases and configuring the FastFix client and server 3 2 5 1 Preparing Applications Instrumentation 3 2 5 1 1 SWT Applications Setting up SWT applications for FastFix error record re play requires no previous instrumentation This is due to the fact that the FastFix SWT fault replication sensor records all the necessary event for SWT replay The only require ment is that the runtime configuration
63. ually by developers or semi automatically The instru mentation approach implemented for the FastFix self healing feature is an example for a Page 40 of FastFix Platform User Manual semi automatic instrumentation because the methods to instrument are specified in an expert file and the actual instrumentation is done automatically It can be reused and extended for other instrumentation approaches In order to integrate a self implemented sensor into the FastFix framework two things have to be done First the sensor implementation has to use the lifecycle described in Deliverable D3 6 The sensor lifecycle allows the Fast Fix client to control sensors e g switch them on or off Second the sensors have to communicate to the FastFix client via RMI or HT TP as described in Deliverable D2 4 This allows self implemented sensors to send monitored events to the FastF ix platform and use FastFix functionality like event storage or event correlation As HTTP is available on almost all devices and it usually can transmit through firewalls this consitutes a general communication mechanism that is available in most environments Also existing instrumentation solutions can be integrated into the FastFix framework in this way 4 2 1 2 Extension of Existing Sensors by Adding Monitoring Code Existing FastFix sensors can be extended to monitor additional types of events or addi tional properties This extension has to be done by adding addi
64. users to regulate the FastFix functionality e g start and stop sensors The client namespaces currently contains 100 classes with 504 methods in 37 packages 10 bundles with a total of 4 632 lines of code in 110 files Figure 2 2 shows a screenshot of the FastFix client UI including three registered sensors J ANSO FastFix Client 0 Sensor ID O http www fastfixproject eu ontologies MonitoringOntology owlX TomcatTextlnputSensor 40815f988 4430 4fbd b7c0 04664b53cc96 O http www fastfixproject eu ontologies MonitoringOntology owl ApplicationStartStopSensor_e84edad2 dced 47c7 a316 7feff55SbecS O http www fastfixproject eu ontologies MonitoringOntology owlWTomcatHttpRequestsensor_04a26d0e d443 414a af36 5e667b7c29e0 E console 3 E pE El i e Console eu fastfix client selfHealing internal ServerDatalistener Registering for messages IIIIIIIDIIIIIIIIITIITIIITIITIITTITTIIITI eu fastfix client communication internal ClientCommManager Instance eu fastfix client communication internal ClientCommManagt eu fastfix client communication internal ClientCommManager no messages in client for ClientSelfHealing bundle ClientWorker 1 client ContextBus Added event Type http www fastfixproject eu ontologies Monitoring ntology owl amp HttpRequest nServerSide ArT A 4 Figure 2 2 FastFix client platform UI 2 2 eu fastfix common The common namespace contains bundles which are needed by both the FastFix
65. ust fulfill All these tags work in conjunction i e only methods that fulfill all the conditions are selected However if tag values are left empty then they have no impact in the filtering process This is useful for instance for methods declared in anonymous class In this case the lt className gt lt className gt tag should remain empty Finally several sets of methods can be described in an expert file using several lt method gt lt method gt tags Finally configuration parameters are also available on the client side and can be set through the FastFix client Figure 3 13 Activate Controller allows to disable monitoring for self healing and consequently any action that this component could make This is only useful in case runtime overhead becomes an issue in order to disable any of the behavior brought through code instrumentation Supervisor Buffer Size represents how Ph Page 23 of FastFix Platform User Manual many method calls are kept in memory at most Therefore it also represents the size of the trace logged when an exception is raised and caught by the self healing component Request Period represents the frequency at which the FastFix client sends a request to the server for new patches Finally Testing Mode for Trace Collection modifies the behaviors of the self healing component at runtime In this mode every method calls that is instrumented is logged in a file This is useful to collect sample of traces
66. vent type associated with a symptom To choose one a selection dialog is provided Mandatory e Symptom Name It represents the name of the symptom It will be used to identify the symptom hence it must be unique Mandatory Ph Page 30 of FastFix Platform User Manual e Operator This field is a combo box which represents the action to be per formed over the next field Value to detect if an event with a concrete event type is a symptom of a pattern Possible values are matches lt gt other Optional but mandatory if Value field has been inserted e Value This field represents the metadata associated to the symptom that will be evaluated by using the last field Operator Optional but mandatory if Operator field has been inserted e Criterion This field is a combo box which represents if it is necessary or not that all the declared symptoms must occur to detect a pattern Possible values are all and any Mandatory 4 Identify the cause In this tab maintenance team can associate a cause to the current pattern Associate a cause to a pattern it is not necessary but all the fields are mandatory in such case Figure 3 18 e Cause Type It represents the type of this cause To choose one a selection dialog is provided e Cause Name This field represents the name of the cause It will be used to identify the cause so it must be unique e Info The information inserted in this field must explain in a general
67. way what this cause means 5 Save the new pattern If any mandatory field is empty a message is shown warning the engineer that must review the tabs and complete all the required information Once a pattern is created it s added automatically to the FastFix correlation system i e it is not necessary to restart the FastFix server to detect this new error General Information Symptoms Cause Pattern Name Subject Summary m ES ur Figure 3 16 General Information Tab Page 31 of 47 FastFix Platform User Manual 200 Pattern Mining Create New Maintenance Pattern General Information Symptoms Cause Event Type Symptom Operator Value Info Reaction Dela Reuse PressButton symptomPressButton matches save If you choose an operator you have to fill the field Value since the operator is used to evaluate it In the same way if you fill the field Value you have to choose an operator If you fill the field Reaction you have to fill the field Delay since reaction event should occur at most as many seconds as specified in Delay field In the same way if you fill the field Delay you have to fill Reaction field Criterion Ly Delete Symptom Add Symptom Figure 3 17 Symptoms Tab gt Pattern Mining Create New Maintenance Pattern General Information Symptoms Cause Choose an existing cause or create a new one Create new Cause Cause
68. ze of error report attachments The TracConnector element configures the remaining parts of the issue tracker configura tion current parameters are used for the TRAC issue tracker but could be used with alternative meanings with other issue trackers The trac server main uri trac login uri and new ticket uri parameters reference the URIs of the three issue tracker webpages respectively the entry page the login page and the ticket insertion page The username and password are the credentials to login into the issue tracker The trac get login parameters informs FastFix whether the issue tracker login page requires an HTTP GET or PUT call The useEmbeddedTruststoreFile identifies if set to false that an external file is used as a truststore to obtain the certificate of the issue tracker server The truststore parameter points to the locatio of the truststore the trac autho rization domain identifies the domain of the issue tracker certificate and the trac truststore password is the password to open the truststore Finally the TicketBrowser element has as its only parameter the TicketURLPrefix which is the URL used to access the page describing a particular ticket in the issue tracker It assumes that if a ticket number is appended to this string this will result in a valid page describing that ticket lt The issue tracker has parameters to configure the issue tracker to which the FastFix server can connect gt I

FastFix Platform User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents