Framework for Synchronous Gathering of Interaction
Contents
1. ells Odyssee Ge ites 24165820 4 1920 lias Ho 1459 24183580 0 104 886555214907 8 4AM A F3 7 Hektor Agamemnon Athene He 6 sisoeeee ait e n 0116 021566181153 95 700 098 8287221275095 0 101 188184806961 5 2 146 561128342455 10 24216825 0 165 628345708005 164 321610453226 0 244 248580600037 128 911084379706 11 1462 24233450 0 624 972300531808 _ 147 480643405333 0660 13145386547 133 520387604494 Handlungen Menelaos Merkma i2 163 24250202 0929 614463324397 108 080431674261 0 735 001710052748 154 150654275782 13 1464 24266823 Dichter Achill Lessing Odysseus 16s 24283450 Homers Diomedes Glaukos Tiere gt ai Tydeus Rede Menschen Handlung 9 1i 2e Endzweck Moral Olymp Patroklos P 21 1472 24400064 2 1473 24416688 0932 065292787447 125 51616759346 98 05005467861 123 639809435554 3 117 509219924523 3 103190460865699 3 113459140336545 3 142 973992481529 3 144 320360404254 3 107 131445139476 3 98 6809702988467 0884 88178144471 0 857 657641181286 0 858 919084442459 0861 879735426191 0 838 85682248685 0836 426773917045 118 787121630248 106 271739636759 140 550085930863 112 718274495455 113 833906130367 137 674811582565 122 986151057466 117 174729593444 128 338037154572 127 759329519722 0 900 033519691788 0 881 719438789005 0 855 202738766384 0 852 311340634333 0 851 426565591828 0851 398136241623 0 850 38862979156 0 863 557041647 0 870 618051949132 0 857 06232. We solved the second design goal by turning the framework into a client server application A small language and UI framework specific library would handle collecting the interaction data and then send it to the server application via a standard TCP socket This also opens up the possibility to run the client application on a different as fo 1 to gwr yp Y Figure 2 Framework Model in UML notation device than the server which could be useful if the framework is extended to other types of devices e g smarthphones where saving and retrieving the data could be a problem 4 1 The Client Library As a part of this project the target language was Java and the targeted UI framework was AWT Swing which meant that we had to use these technologies when implementing the client library To start recording interaction data the client application only has to initialize the library and optionally add the AOIs by calling a method on the tracking state object Mouse clicks are handled sepa rately because they are frequently associated with some metadata The client application must handle these events in some way e g by registering a mouse click listener on the right component and associate them with the appropriate metadata The mouse event and the corresponding metadata is then passed to the client library which will handle the event similarly to the other events as described below 4 1 1 The client library is based on the concept o
3. do not change position during the entire runtime of an application Consequently dynamic AOIs can change form and position at runtime which is especially useful when analyzing user interfaces with moving interface elements 2 RELATED WORK The history of eye tracking dates back to over a century ago where initial observations about the eye movements were made 9 The techniques improved over time and today eye tracking is used for a diverse set of problems e g marketing psychology or usability research in Human Computer Interaction HCI These days and in previous work eye tracking and interaction data are measured separately using different external applications For this reason eye tracking and interaction data have to be synchronized manually afterwards 4 A consequence of merging the tracked data is that this could lead to an inaccurate analysis A modern approach to reduce the workload is called eTaddy 5 an integrative framework that should guide the supervisor in tracking and evaluating the eye tracking data However this framework does not ensure synchronous tracking of eye activity and interaction data Another modern approach of automatic processing is called AdELE a framework for Adaptive E Learning through Eye Tracking 3 This framework tracks the interaction and eye tracking data synchronous as a requirement to enable an effective adaptive e learning 3 MODEL Before we proceed to the documentation of our
4. 65 86982463639275 126 84674293955149 2 Fixation 183525 0 1395 23134808 511 950249118527 206 280721299117 2 Fixation 232576 0 1410 7 23384131 429 102353093476 430 176683734226 1 Fixation 183114 0 1422 23583995 122 5674121185875 362 2882657104585 1 Fixation 333100 0 1459 24183580 101 85763867120825 143 02836260449152 2 Fixation 49870 0 1500 24766415 748 011690697458 91 7054543901941 3 Fixation 16748 0 1632 25899078 986 571398856759 107 387756611934 3 Fixation 16627 0 1658 26215435 980 0156127596711 188 15577486034292 3 Fixation 33373 0 1703 26964996 1162 06377388633 196 621472536299 5 Fixation 3813892 0 2127 30795511 1515 31824158024 299 46108561744 4 Fixation 716198 0 2218 31578328 1108 709957993635 97 81381271204755 3 Fixation 149868 0 LS 2256 31911431 699 546351569152 80 77426290719805 3 Fixation 333102 0 Verzeichnis home maheswvn Dokumente WS15 ProjektINF FixationFilter eyeGaze csv Speichern Figure 5 The GUI of the Fixation Filter 1 Selection of eye gaze and AOI data files 2 Selection of which algorithm should be used to compute the fixations 3 Table showing the results of the algorithm Participant01 Thinking Aloud Chapter Subchapter at x Page f Paragraph fai Text WENEN Participant02 Thinking Aloud Chapter Subchapter gm Page a Paragraph 4 i Text a E E Participant03 Thinking Aloud Chapt apter O A Eed Subchapter u t Page 4 t E Paragraph E a x
5. 782 72901770164 0 808 557639849605 0 811 610192088992 0 884 532023513457 0 884 298025035532 0 859 105915959226 0 882 318984784652 0 850 420330563647 O RGR PRGAIANISASA SFist GHA towel aviv reat Neet oe octont aa Lees cae 1510 Paragraph FIED DINUCS DIGN IGI RUINY FUI wally Thesen Beschreibung Haller und komisch verstanden in einem Sinne jedoch der sich von dem bisher blichen unterscheidet und gleich zu Beginn erkl rt werden mu Der Titel Poetik Parii bedeutet zwar l ngst nicht mehr eine praktische Lehre die Ungeiibte instand setzen soll regelrechte Gedichte Epen und Dramen zu schreiben Aber die neueren Malerei Eigenschaften Teile Terminal Help 3 Figure 1 Representation of the developed framework 1 The application VarifocalReader where interaction and eye tracking data is recorded 2 The resulting data being displayed in a spreadsheet application LibreOffice Calc 3 The commandline interface of the server application managing the recording sessions Abstract The evaluation of user behavior has become a leading aspect when developing user friendly software or analyzing the influence software modifications have on the user experience These days many methods to evaluate this data are available This paper will focus on eye tracking which is a method to evaluate user behavior by measuring the eye movement of a participant during the interaction with a graphical user interface of an applicatio
6. 85662284 0 857 878885878017 0 834 515877588565 0 834_787829103589 Held Vater Dichtung Welt Helden Gott C 23 1474 24433312 0856 279641803558 111 142610415664 3 0844 28B7884455989 96 098774489019 24 1475 24450065 0870 899452544254 109 066442834683 3 0 837 634875545773 136 777466603689 ch Epischer Sti_25 1476 24466685 0776 719090176048 131 09387294171 3 0800 176875370962 118 445378725391 26 1477 24483309 0722 395009978368 76 8476141050087 0 757 544428857509 108 728736696139 Pathos Welt Wallenstein 27 1478 24499934 0737 24055224171 127 376833284052 3 0 748 486000038829 98 9822065633234 120 666424384308 28 1479 24516684 Sophokles Tragische Anstren s gt tis 2184032 1482 24566558 0 781 771600245265 0 773 05774904089 0 746 598766917014 0 739 59992740929 710532126844555 89 7378034959729 117 653308434274 3 918087693627967 0 0 755 583697058028 0 740 653507968527 0 769 861138533015 0 761 360346419388 106 905207807631 132 91153942946 107 162212140156 31 i i 32 1483 24583306 0751 796342009911 66307524122476 0781808487745147 103 82253649932 Mensch Dichter Rede Faustin 33 1484 24599929 0761224778566539 113 369701412557 3 075335522123758 118890884743541 34 1485 24616552 0783184968997812 62 7767406443127 0774 353649902623 105 20357024343 Erwartung Held Leidenschaft 35 1486 24633176 0 764 490783905931 105 925625144573 3 0760 700694622938 106 527809706313 36 1487 24649924 Be
7. Framework for Synchronous Gathering of Interaction and Eyetracking Data Nico Lassig Marius Maal3 Vithunan Maheswaran Universitat Stuttgart lle Frage 1 a In welchem Kapitel finden sich die meisten Instanzen zwischen doppelten Anfulhrungszeichen Chapter Ex Subchapter RE WE Sex Hi Begriffe episch lyrisch dramatisch und allenfalls tragisch a oa 8 n 99 Kehrreim Sprache Dichter Strophen Stimmung Lied Lieb Lyrischen Verse Musik Dichtung Liebe Eingebung Flut Treulieb Leser Straz ROCA E Eichendorffs Herz Gedichte Mond S File Edit Wew Insert Format Jools Data Window Help x sen Vers Versen Gedicht Abend Lieds Was B HO Eon x 9 A S aoe 3 GS Bi gt 2 Mm is mten Wiederholungen Goethe Lieder Abstand ftbeationsans v 0 a ga a a 2245 AeOKS 2 ZEHR F IS Gedichten Nacht Welt Seele Geliebte Grase ka i _ Ris x pA 2 Reime Brentano Freude Takt Eine Worte Eri k z z A Cc 2 F G 1 D Timestamp Left Validity Left x Lett Y Let AOls Right Validity Right X Right Y 2 1453 24083591 0 43 2716783702199 161 683340099717 1 0 85 4338172182906 138 146372773463 3 1454 24100209 08 04077917899122 160 764038309117 1 0 82 3032585936016 105 298251578824 _4 1455 24116958 4 1920 1200 0 0 102 538204346783 118 21687384454 ch Lyrischer Sti 5 1456 24133587 0 147 373753385473 77 5231125825485 0 101 908958093118 93 0709292554639 24150205 4 1920 1200 0 88 5599839335191 98 2425
8. IChangeltem This item is generated when an AOI has changed It contains the id name and the new screen bounds of the AOI e EyeGazeDataltem The used eye tracker captures the loca tion the user looks at every 16 ms This data consists of four IEEE 754 double precision binary floating point numbers that specify the X and Y coordinates of each eye The data is then converted into this data item and the framework also deter mines which AOI the user looked at e KeyboardDataItem Whenever the user types on the key board this data item is generated It contains the pressed keys and modifiers both as a textual description e g Ctr1 Shift for modifiers and F for the pressed key and as a numerical identifier which depends on the used UI framework It also specifies which kind of event has happened by indicating if the button was pressed released or typed Normally one keyboard interaction generates multiple events because a single inter action consists of pressing and releasing the button A typed event is generated if the button was quickly pressed and then released which happens when the user is typing text e MouseClickDatalItem Similarly to the keyboard interactions the mouse clicks are also captured The contents of the data item are similar to the keyboard data but the application being tracked can also associate additional data with each event in order to provide additional data for later analysis e MouseMotionDataltem In addition
9. Notes in Informatics Informatiktage 12 111 114 2013 K A Ericsson and H A Simon Verbal reports as data Psychological review 87 3 215 1980 R Fielding J Gettys J Mogul H Frystyk L Masinter P Leach and T Berners Lee Hypertext Transfer Protocol HTTP 1 1 RFC 2616 Draft Standard June 1999 Obsoleted by RFCs 7230 7231 7232 7233 7234 7235 updated by RFCs 2817 5785 6266 6585 K Holmqvist M Nystr m R Andersson R Dewhurst H Jarodzka and J Van de Weijer Eye tracking A comprehensive guide to methods and measures Oxford University Press 2011 R Jacob and K S Karn Eye tracking in human computer interaction and usability research Ready to deliver the promises Mind 2 3 4 2003 S Koch M John M W rner and T Ertl Varifocalreader in depth visual analysis of large text documents IEEE Transactions on Visual ization and Computer Graphics TVCG 2014 N L ssig Detection of fixations and saccades Eye Tracking in der Visualisierung 2015 D D Salvucci and J H Goldberg Identifying fixations and saccades in eye tracking protocols In Proceedings of the 2000 Symposium on Eye Tracking Research amp Applications ETRA 00 pages 71 78 New York NY USA 2000 ACM M W rner and T Ertl Smoothscroll A multi scale multi layer slider In G Csurka M Kraus L Mestetskiy P Richard and J Braz editors Computer Vision Imaging and Computer Graphics Theory and Applications v
10. USB studio mic The raw data was exported as CSV files Each category Eye Gaze Mouse Movement Mouse Click Mouse Scroll Keystroke and AOI had its own file The keystroke file is irrelevant in this user study as the keyboard is not used in this program In the AOI file the predefined AOI bounds are saved Which AOI is viewed each time can then be looked up in the eye gaze file Later described in Figure 4 The recorded audio files had to be moved manually by us because these files were saved in a subfolder of the client 6 4 2 Participants We conducted the study with six participants one female five male The average age of the participants was 22 8 years min 21 max 25 four of the participants had a slight experience with the program whilst two participants have never seen or used it before Each participant successfully performed an Ishihara test and a Snellen chart to confirm that participants were physically able to accomplish the given task The study took about 25 min depending on the speed and experience of participants 6 4 3 Procedure First the participants were asked to fill out some personal informa tion e g about their age and if they already have used the Varifo calReader Then we gave the participants a brief introduction into Thinking Aloud They were admonished to always tell what they are doing what their intention is as well as to answer the prepared questions We also informed the participan
11. ance the accuracy of the algorithm Finally we have to determine and adopt the focused AOI For that we have to import the data where the bounds of the AOIs are predefined Eye Gaze Computation 5 2 Detection of Fixation and Saccade Points We chose the Velocity Threshold Identification algorithm as the basis for the implementation which we call Fixation Filter We chose it for several reasons it is a fast algorithm which is easy to implement compared to other algorithms and the accuracy of the algorithm is also good 12 The idea of the Velocity Threshold Identification algorithm is that the points get a type fixation or saccade assigned based on their point to point velocities The formula for the calculation is distance velocity timespan distance x coord prev x coord y coord prev y coord timespan timestamp prev timestamp We decided to assign eye movements with a point to point velocity lt 100 deg sec as a fixation otherwise as a saccade Furthermore we categorize all tracked eye movements as a sac cade when the coordinates are not located in a defined AOI This idea is used in the I AOI algorithm 12 After we categorized the fixations and saccades we have to deter mine the duration of the fixations The duration defines how long a participant looks at a particular region Diz 1 There are two steps to identify the duration of a fixati
12. cked function The additional data for this function call was also already recorded This data was converted into a InteractionInfo instance which was also passed to TrackingState onMouseClicked This shows that our design goal of creating an easy to integrate framework has been fulfilled The recorded data also fully meets our expectations It contains all data that was captured by previous tools and also has additional data streams like keyboard events or mouse scroll events Using the data importer presented in section 4 4 we also successfully imported the data into an existing eye tracking data visualization application which was presented in 4 The resulting Integration of the Framework visualization is shown in Figure 6 6 3 Tobii Studio For our approach we work with the tracking system developed by Tobii 2 It is a complete eye tracking studio which provides many features to record analyze and visualize eye tracking data of a participant Before the use case starts Tobii studio is used to calibrate the eye tracker for each user The calibration is necessary to provide accurate test results during a study In the first testing sequence with Tobi Studio we noticed that eye tracking requires some demands on the quality An example would be a user wearing glasses Glasses lead to difficulties in tracking the eye coordinates Using an external dialog offered by Tobi Studio we could examine the tracking status When we noticed that th
13. e track status is under 80 of suc cessful eye tracking it was recommended to cancel the task of a user 6 4 Example User Study For the user study we prepared several questions for the literary texts Grundbegriffe der Poetik english Basic Terms of Poetic by Emil Staiger and Iliad by Homer 6 4 1 The study was performed in a laboratory isolated from external distractions and with the rollers shuttered down The participants had to turn off their mobile phones and other electronic devices which could disturb the user study Environmental Conditions and Technical Setup IFrage 1 a In welchem Kapitel finden sich die meisten Instanzen zwischen doppelten Anflhrungszeichen BEME Chapter Se Subchapter FE Page Kehrreim Sprache Dichter Strophen Stimmung Lied Lieb Lyrischen Verse Musik Dichtung Liebe a Eingebung Flut Treulieb Leser Strophe Eichendorffs Herz Gedichte Mond Subjekt Vers Versen Gedicht Abend Lieds Wasser Wiederholungen Goethe Lieder Abstand 100 Gedichten Nacht Welt Seele Geliebte Grase rauscht Reime Brentano Freude Takt Eine Worte Erinnerung 2 101 3 102 ch Lyrischer Stil Erinnerung llias Homer Zeus Odyssee Gesang Hektor Agamemnon Athene Handlungen Menelaos Merkmal Dichter Achill Lessing Odysseus Homers Diomedes Glaukos Tiere Tydeus Rede Menschen Handlung 3 104 Endzweck Moral Olymp Patroklos Priamos 5 Held Vater Dichtung Welt Helden Gott Gleichnis ch Epischer Stil Vors
14. erate an AreaOfInterest object that contains the ID and the name of the AOI Additionally this object holds all points related to that AOI The final data structure contains a list of AOIs with each individual object again containing a list consisting of the individual AOI points After creating the AOI data structure parsing the actual interac tion data is a simple process Most importers parse the text data into the appropriate number formats e g a 32 bit integer or a IEEE 754 double precision binary floating point and create an object for the event If an event contains information about which AOI was hit then the parser has to do some additional work to cross reference this event with the right AOI object This is done by parsing the list of hit IDs in the format 1 2 into a sequence of integers These represent the IDs of the hit AOIs Then the parser searches in the AOI list for the first AOI object that has the same ID This generates a list of references to the right AOIs After the importer is finished the Dat aBase object contains all data recorded in one session at which point it can be converted into the right data structure suitable for analysis 5 POST PROCESSING During the recording several CSV files are generated which can be used for the analysis The recorded data of the eye movement looks like the following Fig 4 O Each recorded data has its own ID 2 The timestamp of the recorded data It start
15. f having multiple data sources IDataSource which generate a stream of data items IDataltem that have to be recorded By keeping the data sources behind a generic interface adding new data sources at a later time is easy to do without major changes of the client library When a data source generates a new data item the client library uses the IDatalItem interface to serialize it into the OutputStream of the TCP socket so the server can then process the data For tracking the eye movements of the user the Tobi Analytics SDK 1 is used However it s relatively easy to add support for a different kind of eye tracker by implementing a new data source that uses the appropriate SDK The Tobii SDK does not expose a Java API so we created a small JNI library that uses the C API to expose the API functions needed for our framework Recording of user interface events e g mouse movements or keyboard events is done by registering multiple global AWTEventlisteners which get called by AWT whenever a spe cific event has occurred These AWT events are converted into appropriate implementations of Dat alItem and then sent to the server like all other data items To support any possible shape of an AOI we decided to expose this functionality by using an interface The interface has two functions Internal Structure of the Client Library yD e Expose a function that checks if a screen coordinate is within the AOI which has the signature boolean
16. framework we will have a closer look at the model which we have created to describe how the measured data is represented in the framework and how each entry is structured The UML diagram in Figure 2 shows how the data of one session is organized How each individual entry is structured up is described in the following e Session A session contains all data items from one user inter face recording session In the implementation the data items are not kept in the memory but are instead streamed to multiple data files to reduce memory usage as the memory required to store all data items would put unnecessary strain on the system running the framework e Metadata Each recording session has a small data structure that contains additional data related to one session This con tains a title and a description of the test that is currently being conducted Both strings are entered by the test supervisor before the actual recording session begins e DataItem The framework is designed around the idea that each interaction is saved in the form of a single specialized data item that contains all relevant information Every data item has a globally unique numeric identifier and a timestamp that indicates when this event has occurred This timestamp is the zero based number of us that have passed since the beginning of the recording session Although the time is saved in us the actual resolution of the timestamp depends on the operating system e AO
17. graph layer at first then looks at the text and then back at the paragraph again It also can be seen e g that participant 3 reads the text often for a short time whilst the other two participants focus on the text for a longer timespan but not as often as participant 3 The visualization of the complete user study can be analyzed simply because of the visual representation with the same timeline used for each participant and because the interactions are categorized and each category has its own color 7 3 Task Solving e Speed The slightly more experienced users needed 45min less to solve the tasks in comparison to the inexperienced users This is kind of unsurprisingly however the inexperienced users needed about the same time for each task The experienced users though needed more time for each new task because the tasks got harder and more elaborate each e Correctness There was no major difference between the ex perienced and inexperienced users Most tasks were solved correctly but there were few tasks which several probands struggled to solve e Eye Gazes Another significant difference between experi enced and inexperienced users is that the experienced ones nearly did not look on the icons only for short time to se lect them whereas the inexperienced ones watched them more closely to understand what they could mean before selecting them The biggest difference however was that the experienced participants did not
18. hitTest int xy Le y e Handling a set of listeners that get notified if the bounds of an AOI have changed When using AWT every interface element is a subclass of the Component class which has all the necessary information to per form the mentioned hit test and and notify the listeners should the bounds of the AOI component change We used this to implement TAreaOfInterest fora Component instance This makes the AOI usage easier to realize for client applications and can also be used as an example for how an implementation should be written The framework also handles recording an audio stream for when a user is thinking aloud The Java Runtime offers an API for capturing audio from various sources in the system However choosing which source to capture has to be done by test supervisor To minimize the required level of interaction while performing the recording our framework simply captures all audio streams and saves them to multiple files After the user studies are finished the right files can be selected by checking which file contains the right audio Initially we also wanted to capture periodic screenshots or even a full video of the screen the user is looking at However this caused performance issues because the system locked up for a few milliseconds each time the framework took a screenshot This would distort the interaction data captured of a user 4 2 The Server Application The server application is responsible for deserial
19. itten in C and runs on both the Microsoft NET and the open source Mono platforms This it can run on all major desktop platforms without modifications 4 3 Communication Protocol The communication protocol between the client and the server is a central part of our framework It is the basis for future extensions to other programming languages or UI frameworks After the TCP connection is established the client first sends the ASCII string EyeTrackingEvaluation followed by a carriage return CR and a line feed LF character The next line contains a version number currently that is 1 again ended by the CR LF sequence The client library can send some additional options to the server The only option currently supported is Application Name which should identify the application that is used The format of these options is the same as in the HTTP Header 7 After sending all options the client sends an empty line just containing CR and LF to signal the server that all options have been sent The client now waits for the server to send the string Ready r n until proceeding with initializing the interaction tracking This mechanism allows the server to interactively ask the supervisor to enter the metadata for this session After all this data has been gathered the server sends the ready string Ready r n and the client begins transmitting the serialized interaction data in binary form After one session has ended the server begins
20. izing the data re ceived from the client and saving it into multiple Character Separated Values CSV formatted files After deserializing the data items they are handed to a separate thread which demultiplexes them into multi ple streams of the same type e g all eye gaze data items will be in one stream mouse move items in another The data items are then formatted into a line where each value is separated by a character This format is supported by common spreadsheet applications After a client has connected to the server there is the possibility of specifying some metadata for this session that gets saved alongside the recorded data This can be used to give the session a meaningful title and description After the supervisor of the test has entered this data it is saved to a JSON JavaScript Object Notation formatted file which can be read at a later time to reconstruct the data File Edit View Search Terminal Help Figure 3 Server commandline interface running in a terminal emu lator on Ubuntu GNOME Currently the server application only uses a basic commandline interface see Figure 3 which allows to enter the required meta data The internal structure of the server was designed around the idea that the UI could be easily replaced The server uses an interface to communicate with the user interface Creating a graphical user interface for the server could be done without major changes to the server code The server is wr
21. lly synchronize the eye track ing and interaction data the accuracy for the analysis gets lost There is no unified architecture capable of handling both in a synchronous manner which leads to complications when analyzing the recorded data 4 For this reason we have designed implemented and tested a framework which is capable of providing synchronous measure ment and storage of interaction and eye tracking data This makes the analysis much more accurate The challenges and goals of our approach are to create a data model which is necessary to put the in teraction data into a cohesive relationship and to ensure synchronous storage and recording when using the framework There are some important concepts that need to be explained before going into detail about the framework Interaction events There are multiple forms how the user can interact with a given interface This includes mouse movements mouse clicks or keystrokes Thinking Aloud To understand the user behavior data is not always sufficient For this reason the thoughts of the user must be considered This is could be done by additionally asking the user to say their thoughts out loud 6 Area of Interest In many user interfaces there are areas that need special attention These are called areas of interest which means that events occurring within this area should be handled specially when analyzing the interaction data There are static and dynamic AOIs Static AOIs
22. n Tracking the eyes is an important step for gathering eye movement data However without more information about other actions of a user the value of the recorded data is limited To supplement the eye tracking data other data sources like mouse movements or keystrokes can also be recorded In this paper we present a framework that automatically records this data while keeping the different data sources synchronous in order to make analysis as accurate as possible Index Terms Eyetracking Synchronous Interaction Data Collection Thinking Aloud Fixation Filtering 1 INTRODUCTION Recording interaction data is a key aspect when analyzing the us ability of a user interface of an application Additionally tracking the eye of a user yields important data about the user experience 4 Recording interaction data as well as eye tracking data is mostly accomplished by using an external software However the applica tion recording the eye movement data is typically not able to record the interaction data This data has to be recorded using another application So the storage of the measurements occurs separately and without a uniform timestamp As a consequence the measured data has to be merged and synchronized manually Nico L ssig Marius Maaf Vithunan Maheswaran stl03S55 marius maass vithunan maheswaran stud uni studi informatik studi informatik stuttgart de uni stuttgart de uni stuttgart de Consequently by having to manua
23. ns or saccades has an impact on the analysis because bad algorithms rather filter out too many points or too few ID Timestamp Left Validity Left X 1453 24083591 0432716783702199 1454 24100209 0 04077917899122 160 764038309117 1 1455 24116958 4 1920 1200 1456 24133587 0 147 373753385473 77 5231125825485 3 4 19204 24166829 4 1920 24183560 0 104 906555214907 24200202 0 154 991695664066 24216825 0 165 626345798905 24233450 0624 972300531606 24250202 0 929 614463324397 Lett Y 167 683340099717 1 11457 24150205 1456 1459 1460 1461 1462 1463 168 59501693 7231 169 6179964 76643 164 321610453226 147 489643405333 108 0604351674261 Left AOls Right Validity Right X Right Y 138 146372773463 105 298251578824 118 21687384454 93 0709292554639 98 24256474094 95 7001354742715 117 461706271752 146 561128342455 128 911084379706 133 520387604494 154 150654275782 Right Aois 0854338172182906 082 3032585936016 0 102 538204346783 0 101 908958093118 5 068 5599839335191 0 116 021566181153 0 98 8287221275095 0 101 188184806961 0244 248580690037 0660 13145386547 0 735 001710052748 Figure 4 Eye Gaze Data file generated by our framework Described in chapter 5 The raw data we get from our implementation returns the coor dinates and the AOIs seen by the right and the left eye separately Therefore we first have to average the value of both eye gazes The exact process is explained in the next subchapte
24. olume 274 of Communications in Computer and Information Science pages 142 154 Springer Berlin Heidelberg 2013
25. on First fixation points are formed into fixation groups if they fulfill two conditions Condition one is that these points are between two type 2 saccades The other condition is that the recognized AOI is the same in successive fixation points To get the duration of a fixation we take the first and last times tamp of the fixations in the fixation group and compute the differ ence At the end we remove all saccade points as well as fixation groups with a duration lt 200 ms Duration 5 3 GUI and Visualization We implemented a GUI that provides a front end for the previously described algorithm The GUI shown in Figure 5 allows to select an eye gaze data file and an AOI data file which were generated by our framework The user can then select an algorithm for computing the fixations execute it and examine the results which are shown in a table At this time the program provides the described algorithm as well as the I AOI and I VT algorithm itself which are used for it The resulting fixation data is written to a new CSV file which can be used as input for a visualization of the data Figure 6 shows a possible visualization which was generated using the software presented in 4 6 USE CASE In the following we present a user study based on our implementa tion For the user study we used the program VarifocalReader 10 6 1 The VarifocalReader offers a multi layer visualization approach and supports analyst
26. ork for synchronously recording various interaction data from an application This was done creating a model containing the various types of interaction data which share a common time base The implementation uses this model to capture the data and save it in multiple data files but now all these individual data files are sharing a synchronous time base and can be easily analyzed The conducted use case has shown that the framework fulfills this goal adequately REFERENCES 1 2 3 4 5 6 7 8 9 10 11 12 13 Tobi analysis sdk http www tobiipro com product listing tobi1 pro analytics sdk Accessed 2015 06 10 Tobii studio manual http www tobiipro com siteassets tobii pro user manuals user manual tobii studio pdf Accessed 2015 10 04 V M G Barrios C G tl A M Preis K Andrews M Pivec F M dritscher and C Trummer Adele A framework for adaptive e learning through eye tracking Proceedings of IKNOW pages 609 616 2004 T Blascheck M John S Koch K Kurzhals and T Ertl Va2 A visual analytics approach for evaluating visual analytics applications IEEE Transactions on Visualization and Computer Graphics 22 1 2015 T Blascheck M Raschke and T Ertl etaddy ein integratives frame work f r die erstellung durchf hrung und analyse von eyetracking daten Gesellschaft f r Informatik editor GI Edition Lecture
27. r Additionally we define two types of saccades 11 When both eye gazes are not tracked it is a type 1 saccade The entries of type 1 saccades are removed immediately So they are not part of the computation of our algorithm Type 2 saccades are removed at the end of the process and are detected by implemented algorithm Thus the characteristic which determines if an eye gaze is a fixation or saccade depends on the implemented algorithm The detection of the type 2 saccades in our algorithm is described in the subsection Detection and filtering of Fixations and Saccades 5 1 As already mentioned above the raw eye gaze data consists of data from the left and right eye separately For the evaluation and therefore for our algorithm however we need one eye gaze in each entry We first have to calculate the eye gaze via the coordinates of the left and right eye If both eyes are tracked the arithmetic mean of the coordinate is determined If both eyes could not be tracked the entry is deleted immediately as already noted If just one eye is tracked we take the coordinates of the tracked eye Furthermore we have another idea which can be realised as well The x coordinates of the tracked left eye is mostly left to the x coordinates of the tracked right eye Therefore we could find an average distance and then adapt the eye gaze This slight change would make the coordinates of the eye gaze more realistic and therefore it would enh
28. rifocalReader inexperienced and experienced users At the end we want to evaluate the evaluation because the reason for use case was not to test the VarifocalReader instead to test the workload with aid of our implementations 7 1 Hypothesis Our hypothesis for the user study is that inexperienced users need way more time than the experienced ones but that it will become less time difference between the experienced and inexperienced probands for each new task because the inexperienced probands learn more and more about the program in the progress and therefore become slightly experienced probands themselves However we believe that there is not any difference about the correctness of the answers between the probands 7 2 Visualization of Recorded Data Fig 6 shows a sector of the visualization from the eye gaze of the tracked data from our use case after saccades were filtered with the Fixationizer using an AOI Sequence Chart 8 This repre sentation of the data is used for the analysis of the use case Every participant has its own AOI Sequence chart with each AOI having its own timeline This allows an easy comparison of individual partici pants Eye movements are represented as gray rectangles connected by lines whilst interaction data are depicted by colored circles The meaning of each color is described in 4 In this part of the visual representation it can be seen that partici pant 3 has the focus on the para
29. s 6 Text _________ Y fN go O O fN A H eo R E 9 hj t af J ER An Ny FN ee i 2 R A ia 6 a ZV O __ 9 0 ALLIT F P le _ e KA SN tay y Figure 6 Visualization of the recorded data using an AOI Sequence Chart 8 This figure shows AOI sequence charts from three different participants Interaction data are classified and therefore each interaction has a specific color based on the classification All interaction categories and their visual representation are precisely described in 4 Each layer was defined as an AOI independently of the used visualization Hence it made no difference for the AOI if the participants used a bar chart or pictogram These layers are chapters 1 subchapters 2 pages 3 paragraphs 4 and the text itself 5 as depicted in Figure 7 6 2 Integrating the framework into VarifocalReader was a fast process In the source code of VarifocalReader we called InteractionTracker initialize in the main function and stored the returned TrackingState object in a global vari able As soon as the main window was created multiple AOIs for the individual user interface areas see Figure 7 were added From previous studies there was already code present which saved the mouse click interactions into an Excel file We used this code and redirected the MouseEvents to the TrackingState onMouseCli
30. s in exploring and understanding documents based on the inherent structure of a document e g chapters pages and paragraphs 4 The VarifocalReader provides several views to show different abstractions of text annotations appropriate to users needs as well as results of search requests There are different visualizations that can be attached to each layer except the last one which displays the text These are either bar charts pictograms or word clouds Clicking on a word in the word cloud marks all such words The hierarchical perspective on text documents and the navigation concept are based on the SmoothScroll approach 13 It enables an analyst to navigate through the visualization and to keep track of the current position across all layers VarifocalReader FixationFilter o xX Aktion 1 EyeGaze Verzeichnis home maheswvn Dokumente WS15 ProjektINF FixationFilter data eyeGaze csv ffnen Aol Verzeichnis home maheswvn Dokumente WS15 ProjektINF FixationFilter data aoi csv ffnen Algorithm1 Algorithmus ausf hren i ID Timestamp x Coordinate y Coordinate Aol Type Duration ID Timestamp X Coordinate Y Coordinate AOI Type Duration a 1360 22551426 1271 5862165988801 1058 048293086405 3 Fixation 16624 0 1363 22601424 524 7934326166435 282 238948920349 2 Fixation 149866 0 1383 22934532
31. s with 0 when the first data is recorded G The range of the validity is between 0 4 it depicts how good the eye was tracked 0 eye coordinates tracked without any problems 4 the eye movement could not be tracked 4 These are the tracked eye coordinates If the validity equals 4 then the coordinates in the table are default 1920 and 1200 5 The AOI where the coordinates are located If they are located in a none predefined area the default is which means that no AOI is behold This is also the case when the eye gaze could not be tracked As one can see the raw data consists of data from both eyes the data is recorded for each eye independently In this figure the data of the left eye is depicted as red the other one as blue However for our evaluation of the user study not all eye gazes are important to consider There are two general types of eye move ments fixations and saccades Fixations are eye movements where participants focuses on a particular region whilst saccades are rapid eye movements between such fixations For most analyses the fix ations are needed that is why we decided to implement another external program Thus the users can decide if they need every eye movement or just the fixations This program filters out the saccades because they make the dataparsing more complex and the analysis gets more elaborate because more points are considered The algorithm which determines eye gazes fixatio
32. t wohl zu verstehen solang die Poetik den Anspruch erhebt alle je geschaffenen Gedichte Erdebewohner Feldschlacht Todes Komm Ziel Becher Myrmidonen Schrein Sohn Andrang Aufbruch Boros Eudoros Menesthios Perieres Figure 7 VarifocalReader The literary texts are divided into layers showing chapters 1 with word clouds subchapters 2 with bar charts and pictograms pages 3 with bar charts paragraphs 4 with word clouds and the text itself 5 As can be seen in this figure the word Hektors is selected The red marked bar charts in the second layer show in which subchapters the word is used and how often it is used We tracked all eye and mouse movements as well as the interac tions made by the participant Additionally we recorded an audio file where the user described what their intentions are for each action as well as what their answers are for the tasks To start the user study first Tobii Studio and the server had to be started After calibrating the participant we selected the program for our user study Before the program started we had to select an ID for the user and give additional information about him her which we use for the analysis Eye movement data was recorded with a Tobii T60 XL eye tracker sampling rate 60 Hz with a 24 inch screen and a resolution of 1920 x 1200 pixels To calibrate the eye tracker we conducted a five point calibration For the thinking aloud part we used the Rode NT
33. tellung Pathos Welt Wallenstein Sophokles Tragische Anstrengung Mensch Dichter Rede Faustin A Erwartung Held Leidenschaft Bewegung Handlung Rampe Redner Wallensteins Lessing Spannung Antigone Aufwand Corneille Elektra Endlichkeit Richter Soldaten Theater Wille Ziel Kleist Menschen Helden Fabel Homburg Schuld Szene ch Dramatischer Stil Spannung Pflanze Geist Sprache Erinnern s Tait Avimdesialen Ontalaaia DMhaaa 103 TE f uy 4105 106 107 103 epp Paragraph 23 aj Begriffe episch lyrisch dramatisch und allenfalls tragisc DIUCS DIGN IGI NUNY T UI tyYyaiiy f und komisch verstanden in einem Sinne jedoch Thesen Beschreibung Haller der sich von dem bisher blichen unterscheidet und gleich zu Beginn erkl rt werden mu Der Titel Poetik Parii bedeutet zwar l ngst nicht mehr eine praktische Lehre die Unge bte instand setzen soll regelrechte Gedichte par Epen und Dramen zu schreiben Aber die neueren Handlungen Gegenstand Poesi Schriften welche unter dem Namen Poetik gehen gleichen den lteren immerhin darin daB sie das Wesen Teile des Lyrischen Epischen und Dramatischen in bestimmten par Mustern von Gedichten Epen und Dramen Ursache Verbindung Zentrum vollkommen realisiert sehen Diese Art der Betrachtung parf stellt sich dar als Erbe der Antike In der Antike n mlich Malerei Eigenschaften Teile par war jede poetische Gattung erst in einer beschr nkten par f Zahl von Mus
34. tern vertreten Lyrisch etwa hie eine Anblickes Lesbia Ovid Dichtung die nach Anlage Umfang und zumal in der Metrik dem entsprach was die neun klassischen palf Alkman Stesichoros Alkaios Sappho Ibykos Simonides Bacchylides und Pindar geschaffen hatten So konnten die R mer Horaz als Lyriker gelten m lassen Catull dagegen nicht weil er andere Versma e w hlte Seit der Antike haben sich aber die Muster un bersehbar Laokoonfragment Handlungen vermehrt Wenn die Poetik weiterhin allen par f Einzelbeispielen gerecht werden will begegnet sie Stanze Duft Alcinas Corinnas par z x 5 z iiad Schwierigkeiten die kaum zu l sen sind und deren L sung These Zielstrebigkeit Widerstre wenig ErsprieBliches mehr verspricht Sie muB par f UM bei der Lyrik zu bleiben Balladen Lieder Hymnen Oden Sonette Epigramme miteinander vergleichen jede dieser Arten durch ein bis zwei Jahrtausende verfolgen und etwas Gemeinsames als den Gattungsbegriff der Lyrik ausfindig machen Dies aber was parf dann f r alles gilt kann immer nur etwas Gleichg ltiges sein AuBerdem verliert es seine Geltung in dem Augenblick da ein neuer Lyriker auftritt und ein noch unbekanntes Muster vorlegt Die M glichkeit einer Poetik parf ist deshalb nicht selten bestritten worden Man wei sich etwas damit dem historischen Wandel vorurteilslos zu folgen und lehnt jede Art von Systematik als ungeh riges Dogma ab Dieser Verzicht is
35. ther hard nor time expensive As already mentioned in section 6 2 Integration of the Frame work the integration of our framework into the VarifocalReader was a fast process as well The visualization of our data with the visualization software also proceeded without any complications In conclusion with aid of our framework the workload of analysis of user studies is reduced 8 FUTURE WORK The focus for future work should be put on the visualization of the captured data and expansion to other UI frameworks and devices The server application could be extended to provide a more user friendly interface as opposed to the current commandline based interface Furthermore the server could be optimized to consume fewer system resources in order to reduce the impact of the recording on the use case The client library could be changed to conduct the eye tracker calibration without needing to use the Tobii Studio As mentioned above the methods provided by the Java Runtime to record screen shots causes unacceptable performance problems This could be solved or at least reduced by using a dedicated screen recording software that is specifically designed to perform more efficiently than the Java Runtime The current method of recording what the proband says captures all available audio sources on the system The correct audio stream must be selected later by the person doing the analysis Providing a way to choose which audio source to record wo
36. to individual mouse clicks the location of the mouse pointer is recorded This data item only contains the current location of the pointer and the AOIs in which the pointer is currently located e MouseScrollDataItem Typically a mouse also contains a scroll wheel Interaction events from this device are also captured and are represented by a subclass of MouseMotionDatalItem In addition to the properties of the superclass it also contains the amount of how much the user has scrolled with this event e ScreenshotDatalItem When a screenshot is captured then a data item which contains the image data is generated For each screen connected to the system one screenshot is captured The data is a fully encoded image file which can be saved to disk without needing any further encoding 4 IMPLEMENTATION Based on the work shown in previous sections we created a list of design goals that the implementation of this framework would have to fulfill e Collection of the interaction data should require as little inte gration effort as possible e Independence from UI framework or programming language so the framework can be easily extended in the future e Tracking of multiple AOIs that may be added or change posi tion at runtime The first design goal was solved by keeping the programming in terface as minimal as possible Most of the recorded data is captured automatically without further intervention of the application that is being tracked
37. to listen on the same port by default 43248 as before so the next tracking session can begin without interruption 4 4 Data Import Once the data has been recorded it needs to be analyzed somehow The amount of recorded data is commonly too much to be analyzed manually So it needs to be converted into a data structure suitable for automatic processing A common data structure for this kind of task is a database that contains all the data and allows a more efficient retrieval of individual items or a group of related records For this project the database is a Microsoft SQL Server which can be accessed with a NET based language For this reason a generic parser for the generated interaction data was written in C The language features of C make writing a CSV parser easy while keeping it fairly robust At the beginning of the parsing process the saved metadata is con verted into the appropriate data structure using the JSON deserializer available in the standard NET and Mono framework To ensure a proper initialization order the parser first reads the file containing the individual AOI events Most other data items depend on this data so it is important that the AOI data is available when the rest of the data is parsed The AOI events are converted into so called AreaOfInterestPoint data structures which contain the timing and location information If an event contains in formations about a new AOI then the importer will also gen
38. ts on how to scroll in the program Nevertheless they were not informed on how the program exactly works Afterwards we tested their visual ability with an eyesight test as well as a color test Then we calibrated the participant with a five point calibration as mentioned before Each participant had to solve several tasks where each new task was slightly harder than the previous one All questions were based on a single text and with no pause between them for a continuous flow For each task participants had to find out for example where specific words or topics are or where specific words are used most Afterwards participants answered several questions about the VarifocalReader How they found the tasks and the program itself if it is easy to use and how useful they find this program The participants also gave some advice about how to improve the program and had the chance to give final remarks 7 ANALYSIS The main reason for the use case is that we want to test our framework and evaluate the results after we filter out the saccades with help of our fixation filter to see if the evaluation is facile and how big the effort is for the analysis We analyze the results from the participants using three main categories speed correctness and eye gazes Additionally we distinguish other characteristics as well Furthermore we grouped the probands for the analysis into two different classes based on their experience with the Va
39. uld streamline the process of recording interaction data Currently the audio capture system is also limited to saving the recorded au dio on the system the client runs on By sending the audio data to the server via the TCP socket and saving it with the other recorded data the files would not need to be copied by the person conducting the test which would remove another step needed for successfully executing a use case The preparation of the tested application needed before being able to conduct a use case is already pretty minimal but it could be further reduced by creating an external application that could be used to specify the AOIs that should be tracked This could be done by selecting user interface elements and saving this information for later use by the client library The Fixationizer is yet an extern application for detecting and removing saccades and computing the duration of fixations Future work could integrate the filter algorithm in our framework so that saccades will be filtered out internal and automatically The user should decide if saccades are needed or are not needed therefore the filtering should be optional Furthermore there are many approaches for filtering fixations and saccades that is why the algorithm can be optimized and or new algorithms could be added to our existing implementation that the user can choose the preferred filter algorithm 9 CONCLUSION The goal of this project was to create a framew
40. watch the text often because you did not really needed to read or watch it in order to answer the ques tions They just looked on the text to verify if they selected the right word etc The inexperienced users have read pretty much at the beginning of the tasks and also tried to find some solutions in the text e Other Characteristics The inexperienced participants played a little bit around with the program before trying to solve the first question The reason behind this is that they wanted to get familiar with the program first in order to solve the rest of the tasks more quickly One common problem which five out of six probands faced was that they struggled a few times with scrolling because it is an extraordinary technique but it got better the more time they were into the tasks For two participants a mouse scroll csv file was created which means that they still tried to use to scroll with the mouse 7 4 Usability of our Implementations Evaluating the user study is more efficient and less complex because when using our framework the results of the use case are more precise because all data is synchronized automatically Tracking of data with our framework is working without any issues We represent the results with the software from 4 as depicted in Fig 6 With aid of our fixation filter we reduce the exported data to the important informations we need for the evaluation that is why the analysis of the user study is nei
41. wegung Handlung Rampe Red gt 1488 24666546 38 1489 24683172 Wallensteins Lessing Spannung Ar 39 1490 24699920 1491 24716544 Aufwand Corneille Elektra Endlichke 2473008 Richter Soldaten Theater Wille Ziel l 4 isos 2178163 Menschen Helden Fabel Homburg S iiS ch Dramatischer S 47 gt 4506 24833037 A 48 1507 24849782 Pflanze Geist Sprache Erinne 5 150 24866408 50 1509 24883033 PARONGSS 0 778 671872578125 0739 416431030841 0 736 058806031942 0 745 582258530631 0 748 082147373789 88 8332888477635 0 49 0044988314366 98 5494307791669 0 0749 589362145489 0 752 468703444174 0773 41289662756 0751 526100936171 0747 47122841989 0 747 346221424086 0 763 018927266385 0 743 294363693276 0 1149 22138778944 0 872 548143760941 0898 359767912625 0 865 859423807269 0 876 303278709966 0 828 836986682145 0838 203031116718 NARI 438581226375 111 544150386635 99 902677575119 103 782858910472 90 8431834335715 84 5514415626894 96 3259626607396 71 0346623357509 81 7239183358652 106 72039988526 122 47626864837 84 0878394827087 108 435373981138 82 6658564346872 123 636665213417 89 4830158960758 99 RRRN17113H48R 821756938879844 89 3006134777352 83 8632548957321 75 227046538871 101 686990444523 3 111 041231412764 3 117 054386514701 3 82 1215242300281 94 6210847046132 38 28532644402 0 66 4731478819704 127 070895084034 3 G4 6244110001479 N 0 770 861710021854 0 778 145996077656 0
Download Pdf Manuals
Related Search