Home

NAVAL POSTGRADUATE SCHOOL THESIS

Contents

1. Pena Uy Qur tign Scale Frequency Mean Mode Median Question Designator I think most people 1 3 could figure out how 2 3 to use this software Learn_1 3 0 3 54 4 4 with the training 4 14 aids provided 5 4 I think most people l 3 could figure out how 2 5 to use this software Learn_2 3 6 3 4 3 with just the user s 4 9 manual 5 l 1 6 No more training is 2 l1 needed to effectively Learn_3 3 4 2 167 2 2 use this software 4 3 5 0 If given these tasks 5 in one month to 2 2 complete again I Learn_4 3 3 3 22 4 4 feel I could execute 4 10 I could show 1 1 someone who has no 2 2 experience with Learn_5 3 2 3 79 4 4 DaViTo basic 4 15 functionality 5 4 1 More training aids 3 ded f t z 2 eee Lean 6 3 4 35 4 4 effectively learn to 4 10 use the software 5 4 Table 26 Table of learnability questions and associated statistics 62 5 Satisfaction The attribute of satisfaction is linked to the users perception of the software Typically users will perform better when using a satisfying product that meets their needs Rubin amp Chisnell 2008 Questions concerning satisfaction listed in Table 27 had three results with scores less than three These questions are all related to the participants self assessment of their abilities to use DaViTo for basic analysis Sat_1 complex tasks without the user
2. Please explain Ouh DOE Scale Frequency Mean Mode Median I understood where I 1 1 was at in the 2 6 software as I Effect_1 3 3 3 375 4 4 progressed through 4 11 a task 5 3 1 1 The mouse 2 4 interacted with the Effect 2 3 5 3 667 4 4 software as I expected it to 4 12 5 5 1 11 I was satisfied with 2 9 the reaction time of Effect_4 3 1 1 875 1 2 the software 4 2 5 1 I felt there was an 1 3 unnecessary amount 2 8 of mouse clicking to Effect_5 3 7 3 25 4 3 complete the given 4 4 task 5 2 1 2 I made frequent z 9 See Erect 3 5 3 4 3 complete the given 4 3 task 5 5 Was the software Processing lag or crash simple to use Effect_3 Lack of a processing status bar Lack of geographic labels Table 24 Table of effectiveness questions and associated statistics 58 3 Efficiency The attribute of efficiency concerns the speed with which the user can accomplish a goal s While usually measured against a baseline time these questions address the user s perceived effective interaction with DaViTo Rubin amp Chisnell 2008 Questions concerning efficiency listed in Table 25 all had a score of three or greater except Eff_8 which scored a 2 25 Eff_8 directly pinpoints a serious lack of efficiency experienced by multiple users due to software freezes and subsequent crashes This can also be a source of user frustration affecting usefulness if users are dr
3. 60 Table of learnability questions and associated statis tics 62 Table of satisfaction questions and associated stati tics 64 Table of visualization questions and associated stati tics 65 Table of visualization questions regarding icons and associated statistics 66 Table of improvement questions and associated statistics 68 List of recommendations and the basis derived from analysis of the task e sutttt E ASEA A ENA R A a a R 12 xiii Table 32 Table 33 Table 34 Table 35 Table 36 Table 37 Table 38 Table 39 Table 40 Table 41 Table 42 Table 43 Table 44 Table 45 Table 46 Table 47 Table 48 Table 49 Table 50 Table 51 Table 52 Table 53 Table 54 Table 55 List of recommendations and the basis derived from analysis of the graph res ltS ienee haee tone eet ae Me ae ae A sed a ae see 74 List of recommendations and the basis derived from analysis of the three common question survey results 75 List of recommendations and the basis derived from analysis of the usefulness portion of the survey results c ce eeeeeceeeeesseeecseeeeceteeeeseeeenaeeees 76 List of recommendations and the basis derived from analysis of the effectiveness portion of the survey results 77 List of recommend
4. Table 59 Table of free form responses to question Improve_4 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_5 What did you most like about DaViTo Please explain Like ARCGIS the ability to display data on a map is very useful I m not sure how this software would be better than ARCGIS though and that is a widely disseminated and developed product that is significantly less buggy The ability to predefine areas that a user could then open view and process for analysis makes it possible to share work without having to recreate it while using it for different purposes That could be a great potential time saver that would also reduce confusion between units by sharing world views The set of the map with controls to the left Seemed much more logical and I could see more of the area I wanted to see Visual display of event data and layered maps Good visual representation of geographic data I like maps Overlay of chart data by geographic region so you can eyeball outliers trends The general interface It was intuitive Simple I liked that basic graphing information could be loaded to represent trend data for a given location It seemed simple It didn t look overwhelming or intimidating But in the end simplicity wasn t the best way to go Nothing That it is open source and can be improve The ability to populate differen
5. Table 61 Table of free form responses to question Improve_6 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_7 How would you improve the mouse interaction with the software Please explain I don t believe there were positive indications of mouse selection or actions associated with some of the functionality When specific selection is required definitive highlighting might be possible Nothing Unknown No improvement necessary Right click on map gives all operations Nothing I wouldn t It was fine Nothing No comment Just make right clicks faster Also give more ways to navigate with just the mouse like panning I think there is that option but it was too slow I wouldn t It seems that the mouse interaction works properly The mouse is fine It worked fine Nothing works just as expected with left and right clicks for primary and secondary actions Mouse clicks for the distance tool was a bit confusing Scroll map while depressing the left mouse key 135 Q Improve_7 How would you improve the mouse interaction with the software Please explain Integration of map navigation into mouse buttons Change how the chart menu option is brought up by clicking on the location Seemed to take numerous attempts for the menu to pop up No issues with the mouse None Table 62 T
6. 20000 4 _ aw Regions Graph of final session task seven cumulative counts of gaze observations for all subjects Task seven directed subjects to navigate to Iraq without the use of the Map Controls portion of the interface Counts correlate to the number of subject fixations in each region shown in Figure 23 108 Gaze Observations in Each Region nay 1 ow l 200 0 4 i ow na E Figure 33 Graph of final session task eight cumulative counts of gaze observations for all subjects Task eight directed subjects to go back to the previous view Counts correlate to the number of subject fixations in each region shown in Figure 24 Gaze Observations in Each Region 1000 00 400 004 70 00 Repions Figure 34 Graph of final session task nine cumulative counts of gaze observations for all subjects Task nine directed subjects to remove all data sets and Shapefiles then load the Indure_Demo data set and the afghanistan_provincial_boundaries Shapefile Counts correlate to the number of subject fixations in each region shown in Figure 25 109 Figure 35 Figure 36 Count Gaze Observations in Each Region 2000 00 900 00 0 004 _ i Regions Graph of final session task ten cumulative counts of gaze observations for all subjects Task ten directed subjects to answer a question th
7. SSO aS 43 1 Analysis of Session One Graph Data The participants were introduced to the five graph types identified in Table 11 in session one Three identical questions were asked about each graph type listed in Table 12 These questions were similar to those a possible user in the field or analyst would ask such as how many events occurred in this region in a certain month this year when attempting to conduct trend analysis of a large array of data Subject answers were evaluated for time to answer and accuracy of their answer Several subjects exhibited frustration with some of the graph types due to the amount of data contained or the format in which it was displayed Upon completion of session one each graph type was explained to the subjects and their performance reviewed This was to determine if another graph type would perform better with a simple explanation such as one that could be stated as a mouse over instruction programmed into DaViTo The stacked bar format SBF performed well in session one with 80 56 percent correct answers and an average time to answer of 27 8 seconds as shown in Table 13 The SBF outperformed all other chart types by more than 25 percent in percent correct and was second in time to answer to the stacked area format SAF by less than three seconds Despite the slightly slower time to answer the much larger difference in correct answers makes this the better performing format
8. Figure 18 Graph of average response to common question two Note that the y axis correlates to the scale in Table 19 The final common question asked the subjects to list their likes and dislikes about the DaViTo program for that session Unlike the final survey where questions were written with specific regard to the usability attributes described in Table 1 these questions have no content to lead the subject and are strictly free form entry The results 53 of common question three are shown in Tables 20 21 and 22 for sessions one two and three respectively Full transcription of comments can be found in Appendix D Common liked features across all three sessions were ease of use and the user interface layout Navigation functionality simplicity and geo data representation appeared in the top five in more than one session The single common disliked feature across all three sessions was response time Mouse interaction lack of a progress status notification and graph functionality were listed in the top five on more than one occasion It is also of note that some features were liked by some participants while disliked by others Recommendations for improvement based on the liked and disliked features are discussed in Chapter V Top 5 Liked Features ine Top 5 Disliked Features nes Cited Cited Ease of Use 24 Response Time 20 Layout 14 Distance Function 10 Navigation Functionality 8 Layout
9. E Response Figure 46 Session two question one with responses on a scale of one very poor to five very good Session 2 Question 2 How many more training hours are needed vw 2 o 2 2 3 a 5 o 2 E EJ z E Response Figure 47 Session two question two subject s self evaluation of additional training hours needed 121 Session Two Question Three What were your likes and dislikes with your second exposure to DaViTo Likes Overall like the product and potential Dislikes The functionality associated with the shapes gave me a bit of trouble following the tutorial and associated outside the circle left click didn t work at first Likes Easy to understand Helps to have the tutorial run through the steps Would take more time using it to get all the steps correct Just more practice time on the tasks that need to be accomplished Dislikes I did not like that the graph display did not have an easy method of changing the axis increments The recall of graphs also seemed to take a little effort Likes The display of the charts for the different regions is helpful Dislikes The right click function did not work to display the full menu on several occasions on the map and the chart view It would be useful to have an hourglass to show progress when loading data sets Likes I enjoye
10. 0 0 1 3 4 5 6 7 R 9 10 11 Figure 4 Graph comparing the bench mark to average participant mouse clicks for both complete and incomplete session three tasks 2 Analysis of Time Duration Correctness and Completeness of Final Session Tasks The time for every participant to complete each session three task was automatically recorded by the EyeWorks software This was then compared to a 30 benchmark time established by the experiment designers as the most efficient possible time in which to complete the tasks correctly Task descriptions for session three are in Appendix E The summary results for the duration of tasks in session three and percent correct are displayed in Table 10 The most notable result from the analysis of time to complete tasks and similar to the mouse click analysis was the large deviation from the baseline for tasks four and ten shown in Figure 5 Again task four directed the participants to create a rectangle chart on the data point in the southwestern most part of Kenya and display the chart While the bench mark for task four was 62 09 seconds on average it took subjects more than 100 seconds longer to complete it with a standard deviation of 67 02 shown in Table 10 There was little deviation between users who were able to correctly complete task four from those who could not but the large deviation for both is shown in Figure 6 Task ten was a more complex task directing the subjects to
11. In an effort to look more inviting C 26 bland Add more to users software should have visual appeal even contrast in military applications Ghanae the defaiile Masvazanen The current default of gray has no C27 contrast with the interface due to frequent use of data point color different shades of gray Table 40 List of recommendations and the basis derived from analysis of the improvement portion of the survey results D ADDITIONAL AUTHOR RECOMMENDATIONS Throughout the development of the experiment and observing subject sessions the authors of this thesis received numerous hours of exposure to DaViTo Table 41 contains additional recommendations by the authors based on their observations and interaction Recommendation D 1 would increase the usable area of the software for data representation and analysis by allowing users to collapse the menu areas on the left of the user interface Once the files are opened or the options in the segments of the interface on the left side of the screen are complete they are no longer very useful and only remove work space from the user D 2 and D 3 are intended to address future possibilities for the use of DaViTo as discussed with TRAC Monterey It is desirable for the use of this software to spread beyond the realm of the small group of analysts it was designed for and the inclusion of nautical capabilities would advance this Training for the new target audience as well as advanced tr
12. Different graphs have better different graph types performance relative to the question asked Table 32 List of recommendations and the basis derived from analysis of the graph results C RECOMMENDATIONS BASED ON SURVEY RESULTS Survey results constituted the only source of participant feedback on DaViTo Although subjective the majority of participants seemed genuinely interested in providing useful feedback ensuring the hours they devoted without compensation to this research was not wasted 1 Recommendations Based on the Three Common Questions Currently the only training product developed for and distributed with DaViTo is a 26 slide PowerPoint presentation While this covers much of the basic functionality of the software it is not effective at training a user to use the software such that they meet the goals stated when developing DaViTo Implementing recommendations C 1 and C 2 listed in Table 33 would address the current learnability deficiency in the software Recommendation C1 is to develop a tutorial based training product for DaViTo While the tutorial method developed for this experiment was not itself evaluated against the PowerPoint presentation or any other training methods users did rate these sessions highly on exit surveys with an average of 4 55 on a scale of one to five and 21 of 24 74 users were able to complete all final session tasks albeit with only one perfect score Recommendation C2 is to dist
13. please answer the following question what were the total events for IED attacks for 2010 in the most Northwestern provincial boundary Similar to the mouse click data Table 10 shows a large deviation from the baseline of 63 36 seconds for this task The average user completed this task in 141 47 seconds with a standard deviation of 76 17 This time was even longer for subjects who completed the task correctly with Figure 6 showing the deviation between the baseline incorrect and correct task completion Note that task completion refers to finishing the task not answering the question correctly Only one subject was able to answer the question asked in task ten correctly Both tasks required the subjects to make a shape within which a graph would be made Once made the user must click outside of the shape to get the graph to populate within it DaViTo would frequently lag when doing this and without a progress status users would repeatedly click outside the shape in an effort to get the graph to populate Confusion from the delay and no status indication caused some subjects to give up having no idea if the software was processing their command and is indicative of a usability deficiency 31 see kee Sees ain mex staper ore Task 1 10 72 25 38 9 50 78 13 17 17 100 Task 2 42 96 52 66 10 53 202 92 36 46 100 Task 3 9 55 14 86 3 25 48 66 9 27 100 Task 4 62 09 165 69 52 28
14. NAVAL POSTGRADUATE SCHOOL MONTEREY CALIFORNIA THESIS USABILITY TESTING AND WORKFLOW ANALYSIS OF THE TRADOC DATA VISUALIZATION TOOL by Darren Cutler Tyler Rasmussen September 2012 Thesis Advisor Ji Hyun Yang Second Reader James Henry This thesis was performed at the MOVES Institute Approved for public release distribution is unlimited THIS PAGE INTENTIONALLY LEFT BLANK EEO POO UMENTATON T ICE Public reporting burden for this collection of information is estimated to average 1 hour per response including the time for reviewing instruction searching existing data sources gathering and maintaining the data needed and completing and reviewing the collection of information Send comments regarding this burden estimate or any other aspect of this collection of information including suggestions for reducing this burden to Washington headquarters Services Directorate for Information Operations and Reports 1215 Jefferson Davis Highway Suite 1204 Arlington VA 22202 4302 and to the Office of Management and Budget Paperwork Reduction Project 0704 0188 Washington DC 20503 1 AGENCY USE ONLY Leave blank 2 REPORT DATE 3 REPORT TYPE AND DATES COVERED September 2012 Master s Thesis 4 TITLE AND SUBTITLE Usability Testing and Workflow Analysis of the 5 FUNDING NUMBERS TRADOC Data Visualization Tool 6 AUTHOR S Darren W Cutler and Tyler J Rasmussen 7 PERFORMING ORGANIZATION NAME S AND ADDRESS ES 8 PERFO
15. Potential Risks and Diseomforts The potential risks of participating in this study are minimal You will be sitting in front of a large monitor using a conventional mouse and keyboard to imteract with the DaViTo software The eye scan camera systems have infrared lights associated with them These are no more harmful chan normal room lighting There will be many safe guards im place to protect the participant s confidentiality hawever there is always a minimal risk that there mighe be a breach of confidentiality ars cee APPRBVED Updated on 4 10 12 APR 11 2012 EXPIRES SFP An on 144 Anticipated Bemefits Anticipated benefits from this shady are to improved the usability for the DaViTo software and provide visualization data in a clear and understandable way foe the decision maker There is no direct benefit to you for participating Compensation for Participation No tangible compensation will be given to the participant Confidentiality amp Privacy Act Any information that is obtained during this study will be kepe confidential to the full extent permitted by law All efforts within reason will be made 10 Keep your personal information in your research record confidential but total confidentiality cannot be guaranteed All surveys will be protected by a password security and at a secure NPS terminal If it is mecessary to print any surveys all personally identifiable information PID will be scrubbed so the participant c
16. APPENDIX D ADDITIONAL SURVEY DATA 117 A THE THREE COMMON QUESTIONS csccssscssssssssssseccssscssssssssssesees 117 B FINAL SURVEY ssccssssstvessessesssecsasns coves ova sonssudestenwasisessune covenevasdosesversseswudteanes 126 APPENDIX E TASK WORKSHEE T cccccsssssssssossssssssssssssscsosscssssssssssnecsonsssscssensssnsccoss 139 APPENDIX F APPROVED IRB DOCUMENTS cssccssscssscsssccssscssscssccssscssnesceesesses 141 LIS TOF REFERENCES isssaseccestisactiscksineccatascestesenacetasnosecansnsassesncsauteecedncceandetaedberonanetasoane 155 INITIAL DISTRIBUTION LIST P 157 viii Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 Figure 18 Figure 19 LIST OF FIGURES Birds eye view diagram of the equipment configuration Descriptions of GACH Ais Mare dN Table 2i Aansen EE E A N 14 Interface introduction slide from Tutorial Session One This slide shows the breakdown of the DaViTo interface into five functional areas 17 Graph comparing average participant mouse clicks to the bench mark 30 Graph comparing the bench mark to average participant mouse clicks for both complete and incomplete session three tasks 30 Graph comparing average participant task durations to the bench mark 32
17. Regions Graph of final session task three cumulative counts of gaze observations for all subjects Task three directed subjects to go to the Africom view Counts correlate to the number of subject fixations in each region shown in Figure 20 106 Gaze Observations in Each Region 1000 09 Regions 00 4 Figure 29 Graph of final session task four cumulative counts of gaze observations for all subjects Task four directed subjects to create a user defined polygon and display the chart Counts correlate to the number of subject fixations in each region shown in Figure 9 Gaze Observations in Each Region Count 2 Regions Figure 30 Graph of final session task five cumulative counts of gaze observations for all subjects Task five directed subjects to close and delete the chart generated in task four Counts correlate to the number of subject fixations in each region shown in Figure 21 107 Figure 31 Figure 32 Count Gaze Observations in Each Region 300 00 20 4 200 mo 100 00 20 io Graph of final session task six cumulative counts of gaze observations for all subjects Task six directed subjects to hide the Map Controls ribbon bar Counts correlate to the number of subject fixations in each region shown in Figure 22 Gaze Observations in Each Region 200 00 0 0
18. Statistics for all five chart types are shown in Table 13 Figure 13 shows a comparison of the data with top performance in the upper right corner This figure highlights the large difference in performance from the other formats and the small time advantage of SAF Additionally subjects answered the questions quickly with SBF which posted the second fastest average time to answer a question It is also of note that the SBF outperformed the other types in every question in session one with these additional statistics in Appendix C The distant second place format in accuracy was the clustered columns format CCF although it took the longest for users to evaluate While it had much worse performance on average it should be noted that on question B in Table 12 its 44 performance was similar to that of the SBF with additional statistics in Appendix C The CCF and SAF graphs had similar accuracy performance to the LCF The worst performance in accuracy was the SLMF Session 1 Accuracy Correct Time sec CCF 54 17 58 33 SBF 80 56 27 81 LCF 47 22 38 38 Table 13 Overall statistics for all graph types in session one Session One Time vs Accuracy Accuracy correct 0 10 20 30 40 50 60 70 80 90 15 20 25 K SAF oor 30 E SBF E SBF 35 ASIME LCF A LCF g 40 SLMF E 45 50 K SAF 55 60 CCF 65 Figure 13 Plot of session one time to answer versus accurac
19. development process designers may create software from their point of view rather than that of the user resulting in a deficient design Rubin amp Chisnell 2008 Questions concerning usefulness shown in Table 23 all had a score greater than three except Use_6 which scored a 2 66 It is of note that the user s manual was not consulted by any subject despite being available and there is no electronic user s manual or help function accessible from the DaViTo interface While the majority of subjects selected response three neither agree nor disagree the other users based their response on the perceived difficulty of the software rather than by actually referencing the manual This indicates a potential deficiency in usefulness with regards to the ability to get assistance from a manual or help tab within the interface Question Use_2 results 55 also identify usefulness deficiencies in the areas of graph interpretation and construction as well as polygon construction Deficiencies will be addressed by recommendations in Chapter V Usefulness Question SEEM Scale Frequency Mean Mode Median Designator 1 1 The task 7 eee eeu ine a Use_1 3 5 gau Wil A 4 4 14 5 3 1 0 The terminology for 7 4 buttons and Use 3 3 J 3 79 4 4 commands was clear and understandable 4 13 5 5 1 1 A computer novice 7 would be able to use 7 this
20. i regards to satisfaction time and stability Includ h tools t Satisfaction This was the third most frequent C 19 improve graph f f a complaint with regards to satisfaction readability Table 38 List of recommendations and the basis derived from analysis of the satisfaction portion of the survey results 7 Recommendations Based on Visualization Questions The visualization segment of the final survey generated only one unique recommendation found in Table 39 There was some confusion among the users with regard to the identification of the icons discussed in recommendation C 23 Since both have similar functionality it is recommended to use a common icon 80 Additional recommendations in Table 39 are C 20 C 21 and C 22 These are repeats of C 4 B 2 and C 13 respectively These recommendations were derived from participant free form responses to visualization questions and are tailored to improve the interpretability of the displayed output data within DaViTo Designator Sas ata Attribute and Basis for Recommendation Description Visualization Only 12 5 percent of subjects Include graph tools to agreed that output data is clear to interpret and C 20 improve graph the second most frequent response to desired readability changes to the output was more graph manipulation tools Addni optonsioi The most frequent response to C 21 desired changes to software output was m
21. Time Monthly kidnapping Bombing Armed Attack gt Other Arson Firebombing Figure 11 Example line chart format LCF with markers from the DaViTo software 40 Region 13 Afgan 2007 2008 Time Monthly Figure 12 Example cumulative stacked bar format SBF from the DaViTo software Five different graph types were constructed for the data representation portion of the experiment Each format is summarized in Table 11 In the experiment sessions there were questions about each graph format for which the data was randomly generated from session to session These questions are listed in Table 12 with a letter designator This allowed the use of the same questions for each graph but each had a different answer These graph types were chosen because two of them are already available for selection by the novice user and DaViTo has the capability to easily generate the others 41 EventA EventB Event C EventD eta etmon ET PO et das et 3ny etni et unr ethen Event D t Event C E EventB Event A m Event D mEventC m EventB m Event A 5 Etudy eten ETq 4 S etuer 8 zta Z A0N z1 Po zrdas zt 3ny zin ztunr zT AeW zTudy ZTE ZT zr uer l 2 3 9 g9 8 o s o as 8a
22. as expected it to o1 o2 n3 gs os Was the software simple to use Written response was satisfied with the reaction time of the software oi a2 03 a4 os felt there was an unnecessary amount of mouse clicking to complete the given task og D2 o3 Da os I made frequent errors trying to complete the piven task ol n2 a3 og os 149 Quantitative amp Subjective Questions Learnability refers to competence training time confidence relearn an activity 1 i think most peoplecwouk figure out how to use this software with the training aids provided oi o2 o3 o4 os Pthink most people could figure aut haw to use this software with just the user s manual tap o2 o3 o4 os No more training is needed to effectively use this software oi o2 o3 o4 os if given these tasks in one month to complete again feel could execute them ol o2 o3 o4 os I could show sameone who has no experience with DaViTo basic functionality Di n2 o3 D4 os improvements refer to the users suggestions User s manual is easily understandable o1 o2 os a4 os More training aids are needed for me to effectively learn to use the software oi u2 a3 o4 os How would you improve DaViTo Written response What was the most frustrating feature of DaViTo Written response What did you like most about DaViTo Written response What would you change about the color scheme Written response Haw would you improve the mouse interact
23. either neutral or disagreed that a computer novice would be able to use DaViTo with only the user s manual Despite this accessibility to the user s manual would increase usefulness by providing a reference for assistance when problems arise and linking it to the help tab would place it where most users would expect as it is commensurate with commonly used software 75 When subjects were asked if the software is simple to use it was most frequently stated that the three most difficult tasks to accomplish involved graph interpretation graph creation and generating user defined polygons Graph tools such as a scrolling capability and mouse over functionality to display the value of the data point would assist in the implementation of recommendation C 4 which is to include graph tools to improve graph readability C 5 recommends making graphs easier to create This is likely related to the excessive mouse clicking required when drawing a polygon and the subsequent graph creation Finally drawing and populating a polygon when drawn by the user designated recommendation C 6 is identical to A 3 but is based on a usefulness deficiency identified by the participants R dati Designator S l N Attribute and Basis for Recommendation Description Link the user s manual C 3 to the Help tab on the ribbon bar Usefulness Currently the user s manual is not accessible via the DaViTo interface Includ h tools t te Gai Useful
24. in Table 12 it had the lowest performance with the additional statistics in Appendix C Session 2 Accuracy Correct Time sec CCF 83 33 30 38 SBF 64 58 28 88 LCF 50 00 42 79 SLMF 72 92 29 31 SAF 75 00 34 62 Table 14 Overall statistics for all graph types in session two 46 Session 2 Time vs Accuracy Accuracy correct 0 20 40 60 80 100 20 CCF 25 SME E SBF 7 SBF E A LCF g 30 X wr T lt SLMF 35 s K SAF SAF 40 A LCF 45 Figure 14 Plot of session two times to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in the upper right corner 3 Analysis of Session Three Graph Data In session three the participants were again asked a series of questions from Table 12 about each of the five graph types identified in Table 11 Two identical questions were asked about each graph type similar in type to those of sessions one and two Subject answers were evaluated for time to answer and accuracy of their answer Once again several subjects exhibited frustration with some of the graph types due to the amount of data contained or the format in which it was displayed Upon completion of session three there was no review of performance or explanation of graph formatting In this session the SBF and CCF were the top performers in accuracy at 75 percent although the CCF took on average greater than 15 seconds to answer as s
25. integration into map Likes 1 Ease of use Resembles other map programs 2 Ability to quickly toggle between chosen projections without using the zoom feature 3 Theater drop down to focus on the different AORs Dislikes 1 Distance tool line blends in with rest of map lines Not distinguished Likes Ease of use Convenient lay out Dislikes Lack of labels Likes 1 Simple menus 2 Quick feedback when a tool was used distance zoom Dislikes 1 not sure yet Likes user friendly dislikes when using the pan tool the delay caused by moving the map Likes 1 Very user friendly 2 Good graphical user interface 3 Follows standard format like most Windows programs so if you are a Windows user it was easy to understand Dislikes 1 A little slow with zooming out Likes Simplicity Familiarity distance measurement Dislike Scaling back out no scroll wheel use having to come up with three dislikes Likes Ease of use Ease of understanding of the functionality of the tool Simplicity of functions Dislikes Latency of program reaction to user demand D Layout of Screen D Default color scheme should have used Army colors Table 51 Table of likes and dislikes for all 24 subjects in session one Comments taken directly from the EyeWorks software with no modification 120 Session 2 Question 1 How would you rate your second training session 14 12 10 Number Of subjects
26. interaction device of choice In most cases left clicking selects an object or carries out a function right clicking brings up a menu of options In DaViTo a specific converse to this was right clicking outside of a polygon in order for it to be drawn and populated with statistics This additional click was problematic and confusing for some participants Today applications such as Google Earth MapQuest and other mapping tools have a rich user interface Not only are they more visually appealing it also has functionality when trying to understand the terrain of the area you are looking at and keeps users more engaged Recommendations C 25 C 26 and C 27 in Table 40 will incorporate a 3D mapping capability add contrast to the color scheme and change the default data point color from gray to a color with increased contrast This will improve the visualization of DaViTo as well as its usefulness and satisfaction 82 Recommendation Designator Attribute and Basis for Recommendation Description Standardize two Multiple Attributes Follow the standard C 24 button mouse functionality for a two button mouse that most functionality software developers utilize I t ae e ap Visualization Top response when subjects were relief option or C25 me F asked what they would change about the color capability similar to f scheme or user interface layout Google Earth The color scheme is Visualization
27. likes at this time Likes liked the way you can bring up data on specific events Dislikes I didn t like the slowness of the program There appears to be decent amount of lag Overall it seems cluttered lots of stuff to see and not very well organized I don t see how anyone can get anything out of the thumb nails of the graphs that appear on the map Plus they occlude the information under them 122 Session Two Question Three What were your likes and dislikes with your second exposure to DaViTo Like Intuitive to use select a specific area and use shapes to draw graphs The different options to modify plots e g freq vs percent normalized etc Dislike Since there is an inherent lag in the system a user feedback prompt or in progress bar would be helpful to the user Inability to change the line colors in the plots Likes Reviewed previous Session so that quickly got me reoriented to the tool Gained knowledge on the system to include informational Statistical plots Focused was on the different views and map manipulation Likes Clear instructions good examples of tasks Hands on is definitely a benefit to learning the application rather than just watching the video Dislike Is really my personal preference to see the instructions while I m executing the task Since I don t consider myself to be proficient with this software yet I d like the option to refer to the question you re asking as I go
28. mouse clicks should not be necessary at all The procedure of drawing the shape and then right clicking outside the shape to initiate its formation is unnecessary and counter intuitive Like other programs with shape drawing functionality Microsoft Paint PowerPoint Word etc the code should be modified to not require this additional click as recommended by A 3 but rather initiate once the mouse button is released Finally the heat map analysis indicated a large concentration of subjects gazing and clicking in the Map Controls Region Again the file manipulations functionality is counter intuitive to common programs used Recommendation A 4 will increase effectiveness for users Recommendation Designator Attribute and Basis for Recommendation Description Improve software Efficiency When drawing user defined A l processing response polygons there was excessive lag time or the time and stability software crashed Effectiveness When drawing polygons the lag Implement a progress time led subjects to continue clicking outside the A 2 ae status bar polygon because there was no indication of progress D d lat Effectiveness Right clicking outside of the A 3 polygon when drawn es polygon was counter intuitive to users by the user Implement file Effectiveness Heat mapping shows that many A 4 operations in the main users looked to the menu ribbon for file menu ribbon operations Tabl
29. on how to use the left hand mouse for controlling the tutorial on the left screen and how to use the right hand mouse for DaViTo and the EyeWorks software that controlled the experiment when moving from DaViTo to graph and survey questions The actual experiment was now ready to begin The experiment proctor would push record on the EyeWorks interface and instruct the participant to begin For sessions one and two the participants were directed within the EyeWorks window to follow along with the tutorial and it would then exit out to the DaViTo interface For these sessions upon completion of the tutorial they would go back to the EyeWorks window The subject would then proceed through the graphing and survey questions until complete At this time recording in EyeWorks would be stopped the data saved and the subject reminded of their future session times Similarly for the final session the experiment would begin in the EyeWorks window with the initial task The participant would proceed through all of the tasks transition to graphing questions and conclude with a longer series of survey questions The subject was then thanked for their participation and dismissed The proctor saved the information under the participant s number and proceeded to setup the system for the next session 23 F DATA COLLECTION AND ANALYSIS STRATEGY 1 Eye Tracking Data This data collection was method was used during all three sessions The eye tracki
30. parma Attribute and Basis for Recommendation Description C7 Improve the software Effectiveness 83 percent of subjects were not reaction time satisfied with the software reaction time Improve software Effectiveness Subjects stated this as the worst C 8 processing response deficiency that reduced the simplicity of use of time and stability DaViTo J Effectiveness Subjects stated this as the second C 9 i Pg worst deficiency that reduced the simplicity of status bar use of DaViTo Include geographic Effectiveness Subjects stated this as the third C 10 labels in the Map and worst deficiency that reduced the simplicity of Data Display Area use of DaViTo Table 35 List of recommendations and the basis derived from analysis of the effectiveness portion of the survey results 77 4 Recommendations Based on Efficiency Questions Recommendation C 11 in Table 36 is again related to improving software processing latency and is the same as recommendation A 3 Similarly recommendation C 12 including geographic labels is identical to C 10 Recommendation C 13 will allow data sets of different data types to be sorted with different colors Large datasets such as the one used frequently in the experiment WITS csv have multiple data types Additionally when multiple datasets are loaded they all continue to use the default gray color scheme Automatic color coding of the different subsets was recommended by multiple
31. research scope and limitations Chapter II Background This chapter introduces and explains the DaViTo software and usability Additionally the purpose of usability testing generally and with respect to DaViTo is broken down Chapter III Methodology The experimental design is discussed in depth to include training and tasks subject participation and the actual experimental procedure Chapter IV Results and Data Analysis This chapter presents results from the collected data and explains how the various sources are analyzed The findings of the data analysis are also discussed Chapter V Recommendations This is the consummation of the research into recommendations for change and improvement of the software to improve its usability This is input for TRAC Monterey as the software continues to be improved and is broken down into four categories recommendations based on task results graph results survey results and author recommendations Chapter VI Conclusion and Future Work The final chapter will summarize the results and recommendations and detail future recommended work Appendix A Consolidated Recommendations This appendix contains all 37 recommendations to TRAC Monterey in one table Appendix B Additional Task Results This appendix contains additional results from the task analysis portion Appendix C Additional Graph Results This appendix contains additional results fr
32. simple Likes I liked how the tools were easy to see and use on the left panel I ve have experience with other military mapping programs and they have always confused me on how mapping data is loaded and accessible by the user Dislikes I don t like how slow the refresh rate it I would get frustrated using this program in a stressful environment when I would start clicking to many times and get into a do loop because the program was too slow That is about it Likes Multiple options to manipulate redundancy user interface is intuitive color are not distracting and appropriate Dislikes System latency zoom option the squared icon on the tool bar when selecting a location hard to center no country labels provided similarly states provinces districts etc Likes Easy to see the maps on the right display Once I learned the navigation portion it was easy to zoom in on a specific area Easy to load data sets Dislikes I got confused on zooming in with the mouse scroll I went the opposite direction Likes 1 Fairly intuitive 2 Efficient selection of functionality no more a couple clicks to do what I wanted 3 Simplicity functionality I was instructed in during this session was all visible on the screen Dislikes 1 Didn t like the position of lat long and distance down in the left corner my preference would be maybe in the top middle closer to where I was looking ideally the distance would show up in the middle of the great cir
33. software with Use_4 3 4 3 16 4 3 the training 4 11 provided 5 1 1 4 A computer novice would be able to use 2 5 this software with Use_6 3 11 2 66 3 3 just the user s 4 3 manual 5 1 What was the most Reading interpreting the graphs difficult task to Use_2 Creating the graphs complete Drawing polygons Table 23 Table of usefulness questions and associated statistics 56 2 Effectiveness The attribute of effectiveness refers to whether the product behaves as the user expects or that users can use the software to do what they desire it to do While this can be measured against a baseline rate of error these questions addressed the users perceived effectiveness of their interaction with DaViTo Rubin amp Chisnell 2008 Questions concerning effectiveness listed in Table 24 all had a score of three or greater except Effect_4 which scored a 1 875 The results of this question once again address the latency of the program and point to an effectiveness deficiency This can be a source of user frustration also affecting usefulness if users are driven to stop using the software Question Effect_3 also identifies effectiveness deficiencies in the areas of software lag the lack of a processing status bar and the need for geographic labels within the Map and Data Display region Deficiencies will be addressed by recommendations in Chapter V 57 Effectiveness Question
34. to sort data types within a set by color Table 25 Table of efficiency questions and associated statistics 60 4 Learnability The attribute of learnability is a sub attribute of effectiveness and concerns the user s ability to operate the system whether it is after some period of familiarization and or training Additionally this element can refer to a subject s ability to use the software effectively after some period of inactivity Rubin amp Chisnell 2008 Both of these were addressed by the learnability segment of the final questionairre Questions concerning effectiveness all had a score of three or greater except Learn_3 with a score of 2 17 and are presented in Table 26 Similar to the second common question of all three sessions Learn_3 requires the participant to evaluate if they are adequately trained to use the software Like common question two the majority of subjects do think additional training on DaViTo is necessary Also of note is Learn_2 which like Use_6 concerns the use of the user s manual The responses to Learn_2 averaged three and 10 subjects agreed or strongly agreed while Use_6 only scored a 2 66 and only four subjects agreed or strongly agreed Despite these conflicting responses the fact remains that the user s manual was never consulted by any participants and there is little help available via the interface These deficiencies will be addressed by recommendations in Chapter V 61
35. two recommendations listed in Table 31 A 1 and A 2 were touched on by nearly every subject and will be mentioned in multiple sections of Chapter V for a variety of different bases Recommendation A1 requires revising the software code to improve software processing response time and stability Recommendation A2 is to implement a processing status bar to alert the user that the software is operating and is not frozen As discussed in Chapter IV multiple mouse clicks would take place and wasted time elapse as participants waited with growing impatience for the polygon to form and populate This lack of efficiency in the software code resulted in user frustration which the absence of is one of the definitions of usability mentioned in Chapter II Not only is efficiency at issue here but also usefulness and satisfaction As DaViTo performance deficiencies reduce the ability of the user to complete their goals efficiently the user then becomes less likely to use the product at all and their perception of the product becomes more negative Rubin amp Chisnell 2008 Finally most software has lag time at some 71 point as it computes things and the implementation of A 2 would reduce the dissatisfaction of users As the moments pass at least the user will know the program is processing rather than being left guessing as to whether it is simply frozen The excess mouse clicks were the result of frustration and no status of progress however the
36. was no tutorial Prior to use each day it was necessary to check the calibration of the eye tracking cameras and software If required a full system calibration was conducted Additionally stereo and IR camera positions were verified to be correct DaViTo was then started and checked for proper function in case of any system freezes Upon arrival for session one subjects were directed to read the Consent to Participate in Research form contained in Appendix F and to sign and date it if in 22 agreement The form was then signed by the researcher as a witness Following that they were directed to complete a demographic questionnaire contained in Appendix F The researcher instructed the subject to sit in front of the cameras and screens in a comfortable position They were then linked up with faceLAB and eye tracking commenced stepping through the steps of the software to properly track facial components such as the corners of the eyes and mouth in order to optimize the tracking performance FaceLAB tracking was then checked by following a series of dots on the screen This data was saved by taking a screenshot and recording it on the experiment checklist Once complete control was then shared with EyeWorks and an additional dot check was done by having the participant follow dots on the screen This completed setup of the participant with the eye tracking software Before beginning the experiment the subject was given final instruction
37. 2 2012 from Dictionary http www merriam webster com dictionary usability Evangelista P Henry J Buttrey S amp Whitaker L 2012 Data Visualization Tool DaViTo Monterey US Army TRADOC Analysis Center Raytheon BBN Technologies 2005 What is OpenMap Retrieved from OpenMap Open Systems Mapping Technology http openmap bbn com whatis html Rosson M amp Carroll J 2002 Usability engineering San Francisco Morgan Kaufmann Rubin J amp Chisnell D 2008 Handbook of usability testing How to plan design and conduct effective tests Indianapolis Wiley The R Foundation for Statistical Computing n d What is R Retrieved August 4 2012 from The R Project for Statistical Computing http www r project org U S Army TRADOC Analysis Center 2012 January Data Visualization Tool DaViTo user guide Monterey CA U S Army TRADOC Analysis Center 155 THIS PAGE INTENTIONALLY LEFT BLANK 156 INITIAL DISTRIBUTION LIST Defense Technical Information Center Ft Belvoir Virginia Dudley Knox Library Naval Postgraduate School Monterey California TRAC Monterey Naval Postgraduate School Monterey California Dr Ji Hyun Yang Naval Postgraduate School Monterey California MAJ James Henry Naval Postgraduate School Monterey California 157
38. 2 How many more training hours are needed 14 12 z v C 10 2 a 8 5 5 6 2 E 4 3 n 0 8 10 10 a Fresones 5 B a o 2 0 2 Figure 48 Session ee __ question two subject s self evaluation of additional training hours needed Session Three Question Three What were your likes and dislikes with your third exposure to DaViTo Likes The interface works well for navigation and selection Dislikes But some of the analysis through the one chart I know how to make was difficult If there is an icon that indicates the system is processing data I don t know where it is and so would continue to click on different icons with no response Dislikes Just don t remember enough of the tasks It is laid out in a familiar fashion But some of the later tasks I just couldn t remember how to do Dislikes Program responded sluggishly An easier way to access chart information would be beneficial Perhaps a chart display addition to the tool bar Count functions would also be useful i e count occurrences between this date and that date Dislikes The chart for IED attacks was difficult to read it was hard to tell which time frame corresponded to the year and it was hard to trace a data point to a number on the y axis There was no indication that the polygon was loaded and then there was a long delay if the polygon was selected before it was ready Likes Fun Di
39. 2 Plot of session one depicting time to answer versus accuracy of the answer for question C Note the decreasing time on the Y axis Top performance is in upper right Corner eeeseseeeeeseresesreesreserserssresseseresressesee 113 Plot of session two depicting time to answer versus accuracy of the answer for question D Note the decreasing time on the Y axis Top performance is in upper right COMERS 2542252 4cycscndadescosagseaeeeni de scaveqantedezacoeese eae 114 Plot of session two depicting time to answer versus accuracy of the answer for question E Note the decreasing time on the Y axis Top performance is in upper right corner 114 Plot of session three depicting time to answer versus accuracy of the answer for question F Note the decreasing time on the Y axis Top performance is in upper right corner 115 Plot of session three depicting time to answer versus accuracy of the answer for question G Note the decreasing time on the Y axis Top performance is in upper right corner p 116 Session one question one with responses on a scale of one very poor to 0 880 0 Mataeder A mec Mamet eacaiess 117 Session one question two subject s self evaluation of additional training INOUPS NEEM E EEEE 117 Session two question one with responses on a scale of one very poor to five Very sood Jeer luau aE E uate egos en hig emetic 121 xi Figure 47 Session two question two subject s self evaluation o
40. 319 81 67 02 70 Task 5 8 87 17 34 2 30 66 99 15 12 91 Task 6 6 77 31 18 8 68 96 76 21 27 96 Task 7 8 62 28 34 9 02 140 33 34 70 100 Task 8 11 75 33 77 9 23 149 94 35 50 52 Task 9 90 93 90 61 24 18 245 10 46 46 87 Task 10 63 36 141 47 7 17 355 25 76 17 4 Task 11 16 63 16 72 4 45 281 58 57 54 83 Table 10 Summary of time duration statistics for all session three participants v a N e 2 k 3 a Task Duration Sec Tasks Final Session Subjects Average E Bench Mark Figure 5 Graph comparing average participant task durations to the bench mark 32 Task Duration for Complete and IncompleteTasks 250 B 200 A 150 a A 5 Bench Mark 5 100 A A Completed Tasks a E A Incompleted Task 50 m i B a E 5 Fee F_ 7 0 1 2 3 4 5 6 7 8 9 10 11 Tasks for Final Session Figure 6 Graph comparing the bench mark to average participant time duration for both complete and incomplete session three tasks 3 Analysis of Cumulative Gaze Frequencies of Final Session Tasks Along with time duration and number of mouse click to complete each task the EyeWorks Analyze software suite has the capability to generate heat maps and associated graphs The heat maps were useful to display the gaze concentration of each participant or cumulatively for each task The gaze observation bar graphs were then used to compare gaze concentration in the
41. 5 SLMF 45 83 33 00 SAF 79 17 22 70 Table 46 Statistics for session one question B for all graph types Session 1 Question C Time vs Accuracy Accuracy correct 0 10 20 30 40 50 60 80 15 20 25 SAF CCF 30 m ser E S8F T35 X lt SLMF 3 40 __A LCF ee SLMF E 45 50 K SAF 55 I ccF 65 Figure 39 Plot of session one depicting time to answer versus accuracy of the answer for question C Note the decreasing time on the Y axis Top performance is in upper right corner Question Accuracy D Correct SAF 83 33 SLMF 87 50 LCF 25 00 SBF 87 50 CCF 79 17 Table 47 Statistics for session two question D for all graph types 113 Session 2 Question D Time vs Accuracy Accuracy correct 0 10 20 30 40 SO 60 70 80 90 100 15 SBF 20 CCF 25 X SLMF 30 X E SBF f 35 SAE ALCF w 40 ed lt SLMF 45 CEF es K SAF 55 60 ale 65 Figure 40 Plot of session two depicting time to answer versus accuracy of the answer for question D Note the decreasing time on the Y axis Top performance is in upper right corner Question Accuracy E Correct SAF 66 67 SLMF 58 33 Table 48 Statistics for session two question E for all graph types Session 2 Question E Time vs Accuracy Accuracy correct o 10 20 30 40 50 60 70 80 90 100 CCF 20 CCF gt 25 ICF E SBF 8 CF a AL 2 30 F 35 X SLMF SOME SBF Wi
42. 6 Simplicity 7 Map Overlays 6 Data Sets 6 Navigation Function 4 Table 20 Summary statistics of the top five likes and dislikes for session one common question three Top 5 Liked Features He af Top 5 Disliked Features ae P Geo Data Representation 6 Mouse Interaction 7 Ease of Use 5 Response Time T Functionality 3 No Progress Status 3 Layout 2 Creating a Graph 2 Graph Functionality l Graph Functionality 2 Table 21 question three 54 Summary statistics of the top five likes and dislikes for session two common Top 5 Liked Features rae a Top 5 Disliked Features ee Ease of Use 4 Response Time 14 Geo Data Representation 3 No Progress Status 9 Navigation Functionality 2 Graph Functionality 9 Layout 2 Map Labels 3 Simplicity 1 Mouse Interaction 3 Table 22 Summary statistics of the top five likes and dislikes for session three common question three 2 Analysis of the Final Survey The final survey was written utilizing the five usability attributes described in Table 1 from the Handbook of Usability Testing Rubin amp Chisnell 2008 Topics of improvements and visualization were also used to formulate questions All questions analysis and responses are contained in Appendix D 1 Usefulness The attribute of usefulness refers to the ability of a user to achieve their goals and their willingness to use the product If this attribute is not considered in the
43. APPENDIX D ADDITIONAL SURVEY DATA A THE THREE COMMON QUESTIONS Appendix D part A presents additional results from the three questions common to each session The results shown in Figures 44 through 48 are for each individual question Note that the answers to the free form questions in Tables 51 through 53 are not altered in content from what the subjects entered in the survey Session 1 Question 1 How would you rate your first training session a 14 2 12 5 10 N 5 8 2 6 5 4 o 1 2 3 4 5 E Response 0 o 0 9 15 Figure 44 Session one question one with responses on a scale of one very poor to five very good Session 1 Question 2 How many more training hours are needed 3 16 3 14 2 2 12 a 10 2 6 3 4 2 m 0 E _ E 0 1 2 4 5 7 810 1 E Response 4 16 2 2 ie Figure 45 Session one question two subject s self evaluation of additional training hours needed 117 Session One Question Three What were your top three likes and dislikes with your first exposure to DaViTo Likes Visual display of data possible for walk through with a boss it seems like you could create and share databases with others to share the same view of a situation shapes displays colors are customizable Dislikes It might be chugging through a lot of data when you move the map each time
44. C 2 recommends that it is distributed with DaViTo Training should be developed with the DaViTo software objective in mind that a user whom is unfamiliar with statistical methods and Geographic Information Systems could use the software This will reinforce the other software goals of users that can produce useful visualizations that can be grasped and understood by ground commanders Evangelista Henry Buttrey amp Whitaker 2012 R dati Designator OTE Attribute and Basis for Recommendation Description Learnability 75 percent of subjects reported that most people could figure out how to use DaViTo with the tools provided however 58 percent believe more training products are needed Develop a tutorial C 14 based training product for DaViTo Learnability 88 percent of subjects are either neutral or believe that more training is necessary to effectively be able to use DaViTo Distribute a training er product with DaViTo Table 37 List of recommendations and the basis derived from analysis of the learnability portion of the survey results 6 Recommendations Based on Satisfaction Questions The satisfaction segment of the final survey generated only one unique recommendation found in Table 38 Subjects desired that a help capability be designed into the DaViTo interface Currently there is little help functionality only a link to the OpenMap developer s page and no user s manual is acc
45. D for all graph types 113 Statistics for session two question E for all graph types 114 Statistics for session three question F for all graph types 115 Statistics for session three question G for all graph types 115 Table of likes and dislikes for all 24 subjects in session one Comments taken directly from the EyeWorks software with no modification 120 Table of likes and dislikes for all 24 subjects in session two Comments taken directly from the EyeWorks software with no modification 123 Table of likes and dislikes for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification 126 Table of free form responses to question Effect_3 for all 24 subjects in session three Comments taken directly from the EyeWorks software with TNO FOCI ANTONY 8 ica reuse cas r 127 Table of free form responses to question Eff_4 for all 24 subjects in session three Comments taken directly from the EyeWorks software with TO Modification i vans E i E Te agstiva EERE E R E ER A 128 xiv Table 56 Table 57 Table 58 Table 59 Table 60 Table 61 Table 62 Table 63 Table 64 Table 65 Table of free form responses to question Use_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software with DO
46. Design Our Institutional Review Board IRB submitted time constraint was three one hour sessions The goal was to keep both tutorial sessions less than 20 minutes each to allow ample time for task accomplishment data representation questions and questionnaire completion They were also designed with ample breaks and pause points so subjects would not become lost in the tutorials 2 Questionnaire Development The final portion of each session was a series of questions drawn from the extensive list of questions submitted with the IRB package in Appendix F The questionnaire portion of each session began with the same three questions These questions asked the subject to rate the training for the first two sessions how much more training they think is needed to use DaViTo in the field and to list their general likes and dislikes The final survey then had a much more extensive questionnaire consisting of the first three questions and then approximately 40 more questions derived from the usability factors matrix in Table 1 The questions were a mixture of multiple choice and free entry These questions were designed to identify deficiencies with respect to the usability attributes as well as elicit recommendations from the participant that would be directly targeted to each attribute 15 3 Training Session One The first session was limited to software interface introduction The video tutorial was exactly 18 minutes long It began by int
47. Graph comparing the bench mark to average participant time duration for both complete and incomplete session three tasks 33 Final session task one heat map and mouse clicks Task one directed subjects to open the WITS csv data file which required interaction with the Data Sets MERION de Auto einne sat ia 35 Graph of final session task one cumulative counts of gaze observations for all subjects Counts correlate to the number of subject fixations in each region shown in Figure 7 sesssssssssessesssessseeessseessresseesseessseessseessesseeesee 36 Final session task four heat map and mouse clicks Task four directed subjects to create a user defined polygon and display the chart 37 Final session task ten heat map and mouse clicks Task ten directed subjects to answer a question that required the creation of a user defined polygon and to display the chart 38 Example line chart format LCF with markers from the DaViTo software 40 Example cumulative stacked bar format SBF from the DaViTo software 41 Plot of session one time to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in upper right corner 45 Plot of session two times to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in the upper right Eoree 47 Plot of session three times t
48. Integrated data analysis Please explain Simple and intuitive interface What would you Area relief or 3D capability like Google Earth change about the color scheme page Improve_6 Color scheme is bland add contrast layout Please f explain Change default data point color How would you Right click on map shows all available options improve the mouse interaction with the Improve_7 Quicker response to right clicks software Please ey ah oe explain Highlighting or positive indication of selection How would you like Improve_8 Multiple see Appendix D explain Options Frequency What graph did you SBF 14 think conveyed the I 9 CCF 2 information in SLME 6 general the best LCF 0 SAF 2 Table of improvement questions and associated statistics 68 3 Survey Results Summary Multiple usability deficiencies were identified by the output data from both the three common questions for all sessions as well as the extensive session three survey Recommendations for improvement will be made in Chapter V The common questions were more general than the session three survey The first two questions did not identify deficiencies rather were related to the training received and subject opinions of additional training necessary The responses will serve as a basis for improvement recommendations The final question identified general likes and dislikes The dislikes correlate to more specific com
49. K SAF 40 xX 45 Figure 41 Plot of session two depicting time to answer versus accuracy of the answer for question E Note the decreasing time on the Y axis Top performance is in upper right corner 114 Question F Accuracy Correct LCF 12 50 SLMF 33 33 SAF 50 00 SBF 75 00 CCF 50 00 Table 49 Statistics for session three question F for all graph types Session 3 Question F Time vs Accuracy Accuracy correct 0 10 20 30 40 50 60 70 80 90 100 30 40 CCF 50 K SAF SBF E E SBF 60 o x ae ALCF z a SLMF lt SLMF F 90 K SAF 100 110 A LCF 120 Figure 42 Plot of session three depicting time to answer versus accuracy of the answer for question F Note the decreasing time on the Y axis Top performance is in upper right corner Question Accuracy G Correct LCF 87 50 SLMF 62 50 SAF 70 83 SBF 75 00 CCF 100 00 Table 50 Statistics for session three question G for all graph types 115 Session 3 Question G Time vs Accuracy Accuracy correct 0 10 20 30 40 50 60 70 80 90 100 20 25 F CCF 30 35 E SBF g 40 X SAF 45 ECF SUM s0 SLMF Firo x KSAH 60 65 Jo A LCF Figure 43 Plot of session three depicting time to answer versus accuracy of the answer for question G Note the decreasing time on the Y axis Top performance is in upper right corner 116
50. L basic user and DaViTo An analyst with advanced skills would experienced analyst benefit from tailored training that goes beyond the capabilities in the user interface Extend mapping Usefulness Many Navy subjects identified D 3 capability to include additional uses such as counter piracy or sea nautical charts based counter drug operations Develop a standardized Army or Usability Without a formal database and data D 4 service wide database formats the tool itself may suffer reduced and data formats for usability use with DaViTo Table 41 List of author recommendations and the basis derived from the observations of subjects and interaction with DaViTo 85 THIS PAGE INTENTIONALLY LEFT BLANK 86 VI CONCLUSION AND FUTURE WORK A SUMMARY The Data Visualization Tool developed by TRAC Monterey is a powerful piece of software that has the potential to be instrumental in future military operations If properly developed through continuous revision and improvement it will see broader employment The tool itself has wide ranging application throughout the Army and potentially the entire military This thesis was limited in scope to the usability of DaViTo although there are other opportunities for research A formal usability study was not conducted during the development of DaViTo and as such this thesis was done in lieu of that following the initial release The primary objective was to provide actionable recommenda
51. MOGRAPHIC SURVEY ID RANDOM 4 DIGITS Please enter your age in years Please indicate your military service branch Check One o ARMY o NAVY n MARINE CORPS OA FORCE RETIRED MILITARY D INTERNATIONAL MILITARY MEMBER Is English your primary language Check One a ves DNO Please indicate your military grade e g O 2 O 3 0 4 ete __ Please indicate f you are affilisted with the development of the DaViTo software Check One o ves ono Are you now or have you ever been employed as an analyst Check One o YES o NO Have you ever used the DaViTo software before you participated n this study Circle One D YES ono Check the most descriptive unit of measurement that describes the amount of experience you have with computer software 0 None 0 Novice word processing emai web surfing n Intermediate user some programming in at least one python C java ruby jmp etc o Expert Computer Science background or can create complex programs How many hours per week do you spend on a computer Check One oO5heurs 6 20hours 011 15 hours 16 20hours 21 hours Do you have 20 20 vision ar correctable to 20 20 vision OYES oNO Are you color dling DYES ano Have you ever used ARC GIS or any other geo spatial mapping software Check One n YES ono What operating system do you prefer 0 Mee OWindows Other abk iY She EXPIRES SEP 30 2012 146 Quantitative amp Subjective Questions Exi
52. Map Controls portion of the interface 102 J Map Controls 1 213 0 137 4 t vi ll WO 3 T fj j pon L4 Tan eee ASG i ry Big 6 se X O bd List 20 2 20 0 P L i ve Progress Bar 1 al a E FMa 20 4 aa Figure 24 Final session task eight heat map and mouse clicks Task eight directed subjects to go back to the previous view 103 er _oe a 1 WEE a m m Figure 25 Final session task nine heat map and mouse clicks Task nine directed one to remove all data sets and Shapefiles then load the Indure_Demo data set and the afghanistan_provincial_boundaries Shapefile 104 Figure 26 Final session task eleven heat map and mouse clicks Task eleven asked subjects to answer a trend analysis question using the graph generated in the user defined polygon from task ten 105 Figure 27 Figure 28 Gaze Observations in Each Region 20 00 Dw 20 0 4 Regions Graph of final session task two cumulative counts of gaze observations for all subjects Task two directed subjects to load the Kenya Roads Shapefile which required interaction with the Shapefiles region Counts correlate to the number of subject fixations in each region shown in Figure 19 Gaze Observations in Each Region 200 00 5009 4 100 004 a0
53. RMING ORGANIZATION Naval Postgraduate School REPORT NUMBER Monterey CA 93943 5000 9 SPONSORING MONITORING AGENCY NAME S AND ADDRESS ES 10 SPONSORING MONITORING N A AGENCY REPORT NUMBER 11 SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U S Government IRB Protocol number NPS 2012 0037 IR EP7 A 12a DISTRIBUTION AVAILABILITY STATEMENT 12b DISTRIBUTION CODE Approved for public release distribution is unlimited A 13 ABSTRACT maximum 200 words The volume of data available to military decision makers is vast Leaders need tools to sort analyze and present information in an effective manner Software complexity is also increasing with user interfaces becoming more intricate and interactive The Data Visualization Tool DaViTo is an effort by TRAC Monterey to produce a tool for use by personnel with little statistical background to process and display this data To meet the program goals and make analytical capabilities more widely available the user interface and data representation techniques need refinement This usability test is a task oriented study using eye tracking data representation techniques and surveys to generate recommendations for software improvement Twenty four subjects participated in three sessions using DaViTo over a three week period The first two sessions consisted of training followed by
54. Review Board To Dr Ji Hyun Yang MOVES Institute APR 11 202 Mr Jesse Huston MOVES Institute MAJ James Henry TRAC Monterey LT Tyler Rasmussen USN LT Darren Cutler USN SUBJ USABILITY TESTING AND WORKFLOW ANALYSIS OF THE TRADOC DATA VISUALIZATION TOOL DAVITO Emi 1 Approved IRB Protocol 1 The NPS IRB is pleased to infor you that the NPS President has approved your project NPS IRBa NPS 2012 0037 IR EP7 A The approved IRB Protocol is found in enclosure 1 Completion of the CITT Research Ethics Training has been confirmed 2 This approval expires on 30 September 2012 If additional time is required to complete the research a continuing review report must be approved by the IRB and NPS President prior to the expiration of approval At expiration all research subject recruitment data collection analysis of duta containing PIL must cense 3 You are required to report to the IRB any unanticipated problems or serious adverse events to the NPS IRB within 24 hours of the occurrence 4 Any proposed changes in IRB approved research must be reviewed and approved by the NPS IRB amd NPS President prior to implementation except where necessary to climinate apparent immediate hazards to research participants and subjects 5 As the Principal Investigator it is your responsibility to ensure that the research and the actions of all project personnel involved in conducting this study will conform with the IRB approved protocol and IRB re
55. Task three directed subjects to go to the Africom view 99 Region Selection 1 20 1 10 0 A senator Pot clear Remove Sancti 3 Progress Bar 1 L i DaViToNextTask 1 wide iii i a eR Figure 21 Final session task five heat map and mouse clicks Task five directed subjects to close and delete the chart generated in task four 100 Map Controls 1 213 2 329 4 A 2 f Shape Files 1 12 0 10 7 ge Pred Shape File Controls 1 21 1 0 0 Regi on List Region Selection 1 70 9 20 0 Lasts mt tee team e Progress Bar 1 y L DaViToNextTask 1 nd Te TO iM Aeh Figure 22 Final session task six heat map and mouse clicks Task six directed subjects to hide the Map Controls ribbon bar 101 amas O Shape File Controls 1 oTempora 0 6 0 0 e Chartrm Shape Sad Darel iaei aala 4 _ Opaque Charts aa 2 3 0 Region Selection 1 20 20 Lsendion_ _piot clear Remove Selected ae E Progress Bar 1 70 0 E i DaViToNextTask 1 4 IA M j OO Secons 1453 i 3 EA d SS Figure 23 Final session task seven heat map and mouse clicks Task seven directed subjects to navigate to Iraq without the use of the
56. a 8 a A 8 8 2 gt ma ma 5 A E ye o H ge o 5 S28 y Spig seif o 2 B 5 5 g O a a gs Esg 5 8 er D 58 gt 8 OY as GH 65 o J o y a o JB a 2 Ce 2 R 4 Sos g D g 5 D o go 3 5 3 2 g 8 sa a as 8 58 57 om B 5 3 5 me So 8 N S O O v E E a z a E z a J o w E Ea R 5 ar wd f oe amp Q Q R S m 1T a g o 6 m x 7 Sg a lt i EL gz ee 2i lt n lt 42 T d A Description Example Stacked Bar m Event Cumulative stacked m Event C Format mEvent B bar chart 15 m Event A SBF 10 Clustered Columns for each j Columns data point are s A Format clustered side by CCF side Table 11 Table showing the following information for the five graph types constructed for the experiment Type and abbreviation description and an example of each Question Deea oa Session Question Text One What were the total events for December 2012 One What event had the highest frequency in Jun 2013 One What month had the most events Two What month had the least events Two Which month had the highest frequency for event D Three Compare the months of December 2012 and December 2013 What was the increase or decrease in total events G Three What event had the highest sum for the months of February 2012 and February 2013 Table 12 List of questions for each session and their designation letter
57. a e ES EE E rS 138 Worksheet used by experiment proctors for data recording Includes all tasks Tor session hee aTe A aE EE E E 140 XV THIS PAGE INTENTIONALLY LEFT BLANK xvi DaViTo TRAC TRADOC GIS IRB NPS LCF SBF SLMF SAF CCF LIST OF ACRONYMS AND ABBREVIATIONS Data Visualization Tool TRADOC Analysis Center Training and Doctrine Command Geographic Information System Institutional Review Board Naval Postgraduate School Line Chart Format Stacked Bar Format Stacked Line Marker Format Stacked Area Format Clustered Columns Format xvii THIS PAGE INTENTIONALLY LEFT BLANK xviii ACKNOWLEDGMENTS First we have to thank our families Our wives and children have been a source of support inspiration and encouragement as we put in the long hours to complete this endeavor They also made the downtime worthwhile as we tried to remove ourselves from the weight of the task at hand whenever we could Thank you to our support team Jesse Huston whose expertise and knowledge in the eye tracking lab got us off to a great start and through to the home stretch LTC Jon Alt USA and all of TRAC Monterey we thank you for your assistance developing the topic and your support throughout MAJ James Henry USA was instrumental from day one assisting us at becoming experts at DaViTo and with experiment development Even while serving in Afghanistan he continued to provide his guidance with timely thesis reviews thorou
58. ab 6 B Right or left click on hide tool panel Total 7 Navigate to Iraq without map control panel Option 1 7 A Right or left click control Panel 7 B Right or left click on pan 7 C Navigate to Iraq Total Option 2 7 A Right or left click views 7 B Right or left click on Iraq Total 8 Go back to previous view 8 A Right or left click control tab 8 B Right or left click display tool panel 8 C Left click go back previous projection icon Total 9 Remove all the Data sets and Shapefiles load the Indure_Demo data and afganistan_provincial_boundaries Shapefile 10 Now please answer the following question what were the total events for IED attacks for 2010 in the Northwestern most provincial boundary 11 Does it appear that IED attacks are on the rise from 2006 in the same provincial boundary Total Table 65 Worksheet used by experiment proctors for data recording Includes all tasks for session three 140 APPENDIX F APPROVED IRB DOCUMENTS Appendix F contains the IRB approval cover sheet and author generated IRB documents This includes the consent form call for participants demographic survey and question list for the final survey These are the approved documents from the IRB for use in this thesis Al Naval Postgraduate School Human Research Protection Program From President Naval Postgraduate School Via Chairman Institutional
59. able of free form responses to question Improve_7 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_8 How would you like to see the software employed Please explain If this software could be adopted by intel and ops analysis groups it would be awesome to share prepared data That said it would have to be unilaterally adopted within the joint construct or it would be a waste of time because data portability would suffer In TOCs to assist planning Resource utilization based upon event data Allows focused use of limited resources Data analysis for counter piracy operations Intel mission planning I could have used this to track my squadron s IED detection performance P 3C missions up and down Iraq s supply routes report hundreds of hot spots and some get investigated while some do not Those that are investigated may be IEDs but many more are false positives The trends are somewhat geographically related but the data isn t easy to visualize Better data tracking would be required to use DaViTo for this but if you had it you might get the commands to track data appropriately I can think of several uses It would be useful for anything from intelligence events to liberty incidents to logistical tracking Wide variety of possible uses Not sure I know this is mainly an Army software tool but if there could be incorporation of it among other branches of military
60. agram of the equipment configuration Descriptions of each item are in Table 2 Number Description i Diagram P 1 24 Inch LCD Screen Displayed DaViTo software tasks and surveys that were created in EyeWorks 2 Secondary 24 Inch LCD Screen Display for the experiment controller to monitor Eye Works tracking data 3 Gateway Laptop Displayed the video tutorials This was a standalone device and was removed prior to the final session 4 Hewlett Packard Laptop Controlled the faceLAB software This was a standalone device facing the experiment controller 5 Infrared and Stereo Camera Pod Precision located to track the subject s eyes for best results while not obscuring the subjects view 6 Keyboard Input device to respond to survey questions 7 Mouse Input device to allow the subject to interact with the DaViTo and EyeWorks software via the HP desktop Table 2 Description of items in equipment configuration illustrated in Figure 1 14 C DESIGN OF THE EXPERIMENT 1 Training Tutorial Development In order to standardize training for the DaViTo software and stay within time constraints the training materials were designed as a recorded tutorial Microsoft PowerPoint 2010 version 14 0 6123 5001 was used to generate slides showing the subjects what to do The slides were then narrated and recorded using Camtasia Studio version 7 The testing script was developed using EyeWorks
61. aining for the initial audience would 83 increase usefulness for both parties To get the tool in use beyond the army incorporation of mapping features that benefit the other service branches would be helpful Finally recommendation D 4 addresses the development of a database for use with DaViTo as well as standardized data formats The output of DaViTo is only as good as the information entered into it It was stated in Data Visualization Tool that there are vast amounts of data that are not being properly analyzed and the subsequent results getting to decision makers Evangelista Henry Buttrey amp Whitaker 2012 Through conversations with TRAC Monterey there does not currently exist such a database This should be developed concurrently with DaViTo to maximize its potential and ensure the tool is able to be utilized to its fullest potential 84 Recommendation Designator Basis for Recommendation Description Add a collapse capability to the Data Visualization This will expose more of the Map D1 Sets Predefined and Data Display area reducing clutter and Polygons and Regions Selection List areas of the interface allowing the user more capability to work within the primary workspace of the interface Develop training products for both a Usefulness Recommendations C 1 and C 2 address the development of training products There are complex statistical capabilities in
62. annot be idemified All printed surveys will be stored in a locked drawer in a secured facility when not in use In the final publication no names will be wed for privacy reasons and because it adds po vale to the research All references to data collected will be made anonymous Your name will be encoded as a participant number Only principle investigators will have access to this key that translates an identification mamber to your name Points of Contact If you have amy questions or comments about the research or you experience an injury or have questions about any discomforts that you experience while taking part in this study please contact the Principal Investigator Dr Ji Hyun 831 656 3004 jvanl ngsedy Questions about your rights as a research subject or any other concerns may be addressed to the Navy Postgraduste School IRB Chair CAPT John Schmidt 831 656 3364 eschonid inps edy Statement of Consent have read the information provided above I have been given the opportunity to ask questions asd all the questions have been answered to my satisfaction have been provided a copy of this form for my records and agree to participate in this stedy 1 understand that by agreeing to participate in this research and signing this form do not waive any of my legal rights Participant s Signature Date Researcher s Signature Date HPS IRB APPROVED Updated om 4 10 12 APR 11 2012 sth 4 bon 145 iL 12 13 DE
63. ar in its representation Possible hovering mouse shows nearest data point value More options for graphing than are shown in the questions assuming that s all there is The axis labels are confusing and some of the cumulative line graphs are misleading because of their alignment I addressed this a bit already Make the manipulation of the output more intuitive It should be able to be manipulated by right clicking or with first level menu options Perhaps a GUI where the user could ask specific questions like posed in this experiment and the software displays the correct answer in addition to the graph This could be done from pull down menu selections Many current financial websites do this for the customer 137 Q Visual_2 What more would you like to see in the output of this software Basic examples on how to read the data would be beneficial A way to hover over the graphs and get data points amounts would help Or be able to click on the graph and get the actual numbers instead of generalizations done with the graphs For big graphs it is hard to see where the values line up on the y axis If the axis could be slide to the right so it is next to the data that one is interested in it would make interpreting the data easier Also it would be nice to zoom But all of this will be pointless if the program keeps running so slow Instead of redrawing all the lines frame cycle like I know this t
64. at 101 Final session task seven heat map and mouse clicks Task seven directed subjects to navigate to Iraq without the use of the Map Controls portion of the interfeeee luelnyctueuawnbaaneuiaeantansanaaetiaeaaiiabrestuunayae 102 Final session task eight heat map and mouse clicks Task eight directed subjects to go back to the previous View 103 Final session task nine heat map and mouse clicks Task nine directed subjects to remove all data sets and Shapefiles then load the Indure_Demo data set and the afghanistan_provincial_boundaries Shapefile 104 Final session task eleven heat map and mouse clicks Task eleven asked subjects to answer a trend analysis question using the graph generated in the user defined polygon from task ten 105 Graph of final session task two cumulative counts of gaze observations for all subjects Task two directed subjects to load the Kenya Roads Shapefile which required interaction with the Shapefiles region Counts correlate to the number of subject fixations in each region shown in Figure Graph of final session task three cumulative counts of gaze observations for all subjects Task three directed subjects to go to the Africom view Counts correlate to the number of subject fixations in each region shown in PRS UTS 20 28056 alate yee 106 Graph of final session task four cumulative counts of gaze obser
65. at required the creation of a user defined polygon and to display the chart Counts correlate to the number of subject fixations in each region shown in Figure 10 Gaze Observations in Each Region 0 00 bal Regions Graph of final session task eleven cumulative counts of gaze observations for all subjects Task eleven asked subjects to answer a trend analysis question using the graph generated in the user defined polygon from task ten Counts correlate to the number of subject fixations in each region shown in Figure 26 110 APPENDIX C ADDITIONAL GRAPH RESULTS Appendix C contains additional results from the analysis of the data representation portion of the experiment Table 43 presents the data representation questions asked about each graph type The statistical analyses of the questions in Table 43 are presented in Tables 44 through 50 and Figures 37 through 43 aa Session Question Text A One What were the total events for December 2012 B One What event had the highest frequency in Jun 2013 C One What month had the most events D Two What month had the least events E Two Which month had the highest frequency for event D F Three Compare the months of December 2012 and December 2013 What was the increase or decrease in total events G Three What event had the highest sum for the months of February 2012 and February 2013 Table 43 List of questions for eac
66. ations and the basis derived from analysis of the efficiency portion of the survey results 4 4 5 26 a aA 78 List of recommendations and the basis derived from analysis of the learnability portion of the survey results 79 List of recommendations and the basis derived from analysis of the satisfaction portion of the survey results 80 List of recommendations and the basis derived from analysis of the visualization portion of the survey results 81 List of recommendations and the basis derived from analysis of the improvement portion of the survey results 83 List of author recommendations and the basis derived from the observations of subjects and interaction with DaViT0 0 eee eeeeeeseeeeeeeees 85 Consolidated list of recommendations for the improvement of DaViTo and their basis derived from all research methods 95 List of questions for each session and their designation letter This is a repeat of Table 12 in Chapter IV Section B shown here to correlate data TORUS SLOTS an e n e a a ran ee a dante Pom eet 111 Statistics for session one question A for all graph types 111 Statistics for session one question B for all graph types 112 Statistics for session one question B for all graph types 113 Statistics for session two question
67. ations were developed by the thesis authors and are a result of the hours spent designing and carrying out the experiment sessions and the accompanying research They consist of four final big picture recommendations to assist developers with the next revision of DaViTo In the end 37 recommendations have been formulated for dissemination to TRAC Monterey Several of these are repeats however they have a different usability attribute and basis for recommendation The full list of recommendations can be found in Appendix A Ideally the fruits of this thesis will make it into future revisions of DaViTo and succeed in making it a premier software package useful throughout the military and beyond 88 B FUTURE WORK Prospective topics for additional research derived from this thesis are abundant Usability testing and eye tracking both have far reaching capabilities in the future The first is a follow up usability test of the next revision of the DaViTo software Verification of improvements in all attributes of usability would be useful to verify the recommendations resulting from this thesis were indeed applied and beneficial Further research is also necessary in the field of data visualization This thesis was very limited in scope with regards to the techniques tested As data visualization methods advance there is a need for research in this field to continue evolving DaViTo Additionally the software could be further improved and refined t
68. basic reinforcement tasks evaluation of graphical methods and a brief survey The final session was a task oriented session followed by graphical representations evaluation and an extensive survey Results from the three sessions were analyzed and 37 recommendations generated for the improvement of DaViTo Improving software latency providing more graphing options and tools and inclusion of an effective training product are examples of important recommendations that would greatly improve usability 14 SUBJECT TERMS Eye tracking usability data visualization and workflow 15 NUMBER OF PAGES 179 16 PRICE CODE 17 SECURITY 18 SECURITY 19 SECURITY 20 LIMITATION OF CLASSIFICATION OF CLASSIFICATION OF THIS CLASSIFICATION OF ABSTRACT REPORT PAGE ABSTRACT Unclassified Unclassified Unclassified UU NSN 7540 01 280 5500 Standard Form 298 Rev 2 89 Prescribed by ANSI Std 239 18 THIS PAGE INTENTIONALLY LEFT BLANK ii Approved for public release distribution is unlimited USABILITY TESTING AND WORKFLOW ANALYSIS OF THE TRADOC DATA VISUALIZATION TOOL Darren W Cutler Lieutenant United States Navy B S North Carolina State University 2006 Tyler J Rasmussen Lieutenant United States Navy B A B S Georgia State University 2006 Submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN MODELING VIRTUAL ENVIRONMENTS AND SIMULATION MOVES from the NAVAL POSTGRADUATE SCHOOL Septem
69. ber 2012 Author Darren Cutler Tyler Rasmussen Approved by Ji Hyun Yang Thesis Advisor James Henry Second Reader Christian Darken Chair MOVES Academic Committee Peter J Denning Chair Department of Computer Science iii THIS PAGE INTENTIONALLY LEFT BLANK iv ABSTRACT The volume of data available to military decision makers is vast Leaders need tools to sort analyze and present information in an effective manner Software complexity is also increasing with user interfaces becoming more intricate and interactive The Data Visualization Tool DaViTo is an effort by TRAC Monterey to produce a tool for use by personnel with little statistical background to process and display this data To meet the program goals and make analytical capabilities more widely available the user interface and data representation techniques need refinement This usability test is a task oriented study using eye tracking data representation techniques and surveys to generate recommendations for software improvement Twenty four subjects participated in three sessions using DaViTo over a three week period The first two sessions consisted of training followed by basic reinforcement tasks evaluation of graphical methods and a brief survey The final session was a task oriented session followed by graphical representations evaluation and an extensive survey Results from the three sessions were analyzed and 37 recommendations generated for the improv
70. cciusicedevssdssadccssisecsoedessavdesnsters sdesclocstaibesbadesandessstcnveccaadtessntdvasdasscns 19 E EXPERIMENT PROCEDURE 22 F DATA COLLECTION AND ANALYSIS STRATEGY u csccssscssssoseee 24 1 Eyes Tracking Dafa 24 2 ERREN 0 ITPA E E A E TE 24 3 Questionnaires 25 RESULTS AND DATA ANALYSIS 27 A DATA PREPARATION 27 B DAVITO TASK PERFORMANCE AND INTERACTION cccceeeee 27 1 Analysis of Mouse Clicks Correctness and Completeness of Final Session Tasks 28 2 Analysis of Time Duration Correctness and Completeness of Final Session Tas 30 3 Analysis of Cumulative Gaze Frequencies of Final Session Tas sdesseavdsssdinnddidgendeccesecas 33 4 Task Results Summary 39 C GRAPH PERFORMANCE RESULTS 39 1 Analysis of Session One Graph Data 44 2 Analysis of Session Two Graph Data 45 vii 3 Analysis of Session Three Graph Data 47 4 Analysis of Overall Graph Data 48 5 Graph Results Summary 50 D SURVEY RESULTS sidsiessssticecessscds stussvechessseacssbauisacascaviscuyisbushoussvansevinsdavessints 51 1 Analysis of the Three Common Questions from Each Session 52 2 Analysis of the Final Survey 55 L USOPULIEOSS ci seasicassicascncsspsisassadinascaanstedosa ionas ausbanesansvdnnverSiaiosanss 55 2 Effectiveness oseni nenaad ia ie aa aes 57 3 EA A R ETE 59 4 Learnabilit iiss ies individ Wi diaaiasstenssteasiWaeen saosin 61 5 SAUS FACTION
71. cle distance as you measure 2 Zoom box function was refined it worked but didn t allow me to drag down to the lower right like I intended 3 Appears to have limited level functionality compared to something like CPOF where you could analyze in 3D Likes Easy to understand icons intuitive graphical representation mouse over tooltip 2 Clean interface no clutter 3 Controls are similar to normal software e g pan using click and drag zoom using selection box or mouse wheel Dislikes 1 Response Lag especially when click and drag to pan when data sets are loaded 2 Text is a bit small necessary tradeoff to maximize map display area 3 Info bar where it shows the Lat Long Distance at bottom left is small and tucked away Hard to use with the distance measurement function 119 Session One Question Three What were your top three likes and dislikes with your first exposure to DaViTo Dislike Information could be displayed in conjunction with a map built in Did not like the system being jittery when navigating on the map Could lose the location you are looking for when moving around do to the slow response from mouse input Needs to be smoother Slow No country labels Distance ring to thin Distance tools doesn t allow you to make separate waypoints Likes Easy to use Likes 1 Multiple map projections available 2 Clearly defined data set types along with data set previews 3 Ease of data set
72. d learning more of the capabilities of DaViTo Dislikes The right left mouse use when creating and exploring charts Likes Liked the way you could get to a graph so quickly based on just drawing a Shapefile on the chart Very convenient Disliked The way you had to repeatedly right click on the smaller graphs to bring up the options menu Initially all that is available is the create option first four to five clicks Likes Easy to use 2 Shape tools to select particular regions were intuitive Chart output was easy to access and read Dislikes 1 No option for irregular shapes 2 Data analysis tools not explained in depth Likes Seems like an easy way of embedding data into a map to make it more meaningful Dislikes The lag between mouse clicks and the menu appearing is frustrating but not as frustrating as the Right click menus not acting the same each time Sometimes I would get a short list of options other times a longer list of options This should be more consistent Likes I do like the capability of showing terrain on the graph and also the function of showing data per data dot Dislike Also loading time is still slow I think that exposure for more than 15 minutes is required to gain a full grasp Dislikes It was quite a bit harder for me to navigate through the charts but with repetition I should be fine So I disliked having to play seek and find to figure out what I needed I really don t have any
73. designated regions of the user interface similar to those represented in Figure 2 Task descriptions for session three are in Appendix E Heat maps and gaze concentration graphs for all eleven tasks conducted in session three are in Appendix B Note that on the heat map color increasing from blue to yellow and then to red indicates a greater number of fixations Also there are two numbers displayed in each region on the heat map The top number correlates to the percentage of total time the participants looked in that region The bottom number is the percentage of total clicks that were made by subjects in that region 33 Following analysis of the heat maps and gaze observation graphs for each task three of the maps were found to be of specific interest In addition to reinforcing the conclusions reached from the mouse click and time duration data with regards to tasks four and ten an additional deficiency was found in task one Task one was a simple task that had been completed in sessions one and two with the tutorial The user was instructed to open the file WITS csv in order to populate the Map and Data Display Area with a dataset Figures 7 and 8 show the gaze data for task one This required the user to open the file using the Data Set region of the interface Note in Figure 8 that while this was the predominant region where gazes were observed a significant number were also recorded in the Map area as well as the Map Controls area Fi
74. down the menus to confirm I understand the task Like Step by step instructions Dislike Need to memorize each step of task the white screen and no able to go back to reread the instruction Dislikes Still a little difficult to remember how to load data set May have been to the brief exposure Likes Ease of general statistical analysis access to overlays ability to create new graphs on the fly Dislikes Response times to clicks load times for data sets unclear chart option explanations in control panel too many steps to draw a shape Like The ability to create the chart in order to pull data Dislike Remembering the steps to create the chart Dislikes The maps take a while to upload seems okay for this environment but it may be too restrictive in a real work environment that requires more changing uploading Dislikes Maps and charts loaded a little slow but other than that no issues Likes Good graphical user interface and the pop up menus were easy to follow Graph displays were good No dislikes with my second exposure Likes Visual interface Dislike Area selection method Dislikes Deeper you go into the functionality the more you have to remember Tough to do when you get older and your short term memory begins to slip Table 52 Table of likes and dislikes for all 24 subjects in session two Comments taken directly from the EyeWorks software with no modification 123 Session 3 Question
75. e geographical areas or load predefined geographic polygons Once geographic areas of interest have been defined DaViTo provides a suite of data exploration methods that can be applied to the data that resides within these areas Since DaViTo is government owned software built solely with open source software the potential of this software is limited only by the skills and expertise of the user and developer community Evangelista Henry Buttrey amp Whitaker 2012 DaViTo was developed with four goals in mind for the software One was that it be capable of producing useful visualizations and analysis of data with a spatial temporal component This is an essential part of the program as it was desired to convert the vast amounts of data into something that can be interpreted This leads into a second goal representing the data with statistical and graphical output that can be grasped and understood by ground commanders The importance of this was to allow them to take the data input to the tool transform that data into a visual representation that shows the points of significance and present it to decision makers in a logical and comprehensible manner A third goal seen as very important to TRAC Monterey was to distribute the software as open source created from open source programs The point in this goal was to give users the power to modify the tool to meet their specific needs providing an instrument that can be continually adapted b
76. e time and stability software crashed Effectiveness When drawing polygons the lag A2 Implement a progress time led subjects to continue clicking outside the status bar polygon because there was no indication of progress D d lat Effectiveness Right clicking outside of the A 3 polygon when drawn gts Gee polygon was counter intuitive to users by the user Implement file Effectiveness Heat mapping shows that many A 4 operations in the main users looked to the menu ribbon for file menu ribbon operations Make the Stacked Bar Efficiency The SBF graph was the most B 1 Format SBF the conducive to generating a correct answer from default graph type experiment subjects B2 Add more options for Usefulness Different graphs have better graph types performance relative to the question asked 91 Recommendation Designator Attribute and Basis for Recommendation Description Develop a tutorial Learnability Currently there are no training C1 based training product products for DaViTo beyond a basic PowerPoint for DaViTo slide show C2 Distribute a training Learnability There is only a PowerPoint slide i product with DaViTo show distributed with DaViTo currently Link th 4 l Usefulness Currently the user s manual is not C 3 to the Help tab on the accessible via the DaViTo interface ribbon bar Includ h tools t ms i graph 100819 Usefulness G
77. e 31 List of recommendations and the basis derived from analysis of the task results 12 B RECOMMENDATIONS BASED ON GRAPH RESULTS Three of the four development goals for DaViTo discussed in Chapter II involve graphical representation of data The production of useful visualization and data analysis and representing data with graphical output that can be interpreted by ground commanders are the root of two of these Evangelista Henry Buttrey amp Whitaker 2012 The usability of DaViTo is significantly deficient in this regard In addition to the two recommendations in Table 32 there are more concerning the functionality of graphs in the next section generated from the survey data It was identified that the default graphing option Line Chart Format LCF was the lowest scoring option overall in the three sessions Fortunately the highest performer Stacked Bar Format SBF is the one and only other option directly on the DaViTo interface However this requires more from the user such as knowing there is another option and realizing that this option may better convey the data Implementing B 1 will make the SBF the default graph type This will have an immediate impact on the efficiency of the software by giving the user the best chance at successfully interpreting the graph This will also improve the usefulness effectiveness and satisfaction attributes of DaViTo The second recommendation B 2 will add more options for di
78. e to try another way method to accomplish a task I also didn t like that when I was to load a shape of some sort I needed to look for the polygon That didn t make sense to me and it took me a bit with some prompting to get where I needed to go As for likes Likes I seemed to be able to navigate better since this was my third session But there still is a learning curve Dislikes SLOW I spend too much time swimming Did they write this thing in visual basic Learn to code code monkeys Dislikes User feedback indicating loading in progress a progress bar would improve user interface When loading a new data set or Shapefile the system does not pan to that location automatically When using the right click in the map option the system displays different menu option some more complete that others but it is not clear to the user where to select in the map to get the right options Dislikes 1 It takes too long to load the data sets 2 If the system is working there needs to be a message that states it is doing so Still easy to fumble through navigate Charts Awful Likes Once loaded with data the tool provides very detailed charts for several types of data Although small the charts were easy to read based on color coding Dislikes Time to process the data was a little confusing as to whether I had messed up or the processing was taking long computer froze Country labels would have been helpful with the boundaries
79. e was to see how a user would interpret the graphs with no training on their format Again this was not an effort to evaluate participants from a training transfer perspective but rather an attempt to determine which graphs best represented the type of data being used in DaViTo It is recognized that a great deal of research has been done on data representation methods and there are novel concepts in literature that could be applied to future applications however this was an effort to compare the currently used methods with some others that are easily achievable with the current software Complete statistical analysis of all graph data is contained in Appendix C 39 JFreeChart and R have the capability to present data in a number of different graphical formats These different types are accessible within DaViTo by an advanced user however the novice user has two readily accessible graph style options The default option is the line chart format LCF shown in Figure 11 The LCF has options for line chart shapes or markers on default or off as well as the ability to manipulate the line thickness The other option is to use the cumulative stacked bar format SBF shown in Figure 12 These options are selectable in the display region shown in Figure 2 of the DaViTo interface Additionally data can be exported from DaViTo for analysis with other statistical tools such as Excel or JMP Region Afgan 13 2004 2005 2006 2007 2008 2009 2010
80. ement of DaViTo Improving software latency providing more graphing options and tools and inclusion of an effective training product are examples of important recommendations that would greatly improve usability THIS PAGE INTENTIONALLY LEFT BLANK vi Il HI IV TABLE OF CONTENTS TNDRODU CTTON PETA E E A E EE T T T 1 A JADI rA d OSI EAEE EE E A TE E EO E EEEE 1 B OBIECIVEG aerisit aria ia i Eei oe 1 C SCOPE AND LIMITATIONS 2 D THESIS ORGANIZATION sseesoescesseesoesocsseesoesocsscesoesoesocsscesoesocsscesoesoesseesoee 3 BACKGROUND ossceeriesoniionsireesctostie evesai seuss Sostadecansteselucstupecussendeancbawencteassbatseseubecueas 5 A WHATIS DAVITO carie chats tnss tie sssaseee 5 B WHAT IS USABILITY nuset ennie naisen tous eset Eles ae 7 C PURPOSE OF USABILITY TESTING 7 METHODOLOGY seiseisssieisecrens acstiets on 11 A OVERVIEW AND OBIECTIVES 11 B EQUIPMEN Foree oiaro datos saoilear 11 1 APO WAPe i cvetensacverecvsnsduasisvceedaesied senses decaweutsaesckseudsucdiuved sorsedssevsetissstiens 11 2 S LANE Ta P EAE cu EEE EE E 13 3 Physical SOU cosore nic stats teases eten kasirna 13 C DESIGN OF THE EXPERIMENT cscccssscsssscsscssscssssscssccsscscsnecsssscssces 15 1 Training Tutorial Development 15 2 Questionnaire Development 15 3 Training Session One 16 4 Training Session Two 17 5 Session Ihreeee 18 6 Experiment Scheduling Considerations 19 D SUBJECTS s
81. ements such as user interface improvements to provide feedback to the user including labels for maps improving system latency etc I did not like the amount of time that it took to load the data sets If there was a notification that stating loading then maybe it would have been ok Same comments as previous Relatively easy to use software I could pick it up where I left off each session with only a few seconds of menu searching at the most to re familiarize myself with the tools The graphs however are mind numbingly bad They are difficult to read If they were as intuitive as the basic controls for the rest of the software I would recommend this program wholeheartedly As it currently stands I would break my monitor trying to read even a small number of them I can definitely see potential in the analysis features of this software as long as the data is accurate and timely but the performance appeared a little slow to me not knowing what s under the hood I understand I could create smaller sets of data but this might be counterproductive and not provide the big picture depending on the intent scope of this software It was adequate Somewhat slow but that is understandable considering the amount of data involved Functionalities are quite intuitive Inability to zoom into charts is slightly inconvenient No It s jittery unresponsive Takes forever to load information The user help is not very friendly This software needs seriou
82. erformance of the software Please explain I would like to know when it is processing data to know to wait If analysis tools charts and data displays were offered with visual examples of what the products looked like it might assist choosing the correct one for the job Otherwise the software seems like it would be a reasonably good tool for both analysis and visual display of data Not with my own performance no If I had to use this system on the move I would not be happy with my current level of training I would need much more time using the system If this was for tactical operations center I would be able to use it better Be able to also input data easier Based upon the questions posed and my background as a Surface Warfare Officer this software would be highly useful if data could be displayed in maritime regions With this software or similar software and good historical data and piracy and counter narcotics missions that I have performed could have been effective if we had the data and tools software to effectively focus our limited resources Overall it has a lot of lag and lacks fine controls The graphs are hard to read but there may be functionality to make them more readable With some improvement this could be a useful tool for analysis of geographic data Yes but more training is needed and adjustments to the software would be 129 Q Satisfaction_7 Were you satisfied with the performance
83. erstand It took some training to get used to it Yes Yes however labels should be provide Yes no issues with the actual visual portion minus the charts Relatively so Of the maps and controls yes but not the charts as I stated previously the line charts were a pain to read Yes Yes Projecting the charts directly onto the regions increased clutter on the screen and is a bad stylistic choice The map projection was clunky as well and the software would be wise to use a Google Earth type 3d view vice a 2d projection Yes Depending on the level of detail labels of locations would be useful map yes graphs no Yes Yes except for some of the charts the cluster charts were the worst I think Yes except the charts Yes Table 55 Table of free form responses to question Eff_4 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Use_2 What was the most difficult task to complete Selection of individually highlighted areas and the graphs associated was sometimes difficult due to the need to click inside wait click outside delays I found myself wondering if I had done it correctly based on the lag in response Setting up the charts I definitely needed more time using that I know that was a focus last session But I was not practiced enough to remember it correctly Graph generation Reading IED attacks in 2010 fr
84. essible from the interface Recommendation C 16 to link the user s manual to the help tab is a repeat of C 3 If the user s manual provides proper guidance this will increase user satisfaction Similarly recommendation C 17 recommends going further with a comprehensive help capability 79 This feature would put DaViTo on par with more familiar software packages that utilize searchable help databases in addition to online help capabilities Additional recommendations in Table 38 are C 18 and C 19 call for improved software stability and more graphing tools respectively These are repeats of A 1 and C 4 respectively responses to satisfaction questions Both recommendations were derived from participant free form Recommendation Designator Ras Attribute and Basis for Recommendation Description tisfaction 4 t of think th AE E e T Satisfac o 6 percent of users think they gan conduct basic tasks and 25 percent of users think C 16 to the Help tab on the l they can complete complex tasks without the ribbon bar user s manual Satisfaction Only 21 percent of subjects are Develop a comfortable using DaViTo without assistance C 17 comprehensive help W a Additionally this was the second most frequent capability for DaViTo f f complaint with regards to satisfaction I ft E aaa Satisfaction This was the top complaint with C 18 processing response
85. f questions as well as different users While the SBF chart performed well on six out of seven questions there were other formats that performed better on certain questions For example the SBF performed well at 75 percent accuracy on question G however it was outperformed by CCF which had 100 percent accuracy In question by question comparison one can see which types perform better and that a cumulative type graph such as SBF is not always better for questions that involve single event types rather than totals as no mental math is required to derive the number from the stack of bars The results of individual questions can be seen in Appendix C The worst performing graph types were the LCF and SLMF The LCF is the default choice in DaViTo and can also be used with markers like the SLMF While they had overall accuracy averages of less than 50 percent and some of the longest average times they performed well at some questions This is again evidence that having multiple options gives users the capability to choose the type that best conveys the data they are trying to represent D SURVEY RESULTS Each session was concluded with a survey to allow the participant to give feedback about the training and software For sessions one and two there were three common questions which were also the first three questions for the final session The final session concluded with a more extensive survey with questions related to the five usability a
86. f additional training TOURS peede 121 Figure 48 Session three question two subject s self evaluation of additional training FGUES needdeeeee E A a A a AERES 124 xii Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Table 22 Table 23 Table 24 Table 25 Table 26 Table 27 Table 28 Table 29 Table 30 Table 31 LIST OF TABLES Matrix of usability attributes and their descriptions Rubin amp Chisnell TNE Vises issue bashes canine ss es 00 aad 9 Description of items in equipment configuration illustrated in Figure 1 14 Breakdown of subject population by age 20 Breakdown of subject population by rank 21 Breakdown of subject population by rank 21 Subjects assessment of their own computer skill level 21 Subjects reported weekly computer usage 22 Summation of subject populations preferred operating system 22 Summary of mouse click and correctness statistics for all session three participants as ats 29 Summary of time duration statistics for all session three participants 32 Table showing the following information for the five graph type
87. fferent graph types and further improve the usefulness of DaViTo It is clear from the results that while the SBF is the top overall performer different graphs produce better results dependent upon the question Similarly most software with graphing functionality Excel JMP are not limited to two graph types but give their users a plethora of choices as well as provide brief explanations of the strengths of each format Admittedly DaViTo has the capability to produce different graph types either through the use of R or importing the data to Excel Unfortunately this is laborious and counter to one of the goals stated for the development of DaViTo which was that the software would be easy to use for someone unfamiliar with statistical methods Evangelista Henry Buttrey amp Whitaker 2012 Requiring a user that is unfamiliar with statistical methods such as those employed by R or requiring the importation manipulation and eventual graphing of data 73 in Excel is not realistic Including more options for graph types in the user interface such that they are easily accesible will better meet the original development goals of DaViTo R dati Designator pera 3 ane Attribute and Basis for Recommendation Description Make the Stacked Bar Efficiency The SBF graph was the most B 1 Format SBF the conducive to generating a correct answer from default graph type experiment subjects B2 Add more options for Usefulness
88. functions Dislikes 1 speed of map refresh likely a function of the hardware but it was a little annoying to see the lag between a selection that the new map 2 MGRS was misspelled in the dropdown menu was written MRGS Likes 1 My first like is the usability of the navigation functions They are very easy to understand and apply 2 My second like is the use of a importation of data sets from pre existing files 3 My third like is the ease of understandability of how to use the program Dislikes 1 My first dislike is the time lag in the zoom in zoom out function 2 My second dislike is the caliper function of the use of distance This tool needs to be updated to something that is more user friendly and easily shows data that you are trying to find to new users 118 Session One Question Three What were your top three likes and dislikes with your first exposure to DaViTo Likes 1 Liked that it was pretty simple to use 2 There wasn t too much information overload on the screen 3 It also seems familiar with other applications like GPS so that made it easy to figure out Dislikes 1 1 didn t like that it seemed slow to respond I feel like I could easily over task the computer if I was trying to get information quickly 2 It took me a second with help to figure out how to use the measurement tool 3 While I liked that there wasn t information overload at the same time I felt it was too generic and
89. gave up on task ten Both tasks required the subjects to make a shape within which a graph would be made Once the shape is made the user must click outside of the shape to get the graph to populate within it Frequently when doing this the program would lag and without a progress status users would be repeatedly clicking outside the shape in an effort to get the graph to populate Some participants even gave up waiting and moved on to the next task particularly for task ten This is indicative of a usability deficiency within the software a sverige Average Mowe ya tek Na Mouse Clicks Clicks for for All Users Correct Answers Pi 4 6 52 6 52 0 5 19 3 31 3 7 00 7 00 0 4 36 862 3 3 48 3 48 0 1 1 27 10 25 43 22 63 31 86 7 53 1247 5 15 87 35 00 15 1 45 1268 2 3 13 242 6 5 1 23 468 Table 9 Summary of mouse click and correctness statistics for all session three participants 29 Mouse Clicks by Task 30 nn 3 25 O g 20 3 15 Subjects Average 5 Mouse Clicks 2 40 Bench Mark 53 5 D 1 2 3 4 5 6 7 8 9 10 11 Tasks Final Session Figure 3 Graph comparing average participant mouse clicks to the bench mark Mouse Clicks for Complete and Incomplete Tasks 40 35 E 30 25 Average Mouse Click a For Completed Task 20 E Average Mouse Clicks For Incomplete lask 1 gt A n Bench Mark 10 5 Px 4 M a t aa
90. gh comments and very helpful insights Finally a special thanks to our advisor Dr Ji Hyun Yang Your support was crucial from the day you took us on Your knowledge was instrumental throughout the experiment development conduct and analysis as well as the exhaustive thesis writing process We extend a sincere thank you for your persistence in encouraging us all the way to the end xix THIS PAGE INTENTIONALLY LEFT BLANK XX I INTRODUCTION A PURPOSE In the early years of computers and the software to use and operate them most applications were developed by highly trained and specialized personnel Software was designed for a user with commensurate specialized skills in the computing arena Jobs that required the use of computers also required specific training or education in order to use them As computing power began to increase and the use of computers spread across professions and into homes there were more users with little or no education or training in the field Software development evolved with this merging toward a central theme of usability for the end user This enabled computers and software to traverse into all fields of work for a full range of users Usability has many definitions but in software engineering it is the quality of a system with respect to ease of learning ease of use and user satisfaction Rosson amp Carroll 2002 These three usability perspectives are directly reflected in a summary sta
91. ght clicking in the Map and Data Display area This further supports the usability deficiencies previously discussed in this chapter with regard to excessive mouse clicking and missing functionality in the Map Controls ribbon 34 pen Shape File Controls 1 ore 0 9 20 0 Figure 7 Final session task one heat map and mouse clicks Task one directed subjects to open the WITS csv data file which required interaction with the Data Sets region 35 Gaze Observations in Each Region Progress Bar 1 MB Repon Selector 1 Shape File Controls 1 fat Shape Files 1 300 00 Count 200 00 100 00 0004 Figure 8 Graph of final session task one cumulative counts of gaze observations for all subjects Counts correlate to the number of subject fixations in each region shown in Figure 7 36 Figure 9 Final session task four heat map and mouse clicks Task four directed subjects to create a user defined polygon and display the chart 37 _ Progress Bar 1 30 0 ETAN i i i eronenTsk mal 0 7 B04 Figure 10 Final session task ten heat map and mouse clicks Task ten directed subjects to answer a question that required the creation of a user defined polygon and to display the chart 38 4 Task Results Summary Two usability deficiencies were identified by the output data from the ses
92. graduate School Consent to Participate in Research Introduction You are invited to participate in a research study titled Usability Testing and Workflow Analysis of the TRADOC Data Visualization Tool DaViToy This research is focused on evaluating the usability and workflow of the DaViTo tool with an aim of improvement Purpose The purpose of the research is 10 provide recommendations for improvement on DaViTo from user inputs and statistical information collected during human to software interaction sessions These recommendations are intended to improve the workflow usability and visual clarity of DaViTo Procedures We are asking military officers and civilians to participate in a usability study of the Data Visualization Tool DaViTo developed by TRAC Moeterey The target number of participants is 24 The stady will be condected over a three to four week period During the study we will use cameras and software to estimate and record eye gaze direction eye dwell time and mouse and keyboard inputs These inpats will be recorded but the individual participant will not be recorded The participants will be asked to atend two familiarization and training sessions that will last for approximately 30 60 minutes and will be separmed by a one week period then a final evaluation session In the first training and familiarization session the participant will be asked 10 complete a demographic questionnaire and create a profile in the eye
93. guirementsipolicies Afler the experiment is completed the Principal Investigator will submit to the Human Subjects Protection Office all signed informed consent documents unanticipated problem reports adverse event reports and a End of Experiment Report The Human Subjects Research Office will secure these documents for 10 years and then forward to the nearest FRC Daniel T Oliver President Institutional Review Board Naval Postgraduate School 141 Subject Line Call for Participants in MOVES TRAC Monterey Student Thesis Research Please join our study as a volunteer for the following thesis USABILITY TESTING AND WORKFLOW ANALYSIS OF THE TRADOC DATA VISUALIZATION TOOL DaViTo Dear Prospective Participant Would you like to participate in NPS student thesis research to improve a geospatial mapping tool for use by the U S military We are NPS students in the Modeling Virtual Environments and Simulation MOVES Institute conducting a usability test as part of our thesis The purpose of the research is to 1 Evaluate the usability and efficiency of the DaViTo and determine how closely the actual software workflow reflects how the user thinks of the workflow 2 Determine how the DaViTo efficiency and usability can be improved What design suggestions need to be made to achieve these improvements This research opportunity is open to all NPS military students and civilians who are not color blind and were
94. gure 7 also displays a large concentration of users gazing at the Map Controls area and the mouse icons representing mouse clicks there It is expected that there are a large number of gazes in the Map area due to its size and the fact that the user would focus or scan there while waiting for the dataset to populate However the mouse clicks and concentration of gaze in the Map Controls area indicates that participants showed a tendency to go to this region of the interface when trying to open a file This is consistent with other software programs that have a menu ribbon at the top of the page with functionality to open a file but there is no functionality such as this in DaViTo This again indicates a usability deficiency of the DaViTo software Finally the conclusions reached from the task duration and mouse clicks data is further reinforced by the heat maps for task four shown in Figure 9 and task ten shown in Figure 10 Tasks four and ten required subjects to create a user defined polygon and allow it to populate with a graph of activity within the polygon These heat maps show the large number of mouse clicks during these tasks as indicated by the mouse icons in the heat maps This illustrates the unnecessary mouse clicking that was done during these tasks Additionally these tasks have significant gaze concentrations in the Map Controls area indicating that users may be searching there for the shape drawing functionality that requires ri
95. h session and their designation letter This is a repeat of Table 12 in Chapter IV Section B shown here to correlate data to questions Question Accuracy A Correct CCF 41 67 SBF 70 83 LCF 33 33 SLMF 8 33 SAF 33 33 Table 44 Statistics for session one question A for all graph types 111 Session 1 Question A Time vs Accuracy Accuracy correct 0 10 20 30 40 50 60 70 80 90 30 X SAF Oo 10 a SBF CCF X SLMF LCF _ 50 E SBF 3 A LCF J 60 v SLMF 70 K SAF BO CCF 90 Figure 37 Plot of session one depicting time to answer versus accuracy of the answer for question A Note the decreasing time on the Y axis Top performance is in upper right corner Question Accuracy B Correct 75 00 Table 45 Statistics for session one question B for all graph types Session 1 Question B Time vs Accuracy Accuracy correct o 10 20 30 40 50 60 70 80 90 m e a7 X SAF CCF 19 E SBF g A LCF 221 X SLMFE 223 i SLMF E SBF SAF 25 ECF a 27 33 CCF Figure 38 Plot of session one depicting time to answer versus accuracy of the answer for question B Note the decreasing time on the Y axis Top performance is in upper right corner 112 Question Accuracy Time C correct sec CCF 45 83 65 78 SBF 91 67 22 17 LCF 50 00 50 8
96. hing is doing why not write the data to a background buffer Stuff can be put over this buffer but it doesn t need to be recalculated every time Essentially improve flexibility so the user can modify and edit output similar to capabilities provided by other tools including excel Nothing more to add just improve the output of some of the charts meaning the actual display size in general Make the graphs more intuitive as the software itself is Have the ability to toggle quickly between the types of charts to pick which one presents the most clear summary of events for which you are analyzing It would be interesting to add some real analysis features such as extrapolation prediction and so forth to leverage statistics but I m not sure how difficult that would be Bigger charts when required Graphs that a human can understand without performing some kind of advance math problem Explanations available for users that are not proficient with basic statistical analysis would be useful in the output More user friendly charts The graphs should be tailored to the query Nothing else Easier to read charts Pie charts for one Cluster column format needs to be spread out Line the data up with the grid line references to make it easier to follow Table 64 Table of free form responses to question Visual_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software
97. his thesis is to test the usability performance and workflow functionality of the DaViTo software The development of this experiment was driven in large part by definitions designs and procedures outlined in two references Rosson and Carroll cite usability professionals from Digital 7 Equipment Corporation describing usability engineering as using the term to refer to concepts and techniques for planning achieving and verifying objectives for system usability The key idea is that measurable usability goals must be defined early in software development and then assessed repeatedly during development to ensure that they are achieved Rosson amp Carroll 2002 While TRAC Monterey developed goals for the development of the DaViTo software it did not follow this definition with respect to end user usability The Handbook for Usability Testing refers to usability testing in the same manner in which it was desired to test DaViTo in this post development method The authors deem it a process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria Rubin amp Chisnell 2008 Again throughout the development of DaViTo no subject testing was conducted to refine the software and ensure the stated goal of easy to use for someone unfamiliar with statistical methods and Geographic Information Systems Evangelista Henry Butt
98. hown in Table 15 Figure 15 shows that SBF CCF and SAF were clustered near the upper right corner with the best results 47 Session 3 Accuracy Correct Time sec CCF 75 00 54 62 SBF 75 00 39 09 LCF 50 00 87 31 SLMF 47 92 60 16 SAF 60 42 45 69 Table 15 Overall statistics for all graph types in session three Session 3 Time vs Accuracy Accuracy correct o 20 40 60 20 100 20 e 30 SBH va 40 m SBF XK SAF g 50 STMF ALCF z 60 x lt SLMF 70 K SAF 30 90 A LCF 100 Figure 15 Plot of session three times to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in upper right corner 4 Analysis of Overall Graph Data The results in Table 16 clearly indicate that the SBF was the top performer in both accuracy and time at 73 38 percent and 35 06 seconds respectively The SLMF had the worst accuracy and the LCF the worst time also in Table 16 SBF and LCF formats are already in use in DaViTo with the LCF as the default type Figure 16 shows that CCF was a close second in accuracy but on average takes 50 percent more time to answer the question than with the SBF 48 While the SBF was clearly the top performer there are other things to note when looking at the overall performance If only one type of graph were to be used in DaViTo the SBF would be the recommended one however other formats have strong poin
99. hrough additional testing Another is research into the application of eye tracking to usability testing As a relatively new source of data for research eye tracking has boundless potential Determining the most effective ways to utilize the process and resulting data to maximize improvements to the research subject would be beneficial Finally usability testing of other products with the same methods used in this thesis would be useful Confirmation of the methods used to conduct testing on DaViTo could lead to advancement in the field of usability testing and enhance future product analyses 89 THIS PAGE INTENTIONALLY LEFT BLANK 90 APPENDIX A CONSOLIDATED RECOMMENDATIONS Appendix A presents a consolidated list of recommendations for the DaViTo software in Table 42 The designator represents which portion of results it came from Recommendations based on task results have the designator A those from graph results are designated B the C designation is based on survey results and finally the D designator is for author recommendations The attributes are described previously in Table 1 Repeated recommendations are identified with an asterisk in the designator but have a different attribute and or basis R dati Designator Attribute and Basis for Recommendation Description Improve software Efficiency When drawing user defined A l processing response polygons there was excessive lag time or th
100. ics such as gaze patterns and dwell times to identify software inefficiencies The second objective is to identify areas where efficiency and usability can be improved and translate these into actionable recommendations Again a subset of secondary objectives was used to complete the primary objective Pinpoint ways to improve the software s visual palette in a manner that will increase information transfer to users e Identify button improvements such as location and identification that will increase ease of use Identify ways to optimize input device use to increase efficiency The final objective was to determine if there were deficiencies in the use of data representation techniques within DaViTo and how to correct them The secondary objectives to complete this were as follows Identify graph types that best represent the statistical analysis of typical data sets used in DaViTo Determine additional graphing tools needed to improve output statistical analysis by users with little or no statistical training C SCOPE AND LIMITATIONS This thesis is limited to the usability of the DaViTo user interface itself There is no intention to investigate the software behind DaViTo despite one of the software goals being to allow user modification of the code for customization to individual needs DaViTo is a broad program that draws from three different open source software packages R JFreeChart and OpenMap Additionally
101. igure 17 Plot of all questions times to answer versus accuracy of answer The labels are the graph type followed by the question identifier Note the decreasing time on the Y axis Top performance is in upper right corner 5 Graph Results Summary Two usability deficiencies were noted as a result of the data representation portion of the experiment Recommendations are consolidated in Chapter V The first identified deficiency for data representation is related to the attribute of usefulness and was the inability to answer all questions with considerably high performance with the default graph format used by DaViTo The default LCF graph is the lowest performing graph of the five tested Fortunately the highest performing graph the SBF is also readily available for use with the ease of a simple radio button in the Display region of the DaViTo interface Using the results of both of these graph formats all questions were able to be answered with an accuracy that was on average in 50 excess of 70 percent However this puts the user in the position of recognizing that a different graph would relay information better and have the technical prowess to change the graph type This also results in deficiencies in the attributes of efficiency effectiveness and likely satisfaction The second identified deficiency is in the attributes of efficiency and effectiveness As seen in Figure 17 different graphs perform better for different types o
102. ill not be discussed The final session required execution of a series of eight tasks with individual instructions followed by a more complex task divided into three general instructions Of the 24 participants only one achieved a perfect score by completing tasks ten and eleven with correct answers Previous incorrect answers in tasks one through eight would not affect completion of tasks nine through eleven however failure to properly open the data set and Shapefile in task nine would prohibit completion of task ten No partial credit was given Three separate analyses were conducted on the session three task data number of mouse clicks for each task duration of time to complete a task and cumulative heat mapping of the screen area for each task Mouse click data and time duration of task completion were analyzed to determine efficiency of operation with the interface Cumulative gaze density mapping was analyzed for workflow processes showing gaze concentration areas in a heat map to determine if subjects were looking at the appropriate part of the screen for completing a task 1 Analysis of Mouse Clicks Correctness and Completeness of Final Session Tasks Mouse clicks for each participant were recorded automatically by the EyeWorks software This was compared to a benchmark number of mouse clicks established by the experiment designers as the most efficient possible manner to complete the tasks correctly Task descriptions for sess
103. ing the software in my third session I d also have a user friendly help section that we could 131 Q Improve_3 How would you improve DaViTo Please explain click on Additionally I d like to see a way to personalize the display buttons etc for use That way I could put things where I d like them Make it faster Improve user interface issues essentially to make it more intuitive for the user improve the flexibility so the user can modify output and fix system latency Improve the data set loading issue Other than that the tool was excellent Make the graphs easier to read Improved charts fix the lag see previous comments Navigation button could be renamed zoom box or something similar to avoid confusion with the map navigation button user could be given more control of the charts e g zoom in to some portion of the data loading time if possible e g when loading Make it more user friendly and responsive Comments provided in previous questions Better maps and response times essentially Spend more time accomplishing the tasks and improving the user friendly charts Speed it up and improve the graphical outputs Figure a way to remove the delays when loading or creating shapes Again improve some of the charts I thought the cluster charts were not good visual representations of data Charts Table 58 Table of free form responses to question Improve_3 for all 24 sub
104. ion three are in Appendix E The summary results for the tasks in session three are displayed in Table 9 This is also graphically represented in Figures 3 and 4 The most notable result from the mouse click analysis was the large deviation from the baseline for tasks four and ten Task four directed the participants to create a rectangle chart on the data point in the southwestern most part of Kenya and display the chart As seen in Table 9 the benchmark for this task was 10 mouse clicks yet the average number of clicks for subjects was 25 43 with a standard deviation of 12 47 This difference from benchmark to subject average is displayed in Figure 3 While users who completed the task correctly 28 improved to 22 63 clicks the average was far worse for users that could not complete the task correctly with 31 86 clicks as shown in Figure 4 Task ten was a more complex task directing the subjects to please answer the following question what were the total events for IED attacks for 2010 in the most northwestern provincial boundary Table 9 shows that on average subjects clicked 15 87 times to complete the task while the benchmark was 5 clicks with a standard deviation of 12 68 This deviation between the benchmark and subject average is shown in Figure 3 Figure 4 shows that for subjects who did not complete the task correctly they averaged only 15 clicks while those who did averaged 35 clicks This indicates that some subjects
105. ion with the software Written response 150 Quantitative amp Subjective Questions 8 How would you like to see the software employed Written response 151 Graphics Questions Do you know what this icon means D Drawing oPan o Distance 5 Navigation D Gestures D Go forward to next projection D Go back to previous projection O Map Navigation Ot don t know i eee icon means o Drawing oPan Distance a Navigation o Gestures D Go forward to next projection D Go back to previous projection Map Navigation DI don t know EJ Do you know what this icon means o Drawing D Pan O Distance o Navigation o Gestures D Go forward to next projection D Go back to previous projection D Map Navigation D I dort know Do you know what this icon means n Drawing oO Pan Distance o Navigation Gestures D Go forward to next projection D Go back to previous projection D Map Navigation n I don t know 5 Do you know what this icon means o Drawing o Pan o Distance D Navigation Gestures a Go forward to next projection O Go back to previous projection 0 Map Navigation otdow t know 152 Graphics Questions 4 Do you know what this icon means o Drawing OPan o Distance o Navigation o Gestures 0 Go forward to next projection o Go back to previous projection D Map Navigation 0 I don t know 153 THIS PAGE INTENTIONALLY LEFT BLANK 154 LIST OF REFERENCES Merriam Webster Incorporated 2012 Retrieved August 2
106. ions easy Data Display Area to read Make data t f e ERR Efficiency Subjects reported this as a deficiency C 13 within a data set that made visual representations difficult to read sortable by color Learnability 75 percent of subjects reported a most people could figure out how to use DaViTo C 14 based training product f i with the tools provided however 58 percent for DaViTo Bs believe more training products are needed ae Learnability 88 percent of subjects are either Distribute a training f PRP C 15 neutral or believe that more training is necessary product with DaViTo to effectively be able to use DaViTo Link the user s manual Satisfaction 46 percent of users think they can conduct basic tasks and 25 percent of users think C 16 to the Help tab on th eee they can complete complex tasks without the ribbon bar user s manual Satisfaction Only 21 percent of subjects are Develop a 3 comfortable using DaViTo without assistance C 17 comprehensive help oe i e f Additionally this was the second most frequent capability for DaViTo complaint with regards to satisfaction I ft Bi Satisfaction This was the top complaint with C 18 processing response i oe regards to satisfaction time and stability Includ h tools t W R Satisfaction This was the third most frequent C 19 improve graph complaint with rega
107. ions had distinct divisions which made for a natural separation of the data into three sections The purpose of the first interaction with the DaViTo software following the tutorials and initially in the beginning of session three was to conduct tasks which became one grouping of information for data analysis Following task completion the subjects would analyze graphs and answer questions in order to determine the best data representation method This became an additional grouping for analysis Finally the experiment subjects completed a series of questions at the end of each session These questionnaire answers became the third grouping for analysis B DAVITO TASK PERFORMANCE AND INTERACTION The first two session tutorials concluded with a series of tasks and the final session began with similar tasks Task completion meaning the user completed the directed task as well as time and correctness were used to evaluate the ability of the participant to utilize the basic skills learned from the tutorials such as navigation file operations and creating a data representation This was not an effort to evaluate participants from a training transfer perspective but rather to determine the efficiency and workflow with which simple tasks could be completed by a novice user with minimal training on the basic functionality and tools available in DaViTo Tasks in the first two 27 sessions were used to reinforce the skills taught by the tutorial and w
108. iracy or sea nautical charts based counter drug operations Develop a standardized Army or Usability Without a formal database and data D4 service wide database formats the tool itself may suffer reduced and data formats for usability use with DaViTo Table 42 Consolidated list of recommendations for the improvement of DaViTo and their basis derived from all research methods 95 THIS PAGE INTENTIONALLY LEFT BLANK 96 APPENDIX B ADDITIONAL TASK RESULTS Appendix B contains cumulative gaze frequency data for all final session tasks as heat maps in Figures 19 through 26 Figures 27 through 36 show the cumulative counts of gaze observations for all subjects in each region of the interface for the heat maps Heat maps and gaze observation graphs presented in the body of the thesis are not reproduced in this Appendix 97 my 5 inl SEIN S Ha Figure 19 Final session task two heat map and mouse clicks Task two directed subjects to load the Kenya Roads Shapefile which required interaction with the Shapefiles region 98 Map Controls 1 19 6 226 2 Sha 1 i 11 3 10 3 pe Files line C T om hag Start Date End Date Bin size Mont Region Selection List Region 20 1 20 0 DaViToNextTask 1 0 0 Ad oo Figure 20 Final session task three heat map and mouse clicks
109. islikes Program too slow to respond Dislikes Loading the map shapes was very slow Likes It took a few minutes to remember how to navigate around but it seemed to come back to me Good user interface Dislike Is the program is a little slow with generating charts Like None Dislike Not using the map control bar Likes Ability to analyze data in picture format to gain insights Dislikes System latency in loading data Inability to know whether system is working or not i e progress bars or hour glass that indicates that system is working on providing feedback for desired input Table 53 Table of likes and dislikes for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification B FINAL SURVEY Appendix D part B presents additional results from the final session survey free form questions Note that the responses in Tables 54 through 64 are not altered in content from what the subjects entered in the survey Q Effect_3 Was the software simple to use Please explain Some of the finer controls associated with selection suffered due to lag Yes It was relatively easy to use but chart generation seemed more complex than it needed to be Yes with the exception of time delays without status icons to let the user know that information was being processed by the system Yes with training The basic interface is simple enough but some options are harder t
110. it Manufacturer Gateway Processor Mobile AMD Sempron 3300 1 99 GHz RAM 448 MB DVD Player Philips DVD RW model SDVD8441 Screen 16 in LCD display The laptop for faceLAB control had the following specifications System Microsoft Windows XP 32 bit Manufacturer Hewlett Packard Processor Intel R Core 2 CPU T7200 2 00 GHz RAM 99 GB Screen 15 in LCD display The subjects utilized the following input devices a Dell two button mouse with a scrolling wheel in the center as well as a standard Dell QWERTY keyboard Additionally directly in front of the keyboard was a pair of stereo cameras and an IR pod that took input from the subject s eyes The IR pod and stereo camera set were manufactured by Seeing Machines model number IRPD PI R2 12 2 Software Three software packages were used for the experiment The program used to initiate and control subject eye tracking was faceLAB version 5 0 5 The experiment was designed and implemented using the EyeWorks software suite version 3 7 3 145 Finally the DaViTo software package was used by the participants and was the software of interest and evaluation Data was collected automatically via the eye tracking software faceLAB and EyeWorks FaceLAB was the software vehicle to track subject s eyes and facial features It was coupled with the EyeWorks software which would receive the automated data from faceLAB and record it for future analysis The EyeWorks sui
111. it could broaden the horizon of data available I m not sure No idea Currently my experience with the tool is exploring event data but this tool could easily be modified and employed for sustainment and logistics personnel In training and field environments Afghanistan of course I suppose if it were made available for basic research I had never heard of it before Integrate this software into CPOF rather than as a stand alone application The features I ve seen indicate a focus on analyzing data sets for trending that could be applicable to both the tactical and operational levels so it would be better to have a shared platform for looking at the same data Probably useful for the OR and data analysis guys 136 Q Improve_8 How would you like to see the software employed Please explain Used for trend analysis in order to make better decisions at company level and higher Basic analysis of available data for military needs Useful as a planning tool for events in areas being entered Could give a visual indication of the activity expected to be encountered Could be used in a HQ to conduct trend analysis for SIGACTS Would be useful for land and see navigation It seems it would be more useful for land based forces I serve on Navy ships and I do think it would add much value there I have no reference Would be useful in organizational headquarters first in a garrison setting a
112. it is used Common definitions from the Merriam Webster Dictionary are capable of being used or convenient and practicable for use Merriam Webster Incorporated 2012 In the context of computer software development and testing it has a much more specific definition In Rosson and Carroll s Usability Engineering usability is defined as the quality of a system with respect to ease of learning ease of use and user satisfaction Rosson amp Carroll 2002 It is even more generally described in The Handbook of Usability Testing as what makes something usable is the absence of frustration in using it and further defined as when a product is truly usable the user can do what he or she wants to do the way he or she expects to be able to do it without hindrance hesitation or questions Rubin amp Chisnell 2008 Both of these definitions apply to the purpose of this thesis research With the overall goal of developing recommendations for improvements to the usability of the DaViTo software clearly improving the ease of learning ease of use and user satisfaction while minimizing user frustration would be drivers for these recommendations Throughout the development of the experiment and subject trials frustration was frequently seen with the software pointing to areas where improving the ease of use would improve satisfaction C PURPOSE OF USABILITY TESTING The purpose of usability testing in the context of t
113. iven to stop using the software Question Eff_4 results also identify efficiency failures in visual representation as a result of no geographic labeling of countries or regions and the inability within DaViTo to sort different data types within a data set and display their data points with different colors This question also had multiple responses again identifying difficulty reading graphs in DaViTo Deficiencies will be addressed by recommendations in Chapter V 59 to understand Efficiency Question earns Scale Frequency Mean Mode Median The organization of l l the screen matched z a 2 Eff_1 3 5 3 5 4 4 my expectation of button location 10 5 4 1 0 Similar icons and 2 3 activities are Eff_2 3 3 3 83 4 4 consistently placed 4 13 5 5 1 0 Navigation of 2 5 DaViTo was Eff_3 3 3 3 58 4 4 intuitive 4 13 5 3 1 0 The color scheme for 2 3 the software was Eff_5 3 4 3 79 4 4 appropriate 4 12 5 5 The text within 1 0 output graphical 2 8 representations was Eff_6 3 6 3 21 2 3 easy to read and 4 7 match to data 5 3 The software header bars and dropdown Eff 7 3 7 3 83 4 4 lists made 4 16 navigational sense 5 4 1 9 The software was z Eff_8 3 3 2 25 1 2 stable no freezing 4 6 5 0 Was the visual Lack of geographic labels representation easy Eff_4 Charts difficult to read Unable
114. ixations in each region SHOWIN Fis re Jccce 109 Graph of final session task nine cumulative counts of gaze observations for all subjects Task nine directed subjects to remove all data sets and Shapefiles then load the Indure_Demo data set and the afghanistan_provincial_boundaries Shapefile Counts correlate to the number of subject fixations in each region shown in Figure 25 109 Graph of final session task ten cumulative counts of gaze observations for all subjects Task ten directed subjects to answer a question that required the creation of a user defined polygon and to display the chart Counts correlate to the number of subject fixations in each region shown m Fisure TOn eaa leas 6 110 Graph of final session task eleven cumulative counts of gaze observations for all subjects Task eleven asked subjects to answer a trend analysis question using the graph generated in the user defined polygon from task ten Counts correlate to the number of subject fixations in each region shown in Figure 26 110 Plot of session one depicting time to answer versus accuracy of the answer for question A Note the decreasing time on the Y axis Top performance is in upper right corner p 112 Plot of session one depicting time to answer versus accuracy of the answer for question B Note the decreasing time on the Y axis Top performance is in upper right Corner sssesesssessesesessseressseessressersseesseeesseee 11
115. jects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_4 What was the most frustrating feature of DaViTo The lag On screen help I should have used the help button more This would likely be eliminated by more training Sluggishness and chart generation Lag Not knowing if it was frozen or thinking Finding the chart option for Kenya Chart output was difficult to match to months and difficult to add up events Lag time in loading things and lag time between selection and reaction Interpreting charts and lag time for map loading It seemed really slow Lots of delay loading maps and right clicking Sometimes the delay would be counterintuitive because I thought I was doing something wrong but in reality it was just slow It was slow System latency Data set loading time 132 Q Improve_4 What was the most frustrating feature of DaViTo Graphs graphs graphs Lag time when selecting maps data sets Load times Slow and unnecessarily complicated to navigate way around data Confused on where to look to load data took too much time Slow map response times Interpreting the charts Sluggish responses The delay experienced when creating or loading a shape The slow time of generating charts and trying to read and interpret the cluster charts Which mouse button to use and when in what context Latency of display function execution
116. lief option or 3D capability similar to Google Earth Visualization Top response when subjects were asked what they would change about the color scheme or user interface layout C 26 C 27 The color scheme is bland Add more contrast Change the default data point color Visualization In an effort to look more inviting to users software should have visual appeal even in military applications Visualization The current default of gray has no contrast with the interface due to frequent use of different shades of gray 94 Recommendation Designator Attribute and Basis for Recommendation Description Add a collapse capability to the Data Visualization This will expose more of the Map DA Sets Predefined and Data Display area reducing clutter and Polygons and Regions Selection List areas of the interface allowing the user more capability to work within the primary workspace of the interface Develop training products for both a Usefulness Recommendations C 1 and C 2 address the development of training products There are complex statistical capabilities in D 2 basic user and DaViTo An analyst with advanced skills would experienced analyst benefit from tailored training that goes beyond the capabilities in the user interface Extend mapping Usefulness Many Navy subjects identified D 3 capability to include additional uses such as counter p
117. m the single cursor type This will be addressed in Chapter V ELTA SECM Scale Frequency Mean Mode Median Question Designator p z 1 7 The way in which 7 6 Fearne Visual 1 3 8 2375 3 2 displayed is clear to 4 1 interpret 5 z What more would More chart options available to the user you like to see in the Visual 2 Ability to manipulate the charts such as zoom output of this H and view data values on the chart software More functionality to tailor the data Table 28 Table of visualization questions and associated statistics 65 Visualization Question Ques or Dear Icon Options Selected Frequency Pan 21 Map Navigation 1 Visual_15 ay Gestures 2 Correct 87 5 Distance 24 Visual_ 14 B Correct 100 Navigation 11 Pan 4 Visual_15 3 Map Navigation 8 I don t know 1 What does this Correct 45 83 icon mean Forward 21 Projection Visual_16 gt Pan 1 I don t know 2 Correct 87 5 Map Navigation 17 KOA G Navigation 5 Visual_17 ey 8 Pan 5 Correct 70 83 Drawing 24 Visual_18 Correct 100 Table 29 Table of visualization questions regarding icons and associated statistics 7 Improvements The inclusion of the improvements section with the other attributes was an effort to give the participants an opportunity to voice ways to change DaViTo in the contex
118. mated eye tracking to collect mouse clicks reaction time and gaze data such as fixations and scan patterns Following the call for participants and subsequent scheduling the experiment took place over three sessions The first two sessions consisted of a brief training tutorial followed by some simple reinforcement tasks and questions The objective of the tutorials was to present a method of software functionality to the participant on one screen and for them to follow along on their own screen The objectives of the final session were completion of more complex tasks data representation questions and an extensive questionnaire B EQUIPMENT 1 Hardware The main hardware used for the experiment consisted of two laptops one desktop and a set of stereo cameras with an IR tracking pod Additional hardware included multiple displays a keyboard and a mouse For the final session the Gateway laptop was 11 removed because there was no video tutorial The data was kept securely on the desktop computer then moved to a secure folder on the share drive for analysis The desktop had the following specifications Operating System Windows 7 Professional 64 bit Manufacturer Hewlett Packard HP Model HP 2800 Workstation Processor Intel R Xeon R CPU X5560 2 80GHz RAM 6 00GB Screens Two 24 Inch LCD Displays The laptop for tutorial playback had the following specifications Operating System Microsoft Windows XP 32 b
119. maybe put some hot spot labels when you hover over each province country When trying to hide the map controls I reverted back to a Windows mentality looking for or buttons to minimize or hide the map controls Multiple clicks were probably the biggest concern I had during this session since the mouse icon still moved It lead me to doubt if I had selected the proper command s Likes Interface is still manageable despite forgetting some of the commands it is possible to try out and read the tool tips to recall what is the correct action to take Dislikes Slow to load data perhaps good to have a progress bar so that user knows it s still working on it and estimate of how much longer and not that the system is hanging not able to zoom in on charts e g to see just the 2010 data on IEDs making it hard to read off and count the values Dislikes Slow as hell Unresponsive Very frustrating Shouldn t take that long to gather data Forgot how to navigate my way around the system Couldn t remember how 125 Session Three Question Three What were your likes and dislikes with your third exposure to DaViTo to draw a chart Likes I remembered most of the main features and did not have to be retrained Dislikes Slow system response times Likes Seemed easier to use by the third trial Most tasks were relatively easy to recall Dislikes Difficult to right click open the menu for the data chart D
120. ments made in the free form questions of the final survey and will be used to formulate recommendations for the next release of the DaViTo software The final survey was extensive and exhausting Multiple deficiencies were identified by both free form response questions and scaled answer questions These deficiencies will be incorporated into the recommendations in Chapter V 69 THIS PAGE INTENTIONALLY LEFT BLANK 70 V RECOMMENDATIONS The main objective of this thesis was to provide clear and actionable recommendations to TRAC Monterey to improve the usability of the DaViTo software Data analysis was divided into three distinct sections for analysis and results reporting and the recommendations will follow this same format Identical recommendations supported with different bases will be reported in the tables Additionally recommendations contained in the tables in this Chapter are connected to their primary usability attribute in Table 1 If other attributes are related they will be presented in the discussion paragraphs A comprehensive table of recommendations can be found in Appendix A A RECOMMENDATIONS BASED ON TASK RESULTS Eye tracking and interface interaction was the primary source of data for the task results analysis presented in Chapter IV Three data sources were used to generate these recommendations mouse clicks duration of time to complete tasks and heat maps of all participants gaze locations The first
121. modificatggggggg cau N 129 Table of free form responses to question Use_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software with NO modificatissssssssss n EAA E NRE 131 Table of free form responses to question Improve_3 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no M dIfCAtON prira e 132 Table of free form responses to question Improve_4 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification edini e 133 Table of free form responses to question Improve_5 for all 24 subjects in session three Comments taken directly from the EyeWorks software with THO MO CITC ATION 35 555 555 e a A i a EN AA AR E A a 134 Table of free form responses to question Improve_6 for all 24 subjects in session three Comments taken directly from the EyeWorks software with o MO CITC ATION s er a a 135 Table of free form responses to question Improve_7 for all 24 subjects in session three Comments taken directly from the EyeWorks software with jo modifiegtigggg 136 Table of free form responses to question Improve_8 for all 24 subjects in session three Comments taken directly from the EyeWorks software with jo OCT ANTONY ne ie essed ei hese eases A A a ha E N 137 Table of free form responses to question Visual_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software with o modification Aaris a e
122. nd then deployed Garrison use would ensure that both the tool and the user were familiar with the system and the interpretation of the data prior to operational employment Table 63 Table of free form responses to question Improve_8 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Visual_2 What more would you like to see in the output of this software If you could link the data through R Jump or Excel the displays might be more consistent with some of the data display tools we all have to use anyway especially when constrained to an NMCI machine Perhaps both cumulative and incremental charts could be displayed simultaneously to alleviate confusion Expanded graphing such that you could zoom into a graph and discern individual data points including scales that would be appropriately detailed to discern where the data points sit Just a clear definition of whether events are independent or cumulative Also why do the events have to be stacked or put together This is not provide me with more information If I just wanted shootings and not IEDs why would I get all the information all at once I personally want to be able to tailor the information User modifiable graphs Statistical analysis of different portions of the data i e analysis of all IED attacks for a particular month over all of the provinces Data should be more cle
123. ness Graph interpretation was identified 5 i h ci a Te by subjects as the one of the most difficult tasks readability C5 Make graphs easier to Usefulness Graph creation was identified by create subjects as the one of the most difficult tasks Draw and populate a C 6 polygon when drawn by the user Usefulness Polygon functions were identified by subjects as one of the most difficult tasks Table 34 List of recommendations and the basis derived from analysis of the usefulness portion of the survey results 76 3 Recommendations Based on Effectiveness Questions Recommendations C 7 and C 8 are both recommendations to improve software processing latency and stability This recommendation was previously identified with recommendation A 3 Similarly recommendation C 9 to implement a progress status bar is identical to A 2 The Map and Data Display Area contains legal boundaries for countries but there are no labels This was identified by subjects as a deficiency when using DaViTo in the final survey Recommendation C 10 in Table 35 addresses this deficiency by recommending the implementation of geographic labels in the Map and Data Display Area Specifically in the final session participants were directed to find Kenya and load a Shapefile This required geographic knowledge of Africa not necessary when using a typical map with appropriate labeling R dati Designator
124. ng data that was of the highest importance was that of session three particularly the conduct of completing the tasks Task completion for the first two sessions as well as graph evaluation by the subjects was also of importance While there are a large number of analysis techniques available through EyeWorks Analyze only a few were used with the goal of delivering recommendations to TRAC Monterey for improvements to DaViTo Raw mouse click data collected for each subject and session was used extensively particularly in the final session for comparison to an efficiency baseline Subject time to complete tasks was used for comparison to a time baseline Additionally fixation data was utilized in the form of heat maps to determine visual concentration areas of the DaViTo interface 2 Graphs Data representation research was important to see if the graphical methods employed by default in DaViTo are the appropriate way to display data as well as evaluate some similar methods of data representation The current default is the line chart format with the stacked bar format also available but there are a myriad of possibilities with the inclusion of JFreeChart and the R statistical package These include but are not limited to the stacked line with markers format the stacked area format and the clustered columns format as well as many others To evaluate this at the conclusion of each series of tasks the participant was then directed within the E
125. not involved in the development of the DaViTo software Stadent participation is crucial to this thesis and meeting our graduation requirements and is completely voluntary There is no direct benefit to the subject The research will take place in Watkins 212B over 3 sessions approximately one week apart beginning and ending on dates TBD The sessions will take approximately 30 60 minutes Should you choose to participate we will ensure your performance and personally identifying information PL will be secure Collective performance data and survey results will be used in our final analysis and publication however no PII data will be included and has no bearing on our analysis We will take the following precautions to protect participants All email correspondence will be via secure NPS email 2 No digital files containing PL will be saved to our computers 3 Any materials printed will have all PI removed and will be securely locked in a cabinet Thank you in advance for your consideration and participation Please reply to this email to setup your participation Your assistance is personally important to us as we work to complete our graduation requirements Sincerely APS TRB abh faves 11 2012 EXPIRES SEP 30 209 142 Darren Cutler Tyler Rasmussen LT USN LT USN IRB Chair Contact Information CAPT John K Schmidt USN jkschmidi anps edu 831 656 3864 UPS IPP APPROVED APR 11 EXPIRES E SEP 30 2012 143 Naval Post
126. nses The outcome of this portion of the thesis was useful at determining the best ways to present information to decision makers in a manner in which they can understand Finally a huge amount of data directly from the users was collected in brief questioning in the first two sessions followed by a complex and thorough final surery in session three This data was particularly useful for statistical analysis of questions related to the usability attributes identified in Table 1 as well as the development of recommendations directly from subject free form responses Methods to improve DaViTo is the heart of this research Four subsets of recommendations were generated from the results of the data analysis The first subset was derived from the results of subject tasks Recommendations were developed that would improve usability in nearly all attributes The second subset of recommendations were based on the results of subject analyses of five graph types Data visualization and analysis within DaViTo was a large part of this thesis and many improvements were suggested that would increase the usability and assist developers in meeting the initial goals outlined for the software The next subset focused on three common questions across each session as well as an extensive final survey With the wealth of information gathered multiple new and unique recommendations were formulated to improve upon all attributes of usability The final subset of recommend
127. o answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in upper right corner 48 Plot of all three sessions times to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in upper right CODE e Ia andi aa a ll SN as a eae eh 49 Plot of all questions times to answer versus accuracy of answer The labels are the graph type followed by the question identifier Note the decreasing time on the Y axis Top performance is in upper right corner 50 Graph of average response to common question two Note that the y axis correlates to the scale in Table 19 53 Final session task two heat map and mouse clicks Task two directed subjects to load the Kenya Roads Shapefile which required interaction with the Shapefiles region n 98 1X Figure 20 Figure 21 Figure 22 Figure 23 Figure 24 Figure 25 Figure 26 Figure 27 Figure 28 Figure 29 Figure 30 Figure 31 Figure 32 Figure 33 Final session task three heat map and mouse clicks Task three directed subjects to go to the Africom view 99 Final session task five heat map and mouse clicks Task five directed subjects to close and delete the chart generated in task four 100 Final session task six heat map and mouse clicks Task six directed subjects to hide the Map Controls ribbon b
128. o find than others therefore they are more difficult to remember from one task to the next Yes mostly Again the chart output and geographical labeling were issues Other 126 Q Effect_3 Was the software simple to use Please explain than that it was fairly intuitive Yes but the lag created errors and mistakes The software was a little more user friendly with the video tutorial If I had to use DaViTo without the tutorial I would have been lost Generally it was okay It seemed pretty simple at first but I think I would get really frustrated if I had to use this a lot There is definitely room for improvement Yes Things did what they looked like they would do After three sessions it became easier to user to use Yes the software was easy to use Again the delay in loading the map data was the only issue It was relatively intuitive and not complicated to operate While the execution of commands and navigation through menus was simple enough the analysis features i e graphs were sometime difficult to interpret I think this could be solved by adjusting the formatting Yes controls are intuitive and tooltip pop ups were helpful No it sucked It was simple to use but it could definitely be improved with more integration of map navigation into the mouse controls Separate buttons on screen for panning zooming grabbing and data selection was a terrible GUI choice For the most part To be proficient wo
129. ociated statistics 6 Visualizations The visualization attribute was used to identify subject opinions with regard to the visual appearance of DaViTo output and the user interface These trends could then be used to generate recommendations in addition to those generated from the five usability attributes Questions concerning visualization are presented in Table 28 and had one question with a score below three This question Visual_1 had a score of 2 375 and asked users if the output data could be clearly interpreted The large number of negative responses correlates with previous questions regarding the quality of data output in DaViTo and reinforces a deficiency in this area Question Visual_2 results were not new comments and are concurrent with similar comments from other areas These deficiencies will be addressed by recommendations in Chapter V Subjects were also questioned on their ability to identify key icons used in DaViTo Five of the six questions were answered with greater than 70 percent accuracy The navigation icon Visual_15 performed poorly with less than half of users able to identify it While in general icon recognition was not an issue with the exception of the navigation icon there were multiple comments throughout the free form portions of the survey to remove the requirement to change cursor functionality between the four types and instead add right click functionality with options to perform all actions fro
130. of the software Please explain beneficial Yes with more frequent use and a basic help menu this would be useful software Somewhat I feel like this software could be used effectively with training or better software design I recommend including up to date geographical information with the software The information should be able to be updated automatically and manually This way it could be updated automatically with a steady internet connection or manually if on deployment or in the field Also the graph display could be more intuitive There should be an easier way to change the display of the data line bar pie etc by right clicking on the displayed chart The analytical tools that are included should be addressed in training or in the user s manual They should be written so a person with a sixth grade reading and thinking level could understand them Somewhat If the dislikes were fixed it would be very helpful I feel the software is a great tool however without a larger knowledge base of how to run the software I would be at a disadvantage No I had the worst time deciphering the graphs I feel that I could manipulate DaViTo well enough to get the graphs but it s ridiculous to get any usable data from the graphs No too slow And in general the graphs aren t very good Seems to be too much information at once Moderately satisfied while the tool is simpler to use that ArcGIS it is very limited requiring improv
131. om the chart Navigate with map tools hidden Finding the chart function for the Kenyan data Reading the chart output Go to a previous view Creating the charts Deciphering the graphs to get usable information Way too difficult I m still not sure if I read the graphs right 128 Q Use_2 What was the most difficult task to complete The hardest part was remembering what the actual question was Everything else seemed pretty straight forward None Hiding the map controls Reading the ridiculous graphs Line chart interpretation was the most difficult since everything is so closely spaced and you have to look over to the left to judge of events Reading off number of IED attacks because I forgot some of the steps and because the chart was small Reading the graphs Setting up a graph for specific regions since it took a while and the user is not sure if the system is responding or still waiting for an input Making and opening the chart Finding the number of IED attacks in Afghanistan creating the rectangle shape Drawing the polygon around the northwestern provincial district to generate the chart Chart selection Reading the graphs Table 56 Table of free form responses to question Use_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Satisfaction_7 Were you satisfied with the p
132. om the graph analysis portion Appendix D Additional Survey Data This appendix contains additional results from the survey data It is divided into the three common questions for all three sessions data and the final survey and contains all subject free form input Appendix E Task Worksheet This appendix contains the task list worksheet used by the experiment proctor for the final session which lists all 11 tasks Appendix F Approved Institutional Review Board IRB Documents Including the Call for Participants Consent Form Demographic Survey and Final Survey questions This appendix contains the approved Institutional Review Board Protocol cover sheet and author generated documents for the usability test I BACKGROUND A WHAT IS DAVITO The Data Visualization Tool DaViTo was developed by TRAC Monterey to give U S Army personnel a data exploration tool that could be used to discover patterns and conduct analysis of the huge amounts of data that are generated in today s sophisticated military operational environments From the DaViTo project report DaViTo specializes in the exploration of point process data that contains both a spatial and temporal component Pre loaded with simple yet novel data visualization techniques DaViTo can enhance the analyst s ability to understand point processes across time and space DaViTo provides an interface that allows users to tessellate geographic areas define uniqu
133. ore different graph types graph options Visualization Only 12 5 percent of subjects Make data types agreed that displayed output data is clear to C 22 within a data set interpret and the third most frequent response to sortable by color desired changes to the output was a greater capability to tailor the data Alter the cursor navigation button Visualization 54 percent of subjects could not i correctly identify the cursor navigation icon 71 C 23 percent of subjects could correctly identify the to match the map navigation icon Kau be map navigation icon Some incorrect answers for each were the other icons title The functionality associated with these icons is very similar Table 39 List of recommendations and the basis derived from analysis of the visualization portion of the survey results 81 8 Recommendations Based on Improvement Questions The final segment of the session three survey asked a series of questions as to how the subjects would improve DaViTo Many of the recommendations are repeats of previous comments and were mentioned earlier with regards to specific usability attributes so only new recommendations are discussed in Table 40 Recommendation C 24 was mentioned by multiple users as a desire for standard two button mouse functionality Software designed for Microsoft Windows based systems is typically developed in anticipation of a two or three button scrolling mouse being the
134. polygons for specific smaller regions of interest and the accompanying statistical output The next training item covered was the loading and use of map layers within the Map and Data Display Area and another method for user defined polygons Using the accompanying graphs the remaining options for graph modification were covered 5 Session Three The third and final session included no training rather it was focused on conducting a sequence of tasks There were several purposes for the last session The first was to observe the workflow of the participants as they conducted tasks This was done primarily via eye tracking with the EyeWorks software and by the administrators of the experiment recording subject s answers to the final two tasks The complete list of tasks for session three is contained in Appendix E It began with tasks one through eight as a series of step by step simple tasks such as navigating to Africom and Iraq loading data and Shapefiles and creating user defined polygons The intent was to refresh the software and functionality of the interface The ninth task removed all current data sets and Shapefiles and then loaded the sets for the final two tasks Task ten was a more complex and comprehensive task that required the participant to work through several steps without having the step by step questioning entered into the EyeWorks software to direct them through completion The objective was for the subject to complete a ta
135. raph interpretation was identified C 4 improve graph ee ne by subjects as the one of the most difficult tasks readability Make graphs easier to Usefulness Graph creation was identified by C 5 f create subjects as the one of the most difficult tasks D d lat Usefulness Polygon functions were identified C 6 polygon when drawn i by subjects as one of the most difficult tasks by the user C7 Improve the software Effectiveness 83 percent of subjects were not reaction time satisfied with the software reaction time Improve software Effectiveness Subjects stated this as the worst C 8 processing response deficiency that reduced the simplicity of use of time and stability DaViTo J Effectiveness Subjects stated this as the second Cor p worst deficiency that reduced the simplicity of status bar i use of DaViTo Include geographic Effectiveness Subjects stated this as the third C 10 labels in the Map and worst deficiency that reduced the simplicity of Data Display Area use of DaViTo I ft S Efficiency 67 percent of subjects disagreed that CA processing response time and stability the software was stable and not prone to crashing 92 Recommendation Designator Attribute and Basis for Recommendation Description Include geographic Efficiency Subjects reported this as the top C 12 labels in the Map and deficiency that made visual representat
136. rds to satisfaction readability Visualization Only 12 5 percent of subjects Include graph tools to agreed that output data is clear to interpret and C 20 improve graph the second most frequent response to desired readability changes to the output was more graph manipulation tools 93 Designator Recommendation Description Attribute and Basis for Recommendation C 21 C 22 Add more options for graph types Make data types within a data set sortable by color Visualization The most frequent response to desired changes to software output was more graph options Visualization Only 12 5 percent of subjects agreed that displayed output data is clear to interpret and the third most frequent response to desired changes to the output was a greater capability to tailor the data C 23 Alter the cursor navigation button E 47 iy to match the map navigation icon ga Visualization 54 percent of subjects could not correctly identify the cursor navigation icon 71 percent of subjects could correctly identify the map navigation icon Some incorrect answers for each were the other icons title The functionality associated with these icons is very similar C 24 Standardize two button mouse functionality Multiple Attributes Follow the standard functionality for a two button mouse that most software developers utilize C 25 Incorporate an area re
137. rey amp Whitaker 2012 Providing recommendations that would aid TRAC Monterey in meeting this goal in future releases of the DaViTo product was the purpose of this usability test As the thesis was conceived and the experiment methodology began to be formulated it was necessary to break usability into several attributes most of which were derived from The Handbook of Usability Testing Rubin amp Chisnell 2008 The attributes are usefulness efficiency effectiveness learnability and satisfaction which are described in the matrix in Table 1 and provided the basis for the different sections of the surveys developed for the experiment contained in Appendix F In addition to the five attributes identified by Rubin and Chisnell categories for improvements and visualizations were used as well Attribute Description The degree to which a product enables a user to achieve his or her Usefulness goals and is an assessment of the user s willingness to use the product at all The quickness with which the user s goal can be accomplished Efficiency accurately and completely and is usually a measure of time The extent to which the product behaves in the way that users expect Effectiveness it to and the ease with which users can use it to do what they intend A part of effectiveness and has to do with the user s ability to Learnability operate the system to some defined level of competence af
138. ribute an in depth training product with DaViTo A thoroughly prepared and executed training plan could teach the basic use of DaViTo in several hours of training Final responses after session three indicated that users estimated they would require five to seven total hours of training to feel comfortable using DaViTo in the field and 50 percent of users agreed that a novice could use the software with just the training provided R dati Designator ats Attribute and Basis for Recommendation Description Develop a tutorial Learnability Currently there are no training C 1 based training product products for DaViTo beyond a basic PowerPoint for DaViTo slide show C2 Distribute a training Learnability There is currently a PowerPoint product with DaViTo slide show distributed with DaViTo Table 33 List of recommendations and the basis derived from analysis of the three common question survey results 2 Recommendations Based on Survey Usefulness Questions Designers did design a Help tab on the DaViTo menu ribbon however it currently has no functionality beyond providing a link to the OpenMap website help There is no DaViTo specific help and the user s manual is not available from within the software Linking the user s manual to the help tab on the ribbon bar to implement recommendation C 3 in Table 34 would correct this deficiency Without even seeing the user s manual 83 percent of subjects were
139. roducing the thesis topic the software designers and the authors Next the purpose of DaViTo its four goals and six main functions were introduced to give the participants some background information pertaining to the software s design functionality Following the introduction the user interface was introduced The user interface was divided into five functional areas as shown in Figure 2 Tutorial session one focused only on the Map Controls Data Set and Map and Data Display Area portions of the interface with the Shapefiles and Display portions to be covered in session two To begin the Data Set section of the interface was introduced and the participant was shown how to load various data sets into the software They were then instructed on how to manipulate them such as removing data sets or changing colors This section was taught first to facilitate having a data set open in the software for the remainder of the tutorial Once complete instruction moved to the Map Controls portion of the DaViTo interface Here the selectable drop down menus and software control buttons were each introduced continuing in the tutorial style with the participant following along on their own screen To introduce the controls they were each discussed and then some were used to manipulate the Map and Data Display Area Pause points were also inserted and the participant was informed that they could pause the tutorial and take additional time to explore the f
140. s constructed for the experiment Type and abbreviation description and an EiT Ol CAC WN aioli eke ree scious 6 43 List of questions for each session and their designation letter 43 Overall statistics for all graph types in session ONE P 45 Overall statistics for all graph types in session two 46 Overall statistics for all graph types in session three cee eeeeeseeeseeeeeeeeeee 48 Overall statistics for all graph types in all sessions 49 Key used by subjects to answer survey questions that were not free form feee AUPE Pe URIS PUREED AER ONES Pe ae emer NA 52 Common question one results n 52 Evaluation scale for common question two This question asked subjects how many more hours of training do you think would be necessary before you are ready to use DaViTo in the field for analysis eee 53 Summary statistics of the top five likes and dislikes for session one COMMON QUESTION THEE sisisi niiin 54 Summary statistics of the top five likes and dislikes for session two common question three PPnn 54 Summary statistics of the top five likes and dislikes for session three COMMON question eeee 666 55 Table of usefulness questions and associated statistics 56 Table of effectiveness questions and associated statistics 0 0 eee eeeeeeeee 58 Table of efficiency questions and associated statistics
141. s and applets needing geographic information Raytheon BBN Technologies 2005 OpenMap is the GIS portion of DaViTo that provides the mapping capability It is the root of the interface for users allowing interaction with the map and display of data on it Using OpenMap users have the ability to load and manipulate various Shapefiles datasets and maps in multiple formats JFreeChart was the second open sources software tool implemented in DaViTo JFreeChart was developed by David Gilbert of Object Refinery Limited and is a widely used Java chart library It allowed the developers of DaViTo to incorporate professional quality data representations into the project The ability to produce high quality visual output was necessary for both display when using the software and for import to planning briefs and training Statistical analysis was incorporated into DaViTo with the use of the R Project for Statistical Computing This open source software known as R is a language and 6 environment for statistical computing and graphics R provides a wide variety of statistical and graphical techniques and is highly extensible The R Foundation for Statistical Computing By integrating R into DaViTo the ability to conduct complex statistical analysis is given to the user and is only limited by their skill level B WHAT IS USABILITY When the term usability is brought up its actual meaning depends upon the context within which
142. s help While the goals of the software and tools available were helpful I felt that 130 Q Satisfaction_7 Were you satisfied with the performance of the software Please explain functionality could be improved on the map interface It is slow and clunky with non intuitive pan zoom selection buttons Support for mouse wheel scroll for zooming would be very helpful and reducing the delay between data selection and analysis would be helpful as well The software yes The chart interpretation no Not really it s very slow to respond to commands Yes very satisfied I accidently marked answers strongly disagree when I meant strongly agree This software is user friendly and not difficult to navigate around in The time required to load and unload shapes had quite a bit of a delay The software has some good output The type of charts needs to be looked at some charts are better than others and easier to read The speed of the program was a bit slow at times generating charts Series of clicks left or right and where you had to click was not straight forward I assume that the graphs can be made larger Latency of data display is not good If I was using this in an operational environment I would have to get up and smoke a cigarette to kill the time while waiting for data to load If I did not have a smoke I would just beat my machine with a hammer I m confident that once the data is displayed tha
143. s manual Sat_3 and using DaViTo with no assistance Sat_6 Overall this is indicative of a deficiency in satisfaction with their ability to use the software and is likely related to their current level of training and experience Again the user s manual was never consulted by any participants and there is no help available via the interface This deficiency will be addressed by recommendations in Chapter V 63 software Please explain Qu un Scale Frequency Mean Mode Median Question Designator z 1 3 I feel confident in my 7 3 abilities to use DaViTo for basic Sat z D E 3 3 analysis 5 0 1 3 I think I can 2 4 successfully finish basic tasks without 4 3 the user s manual 4 9 5 2 1 8 I think I can successfully 2 6 complete complex Sat_3 3 4 2 375 1 2 tasks without the 4 5 user s manual 5 1 1 0 I think this software 2 2 at 4 3 i 3 458 3 3 missions I have performed 4 9 5 2 1 2 I would recommend 2 2 this software to my Sat_5 3 12 3 167 3 3 command 4 6 5 2 1 3 I feel comfortable 7 ing DaViTo i eS Sat_6 3 8 2 67 3 3 without any 4 4 assistance 5 1 Were you satisfied Software latency lag with the Lack of help functionality performance of the Sat_7 Lack of graphing options and graphs difficult to interpret Table 27 64 Table of satisfaction questions and ass
144. sc iirin veis iesist ee te 63 6 VISUALIZATIONS sienien esineen RR iiis 65 7 Improvements sois sssiecaenticstacinad uatetea staat ccantis cea asdecrextevtee 67 3 Survey Results Summary 69 vV RECOMMENDATIONS 71 A RECOMMENDATIONS BASED ON TASK RESULTS 71 B RECOMMENDATIONS BASED ON GRAPH RESULTG ccscceeees 73 C RECOMMENDATIONS BASED ON SURVEY RESULTG ccccee00 74 1 Recommendations Based on the Three Common Questions 74 2 Recommendations Based on Survey Usefulness Questions 75 3 Recommendations Based on Effectiveness Questions ee sseseessece0 77 4 Recommendations Based on Efficiency Question6 0sscceee 78 5 Recommendations based on Learnability Questions 000 78 6 Recommendations Based on Satisfaction Question6 s 0000e0 79 7 Recommendations Based on Visualization Questions 00 80 8 Recommendations Based on Improvement Questions 00 82 D ADDITIONAL AUTHOR RECOMMENDATIONS sscssssssssssescsssseses 83 VI CONCLUSION AND FUTURE WORK sesessoesesocsossesccsssscssossesocsossesocsossoseossesossos 87 A SUMMARY 87 B FUTURE WORK issosicssis snssensacaastvis sopvsncaseadessbssnaeksasatsladssignssushassevdassvenbaatsasbess 89 APPENDIX A CONSOLIDATED RECOMMENDATION G scssscsssssssscssscsssssscsseeee 91 APPENDIX B ADDITIONAL TASK RESUILTS 97 APPENDIX C ADDITIONAL GRAPH RESULTS 111
145. seems to go more slowly than necessary Otherwise need more exposure to make a better judgment Likes 1 Easy to use 2 The tab identifies are descriptive 3 Quick to learn Dislikes 1 Couldn t remember the zoom in 2 The distance circle is deceiving 3 Map changes slowly Likes 1 Easy to navigate 2 Overlays 3 Quick reference Dislikes 1 Seems like it runs scrolls slowly 2 Unable to directly click on data points when using cursor3 No city or population center overlays Likes All of the data is displayed in one window making it easy to use Good introductory tutorial Tool for zooming in on certain areas of interest is useful Dislikes Lag time in updating the map Map controls would be more useful if on the map Zoom and pan functions could use more fine controls Likes Good Company Fun presentation Cool gadgets Dislikes Slow gui no map labels distance tool should have separate way points Likes Simple navigation toolbar saved views Dislikes Measurement tool graphics leave an artifact sometimes slow refresh rate Likes 1 Easy to use 2 Easy to understand the interface 3 Redundant ways to accomplish the same task Dislikes 1 No preloaded options for US layers geographical boundaries 2 Not clear on how to add create additional layers 3 No built in tutorials Likes 1 common menus and icons to other similar software easier to learn 2 simple plain interface 3 easy to get to the right
146. sion 1 4 6 2 4 5 Table 18 Common question one results The second common question required the subjects to self evaluate the amount of additional training they may need in addition to that already received to feel comfortable using DaViTo in the field This question was evaluated on the scale shown in Table 19 Figure 18 shows that subject confidence decreased between sessions one to two as subjects felt they needed 14 percent more training Following the final session 52 participant s confidence increased and the amount of additional training thought to be needed decreased by 21 percent to its lowest level This translates to a total of five to seven hours of training are thought to be required in addition to the two 20 minute tutorial sessions This response may give TRAC Monterey guidance on the development of future training tools and the amount of additional training needed For full analysis results see Appendix D 0 1 Hours 2 4 Hours 5 7 Hours 8 10 Hours 10 Hours 1 2 3 4 5 Table 19 Evaluation scale for common question two This question asked subjects how many more hours of training do you think would be necessary before you are ready to use DaViTo in the field for analysis All Sessions How Many More Training Hours Needed 3 9 3 7 3 2 3 3 f 3 1 l Average Response 29 Hours Average Response 27 i 2 5 1 2 3 Session
147. sion three tasks The first was identified by the excessive mouse clicks when attempting to generate a graph within a user defined polygon This deficiency is related to the attribute of efficiency The quickness with which a user can complete their goal is reduced when time is wasted awaiting program response Since the product does not react in the expected manner effectiveness is also reduced The survey results will also show that these issues decreased the attribute of user satisfaction The second deficiency was identified by gaze concentrations indicated on heat maps in the improper areas This indicates a lack of functionality that the user expects in the header menu bar ribbon and is indicative of reduced effectiveness of the DaViTo software Recommendations for improvement are discussed in Chapter V and a consolidated list in Appendix A C GRAPH PERFORMANCE RESULTS A series of data representation questions followed the task completion sections for all three sessions This portion of the experiment was used to evaluate the ability of the participant to interpret graphical output similar to that already used in DaViTo or able to be used The purpose was to ensure that data is being represented in the best manner for the user to consistently interpret it quickly and accurately It should be noted that after session one a brief training session was given by the experiment administrators followed by a review of their performance The purpos
148. sk requiring a full series of actions from loading a data set to generating a graph that could be used to answer a theoretical commander s question The participant was directed to determine the total number of IED attacks for 2010 in the most northwestern provincial boundary of Afghanistan Essentially this task began with a blank slate and required loading a dataset and Shapefile then navigating the Map and Data Display Area to Afghanistan by a means of their choice and generating and opening a specific graph Subjects were then required to answer the question about that graph If a subject could 18 not complete the tasks he or she was unable to answer the question The final task was a trend analysis question that also required the completion of all of task number ten to answer Answers to questions ten and eleven were recorded by the administrators of the experiment Following the completion of task eleven all work within DaViTo was complete The participant then proceeded to the final series of graphing questions with answers recorded by EyeWorks Record The final portion of the session consisted of an extensive questionnaire of 50 questions composed from the seven usability factors 6 Experiment Scheduling Considerations The overall design of the experiment was for it to take place over a three week timespan for each participant within a six week window of open scheduling available This gave subjects the flexibility to choose a fo
149. slikes Like playing with the software I did not like the lack of progress display Dislikes Some menu options are in counterintuitive locations opening Shapefiles and map layers I couldn t remember how to create one of the graphs and searching the usual areas didn t give any clues Once the graphs are readily available the overlay onto a geographic area is convenient Likes Simple instructions Tasks were well defined Dislikes Country not labeled wrt task given I didn t know where Kenya was 2 No fantastically narrated training session by LT Cutler3 Couldn t figure out how to change the display of chart from individual event line to cumulative bar 124 Session Three Question Three What were your likes and dislikes with your third exposure to DaViTo Likes Similar interface as other C2 systems in the Army but difficult to learn the first time Dislikes The long load times with any indicator of what the computer was doing led to some mistakes and frustration Overall seemed slow For the geographically challenged adding location names would be helpful Such as labeling the nation names Dislikes Slow for loading Dislikes It sometimes took a while to load maps And after it loaded the maps I was unsure that it had completed This may just be due to a slow processor Additionally the response to my right click was sometimes unresponsive or took too long So it didn t feel right and I would hav
150. subjects to improve the visual representation of data on the Map and Data Display Area R dati Designator S i cin Attribute and Basis for Recommendation Description Improve software C 11 processing response time and stability Efficiency 67 percent of subjects disagreed that the software was stable and not prone to crashing Include geographic Efficiency Subjects reported this as the top C 12 labels in the Map and deficiency that made visual representations easy Data Display Area to read Make data types Efficiency Subjects reported this as a deficiency C 13 within a data set that made visual representations difficult to read sortable by color Table 36 List of recommendations and the basis derived from analysis of the efficiency portion of the survey results 5 Recommendations based on Learnability Questions Most questions in the learnability segment of the final survey reflect positively upon the training developed for the experiment The majority of subjects believe that with this training most people could use DaViTo they could complete the final session tasks again in one month and that they could teach a user the basic functionality of 78 DaViTo Despite this a majority also believe more training products are needed Because of these results Table 37 lists recommendation C 14 identical to C 1 to endorse a tutorial based training program and C 15 identical to
151. t I would be able to use the tool positively Table 57 Table of free form responses to question Use_2 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_3 How would you improve DaViTo Please explain There is something off with the mouse selection of individual territories associated with lag and the need to deselect an area prior to the next selection Perhaps a waiting icon hourglass could be displayed to let users know when to wait prior to entering the next command would be appropriate More hands on training Executing the tasks that are necessary to use the software efficiently I would add chart generation to the toolbar Improved lag on map controls status icons for processes Progress icons more training Find the most commonly used geographic analysis software out there that resembles DaViTo and organize the GUI to resemble that as closely as possible See previous comments Improve the lag problem I would improve DaViTo with a tutorial by an onsite representative to teach the functionality of the program I would also leave a basic video tutorial to ensure that the use of the system was driven home by the user If you hover over an item it should pop up quickly telling you what it is It seemed to be pretty delayed I was still doing a lot of trial and error while manipulat
152. t Survey Please check or answer the following questions about the experience you had using the DaViTo software Please use the following scale for your responses Satisfaction refers to perception feeling and opinion 1 Ifeel confident in my abilities to use DaViTo for basic anabysis 0i n2 a3 og n5 2 think cam successfully finish basic tasks without the user s manual ni o2 o3 o4 as 3 think I can successfully finish complex tasks with the user s manual oi 02 os o4 o5 4 think this software would add value to missions that have performed oui n2 o3 a4 o5 5 would recommend this software to my command oi o2 as o o5 6 tfeel comfortable using DaViTo without any assistance oi a2 o3 o4 os 7 Were you satisfied with the performance of the software Written response Visual refers to representation of output 1 The way in which output data is displayed is clear to interpret g1 o2 33 o4 os 2 What more would you like to see in the output of this software Written response 3 Graphical Representation question 1 147 Quantitative amp Subjective Questions 4 5 6 Actual draphs shi under const cithay wll includ s bar Graphical Representation question 3 2 ine scodder N Sihine Graphical Representation question 4 Cr 328 Usefulness refers to ability to achieve goals case of use and willingness to use 1 The tasks were easy to complete oi o2 os o4 o5 Wha
153. t districts and boundaries for different countries areas The ability to select a specific region and then it provide detailed charts on the data is a great benefit as well Ease of use 133 Q Improve_5 What did you most like about DaViTo Please explain Fairly intuitive Gives a good view of the whole data set at a glance Clicking on a data set and seeing data Integrated data analysis Navigation of the map and the ability to bring up the loaded data about the provinces Nothing really excited me Easy to use Simple easy user interface I am not usually fast with learning computer and or software programs and the fact that I learned some basic functionality with this program in a short amount of time speaks well about its usability Graphical display Ability to visualize data Table 60 Table of free form responses to question Improve_5 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Improve_6 What would you change about the color scheme page layout Please explain Slightly greater contrast might make it more comfortably viewable though it was sufficient For daytime use the color scheme was sufficient The major change would be for detailed use having more area relief This would be useful both on the move and in planning I would know how to plan for movement times Nothing No
154. t of specific questions This prevented generic responses to general questions Questions concerning improvements and some of the results are contained in Table 30 with complete data in Appendix D The question Improve_1 asks about the user s manual While most subjects answered neutrally there was a full spectrum of answers despite the manual never having been consulted When asked in Improve_9 about their preferred graphs the majority of users chose the SBF by a large margin which is concurrent with it exhibiting the top performance overall Deficiencies identified by the subjects in the free form questions will be addressed by recommendations in Chapter V 67 Table 30 Improvement Scale Frequency Mean Mode Median Question Designator 1 2 User s manual is 2 1 easily Improve_1 3 17 3 3 3 understandable 4 3 5 1 1 0 More training aids 7 6 as saint or melo Improve_2 3 4 3 5 4 4 effectively learn to use the software 4 10 5 4 How would you Correct the lag issues improve DaViTo Improve_3 Generally improve the graphs and interaction Please explain Add a progress icon What was the most Lack of on screen help available frustrating feature of Improve_4 Standardize mouse button functionality DaViTo General Lag and charts What did you most Ability to display data geographically like about DaViTo Improve_5
155. t was the most difficult task Written Response The terminology for buttons and commands was clear and understandable o4 oz a3 D os W were an analyst would use this tool for field analysis Di 52 a3 n as A computer novice would be able to use this software with the training provided oi o2 c3 a4 as A computer novice would be able to use this software with just the user s manual oi n2 o3 o4 o5 Efficiency refers to quickness to accurately accomplish goal layout time benchmark and navigation a The organization af the screen matched my expectation of button location bi n2 o3 n os Similar icons and activities are consistently placed pl o2 os a4 os 148 Quantitative amp Subjective Questions 3 Due to my experience with computers navigation of DaViTo was intuitive ol n2 n3 os os Was the visual representation easy to understand Written Response The color scheme for the software was appropriate fap o2 D3 o4 as The text within output graph cal representations was easy to read and match to data oi a2 o3 o4 os The software header bars and dropdown lists made navigational sense o1 o2 33 o4 os The software was stable no freezing oi o2 o3 p4 os Effectiveness refers to software behaves as expected workflow error rate mouse click bench mark 1 I understood where was at in the saftware as progressed through a task ot o2 os a4 os The mouse interacted with the software
156. te was the backbone of the experimental design and is an all inclusive research software package for users of eye tracking The software has three functional areas Design Record and Analyze all of which were used The Design component was used to format the experiment from beginning to end for each session All tasks questions and surveys were built into the Design shell The Record component ran each session The script was started at the beginning of each session and within the Record window all tasks were prompted answers collected and surveys conducted This division of the software package was also used to capture not only eye movement but launch the experiment partition portions of each session for ease of data analysis and record mouse clicks It was also useful for real time observation of the subjects view and where their gaze was directed during the experiment as well as providing a video recording of every session EyeWorks analyze was then used to provide heat maps and graphs of gaze data display mouse clicks and output survey answers to Excel 3 Physical Setup The arrangement of the physical hardware shown in Figure 1 was identical for all three sessions The exception was item 3 in Figure 1 the laptop used for the video tutorials in sessions one and two which was removed prior to session three The layout of all hardware components shown in Figure 1 are described in Table 2 13 Figure 1 Birds eye view di
157. tement of the goals TRAC Monterey stated when developing the Data Visualization Tool DaViTo Their intention was to develop an open source tool that would be easy to use for someone unfamiliar with statistical methods to produce useful visualizations and analysis of spatial temporal data and be able to present that output in a way that could be understood by ground commanders Evangelista Henry Buttrey amp Whitaker 2012 The overall purpose of this thesis is to determine how DaViTo in its current state needs to be improved to meet the usability goals it was originally developed to fulfill B OBJECTIVES The goal is to provide concrete and substantive recommendations to improve the usability and workflow of the DaViTo software Research for this thesis began with two main objectives both with subsets of secondary objectives The primary objective consists of evaluating the usability and efficiency of the DaViTo to determine if the workflow to complete a task matches how a user with 1 minimal training thinks the workflow should be carried out The following subset of secondary objectives was used to complete the first primary objective Identify speed and accuracy to complete tasks by investigating mouse clicking trends task completion times and number of correct answers Identify misunderstandings of the software interface such as button symbology names mouse overs or locations Utilize eye tracking characterist
158. ter some predetermined amount and period of training y e Refers to the user s perceptions feelings and opinions of the Satisfaction product Raw recommendations from the subjects with regard to ways to Improvements improve the software Refers to the user s interpretation of the user interface and used to Visualizations determine if it is inherently usable in its current state Table 1 Matrix of usability attributes and their descriptions Rubin amp Chisnell 2008 THIS PAGE INTENTIONALLY LEFT BLANK 10 Hl METHODOLOGY A OVERVIEW AND OBJECTIVES The overall objective of the experiment was to collect objective and subjective data from participants using the DaViTo software that could then be transformed into actionable recommendations for improvements to the DaViTo software Data collection was conducted through a number of different methods The first was through a demographic survey contained in Appendix F Another method was a set of questionnaires at the conclusion of each session with the final questionnaire being very extensive also contained in Appendix F The questionnaires contained questions with selectable multiple choice responses as well as fill in the blank where subjects could give free form opinionated answers Data was also collected by evaluating subject s ability to evaluate graphical representations in an accurate and timely manner The final method of data collection utilized auto
159. th the mapping and spatial analysis tool ArcGIS and no subjects had any knowledge of the DaViTo software Subjects Age years Min Max Average Mode Median Standard Deviation Table 3 Breakdown of subject population by age 20 Number of Subjects By Rank Table 4 Breakdown of subject population by rank Number of Subjects By Service Army Navy Marine Corps Air Force Retired Table 5 Breakdown of subject population by rank Computer Skill Level Novice Intermediate Table 6 Subjects assessment of their own computer skill level 21 Subjects Hours Per Week Using a Computer 1 Hours 0 5 1 2 Hours 6 10 0 3 Hours 11 15 5 4 Hours 16 20 6 5 Hours 21 12 Average Response 4 17 Mode 5 Median 4 5 Standard Deviation 1 03 Table 7 Subjects reported weekly computer usage Preferred Operating System Mac 4 Windows 19 Other b hee Table 8 Summation of subject populations preferred operating system E EXPERIMENT PROCEDURE The experiment consisted of three sessions and the procedure was common for all three sessions The difference between sessions was content as covered previously For the subjects there was no difference between experimental setup and procedure except for in sessions one and two they controlled the playback of the tutorial whereas in session three there
160. the goal was to test the usability of new users rather than analysts so advanced functionality of the software such as outputting data to R for advanced statistical analysis is not evaluated Due to time 2 constraints study participation is limited to 24 subjects at approximately three hours each This sample size is large enough to thoroughly evaluate resultant data and sufficient for potential loss or corruption of individual data sets yet small enough to be tested within research lab and personnel time restrictions Eye tracking is a key component of the usability testing of this thesis and is not known to have been frequently used previously in this area of research While the eye tracking itself and the statistics associated with it are of interest the scope of this thesis limits the use of eye tracking results They were used solely for developing recommendations for software improvement in the areas of workflow analysis and usability There is a large amount of data available from the EyeWorks software such as blink data saccades and cognitive load based on pupil contraction Eye tracking was only a component of the data evaluated and as such only scan patterns and fixations were used in the end The outcome of the eye tracking performance is not evaluated outside of these areas D THESIS ORGANIZATION Chapter I Introduction This chapter describes the purpose and overall objectives of the thesis as well as identifies the
161. thing Nothing Nothing No Nothing I would add actual colors vice the drab grey I would also have areas of high importance be color coated differently than the surrounding areas to represent their importance Too bland I don t need a lot of colors but if I had to look at it all the time I d like it to be easier to customize The default colors scheme page layout should be a little less bland I get bored just looking at it Less shades of blue Given that used the tool for less than 2 hours I cannot properly recommend a color scheme or page layout No Nothing of note Kind of bland compared to most applications A little more vibrant color scheme and 134 Q Improve_6 What would you change about the color scheme page layout Please explain larger fonts might make it more attractive Compared to dynamic mapping software like Google Earth this is at the lower end of the spectrum not like Apple user interface that slides the map along and incorporates inertia It s fine the way it is perhaps the default color when loading a data set could be something of high contrast e g red to start with instead of dark grey Use colors to depict certain areas on the screen for better navigation Layout was good but utilizing a 3d map like Google Earth would be far more intuitive and helpful No issues Nothing Nothing None Nothing I think it is fine
162. tions to TRAC Monterey that could be used to improve the usability of the tool through future revisions Usability has many definitions but in the context of this thesis it can be best described as the quality of a system with respect to ease of learning ease of use and user satisfaction Rosson amp Carroll 2002 As a tool for the analysis of data with both spatial and temporal components DaViTo has the potential to be instrumental Increasing the usability of DaViTo by improving upon all of its attributes will enhance that potential The experiment itself was developed to rapidly train and familiarize participants with DaViTo Once complete gathering data from subjects was the next step With 24 volunteer participants over 70 man hours were spent training testing and surveying them Data was collected through eye tracking while conducting tasks inspecting graphs and answering numerous survey questions In order to compartmentalize collected data for analysis it was split into three groups The first was the collection of data through eye tracking This data was useful for determining where subjects are focusing their attention while trying to complete a 87 task The software used EyeWorks Record also collected mouse clicks and tracked the time elapsed from task to task The second method of data collection was by presenting participants with multiple graphs asking quesitons about them and then analyzing their respo
163. tracking software then watch a familiarization video of the Da ViTo software During the second training and familiarization session the participant will watch a video that walks through commoe tasks DaViTo was designed to handle The third session will consist of the experiment where the participant will be asked to complete tasks in DaViTo similar t0 the ones covered im the training and familiarization sessions During this session the participants eye gaze eye dwell time and mouse and keyboard inputs will be recorded for usability analysis When the participant has completed the DaViTo tasks they will be asked to complete a survey on DaViTo about the general satisfaction of the tool and opinions on visual formats The experiment will take place one week after the second training am familiarization session and last for approximately 60 minutes We will provide information on the study results if desired The entire study showld take approximately 3 to 4 hours Any information will be used only for this research endeavor Voluntary Nature of the Steady Your participation im this study is strictly voluntary If you choose to participate you can change your mind s any time and withdraw from the study You will not be penalized in any way or lose any benefits to which you would otherwise be entitled if you choose not to participate in this study or to withdraw The alternative to participating im this research is to not participate in this research
164. ts as well and should be utilized Figure 17 shows the performance of all formats for each question Six of the seven questions that used the SBF graph are clustered in the upper right hand corner showing consistent top performance The inconsistent outlier for the SBF was question E at 42 percent which was one of the higher performing questions for the LCF at 75 percent accuracy The CCF also performed well on question E at 87 5 percent This indicates that no one graph type is appropriate for all data representations or questions All Sessions Accuracy correct Time sec CCF 70 83 47 78 SBF 73 38 31 93 LCF 49 07 56 16 SLMF 47 69 40 99 SAF 62 27 35 06 Table 16 Overall statistics for all graph types in all sessions All Sessions Time vs Accuracy Accuracy correct 40 60 A Time sec A uw Sy S S oN Figure 16 Plot of all three sessions times to answer versus accuracy of answer Note the decreasing time on the Y axis Top performance is in upper right corner 49 All Questions Accuracy correct o 10 20 30 40 50 60 70 80 90 100 X SAHK SRF D 20 rere X SIMF3 A icra J X SAFC x sume gece cerr SREB i X 5AFD X suMrc s K SARA mut X SIMIT m Esee an ATTN x Aa sarg cer G E SBE E X SLMIA SAFF t ire ccc X suMrG L Corr i A LCD Ee X SUMEF emi ICEG 80 A TUK A LCFF 120 F
165. ttributes listed in Table 1 as well as categories of improvements and visualizations All questions are contained in Appendix F Unless otherwise noted questions that were not free form text entry by the subject used the number key shown in Table 17 51 Strongly Neither Agree Disagree Agree Strongly Agree Disagree nor Disagree 1 2 3 4 5 Table 17 Key used by subjects to answer survey questions that were not free form text 1 Analysis of the Three Common Questions from Each Session Three standard questions were used for the first two sessions and were contained in the final session questionnaire These questions were standardized across each session to allow for comparison The only exception is question one for session three as it included no training The first common question asked the subjects to evaluate the training during that session The session three question was not evaluated against sessions one and two because there was no training and will not be discussed The training was evaluated on a five point scale from very good to very poor similar to Table 17 The results of common question one for sessions one and two are shown in Table 18 This high valuation may give TRAC Monterey guidance on the development of future training tools to distribute with the software Full analysis can be found in Appendix D Average Response for Question 1 How Session would you rate the training ses
166. uld require daily use in order to not forget steps and functions Somewhat Like using my desktop at home Yes simple user interface Simple yet not always intuitive Mostly intuitive More use would have made it fully intuitive Table 54 Table of free form responses to question Effect_3 for all 24 subjects in session three Comments taken directly from the EyeWorks software with no modification Q Eff_4 Was the visual representation easy to understand Of geographical data and selections yes Of numerical data representations no Yes Something to consider is my level of prior training I was in the 4th Infantry Division in 1998 as we were testing FBCB2 So I have had a lot of exposure to both that and BFT I understand most of this set up from those two systems The charts configuration was new to me Yes Sorting of the data points within a data set with different colors could be helpful Yes Yes for the most part Some option locations were inconsistent with other similar mapping GUIs That makes searching around for it difficult Yes for the most part The charts and lack of geographical labeling were really the 127 Q Eff_4 Was the visual representation easy to understand major problem Yes Yes If we re talking maps then yes If we re talking graphs NO The general display wasn t hard but I wouldn t say it was really easy to und
167. unctionalities just discussed if they desired 16 Figure 2 Interface introduction slide from Tutorial Session One This slide shows the breakdown of the DaViTo interface into five functional areas 4 Training Session Two The second session video tutorial began with a brief review of the initial session followed by more advanced instruction This second tutorial was slightly shorter at only 15 minutes and 45 seconds Following the introduction a brief review of the Map Controls Data Set and Map and Data Display Area portions of the DaViTo interface was conducted with an opportunity for the user to pause the tutorial to interact and refresh themselves with any portions of the interface that they felt necessary Once complete the focus shifted to employing Shapefiles manipulating statistical output and conducting statistical analysis all within the Map and Data Display Area While this required use of the previously covered Map Controls and Data Set areas of the interface it made frequent use of the Shapefiles and Display regions The participants then loaded a similar dataset 17 to the previous session to work with through the first several minutes of the tutorial The first new technique taught was loading a Shapefile with predefined polygons centering the interface on that area of the screen and then viewing and renaming graphs populated in those polygons The subjects were then instructed on methods for making user defined
168. ur week period in which to conduct their first session and still have two weeks following that for the remaining two sessions Subjects were to schedule their sessions no less than six days and no more than eight days apart The goal was seven days between but plus or minus one day was allowed for flexibility Finally sessions were available throughout the Monday through Friday workweek from 0700 to 1800 daily D SUBJECTS In accordance with the approved IRB all subjects were taken from the pool of personnel of the Naval Postgraduate School NPS This included the student population civilians and foreign officers with only two conditions The first condition was that a participant could not be colorblind due to the coloring of the charts used in DaViTo The second required that the individual not be involved in the development of the DaViTo software Participation in the study was completely voluntary and it was made clear during recruitment and throughout the experiment that the subjects were free to stop at any time as well as the fact that there would be no compensation for participation The demographic survey for the subject backgrounds is contained in Appendix F The experiment sessions were scheduled to be conducted over a six week period on weekdays between 0700 and 1800 When scheduling their three sessions subjects 19 were directed to sign up for one hour blocks in three consecutive weeks preferably seven days apart but
169. vations for all subjects Task four directed subjects to create a user defined polygon and display the chart Counts correlate to the number of subject fixations in each region shown in Figure 9 107 Graph of final session task five cumulative counts of gaze observations for all subjects Task five directed subjects to close and delete the chart generated in task four Counts correlate to the number of subject fixations in each region shown in Figure 21 107 Graph of final session task six cumulative counts of gaze observations for all subjects Task six directed subjects to hide the Map Controls ribbon bar Counts correlate to the number of subject fixations in each region SHOWMAN Figure 22 ias cSccacssegansdacassuacagveghesuanenconswedavas 108 Graph of final session task seven cumulative counts of gaze observations for all subjects Task seven directed subjects to navigate to Iraq without the use of the Map Controls portion of the interface Counts correlate to the number of subject fixations in each region shown in Figure 23 108 Graph of final session task eight cumulative counts of gaze observations for all subjects Task eight directed subjects to go back to the previous Figure 34 Figure 35 Figure 36 Figure 37 Figure 38 Figure 39 Figure 40 Figure 41 Figure 42 Figure 43 Figure 44 Figure 45 Figure 46 view Counts correlate to the number of subject f
170. with no modification 138 APPENDIX E TASK WORKSHEET Appendix E contains the list of tasks for the final session as shown in Table 65 This form was used by the experiment session proctors to track subject performance The mouse clicks and times for each task were recorded by EyeWorks The proctor monitored and verified the status of satisfactory task completion Participant Number Completed Task Clicks Time Y N 1 Open WITS csv 1 A Left click button in Data Set 1 B Left click WITS csv 1 C Left clicks WITS csv open button 1 D Left click load button Total 2 Load Kenya_Roads Shapefile 2 A Left click button in Predefined Polygons box 2 B Left click Shapefile Kenya 2 C Left click open Total 3 Go to Africom view 3 A Left click Views 3 B Left click Africom Total 4 Create a rectangle chart on the data Point in the southwestern most part of Kenya and display the chart 4 A Right click on screen to create 4 B Left click on rectangle 4 C Left or right click to draw rectangle 4 D Left click on screen 4 E Right click on rectangle 4 F Left or right click on Display Chart Total 139 5 Close and Delete Chart 5 A Left click red x Button 5 B Right click on rectangle 5 C Left or right click on delete chart or delete all charts Total 6 Hide Map controls 6 A Right or left click on control t
171. within a window of six to eight days between sessions Almost all subjects signed up for sessions with seven days of separation between them and all sessions were scheduled with the six to eight day gap between sessions Every subject s sessions were completed in five weeks and the last week allotted for experiments did not need to be utilized The experiment was limited to 24 subjects in order to have an adequate amount of data for statistical analysis but to also have enough time to conduct 72 hours of sessions Recruitment was conducted through mass e mail word of mouth and via classroom briefs The IRB approved recruitment document is contained in Appendix F The goal of 24 subjects was met through the recruitment efforts with a fairly diverse pool The average participant age as shown in Table 3 was 34 92 consisted almost entirely of O 3s and O 4s presented in Table 4 and the dominant service seen in Table 5 was the U S Navy The demographic survey responses also permitted the subjects to assess their computer skill level and average usage Nearly one third of subjects rated their skill level at intermediate as seen in Table 6 while Table 7 shows that half the subjects indicated usage in excess of 21 hours per week Other items of note from the demographic survey include a majority of subjects having a preference for the Windows operating system shown in Table 8 seven subjects had some unspecified amount of previous experience wi
172. y of answer Note the decreasing time on the Y axis Top performance is in upper right corner 2 Analysis of Session Two Graph Data In session two the participants were asked the same series of questions listed in Table 12 about each of the five graph types identified in Table 11 Two identical questions were asked about each graph type similar in type to those of session one Subject answers were evaluated for time to answer and accuracy of their answer Again 45 several subjects exhibited frustration with some of the graph types due to the amount of data contained or the format in which it was displayed Upon completion of session two there was no review of performance or explanation of graph formatting In session two all chart types showed improved accuracy performance except the SBF which exhibited a significant decrease as seen in Table 14 although it had the shortest time to answer Figure 14 shows that all types except the LCF were clustered near the upper right corner with the CCF resulting in the highest number of accurate answers in commensurate time with the less accurate SBF The format with the second best accuracy was the SAF with 75 percent accuracy and approximately five seconds more required to answer than the CCF While on average it had worse performance than the CCF it should be noted that on question D from Table 12 it had the highest performance and lowest time to answer Converse to this on question E
173. y the development community The final goal was that the software would be easy to use for someone unfamiliar with statistical 5 methods and Geographic Information Systems GIS Evangelista Henry Buttrey amp Whitaker 2012 This goal was the driving force for the conduct of this thesis TRAC Monterey s approach was to conduct a usability test of the software package after its initial release to give recommendations for improvement that could be applied to a future revision of DaViTo that would better meet this goal It was identified that by making the tool more usable it could be utilized further beyond the use of an Army analyst such as in deployed environments and potentially by other branches of the armed forces DaViTo is an open source program that is actually a compilation of three open source software packages The open source aspect was a goal not only to allow for customization but also for fiscal reasons Similar commercial systems such as ArcGIS are very powerful but widespread use is not possible due to the price Additionally commercially available products have the ability to perform some of the tasks that DaViTo does however none provide the merger of functionality that DaViTo possesses As such DaViTo was developed using three open source software tools The first OpenMap developed by Raytheon subsidiary BBN Bolt Beranek and Newman Systems is a Java Beans based toolkit for building application
174. yeWorks Record session to answer similar questions about a number of graphs Their answers were saved by EyeWorks Record Participant answers were later imported to Excel and used in multiple analyses with Excel and JMP The strategy was to evaluate the resultant data for efficiency with time to complete and correctness the parameters of interest 24 3 Questionnaires The participant answers to the surveys were recorded by EyeWorks Record and then imported to Excel for analysis The multiple choice questions were evaluated on the basis of average mode median and frequency The free entry questions were evaluated for similar responses and used to formulate participant recommendations for improvement 25 THIS PAGE INTENTIONALLY LEFT BLANK 26 IV RESULTS AND DATA ANALYSIS A DATA PREPARATION Over 4 5 gigabytes of data was collected throughout all three subject sessions One challenge was determining how to partition the data such that it could be analyzed and presented in a logical and efficient manner Another was which data to use particularly the eye tracking output of the EyeWorks software as there are a number of analytic options and some data was not directly relevant to this thesis or its scope The purpose was to provide actionable recommendations to TRAC Monterey to improve the usability and workflow of the DaViTo software As such not all resultant data was analyzed and will not be discussed here The sess

Download Pdf Manuals

Related Search

Related Contents

ワイヤレスビジュアルプレゼンター MO-1w  Craftsman 10-Drawer Use & Care Manual  取扱説明書 - Panasonic  Traduction et adaptation Daniel Garance , Emmanuelle Stachurski    User's Guide ver. 1.3 EN  PAR LED 1418 Z IP - KARMA ITALIANA Srl  Manual  OWNER`S MANUAL    

Copyright © All rights reserved.

• https://telegra.ph/Wire-Size-Calculator-From-Voltage-Drop-ac7518b9-11-16
• https://telegra.ph/Calculat-Wire-Area-From-Voltage-Drop-fe98d213-11-16
• https://telegra.ph/Concrete-Volume-Calculator-for-Curbs-Gutters-69747c7f-11-16
• https://telegra.ph/Square-Slab-Concrete-Volume-Calculator-0c695c7f-11-13
• https://telegra.ph/rebar-weight-calculator-46c7a59f-11-07
• https://telegra.ph/Wire-Size-Calculator-From-Voltage-Drop-d3629810-11-17
• https://telegra.ph/Calculate-Length-from-Voltage-Drop-f9cb4414-11-18
• https://telegra.ph/AC-Single-phase-Voltage-Drop-Calculator-1cfa136c-11-18
• https://telegra.ph/DC-Voltage-Drop-Calculator-14f8324c-11-18
• https://telegra.ph/Wire-Size-Calculator-From-Voltage-Drop-71ac8695-11-18