InTouch - Computer Science Technical Reports
Contents
1. Se aaas Correcting your mistakes mistakes A PA 54 4 Experienced and inexperienced users needs are taken into 2 8 consideration As a free form information manager InTouchis a free form information As a free form information manager InTouchis InTouch is 3 Ha ae When compared to other information managers you have e 4 used InTouch is 462. not clear what item refers to Interaction user task not Medium Allow user to highlight mapped to the system several records in Records even though functionality List and delete them to support task is present 32 9 i e user can delete individual records but cannot delete multiple records in a single step Names Labels Low Remove from label misleading name label 24 9 25 Edit Problem User can easily find all instances of a given string but cannot easily replace instances of the string User would like to spell check information User wants to use a different font e g bold for portions of the record text Problem User has trouble finding Find in the menubar User can t find a record that is in the file because he doesn t realize that find only searches the current group Potential Solution Error Classification Discussion Functionality missing Lo 0 Modify find dialog to include replace and replace all operations Functionality missing 35 1 Either provide or allow the user to plug in a spell checker functionality 35 1 Useful if you are using InTouch as something other than an address book Functionality missing functionality 35 1 w Importance High Add support for different fonts to InTouch Capability would allow delineating things with fonts it is very desirable when you re using InTouch for somet
3. or rename option to New File Change label to Create Potential Solution Importance High Move Open to a location more intuitive to Mac users i e the File menu Order the buttons in the Oper dialog box in the same manner as the other dialog boxes Low Recent Files Problem User has trouble finding files that were recently opened by InTouch No directory information for files is given If two files with same name were recently used user has no way to distinguish InTouch data files do not appear under Recent Documents in the menu Missing shortcut 34 4 Error Classification Discussion Importance Low Menu organization inappropriately grouping unrelated choices in a menu 19 9 Metaphor doesn t make sense to user 32 4 User does not understand why all InTouch items are in the InTouch menu Name not meaningful enough 24 7 13 Potential Solution Move Recent Files to a location more intuitive to Mac users i e the File menu Preferably include list of recent files at the end of the menu before quit separate with a bar from standard items and quit option Add directory information Ensure InTouch data files appear as recent documents Save Problem User has trouble finding save file command Save in the menubar User confused when asked if wishes to save changes in a file that has not
4. and possibly the 7 InTouch is a product of the Prairie Group P O Box 65820 West Des Moines lowa 50265 8 Originally installation was included in the task list However as the installation process is straightforward but time consuming it was not included in the final task list Prairie Group developers summarized and relayed customer complaints to usability testers 10 5 10 minutes is Prairie Group development goal 11 First half hour is Prairie Group development goal 32 source of customer difficulties potentially leading to recommendations for redesign of the problem area 33 User Profile A total of 5 participants will be tested A total of 6 people will be selected 5 participants and 1 alternates that satisfy the following requirements Characteristic Range Frequency Distribution Macintosh Experience InTouch Experience 100 Novice Intermediate Method The test will consist of 1 A performance test of benchmark tasks plus some representative tasks designed to gather InTouch usability information through remote evaluation 2 A user preference questionnaire designed to determine how successful InTouch was at satisfying user expectations of a free form information manager It will consist of the following three stages 1 General Introduction Participant will be given a short scripted introduction to the test which explains the purpose and objectives of the test Participant will be introduc
5. Add Cancel button to Layout dialog box Potential Solution Simplify icons to facilitate understanding and add labels Place Layout pulldown beneath mechanism to select item to be printed May want to re label Users frequently referred to layout as format Differentiate between the necessary and secondary options Consider placing secondary options at a lower level Record Operations Delete Problem User cannot undo record delete User inadvertently deletes a record Had highlighted a portion of the record and hit the delete button assumed delete would only apply to portion highlighted In the delete confirmation message user asSumes item refers to the portion of record highlighted not the entire record User must delete individual records to delete a group of records Delete on record delete button indicates a dialog box should pop up but dialog box only pops up when Confirms Delete option is active Error Classification Importance Potential Discussion Solution User Action Reversal High Undo action should apply user unable to undo an to InTouch record action 37 1 operations Recommend that all Edit actions be made functional for records Names Labels Problem High Change label to Delete 24 7 wording not Record meaningful enough Feedback Message High Change item to entire Problem 26 5 feedback record message vague
6. Allow users to highlight multiple records in record list and then cut and paste If a user highlights items in the record list the operation applies to the entire record if highlight items in other fields operation only applies to object being manipulated in that field Sort Problem User assumes when he sorted the ALL group all records in all groups are sorted User confused that sort dialog box changed since last use User wants to cancel sort but unsure if done will cancel without doing sort User would like to be able to sort several groups simultaneously Reminders Problem User locates calendar by trial and error User sidetracked by Dateview button and highly annoyed when it points to a product that is not installed Views button as an advertisement and annoyance Did not understand the Each group has it s own Potential Solution Error Classification Discussion Importance Medium Feedback message should make it clear that only one group All was sorted interaction 32 3 Feedback message general rather than specific 26 5 sort order Not apparent to user 32 3 Missing button 20 2 Document preferably on line Low Low Add Cancel button Low Modify selection method to allow multiple groups to be selected High User can sort multiple groups by selecting sort repeatedly but cannot sort multiple groups in one step 32 9 Po
7. Complex Group Search 0 Task Completion Data 7 44 maro tO Save a Fle Undera DifeeniName e Pe LT pe m7 Finda Specie Record A A A Delete Records Berga R Rar IEEE ort a File v wv Saans EEE Mate a NeW 599 CS Print a Fax Cover Pinta Fax Raa ooo Esport Records Belonging oaro IA Find Records Using Simple rop Sen foe 07 Find Records Using a Complex Group Sear Y vfo meor baa tomare OO jee o GosethenReOpennfoush H Y Y 0 7 Percentage of Tasks Successiuly Completed ay Suec 789 992 948 8 100 Task Timing Data Task Time see Save a File Under a Different Name 123 8 SL L 00 18 58 198 1842 CE ES THB EE BH GH DT Make a New Group 508 Find Records Using a Complex Group Search e6 101 124 38 50 758 lose men Reopen man TO SL OT ss Post Test Questionnaire terrible wonderful 21012 4 4 COE aa 45 ER H ST AT ATO NE inadequate power adequate poner H BL OLA LS 8 Sequenosofsowens arpa pa popa E II II ATREA TA EEC Pose ater rogar PT RS Computer terminology is reied oee H H S 27 42 Messages on screen which prompt user ornp HH 3 fife 9 32 Computer Keeps you informed about what tis dong 2 2 S 4 8 Enormessages Te TT popa Teaming ooperaiotheaysem 9 eT TT OL Eaa new features bytalanderor fs 7 eT TTS 48 Remembering names and use of commans H 3 2 7 s 34 Help messages on tesore Ja TOT ALO LT e ee 7 ETA E E
8. great difficulty manipulating the printer Print out of fax covers was substituted as it required no manipulation of the printer 35 2 3 Should already be stocked with paper Should be correctly connected to the Mac An InTouch User s Manual InTouch will be running on the Mac with the The test InTouch file open The InTouch reminder list clear Evaluation Measures The following evaluation measures will be collected and calculated 1 2 Observations and comments for each critical incident Classification of errors associated with critical incidents severity scope and source of error will be indicated whenever possible The time necessary to complete each benchmark task The percentage of participants who successfully complete each task Participant rankings of the functionality and usability of InTouch Test Report Contents and Presentation The report will include the following sections a ron Executive Summary Description of the Test Results which contains a summary of all results in tabular form Findings Explanations and Recommendations Appendices containing Task scenarios The test plan The post test questionnaire Raw data from the test The target audience for this report will be the Prairie Group developers 14 The InTouch HP print drivers are known to be buggy Prairie Group is working on the problem so there is no reason to subject a participant to the problems 15 A task
9. more space for the functionality Design should be iterative and involve substantial user testing Simplify icons to facilitate understanding and add labels Move the units and dimensions closer together May want to use borders or boxes to emphasize connection Common functionality exists across different layouts however implementation details vary Group and label by common functionality Make current record default Layout dialog box does not contain a cancel button Print Dialog Problem User doesn t recognize item icons in Print dialog box User attempts to use the Layout pulldown in the Print dialog box to select the item to be printed Too many options on print print dialog box User spending a lot of time worrying about things she doesn t need to Icons are too small and Screen Layout object is Missing direct manipulation 29 2 User closes dialog box or hits Done button and thereby canceling request but cannot directly cancel request Error Classification Discussion Importance High busy to be easily recognized 20 4 not placed for best effect 19 16 Also task mapping interaction task not mapped well mapping does not reflect the structure of the user s task 32 6 User wants to specify the item to print before specifying the layout Too many objects on Low screen 19 2 Interaction is too complex 32 2 24
10. much as possible Medium Try more descriptive label e g view reminders Medium Possibly use the first event of the day as the default Potential Importance Solution Medium Screen is very busy Needs a better organization Medium Special is not particularly meaningful to users May want new label or simplify underlying interaction Low Add feedback Label not meaningful enough User did not understand interaction 32 4 Need to have an event highlighted for the buttons to be enabled User did not understand interaction 32 4 Error Classification Discussion grouped 19 6 Interaction too complex 32 2 Related features not well Missing feedback 30 Modify Problem Changing a single reminder that was generated by repeating another reminder crashes InTouch and sometimes the user s computer Special Problem User has no clue as to what L stands for in special dialog box When in event dialog box the special by day yields boxes numbered from 1 to 31 user doesn t know which day of the week goes with which day Missing label 24 2 Potential Solution Importance Error Classification Discussion Functionality High Fix bug implemented incorrectly 35 2 Potential Importance Solution Use a descriptive label Medium Label days of the week Error Classification Discussion Label not meaningful enough User did not understand interacti
11. through Find or to the menubar to get to Group Search Group Search dialog box does not contain a cancel button User wants to be search more than one group ata single time Error Classification Importance Potential Discussion Solution Misleading label 24 9 Medium It s just a special type of label indicates searches find Change label to reflect for groups that Consider combining find and group search into single function with group search functions handled as options of the basic find much in the same manner as find file does on the Mac The way results are Medium If attempt to overwrite any formatted is not group other than temporary noticeable 20 7 and marked ask for confirmation first No confirmation for actions with consequences 39 2 Presentation is not Medium Make search results more noticeable noticeable try presenting in System does not guide a Separate window user s attention to the relevant part of the screen to notice results 39 7 Extra steps in task on Medium Consider eliminating group system 32 7 search as a separate operation and make extra capabilities find options Something along the lines of find file on the Macintosh Direct Manipulation Low Add Cancel button to missing direct Group Search dialog box manipulation 29 2 User close dialog box or hit Done and thereby canceling request but cannot directly cancel request ow
12. InTouch Usability Evaluation Faith McCreary Department of Industrial and Systems Engineering Virginia Polytechnic Institute and State University August 9 2001 TABLE OF CONTENTS EXECUTIVE SUMMARY TEST DESCRIPTION Procedure Evaluation Measures RESULTS Performance Test Summary FINDINGS EXPLANATIONS AND RECOMMENDATIONS General Findings and Recommendations Specific Findings and Recommendations Good Points Problems APPENDICES Appendix A Test Plan Purpose Test Objectives User Profile Method Task List Test Environment Equipment Evaluation Measures Test Report Contents and Presentation Appendix B Task Scenarios Appendix C Post test Questionnaire Appendix D Raw Test Data Performance Data Positive Critical Incident Data Post Test Questionnaire Appendix E References EXECUTIVE SUMMARY The main purpose of this test was to asses the performance of an actual customer with little or no previous InTouch experience The usability test measured the total time needed to accomplish information entry and output and record user critical incidents Tasks included routine operations print operations and import export of data Each test session consisted of a performance test where the user performed a series of tasks and a post test interview where the user filled out a brief user preference questionnaire about the functionality and usability of InTouch and was given the opportunity to make comments or ask
13. User task not mapped to L Modify dialog so that user system even though can select more than one functionality to support is group to search present 32 9 17 Import Export Problem User wants to highlight a range of records and export them Not clear what group imported records are placed in User wants to be export more than one group ata single time Installation Problem 2 icons with install in their name on the installation disk user does not know which to use and in what order Error Classification Discussion Functionality missing functionality 35 1 Placed in current group by default Specification not an explicit part of the dialog User must remember to create change group before beginning import Interaction too complex 32 3 Missing object 20 2 User task not mapped to system even though functionality to support is present 32 9 Error Classification Discussion Language problem multiple dissimilar objects named too similarly Clear in documentation but most people don t read documentation Task Facilitation problem system did not guide or direct user s attention to help the user start the task 18 Potential Solution Importance Medium Add capability Medium Specify the group as part of the import process Make current group default Modify dialog so that user can select items belonging to both groups to export Potential Solutio
14. ace feature involved is mission critical or is a major source of explicitly expressed dissatisfaction or if usability problem is a show stopper e g user cannot complete key task Close Quit InTouch Problem When prompted to save changes user cannot cancel operation Potential Importance Solution Error Classification Discussion Low Activate cancel button in save dialog box Missing physical affordance 32 11 Missing button 20 2 10 Documentation Problem No on line help User would like to know which index entry is the primary reference for a topic Balloon help is not available File Operations New Problem User has trouble finding new file command New in the menubar Important information Error Classification Discussion Importance Medium Importance i Functionality missing functionality that the users want 35 1 Users seldom referred to written documentation but often requested on line documentation not emphasized 21 5 Missing user requested information 52 1 System did not help user start or complete task 39 6 Error Classification Discussion Menu organization inappropriately grouping unrelated choices in a menu 19 9 By convention new file command is usually found in the File menu Metaphor doesn t make sense to user 32 4 User does not understand why all InTouch items are in the InTouch men
15. been changed Cause if Confirm Save preference is not selected and user changes data files without saving changes the user will be prompted to save changes when performing a sort on a newly opened file Save As Problem User has trouble finding save as file command in the menubar Error Classification Importance Discussion Menu organization High inappropriately grouping unrelated choices in a menu 19 9 Metaphor doesn t make sense to user 32 4 User does not understand why all InTouch items are in the InTouch menu Misleading feedback Medium message 26 13 Temporal problem how to handle Error Classification Importance Discussion Menu organization High inappropriately grouping unrelated choices in a menu 19 9 Metaphor doesn t make sense to user 32 4 14 Potential Solution Move Save to a location more intuitive to Mac users i e the File menu Message should only consider current file Internal flags should be reset when the next file is opened Potential Solution Move Save As toa location more intuitive to Mac users i e the File menu Group Operations Problem User cannot access group functions from main window User tries to view group by double clicking on group s name in group window Extremely common novice error User doesn t notice checks result from double clicking on group name Fairly com
16. ection contains a summary of the usability test results in tabular form It includes summaries of both the performance test and questionnaire results The performance test results include for each task mean positive negative critical incident counts percentage of users who successfully completed the task and mean time to complete task The post test questionnaire results are the mean user satisfaction scores from the post test questionnaire Performance Test Summary Mean Number Mean Number Subjects That Mean Task Positive Critical Negative Critical Successfully Time Incidents Incidents Completed Task _ Task sec Inserta Record 0o i6 10 1608 seca e Different Name Find a Specific Record 0 2 J 3 o 206 SortaFie J 0 12 10 9902 Add a Reminder O H L H AE Make a New Group 2 4 o 346 4 Printa FaxCover___ 0 ___38 8 3624 ee H E TES Belonging toa Group NS ATA Simple Group Search Find Records Using a Search li ARRE a T a a File Esra EA AO DEE AE A EE InTouch In any re design efforts particular attention should be paid to improving those tasks with a less than 80 completion rate 5 Critical incidents are something that happens while participant is working that has significant effect either positive or negative on task performance or user satisfaction Hartson 1993 Post Test Questionnaire Summary Mean Rating Overall Reactions to InTouch 4A te
17. ects that work together are not close enough to each other 19 11 When the menubar and main window did not abut novices assumed InTouch consisted only of the main window 20 Add a more descriptive label to pop up e g Display Group Can they be incorporated into scrolling device Add Pull Out to placeholder label for a horizontally compressed field Potential Solution InTouch menu should be broken into a smaller menus consisting of related choices e g place all file operations in the File menu At startup the InTouch main window should abut the menubar Quit appears in both the Files and InTouch menus Remove Quit from InTouch menu User did not understand the presence of duplicate quits 32 10 Open InTouch Problem User confused by lack of InTouch under Recent Applications in the menu User cannot open InTouch file by double clicking on an InTouch data file Missing shortcut 34 4 Potential Solution Error Classification Discussion Importance High Ensure InTouch appears By Mac convention under recent applications InTouch should be there Missing direct High Allow user to open InTouch manipulation 29 2 files by double clicking Also missing shortcut 34 4 By Mac conventions users expect to be able to double click on any file Even experienced InTouch users invariably try to open a file by double clicking Not one user
18. ed to the testing environment Participant will be verbally informed that the test session will be observed and videotaped At this time the participant will be asked to read and sign the consent form Participant will be given the pre test instructions Participant will be given detailed information on how tasks will be presented and what is expected of the participant participant will be given a short demo of thinking aloud Participant will be given final verbal reminders In particular he she will be reminded that a The test evaluates the quality of InTouch not the performance of the participant b He she should refer to the manual only if he she gets stuck and is unable to complete the task iS Originally participants with some InTouch experience were to be tested However no one with InTouch experience could be recruited 34 Performance Test The performance test will consist of a series of benchmark and representative tasks that the participants will be asked to perform The test will be videotaped The basic procedure for the test will be After the introduction is completed the participant will escorted to the evaluation room and will sit down at the desk see description of test environment The participant will be guided through a series of tasks by the tester The participant will be encouraged to a Work without input from the tester b Think aloud while working through the task c To refer to
19. er problem area with users Most found the boxes too cluttered and the icons indecipherable Both would benefit from a visual re design and extensive usability testing focusing just on them Specific Findings and Recommendations Good Points Documentation Users gave InTouch documentation high marks Unfortunately users seldom referred to it even when having problems Sort Users also liked the sort exclusion option Find Users liked that find isn t necessarily case sensitive Group Search Users found the option of creating a new group when in the group search dialog box very handy Reminders Users especially liked the sound and repeat options Problems This section contains a tabular listing of the specific problems uncovered along with potential solutions It is organized in terms of existing interface elements Problems are classified in terms of Keenan s usability taxonomy Keenan 1996 They are also assigned an importance rating Importance is a subjective measure of the problems effect on usability and is independent of cost It is determined as follows Importance Description Rating If problem did not impact task performance or dissatisfaction very much e g irritant or cosmetic problem but is still worth listing If user can complete task but ability to perform it properly was seriously hampered e g caused confusion and required extra effort or problem was source of dissatisfaction If interf
20. estion about whether this belongs on main window Consider placing in a menu so can list symbol what it does and shortcut Does this belong on the main window If it does use a better label e g date time stamp If it doesn t place it in a menu and provide a shortcut User not sure how Group pop up controls record display User not sure what the arrows apply to When group list and record list was compressed user thought record group placeholders were labels for the address notes field Menubar Problem User has trouble finding options in menubar menus Novice InTouch user doesn t realize the menubar contains InTouch options when the main window and menubar do not abut Dissimilar objects look User did not understand the interaction 32 3 Label does not indicate full effect of action 24 8 Low Arrows apply to records but are over by address field That work together records list and arrows are not close enough to each other 19 11 too similar i e horizontally compressed panes look like labels 20 12 Error Classification Discussion Importance High Menu organization inappropriately grouping unrelated choices in a menu 19 9 Having all InTouch operations accessible from one giant InTouch menu confused users Also it increased search time and increased the probability that a user would miss the option being searched for Screen layout obj
21. g 1 Learning to operate the system 0 12 3 4 5 6 7 8 9 difficult easy 2 Exploring new features by trial and error 0 1 2 3 4 5 6 7 8 9 difficult easy 3 Tasks can be performed in a straight forward manner 012 3 4 5 6 789 never always 4 Remembering names and use of commands 12 3 4 5 6 7 8 9 difficult easy 5 Help messages on the screen 0 1 2 3 4 5 6 7 8 9 unhelpful helpful 6 Supplemental reference materials 012 3 4 5 6 789 confusing clear E System Capabilities 1 Correcting your mistakes 012 3 4 5 6 7 8 9 difficult easy 2 Experienced and inexperienced users needs are taken into consideration 0 12 3 4 5 6 7 8 9 never always 42 3 As a free form information manager InTouch is 0 1 2 3 4 5 6 7 8 9 useless useful 4 When compared to other information managers you have used InTouch is 0O 1 2 3 4 5 6 7 8 9 difficult easy 43 Appendix D Raw Test Data Performance Data Positive Critical Incident Data EA ST S SERES Name HEH ESA fe EN EA AAA IE T EOS ER E CA NAAA EEES IA EEN IS EA ER HT E A PEA ON Eel ie AS An ON ES ESA CA E OO ESO NOS ER E E A LA ES EA IEA EA EE HEAT TE A E EN EE E TEA E EA EN SEA A E AREA AE ERA AA RET Fae pee SE RA E Negative Critical Incident Data insetaRecod SSCS RL HET EC E OY Save a File Under a Different Name S A HPL OO 3 OS OT PO eT 2 CES E PO POT SY L eT HT OLS LOL BMY Esport Records Berang oaro rpe jo pel i i E Find Records Using a
22. he giant InTouch menu Many grumbled about the lack of access to group functions from the main window and were frustrated by the lack of on line help Probably what frustrated people the most was the giant InTouch menu It was too big to search quickly or accurately Further it contained items generally found in the File or Edit menus e g SaveAs so it was usually the last place users looked This menu should be broken up into a series of smaller menus The File menu should contain file commands the Edit menu should contain editing commands The remaining commands may be parceled out to one or more menus Users can be recruited to sort the commands into groups to help develop a menu grouping that is more intuitive to users Participants also had a great deal of trouble determining the functions of the different panes and buttons in the main window Labeling of the windows and better button labels should alleviate many of the problems as would on line help In tests users infrequently referred to the documentation but frequently went to look for on line help Participants were also disliked InTouch not behaving like other Macintosh applications Its files could not be opened by double clicking Its files do not appear under Recent Documents inthe menu nor did InTouch appear under Recent Applications in the menu No one liked having to select InTouch from the menu to start up the program The print and layout dialog boxes were anoth
23. hing other than an address book Potential Solution Error Classification Discussion Move find to a location more intuitive to Mac users i e the Edit menu Menu organization i inappropriately grouping unrelated choices in a menu 19 9 By convention find is usually found in the Edit menu i 1 Make group an explicit part of the find dialog default current group Error message does not help user solve problem 27 4 User did not understand the interaction 32 3 2 In error message explicitly state the parameters find used 26 User doesn t understand field option on find User doesn t realize that the record wasn t found because he had the whole word only option on Find dialog box does not contain a cancel button Insert Problem After typing a new record the subject hits insert which brings up another new record Occurred frequently with novice users User does not differentiate the fields in the main window did not under stand interaction 32 3 Error message does not help user solve problem 27 4 Medium Medium a Importance Direct Manipulation missing direct manipulation 29 2 User can close dialog box and thereby canceling request but cannot directly cancel request Error Classification Discussion 26 2 Missing feedback Unaware operation is complete 27 Label the fields on the main window Need to be more spec
24. ific in find error messages in particular what options guided the search Add Cancel button to Find dialog box Potential Solution Add feedback not message just have the text appear in the records list as user is typing so they can see that the record has already been inserted Snapshot Problem User repeatedly clicks the snapshot button in the main window trying to figure out what it does When multiple clicks appear to do nothing user is mystified When user hit snapshot the word snapshot appears in the clipboard not the address field Word 6 0 running at time User wants to use cut and paste on entire records without resorting to Snapshot dialog Error Classification Discussion Importance Name label label not meaningful to user 24 7 non message message feedback is needed 23 2 26 2 Also user did not understand the interaction 32 3 Functionality not implemented correctly 35 2 Direct Manipulation Medium missing direct manipulation 29 2 28 Potential Solution 1 Change label icon to a more meaningful term e g clipboard or 2 Clipboard is just a form of cut and paste Consider treating it as such place additional commands under edit i e show clipboard and eliminate snapshot button At the same time improve overall cut and paste operations on records groups etc Fix so functionality matches documentation
25. is classified as successful or unsuccessful No partial measures of completion will be used 36 Appendix B Task Scenarios 1 The file that you see is your family rolodex Put your name and work phone number in the rolodex so other family members can contact you in an emergency Benchmark task Insert a Record 2 Save a copy of your family rolodex for personal use Call the copy MY ROLODEX You will work with MY ROLODEX for the remainder of this session Benchmark task Save a File Under a Different Name 3 Play around with the system exploring anything you like in the InTouch System for as long as you like 4 Ken Landry is already entered in your rolodex He just called you with his new home phone number add it to his entry Ken s home phone number is 500 1234 Benchmark task Find a Specific Record 5 You decide you don t want all of Mary s friends in your rolodex Make a list of Mary s friends Delete Sallie s and Brian s information Benchmark task Delete Records Belonging to a Group 6 Order the entries in your rolodex alphabetically so that people with the same last name appear together with family members listed alphabetically by first name Benchmark task Sort a File 37 7 Congratulations you just found a new ballroom dance class It meets every Wednesday 8 00 10 00 PM for the next 3 months Add the class to your calendar Benchmark task Add a Reminder 8 You notice
26. mon mistake Group Edit Problem User doesn t connect Edit Groups with creating a new group Thinks that it will allow her to specify which records belong in a group User is confused by Delete and Insert buttons not working on groups Error Classification Discussion Importance High Missing physical affordance to access functionality from main window 32 11 Missing direct Medium manipulation 29 2 Mental model not easy to understand 32 5 Importance Medium Check mark signaling group membership is not noticeable 20 6 Interface modes not well distinguished 39 9 Error Classification Discussion Label not meaningful enough 24 7 Inconsistent ways to manipulate similar objects 29 1 Similar tasks structured inconsistently 32 1 15 Potential Solution Add group button on main window Re evaluate button usage on main window eliminate the infrequently used to cut down on clutter in main window Label the window e g record s group membership Use a different symbol Consider highlighting groups to which the current record belongs Potential Solution Try to find a better label e g add delete groups 1 Allow user to delete group shown in Groups List by highlighting group names and hitting Delete button or 2 Change label on button to Delete Record User wants to be able to edit gr
27. n Importance Low Rename icon s or combine functionality Main Window Problem Novice InTouch user has great difficulty discerning function of the different panes User has no idea why Tag would be used Clicks it several times guesses that character might be footnote and finally deletes characters User must click on Stamp to figure out what button does Error Classification Discussion Missing label 24 2 and user did not understand interaction 32 3 During usability testing some users never fully understand the functions of the different panes even after using the product for an hour or more Concept very difficult to uncover by trial and error and most users did not refer to the paper manual although most did look for on line documentation Tag not meaningful enough users frequently do not understand what action is associated with the button 24 7 Also interaction user did not understand interaction 32 3 Stamp not meaningful enough users frequently do not understand what action is associated with the button 24 7 19 Importance Medium Potential Solution Panes should be labeled to facilitate learning process On line help and bubble help should be available to increase the likelinood of users accessing documentation More descriptive label but may be difficult due to combined functionality may want to break up Real qu
28. on 32 4 31 APPENDICES Appendix A Test Plan The following is the test plan for conducting usability tests of version 2 5 2 of the free form information manager InTouch The plan contains the following sections Purpose Test Objectives User Profile Method Task List Test Environment Equipment Evaluation Measures Test Report Contents and Presentation Purpose The main purpose of this test is to asses the performance of an actual customer with little or no previous InTouch experience The usability test will measure the total time needed to accomplish information entry and output and record observed critical incidents Tasks will include routine operations print operations and import export of data Particular attention will be paid to those features that have been the target of significant negative customer feedback Test Objectives Specifically the test will determine if an actual customer with little or no previous InTouch experience can Successfully enter information into InTouch within the first 5 10 minutes of use Successfully output information from InTouch within the first half hour of use Successfully perform routine tasks associated with sorting grouping and retrieving of information Successfully use print and import output dialogs The Prairie Group has received numerous complaints about the printing and import export dialogs The test will attempt to determine the precise nature
29. oup names in Groups List User can move or delete individual groups but can not move or delete the groups in a single step User has to alphabetize list by moving groups cannot alphabetize list in one step User would like a pre defined shortcut for Edit Group Inconsistent ways to manipulate similar objects 29 1 Similar tasks structured inconsistently 32 1 User task not mapped to the system even though functionality to support task is present 32 9 Cognitively indirect manner to manipulate groups 29 8 Interaction user task not mapped to the system even though functionality to support task is present 32 9 Cognitively indirect manner to alphabetize groups 29 8 Missing shortcut 34 4 16 Medium Low Allow user to directly edit group names in Groups List Allow user to highlight several groups and manipulate those groups e g delete Option on Group Edit dialog box to alphabetize or sort list explore possibility of keeping the Temporary and Marked groups in same position and alphabetizing rest of list Add a default shortcut for Edit Group Group Search Problem User believes group search returns all records and group names matching the input specification User destroys records belonging to a group with group search User not sure requested group search operation was done User dislikes having to always go
30. questions about InTouch Evaluation measures included observations and comments for each critical incident classification of errors associated with critical incidents including severity scope and source of error the time necessary to complete each task the percentage of participants who successfully complete each task and user rankings of the functionality and usability of InTouch All participants did fulfill InTouch developer goals of having users succeed in entering information into InTouch within the first 5 10 minutes of use and outputting information from InTouch within the first half hour of use Although many participants did have major problems User likes included the InTouch documentation the exclusion option in sort the option of creating a new group from the group search dialog box the choice of sounds for reminders and the repeat option for reminders User dislikes included having to search the giant InTouch menu the difficulty figuring out the function of the different panes in the main window the lack of on line help and the lack of access to group functions from the main window Users also disliked InTouch not behaving like other Macintosh applications its files could not be opened by double clicking and no one liked having to select InTouch from the menu to start up the program Most users also had problems with the print and layout dialog boxes Problems were rated in terms of severity and scope and whenever po
31. re the participant was given a short introduction to the testing and introduced to the testing environment The subject was also given a short demo of thinking aloud 2 Performance Test which consisted of a series of tasks that the participants were be asked to perform Participants were videotaped while performing tasks 3 Post test debriefing in which the participant filled out a brief user preference questionnaire about the functionality and usability of InTouch and was given the opportunity to ask questions or make comments about InTouch and the testing process Evaluation Measures The following evaluation measures were collected and calculated 1 Observations and comments for each critical incident 2 Classification of errors associated with critical incidents severity scope and source of error were indicated whenever possible 3 The time necessary to complete each benchmark task 1 Originally installation was included in the task list However as the installation process is straightforward but time consuming it was not included in the final task list 2 5 10 minutes is Prairie Group development goal 3 First half hour is Prairie Group development goal 4 The percentage of participants who successfully completed each task 5 Participant rankings of the functionality and usability of InTouch A task is classified as successful or unsuccessful No partial measures of completion will be used RESULTS This s
32. rrible wonderful 4 4 dull stimulating 3 4 Ee A Leming R 25 System Capabilities consideration As a free form information manager InTouch is 4 When compared to other information managers you have 2 4 used InTouch is The post test questionnaire rating scale went from 0 to 9 FINDINGS EXPLANATIONS AND RECOMMENDATIONS This section contains the findings and recommendations along with a brief discussion and explanation of them It has been divided into General Findings and Recommendations followed by Specific Findings and Recommendations The General Findings and Recommendations section contains a high level discussion of the major global issues The Specific Findings and Recommendations contains tabular lists of user likes user wish lists and user problems Finally the solutions presented in this report are just recommendations Any re design should consider alternative solutions Both the problems and potential solutions should be evaluated in light of the total system General Findings and Recommendations Happily all participants did fulfill InTouch developer goals of having users succeed in entering information into InTouch within the first 5 10 minutes of use and outputting information from InTouch within the first half hour of use Unhappily many encountered major problems they had great difficulty figuring out the function of the different panes in the main window and they floundered about in t
33. ssible the source of the problems and potential solutions were indicated The solutions presented in this report are just recommendations Re design efforts should consider alternative solutions with both the problems and potential solutions evaluated in light of the total system TEST DESCRIPTION The main purpose of this test is to asses the performance of an actual customer with little or no previous InTouch experience The usability test measured the total time needed to accomplish information entry and output and record observed critical incidents Tasks included routine operations print operations and import export of data Particular attention was paid to those features that have been the target of significant negative customer feedback Specifically the test attempted to determine if an actual user with little or no previous InTouch experience can Successfully enter information into InTouch within the first 5 10 minutes of use Successfully output information from InTouch within the first half hour of use Successfully perform routine tasks associated with sorting grouping and retrieving of information Successfully use print and import output dialogs The Prairie Group has received complaints about the printing and import export dialogs The test will attempt to determine the precise nature and possibly the source of customer difficulties Procedure Test sessions had three stages 1 General Introduction whe
34. t you forgot something re open My Rolodex Benchmark task Close then ReOpen InTouch 39 Appendix C Post test Questionnaire Directions Please rate each of the items according to the scales provided and circle the appropriate number A Overall Reactions to InTouch 1 0 1 2 3 4 5 6 terrible 2 0 1 2 3 4 5 6 difficult 0 1 2 3 4 5 6 inadequate power 5 0 1 2 3 4 5 6 rigid B Screen 7 8 9 wonderful 7 8 9 easy 7 8 9 stimulating 7 8 9 adequate power 7 8 9 flexible 1 Organization of information on screen 0 1 2 3 4 5 6 confusing 7 8 9 very clear ae Adaptation of the QUIS Questionnaire for User Interface Satisfaction rating scale developed by the University of Maryland 40 2 Sequence of screens 0 1 2 3 4 5 6 789 confusing very clear 3 Characters on the computer screen 0 1 2 3 4 5 6 789 hard to read easy to read 4 Highlighting on the screen simplifies task 0123 4 5 6 78 not at all very much C Terminology and System Information 1 Use of terms throughout system 012 3 4 5 6 7 8 9 inconsistent consistent 2 Computer terminology is related to the task you are doing 0 12 3 4 5 6 7 8 9 never always 3 Messages on screen which prompt user for input 0 12 3 4 5 6 7 8 9 confusing clear 4 Computer keeps you informed about what it is doing 012 3 4 5 6 7 8 9 never always 5 Error messages 0 1 2 3 4 5 6 7 8 9 unhelpful helpful 41 D Learnin
35. tential Solution Error Classification Discussion Change reminder label to a term more meaningful to users During testing users generally referred to the item as a calendar or alarm Change icon accordingly High In preferences allow user to specify whether the Remind button takes the user to the standard calendar or DATEVIEW Label not meaningful 24 7 user s don t understand what the remind corresponas to Also icon is not meaningful 20 9 to user Misleading visual cue 30 7 Dateview Button is largest most salient on screen but functionality not supported Also system did not make erroneous choices unavailable 39 4 29 Novice user has great difficulty determining where to enter new appointment Focuses on the calendar and doesn t see the new button View in Remind dialog box has no meaning for many users User confused by disabling of modify delete buttons on a day she defined a reminder New Problem Subject misses the for part of the calendar repeat Does multiple instances of the reminder by hand User does not understand the repeat for still does it with special Proximity of the two not clear or wording problem User not sure that reminder was created Goes to calendar to check Related features not well Related features not well grouped 19 6 Medium Try putting buttons to manipulate calendar closer to the calendar De clutter screen as
36. that all your relatives have RELATIVE in the notes field You decide to make family a new group and make all your relatives members of the group Benchmark task Make a New Group 9 You decide to print out a fax cover sheet for David A Groom Be sure to include your return address on the cover sheet Benchmark task Print a Fax Cover 10 Uncle Harry a new InTouch user would like you to send him an file containing the names and addresses of all the relatives your family sends Christmas cards to Create an InTouch file by the name of Relatives to send to Uncle Harry Benchmark task Export Records Belonging to a Specific Group 11 You have to go to Washington D C for the weekend You don t want to stay in a hotel You want to stay with someone you know How many people do you know in Washington Benchmark task Find Records Using a Simple Group Search 12 After thinking more about the weekend you decide you want to stay with family Find a relative in Washington Benchmark task Find Records Using a Complex Group Search 38 13 You re in charge of organizing soccer this year Last year s organizer gave you a tab delimited text file Soccer People containing the names and addresses of the relevant people Add this people to your rolodex put them in the Soccer group for easy reference Benchmark task Import Data From a File 14 That s enough work for today Shutdown InTouch 15 Wai
37. the manual only when stuck 3 Post test debriefing After all tasks are completed the participant will be given a short break After which the participant will fill out a brief user preference questionnaire about the functionality and usability of InTouch Finally the participant will be given the opportunity to ask questions or make comments about InTouch and the testing process Following completion of the test the participant will be thanked and dismissed Task List Starting up InTouch Inserting records into an InTouch file Finding records meeting certain criteria Deleting records meeting certain criteria Adding and deleting reminders including repeated reminders Printing a fax cover sheet Exporting information from InTouch Importing information from InTouch Saving an InTouch file under a different name Test Environment Equipment The test will be conducted in the Virginia Tech Computer Science Usability Laboratory During the performance test the participant will sit at a desk in the evaluation room with the door closed If a member of the test team is available he she will remain in the evaluation room with the participant On the desk will reside a Macintosh computer Close to the desk on a side table will be the following items 1 A printer Should not be an HP printer G Originally participants were to use InTouch to print out envelopes However during this operation during pre test subjects had
38. u 11 Potential Solution Add on line help to system Bold primary page in index Add balloon help to application Potential Solution Move New to a location more intuitive to Mac users i e the File menu User has trouble understanding the function of the New command in the InTouch Menu User confused by Save button in New dialog box Open Problem User has trouble finding open file command Open in the menubar Buttons in Open dialog box are ordered differently than buttons in other file related dialog boxes e g Save As dialog box Names labels wording not meaningful enough 24 7 Names Labels system centered terms instead of user task domain terms 24 4 User is creating file system is saving an empty InTouch file Error Classification Discussion Menu organization inappropriately grouping unrelated choices in a menu 19 9 Metaphor doesn t make sense to user 32 4 User does not understand why all InTouch items are in the InTouch menu Inconsistent screen layout 19 1 In Open dialog box buttons are ordered Open to Cancel while in other file related dialog boxes buttons are ordered from Cancel to box action e g Save Similar tasks are structured inconsistently 32 1 12 Place option in File menu which will provide a context for user understanding preferred
39. went to the menu 21 Preferences Problem User unable to specify which data file to be opened at start up User does not understand that Confirm Saves in the General Preferences dialog box will result in the user being prompted to save changes when closing InTouch or switching data files User cannot specify that InTouch open the last data file used at startup Error Classification Discussion Task not well mapped 32 6 User never discovered that start up file could be specified in the Open dialog box User thought of it as a preference not an open operation Names Labels label is misleading 24 9 Confirms Saves implies that system asks for confirmation after a save has been initiated System does not allow user to specify his her preferences in this area 34 1 i e user can specify which file to open at startup but user cannot specify InTouch use the last file opened 22 Importance High Medium Potential Solution Allow the user to specify the startup file as a preference and when opening a file a few experienced users found this handy Re word dialog e g prompt to save changes Add a preferences option that tells InTouch to open the last file opened at startup Printing Layout Dialog Problem Users have great difficulty using layout dialog box some users refuse to even try User doesn t recognize item icons in La
40. yout dialog box User can t determine what units are used for the layout width height User complains of objects jumping around as he explores the different layouts When creating fax cover user doesn t want to have to specify record Would like the default to be current record Numerous visual layout Missing default value Error Classification Discussion Importance problems inconsistent screen layout 19 1 screen clutter 19 2 too many objects on screen 19 2 unaesthetic layout of objects 19 3 not clearly delineating groups of objects with borders boxes or spacing 19 13 object grouping not intuitive to user so user could not use grouping to help locate an item Also interaction is simply too complex 32 3 Object Appearance icon is not meaningful enough 20 9 Icons are too small and busy to be easily recognized Medium Object layout objects that work together i e width height and units are not close enough together 19 11 Inconsistent object layout 19 1 Also system did not guide or direct user s attention to the relevant part of the screen 39 6 36 3 23 Potential Solution Redesign from scratch Look and feel should be more like Print dialog box Instead of making everything visible put the less frequently used groups down a level or in a pull down menu Also try halving the size of the graphic representation of the layout to allow
Download Pdf Manuals
Related Search