Usability Report: Mitsubishi HS
Contents
1. Tester reviews the form and then conducts a short interview with the participant to further explain their answers Answers are recorded on the log Is there anything else you would like to add about your experiences today Susan H Kaltenbach TC 517 Page 16 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation References Gray Wayne D 2000 The nature and processing of errors in interactive behavior Cognitive Science Volume 24 Issue 2 205 248 Schriver Karen Dynamics in document design John Wiley amp Sons Inc 1997 Susan H Kaltenbach TC 517 Page 17 12 12 20012. Mitsubishi HS U500 VCR documentation Appendix E Script for test administrator Check here when completed Participant enters the room Tester introduces herself Hello my name is Susan Kaltenbach and will be the administrator for this usability test Right now am reading from a script By doing so will be able to make sure that every participant has the same interaction with me and the same environment This is important when conducting a study Today will be observing you as you perform two tasks on the Mitsubishi HS U500 VCR will ask you to complete a questionnaire and then will give you two tasks to perform A part of you your hands and shoulders will be videotaped during this procedure but you will remain anonymous Your face and full name will not be shown or used Walks to video camera Would you like to look through the viewfinder of this video camera to see what am filming lf yes You can see that we will be recording only your hands the VCR and the TV screen Let s return to the testing area During this procedure am going to ask you to use the talk aloud protocol This is a very common way for researchers to gather opinions from participants A talk aloud protocol means that you talk aloud as you perform the task will prompt you if you become silent Let me give you an example of the talk aloud protocol as open the user s manual and try to find out how
3. 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Based on video logs of the start and stop time for task performance below are the times recorded for each participant for each of the two tasks along with a mean presented for comparison purposes only Task 1 Setting the time Total Start Time Stop Time Elapsed Participant PST PST Time 5 46 25 5 52 21 0 05 56 5 44 45 5 46 50 0 02 05 A B c 6 09 18 6 12 23 0 03 05 D 6 37 22 6 40 45 0 03 23 Mean 0 03 37 Task 1 Setting the VCR time total time elapsed E 3 o a Q s o E ke C Participant Susan H Kaltenbach TC 517 Page 7 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Task 2 Recording a Program Total Start Time Stop Time Elapsed Participant PST PST Time ees RES A B C 6 13 00 6 14 46 0 01 46 D 6 41 22 6 42 44 0 01 22 Mean 0 01 54 Task 2 Recording a program total time elapsed S E 3 o 2 s o o E ke Cc Participant Satisfaction Results Very Very easy Easy Neither Difficult difficult 1 Overall programming the Mitsubishi HS U500 VCR was all TI Very Not helpful Helpful helpful 2 Was the owner s manual helpful in setting the VCR s time wjf Participant B did not use the owner s manual at any time during the test and thus did not answer questions pertaining to the owner s manua
4. I m going to go to the Table of Contents and look for something telling me how to change the VCR time well must have missed it somewhere so I ll look through the list of chapters again to see if maybe I passed it over after 1 min 20 sec So I guess it doesn t really say time so setting up your VCR for the first time I m assuming that it the setting time instructions will be on page 19 Starts reading on page 19 and flips pages after 1 minute Good lord page 29 finally Participant C Im going to find the table of contents in here and I m going to try to find how to set the time on the VCR and I d expect it to be somewhere near the beginning and and I m looking desperately frantically searching and I m not seeing it l d expect it to be somewhere near the beginning These two participants were unable to find information because of their adherence to their mental model of what new VCR instructions should consist of Participant D however had no problem finding the instructions for setting time There was also a highly diverse response to the objective Which reference method do users rely on the most the user manual or the remote control interface While three of the four participants used the owner s manual Table of Contents to find information participant B never used the owner s manual After reading the task sheet participant B used the remote control in performing ta
5. Usability Study Mitsubishi HS U500 VCR documentation By Susan Kaltenbach Test conducted December 3 and 4 2001 Report prepared December 12 2001 Address inquiries to Susan Kaltenbach Email inheritanceabuse gmail com Usability Study Mitsubishi HS U500 VCR documentation Executive Summary This report documents the findings of a usability study of supporting documents for the Mitsubishi HS U500 VCR A total of four 4 participants were given two 2 tasks to perform on the VCR Measurements were taken of the total elapsed time for task completion for each of the two tasks A post questionnaire was administered and feedback was solicited to generate comments about the Mitsubishi HS U500 VCR Positive aspects of the Mitsubishi HS U500 supporting documentation include e Participants rated using the VCR remote control as easy or very easy e All participants found the VCR s on screen menus helpful e Seventy five percent 75 of participants rated the overall programming of the VCR easy e Participants found on screen menu settings most helpful for recording a program Areas identified as needing improvement in the supporting documentation included e Instructions for setting time in the owner s manual Table of Contents is difficult to find e Fifty percent 50 of users found the layout of the owner s manual Table of Contents text difficult to read e Fifty percent 50 of users were unable to confirm a
6. each question choose from the answers provided and check that box If you want to make additional comments a space has been supplied under each question 1 Overall programming the Mitsubishi HS U500 VCR was Very Easy Easy Neither Easy nor Difficult Difficult Very Difficult Kai T Comment 2 Was the owner s manual was helpful or hindering in setting the VCR s time Very Helpful Helpful Neither Helpful nor Hindering Hindering Very Hindering EJ Comment 3 Was the owner s manual was helpful or hindering in setting the VCR to record a future broadcast Very Helpful Helpful Neither Helpful nor Hindering Hindering Very Hindering Comment 4 Was it easy to find information in the owner s manual Very Easy Easy Neither Easy nor Difficult Difficult Very Difficult 2 Comment 5 Did you find operating the VCR remote control to be Very Easy Easy Neither Easy nor Difficult Difficult Very Difficult Comment 5 Did you find the VCR s on screen menus to be helpful or hindering Very Helpful Helpful Neither Helpful nor Hindering Hindering Very Hindering gt Comment 7 Would you recommend this VCR to someone less technically Yes No experienced than yourself Comment Susan H Kaltenbach TC 517 Page 14 12 12 2001 Usability Study
7. easy to perform using the reference materials None of the manual using participants had trouble referring from the manual to the on screen menu when using the remote control and the two supporting components worked together well Participant C commented that having illustrations of the on screen menu inside the printed owner s manual made referring back and forth much easier This answers another objective Will users be able to successfully refer to the user manual remote control interface during their operations Results were highly diverse for the objective What problem solving approach will they use during the procedure For example participant D was able to quickly find the information in the Table Contents for Task 1 Setting the VCR time while participants A and C had more difficulty These two latter participants viewed the manual as an ongoing narrative and expected the information presented to follow that narrative They both had a mental model that the first step their new VCR would need was to set the VCR time and so such a function would appear at Susan H Kaltenbach TC 517 Page 9 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation the beginning of the manual As a result these participants tried to discover evidence of the narrative that would support their mental model Participant A So lm going to use the user s manual to complete task one and since we re going to change the time on the VCR
8. g note of 1 The time the work started 2 the methodology owner s manual remote control etc being used by the participant to perform the task Procedure Participants received a short scripted introduction and orientation The purpose and objective of the usability test was explained Participants were assured that their personal performance was Susan H Kaltenbach TC 517 Page 5 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation not being measured They were informed that the focus was on possible shortcomings of the device and supporting documentation The participants were informed that they were being observed and videotaped and asked to sign the consent form e The participants were asked to sit down in front of the VCR user manual remote control and television and to complete the pre test questionnaire and the consent form e From atester script the tester explained the nature of the usability test and how it proceeded The tester then demonstrated the talk aloud protocol by performing a task while using the protocol e The participant received the scenario sheet and was asked if they understood it e The tester presented the participant with the first task sheet e The tester sat at a location visible to the participant only by their peripheral vision This allowed the tester to observe the participant and prompt when the think aloud protocol is not being observed e After completion of Task 1 the pa
9. haracteristics were screened via a pre test questionnaire All participants reported five 5 years or more experience using a VCR and were either very comfortable or comfortable using a VCR Used a Participant 1 Participant 2 Participant 3 Participant 4 Table 1 Participant profile for usability study Susan H Kaltenbach TC 517 Page 4 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Tasks Participants were given a scenario sheet describing their role They were identified as sports fans that needed to use their VCR to program an upcoming sports event Two tasks were then performed under this scenario The first task was to set the time for the VCR The second task was to set programming for the VCR to record a future broadcast These two tasks were selected as representative of essential functions in operating a VCR and popular opinion often describes these VCR functions as problematic Participants completed the tasks while being recorded on video The tester observed their performance in a non intrusive manner Participants were also instructed to use the think aloud protocol during task performance The total time elapsed for completion of each task was recorded Tasks were judged accomplished only when the participant identified that they had successfully performed the task All participants were instructed to let the tester know when they were done This self acknowledgement is importan
10. ive you two tasks to perform Please do not turn over the top sheet until ask you to do so Hands task packet to participant Let s get started on your first task Remember that need you to think aloud as you work If you fall silent will give you a prompt to restart talking Since this study is examining the owner s manual you have to rely solely on its contents to perform the task will not be able to answer your questions about this task Please turn over the top sheet and begin working on task 1 only Participant begins work on task 1 Notes are taken on the study log If participant falls silent prompt for restarting the talking Participant indicates that Task 1 is complete Now we are ready to start the second part of the study Please use the same procedures and protocols as the first task Please turn over the top sheet and begin working on task 2 only Participant begins work on Task 2 Notes are taken on the study log If participant falls silent prompt for restarting the talking Participant indicates that task 2 is complete Great You have finished the tasks for this usability study am now going to give you a post test questionnaire to ask you some specific questions about your experience with this VCR If you have any questions on completing this form please let me know Give participant the post test questionnaire form Form is completed and returned to tester
11. l Susan H Kaltenbach TC 517 Page 8 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Very Not helpful Helpful helpful 3 Was the owner s manual helpful in setting the VCR to record a future broadcast a 1 Very Ge ER SECH ec E 4 Was it easy to find information in the owner s manual 5 Did you find operating the VCR remote control to be easy Very Very easy Easy Neither Difficult difficult Very Not helpful Helpful helpful 6 Did you find the VCR s on screen menus to be helpful Discussion All participants were able to perform the assigned tasks thus answering the first objective of this usability test Will users be able to perform the tasks Less certain is how helpful the printed owner s manual was in supporting their task performance Finding information in the manual was a problem for all manual readers Of the three participants who used the printed owner s manual two found the instructions difficult to locate for Task 1 Setting the VCR time Also two out of three participants found it difficult to read the Table of Contents These participants commented on typographical and graphic layout issues that could contribute to the problem Importantly after finding the instructions a majority of the manual using participants found the manual helpful in performing both tasks answering the second objective of this test Will the users think that the assigned tasks were
12. n H Kaltenbach TC 517 Page 3 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Introduction The Mitsubishi HS U500 VCR offers standard VCR functionality for playing and recording a videocassette It has a user interface oriented around the device s remote control The product is packaged with an owner s manual for supporting documentation This product is intended for the consumer market for home use Our usability test focused on determining if the participants needs were met by support documentation supplied by the manufacturer Specific metrics for the usability evaluation are outlined in a later section Test Objectives This usability test focused on specific concerns as outlined below e Will users be able to perform the tasks of setting time and recording a program e Will the users think that the assigned tasks were easy to perform using the reference materials e Will users be able to successfully refer to the user manual remote control interface during their operations e What problem solving approach will they use during the procedure e Which reference method do users rely on the most the user manual or the remote control interface Method Participants A total of four 4 participants were selected through an open recruitment via e mail in the University of Washington Technical Communications department All participants were students currently enrolled in that college and program Participant c
13. rticipant was given the second task sheet e After Task 2 was completed the participant was asked to complete a post test questionnaire e The tester also interviewed the participant for additional comments and opinions Performance evaluation Performance was evaluated by recording the total time elapsed for each assigned task The ending time for the task was considered complete only when the participant self identified that they were done Participant debriefing After the post test questionnaire was completed the participant was debriefed by the tester This debriefing was also videotaped Topics included e The participant s response to questions on the post test questionnaire e Specific errors or problems the tester observed the participant making during the usability test e The participant was asked for spontaneous comments about the test experience Each individual was paid 15 cash for their participation Results Data Analysis Data were collected from the participants statements recorded on video their total elapsed time to complete each task and their responses to the post test questionnaire and debriefing Data for total participant time for each test is presented below Due to the small size of this participant group and the nature of data collected no statistical analysis has been performed on the data except to provide the mean value for each total time elapsed Susan H Kaltenbach TC 517 Page 6
14. s Contact phone number Your experience using a VCR Have you used a VCR in the past Yes No How many years of experience do you have using a VCR 1 Six months 1 to 2 years 3 to 5 years more than 5 years Using the checklist below please let me know what you have used your VCR for in the past Watch pre recorded tapes movies programs etc Record programming as it is playing hitting record button during a program _ Program VCR to record a show in the future How comfortable do you feel using a VCR Very comfortable Somewhat comfortable Neutral Somewhat uncomfortable Very Uncomfortable G Susan H Kaltenbach TC 517 Page 11 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Appendix B Scenario The scenario You are a hockey fan You notice that the 2001 Stanley Cup playoff is scheduled to be broadcast on June 10 at 8 00 pm on channel 13 But you will be out of town visiting your mother in law on that day and you won t be able to watch the game at her house Your mother in law is not a hockey fan You will use your Mitsubishi HS U500 VCR to program the upcoming Stanley Cup match so that you can watch it when you return home later Do you have any questions Susan H Kaltenbach TC 517 Page 12 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Appendix C Tasks Task 1 In order to record fut
15. sks without ever reading the owner s manual This was the participant s natural way of performing tasks Significantly participant B had a dramatically lower elapsed time 0 02 05 to perform Task 1 than the other users When performing Task 2 all users began to rely heavily on the remote control and on screen menus to program a future broadcast Participants A B and D used the remote control exclusively to perform this task while participant C referred back and forth from manual to on screen menu to perform the task It is important to note that when participants used the remote exclusively to record a program without referring to the user s manual their time scores were not only lower than Task 1 scores but they were closer to the mean It took less time to set programming to record a future broadcast than to set the VCR time because they used the remote control exclusively However after performing Task 2 fifty percent 50 of participants were unable to identify that they had completed the task successfully These participants then repeated the same procedural steps until they saw evidence of their prior programming displayed Only then could they identify that they had completed the task Susan H Kaltenbach TC 517 Page 10 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Appendix A Prescreening questionnaire Usability participant questionnaire Your personal information Name e mail addres
16. successful operation when recording a program Based on the usability problems identified during this test we recommend e Updating the organization of the owner s manual so that it is structured more clearly with categories matching the VCR users mental map of functions e Improving the formatting of the Table of Contents for better legibility e Providing a confirmation message after performing a function when using the VCR on screen menus Susan Kaltenbach December 12 2001 Susan H Kaltenbach TC 517 Page 2 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Executive Summary 2 Introduction 4 Test OB OCUVES zosckeu fea cock k ng ee faeces Seen cea Seege Sege eege 4 Method ee eaaa aaa aI Partlel pants 22h hee ee ee EE Ee 4 EE 5 eebe 5 Test Administrator Tools ccccceeeeesssssssnseeeeeeeseeeeeessnneeeeeesseooeasnneeeeseseoooaennas 5 Proceed TEE 5 Performance evaluation cccccccssseeesseeeeeeeeseeceesnseeeeeeeeseecoeasaneeeeeessooooansnaeeeseeseooes 6 Participant debrieting enee eee ee 6 Results RE ET 6 Satisfaction Results ek geess Nee eege SANS codecedeccasecusccavaodecceuesds g ie gei gie eier 8 Discussion 9 Appendix A Prescreening questionnaire 11 Appendix B Scenario 12 Appendix C Tasks 13 Appendix D Post test questionnaire 14 Appendix E Script for test administrator 15 Appendix F Log of actions performed Error Bookmark not defined References 17 Susa
17. t when dealing with feedback of completed functions which will be discussed later Test Facility The usability test took place in Room 201 of Loew Hall at the University of Washington campus This room was a regular classroom that already contained a TV VCR setup in place The Mitsubishi HS U500 VCR was connected to the existing TV All on screen menus were displayed on this TV All tests were performed by the participant and tester with no other room occupants Test Administrator Tools For performance of this test the tester used the following materials see Appendices e Pre test questionnaire The participants were screened when they arrived for the test by use of a pre test questionnaire e Consent form Participants completed this form to provide their consent for use of their voice and image in report s of this test e Scenarios A scenario sheet was given to test participants for performance of the two tasks This scenario identifies their role and what their motivation is in performing tasks e Task sheets Two tasks were presented on two different sheets of paper 1 Set the time on the VCR and 2 Program the VCR to record a future broadcast e Post test questionnaire This form allowed participants to share their opinions on the product s usability e Tester script The tester script was used by the tester for interaction with the participant e Log of actions performed A log sheet for Tasks 1 and 2 was used takin
18. to display English as my language on the VCR Pick up manual perform task while using talk aloud protocol Do you understand how to use the talk aloud protocol Please let me know if you would like to practice before we begin Let s move on to the actual usability study First am going to ask you to complete a consent form This form states that you are giving me permission to use the video tape of this session for purposes of my usability report only Please let me know if you have questions about the consent form Hands form to participant After consent form is completed Next would like you to complete a pre test questionnaire This form helps me determine what your skill level is Please let me know if you have questions as you complete this form When form is returned Thank you We will now begin the actual study am now going to give you a scenario description This will describe what role you will be playing and what you want to do in that role Please read the scenario and let me know if you have any questions about the role you will be playing Let me know when Susan H Kaltenbach TC 517 Page 15 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation you are finished reading the scenario Participant indicates that they are done reading the scenario This is where you learn what tasks you need to perform in your role as the hockey fan am going to g
19. ure broadcasts you will first need to set the correct time on your VCR During this task remember to think aloud as you work 1 Change the time on the VCR to the correct time now NOTE You can approximate what time it is right now 2 Use the owner s manual the remote control or any other materials you have on hand for instructions 3 Do not ask me your tester to give you help following the instructions This task tests whether the materials are sufficient to set the VCR time by yourself only Please tell me when you have completed task 1 Do not proceed until ask you to move on Task 2 You are now ready to program your VCR to record the Stanley Cup match which will be broadcast in the future The information you need to program the VCR is June 10 2001 Time 8 00 pm Channel Channel 13 1 Program the VCR to record this future program 2 Use the owner s manual the remote control or any other materials you have on hand for instructions 3 Do not ask me your tester to give you help following the instructions This task tests whether the materials are sufficient to program the VCR by you only Please tell me when you have completed this task Susan H Kaltenbach TC 517 Page 13 12 12 2001 Usability Study Mitsubishi HS U500 VCR documentation Appendix D Post test questionnaire This post test questionnaire gives me information on how you feel about the Mitsubishi HS U500 VCR documentation For
Download Pdf Manuals
Related Search