Methods and Metrics
Contents
1. Subjective Surveys Subjective surveys were administered at the end of each usability test session Questions were presented to the participants on the screen of simulator while they are simultaneously read aloud by the observer When applicable a Likert Scale will accompany questions on the screen Responses were recorded via audio and video as well as through notes taken by the researcher The participant was given 10 minutes to respond to the survey questions PILOT While seated inside the car with the engine on the moderator asked the questions verbally and the participant answered verbally Survey by User Type New and Alternative Hybrid User e What did you think of the hybrid display e If applicable Why didn t you play around with the car or the manual before starting to drive e Did you find yourself more aware or concerned with your mileage than you normally would be e How do you feel about the placement of the screen itself How easy difficult was it to find information on the display while driving e Of the screen options you saw on the display which one was the easiest to read Why e Of the screen options you saw on the display which one would you least use Why e What did you think of the tracer display e Was there anything in particular you strongly did not like e On a scale of 1 5 please rate your concern for the car s fuel consumption while driving the car 1 very little concern 5 high concern e Ona2. averaged 30 35 minutes The order of session activities varied depending on the participant type as follows Novice User o 2minutes Sign in amp calibrate recording devices o 5 10 minutes Goal based driving acitivity o 10 minutes Task based activity o 10 minutes Post test survey amp participant feedback Prius User o 2minutes Sign in amp calibrate recording devices o 15minutes Interview and Task based activity o 5 minutes Goal based driving activity o 10 minutes Post test survey amp participant feedback e Alternative Hybrid User o 2minutes Sign in amp calibrate recording devices 5 minutes Goal based driving activity 10 minutes Task based activity 5 minutes Free play with interface 10 minutes Post test survey amp participant feedback O O O 0 From the Full Report Each individual protocol can be found in the last section of the Protocols in the in Appendix F New User Appendix G Prius User amp Appendix H Alternative Hybrid User From the Full Report The modifications to the protocol for the pilot test are noted in Appendix Consent Form All participants signed an informed consent form prior to taking part in this usability test The document was presented to the participant upon arrival and was read aloud to them by the researcher as they read it to themselves Within the informed consent form it was explicitly stated that the participant had the right to leave the study at any time for any re
3. basic instructions about how to start the car The first exercise was a free form task A manual was left within reach The user was told to interact with the car in whatever manner they choose and that they were free to explore and perform any driving tasks they Simulator set up in the may choose lab room The researcher then returned to the room and instructed the user on the next task after the designated time is up The next exercise was a goal based activity The participant was given instructions about a local route they Will have to drive on the simulator The participants were told they have flexibility in how they would drive the route and to perform the driving tasks are they normally would in a car T Ml a Participant drives the The researcher then left the participant to complete the simulator in the free form task Upon completion of the goal based portion the session and the researcher returned to the lab room and began the task designated route for the based activity goal based activity Here the participant was led through the features of the interface answering questions posed by the researcher using their understanding and perception of how the interface functioned The researcher then informed the participant that the session activities were complete and that their feedback would be collected The simulator then brought up the survey questions The participant then read the screen In addition th
4. there can be zero accidents Measures of Behavior Ease of learning and remembering where information is diplayed Number of features identified in Task based activity e Efficient to use Amount of time and number of screen changes needed to find information on Task based activity Positive and Negative Responses Tabulation of all behaviors observed including general body posture words spoken honking and quantifying of any repeated behaviors by individual and across participants based on both Goal and Task based activities Measures of Satisfaction The Post test Survey will ecilit user feedback on the interface in terms of e Information clarity and acceptability Fun to use and Purchase worthiness e User preferences and grading of different screens User suggestions on improvements to interface e User perception of their task performance PILOT Sources of Data Information recorded on 3 cameras located in the car and researcher logs for each individual based on appropriate protocols one recording answers and the other behavior Measures of Performance e Situatuational Awareness Probes vis a vis interface Novice and Alternative Hybrid Users protocols only included 1 What is the current miles per gallon ask twice at different stages 2 How fast are you driving 3 Is the car running on gas or electric 4 How much energy is currently left on the battery 5 What is your current miles per hour Fuel Efficiency kno
5. Toyota Prius Hybrid Display System Usability Test Report EXERGY MONITOR Methods and Metrics Author Shobana Subramanian Contributor Daniel Nissenbaum Table of Contents Methods and Metrics Table of Contents Methods and MOeLtFr1CS itae iiie ERIT IER Ru EH HESES EUH GEH ERU H RESET XRERSRS EE RH EREFLDSCE ERR ELS ERAS D eR ERS RS UR R RR nne nnnn nenns 2 Me thods and Pe OCC MM essit in iteuv user DIE iS Datum Stn MD Du use Neri u A IE ensaia 3 Partemant IHIOEIISDO Bonsai nU E tre UDS EIDEM DEINDE M IE E ME 4 aloe CODA UP EIN So 7 POCTE t CONN U AO ubirnon d ran no ene MESE EM EN E MM 9 eei loda Medea ers a LOT K RP 12 po So OMIT J 19 Mo Cd dg E H m 21 General Goals and TaS OR E m 22 ESAT VMS Ue S oeceseesivsesiusga tok cust rtrpm A A E N A E E URINE NE INSIDER EPUM ERIS EEUU 23 PMN STS EDUC Bunsnirsepes ceni cerco E E ENIE TUNES LINER s NU MN SSE ETE DN E eEE TEE NEUE 25 Methods and Procedure Usability testing included a Final and Pilot phase FINAL In order to understand the ways in which different types of people interact with the interface we tested using 3 different groups of people People who have never driven a hybrid people who drive a Prius and people who drive another hybrid other than the Prius By dividing up t
6. ason but that uncompleted sessions would result in a loss of the 100 compensation For the PILOT the consent form process was followed as well A sample of the Informed Consent Form can be found in Appendix E of the Full Report General Goals and Tasks FINAL There was a number of general goals and tasks that were asked of the participant throughout the session During the Free form section of the usability test goals and tasks included e Exploring the interface Optional explore the user manual General goals and tasks during the Goal based section of the usability test included e Driving from Point A to point B e Providing various readings while driving including o Current miles per gallon o Current miles her hour o Whether car running on electric or batter o Amount of power left on battery Maintaining safe driving behavior and situational awareness e Attaining a high level of fuel efficiency General goals and tasks during the Task based portion of the usability test included e Providing various readings including o Odometer o Average miles per gallon o Average miles per hour o Distance until empty fuel tank e Describing to the resarcher how to navigate to specific screens e Describing to the researcher what information is presented on certain screens of the interface General goals ands tasks during the survey and feedback activity included e Describing to the researcher preferences points of complications and reco
7. comfortable do you feel using the other 2 screens On a scale of 1 5 how acceptable was the Hybrid System Indicator Screen 1 not acceptable 5 most acceptable Why On a scale of 1 5 how acceptable was the Energy Monitor Screen 1 2 not acceptable 5 most acceptable Why On a scale of 1 5 how acceptable was the Interval Consumption Screen 1 2 not acceptable 5 most acceptable Why On a scale of 1 5 how acceptable was the Past Record Screen 1 not acceptable 5 most acceptable Why On a scale from 1 5 how valuable to your driving do you find the information displayed to be 5 high What if anything would you add to the interface Why What if anything would you remove from the interface Why Are there any functionalities that you have learned at some point but forget or neglect to use while driving How would you rate the interface design of the Prius Having owned and driven this car could you describe to me how the hybrid system works Is it important for you to know how the hybrid system is working at any given moment Why or why not Any additional concerns that have not been previously addressed
8. dio and video of the session and requires the participant to not disclose anything they would learn and observe during the sessions with another party All recording instruments are switched on in the Car What follows is an overview of 3 separate protocols for each kind of user Appendiix has the modified protocols Participant gets settled into the test car Novice User Procedure The participants were told they would be interacting with a Toyota Prius and will be required to do driving related tasks They were asked to perform the driving tasks as they normally would following normal driving rules They were reminded to wear seat belts and asked to adjust any car settings as per their needs The participants were given basic instructions about how to start the car They were told to take their time getting to know the car and when they were ready to take a few rounds around the parking lot The car manual was left within reach The goal based activity commenced when the participant started to drive The moderator gave basic instructions about how to navigate around the parking lot as required As the participant drove at set intervals the participant answered s Re d questions to situational awareness probes A gt Za Participant answers probes during driving Upon completion of the goal based portion the task moderator asked the participant to park the car in a parking spot The task based acti
9. e researcher verbally read out the questions twice The participant responded verbally Researcher asks the participants questions about the interface and their subjective responses Alternative Hybrid User Procedure The procedure was the same for this user with the exception that the goal based activity was completed first followed by the task based freeform activities and finally feedback gathering The feedback session included additional questions Prius User Procedure The Prius User procedure included similar activities but with a focus on more complex information and tasks The researcher was in the lab room for all the activities providing instructions and asking questions The participant was given a full view of the interface at all times with no interruptions while tasks were being Researcher with Prius completed user Interview about the interface followed by goal based driving activities on the simulator The session started with an interview and task based activities with the participant answering questions involving a complex experience with the system This was followed by the goal based activity with specific driving instructions including directions to drive in a manner that would be considered fuel efficient The procedure then ended with a feedback session conducted in a similar fashion as the other user protocols but with a focus on display preferences and value of the interface for t
10. ected to 20 20 Availability on the testing dates Reliable transfortation to the testing facility Other data collected from potential participants included e Age e Gender Type of car e Average amount of driving per week Experience driving a Toyota Prius or other hybrid car e Driving tendencies uses A C follows speed limit etc A sample of our participant screener can be seen in Appendix C in the Full Report Participant Recruitment Pilot The pilot test relied on a pool of volunteers aquainted with the researchers Final Our participant recuitment advertisements asked for participants to help out for consumer research on a hybrid car and welcomed all current and non users It explained that the test was to be conducted within the safety of a simulator and will of course be fun Compensation was offered to participants who qualify and complete all session in the study Length of individual sessions was descrbed to the potential participants Contact information was also provided Our recruitment mainly took place through local posting websites such as Craigslist org as well as local print ads Advertising was also done online via the Prius forums www priuschat org forums Sample recruitment ads can be found in the Full Report in Appendix B Reminder emails were sent to the participants 48 hours before their session time An example of a reminder email can be seen in Appendix D Participant Compensation For the Final
11. he participants this way we were able to detect different levels of knowledge comfort with the system and ability to use the interface to attain the goal of high gas mileage All three groups were given similar goals and tasks in the usability test with more detailed questions being asked of the Prius users While all three groups recieved similar tasks and goals they were presented in a different order depending on which user group they belong to In order to gain information from realistic driving conditions while providing a safe controllable environment the full usability test was conducted in a simulator rather than a traffic lot By using a simulator the researchers were able to control 15 30 minute long traffic conditions for each session A simulator also allowed for easy collection of a larger range of objective data PILOT To test the viability of the test procotols and procedures a pilot was first conducted The pilot had all the same goals and tasks but used a smaller number of participants The participants were also aquaintances of the reseachers and were not compensated thereby eliminating recrtuiment and screening related costs A test car owned by one of the researchers was used in lieu of the more expensive car simulator Recording devices were also be drawn from the reseachers current resources Participant Information Participant Make up FINAL Table 1 Participant Type Distribution in Final Test Participa
12. he user Check Out When the sessions in the lab room were finished the researcher thanked the participant for their participation and lead the participant to the reception area Here the participant received their compensation and was asked to sign a payment log confirming receipt of payment See Appendix J in the Full Report for sample payment log The researchers then saved all recordings and begin set All sessions completed up for the next participant Recording Participant s entire interaction with the simulator was recorded by the driving simulator software Camtasia screen recorder eye tracker software and the cameras The verbal response of the participants was recorded on the researcher s log on the individual protocols and the cameras and microphone PILOT The modified session procedure used for the Pilot are as follows Pre Session Set up Before the participants arrived for any session the researchers verified protocol and that the test car and recording equipment is set up for the new participant m e Test Car in lot being set up Initial Set up with Participant Introductions were made and the participant was greeted in the parking lot The participant was requested to sit in the driver s seat of the test car Participants were verbally given an introduction to the sessions and activities involved The participant was asked to sign a consent form that gives the client consent to record au
13. ipants arrived for any session the researchers verified that the test lab equipment was set up for a new participant Initial Set up with Participant One of the researchers greeted the participant in the reception office The participant was asked to sign a consent form see Appendix E in the Full Report that gives the client consent to record audio and video of the session and requires the participant to not disclose anything they would learn and observe during the sessions in the lab with another party Participants were verbally given an initial session introduction All recording instruments were then switched on in the simulator and its peripherals and the participant is taken to the test lab What follows is an overview of 3 separate protocols for each kind of user Each individual protocol can be found in the last section of the Protocols in Appendix F New User Appendix G Prius User amp Appendix H Alternative Hybrid User in the Full Report Novice User Procedure In the lab room the participant was told they would be interacting with a driving simulator and would be required to do driving related tasks They were then asked to perform the driving tasks as they normally would following normal driving rules The participant was then seated in driving seat of the Reception Area simulator The participants posture was adjusted so it synchs with the eye tracker and the cameras The participant was given
14. lex experience with the system This was followed by the goal based activity with specific driving instructions including directions to drive in a manner that would be considered fuel efficient Survey interview and feedback gathering finishes off the sessions The procedure ended with a feedback session conducted in a similar fashion as the other user protocols but with a focus on display preferences and value of the interface for the user Check Out When the sessions were complete the researchers thanked the participant for their participation The researchers checked all recordings and V began set up for the next participant N All sessions completed Recording Participants entire interaction was recorded by 3 cameras The verbal response of the participants was also recorded on 2 researcher logs One capturing behavior and time stamping key sections and other log recording answering participant s answers to questions posed The back seat research team Session Details Session Activities The session activities were ordered differently for each of the User Groups but typically consisted of five sections e Introduction consent form Upon arrival participants signed the informed consent form After signing they were brought into the simulator room where all equipment was callibrated to the specific participant e Free form Participants were given basic instructions were then left to interact with
15. mmedations PILOT For the Pilot the same goals and tasks were followed with the following exception The Goal based tasks were conducted in a parking lot Usability Metrics FINAL Sources of Data Information recorded on cameras and microphones located throughout testing area Camtasia screen recordings traffic simulation software eye tracking recordings and researcher s log on indivual protocols Measures of Performance All performance measures were based on the Goal based activity and include e Situational Awareness vis a vis interface o While driving answers to probes on current miles per gallon miles her hour and whether car running on electric or battery amount of power left on battery measured by accuracy number of glances and time taken e Overall Situational Awareness and Allocation of Attention o Overall number of glances towards interface Measured against standard 2 7 glances is maximum allowable o Driving Accuracy Lane control staying close to center of lane maintaining safe distance between cars Turn Exit Errors Lane Change Execution signalling checking traffic executing smooth transitions between lanes without jerks or mishaps Accidents e Fuel efficient Behavior Number of times Amount of time ECO light appears Amount of time power charged CHG versus PWR on HybridSystem Indicator Average miles per gallon on trip e Time taken to complete tasks e Task success rate Measured against standard
16. nt Type Number of Participants No Prius Experience ewe o 06 Total Number of Participants 20 PILOT Table 1 Participant Type Distribution in Pilot Test Participant Type Number of Participants No Prius Experience Total Number of Participants 8 Participant Groups Novice User Participants who have never driven a Prius Prius User Participanst who have driven a Prius for at least 1 year e Alternative Hybrid User Participants who have driven a hybrid other than the Toyota Prius for at least 1 year This group of users may include some participants who have driven a Prius at one point but have not driven one regularly in the past year Certain characteristics were spread evenly across the three participant types in order to counterbalance them These characteristics included e Age e Gender A list of participants which group they belong to and what protocol they were given is shown in the Full Report in Appendix A e Amount of driving done per week PILOT The make up of the group was as follows Amount of Driving per Week Driving Tendencies Female 3 37 969 25 50 41 85 miles per week day per N 4 Windows down Male 5 6396 Avg 35 6 participant N 4 A C on Participant Screening Document In order to be included in this study portential participants had to meet a basic criterion of e A valid driver s license e Knowledge of the area San Jose CA amp surrounding areas 20 20 vision or corr
17. scale of 1 5 please rate how distracting was the hybrid display 1 not distracting 5 very distracting e If above is yes what did you find the most distracting Ona scale of 1 5 how confident are you in driving this car after this experience 1 low confidence 5 high confidence e Ona scale of 1 5 can you rate how appealing the interface was for you 1 unappealing 5 very appealing e On a scale of 1 5 can you rate whether you thought it was fun to drive the car 1 no fun 5 great fun e Having now driven the car could you describe to me how the hybrid system works sitimportant for you to know how the hybrid system is working at any given moment Why or why not e f you could change the display what information would you take out What information would you add Would the hybrid display system affect your decision to purchase a Prius In what ways Any final thoughts Additional Probe for Alternative Hyrbid User How would you compare the Prius to your current Hybrid What are the Pros and Cons Current User How do you feel you did with the tasks Why How do you feel about the placement of the screen itself How easy difficult was it to find information on the display while driving Is there anything you find distracting on the interface What improvements would you make to the hybrid display What is your overall satisfaction with your Prius Of all the screens which is most important to you Why How
18. test participants were given 100 cash at the end of the completed session If the participant did not show up for their session no compensation was given If a participant abandoned their session for any reason compensation was not given Lab Configuration The Simulator The usability test was carried out via a driving simulator A simulator was used in order to be able to effectively control road conditions traffic as well as providing a much safer environment for the participant than driving the real car on the road The simulator was equipped with a Toyota Prius seat steering wheel pedals and a fully functional instrument panel In place of the windshield will be a screen onto which road traffic was simulated Driving Simulator The Observers When the researcher is not in the room with the participant they were in an adjacent room behind a one way mirror observing the sesssion Recording Devices e A Camtasia screen recorder was used to record the interaction with the hybrid display system Acamera was placed on top of the simulator facing the participant to record facial expression e A second camera placed on top of the driving seat over the right shoulder of the participant was focused on the buttons on the steering wheel used to change screens in the interface and also had a view of the interface e An eye tracker that can be installed on the computer was used to record eye movements Microphones were placed on
19. the car simulator on their own Exception Not used for Prius User e Goal based activity Participants performed driving tasks on the simulator in car in parking lot Task based activity Participants were led through the features of the interface performing feature identification and related tasks one task at a time with all participants following the same logical temporal order for each type of protocol e Post test survey and Open ended feedback Participants verbally answered a survey on their overall subjective experience with the interface Session Timing FINAL Each participant session lasted exactly 1 hour The order of session activities varied depending on the participant type as follows Novice User o 10 minutes Sign in amp calibrate simulator o 10 minutes Free play with interface o 20 minutes Goal based driving simulation o 10 minutes Task based activities o 10 minutes Post test survey amp participant feedback Prius User o 10 minutes Sign in amp calibrate simulator o 25 minutes Interview and Task based activities o 15minutes Goal based driving simulation o 10 minutes Post test survey amp participant feedback Alternative Hybrid User o 10 minutes Sign in amp calibrate simulator o 20 minutes Goal based driving simulation o 10 minutes Task based activities o 10 minutes Free play with interface o 10 minutes Post test survey amp participant feedback PILOT Total session time
20. the steering wheel of the simulator to record verbal information from the participants All information from the simulated windshield was recorded by the simulator software e D Video Cameras Eye tracker installable on computers Microphone Pilot Test Configuration Parking Lot The testing was done on weekends in an empty parking lot of an office complex located at the interection of Brokaw Road and Oakland Road in the north San Jose CA The location was easy to find Intersection of Brokaw and Oakland San Jose CA Test Area Empty Parking Lot Se eee Eat ir 1 1 Senor Views of the Parking lot A 1 4 mile long full loop of the parking was used for the driving portions of the test Recording Devices Video cameras were used to record the session activities A video camera was placed on top of the dashboard in a corner to avoid interference with driving and facing the participant to record facial expression A second video camera was set on the front passenger s seat focused on the buttons on the steering wheel used to change screens in the interface and with also with a view of the interface The third camera on a tripod was set up on the back seat floor which captured the the interface itself p N LA 5 21 B le j wm l D Se Session Procedure FINAL The following procedures were used for all participants from each user group Pre Session Set up Before the partic
21. vity was conducted next Here the participant was led through the features of the interface answering questions posed by the moderator to test their understanding and perception of how the interface functions The moderator pointed to particular screens and asked questions sometimes twice to make sure the participant understood The moderator then informed the participant the session activities were complete and that their feedback would be collected The researcher asked questions out aloud sometimes twice to make sure the participant understood The participant responded verbally Alternative Hybrid User Procedure The procedure was the same as the novice user with the following exceptions The alternative hybrid user completed a free form activity for 5 minutes where they explored the display screens by themselves with the car in a parked mode The final survey included questions comparing their experience with their current Cars Participant answers questions about the interface in parked mode The order of activities was also different the goal based activity came first followed by the task based the freeform activity and finally feedback gathering Prius User Procedure The Prius User procedure included similar activities but with a focus on more complex information and tasks The procedure started with an interview and task based activities with the participant answering questions involving a comp
22. wledge with Situtational Awareness Current Users only probes included 1 Begin Driving in most efficient mode EV 2 Switch to ECO mode 3 Check if we are running out of gas 4 What is the Current MPG 5 Go to the energy monitor display Drive to show that the flow of energy is going from the battery to the electric motor to the gas engine 6 Next show the flow of energy going in reverse from the gas engine to the electric motor to the battery 7 Drive in most effective way to recharge battery The data was tabulated for each probe above are as follows o Accuracy of responses o Number of glances towards interface measured against standard 2 7 glances is maximum allowable Eye Pursuit and Head Turns counted o Time Taken to answer Measures of Behavior Ease of learning and remembering where information is diplayed Number of features identified in Task based activity Positive and Negative Responses Tabulation of all behaviors observed including general body posture words spoken and quantifying of any repeated behaviors by individual and across participants based on both Goal and Task based activities Measures of Satisfaction e In the Post test Survey users assessed the interface on the basis of o Information clarity and acceptability Fun to use and Purchase worthiness User preferences and grading of different screens User suggestions on improvements to interface User perception of their task performance O QO O O
Download Pdf Manuals
Related Search