Home

User tests of the final demonstrator applications - Certec

1. sseesssssss 93 Table 3 6 1 5 Ratings of the Toolkit with significance tests for positive ratings 94 Table 5 1 4 Terrain Navigator requirements X designates partial fulfilment not AIIE S MP d MO PCM ron rn 105 Table 3 5 4 Event Guide JUICY BEATS functional Requirements 106 Table A 1 Functional requirements from D 5 1 sssseessussss 109 Table A 2 Non functional requirements smmsessssssrrrrssrrrsserrrrrrrrrrrnnnnnrrrrrr rr enn n rr nn nn nn rna 110 Table A 3 Goals of user testing smnsmnsssessrrrssrressnenerrrrrrrrrnnnnnrnrrrrrrrrrnnn rn n rr rr nr rr rn nn nn rna 110 Table A 4 Extracted requirements table subset of Table 1 and table 2 111 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Executive summary This document covers two main topics a description of the final and summative user field tests of the span of HaptiMap prototypes which are implemented using the HaptiMap programming toolkit and descriptions of user tests carried out to evaluate the use of that toolkit by students and professional developers The description of the end user tests of demonstrators contains an overview and details of this evaluation with respect to the claims and requirements formed in the earlier stages of the HaptiMap project D5 1 The demonstrators spanned historical site guides to navigation tools for blind and par
2. eeesssssss 27 Table 3 1 8 Subjective user rating of the Joined app older users 27 Table 3 2 2 Specifications related to route guidance traversal 29 Table 3 2 3 Characteristics of the evaluation sessions eseesssssss 29 Table 3 2 5 3 Session 1 PaITUCIDAULS arret oet er P xe e bee aerario 32 Table 3 2 5 5 1 Usability scale results ccccccccccccccecececeeeeeeceeeeeeeeceeeeeeeseeeeeeeeeeeeseess 33 Table 3 2 5 5 2 NASA TLX test results unmnssssssssesrrrssreennnnnnrrrrrrrennnnrrnrrrrrrrrrrrnnnn rn r rr rn nn 34 Table 3 2 5 5 4 Semi structured iNtervVieWS sssrrssssesseerrrrrrsrrennnnnrrrrrrrrrnrrrnnnrr nr rr rn nn 34 Table 3 2 5 5 3 Specifications related to the execution of a particular route in open SMNVIF NN ies ht cs Pesca ota tecta l rd EN N E 35 Table 3 2 6 1 Session 2 sample icc eee ote eee eee ee eee 36 Table 3 2 6 3 1 Usability questionnaire results ccccccccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeess 39 Table 3 2 9 Session 3 participants ssmnmssssssseseesrrrrsrrennnnnrrrrrrrrrnnnnrrrrrrrrrrnn nns rr rr rr nn nan 41 Table 3 2 7 5 3 Session 3 participants parts A F mnmsssmmsrrsssrrsssererrrrsrrennnnnrrrrrnrrrnnn nn 44 Table 3 3 1 1 Functional requirements from D 5 1 eesussssss 52 Table 3 3 1 2 Non functional requirements ccce 52 Table 3 3 1 3 Goals of user
3. Present upcoming interesting points within the route Store route data for several routes The user may ask the system to identify its current position Route may be downloaded with ease 3 2 3 Evaluation conditions Due to the complexity of working with visually impaired people tests have been divided in various sessions The characteristics of each session are collected in the Table 3 2 3 Table 3 2 3 Characteristics of the evaluation sessions Type of exercises Type of users involved yp SCIO Session Date N users Totally Partially Deaf Open Blind Sighted Blind Space Trog Session 1 4 X X X X Session 2 2 X X Session 3 4 X X X 3 2 4 Outline of evaluation protocol for end users Informed Consent Form Usability Scale NASA TLX task load test Specifications related to the execution of a particular route Global Functionality for Navigation Semi structured Interview Common HAPTIMAP questionnaire 3 2 5 Session 1 3 2 5 1 The tests were done in three phases 3 2 5 1 1 Introducing the participants with the system indoors 29 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Figure 3 2 1 Familiarization phase The objective of this phase was to get the user familiarised with the device with the application and with the two interactive modes tactile gestures and sound voice The different ways in which the applicatio
4. PRINCIPLE SIX Low Physical Effort The design can be used efficiently and comfortably and with a minimum of fatigue Guidelines 6a Allow user to maintain a neutral body position 6b Use reasonable operating forces 6c Minimize repetitive actions 6d Minimize sustained physical effort PRINCIPLE SEVEN Size and Space for Approach and Use Appropriate size and space is provided for approach reach manipulation and use regardless of user s body size posture or mobility Guidelines 7a Provide a clear line of sight to important elements for any seated or standing user 7b Make reach to all components comfortable for any seated or standing user 7c Accommodate variations in hand and grip size 7d Provide adequate space for the use of assistive devices or personal assistance Notes for use e The principle 7 concerning size and space for approach and use might need adaptation for a mobile context e Instead of ability to reach the device more relevant might be to check if the grip is stable enough to for use while moving e Instead of providing adequate space for assistive devices checking if it can be used with busy hands i e using a cane or with a wheelchair and the body is moving Diverse conditions of use Go back to the list of conditions in the introduction For each of those please state if your application will or will not work You might need to specify if this answer is conditional to specific events o
5. The app was tested with the explore mode in focus during the city tests in Spring 2012 All users succeeded in using the explore mode in the learning phase and all users except one used it to explore freely the historical points they chose Thus we have fulfilled these goals F7b Possible to select objects by pointing to the objects in real life The explore mode lets users physically point in different directions and select points The information about point can also be read from a distance This goal is fulfilled NF1 Fun In the word selection task it was seen that words such as Fun and Stimulating were rated highest Thus we have fulfilled this goal NF2b Possible to use together with others This requirement probably had its origins in a wish to connect devices and the game between devices As such the app is not possible to use in collaboration However there is no barrier to using the app together either with one phone each or by handing the phone back and forth Since the goals are GPS points with a radius they will not start exactly at the same time but the precision is sufficient for goals to be met within the same 1 2 seconds if you walk together and thus you have a shared experience This goal is considered partially fulfilled therefore the parentheses in table 4 F13 new Basic functionality available without vision This requirement can be connected with G2 and G3 The entire application was indeed usable without
6. eese 131 Appendix H Protocol for HaptiMap toolkit evaluation eere 133 Appendix I Word list for word cloud exercise eese 134 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix J Haptimap Developer Evaluation Instruction Sheet 137 Appendix K Parameters and Notation for NavEscort eerte 138 APPENDIX L ML questionarre ssssssssssssssassssasassannsnanannanannnnannn nn nn nan nn KARA S ARKA AR RR RAR RR KR RR 140 APPENDIX M Questions used in professional developer evaluation 141 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications List of Figures Figure 1 2 1 Joined user interfaces smnssessssssseersrrrrrrresnennerrrrrrrrrnnnnrrrrrrr rr rn nn ner r rr nn nn 13 Figure 1 2 2 urban routes for NavEscort tests ssmnmnssssssssesrrrrsrrrrnnnnrrrrrnrrrrnnnnrrr rr nn nn 14 Figure 1 2 3 Time Machine User Interface Screens Splash screen trail point screen ANA Map SCIGGils sissies cust eect 14 Figure 1 2 4 Terrain Navigator sample screen shots ccccceeeeeeeeeeeeteeeeeeeeeeeeeees 15 Figure 1 2 5 Screenshots from the Juicy Beats event guide 15 Figure 3 1 1 Joined screen Shols cec e oc pa di wat dia d ie rr rr mido 23 Ta
7. Nr Requirement In TM Possible to use both on your own and together Possible to check up share afterwards ES Possible to plan in advance NF5 Basic functionality available also without vision Easy to change between modules Easy to create trails and games IEEE Based on the functional and non functional requirements above and taking into account not the available functions but the use of the functions in a wayfinding activity these are the set 52 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications of usability goals that we aimed for with the Time Machine concluded with a column showing if the requirement has been met according to the results Table 3 3 1 3 Goals of user testing Nr Basic usage Fulfilled Sighted users who have Android experience will be able to navigate the interface Users with visual impairment and smart phone experience will be able to navigate the Not tested interface using an Android phone with physical buttons after being asked about their preferred confirmation method select on release or double tap except starting and stopping the application Users will be able to detect and understand the haptic guide concept when introduced to it Users will be able to follow a trail and answer the questions for the different points on the YES trail in a real context G5 The app will be usable both for hiking and for city tours YES e 7 YES A wide a
8. 133 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix I Word list for word cloud exercise Translation to Swedish Tillg nglig Avancerad Tvetydig Irriterande Tilltalande Tillm tesg ende Lockande Omstandlig Trakig Ljus Business like Upptagen Ren Tydlig R rig vertygande Komplex Helt ckande F rvirrande Konsekvent Mots gelsefull Styrbar Bekv m O intuitiv Kreativ Trov rdig Toppmodern Omodern nskv rd Sv r Distraherande Trist L ttanv nd Verkningsfull Effektiv Utan anstr ngning Sj lvst rkande Energisk Accessible Advanced Ambiguous Annoying Appealing Approachable Attractive Awkward Boring Bright Business like Busy Clean Clear Cluttered Compelling Complex Comprehensive Confusing Consistent Contradictory Controllable Convenient Counter intuitive Creative Credible Cutting edge Dated Desirable Difficult Distracting Dull Easy to use Effective Efficient Effortless Empowering Energetic 134 HaptiMap Deliverable D1 5 PU Engagerande Underhallande Sp nnande F rv ntad V lbekant Snabb Defekt Flexibel Fr sch V nlig Frustrerande Rolig Sv ranv nd H g kvalitet Ologisk Imponerande Otillr cklig Obegriplig Inkonsekvent Ineffektiv Innovativ Os ker Skr mmande Intuitiv Irrelevant Meningsfull Vilseledande Motiverande Ny Icke standard Dunkel Gammal Vanlig V lorganiserad verv ldigande ve
9. 3 4 4 Heuristic Evaluation We used the Universal Design Principles checklist to ensure the usability of the Terrain Navigator As the Terrain Navigator includes the audio functionality in addition to the visual maps the application suits users with diverse abilities such as users with reduced eyesight If the users have not been using an iPhone or any touch screen phones before some time might be needed for learning Unnecessary complexity has been removed from the basic user interface but the application includes various options to adjust it for individual preferences 3 4 5 NASA TLX task load test We asked all the eight participants in the first part of the evaluation to fill in a NASA TLX task load test just after the evaluation tasks with the Terrain Navigator For the NASA TLX we used a scale from 1 to 20 The table below shows the average satisfaction with the evaluation of the Terrain Navigator 72 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 4 4 Participants ratings of their subjective task load 14 1 0 mam Mental demand Physical demand Temporal demand Performance Effort Frustration 3 4 6 Analysis of the Semi Structured Interview After the evaluation tasks with the Terrain Navigator and the NASA TLX task load test a semi structured interview was conducted for two participants at a time Two facilitators asked the questions and recorded the answers In the in
10. The system user interface is intuitive and easy to use for the first time user no instruction book needed NFC002 The system is usable for both sighted and visually impaired users NFC010 The system is reliable NFCOI 1 The system is attractive NFC012 The use of the system is socially acceptable like required movements are subtle and enjoyable and cannot interfere with other types of communication Every day life things are more acceptable etc eK x NFC005 The feedback from the system is easy to understand NFC022 The system informs about the user location fast The navigation part of the user hardware gets the first fix fast NFC023 Vibration speech etc do not interrupt hiking unnecessary i e if not really needed NFS024 The system does not allow to accidently download wrong navigational data to the user hardware MD Xx NFC004 Audio messages from the system do not overload the user increase the cognitive workload gt lt NFC016 The system conservers batteries of the user hardware as much as possible NFC003 The system forces the user to communicate with other users FC024 Table 3 4 1 2 The Terrain Navigator s functional requirements User Requirement The system can show the user s location on the map FC013 FC055 The system integrates audio based guiding for indicating route The system informs through the audio if landmar
11. Total 282 3 5 2 4 Discussion and Conclusions An analysis was undertaken of over 5000 uses of the Juicy Beats app at a music festival The app included guiding functionality developed with the HaptiMap Toolkit Over 10 of all uses of the app included use of the guiding functionality The pattern of use of the guiding functionality did follow the overall pattern of use of the app during the festival with an initial increase in use peaking at the beginning of the main festival programme at 16 00 on the 20 07 2011 followed by a decline in use during the main part of the festival and another peak when the main programme ended at 22 00 More detailed analyses were made of the use of the sound Geiger and vibration functionality within the guiding functions About 50 of the time users activated the sound functionality and did not use the vibration functionality at all this had to be explicitly activated About 30 of the time users activated the vibration in addition and kept it on Finally in about 2096 of instances users activated the vibration but then turned it off again From the logging we cannot tell why people would only use sound or turn the vibration on and then off again This can be related to the qualitative interviews where the sound was less liked by the participants than the vibration Follow up interviews with the users would be interesting to clarify further this behaviour However there is also some hint of what
12. At the beginning when I was walking walk through the streets in the urban route because you must know straight on the pavement when to cross Which mode did you think was the most effective Hours of the clock face Sound Vibration 15 2225 1 2 1 1 25 3 so Did you think that any other additional functionality in the system could have improved your execution of the route No What the system does is enough Sometimes it s not very clear when trying to be precise For instance when it says at 2 00 it would be more significant if it said at 3 00 in order to do the turn It should speak out the street names from time to time Higher sound volume for the headphones Did you have any problem when doing the test i e while traveling the route YES 3 NO 1 Ask for help from the experimenters What did you do to solve the problems I stopped and waited for more information Kept trying until I got oriented again When comparing your execution of the route using the system being tested with executing the route as you did before what s your opinion What are the differences What are the similarities It makes you feel much safer It makes me feel safe I would have never tried it without the system because I m very fearful If I had to do it I would ask for help If I was in an unfamiliar place and I didn t have the system with me I would have to a
13. Heuristic BV ALU AION uar tn e taa 72 334 5 NAS A TL X task load teste ete eic cde diuine da or Pero NE 72 3 4 6 Analysis of the Semi Structured Interview eee eene tette ttnnttnntnnttn ttn nnn 73 3 4 6 1 Using the Terrain Navigator on iPhone eene tentent ttn tnnttnntnnttn tnnt 73 3 4 0 2 The Simple Maps trente etre tetendit e ta rep tere aaia 74 E MAG III E 75 3 5 Juicy Beats evaluation cce tnte rere ttm enm ct nan ar RKA RA RAR AN RA RR nac KR RA RR KR AN RA RR KR RA RR KR RR RR nenn 76 3 5 Qualitative evaluation 4 2 sete ee eret rette recent ERT ERA in rer RR RAN REAR 76 3 5 2 Quantitative Analysis of logging data from the Juicy Beats app seen 81 3 6 Developer evaluations t 92 3 6 1 Students in course at Lund University eese tnnt tette tnn tnnt sn rn nnn nn nr nn nn 92 3 6 2 Students at University of Glasgow esee eese ttn tente ttnnttnn tente ttn tien ttt nts tite tentent 95 3 6 3 Evaluation of the HaptiMap Toolkit with professional developers sss 98 4 Analysis Of th results uasa eir rat esa perat Yi np Ix vo nonien COR kk ux cO dr Fa INR FCR p S RC o es RE 104 4I By Eval a tion css C ncc 104 Ae T Vs OVC sua EAE teretes bae EEE 104 4 1 2 NavEscort and mobility trainers isse tte tente tnnt ttn ttn ttn tns 104 4 1 3 The Lund Time Machine eerte nnt t
14. The use domains are so different it is nearly impossible to match results between demonstrators We have captured a count of error help requests and system corrected error states and the qualitative remarks have been noted The most important things to capture have been breakdowns and ah ha experiences The tasks for each demonstrator have naturally been different The initial requirements in D5 1 where the demonstrators were first presented have set the groundwork for creating tasks and the evaluation measures Appendix A presents a walk through on how to set up the final requirements and thus the tasks to test connected to the toolkit features Since mobile use is highly dependent on context we also noted contextual information in some detail Contextual variables related to impairment environment or activity e Biking or walking e Hands needed for a white cane walking dog walking stick e Temperature and use of gloves e Level of noise e Relevant weather conditions o Temperature use of gloves o Sunny glare o Rain Logging In some cases the Virtual Observer or other custom logging methods was used 2 1 5 NASA TLX task load test The raw NASA TLX test provides a validated sore of the workload that the application provides to the end user By getting a summation score for the cognitive and physical workload the evaluators can compare across demonstrators and get a first objective metric qualitative evaluation of the syste
15. 15 minutes after the end of the evaluation It also included a SUS questionnaire a Santa Barbara Sense of direction questionnaire the Haptimap common background questionnaire and the semi structured interview 3 5 1 2 Participants Demographic Information Seven participants filled the common background questionnaire as well as the NASA TLX questionnaire For lack of time the seventh participant marked with could not answer the interview and had to be excluded from the rest of the analysis Three participants were visually impaired The 2 blind participants went accompanied by a sighted person in the festival Table 3 5 1 2 1 Demographics for the qualitative Juicy Beats evaluation Mobile M F Age Vision Hearing Mobility phone GPS use Minor problems Yes Rarely for test 1 IM 30 Blind Minor problems due to vision Nokia purposes only Low 2 F 27 vision Nokia No Yes Once a week to find 3 IM 36 Blind iPhone from A to B Yes Every 6 months in car 4 M 25 Sighted Nokia or pedestrian Yes Once twice a month for SE ME 35 Sighted iPhone work Minor hearing 6 F a aea POP Com Samsun yes Ina car hiking sailing damage g Yes Nearly daily for turn by 7 M 27 Sighted iPhone turn navigation 76 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The 6 participants filled a Santa Barbara sense of direction questionnaire it is shown in the tabl
16. 5 2 3 1 Screen from Juicy Beats app showing guiding function HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Use per hour FP N OU B Ui S S S S S S S y NN OS S S SQ NS Sy WM PS HM LM PH E Total Seq ES E S 40 amp va 3 A 22 Guide un So ec oc Figure 3 5 2 3 3 Total use sequences and use of guiding sequences of Juicy Beats app during the festival 368 of the 554 sequences 66 4 that included the guiding functions included only guiding functions so the user was specifically and only using the app for guiding Two most common sequences were by far the most common The first was e Toolkit Navigation Started e Toolkit Navigation Running e Toolkit Geiger Running This sequence starts the guiding function and then moves to the default option of a visual presentation and a Geiger like sound indicator This sequence accounts for 177 31 9 of all sequences that included guiding and 48 196 of the guiding only sequences From the 85 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications logging data we cannot tell for how long the Geiger function is then activated as there was no logging of when the guiding functions were stopped The second most common sequence accounts for 68 12 3 of all sequences that included guiding and 18 596 of the guiding only sequences This sequence is e Toolkit Navigation Starte
17. 6 3 1 Usability questionnaire results ITEM Did you think application was useful the USER SD1 I didn t understand it very well Vibrations would be better if they had different intensities the short one to be softer and the long one to be more intense User tests of the final demonstrator applications USER SD2 I thought it was useful It provides very good orientation but one must have a lot of patience The sound option was bad because there was a lot of noise on the street Has the orientation been given through the means of vibration been helpful in your case No because I could not understand it properly Only when it vibrated I could understand it I think it would be better to have two vibrations to indicate left and three to indicate right Very well but it should vibrate twice when on a straight line Did you feel comfortable throughout the whole experience Well more or less Yes Do you prefer positive vibration when the device vibrates while you are on the right track or negative vibration vibration only happens when there is a deviation from the correct trajectory direction What enhancements could be added to improve the functionality of the application Negative vibration and two vibrations to indicate right three vibrations to indicate left It could also be one vibration for right and two for left and nothing when one is on the right track Po
18. 8 1 Have you encountered any toolkits similar to the HaptiMap Toolkit 8 2 If yes how would you compare them to the HaptiMap Toolkit both in functionality that they offer and ease of use 9 FINALLY JUST A FEW QUESTIONS ABOUT YOURSELF Your age Gender How long have you been developing mobile apps How many apps have you developed Do you develop for a living as a hobby as part of your studies MANY THANKS FOR TAKING THE TIME TO ANSWER THIS QUESTIONNAIRE WE WILL USE THE RESULTS TO IMPROVE THE HAPTIMAP TOOLKIT 144
19. A F 5 ir e ceca pestered 44 Table 3 2 7 5 4 Semi structured interview results sssseeseessssssss 44 Figure 3 2 9 1 1 TrainerTools SCIeeNSNOt sssssssssrrrrsrrrsseeenrrrrsrrrrnnnnrrrrrrrrrrnnnn rr rr rn nn nn 47 Figure 3 2 9 1 2 A Mobility Trainer working ssmmmsssssrsrrrrrrrssssennrrrrrrrrnnnnnrrrrrrrrrrnnnnrrr rr rn nn 47 Figure 3 3 4 1 Scanning concept in the explore MOCG ssssssrrssrrsssererrrrrrrrrnnnnrrrrrn nn 56 Figure 3 3 4 2 Schematic concept of trail points and trail following 57 Figure 3 3 5 10 1 Result from NASA RTLX rating in the CITY setting by category 63 Figure 3 3 5 10 2 Concentration rating of subtasks Scanning Guiding and using Ls H ec ME Ere E ee 64 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Figure 3 3 5 10 3 Result from NASA RTLX rating in the RURAL setting by category 64 Figure 3 3 3 5 10 4 Concentration rating of subtasks Arriving Guiding and Sound VYING OWS A ioris Het tA aec A hob ncn Ac D Lied 65 Figure 3 4 1 The Terrain Navigator operates on the Apple iPhone 69 Figure 3 4 2 A map of the 2 km walk in the Nuuksio National Park 72 Figure 3 4 6 1 The difficulty of the path would be useful information on the map 74 Figure 3 4 6 2 The participants answered questions regarding the five different maps eratis re ara etd ate etes rote uita ost ie Po FO e ta
20. Consultant highschool 63 Medical Representative retired middle highschool 50 Programmer master 59 Optician middle highschool 58 IC Consultant master 57 6 With regard to means of transportation the second group mostly rely on private cars and walking though not on public transport Table 3 1 6 shows independent travel by older users the answers to question D1 in the HaptiMap questionnaire in appendix B 1 means very often 6 never 26 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 1 6 Independent travel by older Foot Priv Car 6 6 6 6 3 1 6 6 6 6 1 1 6 6 6 5 3 1 1 6 6 6 2 2 6 6 6 6 1 5 Avg 5 6 6 5 8 2 2 The participants were all frequent travellers and with the exception of the senior citizen mostly work related whereas shopping and leisure activities were nearly equal but lower than for the first group Table 3 1 7 shows Activities by users with visual impairment the answers to question D1 in the HaptiMap questionnaire in appendix B I means very often 6 never Table 3 1 7 Activities by users with visual impairment Work Shopping Leisure m m ale Wl NN vo A NiO AN 4 Avg 2 4 3 6 In terms of unknown routes which they all encountered very seldom they all prepared themselves by facilitating paper maps and navigation systems Although t
21. Tasks This document gives guidelines for the evaluation of the demonstrators Those guidelines are to be used by all partners for comparison of the results afterwards Only the parts explicitly marked as optional are not mandatory The mandatory parts should be applicable also for children users Generally the questions should be translated to the language of the region where the test is done The Likert scale is used widely in this test and maybe children can benefit from a more tangible version The subject marks their answer in a to 7 range For the summary evaluation as well as for the test in context you will need to define tasks that are relevant for your demonstrator With 14 different interfaces and task domains it will be impossible to have each group do identical tasks The model for the tasks that the participants will perform can be taken from D1 2 late design prototype evaluations pg 65 so it will be useful to re read that document before designing the tasks for this part of the evaluation The point of this initial evaluation is that These evaluations will evaluate the usability and accessibility of the applications both on its own and compared to the solutions the users currently use and These tests will follow from the D1 2 User studies guidelines for HaptiMap so any field testing should be done from that perspective So from reading the DOW the initial requirement from each testing centre is to provide a brie
22. an excessive amount of effort into the activity 3 5 except one participant who felt that the terrain was hard to cross The rated level of frustration wis more variable The most frustrating experience reported was the loss of vibration signal or the unreliability of it and also the problems arriving at points on several occasions For the three different concentration figures Figure 3 3 3 5 10 4 contrary to the city tests it can be seen that the perceived hardest sub task was to follow the guiding E1 m2 m3 m4 m5 m6 n7 n3 m9 m10 mii E Group mean Figure 3 3 5 10 1 Result from NASA RTLX rating in the CITY setting by category 63 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications E Scanning B Guiding E HUI use 1 2 3 4 5 6 7 8 9 10 11 Mean B Scanning B Guiding m HUI use 1 2 3 4 5 6 7 8 9 1 Figure 3 3 5 10 3 Result from NASA RTLX rating in the RURAL setting by category 11 Mean 64 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications E Arriving B Guiding E Sound windows Pun d i Figure 3 3 3 5 10 4 Concentration rating of subtasks Arriving Guiding and Sound Windows 0 1 2 3 Mean 3 3 5 11 Word cloud The users were asked to check all words in a list Appendix I that they felt appl
23. and audio experiences were not present within the list 3 3 6 Meeting the requirements discussion The usability and functional requirements were set up before the test Each of these requirements deserves a discussion As you will notice some usability goals G coincide with functional requirements F and are treated together G1 Sighted users who have Android experience will be able to navigate the interface 65 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Three out of five users in the final controlled test used Android phones regularly Only the guiding principle and the shake function were explained at the beginning of the test and the first screen was introduced No problems were observed with the visual interface except that one user didn t focus on the screen at all and thus missed the quiz questions entirely Despite this we consider this goal fulfilled G2 Users with visual impairment and smart phone experience will be able to navigate the interface using an Android phone with physical buttons after being asked about their preferred confirmation method select on release or double tap except starting and stopping the application This goal proved itself to be harder to test than expected Only 2 out of 9 participants with severe visual impairment or blindness had in fact used a smart phone In both cases of using an iPhone both claimed not learning to use it Select on release was use
24. and useful The participants found the descriptions of landmarks interesting and wished there would be even more information concerning for example the flora or the geographical history of the area Especially the elderly participants considered simplifying the traditional maps necessary However the accessibility of the paths such as the steepness of the path should be clearly presented 75 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 5 Juicy Beats evaluation 3 5 1 Qualitative evaluation 3 5 1 1 Test procedure The participants gathered at the site of the festival After gathering the tickets and entering the site some time was taken to sign the consent forms and prepare the group A short introduction to the Juicy Beats application was given explaining its purpose as a festival guide The participants also had the opportunity to register other participants as friends so they could later interact through the application The participants were then asked to go on their own for several hours before coming back to the meeting area where the interview would be conducted During this time they should try all the functions of the demonstrator namely explore the festival s program be guided to a scene and be guided to meet some friend The blind users were accompanied by sighted non participants to help navigation in the festival area The interview comported a NASA TLX questionnaire made within
25. applications As can be seen in the figure below Figure 3 3 4 1 a user holds the phone and points with it to scan for historical interest points Within the chosen distance range the pointing direction will decide which of the historical points is read The information read is the name of the point and the distance to it By selecting a point after it has been read the user can be guided to it or ask for more information 3 3 4 2 Rural test trail mode The rural tests have been carried out over a longer period of time 6 months The collaboration with the local cultural historical museum organization has put focus on issues such as the possibility to spread the app and its content the possibility to edit material by lay men as well as remote feasibility testing by laymen Two stages of tests will be reported pre final tests and final tests The pre final test 7 persons uses the same basic app as the city test but the trail module and has only part of its content finished The final test 5 persons has all the content finalized and was a 1 5 km long trail with 19 information points and 12 sound window points In the rural setting there is no routing and therefore guiding is as the crow flies This could pose problems which was discovered at the first pilot test not reported below Therefore there is a special type of point called via point that can be put into the gpx file that composes the trail These via points ar
26. button interaction and by pointing to the objects in real life 109 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications F8 Manage your team add remove members and also get help to find them F9 Allow filtering objects based on distance year and type F10 Not time critical easy to take a pause and restart later F11 Possible to design and experience both on location and from a distance F12 Library of ready made materials available Table A 2 Non functional requirements Nr Requirement NFI Fun NF2 Possible to use both on your own and together NF3 Possible to check up share afterwards NF4 Possible to plan in advance NF5 Basic functionality available also without vision NF6 Easy to change between modules NF7 Easy to create trails and games Based on the functional and non functional requirements above and taking into account not the available functions but the use of the functions in a wayfinding activity a set of usability goals were also set up Table A 3 Goals of user testing Nr Basic usage Gl Sighted users who have Android experience will be able to navigate the interface G2 Users with visual impairment and smart phone experience will be able to navigate the interface using an Android phone with physical buttons after being asked about their preferred confirmation method select on release or double tap except starting and stopping the application G3 Users with
27. cause loss of confidence in the toolkit Keeping tutorials up to date e g ensuring upgraded versions are put on the website and instructions altered as the underlying operating systems are updated will ensure this does not happen Improve discoverability of HCI modules The key issue to arise from our evaluation was that the toolkit provides easy to use and powerful modules to support the integration of multimodal feedback in a lightweight way However discovering what modules were available and how to use them was problematic Participants who progressed to the final part of the evaluation found it difficult to identify the modules available and relied on the bearing module as used in the tutorials Improved visibility of modules should be undertaken 97 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 6 2 4 Conclusions Overall the toolkit provides good straightforward access to support simple integration of complex and difficult to design multimodal feedback techniques for developers In addition it does so in a way that does not overburden the user to design their app architecture to fit the toolkit All of our participants were positive that they would use the toolkit again where appropriate The issues we have identified do not diminish the value of the toolkit but if solved are likely to ensure that users reach the level that our participants did allowing them to understand the value in the toolkit ra
28. experience disability Note Professionals will be asked to do this in their own time They will be encouraged to involve colleagues particularly for the exercise Students can perhaps do this as part of coursework and do the exercise as small group work For the professionals 1 In their own time participants will be asked to do a very simple task with the toolkit just to make sure that they familiarize themselves with it 2 They will be asked to email the solution to that task so we know they have done it to HaptiMap a At least 8 participants will then be asked to do another task with a HaptiMap researcher sitting next to them or on a skype link They will be asked to produce a think aloud protocol They will then be given a short interview about their experience with the toolkit They will also be asked to re take the Attitudes to Disability scale b If we have more participants or for those we can t meet with will be asked to do another task Then complete a questionnaire corresponding to the interview They will also be asked to re take the Attitudes to Disability scale For the Students Participants will be asked to undertake an exercise assignment with the toolkit It would be helpful if the results of the exercise were available so we can see how well participants did Participants will then complete the same questionnaire as the professionals They will also be asked to re take the Attitudes to Disability scale
29. festival map or by selecting items from list views A friend finder allows users to see where their friends are located on the map and also broadcast their current location to selected friends and group of friends A compass mode for navigation is to be used replacing a map and Yes but no turn by turn instructions for guiding visitors to their destinations routing Vibration patterns are to be used to aid users in navigating to Partially not safe previously selected waypoints or locations of friends and to guide without help them safely on the festival grounds Requirement F1 The iPhone VoiceOver made all text information fully accessible via Text to Speech AII the required information was available in the application Requirement F2 It was possible to see the neccessary information on a visual map Without vision it was possible to select a point and be guided to it Some information was missing to make the non visual overview information complete like for example which stages are closest in distance to the participant s position Requirement F3 and F4 All participants with visual impairment could select items for navigation and have an overview of the possible items by going through a list of points 79 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications stages friends or other points of interest Those could not be organised spatially as in the visual map but all points could be fo
30. for Android using the instructions on http www haptimap org downloads haptimaptoolkit gettingstarted html To ensure the toolkit is installed compile and install the Tactile Compass example on your device Task 2 Working through the tutorial On the following page is a tutorial that uses the toolkit to create a simple navigation application Follow the tutorial to build the application install and run it on your device Task 3 Developing a novel app Look through the developer docs for the toolkit Sketch on paper a simple navigation or location based application that utilises the toolkit Then build the application and install it on the device We will talk about the design of your application in the discussion at the end 137 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix K Parameters and Notation for NavEscort Intermediate Points REFERENCE POINTS FOR OPEN ENVIRONMENTS TEST Lamppost 1 Lamppost 2 Lamppost 3 Lamppost 4 Horse Baker house Tourist Information Latitude N 40 41522 Latitude N 40 41524 Longitude O 3 70767 Longitude O 3 70761 Altitude 708 m Latitude N 40 41557 Altitude 703 m Latitude N 40 41559 Longitude O 3 70770 Longitude O 3 70767 Altitude 717m Latitude N 40 41559 Longitude O 3 70705 Altitude 711m Latitude N 40 41557 Longitude O 3 70705 Altitud
31. indicated they would be likely or very likely to use the Toolkit again and that it was well ahead of any other systems currently available This evaluation was based on a relatively short use of the HaptiMap Toolkit particularly be the three freelance developers It would be interesting and useful to conduct a longer evaluation of the use of the Toolkit 103 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 4 Analysis of the results Since this is a report on the final summative evaluation studies of the various demonstrators and the use of the toolkit by developers we will focus here on how well the project fulfilled the claims made by the various applications discussed earlier in the design process So what follows as an application by application summary of the success or failure to fulfil the requirements claims made earlier Following this is a brief discussion of the suggested additions to the system that were brought up by the reports of the projects tested Across the demonstrators evaluated here and with others not reported in this document there was a wide agreement that the requirements generated in the beginning of the Haptimap project and intimated in the original Haptimap proposal were satisfied The evaluations reported 75 requirements and 81 were fulfilled completely if we conceder partial fulfilment typically completely fulfilled in other applications the success number goes to 9296 A re
32. like to raise about the HaptiMap Toolkit Tutorial 2 5 Did you use other documentation about the Toolkit Yes No 2 6 If so what was it 2 7 Was it useful 2 8 Was it clear 2 9 Any other issues you would like to raise about the HaptiMap Toolkit documentation 3 WHAT DID YOU USE THE TOOLKIT FOR 3 1 What did you use the toolkit for How successful were you in using it did you achieve your goal 4 EASE OF USE OF THE HAPTIMAP TOOLKIT 4 What it easy to use the HaptiMap Toolkit 4 2 On a scale from 1 5 where 1 is very difficult and 5 is very easy how would you score the toolkit 4 3 Any particular issues you would to raise about ease or difficulty of using the Toolkit 5 WAS IT WORTH THE EFFORT 5 1 Considering the effort you needed to invest in downloading the Toolkit understanding it and so on was the return worth that effort 6 FUTURE USE 6 1 Would you use the toolkit again in the future 6 2 On a scale from 1 5 where 1 is not at all likely and 5 is very likely how likely is it that you would use the toolkit 6 3 What might you use it for 143 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 7 USE OF SOUND AND VIBRATION IN MOBILE APPS 7 Have you ever wanted to add sound or vibration to an app in the past 7 2 If yes how did you do that 7 3 Do you think sound and vibration will be important aspects of apps in the future 8 ANYTHING SIMILAR TO HAPTIMAP TOOLKIT
33. listed parts above which was the easiest one Which parts of lt FBT gt were difficult to use Please list them Of the listed parts above which was the one most difficult one _ Is there anything you would add to the lt FBT gt Is there anything you would remove from the lt FBT gt _ Did you run in to problems using the lt FBT gt Yes No How did you recover from these Compare using the lt FBT gt with how you accomplished similar activities tasks before Would you use this lt FBT gt again or in other applications How much did you need to concentrate on using the lt FBT gt functionality Likert 1 7 How do you think this affected your experience of the lt application gt Back to top till you ve exhausted the list of lt FBT gt moe mo aos mi a For the whole application 2 Observation and debriefing with user Did the user successfully use the application to accomplish the global task Yes No free text 3 What was the one most difficult part of using the application 4 What was the one easiest part of using the application 5 Is there anything you would add to the application 6 Is there anything you would remove from the application 7 Did you run in to any other problems using the application except the ones you have told about previously Yes No How did you recover from these 8 For the whole experience choose X words from this list 9 Is there anything you want to tell me about your experience or
34. little or no Android or other smart phone experience will be able to navigate the interface after an introduction G4 Users will be able to detect and understand the haptic guide concept when introduced to it G5 Users will be able to follow a trail and answer the questions for the different points on the trail in a real context G6 The app will be usable both for hiking and for city tours G7 A wide age range of users will be able to use the trail part of the app approximately 7 75 years G8 Users will be able to use the explore mode with an introduction to it We extracted a sub set of the functional and the non functional requirements from the tables above that fit to the test tasks and the current functionalities The table has been completed with information about which of the test situations that covers the claim and a column showing if the requirement has been met according to the results see Section 5 110 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table A 4 Extracted requirements table subset of Table 1 and table 2 Nr Requirement F5a Allow explorative navigation select points F5b Allow guided navigation trails F7b Possible to select objects by pointing to the objects in real life NF1 Fun NF2b Possible to use together with others F13 new Basic functionality available without vision Diverse conditions of use When creating the tasks you should take care to incl
35. might be happening in the pattern of use of sound and vibration during the festival through the environmental context It could well be that as the music became louder and more important to the users the use of vibration was also found to be more useful The analysis also looked at the use of the Facebook list function which allowed users to search at the festival for their Friends from Facebook Unlike the use of the guiding functions this showed a somewhat different pattern of use during the festival with more use early on in the festival and less synchronization with the music events This suggests that people were using this function to add friends before the festival main event started which makes good sense This analysis has shown that much useful information can be extracted from logging data from an app in a realistic context of use although a better coordination with the logging team from the start could have yielded more relevant and useful logging regarding user relevant events It was frustrating that more information could not be extracted and we have learnt useful lessons about what is needed in a logging protocol to ensure that the right information is extractable for analysis The first lesson would be to collaborate better and plan better the logging with the team developing the application and to make sure the logging is made on the relevant parts and events of the application Here we mostly had information and events pertain
36. might be interested in trying out the HaptiMap Toolkit for future use 3 participants were recruited this way e Developers within the HaptiMap project 2 participants were recruited this way The first source of developers was clearly the most suitable as these developers had really tried to use the HaptiMap Toolkit for a real project to which they had some considerable commitment they presumably entered the competition as they wanted to win both for the money and the publicity They are also developers who are completely independent of the project This shows an extra benefit of running such a competition to projects EU or otherwise it creates a pool of developers who have used the tool developed in earnest The second source of developers was also useful but harder to recruit One needs to know developers who might be interested in trying out a particular toolkit and one cannot be sure how much effort they will put into learning and using a tool For this evaluation we asked the developers to complete a task with the toolkit and demonstrate it to us to try to ensure that they had really used the toolkit We used a range of personal contacts in the developer community to find appropriate people to ask to participate in the evaluation 98 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The final source of developers are clearly people who have used the HaptiMap toolkit but because they have been in
37. mobile phones also need to be checked for precision In the final rural tests with 5 participants aged 16 to 23 all participants were able to follow the trail 3 of them followed the trail from start to finish without any problems of using the app as such but 2 participants needed to restart at one point because they were unable to reach a point The question On a scale from 1 7 how would you rate your wish to use the app again 1 indicating low wish and 7 a high wish was answered with 5 7 by all of the 5 participants 3 3 5 3 Scanning results city All users except one used the select on release mode in the HUI all interaction screens except the scanning the single user used select on double tap It was observed that the combination of using double tap in the HUI and scanning with the finger resting on the screen with select on release for just that action was confusing to that one user Also some problems were seen for other users with the scanning One user complained that he got too warm and wanted to lift the finger once in a while resulting in a point being selected Another user wanted to actively select the scanning range by lifting the finger when the speech synthesis read the distance also resulting in a selection event Participants quite quickly learned the use of the Android Back button and recovered from those errors easily The participants using a wheel chair had another problem Needing two hands to
38. on the keyboard does not mean that the place is added Sending a place requires to enter a message which is not indicated 2 Some of the elements should have a more meaningful text label announced by Voice Over Sidebar status should be indicated and what it exactly does Yes no button when adding a place Pin red is a very visual analogy and not clear for all users with a visual impairment Friends places maybe announce that a double tab will open a context menu Some buttons were labelled in English 3 Settings as heading is confusing with Profile which should be labelled as Settings 4 A button to turn off vibration sound However the participants made also suggestions on how the app could be improved 1 Use Ariadne as a map 2 Incorporate SiriTM 3 Incorporate the face recognition feature from Apple into Joined 4 Incorporate the Hapto Menu developed by OFFIS 5 Include a search function table index for the list of friends places 6 Use Joined to track public busses 7 Use Joined to find friends indoor 8 Change the vibration sound pattern when one comes closer to the selected friend place In the second group all participants had good vision three of the five needed glasses but reached 10096 with them They came all from different professional and educational backgrounds see figure 5 Table 3 1 5 Participant basic demographics group 2 older users Age Occupation Education 58 IC
39. outlines the conditions in which the specifications of the NavEscort application were tested and the subsequent results The aim of this evaluation was to verify whether the design of the application complied with the specifications on which it was based When we thought of designing the tester we set as objective to emulate the conditions that the design should comply with in order for users with visual difficulties to be able to make use of global positioning systems or GPS The following conditions were identified as the main ones Cartography for vehicles Representation of inaccessible geographical information maps Unintuitive and inaccessible interface A non multi mode approach Inadequate or insufficient basic information given by the system We therefore thought of trying to solve these problems through e Designing our own cartography by means of collecting geographical positioning data concerning the necessary elements for the user to carry out a safe and effective route This data is transferred onto the map and it is executed by the system as a file which has the required characteristics so that the user can make use of the system to follow a particular route e Producing specific information in a sequential format so that when followed by the user allows him or her to execute a particular route e Creating an interface through sound visual and tactile features and with simple menus If these solutions are implement
40. pedestrian zone All participants had to accomplish the same tasks which consisted of 1 Installing and registering 2 Invite a friend 3 Add a place and name it 4 Find a friend with sound and vibration 5 Send a chat message to a friend 6 Send a place to a friend 7 Receive a place and find it with sound and vibration Joined aims at people who want to find friends or locations without looking at the screen of the mobile device while walking This is achieved by the usage of multimodal components like the Geiger counter and vibration in order to indicate the direction If the application is not open no GPS coordinates are sent to the Joined server and when a user would like to find one of his her friends he she has to wake him her up which has to be confirmed by accepting the wake up and thus sharing the current location Only the last coordinate is stored on the Joined server thus it is not possible to create movement profiles 1 3 2 NavEscort Tecnalia The NavEscort has been developed by TECNALIA and has been tested by 4 users in Madrid The testing has been led by TECNALIA and ONCE The NavEscort by TECNALIA amp ONCE is coupled to the TrainerTools Its aim is to actually guide a blind person It takes a route and a user profile both created with the help of the TrainerTools and provides a highly accessible routing experience for blind people 1 3 3 Time Machine The Time Machine is developed by ULUND and
41. project website www haptimap org and the toolkit wiki pages A forum for developer questions has also been set up 15 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The toolkit evaluation is an evaluation of the HaptiMap Toolkit It uses no modules per se but is designed to study how other third party developers external to the project consortium use the toolkit to employ modules within their own development projects and applications This includes ease of setting up and deploying the toolkit The toolkit use evaluation was designed to assess the ability of developers to use and employ the toolkit itself in the creation of novel applications for navigation and location based scenarios 1 3 Definition of user profiles and groups The HaptiMap toolkit aims at making it possible for developers of map type services to add accessible touch gesture and sound interface elements to their applications We think that the fact that we will provide developers with tools that help them add this type of novel and attractive interface elements to their designs will make them more motivated to use these and thus as a wanted side effect improve the accessibility from the HaptiMap DoW p 16 During the demonstrator tests different user groups have been involved see Table 1 3 For the different demonstrators different end user groups are the main targets Since the toolkit should support both design for all a
42. sights nature etc 111 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix B HaptiMap common background questionnaire Participant Identifier Experiment Experimenter Date Please complete the questions below as fully and accurately as possible A Basic Information 1 Gender __ Male Female 2 Age 3 Occupation current past 4 Highest completed education B Hearing Vision and Motor ability 1 Please rate your visual ability with the best possible correction for your glasses __ Full vision __ Minor vision problems __ Moderate vision problems __ Severe vision problems with some residual vision __ Blind 2 Are you colour blind __ Yes __ No 3 If you have vision problems please provide some details 4 Please rate your hearing __ Full hearing __ Minor hearing problems __ Moderate hearing problems __ Severe hearing problems with some residual hearing __ Deaf 5 If you have hearing problems please provide some details 6 Do you have practical musical experience playing instruments singing in a choir etc _ Yes No 7 Please rate your motor abilities __ Fully mobile __ Minor motor problems 112 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications __ Moderate motor problems __ Severe motor problems 8 If you have motor problems please provide so
43. testing 53 Table 3 3 1 4 Extracted requirements table subset of Table 1 and table 2 53 Table 3 3 3 1 Demographics of city Time Machine test ccccceeeeeeeeeeeeeeeeeeeeeees 54 Table 3 3 3 2 1 Participants in pre final 68ls 5 iiio ir Ri i RERREHE ERR DERE 54 Table 3 3 3 2 2 Participants in rural Time Machine testing 55 Table 3 4 1 1 The Terrain Navigator non functional non hardware related zie U Lid 6911 p cc e Tcr 69 Table 3 4 1 2 The Terrain Navigator s functional requirements 70 Table 3 4 1 3 Evaluation topics for the Terrain Navigator sse 71 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 4 4 Participants ratings of their subjective task load 73 Table 3 5 1 2 1 Demographics for the qualitative Juicy Beats evaluation 76 Table 3 5 1 2 2 Sense of direction of the participants 6 means gooQ 77 Table 3 5 1 3 1 1 NASA TLX ratings for the Juicy Beats qualitative evaluation 77 Table 3 5 1 3 1 2 Scores from the System Usability Scale for the Juicy Beats ccm 71 Table 3 5 1 3 3 1 Event Guide JUICY BEATS functional Requirements 79 Table 3 5 1 3 3 2 Event Guide JUICY BEATS non Functional Requirements 80 Table 3 6 1 4 Use of HaptiMap Toolkit components
44. than asking a single person to think aloud The accompanying person was in most cases a test leader who could also help with technical problems but 2 participants had accompanying persons with them The first step was to introduce the evaluation get verbal consent and begin the recording The recording by audio recorder and SenseCam pictures was then started The participant carried an Android phone with the Time Machine a SenseCam camera and an audio recorder An introduction was then given to the participants about the Time Machine and also the use of the back button in Android This introduction was adapted to the user s previous knowledge of smart phones for example the select on release was more carefully explained to those who had previously used an iPhone which has select on double tap These instructions were recorded in order to collect information about the improvement of the help section of the application and also to document the learning process Then the user carried out two learning tasks one to scan and choose a specified point and let themselves be guided to it by the HaptiMap toolkit vibration module and another to scan and choose a historical sight to examine from afar 55 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Half of the participants were given a phone configured with turn by turn routing The routing employed was the adapted kind of routing that routes over open areas a
45. the user evaluation testing the fit of the application to the specification and not using precious user evaluation time in capturing easy to rectify interface problems In this summative evaluation the heuristic results are not reported throughout all demonstrators However they have been a part of the development process We have used the Universal Design Principles with some notes of adaption to the mobile situation see Appendix D 2 1 3 Optional Laboratory based usability accessibility testing These tests the in laboratory task completion System Usability Scale SUS Brooke 1996 can provide additional filtering to capture design flaws before field testing The think aloud protocol can expose menu and widget problems The SUS provides insight into the work practice and dynamic problems that the system has The complete procedure for using the SUS can be found in Appendix E 19 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 2 1 4 In context evaluations This is the core of the demonstrator evaluations These in context evaluations provided final results for each of the applications by exposing design and usage problems in realistic tasks These tests have been videotaped only if taping did not interfere with the test A small pocket sized audio recorder attached to the participant has been used in some cases The observer has used the in pairs walking approach or observed from a short distance
46. trainer at all times during the tests in order to guarantee their safety Furthermore an observer followed the users from a short distance with the purpose of registering any unusual circumstance and informing about it The observer would do so by sending the exact coordinates through wireless technology to them playing in this way the role of a GPS There was also an expert in user studies who wrote down all the relevant data One of the users had a 31 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications significant degree of residual vision and to avoid that this circumstance had an impact on the user s orientation ability he was asked to wear an eye mask and use a cane The complete route was recorded on video 3 2 5 3 Participant overview Table 3 2 5 3 Session 1 Participants FAMILIARISED WITH GENDER VISUAL CONDITION MOBILE TECHNOLOGY Residual M F Blind YES NO 4 36 57 57 40 VISUAL 3 1 2 2 3 1 3 2 5 4 User characteristics 3 2 5 4 1 User El He has only Residual Vision He carried out the test with the aid of a cane which he was not familiar with while wearing an eye mask He found the voice aids headphones and microphone too awkward and uncomfortable In the context of following a particular route through vibrations the user suggested the utilization of wristbands that could indicate the direction to take while walking The user preferred that distances were gi
47. used within a particular use The mean number of calls in a sequence is 16 53 standard deviation 30 96 with 70 of sequences being 15 calls or less The logging data of the use sequences do not include unique identification of the different users that information was recovered from the registration process so the analyses below are presented in terms of uses rather than users 81 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 5 2 2 Distribution of use sequences across the time of the festival nimm S NE GL ye I SES Sos S S SS S Ss XP AS SS AES ATS AS ax V HP Ge d dg Time 30 31 07 2011 CEST Figure 3 5 2 2 1 Total use sequences of Juicy Beats app during the festival To provide a baseline against which to compare the use of particular interesting functions the distribution of all use sequences of the app during the festival was plotted This is shown in Figure 3 5 2 2 1 above The festival started at 12 00 on the 30 July This can be compared with the programme of festival events reproduced in Figure 3 5 2 2 2 From Figure 3 5 2 2 1 we can see that even in the first hour of the festival there were over 100 uses of the app Use then rose steadily to approximately 600 uses per hour by 18 00 as more acts got going Between 18 00 and 22 00 the main part of the festival took place and use of the app declined probably as people were more concerned with the
48. via points hand made route points between information points to guide the user when there is no natural trail to follow or at forks in the road Make sure the organisation or company you are working with understand the media Provide them with a sketch trail that shows all forms of interaction 3 4 The Terrain Navigator 3 4 1 Introduction The Terrain Navigator developed by the FGI aims to demonstrate that good quality raster formatted background maps in combination with vector data and audio can be used to improve accessibility in the hiking context The Terrain Navigator operates on one of the mainstream platforms the Apple iPhone and the iPad Figure 3 4 1 Figure 3 4 1 The Terrain Navigator operates on the Apple iPhone The evaluation of the demonstrator is based on the scenario 1b Hiking in a national park described in the DoW The main user group of the Terrain Navigator is ageing users who have age related deficiencies more specifically problems with vision dexterity and mobility The specific user requirements for the Terrain Navigator from D5 1 p 66 67 are described in Tables 3 4 1 1 and 3 4 1 2 The tables also show the requirements that have been implemented in the final version of the Terrain Navigator Table 3 4 1 1 The Terrain Navigator non functional non hardware related requirements User Requirement 69 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications NFCO001
49. visually impaired during the music festival It gives to all interesting information about the festival and important location features The guiding and the friend finder where considered the most important features of the Juicy Beats application The Friend Finder was seen as a possible application on its own which has been later realized in Joined The guiding was thought to be equilly important but hindered by GPS problems Some workarounds were suggested For users with visual impairment a few features are missing to make the application fully accessible such as adding detailed routing along paths or provide a function to get the overview location information on a visual map Overall the participants rated the application as usable 89 17 mean at SUS and requiring low demand 2 26 on a 1 7 scale at NASA RTLX They could carry out the expected tasks while enjoying the context of the music festival 3 5 2 Quantitative Analysis of logging data from the Juicy Beats app 3 5 2 1 Introduction During the Juicy Beats Music Festival held in Dortmund 30 J uly 2011 approximately 300 people signed up for the Juicy Beats app and gave permission for us to log their use of the app during the festival This has resulted in a corpus of 5035 uses of the app from 291 different users Each use consists of between and 805 calls to different web services Thus we will call each use a use sequence as we discuss the different web services that were
50. 14 Score 10 Item How do you find the possibility of adding to the route informative instructions or instructions on directionality 15 Useful 3 Not very useful 0 Useless 0 49 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Item The personalization of an instruction may be of value 16 For any 0 For users with 2 It is personal and can not be user similar profile applied to other users Item How far in advance and how many times the instruction should be repeated to the user when covering a route 17 Only once at X meters of distance being X a distance which 0 would be configured for each user profile example In X meters steps prepare your self to cross a regulated zebra crossing with sound A first time to prevent him her from the situation and then 2 another time in the moment of the execution In 20 meters steps you will find a regulated zebra crossing with sound and later Cross it Others It depends on the situation Item Overall impression of the whole application 18 e MTs think that the application is useful and easy to handle e It can be of help for both the users and the specialists in orientation and mobility A tool like the one presented could make the work of the MTs easier both indoors and outdoors e Good Note Item number 12 Changes in the profile characteristics listed i
51. 16 17 18 Table 3 3 3 2 2 Participants in rural Time Machine testing Smart phone ID M F _ Age Vision experience GPS use GPS used for what 12 M 7 Sighted Android Y Geocaching 13 M 9 Sighted Android Y Geocaching 14 F 63 Sighted Android X Car 15 F 50 Sighted iPhone Nokia Y Research Google Maps etc 16 F 59 Sighted Android Y On phone 17 F 41 Sighted Android N 18 F 26 Sighted Android X Research Google maps etc 19 1 F 18 Sighted Android Y In cities 20 2 M 16 Sighted Windows Y Walking 21 3 F 21 Sighted No smartphone N 22 4 F 24 Sighted Android N 23 5 M 17 Sighted Android Y Car 3 3 4 In Context evaluation procedure 3 3 4 1 City test explore mode The test in the city was to use the Time Machine app to scan around for points of interest choose a point and be guided to it There were 34 different points to choose from and there were 46 sound windows scattered among the POIs The phones used were SonyEricsson now Sony Xperia Neo and Arc These were chosen because of the physical buttons for back home and menu In the city participants were carrying out the evaluation by walking together with an accompanying person This was done for two reasons first for the safety of the user with visual impairment and second to aim for capturing conversation between the participant and the accompanying person about the usage of the application rather
52. 2 2 3 1 3 2 5 5 3 Achievement of functional and non functional requirements Table 3 2 5 5 3 Specifications related to the execution of a particular route in open environment N Requirement Tested pass fail Through the voice Interface the user is able to use the Interface properly to explore and 1 Navigate through the different functionalities Observation User E2 Although I think it is extremely difficult It s very little time to learn so many things Through the tactile screen Interface the user is able to utilize the Interface properly to 2 a 4 explore and navigate through the different functionalities FUNCTIONAL SPECIFICATIONS RELATED TO NAVIGATION 3 It is able to upload a route saved on the system 4 4 It is able to execute a previously uploaded route 4 3 IT is able to interact with functionalities related to the execution of a particular route 4 6 Itis able to execute a route with the help of a GPS 4 FUNCTIONAL SPECIFICATIONS RELATED TO THE EXECUTION OF A ROUTE 7 Information about the route 4 8 List of instructions for the route 4 9 Upcoming interesting points within the route 4 10 Route Data 4 11 Ask to identify its current position 3 1 12 Route download 4 3 2 6 Session 2 Due to the particularities and difficulties of this group when trying to communicate with the world around them or interact with devices that have been
53. 796 students did not attempt to use the Toolkit at all Students who did not end up using the Toolkit in their project work were asked to comment on why they had not used it Most of the comments were not related to the functionality or usability of the Toolkit e g did not have time functionality was not relevant to the project I was doing but the following comments suggest areas that might need follow up messy API did not get it to work with our existing code from three students lack of documentation API was poorly documented The remaining results will concentrate on the fourteen students who used the HaptiMap Toolkit Table 3 6 1 below shows the number and percentage of these students who used different components of the Toolkit It can be seen that the great majority of students used examples but modified them for their particular project This is undoubtedly a good first use of the Toolkit to understand the functionality of components and to efficiently but flexibly use modules Table 3 6 2 below shows the ratings on a 1 to 5 scale where 1 is completely disagree and 5 is completely agree made by the students on a number of aspects of the Toolkit as represented by statements about the Toolkit The Table also provides results of one sample t tests conducted to investigate whether the ratings were significantly above the mid point of the scale so were students more than randomly positive about t
54. 9 2 1 1 Participant demographics and technical use history eerte 19 2 1 2 Optional Heuristic walkthrough eese tentent ntt ttnnttnn tnnt tn tttntent tentes enn 19 2 1 3 Optional Laboratory based usability accessibility testing een 19 2 1 4 In context evaluatiOns n uertice boite einen er ier tx a trade ren ree aita daran 20 2 1 INASAZTEX task load test eate tentur egt er trt imd tei rei e aiia 20 2 1 6 Semi structured interview esent ortae tenti tta rtt dna rta ta bane aao orn efTe na Kra keR tu aa 21 2 2 Evaluation protocol for toolkit users erre ene enetetetntntntn tnnt tata KR tata KR sas RR 21 3 Field trials and analysis aia ua iubet vacas a EN Y u cuss CaL RAK REB RA ARR nd Dn E d RR 02 22 RA MI DRE 22 S ZINAVESCOMES ONCE 20 MP E 28 3 2 1 Introduction user trials ertt tette red entere ne enata enge ticdu gend 28 3 2 3 Evaluation Conditions siesccisecceescsacsicattectecedasceacansastavticeduesesuatasteastacedusvezindueranesteudtaderanduaran atractsnedanientasuaceateat 29 3 2 4 Outline of evaluation protocol for end users eerte tnnttnntnnttn nnns 29 3 2 5 SeS10f a intend skr enn ASA NIAN AR dado te epo t doe DN ende ut dB ARR M cnt Pea 29 SON di M 35 RM TI 40 3 2 8 Conclusions and recommendations eese tentent tentent tnnt tnnt tnn
55. C and from sunny to cloudy In the second part of the evaluation the participants studied the Simple maps on an iPad and answered questions regarding the different types of maps The maps were studied in similar conditions as would be while hiking i e in an information hut or outdoors 71 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications T 1 7 N F m E 0 25ysfalam es fi EN N cs pr 74 2 E N Figure 3 4 2 A map of the 2 km walk in the Nuuksio National Park 3 4 3 Participant Demographic Information Eight participants five female and three male completed test tasks with the Terrain Navigator The participants were between 55 and 70 years old Most of the participants had specialized education for their diverse professions All of the participants used glasses but did not have any major problems with their vision Problems with hearing or motor abilities were not critical issues either All of the participants had owned a mobile phone for more than two years For the second part of the evaluation further eight participants in addition to the previous eight elderly participants answered the questionnaire about Simple maps These four female and four male participants were between 24 to 39 years of age They all had university education One of the participants was a colour blind otherwise they did not have any major problem with their vision hearing or motor abilities
56. E WITTEBORG amp TOBI KATZE ZWEI DIEILESEN OHNE KONTRABASS FRAU SCHWEINEMETT ANDI SUBSTANZ LAURA REICHEL AUTOMATOPOET TORSTEN STRATER DEEJAY NACHTFALKE DJ SET LMBN SLAM LESEB HNE mesa KN C mammam sesso PHILIPP amp FABIAN DJ LARSE AFRESH amp PASCH SRE Tum STEE asec sv pumueum C mar umen G sue a pu eum 7 rr esse Tessan mw qeu CN ej pem FUNKTRONIX FUNKFATAL pum 00 qm eas Figure 3 5 2 2 2 Programme of events at Juicy Beats Festival 2011 3 5 2 3 Use of the guiding functions The Juicy Beats app provided functions for showing the direction of a particular friend or one of the stages within the festival the bearing and guiding functions The bearing and distance information is shown visually on the screen see Figure 3 5 2 3 1 and this is combined with either sound the Geiger function or vibration Unfortunately the logging data does not discriminate between seeking the direction of a friend or a stage These functions were used 554 times so 11 096 of all use sequences of the Juicy Beats app included some guiding functions Figure 3 4 2 2 shows the distribution of the use of the functions during the festival Figure 3 5 2 3 2 shows the use of the guiding functions in comparison with the overall use of the app and we can see that the pattern of use follows closely with the overall pattern of use Telekom de 12 37 4 87 ES Cancel Navigation Distance 4 8km Geiger Figure 3
57. Eclipse 3 6 1 5 Conclusions The fact that nearly 5096 of students responding to the evaluation questionnaire successfully used the HaptiMap Toolkit in their project work even though it was an optional aspect of the course in itself shows the usefulness of the Toolkit This is reinforced by the significantly positive ratings that the students gave on the usefulness of the modules and the functionality benefit of using the Toolkit Table 3 6 1 5 Ratings of the Toolkit with significance tests for positive ratings Mean rating Toolkit aspect OF T test result P St Deviation 2 78 t 0 81 EASE OF FINDING INFORMATION f 1 17 df 17 It was easy to find the information about the modules a USEFUL FUNCTIONALITY 1075 pm 3 i I found the functionality of the modules useful i E p 0 001 EASE OF SET UP ANDROID a jo s It was easy to set up the Android Specific part no NDK i 5 EASE OF SET UP dos jn is It was easy to set up the platform independent part NDK i E EASE OF UNDERSTANDING s pa 0 94 df 16 It was easy to understand the information about the modules n s FUNCTIONED AS EXPECTED n UM 0 63 df 15 The parts I used worked as expected p lt 0 001 FUNCTIONALITY BENEFIT GAINED 3 88 t 2 97 I think that my project application got a higher level of 1 20 df 15 functionality with the modules than I would have managed to do p 0 01 within the same time frame without it The problems tha
58. HaptiMap Toolkit Finally participants were asked whether they knew or had used any toolkits or other systems similar to the HaptiMap Toolkit and if so how they compared with HaptiMap Only two participants had used anything similar and in both cases they thought HaptiMap was far superior One participant mentioned two other systems Unity a game engine which allows you to do vibration but it is simple on off stuff and the Corona SDK http www coronalabs com products corona sdk has a little functionality for vibration at most short and long vibration P5 When asked to compare these systems with HaptiMap the participant commented HaptiMap is far ahead of what current engines can provide Another participant did not name any other systems but said he was familiar with others but that HaptiMap has more better functionality P1 102 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 6 3 4 Discussion and Conclusions This evaluation of the HaptiMap Toolkit with professional developers was challenging to organize and undertake but it has been successfully undertaken with seven professional developers The information yielded seems an honest assessment of their experience with the HaptiMap Toolkit This experience has generally be very positive although the evaluation highlighted a number of areas where some improvements could be made Most importantly all the developers
59. I can t see how you could make it easier for developers P4 another noted some of the functionality is inherently very complex nevertheless it was easy to use the underlying functionality P1 In addition one participant commented that the toolkit would be easier to understand if it had maybe a better designed homepage less text better overview with some real world and easy to follow use cases and examples P1 In fact the design of the Toolkit homepage and support pages had been addressed since this participant had used them 3 6 3 3 5 Cost benefit ratio 101 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Participants were asked whether considering the effort they had to invest in downloading and learning to use the Toolkit it was worth the effort given the return they received So in other words whether the costs in terms of their effort warranted the benefits the functionality available and the time saved in using the Toolkit over other ways of achieving the same ends in their apps Participants found this rather hard to estimate not surprisingly and answers tended to be brief All participants thought it probably was worth the effort For example one participant commented for the app that I did I can t really say yes or no but for another app that the participant is planning it would be make or break it is difficult to predict when the value return woul
60. Installing the toolkit e Task 2 Working through the tutorial e Task 3 Developing a novel app Following this is a debriefing session collecting programmer s thoughts about the toolkit and their use problems and successes 21 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 Field trials and analysis 3 1 Joined For the final evaluation of the Joined demonstrator within the HaptiMap project we conducted a user study in two parts One group consisted of people with visual impairments and the other of elderly people between 50 and 63 years of age The first group of people with visual impairments consisted of nine male and two female users recruited from the members of a German iPhone accessibility mailing list Ten participants were considered legally blind and one had a remaining vision of 2 They were all experienced iPhone users within a broad age range and diverse professional background see figure 1 Table 3 1 1 Participant basic demographics group 1 and users with visual impairment iPhone experience years Occupation Education 27 2 research assistant master 30 1 government employee middle highschool 51 2 programmer middle highschool 70 1 physio therapist middle highschool 54 2 piano tuner middle highschool 29 1 master student bachelor 34 1 psychologist master 41 1 unemployed highschool 42 1 telephone operator highschool 38 2 programmer m
61. L T ET 8 TASt Of CADICS e MM 10 Executive summary ioca iinetiRo quibm n EUG ort decanted RUE bise E FU OO Ue na EN K R S RR DE aS 12 13 TVG OUNCE OI RR 13 1 1 User re quirements cc cites a snask aa ete cei ean eeepc eee eS hres 13 1 2 List of Demo nstrat t 13 1 2 Jomed GeoMobile 5 c aie tei i r Ep d d s E c t kE FER ke 13 1 2 2 NavEscort T cnalt esce tit en rona db nd a P n Cr dr Ra De 14 1 2 3 Lund Time Machine eiii nia den ar a UR terna 14 1 2 4 Terrain Navigator zen ter tek epo ns oe uk ete pde Ret nce Le De itae laces 15 TDD Juicy Bats zie etie tree rene id gate ei ec YU e ect eee e NA AA rere dort 15 1 2 6 Developer evaluations 5 ite nnne terit tr c o E dede aiia 15 13 Definition of user profiles and groups eere netten tete nen nanna RR KR Rn Rn 16 1 3 1 Jomed GeoMobile 5c ien tio vct oa vi iR cd cr pa i e RR c da 17 1 3 2 NavBscort Tecnalta ia teet o bte nte n a due a n C b eR e Dad ONA 17 1 3 3 Time Machines menia anai RR 17 1 3 4 The Terrain NavigatiOn citet tse rer tne tp re srt i Erg tron cen ronde diarie uto 18 ISAAC E e A E e a e 18 1 3 6 Developer evaluations ttt tnnt aana taninai setinetantaae nner aaneantodeeaarinies 18 Des Test procedure ssania aaan aaau xa sean ara Ci aaeoa aaraa a aaa Du PEE a CL Da C ao 19 2 1 Description of th tests sede esenee inas anea ceana reseau Saso Sa dasa rup de aE Anaa Ea aa ea ania dadandan ei eaS 1
62. MS Mental demand Physical demand Temporal demand Effort Performance Frustration level TOTAL 43 PU User tests of the final demonstrator applications HaptiMap Deliverable D1 5 3 2 7 5 3 Haptimap Common Questionnaire A Basic Information Table 3 2 7 5 3 Session 3 participants parts A F Gender Age Occupation Education 4 O Female 80 68 2 pensioner Administrative Mae 48 49 1 administrative Pon doctorate in geography and 1 Braille specialist history Law degree B Hearing Vision and Motor ability Visual ability Hearing Music Abilities Dominant hand Severe vision problems with Blind Full Minor No Yes Fully Right some residual vision Hearing mobile 2 2 3 1 3 1 4 4 C Experience of Mobile Devices Mobile phone How long mobile phone Model and brand Yes gt 2 years Samsung Nokia N 85 4 4 Nokia 6120 Nokia 6120 D Diary navigation How would you rate your sense of direction please circle the numbers all 4 said 6 of 7 E Online Map Services on desktop computer In an average week how often do you access online map services all 4 said never F Mobile Map Services Have you ever used a GPS navigation device or application All 4 said no 3 2 7 5 4 Semi structured interview results Table 3 2 7 5 4 Semi structured interview results 1 Rate from firs
63. Scoring information follows The basic idea of thinking aloud is very simple You ask your users to perform a test task but you also ask them to talk to you while they work on it Ask them to tell you what they are thinking what they are trying to do questions that arise as they work things they read You can make a recording of their comments or you can just take notes You ll do this in such a way that you can tell what they were doing and where their comments fit into the sequence 121 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator System Usability Scale 1 I think that I would like to use this system frequently 2 I found the system unnecessarily complex 3 I thought the system was easy to use 4 I think that I would need the support of a technical person to be able to use this system 5 I found the various functions in this system were well integrated 6 I thought there was too much inconsistency in this system 7 I would imagine that most people would learn to use this system very quickly 8 I found the system very cumbersome to use 9 I felt very confident using the system 10 I needed to learn a lot of things before I could get going with this system Digital Equipment Corporation 1986 Scoring SUS applications Strongly Strongly disagree agree SUS yields a single number representing a composite measure of the overall usability of the system being studied Note that sco
64. The visual in the application Or more meaningful ways of saying this like the gestures the screen The content that the application gives access to might be a part too Question 7 You can prompt the answers with the following 1 Did you get help from the observer test leader 2 Did you use a recovery mechanism built in in the application 3 Did you rely on the outside world e g A physical map or people around you Analysis Finish initial analysis that day best would be before the next participant This includes taking field notes and recordings from 6 above and making a document that puts together all the sources and is ready for coding which should be done no later than the week of the individual test Fresh memory is important 132 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix H Protocol for HaptiMap toolkit evaluation Protocol for HaptiMap toolkit evaluation with professional developers and students 1 Ask participants to complete the short scale Attitudes to Disability this is a standardized scale 2 Ask participants to complete a short introductory web course on disabled people and technology to take 45 minutes maximum This will include e Watching the HandiSam videos about disability e Reading a short set of slides about people with disabilities e Watching the videos of blind person using Voiceover on a mobile e Completing a short practical empathy exercise to
65. Toolkit This is a return rate of 78 9 which is very satisfactory so the sample represents a robust cross section of the students taking the course Seventeen 56 7 of the students who responded were Information and Communications Engineering students and thirteen 43 3 were Computer Science and Engineering students Five 16 7 of the students have not yet completed all the compulsory programming courses in their degree so can be considered somewhat novice programmers Seven 23 3 of the students have completed all the compulsory programming courses but have not taken any additional courses so can be considered medium level programmers The remaining eighteen 60 0 students have completed all their compulsory programming courses and have taken additional courses so can be considered advanced programmers The students were asked whether they do programming as a hobby or as a job fifteen 50 replied that they do this included one of the novice programmers three of the medium level programmers and eleven of the advanced programmers as classified above These fifteen reported that they programmed on average 7 90 hours a week standard deviation 10 62 hours with a range of between 1 and 35 hours Students were also asked whether they have programmed on a mobile platform before Six 20 0 said they had with 4 reporting that they have programmed on the Android platform only 2 reporting that they have programmed on both the Android a
66. U User tests of the final demonstrator applications Item The fragments of an automatically generated route are valid only for a user or can be applied to multiple users 8 A user 0 Multiple users 3 Item When an MT explains a route to the user which kind of information he she provide to him her 9 Distance the route has a length of 500 meters 1 Approximate time to cover it it takes about five minutes to cover it 1 Shape of the route en forma de U L etc 3 Streets included in the route 2 Incidences in the route Number of crossings traffic directions obstacles which 2 do not meet the standard All the previous ones Others 1 Which one Special characteristics about the route Item Do you consider that the profiles included in the application enough are enough normal reduced mobility visually impaired and customizable 10 Yes 2 No 1 Item Would you add any other profile or characteristic to the profile 11 User height unexpected obstacles squares Item Changes in the profile characteristics listed None offered 12 in the application Item Are enough the different modes for providing turn instructions directions degrees hours 13 Yes 3 No 0 Item Do you think it is easy the way in which instructions can be added to any point of the route rate it from 1 to 4 where 1 is complex
67. W HaptiMap 7 m o iai SEVENTH FRAMEWORK PROGRAMME THEME 3 ICT Information and Communication Technologies FP7 ICT 224675 HaptiMap Haptic Audio and Visual Interfaces for Maps and Location Based Services Large scale integrating project Challenge 7 Independent living and inclusion D1 5 User tests of the final demonstrator applications Due date of deliverable 31 of July 2012 Actual submission date 3 of October 2012 Start date of project 1st September 2008 Duration M48 Lead contractor TECNALIA RBTK HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Project co funded by the European Commission within the Seventh Framework Programme Dissemination Level PU Public Project Number FP7 ICT 224675 Project Title HaptiMap Haptic Audio and Visual Interfaces for Maps and Location Based Services Document Type PU Document Number 1 Contractual date of delivery to the 31 of July 2012 Commission Actual date of delivery to the Commission 3 of October 2012 Title of the Document User tests of the final demonstrator applications Work package contributing 1 Nature of the Document R report Abstract This deliverable summarizes the final summative user tests of the application prototypes and evaluation of developers using the HaptiMap toolbox This deliverable describes the results of the user evalu
68. a summative record of how well the various facets of the HaptiMap project satisfied the claims that resulted from the design process requirement generation sections An example of the process of extracting the assertions claims for each demonstrator is in the instruction part of the experimental protocol in Appendix A Observation of the demonstrators being used by the end users as well as the topics generated for the semi structured interview topics that were specific to each demonstrator were based on these claims 2 1 1 Participant demographics and technical use history The process of user trials started with the explanation and obtaining assent to the HaptiMap informed consent documents Following this the users demographics and technology use history were gathered The purpose for doing this is twofold 1 to ensure that the participants fit the specification for the demonstrators and 2 to allow for across demonstrators comparison This step was based on the D1 1 B1 Basic Questionnaire Long and short versions are provided in Appendix B and Appendix C 2 1 2 Optional Heuristic walkthrough Continuously checking the UI for inconsistencies and usability problems is part of the design process Throughout the design process the UI was walked through based on usability principles The purpose of this is to capture design problems as early as possible in the implementation process to facilitate corrections This initial pass supports
69. ability in the toolkit often having to look at the original source files rather than the api documentation One participant noted that he had been unable to find the api documentation for any of the modules This was found to be frustrating as all participants found the toolkit to be powerful and provided a good start with using multimodal navigation in mobile applications at little cost The participant who had attempted to unsuccessfully compile to toolkit noted Its really cool Pd have no idea how to get started doing those things He further noted that he would use the toolkit again General Results Overall participants were able to employ the toolkit to build applications quickly In the two hours of actual development time participants were able to get through the basic learning of the toolkit and onto at least thinking about using it in real applications Participants were positive on the design agreeing strongly that the toolkit was useful functional and straightforward When asked about their three most positive points on the toolkit the plug and play nature where elements could be selectively chosen and integrated within applications was popular The quality of the HCI modules meant that they were seen as being beneficial to incorporate with participants noting that it would have taken several hours for them to individually build the HCI modules from scratch Therefore our results indicate that the toolkit is both useful for de
70. actory 7 very satisfactory 1 3 13 Would you use this system again NO NEVER YES SURE 4 14 Would you recommend this application to your NO YES friends 1 3 15 Do you want to say anything about your experience or about the application It feels like a prototype and needs to be completed There should be a way to lock the screen in order to use the application without activating other options It should be very easy 3 2 8 Conclusions and recommendations This section collects the main conclusions and recommendations that can be extracted from the experiences carried out The section has been divided into two parts corresponding to the type of users to which the recommendations are targeted The first part covers the conclusions that can be applied in general to all the users that have taken part in the tests whereas the second one only affects to those ones belonging to the group of deafblind people 3 2 8 1 Conclusions of general applicability 1 Regarding the three modalities that were tested in open spaces vibration patterns stereo sound verbal instructions in clock times it can be said that 45 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications a The modality that was worst assimilated by the users was the one based on vibration patters Vibration modality required more mental effort from the users that continuously stopped doing their task in order to interpret the meaning of
71. and stereo sound Initially she was given standard headphones but her sound perception was not very good Later on bone conduction headphones were used with a small amplifier In this occasion she did perceive the sound and she expressed her willingness to try out the sound modality during the tests on the street 3 2 6 2 Test characteristics and observations by each user User SD1 is a 54 year old person who is deaf blind He uses a white cane for mobility assistance He began route 1 with correct orientation almost on a straight line but he deviated from the original trajectory once he reached more or less the half way mark going around the equestrian statue situated in the middle of the square and returning to the original route later on When he came across a grating on the ground he introduced the cane slightly into it and he gets very agitated He thought there was a big slope and this makes him extremely nervous From that moment on he had great difficulties to follow the instructions given through the equipment During the route carried out in the second exercise he was doing well with orientation but when he had executed approximately half of the route he deviated and the test had to be stopped when the evaluators identified that he had no chance of getting back into the right direction On approaching him the evaluators realized that the application was off so there were no vibrations misleading the user to understand that
72. ation program in the city of Lund the HaptiMap partner LUND In the rural setting following a trail was more appropriate This test was a collaboration with a local cultural historical organization responsible for creating the content of the trail With the help of the Virtual Observer the application logged internal variables in the application The test was documented by SenseCam pictures hung on the participant to see 53 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications what he she was turned towards an audio recording to record the surrounding sounds and also the conversation between the participant and an accompanying person Visual observation during test was also carried out After the test the NASA RTLX workload was employed a demographic questionnaire filled in and a semi structured interview carried out Additionally a word list was used aiming to capture the more subjective feelings toward the app The test procedure can be seen in more detail below and in the test procedure section above as also the debriefing and interview session details which were also audio recorded 3 3 3 Participant demographic information 3 3 3 1 City test The test participants were recruited in different ways The participants with visual impairment who carried out the city evaluation were recruited through the local visual impairment organization SRF Skane A notice was sent out with their regular newslet
73. ations in the context of the claims and assertions of the various applications Keyword list User trials Summative evaluation demonstrators haptic interfaces qualitative evaluation quantitative evaluation mixed evaluation navigation demonstrators mobile context visual impairment elderly NASA TLX System Usability Scale HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Function Name Partner M Stefan Carmien TECNALIA Kirsten Rassmus Gr hn ULUND Kirsten Rassmus Gr hn ULUND Delphine Szymczak ULUND Helen Petrie ULUND Charlotte Magnusson ULUND David McGookin UGLAS Ginger Claassen SIEMENS hers Hanna Marika Flink FGI Mari Laakso FGI L Tiina Sarjakoski FGI Jos Antonio Mufioz Sevilla ONCE Stefan Carmien TECNALIA Arantxa Renteria Bilbao TECNALIA Alfonso Dominguez Garcia TECNALIA Internal Reviewer Christos Giachritsis BMT HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Document Status Information Iss rev Date Section Reason of change remarks affected 0 1 5 3 2012 First draft of the document 02 25 7 12 fe Inserted contributions 03 7 812 AM Corrected indexing and added text 0 4 8 812 All Editing bits 0 5 14 8 12 Figure and table numbering editing typos and fixing layout here and there Put Developer Bani Hol evaluation last in all secti
74. ble 3 1 2 Transportation Mode Usage 2 ecceeeeeeeeeeeeeeeeenneeeeeeeeeeeeeeeenaaees 23 Figure 3 1 2 Screen shot of bearing module and friend list in Joined 25 Figure 3 2 1Familiarization plhiasa ur rr Re E ob ert MX 30 Figure 3 2 5 1 2 Orientation phase SCENALNO ssrrrssrreseseerrrrrrrrrnnnnnrnrrrrrrrrrr rr nn nn nr rna 30 PIQUIG 8 252 123 Uran TOUS netee ap aroia eaea 31 Figure 3 2 6 2 1 One of the interpreters communicating with one of the users 37 Figure 3 2 6 2 2 Routes accomplished by user SD1 mssssssssrrsssrrssseerrrrrrrrrnnnnnnrrrrn anna 37 Figure 3 2 6 2 3 One of the participants during the tests ssssesssssssse 38 Figure 3 2 6 2 4 Routes accomplished by user SD2 s ssssssssrrrrssrsssseerrrrrrrrrnnnnnrrrrrr nn 38 Figure 3 2 7 1 1 Familiarization PNASC sssesssssssererrrrrsrresnennnrnrrrrrrrrnnnrrrrrrr rr rn n nr rr rr nn nn 40 Figure 3 2 7 1 2 Orientation plasQ i uoi teret E ERE P arian 40 Figure 3 2 10 Routes accomplished by user E3 sisstsssssssssesesrrrrssrsrsnnnnrnrrrrrrrnnnnnrrr rr rn nn 41 Figure 3 2 11 Routes accomplished by user E4 cccccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 42 Figure 3 2 12 Routes accomplished by user E6 s sstussssssssesesrrrrssrrrsnnnnrrrrrrrrrnnnnrrr rr nanna 42 Figure 3 2 13 Routes accomplished by user E7 ccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 43 Table 3 2 7 5 3 Session 3 participants parts
75. cations in place They also considered interesting to make the application accessible for visually impaired people Although the main target of the application are the MTs who cannot be visually impaired there are some functionalities in the application which can be also very useful for the MT users 50 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The fact that the routes designed for a user can be extrapolated to other users with similar profile has been assessed very well and offers a pretty good idea of the logic and validity of the specifications under which the application has been designed Another aspect that was mentioned is that although the application is very intuitive and easy to use the edition of maps using JOSM is still too complex JOSM requires a great level of know how to handle with it Nowadays MTs often use GoogleMaps to configure and design routes and they believe that it would be interesting to make the process of importing existing files with recorded routes easier Finally they considered that the current level of development of the application is enough in order to use it as an aid in their daily work 3 3 Time Machine evaluations 3 3 1 Claims list The Time Machine TM demonstrator has developed from a basic proof of concept application focused on testing non visual guiding in a local historical context into a location flexible fully non visually accessible tourist g
76. cidos por el investigador o por vd mism Cu l es su Performance R grado de sastisfaccion con su nivel de ejecuci n 6 Nivel De Durante la tarea en qu medida se ha sentido insegur Bajo Alto desalentad Q irritad 9 tens 9 o preocupad o por el contrario Frustraci n Fr se ha sentido segur content 2 relajad y satisfech Estos pesos pueden tomar valores entre O para la dimensi n que no ha sido elegida en ninguna ocasi n y por tanto no se considera relevante y 5 para la dimensi n que siempre ha sido elegida y por tanto se considera que es la fuente de carga m s importante El mismo conjunto de pesos puede utilizarse para variaciones de una misma tarea o para un grupo de subtareas Adem s los pesos dan informaci n diagn stica acerca de la naturaleza de la carga de trabajo impuesta por la tarea ya que proporcionan datos acerca dos fuentes de variabilidad interpersonal a las diferencias interpersonales en la definici n de carga de trabajo en cada tarea considerada b las diferencias en las fuentes de carga de trabajo entre distintas tareas El segundo requisito es adjudicar un valor para cada factor que representa la magnitud de cada factor en una tarea determinada En esta fase de puntuaci n las personas valoran la tarea o subtarea que acaban de realizar en cada una de las dimensiones marcando un punto en la escala que se les presenta Cada factor se presenta en una l nea dividida
77. code repository of the toolkit and that downloading an open source framework only as a ZIP file was not that comfortable P1 In addition one participant commented you never get the feeling of it being a real product like from a company P4 100 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications When asked to explain what he meant he elaborated if you want a commercial product you tend to spend a bit more time on the presentation of the download it probably affected my expectation of the library a bit P4 Participants were also asked about whether they minded having to register with the project to be able to download the toolkit Only two participants raised any objections to this One commented that it potentially scares many developers that are used to other open minded open source projects P1 The other commented that the registration discourages one a bit from downloading P2 3 6 3 3 3 Documentation Participants were asked whether they had used different parts of the documentation available to support the Toolkit how useful they found them and how easy they were to use Four of the seven participants had used the Toolkit tutorials All four found them useful One participant was particularly impressed by the tutorials and commented more time than I expected had been put into the building tutorial which is nice to see P7 Another commented the des
78. continued However the correct direction was the other side of the corner and probably he had found it giving him more sharpened instructions 2 A very important factor to be taken into account is that the application should be preserved as to avoid its deactivation the problem here is that the person carried the device inside his her pocket and when moving he she pushed some of the screen touchable HW buttons involuntarily causing the application to stop As the application was switched off it did not transmit any vibration pattern and the person thought he she was going in the correct way although he she wasn t 3 2 8 2 Conclusions and recommendations when working with deafblind people Regarding the group of deafblind people these are the main conclusions that can be extracted from the experience carried out 1 The first conclusion is that when carrying out tests with deafblind users special attention must be paid to the test protocol It is necessary to have the aid of specialized staff in deaf blindness so that they can give advice and evaluate the test conditions to avoid any difficulties 2 Deaf blind people have a much higher degree of apprehension than people with any other disability which occasionally makes them more suspicious and inconsistent when carrying out tasks The work patterns are slow and with this type of users require the tasks to be as short and specific as possible 3 Before starting with the tests it is advi
79. cription of how to get the material into Eclipse was good explained how to get it to compile P5 However one participant who looked at the tutorials not included in the 4 above felt that the tutorials were perhaps too simple it is just a hello world example but it does explain the ideas behind navigation P7 In commenting on other documentation one participant liked the samples provided commenting that he learnt mainly from these the names of the methods etc are pretty self explanatory there was very little comment but this wasn t much of an issue because the method names were clear scanner and vision example were straight away useful to get me up and running P5 One participant did not like having to generate the documentation himself and comments I had to create the api doc ument with doxygen myself and the an online api documentation would be useful P3 One participant also missed typical help and search functionality of an online help system However he was glad that the samples were available that makes it easier P4 3 6 3 3 4 Ease of use of the Toolkit Participants were asked to rate how easy the toolkit was to use on a scale from 1 very difficult to 5 very easy The average rating was 4 14 so easy with a standard deviation 1 15 This is significantly higher than the neutral midpoint of the rating scale of 3 t 2 49 df 6 p 0 05 One participant noted that
80. ct All the participants were male with ages ranging from 24 to 33 years mean age 29 14 years standard deviation 3 53 In terms of their experience with developing mobile apps they had been developing such apps for between 1 and 6 years mean 3 85 years standard deviation 1 67 and had developed between 7 and 35 apps mean 17 00 standard deviation 12 58 3 6 3 2 2 Materials 99 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Participants were sent a short briefing paper about the study This differed depending on which sample of participants they were in e Participants in the competition group were asked would they be prepared to be interviewed as part of the evaluation of the HaptiMap Toolkit e Participants in the freelance group were sent information about the HaptiMap project and directed to the Toolkit part of the website http www haptimap org toolkit download html They were asked to try out the toolkit either with an appropriate app they were working on or on a set task adding sound and vibration to a game of tic tac toe see http developer android com resources samples TicTacToeMain index html e Participants in the HaptiMap project were asked whether they would complete a questionnaire about their use of the Toolkit in the project The questions used in the evaluation both interview and questionnaire are provided in Annex M Three participants the two HaptiMap developers and o
81. ction headphones to allow continued use of hearing for navigation e Sonar sound correct orientation Less frequency more distance more frequency less distance e Continuous sound turn to the side where the sound is coming from The greater the intensity of the sound is the bigger the turn must be If the sound is coming through both headphones one must do a full turn 3 Through orally transmitted time instructions providing information about turns and distance The turn is announced through the o clock times of a clock Similarly bone conduction headphones with amplification were used 3 2 2 1 3 Urban route a Figure 3 2 2 1 3 Urban route Urban route within a central area in the Madrid of the Austrias It began at house number 65 of Mayor Street and ends at the Tirso de Molina metro station Although the route is almost straight there are intersections and open squares which are difficult to face by the user A false GPS signal or GPS spoofing was used in this case Information about turns was provided to the user in clock format at 11 00 at 2 00 etc Information about nearby steps pedestrian crosses land marks places of interest such as portals or monuments was also provided Bone induction headphones with amplification were used in this case with the aim of minimising the interaction with annoying noises coming from the surrounding environment 3 2 5 2 Test Conditions The users were accompanied by a mobility
82. d Toolkit Navigation Running e Toolkit Geiger Running e Toolkit Vibration Running So in this sequence the user starts the guiding function and then adds the vibration as well as the Geiger sound Again we cannot tell for how long the Vibration function is then activated as there was no logging of when the guiding functions were stopped The next most common sequence is also interesting This sequence was e Toolkit Navigation Started e Toolkit Navigation Running e Toolkit Geiger Running e Toolkit Vibration Running e Toolkit Geiger Running This sequence accounted for 27 4 9 of all the guiding sequences and 7 3 of guiding only sequences In this sequence the user tries the Vibration function but then turns it off again reverting to the sound and visual condition But this happens far less frequently 7 3 versus 18 596 than the user turning on the Vibration and keeping it on until the end of the sequence the previous sequence Following up on this observation we looked at the guiding only sequences that end with sound and visual versus those that ended in vibration plus sound and visual The results are presented in Table 1 below This shows that in just over half the sequences 53 596 of the guiding only sequences the user activated the guiding gets the sound as default and does not activate the vibration In a further 15 5 of sequences the user activated the guiding including activating the vibration but subsequently tu
83. d brand of your mobile telephone D Everyday navigation 26 How would you rate your sense of direction please circle the Very Poor 1 2 3 4 5 6 7 Very good F Mobile Map Services 0 1 Have you ever used a GPS navigation device or application __ Yes numbers No 0 2 If yes how often do you use it 0 3 If yes for what purpose car geocaching biking 118 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix D Checklist for Universal Design Criteria PRINCIPLE ONE Equitable Use The design is useful and marketable to people with diverse abilities Guidelines 1a Provide the same means of use for all users identical whenever possible equivalent when not 1b Avoid segregating or stigmatizing any users 1c Provisions for privacy security and safety should be equally available to all users 1d Make the design appealing to all users PRINCIPLE TWO Flexibility in Use The design accommodates a wide range of individual preferences and abilities Guidelines 2a Provide choice in methods of use 2b Accommodate right or left handed access and use 2c Facilitate the user s accuracy and precision 2d Provide adaptability to the user s pace PRINCIPLE THREE Simple and Intuitive Use Use of the design is easy to understand regardless of the user s experience knowledge language skills or current concentration level Guidelines 3a Eliminat
84. d come in P4 Another participant commented given the demand for apps with sound and vibration that is probably coming I would be willing to commit the effort to learning about the Toolkit more fully to design proper haptic designs and then to implement them HaptiMap can cut down on the latter part to allow more time for proper design P5 And finally another participant mentioned It will make the difference to users coming that is clear from what we are seeing in the current market place even simple vibrations are being used a lot by people on their phones P7 3 6 3 3 6 Future use of the HaptiMap Toolkit Participants were asked how likely it would be that they would use the HaptiMap Toolkit in the future on a scale where not at all likely to 5 very likely and what they might use it for The average rating was 4 42 so midway between likely and very likely with a standard deviation 0 53 This is significantly higher than the neutral midpoint of the rating scale of 3 t 7 07 df 6 p 0 001 When answering about what they would use it for some participants gave examples of particular apps whereas others gave general answers for example haptic feedback is the next big thing coming it will be the next thing that users will want P5 Several participants mentioned the potential of using the HaptiMap Toolkit to create guiding and bearing apps of different kinds 3 6 3 3 7 Similar tools to
85. d throughout except for the first test user and introduction was needed to touch screen usage and the HUI As described above the HUI use was rated as difficult and therefore it is questionable if the HUI is worth updating This functionality should be taken care of by a well written screen reader This goal was not sufficiently tested G3 Users with little or no Android or other smart phone experience will be able to navigate the interface after an introduction As described above several of the users with visual impairments would have wished for more introduction time but in fact all but one participant used the app on their own At some occasions they would ask for help or confirmation but in general they navigated the app on their own Therefore we claim to have reached this goal G4 Users will be able to detect and understand the haptic guide concept when introduced to it This was true in all cases As described above the guiding was rated as the easiest part of the application by the users with visual impairment The user who had trouble with wind claimed to have no problems when in lee Although users in the rural condition had problems in general with wayfinding and the accuracy of the signal all understood the haptic guiding Thus we have fulfilled this goal G5 Users will be able to follow a trail and answer the questions for the different points on the trail in a real context See G6 and F5b for first part The ques
86. e 710m Latitude N 40 41530 Altitude 700 m Latitude N 40 41535 Longitude O 3 70706 Longitude O 3 70708 Altitude 710 m Latitude N 40 41537 Altitude 702 m Latitude N 40 41546 Longitude O 3 70732 Longitude O 3 70726 Altitude 704 m Latitude N 40 41580 Altitude 700 m Latitude N 40 41581 Longitude O 3 70720 Longitude O 3 70730 Altitude 707 m Altitude 691 m 138 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix II ORIENTATION TEST WITHIN AN OPEN ENVIRONMENT i d Oe Fl s z mm rd Symbol Explanation Streetlight e Movement between two points Equestrian statue monument the arrow indicates the direction of the horse s head Ideal path between two points AD Trajectory carried out by the user m gt Directionality of the path between two points streetlight 139 HaptiMap Deliverable D1 5 PU APPENDIX L ML questionnaire User tests of the final demonstrator applications USEFULNESS OF THE APPLICATION Do you think that the application can be useful rate it from 1 to 4 where 1 is not useful 2 Does the application functionality meet your needs regarding route designing rate it from 1 to 4 where 1 is not useful 3 Which similarities differences do you find between the functionality of the application a
87. e of several different wayfinding strategies A user with visual impairment also commented that 60 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications she would need more time to really integrate the new technology also the guiding into her regular strategies for travelling by foot It was observed at the very end of the test that she more naturally felt the direction at the same time as she followed a paved path In the rural test participants followed the guiding from the phone in a very trusting way instead of using the heading as an indication of a general direction The hypothesis prior to the test was that users would make decisions based on the terrain and the physical trails in combination with the heading information and thus not choose paths through heavy underbrush or steep slopes This behaviour from the users was seen from the entire group despite that they had been previously cautioned about the need for making conscious path choices 3 3 5 6 HUI results city The HUI Haptic User Interface has been a way to try to overcome the poor screen reader solutions that exist for Android phones However since it only works within the application it is not the final solution A standardized one is to prefer However the existing screen reader solutions like for example voice over on iPhone are not main stream solutions The idea with a HUI was that it might help all users also sighted to use in t
88. e below We see that the participants with visual impairment rated overall their sense of direction to be lower than the mean while the sighted users rated it higher It seems that the unique question in the common questionnaire gives some difference in the ratings with a mean more that is one point higher than the SBSOD Table 3 5 1 2 2 Sense of direction of the participants 6 means good SBSOD rating Unique Question 3 5 1 3 Results 3 5 1 3 1 NASA TLX The NASA TLX results are shown in the table below The mean of the mental demand ratings was the high Overall the demand stays quite low with a mean of 2 26 for all scales indicating that the use of the application was not too demanding Only two participants JB1 and JB2 rated the pair comparisons and the results were incomplete The results are thus not shown in detail but they confirm that the mental demand was the most important compared to all other scales Table 3 5 1 3 1 1 NASA TLX ratings for the Juicy Beats qualitative evaluation wJB1 OJB2 E JB3 L1JBA i I mJB5 ula dard oth Gie o3JB7 Mental Physical Temporal Performance Effort Frustration Mean E mear System Usability Scale The scores given in the system usability scale are all above 80 on possible values from 0 to 100 This agrees with the reports from all participants that t
89. e dog does not know what to do and awaits instructions from the user 32 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Out of the three modes that were tested in open environments the user worked better with the verbal instructions in clock times Although it is easier to interpret the direction in which the turn must be made through the sound signals turning in the direction indicated by the sound it is easier to be accurate when knowing how much one must turn with the sound mode this information is given by the intensity of the sound and it could be difficult to appreciate the change in sound intensity within open environments The urban route was limited to the area which included pedestrian crosses near closed doors It did not manage to complete the route by following the instructions coming from the device the user moved too quickly to be able to update its position through the MT device Verbal instructions in clock times appear to not work very well in the city Instructions such as turn at 14 00 hours should be substituted by more straight turns such as at 15 00 hours which are suitable for urban routes where straight angles are predominant The user stated that he thinks that the device could be put on a belt with a clip in a way that it would be easy to make the gestures just with one hand The user indicated that he trusted on the device However this trust is to some degree def
90. e in place to keep the HaptiMap toolkit current and useable for some time in the future 106 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 4 2 Future Work The short list of un fulfilled requirements is a good target for further work Also the consistent problem of GPS in many of the exemplary demonstrators needs to be solved before such applications can achieve wide use 4 5 Conclusion The HaptiMap project produced both a versatile cross platform toolkit and tested novel ways of producing user guidance in varied conditions for users with an array of abilities Further the ease of making interfaces for route creation provides functionality which is often missing in navigation systems Finally several AAL projects have indicated that they will use the toolbox and map data stuctures as a basis for their projects 107 User tests of the final demonstrator HaptiMap Deliverable D1 5 PU applications References Kovanen J Oksanen J Sarjakoski L T and T Sarjakoski 2012 Simple Maps A Concept of Plain Cartography in Mobile Context for Elderly Users Proceedings of the GISRUK 2012 April 11 13 2012 Lancaster University UK The Principles of Universal Design http www ncsu edu project design projects udi center for universal design the principles of universal design 108 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix A Notes on Creating
91. e no information points but serve only to give a better guiding along a winding trail They can be placed at sharp bends in a path for example Google hybrid Y Tracks R fshalen till Ruver d 25 juli LJ 2 E Q Center 56 08630 13 31117 777 m 1 ET I Bilder 6201 2 DiytalGiobe GeoEye LantmaterietMcelfia Kafftiate 2012 Boo Figure 3 3 4 2 Schematic concept of trail points and trail following In the figure above a part of the trail is shown in a map and a user icon is seen pointing with a phone It vibrates in the direction of the next point in the trail The user passes the sound windows blue and is led to the information points red Yellow points are via points Pre final testing has been done iteratively and collected qualitative results while on the move 57 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Final testing has been carried out much in the same way as the city tests with introduction test and then post test activities However logging with the Virtual Observer was not done The trail following module is based on guiding mainly and on a multiple choice quiz A shake gesture to go on to the next point is also available 3 3 5 Results Results have been divided into several sections where the observations and the semi structured interview answers are treated together showing High level results both Scanning results city Guiding results bo
92. e unnecessary complexity 3b Be consistent with user expectations and intuition 3c Accommodate a wide range of literacy and language skills 3d Arrange information consistent with its importance 3e Provide effective prompting and feedback during and after task completion PRINCIPLE FOUR Perceptible Information The design communicates necessary information effectively to the user regardless of ambient conditions or the user s sensory abilities Guidelines 4a Use different modes pictorial verbal tactile for redundant presentation of essential information 4b Provide adequate contrast between essential information and its surroundings 4c Maximize legibility of essential information 4d Differentiate elements in ways that can be described i e make it easy to give instructions or directions 4e Provide compatibility with a variety of techniques or devices used by people with sensory limitations PRINCIPLE FIVE Tolerance for Error 119 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The design minimizes hazards and the adverse consequences of accidental or unintended actions Guidelines 5a Arrange elements to minimize hazards and errors most used elements most accessible hazardous elements eliminated isolated or shielded 5b Provide warnings of hazards and errors 5c Provide fail safe features 5d Discourage unconscious action in tasks that require vigilance
93. e very recent release of Android 4 1 Jellybean and we expect that it is not a problem in the future but that it is still important to keep such tutorials up to date lest they cause confusion Novel Application Building All participants got as far as completing the tutorial Only two of the participants progressed to working on a novel app One participant reached the stage of completing and testing it His app HaptiHunt see figure 3 6 1 employed the bearing module of the toolkit the same as used in sample app when installing the toolkit to direct users towards treasure that had been left around the environment The other user who worked on but did not complete his novel application intended to include the bearing module in the same way T s we enm HaptHunt 0 points Sien purger Figure 3 6 2 3 The HaptiHunt application created by one of the participants that employed the Bearing Module During the debrief session both participants noted that the bearing module was familiar to them and that is why they employed it Both participants noted that it was hard to find out what other modules were available what they did and how to use them This given the limited time encouraged them to use the module that was already familiar The lack of 96 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications documentation was cited as an issue by most participants They found a low level of discover
94. e viel Zeitdruck empfanden Sie hinsichtlich der Haufigkeit oder dem Takt mit dem Aufgaben oder Aufgabenelemente auftraten War die Abfolge langsam und geruhsam oder schnell und hektisch niedrig or or or or or 4 0 ee hoh Erfolg der Ausf hrung Wie erfolgreich haben Sie ihrer Meinung nach die vom Versuchsleiter oder Ihnen selbst gesetzen Ziele erreicht Wie zufrieden waren Sie mit Ihrer Leistung bei der Verfolgung dieser Ziele gut Ck s s s s s schlecht Aufwand Wie hart mussten Sie arbeiten um Ihren Grad an PAPAE zu erreichen niedrig rr 04 0 40 e hoch Frustration Wie unsicher entmutigt irritiert gestresst und verargert versus sicher bestatigt zufrieden entspannt und zufrieden mit sich ir f hlten Sie sich wahrend der ip niedrig 0 0 0 0 0 0 l hoh Geben sie an welche Beanspruchungsdimension f r die Gesamtbeanspruchung die Sie empfunden haben bedeutsamer war Kreuzen Sie die wichtigere Dimension an K rperliche A Geistige A Zeitliche A K rperliche A Aufwand Zeitliche A Zeitliche A Geistige A Erfolg K rperliche A Erfolg Geistige A Aufwand K rperliche A Aufwand Erfolg Aufwand Geistige A Frustration K rperliche A Frustration Erfolg 125 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications French Titre Descripteurs Descriptions Quelle a t l importance de l activit mentale et intell
95. eard while exploring and the goal that you had chosen Participants in the city were curious about the sounds and wanted therefore be able to explore the sounds to understand what they were and why they were there The sounds in the rural setting were more closely tied to the trail points in the near distance If you for example heard a sound from the use of scythe you would get information about the harvest taking place there in the 19 century Several participants reported on how the sounds helped to bring an atmosphere to the trail In the pre test in particular two participants commented on how much they appreciated them 3 3 5 9 Participant s subjective task load ratings city All participants filled in the NASA RTLX task load rating for the entire activity i e just once for each participant This limitation was done for practical reasons It would be hard to do a full NASA RTLX for each sub task in the test But in addition questions about the perceived concentration effort needed for the relevant sub task were asked in the post test semi structured interview In the full NASA RTLX chart for the city tests Figure 3 3 5 10 1 you can see that the perceived mental demand on doing the tasks and using the app varies between 2 and 6 The perceived physical demand is rated low 1 3 except for the two sighted users The temporal demand is rated quite low 1 3 by all except participant number 2 who had technical difficulties and was
96. ec dre rum durar tract tek nta tig 105 4 1 4 The terrain navigator seien trente rte tttm tnnt tente ttn tttn ttt ttt ttti tots ra ARR RR nn 105 D Bn dr 106 4 1 7 Toolkit use eyla ons nsina teen rtt aaa nghe nte Rb Fe i 106 4 2 Oe ee 107 4 5 Conclusion 25icicon amice a A a a i aa iaa 107 References iniiaiee aradenn issn dO TU Radon Tec WORD U caa EU Ka Kna KO EEEO aaa aaas 108 Appendix A Notes on Creating Tasks eren tht tenete tenete annan nnns 109 Diverse conditions of SQ aestus orae pte thich fara La M PER eb iata Ra Ta eR inten 111 Appendix B HaptiMap common background questionnaire ess 112 Appendix C HaptiMap common background questionnaire mandatory version 117 Appendix D Checklist for Universal Design Criteria eee 119 N tes FOL USE P AAS 120 Appendix E Laboratory based testing SUS scale eren 121 Appendix F NASA TLX task load test with printable and localized versions 123 LG untur PEDE 125 luii 126 WPAN SH es shade 127 ginti aaa Eaa ENE Tas 129 Swedish 157 Scale ice tto a a e aD 130 Appendix G Semi structured interview guide
97. ectuelle requise ex Exigence De Faible r flexion d cision calcul m morisation observation recherche etc La t che Mentale Elev e vous a t elle paru simple n cessitant peu d attention faible ou complexe n cessitant beaucoup d attention lev e uelle a t l importance de l activit physique requise ex pousser porter Exigence De Faible Q pol x die q P P j tourner marcher activer etc La t che vous a t elle paru facile peu fatigante Physique Elev e 2 parce calme faible ou p nible fatigante active lev e EE uelle a t l importance de la pression temporelle caus e par la rapidit Exigence De Faible Q x np P n P P P 2 n cessit e pour l accomplissement de la t che Etait ce un rythme lent et Temporelle Elev e ae tranquille faible ou rapide et pr cipit lev De Bonne a Quelle r ussite pensez vous avoir eu dans l accomplissement de votre t che Performance MPa Pie A Mauvaise Comment pensez vous avoir atteint les objectifs d termin s par la t che Effort De Faible Quel degr d effort avez vous d fournir pour ex cuter la t che demand e Elev mentalement et physiquement De Faible Pendant l ex cution du travail vous tes vous senti satisfait relax s r de vous Frustration niveau de frustration faible ou plut t d courag irrit stress sans assurance Elev niveau de frustration lev E
98. ed correctly according to the user s specific needs the test should lead to a successful experience for the user in which he or she is able to navigate through an unknown route and consequently identify that the system and the application are both useful Hence the specifications being contrasted must be linked on the one hand to the actual system and to the tools employed and on the other to the route being executed Another important factor to take into account will be the user s perception and opinion of the whole procedure 3 2 2 Functional and Non Functional Requirements 28 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 2 2 Specifications related to route guidance traversal N SPECIFICATION Through the voice Interface the user is able to use the Interface properly to explore and Navigate through the different functionalities N Through the tactile screen Interface the user is able to utilize the Interface properly to explore and navigate through the different functionalities It is able to upload a route saved on the system It is able to execute a previously uploaded route It is able to interact with functionalities related to the execution of a particular route It is able to execute a route with the help of GPS It provides information about the route The system provides a list of instructions waypoints and directions for the route
99. ed in the beginning of the HaptiMap project 1 2 1 Joined GeoMobile Joined is an application which supports finding friends on festivals and similar events Once a friend or location has been selected the direction to walk can be indicated by vibration sound or visually on a map Furthermore one can share locations with friends and stay in touch with them via short chat messages See figure 1 2 1 The application is available for Android and IOS and is fully accessible by people with visual impairments 16 25 16 25 Profile Bearing Nickname Chris Profile image Click to change n ee M profile Distance 3 2 km Accuracy 30 m Vibration Figure 1 2 1 Joined user interfaces 13 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 2 2 NavEscort Tecnalia The NavEscort demonstrator guides a visually impaired user to complete a route previously prepared by his her mobility trainer The application can be managed by the user independently from the visual channel It provides navigation and POI information using audio text to speech with the possibility of integrating a haptic device such as ViFlex See figure 1 2 2 Figure 1 2 2 urban routes for NavEscort tests 1 2 3 Lund Time Machine The Time Machine is a virtual tourist guide application It allows for the following of pre designed trails which also can be shared using a GPX format with a HaptiMap XML schema to sto
100. ed to the device the application and the different sensorial modalities that would be used to help them to orientate themselves when moving 3 2 7 1 2 Orientation in open spaces outdoors 40 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The route was carried out at Plaza Mayor in Madrid The features of this square are large open square with four lampposts one at each of the four corners with arches all around the square and with an equestrian statue in the centre The following inconveniencies were identified commercial vehicles delivering and loading until 11 00am and large amounts of pedestrians due to the touristic nature of the area 3 2 7 2 Test conditions A Mobility Trainer accompanied the users all the time during the tests in order to guarantee their security The complete route was recorded on video 3 2 7 3 Participants Demographics The characteristics of the sample which take part in the tests are shown in the Table 3 2 9 Table 3 2 9 Session 3 participants TOTAL FAMILIAR WITH USERS AGE GENDER VISUAL CONDITION MOBILE TECHNOLOGY 4 36 57 57 40 M F Blind Residual vision Yes No 3 1 2 2 3 1 3 2 7 4 Test characteristics and observations by each user Viewing the recorded videos a detailed analysis of the trajectories followed by the users when moving from a point A to a point B has been done This analysis has been possible thanks to the hor
101. en 20 intervalos iguales puntuaci n que es reconvertida a una escala sobre 100 y limitada bipolarmente por unos descriptores por ejemplo elevado bajo y teniendo presentes las definiciones de las dimensiones 127 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 128 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Finnish NASA Task Load Index NASA TLX mittarin kehitt j on Human Performance Group NASA Ames Research Center Sen avulla mitataan ty n henkist kuormittavuutta jolloin tarkastellaan erikseen seuraavia kuutta rasituksen dimensiota Lopuksi n ist kuudesta lasketaan painotettu keskiarvo 1 Henkinen Rasittavuus 2 Fyysinen Rasittavuus 3 Vaativuus Ajoituksen Puolesta 4 Suorituksen Vaativuus 5 Vaivann k 6 Turhautuminen Mainittujen kuuden dimension keskin inen painoarvo vaihtelee eri t iss joten aluksi pyydet n vastaajan mielipide niiden painoista T m tehd n siten ett vastaaja vertailee eri alueita pareittain Seuraavaksi vastaaja saa arvioida tekem ns ty n rasittavuutta useilla eri asteikoilla joiden tarkempi sis lt on alla 1 Henkinen rasittavuus mental demand kohdassa vastaaja arvioi miten paljon teht v ss tarvittiin henkist ty t kuten ajatteleminen p tt minen laskeminen muistaminen tarkkailu etsiminen jne Pyydet n arvioimaan oliko teht v helppo vai vaativa yksinkertainen va
102. equences with different combinations of sound and vibration during the festival Time 12 13 14 15 16 17 18 19 20 21 22 23 0 01 02 03 S Def 0 3 5 8 10 3 12 7 10 3 10 0 9 7 45 52 24 10 7 83 3 4 42 17 S Opt 1 2 46 13 8 12 6 18 4 115 46 69 3 5 2 3 9 2 58 2 3 1 2 2 3 Vib 1 8 34 5 4 12 1 13 3 85 85 85 12 3 6 10 3 13 9 6 7 12 12 3 5 2 3 Use of the Facebook linking functions Uses per hour e 20 A e Figure 3 5 2 3 1 Use of Facebook list function of Juicy Beats app during the festival 89 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications uw A e Total Seq I z e E o n o n E FaceBook Figure 3 5 2 3 2 Total use sequences and use of FaceBook list of Juicy Beats app during the festival The Juicy Beats app provided a function for users to search for a friend from their Facebook account at the festival This function was used in 282 sequences Figure 9 shows the distribution of the use of this function during the festival Figure 10 shows the use of the guiding functions in comparison with the overall use of the app From these figures we can see that the pattern of use is rather different from the overall pattern of use of the Juicy Beats app The use of the Facebook function shows an earlier peak aro
103. ess if you want to guide users very close to a smaller object particularly if users are visually impaired would you might want to guide them particularly closely But also sighted individuals can have problems with this the 7 year old participant clearly missed a point partly possibly because he was in a hurry to the next point but also because it was unclear what the goal was The point was showing an old well but since it was overgrown you needed to look closely to find it There was a difference in the mode of the app when arriving at a point in the city and rural setting In the rural setting trail mode the user needed to reach a point before reading the information in it In the city test explore mode the user could read it from afar Also the arrival was somewhat different in the trail a notification sound played as you reached a POI but not in the city There was a difference in the comments from users on the arrival behaviour In the city explore mode users commented that they would want to arrive at a point and start the playback of the information when ready The users in the rural setting were happy with the virtual guide starting to speak as soon as they arrived The users in the 61 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications rural setting however commented on that they wished for at end of text notification after the guide had finished speaking In fact one participant mistook a n
104. f description of existing solutions for this domain In creating the task you should evaluate the difficulty by actually using the device and doing the task yourself for example following the route you have chosen This to some degree may bias the test towards making the routes too easy Make the tasks to take around 30 50 min More than that creates the risk of end user fatigue especially in fragile populations and observer loss of attention One of the goals in this last evaluation of demonstrator applications is to verify that you have successfully implemented the accessibility goal that the specifications called for Here is an example Time Machine The first plans for a historical game application had the following requirements Table A 1 Functional requirements from D 5 1 Nr Requirement Fl Provide information multimodally about which direction the user can find things Distance information should also be provided F2 Provide audio and or haptic notifications as the user passes interesting spots Allow for more information on demand F3 Allow the user to personalize the feedback F4 Keep track of scores clues and challenges F5 Allow both explorative and guided navigation F6 Provide the user with multimodal overview information both of the current environment and of how things were before On screen information should be complemented with audio and or haptic feedback F7 Possible to select objects both by screen
105. f points of interest 3 3 5 4 Guiding results The vibration guiding used by all visually impaired participants and all except one sighted participant was well received Comments like Strong and good 1 user Pleasant 2 users and The vibrations were good 1 user were uttered during the interview and 8 users spontaneously answered the guiding when asked about the easiest part in the application One user in the city occasionally had problems feeling the vibrations but it was on a particularly windy day and the problems seemed to occur at windy places In the rural setting participants reported several problems when the guiding direction was a bit off the track and they reported losing the signal at several occasions One user wished for a signal that was more continuous also when standing at a long distance from the goal the frequency increases as one walks closer to the goal In the particular forest and in the surroundings chosen it was also less obvious how to choose paths or if you should walk off the beaten track The difference between having to make a detour in the city or in the forest is substantial A detour in the city is seldom serious but in the forest you may need to walk very rugged terrain with underbrush or steep slopes Regarding routing only city it was observed that users with severe visual impairment in general preferred to have routing on In the case where users didn t have the routing co
106. g Oro Fa ERA a e e EUR EHEVETE 75 Figure 3 5 2 2 1 Total use sequences of Juicy Beats app during the festival 82 Figure 3 5 2 2 2 Programme of events at Juicy Beats Festival 2011 83 Figure 3 5 2 3 1 Screen from Juicy Beats app showing guiding function 84 Figure 3 5 2 3 2 Use of guiding sequences of Juicy Beats app during the festival 85 Figure 3 5 2 3 3 Total use sequences and use of guiding sequences of Juicy Beats app during the fesllVal da eid nce Aem eA kA Hen goce en a e c oan o nd Ea 85 Table 3 5 2 3 1 Breakdown of guiding only sequences into those ending with sound and those ending with vibration seo aeter rr ERE rr eet ERR DAD netta dla M rr rr ran nn 86 Table 3 5 2 3 2 Breakdown of guiding included sequences into those ending with sound and those ending with vibration secs icc e ui Ceu x e kin tenen a Ceci 87 Figure 3 5 2 3 4 Use of guiding only sequences with different combinations of sound and vibration during the TestIVal s si toe rete e p SER p ha eo et Sd 88 Figure 3 5 2 3 5 Use of guiding included sequences with different combinations of sound and vibration during the festival eo e e ret trea ta ema 88 Figure 3 5 2 3 5 Use of both guiding included and guiding only sequences with different combinations of sound and vibration during the festival 89 Table 3 5 2 3 3 Percentage use of both guiding included and guiding only sequences with d
107. ge range of users will be able to use the app approximately 7 70 years YES G8 Users will be able to use the explore mode with an introduction to it Users with little or no Android or other smart phone experience will be able to navigate YES the interface after an introduction We extracted a sub set of the functional and the non functional requirements from the tables above that fit to the test tasks and the current functionalities The table has been completed with information about which of the test situations that covers the claim and a column showing if the requirement has been met according to the results see Section 5 Table 3 3 1 4 Extracted requirements table subset of Table 1 and table 2 Nr Requirement Test Fulfilled F Allow explorative navigation select points City F YES F7b Possible to select objects by pointing to the objects in real YES life n City Rural NF2b Possible to use together with others City Rural YES F13 new Basic functionality available without vision 3 3 2 Brief description of evaluation procedure The evaluation was carried out in context In the city scenario users were asked to use the free explore function where the user scans by pointing with the mobile phone for historical points of interest and chooses one to walk to The points of interest in the city were collected with the help of the archaeology department the culture historical museum and the building preserv
108. has been tested by in total 9 visually impaired adults and 2 sighted participants in the city centre of Lund Sweden scanning for points of interest and walking towards them Additionally 12 persons have used the trail feature of that same application while walking a hiking trail outside Riseberga Sweden The testing was led by ULUND The Time Machine app aims at users with a variety of visual capabilities In the city scenario users were asked to use the free explore function where the user scans by pointing with the mobile phone for historical points of interest and chooses one to walk to In the rural setting following a trail was more appropriate Both users with visual impairments and full vision can follow trails and hear pre recorded information about historical sites as well as explore the surrounding points of interest and can be guided to them by using the multimodal components explicitly developed for the Time Machine application There is also an accessible multiple choice quiz game that was used in the rural test 17 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 3 4 The Terrain Navigation The Terrain Navigator has been developed by FGI and has been tested by 12 users in the national park of Nuuksio Finland The testing was led by FGI It is targeted at all kind of user groups sighted elderly and visually impaired The application guides a visually impaired user to complete a route prev
109. he Toolkit Table 3 6 1 4 Use of HaptiMap Toolkit components Component Number Percentage of students Haptic Guide Tutorial 7 50 0 Examples modified for the project 12 85 7 Modules used without modification 2 14 3 Compiled the NDK version 1 7 396 Used platform independent parts 0 0 0 From this table we can see that the students were significantly positive about the usefulness of the functionality of the Toolkit Mean rating 3 94 significantly higher than the midpoint that the Toolkit functioned as expected Mean rating 4 28 significantly higher than the 93 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications midpoint and that the Toolkit provided a functionality benefit in that they were able to develop an application with a higher level of functionality in the time frame available than they would have managed to do without it Mean rating 3 88 significantly higher than the midpoint For the other aspects of the Toolkit neutral ratings neither positive nor negative were given The students were asked to provide any additional comments that they wished to about their use of the Toolkit Positive comments were e didn t use that many functions but those we did use worked splendidly e Guide Tutorial was good Comments suggesting further work were e would have liked support for Android 2 1 e had some problems compiling the Toolkit in
110. he elderly people had some minor problems to get used to the User Interface and understand the purpose of the application they rated the application with an overall excellent mark see figure 3 1 8 Table 3 1 8 Subjective user rating of the Joined app older users UI Marking App Overall 4 4 2 3 2 1 3 3 2 2 2 2 2 2 1 Avg 2 8 2 6 1 6 In conclusion the participants in the second group were quite satisfied with the application its purpose and the slim UI What made it a bit difficult for them was to immediately understand the purpose of the app and intuitively handle the UI They liked the vibration guiding concept especially when on route to an unknown destination which one can mark on the map by moving it around and thus being able to mark a place other than the current location Using it to find friends for example on a festival area like in the Juicy Beats evaluation did not have such a high importance 27 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Summarised both target user groups liked the application and the guiding concepts via sound and or vibration however for different reasons All participants agreed that directional information via sound or vibration is intuitive to understand and process with the added benefit that vibration offers a hands and eyes free unobtrusive usage 3 2 NavEscort Once 3 2 1 Introduction user trials This report
111. he pocket or in other conditions where screen interaction is hard glare you want to look at the surroundings etc All users were able to use the HUI but they had varying difficulties with it With select on release for the users with visual impairment a higher rate of accidental select events was observed Since the Back button was physical a Sony Ericsson Xperia Neo was used for all tests except one the users easily recovered from the errors However one of the sighted users commented that she had problems remembering to use the back button Six of the users mentioned the HUI navigating the application and understanding the interface as being the hardest part of the application We interpret this in part as being a result of combing different interaction modes in the same application Selecting on releasing buttons in the HUI and choosing distance ranges while keeping the finger on the screen became confusing for some The placement of buttons was also a problem a user suggested a standardised placement with only the names of the buttons changing Because of technical problems it was also not possible to make the interface speak out the screen names when an Android activity started which was requested several times 3 3 5 7 Arriving at a POI results To be less vulnerable to GPS inaccuracies the POI arrival has a 10 m offset 1 e in a radius of 10 m from the position the user is considered to have arrived This is of course a weakn
112. he was following the route correctly absence of vibration so he kept walking on a straight line 36 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications ETTITA TTA riri GELEI a iL p 2 Figure 3 2 6 2 1 One of the interpreters communicating with one of the users Figure 3 2 6 2 2 illustrates a comparison between the ideal routes the user SD1 had to accomplish in green and the paths finally carried out by him in red Some information has been also included about the modality used in the test and the time expended in completing the exercises d A PS ERAT O O o Modality Vibration Modality Vibration Time 3752 Time 6 36 Figure 3 2 6 2 2 Routes accomplished by user SD1 User SD2 is a 35 year old woman who uses a guide dog for mobility assistance She moves with independence and uses public transport if needed Although she has been defined as having a deafblind profile she has some residual hearing which allows her to understand verbal instructions with the aid of an amplifier when she is spoken very close to her ear During the first test 3 24 went by before she figured out how to walk on a straight line up to that point she could only move in circles We interpreted that she was insecure and couldn t make a decision 37 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Figure 3 2 6 2 3 One of the participants during the te
113. hey normally do The tests ended with the verbal instructions modality Modality Vibration Modality Stereo sound Modality Orals instructions Time I 52 Time 1 41 Time 1 55 Figure 3 2 12 Routes accomplished by user E6 42 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 2 7 4 4 User E7 This participant was totally blind however he she had to wear sunglasses in order to avoid dazzle He she used cane The modalities used in the tests were the following ones in the order in which they appear in the list below e Vibration e Stereo sound e Verbal o clock hours orientation After completing the tests the participant showed a lot of enthusiasm and interest The participant had to modify his her natural way of walking because he she went ahead the instructions given by the application The user pointed out that the instructions were slow in relation with her his stops and that he she had to slow down in order to get the instructions properly LET _ Modality Vibration Modality Stereo sound Modality Orals instructions Time 41 Time 1 15 Time 1 48 Figure 3 2 13 Routes accomplished by user E7 3 2 7 5 Test analysis 3 2 7 5 1 Usability Scale results Table 3 2 7 5 1 SUS results USERS E3 E4 E6 E7 DIRECT P 33 38 32 25 CONVERTED P 82 5 95 80 62 5 3 2 7 5 2 NASA TLX task load test Table 3 2 7 5 2 NASA TLX ITE
114. hey were able to use the Juicy Beats application during the festival Table 3 5 1 3 1 2 Scores from the System Usability Scale for the Juicy Beats evaluation 77 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Participant JB1 JB2 JB3 JB4 JB5 JB6 Mean SUS score 87 5 80 92 5 80 100 95 89 17 3 5 1 3 2 Semi structured Interview Overall the Juicy Beats application was well appreciated by the participants Two of them cited the whole application when asked about what was easy All the other participants cited the Friend Finder as one easy part of high importance that could be used as a stand alone see Joined All participants except one could do the tasks confirming the usability of the system both by sighted and visually impaired users One participant could find the stages but not meet friends it seems the friends were moving and a notification that someone was trying to get to them could have helped One issue that is revealed by comparing the answer from sighted and visually impaired participants is that the visual location map and being able to locate things or oneself on it was very important and easy This feature was not cited by visually impaired participants and in fact was not made accessible to them The location map was rated as important by all participants Being able to locate all the music stages also an important point The guiding was given by vibrat
115. i monimutkainen 2 Fyysinen rasittavuus physical demand sis lt ruumiillisen ty n arvioimisen oliko se helppo vai ty l s kevyt vai raskas 3 Vaativuus ajoituksen puolesta temporal demand mittaa ty n ajallista painetta oliko toiminta verkkaista ja rauhallista vai kiihke ja vilkasta 4 Suorituksen vaativuus performance kohdassa vastaaja arvioi miten h n onnistui teht v n vaatimuksissa 5 Vaivann k effort kohdassa vastaaja arvioi miten paljon h n suorituksessa joutui ponnistelemaan henkisesti ja ruumiillisesti 6 Turhautuminen frustration kohdassa vastaaja kertoo leimasiko teht v n suoritusta ep varmuus lannistuminen rsyyntyminen rasittuminen ja huolestuminen vai p invastoin varmuuden itseluottamuksen ja tyytyv isyyden tunteet Henkinen vaatimustaso Valu Paljon Fyysisen vastimustaso Vihia Paljoa Ajallisen vaatimustas iis a m Nofea Suontus nin palmi V h n i mE CENE ucc EE Paljoa Turhautueninen 129 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Swedish 1 7 scale Nedan f ljer sex skattningsskalor som ber r olika aspekter av arbetsbelastning som kan t nkas bidra till den totala arbetsbelastningen under ett arbetspass Var sn ll och indikera Din arbetsbelastning under det gangna arbetspasset genom att ringa in en siffra mellan 1 lag belastning till 7 h g belastning Lag Hog 1 Hur s
116. iPhone application aimed provideing vistiors with information about the annual Juicy Beats open air music festival in Dortmund hosting more than 100 acts on 14 stages See figure 1 2 5 The application gives directions to stages friends and facilities through sound or vibration patterns Details about the events and artists are given on screen and are also made accessible through text to speech EDUC Pe IT GB FriendFinder Juicy Bests Twitter News iphone Guide Antonio Ca vor 16 Tagen 6 Neuigkeiten vom Juicy Beats Team 3 iic 5 2km 3 vor 26 Tagen JUICY BEATS TWEET d rA Lisa Ambb 815m Act of the day Mit einer Tim Ontrup VO 14 Tagen fantastischen Elektro Dub Deep 10 6km FAW Funk Tropical Symbiose angef hrt vor 15 Tagen 6 von QUANTIC und ayhan 3 1km http fo me wdNIRGxx simu Am Samstag den 25 6 beohrt Carsten Helmich der Urvater des rae Juicy Beats Festivals h chst pers nlich das Daddy hte 7 Vr Far Fier dF ieee Figure 1 2 5 Screenshots from the Juicy Beats event guide From left to right First screen main map friends list tweet feed 1 2 6 Developer evaluations In the HaptiMap review in November 2011 the project was requested to more systematically carry out evaluations with developers who were the target of the software toolkit to create more accessible map and navigation applications The description of the toolkit and its uses can be found in deliverables D 4 2 4 3 and on the
117. iddle highschool 48 1 Journalist state exam Avg 42 182 1 364 The second group were comprised of five elderly male users with an average age of 57 and an iPhone level of experience of 2 2 years These participants were selected randomly at the city centre of Paderborn Participants of both groups have been informed about the HaptiMap project and the reason of the test Afterwards all participants signed a written consent form Both groups had to follow the same test procedure Appendix A with the difference that for the second group the time needed to accomplish tasks A to H was not recorded We made this change for two reasons 1 The focus of the test for the second group lay on the evaluation of the user interface and concept for elderly people 2 All participants accomplished the tasks A to H in a rather speedy timeframe 22 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications c e No SIM 0 2 _ i 4 F A j a Friends 2 a Friends Y VP rim Android 11 4 km 16 2 hrs ago vochen 3 18 km 2 days ago 21 m 4 1 hrs ago PE 9 m 39 mins ago Uli Mann Tim Android 60 m 13 mins ago pe 75 4 km 17 2 hrs ago d feng 4 13 m 0 mins ago Places Tim Ontrup 3 m 2 days ago Uli Mann 19 6 km 1 day ago Tim Ontrup 1 J Michael Gerhard F E Mensa 660 m fen Settings 4l 19 m 45 mins ago Profile Deactivated Change your profile here gt Nis T al Deact
118. ied to the app From these they were asked to mark the 5 most important words Making this exercise with a visually impaired participant requires reading all words one by one and deciding whether it applies or not A sighted user would get an overview of all the words and rather chose a subset of these This can be seen by comparing the number of words chosen by the different sub groups The users with visual impairment have chosen in between 45 and 59 words each and the sighted users 6 to 25 words each The choice of the 5 words that applied the most was the hardest part for the group of visually impaired users as it also requires the overview or remembering all the words which was impossible The total checked words were counted and two word clouds were made out of them One used all the checked words the other all checked words and added the most important words on top That way the most important ones were counted twice Seeing from the city test the most prominent describing words using the weighted figure with the important words counted twice were Fun 17 Stimulating 16 and Usable 16 The resulting word clouds in Swedish can be seen in Appendix I as well It was observed during the word choice task that the words which were taken from an existing list were not entirely appropriate to multimodal use and also not for educational use For example users found the word interesting missing and also words for tactile
119. ier trials experienced were present again in the city trials although there was some success using un spoofed GPS in the open space trials Outside of the formal requirements most apparent were the need for much larger amount of training time as the level of ability fell This was demonstrated the most in the blind deaf trials The trials with mobility trainers using the training tools did not match formal requirements to trials but in general the trainers could use the system to generate routes that took into account accessibility needs of individual users A good summary of the state of the applications was the quote 104 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications In general terms the application was considered useful and adapted to the needs of the MTs as well as intuitive and easy to use No item in the questionnaire got its maximum score This fact has been interpreted as positive prudence as everything can be improved There appears to be a need to provide better support to various forms of maps and perhaps support for mobile creation of routes 4 1 3 The Lund Time machine The Lund trials did not test all the formal requirements largely because the trials were not suitable for the missing functionality Some of the requirements were only partially satisfied such as Table 4 1 3 Time machine requirements X designates partial fulfilment Description F6 Provide the user with multi
120. ifferent combinations of sound and vibration during the festival 89 Table 3 5 2 3 Breakdown of frequency of calls to Facebook within a use sequence 91 Figure 3 6 2 3 The HaptiHunt application created by one of the participants that employed the Bearing MOGUule ero teg eo noe cette reta ao ete E conet sepe boda 96 Table 4 1 3 Time machine requirements X designates partial fulfilment 105 Table 5 1 4 Terrain Navigator requirements X designates partial fulfilment not JL E cT 105 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications List of tables Table 1 3 End user participants in demonstrators m ssmmmmsrrrrrsrrsssererrrrrrrrnnnnrrrrrrnr rna nan 16 Table 2 1 5 NASA task demand faCtOrS seesesessseerrrrrrrrennnnnrrrrrrrrrnnnnrrrrrrrr ren nnn rn rr rr rna 21 Table 3 1 1 Participant basic demographics group 1 and users with visual Moanne gm Eere Do microns BA E RS Bel E EIE ORO Gente TRI SLUT ARR EE 22 Table 3 1 3 Results of time to accomplish tasks in minutes 24 Table 3 1 4 Subjective user rating of the Joined QPP ssrrrrsrrssrrssseeerrrrrereennnnnnrrrnn nn 25 Table 3 1 5 Participant basic demographics group 2 older users 26 Table 3 1 6 Independent travel by older seeeeeseeseesssssee 27 Table 3 1 7 Activities by users with visual impairment
121. ined by the fact that the user is accompanied by the technical support group all along the route 3 2 5 4 3 User E2 The subject is completely blind and a cane user He pointed out that the synthesizer s speed was too high and that he could not therefore understand some of the phrases being played The speed was decreased from 2 0 to 1 5 by using the configuration menu The user experimented problems with the verbal instructions in clock times at the beginning He kept the initial reference fixed making turns in relation to this initial reference instead of changing the reference every time he made a turn After a brief explanation the user successfully completed the route in open environments 3 2 5 4 4 User E5 User E5 has some residual vision He carried out the route with the aid of a cane with which he is familiar and wearing an eye mask During the familiarisation phase he indicated that he has problems pronouncing correctly the letter Il However the voice recognition software understood commands such as list of route instructions which include that letter without any problems He liked the bone conduction headphones very much He even asked their price He said that the sound mode was the easiest out of the three modes in open environments In addition to the information about the turn he also says that the variation of sound frequency has been very useful when indicating distance When completing the urban route there were
122. ing to the link between the application and the network while we were interested in the actions related to direct user interaction input or feedback The logging plan needs to be 91 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications defined before the study and in tighter collaboration between the researcher and the developers so that the data can be usable and more related to the questions of interest for the researcher It can well be that this different kind of logging will create technical issues like a delay in the application due to logging An iterative piloting of the application with logging should thus be included in the planning before the real study happens 3 6 Developer evaluations 3 6 1 Students in course at Lund University 3 6 1 1 Introduction The purpose of the evaluation was to identify the usability and practicality of the Haptimap toolkit from the perspective of student developers that have no prior knowledge of the project The evaluation was conducted in the context of the Advanced Interaction Design course taught between October 2011 and December 2011 to students majoring in Information and Communications Engineering and Computer Science Engineering at Lund University 3 6 1 2 Participants Thirty eight students took the Advanced Interaction Design course Thirty students returned the questionnaire distributed at the end of the course asking them to evaluate the HaptiMap
123. io messages from the system are useful and do not overload the user 3 4 2 Description of the Evaluation Process The evaluation process was divided into two parts The first part of the usability evaluation was carried out as field tests in Nuuksio National Park near to the city of Helsinki in Finland in May 2012 Each test situation began by briefly introducing the purpose of the test The participants were asked to fill out a background information form which consisted of questions regarding their hearing vision and motor abilities as well as their experience with mobile devices and mobile map services The test tasks included navigating with the Terrain Navigator and testing the audio functionality of the application on the iPhone while walking in the National Park The Terrain Navigator s audio function can be thought as a metaphor for a museum tour during which additional information is given on interesting subjects After completing the tasks the participants were given a NASA TLX form and semi structured interviews were conducted All the tests were documented by video We organised the tests for two participants at a time Two usability practitioners facilitated the tests that included an approximately 2 km walk in the woods Figure 3 4 2 shows the route on a map Each test situation including tasks with the Terrain Navigator and an interview afterwards took about two hours all together The weather conditions varied from 11 C to 14
124. ion or sound The audio tones were disliked by half of the participants The vibrations were described as pleasant and requiring less focus but being less fine grained than the audio Geiger The GPS connection was also a problem in the guiding Sometimes the GPS would turn off without giving a notification The user would have to figure it out and then wait to get back the connection This problem was a reason of the failure to meet friends and of getting lost for one sighted user who had not constructed a mental map of the area because relying too much on the applications guiding The guiding was given as the crow flies This was a problem for the participants with visual impairment as the environment of the festival had rendered the area difficult to navigate outside of the paths There were also water and trees on site making it impossible to rely only on the applications guiding to navigate without vision Here are some of the proposed missing features Alert your friend of your arrival when guided to them so they expect you Display the name of the closest stage to each friend if visually impaired or display the friends on the visual map Give routing along paths to make independent navigation possible Other parts of the application like the ability to get practical information or to plan for what to view through the detailed program and the My Beats favorites was judged as interesting and used by at least one partici
125. ions using o clock hours given by speech synthesis the participant arrived at destinations without hesitation There were some technical problems during the tests that were solved on the spot by calling a support technician 3 2 7 4 2 User E4 This subject also had some residual vision The participant used the cane a mobility aid to which he she was very familiar The participant walked very quickly and went ahead the instructions given by the application The tests began with the vibration modality They continued with the verbal mode orientation instructions using o clock hours given by speech synthesis They finished with stereo sound modality r EI Modality Vibration Modality Verbal instructions Modality Siero sound Time 1 Time 0 53 Time 2 Figure 3 2 11 Routes accomplished by user E4 3 2 7 4 3 User E6 User E6 was completely blind he she used the cane Before starting the tests the different instructions for each modality were explained The tests began with the vibration modality but the first two attempts were considered invalid due to the lack of comprehension of the instructions given by the application These two attempts served at least to get the participant used to the vibration patterns Then the stereo sound modality was used Along the route the user found an obstacle equestrian statue and he did not know how to face it Finally the mobility trainer gave him her indications to skirt the obstacle as t
126. iously prepared by his her Mobility Trainer MT The main actor group of the demonstrator is composed of ageing users who have age related deficiencies more specifically problems with vision dexterity and mobility However completely blind users are not included in the actor group since one of the FGI s main expertise is in visual spatial information and not the development of haptic devices Consequently one of the main aims of the demonstrator is to prove that good quality raster formatted background maps in combination with vector data and audio can be used to improve accessibility 1 3 5 Juicy Beats The Juicy Beats evaluation was conducted in July 2011 during the Juicy Beats musical festival in Dortmund Germany The Juicy Beats application was available on iPhone for all visitors of the festival to use The application contained information about the festival and its musical program as well as a guiding module to get the directions to the scenes or to your friends through vibrations or audio Soest GeoMobile Siemens and ULUND were involved in the testing A qualitative evaluation was carried on site with 7 invited participants Three of the participants had a visual impairment The participants were asked to walk around the festival while using the application to navigate the event The users reported a good usability of the application with a specific interest for the Friend Finder part which then evolved into the Joined applicati
127. ivated for 2 days Info Figure 3 1 1 Joined screen shots Even though the Joined application is available for Apple s iOS and Google s Android we decided to conduct the user study on the iPhone due to the fact that the accessibility of iOS is much better Voice Over and though Android provides some accessibility features the number of people with visual impairments using Android devices is rather small Table 3 1 1 shows the answers to question D1 in the HaptiMap questionnaire in appendix B 1 means very often 6 never Table 3 1 2 Transportation Mode Usage Bus Train Metro Tram Taxi _ Foot Priv Car Work Shopping Leisure 1 2 0 5 1 4 1 1 3 4 4 1 3 2 3 1 4 1 6 5 1 4 1 5 1 4 4 2 3 3 5 1 6 5 3 5 1 5 1 4 1 1 1 3 1 1 5 6 6 1 4 1 5 1 3 4 6 6 1 6 1 4 4 1 3 5 5 1 5 6 3 2 1 1 4 6 2 5 1 4 3 2 2 1 5 1 3 1 4 1 1 5 1 4 1 2 1 3 3 2 09 3 55 2 644 5 1 18 4 1 81 3 45 2 54 All tests were carried out between June 1 2012 and July 27 2012 in Paderborn For the first group the premises of the Heinz Nixdorf Institut and the neighbouring Heinz Nixdorf Museumsforum was used in order to ensure the safety of the participants The second group as mentioned above carried out the test in the city centre of Paderborn at the town hall and surrounding area pedestrian zone The participants of the first group were all well trained in orientation and mobility O amp M and fre
128. izontal reference lines that exist in the square where the tests took place Completing the information about each participant some figures has been added which illustrate a comparison between the ideal routes the user had to accomplish in green and the paths finally taken by him he in red Some information has been also included about the modality used in the test and the time expended in completing the exercises 3 2 7 4 1 User E3 The participant had some residual vision The participant used the cane a mobility aid to which he she was familiar Before starting the tests the different instructions for each modality were explained The participant used his her visual remain in order to avoid several obstacles in the route The test began with the vibration mode It took him her a lot to orientate using this mode He she moved in circles The first two tests were not valid because of the difficulties experienced by the user in discerning and interpreting the stimulus received by him her gt O e Modality Vibration Modality Verbal instructions Modality Stereo sound Time 1 47 MOM Time 2 30 Time 1 39 Figure 3 2 10 Routes accomplished by user E3 41 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The second modality used was the stereo sound mode with which the user felt himself herself more relaxed and comfortable With the verbal modality orientation instruct
129. ks are located on the left right in front or behind the user FC056 The system deliver route guiding notes in vibrations FC038 The system warns clearly of inaccurate positioning state FC030 The user can define a route to be stored in the system in advance FCO18 A route path may have a difficulty risk rating that is visualised or otherwise indicated to the user FC019 The system calculates different route path alternatives and suggests them to the user FCO16 The system can calculate an optimal route and guide the user trough the route FC037 The system notifies on inaccessible routes FC026 The system can inform the user in which physical direction the North is gt lt FC036 The system provides a possibility to zoom and enlarge text as much as possible Based on the requirements listed above we chose the following topics for the usability evaluation Table 3 4 1 3 70 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 4 1 3 Evaluation topics for the Terrain Navigator Nr Evaluation topic 1 The system user interface is intuitive and easy to use for the first time user no instruction book needed The system is usable for users who have problems with vision The system is reliable The system is attractive The user location is accurate and it is shown fast al mY A oO N Aud
130. le phone we recommend Multimodal guiding signals can be confirmative rather than repellent i e the presence of a signal confirms that you are on the right track and not necessarily you are on the wrong track The guiding signal needs to be pleasant and legible Put effort into designing the signal For example with vibration consider that the signal has to be felt in different situations and weather and maybe also in a pocket Provide multimodal distance information to the goal This can be made in several different levels of detail for example o The vibration signal gets more intense as one nears the goal o ATTS voice speaks the distance with regular intervals Provide distance information according to the precision of the system which in this case should not be more detailed than multiples of 10 m Provide the user with a choice of routing turn by turn or guiding as the crow flies The second will let user s choose a path more freely and possibly help understanding the environment If using turn by turn routing in combination with vibration guiding consider adding a speech notification at sharp turns otherwise the user may lose the signal Provide a means to let the user know his or her current position preferable in address form Arriving at POIs Use a 10 m offset radius around points to enable users to more easily reach information points even if they are slightly off Otherwise points may be st
131. lkit What might you use it for 7 USE OF SOUND AND VIBRATION IN APPS Have you ever wanted to add sound or vibration to an app in the past How did you do that Do you think sound and vibration will be important aspects of apps in the future 8 ANYTHING SIMILAR TO HAPTIMAP TOOLKIT Have you encountered any toolkits similar to the HaptiMap toolkit If so how would you compare them both in functionality that they offer and ease of use 9 JUST A FEW QUESTIONS ABOUT YOURSELF Your age Gender How long have you been developing mobile apps How many apps have you developed Do you develop for a living as a hobby as part of my studies Questionnaire version Questionnaire about use of the HaptiMap Toolkit This questionnaire is completely confidential information will only be passed to the HaptiMap team in an anomymous way grouped across a number of respondents So please feel free to express your views about using the HaptiMap Toolkit 1 DOWNLOAD ISSUES 1 1 How easy was it to download the HaptiMap Toolkit 1 2 Was it easy to find the download page on the HaptiMap website 1 3 Was the process clear and as expected 142 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 4 Any other issues you would like to raise about downloading the HaptiMap Toolkit 2 DOCUMENTATION ISSUES 2 Did you use the Toolkit tutorial Yes No 2 2 Was it useful 2 3 Was it clear 2 4 Any other issues you would
132. m The NASA RTLX workload test was given directly after the field test the original research suggests NATO 2001 not more than 15 minutes from the end of the task See below for an overview and Appendix F for localized versions and printable ones The NASA RTLX is the basic NASA TLX test without the pair wise comparisons and resultant weighing 4 High correlations have been shown between the weighted and unweighted scores Byers Bittner amp Hill 1989 Moroney Biers Eggemeier amp Mitchell 1992 20 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 2 1 5 NASA task demand factors Definition of Task Demand Factor How much mental and perceptual activity was required e g thinking Mental ae 9 demand deciding calculating remembering looking searching etc Was the task easy or demanding simple or complex exacting or forgiving Physical How much physical activity was required e g pushing pulling turning demand controlling activating etc Was the task easy or demanding slow or brisk slack or strenuous restful or laborious Temporal How much time pressure did you feel due to the rate or pace at which the tasks or demand task elements occurred Was the pace slow and leisurely or rapid and frantic How successful do you think you were in accomplishing the goals of the task set by Performance the experimenter or yourself How satisfied were you with yo
133. markable number for a novel framework that was developed over a time span the saw major platform changes as well as changing user expectations as the technological context changed over time Further the iterative nature of the design process made some requirements obsolete 4 1 By Evaluation 4 1 1 Joined This finding and meeting friends and places of interest application largely fulfilled the base requirements of being easy to use installing the app registering to the system finding people in difficult environment The best summary of reaching these goals was Therefore nearly all users came to the conclusion that Joined is what we were missing especially the unobtrusive means of navigation by vibration thus not requiring to interact with the phone or concentrate on auditory instructions The several improvements mentioned in the evaluation were focused on improving small user interface behaviour and adding small amounts of functionality 4 1 2 NavEscort and mobility trainers The evaluation of NavEscort with various types of users tested the 12 specific requirements and with the exception of some small problems with identifying where the user deice was on demand fulfilled all the requirements The main positive result that came from the NavEscort trials was the wide range of users with disabilities that could successfully use the system and the importance of multi modal navigation guidance The same GPS problems that the earl
134. me details 9 Dominant hand ILeft __ Right C Experience of Mobile Devices 1 Do you own a mobile phone lYes __ No 2 If yes how long have you owned a mobile phone __ Less than 6 months l6 months to 1 year I1to2 years __ More than 2 years 3 What is the model and brand of your mobile telephone In a typical week how often do you use you mobile phone to nnn IN NN NN NN pu mm qo uq p cip Ecc NE NN RN NE NNI ee Ne ERN NN CNN NNNM ue EM qu 10 Listen to music MR ee Se 12 Web browsing 13 Other 11 Use the alarm 113 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications D Everyday navigation all optional except question D10 Once twice 2 Please provide some details on your other means of transportation 3 In a typical week how many times do you travel for the following reason 4 Please provide some details on your other reasons for travel 5 How often do you bring the following material on your travels DUM IL dL Lj 114 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Eme 0 6 If other please specify what 7 How often do you travel new unknown routes Less than Once or twice 3 5 times a Bi monthly Monthly Weekly Daily once a year a year year 8 How often do you need to plan your journey in advance Less than Once or 3 5 times a Bi monthly Mo
135. modal overview information both of the current environment and of how things were before On screen information should be X complemented with audio and or haptic feedback F7 Possible to select objects both by screen button interaction and by pointing to the X objects in real life F9 Allow filtering objects based on distance year and type X F11 Possible to design and experience both on location and from a distance X NF2 Possible to use both on your own and together X NF4 Possible to plan in advance X The body of the Time Machine report quite nicely lays out what requirements were satisfied and how this was done A theme of the evaluations was the need to allow further customization of both the interface Multimodal guiding signals should be confirmative rather than repellent and the functionality Provide distance information according to the precision of the system which in this case should not be more detailed than multiples of 10 m 4 1 4 The terrain navigator The terrain navigator trials with seniors in national parks provided a realistic test of the proposed functionalities of the system Most of the functional and non functional requirements were met with the following exceptions Table 4 1 4 Terrain Navigator requirements X designates partial fulfilment not fulfilled Description NFC003 The system forces the user to communicate with other use
136. mp find 02 13 01 00 02 00 02 13 00 35 00 43 02 57 00 41 02 56 00 49 00 47 01 35 00 52 03 05 01 34 02 53 02 34 00 54 02 56 02 18 03 22 03 37 00 53 01 57 01 36 02 41 03 51 01 17 02 05 00 45 00 40 00 53 01 05 02 01 04 24 05 36 00 55 01 19 02 01 02 24 00 30 00 35 01 52 01 18 00 40 01 12 01 08 02 29 02 12 00 43 01 43 01 30 01 38 01 36 02 43 01 51 02 00 01 08 00 21 02 10 00 57 03 11 02 11 01 53 00 53 00 57 01 29 01 20 02 55 00 40 01 12 00 55 00 59 01 04 0 55 Avg 02 40 01 49 02 11 01 22 00 56 01 40 01 24 24 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 15 40 1 02 de gt PET 2 Requested 10 29 D Bearing BA Jochen Click to accept or decline a8 Friends 2223 Ihahn r 2 99 km 1 9 hrs ago A F Tim Android 7 m 0 mins ago IETT La Uli Mann 18 m 7 0 hrs ago Deactivated 1 Chris1 Deactivated for 21 6 hrs Distance 75 4 km Accuracy 65 m Places Settings Vibration Profile l Change your profile here Figure 3 1 2 Screen shot of bearing module and friend list in Joined All participants were of the opinion that they had reached their goal Overall they marked their experience of the user interface UI marking in the table below comprehension of the app functionality app functionality and their general experience of the complete app as high All users stated that they would highly recommend the app to their friends since pre
137. music than the app but the app was still being used over 200 times per hour At 22 00 there was another burst of activity with use increasing to nearly 400 times per hour as the main part of the festival concluded However the app continued to be used until 04 00 as late sessions at the festival continued until that time So the pattern of use of the app corresponds closely with the ebb and flow of the festival 3 5 2 3 Use of the guiding functions The Juicy Beats app provided functions for showing the direction of a particular friend or one of the stages within the festival the bearing and guiding functions The bearing and distance information is shown visually on the screen see Figure 3 and this is combined with either sound the Geiger function or vibration Unfortunately the logging data does not discriminate between seeking the direction of a friend or a stage These functions were used 554 times so 11 096 of all use sequences of the Juicy Beats app included some guiding functions Figure 3 4 2 2 shows the distribution of the use of the functions during the festival Figure 3 5 2 3 2 shows the use of the guiding functions in comparison with the overall use of the app and we can see that the pattern of use follows closely with the overall pattern of use 82 6 1 THE NOTWIST GISBERT ZU KNYPHAUSEN LACHERGA TRA QUANTIC amp HIS COMBO BARBARO GLOBAL PLAYER RESIDENTS SCHLACHTHOF BRONX m er E MER LL EE laa eat M
138. n t able to fulfil all tasks The participants subjective rating on performance is generally high 4 7 with the same outlier participant 2 as for the temporal demand In general participants felt they needed to put some but not an excessive amount of effort into the use of the app 3 5 Two participants rate the frustration higher than middle participant 2 with technical problems and participant 10 who was using a wheelchair and needed both hands to use it while simultaneously using the touch screen to choose scanning distance 62 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications For the three different concentration figures Figure 3 3 5 10 2 it seems that the guiding with vibration is the least demanding task in the city This is in line with the qualitative answers that users gave see above 3 3 5 10 Participant s subjective task load ratings rural In the rural setting the NASA RTLX was also used but since the sub tasks differed the rural setting is reported separately In the full NASA RTLX chart for the rural tests Figure 3 3 5 10 3 Figure 3 3 5 10 1 the perceived mental demand on doing the tasks and using the app varies between 2 and 5 The perceived physical demand is rated quite high 3 7 The temporal demand was rated quite low 1 3 by all The participants subjective rating on performance is generally high 5 7 In general participants felt they needed to put some but not
139. n at the different points seems not to be the major goal just running to catch the next At the pre final test no spoken information was present so this might have changed the behaviour somewhat Forks in the road pose a problem too if the angle between them is not sufficiently wide In this case it may be necessary to put a via point in a strategic place so users are sure not to miss it The via points themselves were seen as a necessary addition particularly when there was no visible trail at all The physical trail is scheduled to be cleared but at testing occasion 3 it wasn t causing some of the test persons ending up on the wrong side of a marsh An easy way to create trails is a high level requirement for the cultural historical museum organization During pre final testing the transfer of GPX files and other material as well as the app itself has worked OK However it is at the same time clear that the making of a multi modal trail for a mobile phone is a new media altogether and the design of it is not as straightforward as it may seem Should this be realized training and a dedicated editing tool should be necessary Furthermore the correct placement of GPX points was seen to be a non trivial matter Hobby handheld GPS devices can provide enough decimals at least 5 decimals when using degrees in the WGS84 system but you need to be aware of the error margins Car GPS systems need to be checked if they can provide the precision and
140. n could be used and the variety of menu options available was shown to the users After doing so they were asked to execute those operations on their own 3 2 5 1 2 Trials in realistic environments et t mom mimm om d Figure 305 1 2 Orientation phase scenario Orientation in open environments the route was laid out at Mayor Square in Madrid The features of this square are it is a large open square with four lampposts one at each of the four corners with arches all around the square and with an equestrian statue in the centre We identified the following inconveniencies commercial vehicles delivering and loading until 11 00am and large amounts of pedestrians due to the touristic nature of the area and to the fact that the main Tourist Office is located there The geographical coordinates are illustrated in Appendix l 30 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The GPS signal was used to verify the support given by both the device and the application to the users in terms of their orientation The orientation tests were carried out by using three out of the four modes incorporated into the application 1 Through vibration patterns providing information about turns e Increase Decrease Left e Decrease Increase Right e Decrease Decrease Turn 180 e Nothing go straight 2 Through stereo sounds providing information about turns and distances We used bone condu
141. n the application was not filled by any of the evaluators When they were asked about this fact they answered that they did not believe that it was necessary to modify anything They pointed out that all the profiles should be configurable in order to add remove items depending on the requirements 3 2 9 3 CONCLUSIONS AND RECOMMENDATIONS It has to be emphasized the attention paid and the enthusiasm shown by the MTs during the whole session After the presentation they were willing to use some of the functionalities of the application by their own In general terms the application was considered useful and adapted to the needs of the MTs as well as intuitive and easy to use No item in the questionnaire got its maximum score This fact has been interpreted as positive prudence as everything can be improved MTs manifested that although TrainerTools and NavEscort are thought to be used for outdoors orientation and navigation the same concepts could be applied to indoors where there is a lack of navigation systems In all these systems when providing navigation instructions to the user it is important to give them regarding the user position make a ninety degrees turn to YOUR left instead of make a ninety degrees turn to left They believed that it could be useful to have the same application for mobile platforms Thus they could edit and configure the routes not only in their desktop but in the street where they could make verifi
142. nations Vibration patterns are to be used to aid users in navigating to Partially not previously selected waypoints or locations of friends and to guide safe without them safely on the festival grounds help This demonstrates that although the basic functionality worked well additional work is need to be done to complerte making the application comfortably useable by persons with visual disabilities An interesting note was that the directions and location were all given as the crow flies not taking into account the physical layout of the event However this has been worked out in the NavEscort application route designer tool so the toolkit could provide the missing functionality 4 1 7 Toolkit use evaluations The evaluation of the elements of the toolkit by programmers did not specifically address a set of claims or requirements so as a result the trials were more of usability and API documentation The two trails were judged successful however the were some difficult spots that came up several times There was difficulty with the installation instructions and the coherency of the API with various Android releases Similarly there was a previewed lack of documentation both generally and in the sense of discovery of HCI modules This will be a big challenge as the Haptimap project ends with the too often typical result of orphaning software Fortunately the project management has been aware of this from the beginning and steps ar
143. nd iOS platforms and 1 student not providing information about which platforms 3 6 1 3 Procedure During the course the participants formed groups of 2 to 4 students designing and programming a vertical prototype of an Android mobile phone app with the purpose of experimenting with multimodal interaction and the various sensors in the phone They were also asked to participate in a course competition meaning that if they used the HaptiMap toolkit in their project apps they could win a prize 92 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The available documentation at the time of the course late 2011 was not the final one and in order to make the step in to the toolkit easier an additional to the ones already in the toolkit code example project was written and a tutorial made to show the use of the GPS sensor and the haptic guide Additionally a step by step instruction on installing was created The students were encouraged by their supervisors to install the toolkit and run the tutorial but it was not a compulsory part of the course 3 6 1 4 Results As noted in the Procedure using the HaptiMap Toolkit was not compulsory for the course Fourteen 46 7 of students on the course reported that they did use the Toolkit as part of their project work for the course A further eight 26 7 reported that they looked at the Toolkit but decided not to use it in their project The remaining 8 26
144. nd the way in which you usually design a route SIMILARITIES DIFFERENCES 4 Please rate the level of ease difficulty in handling with the Easy Difficult Very application 5 Rate in terms of flexibility and usefulness the possibility of editing routes introducing mandatory crossing points rate it from 1 to 4 where 1 is not useful 6 Automatically generated instructions on the route given by the application are considered Incomplete Enough Too complex Generic Others 7 Automatically generated routes by the application are A user Multiple users valid only for a user or can be applied to multiple users 8 The fragments of an automatically generated route are A user Multiple users users valid only for a user or can be applied to multiple 9 When you explain a route to the user which kind of information he she provide to him her Distance the route has a length of 500 meters Approximate time to cover it it takes about five minutes to cover it Shape of the route en forma de U L etc Streets included in the route Incidences in the route Number of crossings traffic directions obstacles which do not meet the standard All the previous ones Others which ones 140 difficult HaptiMap Deliverable D1 5 PU User tests of the final demonstrator application
145. ndition they would comment on the weirdness of being led through houses However even when routing is on this can of course occur due to the inaccuracies in GPS positioning and also occurred at times Both routing condition allowed users to cross over open areas such as squares For a visually impaired user crossing a square is unusual To orient themselves adequately they usually follow the outline of the square and it a regular GPS device is used it will route along the perimeter of the square One user particularly commented on it J was able to walk across the square It was great In the forest via points were used to make an artificial routing Since in the rural setting no detailed maps were present and part of the planned route not even yet fully prepared there was no way to use automatic routing Special route points via points were put in places between POIs to make the route following easier These via points were seen to be important however several of them did not work as expected Because of GPS inaccuracies the radius of via points was also 10 m as the POIs see below This means in effect that you arrive at a point before you actually reach it and if the GPS is a couple of meters off you can arrive at it 20 30 meters before you reach the intended point that shows you the particular fork in the track for example It was observed that it would require more time training or instruction to use the app as on
146. ne of the competition developers answered the questionnaire the other four participants were interviewed either by telephone or by Skype 3 6 3 2 3 Procedure For the interviews participants were thanked for their participation in the study and assured of the confidentiality of the information to be provided The interviewer took the participant through the questions see Annex M but allowed the discussion to flow reasonably freely At the end of the session the interviewer took a minute to review the questions and make sure that all had been answered before again thanking the participant for their time For the questionnaires Participants were sent the questionnaire via email The questionnaire and accompanying email contained assurances about the confidentiality and anonymity of the information provided see Annex M 3 6 3 3 Results 3 6 3 3 1 Downloading the HaptiMap Toolkit Participants were asked about their experience of downloading the HaptiMap Toolkit All 7 participants found it easy and clear to download and no one experienced any particular problems In particular one participant commented I ve had quite a few bad experiences with projects in Google Code and Eclipse this one was flawless P6 Another commented that usually you have to put effort into Eclipse but that the download was excellent for this toolkit P4 One participant commented that it was not that easy to find the location of the source
147. ng the toolkit develop a novel application of their own design Participants used the Android operating system and we gave each a Samsung Galaxy SIII device to develop and test with We time limited participation to 3 hours incorporating a 30 minute debrief session at the end of the evaluation Participants were allowed to work together but were asked to note instances where they received help in an incident diary Each participant was paid 40 for participation 3 6 2 3 Results We break the key findings of the evaluation into three parts each related to one section of the worksheet and some more general results Installing the Toolkit and Running the Demo All bar one participant found installing the toolkit and running the haptic compass demo application to be straightforward The getting started instructions on the webpage were found to be straightforward and easy to get setup with The only issue that participants did have was that some of the files in the sample Android project were not updated for the latest version of Android that the students had on their machines In some cases this required an update of the Android SDK on their machines or simple fixing e g the removal of G override statements and upgrading project definitions These were of a minor nature and were easily overcome by an experienced developer The fourth participant failed to read the instructions properly and started to follow the install txt file in the toolkit d
148. nthly Weekly Daily once a year twice a year year 9 Please provide some details on your normal planning tools strategy 10 How would you rate your sense of direction please circle the numbers Very Poor 1 2 3 4 5 6 7 Very good Online Map Services on desktop computer 1 In an average week how often do you access online map services 0 1 3 4 6 7 10 more than 10 2 If you use online map services please tell us which 115 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 Please also tell us how and why you use online maps F Mobile Map Services 1 Have you ever used a GPS navigation device or application Yes No 2 If yes how often do you use it 3 If yes for what purpose car geocaching biking In a typical week how often do you use 1 Mobile maps 0 1 3 4 6 7 10 more than 10 2 GPS handheld 0O 1 3 4 6 7 10 more than 10 3 GPSs in your car 0 1 3 4 6 7 10 more than 10 4 If you use mobile map services please tell us which 5 If you use a GPS please tell us which 6 Please also tell us how and why you use mobile maps and or GPS 116 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix C HaptiMap common background questionnaire mandatory version Participant Identifier Experiment Experimenter Date Please complete the questions below as fully and accuratel
149. o descriptions that were played automatically at certain locations The participants wished there would have been more information about the trails their current condition and their difficulty in general The type of paths varied quite a lot on the two kilometre test walk from very easy to rather difficult Figure 3 4 6 1 We asked the participants how much they needed to concentrate on using the Terrain Navigator and how it affected their experience On the scale from one to seven the average score was 2 1 According to the participants Terrain Navigator made the hiking experience more interesting and carefree as they did not have to concentrate on the map because the audio functionality assured them they were on the right track 3 4 6 2 The Simple Maps The Simple maps have been developed especially to meet the needs of elderly users For example presbyopia can be taken into account in map design by increasing the contrast between objects and by emphasizing the important features more than traditional cartography does Kovanen et al 2012 The participants studied the Simple maps on an iPad Figure X 4 The thought use case was a planning of the hiking route while sitting and resting in a cottage or by fireplace outdoors We asked them questions regarding the accessibility features of the maps the usefulness of the colour schemes the clarity of text elements and so forth 74 HaptiMap Deliverable D1 5 PU User tests of the final demonst
150. on Additionally some events were logged for all users during the time of the festival and it was possible to make a quantitative evaluation based on the aggregated data 1 3 6 Developer evaluations Evaluations of the usability and usefulness of the HaptiMap Toolkit were undertaken with 26 developers Six were students learning mobile software development at UGLAS who undertook an exercise with the Toolkit Fourteen were students learning advanced interaction design at ULUND who used the Toolkit as part of their project work Six were professional developers who used the Toolkit for their own work four developers or as part of a submission for the HaptiMap Developers Competition two developers A combination of interviews and questionnaires was used to elicit qualitative and quantitative information about their experiences with the Toolkit 18 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 2 Test procedure 2 1 Description of the tests This section describes the protocol of the basic tests applied to all demonstrators The sections are of three types Demographic mandatory and optional The optional tests are applicable to some of the demonstrators but not critical to the protocol By providing a standard set of tests and data to be collected the demonstrators can be evaluated both in reference to their specifications and between systems This iteration of evaluation was focused on providing
151. ons 06 15812 Added text 0 7 Streamlining procedure descriptions and appendices and Sec 2 to contain the necessary parts 08 IS5 812 Adding claims in 3 5 1 0 16 8 12 IE SANI Merging documents 1 1 16 8 16 812 16 812 3 1 Adding Joined evaluation 3 1 1 2 16 8 12 3 6 1 3 Adding student developer evaluation edited the end user total table 13 16 8 12 Adding missing text and images 14 Corrected table 1 3 1 1 5 Accepted changes and added draft of section 4 1 6 Shrunk section 3 2 17 More shrink 18 Wrote section 4 19 22 8 12 Section 3 1 Typos corrected Figure and Table captions inserted 20 24 8 12 Checking editing for internal review 21 30 8 12 AN Adding last missing parts 22 2 0 12 12andl 2 and 1 12andl Added missing text Sections 0 amp 23 3 9 12 4 table 1 3 Exec summary amp table 1 3 and section 4 24 5 9 12 Pan Revisions as per internal review 25 10 92 10 92 Author list and executive summary edit 3 5 2 and 26 10 9 12 Section edit 27 25 9 12 3 5 2 amp 3 5 7 Added last evaluation 28 2 10 12 3 5 7 Very last bits Copyright 2012 HaptiMap Consortium Copying is permitted provided that copies are not made or distributed for direct commercial advantage and credit for the source is given HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table of Contents Listo EISHPOS gesto iaa Di mirada aladves Mickael neri eleg Qr mein BRE
152. ortant to the HaptiMap developers and the competition developers in particular To meet these requirements the following methods were used a Participants in the freelance developers group were asked to download the HaptiMap Toolkit look at the documentation as they needed to and to undertake a task to submit to the evaluator They could either undertake a task in one of their own apps or a task set by the evaluator In fact the evaluator had not interest in looking at the implementations but this was to ensure that the participants really did use the HaptiMap Toolkit Participants in the other two groups had already used the Toolkit so this stage was not needed b A set of questions was prepared as an interview schedule or as a questionnaire It was thought that it would be most time efficient for the competition developers and the freelance developers to undertake an interview by telephone or Skype whereas for the HaptiMap developers a questionnaire provided more anonymity However one of the competition developers asked whether he could answer questions via email as that would be more convenient for him so he was also sent a questionnaire 3 6 3 2 Method 3 6 3 2 1 Participants Seven professional Android developers took part in the evaluation 2 were developers who had entered the HaptiMap competition 3 were freelance developers recruited through personal contacts of the evaluators and 2 were developers within the HaptiMap proje
153. otification of a quiz question for such a notification and didn t understand why it only played at about half of the points there were only quiz questions to some of the points The quiz was appreciated particularly by one of the users but unfortunately it worked only partly The HUI did not work when questions or answers were too long and particularly not when the screen was small Three participants used SonyEricsson Xperia Active phones one used a SonyEricsson Xperia Arc and one a SonyEricsson Xperia Neo In the rural setting you had to either press a button on screen or shake the phone to leave the point and start the guiding to the next one Two users used the on screen button while three users used the shake function One of the users particularly commented on that the shake function was the easiest part of the app 3 3 5 8 Sound window results The sound windows were active at all test occasions even if the app has a check box to turn these off Three of the users reported hearing problems but only one of them asked for the sound windows to be removed when asked about improvements No other participants reported that they in general were annoyed with the sounds Some improvements were asked for in particular in the city where sounds illustrated different activities that had been going on at a particular place in historical times Since the sounds were separate from the POIs there was no natural connection between the sounds you h
154. ownload folder rather than the getting started tutorial as instructed This caused a significant number of issues Notably recompiling the toolkit requires a number of additional tools to be downloaded including the Android NDK Cygwin and other compilation tools However more notable was that the instructions provided with the toolkit often did not reflect the configuration process For example asking the user to edit a properties file that did not exist or look in a subfolder that did not exist Coupled with the sparse instructions for this approach the user spent 40 minutes achieving nothing and becoming frustrated However once the experimenter intervened and pointed out the correct 95 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications instructions on the worksheet he made speedy progress and quickly caught up with the others It is therefore key that developers are directed towards the quick start guide and away from unnecessary compilation of the toolkit Completing the Tutorial As with installing the toolkit participants found the tutorial to be straightforward to complete They found that it provided a good understanding of the toolkit capabilities and how to get started The largest problem was again that some of the instructions referred to standard parts of the Android toolkit that had been subsequently redesigned For example the dialog boxes when creating a new project This is partly due to th
155. pant All users would reuse the application and recommend it to others They also suggested the use in different contexts namely any large open area that is crowded like festivals beaches fairs exhibitions and sports events 3 5 1 3 3 Requirements analysis 78 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The specific user requirements for the Juicy Beats demonstrator from D5 1 p 54 55 are described in Tables 3 5 4 and 3 5 5 The tables also show the requirements that have been implemented in the final version of the Juicy Beats demonstrator Table 3 5 1 3 3 1 Event Guide JUICY BEATS functional Requirements Reached er Requirement 9s IUE implemented Text to speech output for content e g information on artists and Yes information services e g Twitter the Juicy Beats tweet provides implemented current festival news from the event organisers is to be available fully Users need appropriate means for obtaining an overview of what Partially stages and service points are nearby with respect to their current location on the festival map only visual Park entrances the different stages toilets and other points of Yes can be interest are to be locatable for visually impaired people using an guided to them interactive map with tactile feedback or voice over Visually impaired users can select an item as a waypoint for navigation by double tapping on items located on the
156. pants were using a cane it was held in the dominant hand This forced the visually impaired participants to hold and interact with the phone in the non dominant hand being somewhat awkward but still possible The thumb was mostly used for interaction with the screen but also while scanning standing still two hands could be used One user was afraid that the mobile phone would be stolen since she was pointing with it and holding it quite lightly The user s had several suggestions for improvement They requested for example e Regular information about distance to target e Information about what was on the points of interest in present time not just the historical information e A where am I function e Navigation in speech also to notify of turns ahead for example e A function that guides the user back to the starting point to be invoked at any time e Information about the sound windows They captured the user s interest yet there was no additional information about them 58 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 3 5 2 High level results rural Already the pre final tests showed that the trail following module is easy to use after a short introduction The young users 7 and 9 years old had in general no problems to follow the trail However there are some considerations Young children like to run which in some cases result in missing a point while running past it The informatio
157. per than the decrease in the use of vibration For example the use of sound only drops by half from 10 6 to 4 8 of instances whereas the use of vibration only drops about 3096 from 12 796 to 8 596 at the same point see Table 3 for percentages of use So it could well be that with the loud music from the festival users cannot hear the sound signal and use the vibration more to use the guiding 87 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications e v a 9 gt SH GPG GGG GY GY GC Y GS YS Sm HH FSF HH HF FH HF SF FF LS SPP AP SP LOT BT I AB gy PS GY MP A Time Figure 3 5 2 3 4 Use of guiding only sequences with different combinations of sound and vibration during the festival Uses per hour S DBO DH Q9 Q GB GH HGF GF gs geo LHP HDB HP GG PY IH HIS IS FY Y HG FG J NOEN EN DT LOT ANT ABAD QS gy AS d GY Y Time Figure 3 5 2 3 5 Use of guiding included sequences with different combinations of sound and vibration during the festival 88 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications gt WwW un Ww e N ui IA uo S2 z 7 v N 9 SOD LHL HP HM A Sar Gy gr Figure 3 5 2 3 5 Use of both guiding included and guiding only sequences with different combinations of sound and vibration during the festival Table 3 5 2 3 3 Percentage use of both guiding included and guiding only s
158. pplications which are mainstream applications that are useable for a wider audience as well as design for me special needs applications different types of demonstrators have been developed Table 1 3 End user participants in demonstrators With partial visual With Demonstrator Total M F Sighted impairment blindness Age span Joined 16 NavEscort 13 Time Machine 23 Terrain Navigator 16 Juicy Beats 7 Juicy Beats statistics 05 300 15 Developer eval 27 n a Total of users 102 117 57 43 56 All 8 report some kind of residual vision but the quality of that vision ranges from experiencing light and having very limited residual peripheral vision to tunnel vision where the user could see the screen but not the surroundings Because these were based on statistical analysis rather than interviews we counted each participant in this section as 1 20 of a full participant 16 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 3 1 Joined GeoMobile Joined an iteration of Juicy Beats friend finder API developed by the Geomobile GmbH has been evaluated by Siemens with ten blind users one severely visually impaired user and five elderly persons The participants with visual impairments tested the application behind the Heinz Nixdorf Institute and the neighbouring Heinz Nixdorf Museums Forum for safety reasons The elderly people tested the application in the city center
159. quent travellers see figure 3 1 2 O amp M for people with visual impairments means the training in the facilitation of the white cane in wayfinding different techniques and orientation in familiar and unfamiliar environments as well as learning frequently used routes e g to work supermarket train station etc The first task was comprised out of two parts 23 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 downloading and installing the app 2 open the app and register The first part did not cause any problems at all and the second part was by all users rated to be very easy and self explanatory Afterwards the users had five minutes to familiarise themselves with the application by exploring it without any further information by the responsible HaptiMap partner The remaining tasks consisted of 1 Inviting a friend 2 Add the current place and name it 3 Select a friend and get their bearing via vibration and sound 4 Send a message Hello to a friend 5 Send the place added in 2 to a friend 6 Receive a place add it to the map and pile it via vibration and sound None of the tasks above caused any major problems for the participants and all were completed in a satisfying timeframe see table 3 1 3 Table 3 1 3 Results of time to accomplish tasks in minutes register invite friend add place Find friend send msg send place get place a
160. r contexts 120 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix E Laboratory based testing SUS scale If you have not already done so run the lab test suite 30 min 2 4 participants should be enough This is in two stages task completion driven and self reporting Optional Task completion Each prototype should be tested if possible with a user attempting to perform a task while doing the think aloud protocol The qualities of the task should be 1 It makes them access most if not all user interface parts of the application 2 Itforces them to make connections between engaging one part of the interface an then another An example could be replay the last instruction and then redraw the map and rotated he smart phone till its perspective matches what you see 3 If you must make multiple tasks to exercises the various interface details you could make a story linking them together 4 The whole of the task completions tests should not take more than 15 min Tape what they are saying and take notes to fill in details As soon as you can put the notes and tape parts together in a rough first draft analysis of the session The necessary mobile missing parts to doing a task can be simulated for this test using a wizard of Oz approach Self Reporting Have the participant fill out the System Usability Scale SUS usability test this should not take more than 15 min The SUS is just below
161. rator applications Figure 3 4 6 2 The participants answered questions regarding the five different maps Out of the five different maps the majority of the elderly participants said they would like to use either Simple map 1 or Simple map 4 These two maps were the ones where the text was the most distinguishable according to the elderly participants The younger participants would rather use Simple map 3 or the traditional map map number 5 that has not been simplified The Simple map 1 was considered to be the map that is the easiest to read by the elderly participants while the younger participants thought that the map that is the easiest to read is either Simple map 1 or Simple map 3 All the participants agreed that the accessibility of the paths was the easiest too perceive in the Simple maps 4 as the slopes on the paths were marked according to their steepness We asked the participants how important it is to simplify the traditional maps and on the scale from one to five the elderly participants gave an average score of 4 0 while the younger participants gave an average of 3 1 3 4 7 Conclusions The Terrain Navigator increased the feeling of safety of the elderly hikers and ensured they were on the right track while hiking in the National Park The application was thought to make hiking a more carefree occasion and encouraged the users to visit new places The audio functionality of the Terrain Navigator proved to be both attractive
162. rbeskyddande D lig kvalitet Kraftfull F ruts gbar Professionell Relevant User tests of the final demonstrator applications Engaging Entertaining Exciting Expected Familiar Fast Faulty Flexible Fresh Friendly Frustrating Fun Hard to Use High quality Illogical Impressive Inadequate Incomprehensible Inconsistent Ineffective Innovative Insecure Intimidating Intuitive Irrelevant Meaningful Misleading Motivating New Non standard Obscure Old Ordinary Organised Overwhelming Patronising Poor quality Powerful Predictable Professional Relevant 135 HaptiMap Deliverable D1 5 PU Tillforlitlig Lyh rd Rigid Tillfredsstallande Saker Enkel F renklad L ngsam Sofistikerad Stabil Stimulerande Okomplicerad Stressande System orienterad Tids dande Tids besparande F r teknisk Palitlig Oattraktiv Okonventionell Forstaelig Of ruts gbar Of r dlad Anv ndbar Praktisk Vag Reliable Responsive Rigid Satisfying Secure Simple Simplistic Slow Sopbhisticated Stable Stimulating Straightforward Stressful System oriented Time consuming Time saving Too technical Trustworthy Unattractive Unconventional Understandable Unpredictable Unrefined Usable Useful Vague 136 User tests of the final demonstrator applications HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix J Haptimap Developer Evaluation Instruction Sheet In thi
163. re trails Trail following is based on pointing and scanning with a mobile device and receiving feedback with sound and or vibrations Sound windows playing localized historical sounds are used to help global navigation and get a feeling for the history See figure 1 2 3 Additionally the Time Machine app allows for free exploration of the surrounding points of interest also loaded through a sharable GPX file It is then possible to choose one point of interest and be guided to it with the same modalities that the trail following provides Additionally a haptic map is implemented which makes it possible to explore the map by touch feeling vibrations on streets and your own position and hearing the street names read out loud Next point Krognoshuset The Krognos House Figure 1 2 3 Time Machine User Interface Screens Splash screen trail point screen and map screen 14 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 2 4 Terrain Navigator The Terrain Navigator operates on iPhone and aims to demonstrate that combining different modalities for route guiding and well designed maps can improve accessibility in the hiking context The demonstrator also presents how modules of the HaptiMap toolkit can be incorporated into existing mobile map applications See figure 1 2 4 Figure 1 2 4 Terrain Navigator sample screen shots 1 2 5 Juicy Beats The Juicy Beats event guide demonstrator is an
164. res for individual items are not meaningful on their own To calculate the SUS score first sum the score contributions from each item Each item s score contribution will range from 0 to 4 For items 1 3 5 7 and 9 the score contribution is the scale position minus 1 For items 2 4 6 8 and 10 the contribution is 5 minus the scale position Multiply the sum of the scores by 2 5 to obtain the overall value of SU SUS scores have a range of 0 to 100 You can get more information about using the SUS here http www measuringusability com sus php 122 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix F NASA TLX task load test with printable and localized versions Definition of Task Demand Factor Mental How much mental and perceptual activity was required e g thinking deciding demand calculating remembering looking searching etc Was the task easy or demanding simple or complex exacting or forgiving Physical How much physical activity was required e g pushing pulling turning demand controlling activating etc Was the task easy or demanding slow or brisk slack or strenuous restful or laborious Temporal How much time pressure did you feel due to the rate or pace at which the tasks or demand task elements occurred Was the pace slow and leisurely or rapid and frantic How successful do you think you were in accomplishing the goals of the task set by Performance
165. rns it off and ends with the sound and visual configuration But in over a quarter of sequences the user turns on the vibration at some point and keeps it on to the end of the sequence Table 3 5 2 3 1 Breakdown of guiding only sequences into those ending with sound and those ending with vibration Sequence type Number Percentage Sound Default Ends with sound vibration never activated 197 53 596 Sound Optional Ends with sound vibration having been 57 15 596 activated Vibration Ends with vibration 103 27 996 86 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications We also looked at the other use sequences that included the guiding functions to see which of these patterns they followed The results are presented in Table 2 below A very similar pattern of sequences is found in this case Table 3 5 2 3 2 Breakdown of guiding included sequences into those ending with sound and those ending with vibration Sequence type Number Percentage Sound Default Ends with sound vibration never 9 51 496 activated Sound Optional Ends with sound vibration having been 30 16 996 activated Vibration Ends with vibration 56 31 696 We also investigated at how these different sequences varied during the festival Numbers of sequences per hour are much smaller than in the previous analyses being in the 10 s rather than the 100 s so the results need to be treated wi
166. rotate the wheel chair to scan around it wasn t possible to put a finger on the screen Thus it seems that several different ways of actively selecting points and also using the scanning without constant screen contact is needed The pointing around oneself with the phone is relatively intuitive although unusual for the users It was observed at times that participants didn t turn all 360 degrees and also that they tried to move the finger on the screen sideways to scan This was also seen when points were close to each other The distance filtering was well received but still the points of interest 59 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications sometimes occluded each other This led to problems finding a specified point or finding it again when you lost it Additional filtering is needed and 2 users suggested list search and one user regular keyboard search Since it seems that users intuitively move the finger on the screen for scanning a combination of point scanning and screen scanning seems like an interesting approach to pursue in the future Rotating imposes a problem for users with visual impairment The turning forces them to give up their reference direction There are some solutions to the problem one being to add a compass with cardinal directions which was suggested by at least one user during the interview Another solution could be to add personalized reference points in the database o
167. rs FC056 The system deliver route guiding notes in vibrations FC030 The user can define a route to be stored in the system in advance FC019 The system calculates different route path alternatives and suggests them to the user FC016 The system can calculate an optimal route and guide the user trough the route FC037 The system notifies on inaccessible routes 105 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Some of these are more or less trivial NCF003 some of them would require deeper changes in the system FC019 As the tests show the applications as tested was sufficient to provide a satisfying user experience 4 1 5 Juicy beats This evaluation was quite similar to Joined except it was performed in a large music festival environment The large masses of moving people and structures not being laid out in a grid like in city s placed additional stress on the user with visual disabilities The specifics of missed requirements are Table 3 5 4 Event Guide JUICY BEATS functional Requirements Reached User Requirement q implemented Users need appropriate means for obtaining an overview of what Partially stages and service points are nearby with respect to their current only visual location on the festival map A compass mode for navigation is to be used replacing a map Yes but no and turn by turn instructions for guiding visitors to their routing desti
168. s APPENDIX M Questions used in professional developer evaluation Interview question schedule This interview is completely confidential information will only be passed to the HaptiMap team in an anomymous way grouped across a number of respondents 1 DOWNLOAD How easy was it to download the HaptiMap Toolkit Was it easy to find the download page on the HaptiMap website Was the process clear and as expected Any particular issues to raise 2 DOCUMENTATION Did you use the Toolkit tutorial Was it useful Was it clear Any particular issues to raise Did you use other documentation about the Toolkit If so what was it Was it useful Was it clear Any particular issues to raise 3 WHAT WAS THE TOOLKIT USED FOR What did you use the toolkit for How successful were you in using it did you achieve your goal 4 EASE OF USE What it easy to use On a scale from 1 5 where 1 is very difficult and 5 is very easy how would you score the toolkit Any particular issues to raise with using it 5 WAS IT WORTH THE EFFORT 141 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Considering the effort you needed to invest in downloading the toolkit understanding it etc was the return worth that effort 6 FUTURE USE Would you use the toolkit again in the future On a scale from 1 5 where 1 is not at all likely and 5 is very likely how likely is it that you would use the too
169. s evaluation you will be assessing the Haptimap toolkit by developing simple Android applications that use it We are interested primarily on issues in both the documentation code quality and usability of the toolkit when building navigation apps I will get you to complete three distinct tasks in the evaluation setting up the toolkit working through a tutorial and developing a novel application The maximum time taken on these tasks will be 2 5hrs At the end of this time or on task completion we will have an interview to discuss the positive and negative aspects of the toolkit and documentation To assist you in this you have been provided with an incident diary Whenever you hit a particularly large problem or find that the toolkit has made a problem that you though would be complex easy then write a short description of the task you were trying to accomplish and how the toolkit aided or not You may also use the help page on the haptimap website for assistance http haptimap org toolkit help html You may work with others on the tasks and help each other as necessary However please write down any help you received and why it was necessary in the incident diary Please also take screenshots of the finished applications On the Samsung Galaxy S3 you can take screenshots by holding down the power and home buttons at the same time Task 1 Installing the toolkit Go to http www haptimap org downloads html Download the toolkit and install
170. s not Regarding NF1 the quantitative evaluation should address it because a short introduction was given at the beginning of the qualitative evaluation Requirement NF2 Three visually impaired users and three sighted users successfully used the Juicy Beats application during this evaluation It is to be noted that all functionality except the overview on a visual map was accessible to users with visual impairment Requirement NF3 The system was stable in its functionality but the GPS and internet connection could be lost leading to some confusion during use 80 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Regarding NF4 and NF6 the qualitative evaluation showed that the users could attend the festival normally listen to the music navigate the festival area and talk to friends all while using Juicy Beats Requirement NF5 In general the users did not complain about feedback that was hard to understand in the application Most of the negative comments were about missing features and positive comments were made about the existing functionality of the application However the feedback from the application when encountering GPS problems seemed insufficient Participants explained in the interview that the GPS would turn off without giving a notification leading to disorientation of the users Conclusion The Juicy Beats application was successfully used by both users who were sighted and
171. s well as in the road network see also D5 3 The other half of the participants were given a version with directions given as the crow flies After the learning training session the participants explored freely and were asked to chose and walk to at least 2 different points of interest in the city The final point after the 2 freely chosen ones was the place for re union for the debriefing and interview The interview session started with the NASA RTLX and then followed the HaptiMap demographic Basic questionnaire a semi structured interview plan and a word cloud exercise http www userfocus co uk articles satisfaction html to capture the subjective impressions of the application both positive and negative The king s house 210m The University Historical Museum 295 m 100 300 m gt 300 m Figure 3 3 4 1 Scanning concept in the explore mode The test procedure took approximately 1 hour for the introduction and actual test and between 1 and 1 5 hours for the interview The relatively short testing and training period 2 5 hours maximum was chosen for two reasons Although there were new forms of interaction we were hoping that they would be intuitive enough to not need too much training And the other reason is of course that participants may be busy with jobs or other activities end we didn t want to book them for too long sessions 56 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator
172. sable to carry out a pre test or training test so that they can use the device perfectly understanding above all its purpose and its functionality This is apparent in two situations a As it can be appreciated from their answers some of the comments were not directly related to the question This fact shows that probably they had not fully understood the tested functionality to evaluate the usability and reliability of the vibration patterns as an orientation aid A certain degree of misunderstanding regarding the objective and carrying out of the tests was also reflected on the development of such tasks b Analysing the trajectories executed by the users when going from A to B they followed the route correctly in general but they changed direction suddenly as if they had not understood the vibration pattern properly and had therefore misinterpreted it 46 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 2 9 Trainer tools 32 9 1 INTRODUCTION This report collects the contributions and opinions about the application TrainerTools of experts in orientation and mobility techniques for visually impaired users also known as Mobility Trainers MT i L JI Lae I a 1 P Saw Figure 3 2 9 1 1 TrainerTools screenshot Three experts participated in the experience Such a small number of technicians to evaluate the application respond to efficiency and economy of efforts reasons since all
173. several moments during which he could not hear the instructions He thought that an external amplifier with a regulator was needed to adjust the intensity of the sound 3 2 5 5 Test Analysis Note All test results are shown in the corresponding attached Annex J 3 2 5 5 1 Results Usability Scale Table 3 2 5 5 Usability scale results USERS E1 ES E2 ES DIRECT P 23 25 28 32 CONVERTED P 57 5 62 5 70 80 3 2 5 5 2 NASA TLX task load test 33 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 2 5 5 2 NASA TLX test results ITEMS Mental demand Physical demand Temporal demand Effort Performance Frustration level TOTAL 3 2 5 5 4 Semi structured Interview Table 3 2 5 5 4 Semi structured interviews Were you successful at completing the proposed route OBSERVATIONS Open environment Urban route Certain incompatibility between using a guide dog YES NO YES NO and using the device at the 4 0 1 3 same time Interview notes about trial Easy Safe to carry out Difficult Dangerous Mayor Square open environment The final part of the urban route when we were at Cebada Square Because I knew subconsciously that None of them has been dangerous One feels safe when one knows that there was no circulation of traffic there are people around you who are keeping an eye IMore difficult to
174. sk a lot and I wouldn t have been able to do it Choose the words from the following list that best Express your opinion about using the application system Annoyed 0 Disappointed 0 Trustful 3 Happy 1 Surprised 4 Interested 4 Hopeful 3 Bored 0 Curious 3 Alleviated 2 Safe 3 Enjoyable 2 Creative 0 ITEM Questions related to the NAVESCORT application regarding the executed route 34 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 Did you feel safe while executing the route YES 4 NO 2 How much influence do you think the GPS has had in carrying out the route From 1 6 7 6 7 to 7 1 being very little and 7 a lot 3 Do you think you could have executed the route successfully without YES 1 NO 3 the GPS 4 The amount of time you have used to execute the Bigger 4 Smaller route with the system would have been without it without it 5 The degree of safety you have experienced when Higher Lower without 4 executing the complete route with the system would without it it have been 6 The information provided by the system has been bud Sufficient Abundant Very abundant Useless Deficient Of vey little quality good 3 1 3 1 7 The help provided by the system has been vey Sufficient Abundant Very abundant Useless Deficient OF vey little quality good
175. ssibility of using all sensorial modalities of information sound vibration voice For instance three vibrations for traffic lights in red Negative vibration Then short for right and long for left Two short vibrations for a 180 turn Warning about obstacles and the possibility of using sound and vibration simultaneously Change the type of sound For example traffic lights sound for the left ear and musical sound for the right one You can say anything you want or make any comments It s better if it does not vibrate Different vibrations depending on what we have to do 39 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 2 7 Session 3 The application was tested with four users with different visual abilities two of them were blind and the other two were partially sighted and familiarization with technology two of them had never used other technology apart from the mobile phone All of them used a white cane as a mobility aid In this case tests focused on navigation in open environments without geographic references For this reason one of the questionnaires the one corresponding to the semi structured interview was edited to adapt it to the nature of the tests 3 2 7 1 Two Phases of the tests 3 2 7 1 1 Familiarization phase indoors i Figure 3 2 7 1 1 Familiarization phase The objective of this phase was to allow the users to get us
176. sts It has to be said that although the tests were carried out with vibration she constantly stopped at each step in order to confirm she was going in the right direction The sound option was also tried out without success This modality had to be given up after several minutes as the user was not able to distinguish the sound coming from the application via bone conduction from the background noise clown noises a child weaselling acrobats etc Figure 3 2 7 illustrates a comparison between the ideal routes the user SD2 had to accomplish in green and the paths finally carried out by her in red Some information has been also included about the modality used in the test and the time expended in completing the exercises The auditory modality also appears in the figure although it was unsuccessful N b N Modality Vibration Modality sound Modality Vibration Time 1142 Time 5 04 Time 8 45 Figure 3 2 6 2 4 Routes accomplished by user SD2 When the test sessions on the street finished the users went back to the building where the training had been carried out in order to complete a questionnaire It was on purpose that this questionnaire 38 HaptiMap Deliverable D1 5 would have few questions in order to avoid a rejection due to their tiredness or their lack of interest It PU contained the following questions 3 2 6 3 Test Analysis 3 2 6 3 1 Usability evaluation questionnaire Table 3 2
177. t to third place which of Vibration Stereo Sound Verbal the three modalities have you found One first One first instructions easier to use Three third Three second Two first One second One third 2 And the most useful Vibration Stereo Sound Verbal One first Two first instructions Three third One second One first One third Three seconds 44 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 Would you add something to the system In the stereo sound modality a continuous beep indicating that you have arrived at destination would be very useful There should be also a special pattern for vibration modality 7 Do you prefer to do the task with or WITHOUT O WITH 1 1 1 1 without the application APP APP 8 Choose 4 words from the list below that better express your opinion about the application system after having used it Annoyed 0 Surprised 1 Bored 0 Safe 4 Disappointed 0 Interested 4 Curious 2 Enjoyed 0 Confident 1 Hopeful 2 Alleviated 2 Creative 0 Happy 0 9 What level of concentration did you need to LITTLE A LOT use the system application 1 2 10 How do you think the use of this LITTLE ALOT application may influence your daily life 2 2 12 Rate your satisfaction with this experiment LITTLE A LOT 7 test 1 unsatisf
178. t ttn 45 3 2 9 Trainer to0 l81 5 treten retenir te tete arken dns dn est te or pet te rid Aer eta pe nna Pebble ve 47 3 3 Time Machine evaluations eee teret da ai aigen edana paana danaa ssk goaien iddaa tuta Rada en 51 ES NES MEI EE 51 3 3 2 Brief description of evaluation procedure eese tte tentent tentent 53 3 3 3 Participant demographic information eee eene ttn tnnt ttn rr rr nr nn nn ne enn 54 3 3 4 In Context evaluation procedure eese tentent tentent tente tttn tentia ntes tite ttnn nnn 55 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 3 9 RESUS certc noe ter nec tete rara lo Ga vaselin dM ep Ruben e tau aa AAKO Aa Dale La tate ve ARE Decide ta bres 58 3 3 6 Meeting the requirements discussion eee entente tentent tette ttn ttnn tente tentent 65 3 3 7 Conclusions and recommendations eese seen trente tatnen te ttnt tette 67 3 4 The Terrain Navigator e sssssssssesssassass nsnsnnnsssnannnnnannnnnnnan nanna nn tana nnnm snas tasa sa so RA so sa AN ARR KR RA RR KR RN RR RR KR Rn 69 3 4 B eoe URP aaraa aaiae aaiae Eaa ATAATA ea TaTTi ETN 69 3 4 2 Description of the Evaluation Process eene tnnt teniente ttn tent tenttnn tentent 71 3 4 3 Participant Demographic Information iwswswmsssssssssssssssssrsssesesrsssrsr rss sr rr rr rr tnnt tnnt tentent enn 72 3 4 4
179. t were raised about using the Toolkit were not extensive or severe and problems with compiling and getting the Toolkit to work with existing code may have been due to the students relative lack of experience with computing 94 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Overall this was a very successful use of the HaptiMap Toolkit in an educational setting 3 6 2 Students at University of Glasgow The purpose of the evaluation was to identify the usability and practicality of the Haptimap toolkit from the perspective of experienced developers that have no prior knowledge of the project 3 6 2 1 Participants The evaluation involved six graduate students all of whom were experienced in Java programming All had been programming for at least 2 years and had mid 2 on a 1 5 scale to very good experience 4 on a 1 5 scale as mobile developers None of the participants had previously seen or used the Haptimap toolkit 3 6 2 2 Procedure We asked participants to sign up for an account at Haptimap org in advance of the evaluation so that they could download the toolkit At the start of the study after a short verbal introduction about Haptimap and the toolkit as well as signing consent forms participants were invited to complete a worksheet see Appendix A involving three core activities installing the toolkit and running a demo app creating a demo app from the tutorial on the website and usi
180. ter and 10 users answered by calling or e mailing the test leaders Nine of these were able to carry out the test see demographic table below In the city 2 sighted users participated as well internally recruited one of which uses a wheelchair Table 3 3 3 1 Demographics of city Time Machine test Mobile GPS ID M F Age Vision Hearin Mobilit hone use GPS used for what loss uses 1 M 60 low vision hearing aid Nokia N minor wayfinder access loadstone 2 M 73 blind hearing loss Nokia Y Garmin etc 3 F 53 low vision Nokia N minor 4 M 67 lowvision hearing loss Nokia Y 5 F 46 low vision Nokia N Nokia 6 M 69 low vision iPhone Y 7 F 44 low vision Nokia X Trekker Breeze Nokia Positionera iPhone app 8 F 66 low vision iPhone Y puts in reference points 9 M 72 low vision Nokia Y Trekker Breeze 10 M 42 sighted Wheel chair iPhone4 Y car iPhone map for unknown IS ESL SS sighted iPhone4 X places 3 3 3 2 Rural test The pre final tests occurred at four different occasions See demographic table and pre final test table The final test was conducted with 5 participants aged 16 23 See also table below Table 3 3 3 2 1 Participants in pre final tests Test number Participants 54 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 14 2 12 3 12 13 14 4 14 15
181. terview we were especially interested in how the elderly users adopted the novel way of using maps whether the audio improves accessibility and in general whether the users find the Terrain Navigator useful 3 4 6 1 Using the Terrain Navigator on iPhone We were especially interested in how useful the participants found the audio functionality included in the map application It can be thought as a metaphor for a museum tour during which additional information is automatically given on interesting subjects to the user AII eight elderly participants thought the audio descriptions were an enjoyable addition to the application In addition to being pleasant to listen the audio descriptions gave the participants confidence that they are on the right track Safety was one of the main issues with the elderly participants The audio descriptions increased the feeling of safety by ensuring their location The participants considered that with Terrain Navigator they would be more comfortable and confident to go to new places The participants were pleased to find that with one click you can centre your position on the map This ensures you will always find your location 73 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Figure 3 4 6 1 The difficulty of the path would be useful information on the map The participants agreed that the easiest and the most intuitive part of using the Terrain Navigator was the audi
182. th HUI results city Arriving at a POI results both and Sound window results both The Participants subjective task load rating both and the Word cloud both also have separate sections 3 3 5 1 High level results city Of 11 primary participants the accompanying persons are not counted all users except one carried out all test tasks The one person who did not succeed had technical problems and the accompanying person was unable to solve those problems Three participants with visual impairment commented particularly on the need for a longer training period while several others also indicated that they needed more time to really get used to the app Also the question On a scale from 1 7 how would you rate your wish to use the app again 1 indicating low wish and 7 a high wish was answered with 5 7 by all of the participants with visual impairment Although several users with visual impairment would have wished for more training in the interview they commented on their learning while using the app While rating the concentration level using the guiding one user said Jn the beginning I would need to concentrate on a 7 level to follow the guiding in the end only a 2 level Another user while commenting the use of the HUI said It was hard in the beginning otherwise OK Holding a device and pointing with it while walking while also using a cane or guide dog proved to be possible but it had drawbacks When partici
183. th caution Figure 6 shows the distribution of the three types of sequence for guiding only sequences Figure 7 shows the distribution of the three types of sequence for guiding included sequences and Figure 8 shows the guiding only and guiding included combined The figures show a particular sharp increase particularly in the use of the sound option of the guiding function early in the festival a sharper increase than for all types of use of the app compare to Figure 1 This suggests that users are trying out the guiding function early in the festival perhaps because it is interesting perhaps because they want to meet up with friends or find a particular stage There also a correspondingly sharp decrease in the use of the guiding functions when the main events of the festival start at 18 00 probably as users concentrate more on the music and have hopefully found their friends and the stage they want Then at 22 00 when the main events end there is another sharp increase in use of guiding particularly the guiding only functions This suggests that users are again looking for their friends and the stages of the festival before the late night events start There is also an interesting suggestion of a difference in the pattern of use of the sound and vibration Although the initial increase in use of sound and vibration is similar but after 18 00 when the festival main events start the decrease in the use of sound in the guiding app is much shar
184. the application that we could discuss leave lots of space here Semi structured interview guide for the interviewer Question 1 You should start with the few specifics from the DoW see the Claims for last Demonstrators Trial document that you have chosen to evaluate the success of the application on These should be based on the specific functionality that you are testing You will have a short number of specific functionalities claims that your application should be substantiating that you are evaluating Loop over the list of claims doing 1 a q for each item on your list When you have gone over all the items in the claims list continue with the topic 2 till the end 131 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Question 2 Questions 2 are written in their short version to let the participants answer spontaneously If they don t have many ideas you can prompt their response by proposing the categories listed below You should record the interview in order to differentiate what answers have been spontaneous and which have been prompted For this question the answers are free at first but when some parts have been identified the Likert scale for importance should be answered as well as the question why did you choose that part as easy natural or difficult You can prompt the following parts in that order e The audio of the application e The tactile in the application e
185. the experimenter or yourself How satisfied were you with your performance in accomplishing these goals Note that this scale is reversed from perfect to failure Frustration How insecure discouraged irritated stressed and annoyed versus secure gratified level content relaxed and complacent did you feel during the task 123 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications NASA Task Load Index printable format Mental Demand How mentally demanding was the task Very Low Very High Physical Demand How physically demanding was the task Very Low Very High Temporal Demand How hurried or rushed was the pace of the task Very Low Very High Performance How successful were you in accomplishing what you were asked to do V ERN oe ES DE NU ES Perfect Failure Effort How hard did you have to work to accomplish your level of performance A ase defi ess ee det rdi Very Low Very High Frustration How insecure discouraged irritated stressed and annoyed wereyou i3 dod darse qoe gp pq dr Very Low Very High 124 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications German K rperliche Anforderung Wie viel k rperliche Aktivit t war erfolderlich 2 8 Ziehen Dr cken Drehen Steuern Aktivieren War die Aufgabe leicht oder schwer einfach oder anstrenged erholsam oder m hselig niedrig oe 0 0 0 ee hoch Zeitliche Anforderung Wi
186. the operating system or adding a screen reader In the table below the parentheses around a checked requirement mean that it is partly fulfilled 51 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 3 1 1 Functional requirements from D 5 1 Nr Requirement In TM Provide information multimodally about which direction the user can find things Distance information should also be provided Provide audio and or haptic notifications as the user passes interesting spots Allow for more information on demand Virtual Excavator Allow the user to personalize the feedback F4 Keep track of scores clues and challenges Allow both explorative and guided navigation F6 Provide the user with multimodal overview information both of the current environment and of how things were before On screen X information should be complemented with audio and or haptic feedback F7 Possible to select objects both by screen button interaction and by Den X pointing to the objects in real life F8 Manage your team add remove members and also get help to find them Joined Allow filtering objects based on distance year and type X F10 Not time critical easy to take a pause and restart later F11 Possible to design and experience both on location and from a X distance Library of ready made materials available Table 3 3 1 2 Non functional requirements
187. the patterns given from the application b Despite everything expected to the contrary verbal instructions in clock hand times worked very well when orientating visually impaired people in open spaces Both verbal instructions in times and stereo sound modalities were well accepted by the users However participants worked better with the verbal instructions in clock times than with the stereo sound modality Although it is easier to interpret the direction in which the turn must be made through the sound signals turning in the direction indicated by the sound it is easier to be accurate when knowing how much one must turn with the sound mode this information is given by the intensity of the sound and it could be difficult to appreciate the change in sound intensity within open environments Verbal instructions in clock times appear to not work very well in the city Instructions such as turn at 14 00 hours should be substituted by more straight turns such as at 15 00 hours which are more suitable for urban routes where straight angles are predominant it is very difficult for a person to calculate the angle corresponding to hours which are not 12h 3h 6h or 9h When you say them turn 2 o clock they know that they have to turn right but it is very difficult for them to calculate the angle When we were testing the app it told the user turn 2 o clock in a corner He turns until one o clock and as he found a wall in that direction he
188. the technicians in mobility and orientation that work in ONCE have the same formation and use the same techniques and procedures at work Figure 3 2 9 1 2 A Mobility Trainer working The application had been developed taking into account the knowledge and the experience provided by many technicians in previous phases where the user specifications and the application functionality 47 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications were defined What it is achieved involving the experts in these sessions is that the final application more closely meets their needs The application was presented in the ONCE general headquarter in Madrid in the 28 of May of 2012 The application was executed and its functionality was shown to the audience using a computer connected to a projector After the presentation the experts were invited to share their opinions make contributions and suggestions and point out all the problems they detected Later they filled in a questionnaire Further explanations were provided when requested by the evaluators The results of the questionnaire are presented in the next section 3 2 9 2 RESULTS OF THE QUESTIONNAIRE Item Utility of the application for your work rate it from 1 to 4 where 1 is not 10 1 useful maximum possible score 12 2 Does the application functionality meet
189. ther than being discouraged due to a small issue with documentation 3 6 3 Evaluation of the HaptiMap Toolkit with professional developers 3 6 3 1 Introduction If a toolkit is going to be taken up by professional developers in industry it needs to be useful and usable by the target user group even if it is an open source toolkit Thus it should exhibit the qualities of usability that we try to instill in the applications we develop However evaluation of whether tools for developers meet the same criteria we set for users are quite rare particularly evaluations with real users In the HaptiMap project we have undertaken three evaluations of the HaptiMap Toolkit with users two groups of students and an evaluation with professional developers In this section we report on the evaluation with professional developers Because developers tend to be very busy people either working for themselves or for a company it is not easy to ask them to give up time to try out a new toolkit for long enough to give a realistic assessment and then to take part in an evaluation So we needed to find a lightweight and flexible evaluation methodology and professional developers who would be willing to take part In terms of the sample of developers we used three sources of participants e Developers who had entered the HaptiMap competition so had by definition used the toolkit in earnest 2 participants were recruited this way e Freelance developers who
190. thought to be used by visual or audible means it 35 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications was considered interesting to include in the final tests some series suited to their specific profile They are a limited amount of tests due to the complexity of working with this group The tests consisted in providing information on orientation during an outdoor experience being communicated by means of vibration patterns Two deafblind people participated in them A familiarization session was carried out indoors before making the tests on the street The aim of it was to get the participants used to the device and the application In this session they were able to perceive the mobile vibrations and the significance of each vibration pattern with the aim of comprehending its relevance in directional terms 3 2 6 1 Participant overview Table 3 2 6 1 Session 2 sample TOTAL FAMILIARIZED NUMBER AGES GENDER AUDIO VISUAL CONDITION WITH MOBILE OF USERS TECHNOLOGY Residual Residual 2 53 35 M F BLIND VISION HEARING YES NO 1 1 2 0 1 2 0 The collaboration of two interpreters of dactylologycal communication was necessary in order to exchange information with participants who were deaf blind The female user due to the fact that she had some residual hearing was invited to evaluate the two sound modalities of the application verbal instructions timetable
191. tially sighted travellers These demonstrators ensure that a broad spectrum of using and accessing map data are targeted The main purpose driving this evaluation was to test of the presence and functionality of the applications fulfilment of the requirements derived in the beginning and modified in the iterative design process of the HaptiMap project We aimed at covering the kind of usage indicated in the application scenarios described in the HaptiMap Description of Work Thus the demonstrators target planning navigation and exploration in different mobile and stationary contexts The protocol for the evaluations included extracting the functional and non functional requirements and basing evaluation tasks and interviews on them The summative evaluations also gathered demographic data made qualitative observations and used quantitative tests As each demonstrator was intentionally quite different overall the resultant redesign recommendations are largely dependent on the domain of the applications An analysis section explores the results from the requirements perspective determining that as a whole HaptiMap fulfilled 81 of all originally stated requirements Of the remaining 19 of the original requirements some were partly fulfilled if these are added the percentage rises to 92 of the requirements fulfilled Of the remaining requirements some were considered less important based on results of interim iterative evaluations or the formative e
192. tions were used in the city 2011 and in the forest 2012 Some technical problems remain with long questions and answers that do not fit well on screen but in essence this goal is fulfilled G6 Theapp will be usable both for hiking and for city tours F5b Allow guided navigation trails Users have been using the app to follow trails at two different occasions in the spring 2011 and in summer 2012 In 2011 this was conducted in a city environment and in archaeological field and reported in D1 3 All users in that test succeeded in using the app In 2012 it was used in a semi prepared nature environment All users succeeded in using the app and take part of the information in it However the rugged terrain posed some problems connected to wayfinding and the GPS inaccuracies may be a bigger problem in the forest Despite these considerations we have fulfilled these goals G7 A wide age range of users will be able to use the app approximately 7 70 years 66 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications The oldest user who participated in the tests was 72 years and the youngest 7 years The use of the app differs somewhat the kids like to run and catch the next point rather than listening to the virtual guide but they use the same app nevertheless Thus we have fulfilled this goal G8 Users will be able to use the explore mode with an introduction to it F5a Allow explorative navigation select points
193. tor mental och perceptuell aktivitet medf rde uppgiften t ex t nka besluta ber kna komma ih g 1 2 3 4 5 6 7 titta leta etc 2 Hur mycket fysisk aktivitet medf rde uppgiften v nda str cka kontrollera aktivera etc 1 2 3 4 5 6 7 3 Hur mycket tidspress upplevde Du Kunde du genomf ra uppgiften l ngsamt och avsp nt eller 1 2 4 7 snabb och hektiskt 3 j 4 Hur bra tycker Du att Du uppn dde m let med uppgiften Hur n jd r Du med Din prestation 1 2 3 4 5 6 7 5 Hur mycket beh vde Du anstr nga Dig f r att uppn Din prestationsniv 1 2 3 4 5 6 7 6 I vilken utstr ckning blev Du os ker irriterad stressad eller modf lld under uppgiften 1 2 3 4 5 6 7 130 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Appendix G Semi structured interview guide You have developed a short number of specific functionalities claims that your application should be substantiating Loop over the list of claims doing 1 a k for each item on your list When you have gone over all the items in the claims list continue with the topic 2 till the end Functionality being tested FBT Iterate over all the specific functions you are testing in your demo a Observation and debriefing with user Did the user successfully use the lt FBT gt while performing the task Yes No free text Which part of the lt FBT gt did you find easy and natural Please list them Of the
194. uck in buildings or down a dangerous slope n certain areas where GPS positioning is problematic a user invoked arrival simulation may be created meaning that the user can decide to have arrived to a point while still more than 10 m from it Provide POI information in both text and audio Some users may prefer to read on screen while other may not be able to Provide alt texts to pictures As the screen on mobile devices is small consider using a gallery form of providing information flipping between pages with different location based information At historical information points provide both contemporary what is here now and historical information about the place Sound windows historical sound clips Make sound windows possible to turn off Use mainly sounds that are not there naturally Forest birds in a forest should for example be used for a particular bird generally not for providing forest atmosphere Connect the sounds to the available historical information or make sound windows possible to explore by examining them closer to understand what they are and why they are there 68 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Creating trails in rural areas When creating a trail use a GPS device with enough precision 5 decimals in the WQGS84 system using degrees and decimals Hobby GPS devices for geocaching or hunting are generally good enough Use
195. ude some contexts that are not optimal because it will never be optimal in the real usage context A list of possible conditions that might limit the use of your application is given below Optimal conditions 1 2 3 4 5 Non o l 2 Optimal lighting and the user can easily look at the screen Quiet environment without sound restrictions The user holds the device in the hands no gloves or similar The user has nothing else in the hands and can use both two handed and one handed grips The context does not require any attention ptimal conditions Non optimal lighting e g bright sunlight or the user has or wants to look elsewhere while crossing a street negotiating rough terrain etc Noisy environment e g in a crowd by a busy street at train station at festival or fair etc or an environment where sounds are not suitable egg meeting concert theatre bird spotting not to frighten the birds etc Situation which limits your ability to touch the device cold hands using gloves cold weather or keeping the device in a pocket or a bag or external vibrations shaking makes it hard to sense the feedback Situation which limits your ability to manipulate the device like having to hold something else in one or both hands e g umbrella bag take away coffee ice cream pram child etc or shaking vibrations that make it difficult to interact Context that requires attention e g other people traffic
196. uide and quiz application The demonstrator is now mature to be tested in different contexts tied to the scenarios described in the DoW while at the same time it covers all targeted user groups The scenarios we target are e b Hiking in a national park e 3 Visiting Linz tourist scenario We planned to test with a wide range of users ages approximately between 7 and 70 with persons with low vision or blindness as well as fully sighted users We also tested in different contexts by testing in both rural and urban surroundings and both with the free explore mode and the guided trail mode The first plans for a virtual historical guide application had the following requirements presented in D 5 1 The tables have been edited such as they show which requirements were considered and have been attempted to be solved Since the design process has been iterative requirements have continuously changed based on pilot and end user testing As for the requirements that have not been realized in the Time Machine app some have been entered in other demonstrators names of these are also put in the tables below The NF5 Basic functionality available also without vision has moved up to be a functional requirement The entire application should now be inclusive i e usable without vision or in the pocket This does not however cover the starting of the application an issue that is not solved by the HaptiMap toolkit as such but by modifying
197. und 14 00 and then a steady decline with no sharp rise about 22 00 This suggests that people were looking for their Facebook friends before the main activity of the festival started and not so much in the breaks between the main activities Another feature of the sequences including the Facebook function is that they tend to be long sequences the mean length is 28 35 web service calls standard deviation 49 13 in comparison with a mean of 16 53 for all the use sequences A t test shows that this distribution of sequences is very unlikely to be taken randomly from the overall set of use sequences t 4 04 df 281 p 0 001 so this difference reflects an underlying different behaviour A further particularity of the use of the Facebook list is that it is usually only used once or twice in a sequence Table 4 shows the distribution of number of Facebook list calls in the 282 sequences that included such calls This probably reflects the fact that people only had one or two friends from their Facebook list at the festival But it also suggests that the app worked well as they did not repeatedly try that function within a sequence 90 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 5 2 3 Breakdown of frequency of calls to Facebook within a use sequence Number of calls to Facebook list in a use sequence Frequency Percent 1 231 81 990 2 47 16 790 3 3 1 190 4 1 0 490
198. und through the interface Requirement F5 All participants could use the friend finder although the location on the map was only visual Regarding F6 and F7 The guiding was implemented through audio and vibrations in a compass mode with possible goals being music stages friends or other fixed or moving locations It was usable without vision but was giving the direction without taking into account the paths and obstacles The festival area was fairly open but the scenes and some natural obstacles ponds made it difficult to navigate everywhere without seeing the area During the qualitative evaluation participants who were totally blind needed external help for navigation of obstacles while users with some sight or fully sighted could navigate independently Table 3 5 1 3 3 2 Event Guide JUICY BEATS non Functional Requirements The system user interface is intuitive and easy to use See Quantitative for the first time user no user manual is needed Evaluation Yes although some map overview functionality is lost The system is usable for both sighted and visually impaired users Partially GPS problems NF3 The system is stable and reliable interfere with other types of communication The feedback from the system is easy to understand Yes NF6 Vibration and speech output do not disturb user activities The use of the system is socially acceptable the NF4 required interaction is unobtrusive subtle and doe
199. ur performance in accomplishing these goals Note that this scale is reversed from perfect to failure Frustration How insecure discouraged irritated stressed and annoyed versus secure gratified level content relaxed and complacent did you feel during the task 2 1 6 Semi structured interview The semi structured interview will allow the designer to evaluate the system across end users and to a lesser degree across applications Asking the same questions supports setting priorities on interface re design and insight into user needs and interests It is important to keep in mind while doing these interviews that we are not trying to analyse the cognitive process of navigation in the real world nor of what the society needs to do to better fit all its members We are evaluating how well the participant navigated using the system with particular emphasis on breakdowns so most of the questions will have a narrative part and a Likert part e In the narrative part make sure to ask why the participant answers that way e In the Likert questions you will use 7 points scales resembling the following 1234567 A detailed interview guide is presented in Appendix G which should be general enough to adapt to different tasks and contexts 2 2 Evaluation protocol for toolkit users The specifics of the evaluation procedure for programmers using the HaptiMap toolkit are laid out in Appendix H The procedure is in three parts e Task 1
200. valuations reported in D 1 3 and some were not tested The user evaluations of the toolkit aimed at assessing the ease of toolkit use by student and professional developers returned results that demonstrated that with several small caveats the HaptiMap toolkit is ready for broad use This is furthermore demonstrated by the reuse of the toolkit by several AAL projects national projects and graduate courses in the near future 12 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications 1 Introduction 1 1 User requirements Driving the design of the prototype demonstrators are the requirements generated from user studies conducted at the start of the HaptiMap project that culminated in the HaptiMap deliverable D1 1 These requirements are founded in the initial Haptimap vision as expressed in the HaptiMap Description of Work These prototype systems were evaluated using the guidelines expressed in HaptiMap deliverable D1 2 1 2 List of Demonstrators This section briefly presents the demonstrators that have been tested in this round of evaluations to make this a stand alone document for future use The full functionality technical detail and the user manuals of the demonstrators are reported in D5 2 The aim of this summative evaluation in contrast to the last D1 3 formative evaluation was to measure how well the project succeeded in providing useable and accessible functionality for the requirements discover
201. velopers when shown its capabilities they are keen to employ it and it is straightforward to get started building applications with it Nevertheless the evaluation has raised some issues that should be addressed for the toolkit to be fully useful Whilst relatively minor addressing the recommendations in the following section will improve toolkit usability to greater levels Recommendations for Improvements Ensure new developers are steered towards the getting started guide One of our participants attempted to start by compiling the toolkit from scratch This significantly reduced his initial confidence in it Documentation should stress that compiling the toolkit in this way is an advanced function and that new developers will find it easier and faster to use the pre compiled version from the getting started guide Compilation documentation should be updated to stress this and sanity checked to ensure that files and folders being referred to are correct Ensure tutorials remain up to date Whilst the tutorials we used were useful well thought through and got participants working quickly there is inevitable rot as the underlying Android platform and other platforms is updated This manifested as errors that were from the old version of project files in the Haptimap toolkit that needed to be upgraded using the android tools before compiling This is obvious and trivial for a very experienced developer But for a new developer it may
202. ven in steps Modes e Through sounds The user found the sound annoying in closed environments but acceptable within open spaces It was straighter forward than the verbal instructions because the user did not have to think He listened and turned in the direction of the sound e Through vibrations It was a bit harder to interpret the patterns The user seemed more disoriented the user stopped more frequently to see which direction was the correct one The user turned too quickly and felt became disoriented e Through verbal announcement of times The user hesitates less Less turns We conjectured that because it was his first time using the device he paid more attention to it than to the cane When detecting an obstacle in the middle of the trajectory the equestrian statue in Mayor Square the user did not know how to overcome it 3 2 5 4 2 User E8 The participant is completely blind and uses a guide dog During the trials the user s hands were both busy with the guide dog One of the hands is used to hold the short or rigid lead through which the user feels the environment The other hand holds the flexible or soft one which is used to guide the dog prompting it to turn and act The dog is an additional variant to be taken into account within the system e fitis going on a familiar route and it arrives at a cross the dog does what he is used to e f the dog is unfamiliar with the route and it arrives at a cross th
203. viously meeting with people finding a specific location where to meet caused major problems especially with regard to the fact that most of the users have many friends with visual impairments themselves So far the most common methods to meet with friends at a certain location involved phoning arranging a time and place use applications like Ariadne MyPos WhereAml etc Therefore nearly all users came to the conclusion that Joined is what we were missing especially the unobtrusive means of navigation by vibration thus not requiring to interact with the phone or concentrate on auditory instructions The latter one was also of great importance since on the one hand it works in noisy environments and on the other hand is not stigmatising Table 3 1 4 Subjective user rating of the Joined app UI Marking App Functionality Overall Marking 1 2 1 1 2 2 1 3 2 1 2 2 1 1 2 2 2 1 2 3 1 1 1 2 2 3 3 1 2 1 2 1 2 Avg 1 36 2 1 73 25 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Table 3 1 3 shows the user rating for users with visual impairment of the app However the users had many comments regarding the app in general and specific features In terms of improving the accessibility the following comments were made by the users 1 Add send place The cursor does not jump directly into the text field for naming the place Pressing done
204. vision However as the comments on G2 and the results about the HUI above show there were still problems with consistency and ease of use However with enough learning time all parts were available This goal is fulfilled 3 3 7 Conclusions and recommendations From the previous sections it is clear that all 23 participants have used and succeeded to use the different modes of the Time Machine application All goals except one that could not fully be tested have been at least partly fulfilled Since the user interface of the Time Machine app is quite complex and there are very many different levels of improvement it is not the goal of this section to cover every detail of the app We do not cover recommendations for improvements on the provider side hence recommendations that have to do with better positioning precision for example will not be brought forward The baseline for our recommendations is a system that has the precision of GPS systems of today July 2012 The above described results give a background to the claims we consider to have reached Also these results lay a foundation for recommendations or guidelines when designing pedestrian wayfinding apps These are our sketches of such recommendations 67 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Guiding We have investigated multimodal guiding and in respect to particularly vibration guiding using the built in vibration motor in a mobi
205. volved in the project they are probably going to be somewhat biased towards the toolkit Thus we did not want this group to predominate in the evaluation and decided to only recruit two participants from this source However it was probably helpful that this evaluation was conducted by someone who had not been part of the HaptiMap team throughout the project so that these participants were not providing information to the colleagues they had worked with throughout the project It was stressed to these participants that information provided as part of this evaluation was completely confidential between the evaluator and the participant and would only be shown to the HaptiMap team in an aggregated and anonymous format We have adhered scrupulously to that principle So participants should have felt free to provide their honest opinions about the toolkit It was also thought that if participants were given the more anonymous option of answering in a questionnaire rather than in an interview more honest answers would be provided So these participants were asked to complete a questionnaire In terms of the methodology to be used in the evaluation a very flexible approach was clearly needed We wanted to ensure that participants had really used the toolkit important for the freelance developer group we wanted to make the evaluation as time efficient for the participants as possible relevant to all groups and completely confidential and anonymous imp
206. xigence Mentale De Faible Elev e Faible Faible Bonne Faible Elev e Exigence Physique alo slels elslelslolulolslulslelvlslv Elev e Exigence Temporelle alefslelslelslelslolululslulslelvlslv Mauvaise Performance alefslelsl lsl lslsolufa fir r fis das r s os oo Elev e Effort 1 2 3 4 5 Faible 126 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications Spanish Dimensi n Extremos Descripci n Cu nta actividad mental y perceptiva fue necesaria Por Exigencia Mental ejemplo pensar decidir calcular recordar buscar investigar a Baja Alta emper P x M J etc Se trata de una tarea f cil o dificil simple o compleja pesada o ligera a Diol Cu nta actividad f sica fue necesaria Por ejemplo empujar 2 Exigeneia Fisica Baja Alta tirar girar pulsar accionar etc Se trata de una tarea f cil F dif cil lenta o r pida relajada o cansada 3 Exigencia Cu nta presi n de tiempo sinti debido al ge al cual se T 1 T Baja Alta suced an las tareas o los elementos de la tareas Era el ritmo emporal lento y pausado 6 r pido y fren tico 4 Esfuerzo E Bajo Alto En qu medida ha tenido que trabajar f sica o mental mente para alcanzar su nivel de resultados 5 Rendimiento Hasta qu punto cree que ha tenido xito en los objetivos rf Bueno Malo estable
207. y as possible A Basic Information 1 Gender __ Male __ Female 2 Age 3 Occupation current past 4 Highest completed education B Hearing Vision and Motor ability 1 Please rate your visual ability with the best possible correction for your glasses __ Full vision __ Minor vision problems __ Moderate vision problems __ Severe vision problems with some residual vision __ Blind 2 Are you colour blind __ Yes INo 3 If you have vision problems please provide some details 4 Please rate your hearing __ Full hearing __ Minor hearing problems __ Moderate hearing problems __ Severe hearing problems with some residual hearing __ Deaf 5 If you have hearing problems please provide some details 6 Do you have practical musical experience playing instruments singing in a choir etc __ Yes _ No 7 Please rate your motor abilities __ Fully mobile 117 HaptiMap Deliverable D1 5 PU User tests of the final demonstrator applications __ Minor motor problems __ Moderate motor problems __ Severe motor problems 8 If you have motor problems please provide some details 14 Dominant hand __ Left __ Right C Experience of Mobile Devices 1 Do you own a mobile phone _ Yes _ No 2 If yes how long have you owned a mobile phone __ Less than 6 months l6 months to 1 year _ 1 to 2 years __ More than 2 years 3 What is the model an
208. your needs regarding route 10 designing rate it from 1 to 4 where 1 is not useful maximum possible score 12 Item Similarities differences in how they work 3 Similarities Differences The parameters used by the application are The MT includes references he she is able to similar to the ones used by MTs at their work identify in the user given by her his The main advantage is that with this tool they characteristics and or the specific situation of are recorded and can be re used the learning process The application provides general detailed MTs find differences in terms of quality instructions on how when and where an action speed and comfort when they complete the must be executed as well as indicators of task with the application direction lines Item Ease difficulty to handle the application possible values easy difficult very difficult 4 Easy 3 Difficult 0 Very difficult 0 Item Route edition introducing mandatory crossing points rate it from 1 to 4 10 5 where 1 is not useful maximum possible score 12 Item Automatically generated instructions on the route given by the application are considered 6 Incomplete Enough Too complex Generic Others 0 2 0 2 0 Item Automatically generated routes by the application are valid only for a user or can be applied to multiple users 7 A user 0 Multiple users 3 Complete questionnaire can be found in Appendix L 48 HaptiMap Deliverable D1 5 P

User tests of the final demonstrator applications - Certec

Contents

Download Pdf Manuals

Related Search

Related Contents