Home

ETNA Deliverable D7.1 (Plan for Validation of the TN portal) (PDF

1. 4 Which measurements instruments designs are found in the literature for measuring usability of a rating tool Which can be considered to be the more reliable and valid ones for the Portal The same procedure as for the upload tool was followed to find instruments measuring the usability of the rating tool To find qualitative literature the PICO question was formulated as follows Is there any research done regarding the usability of a rating tool of an online database or are there any opinions from done surveys which describe how a rating tool can be made more handy and reliable The search was executed with the following keywords in Pubmed OT Seeker and Google evaluation online databases rating tool for online databases and research rating tool These words were combined with the following threads evalu AND online database AND research OR survey Finally the review of literature was performed in a multilingual way The keywords were used in English Dutch and German Literature was reviewed in Google PubMed and OT seeker by using threads like AND or OR ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 10 23 SHS i ETNA ICTpsp Thematic Network on ATs Results Usability measurement models In the following section the findings are presented in relation to each research question la Which variables are suitable for validating usability of databases and which of these variable
2. a method for the validation of the portal aplan for the validation of the upload tool and aplan for the validation of the rating tool Table 4 shows the validation timetable Part 1 The Portal The heuristic evaluation Annex 1 should be executed by at least 3 experts preferably by different partners in different countries The task oriented co discovery learning Annex 2 approach should be executed at least by 2 end users for each stakeholder group Tasks should be about finding ICT AT resources as defined in deliverable 3 2 through the search engine The tasks should cover all the modalities present in the system i e different ways of searching and searching for different resources It is preferred to have the test done in various countries so as to address multilingualism The PSSUQ Annex 2 on satisfaction can be filled in by the end users after finishing the co discovery learning method Because of the multilingualism of the system the tests should be performed in different languages and in different countries In order to do so the tests and the instructions have to be translated into the native language The inclusion criteria for the various stakeholders are e Ability to perform two tasks at the same time e Knowledge about using a computer minimum 1 hour per week e People with any kinds of disabilities provided they are able to operate a common computer or bring along their own computer appropriately config
3. STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 6 r N A Deliverable D7 1 06 06 2013 Annex 2 pg 19 20 ICT PSP ETNA project 270746 SL Tv ETNA Checklist for researchers m uu oe Set up the location with all documents video cameras the folders of the participants and something to eat and to drink Greet participants and introduce development researchers and observers Explain the usability evaluation in general Let the participants read and sign the informed consent Let the participants fill in the pre test questionnaire Ask the participants the general questions about their expectations regarding the usability evaluation Explain the co discovery learning method and the process to the couples of stakeholders Check if there are any questions Explain the hardware setup to the participants Show the participants several help resources Set all video cameras on and start with the usability evaluation After hours the participants roles change After completing the usability evaluation there is time for a short brake 15 minutes Ask the participants what the top findings are Let the participants fill in the PSSUQ Thanks to the participants and hand out the presents Deliverable D7 1 06 06 2013 Annex 2 pg 20 20 ANNEX 3 ICT PSP ETNA project 270746 Funded by EU s ICT Policy Support Programme Competitiveness and Innovation Framework Programme Led by In Coordination with QemteSreedt RIA
4. 1987 The metaplan method communication tools for planning learning groups Goethestrasse Germany Sellito C amp Burgess S 2005 Towards a weighted average framework for evaluating the quality of web located health information Journal of Information Science 31 4 p 260 272 Shackel B 1991 Usability context framework definition design and evaluation Human factors for informatics usability UK Cambridge University Press Soken B 1993 Methods for Evaluating Usability Section 5B Tashhakkori amp Teddlie 1998 mixed methodology combining qualitative and quantitative approaches Sage applied social research method series volume 46 Wang Pierce Madnick amp Fisher 2005 Information Quality ISBN 978 0 7656 1133 8 Wilson C amp Blostein J 1998 Usability Techniques Pros and Cons of Co participation in Usability Studies Reprinted from Usability Interface Vol 4 No 4 April 1998 Yengin I 2010 E learning success model for instructors satisfactions in perspective of interaction and usability outcomes USA ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 23 23 ICT PSP ETNA project 270746 PI AAN ETN ICTesp Thematic Network on ATs Annex 1 Heuristic evaluation questionnaire Heuristic Evaluation a system checklist 1 Visibility of System Status The system should always keep user informed about what is going on through appropriate feedback within reasona
5. Led by In Coordination with Qretusion practice Pe pet tae Deliverable D7 1 06 06 2013 Annex 3 pg 10 10 ICT PSP ETNA project 270746 oe SHY a ETNA by In Coordination with With support of Fondazione Carle Gnocchi RIATISAaNI I Goue EEE Onlus Theret eter nn Ate sti opon atar I nstruction to test the Rating Tool The Rating Tool The Rating tool is a web based online questionnaire for evaluating measuring the use of products in the ETNA Database We would like all partners of the ETNA project to use the rating tool at this point Feel free to sent it on to your own network to have more participants use the rating tool by In Coordination with With support of Deliverable D7 1 06 06 2013 Annex 4 pg 1 6 ICT PSP ETNA project 270746 The Rating Tool Please perform the following tasks Use the product you already uploaded or another CT kind of product you are familiar with present in the information system Use the rating tool on the product as mentioned above Look for other peoples ratings on products Fill in the online questionnaire The following sheets will show you how to perform the given tasks step by step e g HotSpots 2 a software for accessing the computer in scanning mode European Thematic Network on Assine Information Techncengees T Brief presentation CT om Pe ete eg O ae Feios a i Centres mar Eii ita ime m ee a T ee pee A European neteork
6. Averag gt rating 24 In Coordination with go back to the rated product by clicking on the button at the bottom of the filled in rated form please click on the button Read the Reviews for this Product With support of You will see a screen similar to this Here you are able to read the ratings done by other users You have the option to click on see statistics in more detail With support of gassa ill O Deliverable D7 1 06 06 2013 Annex 4 pg 5 6 ICT PSP ETNA project 270746 Now we would like to ask you to fill in the online questionnaire to give your opinion on the rating tool TIP you might want to use the rating tool a the same time with the questionnaire Use the link below to start the questionnaire http www enquetemaken be toonenquete php id 142120 Led by In Coordination with With support of sontortane e Gnocchi fusion practice o i Be pet te Thank you for your help Your feedback is very much appreciated We will keep you up to date Led by Fondazione Don Carlo Gnocchi Onlus Deliverable D7 1 06 06 2013 Annex 4 pg 6 6
7. Manual Usability Evaluation of the ETNA database Team Zuyd Kenniskring Technologie in de zorg The Netherlands 2013 YD ETNA Thematic Network on ATs Index Oa EO ee tec E sea ae ee ceases psc E E E NE E A EE E ET Z 1B 1a othe o i eo es 21 lt RONCES sene ORE e ETE PN OPTION MOMS SECTEOTU UNE MECEE E RIN Mme MET n INN tr orc r 3 In l sion ANG exclusion Criteria 2 lt ssceccassseorasnbannsaamsecaienctococoaussoraraieretattenatanaleneseseuatecaansdaaaqnateonioacionoeareisvesarasaas 4 ac IY e E E E E E 5 La OTC CO OS CNG e EA E E E E E E E E 8 PHS SUNS onni ae pps E E E E E E S E E E E E N AR 8 Co discovery questions asked by participants during the performance cssssssseeeeceececeeceeeeeeeaaasaeeeeeees 10 SEs TAS CCG i Fr At acess scocasacasetanusanecanaconceasescesscancantansacsananunosacessesseuaatesaatanbacsarenueocbcesaesquantesantaanaaensenenoceconeans 12 OS Ey AO IGE aa saacat E E E E AE AE E E 15 PSSUQ The satisfaction quest onna E ssseccsnctesa ccecaanctindtesetedsadunesetesntaaidedaceacanechsndlesedeieideataaidedacnaetescbseddeseueat 17 ee N EO AGC NOUS A AEN E E E E AE 20 Deliverable D7 1 06 06 2013 Annex 2 pg 1 20 ICT PSP ETNA project 270746 Introduction In this document you will find a manual with different parts of documents to evaluate the usability of the ETNA database It is important to read all parts carefully to perform the evaluation in your own country In this manual
8. according to the sections defined by ISO in categorising ICT AT Multilingual Availability of the textual information in the native language and in English Completeness In how far the information answers the original question Efficiency Completion of task In how far a task can be completed regardless which Few resources are needed during ISO manner or duration the performance has standard use process related Consistency System response predictability for the user in any given Nielsen situation Filtering The number of filters that can be used Search options Multiple search options are available e g ICF coding keywords problem to solution product type product name manufacturer supplier importer Display results Multiple options to display results Concreteness The option to adapt the information in a desired amount e g adjustment of character size selection of specific information Low number of The possibility of finding the desired result in the lowest clicks number of required navigation or selection actions Learnability Instructions Availability of a spoken or written manual providing The possibilities suggested to manuals adequate and simple instructions understand the operation of the system Memorability Registration Options to store relevant information so that the user can Memory from the system to easier quickly restart storing sets of articles making use of use the system predefined user types which specific queries E
9. b According to stakeholders In the following the results of the focus groups and semi structured interviews are described for each stakeholder group an overview of the variables of usability mentioned by the stakeholders is shown in Figure 1 End users End users indicated that they need a clear instruction manual enabling successful use The manual should be found on the homepage of the website and should consist of short step by step instruction This enhances the usability of the site as well as a clear structure End users mentioned a pyramid with some tabs of topics and subtopics leading towards more information without using too much text They preferred navigation involving short texts and frequent mouse clicks instead of taking time to read large quantities of text Specific information should be provided at each step Many end users highlight that speed is important By providing a structured process end users would be able to work accurately and move quickly from A to B Effective search is defined as the ability to enter keywords in the search bar Quality information is defined as information that is up to date When using a database completeness may be important but providing the latest products seems to be even more important Finally end users would like to find information preferably in their native language Professionals Professionals indicated that it is important to find an efficient way to search In practice this
10. easy to remember and information should be quickly retrieved When results are found with a keyword researchers want the database to come up with several options The preferred language is English with no translation needed With respect to products it is important to include user reviews experiences In addition they would also like a description on how to use the product Policy makers Policy makers mentioned that a logical structure is one of the most important aspects when using a database According to policymakers the characters should not be too small not too much text on one page and the descriptions should be short Furthermore the structure should be associated to a logical lay out The structure of a database should offer an obvious and clear view with a user manual explaining how to use it Searching should be possible in different ways namely from problem to solution product oriented on manufacturer supplier importer and with matching keywords The International Classification of Functions ICF could be used to map all sorts of problems and hence to search from problem to solution According to policymakers the desirable language for an international database is English other languages are not needed Like other stakeholder groups policy makers mentioned that product information should be up to date and there should be an indication of availability of the products by region e g the Netherlands preferably by an o
11. means that they want to achieve the result as quickly as possible within a few mouse clicks Possibilities to search should also be presented with several options professionals would like to search by keywords but would also like an advanced search The system should have a filtering option so that targets properties product name theme developers and suppliers can be filtered When the results are displayed the best match should always be presented on top Information provided should be up to date and very broad Information especially on the AT ICT products should include visual support Professionals would like to see how a product is used It 1s important however that there is no information overkill limited information that comes to the point contributes to the efficiency of the use of the database Professionals are willing to pay for a database if it provides useful information and if advertising and other irrelevant information is avoided Payment offers the possibility to register and registration offers several potential advantages like a forum a personal login page a search history storage a newsletter with personal preference settings up to date in formation to up to date information new developments or products relevant literature the top 3 most searched products relevant product reviews Professionals consider it important to have the possibility to trace suppliers of ICT AT within their region In addition it is c
12. on Services Service Dominant Logic Service Science and Network Theory Nielsen J 1994 Scenarios in discount usability engineering In Carroll J M Ed Scenario Based Design Envisioning Work and Technology Nielsen J 1993 Usability engineering San Francisco CA Morgan Kaufmann Nielsen J amp Landauer T K 1993 Usability Testing with 5 Users Jakob Nielsen s Alertbox useit com 13 03 2000 in A mathematical model of the finding of usability problems Proceedings of ACM INTERCHI 93 Conference Amsterdam The Netherlands 24 29 April 1993 Nielsen J Mack R L 1994 Usability Inspection Methods Proceeding CHI 94 Conference Companion on Human Factors in Computing Systems Pages 413 414 ACM New York NY USA Poldma T amp Butler Kisber L 2009 The Power of Visual Approaches in Qualitative Inquiry The Use of Collage Making and Concept Mapping in Experiential Research Journal of research practice Volume 6 Issue 2 2010 Poulson D Ashby M amp Richardson S 1996 User Fit HUSAT Research Intitute Leicestershire United Kingdom European Commission Rosenbaum S 1989 Usability evaluation vs usability testing when and why TEEE transactions on professional communication 32 4 Rosenbaum S 1989 Selecting appropriate subjects for documentations for usability testing Proceedings of the third international conference on human computer interaction Boston Massachusetts Schnelle W Stoltz I
13. portal based on the field tests findings In response to this recommendation the ETNA Consortium decided to start preparation of the validation in September 2012 so as to be ready for the field tests right after the first release of the ETNA Information System This was launched in mid February 2013 the second release is scheduled in July 2013 which means that the test activities will be mainly focused on the first release of the Portal and partly on the second one In order to avoid the risk that possible critical issues encountered in the course of testing activities are made known too late close interaction will be kept with FDCGO responsible for the technical development to promptly inform the technical developers and allow them to fix problems as soon as possible As the ETNA Information System has not been developed from scratch but evolving from the already existing EASTIN system the testing validation activities will concentrate on the additional improvements introduced to this system by the ETNA project rather than on the facilities that have been inherited from the EASTIN system without modification Indeed the latter had been extensively validated in 2005 within the ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 5 23 PI Lan ETNA ICTesp Thematic Network on ATs previous eTEN EASTIN project This means that all tests activities will focus on e contents related to ICT products rather than on assist
14. the current situation in the field of researches and development AT 4 Professionals 2x Have to have some knowledge on AT assistive technology Deliverable D7 1 06 06 2013 Annex 2 pg 4 20 ICT PSP ETNA project 270746 Instruction Manual Before starting the evaluation there are five important components the researcher have to consider Translate all documents you give to the participants in your own language Greet the participants Introduction about the questionnaire Orientation about the evaluation process Performance of the evaluation Debriefing with participant The researcher has the responsibility to make the participants feel as comfortable as possible during and after the test Moreover the researcher should be well prepared before the participant s starts with the performance of the evaluation Below a further description of conducting the test 1s given Arrange and prepare suitable A test lab has to be available for at least Pencils white paper Plan test three hours per session and is large enough for the amount of participants observers AND OR researchers and the equipment which is necessary for conducting the test Provide computers For two participants one computer is Computers with pc laptop with internet needed internet connection connection per two Write down the operating system and participants version of the operating system used Three recording device To record the participan
15. the models of Shackel and Eason are not user centered and since their validity is not proven Hartson 2000 these models were found to be not suitable in this study Userfit by Poulson 1996 is a process oriented model which is user centered This model describes usability with the following concepts effectiveness efficiency safety comfort and satisfaction This model is more directed towards appliances Userfit takes into account the social and physical environment of the user but it focuses less on the system This tool is not mentioned in the literature as a model which can be uniformly used Furthermore no quality indications on reliability and validity were found Therefore this model was found not suitable for this study According to Nielsen 1993 usability of a system is multi faceted and consists of five properties easy to learn efficient to use easy to remember low error rate and meets user satisfaction This model is often used to uniformly evaluate different systems and is user centered Nielsen 1993 Cali et al 2011 This model was found to be suitable for this study According to Sellito amp Burgess 2005 usability is defined in ISO in terms of the results of using the product effectiveness efficiency and satisfaction These are measures of the entire system product user task and environment not only of the product ISO 9241 11 is a product oriented model and is not user centered Barnum 2010 Kunjavsky 2
16. the resulting plan is summarized The answer to the overall research question Which methodology will be suitable for measuring the usability and content in 5 months regarding to ICT AT in the TN Portal for each stakeholder group can be synthesized as follows The most adequate and valid design to evaluate the TN Portal appears to be a combination of testing inspection and inquiry The heuristic evaluation to be used by experts appeared to be the most suitable inspection design while the co discovery learning method with use of heuristics by end users appeared to be the most suitable testing design No suitable inquiry design on objective measurement is found in literature The PSSUQ Post Study Satisfaction of Usability Questionnaire appeared to be the most suitable inquiry method to measure subjective aspects on satisfaction This questionnaire can be used in a testing environment as well as online To measure whether information is found up to date extra items should be added to the design For the validation of the upload tool and the rating tool an online questionnaire has been developed In developing the questionnaires the following points should be considered When using surveys to carry out research an average of all information and phenomena is made Open questionnaires are used to get most information reflections and experiences of people The number of participants should be at least 10 Baarda 2005
17. the task on equal level Top findings After the break the participants and also the researcher are asked for One top positive finding One top negative finding One top surprise aha moment Pencil white paper These findings are mentioned on a white sheet of paper Questionnaire PSSUQ After the test a questionnaire for measuring user satisfaction PSSUQ will be filled in Check if there are further questions and if all topics are filled in by the participants Hand over participation gifts After conducting the test thank and say some ideas Book and say good bye good bye to the participant and hand over Vouchers participation gifts Chocolate Wine Flowers Send all Translate when necessary and sent the documents Written documents can be documents and a sae scanned or faxed other results video s to Netherlands Deliverable D7 1 06 06 2013 Annex 2 pg 7 20 ICT PSP ETNA project 270746 ETNA Thematic Network on ATs Informed consent Statement of agreement Please read this page carefully In context of the ETNA project we ask you to participate in evaluating a new database By participating in this evaluation you will help us to improve the usability of this database We would like to ask you to perform different tasks We are going to record and observe the entire performance of these tasks Furthermore we will ask you to fill in the given questionnaires By signing this fo
18. you will find the following documents Definition of stakeholders Inclusion and exclusion criteria Instruction Manual Informed consent Pre test questionnaire Co discovery questions asked by participants during the performance Task Instruction Plan Observation list PSSUOQO The satisfaction questionnaire Checklist for researchers Deliverable D7 1 06 06 2013 Annex 2 pg 2 20 ICT PSP ETNA project 270746 Definition of stakeholders This definition is given in the Deliverable 2 2 of the ETNA project End users End users look at the portal mainly from a costumer s perspective They need information about assistive technologies because of changes in their functions they may have difficulties in executing activities their functions may have decreased or their environmental context may have changed Because of these changes a need to search for new assistive technologies arises A similar need may arise by family members or caregivers who are looking for new technology to ensure suitable assistance and to reorganize the families life Professionals Professionals need to be familiar with assistive technology and keep themselves constantly up to date at the evolving market of assistive technologies It is common that end users ask professionals for support by selecting and choosing assistive technology which is most appropriate to their needs Not only for their own knowledge but also to give their clients an appropr
19. 010 Cho Kim Chung amp Chung 2006 ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 11 23 AFEN b INA ICT ISO 9241 11 is found to be a standardized model which is qualitatively the most suitable and the most used However ISO 9241 11 1s product oriented and for the evaluation of the Portal it was important to include a user centered view considering the stakeholders As a consequence incorporating only this model to validate the Portal would not be suitable a user centered model should be used in conjunction with it Both models 1 e Nielsen and ISO 9241 11 decompose usability into subjective concepts satisfaction and objective concepts effectiveness efficiency memorability learnability and errors Nenonen et al 2008 Poldma 2009 Erhantengel 2010 Yengin 2010 Subjective concepts are individually determined and objective concepts depend on the system ISO 9241 11 and Nielsen 1993 refer to the concept user satisfaction as purely subjective and therefore difficult to measure The other concepts are attributable to the system objective and are more easily measurable Table 2 provides an overview of the models ISO 9241 11 and Nielsen 1993 including the corresponding concepts variables and the definition as described in the literature In conclusion the Nielsen model and the ISO 9241 11 are the most suitable models for measuring the usability of the TN database The concepts of usabilit
20. Most of the questions should be semi structured by using open ended and closed ended answers Participants should get the chance to fill in the online questionnaire individually Moreover the questions should relate to their own opinion and experiences For the upload tool the questions were structured along the lines of the already existing instruction manual for using the upload tool This instruction manual was modified for the use of the questionnaire The survey was adjusted several times after having expert comments and a pilot test was conducted For the rating tool it was important that the validation should be about the use of the tool and not about whether or not QUEST is a suitable validation instrument All questions of this online questionnaire were based on the content questions of QUEST and on the usefulness of the rating tool In conclusion the questions were structured along the lines of instruction manual for using the rating tool This instruction manual was altered for the use of the questionnaire The survey was modified several times after having expert comments and a pilot test was conducted 5 How will the validation be embedded into the ETNA project What are the contents of the plan the timeframe the methods and tools and how should the results be analyzed The plan consists of three parts ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 19 23 SHS i ETNA ICTpsp Thematic Network on ATs
21. PI aN m ETNA ICTesp Thematic Network on ATs DELIVERABLE Project Acronym ETNA Grant Agreement 270746 number Project Title European Thematic Network on Assistive Information and Communication Technologies D7 1 Plan for Validation of the TN portal Due date of delivery 31 05 13 Actual submission date 06 06 2013 Start date of project 01 01 2011 Duration 36 months Organisation name of lead contractor for this deliverable Hogeschool Zuyd HZ Project co funded by the European Commission within the ICT Policy Support Programme Dissemination Level Confidential only for members of the consortium and the Commission Services ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 1 23 PI LAN z ETNA ICTPpsp Thematic Net Revision History Revision Author Organization Description First Draft Second Draft Statement of originality This deliverable contains original unpublished work except where clearly indicated otherwise Acknowledgement of previously published material and of the work of others has been made through appropriate citation quotation or both Authors Main authors Jeanne Heijkers Gert Jan Gelderblom HZ Contributors Students Hogeschool Zuyd HZ e Dori Mertens e Fabienne Bonten e Marjolein Oosterheert e Lisa Henning e Corinna Laufer e Laura Neubauer e Tamara Steffens Project co ordination team FDGCO e Renzo Andrich e Sabrina Vincenti e Valeri
22. TISAaN I nstruction to test the Upload Tool The upload tool gt The Upload tool is a web based interface for entering managing Res 1 Res 2 data in the EASTIN Central Database Webservice Will be used by Information providers that are not willing to implement Web Services to connect their resources Provider B members of the Community willing to make information on resources they have found online in Provider C and D websites available through the ETNA system Provider A In Coordination with gIanssan I Deliverable D7 1 06 06 2013 Annex 3 pg 1 10 ICT PSP ETNA project 270746 www etna project eu Go to www etna project eu Curopean Thematic Network Click on the button on Asestive information lechnooqies ETNA search 9 engine trial version Brief presentation hass Surg aes to KT aste suonet A European network of beading Institutions mess ad aes snt wagta Arg assistive tedinokoay and eauessh ty Emam m anng podem s paty mt wel JOA o mo tart Pe rested rowed ro s 3 BIO sorie zen the hr of few gedhi tired Teo i G Lunia Sine eT at Rat GOO eeu Bagge da maiis e ala reia cloner paling u miih a web pota of ETid a by anti pek eral or rede orbire aani paih nabiy aii and netted irit Bhp ETUA Thra Mereni an ae hrg i LA n hoon ath ote Tenga tte AA EMA 6 a Mwa who arg In Coordination with Ey aris4at I Upload Tool how it works Log in using the European Inf
23. are suitable for validating usability of databases and which of these variables are applicable to the features of the Portal according to the literature An orientating literature search was performed by using the PICO method The PICO method contains a search question with the different types of stakeholders population system use database intervention and usability outcome Concepts sub attributes were extracted from the literature and results were sorted by quality Three reviewers evaluated the findings in a final selection round Ib Which variables are suitable for validating usability of databases and which of these variables are applicable to the features of the TN database in the opinion of all stakeholder groups Focus groups and semi structured interviews were held in order to answer the question above The methodology of focus groups was chosen because consensus can be accomplished on the theme However a pre condition is that there should be a certain level of homogeneity within a focus group Lukassen amp Olde Hartman 2007 Homogeneity in this study was realized by executing focus group meetings for each stakeholder group whereby consensus was accomplished within the individual stakeholder groups Each group should consist of a minimum of 4 and a maximum of 9 participants Nielsen 1994 Barbour 2007 In addition to focus group meetings semi structured interviews were held to provide additional contributions since a l
24. arger level of immersion can be realized on the specific theme and because one should be able to delve deeper into the individual wishes and demands of the stakeholder Kvale 2007 Participants For the recruitment of participants the stakeholders categorization provided by Deliverable 2 2 of the ETNA project was used Stakeholders are end users professionals suppliers and manufacturers developers researchers and policymakers In addition to that participants should be familiar with ICT AT Both suppliers and manufacturers were approached by email and were asked to participate in the research Professionals were approached in the Netherlands via contacts at Hogeschool Zuyd School of Occupational Therapy and School of Speech Therapy In addition flyers were distributed at central meeting points at Hogeschool Zuyd and in nursing centers Social media such as LinkedIn and Facebook were used to discover organizations involved in ICT AT Researchers at several Dutch universities and knowledge institutes were approached to participate directly via email Policy makers ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 8 23 SHS ETNA ICTpsp Thematic Network on ATs were approached at micro meso as well as macro level For instance municipality counselors and staff working with local and central government bodies were invited Overall approximately 330 stakeholders were approached Data collection by means of f
25. ble D7 1 06 06 2013 pg 6 23 SHS ETNA ICTesp Thematic Network on ATs e The Rating and Comments tool is added to the system for generating a novel type of information produced by end end users while using the database end users can rate products in the database on the basis of their experience This rating tool is a modification of the QUEST Instrument Quebec User Evaluation of Satisfaction with Assistive Technology Demers et al 2000 This tool is a new feature with specific characteristics thus it will be validated by means of specific methods Overall goal The goal of the work described in this deliverable is to develop a plan for validation along with the related methodologies suitable for checking the extent to which the ETNA Information System is actually usable by the intended target groups stakeholders of the ETNA project end users professionals suppliers researchers policy makers The following question needs to be answered Which methodology will be suitable for measuring the usability and contents in five months regarding ICT AT in the Web Portal for each stakeholder group To answer this question the following sub questions were formulated 1 Which variables are suitable for validating usability of databases and which of these variables are applicable to the features of the ETNA Information system a According to the literature b In the opinion of stakeholder groups 2 Which measurements in
26. ble time Does every display begin with a title or header that describes screen contents Is there a consistent icon design scheme and stylistic treatment across the system Is a single selected icon clearly visible when surrounded by unselected icons Do menu instructions prompts and error messages appear in the same place s on each menu In multipage data entry screens is each page labelled to show its relation to others If overtype and insert mode are both available is there a visible indication of which one the user is in If pop up windows are used to display error messages do they allow the user to see the field in error Is there some form of system feedback for every operator action ea After the user completes an action or group of actions does the feedback indicate that the next group of actions can be started Is there visual feedback in menus or dialog boxes about which choices are selectable Is there visual feedback in menus or dialog boxes about which choice the cursor is on now If multiple options can be selected in a menu or dialog box is there visual feedback about which options are already selected Is there visual feedback when objects are selected or moved Is the current status of an icon clearly indicated Deliverable D7 1 06 06 2013 Annex 1 pg 1 1 ANNEX 2 ICT PSP ETNA project 270746 Annex 2 Instruction and questionnaire co discovery learning method Instruction
27. covery learning Nielsen 1993 Performance measurement Nielsen 1993 Soken et al 1993 Retrospective Testing Nielsen 1993 and Shadowing Design Nielsen 1993 are suitable testing methods According to Wilson amp Blostein 1998 and Haller et al 2000 co discovery learning has proven to be a very valid and effective design tool Both inspection designs in combinations with the co discovery learning method are proven reliable Hartson et al 2000 Two instruments appear to be valid and reliable Lewis 1995 to measure subjective elements they are the QUIS Questionnaire Usability Instrument Satisfaction and the PSSUQ Post Study Satisfaction of Usability Questionnaire After consulting with several experts in the field it is concluded that the PSSUQ is the most suitable questionnaire Conversely no valid and or reliable instrument for inquiry was found All aspects as mentioned in table 2 are covered except for up to date information To cover this item as well one item will be added which measures the degree of satisfaction concerning up to date information 3 Which measurements instruments designs can be found in the literature for measuring usability of an upload tool Which can be considered to be the more reliable and valid ones for the Portal No literature was found on suitable instruments or tools to evaluate the upload tool This means that a research tool has to be developed The best way to inv
28. d to the effectiveness of the system Learnability and memorability are mentioned by Nielsen 1994 as the level of effort needed respectively for the first use and the re use of the system As this can be interpreted quite subjectively components explicitly indicated by stakeholders are used Completion of a task as mentioned by Nielsen 1994 in connection with the concept errors is especially associated with the number of error messages generated by the system The variable under this concept is therefore only indicated as low error rate Table 3 offers a combination of results found in the literature and the results obtained from the stakeholders This overview presents concepts variables and definitions suitable for the features of this Portal This information will be used to further define the ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 15 23 PI Lan gt ETNA ICTpsp design of the field test Effectiveness Speed ISO Performance measured in time to get from A to B in het Effort expense contributed to the fastest way TATANO GA NNA N Easy ISO The ease of information retrieval and the amount of time related required to do this Accuracy In how far the retrieved information matches the original query Filtering Availability of filtering out irrelevant information Up to date Availability of the most recent settings and options database Structural Options for hierarchical structuring
29. e ETNA project This plan focuses on the first and the second release of the ETNA Information System respectively expected in February and July 2013 in order to assess its effectiveness in meeting all stakeholders information needs and to work out recommendations for possible improvements to be implemented in the final release expected end 2013 This work is part of Work package 7 Field tests validation and launch it will provide an important contribution to the technical and contents development work that has been carried out in the ETNA project so far A first concept of the validation plan was outlined by the WP7 leader and discussed with the whole Consortium at the ETNA Workshop held in Linz Austria on July 9 10 2012 An advanced concept was developed in the period from September 2012 until the end of January 2013 and presented at the following ETNA Workshop held in Milano Italy on February 21 11 2013 After the first release of the ETNA portal became available in mid February all details of the validation plan were defined including the tools and the methods to be used as reported in this deliverable The test validation activities are focused on the ETNA Information System especially in relation to three components the search interface the data upload tool and the user rating and comments tool In the first chapter the background and the goals for the development of the plan are detailed including the compone
30. en planned for the first and the second release of the Portal in order to assess its effectiveness in meeting all stakeholders information needs This work is executed within Work package 7 Field tests validation and launch and will consist of two steps As a first step a validation plan has been established defining the tools and the methods to be used in the field tests and the way in which the results will be processed This plan is reported in this deliverable Second the field tests will be executed according to the instructions provided in this deliverable and recommendations for improvement will be inferred based on the test results The results of the tests and the recommendations will be reported in D7 3 Validation Report of the Thematic Network Portal The test activities will be chiefly focused on the ETNA Information System whose development is responsibility of the ETNA Consortium especially in relation to the search interface the data upload tool and the user rating and comments tool These components of the Portal are described in Deliverable D1 6 Search engine and interface 1 release In the project DoW Description of Work the test validation activities had been originally planned in the last semester of the project months 31 36 However in the First Year Project Review March 28 2012 it was recommended to bring them forward so that more time would be available to implement changes to the
31. estigate the usability of the upload tool would be a survey According ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 17 23 SHS i ETNA ICTesp Thematic Network on ATs to Baarda 2005 a survey gives an overview or an inventory of different phenomena related to the research It describes opinions emotions thoughts and experiences of participants with regard to a specific topic 4 Which measurements instruments designs are described in literature for measuring usability of a rating tool Which can be considered to be the more reliable and valid ones for the Portal One article was found in Google and is about a kind of an online based evaluation rating tool named Gulliver Wright 2004 This can be an example for a general evaluation tool but this article is not convertible to our research question In conclusion no actual instruments or tools were found to evaluate the rating tool This means that a research tool had to be developed Again the best way to do research about the usability of the rating tool would be a survey An online questionnaire is found to be the most efficient way ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 18 23 SHS i ETNA ICTpsp Thematic Network on ATs Conclusions This deliverable was intended to formulate a plan for the validation of the ETNA Information System and to report on the work that has been done to achieve this plan In this section
32. ex 2 pg 12 20 ICT PSP ETNA project 270746 Search by manufacturer s name Search by manufacturer Nova Flo the product Nova flo a environmental emergency alarm system Search by manufacturer ITECH ApS the product Compact 4 HD a character reading machine Search by insert date Search for the newest software product uploaded yesterday Search for the oldest software product uploaded last week Search by search field The Spanish partner doesn t have to perform this task Search Dragger in the category Input software ISO code 22 36 18 Search JAWS Screen Reader in the category Special output software ISO code 22 39 12 Search by Company name Search company RehaDat REHADAT Institut der deutschen Wirtschaft K ln wheelchair accessories Search for the case study of Andrea 6 years old Autonomy in daily life for a person with cerebral palsy Gt Info request Find the possibility to write an email to a specific organisation but do not write an email Deliverable D7 1 06 06 2013 Annex 2 pg 13 20 ICT PSP ETNA project 270746 Kind of Stakeholder Policy makers Manufacturers suppliers Professionals Researchers developers Deliverable D7 1 06 06 2013 Where to find Get in touch with other database users Newest developments Search for suppliers Reviews by end u
33. f the internet Other Deliverable D7 1 06 06 2013 Age M F Tablet Smartphone How many hours a day do you averagely make use of internet lt than 1 hour 1 3 hours 3 6 hours gt than 6 hours Spreadsheet like Excel Presentation like PowerPoint Desktop Publishing like PageMaker Other Graphics Data storage 1 e data bases Decision Support Answering E Mails Social networks Annex 2 pg 9 20 ICT PSP ETNA project 270746 Co discovery questions asked by participants during the performance Visibility of system status The system should always keep users informed about what is going on through appropriate feedback within reasonable time Did you see what happened with the system Where Why did you look over there Match between system and the real world The system should speak the users language with words phrases and concepts familiar to the user rather than system oriented terms Follow real world conventions making information appear in a natural and logical order Did you understand what the system said Can you explain it to me Do you see the logical order in this Can you explain it to me User control and freedom Users often choose system functions by mistake and will need a clearly marked emergency exit to leave the unwanted state without having
34. he roles will be changed During the change a five minute break is given and institute It is mostly helpful when participants experience hospitality by offering something to drink and some food Informed consent Pre questionnaire Observation list Co discovery questions Task instruction plan PSSUQ Participants have to match with the given definitions see above Preferably look for 2 policymakers 2 manufacturers 2 developers 2 professionals and 10 end users Ss Firstly look for stakeholders individually Secondly design an invitation email for all participants One for the end users and one for the other stakeholders An invitation in which goal and procedure of research the time of testing for a total of 3 hours and dates is described Also confirmation is asked to the appointment Set multiple data one week where aa participants have the opportunity to participate within the stakeholder group Make sure that materials are available and functional Install and put on the video recording devices Introduce the research team Explain the goal and the procedure of the whole evaluation Before testing an informed consent has to be Informed consent filled in by the participants in order to give permission for conducting test and use video Check if there are further questions Everyone of the participants has to fill in the Pre test pre test questionnaire Check if there are further questio
35. i e Companies Projects and Service Providers and the name Modify Delete In Coordination with gyarisdalt lll Organisation basic data Select the type of organisation among Company Project and Service Provider Enter the basic data name Description address Fields marked with are mandatory eAt the end click next Organisation detail 1 basic dataj Ty Companies oe eS Che POGUE UN Sireen In Coordination with With support of Carlo Gmocchd RIATSAAN I Grener pe pet of ae Deliverable D7 1 06 06 2013 Annex 3 pg 6 10 ICT PSP ETNA project 270746 I SO and I CF codes Enter the I SO and I CF codes to indicate the domains in which the organisation is involved Organisation detail 2 assistive technology domains in M03 Assistive products for respiration ASSISTIVE PRODUCTS FOR TRAINING IM SKILLS ORTHOSES AND PROSTHESES E ASSISTIVE PRODUCTS FOR PERSONAL CARE AND PROTECTION ASSISTIVE PRODUCTS FOR PERSONAL MOBILITY ASSISTIVE PRODUCTS FOR HOUSEKEBPING In Coordination with gI atsan Don t forget to click on the Save button at the end In Coordination with gI ATS 4an I Deliverable D7 1 06 06 2013 Annex 3 pg 7 10 ICT PSP ETNA project 270746 Associated information Associated information includes Articles Case descriptions FAQs Forums Ideas News Regulation To enter an associated info click on New To modify data of an existi
36. ian Pw eas pa d E ariei OF i E geieeiohesn dE i H pahaa ay n ta g 2 z Wbt PEr G Era rast comperey O See Sy bami j s aides A aA L ACE CENTRE HOTSPOT Arala input devices TSO Code 2 na ACE CENTRE HOTEPOTE Manutacturer ACE CENTRE SSS ee rn Insert date 2041 2013 Latest yp F F M Alamgir dyap eas UE I i ri FE m mi Ae ChA ote 3 f 4 ACE CENTRE HOTSPOTS A eer ut ieee devices TEC Coste 2 mui pat beak 2 ee i Pome Fr det ee a a ae oe ey ee by In Coordination with With support of Carlo Gnocchi SANS AAM Greer Deliverable D7 1 06 06 2013 Annex 4 pg 3 6 ICT PSP ETNA project 270746 Please log in with Username licolatal Password ETNA2013 j i ati s4 a Assistive Toechniciogics and Inclusive collaborative portal gt R Aboul us Home Hoame Login Login form Solutions for All Assistive lechinolo Search Scarch st Research and developrnient 2 Key actors Note he fields with asterisk are compulsory User Pusswerd Revs irg fewr Sy ATIS 4all E Tha Ory cre pconedioncel kon wil bi Ill prin ory a ee pro oi l pe eer heed ipo ie T RS ore wees ha bierg querions mic Pra oer ce a ea oa E i ieee m Oo rod Twad FEF ee een Got eu a ay E Thana ior au oron Predict icin Coco bee 1 Sum lag barre pre tz pimay wing fea wii ive piei Deriba 2 Wiad ba tt enpre ar petapritire wiih Dea Qon l Mow sabehed are ye
37. iate advice they need to be up to date with the latest information Manufacturers Suppliers Manufacturers Suppliers may need information on what is already available and which persons are other potential candidates for the products They also need a detailed investigation of the business case analysing the size and profile of the potential target population Researchers Developers To initiate and carry out a research the unsatisfied needs have to be emphasized Developers need to know what the available resources for development are They also need to weigh up their prototypes against other existing technologies in order to detect their strengths and weaknesses Both researchers and developers need to exploit results and activate technology transfer processes Policymakers In order to develop new strategies and initiatives policy makers need a good knowledge of the ICT AT environment This includes on going projects market offers involved stakeholders good existing practices possible deficiencies and new needs which occur within the current policy and the legislative framework Deliverable D7 1 06 06 2013 Annex 2 pg 3 20 ICT PSP ETNA project 270746 Inclusion and exclusion criteria The described inclusion and exclusion criteria in general apply to all kind of stakeholders named below Inclusion in general for all stakeholders Multitasking are able to perform two tasks at the same time Have knowledge about usi
38. ile performing stressed the tasks General Observation of the user gt Body language gt Facial expression gt Vocalization PSSUQ The satisfaction questionnaire The Post Study System Usability Questionnaire Items Please read each statement carefully and indicate how strongly you agree or disagree with the statement by circling a number on the scale If a statement does not apply to you circle N A Please write comments to elaborate your answers 1 Overall I am satisfied with how easy it is to use this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 a 6 ri N A 2 It was simple to use this system STRONGLY STRONGLY AGREE DISAGREE 1 Z 4 4 F 6 i N A 3 Icould effectively complete the tasks and scenarios using this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 a 6 r N A 4 I was able to complete the tasks and scenarios quickly using this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 a 6 r N A 5 Iwas able to efficiently complete the tasks and scenarios using this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 gt 6 r N A 6 I felt comfortable using this system Deliverable D7 1 06 06 2013 Annex 2 pg 17 20 ICT PSP ETNA project 270746 10 11 12 13 14 15 16 17 STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 a 6 r N A It was easy to learn to use this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 a 6 F N A I believe I could bec
39. ive products in general e extra resources added to the already existing EASTIN system such as organisations companies projects service providers and additional information ideas literature etc e extra facilities such as the data upload tool and the user rating comment tool Setting the stage for the development of the validation plan As stated before it was recommended in Brussels at the 1 annual review of the ETNA project to have the validation earlier in the process This was discussed within the Consortium and its management team A first concept was prepared by the WP7 leader and presented at the ETNA Workshop held in Linz Austria on July 9 10 2012 where the propositions made were discussed and recommendations ideas and comments from partners were taken into account The timeframe was discussed within the management team and the partners from both the ETNA and the ATIS4all consortia Regarding the timeframe it was agreed that e the actual start of the testing activities would only be possible after completion of the technical work for the first release of the ETNA information system e as the idea was to involve students the study schedules of Hogeschool Zuyd needs to be considered e Sufficient time should be allocated to the partners involved in the field test in order to be able to execute their evaluations time wise As a result from September 2012 until the end of January 2013 a team of students and e
40. luation design is the most appropriate design to test usability Dillon 2001 Within usability research and within qualitative research mainly three types of techniques are discussed to evaluate a website on usability These three classifications inspection testing and inquiry each have their own designs again Nielsen 1993 Ivory amp Hearst 2001 Subjective concepts are individually determined and objective concepts depend on the system ISO 1998 and Nielsen 1993 refer to the concept user satisfaction as purely subjective The other concepts are attributable to the system objective and are measurable The two categories objective and subjective measurements can be used to provide a user centered view on a usability evaluation of a database In order to test the usability of the TN Portal both categories have to be taken into account An analysis follows of which instruments from the three classifications fit the inclusion criteria in relation to the objective measurements Heuristic Evaluation Nielsen 1993 and Scenario Based Approach Nielsen 1994 are suitable inspection methods Hartson 2000 mentions the heuristic evaluation as the most valid and reliable method of all evaluation methods The scenario based approach is more expensive complicated and labor intensive Furthermore there is not much evidence about the quality of this methodology because it is not often used Co dis
41. mparison between NLP interfaces and traditional visual GIS interfaces Procedia Computer Science Volume 5 2011 Pages 920 925 Carbonell N amp Stephanidis C 2003 User Interfaces for all LNCS 2615 pp 1410152 Cho H Kim K I Chung M amp Chung Y 2006 Effective semantic Web services discovery using usability Advanced Communication Technology 2006 ICACT 2006 The 8th International Conference Volume 3 Demers L Weiss Lambrou R Ska B 2000 tem analysis of the Quebec User Evaluation of Satisfaction with Assistive Technology QUEST Assistive Technology 2000 12 96 105 Dillon A 2001 Usability evaluation In W Karwowski ed Encyclopedia of Human Factors and Ergonomics London Taylor and Francis Dumas J S amp Redish J C 1993 A practical guide to usability testing revised edition ISBN 9781841500201 Portland Oregon Intellect Books Erhantengel 2010 Conducting preliminary steps to usability testing investigating the website of Uluda University Grani A Glavinic V amp Stankovy S n d Usability Evaluation Methodology for Web based Educational Systems Faculty of Natural Sciences Mathematics and Education University of Split Nikole Tesle 12 21000 Split Croatia Guillemette R A 1989 Usability in computer documentation design conceptual and methodological considerations IEEE Transactions on professional communication 32 4 Hartson H E Andre T S amp Willis R C 2000 C
42. n be considered to be the more reliable and valid ones for validating the Portal A literature review was conducted Inclusion criteria were applied regarding the characteristics of the Portal and the stage of the project These inclusion criteria were cost effectiveness applicability at any developmental stage multi usability and user centrality Where the literature did not indicate a final choice two experts in the field of usability were consulted 3 Which measurements instruments designs can be found in the literature for measuring usability of an upload tool Which can be considered to be the more reliable and valid ones for the Portal A literature review was conducted for which the following keywords were used upload tool evaluation usability satisfaction database multilingual online questionnaire Besides these keywords two PICO questions were formed First Are there any ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 9 23 SHS i ETNA ICTPsP Thematic Network on ATs instruments already available for satisfaction evaluation by using an upload tool and second Which are the most important aspects for users regarding the usability of an upload tool The keywords also were used as synonyms to find different meanings of one keyword The keywords were used in English Dutch and German Literature was searched for in Google PubMed and OT seeker by using threads like AND or OR
43. nd of stakeholders The second table is only for the policy makers manufacturers suppliers professionals researchers developers During the performance of tasks the stakeholders can ask for support any time Explanation of performance Click on the following link to begin with the tasks http www etna project eu and go to on click on the button ETNA Search Engine trial version For the tasks named below the stakeholders have to choose the following languages Netherlands has to choose gt DUTCH Belgium Italy has to choose gt Italian ITALY Spain has to choose gt Spanish SPAIN United Kingdom has to choose gt English UK The researchers have to translate each task in the language of their country It is important to check if every keyword is available Kind of stakeholder Where to find Task to perform For all kind of stakeholders Products Search by ISO codes Search ITECH ClearView PC 22 by ISO code 22 03 18 Search Infralight Swing Stethoset System by ISO code 22 18 27 Search with keywords Search Humanware Communicator a face to face communication app for deaf blind people by using the keywords search Search Voice Sense a portable note taking devices for Braille by using the keyword search communication software Search Powertel 49 Amplified Big Button Corded Phone a Standard network telephone Deliverable D7 1 06 06 2013 Ann
44. ng a computer minimum hour per week People with all kinds of disabilities can perform the usability evaluation as they are able to work with a common computer or they bring along their own computer including devices Stakeholders are able to concentrate to perform tasks for about 3 hours Exclusion in general for all stakeholders People with auditive disabilities which lead to problems in communication People with visual disabilities which lead to problems in making use of a computer People with several cognitive problems are not able to follow instructions are not able to maintain their concentration for about 3 hours People with communicational deficits caused by aphasia massive stutter or any other 1 To give more information about the stakeholders we look for we will give you the following inclusion criteria End users 8x With limitations in motor functions 2x With cognitive limitations 2x With visual limitations if these problems are not prohibitive while using a common computer 2x Caregivers looking for different kinds of aids for family members who are not able to search on their own 2x Policymakers 2x Knowledge about ICT Have to participate in a political or policymaking environment at this moment 2 Manufacturers Suppliers 2x Knowledge about current AT ICT products and future possibilities which is defined in the deliverable 2 2 named above 3 Researchers Developers 2x Preferably knowledge about
45. ng associated info select it from the drop down list and click on Modify To delete data of an existing associated info select it from the drop down list and click on Delete Associated info is listed in alphabetical order of the type i e Articles Case descriptions FAQs Forums Ideas News Regulation and the title by In Coordination with Corto Gm hi Ry ATIS 4all ih Associated info Basic data Select the type of Associated info among Articles Case descriptions FAQs Forums I deas News Regulation Enter the basic data for the associated info title author In Coordination with With support of gyarisdalt_ ll mem boret tae Deliverable D7 1 06 06 2013 Annex 3 pg 8 10 ICT PSP ETNA project 270746 I SO and I CF codes Enter the I SO and I CF codes to indicate the domains of the associated information Organisation detail 2 assistive technology domains in M03 Assistive products for respiration ASSISTIVE PRODUCTS FOR TRAINING IM SKILLS ORTHOSES AND PROSTHESES ASSISTIVE PRODUCTS FOR PERSONAL CARE gum ASSISTIVE PRODUCTS FOR PERSONAL MOBILITY ASSISTIVE PRODUCTS FOR HOUSEKEEPING In Coordination with gI atsan Don t forget to click on the Save button at the end In Coordination with gI ATS 4an I Deliverable D7 1 06 06 2013 Annex 3 pg 9 10 ICT PSP ETNA project 270746 Thank you for your help Your feedback is very much appreciated
46. ns and if all topics are filled in by the participants questionnaire Ask the user if they have any questions Explain the process of the co discovery learning Pencil Instruction plan Questions of co discovery method Participants perform the task instruction plan Task instruction plan i i Heuristi ions While performing the task instruction plan USUC questio one participant is performing the tasks while the other is raising the heuristics questions Deliverable D7 1 06 06 2013 The second part of the evaluation the other Annex 2 pg 6 20 ICT PSP ETNA project 270746 participant is performing the task while the first participant is also asked the same heuristic questions The researcher will intervene when Participants stagnate Participants have no longer equal input Participants pursue wrong goals While the participants switch the tasks a short break of 5 minutes is given Participants are not allowed to talk about the performed tasks i e coffee juice cake and sandwiches Break of 15 minutes After performing the task instruction plan 1 e coffee juice participants have a break cake and sandwiches Observation lists The observers will need an observers list to Observation list monitor in uniform manner during the observation The observer only fills in the observation forms and will note how many helps and in which extend the participants needed to perform
47. nse to the question Suppliers indicated that up to date information is crucial for them The very latest information on new product developments should be provided As far as the search strategy 1s concerned suppliers mention that they would like to search by keywords by disability by product and have more filter options advanced search such as e g gender prognosis and age Registration is considered important when dealing with sensitive information Sensitive records should be associated to a username and password There should be a possibility to be logged in as a company In the preferred scenario one could log in as a company and then have a shortcut on the desktop to link to the database Multilingualism is recommended when dealing with an international database However it is stated that it 1s important to correctly translate technical terms If this cannot be guaranteed the advice is not to translate these texts In this case English is the most common international language Suppliers stated that for instance Google translate is not accurate Researchers Researchers indicated they want to search by means of a search question Also the ability to search according to the ICF classification is desired Researchers want objective and accurate information from suppliers without endless commercial descriptions about products and features The system should be fast and efficient This means that the structure of the database should be
48. nt cick tie Remove button to ran 22 36 12 Dispesitra di ingresso allematra Bobe Hiii MEA 0 Initlic air E Ges If needed you can add optional ISO codes by selecting them in the drop down list by In Coordination with With support of Carlo Gnocchi Ky ATIS 4al lll Qenctuston Deliverable D7 1 06 06 2013 Annex 3 pg 4 10 ICT PSP ETNA project 270746 Primary and secondary features eYou can add features of the product checking the appropriate boxes or indicating the measurments elf a feature you want to add is not among the Primary features you can click on expand secondary features button on the bottom of the page In Coordination with Qretusion practice Pe pet te Don t forget to click on the Save button at the end Deliverable D7 1 06 06 2013 Annex 3 pg 5 10 ICT PSP ETNA project 270746 Organisations Organisation is the term used searches Whatis Eastin The Eastin partners General info to include Companies Projects and Service Providers Assistive Products Organisations Associated Information To enter a new organisation click on New To modify data of an existing Eueu iy organisation select it from the drop down list and click on Modify To delete a data of an existing organisation select it from the drop down list and click on Modifyidelete an organisation Delete Organisations are listed in Select an organisation alphabetical order of the type
49. nts to be evaluated In the second chapter the methods are explained that were applied in the process of designing the methodology for the validation plan In the third chapter the results of the activities are described providing insight into the considerations underlying the choice for the validation methodology The chapter also gives insight on the concepts of usability and on what methods have proved to be the most suitable for the validation exercise In the final chapter details of the validation plan are provided including the required tools and the time schedule ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 4 23 SHS m ETNA ICTpsp Thematic Network on ATs Background Introduction The overall goal of the ETNA thematic network is to establish a European Web Portal offering information on assistive products based on ICT information and communication technologies and e accessibility solutions which are available in Europe and on related organizations and services The Portal is being developed in collaboration with ATIS4AII Assistive Technologies and Inclusive Solutions for All another Thematic Network belonging to the same cluster It will be composed of two main components the information system which is responsibility of the ETNA network and the community which is the responsibility of ATIS4all In order to support the technical and contents development process field tests have be
50. o Gower ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 2 23 PI Lan ETNA ICTesp Thematic Network on ATs Table of contents ME Uy ry AN Yates seas ee sea sce ce site gets es ea ese are cpm econ E AER AE A 4 BAC OG OUI o E E E T E E E T AE EET E EEE EE 5 TAG O Oe e E E A E E E E A EE E E E E E 5 Setting the stage for the development of the validation plan ssssnnsenssssssoeeesssssssseresssssssssseeressss 6 Honc oDe sano W an E OO ee eT Cote ere errr eevee E eee 6 Overa o aa gett E ec se Sate cpg Se ae gets eet a sant eect E E 7 MoO escalate cect gee secede see ee A A E a caeresesteaavensesea 8 JEE N iE AEREE AEE EEEE AT EEEE EEEE EEEE ososeqetoeseqnacsoencerposcecteooseereseeasa 11 Usability measurement models csos6sssccsecatsvanseavuhenncebbocnstavetesatonssauseavatensnebsossusunebasesebsdaesadvetaaweobsetes 11 Findings of the interviews to stakeholders cccccccccccceccececeeeaeesssseeesseeeeceeeeeeceeeeeeeeeaaaaaaeeesees 13 Synthesis of findings from usability literature ccccceeeeeecccceceeeeeeeeaaaaeeesssseeeeeeeeeeeeeeeeeeeaaas 15 6 0S LE IS 6 a AE EE AE E nS re Tre oe ee E E er ere E eee 19 Me aS Te 0 NC EAE E A E ET EEEE 2a ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 3 23 SHS i ETNA ICTpsp Thematic Network on ATs Executive Summary This deliverable reports on the development of a plan for testing and validating the products of th
51. ocus groups and interviews Focus groups were held lasting approximately two hours including the collection of individual viewpoints and the group based categorization of all data During the focus group individual participants were asked which variables they thought relevant for assessing usability of the portal The additional interviews held were semi structured and items were derived from the ISO 9241 11 Standard which provides a definition of usability and from Nielsen 1993 The interviews allowed for further delving into the subjective concepts and the aspects of user centered design as defined by Nielsen 1993 In total six interviews and three focus groups were executed either at the participants home or at Zuyd University Table Stakeholders Focus group N 10 Interviews N 8 End users 4 Professionals 4 Suppliers 2 2 Policy makers 2 _ Researchers Developers 4 Table 1 Number and type of participants Data analysis focus groups and interviews Each interview and focus group was transcribed and analyzed as described in Baarda et al 2005 After the interviews had been coded and peer reviewed among the several researchers the information obtained was analyzed Both variables from the literature and from the interviews focus groups were defined nominal descriptive value and then merged 2 Which measurements instruments designs are described in the literature for measuring these variables Which ca
52. of eado miiu i et Dechy and Acc l Gma pap fe maia a pill g nia iaa fhe Ne pria d O bart ambye beUiraiog jy ombbind aba n bba irpo prety aT 2 The ThA Theat leer ae at edie Gee ee a The ime de BT deren G mates oe pea ee ben a ap S Bibe ai Emau eea ET a SO i Ti ew a Dba IL jel pmi deve j es et Ty of bh a panja nm hanin ret EO jji Deliverable D7 1 06 06 2013 In Coordination with gyarisdatt Go to www etna project eu Click on the button ETNA search engine trial version GO AT Hill Community fd arin til EWTN Oficis Web Poria 5 Feedgaione cea Carlo Gnocchi nalan W Barston In Coordination with Fondazione emio Smosni EFATISAaN Annex 4 pg 2 6 ICT PSP ETNA project 270746 Click on the button Search by product name Information system trial version poweed by EASTIN i cee comar 2 Look for ese mann em seoa ne your product codon e g HotSpot oes Te Oe eh Diced whee DEH i E nics ETNA PPE Mr Bicik att kiasi p BARTIN abim ee eee COED fear tests E ar In Coordination with With support of emionoesi EJATS4al I G Click on i e ACE Centre HOTSPOT2 Click on the akan wre 2 button Review Saacth primma Canne ia names hotspot Ifyou do not wish to nee information from any of the national datebaces than nenerse the Ook above lts mane by clicking on H th S P rod u ct and then clicking on the Refresh run
53. ome productive quickly using this system STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 2 6 F N A The system gave error messages that clearly told me how to fix problems STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 7 N A Whenever I made a mistake using the system I could recover easily and quickly STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 7 N A The information such as on line help on screen messages and other documentation provided with this system was clear STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 2 7 N A It was easy to find the information I needed STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 7 N A The information provided for the system was easy to understand STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 z 7 N A The information was effective in helping me complete the tasks and scenarios STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 b 7 N A The organization of information on the system screens was clear STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 7 N A The interface of this system was pleasant STRONGLY STRONGLY AGREE DISAGREE l Z 4 4 F 6 r N A I liked using the interface of this system Deliverable D7 1 06 06 2013 Annex 2 pg 18 20 ICT PSP ETNA project 270746 STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 6 ri N A 18 This system has all the functions and capabilities I expect it to have STRONGLY STRONGLY AGREE DISAGREE 1 2 3 4 5 6 ri N A 19 Overall I am satisfied with this system
54. onsidered to be important that products can be reviewed by users Multilingualism is desirable however technical terms should be correctly translated Suppliers manufacturers As well as the other stakeholder groups suppliers mentioned instruction as an important variable They would like to have instructions on how to use the website more ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 13 23 SHS i ETNA ICTpsp Thematic Network on ATs specifically on how the database search facility works in order for them to use the website as adequately as possible They would be pleased if the search results were presented in the same way as they are presented when using Google The results should be found quickly Also the results should be shown clearly and the keywords must be written in bold in a brief caption Too much text about the products should be avoided Additional information should be obtained by clicking on certain resources The situation in which errors appear and no results are found is perceived as very annoying Accuracy is defined as the ability to find products quickly say in no more than three clicks According to suppliers a good way of visualizing the results is to display information based on tabs or various columns In their view this means that there are no text blocks In general suppliers focused on the value of the information The information provided should be complete it should give a 100 respo
55. ormation System on a Change language following powered by EASTIN M credentials Ausitte Produet gt Username What is Eastin The Eastin parmers General info fakeUserl Assrinve Products Organisations Associated Information Submit a product Info regust ETN A Assistive ICT products gt Password aaa 53 Guided search by S0 product classification codes ee SAR Searches WhatisEastn The Eastnparners Generalinio MEZES G 0 to th e U p Od d a Search by comm ae i Assiste Products nformaben taip too pa g e gt Search by manuf Search by inserti insert a new product Advanced search New Modifyidelete a product In Coordination with gyarisdaut ll Deliverable D7 1 06 06 2013 Annex 3 pg 2 10 ICT PSP ETNA project 270746 Upload tool The tool allows to enter 1 Assistive Products please upload at least one product 2 Organisations Companies Projects Service providers please upload at least one information source in one of the three categories above 3 Associated Information Articles Case descriptions FAQs Forums Ideas News Regulations please upload at least one information source in one of the seven categories above 4 Modify an existing product Please modify at least one of the products you have uploaded 5 Delete an existing product Please delete at least one of the products you have uploaded and put it back again Fondazione Carlo Gnocchi Onlus Assi
56. perienced users Allow users to tailor frequent actions e How can Iuse the system in the most efficient way Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility e How do I get the most relevant information Explain Help users recognize diagnose and recover from errors Error messages should be expressed in plain language no codes precisely indicate the problem and constructively suggest a solution e Do you understand this error message Is it easy to understand Help and documentation Even though it is better if the system can be used without documentation it may be necessary to provide help and documentation Any such information should be easy to search focused on the user s task list concrete steps to be carried out and not be too large e Where can I find the manuals of add product of subject e Do you think this manual is understandable Why Deliverable D7 1 06 06 2013 Annex 2 pg 11 20 ICT PSP ETNA project 270746 Task Instruction Plan To be able to evaluate the usability of the ETNA database the participants inter alia have to perform different tasks on the computer which are named below see table All kind of stakeholders have to perform the tasks which are named in the first table for all ki
57. psp Thematic Network on ATs validation As the Portal is still in a test environment there is no ethical problem to rate a product and permission of suppliers or manufacturers was not needed The instruction manual is shown in Annex 4 The partners will get a written instruction manual on how to rate a product After the rating they will fill in the online questionnaire which includes 27 questions about their experiences opinions and satisfaction about the rating tool The questionnaire is available at the following address http www enquetemaken be toonenquete php id 142120 Timeframe Instrument used Which partner 2013 February March July September oco sstecoceoeors yrocso Table 4 Time schedule validation TN Portal ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 21 23 SHS ETNA ICTpsp Thematic Network on ATs References Abrain A et al 2003 Usability Meanings and Interpretations in ISO Standards Software Quality Journal 11 323 336 2003 Kluwer Academic Baarda D B Goede de M P M amp Teunissen J 2005 Basisboek Kwalitatief Onderzoek Groningen Houten Wolters Noordhoff Barnum C M 2010 Usability Testing Essentials Burlington MA Morgan Kaufmann Barbour R 2007 Doing Focus Groups London United Kingdom SAGE Cali D Condorelli A Papa S Rata M amp Zagarella L 2011 Improving intelligence through use of Natural Language Processing A co
58. riteria for evaluating usability evaluation methods Department of Computer Science Department of Industrial and Systems Engineering Virginia Tech Blacksburg VA ISO 9241 11 Guidance on Usability 1998 Retrieved from http www usabilitynet org tools r_international htm John amp Marks 1997 Tracking the effectiveness of usability evaluation methods Behaviour amp Information Technology Kunjavsky M 2003 Observing the user experience A practitioner s guide to user research Elsevier 1 edition April 22 2003 Kvale S 2007 Doing Interviews London United Kingdom SAGE Lewis J R 1995 JBM Computer Usability Satisfaction Questionnaires Psychometric Evaluation and Instructions for Use International Journal of Human Computer Interaction Volume 7 Issue 1 DOT ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 22 23 SHS ETNA ATs Thematic Network on Fa ICTesp 10 1080 104473 19509526110 Lukassen P amp Olde Hartman T 2007 Kwalitatief onderzoek praktische methode voor de medische praktijk Houten The Netherlands Bohn Stafleu en van Loghum Madan A amp Dubey S K 2012 Usability Evaluation Methods A literature review Noida India International Journal of Engineering Science and Technology IJEST ISSN 0975 5462 Vol 4 No 02 February 2012 Nenonen S amp Storbacka K 2008 Business model design conceptualizing networked value co creation The 2009 Naples Forum
59. rm you will give your permission for the ETNA project to be videotaped and for your given answers to be used We will get important information to make the database more useful for all kinds of users We will neither use your full name nor share your personal data with third parties in any case At this moment the database only exists in a test version and therefore has to be evaluated with your support Any information about the ETNA project has to be treated with care and consideration You may quit this evaluation at any time If you agree please sign here Name Date Signature Pre test questionnaire ETNA Deliverable D7 1 06 06 2013 Annex 2 pg 8 20 ICT PSP ETNA project 270746 Usability Participant Questionnaire I Participant Information Name Email What is your profession II Computer Experience What kind of computer do you use Laptop Fixed computer Other How many hours a day do you averagely work on a computer lt than 1 hour 1 3 hours 3 6 hours gt than 6 hours What type of applications do you use Word Processing like Word Database like Access Project Management like MS Project Multimedia player e g music films What do you use your computer for in general Games and Pleasure Accounting Finance Word Processing Programming Sur
60. rrors Low error rate The number of errors occurring while using the system The error messages from the system to prevent a task from being executed Satisfaction Up to date Availability of the most recent information The wishes of the stakeholders that content encourage the use of the system Table 3 Overview of selected concepts and variables related to ICT AT that have been considered applicable for this Portal 2 Which measurements nstruments designs are described in the literature for measuring usability of databases and which ones prove to be the more reliable and ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 16 23 SHS i ETNA ICTpsp Thematic Network on ATs valid ones to be suitable for the TN Portal Literature shows that multiple research studies have been executed in order to select the best evaluation design for measuring usability of a database However there is no design which was proven to be the best John and Marks 1997 Abrain et al 1999 Dillon 2001 Nevertheless it is important to point out that any kind of usability evaluation will improve the final version of the system as long as its results provide appropriate feedback from which further improvements could be achieved Grani n d Usually multiple designs are needed in order to achieve the best results Tashhakkori amp Teddlie 1998 Nielsen 1993 Dillon 2001 Carbonell amp Stephanidis 2003 Triangulation within an eva
61. s are applicable to the features of the Portal according to the literature 68 articles were found in the pre selection During the second selection round a closer examination of the content took place After evaluation in the final selection round 21 articles remained In these 21 articles 6 models were described all defining usability in a different way These models are reported by Eason Poulson ISO 9241 11 ISO 9126 Nielsen and Shackel In general usability is the extent to which a product or a system enables users to achieve specified goals Usability is the ability of a product or system to effectively and efficiently fulfill the needs and specifications of users Usability is essential for user satisfaction and user acceptance of a product or system It is the measure of the quality of the user s experience when interacting with a product or system Dumas amp Redish 1993 Guillemette 1989 Nielsen 1993 2012 Rosenbaum 1989 Rubin 1994 Shackel 1991 Each model defines usability by means of different concepts sub attributes The concepts of the Eason model Eason 1984 are learning ease of use task match frequency openness user knowledge and motivation Shackel 1991 gives a definition of usability that could be used throughout the system development lifecycle According to his description the system must encompass the following criteria effectiveness learnability flexibility and attitude Madan 2012 Since
62. s wt Deliverable D7 1 06 06 2013 oi r e ae e e mi T car e A a paa Pa er in appia Ber aia br ee pes e w i x ah ra piai P Login gt licm pil irv peas P Fum deed toy ICT PSP programms by Fondazione Carlo Gnocchi Onlus In Coordination with Eyarisdatt ll Within this tool there are 11 questions given 2 oe eel Er eects Pp On Wi ones of quation hor F CeO ee hte P bih wil sriti 1 a alabe ont pom ia piecke Bes gib iin i i d te ec dl er pe meda T y eens gt 10 of these questions are mandatory fields and have to be answered by choosing one of the given options Wt yoa wish please sf yaw bre irit comment beiee gt The last question is optional and open To complete the rating click the button Save by Fondazione Carlo Gnocchi Onlus In Coordination with Eyarisdatt I With support of Q Annex 4 pg 4 6 ICT PSP ETNA project 270746 Delails Assistme Producss Aifemare snout teases OO 223 Manuladinei oC CENTRE INSE Gates Swe sea Lateci update 503472042 Source dalabase Siva T LECINICEE GETAG INNUIT AC JUST ENTS Sr ring speed Tma of seang INPUT CE CES Sw tch L4RKGUASES In Coordination with gI Ansan ll Revi2ws list Assistive Produris Aey ema fin this arinka Overal styrbtics fdous dala thet orovisionally Simulate ow the rating system wE work ater t e ATIS4al network Fas completed tre data eriry systern
63. sers Relevant literature New researches Task to perform Search for a possibility to post a message to another user organization Search for the developments about navigation newest acoustic Search the suppliers of input software BIOMOUSE Search for an end user review of Saw 5 Search for literature of Search for the newest research report Annex 2 pg 14 20 ICT PSP ETNA project 270746 ABS ETNA Thematic Network on ATs Observation list Test person number Age Experience novice beginner competent proficient expert classification clarified below Type of stakeholder end user professional manufacturer supplier researcher policy maker Section Is the person able to start using the system quickly Time to perform all tasks in minutes Number of errors Is the person able to recover from error Rate Notes Yes No La 2a ea 5 1 does not need any help 5 needs much help Independent With help from the co discovery Deliverable D7 1 06 06 2013 Annex 2 pg 15 20 ICT PSP ETNA project 270746 Purposefulness in performing the task partner With help from the researcher 1 2 3 4 5 1 less 5 a lot In which frame of 1 2 3 4 5 Deliverable D7 1 06 06 2013 Annex 2 pg 16 20 ICT PSP ETNA project 270746 mind is the user 1 wel balanced 5 wh
64. stive Products In Coordination with Eyaris4at I With support of To enter a new product click Searches Whatis Eastin The Eastin partners Generalinfo WaheEion on New To modify an existing product select the product from the drop down list and click on Insert a new product Modify To delete an existing product Now select the product from the drop down list and click on Delete Existing products are listed in Modityidelete a product alphabetical order of the manufacturer name and the commercial name Modify Assistive Products Organisations Delete Deliverable D7 1 06 06 2013 In Coordination with gyanssan I Associated Information Help With support of Qeetusion Be pet for SAE NTa anufacturer Tattile srl Name oka2000 Code 8 y Annex 3 pg 3 10 ICT PSP ETNA project 270746 Product basic data Enter the basic data pot de ic ete Commercial name description vv primary ISO code The manufacturer field is not entered manually but selected from the list of existing organisations So if the manufacturer is not already present it must be entered in the organisation section eFields marked with are mandatory eAt the end click next by In Coordination with With support of Fondazione Carlo Gnocchi Deecraston Onlus om Optional I SO codes Product detall 2 optional additional 50 codes for the product Selected optional 0 codes ca
65. struments designs are described in the literature for measuring these variables Which can be considered to be the more reliable and valid ones for validating the Portal 3 Which measurements instruments designs can be found in the literature for measuring the usability of an upload tool Which can be considered to be the more reliable and valid ones for the Portal 4 Which measurements instruments designs can be found in the literature for measuring the usability of a rating tool Which can be considered to be the more reliable and valid ones for the Portal 5 How will the validation be executed in practice within the ETNA project What are the contents of the plan the timeframe the methods and tools and how might the results be analysed ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 7 23 SHS i ETNA ICTpsp Thematic Network on ATs Method In this chapter the methods required for gathering data to answer the research questions are explained In order to answer the first question a suitable definition of usability for the ETNA validation had to be established pinpointing the variables to be included in the validation This was done through literature review la and stakeholder consultation 1b I In order to answer the second third and fourth questions literature reviews were carried out In this chapter the details of the methodology are described for each research question la Which variables
66. to go through an extended dialogue Support undo and redo Can you explain to me how I get one step back of this screen Consistency and standards Users should not have to wonder whether different words situations or actions mean the same thing Follow platform conventions Can you explain to me how you get from a to b Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place Either eliminate error prone conditions or check for them and present users with a confirmation option before they commit to the action How can I get from a to b without errors Can you explain it to me If I do it like this I get an error What did I do wrong Recognition rather than recall Minimize the user s memory load by making objects actions and options visible The user should not have to remember information from one part of the dialogue to another Instructions for use of the system should be visible or easily retrievable whenever appropriate Are the objects easily visible and logically presented Deliverable D7 1 06 06 2013 Annex 2 pg 10 20 ICT PSP ETNA project 270746 e Is it clear where I am going when I click on specific objects Flexibility and efficiency of use Accelerators unseen by the novice user may often speed up the interaction for the expert user such that the system can cater to both inexperienced and ex
67. ts and the screen Three cameras optional while performing the tasks we need three cameras which record each participant through three different angles Set up one video camera in front of the participants one camera on the left side of the participants and one on the right Prepare participants folder Collect all papers of each participant in one Folder for each folder a E participant Code the participant as participant 1 participant 2 participant 3 etc Travel expenses declaration Depends on what is usual in your country The recordings appeared to be very time consuming to set up the recordings to translate and to analyze After a discussion with the coordinator it was decided to leave out the by the partners Only in the Netherlands one camera registered the activities Deliverable D7 1 06 06 2013 Annex 2 pg 5 20 ICT PSP ETNA project 270746 pf and institute Depends on what is usual in your country Participation fee Print all the appendix Looking for participants who fit best with the given definitions in deliverable D 2 2 Advertise participants Design invitation Set dates Conduct Setting up test environment usability test Welcome participants Fill in the Informed consent Fill in the Pre test questionnaire Explain the procedure of the task plan and co discovery method Performance of task instruction plan 1 5 hours in general after 40 minutes t
68. ured for their need e Ability to concentrate on the required tasks for about 1 5 hours In order to be able to perform the testing ICT AT information on the different resources should be uploaded ICT AT products are available in the system though not all the resources as defined in D3 2 are present in the system Part 2 The upload tool It was decided by the management team that the upload tool will be evaluated by all 23 partners of the ETNA project because information needs to be uploaded before the start of the validation of the search engine as described in part 1 Each partner should upload one ICT Assistive Product one ICT resource in the Organizations section and one ICT resource in the Associated Information section The instruction manual is shown in Annex 3 The questionnaire is available online at the following address http www enquetemaken be toonenquete php id 1394977 In order to have valid information about satisfaction on the use of the system and the content approximately 90 subjects within the widest possible range of stakeholders should evaluate the system by using the revised PSSUQ linked as an online questionnaire Therefore the questionnaire should be translated into all the native languages Part 3 The rating tool For the rating tool the management team decided that all partners would do the first ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 20 23 SHS z ETN ICT
69. verview of suppliers ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 14 23 IHS Thematic Network on ATs 100 00 90 00 80 00 70 00 60 00 50 00 E manufacturers su 40 00 ppliers N 2 30 00 E policymakers 20 00 N 2 10 00 0 00 E end users N 4 Figure 1 Overall outcomes of the interviews Synthesis of findings from the usability literature and focus interviews The variables found were divided into objective variables and subjective variables The usability concepts found in the literature search are used as pillars to arrange the variables Many variables were found in the literature and these correspond to variables mentioned by stakeholders These variables are shown in Table 3 below The ICT AT context requires localization of these concepts and variables Compared to a database with non ICT resources a database of ICT resources requires different search paths Therefore some variables mentioned by the stakeholders and or found in literature are not adequate for this project Hence a choice has been made in the selection of the variables which don t correspond to the concepts of ISO 1998 and Nielsen 1993 ISO 1998 for example places speed of the system under effectiveness while Nielsen 1994 puts it under efficiency Since effectiveness is defined as the amount of effort needed to achieve the aim and the speed contributes to the level of effort needed speed is couple
70. xperts at Hogeschool Zuyd worked at the design of the field tests This work and its results are explained in chapter 2 and 3 At the ETNA Workshop held in Milano Italy on February 21 22 2013 a first draft of the validation methodology was presented The content was discussed with the management team and all the partners A final timeframe and the content for the field test was decided upon The first release of the ETNA Information System became available in mid February 2013 It is accessible in test environment through the ETNA project website Items to be validated As decided at the workshop in Milano February 2013 the items to be evaluated are e The Web Portal with information on ICT based Assistive Technology products available in Europe and related information such as organizations projects literature etc This means that only contents related to ICT based assistive technology will be evaluated rather than all contents provided by the databases of the existing EASTIN network e The Upload tool which gives authorised organisations the opportunity to directly upload their information resources The upload tool is part of the first release of the ETNA Information System however due to its different characteristics it will be treated with specific methods and timing it is necessary that the upload tool is used first so that there is information present in the database to be evaluated ICT PSP ETNA project 270746 Delivera
71. y according to these models are efficiency effectiveness satisfaction memorability learnability and errors Variables of usability are completion of a task speed ease consistency re use level of effort and first use level of effort Table 2 Model concepts Variables Definition Efficiency Completion of a Accuracy and completeness of the task specified goal Effectiveness Speed Reaching a target in het fastest way Ease Extent how easy information can be retrieved and how much time is required to retrieve the information Satisfaction Subjective Experience in a specified context of use Efficiency Speed Performance in time high level of productivity Memorability Learnability Errors Satisfaction Re use Level of effort First use Level of effort Completion of a task Subjective The user can predict what the system will do in any given situation The extent in which the person is able to use the system when re using the system Any action that does not accomplish the desired goal Experience of the 4 other components Table 2 Overview of selected usability models and their concepts ICT PSP ETNA project 270746 Deliverable D7 1 06 06 2013 pg 12 23 SHS ETNA ATs Thematic Network on Fa ICTesp Findings of the stakeholder interviews Ib What are domains variables for usability of databases and which domains variables are applicable to the features of the Portal

ETNA Deliverable D7.1 (Plan for Validation of the TN portal) (PDF

Contents

Download Pdf Manuals

Related Search

Related Contents