Home

D6.2 – ANNEX 1 – Validation testing and usability results

1. si Competence Bullding The A Metwork for Lifelong Competence Development Project no 027087 TENCompetence Building the European Network for Lifelong Competence Development Project acronym Integrated Project TENCompetence Thematic Priority 2 4 10 D6 2 ANNEX 1 Validation testing and usability results Due date of deliverable 31 05 2008 Actual submission date 12 01 2009 Start date of project 01 12 2005 Duration 4 years University of Bolton Version 1 0 Project co funded by the European Commission within the Sixth Framework Programme 2002 2006 Dissemination Level IC S e Restricted to other programme participants including the Commission Services A Restricted to a group specified by the consortium including the Commission CO Confidential only for members of the consortium including the Commission Services ya Building the European Network For Lifelong Competence Development Competence Building The European Network for Lifelong Competence Development TENCompetence IST 2005 027087 D6 2 Annex 1 Validation testing and usability results Work package Task Date of delivery Code name Type of milestone Security distribution level Contributors Authors Partner Contact Person WP Task responsible EC Project Officer Abstract for dissemination Keywords List WP6 Learning Activities Units of Learning Contractual 31 05 2008 Actual 12 01 2009
2. xo E competence TENCompetence ID 6 5 Validation testing and usability results 1 resource management 2 the structure and concepts of IMS LD All but one of the transcripts of discussions were all classified as relating to one or other of these issues This is a strong finding of the evaluation and provides not only insight into the challenges of the authoring process but also a guide for further improvement in the application This is also borne out in the problems identified in the free text responses for which seven were related to management of resources e had slight problems understanding resources and properties as it was all getting mixed up I think separating most of the resources in a slightly easier way could well better help such as separating pic or links and storing it in one place rather than changing name in 2 different places e Would be difficult when organising a large amount of activities need folders to group files e After you create an environment and then add resources etc it was difficult to remember the name or what environment you are currently working in e think there should be a slight more clarification over resource management and properties e Felt uneasy about did I save it or didn t I e really struggled with the environment and resources Also the fact that a service is just a resource was confusing also e Organiser is it possible to have folders collapsible things in the organi
3. Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 Usability would be enhanced if users could edit resources directly without having to worry about whether files are being over written and other similar file management problems Ideally such issues should only be visible when carrying out resource management tasks 3 Names of resources in properties Evaluator User Evaluator User Comment Recommendation 4 Different kinds Evaluator Can you sum up what just happened I added a forum widget and I wanted to give an instruction to the people using the forum That instruction became a resource So I put in a title introduction to forum and I edited the text but then in the resource column it said resource 2 I wanted to rename it in the properties window and I couldn t So I had to go to the resources When I came to rename it in the resources it told me I already had a file of that name but I didn t because it wasn t in the list already It seemed to be taking the fact that I had given it that title to be the name of the file and it wouldn t let me duplicate it So the impression that you have is that the title of the resource in the properties was conflicting with the name of the resource in the resources Obviously I couldn t use that because its got a space in it so whether that was it but it didn t say that wasn t a valid name it said you
4. 1 Validation testing and usability results xo E competence TENCompetence ID 6 5 Validation testing and usability results flagging when an activity has no environment associated with it The planned inclusion of a template mechanism will also make it easier for users to get to start work with the system although it should be remembered that use of templates gains ease of use at the cost of flexibility There is a difference between the two principal issues identified The aim in the management of resources is straight forward to produce a system which is clear and easy to use The aim in clarifying the structure and concepts of IMS LD is less simple The authoring of sophisticated pedagogical plans is a complex matter even with pencil and paper and an authoring application cannot protect authors from this challenge Consequently the goal is to make this task as easy as possible by identifying those aspects of IMS LD which create barriers in this process and by providing support in the application which reduces or removes those barriers However it should not be expected that any application can or should seek to remove the basic intellectual task of planning pedagogy and so some degree of conceptual challenge will remain in any system such as ReCourse 4 3 Detailed results In this section we provide the following detailed results of the evaluation 3 1 The consolidated results of the Likert scale section 3 2 Responses to the fre
5. D6 2 Version 1 0 Draft L Final Xl Report Public Atenas Georgiev David Griffiths Davinia Hernandez Javier Melero Krassen Steffanov Mar Perez Paul Sharples Phillip Beauvoir Scott Wilson Tim Sodhi Yongwu Miao UB OUNL SU FBM UPF GIUNTI David Griffiths UB David Griffiths UB Mr Martin Maj k This is an annex to TENCompetence D6 2 It contains reports from evaluation activities carried out with the TENCompetence Learning Design Toolkit in Liverpool Sofia and Barcelona The focus was principally on authoring and publishing Units of Learning and the results indicate that while working with IMS LD is demanding for teachers and learning designers it is by no means impossible The results indicate that users with pedagogical experience who are also competent users of information technology but not technical experts can become independent users of the applications if they are given four to eight hours support The evaluations showed that the creation and management of resources was a particular difficulty for users and the insights provided by these evaluations led directly to major changes in the Toolkit IMS Learning Design Authoring assessment specification run time IMS LD services Units of Learning ReCourse Widget server TENCompetence Project Coordination at Open University of the Netherlands Valkenburgerweg 177 6419 AT Heerlen The Netherlands 31 45 5762624 Fax 31 45 5762800 xo E compet
6. The problem can be overcome by means of a link tool enabling users to access to UOL and to register on it themselves 4 2 3 Conclusions The results of the Likert scale questions indicate that the respondents could get to grips with the application with only a two hours of demonstration and presentations Thus it seems clear that within the period of the workshop almost all participants were able to reach the point where they could start working independently with ReCourse The results of the questionnaires suggest that while most respondents found the authoring task challenging there was considerable variation from user to user in the areas of the application with which they had difficulty However transcripts of conversations with users strongly indicate that in most cases the underlying problems in most cases were twofold l resource management 2 the structure and concepts of IMS LD The insight into the nature of users difficulties with resource management provides valuable input for the revision of the ReCourse user interface particularly with reference to management of resources and changes to the application are already underway The difficulties with the structure and concepts which face users can be addressed both by appropriate improvements to documentation and in some cases by the provision of feedback on the state of the Unit of Learning under construction e g TENCompetence IST 2005 027087 Page 15 48 D6 2 Annex
7. The user seems to be confused and think that learning objects should be resources associated with a learning activity description The evaluator was not clear about the function of a resource associated with a Forum a There is an ambiguity in IMS LD about where learning activity instructions should be placed in the activity description or in the associated learning object b If possible it would be advantageous to indicate the distinction between resources to be used by the learner and resources used to describe the activity perhaps by changing terminology TENCompetence IST 2005 027087 Page 24 48 xe PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 5 Problems in creating resources and understanding the process User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator TENCompetence IST 2005 027087 I want to save this under a different file name so I don t overwrite this pre existing learning resources can I do that from here Which learning resource This one this learning resource already existed Why do you not want to overwrite it Because it might be something I don t want to change So would I be able to change the file name from here No Because I we
8. Usability would be enhanced by including a message to tell the uses you can t delete this because every learning object must have at least one learning resource 7 Request for an inspector providing feedback on where elements and resources are being used User Evaluator User User 2 Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User TENCompetence IST 2005 027087 As you are completing it there is not like a list to say like this is what you have put in You have done these tasks now do these tasks Not necessarily when we are in the environments we went into them and when you were like say editing we would forget what the environment was called There was no list of things It was like when the thing popped up at the top wasn t it and it came up like that and I got distracted momentarily and I thought which one are we in I couldn t remember whether I was in the intro or the discussion thingy Obviously the tab thing obviously means something but I wonder if it could have the actual name If you had given it a name Well we did we couldn t remember that we were there It was this you were editing wasn t it So that s the name of the resource That s just telling you the file that you are editing back here If you had built 1t the other way round with a phrase or something And given this a name which was more like these then you d know I m
9. associate the activity with the environment Yes that makes sense Partly because lots of different activities use the same environments so you don t want to duplicate it all over the place The concept of environment is not intuitive to these users but they do not have trouble understanding it once it is explained to them TENCompetence IST 2005 027087 Page 31 48 FAA PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 Recommendation The authors of ReCourse documentation should be aware that this is a problem for users as should organisers of workshops The environment element is a basic part of IMS LD and it is difficult to dispense with it in the interface This may sometimes be achieved by creating a default environment for a new resource but this creates the problem of a generating as many environments as there are resources which is difficult to manage b makes it harder for users to make use of the clarity and reusability which organising resources and services into environments brings to IMS LD Consequently hiding the use of environments in the interface is a strategy which should be used with care 12 What is a learning objective User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User TENCompetence IST 2005 02
10. briefly analyse and explained that the teacher does not have access to activities unless they are explicitly assigned to them TENCompetence IST 2005 027087 Page 41 48 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 4 5 Evaluation instrument I have read this form and I give my informed consent to participation in this evaluation of ReCourse at Liverpool Hope University Name please print A E Signature TENCompetence IST 2005 027087 Page 42 48 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 Appendix 1 Instrument used at Liverpool Hope University July 14th 2008 ReCourse evaluation Liverpool Hope University 14 July 2008 What is ReCourse e ReCourse is an application for creating lesson plans called Units of Learning UOLs e The plans can then be uploaded to a server which coordinates the lesson activities and makes sure that all the participants have access to the right resources and services at the right time e The format of the lesson plans which ReCourse generates is IMS Learning Design an open specification for defining and exchanging Units of Learning What is the session trying to achieve e IMS Learning Design provides teachers and course planners with us much flexibility as possible As a result it can be rather co
11. how needs a change An installer automating as many as possible of all configuration tasks will be strongly needed for use of not experienced with Java users The fact that the some of the servers are on the local machine which is not the case in the real life and some are not like Opendocument causes additional confusion for users It is not clear why we need a player CopperCore engine while creating an UoL with the editor 3 3 START USING RECOURSE For the Multi language versions we strongly recommend the full use of the chosen language not only for some commands or some messages Also in this respect the user documentation also need to be available in advance in the same chosen language There are problems from time to time with novice users caused by unexpected visualisation of various views Sometimes the user can become lost There is a strong need of special section dealing with this problem and explaining users how to become more oriented in this complex environment and how to find the right window again 3 4 CREATING A UoL In the first prototype there were very big problems for some of the users to publish their UoLs even all elements were highlighted as OK and validated with green marks Further on after discussions with developers most of these problems were solved In any case this is a signal that there is a need for quite more user centred documentation about all aspects of the tool LD is a very complex speci
12. in and I couldn t delete it Is it because it s being used somewhere that its checking Could it be used in here learning object Yes you might not be no it shouldn t These resources are associated with these learning objects aren t they So if it was being used in here you d still should You should be able to It s just a reference I mean it may be just a bug So the bug what s happening is you ve opened the learning object you ve assigned some resources to it You ve created these Was this the one that was in there by default User I couldn t delete the default one at all So it looks a bit as if you can t delete the default resource in a learning object I change the resource so its not the default Can you change its name I can change the name and I still cant delete I think what it s telling us is that the application is built in such a way that you must have at least one resource for a learning object And that rather than checking to see if there is one it says I m not going to let you remove this one you must use this Page 26 48 xe PTEN Competence Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 ReCourse is constructed so that every learning object must have at least one least one learning resource This makes sense but caused this user problems because they were not aware of this principal
13. it be shown to the learner Transcript 3 Unless there is a strong argument for providing the user with prominent access to this element it should be made less salient It could perhaps be available as an expert mode or through the resources tab 1 4 There is an ambiguity in IMS LD about where learning activity instructions should be placed in the activity description or in the associated learning object Transcript 4 1 5 It would be advantageous to indicate the distinction between resources to be used by the learner and resources used to describe the activity perhaps by changing terminology Transcript 4 1 6 Editing of resources in situ should be made more transparent and resource management aspects restricted to the Resources tab Transcript 5 1 7 ReCourse is constructed so that every learning object must have at least one least one learning resource Usability would be enhanced by including a message to tell the uses you can t delete this because every learning object must have at least one learning resource Transcript 6 1 8 Users would be helped by providing them with feedback which indicates the state of elements without them having to open them up and inspect them This would enable them to see for example so that they can see for example if it an activity has a role an environment associated with it This would provide an indication to learners of what authoring actions they still need to take and where problems
14. it easy to use giving an average score of 2 7 close to a neutral response but this was not to be expected and it is satisfactory that they did not find it hard to use Three of the seven participants succeeded in building and publishing a Unit of Learning in the relatively short time provided for them to work with the application 90 minutes while one more succeeded in publishing but did not have time to complete debugging The remaining participants made good progress towards this goal as is shown by the score of 4 for the question I can imagine that I could use the software in future by myself This work was carried out with the support of the evaluators but this support was not intensive as shown by the response of 2 to the question How often did you need to ask for help from the session organisers on a scale of never to continuous Thus it seems clear that within the period of the workshop almost all participants were able to reach the point where they could start working independently with ReCourse It is worth noting that this finding relates to ReCourse as an editor for IMS LD Level A Editing of Level B which includes properties and conditions requires expertise in authoring XHTML and will always remain a more challenging task However two considerations should be borne in mind Firstly some of the more frequently used applications of level B can be facilitated with specialised interfaces for example
15. so I can t have this play as my whole course Yes you can But if I do that then how can I say that I don t want it to start until a week after I ve started the unit Your first one lasts a week then the second one will start a week into the course so longs you have put the first one you ve got when the following has been completion rule um why have we Oh yeah the activity isn t it If the activity in the first 1f this activity discussion forum has a week to go then it ll delay the Right So long as you ve clicked all the completion conditions Because unless you click the completion conditions everybody can see everything Well that s right yes So you have to put completion conditions here But that does mean then that if half way through your course you ve just got something you want to last a week and it needs to start at a particular time there s no way that If you just put the completion conditions for the teacher say click and go on then you can Because it will go through the acts Page 36 48 FAA PTEN Competence Right Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator TENCompetence IST 2005 027087 D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 sequentially So 1f t
16. stand alone entity which is not linked to an activity then the users won t see it when they go in won t they It s like giving the privileges but you haven t given them access So I ve said who can do it and I went to resources and told them what to do with it there I think here we ve got a confusion This resource really its only there because the specification says we have to have it there It doesn t make much sense for a forum to have a resource If you remember what I said is that the only important thing is this bit the participants You re thinking of this as the activity instruction Yes but that s not really where you want it No I see I need it in that one which is the forum activity that I created already Page 23 48 xe PTEN Competence Evaluator 2 User Evaluator 2 User Evaluator 2 User Evaluator 1 User Evaluator 1 Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 So that needs to go in there this here The only thing that s important is that when you give it to these people that they have access rights to the forum Otherwise they ll go there and they can t Ir Il spit em out again That was what was confusing me Because there was a resource there and I put it in and then I thought that doesn t make sense because I can t make
17. suppose I could go in and have a look but I m not sure if I ve actually used it or if I ve just created it and not put anything in it yet Like a site map that says Because when you go to here I can see the widgets but not the activities necessarily that I ve associated with them but I go to here don t I Cos I ve got my discussion forums So my welcome activity I thought I d used that Page 28 48 FAA PTEN Competence Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Comment Recommendation TENCompetence IST 2005 027087 D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 somewhere Maybe I replaced 1t with a discussion forum Maybe that s what I did So what would help you here Um Almost I mean I think this does it but you re switching through almost a smaller hierarchical list I d have Welcome and then all of the things associated with that The problem with that is that it may get too big that s the issue and obviously I m just learning And I think what I did possibly is created this Welcome activity What I ve found helps a lot is that if you are more comfortable with the resources tab to make sure that I make all the resources with sensible names And then I find that helps a lot Right so that s what I ve done I ve just
18. these reports two aspects of the application should be remembered Firstly the LD Toolkit is intended for users who want to create relatively complex Units of Learning which contain a number of roles and a flow of activities If this is not the case then it is sufficient to distribute links to resources and an activity instruction to learners and this functionality is supported within TENCompetence by the PCM Rich Client application and by the more recent Web based PDP tool Consequently it is to be expected that with a powerful application such as this will take a certain amount of time for users to become confident in using it Secondly IMS LD includes support for generic XHTML authoring with properties and conditions and while support can be given to non technical users some aspects of this can be will remain the preserve of technical experts TENCompetence IST 2005 027087 Page 2 48 xo E competence D6 2 Annex 1 Validation testing and usability results 2 Evaluation of ReCourse and SLeD with the Widget Server at Universitat Pompeu Fabra 2 1 CONTEXT OF THE EVALUATION The evaluation took place on May 22nd 2008 13 20 15 15 Three learning designers from UPF followed the worksheet Two of them were familiar with LD and had used Reload and CopperCore before 2 2 INSTALLATION e They managed to install and access the widget server through ReCourse o Their expressions were of surprise when seeing for the first time the n
19. with their UOL might lie Transcripts 7 8 14 1 9 It would enhance usability and clarity if the resource names generated by ReCourse were related to the names of the original files Transcript 10 2 Structure and concepts of IMS LD 2 1 Users repeatedly found difficulties with a small number of IMS LD concepts Transcripts 8 9 11 12 13 14 16 18 Most commonly this involved environments e g transcript 11 13 14 and their relationship to activities but roles transcripts 9 18 runs transcript 16 and even learning objectives e g transcript 12 also created difficulties None of these proved difficult to explain and users were able to continue without problems However if help had not been available they might have become blocked Documentation and workshop materials for ReCourse should focus on the small number of principal issues which cause confusion for novice users starting with those identified in these conversations Appropriate error messages could also be of assistance 2 2 The environment element which caused substantial misunderstandings e g transcripts 11 13 14 is a basic part of IMS LD and it is difficult to dispense with it in the interface This may sometimes be achieved by creating a default environment for a new resource but this creates the problem of a generating as many environments as there are resources which 1s difficult to manage TENCompetence IST 2005 027087 Page 14 48 DoS D6 2
20. 7087 I ve got my activities and my resources but what s a learning objective for The learning objective is so the learner knows what on earth they re doing this unit of learning is for So you are telling them what you expect them to gain from it Yeah So it s a learning outcome Or so another teacher knows what 1t was that you designed this thing for OK What did you think 1t might be I dont know I think I just had an extremely long day and lots of 14 year olds that have given me a headache I m glad it s not us This is a holiday is it in comparison Java script and cascading stylesheets for 14 year olds Anyway Though I don t know because for me the learning objective would be I think I was thinking learning outcomes and even now I don t think it s fully clear in my mind exactly how these structures fit together Just think of it as a kind of you know what you put on the front page of the course sheet Yes this is what we want to do But who Why bother is that the question Well no because you always say to people this is what we want to cover today these are the aims or the objectives of this particular session You would always say that to them and you would try to ensure at the end of that that you haven t just covered Page 32 48 FAA PTEN Competence Evaluator User Evaluator Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used
21. Annex 1 Validation testing and usability results E competence TENCompetence ID 6 5 Validation testing and usability results b makes it harder for users to make use of the clarity and reusability which organising resources and services into environments brings to IMS LD Consequently hiding the use of environments in the interface is a strategy which should be used with care Conversation 13 suggests that the preparation and classification of learning materials prior to starting work on the UOL would be of assistance to learners in understanding environments This aspect should be included in the documentation and could also form part of training and workshop activities although this would add to the time required 2 3 In order to make the completion conditions work by means of users indicating completion it is necessary to give either the teacher or the learner the ability to indicate completion Some users did not see that this would be necessary assuming presumably that this would be done automatically If practicable the UOL should be parsed and users given feedback indicating if they have created an inconsistency between completion conditions Two respondents working together transcript 17 commented that for them populating multiple UOLs over the length of a full year course would be a major task while creating a single large UOL would reduce flexibility unacceptably This confirms observations already made within TENCompetence
22. Evaluator So we thought we had got it all right because we wrote text in and we thought it was and when we checked it and we know we wrote some learning objectives but for some reason they are not showing up So we have gone wrong somewhere along the way Well I ll have a sneaky peep at your UOL and work out what it was that has gone wrong TENCompetence IST 2005 027087 Page 34 48 FAA PTEN Competence User Evaluator User Evaluator User Evaluator User Evaluator Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 Because of time constraints we put two acts in but we deleted one so that we could get all our green ticks We wrote text in there and assumed that that would be the text that would come up Which one That one Brainstorm project ideas environments You haven t assigned the environment to the Which I m sure you would have figured out if you had a bit more time to look at it So that was the only thing that was missing All the text was put in the right place it was just checking the box I guess so Do you want to see if it works No you need to get away I guess if you were sitting around doing that you d be reading the user manual anyway wouldn t you But also I ve found myself learning it you just need a few hours to mess around and think Why didn t that work and che
23. So I mean like would you run out of space Yes Because it When you create the environment you have to create all the environments on that screen So say you had a lot of acts You have to have all these environments So would it you know help that they just go over the top of each other or would it just scroll down so you d have more room I wonder if you can do this no you can t You d be scrolling for ages because you can t see them and you might think you d lost them In a normal sort of screen you d be used to seeing a scroll down thing As an uninitiated person I wouldn t know I wouldn t even have thought of dragging them down But if I d seen a scroll down bar I know what that means You used to be able to but you can t at the moment I think that the idea is that you d have a bank of these over here Ideally what you d have what would be really nice would be as you do with any huge document would be to do it in bits and then concatenate at the end I suspect that this tool doesn t do this in which case your comment is absolutely right It d be like where s that box and zzzzzzzzz These users identified a problem with the practical use of IMS LD with the current infrastructure 1 e that the need to provision multiple UOLs over the length of a course is a major task while creating a single large UOL reduces flexibility This problem can be overcome by means of a link tool as discussed in TENCompetence on other
24. age 11 48 D6 2 Annex 1 Validation testing and usability results xo E competence TENCompetence ID 6 5 Validation testing and usability results Again these respondents indicated that independent work with the application is possible for teachers It should however be noted that the respondents were not all convinced that the infrastructure was appropriate for their own practice e can see the potential benefits but probably only for distance learners It seems a lot of work when you have access face to face with students but I would persist if I had distant students e Program seems good for distance learning and courses that have large teaching teams e The complexity of setting up units takes a lot of time and for complex and large unit structures e Very resource intensive program not all classes would require such detail Thus it seems that while teachers can deal with the complexity of authoring IMS LD with ReCourse it is not likely that most of them will actually do this unless they have a particular reason for doing so for example authoring distance learning courses or creating complex learning flows This is in line with the provision of simple authoring functionality in the TENCompetence PDP tool along side the authoring of IMS LD Units of Learning 4 2 2 the nature of the difficulties which users experience a Likert scale questionnaire responses Section 2 of the questionnaire section 3 1 addresse
25. ake notes so that you will remember the things which you found confusing or did not understand The people running the session will be coming around to help you and will record the problems which you are experiencing e The evaluation form audio recordings and the evaluation report will be anonymous How the session will be run approximate timings Consent forms personal introductions introduction to the session 45m An input session providing an introduction to ReCourse and a practical guide to its functionality Guided creation of a Unit of Learning Coffee Break Planning a Unit of Learning with paper and pencil 75 m Building a simple new Unit of Learning with at least e two acts e two environments e two roles e one service Uploading the new Unit of Learning to the server and running it TENCompetence IST 2005 027087 Page 44 48 D6 2 Annex 1 Validation testing and usability results ne PTEN Competence Instrument used at Liverpool Hope University July 14th 2008 ReCourse Evaluation at Liverpool Hope University 143 uly 2008 User feedback form How to fill out this form Please indicate your agreement or disagreement with the statements on the left by putting a cross in the appropriate box of the five point scale to the right 1 General impressions I enjoyed using this software Enjoyable Not enjoyable The software was easy to use IM il Difficult 2 To the extent that you had difficu
26. at Liverpool Hope University July 14th 2008 that but that they actually have coped with that and understand it Which is fine but its jut the way it sits on its own like this I think the reason that it sits on its own like this is that when you ve made your unit of learning this will be tagged up in the XML as the learning objectives That means that any other application looking at this say can interrogate it and find out what the learning objectives are without knowing anything else So when we keep this on a repository then when you look at the repository and you find an object you can see what its learning objectives are because we put this in as a discrete piece of xml If it was just a piece of text you wouldn t know No that s right so it allows you if for instance you were going to make these into reusable learning objects and then someone else comes along for something they can use then they re going to use this list Because this is what they will see you are hoping they will get out of it So you don t even want It s not just a list of tags it s more concise than that isn t it It literally is that this unit should at the end of the day have introduced A B and C kind of thing Yes This user was in doubt about what exactly a learning objective is and if it was the same as a learning outcome This underlines the degree to which terminology which seems transparent may be confusing to users a An brief and easy to
27. branching structures dependent on the result of a test This will be done in future versions of ReCourse and there is no reason to suppose that this type of authoring will prove more challenging than the level A authoring evaluated here Secondly the services forums and messaging which were authored by participants in this evaluation session would in earlier LD authoring tools have required expert authoring at Level B or programming work in integrating services b free text questionnaire responses see section 3 2 The free text questions provide support for the conclusions from the questionnaire specifically the responses to the question Please briefly describe any aspects of ReCourse that you particularly liked This includes the purpose functionality and interface of the application e Overall the package is manageable and useful after some effort in understanding interface e Good quick intro need more time support to actually use e Interface GUI is particularly attractive e Once set up seems like it would be easy e Navigation at first was confusing but after 20 minutes of use this became clear and more logical 2 See for example an earlier evaluation of the Reload LD Editor Barrett Baxendale M amp Groes S 2006 Supporting an English tutor in using IMS Learning Design Retrieved November 14 2008 from http bsd 1 phosphorix co uk ld4p images fcs_englishness pdf TENCompetence IST 2005 027087 P
28. ck through everything These users successfully built and published a UOL but had not linked an environment with a learning activity This is an example of the non intuitive nature of environments They had created the activities and were not conscious of the need to do any more The provision of an inspector of some sort which could indicate when an activity has no environment would help with this difficulty 15 Managing completion conditions Evaluator Eval Evaluator Comment So was there a problem It was the act where was the completion for it it was here he had when the following has been completed but not any conditions I found the completion a bit tricky sometimes because you have to get a lot of different things right in different places It s not really hard but The problem here was that in order to make the completion conditions work by users indicating completion you have to give either the teacher or the learner the ability to indicate completion Some users did not see that this would be necessary assuming presumably that this would be done automatically Recommendation If practicable this problem could be resolved by flagging to the user that they have created an inconsistency between completion conditions TENCompetence IST 2005 027087 Page 35 48 xe PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University Jul
29. combined to work together with very good user documentation backed with good methodology examples and rich library of best practice example UoLs 3 to have more time to prepare meaningful and well elaborated evaluation instruments Concerning the last versions with mixed BG EN menus all testers declare that they were more confused with the last version maybe because they already have experience with the previous one and have read the documentation for that one With the last version almost all testers declare big troubles as they don t know where to look for functionalities described in English but available with the Bulgarian menu TENCompetence IST 2005 027087 Page 8 48 D6 2 Annex 1 Validation testing and usability results xo E competence TENCompetence ID 6 5 Validation testing and usability results 4 Evaluation of ReCourse at Liverpool Hope University 4 1 Introduction Objectives The objectives of this evaluation action were twofold 1 to establish if teachers would be able to use the ReCourse application to author Units of Learning without long term support from the project team 2 to obtain insight into the nature of the difficulties which users experience User profile Within the TENCompetence project two kinds of learning activities can be created Firstly there are simple learning activities which are defined within the Personal Development Plan tool and consist of resources and activity de
30. contexts for example in Moodle an activity is an opportunity for a user to do something Page 29 48 FAA PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 e g a wiki This needs to be clarified in the documentation and in training b An inspector which could show in which activities a resource is used would be a useful tool for authors 9 Understanding the relationships of roles and acts User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator Comment Recommendation PE assign a role to this act All you roles are here You are trying to assign a role to an act but you don t assign a role to an act you assign a role to an activity So you don t have any activities here except a default one But if we drag a default one on then you can assign all these to it That s what I didn t do because So you ve created your roles and they are all appearing here which is a complicated set lots of roles But they don t do anything until you have some activities for them to carry out So you have to define some activities and then associate the roles with the activities Does that make sense Right yes yes These are the kinds of concepts that are hard to get your head around Yes yes Very interesting for me to see the kinds of problems that emerge Do you want any ot
31. created that and I haven t used it so all I need to do is plonk that on here Teacher description Which is fine But hang on we are in Introductory activity here don t you want to be in Welcome No that needs to be changed The introductory activity is actually a forum so the name is not appropriate OK so why don t you change it So I will But I mean not change the name of the activity but change the Or do you want to change the name of the activity Yes because Well maybe no maybe do you call the Forum the activity I suppose you do Its up to you but you can change the name of the Activity or the name of the Play whichever you want In fact this 1f truth be told no ignore that whichever you fancy That would make more sense from what I ve said I need them to do So it s associated with But they don t want that Of course I have a of the environment don t I Environments It s good to have something in it so I presume I put an Learning Object in if it s just going to be a list of files Whatever it s going to need something in it So learning object may just be a placeholder Environment This user thinks of the activities being associated with the widgets rather than vice versa She would like to have a list which shows in which activities her resources have been used a The concept of activities used in IMS LD i e An action taken by a user is different from that used in other
32. d the difficulties experienced by respondents in using the software In no case did the average response to any question fall below 2 5 and for five questions it was above 3 Inspection of the detailed responses however shows that some respondents found difficulties with certain areas For example e two respondents gave a score of 2 to I had difficulty understanding what the different menus and windows were supposed to do while three gave a score of 4 e three respondents gave a score of 2 to I didn t know how to make the elements I needed to build a Unit of Learning while two others gave scores of 5 e one respondent gave a score of 1 to I could not find my way from one part of the application to another while giving high scores to other parts of the interfaces while another gave a score of 4 to that question and lower scores to others Thus the questionnaire suggests that while most respondents found the authoring task challenging there was considerable variation from user to user in the areas of the application with which they had difficulty We speculate that may be the result of varying expectations due to differing previous experience of similar applications and of the teaching process b transcripts of conversations with users indicate that in most cases the underlying problems in most cases were twofold TENCompetence IST 2005 027087 Page 12 48 D6 2 Annex 1 Validation testing and usability results
33. dow e One of them asks whether it is only possible to have a manager and a moderator in a chat e All the designers had problems when creating a resource The resources window has still opportunities to be improved For example the editing icon is available in TENCompetence IST 2005 027087 Page 3 48 xo E competence D6 2 Annex 1 Validation testing and usability results both areas of the window while it can be used in principle just in the right hand area e One of the designers had problems when creating a resource She clicked over the central window and didn t use the left hand window e In general the designers have found some inconsistencies with the references to the sections and to the tables in the text and they commented that it would be useful to have more screenshots showing the options e All the designers comment that when associating a resource to one of the activities the Worksheet makes reference to the first activity in the second act They found that it would be more useful to reference activities in the first act e The activities in the organizer should be deleted when deleting an activity from an act It could be probably an idea to add a confirmation message to delete it completely from the UoL or just from the act so that the activities can be added to other acts Otherwise there can be problems when in the final checking of the UoL e One of the users had problems when checking and asked for cla
34. e file or you can create the file and then assign the resource that you want OK so there are two resources now for this environment For this learning object Page 25 48 FAA PTEN Competence User Evaluator User Evaluator Comment D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 And how does that learning object oh it does have open database technology so this learning object is for this This learning object is in this environment in this unit of learning Everything is in one unit of learning All these files are in one Unit of Learning OK Yeah Do you want to bash on a bit and I ll The user here is seeking to edit the resources for a learning object but does not understand how to do it They are trying the file menu which does not help They do not know how to create a new resource Recommendation As concluded from other interactions editing of resources in situ should be made more transparent and resource management aspects restricted to the Resources tab 6 Deleting default resources User Evaluator User User Evaluator User Evaluator Another Evaluator Evaluator User Evaluator TENCompetence IST 2005 027087 We ve just discovered we can t delete that one Which one If T add a new resource here I found that before I thought I d delete that resource that I ve just put
35. e text questions 3 3 Transcripts of conversations with users TENCompetence IST 2005 027087 Page 16 48 DoS D6 2 Annex 1 Validation testing and usability results EU competence TENCompetence ID 6 5 Validation testing and usability results 4 3 1 Results of the Likert scale section The scores are adjusted so that higher score better result for all questions Question text Reply options score high good General impressions SEE E a i E PEA Enjoyable not I enjoyed using the software enjoyable 413 4 4 3 3 3 24 3 43 To the extent that you had difficulty in using the software please indicate in which areas you had problems and to what extent o A I had difficulty understanding what the different Not a problem Not a problem Not a problem Not a problem I didn t know how to make the elements I needed to build a Unit Not a problem I didn t know how to set the properties of Not a problem Not a problem 6 ad probe managing the resources fo the Unit of Learning _ A majo pron 3_ Building a VOL o EEE Yes No B1 T successful created a UntotLeaming ligero fifi folifofol os How often did you need to ask for help from the session 03 ognon a OS oron ieverconinasy sfofa Lolli ia gt 3 2 I can imagine that I could use the software in future by myself 3 3 Ican imagine making use of the software if I had support TENCompetence IST 2005 027087 Page 17 48 DoS D6 2 Annex 1 Val
36. ecause the activity itself has properties which have resources but they re a DIFFERENT kind of resources So would that be the research question Yes when you give them this activity what would you say to them The words in quotation marks look at this or think about it go in here The material which they are going to study goes into the environment So this is the description of what they are going to do Even more than the description it s the instruction itself Description is a bad word here It s look at these links write me an essay So it s the objectives Mmm Pd reserve objectives for achieve an understanding of such and such a higher level description Think of this as the actual words that you would give to your learners when you give them a task Well I want them to write a 500 word report to describe a database Yes put in write a 500 word to report to describe a database The confusion is that that in the file structure is a resource a piece of HTML but its not a learning resource Because its quite complicated I was thinking that if I had the forum I d also need an activity well I do need an activity but then when the activity is up So you want a forum and what you are saying is that you do need an activity because at the minimum because the forum they g But what confused me is that you can double click on the forum But at the moment that is a
37. ed that you had problems please add any comments which may help us to understand your difficulty more clearly and any suggestions for improvements which you have Please indicate the number of the question which you are writing about e Resources o I had slight problems understanding resources and properties as it was all getting mixed up I think separating most of the resources in a slightly easier way could well better help such as separating pic or links and storing it in one place rather than changing name in 2 different places TENCompetence IST 2005 027087 Page 18 48 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 e Comments Could do with show hide for roles If you click as a dialogue box is open the dialogue just disappears When publishing spell check e Clarification of the terminologies for novice users e When clicking on learning objective can there be a label on the tab that opens that tells you which environment you are in as I keep forgetting e Organiser is it possible to have folders collapsible things in the organiser because it s going to be a very long list of things especially for longer courses e Also some kind of pop up reminder to create activity names that relate v closely to what your week class is e g Week 1 lecture week 1 seminar week 1 tutorial Week no important to have in file name e A
38. ence D6 2 Annex 1 Validation testing and usability results TABLE OF CONTENTS 1 INTRODUCTION nn EE E oca asada 2 2 EVALUATION OF RECOURSE AND SLED WITH THE WIDGET SERVER AT UNIVERSITA F POMPEU FABRA cociente eare ENEE E A aa 3 2 1 CONTEXT OF THE EVALUA TION aaa 3 22 INSTALLATION eoa R ene erat ene AROE OAO OE eee 3 25 START USING RECOURSE 00 a 3 2 4 CREATING A UoL OBSERVATIONS MADE BY TWO SUPPORTING RESEARCHERS THE CREATORS OF THE WORKSHEET A A AAA eaa Nee 3 3 EVALUATION OF RECOURSE AND SLED WITH THE WIDGET SERVER AT THE UNIVERSITY OF SOFIA iaeaea O T nade ENEA TE A EE 7 3 1 CONTEXT OF THE EVALUATION cie e A 7 32 INSTA ELA TION i AI o RETO O A eoten tie 7 3 3 STARTUSING RECOURSE 00d dida 7 3 4 CREATINA DU an en aeons 7 3 5 PLANO A OD e O ATE NTO 8 4 EVALUATION OF RECOURSE AT LIVERPOOL HOPE UNIVERSITY sssssssseeooes 9 4 1 INTRODUCTION a cd 9 4 2 OVERVIEW OF RESULTS talca caro cia 10 4 3 DETATEE DRESS tna a Stadia ia baat aia da dia Colores iO 16 4 4 TRANSCRIPTS OF RECORDED CONVERSATIONS WITH USERS 0sscccsesecceescecaeseeseeceeseeeaaneeeaenes 21 4 5 EVALUATION INSTRUMEN Decente liar tesoros 42 RECOURSE EVALUATION LIVERPOOL HOPE UNIVERSITY 14 JULY 2008 ccseececcceececcceseeececeeeaees 43 TENCompetence IST 2005 027087 Page 1 48 xo if competence D6 2 Annex 1 Validation testing and usability results 1 Introduction The present document is an annex to TENCompetence D6 2 and it contains repor
39. entifying interface See Annex 1 to TENCompetence deliverable D6 1 available at http hdl handle net 1820 1149 TENCompetence IST 2005 027087 Page 9 48 D6 2 Annex 1 Validation testing and usability results xo E competence TENCompetence ID 6 5 Validation testing and usability results problems but it does little to identify conceptual difficulties or to assess the ability of participants to create Units of Learning independently Consequently a smaller but more in depth evaluation session was conducted including interviews in order to get a richer picture of user experience The session was conducted as a workshop with the following approximate timings departing somewhat from the plan in the instrument Introduction to the session and signature of informed consent forms 15 minutes Presentation on the fundamental concepts of IMS LD 45 minutes Hands on training 60 minutes Coffee break 20 minutes Participants development of their own UOLs 90 minutes Completion of questionnaires and close 10 minutes During the time in which the participants were creating their UOLs the evaluators went around discussing the difficulties which they experienced When an interesting interaction occurred this was recorded with participants explaining their difficulties Instructions and briefing to the participants were provided to participants and are reproduced in the following section together with the questionnaire was distr
40. ew graphical design of the authoring tool e There were not problems with the installation Just some questions about the version of the Java JDK that they have to install It would be necessary to specify that you can use any version from 1 5 0 2 3 START USING RECOURSE e On the beginning the user finds that it is necessary to add a screenshot as a figure in the worksheet with the initial screen of Recourse where you can choose Start with ReCourse or Information of ReCourse e When creating a UoL you have to create a folder Specify in the worksheet instructions that it can be anywhere 2 4 CREATING A UoL Observations made by two supporting researchers the creators of the worksheet e In the Unit of Learning Window the prerequisite window cannot be seen if the size of the windows is small No scroll is available for that e One of the users comments that they expect an explanation of the central window Tab distribution the Organizer e When adding information about a play we comment click on the grey area of the Play The user doesn t understand where she has to click Add a signal on the screenshot e Two of the users were lost when creating the role parts for an act They had to click first over the act where they had to create the role parts e One of the users comment when using the palette that she didn t know where of the environments tab she should click on on the organizer or the central win
41. fication with quite a lot hidden semantics which somehow needs to be made more open and clear for the users especially when they do something wrong Even now users can define perfect from the user s point of view UoLs validated from the ReCourse but not able to be published TENCompetence IST 2005 027087 Page 7 48 DoS D6 2 Annex 1 Validation testing and usability results E competence TENCompetence ID 6 5 Validation testing and usability results 3 5 PLAYING A UoL About half of users signal various problems in playing their UoL It is not realistic to expect from novice users to create meaningful and error free UoL in their first experiments In this case we need to have as rich as possible library with existing UoLs which need to be used in the documentation in explaining all variations that can be achieved Most of the users strongly prefer this approach learning by examples and maybe the WP6 partners need to pay special attention in this direction preparing a set of well thought UoLs examples covering most of the LD richness and various LD design styles GENERAL REMARKS The environment is very rich and powerful It is not realistic to be evaluated for less than 2 or 3 months intensive work of experienced evaluators We need to be quite better prepared for this 1 to have clear vision what exactly we want to evaluate and validate 2 to have very stable environment all tools working well and well
42. fter trying out ours we thought we d done it correctly but the text and LO s weren t there Obviously gone wrong e After you create an environment and then add resources etc it was difficult to remember the name or what environment you are currently working in e Need a spell check e No overall list of what has been completed as a reference point You have to make a mental note of how or what you have done e Would be difficult when organising a large amount of activities need folders to group files e Need overall save before publishing c Please briefly describe any aspects of ReCourse that you particularly liked This includes the purpose functionality and interface of the application e I think the factor of validation Checking 1 e Checker and publisher before uploading the document is really exciting as it gives you a flavour of what you have done e Overall the package is manageable and useful after some effort in understanding interface e Good quick intro need more time support to actually use e Interface GUI is particularly attractive e Well linked in one window e can see the potential benefits but probably only for distance learners It seems a lot of work when you have access face to face with students but I would persist if I had distant students e Program seems good for distance learning and courses that have large teaching teams TENCompetence IST 2005 027087 Page 19 48 DoS D6 2 An
43. g I need a darkened room Yes OK I think it requires more than half a day Yes it does its a day I think its a day And also it would be really helpful to say if you are going to do this you need to produce yourself a little course and the resources you d like to be made available and then come and build the unit with materials that you know with a plan in mind because to do it off the top of your head is quite difficult I think if we were if we actually going to use this with learners that s what you d do Or were even thinking about it I felt a bit hesitant last week it s already a big favour for people to come and do stuff No no I mean I was highly tempted to go on the web and just nick some of the database stuff and put that in because that s what we ve already developed and most of the stuff was there and then you d identify gaps because this gives you much more functionality doesn t it I guess captivate and stuff like that you tend to use These users are confused about the role of environments and observe that ReCourse would be easier to work with in the context of an authentic teaching task where real materials were available to be used Where possible training in ReCourse should include the preparation of materials as an initial stage In the documentation suggestions should also be made about preparing materials in advance 14 Failed to associate an environment with an activity User Evaluator User
44. give the name the same letter Page 40 48 FAA PTEN Competence User Evaluator User Evaluator User Evaluator User User Evaluator Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 So 1f the problem is allocating her but you also haven t given the teacher the learning activity You haven t made them available to the teacher No Which might be good I mean maybe the teacher doesn t want to know what is going on Yes it said participant and I thought the teacher isn t going to do this she s only going to look at it Yeah but you have to So do they have to they don t have to be a participant and everything Well in the forum if they can manage and so on then they can But the forum would be OK but any learning activity unless you tick the teacher for that learning activity they won t be able to see it Right Obviously Stupid woman Well no not obviously We assume that the teacher does the teaching and that what they do is a learning activity and so therefore it can be seen But that s not how it works in this When they had published their UOL these users assumed that the teacher would have access to the whole UOL even if the teacher role had not been assigned to activities Once this was pointed out the user it became clear to them The documentation and workshop materials should clearly and
45. he one before the seven days says don t complete this until I tell you so then you ve solved it It s a bit messy that bit isn t it It is messy but If you re running this a number of times then you re relying that the teacher is not going to click it by mistake Well either you want it controlled by time or you want it controlled by somebody saying do 1t I think if I wanted it controlled by time then I d want to create a schedule in one place so I could see all the acts and all of those time constraints for all of them and be able to adjust those so that I know when it starts Because again you then otherwise you are kind of bounded to potentially not starting just before you use it if you dont really want it that structured A time line might help just being able to drag it I think what you are Saying 1s confirming my instinct that this is an area I think what I d want is a timeline and I m quite happy to put all that in but there would also want to be able to say even if I say this is available actually don t start it until this date Because assuming that you are going to be available to make something at exactly the right date is a bit of a pain whereas if you say right I m happy this is done but I don t want it published until You put that date in and all your other constraints remain Trouble is that it can t be a date It has to be a length of time because it is general You see what I mean Yes Yo
46. her hints now No it s OK Next thing to do is to set up some activities Right This one s OK so give it a name edit It says description but actually its more like the instruction you re going to give to the learner Set up a few of these and apply them This interaction is an example of the kind of help which was given to users in the evaluation in order to understand the underlying concepts of Learning Design Issues like this needed to be explained but once clarification had been given the users were able to carry on working The documentation for ReCourse should focus on the small number of principal issues which cause confusion for novice users for example environments activities roles and the relationship between them Appropriate error messages could also be of assistance TENCompetence IST 2005 027087 Page 30 48 xe PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 10 Adding files as resources User Evaluator User Comment Recommendation So you were trying to drag files onto the resources and you hadn t realised that they were already there But then if it was really our own sets of files we d know that we d already done that They d have more sensible names Yes we d know They might have been like Word documents The Resources area was confusing for these users and it was not clear
47. here be a pop up say if you can t see your package here click on the green connect to server icon e Felt uneasy about did I save it or didn t I e The complexity of setting up units takes a lot of time and for complex and large unit structures e Not keen on the term Play Seems strange for what we are doing Not sure what term should replace it e Very resource intensive program not all classes would require such detail e really struggled with the environment and resources Also the fact that a service is just a resource was confusing also TENCompetence IST 2005 027087 Page 20 48 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 4 4 Transcripts of recorded conversations with users 4 4 1 Resource management 1 Resources are the principal problem User Resources that s what I need to get my head around Evaluator But you think that given another session you d get your head around that User Yes and I tend to be somebody that would need to use it for something So with a bit of preparation then I would apply it to something Comment This is an example of the many occasions when users commented that the management of resources was the principal problem which they encountered Recommendation Attention should be paid to improving the support for users in handling resources as discussed in the following points 2 Prere
48. hings You know like when you click on a folder and it s all ch ch ch there In a way I think that s what this is That s the file structure but that s not really what you want You want something Something that works just with that page If you had something that was Oh you mean then you could see 1f you had really forgotten to put something in it Oh dear I ve forgotten to There is a window here that I haven t shown you and I dont really know quite what it does but he s tried to If there are six power points how would I know that I uploaded ReCourse is a highly flexible tool which enables the user to start building their UOL in any way which they wish It does not impose an order of actions or a methodology on the user This means that the users have to hold in their memory which actions they have completed and what names they have used This is not difficult for a user who knows their way around the application but for new users it is challenging The users requested support in seeing the state of elements so that they can see for example if it has a role assigned or an environment associated Some functionality along these lines should be provided 4 4 2 The structure of I MS LD 8 The nature of activities Evaluator User TENCompetence IST 2005 027087 So where can you find where you ve used your activity Oh going backwards Yes cos I ve maybe I didn t plan it very well but I know I
49. ibuted reproduced in the following section and completed by seven of the eight participants one had to leave urgently Treatment e The Likert scale responses from the questionnaires were transcribed into a spreadsheet e The free text responses from the questionnaires were transcribed into a text document e The interviews were transcribed into a text document All results were anonymised The scores from the questionnaire were consolidated and totals and averages calculated The conversations with participants were classified into themes For each conversation a comment and a recommendation was added 4 2 Overview of results 4 2 1 Independent use of ReCourse The first objective of the evaluation was to establish if teachers would be able to use the ReCourse application to author Units of Learning without long term support from the project team We now review the evidence provided by the responses to the questionnaires a Likert scale questionnaire responses see section 3 1 All results are on a five point scale where 5 is the most favourable response The results of the Likert scale questions indicate that the respondents could get to grips with the application with only a two hours of demonstration and TENCompetence IST 2005 027087 Page 10 48 DoS D6 2 Annex 1 Validation testing and usability results E competence TENCompetence ID 6 5 Validation testing and usability results presentations They did not find
50. idation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 4 3 2 Free text responses from questionnaire a If you had a difficulty which was not covered in questions 2 1 to 2 6 please write it down here e Presentation and training materials o A slight slower pace while demonstrating could have been much better for a proper understanding of various terminologies as that s where I struggled initially o The linking of the applications is a bit complicated The documentation should specify section with tasks as to which is to be performed first e Interface design o 2 2 User interface and icons may need improvements o 2 3 Some controls e g Environments appear at more than one place in the interface and it is confusing because each of these can be used for different purposes o Is it possible to have pop up box with message reminding you to save manually as you go along and also when you click on the upload buttons o The steps and layers made it hard to remember what order to complete creating a unit 1 2 3 e Conceptual issues o Did not click teacher as participant therefore things could not be seen o The terminology is a bit difficult to understand in the first instance e Resource management o Unable to delete an unwanted resource this turned out to be the default resource of the learning object b If in your answers to you questions 2 1 to 2 6 you have indicat
51. it I think what it says to me is that the resource should have a default text which says You don t really need to put anything in here unless you are really sure that you want to Given that that resource 1s there does it do anything at all As far as I can see it doesn t I think it is one of these things that follows the specification The specification says that all these things must have resources but a resource for a forum doesn t make much sense No because 1f you put in something like a web link what I don t understand is how they would If you put a resource in here how would they view it can they view it You re right you remember I said at the beginning there are places that you can put things in where actually you don t need to Because this is allowing you to put in everything that the specification says that you can put in even if in this particular sense it doesn t make much sense And then you get confused like you just have So the answer is ignore the resources for the forum The user is confused about different kinds of resources descriptions and instructions Principally it is not clear where the activity instructions go Should they be part of the environment or should they be included in the activity description If they are included in the description then is the term description misleading The fact that an activity has various types of resources e g Prerequisites makes this more complex
52. lty in using the software please indicate in which areas you had problems and to what extent 2 I had difficulty understanding what the different Not a A major problem problem menus and windows were supposed to do I could not find my way from one part of the application to another Not a A major problem problem The terminology used by the software was confusing Not a A major problem problem I didn t know what to do next to build a Unit of Learning Not a A major problem problem I didn t know how to make the elements I needed to build a Unit of Learning Not a A major problem problem TENCompetence IST 2005 027087 Page 45 48 Zia 2 3 A 2 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 I didn t know how to set the properties of Not a A major l roblem roblem the elements of the Unit of Learning i E I had problems managing the resources for the Unit of Learning Not a A major problem problem Building a VOL aa raosan rota eo fa ea 32 How often did you need to ask for help from the session organisers Never Continuously 32 Lean imagine that I could use the software in future by myself Agree Disagree 33 Tan imagine making use of the software if had support agree Disagree TENCompetence IST 2005 027087 Page 46 48 D6 2 Annex 1 Validation testing and usability results xo E compe
53. lve the errors identified by the checker The errors are not well described and they are difficult to interpret However they all consider that it is a stable system Section 6 Portability The three designers agree that the software is easy to install In fact they don t have any problem during the installation process They don t have opinion for the other aspects In conclusion the three designers consider that ReCourse is a suitable accurate and interoperable tool easy to install and efficient in terms on time of response In terms of usability the opinions are more heterogeneous Two of the users consider that in general that the navigation through the menu and through the different screens is not obvious and difficult to understand They also agree that this system makes demand of some familiarity with this type of tools In the contrary the other user agrees with the interface and finds the system easy to use All users agree with the reliability of the systems maturity fault tolerance and recoverability All users disagree with the analyzability of the system The faults are not well described and they are difficult to be interpreted However they all consider that it is a stable system easy to change and to test QUESTIONNAIRES RESULTS Testing the functionalities of ReCourse authoring tool v 1 5 1 Section 0 Installation of the Environment and ReCourse Edition and Activation of the Tools All users strongly agree
54. nex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 e Ability to take units or move or save any place is a good feature e Once set up seems like 1t would be easy e Like time access e Navigation at first was confusing but after 20 minutes of use this became clear and more logical d Please briefly describe any aspects of ReCourse that you particularly disliked and any suggestions you may have for improvements This includes the purpose functionality and interface of the application e think there should be a slight more clarification over resource management and properties e Connection to server may be built into the main window e Conceptually difficult prefer paper based notes e A bit complex terminology used e Takes a long time to understand e Having to manually put each student s e mail in as a user would take me ages I have 130 students in one module alone Will there be a way of importing students emails e t s not very clear what where the run button is Maybe if when you hover over the icon it could also say click on this button when you are ready to upload to the server It would be more user friendly e On the delete act or play options can we have act before play Less likely to accidentally delete the whole play e When you get to the screen with publisher if there s nothing showing your thing isn t there could t
55. nfusing to use with many elements and properties which need to be set In creating ReCourse we have tried to make this process as simple as possible In this session we will evaluate o How far we have been successful in making an IMS Learning Design editor which teachers can use o Which parts of the application users find difficult to understand and why In participating in this evaluation you will be asked to e Listen to an explanation of the application e Create a UOL with guidance e Try to create a UOL of your own e Take notes on the problems you experience and things you do not understand e Explain the problems you experience to an evaluator When convenient we will make an audio recording of the key points as you explain them e Fill in the evaluation form at the end of this document TENCompetence IST 2005 027087 Page 43 48 DoS D6 2 Annex 1 Validation testing and usability results E competence Instrument used at Liverpool Hope University July 14th 2008 What are the evaluators expectations for the session e This is the first session of its type that we have carried out We don t know how difficult the activities will be for people who don t know the application We will be delighted if you manage to make new Units of Learning which will load onto the server but it is not a problem if you do not e The important thing is that we capture the problems which appear As you are listening and working please t
56. not saying that this is weren t you stupid not to realise I m just saying It s not the software that s the problem No I think it may be the software that s the problem it s because we are thinking that way round rather than If we d labelled them properly in our own files going on more meaningful names it would have been more obvious then This is a new thing that we ve put in Making it inverted commas easier it sometimes creates problems but if you impose the kind of methodology of creating the files first I guess what I was thinking was you create the environment and we want these environments with this in the topic and we forgot to tick these But if it was listed here that this was part of this Page 27 48 FAA PTEN Competence Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 have you done all these things to check up So you can t accidentally miss Cos we forgot to Well I m surprised You might want an activity that didn t have one just the text environment talk to your neighbours then you wouldn t want one of these I guess 1f there was like a condensed folder that popped up and said what was in it instead of having to click on all these OK so you want a kind of cleverer inspector for t
57. nt to file and file save as but it doesn t look like it s saving the resource it looks like it s saving the entire thing If you want to That s the web link let s take that out I missed it Let s say No So this resource at the moment says To do That s all 1t says so 1f you replace that no harm done If T wanted to create one If you wanted to create one from scratch what you would do you would go to resources and you d create a new one Oh OK It s a pretty horrible name So long as you give it an html as a file name then it ll come up like that So then you can save that Where s it gone There it is So you need to go through the two stages Now 1f you go back to here It to the list There 1t 1s It s two ways round If you do 1t this way round you can create a new resource here It doesnt like spaces And you can edit 1t here I see And then it ll appear here another new one somewhere It ll be here And the resources can be HTML files Yeah And PDF documents Well 1t can be anything you like Anything The only thing 1s that the runtime system has to be able to understand it If you give it something truly exotic it ll just throw up an error OK This is I think a really tricky thing to understand You ve got the files here the resources here and here you can do the same thing you can create them in which case it kind of works backwards creates the resource and th
58. occasions This would enable users to have access to a UOL and to register on it themselves The comments of these users and others at the session suggest that this should be a priority task Page 39 48 xe PTEN Competence D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 18 Need to assign teachers to learning activities User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator Userl Evaluator User Evaluator User Evaluator User Evaluator TENCompetence IST 2005 027087 The other thing that confused me a little bit with the Dylan Thomas one I was only seeing one thing and I thought that there were more things that I d added that I couldn t see and you certainly had more But you kind of whizzed on and I didn t get round to mentioning that When I logged in I didn t see what I thought I ought to see That s all I got So OK run support activity So I m assuming that I accidentally only allowed them to do one thing each and I hadn t realised I d done it Yes and the other thing you can do is if you put the condition for this that is you can t see anything until the teacher has clicked it but you don t allow the teacher to click it then that would do it Yes that would be why Where s it all gone You see you would be ex
59. pecting to find a little box here when this activity is completed But that s my guess that this is what s happened here That could be really disconcerting for the students What that you don t see anything else below I think that if there is anything else it might be nice for there to be a text that says you know this module is released in portions and this activity will be released in a certain amount of time or something Because I went on that and I thought Oh crap I ve broken it You could have some greyed out ones Well yes so that they know its there but they can t access it Because that doesn t look like its working properly And I d ring up and say I don t really know what I m doing Well You d probably want the first activity The support activity would need to say the first learning activity would say You see I thought I d said You haven t given the teacher access to the learning activity you see that s something that I think often do You assume that the teacher will have access to the learning activity Oh you assume that Louise Louise is the teacher No no Terry is the teacher Louise is my student No Louise is the teacher because you ve made her a teacher cos she s here as a teacher How do you know she s a teacher because if you enter as Louise again Log in It says Role teacher So you must have assigned her as teacher Oh so I did I couldn t remember who was who That s why I always
60. quisites default User When we were in the prerequisites putting the second prerequisite in my brain automatically went to opening up that drop down box and the resource and changing that to prerequisites Evaluator OK so you are looking to put in the default one again User So is it possible or would it make any sense to have options where it said that the original one said prerequisites and the second and third say pre requisite 1 or pre requisite 2 because I think other people s brains may well make that link Because it seems like it is over writing it Another user Because the resource to me means something you access User Because the word says prerequisites my brain says that what you have to select next Comment The resource structure in IMS LD involves the creation of files which are associated with resources which are associated with items While this is logical and has advantages in maintaining UOLs it is too confusing for authors if presented to them directly These problems arise from users being confused about what it is that they are editing They appear to feel that they are editing the element directly but they know that they are not because the interface tells them that they are working with resources But their understanding is partial because they do not discuss the underlying file structure TENCompetence IST 2005 027087 Page 21 48 FAA PTEN Competence Recommendation D6 2 Annex 1
61. re The other two although having a correctly checked UoL cannot upload it the UoLs are attached so that they can be analyzed by the tool creators Section 13 14 Opening Existing UoL and Searching UoL in the repository OpenDocument net All the designers agree with the functionalities regarding the OpenDocument Repository They can easily find and use it They also download a UoL and open it without any difficulty Section 15 Connecting to Sled Sever and Running a UoL All users can connect to the Sled Server Nevertheless only the user that hasn t had any trouble uploading his UoL can navigate through it TENCompetence IST 2005 027087 Page 6 48 D6 2 Annex 1 Validation testing and usability results xo E competence TENCompetence ID 6 5 Validation testing and usability results 3 Evaluation of ReCourse and SLeD with the Widget Server at the University of Sofia 3 1 CONTEXT OF THE EVALUATION The evaluation took place from May 22nd June 2nd 2008 at the University of Sofia A total of 12 learning designers from SU followed the worksheet four University researchers familiar with LD five teachers already followed a course about LD but with limited practical skills and three lecturers experienced in working with various e Learning tools but not familiar with LD 3 2 INSTALLATION It is not clear which versions of the JDK and NB are OK nor how one user can re use the existing Java installations and when and
62. rification to interpret the listed errors e The users disagree in deleting the activities They didn t understand that they have to delete the resources associated to an activity for completely deleting it e Only one of the users was able to upload the UoL successfully at the testing time The three UoLs created in the experience are attached as evaluation data Phil or Paul may check what s wrong with them QUESTIONNAIRES RESULTS Usability Checklist Section 1 Functionality The three designers agree in all the aspects regarding this part Just in the security comments they have no opinion Section 2 Reliability All users agree with all the reliability aspects Section 3 Usability Two of the users disagree with all the usability aspects They found that it is not very intuitive this might be probably because they were used to the previous Reload version In the contrary the other user not familiar to Reload considers the tool strongly understandable and learnable He generally agrees with the attractiveness of the tool Section 4 Efficiency All the users agree with the efficiency of the system maturity fault tolerance and recoverability TENCompetence IST 2005 027087 Page 4 48 xo E competence D6 2 Annex 1 Validation testing and usability results Section 5 Maintainability All users disagree with the analyzability of the system They asked for help to the worksheet authors when trying to so
63. rs involved I think you d have to be quite a good user of that to do that Whereas it might be better for the play to be week one then the acts in it are session one session two session three It d be too complicated to have the whole module as a play for someone who wasn t a very sophisticated user of I don t know it would be nice to be able to do them Perhaps 500 activities in a module that went on that long with that number of people plus two sites I guess it depends how complicated the thing is that you re doing Maybe that can be advice for people who are at different levels of You can organise these things in folders That s fairly easy TENCompetence IST 2005 027087 Page 38 48 FAA PTEN Competence User Evaluator User Evaluator User 2 User 1 User User 2 User Evaluator User User 2 User 1 Evaluator User 2 Evaluator User 2 Comment Recommendation TENCompetence IST 2005 027087 D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 Yes that s something we were asking about That s fairly easy can t be same I m putting it in the wrong one So you can organise this Yes that would help You would have week one week two You d have to do it like that wouldn t you to make it Possibly You d end up with a lot of environments though because you have to create them all on one Yeah
64. scriptions Secondly there are IMS LD Units of Learning which support multiple roles and complex learning flows created using ReCourse Thus ReCourse is not designed to support the simplest learning activities and it is expected that authors will have to make some effort to understand the extensive functionality which is available to them in the application and an interest in pedagogic modelling On the other hand they are not expected to have more technical knowledge than that required to publish a simple web page In this ReCourse is similar to other powerful authoring applications such as the widely used Adobe applications Dreamweaver and Photoshop It is to be expected that users will not be able to sit down with ReCourse and be immediately able to work with the application and will require some initial input either through documentation or through training Respondents The evaluation was carried out at a four hour workshop on the 14 of July 2008 with eight teachers from Liverpool Hope University and St Helens Community College Two evaluators participated Dai Griffiths a member of the project team at IEC the University of Bolton and Mark Barrett Baxendale an expert user from Liverpool Hope University Method In previous evaluations of both the Reload LD Editor and ReCourse respondents have been supported by providing them with a step by step guide to the creation of a Unit of Learning which they can follow This is valuable for id
65. ser because it s going to be a very long list of things especially for longer courses There were also a number of comments on the difficulty of the IMS LD concepts and structure e Clarification of the terminologies for novice users e Conceptually difficult prefer paper based notes e A bit complex terminology used e Not keen on the term Play Seems strange for what we are doing Not sure what term should replace it We now summarise the conclusions drawn from the transcripts regarding resources and the concepts of IMS LD 1 Resources 1 1 Attention should be paid to improving the support for users in handling resources as discussed in the following points Transcripts 1 10 1 2 Usability would be enhanced if users could edit resources directly without having to worry about whether files are being over written and other similar file TENCompetence IST 2005 027087 Page 13 48 DoS D6 2 Annex 1 Validation testing and usability results E competence TENCompetence ID 6 5 Validation testing and usability results management problems Ideally such issues should only be visible when carrying out resource management tasks Transcript 2 1 3 Itis unclear what giving a title to a resource associated with a Forum implies Does it change the title of the resource itself or the file or does it give a name to the reference to the resource Is it for the benefit of the author to remind them of what the resource 1s for or will
66. tence Instrument used at Liverpool Hope University J uly 14th 2008 Please look at your answers to questions 2 1 to 2 6 a If you had a difficulty which was not covered in questions 2 1 to 2 6 please write it down here b If in your answers to you questions 2 1 to 2 6 you have indicated that you had problems please add any comments which may help us to understand your difficulty more clearly and any suggestions for improvements which you have Please indicate the number of the question which you are writing about TENCompetence IST 2005 027087 Page 47 48 D6 2 Annex 1 Validation testing and usability results xo E competence Instrument used at Liverpool Hope University J uly 14th 2008 Please briefly describe any aspects of ReCourse that you particularly liked This includes the purpose functionality and interface of the application Please briefly describe any aspects of ReCourse that you particularly disliked and any suggestions you may have for improvements This includes the purpose functionality and interface of the application TENCompetence IST 2005 027087 Page 48 48
67. thinking of this as a template This user is thinking in terms of specific runs when designing their Unit of Learning They do not find it natural to think in terms of a plan which can be run a number of times and do now understand why the scheduling must be done in periods rather than by dates IMS LD was designed to create plans which can be reused and this is also the aim of ReCourse Consequently it is not appropriate for ReCourse to change this focus although other applications could This conceptual difficulty should be clearly and simply analysed in FAQs training materials and workshops 4 4 3 Provisioning and workflow 17 Splitting the course into sections User 2 Evaluator User Evaluator User Evaluator Evaluator Typically because we would have like a 24 week course and then you might have three things going on per week and within those things you might have three acts a week p raps even that s a lot of things so if you had all your things rolling down there imagine how many you would have You can nest these things You can have a UOL inside a UOL which refers to another OK so there are slightly tidier ways of doing it and you can have different plays but yes I see what you mean There is a limit to how many files you keep in your head I think I would go a little bit mental if I tried to put my entire lets say first year module on that because that d be there s three sessions a week four tuto
68. to them if files had been added or not It would enhance usability and clarity if the resource names generated by ReCourse were related to the names of the original files 11 Understanding the relationship between environments and activities Evaluator 1 User Evaluator 1 User Evaluator 1 User Evaluator User Evaluator Comment How are you getting on Not so well I understand what I what I want to achieve I ve written down what I want to achieve I ve got an activity and I ve set a question And the question I ve set this activity Now I want link to some resources to that activity Have you got any environments here The environments 1s where you create sets of resources learning packs if you like and they go in learning objects You need to give the resources to the learning object OK And then in here the activity in the environments you already have allocated it because by default there is only one environment so it must be that one So when the person gets into the activity they ll see whatever is in here So in environments we now need to put into this the resources that we want Either we assign them from here which is what I did with the poems or we edit is as a piece of HTML and put the links in here Does that make sense It does make sense It s the idea that the files that you re going to use as your learning materials sit in the environment not inside the activity and then you
69. ts from evaluation activities carried out with the TENCompetence Learning Design Toolkit described in that deliverable The evaluations were carried out in Liverpool UK Sofia Bulgaria and Barcelona Spain in the summer of 2008 The focus of the evaluation was principally on the teacher and learning designer tasks in authoring and publishing Units of Learning Of the three evaluations the first two at Barcelona and Sofia were carried out and after May 22 The third that at Liverpool Hope University goes into greater depth moving away from scripted user activities and containing focused interviews with participants on the problems which they experienced providing a more authentic and detailed picture of user experience Consequently the report on the latter evaluation is rather more detailed than for the other two There are two principal results 1 While working with IMS Learning Design LD is demanding for teachers and learning designers it is by no means impossible The Liverpool evaluation indicates that users with pedagogical experience who are also competent users of information technology but not technical experts can become independent users of the applications if they are given four to eight hours support 2 The evaluations showed that the creation and management of resources was a particular difficulty for users and the insights provided by these evaluations led directly to major changes in the Toolkit In considering
70. ty The resources associated to an activity have to be erased before completely deleting the activity itself Section 7 Creation Role s element Any of the three designers use the Role tab to create a new role and there is no information about this point Section 8 Creation of an Environment All users agree with the functionalities related to the Environment creation All the designers successfully carry out the activities regarding the Environment creation The especially like the palette functionality Section 9 Resources The users disagree with the functionalities related to the resource management They don t understand which of the options they should use from the various available However they success when creating the resources and filling their main elements See also the observations in the first page of this report Section 10 Completion of UoL All the designers agree with the functionalities regarding the completion of UoL Section 11 Checking and Validating the UoL This is one of the critical points of the test The users disagree with the Checker view form the Window Show view because they cannot find it easily Only one of them uses this option The other two use the button on the toolbar Section 12 Packaging and Publishing the UoL Two of the users have problems finding the link for packaging the UoL because it was covered by other functionality tabs Only one of the designers can upload his UoL to CopperCo
71. u want it every time you publish it you want it to work You can t be general and specific at the same time No I m not seeing this as a template I m seeing it as a single run It s not going to make sense There s no way of when you do decide to run it that that one event you can actually have something which says this starts on this date You could design you could change the system so that you could type in the day I suppose I m asking for a lot Even then the logic of this is that it is design once and run a number of time OK Unless you do something like I said don t start this until I tell you to watch out for the forum You know which is actually what you d probably do So potentially what you could do is have an extra user and the user would be the person in overall charge That s the kind of supporting role that you would have Page 37 48 FAA PTEN Competence User Evaluator User Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 So you would publish it ready but you would be the only one who could access it You could have a welcome to the thing this course will start on And then the over arching user is the only one who can click to say that this has started That 1s the kind of thing you can do yes So long as they don t get run over by a bus Yes I think that is the problem not
72. understand glossary of terms would be helpful That developed in UNFOLD might be a good starting point for this b Learning objective should be added to the list of user definable terms 13 Resources and environments User User 2 User Evaluator And you could say do the following portfolio activities and you would have support instructions in there Well they might not need them so they are simply available and you might have three or four activities each of which is a portfolio activity and at any stage those programming support files would be used for that and they would have to sit in the environment wouldn t they Are you grouping lots of things in sort of one environment it always confused me Yes that s what I m saying I think that s what s confused me as to why I would want it Why would you want what TENCompetence IST 2005 027087 Page 33 48 FAA PTEN Competence User Evaluator User Evaluator User Evaluator User Evaluator User Comment Recommendation D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 And to be honest I ve completely forgotten completely forgotten where we put Dylan Thomas and his bits and pieces I think that s probably what it is We put him in here In resources We dragged the files on here Yes we did and then we associated that with environment Was it learnin
73. ve already used this once It seems that the user is unclear what giving a title to a resource associated with a Forum actually does Does it change the title of the resource itself or the file or does it give a name to the reference to the resource Is it for the benefit of the author to remind them of what the resource is for or will it be shown to the learner Unless there is a strong argument for providing the user with prominent access to this element it should be made less salient It could perhaps be available as an expert mode or through the resources tab of resources Summarising conversation so far Here what s happening is you ve got this activity and you want to provide some web links for them to use in that activity That s going to go in an environment Your instinct is that the links go inside the activity but they don t go into an activity they don t go into an activity they go into an environment which you share with an activity But TENCompetence IST 2005 027087 Page 22 48 FAA PTEN Competence User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator Evaluator 2 User Evaluator 2 User Evaluator 2 User TENCompetence IST 2005 027087 D6 2 Annex 1 Validation testing and usability results Instrument used at Liverpool Hope University July 14th 2008 unfortunately this gets complicated b
74. with all the aspects They don t have any problem with any of the installation steps neither with the creation of the environments the configuration of the browser or the activation of the CopperCore engine Section 1 Creation of Unit of Learning UoL The three designers agree with the management of the Units of Learning They don t have any difficulty in creating deleting cutting copying or moving to a folder a UoL Section 2 Creation of the Main Elements of the UoL They also strongly agree with all the aspects regarding the creation of the main elements of the UoL They can handle the functionalities related to the Play the Act and the different type of Activities and with those related to the Roles and Groups Section 3 Filling the Main elements of the UoL Once created the main elements of the UoL they fill them They don t have any problem using the elements in the Unit of Learning Tab in the main window for defining the main elements of the UoL TENCompetence IST 2005 027087 Page 5 48 xo E competence D6 2 Annex 1 Validation testing and usability results Section 4 5 and 6 Creation of a Play s Act s and Activities elements The three designers make use always of the properties tab for defining the elements of a Play an Act or an Activity and they all generally agree with the functionalities regarding to these Only one of the designers expresses disagreement with the Activities delete functionali
75. y 14th 2008 16 Completion and relative and absolute time User Evaluator User Evaluator User User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator User Evaluator TENCompetence IST 2005 027087 So what I need to do is take it off that and put it on there which is fine So user choice then That means choose to click that you can go on Right so this then would be appropriate to say to them you ve got a week So what s months So why isn t there a week in there Cos you ve got days Can I type Hours and minutes even So if you do that then how does know Does it do it from the first time you go in it gives you exactly seven days It ll go from when the unit starts running from when you make it available to them Right so its not to do with when they log in its to do with when you free it up when you post it to them Yes you Say I ve launched this course you have eight days So how do I do this It may already be in there but how do I do this so I don t make it available yet So if say the welcome activity is the first week It becomes available when you set up the run so you put it on the server but it s not available You click start a run and 1t becomes available But that s the launch of the whole unit isn t 1t But what 1f this unit is going to run

D6.2 – ANNEX 1 – Validation testing and usability results

Contents

Download Pdf Manuals

Related Search

Related Contents

D6.2 &ndash; ANNEX 1 &ndash; Validation testing and usability results

Contents

Download Pdf Manuals

Related Search

Related Contents

D6.2 – ANNEX 1 – Validation testing and usability results