Home

1.3 Struc - POLITesi - Politecnico di Milano

1. gt Was it a lot of trouble getting the kids to stand in the stage area Did you have to always remind the kids about it v Jill and Ashley did not face much of an issue The kids understood well v 148 However Jess suggests there was a tough time making the kids stay in that area Needed to mark X on floor because even in our marked area it was not reading the actions Kids constantly moving made hard to do solo 5 Final Analysis gt Did you pick one kid at a time turn taking letting the other kids watch at the back of the class or did you let them play all together v This could be followed in both Jill and Ashley s class but was difficult in Alysha s class gt What did the kids do if their gesture wasn t recognized Did they get frustrated and stop or did they wait for it or did they talk to you v v v Ashley s class had problems with the remote controller They tried pretty hard nodding etc but when it didn t work they let go Jill mastered the remote controller so the kids didn t even notice any issues with the gestures Jess points out kids especially Alistor were frustrated in Alysha s class when the gestures did not work This was aggravated since her remote too wasn t working gt What did the other kids in the background do when their peer was playing v v Jill and Ashley s class Kids were cheering and singing in the back and telling the
2. 1 bad recognition of the joints 5 of the cases due to room lighting condition and recognition of other people inside the play area 2 unstable code around 70 of the cases caused by bad coding of gestures and sequential order of them i e zoom in has some segments of clap thus the recognizer output zoom in while I wanted clap 3 different understanding of gestures around 25 of the cases For example a nod nod gesture means to me that the child has to move his hand from right to left while teachers think the child has to move his head from backward to forward Solutions can be easily found For the 5 of the cases the bad recognition caused by many people inside the playground can be solved with a better approach in understanding the first active player figure a while the bad recognition caused by room lighting can just be unraveled by a better resolution of Kinect camera figure b For most of the cases 70 we should deeply focus on understanding the sequential order of gestures counting on well coded frameworks figure c or passing to the gesture recognizer only the gestures we want to recognize fig d The 25 of the cases is just a misunderstanding of gestures From the early moments in the Lionheart School we could easily create ad hoc gestures while when we will try to deploy the system in the real world a new solution must be found A possible one could be to let teachers creates their own g
3. The story is the view of the game To what extent the story should be able to abstract everything from the code side and show the play background objects feedbacks sounds characters mC1 STORY This is just a concept though more details will be given in the design and implementation section As you can read below the common requirement from teachers has been that the story should be interactive and affordable 13 For some children the concept of being inside the story is just too abstract for their current level When the narration paused to prompt and wait for them to perform the appropriate action The teachers discussed the possibility that she didn t expect the system to be interactive but simply an animated progression through the story with musical narration mC2 AVATAR The concept of virtual avatar inside the playground took the idea from some trials with Federico the autistic children with who I spent some time at school during lessons and playtime He is particularly out of the world but playing with Adventures a game made for Kinect XBOX gave him the connection he needed to understand and feel part of the game He reacts well to the way that the game is like a virtual mirror and the system encourages him to move because his movement then has an immediate visual effect an instant feedback to him Moreover teachers brought up the stick figure and posited that seeing the stick figure m
4. foreach var skeleton in skeletons skip the skeleton if it is not being tracked if skeleton TrackingState SkeletonTrackingState Tracked if closestSkeletonIdx 1 closestSkeletonIdx i secondSkeletonIdx 1 else secondSkeletonIdx i i To track the second player we start by initializing some variables closestSkeletonIdx secondSkeletonIdx i Then for each people who is in he playground skeletons we analyze person by person skeleton and understand if it is tracked in that moment SkeletonTrackingState Tracked up to 2 people can be tracked If the first player has not been saved I save its id and the second player is null otherwise I save the second skeleton id Obviously the JSON file to send to the JS via WebSocket will be changed as consequence Now the file will have two lists of skeleton joints along two ids and a gesture for the first player When the second skeleton is not tracked the file will simply have information of the first player 132 4 S COOL s Iterative Development Final conceptual model Children play all together in a turn taking way Each child has his designated time to play and the teacher supports him noninvasively via the remote controller 133 4 S COOL s Iterative Development 4 1 8 Evaluation The 28 of October we installed the system and gathered some feedbacks The following is a 3 person report of that day This was also
5. or full body touchless interaction This paradigm exploits sensing devices which capture track and decipher body movements and gestures while users do not need to wear additional aides e g data gloves head mounted display or remote controllers or body markers Several authors claim that motion based touchless interaction has the potential to be more ergonomic and natural than other forms of interaction the gestural dimension resembles one of the most primary forms of human to human communication body expression body involvement can enhance engagement the come as you are feature removes the burden of physical contact with technology making the user experience more pleasurable Computer vision researchers have long been working on touchless motion and gesture recognition The recent evolution of hardware and software sensing technology e g by Microsoft and Intel as explained in chap 2 3 enables developers to implement applications at low cost and make them accessible to the consumer market While most commercial products are for pure entertainment the research community has started to explore motion based touchless games for children s education Theoretical arguments and empirical results suggest that these tools can be beneficial for regular children Still little is known about whether and how they work for autistic children because of the limited number of experimental studies and our incompl
6. COOL s Iterative Development mD TEACHERS CONTROLLER A teacher controller is the best way to override system s fails and customize story in real time There are different solutions for a remote controller but overall we need something fast with no delays easy to develop due to the oncoming deadline and wireless wearable since teachers do not need to move away from where they are mD1 REMOTE MOUSE Fast easy to develop but most of the time needs a wire Teachers are acting stories out with the children and they don t have the time to go to the pc mD2 MOBILE REMOTE CONTROLLER The remote controller prototype we brought to the Lionheart School was a nice way to let teachers understand the possible features to implement In some classes where the teacher sits down while the story plays it is possible to use a complete features application where the teacher can control the entire setting On the other hand as said before in classes where the teacher plays with children this application fails its main purpose of giving the control of the story Replay Play Pause PAGE Previous Next ACTION Previous Next x tot scenes 119 4 S COOL s Iterative Development Moreover our first remote controller was based on internet connection and database instruction storage as queues which causes a big delay in the communication speed However the mobile remote controller can be implemented also using a Bluetooth connec
7. On this basis the power of the internet to share contents in an easy and fast way and the possibility for teachers to create their own personalized contents must be considered 6 Updatable an easy to update software can enhance its spread of working and updated software and promote its use 86 4 S COOL s Iterative Development As a consequence of the above implementation requirements these are the solutions I found accounting difficulty in learning new software and resulting time loss gain on the basis of my deadline N SOLUTION CRITICAL TIME TO DO 1 3 DIFFICULTY 1 iteration COST 1 3 A STORY CREATION TOOL 1 3 MIT B 1 GESTURE DETECTION 3 2 YES B 2 MULTIPLE USERS 1 p NO DETECTION B 3 LEVELS OF GESTURES 1 2 NO C 1 STORY 3 2 YES C 2 AVATAR 2 2 JUST EXAMPLES C 3 TUTORIAL 1 1 NO D 1 REMOTE MOUSE 3 1 YES D 2 MOBILE REMOTE 2 2 JUST DEMO CONTROLLER E 1 TABLET 1 2 MIT E 2 SPEECH RECOGNITION 1 1 NO E 3 NFC 1 1 NO E 4 ARDUINO 1 p NO 87 4 S COOL s Iterative Development mB1 GESTURE DETECTION Nowadays Microsoft offers different ways of gathering Kinect data input and provide different samples in its SDK toolkit The most relevant programming languages to use with the SDK are C C VB On top of these main solutions some company built other ways and one of the best frameworks is OpenNI Python Unity 3d Node js Air The OpenNI framework is an open source SDK used for the development of 3
8. b8 b9 Crosland K Dunlap G 2012 Effective Strategies for the Inclusion of Children With Autism in General Education Classrooms Harrower J K Dunlap G 2001 Including children with autism in general education classrooms Blumberg et al 2013 Changes in Prevalence of Parent reported Autism Spectrum Disorder in School aged U S Children 2007 to 2011 2012 Dunlap G Iovannone R amp Kincaid D 2008 Essential components for effective educational programs Karagiannis A Stainback S amp Stainback W 1996 Historical overview of inclusion Fryxell Dale Kennedy Craig H Placement along the Continuum of Services and Its Impact on Students Social Relationships Hunt P Farron Davis F Beckstead S Curtis D amp Goetz L 1994 Evaluating the effects of placement of students with severe disabilities in general education versus special classes Wilde Koegel amp Koegel 1992 Increasing success in school through priming a training manual Zanolli Daggett amp Adams 1996 The effects of reinforcement on the spontaneous social initiations of socially withdrawn preschoolers 177 8 References b10 Koegel amp Koegel Frea Fredeen 2001 Identifying Early Intervention Targets for Children With Autism in Inclusive School Settings b11 Ryan Joseph B Hughes Elizabeth M et al 2011 Research Based Educational Practices for Students with Autism Spectrum Disorders b12 Lov
9. s10 http en wikipedia org wiki Xbox_One s11 http www youtube com watch v uuP6d42hK8k 12 http www educationaltoys com can xbox Kinect help autistic children learn 13 http www penny arcade com report article using Kinect and happy action theater as therapy how one school is reaching s14 http en wikipedia org wiki Nintendo_ wii s15 http en wikipedia org wiki Perceptual_ computing s16 http en wikipedia org wiki Leap_ motion s17 http www tobii com rexvip s18 http www ellipticlabs com 19 http www airwriting com s20 http eyesight tech com s21 http mauzup com s22 http www pointgrab com s23 https www thalmic com en myo s24 http Kinectsen wikispaces com s25 https www pinterest com schoolstaff Kinect for special ed and autism 180 8 References s26 http www reactickles org s27 http z vector com s28 http www visikord com s29 http www binaura net s30 https github com trentbrooks Noise Ink s31 http arena openni org OpenNIArena Applications ViewApp aspx app_id 428 s32 http www flight404 com blog p 482 s33 http www scratch saorog com s34 http somantics org s35 http www pictogramas org s36 http thelionheartschool com s37 http en wikipedia org wiki Design_ thinking s38 http en wikipedia org wiki Iterative_and_incremental development s39 http en wikipedia org wiki Authoring syst
10. 100 million dollar grant from Intel Capital In 2013 PerC launched their first product the Senz3D a Time Of Flight depth camera in partnership with Creative Technology The camera is similar to the Microsoft Kinect however it is targeted at gestural interactions within a shorter range PerC has released an SDK to help external developers build games and applications using the multi modal sensing capabilities of the Senz3D device Leap Motion Leap Motion s16 Inc is a company that a oc lt manufactures and markets a computer hardware sensor device that supports hand and finger motions as input analogous to a mouse but requiring no hand contact or touching It uses proprietary advanced motion sensing technology for human computer interaction The Leap Motion controller is a small USB peripheral device which is designed to be placed on a physical desktop facing upward Using two monochromatic IR cameras and three infrared LEDs the device observes a roughly hemispherical area to a distance of about 1 meter 3 feet The LEDs generate a 3D pattern of dots of IR light and the cameras generate almost 300 frames per second of reflected data which is then sent through a USB cable to the host computer where it is analyzed by the Leap Motion controller software using complex math in a way that has not been disclosed by the company in some way synthesizing 3D position data by comparing the 2D frames generated by the two came
11. Ashley s class Y Matching color of the hand to the object color Crossing gestures v Kids should be required to jump up to get objects high up v Suggestions from Ashley s students throw virtual things on screen like pass the monster s head v Try to reach the castle in the background FREEFORM Music in the background see them social interaction image up there On screen around grabbing another student to dance with them Free form on each other SNS They could see their peers on the screen and turned around and get them Occupational Therapist was not present for this session Additional discussion v Longer term effect This new found social interaction with or without the Kinect Will the effect stay or fade taper away Or the need for the game will be eliminated over time Does it wake them up Jill contradicts by saying that when the TV crew comes up we react to it even as an adult Is there something special about Lionheart Or can this be useful in other schools They suggested less in public sectors but a good tool for private and OTs How do we strategically measure these changes 1 Gregory suggested a pedometer on the kid to measure the movements 2 Video recording the sessions 3 A comparative study on response of the kids with and without the Kinect 154 5 Final Analysis gt Screen vs the Mirror An interesting discussion was would there be a similar effect if the child was sai
12. From these results a treatment plan is established using a combination of lower extremity and core strengthening programs During one single activity the player red silhouette has to match the shape behind him gray silhouette for a predefined se g duration or cover area Levels of difficulty depend on movement of coronal plane sagittal plane traverse plane SHAPE TO 5 MATCH upper limbs lower limbs and center of gravity A mix of these movements will create an higher level of difficulty PLAYER The game challenges balance eye hand co ordination and reflexes The graphics and sounds give this the semblance of a competition and even provide a score at the end of each run It becomes a social event when many children participate together 43 2 State of the Art 2 5 3 Kintroller While I was developing Pixel Balance I also made Kintroller The first idea was that Kintroller gave the possibility to use Kinect as a peripheral input device for the pc The body could control the active window and different gestures performed a reaction of the operating system To what extent a person just needed to run the application plug the Kinect open its PowerPoint presentation and enjoy swiping left and right to turn presentation pages However when I noticed that Kintroller could be something more than a motion based interface for the pc I started developing a JavaScript app based on Google Str
13. JS SERVER SIDE MYSQL PHP CLIENT SIDE C AND XAML CHANNEL HTML5 WEB SOCKET n e JSON JSON CLIENT SIDE JS HTML5 FRAMEWORK PHONEGAP FOR TABLET AND SMARTPHONE CHANNEL HTML5 WEB SOCKET JSON PI DI e CLIENT SIDE JS HTML5 FRAMEWORK PHONEGAP 92 4 S COOL s Iterative Development Relevant Code In this section I will explain the most relevant code choices taking into account the previous requirements Collaboration with MIT One of the peculiarities of the entire project is the collaboration with MIT as said before for a complete exchangeable format based on JSON To do this we first have to understand the difference between the touch interaction and the touchless interaction Using a tablet each object red is active and on that it is possible to trigger a touch gesture such as tap double tap rotate zoom in out Each touch gesture corresponds to an animation which is completely addressed to the same object Technically speaking each object has a listener attached to itself For example in this image above many children can touch many objects and each object will react to the child s touch 93 4 S COOL s Iterative Development On the other hand using a Kinect the listener is attached to the gesture to act This means that if the child performs the right gesture the system will listen to it and will activate an
14. Jill O More avatars Closet Wardrobe Vy o More games What kind of games do you want Farm Horses e The farm game How should it work Teacher selects images of the animals and activities that students recollect Occupational Therapist gt Discussing the shape games with the OT gt What kind of postures o One player two players 144 5 Final Analysis 5 Final Analysis Analysis has been conducted following teachers feedbacks logs and video In the first part we will fill out the form we prepared in the last section then we will focus on understanding usage and technical data using logs and finally we will try to find relevant differences between normal school lessons and activities using the system Teachers feedbacks Three systems were installed at the Lionheart School from October 28 2013 to November 11 2013 in three classrooms We began the session by asking the teachers to give an open feedback on their experience with the systems We have grouped their answers in accordance to our planned questions OUTLINE General feedback about their experience v e and ve Benefits Motor Social Verbal Drawbacks or issues Specific feedback on the technology Then we covered the additional features they want in the Story Game and Freeform one by one Teachers present Elizabeth Tamara Victoria Jill Ashley Researchers from Georgia Tech Agata Gregory Mirko Arpita Rene Teachers w
15. Single Player 2 YES 6 F 1b CATCH THE OBJECTS Multiplayer 2 YES 6 F 2 FREE FORM GAME 2 YES 9 F 3 SHAPE GAME just demo G 1 LOG MANAGER 2 YES G 2 VIDEO RECORD 3 NO i G 3 SURVEY amp MEETINGS done in part G 4 AUTOMATIC MEASUREMENTS to decide Other tasks to do e teachers aware of the playground designated area e buy some Kinects to leave them behind 129 4 S COOL s Iterative Development Implementation of Logs manager mG1 As explained in the design part mG1 I found useful to create a log of failures and winnings of the system Since the entire application story and games works with JavaScript and html a solution on this basis had to be created As the complexity of JavaScript applications increase developers need powerful debugging tools to help quickly discover the cause of an issue and fix it efficiently The Chrome DevTools s44 include a number of useful tools to help make debugging JavaScript less painful Since I developed using Crome DevTools and debugged using the console given my first idea was to simply export in a text file the console log But JavaScript is not allowed to access hard drive so I should have used an ActiveXObject and I decided to quit this idea Thinking to the future development I will need a monitoring system online which store my logs and under a data mining process shows the analytical data I want to track So I opted out for something in the middle between writing a text file on the
16. They added that they have a Kinect at home but he likes watching and hasn t learnt motor skills Parents tried to encourage him and play with him but very he gave very less response They said he likes Christmas music We refreshed the game many times and he finally gets excited when he sees that his pink balloon is also detected on the screen He starts laughing and tries to move his balloon but stands at the same position He doesn t pay much attention to the objects appearing on the screen and is interested in his balloon on screen He starts crying when the parents finally say we need to leave and take him away We had two more little kids of 2 3 years of age The objects appeared at a height that kids that small were not able to reach A lady picked up the kid on her lap and was playing while she was asking her kid where do I go now Tell me where the lion is The kid did respond after a few trials Then she also asked her kid if he will play now without mommy and the kid tried some movements She thanked us Some parents tried and said this is great But he just doesn t like to play Parents did well but for the shorter kids objects at the bottom top and corner of the screen were difficult to reach We had to tell the player to shift back at times while they were trying to come towards the screen to get the object conflict in perception Some kids didn t listen and kept coming forward 111 4
17. Vocational Skills Sensory Physiological Motor Functional Assessment Diagnosis Screening Intervention Education Scientific Assessment Parent Clinical Training lt Person with Autism Family Caregiver Peer Clinician Therapist 4 Researcher Educator Home lt School lt Research Lab Clinic Community Autism Specific Social Behavioral Science lt Computing Education Medical Descriptive lt Correlational Quasi Experimental Experimental azi D S S bar B AZo OugIa L FAIe1IU uewog Jasp pug jessie Ay o a Db z lt oddns jeaduy Concept Non Functional Prototype lt Functional Prototype Publicly Available 37 2 State of the Art Technology is just a tool In terms of getting the kids working together and motivating them the teacher is most important Bill Gates 2 4 1 Touchless Technologies and Autism b24 b25 b26 b27 s41 In the last years we have seen an increasing number of technologies in the research literature and the market place that adopt game based learning to promote various skills of children with autism Existing products and prototypes support a variety of interaction modes and are designed for different platforms and input devices from conventional mice or joysticks to multi touch gestures speech recognition devices digitally augmented objects or robots Limited research in this arena has explored the potential of motion based
18. action 102 4 S COOL s Iterative Development S COOL Beta This second iteration starts by explaining the feedbacks we got from the Lionheart School and Gwinnet Race for Autism and to settle down new requirements to work on developing set aside modules of the first iteration and bettering the considered ones After I explain our choices of design and related implementation Finally I will show the evaluation phase which step by step require more structuration 4 1 5 Requirements From both Lionheart School and Gwinnet Race gathered some useful feedbacks which we have organized as following two reports Lionheart School During our visit we have been asked to accommodate in a meeting room where we can show what we prepared A teacher was there to answer all of our questions and provide us feedbacks Some other teachers didn t have the chance to be involved in the entire meeting and attended just a part of that After a brief introduction of the school and its organization we started asking we in normal font teachers italic 1 When you do stories in class what kind of interaction do the children do And what do you want your child to say what do you what him to do Teachers try doing stories that think affect the children like scary The very first thing they do is getting the child familiar with the story using interactive whiteboard Promethean IPad written text and whatever it might be As they are into i
19. identified any particular practices or approaches that work equally well for all children that work better than other interventions or that lead to improved outcomes beyond their specific target area Interventions that have been shown to be effective have been based on broad practices tailored to the individual child rather than a rigid set of specific treatment strategies or methods Using meta analysis to analyze a broad range of studies several researchers have identified some common underlying characteristics of effective ASD interventions and some relevant practices that support those interventions These educational practices include e Applied Behavior Analysis ABA e DIR Floortime e the Picture Exchange Communication System PECS e social stories e TEACHH Applied Behavior Analysis Discrete Trial Training ABA DTT Intervention that focuses on managing a child s learning opportunities by teaching specific manageable tasks until mastery in a continued effort to build upon the mastered skills This approach requires 20 20 hours per week across settings Researches demonstrate DTT increases levels of e Cognitive Skills e Language Skills e Adaptive Skills e Compliance Skills e IQ e Social functioning 15 2 State of the Art Developmental Individual Difference Relationship Based Model Dir Floortime Through challenging yet child friendly play experiences clinicians parents and educators lear
20. they use movement to create visuals and sound without needing the pupils to use equipment or to learn any specific skills first They are natural and intuitive 167 6 Conclusion and future works It is hard for people who do not understand the complex needs of Autism to understand the significance of enabling and motivating some of our pupils to interact meaningfully with anything in their environment The more enabling technologies we use the more chance we have of finding something that will increase motivation creativity interaction and movement when other traditional methods have failed The Kinect is not replacing these other methods but adding a whole new level of opportunity to the tools at our disposal I am certain that given the resourcefulness and inventiveness of the special needs teaching community worldwide and the support of the business and technical community that these opportunities will multiply and flourish 168 6 Conclusion and future works 6 1 Towards a framework Further work is in progress to test the platform s ability to automatically collect and analyze data and to create a modular architecture that will make it reusable and adaptable for different developers and kids needs S COOL is based on a modular architecture as explained in chapter 4 From a developer point of view the c application will be seen as a black box that output gestures joints and recognized voice commands The deve
21. top to bottom e zaxis depth near to far ANKLE RIGHT ANKLE LEFT 100 JSON from KINECT 4 S COOL s Iterative Development The common exchangeable format is JSON on a WebSocket channel As you can see below each joint x y z is passed along the gesture if recognized and the player id tidu tin gesture type WaveRight from Kinect Tormes sf AnkleLeft z 0 4041011 MyM eNaO 30 76425 Meher 529114 AnkleRight ny y 21 3023 71 WoW z 0 564546 0 06565069 hy ElbowLeft memeng 1352711 y 0 05124895 annA 3963879 by ElbowRight Maan 1376931 y 0 06701304 atte 3930579 by FootLeft x 0 4578435 y 0 4110892 z 1 640476 by FootRight Mace 6099673 y 0 04129197 Tall Wi 4687505 by HandLeft tong 26585970 y 0 1266259 tato 7371287 he HandRight tate 28035602 y 0 01907948 z 0 6756439 ky Head Meets EO 109869238 y 0 03575548 monen 3935167 hy HipCenter Wate JOLT 105 mymans0 1549605 a 1 015222 he HipLeft Wy 4287141 moyens 1323563 Ta Vl L74007 he HipRight T 0 2952475 myty 07 12956875 amn 8561926 Fy KneeLeft Vem 0423196 Wes O 3296682 tata TLB063 ty KneeRight wx 5032026 Wy eta 0 226159781 Ware Al eo Bi ShoulderCenter Teme ToT al O
22. 160 5 Final Analysis As you can see Alysha preferred to show video hints to her small and low functioning kids while Jill and Ashley wanted their medium and high functioning kids respectively to think about the gesture to perform without suggestions Story enabled video hints per class 80 70 60 50 m ALYSHA 40 E ASHLEY 30 E JILL 20 10 0 During the days Ashley and Jill had been helped considerably less while Alysha remained almost constant in her aid As the table below shows Jill decreased her aid of 25 of the time and Ashley for the 40 going to not help her children anymore This is possibly due to remote controller failure for a negative side and a big increase of children winning on the positive side Story Teacher s aid during the story per class 0 ALYSHA 8 ASHLEY JILL 161 5 Final Analysis The Little Old Lady story has its peculiarity in repeating gestures as a cycle The story starts asking for a walk than for a walk and a wiggle than for a walk a wiggle and a shake and further on asking at the last page both 5 gestures in order Moreover this table below can show not only the difficulty in kids performing a gesture but also the problem of the system recognizing the right gesture During the focus group we also understood that teacher sometimes wanted to focus on more precise gestures and quickly skip the gestures at the beginning Story helped g
23. 34 in and the vertical field is 63 cm 25 in resulting in a resolution of just over 1 3 mm 0 051 in per pixel The microphone array features four microphone capsules and operates with each channel processing 16 bit audio at a sampling rate of 16 kHz 23 2 State of the Art The new Kinect 810 Xbox One consoles the upcoming will ship with an updated version of Kinect the new Kinect uses a wide angle time of flight camera and processes 2 gigabits of data per second to read its environment The new Kinect has greater accuracy with three times the fidelity over its predecessor and even the ability to see in the dark thanks to its new active IR sensor It has an up to 60 wider field of vision that can detect a user up to 3 feet from the sensor compared to six feet for the original Kinect and will be able to track up to 6 skeletons at once It can also detect a player s heart rate facial expression 25 individual joints even thumbs and the precise rotation of such joints the weight put on each limb and the speed of your movements and track gestures performed with a standard controller The principle of operation of the Kinect 3D camera is described in US patent 5081530 Skeleton FPS 30 07 di i Skeleton Orientation Muscle Force Heartrate Expression 24 2 State of the Art Why Kinect s11 s12 s13 While the educational value of video games has been debated for years and whether
24. A functional prototype has been developed and has been used by the intended users for the target purposes It has been built for use by the developers to answer specific questions but may require assistance with setup use or maintenance Publicly Available The technology is mature enough that it can be used without assistance from the developers or research team This might be a commercial product software that is open source or applications available for download on websites or on mobile marketplaces 2 State of the Art OTOZ Te 39 UAEM ZLOZ IE 19 UAalsayy A AA AAA A OTOZ TE 19 IEM 9002 Te 38 Jadig Z00Z Te 39 UN 6002 TE 12 zquary S00Z TE 12 77u31 ZTOZ Te 3U0H OTOZ TE 33 OuRITH 800Z Te 39 sadey 6002 Je 33 usadpeH 9007 TE 39 UMpoon 6007 Te I9 Jayes lag OTOZ Te 39 Uaques ULA 800Z Te 19 OJeNeL AA AA A AA SAN lt s lt s lt ax lt s s lt _ lt lt lt lt amp lt lt TR 9002 Te 39 Yaqoizq ZLOZ Te 3 Opaqoosy L00Z Te 39 Jadutumneg 9002 Te 39 Aqno rey a Val lt 4 E lt lt lt Jaded 6007 TE 39 TeUIqIy Personal Computers amp Web Video amp Multimedia Mobile Devices Shared Active Surface Virtual amp Augmented Reality lt Sensor Based amp Wearable Robotics Natural Input Social Emotional Skills Language Communication lt Restrictive Repetitive Behaviors Academic Skills Life
25. Iphone le pecs sono delle tesserine che mostrano il disegno dell azione che il bambino dovr svolgere I bambini autistici hanno il problema di astrarre dalla realt e di fare un collegamento logico tra la parola penna e I oggetto penna Mediante Finger Tools possibile creare le proprie pecs fotografando gli oggetti che dovr utilizzare il bambino Sua affermazione il disegno era comodo 20 anni fa ora comodo fare le foto II Dr Moderato andato a visitare alcuni bambini autistici in Eritrea e dice che loro hanno una sorta di purezza della mente Bisognerebbe quindi cercare di far spegnere le televisioni interne che hanno mediante della musica Consigli guardare video Temple Grandin una donna straordinaria leggere rivista Mente amp Cervello AUTISMO Un altra mente Tags agenda iconica pecs comunicazione aumentativa e alternativa caa Conclusioni mostrare a schermo le azioni da eseguire mediante pecs foto immagini permettere al tool di fare foto strutturare un buon archivio di pecs per tag musica ottimo strumento ma attenzione perch a volte succede che il bambino preferisca fare un azione sbagliata perch preferisce un suono piuttosto che fare quella giusta Disponibilit ottima Una delle sue affermazioni stata ho vagonate di bambini su cui far testare 57 3 Preliminary Requirements Ospedale Policlinico Milano 1 Antonio Bianchi Mirko Gelsomini ci
26. animation chain For example in this image above the child will act out red the story performing gestures that will be caught and the reaction will be an animation of different objects green To what extent the tablet listeners can be touch gestures sounds and NFC tags while for touchless interaction can be touchless gesture sounds collision detection and collaborative gesture Each listeners when catch the right action will activate animations and or turn of the page Ex For tablet on tap go to page x for Kinect on walk go to page y The story flow can follow a linear path 1 or a non linear path 2 An example can be seen below Ge Blo aan la SES Plo le For the touchless interaction I thought also about another important feature if the child performs the right gesture the system react with animations of linked objects but do not compulsory change page It changes the sequence number and remains on the same page This feature can be better understood by looking at the following image 94 4 S COOL s Iterative Development i i B uw uw sequence 1 gesture jump reaction move A show B hide A rotate B next page On these bases we decided to level out some critical notions A story as a storybook is composed by pages Each page has some elements inside which can be text images and sounds In particular text and images have also their position in a page Each element has its l
27. as web sites Google images YouTube Bing search Every single child is also pushed to do his best and receive different stimuli teachers voice sounds touchable objects visual images video In 2 classes we saw children acting out fairytales all together singing and dancing While many were completely into the story mimicking the character s actions some were distracted from other stimuli such as one kid was distracted from the pc screen and a teacher switched it off one other was probably excited by our visit and shouted loud for some time a girl was really focused on flapping a shirt which was a real gesture of the story but not the only one a boy was listening to something and performed just some of the actions After visiting classes we came back to the meeting room to meet some teachers and show them our current system As said before some teachers did not have the time to be there for the entire meeting but just for some minutes 105 4 S COOL s Iterative Development We explained new teachers the project in simple words ex Kinect is a special camera stories can be triggered by children s actions and show them how StoryScape is working the current authoring tool Then we showed the S COOL version of The Little Old Lady and the remote controller smartphone app in action After this first demo we gather some feedbacks They remember when some years they saw the same story and th
28. climbed inside to get warm s s s 7 A fox opened the mitten and tip toed walked motion on kinex inside to get warm s s s 8 A bear opened the mitten and stomped inside to get warm s s s 9 A tiny little mouse opened the mitten and wiggled inside to get warm 10 The mouse sat on the bear s nose His whiskers tickled the bear s nose Ahhhhhhhh achooooo sneeze SOUND decibels to make Kinect trigger mitten bursts open with animals coming out The need of the teachers to have a story every one two months will probably effects in a creation of an authoring tool in the next few months 157 5 Final Analysis Log analysis As deeply explained in chap 4 3 2 and 4 3 3 module G1 logs differently from teachers feedbacks which are more cognitive information can help us to understand time related data otherwise impossible to obtain Logs are simply text lines that require a complete understanding of the dynamics inside the class and of the entire system For this hard work I tracked around 10 thousands lines of logs to extract semantic information we needed Each class had its own installed system in its computer and we were able to differentiate statistics among the following three classes 4 Alysha 6 9 years old kids low functioning level 5 Ashley 10 13 years old kids high functioning level 6 Jill 10 13 years old medium functioning level Both classes played along two weeks 11 days of school wi
29. controls all of the parts of a gesture and which one is currently being checked It contains an array of IRelativeGestureSegment which are individual implementations of the IRelativeGestureSegment interface which I will mention later When a skeleton frame is created it is passed through to each Gesture which then passes it through to the current gesture segment When the final segment returns a result of Succeed it raises a gesture recognized event which is caught by the gesture controller 98 4 S COOL s Iterative Development The IRelativeGestureSegment is the final part of a gesture It is essentially the individual segments that make up a gesture Segment 1 hand above elbow l if skeleton Joints JointID HandLeft Position Y gt skeleton Joints JointID ElbowLeft Position Y 221 Sp hand right of elbow 4 if skeleton Joints JointID HandLeft Position X gt skeleton Joints JointID ElbowLeft Position X DE 6 return GesturePartResult Suceed as 8 hand has not dropped but is not quite where we expect it to be frame SE return GesturePartResult Pausing 10 ILS 12 hand dropped no gesture fails 13 return GesturePartResult Fail Segment 2 1 hand above elbow 2 if skeleton Joints JointID HandLeft Position Y gt skeleton Joints JointID ElbowLeft Position Y pausing till next Sire 4 hand right of elbow DE if skeleton Joints JointID HandLeft Position X lt skeleto
30. create a story all together sharing their thoughts The teacher builds the story on live considering kids responses and acting as consequence The main point here is that technology can t do anything without an experienced teacher while a teacher can do without technology but also can do better using it Therefore a real time system to create live stories came to our mind Today s platform is based on Promethean ActiveBoard but as I noticed there are some problems which distract children s attention particularly important when we speak about children with ASD One main problem is that the teacher needs to switch from the software to browser to find a good image If on one hand this is good because children look at many images and choose together on the other hand they are distracted by the switching from the software to the browser and vice versa Further development should consider this fact and maybe build something on this 114 4 S COOL s Iterative Development mB KINECT DETECTION SOFTWARE The Kinect detection software is one of the critical aspects of this project It must be enough ready to support joints skeleton recognition smart gesture detection and multiple users input mB1 GESTURE DETECTION Gestures are fundamental during the play of the story If the child perform an action and the system does not recognize it the entire story fails The causes of unsuccessfulness are mainly related to
31. face interaction style of traditional small group work with the enhancements of digital media Digital technology offers many benefits for educational activities in particular digital technology can help address the problem of having one teacher for many students When students are working on a small group activity the teacher can only assist one group at a time With digital technology however groups can still receive feedback regarding their progress even when the teacher is busy helping other students Allowing students to immediately know they have found the correct answer has pedagogical benefits and increases group efficiency Two factors of particular interest for educational applications is how they impact students levels of participation in the activity and how they facilitate awareness of one s own and others contributions Participation can measure either direct interactions with the software itself or the amount of foreign language conversation produced by each group member since both of these actions align with desired learning outcomes Therefore the point is do we really need our autistic children to be able to communicate with other children through technology or we need that our children thanks to the technology are able to speak with other children in the real world This is one of the challenges this work is going to embrace 45 2 State of the Art Human Computer Interaction HCI as Wikipedia pr
32. forest Jill on the Gingerbread man story It would be better if it read more than one person for the story for example when the fox comes or the pig comes Two characters with the story line one who is the ginger bread man running and the other is chasing Importance of the story O Elizabeth insisted on the fairy tales being solely important for the younger kids Very important medium of learning for Mary Katherine Jill and Alysha s class Younger kids cannot socialize Do not listen to music or watch videos She added that in the fairy tale framework the kids carry the picture in their head eg Kids have a dog at home but they will not talk about it unless they interact with a dog or see it at school Ashley s class doesn t do fairy tales 151 5 Final Analysis SPECIFIC TECHNOLOGY RELATED FEEDBACK Alysha s class had a broken dongle Alysha s angle of Kinect is also an issue change of camera gt Problems with Kinect v Jill Where there was a very long pause when she left the system on and go do other activities it doesn t read the room and they needed to restart v Ashley added that the little box shows that it s reading but when she pressed start it doesn t start v Alysha had problems starting the system and had to restart every time v Mirko suggested put your hand in front and then remove it v The logs suggest they clicked many times so there were multiple inst
33. gt CHOOSE THE AVATAR 2 2 CHOOSE THE THEME di START CHOOSE THE AVATAR CHOOSE THE THEME START START The GAME has 3 parameters to set DURATION time of play 30 secs 1 2 5 mins infinite AVATAR the avatar of the child Skeleton Live image THEME different background and objects related to the theme Jungle Mars The STORY has 2 parameters to set AVATAR the avatar of the child No avatar Skeleton Little Old Lady VIDEO HINTS Video suggestions of the gesture to perform Show Hide The free form has 3 parameters to set AVATARI 1 child s avatar Skeleton Live Img Princess MrBean Iron Man AVATAR2 2 child s avatar Skeleton Princess MrBean Iron Man THEME different background and objects related to the theme Jungle Mars 139 4 S COOL s Iterative Development Beside an easy to use first screen we provided an instructions sheet Camera height 6 ft max Implementation has been done to fix some errors which are non relevant to discuss in this document As shown in the previous page we developed an index page to orientate teachers in their choices 140 4 S COOL s Iterative Development 4 2 3 Evaluation Three systems were installed at the Lionheart School from October gg 2013 to November re 2013 in three classrooms We will begin the session by asking the teachers to give an open feedback on their experience with the systems OUTLINE General feedbac
34. is a computer genius but poor in motor skills o Kevin s wheel barrow run and jumping on the monster s head o They made Gracialla sit and do the movements o Kids explored their own ideas with the white screen Everybody down or everybody up 150 5 Final Analysis gt Game Vs Story v The game was a great success in Jill and Ashley s class but the story did not get them looked as much Here s why the teachers thought so v Jill In the game they enjoyed moving and get rid of animals in the games but in the story they spend a lot of time marching They are excited and waiting for the pants to come up and all that but that but most of the time they are just waiting so it doesn t hook them quite hard Y Agata worried that this is going to be with every story to which Jill says that s when the teacher comes in and tries to amp it up Y Elizabeth explains how Jill will have her class acting out the story putting on costumes and everybody would be involved over and over again in making it kind of a role play as opposed to one child playing it v Ashley suggests The fairytale has different settings so recollecting that one of her students was the princess and was trying to get the princess back to the castle when she got to the castle she was expecting that she would go inside the castle She thought that would be a good idea to transit to the next scene like may be walk out the door to go to the
35. local pc and saving the data directly to an online database the WebSql database The Web SQL database API isn t actually part of the HTML5 specification but it is part of the suite of specifications that allows us developers to build fully fledged web applications The API is well supported on Chrome Opera Safari and Android Browser which is enough since the application will work on a Google Chrome Portable browser I will not go inside the code and explain line by line because it is easy to find good instructions online while I would like to explain which information I wanted to log to transform those in meaningful analytical data 130 4 S COOL s Iterative Development For each session each play of story or game a table is saved with the name logs day _ hour _ min Sessions within the same minute interval are registered inside the same table For each story I log the following parameters par a Title of the story Type of avatar Hints enabled disabled Click on an object Number of page every time it changes SE Number of sequence every time it changes Game paused played Teacher help every time the teacher tries to help the player cer Gesture every time the system is waiting for a new gesture Gesture recognized Unwanted gesture 1 Skeleton joints data And for each action I log also the time when it occurs me Z Doing as above I can discover simple variables resu
36. of Autism and its causes highlighting children in the school environment and educational practices that try to refine their specific weaknesses and enhance their overall strengths After I introduce the power of story telling for educational purposes focusing on its benefits for group of children s activities Then a brief outline of touchless technologies and technologies for ASD kids addressing the Kinect as one of the most powerful device for them My previous experiences with technology related to impaired children are also explained in this chapter along with the statement of a new paradigm of Human to Human Computer Interaction HHCI The third chapter gives a general overview of the project listing stakeholders dividing them in target end users and secondary users the possible domains of the project and its settings a general work plan and feedbacks from some Italian clinics and hospitals The next chapter is the core of the entire project It deeply describes the entire software life cycle based on a fully child centered iterative design Each set of requirements yields its own solution discussed inside and the development of a set of inter dependent modules Chapter 5 assembles the overall feedbacks of this 6month long project and tries to analyze some first statistics gathered from a combination of teachers interviews logs and videos The sixth chapter briefly explains the general instructions to setup the right env
37. otherwise specified PDD NOS and Asperger syndrome ASD can be associated with intellectual disability difficulties in motor coordination and attention and physical health issues such as sleep and gastrointestinal disturbances Some persons with ASD excel in visual skills music math and art 2 State of the Art Autism appears to have its roots in very early brain development However the most obvious signs of autism and symptoms of autism tend to emerge between 2 and 3 years of age There is no medical detection or cure for autism Autism statistics from the U S Centers for Disease Control and Prevention CDC 2013 identify around 1 in 88 American children as on the autism spectrum a ten fold increase in prevalence in 40 years the fastest growing serious developmental disability in the U S Careful research shows that this increase is only partly explained by improved diagnosis and awareness Studies also show that autism is four to five times more common among boys than girls Autism costs a family 60 000 a year on average and lifelong care expenses can be reduced by 2 3 with early diagnosis and intervention Autism is treatable Children do not outgrow autism but studies show that early diagnosis and intervention lead to significantly improved outcomes Autism Occurrence Statistics 1 in 88 1in 110 1 in 125 1 in 150 the burden of proof is upon anybody who feels that there is NOT a real increase here in the num
38. players which object to get They all enjoyed together Alysha s kids couldn t be restricted in the background and interrupted the main player s detection gt Were the children bored of using the system over time or did they look forward to it v The kids weren t bored at all Jill said Moving their bodies in space like that is almost new to them so to them it s a wonderful thing having them bend down and do whatever blup noise that its doing they love that She added that wider movements like going higher to get their feet up the ground and adding more movement area is even better but they have not gotten bored with it She pointed out later that it is good that the game stays at the same level so there can be improvements every day Ashley said Mine have turned it to a game using the counter just see who gets the most She improvised with comparing scores And encouragement in other activities like wheel barrow task by Kevin gt Did they talk about the system Compliment or Complaint While playing While they were NOT playing v Yes Asked the teachers when are we going to play the Kinect They also said they had to cancel a couple of activities since the kids wanted to 149 5 Final Analysis play with Kinect or do similar social activities gt Negatives v Kids fought for their turns which the teachers said was usual for all their activities and there was no abnormal effe
39. professori di l e di qui s proprio loro e non faccio i nomi perch loro lo sanno loro che mi hanno fatto passare momenti buissimi con migliaia di pagine da studiare e momenti indimenticabili come quelli che stanno avvenendo qui ad Atlanta Grazie Sono le educatrici e le terapiste della scuola Lionheart che grazie alla loro continua disponibilit e gentilezza ci hanno dato preziosi consigli nonch regalato dei fantastici momenti con i loro bimbi E poi vorrei dedicare poche righe anche a loro che ora mi proteggono dall alto il nonno la nonna e Alex vi voglio bene v Se anche tu stai leggendo questi ringraziamenti perch in qualche modo le nostre due strade si sono incrociate quindi grazie nel bene o nel male grazie anche a te E infine come ultima opzione potresti essere un ladro che ha rubato questa tesi dalla mia villa alle Seychelles o dal mio yatch alle Maldive in questo caso grazie lo stesso per avermi ricordato quanto sia fortunato E stata dura ma ce l abbiamo fatta diceva una famosa pubblicit ce l abbiamo fatta a tagliare la linea del traguardo e non importa poi se con il massimo dei voti o no quello si vedr ce l abbiamo fatta nel strappare un sorriso e poi due e poi tre perch d altronde la laurea e solo un semplice traguardo di fronte a tutto ci Storie BI C era una volta un bambino nato in mondo strano fatto di luci abbaglianti suoni assordanti e odori nauseanti
40. scope and requirements For every meeting 1 week iteration we needed to produce a good piece of software in which our first stakeholders professors would be able to find risks and suggest us the best ways to handle them After a careful planning and design of the whole system we had been able to break it down and build it incrementally From the beginning we decided to divide the whole project into various builds to better accomplish the constant requirements input we were going to have Therefore we found that the best model to be based on was the incremental model s38 Multiple development cycles take place here making the life cycle a multi waterfall cycle Cycles are divided up into smaller more easily managed modules A working version of software is produced during the first module so we have working software early on during the software life cycle Each subsequent release of the module adds function to the previous release The process continues till the complete system is achieved Build 1 Design amp Development Design amp Development Design amp Development Incremental Life Cycle Model gt Implementation Impl tati mplementation To be clear we decided to call each build version with a Greek alphabet letter ALPHA BETA GAMMA Each module passes through the requirements design implementation and testing phases in iterations 65 4 S COOL s Iterative Development Require
41. still be recognized as a wave as both parts of the gesture were completed in the order they were defined yet I clearly did not perform a wave To solve this problem we came up with three results that a gesture part can return when it checks to see if it has been completed or not This diagram shows these three results and the impact of returning each of them Succeed 97 4 S COOL s Iterative Development A result of Pausing allows the system to identify a movement that does not fulfill the gesture but could be a result of the user moving slowly In short the three results mean the following e Fail The gesture failed The user moved in a way that was inconsistent with the gesture and as such the gesture will start again at the beginning e Pausing The user did not fail the gesture but they did not perform the next part either The system will check again for this part after a short pause A result of pausing can only be returned a maximum of 100 times before the gesture will fail and recognition will start again at the beginning e Succeed the user performed this part of the gesture After a short pause the system will start looking for the next part of the gesture The overall gesture service is made up of three main parts each of which is detailed below Gesture Controller Gesture Gesture bes ica Gea The gesture controller is a way of controlling all of the gestures that a user can perform This
42. that provide support for caregivers educators clinicians and other professionals to further their own learning and education or improve skills Publication Venue is structured in the following way Autism Specific Journals or publication venues specifically relating to understanding autism Examples Autism IMFAR JADD Focus on Autism There is even a new conference series specific to autism and technology International Conference on Innovative Technologies for Autism Spectrum Disorders first held in 2012 Social Behavioral Science Journals or publication venues from areas in Psychology Human Development or Sociology Examples Journal of Consulting and Clinical Psychology Child Development Behavior Research Methods Computing Journals conference publications and other publication venues relating to the fields of computing computer science or human 35 2 State of the Art computer interaction Often included in the ACM or IEEE digital libraries Examples CHI UbiComp CSCW To CHI PUC ASSETS Education Journal articles or publications focusing on education or special education Often included in the ERIC digital library Examples American Journal on Intellectual and Developmental Disabilities Mental Retardation Journal of Mental Health Research in Intellectual Disabilities Medical Journal articles or publications from the medical field including health informatics Often included in the PubMed digital libra
43. the same story where technology is the medium One of our goals is to have children interacting together where technology is just a support and this clearly means we should focus on socialization between children thanks to the technology and not through the technology The brand new notion of HHCI means now more than before that with children in particular with autistic ones the path to follow is having them interacting and being social and if this can be done with the support of the technology is just a point in our favor 121 4 S COOL s Iterative Development mE2 SPEECH RECOGNITION We quickly discarded an idea of speech recognition because classes are so loud to make this working well However speech recognition is the best way to improve verbal skills one of our main goals so new solutions and classroom setting must be found to be able to use this powerful technique mE3 NFC Well understood and accepted from teachers it enhances social interaction but it needs too much time to be deeply explored and developed Since during the story children use to shake and move clothes simulating what s happening in the story teachers were suggesting to put tags inside clothes and have a basket to put them inside For example if in the story it is required to move the shoes the child has to get the shoes and bring them into the bin The tag reader will be positioned inside the bin and the shoes will have a tag Even if this
44. thinking problem solving and decision making Digital citizenship PD ps Ga I Technology operations and concepts 2 State of the Art 2 3 Touchless Technologies Touchless gestures are not just a cool idea they are part of our interactive future The popularity of Nintendo s Wii has demonstrated the need for enhanced motion recognition and digital interaction with display devices And now Microsoft with Kinect have signaled a substantial response to the Wii enabling much more sophisticated interactive capabilities Beside this some other company began developing their own system to prove that motion controlled interfaces are not anymore a concept but reality Below is presented a quick overview of the most relevant today s touchless technologies 2 3 1 Microsoft Kinect s9 XBOX 360 Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs Based around a webcam style add on peripheral for the Xbox 360 console it enables users to control and interact with the Xbox 360 without the need to touch a game controller through a natural user interface using gestures and spoken commands The project is aimed at broadening the Xbox 360 s audience beyond its typical gamer base Kinect competes with the Wii Remote Plus and PlayStation Move with PlayStation Eye motion controllers for the Wii and PlayStation 3 home consoles respectively A version for Windows was released on F
45. web application models adaptive hypermedia human computer interaction while her applied research has mainly addressed the domain of e learning and e culture In the last years part of her research in focusing on story telling learning technologies and autism related activities in collaboration with Matteo Valoriani a PHD student of Politecnico di Milano Q Executive Director of the Health Systems Institute at Georgia Tech In 2008 he founded the Atlanta Autism Consortium a group of researchers interested in autism in Atlanta Georgia He is one of the authors of Human Computer Interaction Prentice Hall a popular human computer interaction textbook Abowd is married to Dr Meghan Burke and they have three children Aidan Blaise and Mary Catherine Two of Abowd s children have been diagnosed with autism which has been a primary inspiration for much of his technology and autism research agenda p2 Gregory Abowd is the distinguished professor in the School of Interactive Computing in the College of Computing He is a member of the GVU Center and directs the Ubiquitous Computing and Autism and Technology research groups Abowd was the founding Director of the Aware Home Research Initiative and is p3 Agata Rozga is a Developmental Psychologist with a research focus on conditions on the autism spectrum particularly with respect to early screening and diagnosis She is interested in the development of social communication skills in
46. 175 Or Fh COLOR O TNC E E E E T EE T 177 Sk Bibliography mennina lella cn a a a o ty Sanat re ci 177 S2 ON iE Ste Di Ah ESA A EA EA TAE EA E AEEA ESTA 180 1 Introduzione Guardatemi male perch sono diverso Io rider di voi perch siete tutti uguali Jonathan Davis 1 Introduzione Il disturbo dello spettro autistico DSA diventato la disabilit pi crescente negli Stati Uniti con tassi d incidenza pari a 1 su 88 nascite CDC 2013 67 bambini sono diagnosticati al giorno equivalentemente a un caso ogni 20 minuti un aumento di 10 volte rispetto a 40 anni fa Questi numeri indicano un bisogno urgente di interventi continuativi e focalizzati Ogni bambino autistico si comporta in modo diverso eppure il disturbo caratterizzato da una triade di sintomi mancanza d interazione e legami sociali deficit nell acquisizione e nell espressione del linguaggio e pattern ripetitivi di comportamento spesso accompagnati da menomazioni motorie Mentre le ultime tecniche di apprendimento hanno trasformato la comunicazione connettivit e sensitivit per i neuro tipici non DSA la comunit autistica stata largamente abbandonata Il contesto generale di questa ricerca lo sviluppo di tecnologie interattive che possano essere integrate sia con attivit scolastiche sia autonomamente utilizzate da insegnanti terapisti per promuovere attraverso il divertimento interazione sociale capacit comunicative e
47. 5 Initial feedbacks My initial knowledge of Autism didn t allow me to conduct a good research from the early phases hence I decided to study and raise awareness in me addressing the first design challenges This two month May June initial research took sources from books papers psychologists and schools Meanwhile I scouted different hospitals and clinics of Milan finding main contributions in e Lucio Moderato chief director of Istituto Sacra Famiglia of Cesano Boscone e Antonio Bianchi member of Unit Operativa di Neuropsichiatria dell Infanzia e dell Adolescenza of Ospedale Policlinico e Cristina Bellosio coordinator of Reparto Neuropsichiatria of Azienda Ospedaliera San Paolo e Gloria speech therapist of Istituto Besta I had a meeting with them explaining the preliminary ideas of the project and receiving from them useful feedback which I paste in the following sections for a more completeness Feedbacks are in Italian because translating them could have result in loose pureness and relevant information However these feedbacks will be summarized in English in the requirements section of the next chapter 56 3 Preliminary Requirements Istituto Sacra Famiglia Milano 11 Lucio moderato Il bambino ha problemi in competenze relazionali sociali abilit sociali esempio orientamento per strada gt spostamento Google Maps abilit lavorative Consigli di guardare Finger Tools applicazione
48. Avere un pannello di controllo di analisi biblioteca personaggi aspetti emotivi 62 3 Preliminary Requirements My Hometown During the first period I had the chance to spend some time with an autistic children Federico and follow him in different moments of his days Under a signed agreement between me the major of Cornaredo my hometown the councilor of education of Cornaredo the principal of the Federico s school and all of his teachers I was able to attend some lessons in his elementary school classroom Furthermore I promised to come back to Italy with some tools to play and learn at the same time Teachers gave me the chance to sit beside his desk and attend the ordinary activities he did I wrote down plenty of actions and suggestions from the support teacher I extended this program after the school end June July in a summer center where Federico used to go and enjoy I ve then been able to attend learning educational activities in the school environment and playful motor activities in the summer center environment 3 stabia fiu aaarnas Z a 63 3 Preliminary Requirements 64 4 S COOL s Iterative Development 4 S COOL s Iterative Development From the previous chapter we understood we needed a working software quickly and early during the entire software life cycle Furthermore without having a complete set of requirements we had to be more flexible and less costly to change
49. D sensing middleware libraries and applications The OpenNI website s42 provides an active community of developers the tools and support a network of potential partners and a distribution platform addressing the complete development lifecycle Application NL Devices A second opportunity is DepthJS It is an open source browser extension and plugin currently working for Chrome that allows the Microsoft Kinect to talk to any web page It provides the low level raw access to the Kinect as well as dina YI AAS high level hand gesture events to simplify development DepthJS is very modular The Kinect driver and computer vision are written on top of OpenNI and NITE This component can output the raw RGB image the raw depth map filtered for the hand as well as the high level events that the computer vision recognizes A native browser plugin think Flash wraps this Kinect code which directly interacts with a JavaScript browser plugin Fortunately in Chrome extensions can contain native code so it will be easy for anyone to install it Safari requires a bit more work with a plugin installer amp needing to go to the extension store if Apple will even permit this 88 4 S COOL s Iterative Development Event handlers in the browser extension may be placed globally in content scripts injected into each web page or pushed via the content script to local DOM elements written by 3rd parties Due to th
50. Made pairs of two and play with toys They go and get friends to dance with them on the screen The feeling stays even after they are done and they pair up to play with toys as well Kids developed their own innovative ways to play with the free form like follow the leader everybody sit stand lead actions etc They were not disturbed by other children walking in front of the Kinect They just wanted the game to resume Jill s class couldn t plan out a strategy for a two player game though Ashley s class planned talked talked out their frustration over the game with each other We saw this in the video too Amelia and Rhys Let s get the sea horse thingy Children s reaction gt How did the children react the first time they used it How hard was it for you to explain to the kid what to do v For Jill s class Jess s notes suggest that they had Trouble reading picking up student at first and hard to clear enough of room so other kids don t interrupt However Jill and Ashley did not point out anything specific The kids were curious about the Kinect gt Was there an improvement in their interaction with the system over time v Ashley and Jill suggested gradual and positive improvement over time gt Did you need to draw or tape a stage area around the Kinect v From videos Every teacher had marked out an area using blue tape Ashley s area was a lot closer to the Kinect
51. Non capiva come mai quella che poi avrebbe imparato a chiamare mamma lo costringesse a indossare abiti che pungevano tanto da farlo impazzire e perch ci tenesse a riempirlo di baci che gli lasciavano le guance cos appiccicose da non dormirci la notte Col passare degli anni questi ed altri fastidi andarono diminuendo e cos l interesse verso l ambiente circostante lo spinse a cercare di comunicare le proprie esigenze agli alieni che gli stavano attorno Pi cercava di esprimersi per pi riceveva ulteriori punizioni Un giorno per esempio decise di uscire di casa con mamma senza protestare perch con lui era cresciuta anche la curiosit di vedere cosa ci fosse l fuori Purtroppo si ritrov ingabbiato in una specie di sedia a quattro ruote che si muoveva producendo un insopportabile rumore metallico Le vibrazioni delle ruote gli creavano un fastidioso prurito lungo tutta la colonna vertebrale e le luci al neon lo rendevano incapace di vedere altro Poteva per sentire le mani che immaginava essere della madre accarezzargli i capelli ed aveva la sensazione che ad ogni gesto gli venissero strappate intere ciocche Fu difficile per questo bambino insegnare alla mamma quanto il mondo da cui sentiva di provenire fosse diverso da quello in cui si ritrovava ma col tempo le cose migliorarono Lui fu in grado di spiegarle le sue difficolt e lei trov il modo di aiutarlo ad adattarsi alla sua strana realt e da quel giorno impararon
52. OL s Iterative Development mG TESTING The testing is one of the critical parts of our projects It requires good knowledge accompanied by certificates of approval to test Luckily we have both our group is formed by different psychologist and me as Agata and Arpita had to take a course and to do an exam to obtain the IRB approval The approval would have given us the possibility to record children from the early phases A better explanation of everything will be given in the testing phase this serves us just to explain the legal and technical requirements needed before coming back to the Lionheart We found different way to test children improvements but since we did not have time to explore all of them we focused on three main modules to design and develop mG1 LOGS MANAGER Before having the IRB approval I was thinking about a way of having children data without incurring in penalties Kinect can record different sources an RGB video a depth video microphone inputs and skeleton joints Since recording RGB and depth could result in requesting an approval I need to find a way to understand children s behaviors even without recording images I found the best solution in a log manager I made up I will analyze this module deeply in the implementation and testing phase but to sum up I should be able to save a log of failures and winnings and calculate on that different data An example could be while Pm waiting a child to per
53. OL project and explore new ways of enhancing children motor skills A purpose game is the best way to elicit children interaction It requires a child to meet a goal which can be score or time trying to reach a record mF1 CATCH THE OBJECTS single player vs multiplayer The game I was referring to above is in particular a catch the object game where the child needs to move and catch some items on the scene using his hands This particular scene is strongly wanted by occupational therapists because in their session they usually try to have the kid doing a movement Obviously if the child feels to play instead of doing boring movements it is very different mF2 FREE FORM GAME One of the results from the Lionheart and Gwinnet experiences is the free form game If the story mostly enhances socialization and verbal skills and the game improve motor skills the free form game link the child to the virtual world and let him understand what he is doing as a mirror A further development as suggested from teachers can be having a closet where children can choose what to be picking clothes and whatever items they would like mF3 SHAPE GAME As explained in section 3 1 2 the shape game called Pixel Balance enhances player s motor skills by having him figure A to match a shape figure B a result in a covered area figure C A B This will be shown as a concept of what we can provide to them 124 4 S CO
54. POLITECNICO DI MILANO Facolt di Ingegneria dell Informazione Corso di Laurea Magistrale in Ingegneria Informatica MOTION BASED INTERACTIVE STORYTELLING FOR CHILDREN WITH ASD Relatore Prof ssa Franca Garzotto Tesi di Laurea di Mirko Gelsomini matricola 786123 Anno Accademico 2013 2014 Abstract Autism spectrum disorder ASD and autism are both general terms for a group of complex disorders of brain development These disorders are characterized in varying degrees by difficulties in social interaction verbal and nonverbal communication and repetitive behaviors often accompanied by sensorimotor impairments For years different techniques have been used to improve the quality of life of people who have various developmental disabilities However the focused use of technology autism continues to receive limited attention despite the fact that it tends to be a high interest area for many of these children This work tries to present a solution in this broad and varied panorama developing innovative interactive technologies for autism that can be integrated with therapeutic and schools activities and can be autonomously used by therapists and teachers to promote through engagement social interaction communication capabilities and motor skills This thesis represents a contribution to the current state of the art in the field by developing and validating an innovative tool that for the first time combines moti
55. RIVEN Instead of having just a story driven by gestures I should consider collision detection between for example hand and an object on the screen or between two children mB and mC GESTURE DETECTION AND STORY Depending on how the student performs during the tutorial the system could then proceed to different versions of the interactive story that require more or less complicated interactions This concept of starting with a simple activity and then proceeding to progressively more advanced activities according to the level and pace of the particular student was suggested by several teachers as a strategy for dealing with the wide range of developmental levels among the target audience of this system This is a link between the development of module C3 tutorial and B3 levels of gestures 81 4 S COOL s Iterative Development MIT COLLABORATION At the same time of my first month in Georgia Tech I met a PHD from the MIT Media Lab Micah Eckhardt He was developing Storyscape b29 an extensive platform that allows for story creation story sharing story remixing and data collection and analysis on multitouch devices tablet surfaces While experimental testing of StoryScape s efficacy in facilitating language acquisition and expression have not been concluded some initial test related to usability are under way Preliminary results indicate that the platform is very intuitive to use for reading and interacti
56. S COOL s Iterative Development A boy of 10 14 years of age asked if the game is childish as I started explaining he confirmed yes this is childish But his mother insisted he tried She said he likes Toy Story The soundtrack for the room was you ve got a friend in me from Toy Story 3 I asked the mother if she would like to try and she volunteered Immediately the boy joined her in the adjacent box and started playing The boy scored more and the mother acknowledged this fact herself in the end Three elderly one woman and two men aged about 60 years old also played One of them wasn t willing to at first and expressed concerns about his fitness so we encouraged him saying this is good exercise Awarding moments When the kid or the parent smiled as they saw themselves on the screen especially when by the end of the game they shouted out this is cool straight from the heart Conclusion We need a game that helps detect at least a parent or a sibling along with the child The kids might not do anything themselves but when they see family or teacher they also will interact When I was discussing this with Anne Pierre on the way back she confirmed this observation saying that kids with autism have problems with initiation 112 4 S COOL s Iterative Development 4 1 6 Design After our first experiences with Atlanta s Autism representatives we started deciding what to implement and what not dependi
57. Thinking produces six distinct phases a 9 53 3 Preliminary Requirements UNDERSTAND Understanding is the first phase of the design thinking process During this phase I immersed myself in learning I talked to experts and conducted research My goal was to develop background knowledge through these experiences I used my developing understandings as a springboard as I began to address design challenges OBSERVE Then I became a keen watcher in the observation phase of the design thinking process I watched how children behave and interact and I observed physical spaces and places I talked to people about what children were doing I asked questions and reflected on what I saw The understanding and observation phases of design thinking helped me develop a sense of empathy POINT OF VIEW In this phase of design thinking I focused on becoming aware of children needs and developing insights The phrase How might we is often used to define a point of view which is a statement of the user need insight This statement ends with a suggestion about how to make changes that would have had an impact on children experiences IDEATE Ideating is a critical component of design thinking I challenged to brainstorm a myriad of ideas and to suspend judgment No idea were too far fetched and no one s ideas were rejected Ideating were all about creativity and fun In the ideation phase quantity is encourage
58. Total average time of play mm ss per class 08 38 07 12 05 46 mALYSHA 04 19 m ASHLEY 02 53 E JILL 01 26 00 00 Free Form 159 5 Final Analysis ALYSHA gt To what extent Alysha preferred her kids being involved with stories in a considered amount of time 50 and less in free form and game activities This proves in a request of young children play well structured and familiar activities Overall Alysha s class did not play enough total of 6 minutes to state any other relevant considerations ASHLEY tree Ashley s kids loved being involved in games sian a and did not considered at all the free form 10 However when they decided to play the story they always did from the beginning to the end for an average time of slightly less than 6 minutes while in the game they played just for a half of the story time Jill really like her kids playing stories among the entire school Furthermore this table demonstrates that they also liked playing games and when they did they played for more than 3 and a half minutes Jill s class is the most active class of the entire school and even if they played free form few times 3 they enjoyed enough to last more than 8 minutes on the average After the above overall introduction of activities we will focus on stories In the story teachers can enable video hints mod C4 and or help their children using the remote controller mod D3
59. a Lionheart ad Alpharetta Atlanta un eccellenza nel panorama autistico stata eseguita una progettazione partecipativa con grande coinvolgimento di terapisti psicologhi educatori e bambini come fruitori di storie Finora questo paradigma interattivo stato applicato a 3 classi di differenti livelli di funzionamento e ha coinvolto 20 bambini di et compresa tra i 5 e i 13 anni Lo studio empirico ancora in fase di elaborazione attraverso osservazioni video focus group interviste semi strutturate e file di log Complessivamente si pu affermare che i bambini hanno reagito positivamente all introduzione del sistema che diventato in poco tempo parte integrante delle attivit della Lionheart In particolare sono stati notati progressi nel movimento e coinvolgimento tra i compagni Oltretutto ergoterapeuta ha percepito un progresso incredibile in alcuni bambini mentre gli educatori hanno confermato un miglioramento della socializzazione e della comunicazione con presenza di pianificazione strategica condivisa mai avvenuta prima Il progetto in continua espansione sviluppi e test iniziali sono promettenti L abilit di creare o modificare il materiale didattico personalizzandolo in base alle abilit e agli interessi del bambino visto come un vantaggio enorme per poterlo coinvolgere nelle attivit sociali Ulteriori sviluppi sono in progresso per testare la capacit della piattaforma di raccogliere e analizzar
60. a door rocking in a chair pulling up bedcovers and whispering to the scarecrow parts 73 4 S COOL s Iterative Development mB2 MULTIPLE USERS DETECTION Nowadays Kinect can track up to 2 people joints Probably in the early 2014 we will be able to recognize up to 6 people thanking the new Kinect sensor version 2 described above in chapter 2 3 1 A multiple user interaction should face the following problem 12 The system can track multiple users simultaneously but the story only proceeds in response to actions by one user deemed the primary interactor Some students were hesitant to participate alone which was why the non primary interactors were included This also allowed us to observe interactions between students in the context of and in response to using the system mB3 LEVELS OF GESTURES The goal is the creation of a rich library of Kinect detectable actions that vary along a continuum of precision and complexity so that the teachers can adjust the story to the various levels of ability of the children in the school and personalize the storytelling experience to address the specific needs and characteristics of each child One of the teachers feedbacks revealed this problem and the solution found 10 Include gradations layered levels of difficulty speed extent of motions specificity of target gestures certain difficult motion types 74 4 S COOL s Iterative Development mC STORIES
61. aas O Ivar 1987 Behavioral treatment and normal educational and intellectual functioning in young autistic children b13 Wieder S amp Greenspan S 2001 The DIR Developmental Individual Difference Relationship Based approach to assessment and intervention planning b14 Bondy A amp Frost L 1994 The Picture Exchange Communication System Focus on Autistic Behavior b15 Gray C amp Garand J 1993 Social stories Improving responses of students with autism with accurate social information b16 Schopler E and Reichler R 1971 Parents as Co therapists in the Treatment of Psychotic Children b17 Garzotto F Bordogna M Paper based Multimedia Interaction as Leaning Tool for Disabled Children b18 Di Blas N and Boretti B 2009 Interactive storytelling in pre school a case study b19 Di Blas N Garzotto F Paolini P et al 2009 Digital Storytelling as a Whole Class Learning Activity Lessons from a Three Years Project b20 Freidus N Hlubinks M 2002 Digital Storytelling for Reflective Practice in Communities of Learners b21 Cao X Sian E L et al 2010 Telling the whole story anticipation inspiration and reputation in a field deployment of TellTable b22 Abowd G Interactive Technologies for Autism b23 Ferguson B Anderson Hanley C Mazurek M O Parsons S Warren Z 2012 Game interventions for autism spectrum disorder b24 Bartoli L Corr
62. abilit motorie Questo lavoro presenta un ecosistema tecnologico centrato sulla nuova nozione di Human to Human Computer Interaction HHCI dove la tecnologia non pi l attore primario come avviene in HCI ma solo e semplicemente un supporto tra la collaborazione umana Questa tesi contribuisce inoltre allo stato dell arte corrente nel campo sviluppando uno strumento innovativo chiamato S COOL che per la prima volta combina interazioni basate sul movimento e un approccio di storytelling per bimbi autistici Lo storytelling pu promuovere coinvolgimento emotivo offrendo trama e personaggi avvincenti immagini profonde e condivise e altri elementi come la ripetizione di parole e frasi Aggiungendo elementi interattivi e riconoscimento dei movimenti del corpo allo storytelling tradizionale si crea un attivit complessa che involve comunicazione motricit e abilit di problem solving facendo leva sulle connessioni sociali 1 Introduzione Il sistema costruito per essere eseguito su uno schermo con l utilizzo della Kinect che combina una videocamera RGB un proiettore infrarossi e un microfono per riconoscere gesti del corpo e comandi vocali Il potenziale dell interazione basata sui movimenti per poter imparare si fonda su approcci teorici che riconoscono la relazione positiva tra attivit fisica e processi cognitivi ed supportata da un evidenza psicologica e neurobiologica La ricerca partita dalla scuol
63. able the software to recognize and adapt to pc configuration changes with little or no intervention by a user Teachers do not have to own a precise knowledge to install the software At the beginning phase I will install the system but at the latest phases the system will be easily downloaded from the internet and installed by teachers with no need of a technician 2 During my visit to some Italian schools I noticed they don t usually have an internet connection My goal is indeed to provide software that does not require an internet connection during the fruition of particular actions 3 Cross platform this is a broad field which I would summarize in some lines and discuss later in another stand alone section chrome apps To sum up school s computer nowadays have different operating systems and different hardware I do not want to discourage teachers nor having their impossibility to install the software on a particular machine 4 Tablet interaction As said on previous chapters I decided to use Kinect as a primary source of interaction I would like to develop in the last phases a system that can be played either on a multi touch surface and eventually test the differences between this powerful and different ways of interaction 5 Web content sharing In the last 10 years the nature of content on the internet has radically changed Nowadays blog and content platforms are generated no more by only experts but also by users UGC
64. adi C Garzotto F Valoriani M Exploring Motion based Touchless Games for Autistic Children s Learning 178 8 References b25 Nielsen M St rring M Moeslund T Granum E 2004 A procedure for developing intuitive and ergonomic gesture interfaces for HCI b26 Grandhi S A Joue G Mittelberg I 2011 Understanding naturalness and intuitiveness in gesture production insights for touchless gestural interfaces b27 Villaroman N Rowe D Swan B 2011 Teaching natural user interaction using OpenNI and the Microsoft Kinect Sensor Proc SIGITE 2011 227 232 ACM b28 Andrew Harbor et al 2011 Interactive Storytelling for Children with Autism b29 Eckhardt M Goodwin M S Picard R W StoryScape A Social Illustrated Primer b30 Makela Bednarik Tukiainen 2013 Evaluating User Experience of Autistic Children through Video Observation 179 8 References 8 2 Site links s1 http www cdc gov ncbddd autism data html s2 http en wikipedia org wiki Autism s3 http www autismspeaks org what autism s4 http ndar nih gov s5 http www georgiaraceforautism com s6 http www autismtoday com articles 20 Ways To Ensure the Successful Inclusion asp s7 http chalkboardproject org wp content uploads 2011 11 Educating Autistic Children pdf s8 http www totalesl com esl teaching using storytelling as a teaching tool for young learners s9 http en wikipedia org wiki Kinect
65. all and logical steps and progress at a rate necessary for conditioned reinforcement The data collected can also be useful for assessing progress in learning In general due to the individualistic nature of the autism experience computer based interventions can be tailored to an individual s needs or even special interests which can potentially enhance learning and maintain interest over time Because of this perceived benefits of using computers they have become an integral part of a number of interventions and educational programs They have also become a good way of supplementing face to face therapies that are time cost and or other resource prohibitive Nowadays different software for ASD have been developed Obviously it is quite impossible to quote all of them thus I categorized them and briefly explained in the next pages 33 2 State of the Art The primary platforms form factors or delivery mechanisms used by technology or application are the following For Personal Computers and the Web Includes applications that use a traditional keyboard mouse and monitor and Internet based applications that are primarily designed for access via a computer based web browser This can also include laptop based technologies but the primary differentiator is those that are intended to be stationary and not mobile Video amp Multimedia Includes the capture storage and or access of a combination of text audio still images
66. an affect or be affected with the system They are mainly divided in primary target end users who interact with the technology and secondary stakeholders who may benefit from the technology but do not actually interact with or use it Target end users Children with autism autistic children with both high and low functioning diagnosis Clinicians Therapists paid professionals who works with individuals with autism May include medical professionals doctors occupational therapists physical therapists speech therapists applied behavior therapist or other allied health professionals Educators Includes those who teach or are otherwise involved in the education of students with autism in schools public or private including teachers administrators school staff etc Family Caregivers Include anyone who is not a professional who cares for or supports an individual with autism May include parents siblings other family friends volunteers Peers Can be adults or children who are peers to individuals with autism Includes both neurotypical individuals as well as those with autism or other cognitive disorders As you can read I decided to focus also in children s families and peers because one of the future goals could be extending the development and inserting these two categories inside the apparatus Secondary stakeholders 48 Universities researchers Anyone intending to collect data or con
67. an evaluation of the current system inside the school setting We installed three systems in three classrooms taught by 1 Ashley 10 13 years old kids high functioning level 2 Alysha 6 9 years old kids low functioning level 3 Jill 10 13 years old medium functioning level Fraser Lionheart s IT in charge unblocked port 8181 for the computers in these classes This resolved the Norton firewall issue we faced last time The new mounts we bought were still not supported on the promethean boards Fraser saved the day by fixing the Kinect mounts on the boards using zip ties He also helped us in explaining the system to the teachers and stayed with us till the end Ashley s class was out So we installed the system and left the instructions and the remote controller on her table We had done a demo with her on our last visit on 17th October so we counted on the fact that she would know how to use it or the other teachers could explain it to her Alysha s class was playing with blocks at the back of the classroom while we were installing the Kinect e As soon as I took out the Kinect one of the kids yelled Is this the Xbox and got all excited about it The teachers had to call him back every time e The promethean board was higher here and the angle level had to be set all the way down Once we got the system running the first thing we put was the live image of the child on white background e Memorable moment As
68. ances of the browser running gt Was the remote controller enough Or do you prefer additional controls v Jill had no complaints with the remote She said she actually enjoyed taking control with the remote based on her judgment how well the child did the action v Alysha had a broken dongle y Ashley couldn t get it to work perfectly gt Did you use the video hints v Jill s class already knew what to do so they didn t use the video hints STORY Gestures v gestures failed v had to stomp really high v shirt shake wasn t working v no nod y Ashley and Alysha faced issues with remote Her class frustratingly tried to get them to work v Jill mastered the remote Y Did not hook them quite as hard as the game did 152 5 Final Analysis Future of the Story gt Disjointed Little old lady avatar Fix it Want it Remove it v Remove in Jill and Ashley s class Not sure for Alysha s class gt What developments are needed in the story Additional features v Alysha said My group doesn t like skeleton images only full body for little old lady v Ashley and Jill wanted Live image and skeleton for the story v Live image will pick up costumes that the kids put on for the story gt Other ways of interaction v Two or more children playing the story just like the game v Remove the little old lady avatar v Live narration of the story in the teacher s voice pace of the rhythm by the teacher v St
69. animation video or interactivity content forms Also includes interactive videos DVDs or other multimedia Mobile Devices includes applications delivered on mobile phones PDAs tablets or other mobile devices intended for personal use Can be used in multiple environments or anywhere the user goes Shared Active Surface includes applications that are intended for multiple users in a co located mostly synchronous interaction such as large displays tabletop computers electronic whiteboards Virtual amp Augmented Reality includes the use of virtual reality augmented reality virtual words and virtual avatars Sensor based amp Wearable Includes the use of sensors e g accelerometers heart rate microphones etc both in the environment and on the body or computer vision to collect data or provide input Robotics includes physical instantiations of digital interactions Includes both humanoid or anthropomorphic robots and general digital devices that carry out physical tasks Includes both autonomous robots and those operated remotely by humans Natural input Includes the use of input devices beyond traditional mice and keyboards such as pens gestures speech eye tracking multi touch interaction etc Also required interaction with a system rather than just providing passive input a complete understanding professor Gregory Abowd in his book Interactive Technologies for Autism b22 has coded 20 papers and defi
70. as been a mobile remote controller with which teachers are able to easily control the entire story from triggering unrecognized actions to editing scenes live Since we had just the idea further details will be given in the next two sections design and implementation 78 4 S COOL s Iterative Development mE OTHER INTERACTIONS Other way of interaction below should accomplish the following feedback 6 Combine different modes of interaction and communication in both the display of the story and the users input In addition some children respond better to verbal instructions while some need a visual representation of instructions mE1 TABLET After the creation of an entire system using touchless gestures the idea could be to deploy the same system with touch interactions This development could elicit research on the field of the differences between touch and touchless interactions with autistic children The design phase should be able to lay a common background on which will be easy to accommodate whatever need mE2 SPEECH RECOGNITION Speech recognition s43 is the translation of spoken words into text To accomplish the goal of giving children the possibility of interacting in different ways we started thinking about speech recognition and subsequent natural language processing to understand spoken words and convert them into reactions of the system Furthermore our system should be speaker independent bec
71. at there was a tiny skeleton figure in the bottom right corner which really caught kids attention Children getting a feedback were moving their body and they were so enthusiastic about this But we had a question having a tiny figure of me in a portion of a screen unrelated to the entire story could be distracting that s why we thought the best solution was transforming the tiny figure in something bigger and that could easily represent the character of the story as well as the kid playing the story Maybe a gesture detecting driving game is too much and maybe touchless interaction or object interaction or vocal command We have ways of putting small chip inside objects and recognize them to trigger some actions We can do many things but probably it is better to understand what can be the initial need We also have thought about a coarse precise mode where the teacher can set up the granularity and levels of difficulty of gestures Teachers want kids moving If there is a sort of accuracy bar slider to refine levels is relevant And then about the remote other than relying on just the Kinect we make the remote easy and fast to drive the story There is anything from low tech to high tech we can do Do you want a tablet based interaction or what We are giving you different ideas but it is really up to you and your needs The idea was to one child in front of the Kinect but other children interacting within the syst
72. ate players from the front as if you were taking a picture of them Make sure the room is brightly lit Some lighting conditions can make it difficult for Kinect to identify you or track your movements For best results avoid positioning either the players or the sensor in direct sunlight 174 7 Appendix 7 3 4 Install the software e Open your internet browser Chrome Firefox Explorer Safari e Type in your address bar www microsoft com en us Kinectforwindows e Download the latest drivers e Plug in your Kinect and wait for the drivers to install 7 3 5 Connect the cables Plug one end of the cable into the USB port on the USB plug of your pc and the other end into an electrical outlet 7 3 6 Use the system for the first time Once your software is installed visit http www scool it and follow the instructions di 8 C http www scoolit tector sis a Lal httpy wwwscool t CE htprwwavscootit I Po amp d http www scoolit a ar http www scoolit a D amp 175 7 Appendix 176 8 References We re not talking numeracy and literacy skills here but encouraging creativity movement engagement and exploration KinectSEN s24 8 References Inside this document bibliography references have the prefix b while site links start with s 8 1 Bibliography b1 b2 b3 b4 b5 b6 b7
73. aumentare l interazione tra bambini utilizzano giochi di finzione con le bambole e ci sono alcuni bambini che sono molto bravi nell inventare storie Per alcuni bambini non accettabile il fatto di condividere la loro storia per altri si Un lavoro che fanno con dei bambini quello di creare insieme un finale della storia Lei chiede di inserire nei giochi dei punteggi ma alcuni non sanno n leggere n confrontare i numeri allora andrebbe bene specificare chi ha vinto con la faccia Se uno immagina di partire da una storia nota es Cappuccetto Rosso il bambino sa gi come va avanti il bambino impara a fare quella storia Per i bambini con autismo lo si vede costantemente che tantissimo per loro passa dall immagine Va fatta una selezione dei bambini su cui applicare questa storia A loro non interessa l aspetto didattico Loro hanno una collaborazione con la scuola di Via Ravenna vicino Corvetto Sono ragazzini con livelli cognitivi bassi Lo scherzo per i bimbi non scontato oppure immedesimarsi e no strumento come questo potrebbe avere degli sviluppi Si potrebbe lavorare su aspetti emotivi quindi sull imprimere al personaggio emozioni Consigli continua a far utilizzare la storia a casa estensione di concetti e idee quotidianamente guarda dei giochi da tavolo per bimbi autistici alcuni usano dei tablet Ipad Consigli andare a guardare le parti di teoria della mente neuroni specchio Conclusioni
74. ause of the different kind of children will play with the story there won t be a training of the recognition of voices mE3 NFC Trying to accomplish the same goal combing different modes of interaction we focused on NFC Near field communication NFC is a set of standards for smartphones and similar devices to establish radio communication with each other by touching them together or bringing them into close proximity usually no more than a few inches Present and anticipated applications include contactless transactions data exchange and simplified setup of more complex communications such as Wi Fi Communication is also possible between an NFC device and an unpowered NFC chip called a tag In the tag can be written a code text and be understood from the system that has the same mime type format What we have in mind is to have written tags chips inside puppets and be able to recognize them when the children will tap the puppet into our board More details will be given in the design and implementation sections TO 4 S COOL s Iterative Development mE4 ARDUINO Arduino is a single board microcontroller to make using electronics in multidisciplinary projects more accessible The hardware consists of an open source hardware board designed around a microcontroller The software consists of a standard programming language compiler and a boot loader that executes on the microcontroller Our idea is to bring virt
75. ber of kids affected 1 in 500 A Dr Thomas Insel 1in1 000 Director of National Institute of Mental Health Head of Interagency Autism Coordinating Committee IACC 1 in 1 0000 1970s 1995 1999 2000 2002 2004 2006 2008 You will probably see a lot of headlines about the 1 in 50 children on the Autism spectrum Some organizations might even try to use those numbers to scare people to talk about an epidemic or a tsunami But if you look at the numbers and the report s48 yow l see that overall the numbers of people born with autism aren t necessarily increasing dramatically It s just that we re getting better and better at counting them The next step is getting better at accepting autistic people seeing their potential and ensuring the supports and resources they need to fulfill that potential 10 2 State of the Art 2 1 1 What Causes Autism Not long ago the answer to this question would have been we have no idea Research is now delivering the answers First and foremost we now know that there is no one cause of autism just as there is no one type of autism Over the last five years scientists have identified a number of rare gene changes or mutations associated with autism A small number of these are sufficient to cause autism by themselves Most cases of autism however appear to be caused by a combination of autism risk genes and environmental factors influencing early brain deve
76. cause he was wearing black colored jeans This had happened with me also in TSRB once We called Alysha to see the demo so we could explain the instructions to her Elizabeth also joined us and watched Jill and Alysha try out the system as she enjoyed her coffee Towards the end Alysha was interrupted by an emergency with one of her kids Tamara had a very strong opinion that they start video recording the kids using the system right away though Fraser expressed some technical concerns She was affirmative that she would get Coner s parents consent to the video recording as well She called her OT Jess to see Mirko s shape games Although the shapes were not visible they understood the idea and received it positively They also suggested a game for the kids with horses since they did regular farm activities They currently do it in writing and would prefer using pictures instead to describe what the kids did in the farm The next story they plan on is the ginger bread man They are pretty willing to let the teachers make their own stories These would be discussed in details in the next meeting with them on 11th November 136 Points to discuss on next visit Feedback on the system How much did they use What worked What didn t Collect log files from the three computers What developments are needed in the story Make gestures easier Ginger Bread man story gestures images sound track What kin
77. cher to tell the story or drive the action 8 In cases when the system is waiting for a specific input from the child there is no option for the teacher to manually trigger progression 9 No possibility to recognize gestures that cross the midline hands move across body to opposite side to fit with existing physical therapy goals 1 Autistic children have different abilities Some can do some gestures while some have more problems 1 Lack of gestures motor 1 The story only proceeds in response to actions by one user Some students were hesitant to participate alone 1 For some children the concept of being inside 3 the story is just too abstract for their current level Gesture must be revisited and recoded if needed The time must be nearly equal to null recognizer Weak and short time solution programmer need to code every single story putting repetition Strong and long time solution easy way for teachers to create stories and putting repetition whenever they want tablet speech recognition NFC near field communication Arduino Weak and short time solution programmer need to code every single story inserting music and texts Strong and long time solution easy way for teachers to create stories and putting music and texts wherever they want Weak and short time solution remote mouse Strong and long time solution mobile remote controller Gesture recognizer must be revisited and
78. ct of course the demand for Kinect was higher they implied Same natural tendency gt Does anyone have Kinect at home v Only one of Jill s students Time gt How did you fit it in your schedule during the day or week Which activity did you replace or modify How did you decide when the kids do the story the game and the freeform Y Ashley used to do it in the group time in the beginning Then she tried different times of the day She had tried the story but mostly kids did the game Her class doesn t do stories v Jill did the story in their story time in the reading time She preferred making the kids dance to music in free form And single player games Effort gt Were you willing excited to use the system Or was it an obligation v They oved using it and looked forward to the new improvements in the kids But the logs show that the teachers mostly tried it the initial days after we left 5 ae 7 and mostly on the i verified from log data gt How much effort additional help in shifting furniture was needed in clearing the stage area Was the effort worth using the system Y As quoted by Ashley this is the fastest the kids have helped me move the desks vo Jill they will all help do it Innovative techniques used by the teachers o Ashley let the kid try with the story running on an ipad and the promethean simultaneously then the kid did the actions that was Chris from ashley s class who
79. ctions We have a remote controller idea with which teachers can easily override the system s fails and have some sort of customization We d like to have something 104 4 S COOL s Iterative Development more collaborative and useful to teachers That s why we had in mind also an authoring tool to give teachers the possibility to create their own personalized stories including a choice of different ways of interaction Doing this stories can easily be moved to different classes taking into consideration different levels of ability yet stories Teacher feel like it is really rich They love the idea that children use technology to be fit and interactive They love they can do THEIR stories and embed pictures and words into stories Furthermore teachers are young and they can easily use or understand these new technologies they can be creative After these brief introductory questions we had a tour and picked some classes to visit Teachers chose to visit classes of Elementary School children getting rid of in a good way some teenagers who were asking to join our research At the Lionheart School there are different classes Every class has up to 8 children and at least 2 teachers Kids do the same activity all together going at the same pace It is also relevant that most of the classes are equipped with the latest technology and we noticed teachers using really well these technologies Promethean board pc as well
80. d I became silly savvy risk taker wishful thinker and dreamer of the impossible and the possible PROTOTYPE Prototyping is a rough and rapid portion of the design process A prototype was a model a way to convey an idea quickly I learnt that it is better to fail early and often as I created prototypes TEST Testing is part of an iterative process that provided me with feedback The purpose of testing was to learn what worked and what didn t and then iterate This meant going back to my prototype and modifying it based on feedback 54 3 Preliminary Requirements Below you can see my initial work plan starting from the first phase In May and ending some day before my graduation One of the main goal was to be ready and prepared as soon as I arrived in America mid of August that s why the ideating phase is half in Italy and half in USA At the beginning of September I was able to prototype code and gather results by the first week of October ACTIVITY May June July August September October November December UNDERSTAND OBSERVE DEFINE INCREMENTAL MODEL IDEATE PROTOTYPE TEST THESIS ITA SUPERVISOR USA SUPERVISOR ITA Supervisor Prof Franca Garzotto p1 USA supervisor Prof Gregory Abowd p2 Dr Agata Rozga p3 55 3 Preliminary Requirements 3
81. d of games do they want Farm Horses Wardrobe In the two player game what kind of interactions they want Hi 5 Hand shake Discussing the shape games with the OT 4 S COOL s Iterative Development 4 2 COOL Gamma This is the last iteration I had the chance to write down More other evolution will be done but they will not be written inside this document 4 2 1 Requirements After a meeting we decided to solve the most relevant pre existing modules problems and not to proceed into the development of new modules Doing this we could easily focus on 1 STORY Little Old Lady a NO AVATAR by default and skeleton shape as a second choice Little Old lady skeleton in the bin We will explain teachers this is not a completely working gesture driven story but we are just monitoring the kind of gestures they usually do with different kind of children However the story will completely works with the one button remote controller b ADD PAGES add pages where the little old lady walks a little faster with animation of going faster c video hints hidden by default d fix remote controlling activities 2 GAME a add the chance to play with infinite time 3 FREE FORM a add a white background b more avatars mr bean iron man OTHER THINGS TO DO a path problem b Norton Business Manager firewall problem with socket port c buy mounts d speed of the showing Agata noticed a speed decrease in the smoothnes
82. d to perform actions in front of the mirror The very concept of seeing themselves on the TV screen When people are in the news or TV effect as described by Jill Get a monster MOTIVE gt Filmed versus real if the o t will be filmed will it be a greater effect than he r actually demonstrating it gt Live camera To give live feedback will that work gt Kinect s off the shelf commercial games Will there be a similar or a better effect with the commercial games available with Kinect Jill suggested that one reason that would fail would be that this allows them to stay at the same level and hope for improvement The commercial games change levels too fast Gregory added they wouldn t consider it sexy to put up things on the white background Gregory s question If I took away the system will you be happy and the teachers unanimously yelled NO Memorable moment Magi peeked in through the glass window and we had her video paused on the big screen As I met her outside she exclaimed there are my favorite visitors Conclusion The feedback and improvement wouldn t have been the same had the teachers not constructively explored the possibilities with the system They used their own ingenious methods to use it to the maximum benefit of the child Also had there been Alysha present at the meeting the emphasis could have been more on the negative aspects based on her notes 155 5 Final Analy
83. duct studies about individuals with autism and publish something generalizable about obtained data Hospitals clinics Any kind of structure which want to take part of the project and try the system with autistic children and gather some data 3 Preliminary Requirements Where we see your children with our hearts and address their needs with our minds I The wo Lionheart School JON dl t The Lionheart School This entire section is completely dedicated to the Lionheart School s36 with which the entire project is based on Lionheart is a school situated in Alpharetta Atlanta with the mission of providing a developmentally appropriate education for children who need a specialized learning environment therapeutic interventions supported social interactions and strategies to accommodate their individual profiles The school serves children from 5 to 21 who have difficulty with academics sensor processing motor planning visual and auditory processing receptive and expressive language attention memory executive function or social interactions Teachers are professionals that work with families and their children who recognize the importance of early intervention and that the right academic setting can lead to lifelong success no matter what the difficulty may be Lionheart s therapeutic interventions are e Occupational therapy for sensory integration e Speech language therapy including the pragmatics a
84. e ea OD Fy ShoulderLeft Tmo moT hb aL Oe WON by ShoulderRight Wight Stes Wage tigi tra On by Spine Wx sO 2620168 myt men L55754 mati 032668 F WristLeft taming 2316126 y 0 05916679 z 0 6303872 ty WristRight Mago 0 2332067 y 0 07285482 Tati to 6275901 101 4 S COOL s Iterative Development 4 1 4 Evaluation Lionheart School and Gwinnet Race for Autism s5 were two big possibilities to show our project and have two different points of view Although I gathered different feedbacks from many hospitals clinics in my country Italy read different papers and publications about this topic attended a 2month program in a center as volunteer for children with autism and deeply understood the strengths and concerns of the previous project I felt that the meeting of the 3 October was the key of my research here in Atlanta After the first month we were ready to bring to the school and ask some preliminary questions to the teachers 1 When you do stories in class what kind of interaction do the children do And what do you want your child to say what do you what him to do 2 Teacher is in a class how does she know who is doing better Is there a sort of turn taking activity 3 Can you have some of the things that happen on the screen be dependent on two kids doing something together For example me and I both have to come up with a shared
85. e dati in maniera automatica e per creare un architettura modulare che permetter di avere un sistema riusabile e adattabile alle esigenze di sviluppatori e in particolare dei bambini 1 Introduction You laugh at me because I m different I laugh at you because you re all the same Jonathan Davis 1 Introduction Autism spectrum disorder ASD has become the fastest growing disability in the United States with current prevalence rates estimated at as many as 1 in 88 children CDC 2013 67 children are diagnosed per day meaning that a new case is diagnosed about every 20 minutes a tenfold increase compared to 40 years ago These numbers indicate an urgent need for focused and ongoing intervention Each child with ASD behaves differently yet the disorder is characterized by a triad of symptoms related to lack of social interaction deficits in the acquisition and expression of language and repetitive patterns of behavior often accompanied by sensorimotor impairments This work presents a technological ecosystem centered on the brand new notion of Human To Human Computer Interaction HHCI where technology is no more the primary actor as suggested by HCI but it is just simply a support between the human collaboration Our initial body of research started from the Lionheart School in Alpharetta Atlanta an excellence in the ASD panorama A participatory design approach with a strong involvement of therapists
86. e effectiveness of motion based touchless applications for autistic children is limited because of the small amount of existing empirical studies and the limits of our current knowledge on autism This thesis contributes to the current state of the art in the field by developing an innovative tool called S COOL that for the first time combines motion based touchless interaction and storytelling approach for autistic children Storytelling can promote emotional engagement with compelling plot and characters vivid imagery and other elements such as repetition of words and phrases Adding interactive elements and body motion recognition to storytelling creates a more complex activity that involves communication motor and problem solving skills while leveraging social connection The potential of motion based interaction for learning is grounded on theoretical approaches that recognize the relationship between physical activity and cognitive processes and are supported by a growing body of evidence from psychology and neurobiology Preliminary evaluations of the prototype at a local school have pinpointed the potential of motion based interactive storytelling for autistic children At the same time they have identified challenging research directions to expand the original project that will be investigated in this thesis 1 Introduction 1 3 Structure The structure of this work is as follows I first begin with a discussion
87. e missed Alysha Jess Each class had its own installed system in its computer and we were able to differentiate statistics among the following three classes 1 Alysha 6 9 years old kids low functioning level 2 Ashley 10 13 years old kids high functioning level 3 Jill 10 13 years old medium functioning level Jill and Ashley had a very positive and encouraging experience but Alysha and Jess have pointed out major technical issues in their notes Unfortunately they were not present in this meeting 145 5 Final Analysis There was a major difference in capabilities of different kids in the groups of students taught by each teacher Alysha s class Her kids are younger and at a lower functioning level They mostly tried the story and had major technical issues with it The game was avoided in this class since it had music and most kids in that class have auditory dysfunction This we had experienced in our last visit too when her student Alistor desperately wanted us to turn the sound off Elizabeth supported the same stating that in that class each kid tries to out shout the other Jill s class was at the next level with older kids She had a group of 12 kids including some from other classes try it in her class They did the story regularly but were benefited more by the game and the free form dancing However her kids were not capable of playing the two player game Ashley s c
88. e software and use the system Further development foresees that all of these instructions will be provided inside the S COOL website in a more organized way 7 3 1 Position the Kinect sensor s45 s46 Kinect needs to see your entire body e Place the sensor near the edge on a flat stable surface e Position the sensor between 2 feet 0 6m and 6 feet 1 8m from the floor Ideally the sensor should be within 6 inches 15 cm above or below your TV e Avoid positioning the sensor in direct sunlight or within 1 foot 3m of audio speakers e Do not manually tilt the sensor it adjusts automatically e Be careful not to drop the sensor 2 ft 6 ft 0 6m 1 8m Sensor mounting adapter required for above TV mounting sold separately Note In smaller rooms try to position the sensor as close to 6 feet 1 8m as possible 173 7 Appendix 7 3 2 Set up your play space Kinect needs to see your entire body e Clear the area between the sensor and the players e One player Stand back 6 feet 1 8 m e Two players Stand back 8 feet 2 4 m Make sure that the play space is at least 6 feet 1 8 m wide and not wider or longer than 12 feet 3 6 m Make sure the room has bright even lighting 6 ft 1 8m 7 3 3 Room lighting s47 Make sure your room has enough light so that your face is clearly visible and evenly lit Try to minimize side or back lighting especially from a window Illumin
89. e time constraints we didn t have the time to precisely focus on each programming language and see its strengths and weaknesses but the best solution we found has been to delegate most of the relevant software features such as view story fruition interaction to a cross platform easy to develop powerful fast programming language such as JavaScript Doing this we just needed to code the communication between JS and the SDK in one of preferred native programming languages Since this entire project cycle has fast iterations the most important goal is to develop something that work in a few time If the native programming language chosen fails in some ways we could easily replace just a piece of native software without rebuilding the entire solution C DirectX C Win32 XNA Windows NET WPF i Forms Kinect Kinect Kinect HTML5 Javascript Enabled App J Enabled App f Enabled App a Kinect Enabled App Kinect Enabled App Il Windows WPF Kinect Controls HTML5 JS Controls i XNA Kinect Forms A Support Kinect Microsoft Kinect Microsoft Samples Direct I pha bi l A Controls Toolkit Controls dll Kinect Webserver dll Support j Controls i Kinect jl Kinect InteractionStream Il Microsoft Kinect Toolkit Interaction dll NULINTERACTION STREAM KinectInteraction170_ dll 32 or 64 This is done with the promise that in the last iterations at the end of the project if we notice a bett
90. ebruary 1 2012 Kinect was launched in North America on November 4 2010 in Europe on November 10 2010 in Australia New Zealand and Singapore on November 18 2010 and in Japan on November 20 2010 The Kinect claimed the Guinness World Record of being the fastest selling consumer electronics device after 21 2 State of the Art selling a total of 8 million units in its first 60 days 24 million units of the Kinect sensor had been shipped as of January 2012 Microsoft released Kinect software development kit for Windows 7 on June 16 2011 This SDK was meant to allow developers to write Kinecting apps in C CLI C or Visual Basic NET Kinect builds on software technology developed internally by Rare a subsidiary of Microsoft Game Studios owned by Microsoft and on range camera technology by Israeli developer PrimeSense which developed a system that can interpret specific gestures making completely hands free control of electronic devices possible by using an infrared projector and camera and a special microchip to track the movement of objects and individuals in three dimensions This 3D scanner system called Light Coding employs a variant of image based 3D reconstruction The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot and is designed to be positioned lengthwise above or below the video display The device features an RGB camera depth sensor and multi array microphone running
91. ed an idea of speech recognition because classes are so loud to make this working well A manual low technology fast wearable necklace button is the best solution After asking how stories were played with the previous project teachers told us that they pulled children out to let them trying stories They had never had an up and running system for the entire class We come up with a solution coming every 2 weeks and giving them an installed system in the class Then some kids of the Lionheart came into the meeting room and we tried the avatar with some positive results one kid recognized the Kinect some teachers suggested we should build something around this some teachers suggested having the background behind the character someone wanted to be someone else one kid who usually doesn t move was moving Teachers can see something Having a closet and choose who to be is amazing We explained also what we were going to create for Saturday and they liked the idea of purpose games We also understood that technology is nothing compared to the teachers job Technology is going to support and not replace teachers 108 4 S COOL s Iterative Development Gwinnet Race for Autism The Setup Kinect did not detect in open sunlight so we shifted inside There was fewer crowds there but we had around 30 people try our system We had two big screens and two Kinect running on two laptops one played the avatar a
92. ed those teachers who were used as informants For this reason we simply played videos during our focus group to raise new comments which for completeness are listed below with their respective skills Vv Amelia usually likes to be by herself most of the day Erin said c mon get in here she responded teacher driven prompt usually but here she was playing gt communication and social interaction v Ashley 2 kids playing deciding together to get the sea horse now the fish and so on gt shared planned strategy v Kevin jumping or stomping Teachers say it s a big thing gt motor skills v Alysha s class Alister left the scene and came back each time to watch the lady walk across gt cognitive skills v Jill s class magi and two other kids dancing on a white background Jill said the boy went and got two of his friends to dance with him gt social If permitted in the future some of these videos will be shared to the community through the S COOL website www scool it 166 6 Conclusion and future works Real progress happens only when advantages of a new technology become available to everybody Henry Ford 6 Conclusion and future works The project is a continuing work initial development and usability testing are promising The ability to easily create or modify learning material around a child s special interest is seen as a major advantage to engaging individuals in social cent
93. eda la possibilit di essere anche in parte scritta dal bambino stesso soprattutto con la presenza di almeno un altro interlocutore Al momento l impostazione pensata da Mirko sembra pi orientata al paradigma del gioco con prove a pi livelli avanzo al livello successivo o nella scena successiva in questo caso se ho compiuto le azioni previste dall editor in fase di scrittura Mirko per venuto all incontro con un sincero desiderio di confronto e di ascolto Raccoglie tutte le considerazioni e le proposte che emergono Viene avanzata l osservazione rispetto all emozione se cio il sistema sar o meno in grado di rilevare espressioni facciali anche queste da confrontare con quelle definite dalla scrittura della storia Mirko risponde che c un limite tecnologico per questa possibilit ma soprattutto sembra che non sia questo il focus del suo progetto questo aggiungo io Sembrano invece essere possibili la gestione degli aspetti di relazione evidenziati come interessanti da Anna considerando almeno due giocatori attori Questo sembra aprire molte possibilit di utilizzo Si accenna al videomodeling sul fatto anche di segmentare in modo piuttosto parcellizzato una storia e sulla possibilit che ci sia un matching dinamico Mirko dice che questa possibilit c Erminia propone un utilizzo anche per una fase che sembra precedente di riconoscimento del corpo delle sue parti di s come figura completa La possibilit offe
94. eet View APIs to visit the world using the body FORWARD SWIPE RIGHT Kintroller supported different gestures and for each gesture something happened For example if the player is walking the app will show him going straight if he swipe right the app rotate the view to the right The rotation of the body controlled the angle of view while the binocular and zoom gestures controlled the field of view Technically speaking the software used C for reading the data from the Kinect and output live to the JavaScript app using JSON as a interchangeable format Inside JSON were written the gestures body angle and hand gestures At that time I did not expect I would have to use the same configuration of this project 44 2 State of the Art Tell me and I forget Teach me and I remember Involve me and I learn Benjamin Franklin 2 6 A new paradigm HHCI The recent introduction of computationally enhanced devices that support simultaneous multi user input has important implications for co located face to face activity Educational applications particularly stand to benefit from this technology which can combine the benefits of small group work with the enhancements offered by digital media Peer to peer interaction provides important opportunities to practice conversational skills Educational activities may benefit significantly from interactive technology both touch and touchless because it combines the face to
95. eletons On November a new Kinect should come out and as rumors said can track up to 6 skeletons But before November is there something you really need Teachers want children interacting together moving and share attention around something They d like to have someone being the old lady and someone being the shoes pants One main character and other beings objects Agata highlighted the fact that before having an authoring tool we should think about having a working story with possibly different inputs and a working and easy remote controller Another feature could be implementing NFC enabled objects The objects will have a chip inside it and they have to be close to the reader to trigger some actions on the screen 107 4 S COOL s Iterative Development Teachers would prefer something more cooperative For example I have a real pair of shoes when I move them something happens We explained that this cannot be done by NFC but using the remote controller Teachers replied that a remote controller can be used in few classes where the teacher sit and let children act the story out but in more classes teachers are involved into the play and they cannot be focused just on the remote the app has many features We understood that most of the time teachers don t really need all of the features but just one single button Even if the teacher moves around the room he can easily push that button to proceed We quickly discard
96. em s40 http en wikipedia org wiki Gesture_ recognition s41 http psychcentral com news 2013 08 20 autistic kids like games that stimulate senses movement 58678 html s42 http www openni org s43 http en wikipedia org wiki Speech_ recognition s44 https developers google com chrome developer tools docs javascript debugging s45 http support xbox com en US xbox 360 Kinect sensor placement s46 https support xbox com en US xbox 360 Kinect Kinect sensor setup s47 http support xbox com en US xbox 360 Kinect lighting s48 http www cdc gov nchs data nhsr nhsr065 pdf s49 http www amazon com The Mitten Jan Brett dp 0399231099 181 8 References 182 8 References See beyond Me my group the Lionheart School as well as some people around the world which have had the time to spend with children with ASD see a potential in them and that s why every day we are proud and excitedly willing to work with and for them 43457 parole ora che sto digitando la stanchezza di questo racconto di 199 pagine e di questi lunghi mesi qui in America comincia a farsi sentire 43483 parole ora che ricomincio questo paragrafetto e Word mi fa notare con una lineetta rossa zigrinata che la parola paragrafetto non esiste Ah gia vero dovrei dire di aver scritto in Latex un ingegnere che scrive in Word Ahia ahia qualcuno dira 43528 B ho lavorato duramente 12 ore al giorno e 7 giorni su 7 me la po
97. em using tablets 106 4 S COOL s Iterative Development Then we decided to show our avatars To what extent an avatar is a way of being inside the story trough an animated figure on the screen which can be Skeleton joints are points and bones are lines Shadow black shadow of the body Green Screen video recording of the player without his real background Google effects augmented green screen with virtual objects Skeleton character segmented image tracking rotation of limbs Static Image image of the character go left right up and down and no more They expressed their interests in this from the beginning and they asked if it was possible to insert this avatar mode inside a story And they also would like to have a tutorial as suggested from the previous feedbacks What can be the most flexible to accomplish most of children s needs Connor a little boy interrupted us and we asked him to play the story He tried the Little Old Lady and some gestures were not be well triggered Arpita tried to help him using the remote controller but the delay between the child s action Arpita understanding of a true negative unrecognized gesture system sending via web the fake gesture was too much 1 2 seconds The system also failed some times because he was recognizing people behind the players yet not triggering the right gestures The current Kinect has some big limitations and can recognize just 2 sk
98. ent mB KINECT DETECTION SOFTWARE mB1 GESTURE DETECTION Gesture recognition software has the goal of interpreting human gestures via mathematical algorithms Gesture recognition can be seen as a way for computers to begin to understand human body language thus building a richer bridge between machines and humans than primitive text user interfaces or even GUIs graphical user interfaces which still limit the majority of input to keyboard and mouse Gesture recognition will enable children to communicate with the machine and interact naturally without any mechanical devices Using the concept of gesture recognition it is possible to move a hand at the computer screen so that the cursor will move accordingly Our ongoing work is in the computer vision field on capturing gestures or more general human pose and movements by a Kinect sensor connected to a computer Our gesture detection software should be able to accomplish the following requests 4 The teachers expressed some concerns that the time between the user performing the gesture and the on screen animation was too long 9 Encourage gestures that cross the midline hands move across body to opposite side to fit with existing physical therapy goals 10 Include gradations layered levels of difficulty speed extent of motions specificity of target gestures certain difficult motion types 11 More triggered gestures interactions including shutting and opening
99. entre for persons with Disabilities 2 4 Technology and autism b22 b23 Adapting existing and developing new interactive technologies for individuals with autism has increased dramatically over the last decade One of the primary motivations for the increased interest is the observation that individuals with autism appear to have great affinity for and frequently use interactive technologies when that are made accessible Computer screens allow information to be abstracted or limited to only relevant information thereby supporting the filtering process Second many individuals with autism are often confused by unpredictability social nuance and rapid changes present in the non computerized physical world Computers are much more predictable than humans and do not require social interactions Additionally computational interactions can be repeated indefinitely until the individual achieves mastery Third computers can provide routines that are explicit have clear expectations and deliver consistent rewards or consequences for responses which can encourage engagement with educational and assistive technologies by allowing an individual to make choices and take control over their rate of learning Fourth content can be selected and matched to an individual s cognitive ability and made relevant to their current environment and photos can be used to help generalize to the real world Finally learning experiences can be broken down into sm
100. eople including two active players for motion analysis with a feature extraction of 20 joints per player However PrimeSense has stated that the number of people the device can see but not process as players is only limited by how many will fit in the field of view of the camera Reverse engineering has determined that the Kinect s various sensors output video at a frame rate of 9 Hz to 30 Hz depending on resolution The default RGB video stream uses 8 bit VGA resolution 640 x 480 pixels with a Bayer color filter but the hardware is capable of resolutions up to 1280x1024 at a lower frame rate and other colour formats such as UYVY The monochrome depth sensing video stream is in VGA resolution 640x480 pixels with 11 bit depth which provides 2 048 levels of sensitivity The Kinect can also stream the view from its IR camera directly as 640x480 video or 1280x1024 at a lower frame rate The Kinect sensor has a practical ranging limit of 1 2 3 5 m 3 9 11 ft distance when used with the Xbox software The area required to play Kinect is roughly 6 m2 although the sensor can maintain tracking through an extended range of approximately 0 7 6 m 2 3 20 ft The sensor has an angular field of view of 57 horizontally and 43 vertically while the motorized pivot is capable of tilting the sensor up to 27 either up or down The horizontal field of the Kinect sensor at the minimum viewing distance of 0 8 m 2 6 ft is therefore 87 cm
101. er at the first meeting a Login screen Username Password b Choose a live story STORIES Broken Bells 1 minute ago Warning 5 minutes ago Wolfgang Amadeus Pho 3 hours ago c Control the flow O Back Story Title Sound Avatar Controls Help GAME Replay Play Pause End PAGE Previous ACTION Previous Next x tot scenes d Set the avatar view Q Back Story Title Sound ET Controls Help x x Shadow Skeleton Character Fixed Image Green Screen Google Effects Skeleton x tot scenes e Help the player s Q Back Story Title Sound Avatar Controls Help Tools HELP next gesture SHOW SUGGEST LEVEL MEDIUM DIFFICULT x tot scenes f Adjust the sound Q Back Story Title Sound Avatar Controls Help SOUND GAME Play Pause SCENE Play Pause CHANGE SOUND x tot scenes 85 4 S COOL s Iterative Development 4 1 3 Implementation As said in the design phase the first implementation phase should take into account the possibility for the next phases to build on top of this early code as if we built a good stable foundation Furthermore beside specific requirements descripted above in the requirements section we must consider general requirements given by different stakeholders 1 Software Plug and Play as a combination of settings that en
102. er way to communicate with the JavaScript we will create a more solid system or framework 89 4 S COOL s Iterative Development mA STORY CREATION TOOL and mE1 TABLET The story creation tool and tablet interaction is something we have decided not to develop on the early versions but to extend the possibility to be built by us or be benefited from other sources MIT StoryScape as an example To do this we collaborated with MIT to create a solid exchangeable format in which mine and their project could gather what they needed mC1 STORY The module story is the view of the entire system It is the one that interacts with teachers and children and abstracts the communication between the native code and the view mC2 AVATAR The avatar is a way of viewing the child in front of the Kinect Even this module can be deployed using JavaScript 5 COOL AVATAR 90 4 S COOL s Iterative Development mD1 REMOTE MOUSE Mouse interaction can be easily triggered by different programming languages one of them JavaScript mD2 MOBILE REMOTE CONTROLLER On the early phases can be done by an internet connection but then following some of the above requirements we will need a Bluetooth communication avoiding the internet connection and possible delays S COOL MOBILE CONTROLLER STORY 91 4 S COOL s Iterative Development Overall communication between modules CLIENT SIDE HTML5
103. estures fig e 115 4 S COOL s Iterative Development Kinect camera input Joints SDK ERROR SOLUTION SOLUTION clap Gestures recognition clap or zoom in Framework ERROR SOLUTION Just clap 1 e no zoom in A clap Joints SDK Gestures recognition ARA Gestures recognition We are here Kinect camera e SOLUTION vee Gesture creator clap 116 4 S COOL s Iterative Development mB2 MULTIPLE USERS DETECTION This is one of the best ways of having many children interact hopefully together New ways of cooperation and collaboration should be explored and developed mB3 LEVELS OF GESTURES A better research and implementation of granularity of gestures should be done We can also think to add a sort of accuracy bar mC STORY The story is the best way to share kids attention and involve them in different levels of sight heard and movement mC1 STORY The biggest problem is that in the class they don t usually do turn taking activities because it is not a natural waiting process A combination between one player in the tracking area all the other children in the class playing around and a teacher remote could be the best mC2 AVATAR Having a tiny figure of me in a portion of the screen could be distracting as in Kinect the Dots and the solution was to create the avatar a bigger figure which represents the child inside
104. estures during the story per class mALYSHA m ASHLEY E JILL Taking into account all the above consideration we can easily state that for Alysha s and Ashley s class the first three gestures were almost perfect with a 20 of aid while the last two caused they to click the controller for more than half of the times In Jill s class as she said at the focus group she decided live what to skip and what not according to her feelings and this causes a click of the remote for an average of more than 60 of the times Overall the walking gesture seems to be the less helped probably because of its high frequency 5 times inside a story while the last two gestures nod and clap seems to require a constant help by teachers probably also because of their low frequency 2 and 1 time respectively and subsequent children difficulty to perform 162 5 Final Analysis Our experience tells us that a children is negatively influenced by the system if its response has a delay of more than 1 second This means that in Jill s class children have a positive experience and when they don t the teacher is ready to help them while Alysha and Ashley help few times and want their children to perform a better motion Story avg children response time s per class 0 00 0 50 1 00 1 50 2 00 2 50 walk wiggle shake MALYSHA MASHLEY JILL As said before the system can fail and not recognize a gesture Sometimes it is a deve
105. ete understanding of the cognitive mechanisms of these subjects 38 2 State of the Art Software s24 s25 There are many interactive programs written by individuals on the internet Most of these programs are for everyone while Somantics and Pictogram Room were particularly designed for special needs pupils Before speaking about these 2 ad hoc software this is a list of interactive programs that have been tested on ASD Children and have resulted in a positive way Reactickles s26 Z Vector s27 Visikord s28 Snowbells s29 Noise Ink s30 Fluid Wall s31 Body Dysmorphia s32 Kinect2Scratch s33 Somantics The program s34 has ten different applications that respond to body movement you can paint with your body make sparkles appear around you and explore different graphic reactions to movement They were designed for ASD pupils in mind but work for any SLD pupils who have good vision they encourage pupils to move create and explore their physical movements Put the pupils in front of it and let them explore there is no task or right or wrong answer to this it is about encouraging movement and exploration of the self and the interactions it creates There is also an iPad app that works on the same principles which can be hooked up to a larger screen if you don t have a Kinect It was designed by Dr Wendy Keay Bright at Cardiff Metropolitan University for use with students with Autistic Spectrum Disorder The pr
106. ete way after this table N DESCRIPTION OF THE PROBLEM 1 Although the students were instructed where DESCRIPTION OF THE SOLUTION A clearly marked area to stand within may to stand they frequently forgot or help with this difficulty disregarded this instruction in the heat of the moment Weak and short time solution Programmers need to draw objects considering this fact 2 The position of objects at the top of the screen made them difficult for many of the children to reach 3 The images used were noticeably different from other imagery being used in the classroom for Jack and the Beanstalk activities and the plot of the story within Kinect the Dots contained some deviations from the story used in the classroom Strong and long time solution Teachers personal page tool to move object on the scene Weak and short time solution programmer need to code every single story with the same images used in classrooms Strong and long time solution easy way for teachers to create stories 69 4 S COOL s Iterative Development 4 The delay between completed gesture and recognition might interfere with the cause and effect connection for some children or cause them to lose interest in the story 5 No repetition of specific words inside the story means no motivation 6 Some children are more physical while some are more verbal in their interaction style 7 The first version of Kinect the Dots required the tea
107. form a gesture save the time where the system starts to accept the gesture and the time where a gesture has been performed Doing this I could have the delta and understand how much time is needed to understand a given gesture On the same basis I could save the gestures which have been recognized but did not cause a reaction because the system was not expecting them or I could record how often the teacher help the student using the one button remote controller Again all of this information will be given in a broader sense during the implementation part 125 4 S COOL s Iterative Development mG2 VIDEO RECORDINGS Most of the current methods for user experience evaluation require that users are able to reflect on and communicate their own experience Such methods however are not suitable when users have limited communication skills Conducting video observations can be a useful technique for evaluating episodic user experience of children with special needs At the University of Western Finland some researchers b30 evaluated this approach and are now able to prove some results They recognized that there are several limitations in using video observation for evaluating user experience As the evaluation is based on observing users interaction with the system a working prototype of a system is required In addition while the results of the evaluation can help in describing the user experience the reasons behind the experience
108. gned to work on the Windows 8 platform and is expected to work on tablets smartphones and even cars Elliptic Labs is not out for consumers to buy as the company is focusing on marketing it to Original Equipment Manufacturers OEM 29 2 State of the Art Airwriting Airwriting s19 is a technology that allows you to write text messages or compose emails by writing in the air Airwriting comes in the form of a glove which recognizes the path your hands and fingers move in as you write The glove contains sensors that can record hand movements What happen is when the user starts airwriting the glove will detect it and send it to the computer via wireless connection The computer will capture it and decode the movements The system is capable of recognizing capital letters and has 8000 vocabulary words For now the glove is only a prototype and it s nowhere near perfect as it still has a 11 error rate However the system will self correct and adapt to the user s writing style pushing the error rate down to 3 Google has awarded the creator Christoph Amma it s Google Faculty Research Award of over 80 000 in hopes that it could help him to developed this system EyeSight EyeSight s20 is a gesture technology which allows you to navigate through g your devices by just pointing at it Much like how you use a remote to navigate your TV you don t have to touch the a screen And get thi
109. ha presentato il suo progetto di sviluppo per la sua tesi per la laurea in Ingegneria Informatica al Politecnico di Milano con la professoressa Franca Garzotto Il suo piano di lavoro prevede una fase di confronto con esperti del campo da maggio a luglio l incontro di oggi dentro questa fase e un periodo da agosto a dicembre presso un universit degli Stati uniti esattamente ad Atlanta col professor Gregory Abowd che Mirko ci ha detto essere anche padre di un figlio con autismo Il riferimento americano del lavoro di tesi sar la ricercatrice Agata Rozga Il progetto gi molto ben presentato dalla lettera inviata da Mirko per quanto riguarda la narrazione La proposta prevede un ambiente di fruizione per il bambino o ragazzo eventualmente con altre persone compagni o adulti e un ambiente di editing dove vengono definiti i contenuti del contesto davanti a cui il bambino si pone interagendo attraverso i suoi movimenti letti e codificati dal dispositivo Kinect con il supporto del software di base del dispositivo le sue librerie e quello applicativo scritto da Mirko Lavori esistenti Mirko ci ha presentato due lavori gi realizzati sempre all interno di questo paradigma di interazione Pixel balance dove dopo avere definito delle figure di riferimento l ambiente invita a riprodurre quella stessa configurazione attraverso il proprio corpo Interessante la possibilit di raggiungere l obiettivo attraverso una cooperaz
110. he story What the teachers would like to be able to change or add themselves by downloading from the internet or taking photos o Background images o Music Record voice o Captions Do you want to change these when you create the story or as the story plays What kinds of other interactions do you want Touch driven promethean board Social driven with humans Collision driven virtually get object on the screen Speech driven Ginger Bread man story gestures images sound track 143 4 S COOL s Iterative Development GAME Bigger objects on the game Use feet to get objects Ashley Jill Music stops in mars In the two player game what kind of interactions you want Hi 5 Hand shake Clap to remove objects for advanced level Ashley Avatars throw a ball back and forth on screen Ashley o Pick scenes that go with our avatars ex power ranger scene to go with iron man etc Ashley s class o Matching color of the hand to the object color Crossing gestures O VV V O O FREEFORM o 2nd character not available on Free Form option No 2nd avatar option to choose Ashley Is that the live Image Interactive objects in the background I ve noticed a lot of the O students are trying to go to different scenes that are on the screen For example walk the princess back to her castle Trying to think of how to incorporate this as it is motivating for them Do you want music for kids to dance
111. her different ways of using technology to make experience interactive Our group is not completely locked into this interaction imaging you are in a stage Kinect has some limitations even if can detect multiple bodies but one two must be the active players of the story Teachers want their kids sharing attention one can have a book one other a book but they both have a common ground They want engagement they want kids are all in together they want them being in the book Doing something with another kid and maybe see myself make something happen is the goal Teachers clearly don t know the ways to have that using technology but they know that fairy tales are the best chance to accomplish this goal with or without technology Example if there is an angry bird right in front of me I may not be able to find a picture of angry bird into my mind while if I share a fairy tale there might be the possibility to have the same pictures shared by all of the kids Can you have some of the things that happen on the screen be dependent on two kids doing something together For example me and I both have to come up with a shared action Some classes could do that because kids are not just about themselves nature of autism is being about own selves Some older in the majority could learn that they could learn a rule but for young it is more complicated harder Teachers love this because is a way of sharing attention images and a
112. hildren are surprised and amazed at some of the results In fact the trend is spreading and even more classrooms and clinics are being outfitted with Kinect gaming systems in an attempt to see a broader base of positive results 25 2 State of the Art A therapeutic device The great news is that the Xbox and Kinect add on still come in cheaper 100 than the cost of many expensive pieces of equipment that were previously used to have the same results Could it be that the results are better from this because it stimulates children visually and feeds upon their natural love of video games For once parents will be thrilled to send their kids off to school and hearing they spent a large amount of that time playing video games While it may not quite be labeled a therapeutic device researchers have a hard time denying there are some serious benefits and advantages here The next step could be to expand on the actual educational games that Microsoft offers for the Kinect gaming system By branching out into subjects such as math and other areas it will be important to see if this type of training also gets through to young Autistic students The bottom line is that teachers therapists and researchers are seeing results from using Kinect that they themselves were not able to get through months of hard work Overall we can collect all the advantages of using Kinect for Children with Autism in the following terms e Social e Phy
113. howed them the little old lady story with the skeleton The kids were all over the place so we just let it play instead of telling them to move away Although the lady s skeleton was fluctuating between kids all the gestures worked without Alysha having to click the remote even once The tech savvy boy told us to put his real image in the story which we have not implemented yet Another notable behavior is that in this class was that the kids were moving more towards the board The Kinect doesn t detect at such close distance so the teachers always had to tell them to move away if they wanted to see themselves on the screen Jill s class was having lunch in the other room We moved the desks installed the system and explained how to use it Her computer s processing power was slower compared to other systems and the software took about 10s to start after double clicking She tried out each feature and went over their pros and cons with us She suggested we keep music with the free form so the kids could dance One of her kids wandered in the room and disrupted the story so she closed the door She also checked up to what distance it was ok to let the other kids stand behind the Kinect s dedicated stage She said she would run it at least twice a day and send us a summary everyday which she did 135 4 S COOL s Iterative Development Kinect was not detecting Fraser s lower body which I figured was be
114. id were Jean and the Beanstalk and The Little Old Lady The story The Little Old Lady was particularly chosen in large part due to the many opportunities for gesture based interaction it presented The main character encounters various items that make up a scarecrow shoes pants shirt gloves hat and a pumpkin head as she walks through the woods Each item has a signature action that is repeated throughout the story such as the shoes clomping on the ground and the gloves clapping together When the audio narration reaches a point in the story where an action occurs it pauses and waits for input When the child performs the action and the application recognizes it a corresponding animation occurs The Little Old Lady 10 x 42 2 State of the Art 2 5 2 Pixel Balance As explained before Pixel Balance was my first project with the Kinect and the entire motion based touchless interaction Professor Franca Garzotto asked me to build something to have people physically rehabilitating with and Pixel Balance was my solution for it L Pixel Balance proposes to enhance the balance of players by giving them the possibility to enjoy and train at the same time Patients who seek therapy to correct balance and gait issues will have a thorough examination by a therapist During this initial examination the therapist will perform one of several activities including but not limited to the easiest one
115. ifferent sensors RGB depth microphones infrareds a good setting of the environment playground without noise lights available space and algorithm If one or more fail or are not accurate the recognition fails as consequence For autistic children create the connection between them and the game as told above is already a difficult undertaking Furthermore it is yet hard to maintain this link constant during the entire gameplay If technology in at least one of its part fails the connection blow up and everything must be restarted To what extent if a child who already knows he is playing performs the correct gesture but the system can t detect it could result in a daunting outcome A teacher remote controller should be able to 8 include the option for the teacher to manually trigger progression you don t want to discourage kids when they re performing the right actions or as close as they can get and the system doesn t recognize it mD1 REMOTE MOUSE The first solution we conveyed was to trigger the teachers remote mouse If the child performs the action but the system does not recognize it the teacher can use a remote mouse to override the gesture recognizer and cause the story to proceed as if the action had been recognized mD2 MOBILE REMOTE CONTROLLER After the remote mouse we were trying to find something more complete which could give teachers the mastership of the story The idea we found h
116. ione ad esempio per matchare la figura dell elefante Sono naturalmente definiti dei livelli a difficolt crescente e un rinforzo audio che commenta l approssimarsi o meno all obiettivo Il matching comunque di tipo statico Facilitazioni sono fornite all ambiente di editing con la possibilit di partire da foto Il software non attualmente distribuito si tratta di un prodotto che Mirko dice di volere mettere a disposizione come anche il resto in modo aperto Interazione con Street View sempre attraverso Kinect gestendo le gesture di cammino svolta guardare in alto o in basso con la corrispondente gestione su Google Maps Oltre a quanto presente gi in Google Maps Mirko sottolinea la possibilit di aumentare la rappresentazione con elementi aggiunti ritenuti interessanti per quel bambino o ragazzo segnalazioni o marcatori Molto interessante nella sua semplicit di concezione 58 3 Preliminary Requirements Il progetto di story telling Rispetto alle due possibili rappresentazioni dell avatar in prima o terza persona Mirko ci ha detto che pensa di usare la terza persona studi sulla validit della prima rispetto a situazioni di autismo sarebbero da rintracciare ma sembra poter portare a un esperienza pi distanziante Un elemento che sembra dipingere almeno due scenari molto diversi quanto la storia che il bambino invitato a vivere sia una storia molto definita con azioni specifiche da compiere o prev
117. ior One very positive aspect of antecedent procedures is that they are proactive Since these strategies all involve altering routines or environments they address challenging behavior prior to its occurrence Antecedent procedures that have been used specifically for students with autism in general education classrooms include priming prompt delivery and picture scheduling Priming Priming or prepractice has been documented as an effective classroom intervention for children with autism Priming consists of previewing information or activities that a child is likely to have difficulties with before the child actually engages in that activity For example if a child is having difficulties during circle activities where the teacher is reading the class a story each day s story could be read to the child individually before the child experiences the story in the presence of the entire class Priming is important in facilitating the inclusion of students with autism in that it links individual instruction to larger classroom group activities a common feature of general education classrooms Research has focused on using priming to improve social interactions between children with autism in regular education classrooms and priming has been shown to be effective in increasing the initiations of social interaction with typical peers Prompt delivery Prompting strategies have been successful in facilitating the inclusion of students with aut
118. ironment and try the software I conclude this work with an overall discussion of challenges and future directions for the project giving a general view of the possibility of being based on a modular architecture framework 1 Introduction 2 State of the Art 2 State of the Art This chapter starts by explaining the Autism Spectrum Disorder and its causes Then after a brief focus on the power of storytelling especially for children I will list the most relevant touchless technologies and technologies for autism The state of the art is also composed by my previous experiences and developed software in this field and by a brand new notion of Human to Human Computer Interaction HHCI When you ve met one child with autism you ve met one child with autism Stephen M Shore 2 1 Autism s1 s2 s3 s3 s5 b3 s48 Autism spectrum disorder ASD and autism are both general terms for a group of complex disorders of brain development These disorders are characterized in varying degrees by difficulties in social interaction verbal and nonverbal communication and repetitive behaviors With the May 2013 publication of the DSM 5 diagnostic manual all autism disorders were merged into one umbrella diagnosis of ASD Previously they were recognized as Aspergers a Autiati distinct subtypes including autistic n a sein disorder childhood disintegrative Riprende disorder pervasive developmental ae disorder not
119. irror their actions helped the students establish and reinforce the notion that their body was being recognized by the system and that moving their body elicited a reaction from the system The avatar seems to be a cardinal point to work and research on It could be a good connection children understand they are prompting the story as much as a good distraction children play just with the avatar without taking care of the story required gestures 75 4 S COOL s Iterative Development To what extent an avatar is a way of being inside the story trough an animated figure on the screen which can be Skeleton joints are points and bones are lines Shadow black shadow of the body Green Screen video recording of the player without his real background Skeleton character segmented image tracking rotation of limbs Google effects augmented green screen with virtual objects Static Image image of the character go left right up and down 76 4 S COOL s Iterative Development mC3 TUTORIAL One feedback was to reinforce the notion of being inside the story and the first thought was about the avatar described above Autistic children more than neurotypical ones non autistic need to link their being to their actions and again to the story The connection we build up easily are non trivial for them Teachers brought up the idea of including an introductory section in the s
120. is a nice idea to be explored we considered not to develop due to two main issues short range distance implies tag and the reader to be very near lt 10cm and time to develop this feature is greater than our time constraints NFC can also be used for other form of interaction as explained in the first Alpha version where a puppet with inside a tag can be positioned on a NFC tag reader and the system can bring the same puppet inside the story The images shows and example of NFC integrated with ARDUINO mod E 4 mE4 ARDUINO Arduino is another unexplored form of interaction that should be taken into account for further development It enhances the truthfulness of the relation between virtual and real causing a better sense of empathy thus experience 122 4 S COOL s Iterative Development mE5 PROMETHEAN BOARD Promethean board is an interactive whiteboard system which enlivens teaching and learning It is designed to focus attention and provide a platform to boost the interactivity of lessons and the essential building blocks for any digitally connected classroom Children of the Lionheart School can use really well Promethean boards The touch interaction and the triggering of the touch should be easy to implement 123 4 S COOL s Iterative Development mF PURPOSE GAMES Thanks to the Gwinnet Race for Autism where we develop in one day a purpose game we can now include it inside the S CO
121. ism Often when teaching children with autism in order to elicit an appropriate response in a targeted academic or behavioral activity one must provide prompts that supplement the general instructional routine Using various prompting strategies is important in facilitating the inclusion of students with autism as these students may not respond to traditional instructions delivered in general education classrooms 13 2 State of the Art Picture schedules Picture schedules are often used as a strategy for increasing predictability and as an alternative to verbal and written instruction Transitioning from one activity to another can be problematic for some students with autism yet is a very common occurrence in general education classrooms Picture schedules can serve as effective cues alerting students with autism to upcoming changes in activities _ Today At School 88 esl Ee Ar Ta A P A IS gym gyndass special activity NAZ Te R 14 2 State of the Art I m a visual thinker not a language based thinker My brain is like Google Images Temple Grandin 2 1 3 Educational practices b11 s7 b12 b13 b14 b15 b16 p17 As said in the previous chapters the continued increase of students identified with ASD has placed significant stressors on public schools and the educators that serve them To date the research that exists on school based interventions for ASDs has not
122. istener called action_trigger_code its reaction action_code and its sequence number sequence_ item We also shared a common notation a story has some descriptive attributes each media object has the following story_ id the unique id inside the database attributes title title of the story action code the animation to do after the description description of the story action is recognized tags relevant keywords z index the z position of the object author name name of the author page _on_ complete page to go after creator_uid unique id of the author in the completion of the animation database color color of the text num_ pages number of pages text content of the text action_ trigger code action to recognize each story has different pages each height height of the object page has width width of the object page _id unique id of the page in the url relative url of the object database object _id id of the object page number number of the page inside the x distance from the left border of the object story y distance from the top border of the object media _objects an array of media objects font size size of the text images sounds text type image sound text sequence_ item sequence number 95 4 S COOL s Iterative Development This is the entire str
123. iteration the product can restart from the requirements analysis for another iteration or can be deployed in the identified market 66 4 S COOL s Iterative Development 4 1 Organization As soon as I met Gregory and Agata my supervisors in Atlanta we scheduled a weekly meeting on Tuesday in which we could speak about on going developing and gathering some useful feedbacks for further single modules development Furthermore every Friday I had the chance to meet with the Ubicomp Group PHDs and listen about their projects a big chance to understand different faces of Autism and generate new possible projects ideas One of the first weeks in one Tuesday meeting Gregory noticed I needed someone to work under my supervision and then could help me doing some assigned tailored tasks He found Arpita Bhattacharya p4 a first year master student in Computer Science be able to work with me Since the first moments me and Arpita found a good agreement in which I worked full time avg 10 hours per day managing the entire process and Arpita worked part time avg 3 hours per day since she had some classes to attend and related exams project to do In particular I split the project in different modules assigning her small easy stand alone modules and letting me the most critical part regarding bigger modules and communication between them Each module will be deeply described in the following sections For a complete li
124. jects are geared toward performance a skill essential for success in the real world They also lend themselves naturally to the form of many common public presentations such as museum docent talks photo essays and documentary films giving students practice in real world skills Encourages creativity helping students open up new ways of thinking about and organizing material This new medium promotes the development of multiple channel intelligence and communication blending intellectual thought research emotion and public communication Works well with portfolio assessment For expert advice on how to use electronic portfolios and digital storytelling for lifelong and life wide learning visit Dr Helen Barrett s web site Promotes digital literacy Becoming proficient in digital skills is fundamental to students success in the 21st century Overall when you combine the power of project learning with the learning power of digital storytelling the educational benefits increase You get motivated energized students and the confidence of knowing that you are helping your students meet national educational standards developed by the International Society for Technology in Education ISTE All six of the 2007 National Education Technology Standards NETS for students are addressed by digital storytelling 20 1 Creativity and innovation Communication and collaboration Research and information fluency Critical
125. k about their experience v e and ve Benefits Motor Social Verbal Drawbacks or issues Specific feedback on the technology Then we cover the additional features they want in the Story Game and Freeform one by one For each we will focus on a Short term goals what is to be fixed in current system b Long term goals What additional features are required e Mandatory can t do without this o Plus it will be great to have this GENERAL FEEDBACK ABOUT THEIR EXPERIENCE gt Benefits gt How did it benefit the teaching goals of Lionheart gt How did the child benefit from the story or game Did they do something they never used to gt Motor skills What are the potential improvements in their motor skills if any gt Emotional development Was there any development in their level of excitement or emotional state gt Social skills Did the children interact look talk plan with other children while playing Did they talk to you while they were playing Children s reaction How did the children react the first time they used it How hard was it for you to explain to the kid what to do Was there an improvement in their interaction with the system over time VVVV Vv Did you need to draw or tape a stage area around the Kinect Was it a lot of trouble getting the kids to stand in the stage area Did you have to always remind the kids about it gt Did you pick one kid at a time turn taking le
126. kills and address certain challenges e Social Emotional Skills generally includes modules that focus on emotion recognition pro social behaviors nuances and figures of speech The software at its first development will include only the acquisition of pro social behaviors and some impersonations e Language Communication Includes modules that focus on learning vocabulary language acquisition skills reading spoken language for communicative purposes semantics syntax morphology or prosody The software mostly in the storytelling part will include this possibility e Restrictive Repetitive Behaviors Includes modules that focus on repetitive circumscribed behaviors interests or play May include both high level cognitive behaviors and low level behaviors such as manipulation of body or objects Story telling is the perfect base to work on repetitive actions e Academic Skills Include modules that focus on skills traditionally taught in educational institutions including math science letters shapes colors etc Language skills would be an academic skill but because they are often a primary focus for other applications I included them in their own category A story can have the perfect flexibility to give the possibility to create a big variety of activities e Life Vocational Skills Includes skills that allow individuals with autism to function in home work or everyday life Includes skills such as clothing toileting mea
127. l sistema stato denominato S COOL II Da al mondo il meglio di te e forse sarai preso a pedate non importa d il meglio di te Madre Teresa di Calcutta Ringraziamenti HI Eccoci qui dopo poco meno di 2 anni a scrivere di nuovo i ringraziamenti Non fraintendermi magari potessi scrivere ringraziamenti ogni anno significherebbe soltanto che ho fatto qualcosa che conta Perch veramente questo il significato della vita no Fare qualcosa che possa contare che ti ripaghi dei tanti sacrifici per cui valga la pena svegliarsi la mattina presto e andare a letto tardi s questo In questi lunghi mesi lontano da casa l ho capito fantastico poter vedere la propria opera fatta con le proprie mani vedersi realizzare di fronte ai tuoi occhi funzionare e ancora meglio avere un riscontro nei visi sorridenti nelle parole nelle emozioni condivise delle persone con cui ho avuto la fortuna di poter lavorare Quindi innanzitutto un grande grazie va a quelle persone che mi hanno sostenuto aiutato e consigliato nella realizzazione di questo grande ambaradam E tu mi chiedi chi sono Sono i miei familiari vicini come la mia preziosa famiglia e i parenti di Milano e lontani come gli zii della Sardegna A loro il mio pi grande grazie Sono gli amici e i compagni universitari delle belle serate passate insieme degli scherzi delle partite memorabili e dei viaggi senza la voglia di tornare Grazie Sono i
128. l times time management transportation safety scheduling and workplace skills If is true that life is made of stories this means that our environment can include different everyday actions e Sensory Physiological Responding Motor Skills Includes modules that focus on an individual s sensory or physiological responding such as perception activation recovery or regulation Also includes modules that focus on an individual s movement including fine motor gross motor motor fluency posture and gestures Our system should include a way to have the children move and percept at the same time 51 3 Preliminary Requirements 3 3 Settings The care of individuals with autism takes place in a number of different settings This category refers to the settings or locations in which technology is intended for use As in the previous section there are common primary settings and secondary settings Primary settings Special schools A place for educating individuals with autism Could be at all levels from pre school through post secondary education General schools A public private place for educating children with autism inside general education classrooms Clinics A place for professional practice that is not intended for education such as a doctor s office therapist s office or a specialty provider Secondary settings 52 Home a person with autism s and or their family s home or personal living s
129. lass was the most high functioning class was which could play the two player game by planning together and give suggestions to improve the game or story Elizabeth also pointed out that younger kids in Alysha and Mary Katherine s class are highly dependent on the fairy tale story and have major motor control issues 146 5 Final Analysis GENERAL FEEDBACK ON THEIR EXPERIENCE Benefits gt How did it benefit the teaching goals of the Lionheart School Y Elizabeth and Tamara confirmed that it addressed their goals of regulation attention collaboration problem solving and abstract critical thinking Elizabeth added we got what we are looking for There were improvements in movement involvement and engagement with peers mainly socializing v They also said that their OT Jess was remarkably happy with the progress in motor movements of the kids like Kevin gt Motor skills What are the potential improvements in their motor skills if any v Kevin is known to experience a lot of fatigue when he has to do complex motor tasks so he prefers sitting He also faces difficulty in jumping rope Ashley saw him jump on Sulli s monster head to get it which she described as being unbelievable The rope jump illusory training A potential idea that Gregory suggested towards the end was if we can use the jump on monster technique to make the kid jump on a rope really slow and then eliminate the monster from the sce
130. layer together need to collaborate and touch the object together Overall only Ashley forced to play collaboratively while the others had few tries Game multi single player per class 100 90 80 70 60 50 m multi player 40 single player 30 20 10 0 ALYSHA ASHLEY 164 5 Final Analysis Among eleven days there has been a big improvement of number of touched objects per second in Jill s and Ashley s class while Alysha used the game just one time In both red and blue lined it is clear how children improved their skills going from around one object touched every 3 seconds to one object touched every second which is a noticeable progress among these two medium and high functioning classes Game caught objects per second per class 1 4 1 2 1 0 0 8 si 0 ALYSHA 0 6 0 ASHLEY JILL 0 4 0 2 0 0 day 5 165 5 Final Analysis Video Most of the current methods for user experience evaluation require that users are able to reflect on and communicate their own experience b30 Such methods however are not suitable when users have limited communication skills Video recordings have proved to be a useful tool especially when designing for children with no or little speech We did not have the time to install the right instruments to systematically record varied and complete session Regarding to this we used the videos as additional voice that complement
131. li o storie ricche Viene avanzata la necessit di attenzione a evitare l utilizzo addestrativo Anna sottolinea l importanza di portare la relazione dentro la scena l aspetto collaborativo anche stando nel paradigma dell azione da compiere per superare il livello potrebbe essere un azione che necessita di una partecipazione di due persone per poter essere compiuta Chiara chiede se sia possibile per le situazioni ad alta necessit di supporto a livello motorio considerare questo supporto ma nascondendo nella rappresentazione del mondo dell applicazione quella figura rendendola una sorta di fantasma Mirko dice che sarebbe possibile eventualmente anche solo rendendo evidente una sola parte della persona che aiuta la mano ad esempio interposta tra il piano del bambino e quello del mondo virtuale Dal gruppo si sollevano alcuni dubbi sull opportunit di questa operazione Oltre a una gestione lineare della storia con una sequenza di scene sono proponibili bivi narrativi che possano o meno ricongiungersi a un certo punto Questo aspetto ha un impatto sul disegno e sull implementazione dell ambiente di editing Mirko dice che di questo requisito progettuale cosciente ma probabilmente in una prima versione da qui a dicembre non sar possibile soddisfarlo Giulia avanza delle perplessit sul possibile problema della rappresentazione di s proprio per quello che riguarda i bambini con autismo che cio questa propos
132. loper s error while most of the times it is due to the disturbance inside the play area This means that if a child enters inside the play area while another child is playing the system has some problems in recognizing the right gesture due to a difficulty of recognizing the right joints The smaller children in Alysha s class are the most unleashed and they use to run inside the play area for more than 15 times per story while Ashley and Jill can easily structure the play area in a more organized way and this causes a less disturbance rate Story avg disturbance rate child story per class 20 m ALYSHA E ASHLEY E JILL 163 5 Final Analysis Unfortunately in two weeks we could not gather enough logs for the free form activity and we simply shown its usage On the contrary the game had been played for enough time to understand some relevant data The game gives the opportunity to select to show the player s with their live image or their colored skeleton and can be played in single or multi player mode Alysha s kids preferred to play using only the live image while Ashley s and Jill s kids played for half of the time with the skeleton and the other half with live image Game selected avatar per class Live image Skeleton ALYSHA ASHLEY Differently from single player mode where the player needs to touch the object alone with the multiplayer mode everything is more difficult because both p
133. loper will just have to understand the preshared format of communication based on JSON and write its code with or without any other modules offered by S COOL or over the internet JAVASCRIPT LIBRARIES AT YOUR own choice Currently some modules are already PERFORMED GESTURES PLAYER S JOINTS available and in YOUR CODE BASED ON some weeks JAVASCRIPT everything will be ready to be deployed RECOGNIZED VOICE KINECT Overall the system is designed to be changed and upgraded easily and this cc gt a pre I i a es a a e amp DIFFERENT MODULES OFFERED BY S COOL and used adaptability adds to its lifetime value We adopt the concept of the economics of the framework and extend it in the field of systems architecture FRAMEWORK lt 3 4 TY SM MODULE 9 MODULE MODULE 4 169 6 Conclusion and future works 170 7 Appendix 7 Appendix 7 1 People pl Franca Garzotto is an Associate Professor of Computer Engineering at the Department of Electronics and Information Politecnico di Milano She has a Degree in Mathematics from the University of Padova Italy and a Ph D in Computer Engineering from the Politecnico di Milano Since she joined the Politecnico di Milano her theoretical research has focused on topics related to hyper document modeling hypermedia design methods usability engineering multichannel
134. lopment In the presence of a genetic predisposition to autism a number of nongenetic or environmental stresses appear to further increase a child s risk The clearest evidence of these autism risk factors involves events before and during birth They include advanced parental age at time of conception both mom and dad maternal illness during pregnancy and certain difficulties during birth particularly those involving periods of oxygen deprivation to the baby s brain It is important to keep in mind that these factors by themselves do not cause autism Rather in combination with genetic risk factors they appear to modestly increase risk A growing body of research suggests that a woman can reduce her risk of having a child with autism by taking prenatal vitamins containing folic acid and or eating a diet rich in folic acid at least 600 mcg a day during the months before and after conception Increasingly researchers are looking at the role of the immune system in autism Each individual with autism is unique Many of those on the autism spectrum have exceptional abilities in visual skills music and academic skills About 40 percent have average to above average intellectual abilities Indeed many persons on the spectrum take deserved pride in their distinctive abilities and atypical ways of viewing the world Others with autism have significant disability and are unable to live independently About 25 percent of i
135. lted from a single par e how often a story is played par a e how often a class choose a type of avatar par b e if the teacher decides to enable hints par c e if children are using the Promethean Board par d e how often a gesture failed par k And more complex variables resulted from a combination of different pars such as when the system is waiting for a gesture to be performed by the child par i I save the skeleton joints par 1 of the child and until happen that 1 the gesture is recognized par j par i 2 the teacher help par h par i Finally I can monitor the entire flow using par e and f For each game I log the following parameters Choice of the background and related objects a Type of avatar b Time of game c Number of players every time it changes d Object collided every time a player gets an object All of these parameters result in simple calculus to obtain meaningful data For example how many objects have been collected alone par c and d how often an avatar is chosen par a 131 4 S COOL s Iterative Development Implementation of Multiple users detection mB2 When we decided to focus on games we had to enable the possibility to track the second player In the ALPHA version the closest skeleton was the player to track and the code was easy while in the BETA version things are getting harder int closestSkeletonIdx 1 int secondSkeletonIdx 1 int i 0
136. may not be evident On the other hand the method has many potential strengths when evaluation user experience of children with special needs It does not rely on users ability to express themselves through speaking or writing but nevertheless keeps than in the centre of the evaluation As such the evaluation can be used as an additional voice that complements those of parents and teachers who are often used as source of information Their method highlights the variability of user needs and makes it possible to compare design alternatives or to evaluate how experience develops over time In addition it does not require lot of resources which makes it a practical design tool mG3 SURVEY amp MEETINGS As you can understand from this document most of the empirical study has been carried out using meetings with teachers and educators of the School Meetings have been mostly conducted by us as organized focus group and somehow interviews about specific questions Since the relationship with the Lionheart School started few time ago we wanted to structure the work by beginning with weekly meetings and then by the time with online surveys on more specific and personalized requirements 126 4 S COOL s Iterative Development mG4 AUTOMATIC MEASUREMENTS During my experience in the Ubicomp group I met Jonathan Bidwell and after he explained me his project we decided to build something together John s project is abo
137. ments Identifications and Analysis This phase starts with gathering the business requirements in the baseline iteration In the subsequent iterations as the product matures identification of system requirements subsystem requirements and unit requirements are all done in this phase This also includes understanding the system requirements by continuous communication between the customer and the system analyst Design Design phase starts with the conceptual design in the baseline iteration and involves architectural design logical design of modules physical product design and final design in the subsequent iterations Implementation Construct phase refers to production of the actual software product at every iteration In the baseline iteration when the product is just thought of and the design is being developed a POC Proof of Concept is developed in this phase to get customer feedback Then in the subsequent iterations with higher clarity on requirements and design details a working model of the software called build is produced with a version number letter These builds are sent shown to customer for feedback Testing Evaluation and Cost Analysis Risk Analysis includes identifying estimating and monitoring technical feasibility and management risks such as schedule slippage and cost overrun After testing the build at the end of first iteration the customer evaluates the software and provides feedback At the end of the
138. ments and subsequent choices create the fundamental design base to be considered although the other possible solutions the ones not chosen in the first iteration should not be blocked from them To what extent the first design phase and later implementation should take into account the possibility to implement the remaining ideas in the near future and if possible trace an easygoing path To do this we must reconsider design requirements for each solution and understand which best overall solution that accomplishes every goal there could be 83 4 S COOL s Iterative Development Conceptual design The teacher green creates the story via a simple tool which gives the possibility to select sounds images from a premade library or upload them select a gesture to be triggered and relative animations Children orange within a designated play area red act out the story in front of both screen and Kinect Their actions correspond to reactions i e animations of the system and its flow During story time the teacher can easily manage on the fly the entire story flow changing pages gesture to be recognized sounds and many other features Children will be able to play the same story on a multi touch device 84 4 S COOL s Iterative Development DESIGN OF THE MOBILE APP PROTOTYPE This is just a first graphical prototype of the mobile app that have been introduced to the teach
139. n Joints JointID ElbowLeft Position X 6 TS return GesturePartResult Suceed 8 9 hand has not dropped but is not quite where we expect it to be pausing till next frame 10 return GesturePartResult Pausing ALLE 12 hand dropped no gesture fails 13 return GesturePartResult Fail 99 4 S COOL s Iterative Development View The story view is the core of the entire story telling application To sum up it e Gets the JSON which represents attributes actions and reactions of the story bottom gt up e Draws the elements on the ground the visible area for each page draw media objects and avatar e Gathers the JSON from the Kinect App skeleton joints gestures e Manages the entire story change page sequence wait animate gt COOL DRAW i JSON AVATAR from O KINECT S tu X On gesture change page recognized Saa sequence K gestures do nothin CONTROLLER In the following page you can see how joints and gestures are sent from C app to the JS app Kinect tracks the entire skeleton of the player from the head to the feet HAND RIGHT HEAD SHOULDER CENTER HAND LEFT x hci and gives you a list of joints shown WRIST_RIGHT WRIST_LEFT 5 y ELBOW RIGHT ELBOW LEFT in this figure SHOULDER_RIGHT SHOULDER LEFT SPINE For each joint the programmer can HIP_RIGHT J read its e x axis horizontal left to right KNEE_RIGHT KNEE LEFT e y axis vertical
140. n about the strengths and limitations of the child therefore gaining the ability to tailor interventions as necessary while strengthening the bond between the parent and child and fostering social and emotional development of the child The time requirement varies from 14 to 35 hours per week This intervention increases levels of e Social functioning e Emotional functioning e Information gathering ati ml aa Picture Exchange Communication System PECS Communication system developed to assist students in building fundamental language skills eventually leading to spontaneous communication The tiered intervention supports the learner in learning to identify discriminate between and then exchange different symbols with a partner as a means to communicate a want 16 2 State of the Art As long as the child is engaged the session proceeds usually 20 30 minutes Studies demonstrate it increases levels of e Speech and Language development e Social communicative behaviors t amp i T O Z i g i sand and water table sidewalk chalk Social stories Personalized stories that systematically describe a situation skill or concept in terms of relevant social cues perspectives and common responses modeling and providing a socially accepted behavior option Time requirements vary per story approximately 5 10 minutes prior to difficult situation Social stories increa
141. n programmer to easily create software with programming features The programming features are built in but hidden behind buttons and other tools so the teacher does not need to know how to program Our ideas of an Authoring System provides lots of graphics interaction and other tools educational software needs Teachers should be able to create their own personalized stories taking the images from the web scanner photo cameras and whatever device sounds from the web or from their own source gestures and animations from a predefined library An authoring system usually includes an authoring language a programming language built or extended with functionality for representing the tutoring system Our goal is to provide the easiest way to associate animations actions to gestures without having visible programming languages The authoring tool will satisfy the following needs and problems 2 Objects in Kinect the Dots were placed close to the top of the screen 3 Kinect the Dots failed to integrate closely with the existing storytelling activities Students respond better to an application that conformed more closely to their existing storytelling activities in look and feel 5 The teachers noted that specific words that are repeated numerous times in a story are especially motivating Repetition and simple language to motivate children in support of educational and therapeutic goals 7 If the story is mostly self contained within
142. nce success in these educational contexts The educational inclusion of students with autism and other disabilities has been a fiercely controversial topic Historically students with disabilities have been segregated from their peers even from society as a whole More recently however there has been an increasing trend to include students with autism and other disabilities in general education classrooms along with their typically developing peers This trend has stemmed largely from theoretical arguments related to social development and legal issues related to the civil rights movement With regard to the potential social outcomes of students with autism schooled in general versus special education settings researchers have evaluated students with autism on a number of dependent variables holding educational placement as the independent variable 12 2 State of the Art For example researchers have documented that students with disabilities including students with autism who are fully included a display higher levels of engagement and social interaction b give and receive higher levels of social support c have larger friendship networks d have developmentally more advanced individualized education plan goals than their counterparts in segregated placements By modifying discriminative stimuli for both appropriate and inappropriate behavior antecedent procedures can be designed to prevent and reduce challenging behav
143. ndividuals with ASD are nonverbal but can learn to communicate using other means 11 2 State of the Art 2 1 2 ASD Children in the school setting s6 b1 b2 b4 b5 b6 b7 b8 b9 b10 The range of impairment and the increase in the prevalence of autism present great challenges to the educational system Across the United States schools have reported an average increase of more than 800 since 1992 in the number of children with ASD being served in the educational system Individuals With Disabilities Education Act IDEA 1997 The research related to educational and behavioral interventions for children with ASD is extensive including various case studies experimental studies and theoretical orientations all of which do not adequately offer guidelines for providing best practice In addition the popular media including celebrities and the Internet will often advocate for new unproven and perhaps unsafe interventions Given that there is no well established consensus regarding the appropriate educational practice for students with autism school districts often need to make difficult decisions about the programs that they choose to implement Children with autism can benefit from participation in inclusive classroom environments and many experts assert that inclusion is a civil right and is responsible for nurturing appropriate social development However most children with autism require specialized supports to experie
144. ne some how y Wheel barrow Kevin never runs with the wheel barrow Ashley encouraged him reminding him about the points he scored getting the monsters Kevin ran with the wheel barrow in his hand Jess is thrilled with Kevin s improvement Y It is very difficult for him to stand but another day he religiously waited for the computer to shut down and restart the Kinect game which the teachers described as requiring a ot of effort on his part v Gracialla in Mary Katherine s class also has constrained movements but when they made her sit on a chair in front of the screen she could do the movements Other kids in her class like Zackery and Mathew also face motor issues gt Emotional development Was there any development in their level of excitement or emotional state y In Jill s class one of the girls saw her live image on the screen and bolted out of the room Jill described it as a positive over whelming reaction They had to put the skeleton on to get her back to the room Y Other kids also displayed a lot of excitement over the game and asked their teachers when are we going to play with the Kinect 147 5 Final Analysis gt Socializing effect Did the children interact look talk plan with other children while playing Did they talk to you while they were playing v This effect was seen maximum in Jill and Ashley s classes Jill s class cannot connect to the two player game but loved dancing together
145. ned refined and tested his own framework 34 2 State of the Art The table in the following pages associates for each papers different parameters based on Interactive Technology Platform described in the previous page Domain described in chap 3 2 Goal described below Target End User described in chap 3 1 Setting described in chap 3 3 Publication Venue described below Empirical Support described below Technology maturity described below Goal section is composed by Functional Assessment applications or project focused on the collection and review of data over time to assess and individual s learning capability or levels of functioning The data collected is intended for end users and or people caring directly for individuals with autism Diagnosis Screening Includes applications that assess the risk of an autism diagnosis in the general population or that assist in helping make or understand the severity of an autism diagnosis Intervention Education Includes applications that attempt to improve or produce a specific outcome in an individual with autism May focus on teaching new skills maintaining or practicing skills or changing behaviors Scientific Assessment Includes applications or projects that use technology in the collection and analysis of data by researchers to understand more about autism and its features or characteristics Parent Clinical Training Includes applications
146. ng on the Lionheart and Gwinnet feedbacks Our meetings took place in the Ubicomp office were we had the time to write down on a big whiteboard where all the people from Ubicomp could say their feelings and suggestions PROS ie Ei gt AVATAR MODE ao Vine rie x Kimect Stony D Auth Took i _ ad column and For each module 1 column we highlighted some strengths 2 weaknesses 31 column finding a solution al column to implement by the next meeting with Lionheart teachers 113 4 S COOL s Iterative Development mA1 STORY CREATION TOOL AUTHORING TOOL The story creation tool is a powerful tool and gives teachers the possibility to support kids individual profiles and academic goals On the other hand a story creation tool should be able to deploy a good working story which in the second iteration is not the case There will be surely a time where we will create an authoring tool because we are aware that teachers know how to use new technologies interfaces but now we are just focusing on providing a well working story mA2 REAL TIME STORIES When we went to the classes most of the teachers were using the Promethean boards explained later mod E 5 In particular during story time they were doing two different kinds of stories 1 The normal story children act the story out running and shouting The best way to improve motor skills as much as verbal skills 2 The real time story children
147. ng with the interactive stories In addition initial test have demonstrated that story creation is accessible to a wide area of people with different technological skill levels This was to me a good opportunity to work with him and to build something together since the early phases To do this we needed a common ground to work on In the design and development phase you will find our meeting points 82 4 S COOL s Iterative Development 4 1 2 Design Since the time to develop was short and the solutions were many we decided to focus on most critical modules and leave the others for the next iterations after a subsequent analysis N SOLUTION CRITICAL 1 TIME DIFFICUL 3 TY COST 1 3 A STORY CREATION TOOL 7 E B 1 GESTURE DETECTION 3 2 B 2 MULTIPLE USERS 1 2 DETECTION B 3 LEVELS OF GESTURES 1 2 C 1 STORY 3 2 C 2 AVATAR 2 2 C 3 TUTORIAL 1 1 D 1 REMOTE MOUSE 3 1 D 2 MOBILE REMOTE 2 2 CONTROLLER E 1 TABLET 1 2 E 2 SPEECH RECOGNITION 1 1 E 3 NFC 1 1 E 4 ARDUINO 1 2 CRITICAL COLUMN 1 uncritical 2 quite critical 3 critical TIME DIFFICULTY COST COLUMN 1 short time easy 2 medium time medium 3 long time difficult All the bold solutions will be implemented in some ways in the first month period September In October 3 we will have a meeting with the Lionheart School showing what we have done bold solutions explaining the further directions and gathering their feedbacks The first require
148. o a volersi davvero bene ed anche ad andare insieme al supermercato senza carrello per IN amp AUT Percezioni sensoriali e comunicazione nell autismo Secondo convegno internazionale Crema 22 23 24 marzo 2012 VII Contents Most of the work ia an Hoglih otherwise specified in Italian I I Gran parte del lavoro in Inglese Abstract altrimenti specificato in Italiano Sommario ll Il Ringraziamenti ll I eden ia 1 Introduction iris lille lait ili 3 LE CONTER elio E E ani 5 12 PUOSC ei a Daddies a A did ea 6 1 3 Structure ee eaae ea a ede 7 RETRO E TOTI TT 9 Lila 9 2 1 1 What Causes Autism solaio la alal 11 2 1 2 ASD Children in the school setting 12 2 1 3 Educational praeti e ile alan 15 2 2 Story Telling and Child iii 18 2 3 Eowuchless Eschnolofpie sassate 21 2 3 1 Microsoft Kinect csrl ilaele la 21 23 2 Nintendo WY 1 aid les 27 2 3 3 Other Touchless Tech logistica lierna aa 28 2 4 Technology and autism ss seus Seen ian eed ie x ene iaia 33 2 4 1 Touchless Technologies and Autism 38 2 5 Previous experiences iii lio 41 25 1 Kinect the Does ee 42 o s A dl es De hE oo see mew Re BEDE oraninin RET treet Ea Albi sd dii dialer 43 23 PWR Pt GI seats eee a ee ee eee eee ees 44 2 6 A new paradioni HACE ssp liana 45 3 Preliminary Requirements ace aes 47 Sl Stakehold rsenct Ack oe ews eine ei a a eee es ens 48 o Domaine vii niet 51 DISPO Dili cpr colin
149. o corolla rin cnr crederai ainsi Suet atl 52 A Work PIO se tree eto eee etn crete eee coder orarie EE AR A siedono OOA 53 Si dmmal iccgbaclssstataeelelleelia 56 S COOL s Iterative Development senta 65 4T rOrsaMZat O e Ea ETE aE ETE een di id i 67 SCOLA reia e a a ennai a a 69 ALAC Requirements oh a Reco Ae eek E A A 69 4423 SOG UE iires are duped eb taped ao Seed alettoni Ari ir 83 4 1 3 Implementati ni ots far acts pil cloro opel sonale onice 86 4 TA Evaluatio GUN eee ee ae en re ee Pet et a Pr eae ne ten eT ee rt eet ee eR PT Tat 102 COOL Belen ella ila 103 4T Requirements gentile elia 103 46 Delli Dei 113 Asl Fe implementation rat IA ta 128 Le Eealiaonacelleclicolielhi ella Oy Oren AES 134 42 SCOOL Gamma iii 137 4 21 Regiirotnieht disertare 137 4 2 2 Design and Implem ntationi rele eee 139 LI A DI ERROR ARR TI 141 Enable 145 Conclusion and a EROE meee ere ee ene mettre eet one ae Nea 167 6 1 Towards a framework seis ol sl hi slash canna casas 169 PRCT E LIRA RR O TT 171 AOSTA 171 TD Mod le tanne pesto prati hn Sas chun a a Wie Sane O A are di 172 Gs User manual airone lei ene riale a 173 7 3 1 Position the Kinect sensor eco ug occas is tad titi Aad Riess Rian tides 173 Ted ett your PAYPAL ASA i 174 oS Room Nobile 174 Toe Install be solbWar seine mney Ener ene rey ici inci ini ei 175 7 3 5 Connect the COD CS see Soret iosa rana lire Samat parl ia 175 7 3 6 Use the system for the first times tge 0se 0 eee se creeranno
150. o do Was the instruction sheet helpful gt Did you use the video hints gt Did you find or use the back button at the bottom left of the screen STORY gt Is this better than their regular story time gt Can this replace your story time videos Or it works better as a separate activity with the story time videos gt Gestures On the whole which gesture had the maximum issue Clomp Wiggle Shirt Shake Clap Nod Worked Failed Too easy 142 4 S COOL s Iterative Development GAME gt gt Did the children plan a strategy to get objects in the two player game Which avatar did they prefer skeleton or the live image And Why gt Live Image Did the children notice any lag in the live image recognition Did they notice that their image wasn t whole missing hair on head Any other glitches that affected their performance FREEFORM Which avatar was most popular Watch the videos the teachers want to share with everyone or with us separately From their perspective we intend to understand any memorable moments What was so special about it Anything that happen which never happened before ADDITIONAL FEATURES PROBLEMS STORY gt gt gt Disjointed Little old lady avatar Fix it want it remove it What developments are needed in the story Additional features Should we make gestures easier or tougher What is the usual tendency of the child Do you want live image in t
151. oject is part of The Centre for Applied Research in Inclusive Art and Design SETTINGS ABOUT 39 2 State of the Art Pictogram room Pictogram Room s35 is a set of educational video games available for free download designed to respond to a set of needs for people with Autism Spectrum Disorders ASD on which until now it was difficult to work Difficulties in understanding body language recognizing oneself imitation or joint attention these are critical skills for the development of a child with ASD which can be addressed in an entertaining way in the Pictogram Room where autistic children and adults with the support of their tutor or in the company of other children without autism learn while they play Using a camera projector system with movement recognition technology the image of the player is reflected on the screen with a number of graphic and musical elements added in order to guide the learning process To use this application you will need a PC running Windows 7 a Kinect sensor device and the projection surface television screen or wall on which to view the Pictogram Room images all technical requirements are listed on the website The Games in the Pictogram Room devised to draw on the strengths of people with autism are arranged into different sections depending on pedagogical level Individual work interaction with tutor self awareness attention imitation and communication All
152. omplexly about it and to communicate what they have learned in a more creative way When students write scripts together for example they have to decide how the blend different languages voices and ideas and they have to agree on what tone and what angle to use Brain researchers say human beings are hardwired to tell stories to organize experience into a meaningful whole that can be shared with others Giving students opportunities to use and direct this natural drive gives them a sense of confidence while it develops fundamental intellectual skills Encouraging your students to create digital stories is not just a ploy to keep them interested digital storytelling has proven educational benefits that help prepare students for success in the 21st century Playing digital stories e Encourages research by helping students invest in issues and engaging them in dynamic interactive processes of learning e Fosters critical thinking skills helping students think more deeply clearly and complexly about content especially when that content is challenging e It gives them practice in the skills of sequencing logic and constructing a persuasive argument Creating storyboards and then editing stories reinforces these skills e Encourages students to write and to work at becoming better writers Many students don t think of themselves as writers or are daunted by the writing process Writing revising and editing scripts for digisto
153. on based touchless interaction and a storytelling approach for autistic children and has been designed using a participatory design approach with a strong involvement of therapists psychologists educators and children as users from the Lionhearth school Atlanta So far this interaction paradigm has been mainly applied to 3 classes of different functional levels and involved 20 children aged from 5 to 13 The empirical study is still being carried but improvements in movement communication involvement and engagement with peers were noticed immediately Further deployment is to create a broad and comprehensive curriculum of educational and therapeutic activities based on a framework specifically designed for children with autism to support their development of motor speech and language cognitive academic and social skills The system has been named S COOL Sommario I Disturbo dello Spettro Autistico DSA e autismo sono entrambi termini generali di un gruppo di patologie complesse dello sviluppo cerebrale Queste patologie sono caratterizzate in vari gradi da difficolt in interazione sociale comunicazione verbale e non verbale e pattern ripetitivi di comportamento spesso accompagnati da menomazioni motorie Per diversi anni differenti tecniche sono state utilizzate per migliorare la qualit della vita delle persone con disabilit dello sviluppo Tuttavia l utilizzo della tecnologia a favore dei bimbi autistici c
154. ong learners Some of the educational benefits of digital storytelling are engagement with real world issues careful analysis excitement about learning investment in students own performance conflict resolution community connections and much more Many teachers have noted their students grades go up when they work on digital storytelling projects They may also be more likely to do their homework and to come to class eager to work Furthermore digital stories give voice to those who don t always participate in class Students who work together on long term projects are less likely to be absent They also develop cooperation and communication skills practice problem solving and critical thinking skills and improve their scores When students integrate technology into their projects these benefits increase Students of all ages enjoy creating stories and more and more students are eager to use technological tools to play those stories Digital story telling the art of combining storytelling with some mixture of digital graphics text recorded audio narration video and music to communicate information about a specific theme or topic enables them to do both at once 18 2 State of the Art But digital storytelling is not just frivolous play it is serious play with a big educational payoff because the process of constructing digital stories inspires students to dig deeper into their subject to think more c
155. ontinua a ricevere un attenzione limitata nonostante che essa tenda ad essere un area molto stimolante per molti di questi bambini Questo lavoro presenta una soluzione in quest ampio e svariato panorama sviluppando tecnologie interattive innovative che possano essere integrate con le attivit terapeutiche e scolastiche e autonomamente utilizzate da terapisti e insegnanti per promuovere attraverso il divertimento interazione sociale capacit comunicative e abilit motorie Questa tesi rappresenta un contributo allo stato corrente dell arte nel campo sviluppando e validando uno strumento innovativo che per la prima volta combina interazioni touchless e storytelling ed stato progettato seguendo un design partecipativo con grande coinvolgimento di terapisti psicologi educatori e bambini come fruitori del sistema dalla scuola Lionheart Atlanta Finora questo paradigma interattivo stato applicato a 3 classi di differenti livelli di funzionamento e ha coinvolto 20 bambini di et compresa tra i 5 e i 13 anni Studi empirici sono ancora in corso ma miglioramenti nel movimento e nel coinvolgimento comunicativo e sociale sono stati notati immediatamente Ulteriori sviluppi mireranno a creare un vasto curriculum di attivit terapeutico educative basate su un architettura modulare e progettate specificatamente per bimbi autistici per supportare il loro sviluppo motorio comunicativo cognitivo accademico e sociale I
156. oposes involves interaction between people and computers As we learnt ASD children more than others tends to stereotypy and dive themselves in other worlds Thus giving them the possibility to communicate through another world is easy to lose them That s why in my brand new notion I wanted to add an H stands for human again because the primary actor must not be the computer as supposed in HCI but should be the child that assisted by the technology is elicited to communicate with another child 46 3 Preliminary Requirements This is just the start of the process It s easy to draw up plans but there will still be a lot of work left to do David Watson Taylor 3 Preliminary Requirements This chapter is dedicated to a general overview of initial phases of my research After this brief introduction I will explain the general stakeholders of the project dividing them in target end users and secondary stakeholders Some space will be given to explain the Lionheart School s36 and my University Supervisors p1 p2 p3 After this once clarified domain and divided settings in primary and secondary I will show the work plan based on Stanford Design Thinking s37 In this chapter are also pasted some reports done in some hospitals in Italy and will be given some motivation of using Kinect 47 3 Preliminary Requirements 3 1 Stakeholders This subsection focuses on the person or persons which c
157. ory reading doesn t have an effect gt Do you want live image in the story Y Yes they found the live image to be the most effective GAME Jill started by saying the game was a high success Under the sea children knew the song from little mermaid relate to it Ashley s class mars was more popular because it had monsters inc characters gt Did the children plan a strategy to get objects in the two player game v Ashley s kids did Jill s did not gt Which avatar did they prefer skeleton or the live image And Why Did the children notice any lag in the live image recognition Did they notice that their image wasn t whole missing hair on head Any other glitches that affected their performance v Live Image v Kids did not mind the glitches with green screen v They don t mind their image pixilated but they mind the disjointed little old lady avatar Other considerations v Allow choosing between feet and hand controls v Bigger objects on the game v Use feet to get objects Ashley Jill v Jess Ashley Jill all feel that feet hand they should be able to select what to use Cross gesture 153 gt 5 Final Analysis In the two player game what kind of interactions you want Hi 5 Hand shake v Clap to remove objects for advanced level Ashley v Avatars throw a ball back and forth on screen Ashley Y Pick scenes that go with our avatars ex power ranger scene to go with iron man etc
158. oyed and appreciated but after they finished they went woosh it was tiring 109 4 S COOL s Iterative Development Boo Technology The green screen had delays and got stuck at times but the players were ok with it and did not complain They were too amazed to see themselves on screen than to complain The active player was the one closer to the Kinect so if parent stood behind the child and tried the child was still the active player We could not ask the child to move back since the parent would block its view Observations Our first visitor was a mom and her daughter aged 10 13 years The mom was very excited about the game and was encouraging her daughter also but the daughter was like stop mom Make it stop But the mother didn t stop and the daughter also liked standing and watching The mother got us our next customers too They were siblings who tried out each on one set up They did well too Keeping scores was encouraging to the player Some kids enjoyed the shadow more as they didn t like to see themselves on the screen Next there were two siblings one aged 10 and the other 8 One played on the shadow and other on green screen Both switched played another turn then start comparing scores They both did well We are not sure but the 10 year old scored very well and is our high scorer 1610 I guess I did not i note his name The parents and kids thanked us and left The 10 years old came back in
159. pace Community technology is intended for use while the user is freely moving in public spaces like oratories and centers for entertainment Research Lab technology is intended for use in a research laboratory under careful observation or for controlled settings 3 Preliminary Requirements 3 4 Work plan A first raw work plan based on the Stanford Design Thinking s37 has been traced from the beginning of my experience May 2013 Basically Design Thinking is a creative and user centered design methodology It differs from traditional design approaches in specific ways described below I choose to approach to this method because it is particularly useful for addressing ill defined or tricky problems Even if I had a report of past feedbacks of the project problem and solution had to be organized and understood in a more systematic engineering way Unlike analytical thinking which is associated with the breaking down of ideas Design Thinking is a creative process based on the building up of ideas Analytical approaches focus on narrowing the design choices while Design Thinking focuses on going broad at least during the early stages of the process In Design Thinking me as designer I did not make any early judgments about the quality of ideas As a result this minimizes the fear of failure and maximizes input and participation in the ideation brainstorming and prototype in the early phases The Stanford Design
160. proprietary software which provide full body 3D motion capture facial recognition and voice recognition capabilities At launch voice recognition was only made available in Japan the United Kingdom Canada and the United States Mainland Europe received the feature later in spring 2011 Currently voice recognition is supported in Australia Canada France Germany Ireland Italy Japan Mexico New Zealand United Kingdom and United States The Kinect sensor s microphone array enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression allowing for things such as headset free party chat over Xbox Live XBOX 360 4 O Infrared optics RGB camera Motorized tilt Multi array microphone 22 2 State of the Art The depth sensor consists of an infrared laser projector combined with a monochrome CMOS sensor which captures video data in 3D under any ambient light conditions The sensing range of the depth sensor is adjustable and the Kinect software is capable of automatically calibrating the sensor based on gameplay and the player s physical environment accommodating for the presence of furniture or other obstacles Described by Microsoft personnel as the primary innovation of Kinect the software technology enables advanced gesture recognition facial recognition and voice recognition According to information supplied to retailers Kinect is capable of simultaneously tracking up to six p
161. psychologists educators and children as players of stories has been carried out with the school So far this interaction paradigm has been mainly applied to 3 classes of different functional levels and involved 20 children aged from 5 to 13 The empirical study is still being carried through video observation focus group semi structured interviews and log files 1 Introduction Overall children reacted positively to the introduction of the new system and nowadays it is constantly integrated into Lionheart s activities Improvements in movement involvement and engagement with peers were noticed Furthermore the Occupational Therapist perceived an unbelievable progress in motor movements in some kids while educators detected not only an enhancement of socialization but also communication and shared strategy planning The project is a continuing work initial development and testing are promising The ability to easily create or modify learning material around a child s special interest is seen as a major advantage to engaging individuals in social centric activities Further work is in progress to test the platform s ability to automatically collect and analyze data and to create a modular architecture that will make it reusable and adaptable for different developers and kids needs 1 Introduction 1 1 Context While cutting edge learning technologies have transformed communication connectivity and sensiti
162. ras It is designed to track fingers or similar items such as a pen which cross into the observed area to a spatial precision of about 0 01 mm 28 2 State of the Art Tobii Rex Tobii Rex s17 is an eye tracking a device from Sweden which works ge with any computer running on Windows 8 The device has a pair of infrared sensors built in that will track the user s eyes Users need just place Tobii Rex on the bottom part of the screen and it will capture eye movements engaging in Gaze Interaction Basically you use your eyes like you would the mouse cursor Wherever you are looking the cursor will appear in the precise spot on screen To select you can use the touchpad Although not entirely touchless at least now you need not move a bulky mouse around It s also a great alternative to using the finger on a touch tablet which blocks off the view of what you want to click or select Elliptic Labs Elliptic Labs s18 allows you to operate your computer without touching it with the Windows 8 Gesture Suite It uses ultrasound so it works not with cameras but with your audio tools Ideally you need 6 speakers and 8 microphones but the dedicated speakers on laptops and a normal microphone could work too The speaker will emit ultrasound which will bounce to microphones so that it could track a user s hand movements which will be interpreted by the Elliptic Labs software This technology is desi
163. recoded if needed It has to able to recognize this kind of gestures Gesture must be revisited and recoded if needed It has to able to recognize different levels of gesture accuracy Gesture recognizer must be revisited and recoded if needed We have to implement more gestures Further solution an authoring tool to setup gestures Track more users recognizer Create a more understandable way for children of being inside the story Stick figure avatar Tutorial For some rows a better analysis of problems and solutions was required before jumping into the design phase Below I organized my analysis ordering each solution with a letter grouped some problems together since they require almost the same solution and wrote down my thoughts about them Each solution represented with a letter will be split into one or more modules on the basis of its task duration difficulty size easy medium difficult 70 4 S COOL s Iterative Development mA1 STORY CREATION TOOL AUTHORING TOOL s39 An authoring tool is a program which has pre programmed elements for the development of interactive multimedia software titles Authoring systems can be defined as software which allows its user to create multimedia applications for manipulating multimedia objects Users in this case are teachers In our case since users are teachers in a school we can speak about the development of educational software that allows a no
164. ric activities In addition the platforms ability to record story interaction is very promising as it allows for the collection of data that may reveal learning ability that otherwise would not be available The development of technology solutions with and for children with autism is internationally and nationally acute as these children still have very limited means to participate in everyday life activities due to their disability We expect that our methodological choices and the establishment of a learning environment that supports active roles for children body and soul will gain new scientific knowledge of the possibilities and limitations of technologies in education The design empowers the participating children and the findings enable them to be integrated into society instead of being trapped in negative and problem based assumptions about children with special needs In conclusion with a little bit of space and a small outlay in money an interactive area can be set up to try to engage our children and help their interactions creativity and movement Different sessions and different programs produce different responses and these responses also evolve over time Some children love it and take to it straight away others need some encouragement and patience others aren t bothered with it at all but in my experience that is true of everything What is important is that these Kinect programs have no real parallel in our schools
165. ries makes this process natural and enjoyable It promotes student initiated revision instead of editing according to a teacher s markups or a grade requirement e Gives students a voice It empowers them to find their own unique point of view and relationship to the material they re investigating and to express that viewpoint more fully and clearly Many students find that sharing their digistories is far less threatening than reading their writing out loud e Tells a personal narrative Enables students to share about themselves such as a key turning point in their life or their family history Digistories can embody the story of someone else where the student takes on their persona and shares from their point of view 19 2 State of the Art Helps students retain knowledge longer Researchers at Georgetown University discovered that the emotional aspect of telling stories improves learning because it helps students remember what they have learned Enhances learning by encouraging students to communicate effectively It also promotes classroom discussion community awareness global awareness and a connection between what students do in the classroom and the wider community Posting students digistory projects on class web sites or school portals reinforces these connections and improves communication Helps students make a connection between what they learn in the classroom and what goes on outside of the classroom Digistory pro
166. round social thinking DIR Floortime developmental individual differences relationship based e Music therapy 49 3 Preliminary Requirements University Supervisors Beside the above stakeholders I strictly worked under the supervision of two Universities in the person of Professor Franca Garzotto p1 Politecnico di Milano provision of initial guidelines for development of the thesis work coordination with Georgia Tech supervisors Prof Gregory Abowd and Dr Agata Rozga for planning the work refining requirements design and implementation solutions remote supervision of student s activities regular Skype meetings guidance to work reporting in the final thesis Prof Gregory Abowd p2 and Dr Agata Rozga p3 Georgia Institute of Technology 50 provision of initial requirements and design concepts for the tools developed in the thesis recruitment of the partner and the participants for the empirical study Lionhearth School 225 Roswell Street Alpharetta GA 30009 devoted to refine requirements and evaluate the thesis outcomes coordination with Polimi thesis supervisor prof Garzotto for planning the work defining requirements specifying design and implementation solutions local supervision of student s activities regular in site meetings 3 Preliminary Requirements 3 2 Domain The software in all of its modules explained in Chapter 4 should help with acquisition of certain s
167. rta dal sistema sembra esserci Probabilmente va meglio definito come si presenta complessivamente il prodotto una suite di applicazioni coordinate da un unica filosofia un ambiente riabilitativo un ambiente prevalentemente creativo o di gioco Mirko cita a questo riguardo ad esempio la grande messe di applicazioni molto specifiche che vede sull Ipad del bambino con autismo con cui sta sviluppando una conoscenza Sarebbe possibile dice integrare l app per apprendere la matematica Io credo che il punto su questo aspetto sia decidere cosa debba essere il prodotto Un profilo necessario altrimenti diventa una nebulosa dentro cui 59 3 Preliminary Requirements tutto sembra possibile ma di cui non si colgono i contorni questo penso io che sia necessario definire i contorni anche larghi ma che ci siano questo il programma lo fa questo non lo fa Nicoletta sembra interessata a un utilizzo pi strettamente riabilitativo in ambito logopedico e sottolinea l importanza della possibilit di utilizzare l azione per disambiguare il significato di una parola in particolare di un verbo associando l espressione verbale a uno schema dinamico proposto e che possa essere riprodotto dal bambino Viene avanzata l osservazione sulla possibile complessit della storia che potrebbe risultare oltre le capacit di gestione di comprensione del bambino Esce quindi il tema della popolazione di dettagli nella storia storie spoglie essenzia
168. ry Examples JAMA JAMIA AMIA Empirical support is arranged as follows Descriptive Study design seeks to observe natural behaviors without affecting them in any way Common approaches include Observational methods e g ethnography Case Study methods and Survey methods Correlational Quasi Experimental Study design involves comparing groups without any random pre selection processes on the same variable s to assess group similarities differences and or determine the degree to which variables tend to co occur or are related to each other They are similar to Experimental Study Designs but lack random assignment of study participants Common approaches include Nonequivalent Groups Design Regression Discontinuity Design Retrospective Designs and Prospective Designs Experimental Study designs seek to determine whether a program or intervention had the intended causal effect on study participants Common approaches include Randomized Controlled Trials Solomon Four Group Design Within Subject Design Repeated Measures Design and Counterbalanced Measures Design Lastly Technology Maturity is categorized in this way 36 Design Concept Non Functional Prototype The technology is not yet functional It may be an idea expressed as a sketch storyboard interface mockup etc May also include non functional but interactive prototypes such as paper prototypes Wizard of Oz prototypes video prototypes etc Functional Prototype
169. s the basic requirement for eyeSight to work is to have a basic 2D webcam even the built in ones work and the software Your screen need not even be one with touch technology To navigate you just move your finger to move the cursor push your finger like how you push a button to click EyeSight does not only work with laptops and computers but it also works with a lot of other devices such as tablets televisions and much more As of now eyeSight is not for consumers use but it is now offering software development kits SDK for the Windows Android and Linux platforms 30 2 State of the Art Mauz Mauz s21 is a third party device that turns your iPhone into a trackpad or mouse Download the driver into your computer and the app to your iPhone then connect the device to your iPhone via the charger port Mauz is connected to the computer via Wi Fi connection Start navigating through your computer like you would a regular mouse left click right click and scroll as normal Now comes the fun part you can use gestures with Mauz too With the iPhone camera on move your hands to the left to bring you a page back on your browser and move it right to bring yourself a page forward If there s an incoming call or a text message simply intercept it and resume using Mauz right after Unfortunately Mauz is not out for consumers to buy just yet PointGrab PointGrab s22 is something similar to eyeSight in that it enables
170. s of the skeleton 137 4 S COOL s Iterative Development FURTHER DEVELOPMENT not soon 1 STORY a static image avatar lt left right gt skeleton shapes shadow from green screen green screen b BETTER GESTURE RECONITION using framework FAAST c black shadow done with green screen input 2 GAME a A way of having crossing gestures b 1 player or 2 player selection box 3 FREE FORM a a sort of closet where teacher can choose what to add skirts pants select images online and choose the joints and the tool will autonomously move the images chosen b 1 player choose to be Mr Bean and the other one iron man 4 NEW STORIES a Anticipate the school with new stories taken from their books b Consider doing a simple authoring tool 138 4 S COOL s Iterative Development 4 2 2 Design and Implementation In this chapter design and implementation have been merged under one single section because reiteration of these phases is not needed The problems noticed by teachers were mostly easy to solve that s why in this section we focus on explaining the concept of S COOL first screen and instructions sheet for teachers First screen As soon as the teacher clicks on START she will see this screen in which she will be able to select what kind of activity GAME STORY FREE FORM and which variables to set CHOOSE THE DURATION CHOOSE THE AVATAR a CHOOSE THE AVATAR 1 VIDEO HINTS
171. s the skeleton shadow and other showed the person live on the screen green screen The game was to give the kid a choice of being in the Jungle Mars fairyland farm room sea Halloween or Christmas and to try to catch objects that randomly appeared on screen when their avatar touched it with their hands Each theme had a popular soundtrack playing in the background We ji T I marked an area with tape where the player had to stand As the players consented to play we explained the rules to them asked them to pick a background and ran the game for one minute on green screen and 2 minutes on shadow Sometimes a parent played on the shadow and the kid played on green screen or two siblings played side by side so it was engaging Some kids made a choice of the background where they would like to be for some we asked the parents what the kid likes while some parents were like wherever it doesn t matter May be they just wanted to see if their child does something It was a mixed experience People who tried our game included parents kids with autism their neuro typical siblings volunteers and we also had a couple of enthusiastic elderly We never asked explicitly if the kid had autism but let everybody play at will When we tried talking to some parents to get them to try it they said he has autism and did not think the game was meant for their kid There were people who did not have the time to Most people enj
172. s to the console using Bluetooth with an approximate 30 ft 9 1 m range and features rumble and an internal speaker The Wii Remote can connect to expansion devices through a proprietary port at the base of the controller The device bundled with the Wii retail package is the Nunchuk unit which features an accelerometer and a traditional analog stick with two trigger buttons In addition an attachable wrist strap can be used to prevent the player from unintentionally dropping or throwing the Wii Remote Nintendo has since offered a stronger strap and the Wii Remote Jacket to provide extra grip and protection The Wii MotionPlus is another accessory that connects to the Wii Remote to supplement the accelerometer and sensor bar capabilities enabling actions to appear on the screen in real time Further augmenting the remote s capabilities is the Wii Vitality Sensor a fingertip pulse oximeter sensor that connects through the Wii Remote 27 2 State of the Art 2 3 3 Other Touchless Technologies Intel Perceptual Computing Perceptual Computing PerC s15 is an organization within Intel Corporation tasked with the research development and productization of technologies for Natural User Interaction The objective of PerC is to explore consumer focused hardware and software applications of close range hand and finger gestures speech recognition face recognition and tracking and augmented reality The organization is funded by a
173. sed levels of prosocial behaviours and they work well for children aged between 2 and 12 years Treatment and Education of Autistic and Communication related handicapped Children TEACHH The method supports task completion by providing explicit instruction and visual supports in a purposefully structured environment planned to meet the unique task needs of the student TEACHH works well during the school day up to 25 hours per week and it demonstrated to increase levels of e Imitation e Perception e Gross motor skills e Hand eye coordination e Cognitive performance 17 2 State of the Art If you want your children to be intelligent read them fairy tales If you want them to be more intelligent read them more fairy tales Albert Einstein 2 2 Story Telling and Children s8 b18 b19 b20 b24 By incorporating digital storytelling projects into learning you can reach today s students and at the same time help them to develop the skills they need to be successful in our complex technology rich world Digital storytelling learning projects may not be a cure all for reluctant learners bored students students who have trouble retaining information or those who are chronically late but the experience of students and teachers in classrooms around the world confirms that this approach to learning is an exciting and compelling way to engage students in the learning process and to inspire them to become lifel
174. sical e Creative e Communication language skills e Game s lack of structure and boundaries e Coordination and body awareness e Playful learning e Highly engaging e Shared attention e Common goal e Safe interaction and predictable environment In one word Kinaesthetic a learning style in which learning takes place by the student carrying out a physical activity rather than listening to a lecture or watching a demonstration 26 2 State of the Art 2 3 2 Nintendo Wii 814 The Wii is a home video game console released by Nintendo on November 19 2006 As a seventh generation console the Wii competes with Microsoft s Xbox 360 and Sony s PlayStation 3 Nintendo states that its console targets a broader demographic than that of the two others As of the first quarter of 2012 the Wii leads the generation over PlayStation 3 and Xbox 360 in worldwide sales in December 2009 the console broke the sales record for a single month in the United States The Wii introduced the Wii Remote controller which can be used as a handheld pointing device and which detects movement in three dimensions The Wii Remote is the primary controller for the console It uses a combination of built in accelerometers and infrared detection to sense its position in 3D space when pointed at the LEDs in the Sensor Bar This design allows users to control the game with physical gestures as well as button presses Ail The controller connect
175. sis Discussion with Gregory and Agata Tentative guidelines for creating games 1 2A No time constraints allows the kid to do the activity taking their time Involve constructive social interactions Hi 5 get the object together shake hands Involve motor movements with wider range of actions jump up to get the object more body parts use of feet hands both or either variety use left hand for one action and right for the other both hand movements use both hands to get two objects together Complex cognition Cross body cross color interaction left hand reach for object on the right Not too complicated or fancy i e different requirements than neurotypical kids Sense of reward with interactive objects on the screen scoring encouragement Visibility and accessibility Bigger objects not too cluttered Keep it simple No complex Level advancement Tentative examples of other games that can be made 156 v Be Se AS Se S Tennis volley ball game Falling objects Running away from the object Collaborative develop plans simply socialize Shape Game OT Motion games Dance live Free form Google Images YouTube Jumping Rope game interacting with real objects in conjunction to virtual objects on the screen Google effects game Virtual Costumes 5 Final Analysis A new story After some days teachers of the Lionheart called us to fix another appointment where we could ha
176. some time and played for another hour till his father came to get him The kid liked the shadow avatar more because he identified that it is faster without glitches and he can score more than the green screen He had some amazing comments and feedback for us while he was playing To quote a few I wish I could earn more seconds This is an exercise game That is so appropriate on LIVE the game written on the green screen gaming area Judging from his performance intelligence and active commenting we figured he doesn t have autism He also said that my parents have come here with my 110 4 S COOL s Iterative Development brother He was also trying to help Aidan when he wasn t responding Aidan was happy to meet us but he did not respond to the Kinect or to seeing himself on the screen He stood inside the box when Gregory or his brother told him to but he would not do any actions or listen to us He would keep going back to Gregory whenever he tried Later in the day Gregory s brother got Aidan to try it again but his response was the same It was like we were forcing him to do something he doesn t like at all So we stopped Again the thought reinforced in my mind like at Lionheart Technology can never do what family can A young child of 2 4 years of age stood in front of the Kinect and did not react when he saw himself on the screen Parents say he just likes watching
177. soon as one of the kids saw himself on the screen he jumped in delight immediately turned back and hugged his teacher O This is the kid that loves fans usually lies down on the desk and kept coming to Agata and her camera on our last visit e One of the kids was pretty tech savvy and started using the software on his own This is the boy who loves jumping on the trampoline His favorite was the Live Image I only had to tell him once how to select it I wanted to help him the next time but backed off when he said I can do it myself The teacher supported saying he knows all the computer stuff 134 4 S COOL s Iterative Development He picked one background at a time from the Freeform options and tried them all We wanted to show them all the other options but he just wouldn t let go of it Finally the teacher got him out of there He jumped on his trampoline only this once in the whole time that we were there We put the jungle game with the soundtrack Lion sleeps tonight to which he started yelling turn the sound off We obeyed We put the Christmas game to which he had the same reaction One behavior to note here is this boy was always facing the computer Now the Kinect and the promethean are placed at the same place but the computer was in a different direction so he got to see only his side profile and he wouldn t look at the camera He also turned off the monitor at times Next we s
178. st of all the modules please take a look in chapter 7 2 67 4 S COOL s Iterative Development 68 4 S COOL s Iterative Development S COOL Alpha This is the first iteration of the project It starts gathering as input the requirements of Kinect the Dots powered by Andrew Harbor in collaboration with the Lionheart School After the requirements phase in which I had to affine those most relevant I began organizing the whole work chunking tasks in small modules and implementing assigning them starting from the most critically fundamental In one month whole September we needed to be ready to show a working version to the School and after that restart from the requirements phase as described from the incremental model section 4 1 1 Andrew brainstormed and compiled a list of preliminary project ideas and areas Requirements for exploration Then he conducted semi structured interviews with teachers and occupational therapists at the Lionheart School followed by observations of existing classroom activities centered on storytelling Based on that I summarized his report highlighting strengths and weaknesses organizing and categorizing feedbacks and possible solutions and finally finding new directions of this project Below you can find a table describing problems and solutions from most critical to less relevant hierarchy provided by teachers I will explain the found solutions in a more compl
179. sto modo sarebbe conservata la gratuit dell ascolto mie considerazioni estemporanee Erminia propone a Mirko di fargli avere degli elementi su queste attivit ad esempio il libro spezzoni di video dove c la lettura in gruppo con modeling e altro materiale sulla caa Mirko propone un raccordo a livello di sperimentazione quando il prototipo sar disponibile e nelle successive fasi di sviluppo e raffinamento mentre si trover ad Atlanta Dal punto di vista dei dispositivi necessari ci sono il Kinect attualmente nella versione 1 del costo dell ordine di 100 EUR sostituibile eventualmente da tablet nelle sue funzioni di videocamera e di interazione tattile Complessivamente l interazione stata molto positiva Mirko ha un atteggiamento di ascolto molto buono e di collaborazione molto creativo Disponibilit ottima Pronto a testare le applicazioni 61 3 Preliminary Requirements Ospedale San Paolo Milano 11 Cristina Bellosio E uno strumento informatico applicabile a qualsiasi tipo di pc Loro seguono sia bambini con buon livello cognitivo quindi integrati sia altri con limiti e disabilit maggiori Non scontato mettersi nei panni del personaggio a video I bimbi autistici sono molto attratti dagli strumenti informatici il rischio che ci sia un assorbimento che non vada a favore nel sviluppare nei bambini con autismo la relazione Se lo strumento utilizzabile da pi bambini allora meglio Loro per
180. t kids are actually acting out the story and the setting is equipped with puppets it s all about involve children in visually auditorally motorically and have them exposed to the story from a lot of different ways 2 Teacher is in a class how does she know who is doing better Is there a sort of turn taking activity First of all there are at least 2 teachers per class and classes are small up to 8 children Each class group children of almost the same need and support individual profiles and academic goals Teachers don t do a lot of turn taking 103 4 S COOL s Iterative Development because it is not a natural waiting process and kids have to wait their turn but in a lot of classes they try to do something in which everybody is involved But in some games there might be a child who is on the spot actually making a move or there might be everybody participating in the room They are all involved because if a kid has wait that is the time when he disconnects from the story and the class Agata adds that this is something that we have to think about our group is interested not just in the sort of camera getting gestures but all the ways we can potentially interact with the story that could be something you say something you do at the board We have focused of detecting the body gestures but to the extent that it might not make sense because the Kinect can really only be driven by one child at a time Maybe thinking of ot
181. t could fit my intentions and did not deviate from her knowledge fields After some universities proposals we agreed that the Georgia Tech project was the one that better tailored mine and her needs Gregory Abowd p2 the distinguished professor in the School of Interactive Computing in the person of Agata Rozga p3 a developmental Psychologist with a research focus on conditions on the autism spectrum proposed us an improvement of a project called Kinect the Dots Al 2 State of the Art 2 5 1 Kinect the Dots Kinect the Dots b28 was an application designed by Andrew Harbor GTech to help teachers therapists and parents engage children in an interactive story experience A specific story is used for several months at a time during which time the children develop a familiarity and emotional connection to the story Activities are structured around the story to promote a variety of developmental skills For example during one activity a teacher reads the story as the children act it out repeating what the characters say and performing their actions The resulting system was designed to augment the teachers use of storytelling in the classroom At the most basic level the prototype acts as a digital storybook projected on a large screen in the classroom The teacher narrates the story and manually advances through a series of images on the screen analogous to turning the pages of the book The first stories Andrew d
182. ta non sia adeguata per loro se non per quelle situazioni in cui il focus sull apprendimento di particolari schemi motori imparare a sciare ad esempio 60 3 Preliminary Requirements La comunicazione aumentativa e alternativa Sembrano essere molto immediati i collegamenti con alcuni strumenti di caa come il passaporto le contingency map le strisce delle attivita soprattutto in vista di un attivit potenzialmente carica di emozione e di paura la risonanza magnetica la visita del dentista Daniela in particolare sottolinea questa possibilit e sembra che possa scrivere con una certa competenza delle brevi storie per questo scopo Rispetto alla lettura con modeling ci si detto che questo strumento potrebbe essere parte di una modalit pi esplorativa e gratuita dell utilizzo dell ambiente Le storie in forma di filmato gi modellizzato potrebbero essere rese disponibili esplorando l ambiente in particolari luoghi evidenti o da scoprire mettendo magari a disposizione delle possibilit di interazione analoghe a quelle che mettiamo nelle tabelle a tema per la lettura qui mappate da opportune gesture Non mi sembra invece proponibile un interazione in cui il modeling sia guidato dal bambino utilizzando l avatar a cui si prestano i gesti Ma forse potrebbe valere per un lettore accanto al bambino il suo avatar fa il modeling nel mondo dell applicazione guidato dal modeling nel mondo reale da parte dell adulto in que
183. th a total of 120 session and 91 valid The validity of each session follows the rules Lines of logs must be more than 10 Free Form and Game must be played at least for 10 seconds Story must be played at least 2 minutes and must go over 2 pages 91 valid sessions means also around 30 sessions per class and along 11 days an average of around 3 plays for each day For the following analysis we have considered to calculate usage and technical statistics starting from a broad view of the three activities and then focusing on each activity More details will be given respectively to each graph 158 5 Final Analysis These tables below describe the total number of times of play per class and the total average time of play per class As you can see Alysha s class played just few times of each activity while the other two classes focused on a precise one Ashley s class stand out for the game activities while Jill s class for the story activities Free form has been played few times in all the classes but in Jill s class when they played they did for an average of 7minutes and a half In Ashley s class preferred to play the game for more than 35 times but just for an average of 2 minutes and a half while in Jill s class they played around 10 times less than Ashley but with an average time of around 4 minutes Total times of play per class 40 35 30 25 m ALYSHA 20 E ASHLEY E JILL Free Form
184. the activities have a common structure and can be played with one or two players student tutor Different visual and operational aspects of the activities can also be customized Pictograms are one of the most commonly used alternative communication systems and we know that most people with autism can learn to use them correctly They can use them both to understand the world around them when they are used to structure their physical space or time via panels or an activity diary and also for communication with others pointing at them in order to ask for things AO 2 State of the Art The universe is made of stories not of atoms Muriel Rukeyser 2 5 Previous experiences Everything started at the beginning of 2013 when I decided to attend the university course HCI provided by Politecnico di Milano under the schooling of professor Franca Garzotto p1 Within that course I had to develop a game for motor impaired children to enhance their skills A deeper view will be given the next section After a good success of Pixel Balance I decided to follow this HCI path asking Franca to extend this work into a thesis During this period I developed Kintroller a self made app with which was possible to discover the whole word through Google Street View and control the system via body motion with Kinect In the early march 2013 my professor contacted different American universities to find a good thesis topic tha
185. the application is better rather than requiring the teacher to tell the story or drive the action Using a narration or music track or having words on the screen with progression either automatic or driven by children s actions interactions would be best 11 Lack of gestures gt authoring tool to setup gestures 71 4 S COOL s Iterative Development Furthermore the story creation tool should be able to allow teachers to import scanned pictures from existing paper based stories as well as multimedia material e g icons images animations and sounds from the internet to select which Kinect based actions they want the story to use and to associate the actions with specific animations in the story As suggested from the teachers of the school in the Kinect the Dots report they wanted to explore this idea further in the future The previous system especially after implementing some of the proposed changes outlined above and conducting further user testing served as a demonstration of the proposed output the story creation tool would be able to produce With the Kinect the Dots demonstration implemented and high levels of enthusiasm on the part of the teachers for more stories of this type continuing the project in the direction of a story creation tool seems more appropriate at this time than before The image above shows the first raw design of the authoring tool user interface 72 4 S COOL s Iterative Developm
186. the first years of life and how the emergence of these skills is disrupted in autism Her current research bridges the fields of psychology and computer science with an aim toward building new tools to measure the full iS range of behaviors relevant to autism p4 Arpita Bhattacharya is a Master Student in Computer Science with a particular focus on Human Computer Interaction with skills in computer programming and application development for desktop web amp mobile devices 171 7 Appendix 7 2 Modules mA mB mC mD mE mF mG 172 Authoring Tool mA1 Story Creation Tool mA2 Real Time Stories Kinect Detection Software mB1 Gesture Detection mB2 Multiple Users Detection mB3 Levels Of Gestures Stories mC1 Story mC2 Avatar mC3 Tutorial mC4 Video Hints Teacher s Controller mD1 Remote Mouse mD2 Mobile Remote Controller mD3 One Button Remote Controller Interaction mE1 Tablet mE2 Speech Recognition mE3 NFC mE4 Arduino mE5 Promethean Board Purpose Games mF1 Catch The Objects mF la Single Player mF 1b Multi Player mF2 Free Form mF3 Shape Game Evaluation mG1 Log Manager mG2 Video Record mG3 Survey amp Meetings mG4 Automatic Measurements 7 Appendix 7 3 User manual The following are common instructions position and plug the Kinect sensor set up the play space taking care of lights install th
187. the story They expressed their interests in this from the beginning and they asked if it was possible to insert this avatar mode inside a story mC3 TUTORIAL An intro tutorial can be the best thing when children do not know what to do and can be developed for children that are approaching to Kinect for the first time In this school some children are already familiar with motionless interaction and few are conscious of what are the gesture to perform they used to play stories with books That s why we decided to focus on a sort of tutorial that can be placed along the execution of the story video hints C 4 117 4 S COOL s Iterative Development mC4 VIDEO HINTS As said in the previous module we found video hints as the best way to show kids which gesture to perform during the story A video hints will be shown just when the system will be waiting for a gesture to perform Some classes don t like this feature because teachers do want children to think about what gestures coming next while some other classes want this feature as an assistance For this reason teachers will be able at the beginning of the story to select whether to hide or not video hints 601 mp4 602 mp4 603 mp4 604 mp4 621 mp4 601 Walk walking gesture with knees high 602 Wiggle movement of the body as a twist 603 Shake shake the shirt near the hip 604 Nod nod move the head forward and backward 605 Clap clap the hands 118 4 S
188. tion which can enhance the communication speed thus reduce the delay between teacher s tap and reaction of the system mD3 ONE BUTTON REMOTE CONTROLLER We understood that most of the time teachers don t really need all of the features but just one single button Even if the teacher moves around the room as usually does he she can easily push that button to proceed A manual low technology fast wearable necklace button is the best solution The one button remote is the easiest way to give a well working system in a good time frame without frenetically work on gesture recognition which is a more delicate aspect 120 4 S COOL s Iterative Development mE OTHER INTERACTIONS As we understood from the Lionheart meeting they really want a story with which it is possible to interact in many different ways Any sort of idea introduces new ways of interaction which need some time to develop Since every single iteration lasts just around 20 days there is no time to develop everything by the deadline but there is time to explore some ways and prototype only some mE1 TABLET The idea is to have one child in front of the Kinect and other children interact within the system using tablets They will be able to add elements inside the story and understand they can do as much as the active player This could be a nice idea but it needs some time to be explored For example do we need children involved all together in
189. tr dedicare questa ultima pagina delle mia lunga carriera universitaria S spero 43556 Ringrazio di nuovo tutte le persone che mi sono state vicine pur lontane fisicamente Ringrazio chi mi ha supportato allo stesso modo di chi mi ha ingannato perch da quell errore ho imparato Per stare in tema con la formalit della tesi dovrei referenziare i miei ringraziamenti cos pag V 43608 Sar pure considerato un folle ma io in questi bambini ci vedo qualcosa vero non sono uno psicologo ne uno studioso di autismo nelle sue profonde radici ma voi quando dite che una torta vi sembra buona senza averla mangiata avete bisogno di un cuoco per esserne sicuri Io credo di no 43660 Ci vedo qualcosa perch se una terapeuta vi dice che quel bambino non ha mai saltato e poi le dimostrate tramite un video che lui salta utilizzando S COOL e anche tanto come si fa a dire il contrario Non si pu 43702 C magia qualcuno ha sussurrato e qualcun altro d accordo con lui ha affermato Si c ma come facciamo a misurare la magia Forse non si deve per forza sempre misurare cos come non dovrei pi contare queste parole Oppure no 43742 Fine 43744 183
190. tting the other kids watch at the back of the class or did you let them play all together 141 4 S COOL s Iterative Development gt What did the kids do if their gesture wasn t recognized Did they get frustrated and stop or did they wait for it or did they talk to you gt What did the other kids in the background do when their peer was playing gt Were the children bored of using the system over time or did they look forward to it gt How long did each kid play How often did they play gt Did they talk about the system Compliment or Complaint While playing NOT playing gt Time gt How did you fit it in your schedule during the day or week Which activity did you replace or modify What time of the day did you use it gt How did you decide when the kids do the story the game and the freeform e Effort gt Were you willing excited to use the system Or was it an obligation gt How much effort additional help in shifting furniture was needed in clearing the stage area Was the effort worth using the system SPECIFIC TECHNOLOGY RELATED FEEDBACK gt How many times did you have to adjust the Kinect angle Did you face difficulty in changing the setting every time gt Did you have to refer to the Kinect s skeleton screen after starting the story or game gt Was the remote controller enough Or do you prefer additional controls gt What problems did you face in understanding what t
191. ual objects to reality I would imagine a story where there is so much wind and at the same time through a complex link a switcher will turn on a fan Or let s imagine a child who needs to touch the sun virtual object and as soon as he touched it a lamp switches on These are just concepts that will be extended if they will be taken into consideration as further development 80 4 S COOL s Iterative Development On top of these solutions I decided to add my own annotations that will be taken into consideration as features to add to modules when they will be developed mA STORY CREATION TOOL SHARED LIBRARY The idea is to create a shared library on the cloud which would enable to upload graphics sounds and organize them by tags and categories Imaging to store not only teachers material but also the entire stories e g of different schools and gestures algorithms could increase the possibility of rapid development of the entire toolkit mC STORY MONITORING SYSTEM What about developing a system to monitor children s results and store them for future analysis This is just an idea that could be deeply considered if the school would like to monitor improvements of its children mC STORY OFFLINE FRUITION During my visit to some Italian schools and clinics I noticed they don t have an internet connection or they have it slow hence the game should be usable offline mC STORY COLLISION D
192. ucture of the story format num_ pages description story id title author name creator uid pages media objects action code z index page on complete color text action trigger code height width url object id ny y font_ size type image sequence item page_ id page number 96 4 S COOL s Iterative Development Recognition of gestures Stories are controlled by predefined actions in front the screen An action which can also be called gesture performed by a child means technically a complete set of procedures that catch the motion and give us the name of that particular movement To be able to recognize gestures it is first important to understand what makes a gesture We concluded that gestures were made up of parts Each part of a gesture is a specific movement that when combined with other gesture parts makes up the whole gesture For example the diagram below shows the two parts of a simple wave gesture and how they can be identified However this is not quite enough to be able to recognize multiple gestures with any degree of accuracy The problem occurs when you think about multiple gestures being recognized at once It s not as simple as looking for the next part of the gesture For example consider the wave gesture shown above If I was to drop my hand between the two parts then it would
193. users to navigate on their computer just by pointing at it PointGrab comes in the form of a software and only needs a 2D webcam The camera will detect your hand movements and with that you can control your computer PointGrab works with computers that run on Windows 7 and 8 smartphones tablets and television Fujitsu Acer and Lenovo has already implemented this technology in their laptops and computers that run on Windows 8 The software comes with the specific laptops and computers and is not by itself available for purchase 31 2 State of the Art Myoelectric Armband Myoelectric armband or MYO armband s23 is a gadget that allows you to control your other bluetooth enabled devices using your finger or your hands How it works is that when put on the armband will detect movements in in your muscle and translate that into gestures that interact with your computer By moving your hands up down it will scroll the page you are browsing in By waving it it will slide through pictures in a photo album or switch between applications running in your system What would this be good for At the very least it will be very good for action games MYO armband is out for pre order at the price of 149 32 2 State of the Art For people without disabilities technology makes things easier for people with disabilities technology makes things possible Mary Pat Radabaugh Director IM National Support C
194. ut fully automating the measurement of children s response to name Measuring child response to name is an important cue for assessing child development and is both widely practiced by clinicians and technically feasible to automate in the wild Our goal is to create an automated system for predicting a child s response to his or her name being called I was interested in the technical perspective of measuring automatically the entire process because if developed inside my system it could elicit the therapist side of the entire project Gather information and record them from a camera is quite easy but extract meaningful information from it is a difficult task But before developing this therapeutic side I should better develop all the other side to have in the future a solid system For this reason me and John met to understand what were the common language to work on He will be able to use my front view Kinect camera and install other cameras around the class After having the recordings it could be difficult to synchronize them hence a log will be created giving information of times actions of children and reactions of the system 127 4 S COOL s Iterative Development 4 1 7 Implementation The requirements and design part of this second build cleared us the ideas of what to build and what to leave for this iteration Regarding to this the group met to clarify the overall implementation choices and
195. ve spoken about the creation of a new story and to what extent a step forward to the understanding of how stories are played teachers and children needs during the game After the meeting we were be able to create a first raw outline listed below of the story pages actions gestures to recognize and elements to insert sounds images and text First of all they chose the The Mitten story s49 for rhythmic repeating language associations repetition Playing the story children will vary the motions and make them relevant to the animal shown on the screen Teachers will simplify or amp up the story construction based on students classes A teacher will read fast and slow versions for speech Based on the framework we built during these months will be easy to write everything inside the common JSON structure described above without coding any single line The story is designed as follows 1 On a very cold day in a forest a little boy lost one of his mittens 2 A little mole found the mitten in the snow The mole opened it and crawled inside to get warm Squish squash squeeze s s s Sound effect and the students will rub their hands together 3 A rabbit opened the mitten and hopped inside to get warm s s s 4 A hedgehog opened the mitten and hurried ran motion for Kinect inside to get warm s s s 5 An owl opened the mitten and flew inside to get warm s s s 6 A badger opened the mitten and
196. video games could ever be called educational toys Kinect may have finally made a compelling argument in its favor Kinect is the latest in gaming technology although no one could have predicted the potential benefits for those young gamers dealing with Autism Kinect is an add on for Microsoft s Xbox and the game play is motion activated or sensitive Before Kinect came Wii but this gaming system is played using a single hand wireless controller With Kinect the entire game is powered by the player s motions For that reason it may be easy to understand how running in place jumping ducking and even swinging at things can help a child with their eye hand coordination as well as other skills Advancing through Avatars On screen the players appear as a cartoon like avatar but the movements they make are powered by the real life movements of the real life players If the player raises his arm his avatar does the same at the same time This process also helps kids grasp body awareness as well as coordination in general All of these things are helpful for any child to develop and improve For Autistic children however this type of learning can be otherwise close to impossible Although the intention of Microsoft gaming development was never to promise to be a device to help Autistic kids be able to learn new concepts no one is complaining Educators and even child therapists who specialize in working with these challenged c
197. vity for neurotypicals non ASD the ASD community has been largely left behind The general context of this research is the development of innovative interactive technologies that can be integrated with both general school classes or personalized therapeutic activities and can be autonomously used by teachers therapists to promote through engagement social interaction communication capabilities and motor skills Specifically this work will extend the existing prototype of an interactive story telling tool that exploits the paradigm of motion based interaction and the teachers could use in their classroom during story time with autistic children The initial prototype of the system consisted of a single story and a limited set of actions that were detected by the Kinect and resulted in animations on the screen The basic idea is to have the children s movements and gestures as detected by motion sensing technology e g the Kinect animate elements of a story being projected on a large screen e g a smartboard in the classroom The system is designed to run on a screen with the use of the Kinect a special device which combines an RGB camera an infrared projector and a microphone to perform gesture and voice command recognition 1 Introduction 1 2 Purpose The work exploits an innovative interaction paradigm motion based touchless interaction to support education and therapy of autistic children Our understanding of th
198. write them down on a whiteboard As you can see inside the PRIORITY column there are some initials which represent Mirko M and Arpita A This assignment of tasks has been given considering our availability time Mirko full time Arpita part time For a better understanding this image had been reported in the next page under the form of organized table showing 128 module code code of the module described in the design phase ex mA1 name of solution name of the solution outlined in the previous phases ex STORY CREATION TOOL time difficulty cost difficulty to build that module with proportional time cost to do or not choice whether to do or not to do in this iteration priority factor if module is on to do list represents the priority with which must be developed 4 S COOL s Iterative Development N NAME OF THE SOLUTION TIME COST TODO PRIORITY 1 3 1 n A 1 STORY CREATION TOOL 3 NO A 2 REAL TIME STORIES 3 NO B 1 GESTURE DETECTION 1 YES 2 change some gestures based on Lionheart B 2 MULTIPLE USERS DETECTION 2 YES 5 B 3 LEVELS OF GESTURES 3 NO C 1 STORY put avatar inside 2 YES 4 C 2 AVATAR 3 YES 4 C 3 TUTORIAL 1 NO C 4 VIDEO HINTS 2 YES 3 D 1 REMOTE MOUSE already implemented D 2 MOBILE REMOTE CONTROLLER only prototypized D 3 ONE BUTTON REMOTE 1 YES 1 CONTROLLER E 1 TABLET i NO E 2 SPEECH RECOGNITION NO y E 3 NFC R NO 8 E 4 ARDUINO NO E 5 PROMETHEAN BOARD 1 YES F la CATCH THE OBJECTS
199. ystem a sort of tutorial to take place before the story begins The tutorial could introduce the students to the idea that their physical actions cause a reaction in the system The tutorial could begin by simply showing the stick figure or a similar representation that mirrors the student s actions to introduce the basic concept The tutorial could proceed to explain the concept of specific actions or gestures as input and walk the student through this process As we thought the tutorial can show the figure in 3 different ways The first setting left is showing the figure to imitate blue on the top right corner This is a generic design to show that the figure can be smaller and positioned in a corner in the first case In the second case center the stick figure to match blue is behind the projection of child itself orange The child will simply need to cover the same figure behind his silhouette The last solution right shows that the screen is split into two figures one next to the other The child will need to look near its silhouette on the screen and try to follow the blue shape TT 4 S COOL s Iterative Development mD TEACHERS CONTROLLER There are different variables in this gesture detection field that could be taken into account to prove that further researches and subsequent valid improvements must be done The recognition of gestures is based on one or more powerful devices with d

1.3 Struc - POLITesi - Politecnico di Milano

Contents

Download Pdf Manuals

Related Search

Related Contents