Home

an interactive system to analyze dietry habits

Contents

1. 2 1 2 An e ee Pro A A ACSS M EM 4 3 MICROSOFT SOL SER VER eis ee ee een 2 54 8101 60 b u Or EN ee ee 4 371 1 Available daya ee ee A 31 2 nee ee qO A lir INTERFACE ee ner RER ER En 2 4 2 a i R A e 442 Adice graphic Inter aCe CODE BERIN D uu u COS een en ee u T 4 5 1 1 Calendar control en wav 5921 era 4 95 2 05DX eo 4 5 2 1 SHOW image clicked function ae na ee 452 2 deleteColumn FUNCTION iue cad 46 4 5 2 3 record food type clicked function 46 4 5 2 4 P OD LUD TING Onu y A A 46 235 9 u EU DUE 47 4 6 PROBLEMS AND SOLUTIONS stated 48 4 6 1 Location Of CATADASE ccccsesesescccsscnassoescccecennatevescossceeenascosssccssconateossccsonanseosssecenenanssonsccssesceenaee 48 BOF DV WNO IMAGES oirir EEEE E a 49 4 6 3 Too many images for screen size a 49 40 4 Error introducing PTT
2. y Ol re E 2 2 2 2 PCC ONS ON UE 2 2 2 3 PASSIVE infrared 5 rs PEES 2 22 DIE a MENU Eee ne ne ae ee en re EEE 2 225 Electret micro OIG so ii it 2220 Tempel st UI EBENSO a 2 2 2 7 edmunds opties lO iii 2 3 CAPTURE UPLOAD AND PROCESSING OF THE IMAGES 723 1 geo ino eei og n usun CMM 2 3 2 Capture aod A AI ra 2 3 LEVE NE Semen ON editada ps 2 2 3 2 Lanamark photo seleciona aaa 2 3303 Calculating event hovelV anreisen 2 3 2 CONCEPT GELOCHON DEION 3 1 DESCRIPTION OF SYSTEM ARCHITECTURE une dida SIT DIC 3 1 2 Second stage image processing 3 1 3 Third Stage Anterac EOD uns A DEVELOPMENT atracar 2 LINTRODUCTION cursillos 4 2 MICROSOFT VISUAL STUDIO 2005 esse ee a ne nenne A A2 A O
3. as 19797 7 Madrid Mes Marzo de 2010 El Ingeniero Jefe de Proyecto Fdo Carolina Camacho Garc a Ingeniero Superior de Telecomunicaci n PLIEGO DE CONDICIONES Este documento contiene las condiciones legales que guiar n la realizaci n en este proyecto de un SISTEMA INTERACTIVO PARA EL AN LISIS DE H BITOS ALIMENTICIOS En lo que sigue se supondr que el proyecto ha sido encargado por una empresa cliente a una empresa consultora con la finalidad de realizar dicho sistema Dicha empresa ha debido desarrollar una l nea de investigaci n con objeto de elaborar el proyecto Esta l nea de investigaci n junto con el posterior desarrollo de los programas est amparada por las condiciones particulares del siguiente pliego Supuesto que la utilizaci n industrial de los m todos recogidos en el presente proyecto ha sido decidida por parte de la empresa cliente o de otras la obra a realizar se regular por las siguientes Condiciones generales 1 La modalidad de contrataci n ser el concurso La adjudicaci n se har por tanto a la proposici n m s favorable sin atender exclusivamente al valor econ mico dependiendo de las mayores garant as ofrecidas La empresa que somete el proyecto a concurso se reserva el derecho a declararlo desierto 2 El montaje y mecanizaci n completa de los equipos que intervengan ser realizado totalmente por la empresa licitadora 3 En la oferta se har constar el p
4. 49 4 6 5 More than one picture of the same 50 4 6 6 Representation Of images the screen 50 O 51 S TESTS AND RESULTS nee nenne 52 SE CONCEPT DETECTION A Pe o een 52 SUA WEER escis 53 VEDI o E 54 DZ Ip Zi AAA 55 E10 15 2 E Aa Er o 57 y nsn 57 o ES 58 DE DVD 60 LUNA A E o PO PR OC O On nus EE EE 61 72 E OIL EORNM 62 5 2 8 1 Annotated quantities rias 62 5 2 8 2 Number of pictures taken from each meal sees enne nennen nennen nnns nnne nnns 62 5 2 8 Forgotten Meals 63 5 284 Meal annotated A oo eem 64 Tp E AI o A e oo A R 64 6 CONCLUSIONS AND FUTURE WORK nee 66 cr 66 MA JUN ie rc m 67 PREFERENCES 69 APPENDIX is COLOR HISTOGRAM AND MANHATTAN DISTANCE pps HOW NIUCILISACSEP ING sensitive vean C Ns e E IV TABLE OF FIGURES FIGURE 1 1 MICROSOFT SENSECA
5. 16 FIGURE 2 16 FALSE BOUNDARY LIGHT SENSOR DUE TO WEARER 18 FIGURE 2 17 FALSE BOUNDARY ACCELEROMETER DUE TO WEARER MOVEMENT 18 FIGURE 2 18 EXAMPLES OF POOR QUALITY SENSECAM IMAGES 19 FIGURE 2 19 NOISE COMPOSITION 2 ars u 20 FIGURE 2 20 MIDDLE IMAGE OF THE NINE FIRST EVENTS IN THE WEARER S DAY 21 FIGURE 2 21 MANHATTAN DISTANCE BETWEEN EVENT O AND 8 OF FIGURE 2 20 WITH THE FIRST 12 EVENTS USING COLORHISTOGRAM ista 21 FIGURE 2 22 DETERMINING EVENT IMPORTANCE 22 22 FIGURE 2 23 ENTIRE PROCESS OF EVENT SEGMENTATION LANDMARK DETECTION AND NOVELTY CALCULA TON FE 23 FIGURE 2 24 IMAGE FROM FLICKR AND THE TAGS TO SEARCH THE PHOTO 24 FIGURE 2 25 DIFFERENT IMAGES SAME CONCEPT nee le 24 FIGURE 2 26 PATTERN CLASSIFICATION BY AN SY Mae ea 25 FIGURE FIRST STAGE iia AAA 27 FIGURE 5 2 SECOND STAGE rios 28 FIG RE 3 3 FIELDS OF IMAGE INFORMATION aran dica 29 FIGURE 3 4 EXTRA FIELDS ADDED FOR OUR SPECIAL 29 FIGURE 3 5 THIRD INTERACTION WITH THE USER sie a 30 FIGURE
6. 12 We can also segment photos using the information that the different sensors provide us This can be called context based analysis The values of sensors can indicate the probability that the wearer may have changed activities For example a change in temperature or in the light sensors can indicate that the user has gone from indoors to outdoors or specific values in the infrared sensors can indicate that a person has crossed in front of the wearer The optimal way to separate the events of a day will be by fusing the output of the content based image analysis with the output corresponding to the context based sensor analysis Therefore to proceed with the segmentation of events we will have to focus on the information retrieved with our SenseCam the results obtained after some image processing and the best way to calculate the similarity between images the metric that will provide optimal results and the calculation of appropriate threshold To segment all the images taken in a day in different events we have to follow some steps 1 Revising the time stamps if found that the camera has been disconnected more than two hours it will be automatically a boundary of different events It is usually registered as a day change as the wearer would have disconnected the camera to go to sleep 2 Compare adjacent images against each other or blocks of images to see how dissimilar they are The TextTiling approach 9 was found to perf
7. 136 206 19 149 all users 00024491 JPG 06 06 2006 6 30 24 5 29 293373 2 3873 http 136 206 19 149 all users 00024492 JPG 06 06 2006 6 30 32 23 32 293374 2 3878 http 136 206 19 149 all users 00024493 JPG 06 06 2006 6 30 40 14 31 293376 2 3878 http 136 206 19 149 all users 00024495 JPG 06 06 2006 6 31 00 14 31 293378 2 3878 http 136 206 19 148 all users 00024497 JPG 06 06 2006 6 31 12 30 26 Figure 3 3 Fields of image information For the correct operation of our software we have added some fields to the database related to the meals taken and the type of food ingested e Type of meal Breakfast lunch afternoon snack dinner or extra meal e How much food is taken from each of the six groups that form the food pyramid o Sweets and oil Milk Meat fish and eggs Vegetables Fruit Bread and cereals O O O O O meal_type Lunch meat vegetables fruit 0 0 0 0 0 0 Figure 3 4 Extra fields added for our special purpose bre Once everything is stored the user can start interacting with the graphical interface Pictures about his diet will be shown and data about the type of meal and the quantities taken will be introduced 992 It is time now of receiving some advises based in the particular dietry habits that the user has 3 1 3 Third stage Interaction In this part of the process the time of the real purpose of the study has arrived Here the user will interact with the sof
8. SET ANSI NULLS ON GO SET QUOTED IDENTIFIER ON Author lt Author Name gt Create date lt Create Date gt Description lt Description gt PROCEDURE dbo Carolina Get Eating Images Add the parameters for the stored procedure here USER ID AS INT GDATE TIME AS DATETIME EATING PROBABILITY AS DECIMAL 6 4 AS BEGIN SET NOCOUNT ON added to prevent extra result sets from interfering with SELECT statements SET NOCOUNT ON Insert statements for procedure here select Image Fey image path meal type sweets milk meat vegetables fruit bread from Aiden Images where user id USER ID and datepart year date time datepart year TIME and datepart dayofyear date time datepart dayofyear TIME and eating gt BEATING PROBABILITY END Figure 4 4 Second stored procedure Eating images The parameters needed to make this stored procedure work are the user identification the date and the eating probability For a given day we will select the images with higher probability of eating occurrence than the probability introduced as a parameter In this project we will use a probability of 0 4 This way we will show in the screen all those images where the probability of eating occurrence is higher than 0 4 organized by the time the SenseCam took the picture If we analyze the text in figure 4 4 we can see that besides image path more fields are selected These
9. user For example next to the quantity slide we can place an icon representing the type of food selected that gets bigger and bigger depending on the servings annotated with the slide Another interesting option could be to make the pyramid change each time a food group is selected this way we can avoid repeating meals already annotated Another function that we can add to our software is a calorie counter Depending on the quantities and the type of food annotated the software would be able to inform 567 about the calories taken in each meal Besides it would be a great idea to use the accelerometer included in the SenseCam to approximately calculate the number of quantities that the user burns due to exercise One of the new options of our software could be to introduce different diet profiles This way we could use a different profile depending on our sex age or weight Besides this could be very helpful if there were more than one member of the family using our SenseCam method Besides we can relate the number of pictures taken in each meal with the quantity of food that we ate in that meal The bigger the amount of food the longest time we need to finish it and the bigger the amount of pictures taken It would be also interesting to create graphics that could illustrate our evolution and the improvement of our diet based on our records s history Trabajo futuro Actualmente las SenseCams son unos dispositivos no demasi
10. we will have to employ content based methods to access digitally stored images quick and without problems A disadvantage of these content based methods is that the computational cost is very high and it will not be reasonable to make the computation each time we want to search for a photo so we can automatically create textual labels based on the image content and do the search from them 23 das boat Gy ALL Shes Bl Uploaded on Apri 21 2006 i A 2 4 Y by Pablo Municto This photo also belongs to T Lal Tags Comments Sa jjorteguilla says Mola Mucho Posted 36 months ago permalink coast Figure 2 24 Image from Flickr and the Tags to search the photo To carry out with success many of SenseCams applications this concept detection is basic We have to take into account that the images will have poor resolution they will be distorted due to the fisheye lens and some images will be more dark than desired because of the lack of flash As we can see concept detection is not an easy process Besides the problems due to the characteristics of SenseCams we will have to deal with other problems due to the following reasons e number of positive samples is very small compared to the negative ones e Many objects are difficult to detect because two same types of objects can look very different to each other due to their position illumination appearance 25 e Some concepts are highly abstrac
11. Cheese Group Meat Poultry Fh Dry 2 3 Servings Beans Eggs amp Nuts Grow 2 3 Servings You should not eat that much sweets fats and oils Use them sparingly Vegetable Group 3 5 Servi N Milk is good for our bones Don t forget it T Figure 4 11 Changes saved 39 4 4 2 Advice graphic interface In this third web form we present the user an easy way to look for some advices based in what nourishments he took in any day he choose The way it works is very easy for the user he just has to select a day in the calendar and the system will show the eating images of that day and the type of food that he was taking in each one This information has to be previously introduced by him Thereupon a collection of advices appear on the screen On top of the screen we can read overall advices of that particular day especially those things that the user has to correct in the future See figure 4 7 If the user took too much sweet during the day or didn t take enough fruit the application will advise him and will give some reasons of why it is important to take certain nourishment At the bottom of the page we can find more advices based on the analysis of each particular meal Breakfast Lunch Afternoon Snack Dinner and Extra Meal See figure 4 8 In the Advise web page we can find a link that guides us to the principal page In case the user notices that he left a picture without information he can easily go
12. City University 16 Mark Young Tom Radcliffe Paul St john Mark Chatterley Improved outcomes software www improvedoutcomes com 17 Mark Montague Javed A Aslam Relevance Score Normalization for Metasearch Department of Computer Science Dartmouth College 18 The AceMedia project http www acemedia org April 2007 19 Aiden R Doherty Alan F Smeaton1 Keansub Lee amp Daniel P W Ellis Multimodal Segmentation of Lifelog Data Centre for Digital Video Proessing amp Adaptive Information Cluster Dublin City University Ireland LabROSA Columbia University New York USA 20 Aiden R Doherty Daragh Byrne Alan F Smeaton Gareth J F Jones and Mark Hughes Investigating Keyframe Selection Methods in the Novel Domain of Passively Captured Visual Lifelogs Centre for Digital Video Processing and Adaptive Information Cluster Dublin City University Glasnevin Dublin 9 Ireland 21 SEAN MCHUGH Cambridge in Colour University of Cambridge United Kingdom 22 Aiden R Doherty and Alan F Smeaton Combining Face Detection and Novelty to Identify Important Events in a Visual Lifelog Centre for Digital Video Processing and Adaptive Information Cluster Dublin City University 23 Thomas Deselaers1 and Allan Hanbury The Visual Concept Detection Task in ImageCLEF 2008 Aachen University Computer Science Department Aachen Germany Inst of Computer Aided Automation Vienna Univ of Technology Austria 24 Juri
13. PICTURE OF EACH EVENT WITH THE CENTRE PICTURE OF Me RES Sorrir I FIGURE A 3 TWO MOMENTS OF OUR MEETING WITH THE NUTRITIONIST IV FIGURE 4 BECAUSE ONE SIZE DOESN T FIT ALL a IV TABLE OF CHARTS TABLE 2 1 SENSECAM HARDWARE SPECIFICATION 2 TABLE 2 2 VECTOR DISTANCE METHODS RESULTS ss a a a 15 TABLE 2 3 NORWIALIZATION ALGORITHMS een een 16 TABLE 2 2 FUSION ALGORITHM S entre 17 TABLE 2 5 PRECISION OF SEGMENTATION WITH DIFFERENT DATA SOURCES 17 TABLE 2 6 IDENTIFIED CHANGE LOCATION WITHIN LOCATION AND SOCIAL SEGMENTATION 18 TABLE 2 7 ACCURACY OF DETECTION FOR SOME CONCEPTS 26 TABLE 1 MANHATTAN DISTANCE OF THE CENTRAL IMAGE OF FIRST 12 EVENTS OF A DAY vil 1 Introduction Having good eating habits is very important and sometimes we are not very concerned about it We need a varied and well balanced diet because there is not only one nourishment able to provide all the nutrients we need A proper diet will increase our quality of life and will help us prevent illnesses However as a result of new styles of life and the development of new technologies in some areas related to diet
14. You can introduce some vegetables in your dinner Y ou should take more food from meat group to complete the 3 servings m one day Afternoon snack Remember afternoon snack has to be light Try with a piece of fruit vou are Figure 5 8 Detailed advices The advices that the software gives are a guideline to have a better diet but it s always necessary to have a doctor following our progress and checking the steps we take In the first part of the advice we have put together all the aspects that weren t correct in the whole diet of the day In the second part we show both correct and incorrect parts of our diet meal by meal This way we can know specifically the things that we are doing wrong One thing that we can introduce in the next version of this software is the creation of not only daily but weekly and monthly advices s65 6 Conclusions and future work 6 1 Conclusions As we can see in the previous section if we use SenseCams it is easier to remember what we took in each of our meals Having pictures of our meals help us to remember not only the type of food we had in a certain day but how much of each nourishment we took is proven that our SenseCam method is more effective and comfortable than the traditional diet notebook Using the notebook we found out that some annotations weren t precise 2696 of error that sometimes we forgot to write down our meal 3296 of the times and that we had difficulties t
15. are presented in the screen using a RadioButtonList created dynamically in the method Add_Checklist from Main aspx cs To select the food and the quantity taken we tried firstly using a javascript prompt However it wasn t very intuitive and the user had to type the quantities Instead of that we found that the best solution was to turn a food pyramid into an HTML image where each step was a button itself This way the user can press the step corresponding with a group of food and automatically a slider where the number of servings can be marked appears on the screen 44 A Array ELN CASEN dT i HTML_Command lt table MS For each image Y 4 k HTML Command button to delete imagepath event id image onClick data stored HTML Command lt table gt HTML Command mm Insert command in label l E I g ets Ln o S Result in the screen Figure 4 12 Presenting data in the screen Once the user has introduced all the data required all this new data has to be saved in the database in the same row as the image they belong to For that we will connect again with the database but this time with a different command one that allows us to update the table UPDATE table in database SET This way all new data will be store in the database and ready for the application to read it and use it to show and to create the
16. back to introduce it properly Choose a day in the calendar to get some advice lt junio de 2006 5 m E a o a o B In iw Nas 4 ha 109 I Io ls m Las p E Iz 4 IS Un IS 1 8 5 LJ gt ee I 9 a 5 N Im 9 Io Is t uo Back to main page Figure 4 6 Advice page before selecting a day in the calendar The background picture that appears in figures 4 5 and 4 6 is a fixed image that remains motionless even though we play with the page press a button or move the page to read the advices They are introduced using css language These particular lines of code appear in the files with the aspx extension In these pages we also have the Javascript code and the html code 40 Choose a day in the calendar to get some advice You should not eat that much sweets fats and oils Use them sparingly Milk is good for our bones forget it Proteins are important You have to take more meat fish or eggs Don t forget a bigger amount of fruit next time Don t forget bread cereals and biscuits next time they are good e E lun mar mi jue vie sab dom 29 30 31 1 2 3 4 5 6 8 2 1 15 16 1 IL 9 2 2 2 2 IN 26 0 2 39 1 2 3 4 5 6 1 8 9 Back to main page Figure 4 7 Top of the a
17. fields contain the information about the meal that is taking place in the picture breakfast lunch dinner and which are the nourishment that the 35 wearer is having These fields have to be filled by the user but this procedure will be detailed later on 4 4 Graphic interface The main goal when designing and developing this application was creating a graphic interface which was easy to interact with regardless of user s expertise Actually the graphic interface is the way the user has contact with all the advantages that the application has to offer so it has to be friendly and with all the information necessary in full view of the user Besides a usable and accessible design would help users interact with the application without having to read a user manual In the present project we needed to present several pages to the user at least one for the main page and another one for the page where the advices are shown For this we used two different web forms An additional web form was used to show a slider but we will explain this later 4 4 1 Principal interface We are going to consider as principal interface the first of the three web forms that we can find in this project When the user launches the application the calendar is initialized and the last day in which we have recorded images is marked See 4 3 1 Next to the calendar we present an image of a food pyramid In this pyramid the user can see the six d
18. porque es bonito saber que para alguien tus alegrias son alegr as al cuadrado y gracias a mi hermana Elena confidente y amiga que aunque de peque as quer as ser como yo ahora soy yo la que quiere ser como tu Tambi n quisiera dar las gracias a mi abuelo Melchor porque con l pis por primera vez la universidad con una hoja de matr cula en la mano y a mi abuela Toni porque nos quiere con locura y yo a vosotros m s Agradezco tambi n a Francisco C novas Vall s futuro presidente por su impagable ayuda en la realizaci n de este proyecto y por ser fuente inagotable de ideas Ideas tan felices como lo soy yo por estar a su lado Gracias por los minutos los segundos y los momentos No me olvido tampoco de todas esas personas con las que he compartido momentos en mi vida que me han enriquecido y acompa ado hasta el lugar donde hoy estoy Y por ltimo dedicar este trabajo a Eugenio Camacho Jim nez Y darle las gracias Gracias a l por todo Por todo siempre Carolina Camacho Garc a MARZO 2010 TABLE OF CONTENTS LINTRODUCTON AAA 1 2 MOTNATIONAND GOALS L es 1 2 DOCUMENT STRUCTURE u add as 2 STATE OF IME AR Di i i Di ern 2 1 THE IMPORTANCE OF LEARNING HOW TO EAT 8 2 E o Bd so ICU CONN Ag TT 2 2 SENSECAMS AND DATA COMPILATION ra eT WATE POG UC SS E 2 2 2
19. represent_image 2 See Problems and solutions number 4 6 6 int 43 Table in database NS d Stored procedure ASP NET application Hr e ee ReDPESET Made ig path medal Array sweets represent Image Fruit inf Use methods to extract information F ed Dread ger pain get SWEBIS get icd get path Figure 4 11 Save information in an array of data type represent image The images will be printed in the screen using a method called print images in Main aspx cs An HTML command is created to draw all the images as a table For each image we will extract its information kept in the array and include it in the HTML command After creating all the HTML command it will be represented on the screen placing it in a label The best thing to understand this process is to take a look at figure 4 12 To see more information of how the images are displayed on the screen see section 4 6 6 To make it easier to the user to analyze the image selected and introduce all data required the picture is shown in a bigger size Next to it we can see some options of meal type breakfast lunch afternoon snack dinner and extra meal The user has to manually select one It can be considered for future work to skip this step and automatically guess which meal of the day it is taken place revising the timestamp of the image and comparing it to the user s profile This different options
20. the pyramid Fats Olls amp Sweets Use Sparingly Shah 2 3 Send EEE Meat Poultry Fish Dry ngs Beans Eggs amp Nuts Group 2 3 Servings Fruit Group Vegetable Grow AS 2 4 Servings 3 5 ings Select the meal of day and what did you take Figure 4 8 introducing data When a food is pressed a Javascript function will open the second web form a slider With this slider we can introduce how many serves we took of the food selected See Appendix B Click to add your meal characteristics Fats Oils amp Sweets Use Sparingly lt 4 4 x Breakfast o Lunch Mil Yogur e Cheese Group ax Nay BA D nes N Beans Eggs amp Nuts Group Afternoon snack N 2 3 Servings Extra meal X Vegetable Group Fruit Group 3 5 Servings Servings 2 4 Add the quantity Win Bread Cereal Rice amp Pasta Group 6 11 Servings d 0 Get slider value Figure 4 9 Slider to introduce quantities 38 4 4 1 1 The slider The slider is made using the second web form This web form is opened with a Javascript function and it shares location with the first web form as we can see in figure 4 9 To make it we used the AJAX Control Toolkit This is an open source project built on top of the Microsoft ASP NET AJAX framework This toolkit contains a lot of controls that make possible to easily create new controls from existing
21. year date time datepart dayofyear date time order by min date time desc END Figure 4 3 First stored procedure Available days As we can see in the figure above if the identification number is introduced correctly the time stamp of the first image in each different day is selected This way we can know which days we have recorded data 4 3 1 2 Eating images This is the second stored procedure created When the user presses a date in the calendar the application must show him the images where he is taking any food When we implemented concept detection See 2 3 4 each image was scored with the 34 likelinood of a certain concept taking place This way we have in the database the probability that grass hands buildings eating and many others concepts are present in each image So to find which are the images where the user is eating we just have to read the likelihood of eating and choose those images where the probability is high enough Of course concept detection is not an exact science and it is possible that some of the images chosen as eating weren t eating images actually We have solved this problem allowing the user to manually delete these incorrect shown images Of course instead of deleting the image what the user is really doing in changing the eating probability to O USE Carolina GO Object StoredProcedure dbo Carolina Get Eating Images Script Date 07 27
22. 4 1 USE OF CSS IN THE PRESENT PROJECT coito a 32 FIGURE 4 2 APPECATON S STAR T ae ee 33 FIGURE 4 3 FIRST STORED PROCEDURE AVAILABLE DAYS inne ie a 34 FIGURE 4 4 SECOND STORED PROCEDURE EATING IMAGES 35 FIGURE 4 5 PRINCIPAL PAGE OF THE APPLICATION puyu l E 36 FIGURE 4 6 JAVASCRIPT POPUP AFTER CLICKING X FROM ERRONEOUS IMAGE 37 FIGURE 4 7 cEATING IMAGES OF A DAN ee ee 3 FIGURE 426 INTRODUCING DATA eek 38 FIGURE 4 S SLEIDER TO INTRODUCE QUANTITIES u lu in 38 FIGURE 4 10 SLIDE USING AJAX CONTROL TOOLKIT sia R L a dia 39 FIGURE 4 11 CHANGES SAVED etnias 39 FIGURE 4 6 ADVICE PAGE BEFORE SELECTING A DAY IN THE CALENDAR 40 FIGURE 4 7 TOP OF THE ADVICE PAGE WITH THE OVERALL ADVISES 41 FIGURE 4 8 BOTTOM OF THE PAGE WITH THE SPECIFIC ADVICES 41 FIGURE 4 9 CONNECTION WITH THE DATABASE TO EXECUTE FIRST STORED PROCEDURE 42 FIGURE IO CLASS REPRESENT IMAGE ze sn er t 43 FIGURE 4 11 SAVE INFORMATION IN AN ARRAY OF DATA TYPE 44 EIGURE 4 12 PRESENTING DATAN THE SCREEN aus tadas 45 FIGURE 4 13
23. 46 181548 153220 8 162484 159606 161384 254322 281452 173986 178390 175400 147106 213392 131100 9 168828 92100 133808 249052 308238 170400 190182 160946 147106 0 172304 151176 10 188752 173918 155922 247226 380936 165794 186190 181548 213392 172304 0 175012 11 115374 161300 126370 252278 349728 172006 171644 153220 131100 151176 175012 0 12 157324 129018 115220 261850 394216 136684 182456 141756 198664 128312 169660 169008 Table A 1 Manhattan distance of the central image of first 12 events of a day 12 157324 129018 115220 261850 394216 136084 182456 141756 198664 128312 169660 169008 0 These images are represented in figure 2 20 and analyzing the graphic we can see that those images that are more similar to events O and 8 have smaller Manhattan distance between their color histograms b How much is a serving To annotate the quantity of food taken in each meal we have to create guidelines of how much a serving is Of course servings will be different depending on which part of the pyramid the food belongs to To have a complete and varied diet we will have to complete all the serving amounts a day that the pyramid shows Fats Oils Sweets Use Sparingly Milk Yogurt amp Cheese Group Meat Poultry Fish Dry ii Beans Eges amp Nuts Group 2 3 Servings Vegetable Group Fruit Group 3 5 Servings 2 4 Servings Bread Ce real Rice amp Pasta In this application this is what we consid
24. ES the html code is kind of rewritten in real time and the image disappears from the screen At the same time its eating probability in the table of the database is changed to O 4 5 2 3 record food type clicked function This function is used to save all the food that the user clicks for an image in the html food pyramid Besides for each time called this function itself calls another Javascript function which will open a popup that includes the slider to annotate the servings taken for the food selected Once the user has terminated introducing new data it can be read from the principal aspx cs page 4 5 2 4 Popitup function This function is the smallest one and its only mission is to open the web form where the slider is placed function popitup url 4 newwindow window open url hola height 930 width 150 l1eft 600 top 600 newwindow focus Figure 4 14 popitup function 46 4 5 3 Advice creation In the graphic beside we can see the steps follow by the application to create the advices Finish Figure 4 15 Create advice flowchart 47 To show personalized advices is the most important goal of this project because it can be use for grown up to improve their eating habits and for children to educate them The advices are created in the Advice page In this page the user selects a day in the calendar Once a day is selected the application reads all the information that the user has introduc
25. NI sra 1 FIGURE 2 1 OBESITY IN SPAN u Qu 4 FIGURE 2 2 TEACH CHILDREN HOW TO EAT IS IMPORTANT FOR THEIR FUTURE 5 FIGURE 2 3 EXAMPLE OF SENSECAM IMAGES u ne anna 6 FIGURE 2 4 MICROSOFT SENSECAM WITH TRANSPARENT PLASTIC CASE 6 FIGURE 2 5 HOW PASSIVE INFRARED SENSOR WORKS 7 rr 8 FIGURE 2 6 FRONT AND BACK OF THE SENSECAM PCB 1 9 FIGURE 2 7 SENSOR TRIGGERED IMAGES VS TIMER TRIGGERED IMAGES 9 FIGURE 2 8 SEGMENTING IMAGES INTO 5 10 FIGURE 2 9 EXAMPLES OF FALSE POSITIVE BOUNDARY 111 11 FIGURE 2 10 THE ONLY FLOWER IMAGE CANNOT BE CONSIDERED AN EVENT 11 FIGURE 2 11 THE CLD EXTRACTION PROCESS STEPS 10 occcoccccncncccncnncocononaconononccnnnnacnonaronnonarononnnoss 12 FIGURE 2 12 CSD STRUCTURING ELEMENT 1312 l t uu seine el 12 FIGURE 2 13 COMPARISON OF BLOCKS OF IMAGES ALL IMAGES BELONG TO THE SAME EVENT 13 FIGURE 2 14 VISUAL FEATURES OF AN IMAGE REPRESENTED IN XML 14 FIGURE 2 15 MANHATTAN AND EUCLIDEAN DISTANCE BETWEEN TWO POINTS
26. SHOW IMAGE CLICKED FUNCTION Ga 46 FIGURE A 14 POPTPUPEUNCTION see a 46 FIGUREA 15 CREATE ADVICE EFLOWEHART s e eed kv ba eei Que d e 47 FIGURE 4 16 OLD VS NEW DATABASE CONNECTION STRING 48 FIGURE 4 17 SENSECAM IMAGES WITH SCROLL BAR nen 49 FIGURE 4 18 CHECKING ERRORS WHEN INTRODUCING DATA 50 FIGURE 4 19 MORE THAN ONE PICTURE IN THE SAME MEAL 50 FIGURE 4 20 CODE TO REPLACE PHYSICAL ADDRESS WITH VIRTUAL ONE 50 FIGURE 4 21 VIRTUAL DIRECTORY FOR ALL_MY_IMAGES 51 FIGURE S E THREE FALSE POSITIVES EATING PICTURES ini 52 FIGURES 2 MY DIET DIARY es AAA a 53 FIGURE S S ANNOTATED OUANTITIES ria a Foie nto doubt bn anes 62 FIGURE SA ANNOTA TEVQUANHTIES see tia 63 FIGURE S gt ANNOTATED OUAN WINE Sanrio 63 FIGURE 5 5 VIEALS ANNOTATED IMIVIEDIATLY ea a 64 FIGURE 5 7 THINGS TO XIMPROVE ADVICES ss ee 64 FIGURE 5 6 DETAILED ADVICES u 0 ann 65 Vi FIGURE A 1 IMPLEMENTATION OF SAX PARSER IN JAVA anni ae FIGURE 2 MANHATTAN DISTANCE BETWEEN THE CENTER
27. UNIVERSIDAD AUTONOMA DE MADRID ESCUELA POLITECNICA SUPERIOR JA VA UNIVERSIDAD AUTONOMA Polit cnica DE MADRID Superior AN INTERACTIVE SYSTEM TO ANALYZE DIETRY HABITS PROYECTO FIN DE CARRERA Carolina Camacho Garcia MARZO 2010 AN INTERACTIVE SYSTEM TO ANALYZE DIETRY HABITS AUTOR Carolina Camacho Garc a TUTOR Noel O Connor PONENTE Jos Mar a Mart nez S nchez Dpto de Ingenier a Inform tica Escuela Polit cnica Superior Universidad Aut noma de Madrid Marzo de 2010 Keywords SenseCams passive capture segmentation algorithms food pyramid diet Abstract The main objective of this PFC is the design and development of a system capable of analyze and follow the dietary habits of a real user providing advises based on the daily diet to be able to construct a more varied and balanced one For that purpose we will use some devices SenseCams Microsoft These passives wereable cameras capture an average of 1 785 images per day so they can provide a fully detailed diary of our day by day life In this project using algorithms and segmentation techniques we will focus on the images where the user is eating The system will be capable of showing the different meals in the final user screen This user will be able to save a diary interact with the images and even receive personalized data and dietary guides Palabras Clave SenseCams captura pasiva algoritmos de segmentaci n pir mide alime
28. a se reserva el derecho a la utilizaci n total o parcial de los resultados de la investigaci n realizada para desarrollar el siguiente proyecto bien para su publicaci n o bien para su uso en trabajos o proyectos posteriores para la misma empresa cliente o para otra 3 Cualquier tipo de reproducci n aparte de las rese adas en las condiciones generales bien sea para uso particular de la empresa cliente o para cualquier otra aplicaci n contar con autorizaci n expresa y por escrito del Ingeniero Director del Proyecto que actuar en representaci n de la empresa consultora 4 En la autorizaci n se ha de hacer constar la aplicaci n a que se destinan sus reproducciones as como su cantidad 5 En todas las reproducciones se indicar su procedencia explicitando el nombre del proyecto nombre del Ingeniero Director y de la empresa consultora 6 Si el proyecto pasa la etapa de desarrollo cualquier modificaci n que se realice sobre l deber ser notificada al Ingeniero Director del Proyecto y a criterio de ste la empresa consultora decidir aceptar o no la modificaci n propuesta 7 Si la modificaci n se acepta la empresa consultora se har responsable al mismo nivel que el proyecto inicial del que resulta el a adirla 8 Si la modificaci n no es aceptada por el contrario la empresa consultora declinar toda responsabilidad que se derive de la aplicaci n o influencia de la misma 9 Si la empresa cliente decide
29. ado pequefios y negros que tienen que ser llevados colgados del cuello Con el tiempo su dise o puede desarrollarse para hacer dispositivos m s peque os y a la moda Por ejemplo podremos encontrar una SenseCam futura tan peque a como un bot n o que parezca un bonito broche De esta manera ser n si cabe m s c modas de llevar y por supuesto m s discretas Como podemos ver la interfaz con la que hemos trabajado es simple y f cil de usar Esta interfaz puede cambiarse para mostrar m s aspectos de la dieta o para ofrecer al usuario m s opciones Por ejemplo al lado de la barra de cantidades podemos colocar un icono que represente el tipo de comida seleccionada y que se haga m s y m s grande dependiendo de la cantidad de raciones anotadas con la barra Otra opci n interesante puede ser crear una pir mide que cambie cada vez que un grupo de comida sea seleccionado de esta manera podemos evitar repetir comidas que ya est n anotadas Otra funci n que podemos afiadir a nuestro software es un contador de calor as Dependiendo de las cantidades y del tipo de comida anotado el software ser capaz de informar sobre las calor as consumidas en cada comida Adem s ser a una muy buena idea usar el aceler metro incluido en la SenseCam para calcular aproximadamente el n mero de calor as que el usuario quema debido al ejercicio Una de las nuevas opciones de nuestro software puede ser introducir diferentes perfiles de dieta De esta m
30. advices 4 5 2 aspx files In these files we keep all what is related to the graphic interface html and css code but also the code needed to use Javascript This client side language makes it easier to perform a lot of actions In the present project we used it several times All its functions are located in the main interface page main aspx 45 4 5 2 1 show image clicked function As we can see in its name this function save the id of the image clicked by the user to be able to show it in a larger size later When this bigger image have to be shown we look for this variable from the aspx cs page identifying the image in the database and showing it again next to the RadioButtonList and the html image function show image clicked event 10 4 document getElementById aiden javascript store value event id document getElementById btnRead javascript store click Figure 4 13 Show image clicked function 4 5 2 2 deleteColumn function This function is used to delete a picture when it is not an eating image When this happens the user clicks a button besides the image see figure 4 12 This button is connected with deleteColumn Javascript function which is called this way because as the images and the information behind them are represented in a table deleting an image is like deleting a column Once this X button is clicked a Confirm box appears asking the user if he wants to delete the image If the answer is Y
31. age represented in XML 14 There are many different metrics that we can choose to compare all the images and to calculate how dissimilar they are As we can read in 15 several metrics have been tested to find out which one has the best results Vector Distance Method Histogram Intersection Euclidean Manhattan Squared Chord Jeffrey Mod KL Bray Curtis Square Chi Squared X2 Statistics Kullback Leiber Canberra Table 2 2 Vector distance methods results The results are reported in terms of the F1 measure This is a measure of the accuracy of a test It is considered the precision p and the recall r correct results correct results p all results results that should have been returned FL 2 precision recall u precision recall The score reaches its best value at F 1 and worst at F 0 As we can see in Table 2 2 the best metric method based on MPEG 7 is Histogram Intersection followed by Euclidean and Manhattan distance The histogram intersection distance compares only the elements which exist in the query 202 E min p q i q The Euclidean distance measures the shortest distance between two points pi q1 po qn This refers to the group of results that should have been returned due to their relevance 16 The Manhattan distance is the distance between two points if a grid like path is follow
32. alysis techniques will take a very important role helping us to distinguish the several events that took part in the development of our day 2 3 2 Capture and upload A SenseCam user normal day starts of course putting on the SenseCam He must wear it all day or just during some remarkable moments from which he wants to save a graphic testimony In our case as we will focus on dietry habits it would be enough to wear the SenseCam during the different meals taken in a day However it would be better to wear it all the time to avoid forgetting meals Once the day is over the camera will be connected to a PC and the photos will be uploaded Then automatic content based analysis commences making possible to organize all the different events without meaning any effort or waste of time 2 3 3 Processing As we explained once the captured photos have been uploaded to a PC different content based image analysis techniques are applied to these photos We will organized all pictures in a way that will be easy for the user to access them and easier for our particular software to distinguish those photos where the user is taken food The processing will start with event segmentation 2 3 3 1 Event Segmentation The first challenge is to divide all the images taken into different groups or events and determine a boundary that means the transition between different events for example having lunch watching TV going to the supermarket etc T
33. and blue RGB filters that can derive the color chromaticity and luminance intensity of ambient light 2 2 2 5 Electret microphone The mayor advantage of these microphones microphones with thin electrets polymer foil suspended over a perforated backplate is that they operate without needing an external power supply and they are available at very low cost 6 2 2 2 6 Temperature sensor To measure ambient temperature and notice if the wearer has moved for example from outdoors to indoors or viceversa It doesn t require external components and has a shutdown mode to minimize power consumption 2 2 2 7 Edmunds optics lens The camera has an ultra wide angle lens incorporated Although it will introduce some spherical distortion to the image the pictures taken will be closer to the real user view than a picture taken from a camera with a normal lens Figure 2 6 Front and back of the SenseCam PCB 1 Different experiments have demonstrated the importance of sensor data as a trigger for the SenseCam In an experiment carried out by Jason R Finley William F Brewer and Aaron S Benjamin from the University of Illinois 3 some participants wore a SenseCam for five consecutive days The cameras were programmed to take pictures either at fixed intervals or triggered by their sensors On two of the five nights the participants reviewed some of the pictures taken at the end of t
34. anera podremos usar un perfil diferente dependiendo del sexo edad o peso del usuario Adem s podemos relacionar el n mero de fotograf as sacadas en una comida con la cantidad de comida que ingerimos Cu nto m s grande sea la cantidad de comida m s tiempo necesitaremos para terminarla y por tanto m s fotos tendremos Adem s esto puede ser muy util si hay m s de un miembro en la familia que est usando el m todo de las SenseCam Ser a interesante crear gr ficos que puedan ilustrar nuestra evoluci n y la mejora de nuestra diera basado en la historia de nuestras fotograf as 68 References 1 Microsoft Research Sensors and devices group SenseCams http research microsoft com en us um cambridge projects sensecam 2 Ken Wood Rowanne Fleck and Lyndsay Williams Playing with SenseCam 3 R Finley William Brewer and Aaron S Benjamin SenseCam Autobiographical Memory University of Illinois at Urbana Champaign Psychonomics 2008 4 ADXL210E datasheet Analog devices Inc 2002 5 Mary Lynn Garcia The Design and Evaluation of Physical Protection Systems Second edition 2007 351 pages 6 Handbook of acoustics By Malcolm J Crocker Wiley interscience John Wiley amp Sons 7 Passive infraredsensor http itp nyu edu physcomp sensors Reports PassivelnfraR edSensor 8 Lee Hyowon and Smeaton Alan F and O Connor Noel E and Jones Gareth J F and Blighe Micha
35. at our specific needs are Obesity is a very serious problem to take into account Hombres Mujeres Total iTe prevalencia s prevalencia prevalencia Poblaci n infanto juvenil Poblaci n adultat 45 54 a os 55 04 a os Poblaci n anciana 65 a os y m s Institucionalizados institucionalizados Estudio EnKid IMC gt para edad y sexo tablas de Orbegoso Hem ndez et Estudio DORICA IMC z 30 t Aranceta et al IMC 30 Kam amp Guti rrez Fisac et al IMC z 30 Kg m Figure 2 1 Obesity in Spain 2 1 2 Diet control Looking good and feeling healthy can be considered an obsession in present times We are used to read and hear tricks everywhere to lose weight and to look thinner But reality is other thing and these tricks are not always effective The fact is that an equilibrate diet and appropriate exercise is the only way to be in a proper shape More than once we have seen people that have diets stuck on their fridges obsessed about light products willing to write down everything they eat to control their weight These proposals last four or five days tops and it usually takes so much effort that they usually give up So how about making some automatic way of controlling our diet without taking too much time or effort from the person who wants to eat healthy This will be an important step in the traineeship of eating And this learning has to start since our very first y
36. calculation The main aim of our project is to make an interactive system to analyze dietry habits Thus our software has to be able to show the user the different images where the wearer is having lunch breakfast or taking any food discarding all other pictures where the activity is not related to the diet In order to do that we have to process our images with the finality of guessing the activity which is taking place in each of the events Eating riding a bicycle outdoors images indoors activities sitting in front of the computer watching television This is called concept detection and we explain how it is done in the following section 2 3 4 Concept detection For a long time searching for images based on text was the optimal method because the number of images was not large and because there were a limited number of people that needed to access huge images databases However the conditions don t remain constant nowadays We have sites in the Internet like Google picassa and Flickr that attract a large amount of visitor everyday and high end imaging professional equipment are cheaper now Nevertheless looking for images is still a difficult task For example Flickr do not allow pictures to be accessed based on their content but in the o e annotation that the user creates This type of access will be sufficiently accurate if the annotations made by the user were perfect Unfortunately this will never happen Therefore
37. ci n t cnica el depositario de efectos el interventor y el jefe de servicio o un representante estampando su conformidad el contratista 22 Hecha la recepci n provisional se certificar al contratista el resto de la obra reserv ndose la administraci n el importe de los gastos de conservaci n de la misma hasta su recepci n definitiva y la fianza durante el tiempo se alado como plazo de garant a La recepci n definitiva se har en las mismas condiciones que la provisional extendi ndose el acta correspondiente El Director T cnico propondr a la Junta Econ mica la devoluci n de la fianza al contratista de acuerdo con las condiciones econ micas legales establecidas 23 Las tarifas para la determinaci n de honorarios reguladas por orden de la Presidencia del Gobierno el 19 de Octubre de 1961 se aplicar n sobre el denominado en la actualidad Presupuesto de Ejecuci n de Contrata y anteriormente llamado Presupuesto de Ejecuci n Material que hoy designa otro concepto Condiciones particulares La empresa consultora que ha desarrollado el presente proyecto lo entregar a la empresa cliente bajo las condiciones generales ya formuladas debiendo a adirse las siguientes condiciones particulares viii 1 La propiedad intelectual de los procesos descritos y analizados en el presente trabajo pertenece por entero a la empresa consultora representada por el Ingeniero Director del Proyecto 2 La empresa consultor
38. color histogram extracted in HSV Hue saturation and value color space computated with 256 bins To reduce the large size of this representation the Haar transform encoding scheme is used The SCD measures the color distribution over an entire image It s often used to look for similarities in multimedia databases and for browsing in large databases 12 Edge Histogram Descriptor EHD It is scale invariant and represents the spatial distribution of five types of edges vertical horizontal 45 135 and non directional The input image is divided in 16 4x4 non overlapping blocks and a 5 bin edge histogram of each subimage can be obtained Each bin value is normalized by the total number of image blocks and finally the normalized bin values are nonlinearly quantized This descriptor provides a better representation of event boundaries as it was less sensitive to Changes in light than scalable color descriptor 11 Color structure CSD It expresses local color structure in an image by use of a structuring element The CSD is calculated by taking the colors present in each of the pixel that form the 8x8 pixel structure Each color is identified CO C7 and the correspondent CSD bins assign to each color suffers an increment for each positive id This method is equivalent to subsampling the image by the power of 2 and then using the structuring element on it 13 amp xB Structuring element Figure 2 12 CSD structuring element 13
39. crosoft Visual Studio 2005 Microsoft Visual Studio 2005 is an Integrated Development Environment IDE from Microsoft This is the one chosen for the development of this project because it has many advantages for creating robust and elegant applications using ASP NET Amongst other advantages it has code completion which means fewer errors and less typing an HTML editor which provides design and HTML views that update each other in real time and an integrated debugger which allows to follow the code step by step making easy to find the functional errors 4 2 1 ASP NET ASP NET is a web application framework developed and marketed by Microsoft With this tool we can build dynamic web sites web applications and web services We can write ASP NET code using any NET languages C JH CH In the case of the present project we are going to use CH for the code behind Web forms are the main building block for application development They are contained in files with an aspx extension These files contain Hipertext Markup Language and with it we will model the look of the web page the way we want Besides in this same page we will have javascript language which will be very helpful with some of the issues of the design In our project we have two web forms one for the principal page and another one for the advice page Each of them has a code behind file Both are written in CH and they show how the program responds to d
40. ctures for HTML markup are elements Elements have two basic properties attributes and content 4 2 1 3 Javascript _ Javascript is a scripting language that is a programming language that allows IS control of one or more software applications It is used in the form of client side Javascript for the development of dynamic websites Javascript supports all the structured programming syntax in C if while for Javascript was designed to look like Java but to be easier for non programmers to work with However it is unrelated to the Java programming language even though both of them have some similarities they both have C influences and JavaScript copies many Java names and naming conventions It was originally developed by Brenda Eich of Netscape under the name Mocha Later the name was changed to LiveScript to end up being named JavaScript Javascript is a trademark of Sun Microsystems 4 2 1 4 CSS Casvading Style Sheets is a style sheet language used to describe the presentation of a document that is written in a markup language In our project we use Css to present our pages with special images in the background lt style type text css gt body background image url fruits jpg background repeat no repeat background atrachment fixed lt style gt Figure 4 1 Use of Css in the present project 39 4 3 Microsoft Sql Server SOL S Microsoft Sql Server is a relational database
41. dar riy initialized Event User N User launches the application selected 4 dl a y al ta h ce Figure 4 2 Application s start 39 The steps where the stored procedures are called are marked in red The first stored procedure is called when the user launches the application and it is necessary to search the available days to initialize the calendar When the pages loads we would see a calendar opened in the day when we took the first picture Then the user must choose a day in the calendar and the second stored procedure will be called The application will show then the images where the user is eating 4 3 1 1 Available days The first stored procedure implemented looks for those days in the database that have information For that we have to examine the images timestamps to realize how many days of information do we have To work it needs a parameter the identification of the user If the identification number is correct the stored procedure will return the information needed USE Carolina GO Object StoredProcedure dbo Carolina Available Days SET ANSI NULLS ON GO SET QUOTED IDENTIFIER ON GO Author Carolina Create date Description PROCEDURE dbo Carolina Available Days BUSER ID AS INT AS BEGIN SET NOCOUNT ON select min date time from Aiden Images where user id USER ID group datepart
42. desarrollar industrialmente uno o varios productos en los que resulte parcial o totalmente aplicable el estudio de este proyecto deber comunicarlo a la empresa consultora 10 La empresa consultora no se responsabiliza de los efectos laterales que se puedan producir en el momento en que se utilice la herramienta objeto del presente proyecto para la realizaci n de otras aplicaciones 11 La empresa consultora tendr prioridad respecto a otras en la elaboraci n de los proyectos auxiliares que fuese necesario desarrollar para dicha aplicaci n industrial siempre que no haga expl cita renuncia a este hecho En este caso deber autorizar expresamente los proyectos presentados por otros 12 El Ingeniero Director del presente proyecto ser el responsable de la direcci n de la aplicaci n industrial siempre que la empresa consultora lo estime oportuno En caso contrario la persona designada deber contar con la autorizaci n del mismo quien delegar en l las responsabilidades que ostente
43. dvice page with the overall advises The next figure shows the bottom of the advice page 2 6 1 Back to main page Breakfast You didn t take milk for breakfast you know how tasty and healthy is milk with cereal or fruit mmmm Cereals and bread are very healthy But try not to abuse just eat them all little by little all day long Fruit and vegetables are good for your health You introduced it in your breakfast well done Come on leave the sweets and food from meat group for luch or for dinner Lunch Remember that you can always have some milk derived product as dessert How about a little of bread in your lunch It s a very good idea You can introduce a piece of fruit in your kunch Vegetables are very healthy But don t take that much for Lunch Remember that meat fisch and eggs are very important for your diet Always remember be careful eating sweets Dinner Milk is important for our bones you can take some milk derived product for Dinner You can eat bread in your dinner than that Great amount of fruit for dinner Great amount of vegetables for dinner You should take more food from meat group to complete the 3 servings in one day Always remember be careful eating sweets Figure 4 8 Bottom of the page with the specific advices 41 4 5 Code behind To carry out all the functions of the application there are many files that work behind the scenes and that make i
44. e F Triggs B Creating efficient codebooks for visual recognition In IEEE International Conference on Computer Vision Beijing China 2005 pp 604 610 25 Duy Dinh Le and Shin ichi Satoh Concept Detection Using Local Binary Patterns and SVM National Institute of Informatics 2 1 2 Hitotsubashi Chiyoda ku Tokyo Japan 26 V Yanulevskaya J C van Gemert K Roth A K Herbold N Sebe J M Geusebroek EMOTIONAL VALENCE CATEGORIZATION USING HOLISTIC IMAGE FEATURES University of Amsterdam Informatics Institute Amsterdam The Netherlands University Clinic of Bonn Department of Medical Psychology Bonn Germany 70 27 Daragh Byrne1 Aiden R Doherty Cees G M Snoek Gareth G F Jones and Alan F Smeaton Validating the Detection of Everyday Concepts in Visual Lifelogs Centre for Digital Video Processing Dublin City University Glasnevin Dublin 9 Ireland 28 Jesse Liberty Dan Hurwitz Programming ASP NET O REILLY x pu Appendix a Color histogram and Manhattan distance In the first three months of this project the main goal was to perfectly understand how the pictures information was stored in the XML documents and how was the distinction between images and events done For that we focused our attention in creating a Java program that was able to read big XML documents and able to extract all the information that we were interested in The number of images within the same XML could be huge so our pro
45. e during this project A general block diagram will be shown and an overall description of each point will be presented 3 1 Description of system architecture Firstly we are going to show a project schematic as we thought about it since the beginning from the time a real user decides to monitor his diet using a Microsoft SenseCam till he receives diet personalized advices from our software For a better and clearer understanding we can divide it in three parts The first and the third one will require the user intervention and the second one will be done automatically 3 1 1 First stage taking pictures This stage of the process is explained in Figure 3 1 A man a woman or a child decides that it would be a good idea to improve his her dietry habits using SenseCams 1 It can be because other used methods have already failed or just because nobody born knowing how to eat correctly and it s never too late to learn Besides the model we are presenting requires little effort from the user making it comfortable to use After placing a SenseCam around his neck it starts taking pictures passively without user interaction Pictures will be taken each 30 second approximately or every time there is a significant change in the data that SenseCam sensors collect Of course if the user is in some kind of relevant situation he can manually take a picture using the SenseCam photo trigger 2 3 See 2 2 1 After a certain period of time we have t
46. e segmentation is only a 3 396 better including MPEG 7 sources As a result all data extracted from our images are perfectly ordered in a database 7 The dietry monitor software that we have installed in our computer is now able to read the database and extract all the information needed to analyze and show it to the user in our case it is especially important the probability of the wearer taking any kind of food The way the data is organized is very intuitive Each row of information is composed by several fields e Key number and a name of image that indentifies it e Number that identifies the event e Anumber that identifies the user e The date and time of the shot e Visual features MPEG 7 vectors color layout color structure edge histogram and scalable color 28 e Accelerometer values that the SenseCam recorded in that instant of time from each of the axis x y and z e Ambient temperature and values of the rest of sensors light and passive infrared one e Trigger information A P means that it was triggered by a change in passive infrared data an L by a change in Light sensor data an M means that it was manually triggered and a T means that the picture was taken after a certain period of time e likelihood of each of the concepts taken place in each of the images buildings face hands eating Image Key user id event id image path image name date time colour layout colo 293372 2 38 8
47. ears Usually a food pyramid is used to show people good eating habits TT 2 Comer de todo te har E Las chuches y dulces 2 no te alimentan y te estar fuerte y sano lt p Qu debemos comer ED estropean los dientes ocasiones especiales gt mejor en clase La comida r pida o Fast Food Con un buen desayuno ser solamente para te concentrar s Las vitaminas de frutas Hacer deporte te y verduras te ayudan ayuda a crecer a estar guapo a C mo repartimos los grupos de alimentos a lo largo del d a desavuno medi a ana AR comida la J F media ma ana Figure 2 2 children how to eat is important for their future Besides we don t have to forget about exercise as an essential part of our development The recommended thing is to choose some sport or practice that entertains us at the same time that we exercise our body In the next pages we are going to present a way to help people with their diet problems ai 2 2 SenseCams and data compilation 2 2 1 Introduction A SenseCam is a wearable digital camera It can be hold around the neck clipped to the belt or directly attached to different clothes However it is recommended to wear it around the neck because it increase stability the right left movements are minimized its relatively comfortable for the wearer and it stays close enough to his eye line to ensure a good ca
48. ed 16 E 2 IP ail i 1 Figure 2 15 Manhattan and Euclidean distance between two points This last distance Manhattan is the one used in the study showed in Appendix A Sensor readings are scalar values and thus the difference between sensor data of adjacent images is calculated using standard subtraction To calculate the overall dissimilarity score for each image it would be necessary to fuse all the sources of information that we have MPEG 7 descriptors accelerometer ambient temperature light level and passive infrared To do so the values have to be normalized This is because data from different sources can probably not be expressed in the same way in different scales or range although all data are real numbers In the following table some normalization algorithms are presented Name Method Standard Shift min to 0 scale max to 1 Sum Shift min to 0 scale sum to 1 ZMUV Shift mean to 0 scale variance to 1 Table 2 3 Normalization algorithms Standard and Sum are shift and scale invariants algorithms The standard one is sensitive to the maximum and minimum score given for each data source and highly sensitive to outliers while the Sum is sensitive only to the min score and more robust In practice the sum algorithm is fairly outlier insensitive 17 The ZMUV algorithm is shift and scale invariant and outlier insensitive It s very robust After normalizing the different scores we have to combine all
49. ed into the database for that day all the food and the quantities taken This data read is kept in an array of meal data data type a class similar to represent image This way it will be easy for the application to access the information and to ponder the quantities to create the required advices It is probable that the application shows more than one image for the same meal and we shouldn t count the same amount of food two times That is why we only count the first picture of each type of meal So if we have more than one picture of the same meal event we just ignore all the extra pictures We can see how this is done in figure 4 15 where a flag for each of the types of meal is used to know if we have already read information from an image in the same event 4 6 Problems and solutions For the development of the project we fixed different goals Little by little Once a point was reached we travelled to the next one then to the next one and so on till we completed the journey However during this long walk we faced different problems that required intelligent ideas and practical solutions Those problems were both logical and programming Sometimes they were in sql server sometimes somewhere in any of our files in ASP NET In this section we will present some of them and their resolution 4 6 1 Location of database The first thing that we created was a web page with a calendar in it When one day in the calendar was selected t
50. el and Byrne Daragh and Doherty Aiden R and Gurrin Cathal 2008 Constructing a SenseCam visual diary as a media process Multimedia Systems Journal 14 6 pp 341 349 ISSN 1432 1882 9 M Hearst and C Plaunt Subtopic structuring for full length document access In SIGIR The 16th Annual ACM SIGIR Conference on Research and Development in Information Retrieval 1993 10 Laila H Shoukry Visualizing the MPEG 7 Color Layout Descriptor Institute for Visualization and Interactive Systems Intelligent Systems Group Stuttgart University 11 Aiden R Doherty Alan F Smeaton Keansub Lee amp Daniel P W Ellis Multimodal Segmentation of Lifelog Data Centre for Digital Video Proessing amp Adaptive Information Cluster Dublin City University Ireland Lab ROSA Columbia University New York USA 12 Vinay Modi Color descriptors from compressed images School of Informatics The University of Edinburgh UK 13 Adis Buturovic MPEG 7 Color Structure Descriptor for visual information retrieval project VizIR Institute for Software Technology and Interactive Systems Technical University Vienna 69 14 M Sezgin and B Sankur Survey over image thresholding techniques and quantitative performance evaluation Journal of Electonic Imaging 13 1 146 168 2004 15 Aiden R Doherty and Alan F Smeaton Automatically Segmenting LifeLog Data Into Events Centre for Digital Video Processing and Adaptive Information ClusterDublin
51. er ajustados a las necesidades de cualquiera De esta manera necesitaremos ir al doctor menos veces que con el m todo tradicional porque simplemente con una visita podemos ajustar el programa a nuestras propias necesidades Adem s el m todo de las SenseCam nos brinda m s posibilidades para controlar nuestra dieta Podemos enviar nuestra informaci n al doctor a trav s de internet y podr seguir nuestra diera mientras nosotros estamos tranquilamente en casa Usando el m todo de las SenseCam me di cuenta de que mi dieta no era tan buena como pensaba que era Desde entonces intento mejorarla d as tras d a En referencia al procesamiento e identificaci n de eventos encontr que la precisi n era m s bien aceptable sin embargo hay algunas fotograf as identificadas como eventos comida que no lo son Para ello hemos dise ado un m todo para eliminar las im genes sobrantes 6 2 Future work Nowadays SenseCams are not very little black devices that must be worn around the neck With the time its design can be developed to make them even smaller and more fashionable For example we could find a future SenseCam as small as a button or a SenseCam that looks like a beautiful brooch This way they will be even more comfortable to wear and of course more discreet As we can see the interface we have worked with is simple and easy to use This interface can be changed to show more aspects of the diet or to give more options to the
52. er as a serving for each the pyramid s groups Group 1 sweets We don t count serves from this group The only advice is to use them sparingly Group 2 milk One serving equals 1 cup of milk or yogurt 60 grams of cheese or 2 cups of ice cream If it is low fat it will be better Group 3 Meat fish eggs One serving equals 30 grams of cooked lean meat poultry or fish 1 4 cup dried beans after cooking 1 egg 1 tablespoon of peanut butter or 15 grams of nuts or seeds Group 4 vegetables One serving equals 1 2 cup of raw or cooked vegetables or vegetable juice or 1 cup of leafy raw vegetables Group 5 fruit One serving equals 1 cup of fruit or 100 percent fruit juice or 1 2 cup of dried fruit Group 6 bread One serving equals 1 slice of bread 1 2 cup of cooked rice pasta or cereal or 30 grams of cold cereal c New food pyramid After showing the software to our DCU nutritionist he told us about a new food pyramid that experts in nutrition were designing Figure A 3 Two moments of our meeting with the nutritionist In this new pyramid we still have the six different groups of food but instead of horizontal separating lines we have vertical ones separating different colors Each color represents one different group of food In this new Pyramid we make a distinction between good fat and less recommendable fat and the importance of exercise is shown Furthermore it teaches us that not everyone needs the sa
53. esentation of images in the screen To be able to show the images in the screen it wasn t enough to just inform the program about our picture s physical address This way we couldn t present our photographs Instead of our SenseCam pictures we had a blank space with a red cross To solve this problem we tried creating a virtual directory mapped to the local folder where we have all our SenseCam images Replacing the physical address with the new virtual one our interface was able to show the pictures normally Replace C nueva carpeta All my SenseCam images http Figure 4 20 Code to replace physical address with virtual one 50 To create the virtual directory we used the Windows Vista tool IIS 7 Internet information services We can see in the following figure the virtual directory created and its content in the physical folder LE Administrador de Internet Information Services 15 PRIN Sitios k Default Web Site F All My Images Archivo Ver Ayuda g 4 1 PRIM Prin Carolina b 3 Grupos de aplicaciones Sitios Default Web Site 1 Images ii 2009 07 30 17 07 13 2009 07 31 16 56 16 gt 2009 08 09 16 46 40 2009 08 14 07 38 40 Desktop Images t 2009 07 30 17 07 13 2009 07 31 16 56 16 Listo Filtre Mombre 3 2009 07 30 17 07 13 2009 07 31 16 56 16 2009 08 09 16 46 40 C 2009 08 14 07 38 40 desktop ini
54. event cannot be selected as representatives due to their poor quality 2 3 3 3 Calculating event novelty The last step in the processing of the images is to calculate how important or how exceptional an event is When the user wants to review his past activities we can understand that it will be more preferable for him to take a look at those images of events that are more unique and unusual It will be worthless to review those pictures in which the user is just sitting at his desk working being more interesting to look through events like his daughter birthday party or meeting an old friend eU ER A S E 2 Importance Calculation Figure 2 22 Determining event importance 22 Event representative vectors composed by the low level MPEG 7 visual features and accelerometer light passive infrared and temperature values are compared one another using the Manhattan distance to determine how similar they are If an event contains a lot of uniformly dark images the novelty score will be zero The novelty selection technique basically consist in picking the most dissimilar event during a certain period of time This period of time can vary In 22 we can read a study about novelty calculation over different windows the previous 7 days previous 3 days previous 14 days events within 2 hours of the event we are trying to compute the novelty of from previous 7 days and same week day in the
55. figure we can see how the program performs this checking 49 if ToSave data selected counter ToSave counter quantity i Response Write You have not introduce data correctly Try again for int z 0 2 lt ToSave data selected counter 2 i ToSave quantity z 0 Figure 4 18 Checking errors when introducing data 4 6 5 More than one picture of the same meal Almost one hundred per cent of the times we will have more than one picture of the same meal This is obvious because the SenseCam takes pictures very often and a lunch time for example lasts between half and one hour Therefore we will have more than one picture with the same information from the same meal of a day To avoid confusion created by repeated data or having wrong results in the cases when two or more pictures are shown for the same meal only data from the first picture will be taken into account int repeat new int 10 meaning 1 breakfast 2 lunch 3 dinner 4 afternoon snack 5 extra meal For this each time that we identify a different type of meal a flag will be placed in the position of the array that corresponds with the meal chosen Before saving any information about a type of meal already introduced the program will check the array of flags Identify breakfast if day images i get meal type Equals Breakfast amp amp repeat 1 0 i Figure 4 19 More than one picture in the same meal 4 6 6 Repr
56. finitiva al a o de haber ejecutado la provisional procedi ndose si no existe reclamaci n alguna a la reclamaci n de la fianza 18 Si el contratista al efectuar el replanteo observase alg n error en el proyecto deber comunicarlo en el plazo de quince d as al Ingeniero Director de obras pues transcurrido ese plazo ser responsable de la exactitud del proyecto 19 El contratista est obligado a designar una persona responsable que se entender con el Ingeniero Director de obras o con el delegado que ste designe para todo relacionado con ella Al ser el Ingeniero Director de obras el que interpreta el proyecto el contratista deber consultarle cualquier duda que surja en su realizaci n 20 Durante la realizaci n de la obra se girar n visitas de inspecci n por personal facultativo de la empresa cliente para hacer las comprobaciones que se crean oportunas Es obligaci n del contratista la conservaci n de la obra ya ejecutada hasta la recepci n de la misma por lo que el deterioro parcial o total de ella aunque sea por agentes atmosf ricos u otras causas deber ser reparado o reconstruido por su cuenta 21 El contratista deber realizar la obra en el plazo mencionado a partir de la fecha del contrato incurriendo en multa por retraso de la ejecuci n siempre que ste no sea debido a causas de fuerza mayor A la terminaci n de la obra se har una recepci n provisional previo reconocimiento y examen por la direc
57. given by the accelerometer at the time the image was taken If the accelerometer registered some kind of motion at that precise moment it is very probable that the resultant image is blurred Besides many pictures suffers from lack of light or from excess of brightness due to ambient light so if we have a look at the values registered by the light sensor we can have an idea of the images that present a poor illumination In the next example we have chosen the middle image of the nine first events on the day of a wearer 20 Event 0 Event 1 Event 2 Event 6 I Event 7 Event 8 As we can see in the previous figure almost every image selected satisfies their role as a representative photo However we can see that there has been a mistake between event O and event 1 as they should take part of the same event Besides the central image of event 6 is perhaps too blurry to be a good representative It is probably caused by a quick movement of the wearer This is why we need to consider the quality of the image chosen to have good representatives x 10 Manhattan distance between event 0 and 8 with the rest of events 5 T T T T Figure 2 21 Manhattan distance between Event 0 and 8 of figure 2 20 with the first 12 events Using colorHistogram 21 Once we have the quality score of the images we can apply the Kapur 14 adaptative thresholding technique to know which of the images in an
58. gmentation algorithms and methods concept detection e Chapter 3 Design of the system e Chapter 4 Development and implementation of the system e Chapter 5 Tests and results 2 State of the art 2 1 The importance of learning how to eat 2 1 1 Introduction Every living being needs besides water nutrients vitamins and proteins to be alive and to carry out all the basic functions that ensure the right performance of our body Nutrition is not the same as diet While nutrition is the physiological process that transforms what we eat into substances that our body can assimilate diet is the act of eating Every living being has its own needs a lion or a wolf for instance couldn t survive eating plants Similarly it is not in the nature of cows elephant or goats to take meat for their diet Other animals as pigs or brown bears eat both meat and plants This last group is called omnivorous just like human beings are We need a mixture of different nutrients to grow up strong and healthy Nevertheless it is not that easy for us to determine what our proper needs are If we take a look around we can see how fast food is taking a very important place in everyone s life and how children suffer from some deceases that wouldn t be possible if they were following a suitable diet Therefore diet is something we have to learn As we can see in the following table we have to put every effort in controlling our diet and learning wh
59. gram must be stable no matter the number of pictures stored in it To see how features information is stored in the XML document see Figure 2 14 To parse the XML document we used a SAX parser Sax parser is faster than Dom one and can handle bigger documents public static void maimsmi sotcring args i LEY 1 JSixParserFactory factory S xParserFactory mnewinstance l SAXParser parser factory newsohzParseri l SaxHandler handler new SaxHandler parser parse image xml handler catch Exception ex ex printo tackTrace System outi Figure A 1 Implementation of Sax parser in Java To compare the different pictures we chose color histogram feature of the centre image of each of the events and made the comparison using the Manhattan distance The reason to choose Manhattan distance as the vector distance method is that Manhattan distance is very easy to implement and the results are more than acceptable in comparison with other vector distance methods See Table 2 2 In the next figure we can see the algorithm used to calculate the Manhattan distance between the centre image of each of the events with the center image of the rest of them for 1120 1 lt 1 1181 12 11 1 1 4 for d i 1 d lt folourList sizel d 14 for a70 data int i a 1 3a4 i if data int i a data int d a lt 0 1 auz data int i a data int d a aux aux i 1 manhattan ca aux ELSE manhatta
60. haciendo uso de las SenseCams En la primera fase se estudiar el formato del fichero XML en que estos dispositivos almacenan los diferentes datos que recogen y se comparar n los histogramas de color para hacer una segmentaci n ptima de los eventos En una segunda fase se realizar detecci n de conceptos SVM De esta forma se dispondr de una base de datos con informaci n sobre cada una de las im genes Ser posible consultar si una determinada imagen es de interiores de exteriores de un paisaje y m s concretamente si se est comiendo o no Posteriormente habr una tercera fase en la que se desarrollar la interfaz gr fica por medio de ASP NET Debe ser una interfaz con la que el usuario pueda interactuar f cilmente Con ella se deber ser capaz tanto de recuperar informaci n de la base de datos del usuario como de insertar nuevos datos aportados En la fase n mero cuatro es el momento de trabajar con un experto en nutrici n y deporte para poder calcular tanto las calor as ingeridas como las calor as gastadas por el usuario en su d a a d a En la quinta fase probaremos el programa Bas ndonos en los resultados de las pruebas modificaremos el programa para que se ajuste a las necesidades de los usuarios reales 1 2 Document structure This document has the following sections e Chapter 1 Introduction motivation and goals e Chapter 2 State of the art about SenseCams and the way they compilate data se
61. he day These participants took memory tests at intervals of 1 3 and 8 weeks The results showed that the participants remembered clearer the days that they had previously reviewed and the images taken in sensor mode than the images taken in time mode 0 60 70 0 55 lt 0 50 B 0 45 25 A 0 40 5 0 35 gt E 0 25 e 0 20 En 1 week 3 weeks 2 weeks 1 week 3 weeks B weeks Recall of day and time was more Picture cued recall responses were accurate for sensor triggered than longer for sensor triggered than timer triggered images timer triggered images Figure 2 7 Sensor triggered images vs timer triggered images 2 3 Capture upload and processing of the images 2 3 1 Introduction As we pointed out before at the end of the day the wearer has approximately 1750 pictures in his SenseCam memory card Thinking about organizing all this information and being able to search for a particular image doesn t seem a very pleasant work to do manually For example looking through the pictures taken on a single day will take approximately 10 minutes expending just 0 3 seconds per photo Of course SenseCam will not be worthy if we have to spend one hour a day just revising the images one by one and manually dividing all the different events That is why we need an automated process to help us organize all the visual material that we will achieve at the end of each period Here is where content based image an
62. he images of that day appeared on screen Initially all these pictures were kept in a database in a DCU server To access the images we used the DCU server ip address Due to this to access the images we needed connection to the internet and be part of the DCU LAN This seems to be a problem when we wanted to access our own images from anywhere outside DCU which happened often To solve this problem we create a table in our local database and move the information needed to our own computer From then on all the images and data would be downloaded to our local database This way we could be independent from a remote database and access our information from everywhere public static string DATABASE CONNECTION STRING Data Source 136 206 19 54 Initial Catalog DCU SenseCam User Carolina Password Madrid public static string DATABASE CONNECTION STRING Data Source local Initial Catalog DCU SenseCam Integrated Security True Figure 4 16 Old vs new Database Connection String 48 4 6 2 Day without images Not all days in the calendar have images At the beginning this was a problem because when a day without images was selected the application returned an error To repair this we made calendar control to return a null value if the day selected had no eating images From the principal web form we can read the null value and write a response in the screen informing the user that there are no images available for the da
63. hen there is a change in the environment and the sensor data itself will facilitate the indexal and retrieval The union of all these data will help to create a better understanding of a situation in every detail All these captured data may be related altogether via time correlation Table 2 1 shows the hardware present in a SenseCam 16 876 8 bit microcontroller EC bus interfaces sensor penpheral devices to the microcontroller ADXL210 2 channel accelerometer 2 axis motion and tilt Passive infrared sensor Seiko detects heat from a person at up to 25m Digital Light Sensor 105230 records overall and RGB intensities Electret Microphone records audio level Temperature sensor LM75 0 to 0C Real time clock to record calendar data of time of image R5232 serial interface for loading data to PC MMC Card 64 MB for sensor data Camera module 125 MB various commercial modules used Edmunds Optics lens 2 2mm f2 ultra wide angle 132 degrees 2 x AA NiMh cells for recharging after 12 hours Table 2 1 SenseCam hardware specification 2 2 2 2 1 SenseCam camera The resolutions of the pictures that are taken are VGA 640x480 pixels This is not a very high resolution and the captured images all suffer from spherical distortion 2 However the purpose of the SenseCam is not to create wonderful and rich media but to aid a user with some memory loss or as a simple reminder of how a certain day we
64. his will have a computational cost of n n nznumber of images of an event e Select the image which is closer to all other images that compose the event but that also is most distinct from all other images present in the rest of the events This approach is computationally the most expensive n n m m number of images in a day e Select the image closest to the average value of the visual features of its own event but also as different as possible from the average values of the pictures of the rest of events The computational cost will be n e ez number of events in day e Select the middle image from each event The most intelligent approach to use is the forth one as it s easier to compute and because in practice the results doesn t seem to be very different to the others approaches However to select a representative image we will have to consider some aspects It s important to choose a photo that is well focused and that has a level of brightness that is appropriate If it is too dark or too bright the chosen image will not be a good representative Because of the nature of a SenseCam a wearable and mobile device many pictures can be taken while the wearer is moving around resulting in a picture more fuzzy and blurry than expected Besides the camera can eventually move to a bad angle receiving the light directly to its lens and taken pictures too bright to be considered representative Thus it s obvious that quality may
65. ifferent events for example a page being loaded or a control being clicked 28 4 2 1 1 CHA C Sharp is a simple modern object oriented programming language It has its Cit roots in the C family of languages and it also looks very similar to Java CH is standardized by ECMA International as the ECMA 334 standard and by ISO EC as the ISO EC 23279 standard These are some of the design goals listed in the Ecma standard e Simple moder general purpose object oriented programming language 94 gt e Support for internationalization e Source code and programmer portability 5 principal designer and lead architect at Microsoft is Anders Hejlsberg who previously was involved in the design of Turbo Pascal CodeGear Delphi and Visual J With this language we will develop the code behind in our project where we specify the action taken when any event occurs and where we will connect with the database to introduce and extract the information needed 4 2 1 2 HTML Hipertext Markup Language is a language for describing web pages lt allows creating structured documents by denoting structural semantics for text such as headings paragraphs lists html images It is not a language itself it is a markup language which is a set of markup tags These markup tags are used by HTML to describe web pages These tags are surrounded by angle brackets like html and normally come in pairs like lt b gt and lt b gt The basic stru
66. ifferent types of groups in which the food is divided and a brief explanation of the correct amount of serves per day for an adult user At this stage the user can decide if he wants to review the eating images of a certain day and introduce the eating information or if on the contrary he prefers to get some advice based on the amount type of food taken a certain day See 4 4 2 Choose a day in the calendar or Click to get some advice Fats Oils amp Sweets Use Sparingly gt M m 5 EA ju i 8 lun e ig E Meat Poultry Fish Dry Beans Eggs amp Nuts Group 2 3 Servings Is P Is gt 5 IS Its l l Fruit Group 2 4 Servings le e R b E w IN Bread Cereal N Rice amp Pasta Figure 4 5 Principal page of the application If the user decides that he wants to revise the photographs taken a certain day he just has to select a day and follow the instructions that will appear in the screen As we can see in figure 4 6 all the eating images are shown with little icons which indicate what food is already annotated for that image in the database If one of the images is not an eating image the probability of eating will be changed to 0 just pressing the X beside it This way next time that we select that day this image will not appear anymore Figure 4 6 Windows Internet Explore
67. igned a method to delete all images that are not needed Conclusiones Como podemos ver en la secci n anterior si usamos SenseCams es m s f cil recordar lo que comimos en cada una de nuestras comidas Tener fotograf as de nuestra dieta nos ayuda a recordar no solo la comida que tomamos un d a cualquiera sino la cantidad que comidos de cada alimento Adem s hay d as en los que comemos fuera de las horas tradicionales La mayor a de estas veces no tom nota de lo que comi sin embargo todo quedaba grabado en mi SenseCam 662 Est probado que el m todo de SenseCams es m s efectivo y c modo que el tradicional bloc de notas para apuntar nuestra dieta Cuando usamos el cuaderno nos dimos cuenta de que muchas anotaciones no eran lo suficientemente precisas con un 26 de error y algunas veces nos olvid bamos de escribir la comida 32 de las veces lo que hac a que tuvi semos dificultades para recordar lo que hab amos tomado despu s de alg n tiempo En cuanto a la creaci n de consejos para mejorar nuestra dieta y por lo tanto nuestra salud podemos observar que la creaci n de consejos por parte de nuestro software es instant nea Sin embargo si usamos un cuaderno para apuntar nuestra dieta necesitar amos visitar a un nutricionista o a un m dico para saber qu pasos tenemos que tomar para seguir una dieta correcta Los consejos creados por el software han sido dise ados por un especialista en nutrici n y pueden s
68. igure 2 17 False boundary accelerometer due to wearer movement Finally we can notice that there are three types of activity boundaries change of activities within the same location wearer involve in social interaction and the movement to a different location For the first one a fusion of image processing and light sensor will provide the most accurate results The segmentation done with accelerometer values gives its best results for activities which implicate moving to different locations and finally the audio sensor alone or the fusion of the image and light processing sources provides better detection of the boundaries that implicate the wearer socializing with colleagues 19 Data Source Precision Data Source Precision DataSource Precision lemperature 17 48 Image processing 26 81 Audio 18 51 Accelerometer 17 45 Light sensor 10 37 Imagellightfusion 20 60 Fusion ofboth 0 3 Fusinofbo 4 7 Fusinofaudioimagellight 112 Table 2 6 Identified change location within location and social segmentation 18 2 3 3 2 Landmark photo selection The second step in the processing of the images taken by the SenseCam involves the selection of a landmark picture for each event An event has on average one hundred photos so it would be helpful if we can choose just one photo that could represent the content of each event To proceed we have four options e Select the image which is closer to all other images that compose the event t
69. iven orientation on a given scale and frequency and they are also sensitive to color planes 26 e Feature and classifier fusion here we employ average fusion of classifier probability scores After this we obtain our final concept detection score Then for each concept we calculate a threshold value We can use the Kapur automatic thresholding technique This is an entropy based non parametric technique that don t require any training so it can be easily applied to a large collection of images All images above this threshold will be considered positives and every image below will be a negative s25 s System Positive Accuracy LEN EE Gm E I eo o vehiclesExternal b z 756 726 726 69 68 64 63 62 61 61 60 ge ge ge 480 470 460 450 LF 1 L Table 2 7 of detection for some concepts Accuracy It was found that the accuracy of the system was more than acceptable From the 95 907 images that are judged manually and with the automatic technique in 27 the 7596 were correctly classified by the system From them the system correctly identified 5796 of true positives and 9396 of negatives Be 3 Design In this chapter we will explain with detail the design of our interactive dietry habits analyzer and the evolution of its architectur
70. m DCU have determined with experimentation that the gap of 3 minutes is the best 15 The MPEG 7 visual features are color structure color layout scalable color and edge histogram Once the image is processed as explained before and all the MPEG 7 visual features are extracted we have to proceed to compare images characteristics and for that we will choose the best metric for our purpose In Figure 2 14 we can see how the visual features of each picture are represented using Extensible Markup Language to facilitate its manipulation Appendix A In the figure the events are already separated and marked ImagelnfoCollection Image blueRegionAverage 14 217396 12 927767 4 7789097 13 228907 10 220811 11 8553 81 1 5571094 2 4198046 2 2613835 bookmark DCU Event 0 colourAverage 45 71277 59 83522 73 47109 colourHistogram 121 106 43 74 301 620 718 1411 2637 4554 44815 18706 1150 232 103 88 34 14 1315 109 112 182 1033 820 15844 3279 5040 11195 33203 3664 1260 378 77 87 115 96 63 0 0 0 0 0 183 532 1444 6412 28961 390 00 0 0 0 0 38450 88 696 546 139 61 48 56 97 893 35750 203 61 94 6 22 22 38 528 12 distance 0 greenRegionAverage 1 1 300309 9 423034 4 6796713 11 157379 8 149681 9 996442 _1 3440951 1 8769531 1 9110873 redRegionAverage 9 255179 7 45305 2 8036199 8 652275 6 0556545 7 7606444 0 80875 1 3782878 1 545306 timestamp 2008 10 06T10 44 04 gt lt ImagelnfoCollection gt Figure 2 14 Visual features of an im
71. me amount of food to have a correct diet A e m e ER S TUNA E Grains Vegetables Fruits Milk Meat amp Beans Moke half your groins whok Vary your veggies Focus on fruits Get your colcium rich foods Go lean with protein Got 3 ovory doy Eat 21 2 cups every day Eat 1V cups every day for kids doe fic amp is Hu Eat 5 oz every day tot 6 every day ot holf Ge whois grogi Find your balance between food and play Fats and sugars know vour limits Figure A 4 Because one size doesn t fit all 1 2 3 4 5 6 7 8 PRESUPUESTO Ejecuci n Material Compra de ordenador personal Software incluido 2 000 SENSE Cd ee O aes 5604 1504 total de ejecucion materias 2 710 Gastos generales 16 7 sobre Ejecuci n Materials u eer si 434 Beneficio Industrial O sobre Ejecucion Material 163 Honorarios Proyecto 900 horasa 15 Nord idea 13500 Material fungible Gastos de IMPresiON iia aia 60 e Encu dernacion suite 200 Subtotal del presupuesto e Subtotal Presupuesto ooo 17067 aplicable e 16 Subtotal 2730 7 Total presupuesto otsliPresQpues 6 usss
72. model server produced erver by Microsoft The model for a relational database is based on first order predicate logic The main unit of data storage is a database a collection of tables with typed columns To retrieve data from an SQL Server database we will have to query it The query is expressed using a variant of SQL called T SQL This way we specify what is to be retrieved and it will be a task for the query processor to figure out the sequence of steps that will be necessary to retrieve the needed data SQL Server has been evolving since the late 1980s Microsoft SQL Server was originated as Sybase SQL Server in 1987 The version used in this project Microsoft SQL Server 2005 was released on November 7 2005 Along this project we have used Microsoft Sgl Server to retrieve needed data in several occasions which will be explained along this document We created a database See 3 1 2 with all the data extracted from the images and from the SenseCam sensors Besides we used stored procedures when our system required the execution of several SQL statements This way our application had to call only the stored procedure We have implemented two stored procedures one for getting the days available days when we took photographs and other to get the images where the user is eating 4 3 1 Stored procedures In the following figure we can see an overall view of how the first part of the development of the system works Shows Etha Calen
73. n data int i a data int d a manh dist i 4 1 manhattan manh dist i d 1 manhattan 0 for 1150 1 lt 10 1181 12 1 1 1 1 for 0 9151111 a 1 14 System ort println Manhattan distance of central image event T atl r manh dist i a LI with event Figure A 2 Manhattan distance between the center picture of each event with the centre picture of the rest of events As we can see the Manhattan distance is stored in a 2D array and the results printed in the screen To prove how this software works we used SenseCam images from twenty eight correlated events and a total of 4633 images The results of the first 12 events are represented in the next table Manhattan distance of the central image of first 12 event of the day 1 2 3 4 5 6 7 8 9 10 11 0 0 196610 118052 216278 348740 159686 178400 101064 162484 168828 188752 115374 1 156610 117726 255200 325728 133526 207332 175610 159606 92100 173918 161300 2 118052 117726 0 255400 342414 123594 193134 94594 161384 133808 155922 126370 3 276278 255260 255400 0 408330 314996 235906 273128 254327 249052 247226 252278 4 348740 325728 342414 408330 0 328652 373930 343634 281452 308238 380936 349728 2 159686 133526 123594 314336 328052 0 198082 154942 173986 170400 165734 172006 6 178400 207332 193134 235906 373930 198082 0 181158 178390 190182 186190 171644 y 101064 175610 94594 273128 343634 154342 181158 0 175400 1609
74. nt by These pictures are stored as compressed jpg files on a flash memory placed inside the device Due to the low resolution of the images over 30 000 images can be fitted en a 1 Gb card Furthermore in the flash memory we also have place for a log file where we can find other sensor data and their correspondent timestamps 1 2 2 2 2 Accelerometer An accelerometer measures the change in velocity over time relative to freefall These devices are able to detect the magnitude and direction of the acceleration The accelerometer present in the SenseCam can measure accelerations with a full scale range of 10g It can measure both dynamic and static acceleration To measure it the device has outputs of analog voltage or digital signals whose ratio of pulse width to period are proportional to acceleration 2 2 2 3 Passive infrared sensor It s called passive because it does not emit any energy All objects emit an infrared radiation that is invisible to the human eye but can be detected by electronic devices designed for such a purpose It is activated with the changes in the thermal energy emitted by a human intruder in comparison with the temperature of the object in front of which this human is passing A wall for example 5 PINS 1 2 ON A HORIZONTAL PLANE FRESNEL LENS HEAT SOURCE MOVEMENT Figure 2 5 How passive infrared sensor works 7 OUTPUT SIGNAL 2 2 2 4 Digital light sensor It has an integrated red green
75. nticia dieta Resumen El objetivo de este Proyecto Fin de Carrera es el dise o y desarrollo de un sistema capaz de monitorizar y analizar los h bitos alimenticios del usuario proporcion ndole consejos basados en su dieta diaria con el fin de hacer de sta una m s sana y equilibrada Para ello contaremos con unos dispositivos SenseCams Microsoft Estas c maras pasivas toman una media de 1785 im genes por d a y si se llevan colgadas del cuello pueden proporcionarnos un diario detallado de nuestro d a a d a En este trabajo y gracias a algoritmos y t cnicas de segmentaci n nos centraremos en aquellas instant neas en las que el usuario est comiendo El sistema ser capaz de mostrar las diferentes comidas en la pantalla del usuario final quien podr guardar un diario de las mismas interactuar con ellas y incluso recibir datos y gu as de alimentaci n personalizadas This Master Thesis has been developed in the Centre for Digital Video Processing research centre belonging to the School of Computing and the School of Electronic Engineering Dublin City University Dublin Ireland Agradecimientos Quisiera agradecer a todas las personas que han hecho posible la realizaci n de este proyecto A Jos Mar a Mart nez porque cuando le propuse mi proyecto en Irlanda todo fueron facilidades y porque siempre he podido encontrarle cuando le he necesitado durante el proyecto y durante los a os de carrera A Noel O Conn
76. o be able to do so we will have to use context based sensor analysis techniques in conjunction with content based image analysis 8 day s SenseCam images 3 684 4 000 _ uU UU UU UU ua uu UU ee Finishing work At the hus Chatting at Skylon Hotel Moving to time On the way lobi y in tha lab stop room back home Figure 2 8 Segmenting images into events 340 Talking about SenseCams if we compare adjacent images we can find two very different images inside the same event This is because the images are taken in a low frequency compared for example with the rate of images taken for a video camera and a photo could be taken when the user turn himself for a moment while sitting in front of the computer This is the most common reason to trigger false events slightly change of position of the user Figure 2 9 Examples of false positive boundary 11 To segment a group of images into events using content based image analysis an adaptation of Heart s Text Tiling approach is used 9 With this technique we take a reference image and then compare the block of images previous to it with the block of images that come afterwards Each block is represented with the average value of the low level MPEG 7 visual features of all images present in that block This way we can solve the problem of intruder images inside a certain event New event Figure 2 1 0 The only flower image cannot be con
77. o remember our diet after certain time every night before going to bed was forced to go through my diary to check if every meal was written Furthermore there are days when we have food outside the traditional hours Most of these times didn t write down what took but everything was recorded in the SenseCam Regarding the creation of advices to improve our diet and therefore our health we can see that the creation of advices with our software is instantaneous However if we are using the diet notebook we will need to visit a nutritionist or a doctor to know which steps we have to take in order to follow a correct diet The advices created by the software were designed by nutrition professionals and can be adjusted to anyone needs This way we will need to go to the doctor less times than with the traditional method because with just one visit we can adjust the program to our own needs Besides the SenseCam method gives us further possibilities for the control of our diet We can send our doctor via Internet all our eating data and he will be able to follow our diet while we are peacefully at home Using the SenseCam method realized that my diet wasn t as good as thought it was Since then try to improve it every day Talking about image processing and identification of events found out that the precision is acceptable however there are sometimes pictures identified as eating events which are not For that we have des
78. o upload the images taken using a personal computer 4 We can upload the photos at the end of the day to let the second part of the process start After uploading the images the user cannot forget to charge the SenseCam battery to have it ready to for next day 2 take pictures passively Upload SenseCam Photos Figure 3 1 First stage 2 It is important to install the Dietry Habits Analizer software in our personal computer This is the tool that we will use to review all our eating images and to introduce all the additional data needed Besides it will give us personalized advices about our diet 3 1 2 Second stage image processing This stage is done automatically without user intervention Our purpose is to prepare and organize all data extracted from the images When the software needs to retrieve any data it could find it just connecting to the database and searching for the field needed i eating d nee 8 INdoors E outdoors Local server Concept detection SVM Event segmentation Figure 3 2 Second stage Once the wearer has uploaded the images using a computer 4 the event segmentation and concept detection software will do their work 5 6 Due to practical reasons we have only used the data provided by the sensors installed in the SenseCam to segment all the events This is because the computational cost is too much lower and the accuracy in th
79. of the pictures are not taken from the proper angle and it can be difficult sometimes to distinguish the correct amount of food Annoted quantities N OS Z N AA eee uh fer m 3 o eee Diary Real q Day1 Days Day3 Dayd Days Dave Day Figure 5 3 Annotated quantities SenseCam method gt 8 error Diet diary method gt 26 error 5 2 8 2 Number of pictures taken from each meal The number of pictures taken in each meal depends on different factors The most important one is the time that the meal lasts Obviously a longer meal will have more pictures Besides the frequency with which the pictures are taken can vary depending on the information collected by the SenseCam s sensors changes in illumination movements changes in temperature 62 N9 of meal pictures Day 1 Day 2 Day 3 Day 4 Day 5 Day Day 7 Figure 5 4 Annotated quantities 5 2 8 3 Forgotten meals The following graphic shows the meals that we have forgotten to annotate in both methods SenseCam and diet diary The two meals that haven t been written down in the diet diary were two snacks taken during the afternoon It wasn t a big amount of food and forgot to write it down at the end of the day However in our SenseCam the pictures were recorded and it was difficult to forget about them Forgotten meals SenseCam B Diary Figure 5 5 Annotated quantities 263 5 2 8 4 Meals annota
80. ones In this case we create a slider from an asp TextBox and a label The TextBox will show the quantity selected and the label will contain the slider itself The values that can be chosen go from 0 to 5 They reference the serves taken of each food To know how much is a serve see Appendix B These values are recorded to count the serves taken from each food each day Then they will be compared with guidelines of the optimal amount of food for each meal and for an entire day to see what the mistakes made are lt form id rorml runat server gt lt asp ScriptManager ID ScriptManagerl runat server gt asp ScriptManager lt asp TextBox ID txtValue runat server gt lt asp TextBox gt lt asp Label ID lblValue runat server Text Label gt lt asp Label gt ccl S5liderExrender ID 5liderExtenderl1 runat server largerControllD txtValue EnableHandleA nimation true BoundConrrollID lblValue Minimum Maximum 5 gt lt ccl 5liderExtender gt Figure 4 10 Slide using AJAX Control Toolkit Once the data for an image is introduced we can save the changes made and continue with other images If we refresh the page we can see that the icons for the new food annotated appear under their picture If we have finished introducing new data we can go to the advice page Changes saved E ee vx D Fats Oils amp Sweets Use Sparingty Milk Your t
81. or que me tuvo en cuenta desde el momento en que llam a su puerta y ha hecho posible que desarrollase este proyecto en la Dublin City University Quisiera agradecer tambi n a todo el equipo del Centre for Digital Video Processing por toda la ayuda prestada y muy en especial a Aiden Doherty Aiden for your help your time your patient your ideas your friendliness and your kindness thanks thanks a million Tambi n agradecer a todo el profesorado de la Escuela Polit cnica Superior y en especial a Jes s Besc s Cano por ser tan cercano y porque siempre nos ha llamado a todos por nuestro nombre lo cual me parece el mayor de los piropos Gracias la Universidad Aut noma de Madrid a la O R l o a quien sea que tenga que agradecer por haber hecho posible que pasase un a o de Erasmus en la M lardalens h gskola de Vester s porque fue un a o en el que simplemente no pod a dejar de ser feliz A mis compa eros y amigos de la universidad que tantas y tantas horas hemos compartido en los ltimos a os por todas las asignaturas y pr cticas que aprobamos juntos y por todos los paseos que de vez en cuando nos hemos dado por el wild side of life Menci n especial merece mi hermanamiga Maribel que en noches de estudio y de fiesta con frio o calor ha estado a mi lado haciendo posible siempre que todo siguiese adelante Agradecer a mi madre Mari Beni por su cari o porque siempre se ha desvivido por mi bienestar y
82. orm better on average than non TextTiling for the MPEG 7 passive infrared and temperature data sources However not the same size of blocks of images were optimal for every technique resulting in an average value of 5 images grouped together for the MPEG 7 source and 8 images for the temperature and passive infrared one This is because temperature and passive infrared values change slower over time the temperature showed by the sensor would remain almost constant if the wearer do not change his location However if we have a source of information which values change quickly over time it would be better to compare the values of adjacent images This could be the case of data from accelerometers and light sensors Figure 2 13 Comparison of blocks of images All images belong to the same event NET 3 Determine a threshold value Images with dissimilarity values higher than the threshold value will be with probability a boundary between different events Some of the techniques to select boundaries are non parametric kapur and parametric Mean 14 4 Remove two event boundaries too close in time to each other Sometimes if we are moving quickly for example riding a bicycle our SenseCam will take pictures that will be very different to each other although they belong to the same event Only one image has to be considered as a boundary But what is the optimal time under which considered boundaries have to be ignored The people fro
83. play an important role in the selection of an appropriate keyframe Figure 2 18 Examples of poor quality SenseCam images The quality measures can be automatically extracted from each image within the collection There are five low level image features that can measure the quality of a given image 19 e Contrast Measure this is the measure of the difference in brightness between the different sections in an image Pictures taken in a place with direct sunlight will have higher contrast than pictures taken in the fog e Color Variance here we calculate the perception of the color richness of the dominant colors in an image e Global Sharpness it measures how sharply focused is an image For this we measure the sharpness based on sections of an image that are in focus e Noise measure it is the random variation of brightness or color information in images The noise is less pronounced in bright tones while it is more appreciable in dark tones It is composed for two elements fluctuations in color and luminance Luminance Noise Chroma Noise Figure 2 19 Noise composition 21 Image Noise e Saliency Measure here we measure the different from the image After the processing take part the result values have to be normalized 20 There are two other values automatically captured by the SenseCam that can help us to predict the quality present in the images taken We can relate to our image the values
84. previous 21 days Although previous 14 days is the window that requires a higher computational cost it is demonstrated that is the optimal one referring to the right amount of previous days to consider However an advantage of the 2h time constrain approach is that the novelty of an event is calculated based on how different it is to all other events and also considering the time of day This is important because if we are sitting in front of the computer in the middle of the night this event is more unique that if we are sitting in front of our computer during working hours The same day of week in the previous 21 days is the method with worst results so a combination of previous 14 days and 2h time constrain approach will be the best way to proceed ds Following these previous three steps event segmentation landmark photo selection and event novelty calculation we finally have our photos segmented into events and ready for the user to review them in a clear and easy way SenseCam Images of day about 7 000 Interactive TITI III MI Il 1 i TM MIME II i Event Segmentation Event Event Comparison within the Multi day Event database Composition of the Browser Qu y BRR OB VL PET TIL deem A Howelty Calculation of Each Ewent Event database containing last n days Events Figure 2 23 Entire process of event segmentation landmark detection and novelty
85. ptation of everything that passes through his point of view A gt Figure 2 3 Example of SenseCam images Unlike traditional digital cameras it doesn t have a display to find and look the taken photos and it is designed to take pictures in a passive way without user interaction It is provided with a wide angle lens fish eye This way we can be sure that most of the details of the wearer view will be recorded by the camera To take the photographs the user can program a timer making the camera take a picture every for example 30 seconds Alternatively a certain event can trigger a photograph to be taken For example a considerable change in the light level in the ambient temperature or a sudden movement would be noticed for one of the SenseCam sensors light intensity light color sensors passive infrared body heat detector temperature sensor or multiple axis accelerometer and a picture will be taken Besides SenseCams have a button to manually take a photo whenever the wearer thinks it s appropriate This way there will not be a detail that could escape from being recorded 1 Figure 2 4 Microsoft SenseCam with transparent plastic case 2 2 2 Data compilation To make a captation as close to reality as possible the SenseCams have a digital fish eye lens camera and several sensors that will record a lot of different data about the wearer situation The sensors are used to trigger pictures w
86. r alu OD 4 j t l weed Choose an image Fatt Sweets Live Sparing 5 5 8 14 Ac dee eti Cheese Group Poultry Fish Dry 2 3 Servings ls m Iz E dE m ls lin du d m fo c F FE g EE gt r lin lie You should not eat that much sweets fats and oils Use them sparingly TA i eed Fue mone hal Ce Eran rl Figure 4 7 Eating images of a day At this point the user reads the screen and finds out that he has to choose an image The image chosen has to be one with incomplete eating information maybe because we never introduced any data on it or because we forgot to introduce something All the information that we need to introduce in all the pictures of a day can be introduced by selecting every image one by one After selecting a certain image the application will show us a larger version of the image selected This way we will see every detail clearly Now we have to select the meal of day and what did we take Thereupon we have to introduce the type of food and the quantity taken The pyramid beside the photograph looks like a normal image but it is an html image We can introduce the food just pressing in the corresponding place of
87. r n Vista Caracteristicas 5 Vista Contenido Contenido de My Images r E Mostrartodo Agrupar por _ Tipo Carpeta de archivos Carpeta de archivos Carpeta de archivos Carpeta de archivos Opciones de configuraci n Figure 4 21 Virtual directory for All My Images 4 6 7 Servings Directorio virtual Visi Cambiar a vista Caracter sticas Explorar Editar permisos Convertir en aplicaci n 24949 E Agregar aplicaci n ltima modificaci n 14 08 2009 8 38 40 Administrar directorio D Actualizar In the first version of our software the servings were presented in the slide as quantities from O to 100 This was ok for testing results but it wasn t easy for a normal user to count servings with accuracy That s why we changed the values of the slide Always thanks to Aiden good advices It now goes from zero to five zb zd Tareas de My Images Agregar directorio virtual Ua Configuraci n avanzada 5 Tests and results In this section we will show with practical examples how useful SenseCams and this interactive software are to analyze dietry habits We will analyze if it guides us correctly to improve our diet and if it is more or less useful than the traditional diet diary Besides we will prove the precision that concept detection techniques have and if this precision is acceptable in practice For this have been wearing a SenseCam fo
88. r one week and at the same time have been writing a traditional diet diary To have an objective point of view every time had to eat asked someone to annotate exactly what food was taken and what the exact quantities were Here we have the results 5 1 Concept detection As we can see in table 2 7 the system misses just the three per cent of the eating images However between all the positives pictures we have more or less half of them that are not real positives That is why we have a very easy procedure to delete all the false positives In the future the quality of the pictures taken by a SenseCam will surely improve as well as the recognition techniques We will have more accurate patterns and the detection will be more and more precise am sure that in a not very far future we will have recognition techniques that not only will tell us if a picture is about eating They will tell us even the type of food that we are taken in a certain picture The most usual failure that the recognition system has is to confuse images from supermarkets or restaurants with eating images This is due to the diversity of colors that both types of images have and because in most supermarket pictures the main character is food Figure 5 1 three false positives eating pictures spj e 5 2 My week with a SenseCam As said before to test how helpful is our software have been wearing a SenseCam for one week and at the
89. recio o ejecute con mayores dimensiones cualquier otra parte de las obras o en general introduzca en ellas cualquier modificaci n que sea beneficiosa a juicio del Ingeniero Director de obras no tendr derecho sin embargo sino a lo que le corresponder a si hubiera realizado la obra con estricta sujeci n a lo proyectado y contratado 12 Las cantidades calculadas para obras accesorias aunque figuren por partida alzada en el presupuesto final general no ser n abonadas sino a los precios de la contrata seg n las condiciones de la misma y los proyectos particulares que para ellas se formen o en su defecto por lo que resulte de su medici n final 13 El contratista queda obligado a abonar al Ingeniero autor del proyecto y director de obras as como a los Ingenieros T cnicos el importe de sus respectivos honorarios facultativos por formaci n del proyecto direcci n t cnica y administraci n en su caso con arreglo a las tarifas y honorarios vigentes 14 Concluida la ejecuci n de la obra ser reconocida por el Ingeniero Director que a tal efecto designe la empresa 15 La garant a definitiva ser del 4 del presupuesto y la provisional del 2 16 La forma de pago ser por certificaciones mensuales de la obra ejecutada de acuerdo con los precios del presupuesto deducida la baja si la hubiera vil 17 La fecha de comienzo de las obras ser a partir de los 15 d as naturales del replanteo oficial de las mismas y la de
90. recio total por el que se compromete a realizar la obra y el tanto por ciento de baja que supone este precio en relaci n con un importe l mite si este se hubiera fijado 4 La obra se realizar bajo la direcci n t cnica de un Ingeniero Superior de Telecomunicaci n auxiliado por el n mero de Ingenieros T cnicos y Programadores que se estime preciso para el desarrollo de la misma 5 Aparte del Ingeniero Director el contratista tendr derecho a contratar al resto del personal pudiendo ceder esta prerrogativa a favor del Ingeniero Director quien no estar obligado a aceptarla 6 El contratista tiene derecho a sacar copias a su costa de los planos pliego de condiciones y presupuestos El Ingeniero autor del proyecto autorizar con su firma las copias solicitadas por el contratista despu s de confrontarlas 7 Se abonar al contratista la obra que realmente ejecute con sujeci n al proyecto que sirvi de base para la contrataci n a las modificaciones autorizadas por la superioridad o a las rdenes que con arreglo a sus facultades le hayan comunicado por escrito al Ingeniero Director de obras siempre que dicha obra se haya ajustado a los preceptos de los pliegos de condiciones con arreglo a los cuales se har n las modificaciones y la valoraci n de las diversas unidades sin que el importe total pueda Vi exceder de los presupuestos aprobados Por consiguiente el n mero de unidades que se consignan en el proyecto o en el pre
91. s amp Swets 24 25 26 27 28 29 30 Use Sparingly 31 1 2 3 4 3 5 Gr verry Time written in diary N2 meal s pictures 58 Breakfast Afternoon snack l l l 59 5 2 6 Day 6 Q eeo 20 4 Ai e x Choose an image Click to get some advice Fats Oils amp Sweets 24 25 26 27 28 29 30 Use Sparingly Oe Forgotten in O out of 3 meals diar EA YH in diary N meal s pictures 60 5 2 7 Day 7 Choose an image Click to get some advice Fats Oils amp Sweets 24 25 26 27 28 29 30 Use Sparingly Su Forgotten in 0 out of 3 meals diar Time written Bro in diary N meal s n CHI TIL 2612 5 2 8 Statistics In this section we are going to present our results in a simple and clear way showing the statistics of the most important points This way we can compare both methods SenseCam and traditional diet diary and decide which one is more useful 5 2 8 1 Annotated quantities In the following graphic we can see the different quantities annotated with both methods and their comparison with the real quantities As we can see the SenseCam method is more accurate than the diary one This is because sometimes we wrote the meals down several hours after eating The SenseCam method is not 100 accurate some
92. same time have been using a food diary To do this test have followed some rules e Wear the SenseCam all the time during the day This way every meal was registered and we can test how well the eating images are distinguished e Always have the diet diary within reach and write down my meals as soon as could or as soon as remembered e Don t fetch SenseCams images until the test week is over e Count on someone else to write down an objective point of view of my meals and the quantities took In the following figure we can see three pictures of the diet diary taken with my SenseCam Figure 5 2 My diet diary To present the results we are going to show with detail three of the seven days that the test lasted We will include the information extracted from the other four days of test in graphics and statistics that will help us to have a general understanding of the results presented In some of the days we happen to have a big amount of pictures so only a representative part of them will be presented in this document These are the legends SC 2 Annotated using SenseCams Br gt Breakfast Ln gt Lunch D gt Annotated using diet diary As gt Afternoon snack Dn gt Dinner R gt Real quantities Em gt Extra meal 53 5 2 1 Day 1 0 ARE Choose an image 7 29 Click to get some advice 3 4 5 z 10 11 12 13 14 15 16 Time written in diary N meal s pictures Breakfa
93. sidered an event The MPEG 7 visual features are color structure color layout scalable color and edge histogram They are calculated making use of the aceToolbox 18 Color Layout Descriptor CLD it is resolution invariant and it is designed to capture the spatial distribution of color in an image or an arbitrary shaped region To extract the color layout descriptor firstly the image is partitioned into 8x8 64 blocks Each block is representated by its average color This result in three 8x8 arrays one for each color component Y luminance Cb Blue difference chrominance components Cr red difference chrominance components Then the DCT transform is applied We now have 3 matrices of coefficients The resulting coefficients are zig zag scanned and the CLD descriptor is formed by only 6 coefficients from the Y DCT matrix and 3 coefficients from each DCT matrix of the two chrominance components The Descriptor is saved as an array of 12 values Finally the remaining coefficients are nonlinearly quantized 10 11 2711 Rep Colorl yer LE Soc j Y Eo eal20 coll poleo calan ds Emm site caris m ao gasses cn J rranstorm E 8 Figure 2 11 The CLD extraction process Steps 10 m a i 1 5 E EL i 1 pri Scalable Color Descriptor SCD this is a
94. st Afternoon SC 5 2 2 Day 2 e cei p ce d Choose an image Click to get some advice ES lun E 3 Ion IN lo e Y H e BS 5 ls un c IS lo Is P Is b Fats Oils amp Sweets Use Sparingly N I N Is Time written in diary pictures 55 Afternoon snack 5 2 3 Day 3 Y x Ad o medo Choose an image Click to vet some advice 1 t D 1 In pa m rt N LJ Iz 2 EN 5 6 7 8 9 10 17 pa m LO N O n Its Fats Oils amp Sweets E tane E c Use Sparingly Forgotten in 1 out of 4 meals diar Time written d AAA in diary WE i iw na he na un m 4 N Ao e N meal s pictures 5 2 4 Day 4 m ms 4 JM fi k Choose an image Click to get some advice Fats 81 5weers Use Sparingly 57 Forgotten in 0 out of 3 meals diar Time written AA in diar N meal s pictures 5 2 5 Day 5 Choose image do 2 Click to get some advice 3 Fats Oil
95. supuesto no podr servirle de fundamento para entablar reclamaciones de ninguna clase salvo en los casos de rescisi n 8 Tanto en las certificaciones de obras como en la liquidaci n final se abonar n los trabajos realizados por el contratista a los precios de ejecuci n material que figuran en el presupuesto para cada unidad de la obra 9 Si excepcionalmente se hubiera ejecutado alg n trabajo que no se ajustase a las condiciones de la contrata pero que sin embargo es admisible a juicio del Ingeniero Director de obras se dar conocimiento a la Direcci n proponiendo a la vez la rebaja de precios que el Ingeniero estime justa y si la Direcci n resolviera aceptar la obra quedar el contratista obligado a conformarse con la rebaja acordada 10 Cuando se juzgue necesario emplear materiales o ejecutar obras que no figuren en el presupuesto de la contrata se evaluar su importe a los precios asignados a otras obras o materiales an logos si los hubiere y cuando no se discutir n entre el Ingeniero Director y el contratista someti ndolos a la aprobaci n de la Direcci n Los nuevos precios convenidos por uno u otro procedimiento se sujetar n siempre al establecido en el punto anterior 11 Cuando el contratista con autorizaci n del Ingeniero Director de obras emplee materiales de calidad m s elevada o de mayores dimensiones de lo estipulado en el proyecto o sustituya una clase de fabricaci n por otra que tenga asignado mayor p
96. t and difficult to judge 24 The concept detection process can be divided in three stages e Supervised learner The detection of concepts is basically the recognition of a pattern Given an image or a part of it we have to measure how similar it is from different patterns which will indicate if a concept is present in the image or not For supervising learning of concepts we use the Support Vector Machine SVM With SVM we view input data as two sets of vectors in an n dimensional space Our method will construct a hyperplane that will maximize the margin between the two data sets The larger this margin is the lower the generalization error of the classifier Input Space Feature Space Figure 2 26 Pattern Classification by an SVM e Visual feature Extraction Here we adopt the codebook novel 24 which represents an image as a distribution over codewords Following 25 we can build this distribution by dividing an image in several regions with rectangular shape To obtain two separates codebook models we will employ two methods in order to extract visual features they measure colored texture o Wiccest features These features utilize natural image statistics to model texture information Distributions of edges describe the texture in an image These features are well suited to detect natural sceneries o Gabor features They are good to detect artificial constructions as they respond to regular patterns in a g
97. t possible to change the interfaces and to make easier for the user to interact with the application In this section we are going to explain the most important steps made in the development of the code the problems faced and their solution and all the different challenges that appeared during all the development process 4 5 1 Main aspx cs The first challenge was to be able to present a calendar initialized in the month when the last picture was taken and with the exact day marked The second challenge was to show all the images kept in the database The images shown are all those with an eating probability higher than 0 4 See 4 3 1 2 Of course there will be some days in the calendar that don t have images maybe because that day the user wasn t wearing the SenseCam or because the camera didn t take any picture where the wearer was eating In that case the application will inform the user that there are no images in the database for the day selected For that a class called calendar control was made Its methods are called from main aspx cs and the calendar control class will return all the data needed for the calendar calendar control will connect to the database 4 5 1 1 Calendar control cs From this class we will call the two stored procedures that we made before eating images and available days First to load the main page the first method in calendar control has to return the days where we have information in a connec
98. t working with an expert in nutrition and sport He can help us with the number of calories in each meal and the amount necessary to have a healthy nutrition so that our Project will be as useful and efficient as possible 1 1 Motivation and goals The motivation of this PFC is to create a tool that can be used as a guide for the user to have a correct nutrition using Sensecams In the first stage we will study the format of the XML file in which the devices store the different data that they collect and we will compare the color histogram to do an ideal segmentation of the events In the second stage we will carry out the concept detection SVM This way we will have a database with information about each of the images We will be able to look up if one image is outdoors indoors a landscape Later there will be a third stage in which we will develop the graphic interface with ASP NET It has to be a graphic interface easy to interact with We will use it both to take information from it and to store new data collected As the fourth stage we will have to work with an expert in nutrition and sport to be able to calculate the calories taken and the ones burnt In the fifth stage our project will be tested Based on the test results we will modify the initial program to fit the needs of real users Motivaci n y objetivos La motivaci n de este PFC es crear una herramienta capaz de guiar al usuario hacia una correcta nutrici n
99. taying in the same environment This is why the variance of the value has to be measured over a predetermined window size If the degree of variance is higher than a certain value it is probable that the user has change the environment maybe going outdoors or moving between rooms Some of the wrong boundaries were due to the time for calibration that the sensor needs once it is activated and due to temperature changes in the same environment a cloud covering the Sun if outdoors the air conditioning turned on off if indoors The light sensor simply measures the intensity of white light present So for example if the wearer is taking an elevator the white light detected would probably decrease reaching its normal value again when the elevator is left behind These changes in the intensity of light will mark a boundary between events ls The most common falsely detected boundaries by light sensor were due to the decrease of light the sensor suffered when the wearer placed some object too close to it and the different light received from the Sun movement Motion of the wearer is calculated by an accelerometer incorporated in the SenseCam The data captured has 3 axes X Y Z and after computing the derivative value 2 2 2 for each axis we use Z to combine the different axis The boundaries falsely determined can be due to some walking around at an airport or due to motion while looking for something in a drawer F
100. ted In the following graphic we can see the percentage of the meals that were immediately annotated in the diet diary and those which weren t Those meals that are not immediately annotated are more likely to be incomplete Meals annotated immediatly in diary Figure 5 6 Meals annotated immediatly 5 3 Advices Our software creates advices based on the quantity and the type of food annotated on each picture These are the advices I got in day 6 gt advice Milk is good for our bones Don t forget it Protems are important Y ou have to take more meat fish or eggs Don t forget a bigger amount of vegetables next time Don t forget a bigger amount of fruit next time Don t forget bread cereals and biscuits next time they are good h dom 2 Figure 5 7 things to improve advices 64 Breakfast You didn t take enough milk this morning Calcmm is friend Fill your glass You can eat more biscuits cereals or bread than You can introduce a piece of fruit m your breakfast Lunch You had milk vell done You can eat bread in vour lunch than that You can take more fruit for lunch you want to You can take more vegetables for lunch if you want to Remember that meat fisch and eggs are very important for your diet Dinner Remember that you can always have some milk derived product as dessert You can eat bread in your dinner than that You can take more fruit for dinner f you want to
101. the actual society is suffering a remarkable evolution in eating habits Nowadays we consume more soft drinks soda and prepared dishes and we also have a very low use of fruits legumes and vegetables That is the reason why we need every aid available to make our eating habits healthy How about having an interactive system in our personal computer that could tell us our goods and mistakes of our diary diet We can use some devices SenseCams It is a small wearable camera It takes approximately 3000 images per day It has an infra red temperature and a light sensor as well as an accelerometer so they can store many types of information a day Figure 1 1 Microsoft SenseCam This device can have many different applications help people with dementia make a diary of a tourist trip or help the users have a well balanced diet analyzing the food that they take We will focus in this last application in the present project With such a large amount of images taken one million of average per year it is important to have an automated process able to segment our images in different events This way it would be much easier for us to find what we are looking for This can be done with different procedures using MPEG 7 descriptor with the SIFT method or with the SURF one Furthermore we have to be able to recognize which of the events are related to the moments in which the user is eating Support Vector Machines We cannot forget abou
102. the data collected s16 Name New relevance score 15 CombMIN minimum of individual rels CombMED median of individual rels CombMAX maximum of individual rels CombSUM of individual rels CombANZ CombSUM num nonzero rels CombMNZ CombSUM x num nonzero rels Table 2 4 fusion algorithms It is empirically determined that Min Max normalization and CombMIN works best If we normalize and combine all data retrieved with our SenseCam we will have the best approach and optimal results with the minimum amount of errors However the process will be slower and more difficult If we fuse data that comes only from sensors sources the process would be much quicker and it would be worse than the fusion of MPEG 7 and sensor data in only a 3 396 15 There are more ways and procedures of segmenting the data collected into events For example we can wear another device to record audio next to our SenseCam However this will not be completely useful due to privacy concerns We can read some studies about how efficient will be the segmentation of events using each source of information separately 19 Data Source Precision Image 21 46 Audio 1 44 Temperature 927 White Light 9 17 Accelerometer 6 16 Table 2 5 Precision of segmentation with different data sources The temperature sensor present in the SenseCam can be used to detect changes in location However it is very sensitive and the values can change even though the wearer is s
103. tion to the database and the execution of available days stored procedure has to be made DATABASE CONNECTION STRING Data Source local Initial Catalog DCU SenseCam Integrated Security Irue SqlConnection con new SqglConnection DATABASE CONNECTION STRING Sq1Command selectCmd new SqglCommand Carolina Available Days con selectCmd CommandType CommandType StoredProcedure selectCmd Parameters Add USER ID SqlDbType Int Value user id con Open sqlDataReader day reader selectCmd ExecuteReader Figure 4 9 Connection with the database to execute first stored procedure Once the calendar is initiated it is time for the user to choose a day in the calendar When a day is selected we need to see all the eating images available for this day To be able to show all images we wrote a method which connects to the database and execute the second stored procedure eating images Besides showing all eating images we will also read and save the data introduced for each of them This is done to be able to show the user what food has he already annotated in each picture 42 All data read from the database will be stored in an ArrayList where each component is the data from one image To help organizing this data we wrote a class represent image Our ArrayList will be composed of this data type Once all data have been retrieved and saved we can represent it as an array of represent_image data type See Fig
104. tware and receive some custom made advices After step one and two we now have all the images in our laptop and all the visual information and data collected by sensors is well organized Now the software created for this project will be launched and due to its friendly and easy interface the user will interact with it without reading any special manual all the information required is right in front of him After introducing the information needed for the software to work some advices will be created regarding the diet that the wearer has had in the days selected See Development The diagram showed below is a continuation of figures 3 1 and 3 2 As we can see in figure 3 5 the user has to introduce some of his diet information 8 using the graphic interface This data will be saved in the database and afterwards it will be read and analyzed to create good dietry advices Finally it will be up to the user to follow them 10 8 4 Introduce data Receive advice Graphic interface Read data Introduce data Figure 3 5 Third step interaction with the user 30 4 Development 4 1 Introduction Now we are going to briefly comment all the technologies that are part of our design and the use that we have done of each of them Also we are going to introduce all the problems presented during the development of the design proposed in 3 Design and how the appropriate solution was found 4 2 Mi
105. ure 4 11 for a representation of how the data is kept and extract Keeping everything well ordered in an array will make it easier to read and the presentation of all information will be quicker 4 5 1 2 Represent_image cs The goal of this class is to be able to save the data read in an ordered way public class represent_image int image ig string image path string meal type int sweets int milk int meat int vegetables int fruit int bread private private private private private private private private private public represent_image int id sweet int mil int mea this image id ig this image path path this meal type meal typ this sweets sweet this milk mil this meat mea this vegetables vegetable t is fruit frui this bread brea public string get path image path int vegetable string path frui string meal_typ int brea image path Replace G8 C NMnueva 11 my SenseCam images http 7 leealhost ATI My images return image path public int get id return image id public public public public public public public string get meal type int int int int in int get sweets return sweets get milk return milk get_meat return meat get_vegetables return vegetables get_fruit return fruit get_bread return bread return meal_type Figure 4 10 Class
106. y selected 4 6 3 Too many images for screen size Sometimes there are days in the calendar that have a big amount of images It was very difficult to see all of them in the screen and sometimes their presentation wasn t organized enough That is why we decided to make them appear in a line of four images and if the number of images is more than four we can just use a scroll bar to visualize all of them To do this in the aspx page where we place the html code we added a different style in the label where the images are placed style overflow scroll This way when the size of images exceed the size of the label a scroll will appear to be able to see the overflow images Figure 4 17 SenseCam images with scroll bar 4 6 4 Error introducing data While the user is introducing the food that he took one particular day some mistakes can be made For example he can select a type of food and forget afterwards to introduce the quantity taken When this happened the software became unstable and a fatal error took place To avoid this type of errors the program checks that the data has been introduced correctly before making further steps To do so we compare the arrays where the type of food and the different quantities are stored lf there are more quantities than types of food the program will reset the quantities already introduced and will inform the user about the mistake asking to insert the data again In the following

Download Pdf Manuals

Related Search

Related Contents

膨脹式ボート特殊基準 - 日本小型船舶検査機構  For shower door Pour porte de douche Para puerta de ducha  DVM Floor Stand-CS6-IM-05451AT00  MaxTech V.34/V.42bis Network Card User Manual  P - Weed Eater World  37_IRLDSP1701_A [Converted]  Behringer Europack UB1202 User's Manual  Gauge install For Subaru Impreza WRX STi  

Copyright © All rights reserved. Failed to retrieve file