Home

Video content analysis & aggression detection system for a train

1. Number Description 1 The file menu Contains Opening video Opening frames saving and loading options reset and exit 2 The main frame where annotations goes 3 The people who are in the scene They can be selected again when desired 4 Objects in the scene They can be selected again when desired 5 The annotations that has been made 6 Deleting an object in the scene after selecting the object from the table 7 Setting the value of the manual aggression level for the current frame 8 Increase manual aggression level 9 Manual aggression level meter 10 Automatic aggression level meter 11 Decrease manual aggression level 65 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation Control panel It will be functional when an XML file is 12 loaded It contains a play button stop forward and rewind frames buttons 13 To add annotations after playing a loaded XML file 14 Undo wrong drawn rectangle 15 Add the current annotation to the database 16 The current annotation 17 The current selected buttons 18 The content of the tabs 19 Overview of the loaded frames The file menu is shown in figure 6 2 Table 6 2 describes the options in the menu File View Help Open video Open Frames Load XML file Save XML file Save Manual levels Load Manual levels Reset Exit Figure 6 2 The file menu M
2. Delete row Figure H 1 Example of mouse over tip After choosing the people type a choice can be made between Expression Relation and Action The syntax is lt People gt lt Expression gt lt People gt lt Relation gt lt People gt lt People gt lt Action gt lt People gt lt Action gt lt Object gt The program will not allow different choices everything that is not allowed to be selected is disabled Note when choosing Relation the second lt People gt have to be already defined before choosing relation e g first you choose 1 A passenger is sitting 2 A hooligan is invading private space of lt passenger you chose in step 1 gt When an annotation line is made the Add button must be clicked to add the annotation to the database the annotation will be shown in the table Click on the desired frame to annotate it Note that you see the current frame number is blue 129 Video content analysis amp aggression detection system for a train environment The Left Right arrows are used to go to the next 5 frames to the right and go back to the first 5 frames to the left When done with annotating the desired frames the XML file can be saved Adding manual levels To analyze the data a manual aggression levels can be stored and loaded again In every frame you choose you can click on the Plus minus buttons to increase decrease the level of the progress bars following that click on Set Va
3. h_makes_sound null h_sound_volume null h_description h_Framenumber 0 h_x 234 h_y 94 h_width 40 h_height 52 h_expression null 8 MAMNATa cuy ASS sins uote tato ect 133 a_action shout a_subject null a_makes_sound null a_sound_volume null a_Framenumber 0 9 MAIN update shouting 30 f 10 MAIN shumans h_id 9 h_is_visible null h_makes_sound null h_sound_volume null h_description white shirt h_Framenumber 1 h_x 181 h_y 75 h_width 40 h_height 69 h_expression fighting 11 MAIN cta e ool aS st soles ao ao ecst null a_action null a_subject null a_makes_sound null a_sound_volume null a_Framenumber 1 12 MAIN update fighting 60 13 MAIN shumans h_id 133 h_is_visible null h_makes_sound null h_sound_volume null h_description h_Framenumber 1 h_x 240 h_y 88 h_width 32 h_height 58 h_expression angry f 14 MAIN update fighting 15 Fighting 85 Sound Played 83 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results 84 Video content analysis amp aggression detection system for a train environment Conclusions amp Recommendations In this chapter the results of the implemented system of the train aggression project will be discussed Next the project will be assessed against the requirements and goals that were defined in the beginning Furth
4. Adding new icons to the GUI of the system should ONLY be done by a Java programmer or by someone who have knowledge of Java and programming languages Icon requirement Size 50x50 pixels Format Gif Location build classes images In general every new icon has to be placed in one of the tabs people expression relation object action and object and when placing a new icon everything has to be copied from an icon of the same type For example adding a new expression has to be done as follow After designing the gif new gif In initComponents 69 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation ExpressionButton X setIcon new Imagelcon getClass getResource images new gif ExpressionButton X setToolTipText NEW ExpressionButton X setBounds X1 X2 50 50 i jDesktopPaneExpression add ExpressionButton X JLayeredPane DEFAULT_LAYER X is the number of the new button we currently have 7 expression buttons so X should be 8 X1 and X2 are the new locations of the button If we compile and run the file we will see the new icon displayed in the expression tab The action listener is auto defined so there is no need to add it We now need to add action to the listener when the user clicks on the new icon In the method actionPerformed ActionE vent e we add if e getSource ExpressionButton X if clicked 0 get
5. circuits and Systems for Video Technology IEEE Transactions on Volume 14 Issue 5 Page s 622 633 Digital Object Identifier 10 1109 TCSVT 2004 826764 May 2004 M Pantic and L J M Rothkrantz Expert system for automatic analysis of facial expressions Image and Vision Computing vol 18 no 11 pp 881 905 2000 Moyer K E 1968 Kinds of aggression and their physiological basis Communications in Behavioral Biology 2 65 87 Ekman P Basic Emotions in T Dalgleish and T Power Eds the Handbook of Cognition and Emotion Pp 45 60 Sussex U K John Wiley amp Sons Ltd 1999 McKenna S J et al Tracking Groups of People Computer Vision and Image Understanding 80 1 42 56 2000 J Russell and J Fernandez Dols The Psychology of Facial Expression New York Cambridge Univ Press 1997 M Pantic L J M Rothkrantz Facial Action Recognition for Facial Expression Analysis from Static Face Images IEEE Transactions on Systems Man and Cybernetics Part B Cybernetics Volume 34 Issue 3 pp 1449 1461 June 2004 Neil Robertson lan Reid A General Method for Human Activity Recognition in Video Computer Vision and Image Understanding 29 June 2006 90 Video content analysis amp aggression detection system for a train environment Reference 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 Jeff
6. saw after the test steps has been completed This is often replaced with a Pass Fail Quite often if a test case fails reference to the defect involved should be listed in this column We created four different test cases these are 1 Threshold for fighting 2 Annotating a hooligan holding a bottle of beer 3 Saving XML file 4 Loading XML file 7 2 7 Testing goals It is important to know why we are testing Below is a summary of the testing goals from the previous chapters To test the user friendliness of the system To test for errors To test the learn ability of the system To test the efficiency amp accessibility of the system 79 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results To test the effectiveness of the system To test the consistency of the system To test the memorability of the system To test the satisfaction of the system To test the functionality of the system To test the system outputs 7 3 Experiment result Before we reveal the testing results we first give a short overview of our test subjects Afterwards we present the expectation and finally the results will be presented 7 3 1 Test subjects For our experiment we had 5 test subjects all of the 5 subjects are students from the MMI group The following table gives a summary of the participant Table 7 2 Test subjects characteristics Am
7. Loading CASAVED_Test xml 104 Video content analysis amp aggression detection system for a train environment JA hooligan is holding beerbottle Add annotation O m KS PASS 105 Video content analysis amp aggression detection system for a train environment 4 Threshold for fighting Initialization a video frames must be opened Annotation has to be made to the system e Finalization None Actions Annotate an aggressive situation where the fighting level goes up Input data None Results O Expected results The fighting level goes up the text Fighting must flash and a sound of police siren must be played o Actual results Project lol x File View Help Current concepts JA hooligan is holding beerbottle A passenger is shouting hooligan is fighting lA passenger is angry Human Begging Fighting Control Shouting Normal Control Panel Add Undo Add annotation b 0 MAIN initial fact f 1 MAIN humans h_id 9 h_is visible null h_makes_sound null h_sound_volume null h_description white shirt h_Framenumber 0 h_x 121 h_y 15 h_width 57 h_height 95 h_expression null 2 MAIN activity a_id 1 a_is_visible null a_object 9 a_action hold a_su
8. Video content analysis amp aggression detection system for a train environment Annotation language for trains This chapter contains a discussion of why we need an annotation language for aggression detection in train s compartments And why we chose the use of CLIPS and a rule base approach Afterwards we continue with a description of the language and its syntax we also discuss the language capabilities and its future prospects Finally an illustrative example on how the language should be used to annotate scenarios in the train compartment is given 4 1 Annotation language One of the goals of the aggression detection project in trains is to annotate the situation on the train compartment To enable the use of context knowledge we have to develop a world model In other words we have to describe all the objects with their characterizing features which play a role in the world of aggression detection in the train compartments therefore an annotation language is needed A good way to describe the world model is using Extensible Markup Language XML 28 or similar languages e g XSL Transformations XSLT 29 Although not everyone has knowledge of XML and we cannot assume that all the users of the system know XML and can describe the world using XML Using speech to describe the world model would be a good choice but the problem here is that every user will describe the world model in a different manner and this will lea
9. objects o_description money objects o_description beggarclothing relation r_relation wear gt assert update begging 70 portaron Moxerejepiioey LA reli defrule begging rule 2 activity a_action is invading private space of gt assert update begging 10 118 Video content analysis amp aggression detection system for a train environment APPENDIX F People tab In the people tab the user can use the following icons to annotate the world model Passenger this icon is used to all the travelers for a train Artist this icon stands for someone who sings act or play music in order to get money from the passengers Beggar someone who ask for money from passengers Conductor the one in charge in the train A conductor checks the tickets Hooligan a football club fan Mostly in groups and they are very violent and aggressive E Police officer Expression tab The expression tab is used to annotate the human facial expression and body language DA Afraid CIG CIG E l O10 l surprised 119 Video content analysis amp aggression detection system for a train environment Giving the middle finger gesture body language In fighting gesture body language Lale P A No ticket gesture body language Actions tab The action tab is used to describe
10. y y z gt gt might be compiled into the following network one input nodes two input nodes terminals fire example 2 The nodes marked x etc test if a fact contains the given data while the nodes marked fire example 3 remember all facts and fire whenever they ve received data from both their left and right inputs To run the network Jess presents new facts to each node at the top of the network as they added to the knowledge base Each node takes input from the top and 98 Video content analysis amp aggression detection system for a train environment sends its output downwards A single input node generally receives a fact from above applies a test to it and if the test passes sends the fact downward to the next node If the test fails the one input nodes simply do nothing The two input nodes have to integrate facts from their left and right inputs and in support of this their behavior must be more complex First note that any facts that reach the top of a two input node could potentially contribute to an activation they pass all tests that can be applied to single facts The two input nodes therefore must remember all facts that are presented to them and attempt to group facts arriving on their left inputs with facts arriving on their right inputs to make up complete activation sets A two input node therefore has a left memory and a right memory It is here in these memories th
11. assert update fighting2 15 116 Video content analysis amp aggression detection system for a train environment p Come cla defrule control rule 1 activity a_action check objects o_description ticket gt assert update control 30 assert update fighting2 20 pora ou Veomicieoil 10 elle defrule control rule 2 activity a_action hold objects o_description ticket gt GISiS Sistem uO clatae el inte 5 leah 7 jomaincoue Veomicwoil 1 lle defrule control rule 3 activity a_action walk objects o_description conductorclothing relation r_relation wear gt assert update control 10 portaron eo 104 wie defrule control rule 4 objects o_description conductorclothing relation r_relation wear gt assert update control 10 Pp Gorila one Veomicieoil 10 reli defrule control rule 5 humans h_expression noticket gt assert update control 30 assert update fighting 10 117 Video content analysis amp aggression detection system for a train environment begging clp defrule begging rule 1 activity a_action hold objects o_description money objects o_description artistclothing relation r_relation wear gt assert update begging 70 p jorealineo mie Mosgeiims 14 cri begging rule 2 activity a_action hold
12. the most appropriate approach since we will be able to use pattern recognizing to classify aggression based on either a priori knowledge or on statistical information extracted from the patterns 5 But because of its simplicity and availability of data we use a rule based system as a first approach It would be nice to know whether Dynamic Bayesian networks will be more accurate and more efficient than using the expert system and a rule based approach that is used in this report It would be clever if we could compare both outputs in one system REFERENCE 47 Sci Tech Encyclopedia Aggression http www answers com topic aggress ion last visited at September 25 2007 48 Agressie in de trein neemt toe Algemeen Dagblad 1 november 2003 49 Spoorvandalisme kost zeker 20 miljoen per jaar http www home nl nieuws binnenlan d artikel 00114122 Last visited at September 25 2007 50 Robert A Baron Deborah R Richardson Human aggression Second edition ISBN 978 0306444586 Plenum Publishers Nov 1993 51 J Russell and J Fernandez Dols The Psychology of Facial Expression New York Cambridge Univ Press 1997 52 Sergios Theodoridis Konstantinos Koutroumbas i Pattern Recognition 3rd Edition ISBN 0123695317 February 2006 53 P Klahr and D Waterman Expert systems techniques tools and applications Addison Wesley Longman Publishing Co Inc Boston MA USA 1986 54 S Fitrian
13. Cognition Thoughtful Apraisal amp Action Outcomes Decision Processes Impulsive Action Figure 2 2 the general aggression model episodic processes They concluded that the current human aggression models coverage on a common set of theoretical assumptions largely because those assumptions have proved useful in organizing the vast research literature and in generating new testable hypotheses concerning the development persistence and change of aggression 2 2 Aggression in train The aggression in train compartments can take many different forms It can be directed towards the interior of the train the passengers conductors and train mechanist The interior can be damaged or painted and sometimes burned or written on it Passengers conductors and train mechanist can be robbed threatened intimidated or physically or mentally hurt The train aggression is causing a decrease in the number of conductors The number of intimidating and threatens towards the conductors is growing In the first 8 months of 2003 the number of aggression cases was more then 15 than a year earlier in the Netherland The statistics are showing that the aggression problems are getting worse in 2001 there were only 6 944 incidents and in 2006 that number was 10 500 The aggression is directed in 75 of the cases towards the conductors and within that 75 1 out of 10 incidents includes 12 Video content analysis amp aggression detection sys
14. Dutch railways NS and the Man Machine Interaction research group at Delft University of Technology started the Aggression detection project The aggression detection project focuses on developing a system that detects human aggression using a multi modal system that fuses sound and video signals recorded in train compartments Image processing techniques are ongoing researches especially the semantic interpretation of images i e the automated recognition of facial expressions or object recognition We want to investigate the fusion between video and audio data to analyse the features that contribute aggression We want to investigate acquire knowledge see how an expert detects aggression To investigate the process of security experts we created an acquisition tool In this thesis we will research the possibility of designing and implementing a system with high usability standards to annotate the actual modal in the train compartments on the other hand we will also implement an rule base expert system to handle the incoming data from the annotating process and the output will give an aggression level Chapter 1 Introduction 1 1 Relevance Surveillance is monitoring the behavior or activities of people Surveillance cameras figure 1 1 are placed in now days almost in every shop school hospital and other public places Aggression detection using the video and audio output stream of surveillance cameras is possible by watchi
15. University of Technology Mekelweg 4 2628 CD Delft the Netherlands E mail md1808 O hotmail com KEYWORDS Aggression detection rule base system expert system train environment video content analysis annotating system ABSTRACT In this paper we investigate the behavior of the human in the train We designed and implemented a system with high usability standards with which users can annotate situations in the train compartments We are particularly interested in aggression The input to the annotating process is images that are captured from video data of recorded scenarios of aggressive and non aggressive situation We implemented a user interface that is connected to a rule based expert system to handle the incoming data from the annotating process The output gives an aggression level and an aggression classification 1 INTRODUCTION Aggression is a behavior that is intended to threaten or inflict physical injuries on another person or organism a broader definition may include such categories as verbal attack discriminatory behavior and economic exploitation 1 To increase passengers safely from aggression in trains the Dutch railways NS and the Man Machine Interaction research group at Delft University of Technology started the Aggression detection project The aggression detection project focuses on developing a system that detects human aggression using a multi modal system that fuses sound and video signal
16. Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains aggressive situation is when a behavior of one of the people on the train that is intended to threaten or inflict physical injuries on another person or organism Annotation has to be done in order to describe the situation on the train compartments This is done to extract the features that have something to do with aggression situations The user will annotate a sequence of frames that came from aggressive scenarios Using the mouse the user has to locate the location of the people involved in the scene and use the keyboard to type in the description of the people involved Next is an example Figure 4 2 of a situation where a beggar is on the train compartment begging for money from a passenger where it end up in a fight A beggar is asking for money A passenger is sitting on a seat 39 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains A beggar is invading private space of the passenger A passenger is angry A passenger is surprised 40 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains A beggar is fighting Figure 4 2 Example of annotating a beggar As we can see th
17. an action the chosen human is performing Checking this icon is used to describe a conductor is checking the tickets Holding this icon can be used when a human is holding an object Hitting Putting legs on this icon should be used when a human is putting his legs on the seat Shouting Sitting Walking Talking Smoking Putting this icon differ from the putting legs on icon this can be used when putting an object like a book bag etc E Ser aLexg 120 Video content analysis amp aggression detection system for a train environment Objects tab The possible objects that can be found in the train compartments are m Baby pram tee Beer bottle Beer can Bag Train tickets Book Cell phone Train seat Cigarette Doll Knife Money Mp3 player Paint 121 Video content analysis amp aggression detection system for a train environment Train table 4 2 5 Relation tab SL a Hitting lv 00100 Shouting at FAN Touching Invading private space Throwing an object at 2 E Note that hitting is also in the action tab but there it is used for hitting an object for example hitting a table and here it is used when hitting another passenger for example 122 Video content analysis amp aggression detection system for a train envi
18. and provided the user of the system with five different classes of Icons in the GUI e People This class contains the people that are in the train compartment e Expression This class contains the emotions and body languages of the people in the train e Actions This class contains the actions or activities people can make in the train e Objects This class contains some static and dynamic objects that are in the train e Relations This class contains relations between people in the train The list of icons is listed in Appendix F 5 3 The Validator component As the Validator component is strongly related to the GUI component We have included it in the GUI package Because it will be easier to validate the user actions when he is interacting with the system by giving immediate feedback on how it should be done when the user try to annotate something incorrect The Validator makes sure that everything that is send to the expert system Jess is correct of syntax and error free 5 4 Reasoning component As mentioned in chapter 4 an expert system will be used for the reasoning We defined five scripts for the aggression levels normal fight shout control and begging Normal The normal level is when everything on the train is normal that is when there is no shouting begging or fighting Fight A fight will occur when 2 or more people are arguing Shout When one or more persons are screaming or talking very loudly the S
19. and Relation on the sequence diagram due to the fact that the diagram will be too large and it will not fit on one page But Objects Activities and Relations also have sets and gets methods and they function the same as setHuman and getHuman Zs ActionPerformed ButtonHumActionPerformed Description mouseClicked mouseReleased CreateSublmage E SetLocation 7 i SetHuman H dorun 7 J getShout setShout print rt UpdateProgressbar Figure 5 8 Sequence diagram annotating process bexccncucescee f 54 Video content analysis amp aggression detection system for a train environment Chapter 5 Design 5 7 2 XML Figure 5 10 shows the sequence diagram for loading an XML file First the method ReadXML Q will read the file The humans objects activities and relations are stored in their corresponding object type of file After setting the XML file data as Java Object type using setHuman and setObjects the method DrawloadedObjects and ListActivities are called to draw the objects and activities on the GUI Note that we did not draw getRelations and getActivity because the figure will be too large Jess is started using the method dorun All the facts that were in the XML file will be send to Jess In the example we only used the fighting aggression level as an example getFight is called to get the level of aggression of fight T
20. graphical model that represents a set of variables and their probabilistic independencies An artificial neural network ANN often just called a neural network NN a mathematical model or computational model based on Biological neural networks It consists of an interconnected group of artificial neurons and process information using a connectionist approach to computation In this project we chose to use a rule base expert system as a reasoning model for the reason that we can represent the situation in a train compartment and put the facts in the knowledge base by annotating the situation 20 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis The expert system needs a model for the representation of the situation in the train compartment Therefore we have to define which objects can be found that play a role in aggression The world is dynamic this means that objects can appear in one certain moment and disappear in another As a result the reasoning mechanism has to use a database to store all objects people and relations in a database and of course the time where objects and people appear or disappear will be stored as well 3 4 An analysis of the train compartment The static context In order to represent the environment to the expert system we must identify the objects within the static environment that can play a role in aggression The static objects are objec
21. gt h Framenumber Ph_expression Figure 5 4 The XML file hierarchy Human attributes The description of the attributes that are in slot name of human activity objects and relation are listed in the tables resp 5 1 5 2 5 3 and 5 4 Table 5 1 Description of the human fact list h_id The unique Id of the human h_is visible Whether the human is visible or not h_makes_sound Making sound or not h_sound_volume The sound level h_description Description of the human entered by the user h_Framenumber In which frames is the human visible h_x The x coordinates of the top left corner h_y The y coordinates of the top left corner h_width Width of the human rectangle to be drawn 49 Video content analysis amp aggression detection system for a train environment Chapter 5 Design h_height Height of the human rectangle to be drawn h_expression The expression of the human Table 5 2 Description of the activity fact list a_id The unique Id of the activity a_is visible Whether the activity is visible or not Default is true a_object The human Id performing this activity a_action The action of the activity a_subject The object Id that comes with the activity a_makes_sound Making sound or not a_sound_volume The sound level a_Framenumber The frame of the acti
22. he she sees someone smoking on the street he she does not think about aggression and that is different when someone is smoking in the train this might lead to aggression situation since smoking is not allowed on the train These examples imply that the environment where the aggressive or non aggressive situation arises is very important In other words the context of the situation is very important The human does not only use the environment to detect aggression the human detect objects such as a knife but simply detecting a knife is not enough e g a person holding a knife is not enough the human identify the activity the person holding the knife the person could be holding a knife to peel an apple which isn t an aggressive situation but holding a knife when the person is angry is a different situation In the examples above we used the word activity and we refer it to peeling an apple this means that the human have to know the activity to detect aggression Also the word angry is used which is an emotion Another example of how human detect aggression is when a human sees a person holding a gun then he does not immediately sees that as an aggressive situation e g a police officer with a gun is not an aggressive situation in fact it is a safe situation since people tend not to do an aggressive acts when a police officer is around Although a passenger or a hooligan holding a gun is referd to be very aggressive sit
23. means that the human have to know the activity to detect aggression Also the word angry is used which is an emotion Another example of how human detect aggression is when a human sees a person holding a gun then he does not immediately sees that as an aggressive situation e g a police officer with a gun is not an aggressive situation in fact it is a safe situation since people tend not to do an aggressive acts when a police officer is around Although a passenger or a hooligan holding a gun is referd to be very aggressive situation This implies that knowing the people plays an important role in the human aggression detection Another important issue is that humans have memory and they can easily memorize the changes in time e g a person was drinking beer ten minutes ago in the train and they saw him shouting at others later they see that person again and sees him walking towards them they immediately remember him and they detect a possible aggression Thus the human reasons about aggressive situations using the knowledge of the environment the people types objects the activity the emotional state of the person doing the act of aggression and finally they use their memory to identify people 46 A computer system needs the same knowledge about the environment to reason about aggression For that reason a world model is needed The context of that world model should be the same set of information the human 19 Video conte
24. they think about the system and from there we analyzed the data and concluded the testing results and the recommendations to improving the system The final result is a system the user can use to annotate a loaded sequence of images captured from a video file After annotating the user will get as a result 4 Video content analysis amp aggression detection system for a train environment Chapter 1 Introduction an aggression level and aggression classification The annotation can be stored and loaded to from an XML file 1 5 Thesis overview The thesis report consists of eight chapters The first chapter gives an introduction about the project and the problem definition that needs to be solved Chapter 2 provides background information about human aggression and especially human aggression on train compartments Chapter 3 defines the problem and lists the requirements Chapter 4 contains an explanation of why we need an annotation language for aggression detection in train s compartments and why we chose the use of CLIPS and a rule base approach Chapter 5 presents the design of system using UML models First the global system design is discussed followed by a detailed overview of the design of the main classes that will be used Finally the user interfaces is discussed Chapter 6 describes the actual implementation of the system An overview of the used software and tools is given and afterwards the implementation of expert sys
25. to some formal grammar A program that produces such trees is called a parser A parse tree is made up of nodes and branches Below figure 4 1 is a linguistic parse tree here representing the English sentence Mike hit the ball Note this is only one possible parse tree for this sentence different kinds of linguistic parse trees exist The parse tree is the entire structure starting from S and ending in each of the leaf nodes Mike hit the ball 37 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains Mike V NP Hit Det N the ball Figure 4 1 Parse tree Mike hit the ball Since we don t need a language that is very complex we limited our syntax so it will be easier to create rules in the expert system lt Det gt lt Noun gt lt Verb gt lt Object gt lt PP gt lt Det gt lt Subject gt A passenger puts legs on the table lt Det gt lt Noun gt lt Verb gt lt Det gt lt Object gt A hooligan holding a knife lt Det gt lt Noun gt lt Verb gt A Conductor walking lt Det gt lt Noun gt lt Verb gt lt Expression gt A beggar is angry lt Det gt lt Noun gt lt Verb gt lt Det gt lt Subject gt A passenger hits a conductor should the user pay attention to The users of the system have to annotate every situation that is related to aggression or it is an aggression situation already An 38
26. verbal behavior intended to harm Aggression can either be directed inward by self mutilation or suicide or directed outwardly at another person There are many things that human aggression has been caused on including broken homes poverty chemical imbalances in the brain toy guns inequality TV violence sexual repression sexual freedom and bad genes 2 Some believe that all of these potential causes have one thing in common unfulfilled human needs and desires Fortunately when most people s needs are not met they do not turn to violence to deal with their frustrations Chapter 2 Background Nevertheless self control sometimes breaks down resulting in aggression ranging from petty theft to murder One of the problems in researching aggression is how to classify the different type of aggression It is an easier task with animals which tend to display stereotyped patterns of violence such as killing to gain food or territory With humans and non human primates classifying aggression becomes more difficult because there is complication of intent Punishment for example represents an especially gray area Should spanking be considered an aggressive act What about capital punishment Indeed almost all acts we consider aggressive have been socially sanctioned by some cultures over the years Moyer 1968 14 presented an early and influential classification of seven different forms of aggression from a biological and evolutiona
27. 0 happiness 26 Classified as 75 disgust 67 disgust 82 disgust 93 disgust 95 disgust 89 disgust 100 happiness 100 happiness 68 anger 83 sadness 70 surprise Average 85 02 We can use a similar table to classify the aggression in this project 15 Video content analysis amp aggression detection system for a train environment Chapter 2 Background 16 Video content analysis amp aggression detection system for a train environment Problem analysis The problem in the train environment is that there are two types of objects static and dynamic In this chapter first we will be discussing some statistical information about the aggression in trains Then we will give a description of the problem of the train environment the static and dynamic objects and activities will be listed Thereafter the requirements will be discussed 3 1 Aggression in the train It is important to know how much aggression occurs in the public transportation and especially in trains The NS year report 12 shows that the passengers who give a score of 7 or higher to the safety in the train are 69 as average over the last 5 years Table 3 1 This means that 31 of the passengers do not feel safe in the train and this is due to aggression Table 3 1 Passengers that give a score of 7 or higher for the safety in the train passenger that give 2002 2003 2004 2005 2006 7 10 or higher for safety So
28. 7 Reasoning Figure 5 5 system main packages The three packages are 1 Reasoning This package contains all the classes to reason about aggression 2 GUI This package contains the user interface and the Validator component 3 XML IO The inputs and outputs of the system XML is used to save and load annotation 5 6 1 GUI package Figure 5 6 shows part of the content of the package GUI We created a Java Object class of each type of the facts we have in the XML file Human Object Activity and Relations As well as for each fact we have a set and a get function this is done so we can easy acquire each fact separately from other Note that we did not draw other parts of the class diagram due to the large size that could not fit in one page 51 Video content analysis amp aggression detection system for a train environment Chapter 5 Design HumanTemplate Figure 5 6 Class diagram of the package GUI ObjectTemplate 52 Video content analysis amp aggression detection system for a train environment Chapter 5 Design 5 6 2 XML IO Package The XML IO component figure 5 7 is where the handling of the XML file is done When loading XML file all the facts that are in the XML file has to be converted back to Java objects and this is done using a binder This package also contains the media panel that is connected with the user interface where the user can play the video of the scenario he she wil
29. IN humans h_id 133 h_is_visible null h_makes_sound null h_sound_volume null h_description h_Framenumber 1 h_x 240 h_y 88 h_width 32 h_height 58 h_expression angry 14 MAIN update fighting 15 Fighting 85 Sound Played PASS 107 Video content analysis amp aggression detection system for a train environment 108 Video content analysis amp aggression detection system for a train environment APPENDIX E Rules aggression clp the general rule engine of aggression level detection This file contains several general definitions that apply to all kinds of aggression The code contains a defclass that is an external address which makes exchange of data between Jess and Java possible The aggression levels that can be detected ar begging fighting control j shouting p normal Upon asserting an action the defclass a Java bean will be updated by asserting the corresponding MS COE defclass aggression RunJess Aggression In the java source the bean is passed to Jess as follows r store AGGRESSION new Aggression r executeCommand bind s fetch AGGRESSION Following we can use the following jess commands f call s getAgr E call s setAgr value and as getters and setters for the bean This will make sure that the globals will keep their values upon issuing a reset A oxeslidcoulc E Mis rost sis
30. Icon null A Stine OE SN HumanButton i setEnabled false ActionButton i setEnabled false ObjectButton i setEnabled false ExpressionButton i setEnabled true RelationButton i setEnabled false jTextArea2 setText jTextArea2 getText is clicked 1 setIcon ExpressionButton 0 getIcon Pressed_Action new h_expression new When clicked disable other tabs Note that we should use instead if the word new the function of the real expression e g crying and we should replace the word new by crying 70 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation h_expression is what will go to the expert system This means that our rule for the new icon should consist of crying e g Rule fighting 101 If a passenger is crying Then update fighting to 20 Then we add the following lines in the fight clp dao ie Op humans h_expression crying gt assert update fighting 20 We are done with inserting this icon in the expression tab The same technique can be used for adding other types of icons 71 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation 12 Video content analysis amp aggression detection system for a train environment Testing amp Experiment results This chapter describes the test plan and presents the
31. JlL UN ciel defrule control rule 1 activity a_action check a_subject ticket gt assert update control 30 orion E esmero I cuele All the script files and rules are located in Appendix E 6 3 GUI amp Screenshots Figure 6 1 shows the user interface on the right we can see what is annotated so far We also can see that people which are currently in the scene are marked with a blue rectangle and also their corresponding button is highlighted with black if we choose another frame the button will not be highlighted We also can see a passenger with a white shirt holding a beer bottle which is drawn in the GUI There are 5 tabs People Expression Relation Actions and Objects These tabs contain People objects relations that can be found on a train Table 6 1 describes the content of the user interface shown in figure 6 1 64 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation i ading private space JA conductor is checking ticket JA conductor is talking A conductor is holding ticket JA conductor is walking lA passenger is afraid lA passenger is surprised JA passenger is happy JA passenger is talking Begging Fighting Control Shouting Normal 14 13 12 Figure 6 1 The user interface Table 6 1 A description of the user interface
32. Play button in the control panel figure H 4 Control Panel a Figure H 4 Control panel play button When clicking on the play button the session will play after 10 seconds and will start from frame 1 and stop on frame 10 the time between the frames is 10 seconds During that period the user can click on the Pause button figure H 5 to pause the scene 131 Video content analysis amp aggression detection system for a train environment Control Panel Figure H 5 Pause button Clicking on the Stop Button will stop the play of the scene and go back to frame 0 After pausing or stopping the scene the Forward and Rewind buttons are available clicking on the forward button will go to the next frame and rewind button will go back to the previous frame Clicking on the button Add annotation will allow you to add annotation in the current scene note that you will have the possibility to choose between all the passengers in the scene 132 Video content analysis amp aggression detection system for a train environment APPENDIX G PAPER VIDEO CONTENT ANALYSIS amp AGGRESSION DETECTION SYSTEM FOR A TRAIN ENVIRONMENT 133 Video content analysis amp aggression detection system for a train environment VIDEO CONTENT ANALYSIS amp AGGRESSION DETECTION SYSTEM FOR A TRAIN ENVIRONMENT Mohannad Ismail Man Machine Interaction Group Faculty of Electrical Engineering Mathematics and Computer Science Delft
33. Recognition ISFER which performs recognition and emotional classification of human facial expression from a still full face image 20 The system consists of two major parts The first one is the ISFER Workbench which forms a framework for hybrid facial feature detection Multiple feature detection techniques are applied in parallel The redundant information is used to define unambiguous face geometry containing no missing or highly inaccurate data The second part of the system is its inference engine called HERCULES which converts low level face geometry into high level facial actions and then this into highest level weighted emotion labels Table 2 1 Distribution of the correct recognition ratio Produced Classified as Produced Classified as Produced Classified as Produced expression expression expression expression 1 2 95 surprise 1 98 sadness 23 17 72 anger 10 17 2 77 anger 4 100 anger 23 26 49 anger 10 25 26 6 98 happiness 3 79 surprise 23 88 anger 10 1 4 5 7 90 fear 7 92 anger 24 17 26 85 anger 9 25 26 1 4 5 93 fear 24417 77 anger 9 17 1 4 7 82 sadness 27 81 surprise 24 26 53 anger 9 1 5 7 96 fear 20 25 26 100 fear 24 92 anger 12 25 26 1 4 79 sadness 20 80 fear 10 16 25 26 75 anger 12 1 5 81 fear 15 25 26 100 sadness 10 17 25 26 79 disgust 16 25 26 1 7 77 sadness 15 100 sadness 9 17 25 26 98 disgust 17 5 7 86 fear 23 17 26 82 anger 12 16 25 26 10
34. Video content analysis amp aggression detection system for a train environment Shout Fight Mohannad Ismail November 2007 G TUDelft Delft University of Technology Faculty of Electrical Engineering Mathematics and Computer Science Mediamatics Man Machine Interaction Video content analysis amp aggression detection system for a train environment Master s Thesis in Media amp Knowledge Engineering Man Machine Interaction Group Faculty of Electrical Engineering Mathematics and Computer Science Delft University of Technology Mohannad Ismail 1233246 November 2007 Il Man Machine Interaction Group Faculty of Electrical Engineering Mathematics and Computer Science Delft University of Technology Mekelweg 4 2628 CD Delft The Netherlands Members of the Supervising Committee Dr drs L J M Rothkrantz Ir H J A M Geers Dr ir C A P G van der Mast Ir Z Yang HI Abstract Video content analysis amp aggression detection system for a train environment Mohannad Ismail student number 1233246 Delft November 2007 Man Machine Interaction Group Members of the Supervising Faculty of Electrical Engineering Mathematics Committee and Computer Science Dr drs L J M Rothkrantz Delft University of Technology Ir H J A M Geers Mekelweg 4 2628 CD Delft The Netherlands Dr ir C A P G van der Mast Ir Z Yang Aggression in trains is increasing annually the aggression is costing the Dut
35. Video content analysis amp aggression detection system for a train environment APPENDIX H User manual Start the application Run the program in NetBeans using F6 Watch a video To watch a video click on the file menu and choose Open video Browse and select the desired video file Open frames Click on the file menu and choose Open frames Browse and select the txt file Note that the filename must be named to a folder containing images in the output folder Load XML file Click on the file menu and choose Load XML file Browse and select the desired XML file Note that the filename must start with SAVED_ lt Name gt Save XML file Click on the file menu and choose Save XML file Browse the desired direction and type in the name and click on OK Save Manual levels Click on the file menu and choose Save manual levels Type the desired name and click on save Load Manual levels Click on the file menu and choose Load manual levels Browse for the file and click open Reset To reset the current session click on the file menu and choose Reset View captured Frames 127 Video content analysis amp aggression detection system for a train environment Choose View View captured frames Annotating Before starting the annotating process frames must be opened When this is done a saved XML file can be loaded to work further on that file or to start a new session Ne
36. When a state of the world matches a rule the rule is said to be fired 5 5 The XML IO component This component contains the inputs and outputs of the system As we mentioned before we will use XML to store the annotations that are made by the user We tried to make the structure of the XML file as readable as possible Figure 5 4 shows the hierarchy of the XML file The attributes under the slot name are different between human activity objects and relations The example shown in figure 5 4 is taken from the human facts list An example of XML file is provided in Appendix G A DTD is the grammar of an XML 33 It is an acronym that stands for Document Type Definition It contains the elements attributes entities and notations used in the XML document The DTD of our XML file is shown below lt xml version 1 0 encoding UTF 8 gt lt ELEMENT fact last fact gt ENT fact slot name gt ENT name PCDATA gt ENT slot value name gt lt ELEMENT value PCDATA gt lt ATTLIST value type CDATA IMPLIED gt 48 Video content analysis amp aggression detection system for a train environment Chapter 5 Design Fact list name gt Human gt Activity gt Objects gt Relations name value type hid gt Integer gt h is visible gt Symbol gt h_ makes sound gt String 2h _sound_volume gt h _ description
37. _visible null a_object 9 a_action hold a_subject 806 a_makes_sound null a_sound_volume null a_Framenumber 0 3 MAIN objects o_id 806 o_description beerbottle o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 4 MAIN objects o_id 100 o_description hooliganclothing o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 5 MAIN relation r_id 1 r_object 9 r_relation wear r_subject 100 r_is_visible null r_makes_sound null x_sound_volume null 103 Video content analysis amp aggression detection system for a train environment 6 MAIN update fighting 10 PASS 2 Saving an XML file Initialization a video frames must be opened Annotation have to be made to the system Finalization None Actions Click on File gt Save XML file Type the file name Input data File name o Results O Expected results the file is saved in C o Actual results Saving file Saved_Test is saved in CA PASS 3 Loading an XML file Initialization a video frames must be opened Finalization None o Actions Click on File gt Load XML file Choose the desired file Input data Choosing the file or typing the name of the file to be loaded Results O Expected results the file is loaded into the system and all the annotation and results must be shown o Actual results O Opening SAVED_Test xml
38. a rule based approach that is used in this report It would be clever if we could compare both outputs in one system 8 2 2 Annotation language The annotation language that was described in Chapter 4 is recent and therefore it is up to date and not too complex By adding more features it can become a language that can be used as a standard language for annotating in a train environment New features could be e Adding and in the sentence form e g the passenger is sitting on a seat and holding a cell phone the current language can only handle the passenger is sitting on a seat and the passenger is holding a cell phone 87 Video content analysis amp aggression detection system for a train environment Chapter 8 Conclusions amp Recommendations e Make distinguish between male and female by adding a gender slot to the human template e Add objects belong to passenger This way we can define a theft when someone pick up an object that belong to other e Grouping people by adding this feature it will be easier to detect which people can be seen as one group and here we can define a group of hooligan e g or a group of passengers that are fighting with one person which is an aggressive situation e Grouping objects This feature can be used along with the belong feature It will be easier to detect theft when it occurs since we know which objects belong to one passenger 88 Video con
39. aar cijfers NS http www ns nl servlet Satellite cid 1 1710104944328 pagename www n s nl 2FPage 2FArtikelPage_www ns nl amp p 1171010494432 amp lang nl amp c Page Last visited at September 25 2007 Centraal Bureau voor de Statistiek Thema veiligheid verkeer www cbs nl Last visited at September 25 2007 Moyer KE Kinds of aggression and their physiological basis Communications in Behavioral Biology 2A 65 87 1968 Reference 15 16 17 18 19 20 21 22 23 24 25 26 Behar D J Hunt A Ricciuti D Stoff and B Vitiello Subtyping Aggression in Children and Adolescents The Journal of Neuropsychiatry amp Clinical Neurosciences 2 1990 189 192 7 Dec 2006 Crews D N Greenberg and M Scott Role of the Amygdala in the Reproductive and Aggressive Behavior of the Lizard Physiology amp Behavior 32 147 151 1984 Craig A Anderson amp L Rowell Huesmann Human Aggression A Social Cognitive View in THE SAGE HANDBOOK OF SOCIAL PSYCHOLOGY 296 323 Michael A Hogg amp Joel Cooper eds 2003 Hisashi Miyamori Shun ichi Iisaku Video Annotation for Content based Retrieval using Human Behavior Analysis and Domain Knowledge Automatic Face and Gesture Recognition 2000 Page s 320 325 Proceedings Fourth IEEE International Conference on 28 30 March 2000 Dorado A Calic J Izquierdo E A rule based video annotation system
40. alk RL walk LR walk RL Figure 3 2 Exemplar set for walking High level behavior recognition is achieved by computing the likelihood that a set of predefined HMMs explains the current action sequence Thus human actions and behavior are represented using a hierarchy of abstraction from person centered actions to actions with spatio temporal context to action sequences and finally general behaviors While the upper levels all use Bayesian networks and belief propagation the lowest level uses non parametric sampling from a previously learned database of actions The combined method represents a general framework for human behavior modeling The use of body language is not standardized globally but locally such as to a particular country continent or region The use of body language is one of the most variable forms of expression in the world just as spoken and written languages are 27 Take insults for example forms of communication that say in your face jerk Aussies citizens of Australia often extend the thumb as if p gt to say yes except as if to mean to say you idiot Meanwhile in North America and Europe they give offensive people the middle finger because it looks like a certain male body part when formed with our hands In other cultures it may insult people with the finger thumb zero sign A OK a v shape formed with the first two fingers after the thumb called the victory sign or
41. ame gt lt value type SYMBOL gt null lt value gt 124 Video content analysis amp aggression detection system for a train environment lt slot gt lt slot gt lt name gt o_sound_volume lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt o_framenumber lt name gt lt value type INTEGER gt 0 lt value gt lt slot gt lt slot gt lt name gt o_endframe lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt fact gt lt fact gt lt name gt MAIN relation lt name gt lt slot gt lt name gt r_id lt name gt lt value type INTEGER gt 1 lt value gt lt slot gt lt slot gt lt name gt r_object lt name gt lt value type INTEGER gt 9 lt value gt lt slot gt lt slot gt lt name gt r_relation lt name gt lt value type SYMBOL gt wear lt value gt lt slot gt lt slot gt lt name gt r_subject lt name gt lt value type INTEGER gt 100 lt value gt lt slot gt lt slot gt lt name gt r_is_visible lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt r_makes_sound lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt r_sound_volume lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt fact gt lt fact list gt 125 Video content analysis amp aggression detection system for a train environment 126
42. ased expert system to handle the incoming data from the annotating process The output gives an aggression level and an aggression classification We designed and implemented a demonstration and have tested the system The model implementation and test results will be described in this thesis IV Acknowledgements First and foremost I would like to thank my thesis supervisors Leon Rothkrantz and Zhenke Yang for their inspirations that they gave me during my thesis project Their wide knowledge and great foresight have provided a good basis for the present thesis Their knowledge and instructions helped me to conquer a lot of challenges in the process of my research Furthermore I would like to thank the other PhD and MSc students and other people from the university who helped me during the project and gave me useful advices during my thesis project Last but certainly not least I would like to thank my parents my brother and my fianc e for supporting me day and night and even in the most difficult times Table of content ABSTRACT innata rios cdas innata IV ACKNOWLEDGEMENTS orocisccszsstesseccevessesaciivssessesessiasssecoseondeas sosaecbeeesseagozedes sasvsdecesessdoscocedasasedsseecabocosabees Vv TABLE OFCONTEN Ero eae ose esetei scoe duscesedavscdsaseasesedevetensbucpedeveeuaeeveae VI LIST OF FIGURES scsvssesdescscascsesesesscessactecicesiccedsebasvecsddelinveedsbedckedesscsaceutessccndevaceusctucebusascssucsdeccsste
43. at the inefficiency described above is avoided A convenient distinction is to divide the network into two logical components the single input nodes comprise the pattern network while the two input nodes make up the join network There are two simple optimizations that can make Rete even better The first is to share nodes in the pattern network In the network above there are five nodes across the top although only three are distinct The second is by modifying the network to share these nodes across the two rules the arrows coming out of the top of the x and y nodes are outputs fire example 2 fire example 3 But that s not all the redundancy in the original network Now we see that there is one joined node that is performing exactly the same function integrating x y pairs in both rules and we can share that also 99 Video content analysis amp aggression detection system for a train environment fire example 2 fire example 3 The pattern and joined networks are collectively only half the size they were originally This kind of sharing comes up very frequently in real systems and is a significant performance booster We can see the amount of sharing in a Jess network by using the watch compilations command When a rule is compiled and this command has been previously executed Jess prints a string of characters something like this which is the actual output from compiling rule example 2 above exa
44. at we didn t annotate all the passengers because they are not involved in this situation 4 5 Future of the language The number of sentences that can be made using this structure is very large due to the fact that we have the choice between different relations and activities between objects and people We believe this is enough to annotate most of the common actions that can be made by people in the train compartments because we covered a variety of scenarios which were aggressive and non aggressive scenarios and we were able to annotate almost everything that is related to aggression However it is very easy to extend this language in the future Because the use of XML XML is easy extendable language since it is readable and well structured The language can also be seen as multimodal language It can be extended in the future so it can split into Spoken text and picture annotations The picture annotation can also be split into Features e g movement and semantic interpretation running away walking figure 4 3 41 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains Multimodal grammar sg Picture annotations Spoken text Features Semantic Movement interpretation Sound Walking Shouting Figure 4 3 overview of the multimodal language 42 Video content analysis amp aggression det
45. bject 806 a_makes_sound null a_sound_volume null a_Framenumber 0 3 MAIN objects o_id 806 o_description beerbottle o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 4 MAIN objects o_id 100 o_description hooliganclothing o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 106 Video content analysis amp aggression detection system for a train environment 5 MAIN relation r_id 1 r_object 9 r_relation wear r_subject 100 r_is_visible null r_makes_sound null r_sound_volume null 6 MAIN update fighting 10 7 MAIN humans h_id 133 h_is_visible null h_makes_sound null h_sound_volume null h_description h_Framenumber 0 h_x 234 h_y 94 h_width 40 h_height 52 h_expression null 8 MAIN activity a_id 133 a_is_visible null a_object 133 a_action shout a_subject null a_makes_sound null a_sound_volume null a_Framenumber 0 9 MAIN update shouting 30 10 MAIN humans h_id 9 h_is_visible null h_makes_sound null h_sound_volume null h_description white shirt h_Framenumber 1 h_x 181 h_y 75 h_width 40 h_height 69 h_expression fighting f 11 MAIN activity a_id 133 a_is visible null a_object null a_action null a_subject null a_makes_sound null a_sound_volume null a_Framenumber 1 12 MAIN update fighting 60 13 MA
46. c as indices have drawbacks that tracking errors of a certain object due to occlusions cause recognition failures and that representation by position information essentially has limited number of recognizable events in the retrieval The authors approach incorporates human behavior analysis and specific domain knowledge with conventional methods to develop integrated reasoning module for richer expressiveness of events and robust recognition Based on the proposed method they implemented content based retrieval system which can identify several actions on real tennis video 2 3 2 A Rule Based Video Annotation System 13 Video content analysis amp aggression detection system for a train environment Chapter 2 Background A generic system for automatic annotation of videos is introduced 19 The proposed approach is based on the premise that the rules needed to infer a set of high level concepts from low level descriptors cannot be defined a priori Rather knowledge embedded in the database and interaction with an expert user is exploited to enable system learning Underpinning the system at the implementation level is pre annotated data that dynamically creates signification links between a set of low level features extracted directly from the video dataset and high level semantic concepts defined in the lexicon The lexicon may consist of words icons or any set of symbols that convey the meaning to the user Thus the lexicon is conti
47. cebasseses VIII LIST OF TABLES sensestesscscescvesseseses cc iscccseasesecsds secs veesvecocscsescacvesevesscs ii Ix 1 INTRODUCTION caian aeteusitisscecabiasteuetescatssseccsadasecedeasasasdceaseazebeleasacacsesssvassenaaezacateieds 1 LARECEVANCE de deh te at Asal 2 1 2 PROBLEM DEFINITION LAIA A a AE A A E EENE OO TAMPA DA A cal ot tle Bi a es Bea ae 4 LS THESIS OVERVIEW a litis 3 Z BACKGROUND 0 0000 cada 7 21 HUMAN AGGRESSION da iia 7 2 2 AGGRESSIONIN TRAIN hd lia 12 2 3 THE CURRENT STATE OF THE ART ccsecccccessssccecssssceseccessccseccessscescccessssssecesssesssesesseeeseseeseeesscsenseeeeess 13 3 PROBLEM ANALYSIS ovscsacssccscasicceccessieaduntiesaissosescessectesavacesossacdscccseecssedoodicssdecenssbssscssoosdedesseessosedesbaues 17 3 1 AGGRESSION IN THE TRAIN ccceeccccesseesceeecesccececccsssceeccesssssscccssssseccssessessceesessseececsseseeeeeesseeeeecenaaeees 17 3 2 AGGRESSION DETECTION BY HUMANS cccccsesssceccesssccecccsssscecccsesssseccsssssseccesessseeeeeessseseeeessuseeeeessaaeeees 18 3 3 AN ANALYSIS OF THE TRAIN COMPARTMENT cccescscecccsssscecccessssscecesessecccessssseeeeeessseseeesesueeeeeessaaeees 20 3 4 AN ANALYSIS OF THE TRAIN COMPARTMENT THE STATIC CONTEXT cccccccssesssccceesssseeeecesseeeecessaeeees 21 3 5 AN ANALYSIS OF THE TRAIN COMPARTMENT THE DYNAMIC CONTEXT ccccessccceeesseceeeessseeeeeeesaeeees 21 3 6 AN ANALYSIS OF THE TRAIN COMPARTMENT ACTIVITIES amp BEHAVIORS cccccc
48. cess ssssessssseseesssssstetsstetstesestesestesseersseses 54 Figure 5 9 Progress bar updating the Control and begging levelS oooonionnoninnnnnnc mommmm 55 Figure 5 10 Sequence diagram Loading XML fil8 oooooconcnncnnncnccnnonocncononononncnncnnonnnonncnnos 56 Figure 5 11 Flowchart of the annotating Process csesceseseeecesreneeeseeeeeeeeeeneeeneeaneeses 57 Figure 5 12 global usage of the System 0 0 aa A AT TA EE AE 58 Figure 6 1 The user interface cs ceeescscsssesececceececesceaeceseeseceeceseeaeeeseeaeeesseseeneeesteanenses 65 Figure 6 22 The file Menu visti cccszceee dd dagen E E fata niente 66 Figure 6 3 The fighting level is above 80 The text fighting flash white and red 67 Figure 7 1 Testing in the MMI lab the observer right the tester left o ononnoninnnn 75 Figure 7 1 Fighting level is flashing ces cseeeesscsecesseeeeseeseeeecesecneeeseeaeeeseeaeeneeeseeaneeaes 82 Vill List of tables Table 2 1 Table 3 1 Table 3 2 Table 3 3 Table 3 4 Table 3 5 Table 3 6 Table 3 7 Table 4 1 Table 5 1 Table 5 2 Table 5 3 Table 5 4 Table 6 1 Table 7 1 Table 7 2 Table 7 3 Distribution of the correct recognition ratio o oonnccnncnncinccnnnanccnnconnannconnoncconcnnnonos 15 Passengers that give a score of 7 or higher for the safety in the train 17 unsafely feeling in public transportati0N coooconcnnconccnoconcnncnnonancconnnnnannnonca
49. ch railroads NS a lot of money due to destroying or drawing graffiti on the interior of the train On the other hand the aggression towards the train conductors is causing conductors to quit their job which is not what the NS wants due to the shortage they have in conductors Passengers are also not feeling safe on the train Current video surveillance systems have limited intelligence necessitating the employment of human operators who are able to interpret the images This job is tedious and has a low efficiency Only a small fraction of the image stream contains interesting information At Delft University of Technology there is an ongoing project on aggression detection in trains by multiple single modality devices that capture data from different modalities hence multimodal Using video and sound input an intelligent system has to be developed that is able to make a context sensitive interpretation of human behavior on a train The aim is to detect aggression as it is about to happen In this thesis we investigate the behavior of the human in the train We designed and implemented a system with high usability standards with which users can annotate situations in the train compartments We are particularly interested in aggression The input to the annotating process is images that are captured from video data of recorded scenarios of aggressive and non aggressive situation We implemented a user interface that is connected to a rule b
50. cial safety in trains 63 66 69 71 74 In addition table 3 2 source the CBS 13 shows that about 7 2 of the total population in The Netherland do not feel safe in public transportation once again due to aggression that is made Chapter 3 Problem analysis Table 3 2 unsafely feeling in public transportation Subject Feels Feels occasionally often Unsafely feelings situations unsafely unsafely Public Public In own On the In transport places house street the mall Netherland 7 2 13 5 3 5 9 4 TT 21 4 8 These numbers show us that people do not like to travel by trains or other public transportations However not everyone has any other option and from the total population we can see in table 3 3 that the average distance traveled per person per day using the train is about 2 45 km Table 3 3 Average travel by train Subject Travel per Distance Travel distance person per per Travel per person per day day Number 0 06 42 65 km 2 45 km 3 2 Aggression detection by humans To understand what is needed to detect aggression automatically it is important to understand how human detect an aggressive situation Humans are very well able to detect aggression in verbal expressions The problem here is that speech recognition techniques are still ongoing researches and there is no technique available that provides enough accuracy to depend on given
51. components and of the profile contour The aggression detection project is a large project it is divided in sub projects and the automated facial recognition part is not part of this sub project However because we want to take the next step in this project and take a look at what we will do when we have the emotion recognition ready we will use emotions as if we have them We used only five of Ekman six basic emotions which are anger fear joy sadness and surprise We didn t use disgust since it s not common on a train compartment Body language also plays a major and important role in aggression detection Body language is a term for communication using body movements or gestures instead of or in addition to sounds verbal language or other communications Neil Robertson and Ian Reid 26 developed a system for human behavior recognition in video sequences Actions are described by a feature vector comprising both trajectory information position and velocity and a set of local motion descriptors Action recognition is achieved via probabilistic search of image feature databases representing previously seen actions Hidden Markov Models HMM which encode scene rules are used to smooth sequences of actions 24 Video content analysis amp aggression detection system for a train environment P7 tonguedip attachment PT tongue lip attachment Chapter 3 Problem analysis walk LR walk LR walk away Walk away walk RL w
52. cssessssseeesssseeeeeeesseeeees 23 3 7 AN ANALYSIS OF THE SYSTEM NEEDED cccccccccccsssssceccesssscecccsssssscccsssesssccessssseseeessseeeseesssueseceessaaeeees 26 3 8 REQUIREMENTS 220s cscuet cooks badbaee A A A adele E 28 4 ANNOTATION LANGUAGE FOR TRAINS uuu cccccccssscccccessscccccessssccccccensccccscensccccssensccccesesssssccseess 33 4 1 ANNOTATION LANGUAGE ccccccccssssssecccsssscecccssscscccsssssscccssssussecssessssscesesseesceessassseeesssusesecesaeseceessaeaees 33 AD REASONING MODE Is 33 035 50025522555 55 a8 E died aiii 35 AB LANGUAGE SYNTAX NO 37 AVA WHA TT OVANNOTATE iasete cacosves Wee tecenia ce soeeshan is A euccie venta AIA E A E E SO 38 4 5 FUTURE OF THE LANGUAGE ca A oa 41 VI S DESIGN ONO 43 SA ARCHITECTURE cdas tido toda eto da dla ld db dot dol ll dr ob ll cold of 43 5 2 THE GUICOMPONENT ii ida 44 5 3 THE VALIDATOR COMPONENT operire i EE e E ER E EEEN RE EE EE E 45 5 4 REASONING COMPONENT ii da 45 95 THE XME 1O COMPONENT e AO E KEE EEN A A a 48 5 6 CLASS DIAGRAMS ir A EE E EET E EEEE EAE E seracts tives ia 50 5 7 DYNAMIC BEHAVIOR rna E E E A EA E 5 8 PROGRAM FLOW AND GLOBAL USAGE 6 IMPLEMENTATION rassssecavscssnedesdicssccesssceszavessodssensscnsdaseosavagesoaseasscacsbeces cecoosacesvedeuotbesbeebeeadesoeesvesedazeesbeuss 59 6 1 TOOLS amp SOFTWARE PREE EE AETA EE E E A EAA E IE ETE A EAE E 59 6 2 IMPLEMENTATION OF THE EXPERT SYSTEM cccccsssssseccessscscccssessceecesessseecesessesceeesssee
53. cts were able to accomplish all the tasks during the test The average grade that was given by the test subjects about the overall system was 7 8 The word Clear was the most mentioned characteristic of the system Table 7 3 provides the top three things the test subject liked or disliked about the system Table 7 3 Top three characteristics 3 characteristics 3 best things about the 3 things disliked that describe the system about the system system 1 Clear Annotation goes quick Sometimes slow once you know the icons 2 Easy to use The aggression level that No Voices in the was given as a result was videos close to the real scenario 3 Nice Icons Too many help windows 8l Video content analysis amp aggression detection system for a train environment Chapter 7 Testing amp Experiment results All the four test cases we passed Below is the output of the test case Threshold for fighting The complete test case results can be found in appendix D e Initialization video frames must be opened Annotation has to be made to the system e Finalization None e Actions Annotate an aggressive situation where the fighting level goes up e Input data None e Results o Expected results The fighting level goes up the text Fighting must flash and a sound of police siren must be played o Actual results Project 0 xj File View Help Current concepts JA hooligan is
54. d to endless amount of possibilities a user can express Chapter 4 Annotation language for trains the model Therefore using speech is too expressive for such a task and is a bad choice after all The use of Object Oriented OO to describe the world model is also a good choice but if we use OO then it will not be easy to define new objects And this means that OO is too restrictive for this task Using logic is also an option Logic programming is like functional programming a declarative way of composing programs In brief declarative programming is much more concerned about what should be computed and much less with how something should be done Moreover an important issue in logic programming is not only to make certain that the syntax of the code is right but also that the semantics exactly correspond to the semantics of the problem whose solution is desired Table 4 1 summarizes the strong and weak points between the possibilities for the annotation language Table 4 1 The different alternatives weak strong points Approach Problem Syntax Clear declarative Flexible solving free code XML Speech 00 Logic The above table shows that XML and Logic have the highest score both with five And this leads us to a clear choice which is a mix between logic and XML approach Logic will be used as a purely declarative representation la
55. ding the manually edited levels form a text file The user should be able to set the aggression level manually Load and saving the manual set levels 3 8 1 End users It is important to identify the end users that will use the system After making interviews with the supervisors and studying the requirements of the system we found three user groups that will use the system these are Computer science or related area students with basic knowledge of programming Java and expert systems People with a personal interest in artificial intelligence that have at least basic programming skills Researchers who will continue working on the project of aggression detection in a train environment To design a system that will be useful to these three groups of people it is crucial that a flexible system is built that can be modified to suit personal preferences As these groups are likely to use the system for different reasons and with different background knowledge it is important that the requirements for all groups are incorporated in the system 3 8 2 Functional requirements Functional requirements define the internal work of the system or in other words what the system will be able to do We created a list of these requirements Giving a list of annotations the system should automatically classify the aggression Giving a list of annotations the system should automatically return an aggression level The us
56. e analyzed the actions that can be made on a train and we found Sitting smoking talking walking hitting checking tickets putting holding shouting and putting legs on the seat Combinations between these actions and objects that are found aggression situation can be created like A hooligan is holding a knife lt very unsafe situation gt A hooligan is holding paint lt damaging the interior gt A passenger is smoking a cigarette lt may cause a fight gt Etc Emotions also play an important role in aggression detection Emotion in its most general definition is a psychophysical process that arises spontaneously rather than through conscious effort and evokes either a positive or negative psychological response and physical expressions often involuntary related to feelings perceptions or beliefs about elements objects or relations between them in reality or in the imagination 24 e g Intimidating aggression Anger While Aggression Fear We need human emotions in the system to reason more accurately about aggression Ekman 18 categorized 6 basic emotions anger disgust fear joy sadness and surprise Automatic recognition is rapidly becoming an area of intense interest in the research field of machine vision Several projects are trying to develop the best possible rate of recognition M Panic and L M Rothkrantz 25 presented an automated system that they developed to recognize facial gestures in
57. each frame 5 31 Save XML file 32 Close the program 33 Start application 34 Open frames using the file 01_associale_hooligan txt 35 Load the saved XML file 36 View captured images 37 Continue annotating staring from frame 6 38 Save XML file 39 Exit Thank you for your time please proceed with the debriefing form 94 Video content analysis amp aggression detection system for a train environment APPENDIX B 95 Video content analysis amp aggression detection system for a train environment 96 Video content analysis amp aggression detection system for a train environment APPENDIX C The Rete Algorithm Jess is a rule based expert system shell In the simplest terms this means that Jess s purpose it to continuously apply a set of if then statements rules to a set of data the knowledge base The typical expert system has a fixed set of rules while the knowledge base changes continuously However it is an empirical fact that in most expert systems much of the knowledge base is also fairly fixed from one rule operation to the next Although new facts arrive and old ones are removed at all times the percentage of facts that change per unit time is generally fairly small For this reason the obvious implementation for the expert system shell is very inefficient This obvious implementation would be to keep a list of the rules and continuously cycle through the list checking each one s lef
58. ecified by the annotations Playing a video file of the scenario to be annotated is also the work of the XML IO component Figure 1 System architecture showing the different components and how they are related 7 REASONING We defined five scripts for the aggression levels normal fight shout control and begging Normal The normal level is when everything on the train is normal that is when there is no shouting begging or fighting Fight A fight will occur when 2 or more people are arguing Shout When one or more persons are screaming or talking very loudly the Shout level will occur Control When the conductor is checking the tickets this will be the result Begging When a beggar is asking for money begging will be the result The reasoning system works with numbers the higher the number the most likely it is the correct case Normal Fighting Shouting Begging or Control Figure 2 shows how the reasoning system works Add Facts Begging Fighting Control Shouting Normal Figure 2 The reasoning system 8 IMPLEMENTATION The scripts have rules when the user is annotating he she will assert facts into the Jess database and from there a result will be given Those rules are similar to CLIPS rules There is also a script defined which provides the methods that are used in the five scripts described above The below rule will add 60 to the fighting level when a beggar is holdin
59. ection system for a train environment Design This chapter presents the design of the video content analysis amp aggression detection system for a train environment First the architecture of the system is given followed by a detailed overview of the reasoning model Then we will discuss the inputs and outputs of the system Subsequently the user interface is discussed Finally the program flow is provided 5 1 Architecture The system has four main components figure 5 1 1 GUI 2 Validator 3 Reasoning 4 XML IO The GUI allows the user to interact with the system All the user commands will go through the GUI The user interface contains facilities for the user to perform all the actions that were shown in the use case diagram in chapter 3 The Validator validates the annotations resulting from user actions Only valid annotations are allowed to the reasoning component invalid annotations are rejected with a message The Validator sends the valid annotations as facts into to the reasoning part where Jess will reason about the aggression and send the output back to the GUI where the output is displayed The XML IO is used for the saving and loading the XML file It is also used for generating JPEG images of the locations of the people or objects as specified by the annotations Playing a Chapter 5 Design video file of the scenario to be annotated is also the work of the XML IO component Figure 5 1 System architec
60. ed in chapter 5 is very common and reasonable for such a system Most of the requirements have been fulfilled The user interface is designed to guide the user through the annotation process giving feedback whenever possible and disabling options that is not logical Many help tips is been implemented in the system so that the user can work efficiently and minimize errors The user interface was designed to be simple intuitive and visually appealing The developed system is flexible new rules can be easy added by editing the CLP file Also new icons of people expressions relations actions and objects can be added to the interface by modifying the GUI file 8 1 3 Design of the reasoning algorithm An important part of the system the reasoning algorithm of the Video content analyze amp aggression detection system for a train environment Based on the annotation that is done by the user an output of aggression level is generated by the expert system This requirement is also fulfilled The developed reasoning model is implemented as a rule based system Inserting facts to the database will return a certain level of aggression and classification between fighting shouting control normal and begging The reasoning model has been tested extensively and the results were quantified which means that the reasoning model is quite accurate 8 1 4 Implementation of the system Based on the design the system was developed incrementally Firs
61. enu option Description Open video A file browser will be opened to locate the video file the user wants to see before starting to annotate Open Frames A file browser will be opened to locate the text file that contains the names of the images frames the user wants to load into the system Load XML file Using a file browser the user can locate the desired XML file to load into the system Save XML file Using a file browser the user can locate the 66 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation desired location to save the XML file after typing the name of the file Using a file browser the user can locate the Save manual levels desired location to save the file that contains manual set levels Using a file browser the user can locate the Load manual levels location of the file to load into the system Reset Reset the current annotating session Exit Exit application Saving and loading manual levels are implemented so it will be easier for the user to compare what he she thinks the aggression should be in a certain situation comparing with what the system gives as output Loading and playing annotations The user can load an annotation file and plays it When playing the file the system will automatically use the loaded XML file to show what was annotated on every frame and wha
62. er should be able to draw a rectangle of the location of the people in the train compartment The drawn rectangles have to generate JPEG pictures The user should be able to see all the generated pictures 29 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis The user should be able to undo a wrong drawn rectangle The system should be able to play a loaded XML file The system should allow to pause stop forward or rewind the demo The system should be able to load 10 frames of a video sequence The system should show the aggression level on every frame individually The user should be able to reset everything 3 8 3 Non functional requirements Non functional requirements are requirements which specify criteria that can be used to judge the operation of a system rather than specific behaviors The interface should be intuitive and easy to use The programming language should be OS independent and preferably in Java The reasoning should be done using an expert system preferably using Jess The aggression classification should be the same as the expected classification within the used scenario The level of aggression should be reasonable and close to the expected values within the used scenario The system should be error free The system should be efficient in use The system should be of a high performance 30
63. eraction research group at Delft University of Technology started a project of aggression detection in 2005 The current human objects tracking and recognition techniques are not very accurate and fully automatic in the sense that one can depend and trust their output Therefore we need a system to annotate the situation on a train compartment Even if those techniques are trustable and accurate there are still some questions What will we do with all the processed data from the multimodal devices How to annotate a certain situation in a train compartment What to annotate What is the aggression level 1 3 Goals The main goal for this project is to design and implement a semi automatic annotation system for a train environment A secondary goal is to design and implement an expert system to detect the level of aggression after annotating a certain situation in a train compartment A summary of the goals is listed below Research the requirements for the system Design annotation system for a train environment Design the user interface Implement an annotation system for a train environment Implement an expert system to detect the aggression level Test the usability of the system Evaluate the system 3 Video content analysis amp aggression detection system for a train environment Chapter 1 Introduction This leads us to formulate the main goal of the project To implement an annotation system for
64. ermore recommendations for future work will be given 8 1 Conclusion This thesis report presents the Video content analyze amp aggression detection system for a train environment The architecture the reasoning system and the annotation language was the main focus of this project The resulting system is a system that makes it possible for the user to e Save an annotation on XML file for a certain scene e See what aggression classification is best fitted in a certain scene based on predefined rules e The level of aggression according to the situation e Load a saved session and continue to analyze or add new annotations e Compare between the automatic level of aggression output that is generated by the system and the manual aggression levels that the user set according to his her thoughts about the situation e Compare the level of aggression between the frames Chapter 8 Conclusions amp Recommendations 8 1 1 Literature The literature survey introduced what aggression is and specially the human aggression This was necessary in this project since it is all about human aggression in a train environment The literature survey also introduced a few examples of annotating tools rule base and expert systems The information that was available in those papers is very good and very common to our problem 8 1 2 Design of the system The system is designed from scratch Therefore the design that has been describ
65. exeilic defrule fight rule 2 activity a_action hit gt assert update fighting 30 assert update normal2 20 defrule fight rule 3 ACTA aa tons mo Sings objects o_description cigarette gt assert update fighting 15 defrule fight rule 4 activity a_action shout gt assert update fighting 10 defrule fight rule 5 activity a_action hold objects o_description beerbottle gt assert update fighting 10 defrule fight rule 6 activity a_action hold objects o_description beercan gt assert update fighting 10 defrule fight rule 7 activity a_action hold objects o_description paint gt assert update fighting 10 114 Video content analysis amp aggression detection system for a train environment defrule fight rule 8 ascii la action Wasi objects o_description hooliganclothing relation r_relation wear gt assert update fighting 25 defrule fight rule 9 humans h_expression fighting gt assert update fighting 60 defrule fight rule 10 humans h_expression afraid gt assert update fighting 10 defrule fight rule 11 humans h_expression angry gt assert update fighting 15 defrule fight rule 12 humans h_expression finger gt assert update fighting 20 defrule fight rule 13 aarlyiey eacui
66. fact base Our discussion of the Rete algorithm is necessarily brief The interested reader is referred to the Forgy paper or to Giarratano and Riley Expert Systems 97 Video content analysis amp aggression detection system for a train environment Principles and Programming Second Edition PWS Publishing Boston 1993 for a more detailed treatment The Rete algorithm is implemented by building a network of nodes each of which represents one or more tests found on a rule LHS Facts that are being added to or removed from the knowledge base are processed by this network of nodes At the bottom of the network are nodes representing individual rules When a set of facts filters all the way down to the bottom of the network it has passed all the tests on the LHS of a particular rule and this set becomes an activation The associated rule may have its RHS executed fired 1f the activation is not invalidated first by the removal of one or more facts from its activation set Within the network itself there are broadly two kinds of nodes one input and two input nodes One input nodes perform tests on individual facts while two input nodes perform tests across facts and perform the grouping function Subtypes of these two classes of node are also used and there are also auxilliary types such as the terminal nodes mentioned above An example is often useful at this point The following rules defrule example 2 defrule example 3 x x
67. g process can be started now During annotating the user should be able to select different kinds of people types objects emotions and relations Because the train environment is dynamic and objects appear at one time and disappear at other the user should be able to remove objects After that the user should be able to save the work that has been made and because there is a save function there must be a load function as well When loading a file the user should be able to play the file and watch what was annotated on each frame and he she should be able to pause stop forward and rewind between the frames 5 AN ANALYSIS OF THE TRAIN COMPARTMENT The environment we will be focusing on is the train compartment In this environment we have specific objects such as seats pathways windows doors and objects that are brought by humans and we also have different type of humans that can be found in a train Humans such as normal passengers conductors police hooligans and beggars play a specific role e g traveler controller and information provider Occasionally unwanted behavior arises for instance aggressive behavior towards conductors or other passengers demolishment of objects Our goal is to design an aggression detecting system by multi modal camera s which are able to observe their environment communicate with the main computer and reason about the observed data and conclude an aggression level as an output I
68. g a knife is annotated defrule fight rule 100 activity a_action hold objects o_ description beggarclothing relation r_relation wear objects o_description knife gt assert update fighting 60 Figure 3 shows the designed user interface We used icons to present the people types objects emotions and relations The use of icons makes it easy to construct annotations that can be easily validated due to the fact that we limited the options in the annotation process in such way that the annotations are error free of syntax Figure 3 The user interface 9 TESTING amp EXPERIMENT RESULTS The testing is carried out by students For our experiment we had 5 test subjects all of the 5 subjects are students from the MMI group We tested different scenarios A task list was giving where testers have to complete a certain series of tasks using the system The first few tasks that were given were relatively easy to do and everything was explained exactly but later on the tasks were open and the user can do what he she thinks is the right action this is done so we can measure the learn ability of the system After finishing the test the users have to fill in a debriefing form where we ask the testers on their opinion regarding the system and to give a grade about several things like difficulty user friendly We also asked the testers to give us three things they l
69. he only aggressive situation that may Doll occur is when a child throws the doll at someone Aggressive use Throwing Wearable audio can be put on the table or listener s lap or in his Mp3 player pockets Turning the value up too loud may irritates others Aggressive use Storing elsewhere Turning the volume up too loud and throwing Money is stored in wallets or in the carrier s clothing Aggressive Money situation may occur when someone tries to steal the money Aggressive use Stealing from the carrier E Painting or writing on the train interior may cause aggressive aint situation Aggressive use Painting on the train interior Table 3 6 People on the train People Description Conductor One who is in charge of a railroad train A conductor checks the tickets of the passengers and provides information s Passenger A person who travels using the train without participating in its operation Beggar One who solicits alms for a living Police officer A member of a law enforcement agency Artist A person whose work shows exceptional creative ability or skill In this project he asks for money after his performance Hooligan A person who treats others violently and roughly Mostly a soccer club fan 22 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis 3 6 An analysis of the train compartment Activities amp behaviors W
70. hen once the object has been modified and operated upon Zeus can be used to convert the Java object back into an XML representation 6 1 4 Version control TortoiseSVN 10 is used for version control TortoiseSVN is easy to use Revision control version control source control software for Windows It is based on Subversion TortoiseSVN provides a nice and easy user interface for Subversion It is developed under the GPL Which means it is completely free including the source code 6 2 Implementation of the expert system 60 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation As described in Chapter 5 the scripts have rules when the user is annotating he she will assert facts into the Jess database and from there a result will be given Those rules are similar to CLIPS rules There is also a script defined which provides the methods that are used in the five scripts described above This script emplate humans laa late swisilo le h_makes_sound h_sound_volume h_description h_Framenumber O h_y h_width h_height h_expression a a SS O 00000000 tb i a ENS Hohe C Cee Cheek eet emplate activity a_id a_is_visible a_object a_action a_subject a_makes_sound a_sound_volume a_Framenumber Oy GY Ge Gy A iy A EN Ol jy Fey FE Sy a A mplate objects ie o_description o_is_visible o_makes_
71. hen setFight is used to set the fighting level in the GUI The GUI now updates the progress bar figure 5 9 to show the fighting level Begging Fighting Control Shouting Normal Figure 5 9 Progress bar updating the Control and begging levels 55 Video content analysis amp aggression detection system for a train environment Chapter 5 Design gt ReadxXML O getFactList getHuman H getObjects setHuman setObjects DrawloadedObjects q ListActivites dorun getFight setFightQ UpdateProgressbar Figure 5 10 Sequence diagram Loading XML file 5 8 Program Flow and Global usage The flowchart in figure 5 11 presents the annotating process The user first chooses a type of people the user will be asked to type a small description of the chosen people type After that using the mouse the user has to draw a rectangle 56 Video content analysis amp aggression detection system for a train environment Chapter 5 Design of the location of the human An expression relation or an action can be given after that Finally clicking the Add button asserts the facts into Jess Figure 5 11 Flowchart of the annotating process 57 Video content analysis amp aggression detection system for a train environment Chapter 5 Design Figure 5 12 shows a global use of the system and the options that can be pe
72. hether these steps are justified or there is a scope of reducing the steps and 76 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results increasing the efficiency Rate of errors by the participants where we will Effectiveness record how many and what kind of errors do the participants make while carrying out the given tasks Retention over time where we will observe and see how well do the participants maintain their E knowledge over the system and the use of it after a Memorability A f demonstration for example if the participants will remember the paths they have taken to do a given task and the way to use the system Subjective satisfaction where we will record how much did the participants like using the different f aspects of the user interface of the system The Satisfaction f eee answer can be determined by a closing interview or a questionnaire that include satisfaction scales and space for free comments It should be ensured that all error and warning messages are intuitive and consistent They should convey meaningful messages that enables user to Consistency a y understand the error condition and remedial actions The output should also be consistent when facing the same scenario 7 2 4 Test procedure When carried out the tests we used the following procedure 1 Give some background info
73. holding beerbottle JA passenger is shouting hooligan is fighting JA passenger is angry Begging Fighting Control Shouting Normal Control Panel Add annotation E z Figure 7 1 Fighting level is flashing Test case PASS System data logging 82 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results 0 MAIN initial fact s MAIN shumans h_id 9 h_is_visible null h_makes_sound null h_sound_volume null h_description white shirt h_Framenumber 0 h_x 121 h_y 15 h_width 57 h_height 95 h_expression null E2 MATINe2a0tivity e eE 1 da e visible null a object 9 a_action hold a_subject 806 a_makes_sound null a_sound_volume null a_Framenumber 0 3 MAIN objects o_id 806 o_description beerbottle o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 4 MAIN objects o_id 100 o_description hooliganclothing o_is_visible null o_makes_sound null o_sound_volume null o_framenumber 0 5 MAIN relation r_id 1 r_object 9 r_relation wear x_subject 100 r_is visible null r_makes_sound null r_sound_volume null 6 MAIN update fighting 10 7 MAIN humans h_id 133 h_is_ visible null
74. hout level will occur Control When the conductor is checking the tickets this will be the result 45 Video content analysis amp aggression detection system for a train environment Chapter 5 Design Begging When a beggar is asking for money begging will be the result The reasoning system works with numbers the higher the number the most likely it is the correct case Normal Fighting Shouting Begging or Control Figure 5 2 shows how the reasoning system works Figure 5 2 The reasoning system When a fact is asserted all the rules that have this fact in the rule base will be activated and waiting for another fact to determine whether the rule will be fired or not some rules have just one fact for example when someone is holding a knife one of the fighting rule will be fired If all facts are asserted at once then the rules that have the highest priority will be fired first There are also rules that subtract a certain amount of aggression from each level this situation occur for example when a hooligan is screaming and making some trouble the fight level will go up two or three frames later a conductor arrives checking the tickets the hooligan is now very quite and behaving well then the fight level will decrease 46 Video content analysis amp aggression detection system for a train environment Chapter 5 Design 5 4 1 Rules design When the user start annotating then this mean that the user is fir
75. ich they might have is not lost when they retire or leave the firm Rule based systems differ from standard procedural or object oriented programs in that there is no clear order in which code executes Instead the knowledge of the expert is captured in a set of rules each of which encodes a small piece of the expert s knowledge Each rule has a left hand side and a right hand side The left hand side contains information about certain facts and objects which must be true in order for the rule to potentially fire that is execute Any rules whose left hand side match in this manner at a given time are placed on an agenda One of the rules on the agenda is picked there is no way of predicting which one and right hand side is executed and then it is removed from the agenda The agenda is then updated generally using a special algorithm called the Rete algorithm Appendix C and new rules are selected to execute This continues until there are no more rules on the agenda Another reasoning model we can use is BDI agents BDI stands for Belief Desire Intention 35 A BDI agent is a particular type of bounded rational 35 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains software agent imbued with particular mental attitudes viz Beliefs Desires and Intentions The BDI model has some philosophical basis in the Belief Desire Intention theory of human practical
76. ie and L J M Rothkrantz An Icon Based Communication Tool on a PDA Euromedia 2005 volume 11 pages 83 90 ISBN 90 77381 17 1 2005 55 P Schooneman ISME Icon based System for Managing Emergencies Master Thesis TU Delft 2005 56 Sabrina Sestito Tharam S Dillon Automated Knowledge Acquisition Prentice Hall of Australia Pty Ltd 1994 57 M Pantic and L J M Rothkrantz Expert system for automatic analysis of facial expressions Image and Vision Computing vol 18 no 11 pp 881 905 2000
77. ifields and multislots The Jess language code is parsed by the class jess Jesp while the actual network is assembled by code in the class jess ReteCompiler The execution of the network is handled by the class Rete The jess Main class itself is really just a small demonstration driver for the Jess package in which all of the interesting work is done 101 Video content analysis amp aggression detection system for a train environment 102 Video content analysis amp aggression detection system for a train environment APPENDIX D Test cases 1 Annotating a hooligan holding a bottle of beer Initialization a video frames must be opened Finalization Clicking the add button if there is something went wrong a popup message giving a warning will appear Actions O Select the hooligan button O Type description of the hooligan O Draw using the mouse a rectangle around the hooligan O Go to the actions tab O Select the icon holding O Go to the objects tabs O Select the beer bottle icon Input data description of the hooligan o Results O Expected results The fighting level will go up O Actual results The fighting level went up 0 MAIN initial fact f 1 MAIN humans h_id 9 h_is_visible null h_makes_sound null h_sound_volume null h_description white shirt h_Framenumber 0 h_x 121 h_y 15 h_width 57 h_height 95 h_expression null 2 MAIN activity a_id 1 a_is
78. ike and dislike about the system This is done to get a better idea about how the testers think about the system and the usability of it The entire test subjects were able to accomplish all the tasks during the test Five different scenarios of aggressive and non aggressive situation on the train could be annotated using the system The aggression that was detected by the system was very close to the aggression level that testers gave to the annotated scenarios The average grade that was given by the test subjects about the overall system was 7 8 10 CONCLUSIONS amp RECOMMENDATIONS People in trains can show aggressive behavior A multimodal camera monitoring the train can recognize this behavior by applying knowledge for people on a train and the objects that are on the train either as interior of the train or brought on the train by passengers With the large variety of objects it is hard to formulate rules that apply to all objects or to classify all objects The multimodal camera needs many skills for its task of identifying aggression it should recognize people as they move on the train identify groups of people i e people that belong together identify gesture body language and recognize speech All these qualities are necessary to properly identify aggression The final goal was to develop a model to annotate a video sequent of images and to detect the level of aggression Dynamic Bayesian networks are probably
79. ing facts into the rule base From there the expert system Jess uses the Rete algorithm to determine an output Figure 5 3 shows how it works Figure 5 3 Reasoning architecture 5 4 2 Rule base Rule based systems also called production systems form a well known architecture for implementing systems based on artificial intelligence techniques 31 The heart of a rule based system consists of a database a rule base and an inference engine The database contains a representation of the state of the environment in asserted facts Upon annotating the environment the user asserts corresponding facts in the fact database The rule base consists of a set of rules each of which maps a specific state in the environment to one or more possible aggression levels and classifications The rules take the following form 47 Video content analysis amp aggression detection system for a train environment Chapter 5 Design if lt list of conditions gt then lt list of aggression level and classification gt Where lt list of conditions gt is associated with asserted facts in the database and lt list of aggression level and classification gt are possible outputs that may update other facts in the database or in the external environment The connection between the facts in the database and the rules in the rule base is made by the inference engine Upon assertion of facts the inference engine considers all rules in the rule base
80. l annotate Also when the user locates the people in the scene a JPEG file is created using this package lt lt interface gt gt lt interface gt lt lt interface gt ssinterface gt gt Value Unmarshallable Slot FactList Fact SO Screenimage MediaPanel CapturedFrames MediaTest Figure 5 7 XML IO Package class diagram 5 7 Dynamic behavior To show what the system does when the user is annotating we used a sequence diagram figure 5 8 Showing the invocation of methods in each object and the order in which the invocations occur This represents the dynamic behavior of the system On the horizontal axis it shows the life of the object that it represents while on the vertical axis it shows the sequence of the creation or invocation of these objects 5 7 1 GUI The user starts by selecting a human icon then the user has to type the description of the human chosen and then the user has to locate the location of the chosen human on the current frame When the location of the human is set the system has to create an image of the human this image is created so it can be 53 Video content analysis amp aggression detection system for a train environment Chapter 5 Design shown when the user desire to see what the chosen human is doing during the whole scene The annotated human will be added to the knowledge base and Jess will print out the output Note that we did not draw Objects Activities
81. lue button figure H 2 Delete row Set Value 3H a Begging Fighting Control Shouting Normal Figure H 2 The manual levels progress bars After completing the levels it can be saved The saved file can be loaded again when starting a new session Loading saved session After opening the frames the saved XML file can be loaded After loading the XML file you will see all the objects and people that are used in the session Figure H 3 130 Video content analysis amp aggression detection system for a train environment DER Concept A hooligan is holding beerbottle A hooligan is hiting a table A conductor is checking ticket A conductor is talking A passenger is afraid A passenger is putting legs on the chair A passenger is siting onthe chair me Delete row Set Value People Expression Relation Acti no Sn RA E H A A SO Begging Fighting Control Shouting Normal Control Panel Add annotation Figure H 3 Loading XML file Clicking on different frames will allow you to see what the aggression levels were on that frame Play session After loading a session you can play the whole session as a scene and see what has been done on every frame To do that click on the
82. mal call n getNormal c defrule update normal2 update normal2 c gt call n setNormal call n getNormal c defrule update control update control c gt call Td sercomezol cr call Fe qerComurial Pe defquery object id query VLOG Wo ace el declare variables id praet s OO EC SN OREA deffunction modify object endframe Modify objects instance endframe id endframe orac Grestllcs rama ausisys Olyjecir ice wie Rate results next Assumes exactly one objects instance with id modify results get fact o_endframe endframe close results akk batch CLP fight clp batch CLP shout clp batch CLP begging clp batch CLP normal clp barca CLP Comeicoll elia pnornall es defrule nomal rule 1 facu Amal tons iO a gt assert update normal 20 AO o A a 111 Video content analysis amp aggression detection system for a train environment defrule nomal rule 2 activity a_action hold objects o_description mp3player gt assert update normal 21 Foca ote ic Viorel criar defrule nomal rule 3 activity a_action hold objects o_description dol1 gt assert update normal 10 P joxesiacouic e inveremetl YY ciel defrule nomal rule 4 activity a_action hold objects o_description bag gt assert update normal 30 2 jexcalacouie e Nstereme Il PY ea defrule n
83. ment than other IDE s The NetBeans IDE is a free Open Source Integrated Development Environment for software developers The IDE runs on many platforms including Windows Linux Solaris and the MacOS Chapter 6 Implementation 6 1 2 Jess Java Expert System Shell Jess is a rule engine and scripting environment written entirely in Java It was originally inspired by the CLIPS expert system shell but has grown into a complete distinct Java in environment of its own Because of its complete implementation in Java the rule engine can be easily embedded within the Java simulation environment For detailed information about Jess 32 6 1 3 XML XML eXtensible Markup Language is a markup language for documents containing structured information XML is used to store the annotation that had been made when using the system Not only had the annotation saved in the XML file but also the configuration of the system Using the system an XML file can be loaded to continue the work on the previous session Xerces 8 is used to parse an XML file and Zeus 9 used as a binder A binder is tool that binds the XML data into Java object models Zeus is in a nutshell an open source Java to XML Data Binding tool It provides a means of taking an arbitrary XML document and converting that document into a Java object representing the XML That Java object can then be used and manipulated like any other Java object in the VM virtual machine T
84. mple 2 1 1 1 1 1 1 2 2 t Each time 1 appears in this string a new one input node is created 2 indicates a new two input node Now watch what happens when we compile example 3 example 3 1 1 1 1 2 t Here we see that 1 is printed whenever a pre existing one input node is shared 2 is printed when a two input node is shared trepresents the terminal nodes being created Note that the number of single input nodes is larger than expected Jess creates separate nodes that test for the head of each pattern and its length rather than doing 100 Video content analysis amp aggression detection system for a train environment both of these tests in one node as we implicitly do in our graphical example No new nodes are created for rule example 3 Jess shares existing nodes very efficiently in this case Jess s Rete implementation is very literal Different types of network nodes are represented by various subclasses of the Java class jess Node Nodel Node2 NodeNot2 NodeJoin and NodeTerm The Nodel class is further specialized because it contains a command member which causes it to act differently depending on the tests or functions it needs to perform For example there are specializations of Nodel which test the first field called the head of a fact test the number of fields of a fact test single slots within a fact and compare two slots within a fact There are further variations which participate in the handling of mult
85. multimodal video recordings of aggression scenarios for a train environment and to return a certain aggression level as an output 1 4 Approach We divided the project into phases 1 The first phase of the project is to research the current techniques to annotate a situation in a certain environment The goal for this phase is to collect enough information so we will be able to decide how we will implement our system 2 The second phase is analyzing all the data we have we watched all the recorded video s of the scenarios that were made by the MMI group The scenarios were predefined and were acted in a real train compartment Notes about the objects and humans relations that can be found in a train compartment were made After that the design of the annotating language is started This phase is necessary to do before the actual design of the system since we need to know the syntax of the language we also need to know what will be annotated and how 3 The third phase is the actual design of the system using UML to make class diagrams and sequence diagrams to understand the system 4 Phase four is the implementation of the system We chose Java as a programming language and Jess as an expert system and XML to store the annotations that were made by the users After the implementation we began to test the system We asked students to test the usability of the system The testers had to fill a debriefing form to tell us what
86. n this project we chose to use a rule base expert system as a reasoning model for the reason that we can represent the situation in a train compartment and put the facts in the knowledge base by annotating the situation The expert system needs a model for the representation of the situation in the train compartment Therefore we have to define which objects can be found that play a role in aggression The world is dynamic this means that objects can appear in one certain moment and disappear in another As a result the reasoning mechanism has to use a database to store all objects people and relations in a database and of course the time where objects and people appear or disappear will be stored as well The static context In order to represent the environment to the expert system we must identify the objects within the static environment that can play a role in aggression The static objects are objects that are always in the train compartment The objects we have found are train seat and train table The dynamic context The dynamic objects are objects that are moving in other words objects that are not always on a train compartment The dynamic objects in this environment can be split into two categories the first one is objects that are brought by people onto the train and the second one is the people who are on the train Objects that are brought by passengers bag cigarette beer can or beer bottle book bab
87. nd will show on every XML file a frame what exactly is happening Rewind The user can rewind or forward a loaded XML file to see a Forward specific frame Load Save The user can use the plus and minus button to manually manual decide what the aggression level on every frame he can levels save these data and load it again in another session Reset Reset all the annotations in the current session The user can select a row from the table and delete it when Delete row an object is disappeared from the scene 3 8 Requirements The goal of the system is to make it possible to the end user to manually annotate the situation on a train compartment The annotation has to be made of a sequence of captured frames from a video file The system should be capable to automatically classify the aggression and come up with an aggression level Other capacities of the system are The user should be able to annotate aggression as well as non aggression situations The user should be able to annotate different types of humans objects actions that are possible on a train environment Saving annotations to XML file Loading annotations from XML file The user should be able to load a sequence of captured frames from a video file Playing a video file Saving the manually edited levels to a text file 28 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis Loa
88. ng the monitors Automatic aggression detection is very young field and from the larger perspective it is just starting up however if significant gains can be made in performance accuracy of detection then a bright future lies ahead in the sense of less incidents and injuries due to aggression Figure 1 1 Surveillance camera Numerous examples can be given where aggression detection can be used to increase the safety For example using aggression detection in a football stadium or other sport stadium where aggression by hooligans may occur Another example is in music or huge pop rock events where drunk people walk around As for now the focus is on aggression detection in a train environment The bottleneck in today s systems is that people have to watch the monitors of the security surveillance cameras all day long And sometimes these are ten or twenty monitors and the human is not able to track everything that happens on all monitors Automatic aggression detection advances can be made in every aspect of this multi disciplinary field from the very start detection of objects detection of 2 Video content analysis amp aggression detection system for a train environment Chapter 1 Introduction groups to a more advanced human sound tracking and facial speech recognition the expert system itself and the rules for detecting the aggression 1 2 Problem definition The Dutch railways NS and the Man Machine Int
89. ngent on the user application time and the entire context of the annotation process The main system modules use fuzzy logic and rule mining techniques to approximate human like reasoning A rule knowledge base is created on a small sample selected by the expert user during the learning phase Using this rule knowledge base the system automatically assigns keywords from the lexicon to non annotated video clips in the database Using common low level video representations the system performance was assessed on a database containing hundreds of broadcasting videos The experimental evaluation showed robust and high annotation accuracy The system architecture offers straightforward expansion to relevance feedback and autonomous learning capabilities Low Level a Feature Extraction L Video J Module Y Difference Metric A i Y OTHER gt Low level Description CLASS 2 LEXICON keyword 2 PS Mining Mining TA keyword 3 gt Module Module lt keywords Rule Knowledge Base CLASS 1 LEXICON keyword 1 Rule Knowledge Representation Module Figure 2 3 Flowchart of the learning unit 14 Video content analysis amp aggression detection system for a train environment Chapter 2 Background 2 3 3 Expert system for automatic analysis of facial expressions This paper discusses an expert system that was made by the authors called Integrated System for Facial Expression
90. nguage and a model generator is used as the problem solver 34 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains 4 2 Reasoning model The simplest form of artificial intelligence AD which is generally used in industry is the rule based system also known as the expert system Before we discuss in details what these are let s take a step back and point out that there are different opinions as to what really constitutes artificial intelligence Some people when they use the term AI are referring to systems which have some ability to learn That is the system will improve its performance over time as it gains experience in solving problems just as a human would Others when they use the term Al are referring just to systems which are capable of exhibiting human level performance in some very narrow area but which are incapable of learning or expanding their expertise Different people are always going to disagree about what Al is but this is fairly simple form of AI which we want to discuss about right now A rule based system is a way of encoding a human expert s knowledge in a fairly narrow area into an automated system There are a couple of advantages to doing so One is that the human expert s knowledge then becomes available to a very large range of people Another advantage is that if you can capture the expertise of an expert in a field then any knowledge wh
91. nnnnnnons 18 Average travel DY TU cisco stes iria tidad 18 Static objects that can be found on a train COMpartMeNt coooonccicnnnconcconnonnconcnnnenos 21 Objects that are brought by passengers ooooconccnoconccncnonnonnconnonncnnconnancconnoncconcnnnanos 21 People on the tati ds a aE N te tetieueeed ee eb va wee aa eee 22 Description of the user actiOns cesesceesseeceesereceececcececsecececceceececaeeneseeeeneees 27 The different alternatives weak strong points oooccnccnocancconnonncnnconcancconcnnnconconnanos 34 Description of the human fact HStecsecresassnsia a 49 Description of the activity fact list 0 0 ce eseeesecececescececeeeececeeceeeeeeceeeenesseeneeees 50 Description of the objects fact list eeeseeeeseeceeceseeeececeececeececeeneceeeenesseeseeees 50 Description of the relation fact list eeeseeecececeeceeeeeececeececeeceeeeeeceenesseeeneees 50 A description of the user interface serenite i iieii r iiiki h 65 Description of the usability measurements oconcnnccnoconccnnconnnonannconcancconnoncconconcanos 76 Test subjects Characteristics renees a E O EAE 80 Top three characteris HES enrio ea aeea e ena ia eee sl Introduction Aggression is a behavior that is intended to threaten or inflict physical injuries on another person or organism a broader definition may include such categories as verbal attack discriminatory behavior and economic exploitation 1 To increase passengers safely from aggression in trains the
92. nose errors e g navigational procedural data entry interpretation etc and 5 List observed usability problems 7 2 6 Test Case We also used a test case to test the functionality of the system The most common definition of a test case is a set of conditions or variables under which a tester will determine if a requirement or use case upon an application is partially or fully satisfied It may take many test cases to determine that a requirement is fully satisfied In order to fully test all the requirements of an application are met there must be at least one test case for each requirement unless a requirement has sub 78 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results requirements In that situation each sub requirement must have at least one test case Test case activity Initialization describes actions which must be performed before test case execution is started For example we should open some files Finalization describes actions to be done after test case is performed For example if test case crashes database tester should restore it before other test cases will be performed Actions step by step to be done to complete test Input data description Results S Expected results contains description of what tester should see after all test steps has been completed S Actual results contain a brief description of what the tester
93. nt analysis amp aggression detection system for a train environment Chapter 3 Problem analysis uses to detect aggression which is people types activity objects relations and emotions 3 3 An analysis of the train compartment The environment we will be focusing on is the train compartment In this environment we have specific objects such as seats pathways windows doors and objects that are brought by humans and we also have different type of humans that can be found in a train Humans such as normal passengers conductors police hooligans and beggars play a specific role e g traveler controller and information provider Occasionally unwanted behavior arises for instance aggressive behavior towards conductors or other passengers demolishment of objects Our goal is to design an aggression detecting system by multi modal camera s which are able to observe their environment communicate with the main computer and reason about the observed data and conclude an aggression level as an output To reason about the observed data in the environment a reasoning model have to be chosen We can choose between an expert system Bayesian networks and neural networks An expert system also known as a knowledge based system a computer program that contains some of the subject specific knowledge and contains the knowledge and analytical skills of one or more human experts A Bayesian network or a belief network is a probabilistic
94. oa ls toute legs Gash objects o_description chair gt assert update fighting 20 assert update normal2 10 pora one Miracle 04 cri defrule fight rule 14 acgrlyiey encerio Mequelnalag gt assert update fighting 15 defrule fight rule 15 Erie la ccCrtiom Masas gt assert update fighting 55 assert update normal2 20 115 Video content analysis amp aggression detection system for a train environment defrule fight rule 16 activity a_action throwing an object at gt assert update fighting 30 assert update normal2 15 derrule atea ile il 7 activity a_action is invading private space of gt assert update fighting 10 assert update normal2 5 defrule fight rule 18 Actin tive amact Lon iol objects o_description knife objects o_endframe null gt assert update fighting2 25 assert update normal 45 defrule fight rule 19 activity a_action hold objects o_description beercan objects o_endframe null gt assert update fighting2 10 defrule fight rule 20 activity a_action hold objects o_description beerbottle objects o_endframe null gt assert update fighting2 10 defrule fight rule 21 activity a_action smoking objects o_description cigarette objects o_endframe null gt
95. omal rule 5 activity a_action hold objects o_description book gt assert update normal 24 ORO A A E defrule nomal rule 6 activity a_action talk objects o_description cellphone gt assert update normal 27 e jorcsincome ic amommal 1 eres defrule nomal rule 7 humans h_expression happy gt assert update normal 15 defrule nomal rule 8 activity a_action hold objects o_description ticket gt assert update normal 10 assert update fighting2 10 portaron i Panorama 101 ieilss 112 Video content analysis amp aggression detection system for a train environment pg slow el defrule shout rule 1 activity a_action shout a_subject cellphone gt assert update shouting 30 defrule shout rule 2 activity a_action shout gt assert update shouting 30 defrule shout rule 3 activity a_action shouting at gt assert update shouting 40 defrule shout rule 4 humans h_expression afraid 2 gt assert update shouting 15 defrule shout rule 5 humans h_expression surprised gt assert update shouting 15 113 Video content analysis amp aggression detection system for a train environment IE ALS PANE AS ILS defrule fight rule 1 ivity a_action hold ts o_description knife update fighting 40 update normal2 30 coure 1 ieaiciae 1
96. oonooconococonoconoconcconcncnnccnoo 12 Figure 2 3 Flowchart of the learning Unit eeeeeesseeeceseeneceeeeseeaeeeseeaeeerseaeeneeeseeaneeaes 14 Figure 3 1 Feature points fiducials of the contours of the face components and of the profile CONLOUN A A E ee ate vat eee dE IE da dada eee 24 Figure 3 2 Exemplar set for walking cesessceccssscseeesseeeeseeeeeecesecneeeseeseeeeeeseeneeeseeaeeeaes 25 Figure 3 2 Use Case diagram toeon nr e oie vantinsedetiateaavanthagedncent Addenda 27 Figure 4 1 Parse tree Mike hit the ball eee eeescesseseceseeneceeeesecneeeseeaeeeeeeseeneeeaeeaeeeaes 38 Figure 4 2 Example of annotating a beggar oo eeeeeesseseceseeseceecesteaeeeseeaeeeeeeseenteeaeeaneeses 41 Figure 4 3 overview of the multimodal language conconnnnnccncnnocnnnnncnconconcnancnncnnnonncnnnonncnnos 42 Figure 5 1 System architecture showing the different components and how they are related 44 PURAS NAS A dbesavweses 46 Figure 5 3 Reasoning architecture esescesesceccesecnececeaeceseeseceeceseeaeeeseeaeeeseeaeeneessteaneeaes 47 Figure 5 4 The XML file hierarchy Human attributes ooonconnoninnnconnnnocnnoncnncnancnncnnnonncnnos 49 Figure 5 5 system main pakapes ea a E EAA 51 Figure 5 6 Class diagram of the package GULI c ooooncnconicnconccccoconononnconncnncnnonononncnnnonncnnnonncnnos 52 Figure 5 7 XML IO Package class diagram oisein anandia Eneee hea A iai Ainaa a Sir 53 Figure 5 8 Sequence diagram annotating Pro
97. orld and save the world model in XML file Jess 30 provides a cohesive tool for handling a wide variety of knowledge with support for rule based programming Rule based programming allows knowledge to be represented as heuristics or rules of thumb which specify a set of actions to be performed for a given situation 36 Video content analysis amp aggression detection system for a train environment Chapter 4 Annotation language for trains 4 3 Language syntax Syntax refers to the sets of rules we use in constructing grammatical understandable phrases and sentences Syntax is generally regarded as being independent of sensibility the sentence Colorless green ideas sleep furiously written by the famous linguist Noam Chomsky certainly does not make any sense but its syntax is correct 4 3 1 Phrase structures Every sentence is composed of some combination of phrases In almost every language including English a sentence must have a noun phrase and a verb phrase Sentences may also have object phrases that specify who did what to whom For example we can parse the sentence Mike hit the ball The noun phrase of this sentence is Mike the verb phrase is hit and the object phrase is the ball Phrases can be moved around for stylistic reasons or to draw more attention to a certain part of the sentence A parse tree or concrete syntax tree is a tree that represents the syntactic structure of a string according
98. ount 5 participants Age Varies from 21 to 28 years old Gender 1x Female and 4x Male Highest level of education Bachelor of science BSc These test subjects presents our user group 7 3 2 Expectations We expect small problems with the following activities Making annotations syntax errors Annotate irrelevant things Annotate too much Not annotate everything that is relevant Drawing a rectangle around the human We also expect that the time to accomplish the task list will be between 10 15 minutes and the debriefing form 10 minutes 80 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results 7 3 3 Results All of our expectations where justified when we looked at the experiment results and even some more came to our attention Some expectation we had were not justified but these where all duration time expectations We expected that the time which the test subject needed to finish the task list would be about 10 15 minutes but it took them 15 20 minutes We expected that the debriefing would take 5 minutes but it took 10 minutes During the first 2 tests we found many bugs and errors we did not find before because we did not make mistakes using the system but the testers made some mistakes and we found those bugs We fixed the bugs and errors and continued with the test with the other three test subjects The entire test subje
99. p Experiment results 7 2 1 Test setup The tests were carried out in the MMI lab figure 7 1 The tester will carry out the test on the computer where the system is installed on The observer will sit next to him her taking notes about what the user is actually doing and where things went wrong The observer also comments on errors when they occur and keeps a timeline about the user actions Figure 7 1 Testing in the MMI lab the observer right the tester left 7 2 2 Usability testing To identify discrepancies between the user interface of the system and the human engineering requirements of users usability testing has to be performed The user interface is the module of the software which comes in direct interaction with the user so its logical organization design and correctness plays an important part in the better usability of product A good way to define usability testing is testing the software from a users point of view which boils down to testing the software to ensure that it is user friendly Usability testing is different from functionality testing as the later involves validating the software against the requirements only In simple words 75 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results usability testing includes ergonomic considerations screen design consistency in user interface proper naming conventions of the menus and op
100. p aggression detection system for a train environment lt slot gt lt fact gt lt fact gt lt name gt MAIN activity lt name gt lt slot gt lt name gt a_id lt name gt lt value type INTEGER gt 1 lt value gt lt slot gt lt slot gt lt name gt a_is_visible lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt a_object lt name gt lt value type INTEGER gt 9 lt value gt lt slot gt lt slot gt lt name gt a_action lt name gt lt value type STRING gt hold lt value gt lt slot gt lt slot gt lt name gt a_subject lt name gt lt value type INTEGER gt 806 lt value gt lt slot gt lt slot gt lt name gt a_makes_sound lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt a_sound_volume lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt a_Framenumber lt name gt lt value type INTEGER gt 0 lt value gt lt slot gt lt fact gt lt fact gt lt name gt MAIN objects lt name gt lt slot gt lt name gt o_id lt name gt lt value type INTEGER gt 806 lt value gt lt slot gt lt slot gt lt name gt o_description lt name gt lt value type SYMBOL gt beerbottle lt value gt lt slot gt lt slot gt lt name gt o_is_visible lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt o_makes_sound lt n
101. reasoning expounded by Michael Bratman Wooldridge 36 lists four characteristics of intelligent agents which naturally fit the purpose and design of the BDI model Situated they are embedded in their environment Goal directed they have goals that they try to achieve Reactive they react to changes in their environment Social they can communicate with other agents including humans One can say that choosing BDI agents as a reasoning system is a good choice since it can situated to the train environment with the goal to detect aggression and they can react to the dynamic changes of the train environment and finally they can communicate with the human when an aggressive situation occur But what would be the beliefs of the agents We are using an annotating system and therefore when someone annotates a situation then these are facts and not beliefs Desire could be to detect aggression but what would be the desire when there will not be an aggressive situation Then the agent will have the intension to try and search for aggression Therefore BDI agent reasoning system is not suited in our system it can be used when the system can control the cameras in the train e g when the agent belief there is aggression he could move the cameras towards the aggressor with the desire to detect the aggressor It is clear that a rule based approach will be used to reason about the world We developed a special language to describe the w
102. rey Davis The Culture of Body Language How a Person s Gestures Take on Meaning in Different Parts of the World The People media company Published Mar 01 2006 Extensible Markup Language XML http www w3 org XML Last visited at September 25 2007 XSL Transformations XSLT http www w3 org TR xslt Last visited at September 25 2007 Jess the Rule Engine for the Java Platform http herzberg ca sandia gov jess Last visited at September 25 2007 Production system http en wikipedia org wiki Production_system Last visited at September 25 2007 Ernest Friedman Hill Jess in Action Java Rule based Systems ISBN 1930110898 2003 Introduction to DTD http www w3schools com dtd dtd_intro asp Last visited at September 25 2007 Sergios Theodoridis Konstantinos Koutroumbas Pattern Recognition 3rd Edition ISBN 0123695317 February 2006 Bratman M E Intention Plans and Practical Reason CSLI Publications ISBN 1 57586 192 5 1987 1999 Wooldridge M Reasoning About Rational Agents The MIT Press ISBN 0 262 23213 8 2000 P Klahr and D Waterman Expert systems techniques tools and applications Addison Wesley Longman Publishing Co Inc Boston MA USA 1986 S Fitrianie and L J M Rothkrantz An Icon Based Communication Tool on a PDA Euromedia 2005 volume 11 pages 83 90 ISBN 90 77381 17 1 2005 P Schooneman ISME Icon based System for Managing Eme
103. rformed The user can watch the video of the scene he she wants to annotate After that the user has to open the frames he she wants to annotate Then the user can load an XML file that contains annotating he she has made before Or he she can start annotating and after that the user can save the XML file Load XML View captured frames Save XML Exit application Figure 5 12 global usage of the system Play XML Rewind Forward 58 Video content analysis amp aggression detection system for a train environment Implementation This chapter describes the actual implementation of the system The system is implemented accordance with the design that is specified in Chapter 5 First an overview of the used software and tools is given and afterwards the implementation of expert system is discussed and finally some screenshots of the GUI will be shown 6 1 Tools amp Software The overall used programming language for this project is Java Java was chosen above other languages because the expert system that will be used for the project Jess is also implemented in Java this mean that the communication between the expert system and the GUI will be easier when the GUI and the whole system is implemented in Java The used Java version is 1 5 6 1 1 NetBeans IDE Netbeans IDE 5 0 is chosen for as a development environment NetBeans provides a rich set of features and a more productive work environ
104. rgencies Master Thesis TU Delft 2005 Sabrina Sestito Tharam S Dillon Automated Knowledge Acquisition Prentice Hall of Australia Pty Ltd 1994 Han Reichgelt Knowledge Representation an Al perspective Ablex Publishing Corporation 1991 Hector J Levesque and Gerhard Lakemeyer The logic of knowledge bases The MIT Press 2000 91 Video content analysis amp aggression detection system for a train environment Reference 43 Peter Jackson Introduction to expert systems 2nd edition Addison Wesley Publishingcompany 1990 44 Waterfall model http en wikipedia org wiki Waterfall_model Last visited at September 25 2007 45 Fuller RW The influence of fluoexetine on aggressive behavior Neuropsychopharmacology 14 77 81 1996 46 Robert A Baron Deborah R Richardson Human aggression Second edition ISBN 978 0306444586 Plenum Publishers Nov 1993 92 Video content analysis amp aggression detection system for a train environment APPENDIX A Tasks list The following list presents the activities the testers have to perform during the test Start the application using F6 Open video file beggar2 mpg located in the desktop Watch the video o Open frames using the file 03_beggar txt We want to annotate the man in the yellow shirt as a beggar holding money 5 Click in the human tab on a beggar 6 Type as description for the beggar
105. rmation about train aggression 2 Introduce the system to the participants what the purpose of the system is and how you can use it 3 Give a short demo and explain what you are doing 77 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing 4 Experiment results 4 Train the participants to think out loud so that you are able to better understand their experiences with the system 5 Begin the test by giving the participants a task list that they must carry out Appendix A 6 Carry out a closing interview to review overall impressions of the Health Pal system 7 Give the participants a questionnaire about the tasks they carried out and about the system it self Appendix B 8 Thank the participants 7 2 5 Data analysis Once the usability test sessions are complete using the saved session by the testers and the data logging of the tests the following activities are carried out to compress the data and present the findings Nominally we expect to perform the following kinds of analyses to measure the objective and subjective performance of the system 1 Evaluate the efficacy of the system information architecture and navigation 2 Seek an understanding of the system strengths and weaknesses regarding branding and information offerings 3 Organize anecdotal comments in a manner that provides a clear picture of the system strengths and weaknesses 4 Classify and diag
106. ronment APPENDIX G lt xml version 1 0 encoding US ASCII gt lt fact list gt lt fact gt lt name gt MAIN initial fact lt name gt lt fact gt lt fact gt lt name gt MAIN humans lt name gt lt slot gt lt name gt h_id lt name gt lt value type INTEGER gt 9 lt value gt lt slot gt lt slot gt lt name gt h_is_visible lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt h_makes_sound lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt h_sound_volume lt name gt lt value type SYMBOL gt null lt value gt lt slot gt lt slot gt lt name gt h_description lt name gt lt value type STRING gt white shirt lt value gt lt slot gt lt slot gt lt slot gt lt name gt h_Framenumber lt name gt lt value type INTEGER gt 0 lt value gt lt slot gt lt slot gt lt name gt h_x lt name gt lt value type INTEGER gt 122 lt value gt lt slot gt lt slot gt lt name gt h_y lt name gt lt value type INTEGER gt 11 lt value gt lt slot gt lt slot gt lt name gt h_width lt name gt lt value type INTEGER gt 56 lt value gt lt slot gt lt slot gt lt name gt h_height lt name gt lt value type INTEGER gt 116 lt value gt lt slot gt lt slot gt lt name gt h_expression lt name gt lt value type SYMBOL gt null lt value gt 123 Video content analysis am
107. roviding a high level view of what the system does 26 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis Identifying the users actors of the system Determining areas needing human computer interfaces OS ad Open video 7 oY rN gt a 4 a QS e frames 3 Start annotating Save XML file a nn View captured frames gt Load XML file Delete row dl Load Save Manual levels Figure 3 2 Use Case diagram To summarize the actions that can be made table 3 7 provides an overview of the functionality of the system Table 3 7 Description of the user actions Action Description The user can view a video before starting the annotating Open Video process When the program starts the user have to load open a set of Open frames frames that has been captured from a video file 27 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis ae PNE The user can load a saved session from a XML file Be FM The user can save a session into a XML file After opening the frames or loading XML file the user can Annotating a now start annotating View Be The user can see what is been captured while annotating P the world model frames Play loaded The user can play the loaded XML file the system will start from frame 1 to frame 10 a
108. ry point of view e Predatory aggression attack on prey by a predator e Inter male aggression competition between males of the same species over access to resources such as females dominance status etc e Fear induced aggression aggression associated with attempts to flee from a threat e Irritable aggression aggression induced by frustration and directed against an available target e Territorial aggression defense of a fixed area against intruders typically conspecifics e Maternal aggression a female s aggression to protect her offspring from a threat Paternal aggression also exists e Instrumental aggression aggression directed towards obtaining some goal considered to be a learned response to a situation Currently there is a consensus for at least two broad categories of aggression variously known as hostile affective or retaliatory aggression versus instrumental predatory or goal oriented aggression Empirical research indicates that this is a critical difference both psychologically and physiologically Some 8 Video content analysis amp aggression detection system for a train environment Chapter 2 Background research indicates that people with tendencies toward affective aggression have lower IQs than those with tendencies toward predatory aggression 15 As can be expected humans express their aggression differently than animals Although humans are similar to animals in some aspects of aggres
109. s recorded in train compartments We want to investigate acquire knowledge see how an expert detects aggression To investigate the process of security experts we created an acquisition tool 2 AGGRESSION IN TRAIN The aggression in train compartments can take many different forms It can be directed towards the interior of the train the passengers conductors and train mechanist The interior can be damaged or painted and sometimes burned or written on Passengers conductors and train mechanist can be robbed threatened intimidated or physically or mentally hurt The train aggression is causing a decrease in the number of conductors The number of intimidating and threatens towards the conductors is growing In the first 8 months of 2003 the number of aggression cases was more then 15 than a year earlier in the Netherland The statistics are showing that the aggression problems are getting worse in 2001 there were only 6 944 incidents and in 2006 that number was 10 500 The aggression is directed in 75 of the cases towards the conductors and within that 75 is 1 out of 10 incidents include physical violence 2 About 10 of the aggression is directed towards the passengers The Dutch railroad needs at least 10 more conductors and they can t achieve that by not doing anything to the aggression on the trains NS tried to solve this problem by hiring guards as conductor assistant during rush hours The train in
110. seeeeeueeeeeeesaeeees 60 6 3 GUL E SCREENSHOTS a EEA 64 E A I ED TED i EA H D E A iaa 68 7 TESTING EXPERIMENT RESULTS ccccssocsscsssccccccssscccccessssccecscessccccscenscccessesssscccsssssccessees 73 PA DIYO E D OA HIN EE EEA E E E o ii 73 LATEST PIAN es a aaa a a aet 74 TB EXPERIMENT RESULT dass 80 8 CONCLUSIONS amp RECOMMENDA TIONG csscccccssssccccccssscccccessssccccccssscccescensscccesensccccsessssscceeees 85 8 L CONCLUSION A eee 85 g2 RECOMMENDATIONS 2 a E n 87 REFERENCE cisne esae i oa e aeaa O E a ae a a aaa Sains 89 APPENDIX Aore eai a a eanas a E aae ar S aaa aas enoar aaria ecesas iaooe oseese es 93 INPPENDIUX EEEE EEE E AT TEE E EE 95 APPENDIX E aa a ea A a O onas ne Eai 97 APPENDIX D occiso iii ai toatoa sessie eses sinies 103 APPENDIX Enies sseense secies aeaeo sasa ed rearea aea orae Oe OaS S duedepcastbcacbanesbecobeauoublenseuaseenes 109 APPENDIX Frrr a eS e aeS e a a OIRE NOE eaS 119 APPENDIX G oieee sosest eee cosi sesiis ieie sensisse orse saoo edoae ieaoo eio oses Soa deio oee oeaio seod esos eraso o aiio ines Sonea isis 123 APPENDIX H dani 127 APPENDIX Gi isa 133 VII List of figures Figure 1 1 Surveillance Camera esceceesscsecssseseceeceseceecseccecsceececsececececaeececseceaeceeseneeaeeseeees 2 Figure 2 1 The location of the human Hypothalamus in the brain oonnocnnccnnnnnonnnonnnonncnns 11 Figure 2 2 the general aggression model episodic processes ooooon
111. sion they differ from most animals in the complexity of their aggression because of factors such as culture morals and social situations A wide variety of studies have been done on these situations Alcohol drugs pain and discomfort frustration and violence in the media are just a few of the factors that influence aggression in humans In this project we will focus on instrumental aggression since it is a planned goal directed aggression Also this form of aggression is known to be the most common aggression that may occur on a train 2 1 2 Instrumental aggression The two cardinal characteristics of instrumental aggression are goal directedness and planning The instrumental aggressor acts to obtain a readily apparent goal such as power money sexual gratification or some other objective beyond inflicting injury on the victim Examples of instrumental aggression include shooting a police officer in the course of a bank robbery stabbing a homeowner during a burglary and strangling a rape victim Instrumental aggression is initiated as a means to an end rather than as an act of retaliation or self defense As a result instrumental aggression involves planning or preparation However in some cases instrumental aggression involves relatively little planning such as in the case of a criminal who engages in an opportunistic offense e g unexpected opportunity to rob someone that involves assaulting the victim In some cases a subjec
112. sociated with emotional response and arousal These structures along with the septum and amygdale may play a role in mediating aggression 3 Pituitary Hypothalamus Figure 2 1 The location of the human Hypothalamus in the brain On the other side Craig A Anderson and L Rowell Huesmann 17 tried to understand the human aggression in a social cognitive view They researched other human aggression model that has been developed in the last 30 years they tried to identify what are the causes of the aggression and in which environments they may occur They used the general human aggression model figure 2 2 which illustrates the point that all of the social cognitive models agree that aggression results from the way in which person variables and current situational variables combine to influence the individual s present internal state Sometimes person and situation variables combine interactively as in K B Anderson et al s 1998 finding the pain and the trait hostility interactively affect aggressive cognitions The present internal state then influences a host of appraisal and decision processes Eventually sometimes very quickly an action emerges which in turn moves the social encounter along to its next cycle 11 Video content analysis amp aggression detection system for a train environment Chapter 2 Background Inputs Person Situation Present Internal State Routes Affect Social Encounter X J
113. software product may have great features and wide array of options but if it is difficult to use and is complicated then there is a strong chance of an otherwise good product to get branded as bad product While developing and testing the software one must take into consideration is that both a novice and a seasoned user are able to use it with relative ease A focused testing effort on usability aspects of the product leads to development of more usable product and satisfied customers The testing will be carried out by students who work at the MMI lab A task list Appendix A is giving where testers have to complete a certain series of tasks using the system The first few tasks that were given were relatively easy to do and everything was explained exactly but later on the tasks were open and the user can do what he she thinks is the right action this is done so we can measure the learn ability of the system After finishing the test the users have to fill in a debriefing form Appendix B where we ask the testers on their opinion regarding the system and to give a grade about several things like difficulty user friendly We also asked the testers to give us three things they like and dislike about the system This is done to get a better idea about how the testers think about the system and the usability of it 74 Video content analysis amp aggression detection system for a train environment Chapter 7 Testing am
114. sound o_sound_volume o_framenumber o_endframe na Gb A Gp JO 0000000 th Gu ar ie tr fer er ore mplate relation 12 all 2 lO ISCIE r_relation r_subject r_is_visible r_makes_sound r_sound_volume Oy GY Gr Gy Wi I A O 0000000 th As we can see that each template has the same attributes as the XML file described in Chapter 5 That is done so it will be easy to bind the XML data into Jess facts and vice versa 61 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation The rules that are in the aggression script which is the engine of the whole expert system is shown below defrule update begging update begging c gt Gall we NS SEBAS MS SEE eq REII defrule update shouting update shouting c gt CHARTS ESO O A mA Cio Oi me e defrule update fighting update fighting c gt SELL Piz sermoe F eel Bie ese rica ZEN defrule update fighting2 update fighting2 c gt call 2 sertie Call Ze garialcime HE defrule update normal update normal c gt call n setNormal call n getNormal c defrule update normal2 update normal2 c gt call n setNormal call n getNormal c defrule update control update control c gt 62 Video content analysis amp aggression detection system for a train environment Chapter 6 Implemen
115. ssion when there are not many seating space available causing people to travel standing up Aggressive use keeping on seats throwing or touching other passengers with It is not allowed to smoke in the train and this might create an Cigarette aggressive situation when someone is smoking Aggressive use smoking when it s not allowed Drinking beer is allowed in the train Aggression situation may occur Beer can or when someone gets drunk beer bottle Aggressive use getting drunk throwing at another passengers These can be put on the table on benches or on the luggage rack Book Aggressive use putting it on fire reading in a way that hinders others throwing at others A pram will take up seating space or stand in the way in the gang Baby pram way but clearly there is no way around this Aggressive use putting on seats throwing at others touching others i Knives are most of the times hidden by the passengers Aggression Knife situation may occur when someone is holding one Aggressive use hitting others with it threatening others Tickets Tickets are used as transportation prove Aggression situation may occur when someone does not have a ticket Phones can be put on the table or in the carrier s clothing or bag Cell phone Some people get irritated by hearing other people conversing over the phone loudly Aggression use Throwing Loud conversation Dolls are brought by children T
116. static frontal and or profile view color face images A multi detector approach to facial feature localization is utilized to spatially sample the profile contour and the contours of the facial components such as the eyes and the mouth From the extracted contours of the facial features they extract ten profile contour fiducial points and 19 fiducial points of the contours of the facial components Based on these 32 individual facial muscle actions AUs occurring 23 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis alone or in combination are recognized using rule based reasoning With each scored AU the utilized algorithm associates a factor denoting the certainty with which the pertinent AU has been scored A recognition rate of 86 is achieved Pt top of the forehead P2 eyebrow arcade P3 root of the nose P4 tip of the nose PS upper jaw P6 upper lip P7 lips attachment if tongue is visible E E1 outer comer of the eyebrow D D1 inner corner of the eyebrow A A1 outer corner of the aye B B1 inner corner of the eye F F1 top of the eye G 61 bottom of the eye H H1 outer corner of the nostril K top of the upper lip L bottom of the lower lip 1 J mouth corner M tip of the chin P7 tip of the tongue P8 lower lip PS lower jaw P10 tip of the chin titit tt Figure 3 1 Feature points fiducials of the contours of the face
117. sueceess eri O asas Ge tar tar cr tar ar a ar O template humans A Stl h_is_visible h_makes_sound h_sound_volume h_description h_Framenumber h_x h_y h_width h_height h_expression CIRO OI i ti CI VIVIR E PE pp fey E e E ps a a 109 Video content analysis amp aggression detection system for a train environment deftemplate activity slot a_id slot a_is visible slot a_object slot a_action slot a_subject slot a_makes_sound slot a_sound_volume slot a_Framenumber deftemplate objects slor rel slot o_description silo 0 18 visiloule slot o_makes_sound slot o_sound_volume slot o_framenumber slot o_endframe deftemplate relation slo 12 tel SIOE 2 ODJECE slot r_relation slot r_subject silo uns wisilolS slot r_makes_sound slot r_sound_volume defrule update begging update begging c gt call s setBeg call s getBeg c defrule update shouting update shouting c gt CAMAS SES OU G Call TR SS MOTE E defrule update fighting update fighting c gt eulil gic Serevent a gt aL Bir CETNLone Be 110 Video content analysis amp aggression detection system for a train environment defrule update fighting2 update fighting2 c gt Call P seuslciae eall PE Ceiewueiane Pe defrule update normal update normal c gt call n setNor
118. t hand side LHS against the knowledge base and executing the right hand side RHS of any rules that apply This is inefficient because most of the tests made on each cycle will have the same results as on the previous iteration However since the knowledge base is stable most of the tests will be repeated You might call this the rules finding facts approach and its computational complexity is of the order of O RF P where R is the number of rules P is the average number of patterns per rule LHS and F is the number of facts on the knowledge base This escalates dramatically as the number of patterns per rule increases Jess instead uses a very efficient method known as the Rete Latin for net algorithm The classic paper on the Rete algorithm Rete A Fast Algorithm for the Many Pattern Many Object Pattern Match Problem Charles L Forgy Artificial Intelligence 19 1982 17 37 became the basis for a whole generation of fast expert system shells OPSS5 its descendant ART and CLIPS In the Rete algorithm the inefficiency described above is alleviated conceptually by remembering past test results across iterations of the rule loop Only new facts are tested against any rule LHSs Additionally as will be described below new facts are tested against only the rule LHSs to which they are most likely to be relevant As a result the computational complexity per iteration drops to something more like O RFP or linear in the size of the
119. t a basic work frame was implemented Next all objects people relations and actions are 86 Video content analysis amp aggression detection system for a train environment Chapter 8 Conclusions amp Recommendations implemented Then the user interface was developed Finally more complicated features such as the capturing images drawing the rectangles saving and loading to XML file were implemented We can say that the approach for the implementation was a good choice since we were able to implement in phases and test in phases Developing the system using the programming language Java made it easy to accomplish the goal that the system should be OS in depended 8 2 Recommendations Throughout the individual chapters of this thesis a number of suggestions for future work have been given This section summarizes these 8 2 1 Approach The final goal was to develop a model to annotate a video sequent of images and to detect the level of aggression Dynamic Bayesian networks are probably the most appropriate approach since we will be able to use pattern recognizing to classify aggression based on either a priori knowledge or on statistical information extracted from the patterns 34 But because of its simplicity and availability of data we use a rule based system as a first approach It would be nice to know whether Dynamic Bayesian networks will be more accurate and more efficient than using the expert system and
120. t may plan a robbery or burglary and when something goes wrong engages in an act of aggression such as shooting someone in order to get away In these cases the coder should consider that the subject s plans include the possibility of violence even if there was no specific plan to shoot someone Instrumental aggression usually involves little or no provocation by the victim In some cases subjects may be provoked into violence in the course of another 9 Video content analysis amp aggression detection system for a train environment Chapter 2 Background crime e g a robbery victim who insults the subject or resists the robbery in some way These acts are still considered instrumental acts of aggression Instrumental aggressors are motivated by goals not emotions It follows that their level of emotional arousal especially anger is relatively low or is secondary to he act Some instrumental aggressors try to calm themselves prior to an offense through drug use or drinking In extreme cases instrumental aggressors are not angry toward their victims and may have a cold business like attitude about their behavior Nevertheless many less hardened instrumental aggressors are nervous and highly aroused while committing a crime even though it is not their arousal which motivates their actions 2 1 3 Biological amp Social research Reading a daily newspaper shows us how serious the problem of violence is in today s societ
121. t the aggression level is The user also can rewind forward frames using the rewind forward buttons Thresholds There are several thresholds implemented in the system The thresholds will trigger an alarm sound and will make the aggression level text to flash when the aggression level is above 80 Figure 6 3 H 4 Begging Fighting Control Shouting Normal Figure 6 3 The fighting level is above 80 The text fighting flash white and red 67 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation 6 4 Flexibility One of the important goals and requirements of the system is the flexibility of the system This means that the system should be easy to modify by inserting new rules editing rules and inserting more icons into the GUI of the system This paragraph provides the basic lines for modifying and adding new rules and inserting new icons 6 4 1 Adding new rules Adding or editing rules is quite easy In the CLP directory six files can be found ageression clp is the main file The other five files are files for every script or ageression classification The main file should be not be edited because it s the engine of the whole expert system the main file consist of templates of the human relations objects and relations Also the main rules for adding subtracting and searching are located
122. tart the analysis about what is needed in the system is to sketch what a typical annotating session will look like The user needs to see the video file he will annotate this means that the system must be able to open a video file Since the annotation will be on static images not moving video the user must be able to load a set of frames pictures that were captured from a video file The annotating process can be started now During annotating the user should be able to select different kinds of people types objects emotions and relations Because the train environment is dynamic and objects appear at one time and disappear at other the user should be able to remove objects After that the user should be able to save the work that has been made and because there is a save function there must be a load function as well When loading a file the user should be able to play the file and watch what was annotated on each frame and he she should be able to pause stop forward and rewind between the frames Now that we know the important aspects of the system a use case diagram can be drawn A Use Case diagrams identify the functionality the system will provide the users who interact with the system actors and the association between the users and the functionality Use Cases are used in the Analysis phase of software development to articulate the high level requirements of the system The primary goals of Use Case diagrams include P
123. tation Cadi ro Serono E Ae Ono REI defquery object id query Loc 9 reee akel Y declare variables id Trace lt oloqeci s ko tel Tich deffunction modify object endfram Modify objects instance endframe 2id endframe Uouline Piresultes bine OOc cte deny Riel results next Assumes exactly one objects instance with id modify results get fact o_endframe endframe close zresults mar JL The rules shown above are rules to update the score of the corresponding script or aggression type For example the rule update begging will update the begging level by first calling the current level and then adding the giving number and finally set the total as an output The query object id looks for the fact id in the database of the inserted object id this is used to update the o_endframe It s used with the rule modify object endframe Each of the 5 aggression scripts normal control begging shouting and fighting has rules that contain facts E g the control rule 4 shown below will detect a conductor and control rule 1 will add 30 points more to the control level since the conductor is checking the tickets 63 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation defrule control rule 4 objects o_description conductorclothing relation r_relation wear gt assert update control 10 orimeout E Mecnteso
124. tem for a train environment Chapter 2 Background physical violence 4 About 10 the aggression is directed towards the passengers 67 of the total aggression is in the form of threatening or swearing The Dutch railroad needs at least 10 more conductors and they can t achieve that by not doing anything to the aggression on the trains NS tried to solve this problem by hiring guards as conductor assistant during rush hours The train interior damage is costing the Dutch railroad about 20 million euros on a yearly basis according to Mr Schultz van Haegen the state secretary at the Dutch ministry of Water and Traffic The damage is mainly caused by vandalism and graffiti that are drawn on the interior of the train The eradication costs of these graffiti cost about 10 million euros The misuse of the fire extinguishers and the cleaning of the damage afterwards are costing the NS about 5 million 6 2 3 The Current state of the art This part provides a list of some studies about human aggression annotating tools rule base and expert systems Below the topics are discussed into chapters 2 3 1 Automated aggression detection Video Annotation for Content based Retrieval using Human Behavior Analysis and Domain Knowledge This paper 18 proposes an automatic annotation method of sports video for content based retrieval Conventional methods using position information of objects such as locus relative positions their transitions et
125. tem is discussed and finally some screenshots of the GUI will be shown Chapter 7 describes the test plan and presents the testing results of the implemented system First an explanation about how the data was collected thereafter the test plan will be described and finally the experiments results will be presented In Chapter 8 the results of the project will be assessed against the requirements and goals that were defined at the start and the results of the implemented system of the train aggression project will be discussed Furthermore recommendations for future work will be given 5 Video content analysis amp aggression detection system for a train environment Background This chapter provides background information about human aggression in general and the aggression types After that we will discuss about the aggression in train Then a discussion about the related projects that concerns aggression annotating usability and expert systems 2 1 Human aggression Aggression is a perplexing phenomenon Because in some situations it is unknown of why are people motivated to hurt each other How does violence help organisms to survive and reproduce After two centuries of theories 21 and technological advances psychologists and other scientists have been able to look deeply into aggression s biological and evolutionary roots as well as its consequences in society 2 1 1 Definition Aggression is defined as physical or
126. tent analysis amp aggression detection system for a train environment Reference 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Sci Tech Encyclopedia Aggression http www answers com topic aggression Last visited at September 25 2007 Aggression in Human Beings Factors of Aggression and What Can Be Done to Lesson it http www associatedcontent com article 4600 aggression_in_human_bein gs html Last visited at September 25 2007 Joanna Schaffhausen The Biological Basis of Aggression Published at http www brainconnection com September 1 2007 Agressie in de trein neemt toe Algemeen Dagblad 1 november 2003 Treinkapitein heeft het zwaar http www nrc nl W2 Lab Spoorwegen 001014a html Last visited at September 25 2007 Spoorvandalisme kost zeker 20 miljoen per jaar http www home nl nieuws binnenland artikel 001 14122 Last visited at September 25 2007 Rete algorithm http en wikipedia org wiki Rete_algorithm Last visited at September 25 2007 Xerces Java Parser http xerces apache org xerces j Last visited at September 25 2007 Enhydra Zeus Project http forge objectweb org projects zeus Last visited at September 25 2007 TortoiseSVN http tortoisesvn net Last visited at September 25 2007 Z Yang L J M Rothkrantz Aggression Detection in Train Compartments Data Collection Plan April 2006 J
127. terior damage is costing the Dutch railroad about 20 million euros on a yearly basis according to Mr Schultz van Haegen the state secretary at the Dutch ministry of Water and Traffic The damage is mainly caused by vandalism and graffiti that are drawn on the interior of the train The eradication costs of these graffiti cost about 10 million euros The misuse of the fire extinguishers and the cleaning of the damage afterwards are costing the NS about 5 million 3 The conclusion is that NS needs an automatic aggression detection system 3 AGGRESSION DETECTION BY HUMANS To understand what is needed to detect aggression automatically it is important to understand how human detect an aggressive situation Humans are very well able to detect aggression in verbal expressions The problem here is that speech recognition techniques is still ongoing researches and there is no technique available that provides enough accuracy to depend on given the large amount of noise occurring in the train Therefore we need to identify other aspects that humans use to detect aggression e g emotions shouting and location Humans detect aggression easily when he she knows the environment e g when a human sees two people fighting in a karate school he she knows this is part of the sport and its probably not a form of aggression but when he she sees two people fighting on a train then this means it is a form of aggression another example is when
128. test results of the implemented system First an explanation about how the data was collected is provided afterward the test plan will be described and finally the experiments results will be presented 7 1 Data collecting Data Collection is an important aspect of any type of research study Inaccurate data collection can impact the results of a study and ultimately lead to invalid results The project group of aggression detecting had collected data using scenarios 11 The MMI project group used real train compartments to perform those scenarios Using multimodal cameras they recorded video and sound of actors acting out a certain predefined scenario There were two types of scenarios recorded Scenarios containing aggressive situation Scenarios containing non aggressive situation The scenarios performed were Shouting smoking hooligans beggar conductor mobile phone drunkard shoe on seat mother with child and graffiti and vandalism Chapter 7 Testing 4 Experiment results Later the videos were analyzed and for every video a number of notes were made these notes helped the design of the system a lot since they describe what is in the scene and what action is being performed by the actors 7 2 Test plan Usability is one of the important aspects which determines product s acceptance by users and its exhaustive use The ease of use is now becoming an increasingly important factor in the purchase of software A
129. the index finger and the small little finger all the way back from the thumb Another common gesture people can mix up in different countries is he is acting like he is crazy motion In North America and Europe they point to their heads and swirl their fingers clockwise to indicate such inferior unintelligent behavior If you do that in Japan however you make it look to locals as if you mean otherwise as if he is thinking intelligently However reversing the motion does indicate to a man native to say Tokyo or Nagasaki that he should correct his behavior to correct standards The point is clear different cultures call for 25 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis different body language standards We choose body language gestures that are used in Europe Giving the middle finger fighting gesture attack signals exposing oneself insulting gestures and no ticket gestures searching pockets knocking the head No running away from the conductor Finally since we have many people on the train compartments we have relations between people in the train shouting at throwing an object at hitting touching and invading one s private space 3 7 An analysis of the system needed One of the main goals of the system is to annotate the situation on the train compartment This is why we need a system to make that wish come true A good way to s
130. the large amount of noise occurring in the train Therefore we need to identify other aspects that humans use to detect aggression e g emotions shouting and location Humans detect aggression easily when he she knows the environment e g when a human sees two people fighting in a karate school he she knows this is part of the sport and its probably not a form of aggression but when he she sees two people fighting on a train then this means it is a form of aggression another example is when he she sees someone smoking on the street he she does not think about 18 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis aggression and that is different when someone is smoking in the train this might lead to aggression situation since smoking is not allowed on the train These examples imply that the environment where the aggressive or non aggressive situation arises is very important The human does not only use the environment to detect aggression the human detect objects such as a knife but simply detecting a knife is not enough e g a person holding a knife is not enough the human identify the activity the person holding the knife the person could be holding a knife to peel an apple which isn t an aggressive situation but holding a knife when the person is angry is a different situation In the examples above we used the word activity and we refer it to peeling an apple this
131. there Adding rules to each script should be located in the corresponding file A new rule e g for fighting should be located in the fighting script fight clp etc The rules form of the new rules should look similar to the rules that are already there e g New fighting rule If a beggar is holding a knife Then the fight level should go up by 60 The rule can be added in the fight clp as follow defrule fight rule 100 activity a_action hold objects o_description beggarclothing relation r_relation wear objects o_description knife gt assert update fighting 60 68 Video content analysis amp aggression detection system for a train environment Chapter 6 Implementation Different update rules can be used e update begging x Update the begging level by adding the x amount to the current level e update shouting x Update the shouting level by adding the x amount to the current level e update fighting Update the fighting level by adding the x amount to the current level e update fighting2 Update the fighting level by subtracting the x amount from the current level e update normal Update the normal level by adding the x amount to the current level e update normal Update the normal level by subtracting the x amount from the current level e update control Update the control level by adding the x amount to the current level 6 4 2 Adding new icons
132. tions The benefits of usability testing are e Overall quality of the software improves as the system becomes easy to use efficient and responsive e Shortens the learning curve and training efforts for new users and hence saves time for the user 7 2 3 User experience measures What to look for and how to ensure that usability is built in the system There are many points which should be kept in mind while writing test plans and creating test cases Every test will include usability measures such as learn ability effectiveness efficiency memorability and satisfaction Those measures can be distinguished and tested in particular evaluations Table 7 1 describes what every measurement is Table 7 1 Description of the usability measurements Measurement Description Time to learn here we will see how long it takes the e participant to learn how to use the system by making Learn ability ar o two similar tasks one at the beginning one at the end and compare the time needed to accomplish the task The product should be efficient and it features and options should be easily accessible The main UI should have all the major functions and options but at Efficiency amp the same time it should not be cluttered in menus All Aeneane the readily used options and important menus should be easily accessible While testing any feature one must check how many steps are required to complete a task Tester should ascertain w
133. ts that are always in the train compartment Table 3 4 provides a list of the static objects Table 3 4 Static objects that can be found on a train compartment Object Description Seats are found in the hallway and in the compartment They may be solid or folding benches Train seat Aggressive use drawing upon putting feet on occupying more then one seat and putting objects on the seat when there are not many seats available Found in compartments on the wall or on the back of a bench Train table where it can be pulled out Aggressive use drawing upon mutilation and folding in aggressive manner 3 5 An analysis of the train compartment The dynamic context The dynamic objects are objects that are moving in other words objects that are not always on a train compartment The dynamic objects in this environment can be split into two categories the first one is objects that are brought by people onto the train table 3 5 and the second one is the people who are on the train table 3 6 Table 3 5 Objects that are brought by passengers Object Description Bag Bags should be stored on people s laps in the luggage rack or under the benches where sometimes a storing room is provided Putting 21 Video content analysis amp aggression detection system for a train environment Chapter 3 Problem analysis bags on a bench occupies seating space and may be considered an act of aggre
134. ture showing the different components and how they are related 5 2 The GUI component The GUI component contains classes for the user interface The design is based on use cases where we defined what the user interaction with the system is and what functionalities the system should provide We try to keep the user interface design simple and easy to use so that users quickly see the breadth of their options grasp how to achieve their goals and do their work Work is carefully and continuously saved with full option for the user to undo any activity at any time The GUI design is close to most windows applications design with a file menu where one can save load work reset exit etc Furthermore we use icons to present the people types objects emotions and relations The use of icons makes it easy to construct annotations according the annotating language that was described in Chapter 4 The GUI cooperates with the Validator to make sure that the user cannot make annotating sentences that are incorrect according to the syntax of the language Note that the icons are used purely to present concepts in the annotation language in a visual way we also could use a normal text for that function and this means that icons are unrelated to the annotating language we use 44 Video content analysis amp aggression detection system for a train environment Chapter 5 Design Following the annotation language we divided the icons into groups
135. uation This implies that knowing the people plays an important role in the human aggression detection Another important issue is that humans have memory and they can easily memorize the changes in time e g a person was drinking beer ten minutes ago in the train and they saw him shouting at others later they see that person again and sees him walking towards them they immediately remember him and they detect a possible aggression Thus the human reasons about aggressive situations using the knowledge of the environment the people types objects the activity the emotional state of the person doing the act of aggression and finally they use their memory to identify people 4 A computer system needs the same knowledge about the environment to reason about aggression For that reason a world model is needed The context of that world model should be the same set of information the human uses to detect aggression which is people types activity objects relations and emotions 4 AGGRESSION DETECTING SYSTEM A good way to start the analysis about what is needed in the system is to sketch what a typical annotating session will look like The user needs to see the video file he will annotate this means that the system must be able to open a video file Since the annotation will be on static images not moving video the user must be able to load a set of frames pictures that were captured from a video file The annotatin
136. vity Table 5 3 Description of the objects fact list oid The unique Id of the object o_description Description of the object o_is_visible Whether the object is visible or not Default is true o_makes_sound Making sound or not o_sound_volume The sound level o_framenumber The frame of the object o_endframe The frame where the object disappear from the scene Table 5 4 Description of the relation fact list r_id The unique Id of the relation r_object The human Id involving in the relation r_relation The relation type r_subject The object involving in the relation r_is_visible Whether the relation is visible or not Default is true r_makes_sound Making sound or not r_sound_volume The sound level 5 6 Class diagrams 50 Video content analysis amp aggression detection system for a train environment Chapter 5 Design Class diagrams identify the class structure of a system including the properties and methods of each class In addition depicted are the various relationships that can exist between classes such as an inheritance relationship The Class diagram is one of the most widely used diagrams from the UML specification Since there are many classes in the package we will discuss the important classes only Moreover we grouped related classes into packages as shown in figure 5 5 XML IO
137. w session The annotating can be started by first choosing the desired frame to annotate usually the annotating start in frame 1 Second choose people type The choice is between Passenger A normal passenger of a train can be used for male and female Hooligan football supporter that makes troubles Beggar someone who asks for money Artist someone who makes an act sings and ask for money Conductor the conductor of the train Police officer a police a cop Note that on almost all buttons in the system there is a mouse over so e g when you move your mouse over the passenger button you will see A passenger When you click on a people button you have to enter a small description about the person you choose and then a rectangle of the location of that person has to be drawn using the mouse To draw a rectangle click and hold the left button of the mouse on the left top corner of the person move the mouse to the right bottom corner and release the button The Undo button can be used when you draw a wrong rectangle The selected person icon will be displayed right to the big frame picture with a mouse over tip of the description and the frames he is in figure H 1 128 Video content analysis amp aggression detection system for a train environment ie Frame Concept q 7 OlHooligan is walking Hooligan with white shirt Frames 0
138. y There is an 80 chance that a person will be the victim of a violent crime during his or her lifetime 3 Researchers claim that we are coming closer to predicting from a brain scan or a blood test whether a person is at risk for committing an act of violence Ethical complications aside a closer look at the neurobiology of aggression shows why we are unlikely to find a conclusive test for potential violent behavior While there are many biological factors associated with aggression their predictive value remains still quite low The area from which all emotion originates is the brain While scientists continue to test various areas of the brain for their effects on aggression two areas that directly regulate or affect aggression have been found The amygdale has been shown to be an area that causes aggression Stimulation of the amygdale results in augmented aggressive behavior Bauman et al 2006 16 Another area the hypothalamus figure 2 1 is believed to serve a regulatory role in aggression The hypothalamus has been shown to cause aggressive behavior when electrically stimulated but more importantly has receptors that help determine aggression levels based on their interactions with the neurotransmitters serotonin and vasopressin 10 Video content analysis amp aggression detection system for a train environment Chapter 2 Background The hypothalamus and pituitary gland are important parts of the brains limbic system as
139. y pram knife tickets cell phone doll Mp3 player money and paint People on the train conductor passenger beggar artist and hooligan Activities amp behaviors We analyzed the actions that can be made on a train and we found sitting smoking talking walking hitting checking tickets putting holding shouting and putting legs on the seat Combinations between these actions and objects that are found aggression situation can be created like A hooligan is holding a knife lt very unsafe situation gt A hooligan is holding paint lt damaging the interior gt A passenger is smoking a cigarette lt may cause a fight gt 6 ARCHITECTURE The system has four main components figure 1 The GUI allows the user to interact with the system All the user commands will go through the GUI The user interface contains facilities for the user to perform all the actions that were discussed earlier The Validator validates the annotations resulting from user actions Only valid annotations are allowed to the reasoning component invalid annotations are rejected with a message The Validator sends the valid annotations as facts into to the reasoning part where Jess will reason about the aggression and send the output back to the GUI where the output is displayed The XML IO is used for the saving and loading the XML file It is also used for generating JPEG images of the locations of the people or objects as sp
140. yellow shirt and click Ok 7 Read the popup message and draw a rectangle around the beggar using the mouse and click ok 8 Click on the action tab and choose walking 9 Click on Add 10 Go to frame 2 11 Choose the beggar you just annotated and draw a rectangle in his new location 12 In the action tab choose Holding and in the objects tab choose money and click on add 13 Click on frame 3 Choose in the human tab a passenger 14 Type as description long hair and draw a rectangle around the passenger with the long hair 15 In the action tab choose sitting and in the object tab a seat And click on Add 16 In frame 4 choose the passenger with the long hair locate the new location and in the expression tab choose afraid and then add 17 In frame 5 the passenger with the long hair is angry 18 In frame 6 the male passenger in the white shirt is surprised 19 In frame 7 the passenger with long hair is fighting 20 In frame 7 the beggar with long hair is fighting 21 In frame 8 the passenger in the white shirt is shouting 22 In frame 9 the beggar is afraid 23 In frame 10 the beggar is walking 24 view captured images 25 save XML file 26 exit We want you to annotate on your own now 27 Start the application using F6 28 Open video file hooligan mpg located in the desktop 29 Open frames using the file O1_associale_hooligan txt 30 Start annotating until you r

Video content analysis & aggression detection system for a train

Contents

Download Pdf Manuals

Related Search

Related Contents

Video content analysis &amp; aggression detection system for a train

Contents

Download Pdf Manuals

Related Search

Related Contents

Video content analysis & aggression detection system for a train