Full report - ECE - Cornell University
Contents
1. Timer the number filled in by user indicates the duration of the program when time is up the program stops automatically in hour File Name name of file to save data if the entered file name does not exist the program creates a folder with the name Acquire ROI acquire region of interest When clicked a snapshot of the camera will show up for the user to manually select the tank so the program only focus on the selected the area tank and anything moves outside tank will be ignored At the same time the background will be initialized Fish position when clicked a snapshot will show up for the user to manually click on the fish s location This is designed because in this project fish rarely moves when program started fish s position could not be found by program since nothing moves so we need to manually locate the fish Video Preview preview the real time video captured by camera Start tracking program starts to track when clicked STOP Program to stop the program manually 5 Test and Result In order to meet the requirement of long time reliability we tested this program 4 times in real setting The test environment was 1 Ze 3 4 Room temperature 20 C PC CUP Intel R Core TM 17 2620M CPU 2 70GHz PC memory 4 00 GB Frame number 20 frames second All videos were recorded and more details are shown in following table 19 Duration2. achieve that goal is to measure the moved distance in a fixed period Since everything is based on pixels and a fair comparison should be based on inches we define a parameter called PixelLength to convert pixels to inches PixelLength is the length in inch two adjacent pixels stands for which is used to convert the pixel distance to real distance in tank For example if the number of pixels fish traversed is N in a time period then the fish moved about N PixelLenght inches in the tank PixelLength is calculated based on the size of tanks as below tank width tank height PixelLength _ elLe gt rer width of ROI pixel height of ROI 2 5 eee 16 Where ROI means region of interest which is selected by user Basically locations in pixels are stored to hard drive every minute and release the memory they occupied which could prevent loss of data when unexpected crash happens and also being memory friendly for long time recording Besides total moving distances of fish for each minute and associated time periods will be stored to hard drive also When program stops what we saved is the location of fish on every frame and the moving distance of the fish for every minute of the monitoring duration The fish s location which is array of x y is saved in mat format Matlab file the array of distance for every minute is saved in both mat file and xls file all of them can be reused
3. Files of type 30
4. Hour Date Total Frames Analyzed Comment 1 24 3 22 2012 1 7 10 1 camera 2 9 4 6 2012 1 3 10 2 cameras 3 72 4 13 2012 1 107 2 cameras 4 24 5 2 2012 5 2 10 3 cameras Table 2 Duration and Data The test result for the last test with 3 cameras is shown below Starting From 2012 5 2 13 59 250 T T T T T q 200 150 F a 100 50b 200 400 600 800 1000 1600 Time Min Figure 7 test result for fish 1 20 Distance inch Starting From 2012 5 2 13 59 120 T T T T T T T al 100 F a 80 F 60 F Distance inch 20 F L L 0 200 400 600 800 1000 1200 1400 1600 Time Min Figure test result for fish 2 Starting From 2012 5 2 13 59 1 80 T T T T T T A 160 F al 140 J 120 F a 100 F El 804 El 40 201 7 0 200 400 600 800 1000 1200 1400 1600 Time Min Figure 9 test result for fish 3 21 We met static error problem Section 3 3 2 accumulated error problem Section 3 3 3 and memory problem when doing tests in lab Memory occupation kept increasing even we deleted everything we used We spent couple of nights debugging the program line by line Finally memory problem was perfectly solved after we re wrote a build in Matlab function called regionprops which kept eating memories regionprops is a function that we used to find the area of maximum pixel intensity however it also has a lot of other function
5. fish made and so on in this project we focus on the distances 2 Requirements In order for the program to function well the following requirements should be met O accuracy Accuracy is crucial since any deviation of the acquired location from the actual location would give us inaccurate data which leads to inaccurate distances In this program we should make sure the red dot acquired fish location always stays on the fish without sudden jump O reliable for 24 hours This program should continuously run for at least 24 hours so any minor flaws may accumulate during this long time period and easily leads to instability to the program and burden to the system Especially this program should be able to release useless memory and be memory friendly O 3 fish in 3 tanks This program should be able to monitor 3 fish in 3 different tanks In order to reach a statistical result this program should provide some efficiency O user friendly Our program is mainly designed for non software background users Therefore a user friendly interface which is easy to use and understood is essential O easy data access for later analysis The data acquired by this program should be saved and accessible in a later time so as to support summation and reanalysis 3 Approach Our project requires both hardware and software support we need hardware to obtain video input and software program to process the input video to find fish locati
6. so we can analyze the data to master fish s activity rhythm Individual Contribution Xinjia Huan O Researched on purchase and constructed the hardware system O Accumulated error problem O Memory problem O Designed data analysis system Zhao Wei O Researched and created the main functions in tracking algorithm O Static error problem O Designed user interface Designed data analysis system Executive Summary This project is the creation of software program to satisfy the specific needs of graduate students in Department of Neurobiology and Behavior to better study the behavior of a special kind of fish The program created in this project is called Fishtrack3 which is run in Matlab to take advantage of 1ts image processing functionality The program is used to study the locomotor rhythm of a sound producing teleost the Plainfin Midshipman Currently most of the methods used to study the daily activity of the fish are very crude and inefficient a more sophisticated and accurate way is to design a vision based real time motion capture system to monitor daily locomotor rhythm of the fish which is what we did in this project The project is a concatenation of hardware and software it takes a infrared camera based video as input followed by a reliable program which can efficiently analyze and save data The program is capable of locating the fish on every frame from the video thus the fish s every move and activity is
7. 3 4 2 Analysis To meet the requirements of later analysis we developed a program for data plotting Bar plot is the default way to present data And the time interval to save data could be changed based on the user s need The bar plot is given below j Figure 1 o amp amp File Edit View Insert Tools Desktop Window Help 90 T T Starting From 2012 4 21 23 18 T T Distance inch o 20 40 60 80 100 120 140 Time min Save xis Ene on Jason Pot Fie Name s Figure 5 Data analysis for fish distance every 5 minutes __ z _t_mrTrI Pm E y 2 Each bar means the distance fish moved in the selected time period in this example it shows the distances fish moved every 5 minutes These data could also be saved into a file in xls format if the user wants 4 User Interface Control In this project a user interface UT is required to successfully run the program since some variable values need to be set manually by user like tracking duration directory and file name to save data We designed the UI such that user can easily understand and use The following figure shows the user interface and UI control r File Edit View Inset Tools Desktop Window Help AcquireROl Fish position Timer File Name Video Preview STOP Program 2 test Figure 6 User Interface gt 18 There are several controls on the interface all with different functionalities 1
8. FISH TRACKING A Design Project Report Presented to the School of Electrical and Computer Engineering of Cornell University in Partial Fulfillment of the Requirement for the Degree of Master of Engineering Electrical and Computer Engineering Submitted by Xinjia Huan Zhao Wei MEng Field Advisor Dr Bruce Land MEng Outside Advisor Ni Feng Degree Date May 2012 Abstract Master of Engineering Program School of Electrical and Computer Engineering Cornell University Design Project Report Project Title Fish Tracking Author Zhao Wei amp Xinjia Huan Abstract In this project we constructed a vision based real time motion capture system to monitor daily locomotor rhythm of Plainfin Midshipman a vocal fish We used infrared cameras to collect input data and a GUI based Matlab program to display and analyze useful information We want to monitor midshipman s locomotion over 24 hour period of time but it is almost impossible for human to manually record for such a long time a reasonable approach is to design a vision based self recording system which is reliable and user friendly Also infrared camera does not impose any physical restrictions on fish which provides a natural way of measuring fish s daily activity In this system every frame of the picture captured by camera is treated as input of Matlab Program With this program we can find the fish s location on every frame Those locations will be saved in a file
9. a of noise also has pixel intensity so we can set a threshold value of pixel intensity if the intensity of area is below this threshold value then they are regarded as noise and ignored In this project we set the threshold value as 5 10 actually most noises could be eliminated and it worked fine Finally the setup of hardware also has effect on noise reduction Properly setup would increase the immunity of the program to noises Refer to Figure 1 to set up the equipments well following steps should be followed 1 Camera should be in the same horizon level with corresponding tank and directly face the tank 2 Backlight should be placed at the back of tank in the same horizon level with tank Choose proper distance with tank so that the whole tank could be lighted up 3 Diffuser should cover the whole back side of tank 4 Light spot should be avoided if not no objects should be in the highlighted area since a small movement of objects in highlighted area would cause large pixel intensity 3 3 2 Static Error Static error exists because most of the time fish does not move so it s difficult to find the fish s location just by subtracting current frame with background When fish does not move there exist some frames in which the calculated location is not for fish but for some other moving object which has largest intensity in that frame it could be a bubble or water ripple An obvious solution is to define a flag with va
10. alities It s recursive and each time called reginprops calculates lots of variables even though what we need is only one or two of them and fails to release those variables Since this is not a typical functional error traditional debug cannot find the problem so we tested every module and put them into a loop to check the memory finally found the origin to memory problem We re wrote a function similar to regionprops but more simplified which solves the memory problem Our test showed that on the PC when running the program for 3 cameras 25 CPU was used 35 memory used the memory occupied by Matlab was less than 1GB The occupancy of CPU and Memory keeps constant during the monitoring and no accumulation happened Depends on these we believe our program can support 10 15 cameras simultaneously on this PC but it also depends on the data processing capability of Matlab It showed that this program functions well for long time duration the longest test lasted for 72 hours Besides accuracy was proved after we compared stored data with recorded video A screenshot of the 3 camera monitoring is 22 FE z ala x Edit View Insert Tools Desktop Window Help EA E E EA E E E E Wednesda 2 Figure 10 3 fish in 3 tanks 6 Conclusion This program works well and all requirements are met O Accuracy comparison between recorded videos and stored data shows high accuracy O Reliability the 4 tests showed ver
11. ecommended e Diffuser A diffuser is recommended to stick at the back of tank to smoothen the light Without a diffuser the camera will see a big light spot and everywhere else black which affects the accuracy A diffuser can distribute the light evenly and light up the whole tank A diffuser could be something like a translucent white plate in this project we used food mat and it worked good TV USB adaptor To transfer the data from camera to computer also to transform analog signal to digital signal for program to process Bill of Materials Name Serial Number Unit Price PC Infrared camera SVAT SVVU5 29 99 Infrared LED backlight IR045 48 95 TV USB adaptor Hauppauge USB Live2 44 95 Table 1 Bill of Materials AA OOoooo nn 9 3 2 Tracking Algorithm The basic idea of the tracking algorithm is to subtract well defined background Ig n from current frame I n resulted in a subtracted frame I n Then we use Matlab to deal with J n to eliminate noises in recording and find the pixel of maximum intensity in I n then find out the centroid of the area with maximum intensity to locate the fish The subtracted frame I n could be like the figure below the circular area means the differences between background and current frame which could be potential fish Among them we then find the area with maximum intensity and find the centroid of this area The centr
12. el with corresponding tank place backlight on the back of tank choose proper distance so that the whole tank could be lighted up place diffuse material on the back of tank try to smoothen light source and avoiding light spot Be cautious to 1 place camera backlight and tank at the same horizontal level 2 try to avoid light spot if can t make sure no objects in highlighted area program is more sensitive to objects in highlighted area 3 FishTrack 1 Run Fishtrack3 m 2 Choose directory to save data Set a directory to save data FE Desktop A Libraries 2 Homegroup B Kevin jE Computer Sig Network B Library Eolder test Make New Folder Cancel bi 3 Set Timer enter the time duration you want to monitor the fish For example for 5 hours enter 5 File Edit View Insert Tools Desktop Window Acquire ROI Fish position Video Preview STOP Program File Edit View Inset Tools Desktop Window Help Acquire ROI Fish position Timer File Name m Start tracking Video Preview STOP Program 26 5 Click Video Preview to preview the real time video and adjust the position of camera 03 45 20 055 529x346 Waiting for manual TRIGGER 1 of 1 6 Click Acquire ROI use mouse to drag the square to select the region of interest and then double click ROI should include the whole tank and be as large as possible File Edit View Inse
13. he video recorded for the first test we found that even the fish is not moving the red dot representing the stored location varied in a small range Although the variation is quite small which is usually couple of pixels the overall accumulated error could be very large for a long time duration when come to the total distance fish moved The accumulated error occurred because of the change of the centroid This happened when fish moves its tail but stays in the same location We attacked this problem with following approach Set first location point as a reference point and put its successive points within 5 pixels in one subset Then the first point which is out of 5 pixel range would be the next reference point and get another associated subset Repeat these procedures until allocate all location points into different subsets Then take average for each subset the resulting set of points are the new modified location points Here is a plot which compares the original location points with modified location points This plot is in 3D format z axis represents time 15 File Edit View Insert Tools Desktop Window Help DGdAds 1 88 9084 2408 20 original modified T 2 6600 v 250 200 0 150 x pixel Figure 4 Comparison between original points blue and modified points red 3 4 Data Storage and Analysis 3 4 1 Data Storage Our objective is to measure the fish s activity level in different time periods A simple way to
14. lue one when fish is moving and vice versa When the flag is one do the calculation and find location otherwise keep the location the same as last frame However it s not that easy to directly define such a flag to identify a moving fish from an image An alternative approach is to define such a flag in an indirect way We define a threshold value of intensity T and pixel distance D Fish is the largest moving object we could easily define a threshold value T which is larger than normal noise but smaller than the fish Noise occurs randomly on the whole image if the calculated location is further than a distance D from previous location and at the same time the intensity is below T then this location would be AAAAAAAAAHHAHAHAHAANA gt S gt S gt AAE o 14 ignored and treated as noise and the flag would be set to zero In this scenario the value of location remains the same as last frame Otherwise the program would do calculation and update the location This mechanism is also intuitive because a sudden jump of location is not expected since fish moves continuously Then the major issue is how to define such threshold T We measured average pixel difference between fish and background is SO black is 0 white is 255 and smallest size of fish is at least 10 pixels by 12 pixels camera is 360 pixels by 240 pixels So the resulting T is 3 10 We tested this T and everything was fine 3 3 3 Accumulated Error When reviewing t
15. nsity of the subset image is calculated and used to calculate the object s x and y locations The equations are given below L A 221 asupln x n yay n T Re n x L 3 ji Asub HA iT sub n max y n y in IX n Je L 4 Asub ji Here i and j correspond to row and column indices respectively Ly and L are the dimensions of max IN the subset image and is the pixel of maximum intensity in the subset image Below is a sample image the red dot on fish is the calculated centroid which is the fish s location AD E figure 1 File Edt View insert Tools Desktop Window Help Acquire RO Timer File Name Video Pres E 120000 322012 Figure 3 Sample Image 3 3 Error Elimination Fish could be found in ideal environment based on the algorithm described in Section 3 2 Unfortunately environment is not always as good as we expected There are some facts which could affect the accuracy of the tracking algorithm 3 3 1 Environmental Noise Noise is inevitable in this project Major noises could come from the process of transforming analog signal to digital signal of the input video small variation of light water ripple and bubble Noises have significant influence in accuracy so eliminating them seems crucial As we can see from Figure 2 on page 10 other than the area of maximum intensity which is the fish there is some other area which is referred as noise go The are
16. oid is the calculated location of fish This algorithm is based on the fact that fish is the largest moving object in the tank Largest not only means large in area but also means large magnitude in intensity more details about intensity is in Section 3 2 2 Maximum Intensity Figure 2 Subtracted frame I n with area of interest Since a frame is represented as a two dimensional matrix in Matlab each number in the matrix represents a pixel and the value of the number represents the color in that pixel so the subtraction between background and current frame is actually subtraction between two matrixes Av 10 So in the above figure darker area means larger difference between background and current frame As stated above the basic idea is to subtract current frame from updating background But we wish this program could be more sensitive to fish such that it would give more weights on black object A direct way is to enlarge the difference between fish and background based on the fact that fish is black and background is white So we squared the result and defined it as I n which is calculated below 1 n n Ig m Y 1 3 2 1 Background Frame The very first problem is how to efficiently and accurately define a background We used the algorithm from Analysis of the Trajectory of Drosophila melanogaster in a Circular Open Field Arena written by Dan Valente an Golani and Partha P Mitra The backgr
17. on 3 1 Hardware Setup Since the main challenge of this project is to develop a sophisticated tracking algorithm we wish we could spend as less time as possible on hardware system As a result we focused on those commercially available off the shelf products for hardware setup After a little bit research we decided to construct the hardware system as follows Camera fs Backlight Figure 1 Hardware Setup The camera is placed to face the side of tank with the largest area which provides a more broad view of the fish Ideally placing another camera on top of the tank will provide more information about the fish in great detail in 3 D but that is more complex and requires a lot more computation we want to simplify the project since we only care about fish s activity level so we use one camera In this setup we need a PC an infrared camera an infrared LED backlight a diffuser and a TV USB adaptor PC To process the input video and save the acquired data from program O Infrared camera To monitor the 24 hour activity of the fish and provide video input for program Infrared camera can capture the fish during both day and night O Infrared LED backlight To provide infrared light to highlight the fish s location for camera also infrared light does not affect the fish s daily activity rhythm The LED backlight should be strong enough so as to penetrate the tank a LED array consists of hundreds of LEDs is r
18. ound can be acquired first by averaging the first 20 frames and then update the background through running average which means we will combine the current background with current frame to get new background the equation can be expressed as Ig n 1 alg n 1 a I n 2 Where 0 9 lt a lt l Besides a subset image Is around the location of fish is extracted from the new background Ig n 1 and replaced with the pixels of the same area in the original background Ig n This process prevents the fish been included into background 3 2 2 Area of Maximum Intensity In order to simplify the calculation we converted frames to gray based images white and black only from RGB based images red green and blue As a result calculation could be based on _ xaj image intensity rather than relative ratio of RGB As shown in Figure 3 fish is black and background is white in such setup After defining background the next step is to find the maximum intensity area The intensity is calculated by summing up the square of the pixel number see equation 1 in the above area So the maximum intensity is associated both with magnitude of area and darkness of the area The area with the largest sum of squared pixel value is our target area Next we find the centroid of this target area as the location of fish A subset image I sp around this pixel is extracted where I sy means our target area The center of inte
19. rt Tools Desktop Window Help JOaas k lt 8o0es4 a 08 e0 27 7 Click Fish position to manually point out the fish position 8 Click Start Tracking the program will start to monitor the fish 9 Click STOP Program to terminate 4 FishTrackPlot 1 run FishTrackPlot m 2 choose distancex mat must be distancex 28 4 14 2012 5 26 PM 4 top_Cyclel 4 14 2012 6 26 PM 4 top_Cycle2 4 14 2012 7 26 PM 4 top_Cycle3 4 14 2012 8 26 PM 4 top_Cycled 4 14 2012 9 26 PM 4 top_Cycle5 4 14 2012 10 26 PM 4 top_Cycle6 4 14 2012 11 26 PM 4 top_Cycle7 4 15 2012 12 26 AM 4 top_Cycle8 4 15 2012 1 26 AM 4 top_Cycle9 4 15 2012 2 26 AM 4 top_CyclelO 4 15 2012 3 26 AM E BE KHKH42AA S EEE 4 File name distance1 bd Files of type MAT ies mat al 3 set Interval must be integer 4 set bar and envelope 5 click Plot 29 File Edit View Insert Tools Desktop Window Help Starting From 2012 4 14 17 26 A peveeieec pace TL TTET A AS AAA ARS AR SSEAERESE SE RS HERESEESSESE A A EA AA PRA RO pean ey A a el o il is a a ERE ccc Distance inch SSS oS eee back to step 3 set another Interval and then Plot set File Name after click Plot modified data could be saved as excel file by click Save xls File Name and Plot must be set before click Save xls figure could be saved by menu bar gt File gt Save As could be jpg or fig by choosing from
20. under monitoring it can also analyze these data to provide plenty of information about the fish like during what time period the fish is most active We designed the program to support tracking 3 fish simultaneously for 24 hour period at the same time be system and memory friendly Accuracy is the most important requirement for this program at all stages of the project other requirements include reliability user friendly interface We are pleased to say they are all well met Analysis and study of animal behavior has always been a big topic in biology this program should see good use for this area and 1t was made to be easily extendable for other functionalities Table of Contents 1 Introd ction raro 6 REQUIEM ENES ran a oe a aa sr A a Oaa O R ao aN 7 EESE E EEE T AE T R Ge ne 8 Jet Hardware SCWIP ee ien A E A os 8 3 2 ela O a a a aad 10 33 Error EMINA ON ss 405443 A ide 13 3 4 Data Storage MA AA E EO EEE EES 16 4 User Interface Control aciei AA RIAs 18 Lescano Rel cada t 19 AECI ON o do O 23 TACO sd 24 RELE A ove ema E onan eames 24 APPENdIX OE 25 1 Introduction The main focus of this project is to study the locomotor rhythm of a sound producing teleost the Plainfin Midshipman The male midshipman shows regular day night behavior cycle during the summer breeding season including vibrating muscles to produce long duration courtship calls at night to attract females and remain quiescent during the day We want to study
21. whether the fish also performs a day night activity cycle during this breeding season It is believed that the midshipman is more active during night than day In order to reach a statistical result we need to monitor multiple fish each with multiple times for 24 hour period which incurs huge amount of data and calculation this leads us to the idea of designing a system concatenated with software and hardware to monitor the fish s activity for at least 24 hours The most straightforward configuration for such a system would be taking a camera based video as input followed by a reliable program which could efficiently analyze and save data Since visible light source would change the activity rhythm of midshipman an infrared based video recording camera is recommended in this system Other than that a Graphical User Interface GUI based program which is easy to read and use is also recommended in this system The video input from camera is analyzed on a frame base this program locates the fish s location on every frame and all the locations during the 24 hours will be saved into a file We will then analyze these data to find the distances the fish moved during the 24 hours and find out during which time period the fish moved the most which corresponds to the most active time period Other behaviors of the fish could also be analyzed after collecting the data like in which area of the tank the fish is most likely be how many turnings the
22. y good reliability of the program it supports 72 hour recording and does not occupy much memory e 3 fish in 3 tanks we could track 3 fish in 3 different tanks simultaneously besides more fish could be tracked just by making a slight change to the program but that is not the scope of the project we focus on 3 fish in 3 tanks here O User friendly the program interface is easy to understand and use for non software background users O Easy data access for later use all the fish locations will be saved into files and fish distances saved into a different file These data is accessible and could be used any time 23 7 Acknowledgements 1 We would like to acknowledge the advice and guidance of our ECE advisor Bruce Land and outside ECE advisor Ni Feng 2 We also want to thank Dawnis Chow from Department of Neurobiology and Behavior for sharing information and materials with us for the project 8 Reference 1 Dan Valente an Golani Partha P Mitra 2007 Analysis of the Trajectory of Drosophila melanogaster in a Circular Open Field Arena PLoS ONE 2 10 e1083 24 Appendix User Manual 1 About This program was created to meet the specific needs of Cornell s Bass lab in the Department of Neurobiology and Behavior All Matlab functions were created and tested in the Matlab 2010b version 2 Hardware Setup connect cameras to PC via adapter properly place cameras 1t should be in the same horizontal lev
Download Pdf Manuals
Related Search