fulltext - DiVA Portal
Contents
1. Estimate Error Pitch Estimate Error Yaw es ee ee a ro Estimate Error Roll 0 200 400 600 800 1000 Figure 4 15 The estimated relative orientation between camera 1 and camera 2 is plotted for each time step Yaw axis test overshoot Estimate Pitch 1 Estimate Yaw 1 Estimate Roll 1 Estimate Pitch 2 Estimate Yaw 2 Estimate Roll 2 Figure 4 16 The estimated orientations relative to the baseline be tween the cameras is plotted for each time step 40 4 2 Test Results from Live Video Streams 10 Yaw axis test overshoot T Estimate Error Pitch Estimate Error Yaw Estimate Error Roll Figure 4 17 The estimated relative orientation between camera 1 and camera 2 is plotted for each time step Al 5 Discussion Overall the chosen calibration model based on minimizing the dis tances between points and their corresponding epipolar lines seemed to be working well The tests have proved the model to be quite precise in some aspects while weaknesses in others have been high lighted This chapter will be divided into sections of discussions on these weaknesses 5 1 Sensitivity of the Estimated Angles The simulation results clearly states that all angles can be estimated with good accuracy The relative angle between the two cameras can be estimated with as low as 0 2 degrees deviation The simulation also shows that the difference in posit2. NTNU Trondheim Norwegian University of Science and Technology Continuous Calibration of 3D Vision Systems John Magne Rge Master of Science in Engineering Cybernetics Submission date June 2013 Supervisor Tor Engebret Onshus ITK Co supervisor Havard Knappskog NOV Norwegian University of Science and Technology Department of Engineering Cybernetics Problem Definition Background 3D vision systems consist of two or more cameras used to recover three dimensional images The applications are numerous spanning from home entertainment to industrial applications where position of objects may be measured Accurate 3D reconstruction requires precise estimation of calibration parameters Such parameters involve rotation and translation be tween cameras as well as lens and camera geometry Although an ac curate calibration may be performed at a given time these parameters are subject to drift or change due to vibration material degradation and misalignment Therefore long term reliable operation depends on updating the calibration data regularly or continuously Main tasks e Background study e Evaluate common methods for multi camera calibration e Investigate develop a method for continuous calibration of mul tiple cameras e Implement computer software to process multiple streams of video e Perform continuous calibration Supervisor Tor Engebret Onshus ITK Co supervisor Havard Knappskog NOV Preface
3. The workflow for the application is basic It runs continuously as fast as the hardware allows it to looping through the different steps of the system structure The graphical user interface allows the user to choose various options and a short user manual is written in Appendix 6 3 4 Test Procedure for Continuous Calibration The test procedure is split up in two parts a simulation and a live test using web cameras The simulation creates virtual feature points based on calculated intrinsic camera parameters and camera rotation based on user input The live test on the other hand uses live images in which it detects and matches features before finally calculating camera rotations 3 4 1 Simulation Procedure Simulating a real system is a good way of checking the correctness of the application A simulation successfully removes the unwanted random variables which is applied by the real world This means that the negative effects of bad feature detection feature matching image noise and more is non existent Thirty two 3D points which are spread out in box like shape are gen erated and equation 2 2 the pinhole camera model is used to generate simulated feature points for both the cameras The feature points will appear slightly different in each of the camera frames based on their position and orientation Figure 3 4 3 5 3 6 and 3 7 show how the 19 Chapter 3 Methods different rotations affect the perspective f
4. Estimate Pitch 2 gata aa k iN cD Estimate Yaw 2 i Ha uk ed V NI Estimate Roll 2 Figure 4 12 The estimated orientations relative to the baseline be tween the cameras is plotted for each time step Pitch axis test Estimate Error Pitch Estimate Error Yaw Estimate Error Roll Figure 4 13 The estimated relative orientation between camera 1 and camera 2 is plotted for each time step 38 4 2 Test Results from Live Video Streams 4 2 3 Yaw Tests There are two yaw tests shown The yaw test was especially sensitive and there was therefore a need of showing the different results In Figure 4 14 and 4 15 the results of the first test is graphed Here the estimate doesn t return completely to 0 degrees relative yaw angle but ends up at about 6 degrees Figure 4 16 and 4 17 shows the second test where the estimate of relative angle overshoots and ends up at 10 degrees These tests also show the same problem with relative position estimates as the simulations and roll test Yaw axis test Estimate Pitch 1 Estimate Yaw 1 R e T Estimate Roll 1 TREET O IIT Estimate Pitch 2 Estimate Yaw 2 Estimate Roll 2 SPOT 0 200 400 600 800 1000 Figure 4 14 The estimated orientations relative to the baseline be tween the cameras is plotted for each time step 39 Chapter 4 Results Yaw axis test
5. When I started out on this assignment it seemed like a simple enough task After a few rounds of literature searches with unhelpful out comes and non working system implementations my opinions changed Yet with guidance and persistence useful literature was found good models were developed and software implemented My perspective has been broadened and I can see that the potential of improvement is tremendous First of all I would like to thank co supervisor Havard Knappskog at National Oilwell Varco NOV for being helpful in every aspect regarding this task He has been discussing various methods of cali bration and has helped me by giving feedback on the report several times I am also grateful for being able to work on my system at NOV s offices in Trondheim and getting to meet the people there Thanks to my supervisor Professor Tor Engebret Onshus for helping me construct a time schedule and giving me advice whenever unsus pected situations occurred Last but not least thanks to Vivi Rygnestad Helgesen for continu ously pushing me forward and helping me finalize this report il Abstract and Conclusion This report is about continuous calibration of 3D camera systems It seeks to make it possible for two freely moving cameras to know each others relative position as long as they are monitoring a common frame The short conclusion of this paper is that it is possible to a certain extent The system presented is capabl
6. d 0 0 Figure 3 2 The world coordinate system is defined to have its origin in C1 the x axis parallel to the baseline from Ci and Cs and the z axis is defined such that the pitch of camera 1 is zero Cra Ri Co C1 When the epipolar line for a given point is defined the distance from the point to the line has to be calculated When the line is defined as it is in equation 2 7 the shortest distance from the point to the line can be computed by using the following expression jax by Va b We can rewrite this equation to define a function which accepts the line and point in vector form a b c fa y SS When the equation for the epipolar line and the distance from a line to a point is in place a optimization problem can be defined In this implementation the sum of the distances is used Each point pair m1 M2 contributes to the sum with the distance from m to its epipolar line min f Ri R gt d Poa Ri R My Mz Ry Ro d x y M M 17 Chapter 3 Methods For solving the optimization problem a open source solver is used 7 The solver used estimates the gradient of the function to be optimized and runs until it reaches a local minima This might be a problem if the problem has multiple local minima No feature matcher is perfect and the method had to include som kind of filter to remove obvious mismatches This was done by assuming that the deviation from the pre
7. KoR Ri Kym F K Ra Ci Ca p2M2 p1 M Pe 2 2 5 Equation 2 5 describes what is called the epipolar line Am is the vanishing point of m in camera 2 while es is the location of camera 1 in camera 2 s view Basically this means that a given point m in camera 1 has to lie on the line between the vanishing point of m in camera 2 and the epipole of camera 1 in camera 2 This is shown graphically in Figure 2 5 z I Pa I 7 I I A i I 7 I I i 7 I I M M a 1 e ros me i N 1 i amp r A N UA image 1 oe N image 2 imag 1 Ai image 2 O C C Figure 2 5 On the left the relation between the point m in camera 1 and the line in camera 2 is shown On the right the epipole of e2 and the vanishing point Am is added Equation 2 5 can be manipulated further That m lies on the line between Am and e means that the three vectors are linearly de pendent The algebraic way of expressing this is by saying that the determinant of the matrix with these vectors as its column vectors is equal to zero Mo 9 Am 2 1 Camera Modelling By using the definition of the cross product this can be written as Mo e Am M ez X Am Expressing the cross product as a matrix multiplication ax b al b x 0 a3 2 Q1 lal a3 0 a a d2 ai 0 a3 has some benefits which makes it possible to write the relation between m and ms as a linear matri
8. Rosten and T Drummond Machine learning for high speed corner detection in Computer Vision ECCV 2006 A Leonardis H Bischof and A Pinz eds vol 3951 of Lecture Notes in Computer Science pp 430 443 Springer Berlin Heidel berg 2006 M Calonder V Lepetit C Strecha and P Fua Brief Binary robust independent elementary features in Computer Vision ECCV 2010 K Daniilidis P Maragos and N Paragios eds vol 6314 of Lecture Notes in Computer Science pp 778 792 Springer Berlin Heidelberg 2010 D E King Dlib ml A machine learning toolkit Journal of Machine Learning Research vol 10 pp 1755 1758 2009 G Bradski The OpenCV Library Dr Dobb s Journal of Soft ware Tools 2000 H Zhuang A self calibration approach to extrinsic parameter estimation of stereo cameras Robotics and Autonomous Systems vol 15 no 3 pp 189 197 1995 D0
9. for keeping the cameras in place might experience degradation over time Material degradation might cause the orientation and position of the cameras to vary and this will in turn affect the quality of the 3D reconstruction The sensitivity of 3D reconstruction to erroneous camera calibration has been estimated in 2 To handle the drifting camera problem a continuous calibration might Chapter 1 Introduction be applied It would be beneficial to be able to estimate new camera positions and orientations without the need of operator interaction An automatic system for calibrating the drifting cameras should then be able to estimate new positions and orientations by analyzing ordi nary video streams watching the same scene 2 Camera Theory In order to use multiple cameras for 3D reconstruction of the scene it is important to have equations which relate the camera frames to the 3D space which is pictured These equations are affected by two sources the inner workings of the camera and the cameras relative position and orientation The parts of the equations affected by the inner workings of the camera is typically called intrinsic parameters while the relative position and orientation is called extrinsic parame ters Both of these will be explained further in this chapter Calibration of a 3D camera system is typically done by analyzing known shapes By using the camera models on points of which the relative position is known the c
10. in advance The structure of the calibration file can be seen in cameracalibration xml However by running the application with the given input videos the correct initial calibration is stored in cameracalibration xml and that file should be selected when prompted by the application The python scripts utilize the matplotlib module of python which will have to be installed in addition to python 2 7 or later The system debug output prints are there to allow the python scripts to make plots and may not make much sense if read in a text editor The source files include the files of the application which consist of the calibration algorithms They rely on a framework which was provided by National Oilwell Varco for running the graphical user interface yet the files can easily be ported to a standalone project 54 Bibliography 1 2 3 R Ng and S U C S Dept Digital light field photography Stan ford University 2006 T Dang C Hoffmann and C Stiller Continuous stereo self calibration by camera parameter tracking Image Processing IEEE Transactions on vol 18 no 7 pp 1536 1550 2009 T Moons L Van Gool and M Vergauwen 3d Reconstruction from Multiple Images Part 1 Principles No pt 1 in Foundations and trends in computer graphics and vision Now Publishers 2009 G Bradski and A Kaehler Learning OpenCV Computer Vision with the OpenCV Library O Reilly Media 2008 E
11. the camera frame perpendicular to the epipolar line During the work on this calibration system the weight has been on the calibration model which has left little attention to testing different feature detection and matching algorithms The features have a huge impact on the quality of the calibration and achieving good features might even make the need for a noise suppression algorithm fade away The distance between the cameras can be calculated by knowing a sin gle distance in the scene If an initial distance is known existing cal ibrations can be used to calculate distances between all matched fea ture points These distances and points could be stored and matched with future features to calculate the updated distances between the cameras If this is attempted a paper from 1995 about self calibration of extrinsic parameters might be interesting 9 Although the paper is old it presents a relevant method of calibration which uses a known distance in the scene While working on the system a relevant paper from 2009 was found 2 The problem described in the article share great commons to the problem described in this report The paper from 2009 had a more complex problem than presented here due to less simplification The algorithm described also had a more complex optimization problem The paper claimed less than 5 3D reconstruction error There was no comparison of actual and estimated extrinsic camera parameters This pap
12. used Default is FAST Feature point threshold How far away a feature point can be from its corresponding epipolar line before being invalidated Default is 100 50 Short User Manual Run continuous Unchecking this will pause the application allowing for taking screenshots or setting parameters without cali bration running ol Contents Overview of CD A CD is stuck to the cover of the physical edition of the thesis It contains the following e A compiled version of the continuous calibration application e Test videos used for the live test e System debug output prints from the live tests and simulations e Python scripts for processing the output prints from tests and simulations e The source files of the application which are relevant for the thesis e An xml file which contains the calibration of the cameras used for the test The structure of the CD is as follows application continuous calibration exe plus the needed dlls for running the application source code FeatureFinder cpp FeatureFinder h Featurestream cpp Featurestream h Optimizer cpp Optimizer h test results live application output process live output py 59 Kapittel Contents Overview of CD 6 output txt files videos 6x2 videos of tests cameracalibration xml simulation process sim output py 4 output txt files If cameras are to be used with the application they would have to be initially calibrated
13. 3 degrees off This ori entation error indicates that the relative position is poorly estimated which is discussed in chapter 5 29 Chapter 4 Results Roll axis simulation D Sim Roll Estimate Roll 1 Estimate Roll 2 Sim Pitch Estimate Pitch 1 Estimate Pitch 2 Sim Yaw Estimate Yaw 1 Estimate Yaw 2 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 1 Simulation output for each time step Sim values are the simulated angles for camera 2 Estimate values are the esti mated values for camera 1 and camera 2 Roll axis simulation CC Estimate Error Roll Estimate Error Pitch Estimate Error Yaw 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 2 Estimation error of the relative orientation from camera 1 to camera 2 30 4 1 Test Results from Simulated Test Roll axis simulation Estimate error Roll 1 Estimate error Roll 2 Estimate error Pitch 1 Estimate error Pitch 2 Estimate error Yaw 1 Estimate error Yaw 2 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 3 Error in the estimated orientation for both camera 1 and camera 2 relative to the baseline between the cameras 4 1 2 Pitch Simulation The plot of the estimated orientation in Figure 4 4 shows that the estimates follow the simulated angles fairly well It can though be seen that the pitch simulation peaks at about 27 degrees where th
14. 7 49 53 55 1 Introduction People use their vision to control actions and movements in many tasks computer vision aims to duplicate this How you pick up a glass of milk is one of an endless amount of examples You have to know where the glass is to pick it up the angle of the glass to ensure that no milk is spilled and more Cameras get images of the three dimensional 3D space by projecting it down to the two dimensional 2D plane This results in a tremen dous loss of information Today there are cameras which can compute the depth in the image by utilizing an extra array of lenses which adds directional information to the light rays arriving at each point in the 2D plane The cameras in question are called light field cameras and more can be read about them in 1 Another way to compute the depth of a scene is by deploying multiple cameras to get multiple per spectives of the same scene When using two cameras this is called stereo vision When using multiple cameras it is possible not only to get depth information in one image but whole 3D objects can be reconstructed since images from every perspective can be used When a multiple camera system is constructed it has to be calibrated The attributes of the camera such as field of view and optical center as well as the position and orientation relative to the other cameras in the system has to be computed This can be done accurately at system setup but the materials used
15. 70099664 points left 9 l Figure 3 9 Screenshots of the test procedure showing roll angle ro tation There was one test for each of the axes roll pitch and yaw For each test camera 2 started at 0 degrees and was rotated to either 45 degrees or 45 degrees depending on which direction kept the checkerboard in sight It was then rotated back to 0 degrees For the roll test the angle of the camera was estimated visually by looking at the orientation of the checkerboard In the pitch test the angle of camera 2 was taken picture of and estimated after the test In the yaw test a piece of paper with a 45 degree angle drawn on it was used to guide the movement of camera 2 These various methods are pictured in Figure 3 12 24 3 4 Test Procedure for Continuous Calibration 4 116631918241984 2 183133665720953 0 2 432488770766075 0 1911042237113118 0 4444552289139219 points left 9 13 06973148191377 6 236331714604531 o 14 3197461489146 3 092134652771874 16 85777209941093 points left 9 l Figure 3 10 Screenshots of the test application showing pitch angle rotation 25 Chapter 3 Methods gt 3 3599454308367477 5 069400253330907 0 0 8805870899222789 14 03171229179372 0 6227703356340353 points left 9 1 8 90773251958834 1 496730631174506 0 6 312083858967935 41 7561088840143 1 742666636814659 points left 9 l l Figure 3 11
16. First of all the equations for the epipolar line has to be established This has been done in chapter 2 1 3 where equation 2 7 is repeated here eal l b Fmi e ax by c 0 The fundamental matrix in this case is calculated from the calibration parameters Equation 2 6 defines that F e2 _ A and the following equation is obtained by inserting matrices from equation 2 5 F K R C1 C2 _ KoR Ri Ky Here K and K is intrinsic parameters which are assumed constant C Ch is defined to be d 0 ole where d is the distance between camera 1 and camera 2 This makes it possible to define the funda mental matrix as a function of R and Ro Regi Ry Re Ry Cy C2 KR Ri Ky The definition of Ci Ca above is caused by the choice of world coordinate system The x axis is chosen to be parallel to the baseline from camera 1 to camera 2 while the z axis is chosen to have the same pitch angle as camera 1 This makes the relative position from camera 1 to camera 2 always be parallel to the x axis The rotation matrix for camera 1 can also be computed by two angles roll and yaw since the pitch is defined to be 0 degrees This world coordinate system is pictured in Figure 3 2 This can easily be translated to orientation and position relative to the orientation and position of camera 1 Rrel R Ro 16 3 2 Details of Chosen Calibration Method R w 4 0 6 R y 8 C 0 0 0 C
17. Screenshots of the test application showing yaw angle rotation 26 3 4 Test Procedure for Continuous Calibration 19 51122405660977 4 967554333139868 0 23 46631147327351 1 185602261341613 3 265059087291899 points left 9 l l a Roll angle visual estimation When looking at the checkerboard angles of 45 degrees increment can be estimated quite well pi b The camera pitch angle was pictured at the start of the test and at max pitch angle A graphical measurement tool was used to measure the relative angle c Guide lines was drawn on a piece of paper to estimate the yaw angle during tests Figure 3 12 Various methods used to estimate the actual angle ro tated during live test 27 Chapter 3 Methods 3 5 Analysis of the Example Video Videos monitoring an offshore installation by cameras mounted on a crane was provided by National Oilwell Varco The video was provided to show the challenges which are present in the target usage area By analysing the example video a few problem areas were easily spotted This analysis is included due to being the systems target area even though no system tests have yet been run on the example video Firstly the cameras were mounted to a crane which was mobile This makes the timing of the images extremely important If the system starts matching a 500 milliseconds old image from camera 1 with a fresh image from camera 2 great errors may arise The cra
18. a tion method but they can be assumed to be constant Requiring calibration objects would make operator interaction necessary during calibration which is undesired Continuous calibration therefore use other algorithms to find corresponding points in two or more camera frames watching the same scene and use these points to update the initial calibration Not all information about the scene can be ex tracted from these corresponding points For instance the cameras may not know the distance between two points This makes it neces sary to make some assumptions if this information cannot be gathered 11 Chapter 2 Camera Theory Figure 2 7 Left 3D calibration object Right 2D calibration object Figure borrowed from 3 by other means 12 3 Methods Calibration of 3D camera systems are divided into two main cate gories active and passive Active calibration means that there are markers in the images which are used to calibrate the camera pa rameters whereas passive has to find points called interest points or feature points for usage in calibration automatically System Structure Calibration Figure 3 1 The different steps needed for online calibration The method which is used in this paper is an initial active calibration where checkerboards are used for finding both nonlinear parameters intrinsic parameters and possibly initial extrinsic parameters The focus of this paper is the continuou
19. alue This is believed to be caused by dominating effects from the angle error in the optimization problem The relative position of the cameras is still a major part of the calibration and further work needs to be done to ensure that the relative position can be calculated with good 44 5 2 Relative Position Estimation accuracy Ideas for solving this problem is presented in chapter 6 The angular direction which seems to produce the most relative po sition error is the roll angle In the simulation the roll error is up to 3 degrees which corresponds to a deviation of relative position of about 5 compared to the distance between the cameras The roll test results shown in Figure 4 10 also suggest that the calibration application is having a hard time estimating relative position when the roll angle is varying 45 6 Further Work The calibration system presented in this report is far from finished The areas of improvement are summarized by the following list e Enhancing the relative position estimation e Improving the noise suppression algorithm e Testing with different feature detection and matching algo rithms e Adding functionality for estimating the distance between cam eras to remove the last free variable Enhancing the relative position estimation is probably the most im portant point and a few ideas of how to do it is presented in this section Adding an extra camera is one of the ideas By using an
20. amera model equations can be solved to reveal both the intrinsic and extrinsic parameters Camera cali bration will also be discussed in this chapter 2 1 Camera Modelling Cameras project the 3D world onto a 2D plane and in the transfor mation there is a certain loss of information In the work described by this report the pinhole camera model is used The pinhole camera model is a linear model of the camera To account for non linearities a non linear pixel transform is applied in addition to the pinhole camera model 2 1 1 Pinhole Camera Model Imagine a black box with a single pinhole which would let light into the box Each point on the opposite side of the pinhole would then Chapter 2 Camera Theory be hit by light rays originating from a direction given solely by the coordinates for that point This is illustrated in Figure 2 1 objects in the scene negative positive image image lens t f f Figure 2 1 The object in the scene is projected to the camera lens The field of view of the camera is decided by the distance f This figure is borrowed from 3 If we define the coordinate system with the origin at the camera po sition and the z axis pointing perpendicular to the image plane out wards from the camera illustrated in Figure 2 2 some equations re lating the object position to the pixel coordinates can be established If the pixel value on the x and y axis is defined by u and v we can derive the f
21. ch will have one solution for data perfectly fitting the linear model and no solution if there is noise present If there are no solutions to the equations a minimization problem can be solved to estimate an optimal solution min P1 P2 p R K7 m C1 p2R2K37 m C2 2 2 Calibration A typical calibration uses known geometrical shapes often called cal ibration objects Such calibration objects are shown in Figure 2 7 A typical calibration algorithm follows the following steps 3 e Construct a calibration object either 3D or 2D Measure the 3D positions of the markers on this object e Take one or more depending on the algorithm pictures of the object with the camera to be calibrated e Extract and identify the markers in the image 10 2 2 Calibration ON ene Cane Figure 2 6 Model of triangulation c is the center of the image and x is the projection of P to the image frame Assuming noiseless image data Z can be computed if f is known Figure is borrowed from 4 e Fit a linearized calibration model to the 3D 2D correspondences found in the previous step e Improve the calibration with a non linear optimization step Continuous calibration is quite different from initial calibration A goal of continuous calibration is to update the extrinsic parameters of an initial calibration which may have changed due to aging The intrinsic parameters can also be updated by a continuous calibr
22. e estimates suddenly stops following the simulated angle This is due to the noise suppression algorithm and will be discussed further in chapter 5 The pitch simulation results shows the same tendencies as the roll simulation results The relative orientation estimate is quite accurate and Figure 4 5 shows that it deviates by less than 0 2 degrees when not counting the situations where the noise suppression algorithm invalidates all the points The orientation relative to the baseline between the cameras is in this case just as accurate since the pitch angle of camera 1 is defined to be 0 degrees 4 1 3 Yaw Simulation The results from the yaw simulation is similar to the results from both roll and pitch simulation Figure 4 7 plots the simulated orientation ol Chapter 4 Results Pitch axis simulation eS CS Sim Roll Estimate Roll 1 Estimate Roll 2 Sim Pitch Estimate Pitch 1 Estimate Pitch 2 Sim Yaw Estimate Yaw 1 Estimate Yaw 2 0 200 400 600 800 1000 1200 1400 Figure 4 4 Simulation output for each time step Sim values are the simulated angles for camera 2 Estimate values are the esti mated values for camera 1 and camera 2 Pitch axis simulation Estimate Error Roll Estimate Error Pitch Estimate Error Yaw 0 200 400 600 800 1000 1200 1400 Figure 4 5 Estimation error of the relative orientation from camera 1 to camera 2 32 4 1 Test R
23. e stored 3D points and update the calibration based on the movement since the last time step This worked well on paper but testing it on live data wasn t successful and the method was shelved In retrospect the reason why these methods failed on live data may have been caused by nothing but erroneously matched points which was passed to the calibration algorithm The last method based on minimizing the distance between feature points and its epipolar lines was chosen due to better performance than the other two It performed well in initial simulations and test ing it on live data seemed to output correct data It was discovered that as soon as the descriptor matcher started passing non matching points to the calibration algorithm the algorithm would fail and the calibration would be corrupted This made it necessary to construct a simple noise suppression algorithm which is described in chapter Od 3 2 Details of Chosen Calibration Method The chosen calibration method relies heavily on the epipolar con straint which is described in section 2 1 3 The epipolar constraint 15 Chapter 3 Methods states that a point in camera frame 1 will appear somewhere on a line in camera frame 2 The feature detector and matcher will compute a list of corresponding points in camera frame 1 and camera frame 2 By using this list of points it is then possible to minimize the sum of distances from points to corresponding epipolar lines
24. e of computing the relative orientation of the cameras which leaves the distance between the cameras as a free variable to be decided by other means Simulations of the calibration model indicates that the relative orien tation of the cameras can be estimated within a 0 2 degree deviation while estimation of the relative position needs to be perfected Live system tests confirm that the relative orientation can be estimated ac curately They also show that accurate estimation of relative position will require more work Utilizing three or more cameras is an alter native way of estimating relative position since the relative position can be calculated by using the relative orientations The yaw angle was estimated with high precision in the simulation while during the live tests the system had trouble estimating the angle correctly about 0 degrees This might be a field for further work but it is believed that having a good spread of matched features will make the problem disappear In the test the feature points was not only close to each other they were also located in the same plane The test environment is believed to have affected the results negatively because of this This master s thesis was initiated by the wish of National Oilwell Var co to be able to estimate the relative orientation of cameras mount ed on their crane systems using the output calibration for depth mapping the environment This is one possibl
25. e usage area but the continuous calibration system is not limited to this application In theory it could be applied to any multiple camera system The sys tem used for testing was two initially calibrated web cameras which demonstrates the minimal requirements for using the continuous cal ibration vi Contents Problem Definition Preface Abstract and Conclusion 1 Introduction 2 Camera Theory 2 1 Camera Modelling 2 1 1 Pinhole Camera Model 2 2 2 1 2 Nonlinear Effects on Camera Model 2 1 3 Epipolar Constraint 4 4 j 4 e 4 wo a RS 2 1 4 Triangulation 2 2 Calibration ci 3 Methods 3 1 Choice of Continuous Calibration Method 3 2 Details of Chosen Calibration Method 3 3 Software Implementation 3 4 Test Procedure for Continuous Calibration 3 4 1 Simulation Procedure 02 2 3 4 2 Live Test Procedure 3 5 Analysis of the Example Video 2 4 Results 4 1 Test Results from Simulated Test 2 4 1 1 Roll Simulation 4 1 2 Pitch Simulation 4 1 3 Yaw Simulation vil 4 2 Test Results from Live Video Streams 4 2 1 Roll Test 4 2 2 Pitch Test 4 2 3 Yaw Tests 5 Discussion 5 1 Sensitivity of the Estimated Angles 5 2 Relative Position Estimation 6 Further Work Short User Manual Contents Overview of CD Bibliography viii 4
26. er was discovered too late to take into account when imple menting the calibration system which is why it is commented here as further work 48 Short User Manual Startup procedure 1 The application is started by running continuous calibration exe 2 The user is then asked about the kind of input to use Either Web cameras or Files can be chosen 3 In case of Web cameras a A new box will ask for camera 1 ID this is usually a number between 0 and 2 depending on the system setup Trying 0 or 1 usually works b Another box asks for camera 2 ID This cannot be the same as camera 1 ID Try incrementing the last answer 4 In case of Videos a A file browser will pop up asking for camera 1 video file Locate the video file for camera 1 and open it If running test videos this would be the file ending with _ 1 b A file browser will pop up asking for camera 2 video file Locate the video file for camera 2 and open it If running test videos this would be the file ending with _ 2 5 A file browser will now pop up asking for a calibration file locate the xml file which has stored the calibration data If running the test videos this will be cameracalibration xml 6 A box will appear asking for Run mode The available options will be 9x9 Checkerboard Simulation and Free Notice that this can be altered by modifying parameters while appli cation is runn
27. esults from Simulated Test Pitch axis simulation eS a Estimate error Roll 1 Estimate error Roll 2 Estimate error Pitch 1 Estimate error Pitch 2 Estimate error Yaw 1 Estimate error Yaw 2 0 200 400 600 800 1000 1200 1400 Figure 4 6 Error in the estimated orientation for both camera 1 and camera 2 relative to the baseline between the cameras versus the estimated and it can be seen in Figure 4 8 that the relative orientation estimate is mostly within 0 05 degrees The orientation relative to the baseline between the cameras shown in Figure 4 9 like the roll simulation also states is less accurate and suggests a deviation in the relative position estimate 33 Chapter 4 Results Yaw axis simulation D Sim Roll Estimate Roll 1 Estimate Roll 2 Sim Pitch Estimate Pitch 1 Estimate Pitch 2 Sim Yaw Estimate Yaw 1 Estimate Yaw 2 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 7 Simulation output for each time step Sim values are the simulated angles for camera 2 Estimate values are the esti mated values for camera 1 and camera 2 Yaw axis simulation Dl Estimate Error Roll Estimate Error Pitch Estimate Error Yaw 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 8 Estimation error of the relative orientation from camera 1 to camera 2 34 4 1 Test Results from Simulated Test Yaw axi
28. extra camera the relative orientation between each of the three cameras can be used to calculate relative positions as long as one of the dis tances is known Adding an extra camera is perhaps the easiest and is likely to give good estimates without much work A downside of this is the need of increased computation power since it has to up date three calibrations as opposed to one The other idea is to add an extra optimization problem or perhaps modifying the existing one The existing one looks for a local minimum which is not guaranteed to take it to the global minimum An algorithm for finding the global minimum may be applied to the current optimization problem but it might be more effective to split the problem in two This can be done by using the relative orientation output by the current optimization algorithm to formulate a similar problem This new problem would consist of three variables instead of the original six which might low er the computational costs A Kapittel 6 Further Work The current noise suppression algorithm which is described in chap ter 3 2 calculates the distance from the point to the corresponding epipolar line and invalidates any points farther away than a given threshold This means that non matching points may be considered valid by as long as they are close to the epipolar line Reducing the threshold would help but this would again possibly invalidate valid points caused by movement of
29. igned with the image sensor in the camera 2 1 Camera Modelling The effect looks similar to the effect seen when taking a picture of a plane which is not parallel to the camera lens examples are shown in Figure 2 4 Cheap CMOS chip Cheap lens Image Cheap camera b Source 4 Figure 2 4 a shows an example of tangential distortion applied to Figure 3 8 while b shows a possible cause of tangential distortion Tangential distortion can be modelled mathematically using Equation 2 4 which is borrowed from 4 Leorrected LT Qpiy p2 r T oa 2 4 Yeorrectted YT p r T 2y 2px As in Equation 2 3 x and y represents the pixel values subtracted by the origin of the camera r is the euclidean distance from the origin pi ie 1 2 are the tangential distortion parameters and 2coprecteq and Yeorrected are the new pixel values without tangential distortion 2 1 3 Epipolar Constraint When two cameras are recording the same object and the camera parameters for both cameras are given a point in one of the frames has to be on a line in the second frame This line is given by the epipolar constraint and can be calculated by using equation 2 2 Pym K Ri M C1 Pus KR M C2 Chapter 2 Camera Theory Solving camera 1 s equation for M and inserting it into the equation for camera 2 results in a new equation 3 P2M2 KR3 P1R K7 m aE C1 C2 PoM2 p
30. in the relative orientation is estimated with good precision but the relative position is not 4 2 2 Pitch Test In the pitch test camera 2 was pitched to about 15 degrees This was because it had to keep the checkerboard in sight which would be more complicated if tested with larger pitch angles Other than the fact that the rotation is not continued to 45 degrees the results shown in Figure 4 12 and 4 13 seem to be correct Notice that the pitch angle of camera 1 is defined to be 0 degrees which makes the relative pitch angle between the cameras the same as the pitch angle relative to the baseline 36 4 2 Test Results from Live Video Streams Roll axis test 30 Estimate Pitch 1 Estimate Yaw 1 Estimate Roll 1 20 Estimate Pitch 2 Estimate Yaw 2 Estimate Roll 2 10f sure PGC i eecee se ee GE E i ale Oi OO eaaa gee 6g a I E T BRR ee gs we T E a Pg ig anoo 2 gt gee we ease Figure 4 10 The estimated orientations relative to the baseline be tween the cameras is plotted for each time step Roll axis test 20 y Estimate Error Pitch Estimate Error Yaw Estimate Error Roll Figure 4 11 The estimated relative orientation between camera 1 and camera 2 is plotted for each time step 37 Chapter 4 Results Pitch axis test 5 E Reesesirneri mt Estimate Pitch 1 aot Estimate Yaw 1 Estimate Roll 1
31. ing If running the test video the correct option would be 9x9 Checkerboard 49 Kapittel Short User Manual 7 The application should now load the appropriate input and at tempt updating the calibration Altering application parameters To start altering parameters select View gt Parameter window while running the application The parameters are stored in a tree structure under Feature Use simulated points This option enables feature point simulation the Euler angle options will then be used for control ling simulated points This option should not be enabled at the same time as Find calibration patterns Euler angle X control the simulated rotation about the world X axis pitch for camera 2 Y control the simulated rotation about the world Y axis yaw for camera 2 Z control the simulated rotation about the world Z axis roll for cam era 2 Find calibration patterns This option makes the FeatureFinder look for a 9x9 checkerboard instead of ordinary feature points This option should not be enabled at the same time as Use simulated points Max feature point matches The max number of feature point match es which should be sent to the Optimizer The N closest matches will be kept DescriptorExtractor Change the algorithm for extracting descrip tors default is BRIEF GoodFeaturesToTrack detector Enable the GFTT detector if unchecked the FAST feature detector will be
32. ion contributes less to the op timization function than the difference in angle of the two cameras Even with simulated measurements with close to no noise the position is not estimated perfectly the only noise present would be caused by the precision of the C float type As shown in chapter 3 2 the rel ative position is closely related to the angle of camera 1 The angles of camera 1 should be stationary at 0 degrees during the simulation yet they deviate by up to 3 degrees as shown in Figure 4 3 4 6 and 4 9 This corresponds to a deviation of the relative position by 5 of the distance between the cameras which is discussed in section 5 2 Looking at the simulation results of the pitch it can be seen that the noise suppression part of the calibration application which is de scribed in section 3 2 invalidated all the feature points at about 27 degrees This makes sense since rotating the camera in the pitch angle moves the feature points further away from the epipolar lines 43 Chapter 5 Discussion compared to rotating the camera in roll or yaw angles The points move further when at high angles because the Z value of the point rel ative to the camera frame decreases at an increasing rate This most likely not be a problem in live setups since the feature point would disappear out of the camera frame before this effect appears A re lated problem with this noise suppressing algorithm should though be kept in mind it wi
33. ll be highly effective at invalidating noise which is perpendicular to the epipolar line while not removing any noise which is parallel to it Another weakness of the chosen calibration model is the yaw sensitiv ity Rotating the camera in yaw angle will not affect the optimization function as much as rotating the camera in any other angle This is a problem which can be observed in the live tests where the yaw estimates deviate by 5 to 10 degrees when the camera yaw angle is returned to 0 degrees This effect might be reduced by using more spread feature points The live test utilizes no more than nine points which are all on the same plane in the 3D space Using points in one plane when calibrating means that we are missing information 4 Only one plane was used in this test to ensure good features using a checkerboard and the lost information should not be crucial since the application is updating a given calibration instead of computing a new one It is possible to detect two checkerboards in different planes this was originally done in the test application but was omitted due to slow computation on the laptop used for testing 5 2 Relative Position Estimation The roll pitch and yaw angles of camera 1 is easily converted to get the relative position between the cameras based on the orientation of camera 1 Figure 4 3 4 6 and 4 9 show that the estimated orientation of camera 1 in the simulation deviates from the correct v
34. ne itself might have rotated 5 degrees and the calibration will then be 5 degrees off Another problem which was identified by the footage was waterdrops on the lens This will not affect the calibration algorithm directly but it will affect the feature detection description and matching which is a dificult problem already 28 4 Results The calibration software was run through two different tests One test used simulated values with no noise and the other used live video streams 4 1 Test Results from Simulated Test The simulated test was run as described in Chapter 3 4 1 In this section figures with plots of the test results will be presented There are three rotation axes roll pitch and yaw rotation about z x and y axis respectively For each rotation axis there are three figures plot of simulated and estimated angles plot of estimation error of camera 2 and a plot of estimation error of the angle between camera 2 and camera 1 4 1 1 Roll Simulation Figure 4 1 plots the simulated versus the estimated angles The esti mates follow the simulated angles closely but some deviation can be noticed The fact that the estimates of camera 1 is non zero impli cates that the relative position estimation is incorrect The relative orientation error is shown in Figure 4 2 and it seems to be within 0 2 degrees The orientation error relative to the baseline between the cameras can be seen in Figure 4 3 it is about
35. nsic parameters and describe the focal length skewness and optical axis of the camera R and C are called extrinsic parameters Chapter 2 Camera Theory 2 1 2 Nonlinear Effects on Camera Model A camera perfectly matching a linear model is theoretically possible to construct but in reality all cameras have some nonlinear distortion Most of the distortions can be described as radial and tangential dis tortions Radial distortions are commonly known as barrel or pincushion effects this is caused by the fact that the camera lens is not perfectly parabolic This causes the pixels away from the image center to appear either farther away or closer than they should examples are shown in Figure 2 3 b Figure 2 3 Examples of radial distortion a shows the barrel effect while b shows the fish eye effect Both are applied to the test scene in Figure 3 8 Radial distortions can be modelled mathematically by using Equation 2 3 which is borrowed from 4 Lcorrected L 1 T kyr BA kar k3r 2 3 Ucorrected Y 1 Ea kar ga kart T kgr l Here x and y are the pixel values subtracted by the origin of the camera r is the euclidean distance from the origin k ie 1 3 are the radial distortion parameters and Zecorrectea ANd Yeorrected are the new pixel values without radial distortion Tangential distortion appears as skewed images and is caused by the lens not being perfectly al
36. ollowing equations X Y U Jz t Pe Where the object in the image at u v is at the position given by Xc Yo Zc in the camera centered reference frame p and p defines the origin of the image frame This can be written on matrix form giving the following relation f O Du pm 0 f py Me 2 1 0 0 1I 2 1 Camera Modelling Where m uv i is the homogenous coordinates of the pixel value Mc Xo Yo Zo is the 3D point projected down to the image plane p is a scaling constant which in this equation equals Zc objects in the scene projection center N Figure 2 2 The camera centered reference frame Figure borrowed from 3 Equation 2 1 assumes that the object in the scene is given in the camera centered coordinate frame this is often not practical Usu ally the camera coordinate frame and the world coordinate frame are different and a transformation of the object point to the camera centered coordinate frame is required This is done by straightfor ward translation and rotation Mc R M C If we put this into Equation 2 1 we end up with the equation given in 2 2 pm KR M C 2 2 Where M X Y Z i is the 3D world point being projected down to 2D C is the position of the camera in world coordinates R is the rotation from the camera coordinate to the world coordinate K is the camera matrix and p is a scalar value The elements of the K matrix is called the intri
37. or eight points in a box like shape R1 0 3753951586849872 0 3915514813457115 0 R2 0 3660512309906027 0 3478478858266601 0 08522720232888147 points left 8 Figure 3 4 Simulation No camera rotation the points represent corners of a 20cm cube The translation of camera 2 can be seen by looking at the perspective of the cube 1 0 02288692146463023 0 02520067115340248 0 9 977868169938631 0 0008802578031719681 0 005180562233190401 points left 8 l Figure 3 5 Simulation The points represent corners of a 20cm cube The roll angle of camera 2 is about 10 degrees 20 3 4 Test Procedure for Continuous Calibration R1 0 9640894470546345 0 8367407894115497 o R2 0 968621000060092031 0 8395701482060908 9 774141704408057 points left 8 Figure 3 6 Simulation The points represent corners of a 20cm cube The pitch angle of camera 2 is about 10 degrees R1 o a a 0a 0 05233639485863097 0 R2 0 0472024093209945 9 95815348504 0 01071711092651884 points left 8 i 5 Figure 3 7 Simulation The points represent corners of a 20cm cube The yaw angle of camera 2 is about 10 degrees negative For example if the camera matrix K and camera position Cs is given 100 0 60 1 K 0 100 40 C 0 0 0 1 0 This camera could for example have 120 times 80 pixels and the opti cal center is at 60 40 To calculate the pixel value of a 3D point M when the came
38. pter the three calibration methods which were a part of the initial test will be briefly mentioned The method which was chosen after the initial testing will be explained in further details The system implementation and test procedures will be explained A short analysis of the videos which the calibration system originally was made for is also included 3 1 Choice of Continuous Calibration Method The background search done during the initial phase of the work did not lead to a conclusion concerning the choice of calibration method this resulted in a trial period where three different approaches were attempted One method is based on approximating the fundamental 14 3 2 Details of Chosen Calibration Method matrix a second method is based on triangulation and the third is based on minimizing the distance between a point on the camera frame and its corresponding epipolar line The first method approximated the fundamental matrix based on the feature points Fostimate and seeked to minimize the difference be tween Fretimate ANd Fraltibration Fealibration is the fundamental matrix computed using the calibration in IR T a Pianoan extrinsic parameters It ran well on tests but was abandoned due to noisy results on live data The second method needed a correct calibration to start with It would compute 3D points based on the feature point matches and the current calibration The next time step it would try to find thes
39. ra is rotated 30 degrees about z axis roll rotation the 21 Chapter 3 Methods rotation matrix have to be calculated and pinhole camera equations applied a 5 R 30 05 3 0 0 0 1 1 T p2 100 0 60 S 05 0 1 1 0 100 40 0 5 2 0 1 0 P 9 0 1 0 0 1 2 0 170 85 167 83 P2 J 1 Without rotation the point would have been 60 90 ial The simulated test is a part of the FeatureFinder algorithm and is activated through setting the Use simulated values option When the simulated test is activated the Euler angle x y and z values which are changed under the FeatureFinder algorithm sets the pitch yaw and roll angles used for making simulated feature points The test was done by manually setting the simulated roll pitch and yaw angles using the mechanics described in the last paragraph The test was done for individually for each axis The roll was first set to 0 then gradually moved to 45 45 and back to 0 again This was repeated for pitch and yaw axis 3 4 2 Live Test Procedure The objective of the test procedure is to verify that the calibration ap plication is able to calibrate the cameras The calibration application consists of multiple steps described in Figure 3 1 and the test proce dure should minimize the effect of poor feature detection description and matching 22 3 4 Test Procedure for Continuous Calibration A way of minimizing the effect of feat
40. s calibration of extrinsic pa rameters which occurs after the initial calibration A common factor 13 Chapter 3 Methods of all the calibration methods is the system structure which is shown in Figure 3 1 First of all feature points has to be found Several algorithms for finding feature points exist the feature finder which is used by default in the current system is an open source implemen tation of the FAST algorithm 5 When the features are detected something called descriptors have to be extracted The descriptors distinguish the points from one another and a descriptor matcher is used to find the points in each camera that seems to be different per spectives of the same point The default descriptor extractor used in the current system is an open source implementation of the BRIEF descriptors 6 The default descriptor matcher is a simple open source implementation of a brute force matcher The parts of the system structure which are not common for all dif ferent approaches attempted in this paper is the noise suppression and the calibration In fact the noise suppression was developed for the chosen calibration approach after the initial tests The noise sup pression is present because the descriptor matcher would often match non matching points which would corrupt the calibration The cali bration part of the system structure is the main focus of this paper and is where most of the work have been focused In this cha
41. s simulation 1 0 SS SS Estimate error Roll 1 Estimate error Roll 2 Estimate error Pitch 1 Estimate error Pitch 2 Estimate error Yaw 1 Estimate error Yaw 2 0 8 0 6 had SA Si if il a K n 0 200 400 600 800 1000 1200 1400 1600 1800 Figure 4 9 Error in the estimated orientation for both camera 1 and camera 2 relative to the baseline between the cameras 30 Chapter 4 Results 4 2 Test Results from Live Video Streams The results from the live tests are presented in figures Each test has two figures One figure plots all the camera angles individually and the other plots the relative angles between camera 2 and camera 1 A total of four tests are shown one pitch test one roll test and two yaw tests All the live tests output noisy calibrations while moving the camera this is because the camera was moved by hand 4 2 1 Roll Test In the roll test camera 2 was rolled to an angle of a little less than 45 degrees in negative direction The reason why it was not rotated to fully 45 degrees was because of limitations in the software when detecting the checkerboard The checkerboard would be processed erroneously when passing 45 degrees and result in corrupted estima tion Figure 4 10 shows that even though only camera 2 is rolled it is estimated that both camera 1 and camera 2 is actuated Figure 4 11 indicates that the relative roll angle is correct This means that yet aga
42. ure detection performance is by making a scene consisting of good features In this case it was decided to use a checkerboard Figure 3 8 shows the kind of checkerboard used The test was first designed to use two checkerboards the reason for this was to avoid having all feature points in the same plane in space When all the feature points are in the same plane the calibration may be less accurate In this case the hardware was too weak for running tests with two checkerboards which caused excessive computation time The tests therefore consist of only one checkerboard in the scene which is pictured in Figure 3 8 Figure 3 8 The test scene Camera 2 is placed about 15 cm to the right of camera 1 The camera setup consisted of two cameras which were called camera 1 left and camera 2 right Camera 1 was stationary while camera 2 was rotated in a controlled fashion There was a lack of time to set up positioning for the cameras therefore the cameras was moved by hand and detailed error analysis could not be made of the results The results did however give a good visualization of the calibration problems local convexity and the correctness of the output calibration 23 Chapter 3 Methods Ri 1 286344462917422 0 5339530892378426 0 R2 1 350585337267705 4 506109090756759 3 429909014058501 points left 9 R1 bort nie 1 100245781577992 0 R2 4 963166680755145 3 842657709318029 0 90661977
43. vious calibration was small Any point which was farther away from the epipolar line than a certain threshold would be invalidated The epipolar line was calculated by using the rotation matrices most recently estimated 3 3 Software Implementation The implementation is a part of a greater framework and its output is shown in a graphical user interface except for debug prints The controlling class is called FeatureStream and it calls the appropriate functions in the member objects featureFinder and optimizer to run the continuous calibration A shallow class overview is shown in Figure 3 3 FeatureStream featureFinder FeatureFinde optimizer Optimizer run void ee eo matchObjects 5etTPointLists TindKeypoints 5etCalibration makeDescriptors optimizeParameters matchDescriptors Figure 3 3 A shallow class diagram showing the important member objects and public functions The FeatureFinder class implements the first three steps from Fig ure 3 1 feature detection descriptor extraction and descriptor match 18 3 4 Test Procedure for Continuous Calibration ing The Optimizer class implements the last two steps noise sup pression and calibration The software implementation uses two third party libraries which are open source OpenCV 8 and Dlib 7 The feature finding algorithms of OpenCV is used and Dlib is used for solving the optimization
44. x equation 3 mz ez x Am ma es Am Where F is called the fundamental matrix This also leads to an alternative way of expressing the line called l in Figure 2 5 a C Where the line is is defined by the following equation a m e Y 1 b ar by c 0 C 2 1 4 Triangulation When the equations relating the camera frame to the 3D space are in place any point on the camera frame can be traced to a line in the 3D space A typical 3D camera system consists of two cameras of which the intrinsic and extrinsic parameters are known When an object can be found in these two separate camera frames they can be traced Chapter 2 Camera Theory back to two lines in the 3D world With a noise free measurement and perfect equations those two lines would intersect at the position of the object and a 3D reconstruction is accomplished Figure 2 6 illustrates three points defined by the two camera positions and the object position in a 2D drawing The lines from the camera positions to the object passes through the points x and x called m and mz in earlier equations on the camera frames and the position of P called M in the earlier equations can be estimated By solving equation 2 2 with respect to M for both camera 1 and camera 2 the following equations arise p R K tm ae Ci M M po RoK m Ce The unknowns in this equations are M p and p2 This results in a total of five variables and six equations whi
Download Pdf Manuals
Related Search