Home

user manual - Brainstorm Technology LLC

1. 68 STOP POSE RECOVERY Clicking on the button in the toolbar stops the camera pose recovery process The user can save the project and resume the pose recovery proces later by clicking the Camera Pose Recovery V icon on 6 9 CAMERA PARAMETER REPORTS Clicking on this icon will bring up a webdialog that shows a list of input image thumbnails and a report of camera calibration parameters 6 10 VIEW SCENE FROM PREVIOUS CAMERA Click on the previous camera icon to jump back to the previous camera and see the scene from that camera s position and orientation The user viewpoint will be placed inside of the previous camera s frustum This effect can also be invoked by pressing the left arrow key 60 6 11 VIEW SCENE FROM CURRENT CAMERA 61 b Figure 6 27 a Selecting the first endpoint of the corresponding line A magnified sample window appears as a visual cue to help the user recall the start end point of the first line segment b Selecting the second endpoint of the corresponding line MAC USERS To enable the use of the left arrow key go to System Preferences Universal Access and turn on Enable access for assistive devices 6 11 p VIEW SCENE FROM CURRENT CAMERA Click on the current camera icon to jump to see the scene from the current camera posi tion and orientation The user viewpoint will be placed inside of the current cameras frustum 6 12 VIEW SCENE FROM NEXT CAMERA Click on the next
2. 6 15 a Overall progress 85 Remaining time 0 min 15 sec Processing photo park_004 jpg Figure 6 14 Feature extraction progress bar After feature extraction is complete the user is prompted prior to continuing with the next step The user may either stop and save the project to continue at another time or immediately commence with the next stage feature matching Feature matching is performed among successive pairs of photos The progress bar shows the percentage completed and the approximate time remaining Fig 6 16 The number 02 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm Feature extraction size 16670 Feature matching size 2805 C Pose recovery error 0 0 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm Feature extraction size 14962 Feature matching size 2984 Cl Pose recovery error 0 0 Image size 1936 x 1288 6 6 FEATURE TRACKING CAMERA POSE RECOVERY 53 IMG 000 jpg IMG 001 jpg Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm Feature extraction size 16670 Cl Feature matching size 0 Cl Pose recovery error 0 0 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm i Feature iiai size 14962 C Feature matching size 0 CI Pose recovery Overall pr
3. camera pose recovery will be activated after creating a new project in Photosketch as described in Sec By clicking the ES icon the plugin prompts the user to choose either the automatic or the manual mode of feature tracking camera pose recovery Fig The automatic and manual modes can be selected with the Yes and No pushbuttons respectively Automatic feature tracking camera pose recovery should be selected as your first choice Be aware however that if the photos are not taken based on the guidelines described in Sec 3 or if the scene is fairly textureless then the automatic mode will likely fail In that case the user should try the manual mode The automatic and manual modes of feature tracking camera pose recovery are described in Sec 6 6 1 and Sec respectively SketchUp a I Pose recovery LI Select Yes for automatic tracking pose recovery Select No for manual tracking pose recovery no oma Figure 6 13 6 6 1 Automatic Feature Tracking Camera Pose Recovery The automatic feature tracking and camera pose recovery module consists of three steps The first step is feature extraction whereby features are extracted from each photo in the path The user can gauge the progress of the computation by inspecting the popup dialog Fig where the percentage completed is shown alongside the approximate time remaining The number of extracted features is displayed in the webdialog report as shown in Fig
4. e Do not mix landscape and portrait modes in a path Fig 3 7 e For interior scenes do not acquire images over concave paths because they may share a single center of projection and produce only marginal parallax This is problematic because parallax is needed for reconstructing depth See the examples in Fig Notice that the solution in Fig 3 8 b Is acceptable 15 16 SCENE CAPTURE Figure 3 5 a b Two input images taken along different lines of sight c Parallax manifests itself as the apparent displacement between images a and b Notice that nearby objects have larger parallax than more distant objects Figure 3 6 Do not walk in a straight line toward the scene 16 3 2 GUIDELINES 17 Figure 3 7 Avoid changing from landscape to portrait mode Figure 3 8 Example of a bad and b good paths for capturing interior scenes 17 18 3 3 SCENE CAPTURE TIPS FOR TAKING BETTER PHOTOS Use the widest field of view in your camera This is the default setting on most consumer grade cameras when the camera is turned on There are two advantages for using this setting First the photos can cover most of the building and capture a sufficient number of image features that can be tracked from frame to frame This is especially helpful in very dense urban areas where viewing positions may be limited Secondly calibrating once with a default zoom setting relieves you from having to remember the zoom factor
5. parameter controls the gridline density along the floor If the scene is too cluttered with gridlines increase this parameter to lower the density The dimensions of the floor are specified by Grid width and Grid height respectively The units for the step width and 48 6 5 CAMERA CALIBRATION 49 height parameters are identical to those in Sketchup If the user for example selected meters as the current units the numbers entered for grid step grid width and grid height are in meters as well Fig 6 8 f BOC 9945 Lx0T 067A SER ERE Ww iw ew ZWecgo984654 x0 074A SSP a BY w ee m y Wh Go Eens 65 SRY wweThY LET AA SHR ao Rega LEY XY wwe EE AAR RE moo s ge SSS ep Rp re fan e ee irea aag mza ia iaae ils a b Figure 6 8 a A floor depicted with a 200m x 200m grid and a 5m step size b A floor depicted with a 50m x 50m grid and a 10m step size The second tab in the preferences form allows the user to change the magnifier window size and zoom factor Fig 6 7 b Fig 6 9 shows the effects of these changes In the third tab the user can set clear the Auto pause flag for use during camera pose recovery Fig le 7 c If this flag is set pose recovery pauses after feature extraction and feature matching to give the user a chance to save the results Setting this flag is most useful in those cases where the user may want to save partial results so that the session may be continued at a future
6. A 7 5 Network Ens G HI X T IMG 010 IMG 010 IMG 010 IMG 010 IMG 010 IMG 010 E EI IMG 010 IMG 010 File name Files of type Figure 6 11 Select any input image to initiate calibration module for all images in directory camera Fig 6 12 The calibration is considered accurate when its error falls below one pixel Image size 1600 x 12093 Maker Canon Model Canon Powershot D 70 I amp Focal Length 3 65 mn Focal lengths 1321 803 1330 852 amp 1 053 1 032 Camera center 506 130 617 791 1 253 0 923 Radial Distortion ki 0 1475335 3084 305 kl 0030477813736375 pik 0 4000915561738492211 pl 0 00251330982457205 Errors 0 263 0 307 Figure 6 12 Calibration module displays input images and recovered intrinsic parameters There is no need to repeat the calibration stage across projects as long as the zoom settings and camera resolution remain the same Just calibrate the camera once and the computed intrinsic parameters can be re used many times If either the zoom settings or camera resolution change for a specific project the software will automatically prompt the user to recalibrate the camera for the new settings Each time the camera is calibrated for a specific zoom resolution setting the intrinsic parameters are saved in the PhotoSketch plugin ol 52 TOOLBAR REFERENCE 6 6 l FEATURE TRACKING CAMERA POSE RECOVERY Feature tracking
7. D end ol m sete L OD ueram ea et a b Figure 6 2 a Thumbnails of an overlapping sequence b Example of an overlapping se quential path 6 1 NEW PHOTOSKETCH PROJECT 45 If the user clicks No Fig l6 1 b the photos are assumed to be overlapping with no specific order in the sequence Fig le 3 a The unordered option is suitable when the user takes photos of the site at one time and then later returns to the site to take additional photos Fig 6 3 b shows photos that are not taken along a specific path Note that this method will try all pairwise matches and will be more time consuming than the sequential method E hum Wes SS axes ny BNer 2954x207 7X FR OL i Sa cE oo ERASE SS BRS ywer LE a 1 IMG 1577 JPG IM 1576 JPG cameo a b Figure 6 3 a Thumbnails of an unordered set b Example of an unordered path After the project name is entered a file browser will appear Navigate to the folder con taining your photos and select any jpeg image in the set and press the Open button Fig 6 4 The system will read all of the jpeg files in the directory PA Import photos Look in F park l e Eg i Name Date modified Type w icd data 1 28 2009 6 56 PM File Folder Recent Places 2 amp park 000 JPG 1 7 2008 11 03 AM JPEG Image 1 015 KB Ww amp park 001 JPG 1 7 2008 11 03 AM JPEG Image 1 037 KB Desktop park 002 JPG 1 7 2008 11 03 A
8. NEC l misi Figure 5 27 a Select all the cameras and point cloud groups b Unlock these groups by right clicking on the selected objects and select Unlock from the context menu c The color of the groups will change from red to blue and d Group the cameras point cloud and model into one group to apply global transformations d n dm e demus uam less edem uds CBHGc oS 025 4 x0t 079 RO LEAST SS ER 3 a e e eh crash ioe Figure 5 28 a Navigate to the top view of the satellite image from where the photographs were taken and b use the Move Rotate and Scale tools in SketchUp to align the footprint of the model or point cloud to the satellite image 42 6 TOOLBAR REFERENCE NASH CO ASAD SA se CS wecn 35 aaaeKe HID New PhotoSketch Project e 5 Snap Floor To 3D Point q i Open PhotoSketch Project Turn On Off Point Cloud mR Apply Texture Save PhotoSketch Project l Advanced Texture Mapping Preferences Line e BB camera Calibration Rectangle Circle e 2 O Spline Taper To Point Taper To Line Taper To Offset View Scene From Current Camera pie ele Pe wt IN Move Floor Plane Up View Scene From Next Camera e dh Move Floor Plane Down e r4 Turn On Off Cameras e A Turn On Off Floor Plane Set My View e ES Line Based Floor Alignment x Bi Point Based Floor Alignment Go To My View j 43 44 TOOLBAR
9. camera icon to jump forward to the next camera and see the scene from that camera s position and orientation The user viewpoint will be placed inside of the next cameras frustum This effect can also be invoked by pressing the right arrow key MAC USERS To enable the use of the left arrow key go to System Preferences Universal Access and turn on Enable access for assistive devices 6 13 TURN ON OFF CAMERAS When the user sees the scene through the current camera frustum the neighboring frus tums may be visible and clutter the view Click on this icon to turn off the neighboring frustums 6 14 MO SET MY VIEW During the modeling step the user may need to see the model outside of the camera frustums in the path This situation specially happens when the user uses the taper tools For example the user may not see the roof completely from the camera point of view at street 61 62 TOOLBAR REFERENCE Figure 6 29 Click the right mouse button and select Done to exit level To switch fast between current camera view to your desire viewpoint use the Sketchup navigation tools to adjust your view and click on the set my view icon Although Sketchup users can achieve the same result by creating pages this tool is useful for the user who wants to stick a single page 6 15 we GO TO MY VIEW The user can quickly jump to the saved viewpoint by clicking this icon If the user has not set the desired viewpoint the plugin warns the user
10. choose automatic or manual tracking pose recovery Fig select Yes for automatic track ing camera pose recovery The three step automatic camera pose recovery process begins The first step is feature extraction whereby features are extracted from each photo in the path The progress of this step appears in the popup window Fig 5 6 with the percent completed shown alongside the approximate time remaining After finishing this step the pose recovery process pauses and prompts you to continue to the next step The webdialog report Fig 5 7 a shows the number of extracted features Click on the Yes button to go to the next step Feature matching is then performed among successive pairs of photos The progress bar shows the percentage completed and the approximate time remaining Fig 5 8 The pose recovery process pauses and prompts you to continue to the next step The webdialog report Fig 5 7 b shows the number of extracted pairwise matches Click on the Yes button and the plugin estimates the position and orientation pose for ah 28 TUTORIAL Pose recovery LJ Select Yes for automatic tracking pose recovery Select No for manual tracking pose recovery no ma Figure 5 5 Select automatic or manual feature tracking modes for camera pose recovery _ Feature extraction Overall progress 85 Remaining time 0 min 15 sec Processing photo park 04 jpg Figure 5 6 Feature extraction progress ba
11. for subsequent sketching it is helpful to reset the world coordinate axes to a convenient corner of the scene using the axes tool Fig 5 19 This alignment will ease the modeling refinement when using the Sketchup tools We dd Vm Cum mem Dem Wen Pm moi OGLES SARC Fe tx ae LZEREERKN TIU ie ee i4 O To RDMDMT S S A i11 LTEM ELIO 4 m c melli RR sssm EDI SIL Oo MOI ILLI Diaea Br E b Pull the face down to the street Figure 5 18 3D volumes are created by extruding faces using the push pull tool 30 36 TUTORIAL e y 7Wec 4254Xx0 92 T2409 69 0m vg SASF GS BK WY wet Figure 5 19 Align the Sketchup axes with respect to the major axes of the red building 5 7 TAPER OPERATIONS In this section we demonstrate how to build m using the Taper to an offset tool First click on the Apply texture projection icon to project the photo of the current view onto the model You can then dive into the scene and see the model from different views Since SketchUp supports dynamic texturing you can refine the model while the photo continues to be projected onto it The faces of all blocks become textured with the single photo of the current camera Projecting this photo is helpful for model refinement This is especially true when your vantage point lies outside the camera frustum and there are no visual cues from the photo to help guide the
12. image in the set and press the Open button Fig 5 2 29 26 TUTORIAL Project name Cancel a Figure 5 1 a Prompt for entering the project name b Click Yes to select overlapping sequence a n d Import photos Look in 4s park ey E Name i Date modified Type ui prd data 1 28 2009 6 56 PM File Folder Recent Places amp park 000 JPG 1 7 2008 11 03 AM JPEG Image 1 015 KB w im park 001 JPG 1 7 2008 11 03 AM JPEG Image 1 037 KB Desktop park 002 JPG 1 7 2008 11 03 AM JPEG Image 1 121 KB amp park 003 JPG 1 7 2008 11 03 AM JPEG Image 1 158 KB p amp park 004 JPG 1 7 2008 11 03 AM JPEG Image 1 162 KB zokai A Computer A Network File name E Files of type Figure 5 2 Select the first input photo The system will read all of the jpeg files in the directory In general the images in a project directory should be limited to those that constitute the facade of the buildings to be modeled Extraneous images may invalidate results The first time a camera is used with a PhotoSketch project it must be calibrated as described in Sec 4 4 However since the camera for this example has already been calibrated the plugin will then read the calibration parameters and display the camera model focal length center of projection and radial lens distortion parameters Fig 5 3 a After the OK button is pressed the plugin will invert the radial lens distortion for eac
13. modeling Kl your view with the orbit pan and zoom tools to arrive at the vantage point shown in Fig 5 20 You can save a preferred view by clicking the Set my view icon 7 Itis then possible to bounce back and forth between the current oun view and the saved view by clicking on the Current camera and Goto my view icons To unclutter the scene feel free to turn off the 3D point cloud by clicking on the 5 i icon Furthermore you may turn off the floor plane by clicking on the BB i icon It is worth noting that the projected photo achieves the desired result on the extruded vol ume of the building in Fig 5 20 However since the roof has not yet been modeled segments of the photo that would have projected properly onto the roof are instead hitting the side wall of the adjacent building This will be corrected once we complete the geometric modeling of the roof We now model the roof of the red building Select the polyline tool to draw a line 36 5 7 TAPER OPERATIONS 3 7 Figure 5 20 Roughly set your view to this vantage point across the roof to split it into two faces as shown in Fig 5 21 a Hold down the Alt key to constrain the line to be parallel to the closest edge Note that the SketchUp line tool could have been used instead In that case the drawn line changes to pink when it is parallel to the edge along which the cursor is moved oelect the push pull tool icon from the Sketchup toolbar We us
14. or focal length for each project Use a moderate camera resolution Any resolution between two and five megapixels should be adequate Higher resolution images require excessive memory and demand more processing time Use a tripod to reduce motion blur This will improve the accuracy of feature matching in the camera pose recovery stage Fig I3 9 a a Figure 3 9 a Avoid motion blur and b glass buildings with a changing view dependent appearance due to a great deal of reflections The best condition for taking photos is an overcast sky This condition eliminates sharp shadows on the facades and yields better view independent textures Try to take pictures during periods of lighter pedestrian traffic such as holidays or off peak times This will help avoid occlusions due to pedestrians or cars Image content is very important for image feature matching The texture which consists of the color pattern of the scene is critical There must be enough texture available on the object Furthermore the appearance of objects must remain the same through the scene capture This precludes buildings with a great deal of reflection Fig 3 9 b The reflections which change based on the viewpoint undermine the use of feature matching for recovering 3D information 18 4 CAMERA CALIBRATION 4 1 INTRODUCTION PhotoSketch derives its utility by leveraging input images to create 3D models Each image is a different view into the
15. points e g corners After releasing the left mouse button the tool switches to the next photo in the path The added feature is drawn with a red plus sign The user needs to pick the same point feature in this new photo as accurately as possible In order to help the user recall the previous selected feature a moving window appears next 28 5 4 FLOOR ALIGNMENT 29 F Qf Preview Lolo x J Qf Preview Image size 1600 x 1200 Maker Canon yc e 1600 x 1200 Model C PowerShot SD870 IS Maker Canon N 4 ittm Model Canon PowerShot SD870 IS Focal Length 4 6 mm 524 Feature extraction size 185 34 p action ias 18534 E Feature matching size 0 Feature matching size 1491 C Pose recovery error 0 0 rem emor 00 Image size 1600 x 1200 Image size 1600 x 1200 Maker Canon Maker Canon Model Canon PowerShot SD870 IS Model Canon PowerShot SD 870 IS Focal Length 4 6 mm Focal Length 4 6 mm z Feature extraction size 19260 park 001 jpg LJ Feature matching size 0 001 Feature matching size 891 C Pose recovery error 0 0 C Pose recovery error 0 0 Image size 1600 x 1200 SketchUp SketchUp Overall progress 51 Remaining time 0 min 23 sec Matching park 001 jpg gt park 003 jpg Figure 5 8 Feature matching progress bar to the cursor showi
16. scene that will help guide the user in performing sketching extrusion and taper operations The camera position and orientation for each of these views is unknown and must be solved using our camera pose recovery module This will serve to bring the images into a single reference frame Only then can the user seamlessly transition across the different views and perform modeling operations in one preferred view that can be properly rendered in the remaining views Pose recovery is possible only after we perform camera calibration to estimate key pa rameters associated with the camera that acquired the images This information known as the intrinsic camera parameters includes the focal length aspect ratio image center I e princi pal point and radial lens distortion coefficients Computing this calibration data will permit us to accurately determine the camera pose and to obtain precise phototextured models Due to geometric lens and mechanical assembly tolerances there are often variations between the actual focal length and the focal length indicated by the manufacturer Therefore the EXIF focal length that is readily available in the image header cannot be treated as an accurate value As a result an otherwise simple calculation to compute the focal length in pixels from the EXIF focal length and the CCD chip size both given in mm is not accurate This problem is further compounded by the lack of information about the pixel aspect rati
17. that camera calibration recovers the focal length of the camera implies that there must be a single setting for this parameter over all the acquired images Therefore freely changing the zoom factor across images would violate this condition and would require Pho toSketch to operate on uncalibrated images Although this is possible it is inherently less accurate and will not account for lens distortion problems Therefore we require that the user maintains a constant focal length zoom setting throughout the calibration and scene capture sessions As described in Sec 4 camera calibration requires the user to take a few choice images of a planar calibration pattern e g checkerboard from several different views and distances Approximately a dozen views is usually sufficient for this purpose The calibration module will take these images and automatically recover the intrinsic camera parameters including the lens distortion coefficients Computing this calibration data will permit us to accurately determine the camera pose in the next stage 6 INTRODUCTION 1 2 3 Camera Pose Recovery Camera pose recovery automatically recovers the camera position and orientation pose as sociated with each input image This pose information constitutes the extrinsic camera param eters During the course of computing these parameters a sparse point cloud representing the 3D positions of the extracted features is found This stage permits us to brin
18. xD 6 16 SNAP FLOOR TO 3D POINT Clicking on this icon lets the user to select a 3D point and move the floor plane to the height of the 3D point For example if a 3D point is at the edge of the rooftop the user can move up the floor plane to the rooftop height and start 2D drawing from top to the bottom 62 6 17 TURN ON OFF POINT CLOUD 63 s BES lice i iy n Figure 6 30 Examples of correspondence points that lie parallel to the ground plane Also this tool lets the user to snap floor plane to the vertex of a face or an edge EM 6 17 TURN ON OFF POINT CLOUD The user can turn on off the point cloud layer This is useful when the scene is cluttered with the point cloud during modeling 618 5 APPLY TEXTURE Clicking on this icon projects texture on the visible faces from the current photo This is useful when the user does modeling outside of the camera frustum and the projected textures guide the user in the extrusion or tapering operation This method is fast but only performs texture mapping based on the current view and does not detect occlusions We have added a new feature to the texture mapping process that allows the user to select a face or group of faces and applies only textures on the selected face s The textures of the selected faces will be frozen and will not change This helps the user to manually select the best camera e g frontal for texture mapping and avoid the influence of very oblique
19. 5 a The standard SketchUp push pull interface is leveraged to perform extrusion PhotoS ketch facilitates further refinement by snapping the faces to sparse 3D points that represent a plane Sketching is not limited to drawing footprints on the ground plane The user may also draw on extruded faces and use the push pull interface to refine the model The user can further edit the model by tapering to a point line or offset This is often used to model rooftops In these cases the user can snap to a featured 3D point that represents the tapered height or dynamically adjust the height for getting an appropriate texture on the visible faces 2 INSTALLATION 2 1 Getting Started The PhotoSketch plugin can be downloaded from http www brainstormllc com Three download options are available e Free 30 day demo version e Commercial license e Academic license The free demo version of PhotoSketch is a full featured version of the plugin After 30 days you will not be able to activate the plugin nor run another downloaded demo version of Pho toSketch You will be required to uninstall PhotoSketch in order to avoid any popup window warning you that the plugin has expired when you open SketchUp Follow the directions be low for installing the demo version or for purchasing a commercial or academic license of PhotoSketch Discounts apply for running PhotoSketch on multiple computers 2 1 1 Installing the Demo Version To install the
20. M JPEG Image 1 121 KB T park 003 JPG 1 7 2008 11 03 AM JPEG Image 1 158 KB p park 004 JPG 1 7 2008 11 03 AM JPEG Image 1 162 KB Files of type Figure 6 4 Select the first input photo PhotoSketch inspects the EXIF tags from the selected photo to identify the camera model focal length and spatial resolution that was used to acquire the image Based on this infor 45 46 TOOLBAR REFERENCE mation the system checks whether this camera configuration has already been calibrated as described in Sec If so there is no need to calibrate again and the system automati cally pops up a new dialog window to display these camera calibration intrinsic parameters as shown in Fig 6 5 a After the OK button is pressed the plugin will invert the radial lens distortion for each image A progress bar will appear to show the rate of completion in processing each input image The corrected images are stored in a subfolder named data Your original photos will not be modified A webdialog report will be generated that displays the input image thumbnails the camera manufacturer the focal length and other information related to the calibration process Fig 6 6 Calibration will be necessary if the new scene images are taken with different image dimensions resolution or focal length than those taken of the checkerboard images In that case PhotoSketch displays the alert shown in Fig 6 5 b Advanced users who already know the calib
21. P ri TT I i F saia nai Ly i SS fp dnm J i gt l Dms S Figure 6 9 Effects of changing the magnifier window size in pixels and zoom factor a Window size 128 and zoom 2 b Window size 256 and zoom 3 GowasMEESANILVYwWwerVAiizsaazux GORSEASPASEIEMUMNGUUISESZAAXME Figure 6 10 A frustum opacity of a 0 5 and b 0 85 folder Do not mix images of checkerboards from different cameras or zoom settings Click on the Ye icon to pop up a file browser Fig 6 11 Navigate to the folder containing the checkerboard images and select any one of the images This is done by either double clicking an image icon or single clicking the icon and pressing Open Selecting just one image will initiate the reading of all images in that directory Once the images are read the calibration process begins automatically The calibration module first extracts features such as corners from the checkerboard images to solve for the focal length aspect ratio center of projection and radial distortion parameters The module then displays the thumbnails of the input images and the recovered intrinsic parameters of the 50 6 5 CAMERA CALIBRATION 51 r 6 d Import calibration photos x Look in d calibration e c Eg t Name Date modified Type Size mu Recent Places p TT E m Ww Nr a M IMG 008 IMG 009 IMG 009 IMG 009 Desktop zokai j A Computer KA o
22. PhotoSketch 3 0 SketchUp Plugin User Guide v TECHNOLOGY Legal Notice Copyright 2015 Brainstorm Technology LLC All rights reserved Information in this document is subject to change without notice The software described in this document is furnished under a license agreement or nondisclosure agreement The soft ware may be used or copied only in accordance with the terms of those agreements No part of this publication may be reproduced stored in a retrieval system or transmitted in any form or any means electronic or mechanical including photocopying and recording for any purpose other than the purchaser s personal use without the written permission of Brainstorm Technol ogy LLC Product information and specifications are subject to change without notice This publication may include inadvertent technical inaccuracies or typographical errors Brainstorm Technol ogy provides this information as is without warranty of any kind either expressed or implied including any implied warranties of merchantability or fitness for a particular purpose All photos Brainstorm Technology LLC Brainstorm Technology LLC 118 West 27th Street 7th Floor New York NY 10001 www brainstormllc com Version 3 0 October 14 2015 INTRODUCTION 1 1 INTRODUCING PHOTOSKETCH Welcome to the PhotoSketch urban 3D modeling plugin for SketchUp PhotoSketch is de signed to make it easy for you to build phototextured 3D models of build
23. REFERENCE 6 1 NEW PHOTOSKETCH PROJECT The first step in constructing any PhotoSketch model is to create a new project The user must begin by creating a folder containing the input photographs Since PhotoSketch will read all of the images in the folder it is important to restrict these photographs to only those that are part of the path as described in Sec 3 Click on the New Project icon to bring up the dialog window shown in Fig 6 1 a It is recommended that the project name be the same as that of the folder Project name Cancel a Figure 6 1 a Prompt for entering the project name b Click Yes to select overlapping sequence After the project name is entered a dialog box will ask the user to choose if the scene photos are in sequence ordered or out of sequence unordered Fig 6 1 b If the user clicks Yes the photos are assumed to be overlapping and ordered along a path around the urban scene Their filenames determine the order of the images so a sensible naming convention is required Most digital cameras achieve this anyway with filenames such as IMG 0000 JPG IMG 0001 JPG etc Fig 6 2 a Fig 6 2 b shows the camera path for the overlapping sequence Please note that the numbers do not have to be sequential as there are likely to be gaps for those images that were manually deleted from the set BOC 9 854xO 74A TO RLS HR N mgr oo CHART SS LK SW or SU 7 a a y EJ DUA iS iiz
24. Sketch plugin the plugin warns you and pre vents the file from being opened Use the native SketchUp File open command instead 6 3 by SAVE PHOTOSKETCH PROJECT save your PhotoSketch project by clicking the Save Project icon y Please note that Do not use sketchup File save to save your PhotoSketch project Although the geometry will be saved but the cameras poses and info related to your project will not be saved in skp file 6 4 s PREFERENCES Click on the Preferences s icon to launch the preferences form Fig 7 a This permits you to customize different aspects of your PhotoSketch project 4 Preferences PA Preferences Led 2 XS Floor Magnifier Pose Recovery Miscellaneous Floor Magnifier Pose Recovery Miscellaneous Grid step 5 Window size 128 Grid width 100 Zoom factor 2 gt Grid Height 100 Ok Cancel Ok Cancel am x J Y Preferences y Qf Preferences Floor Magnifier Pose Recovery Miscellaneous Floor Magnifier Pose Recovery Miscellaneous Auto pause Iv Ok Cancel Frustum opacity 0 65 Circle Arc segments 24 Ok Cancel c d Figure 6 7 Preferences form with tabs for a floor plane b magnifier c pose recovery and d miscellaneous parameters The first tab in the form allows the user to change the floor plane parameters The Grid step
25. agnifying glass during drawing 68 6 20 2D DRAWING TOOLS 69 6 20 1 LINE The user can draw a line segment or polyline on the floor plane or a face by clicking the left mouse button Use the right mouse button to finish adding more vertices or to close the polyline The magnifier window will be available when the user sees the scene through the camera frustum i Extruding the created face and Drawing another closed polyline inside the face Figure 6 39 Example of polyline drawing tool 69 70 TOOLBAR REFERENCE T 6 20 2 RECTANGLE The user can draw a rectangle by clicking the left mouse button and selecting three points This rectangle drawing tool is very useful in the urban scene that three corners of the footprint or rooftop are visible The edges of the rectangle using this tool do not have to be parallel to Sketchup axes The edges of the Sketchup rectangle drawing tool is restricted and are parallel to the axes since it only defines a rectangle by two points 4 F 4 Te m El A A gn ie E v M a ee T PA lI a 4 f LM A zA wn as ve i The floor plane is at the rooftop height we dien The final rectangular face Figure 6 40 Example of rectangle drawing tool 10 6 20 2D DRAWING TOOLS 71 Tos 6 20 3 O CIRCLE This drawing tool allows the user draw a circle on the floor plane or a face using three points Note that the Sket
26. alibration is considered accurate when its error falls below one pixel The calibration output is used to perform camera pose recovery The camera poses as sociated with the 20 checkerboard images used as input to the calibration module are depicted by green frustums in Fig Note the wide range of camera positions and orientations that were used to achieve subpixel accuracy in computing the intrinsic parameters of the camera There is no need to repeat the calibration stage across projects as long as the zoom settings and camera resolution remain the same Just calibrate the camera once and the computed intrinsic parameters can be re used many times If either the zoom settings or camera resolution change for a specific project the software will automatically prompt the user to recalibrate the camera for the new settings Each time the camera is calibrated for a specific zoom resolution setting the intrinsic parameters are saved in the PhotoSketch plugin 23 24 CAMERA CALIBRATION JW nnie parameters it ee Image size 1600 x 1200 Maker Canon Model Cmca Powershot 0870 I5 Focal Length 4 6 mn Focal bomgths 1321 603 1330 842 amp 1 053 1 032 Camera center 806 140 617 791 1 284 0 983 Radial Distortion k 0 143533530841305 kl 0 05305778137356375 po 0 000915561738492211 pl 0 0025133098245 7295 Errors 0 263 0 307 Figure 4 7 Frustums depict recovered camera poses used to acquire input i
27. bration stage The overlap among the photos during this scene capture stage will permit us to easily track features across the frames which will prove to be necessary for the camera pose recovery stage Before beginning the scene capture stage the user is advised to survey the area and find the optimal path and distance from the building This will help avoid altering the zoom set ting during the course of taking the photographs To further guarantee that this setting remains constant we recommend using the widest camera angle particularly for consumer grade cam eras Since most cameras do not provide an accurate measure of the focal length using the widest angle will guarantee that this zoom setting will remain identical during the scene capture and calibration stages Any other intermediate setting may not be reproduced exactly between the two stages especially for cameras that retract the lens and do not automatically restore its previous setting 3 2 GUIDELINES The quality of the images you acquire plays a critical role in a successful camera pose recov ery Here are some guidelines for taking pictures that will make your camera pose recovery successful Results are likely to be invalid if these guidelines are not met e VERY IMPORTANT Do not change the zoom settings or camera resolution It is critical that the lens does not zoom in or out during scene capture Fig 3 1 e Walk with the camera in an arc around the subject and roug
28. chup circle tool requires two points the center and radius This however is not helpful when tracing the 2D footprint of a column since the center of a column in the photo may not be visible However the user can see half of the column and three extreme outer points on the circle s arc Interactively adjust the circle Select third point on edge Final circle Figure 6 41 Example of circle drawing tool 11 72 TOOLBAR REFERENCE 6 20 4 E ARC This drawing tool allows the user to draw an arc on the floor plane or a face by specifying three points Final arc on the face Figure 6 42 Example of arc drawing tool 2 6 20 2DDRAWING TOOLS 73 T 6 20 5 P SPLINE This drawing tool allows the user trace curved footprints that can not be done by arc or circle drawing tool The tool uses spline curve fitting The spline is very popular curve fitting tool that fits a smooth curve using few control points The following example depict a case that the ceiling of building is a curved surface The user traces the path of the ceiling and wall intersection and extrude the wall to form the ceiling www M Lie TERTE mom wg O E c E CLSASPF ASA Set Ye er iii Ah m RE B r SersinOre AX 949937 wawNY t a BASESS SRAY YuSTH LES AAE H athi gm wee cunc Pe ee ees ep om amm utm aam um Trace the curve by adding n more control point The final spline ree Tae y FO TERBEN fet er 675 4 9i ee Lee E asc aFas 2h
29. ction to paint occluded pixels in gray color b Correction to height of roof further resolves occlusion issues and correctly renders the roof 40 5 9 GEOREFERENCE MODEL ON GOOGLE EARTH 41 5 9 GEOREFERENCE MODEL ON GOOGLE EARTH One main application of the PhotoSketch plugin is for users who want to add their content to online mapping systems The user can georeference their model by aligning the model footprint with georeferenced satellite imagery and then uploading it onto Google Earth The following steps must be followed to georeference the cameras point cloud and model 1 2 select all the cameras and point cloud groups Fig a Unlock these groups by right clicking on the selected objects and select Unlock from the context menu Fig b The color of the groups will change from red to blue Fig 5 27 c Group the cameras point cloud and model into one group Fig d Open Google Earth and navigate to the top view of the satellite image from where the photographs were taken Fig a In SketchUp click on the Get Current View icon to import the top view of the satellite image Use the Move Rotate and Scale tools in SketchUp to align the model footprint or point cloud to the satellite image Fig b Select Plugin gt PhotoSketch gt Export to KML from the SketchUp menubar PhotoSketch generates projectName kml in your project folder The file contains the latitude longitude an
30. d altitude of the point cloud and the cameras In addition the heading and tilt angles of the cameras are stored Note that the heading angle refers to the azimuth of the camera in degrees with zero referring to true North The tilt angle refers to rotation in degrees of the camera such that a value of O indicates that the view is aimed straight down toward the earth the most common case A value of 90 indicates that the view is aimed toward the horizon Values greater than 90 indicate that the view is pointed up into the sky Values for tilt are clamped at 180 degrees For more information about KML and the camera parameters please refer to http code google com apis kml documentation mapsSupport html A 42 TUTORIAL ITI EE zi ITI K a Bk dm Was jem mam Reda mi wig Lu Bb BH Vez ium Heu heu ie Pulp L I fBOC VP FR LaF OTTER SOS C ZWec 9854x0f OTAIA DS OF HAGO ERAS FSS LEY Y EMAJ SS OUR SN 6 83 pack LT Tr M A E E E ETE EE TRETEN EY I T Me este Ts MNT PTUS i VAS S erro eee eed gh ad apa n Ea AE e H T EIUS CL cn eds 1 e X EAE M Ne CAEN as sir 1 E i Hey IM HUGUES TEEN aniele f Y RID at FEE ew oe iia eed gt e saki Em e OG am Tamil OD A pacer DE uiu NU Beim Paci 2D ps Thami er E e e C m tomer ie ak iau ie Mae hera Mul J PERI ek Fi el BUE Q1 I eun EN FEM ES o e ie am PA i inis minc pom Ir ult i A 2
31. e advanced texture mapping is a slow process and we recommend to use it when the final model is ready The user can freeze unfreeze the face textures as described in 620 2DDRAWINGTooLsS 4 1 C This section describes the 2D drawing tool set These drawing tools are different from the Sketchup drawing tool The main difference is that it forces the user to draw only on the floor plane or a face This requirement is simplify the modeling of manmade structure such as building As long as the floor plane is at the right height these 2D drawing allow the user to trace the footprint or horizontal profile of the building and with a simple extrusion or taper operation create a 3D volume These drawing tools have two common features when they are activated The first feature is the mouse pointer that highlights the floor plane or the face upon which the user will draw The user can lock on the floor plane or the drawing face pressing the Shift key This allows the user to draw outside of the face boundaries The second feature of the drawing tool is a Che k fex Vut paint Figure 6 37 The 2D drawing tools highlights the selected face as drawing pad 67 68 TOOLBAR REFERENCE magnifier window This is only available when the user sees the scene through the camera frustum The magnifier window helps the user pick the corners of the building more accurately e an a i TT With 51 8574cm Figure 6 38 Example of a m
32. e is no 3D point feature to snap onto 6 25 MOVE FLOOR PLANE DOWN Clicking on the Move floor plane down rN icon causes to move floor down with a small step This useful when the user wants to fine tune the height of the floor plan and there is no 3D point feature to snap onto 6 26 ain TURN ON OFF FLOOR PLANE The user can turn on off the floor plane layer This is useful when the scene is cluttered with the floor plane during modeling or the floor plane preventing to select faces beneath it i
33. e photo as shown in Fig When you draw the first edge of the rectangle try to draw the edge along the vanishing lines In this example they tend to lie along the edges of the windows Lh FI f E p e u d EM d jo fo db A of 4 KJ Pw y py idiom 4 w 3 his Ni E p f a S F eee OC Figure 5 15 Click on the rooftop corners to draw 2D rectangles Select the a first b second and c third corners of the first building Select the d first e second and f third corners of the second building 33 34 TUTORIAL eee o en TEAV b Draw a 2D rectangle 54 es d i e Snap the floor plane to a 3D point f Draw a 2D rectangle i Figure 5 16 Repeat the process of snapping the floor to the rooftop and drawing rectangular footprints by selecting three points 34 5 6 EXTRUSION TO CREATE VOLUMES 35 5 6 EXTRUSION TO CREATE VOLUMES oelect the push pull tool E icon from the Sketchup toolbar Move the cursor to select the rectangular rooftop face of the corner building When the face is highlighted click the left mouse button Fig 5 17 a and pull it down to extrude the face Release the left mouse button when the extruded face reaches the street Fig 5 17 b This extrusion operation needs to be repeated for all footprints to create 3D volumes that are embedded in the scene Fig 5 18 To assist the SketchUp inference engine
34. e the photo as a cue to extrude the narrow face to its proper height This is done by pointing the cursor at the narrow face and dragging it up until it has been extruded to the light colored detail above the bricks Fig 5 21 b Figure 5 21 a Draw a line from edge to edge of the rooftop face b Pull the narrow face up to the shown height c Draw four lines on the extruded narrow ledge This process can be further refined by drawing four lines on the top face of the extruded narrow ledge to split into additional facets Fig c Then select the Taper to offset tool t and pull the middle face up until it reaches the narrow end of the roof Fig 5 22 Draw two parallel lines to break the rooftop face to three faces Fig a The width 37 38 TUTORIAL E Ll 58587 I a an PL bel ind vee F T3 s55 H 75 a 9 E E Ra Y 1 ny Ry i Figure 5 22 Select the Taper to offset tool and pull up the middle face to its proper height a Frustum view b Viewpoint from saved view of the middle face should be approximately the same as the topmost segment of the tapered roof Select the Taper to offset tool t and pull the middle face up to its proper height Fig 5 23 b Note that you will have an opportunity to further refine the placement of the edges later by using the Sketchup move tool Ld Figure 5 23 a Draw two lines on the roof
35. editing deleting task ZETEL Gr wv E a EEEN aa Figure 6 23 Click the right mouse button and select Done to exit 6 7 FLOOR ALIGNMENT Once camera pose recovery is complete the plugin prompts the user to select between the vertical line based and point based floor alignment methods see Fig 6 24 The floor ot 58 TOOLBAR REFERENCE Hoor Alignment Ue Select Yes for vertical line correspondence method Select No for point correspondence method Yes No Cancel Figure 6 24 alignment tools allows the user to manually align cameras and the point cloud with respect to a plane that is parallel to the ground We assume that the building facades are perpendicular to the ground Therefore it is important to find the orientation of the ground plane accurately Otherwise the extrusion of the footprints or rooftops will not pass through the building facades Select Yes and NO respectively for the line based method Sec 6 7 1 and for the point based method Sec These choices are dependent on the input scenes The user must decide which method is easier Often either method is equally convenient However if the result is not satisfactory a user can go back and redo floor alignment again Note Do not add geometry when you don t have satisfactory floor alignment 6 7 4 FLOOR ALIGNMENT VERTICAL LINES This alignment module allows the user to draw line segments on the vertical edges of facades or windows among two p
36. ent within the face Fig 6 45 and Fig 6 46 depict two examples of taper to a point operation Draw line segment Pull up interactively Final result view1 Final result view2 Figure 6 45 Taper to a line tool example 1 oelect first endpoint Select second endpoint Taper to the line result Figure 6 46 Taper to a line tool example 2 19 76 TOOLBAR REFERENCE 6 23 e TAPER TO OFFSET This tool allows the user to select a face within a face normally offset and interactively taper to the desired height There is no restriction on the shape of the base face or inner face When the user hovers over the inner face the face will be highlighted We recommend to use the Sketchup offset tool to create the offset face first Also the user can use other drawing tool to define the inner face however we do not guarantee to get topologically sound or meaningful result Fig 6 45 and Fig 6 46 demonstrate examples of how to taper to a point Draw offset inside face Select offset face Pull up offset face Final result Figure 6 47 Taper to an offset example 1 Circular offset curve Final result Square offset curve Final result Figure 6 48 Taper to an offset example 2 16 6 24 MOVE FLOOR PLANE UP 77 6 24 N MOVE FLOOR PLANE UP Clicking on the Move floor plane up A icon causes to move floor up with a small step This useful when the user wants to fine tune the height of the floor plan and ther
37. ert dialog box for uncalibrated cameras After radial lens distortion is removed from all of the scene images the plugin is ready to recover the camera poses by computing extrinsic camera calibration parameters A webdialog report is then generated that shows input image thumbnails the camera manufacturer the focal length and other information related to the calibration process Fig 6 6 46 6 1 r Qf Preview ce E Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm CI Feature extraction size 0 E Feature matching f size 0 CI Pose recovery error 0 0 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm CI Feature extraction size 0 C Feature matching size 0 CI Pose recovery error 0 0 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm C Feature mum l size 0 C Feature matching size 0 Figure 6 6 Generated report AT NEW PHOTOSKETCH PROJECT 47 48 TOOLBAR REFERENCE 6 2 ug OPEN PHOTOSKETCH PROJECT PhotoSketch uses the native SketchUp skp format to save projects Click on the Open PhotoSketch Project icon ug to open a previously saved PhotoSketch project Select your PhotoSketch skp file using the file browser dialog Please note that if you try to open a oketchUp file that was not created by the Photo
38. face and b pull up the middle face 5 8 MODEL REFINEMENT In this section we demonstrate further refinement of the roof section Select the push pull tool icon from the oketchup toolbar Point the push pull tool at the top narrow face and pull that face up until you see the white cement band on the top of the bricks Fig a Select the Arc tool ros and point the cursor at the side wall to highlight it Click the Shift key to activate drawing anywhere on the face plane Fig b Then draw an arc on that plane as shown 38 5 8 MODEL REFINEMENT 39 in Fig C Select the push pull tool icon from the Sketchup toolbar to extrude the arc across the rooftop Fig d Figure 5 24 a Pull the face up using the push pull tool b Select the arc tool and point at the highlighted face c Draw an arc on that face and d extrude the arc across the rooftop Apply texture mapping by clicking on the 5 icon to project texture onto the newly created faces Fig 5 25 a The full model can be completed by repeating the process of drawing rectangles extruding faces tapering to a line and applying copy paste operations to geometric elements on the roof Fig 5 25 b The use of the standard SketchUp texture mapping continues to present artifacts on occluded faces as shown in Fig 5 26 a Earlier we had noted that incorrect texture may be projected on a face if the model is incomplete However there is an additional problem that is na
39. free 30 day demo version of PhotoSketch 1 Visit www brainstormllc com and click the Try It Now button 2 Download and run the PhotoSketchDemo3 0 exe file to install the plugin files 3 Open SketchUp An activation dialog window Fig 2 1 a will appear DA Click on the blue URL link to take you to an activation request page Fig 2 1 b Submit the completed form to receive an activation code by email Copy and paste the activation code into the activation dialog window to unlock the plugin N O Ol Run the full featured plugin for up to 30 days No additional demo version may be downloaded and run on that machine thereafter 10 INSTALLATION Activation This application requires an Activation Code Click the URL below to start the process now PhotoSketch Activation Page Request Number 1933613616 When you receive your Activation Code paste it in the field below and click the Activate Now button Activate Later a Activation Request Please complete the form below to activate your free 30 day trial version of the PhotoSketch Plugin for Google SketchUp Asterisks indicate required fields Your activation code will be emailed to you so please enter your email address carefully First Name Last Name Company Street City State Country Zip Postal Code Phone Email 1933613616 Request Number Send b Figure 2 1 a Activation dialog window appears upon fir
40. g all of the photographs into a single reference frame in which we will build the 3D model An example is shown in Fig where the collection of frustums represent the five recovered camera poses derived exclusively from the input images in Fig 1 1 fie dt View Camera Draw Took Window Plugins Help s BOC OGLA ASRI WAR co BERAS N 239 9 99 BYU wyy ts xi t ag N ry e f Ka lg b b bi E Bc T VE aA mp he LE c c KL Cee ye 2t4724291223 2 PL VT TREE 2 TI IEET ES Bis Figure 1 3 he five recovered camera poses and the sparse reconstruction of a New York City building on Park Ave and 85th Street Once all the images are brought into a single reference frame any user drawing made upon one of the input images will appear properly aligned in the remaining images The rationale for having multiple calibrated images is to facilitate total coverage of the scene in the presence of occlusions Since each image can be projected back into the scene the texture of all 3D faces will be derived from non occluding views 1 2 4 Floor Alignment The cameras and point cloud derived by camera pose recovery sit in a single reference frame in the world coordinate system Since the absolute position and orientation of the first camera is unknown it is placed at the origin of the world coordinate system With respect to this camera s coordinate system the floor of the sparse 3D point cloud of
41. h image A progress bar will appear to show the rate of completion in processing each input image Fig 5 4 The corrected images are stored in a subfolder named data Your original photos will not be modified A webdialog report will be generated that displays the input image thumbnails the camera manufacturer the focal length and other information related to the calibration process Fig 5 3 b 26 5 3 CAMERA POSE RECOVERY 27 Z Preview X Intrinsic calibration parameters Image size 1600 x 1200 Maker Maker Canon Model Canon PowerShot SD870 IS Model Focal Length 4 6 mm Focal x 1321 60 i Cl Feature extraction size 0 Focal y 1330 85 park 000 jpg Cl Feature matching size 0 Center x 606 14 CI Pose recovery error 0 0 Image size 1600 x 1200 617 79 Center y Maker Canon Model Canon PowerShot SD 870 IS k 0 130 Focal Length 4 6 mm kl 0 06306 Cl Feature extraction size 0 0 00092 park 001 jpg Cl Feature matching size 0 0 00251 Cl Pose recovery error 0 0 Image size 1600 x 1200 Maker Canon b Overall progress 60 Remaining time 0 min 1 sec Processing photo park 003 jpg Figure 5 4 Progress bar for inverting the radial lens distortion 5 3 CAMERA POSE RECOVERY When the photos and intrinsic camera parameters are known the camera pose recovery but ton in the toolbar will be activated By clicking on the ES icon the plugin prompts the user to
42. hat will make your calibration successful Calibration results are likely to be invalid if these guidelines are not followed e VERY IMPORTANT If you are using a consumer grade camera use the widest angle for the zoom setting Since most cameras do not provide an accurate measure of the focal length using the widest angle will guarantee that this setting will remain identical 21 22 CAMERA CALIBRATION during the calibration and scene capture stages Any other intermediate setting may not be reproduced exactly between the two stages especially for cameras that retract the lens and do not automatically restore its previous setting e Use a moderate camera resolution Any resolution between two and five megapixels should be adequate Higher resolution images require excessive memory and demand more processing time e Disable the camera flash to prevent glare Use a well lit room e Use a high shutter speed to avoid motion blur If the camera automatically reduces the shutter speed under low light conditions use a tripod e Make sure the whole checkerboard pattern is visible in each image Fig 4 4 shows several poorly taken images that create problems for the calibration pro cess These examples do not conform to the guidelines given above and should be avoided a Flash reflection b The pattern is not fully visible c Underexposed and dark d Overexposed and saturated e Out of focus Figure 4 4 Examples of bad photo
43. hly point the camera to the 13 14 SCENE CAPTURE Figure 3 1 Avoid changing the zoom setting same part of the scene Fig I3 2 shows such a proper path Figure 3 2 Example of a good path for scene capture e he distance and angle between consecutive images should be less than five feet and greater than 15 degrees respectively as shown in Fig Keep in mind that taking more photos is better than not having enough photos e he image sequence must be serially ordered along a path This guarantees that each pair of images in the sequence have overlapping views e Do not pan the camera from the same location Fig 8 4 The resulting panoramic image sequence does not have 3D information to recover due to its limited parallax Fig 3 5 Recall that parallax manifests itself as the apparent displacement of objects when viewed along different lines of sight Nearby objects have a larger parallax than more distant objects We leverage parallax to infer depth by measuring the image displacements be tween correspondence points among images The greater the displacement the closer 14 3 2 GUIDELINES 15 Figure 3 3 Guideline for taking consecutive pictures the object This effect does not apply when the images are taken from a single center of projection i e viewpoint Figure 3 4 Avoid standing in one location and panning the camera e Do not take photos while walking in a straight line toward the scene Fig 3 6
44. hotos in the path In many scene photos it is often easier to find vertical line segments than finding corresponding points on a horizontal plan The goal of this tool is to draw vertical line segments on a photo and match these segments across photos in the sequence The user is free to start from any photo in the sequence Technically the user only needs to draw one set of line correspondences to align the floor However drawing more than one pair of line correspondences is recommended to distribute the floor alignment error among all photos in the scene The floor alignment tool will be activated once the user clicks on the ES tool icon The first photo of the sequence is then shown The user may go forwards or backwards in the photo sequence by pressing the left or right arrow keys on the keyboard To add a new line segment press the k key Click the left mouse button near a vertical edge in the photo A magnifier window appears to help the user accurately pinpoint the starting point of a vertical edge see Fig 6 25 Releasing the button will anchor the point at the current location and draw a rubberband line that follows the mouse Fig 6 26 a Click the left mouse button to pop up the magnifier window that assists the user to pinpoint the terminal point of the vertical edge Fig 6 26 b Move the mouse and release the left button to select the terminal point and draw the vertical line After drawing the first line segment the tool di
45. iW FY pier iii TT So cnp sscNSMVYYetVLLttAA aAEE EIL Lb LLL LLL d Extrusion of the back wall The Surface of the ceiling from a novel view Figure 6 43 Example of spline drawing tool 3 74 TOOLBAR REFERENCE 6 21 e TAPER TO POINT This tool allows the user to select a point on a faces normally rooftops and interactively taper to the desire height There is no restriction on the shape of the base face When the user hover over a face the face will be highlighted and when the mouse get close to the center of face the position of the center will be marked by a pink square to facilitate the selecting of the center point We recommend the users to pick center point since the most of manmade building taper symmetrically Fig Fig 6 44 depict three examples of taper to a point operation Select center of N sided face Taper to the point result Figure 6 44 Taper to a point tool examples 14 6 22 TAPER TO LINE 75 6 22 a TAPER TO LINE This tool allows the user to draw a line within a face normally rooftops and interactively taper to the desire height There is no restriction on the shape of the base face When the user hovers over a face the face will be highlighted The user requires to select two points on the face and pulls the line defined by two points up or down The user can snaps to the endpoint or midpoint of the edges of the face The line does not have to be from edge to edge and can be a line segm
46. iew the following key stages in the PhotoSketch modeling process 1 scene capture 2 calibrate camera 3 recover camera poses 4 align floor 5 sketch footprints and 6 extrude taper footprints to form volumes 1 2 1 Scene Capture The input to the system is a collection of overlapping images of a scene For best results it is best to conform to the scene capture guidelines described in Sec The key rule for acquiring the input images is to serially order photographs as you walk around the scene The photographs should have at least 50 overlap and be taken with a constant focal length zoom camera setting The top row of Fig 1 1 depicts a set of input images that satisfy these requirements The overlap among the photos will permit us to easily track features across the frames which will prove to be necessary for the subsequent camera pose recovery stage 1 2 2 Camera Calibration The purpose of camera calibration is to estimate key parameters associated with the camera that acquired the images This information constitutes the intrinsic camera parameters They include the focal length image center i e principal point skew and radial lens distortion coefficients Recovering lens distortion parameters is necessary to invert the warp induced by lens aberrations Since every camera has unique distortions it is necessary to calibrate each camera used to acquire photographs even if they all share the same model number The fact
47. ing you for your serial number Copy and paste your serial number to activate the plugin 8 Enjoy PhotoSketch 2 1 4 Feedback We are eager to incorporate any suggestions you have to improve PhotoSketch Feel free to send email to support brainstormllc com with any comments bug reports or installation issues We will make every effort to incorporate your feedback into future updates All owners of commercial and academic licenses are entitled to free updates You will be notified about new updates automatically when you open SketchUp Future upgrades to major releases will be available at a discounted rate for current PhotoSketch users 11 12 INSTALLATION 12 3 SCENE CAPTURE 3 1 INTRODUCTION PhotoSketch operates on a set of overlapping photos of an urban scene to produce lightweight phototextured 3D models We refer to a sequence of overlapping photos as a path The user needs to acquire the photos along the path in an orderly fashion following a set of prescribed guidelines described in Sec 3 2 Failure to adhere to these guidelines will reduce the accuracy of the modeling or possibly invalidate the camera pose recovery results The key rule for acquiring the input images is to serially order photographs as you walk around the scene The photographs should have at least 50 overlap and be taken with constant focal length zoom and camera resolution settings These settings must be identical to those used during the cali
48. ings directly from photographs We simplify the 3D modeling workflow by leveraging the SketchUp push pull in terface with computer vision and photogrammetry techniques The PhotoSketch plugin merges the benefits of automatic feature extraction camera pose recovery an intuitive sketching inter face and texture mapping to produce lightweight photorealistic 3D models of buildings These models which typically consist of a few hundred polygons each makes them well suited for use in high performance visualization online gaming and web based applications such as Google Earth Sketch l extrude taper ops Figure 1 1 PhotoSketch operates on a set of overlapping photos of an urban scene to produce lightweight phototextured 3D models 4 INTRODUCTION Generating a model of a building from its photograph requires us to embed basic 3D primitives such as cubes or cylinders directly into the image These primitives can then be refined using 3D extrusion or tapering operations to best match the scene geometry Photo oketch simplifies this process Architects designers and entertainment content creators can now employ a new approach to create ightweight 3D models of existing buildings Rather than having a user fumble with the difficult process of orienting a 3D primitive into a 2D photo of the 3D scene the user is now able to directly draw upon the image along principal planes A model is constructed by drawing a footprint on the ground pla
49. ition FHG CABS LxD 8 ISA Soe SF Pramo tans Ywaet iifAAX Figure 6 20 The motion path of the tracked feature in the sequence After tracking the feature in the photo sequence the user is required to stop and close the track To stop press the k key on the keyboard Typically you will need to pick at least eight features between the first and second photos in the sequence and at least six features in each photo that has common features in the adjacent images When enough features are found to be tracked the system attempts to recover the camera poses and displays the average and max reprojection error in the SketchUp status bar see Fig le 21 If the average error is high e Photo 1 Amsterdam_001 jpg Tracks 9 Projection error 0 33 avg 0 88 max Figure 6 21 The tracking status is shown in the SketchUp status bar e g more than 4 or 5 pixels the user can go back and delete edit a track or add more tracked features To delete a track select the desired track by clicking the mouse button near it The track color will change from yellow to cyan Then press the letter d to remove the track To edit a track select the desired track The track color will change from yellow to cyan Then press the letter k to enter track editing mode If the user presses the letter d the corresponding feature point on the current photo will be removed If the user clicks the left mouse button near a fea
50. mages 24 o TUTORIAL 5 1 INTRODUCTION This tutorial teaches the basics of Photosketch You will learn how to create a new project import a photo sequence and use the photos to create footprints that can be extruded and tapered to build lightweight phototextured 3D models This tutorial includes how to 1 Create a new PhotoSketch project and import a photo sequence 2 Determine the camera poses and 3D point cloud from the photo sequence 3 Align the floor plane of the camera with respect to the ground 4 Sketch 2D footprints by tracing over input photos 5 Extrude each footprint to create a volume 6 Taper faces to points lines and curves to refine the model 7 Perform additional model refinement operations on the faces of the model We assume familiarity with the Sketchup modeling tools Readers who need to gain proficiency in SketchUp are advised to consult the wealth of videos manuals and books available on the subject 5 2 CREATE A NEW PROJECT To create a new PhotoSketch project simply click on the new project icon to bring up the dialog window shown in Fig 5 1 a Type Park Ave in the project name and click the OK button After the project name is entered a dialog box will ask the user to choose if the scene photos are in sequence ordered or out of sequence unordered Fig 5 1 b Click Yes to select input photos from the file browser Navigate to the C PhotoSketch example folder select any jpeg
51. ment and applies it on the camera frustums and the point cloud 6 7 2 FLOOR ALIGNMENT POINTS ON PLANE The user interface of this section is similar to manual tracking tool see Sec 6 6 2 e Press k to add or edit a tracked feature e Press d to delete a tracked feature e Press x to turn on or off the feature flow lines Floor alignment begins by having the user track three or more correspondence points fea tures lying on the same plane across multiple photos The tracked points are not required to start from the first photo A popup magnifier helps the user pinpoint the correspondence points Be sure to select points that all lie on the same plane and have the same height For example 59 60 TOOLBAR REFERENCE Figure 6 26 a The first endpoint is set and the second endpoint is moving b The second endpoint of the line segment is selected these points can be picked from the building footprint or rooftop whichever is more visible in the photos In some cases the corners of a row of windows on two adjacent walls may be adequate A mistake may be undone by pressing ctrl Z After three or more correspondence points are selected click the right mouse button and select Done Fig shows several correspondence points that form the plane parallel to the ground and has the same heights The webdialog report Fig shows the average projection error in pixel For an accu rate calibration these values should be less than one pixel
52. ne and extruding it to the proper height of the object to form a volume The walls of that volume now constitute principal planes upon which additional sketches may be drawn In this manner drawing can be constrained to the walls and floor of the scene to yield footprints that can then be extruded to form increasingly refined volumes We leverage the intuitive push pull interface of SketchUp for this purpose An example is given in Fig 1 2 oince footprints on the ground may often be occluded parallel footprints such as those on a roofline may be used instead BO 95L4x0 074 A29 9v9 F wey Li Arb SSCENSOMYTOeUtVLLLAA ARAE ET Figure 1 2 a The image acts as a stencil upon which the user sketches building rooftops black boxes and performs b extrusion operations to generate a lightweight 3D model The black cubes denote the generated volumes c Final phototextured model The automatically recovered camera position and orientation for each of the five input photographs are depicted using frustums The user drawn sketches are interpreted to lie in an upright 3D coordinate system that can be properly rendered from any viewpoint This transformation into the upright coordinate System is facilitated by advanced computer vision algorithms including our camera calibration camera pose recovery and floor alignment modules They are respectively responsible for estimating the intrinsic camera parameters e g focal length principal
53. ng the image of the feature neighborhood from the previous photo After tracking the corner in all photos the user is required to stop and close this track To stop press K on the keyboard Repeat the same procedure for the two corners that remain A mistake may be undone by pressing ctrl Z After three corner points are tracked click the right mouse button and select Done The plugin rotates the cameras and sparse point cloud so that they are aligned with the SketchUp coordinate system PhotoSketch then displays the scene through the first photo Fig 5 14 a You can change your view from one frustum to the next by clicking on the Next camera or Previous camera icons For convenience the user may also press the left or right arrow keys on the keyboard to move between images Additional information can be found in Sec 6 7 29 30 TUTORIAL 3 DOS CCOV Perier A A ts Overall progress 17 ER Processing park 001 jpg gt park 002 jpg Camera pose reco ver Overall progress 7626 Adding photo park_001 jpg EM Figure 5 9 Pose recovery progress bar The pose recovery step successefully estimated 5 out of 5 camera poses Press OK to start floor alignment step Figure 5 10 After camera pose recovery is done the plugin shows the number of photos whose poses were successfully recovered Floor Alignment WE Select Yes for vertical line correspondence method Select No for
54. o and the center of projection It is false to assume that the aspect ratio is 1 0 and the center of projection corresponds to the center of the photo These deviations are particularly true for lower grade consumer cameras 4 2 PROCEDURE The procedure for performing camera calibration is very simple The user must first take several pictures of a flat checkerboard image from different views and distances Please visit http www brainstormllc com uploads checkerboard pdf to download the checkerboard image It is suggested that this checkerboard image be printed on a matte paper stock with a high quality printer Do not use glossy paper because it introduces glare Avoid using plain 19 20 CAMERA CALIBRATION Figure 4 1 Checkerboard for camera calibration paper since ink or heat from the printer tends to warp and buckle the paper surface If you do print the image on plain paper make sure to adhere it to a flat cardboard surface and be careful to prevent the paper from bubbling Using clear tape mount the board on a wall floor or table Fig 4 2 Make sure that you mount the checkerboard at a position from which you can comfortably take 15 to 20 pictures from different angles and distances Figure 4 2 Checkerboard setup for camera calibration Begin by taking one picture in which the camera lies parallel to the board and is close enough to have the checkerboard fill its entire field of view Take the second image by moving
55. ogress 51 Remaining time 0 min 23 sec Matching park 001 jpg gt park 003 jpg Figure 6 16 Feature matching progress bar of extracted pairwise matches is displayed in the webdialog report as shown in Fig 6 15 b If the number of pairwise matches falls below 100 it is likely that camera pose recovery will fail After pose recovery completes its task the module prints the number of photos whose poses were successfully recovered If camera pose recovery fails then the user is advised to acquire additional scene photos to yield a higher density of images along the path The user may otherwise create a new project and select No when asked Are the photos in overlapping sequence If that proves inadequate as well then manual mode Sec 6 6 2 must be employed to produce the desired results Feature extraction and matching are the most time consuming modules The amount of time required to complete these steps depends on the dimensions of the photos and their content The user can interrupt feature extraction or matching at any time by clicking the 09 54 TOOLBAR REFERENCE otop Pose Recovery e icon The requested interrupt waits until the current image is fully processed to avoid corrupting the saved data After feature matching is complete the user is prompted prior to continuing with the next step The user may either stop and save the project to continue at another time or immediately commence with camera pose reco
56. orth in the image sequence to do inspection It is strongly recommended that features be picked across a wide distribution of the scene to avoid bias in computing recovered camera poses Therefore refrain from picking most of the features from one planar facade We define a track to be a list of correspondence points of a single feature that is visible across the image sequence Due to occlusions we permit the tracked feature to be visible across non consecutive image frames in the sequence To start a new track press the k key i Notice that the shape of the cursor changes from to to indicate that the tool is ready to accept input Move the cursor near the feature and click the left mouse button to pop up a 54 6 6 FEATURE TRACKING CAMERA POSE RECOVERY 55 ie dub Vine mn Dres Dash Wes epa ii BOC 29254x0f 97 AQGL PtP J A k EEN FYFE p ou ad oAegaaxSSUXLST Soo D Fai 11 ate pOL gg Dabei raahaa wert Wh Det NIA Leod I I Figure 6 18 magnifying window While the left mouse button is held down the user can slide the magnified image to accurately pinpoint the feature see Fig a a um ee toe mam haus Bm usu UN Bia um umm a be e mius p mm Wer OPES LAO 974A SS SF Wer COP ES LAO 974A Ge EF maio kena Pas SRY YAra LEE ah Sys ho Lena is SRY Fee Est AAA Figure 6 19 a Selecting the feature in the first frame b The path of tracked fea
57. point and radial lens distortion extrinsic camera parameters position and orientation and global alignment with respect to the ground plane Integrating these modules alongside the sketching and push pull paradigm of SketchUp facilitates a superior user interface experience It is this cohesion that helps make PhotoSketch a distinctly intuitive urban 3D modeling tool PhotoSketch is a fully integrated plugin for SketchUp 15 or later The plugin works with the free or Pro versions of SketchUp on the Windows and Mac platforms All PhotoSketch commands which interact seamlessly with SketchUp are accessible via the menubar or a dedicated PhotoSketch toolbar that lies alongside the standard SketchUp toolbars 1 2 PHOTOSKETCH WORKFLOW 5 1 2 PHOTOSKETCH WORKFLOW Conventional modeling tools often use photographs as a postprocess to phototexture a model after the model is generated PhotoSketch however uses photographs as the starting point before the model is constructed from multiple views Indeed our workflow treats photographs as tracing paper upon which 2D shapes are defined prior to extruding them into 3D models The very photographs that serve as the basis for the models automatically serve as the texture elements for them as well thereby facilitating photorealistic visualization In this section we describe the PhotoSketch modeling workflow and demonstrate how its design simplifies the user experience for modeling urban areas We now rev
58. point correspondence method e esc Figure 5 11 Select correspondence method for floor alignment 30 5 4 FLOOR ALIGNMENT 31 Figure 5 12 Floor alignment is achieved by tracking at least three correspondence points that all have the same height among two input photos Sas E GLELxXOF ST9 45 DDO SF GLELKOF HS7A2K DS CP LO re gt a SY YwwtN LES aaa UO beneh 5 SRS Y Ywety LES Adee Figure 5 13 a A popup magnifier helps the user pinpoint the correspondence points b Three corners of the building are tracked in all photos and the motion of the tracked corners are depicted with yellow lines l the sparse point cloud is shown through the first camera a After floor alignment TUTORIAL 32 AMOC OP ESL il o E r B 5 14 frustum b Click on the 3D point as highlighted by the cursor to snap the floor to that height 32 Figure 5 5 DRAWING 2D FOOTPRINTS 33 5 5 DRAWING 2D FOOTPRINTS Click on the Snap to 3D point icon ue and select a 3D point to snap the floor plane to that position as shown in Fig b Now you can use 2D drawing tools to draw the footprint from the top Click on the Rectangle icon E and draw two rectangles To draw a 2D rectangle you need to click on three corners near the roof as shown in Fig This process of snapping the floor plane and drawing the rectangles is repeated for all of the footprints that are visible in th
59. r each camera The progress bar shows the percentage completed Fig 5 9 At the end of this step the user is required to perform floor alignment click OK to do the floor alignment see Fig 5 10 5 4 FLOOR ALIGNMENT Once camera pose recovery is complete the plugin prompts the user to select between two floor alignment methods line based or point based correspondence Fig 5 11 Click No to initiate the floor alignment module that operates by establishing corresponding points on the floor plane The first photo from the path is now displayed requiring you to track at least three points from photos in the path Be sure to select points that have the same heights and all lie on a plane An example of three correspondence points that all have the same height is shown in Fig 5 12 To start a new track press k on the keyboard The shape of the cursor changes from i to Move the cursor near the feature and click the left mouse button A magnifying window pops up While the left mouse button is down the user can slide the magnified image to accurately pinpoint the feature see Fig 5 13 a When you select any point on the image a window pops up to show a magnified portion of the image The mouse may be dragged within the window to inspect the magnified neighborhood more closely Once the mouse button is released the central point under the cross hair will be selected This permits you to accurately pinpoint the interest
60. ration parameters listed in Fig 6 5 a should click OK to enter these values manually Otherwise the user should click Cancel and select the Camera Calibration Ye icon to supply a set of checkerboard images acquired by this camera Once the camera is calibrated the user should click on the New Project icon and start the procedure again This time the system will have found the camera calibration information and directly show you the calibration input dialog shown in Fig 6 5 a Camera Calibration Parameters N g y vic sa MIN IB at lo amp 52 Maker No calibration information found for the following camera C EOS DIGITAL REBEL x aS eue RN Model Canon EOS DIGITAL REBEL XTi Focal x 1589 36 Resolution 1944x1296 Focal length 46 0 mm Focal y 1593 79 This camera must now be calibrated If you are an advanced user and already know the focal length Center y 625 97 in pixels center of projection and radial lens distortion parameters then click OK and enter these values manually Otherwise please click Cancel and select the Camera kl 0 04128 Calibration icon 3X to supply a set of checkerboard images acquired by this camera Center x 972 63 k0 0 14643 po 0 00468 p1 0 00405 See Section 3 of the PhotoSketch user for more details ok Cancel a b Figure 6 5 a Calibration input dialog b Al
61. raw Took W ndow Plogins Help is BOC e FREISAOE 2 ATUS Aper m o D e Select obsects Shift to extend select Drag mouse t Select multiple Measurements Figure 6 33 Only side faces are selected Vike bait View Camera Dow loos W ndow Pagan Heip WeCcosgp54x90 SAAR AE 8Lo CP BY Try EY 4 3 60 SERB RANK S WV etn LEE dm M DE e Freeze textures of selected Faces GO GP ready Mastoremarts Figure 6 34 The user can decide to freeze the textures of the selected faces 65 66 TOOLBAR REFERENCE Tas r eee Pore Ss 4NOCco4 54XxOF SZAR GR 8L0 8 9777w9 HY KOOTERS SS AE Ww OY DOCH SES d de M k Oo CF select objects Shift to extend select Drag mouse to select multiple Measurement Figure 6 35 The final result of the texture mapping EI onore ils S file CAM View Came Dew Toos Window Hel ZWO C o425 x OCeozeacsgp amp toetS W v7wWU PEOTI EELEE AIS TENYT Seo oO Figure 6 36 The user can unfreeze the faces textures 66 6 19 ADVANCED TEXTURE MAPPING 67 6 19 Ww ADVANCED TEXTURE MAPPING Clicking on this icon projects texture on the visible faces This method calls our advanced texture mapping that uses selected photos to texture map faces and also detects occluding areas Since the Sketchup API does not allow us to access the materials content directly we need to generate new supertextures and store them on disk and read them back into oketchup Therefore th
62. s for the camera calibration 4 4 CAMERA CALIBRATION IN PHOTOSKETCH After acquiring the checkerboard images create a new folder and store the images in the folder Do not mix images of checkerboards from different cameras or zoom settings Once in Sketchup click on the X icon to popup a file browser Fig 4 5 Navigate to the folder containing the checkerboard images and select any one of the images This is done by either 22 4 4 CAMERA CALIBRATION IN PHOTOSKETCH 23 f Import calibration photos m Look in i calibration T ci Eg Name Date modified Type Size ay Recent Places IMG 009 IMG 009 Desktop IMG 009 IMG 009 Computer D Network 3 D m 5 Hi IMG 010 IMG 010 IMG 010 IMG 010 IMG 010 IMG 010 IMG 010 Files of type Cancel Figure 4 5 Select any input image to initiate calibration module for all images in directory double clicking an image icon or single clicking the icon and pressing Open Selecting just one image will initiate the reading of all images in that directory Once the images are read the calibration process begins automatically The calibration module first extracts features such as corners from the checkerboard images to solve for the focal length aspect ratio center of projection and radial distortion parameters The module then displays the thumbnails of the input images and the recovered intrinsic parameters of the camera Fig 4 6 The c
63. splays the next photo in the sequence The user must then define the corresponding line segment in the same manner To help the user pick the line endpoints the tool shows a sample window from the previous line segment Fig 627 58 6 7 FLOOR ALIGNMENT 59 Figure 6 25 Click the left mouse button to pop up a magnifier window to help select an endpoint on a vertical edge The user can repeat the above procedure to add more corresponding line segments from different parts of the scene by pressing the k key To delete a line segment from the database first select the line segment by clicking near the line When a line segment is selected the color changes from yellow to cyan Then press d on the keyboard and the line segment and its corresponding line segment will be deleted To edit the start or end of a line segment first select the line segment by clicking near the line When a line segment is selected the color changes from yellow to cyan Then press K on the keyboard to activate the editing mode Then click near the starting or ending point and the tool allows the user to reposition the point A sample window will appear and helps the user recall the position of the corresponding line segment To exit from editing mode press K on the keyboard see Fig 6 28 To exit from this floor alignment tool click the right mouse button and select Done Fig 6 29 The tool then computes the necessary transformation for the floor align
64. st starting SketchUp after the Pho toSketch plugin is installed The request number in the window is uniquely derived from your computer Click on the blue URL link to take you to b an activation request page Notice that the request number is automatically populated on the activation request page Once you complete the form and click Send you will receive an email with your activation code Copy and paste that code into the activation dialog window in a to unlock the plugin 2 1 2 Purchasing and Installing the Commercial Version of PhotoSketch To purchase and install the commercial version of PhotoSketch 1 Visit www brainstormllc com and click the commercial Buy Now button on the left 2 Purchase a commercial PhotoSketch license using a credit card or a PayPal account 3 You will immediately receive a receipt from PayPal A second email containing your serial number and a link to PhotoSketch3 0 exe will arrive within one business day 4 Wait to receive the second email before you uninstall any demo version of PhotoSketch on your machine On Windows click Start gt Programs gt PhotoSketch gt Uninstall 5 Download and run the PhotoSketch3 0 exe file to install the plugin files 6 Open SketchUp A registration dialog window Fig will appear prompting you for your serial number Copy and paste your serial number to activate the plugin 7 Enjoy PhotoSketch 2 1 Getting Started 11 Activation imm m A Serial N
65. tangle polyline circle ellipse or spline and outline the visible footprint of the building This process only requires the user to click on the corners of the building facades To assist the user in this process we provide a magnifying glass to help the user accurately pinpoint the corners Our system allows the user to switch from one viewpoint to another during sketching to add points from corners that are occluded in the current view Fig 1 5 a shows the footprints shown in black Since the cameras are calibrated a footprint drawn in one viewpoint will appear registered in the other viewpoints and will be aligned with the sparse 3D point cloud 1 2 6 Extrusion taper footprints to form volumes The basis of our work assumes that a simple set of extrusion and taper operations is adequate to model a rich set of urban structures The simplest available operation in our toolset is ri 8 INTRODUCTION Figure 1 5 a The user has sketched a 2D footprint of the building on one of the images The 2D footprint is shown in black in the different camera views b Volume blocks are created as a result of extrusion operations extrusion from footprints The user only needs to drag the footprint to the desired height This can be done either by snapping to the height of a 3D point from the point cloud or to a visual cue on the image based on dynamic texturing Fig I1 5 b shows the result of an extrusion operation on the footprint of Fig 1
66. the camera away from the board while keeping the orientation steady All subsequent pictures should be taken across a wide range of angles and distances from the checkerboard Feel 20 4 5 GUIDELINES 21 free to tilt the camera with respect to the checkerboard and to rotate it about its optical axis Fig 4 3 shows a typical set of camera poses from which to acquire images of the calibration pattern Figure 4 3 Suggested camera poses for acquiring the checkerboard image for calibration 4 3 GUIDELINES There is one requirement that must apply to the images of the calibration pattern do not change the zoom settings or camera resolution They must be the same settings that are applied during scene capture It is important to note that there is no need to repeat the calibration stage across projects as long as the zoom settings and camera resolution remain the same Just calibrate the camera once and the computed intrinsic parameters can be re used many times If either the zoom setting or camera resolution changes for a specific project the software will automatically detect this change and prompt the user to recalibrate the camera for the new settings Each time the camera is calibrated for a specific zoom resolution setting the intrinsic parameters are saved in the PhotoSketch plugin The quality of the images you acquire plays a critical role in a successful calibration Here are some tips for taking pictures of the checkerboard image t
67. the model now appears 1 2 PHOTOSKETCH WORKFLOW 7 tilted as shown in Fig 1 4 a A ground plane alignment stage is necessary to properly rotate the camera and the sparse point cloud as shown in Fig 1 4 b This leaves the floor parallel to the ground plane Floor alignment is achieved by manually selecting at least three corresponding image points in two views that correspond to the floor or roofline in the image The 3D position of these selected image points can be determined by triangulation since the camera poses are known A plane is fitted to these 3D points The angle between the fitted plane and the ground plane of the world coordinate system determines the rotation angle necessary to rigidly rotate the 3D point cloud and cameras to leave the floor parallel to the ground plane a Before floor alignment b After floor alignment Figure 1 4 Since the multiview geometry does not have knowledge of ground orientation the structure and poses are not aligned with respect to floor Therefore we need a tool to properly align the ground and floor 1 2 5 Footprint sketching using a 2D drawing toolset After the cameras and the floor are aligned to the ground plane the user can select images from the input set and look at the 3D scene through their respective camera frustums The user then sketches on the ground plane That sketch is properly projected into all of the remaining views The user can select a 2D drawing tool such as a rec
68. time without having to recompute this data If however the user wishes to leave the job running unsupervised then the flag should be cleared so that the computation may proceed uninterrupted after the webdialog reports issued during camera pose recovery are updated In the fourth tab the user can specify miscellaneous options Fig 6 7 d The frustum opacity value lies in the O 1 range with the default value being 0 65 A value of 0 1 denotes a fully transparent opaque photo in the frustum Fig 6 10 shows the effect of changing the frustum opacity Another miscellaneous option the user can control is the number of segments for gener ating arcs and circles A higher number of line segments causes the extrusion operation to generate many narrow slabs which can increase the number of faces in the model unneces sarily Try to keep this value in the 10 36 range 6 5 a CAMERA CALIBRATION The procedure for performing camera calibration is described in Sec 4 This stage re quires the acquisition of checkerboard images from multiple viewpoints using a single focal length After these images are acquired create a new folder and store the images in the 49 50 TOOLBAR REFERENCE Es c Ywetn tit aaa p o meom B epa o HOS NW TET P ee zF a af fd sammi bs ey hoa M eg ant e La p uu r 2 44 7 Lr 1 ee PES m UPS am Om i E i n P J T wv e gt T LI d uo E i LI vA ME
69. tive to SketchUp occluding faces do not terminate the projected rays from striking occluded faces This is clearly visible on the side wall of the rear building and along the sloped faces of the roof PhotoSketch includes an advanced texturing mapping mode that performs occlusion detection to identify occluded pixels and paint them with a gray color This advanced 39 40 TUTORIAL Figure 5 25 a Textured version of Fig 5 24 d Additional geometric elements added to the roof as seen from b the camera frustum and c another view that lies outside the frustum The view in c illustrates the drawbacks with standard texture mapping in SketchUp which does not perform occlusion detection to prevent occluded faces from receiving projected tex ture This problem is fixed with the advanced texture mapping feature in PhotoSketch texture mapping can be applied by clicking on the icon This produces the result shown in Fig 5 26 a Although many occluded pixels are now painted gray there are still incorrectly rendered pixels due to errors in the geometry Since the roof was modeled higher than it actually is the ground based photo of the building manages to project onto the roof when in fact the front face of the building would normally have occluded it Therefore by lowering the top face of the roof to its proper height the result is correctly rendered Fig 5 26 b Figure 5 26 a Advanced textured mapping performs occlusion dete
70. ture in the sequence After releasing the left mouse button the tool switches to the next photo in the path The previously selected feature is now depicted with a red plus symbol The user needs to pick the same feature point in the new photo as accurately as possible In order to help the user recall the previously selected feature a window depicting the neighborhood around that feature in the previous photo is attached alongside the cursor see Fig 6 19 b The window is intended to be a visual cue for searching the corresponding feature in the current photo That visual cue disappears once the left mouse button is pressed to expose the magnifier window for selecting the corresponding feature point in the current photo Continue tracking the same feature throughout the sequence As the track is grown with each input frame the user will see an updated polyline that connects the red plus symbols to denote the movement of the selected feature If the feature is occluded in a frame the user 59 56 TOOLBAR REFERENCE can simply skip that frame The user can jump back and forth in the sequence at any time by pressing the left or right arrow keys on the keyboard Fig l6 20 depicts the path of the tracked feature window corner in the photo sequence The position of the tracked feature is denoted by a red plus symbol If that feature point is associated with the current frame however a red square is used instead to indicate the feature s pos
71. ture point the position of that point is modified to lie below the cursor However if the click does not occur near a feature point a new feature point will be added Press the k key to exit the editing mode If the yellow tracked lines clutter the scene the user can toggle them to turn on or off by pressing the x key see Fig b When the pose of a photo is recovered the reconstructed 3D points are projected on the 2D photo and displayed with green X s In order to gauge the accuracy of camera pose 56 6 7 FLOOR ALIGNMENT 57 recovery these projected points must coincide with the tracked feature points that are depicted with red plus symbols or red squares 1354 XxO 622X 9 Joep ISSLXOC OZARD es tare gt Ypoetr Lit Asad 5 RTTY Ywernrn Lis AATA Figure 6 22 a The final tracked features manually by the user b Pressing the x key will turn off the display of the polylines depicting the tracked features To exit from the manual tracking and to finalize the pose recovery click the right mouse button and a context menu will appear Select done to exit see Fig 6 23 The manual tracking tool will then compute the overall reprojection error If the error is too high the tool simply quits If the error is reasonable the error is displayed to the user and prompts the user whether to go ahead do the final pose recovery followed by floor alignment A one level undo ctrl Z feature is supported for undoing the last
72. umber First Name Last Name Company Street City State Province Country Zip Postal Code Phone E mail i Activate Later Figure 2 2 Registration dialog window appears upon first starting SketchUp after you run PhotoSketch3 0 exe to install the PhotoSketch plugin Copy and paste the serial number that was emailed to you into this window and complete the form to unlock the plugin 2 1 3 Purchasing and Installing the Academic Version of PhotoSketch To purchase and install the academic version of PhotoSketch 1 Visit www brainstormllc com and click the academic Buy Now button on the right 2 Purchase an academic PhotoSketch license using a credit card or a PayPal account 3 You Will immediately receive a receipt from PayPal Forward a copy of the receipt with a scanned copy of your academic ID in pdf or jpg format to sales brainstormllc com 4 A second email containing your serial number and a link to PhotoSketch3 0 exe will arrive within two business days after we receive a scanned copy of your academic ID You will get a full refund if we do not approve validate your ID 5 Wait to receive the second email before you uninstall any demo version of PhotoSketch on your machine On Windows click Start gt Programs gt PhotoSketch gt Uninstall 6 Download and run the PhotoSketch3 0 exe file to install the plugin files 7 Open SketchUp A registration dialog window Fig will appear prompt
73. very to estimate the position and orientation of the cameras that acquired the input images At the end of this camera pose recovery step the system displays the number of photos whose poses were successfully recovered Fig 6 1 Camera pose recovery Overall progress 1 726 Processing park_001 jpg gt park 002 jpg Overall progress 7526 Adding photo park 001 jpg Figure 6 17 Pose recovery progress bar Camera pose recovery brings the photos and their associated cameras into a single co ordinate system The principal plane of that system however may not coincide with the floor In order to achieve this necessary alignment click OK to initiate the floor alignment module 6 6 2 Manual Feature Tracking Camera Pose Recovery When the user selects manual feature tracking camera pose recovery its tool will be acti vated and the first photo of the sequence will be shown see Fig 6 18 The user can move forward or backward in the photo sequence by pressing the left or right arrow keys on the key board The goal of this tool is to pick a salient feature such as a corner on a window rooftop or footprint from one photo and track it among the rest of the photos in the sequence Note that it is important to pick features that can be tracked over the longest set of images in the sequence Therefore the user needs to inspect the feature before starting a new track Simply press the left or right arrow keys to go back and f
74. views Therefore the textures will be sharp Fig 6 32 shows such a scenario Notice that the side of building is visible by all cameras However only one camera view provides the best texture for the side faces The user can select the side faces using the SketchUp selection tool and hide all but the best camera as shown in Fig When the user clicks on S the user can freeze the face textures by clicking Yes on the popup dialog Fig 6 34 Fig 6 35 shows the final result The user can unfreeze the selected face textures by clicking the right mouse button and selecting Unfreeze texture from the context menu Fig 6 36 63 64 TOOLBAR REFERENCE Vie d View Camera Draw Took Window Plugins Help L ae M Dno Se xoroz aAGwSELO Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 z Feature extraction size 16670 E Feature matching size 2805 E Pose recovery error 377 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm z Feature extraction size 14962 E Feature matching size 2984 z Pose recovery error 2 99 Image size 1936 x 1288 Maker Canon Model Canon EOS DIGITAL REBEL XS Focal Length 18 0 mm QD O select objects shit to extend select Drag mouse to select multiple Figure 6 32 Example of many cameras see the side faces 64 6 18 APPLY TEXTURE 65 me Pda View Comers D

user manual - Brainstorm Technology LLC

Contents

Download Pdf Manuals

Related Search

Related Contents