COMP 4010 Lecture10: AR Tracking
â˘
9 likesâ˘4,264 views
Lecture prepared by Mark Billinghurst on Augmented Reality tracking. Taught on October 18th 2016 by Dr. Gun Lee as part of the COMP 4010 VR class at the University of South Australia.
![Augmented Reality Definition
â˘âŻ Defining Characteristics [Azuma 97]
â˘âŻCombines Real andVirtual Images
â˘âŻBoth can be seen at the same time
â˘âŻInteractive in real-time
â˘âŻThe virtual content can be interacted with
â˘âŻRegistered in 3D
â˘âŻVirtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to COMP 4010 Lecture10: AR Tracking
Similar to COMP 4010 Lecture10: AR Tracking (20)
More from Mark Billinghurst
More from Mark Billinghurst (20)
Recently uploaded
Recently uploaded (20)
COMP 4010 Lecture10: AR Tracking
- 1. LECTURE 10: AR TECHNOLOGY: TRACKING COMP 4010 â Virtual Reality Semester 5 â 2016 Bruce Thomas, Mark Billinghurst University of South Australia October 18th 2016
- 2. Augmented Reality Definition â˘âŻ Defining Characteristics [Azuma 97] â˘âŻCombines Real andVirtual Images â˘âŻBoth can be seen at the same time â˘âŻInteractive in real-time â˘âŻThe virtual content can be interacted with â˘âŻRegistered in 3D â˘âŻVirtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 3. Augmented RealityTechnology â˘âŻ Combining Real and Virtual Images â˘âŻ Display technologies â˘âŻ Interactive in Real-Time â˘âŻ Input and interactive technologies â˘âŻ Registered in 3D â˘âŻ Viewpoint tracking technologies Display Processing Input Tracking
- 5. AR RequiresTracking and Registration â˘âŻRegistration â˘âŻ Positioning virtual object wrt real world â˘âŻ Fixing virtual object on real object when view is fixed â˘âŻTracking â˘âŻ Continually locating the users viewpoint when view moving â˘âŻ Position (x,y,z), Orientation (r,p,y)
- 6. AR TRACKING
- 7. Tracking Requirements â˘âŻAugmented Reality Information Display â˘âŻ World Stabilized â˘âŻ Body Stabilized â˘âŻ Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
- 8. Tracking Technologies !⯠Active â˘âŻ Mechanical, Magnetic, Ultrasonic â˘âŻ GPS, Wifi, cell location !⯠Passive â˘âŻ Inertial sensors (compass, accelerometer, gyro) â˘âŻ Computer Vision â˘âŻ Marker based, Natural feature tracking !⯠Hybrid Tracking â˘âŻ Combined sensors (eg Vision + Inertial)
- 10. MechanicalTracker â˘âŻIdea: mechanical arms with joint sensors â˘âŻ++: high accuracy, haptic feedback â˘âŻ-- : cumbersome, expensive Microscribe
- 11. MagneticTracker â˘âŻIdea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. â˘âŻ++: 6DOF, robust â˘âŻ-- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
- 12. InertialTracker â˘âŻIdea: measuring linear and angular orientation rates (accelerometer/gyroscope) â˘âŻ++: no transmitter, cheap, small, high frequency, wireless â˘âŻ-- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote
- 13. UltrasonicTracker â˘âŻIdea: time of Flight or phase-Coherence Sound Waves â˘âŻ++: Small, Cheap â˘âŻ-- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600
- 14. Global Positioning System (GPS) â˘âŻCreated by US in 1978 â˘âŻCurrently 29 satellites â˘âŻSatellites send position + time â˘âŻGPS Receiver positioning â˘âŻ4 satellites need to be visible â˘âŻDifferential time of arrival â˘âŻTriangulation â˘âŻAccuracy â˘âŻ5-30m+, blocked by weather, buildings etc.
- 15. Mobile Sensors â˘âŻInertial compass â˘âŻEarthâs magnetic field â˘âŻMeasures absolute orientation â˘âŻAccelerometers â˘âŻMeasures acceleration about axis â˘âŻUsed for tilt, relative rotation â˘âŻCan drift over time
- 16. OPTICAL TRACKING
- 17. Why Optical Tracking for AR? â˘âŻ Many AR devices have cameras â˘âŻ Mobile phone/tablet, Video see-through display â˘âŻ Provides precise alignment between video and AR overlay â˘âŻ Using features in video to generate pixel perfect alignment â˘âŻ Real world has many visual features that can be tracked from â˘âŻ Computer Vision well established discipline â˘âŻ Over 40 years of research to draw on â˘âŻ Old non real time algorithms can be run in real time on todays devices
- 18. Common AR Optical Tracking Types â˘âŻMarker Tracking â˘âŻ Tracking known artificial markers/images â˘âŻ e.g. ARToolKit square markers â˘âŻMarkerless Tracking â˘âŻ Tracking from known features in real world â˘âŻ e.g. Vuforia image tracking â˘âŻUnprepared Tracking â˘âŻ Tracking in unknown environment â˘âŻ e.g. SLAM tracking
- 19. Marker tracking â˘âŻ Available for more than 10 years â˘âŻ Several open source solutions exist â˘âŻ ARToolKit,ARTag,ATK+, etc â˘âŻ Fairly simple to implement â˘âŻ Standard computer vision methods â˘âŻ A rectangle provides 4 corner points â˘âŻ Enough for pose estimation!
- 21. Marker Based Tracking: ARToolKit http://www.artoolkit.org
- 22. Goal:Find Camera Pose â˘âŻ Goal is to find the camera pose in maker coordinate frame â˘âŻ Knowing: â˘âŻ Position of key points in on-screen video image â˘âŻ Camera properties (focal length, image distortion)
- 23. Coordinates for Marker Tracking
- 24. Coordinates for Marker Tracking Marker Camera â˘âŻFinal Goal â˘âŻRotation & Translation 1: Camera Ideal Screen â˘âŻPerspective model â˘âŻObtained from Camera Calibration 2: Ideal Screen Observed Screen â˘âŻNonlinear function (barrel shape) â˘âŻObtained from Camera Calibration 3: Marker Observed Screen â˘âŻCorrespondence of 4 vertices â˘âŻReal time image processing
- 26. MarkerTracking â Fiducial Detection â˘âŻ Threshold the whole image to black and white â˘âŻ Search scanline by scanline for edges (white to black) â˘âŻ Follow edge until either â˘âŻ Back to starting pixel â˘âŻ Image border â˘âŻ Check for size â˘âŻ Reject fiducials early that are too small (or too large)
- 27. MarkerTracking â Rectangle Fitting â˘âŻ Start with an arbitrary point âxâ on the contour â˘âŻ The point with maximum distance must be a corner c0 â˘âŻ Create a diagonal through the center â˘âŻ Find points c1 & c2 with maximum distance left and right of diag. â˘âŻ New diagonal from c1 to c2 â˘âŻ Find point c3 right of diagonal with maximum distance
- 28. MarkerTracking â Pattern checking â˘âŻ Calculate homography using the 4 corner points â˘âŻ âDirect Linear Transformâ algorithm â˘âŻ Maps normalized coordinates to marker coordinates (simple perspective projection, no camera model) â˘âŻ Extract pattern by sampling and check â˘âŻ Id (implicit encoding) â˘âŻ Template (normalized cross correlation)
- 29. MarkerTracking â Corner refinement â˘âŻ Refine corner coordinates â˘âŻ Critical for high quality tracking â˘âŻ Remember: 4 points is the bare minimum! â˘âŻ So these 4 points should better be accurate⌠â˘âŻ Detect sub-pixel coordinates â˘âŻ E.g. Harris corner detector â˘âŻ Specialized methods can be faster and more accurate â˘âŻ Strongly reduces jitter! â˘âŻ Undistort corner coordinates â˘âŻ Remove radial distortion from lens
- 30. Marker tracking â Pose estimation â˘âŻ Calculates marker pose relative to the camera â˘âŻ Initial estimation directly from homography â˘âŻ Very fast, but coarse with error â˘âŻ Jitters a lot⌠â˘âŻ Iterative Refinement using Gauss-Newton method â˘âŻ 6 parameters (3 for position, 3 for rotation) to refine â˘âŻ At each iteration we optimize on the error â˘âŻ Iterat
- 31. From MarkerTo Camera â˘âŻ Rotation & Translation TCM : 4x4 transformation matrix from marker coord. to camera coord.
- 32. Tracking challenges inARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
- 34. But - You canât cover world with ARToolKit Markers!
- 35. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking â˘âŻ No more Markers! "Markerless Tracking Mechanical Tracker
- 36. Natural Feature Tracking â˘âŻUse Natural Cues of Real Elements â˘âŻ Edges â˘âŻ Surface Texture â˘âŻ Interest Points â˘âŻModel or Model-Free â˘âŻNo visual pollution Contours Features Points Surfaces
- 37. TextureTracking
- 38. Tracking by Keypoint Detection â˘âŻ This is what most trackers do⌠â˘âŻ Targets are detected every frame â˘âŻ Popular because tracking and detection are solved simultaneously Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
- 39. What is a Keypoint? â˘âŻ It depends on the detector you use! â˘âŻ For high performance use the FAST corner detector â˘âŻ Apply FAST to all pixels of your image â˘âŻ Obtain a set of keypoints for your image â˘âŻ Describe the keypoints Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
- 40. FAST Corner Keypoint Detection
- 42. Descriptors â˘âŻ Describe the Keypoint features â˘âŻ Can use SIFT â˘âŻ Estimate the dominant keypoint orientation using gradients â˘âŻ Compensate for detected orientation â˘âŻ Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmals<eg D., Real-Time Detec<on and Tracking for Augmented Reality on Mobile Phones. IEEE Transac<ons on Visualiza<on and Computer Graphics, May/June, 2010
- 43. Database Creation â˘âŻ Offline step â create database of known features â˘âŻ Searching for corners in a static image â˘âŻ For robustness look at corners on multiple scales â˘âŻ Some corners are more descriptive at larger or smaller scales â˘âŻ We donât know how far users will be from our image â˘âŻ Build a database file with all descriptors and their position on the original image
- 44. Real-time tracking â˘âŻ Search for known keypoints in the video image â˘âŻ Create the descriptors â˘âŻ Match the descriptors from the live video against those in the database â˘âŻ Brute force is not an option â˘âŻ Need the speed-up of special data structures â˘âŻ E.g., we use multiple spill trees Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
- 45. NFT â Outlier removal â˘âŻ Removing outlining features â˘âŻ Cascade of removal techniques â˘âŻ Start with cheapest, finish with most expensive⌠â˘âŻ First simple geometric tests â˘âŻ E.g., line tests â˘âŻ Select 2 points to form a line â˘âŻ Check all other points being on correct side of line â˘âŻ Then, homography-based tests
- 46. NFT â Pose refinement â˘âŻ Pose from homography makes good starting point â˘âŻ Based on Gauss-Newton iteration â˘âŻ Try to minimize the re-projection error of the keypoints â˘âŻ Part of tracking pipeline that mostly benefits from floating point usage â˘âŻ Can still be implemented effectively in fixed point â˘âŻ Typically 2-4 iterations are enoughâŚ
- 47. NFT â Real-time tracking â˘âŻ Search for keypoints in the video image â˘âŻ Create the descriptors â˘âŻ Match the descriptors from the live video against those in the database â˘âŻ Remove the keypoints that are outliers â˘âŻ Use the remaining keypoints to calculate the pose of the camera Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
- 48. Demo: Vuforia Texture Tracking https://www.youtube.com/watch?v=1Qf5Qew5zSU
- 49. Edge Based Tracking â˘âŻ Example: RAPiD [Drummond et al. 02] â˘âŻ Initialization, Control Points, Pose Prediction (Global Method)
- 50. Demo: Edge Based Tracking https://www.youtube.com/watch?v=UJtpBxdDVDU
- 51. Line Based Tracking â˘âŻ Visual Servoing [Comport et al. 2004]
- 52. Model BasedTracking â˘âŻTracking from 3D object shape â˘âŻExample: OpenTL - www.opentl.org â˘âŻ General purpose library for model based visual tracking
- 53. Demo: OpenTL Model Tracking https://www.youtube.com/watch?v=laiykNbPkgg
- 54. Demo: OpenTL Face Tracking https://www.youtube.com/watch?v=WIoGdhkfNVE
- 55. Marker vs.Natural FeatureTracking â˘âŻ Marker tracking â˘âŻ Usually requires no database to be stored â˘âŻ Markers can be an eye-catcher â˘âŻ Tracking is less demanding â˘âŻ The environment must be instrumented â˘âŻ Markers usually work only when fully in view â˘âŻ Natural feature tracking â˘âŻ A database of keypoints must be stored/downloaded â˘âŻ Natural feature targets might catch the attention less â˘âŻ Natural feature targets are potentially everywhere â˘âŻ Natural feature targets work also if partially in view
- 56. Tracking from an Unknown Environment â˘âŻ What to do when you donât know any features? â˘âŻ Very important problem in mobile robotics - Where am I? â˘âŻ SLAM â˘âŻ Simultaneously Localize And Map the environment â˘âŻ Goal: to recover both camera pose and map structure while initially knowing neither. â˘âŻ Mapping: â˘âŻ Building a map of the environment which the robot is in â˘âŻ Localisation: â˘âŻ Navigating this environment using the map while keeping track of the robotâs relative position and orientation
- 57. Visual SLAM â˘âŻ Early SLAM systems (1986 - ) â˘âŻ Computer visions and sensors (e.g. IMU, laser, etc.) â˘âŻ One of the most important algorithms in Robotics â˘âŻ Visual SLAM â˘âŻ Using cameras only, such as stereo view â˘âŻ MonoSLAM (single camera) developed in 2007 (Davidson)
- 59. How SLAMWorks â˘âŻ Three main steps 1.⯠Tracking a set of points through successive camera frames 2.⯠Using these tracks to triangulate their 3D position 3.⯠Simultaneously use the estimated point locations to calculate the camera pose which could have observed them â˘âŻ By observing a sufficient number of points can solve for for both structure and motion (camera path and scene structure).
- 60. SLAM Optimization â˘âŻ SLAM is an optimisation problem â˘âŻ compute the best configuration of camera poses and point positions in order to minimise reprojection error â˘âŻ difference between a point's tracked location and where it is expected to be â˘âŻ Can be solved using bundle adjustment â˘âŻ a nonlinear least squares algorithm that finds minimum error â˘âŻ But â time taken grows as size of map increases â˘âŻ Multi-core machines can do localization and mapping on different threads â˘âŻ Relocalisation â˘âŻ Allows tracking to be restarted when it fails
- 61. Evolution of SLAM Systems â˘âŻ MonoSLAM (Davidson, 2007) â˘âŻ Real time SLAM from single camera â˘âŻ PTAM (Klein, 2009) â˘âŻ First SLAM implementation on mobile phone â˘âŻ FAB-MAP (Cummins, 2008) â˘âŻ Probabilistic Localization and Mapping â˘âŻ DTAM (Newcombe, 2011) â˘âŻ 3D surface reconstruction from every pixel in image â˘âŻ KinectFusion (Izadi, 2011) â˘âŻ Realtime dense surface mapping and tracking using RGB-D
- 63. LSD-SLAM (Engel 2014) â˘âŻ A novel, direct monocular SLAM technique â˘âŻ Uses image intensities both for tracking and mapping. â˘âŻ The camera is tracked using direct image alignment, while â˘âŻ Geometry is estimated as semi-dense depth maps â˘âŻ Supports very large scale tracking â˘âŻ Runs in real time on CPU and smartphone
- 65. Direct Method vs.Feature Based â˘âŻ Direct uses all information in image, cf feature based approach that only use small patches around corners and edges
- 66. Applications of SLAM Systems â˘âŻ Many possible applications â˘âŻ Augmented Reality camera tracking â˘âŻ Mobile robot localisation â˘âŻ Real world navigation aid â˘âŻ 3D scene reconstruction â˘âŻ 3D Object reconstruction â˘âŻ Etc.. â˘âŻ Assumptions â˘âŻ Camera moves through an unchanging scene â˘âŻ So not suitable for person tracking, gesture recognition â˘âŻ Both involve non-rigidly deforming objects and a non-static map
- 67. Hybrid Tracking Combining several tracking modalities together
- 68. SensorTracking â˘âŻ Used by many âAR browsersâ â˘âŻ GPS, compass, accelerometer, gyroscope â˘âŻ Not sufficient alone (drift, interference) Inertial Compass Drifting Over Time
- 69. Combining Sensors andVision â˘âŻ Sensors â˘âŻ Produces noisy output (= jittering augmentations) â˘âŻ Are not sufficiently accurate (= wrongly placed augmentations) â˘âŻ Gives us first information on where we are in the world, and what we are looking at â˘âŻ Vision â˘âŻ Is more accurate (= stable and correct augmentations) â˘âŻ Requires choosing the correct keypoint database to track from â˘âŻ Requires registering our local coordinate frame (online- generated model) to the global one (world)
- 70. Example: Outdoor Hybrid Tracking â˘âŻ Combines â˘âŻ computer vision â˘âŻ inertial gyroscope sensors â˘âŻ Both correct for each other â˘âŻ Inertial gyro â˘âŻ provides frame to frame prediction of camera orientation, fast sensing â˘âŻ drifts over time â˘âŻ Computer vision â˘âŻ Natural feature tracking, corrects for gyro drift â˘âŻ Slower, less accurate
- 71. OutdoorARTracking System You, Neumann,Azuma outdoor AR system (1999)
- 72. Robust OutdoorTracking â˘âŻHybrid Tracking â˘âŻ ComputerVision, GPS, inertial â˘âŻGoing Out â˘âŻ Reitmayr & Drummond (Univ. Cambridge) Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reaity. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
- 73. Handheld Display
- 74. REGISTRATION
- 76. The Registration Problem â˘âŻVirtual and Real content must stay properly aligned â˘âŻIf not: â˘âŻBreaks the illusion that the two coexist â˘âŻPrevents acceptance of many serious applications t = 0 seconds t = 1 second
- 77. Sources of Registration Errors â˘âŻStatic errors â˘âŻOptical distortions (in HMD) â˘âŻMechanical misalignments â˘âŻTracker errors â˘âŻIncorrect viewing parameters â˘âŻDynamic errors â˘âŻSystem delays (largest source of error) â˘âŻ1 ms delay = 1/3 mm registration error
- 78. Reducing Static Errors â˘âŻDistortion compensation â˘âŻFor lens or display distortions â˘âŻManual adjustments â˘âŻHave user manually alighn AR andVR content â˘âŻView-based or direct measurements â˘âŻHave user measure eye position â˘âŻCamera calibration (video AR) â˘âŻMeasuring camera properties
- 79. View Based Calibration (Azuma 94)
- 80. Dynamic errors â˘âŻ Total Delay = 50 + 2 + 33 + 17 = 102 ms â˘âŻ 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
- 81. Reducing dynamic errors (1) â˘âŻReduce system lag â˘âŻFaster components/system modules â˘âŻReduce apparent lag â˘âŻImage deflection â˘âŻImage warping
- 82. Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
- 83. ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280 x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
- 84. Reducing dynamic errors (2) â˘âŻMatch video + graphics input streams (video AR) â˘âŻDelay video of real world to match system lag â˘âŻUser doesnât notice â˘âŻPredictive Tracking â˘âŻInertial sensors helpful Azuma / Bishop 1994
- 85. PredictiveTracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now
- 87. Wrap-up â˘âŻTracking and Registration are key problems â˘âŻRegistration error â˘âŻMeasures against static error â˘âŻMeasures against dynamic error â˘âŻAR typically requires multiple tracking technologies â˘âŻComputer vision most popular â˘âŻResearch Areas: â˘âŻSLAM systems, Deformable models, Mobile outdoor tracking