LECTURE 9: EXPLORING AR TECHNOLOGY
•
8 likes•4,388 views
COMP 4010 Lecture 9 providing an overview of Augmented Reality Technology. Taught by Mark Billinghurst on October 8th 2019 at the University of South Australia.
![Lecture 8: Recap
• Augmented Reality
• Defining Characteristics [Azuma 97]:
•Combines Real and Virtual Images
• Both can be seen at the same time
•Interactive in real-time
• The virtual content can be interacted with
•Registered in 3D
• Virtual objects appear fixed in space](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to LECTURE 9: EXPLORING AR TECHNOLOGY
Similar to LECTURE 9: EXPLORING AR TECHNOLOGY (20)
More from Mark Billinghurst
More from Mark Billinghurst (18)
Recently uploaded
Recently uploaded (20)
LECTURE 9: EXPLORING AR TECHNOLOGY
- 1. LECTURE 9: AR TECHNOLOGY COMP 4010 – Virtual Reality Semester 5 – 2019 Bruce Thomas, Mark Billinghurst, Gun Lee University of South Australia Oct 8th 2019
- 2. Lecture 8: Recap • Augmented Reality • Defining Characteristics [Azuma 97]: •Combines Real and Virtual Images • Both can be seen at the same time •Interactive in real-time • The virtual content can be interacted with •Registered in 3D • Virtual objects appear fixed in space
- 3. Augmented Reality 1977 – Star Wars
- 4. History Summary •1960’s – 80’s: Early Experimentation •1980’s – 90’s: Basic Research • Tracking, displays •1995 – 2005: Tools/Applications • Interaction, usability, theory •2005 - : Commercial Applications • Games, Medical, Industry
- 5. 2005 - Mobile Phone AR •Mobile Phones • camera • processor • display •AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction
- 6. 2008 - Browser Based AR • Flash + camera + 3D graphics • ARToolKit ported to Flash • High impact • High marketing value • Large potential install base • 1.6 Billion web users • Ease of development • Lots of developers, mature tools • Low cost of entry • Browser, web camera
- 7. • Weak AR • Imprecise tracking • No knowledge of environment • Limited interactivity • Handheld AR • Strong AR • Very accurate tracking • Seamless integration into real world • Natural interaction • Head mounted AR Strong vs. Weak AR
- 8. Augmented Reality Today •Key Technologies Available • Robust tracking (Computer Vision, GPS/sensors) • Display (Handheld, HMDs) • Input Devices (Kinect, etc) • Developer tools (Vuforia, ARToolKit) •Commercial Business Growing • Gaming, GPS/Mobile, Online Advertisement • >$3Billion USD in 2017
- 9. AR Business Today • Marketing • Web-based, mobile • Mobile AR • Geo-located information and service • Driving demand for high end phones • Gaming • Mobile, Physical input (Kinect, PS Move) • Upcoming areas • Manufacturing, Medical, Military
- 12. • Web based AR • Flash, HTML 5 based AR • Marketing, education • Outdoor Mobile AR • GPS, compass tracking • Viewing Points of Interest in real world • Eg: Junaio, Layar, Wikitude • Handheld AR • Vision based tracking • Marketing, gaming • Location Based Experiences • HMD, fixed screens • Museums, point of sale, advertising Typical AR Experiences
- 13. AR TECHNOLOGY
- 14. Augmented Reality 1977 – Star Wars
- 15. Technology Requirements • Combining Real and Virtual Images • Needs - Display technologies • Interactive in Real-Time • Needs - Input and interactive technologies • Registered in 3D • Needs - Viewpoint tracking technologies Display Processing Input Tracking
- 16. AR DISPLAYS
- 17. Display Technologies • Types (Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor
- 18. Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. DisplayTaxonomy
- 19. Great Books for Background • Spatial Augmented Reality, Displays; Fundamentals & Applications • Both by Oliver Bimber
- 21. Head Mounted Displays (HMD) • Display and Optics mounted on Head • May or may not fully occlude real world • Provide full-color images • Things to considerations • Wearability • Brightness • Power consumption • Resolution, Field of View • Cost
- 22. Types of Head Mounted Displays Occluded See-thru Multiplexed
- 23. Optical See-Through HMD (OSTHMD) Virtual images from monitors Real World Optical Combiners
- 25. Epson Moverio BT-35E ▪ Stereo see-through display ($1,200 AUD) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, ▪ USB-C, HDMI connectors ▪ 5mp camera, GPS, gyro, accelerometer
- 26. View Through Optical See-Through HMD Big data visualization on Microsoft HoloLens
- 27. Other Optical See-Through Displays • Microsoft HoloLens - $3,000 USD • Wearable computer • Waveguide displays • https://www.microsoft.com/hololens • Meta Meta2 - $1,500 USD • Wide field of view (90 degrees) • Tethered display • https://www.metavision.com/ • Mira Prism - $100 USD • Smart phone based • Wide field of view • https://www.mirareality.com/
- 28. Project North Star (Leap Motion) • http://blog.leapmotion.com/northstar/ • Open source OSTHMD • 100 degree FOV, 120 Hz, 2 x 1600x1440 pixel displays • Hand tracking using Leap Motion hardware
- 29. Demo: Project North Star • https://www.youtube.com/watch?v=yN0EGBo74j8
- 30. Strengths and Weaknesses • Pros • Simpler design (cheaper) • Direct view of real world • Full resolution, no time delay (for real world) • Safety, Lower distortion • No eye displacement • Cons • No occlusion of real world (image see-through) • Wide field of view can be difficult
- 31. Optics Design for OSTHMD
- 32. DisplayTechnology • Curved Mirror • off-axis projection • curved mirrors in front of eye • high distortion, small eye-box • Waveguide • use internal reflection • unobstructed view of world • large eye-box
- 33. Example: Sony Waveguide Display • https://www.youtube.com/watch?v=G2MtI7asLcA
- 34. See-throughThin Displays (Waveguide) • Waveguide techniques for thin see-through displays • Wider FOV, enable AR applications • Social acceptability Opinvent Ora Lumus DK40
- 35. Example: Sony Smart EyeGlasses • https://www.youtube.com/watch?v=kYPWaMsarss
- 36. Video See-Through HMD (VSTHMD) Video cameras Monitors Graphics Combiner Video
- 38. Vuzix Wrap 1200DXAR (Discontinued) ▪ Stereo video see-through display ($1500) ■Twin 852 x 480 LCD displays, 35 deg. FOV ■Stereo VGA cameras ■3 DOF head tracking ■https://www.vuzix.com/Products/LegacyProduct/4
- 39. View Through a Video See-Through HMD
- 40. Varjo XR-1 • https://varjo.com/products/xr-1/ • Video pass through • 2 x 12 megapixel cameras at 90 Hz, < 15 ms latency • 80 degree FOV, 1440 x 1600 resolution per eye
- 42. • Pros • True occlusion • AR pixels block video pixels • Digitized image of real world • Flexibility in composition, match time delays • Wide FOV is easier to support • Wide FOV camera • Cons • No direct view, latency Strengths and Weaknesses
- 43. VST vs. OST HMD VSTHMD OSTHMD
- 44. Eye Multiplexed Display • See content in periphery • Virtual image “inset” into the real world
- 48. Vuzix M-100 ▪ Monocular multiplexed display ($1000) ■852 x 480 LCD display, 15 deg. FOV ■5 MP camera, HD video ■GPS, gyro, accelerometer
- 50. Spatial Augmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”
- 51. Projector-based AR Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
- 52. Example of Projector-BasedAR Ramesh Raskar, MIT
- 53. Example of Projector-BasedAR Ramesh Raskar, MIT
- 54. Head Mounted Projector • NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com
- 55. HMD vs.HMPD Head Mounted Display Head Mounted Projected Display
- 56. Tilt5 - https://www.tiltfive.com/ • Stereo head worn projectors • Interactive wand • Rollible retro-reflective sheet • Designed for shared interaction
- 59. Video MonitorAR Video cameras Monitor Graphics Combiner Video Stereo glasses
- 60. Examples
- 61. Alternate Displays LCD Panel Laptop PDA
- 62. Handheld Displays •Mobile Phones • Camera • Display • Input
- 63. OtherTypes ofAR Display • Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch
- 64. AR INTERACTION
- 65. Interacting with AR Content • You can see AR images .. how can you interact with it?
- 66. Typical Interaction Methods • Handheld Display (phone, tablet) • Touch screen, device motion • Head mounted display • Handheld controller • Speech, gesture, touch • Physical manipulation • Spatial AR • Body motion • Physical manipulation
- 67. Gestural interfaces • 1. Micro-gestures • (unistroke, smartPad) • 2. Device-based gestures • (tilt based examples) • 3. Embodied interaction • (eye toy)
- 68. Mobile Device Motion • Interact by moving handheld display • Use IMU to detect shaking, rotation, etc • Treat device as controller
- 69. Example: AR Paparazzi Game • https://www.youtube.com/watch?v=MIGH5WGMnbs
- 70. Tangible Interaction • Use objects in the real world to interact with AR content • Attach AR content to real objects
- 72. HMD Gesture Interaction • Simple click gestures • Head point and air tap • Natural two handed gestures • Full hand tracking, direction manipulation
- 73. Hololens Gestures • Set of basic gestures • Air Tap = select, Bloom = open • Easy to remember and execute BloomAir Tap
- 75. Embodied Interaction • Use whole body to interact with AR • Popular for “Magic Mirror” Interface • Natural Interaction
- 78. AR RequiresTracking and Registration • Registration • Positioning virtual object wrt real world • Fixing virtual object on real object when view is fixed • Tracking • Continually locating the users viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)
- 79. AR TRACKING
- 80. Tracking Requirements • Augmented Reality Information Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
- 81. Tracking Technologies § Active • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location § Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking § Hybrid Tracking • Combined sensors (eg Vision + Inertial)
- 83. MechanicalTracker •Idea: mechanical arms with joint sensors •++: high accuracy, haptic feedback •-- : cumbersome, expensive Microscribe
- 84. MagneticTracker • Idea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
- 85. InertialTracker • Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote
- 86. UltrasonicTracker • Idea: time of Flight or phase-Coherence Sound Waves • ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600
- 87. Global Positioning System (GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc.
- 88. Mobile Sensors • Inertial compass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time
- 89. OPTICAL TRACKING
- 90. Why Optical Tracking for AR? • Many AR devices have cameras • Mobile phone/tablet, Video see-through display • Provides precise alignment between video and AR overlay • Using features in video to generate pixel perfect alignment • Real world has many visual features that can be tracked from • Computer Vision well established discipline • Over 40 years of research to draw on • Old non real time algorithms can be run in real time on todays devices
- 91. Common AR Optical Tracking Types • Marker Tracking • Tracking known artificial markers/images • e.g. ARToolKit square markers • Markerless Tracking • Tracking from known features in real world • e.g. Vuforia image tracking • Unprepared Tracking • Tracking in unknown environment • e.g. SLAM tracking
- 92. Marker Tracking • Available for more than 10 years • Several open source solutions • ARToolKit, ARTag, ATK+, etc • Fairly simple to implement • Standard computer vision methods • A rectangle provides 4 corner points • Enough for pose estimation!
- 94. Marker Based Tracking: ARToolKit http://www.artoolkit.org
- 95. Tracking challenges inARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
- 97. But - You can’t cover world with ARToolKit Markers!
- 98. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking • No more Markers! èMarkerless Tracking Mechanical Tracker
- 99. Natural Feature Tracking • Use Natural Cues of Real Elements • Edges • Surface Texture • Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces
- 100. TextureTracking
- 101. Tracking by Keypoint Detection • This is what most trackers do… • Targets are detected every frame • Popular because tracking and detection are solved simultaneously Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
- 102. What is a Keypoint? • It depends on the detector you use! • For high performance use the FAST corner detector • Apply FAST to all pixels of your image • Obtain a set of keypoints for your image • Describe the keypoints Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
- 103. FAST Corner Keypoint Detection
- 105. Descriptors • Describe the Keypoint features • Can use SIFT • Estimate the dominant keypoint orientation using gradients • Compensate for detected orientation • Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real-Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
- 106. Demo: Vuforia Texture Tracking https://www.youtube.com/watch?v=1Qf5Qew5zSU
- 107. Edge Based Tracking • Example: RAPiD [Drummond et al. 02] • Initialization, Control Points, Pose Prediction (Global Method)
- 108. Demo: Edge Based Tracking
- 109. Line Based Tracking • Visual Servoing [Comport et al. 2004]
- 110. Model BasedTracking • Tracking from 3D object shape • Example: OpenTL - www.opentl.org • General purpose library for model based visual tracking
- 111. Demo: OpenTL Model Tracking
- 112. Demo: OpenTL Face Tracking https://www.youtube.com/watch?v=WIoGdhkfNVE
- 113. Marker vs.Natural FeatureTracking • Marker tracking • Usually requires no database to be stored • Markers can be an eye-catcher • Tracking is less demanding • The environment must be instrumented • Markers usually work only when fully in view • Natural feature tracking • A database of keypoints must be stored/downloaded • Natural feature targets might catch the attention less • Natural feature targets are potentially everywhere • Natural feature targets work also if partially in view
- 114. Tracking from an Unknown Environment • What to do when you don’t know any features? • Very important problem in mobile robotics - Where am I? • SLAM • Simultaneously Localize And Map the environment • Goal: to recover both camera pose and map structure while initially knowing neither. • Mapping: • Building a map of the environment which the robot is in • Localisation: • Navigating this environment using the map while keeping track of the robot’s relative position and orientation
- 115. Visual SLAM • Early SLAM systems (1986 - ) • Computer visions and sensors (e.g. IMU, laser, etc.) • One of the most important algorithms in Robotics • Visual SLAM • Using cameras only, such as stereo view • MonoSLAM (single camera) developed in 2007 (Davidson)
- 117. How SLAMWorks • Three main steps 1. Tracking a set of points through successive camera frames 2. Using these tracks to triangulate their 3D position 3. Simultaneously use the estimated point locations to calculate the camera pose which could have observed them • By observing a sufficient number of points can solve for for both structure and motion (camera path and scene structure).
- 118. Evolution of SLAM Systems • MonoSLAM (Davidson, 2007) • Real time SLAM from single camera • PTAM (Klein, 2009) • First SLAM implementation on mobile phone • FAB-MAP (Cummins, 2008) • Probabilistic Localization and Mapping • DTAM (Newcombe, 2011) • 3D surface reconstruction from every pixel in image • KinectFusion (Izadi, 2011) • Realtime dense surface mapping and tracking using RGB-D
- 120. LSD-SLAM (Engel 2014) • A novel, direct monocular SLAM technique • Uses image intensities both for tracking and mapping. • The camera is tracked using direct image alignment, while • Geometry is estimated as semi-dense depth maps • Supports very large scale tracking • Runs in real time on CPU and smartphone
- 122. Direct Method vs.Feature Based • Direct uses all information in image, cf feature based approach that only use small patches around corners and edges
- 123. Applications of SLAM Systems • Many possible applications • Augmented Reality camera tracking • Mobile robot localisation • Real world navigation aid • 3D scene reconstruction • 3D Object reconstruction • Etc.. • Assumptions • Camera moves through an unchanging scene • So not suitable for person tracking, gesture recognition • Both involve non-rigidly deforming objects and a non-static map
- 124. ARKit – Visual Inertial Odometry • Uses both computer vision + inertial sensing • Tracking position twice • Computer Vision – feature tracking, 2D plane tracking • Inertial sensing – using the phone IMU • Output combined via Kalman filter • Determine which output is most accurate • Pass pose to ARKit SDK • Each system compliments the other • Computer vision – needs visual features • IMU - drifts over time, doesn’t need features
- 125. ARKit –Visual Inertial Odometry • Slow camera • Fast IMU • If camera drops out IMU takes over • Camera corrects IMU errors
- 127. REGISTRATION
- 128. Spatial Registration
- 129. The Registration Problem • Virtual and Real content must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications t = 0 seconds t = 1 second
- 130. Sources of Registration Errors •Static errors • Optical distortions (in HMD) • Mechanical misalignments • Tracker errors • Incorrect viewing parameters •Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error
- 131. Reducing Static Errors •Distortion compensation • For lens or display distortions •Manual adjustments • Have user manually alighn AR andVR content •View-based or direct measurements • Have user measure eye position •Camera calibration (video AR) • Measuring camera properties
- 132. View Based Calibration (Azuma 94)
- 133. Dynamic errors • Total Delay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
- 134. Reducing dynamic errors (1) •Reduce system lag •Faster components/system modules •Reduce apparent lag •Image deflection •Image warping
- 135. Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
- 136. ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280 x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
- 137. Reducing dynamic errors (2) • Match video + graphics input streams (video AR) • Delay video of real world to match system lag • User doesn’t notice • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994
- 138. PredictiveTracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now
- 140. Wrap-up • Tracking and Registration are key problems • Registration error • Measures against static error • Measures against dynamic error • AR typically requires multiple tracking technologies • Computer vision most popular • Research Areas: • SLAM systems, Deformable models, Mobile outdoor tracking