Lecture 10 from the COMP 4010 course on AR/VR. This final lecture talks about future research directions in AR/VR. Taught on October 30th 2018 at the University of South Australia.
How to Troubleshoot Apps for the Modern Connected Worker
Ā
COMP 4010 Lecture10 AR/VR Research Directions
1. LECTURE 10: RESEARCH
DIRECTIONS IN AR AND VR
COMP 4010 ā Virtual Reality
Semester 5 ā 2018
Bruce Thomas, Mark Billinghurst
University of South Australia
October 30th 2018
2. Key Technologies for VR Systems
ā¢ Visual Display
ā¢ Stimulate visual sense
ā¢ Audio/Tactile Display
ā¢ Stimulate hearing/touch
ā¢ Tracking
ā¢ Changing viewpoint
ā¢ User input
ā¢ Input Devices
ā¢ Supporting user interaction
3. Many Directions for Research
ā¢ Research in each of the key technology areas for VR
ā¢ Display, input, tracking, graphics, etc.
ā¢ Research in the phenomena of VR
ā¢ Psychological experience of VR, measuring Presence, etc..
ā¢ Research in tools for VR
ā¢ VR authoring tools, automatic world creation, etc.
ā¢ Research in many application areas
ā¢ Collaborative/social, virtual characters, medical, education, etc.
4. Future Visions of VR: Ready Player One
ā¢ https://www.youtube.com/watch?v=LiK2fhOY0nE
5. Today vs. Tomorrow
VR in 2018 VR in 2045
Graphics High quality Photo-realistic
Display 110-150 degrees Total immersion
Interaction Handheld controller Full gesture/body/gaze
Navigation Limited movement Natural
Multiuser Few users Millions of users
6. Augmented Reality
ā¢ Defining Characteristics [Azuma 97]
ā¢ Combines Real andVirtual Images
ā¢ Both can be seen at the same time
ā¢ Interactive in real-time
ā¢ The virtual content can be interacted with
ā¢ Registered in 3D
ā¢ Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
7. Future Vision of AR: IronMan
ā¢ https://www.youtube.com/watch?v=Y1TEK2Wf_e8
8. Key Enabling Technologies
1. Combines Real andVirtual Images
Display Technology
2. Registered in 3D
Tracking Technologies
3. Interactive in real-time
InteractionTechnologies
Future research can be done in each of these areas
10. ā¢ Past
ā¢ Bulky Head mounted displays
ā¢ Current
ā¢ Handheld, lightweight head mounted
ā¢ Future
ā¢ Projected AR
ā¢ Wide FOV see through
ā¢ Retinal displays
ā¢ Contact lens
Evolution in Displays
11. Wide FOV See-Through (3+ years)
ā¢ Waveguide techniques
ā¢ Wider FOV
ā¢ Thin see through
ā¢ Socially acceptable
ā¢ Pinlight Displays
ā¢ LCD panel + point light sources
ā¢ 110 degree FOV
ā¢ UNC/Nvidia
Lumus DK40
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide
field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014
Emerging Technologies (p. 20). ACM.
17. Evolution of Tracking
ā¢ Past
ā¢ Location based, marker based,
ā¢ magnetic/mechanical
ā¢ Present
ā¢ Image based, hybrid tracking
ā¢ Future
ā¢ Ubiquitous
ā¢ Model based
ā¢ Environmental
18. Model Based Tracking (1-3 yrs)
ā¢ Track from known 3D model
ā¢ Use depth + colour information
ā¢ Match input to model template
ā¢ Use CAD model of targets
ā¢ Recent innovations
ā¢ Learn models online
ā¢ Tracking from cluttered scene
ā¢ Track from deformable objects
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013).
Model based training, detection and pose estimation of texture-less 3D objects in heavily
cluttered scenes. In Computer VisionāACCV 2012 (pp. 548-562). Springer Berlin Heidelberg.
20. Environmental Tracking (3+ yrs)
ā¢ Environment capture
ā¢ Use depth sensors to capture scene & track from model
ā¢ InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
ā¢ Real time scene capture on mobiles, dense or sparse capture
ā¢ Dynamic memory swapping allows large environment capture
ā¢ Cross platform, open source library available
24. Wide Area Outdoor Tracking (5+ yrs)
ā¢ Process
ā¢ Combine panoramaās into point cloud model (offline)
ā¢ Initialize camera tracking from point cloud
ā¢ Update pose by aligning camera image to point cloud
ā¢ Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
25. Wide Area Outdoor Tracking
https://www.youtube.com/watch?v=8ZNN0NeXV6s
26. Outdoor Localization using Maps
ā¢ Use 2D building footprints and approximate height
ā¢ Process
ā¢ Sensor input for initial position orientation
ā¢ Estimate camera orientation from straight line segments
ā¢ Estimate camera translation from faƧade segmentation
ā¢ Use pose estimate to initialise SLAM tracking
ā¢ Results ā 90% < 4m position error, < 3
o
angular error
Arth, C., Pirchheim, C., Ventura, J., Schmalstieg, D., & Lepetit, V. (2015). Instant outdoor
localization and SLAM initialization from 2.5 D maps. IEEE transactions on visualization and
computer graphics, 21(11), 1309-1318.
30. Natural Gesture (2-5 years)
ā¢ Freehand gesture input
ā¢ Depth sensors for gesture capture
ā¢ Move beyond simple pointing
ā¢ Rich two handed gestures
ā¢ Eg Microsoft Research Hand Tracker
ā¢ 3D hand tracking, 30 fps, single sensor
ā¢ Commercial Systems
ā¢ Meta, MS Hololens, Occulus, Intel, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).
32. Example: Eye Tracking Input
ā¢ Smaller/cheaper eye-tracking systems
ā¢ More HMDs with integrated eye-tracking
ā¢ Research questions
ā¢ How can eye gaze be used for interaction?
ā¢ What interaction metaphors are natural?
ā¢ What technology can be used for eye-tracking
ā¢ Etc..
33. Eye Gaze Interaction Methods
ā¢ Gaze for interaction
ā¢ Implicit vs. explicit input
ā¢ Exploring different gaze interaction
ā¢ Duo reticles ā use eye saccade input
ā¢ Radial pursuit ā use smooth pursuit motion
ā¢ Nod and roll ā use the vestibular ocular reflex
ā¢ Hardware
ā¢ HTC Vive + Pupil Labs integrated eye-tracking
ā¢ User study to compare between methods for 3DUI
Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017, March).
Exploring natural eye-gaze-based interaction for immersive virtual reality. In 3D User
Interfaces (3DUI), 2017 IEEE Symposium on (pp. 36-39). IEEE.
34. Duo-Reticles (DR)
Inertial Reticle (IR)
Real-time Reticle (RR) or Eye-gaze Reticle (original
name)
A-1
As RR and IR are aligned,
alignment time counts
down
A-2 A-3
Selection completed
38. Multimodal Input (5+ years)
ā¢ Combine gesture and speech input
ā¢ Gesture good for qualitative input
ā¢ Speech good for quantitative input
ā¢ Support combined commands
ā¢ āPut that thereā + pointing
ā¢ Eg HIT Lab NZ multimodal input
ā¢ 3D hand tracking, speech
ā¢ Multimodal fusion module
ā¢ Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.
39. HIT Lab NZ Multimodal Input
https://www.youtube.com/watch?v=DSsrzMxGwcA
40. Intelligent Interfaces (10+ years)
ā¢ Move to Implicit Input vs. Explicit
ā¢ Recognize user behaviour
ā¢ Provide adaptive feedback
ā¢ Support scaffolded learning
ā¢ Move beyond check-lists of actions
ā¢ Eg AR + Intelligent Tutoring
ā¢ Constraint based ITS + AR
ā¢ PC Assembly (Westerfield (2015)
ā¢ 30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.
42. Collaborative VR Systems
ā¢ Directions for research
ā¢ Scalability ā towards millions of users
ā¢ Graphics ā support for multiple different devices
ā¢ User representation ā realistic face/body input
ā¢ Support for communication cues ā messaging, recording, etc
ā¢ Goal: Collaboration in VR as good/better than FtF
Altspace VR Facebook Spaces
49. The MagicBook
ā¢ Using AR to transition along Milgramās continuum
ā¢ Moving seamlessly from Reality to AR to VR
ā¢ Support for Collaboration
ā¢ Face to Face, Shared AR/VR, Multi-scale
ā¢ Natural interaction
ā¢ Handheld AR and VR viewer
Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The MagicBook: a transitional
AR interface. Computers & Graphics, 25(5), 745-753.
52. Example: Visualizing Sensor Networks
ā¢ Rauhala et. al. 2007 (Linkoping)
ā¢ Network of Humidity Sensors
ā¢ ZigBee wireless communication
ā¢ Use Mobile AR toVisualize Humidity
Rauhala, M., Gunnarsson, A. S., & Henrysson, A. (2006). A novel interface to sensor
networks using handheld augmented reality. In Proceedings of the 8th conference on
Human-computer interaction with mobile devices and services(pp. 145-148). ACM.
59. Example:SocialAcceptance
ā¢ People donāt want to look silly
ā¢ Only 12% of 4,600 adults would be willing to wear AR glasses
ā¢ 20% of mobile AR browser users experience social issues
ā¢ Acceptance more due to Social than Technical issues
ā¢ Needs further study (ethnographic, field tests, longitudinal)
65. Research Needed in Many Areas
ā¢ Collaborative Experiences
ā¢ VR/AR teleconferencing
ā¢ Social Acceptance
ā¢ Overcome social problems with AR/VR
ā¢ Cloud Services
ā¢ Cloud based storage/processing
ā¢ Authoring Tools
ā¢ Easy content creation for non-experts for AR/VR
ā¢ Etc..