SIGGRAPH Asia Slides from VR & AI Workshop by Mark Billinghurst

1. EMPATHIC COMPUTING
New Directions for AR and VR Research
Mark Billinghurst
mark.billinghurst@auckland.ac.nz
November 17th 2019

4. Evolution of Communication Tools

5. Modern Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion

6. Experience Capture
• Moving from desktop to headmounted cameras
• 360 camera, 3D scene scanning and capture

7. Increase Network Bandwidth
• Video conference on phones
• 5G increased bandwidth

8. Implicit Understanding
• Natural user interfaces (gaze, gesture, body motion)
• Recognizing user intent from actions and physiological cues

9. Natural
Collaboration
Implicit
Understanding
Experience
Capture

10. Natural
Collaboration
Implicit
Understanding
Experience
Capture
Empathic
Computing

11. Research Focus
Can we develop systems
that allow us to share what
we are seeing, hearing and
feeling with others?

12. Example Projects
• Remote collaboration in Wearable AR
• Sharing of non-verbal cues (gaze, pointing, face expression, emotion)
• Shared Empathic VR experiences
• Use VR to put a viewer inside the players view
• Measuring emotion
• Detecting emotion from heart rate, GSR, eye gaze, etc.

14. AR View Remote Expert View

15. Empathy Glasses
• Combine together eye-tracking, display, face tracking
• Implicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear

16. Remote Collaboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

18. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User

20. 3D Live Scene Capture
• Use cluster of RGBD sensors
• Fuse together 3D point cloud

21. Platform Features

22. AR View Remote Expert View

23. Live 3D Scene Capture

25. View Sharing Evolution
• Increased immersion
• Improved scene understanding
• Better collaboration
2D 360 3D

26. Sharing VR Experiences
• HTC Vive HMD
• Empathic glove
• Empatica E4

27. VR Environments
• Butterfly World: calm scene, collect butterflies
• Zombie Attack: scary scene, fighting zombies

28. Demo Video

29. AR and VR for Empathic Computing
• VR systems are ideal for trying experiences:
• Strong story telling medium
• Provide total immersion/3D experience
• Easy to change virtual body scale and representation
• AR systems are idea for live sharing:
• Allow overlay on real world view/can share viewpoints
• Support remote annotation/communication
• Enhance real world task

30. Technology Trends
• Advanced displays
• Wide FOV, high resolution
• Real time space capture
• 3D scanning, stitching, segmentation
• Natural gesture interaction
• Hand tracking, pose recognition
• Robust eye-tracking
• Gaze points, focus depth
• Emotion sensing/sharing
• Physiological sensing, emotion mapping

31. • Advanced displays
• Real time space capture
• Natural gesture interaction
• Robust eye-tracking
• Emotion sensing/sharing
Empathic
Tele-Existence

32. Empathic Tele-Existence
• Move from Observer to Participant
• Explicit to Implicit communication
• Experiential collaboration – doing together

33. Brain Synchronization

34. Pre-training (Finger Pointing) Session Start

35. Pre-training (Finger Pointing) Session End

36. Post-Training (Finger Pointing) Session End

37. VR Copy of Real World

38. Viewpoint Sharing

39. Early Results
• Face to Face in Real World
Number of Connections
Freq Before After P value Chi-
square
Alpha 173 191 > .05
Beta 169 210 0.02 4.7
Delta 133 160 > .05
Gamma 187 157 > .05
Theta 160 209 < .001 6.9

40. VR Results
Number of
Connections
Freq Before After P value Chi-square
score
Alpha 154 197 0.01 5.58
Beta 179 165 > 0.05
Delta 151 155 > 0.05
Gamma 199 173 > 0.05
Theta 175 152 > 0.05
Number of
Connections
Freq Before After P value Chi-square
score
Alpha 154 186 > 0.05
Beta 141 169 0.002 9.50
Delta 137 160 > 0.05
Gamma 202 145 0.001 9.92
Theta 202 145 0.001 9.92
VR – Face to Face VR – Shared Viewpoint

42. Conclusions
• Trend towards Empathic Computing
• Understanding, Experiencing, Sharing
• AR Enables Empathic Experiences
• Changes perspective
• Sharing space/experience
• Supports annotation/communication
• Many directions for future research

43. ARIVE
• Australasian Researchers in Interactive and Virtual Environments
• Collecting together all the best AR/VR researchers in Australia and NZ
• 8 institutions, 260 researchers, > 30 million USD in funding
• Looking for Industry Partners for research consortium

44. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz