VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences
â˘
3 likesâ˘1,746 views
Keynote talk given by Mark Billinghurst at the VSMM 2016 conference on October 19th 2016.This talk was about how AR and VR can be used to create Empathic Computing experiences.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences
Similar to VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences (20)
More from Mark Billinghurst
More from Mark Billinghurst (20)
Recently uploaded
Recently uploaded (20)
VSMM 2016 Keynote: Using AR and VR to create Empathic Experiences
- 1. USING AR AND VR TO CREATE EMPATHIC EXPERIENCES Mark Billinghurst mark.billinghurst@unisa.edu.au October 19th 2016 VSMM 2016
- 2. â˘âŻ daf
- 3. Hiroshi Ishii â AWE 2014
- 4. 1977 â StarWars Augmented Reality
- 5. Augmented Reality â˘âŻDefining Characteristics â˘âŻCombines Real andVirtual Images â˘âŻBoth can be seen at the same time â˘âŻInteractive in real-time â˘âŻThe virtual content can be interacted with â˘âŻRegistered in 3D â˘âŻVirtual objects appear fixed in space
- 6. ARToolKit (1999) First open source AR SDK http://artoolkit.sourceforge.net/
- 7. Shared Space (1999) â˘âŻFace to Face interaction,Tangible AR metaphor â˘âŻEasy collaboration with strangers â˘âŻUsers acted same as if handling real objects
- 8. MagicBook (2001) â˘âŻFirst AR story book â˘âŻTransitional AR to VR interface
- 9. AR Tennis (2005) â˘âŻFirst collaborative AR game on mobile phone
- 10. Mobile AR Advertising (2007) â˘âŻFirst mobile AR ad campaign (Saatchi & Saatchi)
- 11. GeoBoids (2012) â˘âŻOutdoor AR game â˘âŻCapture creatures â˘âŻUse whistling to attract creatures â˘âŻGoogle Glass/Handheld AR
- 13. Pokemon GO..
- 14. Pokemon GO Effect â˘âŻ Fastest App to reach $500 million in Revenue â˘âŻ Only 63 days after launch, > $1 Billion in 6 months â˘âŻ Over 500 million downloads, > 25 million DAU â˘âŻ Nintendo stock price up by 50% (gain of $9 Billion USD)
- 15. AR Business Today â˘âŻAround $600 Million USD in 2014 (>$2B 2015) â˘âŻ70-80+% Games and Marketing
- 16. âDo you want to sell sugar water for the rest of your life or do you want to come with me and change the world?" Steve Jobs to John Sculley 1983
- 17. Markâs Midlife Crisis â˘âŻGet Married â˘âŻSabbatical at Google â˘âŻLooking for new opportunities â˘âŻResigned my job â˘âŻMoved to a new country â˘âŻNow creating a new research group
- 18. â˘âŻ df From Hiroshi Ishii
- 19. â˘âŻ asfas
- 20. Interaction Technology Natural Time Punch Card Keyboard Mouse Speech Gesture Emotion 1950 1960 1980 1990 2000 2010 Thought
- 22. Interaction Technology Natural Time Punch Card Keyboard Mouse Speech Gesture Emotion 1950 1960 1980 1990 2000 2010 Thought Implicit Explicit
- 23. Content Capture Realism Time Photo Film Live Video Panorama 360 Video 3D Space 1850 1900 1940 1990 2000 2010
- 24. Live Streaming/3D Space Capture Google Project TangoSamsung Project Beyond
- 25. Content Capture Realism Time Photo Film Live Video Panorama 360 Video 3D Space 1850 1900 1940 1990 2000 2010 2D Static Immersive Live Experience
- 26. Networking Log (b/s) Time 100 b/s 10 Kb/s 1 Mb/s 1980 1985 1990 1995 2000 2010 100 Mb/s 2005
- 28. Networking Log (b/s) Time 100 b/s 10 Kb/s 1 Mb/s 1980 1985 1990 1995 2000 2010 100 Mb/s 2005 Text Audio Natural Video
- 29. Holoportation â˘âŻ Augmented Reality + 3D capture + high bandwidth â˘âŻ http://research.microsoft.com/en-us/projects/holoportation/
- 33. EMPATHIC COMPUTING Systems that Create and Share Understanding
- 34. Empathy âSeeing with the Eyes of another, Listening with the Ears of another, and Feeling with the Heart of another..â Alfred Adler
- 36. Empathic Computing 1. Understanding: Systems that can understand your feelings and emotions 2. Experiencing: Systems that help you better experience the world of others 3. Sharing: Systems that help you better share the experience of others Sensors VR AR
- 37. 1. Understanding: Affective Computing â˘âŻRos Picard â MIT Media Lab â˘âŻSystems that recognize emotion
- 38. Appliances That Make You Happy â˘âŻJun Rekimoto â University of Tokyo/Sony CSL â˘âŻSmile detection + smart appliances
- 40. 2. Experiencing: Virtual Reality "Virtual reality offers a whole different medium to tell stories that really connect people and create an empathic connection." Nonny de la PeĂąa http://www.emblematicgroup.com/
- 41. Using VR for Empathy â˘âŻ USC Project Syria (2014) â˘âŻ Experience of Terrorism â˘âŻ Project Homeless (2015) â˘âŻ Experience of Homelessness
- 43. CHILDHOOD â˘âŻ Kenji Suzuki, University of Tsukuba â˘âŻ What does it feel like to be a child? â˘âŻ VR display + moved cameras + hand restrictors
- 45. 3. Sharing Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 46. â˘âŻ sdfs
- 47. â˘âŻ sdfgs
- 49. â˘âŻ axcvxca
- 50. Movies are like a machine that generates Empathy Roger Ebert
- 51. Technical Requirements â˘âŻBasic Requirements â˘âŻMake the technology transparent â˘âŻWearable, unobtrusive â˘âŻTechnology for transmitting â˘âŻSights, Sounds, Feelings of another â˘âŻAudio, video, physiological sensors
- 52. Wearable AR for Empathic Interfaces â˘âŻWearable AR can: â˘âŻBe unobtrusive â˘âŻCapture emotion â˘âŻShare sights and sounds â˘âŻProvide two way communication â˘âŻEnhance interaction in the real world
- 53. Changing Perspective â˘âŻCamNet (1992) â˘âŻBritish Telecom â˘âŻWearable Teleconferencing â˘âŻaudio, video â˘âŻRemote collaboration â˘âŻSends task space video â˘âŻSimilar CMU study (1996) â˘âŻcut performance time in half
- 54. WACL:Remote Expert Collaboration â˘âŻWearable Camera/Laser Pointer â˘âŻIndependent pointer control â˘âŻRemote panorama view
- 55. WACL:Remote Expert Collaboration â˘âŻRemote ExpertView â˘âŻPanorama viewing, annotation, image capture Kurata, T., Sakata, N., Kourogi, M., Kuzuoka, H., & Billinghurst, M. (2004, October). Remote collaboration using a shoulder-worn active camera/laser. In Wearable Computers, 2004. ISWC 2004. Eighth International Symposium on (Vol. 1, pp. 62-69).
- 56. Example:Google Glass â˘âŻCamera + Processing + Display + Connectivity â˘âŻEgo-Vision Collaboration (But with FixedView)
- 57. Social Panoramas (ISMAR 2014) â˘âŻCapture and share social spaces in real time â˘âŻSupports independent views into Panorama
- 58. Implementation â˘âŻGoogle Glass â˘âŻCapture live image panorama (compass + camera) â˘âŻRemote device (tablet) â˘âŻImmersive viewing, live annotation
- 61. Lessons Learned â˘âŻGood â˘âŻCommunication easy and natural â˘âŻUsers enjoy have view independence â˘âŻVery natural capturing panorama on Glass â˘âŻSharing panorama enhances the shared experience â˘âŻBad â˘âŻDifficult to support equal input â˘âŻNeed to provide awareness cues
- 62. JackIn â Live Immersive Video Streaming â˘âŻJun Rekimoto â University of Tokyo/Sony CSL
- 63. JackIn Hardware â˘âŻ Wide angle cameras â 360 degree video capture â˘âŻ Live video stitching
- 65. JospehTame â Tokyo Marathon â˘âŻLive streaming from Tokyo marathon â˘âŻhttp://josephta.me/en/tokyo-marathon/
- 66. â˘âŻ zxcz
- 67. Capturing Space:RealWorld Capture â˘âŻHands free AR â˘âŻPortable scene capture (color + depth) !âŻProjector/Kinect combo, Remote controlled pan/tilt â˘âŻRemote expert annotation interface
- 69. Example: CoSense (CHI 2015) â˘âŻReal time sharing - Emotion, video, and audio â˘âŻWearable (sender) â Send emotion and view â˘âŻDesktop (receiver) - See remote view and emotion Google Glasse-Health 2.0 board +
- 70. Implementation Data Capture Feature Detection Emotion Recognition Emotion Representation Empathic User Interface Hardware User Interface
- 71. Wearable Interface â˘âŻGoogle Glass + e-Health + Spydroid + SSI â˘âŻMeasure GSR, pulse oxygen, ECG, voice pitch â˘âŻShare video and audio remotely â˘âŻRepresentative emotions sent back to Glass user ! !
- 73. Lessons Learned â˘âŻGood â˘âŻSystem was wearable â˘âŻSender and receiver mirrored emotion â˘âŻMinimal cues provided best experience â˘âŻBad â˘âŻSystem delays â˘âŻNeed for good stimulus â˘âŻDifficult to represent emotion
- 74. Gaze and Video Conferencing â˘âŻGaze tracking â˘âŻ Implicit communication cue â˘âŻ Shows intent â˘âŻTask space collaboration â˘âŻ HMD + camera + gaze tracker â˘âŻExpected Results â˘âŻ Gaze cues reduce need for communication â˘âŻ Allow remote collaborator to respond faster
- 75. Equipment â˘âŻ Custom eye-tracker â˘âŻ Head mounted camera â˘âŻ Head mounted display
- 76. Experiment Set Up â˘âŻ Lego assembly â˘âŻ Two assembly areas â˘âŻ Remote expert
- 77. Experiment Design â˘âŻ4 conditions varying eye-tracking/pointer support â˘âŻ13 pairs subjects â˘âŻMeasures â˘âŻ Performance time â˘âŻ Likert scale results, Ranking results, User preference
- 78. Task Performance â˘âŻ Performance Time (seconds)
- 79. Ranking â˘âŻ Median Ranking Values
- 80. Key Results â˘âŻ Both the pointer and eye tracking visual cues helped participants to perform significantly faster â˘âŻThe pointer cue significantly improved perceived quality of collaboration and co-presence â˘âŻ Eye-tracking improved the collaboration quality, and sense of being focused for the local users, and enjoyment for the remote users â˘âŻThe Both condition was ranked as the best in user experience, while the None condition was worst.
- 81. Empathy Glasses (CHI 2016) â˘âŻ Combine together eye-tracking, display, face expression â˘âŻ Impicit cues â eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
- 82. AffectiveWear â Emotion Glasses â˘âŻ Photo sensors to recognize expression â˘âŻ User calibration â˘âŻ Machine learning â˘âŻ Recognizing 8 face expressions
- 83. Integrated System â˘âŻLocal User â˘âŻVideo camera â˘âŻEye-tracking â˘âŻFace expression â˘âŻRemote Helper â˘âŻRemote pointing
- 84. System Diagram â˘âŻ Two monitors on Remote User side â˘âŻ Scene + Emotion Display
- 86. Ranking Results "I ranked the (A) condition best, because I could easily point to communicate, and when I needed it I could check the facial expression to make sure I was being understood.â Q2: Communication Q3: Understanding partner HMD â Local User Computer â Remote User
- 87. Lessons Learned â˘âŻPointing really helps in remote collaboration â˘âŻ Makes remote user feel more connected â˘âŻGaze looks promising â˘âŻ Shows context of what person talking about â˘âŻ Establish shared understanding/awareness â˘âŻFace expression â˘âŻ Used as an implicit cue to show comprehension â˘âŻLimitations â˘âŻ Limited implicit cues â˘âŻ Task was a poor emotional trigger â˘âŻ AffectiveWear needs improvement
- 88. Empathic VR Environments â˘âŻ Player and Viewer â˘âŻ Viewer slaved to player â˘âŻ Share emotional signals â˘âŻ Heart rate, GSR â˘âŻ Remote affect measuring
- 89. Emotion Classification â˘âŻClassify emotions from physiological data â˘âŻ 8 emotional states â˘âŻCollaboration with Sensaura â˘âŻ http://www.sensauratech.com
- 90. Heart Rate for Different VR Apps. â˘âŻ Games: theBlu, BrookHaven, Night CafĂŠ quiet scene â scary scene â peaceful scene â˘âŻ Devices: HTV VIVE, Empatica E4 Heart Rate The Blu BrookHaven Night CafĂŠ Mean 84.89 BPM 98.23 BPM 73.01 BPM
- 91. Sharing VR Experiences â˘âŻ Player controls viewer position (not view) â˘âŻ Measure and share physiological cues Viewer Player
- 92. Demo Video
- 93. AR and VR for Empathic Computing â˘âŻVR systems are ideal for trying experiences: â˘âŻ Strong story telling medium â˘âŻ Provide total immersion/3D experience â˘âŻ Easy to change virtual body scale and representation â˘âŻAR systems are idea for live sharing: â˘âŻ Allow overlay on real world view/can share viewpoints â˘âŻ Support remote annotation/communication â˘âŻ Enhance real world task
- 94. Looking to the Future Whatâs Next?
- 95. Scaling Up â˘âŻSeeing actions of millions of users in the world â˘âŻAugmentation on city/country level
- 96. AR + Smart Sensors + Social Networks â˘âŻTrack population at city scale (mobile networks) â˘âŻMatch population data to external sensor data â˘âŻMine data for applications
- 97. Example: MIT SENSEable City Lab http://senseable.mit.edu/wikicity/rome/
- 98. Example: CSIRO WeFeel Tool â˘âŻEmotionally mining global Twitter feeds â˘âŻhttp://wefeel.csiro.au
- 99. European Cup â July 10th GOAL!
- 100. Research Challenges â˘âŻHow to capture emotion? â˘âŻHow to measure empathy? â˘âŻInterface/interaction models? â˘âŻHow to communicate emotion? â˘âŻHow to create strong empathic bonds? â˘âŻHow to scaling up to city/country scale?
- 101. Potential Applications â˘âŻEducation â˘âŻSports training â˘âŻRich life logging â˘âŻRemote meeting support â˘âŻPsychological treatments â˘âŻVirtual Travel/Entertainment â˘âŻSurrogate Adventure Tourism â˘âŻFirst responders (stress, team cohesion)
- 102. CONCLUSION
- 103. Harvard Grant Study â˘âŻ$20 million, 75 years study â˘âŻ268 Harvard graduates â˘âŻ456 disadvantaged people â˘âŻLed by George Valliant â˘âŻWhat makes us happy? â˘âŻwarmth of relationships throughout life have the greatest positive impact on "life satisfaction".
- 104. âThe seventy-five years and twenty million dollars expended on the Grant Study points to a straightforward five-word conclusion: Happiness is love. Full stop.â George Valliant
- 105. Conclusions â˘âŻEmpathic Computing â˘âŻSharing what you see, hear and feel â˘âŻAR/VR Enables Empathic Experiences â˘âŻRemoving technology â˘âŻChanging perspective â˘âŻSharing space/experience â˘âŻMany directions for future research