Human-Centered Design for Immersive Interaction - Jason Jerald
•Download as PPTX, PDF•
1 like•365 views
Jason Jerald's talk from VR With The Best on Human-Centered Design for Immersive Interactions Jason Jerald, PhD
Recommended
Recommended
More Related Content
Similar to Human-Centered Design for Immersive Interaction - Jason Jerald
Similar to Human-Centered Design for Immersive Interaction - Jason Jerald (6)
More from WithTheBest
More from WithTheBest (20)
Recently uploaded
Recently uploaded (20)
Human-Centered Design for Immersive Interaction - Jason Jerald
- 1. Human-Centered Design for Immersive Interactions Jason Jerald, PhD
- 2. 2@TheVRBookJason Jerald, Phd 60+ VR Projects, 30+ Organizations2
- 3. 3@TheVRBookJason Jerald, Phd TheVRBook.net Win a free book! @TheVRBook #VRWTB
- 4. 4@TheVRBookJason Jerald, Phd VR is not New! • 1830s to 1990s Images Courtesy of The VR Book (with permissions from original sources)
- 5. 5@TheVRBookJason Jerald, Phd The Most Exciting VR Tech
- 6. 6@TheVRBookJason Jerald, Phd The Most Exciting VR Tech • The Human Brain! – ~100B Neurons – Thousands synapses/neuron – ~1B Synapses/mm^3 – 150km of nerve fibers – Six degrees of separation
- 7. 7@TheVRBookJason Jerald, Phd Perception • The Human Senses – Sight – Hearing – Balance and physical motion – Touch – Proprioception – Smell and Taste – Multimodal – Sensory Substitutions
- 8. 8@TheVRBookJason Jerald, Phd What We Perceive is not Reality
- 9. 9@TheVRBookJason Jerald, Phd The Iterative Perceptual Process • With VR, we hijack the human perception-action system – Human Output = VR Input/Tracking – VR Output = Human Input Image Courtesy of The VR Book (adapted from Goldstein [2007])
- 10. 10@TheVRBookJason Jerald, Phd • Latency • Calibration • Tracking accuracy & precision • Field of View • Refresh rate • Judder & Flicker • Display response & persistence • Vergence/Accommodation • Stereoscopic cues • Real world peripheral vision • Fit/Weight/center of mass • Motion platforms • Hygiene • Temperature • Dirty screens Factors of Adverse Health Effects System Individual • History of motion sickness • VR experience • Health • Thinking about Sickness • Belief • Gender • Age • Mental model/expectations • Interpupilary distance • Not knowing • Sense of balance • Flicker fusion frequency threshold • Real-world task experience • Migraine history Application • Frame rate • Locus of control • Visual acceleration • Physical head motion • Duration • Vection • Binocular-occlusion conflict • Virtual rotation • Gorilla arm • Standing vs sitting • Rest frames • Binocular disparity • VR entrance & exit • Luminance • Repetitive strain
- 11. 11@TheVRBookJason Jerald, Phd Motion Sickness—The Basics • The biggest problem is motion sickness • Varies greatly from person to person • As users: – Stop at the first sign of sickness – Be well-hydrated and not hungry, not hung over – Watch out for colds and sinus congestion • As creators: • Maintain the frame rate of the display hardware 100% of the time • Make interactions comfortable – Initial signs of discomfort may very well lead to sickness • Minimize accelerations (e.g., roller coaster motions) and virtual rotations • Remind your users not to “tough it out” • Wipe down equipment periodically
- 12. 12@TheVRBookJason Jerald, Phd VR Hardware Beyond Displays12
- 13. 13@TheVRBookJason Jerald, Phd An Example VR Device VR Apocalypse by & Virtuix
- 14. 14@TheVRBookJason Jerald, Phd Input Device Classes
- 15. 15@TheVRBookJason Jerald, Phd The Most Important Input Device?
- 16. 16@TheVRBookJason Jerald, Phd The Most Important Input Device! The Human Hands! – Large proportion of sensory and motor cortex devoted to the hands – Hand-tracking technology is simply the mediator that brings the hands into VR 16
- 17. 17@TheVRBookJason Jerald, Phd Interaction Patterns • Hand Selection • Pointing • Image- Plane • Volume Based Selection • Walking • Steering • 3D Multi- Touch • Automated Viewpoint Control • Direct Hand • Proxy • 3D Tool Manipulation • Widgets & Panels • Non Spatial Indirect Control • Pointing Hand • World-in- Miniature • MultiModal Compound
- 18. 18@TheVRBookJason Jerald, Phd The First Hand Technique Trainexus by
- 19. 19@TheVRBookJason Jerald, Phd Hand Selection/Manipulation Patterns • Hands with Arms Cure Fred by
- 20. 20@TheVRBookJason Jerald, Phd Audio Feedback Cure Fred by
- 21. 21@TheVRBookJason Jerald, Phd The Pointing Pattern Zombie Apocalypse by
- 22. 22@TheVRBookJason Jerald, Phd Interface in the Hands Trainexus by
- 23. 23@TheVRBookJason Jerald, Phd Realistic vs. Nonrealistic Hands • Hands without arms Trainexus by Zombie Apocalypse by
- 24. 24@TheVRBookJason Jerald, Phd The Image-Plane Pattern • The Head-Crusher Technique From Pierce et al. [1997]
- 25. 25@TheVRBookJason Jerald, Phd Widgets & Panels Pattern • 2D Desktop Integration Technique From Taylor et al. [2010]
- 26. 26@TheVRBookJason Jerald, Phd Hand-held panels • Buttons • Sliders • Dials • Color cubes Images Courtesy of Sixense & Digital ArtForms
- 27. 27@TheVRBookJason Jerald, Phd 3D Multitouch Pattern • Appropriate for abstract non-realistic interactions • Content/Data independent – Emergency preparedness – Medical Visualization – MakeVR • Solves Gorilla Arm • Reduces Sim Sickness
- 28. 28@TheVRBookJason Jerald, Phd 3D Multitouch Zombie Apocalypse by
- 29. 29@TheVRBookJason Jerald, Phd 3D Multitouch Medical Visualization iMedic by
- 30. 30@TheVRBookJason Jerald, Phd 3D Multitouch 30 MakeVR by &
- 31. 31@TheVRBookJason Jerald, Phd Computer-Aided Design MakeVR by &
- 32. 32@TheVRBookJason Jerald, Phd What is Common to All VR Dev? Iteration!
- 33. 33@TheVRBookJason Jerald, Phd Expert Evaluations 33
- 34. 34@TheVRBookJason Jerald, Phd Summary • Widespread access to tracked hand-held controllers will take VR to the next level – But only for applications that are designed well • There are no universal answers – Selecting devices and interaction patterns are project dependent – Understand tradeoffs and where what is appropriate • Iterative. Iterate. Iterate. • Have fun!
- 35. 35@TheVRBookJason Jerald, Phd @TheVRBook 15% Discount Code TheBest16 TheVRBook.net
- 37. 37@TheVRBookJason Jerald, Phd References • Jerald, J. [2015] The VR Book: Human-Centered Design for Virtual Reality. Morgan & Claypool Publishers and ACM Books. • Gabbard J. (2014). Usability Engineering of Virtual Environments. In K. S. Hale and K.M. Stanney (Eds), Handbook of Virtual Environments (2nd ed., pp. 721-747). CRC Press. • Norman D. [2013] The Design of Everyday Things, Expanded and Revised Edition. Basic Books. • Pierce, J., Forsberg, A., Conway, M., Hong, S., Zeleznik, R., and Mine, M. R. (1997). Image Plane Interaction Techniques in 3D Immersive Environments. In ACM Symposium on Interactive 3D Graphics (pp. 39–44). ACM Press. • Taylor, R., Jerald, J., VanderKnyff, C., Wendt, J., Borland, D., Marshburn, D., Sherman, W., and Whitton, M. (2010). Lessons about Virtual Environment Software Systems from 20 Years of VE Building. Presence: Teleoperators and Virtual Environments, 19(2), 162–178.
- 38. 38@TheVRBookJason Jerald, Phd Additional Slides
- 39. 39@TheVRBookJason Jerald, Phd A New Artistic Medium • The screen disappears • The best VR applications are designed from the beginning for VR • Today the technology is being sold • Tomorrow the story and emotion will be sold
- 40. 40@TheVRBookJason Jerald, Phd Adverse Health Effects—Challenges • Motion sickness • Eye strain – Flicker – Accommodation-vergence conflict – Occlusion-binocular Conflict • Aftereffects • Seizures • Physical Fatigue • Headset Fit • Injury • Hygiene
- 41. 41@TheVRBookJason Jerald, Phd Reference Frames • Head • Eyes • Torso • Virtual World • Real World • Hands Image Courtesy of NextGen Interactions
- 42. 42@TheVRBookJason Jerald, Phd Example Pattern • Walking Pattern – a form of viewpoint control – Specific walking techniques • real-world walking • redirected walking • walking in place • treadmill walking
- 43. 43@TheVRBookJason Jerald, Phd World-in Miniature Pattern iMedic by
- 44. 44@TheVRBookJason Jerald, Phd 3D Multitouch Medical Visualization iMedic by
- 45. 45@TheVRBookJason Jerald, Phd 3D Tools Pattern • The Jigs Technique MakeVR by &
Editor's Notes
- What makes me qualified to give a tutorial on VR interactions? 20 years of VR experience on over 60 projects with over 30 organizations ranging from academics to startups to Fortune 500 Companies. What I’ve learned is that every VR project is different. There are few absolute truths when it comes to VR. But theory, processes, and guidelines can be useful.
- If you want to learn more about human-centered design for VR then check out my book at TheVRBook.net. 20 years of research summarized into 600 pages Not necessarily a leisurely Sunday afternoon read, but dramatically simplified. No equations, no code—the focus is on the design from a human-centered approach. Topics include human perception, understanding and reducing sickness, environmental design, interaction patterns, and iteration concepts. Over 600 guidelines for designing VR applications, hundreds definitions, and over 300 references pointing to more detail. Go to TheVRBook.net to learn more about applying VR design concepts to your application If you are teaching a VR course, then talk with me about getting a free evaluation copy.
- I’d like to point out VR is not new! Depending how you define it you could go back to prehistoric times of cavemen drawing on walls. Here are some systems ranging from the 1830s to the 1990s. For example, Wheatstone stereoscope Brewster stereoscope from the 1860s Flight simulators were really what got VR going in the 1900s Sensorama in the 1960s, included a stereoscopic display, wide field of view, color, stereo sound, wind, smell, a motion/vibration platform Tom Furness and Ivan Sutherland creating head-tracked computer rendered head-mounted displays in the 1960s
- There has been quite a bit of VR over the years. However, for me there is one tech that is most exciting. Many people are only vaguely aware of this. Absolutely essential, without this VR is not possible. Has been available in various forms for thousands of years.
- The Human Brain! A massive number of neurons—the basic unit of the nervous system. However, the connection between those neurons is what is most impressive. On average, there are thousands of connections (synapses) from a single neuron connecting to other neurons. There is a billion of these synapses for a cubic millimeter of brain matter. 150km of nerve fibers is enough to circle the earth multiple times! A neuron can reach any other neuron on average by traveling through only six other neurons. This results in a massive parallel machine. Today’s computers cannot compare, some do but that is comparing apples to oranges. The result of this massive parallel machine called a brain integrated with the body is a perception and action entity that enables us to interact with the world
- Our interactions with the real and virtual worlds start with perception. VR is much more than just graphics! Especially when it comes to interacting. Most VR creators focus primarily on visuals. Some do audio, but mostly as an afterthought. Lack of physical touch is one of the greatest challenges of VR. Proprioception is extremely important for VR—the sense of where your body parts are located (e.g., close your eyes and touch your nose). Normally don’t consciously think about proprioception but when its missing it’s a problem. Smell and taste are important for some applications but is difficult to do well with VR (although research and proof of concepts have shown this to be doable). Bringing the senses together in a coherent manner is important to induce a sense of presence in users and for interactions to be intuitive. Spatial and temporal compliance are most important (e.g., proprioception matches visuals). Multimodal input/output can be challenging but effective.
- This perception of the real (or virtual) world is not reality or “the truth” Our perceptions are an interpretation of reality. We do not perceive objects directly, for example, objects aren’t poking into our retinas, we instead perceive the photons bouncing off those objects into our eyes. For VR, we hijack the human senses and replace real world cues with artificially created cues that the mind interprets as a form of reality. Our sense of reality can be quite distorted, different for different people and under different circumstances A song can be quite emotional and have some significant meaning for one person while not for another person.
- Here is a version of Goldstein’s iterative perception-action process This is a model of how distal stimuli out in the world reaches our receptors and how we process it We can replace the real world distal stimuli with computer generated stimuli and sense the users actions in order to update that stimuli Human output becomes VR Input VR output becomes human input
- Many VR “experts” argue what causes VR sickness as if there were one single factor. The ones listed here are the ones we know about, many more have been hypothesized. The leading hardware manufacturers are solving many of the system challenges. E.g., latency is not much of a problem any more for the best HMD systems. But we should still be aware of them to select the best hardware and to recognize problems when they occur. Unfortunately we don’t have much control over individual factors. But we should understand our target audience. We can also influence their mental model. The factors that we have the most control over are application factors. Especially important for navigation schemes. But bad non-navigation interfaces can cause problems as well. For example put the interface in a comfortable position in the torso reference frame near the waste. We will talk about some of these throughout the talk.
- There is much more to VR beyond displays Hardware is varied and depends upon project goals VR hand input is varied and depends upon project goals. E.g., location based entertainment can be very different than games in the home
- Here is an example of some work we did with Virtuix as shown on ABC’s Shark Tank that uses an omni-directional treadmill to navigate through the virtual world. It is not clear if and how much this treadmill technique reduces motion sickness A great area of research anyone in this audience could study. More interaction patterns and techniques will be presented later in the course.
- No single input device class has all qualities or is the ideal solution for all applications. Most appropriate device depends on goals and the application/experience. Hybrid and multimodal interaction can be ideal (if designed and integrated well), but more expensive and difficult to implement. Currently tracked hand-held controllers are the most appropriate for most, but not all, VR experiences
- I’ve seen some crazy VR input devices. However, there is one type of input that is especially important for VR Clue: Shown on previous slides. Multiple times Turns out these input device is important for the real world as well.
- The human hands! A relatively large part of the human brain is dedicated to input/output of the hands Hardware simply enables us to bring the hands into and interact with the experience.
- Now lets move on to some common interaction themes that occur across VR applications. I looked at ~100 interaction techniques and categorized them into interaction patterns. Very different from software design patterns. These patterns are from the user’s point of view. I organize the patterns into 5 overarching patterns. These 16 patterns can be further broken down into more specific interaction techniques.
- - The first thing people do when hand tracking becomes available is direct selection and manipulation. Here is what I built back in 2013 just after the Oculus Kickstarter DK1 shipment Quite a bit can be done just by tracking three points—the head and the two hands. Here, I’m threatening Paul Mlyniec, the President of Digital ArtForms. Since his hands are tracked he doesn’t have to think of where the surrender button is. He just naturally raises his hands to non-verbally say “don’t shoot!” Very similar to how we interact in the real world Other parts of the body can be moved with inverse kinematics and animations. For example the feet move with a walking animation when pressing forward with joystick on a hand-held controller. Inverse kinematics can be extremely compelling if done well and where appropriate.
- Example of some work we did with Digital ArtForms, Sixense, and the Wake Forest School of Medicine to teach kids about how their brains work. Learn by doing Simply reach out to grab the brain parts-intersect the hand with the object and then push a button Can do so much more than you can with a real-world puzzle. E.g., integrating with digital media, smarter interactions, special effects. Play movie
- Pointing is normally considered to be non realistic (unless simulating a laser pointer) Enables selection at a distance Here we have an exocentric view of the world from an egocentric perspective. Here, pointing enables the user to shoot electricity from his hands to protect a friend in a smaller world (Work with Digital ArtForms)
- Why do VR developers think user interface elements should go in the head or world reference frame? A simple solution: Put UI signifers/labels in the hands. Simply look at your hands when performing an action until you get good at the task—similar to real reality. The signifiers are simple transparent textures with labels attached to a Razer Hydra. A 3d model matching what the user physically feels would be better. No finger tracking, but close enough for users to intuitively know what button does what by touch. Enable users to turn on/off the visuals. Important for both novice and expert users.
- VR experts argue if arms or realistic looking hands are better. Like many other great VR debates, the answer is it depends! Arms are most appropriate when: Realism is important It can be assumed the user will always be facing in one direction or the torso is tracked. Arms are not appropriate when: Reaching further than the physical length of your arms is important For example: Reaching to pick up items off the floor from a seated position You want to reach out further into the distance than you could if you were physically constrained with how far your arms can reach. The Go-Go Techniques is a hand selection technique that has 1to1 mapping within personal space at 2/3 of your physical arm length. Beyond 2/3 of your arm length, your arm reach expands exponentially enabling selection at a distance. Hand selection/manipulation can be quite compelling even when the hands don’t look like hands (i.e., 3d cursors). With appropriate spatial and temporal compliance, 3D cursors feel like they are your hands—even though they don’t look at all like your hands.
- The Image-plane pattern is rarely used pattern but can work quite well for selecting objects at a distance. Shown here is the Head-Crusher Technique. Think of the world as a 2D plane in front of the user. Hold the hand up and place the two fingers of the dominant hand around the object of interest Push a button on the non-dominant hand or verbally say “select”. Requires closing one eye and can result in gorilla arm if used often.
- Widgets and panels often take desktop applications or metaphors and bring them into the virtual world Often not ideal, but easy to do. Hand pointing via ray selection is most common and most usable. Here I’m bringing in arbitrary desktop applications into a CAVE VR system. If you want to do a quick mathematical calculation, you don’t want to have to exit the virtual world to bring up a calculator app and its not worth writing an immersive 3D calculator app. Instead of exiting the virtual world, bring the existing application into VR. Many other forms of immersive widgets and panels. For example hand-held panels, panels above the head, finger menus, marking menus, etc.
- Putting a virtual panel on the hand works surprisingly well Provides double dexterity where the nondominant hand provides a reference frame to work in. Interface always available Turn on/off with the click of a physical button Examples here include traditional GUI elements (buttons, sliders, drop down menus) as well as more interesting GUI elements (dials, marking menus, color cube)
- Similar to 2D multitouch Think of manipulating the world as an object—grab with both hands to translate, orient, and scale the world Thinking about manipulating the world as an object results in the user thinking of the world moving instead of moving through the world, resulting in reduced motion sickness. Appropriate for abstract non-realistic interactions Data independent. We’ve used this across a variety of applications including volumetric data—no polygons required because no intersection computations are required. Solves gorilla arm. A focus study found users could go 4 hours with no report of arm fatigue. To translate, simply grab the world with one or both hands To scale/zoom, “pinch” with two hands instead of two fingers To rotate, grab the world with both hands and orbit about the midpoint between the hands as if it were a globe Can do translation, scale, and rotation simultaneously with a single gesture interaction is direct & immediate—starts as soon as you push the button—no need to wait for the system to recognize the motion Results in the feeling that you are are crawling through and stretching space
- An example of 3D Multitouch shown at SIGGRAPH 2011. Non-HMD here but the 3D multitouch actually works better with HMDs as that is what it was originally designed for. Play movie
- The same interface is used with MakeVR (new company should be formed by VRDC 2016). An immersive CAD modeling program. 3D content creation and world building intuitive and accessible by anyone. Currently works with Sixense STEM, Razer Hydras, and SpaceGrips. Support coming for Oculus Touch controllers and HTC Vive controllers
- For those not familiar with MakeVR, here is an example video of what is possible.
- What is common to all VR Development? Iteration is absolutely essential! More so than for creating any other product If you get it wrong here you can make users sick or injure them Will get it wrong on first attempts What you think would work great often does not
- A big part of what we are working at NextGen Interactions is to creating processes for providing better feedback to VR creators for their specific projects. That starts with task analysis to really understand what is trying to be accomplished That leads to expert guidelines-based evaluations that are the first steps to making sure the most important items are being done well. That leads to formative usability analysis which is using techniques such as speak-out-loud protocols where the user explains out loud what he is doing and thinking, to observational analysis of breaks in presence. Then those that want to get more formalized results can conduct formal scientific studies to compare different techniques or conditions.
- Widespread access to tracked hand-held controllers not only provides a large user base but provides developers with quality input devices that will lead to innovative interfaces that nobody has yet considered. Just because anything is possible doesn’t mean everything will work. Viewpoint control especially requires caution due to risk of sickness and injury. Iteration is absolutely essential for VR. Most importantly have fun! Otherwise what is the point of reality, whether virtual reality or real reality?
- If you want to learn more about human-centered design for VR then check out my book at TheVRBook.net. 20 years of research summarized into 600 pages Not necessarily a leisurely Sunday afternoon read, but dramatically simplified. No equations, no code—the focus is on the design from a human-centered approach. Topics include human perception, understanding and reducing sickness, environmental design, interaction patterns, and iteration concepts. Over 600 guidelines for designing VR applications, hundreds definitions, and over 300 references pointing to more detail. Go to TheVRBook.net to learn more about applying VR design concepts to your application If you are teaching a VR course, then talk with me about getting a free evaluation copy.
- There are multiple health challenges of VR. For now, we will focus on the worst offender—motion sickness.
- Thinking about and understanding reference frames is important for human-centered interaction design. Many of you computer graphics experts are intimately familiar with reference frames. Screen space Object space World space Reference frames define: The axii that determine translation along x, y, & z and also rotation along yaw, pitch, and roll. How objects move relative to other objects (e.g., attached to the head, hand, or world). The image here looks complex in 2D but becomes intuitively obvious when immersed. Head reference frame—similar to screen coordinates in 2D development. Blue Arrows in torso reference frame represent the forward direction. It is important for usability and comfort to place your interfaces in the appropriate reference frames. We will discuss these reference frames and how they are best used in more detail. In some cases reference frames are equivalent to each other. E.g., for a fully walkable system (e.g., HTC Vive) the virtual world and real world ref frames can be made to be equivalent
- Viewpoint control patterns are especially important due to challenges of motion sickness. An example of a viewpoint control pattern is The walking pattern. There are various ways to “walk” within VR. Example walking techniques are: Real World walking, such as with the HTC Vive. Redirected walking, such as with The Void. Walking in Place, such as with Stompz. Treadmill walking, such as with the Virtuix Omni.
- A WIM is basically a real-time map of the world a user is in. Can hold in the hand or have it attached in the world or to your body (torso reference frame) Can reach into the map and move things that then moves the corresponding object in the larger world
- Take on role of robot. Crawl through a body with a friend Green user has carved out the midsection of a body to take a closer look inside. Blue user follows him Medical teleconsultation—2 doctors at remote locations could examine a dataset and point things out that would be difficult to do in 2D setting.
- Precision is normally extremely difficult in VR. No physical constraints as there is with a mouse on a desk. So instead add artificial constraints—what we call jigs. Shown here as used with MakeVR. Think of carpenter tools to provide the user more precise interaction. E.g., grids can be attached to polygon faces and then the grid spacing can adjusted. Jigs are constraint tools, similar to tools that real world carpenters use to build things. They can constrain objects to a plane or a models surface. Or snap objects to grid points that can be spaced as needed. Or snapping an object to another object.