The Future of Human Machine Interfaces (HMI)

Future of Human-Machine
Interfaces (HMI)
Virtual Reality
Augmented Reality
Artificial Intelligence
February 2016
Daniel Zahler

©2016 Daniel Zahler | All Rights Reserved.
What is a Human-Machine Interface?
2
System
hardware
controls
Graphical
user
interfaces
Touch screens
• Gesture-based
• Virtual Assistants
• Virtual Reality
1970s 1980s 1990s 2000s 2010s & beyond

A Human-Machine Interface (HMI) is defined
by its input and output mechanisms
3
> Input - A human user needs
some way to tell the machine
what to do, make requests. or
adjust the machine. E.g
keyboards, touch screens,
joysticks, and mice.
> Output - Allows the machine to
keep the human user updated on
the progress of commands, or to
execute commands in physical
space. E.g. computer screen,
machine activity.
Type Complexity
Hardware
Graphical user
interface (GUI)
Touchscreens
Voice-based
Gesture-based
Brain-machine
interface (BMI)
Examples of common HMIs
Less
complex
More
complex

Artificial intelligence is enabling a new
generation of “smart” virtual assistants
4
• “Dumb” software assistants with
limited functionality
• No ability to learn from human
behavior and adapt to new situations
• Intelligent virtual assistants that
learn from a user’s behavior and
anticipate their information needs in
real-time
• “Anticipatory intelligence”
Before… Now…
There is a major shift in how computers deliver information to
humans as a result of:
• Artificial intelligence
• New input mechanisms (tablets, voice, gesture)
• Adaptive-learning software applications

Intelligent virtual assistants are being
embedded in products across every industry
5
Netflix recommendation engine
> Suggests movies & TV shows you might like based on your
previous viewing behavior and movie ratings
Amazon Echo
> Wireless speaker and voice command device. The device is
capable of voice interaction, music playback, making to-do
lists, set alarms, stream podcasts, play audiobooks, provide
weather, traffic and other real time information. It can also
control several smart devices.
Siri (Apple)
> Intelligent personal assistant and knowledge navigator – uses
a natural language user interface to answer questions, make
recommendations, and perform actions by delegating
requests to a set of web services

The next wave of virtual assistants will be
able to learn and improve over time
6
How can an intelligent machine teach users
how to interact with it?
> Soften lessons by using emotional cues
> E.g. Video game that delivers hints when it sees a
player is struggling
> Halo – when user first sets up joystick, receives
prompt: “Which direction is up?” Game then decides
which direction you think is up, lets user settle into
environment
“With learning agents, what they say
back to you doesn’t always reflect
who you think you are – nobody wants
to hear “You’re a shit driver.” How
does a system tell you things in a way
that doesn’t creep you out or violate
your sense of self?”
– Director of Engineering, Intel
Next-generation Virtual Assistants are deploying novel AI algorithms to learn about
human behavior and adjust to it – they become smarter, more empathetic, better at
understanding what a user really wants
> E.g. From Siri to the human-like computer
voiced by Scarlett Johansson in “Her”
> From HMI to “human computer
relationships”
> Goal is an agent that is learning,
participatory, engaging on your behalf.
Can a machine anticipate what you want?
> E.g. Interstellar – astronauts control
TARS robot’s level of sarcasm. Be able to
adjust the avatar, let it know how to
behave

Artificial Intelligence: opportunities & risks
7
Types of Artificial Intelligence:
> Narrow AI - intelligence applied to a
particular problem. This type of AI can
accomplish specific tasks that we
historically had a hard time to get
computers to do well, such as recognize
faces, categorize an image, drive a car,
make a robot walk, etc
> General AI – aka “Strong AI” – the
ability of software / machines to learn
across domains, ultimately becoming
able to perform any intellectual task a
human being can
“In the next 5 years, AI will go from a
“category” to a “layer” that’s
embedded in everything. Look at Siri
and its potential to understand intent
– AI products will provide assistance
when needed. It’ll be like you have an
encyclopedia attached to your head.”
– AI Researcher, Tempo
Potential Issues
> Privacy: How much information can we entrust
with our Virtual Assistant?
> Control: Does control of the user experience (UX)
and personal information reside with you, or with
the product?
> Open or Closed Source: Can you view & alter the
data sources / rankings of your assistant? E.g.
Yelp results from Siri, OpenTable in Google,
TuneIn radio from Alexa
IBM’s Watson: The future of AI?
Watson uses natural language processing and
machine learning to reveal insights from large amounts
of unstructured data

Gesture-based HMI
8
Gesture-based human-machine interfaces accept inputs in the form of hand or body
gestures. These interfaces are highly intuitive but face several challenges.
Potential barriers to adoption
> How will a gesture-based interface assign
meaning to particular gestures – e.g. What
was our gesture for closing a window, and
does it overlap with another gesture?
> Our current technology solves this problem
with affordances – e.g. a remote control has
buttons – but gestures are more difficult to
interpret and distinguish
> Gestures are expressive as they are codified –
system should be aware of this, and learn new
gestures for new users
High-growth market
Analysts forecast the market
for gesture recognition
platforms to grow 30%
annually to $15B by 2018
$5 B
$15 B

Product Description
Oculus Rift
> Virtual reality head-mounted display developed by
Oculus VR. It has a resolution of 1080×1200 per
eye, a 90 Hz refresh rate, and a wide field of view.
> Shipping in Q1 2016 ($599 retail price)
HTC Vive
> Virtual reality head-mounted display being
developed in co-production between HTC and Valve
Corporation
Samsung
Gear VR
> Virtual reality viewer for Samsung phones
> Features an accelerometer, a gyroscope and
proximity-based sensors -- on board to reduce
latency and increase the performance
Virtual Reality (VR): Immersive sensory &
viewing experiences
Virtual reality (VR) replicates an environment that simulates physical presence in places
in the real world or imagined worlds and lets the user interact in that world.

Investment in VR-related companies in 2015
More than $384 M
invested in at least
74 VR companies

What will early Virtual Reality look like?
The amount of VR content has exploded in recent years, as companies develop new
VR-compatible apps and immersive viewing experiences
> Samsung Gear VR users
have uploaded more than
20,000 360-degree videos to
the service
> “Dynamic streaming” –
increases resolution quality of
360 streaming video in VR,
while reducing the amount of
required bandwidth by four
times
> Vrse Works creates original
VR content through a
dedicated headset and app
> Projects include mini-
documentaries “Waves of
Grace” and “Clouds Over
Sidra,” produced in
partnership with the United
Nations.
“We talk about virtual
reality as ‘the last
medium’ … Ultimately,
what we’re talking
about is a medium
that disappears,
because there is no
rectangle on the wall,
there is no page
you’re holding in your
hand. It feels like real
life.”
– Chris Milk, Vrse
CEO

Virtual Reality: potential applications
12
Application Description
Video
games
> Oculus Rift and Samsung Gear VR will come bundled
with an Xbox One controller as well as at least two
games, space fighter EVE: Valkyrie and platformer
Lucky’s Tale
Social
> Immersive VR-based social sharing enables friends to
experience a 360-degree video simulation of their native
environment
> Enhanced collaboration and information sharing as
face-to-face VR conversations connect remote workers
Automotive
> Driver-training tools for simulating real-life situations and
teaching safe driving
> Sales & marketing - customers can simulate the driving
experience of different cars through VR
Real Estate
> Virtual tours for residential or commercial real estate –
VR applications could let buyers explore properties
without traveling to the site
Architecture
> Ability to show clients a 3D walk-through experience of
what a building will look like before construction has
begun
“Why go to
conferences, school,
or travel for business
if you can have
richer, deeper
experiences from the
comfort of your living
room?”
– Philip Rosedale,
Creator, Second
Life
VR could dramatically
reduce the need for
physical travel

Challenges for VR Developers
13
> The first & most popular
applications for VR will
be video gaming; few
other applications
available yet
> Wider variety of VR-
ready games likely won’t
be rolled out until 2017
> Adult VR content could
be a significant early
application
Lack of content
> Consumers may be
unwilling to wear VR
headsets in public due to
bulky size (as with
Google Glass)
> Large size of processors
and 3D displays may
make it difficult to make
VR viewers smaller and
less cumbersome
Form factor
> Resolution, latency
> Potential interoperability
problem of delivering VR
content from multiple
developers across
different devices
> Custom chips may be
needed for rendering 3D
content on a VR device
Hardware challenges
Adoption of VR technology could be slowed by price, lack of compelling VR content,
concerns about bulky headsets, and a failure to deliver consistently high-quality,
immersive 3D experiences

Product Description
Microsoft
HoloLens
> Smart glasses headset with advanced sensors, a
high-definition stereoscopic 3D optical head-
mounted display, and spatial sound. User interacts
with headset through gaze, voice, hand gestures
Magic Leap
> Head-mounted virtual retinal display superimposes
3D computer-generated imagery over real world
objects, by projecting a digital light field into the
user's eye
META
> Smart glasses headset that generates 3D digital
content that can be manipulated with hand gestures
> Early prototypes being tested with game developers
Daqri
> Smart helmet & safety glasses provide mixed-reality
work instructions, safety information and mapping
for industrial workers
> Currently being tested with Autodesk, GE,
Hyperloop
Augmented Reality (AR): Bringing digital
content into the physical world
14
Augmented reality (AR) is a live direct or indirect view of a physical, real-world
environment whose elements are augmented (or supplemented) by computer-generated
sensory input such as sound, video, graphics or GPS data.

Augmented Reality: potential applications
15
Application Description
Healthcare
> Simulations & training to help medical students learn
about the human body
> Telemedicine applications, e.g. consults with doctors
in far off places
> Less-invasive surgeries driven by physician’s ability
to “see” into a patient without opening them up
Design /
Architecture
> Tools to let a designer or architect “see” into the
physical space of the structure they are creating
> Let designers provide interactive 3D display of
physical projects to clients
Logistics
> Improved safety and efficiency for workers
> Show worker location of an item then send the
person and the item to the right place for packing
and shipment
Manufacturing
> Enable engineers in different factory locations to
view and interact with the machinery from wherever
they happen to be in the world
Services
> E.g. for home improvement project, AR system
provides “expert” who shows you what equipment to
use and walks you through all the necessary steps
“Your Magic Leap
headset will allow
you to view a virtual
TV anywhere, on
any wall… or a
mobile phone screen
on the palm of your
hand, or in the air in
front of you.”
– Philip Rosedale,
Creator, Second
Life
AR could replace all
displays and screens

Challenges for AR Developers
16
“Creating an ecosystem
where developers can
build and sustain useful
apps will be key. What
works on a phone, tablet,
or notebook screen will
not necessarily work in
an AR scenario. You
need great hardware to
entice developers to
create apps for that
hardware, but you need
great apps to get people
and companies to invest
in the necessary
hardware.”
– Product Manager, AR
“Key development
challenges are size
(bulky form factor),
latency, and pixel density.
There’s a tradeoff
between size and
immersiveness of field of
view.”
– Head of Product,
META
“10 years out, AR
technology will be small
enough to wear all the
time. One challenge will
be social acceptance of
wearables and sorting out
issues around privacy. But
the potential benefits are
huge. Imagine if we don’t
have to make monitors
anymore – there would be
a great environmental
impact.”
– Head of Product,
META
Hardware challenges Software design ecosystem
Social acceptance of
AR & wearables

Computer Vision:
Intel Real Sense 3D Camera
17
RealSense 3D Camera
> Breakthrough in computer vision
technology: Next-generation facial tracking
human-machine interface
> RealSense Chip generates 2 pictures: Color
picture, and 3D measure of distance in 3D
space – these two are synchronized in 3D time
and space
> Features world’s smallest laser module (1.8
mm) and smallest video projector (4mm)
• Power consumption = 1 W
• 2 Megapixel camera, takes 600 frames per
sec
Applications
> 3D Face recognition: Captures 78 different
3D points on face. Coming to market with
Windows 10 – most secure facial based
recognition login
> Videoconferencing – capture personal 3D
image from a Skype call, and present your
image against a different background
> Next-generation version for autonomous
operation of UAVs / drones – 4 M range
(world facing usages) – drone-mounted
cameras generate 360 degree depth map for
panoramic view of surroundings to avoid
collisions. E.g. Enables a drone to navigate
through forest or combat area
“3D sensing will have compelling use
cases soon – cars, security –
anywhere you can benefit from real-
time human vision.”
– Director of Engineering, Intel

Augmented reality and next-generation cars
18
> Useful data will be displayed on the dashboard, e.g. speed,
navigation, potential hazards
> An AR display is preferable to having a 2nd screen and recreating
the environment right in front of your eyes. Driver should not take
eyes off road. Let people use peripherals safely while driving, and
make the info meaningful.
> However, this presents many challenges, e.g. How to make
connection between what’s in front of you and then provide
information relevant to this context?
> Goal is for user interface to blend into the background, not intrude
on engaging directly in physical world.
“The Holy grail for auto makers
is an Augmented Reality
display on the windshield.”
– Lead Engineer, Stanford
Automotive Innovation Lab
AR “field of vision” displays are poised to transform the driving experience:

Volkswagen Automotive Innovation Lab
(VAIL): X1 electric vehicle
> Each wheel can be steered and braked independently
> Provides haptic feedback to driver via steering wheel
> Can drive autonomously or share control with driver
Toward a world of semi-autonomous vehicles
19
Driver Monitoring & Feedback
> Visual sensors - 3D cameras detect
upcoming obstacles, bumps in road, etc.
> Haptic feedback – e.g. steering wheel
shakes when you’re about to have
collision or drive off-road
> Facial tracking system - measures
driver awareness, heartrate,
sleepy/distracted. Can extract biometric
data – is the driver stressed?
> Goal is to translate this biometric data
into car behavior – e.g. If stressed,
don’t deliver a phone call to driver
“The car may have great features, but
what if I’ve got no idea how to access
them when I’ve already begun my
journey? I don’t want to feel like an idiot
when I’m in my car – I want to feel like
Tony Stark with Jarvis.”
– Product Manager, Stanford
Automotive Innovation Lab

How AI and machine learning will transform
every industry

Notable HMI technologies in development
21
Type of HMI Example companies Products
Gesture-based
> Leap Motion
> Microsoft Kinect
> Sony SoftKinetic
Natural Language
> Amazon Echo
> Apple Siri
> Google Now
Wearables
> Apple
> FitBit
> 5th Dimension Technologies
Virtual Reality
> HTC Vive
> Oculus VR
> Vrse.works
Augmented Reality
> Magic Leap
> Microsoft HoloLens
> Daqri

Danzahler.wordpress.com
zahler@gmail.com
@DanZahler
https://www.linkedin.com/in/daniel-
zahler-64b0012

The Future of Human Machine Interfaces (HMI)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to The Future of Human Machine Interfaces (HMI)

Similar to The Future of Human Machine Interfaces (HMI) (20)

Recently uploaded

Recently uploaded (20)

The Future of Human Machine Interfaces (HMI)