UX Fellows Gesture Research | gotomedia | SXSW | IEEE

#IEEEGoto
Kelly Goto
@go2girl
#IEEEGoto
Thumbs Up to Natural
Gesture Interfaces?
Can We Create a Global Gesture Language?
Tuesday, August 12, 14

#IEEEGoto
Gestures their Origins and Distributions
Morris, Collette, Marsh, O’Shaughnessy
pregnancyandbaby.comparamount.com

#IEEEGoto

#IEEEGoto
Gestuno » International Sign Language
http://www.handspeak.com
Global Gestures
A native language Gestuno is not. But it is a barrier breaking vocabulary that allows a select few to
bridge an international language gap... the need to bridge that international barrier has aroused a
sort of lingua franca rooted deep in the trade industry of modern Europe.
http://www.deaflibrary.org
“Government” in diﬀerent cultures:

#IEEEGoto
Global Gestures
Unlike human-to-human gesturing,
human-machine gesturing does not
have a foundation of thousands of
years of history to serve as a context.
- Silicon Labs

#IEEEGoto
Freehand Gestures Are Entering Our Lives

#IEEEGoto
9,000
8,000
7,000
6,000
5,000
4,000
3,000
2,000
1,000
$Million
2012 2013 2014 2015 2016 2017 2018
Gesture
Recogni<on Touchless
Sensing
Global Touchless Sensing & Gesture Recognition
Market Revenue, $million, (2013-2018)
• Expected
to

reach
$15.02

billion
by
2018.

• Market
value
in
2012

was
approximately
$2

billion.
• Consumer
electronics

applica<on

contributes
to
more

than
99%
of
the
global

gesture
recogni<on

market.

Global Market Revenue Projections

#IEEEGoto
Global Market Revenue Projections
#IEEEGoto

#IEEEGoto
Pointing Gestures

#IEEEGoto
The UX Fellows Gesture Study
#IEEEGoto#IEEEGoto

#IEEEGoto
We Compared Gestures in 18 Countries.

#IEEEGoto
The UX Fellows Gesture Study
Which natural gestures would people from
various cultures spontaneously use to control
consumer electronic devices?
• Country diﬀerences?
• Country clusters?
• Underlying metaphors?
• Acceptance?

#IEEEGoto
Setting
15 min
Warm-up: everyday gestures
#IEEEGoto

#IEEEGoto
Warmup: Everyday ‘Natural’ Gestures
C: I like!
B: I guess I ate
something wrong
A: I think I need to
see a dentist
What are these people trying to tell us?
D: That meal is really
tasty
D: That meal is really
tasty

#IEEEGoto
Examined Commands
Switch on device
Volume up
Channel up
Context info for current program
Channel down
Share current program
Volume down
Mute volume
Pause current program
Homepage
EPG
P+
P-‐
Backward navigation in menus
Conﬁrm on-screen dialog (OK)
Deny on-screen dialog (cancel)
Pause video-on-demand (VOD)
Restart VOD
Fast forward VOD
Rewind VOD
Skip chapter VOD
Previous chapter VOD
Stop VOD
Switch oﬀ device

#IEEEGoto
Participant Sample
Persons who are able to imagine a gesture-control
scenario for a TV.
• Regular usage of a recent HD ﬂat TV (max. 5-6
years of age)
and other CE devices and services
• High interest in consumer electronics
• Regular usage of a touch device
• No Xbox Kinect and gesture-enabled TV users
• Potential buyers of international "premium" CE
brands
(no national, retailer or "no name" brands)

#IEEEGoto
Participant Sample
sample
socio-economic
background
country

#IEEEGoto
Participant Sample
51%49%
18 countries
x 20 subjects
= 360 participants
16
65 years
32
min
max
mean

#IEEEGoto
The Results
#IEEEGoto#IEEEGoto

#IEEEGoto
Difficulty of TV-Related Gestures
Self-‐reported
diﬃculty,
5
=
very
diﬃcult,

1
=
very
easy,
n
=
360
tech-‐savvy
CE
consumers
in
18
countries

easydifficult

#IEEEGoto
Commonality of Gestures
no.
of
countries
with
the
same
main
gesture
High Medium Low

#IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 18
of
18
Diﬃculty
(1-‐5) 1.8
Q:
How
do
you
turn
down
the
volume?

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 18
of
18
Diﬃculty
(1-‐5) 1.8
Q:
How
do
you
turn
up
the
volume?

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 17
of
18
Diﬃculty
(1-‐5) 1.5
Q:
How
do
you
pause
viewing
a
ﬁlm?
NL

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 17
of
18
Diﬃculty
(1-‐5) 2.0
Q:
How
would
you
pause
the
current
TV
program?
NL

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 15
of
18
Diﬃculty
(1-‐5) 1.7
Q:
How
do
you
rewind?
(with
an
image
being
shown)
AU,
UK
NL

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 15
of
18
Diﬃculty
(1-‐5) 1.9
Q:
You
are
watching
a
boring
scene,
how
do
you
fast-‐forward
(with
an
image
being
shown)?
AU,
UK
NL

#IEEEGoto
Volume down
Volume up
Pause VOD
Rewind VOD
Fast forward VOD
Commonality 15
of
18
Difficulty
(1-‐5) 1.9
Q::
You
have
chosen
a
film
from
an
online
video
library.
The
TV
asks
you
"do
you
really
want
to
buy
the
film?"
How
do
you
answer
"yes"?
CN,
IT,
KR

#IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backward navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
(cancel)
Skip chapter VOD
Commonality 13
of
18
Diﬃculty
(1-‐5) 1.8
Q:
You
are
zapping
through
the
TV
programs:
How
do
you
change
from
channel
15
to
channel
16
for
example?
P+
AU,
ES,
KR,
US
CA

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 12
of
18
Diﬃculty
(1-‐5) 1.6
Q:
You
are
zapping
through
the
TV
programs:
How
do
you
change
from
channel
16
to
channel
15
for
example?
AU,
ES,
KR,
US
CA
P-‐

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 13
of
18
Difficulty
(1-‐5) 2.1
Q:
How
would
you
stop
watching
the
film
altogether
(because
you
want
to
do
something
different)?
CN,
KR
FI,
RU
NL

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 11
of
18
Diﬃculty
(1-‐5) 2.0
Q:
How
do
you
set
the
volume
to
"mute"?

AU,
CN,
MX FI,
RU
BE
UK

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 10
of
18
Diﬃculty
(1-‐5) 2.3
Q:
Imagine
that
you
were
looking
for
a
ﬁlm
in
an
online
video
library.
You
have
gone
to
the
comedies
sec<on
and
have
chosen
the
descrip<on
of
a
single
movie.
Now
you
want

to
return
to
the
previous
set
of
op<ons
(all
of
the
comedies).
Which
gesture
would
you
use
for
that?
AU,
BE,
CA
AR,
RU
ES,
MX,
US

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 9
of
18
Diﬃculty
(1-‐5) 1.8
Q:
How
do
you
express
that
you
don’t
want
to
buy
it?

AU,
CN,
RU,
TR
KR
AR,
BE,
IT,
MX

#IEEEGoto
Channel up
Channel down
Stop VOD
Mute volume
(cancel)
Skip chapter VOD
Commonality 9
of
18
Diﬃculty
(1-‐5) 2.2
Q:
How
would
you
skip
to
the
next
chapter
of
the
ﬁlm?
BE,
CN,
ES,
FI,
FR,
IN,
MX,
UK
NL

#IEEEGoto
Commands with Low Commonality
Switch on device
Switch oﬀ device
Restart VOD
EPG
Context info for current
program
Homepage

#IEEEGoto
Commonality 8
of
18
Diﬃculty
(1-‐5) 2.0
Q:
How
would
you
switch
on
the
TV?
Switch on device
Switch oﬀ device
Restart VOD
EPG
program
Homepage
BE,
FR,
US
KR
DE,
IT
TR
CN
CA
FI

#IEEEGoto
Commonality 7
of
18
Difficulty
(1-‐5) 2.9
Q:
How
would
you
switch
off
the
TV?
Note:
No
common
gesture
in
Italy
Switch on device
Switch off device
Restart VOD
EPG
program
Homepage
ES,
MX,
NL
IN,
KR
RU
TR
CA
AR

#IEEEGoto
Commonality 8
of
18
Difficulty
(1-‐5) 2.0
Q:
How
would
you
return
to
the
previous
chapter
of
the
film?
Switch on device
Switch off device
Restart VOD
EPG
program
Homepage
BE,
CN,
ES,
FI,
FR,

MX,
NL
IN,
US
UK

#IEEEGoto
Commonality 7
of
18
Difficulty
(1-‐5) 2.0
Q:
How
do
you
start
it
(film)
again?
Note:
No
common
gesture
in
UK
Switch on device
Switch off device
Restart VOD
EPG
program
Homepage
AU,
ES,
KR,
MX
BE
NL RU
US
AR,
DE

#IEEEGoto
Commonality 6
of
18
Diﬃculty
(1-‐5) 3.2
Q:
How
would
you
share
with
your
friends
what
you
are
currently
watching
or
listening
to?
Note:
No
common
gesture
in
UK
Switch on device
Switch oﬀ device
Restart VOD
EPG
program
Homepage
CN,
FR,
IN
FI
AR,
TR
MX
NL
KR
RU
US

#IEEEGoto
Commonality 5
of
18
Diﬃculty
(1-‐5) 2.8
Q:
You'd
like
to
see
an
overview
of
tonight's
TV
programs
of
the
main
channels
(EPG).
How
do
you
ac<vate
this
func<on
with
a
hand
gesture?

Note:
No
common
gesture
in
Italy
Switch on device
Switch oﬀ device
Restart VOD
EPG
program
Homepage
AU,
US CN,
FI,
IN,

KR,
RU
BE,
UK
TR
NL
MX

#IEEEGoto
Commonality 4
of
18
Diﬃculty
(1-‐5) 2.8
Q:
How
would
you
get
more
informa<on
for
the
running
TV
program?
Switch on device
Switch oﬀ device
Restart VOD
EPG
program
Homepage
FR,
KR,
RU
AU,
FI
CN,
IN,
TR
AR
DE
BE
ES
US
UK

#IEEEGoto
Commonality 4
of
18
Difficulty
(1-‐5) 2.9
Q:
Now
you
want
to
go
to
the
ini<al
set
of
op<ons
-‐
that
means
the
first
op<ons
you
see
when
you
switch
on
the
device.
Note:
No
common
gesture
in
Italy
Switch on device
Switch off device
Restart VOD
EPG
program
Homepage
AR,
AU,
MX
DE
NL,
UK
FI
US
BE,
IN
CA,
ES,
TR

#IEEEGoto
Some General Insights
#IEEEGoto

#IEEEGoto
Direct vs Abstract Gestures
We observed two kinds of natural gestures.
● "Direct" gestures
¡ strongly inﬂuenced by touch devices
● Abstract gestures
¡ basis in language, iconography and metaphors
¡ associated with more complex tasks
¡ lower intercultural commonality

#IEEEGoto
1.0
1.5
2.0
2.5
3.0
TR FR MX AR IN FI CN RU NL DE AU US BE ES CA IT KR UK
Everyday
gestures CE
gestures
diﬃculty
min
max

5.0
Average Gesture Difficulty by Country
…

#IEEEGoto
Average Gesture Difficulty by Country
1.0
1.5
2.0
2.5
3.0
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
CE
gestures
…
diﬃculty
min
max

5.0

#IEEEGoto
General Acceptance
§ Our 360 tech-savvy participants responded very
positively to the idea of semantic gesture input for TVs.
"fun to use", "convenient", "coolness factor”
§ Participants would like to use gestures to control other
devices as well (e.g. audio).

#IEEEGoto
General Acceptance
Can CE gestures be deducted from everyday gestures?
USA
Inﬂuenced by touch screens —
most thought gestures should be
customizable.
Canada
Personal hobbies eg: gestures
inspired by skiing, kick-boxing etc.
Sign language
Russia.
Not a lot of everyday gestures used.
Replicating alphabets, creating new
gestures and pointing used.
No.
Yes.
Mexico
Smart phones and touch screen
devices eg: sliding the menu up from
the bottom, swiping gestures etc.
Spain | Mexico | France | Netherlands
Everyday gestures and touch screen gestures
Australia | Germany | Italy
Everyday gestures
Argentina | China | Korea
Touch screen devices
India
Analog gestures, touch screen
gestures and icon based gestures
Begium
Everyday gestures, gestures while using
electronic devices and touch screen gestures
Finland
Everyday gestures and touch
screen gestures

#IEEEGoto
Private vs Public Display
§ Considerations
§ Ergonomics
§ “Gorilla Arm”
§ Recognition
privatepublic

#IEEEGoto
Operation
Many participants felt that voice and sound
could be a good support for gesture input.
waking
up
the
device
with
a
sound
Can the TV
hear me?
alterna4ve
op4on
based
on
context
and
need

#IEEEGoto
Operation
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
Seman<c
Poin<ng
Many users would prefer switching between gesture types.
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Seman<c l l l l
l l l
l l l l l l
l l 15
Poin<ng
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Seman<c l l l l
l l l
l l l l l l
l l 15
Poin<ng
l

l l l
l
l

l l l
9
associated

with
more

advanced

commands alterna4ve
op4on
based
on
context
and
need

#IEEEGoto
Typical Concerns of Users
§ Learning and retention of gestures
(but: would be easy if based on implicit
behavior repertory)
§ Sensitivity and accuracy of devices
§ Interference by other gestures in the
room
§ Compatibility with lean-back disposition

#IEEEGoto
Conclusions
1. For basic CE commands a common language for natural
gestures already exists across cultures.
2. More complex commands could be realized by either
a. customizing these gestures for each individual market
b. customizing individually
c. proposing a set of semantic gestures that need to be
learned (the study provides some suggestions)
d. switching to pointing gestures in combination with a
GUI
3. It is necessary to consider cultural diﬀerences at a
country level

#IEEEGoto
Future of communication
Do we want to create a better remote
control, or do we want to create a 2-
way language of communication?
- go2girl

#IEEEGoto
LG Magic Remote

#IEEEGoto
The problem: no standard
gestures currently exist.
Many new devices in the world of
‘touchless tech’ are using completely
diﬀerent gestures for the same
common functions.

#IEEEGoto
Can we utilize this methodology to
start a conversation and bring the deaf
community, the large technology
giants and standards together to
develop a universal gesture language?
source: autocarhire.com

#IEEEGoto

#IEEEGoto
The governance
crisis point
Quantity & complexity of
information
Ability to deal with
quantity & complexity
of information
Zap Think LLC
TIME
(innovation) COMPLEXITY vs. COPING (desire)

#IEEEGoto
Remember: People don’t advance as quickly
as technology.
BEYOND USABLE

#IEEEGoto
CONTEXT

#IEEEGotosource: wikipedia
the future?

#IEEEGoto
Find all results on
www.uxfellows.com
Drop us a line:
kelly@gotomedia.com
Michael Woermann
m.woermann@facit-digital.com
www.3bridgesaustin.com

#IEEEGoto
● UX design + research consulting
● San Francisco, CA
● Global mobile focus
#IEEEGoto

UX Fellows Gesture Research | gotomedia | SXSW | IEEE

Recommended

Recommended

More Related Content

Similar to UX Fellows Gesture Research | gotomedia | SXSW | IEEE

Similar to UX Fellows Gesture Research | gotomedia | SXSW | IEEE (9)

Recently uploaded

Recently uploaded (20)

UX Fellows Gesture Research | gotomedia | SXSW | IEEE