Seven Master of Arts students from Constance at the University of Applied Sciences Communication Design faculty are working on design research concerning multi-touch interfaces during summer term 2008. Studying a research paper ...

1. Touch Research
A project for Communication Design M1
HTWG Constance
Summer Term 2008
Part 3: Reading a Research Paper
How Bodies Matter: Five Themes for Interaction Design, DIS 2006,
June 26–28, 2006

2. With the graphical user interface (GUI) so popular these days ...

3. ... this reduced “body” symbolizes how we use our computers. With
keyboard (fingers),
mouse (fingers),
monitor (eyes), and
speakers (ears).

4. Are richer interaction paradigms possible?

5. The tangible user interface (TUI)!
A tangible user interface (TUI) is a user interface in which a
person interacts with digital information through the physical
environment. The initial name was graspable user interface, which
no longer is used.
Image: Nintendo Wii

6. Bodies matter!
Consider riding a bicycle: one is simultaneously navigating,
balancing, steering, and pedaling; yet it is not possible for
bicyclists to articulate all of the nuances of an activity that
they successfully perform.
Perhaps the most remarkable aspect of this is that riding a
bicycle is just one of thousands of activities that our bodies
can do.
Our physical bodies play a central role in shaping human
experience in the world, understanding of the world, and
interactions in the world.
This paper draws on theories of embodiment — from psychology,
sociology, and philosophy — synthesizing five themes we believe
are particularly salient for interaction design.

7. Contrast the richness, subtlety, and coordination of tasks at
several levels of concern that bicycling offers with the
graphical user interface that we use today.

8. Douglas C. Engelbart, oN-Line System (NLS) and computer mouse,
1960s
A revolutionary computer collaboration system designed by Douglas
Engelbart and the researchers at the Augmentation Research Center
(ARC) at the Stanford Research Institute (SRI).
The first system to employ the practical use of hypertext links,
the mouse (co-invented by Engelbart and colleague Bill English),
raster-scan video monitors, information organized by relevance,
screen windowing, presentation programs, and other modern
computing concepts.
Features: The mouse, 2-dimensional display editing, in-file
object addressing, linking, hypermedia, outline processing,
flexible view control, multiple windows, cross-file editing,
integrated hypermedia email, hypermedia publishing, document
version control, shared-screen teleconferencing, computer-aided
meetings, formatting directives, context-sensitive help,
distributed client-server architecture, uniform command syntax,
universal "user interface" front-end module, multi-tool
integration, grammar-driven command language interpreter,
protocols for virtual terminals, remote procedure call protocols,
compilable "Command Meta Language”

9. Altair 8800, 1975
The MITS Altair 8800 was a microcomputer design from 1975 based
on the Intel 8080 CPU and sold by mail order through
advertisements in Popular Electronics, Radio-Electronics and
other hobbyist magazines.
The Altair is widely recognized as the spark that led to the
microcomputer revolution of the next few years: The computer bus
designed for the Altair was to become a de facto standard in the
form of the S-100 bus, and the first programming language for the
machine was Microsoft's founding product, Altair BASIC.
Programming the Altair was an extremely tedious process. The user
toggled the switches to positions corresponding to an 8080
microprocessor instruction or opcode in binary, then used an
'enter' switch to load the code into the machine's memory, and
then repeated this step until all the opcodes of a presumably
complete and correct program were in place.
When the machine first shipped the switches and lights were the
only interface, and all one could do with the machine was make
programs to make the lights blink.

10. Xerox Star, 1981
The Star workstation, officially known as the Xerox 8010
Information System, was introduced by Xerox Corporation in 1981.
It was the first commercial system to incorporate various
technologies that today have become commonplace in personal
computers, including a bitmapped display, a window-based
graphical user interface, icons, folders, mouse, Ethernet
networking, file servers, print servers and e-mail.
The key philosophy of the user interface was to mimic the office
paradigm as much as possible in order to make it intuitive for
users. The concept of WYSIWYG was considered paramount. Text
would be displayed as black on a white background, just like
paper, and the printer would replicate the screen using
InterPress, a page description language developed at PARC.
The user would see a desktop that contained documents and
folders, with different icons representing different types of
documents. Clicking any icon would open a window. Users would not
start programs first (e.g. a text editor, graphics program or
spreadsheet software), they would simply open the file and the
appropriate application would appear.

11. Apple iMac, 2008 edition
Members of the Apple Lisa engineering team saw Star at its
introduction at the National Computer Conference (NCC '81) and
returned to Cupertino where they converted their desktop manager
to an icon-based interface modeled on the Star. Among the
developers of the Gypsy editor, Larry Tesler left Xerox to join
Apple in 1980 and Charles Simonyi left to join Microsoft in 1981
(whereupon Bill Gates spent $100,000 on a Xerox Star and laser
printer),[9] and several other defectors from PARC followed
Simonyi to Microsoft in 1983.
Some people feel that Apple, Microsoft, and others plagiarized
the GUI and other innovations from the Xerox Star, and believe
that Xerox didn't properly protect its intellectual property.

12. Dan Saffer, reviewing the paper:
“With the current keyboard-mouse-monitor set-up, we do every
task, no matter if it is writing a paper or editing a movie or
even playing a game, all the same way. Pointing, clicking,
dragging and dropping, etc. The work has become ‘homogenized‘.”
http://www.odannyboy.com/blog/new_archives/2008/05/
review_five_the.html

13. Five themes for interaction design:
1. Thinking through doing
2. Performance

14. 1. Thinking through doing
describes how thought (mind) and action (body) are deeply
integrated and how they co-produce learning and reasoning.
Individual corporeality.
Unlike theories of information processing and human cognition
that focus primarily on thought as something that only happens in
the head, theories and research of embodied cognition regard
bodily activity as being essential to understanding human
cognition.
Dan Saffer: “There are a lot of skills you simply cannot learn by
reading or listening alone. You have to try them out. Gestures
aren't just for embellishment to communication, they can also be
an aid to learning and understanding. Manipulation of items
allows for greater understanding of the item. Artifacts have
their own characteristics, and their "backtalk" uncovers problems
or can suggest new designs.”

15. 1.1 Learning Through Doing
Montessori toys employ bodily engagement with physical objects to
facilitate active learning.
Being able to move around in the world and interact with pieces
of the world enables learning in ways that reading books and
listening to words do not.

16. Having the hands stuck on a keyboard are likely to hinder the
thinking and communication.
1.2 The Role of Gesture
From studies of gesturing in face-to-face interactions, we know
that people use gesture to conceptually plan speech production
and to communicate thoughts that are not easily verbalized.
Gesturing has been shown to lighten cognitive load for both
adults and children.
Less constraining interaction styles are likely to help users
think and communicate.

17. 1.3 Epistemic Action
Distinguishing pragmatic action — manipulating artifacts to
directly accomplish a task — from epistemic action — manipulating
artifacts to better understand the task’s context.
Puzzle players manipulate pieces to understand how different
options would work.

18. 1.4 Thinking through Prototyping
Reflective practice, the framing and evaluation of a design
challenge by working it through, rather than just thinking it
through, points out that physical action and cognition are
interconnected.
Successful product designs result from a series of “conversations
with materials.”
The epistemic production of concrete prototypes provides the
crucial element of surprise, unexpected realizations that the
designer could not have arrived at without producing a concrete
manifestation of her ideas.

19. 1.5 On Representation
The representation of a task can radically affect our reasoning
abilities and performance.
Tangibility offers both direct familiarity and a set of common
metaphors to leverage in interaction.
The most common stated purpose of tangibility is that these
interfaces provide “natural” mappings and leverage our
familiarity with the real world, e.g., virtual objects are
positioned in virtual space by moving physical handles in
physical space.

20. 2. Performance
describes the rich actions our bodies are capable of, and how
physical action can be both faster and more nuanced than symbolic
cognition . Individual corporeality.
One of the most powerful human capabilities relevant to designers
is the intimate incorporation of an artifact into bodily practice
to the point where people perceive that artifact as an extension
of themselves; they act through it rather than on it.
While much of the recent TUI literature has focused on “walk up
and use” scenarios which require a low use threshold, this
section describes how designing for skilled bodies can yield
interfaces for expert performance. Physical interfaces with
dedicated (i.e., spatially multiplexed) controls and dedicated
actions can leverage this skill to improve interaction speed and
reliability.

21. Dan Saffer: “We should design products for expert users, able to
use their hands and motor memory to perform action-centered
skills. Thinking can be too slow; experiential cognition (learned
skillful behavior like driving a car) can be more rapid and
powerful than reflective cognition.”
2.1 Action-centered Skills
The tacit knowledge that many physical situations afford plays an
important role in expert behavior.
We draw attention to the importance of tacit knowledge because
computerization can, often accidentally, inhibit it.

22. 2.2 Motor Memory
We are able to sense, store and recall our own muscular effort,
body position and movement to build skill.
It is this motor, or kinesthetic, memory that is involved in
knowing how to ride a bicycle, how to swim, how to improvise on
the piano.
Traditional GUI interfaces employ the same bodily actions for a
wide variety of tasks - this universality is both a strength and
a weakness.
For any given application, kinesthetic memory can only be
leveraged to a limited extent since the underlying actions are
the same across applications.

23. 2.3 Reflective Reasoning
Beyond reliability and robustness of kinesthetic recall, speed of
execution also favors bodily skill for a class of interactive
systems that require tight integration of a human performer “in
the loop.”
Many daily actions such as driving a car or motorcycle, operating
power tools, or engaging in athletic activities require complex
yet rapid bodily responses for which planning through explicit
cognition is simply too slow.
Tangible interfaces that engage the body can leverage body-
centric experiential cognition. To date, computer game
controllers have been the most commercially successful example of
such interfaces.

24. 2.4 Hands
Simultaneously a means for complex expression and sensation:
Hands allow for complicated movement but their skin also has the
highest tactile acuity of our extremities.
Many of the complex motions that we perform are bi-manual and
asymmetric.

25. 3. Visibility
describes the role of artifacts in collaboration and cooperation.
Social affordance.
The extent to which the activities of a practice are made visible
to colleagues and onlookers through the performance of the
activity.
Dan Saffer: “Through the performance of an activity, that
activity can be made visible to others easily, so that
collaboration and situated learning can occur spontaneously. ”

26. Visibility facilities
3.1 Coordination
The visibility provided through collocated practice with task-
specific artifacts is also successful in supporting synchronous
collaboration, and can be especially useful in mission-critical
systems.

27. 3.2 Situated Learning
The invisibility of work practice that the GUI has brought about
inhibits peripheral participation.
A child watching to see what an adult is doing in front of a
laptop, and then copying those motions is not learning anything.
With the graphical interface, there is no mechanism to be aware
of the practices of experts; it all looks the same.

28. That’s what
3.3 Live Performance
is about
The value we place in visibility of creative production is
exemplified by live musical performance.
While the music itself is more intricate and polished in studio
recordings, audiences still pack concert venues because live
perform-ances permit listeners to witness the act of performance
as well as co-produce the event (musician and audience respond to
each other through mutual feedback).

29. 4. Risk
explores how the uncertainty and risk of physical co-presence
shapes interpersonal and human-computer interactions. Social
affordance.
Dan Saffer: “Most products are designed to decrease risk, but
retaining some risk can be beneficial. With risk comes trust,
responsibility, and attention.”

30. 4.1 Physical Action is Characterized by Risk
Risk is having to choose an action which cannot be undone while
the consequences of the action are not fully knowable ahead of
time.
One cannot undo a social faux pas in face to face interactions;
technology mitigates against this risk: one can delete sentences
before sending them to friends over IM or email.

31. 4.2 Trust and Commitment
Though risk can make people feel more anxious about interactions
with others, it can also engender the kind of trust necessary for
successful distance collaborations.
Situations that involve more risk can also stimulate more
committed involvement by participants of the interaction.
Painting in watercolor requires more commitment to each stroke
than working in Adobe Photoshop.

32. 4.3 Personal Responsibility
There are situations where the decision-makers should not be
subject to the overwhelming repercussions of their decisions,
e.g., natural disaster response planning.

33. 4.4 Attention
Situations of higher risk cause people to feel more emotion-ally
negative and, therefore, more focused, paying closer attention to
detail, while situations of low risk allow people to feel more
emotionally positive, relaxed, curious, and creative.

34. 5. Thickness of practice
suggests that because the pursuit of digital verisimilitude is
more difficult than it might seem, embodied interaction is a more
prudent path.
Dan Saffer: “Because there is so much benefit to the real world,
we should be careful with replacing physical artifacts with
digital ones. The best case scenario is to augment the physical
world with digital behaviors, and thus "admitting the
improvisations of practice that the physical world offers."”

35. 5.1 Final Scratch
From a design perspective, solutions that carefully integrate the
physical and digital worlds — leaving the physical world alone to
the extent possible — are likely to be more successful by
admitting the improvisations of practice that the physical world
offers.
Clearly, the digital world can provide advantages. To temper
that, we argue that because there is so much benefit in the
physical world, we should take great care before unreflectively
replacing it.

36. 5.2 “Embodied Virtuality” Rather Than Virtual Reality.
Designing interactions that are the real world instead of ones
that simulate or replicate it hedges against simulacra that have
neglected an important practice.

37. “In this paper we developed our view of the affordances of
physicality and concreteness for the design of interactive
systems.
We believe the five themes presented in this paper will be of
value both generatively — helping designers come up with new
solutions — and for evaluation — providing a rich set of axes for
analyzing the benefits of systems.”
Scott R. Klemmer, Björn Hartmann, Leila Takayama

38. Credits

39. Via Creative Commons on flickr:
Javier Martínez www.flickr.com/hyoga/1165367241/
raysto www.flickr.com/raysto/252630833/
duncan c www.flickr.com/duncan/92623025/
Kit Cowan www.flickr.com/kitcowan/712113879/
Chel's piccies www.flickr.com/mich_chel/1858616605/
Stefan Sonntag www.flickr.com/zerega/1029076197/
foreversouls/ tree & j hensdill www.flickr.com/foreversouls/
2083165757/
Derrick Mealiffe www.flickr.com/dmealiffe/171720479/
Thomas Hawk www.flickr.com/thomashawk/2492298772/
François @ Edito.qc.ca www.flickr.com/francois/1161267539/
Philip Milne www.flickr.com/pamilne/1392285543/
Tom Adriaenssen www.flickr.com/inferis/266391949/
missionbycicles www.flickr.com/mission_bicycles/2108803461/
Chris Gansen www.flickr.com/daychokesnight/2041048614/
Connie Arida/ Connie Sec www.flickr.com/conniesec/2211700732/
Bruce Beh www.flickr.com/brucebeh/120075758/
all in green/ Shana www.flickr.com/isoldesmom/446843362/

40. Via Creative Commons on flickr:
SOCIALisBETTER www.flickr.com/27620885@N02/2597329151/
"Cowboy" Ben Alman www.flickr.com/rj3/2562634563/
Loran Tatooine www.flickr.com/noloran/641931694/
Luca Casamassima www.flickr.com/tyler89/2364065361/
Axel Pfaender www.flickr.com/axor/2295802927/
rickz www.flickr.com/rickz/3071093/
Jenny Downing www.flickr.com/jenny-pics/2583485800/
Daniel Y. Go www.flickr.com/danielygo/1781777706/
Marc A. Garrett www.flickr.com/since1968/9923894/
Cortiça CenaCarioca www.flickr.com/cenacarioca/355888627/
Cheon Fong Liew www.flickr.com/liewcf/894035077/
Luca Mascaro www.flickr.com/lucamascaro/523822075/

41. Additional thanks to:
Wonderbrains.com
Apple.com
OLPC
Perceptive Pixel

42. How Bodies Matter: Five Themes for Interaction Design
Scott R. Klemmer, Björn Hartmann, Stanford University HCI Group,
Computer Science Department
Leila Takayama, Stanford University CHIMe Lab, Communication
Department
DIS 2006, June 26–28, 2006, University Park, Pennsylvania, USA.
Copyright 2006 ACM 1-59593-341-7/06/0006.
http://hci.stanford.edu/publications/2006/HowBodiesMatter-
DIS2006.pdf
Permission to make digital or hard copies of all or part of this
work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or
commercial advantage.

43. Dan Saffer, reviewing the paper
http://www.odannyboy.com/blog/new_archives/2008/05/
review_five_the.html
All the rest
http://wikipedia.org
University of Applied Sciences Constance, Faculty for
Communication Design, Project “Touch Research”
http://www.htwg-konstanz.de
http://www.kd.fh-konstanz.de/dina8/daten_e.php?wodenn=will
http://www.felgner.ch/2008/04/touch_research.html