Brain and Body Interfaces (BBI) were discussed in this workshop. Its goal was to provide a platform for creating synergies between two related and emerging HCI disciplines (PC and BCI). Find out more at the workshop website: http://www.eecs.tufts.edu/~agirou01/workshop/
ICT Role in 21st Century Education & its Challenges.pptx
Brain, Body, and Bytes CHI 2010 Workshop Presentations
1.
2. BBB Workshop Overview
Audrey Girouard, Erin Treacy Solovey,
Regan Mandryk, Desney Tan,
Lennart Nacke, Robert J.K. Jacob
3. BBB Workshop: Bringing Together
BCI& Physiological Computing
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 3
4. Workshop Goals
1. Provide a platform for creating synergies
between two related and emerging HCI
disciplines (PC and BCI).
2. Kick-start BBB research in HCI by identifying
key research questions and application areas
3. Enhancing HCI methodologies by adding these
new methods and techniques to the toolbox of
HCI researchers
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 4
5. About Us
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 5
6. About you
• 25 Participants • X EEG papers
• Y fNIRS papers
• Z EMG papers
• 9 Brain papers
• A GSR
• 19 Body papers
• B Eye tracking
Brain Body
32%
68%
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 6
7. Workshop Discussion Questions
<on your tables>
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 7
8. Workshop Discussion Questions
What terms/acronyms should we use to
describe our work?
Psychophysiological computing seems a bit long…
BBB?
BBI?
Other suggestions?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 8
9. Today’s Schedule
09:00-10:30 Introduction of organizers & participants, CHI madness style
10:30-11:00 Break
11:00-12:30 Group Discussion (assign groups + topic)
12:30-14:00 Lunch
14:00-15:30 Small group discussion (1h of discussion + 30 minutes of reporting)
15:30-16:00 Break
16:00-17:00 Report from small groups, recap, group discussion – barriers to
publication in HCI
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 9
10. Staying connected
Website http://www.eecs.tufts.edu/~agirou01/workshop/
Twitter http://twitter.com/bbbCHI2010
Twitter #bbbCHI2010
Hashtag
Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-
Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786
Flickr http://www.flickr.com/groups/bbbchi/
Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-
user-interaction
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 10
11. And now… 2 Minute Introductions
• About your research:
– Application or domain space? (e.g. video games)
– Signal you are measuring? (e.g. fNIRS)
– Device/hardware you are using? (e.g. ISS, Inc)
• What would you like to get out of workshop?
– What problems are you facing?
– What do you see in common with others here?
– What device would you like to know more about?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 11
12. List of Presentations
1. Adriane Randolph 16. Katarzyna Wac
2. Krzysztof Z. Gajos 17. Feng Tian
3. Saraswathi Bellur 18. Veronica Zammitto
4. Danny Plass-Oude Bos 19. Johanna Octavia
5. Kai Kuikkaniemi 20. Bob Wray
6. Mitchel Benovoy 21. Marjolein van der Zwaag
7. Yee Chieh Chew
8. Martha Crosby
9. Jan Kallenbach
10. Eva Oliveira
11. Lucia Filgueiras
12. Marc Grootjen
13. Melody Moore Jackson
14. Johann Schrammel
15. Daniel Sjölie
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 12
13. Brain-Computer Interfaces as Child’s
Play: Using Mental Training to Improve
Motor Skills in Children
Adriane B. Randolph, Ph.D.
Ashley Ingraham
Kennesaw State University
14. Outline
• Research Position
– Examine ties between sensorimotor area and physical ability to uncover a
training tool to aid children’s motor skill development
• Pilot Study
– 3 able-bodied adults (2 Male, 1 Female)
– Acquire baseline Wii tennis score
– Conduct mental training with BCI2000 mu training protocol for 5 days
– Retest Wii tennis score for improvements Participant Session One Session Five
– Mixed results AZ 98 60
CM 70 64
MW 103 118
• Future Work
– Extend study with 20 able-bodied adults (currently in progress)
– Analyze link between training and baseline ability
– Test with elementary-aged children
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 14
15. Screening for Mu
Ashley prepares participant to use mental imagery of hand and foot
movements to determine his best channels for use in mu BCI training before
he plays a match of Wii Tennis.
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 15
16. Thank you
Adriane B. Randolph, Ph.D.
Kennesaw State University BrainLab
(770) 423-6083
brainlab@kennesaw.edu
http://coles.kennesaw.edu/brainlab/
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 16
17. Understanding How to Design Complex
Brain-Controlled Applications
Krzysztof Z. Gajos
18. How to Design Brain-Controlled
Applications?
• Basic operations
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 18
19. How to Design Brain-Controlled
Applications?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 19
20. How to Design Brain-Controlled
Applications?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 20
21. How to Design Brain-Controlled
Applications?
• Basic operations
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 21
22. How to Design Brain-Controlled
Applications?
• Interaction between primary task and BCI
performance
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 22
23. How to Design Brain-Controlled
Applications?
• Error prevention, recovery, and actions
with side effects
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 23
25. Psychophysiological Responses
to Media Interfaces
Saraswathi Bellur S. Shyam Sundar
Media Effects Research Lab
http://www.psu.edu/dept/medialab
Pennsylvania State University
26. Biopac MP 35 & MP 150
ACQknowledge software
Skin Conductance (EDA)
Heart Rate (ECG)
Facial EMG
Brain Wave (EEG)
Eye Movement (EOG)
Application Areas
Interface presentation
Information Processing
Usability Testing &
Evaluation
User Experience
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 26
27. Physiological Measures: Affect, Arousal & Attention
3. ECG: Heart
Rate recording
1. EEG: Bipolar recordings Occipital Alpha Blocking
Orienting
Response via
BPM change
2. Skin Conductance (EDA) Tonic arousal levels 4. Facial EMG:
Corrugator &
Zygomatic
recordings
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 27
28. Theoretical Frameworks
• The Media Equation: Mediated Life = Real Life (Reeves & Nass,
1996).
• Structural aspects of media technologies: Automatic or
Controlled processing? Voluntary or Involuntary responses?
• Specific cues triggered via interactions with four types of
media affordances, MAIN model (Sundar, 2008)
Animation
M: MODALITY Customization Heuristics
A: AGENCY
Information Perceptual
Scent Bandwidth
I: INTERACTIVITY
N: NAVIGABILITY Contingency UGC & Sourceness
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 28
29. Thank you
Saras Bellur (saras@psu.edu)
S. Shyam Sundar (sss12@psu.edu)
Media Effects Research Lab
http://www.psu.edu/dept/medialab
Pennsylvania State University
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 29
30. BCI for Games
Danny Plass-Oude Bos
Boris Reuderink, Bram van de Laar,
Hayrettin Gürkök, Christian Mühl,
Mannes Poel, Anton Nijholt, Dirk Heylen
HMI, University of Twente
31. Target Group: the general population
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 31
32. BCI: issues
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 32
33. BCI: new information, new features
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 33
34. Acceptance: Usability & Realistic Settings
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 34
35. Subjects Users
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 35
38. Emoshooter
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 38
39. Emoshigeru - cinema games
Real-time
Public display
Biosignal adaptation
Massive co-located
Shooter / Strategy
Part of Shigeru-games
local
wlan
Public display
Connected to game
server
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
40. Presemo – presentation engine
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
41. Emolisten – biosignal sonification
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
42. Research statement
• Social Games and Playful Applications,
which Utilize Biosignals
• Grey Area between Adaptation and
Biofeedback
– Implicit and Explicit Biofeedback
• Biosignal Synchrony (Inter-Personal)
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 42
43. Our team and projects
Professor Marko Turpeinen
Professor Timo Saari
Ilkka Kosunen, Toni Laitinen, Petri Lievonen, Pauli Ojala
Fun of Gaming (EU FP6) (2006-2009)
Enactive Social Media and Games (Aalto) (2009)
Emokeitai (Tekes – National Innovation Agency) (2009-2011)
Thanks:
CKIR / Niklas Ravaja
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 43
44. Psychophysiological signal
analysis and classification
Mitchel Benovoy
Centre for Intelligent Machine, McGill University
45. Outline
• Signals
• Hardware
• Application domains
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 45
47. Hardware
• Thought Technology ProCom Infinity
Images via www.thoughttechnology.com
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 47
48. Application Domains
• Affective Computing
– Interaction evaluation (UI, devices, latency)
– Music & emotion research
– Psychoneurology
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 48
49. Thank you
benovoym@cim.mcgill.ca
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 49
50. BCI and Creativity
Yee Chieh (Denise) Chew
Georgia Institute of Technology
51. Outline
• fNIR and Creative Expression
• EEG and Music
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 51
52. Character Templates for fNIR
Created with a mouse, these are templates that subjects traced
using fNIR
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 52
53. Thank you
ychew@gatech.edu
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 53
54. Physiological Measures Used for
Identification of Cognitive States and
Continuous Authentication
Curtis S. Ikehara & Martha E. Crosby
University of Hawaii at Manoa
55. Outline
• Overview of our Research Using
Physiological sensors
– Assessing Cognitive States
– Continuous Authentication
• Possible Applications
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 55
56. Multi-modal Physiological Sensors System
Eye Tracker
(Gaze Position, Fixation Number, Fixation
Duration, Repeat Fixations, Search Patterns,
MTF Pupil Size, Blink Rate, Blink Duration)
(Moving Target Fractions) HUMAN
Pressure Mouse
(Pressures on Mouse Button and Body)
Laptop
(Data Collection, Analysis & Display)
EDA / TSA Finger Sensors
(Electrodermal and Temperature Sensor Activity)
Oximeter
(Pulse, Blood Oxygen)
Custom Electronics
(Analog to Digital Converter)
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 56 56
57. A Composite of Over 20 Mouse Clicks of
Three People (Click Signature)
US Patent # 7,245,218
Subject 4
Subject 5 Subject 7
57
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
58. Possible Applications
• The student’s confidence can be assessed by
evaluating the distortions of the mouse click
signature.
• High Value Vehicle Control
– Pressure and other physiological sensors can be
attached to a steering (e.g. aircraft yoke or
steering wheel) .
– Sensors can be used to indicate that:
• The authorized pilot or driver is in control.
• The cognitive state of the pilot is consistent with the
vehicle’s activity.
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 58
59. Thank you
Curtis S. Ikehara Martha E. Crosby
cikehara@hawaii.edu crosby@hawaii.edu
University of Hawaii at Manoa
1680 East-West Road - POST 317
Honolulu, Hawaii 96822 USA
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 59
60. Measuring Interaction Experiences:
Integration of Multiple
Psychophysiological Methods
Jan Kallenbach
Aalto University
Helsinki, Finland
61. Outline
User Experience
=
Choice & Interaction
+
Cost-Benefit Evaluations
Measuring Interaction Experiences?
Software Framework
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 61
62. Multimodal
Measurements
Client Application for UX Research
Sampling frequency: max 1000Hz
Psychophysiological Measures: Gaze, Pupil Size, EDA, EMG, EEG, …
Behavioral Measures: Keyboard, Mouse, Slider, …
System Measures: Window Events
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 62
63. Thank you
Jan Kallenbach
Research Scholar
Aalto University
School of Science & Technology
Department of Media Technology
P.O. Box 15500
FIN - 00076 Aalto
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 63
64. Towards enhanced video access and
recommendation through emotions
Eva Oliveira
Nuno Ribeiro
Teresa Chambel
IPCA | LASIGE
65. We Want
• To access to videos based on emotions
– user actual emotional (psychophysiological) state
– video emotional content
– director’s perspective
• To recommend videos based on emotions
– create a set of rules from every affective
classification perspective (user and video, director)
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 65
66. Affective video recommendation
system requirements ()
• Every video must be classified affectively from the user
and from the content perspective.
• Every user has an emotional profile constituted by all his
implicit (physiological) classification of every classified
movie.
• User physiological signals can change the way we watch
the videos
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 66
67. User Classification Issues
• Automatic Classification
– Users emotional state
• emotional model
– Classification techniques
• Result output
• Accuracy
• Lab contexts
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 67
68. Current State
• Capturing users phychophysiological signals
(BIOPAC System) while watching movies
This is my exciting picture
• Next step: several classification techniques to
convert signal result into emotional categories
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 68
69. Thank you
• Eva Oliveira :
• eva.oliveira@gmail.com
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 69
70. Your users can’t talk!
How can you tell they liked
your system?
Lucia Filgueiras and Claudia Tambascia
Escola Politecnica
Universidade de São Paulo
71. What we want to do
• Develop products for cerebral palsy users
• Perform participatory design
• Understand non-verbal communication
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 71
72. Direct measures and usability evaluation
HRV and workload measures Eye tracking in iTV apps
in control room operability evaluation
evaluation
Comparison of eye tracker
NASA TLX, SWAT, Heart Rate
findings to conventional
Variability usability tests results
Variabilidade da frequência cardíaca OP13
LF nu / HF nu LSC Média
18,0
Operador analisa sistema após vários
Índice do Balanço Simpático? Parassimpático
Ação do operador para alarmes: temperatura alta (10:29); Ação do operador para
16,0
correção nível alto (10:32) e segurança correção
14,0 (10:32; 10:37; 10:43) 14,1
12,0 Alarme de 11,7
temperarura alta:
10,0 10:15
9,1
8,3 8,5
8,0
7,3
6,6
6,0 5,9
5,2 5,3 5,6
4,6 5,0
4,0 3,9 3,8 4,0 4,4 4,1
2,9 3,3
2,6 2,6 2,6 2,3 2,4
2,0
0,0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
0
5
:3
:4
:4
:5
:5
:0
:0
:1
:1
:2
:2
:3
:3
:4
:4
:5
:5
:0
:0
:1
:1
:2
:2
:3
:3
09
09
09
09
09
10
10
10
10
10
10
10
10
10
10
10
10
11
11
11
11
11
11
11
11
HORA
Heart Rate Variability of an operator in a unstable unit situation
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
73. Our questions:
• How personalized are body expressions for cerebral palsy patients?
• Are there general body expressions that can mean the same for everyone?
• Is it possible to determine a generic way of measuring these expressions?
• Is there any other liaison besides gestures and facial expressions , like an
unconscious connection of limbic systems that can be responsible for the
mutual sensitive communication?
• Is there a reference body signal that can be used as a reliable parameter to
evaluate body expressions?
• How can we establish a correlation between body expressions and
patient´s opinion of approval or disapproval of a design solution?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 73
74. Thank you!
Contact us:
Lucia V. L. Filgueiras
Escola Politecnica, Universidade de São Paulo
lucia.filgueiras@poli.usp.br
Claudia A.Tambascia
Fundação CPqD
claudiat@cpqd.com.br
http://lts-i.pcs.usp.br
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 74
76. Outline
• Symbiosis & mutual understanding
• Information models
• Interfaces
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 76
77. Maximum task performance
Symbiotic entity
Disturbance
Mutual
human understanding computer
Tasks (goals)
Human-computer symbiosis with mutual understanding
as key issue for maximum task performance
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 77
78. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 78
80. Extended Home-based BCI study
• fNIR-based BCI for communication
• 32 People with ALS (so far) over 1 year
• 78% (25) could operate the device on
initial visit with accuracy > 70%
• Accuracy to 100%, average 76.5%
• Issues:
– Motivation
– Data Collection
– Remote Support
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 80
81. The Kokoro Gatari
Brain Computer Interface based on Functional Near Infrared
imaging
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 81
82. Thank you
Melody Moore Jackson
melody@cc.gatech.edu
404-277-1324
Ian McClendon
imclend@cc.gatech.edu
404-641-7712
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 82
83. Get up, move on! Using
Electromyography to Explore
the Relationship of Experience
and Motion
Johann Schrammel
Centre for Usability Research and
Engineering
84. Outline
• Compare experiences and emotions in
conditions with and without movements
with Electromyography(EMG) and other
methods
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 84
85. Outline
• Can EMG successfully be used to
estimate a user‘s emotional experience in
motion contexts (artefacts, feasibility,
etc.)?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 85
86. Outline
• Is a more embodied way of controlling
interaction actually amplifying experience?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 86
87. Results
• EMG is suitable for motion contexts.
• Both EMG and self-report data indicate
that natural body movement intensifies
emotions.
• It’s crucial that body movements are in
accordance with activity.
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 87
88. Thank you
schrammel@cure.at
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 88
90. Reality-Based Brain-Computer
Interaction
• Complex, reality-based, interaction with the
ability to adapt to the user in real-time
– Using brain measurements to evaluate
• Direct brain-computer interaction
– The user can be unaware
• We’re using a combination of virtual
reality (VR) and functional MRI (fMRI)
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 90
91. HCI and cognition
• An improved understanding of the brain in this
interactive and complex context is needed
• Build on theories of cognition as a foundation
for the development of advanced HCI
– As interaction moves towards being reality-based
we move closer to “interacting with reality”, and
thus to general cognition
• Cognitive neuroscience and computational
models of the brain are important tools
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 91
92. Working with fMRI and Virtual Reality
Working with fMRI requires specialist competence and forces us to
deal with powerful magnetic fields and restricted mobility, but the
ability to measure the whole brain is an invaluable research tool.
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 92
93. Thank you
Daniel Sjölie, daniel@cs.umu.se
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 93
94. emoBAN:
Improving Quality of Life
via Psychophysiological Mobile
Computing
Katarzyna Wac, Anind K. Dey
Carnegie Mellon University, HCII
95. emoBAN idea
• “what always speaks silently is the body” (Brown, 1990)
– measure QoL via psychophysiology
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 95
96. emoBAN: Usable Useful Functional
• Body Area Network
– sensors
• vital signs, e.g., ECG, temp, resp
• context, e.g., location, time, activity
– actuators
• feedback via, e.g., AV, tactile, light
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 96
97. Thank you
Dr Katarzyna Wac
Carnegie Mellon University
Human-Computer Interaction Institute
katewac@cmu.cs.edu
www.cui.unige.ch/~wac
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 97
98. Leveraging Psychophysical Data in
Monitoring and Analyzing the
States of Badminton Players
Feng Tian, Wencan Luo, Jing Dai, Reza Naima, Yongmin Cheng, Hongan
Wang, Guozhong Dai
Institute of Software, Chinese Academy of Sciences
Physical Education Collage, Beijing Sport University
Berkeley Institute of Design, UC. Berkeley
China Badminton Team, General Administration of Sport of China
4/20/2010
99. Traces and physical
actions
- Camera, accelerometers
Psychophysical data
- Berkeley Tricorder
ECG, EMG, Pulse Oximetry,
BioImpedance Respiration
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
100. Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010
101. Exploring Quantitative Methods
for Evaluating Sports Games
Veronica Zammitto
Simon Fraser University
Magy Seif El-Nasr
Simon Fraser University
Paul Newton
Electronic Arts Canada
102. Current Study: NBA Live 10
• Eye tracking
– Eye Tech TM3
• Psycho-physiological data
– ProComp5 Infiniti
– EMG (corrugator supercilii and zygomatic major),
EKG (heart rate), GSR and BVP
• Gameplay Telemetry Data
– event-based RPC telemetry collection system
• Triangulation
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 102
103. What was the user
the
Data Triangulation - NBA Live 10 emotional response?
looking at?
Game user experience
Event, when
Was positive?
Where user’s
Was negative?
attention was?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 103
104. Data Triangulation - NBA Live 10
Game user experience
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 104
106. Exploring Psycho-physiological Measures for
the Design and Behavior of Socially-Aware
Avatars in Ubicomp Environments
Johanna Renny Octavia
Hasselt University, Belgium
107. Avatars in Ubicomp Environment
User’s representation Mediator between system and user (companion)
To design a socially-aware avatar that adapts its behavior according to user’s emotion
Any incorrect actions of the avatar can lead to loss of identification of the user
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 107
108. Our informal experiment
ProComp Infiniti device, measured GSR, EMG and BVP
Higher physiological values when playing against a friend
How different social relationships between players influence those values?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 108
109. Thank you
Expertise Centre for Digital Media (EDM)
Web: www.edm.uhasselt.be
Email: johanna.octavia@uhasselt.be
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 109
111. Why I need Psychophysiological
Computing
• Practice environments are becoming
common-place
• Adaptive tailoring of practice is important for
individualized experience/learning
• Need more than traditional adaptations (e.g.,
scaffolding)
– Adapting to engage learner / maintain interest
– Adaptation to reducing “gaming” in environment
– Adaptation to elicit learner affect/arousal (“real”)
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 111
113. Thank you
Bob Wray
Soar Technology
wray@soartech.com, 919.967.5079
Brian Magerko
Georgia Institute of Technology
magerko@git.edu
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 113
114. Guidelines to biosignal
driven HCI
Marjolein van der Zwaag
Philips Research Europe, University of
Groningen
115. Guidelines to biosignal
driven HCI
Marjolein van der Zwaag
Egon L. van den Broek
Joris H. Janssen
116. Outline
• Signal(s) measured: psycho-physiological
• Application domain: Affective computing
• Device used: Nexus 10, mind media
• Guidelines for processing biosignals
– Triangulation
– Identification of users
– Biosignal characteristics
– Temporal construction
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 116
117. Unobtrusive affective computing
Music direct your mood concept
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 117
120. Workshop Discussion Questions
What is brain/body computing good for?
How can we think systematically about this
space?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 120
121. Workshop Discussion Questions
What are the major themes of HCI that can
be explored using brain/body interaction?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 121
122. Workshop Discussion Questions
What specific brain & body sensing modalities
are interesting and how can we use them?
What are most innovative technologies in this
space?
How do these compare with existing practices
(NASA-TLX, Likert-style questionnaires, etc)?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 122
123. Workshop Discussion Questions
How can BCI and PC be defined within an HCI
context?
How is a HCI context different from
neuropsychology, biomedicine or biomedical
engineering?
How does HCI contribute to
psychophysiological computing (or more
specifically neural engineering, bio-
engineering, etc)?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 123
124. Workshop Discussion Questions
If we could measure anything on brain and
body, what would we want to measure and
why?
What cognitive states and emotions have
been investigated with sensors?
What methodologies are best for what
signals?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 124
125. Workshop Discussion Questions
How can accurate physiological,
psychological and neurophysiological
measurements improve HCI?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 125
126. Workshop Discussion Questions
With what software techniques and
hardware technologies could these signals
be optimally processed?
How can software account for the
combination of user actions taken to
accomplish a task versus actions that
would surprise the user, yet optimize their
experience?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 126
127. Workshop Discussion Questions
What are the issues that must be considered
when using brain/body computing for
direct control, passive sensing, adaptation,
etc?
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 127
128. Groups
• How to go from raw data to something
else
– Emotions: TABLE 1
– Data Fusion: TABLE 4
• How to use these things in interactive
interfaces: TABLE 3
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 128
129. BBB Workshop: Bringing Together
BCI& Physiological Computing
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 129
130. Staying connected
Website http://www.eecs.tufts.edu/~agirou01/workshop/
Twitter http://twitter.com/bbbCHI2010
Twitter #bbbCHI2010
Hashtag
Facebook http://www.facebook.com/home.php#/pages/Atlanta-GA/Brain-Body-and-
Bytes-Psychophysiological-User-Interaction-at-CHI-2010/136581171786
Flickr http://www.flickr.com/groups/bbbchi/
Grouplist http://groups.google.com/group/brain-body-interfaces/
Slide Share http://www.slideshare.net/group/brain-body-and-bytes-psychophysiological-
user-interaction
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 130
131. BBB at CHI2010
Brains and Brawn session: Tuesday 11:30-1pm
Making Muscle-Computer Interfaces More Practical
T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, James A. Landay
University of Washington, USA
Microsoft Corporation, USA
A Novel Brain-Computer Interface Using a Multi-Touch Surface
Beste F. Yuksel, Michael Donnerer, James Tompkin, Anthony Steed.
University College London, UK
The Influence of Implicit and Explicit Biofeedback in First-Person Shooter Games
Kai Kuikkaniemi, Toni Laitinen, Marko Turpeinen, Timo Saari, Ilkka Kosunen, Niklas Ravaja,
Helsinki Institute for Information Technology, Finland
Center for Knowledge and Innovation Research, Finland
Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual
Environment
Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, Lars Nyberg
Umeå University, Sweden
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 131
132. THANK YOU
• Keep in touch!
Brain, Body and Bytes: Psychophysiological User Interaction Workshop at CHI 2010 132