Please tweet to us using: @Affectiva and #EmoDev16 Key Websites: Affectiva: http://affectiva.com Developer Portal: http://developer.affectiva.com Affectiva Demo: http://go.affectiva.com/affectiva-demo Emotion AI Developer Day brought together the largest remote conference of emotion recognition developers in the world, including Affectiva staff, affective computing thought leaders and companies offering complementary technologies. Emotion AI Developer Day provided opportunities for attendees to learn, as we as to help shape the future of Affectiva. Find us on: Facebook: https://www.facebook.com/Affectiva/ Twitter: https://twitter.com/Affectiva LinkedIn: https://www.linkedin.com/company/affectiva_2

1. Socially Assistive Robots, Educational
Tutoring, and Affective Computing
Sam Spaulding
MIT Media Lab

2. Social Robots
Inventing our Future while Learning about Ourselves
Robotic
Engineering
Artificial
Intelligence
Studies of Human
Behavior
Human-Robot Interaction
Educational
Companion
Aging-in-Place HelperAssembly-mate

3. Socially Assistive Robots
(SAR)
Socially Assistive Robots are designed to
leverage their social and affective attributes to
provide social support to people in order to
sustain engagement, motivate, coach, monitor,
educate, or facilitate communication &
teamwork for improved outcomes.

5. Embodiment matters!
Robots produce
higher learning
gains
Robots are more able
to form long-term
bonds
Robots produce
greater
compliance
Leyzberg et al. (2012)
Bainbridge et al. (2012)
Kidd & Breazeal (2008)
Compared to screen-based representations...

6. Intelligent Tutoring Systems
Use domain-general
inference and modeling
algorithms
Have been extensively
tested in real-world
environments over long
periods of time

7. Intelligent Tutoring SystemsRobotic Tutors
The best of both...
Social Presence
Onboard Sensing
Agent-based Interaction
Adaptive Personalization
Data-driven Student Models
Physically embodied robot tutors that:
sense and understand emotions
build models of students based on affective data
act intelligently as a result of the model info

8. Affect-aware Student Models
for Robot Tutors
- Child and Robot
interact through shared
“Storymaker” game
context
- Robot framed as peer,
periodically asks child to
demonstrate reading ability
(to be presented at AAMAS ’16)

9. Knowledge State Estimation
- A key challenge for adaptive,
computational tutors is -
“how to personalize
experience?”
- In order to provide
personalized curriculum, the
tutor must first determine
students’ initial knowledge state
Task
Difficulty
Student Ability
Boredom
Overwhelmed
Flow: Optimal
Challenge
“Flow”

10. Bayesian Knowledge Tracing
(BKT)
Bayesian Knowledge Tracing
(BKT)
P(Li
t)
Correctt
P(Li
t+1) P(Li
t+2)
... ...
Correct t+1 Correct t+2
Background Model Domain Evaluation Contributions
Sparse channel for knowledge,
but widely studied
Each ‘traced’ skill
modeled by an HMM

11. Affective Bayesian Knowledge
Tracing
Affective data drawn from 5s before question asked to
5s after question answered

12. Affect-aware Student Models
for Robot Tutors
- 25 children came and
played with Dragonbot
- 13 children did same
interaction with Tablet
only
- Experiment conducted in
Summer 2014, affective
analysis completed 6mo.
later
(to be presented at AAMAS ’16)

13. Are Children More Emotionally Expressive
When Interacting with A Robot?
... ... ... ...
} {
Avg Smile: 18
Avg BrowFurrow: 7
Avg BrowRaise: 62
Avg LipDepress: 4
AvgValence: -24
Avg Engage: 67
} {Median
Filter
} {Mean
Metric
Value
Session Footage
Raw Affdex Measurements
Median-smoothed Affdex Data
Subject Bag
Average MetricValue
over Interaction

14. Are Children More Emotionally Expressive
When Interacting with A Robot?
*
*
*
* p < .05
n=25 “Robot” condition
n=13 “Tablet condition

15. Training and Evaluating Skill Models
BKT and Aff-BKT models
trained for 4 Skills via
Expectation Maximization
Model classes evaluated via
log-likelihood comparison, with
Leave-one-out cross-validation

16. Does Including Affective Data in Training
Yield Better Models?
...
}Affective + Right/Wrong
Training Data
Subject Bag
Expectation
Maximization
{
P(Li
t)
CorrecttSmilet
P(Li
t+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...
P(Li
t)
CorrecttSmilet
P(Li
t+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...
}
Training Data Trained Model Trained Model
Subset
{Model
Subset
P(Li
t)
CorrecttSmilet
P(Li
t+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...
P(Li
t)
Correctt
P(Li
t+1)
...
Correctt+1
...
Held-out Right/Wrong
Test Data, D'
Likelihood of Model,
Given Test Data D’
}
D’
P(D’| θaff)
θaff = maxθ P(Daff|θ)
θaff, a subset of θaff, containing only
BKT model parameters
θaff
^
^
^
Daff

17. Does Including Affective Data in Training
Yield Better Models?
[
[
[
[
Exact-Correct
BKT Aff-BKT BKT Aff-BKT BKT Aff-BKT BKT Aff-BKT
First-Letter Length Last-Letter

18. Does Including Affective Data in Training
Yield Better Models?
p < 1.0 x 10-4
*
*
**
*
*
**
**p < 1.0 x 10-5
p < 1.0 x 10-6
*
***

19. Are children more
emotionally expressive
when interacting with
robots?
Can we leverage emotional
expression data to create
better student models?
Two main questions:

20. Are children more
emotionally expressive
when interacting with
robots?
Can we leverage emotional
expression data to create
better student models?
Two main questions:

21. Intelligent Tutoring SystemsRobotic Tutors
The best of both...
Social Presence
Onboard Sensing
Agent-based Interaction
Adaptive Personalization
Data-driven Student Models
Physically embodied robot tutors that:
sense and understand emotions
build models of students based on affective data
act intelligently as a result of the model info

22. Meet Tega!

23. Tega: a “real-world ready” social robot!
Student models allow us to personalize curricular
content. How do we personalize affective support?
Affective Personalization

24. Affective Personalization of a
Social Robot Tutor for SSL
(presented at AAAI ’16)
•What’s it all about?
•Children learning Spanish as a second language with a
robot companion
•An Integrated System
•Tega Robot
•Custom Educational Game
• Affdex affective sensor
•All synchronized and coordinated through a ROS-
based cognitive architecture
•The Study
•Long-term, in-the-wild, fully autonomous interaction
•Personalization of affective response

25. Integrated System: Software
Game:
- Unity-based sprite game
- 8 sessions of content + review
- Fully autonomous play
- Virtual “instructor” character, robot as peer
Affdex phone:
- Real-time detection of facial
expressions
- Valence / Engagement used as
reward to RL algorithm

26. Educational Context
Robot again framed as peer,
with ‘bilingual’ Toucan as
teacher
As the student plays through
the game, the robot provides
affective support through
verbal + nonverbal actions

27. Reinforcement Learning on Affective Data
SARSA Algorithm
Reward = .4(Engagement) +.6( (Valence+100) )
2
State Space = 3 x 2 x 2 x 2 = 24 states total
Neg./Neut./Pos.
Valence
Hi/Lo
Engagement
On/Off
Task
Right/Wrong
Last Question
Action Space = 3 x 2 + No-action = 7 action classes total
ɛ-greedy algorithm, with ɛ decreasing across sessions

28. Sample Actions

29. Timeline
• 8 Weeks of In-Class deployment
• 1 Pre-test Session
• 6 Study Sessions (part review, part new content)
• 1 Post-test Session

30. Affective Personalization
***
**
*
*
***

31. Affective Personalization
*
Appropriate affective responses are
critical to avoiding “novelty effect”

32. Did they actually learn?
*

33. Contributions
• Novelty of current study:
• Long-term interaction (8 sessions)
• Fully autonomous social robot (Tega)
• In-the-wild experiment (inside a classroom)
• Affective personalization (Affdex)
• Age of participants (3-6 yrs)
Social robot personalized its affective response, thus
increasing children’s valence during long-term
interaction.

34. Looking ahead…
Old challenges are becoming
tractable: sensing,
deployment, robust systems.
New challenges are conceptual
and computational - i.e. how to
fully integrate emotions into an
agent’s cognition

35. Collaborators and Supporters
This research was supported by the National
Science Foundation
(NSF) under Grant CCF-1138986 and
Graduate Research
Fellowship Grant No 1122374.
Luke Plummer JinJoo Lee
Goren Gordon Jacqueline KoryProf. Cynthia Breazeal