This is the final presentation of my PhD defence which took place on the 4th September 2020 at the Open University of The Netherlands. Abstract This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor}, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with a machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human-annotated samples. The main contributions of this thesis are a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks. Link to youtube presentation https://youtu.be/b1kDSORpV8A Link to the PhD Thesis manuscript https://research.ou.nl/en/publications/the-multimodal-tutor-adaptive-feedback-from-multimodal-experience

2. The Multimodal Tutor
Adaptive Feedback from Multimodal Experiences
4th September 2020
Daniele Di Mitri

3. How do people learn nowadays?
The Multimodal Tutor – Motivation 3

4. Multimodal Learning Analytics (MMLA)
4
+
Learning Analytics approach
"measurement, collection, analysis and reporting of
data about learners”
The Multimodal Tutor – Motivation
data from multimodal and multi-sensor
interfaces
= a more accurate representation of the
learning process

5. PhD journey
I. Exploratory mission
• Learning Pulse study
II. Map of Multimodality
• Literature survey, conceptual model, the big five challenges
III. Preparation of the Navy
•Visual inspection Tool, Multimodal Pipeline, mistake detection
IV. Conquest mission
•Real-time feedback with the CPR Tutor
5The Multimodal Tutor – Structure

6. “Are you in the Flow?” (CHAPTER 1)
6Chapter 1 – Learning Pulse

7. What is multimodal data? (CHAPTER 2)
7Chapter 2 – From Signals to Knowledge

8. Titel van de presentatie 8
s
Input space
Hypothesis space

9. Multimodal
Learning Analytics
Model
(CHAPTER 2)
Titel van de presentatie 9

10. The Big Five Challenges (CHAPTER 3)
10
Multimodal
Feedback
Loop
Chapter 3 – The Big Five
• Multimodal data is messy, most
studies stand at the level of
data geology
• No clear picture how MMLA can
support learning
• We identify five big challenges.

11. Emotiv
Empatica
Myo
Leap
Motion
Data collection challenge
Modalities Sensors Controllers
Multimodal
Learning
Hub
example of serialisation
of Myo data in JSON
External research – The Multimodal Learning Hub

12. Data annotation challenge (CHAPTER 4)
Chapter 4 – Read Between the Lines 12

13. The Multimodal Pipeline (CHAPTER 5)
13Chapter 5 – The Multimodal Pipeline
• To the reduce data manipulation over-
head and focus on the data analysis
• A technological framework composed
by generic solutions for the big five
challenges
• The aim is to support researchers in
setting up experiments more quickly

14. Cardiopulmonary Resuscitation (CPR)
Why CPR?
• It’s taught singularly to one learner
• It is a highly standardized procedure
• It has clear and well-defined criteria to
measure the quality
• It is a highly relevant skill
14Chapter 6 – Detecting CPR Mistakes

15. CPR mistake detection (CHAPTER 6)
15
Indicator Ideal value
Compression rate 100 to 120 compr./min
Compression depth 5 to 6 cm
Compression release 0 - 1 cm
Arms position Elbows locked
Body position Using body weight
assessed by the ResusciAnne manikin
not measured by the ResusciAnne manikin Chapter 6 – Detecting CPR Mistakes

16. CPR Tutor – 1st iteration (CHAPTER 6)
Hardware setup:
• Microsoft Kinect v2
• Myo armband
• Laerdal ResusciAnne manikin
Dateset collected:
• ~5500 chest compressions from 14 experts
• Each CC tagged with 5 classes
Trained 5 neural networks to classify CPR mistakes
Chapter 6 – Detecting CPR Mistakes 16

17. CPR Tutor – 2nd iteration (CHAPTER 7)
Chapter 7 – Real Time Multimodal Feedback 17
Lock your
arms!
Use your
body weight!
Release
the
compression!
*Metronome
sound 110bpm*
Check
compression
depth!

18. Real-time feedback architecture (CHAPTER 7)
18
Sensors
CPRTutor
C# app
SharpFlow
Python3
TCP
client
TCP
server
Chunk
(1 CC – 0.5
sec)
Classification ML models
ClassRate
ClassRelease
ClassDepth
ArmsLocked
BodyWeight
Feedback
Chapter 7 – Real Time Multimodal Feedback

19. Positive effect of feedback (CHAPTER 7)
Chapter 7 – Real Time Multimodal Feedback 19
error rates drops shortly after
the feedback is fired

20. Conclusions
1) Sensors cannot reason about the data they
collect. Machine learning & human annotation
can help for automatic reasoning.
The Multimodal Tutor – Conclusions 20
2) Multimodal Tutors can support the learners
when the human instructor is not available.
3) Multimodal Tutors can help closing the
feedback loop using MMLA.
4) The Multimodal Tutor is an example that
paves the way to human-AI cooperation.

21. Thanks for your attention.