Doctoral consortium presentation at the 17th Conference in Artificial Intelligence in Education in Poznań, Poland 2019 Abstract: In this doctoral consortium paper, we introduce the CPR Tutor, an intelligent tutoring system for cardiopulmonary resuscitation (CPR) training based on the analysis of multimodal data. Using a multi-sensor setup, the CPR Tutor tracks the CPR execution of the trainee and generates automatic adaptive feedback to improve the trainee's performance. This research work is part of a PhD project entitled ``Multimodal Tutor: adaptive feedback from multimodal experience capturing'', a project which investigates how to use multimodal and multi-sensor data to generate personalised feedback for training psycho-motor skills at the workplace or during medical simulations. In the CPR Tutor, we use Microsoft Kinect and Myo to track trainee's body position and the ResusciAnne QCPR manikin to get correct CPR performance metrics. We then use a validated approach, the Multimodal Pipeline, for the collection, storage, processing, annotation of multimodal data. This paper describes the preliminary results obtained in the first design of the CPR Tutor.

1. Detecting Medical Simulation Mistakes with
Machine Learning and Multimodal Data
Daniele DI MITRI^
Jan SCHNEIDER*, Marcus SPECHT^, Hendrik DRACHSLER^*
SafePAT CM - 20180206
^ Open University of The Netherlands
* DIPF — German Institute for International Educational Research
Poznań, Poland 26th June 2019 – Artificial Intelligence in Medicine

2. AI in Medicine vs AI in Education
Term AI in Medicine AI in Education/LA
Subject Patient Learner
Hypothesis Died, treatment work
Grade, correct answer,
mistake
Feedback to
subject
Not needed Always needed
Data point One patient One learner, Time update
Models Predictive To generate feedback

3. Learning through Medical Simulations

4. Intelligent Tutoring Systems (ITS)
ITS were mostly developed for
computer desktop interfaces
easy to distinguish <who did what?>

5. Multimodal Learning Analytics (MMLA)
Learning Analytics approach
Measurement, collection, analysis and
reporting of data about learners
+
Multimodal data and interfaces
=
More authentic representation of
the learning process

6. Multimodal Tutoring Systems

7. Why CPR training?
• CPR can be taught singularly to
one learner
• CPR is a highly standardized
procedure
• CPR has clear and well-defined
criteria to measure the quality
• CPR is a highly relevant skill

8. Multimodal Tutor for CPR
RESEARCH QUESTIONS
RQ1 Validation: how accurately
can we detect common
mistakes in CPR training with
multimodal data?
RQ2 Additional mistake
detection: can we use
multimodal data to detect
additional CPR training
mistakes?
EXPERIMENTAL SETUP

9. Selected CPR Performance indicators
Indicator Ideal value
Compression rate 100 to 120 compr./min
Compression depth 5 to 6 cm
Compression release 0 - 1 cm
Arms position Elbows locked
Body position Using body weight
not measured by the ResusciAnne manikin

10. Study procedure
• Experiment at Uniklinik Aachen, Germany
• Collected data from 14 experts (medical
students)
• Data used 22 sessions from 11 participants
~5200 chest compressions, ~50 attributes
• Technological prototypes used, part of the
Multimodal Pipeline
– LearningHub for data collection and storage
– Visual Inspection Tool for data annotation
– DataFlow for processing the data
• Trained 5 Recurrent Neural Networks (LSTM

11. Methodology: the Multimodal Pipeline
Target Classes Accuracy
ClassRate 0.8650
ClassDepth 0.7791
ClassRelease 0.7220
ArmsLocked 0.9344
BodyWeight 0.9781
1. Multimodal Learning Hub
(Schneider et al., 2018)
data collection, data storing
2. Visual Inspection Tool
(Di Mitri et al., 2019)
data annotation

12. Input space
Time slice
t1 t2 t3 t4 t5 t6 t7 t8
Time-bins (s=8)
a0
a1
a2
a…
aq
Attributes
(Q=41)
i1
i2
i3
i3
i…
in
Intervals
(N=5254)
Resampled
Time-series
Training sample
3D tensor of shape
(5254 intervals, 41
attributes, 8 time bins)

13. Hypothesis Space
Indicator className ClassValues
Compression rate classRate 0 < 100; 100< 1 <120; 2 > 100
Compression depth classDepth 0 < 5; 5 < 1 <5; 2 > 6
Compression release classRelease 0 – incorrect; 1 – correct
Arms position armsLocked 0 – incorrect; 1 – correct
Body position bodyWeight 0 – incorrect; 1 – correct

14. Mistakes Variability among subjects
SafePAT CM - 20180206

15. LSTM configuration
LSTM hidden
units: 128
Input shape: (8, 41)
(time bins, input dim)
Softmax output
shape: 2 – 30 1 2
h1 h2 h3 h4 h5 h6 h7 h… h128
x1,1 x2,1 x3,1 x4,1 x..,1 x8,1
Training set
66.6%
(3520, 8, 41)
Test set
33.4%
(1734, 8, 41)

16. Results classes
Target Classes Accuracy
ClassRate 0.8650
ClassDepth 0.7791
ClassRelease 0.7220
ClassRate acc 0.8650 ClassDepth acc 0.7791 ClassRelease acc 0.7220

17. Results manually annotated classes
ArmsLocked
acc. 0.9344
ArmsLocked
acc. 0.9781

18. Conclusions Future works
• In the first study we focused on modelling CPR mistakes with the
Multimodal Tutor for CPR
• However in learning, feedback to the learner is the end goal
• Next study will try to implement the models in a real-time feedback
system
– We need a model to detect compressions
– Multimodal runtime feedback engine (architectural challenge)
– We need to know how to send the feedback
• The Multimodal Pipeline is a generic approach for modelling
learning behavior
• We need to validate it in more practical learning scenarios

19. SafePAT CM - 20180206