Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digital Learning Projection - Learning state estimation from multimodal learning experiences.

Presentation prepared for the Doctoral Consortium of the Learning Analytics & Knowledge conference held in March 2017 in Vancouver, Canada

  • Login to see the comments

Digital Learning Projection - Learning state estimation from multimodal learning experiences.

  1. 1. Doctoral consortium LAK17 14th March 2017, Vancouver, Canada Digital Learning Projection Daniele DI MITRI Learning state estimation from multimodal learning experiences.
  2. 2. Pagina 2 Where does learning happen? “Online learning does not happen online, it happens where the learner is. It can’t happen where the learner isn’t.” - Peter Goodyear
  3. 3. Background takeaways 1. Learning data available are not enough 2. Learning happens everywhere: ubiquitous and incidental 3. Sensors can collect multimodal data 4. Machine learning can help analysing data 5. Automatically learn how to estimate “learning states” 6. It can generate feedback and recommend actions Pagina 3
  4. 4. Machine Learning approach Pagina 4 Learning States Multi modal data Learning state estimation
  5. 5. Pagina 5 Proposed Framework Blueprint of Cognitive Inference Back-track the intangible by projecting the tangible.
  6. 6. Input space Pagina 6 Multimodal Data Involuntary EEG / Focus HeartRate Sweat Deliberated Step count Gaze direction Head position Hands position • Observable behavior! • Taxonomy • Several way to divide events – Deliberated vs involuntary – Deterministic vs stochastic (random) – Endogenous vs exogenous – Interactive vs Reflective
  7. 7. Output space Pagina 7 Can be defined as: • Learning Gain • Learning Progress • Learning Performance The spaces can be divided into segments that are the Learning States Affective- Behaviour- Cognition (ABC) Learning Gains Project Open University U.K. 0 10 20 30 40 50 60 70 80 Affective (emotions) Cognition (feelings) Behaviour (actions) Learning State (idea) LS1 LS2 LS3
  8. 8. Research Challenges C1 – Availability of Labels – defining and sampling the output space. C2 – Architectural Design– designing an architecture which collects different multiple heterogeneous modalities C3 – Sensor Fusion – aligning the multimodal data for analysis and prediction C4 – Appropriate Feedback – designing some sort of feedback which maximises learning Pagina 8
  9. 9. Research Questions • Q1 – Learning States Is it possible to represent the learning process into numerable learning states which can be predicted? • Q2 – Data Collection What are the requirements for a multimodal sensor fusion architecture? • Q3 – Data Analysis Do multimodal data streams combined with learner’s action sequences improve the prediction of learning states? • Q4 – Feedback Generation How can we use multimodal data to generate feedback to support learning? Pagina 9
  10. 10. Research Tasks Pagina 10 • T1 – Literature Review on multimodal data for learning • T2 – 1st experiment Learning Pulse • T3 – 2nd experiment WEKIT prototype • T4 – 3rd experiment using WEKIT for Learning States
  11. 11. Mapping tasks with questions & challenges Pagina 11 Q1: Learning States Q3: Data Analysis Q2: Data Collection Literature Review 1st Experiment 2nd Experiment 3rd Experiment C3:Sensor Fusion C2: Architecture Design C1: Availability Labels Tasks Questions Challenges C4: Appropriate Feedback Q4: Feedback Generation
  12. 12. Literature review: Learning Blueprint • Taxonomy of Multimodal Data for Learning • Target: JCAL special issue on MMLA • Look for 1. similar experiments 2. techniques used to collect data 3. multimodal data chosen in related studies 4. learning performance indicators used 5. data analysis approaches used 6. results obtained. Pagina 12
  13. 13. 1st Experiment: Learning Pulse LAK17 Pagina 13 Di Mitri, D., Scheffel, Drachsler, H., M., Börner, D., & Specht, M. (2017). Learning Pulse : a machine learning approach for predicting performance in self-regulated learning using multimodal data.
  14. 14. 2nd Experiment: WEKIT Pagina 14 • Wearable Enhanced Knowledge Intensive Training • Main task now: designing & building prototype • Design of a multimodal architecture • For more info:
  15. 15. 3rd Experiment: First Aid Training with manikins • full monitoring of the learning environment • clear start and end • performance measurement • practical learning over more cognitive ones • close set of actions. • High practical significance! Pagina 15
  16. 16. 3rd Experiment: Option B) Indigenous shelter building • Proposal of collaboration • Land & Learning Indigenous Technology Experience (LLITE) with Canadian partners • Fit for WEKIT AR technology • 3D holograms can be used • Case scenario must be defined well Pagina 16
  17. 17. Unique Selling Points • Generative approach vs data already available • Machine learning approach learn from history – Input space: multimodal data – Output space: notion of Learning State • “What to do with data?” vs “what data can be collected?” • Real-time data collection & analysis vs ex-post • Random Action sequences • AR technologies Pagina 17
  18. 18. S.W.O.T. Pagina 18 Strengths Weaknesses Data driven Not enough data Opportunities Threats Personalisation Garbage-in-garbage-out
  19. 19. Ethics & Privacy - ‘2084’ • Out of scope of my PhD • But still really relevant! • Great opportunity vs great risks • Expand the LA Framework • Follow-up of the project Pagina 19
  20. 20. Q&A Thanks for listening! Daniele Di Mitri @dimstudi0 Pagina 20