Here the slides of the workshop "Multimodal Machines" presented at the JTEL Summer School 2017 in Aveiro, Portugal. For more info about it http://www.prolearn-academy.org/Events/summer-school-2017/index_html

1. Workshop @ JTELSS17
12th October 2017 – Aveiro, Portugal
Multimodal Machines
Daniele DI MITRI & Jan SCHNEIDER

2. What is multimodality?
Pagina 2
Group assignment: divide into groups and find a picture which best
describe the concepts of Multimodality.

3. What is multimodality?
Pagina 3
https://www.youtube.com/watch?v=rn9V0cN4NWs https://www.youtube.com/watch?v=3kEHg7cYlP0

4. Pagina 4
How does multimodality work in learning?

5. Multimodality for robots
Pagina 5
Group assignment:
equip your robot with
multimodal capabilities

6. They encode
messages
Pagina 6
Multimodality for computers
e.g. 3D models in AR
They decode
sensor inputs
e.g modern AR
headsets like
Hololenes

7. Observability
Line
INPUT SPACE
OUTPUT SPACE
Observable dimensions,
can be tracked
with sensors
Unobservable
dimensions,
require human interpretation
assessment
Di Mitri, D., Drachsler, H., Specht, M. (2017) From signals to knowledge. A conceptual model for
multimodal learning analytics. In press.

8. Multimodal data tree
Pagina 8

9. Famous multimodal
experiments

10. Put-that-there
Pagina 10
Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface (Vol. 14, No. 3,
pp. 262-270). ACM.ISO 690
https://www.youtube.com/watch?v=RyBEUyEtxQo

11. Emotion sensors go to school
Pagina 11
Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009,
July). Emotion Sensors Go To School. In AIED (Vol. 200, pp. 17-24)

12. Multimodal action based assessment
Pagina 12
Worsley, M., & Blikstein, P. (2013, April). Towards the development of multimodal action based
assessment. In Proceedings of the third international conference on learning analytics and
knowledge (pp. 94-101). ACM.ISO 690

13. Analytics meet Patient Manikins
Pagina 13
Martinez-Maldonado, R., Power, T., Hayes, C., Abdipranoto, A., Vo, T., Axisa, C., and
Buckingham-Shum, S. (2017) Analytics Meet Patient Manikins: Challenges in an Authentic
Small-Group Healthcare Simulation Classroom. International Conference on Learning Analytics
and Knowledge, LAK 2017, 90-94.

14. Multimodal experiments
@OUNL

15. Learning Pulse – are you in the Flow?
Pagina 15
(Csikszentmihalyi, 1972)
Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning
Pulse: a machine learning approach for predicting performance in self-regulated learning using
multimodal data.ISO 690

16. Presentation Trainer
Pagina 16
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015, November). Presentation
trainer, your public speaking multimodal coach. In Proceedings of the 2015 ACM on
International Conference on Multimodal Interaction (pp. 539-546). acm.ISO 690
https://www.youtube.com/watch?v=ElB6OvbL8fA

17. WEKIT.eu project
Pagina 17
https://www.youtube.com/watch?v=RkGXdpSB_AM

18. Calligraphy Learning
Pagina 18

19. Q&A
Thanks for listening!
Daniele Di Mitri & Jan Schneider