1. Automatic Assessment of Collaborative Chat Conversations with PolyCAFe Traian Rebedea 1 , Mihai Dascalu 1 , Stefan Trausan-Matu 1 , Gillian Armitt 2 & Costin Chiru 1 1 - Politehnica University of Bucharest 2 - University of Manchester

12. Technical architecture

13. Utterance evaluation

14. Utterance Evaluation & Threads Utterance Past Future Thread Coherence Future Impact Content In-Degree Out-Degree Social Current Relevance Completeness Centrality

15. Conversation Visualization

16. Utterance Feedback

21. Quality and Consistency of Feedback Validation statement Mean Standard deviation % Agree / Strongly agree The information the system provides me is accurate enough for helping me perform my learning tasks. 3.7 0.52 60% P olyCAFe's feedback is sufficiently accurate to inform my study activities. 3.8 0.88 64% PolyCAFe provides feedback that is useful to my study activities. 3.8 0.85 72% P olyCAFe provides feedback that is relevant to my study activities. 3.9 0.91 72% I trust PolyCAFe to provide helpful feedback. 4.0 0.87 80%

23. Quality of Educational Output Experimental group Control group Improvement over control group Average score for a chat conversation (collaboration + content) 6.80 6.37 6.8% Average importance of the most important 20 concepts 0.194 0.192 1.2% Average number of utterances 351 338 3.8% Average distribution of (implicit and explicit) links between utterances 1.12 0.87 29%

26. Participant Ranking Rank Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Student 3 2 3 - 1 4 Student 4 1 2 3 - 4 Student 5 1 2 4 3 - Student average 2 3 4 1 5 Tutor 1 4 1 5 2 3 Tutor 2 4 2 5 1 3 Tutor average 4 1-2 5 1-2 3 PolyCAFe 4 2 5 1 3

27. Participant Ranking Rankings compared Correlation Precision Average distance Tutors – System 94% 77% 0.23 Students – System 84% 66% 0.43 Tutors – Students 84% 71% 0.40