. The emergence of learning analytics afforded for the analysis of digital traces of user interaction with technology. This analysis offers many opportunities to advance understanding and enhance learning and the environments in which learning occurs. Existing research has shown how learning analytics can provide contributions to different areas of education such as prediction of student success, uncovering learning strategies, understanding affective states, and unpacking the role social networks in learning. While these results have shown much promise, one critical challenge remains unclear – how learning analytics can help track learning progression and inform assessment especially from the perspective of the 21st century skills. This talk will explore opportunities and challenges for the integration of methods commonly used in learning analytics to analyze different digital traces with methods commonly used in assessment and psychometric research. The paper particularly focuses on open learning environments where analytics-based assessment is rather underexplored in contrast to assessment in specialized (intelligent tutoring) systems where the combined use of data mining and psychometric techniques has been established for some time now.

1. Towards Strengthening Links between
Learning Analytics and Assessment
Dragan Gašević
@dgasevic
ETCPS 2017
November 16, 2017
Iowa City, USA

2. Feedback loops between
students and instructors
are missing/weak!

3. Learning
environment
Educators
Learners
Student
Information
Systems

4. Blogs
Videos/slides
Mobile
Search
Educators
Learners
Networks
Student
Information
Systems
Learning
environment

5. Blogs
Mobile
Search
Networks
Educators
Learners
Student
Information
Systems
Learning
environment
Videos/slides

6. Data in education not new, but…
Real-time insights and longitudinal nature

7. Open ended and general purpose
environments
Different from custom made environments (e.g., ITS)

8. Assessment and feedback for learning

9. Student retention
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Year 1 Year 2 Year 3 Year 4
Course Signals
No Course Signals
Arnold, K. E., & Pistilli, M. D. (2012, April). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd
International Conference on Learning Analytics and Knowledge (pp. 267-270).

10. Current state
Understanding and supporting learning
Moving away from deficit models

11. http://ontasklearning.org
Personalized feedback at scale

12. Analytics-based feedback
Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British
Journal of Educational Technology

13. Analytics-based feedback
Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British
Journal of Educational Technology

14. Analytics-based feedback
Pardo, A., Jovanovic, J. Dawson, S., Gasevic, D., Miriahi. N. (in press). Using learning analytics to scale the provision of personalised feedback. British
Journal of Educational Technology

15. High interest in adoption of
learning analytics

16. Is everything all that shiny?

17. CHALLENGES IN
LEARNING ANALYTICS

18. Challenge
Validity of learning analytics
Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into
Score Meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x

19. Generalizability
Many inconsistent results

20. Open Academic Analytics Initiative
http://nextgenlearning.org/grantee/marist-college

21. Consequentiality
Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers &
Education, 57(4), 2414-2422.
Can teaching be improved?

22. Inconsistent associations of
network centrality on performance
External validity
Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance:
importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge (pp. 314-323). ACM.

23. Purposeful measurement
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions
in predicting learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002
How can we act based on
the count of logins?

24. Purposeful measurement
Who makes decisions about
instrumentation?

25. Structural validity
Do existing* measures correspond to
trace-based measures?
*mostly self-reported
Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic
Performance. Journal of Learning Analytics, 4(2), 113–128.

26. Understanding learning strategies
Detection of learning tactics Detection of learning strategy
Fincham, E., Gašević, D., Jovanović, Pardo, A. (2017). Seeing the Invisible: Learning Analytics to Measure the Effect of Interventions on Learning
Strategies. IEEE Transactions on Learning Technologies (submitted).

27. Progression
How do we measure progression?

28. Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., Siemens, G. (2016). Towards Automated Content Analysis of Discussion
Transcripts: A Cognitive Presence Case,” Proceedings of the 6th International Conference on Learning Analytics & Knowledge (pp. 15-24).
Cognitive presence

29. DIRECTIONS

30. Strengthening links between
learning analytics and assessment

31. Critical dimensions
Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice.
Learning: Research and Practice, 3(2), 63-78. doi:10.1080/23735082.2017.1286142

32. Gašević, D., Dawson, S., Rogers, T., Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of course-specific technology
use in predicting academic success. The Internet and Higher Education, 28, 68–84.
Generalizability
Instructional conditions shape
learning analytics results

33. Purposeful measurement
Extraction of
theoretically informed traces
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

34. Purposeful measurement
Siadaty, M., Gašević, D., & Hatala, M. (2016). Trace-based Micro-analytic Measurement of Self-Regulated Learning Processes. Journal of Learning
Analytics, 3(1), 183–214. https://doi.org/10.18608/jla.2016.31.11

35. Structural validity
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–
419. doi:10.1016/j.learninstruc.2012.03.004
Achievement goal
orientation (2x2)

36. Quid pro quo

37. Quid pro quo

38. External validity
Network centrality with weak ties
creates advantage only
Joksimović, S., Manataki, A., Gašević, D., Dawson, S., Kovanović, V., & De Kereki, I. F. (2016, April). Translating network position into performance:
importance of centrality in different network configurations. In Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge (pp. 314-323). ACM.

39. Measurement of engagement
Joksimović, S., Poquet, O., Kovanović, V., Dowell, D., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., Brooks , C. (2017). How do we measure learning
at scale? A systematic review of the literature. Review of Educational Research (in press).

40. Tracking progression
Trace data based measures of
the crowd-sourced learning skill
E.g., Dreyfus model of skill acquisition
Milligan, S. (2015). Crowd-sourced learning in MOOCs: learning analytics meets measurement theory. In Proceedings of the 5th International Conference
on Learning Analytics And Knowledge (pp. 151-155). ACM.

41. Tracking progression
Topic modeling to extract
Guttman scales from online discussions
He, J., Rubinstein, B. I., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016). MOOCs Meet Measurement Theory: A Topic-Modelling
Approach. Proceedings of the 30th AAAI Conference on Artificial Intelligence (pp. 1195-1201).

42. FINAL REMARKS

43. Opportunities afforded by
continuous streams of data

44. Data science methods can be helpful
but not sufficient
von Davier, A. A. (2016). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement,
54(1), 3-11.

45. Links with methods from assessment
and psychometric needed

46. Tolerance to
some measurement imperfections

47. Generalizability in machine learning is
an open research challenge too

48. Towards Strengthening Links between
Learning Analytics and Assessment
Dragan Gašević
@dgasevic
ETCPS 2017
November 16, 2017
Iowa City, USA