This presentation explores shortcomings of learning analytics for the wide adoption in educational organisations. It is NOT about ethics and privacy rather than focuses on shortcomings of learning analytics for teachers and students in the classroom (micro-level). We investigated if and to what extend learning analytics dashboards are addressing educational concepts. Map opportunities and challenges for the use of Learning Analytics dashboards for the design of courses, and present an evaluation instrument for the effects of Learning Analytics called EFLA. EFLA can be used to measure the effects of LA tools at the teacher and student side. It is a robust but light (8 items) measurement to quickly investigate the level of adoption of learning analytics in a course (micro-level). The presentation concludes that Learning Analytics is still to much a computer science dicipline that does not fulfill the often claimed position of the middle space between educational and computer science research.

1. Fighting level 3:
From the LA framework to LA practice on
the micro-level
Hendrik Drachsler, @hdrachsler
LASI Spain, JULY 4-5, 2017
CC-by: https://www.flickr.com/photos/juandelpozo/6110435588/

2. 3
WhoAmI
Full Professor of
Educational Technologies and
Learning Analytics
School of Computer Science
Goethe University Frankfurt
German Institute for international
Educational Research (DIPF)
Frankfurt am Main, Germany
Welten Institute
Research Centre for Learning,
Teaching and Technology
Open University, Heerlen, The
Netherlands

3. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 3 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?

4. 4
Greller, W. & Drachsler, H. (2012). Turning Learning into Numbers. Toward a Generic Framework for
Learning Analytics. Journal of Educational Technology & Society.
http://ifets.info/journals/15_3/4.pdf
Learning Analytics Framework

6. Let‘s not forget, LA is for teachers and
learners
Learner’s and teachers are most
of the located on the micro-level
with their needs and skills

7. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 7 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?

8. Sophistican model
Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of
the Higher Education Sector – Policy and Strategy for Systems-Level Deployment of
Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian
Government. Retrieved from http://solaresearch.org/Policy_Strategy_Analytics.pdf
LA Sophistication Model
1
2
3
4
5
Who of you arrived at
level 3 with their
organisation?
How long you are already
in level 1 and 2?

9. Maturing, Roll-Out, Uptake?
VERY URGENT for uptake of LA
• Ethics & Privacy (Dragan keynote)
• Stakeholder needs & Competences (Hendrik)
• LA supported learning design (Hendrik)
LESS URGENT for uptake of LA (but lots of fun)
• Infrastructure
• Data (Katrien keynote)
• Dashboards (Katrien keynote)

10. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 10 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?

11. STOTA Dashboards
1. Verbert, et al., (2013) Learning analytics
dashboard applications. American
Behavioral Scientist 57(10), 1500
2. Verbert, et al. (2014) Learning
dashboards: an overview and future
research opportunities. Personal and
Ubiquitous Computing 18(6), 1499
3. Yoo, Y., Lee, H., Jo, I.H., Park, Y.:
Educational dashboards for smart
learning: Review of case studies. In:
Emerging Issues in Smart Learning, pp.
145. Springer (2015)
4. Schwendimann, et al., Perceiving learning
at a glance: A systematic literature review
of learning dashboard research. IEEE
Transactions on Learning Technologies
(2016)
5. Bodily, R., Verbert, K.: Trends and issues in
student-facing learning analytics
reporting systems research. In: Proc. of
LAK'17. pp. 309{318. ACM (2017)
For the question of uptake we have been
investigating 26 out of 95 papers that
have grounding in learning sciences:
According to which educational concepts
are learning analytics dashboards
designed?

12. Educational concepts mentioned in LA dashboard
papers
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
18
10
1
7
5
12

13. Competences mentioned in LA dashboard papers
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
34
8
12
5
13

14. Relation between educational concepts and
competencies

15. Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
Reference frames mentioned in LA
dashboard papers
24
16
10

16. • Design of LA Dashboard are technology driven (only 26 out
of 95 papers use edu. concepts)
• Mostly to support awareness and reflection
• According to McAlpine & Weston, refection is a mechanism
to facilitate learning & teaching not an goal in itself.
• More than half of the dashboards rely on SRL but none of
the tool really support the cycle:
fore-thought -> planning -> monitoring -> self-evaluation.
• Social framing is more common than achievement framing
According to which educational concepts are
learning analytics dashboards designed?
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.

17. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 17 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?

18. Learning Analytics supported
Learning Design
Three main opportunities:
1. Using LA indicators for on demand measures
of the learning design
2. Intervening during the runtime of a course
due to LA indicators
3. Increasing students learning outcomes and
interventions through earlier interventions
with LA indicators

19. Opportunities and challenges for using LA in LD
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and
challenges in using Learning Analytics in Learning Design. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.

20. Opportunities and challenges for using LA in LD
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and
challenges in using Learning Analytics in Learning Design. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.

21. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 21 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?

22. Learning Analytics Readiness Instrument (LARI)
Oster, M., Lonn, S., D. Pistilli, M., and Brown., M., G. (2016). The learning analytics
readiness instrument. LAK '16. ACM, DOI: https://doi.org/10.1145/2883851.2883925A

23. Evaluation of the effect of LA
• If you want to measure uptake
of LA you need an instrument to
explore progress and adoption
• It needs to be a light way
instrument as students and
teachers don’t fill overwhelming
long surveys again and again.

24. We were looking for something quick
and robust like ...
... the SUS scale.
But than dedicated
for LA.

25. Scheffel, M., Drachsler, H.,
Stoyanov, S., and Specht, M.
(2014). Quality Indicators
for Learning Analytics.
Educational Technology &
Society, 17 (4):117–132.
Scheffel, M., Drachlser, H.,
Specht, M. (2015). Developing
an evaluation framework of
quality indicators for learning
analytics. LAK ’15, NY, USA.
ACM.
Scheffel, M., Drachsler, H., de
Kraker, J., Kreijns, K.,
Slootmaker, A., Specht, M.
(2017). Widget, widget on the
wall, am I performing well at
all? IEEE Transactions on
Learning Technologies, 10
(1):42–52.
Evaluation Framework for Learning Analytics (EFLA)

26. Evaluation Framework for Learning Analytics (EFLA)
Scheffel, M., Drachsler, H., Kreijns, K., de Kraker, J., Specht, M. (2017). Widget, widget as you lead, I am
performing well indeed!: Using results from an exploratory offline study to inform an empirical online
study about a learning analytics widget in a collaborative learning environment, LAK ’17, NY, USA. ACM.
Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining
Validity and Reliability of the Evaluation Framework for Learning Analytics. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.

27. Evaluation Framework for Learning Analytics (EFLA)
Quick evaluation of LA
tools in micro-settings.
We are looking for
partners to try EFLA and
further develop it.
A great outcome would be
N numbers of LA case
studies that applied EFLA
and share their
experiences.
Can we work towards an
EFLA score like the SUS
score?http://www.laceproject.eu/evaluation-
framework-for-la/

28. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 28 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
CONCLUSIONS
for Uptake of LA
LA is about
learning!
LA is about
learning!

29. The problem for uptake of LA
and mastering level 3 is ...
Educational
Science
Computer
Science
Learning
Analytics
MIDDLE SPACE

30. The solution, LA as a real middle space
Educational
Science
Computer
Science
Learning
Analytics
MIDDLE SPACE

31. 31
@hdrachsler
drachsler@dipf.de | hendrik.drachsler@ou.nl
linkedin.com/in/hendrikdrachsler
Many thanks for the invitation!
Questions now or later Looking for PhD or
PostDoc positions
in computer science
on Learning Analytics?
Get in contact!