Slides from the LAK17 SHEILA workshop "LA Policy: Developing an Institutional Policy for Learning Analytics using the RAPID Outcome Mapping Approach"

1. Supporting Higher Education to Integrate Learning Analytics
http://sheilaproject.eu/
LA Policy: Developing an Ins4tu4onal Policy for Learning Analy4cs using the RAPID
Outcome Mapping Approach
LAK’17 Workshop
13th March 2017
Yi-Shan Tsai, Dragan Gašević, Pedro J. Muñoz-Merino, Shane Dawson, Maren Scheffel,
Alexander Whitelock-Wainwright

2. Workshop schedule
Time Schedule
13.00-13.05 Welcome & introduc5on
13.05-14.30 Five presenta5ons & discussions
1. Overview & LA in Australia: Prof Dragan Gasevic (University
of Edinburgh)
2. LA in Europe (SHEILA Project): Prof Pedro J. Muñoz-Merino
(Universidad Carlos III de Madrid)
3. SHEILA – group concept mapping: Maren Scheffel (Open
University of the Netherlands)
4. SHEILA – ins5tu5onal interviews & survey: Dr Yi-Shan Tsai
(University of Edinburgh)
5. SHEILA – student survey: Alexander Whitelock-Wainwright
(University of Liverpool)
14.30-15.00 A^ernoon Tea
15.00-16.00 Group work on the assessment of ins5tu5onal readiness
16.00-17.00 LA Policy dra^ing

3. National project to benchmark
LA status, policy and practices for
Australian Universities
Dragan Gasevic
(thanks Shane Dawson for the slides!)

4. Introduction
Massive interest and investment in data and
analy5cs
• Academic performance
• Student reten5on
• Pastoral care
• Academic literacies
• Social networks – collabora5ons

5. Introduction
Yet in terms of wide-scale ins5tu5onal adop5on
there are few examples
Why?

6. Aims
understand current LA practice in Australia
unpack the challenges to institutional adoption
identify practices that can aid the implementation
of LA

7. Approach
2 complementary but separate studies
Study 1 – interviews with senior institutional leaders
Study 2 – concept mapping with LA expert panel

8. Study 1
First study
Interviews with 32 Universities:
Identification of current practice, methods and
approaches
Identification of key drivers for institutions, stage of
development, process for implementation, project leads

9. Study 1
Much interest in LA
Stated organisational priority
LA projects were in the early phases of implementation
and small scale (at time of interview July 2014)
2 distinct clusters across variables such as:
implementation, conceptualisation, readiness
Cluster 1 (n=15) – Solutions focused
Cluster 2 (n=17) – Process focused

10. Strategic capability

11. Strategic capability
1. Solutions focused
› LA to address a pressing need
› Time sensitive
2. Process focused
› Networked and integrated model
› Minimal time pressures
› Innovation and experimentation

12. Study 2
What are the ideal dimensions for long term
sustainable uptake of LA?
Invite to Australian and international LA experts
28 completed the entire concept mapping phases
Prompt: ‘for LA to make a continued impact on
learning and teaching it would need to…’
3 phases – brainstorming; sorting and ranking
of statements

13. Study 2

14. Bringing it together
Study 1 – 2 clusters
Study 2 – 7 clusters
Essentially – how an organisation approaches its
conceptualisation of LA underpins (2 clusters) the
method for deployment and adoption (7 clusters)

15. Systems Model

16. Systems Model

17. Systems Model

18. Bringing it together
Challenges to be addressed:
Leadership awareness
Teams are seldom interdisciplinary
IT driven and system focused
Scale versus understanding
Capabilities and skills deficit.
Over reliance on current research – requires further
validation across different contexts to demonstrate
transportability of models

19. Complexity
Leveraging the outcomes of short term goals for long
term gain
How do we merge both models to gain both short and
long term impact?

20. Conclusion
LA requires alternate models for implementation
and leadership
› Enabling leadership
› Whole of organisation
› Models that are agile and research informed
Working in complexity creates friction
› Embrace the friction – generates innovation

21. Conclusion
A solutions based model can drive change –
but need to be mindful of responding to changing
organsational needs
Process based model can drive innovation and
interest – but need to be mindful of how to scale

22. Conclusion
Combined model framed in the organisational context
› Small, diffuse pockets of innovation to build
capacity and build interest
› View to scale adoption – demonstration of
impact (technical, pedagogical)
› Distributed enabling leadership (complexity
leadership)

23. Conclusion
Any “successful” adoption of LA will be dependent
on an institution’s ability to rapidly recognise and
respond to the organisational culture and the
concerns of all stakeholders.

24. Thank you

25. SHEILA Project
- Overview -
Dragan Gašević,
Yi-Shan Tsai,
Maren Scheffel,
Shane Dawson,
Pedro J. Muñoz-Merino
March, 13, 2017
LAK SHEILA Workshop
Vancouver, Canada
hgp://sheilaproject.eu/

26. SHEILA Partners
• University of Edinburgh (UE) (Coordinator)
• Brussels Educa5on Services (BES)
• Open University Netherlands (OUNL)
• Tallinn University (TLU)
• Universidad Carlos III Madrid (UC3M)
• European Associa5on for Quality Assurance in
Higher Educa5on (ENQA)
• Erasmus Student Network (ESN)

27. Policy development framework
Used by different higher educa5on ins5tu5ons
Recognizes ins5tu5on specific drivers
Promotes learning analy5cs for forma5ve assessment
hgp://sheilaproject.eu/

28. hgp://sheilaproject.eu/

29. Project approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP2

30. Project approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP1 and WP3

31. Project approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP3

32. Project approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP3

33. Project approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP4

34. SHEILA approach
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the
sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28.
WP4

35. Project ac5vi5es
Literature review
hgp://sheilaproject.eu/

36. Tsai, Y-S., Gašević, D. (2016). Adop+on of Learning Analy+cs in Higher Educa+on: State, Challenges, and Policies
(Execu+ve summary). SHEILA Project Report, hgp://bit.ly/sheila_lr_es

37. Project ac5vi5es
Literature review
Interviews with decision makers
hgp://sheilaproject.eu/

38. Project ac5vi5es
Literature review
Interviews with decision makers
Group concept mapping with experts
hgp://sheilaproject.eu/

39. Project ac5vi5es
Literature review
Interviews with decision makers
Group concept mapping with experts
Surveys and focus groups with
students and teaching staff
hgp://sheilaproject.eu/

40. Expected 5meline
Policy development framework (v1) – June 2017
Four ins5tu5onal policies (v1) – June 2017
hgp://sheilaproject.eu/

41. Expected 5meline
Policy development framework (v1) – June 2017
Four ins5tu5onal policies (v1) – June 2017
Policy development framework (v2) – March 2018
Four ins5tu5onal policies (v2) – March 2018
hgp://sheilaproject.eu/

42. Interested in joining?
Associate partners as early adopters of
the policy development framework are welcome
hgp://sheilaproject.eu/

43. Many thanks!
hgp://sheilaproject.eu/

44. Group
Concept
Mapping
Maren Scheffel
Open Universiteit Nederland
maren.scheffel@ou.nl
@m_a_s_c

45. Group Concept Mapping
Phase 1:
Brainstorming
Phase 2:
Sor5ng
Phase 3:
Ra5ng
An essen5al feature of a higher educa5on
ins5tu5on's learning analy5cs policy should be ...

46. Point Map
1 2
3
4 5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39 40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79 80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98 99

47. Cluster Replay Map

48. Cluster Replay Map

49. Cluster Replay Map

50. Cluster Map
1. privacy & transparency
2. roles & responsibili4es
(of all stakeholders)
3. objec4ves of LA
(learner and teacher support)
4. risks & challenges
5. data management
6. research & data analysis

51. Ra5ng Map – Importance
1. privacy & transparency
2. roles & responsibili4es
(of all stakeholders)
3. objec4ves of LA
(learner and teacher support)
4. risks & challenges
5. data management
6. research & data analysis
Cluster Legend
Layer Value
1 5.08 to 5.27
2 5.27 to 5.46
3 5.46 to 5.65
4 5.65 to 5.84
5 5.84 to 6.03

52. Ra5ng Map – Ease
1. privacy & transparency
2. roles & responsibili4es
(of all stakeholders)
3. objec4ves of LA
(learner and teacher support)
4. risks & challenges
5. data management
6. research & data analysis
Cluster Legend
Layer Value
1 3.79 to 4.12
2 4.12 to 4.45
3 4.45 to 4.78
4 4.78 to 5.11
5 5.11 to 5.44

53. Ra5ng Ladder Graph
importance ease
privacy & transparency
privacy & transparency
risks & challenges
risks & challenges
roles & responsibili5es (of all stakeholders)
roles & responsibili5es (of all stakeholders)
objec5ves of LA (learner and teacher support)
objec5ves of LA (learner and teacher support)
data management
data management
research & data analysis
research & data analysis
3.79 3.79
6.03 6.03
r = 0.66

54. Go Zone – Privacy & Transparency
1 7
20 31
43 86
2
10
17 24
45
64 65
88
92
9
15
56
60
74
87
34
69
96
6.08
5.44
3.12
ease
3.83 6.03 6.59
importance
r = 0.45

55. Go Zone – Privacy & Transparency
1 7
20 31
43 86
2
10
17 24
45
64 65
88
92
9
15
56
60
74
87
34
69
96
6.08
5.44
3.12
ease
3.83 6.03 6.59
importance
r = 0.45
2. transparency, i.e. clearly informing students of how their data is collected, used and protected
88. a clear descrip5on of data protec5on measures taken
10. a clear descrip5on of data usage
17. being clear about the purpose for collec5on certain types of data
24. aligned with data protec5on regula5ons (ins5tu5onal, na5onal, interna5onal)
34. to assure that the collected data is used only for the purpose of improving learning and instruc5on
96. an agreement between learners, teachers and policy makers on regula5ng a proper use of data

56. Go Zone – Roles & Responsibili5es
5
38
62
11
19
22
33
39 48
70
91
25
28
37
40
55
61
66
27
47 49
6.08
4.72
3.12
ease
3.83 5.48 6.59
importance
r = 0.26

57. Go Zone – Roles & Responsibili5es
5
38
62
11
19
22
33
39 48
70
91
25
28
37
40
55
61
66
27
47 49
6.08
4.72
3.12
ease
3.83 5.48 6.59
importance
r = 0.26
55. being clear about the purpose of learning analy5cs
61. a clear ar5cula5on of roles and responsibili5es when it comes to the use of ins5tu5onal data
39. to promote broad adop5on of learning analy5cs by specifying suppor5ve regula5ons and case law

58. Go Zone – Objec5ves of LA
3
80
84
99
12
78 98
35
50
93
42
44
58
63 76
79
94
6.08
3.79
3.12
ease
3.83 5.44 6.59
importance
r = -0.12

59. Go Zone – Objec5ves of LA
3
80
84
99
12
78 98
35
50
93
42
44
58
63 76
79
94
6.08
3.79
3.12
ease
3.83 5.44 6.59
importance
r = -0.12
63. to use learning analy5cs for improving the quality of teaching
58. to allow for proac5ve behavior of students towards their educa5on, with sevng personal goals
44. to support learners learning to learn and improve skills
98. to encourage meaningful rela5onships between students

60. Go Zone – Risks & Challenges
18
41
83
26
46
97
53
59
90
95
6.08
4.16
3.12
ease
3.83 5.52 6.59
importance
r = 0.52

61. Go Zone – Risks & Challenges
18
41
83
26
46
97
53
59
90
95
6.08
4.16
3.12
ease
3.83 5.52 6.59
importance
r = 0.52
90. to draw agen5on to what the "big picture" end goal of the data use is
59. to demarcate clearly between different uses of analy5cs in ins5tu5onal sevngs
95. to ensure the benefits to students outweigh the risks
53. to ensure adequate representa5on during high-stakes decision processes
18. integrated with a reward system for the use of evidence-based improvements to teaching and learning
41. to discourage students and faculty from gaming the system

62. Go Zone – Data Management
4
14
89
23
67
71
85
6.08
4.5
3.12
ease
3.83 5.16 6.59
importance
r = -0.54

63. Go Zone – Data Management
4
14
89
23
67
71
85
6.08
4.5
3.12
ease
3.83 5.16 6.59
importance
r = -0.54
23. to define clear rules for collabora5on with other researchers

64. Go Zone – Research & Data Analysis
6
30
57
68
72
75
77
81
8
32
52 73
82
13
51
16
21
29
36
54
6.08
4.24
3.12
ease
3.83 5.08 6.59
importance
r = 0.27

65. Go Zone – Research & Data Analysis
6
30
57
68
72
75
77
81
8
32
52 73
82
13
51
16
21
29
36
54
6.08
4.24
3.12
ease
3.83 5.08 6.59
importance
r = 0.27
57. to assure that instruc5onal interven5ons are based on well-studied and empirically validated analy5cal methods and algorithms
75. to encourage the development of dashboards with meaningful and understandable outcomes
68. to recognise the limita5ons of learning analy5cs
72. to provide clear examples of how to interpret data, especially when there may be interac5ons
54. to enhance the connec5on between the academic and the society and industry, showing what is being done at the ins5tu5ons is related to
the educa5on and training
21. to make explicit maths from innova5ve development / use of analy5cs in individual classrooms to scaling up

66. Thank You!
Maren Scheffel
Open Universiteit Nederland
maren.scheffel@ou.nl
@m_a_s_c

67. Institutional interviews
• Methodology
• Topics of the ques5ons
• Analysis
• Preliminary findings
Researchers:
Yi-Shan Tsai (University of Edinburgh)
Ioana Jivet (Open University of the Netherlands)
Pedro Manuel Moreno Marcos(Universidad Carlos III de Madrid)
Kairit Tammets (Tallinn University)

68. Interviews – methodology
• Purpose: to understand the adop5on of Learning
Analy5cs in higher educa5on ins5tu5ons through
direct conversa5ons with decision makers.
• Sampling: deans, vice-deans, vice-rectors, vice
principals, heads of IT or eLearning, and senior
researchers or project managers of learning
analy5cs.
• Length of interviews: 25 ~ 60 mins
• Period of data collec5on: August 2016 ~ January
2017

69. Interviews – methodology
• 64 interviews
• 51 HEIs
• 16 countries
* Two addi5onal interviews were carried out with a na5onal collabora5ve ICT organisa5on in Netherlands and the
Ministry of Educa5on and Science in Estonia.

70. Interviews – topics
• LA projects, scope, mo5va5ons, and goals
• LA strategy
• Progress and achieved goals
• Challenges
• Ethics and privacy

71. Interviews – analysis
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the
sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9(Winter 2014), 17-28.

72. Interviews – preliminary findings
• Current state of adop4on
NO PLANS
IN PREPARATION
IMPLEMENTED
9 7 5
12
18
THE ADOPTION OF LEARNING ANALYTICS
Ins5tu5on-wide Par5al/ Pilots Data explora5on/cleaning N/A

73. Interviews – preliminary findings
• Step 1 – Map poli4cal context
LA
Learner
Teaching Ins5tu5onal
External
q Data protec5on regula5ons
• 2018 GDPR
• The strictness of exis5ng DP
regula5ons à A stopper
q Pressure to adopt LA
q Exis5ng solu5ons focus on
addressing reten5on
problems.
q No one-size-fits-all solu5ons

74. Interviews – preliminary findings
• Step 2 – Iden4fy key stakeholders
Managers
Students
Teachers
q 25 ins5tu5ons have
established formal
working groups.
q Not all ins5tu5ons have
planned to provide
analy5cs data to students.
• Concerns about
demo5va5on.
Professional
Supports

75. Interviews – preliminary findings
• Step 3 – Iden4fy desired behaviour changes
LA
Learner
Teaching
Ins5tu5onal
q Interven5ons
• E-mail alerts and personal
contacts with students (7
HEIs)
• Teaching reports (1 UK HEI)
q Feedback
• 3 HEIs received posi5ve
feedback
• Low response rates
How can ins5tu5ons engage
students with LA?

76. No defined strategy
Interviews – preliminary findings
• Step 4 – Develop engagement strategy
LA
Digitalisa5on strategies
Teaching & learning strategies
q Consulta4ons with primary
stakeholders
• 23 HEIs
• Surveys
• focus groups
• Workshops
• Annual staff development
conference
• Self-help toolkit
q Visualised dashboard
• 24 HEIs
Danger of being data driven

77. Interviews – preliminary findings
• Step 5 – Analyse internal capacity to effect change
Managers
Students
Teachers Professional
Supports
Ins5tu5onal context
q Technological
resources
q Human
resources
q Funding
q Ins5tu5onal
culture
• Awareness
• Buy-in
• Common
understanding
• Analy5cs skills
Ø Clear value and relevance
Ø Ins5tu5onal priori5es Ø Workload
Ø Teaching styles
Ø Monitoring
Ø Confidence in handling new
technology & analy5cs data
Ø Passive engagement with studies
Ø Lack of interest in certain subjects
Ethics &
Privacy
Trust in
LA

78. Interviews – preliminary findings
• Step 6 – Establish monitoring and learning
frameworks
• Few were able to talk about plans for evalua5on.
• It is difficult to isolate and define the success of a
learning analy5cs project that is implemented
alongside other projects with the same goals to
enhance learning and teaching.
• 12 ins5tu5ons have developed or planned to
develop a policy to accommodate the use of
learning analy5cs.

79. Conclusion
• Success to date:
• Scaled up the ins5tu5onal capacity (step 5)
• Beger understanding of challenges
• Gained experience
Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Produc5vity of the Higher Educa5on Sector - Policy and Strategy for Systems-Level Deployment
of Learning Analy5cs. Canberra, Australia: Office of Learning and Teaching, Australian Government.

80. Institutional survey
• Topics inves4gated
The development of LA
• Current adop5on
• Ins5tu5onal infrastructure
and capacity
• Strategy and policy
• Legal and ethical
considera5ons
• Evalua5on
Self-evalua5on of LA maturity
and Ins5tu5onal readiness
• LA maturity
• LA success
• Culture
• Data and research
capabili5es
• Legal and ethical
considera5ons
• Training and
communica5on

81. Institutional survey
• Distribu4on: six consor5ums in Europe – EADTU
(European Associa5on of Distance Teaching
Universi5es), EUA (European University
Associa5on), HeLF (Heads of e-Learning Forum in
the UK), EUNIS (European University Informa5on
Systems), SNOLA (Spanish Network of Learning
Analy5cs) and the eMadrid Network.
• Survey 4me: September 2016 ~ February 2017
• Returns: 46 ins5tu5ons (response rate: 15%)
European HEIs in general were new to LA and hence did not feel equipped with
sufficient experience or knowledge to answer the survey??

82. Institutional survey
• 22 countries
NO PLANS
IN PREPARATION
IMPLEMENTED 2 13
15
16
The adop4on of LA
Ins5tu5on-wide Small scale N/A

83. Institutional survey
• Mo4va4ons for adop4ng LA
Among the 11 op5ons for mo5va5ons specific to
learning and teaching, the top 5 drivers are:
§ To improve student learning performance (16%)
§ To improve student sa5sfac5on (13%)
§ To improve teaching excellence (13%)
§ To improve student reten5on (11%)
§ To explore what learning analy5cs can do for our
ins5tu5on/ staff/ students (10%)

84. Institutional survey
• Strategy and evalua5on
• Among ins5tu5ons that have implemented LA:
§ There is a strategy: 20%
§ In the process of developing a strategy: 20%
§ No clear strategy: 46.7%
§ Have developed success criteria: 27%

85. Conclusion
• ~ Half of the European countries inves5gated have
HEIs that are either observing the development of
learning analy5cs or have engaged with it
prac5cally.
• The early adopters are likely to scale up the culture
for learning analy5cs in Europe.

86. • SHEILA will assist with the development of learning
analy5cs policies in four ins5tu5ons drawing upon
the output of the project.
• SHEILA will push the development of a long term
learning analy5cs policy agenda and community
among higher educa5on ins5tu5ons.
What next?

87. • Become an associate partner of the SHEILA
project?
• Visit: hgp://sheilaproject.eu/ Yi-Shan Tsai
Research associate
University of Edinburgh
@yi_shan_tsai
yi-shan.tsai@ed.ac.uk

88. What do students want? Towards an
instrument for students’ evalua6on of
quality of learning analy6cs services
Alexander Whitelock-Wainwright, Dragan Gašević, &
Ricardo Tejeiro
A.Wainwright@Liverpool.ac.uk

89. Aims
• Highlight the importance of service quality in
learning analy5cs.
• Develop an instrument to explore student
expecta5ons towards learning analy5cs.
• Steps towards a model of learning analy5cs use.

90. What is service quality?
• Subjec5ve assessment of the degree to which a
service user’s needs or expecta5ons were met
(Parasuraman, Zeithaml, & Malhotra , 2005).
Expecta5
ons
Service
Usage/
Exposure
Percep5
ons
Avtu
de

91. What is service quality? (Contd.)
• Encourages users to use own service over
compe5tors (Parasuraman, Zeithaml, & Berry,
1988).
• Service quality in higher educa5on (Spooren,
Brockx, & Mortelmans, 2013).
• Ideological gap (Ng & Forbes, 2009).

92. Service Quality in Learning
Analy6cs
• Learning analy5cs services designed to support
learning.
• Various stakeholder groups within learning
analy5cs (e.g., students, teachers, managers)
(Clow, 2012).
• Quality indicators of learning analy5cs tools
(Scheffel, Drachsler, Stoyanov, & Specht, 2014).

93. Ques6onnaire Development
• Measuring student expecta5ons of learning
analy5cs.
• Development of ins5tu5onal policies (SHEILA).
• Theories of human behaviour.

94. Ques6onnaire Development
(Contd.)
• Iden5fying themes within past literature (Ifenthaler
& Schumacher, 2016; Sclater, 2016; West, Heath, &
Huijser, 2016):
• Ethics and Privacy
• Meaningfulness
• Agency
• Interven5ons

95. Ques6onnaire Development
(Contd.)
Expecta5
ons
Predic
5ve
Desir
ed
Swan & Trawick
(1980)

96. Ques6onnaire Development
(Contd.)
Created 79
Items
Peer Review
Pilot Study
with 37 Items
Exploratory
Factor Analysis
Redistribute
the
Ques5onnaire

97. Pilot Study Results
• Instrument reduced to 19 items.
• Two factor solu5on for both scales:
• Service expecta5ons:
o Desires scale – 0.88 alpha.
o Predic5ve scale – 0.88 alpha.
• Ethical expecta5ons:
o Desires scale – 0.82 alpha.
o Predic5ve scale – 0.86 alpha.

98. Pilot Study Results (Contd.)

99. Pilot Study Results (Contd.)

100. Pilot Study Results (Contd.)

101. Pilot Study Results (Contd.)

102. Pilot Study Results (Contd.)

103. Future Direc6ons
• Develop the corresponding percep5ons scale.
• Model inten5ons towards using learning analy5cs.
Avtu
des
Social
Norm
s
Inten5ons to Use
Learning Analy5cs
Perceive
d
Behaviou
ral
Control

104. Ques6ons?

105. Workshop schedule
Time Schedule
13.00-13.05 Welcome & introduc5on
13.05-14.30 Five presenta5ons & discussions
1. Overview & LA in Australia: Prof Dragan Gasevic (University
of Edinburgh)
2. LA in Europe (SHEILA Project): Prof Pedro J. Muñoz-Merino
(Universidad Carlos III de Madrid)
3. SHEILA – group concept mapping: Maren Scheffel (Open
University of the Netherlands)
4. SHEILA – ins5tu5onal interviews & survey: Dr Yi-Shan Tsai
(University of Edinburgh)
5. SHEILA – student survey: Alexander Whitelock-Wainwright
(University of Liverpool)
14.30-15.00 A^ernoon Tea
15.00-16.00 Group work on the assessment of ins5tu5onal readiness
16.00-17.00 LA Policy dra^ing

106. • Name
• Organisa5on
• Your experience with LA
• Expecta5ons of the workshop
Tell us about you

107. Assessment of institutional readiness
• Use the ROMA framework to map your progress
towards deploying learning analy5cs.
Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the
sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9 (Winter 2014), 17-28.
1) Map poli4cal context
2) Iden4fy key stakeholders
3) Iden4fy desired behaviour
changes
4) Develop engagement
strategy
5) Analyse internal capacity to
effect changes
6) Establish monitoring and
learning frameworks

108. • Discuss in your group what LA projects and
developments have taken / are taking/ will take
place in your organisa5ons
LA projects in your organisations ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

109. • Is there government/teacher/student/
administrator pressure to deploy LA in your
organisa5on? Or pressure not to deploy it?
1) Map political context ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

110. • Who / which people inside and outside your
organisa5on would be / are the key stakeholders
for the use of LA?
• Who will benefit from the use of LA?
2) Identify key stakeholders ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

111. • What are your reasons for adop5ng LA?
• What benefits is it expected to bring / have
brought?
3) Identify desired behaviour
changes ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

112. • How well prepared do you think your organisa5on is to
deploy LA?
• What needs to be done to achieve the desired changes
iden5fied previously? Areas to consider are ethical/
legal, technical, pedagogical, and cultural.
4) Develop engagement strategy
~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

113. • Does your organisa5on has the capacity to
implement the planned strategy? Do you need to
acquire any skill-sets, funding, or technological
systems?
5) Analyse internal capacity to effect
changes ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

114. • How will you track and evaluate the progress? How
will this process inform the next step of
implementa5on? Will there be any success criteria?
6) Establish monitoring and learning
frameworks ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

115. • Review the 6 ROMA steps. Can you iden5fy any
challenges that may come up in any steps? (Put
your notes in the same boxes.)
Challenges ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

117. • Choose one case to work on as a group or work
individually and share ideas in your group.
Draft a LA policy
1. Audience
2. Purpose
3. Process
§ Data collec5on
§ Data management
§ Stakeholder engagement
§ Evalua5on
4. Policy evalua5on
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

118. • 1. Audience
• Who is this policy for? Consider whose working
ac5vi5es the policy will shape.
• 2. Purpose
• What are your policy objec5ves? Consider your
objec5ves for LA and the changes you seek to achieve.
The policy will set out as the reason for introducing LA.
Audience & Purpose ~5mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

119. • 3. Process
• a) Data collec5on
• b) Data management
• c) Stakeholder engagement
• d) Evalua5on
Process ~15mins
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

120. • 4. Policy management
Policy management ~10mins
Final check-ups
• Will the policy need to state anything that should NOT be done in rela5on to
LA?
• Are there any limita5ons or poten5al challenges and risks about LA that
should be addressed in the policy?
• Can you narrow the policy down to a short list of principles?
Keep your group notes here:
Group 1: hgps://goo.gl/CcGlTk
Group 2: hgps://goo.gl/zrMsyi
Group 3: hgps://goo.gl/3WNr62
Group 4: hgps://goo.gl/WzLmQc

122. • Become an associate partner of the
SHEILA project?
• Visit: hgp://sheilaproject.eu/