ASCILITE 2019 Presentation

1. A three tier model to promote
the institutional adoption of
learning analytics
Steven Warburton, University of New England, NSW
Irina Elgort, Victoria University of Wellington, NZ
Derek White, Victoria University of Wellington, NZ
ASCILTE 2019
2nd to 5th December
Singapore

2. Out of scope: deep dive technical
Lundqvist, K. and Warburton, S. (2019). Visualizing
Learner Flows in MOOCs. Proceedings LWMOOCS VI, 23-
25 October, 2019, Milwaukee, WI, USA.
Commend everyone to the Journal of Learning Analytics
https://epress.lib.uts.edu.au/journals/index.php/jla
In scope: organizational drivers
Why invest? Who owns analytics at the University?
Do we need a business case for learning analytics?
Solving the people, process and technology bit.

3. Context
• Victoria University of Wellington is a
mid-size traditional, devolved research-
intensive organisation with nine faculty
and a student body of around 23,000
learners.
• The university maintains a strong
commitment to its’ bicultural heritage
and the advancement of Māori and
Pasifika student pathways into, and
through, tertiary education.

4. • Retention versus acquisition: unfavourable
demographics counter a growth strategy;
• Attrition rates for Māori and Pasifika are about
24% in the first year and 40% over three years;
• First year attrition rate was relatively stable
across the four cohorts between 18.8% and
19.7%;
• 2-3% increase in retention financially
attractive.
The retention challenge

5. Bigger Picture

6. Wide interest in what
benefits [learning]
analytics could
unlock.

7. TOP
BOTTOM
Leadership led; large scale; high
tech, low staff uptake.
Emergent; strong consultation;
scaling up difficult; communicating
value hard.
Dawson et al. (2018),
Considering an approach

8. Learning Analytics Framework
and Governance Model
Small Scale Projects
Community of Interest
1
2
3

9. Learning Analytics Framework and Governance Model
What about …
- Ethics
- Access
- Governance
- Accountability
-> LAPI … Learning Analytics
Policy Initiative
-
Data confusion

11. ROMA: Rapid Outcomes Mapping Approach
ROMA is an approach to improving policy engagement
processes, to influence change. It comprises a suite of
tools that any organisation can use at any stage in their
policy engagement process.
https://www.odi.org/features/roma/home

12. 2.Identify key
stakeholders
1.Map political
context
3.Identify desired
behavior changes
4.Develop
engagement
strategy
5.Analyze internal
capacity for
change
6.Establish
monitoring &
learning
framework Define and
redefine policy
objectives
Adapted version of the ROMA model
used by the SHIELA project (Tsai et al.,
2018) and by Hainey et al. (2018)

13. Series of semi-structured interviews, focus groups and workshops with
members of the University Senior Leadership Team (n=12), Faculty
Deans and Associate Deans, academic and professional staff of the
University Faculties and Central Service Units (n=39), and students
(n=6). The data collected was analysed along the six dimensions,
Piggyback on ongoing policy reform and
rationalization.

14. Three key outcomes
1. Able to identify solutions to support the aspirations
of the learning and teaching vision & strategy and,
enrol academic and professional staff on the journey.
2. Learning Analytics Principles and Framework
document – drawing on well received work at the
University of Edinburgh, UK.
3. A major step towards an agreed institution level
governance model and defined strategic drivers.

15. Strategic drivers
(sponsors)
Student success (SAS)
Revenue (COO)
Learning and teaching
(VPA)
Capability development
(Provost/HR/COO)
Governance
(bringing analytics safely to
scale)
Whole of journey approach
(retention and completion)
Learning Analytics Principles
Data ethics, policy and
framework
Operating Model
Data access
Interfaces
Agency & authority
Transparent & explainable
Assurance: operationalization
of data ethics / principles
Indicators and metrics

16. Governance for Data / Learning Analytics (Board & Terms of
Reference)
Data Use / Ethics Policy
Learning Analytics principles & framework
Workstreams
Learning Analytics guidelines
Projects
Pilots / Proof of
Concept
Research informed
Data / Learning Analytics: Vision, purpose, success & outcomes
Senior Leadership
Team
University strategic
priorities
Related policy;
privacy notice
Related cross
university activity
Related research

17. • The use of Learning Analytics will benefit the University culture of
teaching and learning (with a special emphasis on Akoranga –
collective responsibility for learning).
• Student agency in Learning Analytics is acknowledged and
supported.
• Learning Analytics will be used in an ethical and transparent way.
• Learning Analytics will be practiced responsibly, in line with the
principle of Kaitiakitanga (Protection)
• Good Governance (Kāwanatanga) will be core to our approach to
Learning Analytics.
Elements of LA Principles and Framework…

18. LMS and Lecture Capture embedded tools
StudentVis homegrown visualization tool
OnTask personalized feedback
Acawriter natural language processing tool
Quantext insights through text analysis
Ecosystem of Rapid PoC Small Scale Projects

19. LMS built-in tools – e.g. Retention Centre

20. http://dx.doi.org/10.18608/jla.2018.53.15
OnTask: Delivering Data-Informed, Personalized Learning Support
Actions – in Journal of Learning Analytics. Vol. 5: No.3. 2018

22. LMS - Tools are too complex and don’t meet the needs of lecturers to monitor students’ progress.
StudentVis - Provides a history of practice that can be drawn on to support efforts around monitoring and
responding to student progress.
OnTask - Enthusiastic about the tool, recognizing its power to administer personalised feedback within
large courses to specific subsets of students based on assessment and activity conditions.
- The learning curve to use the tool is significant e.g. around importing data into the tool, setting
up the conditional rules etc.
Acawriter - Students noted the value of the tool to support student agency and timely feedback.
- Participants see the tool supporting a number of use cases.
Quantext - Potential to offer new insights into how students learn in the course, their levels of
understanding of subject-specific concepts and terms.
- Best used in conjunction with an academic developer

23. Community of Interest
Fischer, 2001

25. • Convene an open forum for sharing
practice – regular meetings
• Develop anthologies of meaning for
a common cooperative language and
goal finding (Weiseith et al., 2006)
• Leading to social practice directed
towards institutional learning
• Validation of the learning analytics
principles through repeated
exposure
• Champions within Faculty and CSUs
• Membership of SOLAR provided
access to further capability
development (e.g. LASI)

26. Student focused
institutional analytics
•Macro / meso level
•Retention
•Defined success
•Completion of course
across the university
as a whole
•Audience:
•Support services
•Student
•Governmental level
reporting
•Crosses prospects
and current students
Learning Analytics
•Micro level (e.g.
specific courses and
degree programmes)
•Focus on the learning
and teaching practices
•Audience:
•Staff (e.g. lecturers,
tutors, course
administrators)
•Students
•Maturity in LA
principles and
framework is an
important enabler
•Staff capability and
engagement are critical
success factors
Data Analytics
•Macro level
•Educational data
mining
•Predictive modelling
•Academic Analytics
•Maturity in data
governance is an
important enabler
Research
•Individual / Group
•Encouraged and
supported
•Human Ethics
Committee line of
sight
•Driven institutionally
by research strategy
and associated
priorities
•Can inform VUW data
analytics activity
SSTP
Small scale
pilots
Predictive
modelling
MOOC learner
flows
Project areas:
Space for everyone …

27. Conclusions
• Need to attend to people & culture, process and technology to move from a
data siloed to a data informed institution:
• Technological readiness;
• Leadership;
• Organisational culture;
• Staff and institutional capacity;
• Strategy.
• All levels of activity can co-exist but need to be coordinated
• Academics desire a just-in-time, one-to-one support model that can help them
explore options for meeting their analytics needs as well as provide how-to
support.
• Students also need support to learn how to use LA tools effectively to support
their learning needs.
Colvin et al. (2017)

28. CO-CURRICULAR CURRICULAR
RUN Student Advisor Scheme Pilot
School of Education, Bespoke
Cohort Coach Pilot*
(Low SES)
Consolidated Support
Peer Tutor Pilots
(Vygo = Education
Studiosity = Health)
1:1
PEER
TUTORING
2020 STUDENT ADVISING
*Pending HEPPP Allocation

29. diffuse
design
expert design
Manzini, 2015
Iterative, dynamic and sustained

30. CO-CURRICULAR CURRICULUM
Advising
MCINTOSH STUDENT ADVISING MODEL

31. When you start to predict ‘accurately’ … possibility

32. General LA pilots’ findings and conclusions
 For LA practices to be successful, capability development as well as technology
implementation need to be addressed. Though participants in the pilots were largely hand-
picked from engaged academics and students, many complained of lacking the time or
capacity to learn how to use the tools, interpret and act on them effectively. This is true
not only of lecturers but also students who need support to learn how to use LA tools
effectively to support their learning.
 Academics desire a just-in-time, one-to-one support model that can help them explore
options for meeting their analytics needs as well as provide how-to support.
 Overall, there is a need to coordinate efforts in the LA space across different university
strategic drivers and service areas. We noted overlaps between StudentVis, CRM Advice
(another system being piloted within student academic services) and OnTask. While the
drivers and scope may be different, participants in the LA pilots saw these tools serving a
similar need and are seeking holistic approaches to monitoring student progress and
support.
 Effective LA tools are characterised as easy to use, fast, customisable, accurate, intuitive
and, preferably, aggregated in a single location.