An institutional perspective on analytics that focusses on a particular tool developed using an agile methodology to visualise learner behaviours in MOOCs via Sankey diagrams.

1. Visualising Learner
Behaviour in MOOCs
An Institutional perspective
Steven Warburton
Karsten Lundqvist
Michael Godinez
University of Wellington, NZ. ASCILITE 2018

2.  Describe the institutional context;
 Explore drivers for activity in learning analytics
and MOOCs;
 Demonstrate a visualization tool for learner
behavior in MOOCs;
 Consider the value and sustainability of such
efforts against wider imperatives
Aims of
this talk

3. Exploring the
institutional
context
• WHAT IS THE CONTEXTUAL
BACKDROP TO THIS PROJECT?
• WHAT ARE THE STRATEGIC DRIVERS?
• WHY ARE WE INTERESTED IN DATA
AROUND LEARNER BEHAVIOURS?

4. The university digital
roadmap project
highlighted the need
to build capability in
the area of agile and
design thinking to
enable digital change
initiatives to be
successfully realised.

5. Organisational
capability
development
work needs to be
done …

6. Customer
Customer
Customer
Communities of interest
Finding channels and
initiatives to engage
external audiences with
the university and help
grow the student body.

7. New opportunity space for
micro-credentials in New
Zealand

8. Developing
Massive Open
Online Courses
offered strategic
possibilities and
practical benefits to
respond to
institutional needs

9. The growth and diversification
of MOOCs has opened up
possibilities for a broader and
more flexible portfolio of
educational products.
Top 5 by registered users

10. MOOCs – from drivers to success
• Pressure to extend our offerings to international students.
• Revenue generation becoming more critical.
• Understanding our end to end market is becoming core strategy.
• Getting learners to sign up is only part of the equation – we want to see our
target learner profile reach the end of the course.
• A quality learning experience matters.
• Therefore it becomes critical to gain intelligent insights into our learner
behaviours and, the multiple touchpoints they have with the institution.

11. From “Insights” to … insights

12. Current analytical tools are
useful for reporting but not
detailed enough to help
learning designers.

13. MOOC analytics background
• Early studies on MOOCs used mostly interview or surveys around the UX,
participant demographics, metrics of learner progression.
• Overtime, participation size and completion rate have become more popular
metrics.
• Visualisation tools for these common metrics have included box charts scatter
diagrams etc.
• These tools are useful for comparing MOOCs but hide complexities of learner
behaviours that could be used for learning designers.

14. Our design space: analyzing learner behaviours
• Clear need (current toolset limited, contextual drivers strong)
• Low hanging fruit (easy to get data)
• Ethically light touch
• Visualisation important (our Learning Designers want graphical
tools to help uncover the complexities of learner behaviours)
• Multiple wins possible (learning design, research, marketing, and
co-design)

15. Sankey diagrams for visualisation
• Visualise flows from one state to another using the
arrow width to indicate quantity
• Coffrin et al. (2014) used a specialized type of Sankey
state transition diagram to illustrate user views of
videos in a MOOC
• Google analytics uses Sankey diagrams to visualize
transitions on websites (session based activity)

16. An agile
approach:
rapid
prototyping
paradigm
Connell and Shafer, 1989
edX reference group*
provided structured
feedback
for each iteration of
the cycle.
* a cross functional group w/ learning designers, academics, marketing, content developers,.

17. Initial
prototype:
a visual tool
that provided
an overview of
participant
behaviour
The visualizer tool

18. ⍋ Paying versus
⍒Non-paying participants
A view for paying and non-paying students revealed discernable behavioural differences:

19. • Although 38.7% of the paying customers jumped forward to the first quiz, 65.9% of
these were short jumps (topic 3.1 or after), and they seemingly are engaging with
the content more, and the quizzes are used to engage with previously viewed
material throughout the MOOC.
• The non-paying participants followed the end of course linearly with only a few
jumps. For example under 4% jumped ahead to the last quiz and 2.9% jumping back
back to previous material from that quiz. This indicates a change in behaviour.
• The paying participants kept jumping around the material throughout the
For instance 19.8% jumped to the last quiz, with 20.4% jumping back to explore the
material further, indicating a continued engagement with the material.

20. Feedback on prototype 1 (interpretability and data)
• Consensus that this was an ‘Interesting’ approach. With little prompting the
reference group were able to make observations about learner flows e.g. at
exactly which points leaners were leaving the MOOC.
• Design changes for next iteration:
• Use edX log files (more information than database files, and future proof)
• Use a two bar visualization to differentiate unique visits from subsequent visits

21. Prototype 2 – two bar visualisation

22. Visible learner
behaviour
differences after
a platform
navigation
design change

23. Feedback on Prototype 2 (UX and granularity)
• Balance expressiveness of the visualisation with the UX
• Prepare new views based on segments of users e.g. by qualification
• Create views based on users who followed a particular path
• Minor visual update on the granularity setting to accurately portray
all visits in the vertical long bar
• User can set the minimum number of paths shown

24. 3rd and 4th prototypes (in brief)
• Technical changes: data extraction method, speed up data
processing
• Some UX features that included movable vertical to help visualize
pathways more clearly
• GUI for selecting data files to run in the visualizer tool

25. Filters and settings for visualisations (includes: education level;
paying/non-paying; date window etc.)

26. Future work on the visualiser tool
• More user testing and UX design work. Both internal and external.
• Develop new visualisations.
• Add further statistical tests.
• Community visibility for input from other developers and designers to help test
and iterate the tool.
• Loop back into edX and other providers activities to potentially draw funding to
develop further.
• To carry out comparative research studies on course reruns to measure the effect
of interventions.

27. Stepping back
– the institutional
picture

28. Institutional mapping of all analytics
related activity revealed:
• Emerging ethical issues
• Lack of governance
• Policy vacuum

29. Learning Analytics Policy Initiative
… led to the launch of the

30. Backward approach to planning and implementing Learning Analytics
Why
• Reasons, needs, motivations for the adoption of LA at Victoria
• Expectations, aspirations, opportunities, threats, concerns
How
• SHEILA policy framework (Tsai, Dragan Gašević & colleagues)
• Principles based
• Individual and focus group semi-structured interviews
What
• Identify, plan and implement pilot LA projects (2019)
• Design an approach to evaluate the LA pilots (2019)
• University LA development and implementation approach (2020-21)
• Test the guidelines with institutions currently using LA such as MIT, Waikato University and Raukawa and evaluate their use at a national level
(2019 – 2020)

31. Progress so far
Phase I
• Interviews conducted with 12 members of the senior leadership team
• Focus group conducted with 6 student leaders
• Focus groups interviews with Faculties and CSUs (underway)
Phase II
• Mapping responses onto the six dimensions of the SHEILA framework
• For each dimension, code Actions, Challenges and Policy questions (with further sub-themes)
• Develop Principles, Purpose and Code of Practice for the University implementation of LA

32. Based on the ROMA: Rapid Outcomes Mapping
Approach
“ROMA is an approach to improving policy
engagement processes, to influence change. It
comprises a suite of tools that any organisation can
at any stage in their policy engagement process to
improve how they diagnose the problem, understand
the types of impact their work could have on policy-
making, set realistic objectives for policy influence,
develop a plan to achieve those objectives, monitor
learn from the progress they are making and reflect
learning back into their work.”
http://www.roma.odi.org/introduction.html

33. Building on agile / co-design success

34. diffuse
design*
expert design
Manzini, 2015
Success in this
design space is
helping promote
more whole of
organizational
appreciation of co-
design as an
approach to building
change capability
across the
organization.
*design performed by everybody

35. Final Thoughts
• Tangible benefits emerge in linking imitative/s back to strategy.
• Has helped build traction in design focused agile approaches.
• Brought together a cross functional design teams (see also: “Third Space”
working (Whitchurch, 2008).
• Stitching initiatives together, developing communities of interest, helps drive an
organisational capability shift.
• Policy development and transparent governance helps release activity.
• Need to enrol stakeholders across multiple levels of the organization – using
the ROMA approach is helping us do this.