Keynote Address, Ascilite 2011, 6 Dec., Hobart, Tasmania http://simon.buckinghamshum.net

1. Keynote Address, Ascilite 2011, 6 Dec., Hobart, Tasmania
www.leishman-associates.com.au/ascilite2011
Learning Analytics
Dream, Nightmare or Fairydust?
Simon Buckingham Shum
Knowledge Media Institute, The Open University UK
simon.buckinghamshum.net
@sbskmi
1

2. Introducing a new
Analytics Platform…
2

3. From a recent review…
“Some have tried to argue that
this technology doesn't work out
cost effectively when compared to
conventional tests... but this
misses a huge point. More often
than not, we test after the event
and discover the problem — but
this is too late..”
3

4. Aquarium Analytics!
4

5. Aquarium Analytics!
5

6. How is your aquatic ecosystem?
“This means that the keeper can be notified before water
conditions directly harm the fish—an assured outcome of
predictive software that lets you know if it looks like the
pH is due to drop, or the temperature is on its way up.
This way, it’s a real fish saver, as
opposed to a forensic examiner,
post-wipeout.”
(From a review of Seneye, in a hobbyist magazine)
6

7. How is your learning ecosystem?
This means that the teacher can be notified before
learning conditions directly harm the students — an
assured outcome of predictive software that lets you
know if it looks like engagement is due to drop, or
attainment is on its way up.
This way, it’s a real student saver,
as opposed to a forensic
examiner, post-wipeout.
7

8. First-to-market immaturity, tricky install process…
…but when it’s done calibrating and the dashboard
springs to life, there’s an exciting sense of control
– BUT you still need to know what ‘good’ looks like 8

9. What do we mean by
Learning Analytics?
9

10. Learning analytics
“Learning Analytics is concerned
with the collection, analysis and
reporting of data about learning in a
range of contexts, including informal
learning, academic institutions, and
the workplace.
It informs and provides input for
action to support and enhance
learning experiences, and the
success of learners.”
2nd Int. Conf. Learning Analytics & Knowledge 2012

11. Learning “vs” Academic Analytics
Penetrating the Fog: Analytics in Learning and Education – George Siemens and Phil Long
http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume46/
11
PenetratingtheFogAnalyticsinLe/235017

12. “Academic Analytics”
• Stage 1—Extraction and reporting of
transaction-level data
• Stage 2—Analysis and monitoring of
operational performance
• Stage 3—What-if decision support (such as
scenario building)
• Stage 4—Predictive modeling and simulation
• Stage 5—Automatic triggers of business
processes (such as alerts)
Goldstein, P. J. (2005). Academic Analytics: The Uses of Management Information and Technology in Higher
Education: Key Findings. Boulder, Colorado: Educause Center for Applied Research
http://net.educause.edu/ir/library/pdf/EKF/EKF0508.pdf
12

13. What this talk is not…
The future is
analytics!
Step this
way…
13

14. The future is
analytics!
Step this
way…
14

15. Power
15

16. Power
Learning
Analytics
16

17. dream
17

18. dream
Nightmare!

19. dream
Nightmare!
Fairydust
19

20. Big Data & Analytics:
Very Big Buzzwords
20

21. SAS and Education
http://www.sas.com/industry/education/highered 21

22. IBM TechTrends2011 report:
Analytics fastest growing tech
22

23. IBM TechTrends2011 report:
Analytics fastest growing tech
…but do they know anything about
the roles of language in learning? 23

24. Personal, volunteered
Big Data
is about to explode
24

25. Explosion of data sharing about
ourselves: personal health
25

26. Explosion of data sharing about
ourselves: location

27. Big Data & Analytics:
Very Big Buzzwords
Educational
Institutions
27

28. Your Educational
Institution’s Response?
28
http://mcseavey.org/blog/2007/12/11/deer-in-headlights

29. Response?
Your Learners’
29
http://mcseavey.org/blog/2007/12/11/deer-in-headlights

30. NMC Horizon 2011 Report:
Learning Analytics (4-5yrs adoption)
L. Johnson, R. Smith, H. Willis, A. Levine, and K. Haywood, The 2011 Horizon Report (Austin, TX: The New
Media Consortium, 2011), http://www.nmc.org/pdf/2011-Horizon-Report.pdf 30

31. EDUCAUSE
http://www.educause.edu/Resources/Browse/Analytics/16961
31

32. Right here
Right now
32

33. Analytics in English schools
RAISEonline cohort visualization
33

34. Analytics in your VLE:
joules: analytics for Moodle
http://manuals.moodlerooms.com/display/JOULE2/joule+Reports+Manual
34

35. Analytics in your VLE:
OU VLE Reporting & Tracking tools
Summary stats on number of views, edits and
posts to: Forum, Wiki, Blog, Structured Content,
Quiz, External Quiz, Google Docs, Elluminate
sessions…
35

36. Analytics in your VLE:
Blackboard: feedback to students
http://www.blackboard.com/Platforms/Analytics/Overview.aspx
36

37. Analytics in your VLE:
Grockit: feedback to students prepping for standardized tests
https://grockit.com/research
37

38. Analytics in your VLE:
Desire2Learn visual analytics
http://www.desire2learn.com/demos/Analytics/
Students
Online tools
38

39. Social analytics start to become a
commodity service
http://www.mzinga.com/software/tour.asp 39

40. My OU Story app: emotional state
Tony Hirst, Liam Green-Hughes, Stuart Brown
http://apps.facebook.com/myoustory
40

41. context
context
context
41

42. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
Session
2
3
42

43. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
2
3
43

44. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
2
3
44

45. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
Mentor 1 is doing the best
job: at this introductory
2 level, students need
intensive input and
flounder if left
3
45

46. Beyond data dashboards:
predictive modelling
46

47. OU Analytics: Predictive modelling
§ Probability models help us to
identify patterns of success that vary
between:
§ student groups / areas of
curriculum / study methods Best predictors of
§ Benefits future success:
previous OU study
§ provide a more robust comparison of data – quantity
module pass rates and results
§ support the institution in identifying
aspects of good performance that
can be shared, and aspects where
improvement could be realised
OU Student Statistics & Surveys Team, Institute of Educational Technology 47

48. Purdue University Signals
Real time traffic-lights in the student
VLE, based on a predictive model
http://www.itap.purdue.edu/studio/signals 48

49. Purdue University Signals
Real time traffic-lights for students
based on a predictive model
Premise: academic success is defined as a function of
aptitude (as measured by standardized test scores and
similar information) and effort (as measured by participation
within the CMS).
Using factor analysis and logistic regression, a model was
tested to predict student success based on:
• ACT or SAT score
• Overall grade-point average
• CMS usage composite
Predicted 66%-80%
• CMS assessment composite
of struggling
• CMS assignment composite
students who
• CMS calendar composite
needed help
Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE
Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x
49

50. this all feels rather traditional —
isn’t there a learning revolution going on?!
50

51. Learning analytics for this?
“We are preparing students for jobs
that do not exist yet, that will use
technologies that have not been
invented yet, in order to solve
problems that are not even
problems yet.”
“Shift Happens”
http://shifthappens.wikispaces.com
51

52. Learning analytics for this?
“The test of successful education
is not the amount of knowledge
that pupils take away from school,
but their appetite to know and
their capacity to learn.”
Sir Richard Livingstone, 1941
52

53. Tectonic shifts in the learning landscape...
TECH: cloud,personalised, real Taken together, these are
time, multimedia, mobile... profound shifts in
power,
FREE/OPEN:is increasingly
relationships,
expected – I might pay later
economics,
SOCIAL LEARNING: innovation now infrastructure...
depends on it
VALUES:autonomy, diversity, self-
expression, participation
POST-INDUSTRIAL: new institutional As these reshape
roles in post-industrial education our conception of the
future of learning
–they must also
reshape our conception of the
future of learning analytics 53

54. analytics for C21 skills?
learning how to learn?
authentic enquiry?
social capital critical questioning argumentation
citizenship habits of mind resilience
collaboration creativity metacognition
identity readiness sensemaking
engagement motivation emotional intelligence
54

55. Analytics for
learning dispositions
and transferable skills
55

56. ELLI: Effective Lifelong Learning Inventory
Web questionnaire 72 items (children and adult versions: used
in schools, universities and workplace)
56

57. Validated as loading onto
7 dimensions of “Learning Power”
Being Stuck & Static Changing & Learning
Data Accumulation Meaning Making
Passivity Critical Curiosity
Being Rule Bound Creativity
Isolation & Dependence Learning Relationships
Being Robotic Strategic Awareness
Fragility & Dependence Resilience
Univ. Bristol and Vital Partnerships provides practitioner resources
and tools to support their application in schools and the workplace 57

58. ELLI generates a 7-dimensional spider
diagram of how the learner sees themself
Basis for a mentored-
discussion on how the
learner sees him/herself,
and strategies for
strengthening the profile
58
Bristol and Open University are now embedding ELLI in learning software.

59. Adding imagery to ELLI dimensions to
connect with learner identity
59

60. Working with Gappuwiyak School, N. Territory AUS
(Ruth Deakin Crick, University of Bristol) http://bit.ly/srUSHE
Changing & Learning: Strategic Awareness:
The Drongo - Guwak Emu - Wurrpan
Meaning Making:
The Pigeon - Nabalawal
Critical Curiosity:
Sea Eagle - Djert
Resilience:
Brolga - Gudurrku
Learning Relationships: Creativity:
The Cockatoo - Ngerrk Bower Bird - Djurwirr 60

61. ELLI generates cohort data for each dimension
61

62. …drilling down on a specific dimension
62

63. EnquiryBlogger:
Tuning Wordpress as an ELLI-based learning journal
Standard Wordpress editor
Categories from ELLI
Plugin visualizes
blog categories,
mirroring the ELLI
spider
63

64. EnquiryBlogger
dashboard

65. LearningEmergence.net
more on analytics for learning to learn and authentic enquiry
65

66. Analytics for
learning conversations
66

67. Discourse Learning Analytics
Effective learning conversations display some
typical characteristics which learners can and
should be helped to master
Learners’ written, online conversations can be
analysed computationally for patterns
signifying weaker and stronger forms of
contribution
67

68. Socio-cultural discourse analysis
(Mercer et al, OU)
• Disputational talk, characterised by disagreement and
individualised decision making.
• Cumulative talk, in which speakers build positively but
uncritically on what the others have said.
• Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
68

69. Socio-cultural discourse analysis
(Mercer et al, OU)
• Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
• Statements and suggestions are offered for joint consideration.
• These may be challenged and counter-challenged, but challenges are
justified and alternative hypotheses are offered.
• Partners all actively participate and opinions are sought and considered
before decisions are jointly made.
• Compared with the other two types, in Exploratory talk knowledge is made
more publicly accountable and reasoning is more visible in the talk.
Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
69

70. Analytics for identifying Exploratory talk
Elluminate sessions can
be very long – lasting for
hours or even covering
days of a conference
It would be useful if we could
identify where quality learning
conversations seem to be taking
place, so we can recommend
those sessions, and not have to
sit through online chat about
virtual biscuits
Ferguson, R. and Buckingham Shum, S. Learning analytics to identify exploratory dialogue within synchronous text chat. 70
1st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011)

71. Defining indicators of Exploratory Talk
Category Indicator
Challenge But if, have to respond, my view
Critique However, I’m not sure, maybe
Discussion of Have you read, more links
resources
Evaluation Good example, good point
Explanation Means that, our goals
Explicit reasoning Next step, relates to, that’s why
Justification I mean, we learned, we observed
Reflections of Agree, here is another, makes the
perspectives of others point, take your point, your view
71

72. Extract classified as Exploratory Talk
Time Contribution
2:42 PM I hate talking. :-P My question was whether "gadgets" were just
basically widgets and we could embed them in various web sites,
like Netvibes, Google Desktop, etc.
2:42 PM Thanks, that's great! I am sure I understood everything, but looks
inspiring!
2:43 PM Yes why OU tools not generic tools?
2:43 PM Issues of interoperability
2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you
can add various widgets, similar to those existing web start pages.
2:43 PM What if we end up with as many apps/gadgets as we have social
networks and then we need a recommender for the apps!
2:43 PM My question was on the definition of the crowd in the wisdom of
crowds we acsess in the service model?
2:43 PM there are various different flavours of widget e.g. Google gadgets,
W3C widgets etc. SocialLearn has gone for Google gadgets 72

73. Discourse analysis (Xerox Incremental Parser)
Detection of salient sentences based on rhetorical markers:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct … little is known …
evidence ... … role … has been elusive
… the previously proposed …
... we suggest a new ... approach ... Current data is insufficient …
… is universally accepted ...
... results define a novel role ...
CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided The goal of this study ...
paradoxes … important advances Here, we show ...
In contrast with previous Knowledge ... is crucial for ... Altogether, our results ...
hypotheses ... understanding indicate
... inconsistent with past valuable information ... from
findings ... studies
GENERALIZING: SURPRISE:
... emerging as a promising We have recently observed ...
approach surprisingly
Our understanding ... has grown We have identified ... unusual
exponentially ... The recent discovery ... suggests
Ágnes Sándor & OLnet Project:
http://olnet.org/node/512
... growing recognition of the intriguing roles
importance ...
De Liddo, A., Sándor, Á. and Buckingham Shum, S. (In Press). Contested Collective Intelligence: Rationale,
Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work Journal

74. KMi’s Cohere:
a web deliberation platform enabling semantic social
network and discourse network analytics
Rebecca is playing
the role of broker,
connecting 2 peers’
contributions in
meaningful ways
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011)

75. Learning Analytics
stories from the
near future
75

76. Dream? Student’s analytics dashboard
1 2 3
In your last discussion with your mentor, you decided
to work on your resilience by taking on more learning
challenges
Your ELLI Spider
5 shows that you
have made a start 4
Your most on working on
recent mood
comment:
your resilience,
“Great, at and that you are
last I have also beginning to
found all the
resources
work on your
that I have creativity, which
been you identified as
looking for,
another area to
thanks to!
Steve and work on.
Ellen.!
Based on: Buckingham Shum, S. and Ferguson, R. (2011). Social Learning Analytics. Available as: Technical Report
KMI-11-01, Knowledge Media Institute, The Open University, UK. http://kmi.open.ac.uk/publications/pdf/kmi-11-01.pdf

77. Mr A. Administrator (2015)
Course Application: Analytics Report
Application from Ali Bloggs to study Z0001
This applicant has a high risk profile:
1. No academic study for last 15 years
2. Low socio-economic background
3. English as a second language
4. Weak ICT skills
5. His responses to the learning styles survey indicate a
loner, rather than a collaborative learner, known to be a
disadvantage on this course
Without a Grade 3 tutor (advanced skills in 1-1 support), based
on the last 5 years data there is a 37% chance of dropping out
by Week 6. Recruiting this tutor will take you over budget.
Recommended decision: REJECT
[ACCEPT] [REJECT]
77

78. Ms. S. Student (2015)
“Hi Sue,
In the last 2 weeks, it looks like you’ve been really stretching
yourself. You seem to have been working on your critical
thinking, with that challenge to Mike’s assumption, and the
evidence-based claim about nuclear waste in your blog.
Check out Donna Winter, who seems to have very different
views to yours on global warming. How would you assess her
position?
In your next video conference tutorial, try to improve on the
last three when you seem to have contributed only once each
time.”
78

79. Ms. S. Student (2015)
“Did you know that two other people you know have used
the Smith & Jones 2009 framework graphic?
Finally, you seem to have really become a pivotal member
of the Local-Global Climate Network. Good work: only a
month ago you were on the edge!
Why not reflect in your blog on how these groups are
helping you in your long term goal to Work for the UN in
Africa?”
79

80. Prof. R. Researcher (2015)
Learner 2020 Project:
Analytics report to Rachel Richards:
3761 students are working from 27 resources, 4 of them
provided by the course.
Presentation of SNA viz. to Experimental Grp.1 resulted in
79% of outlying community members taking at least 1 of the
recommended actions and moving themselves to more strategic
positions in the network: 35% towards the centre, 17% to
broker with another relevant network, and 27% with a low-
relevance network.
Currently, this represents a statistically significant effect
according to Hypothesis 1 (p<0.05)
80

81. Dr L. Lecturer (2015)
Z0001: Week 4 Analytics Report
Dear Les,
Please find below key headlines [full report]:
• 29 students have engaged in at least one empathic
conversation with a peer. This is below the 5 year mean for
2012-17, but 10% better than last year. Your arrival on the
course has made an impact!
• 13 students display low resilience when stretched, at levels
predictive of failure to complete. Please review the
mentoring recordings to verify that appropriate
interventions are in place.
• Biometric data indicate abnormally high levels of distress
in 3 students whenever team working is mentioned.
81

82. Closing thoughts
82

83. and hold, the
magnifying glass?
Who gets to define,
83
http://www.flickr.com/photos/somegeekintn/3709203268/

84. http://www.flickr.com/photos/centralasian/6396004353/sizes/m/in/photostream/
Learning Analytics should
provide mirrors for
learners to become more
reflective, and less
dependent
84

85. As Big Data and Analytics mature we
will see the convergence of…
institutional data sources!
85

86. As Big Data and Analytics mature we
will see the convergence of…
+ voluntarily!
shared learner data!
86

87. As Big Data and Analytics mature we
will see the convergence of…
+ scavenging in the wild!
clouds for relevant signals!
87

88. Power
Learning
Analytics
88

89. dream
timely interventions empowered, reflective learners
a living evidence-base open interoperable LA
Nightmare!
regressive pedagogy learners disempowered
HEI “vs” learner interests
Fairydust
over-optimistic, decontextualised
claims about predictive power 89

90. SoLAResearch.org
Sign up for news
twitter @solaresearch
90

91. International Conference on
Learning Analytics & Knowledge 2012
Vancouver, 29 April – 2 May 2012
http://lak12.sites.olt.ubc.ca 91

92. Simon Buckingham Shum
Knowledge Media Institute, The Open University UK
Slides now up: simon.buckinghamshum.net
@sbskmi
This keynote is dedicated to my Father
Jonathan Chung-Hin Shum
4 Jan. 1930 – 27 Nov. 2011