1. Formation and
Learning Analytics?
Simon Buckingham Shum
Knowledge Media Institute, The Open University
Visiting Fellow, Grad. School of Education, University of Bristol
From August: Connected Intelligence Centre, University of Technology Sydney
http://simon.buckinghamshum.net
http://twitter.com/sbskmi
Formation by Design (FxD) Symposium
Georgetown University, Washington, June 30 – July 2, 2014
https://futures.georgetown.edu/formation
1
Emergence
Learning
2. It’s out of the labs and into products: every learning
tool now has an “analytics dashboard” (a Google image search)
2
5. 5
c.f.
“Data
is
the
new
oil”
Scaleable
Precise
Quan?fiable
Reprocessable
6. For Morozov, analytics is
where technological
solutionism hits education:
6
“This flight from thinking
and the urge to replace
human judgments with
timeless truths produced by
algorithms is the underlying
driving force of
solutionism.”
7. If something like this is the pedagogical architecture
for (trans)formational learning…
7
Structuring
Knowledge
Performing
Generating
Learning
Power
Purposing
8. If something like this is the pedagogical architecture
for (trans)formational learning…
8
Structuring
Knowledge
Performing
Generating
Learning
Power
Purposing
…what roles can computational
learning analytics play?
The
worldview
clash
is
what
makes
this
fascina?ng!
9. For discussion on how analytics shape
educational worldviews
9http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote
12. Liminal space = opportunity for transformation, or withdrawal
12
So while we initially strive to make our students
feel comfortable on campus, we then must help them balance
their desire for security with the need to take risks and explore new
ideas and possibilities. Rather than attempting to resolve the tension,
a college should help students find their place on this
precarious threshold, in the liminal space
between the familiar and strange, the old and the new.
Johansson & Felten, 2014
13. Liminal space = opportunity for transformation, or withdrawal
13
“It is when you have left the tried and true, but have not yet
been able to replace it with anything else. [...] It is when you are
between your old comfort zone and any possible new answer. If you
are not trained in how to hold anxiety, how to live
with ambiguity, how to entrust and wait, you will
run... anything to flee this terrible cloud of unknowing.”
Richard Rohr O.F.M. on the spirituality of liminal space
17. Social Network Analysis to gain insight into peer-peer
and peer-mentor dynamics (SNAPP tool)
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference
on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.
18. Social Network Analysis (SNAPP)
18
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
2 learners connect
otherwise separate
clusters
tutor only engaging
with active students,
ignoring disengaged
ones on the edge
19. Visualizing and filtering online social ties: by topic and type of tie
Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept
Demonstrator. Proc. 3rd Int. Conf. on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891
26. Observation
informal and formal
Self-Diagnostic
informal and formal
Behavioural
Analytics
Assessing Learning Dispositions/Mindsets
Future sweetspot...
multiple lenses to
provoke self-reflection
http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets
27. Self-report through reflective blogging
9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK
EnquiryBlogger Wordpress Multisite plugins
http://learningemergence.net/tools/enquiryblogger
27
28. Masters level EnquiryBloggers
Graduate School of Education, University of Bristol
EnquiryBlogger: blogging for Learning Power & Authentic Enquiry
http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
28
31. http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets
Dispositional profile from behavioural traces, to
complement self-report?
Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn
http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium
Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search
Social network patterns,
teamwork effectiveness
and initiation of
relationships
Questioning, arguing
and search behaviours
reveal intrinsic curiosity
and epistemic
commitments
Tagging/sharing/
blogging/social patterns
reveal how you see
connections between
ideas
Behavioural and somatic
traces associated with
perseverance, grit,
tenacity; overcoming
panic/stress when
stretched
32. narra7ve/
discourse
analy7cs
32
The
Storymaking
Project:
PhD
by
Joanna
Kwiat
hLp://projects.kmi.open.ac.uk/storymaking
33. 1st International Workshop on
Discourse-Centric Learning Analytics
analytics that look
beneath the surface, and
quantify linguistic proxies
for ‘deeper learning’
Beyond number / size / frequency of posts; ‘hottest thread’
http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpg
http://solaresearch.org/events/lak/lak13/dcla13
34. Why are stories so important to formation?
34
“Fundamentally different to logical argument, the
story positively demands those things that the other
either denies or restricts: emotion, agency,
character, perspective, and so on. To understand a
story on some level is to become engaged in its
telling and this will normally be achieved by
allowing oneself to empathise or identify with the
story’s characters or its narrator.”
35.
36.
37. Discourse analytics on webinar textchat
Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference
on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM, pp. 99-103. Open Access Eprint http://oro.open.ac.uk/28955
Can we spot the
quality learning
conversations in a
2.5 hr webinar?
40. Discourse analytics on webinar textchat
-100
0
100
9:28
9:40
9:50
10:00
10:07
10:17
10:31
10:45
11:04
11:17
11:26
11:32
11:38
11:44
11:52
12:03
Averag
Classified as
“exploratory
talk”
(more
substantive
for learning)
“non-
exploratory”
…language is used in a manner more
akin to “Exploratory Talk” (Neil
Mercer)
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc.
3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
41. Rhetorical discourse analytics
41
OPEN QUESTION:
“… little is known …”
“… role … has been elusive”
“Current data is insufficient …”
CONTRASTING IDEAS:
“… unorthodox view resolves …”
“In contrast with previous
hypotheses ...”
“... inconsistent with past
findings ...”
SURPRISE:
“We have recently observed ... surprisingly”
“We have identified ... unusual”
“The recent discovery ... suggests intriguing
roles”
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
43. Rhetorical discourse analytics
43
Human analyst Computational analyst
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
44. Rhetorical discourse analytics
44
Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/
Glimpses of analytics capable of
detecting higher order thinking.
But humans will always read
differently to machines
Can we correlate this with
“academic writing”, and can such
analytics be used as formative
feedback on drafts?
45. Rhetorical discourse analytics
45
Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
CONTRAST
SUMMARY &
CONTRIBUTION
46. Evidence Hub: semantic storytelling for
students, practitioners and researchers
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
A wizard guides the user through
the submission of a structured
story:
• What’s the Issue?
• What claim are you making/
addressing?
• What kind of evidence
supports/challenges this?
• Link it to papers/data
• Index it against the NIC’s core
themes
47. Evidence Hub:
Argument Maps
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
The wizard then generates a
structured Knowledge Tree
showing evidence-based
claims (and disagreements)
51. De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf.
Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
Argumentation analytics (in Debate Hub & Cohere apps)
what epistemic contributions are learners making in the network?
51
Rebecca is playing the
role of broker,
connecting different
peers’ contributions in
meaningful ways We now have the basis
for recommending that
you engage with
people NOT like you…
52. Generates novel
visualizations to summarise
community arguments on a
given issue
http://consider.it
Structured Deliberation/Argument Maps
53. Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search
Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment
Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254
Learning sciences research into
Epistemic Commitments/
Cognition combined with
‘researching search’
methodologies.
54. Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
Dimensions of Epistemic Belief
Certainty The degree to which knowledge is conceived as stable or changing, ranging from
absolute to tentative and evolving knowledge
Simplicity The degree to which knowledge is conceived as compartmentalised or
interrelated, ranging from knowledge as made up of discrete and simple facts to
knowledge as complex and comprising interrelated concepts
Source The relationship between knower and known, ranging from the belief that
knowledge resides outside the self and is transmitted, to the belief that it is
constructed by the self
Justification What makes a sufficient knowledge claim, ranging from the belief in observation
or authority as sources, to the belief in the use of rules of inquiry and evaluation
of expertise
Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where
learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226
55. Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
Behavioural Proxies of Epistemic Belief?
Certainty
Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes;
is inconsistent with).
Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread).
Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains
an unresolved issue)
Simplicity
Standard network analytics describing the topography of the structure
Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative),
and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to).
Semantic analysis of tags on nodes and links
Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct
The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy;
argument from precedent)
In collaborative maps, the social network view showing who is connected to whom, but what kinds of links
Source
Presence within the text of nodes of ‘I think’ or other restatements of fact
Challenges to claims through negative link types and meta-discourse markers in node text
Few additional nodes made other than those created as quotations.
Justification
Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’.
Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’)
or publications based on their status (‘valuing peer review’)
Rhetorical constructions in node text which show a critical use of sources
56. analytics for wider
educational outcomes
Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for
Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/
57. Providing a more complete analytic on the
health of the educational community
57
“Test grades are just ONE of a dozen indicators in
our vision of a healthy educational community”
• life-long/life-wide learners • citizenship
• self-confidence • teamwork • emotional
intelligence • “wellbeing”…
58. Hierarchical Process Modelling
(Univ. Bristol Perimeta tool)
58
Seeing a learning community as a complex adaptive system requires
the voices of learners, teachers, leaders, parents and ‘external’
stakeholders
61. ‘Italian Flag’ visual analytic
61
Degree of green / white / red reflects current
(un)certainty over availability of evidence:
supporting / unknown / challenging
63. Your most
recent mood
comment:
“Great, at
last I have
found all the
resources that
I have been
looking for,
thanks to
Steve and
Ellen.
In your last discussion with your mentor, you decided
to work on your resilience by taking on more
learning challenges
Your ELLI Spider
shows that you
have made a start
on working on
your resilience,
and that you are
also beginning to
work on your
creativity, which
you identified as
another area to
work on.
1 2 3
45
Envisioning a social learning analytics dashboard
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics &
Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910
63
64. liminal
space
analy7cs?
Use of computational techniques to reflect back
to learners apparent shifts in • how they engage
in a community of enquiry/practice • how they
narrate their learning journeys • how they
negotiate meaning • how their worldview seems
to have shifted • how they respond cognitively
(even somatically) when challenged.
66. towards a networked
improvement community
(NIC) for FxD?
hLp://learningemergence.net/2014/05/29/learning-‐analy?cs-‐plus-‐nics-‐for-‐systemic-‐edu-‐improvement
67. Defining features of a NIC
‘B’ + ‘C’ Level improvement
‘A’
—
what
prac??oners
do
in
a
given
ins?tu?on/
context
‘B’
—
trying
to
improve
local
prac?ce
‘C’
—
the
network
of
‘B’
people
sharing
insight
and
coordina?ng
innova?on
Based
on
Doug
Engelbart’s
forma?ve
work:
hLp://www.dougengelbart.org/about/nics.html
68. Defining features of an Educational-NIC
a
sociotechnical
infrastructure
and
modus
operandi
for
coordinated
aLack
on
educa?onal
challenges
an
inten?onal
community
facilitated
by
a
coordina?ng
hub
using
agreed
representa?ons
of
the
system
and
improvement
strategies
in
order
to
orchestrate
ac?on
in
pursuit
of
measurable
goals
by
sharing
data,
insights
and
arguments
in
order
to
build
evidence-‐based
claims
Bryk
et
al,
Carnegie
Founda?on
for
the
Advancement
of
Teaching:
hLp://www.carnegiefounda?on.org/sites/default/files/bryk-‐gomez_building-‐nics-‐educa?on.pdf
69. A NIC is in essence a “Collaboratory”
“a collaboratory is more than an elaborate collection of ICT; it is
a new networked organizational form that also
includes social processes; collaboration
techniques; formal and informal
communication; and agreement on norms,
principles, values, and rules” (Cogburn, 2003, p. 86).
Cogburn, D. L. (2003). HCI in the so-called developing world: what’s in it for everyone, Interactions, 10(2), 80-87, New York: ACM Press.
http://dx.doi.org/10.1145/637848.637866
70. Building blocks of an Educational-NIC
NIC Data
Hub
Collective
Sensemaking
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Human + Machine
Analytics
Educational
ContextEducational
Context
71. Building blocks of an Educational-NIC
NIC Data
Hub
Collective
Sensemaking
Human + Machine
Analytics
Educational
ContextEducational
Context
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
72. Shared Conceptual Maps:
of system / problem / strategies
http://www.carnegiefoundation.org/elibrary/getting-ideas-action-
building-networked-improvement-communities-in-education
http://www.carnegiealphalabs.org/persistence/
73. Shared Conceptual Maps:
for knowledge navigation
People / Organizations / Projects / Claims / Evidence
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
74. Shared Conceptual Maps:
for knowledge navigation
People / Organizations / Projects / Claims / Evidence
Evidence Hub for Research by Children & Young People: http://rcyp.evidence-hub.net
75. Building blocks of an Educational-NIC
NIC Data
Hub
Human + Machine
Analytics
Educational
ContextEducational
Context
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Coordinating
Maps
(of the system,
problem, and
strategies)
Collective
Sensemaking