Presentation for an e/merge Africa Webinar on 4 May 2017 - http://emergeafrica.net/4-may-learning-analytics-opportunities-and-dilemmas/

1. Webinar 4 May 2017 – e/merge Africa
http://emergeafrica.net/4-may-learning-analytics-opportunities-and-dilemmas/
Paul Prinsloo (University of South Africa, Unisa)
Learning analytics:
Opportunities and dilemmas

2. Acknowledgements
I do not own the copyright of any of the images in this
presentation. I acknowledge the original copyright and licensing
regime of every image used.
This presentation (excluding the images) is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International
License
2

3. Confession(s)
I am a bit embarrassed
with the fact that this
presentation is so ‘text-
heavy’ as I normally
enjoy designing with
less text and rely more
on visual elements to
convey meaning.
The reason for the amount of text in this presentation is to allow this
presentation to stand on its own as a possible resource for whoever
may find the content or parts of the content usable/informative
Image credit: https://www.flickr.com/photos/sheila_sund/23835028019
3

4. 1. Is bigger data better data? What evidence can such data
provide and what are some of the shortcomings?
2. What are some of the ethical dilemmas involved in uses of
student data?
3. Is the hype over learning analytics based on idealism rather
than reality? How can we move beyond the hype of learning
analytics?
4. And…
In preparation for this presentation, the
organisers posed the following questions:
4

5. 4. Are lessons learnt from the Global North about uses of
learning analytics a useful starting point for educators in African
higher education? What do we adopt and where do we adapt?
Of these questions, the fourth one fascinates
me and exposes some of the dilemmas but also
unique opportunities for learning analytics in
the Global South/developing world context…
5

6. • We, in the Global South, see ‘data’ differently – and that we are very aware of how
certain variables and characteristics were selected and used as basis for
dehumanising many identified individuals and groups
• We are very aware of the challenge of dealing with the inter-generational legacy of
classification systems where the potential for advancement and access to resources
were determined based on assumptions and stereotypes of race, gender and
culture
• We ask how learning analytics functions in environments where online access and
participation are unevenly distributed and where login, download and participation
data say more of the legacy of apartheid and colonialism, than of students’
potential, motivation or aspirations
• We wonder how to engage with the number of logins and downloads, and the
number and quality of online engagements where evidence suggests that these
numbers correlate with socio-economic and prior learning circumstances, and not,
per se, with student potential, motivation or aspirations
Considering our locations and histories (past and
present)…
6

7. • Our geopolitical and institutional contexts and (in)efficiencies impact more on
individual performance than individual students’ agency
• Higher education institutions in the Global South often (mostly?) lack the
resources and infrastructure to establish and maintain integrated systems,
processes and policies to enable appropriate and ethical data collection, analysis
and use
• Many (most?) institutions in the Global South do not have capable and well-
resourced human resources (e.g. data scientists) to optimise the potential of
learning analytics or have the resources to respond ethically and appropriately to
identified needs
• The gender and race of those who code and develop algorithms have
internalised the assumptions and beliefs of the North Atlantic and perpetuate
these assumptions and beliefs through code
7
Considering our locations and histories (past and
present)… (cont.)

8. African higher education institutions are for sale to the highest
bidder – those commercial vendors and the apostles of Silicon Valley
who regard a free IPad and a dumbed-down version of the internet
as a fair exchange for access to students’ data?
And finally, considering our locations and
histories (past and present), is it possible
that...
Image credit: https://pixabay.com/en/photos/for%20sale/
8

9. Image credits: https://pixabay.com/en/photos/spaghetti/; https://www.flickr.com/photos/stevendepolo/30801087140;
http://www.offutt.af.mil/News/Art/igphoto/2000395404/; https://www.flickr.com/photos/stevendepolo/4312332671
https://www.flickr.com/photos/donhomer/12862086683; https://www.flickr.com/photos/tim_norris/1216942456
Making sense of the collection, analysis and use of
learning analytics is [increasingly] like…
9

10. Overview of the presentation
• A short introduction to learning analytics
• Mapping the socio-technical imaginary of evidence, (student) data and
the need for more data
• How do we engage with the potential, dangers and challenges in the
collection, analysis and use of data when…?
• Disclosing my own position regarding technology [data]
• Mapping student data – what data do we have, don’t have but can
collect, and what data we may never have…
• Pointers for considering the potential, limitations and challenges in
learning analytics
• What about the ethics of [not] collecting, analysing and using student
data?
• (In)conclusions
10

11. “Learning analytics refers to the
measurement, collection, analysis and
reporting of data about the progress of
learners and the contexts in which
learning takes place”
(https://www.jisc.ac.uk/reports/learning-analytics-in-higher-education)
11

12. Page credit: https://www.jisc.ac.uk/reports/learning-analytics-in-higher-education
12

13. Page credit: http://ignatiawebs.blogspot.co.za/2017/04/using-learninganalytics-to-inform.html
13

14. “Learning analytics refers to the
measurement, collection, analysis and
reporting of data about the progress of
learners and the contexts in which
learning takes place”
(https://www.jisc.ac.uk/reports/learning-analytics-in-higher-education)
It is about ‘learning’ and
informed about our
beliefs about what
constitutes learning…
What do we measure?
How do we measure?
When do we measure?
What don’t we measure?
Where do we collect,
what, how often, for
what purpose and who
does the collection?
To whom
do we
report?
What
then?
Who will
act?
What if
we
cannot
act?
What if
students
don’t
act?
Who
analyses
the data,
using what,
what skills
are
required,
who
verifies the
analysis?
How do we define ‘progress’? How do
we involve students in making sense of
the data, of their progress, of their
journeys?
Do we assume that all their learning
takes place on the LMS? What do we
assume about their logins, their
downloads, their clicks?
14

15. … has become saturated with data – ranging from automatically
collected, analysed and used, purposefully collected, analysed
and used and volunteered on social media; in exchange for
(perceived) benefits despite concerns about privacy, the
uncertainty of how the data will be used downstream and
combined with other sources of data; and in the context where
our trust in the collectors of data is often misplaced, irrational or
wishful thinking (See Kitchen, 2013, pp. 262-263)
How do we talk about the
collection, analysis and use of
student data in a world that…
Image credit: https://commons.wikimedia.org/wiki/File:Big_Hand_-_geograph.org.uk_-_644552.jpg
15

16. Page credit: http://www.govtech.com/education/How-Data-Driven-Learning-is-Changing-Education.html
“Success here derives from access to data,
or big data as it’s sometimes called.”
“Data matters.”
16

17. Page credit: http://www.baltimoresun.com/news/maryland/education/bs-md-college-analytics-20160611-story.html
17

18. Imagecredit:https://www.flickr.com/photos/haydnseek/2534088367
Student data is seen as the ‘new black’,
as a resource to be mined, scraped &
abused/misused
Image credit: http://fpif.org/wp-content/uploads/2013/01/great-oil-swindle-peak-oil-world-
energy-outlook.jpg
18

19. Page credit: http://insider.foxnews.com/2016/01/31/oklahoma-college-forcing-students-wear-fitbits
19

20. Page credit: https://inews.co.uk/essentials/news/education/university-monitor-student-social-media-gauge-well/
20

21. Higher education is
mesmerized/seduced
by the potential of
the collection,
analysis and use of
student data
Image credit: http://en.wikipedia.org/wiki/Medusa
(Student) data as Medusa – techno-solutionism
in action
21

22. Page credit: http://www.ft.com/cms/s/2/634624c6-312b-11e5-91ac-a5e17d9b4cff.html#slide0
22

23. Page credit: http://hechingerreport.org/students-worry-education-technology-might-predict-failure-chance-succeed/
23

24. How do we talk about the collection, analysis and
use of student data when…
• We have access to more data (often/increasingly real-time, granular and
very personal) of [some] students while a lot of our data about [most?]
students are but proxies for their circumstances [e.g. addresses] and
potential [e.g. prior learning experiences]
• We collect, measure, analyse and use student data in isolation from issues
surrounding epistemological access, institutional operational (in)efficiencies,
pedagogical approaches, faculty and support staff (in)experience and
(dis)engagement
• Current research suggests that many [most?] of the variables impacting on
students’ chances of success (i.e. socio-economic circumstances, political
and economic (in)stability] fall outside the locus of control of students,
faculty and the institution
24

25. How do we talk about the collection, analysis and
use of student data when… (cont.)
• We already have a lot of data that is scattered all over the institution, in a
variety of formats, ranging in quality, collected for a variety of purposes
[with and without ethical clearance], accessed by a variety of staff for a
variety of purposes, and combined with other sources of data collected for
other purposes and used for (un)related needs
• We value quantitative data and ignore qualitative data; we celebrate big
data and ignore small data; and we value surface-level trends rather than
thick descriptions. We are obsessed with ‘what?’ and not ‘why?’
• Students don’t know that/how we collect, analyse and use their data and
don’t have access to their digital dossiers to question, verify, add context
and/or opt out
• The effectiveness and appropriateness of our responses to our analyses
depend on our own locus of control, our resources, our understanding of the
data and a political will [or lack of]
25

26. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
How do we collect, analyse and use student
data while recognising that their data are not
indicators of their potential, merit or even
necessarily engagement but the results of the
inter-generational impact of the skewed
allocation of value and resources based on race,
gender and culture?
26

27. A good place to start…
27

28. “Science [and technology] increases human
power – and magnifies the flaws in human
nature. It enables us to live longer and have
higher living standards than in the past. At
the same time it allows us to wreak
destruction – on each other and the Earth –
on a larger scale than ever before”
John Gray – Straw dogs (2002, p. xiii-xiv)
28

29. “… ‘educational technology’ needs to be
understood as a knot of social, political,
economic and cultural agendas that are riddled
with complications, contradictions and
conflicts”…
(Selwyn, 2014, p. 6)
If we accept that
…what are the implications for the
collection, analysis and use of student
data?
29

30. Data always mattered…
• The origins of the word "census" can be traced back to Rome
from the Latin word censere - "to estimate.“ Used to determine
taxes, counted ‘citizens’ (and defined citizens); used to
determine citizens’ and “others’”; used to allocate rights and
privileges; and used to assess the number of arms-bearing men
(sic) in preparation for war
• India, 300 BCE – Census during the reign of Emperor
Chandragupta Maurya
• China, ACE 2 – the Han Dynasty “one of the world's earliest
preserved censuses”
• Bible – a number of censuses – to determine taxes, to count
the number of ‘foreigners’ living among Israelites
30
Source: https://en.wikipedia.org/wiki/Census

31. Data = Power
Throughout the ages
…those who had the power, decided on what data was needed, for
what purposes, how the data was defined, what the data meant, who
had access to the data, and who would verify the correctness and the
meanings of the data
Data has always been used to
• Control
• Solve (or perpetuate) societal problems/horrors
• To allocate (or withhold) resources/support
• To safeguard the survival of those who collected the data; and
ensure adherence to the assumed rules and conventions determined
by those who collected the data
31

32. Examples include…
Image credit: http://commons.wikimedia.org/wiki/File:Penetentiary_Panopticon_Plan.jpg
‘Panopticon’
Jeremy Bentham,
1873
Greek mythology –
Argus Panoptes –
A giant with 100
eyes
32

33. Image source: https://www.mpiwg-berlin.mpg.de/en/news/features/feature14 Copyright
could not be established
• 1749 Jacques Francois
Gaullauté proposed “le
serre-papiers” – The
Paperholder – to King Louis
the 15th
• One of the first attempts to
articulate a new technology
of power – one based on
traces and archives
(Chamayou, n.d)
• The stored documents
comprised individual
reports on each and every
citizen of Paris
The technology will allow the sovereign “…to know
every inch of the city as well as his own house, he will
know more about ordinary citizens than their own
neighbors and the people who see them everyday (…)
in their mass, copies of these certificates will provide
him with an absolute faithful image of the city”
(Chamayou, n.d)
The Paperholder – “le serre papiers” (1749)
33

34. Image credit:
http://iconicphotos.wordpress.com/2010/07/29/the-
great-ivy-league-photo-scandal/
“… a person’s body, measured
and analysed, could tell much
about intelligence, moral worth,
and probably future achievement…
The data accumulated… will
eventually lead on to proposals to
‘control and limit the production of
inferior and useless organisms’”
(Rosenbaum, 1995)
The great Ivy League
photo scandal 1940-
1970
34

35. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Image credit: http://za.geoview.info/apartheid_museum_entrance,83879989p
The collection, analysis and use of data
are political acts and serve declared and
hidden assumptions about the purpose
of higher education and the masters it
serves
(Apple, 2004, 2007; Grimmelman, 2013; Watters, 2015)
35

36. Revisiting (our beliefs re)Data
Page credit: https://randomdatablog.com/2017/04/17/yes-data-can-lie/
“Data do lie on occasion. They can lie for a whole
bunch of reasons, from the simple to the complex.
The lies can begin at point of collection and continue
on through aggregation and analysis.”
36

37. “Data are like Play-Doh and can take all sorts of shapes
and dimensions. It can be worked and reworked for
endless variety. But, it can only stretch so far before it
breaks and becomes separate pieces. This is what
happens with data when you stretch the definition and
structure too far, original meaning is lost and the
provenance is broken. Small pieces can be lost during
this shaping, or blended with other “colours” creating
something new, but increasingly more abstract than
the original data.” (Tod Massa)
Page credit: https://randomdatablog.com/2017/04/17/yes-data-can-lie/
37

38. What student data do we currently have,
where is it stored, in what format, what is
the quality of this data, who has access to
the data under what circumstances, and
for what purposes?
38

39. Data from student
inquiries prior to
(application for)
registration -
telephonic, in-
person, via email, via
social media
Data gathered during the
registration process –
location of registration,
demographic/ socio-
economic data, prior
learning, uploaded (or not)
documents, financial status
LMS data – how long after
registration did they log in, what
resources did they download, how
often, who did they interact with
(if), for what purpose, what time of
day, for how long , how many
requests for password resets…
‘Class’ data –
attendance
registers, Clicker
data
Library data –
physical or online –
what resources
downloaded,
access, requested,
how often
Academic support
data – phone
calls/emails/visits to
lecturers/tutors
Administrative
support data – phone
calls/emails/visits – re
payments, assignment
marks, examination
dates, remarksAffective support
data – phone
calls/emails/visits to
support staff -
counselling advisory
services
Internal/external
requests for
student data – who
wanted to know
what, for what
purpose, under
what conditions,
ethical clearance
required?
Social media –
Twitter, Facebook,
Linkedin, Own Your
Own Domain
39

40. Where is this data stored, in what format,
what is the quality of this data, who has
access to the data under what
circumstances, for what purposes?
40

41. Student data [in all its varieties, and in different
combinations] are [currently] used to…
• Access - What are the assumptions about and the purpose of the data?
How sure are we that the data means what we think it means? What are
the purpose of controlling access – resources, placements, reputation?
• Allocation of resources – placements in programmes, staff: student ratios,
educational triage
• Personalisation of curricula, assessment, feedback, support and
shortened/extended programs
• Marketing
• Reporting – Internal/External
• Curriculum (re)development
• Quality assurance – who determines quality?
• Addressing strategic objectives – e.g. addressing the legacy of colonialism
and apartheid or dancing to the drum beat of the market?
41

42. Pointers for a way forward
• Students’ digital lives and our data sets are but a minute part of a bigger
whole – so we should not pretend as if our data represent the whole
• The data we collect are never ‘raw’, ‘uncontaminated’, or just ‘scraped’…
Our samples, choices, timing and tools change and impact on data. “Data
are in fact framed technically, economically, ethically, temporally, spatially
and philosophically. Data do not exist independently of the ideas,
instruments, practices, contexts and knowledges used to generate, process
and analyse them” (Kitchen, 2014, p. 2)
• Data have contexts. To re-use data outside of the original context and
purpose for which it was collected impacts on the contextual integrity.
• Knowing ‘what’ is happening, does not necessarily tell us the ‘why’…
• Education is an open, recursive system (Biesta 2007, 2010) where multiple
variables not only intersect but often also constitute one another. Let us
therefore tread carefully between correlation and causation…
42

43. Caught between correlation and causation
Image credit: http://www.tylervigen.com/spurious-correlations
43

44. Caught between correlation and causation
(cont.)
Image credit: http://www.tylervigen.com/spurious-correlations
44

45. The collection, analysis and use of student
data: some pointers
1. What are our (management, administrative, faculty and
support staff’s) beliefs about knowledge, learning, assessment,
data, and evidence?
2. What student data do we already have, why was it collected, in
which format is it stored, who has access to the data, how is
the data used by whom, and do students know this, have
access to it, and know how it influences our and their choices?
3. What data do students currently have access to about their
learning and about our choices pertaining to their learning?
45

46. The collection, analysis and use of student
data: some pointers (cont.)
4. What data don’t students currently have access to, but we
have, that will help them to plan their time and resources in
order to maximize their chances of success?
5. What student data don’t we have, but need in order to teach
better, allocate resources, and support students? Is this data
available, under what conditions will we be able to access it,
how will we govern its-- storage, combination with other
sources of data, who will have access to it and under what
conditions?
46

47. 1. There must be no personal-date record-keeping systems
whose very existence is secret.
2. There must be a way for an individual to find out what
information about him/her is in the record and how it is used.
3. There must be a way for an individual to prevent information
obtained about him/her for one purpose for being used or
made available for other purposes without his/her consent.
4. There must be a way for an individual to correct or amend a
record of identifiable information about him/her.
5. Any organisaton creating, maintaining, using, or disseminating
records of indentifiable personal data must assure the
reliability of the data for their intended use and must take
reasonable precautions to prevent misue of the data.
1973 Code of fair information practices
47

48. What are the ethical implications for the
collection, analysis and use of student (digital)
data?
1. The duty of reciprocal care
• Make TOCs as accessible and understandable (the latter may
mean longer…)
• Make it clear what data is collected, when, for what
purpose, for how long it will be kept and who will have
access and under what circumstances
• Provide users access to information and data held about
them, to verify and/or question the conclusions drawn, and
where necessary, provide context
• Provide access to a neutral ombudsperson
(Prinsloo & Slade, 2015)
48

49. What are the ethical implications …? (2)
2. The contextual integrity of privacy and data – ensure the contextual
integrity and lifespan of personal data. Context matters…
2. Student agency and privacy self-management
• The fiduciary duty of higher education implies a social contract of
goodwill and ‘do no harm’
• The asymmetrical power relationship between institution and
students necessitates transparency, accountability, access and
input/collaboration
• Empower students – digital citizenship/care
• The costs and benefits of sharing data with the institution should be
clear
• Higher education should not accept a non-response as equal to
opting in…
(Prinsloo & Slade, 2015)
49

50. What are the ethical implications …? (3)
4. Future direction and reflection
• Rethink consent and employ nudges – move away from
thinking just in terms of a binary of opting in or out – but
provide a range of choices in specific contexts or needs
• Develop partial privacy self-management – based on
context/need/value
• Adjust privacy’s timing and focus - the downstream use of
data, the importance of contextual integrity, the lifespan of
data
• Moving toward substance over neutrality – blocking
troublesome and immoral practices, but also soft,
negotiated spaces of reciprocal care
(Prinsloo & Slade, 2015)
50

51. • Knowing
• Not knowing
• Knowing what we don’t know
• Knowing what we may never know
• Knowing more?
The solution is not only (or necessarily?) in knowing more, but
ensuring that once we know, we respond in ethical, caring,
discipline and context-appropriate ways
What are the ethical implications of …
51

52. (In)conclusions
52
“Technology is neither good or bad; nor is it neutral…
technology’s interaction with social ecology is such that
technical developments frequently have
environmental, social, and human consequences that
go far beyond the immediate purposes of the technical
devices and practices themselves”
Melvin Kranzberg (1986, p. 545 in boyd & Crawford, 2012, p. 1)

53. Thank you
Paul Prinsloo (Prof)
Research Professor in Open Distance Learning (ODL)
College of Economic and Management Sciences,
Office number 3-15, Club 1, Hazelwood, P O Box 392
Unisa, 0003, Republic of South Africa
T: +27 (0) 12 433 4719 (office)
prinsp@unisa.ac.za
Skype: paul.prinsloo59
Personal blog: http://opendistanceteachingandlearning.wordpress.com
Twitter profile: @14prinsp
53

54. Useful resources
54
Source: https://press.anu.edu.au/publications/series/centre-aboriginal-economic-policy-research-caepr/indigenous-data-
sovereignty

55. Useful resources (cont.)
55
Source: http://www.nature.com/news/south-africa-s-san-people-issue-ethics-code-to-scientists-1.21684

56. Useful resources (cont.)
56
Source: http://journals.sagepub.com/doi/abs/10.1177/0002764213479366

57. Useful resources (cont.)
57
Source: http://oro.open.ac.uk/47251/

58. Useful resources (cont.)
58
Source: https://the-other-jeff.com/wp-content/uploads/2017/02/Structural-Justice-in-Learning-Analytics-ESS.pdf

59. Useful resources (cont.)
59
Source: http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005399

60. Useful resources (cont.)
60
Source: https://na-production.s3.amazonaws.com/documents/Promise-and-Peril_4.pdf

61. Useful resources (cont.)
61
Source: https://na-production.s3.amazonaws.com/documents/Predictive-Analytics-GuidingPractices.pdf

62. Useful resources (cont.)
62
Source: https://www.slideshare.net/abelardo_pardo/the-role-of-data-in-the-provision-of-feedback-at-scale