Collecting, measuring, analysing and using student data in open distance/distributed learning

Collecting, measuring, analysing and
using student data in open
distance/distributed learning
Image credit: https://www.flickr.com/photos/themonk/5948439988
Unisa UNESCO Chair on Open Distance Learning Series,
15 June 2019, University of South Africa
By Paul Prinsloo
14prinsp

Acknowledgement
I do not own the copyright of any of the images in this presentation. I
therefore acknowledge the original copyright and licensing regime of
every image used.
This presentation (excluding the images) is licensed
under a Creative Commons Attribution 4.0 International License.

Collecting data in higher education: scope
Higher
education
institution
Pre-
enrolment/
enquiries/
visits
Enrolment
Demographic
data
Pre-entry
academic
data
Engagement
with student
advising/
counselling/
finance/
faculty/
library
Courses
Learning
management
system
Engagement with
student advising/
counselling/
finance/ faculty/
library
Re-
enrolment/
enquiries/
visits
Post-
enrolment
From pre-enrolment to re/post-enrolment we collect data/they share data

We are not the only ones
collecting, measuring,
analysing student data

The world of (student) data
Academic Analytics
Learning Analytics
(Higher)
Education
• Individuals
• Corporates
• Governments
• Data brokers
• Fusion centers
• Directed
• Automated
• Gifted

How/why is the collection, measurement, analysis and
use of student data in distance/online/distributed
learning different/more important(?) than in
residential/traditional higher education?

Distance
education
institution
The impact of
distance:
• Place
• Space
• Time
• Emotional
• Flexibility/
• choice
Fully offline/ digitally supported/ internet supported/ internet dependent/ fully online*
* Department of Higher Education and Training. (2014). Policy for the provision of distance education in South African universities
in the context of an integrated post-school system. Retrieved from https://www.gov.za/ss/documents/higher-education-act-policy-
provision-distance-education-south-african-universities
Pedagogy
• Teaching period: 10-14 weeks
• Formative assessment?
• Summative assessment
• Guided pedagogic conversation
What data do we have/need to inform our teaching
and support/evaluate students’ learning?

Image credit: https://www.flickr.com/photos/codnewsroom/25137910512
What do you/we know about (y)our
students?

Image credit: https://pixabay.com/photos/bboy-breakdance-dancing-hip-hop-1450053/
Who at the institution, knows what about (y)our
students? How is this shared? For what purposes?
And if not, why not?

What don’t you/we know about (y)our
students?
Image credit: https://pixabay.com/photos/street-africa-ghana-city-streets-3644374/

What do you/we know about
• How they learn
• What times of the day they engage with their studies
• How often and for what purpose they engage with peers/faculty, etc.
Image credit: https://pixabay.com/photos/train-wagon-people-the-crowd-feet-2373323/

Why do you/we want/need to know about
them, their lives, their learning journeys?
Image credit: https://pixabay.com/photos/red-nose-color-splatter-joy-women-1675188/

Do they know you/we are collecting, measuring, analysing
their data and using that data to evaluate and support their
learning?
Imagecredit:https://pixabay.com/photos/sculpture-bronze-the-listening-2275202/

And under what conditions are the collection,
measurement, analysis and use of student data …

And if you know, what can/will you do
about the challenges they face?
Image credit: https://pixabay.com/photos/train-station-transportation-people-691176/

Image credit: https://www.deviantart.com/kovangfx/art/Being-alone-526166616
What information do they have, that if
they had a way or a reason to share it
with you/us, this will help you/us
to understand their learning
journeys better, support
them (more)
appropriately,
and evaluate
more fairly?

#Recap
1. What student (learning) data do we/you currently have?
2. Where/how are these data stored, in which formats, for what
purpose were/are they collected, by whom and who has access to
these data for what purposes?
3. What data don’t we/you have, that if we/you have access to
these data, if will help us/you to teach better?
4. What data do we/you currently use to teach better and make
more informed decisions, and if not, why not?
5. What data do students need to learn better, more make more
appropriate decisions?
6. What data do they have, that, if they would share it with us, can
help us to teach better and make more appropriate support
decisions?

Overview of the (rest of the)presentation
1. Mapping the evolution in the use of student data
2. How we use student data
3. Who/what is doing the collection, measurement, analysis of
and use student data?
4. How does this scale? The potential of algorithmic-decision
making systems
5. Making sense of my students’ data: two case studies
6. Using student data as surveillance: concerns and ethical
issues
7. Some principles to consider
8. (In)conclusions

(Higher) education has always collected, analysed and used
student data – so what has changed?
Image credit: https://upload.wikimedia.org/wikipedia/commons/7/79/A_Medieval_Classroom.jpg

Source credit: https://tekri.athabascau.ca/analytics/
“Learning institutions and corporations make little use of
the data learners "throw off" (sic) in the process of
accessing learning materials, interacting with educators
and peers, and creating new content” (emphasis added)

Source credit: https://tekri.athabascau.ca/analytics/
“Learning analytics is the measurement, collection,
analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimising
learning and the environments in which it occurs”.

Source credit: https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education

IN THE PAST AT PRESENT
Data sources Demographic and learning data at
specific points in the learning
journey: data application,
registration, class registers,
assignments, summative
assessment, personal
communication
Continuous, directed, gifted
and automated collection of
data from a range of data
sources – student
administration, learning
management system (LMS),
sources outside of the LMS
Data use Reporting purposes, operational
planning on cohort, group level by
management, institutional
researchers
Descriptive, diagnostic, predictive
and prescriptive on group/cohort
level
Plus individualised, often real-
time use of data to inform
pedagogy, curriculum,
assessment, student support by
faculty, students and support
staff
Who used the
data
(officially)?
Management, institutional
researchers, planners, quality
assurance and HR departments
Plus researchers, faculty,
students and support staff

IN THE PAST AT PRESENT
Who did the
collection,
analysis and
who used the
data
Humans Increasingly humans in
combination with algorithmic
decision-making processes
Temporal aim Retrospective/historical data to
make predictions with regard to
budget, future enrollments &
resource allocation on
institutional level
Plus real-time data for real-time
interventions
Default Forgetting Remembering
Personal
identifiers
Anonymised, aggregated data Plus re-identifiable data
Personal/ised data
Oversight/
data
governance
Broad institutional oversight.
Ethical Review Board (ERB)
approval for research purposes
Approval, oversight and
governance highly complex and
contested

We know, take into account and we measure: age,
gender, race, street address and zip code,
occupation, pre-enrolment educational data,
registration data, engagement data, academic data,
library data, financial aid data, behavioural data,
location data, who-are-in-their-networks-data,
their chances of failing, dropping out, stopping
out…
Image credit: https://pixabay.com/en/side-profile-black-male-student-1440176/
And we use this data to…
Image credit: https://pixabay.com/en/girl-library-education-student-1721436/

Source credit: http://timoelliott.com/blog/2013/02/gartnerbi-emea-2013-part-1-analytics-moves-to-the-core.html

Students (with a particular habitus and
demographics) apply to register, choose a
qualification and modules, and start…
Macro-societal factors, e.g. economic, political,
social, technological, environmental and legal
factors.
Institutional/lecturer inactions, or inefficiencies
impacting and shaping their behavioral data, their
chances of failing, dropping out, stopping out…

Macro-societal factors, e.g. economic, political,
social, technological, environmental and legal
factors.
Institutional/lecturer inactions, or inefficiencies
impacting and shaping their behavioral data, their
chances of failing, dropping out, stopping out…
Data
points

Processes
Inter & intra-
personal
domains
Modalities:
• Attribution
• Locus of control
• Self-efficacy
Processes
Modalities:
• Attribution
• Locus of
control
• Self-efficacy
Domains
Academic
Operational
Social
TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES
THE STUDENT AS AGENT
IDENTITY, ATTRIBUTES, HABITUS
Success
THE INSTITUTION AS AGENT
SHAPING CONDITIONS: (predictable as well as uncertain)
Choice,
Admission
Learning
activities
Course
success
Gradua-
tion
THE STUDENT WALK
Multiple, mutually constitutive
interactions between student,
institution & networks
F
I
T
FIT
F
I
T
FIT
Employ-
ment/
citizenship
TRANSFORMED STUDENT IDENTITY & ATTRIBUTES
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
Retention/Progression/Positive experience

Source credit: https://www.tandfonline.com/doi/abs/10.1080/01587919.2011.584846

Image credit: https://pixabay.com/illustrations/technology-programming-binary-robot-2062712/
Who/what is doing the collection, measurement,
analysis and how are these analyses implemented
and used?
Collect/measure/analyse
• Institutional analysts
• Researchers
• Faculty
• Support staff
• Administrators
• Management
• Algorithms
Use
• Researchers
• Faculty
• Support staff
• Administrators
• Management
• Students
• Algorithms

Image credit: https://pixabay.com/illustrations/technology-programming-binary-robot-2062712/
In an open distance and distributed learning
environment, with thousands of students registered
for any particular module, how does this scale?
Collect/measure/analyse
• Researchers
• Faculty
• Support staff
• Administrators
• Management
• Algorithms
Use
• Researchers
• Faculty
• Support staff
• Administrators
• Management
• Students
• Algorithms

(1)
Humans
perform the
task
(2)
Task is
shared with
algorithms
(3)
Algorithms
perform task:
human
supervision
(4)
Algorithms
perform task:
no human
input
Seeing Yes or No? Yes or No? Yes or No? Yes or No?
Processing Yes or No? Yes or No? Yes or No? Yes or No?
Acting Yes or No? Yes or No? Yes or No? Yes or No?
Learning Yes or No? Yes or No? Yes or No? Yes or No?
Danaher, J. (2015). How might algorithms rule our lives? Mapping the logical space of algocracy. [Web log post]. Retrieved from
http://philosophicaldisquisitions.blogspot.com/2015/06/how-might-algorithms-rule-our-lives.html
Can Artificial Intelligence (AI) and algorithmic decision-
making systems help?

Image credit: https://pixabay.com/photos/ceramic-clay-pottery-art-old-3050615/
Working with data: a personal journey

CASE STUDY 1: Fully Online (structured) German Masters
degree
Pedagogy
• 15 weeks
• 2 instructors, one support staff, 20 students
• Weekly structure with assigned readings, activities, discussion forum requirement
• In total: 3 tasks, 2 essays, 6 skill builder exercises, group work, structured learning
journal
• Students expected to actively participate in the class – at least one substantive posting
per week, plus respond in a substantive way to other students’ posts
Student data
• Gender
• Employment
• Educational background (to some extent)
• Last 20 logins
• Gifted information (“I hurt my back last week”)

Male
Mid-thirties
Little to no engagement
Average assignment
Female, mid-thirties
Contributes frequently
Excellent assignments
Female
Late forties
Little to no engagement
Failed first assignment

While I did not know what it meant, it did not
prevent me from responding…
Email sent on 23 March in
response to no activity for 5 days

Email sent on 31 March in
response to no activity for 5 days

Reflection: Case Study 1
• The learning/pedagogy design/engagement is immensely intensive for both
instructors and students
• Students could not afford to fall behind
• Students found the high level of engagement and amount of readings
overwhelming
• The only data I had access to was their last 20 logins, and information they
‘gifted’
• I wish some form of algorithmic automated system could have alerted me
when students did not login for three days
• While having access to login data proved immensely helpful in the context
of this course, looking for the data and responding to the data were also
very taxing and time-consuming
• Yes, the instructor: student ratio of 1:20 made a huge difference

CASE STUDY 2: Fully Online (structured) US Masters
degree
Pedagogy
• 15 weeks
• 2 instructors, one support staff, 20-25 students
• Weekly structure with assigned readings, activities, discussion forum requirement
• In total: 3 tasks, 2 essays, 6 skill builder exercises, group work, structured learning
journal
• Students expected to actively participate in the class – at least one substantive posting
per week, plus respond in a substantive way to other students’ posts
Student data
• Gender
• Employment
• Educational background (to some extent)
• A range of login data – number, time-on-task, responses, scores, progress, etc
• Gifted information (“My partner lost his job”)

Image credit: https://pixabay.com/en/person-listening-headphones-eyes-2157916/
What does this mean?

Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from
https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave-
behind/
Reflection: Case Study 2

Image credit: https://pixabay.com/en/buddha-statue-stone-statues-speak-546458/
If these data tell a story, whose story is it? Is this the
whole story? Who is listening? What do we miss?

Collecting student data: some
concerns
Image credit: https://pixabay.com/photos/monitoring-safety-surveillance-1305045/

Source credit: https://www.washingtonpost.com/news/the-switch/wp/2015/12/28/google-is-tracking-students-
as-it-sells-more-products-to-schools-privacy-advocates-warn/?noredirect=on&utm_term=.6c0bfa947fc2

Source credit:https://www.insidehighered.com/news/2018/10/09/coming-soon-ivy-league-campuses-free-coffee-privacy-not-included

Source credit: https://www.theguardian.com/higher-education-network/2016/aug/03/learning-analytics-universities-data-track-students
2016

Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Page credit: https://www.edsurge.com/news/2017-06-13-from-high-school-to-harvard-students-urge-for-clarity-on-privacy-
rights?utm_content=buffer8dd71&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
3 June 2017

Source credit: https://campustechnology.com/articles/2018/05/02/when-learning-analytics-violate-student-privacy.aspx
2018

Image credit: https://pixabay.com/en/travel-sculpture-man-stone-move-3034459/
Knowing more also means having greater
responsibility to make informed, tentative,
appropriate and ethical decisions

Ethics in learning analytics: Selected examples
2013-2017

Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral
Scientist 57(1), 1509–1528.

Source credit: https://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web-content/ethical-
use-of-student-data-policy.pdf
2014

Principle 1: Learning analytics is an ethical practice that should align with core
organisational principles, such as open entry to undergraduate level study.
Principle 2: The OU has a responsibility to all stakeholders to use and extract
meaning from student data for the benefit of students where feasible.
Principle 3: Students should not be wholly defined by their visible data or our
interpretation of that data.
Principle 4: The purpose and the boundaries regarding the use of learning
analytics should be well defined and visible.

Principle 5: The University is transparent regarding data collection, and will
provide students with the opportunity to update their own data and consent
agreements at regular intervals.
Principle 6: Students should be engaged as active agents in the implementation of
learning analytics (e.g. informed consent, personalised learning paths,
interventions).
Principle 7: Modelling and interventions based on analysis of data should be sound
and free from bias.
Principle 8: Adoption of learning analytics within the OU requires broad acceptance
of the values and benefits (organisational culture) and the development of
appropriate skills across the organisation.

Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: A typology
derived from a cross-continental, cross-institutional perspective. Educational Technology Research and
Development, 64, 881-901. DOI: 10.1007/s11423-016-9463-4 http://link.springer.com/article/10.1007/s11423-
016-9463-4
Who will provide oversight over the ethical
issues in learning analytics?
An interpretative multiple-case study: Indiana University, Open University (UK) and
the University of South Africa (Unisa)
2016

Source credit: https://www.siyaphumelela.org.za/documents/5a61c7b737ff5.pdf
2017

Guiding principles for an ethics of care:
Principle 1: The moral, relational duty of learning
analytics
Principle 2: Defining student success in the nexus of
student, institution and macro-societal agencies and
context
Principle 3: Understanding data as framed and framing
Principle 4: Student data sovereignty
Principle 5: Accountability
Principle 6: Transparency
Principle 7: Co-responsibility

Prinsloo, P. (2018). Include us all! Directions for the adoption of learning analytics in the Global South: An
African perspective. Digital Learning for Development (DL4D). Available at: http://dl4d.org/portfolio-
items/learning-analytics-for-the-global-south/

Image credit: https://pixabay.com/photos/compass-hand-travel-direction-1753659/
Some guiding
principles

“If you have come to help us, you can go
home. If you have come to accompany us,
please come. We can talk”
Glesne, C. (2016). Research as solidarity. In T. Kukutai and J. Taylor. (Eds), Indigenous data sovereignty. Toward an agenda (pp. 169-178). Canberra,
Australia: Australian National University Press. Retrieved from https://press.anu.edu.au/publications/series/centre-aboriginal-economic-policy-
research-caepr/indigenous-data-sovereignty
Principle 1: The moral, relational duty of learning analytics
Image credit: https://pixabay.com/en/sculpture-bronze-2196139/

Principle 2: Student success as
entangled
Image credit: https://pixabay.com/en/rope-knot-string-strength-cordage-3052477/

THE STUDENT AS AGENT
Success
THE INSTITUTION AS AGENT
THE STUDENT WALK
Multiple, mutually constitutive interactions between
student, institution & networks
FIT
FIT
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
F
I
T
Subotzky, G., & Prinsloo, P. (2011). Turning the tide: a socio-critical model and framework for improving student success in open distance
learning at the University of South Africa. Distance Education, 32(2): 177-19.

Principle 3: Understanding data as
broken and framed but also as acting
and framing
Image credit: https://pixabay.com/en/eyeglasses-broken-glasses-sight-366446/

Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an emerging
world. Big Data & Society, 5(1), 2053951717753228.
We have to recognize that
our data are broken

“Data is not necessarily accurate,
complete or full aggregated
representations of what individuals
or societal groups have done, or
able to predict what they will do”
(Pink et al., 2018, p. 10)
Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an emerging
world. Big Data & Society, 5(1), 2053951717753228.

Student data are not something separate from
students’ identities, their histories, their beings.
Data are an integral, albeit informational part of
students being.
In the light of the view that data are not
something students own but rather who they
are; what are we assuming when we say we
‘collect’ their data?
E.g. Floridi, L. (2005). The ontological interpretation of informational privacy. Ethics
and Information Technology, 7(4), 185-200.
Principle 4: Student data sovereignty

Ctrl Alt Del
• How much control do students have to determine
what data institutions harvest; to challenge the meaning
of their data and our categories of analysis?
• Can students (re)define/alter interpretations of data
and definitions/categories? Can they offer us counter-
narratives to our understanding of their learning and
their life-worlds?
• Can students opt out of personalised data collection,
analysis and use and have their data deleted?

Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: A typology derived from a cross-
continental, cross-institutional perspective. Educational Technology Research and Development, 64, 881-901. DOI: 10.1007/s11423-
016-9463-4 http://link.springer.com/article/10.1007/s11423-016-9463-4
Principle 5: Accountability
An interpretative multiple-case study: Indiana University, Open
University (UK) and the University of South Africa (Unisa)

Principle 6: Transparency
If they don’t know that we collect their data, the scope and
purpose of the collection, how we will use their data and how
it will impact on their learning journeys, how is this ethical?
Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2209152/

Principle 7: Co-responsibility
Our students’ journeys are intimately weaved into our
(institutional) stories. What happens in their lives,
impact ours. And vice versa.
Image credit: https://pixabay.com/en/weave-hand-labor-samoa-exotic-55/

Student data are an invitation to start a
conversation
Image credit: https://pixabay.com/photos/man-sculpture-art-wonders-talk-1483479/
Image credit: https://pixabay.com/photos/man-sculpture-art-wonders-talk-1483479/
Principle 8: Student data as conversation

Principle 9: You cannot unknow

Prinsloo, P., & Slade, S. (2017, March). An elephant in the learning analytics room: the obligation to act. In
Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 46-55). ACM.

Prinsloo, P., & Slade, S. (2014). Educational triage in higher online education: walking a moral tightrope.
International Review of Research in Open Distributed Learning (IRRODL), 14(4), 306-331.
http://www.irrodl.org/index.php/irrodl/article/view/1881

Image credit: https://pixabay.com/en/stairs-boots-old-shoes-1683118/
Collecting, analysing and using student data may
assist them and us, to make more appropriate, and
responsible choices

Imagecredit:https://pixabay.com/en/sculpture-bronze-child-boy-1392529/
Thank you
Paul Prinsloo (Prof)
Research Professor in Open Distance
Learning (ODL), University of South
Africa, Unisa,
T: +27 (0) 12 433 4719 (office)
prinsp@unisa.ac.za
Skype: paul.prinsloo59
Personal blog:
http://opendistanceteachingandlearning.wordpress.com
Twitter profile: @14prinsp

Collecting, measuring, analysing and using student data in open distance/distributed learning

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to Collecting, measuring, analysing and using student data in open distance/distributed learning

Similar to Collecting, measuring, analysing and using student data in open distance/distributed learning (20)

More from University of South Africa (Unisa)

More from University of South Africa (Unisa) (20)

Recently uploaded

Recently uploaded (20)

Collecting, measuring, analysing and using student data in open distance/distributed learning