Tutorial on qualitative approaches to learning analytics given by Rebecca Ferguson of The Open University UK at the Learning Analytics Summer Institute (LASI) run by the Society for Learning Analytics Research (SoLAR) at the University of British Columbia (UBC) in Vancouver, Canada, on 17 June 2019

1. Qualitative approaches
to learning analytics
Rebecca Ferguson
The Open University, UK
Learning Analytics
Summer Institute
Vancouver, June 2019

2. Why focus on
qualitative?

3. Learning analytics help us
to identify and make sense
of patterns in the data
to improve our teaching,
our learning and
our learning environments

4. Why do
learners and
teachers act
as they do?
What do
we know
about the
contexts in
which
learning
analytics
are
employed?
How can we
increase the value
of learning analytics
tools and methods?
What do learners and
teachers want from
learning analytics?

5. Latent Dirichlet Allocation
Word six
Word five
Word four
Word seven
Qualitative Analysis
Used top words to identify initial themes
Looked at top 10 responses
Then updated themes, adding new aspects
Document
One (D1)
Word three
Word two
Word one
Document
Two (D2)
Used this method to
identify underlying topics
and themes from the
responses
Document
Three (D3)
Word ten
Word nine
Word eight
D1
Topic 1
Word 1
Word 2
Word 3Topic 2
D2
Topic 3
Topic 4
Word 4
Word 5
Word 6
Word 7
D3
Topic 5
Topic 6
Word 8
Word 9
Word 10
Word four
Word five
Word six
Word seven
We conducted quantitative
and qualitative studies to
examine systematically the
algorithms’ acceptance,
accuracy and impacts.
The feedback was positive.
Here are two examples.
Transcripts from all the
focus groups were
then created and
common themes were
identified
Summaries of
responses are
given below

6. ACTIVITY
• Think of a time when you [could] have used a
qualitative approach as part of your work on
learning analytics.
• Share your example with those on your table.
• Do these examples have any common features?
When aren’t numbers enough?

7. What is a qualitative approach?
There exists a fundamental distinction between
two types of data: qualitative and quantitative.
The way we typically define them, we call data
‘quantitative’ if it is in numerical form and
‘qualitative’ if it is not.
https://www.simplypsychology.org/qualitative-quantitative.html

8. What is a qualitative approach?
The question is not whether you are using a
mixture of numerical and non-numerical data, but
how that data is being viewed.
Within a qualitative methodology both numerical
and non-numerical data are viewed in the same
way; all data is a symbolic representation, which
needs to be interpreted and thus its meaning is
subjective and context dependent.
Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C.-C. (2017). Some guidance on
conducting and reporting qualitative studies. Computers & Education, 106, A1-A9.

9. ACTIVITY
• When are these statements correct?
• What do you need to know about context in order
to decide whether these statements are correct?
Putting figures in context
(a) l + l = l0
(b) l + l = ll
(c) l + l = l
(d) l + l = 2

10. Nomothetic = relating to the study or discovery of general scientific laws
Hermeneutic = concerned with interpretation
Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C.-C. (2017). Some guidance on
conducting and reporting qualitative studies. Computers & Education, 106, A1-A9.

11. A qualitative approach in LA (1)
RQ1: What are the individual perceptions of 20 educational
managers within a distance learning HE institution about
the adoption of predictive learning analytics in the
organisation?
RQ2: What are the challenges that may inhibit wider
adoption of PLA across the institution?
We need more
information to know
how and when to
use it best
The tools need to be
able to sell
themselves
Using it on a voluntary
basis does not work... we
need to be clear on how
much easier we can
make life for teachers
with it
Plethora of data
and sometimes
contradictory
outputs
Herodotou, C; Rienties, B; Verdin, B; and Boroowa, A.(2019). Predictive learning analytics ‘at scale’: Guidelines to successful
implementation in Higher Education based on the case of The Open University UK. Journal of Learning Analytics, 6(1) pp. 85–95.

12. Methods
• Research questions identified
• Method of data gathering
identified and justified
• Sample described and justified
• Broad topics identified
• Sample questions provided
• Data collection clarified
• Method of data analysis
identified: thematic analysis
• Themes presented clearly
• Explained how interpretations
were checked
Herodotou, C; Rienties, B; Verdin, B; and Boroowa, A.(2019). Predictive learning analytics ‘at scale’: Guidelines to successful
implementation in Higher Education based on the case of The Open University UK. Journal of Learning Analytics, 6(1) pp. 85–95.

13. What
would your
superpower
be?
Holstein et al, Journal of Learning Analytics, in press

14. A qualitative approach in LA (2)
Initial needs analysis and concept generation, including design
activities such as generative card sorting exercises, directed
storytelling, semi-structured interviews with teachers, and field
observations
Initial concept validation via speed dating sessions with teachers
Iterative lower-fidelity prototyping, gradually increasing the fidelity
of both prototypes and simulated use contexts, using methods such
as role-playing and bodystorming exercises, participatory sketching
and comicboarding, and behavioral mapping
Iterative higher-fidelity prototyping with replay-based simulation
exercises, using Replay Enactments
Iterative classroom piloting and experimental evaluation, using
field observations, pre-post assessments of student learning, semi-
structured interviews with teachers and students, and behavioral
mapping.
Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool
to support teacher-AI complementarity. Journal of Learning Analytics.

15. Methods used
Generative card sorting exercises Come up with superpower ideas,
sort by subjective priority, think aloud about reasoning. Can add or
align with cards from other teachers
Directed storytelling Talk through specific recent events. Refflect on
how challenges could be overcome
Speed dating sessions Rapidly explore possible futures to probe the
boundaries of what individuals will find acceptable
Role playing Bringing future scenarios to life
Bodystorming exercises Design questions are asked in relevant
locations where activities can be acted out
Participatory sketching Sketching out new ideas
Comicboarding Leaving the final panel(s) of a story blank
Behavioral mapping Observing and recording activity in a certain
place or over a certain period of time
Replay Enactments Use authentic sets of data and algorithms to
prototype a scenario

16. Detailed account of analysis
Two researchers then worked though transcriptions of
approximately 5 hours of video and audio recorded
interviews, to synthesize design findings using two
standard techniques from Contextual Design:
interpretation sessions and affinity diagramming (Beyer &
Holtzblatt, 1997; Hanington & Martin, 2012).
Interpretation sessions are aimed at helping design
teams develop a shared understanding of collected
interview and think-aloud data, by collaboratively
extracting quotes representing key issues.
Affinity diagramming is a widely used, bottom-up
synthesis method, aimed at summarising qualitative
patterns across study participants’ responses, by
iteratively clustering participant quotes into successively
higher-level themes (Beyer & Holtzblatt, 1997; Hanington
& Martin, 2012).
Following several interpretation sessions, the resulting
301 extracted quotes were iteratively synthesized into 40
level-1 themes, 10 level-2 themes, and 4 level-3 themes.
Size and
scope of the
dataset
Methods of
analysis
explained,
referenced
and justified
Broad
overview of
the outcomes
Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool
to support teacher-AI complementarity. Journal of Learning Analytics.

17. Making themes clear to readers
• Help me to intervene where, when, and with what I am
most needed.
• Make sure the technology does not draw my attention
away from my students!
• How can I know whether what I’m doing is actually
working?
• Help me understand the “why”, not just the “what”.
• I’m just one person: help ease my load.
• But how do I judge whether my students are really
doing well?
• Help me monitor and manage student motivation.
• What can you tell me about my students that I do not
already know?
• Allow me to customize the technology to meet my
needs.
• Allow me to override the technology.
Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool
to support teacher-AI complementarity. Journal of Learning Analytics.
Teachers want
to stay in control
and provide
value.
They want
analytics that
provide
information they
don’t already
know.
They are
concerned that
analytics could
do more harm
than good

18. Trying a qualitative approach
Note: Qualitative data analysis is typically a lengthy
process. The approach used here has been selected as a
structured approach that can be trialled quickly in the time
available using data generated by participants.
Scupin, R., 1997. The KJ method: A technique for analyzing data derived from Japanese ethnology.
Human Organization, pp.233-237. (Method developed by Jiro Kawakita).
The KJ Method (similar to affinity diagramming)
• Identify the problem and state it clearly to the group
• Create labels – one idea per post-it note
• Randomise the ideas (shuffle them)
• Work together to group the ideas into teams that fit together
• Give each team a title
• Group the teams into families (no more than ten)
• Give each family a title
• Pattern them into a chart, indicating relationships between them
• Explain the chart in words.

19. ACTIVITY
What are the barriers to the use of qualitative
approaches in learning analytics
and how can they be overcome?
Trying a qualitative approach (1)
Use a thick
pen in a dark
colour, so
your idea is
visible
https://medium.com/design-research-methods/how-to-use-post-it-notes-9ca0904a03d1
One idea,
concept or
question per
Post-It note
Be concise
The more
ideas the
better

20. ACTIVITY
Building the teams
Trying a qualitative approach (2)
Group the
ideas that go
together in
‘teams’
Shuffle the
notes
Give each
team a label
Discuss your
decisions and
share
responsibility

21. ACTIVITY
Building the families; creating a chart
Trying a qualitative approach (3)
All your ideas
need to be
associated
with a family
Group the
teams into
bigger
families
No more than
ten families
Represent
the
relationships
between the
families
If there is time, explain your joint interpretation of the data

22. Trying the KJ Method
https://uxdict.io/design-thinking-methods-affinity-diagrams-357bd8671ad4
Note: Several steps are missing here. No consideration of sample
size and selection, briefing, ethics, or original choice of question.
• Provides a structured way of dealing with disparate data
• The route to analysis can be explained clearly
• Data are originally given equal weightings. No voice is privileged
• Rapid means of organising ideas and information
• Enables analysts to work together
• Does not automatically privilege interpretation of one analyst
• Makes analysis visible to others
• Analysts can compare interpretations
• Analysts can relate interpretations to each other
Affinity diagramming typically applies this approach to a much
broader set of data, first breaking it down into separate concepts.

23. Carrying out
qualitative analysis

24. Case study is
a method.
A case is not
automatically a
case study.
Photo by
Marvin Esteve
on Unsplash
Grounded theory is a
complex set of methods.
It isn’t simply a way of
saying your codes are
grounded in your data
Themes do not emerge
from the data. They are
generated by the
researcher(s)
Don’t reference
Glaser, Corbin,
Charmaz, or
Strauss unless you
have read them.
Their views change
over time and they
don’t agree.

25. First, transcribe your data
Audibly
breathes in
T1 Liz: Anyway, I er was hopin’ I could pop by tomorrow (.) if possible
T2 (2.0)
T3 Liz: Probably not, hey
T4 George: .hh. =
T5 Liz: =That's OK?
T6 George: Yeah
T7 Liz: ?I won’t stay long?
Very short gap
Non-word
Representation of accent
0.1 sec gap
2 sec gap
At the same time
In a lower tone
Direction of gaze Facial expression
Gesture
People don’t
naturally speak in
punctuated
sentences.
Words only make up
part of our
communication.
Acknowledge the decisions you have made

26. Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
Thematic analysis
Thematic analysis is a
method for identifying,
analysing and reporting
patterns (themes) within
data. It minimally organizes
and describes your data set
in (rich) detail. However,
frequently if goes further
than this, and interprets
various aspects of the
research topic
Inductive
(working from the data up)
Deductive
(theory led)
Experiential
(how people report their lives)
Critical
(what is going on in the data)
[Critical] realist
(Insight to the truth)
Constructionist
How topic is framed and understood
You need to make
decisions and justify them

27. Thematic analysis
Straightforward analysis
• Describes
• Summarises
• Gives voice
Sophisticated analysis
• Tells a story
• Interprets
• Makes an argument
• Locates data and
participants within wider
– social
– cultural
– historical
– political
– ideological context
Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY

28. Reflexivity
As a researcher, you engage with the data
You make conscious decisions
Reflexivity is an ongoing process
• What are you doing?
• Why are you doing it?
• What are your assumptions?
Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
Ask people to check, and query, your interpretations

29. Immersion
Immerse yourself
in your data
Read
• actively
• analytically
• critically
Identify things of
interest
Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
Immersion
Reflect on what you bring to your interpretation

30. Generate codes
A code is a label that captures
something of interest
• It should include a few words
• It should be able to stand alone
Code your data comprehensively
and systematically
Then repeat the process
End this phase with a list of codes
plus the data connected with them
Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
Semantic codes
capture surface
meaning
Latent codes
capture implicit
meaning
and assumptions

31. Generate themes
Another active stage
• Organise the codes into themes
• What are the bigger patterns of meaning that cut across
the dataset?
• Cluster similar codes together
• Go from the codes to the coded data and check
• Don’t just summarise
• Good themes are distinctive and part of a larger whole

32. Review, define, name themes
What is the name of this theme?
What is the quality of this theme?
What are the boundaries of this theme?
Are there enough data to support this theme?
Are the data too diverse?
Does the theme work in relation to the extracts?
Does it work in relation to the whole dataset?
What is the overall story of the analysis?
Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
Six themes
are
probably
enough
Be prepared to let things go!

33. Produce the report
Analysis should include:
• commentary
• data extracts
• themes
Look for vivid and compelling examples for each theme –
not all from the same eloquent person!
Relate analysis to
• research questions
• wider literature
• wider context
Thematic analysis: an introduction (June 2018) https://www.youtube.com/watch?v=5zFcC10vOVY

34. ACTIVITY
Identify some semantic codes (based on what the students say)
that could be applied to these data
Identify some latent codes (based on what the students
assume or imply) that could be applied to these data
Semantic and latent themes
Comments
from students
studying
Creative
Writing
Looking for
ideas about
analytics to
support forum
use
What they
liked or didn’t
like about the
course /
forums
Data are fake
but based on
real data

35. Trustworthy research
Quantitative: Is this generalisable?
Empirical: this setting is typical
Explanatory: this explains what happens in cases with
similar characteristics
Theoretical: there are necessary relationships between
sets of phenomena
Qualitative: Is this transferable?
Are these insights helpful in other contexts?
Does this rich account add to our understanding of a
broader picture?

36. Trustworthy research
Quantitative: Is this reliable?
Would these measures give the same results in the same
circumstances?
Qualitative: Is this credible?
Is your line of reasoning clear and transparent? Do your
conclusions make sense to participants?
Qualitative: Is this plausible?
Have you shown clear and multiple connections with the
data? Have you made a persuasive case for the ways in
which you engaged with the data?
Qualitative (more positivist stance): Is this reliable?
Can you demonstrate strong inter-rater reliability or
intra-rater reliability?

37. Trustworthy research
Quantitative: Is this valid?
How well does this piece of research reflect the
reality that it claims to represent?
Qualitative: Is this trustworthy?
Have you handled the data carefully? How can you
make it clear you have done a good job of capturing
the views of others?
Qualitative: Is this dependable?
Does the research account for the setting in which it
takes place?
Qualitative: Is this confirmable?
Can your results be confirmed or corroborated by
others?

38. Why do
learners and
teachers act
as they do?
What do
we know
about the
contexts in
which
learning
analytics
are
employed?
How can we
increase the value
of learning analytics
tools and methods?
What do learners and
teachers want from
learning analytics?

39. slideshare.net/R3beccaF
r3beccaf.wordpress.com/
twitter.com/R3beccaF