APL: Academic & Professional Literacies Forum Talk
1. USING LEARNER ANALYTICS
TO SUPPORT THE ACADEMIC WRITING
IN HIGHER EDUCATION
DUYGU SIMSEK
Academic and Professional Literacies Forum, The Open University, UK 07th May, 2014
people.kmi.open.ac.uk/simsek
duygu.simsek@open.ac.uk
simsekduygu_
Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo
Dr. Ágnes Sándor, Xerox Research Centre Europe
2. ABOUT ME
Ankara, Turkey
BA&MA in Computer and Instructional Technologies
Bilkent University, Turkey
MSc. Software Engineering
University of Southampton, UK
2nd year PhD Research Student KMi
The Open University, UK
Duygu Simsek, APL Forum Talk
2
3. OUTLINE –
Using Learner Analytics to Support
the Academic Writing in Higher Education
Research Aim
Where this research sits?
Academic Writing
Learning Analytics
Computational Text Analysis
Research Questions
Research Methods
Duygu Simsek, APL Forum Talk
3
4. RESEARCH AIM
To investigate
whether computational techniques can automatically identify
the attributes of good academic writing in as correlated with
grades of the essay and as identified in the literature
if this proves possible, how best to feed back actionable
analytics to support students and educators
whether this feedback has any demonstrable benefits
Duygu Simsek, APL Forum Talk
4
5. WHERE THIS RESEARCH SITS?
Duygu Simsek, APL Forum Talk
5
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
6. WHERE THIS RESEARCH SITS?-
ACADEMIC WRITING
Key aim of academic
writing is to convince
readers about the validity
of the claims and
arguments put forward
through an effective
narrative.
Duygu Simsek, APL Forum Talk
6
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
7. WHERE THIS RESEARCH SITS?-
META-DISCOURSE
This effective narrative is
signalled through meta-
discourse!
Duygu Simsek, APL Forum Talk
7
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
8. META-DISCOURSE
Meta-discourse refers to the features of text that convey the author’s intended
meaning and intention. It provides cues to the reader which explicitly express a
viewpoint, argument and claim, and signals the writer's stance.
Duygu Simsek, APL Forum Talk
8
Fig. 1 Meta-discourse that convey summary statements
CuestoSummary
statements
9. EXAMPLES OF META-DISCOURSE CUES THAT
SIGNAL ACADEMIC/ANALYTICAL RHETORICAL MOVES
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
Duygu Simsek, APL Forum Talk
9
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
10. WHERE THIS RESEARCH SITS?-
META-DISCOURSE
In order to assess students’
writing therefore, educators
will be examining students’
use of meta-discourse which
make their students’ thinking
visible.
However, students find it
challenging to learn to write
in an academically sound
way.
They need to learn how to
make their thinking visible by
recognising and deploying
meta-discourse.
Duygu Simsek, APL Forum Talk
10
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
11. WHERE THIS RESEARCH SITS?-
COMPUTATIONAL TEXT ANALYSIS
Meta-discourse cues
are automatically
identifiable.
This PhD investigates
whether it is possible to
provide automatic
meta-discourse analysis
of student writing
through the use of a
particular rhetorical
parser, XIP.
Duygu Simsek, APL Forum Talk
11
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
12. EXAMPLE OF A RHETORICAL PARSER:
INCREMENTAL PARSER (XIP)
Natural Language Processing (NLP)* product which includes a
rhetorical parser detecting meta-discourse in academic texts.
XIP extracts salient sentences based on their rhetorical functions:
Background Knowledge
Summarising
Tendency
Novelty
Significance
Surprise
Open Question
Contrasting Ideas
Duygu Simsek, APL Forum Talk
12
*language processing by computers that enable computers to derive meaning from natural language input
14. RHETORICAL FUNCTIONS CLASSIFIED BY XIP
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
Duygu Simsek, APL Forum Talk
14
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
16. WHY XIP? –
KEY FEATURES OF ACADEMIC WRITING?
Duygu Simsek, APL Forum Talk
16
Relevance
Understanding & Knowledge
Structure & Organisation
Linguistic Accuracy
Illustrations
Referencing
Argumentation
17. WHY XIP?
There is a mapping between good and strong features of academic
writing and the XIP’s rhetorical functions.
Duygu Simsek, APL Forum Talk
17
18. WHERE THIS RESEARCH SITS?-
Learning Analytics
XIP is a parser with
potential, if it can be
embedded in a more
complete learning
analytics (LA) approach.
Duygu Simsek, APL Forum Talk
18
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
19. WHERE THIS RESEARCH SITS?-
Learning Analytics (LA)
Learning moves online (i.e. pure online courses –OU, MOOCs –
FutureLearn, Coursera)
Digital learners leave data trails in online activities
Learner-produced online student data (i.e. data from
Learning Management Systems)
Collecting traces left behind to improve learning
LA is “measurement, collection, analysis and reporting of data
about learners and their contexts, for purposes of understanding
and optimising learning and the environments in which it
occurs”.[1]
Duygu Simsek, APL Forum Talk
19
[1] Call for Papers of the 1st International Conference on Learning Analytics & Knowledge (LAK 2011). Retrieved 12 February 2014.
20. WHERE THIS RESEARCH SITS?-
Learning Analytics (LA)
LA stakeholders can be educators, learners and administrators.
educators can learn about student activities and progress
learners can get feedback about their own progress,
understand their own learning habits and clearly see the
impact of the activities helping them more
administrators can make departmental and institutional
level adjustments based on the data collected and
analysed
The most common use of LA is to spot learners who appear to
fail and to make interventions in order to help those students.
Duygu Simsek, APL Forum Talk
20
21. WHERE THIS RESEARCH SITS?-
Discourse-centric Learning Analytics
How should a DCLA
approach be validated?
Duygu Simsek, APL Forum Talk
21
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
(DCLA)
Meta-
discourse
in Student
writing
22. MAIN RESEARCH QUESTION
Duygu Simsek, APL Forum Talk
22
To what degree can computational text analysis
and visual analytics be used to
support the academic writing of
students in higher education?
23. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
23
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
RQ1
24. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
24
RQ1
XIP Highlighted Student Writing
Any correlation between
the final grade of writing & XIP findings?
Pearson for
Total number of salient sentences vs. Grade
Generalised Multiple Regression
How strongly each rhetorical sentence type
influences the final grade
Grades
25. The OU’s S288 Practical Science
1st year undergraduates
5 strands:
Physics and Astronomy
Chemistry and Analysis
Environmental Sciences
Earth and Environment
Biology and Health
Collaborative scientific writing assignment
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh
July 5, 2013
DATA
Duygu Simsek, APL Forum Talk
25
26. ~300 1st year undergraduate students in 2012 (33 reports)
~600 in 2013 (69 reports)
Not a meaningful correlation between total number of
sentences found and grades.
Weak negative correlation between summary & grades.
Pearson: -0.28
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh
July 5, 2013
PRELIMINARY RESULTS
Duygu Simsek, APL Forum Talk
26
27. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
27
RQ1
What is the overlap
between XIP’s output and
how tutors judge quality?
Tutor Highlighted Student WritingXIP Highlighted Student Writing
28. IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END
USERS (STUDENTS AND EDUCATORS)?
Duygu Simsek, APL Forum Talk
28
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
RQ2
29. IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END
USERS (STUDENTS AND EDUCATORS)?
1st Year
Pilot study
Duygu Simsek, APL Forum Talk
29
RQ2
1 min. Intro. Video: http://goo.gl/WGIkDs
5 mins. Demo: http://goo.gl/Km1di5
30. TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S
ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK
WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?
Duygu Simsek, APL Forum Talk
30
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
RQ3
31. TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S
ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK
WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?
Duygu Simsek, APL Forum Talk
31
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
RQ3
32. TO WHAT EXTENT DO STUDENTS VALUE THE RESULTS OF XIP’S
ANALYSIS AS FORMATIVE FEEDBACK ON THEIR WRITING?
Duygu Simsek, APL Forum Talk
32
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
What students think
RQ4
33. YOUR VIEW ON…
1. Your view on the distilled summary of key features of academic
writing & argumentation.
2. Any graded student writing you have or any OU module that you
can suggest me to contact for data collection?
3. Any feedback on my studies with ALs?
Duygu Simsek, APL Forum Talk
33