Presentation for ECTEL 2015, Toledo, Spain (the detailed version). The related, shorter, presentation is at http://www.slideshare.net/dougclow/moving-through-moocs

1. Moving through MOOCs: Pedagogy,
Learning and Patterns of Engagement
Rebecca Ferguson, Doug Clow (OU)
Russell Beale, Alison J Cooper (Birmingham)
Neil Morris (Leeds)
Siân Bayne, Amy Woodgate (Edinburgh)

2. Current context
Students seek not
merely access, but
access to success
“
John Daniel, 2012
% complete from: www.katyjordan.com/MOOCproject
”

3. Patterns of engagement: Coursera
● Sampling
learners explored some course
materials
● Auditing
learners watched most videos, but
completed assessments rarely, if at all
● Disengaging
learners completed assessments at the
start of the course and then reduced
their engagement
● Completing
learners completed most assessments
MOOC designers can apply this
simple and scalable
categorization to target
interventions and develop
adaptive course features
“
”
Coursera study Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement:
analyzing learner subpopulations in massive open online courses. LAK13

4. 4
Replication
Open University FutureLearn data
Replication
MOOC1 MOOC2 MOOC3 MOOC4
Subject area Physical
sciences
Life
sciences
Arts Business
M 51% 39% 32% 35%
F 48% 61% 67% 65%
Participants 5,069 3,238 16,118 9,778
Fully participating 1,548 684 3,616 1,416
Participation rate 31% 21% 22% 14%

5. 5
Calculating an activity profile
Replicating the method
● T = on track (3)
undertook the assessment on time
● B = behind (2)
submitted the assessment late
● A = auditing (1)
engaged with content but not
assessment
● O = out (0)
did not participate
Replication

6. 6
Replication
Identifying dissimilarity between engagement patterns
Assigned numerical value to each label
● On track = 3
● Behind = 2
● Auditing = 1
● Out = 0
Calculated L1 norm for each
engagement pattern
Used that as the basis for
one-dimensional k-means clustering
Repeated clustering 100 times
and selected solution with
highest likelihood
Focused on extracting four clusters
Replication

7. 7
Replication
Coursera and FutureLearn results were different
● Sampling
learners explored some course materials
● Auditing
learners watched most videos, but completed assessments
rarely, if at all
● Disengaging
learners completed assessments at the start of the course
and then reduced their engagement
● Completing
learners completed most assessments
√
√
x
x
They also differed when we tried
●Different values for k
●A one-dimensional approach
●Running k means directly on engagement profiles
Replication

8. 8
FutureLearn is different
Conversation is a central feature
Sharples, M., & Ferguson, R. (2014). Innovative Pedagogy at Massive Scale:
Teaching and Learning in MOOCs. ECTEL 2014.

9. 9
Revising the numeric values
OU FL study
1 only visited content (for example, video, audio, text)
2 commented but visited no new content
3 visited content and commented
4 did the assessment late and did nothing else that week
5 visited content and did the assessment late
6 did the assessment late, commented, but visited no new content
7 visited content, commented, late assessment
8 assessment early or on time, but nothing else that week
9 visited content and completed assessment early / on time
10 assessment early or on time, commented, but visited no new content
11 visited, posted, completed assessment early / on time

10. 10
Typical engagement profiles
These profiles apply to an eight-week course
● Samplers visit only briefly
[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content
● Strong Starters do first assessment
[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time
● Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]
● Mid-way Dropouts
[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late
● Nearly There drop out near the end
[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time
● Late Completers finish
[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late
● Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]
OU FL study

11. 11
Samplers & Strong starters
Samplers (1, 0, 0, 0, 0, 0, 0, 0)
● The largest group in all MOOCs
● Typically accounted for 37% – 39% of
learners
● Visited the materials, but only briefly
● Active in a small number of weeks
● 25% – 40% joined after Week 1
● Very few Samplers posted
comments (6% – 15%)
● Almost no Samplers submitted
any assessment
Strong starters (9, 1, 0, 0, 0, 0, 0, 0)
●All Strong Starters submitted the first
assignment
●Engagement dropped off sharply after that
●A little over a third of them posted
comments
●Typically posted fewer than four comments
OU FL study

12. 12
Returners & Mid-way dropouts
Returners (9, 9, 0, 0, 0, 0, 0, 0)
● Completed the assessment in the
first week
● Completed the assessment in the
second week
● Then dropped out
● Over 97% completed those two
assessments, although some
submittted late
● No Returner explored all
course steps
● Average amount of steps visited
varied (23% – 47%)
Mid-way dropouts (9, 9, 9, 4, 1, 1, 0, 0)
●A much smaller cluster (6% of learners on
MOOC1, 7% on MOOC4)
●These learners completed three or four
assessments
●They dropped out around halfway
through the course
●Mid-way dropouts visited about half
the steps on the course
●Just under half posted comments
●Posted just over six comments on average
OU FL study

13. 13
Nearly There
Nearly there (11, 11, 9, 11, 9, 9, 0, 0)
● Another small cluster (5% – 6% of learners)
● Consistently completed assessments
● Dropped out just before the end of the course
● Visited around 80% of the course
● Submitted assignments consistently (>90%) and
typically on time until Week 5
● Activity then declined steeply
● Few completed the final assessment
● None completed the final assessment on time
OU FL study

14. 14
Late completers & keen completers
Late completers (5, 5, 5, 5, 5, 9, 9, 9)
● Submitted the final assessment
● Submitted most other assessments
● However, either submitted late or
missed some assessments
● Each week, more than 94% of this
cluster submitted their assessments
● More than three-quarters submitted
the final assessment on time
(78% – 90%)
● Around 40% of them posted
comments (76% did so on MOOC3)
Keen completers (11, 11, 9, 9, 11, 11, 9, 9)
●Accounted for 7% – 23% of learners
●All Keen Completers submitted all
assessments
●More than 80% of these were submitted
on time
●Typically, Keen Completers visited more
than 90% of course content
●Over two-thirds contributed comments
(68% – 73%)
●Mean number of comments varied from
21 to 54
OU FL study

15. 15
Cross-university dataset
FutureLearn data from four universities
Cross-university
Name Duration University Discipline Active
learners
LongMOOC1 8 OU Hard
science
5,069
LongMOOC2 7 Edinburgh Hard
science
10,136
TalkMOOC3 6 Edinburgh Politics 6,141
ShortMOOC4 3 Birmingham Medical
science
6,839
ShortMOOC5 3 Leeds Medical
science
4,756

16. 16
Values for k used in this study
Different values were used for the three study phases
Cross-university
Name Phase 1 Phase 2 Phase 3
LongMOOC1 7 – –
LongMOOC2 7 – –
TalkMOOC3 – 7 3
ShortMOOC4 – 7 4
ShortMOOC5 – 7 5
Phase 1: Best-fit value for k aligned with OU study
Phase 2: Testing k=7 where this was not the best fit
Phase 3: Most suitable value for k in each set of data

17. 17
Phase two: k ≠ 7
Why k≠7 in Talk MOOC3
Phase two
The absence of assessment in TalkMOOC3 limited its coding profile
1 only visited content (for example, video, audio, text)
2 commented but visited no new content
3 visited content and commented
4 did the assessment late and did nothing else that week
5 visited content and did the assessment late
6 did the assessment late, commented, but visited no new content
7 visited content, commented, late assessment
8 assessment early or on time, but nothing else that week
9 visited content and completed assessment early / on time
10 assessment early or on time, commented, but visited no new content
11 visited, posted, completed assessment early / on time

18. 18
Phase two: k ≠ 7
Why k≠7 in ShortMOOC4 and ShortMOOC5
Phase two
Three clusters are indistinguishable in a three-week MOOC
Returners who come back in Week 2
Mid-way Dropouts who drop out mid-course
Nearly There who drop out near the end
With only three opportunities for late submission, there are no
Late Completers (who typically submit assessment late five times)
The three-week course design means other clusters emerge, such as:
Surgers concentrate their effort after the first week of a three-week course
Improvers fall behind in Week 1, begin to catch up in Week 2 and complete on time

19. 19
Phase three: suitable values for k
TalkMOOC3: k=3
Phase three
Quiet (1, 0, 0, 0, 0, 0)
● The largest cluster
● Visit a quarter of course
steps
● Do not comment in first
week
● Only 7% comment at all
● Only 9% engage with
second half of course
Contributors (3, 1, 1, 0, 0, 0)
● 19% of cohort
● Visit 38% of course
steps
● Every cluster member
posts in first week of
course
● Half do not comment
again
Consistent engagers
(3, 3, 3, 3, 1, 1)
● 11% of cohort
● Visit 82% of course steps
● Engage throughout course
● Every cluster member posts
a comment
● 95% contribute more than
three comments
● 7% contribute more than
100 comments

20. 20
Phase three: suitable values for k
ShortMOOC4: k=4
Phase three
Very weak starters (2, 1, 0)
● The largest cluster
● Visit 20% of steps
● 20% do not engage in
first week
Strong starters (truncated)
(10, 1, 0)
● 17% of cohort
● Submit week 1 assessment
● Do not submit another
assessment
● Almost half post comment
Returners (truncated)
(3, 3, 3, 3, 1, 1)
● Most submit week 1
assessment
● All submit week 2
assessment
● Half submit
at least one
comment
Keen completers
(truncated) (9, 9, 9)
● Visit more than 90% of
steps
● Submit work on time
● Engage throughout

21. 21
Phase three: suitable values for k
ShortMOOC5: k=5
Phase three
Samplers (truncated)
(1, 0, 0)
● Visit few steps
● Includes many latecomers
(>25%)
● Very few submit
assessment
Strong starters (truncated)
(9, 1, 0)
● Submit week 1 assessment
● Do not submit another
assessment
Returners (truncated)
(8, 8, 2)
● Most submit week 1
assessment
● All submit week 2
assessment
Keen completers
(truncated) (9, 9, 9)
● Visit more than 90% of
steps
● Submit work on time
● Engage throughout
Improvers (5, 6, 9)
● Activity increases each
week
● Final assessment submitte
on time

22. 22
Learning design and pedagogy
● The Coursera Study suggested that MOOC designers would be
able to apply the four engagement patterns they had identified
‘to target interventions and develop adaptive course features’
● These subsequent studies show that this is not necessarily the
case –engagement patterns are not consistent across MOOCs
● Changes to the basic pedagogic elements of a course are
associated with shifts in patterns of engagement.
● Shifts in pedagogic approach can change the elements of a
course that can be regarded as key
● Changes to some elements of learning design can change
learners’ patterns of engagement with a MOOC

23. 23
Shorter courses
● Reducing course length does not necessarily increase
engagement
● Many learners do not approach a three-week course in the
same way as an eight-week course
● Many focus their attention on later weeks and may miss out the
content and activities in the first week
● A three-week course offers limited opportunities to get ahead of,
or behind, the cohort
● It is possible to dip in at different points without losing the sense
of being a cohort member

24. 24
Improving learning and learning environments
Closing the loop
● Previews of course material would allow Samplers to make a
more informed decision about whether to join the course
● Sign-up pages could draw attention to the problems
experienced by those who are out of step with the cohort
● Discussion steps for latecomers could support those who fall
behind at the start
● Prompts might encourage flagging learners to return and
register for a subsequent presentation
● Bridges between course weeks could indicate links and point
learners forward

25. 25
View these slides at www.slideshare.net/R3beccaF
Rebecca Ferguson @R3beccaF
Doug Clow @dougclow