Presentation at the 2015 Learning Analytics Conference at Poughkeepsie, NY, USA on March 19, 2015

1. Penetrating the Black Box of Time-on-task Estimation
http://bit.do/lak tot
Vitomir Kovanovi´c
School of Informatics,
University of Edinburgh,
Edinburgh, United Kingdom
v.kovanovic@ed.ac.uk
Dragan Gaˇsevi´c
Schools of Education and
Informatics,
University of Edinburgh,
Edinburgh, United Kingdom
dgasevic@acm.org
Shane Dawson
Learning and Teaching Unit,
University of South Australia,
Adelaide, Australia
shane.dawson@unisa.edu.au
Sre´cko Joksimovi´c
School of Interactive Arts
and Technology,
Simon Fraser University,
Burnaby, Canada
sjoksimo@sfu.ca
Ryan S. Baker
Teachers College,
Columbia University,
New York, USA
baker2@exchange.tc.columbia.edu
Marek Hatala
School of Interactive Arts
and Technology,
Simon Fraser University,
Burnaby, Canada
mhatala@sfu.ca
March 19, 2015
Marist College,
Poughkeepsie, NY, USA

2. Introduction
Time-on-task
“All learning, whether done in school or
elsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28

3. Introduction
Time-on-task
“All learning, whether done in school or
elsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28

4. Introduction
Time-on-task
However, there is a big difference between elapsed time,
and time student actually spent on learning (Carroll,
1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28

5. Introduction
Time-on-task
However, there is a big difference between elapsed time,
and time student actually spent on learning (Carroll,
1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28

6. Introduction
Origins of time-on-task in educational research
• Seminal work by J. Carroll: “A model of school learning” in 1963 put a
direct link between time and learning outcomes.
• Carroll (1963) differentiaded between elapsed time and time spent on
learning.
• How time is spent is what ultimately matters!
• Increase of time-on-task was one of the key principles of effective
education (Chickering and Gamson, 1989).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 3 / 28

7. Introduction
Challenges with time-on-task measures
• Many different operationalizations and methods of measurement
• Days in school,
• Number of lectures attended,
• Observing students’ behavior every X minutes.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 4 / 28

8. Introduction
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 5 / 28

9. Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28

10. Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28

11. Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28

12. Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28

13. Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation
• Typycally simple heuristics are used (limit action duration to X minutes)
• Problems for replications
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28

14. Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation
• Typycally simple heuristics are used (limit action duration to X minutes)
• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28

15. Introduction
Idea for the study
• Research study that involved clustering using time-on-task measures,
• Adopted one of the heuristics,
• Still, it didn’t feel totally right.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 8 / 28

16. Introduction Research Questions: Effects of time-on-task measuring on analytics results
Research Questions
Drawing on the Karweit and Slavin (1982) research, we conducted a study to
answer the following questions:
• What effects do different methods for estimation of time on-task-measures
from LMS data have on the results of analytical models?
• Are there differences in their statistical significance and overall conclusions
that can be drawn from them?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 9 / 28

17. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

18. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

19. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

20. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

21. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

22. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0s
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

23. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

24. Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28

25. Methods Dataset
Dataset: Trace data
• 6 offers of 13-week research-intensive fully online masters course at Canadian
public university.
• Moodle used as LMS platform: total of 81 students and 167,261 log records.
Students Actions Messages
Winter 2008 15 33,976 212
Fall 2008 22 49,928 633
Summer 2009 10 21,059 243
Fall 2009 7 11,346 63
Winter 2010 14 31,169 359
Winter 2011 13 19,783 237
Average (SD) 13.5 (5.1) 27,877 (13,561) 291.2 (192.4)
Total 81 167,261 1747
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 11 / 28

26. Methods Dataset
Time-on-task measures / count measures
• Only used activities that were planned by the course design
• Viewing assignments
• Viewing resources
• Viewing discussions
• Posting to discussions
• Updating discussion messages
• Extracted both count and time-on-task measures
• Count measures used as baseline
• Time-on-task measures extracted in 15 different ways
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 12 / 28

27. Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28

28. Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28

29. Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28

30. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

31. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

32. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

33. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

34. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%
1 Triggering Event 308 17.63%
2 Exploration 684 39.17%
3 Integration 508 29.08%
4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

35. Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%
1 Triggering Event 308 17.63%
2 Exploration 684 39.17%
3 Integration 508 29.08%
4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28

36. Methods Dataset
Outcome measures
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
5 CoIHigh: number of integration and resolution messages
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 15 / 28

37. Methods Dataset
Measures summary
# Name Description
Count measures
1 AsignmentViewCount Number of assignment views.
2 ResourceViewCount Number of resources views.
3 DiscussionViewCount Number of course discussion views.
4 AddPostCount Number of posted messages.
5 UpdatePostCount Number of post updates.
Time-on-task measures
6 AsignmentViewTime Time spent on course assignments.
7 ResourceViewTime Time spent reading course resources.
8 DiscussionViewTime Time spent viewing course discussions.
9 AddPostTime Time spent posting discussion messages.
10 UpdatePostTime Time spent updating discussion messages.
Performance measures
11 TMA2Grade Grade for literature review paper.
12 TMA3Grade Grade for journal papers readings.
13 ParticipationGrade Grade for participation in course discussions.
14 FinalGrade Final grade in the course.
15 CoIHigh Integration and resolution message count.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 16 / 28

38. Methods Experimental procedure
Time-on-task estimation strategies
# Name Description
Group 1: No outliers processing, different processing of last actions
1 x:x No outliers and last action processing.
2 x:ev No outliers processing, estimation of last action duration.
3 x:rm No outliers processing, removal of last action.
4 x:l60 No outliers processing, 60 min last action duration limit.
5 x:l30 No outliers processing, 30 min last action duration limit.
6 x:l10 No outliers processing, 10 min last action duration limit.
Group 2: Thresholding outliers and last actions
7 l60 60 min duration limit.
8 l30 30 min duration limit.
9 l10 10 min duration limit.
Group 3: Thresholding outliers and estimating last actions
10 l60:ev 60 min duration limit, last actions estimated.
11 l30:ev 30 min duration limit, last actions estimated.
12 l10:ev 30 min duration limit, last actions estimated.
Group 4: Estimating outliers and last actions
13 +60ev Estimate last actions and actions longer than 60 min.
14 +30ev Estimate last actions and actions longer than 30 min.
15 +10ev Estimate last actions and actions longer than 10 min.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 17 / 28

39. Methods Experimental procedure
Statistical Analysis
• For each of the five outome measures, we constructed 16 multiple regression
models:
• 1 model using count measures
• 15 using diferently extracted time-on-task measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 18 / 28

40. Results R2 variations of the overall models
Results
R2
Performance Measure Min Max Range Mean SD
TMA2Grade 0.08 0.26 0.18 0.14 0.04
TMA3Grade 0.04 0.17 0.12 0.09 0.04
ParticipationGrade 0.23 0.37 0.13 0.3 0.04
FinalGrade 0.06 0.28 0.23 0.16 0.05
CoIHigh 0.21 0.28 0.07 0.26 0.02
Variation in R2
values across five outcome measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 19 / 28

41. Results R2 variations of the overall models
Time-on-task extraction configuration
R
2
0.150.250.35
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Higher levels of cognitve presence (Integration + Resolution)
0.050.150.25
Final percentage grade
0.200.300.40
Course participation grade
0.000.100.20
TMA3 grade: journal readings
Group 1:
No outlier processing
Group 2:
Duration limit
Group 3:
Duration limit + estimation
Group 4:
Estimation above limit
0.050.150.25
TMA2 grade: literature review
Counts
Time-on-task
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 20 / 28

42. Results Significance of individual predictors
Group 1 Group 2 Group 3 Group 4
DV IV x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
TMA2Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
TMA3Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
Part.Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
FinalGrade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
CoIHigh Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Significant model at p < .05
Significant β > 0
Significant β < 0
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 21 / 28

43. Discussion
Discussion
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 22 / 28

44. Discussion
History repeats itself
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 23 / 28

45. Discussion
Slipery road ahead
• We need more caution when using time-on-task measures for building
learning analytics models.
• We need to provide details of how time-on-task has been estimated.
• Supplemetary materials are great for this!
• Source code repositories with time-on-task estimation code.
• Develop plugins for time-on-task extraction from popular platforms.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 24 / 28

46. Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28

47. Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28

48. Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28

49. Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,
• too much emphasis on p-values, what goes into the model counts!
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28

50. Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,
• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,
• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28

51. Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28

52. Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
Especially for big senstationalistic claims that conflict
existing literature!
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28

53. Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
“Extraordinary claims require extraordinary
proof”(Truzzi, 1978, p. 11)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28

54. Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
• Validate results by adopting several different methods
• Results in loss of test power,
• Too much focus on small effects dependent on particular method being
adopted.
• Conduct replication studies.
• Avoid p-hacking and HARKing (Hypothesizing After the Results are Known)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28

55. Discussion Limitations
Limitations
• Ca not provide a definitive recommendation for practice,
• One statistical model,
• Despite 160,000 log records, it is still one dataset, and
• There are many more time-on-task estimation strategies.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 27 / 28

56. Discussion Future work
Future work
• Use DALMOOC data from Prosolo learning platform:
• First six weeks 15 min innactivity logout.
• Second six weeks 60 min innactivity logout.
• How many of students returned (false positive), and how many did not (true
positive)?
• Looking upon ITS research, provide a gold-standard data.
• LMS plugin that through javascript keeps a track of user activity.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 28 / 28

57. Thank you
Vitomir Kovanovic
vitomir.kovanovic.info
v.kovanovic@ed.ac.uk

58. References I
Bloom, Benjamin S. (1974). “Time and learning”. In: American Psychologist 29.9, pp. 682–688.
Carroll, Jb (1963). “A Model of School Learning”. English. In: Teachers College Record 64.8.
WOS:A1963CAJ4400010, pp. 723–733.
Chickering, Arthur W and Zelda F Gamson (1989). “Seven principles for good practice in
undergraduate education”. In: Biochemical Education 17.3, pp. 140–141.
Karweit, Nancy and Robert E. Slavin (1982). “Time-on-task: Issues of timing, sampling, and
definition”. In: Journal of Educational Psychology 74.6, pp. 844–851.
Truzzi, Marcello (1978). “On the Extraordinary: An Attempt at Clarification”. In: Zetetic Scholar 1.1.