These slides present the iterative design and evaluation the student activity meter. Presented for students of the KULeuven CHI course of prof. Erik Duval.

1. THE STUDENT ACTIVITY
METER FOR AWARENESS AND
SELF-REFLECTION
Sten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo.
SigCHI 2012, Long Case Study.
1
Tuesday 20 March 12

2. http://bit.ly/stenpub
2
Tuesday 20 March 12

3. OVERVIEW
• Problem statement
• Objectives
• Design & Implementation
• Evaluations
• Conclusion
3
Tuesday 20 March 12

4. PROBLEM
http://www.ypfp.org/classroom
4
Tuesday 20 March 12

5. PROBLEM
http://www.ypfp.org/classroom
4
Tuesday 20 March 12

6. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning
resource use and
recommendations
http://free-extras.com/images/obama_hope_progress-2848.htm
• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
5
Tuesday 20 March 12

7. STUDENT ACTIVITY METER
(SAM): DEMO.
http://dilbert.com/strips/comic/2000-02-24/
6
Tuesday 20 March 12

8. STUDENT ACTIVITY METER
7
Tuesday 20 March 12

9. STUDENT ACTIVITY METER
8
Tuesday 20 March 12

10. STUDENT ACTIVITY METER
9
Tuesday 20 March 12

11. TASK
• How would you evaluate SAM?
• how/where do you get data from?
• which criteria could you evaluate?
• which methods use to evaluate
those criteria?
• iterative? which iterations? http://callofduty.wikia.com/wiki/Task_Force_141
10
Tuesday 20 March 12

12. METHODOLOGY.
• design-based research [Obrenović 2011]: give
students & teachers a tool and evaluate
• iterative process of design & evaluation
• paper mockups
• usability studies
• surveys on usefulness
http://funnytogo.com/pictures/evolution/timeline.htm
11
Tuesday 20 March 12

13. iter. evaluation goal methodology demo-graphics data
usability, satisfaction, task-based interview
12 CS students timetracking
I. preliminary with think-aloud, SUS
at KULeuven via Twitter
usefulness & MSDT
assessing teacher online survey with 19 teachers &
II. needs, use & Likert items, yes/no TA’s with Moodle logs
usefulness and open questions CGIAR courses
assessing teacher
online survey with 12 participants
needs, expert
III. Likert items, yes/no from LAK Moodle logs
feedback, use &
and open questions MOOC
usefulness
structured face-to-face 11 teachers & VM with
use, usefulness &
IV. interview with tasks & TA’s at Carlos III extensive
satisfaction
open questions Uni. Madrid tool tracking
12
Tuesday 20 March 12

14. ITERATION ONE
• usability and user satisfaction evaluation
• 12 CS students, using a -
based time tracker
•2 evaluation sessions:
• task based interview with think aloud
(after 1 week of tracking)
• user satisfaction (SUS & MSDT)
(after 1 month)
Tuesday 20 March 12

15. LEARNABILITY, ERRORS &
EFFICIENCY
• in general, people understand the visualizations well!
• some issues were uncovered...
Tuesday 20 March 12

16. LEARNABILITY, ERRORS &
EFFICIENCY
• in general, people understand the visualizations well!
• some issues were uncovered...
Tuesday 20 March 12

17. LEARNABILITY, ERRORS &
EFFICIENCY
• in general, people understand the visualizations well!
• some issues were uncovered...
Tuesday 20 March 12

18. LEARNABILITY, ERRORS &
EFFICIENCY
• in general, people understand the visualizations well!
• some issues were uncovered...
Tuesday 20 March 12

19. LEARNABILITY, ERRORS &
EFFICIENCY
• in general, people understand the visualizations well!
• some issues were uncovered...
Tuesday 20 March 12

20. USER SATISFACTION
• average SUS score: 73%
[Bangor et al. 2008]
Tuesday 20 March 12

21. USER SATISFACTION
• average SUS score: 73%
[Bangor et al. 2008]
Tuesday 20 March 12

22. USER SATISFACTION
• average SUS score: 73%
[Bangor et al. 2008]
Tuesday 20 March 12

23. USER SATISFACTION
• average SUS score: 73%
[Bangor et al. 2008]
Tuesday 20 March 12

24. demogra evaluation design
negative positive
phics goal changes
•↑learnability
usability,
•↓errors
12 CS satisfaction, small usability
I. 1st iteration •good satisfaction
students preliminary issues
•usefulness
usefulness
positive
Tuesday 20 March 12

25. ITERATION TWO
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.
• An online survey about usefulness, teacher issues and how
the tool can resolve these.
• on average: 40 mins are spent using SAM.
17
Tuesday 20 March 12

26. CGIAR CASE STUDY
1 2 3 4 5 1 issue for
2 3 4 5
teacher addressed
Provide feedback to the
Provide students to the students
feedback ✔ ?
✔ ✔
Being aware of what
Being aware of are doing
students what students are doing
✗
Knowing about collaboration
Knowing about collaboration and communication
and communication * ✔
Knowing which documents
Knowing which and how muchare used and how much
are used documents ✔ ✔
?
Knowing how and when
Knowing howtools have been used tools have been used
online and when online ✔
Finding the students who
are not doing well
Finding the students who are not doing well ✔ ?
Finding the best students
Finding the besthow much time
Knowing
students ? ?
students spent
Knowing how much time students
Knowing if external learning spent ? ✔
?
resources are used
Knowing if external learning resources Issues
are used ✔
Addressed by SAM
18
Tuesday 20 March 12

27. CGIAR CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ?
Being aware of what students are doing ✔ ✔
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ✔
Knowing how and when online tools have been used ? ✔
Finding the students who are not doing well ✔ ?
Finding the best students ? ?
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ✔ ? 18
Tuesday 20 March 12

28. CGIAR CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ?
Being aware of what students are doing ✔ ✔
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ✔
Knowing how and when online tools have been used ? ✔
Finding the students who are not doing well ✔ ?
Finding the best students ? ?
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ✔ ? 18
Tuesday 20 March 12

29. CGIAR CASE STUDY
19
Tuesday 20 March 12

30. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
19
Tuesday 20 March 12

31. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
19
Tuesday 20 March 12

32. demogra evaluation design
negative positive
phics goal changes
usability, •↑learnability
12 CS satisfaction, •↓errors
I. 1st iteration small usability issues
students preliminary •good satisfaction
usefulness •usefulness positive
•provides
assessing
19 resource awareness
teacher
II. teachers help function recomm. not •all vis. useful
needs, use &
& TA’s useful •many uses
usefulness
•90% wants it
Tuesday 20 March 12

33. ITERATION THREE
• opencourse on learning and knowledge analytics, http://bit.ly/
dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
21
Tuesday 20 March 12

34. ITERATION THREE
• opencourse on learning and knowledge analytics, http://bit.ly/
dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
21
Tuesday 20 March 12

35. LAK CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ✔
Being aware of what students are doing ✔ ?
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ?
Knowing how and when online tools have been used ✗ ?
Finding the students who are not doing well ✔ ?
Finding the best students ? ✗
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ? ? 22
Tuesday 20 March 12

36. LAK CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ✔
Being aware of what students are doing ✔ ?
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ?
Knowing how and when online tools have been used ✗ ?
Finding the students who are not doing well ✔ ?
Finding the best students ? ✗
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ? ? 22
Tuesday 20 March 12

37. LAK CASE STUDY
issue for
teacher addressed
Provide feedback to the students ✔ ✔
Being aware of what students are doing ✔ ?
Knowing about collaboration and communication ✔ ✗
Knowing which documents are used and how much ✔ ?
Knowing how and when online tools have been used ✗ ?
Finding the students who are not doing well ✔ ?
Finding the best students ? ✗
Knowing how much time students spent ? ✔
Knowing if external learning resources are used ? ? 22
Tuesday 20 March 12

38. LAK CASE STUDY
23
Tuesday 20 March 12

39. LAK CASE STUDY
24
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Tuesday 20 March 12

40. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
24
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Tuesday 20 March 12

41. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
24
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Tuesday 20 March 12

42. IDEA’S FROM EXPERTS
the used resource types
detailed information per student
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
25
Tuesday 20 March 12

43. IDEA’S FROM EXPERTS
2 the used resource types
5 detailed information per student
4 detailed information of 2 students
3 detailed usage stats of resources
1 stats or vis. on content creation
25
Tuesday 20 March 12

44. IDEA’S FROM EXPERTS
2 the used resource types
5 detailed information per student
4 detailed information of 2 students
3 detailed usage stats of resources
1 stats or vis. on content creation
25
Tuesday 20 March 12

45. demogra evaluation design
negative positive
phics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
•provides awareness
assessing teacher resource
19 teachers •all vis. useful
II. needs, use & help function recomm. not
& TA’s •many uses
usefulness useful
•90% want it
•provides awareness
assessing re-orderable
most and feedback
12 teacher parallel
addressed •many uses
III. participan needs, expert coordinates
needs are •66% want it
ts feedback, use with
indecisive •recomm. can be
& usefulness histograms
useful
Tuesday 20 March 12

46. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitativestudy using a structured face-2-face interview
with 25 open questions.
27
Tuesday 20 March 12

47. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitativestudy using a structured face-2-face interview
with 25 open questions.
27
Tuesday 20 March 12

48. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
http://www.tdpri.com/forum/telecaster-discussion-
forum/48431-ic503z-cap-52-ri-blackguard-book.html
28
Tuesday 20 March 12

49. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
http://www.tdpri.com/forum/telecaster-discussion-
forum/48431-ic503z-cap-52-ri-blackguard-book.html
28
Tuesday 20 March 12

50. USER SATISFACTION
• average SUS score: 69,69%
[Bangor et al. 2008]
29
Tuesday 20 March 12

51. USER SATISFACTION
• average SUS score: 69,69%
all: want to continue using it
9/11: give it to students
[Bangor et al. 2008]
29
Tuesday 20 March 12

52. demo- evaluation design
negative positive
graphics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
•provides awareness
19 assessing teacher resource
•all vis. useful
II. teachers needs, use & help function recomm. not
•many uses
& TA’s usefulness useful
•90% want it
•provides awareness and
assessing teacher most
12 re-orderable feedback
needs, expert addressed
III. participan PC with •many uses
feedback, use & needs are
ts histograms •66% want it
usefulness indecisive
•recomm. can be useful
filter & •provides time overview
search, conflicting •provides course
11 icons, visions of overview
use, usefulness
IV. teacher zooming in students •PC assist with detecting
& satisfaction
s & TA’s line chart, doing well problems
editing PC or at risk •many uses & insights
axes •100% want it
Tuesday 20 March 12

53. CONCLUSION
• SAM enables:
• to find students doing well and at risk
• a better course overview
• understanding student time spending
• almost all participants want to continue using SAM
• SAM enables to find a wide variety of
new insights
31
Tuesday 20 March 12

54. 2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 32
Tuesday 20 March 12