presentation of the student activity meter at SigCHI 2012 in Austin, TX.

1. THE STUDENT ACTIVITY
METER FOR AWARENESS AND
SELF-REFLECTION
Sten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo.
SigCHI 2012, Long Case Study.
1

2. our team:
Human-Computer Interaction
• technology enhanced learning
• music
• research 2.0
2

3. PROBLEM
http://www.ypfp.org/classroom
3

4. PROBLEM
http://www.ypfp.org/classroom
3

5. OBJECTIVES
• self-monitoring for learners
http://free-extras.com/images/obama_hope_progress-2848.htm
4

6. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
http://free-extras.com/images/obama_hope_progress-2848.htm
4

7. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
http://free-extras.com/images/obama_hope_progress-2848.htm
4

8. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning
resource use and
recommendations
http://free-extras.com/images/obama_hope_progress-2848.htm
4

9. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning
resource use and
recommendations
http://free-extras.com/images/obama_hope_progress-2848.htm
• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
4

10. STUDENT ACTIVITY METER
(SAM): DEMO.
http://dilbert.com/strips/comic/2000-02-24/
5

11. http://research.uow.edu.au/learningnetworks/seeing/snapp/index.html
SNAPP

12. GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html

13. The Participation Tool http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdf

14. METHODOLOGY.
• design-basedresearch [Obrenović 2011]: design a tool, give
students & teachers and evaluate
• iterative process of design & evaluation
• paper mockups
• usability studies
• surveys on usefulness
http://funnytogo.com/pictures/evolution/timeline.htm
9

15. iter. evaluation goal methodology demo-graphics data
usability, satisfaction, task-based interview
12 CS students timetracking
I. preliminary with think-aloud, SUS
at KULeuven via Twitter
usefulness & MSDT
assessing teacher online survey with 19 teachers &
II. needs, use & Likert items, yes/no TA’s with Moodle logs
usefulness and open questions CGIAR courses
assessing teacher
online survey with 12 participants
needs, expert
III. Likert items, yes/no from LAK Moodle logs
feedback, use &
and open questions MOOC
usefulness
structured face-to-face 11 teachers & VM with
use, usefulness &
IV. interview with tasks & TA’s at Carlos III extensive
satisfaction
open questions Uni. Madrid tool tracking
10

16. 11

17. 11

18. demogra evaluation design
negative positive
phics goal changes
•↑learnability
usability,
•↓errors
12 CS satisfaction, small usability
I. 1st iteration •good satisfaction
students preliminary issues
•usefulness
usefulness
positive

19. ITERATION TWO &
THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how
the tool can resolve these.
13

20. ITERATION TWO &
THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how
the tool can resolve these.
13

21. CGIAR LAK
1 2 3 4 issue for2 add- issue for add-
5 1 3 4 5
teacher ressed teacher ressed
Provide feedback to the
Provide feedback to the students
students ✔ ? ✔ ✔
Being aware of what
Being aware of whatdoing
students are students are doing ✔ ✔ ✔ ?
✗ ✗
Knowing about collaboration
Knowing and communication * and communication
about collaboration ✔ ✔
?
Knowing which documents
Knowing are used and how much used and how much
which documents are ✔ ✔ ✔
? ✗ ?
Knowing how and when
Knowing online and whenbeen used have been used
how tools have online tools ✔
Finding the students who
are not doing well
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students
Finding the best students
Knowing how much time
? ? ? ✗
? ✔ ?
students spent
Knowing Knowing if externalstudents
how much time learning spent ✔
? ? ?
resources are used
Knowing if external learning resources areIssues
used ✔Addressed by SAM
14

22. CGIAR LAK
issue for add- issue for add-
teacher ressed teacher ressed
Provide feedback to the students ✔ ? ✔ ✔
Being aware of what students are doing ✔ ✔ ✔ ?
Knowing about collaboration and communication ✔ ✗ ✔ ✗
Knowing which documents are used and how much ✔ ✔ ✔ ?
Knowing how and when online tools have been used ? ✔ ✗ ?
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students ? ? ? ✗
Knowing how much time students spent ? ✔ ? ✔
Knowing if external learning resources are used ✔ ? ? ?
14

23. CGIAR LAK
issue for add- issue for add-
teacher ressed teacher ressed
Provide feedback to the students ✔ ? ✔ ✔
Being aware of what students are doing ✔ ✔ ✔ ?
Knowing about collaboration and communication ✔ ✗ ✔ ✗
Knowing which documents are used and how much ✔ ✔ ✔ ?
Knowing how and when online tools have been used ? ✔ ✗ ?
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students ? ? ? ✗
Knowing how much time students spent ? ✔ ? ✔
Knowing if external learning resources are used ✔ ? ? ?
14

24. CGIAR CASE STUDY
15

25. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
15

26. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1
comparison tool use for course more metrics
design optimization wanted
15

27. LAK CASE STUDY
16
http://maranash.blogspot.com/2011/01/conference-shout-out.html

28. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
16
http://maranash.blogspot.com/2011/01/conference-shout-out.html

29. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
16
http://maranash.blogspot.com/2011/01/conference-shout-out.html

30. demogra evaluation design
negative positive
phics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
assessing •provides awareness
19 resource
teacher •all vis. useful
II. teachers help function recomm. not
needs, use & •many uses
& TA’s useful
usefulness •90% want it
•provides awareness
assessing re-orderable
most and feedback
12 teacher parallel
addressed •many uses
III. participan needs, expert coordinates
needs are •66% want it
ts feedback, use with
indecisive •recomm. can be
& usefulness histograms
useful

31. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative
study using a structured face-2-face interview
with 25 open questions.
18

32. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative
study using a structured face-2-face interview
with 25 open questions.
18

33. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
http://www.tdpri.com/forum/telecaster-discussion-
forum/48431-ic503z-cap-52-ri-blackguard-book.html
19

34. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
http://www.tdpri.com/forum/telecaster-discussion-
forum/48431-ic503z-cap-52-ri-blackguard-book.html
19

35. USABILITY & SATISFACTION
• average SUS score: 69,69%
[Bangor et al. 2008]
20

36. USABILITY & SATISFACTION
• average SUS score: 69,69%
all: want to continue using it
9/11: give it to students
[Bangor et al. 2008]
20

37. demo- evaluation design
negative positive
graphics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
•provides awareness
19 assessing teacher resource
•all vis. useful
II. teachers needs, use & help function recomm. not
•many uses
& TA’s usefulness useful
•90% want it
•provides awareness and
assessing teacher most
12 re-orderable feedback
needs, expert addressed
III. participan PC with •many uses
feedback, use & needs are
ts histograms •66% want it
usefulness indecisive
•recomm. can be useful
filter & •provides time overview
search, conflicting •provides course
11 icons, visions of overview
use, usefulness
IV. teacher zooming in students •PC assist with detecting
& satisfaction
s & TA’s line chart, doing well problems
editing PC or at risk •many uses & insights
axes •100% want it

38. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
22

39. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
22

40. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
22

41. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
22

42. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
22

43. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights
22

44. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights
• almost all want to continue using SAM
22

45. LESSONS LEARNED
• worked well:
23
http://slurmed.com/fanart/dexus5/005_professor.png

46. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
23
http://slurmed.com/fanart/dexus5/005_professor.png

47. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
23
http://slurmed.com/fanart/dexus5/005_professor.png

48. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
• Visualisations are powerful tool for creating
awareness
23
http://slurmed.com/fanart/dexus5/005_professor.png

49. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
• Visualisations are powerful tool for creating
awareness
• worked less well:
23
http://slurmed.com/fanart/dexus5/005_professor.png

50. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings
23
http://slurmed.com/fanart/dexus5/005_professor.png

51. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings
• getting tracking data
23
http://slurmed.com/fanart/dexus5/005_professor.png

52. LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
• Use real-world settings
• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings
• getting tracking data
• longitudinal study would be useful
23
http://slurmed.com/fanart/dexus5/005_professor.png

53. http://www.fitbit.com/
http://media.cnbc.com/i/CNBC/Sections/
News_And_Analysis/_News/_SPECIAL_REPORTS/
BetterYourBusiness_2012/BYB-Digital%20- http://quantifiedself.com/
%20April/Slideshows/Digital_Appliances_ab/10-
Smart-Dig-Appliances-Wifi-scale-.jpg
http://nikerunning.nike.com/nikeos/p/nikeplus/en_EMEA/sportband
24

54. LAK13: Learning Analytics and Knowledge Conf.
8-12 April 2013 Leuven, Belgium
http://lakconference.org/

55. 2
contact: sten@cs.kuleuven.be
slides will appear on http://www.slideshare.net/stengovaerts
@erikduval
slides will appear on http://www.slideshare.net/stengovaerts 26

56. If you are interested in visualisations and
2
tabletops, we have a PhD position available.
contact: sten@cs.kuleuven.be
slides will appear on http://www.slideshare.net/stengovaerts
@erikduval
slides will appear on http://www.slideshare.net/stengovaerts 26