Nikolaos Floratos, Teresa Guasch and Anna Espasa Open University of Catalonia GO-GN Network

1. Recommendations on Formative
Assessment and Feedback Practices for
stronger engagement in MOOCs
By
Nikolaos Floratos, nfloratos@uoc.edu, Teresa Guasch,
tguaschp@uoc.edu and Anna Espasa, aespasa@uoc.edu
Open University of Catalonia
GO-GN Network

2. Around 90% the drop rate in MOOCs
(Clow, 2013; Lewin, 2013)

3. Some interesting facts about MOOCs
• 8.2 Million participants in Coursera
• 700 plus courses by more than 100 universities
in Coursera
• Strong MOOC activity around the globe
• The numbers are increasing continually
!!!!but also drop rates are increasing

4. Without this
• MOOCs do not create impact
• Open Education may fail
With this
• MOOCs will make a real difference in
education
• Open Education will transcend

5. Student Engagement (Adamopoulos, 2013)
depends on non-didactic and didactic factors
Student
profiles
Instructors’
profiles
Institution
Reputations
demographics
Course
structure
Training
content
Self-paced
or not
workload
Course
duration
Course topic
Type of
examsAssessment
& Feedback
types
Interaction with
students &
instructors

6. Two contradictory points
The type of assessment and feedback methods
as well as interaction with students and
instructors is one of the main elements for
student engagement
However,
very limited research so far on the proper
assessment as well as feedback practices to be
applied specifically in MOOCs for enhancing
student engagement

7. Therefore the scope of this study
1. to formulate a framework of
requirements/conditions for feedback and
formative assessment practices that support
student engagement in MOOCs (on literature
review)
2. to suggest related recommendations on formative
assessment and feedback practices

8. Literature review Findings
• no research evidence yet on specific feedback
and formative assessment practices directly
applicable in MOOCs for advancing Student
Engagement
• Such research now applicable only in traditional
learning environments (mainly in f2f learning)
– Nicol and Macfarlane-Dick (2006): 7 principles on
feedback and formative assessment practices for
student engagement
– Gibbs and Simpson, 2004: Assessment tasks should
satisfy 3 specific conditions and feedback 6 conditions
respectively

9. Student Engagement in MOOCS?
(Hew 2015) defines student engagement in
MOOCs as the level of a student’s engagement
in a learning activity

10. How engagement is driven
Hew 2015, Deci and Ryan, 1991; Deci and Ryan,
2000; Hardre and Reeve, 2003;(Self Determination
Theory SDT)
• Need of Autonomy (the need for students to
sense they are non dependent on other
peoples' actions
• Need of relatedness (the need for students to
connect with other people)
• Need of competence (the need for students to
master specific knowledge)

11. Based on literature review (Nicol and
Macfarlane-Dick 2006, Gibbs and Simpson 2004,
Hew 2015, Deci and Ryan, 1991; Deci and Ryan,
2000; Hardre and Reeve, 2003;
17 Requirements/conditions for
formative assessment and feedback
practices to support student
engagement in MOOC

12. 17 Requirements/conditions
1. Requirement 1: Autonomy e.g. a MOOC student
should be able to assess their own work, assignments
and progress
2. Requirement 2: Relatedness e.g.a student should be
able to assess the work and assignment of other
peers
3. Requirement 3: Competence e.g. Assessment to
generate feedback on performance and accelerate
learning
4. Requirement 4: Students should know in advance
what is a good performance and based on that to
improve further
5. Requirement 5: Student should know in advance the
steps needed for reaching good performance

13. 17 Requirements/conditions cont.
6. Requirement 6: Formative assessment and feedback
practices should help students clarify what good
performance is and how much different this is with
respect to their current performance
7. Requirement 7: Formative assessment and feedback
practices should encourage dialogue around learning
between peers and teachers
8. Requirement 8: Formative assessment and feedback
practices should advance self-esteem
9. Requirement 9: Formative assessment and feedback
practices should create positive motivational beliefs
and self-esteem

14. 17 Requirements/conditions cont.
10. Requirement 10: Formative assessment and feedback
practices should provide information to teachers for
improving their teaching approach
11. Requirement 11: Assessment tasks should be
sufficient enough in order to require students to
dedicate appropriate study time for addressing them
12. Requirement 12: Assessment tasks should orientate
students to allocate appropriate amounts of time and
effort to the most important aspects of the course
13. Requirement 13: Assessment tasks should engage
students with related productive learning activities

15. 17 Requirements/conditions cont.
14. Requirement 14: Feedback should be provided
in sufficient detail and frequent enough
15. Requirement 15: Feedback should focus on
students’ performance on their learning
16.Requirement 16: Feedback should be delivered
in time to students so that it still matters and it
is well received Requirement
17. Requirement 17: Feedback should be aligned
with the purpose of the related assignment and
advance future learning and use by the student

16. Approach on formulating related
recommendations
• As basis reviews from the special review platform for MOOCs, coursetalk
www.coursetalk.com
• Reviews considered only from specific MOOCs with all the following criteria
– Offered for free
– Offered by Universities
– Top-rated
– Received more than 100 reviews
• 7 MOOCs have been identified with total 4040 reviews
– An introduction to Interactive Programming in Python – offered by Coursera
(1863 reviews)
– A beginner's Guide to Irrational Behavior offered by Coursera (375 Reviews)
– Epidemics - the Dynamics of Infectious Diseases offered by Coursera (315
Reviews)
– Design: Creation of Artifacts in Society offered by Coursera (204 Reviews)
– Modern and Contemporary American Poetry offered by Coursera (170 Reviews)
– An introduction to Operations Management offered by Coursera (104 Reviews)
– Think101x: The Science of Everyday Thinking offered by edX (1037 Reviews)
• 5 steps approach

17. Our 5 Steps Approach
1. Step 1: Filtering and Selection of reviews only related to feedback and
formative assessment (418 reviews)
2. Step 2: Additional Filtering for selecting just those reviews from above that
contribute with some input to recommendations to be formulated (237
reviews)
3. Step 3: Examine the 237 reviews and formulate 14 recommendations that
respect our identified requirements
4. Step 4: Examining again the 237 reviews and checking
– how many times each review contributed as input to a formulated
recommendation
– Whether there were any new recommendations that may have been missed
N.B. Validation: Two additional judges have applied steps 3 and 4 in a sample of
reviews in order to ascertain agreement in findings
5. Step 5: Significance Ranking - Based on the number of reviews contributed
as input to each of the formulated recommendations, the significance of
each recommendation is derived based on the weighted average method

18. No limitation on number of attempts in quizzes

19. Feedback to be provided in time before the next assignment

20. Peer assessment always by the same students (no random allocation)

21. Students to be able to modify their own related assignment work even after its
submission based on feedback received).

22. Peer assessment can be supplemented with forum discussions for interaction and
offer a communication channel for clarifications.

23. Interaction by human experts say professors’ assistants that intervene and provide
comments and views in discussion forums or even in students’ work

24. Immediate feedback mainly from quizzes is appreciated by students since it
confirms their understanding about what they have learnt. The appreciation is
even higher in cases that some further feedback on the given answer or on
the correct answer is provided. (18 reviews supported this
recommendation/finding with 8 as weighted average)

25. Peer assessment in MOOCs does not always deliver accurate results and in
many cases it creates frustrations or negative feelings since it doesn’t
provide any feedback on the work assessed or there are concerns on the
competencies of the peer-assessor. (26 reviews supported this
recommendation/finding with 9 as weighted average)

26. Peer assessment was more welcome in those MOOCs that there was no
mark given but also or alternatively feedback on the quality of the related
assignment. (23 reviews supported this recommendation/finding with 10
as weighted average)

27. Assessment models, such as self assessments (students judge their own
work) and peer-assessments (students judge the work of their peers) can
be relatively complex in MOOCs and a comprehensive rubric should be
provided in all MOOCS that involve peer assessments (29 reviews
supported this recommendation/finding with 13 as weighted average)

28. Discussion forums are appreciated and are a very good way of students to
receive support, sympathy, formative feedback and clarification as well as
share ideas on their work as long as there is an effective mechanism on
managing and accessing the discussion threads. (36 reviews supported this
recommendation/finding with 18 as weighted average)

29. A well structured course syllabus with information on what assignments
are required per week/module by each participant, by when as well as its
related training content was appreciated (35 reviews supported this
recommendation/finding with 21 as weighted average)

30. Although MOOCs can automatically assess quizzes, self and peer
assessments must deal with more comprehensive assignments. However,
self- and peer assessment should be formative as well, i.e. specifically
intended to generate feedback on performance and improve and
accelerate learning rather than just providing a mark. (48 reviews
supported this recommendation with 30 as weighted average)

31. Assignments can be based on practical problems with clear expected
output that give the sense of completeness to the students and are not
that easy to address but challenging and interesting enough (65 reviews
supported this recommendation/finding with 42 as weighted average)

32. Let’s all move from MOOC to
MOOL!

33. Thank you for your attention
Nikolaos Floratos
PhD Researcher
OUC University
GO-GN Member
www.NikolaosFloratos.com

34. Bibliography
Deci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits: Human needs and the self-determination of
behavior. Psychological inquiry, 11(4), 227-268. http://goo.gl/Jf7wzV
Deci E.L. & Ryan R. M., (1991) “A motivational approach to self: Integration in personality,” in Nebraska symposium on
motivation: Perspectives on motivation, R. Dienstbier, Ed., Lincoln, NE: University of Nebraska Press, vol. 38, pp. 237-
288
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the
evidence. Review of educational research, 74(1), 59-109 http://goo.gl/JWf61L
Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’ learning. Learning and teaching
in higher education, 1(1), 3-31
Guasch, T., Espasa, A., Alvarez, I.M. & Kirschner, P.A. (2013). Effects of Teacher and Peer Feedback on Collaborative
Writing in an Online Learning Environment. Distance education, 34 (3), 324-338, DOI: 10.1080/01587919.2013.835772
Helme, S., & Clarke, D. J. (1998). We really put our minds to it: Cognitive engagement in the mathematics classroom.
Teaching mathematics in new times, Brisbane, Qld: Mathematics Education Research Group of Australasia, pp. 250-257.
Hew, K.F. (2015). Towards a Model of Engaging Online Students: Lessons from MOOCs and Four Policy Documents.
International Journal of Information and Education Technology, 5 (6), 425-431
Nicol, N. & Macfarlane, D. (2006) Formative Assessment and self regulated learning: a model and seven principles of
good feedback practice. Studies in Higher Education, 31 (2), 199-218 http://goo.gl/aW1Ybs
Reeve, J. (2012). A self-determination theory perspective on student engagement. In Handbook of research on student
engagement (pp. 149-172). Springer US, http://goo.gl/OyCOKm
Skinner, E. A., Kindermann, T. A., Connell, J. P., & Wellborn, J. G. (2009). Engagement and disaffection as organizational
constructs in the dynamics of motivational development. Handbook of motivation at school, 223-245.
http://goo.gl/bnTOI0
License and Citation
This work is licensed under the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0/.
Please cite this work as: Floratos, N., Guasch, T., Espasa, A., (2015). Recommendations on Formative Assessment and
Feedback Practices for stronger engagement in MOOCs. In Proceedings of Open Education Global 2015: Innovation and
Entrepreneurship.