A presentation given at the deferred UCT 2015 Teaching and Learning Conference held at UCT on 30 March 2016. This presentation briefly outlines some of the evaluation work conducted on ROER4D’s research capacity building webinars, focusing on the first 12-webinar series conducted in 2014.

1. Sarah Goodier
ROER4D Evaluation Advisor
UCT Teaching & Learning Conference
30 March 2016
User focused evaluation:
Feedback on research capacity building webinars
3/30/20161

2. Outline
 The project being evaluated: ROER4D
 The evaluation work: what & how
 The key evaluation question
 Findings
 Recommendations & next steps
 Lessons learnt

3. THE ROER4D PROJECT

4. in the Global South
 In what ways, and under what
circumstances can the adoption of
OER address the increasing
demand for accessible, relevant,
high-quality and affordable
education and what is its impact in
the Global South?
Research on Open Educational
Resources for Development
http://roer4d.org/

5. ROER4D has 86 researchers across 18 sub-projects in 26 countries
across 16 time zones

6. EVALUATING ROER4D

7. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Key Evaluation Areas
EVALUATION
Using the utilization
focused evaluation (UFE)
framework

8. What is utilization focused evaluation (UFE)
UFE is centered around intended use by intended
users
- What do the users (key stakeholders) want to know?
- How will answers to the evaluation questions
potentially help to improve the project?

9. UFE in 12 steps:
Steps are iterative,
not linear
Utilization Focused Evaluation Framework
(Adapted from Ramirez & Brodhead, 2013)

10. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Key Evaluation Areas
Webinars

11. Webinar evaluation timeline
2014 series (Feb – Oct) 2015 (Feb – July)
Evaluation
Data presented today
This 12-session series focused on the
harmonisation of survey questions that
were to be used across the different
ROER4D sub-projects
Iterative
evaluation
work during
this series
of webinars

12. Key Evaluation Question
 To what extent have ROER4D Network
Hub webinars been a mechanism to
potentially grow the research capacity of
researchers working on the ROER4D
project?
[Has the process worked & how can we improve it?]

13.  To answer the question, focused on:
◦ Recorded attendance (taken at each
webinars)
◦ Survey results of the sub-project researchers
regarding the webinars (sent after the
completion of the webinar series)

14. FINDINGS

15. 2014 Session Attendance
• Overall, the trend was a decrease in
attendance over the course of the sessions

16. 2014 Session Attendance
• For all but one of
the last five
sessions, the
network hub
team made up
half or more of
the attendees

17. 2014 Session Attendance
• Approximately half of the ROER4D network
didn’t attend any of the sessions;
• Of those who did attend at least one session,
most attended only up to six sessions

18. Research Capacity and Question
Harmonization Survey Findings
Key findings:
• Those who did attend/view the session found them useful
• Scheduling issues were a factor for those who didn’t attend any
of the live sessions.
• The fact that these sessions were voluntary was flagged as a less
valuable point about the question harmonization process. This
was seen to potentially create gaps in terms of research capacity
across the project.
• Having recordings and the research questions and concept
definitions available was useful.

19.  In summary, this first webinar process has
experienced varying degrees of success:
◦ Useful for those who attended/viewed the
sessions
◦ Several barriers (e.g. timezones) flagged by
the researchers & engagement in the
webinars was sporadic across the sub-projects

20. RECOMMENDATIONS &
NEXT STEPS

21. Evaluation recommendations on
webinars
– Encourage each subproject to have at least 1
researching member attend the live webinar
– Keep recording webinars and point the
subprojects to the recordings after the
session as a reminder
– Have a forum for questions/discussion
(before and) after the webinar so that those
who don’t attend live can also benefit from
shared discussion
– Have session repeated (2x) to incorporate
different time zones (if there is capacity)

22. *Update: several of the recommendations were picked up in
the 2015 series (e.g. having repeat webinars scheduled at
different times)
Constant dialogue with the ROER4D team and users as the
evaluation work continues
Next steps

23. Some lessons from evaluating
ROER4D
Find a framework for your evaluation (e.g. UFE):
valuable guiding structure
For both internal evaluation and external
evaluations, never underestimate the importance of
iterative engagement :
more engagement = better understanding
of project & what matters to users

24. This work is licensed under a Creative Commons Attribution 4.0 International License.
Website: http://roer4d.org
Contact author: Sarah Goodier
sarah.goodier@uct.ac.za
@SarahGoodier
Excluding images,
screenshots and
logos and/or
unless otherwise
indicated on
content
Thank you!

25. Further reading:
About ROER4D:
• Hodgkinson-Williams, C. (2013). Research on Open
Educational Resources for Development in Post-secondary
Education in the Global South (ROER4D) - Scoping Document.
Available online: http://hdl.handle.net/11427/8430 [Last
accessed 10 February 2015].
• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on
Open Educational Resources for Development in the Global
South: 1st Technical Report 23 June 2013 to 27 August 2014.
Available online: http://hdl.handle.net/11427/9695 [Last
accessed 11 February 2015].
About evaluation:
• Patton, M. Q. (2008). Utilization-focused evaluation.
California: Sage Publications Inc.
• Ramirez, R. and Brodhead, D. (2013). Utilization Focused
Evaluation: A Primer for Evaluators. Penang: Southbound.
• Scriven, M. (1991). Evaluation Thesaurus. California: Sage
Publications Inc.