Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

User focused evaluation: Feedback on research capacity building webinars

A presentation given at the deferred UCT 2015 Teaching and Learning Conference held at UCT on 30 March 2016. This presentation briefly outlines some of the evaluation work conducted on ROER4D’s research capacity building webinars, focusing on the first 12-webinar series conducted in 2014.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

User focused evaluation: Feedback on research capacity building webinars

  1. 1. Sarah Goodier ROER4D Evaluation Advisor UCT Teaching & Learning Conference 30 March 2016 User focused evaluation: Feedback on research capacity building webinars 3/30/20161
  2. 2. Outline  The project being evaluated: ROER4D  The evaluation work: what & how  The key evaluation question  Findings  Recommendations & next steps  Lessons learnt
  3. 3. THE ROER4D PROJECT
  4. 4. in the Global South  In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South? Research on Open Educational Resources for Development http://roer4d.org/
  5. 5. ROER4D has 86 researchers across 18 sub-projects in 26 countries across 16 time zones
  6. 6. EVALUATING ROER4D
  7. 7. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Key Evaluation Areas EVALUATION Using the utilization focused evaluation (UFE) framework
  8. 8. What is utilization focused evaluation (UFE) UFE is centered around intended use by intended users - What do the users (key stakeholders) want to know? - How will answers to the evaluation questions potentially help to improve the project?
  9. 9. UFE in 12 steps: Steps are iterative, not linear Utilization Focused Evaluation Framework (Adapted from Ramirez & Brodhead, 2013)
  10. 10. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Key Evaluation Areas Webinars
  11. 11. Webinar evaluation timeline 2014 series (Feb – Oct) 2015 (Feb – July) Evaluation Data presented today This 12-session series focused on the harmonisation of survey questions that were to be used across the different ROER4D sub-projects Iterative evaluation work during this series of webinars
  12. 12. Key Evaluation Question  To what extent have ROER4D Network Hub webinars been a mechanism to potentially grow the research capacity of researchers working on the ROER4D project? [Has the process worked & how can we improve it?]
  13. 13.  To answer the question, focused on: ◦ Recorded attendance (taken at each webinars) ◦ Survey results of the sub-project researchers regarding the webinars (sent after the completion of the webinar series)
  14. 14. FINDINGS
  15. 15. 2014 Session Attendance • Overall, the trend was a decrease in attendance over the course of the sessions
  16. 16. 2014 Session Attendance • For all but one of the last five sessions, the network hub team made up half or more of the attendees
  17. 17. 2014 Session Attendance • Approximately half of the ROER4D network didn’t attend any of the sessions; • Of those who did attend at least one session, most attended only up to six sessions
  18. 18. Research Capacity and Question Harmonization Survey Findings Key findings: • Those who did attend/view the session found them useful • Scheduling issues were a factor for those who didn’t attend any of the live sessions. • The fact that these sessions were voluntary was flagged as a less valuable point about the question harmonization process. This was seen to potentially create gaps in terms of research capacity across the project. • Having recordings and the research questions and concept definitions available was useful.
  19. 19.  In summary, this first webinar process has experienced varying degrees of success: ◦ Useful for those who attended/viewed the sessions ◦ Several barriers (e.g. timezones) flagged by the researchers & engagement in the webinars was sporadic across the sub-projects
  20. 20. RECOMMENDATIONS & NEXT STEPS
  21. 21. Evaluation recommendations on webinars – Encourage each subproject to have at least 1 researching member attend the live webinar – Keep recording webinars and point the subprojects to the recordings after the session as a reminder – Have a forum for questions/discussion (before and) after the webinar so that those who don’t attend live can also benefit from shared discussion – Have session repeated (2x) to incorporate different time zones (if there is capacity)
  22. 22. *Update: several of the recommendations were picked up in the 2015 series (e.g. having repeat webinars scheduled at different times) Constant dialogue with the ROER4D team and users as the evaluation work continues Next steps
  23. 23. Some lessons from evaluating ROER4D Find a framework for your evaluation (e.g. UFE): valuable guiding structure For both internal evaluation and external evaluations, never underestimate the importance of iterative engagement : more engagement = better understanding of project & what matters to users
  24. 24. This work is licensed under a Creative Commons Attribution 4.0 International License. Website: http://roer4d.org Contact author: Sarah Goodier sarah.goodier@uct.ac.za @SarahGoodier Excluding images, screenshots and logos and/or unless otherwise indicated on content Thank you!
  25. 25. Further reading: About ROER4D: • Hodgkinson-Williams, C. (2013). Research on Open Educational Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015]. • Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015]. About evaluation: • Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage Publications Inc. • Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A Primer for Evaluators. Penang: Southbound. • Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications Inc.

×