Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sarah Goodier
ROER4D Evaluation Advisor
EDN4502W: Research & Evaluation of Emerging Technologies
9 September 2015
Evaluati...
WHAT IS EVALUATION?
… the systematic determination of the quality or value
of something (Scriven, 1991)
Evaluation is…
Scriven, M. (1991). Eva...
… the systematic determination of the quality or value
of something (Scriven, 1991)
• It is done according to a specific p...
… the systematic determination of the quality or value
of something (Scriven, 1991)
• It is done according to a specific p...
… the systematic determination of the quality or value
of something (Scriven, 1991)
• It is done according to a specific p...
• Assessment of merit or worth
Are specific standards met?
• Improvement
What can be improved? And how?
• Oversight and co...
8
Research vs Evaluation
Add knowledge to a
specific field
Modified from: http://aea365.org/blog/john-lavelle-on-describin...
Exercise:
 Come up with a example of 1) Research and 2)
Evaluation that show the difference between
the two
(See the prev...
EVALUATING ROER4D
in the Global South
 In what ways, and under what
circumstances can the adoption of
OER address the increasing
demand for...
1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
...
1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
...
ROER4D Evaluation and Communication
supported by DECI-2
DECI-2
An IDRC funded research project to build and mentor Communi...
UTILIZATION FOCUSED
EVALUATION (UFE)
What is utilization focused evaluation (UFE)
UFE is centered around intended use by intended
users
What do the users (key ...
UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. ...
UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. ...
UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. ...
DEVELOPING AN
EVALUATION STRATEGY
ROER4D Process of developing evaluation strategy
1. Understand what is needed in terms of the scope of evaluating the
ROER...
ROER4D Process of developing evaluation strategy
ROER4D Process of developing evaluation strategy
ROER4D Process of developing evaluation strategy
4. Connect with members of the ROER4D, where needed (surveys,
interviews,...
1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
...
UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. ...
Next steps
 More data collection and analysis (ongoing for webinars, Twitter as well as other
social media platforms)
 F...
SUMMARY
… the systematic determination of the
quality or value of something (Scriven, 1991)
• It is done according to a specific
p...
Some lessons from evaluating
ROER4D
 Find a framework for your evaluation (e.g. UFE)
 For both internal evaluation and e...
Further reading:
About ROER4D:
• Hodgkinson-Williams, C. (2013). Research on Open
Educational Resources for Development in...
Links and license
This work is licensed under a Creative Commons Attribution 4.0 International License.
Website: http://ro...
Upcoming SlideShare
Loading in …5
×

Evaluation: a means to gain insights into and improve the ROER4D project

A shortened version of a presentation given to the EDN4502W: Research & Evaluation of Emerging Technologies PGDip course on the evaluation strategy for ROER4D.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Evaluation: a means to gain insights into and improve the ROER4D project

  1. 1. Sarah Goodier ROER4D Evaluation Advisor EDN4502W: Research & Evaluation of Emerging Technologies 9 September 2015 Evaluation: a means to gain insights into and improve the ROER4D project 9/11/20151
  2. 2. WHAT IS EVALUATION?
  3. 3. … the systematic determination of the quality or value of something (Scriven, 1991) Evaluation is… Scriven, M. (1991). Evaluation Thesaurus. Sage.
  4. 4. … the systematic determination of the quality or value of something (Scriven, 1991) • It is done according to a specific plan/method, in a thoughtful way Evaluation is… Scriven, M. (1991). Evaluation Thesaurus. Sage.
  5. 5. … the systematic determination of the quality or value of something (Scriven, 1991) • It is done according to a specific plan/method, in a thoughtful way • It measures merit/worth Evaluation is… Scriven, M. (1991). Evaluation Thesaurus. Sage.
  6. 6. … the systematic determination of the quality or value of something (Scriven, 1991) • It is done according to a specific plan/method, in a thoughtful way • It measures merit/worth • It is focused on a specific programme/intervention/technology Evaluation is… Scriven, M. (1991). Evaluation Thesaurus. Sage.
  7. 7. • Assessment of merit or worth Are specific standards met? • Improvement What can be improved? And how? • Oversight and compliance Is the technology cost-effective? The purposes of evaluation
  8. 8. 8 Research vs Evaluation Add knowledge to a specific field Modified from: http://aea365.org/blog/john-lavelle-on-describing-evaluation/ Purpose of the study Make judgments/decisions in a specific context Who decides what you study? Researcher derives the questions (researcher-focused) Funder/programme specifies the questions Testing a hypothesis/hypotheses Answering evaluation questions Social science methods & analyses Results The questions Sharing findings Research findings (and some explanation) Findings inform recommendations based on evaluation questions Publish – share with research community in the field Evaluation report to stakeholders (may also publish)
  9. 9. Exercise:  Come up with a example of 1) Research and 2) Evaluation that show the difference between the two (See the previous slide)
  10. 10. EVALUATING ROER4D
  11. 11. in the Global South  In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South? Research on OER for Development
  12. 12. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Objectives
  13. 13. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Key Evaluation Areas EVALUATION
  14. 14. ROER4D Evaluation and Communication supported by DECI-2 DECI-2 An IDRC funded research project to build and mentor Communication and Evaluation for IDRC flagship projects http://evaluationandcommunicationinpractice.net/
  15. 15. UTILIZATION FOCUSED EVALUATION (UFE)
  16. 16. What is utilization focused evaluation (UFE) UFE is centered around intended use by intended users What do the users (key stakeholders) want to know? How will answers to the KEQs help to improve the project? Remember that evaluation is about assessing worth of something
  17. 17. UFE in 12 steps: 1. Assessing program readiness 2. Assessing evaluator readiness 3. Identifying primary intended users 4. Situational analysis 5. Identification of primary intended uses 6. Focusing the evaluation 7. Evaluation design 8. Simulation of use 9. Data collection 10. Data analysis 11. Facilitation of use 12. Meta evaluation Steps in the Utilization Focused Evaluation (UFE) process
  18. 18. UFE in 12 steps: 1. Assessing program readiness 2. Assessing evaluator readiness 3. Identifying primary intended users 4. Situational analysis 5. Identification of primary intended uses 6. Focusing the evaluation 7. Evaluation design 8. Simulation of use 9. Data collection 10. Data analysis 11. Facilitation of use 12. Meta evaluation Progress towards fine-tuning evaluation Steps are iterative, not linear In progress
  19. 19. UFE in 12 steps: 1. Assessing program readiness 2. Assessing evaluator readiness 3. Identifying primary intended users 4. Situational analysis 5. Identification of primary intended uses 6. Focusing the evaluation 7. Evaluation design 8. Simulation of use 9. Data collection 10. Data analysis 11. Facilitation of use 12. Meta evaluation Progress towards fine-tuning evaluation In progress Different ROER4D objectives are at different stages in the UFE process
  20. 20. DEVELOPING AN EVALUATION STRATEGY
  21. 21. ROER4D Process of developing evaluation strategy 1. Understand what is needed in terms of the scope of evaluating the ROER4D project – the evaluation work is iterative by nature. 2. In collaboration with the ROER4D network hub team, formulate an evaluation plan, including what to evaluate and how. > The experience of the evaluation process and the effect this has is a key component of the evaluation. 3. Get feedback from DECI-2 around the evaluation work and incorporate this into the process.
  22. 22. ROER4D Process of developing evaluation strategy
  23. 23. ROER4D Process of developing evaluation strategy
  24. 24. ROER4D Process of developing evaluation strategy 4. Connect with members of the ROER4D, where needed (surveys, interviews, etc.) 5. Assess the findings. 6. Share products of the evaluation work (e.g. slides around process and results, reports, etc.) timeously to allow the findings and recommendations to effect change. 7. Be aware of all components of the evaluation work and collaborate/ share information where possible and where needed.
  25. 25. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Key Evaluation Areas – involving technology use Webinars Social media
  26. 26. UFE in 12 steps: 1. Assessing program readiness 2. Assessing evaluator readiness 3. Identifying primary intended users 4. Situational analysis 5. Identification of primary intended uses 6. Focusing the evaluation 7. Evaluation design 8. Simulation of use 9. Data collection 10. Data analysis 11. Facilitation of use 12. Meta evaluation In progress Focus on: • Data collection and analysis • Facilitation of use by the ROER4D team Progress in the evaluation
  27. 27. Next steps  More data collection and analysis (ongoing for webinars, Twitter as well as other social media platforms)  Feedback from the facilitation of use into refinements in the identification of primary intended uses and users to help focus the evaluation further  Constant dialogue with the ROER4D team and PIUs
  28. 28. SUMMARY
  29. 29. … the systematic determination of the quality or value of something (Scriven, 1991) • It is done according to a specific plan/method, in a thoughtful way • It measures merit/worth • It is focused on a specific programme/intervention/technology Evaluation is… Scriven, M. (1991). Evaluation Thesaurus. Sage. • Prepare an evaluation plan • Choose what to measure and how (indicators) • Collect the data (evidence) Research and Evaluation are different but share methods
  30. 30. Some lessons from evaluating ROER4D  Find a framework for your evaluation (e.g. UFE)  For both internal evaluation and external evaluations, never underestimate the importance of iterative engagement – more engagement = better understanding of the project and what matters to the evaluation users  Keep feeding back from the facilitation of use into refinements in the evaluation to help focus the evaluation further
  31. 31. Further reading: About ROER4D: • Hodgkinson-Williams, C. (2013). Research on Open Educational Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015]. • Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015]. About evaluation: • Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage Publications Inc. • Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A Primer for Evaluators. Penang: Southbound. • Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications Inc.
  32. 32. Links and license This work is licensed under a Creative Commons Attribution 4.0 International License. Website: http://roer4d.org Contact author: Sarah Goodier sarah.goodier@uct.ac.za @SarahGoodier Excluding images, screenshots and logos and/or unless otherwise indicated on content

    Be the first to comment

    Login to see the comments

A shortened version of a presentation given to the EDN4502W: Research & Evaluation of Emerging Technologies PGDip course on the evaluation strategy for ROER4D.

Views

Total views

686

On Slideshare

0

From embeds

0

Number of embeds

9

Actions

Downloads

8

Shares

0

Comments

0

Likes

0

×