Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches for media development
Similar to FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches for media development (20)
Recently uploaded
Recently uploaded (20)
FoME Symposium 2015 | Workshop 8: Current Evaluation Practices and Perspectives on more Effective Media Development Evaluation | Jessica Noske-Turner: Towards more effective approaches for media development
- 1. Towards more effective evaluation approaches for media development
- 2. Current projects Evaluating Communication for development: supporting adaptive and accountable development Mobilising Media for Sustainable Outcomes in the Pacific Region RMIT University School of Media & Communication 2
- 3. PhD Research Question & Design Research Question: How can the impacts of Australian media assistance on social change and governance be most effectively evaluated and understood? Phase 1: Document analysis: Media Assistance Evaluations between 2002-2012 (total of 47) Phase 2: Interviews Interviews with media assistance evaluators (consultants and in-house, total of 10) Phase 3: Case study: Cambodian Communication Assistance Project (CCAP), funded by AusAID-funded project, managed by ABC International Development.
- 4. Findings: Key challenges • Conceptual ambiguities • Bureaucratic systems • Complex epistemological and political undercurrents in the evaluation discipline RMIT University School of Media & Communication 4
- 5. 1. Conceptual ambiguities RMIT University School of Media & Communication 5
- 6. 2. Bureaucratic systems the codification of approaches that are meant to accomplish positive outcomes into mechanical checklists and templates that not only fail to achieve their intent but actually lead to even worse outcomes. (Anderson, Brown and Jean 2012: 67) Proceduralization:
- 7. 2. Bureaucratic systems Authored by Total Commissioned (/required) by donor Commissioned (/required) by project External consultant 27 19 8 Donor 5 5 Project 5 2 3 Consultant + Donor 2 2 Consultant + Project 1 1 Donor + Project 1 1 unknown 6 6 Total 47 35 12
- 10. • ‘Nuancing’ reports • Being ‘circumspect’ • Pre-cooking evaluations Dependence on Independence 2. Bureaucratic systems
- 11. •Simplicity and complexity •Proving vs improving •Positivism vs participation . Evaluation pistemologies
- 12. Adapted from Manyozo (2012) Media Development Focus on industry Aims to improve governance. Media for Development Focus on content Aims to educate and inform. Participatory Communication: Focus on dialogue Aims to foster participation and self- determination. Conceptual ambiguities
- 13. Prototype
- 14. Prototype
- 15. C4D Design & Evaluation in the Pacific RMIT University School of Media & Communication 15 • Workshops, field visits, pilot – Capacity building – ‘Innovation’ – Proportionate – Piloting the ‘LEAD4innovation toolkit’or C4D IDEAS Toolkit – Incudes a facilitator’s guide Rainbow Framework Toolkit Module Objectives and Activities MANAGE an evaluation or evaluation system Module 2: Stakeholder Mapping Who who has an interest or role in our project and evaluation? How can we communicate with them? Module 3: Ethics What are the potentially ethical issues in media and communication projects and evaluation? How can we prevent these from becoming problems? REPORT & SUPPORT USE of findings Module 4: Planning Data Use What is our macro-timeline for sharing evaluation insights? How can we build in continuous learning? What are the most appropriate ways to share our insights? DEFINE what is to be evaluated Module 5: Define What are we trying to do, how are we trying to do it? FRAME the boundaries for an evaluation Module 6: Ask Questions Based on our main goals, what are our key evaluation questions? DESCRIBE activities, outcomes, impacts and context Module 7: Planning Information and Data Collection Which data collection methods will give us the best information to answer our questions? Which methods are realistic? UNDERSTAND CAUSES of outcomes and impacts Module 8: Making Sense of Data How do we manage our data? How can we make sense of and analyse different kinds of data? (SYNTHESISE data from one or more evaluations) (Included in the Facilitator’s Guide) (Meta-analysis of groups of projects) Contents of the LEAD4innovation/ C4D IDEAS Toolkit against the Rainbow Framework
- 16. RMIT University School of Media & Communication 16
- 17. Conventional Baselines Living Baselines Originates from experimental designs Challenges conventional notion of ‘baselines’ Assumes controlled environment Acknowledges complexity (feedback loops, interrelationships) Simple model of cause and effect Social change occurs through interaction between multiple factors and agents Anticipates linear trajectories of change Anticipates multiple, emergent changes which are unknowable in advance For measuring For generating insights and learning Research is undertaken twice (before and after), or at most, three times (if there is a midline) Research is undertaken periodically Fixed, contained Adaptive, expanding Positions development knowledge in the domain of experts Privileges local knowledge “Living Baseline”
- 18. RMIT University School of Media & Communication 18 Noske-Turner, J. (2015). 10 Years of Evaluation Practice in Media Assistance: Who, When, Why and How?. In Nordicom Review (Vol. 36, No. Special Issue, pp. 41-56) Noske-Turner, J. (2014). Evaluating the impacts of media assistance: problems, and principles. Global Media Journal German Edition, 4(2), 1-21. Publications
Editor's Notes
- Break after this slide – to find out about the people in the room and their interests/expectations For small group ask everyone to introduce themselves, who they are, what they know, what they want to find out In a larger group, ask what people are interested in/questions, write up a flipchart paper. As they answer, ask them to introduce themselves. Can also get a show of hands in answers to questions.
- Overview of research design. Introduce the research question, skip over methods & come back to it if there are questions about it.
- I found that rather than there being an issue of a lack of suitable methods, the problems for media assistance evaluation had more to do with the difficulty in articulating the theory of change, the accountability-focused bureaucracy within evaluation practices operate, and the difficulties of navigating the myriad of evaluation approaches. Open up for discussions about the participants’ experiences and challenges in media assistance evaluation.
- (After open discussion about the challenges, go through the challenges and weaknesses found through the research) Introduce weaknesses in developing theories/models of change – many goals and objectives are very high level (democracy) and abstract (press freedom), very hard to evaluate and this tends to lead to us resorting to counting outputs (how many journalists trained, how many articles written etc.) and not being able to make a link to impact.
- The timing of evaluations and the methods used were not deliberate decisions based on an idea of this being ‘best practice’, but rather was an outcome of the evaluation systems. Procedures are intended to standardise practice, ensure best practices are used across the agency, but in practice these can often result in less time and less flexibility. I was drawn to Anderson, Brown and Jean’s definition of ‘proceduralisation’ as a way to describe what seems to be happening in media assistance evaluation. They say the proceduralisation is “the codification of approaches that are meant to accomplish positive outcomes into mechanical checklists and templates that not only fail to achieve their intent but actually lead to even worse outcomes.” This description resonated strongly with the accounts by evaluators of doing evaluations, and of the project stakeholders interviewed. Evaluation processes and decisions are increasingly driven by standardised systems and templates, and less time is made available to engage with the evaluation design process.
- Not representative – but of those evaluations published it indicates that most evaluations are undertaken by consultants. And most are commissioned or required by the donor. This was broadly supported by the evaluators, although in-house researchers and evaluators commented that many of their evaluations remain internal and unpublished.
- Second broad area of current practice I observed was the patterns of when an evaluation might be undertaken. What is striking about this graph is that 11 evaluations of the 47 were taken undertaken 3 years after the project had started, 10 were undertaken after 5 years, but only 1 was undertaken after four years. As I go on to show, the timing patterns of evaluation have more to do with pre-determined project cycles than on any considered decision about when impacts might be observable.
- Next I looked at the methods used – there was a high degree of repetition. Most used qualitative methods only. These were generally typified as using a document review with key informant or stakeholder interviews. In addition to those 29, 8 of the 14 mixed methods were also based on this combination. Therefore in all, 37 of the 47 evaluation reports in the sample were based on a document review with key informant interviews. The consistency of this template – an externally authored evaluation report, undertaken after 3 or 5 years, using stakeholder interviews and document analysis – led me to question why this pattern of evaluation practice was so pervasive. I began to consider the evaluation documents in terms of their materiality – their function and construction. The centrality of the document in the system of evaluation became more apparent as I continued the research – evaluators accounts, and in the case-study.
- Operationalising accountability – consultant evaluators. Repeatable technique, systematic solution for quality assurance processes. ‘Nuancing’ reports Being ‘circumspect’ Pre-cooking evaluations
- The evaluation field is a huge and dynamic field in its own right, with several academic journals and shelves of books. In my reading of the literature I found three polemic debates. The first relates to how to manage complexity. Our kinds of projects are often referred “complex”. But there are two reactions to this complexity – do we simplify it, to reduce it to basic elements that can be predicted and measured, or do we to retain the complexity by allowing for exploration of unpredictable impacts, and adaptation throughout? The second debate is between whether evaluations should prove impacts or improve the program. The third debate relates to positivist (very scientific approaches, seeking objectivity), contrasted with participatory approaches to evaluation. We often feel a pressure to produce objectivity and quantification. However, there has been a push towards involving local communities and counterparts in evaluating, and judging the worth of development projects. Sometimes evaluation designs can be very confused about these different approaches.
- Manyozo’s Framework
- As part of my thesis I created a prototype of a tool to help with this process in the form of a set of evaluation design cards. The cards are colour coordinated according to the model of change – that is whether the model of change refers to content (which is red), dialogue (which is yellow) or system or industry approaches (which is blue). Each card has one question on it that relates to an aspect of that model of change. The idea is that in a workshop project stakeholders could choose the questions and they consider important, and rank them accordingly, then compare these with others. In doing so they would be aware of how the questions relate to the models of change. On the back of the cards are some suggested methods that could answer the evaluation question, with an indication of the required resources for each option.
- There are two other types of question in addition to impact-questions. These are interpretive questions, which related to interpreting the data generated about impact and making judgements – so for example, questions about strengths and weaknesses. This helps to make sure that evaluation plans account for questions about proving impacts, and about how to improve, and shows how these can complement eachother. And there are what I’ve called accountancy questions, which need usually to be addressed for formal reporting requirements. Ranking these could help to generate insights among all stakeholders of the priorities, demands and shared understandings of the project.
- As part of one of the current research projects I’ve been working with colleagues at RMIT we’ve been working on a toolkit intended for use in innovation funds to assist small-scale project teams navigate this take in a way that promotes learning by linking the process of designing and refining a project and the designing of the evaluation. There are 8 modules which basically follow the Rainbow Framework by Better Evaluation.
- This is a tool from one of the modules in the toolkit that intends to help teams articulate a conceptual understanding of the project. It is based on the three component by Manyozo but separates Infrastructure and Capacity into two different components.
- Finally, my colleague Jo Tacchi and I have developed a concept of a Living Baseline.