1.
Result Based Monitoring andEvaluation for Agriculture and Rural Development Safari Park Hotel, Nairobi on 25th- 26th June 2012 Stella Clara Massawe ReSAKSS-ECA, ILRI

2.
Session One

3.
Learning Objectives Session 1o Discuss the concept of Result Based M&Eo To discuss the role of M&E in agriculture and rural development interventionso Developing a theory of change in M&E contexto Practical Assignment

4.
Monitoring• Tracks what is happening within an intervention.• Continuous process of observation, interpretation and institutional learning• Is an internal management tool whose purpose is to: • to take timely corrective action in the day-to-day management decisions • Track progress, or lack thereof, in the achievement of results. • Facilitate subsequent evaluation

5.
Evaluation 1• Periodic• Systematic assessment of ongoing or completed interventions• Looks at their design, implementation and results• Deal with strategic issues such as: Performance, quality, relevance, effectiveness, efficiency, and sustainability.• Addresses various questions:

6.
Evaluation 2• Addresses various questions: – Why: What caused the changes – How: What was the sequence or processes that led to successful (or not) outcomes – Compliance/ Accountability: Did the promised activities actually take place and as they were planned? – Implementation process: Was the implementation process followed as anticipated, and with what consequences?

7.
Why M&E System for Agriculture?• Accountability & Transparency: development effectiveness….national SHs, donors, beneficiaries …..• Learning: Decision making, learning…… – Key SHs: accountability & development effectiveness – Management: evidence decision making, right strategy? – Implementers: delivering on time? Budget? and target?• Adaptive Management/ Managing for impact: Respond to changing circumstances, increased understanding from knowledge generated by M&E• If done well contributes ++ to team morale and foster an atmosphere of transparency and professionalism.

8.
What to Monitor and Evaluate• Best to develop at the beginning of the design process• Theory of change indicates intervention-to-impact pathways• Need to define key indicators along the pathways, including their measurements, methods, and data requirements• Laying out the theory of change for an intervention is important when deciding what to monitor and evaluate

9.
Underlaying Logic or Theory of change• Theory of how an initiative leads to the desired results,• Describes the causal logic of how and why a particular intervention will reach its intended outcomes• Often developed during project conceptualization.• Involvement of stakeholders to put together a TOC improve programme design

10.
Thinking about impact pathways• Example from the regional framework document• Channels of impact on poverty and food security• A key step in reaching the goals (reducing poverty and increasing food security) is by improving the productivity of existing assets and/or by accumulating more assets

11.
Channels of impact: Examples INTERVENTION AREA OUTCOMES / IMPACT CHANNELS •Outcomes: (i) lower transactions cost, (ii) higher marketing margins, (iii) lower postharvest losses, (iv) Market Access and greater volume/value of trade Trade Facilitation •IP: (i) direct via higher income, and (ii) indirect via higher output-input price ratios and/or greater access to production technologies •Outcomes: (i) greater adoption of profitable technologies and enterprises, (ii) higher productivity, output, and NRM & Agricultural marketable surplus, and (iii) more sustainable production Production activities •IP: (i) direct via higher food supply and income, and (ii) indirect via higher real income (lower food prices) •Outcomes: greater protected productive assets and Early Warning and investments in times of disasterDisaster Preparedness •IP: (i) direct via reduced negative effect of disaster on production, consumption, marketable surplus, and trade

12.
Components of a Theory of Change Model Should include the basic elements of theory of change: causal chain, outside conditions and influences key assumptions

13.
A theory of change must• Depict a sequence of inputs, the project or policy will use, activities, outputs, outcomes and impacts expected• Identify events or conditions that may affect obtaining the outcomes• Identify the assumptions the programme is making about cause and effects• Identify critical assumptions that, based on the policy and environmental context and review of literature, the evaluation needs to examine

14.
Components of TOC Your Planned work (Implementation)• Describes what resources you think you need to implement your program and what you intend to do. – Resources/ Inputs: include the human, financial, organizational, and community resources a program has available to direct toward doing the work. – Activities: What we do. Actions taken to convert inputs to outputs. Stated with a verb (“market”, “train”, “deliver”). – Outputs: What we produce. Are the direct products (tangible goods and services) resulting from program activities. These are usually expressed as nouns.

15.
Components of TOC Your Intended Results• All of the program’s desired results – Outcomes: Why we do it. Are the specific changes in program participants’ behavior, knowledge, skills, status and level of functioning. – Results likely to be achieved once the beneficiaries use project outputs (usually achieved in short-to medium term) – Final outcomes/ Impact: Long-term changes that result from accumulation of outcomes. – Is the fundamental intended or unintended change occurring in organizations, communities or systems as a result of program activities. Can be similar to strategic objectives.

16.
Modeling Theory of change• Gertler et al., 2010 indicates that theory of change can be modeled in various ways e.g. – Using theoretical models, – Logic models, – Logical frameworks and outcome models, – Outcome models, – Result chain

17.
A result Chain Goal • Long-term, widespread improvement (Impacts) in societySupply Results +Supply • Intermediate effects of outputs on Outcomes clients • Products and services produced Outputs ImplementationSupply • Tasks personnel undertake to Side Activities transform inputs to outputs • Financial, human, and material Inputs resources

18.
Results-Based Monitoring:Agriculture Extension Programme Goal • Higher yields, higher income levels; (Impacts) increase access to food Outcomes • Increased better farming skills; increased adoption of new technologies Outputs • Number of farmers visited, number trained in field days Activities • Farm visits, farmer field days, distribute handouts Inputs • Extension workers, materials

19.
Theory of Change Template Strategies (5) Assumptions (6)Influential factors (4) Problem Issue (1) Desired results (3)• Foreseen budget for Low productivity, poor maize (Outputs, outcomes and fertilizer subsidies is harvest, inadequate food, need impacts) available in time for food aid • Fertilizer is more• Climatic conditions accessible• Government • Fertilizer is more policies affordable• Macro-economic Community Needs/ Assets (2) • use of fertilizer increases conditions • Almost 38% of the population needed emergency food aid. • soil able to support• Programme increase food availability implementation • Fertilizer is not affordable • Reduce hunger capacity • Increased income (from sales)

20.
Session two

21.
Session 2: M&E framework• To discuss the meaning of M&E framework• To discuss the relationship between TOC and M&E framework• Discuss examples of M&E frameworks• Developing an overarching FW

22.
Participatory Discussion• What is the M&E system?• Using the case of Agriculture M&E framework in your country

23.
What is a M &E Framework?• Builds on the Theory of change• Lists each indicator from the program logic model• Presents how indicators are defined and calculated• Defines Who, What, When, How by:• Identifying who is responsible for data collection• What the data source is ( the methods we use to collect data, report and disseminate)• When and how often an indicator is measured

24.
M&E Frameworks (1)• M&E Framework outlines the plan for monitoring in concrete steps providing the who, what, where and when• Define relationships among inputs, activities, outputs, outcomes, impacts and external factors.• Assist with thinking through programmatic strategies, objectives and planned activities, and whether they are indeed the most appropriate ones to implement.• Plan about implementation of M&E activities

25.
M&E Frameworks (2)• Developing a corresponding M&E plan that acts as a monitoring tool by defining how information from the programme will be tracked.• Develop the framework and plan before activities are implemented.• Different kinds of interventions will need different kinds of frameworks, tools and indicators.• If M&E activities and tools are considered and built into programmatic work from the start, the resource and time burden is minimized.

26.
Designing, Building and Sustaining agriculture M&E system•Borrow from the model: Ten steps to Designing, Building &Sustaining a Result-Based M&E System –Kustel and Rist,2004

27.
Step 1: Readiness assessment should be conducted to understand:• Willingness of the stakeholders• Incentives and Demand for M&E, what disincentives may hinder• Organizational Capacity for M&E (technical, financial, managerial)• key players that will be responsible for program implementation.• Roles and responsibilities• Barriers M&E technical staff, project managers and senior managers involved in strategy development should be involved

28.
Step 2: Agreeing on Specific Outcomes to Monitor and Evaluate• Generate interest in assessing the outcomes and impacts in addition to the implementation issues• Need to be derived from strategic priorities of the country• Involves all relevant stakeholders to arrive at a mutually agreed set of goals and objectives for the program.• Linkages: Programme strategy-- theory of change—translate into key set of indicators• Clearly setting outcomes-and deriving indicators based on them-essential to designing and building result-based M&E system M&E technical staff, project managers and senior managers involved in strategy development should be involved

29.
Step 3: Select Key indicators to monitor outcomes• Numerical measurement that tracks progress (or not) toward achieving an outcome• SMART---Specific, Measurable, Achievable, Relevant and Time- bound• Decide how trends in agreed outcomes will be measured. – For example, if children’s schooling were an important outcome for a program: use the proportion of children enrolled in school, test scores, school attendance, or another metric?• Indicator development drives: data collection, analysis and reporting• The costs of measurement need to be considered• Where possible reduce costs by using already available indicators

30.
Minimum set of indicators (2)• Selecting indicators• All indicators should be neutrally stated, not as “increase in” or “decrease in”• Feasibility of deploying each indicators: Completing each cell in the matrix belowMatrix for indicatorsindicator Data Data Who will Frequency Cost to Difficulty Who will Who will source collectio collect? of collect to analyze use the n collection data collect and data method data report data

31.
Step 4: Baseline data, where are we today• Measurement of initial conditions, preprogram data• Very helpful in assessing the program’s impact• Provided qualitative and quantitative information• Sources of baseline data can be primary or secondary data• Decide who is going to collect data and how, develop data collection instruments

32.
Step 5: Planning for Improvements Establish Realistic Result Targets:• These targets can also be used to monitor results.• This effort includes setting periodic targets over time (for example, annually or every two years).• Consider the duration of the likely effects of the program• Important to consider other factors (political..)

33.
Step 6: Monitoring results• Involves the collection of good-quality data.• Data sources: Primary,& Secondary data sources, Project data, Spatial data• Establish data collection system ---------Appropriate forms for record & data management systems …Staffing: enumerators, data entry• Leverage Partnerships in data collection management, analysis and dissemination

34.
Step 7: Using Evaluations• Evaluation important in supplementing information require to gauge whether there are achievements with regard to impacts• Evaluation helps in in indicating whether: – We are doing right things (strategy) – We are doing things right (operation) – There are better ways of doing it (learning)

35.
Data Collection Methods Panel Key informant Surveys interviewsConversationwith Focus One-Time concerned Group Surveyindividuals Interviews Participant Community Observation Direct Census Interviews observation Reviews of official records (MIS and admin Field Field data) experiments visits Questionnaires Informal/Less Structured Methods More Structured/Formal Methods

36.
Steps 8-10• Step 8: Reporting: Feedback sessions & strategic communication activities, appropriately targeted to different audiences--------- M&E, COM, KM, PRO, MA Reports, newsletters, issue briefs, website, CDs• Step 9 & 10: Using findings and sustaining the system

37.
Session ThreeCase Study: CAADP M&E Framework

38.
Rationale for common M&E framework:The need to assess impact more broadly, adding up to more than of progress and performance associated with individual pillars:(i) to comprehensively assess whether and how investments and policies surrounding the CAADP implementation are having their desired impact on targeted growth and poverty reduction goals(i) individual pillar M&E frameworks focus on specific targets without considering how they interact with those of other pillars (e.g. through price effects) to affect the overall CAADP goals and objectives

39.
Idea behind M&E framework

40.
Rationale for common M&E framework• A mechanism by which processes put in place, commitments etc can be regularly and transparently measured against stated targets and if necessary can lead to the revision of the processes – Are countries achieving the targeted growth rates? (6%) – Are countries investing at the targeted level? (10%) – Are these investments having their intended impacts on poverty and hunger?• To bring cohesion across the different systems being developed to track specific components of CAADP, including the individual CAADP pillar M&E systems, APRM and MAF• To inform the review processes established by CAADP-PP (mutual, peer and progress reviews)• To further inform policy-making and dialogue

41.
http://www.resakss.org/

42.
THANK YOU

43.
References• World Bank (2004). Designing and Building a Results-Based Monitoring and Evaluation System: A Tool for Public Sector Management, A Workshop for Government Officials and Their Development Partners, Presentation made on 20th October, 2004• Gertler,PJ., Martinez, S.,Premand,P., Rawlings,LB., Vermeersch, CMJ. (2011) ImpactEvaluation in Practice, The International Bank for Reconstruction and Development / The World Bank, Washington DC., USA• Kusek JZ. and Rist RC (2004).Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners, The International Bank for Reconstruction and Development / The World Bank, Washington DC., USA• Benin, S.; Johnson, M.; Omilola, B.; Beintema, N.; Bekele, H.; Chilonda, P.; Davis, K.; Edeme, J.;Elmekass, A.; Govereh, J.; Kakuba, T.; Karugia, J.; Makunike, R.; Massawe, S.; Mpyisi, E.; Nwafor, M.;Olubode-Awosola, F.; Sanyang, S.; Taye, B.; Wanzala, M.; Yade, M.; Zewdie, Y. Monitoring and Evaluation (M&E) System for the Comprehensive Africa Agriculture Development Programme (CAADP). ReSAKSS Working Paper No. 6. International Food Policy Research Institute (IFPRI). Washington, D.C.:

44.
Exercise 1 Constructing and Using a Theory of ChangeTask objective: to enable the participants practice how to develop a theory of change for their programme Estimated time: 30 minutes

45.
Task 1• Consider a programme you are currently working with or one which you are familiar but not directly involved. Develop a graphic showing of the theory of change for this programme or project.• Be sure to identify the assumptions underlying the programme or project• Use simple theory of change template (please only include a few outputs and activities)• Reference pages 150-171