Presented at AIDS2012

1. Evaluating impact of OVC
programs: standardizing our
methods
Jenifer Chapman, PhD
Futures Group/MEASURE Evaluation

2. Overview
 Evaluation vs. monitoring vs. case management
 Update on the MEASURE Evaluation OVC
program evaluation tool
 Key questions

3. Evaluation vs. monitoring vs. case
management
Case management questions
 What are the child’s immediate priority needs?
 Is child in school now?
 Have earlier priority needs abated?
Monitoring questions
 How many children are receiving services?
 How many social workers have been trained?
 Is the program being implemented as planned?
3

4. Evaluation vs. monitoring vs. case
management
Evaluation questions:
 Did program beneficiaries show improved well-
being over time?
 What proportion of households are food secure,
compared to 5 years ago?
 Can improvements in well-being be attributed to
the program?
4

5. Tools for answering these
questions
 We have case management tools, and
monitoring tools…
 We lack a standardized approach and tool for
evaluating the impact of OVC programs globally

6. An evaluation tool, in context
 Purpose: To assess the impact of a program (on
child/household wellbeing) over time
 Who uses it?: Trained data collectors /
researchers
 Among whom is it applied?
 A sample of beneficiaries
 Sometimes a matched group of potential
beneficiaries

7. What is an evaluation tool?
 What information is collected?
 Client contact information / demographics
 Wellbeing information that is changeable over time
 Services provided
 Information must be useful at program/national
level (different information than needed for case
management/care planning)…

8. Evaluation protocols
 Evaluation requires a protocol detailing:
 Sampling and recruitment strategy
 Data collection procedures
 Data management and data analysis
 Dissemination
 Ethical approval is required. Evaluation is not
part of routine QI involving service providers.
8

9. MEASURE Evaluation OVC
Program evaluation toolkit
9

10. The problem
Lack of standardized, objectively verifiable,
impact indicators of child wellbeing
10

11. The vision
 Quantitative child outcomes and household
outcomes measurement tools, valid across
countries at the population level
 With survey protocol, analysis plan and report
template, and training materials
 Progress:
 final shortlist of 12 child & 3 caregiver/HH impact
indicators
 Draft tools in discussion
 Piloting in Nigeria, Mozambique, Zambia & Ethiopia
11

12. Methods
 Extensive research (lit reviews, international/national child
wellbeing tools and indicators, OVC program evaluation
tools, national OVC M&E plans, DHS, etc.)
 Result: 100s of child & household wellbeing indicators
 Indicator analysis against defined criteria & wide
stakeholder discussions
 Result: shortlist of indicators

13. Inclusion criteria
1. Measures impact/outcomes
2. Amenable to change from program interventions
3. Relevant across a wide range of interventions
4. Contributes to a holistic vision of child wellbeing
5. Objectively verifiable
6. Easy to implement
7. Relevant across different regions / countries
8. Relevant or easily adapted across age and sex
13

14. 12 Child wellbeing indicators
1. Percent of children malnourished
 6-59 months old: measurement of middle-upper
arm circumference (MUAC)
 >60 months old: measurement of weight and
height
2. Percent of children with recent diarrhea
 Have you / has the child had diarrhea in the two
weeks preceding the survey?
14

15. 12 Child wellbeing indicators
3. Percent of children with recent fever
 Have you / has the child had a fever in the two
weeks preceding the survey?
4. Percent of children who are too sick to
participate in daily activities
 In the last 2 weeks, have you / has the child been
too sick to participate in daily activities?
15

16. 12 Child wellbeing indicators
5. Percent of children reporting irregular food
intake
 Have you / has the child gone a whole day or night
without eating in the last 4 weeks?
6. Percent of children fully immunized
 Has child received all age-appropriate
immunizations?
16

17. 12 Child wellbeing indicators
7. Percent of children with basic shelter
 Is the place that you / the child slept last night
protected from the weather?
8. Percent of children with basic social support
 Is there someone you / the child can go to, to help
solve a problem?
17

18. 12 Child wellbeing indicators
9. Percent of children who have a birth
certificate / identification card
 Does the child have a birth certificate or
registration / ID card?
10. Percent of children currently enrolled in
school
 Are you / is the child currently enrolled in school?
18

19. 12 Child wellbeing indicators
11. Percent of children regularly attending
school
 During the last school week, did you / the child
miss any school days for any reason?
12. Percent of children who progressed in
school over time
 What is the highest level of education you have /
the child has completed?
19

20. 3 Household wellbeing indicators
1. Percent of households in which caregiver
reports basic social support
 Is there someone you can go to, to help solve a
problem?
2. Percent of households that are food
insecure due to lack of resources
 In the past 4 weeks, was there ever no food to eat
of any kind in your household because of a lack of
resources to get food?
20

21. 3 Household wellbeing indicators
3. Percent of households able to access
money to meet important family needs
 If you needed money to meet an important family
need, such as to pay for school fees, pay for
transportation, or purchase for food, how would
you pay? (smaller expenses)
 If you needed money to meet an important family
need, such as to pay for a family emergency, pay
for a house repair, or pay for medical treatment,
how would you pay? (larger expenses)
21

22. And the tool?
 Indicators form the basis of two tools:
1. Child questionnaire (ages 10-17)
2. Caregiver questionnaire (including questions on
caregiver wellbeing, household, child wellbeing)
 Draft tools available Autumn 2012
 Tools will be piloted in Autumn/Winter 2012
 Final tools & guidance documents available
mid-2013

23. For discussion
 PSS and HES indicators – your experience,
please!
 Methodological fine points (or not so fine
points)…what works, what does not?
 How is your organization currently evaluating
programs?
 How will this tool support you in evaluation?
 What more is needed?
23

24. The research presented here has been supported by the
President’s Emergency Plan for AIDS Relief (PEPFAR)
through the United States Agency for International
Development (USAID) under the terms of MEASURE
Evaluation cooperative agreement GHA-A-00-08-00003-
00. Views expressed are not necessarily those of
PEPFAR, USAID or the United States government.
MEASURE Evaluation is implemented by the Carolina
Population Center at the University of North Carolina at
Chapel Hill in partnership with Futures Group, ICF
International, John Snow, Inc., Management Sciences for
Health, and Tulane University.

25. Round-up

26. Our Community of Practice
 There is lots going on in OVC M&E but cross-
country dialogue is lacking
 ChildStatusNet is being revamped as an OVC
M&E Community of Practice (or even M&E of
community-based interventions)
 Forum to exchange ideas & share tools /
indicators / methods
 Regular webinars with guest speakers

28. For more information, contact:
Jenifer Chapman: jchapman@futuresgroup.com
Karen Foreit: kforeit@futuresgroup.com
http://www.cpc.unc.edu/measure/