Jen Chapman presents on the Orphans and Vulnerable Children Program Evaluation Tool Kit, which supports PEPFAR-funded programs and helps fulfill the aims presented in the USAID Evaluation Policy.

1. Evaluating the
impact of OVC
programs:
standardizing
our methods
Jenifer Chapman, PhD
Senior OVC Advisor
MEASURE Evaluation

2. Overview
 Background
 Purpose of the OVC Survey Tools
 Process of development
 Guiding principles
 Structure and content
 When to use the Tools
 When not to use the Tools
 Using the data

3. Tools in a Toolbox
 There is no single data collection tool that can
meet all OVC program targeting, case
management and M&E requirements.
 This set of survey tools responds to distinct
information needs related to program planning
and evaluation, and fills a tools gap.

4. The problem
 High investment in OVC programs BUT impact
is unclear & questions regarding “what works”
in improving household well-being
 Part of the challenge: lack of standardized
measures and tools for child and household
outcomes (well-being)
4

5. A proposed solution
Standardized questionnaires for use in a survey of
children ages 0-17 years and their adult caregivers

6. The purpose
 Standardize population-level child and caregiver well-
being data beyond what is available from routine
surveys
 Produce actionable data to inform programs and
enable mid-course corrections
 Enable comparative assessments of child and
caregiver well-being and household economic status
across a diverse set of interventions and regions

7. Who are these tools for?
 Local and international research institutions and
other implementing organizations with evaluation
agenda

8. Our Process
 Two step, participatory process:
 Build consensus around core impact indicators for
PEPFAR-funded OVC programs
 Develop OVC program evaluation (survey) tools

9. Distilling the core indicators
Step 1: Extensive literature search
Step 2: Gaps (HES, PSS) filled through targeted
research
 Result: >600 child/HH wellbeing
questions/indicators
Step 3: Analysis against 8 criteria
 Result: shorter list of questions for discussion
9

10. Inclusion criteria
1. Measures impact/outcomes
2. Amenable to change from program interventions
3. Relevant across a wide range of interventions
4. Contributes to a holistic vision of child wellbeing
5. Verifiable through another source
6. Easy to implement
7. Relevant across different regions / countries
8. Relevant or easily adapted across age and sex
10

11. Finalizing the core indicators
 External working group: solicited review from 49
stakeholders
 Received feedback from > 25 individuals/groups
 Finalized core set of 12 child and 3 household
measures
11

12. From indicators to tools:
Guiding principles
 Questionnaires measure program outcomes
 Program outcome data should be collected by
trained data collectors
 A documented protocol is required
 Protocol with tools needs to undergo ethical approval
both in the country of data collection and in the US
 Tools require pilot testing in new settings before use

13. Developing the tools
 Tools drafted with
strong stakeholder input
 Draft tools piloted in
Zambia (and Nigeria)
 Cognitive interviews to
test key concepts (e.g.
social support)
 Household pre-test of
full tools, procedures

14. Structure and content
1. Caregiver questionnaire (including questions on
household)
2. Child questionnaire (ages 0-9 years),
administered to caregiver
3. Child questionnaire (ages 10-17), administered
to child with parental consent & child assent

17. Sections Core questions Optional modules
Section 1: Household
schedule
• Household schedule* (10)
• Changes in household
composition (4)
Section 2: Background
Information on Caregiver
and Household
• Demographic information* (7)
• Work* (3)
• Access to money (3)
• Shelter (1)
• Household Economic Status
(forthcoming)
• Progress out of Poverty
Index or similar (country
specific)
Section 3: Food Security • Household food security (6) • Dietary Diversity (1)
Section 4: Caregiver Well-
being and Attitudes
• General health (2)
• Caregiver support (4)
• Parental self-efficacy (1)
• Perceptions and experience
of child discipline, violent
discipline (forthcoming)
• Gender roles and
decisionmaking power* (9)
Section 5: HIV/AIDS Testing,
Knowledge, Attitudes
• Basic HIV/AIDS knowledge* (7)
• HIV testing* (3)
• Attitudes to condom educ (1)
• HIV/AIDS attitudes* (4)
Section 6: Access to HIV
Prevention, Care & Support
• Household access to services
(1)
*DHS, bold=core indicator
Caregiver questionnaire

18. Sections Core questions Optional modules
Section 1: Child Health and
Welfare
• Confirm demographics (5)
• General health & disability (4)
• Birth certificate (2)
• Vaccinations (11)
• Fever (<5 years)* (1)
• Diarrhea (<5 years)* (1)
• Experience of neglect (2)
• Slept under mosquito net* (1)
• HIV testing experience* (2)
• Fever: extended* (4)
• Diarrhea: extended* (3)
• Health for children
living with HIV/AIDS
(forthcoming)
Section 2: Education and
Work
• School attendance*,
progression/repeats, drop-outs,
missed school days (5+ years) (9)
• Work for wages (2)
• Early childhood stimulation (2)
Section 3: Food
Consumption
• Food consumption (2+ years) (8) • Dietary diversity (1)
Section 4: Access to HIV
Prevention, Care & Support
• Child access to services (1)
Section 5: Anthropometric
Measures (of Children)
• Weight*, Height*, MUAC
Child questionnaire (ages 0-9)
*DHS, bold=core indicator

19. Child questionnaire (ages 10-17)
Sections Core questions Optional modules
Section 1: Background
Information on Child
• Confirm demographics* (5)
• Identity of caregiver (1)
Section 2: Diary • Daily log (6)
Section 3: Education • School attendance*,
progression/repeats, drop-
outs (9)
Section 4: Chores & Work • Chores (3)
• Work (7)
Section 5: Food & Alcohol
Consumption
• Food consumption (8)
• Alcohol consumption (3)
• Dietary diversity (1)
Section 6: Health, Support &
Protection
• Birth certificate (2)
• General health & disability (3)
• General support (4)
• Health for children living with
HIV/AIDS (forthcoming)
• Perceptions/experience of
violence (forthcoming)
Section 7: HIV Testing,
Knowledge, and Attitudes
• Basic HIV/AIDS knowledge* (7)
• HIV testing * (3)
• Child development knowledge (6)
• HIV/AIDS attitudes and beliefs (4)
Section 8: Sexual Experience • Sexual behavior (13-17 yrs) (5)
Section 9: Access to HIV
Prevention, Care & Support
• Child access to services (1)
Section 10: Anthropometric
Measures: Weight and Height
• Weight, Height, MUAC

20. When are these the right tools?
Tools are useful if your question is:
1.Is my program having, or did my program have an impact on the
children and households it reached?
2.What are the characteristics of children and their caregivers in
my country, state/province or district/area, in terms of education,
health, protection, and psychosocial support?
3.Where do the children most in need of program support live?
4.Approximately how many children need services or support?
5.What are the needs of my program’s registered beneficiaries, in
terms of education, health, protection, and psychosocial support?

21. These are not the right tools for you if…
You want to know:
Which children in selected communities to target
How a particular child/household is faring
Which households, children or caregivers are worst off
What services to provide or refer for a particular child /
household
The number of children/households that are receiving
program support, and the types of support received
Whether staff are carrying out their responsibilities
Whether interventions are being implemented as planned

22. And, why a special OVC survey?
 DHS and MICS take a general population sample
 difficult to discern the program’s contribution
 DHS and MICS include some, but not all of the OVC
core indicators

23. Using the Data
 Representative sample of program beneficiaries
 Data collected at one point in time
 Data collected at two points in time
 Representative sample of the general population

24. Beneficiary sample: 1 point in time
 Often called: Baseline, Midline or Endline
 If baseline or midline: Use data for program
planning or design, or mid-course corrections
 Example: high food insecurity found at baseline
 Result: Change in workplan, PMP agreed between
partner and USG, emphasizing food security
 If endline: Use data to inform follow-on activities

25. Beneficiary sample: 2 (or more)
points in time
 Commonly referred to as an “evaluation”
 Baseline data should be used immediately
 Evaluation results inform future programming, policy
 But, change in wellbeing from time 1 to time 2, does
not mean program is 100% responsible
 Much stronger result if:
 Comparison group is added (counterfactual)
 Panel study / cohort

26. General population sample
 Commonly called a Situation Analysis
 Use data for needs-based resource allocation at
national or sub-national level (not individual or
community level)
 Example:
 Nigeria OVC Situation Analysis

27. Triangulate
 Analyze data alongside DHS and MICS data
 Check for differences and similarities
 Determine whether differences or similarities
make sense, or point to a data quality issue

28. Implementation so far
 Zambia: Impact evaluation of savings and
internal lending communities on OVC wellbeing
 Baseline data available early Fall
 Nigeria: Baseline survey of OVC umbrella grant
mechanism beneficiaries in 10 states (planning
phase)

29. You said a toolkit?
 Tools & Manual
 Data analysis guide
 Template protocol with
consent/assent forms
 Data collector training
materials
 And then what?
 Revising as we learn
 Supporting countries to
implement

30. Where can I find out more?
Go to our website:
http://www.cpc.unc.edu/measure/our-work/ovc
Keep in touch on Child Status Net:
http://childstatus.net/
Email:
Jenifer Chapman: jchapman@futuresgroup.com
and Janet Shriberg: jshriberg@usaid.gov

31. The research presented here has been supported by the
President’s Emergency Plan for AIDS Relief (PEPFAR)
through the United States Agency for International
Development (USAID) under the terms of MEASURE
Evaluation cooperative agreement GHA-A-00-08-00003-
00. Views expressed are not necessarily those of
PEPFAR, USAID or the United States government.
MEASURE Evaluation is implemented by the Carolina
Population Center at the University of North Carolina at
Chapel Hill in partnership with Futures Group, ICF
International, John Snow, Inc., Management Sciences for
Health, and Tulane University.

32. Questions?