Presented at the International AIDS Conference in July 2014.

1. Evaluating Orphan and Vulnerable Children Outcomes
Innovative Methodology for, and Results of, Pilot Testing a New Toolkit
JENIFER CHAPMAN Futures Group/MEASURE Evaluation, USA // LISA PARKER Futures Group/MEASURE Evaluation, USA // STANLEY AMADIEGWU Futures Group/MEASURE Evaluation, Nigeria // SHEHU SALIHU Futures Group/MEASURE Evaluation, Nigeria // MWILA KANEMA Futures Group, Zambia
BACKGROUND
Despite high donor investment, the impact of orphan and vulnerable children programs is
unclear. Part of the challenge was a lack of standardized and tested measures and tools for
evaluating orphan and vulnerable children (OVC) program outcomes. To fill this gap, MEASURE
Evaluation produced three questionnaires to enable the collection of actionable data for
programs and comparative assessments of outcomes across interventions and regions. The
questionnaires were developed with strong stakeholder input, and aligned to the 2012 PEPFAR
OVC Programming Guidance and the U.S. Government Children in Adversity Action Plan (see
Box 1).
Box 1—MEASURE Evaluation OVC Survey Toolkit
The MEASURE Evaluation OVC survey toolkit includes three questionnaires that measure
the following:
1. Household outcomes and caregiver wellbeing (administered to a caregiver).
2. Wellbeing among children aged 0–9 years (administered to a caregiver).
3. Wellbeing among children aged 10–17 years (administered to a child with
guardian consent and child assent).
The questionnaires were designed to measure changes in child, caregiver and household
well-being that can reasonably be attributed to PEPFAR-funded program interventions.
The questionnaires may be applied in evaluation, situation analysis, or in other research.
We pilot tested the questionnaires in 2013 in Zambia and Nigeria. In Zambia, we piloted tested
the questionnaires under the USAID-funded impact evaluation of savings and internal lending
communities (SILC) on child well-being, led by Futures Group with World Vision and Catholic
Relief Services. In Nigeria, the pilot test was conducted in partnership with the PACT Rapid and
Effective Action Combatting HIV/AIDS (REACH) program and the Catholic Dioceses of Lafia in
Akwanga, Nasawara State.
OBJECTIVES
• Test the construct validity of some questions and concepts.
• Pre-test the reliability of scales.
• Determine whether any questions may be duplicative and, in some instances, to enable a
choice between different question versions.
• Test the clarity of the question sets as an entirety.
• Assess the reliability of recall periods and, in some cases, assess different recall periods; and
test field application of the tools.
METHODS
We applied a three-step methodology:
1. Validation of the translation of the questionnaire with data collectors in a training setting.
2. Cognitive interviews with potential respondents.
3. Pilot-testing the full questionnaires at the household level. These studies were approved
by Health Media Labs, Inc., in the United States, the Biomedical Research IRB in Lusaka, and
the National Health Research Ethics Committee in Abuja.
1. Validation of Translation
During the data collector training for each pilot test, the trainer led a discussion on each
questionnaire. The purpose was to orient data collectors to the questionnaire, enable them to
seek clarification on measures, and validate the translation. Changes were made by the original
questionnaire translator.
2. Cognitive Interview
Cognitive interviewing is a qualitative research technique used to help design questionnaires
by determining whether respondents understand the questions and are able to produce
expected responses (de Leeuw, Borgers & Smits, 2004). In Zambia, we conducted cognitive
interviews with 12 adults and 16 children purposively sampled from the program beneficiary
list in one ward. In Nigeria, we selected a purposive sample of 12 caregivers and 16 children
program beneficiary lists. After gaining participant consent, trained data collectors read
each question and recorded the participant’s response. The data collector then probed for
participant’s understanding of the question.
3. Household Survey Pilot Test of Complete Tools
We pilot tested the complete data collection tools and a few additional measures through
a household survey pre-test to assess respondent’s understanding, question flow, and to
determine the time needed to complete a full questionnaire, including recruitment. We pilot
tested the data collection process from start to finish, including informed consent, application
of the Kish Grid (Kish, 1949) for sampling at the household level, and administration of all
survey tools among 21 households in Zambia and 20 households in Nigeria. Households were
purposively sampled from program beneficiary lists. In Nigeria, data collectors returned to 10
of the 20 households the day after the survey to conduct a reliability check of 16 key measures.
After gaining participant consent, trained data collectors administered the questionnaires.
RESULTS
Overall, items were well understood and the process of data collection was efficient.
Data collectors were successful in conducting informed consent, using the Kish Grid, and
administering the survey tools with both adults and children.
Based on results and lessons learned from piloting, we made changes to a few questions. For
example, for the questions on access to money we changed the formulation of the question
to focus on a respondent’s specific experience accessing money to improve data quality. We
adapted questions requesting whether household expenditures had increased or decreased
by splitting the original questions and including additional skip patterns. We changed three
of four questions on social support based on a reliability analysis conducted on the full scale
from the Rand Medical Outcomes study. Finally we changed various recall periods to improve
participant understanding and response categories to add frequent responses not originally
included.
Of note, Likert scales were not well understand and were replaced in the final tools with binary
response options.
We pilot tested a variety of measures and questions that were omitted from the final tools
because either respondents did not understand the questions or concepts in the questions
well or they demonstrated poor variability, including: an extended asset schedule, two social
capital questions, the Hope Scale (Snyder et al, 1997) and two additional hope questions, two
self-esteem questions from the Rosenburg Self-Esteem Scale (Rosenburg, 1965), the General
Self-Efficacy scale (Schwrzer & Jerusalem, 1995), a stand-alone self-efficacy question, three
parental self-efficacy questions from the Parental Stress Scale (Berry & Jones, 1995), and the
strengths and difficulties questionnaire for children (Goodman, 1997).
There were very few questionnaire revisions after the Nigeria pilot, suggesting that the
majority of the problematic questions had been corrected prior to this second round of
piloting.
Pre-testing raised concern about how to best address child-headed households, as the
caregiver questionnaire is tailored to an adult respondent, and both child questionnaires
assume a previous interview with the adult caregiver.
CONCLUSIONS
Pilot testing of new tools is critical, yet rarely done. Results from these pilot tests informed
revision of the questionnaires, which are now ready for public use. We highly recommend this
three-pronged approach to pilot testing, with some suggested adaptations to the Cognitive
Interviews (Parker et al., 2014), to colleagues seeking to validate their data collection tools.
REFERENCES
Berry JO, & Jones WH (1995). The Parental Stress Scale: Initial psychometric evidence. Journal of
Social and Personal Relationships, 12, 463–472.
de Leeuw ED, Borgers N, & Smits A. (2004). Pretesting questionnaires for children and
adolescents. In S Presser, et al., (Eds.), Methods for testing and evaluating survey
questionnaires. Hoboken, NJ: Wiley & Sons.
Goodman R (1997) The Strengths and Difficulties Questionnaire: A Research Note. Journal of
Child Psychology and Psychiatry, 38, 581–586.
Kish L (1949). A procedure for objective respondent selection within the household. Journal of
the American Statistical Association 44(247): 380–87.
Parker L, Chapman J, Amadiegw S, Salihu S (2014). Using cognitive interviews in Nigeria to pre-test
child, caregiver, and household well-being survey tools for orphan and vulnerable children
programs. AIDS 2014, Melbourne, Australia, July 20–25, 2012.
RAND. Medical Outcomes Study Social Support Survey. Arlington, VA: RAND Corporation.
Available at: http://www.rand.org/health/surveys_tools/mos/mos_mentalhealth.html
(last accessed April 2013).
Schwarzer R, & Jerusalem M (1995). Generalized Self-Efficacy scale. In J. Weinman, S. Wright, &
M. Johnston, Measures in health psychology: A user’s portfolio. Causal and control beliefs
(pp. 35–37). Windsor, England: NFER-NELSON.
Snyder CR, et al., (1997). The Development and Validation of the Children’s Hope Scale. J Pediatr
Psychol. 22(3):399–421.
Rosenberg M. (1965). Society and the Adolescent Self-image. Princeton, NJ: Princeton University
Press.
ACKNOWLEDGEMENTS
This research has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR)
through the United States Agency for International Development (USAID) under the terms of
MEASURE Evaluation cooperative agreement GHA-A-00-08-00003-00 which is implemented by
the Carolina Population Center, University of North Carolina at Chapel Hill with Futures Group,
ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.
Views expressed are not necessarily those of PEPFAR, USAID, or the United States government.
CONTACT
Jenifer Chapman, Senior OVC Advisor, MEASURE Evaluation
Email: jchapman@futuresgroup.com
Website: measureevaluation.org
Evaluation