Presented at the International AIDS Conference in July 2014.

1. Using Cognitive Interviews in Nigeria to Pre-test Child, Caregiver, and
Household Well-Being Survey Tools for Orphan and Vulnerable Children Programs
LISA PARKER Futures Group/MEASURE Evaluation, USA // JENIFER CHAPMAN Futures Group/MEASURE Evaluation, USA // STANLEY AMADIEGWU Futures Group/MEASURE Evaluation, Nigeria // SHEHU SALIHU Futures Group/MEASURE Evaluation, Nigeria
BACKGROUND
MEASURE Evaluation recently developed standardized data collection tools to measure
changes in child, caregiver, and household well-being. These data collection tools are included
in a tool kit (Box 1) and may be applied in the context of an evaluation, situation analysis, or
other research to produce actionable data and enable comparative assessments of outcomes
across interventions and regions. To pre-test these data collection tools, we conducted
cognitive interviews (CIs), a qualitative research technique used to help design questionnaires
by determining whether respondents understand the questions and response categories (de
Leeuw, Borgers & Smits, 2004), with potential respondents in Nasarawa State, Nigeria.
Box 1—MEASURE Evaluation OVC Survey Toolkit
The MEASURE Evaluation OVC survey toolkit includes three questionnaires that measure:
1. household outcomes and caregiver wellbeing (administered to a caregiver);
2. wellbeing among children aged 0–9 years (administered to a caregiver); and
3. wellbeing among children aged 10–17 years (administered to a child with
guardian consent and child assent).
METHODS
We conducted CIs with caregivers (n=12), children ages 10–12 years (n=8), and children ages
13–17 years (n=8) in one rural and one urban site (Table 1). All participating households
were registered to receive services from the USAID-funded PACT Rapid & Effective Action
Combating HIV/AIDS program. This study was approved by Health Media Labs, Inc., in the
United States and the National Health Research Ethics Committee in Abuja
Table 1—Cognitive Interview Sample
Location
Adults Children aged 10–12 Children aged 13–17
Male Female Male Female Male Female
Rural 2 4 2 2 2 2
Urban 0 6 2 2 3 1
Trained interviewers obtained informed consent and posed between 16–26 questions to each
participant and after each question recorded the participant’s response. The interviewer then
assessed the participant’s understanding of the question by asking four questions and related
probes (see Box 2).
Box 2
Question 1: What do you think this question is asking you? Probes: Can you repeat the
question I just asked in your own words? What does the phrase/word [use a phrase/word
in the question] mean to you? What do you think we meant by [use a phrase/word in the
question]?
Question 2: Can you tell me what you were thinking about when you gave this response?
Probes: How did you arrive at that answer? How did you come up with/remember that
response?
Question 3: Thinking about people like you who might be asked this question, what is
unclear about this question? Probes: Was the question easy or hard to answer?
Question 4: Did you hope you could give a different kind of response than the choices you
were given? Probes: I noticed that it took some time for you to think of how you wanted to
respond to this question. Was it because the choices you had for responding were not clear
to you or for some other reason?
At the end of the interviews, interviewers completed a Summary Form, recording observations
and patterns across all interviews. Four questions were included on the Summary Form (Box 3).
Box 3
Question 1: Were there certain types of questions or concepts in these sections that most
respondents had difficulty understanding?
Question 2: Were there certain types of response choices in these sections that most
respondents had difficulty understanding?
Question 3: How can we improve these sections of the questionnaire so that respondents
will be better able to understand the questions and give appropriate responses?
Question 4: Were there certain types (male, urban, etc.) of respondents that had the most
difficulty with this question?
RESULTS
The cognitive interviews were used to:
1. validate the translation,
2. refine the response categories,
3. better understand respondents’ understanding of recall periods, and
4. assess respondents’ understanding of certain concepts.
1. Validating the translation
The cognitive interviews were very useful in validating the translations of specific terms. For
example, the phrases “being too sick to participate in daily activities,” and “leaving a child alone
with no adult present for more than one hour” were adapted to more accurately reflect the
intent of the question.
2. Refining the response categories
We were able to use the cognitive interviews to add response categories if a certain number of
respondents’ answers included categories not available; however, participants struggled with
making their own suggestions as to whether response categories should be different. Likewise,
the cognitive interviews helped us to learn that rural respondents had difficulty with four-level
agree/disagree Likert scale response categories. They were unable to distinguish between
strongly agree and agree and between strongly disagree and disagree. Therefore, we adapted
specific response categories to include only 2-level agree/disagree options.
3. Better understanding respondents’ understanding of recall periods
The cognitive interviews were an effective means of determining the best recall periods to
include in the questionnaires. We tested recall periods for the food security questions of 1
month and 4 weeks. Respondents better understood “the past 4 weeks” versus “the last one
month.” The last one month was interpreted to be the previous month, not necessarily the last
four weeks prior to the date of the survey administration, which was the intent of the question.
Likewise, respondents had difficulty understanding “the last year,” not knowing if it referred to
the previous 12 months or the last calendar year.
4. Assessing respondents’ understanding of certain concepts
The cognitive interviews allowed us to better grasp the respondents’ interpretations of
specific concepts. In general, the adults were able to understand the intended meaning of
the questions posed. However, in a few cases, adults misinterpreted the intention of specific
words. For example, a few participants misinterpreted the question asking if their child was
enrolled in school to mean that their child was in school at the exact time of the interview.
Likewise, a few respondents had difficulty with a question from the Rand Medical Outcomes
Study social support scale (RAND) “Someone whose advice you really want”. They reported not
understanding what it meant to really want advice.
A number of questions were difficult for the children aged 10–17 to understand. Many
children 13–17 understood the question, “Has anyone has ever talked to you or taught you
about how children grow and develop?” as pertaining solely to hygiene and cleanliness. Many
children 13–17 also struggled with the question, “Has someone ever talked to you or taught
you about sex or sexual behavior?” likely due to their lack of exposure to sexual education or
embarrassment related to this topic. Some children 10–17 had trouble understanding the
intent of the social support question, “Do you have someone in your life that can take you to
the doctor if you needed it?” (RAND). Some interpreted the question to be asking if they had
access to health care or if their parents would take them to the doctor or the pharmacy if they
were sick, rather than assessing if they had someone to provide instrumental social support.
CONCLUSIONS
We effectively used the data generated from the Cognitive Interviews to adapt the final
questionnaires and aid in data interpretation. However, we faced a few challenges with the
methodology. Some respondents grew frustrated with the repetitive nature of probing while
many misunderstood the intention of asking what they were thinking and responded with
unrelated personal thoughts. In addition, few respondents were able to provide nuanced
responses concerning what others might find unclear. Previous use of CIs in sub-Saharan
Africa is limited and there is little published research on their application in less-developed
settings. Our results indicate that cognitive interviewing is a challenging, yet potentially useful,
methodology for this context and that adaptations to the methodology should be considered,
including selecting a very limited set of questions and providing specific probes for individual
questions rather than using a general set of probes.
REFERENCES
de Leeuw E D, Borgers N, & Smits A (2004). Pretesting questionnaires for children and
adolescents. In S. Presser et al., (Eds.), Methods for testing and evaluating survey
questionnaires. Hoboken, NJ: Wiley & Sons.
RAND Corporation. Medical Outcomes Study Social Support Survey. Arlington, VA: RAND
Corporation. Available at:
http://www.rand.org/health/surveys_tools/mos/mos_socialsupport.html
ACKNOWLEDGEMENTS
We would like to acknowledge and thank our colleagues from the Centre for Research
Evaluation, Resources & Development, PACT Nigeria, the Catholic Dioceses of Lafia, and
all of the adult and child participants. This research has been supported by the President’s
Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International
Development (USAID) under the terms of MEASURE Evaluation cooperative agreement
GHA-A-00-08-00003-00. Views expressed are not necessarily those of PEPFAR, USAID, or the
United States government.
CONTACT:
Dr. Lisa Parker, Monitoring and Evaluation Technical Advisor, MEASURE Evaluation
Email: lparker@futuresgroup.com
Website: measureevaluation.org
Evaluation