A presentation by Rachel Steinacher, Research Manager for IPA-Kenya (Innovations for Poverty Action), on IPA and RCTsThis was presented at the Commission for Science and Technology (COSTECH) in Dar es Salaam, Tanzania, on June 19, 2014, to an audience of researchers.

1. Innovations for Poverty Action –
Generating and Using Evidence

2. Outline
 Introduction to Innovations for Poverty Action
 What we do and why
 Different Types of Evidence
 Process evaluation
 Qualitative data
 Experimental evidence
 Why research impact
 Determining and defining impact
 What can we use the results of RCTs for?
 Example: Primary School Deworming
 Conclusion
 Opportunities at Innovations for Poverty Action

3. Innovations for Poverty
Action
 Founded in 2002, we partner with academics
to design and evaluate potential solutions to
global poverty problems using randomized
evaluations.
 We also work to mobilize and support these
decisionmakers to use this evidence / these
solutions to build better programs and policies
at scale.
 In collaboration with over 250 leading
academics and implementing organizations,
we have evidence from over 175 completed
studies with over 225 in progress around the

4. Innovations for Poverty
Action
 Our Vision
 More Evidence, Less
Poverty
 Our Mission
 To discover and
promote effective
solutions to poverty
problems around the
world

5. Different Types of
Evidence
 We’ve been referring to ‘evidence’ but what we
mean might not be clear as there are many
different types of evidence
 Types of evidence include (but are not limited
to!)….
 Process Evaluation Data (Monitoring and
Evaluation)
 Qualitative Data
 Experimental Evidence

6. Process Evaluation
 Process evaluation analyzes the extent to which program
operations, implementation, and service delivery are being
implemented properly.
 When process evaluation is ongoing it is called program
monitoring (as in Monitoring and Evaluation: M&E).
 Process evaluations help us determine, for example:
 Whether services and goals are properly aligned
 Whether services are delivered as intended to the appropriate recipients
 How well service delivery is organized
 The effectiveness of program management
 How efficiently program resources are used
 Process evaluations are often used by managers as benchmarks to
measure success, for example: the distribution of chlorine tablets is
reaching 80% of the intended beneficiaries each week.
 These benchmarks may be set by program managers and
sometimes by donors.

7. Qualitative Data
 Qualitative data is data that is captured in a non-numerical
way.
 Collection can happen in many different ways, including:
 In-depth interviews
 One-on-one interviews with a participant with the purpose of probing
the thoughts, ideas, knowledge, etc. of the person being interviewed.
These will often be recorded and transcribed for analysis later.
 Focus groups
 Same as an in-depth interview, but includes multiple participants
 Direct observation (ethnography, etc.)
 This differs from interviewing as the person collecting the data does
not actively try to ask questions to the participant about the topic
being studied. They simply watch and observe, with the goal of
recording and interpreting what they say, their behaviour and
attitudes. Sometimes this involves recording (video and/or audio) or
writing notes about what they observe for analysis later.
 Etc…

8. Qualitative Data
 Qualitative data is useful to answer questions like:
 Does this intervention ‘make sense’ in this context?
 Is this intervention socially, politically, religiously
acceptable?
 How do people perceive / understand this intervention
and the problem(s) it is meant to solve?
 Hypothesis generation: what should we be studying /
collecting data on to better understand the intervention or
topic we’re interested in studying? Are we asking the
right questions?

9. Experimental Evidence –
Why Research Impact?
 Experimental evidence allows us to determine
impact. By impact, we mean changes / outcomes
that have happened because the intervention
happened.
 This let’s us know what interventions…..
 Are effective [How effective is it? In what context?
For what demographic(s)? At what cost?]
 Have no impact (Why wasn’t it effective? Was the
program implemented properly?)
 Are harmful (Even something that seems like
common sense can have devastating consequences;
see the ‘Campbell-Somerville Study’)

10. Why Research Impact –
Influencing Programs and
Policy
 We can use this information to try to influence
the decisions being made by governments,
NGOs, and other policymakers / program
implementers to make evidence-based
decisions

11. Determining Impact
 Different research methods yield
different types of evidence….
 Some of this evidence
demonstrates the impact of a
program
 Some evidence cannot be used to
demonstrate impact. in fact, it can
lead us to believing things that
aren’t true, even though the data
seems to support a certain
conclusion….
 Correlation does not imply
causation

12. Evidence that Pirates Prevent Global
Warming

13. Determining Impact
 Impact is captured by analyzing a comparison between:
1. the outcome some time after the program has been
introduced
2. the outcome at that same point in time had the
program not been introduced (the ”counterfactual”)

14. Impact: What is it?
Time
PrimaryOutcome
Impact
Intervention

15. Impact: What is it?
Time
PrimaryOutcome
Impact
Intervention

16. Constructing the
Counterfactual
 Counterfactual is often constructed by
selecting a group not affected by the
program
 Randomized:
 Use random assignment of the program to
create a control group which mimics the
counterfactual.
 Non-randomized:
 Argue that a certain excluded group mimics the
counterfactual.

17. Counterfactual
 The counterfactual represents the state of the
world that program participants would have
experienced in the absence of the program
(i.e. had they not participated in the program)
 Problem: Counterfactual cannot be observed
 Solution: We need to “mimic” or construct the
counterfactual

18. Non-Random Assignment
HQMonthly income, per
capita
1000
500
0
Treatment Control
1457
947

19. Demonstrating Impact through
RCTs
 RCTs are by far the most powerful research
methodology by allowing us to determine the
impact of a program by minimizing the
likelihood that the treatment and control
groups are meaningfully different from each
other

20. Random Assignment
Monthly income, per
capita
1000
500
0
Treatment Control
1257 1242
HQ

21.  An example from IPA…
What can we use RCT results
for?

22. Primary School
Deworming
 One of Innovations for Poverty Action’s most
famous studies, that is currently be scaled out
across the Kenya

23. Primary School
Deworming
 Principal Investigators: Michael Kremer and
Edward Miguel
 Partners:
 Investing in Children and their Societies (ICS)
 Location: Western Kenya
 Sample: 30,000+ primary school children, 6-
18 years old
 Timeline: 1997-2001

24. Primary School
Deworming
 Policy Issue
 Intestinal worms infect more than one in four
people worldwide and are particularly prevalent
among school-aged children in developing
countries.
 These intestinal worms are believed to have a
negative impact on education, hindering child
development as well as school attendance and
reducing income later in life.

25. Primary School
Deworming
 Context of the Evaluation
 Busia district is a poor and densely-settled
farming region in western Kenya adjacent to Lake
Victoria. It has some of the country’s highest
worm infection rates.
 One quarter of Kenyan student absenteeism is
attributed to abdominal pains which likely due to
intestinal worm infections. In addition, older
children may miss school to take care of siblings
who are sick with worm infections

26. Primary School
Deworming
 Details of the Intervention
 This study evaluated the Primary School Deworming
Project (PSDP), which was carried out by
International Child Support in cooperation with the
Busia District Ministry of Health.
 The program randomly divided 75 schools into three
equal groups which were phased into treatment over
three years.
 Within each group, a baseline parasitological survey
was administered to a random sample of pupils.
Schools with worm prevalence over 50% were mass
treated with deworming drugs every six months.

27. Primary School
Deworming
 Details of the Intervention
 In addition to medicine, treatment schools
received regular public health lectures, wall charts
on worm prevention, and training for one
designated teacher.
 The lectures and teacher training provided information
on worm prevention behaviors—including washing
hands before meals, wearing shoes and not swimming
in fresh water.

28. Primary School
Deworming
 Results and Policy Lessons:
Impact on Infection Intensity
 Deworming reduced serious worm infections by
half amongst children in the treatment groups.
 Pupils that received treatment reported being sick
significantly less often, had lower rates of severe
anemia, and showed substantial height gains,
averaging 0.5 centimeters.

29. Primary School
Deworming
 Results and Policy Lessons:
Impact on School Attendance
 Deworming increased school participation by at
least 7 percentage points, which equates to a
one-quarter reduction in school absenteeism.
 When younger children were dewormed, they
attended school 15 more days per year, while
older children attended approximately 10 more
school days per year.

30. Primary School
Deworming
 Results and Policy Lessons:
Treatment Spillover
 The entire community and those living up to 6
kilometers away from treatment schools benefited
from “spillovers” of the deworming treatment.
 Spillover effects occur because medical treatment
reduces the transmission of infections to other
community members.
 Reductions in infection in non-treated children
resulted in an additional 3 to 4 days of schooling
per year.

31. Primary School
Deworming
 Results and Policy Lessons
 Including the spillover benefits of treatment, the
cost per additional year of school participation is
US$3.27, considerably less than the cost of many
alternative methods of increasing primary school
participation.
 Creation of the ‘National School-based
Deworming Program’ and ‘Deworm the World’

32. National School-Based
Deworming Programme
National School Health Programme
Ministry of Education
Science and
Technology
Ministry of Health

33. National School-Based
Deworming Programme
GOAL:
Treat 5 million children
each year for 5 years
TARGET:
Treat ALL at risk children
aged 2-14

34. Primary School
Deworming
 Scaling-up the Program
 Additionally, in 2007 the ‘Deworm the World
Initiative (DtW)’ was founded. It was managed
and operated through Innovations for Poverty
Action up until the end of 2013, when it our sister
organization, ‘Evidence Action’ spun off with the
goal of scaling evidence-based programs –
including DtW
 To date, DtW’s programs have reached 37 million
children in 27 countries
 Impact evaluation can and has had a profound
impact on the world

35. In Sum
 There are many types of evidence that one
can collect and use to better understand an
intervention or topic
 Experimental evidence generated by RCTs is
a powerful tool that we can use to promote
evidence-based development work by
influencing policymaking and program design.

36. Innovations for Poverty
Action Moving Forward
 Innovations for Poverty
Action is a rapidly growing
organisation with plans to
expand in Tanzania (we
currently have 1 project
here)
 We are always looking for:
 Talented, innovative
researchers to partner with
on new research studies
 For young researchers, this
typically means partnering
with a senior researcher to
allow for mentoring and
expert oversight

37. Innovations for Poverty
Action Moving Forward
 We are also looking for bright, eager people
interested in working in working on our research
projects as:
 Project Associates (entry-level managers with a
Bachelors degree + 0-2 years experience)
 Project Managers (early to mid-career professionals
with a Bachelors degree + 2-5 years experience)
 Research Managers (early to mid-career managers
with a Masters degree + 3-5 years of experience)
 For more information, visit:
 http://www.poverty-action.org/getinvolved/jobs