Presented at the the 2014 American Evaluation Association Conference.

1. Evaluation of gender aware health
interventions in South Asia:
What do we know and what do we need to
know?
American Evaluation Association Conference
Denver, CO
October 17, 2014
Arundati Muralidharan, DrPH, PHFI Kaveri Mayra, PHF; Lara Lorenzetti, MEval-III;
Jessica Fehringer, PhD, MHS, MEval-III; Carolina Mejia, PhD, MPH, MEval-III;
Mahua Mandal, MEval-III; Emily Mangone MS; Lakshmi Gopalakrishnan
Elisabeth Rottach, MA, HPP; Sara Pappa, MA, HPP

2. PhotobyArundatiMuraldharan
 Introduction
 Methods
 Findings
 Discussion Questions
Outline

3.  Rationale
 Guided by belief that health programs must employ evidence-
based strategies that promote gender equity and empower
women to achieve and enjoy better health
 So What? (2000-2004)
 Gender Perspectives (2004-2008)
 Transforming Gender Norms, Roles, and Power Dynamics for Better Health
(2000-2013)
 Reviews found that evaluations of gender aware programs have been
inconsistent and highly variable in terms of approach, measures, and
level of detail published
 Structure
 Phase I - Focused on South Asia
 Phase 2 – Global evidence
Introduction

4. 1. Review the methodological approaches used to
evaluate the effectiveness of gender aware health
interventions in South Asia
2. Provide recommendations for improving future
evaluations
Objectives

5. Source: Interagency Gender Working Group (IGWG). 2013. Adapted from a framework drawing on a range of efforts that have used a
continuum of approaches to understanding gender, especially as they relate to HIV/AIDS. See Geeta Rao Gupta, “Gender, Sexuality and
HIV/AIDS: The What, The Why and The How” (Plenary Address at the XIII International AIDS Conference), Durban, South Africa: 2000;
Geeta Rao Gupta, Daniel Whelan, and Keera Allendorf, “Integrating Gender into HIV/AIDS Programs: Review Paper for Expert
Consultation, 3–5 June 2002,” Geneva: World Health Organization 2002

6. Methods

7. Methodology
Step 1:
Establishing Evidence Review Committee (ERC)
and search for publications
Step 2:
Establishing relevancy
Step 3:
Data abstraction & effectiveness rating
Step 4:
Synthesis and analysis
Step 5:
Report writing and dissemination

8. Step 1: Search: 948 articles
 S. Asia, interventions, target outcomes, gender-integrated
 Date search dep. on topic: CH <5, son preference, TB, and UHC, >
1/1/2000; others >1/1/2009
 Plus references from 2 key previous gender & health reports
Step 2: Establishing relevancy:
1) Intervention is gender-aware, per IGWG definitions
 Avahan: community mobilization, collective identity, or community-led
structural interventions
2) Outcomes: RNMCH+A, HIV/STIs, nutrition, GBV, TB, UHC
 84 relevant articles – 57 non-Avahan and 24 Avahan
Methods

9. Step 3: Data abstraction plus ratings
 Full text of relevant articles read, abstracted for key
components
 Rated on level of gender integration (accommodating vs.
transformative)
Step 4: Synthesis and analysis
 Thematic, each data abstraction field constituted a theme
 Tables to identify patterns; e.g., differences in types of health
outcomes achieved by accommodating vs. transformative
interventions
Steps 5 & 6: Report writing & external review
Methods

11. Preliminary Findings

12. Theory of Change
Why is this important? Preliminary Results
• The theory of change presents
the theoretical or observed
linkage between two concepts,
provides a systematic way of
understanding events or
relationships, and often serves
as a platform for designing
strategies that promote change
• 19.3% of all articles discussed
a theory of change
• Examples of theories:
• Social cognitive theory
• Grounded theory
• Health belief model
• Integrative model of
behavioral prediction

13. Study Design and Methods
0
5
10
15
20
25
30
35
40
Study Designs
Transformative
Accommodating
13
13
31
Mixed Methods: All Evaluations
Quasi-experimental
MM
Non-experimental MM
No MM

14. Sampling Methods
Why is this important? Preliminary Results
• Sampling methods have
important implications for:
• Bias caused from
differences in group
characteristics
• Ability to detect meaningful
differences between
groups
• Interpretation of results
• Power assumptions help
determine an appropriate
sample size for hypothesis
testing
• Nearly all evaluations recorded
sample sizes at baseline and
endline
• Sampling methods were
described in 38% of articles.
Examples include:
• Two-stage cluster
sampling
• Purposive sampling
• Just 20% of articles provided
information on power
assumptions

15. Control Groups
Why is this important? Preliminary Results
• Group differences (intervention
vs. control) can lead to biased
estimates of program impact
• Ideally, program participants
are identical to non-participants
except for participation in
intervention; however, this is
rarely the case
• Some study designs can
control for these differences
while others cannot
• Half of evaluations included a
control group, but information
on how group was selected and
baseline group characteristics
varied considerably across
studies.
• Of studies that did not include a
control group, 4 noted limitation
of not having control group
• i.e. difficult to attribute
change solely to program
• 2 articles discussed
implications of spillover

16. Measurement of Gender
Why is this important? Preliminary Results
• Experts recommend
implementing gender-integrated
interventions to improve health
outcomes; however, evidence
base of program effectiveness is
lacking.
• Challenges in gender research
include defining complex
constructs (i.e. gender,
empowerment) and creating
validated measures
• Accommodating interventions
tended to look at increased
support from partners and
community
• Transformative included and
went beyond these outcomes,
focusing on changing gender
equitable attitudes, beliefs, and
behaviors
• Half of articles discussed specific
gender measures used
• Gender Equitable Men scale
most common
• Individual indicators in
household survey

17. Analysis Plan
Why is this important? Preliminary Results
• Important to provide details on
how the analysis was conducted
so that other researchers can
replicate and verify results if need
be
• Depth of information on analysis
plans was somewhat lacking
across both categories
• Wide variety of quantitative
methods used:
• Basic cross tabs
• T-tests to compare means
between groups
• OLS or logistic regression
• Difference-in-differences
• Structural equation modeling
• Qualitative methods were less
often described but did include
content analysis.

18. Level of Impact
Why is this important? Preliminary Results
• Individual and community-level
gender and cultural norms
affect the overall health and
well-being of women
• Analyzing outcomes at the
individual and community level
is important for understanding if
and how the intervention truly
affects change
• Most accommodating
interventions (85%) evaluated
outcomes at the individual level
• By comparison, evaluations of
transformative interventions
tended to explore both:
• Individual: 46%
• Community: 32.4%
• Both: 21.6%
• Qualitative methods can be
helpful in capturing outcomes at
both levels

19. Multiple Endlines
Why is this important? Preliminary Results
• Most evaluations assess
change in outcomes between
baseline and endline
• Multiple endlines can
strengthen the evaluation
design by describing impacts
over time
• Multiple endlines might be used
to assess program
sustainability, factors
associated with scale-up, or
identify outcomes with delayed
effects
• 3 accommodating interventions
conducted post-intervention
evaluations using a second
endline. Example:
• Post-period analysis of community-
based interventions on maternal
indicators in Balochistan, Pakistan
showed that women in original
intervention groups continued to
have better maternal outcomes
• Several transformative
interventions also included
multiple endlines

20. Cost Effectiveness
Why is this important? Preliminary Results
• Even if we see a positive effect,
does the program justify the
costs?
• Cost-effectiveness compares
the cost of implementing an
intervention against a gain in a
specific health outcome
• Important implications for
program sustainability
• Providing basic cost data gives
an idea of how much it would
cost to replicate or take
intervention to scale
• No evaluations conducted cost-
effectiveness analyses
• However, 7 articles (12.3%)
provided basic cost information.
For example:
• Swaasthya intervention
conducted analysis to
estimate cost for
replicating intervention
• Lady Health Worker
intervention in Pakistan
showed that cost/LHW had
increased over time and
concluded program was
not underfunded

21. Discussion Questions

22.  Do these findings match your experience with gender-related
evaluations?
 What are measurements of gender that you often use? What
methods or study designs do you most often use?
 We acknowledge that gender is a difficult construct to
measure. Why is it so challenging to measure? How can we
address these challenges in future work?
 Are there other aspects of gender evaluations that you don’t
see here but would be important to include?
 How would you deal with the inclusion of grey literature in this
type of review?
 For which audience is this report best suited?
Discussion Questions

23.  Scant literature in South Asia
 Depth of information provided and quality varied,
challenging to draw broad conclusions
 Only a few interventions were evaluated using
randomized control trials
 RCTs often inappropriate for these topics/interventions
 Limited #s of interventions, so cannot quantify strength of
evidence
 English-language only documents
Limitations
PhotobyArundatiMuraldharan

24.  Finalize analysis and drafting of manuscript
 Include Avahan
 Determine potential journals for submission
 Incorporating evidence from the global review
 Dissemination
 Roundtable discussions with Mission and stakeholders
Next Steps

25. Photo by Arundati Muraldharan
For more information, contact:
 Arundati Muralidharan
(arundati.muralidharan@phfi.org) and Kaveri Mayra
(kaveri.mayra@phfi.org) of the Public Health
Foundation of India (PHFI),
 Jessica Fehringer (jessica_f@unc.edu), Lara
Lorenzetti (llorenz@unc.edu), Carolina Mejia
(cmejia@.unc.edu), Emily Mangone
(mangone@live.unc.edu), Lakshmi Gopalakrishnan
(gopalakr@live.unc.edu), and Mahua Mandal
(mmandal@email.unc.edu) of MEASURE
Evaluation
 Elisabeth Rottach (erottach@futuresgroup.com) and
Sara Pappa (SPappa@futuresgroup.com) of the
Health Policy Project.
Thank you!

26. www.measureevaluation.org