Presented at the 2016 AEA conference.

1. Evaluations of Structural
Interventions for HIV
Prevention
Mahua Mandal, MPH, PhD
Brittany S. Iskarpatyoti, MPH
Jill Lebov, PhD, MSPH
Jim Thomas, MPH, PhD
MEASURE Evaluation
University of North Carolina-Chapel Hill
October 27, 2016
American Evaluation Association Annual
Conference

2. Health processes and outcomes are embedded
in and influenced by structural factors: the
social, economic, legal-political, and built
environment
Background
What are structural factors?

3. Background

4. Structural interventions intervene on
distal factors
Background
What are structural interventions?

5. • Contextual factors
• Generalizability
• Complexity
• Non-linear relationships
• Random assignment
• Time horizon
Background
Challenges in evaluations of structural
interventions

6. Systematic review of outcome and impact
evaluations of structural interventions of HIV
prevention, focused on
• Economic strengthening
• Formal and informal education
• Substance abuse
Introduction to study
Aim of study

7. Methods

8. Examined characteristics, methods, and
rigor of evaluations in peer-reviewed
literature
Methods

9. • Theory of change
• Mixed methods (included in paper
or referenced)
• Random assignment
• Length of study period
• Use of cohort
• Comparison/control group
• Equivalence at baseline
• Pre/post data
• Follow-up rate
• Statistical significance testing
• Report of intervention details
Methods
Checklist – rigor of evaluations
Weak: 0-4
Moderate: 5-9
Strong: 10-14

10. Results
Total # evaluations = 27

11. Results
Study Design
0
2
4
6
8
10
12
RCT Quasi-Experimental Non-Experimental
NumberofPapers
Study Design by Level of Evaluation
Outcome
Outcome +
Impact

12. • 21 papers included a TOC (>75 percent)
• 27 included basic sampling information
• Length of intervention: 10 months – 10 yrs
(median=12 months)
• 7 did not measure intermediate variables
• 26 papers included study limitations
• 14 discussed replicability and generalizability
Results
Components of evaluations

13. • HIV/AIDS and sexual knowledge (n=11),
attitudes (9) and behaviors (24)
• Couples communication and condom
negotiation (8)
• Gender-based violence (6)
• Gender attitudes and norms (6)
• HIV biomarkers (7)
• STI biomarkers (2)
Results
Outcome measures reported

14. Results
Rigor of evaluations
Number of Evaluations by Quality Rating
Strong Moderate Weak
9
14
4

15. • Generally, evaluations of good quality
• Most used only traditional epidemiological
methods
• Only 5 evaluations either included or
referenced qualitative methods
• FGDs, IDI, and KIIs
Discussion

16. Discussion
Future of evaluations

17. • Longitudinal qualitative methods
• Most significant change
• Multi-level analysis
• Social network analysis
• Systems dynamics (agent-based
modeling, discrete event modeling)
Discussion
Future of evaluations: “novel” methods

18. Discussion
What novel or lesser known methods
have you used?
How have you simultaneously
maintained flexibility and rigor?
What methodological, logistical and
ethical challenges have you faced?
How did you address these challenges?

19. This presentation was produced with the support of the United
States Agency for International Development (USAID) under the
terms of MEASURE Evaluation cooperative agreement AID-
OAA-L-14-00004. MEASURE Evaluation is implemented by the
Carolina Population Center, University of North Carolina at
Chapel Hill in partnership with ICF International; John Snow, Inc.;
Management Sciences for Health; Palladium; and Tulane
University. Views expressed are not necessarily those of USAID
or the United States government.
www.measureevaluation.org