During a September presentation at South Africa’s Department of Planning, Monitoring and Evaluation, Dr. Jason Smith shared experiences and lessons learned on evaluating impact from MEASURE Evaluation Phase III implementation

1. Evaluating Impact: Lessons Learned
from MEASURE Evaluation
Jason B Smith, PhD, MPH
Director for Health Systems Strengthening

2. Presentation Overview
• What is Evaluation?
• What is MEASURE Evaluation?
• Lessons learned from Phase 3
• Special topics:
– Strategic Information for South Africa (MEval-SIFSA)
– GEMNET-Health and University of Pretoria
– Evaluation Resources
• Questions, Comments and Dialogue

3. What is Evaluation anyway?
• evaluation [e-val″u-a´shun]
– a critical appraisal or assessment;
– a judgment of the value, worth, character, or effectiveness of
something;
– measurement of progress.
From : The Free Dictionary (online)

4. Evaluation: USAID Policy
• “Evaluation is the systematic collection and analysis of
information about the characteristics and outcomes of
programs and projects as a basis for judgments, to
improve effectiveness, and/or inform decisions about
current and future programming”
USAID Evaluation Policy, January 2011
“The two purposes of evaluation are to provide information
for decision-making and contextual learning, and to
demonstrate accountability for resources”
Evaluation of USAID, November 2013

5. Why Evaluate Programs?
• Determine if an intervention was delivered as per plan
• Discover if the intervention had the desired effect(s), and
• Explore whether there are ways that the intervention
could be delivered faster or better or at lower cost

6. Basic Components
The basic components of evaluation are:
• identifying the parameters
• developing criteria
• data gathering
• measuring the data against the criteria, and
• employing the results
From : The Free Dictionary (online)

7. Types of Evaluation Research
• Impact
• Outcome
• Public Health
• Methodological Studies
• Population-based Surveys and Surveillance

8. What is Measure Evaluation?
• MEASURE Evaluation is the USAID Global Health
Bureau’s primary vehicle for supporting improvements in
monitoring and evaluation in population, health and
nutrition worldwide.

9. MEASURE Evaluation Timeline
• 1991-1997 - The Evaluation Project
• 1997-2003 - MEASURE Evaluation Phase 1
• 2003-2008 - MEASURE Evaluation Phase 2
• 2008-2014 - MEASURE Evaluation Phase 3
• 2014-2019 - MEASURE Evaluation Phase 4

10. The MEASURE Evaluation Team

12. MEASURE Evaluation Phase III
Evaluation Activities
North Atlantic Ocean
Arctic Ocean
Indian Ocean
Arctic Ocean
North Pacific Ocean
Nepal
Nigeria
Senegal
Ghana
Ethiopia
Uganda
Rwanda
South Africa
Madagascar
Mali
Tanzania
Benin
Mozambique
Kenya
India Bangladesh
Malawi
Angola Zambia
Guatemala
Mexico
Haiti
Ukraine

13. MEASURE Evaluation
Guiding Principles for Evaluations
• Planning in parallel with program implementation.
• Work with country programs as well as global partners
• Partnership with local institutions to build their capacity
• Where appropriate, cost components are included
• Dissemination and use of study results incorporated from
the outset.

14. MEASURE Evaluation
Criteria for prioritizing evaluation activities (1)
• Comparative advantage
• Theory driven
• Clearly adds to evidence base
• Produces actionable finding
• Reasonable relative cost
• Balance of cost with ability to achieve objectives

15. MEASURE Evaluation
Criteria for prioritizing evaluation activities (2)
• Incorporates capacity building, data use, and knowledge
management principles
• Follows protection of human subjects principles
• Generalizability of results
• Other pragmatic elements

17. Lessons Learned: Ongoing Challenges
• Centralized evaluation agendas hard to implement
• Evaluations are slow – lose momentum and visibility
• Need to improve publication, communication and data
use
• Designs are compromised by field realities – “real
world” evaluation
• “Evaluation” means different things to different people

18. Lessons Learned: Implementing Evaluation
• Begin with the end in mind (but easier said than done)
• Timing is everything
• Size matters
• Pressure for rapid results to inform programs now
• Be prepared to manage expectations
• The need for ownership
Archive of Phase 3 End of Project Event Sessions
www.measureevaluation.org/eop

19. A Message of Hope

20. Special Topics
• Strategic Information for South Africa (MEval – SIfSA)
• GEMNet-Health and University of Pretoria
• Evaluation Resources

21. Current Associate Awards

22. MEASURE Evaluation in South Africa
Strategic Information for South Africa (MEval–SIFSA)
Supports the Department of Health (DoH) at national,
provincial and district levels to improve strategic
information for evidence-based management of HIV and
related health programs.
MEval-SIFSA seeks to ensure that the health system in
South Africa produces useful and high quality information
that contributes to sustainable policy, planning, and
programmatic decision making.

23. SIfSA Strategic Approaches
• System-wide approach
• Leveraging technology
• Data use
• Commitment to sustainability

25. University of Pretoria
• MEASURE Evaluation Regional Training
Partner since 1999
• Pre-service Education in M&E at MPH level
• In-Service Workshops
• Provision of Technical Assistance to others
• Support for Capacity-Building under
MEval-SIfSA
• Founding member of GEMNet-Health

27. Acknowledgements
This presentation is not all of my own original thinking. A
number of my colleagues have contributed to this
presentation in person, in print or by providing presentation
materials. These colleagues include Sian Curtis, Jim,
Thomas, Heidi Reynolds, Tara Nutley, Peter Lance,
Gustavo Angeles, Derek Kunata and Hemali Kulatilaka. My
apologies to anyone I may have forgotten.

28. MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) through cooperative
agreement GPO-A-00-03-00003-00 and is implemented
by the Carolina Population Center at the University of
North Carolina at Chapel Hill, in partnership with Futures
Group, ICF International, John Snow, Inc., Management
Sciences for Health, and Tulane University. The opinions
expressed in this presentation are those of the authors
and do not necessarily reflect the views of USAID or the
U.S. government.
Visit us online at http://www.cpc.unc.edu/measure

29. Questions, Comments and Dialogue
Visit our website at:
www.measureevaluation.org