Presented by Valerie Flax at the 2016 AEA conference.

1. Lessons Learned Collecting
Most Significant Change Stories in
Impact Evaluation in Malawi
Valerie Flax, PhD, MPH
MEASURE Evaluation
October 29, 2016
American Evaluation
Association

2. Integrating nutrition in value chains
 Mixed-methods evaluation
• Overall question: Is an integrated value
chain + nutrition program more effective
in improving nutritional status of children
< 3 years than a value chain alone?
 Program impact measured through
population-based household surveys
 Program implementation measured, in part,
through Most Significant Change (MSC)
stories
Background
Malawi Feed the Future Impact Evaluation
Target
Districts

3. Nutrition and Health Investments
Improved diet
and feeding/care
practices
Increased legume
production
Increased Food
Security
Increased
household income
Demonstrations:
Gardens/Cooking
Improved Nutrition
Reduced chronic
malnutrition among
children U3
Impact
Community Care
Groups
Training care group
volunteers on:
 Health screening
and services
 Essential nutrition
actions (ENA) and
scaling up nutrition
(SUN)
 Food production,
processing, and use
Agriculture Value Chain Investments
Agricultural Inputs:
• Improved Seed
• Technology
• Fertilizers
• Irrigation
Agricultural
Practices:
• Diversification
• Conservation
farming
• Demonstration
farms & gardens
Processing &
Marketing:
• Storage
• Processing
• Aggregation and
Distribution
• Retailing
Increased farm
income
Increased Food
Diversity
Increase Use of
Health ServicesNutrition Screening
& Service Referrals
Improved nutrition
knowledge
Nutrition Education:
ENA and SUN

4. DAI - Prime
Univ of Michigan -
partner
Save the Children -
partner
NASFAM FUM CADECOM MMPA
Nkhoma
Synod
AG Technical Service Providers
Nutrition Technical
Service Provider
Farmer’s
Assoc.
Farmer’s
Assoc.
Farmer’s
Club
Farmer’s
Club
Farmer’s
Club
Lead Farmer
supports Club
Soy
Grp
GN
Grp
Gender, Nutrition,
HIV Group
CCGCCG
Volunteer
Leader
Volunteer
Leader
Volunteer
Leader
INVC Integration Model
HH: U5
& PLW
HH: U5
& PLW
HH: U5
& PLW
HH: U5
& PLW
HH: U5
& PLW
HH: U5
& PLW
Community
Care Group
(CCG) Nutrition
Promoter (120)
1 Promoter
supports 3 CCGs
(360 CCGs)
Mentors 12 HH
1 CCG=12
Volunteer Leaders
(4320 volunteers)
Mentors 12 HH Mentors 12 HH
District Coord. (3)
Nutrition Asst. (6)
Soy
Grp
Soy
Grp
Chili
Grp
Farming /
Marketing Groups
Group Action Committees
Farmer Farmer Farmer Farmer
Farmer Farmer Farmer
Farmer’s
Assoc.
Assoc. Field Officers
train Lead Farmers
GVH/GACVillageDistrict
Integration point
of entry

5. Davis, Rick and Jess Hart (2005). The ‘Most Significant Change’ (MSC) Technique: A Guide to its Us
http://www.mande.co.uk/docs/MSCGuide.pdf. Photo: Elizabeth T. Robinson/MEASURE Evaluation
 From participants’ perspective, what aspects of the
integrated program had the biggest impact on
their lives?
 How did the impacts of the program on
participants’ lives change over time?
Methods: Why MSC stories?

6.  26 nutrition promoters in
two districts trained to
collect stories
 Each promoter collected
one MSC story per month
 Promoters met quarterly to
review stories, select most
significant, and document
reasons selected
 Data collected for
one year
 Data coded in Dedoose
Methods

7. Word cloud
Improved
household
health status
Improved
nutrition
knowledge
Improved
farming
practices
More
time or
money
Improved
hygiene
Increased
sales of
agricultural
products
Improved
infant and
young child
feeding
practices
Improved
health
knowledge
Results

8. “Even though my wife is the
group member, I can still see
changes in our family. Our
child is healthy and we use
soya to make milk and snacks.
We also have porridge each
morning that is making us
healthy. In the past, we didn’t
know much about soya and
used to just sell it. Now we use
it for nutrition and my child is
healthy.”
Improved health status
and nutrition knowledge
Results

9. “We are now able to save
money because our
children no longer get
sick regularly and they
are growing well. This is
very significant because I
am able to do my
household chores. This
year, I have been able to
cultivate in the garden
because my children are
healthy.”
More time and money
Results

10.  Lead mothers get materials,
but cluster members do not
 Promoters are not getting
their bicycle allowance
 Competition between INVC
and other projects
 Lack of supervision of
promoters
 Inadequate male
involvement in program
 Participants and promoters
want incentives for MSC
stories
Issues raised by
nutrition promoters
Results

11. Challenges to Using MSC
 Length of program implementation before
MSC collection
 Level of education of nutrition promoters
 Volunteer nature of program
 Lack of “thick” description
 Repetitive stories
 Use of stories to highlight health promoter
needs
 Issues with voting on most significant stories

12.  MSC stories tell us what
matters to participants, even
if descriptions
are “thin”
 Quarterly meetings
with promoters help
to understand program
implementation
 The process may need to be
adjusted to education level of
data collectors
 Consider utility of continuous
MSC data collection
Conclusions

13. Acknowledgements
Centre for Public Health Research and Development
 Agatha Bula
 David Chilongozi
 Jennifer Tseka
MEASURE Evaluation
 Gustavo Angeles
 Milissa Markiewicz

14. This presentation was produced with the support of the United
States Agency for International Development (USAID) under
the terms of MEASURE Evaluation cooperative agreement AID-
OAA-L-14-00004. MEASURE Evaluation is implemented by the
Carolina Population Center, University of North Carolina at
Chapel Hill in partnership with ICF International; John Snow,
Inc.; Management Sciences for Health; Palladium; and Tulane
University. Views expressed are not necessarily those of USAID
or the United States government.
www.measureevaluation.org