This document discusses the "Science of Delivery in Education" method known as deliverology. It summarizes key points made about deliverology's reported results in improving education in Pakistan. While initial reports showed large gains, upon further analysis the data does not provide clear or consistent evidence that deliverology alone caused such improvements. Enrollment data does not clearly show breaks from prior trends and learning data lacks comparable measures over time. More independent and rigorous data is still needed to fully evaluate deliverology's effectiveness.

1. Discussion
Jishnu Das
(DECRG)
Science of Delivery in Education
This discussion is based on “The Good news from Pakistan”, by Sir Michael Barber and his
presentation at The World Bank on 6th June 2013. The material for the discussion is partly
from the LEAPS project with Tahir Andrabi (Pomona College) and Asim Khwaja (Harvard). I
also thank the South Asia Education Team, and in particular, Dhushyanth Raju, for their
valuable input.

2. Everyone agrees that something
must be done
 Children face “double-burden” of low schooling attainment and
very low learning
 Primary Enrollments increased in recent years
 But test-scores remain very low (1sd below OECD averages)
 Teachers big part of the solution
 Solving problems of high teacher absence rates (15-
30%), poor time-on-task, low attention can significantly
improve scores
 Poor infrastructure may be a contributing factor

3. What I agree with
 Managing teachers is responsibility of managers, not parents
 School Principals in private schools
 Schools Principals and the educational administration in
public schools
 Contrast: Not the job of parents to ensure that teachers come
to work. No private firm forces customers to ensure that staff
come to work—that’s the work of management
 Quote: “Today you ask us why we don’t get teachers to come.
The NGO says that we should file an RTI, meet the district
officer, hold up traffic and force the teachers to come. We have
to do this for the schools, for the electricity, for the doctors, for
the roads, for the garbage and for anything at all. You tell me—
when should I work in my fields? Do you see that shop selling
Pepsi? I go there and buy the Pepsi. If it’s not there, I don’t
have to do anything. Yet, it’s always there.”

4. What I also agree with
 Some states have lost (or maybe never had) the
ability to govern frontline workers
 Lant Pritchett calls this “The Flailing State”
 This is hard and has to be learnt
 We don’t know what it takes, but know it can be
done
Source: Pritchett, Lant. 2009. “Is India a Flailing State: Detours on the Four Lane Highway to Modernization.”, HKS Faculty
Research WP, RWP09-013

5. An Indian Example: Bihar
 Re-election platform of better service delivery plus
 Critical coalition building and politics to obtain the legitimacy
to use sticks and carrots to govern
 That’s what good politicians do
One of India’s worst
governed states: Security
issues, low growth and
unchecked corruption.
Absence rates for
doctors/nurses 70%, for
teachers 40%
Nitish Kumar
elected CM
200
3
200
5
201
0
Complete turn-
around: Greater
security, high growth.
Absence rates for
doctors/nurses
30%, for teachers
28%
Source: Muralidharan, Karthik, Jishnu Das, Alaka Holla, Michael Kremer and Aakash Mohpal, 2013. “The Fiscal Cost of Weak
Governance: Evidence from Primary Education in India.” In Process.

6. An Indian Example
 High benefit-cost ratios of top-down monitoring
 Increasing inspection in last 3 months by 10 p.p.
 Costs Rs.450 million
 Saves Rs.4,048 million in reduced teacher absence
 To produce same effective student-teacher ratio through extra
teacher hiring costs Rs.5,204 million
 This is a longitudinal, nationwide study
 Gathering evidence that top-down monitoring when there is political
backup works in improving services
 Theory: James Wilson, Paul Grout’s work on Bureaucracies
 Changes at the top can have large cascading effects at the bottom
Source: Muralidharan, Karthik, Jishnu Das, Alaka Holla, Michael Kremer and Aakash Mohpal, 2013. “The Fiscal Cost of Weak Governance: Evidence
from Primary Education in India.” In Process.
Wilson, James Q. 1991.“What Governments Do, and Why They Do it.” Basic Books Classics.
Grout, Paul . 2013. “Collapsing Morale in Bureaucratic Environments.” University of Bristol

7. The Service Delivery Problematic: What does
deliverology contribute?
Problem #1: No Political Will to improve education outcomes: Top-down monitoring can fail
badly (Example: Banerjee and others: Nurses break time-stamp machines when wages are
tied to attendance). Building political will for better services is the holy grail.
Deliverology: Could be that deliverology helps build that political will
Not claimed: Political will a pre-condition (Pakistan, but also Malaysia and Sierra Leone)
Problem #2: Political Will, but inability to govern the frontline
Deliverology: Could be that deliverology provides the governability
Claimed: This would be an important step forward in these situations
Problem #3: Political Will and ability to govern
Deliverology: Deliverology is one among several paths that the top manager chooses, but
any number of methods could work
Source: Banerjee, Abhijit, Rachel Glennerster and Esther Duflo. 2008. “Putting a Band-Aid on a Corpse: Incentives for Nurses in the Indian Public Health
Care System. Journal of the European Economic Association

8. Reported results
Better infrastructure
Greater
enrollment and
learning
Between August 2011 and January 2013, Sir Michael’s and the Punjab
education leaders’ efforts achieved the following results (pp. 12-29):
Approaching one and a half million extra children enrolled in school.
Facilities with functioning electricity, drinking water, toilet and boundary
walls increased from 69 per cent to 91 per cent.
Student attendance increased from 83 per cent to 92 per cent.
Teacher attendance increased from 81 per cent to 91 per cent.
81,000 new teachers hired on merit.
Simple, easy-to-use lesson plans for every teacher and new textbooks for
every student
Growing evidence that learning outcomes are rising too; whereas two
years ago Punjab-India and Punjab-Pakistan were roughly on a par, now
Punjab-Pakistan is out in front.
A voucher scheme which enables over 140,000 out-of- school children of
poor families to attend private schools; this is the fastest growing voucher
schemes in the world.
New Programs
Source: Barber, Sir Michael. 2013. “The Good News from Pakistan.” Content accessed from
http://www.reform.co.uk/content/20419/research/education/the_good_news_from_pakistan

9. What can the data tell us?
 Problems of counterfactuals (program is province-wide)
 Researchers very good at dealing with this using innovative methods
 More seriously, critical lack of data
 Enrollment: Consistent government data till 2010/11, subsequent rounds not
released
 Nielsen household survey 2011-2013, commissioned by DFID
 Data not publicly available, not compared to other data sources
 Government data through annual census of schools
 Monitoring data may have problems; covers only public schools
 LEAPS dataset, selected villages in 3 districts: 2003-2010
 School Inputs: Projects own monitoring database
 Not publicly available nor triangulated with other data sources
 LEAPS dataset, selected villages in 3 districts: 2003-2010
 Learning: No consistent, comparable data over time
 Annual Survey of Education Report (ASER): Volunteer-army model, wide
coverage, very short test, home-based tests.
 Good for overall picture in a year, but not for year-on-year changes
 LEAPS dataset: Consistent, comparable data for 2003-2010 for 3 districts
 At this point, what can the data tell us?

10. Enrollment: Household Surveys5060708090
1995 2000 2005 2010 2015
year1
All Children Rural Only
All and Rural Enrollment: Punjab, Pakistan
Enrollment rate (primary) 2003 2004 2005 2006 2010
LEAPS Villages 54 61 68 69 71
Government HH survey
54
(2002) 66 64 70 70
Complete stagnation from
verified, single data
source
Deliverology begins:
Data source changes;
15 p.p jump from 70 to
85% questionable
Age of Deliverology: Small
increases relative to
previous experience; slow
down between 2011 and
2012. 85% in 2011, 86% in
2012, 86.8% in 2013. No
standard-errors in report.
Age of large
increases, from
verified, single data
source. Identical in
LEAPS data
Large WB program
in Punjab New release of
government data shows
74% NER

11. Enrollment: Triangulate
Government Monitoring Data
•Annual School census based on the monitoring division: Same data used to show input improvements (but
not enrollment in report)
•Decreasing number of schools due to consolidation and merger policy
•No change in public school enrollment at all: Insignificant decline of 4000 children
•Note: Student-teacher ratio declined: Good if it increased learning, bad if it only increased costs
•Parallel work from India shows very small impacts of decreasing STR on learning, partly due to increased
absence!
•Triangulation does not provide consistent evidence. Effect of deliverology could have been positive, nothing
or negative depending on counterfactual; in all likelihood effects small in either direction
Nielsen Household Survey: Enrollment increased, share of public schools
increased. This implies that public enrollment numbers must have increased
Variable 2009/10 2010/11 2011/12 2012/13
Schools 61,474 60,494 59,461 57,425
Functional
Schools
59,029 58,187 56,867 54,392
Teachers 165,086 161,947 158,386 165,600
Total Enrollment 10,644,408 10,619,575 10,632,546 10,640,159

12. Enrollment: Triangulate
 (Slide added post-discussion)
 Post discussion, I became aware of the release of the Government’s household
survey, the PSLM for 2011-2012
 According to PSLM 2011-2012
 The Net Enrollment Rate (NER) for children 6-10 in primary education was
74% in 2011-2012
 Much of the increase comes from an increase in enrollment rates in private
schools
 42% in 2007/08; 37% in 2010/2011 and 44% in 2011/2012
 This much smaller increase is completely consistent with the constant
government enrollment numbers in the monitoring data
 Severe discrepancy between PSLM and Nielsen for 2011/2012: Again
emphasizes need to understand how data were collected!
Source: PLSM 2011/2012: Accessed from http://www.pbs.gov.pk/sites/default/files/pslm/publications/pslm2011-
12/complete_report_pslm11_12.pdf

13. Inputs
Using Government project monitoring data, there have been improvements in inputs
Triangulation: ASER, same story
Availability for
functional
schools
2009/10 2010/11 2011/12 2012/13
Electricity 61.8 66.2 69.5 78.3
Drinking Water 87.3 88.9 92.4 97.6
Boundary wall 82.2 84.7 87.9 89.7
Toilets 80.3 84.8 89.6 95.4
All Four 49.7 54.5 60.3 71.5

14. Inputs
Continues long-standing upwards trend, dating back to 2003, data based on LEAPS villages
2003 2004 2005 2006 2010
Number of permanent classrooms 3.89 4.05 3.81 3.41 5.67
Number of semi-permanent
classrooms 0.55 0.47 0.99 1.19 0.04
Number of staff rooms 0.22 0.29 0.20 0.32 0.22
Number of boys' toilets 0.71 0.77 0.53 0.58 1.25
Number of girls' toilets 0.41 0.42 0.89 0.66 1.23

15. Learning
Changes in Grade 4 test-scores, 2004-2011. Using Equated tests in
LEAPS villages
2004 2006 2011
Math -0.29 -0.21 -0.44
English -0.55 -0.22 -0.12
Urdu -0.41 -0.20 -0.34
Math scores worsened, English improved, Urdu remained the same.
There are currently no data from after 2011 in LEAPS, turn to ASER in 2011
and 2012

16. Learning
ASER data from Punjab show large improvements in single year: may be as large as 1 standard-
deviation equivalents: No comparable results from any other countries!
Note: Identical results from province of KP, without deliverology. Mostly likely hypothesis: survey
changes, rather than real learning. Needs to be independently verified!

17. Innovative new programs
 Vouchers and Teacher merit hiring are extensions of pre-existing
programs and discussions
 Punjab Education Foundation was set up around 2007/08 and
has been leading innovative methods for private school
financing
 Teacher hiring has been under discussion for a long time.
 Merit hiring may or may not be a good thing; needs to be
evaluated
 Why? Because teacher value-added has a very small
correlation with observed teacher attributes

18. Does deliverology deliver?
 Initially very excited about the good news from Pakistan
 Perhaps this new method could improve governance when there was
political will
 Reassessment
 Very little support for enrollment increases. Story does not
triangulate, trend not supportive of break
 Inputs have improved, and continue a trend started back in 2003 with
Padha-Likha Punjab (similar to Education for All)
 No consistent data on learning: Improvements in ASER very large to
anything we have ever seen, and identical in KP and Punjab;
potentially survey changes
 Reassessment: No consistent story of large improvements that are
off-trend. Deliverology one of many potential methods that are being
tried, each with (potentially) some effects
 Reassessment: Punjab made huge strides between 2003 and
2010, both in enrollment and learning

19. Next Steps
 Biggest issue is lack of consistent, credible data to evaluate and
understand what deliverology does
 Key components currently missing are legislated data
independence, regular public scrutiny of improvements, third-
party evaluations
 Contrast with evidence from several rigorous randomized
control evaluations in Pakistan, that are now entering the
scale-up phase
 The lack of data in support is surprising given the emphasis in
the method on data