Presented at the Strategic Policy Planning, Implementation and Evaluation training workshop held at Source of the Nile Hotel, Jinja. Oct 1,2015

1. Professional
Certificate in Policy
Implementation and
Evaluation
Ian “Ren” Rennie

2. INTRODUCTION TO POLICY
EVALUATION

3. • Who Evaluates?
• Elites:
– Government Agencies, Think Tanks, National or
International Audit institutions, Regulators
• Interest Groups:
– Trade Associations, Business groups, Pressure groups,
Media
• Citizens:
– Complaints, Whistle Blower laws, Public Opinion, Voting
• Of course, some agencies may be shielded from
evaluations, some may have repeated evaluations
thrust upon them.
Politics of public policy evaluation

4. • Determine policy outcomes
• Identify policy strengths
• Identify and improve weaknesses
• Justify use of resources
• Increased emphasis on accountability
• Professional responsibility to show
effectiveness.
Why evaluate?

5. • Outcome
• Implementation
• Formative
• Summative.
Types of evaluation

6. • Summary process.
Evaluation
Identify
scope and
stakeholders
Analysis
Information
gathering
Identify
sources of
information
Team
Selection
Compare with:
business strategy
business case
Use the information for more effective business
operations

7. • Why are you doing the evaluation?
– mandatory?
– policy outcomes?
– policy improvement?
• What is the scope? How large will the effort be?
– large/small
– broad/narrow
• How complex is the proposed evaluation?
– many variables, many questions?
• What can you realistically accomplish?
Scope of evaluation

8. • There are a number of common challenges that
may be encountered
• the evaluation team need to be aware of these
• although they may not be able to solve them.
Common challenges

9. • more than one organisation involved, no common standard
for measuring and recording benefits and costs
• lack of documentation
• inadequacy of baseline measures - measures of success can
only be made accurately by comparing the level of
performance before implementation
• sensitivities – examining performance may lead to feelings
of insecurity or grievance for those involved
• the organisation is too busy and it never gets it done (there
should be policies to ensure that it is normal practice)
• lack of cooperation by involved agencies.
Challenges:

10. • There is one most important question
• What is it that you want to know about your
policy?
• Once you have defined this make it
measurable
• Do not move forward if you cannot answer
this question!
Ensuring effective and efficient
monitoring

11. • What is the purpose?
• What should we measure to ensure that the policy:
• is producing the required products
• is being carried out to schedule and in accordance with its
resource and cost plans
• remains viable against its business case
• Monitoring activities facilitate the checking and
reporting on progress against the plan
• Control activities promote revisions to be made to the
plan in response to issues discovered during
monitoring.
Monitoring ... for control

12. • To benchmark is to compare performance against a
standard
• To improve the effectiveness of monitoring and
evaluation (M&E) in the policy cycle, benchmarking can
be useful in three ways
1. can help place an outcome in context - was the
achievement good, bad, or indifferent?
2. can help assess the reasonableness of targets that
may be set
3. can help identify specific regions or subgroups whose
exceptionally good or poor results hint at what factors
drive performance.
International comparisons and
benchmarking

13. • Today, there is much online
availability of large
internationally comparable data
sets (such as those in the World
Bank’s Data Development
Platform)
• Plus widely available national
survey data
• It is now considerably easier to
use international data for
benchmarking exercises.
Benchmark with what?

14. Range of institutions

15. LINKING POLICY IMPLEMENTATION
AND POLICY EVALUATION

16. • Give an example of a performance indicator
for your area of responsibility ...
Exercise

17. • Consider these assumptions:
• Indicators will provide clear answers to
questions about policy efficiency and
effectiveness
• Indicators will help explain how decisions and
decision outcomes are linked
• Having the right indicators will result in better
decisions.
Performance indicators

18. • Political use – indicators used to support a
predetermined position
• Symbolic use – ritualistic assurance about appropriate
attitudes towards decision-making (“we even
developed indicators”)
• Tactical use – indicators and the process used as a
delaying tactic or substitute for action (“we will do it
later once we have indicators”)
• Conceptual use for enlightenment – change the
understanding of an issue
• Instrumental – direct relationship between indicators
and decision outcomes.
Possible motives

19. • Indicators play an important role in the policy cycle
and can serve a useful purpose
• They can
– Help assimilate and better understand stakeholder views
– Add to the process of governance
– Help guide and mould policy decisions
• Indicators are linked to the governance process ...
• “which can act like a magic bullet causing decision-
making to become instantly objective and scientific”.
Indicators in the Policy Cycle

20. • Saliency
– will anyone care?
• Legitimacy
– can people trust it?
• Credibility
– can people believe it?
• Usability
– can people understand it?
Observing agency outputs

21. Standards for an M&E Plan
• Utility
– serve practical information needs of intended
users
• Feasibility
– be realistic, prudent, diplomatic and frugal
• Propriety
– conducted legally, ethically, and with regard to
those involved in and affected by the evaluation
• Accuracy
– reveal and convey technically accurate
information.

22. Factors to Consider When
Setting Targets
• What can realistically be achieved given the
resources and the program context?
– Baseline levels
– Past trends
– Needs and gaps in services
– Capacity and logistics
– International benchmarking.

23. Approaches to Setting Targets
• Establish final target then plan progress in
between
• Establish intermediate targets
• Assess progress in attaining targets and
readjust, if necessary.

24. • Past trends
• Expert opinion
• Research findings
• What has been accomplished elsewhere
• Client expectations.
Useful Information for Setting Targets

25. • Advocate for the need for M&E
• Understand program goals and objectives
• Identify user needs and perspectives
• Learn about existing data collection systems &
their quality
• Understand indicators that are being collected
and used (or not used)
• Determine capacity for collecting and using data.
Stakeholder Consultation

26. • Developing M&E framework
• Selecting indicators
• Setting targets
• Reviewing results
• Requires building consensus & commitment
and maintaining effective relationships with
intended users.
Stakeholder Participation

27. DEVELOP A DATA COLLECTION MATRIX FOR YOUR PROGRAM. CONSIDER
THE FOLLOWING ISSUES:
• Who will be responsible for data collection and its supervision?
• Who will be responsible for ensuring data quality at each stage?
• How will data quality be checked at every stage?
• How often will the data be collected, compiled, sent, and analyzed?
• What indicators will be derived from each data source?
• How will the data be sent (raw, summary)?
• What tools/forms will be used, if any?
• What resources (staff, office supplies, computers, transportation) will
be needed at each stage?
• Who will analyse the data? How often will analysis occur?
• How often will the results be compiled into reports?
• To whom and how often will the results be disseminated?
Data Collection Plan

28. Developing & Implementing an
M&E Plan: Logic Model
Inputs
•Human
resources
•Understanding
of the program
•Authority and
mandate
•Stakeholders
Processes
•Advocate
•Assess strategic
information needs
•Assess information
systems capabilities
•Achieve consensus
and commitment
•Develop
mechanism for M&E
plan review
•Prepare document
for final approval
Outcomes
Output
•M&E Plan
Document
Short-term
•M&E System for obtaining
Strategic Information
decision making
Long-term
•Evidence-based decisions
for improving programs
Impacts
•Policy delivery

29. • Start early
• Involve stakeholders at all stages in the process
• Assess strategic information needs for intended users
• Assess current capacity and use what is already available
• Avoid duplication of data collection and reporting
• Do not collect information that will not be used
• Review progress / results regularly and make adjustments
to M&E plan, if necessary.
Summary: The Basics

30. DIRECT VS INDIRECT POLICY
IMPACTS

31. Dandelion dilemma

32. • Case study 1
– Continuously troublesome students are excluded from
school to prevent disturbance of well behaved
students
• Case study 2
– in 18th- and 19th-century northern Europe, there was
a “window tax”
– a property tax whose amount depended upon the
number of windows in a house
– an explicit “income tax” was politically unpopular, so
the intention was to create an income tax without
explicitly taxing on the basis of income.
First order and second order
impacts

33. • Unintended consequences arise because we
live in an interconnected and complex world
• Interconnectedness and complexity of the
world imply that a small change in one part of
a system can potentially generate far-reaching
effects in another part of the system
• Also policymakers may be unable to anticipate
or understand how their policies affect the
incentives of individuals within the system.
Complex systems

34. • public policy attempts to address social
problems, often without understanding
the policy’s effects on individuals’
incentives
• systematic consideration of a policy’s
intended and unintended effects is
imperative
• just because a policy could theoretically
affect individuals’ incentives doesn’t mean
that the policy actually will
• the extent to which individuals respond to
incentives is an empirical question, which
requires an empirical answer.
Empirical answers

35. POLICY EVALUATION
METHODOLOGIES

36. • The good evaluation of a project or a policy
needs on the one hand thorough application
of methodology up to the highest professional
standards ...
• and is on the other hand a creative thought
exercise: what do we really want to know and
how to find out.
Evaluation methodology

37. • If policies are meant to improve on outcomes, they should not be
based on ideology or wishful thinking, but rather on proven
effectiveness
• But obtaining convincing evidence on the effects on specific policies
is not an easy task
• Relevant data on possible outcomes has to be gathered
• Showing a mere correlation between a specific policy or practice
and potential outcomes is not proof that the policy or practice
caused the outcome
• For policy purposes, mere correlations are irrelevant, and only
causation is important
• What would really happen if a specific policy is introduced – would
it really change any outcome that society cares about?
• Evidence-based policy require answers to causal questions.
Overview

38. • The need for evidence-based policy is increasingly
recognised (e.g., Commission of the European Communities
2007)
• Providing empirical evidence suitable for guiding policy is
not an easy task
• It refers to causal inferences that require special research
methods
• Not always easy to communicate due to technical
complexity
• Econometrics is a set of quantitative techniques that are
useful for making "economic decisions“
• Of course, economic decisions are not restricted to those
made by economists ...
Econometrics

39. • The decision of a student to stay an extra year at
university to earn a double degree
• The decision of a spice seller to open a shop in
this village, or another
• The decision of a bank to approve a loan
application
• The decision of a power company to start-up an
extra generator on a certain day
• The decision of a Government to introduce a
purchase tax in order to reduce income taxes.
Examples:

40. • The "efficient allocation of resources" is the subject
matter of economics
• The traditional role of econometricians is to "test"
alternative economic theories
• To confront theories with real data and to reject those
that are not compatible with reality
• It is what transformed economics from an "art" into a
"science“
• However, it only gave the econometrician an indirect
role as an economist's sidekick
• If economists didn't come up with new theories,
econometricians didn't have anything to do!
What is old?

41. • Regardless of what economists do, economic problems
persist in reality and decisions have to be made
• The modern role of an econometrician is therefore more
direct
• Econometricians extract information from the available
data in order to provide support for an economic decision
• The raw material that the econometrician uses is more
often real observed data, instead of economic theory
• In any specific problem if an economic theory that has been
validated by data exists, it will certainly be used
• If no such model exists, however, econometrics must still
make a statistical model for the data, and provide
intelligent support for the decision process.
What is new?

42. • In government decision-making there are always
competing priorities for funds
• The purpose of a CBA is to allow competing policy
priorities to be compared in a consistent way, and for
their economic, social and environmental impacts to be
assessed
• In all areas of policy, the function of the CBA is to assist
policymakers to identify the best way to deliver the
strategic objectives of governments
• And then to assist with the evaluation of the policy.
Cost Benefit Analysis (CBA)

43. 1. needs to be future looking
2. needs to be objective
3. needs to consider implementation risks
4. to be easily understood so it can be subject to a
degree of contestability.
What does a good CBA look like?

44. 1. establishing the base case – or the ‘no change’ policy
option
2. defining the policy options to be evaluated and
compared against the base case and against each
other
3. laying out the estimates and assumptions for external
factors affecting the policy outcomes
4. defining and estimating the costs of a policy proposal
5. defining and estimating the benefits of a policy
proposal
6. drawing a conclusion.
Key steps in a CBA

45. • An initial question ‘what would happen if this investment
did not take place?’
• allows the CBA to compare two futures: the future with the
investment and that without the investment
• The base case does not mean ‘how things have been in the
past’ and should include any observed long term trends
• In particular, the base case needs to account for future
population and economic growth
• All forecasts contain an element of uncertainty
• In order to address this uncertainty, forecasters need to
develop different forecast scenarios (usually low, mid-range
and high scenarios).
1. Establishing the base case

46. • The purpose of any investment proposal is to achieve an
outcome that improves the wellbeing of the community
• In many instances there are multiple approaches that could
achieve the outcome
• Similar to the base case, the policy options in a CBA should
be clearly articulated and reasonable
• It is also important to consider whether there are any
policy options that may have been overlooked and not
included in the CBA
• A well thought-out CBA includes all the feasible options and
provides justification where options have not been covered
• Avoid “Goldilocks” analysis.
2. Defining the policy options

47. • Analysis will be built up from a range of estimates and assumptions
• Estimates are variables where there is existing evidence, preferably
from multiple sources
• Assumptions are variables where there is less robust data and the
team developing the CBA has had to use judgement
• Estimates are clearly preferable to assumptions
• A CBA, however, almost always includes assumptions because not
all of the information needed to perform the analysis is available
• Just because it is necessary to make some assumptions to conduct a
CBA does not mean it is not worth doing as all analysis involves
assumption about the future
• One strength of CBA is that it makes it clear what the key
assumptions are and allows for the systematic testing of
assumptions.
3. Estimates and assumptions

48. • What assumptions need to be made to conduct the
analysis?
• consider the impacts of a proposal across all sectors of
the economy by thinking about the inter-relationships
between sectors (economists term this ‘general
equilibrium analysis’)
• include all intangible costs and benefits where they can
be reliably estimated
• include all subsequent or contingent costs/investments
• analyse over the economic life of the underlying
proposal or assets, subject to a maximum of 20 years.
3. Estimates and assumptions

49. • Takes into account all the impacts of the policy
options that produce undesirable impacts
• A useful way of looking at costs is to identify which
individuals or groups in the community are worse-off
as a result i.e.
– Government
– Business
– The community
• The full range of costs in a CBA should be identified,
even those that may be difficult to measure -
understating or overlooking a cost that is difficult or
impossible to quantify is a common fault with CBAs
• Everything is relative to the base case – that is, the
costs that are additional to the base case.
4. Costs

50. • Helpful to ask, ‘who is made better off as a result of the investment?’ i.e.
– Government: time or money saved through more efficient programs or
processes
– Businesses: reduction in time or cost, particularly with respect to
administration, or improvements in efficiency, productivity and innovation
– The community: increase in welfare, safety, participation and/or connectivity
• Many CBAs will also look at benefits according to the ‘triple-bottom-line’
which covers the positive economic, environmental and social outcomes
of the investment
• While there is nothing wrong with classifying benefits under these
headings, it is crucial to avoid double-counting
• Like the cost side of the CBA, it is important to identify all the benefits
related to the policy options and measure them relative to the base case.
5. Benefits

51. • Finally calculate the value of each policy option taking into account
all future discounted benefits and all future discounted costs
• The final overall value is usually expressed in one of three ways:
– Net present value or Internal rate of return or Benefit-cost ratio
• In a CBA some options may be easier to value than others
• Although CBA places an emphasis on quantifying in financial terms
the impacts of potential options, the CBA should not be biased
towards an option just because it is easy to value
• This does not necessarily make it the best approach
• However, where difficult-to-value impacts are included, document
carefully and transparently the manner in which that has been
done.
6. Drawing a conclusion

52. • An important and often overlooked step, and
purpose is to:
– Identify whether the expected benefits of the
project have been realised
– Assess whether any aspects of the current project
require remedy
– What lessons can be learned from the project for
future investment projects.
Post implementation

53. • Quantitative observations are made using scientific
tools and measurements
• The results can be measured or counted
• Any other person trying to quantitatively assess the
same situation should end up with the same results
• An example of a quantitative evaluation would be
"this year our government paid 4m citizens a pension
totalling $4,000m at an average of $1000 per
pensioner.“
Quantitative

54. • More subjective than quantitative evaluation, qualitative
observations are defined in science as any observation
made using the five senses
• Because people often reach different interpretations when
using only their senses, qualitative evaluation becomes
harder to reproduce with accuracy
• Two individuals evaluating the same thing may end up with
different or conflicting results
• In research and business, qualitative evaluations may
involve value judgments and emotional responses
• An example of a qualitative evaluation is "our government
provided a better pension arrangement last year than this
year."
Qualitative

55. • Both evaluation methods have their benefits, though one is usually
more appropriate than the other in any given situation
• Quantitative evaluation can help remove human bias from a
statistic, making it more of a reliable fact than any piece of
information gathered qualitatively
• Thus, accurate quantitative evaluations can be relied upon as truth
• Qualitative evaluations may also entail truths, but these truths are
harder to get at, and evaluators may not always agree
• Still, qualitative information is invaluable precisely because it
involves human interaction and interpretation
• Many fields need this interpretation in order to decide how to move
forward or to judge the value of past and future sources of data.
Benefits

56. • Individuals or groups are randomly assigned to
either the policy intervention or non-intervention
(control) group and the outcomes of interest are
compared
• There are many methods of randomisation from
field experiments to randomised control trials
• Random allocation design means that systematic
differences between groups are unlikely and so
any differences and changes in outcomes
between the two groups can be confidently
attributed to the policy intervention.
Control groups

57. Random allocation design

58. • Increasing recognition that rigorous evaluation of public
interventions should feature in the policy decision making process
• Yet there is frequently a gap between the desire for information on
the effectiveness of programs and an understanding of the
potential and the limitations of evaluation tools
• What questions can evaluations answer?
• What administrative structures are required to implement them?
• What are the political and social factors surrounding the acceptance
of evaluations by target groups and the public?
• How much do evaluations cost?
• How long do they take to complete?
Cost-implications of different
methodologies

59. • For international aid programmes it is suggested that
the cost of an evaluation is between about 5 and 7
percent of the cost of the program being evaluated as a
rule of thumb (W. K. Kellogg Foundation, 1998)
• Other evidence suggests that there is very little
correlation between the cost of an impact evaluation
and the size of the program being evaluated.
Evaluation costs

60. • Objectives
– the policy questions to be addressed determine how complex
the study design must be and consequently how costly
– simpler is usually better
• Availability of representative socioeconomic data
– if national data exists, alternative methods of selecting a
comparison group can be considered which may be cheaper
than the ideal random assignment design
• Timeframe
– evaluators frequently do not have the luxury of months or years
for data collection
– this lack of time necessitates more creative use of data and
more complicated statistical techniques, potentially raising the
overall price of evaluation.
Cost factors

61. IMPACT ANALYSIS

62. • Conduct a gap analysis
• Determine whether the project
goals were achieved
• Determine the satisfaction of
stakeholders
• Determine the project's costs and
benefits
• Identify areas of further
development
• Report findings and
recommendations.
Questions for post-
implementation review

63. • An equality impact assessment (EIA) is a tool that helps
public authorities make sure their policies, and the ways
they carry out their functions, do what they are intended to
do and for everybody
• Carrying out an EIA involves systematically assessing the
likely (or actual) effects of policies on people in respect of
disability, gender, including gender identity, and racial
equality and, where you choose, wider equality areas
• This includes looking for opportunities to promote equality
that may have previously been missed or could be better
used, as well as negative or adverse impacts that can be
removed or mitigated, where possible.
What is an Equality Impact
Assessment

64. • In many countries, if any
negative or adverse impacts
amount to unlawful
discrimination, they must be
removed
• Typically there are four possible
outcomes and more than one
may apply to a single policy:
– No major change
– Adjust the policy
– Continue the policy (justification)
– Stop and remove the policy.
Possible Outcomes

65. • Statutory Equality Duties Guidance
• Competition Assessment
• Small Firms Impact Test
• Greenhouse Gas Impact Assessment
• Wider Environmental Impact Test
• Health and Well-Being
• Human Rights
• Justice Impact Test
• Rural Proofing
• Sustainable Development Impact Test.
Other impact tests include ...

66. • http://webarchive.nationalarchives.gov.uk/2
0100503135839/http://www.ogc.gov.uk/doc
uments/policy_to_delivery_A4_Guidance_P
1_(2)_(2).pdf
• https://www.gov.uk/government/uploads/sy
stem/uploads/attachment_data/file/220541
/green_book_complete.pdf
• https://www.gov.uk/government/uploads/sy
stem/uploads/attachment_data/file/220542
/magenta_book_combined.pdf
• http://webarchive.nationalarchives.gov.uk/2
0130129110402/http://www.hm-
treasury.gov.uk/d/orange_book.pdf
• http://webarchive.nationalarchives.gov.uk/2
0130129110402/http://www.hm-
treasury.gov.uk/d/riskmanagement_assessm
ent_framework220709.pdf
References

67. QUESTIONS AND CLOSE