Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Demand and Use Workshop

MEASURE Evaluation presentation by Isabel Brodsky and Verne Kemerer

  • Login to see the comments

Data Demand and Use Workshop

  1. 1. Data Demand and Use Workshop Isabel Brodsky and Verne Kemerer MEASURE Evaluation October 24–27, 2017 Freetown, Sierra Leone
  2. 2. Knowledge learning objectives  Describe the role of data demand and use (DDU) in a functional M&E system  Define DDU  Understand the context of data use  Identify barriers to data use and develop recommendations to overcome them  Identify locally relevant programmatic questions and link those questions to available data  Understand types of descriptive data analysis Skills learning objectives  Use tools and strategies to facilitate data use during decision making Workshop Objectives
  3. 3.  Framework for linking data with action  Action plan for overcoming barriers to data use and improving information flow  Three commitments to improve data use in your job Workshop Outputs
  4. 4. Session 1: Monitoring and Evaluation Concepts
  5. 5. Session 1: Learning Objectives Knowledge learning objective:  Describe the role of data demand and use in a functional monitoring and evaluation (M&E) system
  6. 6. DDU’s Role in a Functional M&E System
  7. 7. Strategic Information  Are we making progress towards or are we achieving our goals, objectives, and targets?  How does service delivery vary across facilities?  What are we doing well and how are we under-performing?  Can we improve service coverage with existing resources?  How should we allocate resources?  Do we need to advocate for additional resources or for changes to policies and programmes?
  8. 8. Review of Basic M&E Concepts  Goals  Objectives need to be smart • Specific • Measureable • Achievable • Realistic • Time bound
  9. 9. Results Chain Inputs Activities Outputs Outcomes Impacts Processes Results Objectives Goal(s)
  10. 10. Source: UNAIDS RST-ESA, 2010 Results Chain
  11. 11. Results Framework Result The outputs, outcomes, impacts identified in the results chain, that is, what the programme plans to achieve at different result levels. Result level  Outputs are the products, like the number of people enrolled on antiretroviral therapy (ART) during the past 12 months.  Outcomes reflect the programme’s objectives to change knowledge, attitudes, behaviour, and biomedical markers like viral load and service coverage.  Impacts align with the goal(s) of the programme and describe changes in morbidity and mortality, typically at the national level. Indicator An indicator is a quantitative or qualitative variable that provides a valid and reliable way to measure achievement, assess performance, or reflect changes connected to an intervention. Single indicators are limited in their utility for understanding programme effects and must be collected and interpreted as part of a set of indicators. Baseline The first measurement of an indicator that measures the current condition against which we can track future changes. Target Targets state the desired level of performance the program wants to achieve within a specified period. Each indicator needs a target. Setting targets understanding what can be realistically achieved with available resources.
  12. 12. Indicator  Measures an aspect of a program’s performance  Measures changes over a period of time • # of new family planning users • # of clients currently on ART  Expressed as a number or percentage
  13. 13. Baselines Result The outputs, outcomes, impacts identified in the results chain, that is, what the programme plans to achieve at different result levels. Result level  Outputs are the products, like the number of people enrolled on ART during the past 12 months.  Outcomes reflect the programme’s objectives to change knowledge, attitudes, behaviour, and biomedical markers like viral load and service coverage.  Impacts align with the goal(s) of the programme and describe changes in morbidity and mortality, typically at the national level. Indicator An indicator is a quantitative or qualitative variable that provides a valid and reliable way to measure achievement, assess performance, or reflect changes connected to an intervention. Single indicators are limited in their utility for understanding programme effects and must be collected and interpreted as part of a set of indicators. Baseline The first measurement of an indicator that measures the current condition against which we can track future changes. Target Targets state the desired level of performance the program wants to achieve within a specified period. Each indicator needs a target. Setting targets understanding what can be realistically achieved with available resources.
  14. 14.  What are the sources of data?  What are the data collection methods?  Who collects the data?  How often are data collected?  Who analyses these data?  Who reports these data?  Who uses these data? Establishing a Baseline
  15. 15. Targets Result The outputs, outcomes, impacts identified in the results chain, that is, what the programme plans to achieve at different result levels. Result level  Outputs are the products, like the number of people enrolled on ART during the past 12 months.  Outcomes reflect the programme’s objectives to change knowledge, attitudes, behaviour, and biomedical markers like viral load and service coverage.  Impacts align with the goal(s) of the programme and describe changes in morbidity and mortality, typically at the national level. Indicator An indicator is a quantitative or qualitative variable that provides a valid and reliable way to measure achievement, assess performance, or reflect changes connected to an intervention. Single indicators are limited in their utility for understanding programme effects and must be collected and interpreted as part of a set of indicators. Baseline The first measurement of an indicator that measures the current condition against which we can track future changes. Target Targets state the desired level of performance the program wants to achieve within a specified period. Each indicator needs a target. Setting targets understanding what can be realistically achieved with available resources.
  16. 16. Calculating Targets Things to consider  Existing capacity of staff and facilities  Staff turn-over  Budget allocations through the Ministry of Health (MOH)  Other funding streams like bilateral and multilateral donors
  17. 17. Learning Activity #1 Setting targets
  18. 18. Learning Activity #2 Session 1 review
  19. 19. Session 2: DDU Concepts
  20. 20. Knowledge learning objective:  Define what data demand and use is Session 2: Learning Objectives
  21. 21. Data use  Create or revise a program or strategic plan  Develop or revise a policy  Advocate for a policy or program  Allocate resources  Monitor a program Data Demand and Use Defined Data demand  Decision-makers specify what kind of information they need to inform the decision-making process and then seek it out
  22. 22. Data-Informed Decision-Making Cycle
  23. 23. Why Improve DDU?
  24. 24. Learning Activity 3: Session 2 review
  25. 25. Session 3: Context of Decision Making
  26. 26. Knowledge learning objective:  Understand the context of data use Session 3: Learning Objective
  27. 27. Data-Informed Decision Making Not an event, but a process Data x Stakeholders x Decisions
  28. 28. “We are always giving patient forms and data to our M&E Unit, who then gives data to donors and the government. I am the head doctor and I never have the chance to look through the data before they go up. We just keep giving data up and up, and we never hear back about it…” Head of ART facility, Nigeria
  29. 29.  Census  Vital events data  Surveillance data  Household surveys  Facility-level service statistics  Financial and management information  Modeling, estimates, and projections  Health research Data and Information
  30. 30.  Paves path between data collectors and users  Leads to greater appreciation of data  Important element of management and supervision Working toward a Culture of Data Use
  31. 31. Data x Stakeholders x Decisions Data-Informed Decision Making Not an event, but a process
  32. 32.  View activities from different perspectives  Have different degrees of understanding  Need/want different information  Need information at different levels of complexity  Have different intensities of interest  Have different roles in the decision-making process Knowing Your Stakeholders Why is it important?
  33. 33. Learning Activity 4 Stakeholder analysis matrix Stakeholder Description Role Knowledgelevel Commitment Resources Constraints Government Sector Political Sector Commercial Sector Source: MEASURE Evaluation, 2011
  34. 34. Involving Stakeholders in the Data Use Process Relevance of data Ownership of data Appropriate dissemination of data Use of data
  35. 35. Data-Informed Decision Making Not an event, but a process Data x Stakeholders x Decisions
  36. 36.  Program design and evaluation  Program management and improvement  Strategic planning  Advocacy and policy development Decision Areas
  37. 37. Learning Activity 5 Session 3 review
  38. 38. Session 4: 7 Steps for Using Information
  39. 39. Knowledge learning objectives:  Define data demand and use  Identify barriers to data use and develop recommendations to overcome them  Identify locally relevant programmatic questions and link those questions to available data Session 4 Learning Objective
  40. 40. No Need to Fish Photo by Bread for the World
  41. 41. 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators 7 Steps to Use Information
  42. 42.  Should we expand care and treatment programs in Njombe province?  Am I meeting the HIV prevention needs of my key populations?  Are men and women seeking HIV care equally?  What is the quality of my HIV prevention service? Step 1: Identify Questions Examples
  43. 43. Should we expand care and treatment programs in Njombe province?  What is the burden of HIV in the province over time?  How many facilities are offering counseling and testing (C&T) in the region?  How many people are being seen at each facility?  Are there key facilities that have capacity to expand C&T to reach gaps? Step 2: Prioritize Questions
  44. 44.  Demographic and health surveys (DHS), AIDS indicator surveys (AIS), etc.  Routine health information system (RHIS)  External data sources: behavioral surveillance systems, mapping exercises, surveys  Projections Step 3: Identify Data Sources
  45. 45. Step 4: Transform Data into Information Source: Key Findings of the 2013 Sierra Leone Demographic & Health Survey, page 9
  46. 46. Step 5: Interpret Information and Draw Conclusions Relevance of finding Reasons for finding Consider other data Conduct further research
  47. 47.  Gather additional information  Prioritize Njombe in national expansion strategy  Expand voluntary counseling and testing (VCT) and C&T in high priority facilities  Current VCT facilities do outreach to hotspots  Create referral protocol to ensure those tested are entering into C&T Step 6: Craft Solutions and Take Action
  48. 48.  Monitor implementation of action plan  Consider frequency and duration of monitoring  Develop tool for monitoring Step 7: Continue to Monitor Key Indicators
  49. 49. Learning Activity 6 Session 4 review
  50. 50. Session 5: Linking Data to Action
  51. 51. Knowledge learning objective  Identify locally relevant programmatic questions and link those questions to available data Skills learning objective  Use tools and strategies to facilitate data use during decision making Session 5: Learning Objectives
  52. 52. 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators 7 Steps to Use Information
  53. 53. Questions  Are we implementing the program as planned?  Are we fully meeting the needs of our clients?  What additional resources are needed to meet the needs of our clients?  Are our interventions making a difference? Questions Lead to Decisions Decisions  Allocation of resources across districts/community-based organizations (CBOs)  Revising program approaches to emphasize results  Addition of outreach education programs to most- at-risk populations  Advocacy for new program areas
  54. 54.  Brainstorming about what different staff are interested in knowing  Participatory discussion of indicators/data — desire to know more  Gathering feedback from clients  Preparing for upcoming decisions that have to be made or planning exercises  External factors — donors questions How Do I Identify Questions of Interest?
  55. 55. 7 Steps to Use Information 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators
  56. 56.  Relevance  Answerable  Actionable  Timeliness Criteria for Prioritizing Questions
  57. 57. 7 Steps to Use Information 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators
  58. 58.  Creates a time-bound plan for data-informed decision making  Encourages greater use of existing information/generated demand for data  Monitors the use of information in decision making Framework for Linking Data with Action
  59. 59. Learning Activity 7: Framework for linking data with action Question Data source/ indicator Person responsible (analysis/ research) Timeline Communication channel Possible decision/ action
  60. 60. Learning Activity 8: Session 5 review
  61. 61. Session 6: Transforming Data into Information
  62. 62. 7 Steps to Use Information 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators
  63. 63. Knowledge learning objective  Understand types of descriptive data analysis Session 6: Learning Objective
  64. 64. Descriptive Analysis  Turns raw data into useful information  Provides answers to questions  Describes the sample/target population (demographic and clinic characteristics)  Does not define causality – tells you what, not why
  65. 65.  Question: Is my program meeting its objectives?  Analysis: Compare program targets and actual program performance to learn how far you are from target  Interpretation: Why you have, or have not, achieved the target and what this means for your program Descriptive Analysis Answers Programmatic questions
  66. 66.  Service delivery descriptive analysis  Service coverage analysis  Unit cost analysis  Trend analysis Types of Descriptive Analyses Using Routine Data
  67. 67. Service Delivery Descriptive Analysis Selection of ARV Indicators Private sector Public health system Civil society # of clients who have refilled their prescriptions on-time at a fee-for-service pharmacy √ # of clients on ARVs who have been admitted into a care facility for end-of-life support in the past six months √ √ √ # of clients on ARVs who have been counselled on ARV adherence in the past three months √ √ √ # of clients who have had CD4 and VL tested in the past six months √ √ √ # of clients enrolled in ARV programme √ √ # of client eligible for ARVs √ √ √ # of clients who have refilled their prescriptions on time at a subsidized pharmacy √ √ # of CD4 test kits procured through bilateral development partners and distributed by NGOs and INGOs to public-sector ARV clinics √ √
  68. 68. Service Coverage Analysis  Supply of services data  Need for HIV services  Calculate: Service coverage = Supply of services Demand for services
  69. 69. Service Coverage Analysis Service coverage = # of individuals who have received the services # of individuals in need of the service Service coverage = 467 individuals have received HIV testing and counselling (HTC) services 1,000 individuals in need of HTC services = 47%
  70. 70. Unit Cost Analysis What is the unit cost of delivering antenatal care (ANC) services through an existing mobile health unit that provides maternal and child health (MCH) services? Item Amount Fixed costs per month Overhead SLL 7.8M Personnel SLL 30.5M Variable costs per month Commodities SLL 7.8M Other SLL 15M # MCH clients/month 100 # MCH clients/month receiving ANC services 23 Cost data adapted from 10.1371/journal.pone.0119236
  71. 71.  Observe the overall pattern of change  Reveal patterns due to seasonal variations  Identify outliers and high/low performance in certain time periods  Compare the effect before or after an event  Produce projections to help with programme planning and target setting Descriptive Statistics Trend analysis
  72. 72. Trend Analysis Comparing time periods and sites Source: 2016 Sierra Leone Malaria Indicator Survey Full Report, page 51
  73. 73.  Display trends, relationships, and comparisons  Simple and self-explanatory Charts and Graphs
  74. 74. Column Charts Source: Key Findings of the 2013 Sierra Leone Demographic & Health Survey, page 8
  75. 75. Stacked Bar Chart Source: Key findings of the 2013 Sierra Leone Demographic & Health Survey, page 10
  76. 76. Line Graph Source: 2016 Sierra Leone Malaria Indicator Survey Full Report, page 33
  77. 77. Pie Chart Source: Key Findings of the 2013 Sierra Leone Demographic & Health Survey, page 11
  78. 78. Learning Activity 9: Trend analysis
  79. 79. Session 6 Review: Interpreting Data Relevance of finding Reasons for finding Consider other data Conduct research  Have we met our target?  How does performance compare to other time periods? Facilities?  What could explain the observation?  Talk to beneficiaries and community leaders  Ask program managers  Analyze other data sources • Do they confirm or reject your finding? o Why?  If there are no data, need for research  Methodology depends on question and available resources
  80. 80. Learning Activity 10: Session 6 review
  81. 81. Session 7: Craft Solutions, Take Action, and Monitor Progress
  82. 82. 7 Steps to Use Information 1. Identify questions of interest 2. Prioritize key questions of interest 3. Identify data needs and potential sources 4. Transform data into information 5. Interpret information and assess findings 6. Craft solutions and take action 7. Continue to monitor key indicators
  83. 83. Knowledge learning objective  Identify barriers to data use and develop recommendations to overcome them Skills learning objective  Use tools and strategies to facilitate data use during decision making Learning Objectives
  84. 84. What Determines DDU? * Based on PRISM analytical framework (LaFond, Fields et al. (2005). The PRISM: An analytical framework for understanding performance of health information systems in developing countries. MEASURE Evaluation). Organizational Technical Behavioral
  85. 85. Organizational Technical Behavioral Politics Culture Society What Determines DDU?
  86. 86. Technical  Have you ever had an experience while making a policy or program-related decision when you were concerned about the quality of the information being used? Behavioral  What specific challenges have you experienced among your staff when it comes to using data? Organizational  How does your organization support having the necessary information to make decisions? Learning Activity 10: Identifying barriers to data use: discussion questions
  87. 87. Learning Activity 11: Developing an action plan to address barriers to data use Barrier Action Steps Responsible person Timeline Indicators 1. 1a. 1b. 1c 1a. 1b. 1c. 1a. 1b. 1c. 1a. 1b. 1c. 2. 2a. 2b. 2c. 2a. 2b. 2c. 2a. 2b. 2c. 2a. 2b. 2c.
  88. 88.  Plan, plan, plan!!!  Engage key stakeholders to fully understand: • Decisions they make • Information they need • Appropriate dissemination Building Data Use into Your Work
  89. 89.  Institute regular meetings  Document and communicate data use successes Practical Steps to Integrate DDU  Institutionalize tools (ex., framework for linking data with action)
  90. 90.  Identify a group leader  Identify 2–3 things your group can do to facilitate data use within the first two months of returning back to work, based on your action plan  Small group discussion: 20 mins  Group presentations: 15 mins Make Commitments
  91. 91. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L- 14-00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org

×