Strengthening an Organization’s Capacity to Demand and Use Data
Bringing data users and producers together, applying tools to improve data use, and training health professionals in data-informed decision making cannot be sustained unless mechanisms are put in place to ensure that host organizations are supportive and committed to continued data use.
We are all aware of the challenges involved in providing quality health services in the contexts where we work. In many countries health programs are facing a high disease burden, a growing population, inadequate numbers and poor distribution of qualified health workers, and inadequate health systems to support the distribution of services. It is in this situation that it becomes extremely important for to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on quality data and information is urgent.
The importance of data-informed decision making is expressed on this slide by a national-level policymaker in Nigeria who participated in a data use assessment conducted by MEASURE Evaluation. The assessment involved interviews with a range of professionals at the national, regional, and facility levels. The policymaker interviewed, stated… (READ SLIDE)“… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” This statement nicely summarizes why we are here today to discuss the importance of improving data-informed decision making.
Not reporting or disseminationREVIEWING & DISCUSSING
When we talk about improving the use of and demand for data in decision making we talk about it as a cycle – not a one-time event. The idea of a cycle of evidence-based decision making is the framework on the slide. It starts with basic M&E systems and the collection of information – including ensuring that the information is available and in a format that is easily understood by relevant stakeholders so that the information can be interpreted and used to improve policies and programs. The cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. This leads to repeated data use. You will note that this cycle is supported by coordination and collaboration. This coordination is among data users and data producers as well as between management systems and other organizational supports that facilitate and support data informed decision making. Lastly, the cycle is supported by improving capacity to ensure that individuals are equipped with the skills to collect and use data. All of these supports are critical to ensure that the cycle continues functioning to create a culture of data use. Yet, we all know that cycles that rely on multiple inputs, activities and systems to function effectively – often don’t. In the best designed M&E systems you often find lackluster data use. Data is not being used as often as it should be.
How do we improve DDU?Firstly, build upon a commitment and ongoing efforts to improve M&E and information systems – this is the foundation of all data use improvement interventions.Identify and engaging data users and data producers is also critical. By data users we are referring to those whose primary function is to manage data systems and by data users we are referring to those whose primary function is to use data to monitor and improve health service delivery. These two groups don’t always work closely together. For data use to function as we saw on the previous slide, regular collaboration between these two groups is critical. It is also important to apply tools, build capacity and strengthen organizational systems to support data informed decision making. In this webinar series we will be discussing tool application (the pink box) and the types of tools MEASURE Evaluation has developed to facilitate DDU. The last webinar session of this series will address capacity building and at a later date we will offer a webinar on strengthening organizational supports to improve data demand and sue. The combination of tool application, capacity building and strengthening organizations are all complimentary and necessary elements of any strategy to improve the use of data in decision making. The DDU approach is built upon M3’s ongoing work to strengthen and improve data systems. The approach is overlayed on other M&E improvements – what M3 is already doing in a country.The approach consists of three elements: Rapid assessment, capacity building and organizational support. But sometimes it’s necessary to re-assess capacity needs and thus this process isn’t linear. Organizational support plan that we iplement supported by DDU tools.Finally an essential element of the appraoch at all stages is the engagement of data users and data producers.
Note curricula that exist around these topicsPackaging and refining
Seeking an opportunity to use our curriculum and tools at the facility level and with a service delivery project.
Collaborating with AIDSRelief on trainings, etc., and they informed us that…
Using tools in two different contexts – in independent assessments as well as self-assessments, talk about adaptationsPhase II: Conducted during workshopUsed Assessment of Data Use Constraints as a discussion guide in small groups
Global DDU training resources adapted to the project and needs of trainee groups
The following recommendations were presented to the full AIDSRelief/Nigeria team the next day: Currently the SI Team and senior leadership do most of the talking and reporting at the meetings. While it may take more time out of the meeting, it would be more effective for program managers to present the data for their own sites. This would require the program manager to read the dashboard and understand it before presenting the information. It would also allow the program manager the opportunity to comment on the particular needs/issues at the site. This will build accountability and ownership among program managers of their sites and is more likely to result in the program manager using data regularly in facility supervision/mgmt. SI Team develop standard reporting package to deliver quarterly along with Dashboard; Share lessons learned from other facilities to address issues – Dashboard an opportunity. Sharing lessons learned can demonstrate that facilities overcame problems, and can help motivate facilities by seeing what others are achieving. For instance, what have been facilitating factors to reaching a 10% pediatric target? Can other facilities take lessons? Present the information graphically when possible with an accompanying table; For instance, when presenting the monthly report (a very large table that is hard to read on the projector), it may be more effective to show graphs of the data on the projector and provide the full tables as handouts
Total - 21 respondents out of 35 trained (60% response rate)1) Question: Since attending the workshop, are you familiar with any instances of staff at your LPTF using information provided to them through the monthly reports or other information requests?85% or 17 out of 20 said yes2) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you been able to explain to the LPTF staff the meaning of indicators and their relevance to the delivery of services?20 out of 21 said yes (95%)3) Question part a: As a result of your participation in the M&E Forum Data Use Training workshop, have you been able to identify barriers to data use in your LPTF?17 out of 21 identified barriers (81%)Question part b: If yes, were you able to identify solutions to overcome the barriers to data use?14 out of 17 identified solutions (82%)Question part c: If yes, were you able to implement solutions to overcome the barriers to data use in your LPTF?12 out of 14 implemented solutions (86%)4) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you assisted decision makers with data interpretation?16 out of 21 (76%) said yes5) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you presented information in different ways (graphically, etc.)?11 out of 20 said yes (55%)
One of the success stories documented through the online survey
Forthcoming step in evaluation plan is on organizational element
Specific staff member for DDIU – analyzes information – describe other tasksInvite M&E staff together from facilities and from agencies to review data, discuss best practices – quarterly M&E forumEach site – officer assigned to site. When they send report, we capture important parts, put them into graphs, feedback template that we send back to them. Indicators around quality – different indicators have different threshholds of success or quality. When an indicator shows up in red, it means that it’s something that requires attention. Includes performance graphs, so that they can look at performance over time. Shows how they are performing on those indicators as well. At the sites, they would have the PMT and the M&E officer uses feedback reports at those meetings to show them how they are doing and what they need to address. Make decisions on how to improve and what they need to look at. EXAMPLE. Never had an inkling that they had a problem, but because of these reports, the constant red, they realized it was an issue…had to think about how to get people on cotrimoxazle prophylaxis. Another facility there was a problem with home visits and because they saw red on the performance indicators, they realized it was a problem with home visits – they didn’t have a system.Always used reports from sites when they have the CTCT meeting from IHV, CRS, SI. Have trends…DDU officer picks out what’s going well and not going well and use that to discuss. IHV requests information and ask for detailed reports on things that they don’t routinely report on. Now at CTCT meeting – discuss best practices.
Strengthening an Organization’s Capacity to Demand and Use Data
Data Demand & Use:Strengthening and Organization’s Capacity to Use Data Webinar Series #8 Tuesday, March 6, 2012 Presenters: Tara Nutley, Nicole Judice and Dauda Sulaiman Dauda
Troubleshooting If you lose connectivity Re-enter the meeting room by clicking on the webinar link provided If you have trouble with audio Refer to the conference call instructions in the upper right hand corner Send an email to firstname.lastname@example.org
Tips for Participating in theDiscussion To comment, raise your hand by clicking on the icon. Speak into your microphone (be sure it is enabled by clicking on the icon at the top of the screen). Type questions in Q&A window located at the top of your screen. A recording of the webinar will be made available at www.measureevaluation.org/ddu
Agenda• Welcome - webinar tips• Brief overview of Data Demand and Use• Presentation of Tools• Field Application of Tools• Questions and Answers• Wrap up
Why improve data-informeddecision making? Pressing need to develop health policies, strategies, and interventions
“… without information, things are donearbitrarily and one becomes unsure ofwhether a policy or program will fail orsucceed. If we allow our policies to be guidedby empirical facts and data, there will be anoticeable change in the impact of what wedo.” National-level Policymaker, Nigeria
Definitions Data use – Using data in the decision making process monitor a program create or revise a program or strategic plan develop or revise a policy advocate for a policy or program allocate resources Data Demand - decision makers specify what kind of information they want & seek it out
Improving Data-informed Decision Making Data Users & Data ProducersTool Application Capacity Building Organizational Support Monitoring & Evaluation System Improvements
Rapid Assessment Stakeholder Engagement Tool Assessment of Data Use Constraints Information Use Mapping Framework for Linking Data with Action Observation of management meetings and processes Discussions with SI Team and others Site visits
Capacity Building Global DDU Training Resources (curricula) Data informed decision making – key concepts Data analysis, interpretation, presentation, communication and use Providing feedback and sharing information Developing action plans Identifying commitments to strengthening data use Tools to facilitate data use
Organizational Support Mentoring Strengthening feedback mechanisms Data use supportive supervision checklists Institutionalizing the Information Use Map and Framework for Linking Data to Action Documenting data use
PEPFAR ServiceDelivery ProjectStrengthens Data Use
PEPFAR Service Delivery Project Large amount of data collected, feeding NNRIMS Data were not being used effectively at sites or within project Pervasive mistrust of data Lack of understanding of how RHIS data could be used Lack of understanding of how key indicators calculated and use for program improvement
Approach with Service Delivery Project Data Users & Data ProducersRapid Assessment Capacity Building Organizational Support Monitoring & Evaluation System Improvements
Rapid Assessment Phase I: Pre-workshop assessment Discussion with AIDSRelief SI Team Observation of “Dashboard Meeting” Information Use Map Two site visits Phase II: Self-assessment Small group discussions using the Assessment of Data Use Constraints and Framework for Linking Data with Action
Capacity Building Three workshops with data producers and data users: Central AIDSRelief/Nigeria staff Facility-based M&E Officers Facility-based Program Managers (“Program Coordinators”)
Capacity Building DDU Global Training Resources: Small group discussion using Assessment of Data Use Constraints and develop action plan; Discussion of importance of M&E in program management; Framework for Linking Data with Action; Hands-on exercises in analysis and interpretation; Addressing feedback mechanisms; Full explanation of performance indicators.
Organizational Support Recommendations provided to central office and SI Team Collaborative approach to supervision and feedback DDU Supportive Supervision Checklist
Evaluation Six and 12 month evaluations conducted on-line with workshop participants Surveys were based on retrospective reporting about improvements compared to before the workshop
Improvements in Data Use Since the Training 95%100% 85% 86% 76%80% 55%60%40%20% 0% Presented data Assisted Aware of LPTF Implemented Explain graphically decision makers using info from solutions to indicators with MR identified relevance to interpretation barriers service delivery
Success Story“I presented a report which showed a high numberof clients lost to follow up, and told the facilitymanagers that this shows we are not actually doinghome visits appropriately to track clients. I askedfor permission to coordinate home visit activities,and now there is an active home visit team in thefacility.” M&E Officer, ART Facility Nigeria
Project commitment to data use AIDSRelief fully funded the Capacity Building and Training strategy New project leadership promotes SI and use of data for program improvement Atmosphere and communication at management meetings are more productive and collaborative
Organizational changes toimprove data use Specific staff member is responsible for DDU Quarterly M&E Forum Feedback provided to sites and reviewed in PMT meetings Service delivery improvements Central team meetings – discuss data and exchange best practices Clinical advisors routinely request data
MEASURE Evaluation DDU Resources www.measureevaluation.org/ddu Data Demand and Use Tool Kit Data Demand and Use Training Resources
Join Data Use NetSend an email to email@example.com. Leave thesubject field blank and in the body of the messagetype „subscribe DataUseNet.‟ For example:To: firstname.lastname@example.orgFrom: email@example.comSubject:Subscribe Data Use Net
MEASURE Evaluation is funded by the U.S. Agency forInternational Development (USAID) and implemented by theCarolina Population Center at the University of North Carolinaat Chapel Hill in partnership with Futures Group International,ICF International, John Snow, Inc., Management Sciences forHealth, and Tulane University. Views expressed in thispresentation do not necessarily reflect the views of USAID or theU.S. government.MEASURE Evaluation is the USAID Global Health Bureausprimary vehicle for supporting improvements in monitoring andevaluation in population, health and nutrition worldwide