Bringing data users and producers together, applying tools to improve data use, and training health professionals in data-informed decision making cannot be sustained unless mechanisms are put in place to ensure that host organizations are supportive and committed to continued data use.
Recording: http://universityofnc.adobeconnect.com/p9k8b5x0d9g/
Strengthening an Organization’s Capacity to Demand and Use Data
1. Data Demand & Use:
Strengthening and Organization’s
Capacity to Use Data
Webinar Series #8
Tuesday, March 6, 2012
Presenters: Tara Nutley, Nicole Judice and
Dauda Sulaiman Dauda
2. Troubleshooting
If you lose connectivity
Re-enter the meeting room by clicking on the webinar link
provided
If you have trouble with audio
Refer to the conference call instructions in the upper right
hand corner
Send an email to leah.gordon@unc.edu
3. Tips for Participating in the
Discussion
To comment, raise your hand by clicking on the
icon.
Speak into your microphone (be sure it is enabled by clicking
on the icon at the top of the screen).
Type questions in Q&A window located at the top of your
screen.
A recording of the webinar will be made available
at www.measureevaluation.org/ddu
4. Agenda
• Welcome - webinar tips
• Brief overview of Data Demand and Use
• Presentation of Tools
• Field Application of Tools
• Questions and Answers
• Wrap up
6. “… without information, things are done
arbitrarily and one becomes unsure of
whether a policy or program will fail or
succeed. If we allow our policies to be guided
by empirical facts and data, there will be a
noticeable change in the impact of what we
do.”
National-level Policymaker, Nigeria
7. Definitions
Data use – Using data in the decision making
process
monitor a program
create or revise a program or strategic plan
develop or revise a policy
advocate for a policy or program
allocate resources
Data Demand - decision makers specify what kind of
information they want & seek it out
9. Improving Data-informed Decision
Making
Data Users & Data Producers
Tool Application Capacity Building Organizational Support
Monitoring & Evaluation System Improvements
10. Rapid Assessment
Stakeholder Engagement Tool
Assessment of Data Use Constraints
Information Use Mapping
Framework for Linking Data with Action
Observation of management meetings and
processes
Discussions with SI Team and others
Site visits
11. Capacity Building
Global DDU Training Resources (curricula)
Data informed decision making – key concepts
Data analysis, interpretation, presentation,
communication and use
Providing feedback and sharing information
Developing action plans
Identifying commitments to strengthening data use
Tools to facilitate data use
12. Organizational Support
Mentoring
Strengthening feedback mechanisms
Data use supportive supervision checklists
Institutionalizing the Information Use Map and
Framework for Linking Data to Action
Documenting data use
14. PEPFAR Service Delivery Project
Large amount of data collected, feeding NNRIMS
Data were not being used effectively at sites or
within project
Pervasive mistrust of data
Lack of understanding of how RHIS data could
be used
Lack of understanding of how key indicators
calculated and use for program improvement
15. Approach with Service Delivery
Project
Data Users & Data Producers
Rapid Assessment Capacity Building Organizational Support
Monitoring & Evaluation System Improvements
16. Rapid Assessment
Phase I: Pre-workshop assessment
Discussion with AIDSRelief SI Team
Observation of “Dashboard Meeting”
Information Use Map
Two site visits
Phase II: Self-assessment
Small group discussions using the Assessment of
Data Use Constraints and Framework for Linking
Data with Action
17. Capacity Building
Three workshops with data producers and data
users:
Central AIDSRelief/Nigeria staff
Facility-based M&E Officers
Facility-based Program Managers (“Program
Coordinators”)
18. Capacity Building
DDU Global Training Resources:
Small group discussion using Assessment of Data
Use Constraints and develop action plan;
Discussion of importance of M&E in program
management;
Framework for Linking Data with Action;
Hands-on exercises in analysis and interpretation;
Addressing feedback mechanisms;
Full explanation of performance indicators.
19. Organizational Support
Recommendations provided to central office and
SI Team
Collaborative approach to supervision and
feedback
DDU Supportive Supervision Checklist
20. Evaluation
Six and 12 month evaluations conducted on-line
with workshop participants
Surveys were based on retrospective reporting
about improvements compared to before the
workshop
21. Improvements in Data Use Since
the Training
95%
100% 85% 86%
76%
80%
55%
60%
40%
20%
0%
Presented data Assisted Aware of LPTF Implemented Explain
graphically decision makers using info from solutions to indicators
with MR identified relevance to
interpretation barriers service delivery
22. Success Story
“I presented a report which showed a high number
of clients lost to follow up, and told the facility
managers that this shows we are not actually doing
home visits appropriately to track clients. I asked
for permission to coordinate home visit activities,
and now there is an active home visit team in the
facility.”
M&E Officer, ART Facility
Nigeria
23. Project commitment to data use
AIDSRelief fully funded the Capacity Building
and Training strategy
New project leadership promotes SI and use of
data for program improvement
Atmosphere and communication at management
meetings are more productive and collaborative
24. Organizational changes to
improve data use
Specific staff member is responsible for DDU
Quarterly M&E Forum
Feedback provided to sites and reviewed in PMT
meetings
Service delivery improvements
Central team meetings – discuss data and
exchange best practices
Clinical advisors routinely request data
26. MEASURE Evaluation DDU Resources
www.measureevaluation.org/ddu
Data Demand and Use Tool Kit
Data Demand and Use Training Resources
27. Join Data Use Net
Send an email to listserv@unc.edu. Leave the
subject field blank and in the body of the message
type „subscribe DataUseNet.‟ For example:
To: listserv@unc.edu
From: youremail@youremail.com
Subject:
Subscribe Data Use Net
29. MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) and implemented by the
Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF International, John Snow, Inc., Management Sciences for
Health, and Tulane University. Views expressed in this
presentation do not necessarily reflect the views of USAID or the
U.S. government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide
Editor's Notes
We are all aware of the challenges involved in providing quality health services in the contexts where we work. In many countries health programs are facing a high disease burden, a growing population, inadequate numbers and poor distribution of qualified health workers, and inadequate health systems to support the distribution of services. It is in this situation that it becomes extremely important for to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on quality data and information is urgent.
The importance of data-informed decision making is expressed on this slide by a national-level policymaker in Nigeria who participated in a data use assessment conducted by MEASURE Evaluation. The assessment involved interviews with a range of professionals at the national, regional, and facility levels. The policymaker interviewed, stated… (READ SLIDE)“… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” This statement nicely summarizes why we are here today to discuss the importance of improving data-informed decision making.
Not reporting or disseminationREVIEWING & DISCUSSING
When we talk about improving the use of and demand for data in decision making we talk about it as a cycle – not a one-time event. The idea of a cycle of evidence-based decision making is the framework on the slide. It starts with basic M&E systems and the collection of information – including ensuring that the information is available and in a format that is easily understood by relevant stakeholders so that the information can be interpreted and used to improve policies and programs. The cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. This leads to repeated data use. You will note that this cycle is supported by coordination and collaboration. This coordination is among data users and data producers as well as between management systems and other organizational supports that facilitate and support data informed decision making. Lastly, the cycle is supported by improving capacity to ensure that individuals are equipped with the skills to collect and use data. All of these supports are critical to ensure that the cycle continues functioning to create a culture of data use. Yet, we all know that cycles that rely on multiple inputs, activities and systems to function effectively – often don’t. In the best designed M&E systems you often find lackluster data use. Data is not being used as often as it should be.
How do we improve DDU?Firstly, build upon a commitment and ongoing efforts to improve M&E and information systems – this is the foundation of all data use improvement interventions.Identify and engaging data users and data producers is also critical. By data users we are referring to those whose primary function is to manage data systems and by data users we are referring to those whose primary function is to use data to monitor and improve health service delivery. These two groups don’t always work closely together. For data use to function as we saw on the previous slide, regular collaboration between these two groups is critical. It is also important to apply tools, build capacity and strengthen organizational systems to support data informed decision making. In this webinar series we will be discussing tool application (the pink box) and the types of tools MEASURE Evaluation has developed to facilitate DDU. The last webinar session of this series will address capacity building and at a later date we will offer a webinar on strengthening organizational supports to improve data demand and sue. The combination of tool application, capacity building and strengthening organizations are all complimentary and necessary elements of any strategy to improve the use of data in decision making. The DDU approach is built upon M3’s ongoing work to strengthen and improve data systems. The approach is overlayed on other M&E improvements – what M3 is already doing in a country.The approach consists of three elements: Rapid assessment, capacity building and organizational support. But sometimes it’s necessary to re-assess capacity needs and thus this process isn’t linear. Organizational support plan that we iplement supported by DDU tools.Finally an essential element of the appraoch at all stages is the engagement of data users and data producers.
Note curricula that exist around these topicsPackaging and refining
Seeking an opportunity to use our curriculum and tools at the facility level and with a service delivery project.
Collaborating with AIDSRelief on trainings, etc., and they informed us that…
Using tools in two different contexts – in independent assessments as well as self-assessments, talk about adaptationsPhase II: Conducted during workshopUsed Assessment of Data Use Constraints as a discussion guide in small groups
Global DDU training resources adapted to the project and needs of trainee groups
The following recommendations were presented to the full AIDSRelief/Nigeria team the next day: Currently the SI Team and senior leadership do most of the talking and reporting at the meetings. While it may take more time out of the meeting, it would be more effective for program managers to present the data for their own sites. This would require the program manager to read the dashboard and understand it before presenting the information. It would also allow the program manager the opportunity to comment on the particular needs/issues at the site. This will build accountability and ownership among program managers of their sites and is more likely to result in the program manager using data regularly in facility supervision/mgmt. SI Team develop standard reporting package to deliver quarterly along with Dashboard; Share lessons learned from other facilities to address issues – Dashboard an opportunity. Sharing lessons learned can demonstrate that facilities overcame problems, and can help motivate facilities by seeing what others are achieving. For instance, what have been facilitating factors to reaching a 10% pediatric target? Can other facilities take lessons? Present the information graphically when possible with an accompanying table; For instance, when presenting the monthly report (a very large table that is hard to read on the projector), it may be more effective to show graphs of the data on the projector and provide the full tables as handouts
Total - 21 respondents out of 35 trained (60% response rate)1) Question: Since attending the workshop, are you familiar with any instances of staff at your LPTF using information provided to them through the monthly reports or other information requests?85% or 17 out of 20 said yes2) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you been able to explain to the LPTF staff the meaning of indicators and their relevance to the delivery of services?20 out of 21 said yes (95%)3) Question part a: As a result of your participation in the M&E Forum Data Use Training workshop, have you been able to identify barriers to data use in your LPTF?17 out of 21 identified barriers (81%)Question part b: If yes, were you able to identify solutions to overcome the barriers to data use?14 out of 17 identified solutions (82%)Question part c: If yes, were you able to implement solutions to overcome the barriers to data use in your LPTF?12 out of 14 implemented solutions (86%)4) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you assisted decision makers with data interpretation?16 out of 21 (76%) said yes5) Question: As a result of your participation in the M&E Forum Data Use Training workshop, have you presented information in different ways (graphically, etc.)?11 out of 20 said yes (55%)
One of the success stories documented through the online survey
Forthcoming step in evaluation plan is on organizational element
Specific staff member for DDIU – analyzes information – describe other tasksInvite M&E staff together from facilities and from agencies to review data, discuss best practices – quarterly M&E forumEach site – officer assigned to site. When they send report, we capture important parts, put them into graphs, feedback template that we send back to them. Indicators around quality – different indicators have different threshholds of success or quality. When an indicator shows up in red, it means that it’s something that requires attention. Includes performance graphs, so that they can look at performance over time. Shows how they are performing on those indicators as well. At the sites, they would have the PMT and the M&E officer uses feedback reports at those meetings to show them how they are doing and what they need to address. Make decisions on how to improve and what they need to look at. EXAMPLE. Never had an inkling that they had a problem, but because of these reports, the constant red, they realized it was an issue…had to think about how to get people on cotrimoxazle prophylaxis. Another facility there was a problem with home visits and because they saw red on the performance indicators, they realized it was a problem with home visits – they didn’t have a system.Always used reports from sites when they have the CTCT meeting from IHV, CRS, SI. Have trends…DDU officer picks out what’s going well and not going well and use that to discuss. IHV requests information and ask for detailed reports on things that they don’t routinely report on. Now at CTCT meeting – discuss best practices.