The 7 Steps to Improve HIV/AIDS Programs Guide presents concrete steps and illustrative examples that can be used to facilitate the use of information as a part of the decision-making processes guiding program design, management and service provision in the health sector.
Tool Download: https://www.cpc.unc.edu/measure/our-work/data-demand-and-use/resolveuid/29590bfbaed536fb7f35c8d76d1596dc
Webinar Recording: http://universityofnc.adobeconnect.com/p9csbqoj19x/
When welcoming folks, thank them for volunteering for the webinar. Introduce presenters and participants
We are all aware of the challenges involved in providing quality health services in the contexts where we work. In many countries health programs are facing a high disease burden, a growing population, inadequate numbers and poor distribution of qualified health workers, and inadequate health systems to support the distribution of services. It is in this situation that it becomes extremely important for to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on quality data and information is urgent.
The importance of data-informed decision making is expressed on this slide by a national-level policymaker in Nigeria who participated in a data use assessment conducted by MEASURE Evaluation. The assessment involved interviews with a range of professionals at the national, regional, and facility levels. The policymaker interviewed, stated… (READ SLIDE) “… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” This statement nicely summarizes why we are here today to discuss the importance of improving data-informed decision making.
Not reporting or dissemination REVIEWING & DISCUSSING
When we talk about improving the use of and demand for data in decision making we talk about it as a cycle – not a one-time event. The idea of a cycle of evidence-based decision making is the framework on the slide. It starts with basic M&E systems and the collection of information – including ensuring that the information is available and in a format that is easily understood by relevant stakeholders so that the information can be interpreted and used to improve policies and programs. The cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. This leads to repeated data use. You will note that this cycle is supported by coordination and collaboration. This coordination is among data users and data producers as well as between management systems and other organizational supports that facilitate and support data informed decision making. Lastly, the cycle is supported by improving capacity to ensure that individuals are equipped with the skills to collect and use data. All of these supports are critical to ensure that the cycle continues functioning to create a culture of data use. Yet, we all know that cycles that rely on multiple inputs, activities and systems to function effectively – often don’t. In the best designed M&E systems you often find lackluster data use. Data is not being used as often as it should be.
How do we improve DDU? Firstly, build upon a commitment and ongoing efforts to improve M&E and information systems – this is the foundation of all data use improvement interventions. Identify and engaging data users and data producers is also critical. By data users we are referring to those whose primary function is to manage data systems and by data users we are referring to those whose primary function is to use data to monitor and improve health service delivery. These two groups don’t always work closely together. For data use to function as we saw on the previous slide, regular collaboration between these two groups is critical. It is also important to apply tools, build capacity and strengthen organizational systems to support data informed decision making. In this webinar series we will be discussing tool application (the pink box) and the types of tools MEASURE Evaluation has developed to facilitate DDU. The last webinar session of this series will address capacity building and at a later date we will offer a webinar on strengthening organizational supports to improve data demand and sue. The combination of tool application, capacity building and strengthening organizations are all complimentary and necessary elements of any strategy to improve the use of data in decision making.
Why should you use the 7 step approach? Because it provides concrete steps that lead to data-informed decision making. The approach encourages more strategic and effective use of data. And finally, it ensures the involvement of both data users and producers. The 7 Step approach provides concrete steps to a process that is often ill defined. Yesterday we talked about the decision making process and the 3 elements of data-informed decision making that needs to be in place for that process to function. Today we will discuss the concrete steps that you can follow once you sit down with your stakeholders and your data to address your decision making needs.
NOTE to facilitator : READ SLIDE
What are the 7 steps? MEASURE Evaluation has identified 7 steps to facilitate use of information. Step 1 – First you identify your questions of interest. Yesterday we talked about decisions. Decisions are choices you will have to make to implement health programs. All decisions are informed by questions. Step 1 involves identifying your questions that will inform your decision. Step 2 – Involves prioritizing your questions to ensure you are spending your limited resources answering a question that has high programmatic relevance. Step 3 – identifies the data needed to answer the question. In some cases you may need to consult multiple, existing data sources. In others, there may be a data gap – and the data you need may not exist. In this case you may consider using proxy data, or if the question is important enough, implement a research effort to gather the required data.
Step 4 – Once the data sources have been obtained, it needs to be transformed into information through analysis. Analysis is the act of turning raw data into more useful information. Depending on the question, analysis can be simple math or more sophisticated computations. The resulting information then needs to be presented in a way that is clearly understandable – via graphs, charts and other visual aids. Step 5 - The terms analysis (steps 4) and interpretation (step 5) are often considered synonymous. We like to separate them into distinct steps. Interpretation is the consideration of the analysis – the potential reasons for the findings – and possible next steps. In this process, we move from the ‘what’ is happening in our programs to the ‘why’ it is happening. Step 6 – Involves discussing and agreeing upon recommendations for change based on your conclusions. And lastly, step 7, involves a continued commitment to the data-informed decision making cycle. If changes are implemented based on the data analysis, it is important to continue to track the outcome of those changes. As discussed in day one, the data informed decision making process is a cycle where successful use of data leads to demand for future data.
With all these data, it may be tempting to “fish out” whatever data are readily available and try to figure out how to use them. This is not the most effective way to strengthen data use. Instead, it is more productive and valuable to answer questions that are of real interest at the facility/program/community/organization level. Vast amounts of data are available to many HIV/AIDS service delivery sites. However to compile, analyze, interpret, and use these data can be a very daunting task that requires both time and skill. Rather than embarking on a fishing expedition, a team of data users and data producers can use its time more efficiently by first identifying and then prioritizing key questions of interest. Available data can then be analyzed in a targeted way to begin to answer these questions. Programmatic questions of interest can be identified by: • Participatory discussions of indicators that demonstrate program success; • Discussions of observed or anecdotal problems that program managers face; • Discussions of upcoming planning decisions that need to be made – and what questions will inform those decisions; • Gathering feedback from clients; and • Assessing external factors, such as audits, program evaluations, and donor’s questions. Any of these methods can generate interesting and useful questions.
You can consider these questions as you brainstorm a list of questions of interest.
After generating a list of potential questions of interest, it is important to prioritize the questions to ensure that you are addressing the most important issues and problems first. To prioritize these questions, a team must consider specific criteria and discuss each question in depth. Programmatic relevance: Is the question programmatically relevant and/or of a public health interest? Are others in the community interested in the information? Answerable: Is it possible to answer this question or measure performance with existing data or data that could be collected? Actionable: Does your organization have the authority to act upon the answers to the key question? That is, if data indicate a need for a change in the current course of action, can your organization make the required changes? If not, can your organization influence those with the authority or ability to effect change? Timeliness of the question: Is there a timeline for answering this question or making a decision about the issue at hand? Can some questions be tabled for discussion later to allow the group time to focus on questions that must be addressed more quickly? All of these questions should be considered before embarking on conducting your analysis.
MEASURE Evaluation has a matrix that can be used to prioritize questions. A copy of this tool, called the Priority Questions Scoring Worksheet, can be found in your packet of Handouts. This matrix is a useful tool to facilitate discussion about each question and to reduce the influence of special interests or agendas. The criteria can be defined differently or new criterion can be added. It’s not necessary to use this tool, but may be helpful in some contexts.
Once the group has prioritized and refined the questions of interest, it is time to bring data into the picture. Finding the answer to a question may require one indicator or it may require the triangulation of several different performance indicators from multiple data sources. The following must be considered in the process of identifying and focusing on specific data needs and sources: • How frequently or at what intervals do we need this information? • Do the data already exist and are they readily available? • Are the data of sufficient quality?
Once specific data sources have been identified and obtained to answer your question of interest (as we saw in Step 3), the data can be transformed into information to facilitate decision making and action. Transforming data into information involves: Isolating the required indicators and data elements; Reviewing and examining data and transforming them into useful information. Depicting data in an image –a graph, chart, table A variety of analysis techniques are available to facilitate decision making. They range from simple to complex. They entail reviewing and examining data and transforming them into useful information, usually a visual image such as a graph, chart, or table. Many potential data users are more attentive to and have a better understanding of numbers when they are presented in a graph or table. For example, some data users find it easier to understand the proportion of a whole through a pie chart rather than through raw data. This helps data users to interpret the meaning behind the data.
The terms and concepts of analysis and interpretation are sometimes considered synonymous and are often combined into one process. In the Seven Steps, these processes are separated into distinct steps (Steps 4 and 5) because analysis can be conducted effectively by one person or by a team of people with different backgrounds, but interpretation is most productive when a group is involved. Step 5 – interpretation and drawing conclusions – is a process by which key stakeholders discuss the meaning of a specific finding and draw conclusions about this information through group discussion. They move from the ‘what’ is happening in a program to understanding ‘why’ it is happening. We will practice interpreting and discussing data in the next session.
Data interpretation is the process of making sense of the information. It allows us to ask: What does this information tell me about the program? Here, you see a flow chart of the steps involved in interpreting data. When interpreting data you want to consider: The relevance of the findings The reasons for the findings Consider other data sources relevant to the findings Conduct further research if needed. We start by wanting to know the relevance of our findings. Seeking the relevance of a finding is to: Add meaning to information by making connections and comparisons as well as explore the causes and consequences. Is there anything that surprises you in the data? Are there any highs and lows in the data? How does the indicator compare to other time periods, other facilities? How does the indicator compare to the target/ideal? How far from the target/ideal is it? Asking these questions will help you to put the data in the context of your program. When seeking potential reasons for the finding, we often will need additional information that will put our findings into the context of the program. Supplementing the findings with expert opinion is a good way to do this. For example, talk to others with knowledge of the program or target population, who have in-depth knowledge about the subject matter, and get their opinions about possible causes. For example, if your data show that you have not met your targets, you may want to know if: The community or target population is aware of the service? Have a sufficient number of awareness campaigns been implemented? To answer your interpretation questions you may need to bring in additional data to look at comparisons and targets. What we mean by this stage is that if additional data is available to verify your conclusions, it is always good to have multiple sources to strengthen credibility. For example, if there was recent qualitative research available that helped to further explain or verify your findings, it would be important to include them in the interpretation. Once you review additional data, it may become apparent that these data are not sufficient to explain the reasons for your findings – that a data gap exists. In these instances, it may be necessary to conduct further research or data collection. The types of research designs that are applied will depend on the questions that need to be answered, and of course will be tempered by the feasibility and expense involved with obtaining the new data.
So, we have now identified our questions, pulled in the relevant data, interpreted these data and developed conclusions based on the data. Step 6 entails convening a meeting with relevant data users and data producers to: • use the conclusions identified in the previous step to brainstorm potential solutions; • further specify, craft, and prioritize these solutions to respond to the problem; and • develop an action plan for implementing each of these solutions.
Teams can use a matrix, such as the one on this slide to document their action plan for implementing a response to address problems identified through the Seven Steps process.
Step 7 entails monitoring of key indicators. If your data use exercise indicated a programmatic change you will need to monitor the effect of that change on the indicator in question. Your program may choose to analyze and interpret data once and take action, or your program or site may need to monitor several indicators over time to develop, test, and validate solutions. The course you choose will depend on a variety of factors, including the size of the program or facility, the nature of the priority questions of interest, and whether or not any problem was highlighted during the process of interpretation (Step 5). Many programs have developed their own framework for improving the quality of their program or service and have designed tools, such as spreadsheets and dashboards, to monitor their efforts at program implementation and program improvement. A basic table or graph can be used to monitor an indicator over time. Is this indicator included in monthly reports? What is your target for satisfactory performance? How often can you reliably monitor the indicator? How long do you expect it to take to improve performance? Who needs to know about progress and improvement in performance? Spreadsheets and dashboards Wall charts and graphs Staff meetings to discuss
Here is an example of a simple monitoring table or spreadsheet. The indicator is documented, the numerator and denominator are specified, and we see that the team intends to monitor these data over the course of a calendar year. The team has inserted data for the first five months of the year.
The Framework for Linking Data with Action is a management tool—a combination of template and process—that can help to document the Seven Steps process and serves three key purposes: 1) Creates a time-bound plan for data-informed decision making by setting dates by which data should be reviewed in relation to key programmatic questions and upcoming decisions. 2) Encourages greater use of existing information by identifying existing data resources and linking that information with the programmatic questions that need answers to support evidence-based decision making. Last, it provides you with a data-informed decision-making ‘record’ so that you can— 3) Monitor the use of information in decision making— Provides a timeline for conducting analyses and making decisions.
Here is the template for the Framework for Linking Data with Action. Note to facilitator : Mention each column
7 Steps to Improve HIV/AIDS Programs Guide
Data Demand & Use: Seven Steps to Use Routine Information to Improve HIV/AIDS Programs Webinar Series #6 Tuesday, February 21, 2012 Presenters: Tara Nutley and Nicole Judice
Troubleshooting <ul><li>If you lose connectivity </li></ul><ul><ul><li>Re-enter the meeting room by clicking on the webinar link provided </li></ul></ul><ul><li>If you have trouble with audio </li></ul><ul><ul><li>Refer to the conference call instructions in the upper right hand corner </li></ul></ul><ul><li>Send an email to [email_address] </li></ul>
Tips for Participating in the Discussion <ul><li>To comment, raise your hand by clicking on the icon. </li></ul><ul><ul><li>Speak into your microphone (be sure it is enabled by clicking on the icon at the top of the screen). </li></ul></ul><ul><ul><li>Type questions in Q&A window located at the top of your screen. </li></ul></ul><ul><ul><li>A recording of the webinar will be made available at www.measureevaluation.org/ddu </li></ul></ul>
Agenda <ul><li>Welcome - webinar tips </li></ul><ul><li>Brief overview of Data Demand and Use </li></ul><ul><li>Presentation of Tools </li></ul><ul><li>Field Application of Tools </li></ul><ul><li>Questions and Answers </li></ul><ul><li>Wrap up </li></ul>
Why improve data-informed decision making? Pressing need to develop health policies, strategies, and interventions
“… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our policies to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” National-level Policymaker, Nigeria
Definitions <ul><li>Data use – Using data in the decision making process </li></ul><ul><ul><li>monitor a program </li></ul></ul><ul><ul><li>create or revise a program or strategic plan </li></ul></ul><ul><ul><li>develop or revise a policy </li></ul></ul><ul><ul><li>advocate for a policy or program </li></ul></ul><ul><ul><li>allocate resources </li></ul></ul><ul><li>Data Demand - decision makers specify what kind of information they want & seek it out </li></ul>
Core Data Demand and Use Principle <ul><li>Data users and data producers can work together to identify key programmatic questions and concerns and to link these questions to the data available in their respective settings. </li></ul>
Why Use the 7 Step Approach <ul><li>Provides concrete steps to data-informed decision making </li></ul><ul><li>Encourages more strategic and effective use of data </li></ul><ul><li>Ensures involvement of data users & producers </li></ul><ul><li>Guide includes examples, job aids, templates </li></ul>
Stepwise guidance helps to… <ul><li>Identify and understand trends and needs; </li></ul><ul><li>More effectively plan and set priorities; </li></ul><ul><li>Support changes in program and service delivery; </li></ul><ul><li>Support requests for additional resources; </li></ul><ul><li>Justify changes in policies affecting service delivery; </li></ul><ul><li>Provide evidence-based clinical decision making; </li></ul><ul><li>Facilitate accountability for expended resources; </li></ul><ul><li>Communicate importance of HIV/AIDS services to the community. </li></ul>
Seven Steps Approach <ul><li>1 - Identify questions of interest </li></ul><ul><li>2 - Prioritize key questions of interest </li></ul><ul><li>3 - Identify data needs and potential sources </li></ul>
Seven Steps Approach <ul><li>4 - Transform data into information </li></ul><ul><li>5 - Interpret information and draw conclusions </li></ul><ul><li>6 - Craft solutions and take action </li></ul><ul><li>7 - Continue to monitor key indicators </li></ul>
Step 1 – Identify questions <ul><li>No need to go fishing….instead answer question that respond to true needs and priorities </li></ul>
Defining Program Success - Questions to Consider <ul><li>What do you want or need to know in order to say your program is working? </li></ul><ul><li>How do you know that your program or service is working? </li></ul><ul><li>Is your program or service improving client’s health? </li></ul><ul><li>How do you know if there are problems or that your program is not achieving its pre-determined objectives? </li></ul>
Step 2 – Prioritize Key Questions of Interest <ul><li>Programmatic relevance </li></ul><ul><li>Answerable </li></ul><ul><li>Actionable </li></ul><ul><li>Timeliness of the question </li></ul>Ensure question is specific
Prioritizing Questions PROJECT/ORGANIZATION POTENTIAL SOLUTIONS PROGRAMMATIC RELEVANCE ANSWERABLE ACTIONABLE TIMELINESS OF THE QUESTION TOTAL Please list your proposed solutions, and rank them according to each criterion. Highly relevant=4 Somewhat=3 Little Relevance =2 None=1 Easy to answer=4 Feasible with routine data=3 May require non routine data=2 Significant data collection=1 Highly actionable=4 Potential barriers=3 Low chance of action=2 Little to no chance of action=1 Immediate=4 Next quarter =3 Next month =2 Distant future =1 1. 2. 3.
Step 3 – Identify Data Needs and Potential Sources <ul><li>Enlist M&E Officer </li></ul><ul><li>How frequently or at what intervals do we need this information? </li></ul><ul><li>Do the data already exist and are they available? </li></ul><ul><li>Are the data of sufficient quality? </li></ul>
Step 4 – Transform Data into Information <ul><li>Isolate required indicators and/or data elements </li></ul><ul><li>Analyze the data and calculate the indicator </li></ul><ul><li>Depict data in an image (graph, chart, table) </li></ul>
Step 5 – Interpret Information and Draw Conclusions <ul><li>Convene group or team </li></ul><ul><li>Review graphs, tables, and information </li></ul><ul><li>Discuss the meaning of these analyses for organizations, programs and facilities </li></ul>
Interpreting data <ul><li>Adding meaning to information by making connections and comparisons and exploring causes and consequences </li></ul>
Step 6 – Craft Solutions and Take Action <ul><li>Engage variety of stakeholders to craft solutions </li></ul><ul><li>Discuss conclusions from interpretation </li></ul><ul><li>Brainstorm potential solutions </li></ul><ul><li>Further specify, craft and prioritize these solutions </li></ul><ul><li>Develop an action plan </li></ul>
Step 7 – Continue to Monitor Key Indicators <ul><li>Monitor implementation of action plan </li></ul><ul><li>Consider frequency and duration of monitoring </li></ul><ul><li>Develop tool for monitoring </li></ul>
Example of Monitoring Spreadsheet Site name: 2008 N/D Indicator J F M A M J J A S O N D Numerator Denominator Indicator % of eligible clients placed on ART Numerator # of new clients on ART 102 102 103 115 125 Denominator Sum of # of new clients on ART and clients on ART waiting list 151 162 169 171 177 Indicator Numerator Denominator
Framework for Linking Data with Action <ul><li>Documents the overall Seven Steps process </li></ul><ul><li>Creates a time-bound plan for information-informed decision making </li></ul><ul><li>Encourages greater use of existing information </li></ul><ul><li>Monitors the use of information in decision making </li></ul>
Framework for Linking Data with Action Decision/ Action Program/ Policy Question Decision Maker (DM), Other Stakehold-ers (OS) Indicator/Data Data Source Timeline (Analysis) (Decision) Commu-nication Channel
Question of Interest <ul><li>Organization is concerned about retaining clients in their facilities </li></ul><ul><li>Management team identified the question: </li></ul><ul><ul><li>What percentage of PMTCT clients are counseled and tested? </li></ul></ul>
PMTCT Summary Form Summary Data from General ANC, VCT and Partner Registers Variables Number 1. New ANC clients 6 A. Post-test counseled - Positive 6 B. Post-test counseled – Negative
Additional Questions <ul><li>Which facility is performing better/worse than expected? </li></ul><ul><li>What is the trend over time for these facilities? </li></ul><ul><li>How would you assess each facility’s performance based on the data? </li></ul><ul><li>What other data or information should you consider in providing recommendations or guidance to the facilities? </li></ul>
Applications <ul><li>Nigeria – training workshops at facility, organizational and state levels </li></ul><ul><li>Tanzania – training workshops and coaching at organizational level </li></ul><ul><li>Introduced and shared with different organizations and in electronic forums </li></ul><ul><ul><li>Catholic Relief Services </li></ul></ul><ul><ul><li>TB Care Data and Analysis Forum </li></ul></ul><ul><ul><li>Virtual Leadership Development Program </li></ul></ul>
MEASURE Evaluation DDU Resources <ul><li>www.measureevaluation.org/ddu </li></ul><ul><ul><li>Data Demand and Use Tool Kit </li></ul></ul><ul><ul><li>Data Demand and Use Training Resources </li></ul></ul><ul><li>Next webinar will be on February 28, 2012 at 9:00 am Data Demand and Use Training Resources </li></ul>
Join Data Use Net <ul><li>Send an email to [email_address] . Leave the subject field blank and in the body of the message type ‘subscribe DataUseNet.’ For example: </li></ul><ul><li>To: firstname.lastname@example.org </li></ul><ul><li>From: [email_address] </li></ul><ul><li>Subject: </li></ul><ul><li>Subscribe Data Use Net </li></ul>
<ul><li>MEASURE Evaluation is funded by the U.S. Agency for </li></ul><ul><li>International Development (USAID) and implemented by the </li></ul><ul><li>Carolina Population Center at the University of North Carolina </li></ul><ul><li>at Chapel Hill in partnership with Futures Group International, </li></ul><ul><li>ICF International, John Snow, Inc., Management Sciences for </li></ul><ul><li>Health, and Tulane University. Views expressed in this </li></ul><ul><li>presentation do not necessarily reflect the views of USAID or the </li></ul><ul><li>U.S. government. </li></ul><ul><li>MEASURE Evaluation is the USAID Global Health Bureau's </li></ul><ul><li>primary vehicle for supporting improvements in monitoring and </li></ul><ul><li>evaluation in population, health and nutrition worldwide </li></ul>