Presented by Manish Kumar and James Kariuki

1. How to Assess and
Continuously Improve Maturity
of Health Information Systems
to Achieve Better Health Outcomes
Manish Kumar, STS-Health Systems Strengthening
MEASURE Evaluation
James Kariuki, CDC
August 2018

2. Overview
• MEASURE Evaluation
• Collaboration for health information system
(HIS) improvement
• Why measure HIS stages of continuous
improvement?
• What are the stages of continuous
improvement and how are they measured?
• Introducing assessment tool
• Creating HIS improvement plan
• Questions and discussion

3. Global, five-year, $232M cooperative agreement
6 partners, led by the University of North Carolina at
Chapel Hill (UNC)
Strategic objective
Strengthen capacity in developing countries to gather,
interpret, and use data to improve health

4. Local partners and capacity
building are key
• The prime is UNC and its partners are:
o ICF
o John Snow, Inc.
o Management Sciences for Health
o Palladium
o Tulane University
• MEASURE Evaluation works with more than 72
smaller subawardees in more than 27 countries
• Over 26 percent of project funding goes back to
minor subawardees

5. Countries where we work

6. Intermediate results (Phase IV)
Strengthened collection, analysis, and use of routine
health data
Improved country capacity to manage health
information resources and staff
Methods, tools, and approaches improved and
applied to address health information challenges
and gaps
Increased capacity for rigorous evaluation

7. HIS stages for continuous
improvement

8. Collaboration for HIS improvement
• HIS improvement is not possible in isolation
• Collaboration between Health Data Collaborative, Centers
for Disease Control and Prevention, and MEASURE
Evaluation
• Ensures learning from robust areas (e.g., data quality)
• Prioritizes areas for
improvement (e.g., HIS
for health outcomes
measurement, data
exchange)
• Ensures HIS resources
are targeted to highest
priority and used
effectively

9. Why measure HIS stages of continuous
improvement?
Existing:
• HIS assessment methods and tools focus on individual
components
o For example, these tools focus on data use, data quality,
standards, etc., in isolation
• HIS assessments prioritize “as is” status
o PRISM, interoperability toolkit
Needs:
• Assess “as is” status and identify “future goals” for each of
the HIS components to develop required capabilities
• Build methodological rigor while balancing practice
needs
• A tool that can be applied periodically to track progress

10. HIS for health system improvement
The HIS Stages
of Continuous
Improvement
toolkit
Intervention tools
and packages for
HIS strengthening
Assessment
tools for
specific HIS
areas
Interoperability
toolkit
Health system
Service
delivery
A strong health information system (HIS)
What is a
strong HIS?
What are the
stages of
progression to a
strong HIS?
Leadership /
governance
Health
care
financing
Health
workforce
Medical
products,
technologies
Information
and
research
PRISM
Data
Demand
and Use

11. What to measure for ensuring
continuous improvement?

12. HIS core domains HIS components
• HIS strategic plan or HIS strategy
• Policy, legal, and regulatory framework, and compliance
• HIS leadership and governance organizational structures
and functions
HIS governance
and leadership
• HIS workforce capacity and development
• Financial management
HIS management
and workforce
• Operations and maintenance
• Communication network (LAN and WAN)
• Business continuity
HIS ICT
infrastructure
• Standards and guidelines
• HIS core services
• Interoperability (data exchange)
HIS standards and
interoperability
• Data quality assurance
• Data use
Data quality and
use

13. HIS components HIS subcomponents
HIS strategic plan or HIS
strategy
HIS strategic plan, M&E plan
Policy, legal, and regulatory
framework and compliance
HIS policies and legislation, policy compliance
enforcement
HIS leadership and
governance organizational
structures and functions
HIS leadership and coordination, organizational
structure and functions
HIS workforce capacity and
development
HIS competencies, training and education, HR
policy
Financial management HIS financing plan, resource mobilization
HIS components Slide 1 of 2

14. HIS components Slide 2 of 2
HIS components HIS subcomponents
Operations and maintenance Reliable power/electricity, ICT business
infrastructure support, hardware
Communication network
(LAN, WAN)
Networks and connectivity
Standards and guidelines Standards, guidelines, dataset definitions, data
and exchange standards
HIS core services MFL, indicator registry, terminology management,
unique person identifiers, enterprise architecture
Interoperability (data
exchange)
Person data exchanges, aggregate data
exchange, commodity management data
exchange, data exchange security
Data quality assurance DQ assurance and quality control, data
management
Data use Data use availability strategy, competencies, user
engagement, data synthesis/communication,
data use impact, data collection alignment with
workflow, Decision support

15. How do we measure HIS stages
of continuous improvement?

16. Five-point measurement scale
Stage Description
Emerging/
ad hoc
Formal processes, capabilities, experience, or understanding
of HIS issues/activities are limited or emerging.
Formal processes are not documented, and functional
capabilities are at the development stage.
Success depends on individual effort.
Repeatable Basic processes are in place, based on previous activities or
existing and accessible policies.
The need for standardized processes and automated
functional capabilities is known.
There are efforts to document current processes.
Defined There are approved documented processes and guidelines
tailored to HIS projects or activities.
There is increased collaboration and knowledge sharing.
Innovative methods and tools can be implemented and used
to extend functional capabilities.

17. Five-point measurement scale
Stage Description
Managed Activities are under control using established processes.
Requirements/goals are developed, and a process is in place
to ensure that they are met through feedback.
Detailed measures for processes and products are collected.
Optimized Best practices are applied and capable of learning and
adapting.
The system uses experiences and feedback to correct
problems and continuously improve processes and
capabilities.
Future challenges are anticipated, and a plan is in place to
address them through innovation and new technology.
Processes are in place to ensure review and incorporation of
relevant innovation.

18. Assessment tool
• Part of the toolkit that countries or organizations
can use to assess, plan, and prioritize investments
to strengthen an HIS
• Maps a path toward improvement
• Document successes and challenges in HIS
progression

19. Who should use this tool
This version of the assessment tool is meant to be
used for national-level HIS improvement. This tool
can be used by HIS leaders of the ministries and
international organizations interested in improving
HIS capabilities.
The process should include stakeholders:
• From across the health system
• At various levels
• Who have expertise in one or more of the HIS
components

20. Who should use this tool
• Can be used by a team that is
o Assessing the full HIS in a country
o Examining a subset of the HIS system
For example:
▪ The part of HIS managed by the MOH
▪ All malaria data sources
(Continued)

21. How to use this tool
The assessment can be administered one of three
ways:
1. Self-administered
2. External facilitator administered
3. A hybrid of self- and facilitator-administered
We recommend and have designed the tool for
the third approach—the hybrid approach—to
minimize the drawbacks of self-reporting bias and
documentation burden and maximize local
ownership as well as objectivity.

22. Method
1. Determine scope and target of the
assessment
2. Key informant interviews
3. Consultative workshop
4. Roadmap

23. Scope and target
Explicitly define:
• What system or subsystem of the HIS is the
target for the assessment?
• Who will be the primary user of the
resulting roadmap?

24. Key informant interviews
The facilitator will work with the HIS manager/owner
to identify key informants with expertise in one or
more of the domains.
During interviews, participants will
• Evaluate the status of each
component/subcomponent
• Identify future capability for each subcomponent
• Share documentary evidence where applicable
Individual/aggregated scores are generated using
an algorithm and populate data visualization graphs

25. Consultative workshop
Analysis and workshop: Interview results will be
analyzed and presented in a one-day stakeholder
workshop.
• Discuss and decide final scores for each
subcomponent and the overall HIS using a
participatory process guided by a facilitator
Roadmap: During the one-day workshop, participants
will develop an action-oriented roadmap based on
the current and future
stages of their HIS.

26. Assessment tool
Let’s take a look!

27. What’s next?
Ongoing
Feedback
and revisions
2018
Refine
methodology
Pilot
Other toolkit
pieces
Post
2018
Continuous
improvement
Online
version 2.0

28. Questions and discussion

29. Learning
Resource Center: a platform to assemble and share HIS theory, tools,
performance profiles, resources, definitions, and interventions.
https://www.measureevaluation.org/his-strengthening-resource-center

30. This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of
MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004.
MEASURE Evaluation is implemented by the Carolina Population
Center, University of North Carolina at Chapel Hill in partnership with
ICF International; John Snow, Inc.; Management Sciences for Health;
Palladium; and Tulane University. Views expressed are not necessarily
those of USAID or the United States government.
www.measureevaluation.org