The key to successfully leveraging artificial intelligence (AI) in healthcare rests not wholly in the technical aspects of predictive and prescriptive machines but also in change management within healthcare organizations. Better adoption and results with AI rely on a commitment to the challenge of change, the right tools, and a human-centered perspective. To succeed in change management and get optimal value from predictive and prescriptive models, clinical and operational leaders must use three perspectives: Functional: Does the model make sense? Contextual: Does the model fit into the workflow? Operational: What benefits and risks are traded?

1. Artificial Intelligence in Healthcare:
A Change Management Problem
Health Catalyst Editors

2. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Jason Jones, PhD
Chief Data Scientist
Health Catalyst
This report is based on a webinar presented by Jason Jones, PhD, Chief
Data Scientist at Health Catalyst February 27, 2019, titled, “3 Perspectives
to Better Apply Predictive & Prescriptive Models in Healthcare.”
Artificial Intelligence in Healthcare

3. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Artificial Intelligence in Healthcare
Artificial intelligence (AI) is a hot topic in
health and healthcare today. Most of the
focus has been on the technical challenges
of building and deploying predictive and
prescriptive models.
While there is always room for improve-
ment, algorithms have advanced to a
point that allows us to focus on the bigger
challenge: change management.
By acknowledging that change is hard,
“tooling up,” and maintaining a human-
centered perspective, we can achieve
better adoption and results with AI.

4. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Artificial Intelligence in Healthcare
Predictive and prescriptive models can
enhance health and healthcare, which will
ultimately help improve Quadruple Aim
outcomes: population health, patient
experience, reduced cost, and positive
provider work life.
However, successful adoption of predictive
and prescriptive models heavily depends
upon behavior change, which requires
more than technical accuracy.

5. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Artificial Intelligence in Healthcare
While predictive algorithms abound, tools to
facilitate change management remain scarce.
This article will help clinical and operational
leaders obtain value from predictive and
prescriptive models using three perspectives:
functional, contextual, and operational.
Investing time and effort to ensure these
three levels of model understanding is
necessary for broad-scale AI adoption.

6. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Predictive Versus Prescriptive Models
Healthcare leaders currently delving into AI must understand what predictive
and prescriptive models are and how they differ:
Data scientists try to anticipate a
person’s behavior or a circumstance
before it happens—ideally, with
enough time to act—but it does not
necessarily tell data scientists what
action to take. For instance, a
predictive model might reveal a
patient’s prognosis and a 30 percent
chance of readmission within 30
days (but not how to manage the
patient’s risk).
Predictive Model
Not only does the model tell data
scientists what is going to happen, it
advises the specific course of action
that should be take in response. In
the readmission example, a
prescriptive model would reveal the
prognosis and readmittance odds but
would advise a phone call to the
patient one day after discharge and
a multidisciplinary visit three days
after discharge.
Prescriptive Model

7. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Artificial Intelligence in Healthcare
Healthcare leaders and data scientists
have many predictive and prescriptive
modeling resources available, including
some free clinical prediction models.
In fact, these resources teach healthcare
professionals that developing predictive
models is no longer a question of
technical aptitude but whether a health
system is prepared to deploy them.

8. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
A Framework for AI Change Management
The first question clinical and operational
leaders need to answer is, “What are we
trying to achieve?”
Algorithms and predictive models are
commodities employed to achieve specific
outcomes. It’s important to know the
desired outcome before setting out to
build a predictive model.
If you don’t know where you’re going,
any road will take you there.”
- George Harrison

9. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
A Framework for AI Change Management
Healthcare leaders often disagree over the best course
of action to affect change within an organization.
To successfully adopt AI technologies, frontline leaders
can use a framework for change management that sets
expectations and facilitates productive dialogues to
encourage consensus and buy-in.
The framework includes three levels of understanding:
1. Functional
2. Contextual
3. Operational

10. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
A Framework for AI Change Management
1. Functional: Does the model make sense?
If a patient has multiple comorbid conditions,
does it make more sense that he’s going to be
readmitted, or that he’s taken care of because
he’s receiving care from a group of specialists?
The functional level of understanding ensures
data scientists and healthcare leaders recognize
the relationship between the input feature or
parameter and what they’re trying to predict or
suggesting as an action item.

11. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
A Framework for AI Change Management
2. Contextual: How does it fit into the workflow?
A healthcare analytics team has built the world’s
most accurate predictive model, but the team
doesn’t know what to do with the results.
Analytics teams can’t lose sight of the fact
that the results of this prediction will drive
patient action.
It is imperative that the team understands
who the patient is, what she’s going to do
with the prediction, and when it would
make sense for her to do it.

12. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
A Framework for AI Change Management
3. Operational: What benefits and risks are traded?
Healthcare professionals and clinicians make
very few choices that are risk-free and don’t
require some level of trade-off.
How can analytics teams understand those
trade-offs and make the best decisions for the
organization?
And how can the organization make those
trade-offs with eyes wide open and accurately
measure if they’re achieving success?

13. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Levels of Predictive Model Understanding
Once healthcare leaders have the three-
level framework for change management,
they can begin to deploy predictive and
prescriptive models.
These three levels of understanding
create the framework for change
management that enables successful
deployment and adoption of predictive
and prescriptive models.

14. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Levels of Predictive Model Understanding
Attaining functional, contextual, and
operational understanding helps ensure
that teams don’t lose track of the problem
they’re trying to solve and bolsters their
confidence within the organization to
move decision makers toward agreement.

15. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
To demonstrate, this article will examine three examples of how teams can
move through the three levels of understanding.
Functional Model
Understanding
Contextual Model
Understanding
Operational Model
Understanding

16. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
In the first example, an analytics teams
wants to build a predictive model that
determines which patients in the
emergency department (ED) are at
greatest risk for progressing to severe
sepsis or septic shock within 24 hours.

17. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
In functional model understanding, the team might
begin by asking questions to achieve a basic
understanding of how the model should work:
Does a patient’s risk for severe sepsis or
septic shock increase when his temperature
is low or high?
If the patient’s temperature is low, should
his risk go up or down?
If the patient’s temperature is high, should
his risk up or down?
If so, by how much?
>
>
>
>

18. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
The analytics team needs help answering
these questions; they need input from
clinicians to build a model that makes sense.
Figure 1 illustrates a sample model for
predicting whether a patient will progress to
severe sepsis or septic shock within a 24-
hour period, using six key parameters.
In this example model, the patient’s risk of
developing sever sepsis or septic shock
moves up or down based on how the
parameters change.

19. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
Figure 1: Functional model understanding—sample model for predicting whether a
patient will progress to severe sepsis or septic shock within a 24-hour period.

20. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
Clinician input and understanding are key to
getting the healthcare organization to embrace
and adopt this model down the road.
Data analysts and data scientists need to make
the predictive model transparent, so clinicians
understand how it works, and, in turn, support
the model—a critical step for adoption.

21. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Functional Model Understanding
George Box, a British statistician, famously said,
“All models are wrong. Some models are useful.”
The team is trying to build a useful model, and
the utility will not just be how well it predicts the
event, but what somebody does with the newly
discovered information.
Obtaining functional model understanding
is the first step.

22. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Contextual Model Understanding
Taking the Sepsis example above, obtaining
contextual model understanding requires the
team to ask a different set of questions:
Are we finding the right patients at
the right time?
Who should take action?
>
>

23. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Contextual Model Understanding
The next question the team might ask for
contextual understanding is, “Who’s going
to take action?”
Should a phlebotomist draw a lab with an
automated order?
Should the patient receive fluids earlier?
Figure 2 depicts an example I-Chart that
provides details leading up to an incident
that can be used for contextual model
understanding.

24. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Contextual Model Understanding
Figure 2: Sample I-Chart that provides contextual model understanding.

25. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Contextual Model Understanding
To obtain contextual model understanding, the
team would look at patients who may be at
risk for severe sepsis or septic shock.
The I-Chart provides additional information
about diagnoses, comorbidity, and recent
clinical encounters (e.g., whether the patient
had recently been in an ambulatory setting).
The chart shows the first 24 hours, from the
time the patient checks into the ED, and the
six parameters identified above by the
analytics team in functional model
understanding.

26. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Contextual Model Understanding
This allows the team to understand the
complete context for this patient in
relationship to the predictive model.
With this information, and the questions they
ask in the proper context, the team can
make adjustments to improve the model’s
utility and further gain clinician buy-in.

27. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Operational Model Understanding
Using a different example to demonstrate
operational model understanding, the team
wants to identify patients who will develop
congenital heart disease (CHD).
Unlike the sepsis example, the team is not
limited to events in a specific set of hours or
days but decades worth of information.

28. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Operational Model Understanding
In building this model, the team will want to
consider the following questions:
What percentage of patients will the
model capture?
If the patient is prescribed a statin, what
is the chance she will develop CHD?
>
>

29. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Operational Model Understanding
To build a useful model, the team will not only
need to think about which patients might
develop heart disease over the next decade
but also what to do with that information.
One possible action is to prescribe a statin for
the patient. Prescribing a statin is a relatively
low-intensity action but also comes with
possible side effects and associated costs.
Operational model understanding compels
teams to examine the cause and effects
and weigh the tradeoffs that often come
with medical decision making.

30. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Operational Model Understanding
If the team’s goals for this model are to
capture 50 percent of patients who develop
CHD in the next 10 years and, for those
patients identified, have 50 percent of statin
prescriptions associated with CHD, how can
they set expectations appropriately for the
tradeoffs they’re making?

31. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Three Example Predictive Models to
Demonstrate Understanding
Operational Model Understanding
Can they gain clinician and leadership buy-in
for a model that would benefit 50 percent of
identified patients?
If not, how can they modify the model to get
the buy-in they need for adoption?
This type of organizational dialogue and
value balancing facilitates operational model
understanding.

32. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Supporting Leadership Decision Making
To successfully adopt predictive and prescriptive
models, most healthcare organizations will do
the heavy lifting in change management and
efforts to overcome organizational inertia.
The analytics teams can leverage three specific
tool sets to facilitate the change management
process: functional understanding, contextual
understanding, and operational understanding.
With these three tools, teams are more likely
to successfully deploy predictive and
prescriptive models.

33. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Supporting Leadership Decision Making
An important part of the success of this
model is ongoing collaboration, along
with the goal of creating something that
everyone (from data scientists to
clinicians) can understand, trust, and
utilize.
Successful deployment ultimately
depends on the usefulness and
level of adoption of the model.

34. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
For more information:
“This book is a fantastic piece of work”
– Robert Lindeman MD, FAAP, Chief Physician Quality Officer

35. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
More about this topic
Link to original article for a more in-depth discussion.
Artificial Intelligence in Healthcare: A Change Management Problem
Meaningful Machine Learning Visualizations for Clinical Users: A Framework
Valere Lemon, MBA, RN, Senior Subject Matter Expert; Alejo Jumat, User Experience Designer, Sr.
Healthcare Data Management: Three Principles of Using Data to Its Full Potential
Sean Whitaker
Machine Learning in Healthcare: What C-Suite Executives Must Know to Use it Effectively in Their
Organizations Eric Just, SVP and General Manager, Product Development; Levi Thatcher, VP, Data Science
Tom Lawry, Director, Worldwide Health, Microsoft
A New Era of Personalized Medicine: The Power of Analytics and AI
Health Catalyst Editors
Machine Learning, Predictive Analytics, and Process Redesign Reduces Readmission Rates by 50
Percent Health Catalyst Success Stories

36. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Other Clinical Quality Improvement Resources
Click to read additional information at www.healthcatalyst.com
Jason Jones currently serves as Chief Data Scientist at Health Catalyst. Previously, he served at
Kaiser Permanente (KP) in various roles including Research Scientist and VP, Information Support
for Care Transformation. Prior to KP, Jones was a Medical Informaticist for Intermountain
Healthcare. Other roles have included analytic and marketing leadership positions at Bayer
HealthCare, data and information product development at UnitedHealth Group, and various
academic adjunct faculty positions. Jones received his PhD in Biostatistics from the University of
Southern California in 2001. His mission is to leverage data to achieve the Quadruple Aim.
Jason Jones, PhD

37. © 2020 Health Catalyst
Proprietary. Feel free to share but we would appreciate a Health Catalyst citation.
Other Clinical Quality Improvement Resources
Click to read additional information at www.healthcatalyst.com
Health Catalyst is a mission-driven data warehousing, analytics and outcomes-improvement company
that helps healthcare organizations of all sizes improve clinical, financial, and operational outcomes
needed to improve population health and accountable care. Our proven enterprise data warehouse
(EDW) and analytics platform helps improve quality, add efficiency and lower costs in support of more
than 65 million patients for organizations ranging from the largest US health system to forward-thinking
physician practices.
Health Catalyst was recently named as the leader in the enterprise healthcare BI market in
improvement by KLAS, and has received numerous best-place-to work awards including Modern
Healthcare in 2013, 2014, and 2015, as well as other recognitions such as “Best Place to work for
Millenials, and a “Best Perks for Women.”