The document discusses 3 perspectives - functional, contextual, and operational understanding - that are important to better apply predictive and prescriptive models in healthcare. Functional understanding means ensuring the model makes clinical sense. Contextual understanding is about how the model fits into clinical workflows and who will act on results. Operational understanding examines the benefits and risks of model deployment, such as sensitivity and predictive value. Providing these three levels of understanding can help support leadership decision making about predictive model deployment as it is more a change management challenge than technical issue.

1. 3 Perspectives to Better Apply Predictive
& Prescriptive Models in Healthcare
February 27, 2019
Jason Jones, PhD
Chief Data Scientist, Health Catalyst

2. Learning
Objectives
#4: The next time you have a melt down,
literally melt down. It is very impressive.
• Understand why a predictive or
prescriptive model endeavor is
more a change management
challenge than a technical one.
• Learn how to apply three types of
model understanding to a use case
in your own organization.

3. © 2018
Health
Catalyst
How many predictive model deployments have you been a part of?
1. None — 50%
2. 1-4 — 37%
3. 5-10 — 8%
4. 11+ — 5%
Poll Question #1
3

4. © 2018
Health
Catalyst
4
Chaotic
Complicated
Complicated
Simple
Complex
Close to Certainty Far from Certainty
ClosetoAgreementFarfromAgreement
Agreement
Certainty

5. © 2018
Health
Catalyst
5
And if you don’t know where you’re going, any
road will take you there.
— Lewis Carroll, George Harrison
Algorithms and predictive models are commodities.

6. © 2018
Health
Catalyst
What You’ll Get
• Algorithms
• AUC Curves
6
Nature 2017-06-23

7. © 2018
Health
Catalyst
Choose a problem you are going to solve and for which you think a
predictive or prescriptive model might help…
What is the primary intended benefit?
1. Clinical (e.g., hospital readmission) — 52%
2. Patient Experience (e.g., preemptive service recovery) — 14%
3. Operational (e.g., staffing) — 12%
4. Financial (e.g., collections) — 8%
5. Care Delivery Planning (e.g., virtual vs physical visits) — 15%
Poll Question #2
7

8. © 2018
Health
Catalyst
For the problem you just selected…
When do you plan to deploy the predictive/prescriptive model?
1. Within 1 month — 15%
2. Within 3 months — 27%
3. Within 12 months — 44%
4. Beyond 12 months — 14%
Poll Question #3
8

9. © 2018
Health
Catalyst
What Would Help
3 Levels of Predictive Model Understanding
9
Functional
Does the
model make
sense?
Contextual
How does it fit
into the
workflow?
Operational
What benefits
and risks am I
trading?

10. © 2018
Health
Catalyst
Sepsis
• Does it make sense that risk
increases when temperature is
low or high?
• Do we want to capture arterial
blood gasses for pulmonary
function?
10
Functional Sample Questions

11. © 2018
Health
Catalyst
Functional Model Understanding
11
Instructions:
• Move sliders to adjust clinical
parameters
• Observe change in risk scores (SIRS
& Model)
• Observe change in plots
• Adjust “Threshold” to modify alerting
level
• Press “Reset” to reset values
Plots:
• Y-axis is risk of severe sepsis
• X-axis is value of the parameter
• Blue line shows risk
• Blue dot is point nearest current
selected value
• Red line shows risk threshold (user
set)
• Yellow bars show SIRS (if applicable)
• Risk curves are interdependent
across plots

12. © 2018
Health
Catalyst
Functional Model Understanding
12
Instructions:
• Move sliders to adjust clinical
parameters
• Observe change in risk scores (SIRS
& Model)
• Observe change in plots
• Adjust “Threshold” to modify alerting
level
• Press “Reset” to reset values
Plots:
• Y-axis is risk of severe sepsis
• X-axis is value of the parameter
• Blue line shows risk
• Blue dot is point nearest current
selected value
• Red line shows risk threshold (user
set)
• Yellow bars show SIRS (if applicable)
• Risk curves are interdependent
across plots

13. © 2018
Health
Catalyst
Sepsis
• Are we finding the right patients
at the right time?
• Who should take action?
13
Contextual Sample Questions

14. © 2018
Health
Catalyst
For the problem you selected…
Who will act upon the predictive/prescriptive model result?
1. Clinical (support) staff — 60%
2. Administrative staff — 6%
3. Managers or leaders — 21%
4. A machine — 3%
5. I’m not sure — 10%
Poll Question #4
14

15. © 2018
Health
Catalyst
Contextual Model Understanding
15

16. © 2018
Health
Catalyst
Contextual Model Understanding
16

17. © 2018
Health
Catalyst
Coronary Heart Disease
• What percentage of patients will
we capture?
• If we start a person on statin,
what’s the chance that person
would have developed CHD?
17
Operational Sample Questions

18. © 2018
Health
Catalyst
For the problem you selected…
What are you most concerned about?
1. Missing an important case (e.g., someone who gets readmitted) — 23%
2. Misuse of scarce or expensive resources — 19%
3. Customer trust (e.g., “this model is wrong!”) — 22%
4. No change in processes or outcomes — 36%
Poll Question #5
18

19. © 2018
Health
Catalyst
Operational Model Understanding
Wanted…
• Find 50% of people who
develop CHD in 10 years.
(Sensitivity)
• 50% of our statin
prescriptions would have
been associated with CHD.
(Positive Predictive Value)
…but not possible now.
19

20. © 2018
Health
Catalyst
Operational Model Understanding
Are you okay with…
• Find 50% of people who
develop CHD in 10 years.
(Sensitivity)
• 35% of our statin
prescriptions would have
been associated with CHD.
(Positive Predictive Value)
20

21. © 2018
Health
Catalyst
Supporting Leadership Decision Making
3 Levels of Predictive Model Understanding
21
Functional
Does the
model make
sense?
Contextual
How does it fit
into the
workflow?
Operational
What benefits
and risks am I
trading?
Deploying a predictive model is a change
management activity that tools can help.

22. Q&A

23. Thank You!