By Sandy Ryza, Clover Health How do you get people with chronic heart conditions to take their medication? Or diagnose complications as early as possible? Healthcare operations--the set of actions that organizations like insurers take to interact with their members--sit in some sort of nebulous shadow realm between social science, medicine, and corporate bureaucracy. In this talk, Sandy will throw some additional nouns that seem more at home in the modern web era, like "machine learning" and "A/B testing," into the mix. He'll also walk attendees through an example of now Clover Health builds and tests models for predicting which of diabetic members are likely to develop complications.

1. Driving Healthcare
Operations with Data Science

2. "literally a health insurance company"

3. "Operations"
Clinical Operations
● Close member "gaps in care"
○ Not taking their meds
○ Not seeing their doctors
○ Not getting tested
● Document conditions
Insurance Operations
● Approve / deny claims
● Approve / deny authorizations
● Catch fraud

4. "Operations"
Clinical Operations
● Close member "gaps in care"
○ Not taking their meds
○ Not seeing their doctors
○ Not getting tested
● Document conditions
Insurance Operations
● Approve / deny claims
● Approve / deny authorizations
● Catch fraud

5. E.g.

7. Data ScienceEnter

8. Data Science
What should we do?
For whom?
Did it work?
Enter

9. Case Study: Whom to Call for Home
Visits?

11. Can we predict which of our diabetic
members will have complications in the
next 6 months?

12. Time
Observation Interval Prediction Interval

13. Time
Observation Interval Prediction Interval
Demographic info, lab
tests, medications,
other diagnoses
Diagnosed with
diabetes
complications?

14. Features Labels
Member Age Hypertension hba1c
CP001 65 Yes 6.5
CP002 77 No 8.3
CP002 84 Yes 7.4
Diagnosed with
Complication in 6-
month Interval
Yes
No
Yes

15. Challenge: High Class Imbalance
● Historically, only 8% of diabetic members have been diagnosed with
complications over a 6-month period.
● Easy to get "high" accuracy, but hard to get decent precision/recall tradeoff.

16. Approach: High Class Imbalance
● Evaluate using area under ROC curve.
● Empirically, tree ensemble models appear to handle the imbalance better
than logistic regression.

17. Challenge: Missing Data
● Glycated hemoglobin clearly an important feature… but we only have
measurements for ~60% of members.
● Whether we have a measurement correlates with both:
○ Diabetes complications.
○ How well a model trained without the lab measurement performs.

18. Approach: Missing Data
● Simply hardcode all missing values to something outside the measurement
range.
○ In our case, 0.0.
● This way, tree models can split on "have a measurement" vs. "don't have a
measurement".

19. Final Model: Gradient Boosting Tree Ensemble

20. Evaluation
AUROC: 0.8
Precision: 24%
Recall: 66%

21. Most Predictive Features
Glycated Hemoglobin
Age
Hypertension
Takes Insulin

23. Did it work?

24. Do we catch more complications if we
make calls using the model?

25. Control Group Treatment Group

26. Control Group Treatment Group
Call Group
(Chosen at Random)
Call Group
(Chosen by Model)

27. Found Complications Didn't Find Complications
Control Group 8 92
Treatment Group 24 76
FAKE RESULTS

28. Found Complications Didn't Find Complications
Control Group 8 92
Treatment Group 24 76
FAKE RESULTS