It is and it is not about the model. In this presentation, learn about a framework and some case studies that apply the framework on.

1. A METHOD TO
AI MADNESS
Vishwa Kolla
Head, Advanced Analytics
John Hancock Insurance

2. TOPICS
Background Framework Case Studies
2

3. BUILD ME A MODEL TO …
3
REDUCE
COMPLAINTS
GROW
WALLET-
SHARE
GROW
CSAT
REDUCE
CHURN
REDUCE
COST TO
TARGET
GROW
BOTTOM-LINE
GROW
TOP-LINE
REDUCE
COST TO
ACQUIRE

4. MODEL BUILD IS THE PATH OF LEAST RESISTANCE
4
Platforms
R
P
H20
SPARK
TENSOR
FLOW
TBD
SUPERVISED
UN
SUPERVISED
NLTK
NUM
PY
PAN
DAS
PLOT
LY
PLATFORMS
ALGORITHMS
PACKAGES
PYO
DBC
CRY
PTO
PYPD
F
SCI
KIT
TOR
NAD
O
ZICT
BAB
EL
BLA
ZE

5. A THOUGHTFUL APPROACH CAN YIELD BETTER OUTCOMES
BUSINESS
USE CASES
DATA MATH
TECHNICAL
IMPL.
BUSINESS
IMPL.
FEED
BACK
5

6. TOPICS
Background Framework Case Studies
6

7. “A” MODEL BUILD FRAMEWORK
7
DATA TARGET CONSTRUCTION EVALUATION PERFORMANCE
SOURCES
DISTANCE FROM
SIGNAL
SAMPLING
METHOD
SAMPLE
SIZE
SIGNAL SIZE
PREDICTION
HORIZON
UNIT OF
ANALYSIS
ONE MODEL vs.
STRATIFIED
ONE MODEL vs.
SEVERAL MODELS
TARGET
DEFINITION
PRESENCE OR
ABSENCE
BLACK BOX vs.
CLEAR BOX
RECENCY
FREQUENCY
SEVERITY
FEATURE
SELECTION
MODELING
STRATEGY
MODEL
STRENGTH
EXPLANATORY vs.
IMPORTANCE
ACCURACY vs.
SENSITIVITY vs.
SPECIFICITY
ECONOMIES OF
SCOPE
MODEL
FIT
BAGGING
ENSEMBLE
SINGLE vs.
MULTIPLE STAGES
PREDICTION &
OPTIMIZATION
BOOSTING

8. TOPICS
Background Framework Case Studies
8

9. Business
9

10. IT (ALWAYS) STARTS WITH A BUSINESS PROBLEM
PROSPECTING NURTUREACQUISITION
MARKET
SEGMENTS
CUSTOMER
SEGMENTS
LIKELY TO [*]
MEDIA
MIX
CHANNEL
SURVEY
ANALYTICS
CROSS / UP-
SELL
OCR
MISREP
LIKELIHOOD
MORTALITY
APS
SUMMARY
FLUIDLESS
SMOKER
LIKELIHOOD
MORBIDITY
CHURN
NEXT BEST
OFFER
CLAIM
LIKELI-
HOOD
JOURNEY
CLAIM
SEVERITY
NEXT BEST
ACTION
FRAUD
>>
TEXT
ANALYTICS
OPTIMIZE
NEXT LIKELY
ACTION
WELLNESS
IOT
ANALYTICS
NPS
ANOMALY
>>
10

11. FOCUS ON INCREMENTAL VALUE KEPT US GROUNDED
BUSINESS CASE OPTICAL REALIZABLE SHARED INCREMENTAL

12. IN PROSPECTING, TARGET OPTIMIZAITON IS A JOURNEY
12
… LOWER CUSTOMER TARGETING COSTSA SERIES OF OPTIMIZATION TARGETS …
Prospects
Leads
Apps
Issued
Placed
CPL
CPA
CPP
CP[*]
CHANNEL
MIX

13. Data
13

14. PLANS ARE NOTHING ; PLANNING IS EVERYTHING
14
EDA
USEABLE
USEFUL
DERIVATIVES
BI-VARIATE CROSSTAB PRINCOMP JOURNEY

15. A DATA STITCH IN TIME SAVES NINE
15
CLAIM
TERMIN
ATION
CLAIM
ACTV.
DEMOS
CALLS
INTERA
CTION
CLAIM
INIT.
CUSTOMER
MONTH
FRAUD
DETECTION

16. UNDERSTANDING DATA SAVES (NOT WASTES) TIME
16
Signal
Distribution
Pop. Incidence Rate
Skews Model Inclusion

17. Math
17

18. FLEXIBILITY IN TARGET DEFINITION IMPROVED ACTIONABILITY
18
20172014
Predict incidence
In next 3 years
2007
20172014
Predict incidence
3 years out
2007
Vs.

19. RIGHT SIZING SIGNAL CAN YIELD BETTER OUTCOMES
19
SIGNAL
DILUTION
SIGNAL
AMPLIFICATION
1%
99%
40%
60%

20. SIMPLE MODELS CAN HELP US EXPLAIN BIG DRIVERS
20
PREDICTORS
PRESENCE
RECENT
FREQUENT
SEVERE

21. QUANTIFICATION OF INFORMATION GAP IS A GOOD FIRST STEP
© Andrew Ng
INFORMATION GAP

22. WINNING
MODEL
CHALLENGING CHAMPIONS HELPS US UP THE ANTE
22
DATA TARGET
SOURCES
DISTANCE FROM
SIGNAL
SAMPLING
METHOD
SAMPLE
SIZE
SIGNAL SIZE
PREDICTION
HORIZON
UNIT OF
ANALYSIS
ONE MODEL vs.
STRATIFIED
ONE MODEL vs.
SEVERAL MODELS
TARGET
DEFINITION
METHODS
LINEAR
TREES
DEEP-
LEARNING
EVALUATION
MODEL
STRENGTH
EXPLANATORY vs.
IMPORTANCE
ACCURACY vs.
SENSITIVITY vs.
SPECIFICITY
ECONOMIES OF
SCOPE
MODEL
FIT

23. Technical Implementation
23

24. MULTI-STAGED MODELS PROVIDED IMPLEMENTATION FLEXIBILITY
24
9-101-8
1-7 8-10
Likely to
Qualify
Likely to
Respond
Sweet
Spot
DESIRED
SIGNAL
MODEL
MIS-
CLASSIFICATION
MODEL
EXCLUDE NOISE1
INCLUDE MIS-CLASSIFIERS2
STAGES

25. Business Implementation
25

26. A CULTURE OF MEASUREMENT, TEST AND LEARN IMPROVES VALUE
26
Measure
TestLearn
Build
CONTINUOUS
IMPROVEMENT