Data Science Bridging Gap Data Generation Comprehension

Data Science:
Bridging the Gap Between Data Generation
and Data Comprehension
Dr Carsten Riggselsen
Principal Data Scientist
Pivotal

2© Copyright 2015 Pivotal. All rights reserved.
Analyzing data is nothing new

“Their Data”“Our Data”“My Data”
“Data”
“The Data”
“Data (Big)”

“Data” vs. “Data-Driven”
Deploy analytic apps
and automation at scale
Store any type
and size of data
Discover insights
Create analytics algorithms

Data Science
Product Management
Product Design
Engineering
Continuous Improvement
Data Science

Isolated Data Science
I don’t think (Big) Data
is valuable, it’s a hype
– prove me wrong.
We do BI and stuff
already. Data Science is a
hype – prove me wrong.

Data Science
Product Management
Product Design
Engineering
Data Science

Data Science
Product Management
Product Design
Engineering

“Mere” convenience through Apps
Automate mundane or tedious tasks
Present information at a glance in an app
User Interaction with the app
Consistency and unbiasedness
24-7 availability
Scalability
Platform independence
Easy Provisioning

Smart Apps – Data Science Powered
Combining/link data sources/streams across areas and domains
There is an element of prediction involved based on accumulated data/info
Inferring (ab)normal patterns, e.g., profiling users, usage patterns
There is an element of root-cause identification involved

DS-Cheat-Sheet - Is it a SMART App?
q  Can past knowledge potentially improve on how to inform or act in the
future?
q  Is past knowledge based on data/info from different domains?
q  Do you need to affect outcomes in real-time?
q  Are (ab)normal patterns to be inferred?
q  Is the reason or cause for an action or a pattern unclear yet an important
thing to know?
q  Is the solution highly personalised?
q  Is “crowdsourcing” knowledge (data/information) beneficial?

The Car Unlock
Button – Press it!

“Siri or OK Google – unlock
my car…
UnnnLoooock my Caaaar…”
“OK – I will unlock your house”

SMART Unlock
Access to your Calendar/Agenda
Infer where/when you usually go by car
Awareness of Bank Holidays etc.
Knows where you parked your car
Knows where you are (GPS)

Works Efficient Convenient Smart
The Car-Unlock Experience
I unlocked
your car!

Examples

Obstruction Duration Prediction
•  Predict duration of road incidents in London
•  Android app developed on top of the model
•  http://ds-demo-transport.cfapps.io

R E A LT I M E
DASHBOARD
Driving
Prediction
https://youtu.be/5gySgGWJMHA

Time to Delivery
Ÿ  Three sub problems
–  Time to delivery estimate
–  Time slot availability
–  Courier scheduling
Ÿ  Courier scheduling and
time to delivery estimate
may have mutual
feedback
Logistics Comp. Logistics Comp.

Telco: Protecting Minors - Age Prediction
Estimate age of the customer based
on their calling habits
Can distinguish minors in with an
accuracy of >80%
•  Call records from March-Aug 2014
•  Corresponds to ~3TB data
•  Attributes are
•  Calling party ID
•  Called party ID
•  Date
•  Time
•  Duration at start/end
•  Location
•  Type of call and bearer
•  TAC
•  Data
•  Call records from March-Aug 2014
•  Corresponds to ~3TB data
•  Attributes are
•  Calling party ID
•  Called party ID
•  Date
•  Time
•  Duration at start/end
•  Location
•  Type of call and bearer
•  TAC
•  Data
CDR CRM Data
Feature Importance Observation
Calls (holidays-schooltime) 0.08-0.06 Minors call less in school holiday
Average call length 0.07 Minors make shorter calls
Call timing (night-day) 0.07-0.03 Minors call more at nighttime
Number of phone uses 0.05 Minors use the phones less
Percentage of text use 0.05 Minors text less
Number of contacts 0.05 Minors less likely to have 1 contact
Percentage of calls to minors 0.04 Minors call other minors more
Percentage of voice use 0.04 Typical
Caller-Callee ratio 0.04 Minors receive more calls than make
Fri/Sat/Thurs ratio 0.04-0.03 Minors call more at weekends
Number of locations 0.04 Minors more likely to have 2 locs

Internal Transaction Fraud Detection
Beyond signatures
Beyond simple metrics for thresholding
Beyond manual engineering of rules
Monitor each and every entity in its environmental context

2
5
3
3
3,25
UserID and Data Experts analyze Overall vote is determined
S(id) = w1 · M1(id) + ... + wj · Mj(id)
X
i
wi = 1
s.t.
Weights are a measure of “importance” for
model expert j. Initially uniform across all
experts.
Mixture of Experts Metaphor

Anomalous User Behavior Comparison
Mean Anomaly Scores Users
Transaction
Anomaly
SoD
Risk
Terminated
Employees
CDHDR
Access
Anomaly
VPN
Access
Anomaly
Cluster
Outlier
Total
Score
# %
Reg B
Red 0.6 0.6 0.1 0.2 0.1 0.6 2.3 26 0.3%
Amber 0.4 0.5 0.1 0.1 0.1 0.6 1.7 73 0.8%
Green 0.0 0.0 0.0 0.0 0.1 0.0 0.1 8,765 98.9%
Reg A
Red 0.1 - - 1.0 0.4 0.9 2.4 1 0.01%
Amber 0.4 0.2 0.0 0.1 0.2 0.7 1.7 25 0.4%
Green 0.0 0.0 0.0 0.0 0.1 0.0 0.2 6,853 99.6%

Add SMARTness to your app by leveraging data
Don’t think of Data Science in an isolated fashion
Move beyond POCs on Big Data
Start with a minimal viable product/solution
Get the right platform and resources in place
Collaborate and interact
Conclusions

Digital Transformation Forum
Disrupt or Be Disrupted
19 OCTOBER · BMW WELT EVENT CENTRE · MUNICH

Data Science Bridging Gap Data Generation Comprehension

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Data Science Bridging Gap Data Generation Comprehension

Similar to Data Science Bridging Gap Data Generation Comprehension (20)

More from VMware Tanzu

More from VMware Tanzu (20)

Recently uploaded

Recently uploaded (20)

Data Science Bridging Gap Data Generation Comprehension