Anomaly Detection: How to find what you didn’t know to look for

© 2016 MapR Technologies 2
Anomaly Detection:
How To Find What You Didn’t Know to Look For
Ted Dunning, Chief Applications Architect MapR Technologies
Email tdunning@mapr.com tdunning@apache.org
Twitter @Ted_Dunning
Ellen Friedman, Consultant and Commentator
Email ellenf@apache.org
Twitter @Ellen_Friedman

e-book available courtesy of MapR
http://bit.ly/1jQ9QuL
A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)

Practical Machine Learning series (O’Reilly)
• Machine learning is becoming mainstream
• Need pragmatic approaches that take into account real world
business settings:
– Time to value
– Limited resources
– Availability of data
– Expertise and cost of team to develop and to maintain system
• Look for approaches with big benefits for the effort expended

Anomaly Detection

Who Needs Anomaly Detection?
Utility providers using
smart meters

Feedback from
manufacturing assembly
lines

Monitoring data traffic on
communication networks

What is Anomaly Detection?
• The goal is to discover rare events
– especially those that shouldn’t have happened
• Find a problem before other people see it
– especially before it causes a problem for customers
• Why is this a challenge?
– I don’t know what an anomaly looks like (yet)

Spot the Anomaly

Spot the Anomaly
Looks pretty
anomalous
to me

Spot the Anomaly
Will the real anomaly
please stand up?

Basic idea:
Find “normal” first

Steps in Anomaly Detection
• Build a model: Collect and process data for training a model
• Use the machine learning model to determine what is the normal
pattern
• Decide how far away from this normal pattern you’ll consider to
be anomalous
• Use the AD model to detect anomalies in new data
– Methods such as clustering for discovery can be helpful

How hard is it to set an alert for anomalies?
Grey data is from normal events; x’s are anomalies.
Where would you set the threshold?

Basic idea:
Set adaptive thresholds

What Are We Really Doing
• We want action when something breaks
(dies/falls over/otherwise gets in trouble)
• But action is expensive
• So we don’t want too many false alarms
• And we don’t want too many false negatives
• What’s the right threshold to set for alerts?
– We need to trade off costs

A Second Look

A Second Look
99.9%-ile

Cool algorithm: t-digest

Online
Summarizer
99.9%-ile
t
x > t ? Alarm !
x
How Hard Can it Be?

Using t-Digest
• The t-digest is an on-line percentile estimator
– very high accuracy for extreme tails
• t-digest also available everywhere
– in ElasticSearch, in Solr
– in streamlib (open source library on github)
– in Mahout Math (open source library on github)
– standalone (github and Maven Central)
• Very handy for general distributions, few assumptions
• For latency, exponential binning may be useful
– See, for instance, hdrhistorgram

So are we all done?

What About This?
0 5 10 15
−20246810
offset+noise+pulse1+pulse2
A
B

Model Delta Anomaly Detection
Online
Summarizer
δ > t ?
99.9%-ile
t
Alarm !
Model
-
+ δ

Spot the Anomaly
Anomaly?

Maybe not!

Where’s Waldo?
This is the real
anomaly

Normal Isn’t Just Normal
• What we want is a model of what is normal
• What doesn’t fit the model is the anomaly
• For simple signals, the model can be simple …
• The real world is rarely so accommodating
x ~ m(t)+ N(0,e)

We Do Windows

Windows on the World
• The set of windowed signals is a nice model of our original signal
• Clustering can find the prototypes
– Fancier techniques available using sparse coding
• The result is a dictionary of shapes
• New signals can be encoded by shifting, scaling and adding
shapes from the dictionary

Most Common Shapes (for EKG)

Reconstructed signal
Original
signal
Reconstructed
signal
Reconstruction
error
< 1 bit / sample

An Anomaly
Original technique for finding
1-d anomaly works against
reconstruction error

Close-up of anomaly
Not what you want your
heart to do.
And not what the model
expects it to do.

A Different Kind of Anomaly

Online
Summarizer
δ > t ?
99.9%-ile
t
Alarm !
Model
-
+ δ

The Real Inside Scoop
• The model-delta anomaly detector is really just a sum of random
variables
– the model we know about already
– and a normally distributed error
• The output (delta) is (roughly) the log probability of the sum
distribution (really δ2)
• Thinking about probability distributions is good

Some k-means Caveats
• But Eamonn Keogh says that k-means can’t work on time-series
• That is silly … and kind of correct, k-means does have limits
– Other kinds of auto-encoders are much more powerful
• More fun and code demos at
– https://github.com/tdunning/k-means-auto-encoder
http://www.cs.ucr.edu/~eamonn/meaningless.pdf

The Limits of Clustering as Auto-encoder
• Clustering is like trying to tile your sample distribution
• Can be used to approximate a signal
• Filling d dimensional region with k clusters should give
• If d is large, this is no good
e » 1/ kd

0 500 1000 1500 2000
−2−1012
Time series training data (first 2000 samples)
Time
Test data
Reconstruction
Error

0 500 1000 1500 2000
0.000.050.100.15
Reconstruction error for time−series data
Centroids
MAVError
Training data
Held−out data

Another Example
• Take points randomly in , project non-linearly into
• Approximation using clustering should give

0 500 1000 1500 2000
0.00.51.01.52.0
Reconstruction error for random points
Centroids
Error
Training data
Held−out data

0 500 1000 1500 2000
0.00.51.01.52.0
Error is approximately cube root of k
k
Error Actual
Cube root model

Moral For Auto-encoders
• The simplest auto-encoders can be good models
• For more complex spaces/signals, more elaborate models may
be required
• Consider deep learning, recurrent networks, denoising

Anomalies among sporadic events

Sporadic Web Traffic to an e-Business Site
It’s important to know if traffic is stopped or
delayed because of a problem…
But visits to site normally come at
varying intervals.
How long after the last event
should you begin to worry?

Sporadic Web Traffic to an e-Business Site
It’s important to know if traffic is stopped or
delayed because of a problem…
But visits to site normally come at
varying intervals.
And how do you let your CEO
sleep through the night?

Basic idea:
Time interval between events is how
to convert to something useful you
can measure

Sporadic Events: Finding Normal and Anomalous Patterns
• Time between intervals is much more usable than absolute times
• Counts don’t link as directly to probability models
• Time interval is log ρ
• This is a big deal

Event Stream (timing)
• Events of various types arrive at irregular intervals
– we can assume Poisson distribution
• The key question is whether frequency has changed relative to
expected values
– This shows up as a change in interval
• Want alert as soon as possible

Converting Event Times to Anomaly
99.9%-ile
99.99%-ile

But in the real world, event
rates often change

Time Intervals Are Key to Modeling Sporadic Events
0 1 2 3 4
02468
t (days)
dt(min)

Poisson Distribution
• Time between events is exponentially distributed
• This means that long delays are exponentially rare
• If we know λ we can select a good threshold
– or we can pick a threshold empirically
Dt ~ le-lt
P(Dt > T) = e-lT
-logP(Dt > T) = lT

After Rate Correction
0 1 2 3 4
0246810
t (days)
dt/rate
99.9%−ile
99.99%−ile

Model-Scaled Intervals Solve the Problem

Online
Summarizer
δ > t ?
99.9%-ile
t
Alarm !
Model
-
+ δ
log p

Detecting Anomalies in Sporadic Events
Incoming
events
99.97%-ile
Alarm
Δn
Rate
predictor
Rate
history
t-digest
δ> t
ti δ λ(ti- ti- n)
λ
t

Slipped Week: Simple Rate Predictor
Nov 02 Nov 07 Nov 12 Nov 17 Nov 22 Nov 27 Dec 02
0100200300400500
Main Page Traffic
Date
Hits(x1000)
A B C D

Seasonality Poses a Challenge
Nov 17 Nov 27 Dec 07 Dec 17 Dec 27
02468
Christmas Traffic
Date
Hits/1000

Something more is needed …
Nov 17 Nov 27 Dec 07 Dec 17 Dec 27
02468
Christmas Traffic
Date
Hits/1000

We need a better rate predictor…
Incoming
events
99.97%-ile
Alarm
Δn
Rate
predictor
Rate
history
t-digest
δ> t
ti δ λ(ti- ti- n)
λ
t

Idea: Predict log(rate) from lagged log(rate)
• Predict log because
– Peak to valley ratio
– Traffic grew by 30 %
– All rates are positive

– Just because I said so

– Just because I said so
• Let model see many lagged values
• Use L1 regularized linear model to pick important historical
values
– We would have moved to something fancier if this hadn’t worked

A New Rate Predictor for Sporadic Events

Improved Prediction with Adaptive Modeling
Dec 17 Dec 19 Dec 21 Dec 23 Dec 25 Dec 27 Dec 29
02468
Christmas Prediction
Date
Hits(x1000)

Some days the magic works
Some days ...
We use slightly different magic

Streaming Micro-Service Scenario
File upload
web service
Files
Thumbnail
extraction
Transcoding
uploads
thumbs
recodes
Files

Let’s Assume a Good Micro-Architecture
Thumbnail
extraction
uploads
thumbs
metrics
exceptions
checkpoints
Input
Output
Monitoring
Restart

How Can We Monitor This?
• We want to detect more than just cascading total failure
• Arrival time is good for detecting upstream complete loss
• What about misbehavior of a black box?

0 100 200 300 400 500 600
0.00.20.40.60.8
End time
Deltatime

0 100 200 300 400 500
0200400600
Start
End

0 100 200 300 400 500 600
051015202530
End time
Elapsedtime

Some Models Must Model Internal Operations
• Computational architecture can help modeling
• Models need to be built with knowledge of intent and structure
• Pure black box is rarely sufficient … you need some intuitions

Anomaly Detection + Classification  Useful Pair
• Use the AD model to detect anomalies in new data
– Methods such as clustering for discovery can be helpful
• Once you have well-defined models in your system, you may
also want to use classification to tag those
• Continue to use the AD model to find new anomalies

Recap (out of order)
• Anomaly detection is best done with a probability model
• -log p is a good way to convert to anomaly measure
• -log p takes different forms in different systems
• Simplistic distributions insufficient in practice
– Need mixture distributions
– Resampled live data
• Adaptive quantile estimation (t-digest) works for auto-setting
thresholds

Recap
• Different systems require different models
• Continuous time-series
– sparse coding to build signal model
• Events in time
– rate model base on variable rate Poisson
– segregated rate model
• Events with labels
– language modeling
– hidden Markov models

Why Use Anomaly Detection?

Keep in mind…
• Model normal, then find
anomalies
• t-digest for adaptive threshold
• Probabilistic models for
complex patterns
-
0 5 10 15
−20246810
offset+noise+pulse1+pulse2
A
B

Dec 17 Dec 19 Dec 21 Dec 23 Dec 25 Dec 27 Dec 29
02468
Christmas Prediction
Date
Hits(x1000)
Keep in mind…
• Time intervals are key for
sporadic events
• Complex time shift to predict
rate with seasonality
• Sequence of events reveals
phishing attack

e-book available courtesy of MapR
http://bit.ly/1jQ9QuL
A New Look at Anomaly Detection
by Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)

Read online mapr.com/6ebooks-read
Download pdfs mapr.com/6ebooks-pdf
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams
6 Free ebooks
Streaming
Architecture
Ted Dunning &
Ellen Friedman
and MapR Streams

Thank you for coming today!

bit.ly/sdaml-june2016
Find my slides & other related materials to this talk here:
or search:

…helping you put data technology to work
● Find answers
● Ask technical questions
● Join on-demand training course
discussions
● Follow release announcements
● Share and vote on product ideas
● Find Meetup and event listings
Connect with fellow Apache
Hadoop and Spark professionals
community.mapr.com

Anomaly Detection: How to find what you didn’t know to look for

Recommended

Recommended

More Related Content

Similar to Anomaly Detection: How to find what you didn’t know to look for

Similar to Anomaly Detection: How to find what you didn’t know to look for (20)

More from Ted Dunning

More from Ted Dunning (20)

Recently uploaded

Recently uploaded (20)

Anomaly Detection: How to find what you didn’t know to look for