How Netflix approaches incident response and the Trainman User Behavior Analytics platform.

1. Security Intelligence
and Response Team
@maestretti
jobs.netflix.com/teams/security

2. SIRT
TECH & CULTURE
CULTURE
DETECTION
Technology
Culture
medium.com/netflix-techblog/
jobs.netflix.com/culture

3. CULTURE
FnR
Freedom and Responsibility
Our goal is to inspire people more than manage them. We
trust our teams to do what they think is best for Netflix
There are a few important exceptions to our anti-rules
pro-freedom philosophy. ... keeping our members’
payment information safe, have strict controls around
access. Transferring large amounts of cash from our
company bank accounts has strict controls. But these are
edge cases.
In general, freedom and rapid recovery is better than
trying to prevent error. We are in a creative business, not a
safety-critical business. Our big threat over time is lack of
innovation…

4. CULTURE
CONTEXT
Context Not Control
There are some minor exceptions to “context not control,” such as an urgent
situation…
FEMA Incident Command System - https://training.fema.gov/

5. CULTURE
FULL CYCLE DEVELOPERS
https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

6. https://medium.com/netflix-techblog/how-we-build-code-at-netflix-c5d9bd727f15
PRODUCT
CI/CD TECH STACK
‘Baking’ Virtual Machine images, called Amazon Machine Images (AMIs),
from source (instead of configuring servers on the fly as you would with
Chef/Puppet) provides a strong baseline for forensics.
Any changes to be made to a server (instance), are made in code, checked
into source control, and built into a new AMI - then new servers (instances)
are deployed from this new AMI.
Containers deploy the same way.

7. https://www.spinnaker.io/
PRODUCT
MICROSERVICES
Deploying multiple copies of
the same AMI not only scales
load, but creates a peer
group to compare against,
allowing us to surface
suspicious differences in our
fleet.
https://github.com/Netflix-Skunkworks/diffy

8. CORPORATE
LISA
Our corporate model relies
heavily on SaaS and the
services we do run are
launched in our cloud the
same way as our product.
We isolate and devalue our
user endpoints, then seek to
protect our core assets in the
cloud.
No lateral network access
(LISA), no Active Directory, no
network shares (GDrive). https://www.slideshare.net/BryanZimmer/location-independent-security-approach-lisa

9. CORPORATE
SCOPE
Identity is our perimeter. We
seek to control access to our
cloud resources through Single
Sign On and User Behavior
Analytics.
We make access decisions
based on strong identity and
device health checks.
https://github.com/Netflix-Skunkworks/stethoscope-app

10. Technology and Culture are formative.
Our technology stack supports new approaches to security problems. We try to
solve the easy problem, instead of the hard one.
Our culture enables smart risk taking and aligns incentives to produce positive
outcomes.
SUMMARY

12. Trainman -
Learnings from a
detection platform
Security Data Science Colloquium - 06/11/2018
Siamac Mirzaie, Science & Analytics

13. Motivation

14. Corporate Apps
2-year Growth

15. 40%

16. Corporate App Users
2-year Growth

17. 460%

18. 2018 Content Investment

19. $ 8B

21. Share learnings for better
collaboration

23. Stack

24. Visualization
There is more to it than just detection
Ingestion Detection Post-Processing

27. Learnings

28. “Can you folks do some machine learning
on my app’s data?”

29. Three components to a viable use case
Business
impact
Audit log
data
Analytically
tractable

30. “The thing is, we don’t have past examples of
malicious behavior”

31. Compensating for the lack of ground truth
Security
analyst
feedback
Red team testing

32. “Wait, why was this categorized as
abnormal?”

33. Making an output explainable
Data
enrichment
Algorithms
transparency
Decomposable
ranking of
anomalies

34. “Also, some people always use this resource,
others don’t”

35. Working around data sparsity
More complex
feature engineering
Uncovering
entity
personas
Picking the right
model

36. “This is a mathematical anomaly,
not a business one”

37. Curbing false positives
Ensemble
approach
Post-processing
of anomalies

38. “This used to be an anomaly…
but not anymore”

39. Keeping up with behavioral drift
Dynamic
models/thresholds

40. “That finally looks good. Can we make it
faster?”

41. Shrinking time-to-detection
Stream
processing
Combination of
simpler anomaly
detectors

42. “We have a new use case, can you build
another detector?”

43. Handling use case quantity and variety
Repeat
Identify
use case
categories
Pre-implemented
functionality

44. Thank You.

45. Backup Slides.

46. CULTURE
Netflix Culture Memo - jobs.netflix.com/culture
1. Encourage independent decision-making by employees
2. Share information openly, broadly and deliberately
3. Are extraordinarily candid with each other
4. Keep only our highly effective people
5. Avoid rules
Our core philosophy is people over process. More specifically, we have
great people working together as a dream team. With this approach, we
are a more flexible, fun, stimulating, creative, and successful organization.
Freedom and Responsibility
Our goal is to inspire people more than manage them. We trust
our teams to do what they think is best for Netflix
There are a few important exceptions to our anti-rules
pro-freedom philosophy. ... keeping our members’ payment
information safe, have strict controls around access. Transferring
large amounts of cash from our company bank accounts has
strict controls. But these are edge cases.
In general, freedom and rapid recovery is better than trying to
prevent error. We are in a creative business, not a safety-critical
business. Our big threat over time is lack of innovation…
Context Not Control
There are some minor exceptions to “context not control,” such
as an urgent situation...
Full Cycle
Developers
https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

47. CULTURE
MEMO - 1
Netflix Culture Memo - jobs.netflix.com/culture
1. Encourage independent decision-making by employees
2. Share information openly, broadly and deliberately
3. Are extraordinarily candid with each other
4. Keep only our highly effective people
5. Avoid rules
Our core philosophy is people over process. More specifically, we have great people
working together as a dream team. With this approach, we are a more flexible, fun,
stimulating, creative, and successful organization.

48. SECURITY LEARNING
ORGANIZATION

49. Mary Landesman - Threat Intelligence
https://www.linkedin.com/in/marylandesman/
Forest Monsen - SIRT Security Engineer
https://www.linkedin.com/in/forestm/
Steve Zenone - SIRT Security Engineer
https://www.linkedin.com/in/zenone/
BROCADE
Alex Maestretti - SIRT Manager
https://www.linkedin.com/in/maestretti/
Swathi Joshi - TPM Response
https://www.linkedin.com/in/joshiswathi/
Kevin Glisson - SIRT Security Engineer
https://www.linkedin.com/in/joshiswathi/