lightning talks, overview and cloud prize update

1. NetflixOSS
Season 1, Episode 3

2. Lightning talks

3. Pytheas
● Web based resource
and rich UI framework
● Dashboards
● Web console
● Exploring / visualizing
structured + unstructured
data

4. Pytheas - based on OSS frameworks
● Guice
● Jersey
● FreeMarker
● JQuery
● DataTables
● D3
● JQuery-UI
● Bootstrap

6. Pytheas - Modular Design
● Single jar per module
● Bundles
○ Guice Module
○ Jersey Resources
○ FreeMarker Templates
○ CSS + Img + JS code
○ Properties configuration
○ Pytheas Module class
● Minimal bootstrap code to wire modules
together into a dashboard

7. Pytheas
What's next
- Angular JS
- css/js preprocessing
- Scaffold generation
https://github.com/Netflix/pytheas

8. Conformity Monkey
Help keep your instances
and clusters following
best practices

9. Conformity Monkey
● Mark and Notify approach
● Customize rules
● Customize rule set
● Add new rules
● Customize execution time and frequency
● Can migrate to other cloud platforms

10. Sample Conformity Rules
● Security Groups
● Instance Configuration In Eureka
● ELB and ASG mismatch
● Cluster Naming Conventions

11. Conformity Monkey
https://github.com/
Netflix/SimianArmy

12. Zuul
● Netflix's edge tier
○ In place of Nginx, Apache Http or Traffic Server
● Dynamic filters that act on HTTP requests
● Filters written in Groovy / Dynamically
updated
○ Any JVM language Supported

13. Zuul within Netflix
ELB
Zuul
API Service
Netflix Services
ELB
Zuul
Streaming
Service
ELB
Zuul
Website Service

14. Request Lifecycle
HTTP Request
"pre" filters "routing" filter(s) "post" filters
Origin Server
"custom" filters
Http Request Http Response
"error" filters

15. How we Use Zuul
● Authentication
● Insights
● Stress Testing
● Canary Testing
● Dynamic Routing
● Load Shedding
● Security
● Static Response handling
● Isthmus

16. Zuul Components

17. https://github.com/Netflix/zuul
http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html

18. ICE: AWS Usage Tool
● Background
● Design
● Usage
● Next Steps

19. Background
● There is increased resource usage in the cloud
● That is very decentralized
● But lacks broad and deep insights about resource usage

20. ICE Design Approach
● Cloud resource usage has many dimensions: account,
region, zone, product, usage type, user-defined tags
● ICE writer tags each line item in billing file with different
tags
● ICE UI provides a way for users to query cloud resource
usage freely in arbitrary dimensions

22. ICE Usage
● ICE provides a birds-eye view of large and complex
cloud landscape from a usage and cost perspective.
● ICE UI is easy and intended to be used by:
● Managers
● Reservation Planners
● Engineers
● ICE OSS at https://github.com/Netflix/ice

23. ICE Next Steps
● Accessing billing files though IAM role and allow
multiple payer accounts
● Create Wiki page and Google Group
● Configurable cost alerts

24. Genie - Motivation

25. Genie - Configuration Service
● Abstraction of physical details of
back-end Hadoop clusters in the
cloud
○ Status: up, out of service, or
terminated
○ Cluster naming/tagging for job
submissions
○ Site-specific Hadoop, Hive and
Pig configurations

26. Genie - Job Execution Service
● Simple API to run Hadoop, Hive and
Pig jobs
○ Auto-magic submission of jobs to
the right Hadoop cluster
○ Abstracting away cluster details
from clients

27. Genie - Use Cases Enabled at Netflix
● Running 100% of all SLA jobs, and a high
percentage of ad-hoc jobs
● Running nightly short-lived “bonus” clusters to
augment ETL processing
● Re-routing traffic between clusters
● “Red/black” pushes for clusters
● Attaching stand-alone gateways to clusters

28. http://techblog.netflix.com/2013/06/genie-is-out-of-bottle.html
https://github.com/Netflix/genie

29. Lipstick
Visualization of Pig workflows at a logical level.
● quickly spot common errors / inefficiencies
● visually monitor execution
● empowers users to support themselves
● facilitates communication between
infrastructure team and users

30. Lipstick

31. Lipstick in our Architecture

32. Lipstick UI

33. Lipstick UI - Pig Workflow Closeup

34. https://github.com/Netflix/Lipstick
http://techblog.netflix.com/2013/06/introducing-lipstick-on-apache-pig.html

35. Overview and Roadmap

36. 34 Projects and counting...

37. Excellent mindshare and vendor adoption
Too hard to get started for individual users
State of NetflixOSS

38. Where to Start?
● Recipe app(s)
● AMI's - Asgard available, more coming!
● Hangouts
● Workshop(s)
● One button install.... not there yet.

39. Build,
Deploy,
Tools
Libraries Libraries
, Cont-d.
Services ZK / C* Big Data Recipe
Apps
Aminator Archaius Karyon Eureka Curator Genie Recipes-
RSS
Asgard Blitz4J Netflix
Graph
Edda Exhibitor Lipstick Flux
Capacitor
GCViz Denominator Ribbon EvCache Astyanax ACME Air
Ice EvCache RxJava Turbine Priam
Simian
Army
Feign Servo Zuul CassJMeter
Pytheas Governator Hystrix
Categories

40. 3rd Party demos

41. Acme Air OSS Sample and Benchmark
● Fictitious airline sample application
● Mobile app (iOS/Android) and desktop browser versions
● NodeJS and Java versions with multiple NoSQL data tier impls
● Benchmarking framework that allows “Web Scale” runs
● Previously run at 4.3 Billion customer facing REST calls per
second (50,000 req/sec) using IBM middleware and cloud
● Re-implemented using Netflix OSS
● Microservices architecture
● Eureka, Hystrix, Karyon, Ribbon
● Asgard
● Run at (peak) ~ 13,000 req/sec – 1.1 Billion per day
● Replicate this yourself in ~ 4 hours with the OSS
Learn more at http://bit.ly/acmeairblog and from @aspyker

42. Eucalyptus, AWS, and NetflixOSS
Asgard manages autoscaling across both clouds.
Eucalyptus/AWS compatibility makes it possible!
Some
NetflixOSS
services
can run in
either cloud
Simian Army
Edda
Aminator
Priam
Servo
30+ NetflixOSS services
...and more on the way!

43. Confidential and Proprietary
● Multi Data Center Support
● Single Dashboard to manage OpenStack & AWS
● Access multiple Tenants in Single Login
● One-Stop-Shop to manage
● Compute, Storage, Network
● Security
● Cloud Templates
● Asgard Framework and Architecture
● Why Asgard?
● Janitor Monkey for OpenStack
● Future Plans
● Demo
HIGHLIGHTS

44. Riot Games rCloud
●Aminator +chef-solo provisioner
●Asgard
●Edda +Security Audits
●Simian Army
●Ice
●Eureka
●Archaius + Configuration Service
●Ribbon, Karyon, Hystrix
●And more

45. Progress and Updates
NetflixOSS Cloud Prize

46. State of the Cloud Prize
Forks of github.com/netflix/cloud-prize ~700
Mailchimp signup form complete ~120
Public active submissions ~20
Two more months for procrastination!
Stickers in return for Mailchimp signup...

47. Bootcamp/Workshops
AMIs, Charms, Cookbooks, Ansible Models
More incentives and joint prizes
Make it easier to get
started

48. Juju Charm Championship
https://juju.ubuntu.com/charm-championship/
Helping solve ease-of-deploy for NetflixOSS
Mark Shuttleworth joins Netflix Cloud Prize judges
Adrian Cockcroft joins Charm Championship judges
Ubuntu Juju

49. Citrix are offering $10K prize for best
integration with NetflixOSS
Apache Cloudstack

50. First prototype!
Cloud Prize Trophy

51. Inspiration

52. Cloudified

53. Cloud
Monkey
It Bleeps!
Designed by
bleeplabs.com

54. Q+A and Demos
Food and Drink in Building B