Zero to the Cloud with @NetflixOSS
•
18 likes•4,257 views
Getting started on AWS is easy, but building a scalable, reliable and performant product in the cloud can be a challenge for startups and enterprises alike. Netflix has famously migrated all our services to the cloud. Along the way, we have open sourced large portions of our platform that helped make this a reality. In this talk, Mike McGarr (Manager, Netflix Build Tools) will provide a survey of the @NetflixOSS products available. Mike will also share patterns and lessons Netflix learned migrating to the cloud. Lastly, Mike will leave you with a roadmap for how to get started with @NetflixOSS on your cloud today. This talk will cover the following @NetflixOSS products, and more!
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Zero to the Cloud with @NetflixOSS
Similar to Zero to the Cloud with @NetflixOSS (20)
More from Mike McGarr
More from Mike McGarr (12)
Recently uploaded
Recently uploaded (20)
Zero to the Cloud with @NetflixOSS
- 1. Zero to the Cloud with @NetflixOSS Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.MikeMcGarr.com
- 2. about me • J. Michael (Mike) McGarr • Engineering Manager, Netflix Build Tools • Founder, DC Continuous Delivery Meetup • Former Director of DevOps, Blackboard • Excella Consulting • Booz Allen Hamilton 11/17/14 @SonOfGarr 2
- 5. @NetflixOSS 11/17/14 @SonOfGarr http://netflix.github.io 5
- 7. The Netflix Way Everything is “built for three” Fully automated build tools to test and make packages Fully automated machine image bakery 11/17/14 @SonOfGarr 7
- 8. The Netflix Way Fully automated image deployment Service teams responsible for both “Dev” and “Ops” Redundancy through multi-region deployment 11/17/14 @SonOfGarr 8
- 9. cloud native 11/17/14 @SonOfGarr 9
- 10. microservices (soa) 11/17/14 @SonOfGarr 10
- 12. Zero to the Cloud • Stand-up your own NetflixOSS cloud • Aminator • BaseAMI • Asgard • Edda • Eureka http://nflx.it/zerotocloud 11/17/14 @SonOfGarr 12
- 13. foundations • account setup • user, roles, security groups • jumphost • foundation AMI 11/17/14 @SonOfGarr 13
- 15. immutable servers • new os image per code change • never change running instances • avoid configuration drift http://martinfowler.com/bliki/ImmutableServer.html 11/17/14 @SonOfGarr 15
- 16. Nebula • Netflix Build Language • Gradle plugins http://github.com/nebula-plugins 11/17/14 @SonOfGarr 16
- 17. Nebula Plugins • gradle-ospackage-plugin • gradle-dependency-lock-plugin • gradle-extra-configurations-plugin • gradle-blacklist-plugin • gradle-override-plugin • gradle-stash-plugin • gradle-scm-plugin • gradle-contacts-plugin • gradle-info-plugin • nebula-ospackage-plugin • nebula-test • nebula-project-plugin • nebula-release-plugin • nebula-bintray-plugin • nebula-publishing-plugin • nebula-metrics-plugin 11/17/14 @SonOfGarr 17
- 18. gradle-ospackage-plugin • produce rpm or debian packages from Gradle build • can be used for non-java apps 11/17/14 @SonOfGarr 18
- 20. Baking • app. code installed as OS package • unit of deployment = AMI 11/17/14 @SonOfGarr 20
- 21. BaseAMI 11/17/14 @SonOfGarr 21
- 22. Aminator • python library used to “bake” EBS AMI’s • must bake per region • works with system packages, Chef, Puppet, Ansible, etc 11/17/14 @SonOfGarr 22
- 23. sudo aminate -e ec2_aptitude_linux -b ubuntu-foundation -n ubuntu-base-ami baseami/build/distributions/baseami_1.0.0_all.deb 11/17/14 @SonOfGarr 23
- 25. AWS “applications” 11/17/14 @SonOfGarr 25
- 26. AWS “applications” 11/17/14 @SonOfGarr 26
- 27. asgard • Netflix AWS console • Provides “app” abstraction • Hides AWS keys • NetflixOSS https://github.com/Netflix/asgard 11/17/14 @SonOfGarr 27
- 28. red/black deployments 11/17/14 @SonOfGarr 28
- 29. red/black deployments 11/17/14 @SonOfGarr 29
- 30. red/black deployments 11/17/14 @SonOfGarr 30
- 31. canary releases 11/17/14 @SonOfGarr 31
- 32. inventing clusters 11/17/14 @SonOfGarr 32
- 33. without asgard aws ec2 create-security-group --group-name asgard-sg aws ec2 authorize-security-group-ingress --group-id sg-123456 --protocol tcp --port 22 --cidr 0.0.0.0/0 aws elb create-load-balancer --load-balancer-name asgard-lb –listeners Protocol=string,LoadBalancerPort=80,InstanceProtocol=http,InstancePort=7001 aws autoscaling create-launch-configuration --launch-configuration-name asgard-lc --image-id ami-XXX --key-name zerotocloud --security-groups asgard-sg --instance-type m3.medium --iam-instance-profile jumphost aws autoscaling create-auto-scaling-group --auto-scaling-group-name asgard-v000 --launch-configuration-name asgard-lc --min-size 1 --max-size 1 --desired-capacity 1 --availability-zones us-west-2b --load-balancer-names asgard-lb --tags ResourceId=string,ResourceType=string,Key=string,Value=string,PropagateAtLaunch=boolean 11/17/14 @SonOfGarr 33
- 34. CLOUD INFRASTRUCTURE 11/17/14 @SonOfGarr 34
- 35. service discovery? • ephemeral nature of the cloud – instances come and go – ASG’s come and go • need to register/de-register a server • load balance middle-tier services w/o public exposure 11/17/14 @SonOfGarr 35
- 36. eureka • REST-based server • Java-based client https://github.com/Netflix/eureka 11/17/14 @SonOfGarr 36
- 37. eureka 11/17/14 @SonOfGarr 37
- 38. state of the cloud? • your cloud is dynamic: – new instances – IP address reuse – new firewalls • I want to track changes over time – not provided by AWS 11/17/14 @SonOfGarr 38
- 39. edda • polls AWS resources • quick search via REST API • changes over time • architecture: – Scala – MongoDB backend 11/17/14 @SonOfGarr 39
- 40. cloud persistence • a variety of NoSQL persistence needs • utilize open source • Problems: – not cloud native – poor operational features http://techblog.netflix.com/2011/01/nosql-at-netflix.html 11/17/14 @SonOfGarr 40
- 41. sidecar 11/17/14 @SonOfGarr 41
- 42. priam Cassandra instance sidecar – backup and recovery – token management – Seed discovery – Configuration – Support AWS environment https://github.com/Netflix/priam 11/17/14 @SonOfGarr 42
- 43. raigad ElasticSearch instance sidecar – backup and restore – configured deployments – tribe node deployments – publish metrics – support AWS environment https://github.com/Netflix/raigad 11/17/14 @SonOfGarr 43
- 44. exhibitor ZooKeeper instance sidecar – Performs periodic backups – Perform periodic cleaning of ZK log directory – A GUI explorer for viewing ZK nodes – A rich REST API 11/17/14 @SonOfGarr 44
- 45. SERVICE ARCHITECTURE 11/17/14 @SonOfGarr 45
- 46. Java microservices 11/17/14 @SonOfGarr 46
- 47. ribbon • cloud IPC library – pluggable load balancing rules – eureka integration – archauis integration – cloud enabled https://github.com/Netflix/ribbon 11/17/14 @SonOfGarr 47
- 49. hystrix • Latency & fault tolerance for distributed systems – circuit breaker pattern – realtime operations https://github.com/Netflix/Hystrix 11/17/14 @SonOfGarr 49
- 51. hystrix dashboard https://github.com/Netflix/Hystrix/wiki/Dashboard 11/17/14 @SonOfGarr 51
- 52. archaius • Java library for dynamic runtime properties https://github.com/Netflix/archaius 11/17/14 @SonOfGarr 52
- 53. evcache • distributed in-memory caching – based on memchached & spymemcached • EVCache: – Elastic – Volatile – Cache https://github.com/Netflix/EVCache 11/17/14 @SonOfGarr 53
- 54. servo • expose applications metrics – gauges – counters – timers https://github.com/Netflix/servo 11/17/14 @SonOfGarr 54
- 55. astayanax • Java Cassandra client • fluent API • cloud native https://github.com/Netflix/astyanax 11/17/14 @SonOfGarr 55
- 56. governator • Google Guice extensions – classpath scanning – lifecycle management – field validation – lazy singleton https://github.com/netflix/Governator 11/17/14 @SonOfGarr 56
- 57. karyon • nucleus of Netflix platform – governator bootstrap – admin console – health checks – discovery https://github.com/Netflix/karyon 11/17/14 @SonOfGarr 57
- 58. prana Generic sidecar w/ HTTP access to Java clients for: – Eureka – Ribbon – Archauis – health checks 11/17/14 @SonOfGarr 58
- 59. CHAOS 11/17/14 @SonOfGarr 59
- 60. chaos monkey 11/17/14 @SonOfGarr 60
- 61. simian army • chaos monkey • janitor monkey • compliance monkey • security monkey https://github.com/Netflix/SimianArmy 11/17/14 @SonOfGarr 61
- 64. We are Hiring! Senior Cloud Tools Engineer http://nflx.it/140BnJN Senior Tooling Integration Engineer Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.mikemcgarr.com http://nflx.it/1xyhkeD 11/17/14 @SonOfGarr 64
Editor's Notes
- ----- Meeting Notes (10/9/14 12:28) ----- voice track here
- 47 repos
- ----- Meeting Notes (11/17/14 13:47) ----- redraw as two separate Amis
- -e = environment -b = starting AMI -n = use a named image aminator adds the –ebs to the AMI name
- problems: ZK statically configured limited tools to manage the ensemble backup/restore visualization periodic cleanup
- Ephemeral - The data stored is for a short duration as specified by its TTL(Time To Live). Volatile - The data can disappear any time (Evicted). Cache – An in-memory key-value store.