Freeing the Whale: How to Fail at Scale
•
0 likes•395 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2n9SPpw. Oliver Gould discusses Finagle, a library providing a uniform model for handling failure at the communications layer, enabling Twitter to fail, safely and often. Filmed at qconsf.com. Oliver Gould is the CTO of Buoyant, where he leads open source development efforts. Prior to joining Buoyant, he was a staff infrastructure engineer at Twitter, where he was the technical lead of Observability, Traffic, and Configuration & Coordination teams.
Recommended
Recommended
More Related Content
Similar to Freeing the Whale: How to Fail at Scale
Similar to Freeing the Whale: How to Fail at Scale (20)
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Freeing the Whale: How to Fail at Scale
- 1. Freeing the Whale How to Fail at Scale oliver gould cto, buoyant QConSF, November 9, 2016 from
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ twitter-finagle
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 7. Twitter, 2010 107 users 107 tweets/day 102 engineers 101 ops eng 101 services 101 deploys/week 102 hosts 0 datacenters 101 user-facing outages/week https://blog.twitter.com/2010/measuring-tweets
- 9. objective reliability flexibility solution platform SOA + devops i.e. “microservices”
- 10. Resilience is an imperative: our software runs on the truly dismal computers we call datacenters. Besides being heinously complex… they are unreliable and prone to operator error. Marius Eriksen @marius RPC Redux
- 11. software you didn’t write hardware you can’t touch network you can’t trace break in new and surprising ways and your customers shouldn’t notice
- 12. freeing the whale photo: Johanan Ottensooser
- 13. mesos.apache.org UC Berkeley, 2010 Twitter, 2011 Apache, 2012 Abstracts compute resources Promise: don’t worry about the hosts
- 14. aurora.apache.org Twitter, 2011 Apache, 2013 Schedules processes on Mesos Promise: no more puppet, monit, etc
- 15. timelines Aurora (or Marathon, or …) host Mesos host host host host host users notifications x800 x300 x1000
- 16. timelines Aurora (or Marathon, or …) host Mesos host host host host users notifications x800 x300 x1000 🔥
- 17. service discovery timelines users zookeeper create ephemeral /svc/users/node_012345 {“host”: “host-abc”,“port”: 4321}
- 18. service discovery timelines users zookeeper watch /svc/users/*
- 20. service discovery timelines users zookeeper uh oh. GetUser(olix0r)
- 21. service discovery timelines users zookeeper client caches results GetUser(olix0r)
- 22. service discovery timelines users zookeeper GetUser(olix0r) zookeeper serves empty results?!
- 23. service discovery timelines users zookeeper service discovery is advisory GetUser(olix0r)
- 24. github.com/twitter/finagle RPC library (JVM) asynchronous built on Netty scala functional strongly typed first commit: Oct 2010
- 25. datacenter [1] physical [2] link [3] network [4] transport kubernetes, mesos, swarm, … canal, weave, … aws, azure, digitalocean, gce, … business languages, libraries[7] application rpc [5] session [6] presentation json, protobuf, thrift, … http/2, mux, …
- 26. “It’s slow” is the hardest problem you’ll ever debug. Jeff Hodges @jmhodges Notes on Distributed Systems for Young Bloods
- 27. observability counters (e.g. client/users/failures) histograms (e.g. client/users/latency/p99) tracing
- 28. tracing
- 33. retries typical: retries=3 worst-case: 300% more load!!!
- 34. budgets typical: retries=3 better: retryBudget=20% worst-case: 300% more load!!! worst-case: 20% more load
- 35. load shedding via cancellation timelines users web db timelines users web db timeout!
- 36. load shedding via cancellation timelines users web db timelines users web db timeout!
- 37. backpressure timelines users web db timelines users web db 1000 requests 100 requests 1000 requests
- 39. backpressure timelines users web db 100 ok 100 ok 100 ok + 900 failed/redirected/etc
- 40. lb algorithms: • round-robin • fewest connections • queue depth • exponentially-weighted moving average (ewma) • aperture request-level load balancing
- 42. So just rewrite everything in Finagle!?
- 43. linkerd
- 44. github.com/buoyantio/linkerd service mesh proxy built on finagle & netty suuuuper pluggable http, thrift, … etcd, consul, kubernetes, marathon, zookeeper, … …
- 45. Linkers and Loaders, John R. Levine, Academic Press
- 46. linker for the datacenter
- 47. logical naming applications refer to logical names requests are bound to concrete names delegations express routing /s/users /#/io.l5d.zk/prod/users /#/io.l5d.zk/staging/users /s => /#/io.l5d.zk/prod
- 48. per-request routing: staging GET / HTTP/1.1 Host: mysite.com l5d-dtab: /s/B => /s/B2
- 49. per-request routing: debug proxy GET / HTTP/1.1 Host: mysite.com l5d-dtab: /s/E => /s/P/s/E
- 50. linkerd service mesh transport security service discovery circuit breaking backpressure deadlines retries tracing metrics keep-alive multiplexing load balancing per-request routing service-level objectives Service B instance linkerd Service C instance linkerd Service A instance linkerd
- 56. linkerd roadmap • Battle test HTTP/2 • TLS client certs • Deadlines • Dark Traffic • All configurable everything
- 57. more at linkerd.io slack: slack.linkerd.io email: ver@buoyant.io twitter: • @olix0r • @linkerd thanks!
- 58. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ twitter-finagle