The Docker ecosystem and the future of application deployment
•
18 likes•2,940 views
Ten years ago, virtualization ignited a revolution which gave birth to the Cloud and the DevOps initiative. Today, with containers, we are at the dawn of a similar breakthrough. How can we capture the value of containers? How can we use their features to implement microservices and immutable infrastructures, while retaining as much as possible of our existing practices? The answer is in the rich ecosystem that developed around Docker, an open-source platform to build, ship, and run applications in containers. In this keynote we’ll explore what the applications of tomorrow will look like, how they’ll be deployed and distributed – and how to leverage those tools today.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (16)
Similar to The Docker ecosystem and the future of application deployment
Similar to The Docker ecosystem and the future of application deployment (20)
More from Jérôme Petazzoni
More from Jérôme Petazzoni (10)
Recently uploaded
Recently uploaded (20)
The Docker ecosystem and the future of application deployment
- 1. The Docker ecosystem and the future of application deployment 1 / 110
- 2. Bom dia! I'm Jérôme. 2 / 110
- 3. Bom dia! I'm Jérôme. First time in Brazil! 3 / 110
- 4. Bom dia! I'm Jérôme. First time in Brazil! Unfortunately, I don't speak Portuguese ☹ 4 / 110
- 5. Bom dia! I'm Jérôme. First time in Brazil! Unfortunately, I don't speak Portuguese ☹ Therefore, I will continue in English. Thank you! 5 / 110
- 6. Who am I? French software engineer living in California I have built and scaled the dotCloud PaaS I know a few things about running containers (in production) I ♥ Open Source I think Docker will help us to build better apps! 6 / 110
- 7. Outline Evolution vs Revolution Containers: evolution or revolution? How Docker can help us today How Docker will help us tomorrow 7 / 110
- 9. 9 / 110
- 10. First transatlantic flight Vickers Vimy modified bomber from World War I Altitude: 12,000 ft Speed: 115 mph Cross the Atlantic in hours instead of days 10 / 110
- 12. Revolution Paved the way for air travel Air travel was initially very expensive Ultimately became the main method for long-distance passenger travel 12 / 110
- 13. 13 / 110
- 14. First transatlantic supersonic jet Concorde Altitude: 60,000 ft Speed: 1,350 mph Cross the Atlantic even faster 14 / 110
- 16. Evolution Awesome technological achievement (Paris/New York in 3.5 hours!) We haven't done better since 50 years ago But: did not replace subsonic jets did not turn the impossible into possible 16 / 110
- 17. 17 / 110
- 18. Apollo missions Saturn V Altitude: the Moon (~1,300,000,000 ft) Speed: 24,500 mph Sends people to the Moon, and back 18 / 110
- 20. Revolution Except for those people who don't believe that we really went to the moon, of course... 20 / 110
- 21. We sent people to the Moon... Now what? 21 / 110
- 22. 22 / 110
- 23. Space program Altitude and speed: irrelevant Benefits: put satellites into orbit land rovers and probes on Mars, Titan... examine planets, comets, and more launch probes outside of the solar system Picture: Delta Rocket launching a GPS satellite 23 / 110
- 25. Revolution Numerous scientific breakthroughs And consumer applications: GPS satellite TV internet access in planes (!) etc. 25 / 110
- 26. How does this apply to us, developers? 26 / 110
- 27. 27 / 110
- 28. Physical machines Order time: hours Delivery time: days Install time: hours 28 / 110
- 29. 29 / 110
- 30. Virtual machines Order time: minutes Delivery time: minutes Install time: hours 30 / 110
- 31. 31 / 110
- 32. Virtual machines with an API Order time: seconds Delivery time: seconds Install time: minutes This is typically called "IaaS" or "Cloud" ☺ 32 / 110
- 34. 34 / 110
- 35. Problem Traffic patterns have daily peaks Low traffic during the night (and sometimes, low traffic during week-ends) Must provision servers according to the peaks Note: if you are dealing with worldwide traffic, its distribution changes during the day. 35 / 110
- 37. 37 / 110
- 38. Solution: autoscaling Add servers automatically, when load increases Remove them automatically, when load decreases Requires on-demand servers: available in minutes (not hours or days) elastic pricing model ("pay for what you use") automation (API) This was not possible before. 38 / 110
- 40. Problem Old model: developers write code, ops put it in production and run it Applications are increasingly complex It's increasingly hard for ops to fix problems It's increasingly hard for devs to anticipate them 40 / 110
- 42. 42 / 110
- 43. Solution: DevOps Design capacity and scaling upstream Devs and Ops in the same team Everybody is on call No more manual operations (racking machines, deploying OS...) Everything can be scripted, automated This was not possible before. 43 / 110
- 45. Problem Deploying new code is risky Consequence: we are afraid of deploying Consequence: we deploy less often Consequence: when we deploy, we deploy bigger changes Consequence: deploying new code is even riskier! 45 / 110
- 47. 47 / 110
- 48. Solution: blue/green deployment "Blue" servers are in production Deploy new release to "green" servers Progressively move traffic from blue to green If anything goes wrong: go back to blue Requires: 2x the number of servers (for a short period of time) ... or servers on demand (Cloud!) This was not possible/realistic before. 48 / 110
- 51. 51 / 110
- 52. Kitematic OS X GUI to start containers Order time: seconds Delivery time: seconds Install time: seconds 52 / 110
- 53. 53 / 110
- 54. 54 / 110
- 55. Docker CLI and API Order time: milliseconds Delivery time: milliseconds Install time: milliseconds 55 / 110
- 57. Containers seen by hosting industry Containers = lightweight virtual machines Higher density (3x-10x better than VMs) Faster boot times Possibility of bare metal performance $$$ 57 / 110
- 59. Come on, can we do better? 59 / 110
- 60. 60 / 110
- 61. 61 / 110
- 62. 62 / 110
- 63. Testing, continuous integration Testing environments must be: short-lived (tests can run in seconds) disposable (destroyed after the tests) quick to boot (so that tests can start rapidly) Containers are the perfect fit! 63 / 110
- 64. When a test depends on a database Plan A: use a common "test database" undefined state needs to be reloaded before each test slow (especially on big test sets) still vulnerable to concurrent access 64 / 110
- 65. When a test depends on a database Plan B: spin multiple copies of the DB in VMs expensive (spin up 100 VMs if you have 100 tests) slow (spin up 1 VM, test, repeat) complex to implement (even with the help of IAAS or Vagrant) 65 / 110
- 66. When a test depends on a database Plan C: spin multiple databases in containers cheap, low resource usage (even with big databases) fast boot times (even with big databases) easy to set up (thanks to CLI and API) 66 / 110
- 67. Testing / CI platforms were early adopters of Docker, before it was stable. Why? 67 / 110
- 68. 68 / 110
- 69. Testing is critical to release quality, but it does not affect uptime. (At least, not directly.) 69 / 110
- 70. I believe this is revolution. What do you think? 70 / 110
- 71. Docker can do more. 71 / 110
- 72. How Docker can help us today 72 / 110
- 73. 73 / 110
- 74. Software deployment before Docker Install packages A, B, C Install libraries X, Y, Z It doesn't work: wrong versions Try again Wasted time, frustration 74 / 110
- 75. Software deployment with Docker dockerrunjoaodubas/orientdb It works! Avoid "dependency hell" 75 / 110
- 76. Software dependencies before Docker This software depends on: library A version X library B version Y library C version Z other hidden, unspecified, unknown dependencies 76 / 110
- 77. Software dependencies with Docker This container image depends on: Intel 64 bits CPU instructions Linux kernel ABI Those things are very, very stable. 77 / 110
- 78. Software packaging before Docker Packaging is distribution-specific (different for Red Hat, Debian, OS X...) Packaging can also be language-specific (Node.js npm, Ruby gems, Python eggs...) Packaging is a useful, but uncommon skill (who here knows how to make a package?) Packaging is often done by separate people (ops, maintainers...) 78 / 110
- 79. Software packaging with Docker Developers can build container images easily Images are described by a Dockerfile If you know how to write a shell script, then you know how to write a Dockerfile! 79 / 110
- 81. Another example: tech support 81 / 110
- 82. Tech support before Docker 82 / 110
- 83. Tech support before Docker 83 / 110
- 84. Tech support before Docker "Works For Me" "Can't reproduce" "I would appreciate if you could test between 3 and 4am" "So to trigger the bug you have to install X and Y then configure A, B, and C, then download the extra file, put it in this directory (which doesn't exist?!?) and then if you restart three times in approximatively 5 minutes but sometimes it takes longer you will see that the images are shifted by a few pixels but if it doesn't work try to upgrade Y to version Z and try all over again..." 84 / 110
- 85. Tech support with Docker Get a well-defined, reproducible environment Define this environment in a Dockerfile Build this Dockerfileinto a container image Run this container image anywhere Get exactly the same behavior 85 / 110
- 87. Blue/green deployment before Docker Must create server images (slow) Server images are often specific to IAAS (cannot run them locally for testing) Servers must be stateless (data files are wiped out at each deploy) 87 / 110
- 88. Blue/green deployment with Docker Container image creation is fast Container images can be deployed anywhere (local dev env, physical servers, VMs...) Blue/green can be on the same machine, so data can be preserved across deploys 88 / 110
- 89. Onboarding new team members 89 / 110
- 90. Onboarding before Docker 90 / 110
- 91. Onboarding with Docker Hire developer* Give them a computer Install Docker gitclone... docker-composeup... Your stack is up and running *Actually the most difficult part. 91 / 110
- 93. Cold, hard data How long does it take for a developer to join a new project? Before Docker: 2 days After Docker: 2 hours (Source: Worldline) 93 / 110
- 95. Microservices before Docker Different languages, frameworks, build systems Service discovery challenge Differences between dev and prod environments: dev = many processes on one VM prod = many processes on many VMs different network layout network layout is visible to the processes 95 / 110
- 96. Microservices with Docker Everybody ships containers docker-composedoes service discovery in dev, and offers reliable setup of the whole stack Differences between dev and prod environments: dev = many containers on one VM prod = many containers on many VMs different network layout network layout is invisible to the processes 96 / 110
- 97. I vote for: revolution 97 / 110
- 98. How Docker will help us tomorrow 98 / 110
- 99. Docker Machine 99 / 110
- 100. Docker Machine Create Docker hosts with a single tool docker-machinecreate-dvirtualboxmy-dev docker-machinecreate-ddigitaloceanprod-001 docker-machinecreate-damazonec2prod-002 Work in progress! 100 / 110
- 101. Docker Swarm 101 / 110
- 102. Docker Swarm Control a cluster of Docker hosts, as if it were a single host Talk to Swarm using the Docker API Use the Docker CLI or any existing Docker client Swarm talks to Docker hosts with the same API No change to existing clients and hosts! 102 / 110
- 103. Scaling Swarm with Mesos On large-scale clusters (100s, 1000s of nodes), scheduling is a hard problem Mesos solves the scheduling problem elegantly Swarm and Mesos will integrate: you talk to Swarm with the Docker API, Swarm offloads the hard job to Mesos 103 / 110
- 104. Network extensions Build overlay networks to connect containers across multiple hosts and multiple clouds Integrate Docker containers with existing networks (Open vSwitch, VLANs, VXLAN, etc.) Some extensions: flannel pipework socketplane weave 104 / 110
- 105. Storage extensions Stateful containers (databases) are challenging Moving a stateless container is easy (deploy image; start container; done) Moving a stateful container is hard See: Flocker by ClusterHQ 105 / 110
- 106. Docker on the Desktop What for? better dependency management for proprietary binaries (e.g. Chrome, Skype, Spotify...) better control over resources (both quantitative and qualitative) Check Jessica Frazelle's blog: https://blog.jessfraz.com/posts/docker-containers-on-the-desktop.html 106 / 110
- 107. CRIU (Checkpoint/Restore In Userspace) Save process state (not just filesystem state!) Obvious use-case: live migration of containers Less obvious, but easier use-cases: slow start processes JIT compilers 107 / 110
- 108. Recap 108 / 110
- 109. Recap Build with Dockerfiles. Test rapidly. Assemble complex stacks with docker-compose. Implement powerful software patterns easily: immutable infrastructure, microservice architectures. Manage machines with docker-machine. Orchestrate clusters with Swarm. Leverage the numerous tools in the ecosystem. Last but not least: all of this is Open Source. 109 / 110