Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Anatomy of Continuous Deployment
At Scale
John Viner (Dev Manager)
Mario Visic (Lead Developer of Delivery Engineering...
Who is Envato?
A two-sided Marketplace for creative assets that
has paid out $400 million to our “Authors” in the last
10 years
7 Sites w...
Themeforest.net
Show me the Tech Stack
Ruby on Rails (100K loc, 25 web servers)
MySql (175 tables)
Memcached + Redis
Rackspace / VMWare
Am...
Show me the Numbers!
Peak of 25,000 requests p/min through Rails
160m Page Views per Month
500,000 Database calls p/min
Se...
How do we deliver change?
By the numbers … for the month of June 2016
~ 40 developers
3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Clo...
Statistics over Time
Size of a Pull Request (median)
37 lines changed
3 files changed
Cycle Time of a PR (first commit to ...
Pretty Charts - Builds
Deployments
Pull Requests
PR Cycle Time (First Commit to Deployment)
Building Envato Market
Feature request
• 2 week sprints
• Work with product team to break down work
• No task larger than two days
• Feature togg...
Feature request
Implementation
• Write the tests (sometimes first)
• Write the code
• Peer review
• Get it into production
Implementation
Tests live alongside code, stored in Github
Implementation
Github Pull Request are the unit of change, 2 thumbs up by reviewers
means the code is good to ship to prod...
Implementation
Implementation
Buildkite is our CI tool, a merge on master automatically runs a build.
A successful build automatically tr...
Deployment
• Entirely Automated
• Post deploy monitoring
• Quick rollback
• Multiple time per hour
Deployment
Samson (open source) is our deployment tool, successful master
builds triggers a deploy which is logged in Sams...
Deployment
We can run manual deployments from Samson and view all PRs in a
given deployment.
Deployment
We can trace back to specific changes in a deploy.
Deployment
… including the amount of lines changed. A typical PR changes only
37 lines of code across 3 files
Deployment
Why do we do this?
Lowers Risk
Size of Change is Small means …
Easier to identify any failure
Easier to rollback any failure
Reduces human er...
And also
Reduces Waste:
Fewer handovers from Development to QA to Ops
Increases Product Agility:
New features go to custom...
How to get there
Start with Culture
Foster an Ownership Culture in Developers
http://webuild.envato.com/expectations/collaboration/4-i-take...
Investment in automation of unit tests, functional tests
Tests to cover legacy systems
Add Tests whenever un-tested code i...
Automate Deployments and Enable Feature Toggles
Automate your manual run sheet one line at a time
Start with Push Button
F...
Start Small
Use a less risky or complicated system first
Investment in monitoring and rollback
Knowing that the system is working within seconds becomes
critical
Developers need t...
Thank you!
Upcoming SlideShare
Loading in …5
×

The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016

565 views

Published on

The Envato market development team runs a two sided marketplace platform that powers sites such as themeforest.net and graphicriver.net. This presentation describes how they deploy the application up to 25 times a day while serving up to 200 million requests a week.

Published in: Technology
  • Login to see the comments

The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016

  1. 1. The Anatomy of Continuous Deployment At Scale John Viner (Dev Manager) Mario Visic (Lead Developer of Delivery Engineering) 100 Deploys a Week at Envato
  2. 2. Who is Envato?
  3. 3. A two-sided Marketplace for creative assets that has paid out $400 million to our “Authors” in the last 10 years 7 Sites where digital professionals can buy web templates, video, audio, code, graphics and 3d models 7 million registered users 10 million items for sale
  4. 4. Themeforest.net
  5. 5. Show me the Tech Stack Ruby on Rails (100K loc, 25 web servers) MySql (175 tables) Memcached + Redis Rackspace / VMWare Amazon AWS
  6. 6. Show me the Numbers! Peak of 25,000 requests p/min through Rails 160m Page Views per Month 500,000 Database calls p/min Serving 200TB of data a month 22,000 Unit Tests, 2,000 Integration Tests
  7. 7. How do we deliver change?
  8. 8. By the numbers … for the month of June 2016 ~ 40 developers 3000 total builds (2.5 builds p/dev/day) 500 Pull Requests Closed 400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25) 4.5 hour cycle time (from first commit to deploy - working hours)
  9. 9. Statistics over Time Size of a Pull Request (median) 37 lines changed 3 files changed Cycle Time of a PR (first commit to deploy) average: 10.8 hours (working hours) median: 2.6 hours (working hours) Deployments 8400 deployments in 2 years
  10. 10. Pretty Charts - Builds
  11. 11. Deployments
  12. 12. Pull Requests
  13. 13. PR Cycle Time (First Commit to Deployment)
  14. 14. Building Envato Market
  15. 15. Feature request • 2 week sprints • Work with product team to break down work • No task larger than two days • Feature toggle can be used to push incomplete fe
  16. 16. Feature request
  17. 17. Implementation • Write the tests (sometimes first) • Write the code • Peer review • Get it into production
  18. 18. Implementation Tests live alongside code, stored in Github
  19. 19. Implementation Github Pull Request are the unit of change, 2 thumbs up by reviewers means the code is good to ship to production
  20. 20. Implementation
  21. 21. Implementation Buildkite is our CI tool, a merge on master automatically runs a build. A successful build automatically triggers a production deploy
  22. 22. Deployment • Entirely Automated • Post deploy monitoring • Quick rollback • Multiple time per hour
  23. 23. Deployment Samson (open source) is our deployment tool, successful master builds triggers a deploy which is logged in Samson.
  24. 24. Deployment We can run manual deployments from Samson and view all PRs in a given deployment.
  25. 25. Deployment We can trace back to specific changes in a deploy.
  26. 26. Deployment … including the amount of lines changed. A typical PR changes only 37 lines of code across 3 files
  27. 27. Deployment
  28. 28. Why do we do this?
  29. 29. Lowers Risk Size of Change is Small means … Easier to identify any failure Easier to rollback any failure Reduces human errors due to reliance on automation Security patches can be deployed within minutes
  30. 30. And also Reduces Waste: Fewer handovers from Development to QA to Ops Increases Product Agility: New features go to customers sooner to get earlier feedback Increases developer ownership and engagement: More direct connection between your creation and it being used
  31. 31. How to get there
  32. 32. Start with Culture Foster an Ownership Culture in Developers http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html
  33. 33. Investment in automation of unit tests, functional tests Tests to cover legacy systems Add Tests whenever un-tested code is touched Set Targets and Measure Test Coverage and # Tests
  34. 34. Automate Deployments and Enable Feature Toggles Automate your manual run sheet one line at a time Start with Push Button Feature Toggles help deploy features without showing them
  35. 35. Start Small Use a less risky or complicated system first
  36. 36. Investment in monitoring and rollback Knowing that the system is working within seconds becomes critical Developers need to be “on the pager” Invest in the ability to Roll Back Optimise for Mean Time to Recover
  37. 37. Thank you!

×