The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016
•Download as PPTX, PDF•
4 likes•662 views
The Envato market development team runs a two sided marketplace platform that powers sites such as themeforest.net and graphicriver.net. This presentation describes how they deploy the application up to 25 times a day while serving up to 200 million requests a week.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016
Similar to The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016 (20)
Recently uploaded
Recently uploaded (20)
The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016
- 1. The Anatomy of Continuous Deployment At Scale John Viner (Dev Manager) Mario Visic (Lead Developer of Delivery Engineering) 100 Deploys a Week at Envato
- 3. A two-sided Marketplace for creative assets that has paid out $400 million to our “Authors” in the last 10 years 7 Sites where digital professionals can buy web templates, video, audio, code, graphics and 3d models 7 million registered users 10 million items for sale
- 5. Show me the Tech Stack Ruby on Rails (100K loc, 25 web servers) MySql (175 tables) Memcached + Redis Rackspace / VMWare Amazon AWS
- 6. Show me the Numbers! Peak of 25,000 requests p/min through Rails 160m Page Views per Month 500,000 Database calls p/min Serving 200TB of data a month 22,000 Unit Tests, 2,000 Integration Tests
- 7. How do we deliver change?
- 8. By the numbers … for the month of June 2016 ~ 40 developers 3000 total builds (2.5 builds p/dev/day) 500 Pull Requests Closed 400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25) 4.5 hour cycle time (from first commit to deploy - working hours)
- 9. Statistics over Time Size of a Pull Request (median) 37 lines changed 3 files changed Cycle Time of a PR (first commit to deploy) average: 10.8 hours (working hours) median: 2.6 hours (working hours) Deployments 8400 deployments in 2 years
- 10. Pretty Charts - Builds
- 11. Deployments
- 12. Pull Requests
- 13. PR Cycle Time (First Commit to Deployment)
- 15. Feature request • 2 week sprints • Work with product team to break down work • No task larger than two days • Feature toggle can be used to push incomplete fe
- 16. Feature request
- 17. Implementation • Write the tests (sometimes first) • Write the code • Peer review • Get it into production
- 18. Implementation Tests live alongside code, stored in Github
- 19. Implementation Github Pull Request are the unit of change, 2 thumbs up by reviewers means the code is good to ship to production
- 20. Implementation
- 21. Implementation Buildkite is our CI tool, a merge on master automatically runs a build. A successful build automatically triggers a production deploy
- 22. Deployment • Entirely Automated • Post deploy monitoring • Quick rollback • Multiple time per hour
- 23. Deployment Samson (open source) is our deployment tool, successful master builds triggers a deploy which is logged in Samson.
- 24. Deployment We can run manual deployments from Samson and view all PRs in a given deployment.
- 25. Deployment We can trace back to specific changes in a deploy.
- 26. Deployment … including the amount of lines changed. A typical PR changes only 37 lines of code across 3 files
- 27. Deployment
- 28. Why do we do this?
- 29. Lowers Risk Size of Change is Small means … Easier to identify any failure Easier to rollback any failure Reduces human errors due to reliance on automation Security patches can be deployed within minutes
- 30. And also Reduces Waste: Fewer handovers from Development to QA to Ops Increases Product Agility: New features go to customers sooner to get earlier feedback Increases developer ownership and engagement: More direct connection between your creation and it being used
- 31. How to get there
- 32. Start with Culture Foster an Ownership Culture in Developers http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html
- 33. Investment in automation of unit tests, functional tests Tests to cover legacy systems Add Tests whenever un-tested code is touched Set Targets and Measure Test Coverage and # Tests
- 34. Automate Deployments and Enable Feature Toggles Automate your manual run sheet one line at a time Start with Push Button Feature Toggles help deploy features without showing them
- 35. Start Small Use a less risky or complicated system first
- 36. Investment in monitoring and rollback Knowing that the system is working within seconds becomes critical Developers need to be “on the pager” Invest in the ability to Roll Back Optimise for Mean Time to Recover
- 37. Thank you!
Editor's Notes
- So what’s all this about?
- The survey also contained an additional optional question that entered responses into the prize pool draw – over 3,500 people submitted responses to the question “How has Tuts+ changed your life.”
- A portion of the survey questions focussed on the cross over between all Envato brands. This is a quick wrap on some of those response stats – if you would like additional info please let us know.
- 42% of Tuts+ users have purchased an Envato Market item. And 6% have used a Studio service. We also asked them outside of Envato which other sites they had used – these are the top responses. 20% Shutterstock 18% iStock 14% WooThemes 11% Creative Market 11% Typekit 9% Template Monster 8% Getty Images 4% Mojo Themes
- 42% of Tuts+ users have purchased an Envato Market item. And 6% have used a Studio service. We also asked them outside of Envato which other sites they had used – these are the top responses. 20% Shutterstock 18% iStock 14% WooThemes 11% Creative Market 11% Typekit 9% Template Monster 8% Getty Images 4% Mojo Themes
- 42% of Tuts+ users have purchased an Envato Market item. And 6% have used a Studio service. We also asked them outside of Envato which other sites they had used – these are the top responses. 20% Shutterstock 18% iStock 14% WooThemes 11% Creative Market 11% Typekit 9% Template Monster 8% Getty Images 4% Mojo Themes
- 42% of Tuts+ users have purchased an Envato Market item. And 6% have used a Studio service. We also asked them outside of Envato which other sites they had used – these are the top responses. 20% Shutterstock 18% iStock 14% WooThemes 11% Creative Market 11% Typekit 9% Template Monster 8% Getty Images 4% Mojo Themes
- 42% of Tuts+ users have purchased an Envato Market item. And 6% have used a Studio service. We also asked them outside of Envato which other sites they had used – these are the top responses. 20% Shutterstock 18% iStock 14% WooThemes 11% Creative Market 11% Typekit 9% Template Monster 8% Getty Images 4% Mojo Themes
- So, throughout the survey we had a number of open ended questions. One of which was…