SlideShare a Scribd company logo
1 of 31
Download to read offline
10 Deploys a Day
a case study of

continuous
delivery at
envato
AT

john viner
PRESENTATION TEMPLATE from Envato’s Graphic River - http:/
/graphicriver.net/item/karbon-keynote-presentation-template/2580765
envato ABOUT US
7 years ago - a designer scratching an itch
creates a flash-plugin marketplace

and now …
‣
‣
‣

3.2m members
4.6m Items
10 Authors sold > $1m

‣
‣
‣

130m PVs p/m
20M Visits p/M
Item sold every 10 sec
Envato

our marketplace Sites
Envato

wordpress - our biggest market
> 3400+ Themes for Sale

> Most Popular Theme has sold 37,500 times and generated $2m in
gross revenue

> Average price of a Theme is $42

> Average sales per TF author is 2,075

> Provides a Passive Income source

!

19% of the web runs on Wordpress! (4B monthly page view)
Envato

we’ve grown - and continue to grow
> Themeforest is now Ranked ~ #180 on the Alexa Rankings

> My development team was 8, 15 months ago and is now 24
Envato

the marketplace team
40 - in melbourne
‣ 30 - remote around the world
‣

‣

delivery team
- 19 Back End Developers

- 2 Front End Developers

- 1 Operations Developer

- 5 Product Managers and UX Designers


6 Teams
‣ finance,
‣ back office,
‣ front end,
‣ maintenance
‣ search
‣ 10x
‣
Envato

marketplace tech stack
ruby on rails web app
‣ mysql database
‣ elastic Search
‣ rackspace managed hosting in USA
‣

> 90M App Server Requests per Week

> Backend Response time of 148 ms

> Front end Response time of 2.5sec

> 45 Virtual Machines 

> 4 Physical Machines (DBs, NFS)
> Avg 6 deploys a day - Peak of 10

> Average 100 commits a day 

> Peak Load of 15,000 req per minute

> 4 Physical Machines (DBs etc…)
THIS IS HOW WE ROLL
Envato

pull work, talk to Product, write code, test code
deploy code, verify production
repeat … about twice a week per developer
> Story on the wall
> flip or no flip?
> Create a Branch
> write failing test
> write code
> Run Local Tests
> Run full personal build
> create Pull request
> advertise pull request (IRC)
!

optional
> deploy to staging
> show product owner
> manual test

> wait for +1s (code review)
> Merge Pull request
> Full Master Build Green
> notify team in IRC
> deploy master to Production
> watch System - monitoring tools
> rollback if failed
> Rinse and Repeat
ENVATO

in pictures

Story
Tests
Code
ENVATO

flippin good
>Flip allows us to roll out changes to % of users

> Something might be flipped off for months

> But we Always Deploy It
ENVATO

quality
Code Review

CI Build

Deploy
coordinating reviews
and deploys .. by irc
You Need Deep Dev and
Operational Experience right?
> Luke - Graduate with 4 months Exp

> Rakesh - Swinburne Industry Student

> Ben - Front End Developer

> Emmanuel - 2 yrs Dev Exp
4 Other Developers Mentioned

Across 4 Separate Teams
ENVATO

now check the systems

New Relic
Deploy

With Change
Traceability
things can break …
And Do
ENVATO

deploys gone bad

Rollbar Error Logging
Envato

the post incident review recommends?
>1 - Improve our ability to Automatically rollback on monitored 500 errors

!

> 2 - NO Recommendation to Improve Preventative Testing

!

> Why? - Too Costly and complex to simulate production

> Why? - Would slow development flow
so who operates this
site anyway?
Envato

1 developer all developers

you build it
you run it

> 16 Developers on a Weekly Rotation

> They come out of their team

> They are Trained, have a Buddy for First Week

> Supported by a Secondary 

> They are First Line of Support for 168 hours straight
Tooling is:

> Scout , New Relic and Pingdom monitoring and generating alerts

> Pagerduty forwarding to them On-Call Developer

> Rollbar to track Errors

> IRC to find out what went wrong

> ALL Alerts are also pumped into IRC so everyone knows whats going on and to get Help!
Envato

we reflect on failures

and act to correct
Envato

but we aren’t great at operations

> We are just Good Enough right now, Just

> The rotation leaves without a continuous view of system health

> We don’t always clean up behind us, root causes often come back

> Monitoring, Metrics, Performance insights are all Operational Skills
that many developers don’t have
> We are hiring Operational Specialists for the first time

> Our Single DevOps guy is actively training all the devs - AJs Cafe
Envato

so what do we not do
> No - testing role or formal testing gate
> NO - change control gate
> No - formal comms of every change
> NO - release management
> No - separate deployment team
> NO - Separate operations team

all of these are not feasible
at 10 deploys a day
Envato

how do we solve the problems that these practices solve ?
‣

No Testing Capability

> TDD and 1000s of automated tests

> Use Production Users as your test team

> Flips help us control the rollout

> We test major infrastructure and architecture
changes in production as well … using Flips

> We Fail Fast and Loud
‣

No release management

> Releases are SMALL, no need to define
or document

> Release self-document and radiate
through IRC

‣

No change control
> Operations (i.e. Developers) know of changes
before they happen because they review them as PRs

> There are changes we cannot “signoff” as testing in
pre-prod environments

> A green build is our sign off

> Devs have skin-in-game so are incentivised for the
code to work!

> Small Batch efficiency (small deltas are easier to
identify and rollback)
Envato

OK How did we do it?
‣

it was our startup dNA
> The Team started and stayed very small (less
than 8 devs) for 4 years

> They started with this structure and process
and have never changed.

‣

it scales

> We’ve tripled the team, and split into 5 streams

> We’ve been able to scale this model
‣

but with a lag

> But it took 3 months to see the results
Envato

so do you want to drink the kooL-aid?
‣

it might not be for you

‣

> We have a tolerant savvy users

> We have users that come back to
complete a purchase if it fails

> We are in CONTROL of our full stack

‣

If you want it, you have to change

> people
‣ > processes
‣ > assumptions

‣‣

> Tech is the easy part

> You can’t optimise your “gatekeeping” processes to 10 deploys


a leap is required

> Existing processes can get you close, say once a fortnight or a week

> A leap is required to get to many times a day
Envato

distributed collaboration vs central control
quality
‣ release planning
‣ deployment
‣ operations
‣

!
did I say Continuous
Delivery or deployment?

meet the real
rockstars at
Envato …
Envato

the microlancer team
automatically deploy every
commit to master … always
Envato

whats the difference?
Envato

what our next challenges?
continuous automated reactive deployments
‣ flips + split testing and with every product feature
‣ flexibility of our infrastructure
‣ improved operations
‣

!
envato THANK YOU
@envato
@veztek
webuild.envato.com

More Related Content

What's hot

The everyday developer's guide to version control with Git
The everyday developer's guide to version control with GitThe everyday developer's guide to version control with Git
The everyday developer's guide to version control with GitE Carter
 
Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Noa Harel
 
Git One Day Training Notes
Git One Day Training NotesGit One Day Training Notes
Git One Day Training Notesglen_a_smith
 
Introduction to git hub
Introduction to git hubIntroduction to git hub
Introduction to git hubNaveen Pandey
 
GIT presentation
GIT presentationGIT presentation
GIT presentationNaim Latifi
 
Git and Github slides.pdf
Git and Github slides.pdfGit and Github slides.pdf
Git and Github slides.pdfTilton2
 
Introduction to React JS
Introduction to React JSIntroduction to React JS
Introduction to React JSArno Lordkronos
 
Dealing with Merge Conflicts in Git
Dealing with Merge Conflicts in GitDealing with Merge Conflicts in Git
Dealing with Merge Conflicts in Gitgittower
 
Introduction to Gitlab | Gitlab 101 | Training Session
Introduction to Gitlab | Gitlab 101 | Training SessionIntroduction to Gitlab | Gitlab 101 | Training Session
Introduction to Gitlab | Gitlab 101 | Training SessionAnwarul Islam
 

What's hot (20)

Git training v10
Git training v10Git training v10
Git training v10
 
GitHub Presentation
GitHub PresentationGitHub Presentation
GitHub Presentation
 
The everyday developer's guide to version control with Git
The everyday developer's guide to version control with GitThe everyday developer's guide to version control with Git
The everyday developer's guide to version control with Git
 
Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Introducing GitLab (June 2018)
Introducing GitLab (June 2018)
 
Git One Day Training Notes
Git One Day Training NotesGit One Day Training Notes
Git One Day Training Notes
 
Git and github 101
Git and github 101Git and github 101
Git and github 101
 
Git & GitHub for Beginners
Git & GitHub for BeginnersGit & GitHub for Beginners
Git & GitHub for Beginners
 
Introduction to Git and Github
Introduction to Git and GithubIntroduction to Git and Github
Introduction to Git and Github
 
Git basic
Git basicGit basic
Git basic
 
Git Rebase vs Merge
Git Rebase vs MergeGit Rebase vs Merge
Git Rebase vs Merge
 
Introduction to git hub
Introduction to git hubIntroduction to git hub
Introduction to git hub
 
Git and Github Session
Git and Github SessionGit and Github Session
Git and Github Session
 
GIT presentation
GIT presentationGIT presentation
GIT presentation
 
Git and Github slides.pdf
Git and Github slides.pdfGit and Github slides.pdf
Git and Github slides.pdf
 
Introduction to React JS
Introduction to React JSIntroduction to React JS
Introduction to React JS
 
Intro to Git and GitHub
Intro to Git and GitHubIntro to Git and GitHub
Intro to Git and GitHub
 
Introduction git
Introduction gitIntroduction git
Introduction git
 
Dealing with Merge Conflicts in Git
Dealing with Merge Conflicts in GitDealing with Merge Conflicts in Git
Dealing with Merge Conflicts in Git
 
Git Pull Requests
Git Pull RequestsGit Pull Requests
Git Pull Requests
 
Introduction to Gitlab | Gitlab 101 | Training Session
Introduction to Gitlab | Gitlab 101 | Training SessionIntroduction to Gitlab | Gitlab 101 | Training Session
Introduction to Gitlab | Gitlab 101 | Training Session
 

Similar to 10 Deploys a Day - A Case Study of Continuous Delivery at Envato

Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...
Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...
Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...AgileNetwork
 
DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015Yuval Yeret
 
Continues Deployment - Tech Talk week
Continues Deployment - Tech Talk weekContinues Deployment - Tech Talk week
Continues Deployment - Tech Talk weekrantav
 
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015Yuval Yeret
 
DevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsDevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsNitin Bhide
 
DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)Nitin Bhide
 
Continuously delivering software to big brands (fullscreen edition)
Continuously delivering software to big brands (fullscreen edition)Continuously delivering software to big brands (fullscreen edition)
Continuously delivering software to big brands (fullscreen edition)Paul Boocock
 
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Tieturi Oy
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous DeploymentSmartling
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code DeploysDevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code DeploysAndreas Grabner
 
Unit Test + Functional Programming = Love
Unit Test + Functional Programming = LoveUnit Test + Functional Programming = Love
Unit Test + Functional Programming = LoveAlvaro Videla
 
Jonny wooldridge DevOps Large and Small
Jonny wooldridge DevOps Large and SmallJonny wooldridge DevOps Large and Small
Jonny wooldridge DevOps Large and SmallJwooldridge
 
Frappé / ERPNext Open Day February 2016
Frappé / ERPNext Open Day February 2016Frappé / ERPNext Open Day February 2016
Frappé / ERPNext Open Day February 2016Anand Doshi
 
How to approach Enterprise transformation
How to approach Enterprise transformationHow to approach Enterprise transformation
How to approach Enterprise transformationMirco Hering
 
Jan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsJan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsAgile Lietuva
 
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.02014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0Joakim Lindbom
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOpsJoão Miranda
 
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...Gene Kim
 
How NOT to scale agile
How NOT to scale agileHow NOT to scale agile
How NOT to scale agileRenato Willi
 

Similar to 10 Deploys a Day - A Case Study of Continuous Delivery at Envato (20)

Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...
Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...
Agile Gurgaon 2016 | Thinking Beyond :: Marry Agile and DevOps for Phenomenal...
 
DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015
 
Continues Deployment - Tech Talk week
Continues Deployment - Tech Talk weekContinues Deployment - Tech Talk week
Continues Deployment - Tech Talk week
 
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015
DevOps - the Future of Agile - Why/What/How - from Enterprise DevOps Israel 2015
 
DevOps - Understanding Core Concepts
DevOps - Understanding Core ConceptsDevOps - Understanding Core Concepts
DevOps - Understanding Core Concepts
 
DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)
 
Continuously delivering software to big brands (fullscreen edition)
Continuously delivering software to big brands (fullscreen edition)Continuously delivering software to big brands (fullscreen edition)
Continuously delivering software to big brands (fullscreen edition)
 
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
Testaus 2014 -seminaari: Arto Kiiskinen, Mirasys Oy. Case Mirasys: Toiminnoil...
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code DeploysDevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
 
Enterprise DevOps
Enterprise DevOps Enterprise DevOps
Enterprise DevOps
 
Unit Test + Functional Programming = Love
Unit Test + Functional Programming = LoveUnit Test + Functional Programming = Love
Unit Test + Functional Programming = Love
 
Jonny wooldridge DevOps Large and Small
Jonny wooldridge DevOps Large and SmallJonny wooldridge DevOps Large and Small
Jonny wooldridge DevOps Large and Small
 
Frappé / ERPNext Open Day February 2016
Frappé / ERPNext Open Day February 2016Frappé / ERPNext Open Day February 2016
Frappé / ERPNext Open Day February 2016
 
How to approach Enterprise transformation
How to approach Enterprise transformationHow to approach Enterprise transformation
How to approach Enterprise transformation
 
Jan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wantsJan de Vries - How to convince your boss that it is DevOps that he wants
Jan de Vries - How to convince your boss that it is DevOps that he wants
 
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.02014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...
DOES14 - Jonny Wooldridge - The Cambridge Satchel Company - 10 Enterprise Tip...
 
How NOT to scale agile
How NOT to scale agileHow NOT to scale agile
How NOT to scale agile
 

Recently uploaded

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 

Recently uploaded (20)

Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 

10 Deploys a Day - A Case Study of Continuous Delivery at Envato

  • 1. 10 Deploys a Day a case study of continuous delivery at envato AT john viner PRESENTATION TEMPLATE from Envato’s Graphic River - http:/ /graphicriver.net/item/karbon-keynote-presentation-template/2580765
  • 2. envato ABOUT US 7 years ago - a designer scratching an itch creates a flash-plugin marketplace and now … ‣ ‣ ‣ 3.2m members 4.6m Items 10 Authors sold > $1m ‣ ‣ ‣ 130m PVs p/m 20M Visits p/M Item sold every 10 sec
  • 4. Envato wordpress - our biggest market > 3400+ Themes for Sale > Most Popular Theme has sold 37,500 times and generated $2m in gross revenue > Average price of a Theme is $42 > Average sales per TF author is 2,075 > Provides a Passive Income source ! 19% of the web runs on Wordpress! (4B monthly page view)
  • 5. Envato we’ve grown - and continue to grow > Themeforest is now Ranked ~ #180 on the Alexa Rankings > My development team was 8, 15 months ago and is now 24
  • 6. Envato the marketplace team 40 - in melbourne ‣ 30 - remote around the world ‣ ‣ delivery team - 19 Back End Developers - 2 Front End Developers - 1 Operations Developer - 5 Product Managers and UX Designers 6 Teams ‣ finance, ‣ back office, ‣ front end, ‣ maintenance ‣ search ‣ 10x ‣
  • 7. Envato marketplace tech stack ruby on rails web app ‣ mysql database ‣ elastic Search ‣ rackspace managed hosting in USA ‣ > 90M App Server Requests per Week > Backend Response time of 148 ms > Front end Response time of 2.5sec > 45 Virtual Machines > 4 Physical Machines (DBs, NFS) > Avg 6 deploys a day - Peak of 10 > Average 100 commits a day > Peak Load of 15,000 req per minute > 4 Physical Machines (DBs etc…)
  • 8. THIS IS HOW WE ROLL
  • 9. Envato pull work, talk to Product, write code, test code deploy code, verify production repeat … about twice a week per developer > Story on the wall > flip or no flip? > Create a Branch > write failing test > write code > Run Local Tests > Run full personal build > create Pull request > advertise pull request (IRC) ! optional > deploy to staging > show product owner > manual test > wait for +1s (code review) > Merge Pull request > Full Master Build Green > notify team in IRC > deploy master to Production > watch System - monitoring tools > rollback if failed > Rinse and Repeat
  • 11. ENVATO flippin good >Flip allows us to roll out changes to % of users > Something might be flipped off for months > But we Always Deploy It
  • 13. coordinating reviews and deploys .. by irc You Need Deep Dev and Operational Experience right? > Luke - Graduate with 4 months Exp > Rakesh - Swinburne Industry Student > Ben - Front End Developer > Emmanuel - 2 yrs Dev Exp 4 Other Developers Mentioned Across 4 Separate Teams
  • 14. ENVATO now check the systems New Relic Deploy With Change Traceability
  • 15. things can break … And Do
  • 17. Envato the post incident review recommends? >1 - Improve our ability to Automatically rollback on monitored 500 errors ! > 2 - NO Recommendation to Improve Preventative Testing ! > Why? - Too Costly and complex to simulate production > Why? - Would slow development flow
  • 18. so who operates this site anyway?
  • 19. Envato 1 developer all developers you build it you run it > 16 Developers on a Weekly Rotation > They come out of their team > They are Trained, have a Buddy for First Week > Supported by a Secondary > They are First Line of Support for 168 hours straight Tooling is: > Scout , New Relic and Pingdom monitoring and generating alerts > Pagerduty forwarding to them On-Call Developer > Rollbar to track Errors > IRC to find out what went wrong > ALL Alerts are also pumped into IRC so everyone knows whats going on and to get Help!
  • 20. Envato we reflect on failures and act to correct
  • 21. Envato but we aren’t great at operations > We are just Good Enough right now, Just > The rotation leaves without a continuous view of system health > We don’t always clean up behind us, root causes often come back > Monitoring, Metrics, Performance insights are all Operational Skills that many developers don’t have > We are hiring Operational Specialists for the first time > Our Single DevOps guy is actively training all the devs - AJs Cafe
  • 22. Envato so what do we not do > No - testing role or formal testing gate > NO - change control gate > No - formal comms of every change > NO - release management > No - separate deployment team > NO - Separate operations team all of these are not feasible at 10 deploys a day
  • 23. Envato how do we solve the problems that these practices solve ? ‣ No Testing Capability > TDD and 1000s of automated tests > Use Production Users as your test team > Flips help us control the rollout > We test major infrastructure and architecture changes in production as well … using Flips > We Fail Fast and Loud ‣ No release management > Releases are SMALL, no need to define or document > Release self-document and radiate through IRC ‣ No change control > Operations (i.e. Developers) know of changes before they happen because they review them as PRs > There are changes we cannot “signoff” as testing in pre-prod environments > A green build is our sign off > Devs have skin-in-game so are incentivised for the code to work! > Small Batch efficiency (small deltas are easier to identify and rollback)
  • 24. Envato OK How did we do it? ‣ it was our startup dNA > The Team started and stayed very small (less than 8 devs) for 4 years > They started with this structure and process and have never changed. ‣ it scales > We’ve tripled the team, and split into 5 streams > We’ve been able to scale this model ‣ but with a lag > But it took 3 months to see the results
  • 25. Envato so do you want to drink the kooL-aid? ‣ it might not be for you ‣ > We have a tolerant savvy users > We have users that come back to complete a purchase if it fails > We are in CONTROL of our full stack ‣ If you want it, you have to change > people ‣ > processes ‣ > assumptions ‣‣ > Tech is the easy part > You can’t optimise your “gatekeeping” processes to 10 deploys a leap is required > Existing processes can get you close, say once a fortnight or a week > A leap is required to get to many times a day
  • 26. Envato distributed collaboration vs central control quality ‣ release planning ‣ deployment ‣ operations ‣ !
  • 27. did I say Continuous Delivery or deployment? meet the real rockstars at Envato …
  • 28. Envato the microlancer team automatically deploy every commit to master … always
  • 30. Envato what our next challenges? continuous automated reactive deployments ‣ flips + split testing and with every product feature ‣ flexibility of our infrastructure ‣ improved operations ‣ !