A presentation to the BankWest Solution Delivery team and the Perth DevOps Meetup describing the delivery processes at Envato that enable us to deploy 10 times a day.
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
10 Deploys a Day - A Case Study of Continuous Delivery at Envato
1. 10 Deploys a Day
a case study of
continuous
delivery at
envato
AT
john viner
PRESENTATION TEMPLATE from Envato’s Graphic River - http:/
/graphicriver.net/item/karbon-keynote-presentation-template/2580765
2. envato ABOUT US
7 years ago - a designer scratching an itch
creates a flash-plugin marketplace
and now …
‣
‣
‣
3.2m members
4.6m Items
10 Authors sold > $1m
‣
‣
‣
130m PVs p/m
20M Visits p/M
Item sold every 10 sec
4. Envato
wordpress - our biggest market
> 3400+ Themes for Sale
> Most Popular Theme has sold 37,500 times and generated $2m in
gross revenue
> Average price of a Theme is $42
> Average sales per TF author is 2,075
> Provides a Passive Income source
!
19% of the web runs on Wordpress! (4B monthly page view)
5. Envato
we’ve grown - and continue to grow
> Themeforest is now Ranked ~ #180 on the Alexa Rankings
> My development team was 8, 15 months ago and is now 24
6. Envato
the marketplace team
40 - in melbourne
‣ 30 - remote around the world
‣
‣
delivery team
- 19 Back End Developers
- 2 Front End Developers
- 1 Operations Developer
- 5 Product Managers and UX Designers
6 Teams
‣ finance,
‣ back office,
‣ front end,
‣ maintenance
‣ search
‣ 10x
‣
7. Envato
marketplace tech stack
ruby on rails web app
‣ mysql database
‣ elastic Search
‣ rackspace managed hosting in USA
‣
> 90M App Server Requests per Week
> Backend Response time of 148 ms
> Front end Response time of 2.5sec
> 45 Virtual Machines
> 4 Physical Machines (DBs, NFS)
> Avg 6 deploys a day - Peak of 10
> Average 100 commits a day
> Peak Load of 15,000 req per minute
> 4 Physical Machines (DBs etc…)
9. Envato
pull work, talk to Product, write code, test code
deploy code, verify production
repeat … about twice a week per developer
> Story on the wall
> flip or no flip?
> Create a Branch
> write failing test
> write code
> Run Local Tests
> Run full personal build
> create Pull request
> advertise pull request (IRC)
!
optional
> deploy to staging
> show product owner
> manual test
> wait for +1s (code review)
> Merge Pull request
> Full Master Build Green
> notify team in IRC
> deploy master to Production
> watch System - monitoring tools
> rollback if failed
> Rinse and Repeat
13. coordinating reviews
and deploys .. by irc
You Need Deep Dev and
Operational Experience right?
> Luke - Graduate with 4 months Exp
> Rakesh - Swinburne Industry Student
> Ben - Front End Developer
> Emmanuel - 2 yrs Dev Exp
4 Other Developers Mentioned
Across 4 Separate Teams
17. Envato
the post incident review recommends?
>1 - Improve our ability to Automatically rollback on monitored 500 errors
!
> 2 - NO Recommendation to Improve Preventative Testing
!
> Why? - Too Costly and complex to simulate production
> Why? - Would slow development flow
19. Envato
1 developer all developers
you build it
you run it
> 16 Developers on a Weekly Rotation
> They come out of their team
> They are Trained, have a Buddy for First Week
> Supported by a Secondary
> They are First Line of Support for 168 hours straight
Tooling is:
> Scout , New Relic and Pingdom monitoring and generating alerts
> Pagerduty forwarding to them On-Call Developer
> Rollbar to track Errors
> IRC to find out what went wrong
> ALL Alerts are also pumped into IRC so everyone knows whats going on and to get Help!
21. Envato
but we aren’t great at operations
> We are just Good Enough right now, Just
> The rotation leaves without a continuous view of system health
> We don’t always clean up behind us, root causes often come back
> Monitoring, Metrics, Performance insights are all Operational Skills
that many developers don’t have
> We are hiring Operational Specialists for the first time
> Our Single DevOps guy is actively training all the devs - AJs Cafe
22. Envato
so what do we not do
> No - testing role or formal testing gate
> NO - change control gate
> No - formal comms of every change
> NO - release management
> No - separate deployment team
> NO - Separate operations team
all of these are not feasible
at 10 deploys a day
23. Envato
how do we solve the problems that these practices solve ?
‣
No Testing Capability
> TDD and 1000s of automated tests
> Use Production Users as your test team
> Flips help us control the rollout
> We test major infrastructure and architecture
changes in production as well … using Flips
> We Fail Fast and Loud
‣
No release management
> Releases are SMALL, no need to define
or document
> Release self-document and radiate
through IRC
‣
No change control
> Operations (i.e. Developers) know of changes
before they happen because they review them as PRs
> There are changes we cannot “signoff” as testing in
pre-prod environments
> A green build is our sign off
> Devs have skin-in-game so are incentivised for the
code to work!
> Small Batch efficiency (small deltas are easier to
identify and rollback)
24. Envato
OK How did we do it?
‣
it was our startup dNA
> The Team started and stayed very small (less
than 8 devs) for 4 years
> They started with this structure and process
and have never changed.
‣
it scales
> We’ve tripled the team, and split into 5 streams
> We’ve been able to scale this model
‣
but with a lag
> But it took 3 months to see the results
25. Envato
so do you want to drink the kooL-aid?
‣
it might not be for you
‣
> We have a tolerant savvy users
> We have users that come back to
complete a purchase if it fails
> We are in CONTROL of our full stack
‣
If you want it, you have to change
> people
‣ > processes
‣ > assumptions
‣‣
> Tech is the easy part
> You can’t optimise your “gatekeeping” processes to 10 deploys
a leap is required
> Existing processes can get you close, say once a fortnight or a week
> A leap is required to get to many times a day
30. Envato
what our next challenges?
continuous automated reactive deployments
‣ flips + split testing and with every product feature
‣ flexibility of our infrastructure
‣ improved operations
‣
!