How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team. Reprised version of the talk given at the Continuous Delivery NYC meetup October 2013.

1. Scaling Deployment at Etsy
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Thursday, October 10, 13

2. Thursday, October 10, 13

3. August 2013
•1.8 billion page views
•5,483,399 items sold
•$109.1 million of goods sold
•> 30 million members
•> 1 million active shops
http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
Thursday, October 10, 13

4. LAMMP
Item by TheBackPackShoppe
Thursday, October 10, 13

5. Item by FrankelPhotos
Monolithic App
Thursday, October 10, 13

6. No Branching
Item by NurseryWallArt
Thursday, October 10, 13

7. Deploy Frequency
Thursday, October 10, 13

8. First Day
Item by flowersandfleurons
Thursday, October 10, 13

9. Thursday, October 10, 13

10. IRC
Thursday, October 10, 13

11. Thursday, October 10, 13

12. DeveloperVMs
•KVM
•Dev version of full Etsy stack
•Chef
•DevTools
Thursday, October 10, 13

13. Thursday, October 10, 13

14. Item by codecards
Thursday, October 10, 13

15. % review -r dschauenberg
Thursday, October 10, 13

16. automatically assignedautomatically assigned
Thursday, October 10, 13

17. Try
Item by CSSDesign
Thursday, October 10, 13

18. Actually ...
Thursday, October 10, 13

19. The Bobs
Item by Signz
Thursday, October 10, 13

20. The Bobs
•LXC containers on buildtests
•Multiple SSDs
•Labels for heavy/any execution
•One heavy executor per disk
Thursday, October 10, 13

21. CI/Try
•~260 Bobs
•Mostly for try
•Constant monitoring for slow tests
Thursday, October 10, 13

22. push train
Item by decomodwalls
Thursday, October 10, 13

23. #push
•IRC channel to organize push trains
•Join a train if you want to deploy changes
•Schedule is planned via the channel topic
•First in the train is the driver (controls the deploy)
•Opening hours: 7am - 10pm NYC time
Thursday, October 10, 13

24. #push
<prod> kseever* + jameslee | jpaul | DanielConvissor (c)
Thursday, October 10, 13

25. #push
bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
+ jlaster (c) | dawa + corey + sandosh + jklein + magera
+ seth_home + mpascual + nathan | bateman | russp (c)
Thursday, October 10, 13

26. pushbot
•.join
•.in
•.good
•.done
Thursday, October 10, 13

27. pushbot
Thursday, October 10, 13

28. Item by EsalonPhotography
Thursday, October 10, 13

29. Deployinator
Thursday, October 10, 13

30. Thursday, October 10, 13

31. Thursday, October 10, 13

32. stale commits
Thursday, October 10, 13

33. version checks
buttons disabledbuttons disabled
Thursday, October 10, 13

34. version checks
Thursday, October 10, 13

35. lock down deploys
Thursday, October 10, 13

36. https://github.com/etsy/deployinator
Thursday, October 10, 13

37. Downsides
•Deploys not atomic on the request level
•Limbo during the time of the local rsync
•Common strategy was to split commits into 3 deploys
Thursday, October 10, 13

38. Item by Geographicsart
Thursday, October 10, 13

39. Atomic Deploys
Thursday, October 10, 13

40. Basic Idea
Yin Yang
Active Docroot
Thursday, October 10, 13

41. Basic Idea
Yin Yang
Active Docroot
rsync
Thursday, October 10, 13

42. Basic Idea
Yin Yang
Active Docroot
Thursday, October 10, 13

43. Basic Idea
Yin Yang
Active Docroot
Thursday, October 10, 13

44. Problems
•Symlink swap during requests
•Code needs to be guaranteed to finish on the docroot
it started
•Code inclusion mid request
Thursday, October 10, 13

45. etsy/mod_realdoc
•Apache post_read_request hook
•Whole request works on realpath of docroot
•Caches realpath for 2s
Thursday, October 10, 13

46. ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');
Thursday, October 10, 13

47. etsy/incpath
•PHP module to set the incpath
•Gets docroot from Apache or realpath() itself
•Looks for a pattern to replace in include_path
•Restores include_path at the end of the request
Thursday, October 10, 13

48. What did we get?
•Remove functions and call site in same deploy
•No restarts necessary
•Opcode caches stay warm for files that don’t change
between 2 deploys
Thursday, October 10, 13

49. Things to watch out for
•Code that uses full path names to scripts
•Atomic symlink swapping with `mv -T`
•Realpath caching to not stress the filesystem
•Opcode cache needs to fit 2x code size
•Only request atomicity
Thursday, October 10, 13

50. The Plateau
Item by finandfancy
Thursday, October 10, 13

51. The Plateau
•Regular deploys took ~15 mins
•Config deploys about half
•10am - 6pm => ~ 32 deploys
•Long waiting times
Thursday, October 10, 13

52. Item by KlaireWarren
Thursday, October 10, 13

53. Waiting for push queueWaiting for push queue
Thursday, October 10, 13

54. Split The Queues
Item by KlaireWarren
Thursday, October 10, 13

55. HELLO SPLIT QUEUESHELLO SPLIT QUEUES
Thursday, October 10, 13

56. Dashboards
deploy linesdeploy lines
Thursday, October 10, 13

57. Supergrep
Thursday, October 10, 13

58. Thursday, October 10, 13

59. Summary
•Current setup has scaled to ~150 people
•Constantly trying to improve the speed of deployment
•Find weak parts in the process and make them more
robust/faster
•Bring Dev closer to Prod
•Not being able to deploy has the same status as the site
being down
Thursday, October 10, 13

60. codeascraft.etsy.com
www.etsy.com/codeascraft/talks
etsy.github.com
www.etsy.com/careers
Thursday, October 10, 13

61. Questions?
Thursday, October 10, 13

62. Scaling Deployment at Etsy
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Thursday, October 10, 13