Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
•
12 likes•2,025 views
A look at how we try to make our architecture robust, resilient and fun to work with: Namshi is not github or spotify but... ...imitation is the sincerest form of flattery!
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Similar to Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin (20)
More from Alessandro Nadalin
More from Alessandro Nadalin (20)
Recently uploaded
Recently uploaded (20)
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
- 1. Deploying 3 times a day without any downtimeAlessandro Nadalin - VP Tech @ Namshi.com Rocket Tech Summit 2015 - September 2015 - Berlin
- 2. When was the last time that *.namshi.com was down?
- 3. I don’t know
- 7. SOA
- 11. SOA ~40 services docker heavy on JS devoted to experimenting
- 12. SOA ~40 services docker heavy on JS devoted to experimenting Result?
- 14. A bit of history...
- 15. “When are you guys gonna deploy it?”
- 19. Now.
- 20. This is how it works @ Namshi
- 21. We don’t like centralization
- 22. Single Point of Failure
- 23. On error, rollback all the things!
- 25. Ship it
- 26. Don’t wait tomorrow for what you can get today
- 27. Result?
- 28. Microservices
- 30. ~40 services * ⅔ people = ~100 engineers
- 32. ~15 people rotate, assume different roles
- 35. “adding manpower to a late project makes it later” https://goo.gl/Bc0uBf
- 36. Communication overhead takes time to get used to the whole thing
- 37. keep things small, flat, plain and simple
- 38. means small teams, simple software
- 39. What is the simplest way to get devs their development environment?
- 40. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
- 41. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js Docker
- 42. How long does it take to get started with any of the microservices?
- 43. git clone git@github.com:namshi/stuff.git cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
- 44. git clone git@github.com:namshi/stuff.git (20s) cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
- 45. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
- 46. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up
- 47. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up (depends on the image)
- 48. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up (~5s with alpine)
- 49. ~ (master ✔) docker images | grep watchdog watchdog_web [...] 38.74 MB
- 50. Make it so that it takes <2m to start hacking on your services
- 51. Make it simple.
- 52. How do we get this stuff to production?
- 53. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
- 54. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js It’s that simple.
- 55. Ok, but how do I get to deploy the container?
- 56. There’s a bot for everything
- 57. @c3po deploy app1 patch-1 to staging
- 58. All it takes is a 100 locs or so
- 59. Is it really without a downtime?
- 60. We’re actually down every time we deploy
- 61. it’s just that no one notices
- 62. keep a secondary setup up&running
- 63. deploy there (with downtime), switch the LB
- 65. Everything seems simple, but how do you glue all of this?
- 70. We build following few principles:
- 71. Resilient
- 72. "Resilience is accepting your [...] reality" https://en.wikipedia.org/wiki/Elizabeth_Edwards
- 73. Robust
- 74. "Be conservative in what you send, be liberal in what you accept" https://en.wikipedia.org/wiki/Robustness_principle
- 75. Simple
- 76. "Innovation is saying no to 1,000 things" http://www.goodreads.com/quotes/629613-people-think-focus-means-saying-yes-to-the- thing-you-ve
- 77. Safe
- 78. "Each calling function must check non-void function return values, and the validity of parameters must be checked inside each function." http://sdtimes.com/nasas-10-rules-developing-safety-critical-code/
- 79. Pragmatic
- 80. "Each calling function must check non-void function return values, and the validity of parameters must be checked inside each function, unless you are 1000% freaking sure" http://sdtimes.com/nasas-10-rules-developing-safety-critical-code/
- 81. How does the typical workflow look like?
- 83. spend time spec-ing things out before accepting them in the backlog
- 84. “if it’s not simple enough, it’s wrong”
- 86. +1 from a lead, less formal review from peers
- 87. “more deletions, less additions”
- 89. “we process 5 builds in parallel, don’t piss others off”
- 91. “don’t test everything, just what you fear”
- 92. Then, Go Live.
- 93. Takeaways?
- 95. MVA
- 96. MVA novice engineers are the ones who can’t abstract
- 97. MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract
- 98. MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract great engineers are the ones who know when to abstract
- 100. avalanche of go-live
- 101. failure is always an option
- 104. Challenges?
- 106. parallel deployments based on cookie
- 107. everything is new
- 108. use the internal infra as a testing platform
- 109. manual QA
- 110. simplify the dangerous stuff, think 3ice before touching
- 111. think mobile
- 112. Desktop is the new Blackberry
- 113. Whether you are running everything through an isomorphic JavaScript framework
- 114. every day https://eng.asana.com/2014/02/scaling-asana-com/ & https://asana.com/luna
- 115. or fatass ruby frontends
- 116. 30 times a day https://github.com/blog/1241-deploying-at-github
- 117. or you’re just that crazy
- 120. Roll it out live.