To go faster in a car, you need not only a powerful engine, but also safety mechanisms like brakes, air bags, and seat belts. This is a talk about the safety mechanisms that allow you to build software faster. It's based on the book "Hello, Startup" (http://www.hello-startup.net/). You can find the video of the talk here: https://www.youtube.com/watch?v=4fKm6ImKml8

1. AGILITY
requires
SAFETY

2. Every startup has the
same story:

3. “We don’t have time for
best practices.”

10. You can’t go faster by being
reckless

11. Think of cars on a highway

12. What happens if everyone jams
down on the gas?

14. To go fast, a car needs not only a
powerful engine…

15. But also powerful brakes.

16. As well as seat belts, airbags,
bumpers, and auto-pilot

17. For cars and for software, speed
is limited by safety

18. What are the seat belts, brakes, &
self-driving cars of software?

19. This talk is about
safety mechanisms

20. That make it possible to
build software quickly

21. I’m
Yevgeniy
Brikman
ybrikman.com

22. Founder
of
Atomic
Squirrel
atomic-squirrel.net

23. PAST LIVES

24. Author of
Hello,
Startup
hello-startup.net

25. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

26. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

27. Good brakes stop your car before
you run into something

28. Continuous integration stops buggy
code before it goes into production

29. Imagine your goal is to build the
International Space Station

30. Each team designs and builds
their component in isolation

31. You launch everything into space
and hope it all comes together

33. I thought the Russians were going
to build the bathrooms?

34. Weren’t the French supposed to
do the wiring?

35. Everyone is using the metric
system, right?

36. Teams working for a long time
with incorrect assumptions

37. Finding this out when you’re in
outer space is too late

38. This is the result of
“late integration”

39. Lots of teams working in isolation
on separate branches

40. Before attempting a massive
merge at the very end

41. MERGE
CONFLICT

42. The alternative is
“continuous integration”

43. Where everyone regularly merges
their work

44. The most common approach is
trunk-based development

45. Everyone works on a
single branch (trunk)

46. That can’t possibly scale to a lot
of developers, can it?

47. Uses trunk-based development for
1,000+ developers

48. Uses trunk-based development for
4,000+ developers

49. Uses trunk-based development for
20,000+ developers

51. Wouldn’t you have merge conflicts
all the time?

52. If you merge (commit) regularly,
conflicts are rare.

53. And those that happen are from a
day of work—not months.

54. Commit early and often.

55. Small commits are easier to
merge, test, revert, review

57. Wouldn’t there constantly be
broken code in trunk?

58. Build Build Build Build
Not if you run a self-testing build
after every commit
Build Build Build Build Build Build Build

59. Build Build Build Build
It should compile your code and
run your automated tests
Build Build Build Build Build Build Build

60. Build Build Build Build
If a build fails, a developer must
fix it ASAP or revert the commit
Build Build Build Build Build Build Build

61. Of course, this depends on
having good automated tests

62. Tests give you the confidence to
make changes quickly

63. JUnit version 4.11
...
Time: 6.063
OK (259 tests)
How long would it take you to do
259 tests manually?

64. What should you test?

65. Everything!

66. Everything!

67. It’s a trade-off between:
1. Likelihood of bugs
2. Cost of bugs
3. Cost of testing

68. Likelihood of bugs is higher for
complex code and large teams

69. Cost of bugs is higher for some
systems (payments, security)

70. Cost of tests is higher for
integration and UI tests

71. “Without continuous
integration, your software is
broken until somebody
proves it works, usually
during a testing or
integration stage.

72. With continuous integration,
your software is proven to
work (assuming a sufficiently
comprehensive set of
automated tests) with every
new change—and you know
the moment it breaks and can
fix it immediately.”

73. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

74. Ships have bulkheads to try to
contain flooding to one area.

75. You can split up a codebase to
contain problems to one area.

76. Code is the enemy: the more you
have, the slower you go

77. Project Size
Lines of code
Bug Density
Bugs per thousand lines
of code
< 2K 0 – 25
2K – 6K 0 – 40
16K – 64K 0.5 – 50
64K – 512K 2 – 70
> 512K 4 – 100

78. As the code grows, the number of
bugs grows even faster

79. “Software
development doesn't
happen in a chart, an
IDE, or a design tool;
it happens in your
head.”

80. The mind can only handle so
much complexity at once

81. One solution is to break the code
into multiple codebases

82. Instead of depending on the
source of another module
/moduleA
/moduleB /moduleC /moduleD
/moduleE

83. You depend on a versioned
artifact from that module
moduleA-0.3.1.jar
moduleB-3.1.0.jar moduleC-9.8.0.jar moduleD-1.4.3.jar
moduleE-0.5.6.jar

84. This provides isolation from
changes in other modules
moduleA-0.3.1.jar
moduleB-3.1.0.jar moduleC-9.8.0.jar moduleD-1.4.3.jar
moduleE-0.5.6.jar

85. You already do this: guava-
18.0.jar
jquery-2.2.0.js

86. Advantages of artifacts:
1. Isolation
2. Decoupling
3. Faster builds

87. Disadvantages of artifacts:
1. Dependency hell
2. No continuous integration
3. Hard to make global changes

88. Another option is to break the
codebase into services

89. In a monolith, you use function
calls within one process
A.a()
B.b() C.c() D.d()
E.e()

90. With services, you pass messages
between processes
http://A/a
http://B/b
http://C/c
http://D/d
http://E/e

91. Advantages of services:
1. Technology agnostic
2. Scalability
3. Isolation

92. Disadvantages of services:
1. Operational overhead
2. Performance overhead
3. I/O, error handling
4. Backwards compatibility
5. Hard to make global changes

93. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

94. Autopilot prevents accidents
caused by human error

95. Automated deployments prevent
accidents caused by human error

96. Deploying code can be painful

97. “If it hurts, do it
more often.”
– Martin Fowler

98. The deployment process should
be:

100. That means you should never
deploy or configure manually

101. > ssh ec2-user@12.34.56.78
__| __| __|
_| ( __ Amazon ECS-Optimized Amazon Linux AMI 2015.09.d
____|___|____/
[ec2-user ~]$ sudo apt-get install ruby
Don’t do this

102. Or this

103. Instead, automate everything

104. The gold standard is the
blue-green deployment

105. Let’s say you have version 0.0.1 of
your app deployed

106. First, deploy version 0.0.2 on a
duplicate set of servers

107. If everything looks good, switch
the load balancer over to 0.0.2

108. Four main categories of
deployment automation tools:

109. 1. Configuration management:
Chef, Puppet, Ansible, Salt

110. - name: Install httpd and php
yum: name={{ item }} state=present
with_items:
- httpd
- php
- name: start httpd
service: name=httpd state=started enabled=yes
- name: Copy the code from repository
git: repo={{ repository }} dest=/var/www/html/
Imperative scripts to configure
servers and deploy code

111. 2. Provisioning tools: Terraform,
CloudFormation, Heat

112. resource "aws_instance" "example" {
ami = "ami-b960b1d"
instance_type = ["t2.micro"]
}
resource "aws_eip" "ip“ {
instance = "${aws_instance.example.id}"
depends_on = ["aws_instance.example"]
}
Declarative templates that define
your infrastructure

113. 3. Virtual machines: VMWare,
VirtualBox, Packer, Vagrant

114. {
"builders": [{
"type": "amazon-ebs",
"source_ami": "ami-de0d9eb7",
"instance_type": "m1.medium",
"ami_name": "example-packer-ami-{{timestamp}}"
}],
"provisioners": [{
"type": "shell",
"inline": [
"sudo apt-get -y update",
"sudo apt-get -y install httpd php”
]
}]
}
Images of configured servers

115. 4. Containers: Docker, rkt, LXD

116. FROM ubuntu:12.04
RUN apt-get update && apt-get install -y apache2 php
ENV APACHE_RUN_USER www-data
ENV APACHE_LOG_DIR /var/log/apache2
EXPOSE 80
CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]
Lightweight images of configured
servers

117. These tools allow you to define
your infrastructure as code

118. That way, you can version it,
review it, test it, and reuse it.

119. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

120. Elisha Otis
demoing
elevator
free-fall
safety in 1854

121. The safety
elevator
patent

122. The safety
catches are
locked by
default

123. Only an
intact cable
can unlock
the
latches

124. This
elevator
provides
safety by
default

125. Feature
toggles
provide
safety by
default

126. New
feature,
part 1
New
feature,
part 2
New
feature,
part 3
If a large new feature takes many
commits, wouldn’t a user see it in
an unfinished state?

127. <section id="new-section">
<!-- Code for new section-->
</div>
<section id="original-section">
<!-- Code for original section-->
</section>
Let’s say you were adding a new
section to your website.

128. <% if toggles.enabled("new-section") %>
<section id="new-section">
<!-- Code for new section-->
</div>
<% end %>
<section id="original-section">
<!-- Code for original section-->
</section>
Wrap new code in a conditional
that looks up a feature toggle

129. <% if toggles.enabled("new-section") %>
<section id="new-section">
<!-- Code for new section-->
</div>
<% end %>
<section id="original-section">
<!-- Code for original section-->
</section>
Toggles are off by default, so
users won’t see unfinished work

130. development:
feature_toggles:
new-section: true
production:
feature_toggles:
new-section: false
You can enable feature toggles in
a config file.

131. > curl http://feature.toggles/
{
"development": { "new-section": true },
"production": { "new-section": false }
}
Or you could create a web service
for feature toggles.

132. > curl http://feature.toggles/?user=123
{
"development": { "new-section": "A" },
"production": { "new-section": "B" }
}
It could return different, complex
values for each user.

133. And provide a web UI for
configuring toggles.

134. This allows you to quickly turn
features on or off.

135. <% if toggles.get("new-section") == "A" %>
<section id="new-section-bucket-a">
<!-- Code for new section, version A -->
</div>
<% elsif toggles.get("new-section") == "B" %>
<section id="new-section-bucket-b">
<!-- Code for new section, version B -->
</div>
<% end %>
This allows A/B testing

136. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

137. A speedometer tells you how fast
you’re driving

138. Monitoring tells you how your
product is performing

139. “If you can’t
measure it, you
can’t fix it.”
– David Henke

140. There are many types of
monitoring

141. Availability metrics: is my product
up or down?

142. Useful tools: Keynote, Pingdom,
Uptime Robot, Route53

143. Business metrics: what are my
users doing in the product?

144. Useful tools: Google Analytics,
KISSMetrics, Mixpanel

145. Application metrics: how is my
application performing?

146. Useful tools: New Relic,
CloudWatch, Datadog

147. 127.0.0.1 - - [10/Oct/2000:13:55:36] "GET /apache_pb.gif HTTP/1.0" 200 2326
64.242.88.10 - - [07/Mar/2004:16:05:49] "GET /twiki/bin/ HTTP/1.1" 401 12846
127.0.0.1 - - [28/Jul/2006:10:22:04] "GET / HTTP/1.0" 200 2216
64.242.88.10 - - [07/Mar/2004:16:06:51] "GET /twiki/bin/Twiki/" 200 4523
64.242.88.10 - - [07/Mar/2004:16:10:02] "GET /mailman HTTP/1.1" 200 6291
127.0.0.1 - - [28/Jul/2006:10:27:32] "GET /hidden/ HTTP/1.0" 404 7218
192.168.2.20 - - [28/Jul/2006:10:27:10] "GET /cgi-bin/try HTTP/1.0" 200 3395
64.242.88.10 - - [07/Mar/2004:16:11:58] "GET /twiki/bin/view/" 200 7352
64.242.88.10 - - [07/Mar/2004:16:20:55] "GET /twiki HTTP/1.1" 200 5253
Log files are also a form of
application-level monitoring

148. 127.0.0.1 - - [10/Oct/2000:13:55:36] "GET /apache_pb.gif HTTP/1.0" 200 2326
64.242.88.10 - - [07/Mar/2004:16:05:49] "GET /twiki/bin/ HTTP/1.1" 401 12846
127.0.0.1 - - [28/Jul/2006:10:22:04] "GET / HTTP/1.0" 200 2216
64.242.88.10 - - [07/Mar/2004:16:06:51] "GET /twiki/bin/Twiki/" 200 4523
64.242.88.10 - - [07/Mar/2004:16:10:02] "GET /mailman HTTP/1.1" 200 6291
127.0.0.1 - - [28/Jul/2006:10:27:32] "GET /hidden/ HTTP/1.0" 404 7218
192.168.2.20 - - [28/Jul/2006:10:27:10] "GET /cgi-bin/try HTTP/1.0" 200 3395
64.242.88.10 - - [07/Mar/2004:16:11:58] "GET /twiki/bin/view/" 200 7352
64.242.88.10 - - [07/Mar/2004:16:20:55] "GET /twiki HTTP/1.1" 200 5253
Useful tools: loggly, logstash,
Papertrail, Sumo Logic

149. Server metrics: how is my server
performing?

150. Useful tools: Nagios, Icinga,
Munin, collectd, CloudWatch

151. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

152. Warning lights notify you if
something is wrong

153. Alerting systems notify you if
something is wrong

154. You can’t look at metrics 24/7.
Alerting systems can.

155. Useful tools: PagerDuty,
VictorOps

156. For a full list of monitoring and
alerting tools, see:
hello-startup.net/resources

157. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

158. Seat belts help you survive
crashes

159. High availability helps you survive
crashes

160. Stateless servers: multiple
instances, multiple zones

161. Load balancer routes around
server or zone outages

162. Auto-recovery mechanism brings
server back after outage

163. Stateful servers: multiple
instances, multiple zones

164. Replication to one or more
standby servers

165. Load balancer switches to
standby server in case of outage

166. Auto-recovery mechanism brings
server back after outage

167. Test your recovery process
regularly.

168. 1. Brakes
2. Bulkheads
3. Autopilot
4. Safety catch
5. Speedometer
6. Warning lights
7. Seat belt
Outline

169. Speed is limited by safety

170. Two cars can drive at 80mph in
opposite directions safely…

171. Because of two yellow lines

172. It’s worth the time to put these
safety mechanisms in place

174. For more
info, see
Hello,
Startup
hello-startup.net

175. Questions?

176. F1 racecar: Takayuki Suzuki
Highway traffic: Oran Viriyincy
Car accident: ER24 EMS (Pty) Ltd.
Road: Nicolas Raymond
BWM: Andy Durst
Self-driving car: Steve Jurvetson
Bus: Roland Tanglao
Tail lights: Tony Webster
USS South Dakota: Wikimedia
Crash test dummy: Wikimedia
Elisha Otis: Wikimedia
Otis Elevator: Wikimedia
Speedometer: Dawn Hopkins
Dashboard lights: Jim Larrison
Seat belt: Wikimedia
Google repo stats: Rachel Potvin
ISS: Wikimedia
Fire: Pete
Martin Fowler: Wikimedia
Image credits