What do deployment pipelines look like when your system consists of 10s of different types of services? How do you know what to test before deployment? Should you release a service at a time, or bunch them up? This talk goes into the nitty gritty of managing build,test and release of micro services and also covers the often ignored tradeoff between testing before deployment, and testing afterwards. Presented at GeeCon 2014

1. Testing & Deploying Microservices
Sam Newman

2. @samnewman#geecon
2

3. @samnewman#geecon
3
http://www.thoughtworks.com/radar/

4. @samnewman#geecon
4
Sam Newman
Building
Microservices
DESIGNING FINE-GRAINED SYSTEMS

5. @samnewman#geecon
5

6. @samnewman#geecon
5
Web Shop
Shopping
Cart
Catalog
Registration
Customer
Finance

7. @samnewman#geecon
5
Web Shop
Shopping
Cart
Catalog
Registration
Customer
Finance
1000 lines of code or less

8. @samnewman#geecon
6

9. @samnewman#geecon
6

10. @samnewman#geecon
7
V1

11. @samnewman#geecon
7
V2

12. @samnewman#geecon
8
Go
Java
NodeJS
Ruby

13. @samnewman#geecon
8
Go
NodeJS
Ruby
Clojure!

14. @samnewman#geecon
9

15. @samnewman#geecon
9

16. @samnewman#geecon
10
Accounts
Returns
Invoicing
Shipping
Inventory
Customer
Service

17. @samnewman#geecon
10
Accounts
Returns
Invoicing
Shipping
Inventory
Customer
Service

18. @samnewman#geecon
11

19. @samnewman#geecon
11

20. @samnewman#geecon
11
DB

21. @samnewman#geecon
12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID

22. @samnewman#geecon
12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
Increasing Scope
More Confidence

23. @samnewman#geecon
12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
Increasing Scope
More Confidence
Faster!
Better Isolation

24. @samnewman#geecon
12
Unit
Service
UI
MIKE COHN’S TEST PYRAMID
???
???
Increasing Scope
More Confidence
Faster!
Better Isolation
???

25. @samnewman#geecon
13
Small
Medium
Large
GOOGLE’S TEST PYRAMID
Increasing Scope
More Confidence
Faster!
Better Isolation

26. @samnewman#geecon
14
DB
Small
Large
Medium

27. @samnewman#geecon
14
DB
Small
Large
Medium

28. @samnewman#geecon
15
DB
Small
Large
Medium

29. @samnewman#geecon
15
DB
Small
Large
Medium

30. @samnewman#geecon
16
DB
Small
Large
Medium

31. @samnewman#geecon
16
DB
Small
Large
Medium

32. @samnewman#geecon
16
DB
Small
Large
Medium

33. @samnewman#geecon
1717
Small
Medium
Large
TEST SNOWCONE

34. @samnewman#geecon
18
Small
Medium
Large

35. @samnewman#geecon
19
Small
Medium
Large
Build
Tests
Tests
Source
Control
…
BUILD PIPELINE

36. @samnewman#geecon
19
Small
Medium
Large
Build
Tests
Tests
Source
Control
Faster Feedback
…
BUILD PIPELINE

37. @samnewman#geecon
19
Small
Medium
Large
Build
Tests
Tests
Source
Control
Faster Feedback
Binary Artifact(s)
…
BUILD PIPELINE

38. @samnewman#geecon
19
Small
Medium
Large
Build
Tests
Tests
Source
Control
Faster Feedback
Binary Artifact(s)
…
BUILD PIPELINE

39. @samnewman#geecon
20
DB

40. @samnewman#geecon
20
DB

41. @samnewman#geecon
21
S/M TestsBuild Large Tests UAT Prod

42. @samnewman#geecon
21
S/M TestsBuild Large Tests UAT Prod
DB
Machine CI Node
Large Tests Environment
Large Tests

43. @samnewman#geecon
22
S/M TestsBuild Large Tests UAT Prod
DB
Machine
UAT Environment
Machine

44. @samnewman#geecon
23
S/M TestsBuild Large Tests UAT Prod
Master
DB
Machine
Production Environment
Machine Machine Machine
Slave
DB

45. @samnewman#geecon
24
S/M TestsBuild Large Tests UAT Prod
More Production Like

46. @samnewman#geecon
24
S/M TestsBuild Large Tests UAT ProdLarge Tests
More Production Like

47. @samnewman#geecon
24
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like

48. @samnewman#geecon
25

49. @samnewman#geecon
26
Customer Service

50. @samnewman#geecon
26
Customer Service
S/M TestsBuild Large Tests

51. @samnewman#geecon
26
Customer Service
S/M TestsBuild Large Tests

52. @samnewman#geecon
26
Customer Service
S/M TestsBuild Large Tests

53. @samnewman#geecon
27
Customer
Service
Web Shop
Fulfilment
Service

54. @samnewman#geecon
27
Customer
Service
Web Shop
Fulfilment
Service
Large
Medium
Small

55. @samnewman#geecon
28
Customer
Service
Fulfilment
Service
Large
Medium
Small

56. @samnewman#geecon
28
Customer
Service
Fulfilment
Service
Large
Medium
Small

57. @samnewman#geecon
28
Customer
Service
Large
Medium
Small
Fulfilment
Service
Stub

58. @samnewman#geecon
29
Mountebank
http://www.mbtest.org

59. @samnewman#geecon
30
mountebank
:2525

60. @samnewman#geecon
30
mountebank
:2525

61. @samnewman#geecon
30
mountebank
:2525
:5555

62. @samnewman#geecon
30
mountebank
:2525
:5555
Customer
Service

63. @samnewman#geecon
31
S/M TestsBuild Large Tests

64. @samnewman#geecon
31
S/M TestsBuild Large Tests
Customer
Service
Fulfilment
Service
Stub

65. @samnewman#geecon
32
Customer
Service
V1
Web Shop
Fulfilment
Service

66. @samnewman#geecon
32
Web Shop
Customer
Service
v2
Fulfilment
Service

67. @samnewman#geecon
32
Web Shop
Customer
Service
v2
Fulfilment
Service

68. @samnewman#geecon
32
Web Shop
Customer
Service
v2
Fulfilment
Service

69. @samnewman#geecon
33
S/M TestsBuild Large Tests
Customer
Service
Customer
Service
v1
Web Shop
v1
Production

70. @samnewman#geecon
33
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v1
Web Shop
v1
Production

71. @samnewman#geecon
33
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v1
Web Shop
v1
Production
Customer
Service
v2
Web Shop
v1
Integration
Test

72. @samnewman#geecon
34
Customer
Service
v1
Web Shop
v1
Production

73. @samnewman#geecon
34
Customer
Service
v1
Web Shop
v1
Production
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v2

74. @samnewman#geecon
34
S/M TestsBuild Large TestsWeb Shop
Customer
Service
v1
Web Shop
v1
Production
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v2

75. @samnewman#geecon
34
S/M TestsBuild Large TestsWeb Shop
Customer
Service
v1
Web Shop
v1
Production
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v2
Web Shop
v2

76. @samnewman#geecon
34
S/M TestsBuild Large TestsWeb Shop
Customer
Service
v1
Web Shop
v1
Production
S/M TestsBuild Large Tests Integration Test
Customer
Service
Customer
Service
v2
Web Shop
v2
???

77. @samnewman#geecon
35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large Tests
Customer
Service

78. @samnewman#geecon
35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large Tests
Customer
Service
Integration Test

79. @samnewman#geecon
35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large Tests
Customer
Service
Integration Test
S/M TestsBuild Large Tests
Invoice
Service

80. @samnewman#geecon
35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large Tests
Customer
Service
Integration Test
S/M TestsBuild Large Tests
Invoice
Service
S/M TestsBuild Large TestsBasket

81. @samnewman#geecon
35
S/M TestsBuild Large TestsWeb Shop
S/M TestsBuild Large Tests
Customer
Service
Integration Test
S/M TestsBuild Large Tests
Invoice
Service
S/M TestsBuild Large TestsBasket
S/M TestsBuild Large TestsFulfilment

82. @samnewman#geecon
36

83. @samnewman#geecon
36
Browsers

84. @samnewman#geecon
36
Timing
Browsers

85. @samnewman#geecon
36
Provisioning of Environments
Timing
Browsers

86. @samnewman#geecon
36
Provisioning of Environments
Networks
Timing
Browsers

87. @samnewman#geecon
36
Deployment
Provisioning of Environments
Networks
Timing
Browsers

88. @samnewman#geecon
36
Deployment
Provisioning of Environments
Networks
Timing
Browsers Diagnosis

89. @samnewman#geecon
37

90. @samnewman#geecon
37
Integration Test

91. @samnewman#geecon
37
Integration Test Prod…

92. @samnewman#geecon
37
Integration Test Prod…

93. @samnewman#geecon
37
Integration Test Prod…

94. @samnewman#geecon
37
Integration Test Prod…

95. @samnewman#geecon
38

96. @samnewman#geecon
38

97. @samnewman#geecon
39
John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

98. @samnewman#geecon
39
John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

99. @samnewman#geecon
40
Integration Test Prod…
v1v2
v6
v4 = v10

100. @samnewman#geecon
40
Integration Test Prod…
v1v2
v6
v4 = v10

101. @samnewman#geecon
41
Danger Will Robinson!

102. @samnewman#geecon
42
Show Tangle

103. @samnewman#geecon
43
Golden Rule:
Get good at releasing services independently

104. @samnewman#geecon
44
SO NO INTEGRATION TESTS?

105. @samnewman#geecon
45

106. @samnewman#geecon
45

107. @samnewman#geecon
45
SEMANTIC MONITORING

108. @samnewman#geecon
46
Customer
Service
Web Shop

109. @samnewman#geecon
46
Customer
Service
Web Shop
Small
Medium
Large

110. @samnewman#geecon
46
Customer
Service
Web Shop
Small
Medium
Large

111. @samnewman#geecon
46
Customer
Service
Web Shop
Small
Medium
Large
Consumer Driven Contracts

112. @samnewman#geecon
47
Customer
Service
Web Shop

113. @samnewman#geecon
47
Customer
Service
Web Shop
Expectations

114. @samnewman#geecon
47
Customer
Service
Web Shop
Expectations

115. @samnewman#geecon
47
Customer
Service
Web Shop
Expectations
Prod

116. @samnewman#geecon
47
Customer
Service
Web Shop
Expectations
Prod

117. @samnewman#geecon
48

118. @samnewman#geecon
48
https://github.com/realestate-com-au/pact

119. @samnewman#geecon
49
Prod
Prod
Prod
Prod

120. @samnewman#geecon
49
Prod
Prod
Prod
Prod

121. @samnewman#geecon
49
Prod
Prod
Prod
Prod
QA

122. @samnewman#geecon
49
Prod
Prod
Prod
Prod
QA
Good Monitoring

123. @samnewman#geecon
49
Prod
Prod
Prod
Prod
QA
Good Monitoring
Fast Remediation

124. @samnewman#geecon
49
Prod
Prod
Prod
Prod
QA
Good Monitoring
Fast Remediation

125. @samnewman#geecon
50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like

126. @samnewman#geecon
50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like

127. @samnewman#geecon
50
S/M TestsBuild Large Tests UAT ProdLarge Tests
Faster Feedback
More Production Like

128. @samnewman#geecon
51
DB
Machine CI Node
Large Tests Environment
S/M TestsBuild Large Tests UAT ProdLarge TestsLarge Tests

129. @samnewman#geecon
51
DB
Machine CI Node
Large Tests Environment
DB
Machine
UAT Environment
Machine
S/M TestsBuild Large Tests UAT ProdLarge Tests UAT

130. @samnewman#geecon
51
DB
Machine CI Node
Large Tests Environment
DB
Machine
UAT Environment
Machine
Master
DB
Machine
Production Environment
Machine Machine Machine
Slave
DB
S/M TestsBuild Large Tests UAT ProdLarge Tests Prod

131. @samnewman#geecon
52
Faster Feedback
More Production Like
S/M TestsBuild Large Tests UAT ProdLarge Tests

132. @samnewman#geecon
52
Faster Feedback
More Production Like
S/M TestsBuild Large Tests UAT ProdLarge Tests

133. @samnewman#geecon
53

134. @samnewman#geecon
54

135. @samnewman#geecon
55
Ansible
Puppet
Chef

136. @samnewman#geecon
55
Ansible
Puppet
Chef

137. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS

138. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean

139. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack

140. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare

141. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant

142. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers

143. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up

144. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic

145. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic
Feedback Can Suffer

146. @samnewman#geecon
55
Ansible
Puppet
Chef
AWS
Digital Ocean
OpenStack
VMWare
Vagrant
Immutable Servers
Fast Spin-up
Provider Agnostic
Feedback Can Suffer
Cycle Time

147. @samnewman#geecon
56
Prod
Prod
Prod
Prod

148. @samnewman#geecon
56
Prod
Prod
Prod
Prod
Packer Images

149. @samnewman#geecon
56
Prod
Prod
Prod
Prod
“Machine”
Service
Packer Images

150. @samnewman#geecon
56
Prod
Prod
Prod
Prod
“Machine”
Service
“Machine”
Service Service
Packer Images

151. @samnewman#geecon
56
Prod
Prod
Prod
Prod
“Machine”
Service
“Machine”
Service Service
Packer Images

152. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests

153. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS

154. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare

155. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant

156. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant

157. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant

158. @samnewman#geecon
57
S/M TestsBuild Large Tests UAT ProdLarge Tests
AWS
VMWare
Vagrant

159. @samnewman#geecon
58
“Machine”
Service

160. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About

161. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)

162. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects

163. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!

164. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!
AWS
Digital Ocean
OpenStack

165. @samnewman#geecon
58
“Machine”
Service
Much Easier To Reason About
Easier To Provision (Or Decommission)
Fewer Side-effects
Cost & Management Overhead!
AWS
Digital Ocean
OpenStack

166. @samnewman#geecon
59
STANDARD VIRTUALISATION

167. @samnewman#geecon
59
Machine
STANDARD VIRTUALISATION

168. @samnewman#geecon
59
Machine
Base OS
STANDARD VIRTUALISATION

169. @samnewman#geecon
59
Machine
Base OS
Hypervisor
STANDARD VIRTUALISATION

170. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
STANDARD VIRTUALISATION

171. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
OS
STANDARD VIRTUALISATION

172. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
OS
Apps
STANDARD VIRTUALISATION

173. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
OS
Apps
Packer Image
STANDARD VIRTUALISATION

174. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
OS
Apps
VM
OS
Apps
Packer Image
STANDARD VIRTUALISATION

175. @samnewman#geecon
59
Machine
Base OS
Hypervisor
VM
OS
Apps
VM
OS
Apps
VM
OS
Apps
Packer Image
STANDARD VIRTUALISATION

176. @samnewman#geecon
60

177. @samnewman#geecon
61
CONTAINER VIRTUALISATION

178. @samnewman#geecon
61
Machine
CONTAINER VIRTUALISATION

179. @samnewman#geecon
61
Machine
Base OS
CONTAINER VIRTUALISATION

180. @samnewman#geecon
61
Machine
Base OS
Container
CONTAINER VIRTUALISATION

181. @samnewman#geecon
61
Machine
Base OS
Container
OS
CONTAINER VIRTUALISATION

182. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
CONTAINER VIRTUALISATION

183. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION

184. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION

185. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Linux Only

186. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux Only

187. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux OnlyFine-grained control

188. @samnewman#geecon
61
Machine
Base OS
Container
OS
Apps
Container
OS
Apps
Container
OS
Apps
CONTAINER VIRTUALISATION
Same Kernel
Linux OnlyFine-grained control
Very fast to provision

189. @samnewman#geecon
62

190. @samnewman#geecon
63
DOCKER

191. @samnewman#geecon
63
DOCKER
Machine

192. @samnewman#geecon
63
DOCKER
Machine
Base OS

193. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker

194. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker
Apps

195. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker
Apps Apps

196. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps

197. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps
Docker Image Registry

198. @samnewman#geecon
63
DOCKER
Machine
Base OS
Docker
Apps Apps Apps
Docker Image Registry

199. @samnewman#geecon
64

200. @samnewman#geecon
64

201. @samnewman#geecon
65
S/M TestsBuild Large Tests UAT ProdLarge Tests

202. @samnewman#geecon
65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image

203. @samnewman#geecon
65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image

204. @samnewman#geecon
65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image

205. @samnewman#geecon
65
S/M TestsBuild Large Tests UAT ProdLarge Tests
Docker Image Registry
Docker Image

206. @samnewman#geecon
66

207. @samnewman#geecon
66
Be aware of - and balance - your test Pyramid

208. @samnewman#geecon
66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation

209. @samnewman#geecon
66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time

210. @samnewman#geecon
66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time
Consider consumer-driven contracts over integration tests

211. @samnewman#geecon
66
Be aware of - and balance - your test Pyramid
Understand the balance between testing & rapid remediation
Deploy one thing at a time
Consider consumer-driven contracts over integration tests
Explore image-based deployments to reduce environment differences

212. Any questions:
@samnewman
snewman@thoughtworks.com
THANKS!