HighLoad++ 2013

1. Towards FutureOps:
Stable, repeatable,
environments from dev to prod.

2. I’m Mitchell Hashimoto
Also known as @mitchellh

3. I build tools
http://hashicorp.com

4. Vagrant
http://www.vagrantup.com
Packer
http://www.packer.io
SERF
http://www.serfdom.io

5. History of Ops
Or, the history of pretending that anything works.

6. Single
Dedicated
Server

7. Single Dedicated Server
•
Manually setup

8. Single Dedicated Server
•
•
Manually setup
Expensive to replace
(time and money)

9. Single Dedicated Server
•
•
•
Manually setup
Expensive to replace
(time and money)
Professionals, usually

10. What is hard/slow/expensive?
•
•
•
•
Acquiring hardware
Manual setup
Expertise
Downtime, Failures

11. Multiple
Dedicated
Servers

12. Multiple Dedicated Servers
•
Golden images

13. Multiple Dedicated Servers
•
•
Golden images
Manually made

14. Multiple Dedicated Servers
•
•
•
Golden images
Manually made
Slow to change,
expensive to grow

15. Multiple Dedicated Servers
•
Eventually: config
management (CFEngine,
Puppet, Chef)

16. What is hard/slow/expensive?
•
•
•
•
Acquiring hardware
Making images
Expertise
Downtime, Failures

17. The
Low-Cost
VPS Server

18. The VPS Server
•
Manually made (cargo
cult, usually)

19. The VPS Server
•
•
Manually made (cargo
cult, usually)
Cheaper to replace

20. The VPS Server
•
•
•
Manually made (cargo
cult, usually)
Cheaper to replace
Hobbyist, startup

21. What is hard/slow/expensive?
•
•
•
Expertise
Downtime, Failures
Rebuilding it

22. The
Cloud

23. The Cloud
•
Many cheap servers

24. The Cloud
•
•
Many cheap servers
Seconds to create

25. The Cloud
•
•
•
Many cheap servers
Seconds to create
Configuration
management

26. What is hard/slow/expensive?
•
•
•
•
Configuration mgmt
Portability
Repeatability
Can be expensive

27. The Ops of Tomorrow
Better, faster, stronger

28. Immutable
Infrastructure

29. Immutable Infrastructure
Instead of modifying or maintaining
a server, replace it with a new one.

30. Immutable Infrastructure
Servers don’t change anymore,
they’re pre-built, static images that
are tested/verified.

31. Immutable Wins
•
•
•
Super fast server deployment
Failure expected, not a big deal
Testable, more stable

32. Automation Critical
•
•
•
•
Machine Images
Software install/configure
Failure detection, recovery
etc…

33. Machine Images
Easily build machine images for
multiple platforms with Packer.

34. Machine image: A single
deployable unit that
contains a pre-configured
OS and software.

35. Ew... Images.
The anti-image stigma

36. DevOps has historically
been against machine
images.

37. Important to understand
why DevOps has
historically been against
machine images.

38. Golden images
used to be the way.

39. Quarterly, unchanged,
blessed image.

40. Getting any changes (ops
or dev) in was slow and
frustrating.

41. Necessary evil while
tooling wasn’t as good
as it is today.

42. With modern config
management, easy to
simply ignore images.

43. Machine images also have
a ton of benefits.

44. Super fast
infrastructure deployment.

45. Multi-cloud portability

46. Stability and
Testability

47. Analogy: Ops without
machine images is like
applications without binaries

48. Put it another way:
There is no promise of
reproducible builds in ops,
and no binary-like snapshots

49. Source Code
Binary

50. Source Code
libA 1.0
Binary
libB 1.0
libC 1.0

51. Source Code
libA 2.0
Binary
libB 1.0
libC 1.0

52. Source Code
libA 2.0
COMPILE FAILED
libB 1.0
libC 1.0

53. Chef, Puppet, Scripts, etc.
New Server
Ready Server

54. Chef, Puppet, Scripts, etc.
New Server
Packages
Ready Server
Network
Chef/Puppet
Changes

55. Chef, Puppet, Scripts, etc.
SERVER SETUP
FAILED
New Server
Package
Changes
Network
Chef/Puppet
Changes

56. Chef, Puppet, Scripts, etc.
SERVER SETUP
FAILED
New Server
Packages
Network
Unreliable
Chef/Puppet
Changes

57. Chef, Puppet, Scripts, etc.
SERVER SETUP
FAILED
New Server
Packages
Network
Chef/Puppet
Changed

58. Machine Image
New Server
Ready Server

59. Machine Image
New Server
Ready Server

60. Packer
Embraces modern best practices
and automates building images

61. Takes template input and
repeatably produces
image output.

62. $ packer build !
-var ‘aws_access_key=YOUR KEY’ !
-var ‘aws_secret_key=YOUR SECRET’ !
template.json!
...

63. Automated, repeatable,
fast, uses Chef/Puppet/
Shell/etc. Gets rid of old
downsides.

64. Can also produce
Vagrant boxes for
development, at the
same time.

65. Workflows
What does this look like in practice?

66. Ops
Ops Push
Packer Build
Machine Image(s)

67. Ops
Ops Push
Packer Build
Machine Image(s)
Setup users, install
software, configure
software, configure
services. No
orchestration.

68. Server Deploy
Request Server
Running Server

69. Server Deploy
•
Server started with Packerbuilt image
•
Running server has all preinstalled software
•
Service orchestration runs
now
Request Server
Running Server

70. Development
“vagrant up”
Ready to work

71. Development
•
Vagrant downloads the latest
machine image created by
Packer
•
“vagrant up” takes seconds
to minutes and runs, nearly
identical to production
“vagrant up”
Ready to work

72. Packer Wins
•
•
•
•
Automated, repeatable images
Fast deployment from dev to prod
Testable, very stable
Embraces configuration mgmt

73. SERF
Service orchestration, discovery,
failure detection.

74. Remember, Chef/Puppet/etc.
aren’t doing service
orchestration for the image.

75. Service Discovery:
Web servers in a load balancer,
Memcache in a cluster, MySQL
master/slave, etc.

76. Service Orchestration:
Add this node to your cluster,
deploy this application, restart,
etc.

77. Serf is a decentralized, highly
available, fault tolerant, and
lightweight solution to service
orchestration and discovery.

78. Example
Web servers in a load balancer

79. Run the Agent
$ serf agent !
! -event-handler=“handle.sh”!
…

80. Join a Cluster
$ serf join 10.0.1.5!
…

81. Serf Events
Serf Agent
Handle.sh
Add or Remove
Node from LB

82. Serf Events
Serf Agent
handle.sh
Add or Remove
Node from LB
Serf invokes event
handlers for any
event: member-join,
member-leave,
member-failed, user.

83. How?
How does Serf do what it does?

84. GOSSIP-BASED MEMBERSHIP

85. FAILURE DETECTION

86. CUSTOM EVENTS

87. Config Management
Chef/Puppet Run
Configure Serf
Agent Services
Don’t start Serf. Serf
starts on boot.

88. Config Management
Chef/Puppet Run
Configure Serf
agent init script
Don’t start Serf. Serf
starts on boot.
Chef and Puppet just
configure the Serf init
script to start the
agent.

89. Serf Wins
•
•
•
No orchestration in image build
Fast membership detection, don’t
wait for Chef/Puppet
Easily and infinitely extensible

90. The Future is Immutable
Big wins, little downsides.

91. As servers become cheaper, more ondemand, and infrastructures become
more distributed, immutable
infrastructure will win.

92. Important to remember:
You don’t have to go all in on
immutable. You can start with certain
servers. Example: databases are hard,
just ignore that for awhile.

93. Mostly Immutable
There are still big benefits for going
mostly immutable. Things like quick
config changes and deploys can still
be mutable. Serf is good for this.

94. The Wins
•
•
•
•
•
Super fast deployment
Repeatability
High failure tolerance
Improved stability and testability
Versioning

95. The Downsides
•
•
•
Requires proper mindset
Tooling is new
Deploys, simple config changes

96. Thank you!

97. Vagrant
http://www.vagrantup.com
Packer
http://www.packer.io
SERF
http://www.serfdom.io