In this talk from SWE Days 2013 you will get a solid understanding of what “Infrastructure-As-Code” means. You will learn about tools like Chef, Vagrant and Librarian and their role in automating your infrastructure. Lots of concrete code examples and short demos are waiting for you. You WILL want to treat your infrastructure as code at the end of this talk! :-)

1. © Zühlke 2013
Torben Knerr
Cooking with Chef, Knife,
Vagrant & Co
4 Ingredients for tasty “Infrastructure-As-Code”
27. September 2013
Slide 1

2. © Zühlke 2013
With typical components:
• Application Servers
• Database Servers
• Caching Servers
• Load Balancer / Reverse Proxy
• Clustering & Failover
A Typical Web App Project…
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
nginx
Tomcat Tomcat
DB Master
DB Slave
App App
memcached
27. September 2013 Slide 2

3. © Zühlke 2013
With a typical Deployment Pipeline:
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
Local DEV TEST PROD
A Typical Web App Project…
27. September 2013 Slide 3

4. © Zühlke 2013
With typical (?) problems:
• All environments are somehow different
• Nobody knows the differences in detail
• Setting up a new environment takes weeks
• And again it looks different from the others
• Totally no Dev/Prod Parity
• And last but not least…
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
A Typical Web App Project…
27. September 2013 Slide 5

6. © Zühlke 2013
First Approach:
Hey, let’s build VMs!
This already helps:
• Use VMs for a consistent work environment
• Manually set up a local developer VM with Tomcat,
PostgreSQL, nginx, etc…
• Carefully hand-crafted with love!
• Distribute it to the team
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 7

7. © Zühlke 2013
But…
• Manual management of VMs is cumbersome
• How do you version a 40GB VM?
• How do you take care of updates?
• Can you diff the changes?
• Once it works locally, how can you apply this to the
rest of the deployment pipeline?
• How do you parameterize such VMs?
Better than nothing, but still far from perfect!
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 8
First Approach:
Hey, let’s build VMs!

8. © Zühlke 2013
Why Scripting / Automation?
• Lightweight scripts vs. fat VMs
– Allows for proper versioning and diffing!
– Easy to share
• Infrastructure can be reliably recreated
– Minimize manual (= boring & error prone) tasks
– Basis for “clean” infrastructure deployments
– Anywhere in the deployment pipeline
• Explicit design of infrastructure
– Described as an executable specification
– Not formed “by accident”
Treat your infrastructure like code!
Better Solution:
Automate your VMs!
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 9

9. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Welcome to the DevOps
Kitchen!
Infrastructure á la Carte
27. September 2013
Slide 10

10. © Zühlke 2013
Goal: set up a simple server infrastructure, fully automated
• For example on a developer machine
• Separate VMs for app server and database server
• Anywhere reproducible („clean setup“)
Our Example Scenario
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
App Server VM
Tomcat
Database Server VM
PostgreSQL
.war
27. September 2013 Slide 11

11. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Ingredient #1
We need VMs
27. September 2013
Slide 12

12. © Zühlke 2013
Ingredient #1: we need VMs
Enter Vagrant
What is Vagrant?
• Vagrant is “automation for VMs”
• Different VM providers: VirtualBox, aws, rackspace, etc..
• The configuration of VMs is described in a Vagrantfile
• Simple commands for interacting with VMs:
– vagrant up
– vagrant ssh
– vagrant halt
– …
• http://vagrantup.com
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 13

13. Ingredient #1: we need VMs – A simple Vagrantfile

14. Ingredient #1: we need VMs – A simple Vagrantfile
Basebox

15. Ingredient #1: we need VMs – A simple Vagrantfile
App Server VM

16. Ingredient #1: we need VMs – A simple Vagrantfile
Database Server VM

17. Ingredient #1: we need VMs – Interacting with Vagrant VMs
vagrant up

18. Ingredient #1: we need VMs – Interacting with Vagrant VMs
vagrant ssh

19. © Zühlke 2013
Ingredient #1: we need VMs
A quick look at the VirtualBox GUI
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
Is this the App Server VM?
Or the Database Server VM?
27. September 2013 Slide 20

20. Ingredient #1: we need VMs – More VM configuration

21. Ingredient #1: we need VMs – More VM configuration

22. Ingredient #1: we need VMs – More VM configuration
Name, CPUs, RAM, etc…

23. Ingredient #1: we need VMs – More VM configuration
vagrant reload

24. © Zühlke 2013
Ingredient #1: we need VMs
Looking at the VirtualBox GUI again
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
and the VM Parameters are applied too
Better names 
27. September 2013 Slide 25

25. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Ingredient #2:
Provisioning
27. September 2013
Slide 26

26. © Zühlke 2013
Ingredient #2: Provisioning
Back to our Example Scenario
“Naked” VMs are not enough
• On the database server VM we need PostgreSQL
• On the app server VM we need Tomcat
We need a tool (“provisioner”) for installing
and configuring software components on top
of naked VMs
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 27

27. © Zühlke 2013
Ingredient #2: Provisioning
Enter Chef
What is Chef?
• Chef is a “Configuration Management” system
• Installs & configures software on nodes
• Provides a DSL for describing the configuration of a
node in a readable and platform-independent way
More information:
• http://www.opscode.com/chef/
Vagrant supports provisioning of VMs
via Chef out-of-the-box!
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 28

28. Ingredient #2: Provisioning – Installing PostgreSQL

29. Ingredient #2: Provisioning – Installing PostgreSQL
run the PostgreSQL Server recipe

30. Ingredient #2: Provisioning – Installing PostgreSQL
configuration parameters

31. vagrant up db

32. What’s inside the PostgreSQL
Recipe?

34. install the “postgresql“ package

35. create the config file

36. Erzeuge die Konfig-Datei

37. create another config file

38. start the „postgresql“ service

39. we also need the postgresql client…

41. Ingredient #2: Provisioning – Installing Tomcat

42. Ingredient #2: Provisioning – Installing Tomcat
run the Tomcat default recipe
(short for “tomcat::default“)

43. Ingredient #2: Provisioning – Installing Tomcat
tomcat specific configuration

44. vagrant up web

45. © Zühlke 2013
Chef Philosophy
Specification of the target state
• Idempotence
• Convergence (“self-healing“)
Chef DSL abstracts the OS-specific details via
• Resources
• Providers
Naming: kitchen metaphor
• Chef, Cookbook, Recipe, Knife, …
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 46

46. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Ingredient #3:
Managing Cookbooks
27. September 2013
Slide 47

47. © Zühlke 2013
Ingredient #3: Managing Cookbooks
Back to our Example Scenario
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
where is the PostgreSQL cookbook /
recipe coming from???
27. September 2013 Slide 48

48. © Zühlke 2013
Ingredient #3: Managing Cookbooks
Typical Sources for “Community Cookbooks”
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
http://community.opscode.com/cookbooks/
https://github.com/search?q=cookbook
27. September 2013 Slide 49

49. © Zühlke 2013
Ingredient #3: Managing Cookbooks
Enter Librarian
What is Librarian?
• Librarian is a Dependency Management tool for Cookbooks
• Downloads and extracts cookbooks for you
Dependencies are specified in a Cheffile
• Transitive dependencies are resolved as well
• Different cookbook sources and version constraints are respected
More information:
• https://github.com/applicationsonline/librarian-chef
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 50

50. © Zühlke 2013
Ingredient #3: Managing Cookbooks
A simple Cheffile
Cheffile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 51

51. Ingredient #3: Managing Cookbooks – A simple Cheffile
Librarian „installs“ the cookbooks

52. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Ingredient #4:
Application Cookbooks
27. September 2013
Slide 53

53. © Zühlke 2013
Ingredient #4: Application Cookbooks
Back to our Example Scenario
Re-using Community Cookbooks is not enough
• We need a specific database and database user
for our application
• We need to deploy our application’s .war file
in Tomcat
• Existing Community Cookbooks don’t “know” our
specific requirements!
We need to write our own “Application
Cookbook” for this purpose!
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 54

54. © Zühlke 2013
Did you know? You can create cookbooks with a knife!
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
Ingredient #4: Application Cookbooks
Enter Knife
27. September 2013 Slide 55

55. © Zühlke 2013
Ingredient #4: Application Cookbooks
Structure of our “myapp” cookbook
Single cookbook with separate recipes for setting up
the database and webapp
Goal: hide implementation details from the user
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
myapp
attributes
recipes
db.rb
web.rb
db.rb
default.rb
web.rb
27. September 2013 Slide 56

56. © Zühlke 2013
Ingredient #4: Application Cookbooks
Structure of our “myapp” cookbook
Attributes files expose the “configuration parameters”
of a cookbook and supply sensible defaults
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
myapp
attributes
recipes
db.rb
web.rb
db.rb
default.rb
web.rb
27. September 2013 Slide 57

57. © Zühlke 2013
Ingredient #4: Application Cookbooks
Structure of our “myapp” cookbook
• myapp/attributes/db.rb
• myapp/attributes/web.rb
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 58

58. © Zühlke 2013
Ingredient #4: Application Cookbooks
Structure of our “myapp” cookbook
The “db” recipe first installs PostgreSQL, then creates
the database and database user
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
myapp
attributes
recipes
db.rb
web.rb
db.rb
default.rb
web.rb
27. September 2013 Slide 59

60. first install the PostgreSQL server

61. installs prerequisites for interacting with
PostgreSQL from within this recipe

62. creates the database user

63. creates the database

64. © Zühlke 2013
Ingredient #4: Application Cookbooks
Use “myapp::db” in the Vagrantfile
Vagrantfile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 65

65. © Zühlke 2013
Ingredient #4: Application Cookbooks
Use “myapp::db” in the Vagrantfile
Vagrantfile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
instead of “postgresql::server”
27. September 2013 Slide 66

66. vagrant provision db

67. © Zühlke 2013
Ingredient #4: Application Cookbooks
Extended Cheffile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 68

68. © Zühlke 2013
Ingredient #4: Application Cookbooks
Structure of our “myapp” cookbook
The “web” recipe first installs tomcat, then configures
and deploys the our web app
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
myapp
attributes
recipes
db.rb
web.rb
db.rb
default.rb
web.rb
27. September 2013 Slide 69

70. we need to install Tomcat first

71. there is our .war file located

72. database configuration (via context.xml)

73. deploy the webapp to Tomcat

74. © Zühlke 2013
Ingredient #4: Application Cookbooks
Use “myapp::web” in the Vagrantfile
Vagrantfile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 75

75. © Zühlke 2013
Ingredient #4: Application Cookbooks
Use “myapp::web” in the Vagrantfile
Vagrantfile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
instead of the “tomcat::default“ recipe
27. September 2013 Slide 76

76. © Zühlke 2013
Ingredient #4: Application Cookbooks
Use “myapp::web” in the Vagrantfile
Vagrantfile
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
C
connect to this database host
(here: ip address of the db vm)
27. September 2013 Slide 77

77. vagrant provision web

78. Is it running now?

79. Or whatever is in your context…
(we have borrowed the “yabe“ example from Play! here:
http://www.playframework.org/documentation/1.0/samples)

80. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
The DevOps Kitchen –
Recap
27. September 2013
Slide 81

81. © Zühlke 2013
Recap:
Ingredients for cooking in the DevOps Kitchen
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr
Ingr. #1 We need VMs
Ingr. #2 Provisioning
Ingr. #3 Cookbook Management
Ingr. #4 Application Cookbooks
+
+
+
27. September 2013 Slide 82

82. © Zühlke 2013
Fully Automated Infrastructure via Textual Specification
• No shell scripts, but specification of the target state via Chef DSL
• Crisp and concise, but not cryptic
• SCM-friendly: Kilobytes instead of Gigabytes, diffable
• Complex infrastructure available at the push-of-a-button
• Reproducible anywhere in the deployment pipeline
• Good software engineering principles applied to infrastructure
What have we achieved?
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 83

83. © Zühlke 2013
We have barely touched the surface…
There’s much more of Chef, Vagrant & Co we have not seen today:
• More Vagrant features
– Alternative Providers (e.g. vmware, aws, rackspace, managed-servers, etc…)
– Alternative Provisioners (e.g. puppet, shell, ansible, salt, etc…)
– Vagrant Plugins (e.g. vagrant-omnibus, vagrant-berkshelf, vagrant-cachier, …)
– …
• More Chef concepts
– Writing Custom Resources und Providers (LWRPs)
– Data bags
– Environments
– Search
– Chef Solo vs. Chef Server
– …
• And most importantly…
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 84

84. © Zühlke 2013Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 85
…if we treat infrastructure as code, then
we have to test!
Linting
• foodcritic
Unit-Level Testing
• chefspec
• fauxhai
Integration Testing (from the inside)
• chef-minitest-handler
• test-kitchen
Acceptance Testing (outside-in)
• cucumber-nagios

85. © Zühlke 2013
Resources
Some useful stuff to check out…
• Code Examples from this talk
https://github.com/tknerr/oop-chef-demo
• Newer Example based on Vagrant / Chef / Berkshelf
https://github.com/tknerr/sample-application-cookbook
• Bills Kitchen – all you need for cooking with Chef & Vagrant on Windows
https://github.com/tknerr/bills-kitchen
https://ikm.zuehlke.com/topics/Pages/Bill%27s%20Kitchen.aspx
• “Chef, Vagrant & Co” Yammer Group
https://www.yammer.com/zuehlke.com/#/threads/inGroup?type=in_group&feedId=2051695
• “Test-Driven Infrastructure with Chef (2nd ed.)” – highly recommended!
http://shop.oreilly.com/product/0636920030973.do
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 86

86. © Zühlke 2013
Resources
More of the Vagrant Ecosystem…
• Bindler – Manage your Vagrant Plugins via `plugins.json`
https://github.com/fgrehm/bindler
• Selection of useful Vagrant plugins
https://github.com/schisamo/vagrant-omnibus – installs Chef on a “provisionerless” basebox
https://github.com/riotgames/vagrant-berkshelf – resolves cookbook dependencies on `vagrant up`
https://github.com/fgrehm/vagrant-cachier – transparently caches downloaded packages
https://github.com/tmatilai/vagrant-proxyconf – sets the proxy inside the VM
…
• Selection of Vagrant Providers
https://github.com/mitchellh/vagrant-aws – provision EC2 instances in the AWS cloud with Vagrant
https://github.com/tknerr/vagrant-managed-servers – provision “non-VMs” with Vagrant
https://github.com/fgrehm/vagrant-lxc - provision LXC containers
…
Cooking with Chef, Knife, Vagrant & Co | Torben Knerr 27. September 2013 Slide 87

87. © Zühlke 2013
Cooking with Chef, Knife,
Vagrant & Co
Torben Knerr
Discussion Time!
20. Juli 2012
Puppet, Berkshelf,
Foo, Bar, Baz
DevOps, NoOps,
Blablabla
Vagrant, Chef,
Deployment Stuff
Holy Cloud!
Slide 88

88. Thank You! 
And let’s continue the discussion here: