Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

© Zühlke 2012
Torben Knerr
Infrastructure-As-Code and
Cloud Deployments with
Opscode Chef & Co
20. Juli 2012
Slide 1

© Zühlke 2012
Topics
Motivation
 DevOps & Infrastructure-As-Code
Introduction to Chef
 Terms & Concepts (with Examples)
 Chef vs. Puppet
Cloud Deployment
 Different Approaches for Cloud Provisioning
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr 20. Juli 2012 Slide 2

© Zühlke 2012
Opscode Chef & Co
Torben Knerr
Motivation
DevOps & Infrastructure-As-Code
20. Juli 2012
Slide 3

© Zühlke 2012
Motivation
DevOps & Infrastructure-As-Code
Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co | Torben Knerr
DevOps
Drivers:
 Cloud & Virtualisation
 Mass Server Management
 Agile & Continous Delivery
Philosophy:
 Collaboration between Devs and Ops
 Devs understand the needs of Ops
 Ops learn from Devs (Infrastructure-As-Code)
 Ops are part of the (SCRUM) team
Disciplines:
 Operations, Monitoring
 Automation
 Programming (Ruby)
 Configuration Management
Goals:
 Traceability, Auditability
 Maintainability
 Reproducability
 Speed / Time-To-Market
Infrastructure-
As-Code
20. Juli 2012 Slide 4

© Zühlke 2012
Opscode Chef & Co
Torben Knerr
Knife, Cookbooks, Recipes, etc…
20. Juli 2012
Slide 5

© Zühlke 2012
What‘s in a Chef Repository?
chef-repo

© Zühlke 2012
chef-repo
cookbooks
databags
environments
roles

© Zühlke 2012
recipes
attributes
files
templates
chef-repo
cookbooks
databags
environments
roles

© Zühlke 2012
Chef Solo vs. Chef Server
node
couchdb
cookbooks
roles
databags
nodes
…Chef Server
node
node
node
node
chef-client
poll for node’s run_list
and converge node
node
cookbooks
roles
databags
solo.rb
chef-solo
read run_list from solo.rb
and converge node
Chef Solo
 Runs chef-solo locally
 No server required
 Easier to get started with
 Good for few nodes, onetime
bootstraps
chef-
repo
upload
to node

© Zühlke 2012
Chef Solo vs. Chef Server
node
couchdb
cookbooks
roles
databags
nodes
…Chef Server
node
node
node
node
chef-client
poll for node’s run_list
and converge node
node
cookbooks
roles
databags
solo.rb
chef-solo
read run_list from solo.rb
and converge node
Chef Server
 chef-client polls regularly
 More complex workflow
 Additional features:
Search, WebUI
 Good for ongoing
operations of many nodes
chef-
repo
upload to
chef server

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
The Chef Repository is the place where cookbooks,
roles, config files and other artifacts for managing
systems with Chef will live, typically stored in a version
control system.

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Cookbooks are the fundamental units of distribution in
Chef. They encapsulate all the resources you need to
automate your infrastructure and are easily sharable
with other Chef users.

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Attributes are node data such as the IP address,
hostname, etc... New attributes can be defined on
cookbook, environment, role and node level.
cookbooks/apache2/attributes/default.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Recipes encapsulate collections of resources which
are executed in order to configure the system.
cookbooks/apache2/recipes/default.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Recipes encapsulate collections of resources which
are executed in order to configure the system.
Resources
cookbooks/apache2/recipes/default.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Resources are the cross platform abstraction of the
thing you're configuring on the node.
Examples:

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Examples:

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Examples:
 Many, many more in the Opscode Wiki:
http://wiki.opscode.com/display/chef/Resources
 You can also write your own:
http://wiki.opscode.com/display/chef/Lightweight+Resources+and+Providers+
%28LWRP%29

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Data bags provide an arbitrary stores of globally
available JSON data…
chef-repo/data_bags/users/bofh.json

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
…which are then used in Recipes:
cookbooks/users/recipes/default.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
A role provides a means of grouping similar features
of similar nodes, providing a mechanism for easily
composing sets of functionality.
chef-repo/roles/webserver.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Environments in Chef provide a mechanism for
managing different environments such as production,
staging, development, testing, etc
chef-repo/environments/dev.rb

© Zühlke 2012
 Chef Repo
 Cookbooks
 Attributes
 Recipes
 Resources
 Databags
 Roles
 Environments
 Nodes
Terms & Concepts
Nodes are the thing that Recipes and Roles are applied
to. The primary features of a node, from Chef's point
of view, are its Attributes and its run list.
chef-repo/nodes/web.example.com.rb

© Zühlke 2012
How to get started?
• If you are working on Windows use Bill’s Kitchen
https://ikm.zuehlke.com/topics/Pages/Bill's%20Kitchen.aspx
• Start with Vagrant and Chef-Solo (you will love Vagrant…)
http://vagrantup.com/v1/docs/provisioners/chef_solo.html
• Get familiar with cookbook development and testing
https://github.com/tknerr/bills-kitchen/blob/master/COOKBOOK_DEVELOPMENT.md
• Set up a Chef Server using knife-server
http://fnichol.github.com/knife-server/
• Use Vagrant with Chef-Server Provisioning
http://vagrantup.com/v1/docs/provisioners/chef_server.html
• Get familiar with knife and use advanced features like search, bootstrap, etc..
http://wiki.opscode.com/display/chef/Knife
Getting Started with Chef Development

© Zühlke 2012
VirtualBox Automation with Vagrant
Vagranfile

© Zühlke 2012
Cookbook Dependency Management
with Librarian
Cheffile

© Zühlke 2012
Test-Driven Infrastrucure!
Lint Checking
• foodcritic
Unit-Level Testing
• chefspec
• fauxhai
Smoke Tests (from the inside)
• chef-minitest-handler
Acceptance Testing (from the outside)
• cucumber-nagios

© Zühlke 2012
It doesn’t really matter (for us)
 Same concepts, slightly different terminology
 External vs. internal DSL (like Maven vs. Gradle)
 Both have solo / standalone mode
 There are more Puppet books than Chef books
 Both have a very active community (mailing list, IRC, etc..)
 Do you have in-house knowledge of Chef/Puppet?
Chef vs. Puppet

© Zühlke 2012
It doesn’t really matter (for us)
 Same concepts, slightly different terminology
 External vs. internal DSL (like Maven vs. Gradle)
 Both have solo / standalone mode
 There are more Puppet books than Chef books
 Both have a very active community (mailing list, IRC, etc..)
 Do you have in-house knowledge of Chef/Puppet?
Chef vs. Puppet
Use the tool that “operations” understands!

© Zühlke 2012
Opscode Chef & Co
Torben Knerr
Cloud Deployment
Different Approaches for Provisioning Cloud
Instances
20. Juli 2012
Slide 33

© Zühlke 2012
Cloud Deployment
The Basic Mechanism: EC2 User Data
You can pass arbitrary user data when launching an instance…
…and access it on the instance via the metadata service:
Notes:
 User data is limited to 16K in raw form (before base64 encoding)

© Zühlke 2012
Cloud Deployment
Option 1: Provisioning with Shell Scripts
Convention: if the user data script starts with a `#!` it will be run at first boot
Notes:
 Might not be supported on all AMIs (for the official Ubuntu AMIs it is)
path/to/bootstrap.sh

© Zühlke 2012
Cloud Deployment
Option 1: Provisioning with Shell Scripts
Convention: if the user data script starts with a `#!` it will be run at first boot
Notes:
 Might not be supported on all AMIs (for the official Ubuntu AMIs it is)
path/to/bootstrap.sh
 easiest method
 good if bootstrapping is simple
 one-off bootstrap (first boot)

© Zühlke 2012
 VM instances are bootstrapped
with Chef/Puppet via Cloud-init
 The rest is just AWS
CloudFormation Templates
 Supports Chef Server, Chef Solo and
Puppet Client Server
Cloud Deployment
Option 2: AWS CloudFormation Templates

© Zühlke 2012
 VM instances are bootstrapped
with Chef/Puppet via Cloud-init
 The rest is just AWS
 Supports Chef Server, Chef Solo and
Puppet Client Server
Cloud Deployment
Option 2: AWS CloudFormation Templates
 good for more complex
scenarios
 official, AWS supported
documentation available
 not meant for humans to read

© Zühlke 2012
 Vagrant-like approach for
managing VMs in the cloud
 Supports Chef, Puppet and
Shell provisioning
 Currently focused on AWS
but other providers possible
 Supports stacks via AWS
Cloud Deployment
Option 3: Mccloud

© Zühlke 2012
 Vagrant-like approach for
managing VMs in the cloud
 Supports Chef, Puppet and
Shell provisioning
 Currently focused on AWS
but other providers possible
 Supports stacks via AWS
Cloud Deployment
Option 3: Mccloud
Yay! :-)

© Zühlke 2012
 knife-ec2
 knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef
 Works only with Chef Server – no Chef Solo support
 Puppet Cloud Provisioner
 Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet
 Works only with Puppet Master – no Puppet Standalone mode (?)
Cloud Deployment
Option 4: Native Chef / Puppet Tools

© Zühlke 2012
 knife-ec2
 knife plugin to let you launch, bootstrap and provision EC2 VMs with Chef
 Works only with Chef Server – no Chef Solo support
 Puppet extension to launch, bootstrap and provision EC2 VMs with Puppet
 Works only with Puppet Master – no Puppet Standalone mode (?)
Cloud Deployment
Option 4: Native Chef / Puppet Tools
 You can’t have both: it’s Chef OR Puppet
 Fallback if the generic tools (Option 2 & 3)
don’t work out as expected
 No Solo / Standalone mode

© Zühlke 2012
 EC2 User Data Scripts
http://alestic.com/2009/06/ec2-user-data-scripts
https://help.ubuntu.com/community/CloudInit
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
 AWS CloudFormation Templates with Chef/Puppet
https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithOpscodeChef.pdf
https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithPuppet.pdf
 Mccloud
https://github.com/jedi4ever/mccloud
https://github.com/jedi4ever/mccloud-demo
https://gist.github.com/3175267
 knife-ec2
http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife
http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html
Cloud Deployment
Overview & Resources

© Zühlke 2012
Opscode Chef & Co
Torben Knerr
Discussion Time!
Puppet, Mccloud,
Foo, Bar, Baz
DevOps, NoOps,
Blablabla
Vagrant, Chef,
Deployment Stuff
Cloudy Cloud!
20. Juli 2012
Slide 45

Thank You! 
And let’s continue the discussion here:

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co

Similar to Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co (20)

More from Torben Knerr

More from Torben Knerr (7)

Recently uploaded

Recently uploaded (20)

Infrastructure-As-Code and Cloud Deployments with Opscode Chef & Co