SlideShare a Scribd company logo
1 of 68
Download to read offline
INFRASTRUCTURE AS CODE
                         Managing System Complexity with Chef
                            Robert J. Berger - CTO Runa, Inc.
                          rberger@runa.com http:/ /blog.ibd.com




Friday, April 22, 2011                                            1
Image: http://techcrunch.com/2010/03/16/big-data-freedom/
MOORE’S LAW




Friday, April 22, 2011                                       2
Graph: http://www.gotw.ca/publications/concurrency-ddj.htm
MOORE’S LAW
           HITS A
           SPEED
           BUMP
              CPU speed
               growth
                 has
              STOPPED


Friday, April 22, 2011                                       2
Graph: http://www.gotw.ca/publications/concurrency-ddj.htm
BUT WE GET:
          GROWTH OF CORES &
       FOOTBALL FIELDS OF SERVERS
Friday, April 22, 2011                                                                             3
Data Center Image: http://www.datacenterknowledge.com/inside-microsofts-dublin-mega-data-center/
Chart: http://news.cnet.com/8301-13924_3-10101987-64.html
BUT WE GET:
          GROWTH OF CORES &
       FOOTBALL FIELDS OF SERVERS
Friday, April 22, 2011                                                                             3
Data Center Image: http://www.datacenterknowledge.com/inside-microsofts-dublin-mega-data-center/
Chart: http://news.cnet.com/8301-13924_3-10101987-64.html
VON NEUMANN SEQUENTIAL
                   TRANSFORMS TO...
Friday, April 22, 2011                      4
Image: http://www.necessarysufficient.com
ACTORS,
Friday, April 22, 2011                                                                         5
Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.html
Map Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.html
Nanites: http://www.startrek.com/database_article/nanites
Runa Infrastructure Diagram: Robert J. Berger
Hadoop / HBase                             Cheshire / Swarmiji
                                                                           Map / Reduce
                                                                           Petabyte Store
                                                                                                               Dynamic     Monitor &     HTTP
                                                                                                               Runtime     Recovery    Dispatchers



                                                                            HBase HBase
                                                                               HBase
                                                                             HBase HBase
                                                                                HBase
                                                                              HBase HBase
                                                                                  HBase       Redis
                                                                            HBase HBase
                                                                               HBase HBase
                                                                                HBase
                                                                                   HBase        Redis
                                                                                              Mem
                                                                             HBase HBase
                                                                                HBase            Redis
                                                                                                Mem
                                                                                              CacheRedis
                                                                                                                            Queue
                                                                              HBase HBase
                                                                                  HBase           Mem
                                                                                               CacheRedis
                                                                            HBase HBase
                                                                               HBase HBase
                                                                                HBase
                                                                                   HBase            Mem
                                                                                                 Cache
                                                                             HBase HBase
                                                                                HBase                 Mem
                                                                                                   Cache
                                                                              HBase HBase
                                                                                  HBase              Cache
                                                                                HBase HBase
                                                                                   HBase


                                                                                                     Shared
                                                                                                               Reporting   Analytics     Data
                                                                                                     Session
                                                                                                                                       Collectors
                                                                                                     Memory




               ACTORS, MAP/REDUCE,
               NANITE... PARALLELISM!
Friday, April 22, 2011                                                                                                                               5
Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.html
Map Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.html
Nanites: http://www.startrek.com/database_article/nanites
Runa Infrastructure Diagram: Robert J. Berger
COMPLEXITY EXPLOSION




Friday, April 22, 2011                                      6
Complexity Background: http://www.c0d3m0nk3y.com/gallery/
COMPLEXITY EXPLOSION
      Lots of moving heterogeneous parts

                                                                Not just in the LAMP stack

        Auto scaling
                                                                     Horizontal scaling


                         Clouds

                                                            Monitoring & Management


Friday, April 22, 2011                                                                    6
Complexity Background: http://www.c0d3m0nk3y.com/gallery/
CODE YOUR
                          INFRASTRUCTURE




Friday, April 22, 2011                                              7
Background: http://www.flickr.com/photos/jason-samfield/4398151919/
CODE YOUR
                          INFRASTRUCTURE
         Programming is all about managing complexity

         Apply same techniques to complexity of infrastructure

         Revision control

         Keep it DRY

         Build on the shoulders of others

Friday, April 22, 2011                                              7
Background: http://www.flickr.com/photos/jason-samfield/4398151919/
CHEF: THE RUBY WAY TO
         MANAGE INFRASTRUCTURE
         Plain Old Ruby programming

         External DSL adds language constructs for service
         deployment

         Services to support Configuration Management

         Community Cookbooks



Friday, April 22, 2011                                                   8
Background Image: http://www.flickr.com/photos/steveritchie/3779779586/
CHEF: THE RUBY WAY TO
         MANAGE INFRASTRUCTURE
         Plain Old Ruby programming

         External DSL adds language constructs for service
         deployment

         Services to support Configuration Management

         Community Cookbooks
         ENABLES YOU TO CODE YOUR INFRASTRUCTURE


Friday, April 22, 2011                                                   8
Background Image: http://www.flickr.com/photos/steveritchie/3779779586/
WHAT’S IN THE KITCHEN
   Chef Server
                             Authoritative




                                                       Chef-Client
                                                                     Chef-Client

                                                                                   Chef-Client
                             Data bags
                             Roles
                             Cookbooks
                             Node states                                                                       Remote
                                                    Deployed Infrastructure                                     Repo




                                                                               Local Project Repo of Cookbooks
                                                                                 Do your cooking (editing)
                                                                               Chef Knife client
                                                                                 Push Data Bags, Roles, Cookbooks
                                     PowerBook G4




                                                                                 Launch/Bootstrap instances
                         Your Dev Environment
Friday, April 22, 2011                                                                                                  9
CHEF SERVER

    •    Manage & Authenticate    •   Support search on Data
         clients (nodes, dev,         Bags, Roles and Node
         admin)                       states.

    •    Assign roles &           •   Serve a RESTful API for
         Configurations to nodes       humans and machines

    •    Track states of Nodes    •   Can run your own or use
                                      Opscode Platform


Friday, April 22, 2011                                          10
PROJECT REPO

         Data Bags                               Cookbooks
           -App Level                              -"Standards" from
            Attributes                              cookbooks.Opscode.com
         Roles                                   Site-Cookbooks
           -A Nodes Purpose                        -Your own cookbooks
         Environments                              -Over-rides of
           -Staging, Production   PowerBook G4
                                                     Standard Cookbooks
           -New in 0.10.x


Friday, April 22, 2011                                                      11
COOKBOOKS

         Attributes                                                      Recipes
         -Key-Value pairs                                                -Where the Action is
           associated with a node                                        -Specifies resources
         -Hierarchy of overrides                                           that the chef client
         Files                                                             should take action on
         -Static Files to be                                             Templates
           copied to node                                                -ERB files to build
         Library                                                           config files on node
         -Place to DRY up Ruby                                           Providers / Resources
           code used in Recipes                                          -DIY DSL Extensions
Friday, April 22, 2011                                                                             12
You only need Recipes. Most cookbooks are just Recipe[s] and Templates
Background: Clipart Library
PLETHORA OF EXISTING
                       COOKBOOKS




Friday, April 22, 2011                                                                                                  13
These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one
PLETHORA OF EXISTING
                       COOKBOOKS
               activemq               erlang                    maven                      pxe_dust                     stompserver
               ant                    fail2ban                  memcached                  python                       subversion
               apache2                gems                      mercurial                  quick_start                  sudo
               apparmor               git                       munin                      rabbitmq                     teamspeak
               application            glassfish                 mysql                      rabbitmq_chef                teamspeak3
               apt                    gnu_parallel              nagios                     radiant                      thrift
               aws                    god                       nanite                     rails                        tmux
               bluepill               gunicorn                  nginx                      rails_enterprise             tomcat
               boost                  hadoop                    nscd                       redmine                      tomcat6
               build-essential        haproxy                   ntp                        reprepro                     trac
               capistrano             heartbeat                 ohai                       resolver                     transmission
               chef                   imagemagick               one-shot                   riak                         ubuntu
               chef-client            instiki                   openldap                   rsync                        ucspi-tcp
               cloudkick              iptables                  openssh                    rsyslog                      unicorn
               couchdb                java                      openssl                    ruby                         users
               cron                   java_sun                  openvpn                    ruby_enterprise              varnish
               daemontools            jetty                     ossec                      rubygems                     vim
               database               jira                      packages                   runit                        wordpress
               django                 jpackage                  pacman                     rush                         xfs
               djbdns                 keepalived                passenger_apache2          samba                        xml
               dmg                    kickstart                 passenger_enterprise       sbuild                       zenoss
               drbd                   logrotate                 pdns                       screen                       zlib
               dynect                 logwatch                  perl                       snort                        zsh
               dynomite               lvm                       php                        solr
               ec2                    man                       postfix                    sqlite
               emacs                  maradns                   postgresql                 ssh_known_hosts


Friday, April 22, 2011                                                                                                                 13
These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one
MOST IMPORTANT WIKI PAGE
        WIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES




Friday, April 22, 2011                                                                14
Whole range of Directives. Everything from copying files from file systems, URLs, etc
to deploy from Git/SVN... Similar to Capistrano. But its pull instead of push
MOST IMPORTANT WIKI PAGE
        WIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES




                                                                                         Chef
                                                                                         DSL
                                                                                      Directives




Friday, April 22, 2011                                                                             14
Whole range of Directives. Everything from copying files from file systems, URLs, etc
to deploy from Git/SVN... Similar to Capistrano. But its pull instead of push
A SIMPLE COOKBOOK:
                              RABBITMQ




Friday, April 22, 2011                        15
RECIPES:
                               default.rb

        package "rabbitmq-server" do
          action :install
        end

        service "rabbitmq-server" do
          supports :status => true, :restart => true, :reload => true
          action [ :enable, :start ]
        end




Friday, April 22, 2011                                                  16
RECIPES:
                               default.rb
                                             Install Packages
                                       (Knows about many package formats)
        package "rabbitmq-server" do
          action :install
        end

        service "rabbitmq-server" do
          supports :status => true, :restart => true, :reload => true
          action [ :enable, :start ]
        end




Friday, April 22, 2011                                                      16
RECIPES:
                               default.rb

        package "rabbitmq-server" do
          action :install
        end

        service "rabbitmq-server" do
          supports :status => true, :restart => true, :reload => true
          action [ :enable, :start ]
        end
                                 Defines a service that other
                                     recipes can trigger
                                (Knows about many init style mechanism)

Friday, April 22, 2011                                                    16
RECIPES: default.rb pt 2

             case node[:platform]
             when "ubuntu"
               template "/etc/rabbitmq/rabbitmq.conf" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             else
               template "/etc/rabbitmq/rabbitmq.config" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             end
Friday, April 22, 2011                                                          17

ERB: Embedded Ruby
RECIPES: default.rb pt 2
                          Tweak based on target OS
                                      (Its just Ruby!)

             case node[:platform]
             when "ubuntu"
               template "/etc/rabbitmq/rabbitmq.conf" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             else
               template "/etc/rabbitmq/rabbitmq.config" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             end
Friday, April 22, 2011                                                          17

ERB: Embedded Ruby
RECIPES: default.rb pt 2

                                      Chefisms tend to be blocks with
             case node[:platform]
             when "ubuntu"                 optional parameters
               template "/etc/rabbitmq/rabbitmq.conf" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             else
               template "/etc/rabbitmq/rabbitmq.config" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             end
Friday, April 22, 2011                                                          17

ERB: Embedded Ruby
RECIPES: default.rb pt 2

             case node[:platform]
             when "ubuntu"
               template "/etc/rabbitmq/rabbitmq.conf" do
                  source "rabbitmq.config.erb"
                  owner "root"                  Creates files using ERB
                  group "root"
                  mode 0644
                                                   (ERB Params could be set     here)
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             else
               template "/etc/rabbitmq/rabbitmq.config" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             end
Friday, April 22, 2011                                                                  17

ERB: Embedded Ruby
RECIPES: default.rb pt 2

             case node[:platform]
             when "ubuntu"
               template "/etc/rabbitmq/rabbitmq.conf" do
                  source "rabbitmq.config.erb"
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             else                                Using the service defined
               template "/etc/rabbitmq/rabbitmq.config" do
                  source "rabbitmq.config.erb"            earlier
                  owner "root"
                  group "root"
                  mode 0644
                  notifies :restart, resources(:service => "rabbitmq-server")
               end
             end
Friday, April 22, 2011                                                          17

ERB: Embedded Ruby
ATTRIBUTES:
                          default.rb


 default[:rabbitmq][:nodename] = "rabbit"
 default[:rabbitmq][:address] = "0.0.0.0"
 default[:rabbitmq][:port] = "5672"
 default[:rabbitmq][:erl_args] = "+K true +A 30 
 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] 
 -kernel inet_default_connect_options [{nodelay,true}]"
 default[:rabbitmq][:start_args] = ""
 default[:rabbitmq][:logdir] = "/var/log/rabbitmq"
 default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia"
 default[:rabbitmq][:cluster] = "no"
 default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config"
 default[:rabbitmq][:cluster_disk_nodes] = []




Friday, April 22, 2011                                                             18
ATTRIBUTES:
                          default.rb

  Sets the default value
 default[:rabbitmq][:nodename] = "rabbit"
 default[:rabbitmq][:address] = "0.0.0.0"
 default[:rabbitmq][:port] = "5672"
 default[:rabbitmq][:erl_args] = "+K true +A 30 
 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] 
 -kernel inet_default_connect_options [{nodelay,true}]"
 default[:rabbitmq][:start_args] = ""
 default[:rabbitmq][:logdir] = "/var/log/rabbitmq"
 default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia"
 default[:rabbitmq][:cluster] = "no"
 default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config"
 default[:rabbitmq][:cluster_disk_nodes] = []




Friday, April 22, 2011                                                             18
ATTRIBUTES:
                          default.rb
                                                     Precedence
                                                    -default
                                                    -set
 default[:rabbitmq][:nodename] = "rabbit"
                                                    -override
 default[:rabbitmq][:address] = "0.0.0.0"           -File type
 default[:rabbitmq][:port] = "5672"
 default[:rabbitmq][:erl_args] = "+K true +A 30 
 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] 
 -kernel inet_default_connect_options [{nodelay,true}]"
 default[:rabbitmq][:start_args] = ""
 default[:rabbitmq][:logdir] = "/var/log/rabbitmq"
 default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia"
 default[:rabbitmq][:cluster] = "no"
 default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config"
 default[:rabbitmq][:cluster_disk_nodes] = []




Friday, April 22, 2011                                                             18
ATTRIBUTES:
                          default.rb
                                                      Automatic
                                            Ohai discovers OS & HW
                                         instance attributes at runtime
 default[:rabbitmq][:nodename] = "rabbit"
 default[:rabbitmq][:address] = "0.0.0.0"
 default[:rabbitmq][:port] = "5672"
 default[:rabbitmq][:erl_args] = "+K true +A 30 
 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] 
 -kernel inet_default_connect_options [{nodelay,true}]"
 default[:rabbitmq][:start_args] = ""
 default[:rabbitmq][:logdir] = "/var/log/rabbitmq"
 default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia"
 default[:rabbitmq][:cluster] = "no"
 default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config"
 default[:rabbitmq][:cluster_disk_nodes] = []




Friday, April 22, 2011                                                             18
Templates:
                         rabbitmq.config.erb


    NODENAME=<%= node[:rabbitmq][:nodename] %>
    NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %>
    NODE_PORT=<%= node[:rabbitmq][:port] %>
    SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %>
    CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %>
    LOG_BASE=<%= node[:rabbitmq][:logdir] %>
    MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %>
    SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %>




Friday, April 22, 2011                                            19
Templates:
                            rabbitmq.config.erb

                   Creates configuration files using attributes
    NODENAME=<%= node[:rabbitmq][:nodename] %>
    NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %>
    NODE_PORT=<%= node[:rabbitmq][:port] %>
    SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %>
    CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %>
    LOG_BASE=<%= node[:rabbitmq][:logdir] %>
    MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %>
    SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %>




Friday, April 22, 2011                                            19
Templates:
                           rabbitmq.config.erb
                  node object created when chef-client runs
                       (attribute precedence resolved)
    NODENAME=<%= node[:rabbitmq][:nodename] %>
    NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %>
    NODE_PORT=<%= node[:rabbitmq][:port] %>
    SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %>
    CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %>
    LOG_BASE=<%= node[:rabbitmq][:logdir] %>
    MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %>
    SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %>




Friday, April 22, 2011                                            19
Templates:
                          rabbitmq.config.erb
                Final File is rendered and copied to location
                     specified by the Template Directive
    NODENAME=rabbit
    NODE_IP_ADDRESS=0.0.0.0
    NODE_PORT=5672
    SERVER_ERL_ARGS=+K true +A 30 -kernel inet_default_listen_options
    [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel
    inet_default_connect_options [{nodelay,true}]
    CLUSTER_CONFIG_FILE=/etc/rabbitmq/rabbitmq_cluster.config
    LOG_BASE=/var/log/rabbitmq
    MNESIA_BASE=/var/lib/rabbitmq/mnesia
    SERVER_START_ARGS=



Friday, April 22, 2011                                                  19
ROLE: rabbitmq.rb
     name "rabbitmq"
     description "Deploy rabbitmq instance"
     recipes "ulimits", "rabbitmq::default"
     override_attributes({
        :rabbitmq => {
           :port => "8888"
        },
        "ulimits_list" => [
           {
           :domain => "rabbitmq",
           :type => "soft",
           :item => "nofile",
           :value => 32768
           }
        ]
     })

Friday, April 22, 2011                        20
ROLE: rabbitmq.rb
     name "rabbitmq"
                                       Specify recipes needed
     description "Deploy rabbitmq instance"
     recipes "ulimits", "rabbitmq::default"
     override_attributes({
        :rabbitmq => {
           :port => "8888"
        },
        "ulimits_list" => [
           {
           :domain => "rabbitmq",
           :type => "soft",
           :item => "nofile",
           :value => 32768
           }
        ]
     })

Friday, April 22, 2011                                          20
ROLE: rabbitmq.rb
     name "rabbitmq"
     description "Deploy rabbitmq instance"
     recipes "ulimits", "rabbitmq::default"
     override_attributes({
        :rabbitmq => {
           :port => "8888"          Attribute Overrides
        },
        "ulimits_list" => [
           {
           :domain => "rabbitmq",
           :type => "soft",
           :item => "nofile",
           :value => 32768
           }
        ]
     })

Friday, April 22, 2011                                    20
A TASTE OF A SPICIER COOKBOOK:
                                                APPLICATION/
                                                  DATABASE




Friday, April 22, 2011                                         21
Just touch on the most powerful Cookbook pair

Can use the same scafolding for many applications
Image: http://www.theepochtimes.com/n2/content/view/46924/
DATA BAG DRIVEN:
                  MAP RECIPES TO ROLES IN BAG

  {
       "id": "my_app",
       "server_roles": [
          "my_app"
       ],
       "type": {
          "my_app": [
            "rails",
            "postfix",
            "unicorn"
          ]
       }, ...



Friday, April 22, 2011                           22
Snippet of a JSON representation of a Data Bag
DATA BAG DRIVEN:
                  MAP RECIPES TO ROLES IN BAG

  {
       "id": "my_app",
                                                 Application specific role(s),
       "server_roles": [                         typically the name of the app
          "my_app"
       ],
       "type": {
          "my_app": [
            "rails",
            "postfix",
            "unicorn"
          ]
       }, ...



Friday, April 22, 2011                                                           22
Snippet of a JSON representation of a Data Bag
DATA BAG DRIVEN:
                  MAP RECIPES TO ROLES IN BAG

  {
       "id": "my_app",
       "server_roles": [
          "my_app"
       ],
       "type": {
          "my_app": [
            "rails",
                                                 Recipes in application
            "postfix",                           cookbook to run for this role
            "unicorn"
          ]
       }, ...



Friday, April 22, 2011                                                           22
Snippet of a JSON representation of a Data Bag
DEFAULT RECIPE:


      search(:apps) do |app|
        (app["server_roles"] &
      node.run_list.roles).each do |app_role|
          app["type"][app_role].each do |thing|
            node.run_state[:current_app] = app
            include_recipe "application::#{thing}"
          end
        end
      end

      node.run_state.delete(:current_app)




Friday, April 22, 2011                                                                   23
This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.
DEFAULT RECIPE:
                                         ITS KIND OF META


      search(:apps) do |app|
        (app["server_roles"] &
      node.run_list.roles).each do |app_role|
          app["type"][app_role].each do |thing|
            node.run_state[:current_app] = app
            include_recipe "application::#{thing}"
          end
        end
      end

      node.run_state.delete(:current_app)




Friday, April 22, 2011                                                                   23
This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.
DEFAULT RECIPE:
  EXECUTES ON EACH NODE BY CHEF-CLIENT
     Search thru all ‘apps’ databag

      search(:apps) do |app|
        (app["server_roles"] &
      node.run_list.roles).each do |app_role|
          app["type"][app_role].each do |thing|
            node.run_state[:current_app] = app
            include_recipe "application::#{thing}"
          end
        end
      end

      node.run_state.delete(:current_app)




Friday, April 22, 2011                                                                   23
This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.
DEFAULT RECIPE:
  EXECUTES ON EACH NODE BY CHEF-CLIENT

                                                                Loop thru roles that
      search(:apps) do |app|
        (app["server_roles"] &
                                                                match running node’s run_list
      node.run_list.roles).each do |app_role|
          app["type"][app_role].each do |thing|
            node.run_state[:current_app] = app
            include_recipe "application::#{thing}"
          end
        end
      end

      node.run_state.delete(:current_app)




Friday, April 22, 2011                                                                      23
This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.
DEFAULT RECIPE:
  EXECUTES ON EACH NODE BY CHEF-CLIENT


      search(:apps) do |app|
        (app["server_roles"] &
      node.run_list.roles).each do |app_role|
          app["type"][app_role].each do |thing|
            node.run_state[:current_app] = app
            include_recipe "application::#{thing}"
          end
        end                                                                 Loop thru & run recipes
      end
                                                                            for each role nginx
      node.run_state.delete(:current_app)                                                unicorn
                                                                                         rails
                                                                                         postfix

Friday, April 22, 2011                                                                                23
This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.
CHEF-CLIENT
                           (RUNTIME)




Friday, April 22, 2011                 24
CHEF-CLIENT
                            (RUNTIME)

    •    All work done by client on instances (not server)

    •    Runs on each instance regularly (every 1/2 by default)

    •    Roles, Cookbook, Data Bag, Attribute changes
         propagate from Chef Server to instances via the
         chef-client

    •    Client sends attributes to Chef Server making them
         searchable

Friday, April 22, 2011                                            24
THE AMAZING KNIFE
                            CLI for Chef Server




                                   PowerBook G4




Friday, April 22, 2011                            25
THE AMAZING KNIFE
                            CLI for Chef Server

         Manage Cookbooks                         Manage Data Bags
          -Create                                 Node Mgmt & Status
          -Upload                                 Role Management
          -Syntax Check                           Client Management
          -Generate Metadata                      Search
          -Access Public                            -Nodes, Roles, Data
           Cookbooks                                  Bags
          -Git Vendor Pulls
                                   PowerBook G4




          -Publish

Friday, April 22, 2011                                                    25
PowerBook G4




Friday, April 22, 2011                  26
AND THAT’S NOT ALL!

         Manage Knife Config                     Create, List, Delete Cloud
         Multi-ssh                              Instances
           -Select by Search                    - EC2
         Bootstrap Servers                      - Rackspace
         -From Bare Basic OS                    - Slicehost
                                                - Terremark

                                 PowerBook G4




Friday, April 22, 2011                                                       26
PUSH YOUR COOKBOOKS
                         TO THE CHEF SERVER




Friday, April 22, 2011                        27
PUSH YOUR COOKBOOKS
                          TO THE CHEF SERVER



                         knife cookbook upload -a

      Pushes all cookbooks in your repo to chef server




Friday, April 22, 2011                                   27
PUSH YOUR ROLES
                          TO THE CHEF SERVER




Friday, April 22, 2011                         28
PUSH YOUR ROLES
                          TO THE CHEF SERVER



               knife role from file roles/rabbitmq.rb

               Pushes the rabbitmq role to the chef server




Friday, April 22, 2011                                       28
LAUNCH RABBITMQ ON EC2


  knife ec2 server create 
  'role[production]' 'role[base]' role[rabbitmq]' 
  -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu 
  -G production,rabbitmq -i ami-88f504e1 -f m1.large




Friday, April 22, 2011                                 29
LAUNCH RABBITMQ ON EC2
             Uses Fog to issue EC2 api calls to configure
            and create an instance with these parameters
  knife ec2 server create 
  'role[production]' 'role[base]' role[rabbitmq]' 
  -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu 
  -G production,rabbitmq -i ami-88f504e1 -f m1.large




Friday, April 22, 2011                                     29
LAUNCH RABBITMQ ON EC2
                    “Run List” specifies roles for this instance

  knife ec2 server create 
  'role[production]' 'role[base]' role[rabbitmq]' 
  -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu 
  -G production,rabbitmq -i ami-88f504e1 -f m1.large




Friday, April 22, 2011                                            30
LAUNCH RABBITMQ ON EC2
                    “Run List” specifies roles for this instance

  knife ec2 server create 
  'role[production]' 'role[base]' role[rabbitmq]' 
  -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu 
  -G production,rabbitmq -i ami-88f504e1 -f m1.large


                            Works with other Clouds:
                     Rackspace, Terramark, Slicehost, Bluebox


Friday, April 22, 2011                                            30
LIKE BEING AT BENIHANA’S




Friday, April 22, 2011                                                                            31
Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/
LIKE BEING AT BENIHANA’S
               Launches ec2 instance[s] with AMIs you specified
               Installs Ruby & packages for building RubyGems
               Installs RubyGems from source
               Installs Chef from RubyGems
               Creates the /etc/chef directory
               Writes validation cert to /etc/chef/validation.pem
               Writes an /etc/chef/client.rb config file
               Writes a JSON file, /etc/chef/first-boot.json with
               roles and recipes specified as the run list
               Executes chef-client with the first-boot.json,
               connecting to the chef server in the client.rb

Friday, April 22, 2011                                                                            31
Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/
KEEP YOUR INFRASTRUCTURE
  UNDER CONTROL WITH CHEF!
Friday, April 22, 2011                                                                                       32
Image: http://forums.brickarms.com/yaf_postst18328_The-Evil-Chefs-Are-Back-and-With-Even-More-Weapons.aspx

More Related Content

More from Robert Berger

Scaling Runa Inc Big Data e-commerce service with AWS
Scaling Runa Inc Big Data e-commerce service with AWSScaling Runa Inc Big Data e-commerce service with AWS
Scaling Runa Inc Big Data e-commerce service with AWSRobert Berger
 
Personal Object Technology
Personal Object TechnologyPersonal Object Technology
Personal Object TechnologyRobert Berger
 
Facilitating Distributed Last Mile Broadband Access Providers
Facilitating Distributed Last Mile Broadband Access ProvidersFacilitating Distributed Last Mile Broadband Access Providers
Facilitating Distributed Last Mile Broadband Access ProvidersRobert Berger
 
The Linux Probability Wave
The Linux Probability WaveThe Linux Probability Wave
The Linux Probability WaveRobert Berger
 
802.11: Ethernet Marches On
802.11: Ethernet Marches On802.11: Ethernet Marches On
802.11: Ethernet Marches OnRobert Berger
 
Wireless commonsense fontsfixed
Wireless commonsense fontsfixedWireless commonsense fontsfixed
Wireless commonsense fontsfixedRobert Berger
 
Orchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and ChefOrchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and ChefRobert Berger
 
Chef 0.8, Knife and Amazon EC2
Chef 0.8, Knife and Amazon EC2Chef 0.8, Knife and Amazon EC2
Chef 0.8, Knife and Amazon EC2Robert Berger
 

More from Robert Berger (9)

Scaling Runa Inc Big Data e-commerce service with AWS
Scaling Runa Inc Big Data e-commerce service with AWSScaling Runa Inc Big Data e-commerce service with AWS
Scaling Runa Inc Big Data e-commerce service with AWS
 
Personal Object Technology
Personal Object TechnologyPersonal Object Technology
Personal Object Technology
 
Facilitating Distributed Last Mile Broadband Access Providers
Facilitating Distributed Last Mile Broadband Access ProvidersFacilitating Distributed Last Mile Broadband Access Providers
Facilitating Distributed Last Mile Broadband Access Providers
 
The Linux Probability Wave
The Linux Probability WaveThe Linux Probability Wave
The Linux Probability Wave
 
802.11: Ethernet Marches On
802.11: Ethernet Marches On802.11: Ethernet Marches On
802.11: Ethernet Marches On
 
Open spectrum
Open spectrumOpen spectrum
Open spectrum
 
Wireless commonsense fontsfixed
Wireless commonsense fontsfixedWireless commonsense fontsfixed
Wireless commonsense fontsfixed
 
Orchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and ChefOrchestrating HBase Cluster Deployment with Ironfan and Chef
Orchestrating HBase Cluster Deployment with Ironfan and Chef
 
Chef 0.8, Knife and Amazon EC2
Chef 0.8, Knife and Amazon EC2Chef 0.8, Knife and Amazon EC2
Chef 0.8, Knife and Amazon EC2
 

Recently uploaded

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 

Recently uploaded (20)

Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 

Deploying infrastructure with Opscode Chef

  • 1. INFRASTRUCTURE AS CODE Managing System Complexity with Chef Robert J. Berger - CTO Runa, Inc. rberger@runa.com http:/ /blog.ibd.com Friday, April 22, 2011 1 Image: http://techcrunch.com/2010/03/16/big-data-freedom/
  • 2. MOORE’S LAW Friday, April 22, 2011 2 Graph: http://www.gotw.ca/publications/concurrency-ddj.htm
  • 3. MOORE’S LAW HITS A SPEED BUMP CPU speed growth has STOPPED Friday, April 22, 2011 2 Graph: http://www.gotw.ca/publications/concurrency-ddj.htm
  • 4. BUT WE GET: GROWTH OF CORES & FOOTBALL FIELDS OF SERVERS Friday, April 22, 2011 3 Data Center Image: http://www.datacenterknowledge.com/inside-microsofts-dublin-mega-data-center/ Chart: http://news.cnet.com/8301-13924_3-10101987-64.html
  • 5. BUT WE GET: GROWTH OF CORES & FOOTBALL FIELDS OF SERVERS Friday, April 22, 2011 3 Data Center Image: http://www.datacenterknowledge.com/inside-microsofts-dublin-mega-data-center/ Chart: http://news.cnet.com/8301-13924_3-10101987-64.html
  • 6. VON NEUMANN SEQUENTIAL TRANSFORMS TO... Friday, April 22, 2011 4 Image: http://www.necessarysufficient.com
  • 7. ACTORS, Friday, April 22, 2011 5 Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.html Map Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.html Nanites: http://www.startrek.com/database_article/nanites Runa Infrastructure Diagram: Robert J. Berger
  • 8. Hadoop / HBase Cheshire / Swarmiji Map / Reduce Petabyte Store Dynamic Monitor & HTTP Runtime Recovery Dispatchers HBase HBase HBase HBase HBase HBase HBase HBase HBase Redis HBase HBase HBase HBase HBase HBase Redis Mem HBase HBase HBase Redis Mem CacheRedis Queue HBase HBase HBase Mem CacheRedis HBase HBase HBase HBase HBase HBase Mem Cache HBase HBase HBase Mem Cache HBase HBase HBase Cache HBase HBase HBase Shared Reporting Analytics Data Session Collectors Memory ACTORS, MAP/REDUCE, NANITE... PARALLELISM! Friday, April 22, 2011 5 Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.html Map Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.html Nanites: http://www.startrek.com/database_article/nanites Runa Infrastructure Diagram: Robert J. Berger
  • 9. COMPLEXITY EXPLOSION Friday, April 22, 2011 6 Complexity Background: http://www.c0d3m0nk3y.com/gallery/
  • 10. COMPLEXITY EXPLOSION Lots of moving heterogeneous parts Not just in the LAMP stack Auto scaling Horizontal scaling Clouds Monitoring & Management Friday, April 22, 2011 6 Complexity Background: http://www.c0d3m0nk3y.com/gallery/
  • 11. CODE YOUR INFRASTRUCTURE Friday, April 22, 2011 7 Background: http://www.flickr.com/photos/jason-samfield/4398151919/
  • 12. CODE YOUR INFRASTRUCTURE Programming is all about managing complexity Apply same techniques to complexity of infrastructure Revision control Keep it DRY Build on the shoulders of others Friday, April 22, 2011 7 Background: http://www.flickr.com/photos/jason-samfield/4398151919/
  • 13. CHEF: THE RUBY WAY TO MANAGE INFRASTRUCTURE Plain Old Ruby programming External DSL adds language constructs for service deployment Services to support Configuration Management Community Cookbooks Friday, April 22, 2011 8 Background Image: http://www.flickr.com/photos/steveritchie/3779779586/
  • 14. CHEF: THE RUBY WAY TO MANAGE INFRASTRUCTURE Plain Old Ruby programming External DSL adds language constructs for service deployment Services to support Configuration Management Community Cookbooks ENABLES YOU TO CODE YOUR INFRASTRUCTURE Friday, April 22, 2011 8 Background Image: http://www.flickr.com/photos/steveritchie/3779779586/
  • 15. WHAT’S IN THE KITCHEN Chef Server Authoritative Chef-Client Chef-Client Chef-Client Data bags Roles Cookbooks Node states Remote Deployed Infrastructure Repo Local Project Repo of Cookbooks Do your cooking (editing) Chef Knife client Push Data Bags, Roles, Cookbooks PowerBook G4 Launch/Bootstrap instances Your Dev Environment Friday, April 22, 2011 9
  • 16. CHEF SERVER • Manage & Authenticate • Support search on Data clients (nodes, dev, Bags, Roles and Node admin) states. • Assign roles & • Serve a RESTful API for Configurations to nodes humans and machines • Track states of Nodes • Can run your own or use Opscode Platform Friday, April 22, 2011 10
  • 17. PROJECT REPO Data Bags Cookbooks -App Level -"Standards" from Attributes cookbooks.Opscode.com Roles Site-Cookbooks -A Nodes Purpose -Your own cookbooks Environments -Over-rides of -Staging, Production PowerBook G4 Standard Cookbooks -New in 0.10.x Friday, April 22, 2011 11
  • 18. COOKBOOKS Attributes Recipes -Key-Value pairs -Where the Action is associated with a node -Specifies resources -Hierarchy of overrides that the chef client Files should take action on -Static Files to be Templates copied to node -ERB files to build Library config files on node -Place to DRY up Ruby Providers / Resources code used in Recipes -DIY DSL Extensions Friday, April 22, 2011 12 You only need Recipes. Most cookbooks are just Recipe[s] and Templates Background: Clipart Library
  • 19. PLETHORA OF EXISTING COOKBOOKS Friday, April 22, 2011 13 These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one
  • 20. PLETHORA OF EXISTING COOKBOOKS activemq erlang maven pxe_dust stompserver ant fail2ban memcached python subversion apache2 gems mercurial quick_start sudo apparmor git munin rabbitmq teamspeak application glassfish mysql rabbitmq_chef teamspeak3 apt gnu_parallel nagios radiant thrift aws god nanite rails tmux bluepill gunicorn nginx rails_enterprise tomcat boost hadoop nscd redmine tomcat6 build-essential haproxy ntp reprepro trac capistrano heartbeat ohai resolver transmission chef imagemagick one-shot riak ubuntu chef-client instiki openldap rsync ucspi-tcp cloudkick iptables openssh rsyslog unicorn couchdb java openssl ruby users cron java_sun openvpn ruby_enterprise varnish daemontools jetty ossec rubygems vim database jira packages runit wordpress django jpackage pacman rush xfs djbdns keepalived passenger_apache2 samba xml dmg kickstart passenger_enterprise sbuild zenoss drbd logrotate pdns screen zlib dynect logwatch perl snort zsh dynomite lvm php solr ec2 man postfix sqlite emacs maradns postgresql ssh_known_hosts Friday, April 22, 2011 13 These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one
  • 21. MOST IMPORTANT WIKI PAGE WIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES Friday, April 22, 2011 14 Whole range of Directives. Everything from copying files from file systems, URLs, etc to deploy from Git/SVN... Similar to Capistrano. But its pull instead of push
  • 22. MOST IMPORTANT WIKI PAGE WIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES Chef DSL Directives Friday, April 22, 2011 14 Whole range of Directives. Everything from copying files from file systems, URLs, etc to deploy from Git/SVN... Similar to Capistrano. But its pull instead of push
  • 23. A SIMPLE COOKBOOK: RABBITMQ Friday, April 22, 2011 15
  • 24. RECIPES: default.rb package "rabbitmq-server" do action :install end service "rabbitmq-server" do supports :status => true, :restart => true, :reload => true action [ :enable, :start ] end Friday, April 22, 2011 16
  • 25. RECIPES: default.rb Install Packages (Knows about many package formats) package "rabbitmq-server" do action :install end service "rabbitmq-server" do supports :status => true, :restart => true, :reload => true action [ :enable, :start ] end Friday, April 22, 2011 16
  • 26. RECIPES: default.rb package "rabbitmq-server" do action :install end service "rabbitmq-server" do supports :status => true, :restart => true, :reload => true action [ :enable, :start ] end Defines a service that other recipes can trigger (Knows about many init style mechanism) Friday, April 22, 2011 16
  • 27. RECIPES: default.rb pt 2 case node[:platform] when "ubuntu" template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end else template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end end Friday, April 22, 2011 17 ERB: Embedded Ruby
  • 28. RECIPES: default.rb pt 2 Tweak based on target OS (Its just Ruby!) case node[:platform] when "ubuntu" template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end else template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end end Friday, April 22, 2011 17 ERB: Embedded Ruby
  • 29. RECIPES: default.rb pt 2 Chefisms tend to be blocks with case node[:platform] when "ubuntu" optional parameters template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end else template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end end Friday, April 22, 2011 17 ERB: Embedded Ruby
  • 30. RECIPES: default.rb pt 2 case node[:platform] when "ubuntu" template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" Creates files using ERB group "root" mode 0644 (ERB Params could be set here) notifies :restart, resources(:service => "rabbitmq-server") end else template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end end Friday, April 22, 2011 17 ERB: Embedded Ruby
  • 31. RECIPES: default.rb pt 2 case node[:platform] when "ubuntu" template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end else Using the service defined template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" earlier owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") end end Friday, April 22, 2011 17 ERB: Embedded Ruby
  • 32. ATTRIBUTES: default.rb default[:rabbitmq][:nodename] = "rabbit" default[:rabbitmq][:address] = "0.0.0.0" default[:rabbitmq][:port] = "5672" default[:rabbitmq][:erl_args] = "+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}]" default[:rabbitmq][:start_args] = "" default[:rabbitmq][:logdir] = "/var/log/rabbitmq" default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia" default[:rabbitmq][:cluster] = "no" default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config" default[:rabbitmq][:cluster_disk_nodes] = [] Friday, April 22, 2011 18
  • 33. ATTRIBUTES: default.rb Sets the default value default[:rabbitmq][:nodename] = "rabbit" default[:rabbitmq][:address] = "0.0.0.0" default[:rabbitmq][:port] = "5672" default[:rabbitmq][:erl_args] = "+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}]" default[:rabbitmq][:start_args] = "" default[:rabbitmq][:logdir] = "/var/log/rabbitmq" default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia" default[:rabbitmq][:cluster] = "no" default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config" default[:rabbitmq][:cluster_disk_nodes] = [] Friday, April 22, 2011 18
  • 34. ATTRIBUTES: default.rb Precedence -default -set default[:rabbitmq][:nodename] = "rabbit" -override default[:rabbitmq][:address] = "0.0.0.0" -File type default[:rabbitmq][:port] = "5672" default[:rabbitmq][:erl_args] = "+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}]" default[:rabbitmq][:start_args] = "" default[:rabbitmq][:logdir] = "/var/log/rabbitmq" default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia" default[:rabbitmq][:cluster] = "no" default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config" default[:rabbitmq][:cluster_disk_nodes] = [] Friday, April 22, 2011 18
  • 35. ATTRIBUTES: default.rb Automatic Ohai discovers OS & HW instance attributes at runtime default[:rabbitmq][:nodename] = "rabbit" default[:rabbitmq][:address] = "0.0.0.0" default[:rabbitmq][:port] = "5672" default[:rabbitmq][:erl_args] = "+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}]" default[:rabbitmq][:start_args] = "" default[:rabbitmq][:logdir] = "/var/log/rabbitmq" default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia" default[:rabbitmq][:cluster] = "no" default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config" default[:rabbitmq][:cluster_disk_nodes] = [] Friday, April 22, 2011 18
  • 36. Templates: rabbitmq.config.erb NODENAME=<%= node[:rabbitmq][:nodename] %> NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %> NODE_PORT=<%= node[:rabbitmq][:port] %> SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %> CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %> LOG_BASE=<%= node[:rabbitmq][:logdir] %> MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %> SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %> Friday, April 22, 2011 19
  • 37. Templates: rabbitmq.config.erb Creates configuration files using attributes NODENAME=<%= node[:rabbitmq][:nodename] %> NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %> NODE_PORT=<%= node[:rabbitmq][:port] %> SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %> CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %> LOG_BASE=<%= node[:rabbitmq][:logdir] %> MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %> SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %> Friday, April 22, 2011 19
  • 38. Templates: rabbitmq.config.erb node object created when chef-client runs (attribute precedence resolved) NODENAME=<%= node[:rabbitmq][:nodename] %> NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %> NODE_PORT=<%= node[:rabbitmq][:port] %> SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %> CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %> LOG_BASE=<%= node[:rabbitmq][:logdir] %> MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %> SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %> Friday, April 22, 2011 19
  • 39. Templates: rabbitmq.config.erb Final File is rendered and copied to location specified by the Template Directive NODENAME=rabbit NODE_IP_ADDRESS=0.0.0.0 NODE_PORT=5672 SERVER_ERL_ARGS=+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}] CLUSTER_CONFIG_FILE=/etc/rabbitmq/rabbitmq_cluster.config LOG_BASE=/var/log/rabbitmq MNESIA_BASE=/var/lib/rabbitmq/mnesia SERVER_START_ARGS= Friday, April 22, 2011 19
  • 40. ROLE: rabbitmq.rb name "rabbitmq" description "Deploy rabbitmq instance" recipes "ulimits", "rabbitmq::default" override_attributes({ :rabbitmq => { :port => "8888" }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ] }) Friday, April 22, 2011 20
  • 41. ROLE: rabbitmq.rb name "rabbitmq" Specify recipes needed description "Deploy rabbitmq instance" recipes "ulimits", "rabbitmq::default" override_attributes({ :rabbitmq => { :port => "8888" }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ] }) Friday, April 22, 2011 20
  • 42. ROLE: rabbitmq.rb name "rabbitmq" description "Deploy rabbitmq instance" recipes "ulimits", "rabbitmq::default" override_attributes({ :rabbitmq => { :port => "8888" Attribute Overrides }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ] }) Friday, April 22, 2011 20
  • 43. A TASTE OF A SPICIER COOKBOOK: APPLICATION/ DATABASE Friday, April 22, 2011 21 Just touch on the most powerful Cookbook pair Can use the same scafolding for many applications Image: http://www.theepochtimes.com/n2/content/view/46924/
  • 44. DATA BAG DRIVEN: MAP RECIPES TO ROLES IN BAG { "id": "my_app", "server_roles": [ "my_app" ], "type": { "my_app": [ "rails", "postfix", "unicorn" ] }, ... Friday, April 22, 2011 22 Snippet of a JSON representation of a Data Bag
  • 45. DATA BAG DRIVEN: MAP RECIPES TO ROLES IN BAG { "id": "my_app", Application specific role(s), "server_roles": [ typically the name of the app "my_app" ], "type": { "my_app": [ "rails", "postfix", "unicorn" ] }, ... Friday, April 22, 2011 22 Snippet of a JSON representation of a Data Bag
  • 46. DATA BAG DRIVEN: MAP RECIPES TO ROLES IN BAG { "id": "my_app", "server_roles": [ "my_app" ], "type": { "my_app": [ "rails", Recipes in application "postfix", cookbook to run for this role "unicorn" ] }, ... Friday, April 22, 2011 22 Snippet of a JSON representation of a Data Bag
  • 47. DEFAULT RECIPE: search(:apps) do |app| (app["server_roles"] & node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end end end node.run_state.delete(:current_app) Friday, April 22, 2011 23 This happens when chef-client runs on an instance and node has all resolved attributes Node may have one role or many roles. For instance having the web server, rails app and db roles on one server.
  • 48. DEFAULT RECIPE: ITS KIND OF META search(:apps) do |app| (app["server_roles"] & node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end end end node.run_state.delete(:current_app) Friday, April 22, 2011 23 This happens when chef-client runs on an instance and node has all resolved attributes Node may have one role or many roles. For instance having the web server, rails app and db roles on one server.
  • 49. DEFAULT RECIPE: EXECUTES ON EACH NODE BY CHEF-CLIENT Search thru all ‘apps’ databag search(:apps) do |app| (app["server_roles"] & node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end end end node.run_state.delete(:current_app) Friday, April 22, 2011 23 This happens when chef-client runs on an instance and node has all resolved attributes Node may have one role or many roles. For instance having the web server, rails app and db roles on one server.
  • 50. DEFAULT RECIPE: EXECUTES ON EACH NODE BY CHEF-CLIENT Loop thru roles that search(:apps) do |app| (app["server_roles"] & match running node’s run_list node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end end end node.run_state.delete(:current_app) Friday, April 22, 2011 23 This happens when chef-client runs on an instance and node has all resolved attributes Node may have one role or many roles. For instance having the web server, rails app and db roles on one server.
  • 51. DEFAULT RECIPE: EXECUTES ON EACH NODE BY CHEF-CLIENT search(:apps) do |app| (app["server_roles"] & node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end end Loop thru & run recipes end for each role nginx node.run_state.delete(:current_app) unicorn rails postfix Friday, April 22, 2011 23 This happens when chef-client runs on an instance and node has all resolved attributes Node may have one role or many roles. For instance having the web server, rails app and db roles on one server.
  • 52. CHEF-CLIENT (RUNTIME) Friday, April 22, 2011 24
  • 53. CHEF-CLIENT (RUNTIME) • All work done by client on instances (not server) • Runs on each instance regularly (every 1/2 by default) • Roles, Cookbook, Data Bag, Attribute changes propagate from Chef Server to instances via the chef-client • Client sends attributes to Chef Server making them searchable Friday, April 22, 2011 24
  • 54. THE AMAZING KNIFE CLI for Chef Server PowerBook G4 Friday, April 22, 2011 25
  • 55. THE AMAZING KNIFE CLI for Chef Server Manage Cookbooks Manage Data Bags -Create Node Mgmt & Status -Upload Role Management -Syntax Check Client Management -Generate Metadata Search -Access Public -Nodes, Roles, Data Cookbooks Bags -Git Vendor Pulls PowerBook G4 -Publish Friday, April 22, 2011 25
  • 57. AND THAT’S NOT ALL! Manage Knife Config Create, List, Delete Cloud Multi-ssh Instances -Select by Search - EC2 Bootstrap Servers - Rackspace -From Bare Basic OS - Slicehost - Terremark PowerBook G4 Friday, April 22, 2011 26
  • 58. PUSH YOUR COOKBOOKS TO THE CHEF SERVER Friday, April 22, 2011 27
  • 59. PUSH YOUR COOKBOOKS TO THE CHEF SERVER knife cookbook upload -a Pushes all cookbooks in your repo to chef server Friday, April 22, 2011 27
  • 60. PUSH YOUR ROLES TO THE CHEF SERVER Friday, April 22, 2011 28
  • 61. PUSH YOUR ROLES TO THE CHEF SERVER knife role from file roles/rabbitmq.rb Pushes the rabbitmq role to the chef server Friday, April 22, 2011 28
  • 62. LAUNCH RABBITMQ ON EC2 knife ec2 server create 'role[production]' 'role[base]' role[rabbitmq]' -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu -G production,rabbitmq -i ami-88f504e1 -f m1.large Friday, April 22, 2011 29
  • 63. LAUNCH RABBITMQ ON EC2 Uses Fog to issue EC2 api calls to configure and create an instance with these parameters knife ec2 server create 'role[production]' 'role[base]' role[rabbitmq]' -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu -G production,rabbitmq -i ami-88f504e1 -f m1.large Friday, April 22, 2011 29
  • 64. LAUNCH RABBITMQ ON EC2 “Run List” specifies roles for this instance knife ec2 server create 'role[production]' 'role[base]' role[rabbitmq]' -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu -G production,rabbitmq -i ami-88f504e1 -f m1.large Friday, April 22, 2011 30
  • 65. LAUNCH RABBITMQ ON EC2 “Run List” specifies roles for this instance knife ec2 server create 'role[production]' 'role[base]' role[rabbitmq]' -S my-aws -I ~/.ssh/my-aws.pem -x ubuntu -G production,rabbitmq -i ami-88f504e1 -f m1.large Works with other Clouds: Rackspace, Terramark, Slicehost, Bluebox Friday, April 22, 2011 30
  • 66. LIKE BEING AT BENIHANA’S Friday, April 22, 2011 31 Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/
  • 67. LIKE BEING AT BENIHANA’S Launches ec2 instance[s] with AMIs you specified Installs Ruby & packages for building RubyGems Installs RubyGems from source Installs Chef from RubyGems Creates the /etc/chef directory Writes validation cert to /etc/chef/validation.pem Writes an /etc/chef/client.rb config file Writes a JSON file, /etc/chef/first-boot.json with roles and recipes specified as the run list Executes chef-client with the first-boot.json, connecting to the chef server in the client.rb Friday, April 22, 2011 31 Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/
  • 68. KEEP YOUR INFRASTRUCTURE UNDER CONTROL WITH CHEF! Friday, April 22, 2011 32 Image: http://forums.brickarms.com/yaf_postst18328_The-Evil-Chefs-Are-Back-and-With-Even-More-Weapons.aspx