This document discusses Chef Cookbook workflow testing. It begins with an introduction to Chef and its core components like resources, recipes, and cookbooks. It then emphasizes the importance of testing infrastructure code like Chef cookbooks. Various testing techniques for Chef cookbooks are presented, including linting with Rubocop, style checking with FoodCritic, unit testing with ChefSpec, and integration testing using Test Kitchen. The document stresses treating infrastructure code like any other codebase by implementing practices like version control, continuous integration, and separation of concerns. It provides examples of implementing some of these testing techniques and outlines an example pipeline for testing and releasing Chef cookbooks.

1. Chef Cookbook Workflow
Testing your infrastructure code
Alex Manly - @apmanly

2. Instructor
• Alex Manly
• Solutions Architect, Chef
• Developer (Java, oh and some Ruby)
• @apmanly
• am@chef.io

3. What is Chef?
Short Answer: An Automation Framework
for automating infrastructure and applications

4. Traditional Systems Management
Config 1
Config 2
Config 3

5. Traditional Systems Management
Config 1
Config 2
Config 3
Automation

6. Idea of Services
Service
Config 1
Config 2
Config 3
Automation

7. Abstraction of Services
Service
Config 1
Config 2
Config 3
Automation

8. Chef is Infrastructure as Code
•Programmatically
provision and configure
components
http://www.flickr.com/photos/louisb/4555295187/

9. Chef is Infrastructure as Code
•Treat like any other
code base
http://www.flickr.com/photos/louisb/4555295187/

10. Chef is Infrastructure as Code
•Reconstruct business
from code repository,
data backup, and
compute resources
http://www.flickr.com/photos/louisb/4555295187/

11. Chef is Infrastructure as Code
•Programmatically provision
and configure components
•Treat like any other code
base
•Reconstruct business from
code repository, data
backup, and compute
resources
http://www.flickr.com/photos/louisb/4555295187/

13. Building Blocks: What is a Resource?
• A Resource is a system state you define
Example: Package installed, state of a service, configuration file existing
• You declare what state you want the resource in.
Chef automatically determines HOW that state is achieved
On Linux based OSes: On Windows based OSes:

14. Resource Example
#windows
dsc_resource 'webserver' do
resource :windowsfeature
property :name, 'Web-Server'
property :ensure, 'Present’
end

15. Resource Example
#linux
package "httpd" do
action :install
end

16. Building Blocks: What is a Recipe?
• An abstraction of a Service that consists of a set of
Resources to deliver that Service
• Resources are executed in the order they are listed.

17. Recipe Example
#linux
package "httpd" do
action :install
end
include_recipe "apache::fwrules”
service "httpd" do
action [ :enable, :start ]
end

18. Recipe Example
#windows
include_recipe "fourthcoffee::dsc”
include_recipe "iis::remove_default_site”
remote_directory node['fourthcoffee']['install_path'] do
source 'fourthcoffee’
action :create
end
iis_pool 'FourthCoffee' do
runtime_version "4.0"
action :add
end
iis_site 'FourthCoffee' do
protocol :http
port 80
path node['fourthcoffee']['install_path']
application_pool 'FourthCoffee'
action [:add,:start]
end

19. Cookbooks
• A Higher Level Abstraction of a Service
• A set of Recipes and Data Attributes
required to deliver one or multiple
Services

20. Define cookbook attribute
#attributes.rb
default['fourthcoffee']['port'] = 80

21. Consume cookbook attribute
iis_site 'FourthCoffee' do
protocol :http
port node['fourthcoffee']['port']
path node['fourthcoffee']['install_path']
application_pool 'FourthCoffee'
action [:add,:start]
end

22. Yes!

23. That’s cool but…
• Things break
• Chef is a language (based on Ruby)
• How can you rapidly develop recipes and cookbooks?

24. Let’s step back…

25. Testing is Hard - FACT
Manual Tests

26. Also known as…
You’ll never get to
Continuous Deployment
clicking a GUI

27. Theory of testing…
Testing builds safety

28. Feedback loops…
• Tell us we’re doing the right thing
• At the right time
• With the right results

29. Feedback loops…
Measurements we take to ensure the
“experiment” is behaving as expected

30. Tests are essentially feedback loops

31. Remember…
Chef is “Infrastructure as Code”

32. Remember…
“Infrastructure as Code” should be
treated like ANY other codebase.

33. Treated Any Other Codebase…
• Stored in SCM
• Testing Coverage
• Part of your CI pipelines

34. Testing in Chef
• Chef recipes need tested
Linting
Static Analysis
Unit Testing
Functional Testing

35. DevOps is a Two-Way Street
• It’s great when
developers care about
• Uptime!
• Scaling!
• Deployment!
• Put them on call! etc.
etc. etc.

36. DevOps is a Two-Way Street
• Operations also has as much or more
to learn from developers as well

37. Software Development Workflow
• Write some code
• <ad-hoc verification here>
• Go to pre-production
• <ad-hoc verification here>
• Go to production
• Production failure

38. Software Development Workflow
• Write some code
• Write and run some unit tests
• Go to pre-production
• Run some
integration/acceptance tests
• Go to production
• Lowered chance of
production failure

39. Old Chef Cookbook Workflow
• Write some cookbook code
• <ad-hoc verification here>
• Go to pre-production
• <ad-hoc verification here
• Go to production
• Whoops, broke production

40. Chef Testing
• Did chef-client complete successfully?
• Did the recipe put the node in the desired state?
• Are the resources properly defined?
• Does the code following our style guide?

41. New Chef Cookbook Workflow
• Write some cookbook code
• Check for code correctness
• Write and run some unit tests
• Go to pre-production
• Run some integration tests
• Go to production

42. ChefDK: TDI In a Box
Code Correctness
Unit Tests
Deploy sample environments
A wrapper to tie it all together
Not just syntactic correctness but quality gates too!

43. Rubocop
Because Ruby
• Identify potential Ruby errors
• Unclosed strings, etc.
• Identify style/convention that helps write better code
• Single quotes vs. double quotes
• This is useful for new Chefs, and helps make the code more readable
• Exclude rules that do not apply
• Not everyone is the same – some tests won’t work for your organization or for Chef code
• Run against static code, so tests are very fast (<5 seconds to run)
Code Correctness

44. Rubocop Example
def badName
if something
test
end
end
test.rb:1:5: C: Use snake_case for methods and variables.
def badName
^^^^^^^
test.rb:2:3: C: Favor modifier if/unless usage when you have
a single-line body. Another good alternative is the usage of
control flow &&/||.
if something
^^
test.rb:4:5: W: end at 4, 4 is not aligned with if at 2, 2
end
^^^
1 file inspected, 3 offenses detected
Code Correctness

45. Food Critic
Test Your “Chef Style”
• Flag problems that might cause your Chef cookbooks to fail
on converge
FC010: Invalid search syntax
• Identify style/convention that has been adopted by the Chef
community
FC004: Use a service resource to start and stop
services
• Create custom rules for your own organization’s
compliance/standards
COMP001: Do not allow recipes to mount disk volumes
• Run against static code, so tests are very fast (<5 seconds to
run)
Code Correctness

46. FoodCritic Examples
• FC001: Use strings in preference to symbols to access node attributes
• FC002: Avoid string interpolation where not required
• FC003: Check whether you are running with chef server before using server-
specific features
• FC004: Use a service resource to start and stop services
• FC005: Avoid repetition of resource declarations
• FC006: Mode should be quoted or fully specified when setting file
permissions
• FC007: Ensure recipe dependencies are reflected in cookbook metadata
Code Correctness

47. ChefSpec
Simulate Chef
• Did I send a properly formed piece of code to Chef?
• Especially important if there is mutability in your code
• This can cover things like feature flags, or behavior that changes on a
per-OS basis
• "If on Debian-based Linux I need to install package apache2, but on EL
variants I need to install package httpd.”
• Tests are very fast to run
• Even with simple code, it is useful for new Chefs to find simple errors
quickly
• Useful for regression testing – to make sure new changes
don’t break stuff that worked before.
Unit Tests

48. ChefSpec Example
require 'chefspec'
describe 'example::default' do
let(:chef_run) { ChefSpec::SoloRunner.converge(described_recipe) }
let(:cheftest) { chef_run.node['cheftest'] }
it 'installs foo' do
expect(chef_run).to install_package('foo')
end
it 'sets the default attributes correctly' do
expect(cheftest['package_name']).to eq('httpd')
expect(cheftest['service_name']).to eq('httpd')
expect(cheftest['web_root']).to eq('/var/www/html')
end
end
Unit Tests

49. Test Kitchen
Let’s do this (almost) for real
• “Signal Out” testing for Chef code
• Just because I said something and it was interpreted correctly, doesn't mean that
what I said is what I meant.
• Executes your Chef code on an actual, factual node
• These nodes should be disposable (local virtualization, cloud
instances, etc.)
• Use any number of “signal out” testing products to ensure expected
results
• BATS
• ServerSpec
• Pester
• Can pull in other cookbook dependencies as well, and execute against
a machine that looks the same as your standard image
Deploy sample environments

50. Test Kitchen
it 'contains the default configuration settings' do
file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^chef_server_url')
file(File.join(node['chef_client']['conf_dir'], 'client.rb')).must_match('^validation_client_name')
end
it 'should configure the server with a started webserver' do
expect(service('httpd')).to be_running
end
it 'should configure the server with an enabled webserver' do
expect(service('httpd')).to be_enabled
end
it 'should configure the server to listen on TCP port 80' do
expect(port(80)).to be_listening
end
Deploy sample environments

51. Testing for Compliance

52. Chef Analytics Ensures Compliance
Separation of concerns is a thing
• If my tests are failing, I can just re-write the tests, right?
• By using the Audit Mode of Chef Analytics in your pipeline, cookbooks have to pass
your custom compliance rules in order to be published.
• These are the same compliance rules that run against existing nodes, so you don’t
need to write separate tests.
• Compliance rules in Chef Analytics can (and should) be stored in a
separate SCM repository than the code itself
• Those who write the code, shouldn’t be able to write the rule check

53. Process and Workflow Enforces Standards
• Chef enforces appropriate checks are being made and standards are being
enforced using FoodCritic.
• This example will identify any code that tries to mount disk volumes. If code is
identified, it will be audited and then workflow can control the action of this
deviation to standards.

54. Just because it’s published, doesn’t mean it’s used
• Chef Environments should be constrained to specific cookbook versions
• If my Production environment is constrained to version 1.0.1 of the “MyApp” cookbook, it will
only use that version, even if a newer version is published to the Chef server
• Incrementing the cookbook constraint can be done using your existing change
control mechanisms and separation of duties
• A well-constructed pipeline will never overwrite an existing version of a cookbook
with new code/policy
• BEST PRACTICE – have your pipeline auto-increment versions of cookbooks,
tied to build numbers, to have traceability all the way back to your SCM

55. Chef Workflow with Jenkins

56. Infrastructure Change Control Pipeline

57. Local Development
Local Workstation
master
Clone Push
Commit
<USER>/<COOKBOOK>:master
DevOps/<COOKBOOK>:master
Fork

58. Pipeline Shape
We want to ensure that our changes
are relevant and well tested before we
deploy.
Verify Review Accept Ship

59. Verify Phase
During the verify phase, we want to do basic testing
to build confidence that our changes are sane and at
the very least, won’t break the build process
Verify Review Accept Ship

60. Jenkins
master
<USER>/<COOKBOOK>:master
DevOps/<COOKBOOK>:master
Pull Request
Fail
Fail
Pass
Pass
Pass
Test
Fail
Verify Phase

61. Review Phase
Two heads are better than one! Ask
your team members to double check
your work.
Verify Review Accept Ship

62. Accept Phase
We need to test this in a production-like environment
before we ship it.
Verify Review Accept Ship

63. Accept Job
Jenkins
masterDevOps/<COOKBOOK>:master
Fail
Pass
Pass
Test
Fail
Checkout
Push

64. Ship Phase
The very last thing we need to do is
start using the cookbook in our
environments.
Verify Review Accept Ship

65. CHEF DELIVERY

66. VALIDATED IN OUR ENGAGEMENTS WITH
ENTERPRISE AND BIG WEB CUSTOMERS.
WE'VE IDENTIFIED A PROVEN
PIPELINE

67. Steps
manual automated
Workstation
Create New
Change1
Test Change
Locally2
Chef Delivery
Approve
Change5
Deliver
Change6
Submit
Change3
Review
Change4
Stable Pipeline

68. Steps
manual automated
Verify
Lint
Syntax
Unit
Build
Merge
Lint
Syntax
Unit
Quality
Security
Publish
Provision
Deploy
Smoke
Functional
Acceptance Union
Provision
Deploy
Smoke
Functional
Rehearsal
Provision
Deploy
Smoke
Functional
Delivered
Provision
Deploy
Smoke
Functional
Stages
customizable
Verify Build
Acceptance
Union
Rehearsal
Delivered
Submit
Change3
Review
Change4
Approve
Change5
Deliver
Change6
Chef Delivery
Create a new
change1
Test Change
Locally2
Workstation
Common Pipeline

69. BUILD COOKBOOK
├── recipes
├── default.rb
├── lint.rb
├── syntax.rb
├── unit.rb
├── quality.rb
├── security.rb
├── publish.rb
├── provision.rb
├── deploy.rb
├── smoke.rb
└── functional.rb

70. PHASE EXEC
log "Running unit"
repo = node['delivery_builder']['repo']
delivery_builder_exec “run my junit tests" do
command "mvn test"
cwd repo
end

71. MANY ACCEPTANCE PIPELINES
ONE DELIVERY PIPELINE

72. ONE PIPELINE
One
Pipeline
Delivery Pipeline
union rehearsal delivered
Acceptance Pipelines
review approve deliverChange
Cookbook [A]
review approve deliverChange
Cookbook [B]
review approve deliverChange
Application [A]
review approve deliverChange
Application [B]
Infrastructure & Applications

73. Cookbook Workflow
Supermarket
Chef Server
review approve deliverChange
Cookbook
Node Node Node
Node Node Node
Node Node Node

74. Application Workflow
review approve deliverChange
Application
Node Node Node
Node Node Node
Node Node Node
Deploy
1 2 3
2 2 3
3 3 3

75. Chef Community Summit – London
London, etc. Venues Monument – November 3rd & 4th
Why your participation matters
• Influence the path of the Chef roadmap
• Contribute to the formation of best practices and the avenues to best share them
• Share your experiences transforming your business
• Demonstrate your DevOps Kung Fu
Network with awesome engineers in the Community
• Engage with a community of people actively using Chef to automate their workflow
• Discuss “what keeps you up at night” with a passionate engaged audience
• Meet with CHEF engineers IRL
**Use the code MEETUP and save 20%
http://summit.chef.io

76. What Questions Can I Answer For
You?
Thank you!
@apmanly