Slides from my "Chef at Etsy" talk at the London Chef Meetup on Thurs Oct 10th, 2014

1. Chef at Etsy

2. @jonlives
Jon Cowie
Sr Operations Engineer

4. 30Million Members
4
1Million Active Shops

5. 20Million Items Listed
5
60Million Monthly Unique Visitors

6. @jonlives
We Love Chef!

7. @jonlives
Absorb what is
useful.
Discard what is
useless.

8. @jonlives
“I am not smart enough to
build an ontology … that
can encompass all the
variations in infrastructure.
Nobody is, the world
moves too fast.”

9. @jonlives
There is no
magic pill.

10. @jonlives
You are the
expert.

11. @jonlives
Chef at Etsy
• Chef Server 11.1.4
• ~2000 Nodes
• CentOS, some Mac OS X

12. @jonlives
Beginning of 2010 Today

13. @jonlives
Chef at Etsy

14. @jonlives
Evolution of Chef

15. @jonlives
2010: The Beginning
• ~250 Nodes (Ubuntu & CentOS
• The first cookbooks
• Out of the box workflow

16. @jonlives
2011: Growth
• ~400 Nodes (CentOS)
• Chef still pretty specialised knowledge
• Handlers added

17. @jonlives
2012: A big year
• ~800 Nodes (CentOS & MacOS X)
• More in-house Chef expertise
• Workflow tooling
• Debugging tooling
• Monitoring

18. @jonlives
2013: Chef at Etsy
• ~1500 Nodes
• Workflow tooling enhancements
• Feature flags in Chef
• Chef performance - Chef 11 upgrade

19. @jonlives
2014: Chef at Etsy
• ~2000 nodes
• Consolidation
• CI with Chef
• Omnibus
• Work-in-Progress tooling

20. @jonlives
Patterns & Workflows

21. @jonlives
Cookbook Workflow

22. @jonlives
$> review -r jcowie --cc ops

23. @jonlives
knife-spork
• https://github.com/jonlives/knife-spork
• Workflow tool
• Helps multiple chefs avoid clashing
• Visibility into changes
• Plugins

24. @jonlives
knife-spork
• knife spork bump
• knife spork upload
• Test change

25. @jonlives
Test Change
• https://github.com/jonlives/knife-flip
• knife node flip foo.etsy.com testing
• knife role flip MyRole testing

26. @jonlives
Test Change
• https://github.com/mrtazz/knife-wip
• Uses node tags
<irccat> CHEF: bburry started work
cent7 package bugfixing on
deploy01.ny5.etsy.com

27. @jonlives
knife-spork
• knife spork bump
• knife spork upload
• Test change
• knife spork promote --remote
• git commit and push

28. @jonlives
Monitoring & Debugging

29. @jonlives
knife-spork & CI Job
<irccat> CHEF: Jon Cowie uploaded pentaho@0.1.8
<irccat> CHEF: Jon Cowie promoted pentaho@0.1.8 to
production <snip>
<irccat> Git PUSH -> Sysops/chef <snip>
<Jenkins> Starting build #5649 for job chef-
server-git-sync
<Jenkins> Project chef-server-git-sync build
#5649: SUCCESS in 2 min 36 sec: http://
ci.etsycorp.com/job/chef-server-git-sync/5649/

30. @jonlives
IRC Handler
<irccat> Chef run failed on
officebackup01.office.etsy.com gist
failed, see /var/log/chef/client.log on
the host
!
<irccat> Still Failing on
dbnest01.ny4.etsy.com since 2 days ago
https://github.etsycorp.com/gist/
656d8914fbef5a6bd9aa

31. @jonlives
Lastrun Data
• https://github.com/jgoulah/knife-lastrun
• knife node lastrun foo.bar.com

32. @jonlives
Lastrun Data
%
knife
node
lastrun
dbnest01.ny4.etsy.com
Status
failed
Elapsed
Time
29.055892
Start
Time
2014-­‐10-­‐06
12:54:51
+0000
End
Time
2014-­‐10-­‐06
12:55:20
+0000
!
<snip>
!
Exception
<snip>
Installed
package
backupd-­‐1.4-­‐1.365657d.el5.centos
is
newer
than
candidate
package
backupd-­‐1.2-­‐1.99ddb8e.el5

33. @jonlives
Dashboards

34. @jonlives
Dashboards

35. @jonlives
Dashboards

36. @jonlives
Monitoring & Debugging
• https://github.com/etsy/chef-handlers
• https://github.com/etsy/dashboard
• https://github.com/jgoulah/knife-lastrun
• https://github.com/bmarini/knife-inspect

37. @jonlives
Feature Flags

38. @jonlives
Downsides of Existing Approach
• Holding cookbook in testing is blocking
• Accidental promotions
• Testing env affects all cookbooks
• “Upgrade” envs often used
• How to make it more “Etsy”?

39. @jonlives

40. @jonlives
chef-whitelist
• https://github.com/etsy/chef-whitelist
• Databag driven
• Cookbook library
• Feature flags!

41. @jonlives
chef-whitelist
{
"id": "php-5-5-17",
"patterns": [
"statsd*.ny5.etsy.com",
"deploy*.ny5.etsy.com",
<snip>
]
}

42. @jonlives
chef-whitelist
if node.is_in_whitelist? "php-5-5-17"
package "php-pecl-opcache" do
action :remove
end
end

43. @jonlives
Configuration Data

44. @jonlives
Keep cookbooks:
• Simple
• Modular
• Scalable
• Maintainable

45. @jonlives
Environments
• Cookbook version constraints

46. @jonlives
Roles
• Group-level config
• Syslog-ng
• Iptables
• Sudoers

47. @jonlives
Roles - iptables
“firewall": {
"ports": {
"11211": {
"subnet_group": "prod_subnets"
},
<snip>
}
}

48. @jonlives
Roles - Syslog-ng
"syslog": {
"web": {
"web_apache_access_log": {
"source": "/var/log/httpd/access_log",
"source_program_override": "APACHEACCESS: ",
"destination": "/data/syslog/current/web/access.log",
"destination_filters": [
"host('^(web0|dlweb)')",
"match('APACHEACCESS')"
]
}
}

49. @jonlives
Data Bags
• Global / Datacenter specific Config
• Ganglia
• Cobbler
• VOIP
• Data Storage

50. @jonlives
Data Bags - Ganglia
{
"id": "config_se5",
"grid_name": "EtsySE5",
"authority": "http://gangliase5.etsycorp.com",
"trusted_hosts": <snip>,
"groups": {
"Utilities": "239.2.11.71",
<snip>
}
<snip>
}

51. @jonlives
Data Bags - Cobbler
{
"id": "config_corp",
"cobbler_server": "corpking02.corp.etsy.com",
"dns_servers": [ “10.x.x.x", “10.x.x.x" ],
"dhcp_ranges": {
"10.100.x.0": {
"routers": "10.x.x.1",
"mask": "255.255.255.0",
"range": "10.x.x.11 10.x.x.250"
}
}
}

52. @jonlives
Write cookbooks you’ll
thank yourself for.

53. @jonlives
!
http://jonliv.es/book
!
Discount Code: AUTHD
!
40% off Print
50% off Digital

54. @jonlives
Thanks!
Questions?
!
@jonlives / http://jonliv.es / jon@etsy.com