A bit of history, frustration-driven development, and why and how we started looking into Puppet at Opera Software. What we're doing, successes, pain points and what we're going to do with Puppet and Config Management next.

1. Puppet at Opera
Puppet Camp Oslo 2013
cosimo@opera.com

2. devs sysadmin

3. devs sysadmin
DevSys?

5. FDD
Frustration Driven Development

6. # LVS main config file
#
# Last modified:
# 2012-12-10 Commented out all wlb servers, as they haven't been in use …
# 2012-XX-XX Tons of shifting around servers, upgrading and problems (Everyone)
# 2011-04-01 Removed all old b#-servers (N.....)
# 2010-03-24 Bye bye bigma. (M..../Cosimo)
# 2010-03-03 Restore pre Feb 26th config that seems to ensure stability (Cosimo)
# When adding bigboy/bigcat, bad site lockups happen
# 2010-03-03 Reducing weight on b12 as it is less powerfull (M....)
# 2010-02-26 re-adding bigdog, and lowering bigunc, also vamping up b12 to 100%
# 2010-02-26 Bigdog is crashing, removing from lvs (M......)
# 2010-02-03 Enabled f8 and b7, first b7, then some hours later f8 … (N......)
# 2010-01-19 Bigant ready to rock and roll! (Cosimo)
# 2010-01-13 Removed bigpa, fatgirl from database pool (Cosimo)
# 2010-01-07 Added b8 to backend pool (Cosimo)
# 2010-01-05 Added bigant to the My Opera databases (Cosimo)
# 2009-11-22 Added bigdog to the My Opera databases (Cosimo)
# 2009-11-18 Added b7 and f8 as back-end servers (M.....)
# 2009-11-18 Removed p23-02 backend, moved to auth (Cosimo)
# 2009-11-12 Removing b7 and f8 from Mysql Load balancers (Cosimo)
# 2009-11-11 Added Lenny backend p23-02 (Cosimo)
# 2009-10-11 phased-in InnoDB-powered bigma in production (Cosimo)
# 2009-09-23 phased-in InnoDB-powered bigma in production (Cosimo)
# 2009-06-27 switched master from bigma to bigsis (w-mlb) o/ (N.....)
# 2009-06-23 shifting load away from bigbro. it's dying? (Cosimo)
# 2009-03-18 pushing bigbro as much as we can, to test it out (Cosimo)
global_defs {
lvs_id MY_LVS
…
}

7. innodb_buffer_pool_size = 128M
# was 64M # was 16M # was 32M

8. The Pilot – Goals
●
New deployment procedure
●
Sane configuration files
●
Configuration management

9. CM Tools Evaluation (2009)
CFEngine 2
BCfg2
Puppet 0.25.4
LCFG

10. CM Tools Evaluation
CFEngine 2
BCfg2
Puppet 0.25.4 → 2.6.2 → 2.7.14
LCFG

11. The very beginning...
commit 9c54321f51bf969940b63b48d055743ac504035e
Author: Cosimo Streppone <cosimo@opera.com>
Date: Thu Jan 14 13:21:40 2010 +0000
Generic puppet recipes. To be continued.

12. Our approach

13. A “conservative” approach, surely
• Keep it simple. No concat/append/modify
• As few dependencies as possible
• Stability and reliability is critical
• No pulls from github or external URLs
• We don't use puppet for deployment
• Even realize() gets me into panic mode

14. Three Years In
• Modules repository, with 60+ mods
• Some custom facter plugins
• Shared projects conventions & structure
• Shared deployment procedures and libs
• Good server baseline configuration
• Our team, ~200 nodes
• Opera Mini Ops team, thousands of nodes

15. Datacenters

16. It's Modules all the way down...
Tomcat base_packages
Apache PowerDNS
Bash
Statsd Munin
Postfix Cassandra
security_upgrades Ssh RRDCached
Django
Solr 4.0
RabbitMQ Varnish

18. Projects structure
Master config file /config/production.json
Role-specific files /config/role/<role>/
Puppet manifests /config/puppet/
Deployment scripts /deploy/

19. Master configuration file
{ "master_rev" : "20130129",
"application" : "geodns",
"environment" : "production",
"domain" : "localdomain",
"contact" : "cosimo@opera.com",
"puppet_vars" : { # Available in manifests
"some-password" : "hola/amigos" },
"systems" : { # List of all hostnames and their roles
"node01" : {
"puppet_class" : [ "geodns::backend" ]
},
"node02" : {
"puppet_class" : [ "geodns::frontend" ],
"puppet_vars" : { … },
},
…
}

20. Puppet master layout
/etc/puppet →
puppet.conf (master configuration file)
fileserver.conf
files →
{auth, geodns, opcdn} (local project files)
modules → (shared generic modules)
{ntp, apache, varnish, nginx, ...}
manifests → (generic and project specific manifests)
classes/
{basenode, backend, frontend}.pp
classes/ <project> /
<anything goes, project-specific>

21. Puppet master - site.pp
/etc/puppet/manifests/site.pp
$server = "puppetmaster.opera.com"
import "os/*.pp"
import "classes/*.pp" # generic classes
import "classes/*/*.pp" # project classes
node default { include basenode }
filebucket { "main": server => $server }
File {
ignore => ['.svn', '.git', 'CVS' ],
backup => "main",
}

22. Puppet master – no nodes.pp
/etc/puppet/puppet.conf
external_nodes = /etc/puppet/bin/puppet-node-classifier
node_terminus = exec
/etc/puppet/manifests/nodes/geodns-production.json
{ "application" : "geodns",
"environment" : "production",
"domain" : "localdomain",
"systems" : {
"node01" : {
"puppet_class" : [ "geodns::backend" ],
}, …
}
}

23. Facter
$ facter --puppet
architecture => amd64
datacenter => nerv
domain => opera.com
facterversion => 1.5.7
fqdn => node01.int.opera.com
hardwareisa => unknown
hardwaremodel => x86_64
hostname => node01
id => root
interfaces => eth0,eth1
ipaddress => 1.2.3.4
ipaddress_eth0 => 1.2.3.4
…

24. Facter – custom plugins
facter/datacenter.rb
Facter.add("datacenter") do
setcode do
datacenter = "unknown"
# Get current ip address from Facter's own db
ipaddr = Facter.value(:ipaddress)
if ipaddr.match("^1.2.3.")
datacenter = "dc1"
elsif ipaddr.match(...)
…
end
end
end

25. Facter – custom plugins
case $datacenter {
"dc1" : { include opera::datacenters::dc1 }
"dc2" : { include opera::datacenters::dc2 }
"dc3" : { include opera::datacenters::dc3 }
…
default: { include opera::datacenters::base }
}

26. Basenode class
class basenode {
include opera
# Opera-specific data-center based settings
case $datacenter {
"dc1" : { include opera::datacenters::dc1 }
…
default: { include opera::datacenters::base }
}
include apt-opera
include base_packages
include locales
include logcheck
include munin
include nagios
include cron
include perl
include python
include puppet
include ntp
include timezone
…
}

27. Bootstrap script
autosign
+ some preinstalled packages
+ internal apt repository
+ a bit of shell scripting

28. Real world examples – 1 Project
class geodns::backend {
include opera::admins::devops
include security-upgrades
include powerdns
include geoip::city
include memcache
package { [ 'libjson-xs-perl', … ]:
ensure => 'present'
}
bash::prompt { '/root/.bashrc':
description => 'geodns', color => 'red',
}
munin::plugin::custom { 'geodns_': }
munin::plugin {
[ 'geodns_country', 'geodns_errors', … ]:
plugin_name => 'geodns_',
}
}

29. Real world examples – 2 Varnish
varnish::config { "project-varnish-config":
vcl_conf => "tvstore.vcl",
storage_type => "malloc",
storage_size => "512M",
listen_port => 8100,
sess_workspace => 131072,
ttl => 60,
thread_pools => 2,
thread_min => 400,
thread_max => 3000,
# Needed for GeoIP support in varnish:
# http://stackoverflow.com/questions/5906603/
cc_command => "exec cc -fpic -shared -Wl,-x
-L/usr/include/GeoIP.h -lGeoIP -o %o %s"
}

30. Real world examples – 3 Munin
include munin::server
file { '/etc/munin/munin-conf.d/project-settings.conf':
…
}

31. Real world examples – 4 Solr
include solr4
solr4::core { 'core1':
config => '.../core1/solrconfig.xml',
properties => '.../core1/solrcore.properties',
schema => '.../core1/schema.xml',
}
solr4::config { 'solr-search-config':
cores => ['core1', … ],
}

32. Pain points AKA wish-list

33. Speed!
~60 s runtime → ~600 resources
TOO SLOW!

34. Resources that don't go away
notice: /Stage[main]/Django/Package[Django]/ensure:
ensure changed '1.4.3' to '1.4.2'
notice: /Stage[main]/Package[cython]/ensure:
created
notice: /Stage[main]/Java::Sun_java6/Exec[debconf-set-selections-sun-java6-bin]
/returns: executed successfully
notice: /Stage[main]/Java::Sun_java6/Exec[debconf-set-selections-sun-java6-jre]
/returns: executed successfully

35. Shared resources
cron::logcleanup { … }
• Used by both Apache and Nginx modules
• Getting conflicts if you pull both

36. Shared environment
Many projects run under the same master.
A syntax error anywhere blocks everyone.

37. Testing
Would be awesome to be able
to test our modules and manifests.
Locally.
Without a puppetmaster.

38. Future directions

39. Things we'd like to look into...
• PuppetDB
• Better systems inventory
• Better Nagios integration
• Testing manifests and modules

40. Q&A

41. @cstrep
cosimo@opera.com
https://github.com/cosimo/
http://w w w.streppone.it/cosimo/blog/