Deepak Giridharagopal - "Puppet DB: Higher-Order Puppet". Video at http://youtu.be/fDi-_NpeUcA PuppetCamp LA, May '12

1. PuppetDB
Higher-order Puppet
Deepak Giridharagopal
Lead Engineer @ Puppet Labs
deepak@puppetlabs.com
grim_radical, #puppet
Monday, May 21, 12

2. Let’s talk
about
data
Monday, May 21, 12

3. Monday, May 21, 12

4. Data!
Puppet generates a lot of it, in many
delicious flavors!
Persisted, ephemeral, machine local,
centralized, meticulously structured, totally
free-form, human readable, machine
optimized...
Monday, May 21, 12

5. Catalogs
“The Graph”
Containment edges, dependency edges,
classes, tags, resources, resource
parameters, metadata
Monday, May 21, 12

6. file {“/tmp/foo”: content => “This is a test”}
target: &id063 !ruby/object:Puppet::Resource
catalog: *id001
exported: false
file: /etc/puppetlabs/puppet/manifests/site.pp
line: 44
parameters:
!ruby/sym content: This is a test
!ruby/sym backup: main
reference: "File[/tmp/foo]"
tags:
- file
- node
- default
- class
title: /tmp/foo
type: File
Monday, May 21, 12

7. Group[peadmin]
User[peadmin]
Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]
peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha
Relationships
Monday, May 21, 12

8. Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data]
User[peadmin] User[puppet-dashboard]
File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo
/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/
Service[mcollective]
Relationships
Monday, May 21, 12

9. Group[peadmin]
User[peadmin]
File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]
le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
Monday, May 21, 12

10. Facts
Everything detected by facter
Facts for hardware, software, networking,
CPUs, memory, virtualization, manufacturer
info, custom facts...
Coming soon: structured facts!
Monday, May 21, 12

11. netmask_lo: 255.0.0.0 kernelrelease: 2.6.32-5-686
augeasversion: 0.10.0 ipaddress: 172.16.245.128
fqdn: pe-debian6.localdomain processor0: Intel(R) Core(TM)
manufacturer: "VMware, Inc." i7-2635QM CPU @ 2.00GHz
processorcount: "1" lsbdistrelease: 6.0.2
productname: VMware Virtual uniqueid: 007f0101
Platform hardwaremodel: i686
physicalprocessorcount: 1 kernelversion: 2.6.32
facterversion: 1.6.7 operatingsystem: Debian
boardproductname: 440BX architecture: i386
Desktop Reference Platform lsbdistdescription: Debian GNU/
kernelmajversion: "2.6" Linux 6.0.2 (squeeze)
hardwareisa: unknown lsbmajdistrelease: "6"
timezone: PDT interfaces: "eth0,lo"
puppetversion: 2.7.12 (Puppet ipaddress_lo: 127.0.0.1
Enterprise 2.5.1) uptime_days: 0
lsbdistcodename: squeeze lsbdistid: Debian
is_virtual: "true" rubysitedir: /opt/puppet/lib/
operatingsystemrelease: 6.0.2 site_ruby/1.8
virtual: vmware rubyversion: 1.8.7
type: Other osfamily: Debian
domain: localdomain memorytotal: &id001 502.57 MB
hostname: pe-debian6 memorysize: *id001
selinux: "false" boardmanufacturer: Intel
kernel: Linux
Monday, May 21, 12
Corporation

12. Reports
Catalogs say what you want, reports say
what you got.
Desired state, actual state, events, duration,
timestamps...
Monday, May 21, 12

13. "File[/tmp/foo]": !ruby/object:Puppet::Resource::Status
change_count: 1
changed: true
evaluation_time: 0.001869
events:
- !ruby/object:Puppet::Transaction::Event
audited: false
desired_value: !ruby/sym file
historical_value:
message: *id006
name: !ruby/sym file_created
previous_value: !ruby/sym absent
property: ensure
status: success
time: 2011-10-25 18:51:37.143970 -07:00
failed: false
file: *id007
line: 44
out_of_sync: true
out_of_sync_count: 1
resource: "File[/tmp/foo]"
resource_type: File
skipped: false
tags:
- file
- node
- default
- class
time: 2011-10-25 18:51:37.143396 -07:00
title: /tmp/foo
Monday, May 21, 12

14. Why
bother?
Monday, May 21, 12

15. “There's a war out there, old
friend. A world war. And it's
not about who's got the most
bullets. It's about who controls
the information. What we see
and hear, how we work, what
we think... it's all about the
information!”
-- Sneakers
Monday, May 21, 12

16. Storeconfigs
Centralized storage of the configuration of
all your nodes.
All resources, all parameters, all classes, all
tags, all stages...
Enables use of exported resources
Monday, May 21, 12

17. class exporter {
@@file {
"/var/lib/puppet/nodes/$fqdn":
content => "$ipaddressn",
tag => "ip"
}
}
node "export1.daysofwonder.com" {
include exporter
}
node "export2.daysofwonder.com" {
include exporter
}
node "collector.daysofwonder.com" {
File <<| tag == "ip" |>>
}
http://www.masterzen.fr/2009/03/08/all-about-puppet-storeconfigs/
Monday, May 21, 12

18. public key distribution
monitoring checks
clustered services
master/slave replication
load balancers
shared filesystems
firewall rules
...
Monday, May 21, 12

19. Query
Interrogation, investigation, correlation
Use Puppet-generated data in scripts or for
integration with other tools
Monday, May 21, 12

20. Higher
order
Puppet
Monday, May 21, 12

21. Volume
Every node, on every puppet run, generates
data
We have customers generating over 750G of
data a day. Even storing a small subset of
that much information adds up...
Monday, May 21, 12

22. (demo)
Monday, May 21, 12

23. Slow = :(
When data storage is slow, it makes baby
Deepak cry!
Slows down catalog compilation,
More quickly saturates a Puppetmaster,
Thrashes disk,
Bad news!
Monday, May 21, 12

24. API
Current APIs are limited
Hard to get at the data, and performance
concerns discourage use.
We need better ways of searching, filtering,
and correlating data.
Monday, May 21, 12

25. Paradox
Seemingly contradictory goals
We want to store as much data as we can,
and allow for better querying, but without
slowing stuff down or reducing reliability.
Monday, May 21, 12

26. We need
An information clearinghouse
Something that evolves the Puppet Data
Library. A scalable, safe place to store the
information Puppet collects and generates.
This is a hard problem!
Monday, May 21, 12

27. PuppetDB
Definitely Better!
Monday, May 21, 12

28. Grayskull
Monday, May 21, 12

29. PuppetDB
Monday, May 21, 12

30. PuppetDB is
Fast storage of current catalogs and current
facts,
100% compatible with storeconfigs and
inventory service,
REST APIs for resource, fact, and node
retrieval,
...and other things, even!
Monday, May 21, 12

31. science
&
secret alien
technology!
Monday, May 21, 12

32. Message Queue
"new catalog" "new catalog"
"new facts" "new facts"
"delete node" "delete node"
Puppetmaster
Compiler
Command Handler
Storeconfigs Parsing
Transformation
Validation
Storeconfigs,
Catalogs, Facts
REST
Puppet (SCF)
"inventory query"
Enterprise Domain
"interactive query"
Console objects
Query handling
CLI &
Other
Tools
Monday, May 21, 12

33. (export)
Monday, May 21, 12

34. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
Monday, May 21, 12

35. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

36. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

37. PuppetDB Server DLO
DB Workers
HTTP MQ
F
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

38. PuppetDB Server DLO
DB Workers
HTTP MQ
C F
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

39. PuppetDB Server DLO
DB Workers
F
HTTP MQ
C
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

40. PuppetDB Server DLO
DB Workers
F C
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

41. PuppetDB Server DLO
DB Workers
F C
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

42. (collection)
Monday, May 21, 12

43. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
Monday, May 21, 12

44. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

45. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

46. PuppetDB Server DLO
DB Workers
HTTP MQ
F
Agent Master
Facts Catalog Resrc
F ?
Monday, May 21, 12

47. PuppetDB Server DLO
DB Workers
HTTP MQ
? F
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

48. PuppetDB Server DLO
DB Workers
? F
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

49. PuppetDB Server DLO
DB Workers
F
HTTP MQ
?
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

50. PuppetDB Server DLO
DB Workers
F
HTTP MQ
Agent Master
Facts Catalog Resrc
F
?
Monday, May 21, 12

51. PuppetDB Server DLO
DB Workers
F
HTTP MQ
Agent Master
Facts Catalog Resrc
F ?
Monday, May 21, 12

52. PuppetDB Server DLO
DB Workers
F
HTTP MQ
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

53. PuppetDB Server DLO
DB Workers
F
HTTP MQ
C
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

54. PuppetDB Server DLO
DB Workers
F
HTTP MQ
C
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

55. PuppetDB Server DLO
DB Workers
F C
HTTP MQ
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

56. PuppetDB Server DLO
DB Workers
F C
HTTP MQ
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

57. (failure)
Monday, May 21, 12

58. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
Monday, May 21, 12

59. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

60. PuppetDB Server DLO
DB Workers
HTTP MQ
Agent Master
Facts Catalog Resrc
F
Monday, May 21, 12

61. PuppetDB Server DLO
DB Workers
HTTP MQ
F
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

62. PuppetDB Server DLO
DB Workers
HTTP MQ
C F
Agent Master
Facts Catalog Resrc
F C
Monday, May 21, 12

63. PuppetDB Server DLO
DB Workers
F
HTTP MQ
C
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

64. PuppetDB Server DLO
DB Workers
C
HTTP MQ
F
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

65. PuppetDB Server DLO
DB Workers
C F
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

66. PuppetDB Server DLO
DB Workers
C
HTTP MQ
F
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

67. PuppetDB Server DLO
DB Workers
C F
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

68. PuppetDB Server DLO
DB Workers
C
HTTP MQ
F
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

69. PuppetDB Server DLO
DB Workers
C F
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

70. PuppetDB Server DLO
F
DB Workers
C
HTTP MQ
Agent Master
Facts Catalog Resrc
C
Monday, May 21, 12

71. PuppetDB Server DLO
DB Workers
HTTP MQ
Monday, May 21, 12

72. PuppetDB Server
Workers DLO
DB
HTTP MQ
Monday, May 21, 12

73. PuppetDB Server
Workers DLO
HTTP DB
Proxy
(SSL)
HTTP MQ
Monday, May 21, 12

74. (launch)
Monday, May 21, 12

75. Reliable!
We work very hard to persist everything we
accept
Acknowledgements with UUIDS,
Checksums,
Queueing,
Automatic retry and reconnect,
and the Dead Letter Office if all else fails!
Monday, May 21, 12

76. APIs!
We don’t cheat
Anything Puppet does with PuppetDB, you
can do to
Query your own resources, upload new fact
sets, create catalogs, inspect facts...all part
of the Puppet Data Library
Monday, May 21, 12

77. #> curl
-H "Accept: application/json"
"http://puppetdb/metrics/mbean/
com.puppetlabs.puppetdb.command:type=global,name=processing-time"
{
"50thPercentile": 209.05,
"75thPercentile": 236.5865,
"95thPercentile": 428.3065999999959,
"98thPercentile": 750.53696,
"999thPercentile": 1246.722744999993,
"99thPercentile": 818.9180600000001,
"Count": 3322,
"EventType": "calls",
"FifteenMinuteRate": 1.1500295609205015e-06,
"FiveMinuteRate": 1.387569444096042e-18,
"LatencyUnit": "MILLISECONDS",
"Max": 26514.032,
"Mean": 314.1111032510536,
"MeanRate": 0.21577717049577358,
"Min": 185.53,
"OneMinuteRate": 3.390107448865515e-90,
"RateUnit": "SECONDS",
"StdDev": 833.6079354075728
}
Monday, May 21, 12

78. curl
-H "Accept: application/json"
"http://puppetdb/facts/host.my.net"
Monday, May 21, 12

79. curl
-H "Accept: application/json"
"http://puppetdb/resources?query=..."
Monday, May 21, 12

80. Transparent!
We care about operational visibility
Ships with a real-time dashboard,
Dozens of metrics and gauges,
Correlate-able logs,
Easy to integrate with monitoring systems
Monday, May 21, 12

81. Speedy!
PuppetDB is much, *much* faster than the
previous storeconfigs and inventory services
At Puppet Labs, we’ve seen huge reductions
in compile times, resource collection times,
time to persist catalogs and facts, etc.
Monday, May 21, 12

82. Design
decisions
Monday, May 21, 12

83. Posit:
Hosts are not
entirely unique
snowflakes
Monday, May 21, 12

84. Therefore:
A resource often
exists across
multiple hosts
Monday, May 21, 12

85. Feature:
Single-instance
resource storage
Monday, May 21, 12

86. Resource dedupe
Compute unique hashes for resources
We quickly hash all the resources in a
catalog, and use bulk operations to compare
them to hashes stored.
Monday, May 21, 12

87. Resource dedupe
Significant speed improvement!
Internal to Puppet Labs, we see ~83%
resource duplication; this number is
consistent with what we’ve seen in most
customer environments.
Monday, May 21, 12

88. Posit:
Puppet runs
frequently, but
catalogs change
infrequently
Monday, May 21, 12

89. Therefore:
We’ll often receive
the same catalog for
a host
Monday, May 21, 12

90. Feature:
Single-instance
catalog storage
Monday, May 21, 12

91. Catalog dedupe
Compute unique hashes for catalogs
We use a Merkle Tree approach (hash tree)
for quick comparisons.
Puppet Labs sees ~88% catalog duplication
Big savings!
Monday, May 21, 12

92. Posit:
You have more than
one core, though
storeconfigs is
single-threaded
Monday, May 21, 12

93. Therefore:
Throughput is not
maximized
Monday, May 21, 12

94. Feature:
Massively parallel
operation
Monday, May 21, 12

95. Parallel
We can pat our heads and rub our tummies
at the same time
Database operations don’t block MQ
operations don’t block HTTP operations
don’t block hash computation operations
don’t block metric calculations don’t block...
Dozens of threads, zero locks
Monday, May 21, 12

96. Monday, May 21, 12

97. science
&
secret alien
technology!
Monday, May 21, 12

98. The Future
Monday, May 21, 12

99. http://github.com/
puppetlabs/
puppetdb
Monday, May 21, 12

100. http://
docs.puppetlabs.com/
puppetdb
Monday, May 21, 12

101. Use it, and tell us
about it!
Monday, May 21, 12

102. PuppetDB
Thanks for your time!
Deepak Giridharagopal
Lead Engineer @ Puppet Labs
deepak@puppetlabs.com
grim_radical, #puppet
Monday, May 21, 12