Puppet runs can be deconstructed into data at each step: 1) Facter gathers facts about the agent node. 2) The ENC defines the node's configuration. 3) The catalog compiler generates a catalog based on facts and ENC definitions. 4) The agent applies the catalog and generates a report. All of this data, including facts, catalogs, reports, can be accessed and manipulated programmatically for use cases like pre-compiling catalogs, inspecting state changes, or analyzing failures. Interacting with Puppet's internal data structures provides power and flexibility beyond traditional Puppet runs.

1. Puppet as Data
Transformations
Dan Bode| Puppet Labs
dan@puppetlabs.com
bodepd <on> [twitter,freenode]

2. What is it about?
• Deconstructing Puppet to data
• Why you should care
# puppetcamp # puppetize @ puppetlabs

3. Part One
Dissecting a Puppet Run

4. Facter, who am I?
Hi! your facts are:
kernel=linux
ipaddress=10.0.0.3
Agent macaddress=…
# puppetconf # puppetize @ puppetlabs

5. facts
Hi Mr. Master,
I need a catalog. Here
Agent are my facts
http://www.dgcomputers.org/testimonials.php
# puppetconf # puppetize @ puppetlabs

6. facts
Thanks for you facts.
Agent store them in
I’ll just
PuppetDB
PuppetDB
# puppetconf # puppetize @ puppetlabs

7. Mr. ENC, is this host
defined as an external Yep, he should be an
node? apache server. Here is
the definition
nodes
ENC
ENC
Agent PuppetDB
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

8. catalog
Just compiled your
Agent One sec while
catalog. PuppetDB
PuppetDB
I store it in PuppetDB.
facts
# puppetconf # puppetize @ puppetlabs

9. Here is your
catalog. Send me
a report and let
me know how it
went!
catalog
Agent PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

10. I hate to be a
bother, but can
you compute
the md5sums of
this file?
catalog
Agent PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

11. Oh, I need that
one! Can I get
the file.
catalog
Agent PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

12. Thanks, now
let’s just do that
1000 more
times.
catalog
Agent PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

13. You know
‘content’ will
embed the
contents in the
catalog?
catalog
Agent PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

14. report
Just finished applying.
Agent are the results.
Here PuppetDB
PuppetDB
catalog
facts
# puppetconf # puppetize @ puppetlabs

15. Part Two
Puppet’s internal data language (indirectory/termini)

16. facts find from
terminus facter
Agent
# puppetconf # puppetize @ puppetlabs

17. facts
catalog find from
terminus rest
Agent
http://www.dgcomputers.org/testimonials.php
# puppetconf # puppetize @ puppetlabs

18. facts
facts save to terminus
Agent puppetdb
PuppetDB
# puppetconf # puppetize @ puppetlabs

19. node find from
terminus exec (or
ldap)
nodes
ENC
ENC
Agent PuppetDB
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

20. catalog
catalog find from
Agent
terminus compiler PuppetDB
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

21. catalog
catalog save to
Agent puppetdb
terminus PuppetDB
PuppetDB
facts
# puppetconf # puppetize @ puppetlabs

22. Disecting a Puppet Run
Facter
Facter Facts ENC Nodes/
ENC
Manifest
Compiler
Compiler
Reports Config Catalogs
# puppetconf # puppetize @ puppetlabs

23. CLI commands
# puppetconf # puppetize @ puppetlabs

24. CLI Puppet Facts
# mkdir –p /tmp/yaml/facts
# puppet facts find node_name --render-as yaml
> /tmp/yaml/facts/node_name.yaml
# puppetcamp # puppetize @ puppetlabs

25. Creating a node (optional):
# mkdir –p /tmp/yaml/nodes
# puppet node find node_name
--node_terminus=exec
--external_nodes=/etc/puppet/nodes.sh
--facts_terminus=yaml
--clientyamldir=/tmp/yaml/ --render-as=yaml
> /tmp/yaml/nodes/node_name.yaml
# puppetcamp # puppetize @ puppetlabs

26. Applying a catalog:
# puppet catalog find node_name
/tmp/catalog.yaml
# puppetcamp # puppetize @ puppetlabs

27. Creating a catalog:
# puppet apply –catalog /tmp/catalog.yaml
(its easy assuming you are not using file sources)
# puppetcamp # puppetize @ puppetlabs

28. Fun with IRB
# puppetconf # puppetize @ puppetlabs

29. IRB Facts
irb:> require ‘puppet/face’
> Puppet.parse_config # required Puppet > 3.0
> facts=Puppet::Face[:facts, :current].find('node_name')
# puppetcamp # puppetize @ puppetlabs

30. Access a Fact value (irb):
…
> facts.values['ipaddress']
=> "10.0.2.15"
# puppetcamp # puppetize @ puppetlabs

31. Creating a node (from irb):
…
> node=Puppet::Node.new('node_name',
{:classes => {:foo => {:bar => :baz}}})
>node.merge(facts.values)
# puppetcamp # puppetize @ puppetlabs

32. Creating a catalog:
…
irb> catalog=Puppet::Face[:catalog, :current].
find('node_name', :extra => { :node => node})
# puppetcamp # puppetize @ puppetlabs

33. Use Cases
Interacting with Puppet’s Data

34. Inspecting the catalog:
• What types are in the catalog?
irb> catalog.resources.collect {|r| r.type }.uniq
• Gimme a resource:
irb>catalog.resource(‘Package[httpd]’)
# puppetcamp # puppetize @ puppetlabs

35. Rspec Puppet:
let :facts do
{:operatingsystem => ‘Redhat’}
end
let :params do
{:bind_address => ‘0.0.0.0’
end
it { should contain_file(‘/tmp/foo.conf’) }
# puppetcamp # puppetize @ puppetlabs

36. Thundering Herd
Pre-compile catalogs for faster auto-scaling
# puppetcamp # puppetize @ puppetlabs

37. Applying pre-compiled
catalogs:
• Gather facts from an ec2 instance
• Generate a single catalog
• Apply that catalog to all hosts
puppet apply --catalog /tmp/catalog.json –server
puppet-fileserver
# puppetcamp # puppetize @ puppetlabs

38. DMZ
tcp over USB
# puppetcamp # puppetize @ puppetlabs

39. Use Cases
Interacting with Puppet’s Data

40. Hacking reports
Everything in Puppet is a state transition
User[‘dan’] : absent -> present
User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
# puppetcamp # puppetize @ puppetlabs

41. Setting up the agent:
[agent]
report=true
# puppetcamp # puppetize @ puppetlabs

42. Archive reports in your
yamldir
[master]
reports = store
# puppetcamp # puppetize @ puppetlabs

43. Puppet reports
$ cd `puppet config print reportdir`
$ ls
node1 node2 node3
$ ls node1
# puppetcamp # puppetize @ puppetlabs

44. Every report from every run
ever
$ ls node1
201206060256.yaml 201206060303.yaml
201206060519.yaml 201206122349.yaml
201206122354.yaml 201206130002.yaml
# puppetcamp # puppetize @ puppetlabs

45. Lets crack one open!
Irb > require ‘yaml’
>reports=YAML.load_file('201206130002.yaml')
# puppetcamp # puppetize @ puppetlabs

46. Have a look
>(reports.methods - Object.methods).sort
Notice the following methods:
# puppetcamp # puppetize @ puppetlabs

47. High level data
> reports.exit_status
⇒0
> reports.status
=> "unchanged"
> reports.host
⇒”node1”
# puppetcamp # puppetize @ puppetlabs

48. metrics
> reports.metrics.keys
⇒["resources", "events", "changes", "time"]
> reports.metrics['resources']
⇒[‘failed’, 0],[ ‘changed’, ‘7’]
# puppetcamp # puppetize @ puppetlabs

49. And the awesome sauce
> reports.resource_statuses.keys
=> ["Package[xinetd]", "File[/srv/node/1]",
"Package[swift]", "Exec[compile fragments]",
"Package[swift-container]", "File[/var/opt/lib/pe-
puppet/concat/_etc_swift_object-server.conf]",
"File[/etc/rsync.d/frag-account]”]
# puppetcamp # puppetize @ puppetlabs

50. And the awesome sauce
> status = reports.resource_statuses
> status.keys
=> ["Package[xinetd]", "File[/srv/node/1]",
"Package[swift]", "Exec[compile fragments]",
"Package[swift-container]", "File[/var/opt/lib/pe-
puppet/concat/_etc_swift_object-server.conf]",
"File[/etc/rsync.d/frag-account]”]
# puppetcamp # puppetize @ puppetlabs

51. And the awesome sauce
>events = status["File[/etc/swift/swift.conf]"].events
> events.first.status
⇒"success”
> events.first.desired_value
⇒:present
> events.first.previous_value
=> :absent
# puppetcamp # puppetize @ puppetlabs

52. amp
Thank You
Dan Bode| Puppet Labs
dan@puppetlabs.com