Overview of the modular sub-systems in SaltStack, and basics for writing custom execution modules and state modules for Salt. Presented to the Portland SaltStack Users Group (http://www.meetup.com/Portland-SaltStack-Users-Group/) by me (Jason Denning) on 04/21/2015.

1. Understanding Salt
Modular Sub-Systems and Customization

2. About Me (Jason Denning)
• DevOps Engineer
• SaltStack user for ~ 3 years
• SaltStack Certified Engineer #2
• AWS Certified Architect
• OSS Advocate (first Linux install ~ 1998)
• Pythonista
• jasondenning on GitHub / @jason_denning on Twitter

3. ● Provide analytics and
marketing/segmentation tools for
mobile app developers
● Hosted backend APIs available world-
wide
● Lots of traffic
● Big fans of SaltStack
● upsight.com

4. Frustratingly Flexible
• Salt is extremely flexible and extensible
• Highly modular / easy to customize
• Can be difficult (at first) to know where your custom code should
go
• Occasionally confusing nomenclature
• Pillar
• Grains
• Mine
• Reactor
• ...

5. But once you know your way around..
...Salt’s pretty awesome
• Easy to customize
• Scalable
• Secure
• Doesn’t suffer from the Pareto Principal (80% rule)

6. This Talk
You’ll (hopefully) understand:
• When to use different functionality
• How all the pieces fit together
• How to get started customizing Salt
• Where to go next

7. Salt’s not Configuration Management
• Salt is a remote execution framework
• Run arbitrary code on remote hosts
• Tools to make this easy and flexible
• Configuration Management is just a common use-case
• Originally intended to be a tool for managing cloud-based
infrastructure

8. Remote Execution 101
Basic remote execution:
$ ssh jason@myhost "ls /home/jason"
it_worked.txt
Multiple hosts?
$ for h in myhost1 myhost2 myhost3;
> do ssh jason@myhost "ls /home/jason";
> done

9. Remote Execution 101
Ok, but what if:
• I want to run it on 100 hosts?
• My login credentials are different on some hosts?
• I actually want to do something with the output?
• I want to do this many times?

10. Remote Execution 102
How about a script?
#!/bin/bash
# Make sure your SSH config (~/.ssh/config) is setup!
command=$1
host_list=$2
for h in host_list; do
ssh "$h" "$command"
done
# FIXME: Add output handling
# FIXME: Add error handling
# FIXME: Add logging
# TODO: FIND A BETTER WAY TO DO THIS!!!!

11. Looks like we need:
A script that can handle:
● Executing arbitrary commands
● ...on one or more remote hosts
● …with a sane way to get the output (STDOUT? Write a file?
Text output? JSON? YAML??)
● …and graceful error handling
● …and logging
● …and authentication
● …and everything else I haven't thought of yet!

12. And then we realize..
● We only want to run a command if the host is in a particular
state...
● the command is different for some hosts because they have a
different OS...
● we need to configure host X before we configure host Y...
● we want to generate a file from a template, using data from an
external database
● we need to keep that external data secret

13. You want me to maintain a 5,000 line BASH Script?

14. I’m convinced - let’s use a framework
This is why Salt has so many modular subsystems:
• Each type of component scratches a particular itch (there’s
some overlap)
• ⇒ Modular
• Someone is inevitably going to want something slightly different
• ⇒ Extensible
• We’re almost always going to want things like error handling
• ⇒ One framework instead of several mini-frameworks

15. Salt Basics
• A host can be a “master”, “minion”, “masterless”, or any combination
• a master can instruct minions to run code (via ‘salt’ cmd)
• a minion can trigger actions on itself using code/configuration
stored on its master (via ‘salt-call’ cmd)
• a minion can run in isolation, called “masterless” (via ‘salt-call’)
• a single machine can run both master and minion daemons
The point: Some subsystems run on the master, some on the minion

16. The Event Sub-System
Salt transfers commands and other data (including files)between
hosts via the Event Sub-System.
• Publish/Subscribe message queue (based on ZeroMQ)
• Master can publish messages on minion queues, and vice-versa
• Code execution on the minion is triggered via Event messages
• Output is sent back to the master from the minion in the same
fashion
• Messages are encrypted, and authenticated with Salt’s Public
Key Infrastructure (PKI)
• Reactor: Watches the message queue on the master, and
triggers code execution when it sees specific events

17. How Salt Works (simplified)
$ salt minion1 cmd.run
Salt Master
<do stuff>
minion1
Master Message Bus
(zmq / RAET)
Message
(Event)
Message
(Event)
Minion Message Bus
(zmq / RAET)
Message
(Event)
Message
(Event)
<Encrypted>

18. Code Execution Sub-Systems
• Execution Modules (a.k.a. “modules”): Code that executes on
the minion
• Runner Modules: Code that executes on the master
• State Modules: Code that executes on the minion, depending
on the current state of the host (typically utilize execution
modules)

19. Code Execution: CLI
• ‘salt’ command: (from master) instruct minion(s) to invoke an
execution module, which executes code on the minion
• ‘salt-call’ command: (from minion) invoke an execution module,
which executes code on the minion
• ‘salt-run’ command: (from master) invoke a runner module,
which executes code on the master

20. Data Sub-Systems
• Pillar: Data which is passed to a particular minion via the master
• Pillar Modules: Code that runs on the master which
generates Pillar data
• Grains: Data which is set directly on the minion
• Grains Modules: Code that runs on the minion which
generates Grains data
• Mine: Data collected from minion(s) which is stored on the
master, and made available to all other minions

21. Major Subsystems: Output Handling
• Output Modules: Format output from the CLI
• e.g. text (human-readable), YAML, JSON
• Returner Modules: Send output to external data-stores
• e.g. syslog, MySQL, etcd
• Note: Minions always return output to the master on the
Event bus, regardless of whether Returner modules are
used or not

22. SLS Files
• Human-readable files, with the .sls extension
• Processed by the Renderer subsystem to create data structures
that are understood by other sub-systems
• Most users interact with Salt via SLS files
• Renderer Modules: Convert SLS files into usable data
structures (execute on both master and minion)
• Composable (e.g. Jinja + YAML)
• Can access Pillar and Grains data

23. Targeting Minions
• A salt-master can target minions based on the minions’ ids,
Grains, or Pillar data
• States and Pillar data are assigned to via the Master Tops sub-
system (typically, this is just a file, called top.sls)
• Salt-SSH targets hosts (which aren’t running the minion
daemon) via the Roster sub-system

24. Other Modular Sub-Systems
• Auth : Enable external authentication systems
• Fileserver: File storage backends used by the master
• Wheel: Used to manage the master’s configuration
• Net-API: Access the master via the web
New in 2015.2:
• Engine: Long-running code on either master or minion
• Beacons: Code running on minions, used to translate external
events to the Event bus

25. More Stuff!?!?
• Salt Cloud: Manage cloud (IaaS) infrastructure
• Salt SSH: Apply salt states on hosts without a minion daemon,
via SSH
• Salt Virt: Manage virtual hosts and hypervisors
• Proxy-Minions: Manage “dumb” hardware, such as network
hardware

26. That’s a lot of sub-systems!

27. Let’s Get Customizin’
• First, make sure that you actually need to write code
• http://salt.readthedocs.org/en/latest/ref/modules/all/index.html
• http://salt.readthedocs.org/en/latest/ref/states/all/index.html
• … etc.
• https://github.com/saltstack/salt-contrib
• Look at existing module code:
• https://github.com/saltstack/salt/tree/2015.2/salt/modules
• https://github.com/saltstack/salt/tree/2015.2/salt/states
• … etc.

28. Custom Execution Modules
# _modules/hello_world.py
‘’’
This comment will print if you run $ salt-call sys.doc hello
‘’’
__virtualname__ = ‘hello’
def world():
‘’’ Prints “Hello World!” - this will print if you run $salt-call sys.doc hello.world’’’
print(“Hello World!”)

29. Custom Execution Modules
• Need to determine if the module can run on this minion?
• write a function called “__virtual__()”
• if __virtual__() returns False, the module will not be
available
• Useful if your code has external dependencies
• Need a “private” function?
• Prefix the function name with an underscore
• i.e. _this_is_invisible_to_the_user()
• Any function that doesn’t begin with an underscore will be
callable

30. Custom Execution Modules - Imports
# A useful pattern if you need to import an external library
try:
import foomodule
HAS_FOO = True
except ImportError:
HAS_FOO = False
def __virtual__():
‘’’Only load the execution module if we could import foomodule’’’
if HAS_FOO:
return True
else:
return False

31. Accessing Other Modules
• When Salt loads your module, it will create a couple of magic
dicts, just like in SLS files
• __salt__ : use to call other execution modules
• e.g. __salt__[‘cmd.run’](“ls /”)
• __grains__ : use to access grains
• e.g. minion_id = __grains__[‘id’]

32. Custom State Modules
• Much like execution modules
• Need to return a dict with this format:
{ ‘result’: True,
‘comment’: “Info about the state execution”,
‘changes’: { ‘old’: ‘the state before function was applied’,
‘new’: ‘state after the function was applied’
}
}
• ‘result’ should be True if the state is correct, False if there was
an error, or None if this was a test run

33. Custom State Modules - Tips
• State functions must accept a ‘name’ argument, at minimum
• Enable test mode! ($ salt-call state.sls mystate.foo test=True)
• Write a function called “mod_init(low)” which accepts the Low
State Data Structure to do one-time setup and initialization
• If you want to enable the ‘watch’ requisite, write a function called
“mod_watch()” - see the service state for a good example

34. Example State
From the documentation:
http://docs.saltstack.com/en/latest/ref/states/writing.html#example-state-module

35. Recap
• Salt makes it easy to run custom code on your infrastructure,
with lots of icing
• Lots of modular sub-systems, but they’re useful!
• Write custom code once, run many times
• No 5,000 line BASH scripts!

36. Now we can run whatever code we want!

37. Thanks!
Questions?
salt@jasondenning.com
GitHub: jasondenning
Twitter: @jason_denning