Saltstack is by it's design a event driven configuration management tool. In talk will do a deep dive into salt reactor, runners and beacon systems. Talk will also cover a demo of event driven application releases process.

1. Orchestrate Event-­‐Driven
Infrastructure with SaltStack
Love Nyberg
Site reliability & Automa@on Engineer

2. SaltStack
SaltStack is extremely fast and scalable systems and
configura@on management soDware for predic@ve
orchestra@on, cloud and data center automa@on, server
provisioning, applica@on deployment and more.

3. Agenda
salt @ bloglovin
Terminology
Orchestrate
Reactor
Beacons

4. salt @ bloglovin
tens of millions of monthly unique visitors
hundred of millions requests per day
couple of million stories parsed per day
hybrid bare metal servers/cloud vm’s
2 salt masters
~90 salt minions

5. salt @ bloglovin
Parts we use!
States/Pillars/Grains
Orchestrate
Reactor
Salt Api
Salt Cloud

6. Terminology
Configura)on
Master/Minion
Masterless
Transport
ZeroMQ
Raet
SSH
Parts
States
Modules
Grains
Pillars

7. Execu@on
salt '<target>' <function> [arguments]

8. Execu@on modules
root@master:/# salt minion_love network.interface eth0
minion_love:
|_
----------
address:
172.17.0.9
broadcast:
None
label:
eth0
netmask:
255.255.0.0

9. Execu@on modules
root@master:~# salt minion1 cmd.run "ip addr show eth0"
minion1:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc
pfifo_fast state UP group default qlen 1000
link/ether 12:4b:af:05:be:cd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.206/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::104b:afff:fe05:becd/64 scope link
valid_lft forever preferred_lft forever

10. State
/srv/salt/state/sshd/init.sls
ssh:
pkg.latest
- name: openssh-server
salt test_minion state.sls ssh

11. State/Execu@on
/srv/salt/state/sshd/init.sls
ssh:
pkg.latest
- name: openssh-server
root@master:/# salt test_minion pkg.latest_version openssh-server

12. {%- set ips = salt['network.ip_addrs'](None, False, '172.17.0.0/24')
%}
ssh:
pkg.latest
- name: openssh-server
service.running:
- enable: True
- watch:
- file: ssh
- pkg: ssh
file.managed:
- name: /etc/ssh/sshd_config
- source: salt://sshd/files/sshd_config
- template: jinja
- context:
ips: {{ ips }}

13. Grains
via state
root@mon1:/# salt minion_love grains.get ip_interfaces:eth0
minion_love:
- 172.17.0.9
within a state file
{%- set eth0 = salt['grains.get'](ip_interfaces:eth0, '') %}

14. Pillar
root@master:/# salt minion_love pillar.item secrets
minion_love:
----------
secrets:
----------
foo:
bar

15. Apply state
root@master:/# salt minion_love state.highstate
salt '<target>' <function> [arguments]

16. Event system
Event system is a local ZeroMQ PUB interface which fires salt events
Every event has a tag and a data structure

17. Event chain `test.ping`
Event fired at Wed May 27 22:00:14 2015
*************************
Tag: new_job
Data:
{'_stamp': '2015-05-27T20:00:14.774445',
'arg': [],
'fun': 'test.ping',
'jid': '20150527220014773556',
'minions': ['minion_love'],
'tgt': 'minion_love',
'tgt_type': 'glob',
'user': 'sudo_love'}

18. Event chain `test.ping`
Event fired at Wed May 27 22:00:14 2015
*************************
Tag: salt/job/20150527220014773556/ret/mon1
Data:
{'_stamp': '2015-05-27T20:00:14.981640',
'cmd': '_return',
'fun': 'test.ping',
'fun_args': [],
'id': ‘minion_love',
'jid': '20150527220014773556',
'retcode': 0,
'return': True,
'success': True}

19. Reactor
Ability to execute commands
Allow events to trigger ac@ons

20. # This script is to be called from the reactor system
{%- set application = data['data']['application'] %}
{%- set container = data['data']['container'] %}
{%- set node = data['data']['salt-node'] %}
{%- set tag = data['data']['tag'] %}
sync-states:
cmd.saltutil.sync_states:
- tgt: {{ node }}
deploy-containers:
cmd.state.sls:
- tgt: {{ node }}
- arg:
- containers.{{ container }}
- kwarg:
pillar:
containers:
imagetag:
{{ application }}: {{ tag }}

21. Salt Api
Yes! There’s an rest-­‐api
curl -sS salt.jacksoncage.se/run
-H 'Accept: application/x-yaml'
-d client='local'
-d tgt='*'
-d fun='test.ping'
-d username='saltdev'
-d password='saltdev'
-d eauth='pam'

22. Salt Api
# Example start creation dev box via curl:
curl -H 'X-Salt-Key: Secret-Key-Here'
-H 'Content-Type: application/json'
-d '{"username":"love"}'
https://salt.jacksoncage.se/hook/api/devbox/create

23. Orchestrate
Configura@on order
Sequenced applied states
Orchestrate deployment in order on mul@ple systems

24. # Create AWS EC2 instance
create-instance:
salt.state:
- tgt: {{ master }}
- sls: devlovin.cloud.create
- concurrent: True
- pillar:
devlovinreact:
instancename: {{ basename }}{{ username }}
# Run highstate
highstate-run:
salt.state:
- tgt: {{ basename }}{{ username }}
- highstate: True
- pillar:
devlovinreact:
username: {{ username }}
- require:
- salt: create-instance

25. Beacons
Hook into system processes
Translate external events into the salt event bus
Events can then be responded by Reactor

26. Beacons
DEMO

27. Ques@ons?
Love Nyberg
@jacksoncage
love@bloglovin.com

28. Thanks!
Love Nyberg
@jacksoncage
love@bloglovin.com