talk at monitoring workshop 2014

1. monitoring configuration,
release & configuration management
Christoph Oelmüller
<christoph.oelmueller@epost-dev.de>
*

2. *
*say hello
*monitoring design considerations
*monitoring configuration management
*the problem
*two approaches to solve the problem
*1:n approach
*n:m approach
*a quick n:m approach howto
*summary: 1:n versus n:m

3. *
*physicist
*2008 - 2013: linux consultant
*2014 : operations engineer at EPost-
Development GmbH
*linux infrastructure, identity management,
configuration management, monitoring...

4. *

5. *
*requirement analysis
*technical requirements
*revision controlled configurations
*rollback
*automization
*organizational requirements
*feasable workflow
*keep it simple stupid (KISS)

6. *
*infrastructural requirements
*design your staging
*complete?
*identical?

7. *
*time & material & risk
*cost of a false alarm
*cost of a grumpy admin

8. dev
monitor
tst
monitor
prd
monitor
*
separate your stages!

9. *
what to configure?

10. *
*hosts
*add host to monitoring
master
*services
*add service to
monitoring master
*deploy check/plugin
on node

11. dev-master
tst-
master
prd-
master
dev-node
tst-
node
prd-
node
*

12. *
*all stages are identical
*all stages are complete
*one monitoring master/stage
*call it 1:n approach
*1 master
*n nodes
CODE YOUR CONFIG
CLONE YOUR CONFIG 

13. *
*stages aren‘t identical
*stages are incomplete
*stages are only implemented via different
monitoring masters
*call it n:m approach
*n master
*m nodes

14. *
dev-
master
dev-
node 1
dev-
node 2
tst-
master
tst-
node 1
tst-
node 2
prd-
master
prd-
node 1
prd-
node 2
nodes
dev-
master
tst-
master
prd-
master

15. *
*no staging related false alarms
*checks can‘t go amok in production
*configuration is tested properly in prd
*we‘ve talked about this...
*puppet & icinga, exported resources
monitoring configuration is completely described by
puppet

16. *
*stages are only reflected in monitoring masters
*non-prod doesn‘t alarm
*agent configuration is done via
puppet/package/however
*master configuration is done natively
*KISS 
CLONE YOUR CONFIG!

17. *
a quick howto

18. *
1. transfer prd stage to dev stage (files...)
2. edit configuration files/generate files (check_MK!)
3. sync files to VCS repository
*add/remove/modify
4. merge/copy dev-repo to prd-repo
*svn copy/svn merge/git branching
5. sync prd-repo to prd-filesystem
*again: add/remove/modify
workflow is done on dev-master

19. *

20. *
*local operations (on dev-master)
*checkout VCS
*checkin VCS
* remote operations (on prd-master)
*checkout VCS
*VCS operations
*transfer repositories (svn merge/git merge/...)
*operations are wrapped
*directly called for local operations
*puppet exec resources for remote operations

21. *
*checkout VCS
*from which repository?
*use a fact „stage“!
*a monitoring master can only checkout/sync its own stage
repository
*transfer repositories
*from where to where? the workflow knows...
*puppet runs on remote hosts are triggered by
mCollective
*addressing is done via „stage“ fact

22. *
*remote execution framework
*a simple ssh loop won‘t do it...
*message subscribe architecture
*parallel job execution
*a job => mCollective plugin
*ping, service, puppet, your own plugin
*watch out
*mco agent = subscriber
*mco client = message sender

24. *
*each directory is a repository
*??? => monitoring configuration is spread over the
filesystem
*ask cmk --paths
*the assignment dir to repo is configured in
„locations.cfg“
*configuration directories are working copies

25. *
file{‘/etc/mon/locations.cfg‘:
ensure => present,
source => puppet:///...,
notify => Exec[‘mon_checkout‘]
}
exec{‘mon_checkout‘:
command => ‘/usr/local/bin/mon checkout‘,
notify => Service[‘nagios‘],
}
service{‘nagios‘:
ensure => running,
}

26. *
*dirty little secrets
*only complete directories
*puppet run feedback?
*file permissions
*distributed/simultaneous work

27. *
1:n versus n:m – the battle...

28. 1:n n:m
separation complete separation partial separation
transfer
dev2prd
via release management
tools
on request
puppetization
level required
high low
risk low ...a bit more
complexity
(KISS)
high low
configuration via puppet natively

29. *
*risk versus implementation effort
risk
implementation effort
1:n
n:m

30. *
mailto:christoph.oelmueller@epost-dev.de
http://slideshare.net/christophoelmuller/