Puppet Camp London Fall 2015 - Service Discovery and Puppet
•
9 likes•1,915 views
Presentation given at the London Puppet Camp in London on Nov 3rd 2015.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Puppet Camp London Fall 2015 - Service Discovery and Puppet
Similar to Puppet Camp London Fall 2015 - Service Discovery and Puppet (20)
More from Marc Cluet
More from Marc Cluet (18)
Recently uploaded
Recently uploaded (20)
Puppet Camp London Fall 2015 - Service Discovery and Puppet
- 5. Engineer based in London Co-Founder of Ukon Cherry Contractor at TrainLine 19 years of experience as a SysAdmin Founding member of Juju and MAAS while at Canonical Built a DevOps Engineering Team at Rackspace Been DevOps’in for the last 6 years WhoamI?
- 9. Cloud is hard! • Old style DNS • TTL is a b*tch • Health Checks • Metadata Storage TheProblem
- 11. Automatically define your services • Active Health Checking • Dynamically updated service lists • Can be DNS accessible (if needed) • API accessible (win!) ServiceDiscovery
- 13. ServiceDiscovery Service Publication Service Node A Service Node B Health Check Health Check Discovery AgentDiscovery Agent
- 14. ServiceDiscovery Service Publication Service Node A Service Node B Health Check Health Check Discovery AgentDiscovery Agent
- 15. ServiceDiscovery Service Publication Service Node A Service Node B Health Check Health Check Discovery AgentDiscovery Agent
- 16. ServiceDiscovery Service Publication Service Node A Service Node B Health Check Health Check Discovery AgentDiscovery Agent
- 19. ServiceDiscovery Service Node A Service Node C Service Node B Service: web 10.10.10.1 10.10.10.2 10.10.10.3
- 20. ServiceDiscovery Service Node A Service Node C Service Node B Service: web 10.10.10.1 10.10.10.2 10.10.10.3
- 21. ServiceDiscovery Service Node A Service Node C Service Node B Service: web 10.10.10.1 10.10.10.2 10.10.10.3
- 26. ServiceDiscoverySolutions API + DNS publication k/v Strongly Consistent Host + Service checks API publication k/v Consistent Container checks API publication Container checks Auto-Heal Puppet module Hiera access Puppet module Hiera access Puppet module
- 28. It's a Service Discovery System (duh!) • Service Publications (DNS + API) • k/v Storage (strongly consistent) • Health Checks • With encryption! (whaaaaaat) WhatisConsul?
- 32. ConsulArchitecture Consul Server A Consul Agent Consul Server B Consul Server C Consul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul Agent
- 33. ConsulArchitecture Consul Server A Consul Agent Consul Server B Consul Server C Consul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul Agent Consul Servers Another DC
- 35. service name: web zone: eu-west-1 <servicename>.service.<zone>.consul web.service.eu-west-1.consul ConsulDNSpublisher
- 37. { "check": { "id": "mem-util", "name": "Memory utilization", "script": "/usr/local/bin/check_mem.py", "interval": "10s" } } ConsulHealthChecks
- 45. Puppet+Consul You can integrate at several levels • Puppet Module • KyleAnderson-consul • Hiera • lynxman-hiera_consul
- 46. Puppet+Consul You can integrate at several levels • k/v access • venmo-consulr • Templates with Consul • ghdbaston-consul_template
- 47. PuppetModule-Install Puppet Forge makes it very easy to install $ puppet module install KyleAnderson-consul
- 48. PuppetModule-DNS $ puppet module install saz-dnsmasq include dnsmasq dnsmasq::conf { 'consul': ensure => present, content => 'server=/consul/127.0.0.1#8600', }
- 49. PuppetModule-NodeChecks consul::check { 'disk_space': script => '/usr/lib/nagios/plugins/check_disk -w 5% -c 1% -x /run/lock -x /run/shm -x /run/user -x /dev -x /run -x /sys/fs/cgroup', interval => '30s', }
- 50. PuppetModule-ServiceChecks consul::service { 'nginx': port => '80', checks => [ { script => "/usr/lib/nagios/plugins/check_http -H localhost -w 20 -c 60 -t 20", interval => '30s', }, ], }
- 52. PuppetModule-Install Puppet Forge makes it very easy to install $ puppet module install lynxman-hiera_consul
- 53. PuppetHiera-Config :backends: - yaml - consul :yaml: :datadir: /etc/puppet/hieradata :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: - /v1/catalog/service - /v1/catalog/node
- 54. PuppetHiera-Config :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: - /v1/catalog/service - /v1/catalog/node
- 55. PuppetHiera-Arrays notice('Generating rabbitmq cluster members based on Consul information') $consul_service_array = hiera('rabbitmq',[]) $mq_cluster_nodes = consul_info($consul_service_array, 'Address') notice("Result: ${mq_cluster_nodes}")
- 56. PuppetHiera-Arrays notice("Generating neo4j_ha cluster members based on Consul information") $consul_service_array = hiera('neo4j_ha',[]) $consul_fields = [ 'Address', 'ServicePort' ] $consul_ha_initial_hosts = consul_info($consul_service_array, $consul_fields, ':') $ha_initial_hosts = join($consul_ha_initial_hosts, ',') notice("Result: ${ha_initial_hosts}")
- 57. PuppetHiera-k/vaccess Long live to the API! :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: - /v1/kv/
- 58. PuppetHiera-k/vaccess Long live to the API! :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: - /v1/kv/web/
- 59. PuppetHiera-k/vaccess Long live to the API! :consul: :host: 127.0.0.1 :port: 8500 :failure: graceful :paths: - "/v1/kv/%{env}/"
- 62. PuppetSecurity
- 63. PuppetSecurity git commit -m "All my passwords" git push
- 71. Hieraeyaml Usage: eyaml <subcommand> Please use one of the following subcommands or help for more help: createkeys, decrypt, edit, encrypt, recrypt, version
- 76. EXTRATIME!
- 79. Vault$ vault write secret/foo value=bar Success! Data written to: secret/foo $ vault read secret/foo Key Value lease_id secret/foo/9c5f3cf1-1239-0160-4311-d6544fd1018c lease_duration 2592000 value bar $ vault delete secret/foo Success! Deleted 'secret/foo'
- 80. Vault $ vault read postgresql/creds/production Key Value lease_id postgresql/creds/production/8ade2cde-5081- e3b7-af1a-3b9fb070df66 lease_duration 3600 password 56b43bc3-b285-4803-abdf-662d6a105bd0 username vault-root-1430141210-1847