Empowering developers to deploy their own data stores using Terrafom, Puppet and rage. A talk about automating server building and configuration for Elasticsearch clusters, using Hashicorp and puppet labs tool. Presented at Config Management Camp 2016 in Ghent
➥🔝 7737669865 🔝▻ mahisagar Call-girls in Women Seeking Men 🔝mahisagar🔝 Esc...
Empowering developers to deploy their own data stores
1. Empowering developers to deploy their
own data stores.
A story of Terraform, Puppet and rage
Tomas Doran
@bobtfish
2. • Iterate on the things you do often
• Hide complexity
• Empower others
2
Devops = Workflow
3. • A thing of the past (mostly)
• Need to be able to scale up and down in hours
• If not minutes
• Need to allow people to experiment
• Cloud is expensive, unless you use it!
3
Artisanal hand-crafted servers
5. • Remembering the . on PTR records
• For some people!
• Why make them do this?
5
The hardest thing
6. • Datastore PAAS
• Elasticsearch clusters are the ‘easy’ case
• No ‘master’ - all machines are equal
• Automatic sharding/replication
• ASG + ELB
• Zookeeper for discovery
6
Next logical step
9. • Hostname: search1-reviews-uswest1aprod
• Parse out cluster name
elasticsearch_cluster { ‘reviews’: }
puppet/modules/elasticsearch_cluster/data/cluster/
reviews.yaml
• Can locate the ‘data’ directory somewhere else!
• Reuse the same YAML for service discovery + provisioning
• Commit hook validation
9
puppet data in modules
11. • Bad abstraction for contextual information
• Which db server is the master? Does it have ‘master’ in it’s FQDN?
• If it does, what happens when you promote another machine?
• Need key => value for cattle not pets
• Customize your monitoring system to actually tell you what’s wrong!
• ‘The master db has crashed’ vs ‘A db has crashed’
• ‘10-46-11-54 is dead’ vs ‘zookeeper::10-46-11-54 is dead`
11
Hostnames
12. • Got most of the pieces
• Machines auto-configure themselves after launch.
• Remaining step is actually launching machines
• Terraform is awesome…
• IF you treat it as a low level abstraction
• IF you keep things in composeable units
• IF you add enough workflow to not run with scissors
12
Terraform
16. • Terraform the most generic abstraction possible
• Map JSON (HCL) DSL => CRUD APIs
• Cannot do implicit mapping
• But puppet / ansible / whatever can???
• ‘Name’ tag => namevar
• Only works in some cases - not everything has tags!
• Implicit mapping is evil
• Duplicates will screw up your day
16
Low level
23. • Reusable abstraction (in theory)
• Don’t try to use like puppet!
• Flat hierarchy (do not nest modules)
• Use version tags
• Use other git repos
• Or just generate resources as JSON
• KISS
23
Terraform modules
24. • Why even is state?
• How to cope with state
• Atlas
• Workflow (locking!) is your problem
• Remote state
• Shard terraform for (team) concurrency
• S3 store
• Many read, few write
• Wrap it yourself (make, Jenkins, don’t install terraform in $PATH)
24
State
25. • Provides the workflow
• ‘awsadmin’ machine + IAM Role as slave
• Makefile based workflow
• Jenkins job builder to template things
25
Jenkins
26. • Refresh state (upload refreshed state)
• Plan + save as artifact
• Filter plan!
• Approve plan
• Apply plan, save state
26
Split up the steps
27. • Commit some files to git.
• Push to a branch
• Jenkins runs
• Gated approval/application process
• Abstract away the scary parts
• Enforce workflow
27
Cluster provisioning workflow
28. • Self service cluster provisioning
• Developers define their own clusters
• 1 click from OPs to approve
• Owning team gets accounted
• AWS metadata added as needed.
• All metadata validated.
• Clusters built around best practices
• Can abstract further in future
28
Nirvana