BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. It handles packaging, infrastructure provisioning across multiple clouds, application deployment with features like canary releases and rolling updates, configuration management of services rather than individual servers, and monitoring of processes, nodes, and the overall service. The presentation demonstrated deploying an ELK stack on AWS using BOSH and showed how it can scale the deployment vertically, horizontally, add more disk space, replace nodes, and add monitoring.

1. Persistence in the Cloud
with BOSH
Michael Richardson - @m_richo

2. Agenda
1.What is BOSH?
2.Demo - ELK in AWS

3. What is BOSH?

4. * BOSH is an open source tool for
release engineering, deployment,
lifecycle management, and monitoring
of distributed systems.
* source https://bosh.io
What is BOSH?

5. * BOSH is an open source tool for
release engineering, deployment,
lifecycle management, and monitoring
of distributed systems.
* source https://bosh.io
What is BOSH?

6. WTF??

7. Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning

8. Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning
fpm
/bin/bash

9. Packaging
Config Management Monitoring
Application
Deployment
Infrastructure Provisioning
fpm
/bin/bash
BOSH

10. 1.Packaging
2.Infrastructure Provisioning
3.Application Deployment
4.Configuration Management
5.Monitoring

11. Packaging

12. Packaging
Piece of code or pre-compiled binaries
required at either build-time or run-time

13. Infrastructure
Provisioning

14. Infrastructure Provisioning - Resources
Terraform CloudFormation
100s of cloud and
SaaS resources
100s of cloud and
SaaS resources

15. Infrastructure Provisioning - Resources
Terraform CloudFormation BOSH
100s of cloud and
SaaS resources
100s of cloud and
SaaS resources
● Compute
● Block store
● ELB & EIP
integration

16. Infrastructure Provisioning - Cloud Support
Terraform CloudFormation
Multi-cloud
● AWS
● Openstack
● Azure
● Google
● VMWare
● And many
more
AWS only

17. Infrastructure Provisioning - Cloud Support
Terraform CloudFormation BOSH
Multi-cloud
● AWS
● Openstack
● Azure
● Google
● VMWare
● And many
more
AWS only Multi-cloud
● AWS
● Openstack
● Azure
● Google
● VMWare

18. Application
Deployment

19. Application Deployment
Canary releases (canaries)
Rolling Updates (max_in_flight)

20. Configuration
Management

21. Configuration Management - Resources
Puppet/Chef/Ansible
100s of resources

22. Configuration Management - Resources
Puppet/Chef/Ansible BOSH
100s of resources 3 resources
● packages
● file template
● service

23. Configuration Management
Traditional CM tool (puppet, chef, ansible)
Configuration management for a server

24. Configuration Management
Traditional CM tool (puppet, chef, ansible)
Configuration management for a server
BOSH
Configuration Management for a service

25. Monitoring

26. Process monitoring (monit)
Node (VM) monitoring
Notifications
Email
Datadog
PagerDuty
Monitoring

27. Process monitoring (monit)
Node (VM) monitoring
Notifications
Email
Datadog
PagerDuty
Monitoring

28. Demo
ELK in AWS

29. Demo - ELK stack in AWS
AWS VPC
Availability zone a
BOSH subnet
Availability zone b
Public subnet
Private subnet
Public subnet
Private subnet
NAT
Gateway
NAT
Gateway
Internet Gateway
S3 bucket - bosh artifacts
BOSH Director
https://github.com/AusDTO/aws-infrastructure-provision

30. Demo - ELK stack in AWS
https://github.com/logsearch/logsearch-boshrelease
parser
ingestor
queue
ls-router
haproxy
Logstash
redis
logstash
ES cluster
ES data
EBS
EBS
EBS ES data EBS
ES master EBS
kibana
ES cluster monitor
ES maintenance
EBS
Logstash
S3 upload

31. Demo - Choose your own Adventure
1. Scale vertically
2. Scale horizontally
3. More disk space please
4. Kill a node with data
5. Add some more monitoring for your service