devOpsDays Galway 18 Nov 2019 https://devopsdays.org/events/2019-galway/program

1. Pipeline your
Pipelines!

2. Hardware spec:
1KB RAM (upg. 16KB)
4KB ROM
First computer Past Companies Communities
Giulio Vian Principal Engineer
& DevOps Lead

3. Session framing
Introductory level
Presentation deck on SlideShare
Blog series with examples
GitHub Repository with examples
Is my timer running?

4. There are no
development
environments

5. It all starts
with a itch
90% of our builds were running on a
physical server

6. What if I lose
my build
infrastructure?

7. I stop
deploying
fixes in
production!

8. Principle #1:
Development
is Production

9. Infrastructure: Continuous Integration (build)
Orchestrator
Server
Build Server
Agent
Build Server
Agent
Artifact Store

10. Did I say Agent?

11. Pick your favourites
slave
agent
runner
agent
remote agent
executor
master
server
node
server
server
server
Jenkins
Azure DevOps
GitLab
TeamCity
Bamboo
CircleCI
orchestrator agent

12. Infrastructure: Continuous Delivery (deploy)
Orchestrator
Server
Artifact StoreServer
Server
Agent
Deploy Server
Agent

13. Infrastructure: CI+CD
(build & deploy)
Orchestrator
Server
Build Server
Agent
Build Server
Agent
Artifact Store
Server
Server
Agent
Deploy Server
Agent

14. Why not using
Build
as‐a‐Service?

15. Principle #2:
Don’t share
any secret

16. Secrets?
Production passwords
Keys
SSL Certificates
Code signing certificates

17. What is
disposable?

18. Recycle scale
Source
code
Keys Deploy
data
Testing
data
Build data Build
infrastructure
Precious Disposable

19. Principle #3:
Automate
everything

20. Pattern #1 – pure Docker
Host
Container
Agent
AgentOrchestrator
Server
1
3
Pipelines:
1. Create the Host VM with Agent installed (require bootstrap)
2. Build and Push Docker images for Build Agents
3. Start Docker Containers hosting Build Agents
Docker
Registry
2

21. Design #2 – Kubernetes
Kubernetes Cluster
Container
Agent
AgentOrchestrator
Server
Docker
Registry
Pipelines:
1. Create the Kubernetes cluster (require bootstrap)
2. Build and Push Docker images for Build Agents (bootstrap)
3. Deploy Docker Containers hosting Build Agents to cluster
1
3
2

22. Step 0: Bootstrapping
Aim to minimal requirements, e.g.
• Cloud credentials
• Cloud toolkit (e.g. AWS or Azure CLI)
• Script that setup a VM
• Installs the server
• Restore backups

23. Step 1: Hosts
Physical machine(s)
VMs
Kubernetes cluster
Metal-as-a-Service
Microsoft Deployment Toolkit
Infrastructure-as-code
Terraform, Pulumi
CloudFormation, ARM Templates

24. Toolchain + Agent
Update regularly
Stateless Builds
Step 2:
docker-ize the agent

25. Step 3: pipeline the
creation of agents
Automate deployment of new / existing agents
Agent on host launches new agents in containers
Hurdle: manual admin grant

26. Step 4: pipeline the
creation of hosts
Script to create host
Create VM
Install Agent
Run on master or hosted agent
Autoscale?

27. Alternate path:
K8S hosting
Why bother managing machines?
Still need a cluster…
Illustration © SignMission 2018

28. Benefits and
collaterals
Documenting build and
deploy infrastructure
Re-create on any cloud
provider
Engineers familiarize
with Docker &
Kubernetes

29. Backup
Files
Databases
Printed instructions

30. Monitoring

31. The more you automate,
the more everything
becomes production

32. Series of
posts

33. Links
http://blog.casavian.eu/tags/pipelines/
https://medium.com/@giuliovdev
https://github.com/giuliov/pipeline-your-pipelines
https://www.slideshare.net/giuliov
giulio.dev@casavian.eu
@giulio_vian

34. End of transmissions
34