At GOTO Amsterdam in 2019 I presented how to create an effective cloud native developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I'll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software. In this talk, you will: - Learn why the core tenets of continuous delivery -- speed and safety -- must be considered in all parts of the cloud native SDLC - Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis - Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger) - Explore the importance of cultivating an effective cloud platform and associated team of experts - Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes

1. Cloud Native Development Without the Toil:
Key Practices and Tooling
Daniel Bryant

2. @danielbryantuk
tl;dr
2
• Continuous delivery requires safety and speed throughout the engineering workflow
• Bringing old mental models and tools can increase toil when building, testing, and
running cloud native services
• Artifact syncing, dev environment bridging, and GitOps increase both safety and speed

3. @danielbryantuk
3

4. @danielbryantuk
4
The quest for happy users
Why we’re all here: Happy users
https://dilbert.com/strip/2014-02-25

5. @danielbryantuk
Continuous delivery of value
5
“Continuous delivery is achieved when stability and speed
can satisfy business demand.
Discontinuous delivery occurs when stability and speed
are insu
ff
icient.”
- Steve Smith (@SteveSmith_Tech)

6. @danielbryantuk
Continuous delivery of value
6
“Continuous delivery is achieved when stability and speed
can satisfy business demand.
Discontinuous delivery occurs when stability and speed
are insu
ff
icient.”
- Steve Smith (@SteveSmith_Tech)

7. @danielbryantuk
Continuous delivery of value
7
“Continuous delivery is achieved when stability and speed
can satisfy [your] business demand.
Discontinuous delivery occurs when stability and speed
are insu
ff
icient.”
- Steve Smith (@SteveSmith_Tech)

8. 8
Two modern so
ft
ware trends
for speed and safety

9. @danielbryantuk
Trend #1: Microservices
9
• Modularisation
• Aka “microservices”
• Independently:
• Build
• Release
• Scale
• Operate
• Strive for
• High cohesion
• Loose coupling

10. @danielbryantuk
Trend #2: Kubernetes
10
• Automated orchestration of container-based services
• Declarative config (with reconciliation loops)
• Common API/platform to support and extend
• But… a non-trivial set of abstractions/components for developers
• And it’s not a PaaS

11. @danielbryantuk
tl;dr
11
With cloud native, the focus is o
ft
en on speed

12. The Kubernetes Migration Journey(s)
12
Exploration
Organizational buy-in Cloud-native organization
(Loosely coupled, highly aligned)
We’re not actually moving faster
Let’s release our microservices as a monolith
Centres of excellence
No coordination results in disasters and blame
Investing in “paved road” platforms
And new dev workflows
Clinging to old mental models
and “the way it’s always been done”

13. 13
Safety
The ability to quickly ship
changes to production,
without fear

14. Safety requires multiple strategies
14
Prevention
Mitigation
Detection
Response
Most common strategy (e.g., testing, QA, gated release)
Critical strategies for cloud-native organizations when
the application is a “complex adaptive system”
}
}

15. Safety Strategies in the Cloud
15
Prevention
Mitigation
Detection
Response
• Testing: integration tests, user acceptance tests, …
• High-fidelity replica environments for dev & test
• Engineering for resilience
• Progressive delivery: canary release, blue/green rollouts, …
• Observability
• Chaos engineering
• Instant rollback & traceability
• Runbooks, fast dev / test loops
• Blameless postmortems
• Game days

16. Safety needs to
be part of your
workflow.
16
Develop
Test
Deploy/
Release
Run
Progressive Delivery
Rollback
Observability
Chaos Engineering
Runbooks
Fast dev loop
Fast feedback
Realistic test
environments

17. 17
Develop
Test
Deploy/
Release
Run
Progressive Delivery
Rollback
Observability
Chaos Engineering
Runbooks
Fast dev loop
Fast feedback
Realistic test
environments
Developer
Self-Service

18. @danielbryantuk
18
Patterns, practices, and tooling

19. @danielbryantuk
Pattern: Artifact Syncing
19
• Pain points:
• Can’t run all required dependent services locally
• Sick of code, build image, push to remote registry slow dev loop
• Integration tests rely on mocks (with implicit assumptions)
• Solution
• Deploy all services to remote environment and sync local changes (combine with buildpacks)
• O
ft
en combined with development environment bridging
• Example tool
• Ska
ff
old (ksync, Tilt, Garden)

20. @danielbryantuk
Ska
ff
old: “Source to K8s” CLI Tool
$ skaffold dev
20
Listing files to watch...
- skaffold-example
Generating tags...
- skaffold-example -> skaffold-example:v1.1.0-113-g4649f2c16
Checking cache...
- skaffold-example: Not found. Building
Found [docker-desktop] context, using local docker daemon.
Building [skaffold-example]...
Sending build context to Docker daemon 3.072kB
Step 1/6 : FROM golang:1.12.9-alpine3.10 as builder
---> e0d646523991
Step 2/6 : COPY main.go .
---> Using cache
---> e4788ffa88e7
Step 3/6 : RUN go build -o /app main.go
---> Using cache
---> 686396d9e9cc
Step 4/6 : FROM alpine:3.10
---> 965ea09ff2eb
Step 5/6 : CMD ["./app"]
---> Using cache
---> be0603b9d79e
Step 6/6 : COPY --from=builder /app .
---> Using cache
---> c827aa5a4b12
Successfully built c827aa5a4b12
Successfully tagged skaffold-example:v1.1.0-113-g4649f2c16
Tags used in deployment:
- skaffold-example -> skaffold-example:c827aa5a4b12e707163842b803d666eda11b8ec20c7a480198960cfdcb251042
local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
- pod/getting-started created
Watching for changes...
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!

21. @danielbryantuk
Ska
ff
old: “Source to K8s” CLI Tool
21

22. @danielbryantuk
Ska
ff
old: “Source to K8s” CLI tool
22
skaffold dev & skaffold debug
• Detects changes in your source code and handles the pipeline to build,
push
skaffold build and skaffold deploy
• Execute as part of your CI/CD pipeline, or ska
ff
old run end-to-end
skaffold render
• Build your images and render templated Kubernetes manifests for use
in GitOps workflows

23. @danielbryantuk
Pattern: Dev Environment Bridging/Extension
23
• Pain points:
• Can’t run all required dependent services locally
• Integration tests rely on mocks (with implicit assumptions)
• Need “hot reload” coupled to remote services/resources
• Solution
• Deploy all services to remote environment and proxy/re-route tra
ff
ic to/from
a local running copy of a service (or subset of services)
• Example tool
• Telepresence (kubefwd, Okteto)

24. @danielbryantuk
Telepresence: Local-to-Remote bridge
$ telepresence intercept dataprocessingnodeservice --port 3000
24

25. @danielbryantuk
Telepresence: Local-to-Remote bridge
25
www.youtube.com/watch?v=W_a3aErN3NU

26. @danielbryantuk
Telepresence: Local-to-Remote bridge
26
telepresence connect
• Open a tunnel to the remote cluster; exposes “in-cluster” services/network/DNS
telepresence intercept my-service —port 3000
• Re-routes (intercepts) tra
ff
ic to my-service in the remote cluster to my local machine
telepresence login & telepresence intercept
• Create preview URL to isolate and share results of the intercept

27. @danielbryantuk
Telepresence: Local-to-Remote bridge
$ telepresence intercept dataprocessingnodeservice --port 3000
27
Using deployment dataprocessingnodeservice
intercepted
State : ACTIVE
Destination : 127.0.0.1:3000
Intercepting: HTTP requests that match all of:
header("x-telepresence-intercept-id") ~= regexp ("76a1e848-1829-74x-1138-e3294c1e9119:dataprocessingnodeservice")
Preview URL : https://[random-subdomain].preview.edgestack.me
www.youtube.com/watch?v=W_a3aErN3NU

28. @danielbryantuk
Pattern: GitOps
28
• Pain points:
• No repeatable, auditable, & understandable deployment of infra and apps
• Slow hand-o
ff
s and manual deploys
• Solution
• Use version control (git) as single source of truth
• Declarative config; reconciled against target cluster
• Example tool
• Argo CD (Flux, JenkinsX, werf)
https://www.weave.works/blog/what-is-gitops-really

29. @danielbryantuk
Argo CD: GitOps for K8s
29
$ argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git /
--path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default

30. @danielbryantuk
GitOps in a Nutshell
30
https://www.weave.works/blog/the-gitops-pipeline

31. @danielbryantuk
Argo Rollouts: K8s Progressive Delivery Controller
31

32. @danielbryantuk
Putting it All Together: Safety and Speed
32
https://www.weave.works/blog/what-is-gitops-really
Development Production
Artifact syncing, Dev Env Bridging GitOps
Speed
Safety

33. @danielbryantuk
33
What about the platform
and the people?

34. @danielbryantuk
Workflow: Full Cycle Development
34
• App teams have full responsibility (and authority) for
delivering a service, and ultimately, value to users
• Increase agility by accelerating the feedback loop
• https://netflixtechblog.com/full-cycle-developers-at-netflix-a08c31f83249

35. app.getambassador.io
thenewstack.io/learning-kubernetes-the-need-for-a-realistic-playground

36. @danielbryantuk
Conclusion
36
• Continuous delivery of value to users requires a focus on safety and
speed throughout the engineering workflow
• Bringing old mental models and tools can add toil to the process of
building, deploying, and operating cloud native services
• Adopt best practices first, such as artifact syncing, dev environment
bridging, and GitOps to increase both safety and speed

37. @danielbryantuk
• Learn more:
• www.getambassador.io/resources
• www.getambassador.io/use-case/local-kubernetes-development
• www.infoq.com/profile/Daniel-Bryant
• Find me in:
• Datawire OSS Slack: d6e.co/slack
• Twitter @danielbryantuk
Many thanks!
37