Shown in Docker, Inc.'s sponsor booth at KubeCon US 2019 (Nov 21, San Diego)

1. Tibor Vass, Docker
Akihiro Suda, NTT
Akihiro Suda, NTT
Distributed Builds on Kubernetes
with BuildKit and Docker BuildX
1

2. Why build images on Kube?
2
Two kinds of motivation
1. CI/CD
2. Developer Experience

3. About us
3
Why build images on Kube?
3
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD

4. About us
4
Why build images on Kube?
4poor CPU, RAM, Wi-FI, battery
2. Developer Experience
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD
4

5. About us
5
Why build images on Kube?
5poor CPU, RAM, Wi-FI, battery
2. Developer Experience
BK Pod
BK Pod
BK Pod
Some
Pod
Some
webhook
1. CI/CD
5
BuildX

6. Kubernetes driver for BuildX
6
$ docker buildx create
--driver kubernetes
--driver-opt replicas=3
--use
$ docker buildx build -t foo --load .
Set up BuildKit cluster
on Kube automatically
No YAML mess
No TLS certificate mess

7. Kubernetes driver for BuildX
7
$ docker buildx create
--driver kubernetes
--driver-opt replicas=3
--use
$ docker buildx build -t foo --load .
Same user experience
as legacy docker build
But specify --load to load
the result to local Docker
(or --push to push)

8. Rootless mode
8
• You don’t want to run privileged pods? Don’t
worry
• BuildKit pods can be executed as a non-root user
(UID 1000) without securityContext.privileged
− Protect the host from potential BuildKit vulns
− Friendly to PSP-restricted clusters
− Fake root is emulated for Dockerfile RUNs
docker buildx create … --driver-opt rootless=true

9. About us
9
Demo

10. Caching
10
• Cache can be pushed to a registry as well as image
BK Pod
BK Pod
BK Pod
BuildX Registry
Request Cache

11. Caching
11
• Cache can be pushed to a registry as well as image
BK Pod
BK Pod
BK Pod
BuildX Registry
CacheRequest

12. Caching
12
• But remote cache might be slow compared to
the BuildKit daemons’ local cache
(/var/lib/buildkit in each of the Pods)
• Example:
− No cache: 2m50s
− Remote cache: 36s
− Daemon-local cache: 0.5s

13. Caching
13
Sticky mode allows
sticking a build request
to a specific Pod based
on the Dockerfile path
• Always hit cache
• But non-optimal LB
buildkitd-1
buildkitd-0
buildkitd-2
foo/Dockerfile
bar/Dockerfile
baz/Dockerfile
--driver-opt loadbalance=(sticky|random)
Circular hashing space

14. Parallelism
14
• Multiple Dockerfiles:
built in parallel using multiple nodes
• Multiple stages in a single Dockerfile:
built in parallel using a single node
− Future work: scatter stages across multiple
nodes in parallel

15. About us
15
github.com/docker/buildx