ISC HPCW talks

Copyright©2019 NTT Corp. All Rights Reserved.
Akihiro Suda ( @_AkihiroSuda_ )
NTT Software Innovation Center
My ISC HPCW Talks
1. Current state of rootless dockerd
2. Rootless build with BuildKit
3. OCI Image Spec & Distribution
5th Annual High Performance Container Workshop, ISC (June 20, 2019)

Current state of rootless dockerd

3 
What is rootless dockerd?
• Run Docker daemon (and also containers of course) as a
non-root user
• Don’t confuse with:
• sudo
• usermod -aG docker penguin
• docker run --user
• dockerd --userns-remap
• Experimentally supported since Docker v19.03
https://get.docker.com/rootless
Image: https://xkcd.com/149/

4 
Why?
• For Cloud-Native envs:
• To mitigate potential vulnerability of container runtimes and
orchestrator
• For HPC envs:
• To run containers without the risk of breaking other users
environments

5 
How it works: User Namespaces
• User namespaces allow non-root users to pretend to be
the root
• Root-in-UserNS can have “fake” UID 0 and also create
other namespaces (MountNS, NetNS..)
• Unlike Singularity, NetNS can be unshared
• By using either usermode TCP/IP stack (VPNKit, slirp4netns) or
SETUID binary (lxc-user-nic)

6 
System requirements: /etc/{subuid,subgid}
• If /etc/subuid contains “1001:100000:65536”
• Having 65536 sub-users should be enough for most
containers
0 1001 100000 165535 232Host
UserNS
primary user
sub-users
start
sub-users
length
0 1 65536

7 
Unresolved issues (Contribution wanted!)
• Hard to maintain subuid & subgid in LDAP/AD envs
• NSS module is being under discussion
https://github.com/shadow-maint/shadow/issues/154
• Single-mapping mode w/o subuid & subgid is also under
discussion
• uses ptrace and xattrs
(slow!)
• seccomp could be
used for acceleration
https://github.com/rootless-containers/runrootless

8 
Unresolved issues (Contribution wanted!)
• Lacks cgroup
• cgroup2 (uniﬁed-mode) supports unprivileged mode but
migration may take a few years… or even more
• For cgroup1, pam_cgfs could be used instead, but not available
in Fedora / RHEL due to a security concern
• Kernel / VM / HW may have vulns
• Not suitable for real multi-tenancy
• gVisor might able to mitigate some of them

Rootless build with BuildKit

10 
What is BuildKit?
• Next-generation docker build with focus on performance
and security
• Accurate dependency analysis
• Concurrent execution of independent instructions
• Support injecting secret ﬁles...
• Integrated to Docker since v18.06
(export DOCKER_BUILDKIT=1)
• Non-Docker standalone BuildKit is also available
• Works with Podman and CRI-O as well :P

11 
Rootless mode
• Rootless mode allows building images as a non-root user
• Dockerﬁle RUN instructions are executed as a “fake root” in
UserNS (So apt-get/yum works!)
• Produces Docker image / OCI image / raw tarball
• Compatible with Rootless Docker / Rootless Podman / …
whatever
• Even works inside a container
• Good for distributed CI/CD on Kubernetes
• Works with default securityContext conﬁguration
(but seccomp and AppArmor needs to be disabled for nesting containers)

12 
Rootless BuildKit vs kaniko
• https://github.com/GoogleContainerTools/kaniko
• Kaniko runs as the root but “unprivileged”
• No need to disable seccomp and AppArmor because kaniko
doesn’t nest containers on the kaniko container itself
• Kaniko might be able to mitigate some vuln that Rootless
BuildKit cannot mitigate - and vice versa
• Rootless BuildKit might be weak against kernel vulns
• Kaniko might be weak against runc vulns

OCI Image Spec & Distribution

14 
Open Containers Initiative Speciﬁcations
• OCI Runtime Spec
• How to create container from conﬁg JSON and rootfs dir
• Based on Docker libcontainer (now runc)
• OCI Image Spec
• How to represent image layers for OCI runtimes
• Based on Docker Image Manifest V2, Schema 2
• OCI Distribution Spec
• How to distribute OCI images
• Based on Docker Registry HTTP API

15 
Image layout
/blobs/sha256/e692418e...
/blobs/sha256/b5b2b2c5...
/blobs/sha256/61be55a8...
/blobs/sha256/3c3a4604...
JSON
JSON
tar.gz
tar.gz
tar.gz
Manifest
• Merkle DAG structure ensures reproducibility of
docker pull foo@sha256:e692418e…
Container Conﬁg
AUFS layer archives
(for each Dockerﬁle
FROM and RUN)
v1.0Manifest list latest

16 
Image layout
latest
amd64
JSON
JSON
tar.gz
tar.gz
tar.gz
JSON
Manifest list
Manifest
• Supports multi-arch (use BuildKit to build)
Container Conﬁg
latest
arm64
AUFS layer archives
FROM and RUN)

17 
Image layout
latest
Ice Lake
JSON
JSON
tar.gz
tar.gz
tar.gz
JSON
Manifest list
Manifest
• And even multi-microarchitectures via qnib/metahub
• https://metahub.qnib.org
Container Conﬁg
latest
Broadwell
Tesla M60
AUFS layer archives
FROM and RUN)

18 
Post-OCI image format?
• Issues of current OCI v1
• Too coarse deduplication granularity
• Containers cannot be started until the entire image is pulled
• An alternative: CernVM-FS
• Supports ﬁle-level deduplication rather than layer-level
• Files are lazy-pulled on demand using FUSE
• Integrating CernVM-FS to containerd is under discussion
https://github.com/containerd/containerd/issues/2943

19 
Post-OCI image format?
• ”OCI v2” https://github.com/openSUSE/umoci/issues/256
• Much ﬁner deduplication granularity
• No implementation yet
• Container Registry Filesystem https://github.com/google/crfs
• Focus on lazy-pulling CI images
• IPCS https://github.com/hinshun/ipcs
• IPFS integration for containerd

ISC HPCW talks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to ISC HPCW talks

Similar to ISC HPCW talks (20)

More from Akihiro Suda

More from Akihiro Suda (20)

Recently uploaded

Recently uploaded (20)

ISC HPCW talks