Containers in depth – Understanding how containers work to better work with containers

2
© 2018 Brent Laster@BrentCLaster
2
Containers in Depth
Understanding how containers work to work
better with containers
Brent Laster

3
3
@BrentCLaster
About me
§ R&D Director
§ Global trainer – training (Git, Jenkins, Gradle, Gerrit,
Continuous Pipelines, Containers, Kubernetes, Helm,
Istio)
§ Author -
§ OpenSource.com
§ Professional Git book
§ Jenkins 2 – Up and Running book
§ Continuous Integration vs. Continuous Delivery vs.
Continuous Deployment mini-book on Safari
§ https://www.linkedin.com/in/brentlaster
§ @BrentCLaster

4
4
@BrentCLaster
Book – Professional Git
§ Extensive Git reference, explanations, and
examples
§ First part for non-technical
§ Beginner and advanced reference
§ Hands-on labs

5
5
@BrentCLaster
Jenkins 2 Book
§ Jenkins 2 – Up and Running
§ “It’s an ideal book for those who are new to
CI/CD, as well as those who have been
using Jenkins for many years. This book will
help you discover and rediscover Jenkins.”
By Kohsuke Kawaguchi, Creator of Jenkins

6
6
@BrentCLaster
Agenda
§ What containers are and the benefits they provide
§ How containers are constructed
§ The differences between layers, images, and containers
§ What does immutability really mean
§ The core Linux functionalities that containers are based on
§ How containers reuse code
§ Storage drivers for Dockers
§ Overlay2 storage drivers
§ The Open Container Initiative

7
7
What’s in a container?
Settings
AppRuntimeDependencies
System ToolsSystem Libraries
• A container is a standard
unit of software that
functions like a fully
provisioned machine
installed with all the
software needed to run an
application.
• It’s a way of packaging
software so that
applications and their
dependencies have a self-
contained environment to
run in, are insulated from
the host OS and other
applications - and are
easily ported to other
environments.
• A container is NOT a VM.
A container leverages
several features of the
Linux OS to “carve out” a
self-contained space to
run in. Containers are running instances of images.
Images define what goes into a container.
Containers are built from images.
What are Containers?

8
Benefits of Containers
§ Easy to create and deploy once image is defined.
§ Best practices of continuous development, integration, and deployment allow for
frequent and reliable builds and deployments.
§ Quick/easy rollbacks due to image immutability.
§ Created at build/release time instead of at deployment time – so applications are
decoupled from the infrastructure.
§ Provide observability through application health and signals – not just from the
OS.
§ Because environment is self-contained, runs the same on all infrastructure –
laptop, desktop, cloud, etc.
§ Portable across any OS or cloud that supports containers.
§ Manage application instead of infrastructure
§ Allows applications such as microservices to be loosely coupled, distributed, and
managed and deployed dynamically. Removes the need for monolithic stacks.
§ Resource isolation and (hopefully) effective utilization.
(Paraphrased from: https://kubernetes.io/docs/concepts/overview/what-is-
kubernetes/)

9
What is a container image?
§ A container image is a read-only template used to create a
container.
§ Container images are like a snapshot of a container.
§ Images are stored in a “registry” – like repository for images.
§ Images are generally considered to be “immutable”. That is,
once they are created, the image should not be changed. If a
change is needed, a new image should be created.
§ Immutability does not imply that containers can’t be changed
by configuration or environment, etc.
§ Container images are built up from layers.
§ The docker “build” command is used to create images.

10
What is an image layer?
§ A Docker image is built up from a
series of layers.
§ Each layer results from an instruction
(modification) in the Dockerfile.
§ Layers are “stacked” on top of ones
before.
§ Each layer is only the set of
differences from the last one.
§ Think of a base layer as being an OS
image.
§ Each layer is R/O except for last one.
§ Last/top layer is “container layer”.
§ Container layer is thin R/W layer.
§ All changes made to running container
go in that layer.
§ From https://docs.docker.com/storage/storagedriver/

11
Creating and tracking layers to images
3f00bed1e02a | COPY dir:6a0…
ba16edb35eb9 | mysql:5.5.45
24f4da862742 | ENTRYPOINT
["/ent……
19e729dca1ee | CMD ["mysqld"]
* ”<missing>” in history means layers are built on a
different system and not available locally

12
How do layers & images intersect with the
operating system?
§ Layers can be viewed as single union of all
filesystems - Linux’s Union File System allows
you to stack different file systems and look
through
§ R/O files can be modified via Copy on Write (cow)
– pulling them up to R/W layer when needed
§ Other Linux facilities like cgroups and
namespaces help us keep our containers
separate
ccae61fe0274
read-only
}read
write
UNION FILE
SYSTEM
SERVICE
UNION
FILE
SYSTEM
• Layers are basically like files to the O/S
• Image layers are R/O (aka – “immutable”)
• We create a container from an image via the run command. Adds
another layer – a R/W one – changes can be made here (“mutable”)

13
Managing Content Across Containers
§ Major difference between a container
and an image is the top writeable
layer.
§ All writes (new content or modifying
content) are stored in the writable
layer.
§ So multiple containers can share
access to the same underlying image
and yet have their own data.
§ Layering also simplifies
rebuilding/updating images – only
layers that changed need to be
updated
§ Layers stored locally on disk (usually
/var/lib/docker)
§ Docker uses storage drivers to
manage the contents of the image
layers and the writeable layer
§ From https://docs.docker.com/storage/storagedriver/

14
What is Docker?
§ (Marketing) From docker.com
An open platform for distributed applications for developers and
sysadmins.
Open source project to ship any app as a lightweight
container
§ (Technical)
Thin wrapper around Linux Container technology
Leverages 3 functionalities from Linux to provide isolated
environment in the OS
» union filesystem - data
» namespaces - visibility (pid)
» cgroups - control groups (resources)
§ Provides restful interface for service
§ Provides description format for containers
§ Provides API for orchestration

15
cgroup
cgroups (control groups)
§ Groups processes for the purpose of system resource management
§ Limits applications to specific set of system resources
§ Feature of Linux kernel to track, isolate, and limit resource usage for a set of
processes
§ Leveraged by Docker engine to share system resources among containers (and
constrain them)
§ Resources include network, memory, I/O, etc.
System Resources
CPU Memory I/O Network

16
namespaces
(credit: https://access.redhat.com/documentation/en-
us/red_hat_enterprise_linux_atomic_host/7/html/overview_of_containers_in_red_hat_systems/introduction_to_linu
x_containers)
§ Provides isolated instance of a global resource
§ Appears as separate instance of resources within namespace
PID
(process IDs)
process
identifiers,
lists of
processes
Namespace
A
Namespace
B
Network
network
interfaces
Namespace
A
Namespace
B
Mount
file system
mount points
(R/O, etc.)
Namespace
A
Namespace
B
IPC
(inter-process
communication)
access to inter-
process
communication
Namespace
A
Namespace
B
User
Isolates
userids
Namespace
A
Namespace
B
UTS
(Unix
Timesharing
service)
kernel and
version
identifiers
Namespace
A
Namespace
B
Isolates the set of file system mount points seen by a group of processes so that processes in
different namespaces can have different views of the file system hierarchy. For example,
each containers can have its own /tmp or /var directory.Isolates two system identifiers - nodename and domainname per calls to uname(). Allows
each container to have its own hostname and NIS domain name.Isolates certain interprocess communication (IPC) resources, such as System V IPC objects
and POSIX message queues. This means that two containers can create shared memory
segments and semaphores with the same name, but are not able to interact with other
containers memory segments or shared memory.
Allows processes in different containers to have the same PID. Each container has its own
PID1 process to manage initialization tasks as well as its own /proc directory.
Provides isolation of network controllers, system resources associated with networking,
firewall and routing tables. This allows container to use separate virtual network stack,
loopback device and process space.
Allows specifying a range of host UIDs dedicated to the container. Consequently, a process
can have full root privileges for operations inside the container, and at the same time be
unprivileged for operations outside the container.

17
What is a Dockerfile?
§ Way to create a Docker
image ( a plan)
§ A text file that contains (in
order) all of the instructions
to build an image
§ Has a specific format and
structure
§ Each instruction in a
Dockerfile can create a
read-only layer

18
18
@BrentCLaster
Dockerfile to Image to Container
Build Run

19
Docker Commands

20
20
How do we think about this?
§ Consider analogy of installing software on a
machine…
§ And then provisioning systems for users
User R/W
User R/W

21
21
Diving Deeper into Layers

22
Containers can reuse layers
§ Docker commands can reuse layers if they already exist
in the docker file structure
$ docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
89d9c30c1d48: Pull complete
Digest: sha256:c19173c5ada610a5989151111163d28a67368362762534d8a8121ce95cf2bd5a
Status: Downloaded newer image for alpine:latest
$ docker pull alpine/helm
latest: Pulling from alpine/helm
89d9c30c1d48: Already exists
c14a8f06d505: Pull complete
7de226a1041c: Pull complete
Digest: sha256:ae3be4cdbaf5c6ca5f88c4a28c888fe99dcb1416de41483a2554e9bccec08503
Status: Downloaded newer image for alpine/helm:latest

23
Mapping layers to the file system (before 1.10)
(credit: winsock.io/explaining docker-image-ids/)
§ Layer – alternative definition
§ A layer is a delta or “diff” that results when commands are run during the
image build process that produce new or changed files/directories
§ The new or changed files and directories are stored or ‘committed’ as a
new layer
§ Prior to Docker 1.10, images and layers were synonymous
ID: fdf9d70e7ba7
Parent: dc89c90bfea
Name: alpine/helm:latest
ID: dc89c90bfea
Parent: ead53b4e9ff
Name: “ ”
ID: ead53b4e9ff
Parent: “ “
Name: “ ”
/var/lib/docker/aufs/diff
fdf9d70e7ba7 …
dc89c90bfea …
ead53b4e9ff …
Image Layers
$ docker inspect alpine/helm:latest
[
{
“Id”: “fdf9d70e7ba72e3a74b
“Parent”: “dc89c90bfea235
…
…
}
]
Inspect

24
Mapping layers to the file system (1.10+)
§ Layers equal to image had challenges
§ Security – how to tell if image tampered with during
push/pull, etc.
§ New approach – “Content Addressable IDs”
§ Layers have no affiliation or notion of any image
» Just collections of directories and files
§ Layers identified by digest of the form “algorithm:hex-
value”
» Algorithm is method for computing hex value from
layer’s contents (hashing method)
» Hex-value is string computed from applying
algorithm to layer’s contents (hash)
» Example:
sha256:77cae8ab23bf486355d1b31912597053…
§ Docker image has
» Configuration object – includes ordered list of layer
digests (used to assemble filesystem in container
instead of series of parent images)
» Image ID is digest – computed hash of
configuration object
$ docker inspect alpine/helm:latest
[
{
"Id": "sha256:fdf9d70e7ba76a6af25843
…
"Layers": [
"sha256:77cae8ab23bf486355d1b3
"sha256:fad447c6845e910b04e9391
"sha256:fcaa7782b133863c3ed0f68
]
…
}
]
Inspect

25
Getting Info about Docker Storage
§ docker info provides basic information about configuration
§ Output shows storage driver and docker root directory
diyuser3@training1:~$ docker info
Containers: 111
Running: 98
Paused: 0
Stopped: 13
Images: 32
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
…
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 9.529GiB
Name: training1
ID:
SLSD:QYJ3:YRX4:36UG:FUY3:F7CP:5K55:G34L:SBNC:FBR2:
7CDB:FVI4
Docker Root Dir: /var/lib/docker

26
Docker on the Mac
§ Docker not compatible with macOS
§ Uses Hyperkit and virtual image
§ Located at
~/Library/Containers/com.docker.docker/Data/vms/0
§ Inside of virtual machine, Docker root is /var/lib/docker

27
Docker on Windows
§ Multiple approaches
§ Native Windows containers
§ Linux containers
» Run in simple Hyper-V based virtual environment
» Config and info about virtual image to run stored in default
Docker root on Windows – C:ProgramDataDockerDesktop
» Within image, uses expected location

28
Root Folder
§ Contains data on containers, builds, networks, volumes,
etc.
$ sudo ls -la /var/lib/docker
total 136
drwx--x--x 15 root root 4096 Oct 12 22:45 .
drwxr-xr-x 72 root root 4096 Jul 5 21:13 ..
drwx------ 2 root root 4096 Jun 3 2019 builder
drwx------ 4 root root 4096 Jun 3 2019 buildkit
drwx------ 202 root root 28672 Oct 12 22:46 containers
drwx------ 3 root root 4096 Jun 28 2019 image
drwxr-x--- 3 root root 4096 Jun 3 2019 network
drwx------ 353 root root 49152 Oct 12 22:46 overlay2
drwx------ 4 root root 4096 Jun 3 2019 plugins
drwxr-xr-x 3 root root 4096 Jul 11 2019 registry
drwx------ 2 root root 4096 Oct 12 22:45 runtimes
drwx------ 2 root root 4096 Jun 3 2019 swarm
drwx------ 2 root root 4096 Oct 12 22:45 tmp
drwx------ 2 root root 4096 Jun 3 2019 trust
drwx------ 6 root root 4096 Oct 12 22:45 volumes

29
Storage Drivers (aka Graphdrivers)
§ A local instance of a Docker engine has cache of image
layers (pieces put together to form an image)
§ Layers come from docker pull, docker build, etc.
§ Driver is required to manage the layers at runtime
§ Specifically to mount the layers into a consolidated root
filesystem – creates the mount namespace of the
container
§ Creates a kind of “image graph” that is the relationship
between the various layers

30
Storage Driver Examples
Storage Driver Underlying Technology Notes
aufs AUFS (another/advanced union filesystem) –
union filesystem
Oldest and most mature
fast to startup and efficient with resources; available
in certain Debian distros
Rejected for merging into mainline Linux
Previous default before overlay
btrfs Btrfs (b-tree filesystem) - snapshots Performance impacts and higher memory usage
Easier management tooling/more dense storage
Questionable support often characterized as buggy
devicemapper Device Mapper Operates on block device where Docker is
Creates sparse files
Needs to be combined with other tooling for
production use
Former default for Centos / RHEL – before overlay2
overlay OverlayFS (overlay filesystem) – union
filesystem
Faster and simpler than aufs
Can only run on top of ext4/xfs filesystems
Can cause heavy inode* usage due to file copies
overlay2 OverlayFS (overlay filesystem) - union
filesystem
Same as overlay generally
Improvements to reduce inode usage
vfs VFS – (not a filesystem) Simple approach - makes copies of layers; no union
fs and no CoW
Lower performance and more space on disk
Robust, stable –useful for docker dev/testing
zfs ZFS (filesystem) - snapshots Generally same as btrfs
* inode – contains metadata for each file on filesystem; max count is hardcoded

31
Storage Driver Tech
§ Union mount filesystem (overlay) – combines multiple
mount points into one – layers directories on top of each
other to look like one – like PATH variable
§ Snapshot – R/O point-in-time copy of filesystem or
volume – created quickly but consume more space over
time
§ Device mapper – framework for disk volume
management; used to map physical blocks to higher-
level virtual blocks (created logical disk volumes from
blocks on disk)

32
AUFS Model
§ Oldest, most mature
§ Preferred for Docker 18.06 and earlier on Debian where
no support for overlay2
Docker Concepts OverlayFS Concepts
Container Layer
Image Layer 2
Image Layer 1 Branch 2
Branch 1
Union Mountfile a file b
file a
file b
file c
file d
file c file d .wh
file b
Docker Concepts
Image Base Layer Branch 3
aufs Concepts
file d

33
Overlay Model
§ Only has a single lower OverlayFS layer
§ Requires hard links for implementation of multi-layered
images
§ Deprecated in Docker 18.09
Container mount
Container layer
Image layer “lowerdir”
“upperdir”
“merged”file a file b
file a file b
file c
file d
file c file d
file b

34
Overlay2 Model
§ Natively supports multiple OverlayFS layers (up to
around 128)
Container Layer
Image Layer 2
Image Layer 1 Lower Dir 1
Upper Dir
Merged Viewfile a file b
file a
file b
file c
file d
file c file d
file b
Docker Concepts
Image Base Layer Lower Dir 2
OverlayFS Concepts
file c .wh

35
Devicemapper Model
§ Manages at level of storage blocks (not file based)
§ Deprecated as of 18.09
Container Layer
Image Layer 1
Image Base Layer Snapshot 1
Snapshot 2
Snapshot 3block a block b block c
Docker Concepts
Device
Device Mapper
Concepts

36
Btrfs/zfs Model
§ Uses filesystem constructs - snapshots
Container Layer
Image Layer 2
Image Layer 1 Snapshot 1
Snapshot 2
Snapshot 3
file a
file b
file c
file d
file b
Docker Concepts
Image Base Layer Subvolume
btrfs/zfs Concepts
file bfile a file c

37
37
Diving Deeper into Overlay2 Storage Driver

38
Overlay Filesystem Directories
§ LowerDir
§ R/O image layers
§ Separated by “:” from top-most to bottom-most
§ UpperDir
§ R/W layer where changes are written
§ MergedDir
§ Merged view of all the layers
§ WorkDir
§ Used specifically by the underlying Linux OverlayFS
§ Standard OS functionality if OverlayFS in kernel
$ sudo mount -t overlay -o lowerdir=base,upperdir=diff,workdir=workdir overlay
overlay
$ mount | grep overlay
overlay on /home/diyuser3/overlay type overlay
(rw,relatime,lowerdir=base,upperdir=diff,workdir=workdir)

39
What happens when we pull an image?
$ docker inspect 914 | jq’ .[0].GraphDriver.Data'
{
"LowerDir":
"/var/lib/docker/overlay2/723678378c68c9c8d5e8e0fa7ab
37f60f560cfd36dc0e04d5d12b9a9d253af36/diff:/var/lib/d
ocker/overlay2/db17df65f2091be31103217310a3e75cdf955f
bc94d61d70cca00ef9fb179c69/diff",
"MergedDir":
"/var/lib/docker/overlay2/4d4aa3ddd85e56e583e62122544
55d40e84b283c6d72e17723a026bbd1e457d4/merged",
"UpperDir":
55d40e84b283c6d72e17723a026bbd1e457d4/diff",
"WorkDir":
55d40e84b283c6d72e17723a026bbd1e457d4/work"
}
$ docker pull ubuntu
latest: Pulling from library/ubuntu
d72e567cc804: Pull complete
0f3630e5ff08: Pull complete
b6a83d81d1f4: Pull complete
Digest:
sha256:bc2f7250f69267c9c6b66d7b6a81a54d
3878bb85f1ebb5f951c896d13e6ba537
Status: Downloaded newer image for
ubuntu:latest
$ docker inspect 914 | jq '.[0].RootFS'
{
"Type": "layers",
"Layers": [
"sha256:d42a4fdf4b2ae8662ff2ca1b695eae571c6
52a62973c1beb81a296a4f4263d92",
"sha256:90ac32a0d9ab11e7745283f3051e9900546
16d631812ac63e324c1a36d2677f5",
"sha256:782f5f011ddaf2a0bfd38cc2ccabd634095
d6e35c8034302d788423f486bb177"
]
}

40
What’s in /var/lib/docker/overlay2 ?
$ sudo ls -l /var/lib/docker/overlay2 | grep -v init
total 52
drwx------ 4 root root 4096 Oct 17 15:54
4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4
723678378c68c9c8d5e8e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36
db17df65f2091be31103217310a3e75cdf955fbc94d61d70cca00ef9fb179c69
drwx------ 2 root root 28672 Oct 17 15:54 l
$ sudo ls -l /var/lib/docker/overlay2/l | grep -v init
total 20
lrwxrwxrwx 1 root root 72 Oct 17 15:54 4AMPGY4G3AEDPZKVDRMZFKXWQK ->
../4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4/diff
lrwxrwxrwx 1 root root 72 Oct 17 15:54 LUZL4AJULCXE3BKOC5VXIO4LWF ->
../db17df65f2091be31103217310a3e75cdf955fbc94d61d70cca00ef9fb179c69/diff
lrwxrwxrwx 1 root root 72 Oct 17 15:54 QUY32BQAOJGFMP4CJCSQLPYAHS ->
../723678378c68c9c8d5e8e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36/diff
• High-level directory per layer
• l (lowercase L) directory contains shortened layer identifiers as symbolic
links
• Used to avoid hitting page size limit on args to the mount command

41
Looking at layers #1 (LowerDir)
§ Lowest layer (LowerDir)
§ List, separated by ‘:”
§ Ordered from highest to lowest
§ Contains the link to the main directory and the "diff" directory that has the contents
"LowerDir":
"/var/lib/docker/overlay2/723678378c68c9c8d5e8e0fa7ab37f60f560cfd36dc0e04d5d12b9a9
d253af36/diff:/var/lib/docker/overlay2/db17df65f2091be31103217310a3e75cdf955fbc94d
61d70cca00ef9fb179c69/diff",
$ sudo ls
/var/lib/docker/overlay2/db17df65f2091be31103217310a3e
75cdf955fbc94d61d70cca00ef9fb179c69
diff link
$ sudo ls
75cdf955fbc94d61d70cca00ef9fb179c69/diff
bin dev home lib32 libx32 mnt proc
run srv tmp var
boot etc lib lib64 media opt root
sbin sys usr
$ sudo cat
75cdf955fbc94d61d70cca00ef9fb179c69/link
LUZL4AJULCXE3BKOC5VXIO4LWF
Lower Dir 2
bin dev home
run srv tmp …
diff
LUZL4A…->
link
Image Base Layer

42
Looking at layers #2 (LowerDir)
§ Next layer contains
§ a "lower" file that stores the layer composition info (pointing to the other layer)
§ its link
§ the diff containing the contents,
§ a work area
"LowerDir":
"/var/lib/docker/overlay2/723678378c68c9c8d5e8e0fa7ab37f60f560cfd36dc0e04d5d12b9a9
d253af36/diff:/var/lib/docker/overlay2/db17df65f2091be31103217310a3e75cdf955fbc94d
61d70cca00ef9fb179c69/diff",
Lower Dir 2
LUZL4A…->
link
Image Base Layer
Lower Dir 1
QUY32…->
link
$ sudo ls
/var/lib/docker/overlay2/723678378c68c9c8d5e8
e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36
diff link lower work
diyuser3@training1:~$ sudo ls
e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36/
diff
etc usr var
diyuser3@training1:~$ sudo cat
e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36/
link
QUY32BQAOJGFMP4CJCSQLPYAHSd
sudo cat
4d5d12b9a9d253af36/lower
l/LUZL4AJULCXE3BKOC5VXIO4LWF
bin dev home
run srv tmp …
diff
etc
usr
var
diff
l/LUZL4A…
lower work
Image Layer 1

43
Looking at layers #3 (UpperDir)
§ Upper dir / layer contains
§ a "lower" file that stores the layer composition info (pointing to the other layer)
§ its link
§ the diff containing the contents
§ the merged view
§ the work area
$ sudo ls
/var/lib/docker/overlay2/4d4aa3ddd85e56e583e6212
254455d40e84b283c6d72e17723a026bbd1e457d4
diff link lower work
$ sudo ls
254455d40e84b283c6d72e17723a026bbd1e457d4/diff
run
$ sudo cat
254455d40e84b283c6d72e17723a026bbd1e457d4/lower
l/QUY32BQAOJGFMP4CJCSQLPYAHS:l/LUZL4AJULCXE3BKOC
$ sudo cat
254455d40e84b283c6d72e17723a026bbd1e457d4/link
4AMPGY4G3AEDPZKVDRMZFKXWQK
"MergedDir":
"/var/lib/docker/overlay2/4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4/merged",
"UpperDir":
"/var/lib/docker/overlay2/4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4/diff",
"WorkDir":
"/var/lib/docker/overlay2/4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4/work"
Lower Dir 2
LUZL4A…->
link
Image Base Layer
bin dev home
run srv tmp …
diff
Lower Dir 1
QUY32…->
link
etc
usr
var
diff
l/LUZL4A…
lower work
Image Layer 1
Upper Dir
4AMPG…->
link
run
diff
l/QUY32…
lower work
Image Layer 2

44
Mapping History to Filesystem
§ Note that if we do a history, we see a "mkdir -p /run/systemd" in lower layer
§ Then if we look at the "diff" of the second layer, we see the run/systemd structure
$ docker history 914
IMAGE CREATED CREATED BY SIZE
9140108b62dc 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 3 weeks ago /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> 3 weeks ago /bin/sh -c [ -z "$(apt-get indextargets)" ] 0B
<missing> 3 weeks ago /bin/sh -c set -xe && echo '#!/bin/sh' > /… 811B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:da80f59399481ffc3… 72.9MB
sudo ls -R
/var/lib/docker/overlay2/4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4
/diff
/diff:
run
/diff/run:
systemd
/diff/run/systemd:
container
Doc

45
What happens when we create a container
§ Use docker run to start a container
§ If we look in overlay2, we’ll see a new layer
§ Now suppose we create a new file…
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 9140108b62dc 3 weeks ago 72.9MB
$ docker run -it ubuntu
root@97cc42aa7e75:/#
$ sudo ls /var/lib/docker/overlay2 | grep -v init
4d4aa3ddd85e56e583e6212254455d40e84b283c6d72e17723a026bbd1e457d4
723678378c68c9c8d5e8e0fa7ab37f60f560cfd36dc0e04d5d12b9a9d253af36
b5065bbe44006efc14a1c4cfaa3c284f36da69874ca66a12cf0f7d89562c6957
db17df65f2091be31103217310a3e75cdf955fbc94d61d70cca00ef9fb179c69
root@97cc42aa7e75:/# echo "stuff" > file1

46
Contents of container layer
§ If we look in the new layer we see it has “diff” and “merged”
§ Diff has the file we created
§ Merged has the merged content
~$ sudo ls
/var/lib/docker/overlay2/b5065bbe44006efc14a1c4cfaa3c284f36da69874ca66a12cf0f7d8
9562c6957
diff link lower merged work
~$ sudo ls
9562c6957/diff
file1
~$ sudo ls
9562c6957/merged
bin dev file1 lib lib64 media opt root sbin sys usr
boot etc home lib32 libx32 mnt proc run srv tmp var

47
Contents of container directory
§ Containers info
$ sudo ls -la
/var/lib/docker/containers/97cc42aa7e75ff1aea9da4d90e75dfc3aceb8a3499e50284f1
4ed9b0f1dd8f60
total 72
drwx------ 4 root root 4096 Oct 17 16:52 .
drwx------ 92 root root 28672 Oct 17 16:47 ..
-rw-r----- 1 root root 578 Oct 17 16:52
97cc42aa7e75ff1aea9da4d90e75dfc3aceb8a3499e50284f14ed9b0f1dd8f60-json.log
drwx------ 2 root root 4096 Oct 17 16:38 checkpoints
-rw------- 1 root root 2557 Oct 17 16:52 config.v2.json
-rw-r--r-- 1 root root 1421 Oct 17 16:52 hostconfig.json
-rw-r--r-- 1 root root 13 Oct 17 16:38 hostname
-rw-r--r-- 1 root root 174 Oct 17 16:38 hosts
drwx------ 3 root root 4096 Oct 17 16:38 mounts
-rw-r--r-- 1 root root 589 Oct 17 16:38 resolv.conf
-rw-r--r-- 1 root root 71 Oct 17 16:38 resolv.conf.hash

48
Open Container Initiative (OCI)

49
Other Players in the Container Space

50
50
That’s all - thanks!

Containers in depth – Understanding how containers work to better work with containers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Containers in depth – Understanding how containers work to better work with containers

Similar to Containers in depth – Understanding how containers work to better work with containers (20)

More from All Things Open

More from All Things Open (20)

Recently uploaded

Recently uploaded (20)

Containers in depth – Understanding how containers work to better work with containers