Machine Learning Software Engineering Patterns and Their Engineering
KubeCon NA 2020ふりかえり:「“Speeding Up Analysis Pipelines with Remote Container Images” - Ricardo Rocha & Spyridon Trigazis, CERN 」
1. Copyright(c)2020 NTT Corp. All Rights Reserved
“Speeding Up Analysis Pipelines with Remote Container Images”
- Ricardo Rocha & Spyridon Trigazis, CERN
2020/12/22
Kubernetes Meetup Tokyo #37 KubeCon+CloudNativeCon NA 2020
2. Copyright(c)2020 NTT Corp. All Rights Reserved
GitHub:@ktock / Twitter:@TokunagaKohei
Container Runtime Meetup 2
LT
https://runtime.connpass.com/event/198071/
3. Copyright(c)2020 NTT Corp. All Rights Reserved
CERN Grid Computing
l LHC(Large Hadron Collider; )
50 70 /
l 200 site 100
l
3 https://sched.co/ekDj
CERN
6. Copyright(c)2020 NTT Corp. All Rights Reserved
containerd
kubectl apply
pull
Pod
CRI
pull OCI
Docker, containerd, CRI-O
runc, gVisor, Kata Containers
OCI
l CNCF graduated
l GKE[1] AWS Fargate[2] AKS(preview)[3] IKS[4]
l CRI CRI Kubernetes
l 6
• lazy pulling
[1] https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd ; [2] https://aws.amazon.com/jp/blogs/containers/under-the-hood-fargate-data-plane/ ; [3] https://azure.microsoft.com/ja-
jp/updates/azure-kubernetes-service-aks-support-for-containerd-runtime-is-in-preview/ ; [4] https://www.ibm.com/cloud/blog/ibm-cloud-kubernetes-service-supports-containerd
apiserver
kubelet
7. Copyright(c)2020 NTT Corp. All Rights Reserved
eStargz Lazy Pulling
l containerd Stargz Snapshotter
l eStargz lazy pulling 7 ( )
l containerd eStargz Lazy pulling ( )
https://sched.co/ekDj
l Pull
l
l
l eStargz 45
Lazy Pulling Pull
https://github.com/containerd/stargz-snapshotter
8. Copyright(c)2020 NTT Corp. All Rights Reserved
containerd eStargz lazy pulling
l eStargz 8 push
l containerd Stargz Snapshotter
l pull
• eStargz pull
• 8 NW lazy pulling 8
Stargz
Snapshotter
rootfs FUSE
Lazy
pull
eStargz
pull
9. Copyright(c)2020 NTT Corp. All Rights Reserved
CERN lazy pulling
l
l CERN eStargz
Ø lazy pull
l site eStargz
Ø site site 9lazy pulling
https://youtu.be/j4eIgdDkI9I?t=1325
https://sched.co/ekDj
10. Copyright(c)2020 NTT Corp. All Rights Reserved
( GB )
https://sched.co/ekDj
l pull
l Pull NW /
l 1 0
Stargz
Snapshotter
Kubelet, etc
OCI
Container Registry
lazypull
eStargz
images
eStargz Stargz Snapshotter
lazy pulling
l eStargz lazy pulling
l Stargz Snapshotter 0
containerd lazy pulling
l site
CERN
1
https://github.com/containerd/stargz-snapshotter
11. Copyright(c)2020 NTT Corp. All Rights Reserved
(Kubernetes ) BuildKit base image lazy pull
FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev
COPY ./hello.go /hello.go
RUN go build -o hello /hello.go
lazy
pull
• /usr/local/go/bin/go
• /usr/local/go/src/fmt/…
etc...
Pull COPY RUN
golang:1.15.3-esgz pull
eStargz
l BuildKit v0.8.0 eStargz lazy pulling experimental
l eStargz BuildKit lazy pull
Ø FROM COPY RUN pull
l CI 1
https://medium.com/nttlabs/buildkit-lazypull-66c37690963f