SlideShare a Scribd company logo
1 of 32
Download to read offline
Kubernetes “Node”
Past, now, and the future
by Harry Zhang @resouer
What is “Node”?
Where the
container lives
Kubernetes cluster is bootstrapped
container centric features are
implemented
docker/rkt/runV/runC is plugged in
networking is implemented
volume is enabled
sig-node
Why “Node” Matters?
Kubernetes
a bottom-up design of container
cloud
with special bonus from Google …
Node
Container
Control Panel
Pod
Replica
StatefulSet
Deployment
DaemonSet
Job
…
How Kubernetes is created?
?
Containers
Borg
Borg = engineer oriented deployment scheduling & management system
Google internal is massively using cgroups container, not Container :)
Kubernetes = re-innovate Borg with Container
Node
Unsung hero to bridge Borg
control panel with container
Kubelet Overview
kubelet
SyncLoop
kubelet
SyncLoop
proxy
proxy
1 Pod created
etcd
scheduler
api-server
Kubelet Overview
kubelet
SyncLoop
kubelet
SyncLoop
proxy
proxy
2 Pod object added
etcd
scheduler
api-server
Kubelet Overview
kubelet
SyncLoop
kubelet
SyncLoop
proxy
proxy
3.1 New pod object detected
3.2 Bind pod with node
etcd
scheduler
api-server
Kubelet Overview
kubelet
SyncLoop
kubelet
SyncLoop
proxy
proxy
4.1 Detected pod bind with me
4.2 Start containers in pod
etcd
scheduler
api-server
Pod
“Alloc” in Borg
The atomic scheduling unit in Kubernetes
Process group in container cloud
Implemented in Node
But why?
Are You Using Container Like This?
1.use supervised/systemd to manage multi-apps in one container
2.ensure container order by tricky scripts
3.add health check for micro-service group
4.copy files from one container to another
5.connect to peer container across whole network stack
6.schedule super affinity containers in cluster
Multiple containers
Multiple Apps in One Container
Master Pod
kube-apiserver
kube-scheduler
controller-manager
InitContainer
Ensure Container Order
Health Check for Containers
Liveness probe
Pod will be reported
as Unhealthy
Master Pod
kube-apiserver
kube-scheduler
controller-manager
Copy Files from One to Another
Pod volumes is shared
Master Pod
kube-apiserver
kube-scheduler
controller-manager
/etc/kubernetes/ssl
Connect to Peer Container
Pod network is shared
Master Pod
kube-apiserver
kube-scheduler
controller-manager
network namespace
Pod is atomic scheduling unit
• controller super affinity apiserver
• Request:
• controller: 1G, apiserver: 0.5G
• Available:
• Node_A: 1.25G, Node_B: 2G
• What happens if controller is scheduled to
Node_A first?
Schedule Super Affinity Containers
So, this is Pod
Design pattern in container world
decoupling
reuse & refactoring
describe more complex workload by container
e.g. ML
kubelet
register listers
diskSpaceManager
oomWatcher
InitNetworkPlugin
chooseRuntime
(build-in, remote)
InitNetworkPlugin
NewGenericPLEG
NewContainerGC
AddPodAdmitHandler
HandlePods
{Add, Update, Remove, Delete, …}
NodeStatus
Network
Status
status
Manager
PLEG
SyncLoop
<-chan kubetypes.PodUpdate*
<-chan *pleg.PodLifecycleEvent
periodic sync events
housekeeping events
Pod Update Worker (e.g.ADD)
• generale pod status
• check volume status
• call runtime to start containers
volume
Manager
PodUpdate
*4-sources
api-server (primary, watch)
http endpoint (pull)
http server (push)
file (pull)
image
Manager
Eviction
Prepare Volume
Volume
Manager
desired
World
reconcile
Find new pods
createVolumeSpec(newPod)
Cache volumes[volName].pods[podName] = pod
• Get mountedVolume from actualStateOfWorld
• Unmount volumes in mountedVolume but not in
desiredStateOfWorld
• AttachVolume() if vol in desiredStateOfWorld and not
attached
• MountVolume() if vol in desiredStateOfWorld and not
in mountedVolume
• Verify devices that should be detached/unmounted are
detached/unmounted
• Tips:
1. -v host:path
2. attach VS mount
3. Totally independent from container
management
Eviction
Guaranteed
Be killed until they exceed their limits
or if the system is under memory pressure and there
are no lower priority containers that can be killed.
Burstable
killed once they exceed their requests and no Best-
Effort pods exist when system under memory pressure
Best-Effort
First to get killed if the system runs out of memory
Management
kubelet
CRI
Workloads
Orchestration
kubelet
SyncLoop
Scheduling api-server
Etcd
bind
pod, node list
pod
GenericRuntime
SyncPod
CRI grpc
dockershim
remote
(no-op)
Sandbox
Create
Delete
List
Container
Create
Start
Exec
Image
Pull
List
shim
client api
dockerd
runtime
pod
CRI Spec
CRI Runtime Shim
dockershim: docker
frakti: hypervisor container (runV)
cri-o: runC
rktlet: rkt
cri-containerd: containerd
NODE
Container Lifecycle
Pod foo
container
A
container
B
1. RunPodSandbox(foo)
Created Running Exitednull null
CreatContainer() StartContainer() StopContainer() RemoveContainer()
$ kubectl run foo …
A B foo
foo (hypervisor)
A B
2. CreatContainer(A)
3. StartContainert(A)
4. CreatContainer(B)
5. StartContainer(B) docker runtime hypervisor runtime
Streaming (old version)
kubelet becomes bottleneck
runtime shim in critical path
code duplication among runtimes/shims
kubectl apiserver kubelet runtime shim
1. kubectl exec -i
2. upgrade connection
3 stream api
Design Doc
Streaming (CRI version)
kubectl apiserver kubelet runtime shim
1. kubectl exec -i
2. upgrade connection
3. stream api
serving process
4. launch a http2 server
6. URL: <ip>:<port>
7. redirect responce
8. update connection
CRI
Design Doc
5. response
Streaming in Runtime Shim
kubelet
frakti
Streaming
Server
Runtime
apiserver
url of streaming server
CRI Exec()
url
Exec() request
$ kubectl exec …
"/exec/{token}"
stream resp runtime
exec api
Stream Runtime
Exec()
Attach()
PortForward()
CNI Network in Runtime Shim
Workflow in runtime shim (may vary from different runtimes):
1. Create a network NS for sandbox
2. plugin.SetUpPod(NS, podID) to configure this NS
3. Also checkpoint the NS path for future usage (TearDown)
4. Infra container join this network namespace
1. or scanning /etc/cni/net.d/xxx.conf to configure
sandbox
Pod
A
B
eth0
vethXXX
Physical Server
frakti
Mixed Runtimes: IaaS-less Kubernetes
Hypervisor container + Docker
Handled by:
https://github.com/kubernetes/frakti/
1.Share the same CNI network
2.Fast responsiveness (no VM host needed)
3.High resource efficiency (k8s QoS classes)
4.Mixed run micro-services & legacy application
1. independent kernel + hardware virtualization
2. high I/O performance + host namespace
hyper runtime
dockershim
CRI grpc
hypervisor
NFV
monitor
hypervisor
NFV
logger
docker
docker
docker
docker
docker
Node & Kubernetes is Moving Fast
GPU isolation
libnvidia-container is proposed
CRI enhancement
cri-containerd (promising default), cri-tools, hypervisor based secure container
CPU pin (and update) and NUMA affinity (CPU sensitive workloads)
HugePages support for large memory workloads
Local storage management (disk, blkio, quota)
“G on G”: run Google internal workloads on Google Kubernetes
Recently
Kubernetes
CRI enhancement, equivalence class scheduler (Borg), NodeController, StatefulSet
https://github.com/kubernetes/frakti (Secure container runtime in k8s)
Mentoring
cri-containerd, cri-tools
Unikernels & LinuxKit + k8s (Google Summer of Code 2017)
ovn-kubernetes (coming soon)
Newly started: Stackube
Stackube
https://github.com/openstack/stackube (Hypernetes v2)
100% upstream Kubernetes + OpenStack plugins + Mixed Runtime
A IaaS-less, multi-tenant, secure and production ready Kubernetes distro
Milestone: 2017.9

More Related Content

What's hot

Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container ServiceScaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container ServiceBen Hall
 
Kubernetes automation in production
Kubernetes automation in productionKubernetes automation in production
Kubernetes automation in productionPaul Bakker
 
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka Mario Ishara Fernando
 
Hands-On Introduction to Kubernetes at LISA17
Hands-On Introduction to Kubernetes at LISA17Hands-On Introduction to Kubernetes at LISA17
Hands-On Introduction to Kubernetes at LISA17Ryan Jarvinen
 
Kubernetes - introduction
Kubernetes - introductionKubernetes - introduction
Kubernetes - introductionSparkbit
 
Scaling Docker with Kubernetes
Scaling Docker with KubernetesScaling Docker with Kubernetes
Scaling Docker with KubernetesCarlos Sanchez
 
Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6Opcito Technologies
 
Docker and kubernetes
Docker and kubernetesDocker and kubernetes
Docker and kubernetesDongwon Kim
 
Docker and Kubernetes 101 workshop
Docker and Kubernetes 101 workshopDocker and Kubernetes 101 workshop
Docker and Kubernetes 101 workshopSathish VJ
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMartin Etmajer
 
What's new in Kubernetes
What's new in KubernetesWhat's new in Kubernetes
What's new in KubernetesDaniel Smith
 
Docker Madison, Introduction to Kubernetes
Docker Madison, Introduction to KubernetesDocker Madison, Introduction to Kubernetes
Docker Madison, Introduction to KubernetesTimothy St. Clair
 
Kubernetes 101 for Beginners
Kubernetes 101 for BeginnersKubernetes 101 for Beginners
Kubernetes 101 for BeginnersOktay Esgul
 

What's hot (20)

Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container ServiceScaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
 
Kubernetes automation in production
Kubernetes automation in productionKubernetes automation in production
Kubernetes automation in production
 
DevOps with Kubernetes
DevOps with KubernetesDevOps with Kubernetes
DevOps with Kubernetes
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
kubernetes for beginners
kubernetes for beginnerskubernetes for beginners
kubernetes for beginners
 
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
Microservices , Docker , CI/CD , Kubernetes Seminar - Sri Lanka
 
Hands-On Introduction to Kubernetes at LISA17
Hands-On Introduction to Kubernetes at LISA17Hands-On Introduction to Kubernetes at LISA17
Hands-On Introduction to Kubernetes at LISA17
 
Kubernetes - introduction
Kubernetes - introductionKubernetes - introduction
Kubernetes - introduction
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
Scaling Docker with Kubernetes
Scaling Docker with KubernetesScaling Docker with Kubernetes
Scaling Docker with Kubernetes
 
Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6Kubernetes Introduction & Whats new in Kubernetes 1.6
Kubernetes Introduction & Whats new in Kubernetes 1.6
 
Rex gke-clustree
Rex gke-clustreeRex gke-clustree
Rex gke-clustree
 
Docker and kubernetes
Docker and kubernetesDocker and kubernetes
Docker and kubernetes
 
Intro to Kubernetes
Intro to KubernetesIntro to Kubernetes
Intro to Kubernetes
 
Docker and Kubernetes 101 workshop
Docker and Kubernetes 101 workshopDocker and Kubernetes 101 workshop
Docker and Kubernetes 101 workshop
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on Kubernetes
 
What's new in Kubernetes
What's new in KubernetesWhat's new in Kubernetes
What's new in Kubernetes
 
Docker Madison, Introduction to Kubernetes
Docker Madison, Introduction to KubernetesDocker Madison, Introduction to Kubernetes
Docker Madison, Introduction to Kubernetes
 
Kubernetes 101 for Beginners
Kubernetes 101 for BeginnersKubernetes 101 for Beginners
Kubernetes 101 for Beginners
 
Docker Overview
Docker OverviewDocker Overview
Docker Overview
 

Similar to Kubernetes Node Deep Dive

Kubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKel Cecil
 
Kubernetes extensibility
Kubernetes extensibilityKubernetes extensibility
Kubernetes extensibilityDocker, Inc.
 
Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)HungWei Chiu
 
Container Deployment and Management with kubernetes
Container Deployment and Management with kubernetesContainer Deployment and Management with kubernetes
Container Deployment and Management with kubernetessiuyin
 
Lifecycle of a pod
Lifecycle of a podLifecycle of a pod
Lifecycle of a podHarshal Shah
 
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...Paris Open Source Summit
 
Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 introTerry Cho
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developerPaul Czarkowski
 
Learn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLearn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLarry Cai
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesPhil Estes
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingPiotr Perzyna
 
Pro2516 10 things about oracle and k8s.pptx-final
Pro2516   10 things about oracle and k8s.pptx-finalPro2516   10 things about oracle and k8s.pptx-final
Pro2516 10 things about oracle and k8s.pptx-finalMichel Schildmeijer
 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes IntroductionEric Gustafson
 
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...ssuser92b4be
 
containerD
containerDcontainerD
containerDstrikr .
 
Spark with kubernates
Spark with kubernatesSpark with kubernates
Spark with kubernatesDavid Tung
 
JDD2015: Kubernetes - Beyond the basics - Paul Bakker
JDD2015: Kubernetes - Beyond the basics - Paul BakkerJDD2015: Kubernetes - Beyond the basics - Paul Bakker
JDD2015: Kubernetes - Beyond the basics - Paul BakkerPROIDEA
 
Introduction to Kubernetes
Introduction to KubernetesIntroduction to Kubernetes
Introduction to KubernetesPaul Czarkowski
 
Managing Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayManaging Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayQiming Teng
 

Similar to Kubernetes Node Deep Dive (20)

Kubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of Containers
 
Kubernetes extensibility
Kubernetes extensibilityKubernetes extensibility
Kubernetes extensibility
 
Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)Build Your Own CaaS (Container as a Service)
Build Your Own CaaS (Container as a Service)
 
Container Deployment and Management with kubernetes
Container Deployment and Management with kubernetesContainer Deployment and Management with kubernetes
Container Deployment and Management with kubernetes
 
Lifecycle of a pod
Lifecycle of a podLifecycle of a pod
Lifecycle of a pod
 
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...
Keynote #Tech - Google : aperçu de la gestion des services distribués chez Go...
 
Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 intro
 
Kubernetes for the PHP developer
Kubernetes for the PHP developerKubernetes for the PHP developer
Kubernetes for the PHP developer
 
Learn kubernetes in 90 minutes
Learn kubernetes in 90 minutesLearn kubernetes in 90 minutes
Learn kubernetes in 90 minutes
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use Cases
 
Introduction to Tekton
Introduction to TektonIntroduction to Tekton
Introduction to Tekton
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals Training
 
Pro2516 10 things about oracle and k8s.pptx-final
Pro2516   10 things about oracle and k8s.pptx-finalPro2516   10 things about oracle and k8s.pptx-final
Pro2516 10 things about oracle and k8s.pptx-final
 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes Introduction
 
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...
kubeadm Cluster Creation Internals_ From Self-Hosting to Upgradability and HA...
 
containerD
containerDcontainerD
containerD
 
Spark with kubernates
Spark with kubernatesSpark with kubernates
Spark with kubernates
 
JDD2015: Kubernetes - Beyond the basics - Paul Bakker
JDD2015: Kubernetes - Beyond the basics - Paul BakkerJDD2015: Kubernetes - Beyond the basics - Paul Bakker
JDD2015: Kubernetes - Beyond the basics - Paul Bakker
 
Introduction to Kubernetes
Introduction to KubernetesIntroduction to Kubernetes
Introduction to Kubernetes
 
Managing Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayManaging Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native Way
 

Recently uploaded

Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdfShreedeep Rayamajhi
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpressssuser166378
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilitiesalihassaah1994
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 

Recently uploaded (12)

Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpress
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilities
 
Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 

Kubernetes Node Deep Dive

  • 1. Kubernetes “Node” Past, now, and the future by Harry Zhang @resouer
  • 2. What is “Node”? Where the container lives Kubernetes cluster is bootstrapped container centric features are implemented docker/rkt/runV/runC is plugged in networking is implemented volume is enabled sig-node
  • 3. Why “Node” Matters? Kubernetes a bottom-up design of container cloud with special bonus from Google … Node Container Control Panel Pod Replica StatefulSet Deployment DaemonSet Job … How Kubernetes is created? ? Containers
  • 4. Borg Borg = engineer oriented deployment scheduling & management system Google internal is massively using cgroups container, not Container :) Kubernetes = re-innovate Borg with Container
  • 5. Node Unsung hero to bridge Borg control panel with container
  • 8. Kubelet Overview kubelet SyncLoop kubelet SyncLoop proxy proxy 3.1 New pod object detected 3.2 Bind pod with node etcd scheduler api-server
  • 9. Kubelet Overview kubelet SyncLoop kubelet SyncLoop proxy proxy 4.1 Detected pod bind with me 4.2 Start containers in pod etcd scheduler api-server
  • 10. Pod “Alloc” in Borg The atomic scheduling unit in Kubernetes Process group in container cloud Implemented in Node But why?
  • 11. Are You Using Container Like This? 1.use supervised/systemd to manage multi-apps in one container 2.ensure container order by tricky scripts 3.add health check for micro-service group 4.copy files from one container to another 5.connect to peer container across whole network stack 6.schedule super affinity containers in cluster
  • 12. Multiple containers Multiple Apps in One Container Master Pod kube-apiserver kube-scheduler controller-manager
  • 14. Health Check for Containers Liveness probe Pod will be reported as Unhealthy Master Pod kube-apiserver kube-scheduler controller-manager
  • 15. Copy Files from One to Another Pod volumes is shared Master Pod kube-apiserver kube-scheduler controller-manager /etc/kubernetes/ssl
  • 16. Connect to Peer Container Pod network is shared Master Pod kube-apiserver kube-scheduler controller-manager network namespace
  • 17. Pod is atomic scheduling unit • controller super affinity apiserver • Request: • controller: 1G, apiserver: 0.5G • Available: • Node_A: 1.25G, Node_B: 2G • What happens if controller is scheduled to Node_A first? Schedule Super Affinity Containers
  • 18. So, this is Pod Design pattern in container world decoupling reuse & refactoring describe more complex workload by container e.g. ML
  • 19. kubelet register listers diskSpaceManager oomWatcher InitNetworkPlugin chooseRuntime (build-in, remote) InitNetworkPlugin NewGenericPLEG NewContainerGC AddPodAdmitHandler HandlePods {Add, Update, Remove, Delete, …} NodeStatus Network Status status Manager PLEG SyncLoop <-chan kubetypes.PodUpdate* <-chan *pleg.PodLifecycleEvent periodic sync events housekeeping events Pod Update Worker (e.g.ADD) • generale pod status • check volume status • call runtime to start containers volume Manager PodUpdate *4-sources api-server (primary, watch) http endpoint (pull) http server (push) file (pull) image Manager Eviction
  • 20. Prepare Volume Volume Manager desired World reconcile Find new pods createVolumeSpec(newPod) Cache volumes[volName].pods[podName] = pod • Get mountedVolume from actualStateOfWorld • Unmount volumes in mountedVolume but not in desiredStateOfWorld • AttachVolume() if vol in desiredStateOfWorld and not attached • MountVolume() if vol in desiredStateOfWorld and not in mountedVolume • Verify devices that should be detached/unmounted are detached/unmounted • Tips: 1. -v host:path 2. attach VS mount 3. Totally independent from container management
  • 21. Eviction Guaranteed Be killed until they exceed their limits or if the system is under memory pressure and there are no lower priority containers that can be killed. Burstable killed once they exceed their requests and no Best- Effort pods exist when system under memory pressure Best-Effort First to get killed if the system runs out of memory
  • 22. Management kubelet CRI Workloads Orchestration kubelet SyncLoop Scheduling api-server Etcd bind pod, node list pod GenericRuntime SyncPod CRI grpc dockershim remote (no-op) Sandbox Create Delete List Container Create Start Exec Image Pull List shim client api dockerd runtime pod CRI Spec
  • 23. CRI Runtime Shim dockershim: docker frakti: hypervisor container (runV) cri-o: runC rktlet: rkt cri-containerd: containerd
  • 24. NODE Container Lifecycle Pod foo container A container B 1. RunPodSandbox(foo) Created Running Exitednull null CreatContainer() StartContainer() StopContainer() RemoveContainer() $ kubectl run foo … A B foo foo (hypervisor) A B 2. CreatContainer(A) 3. StartContainert(A) 4. CreatContainer(B) 5. StartContainer(B) docker runtime hypervisor runtime
  • 25. Streaming (old version) kubelet becomes bottleneck runtime shim in critical path code duplication among runtimes/shims kubectl apiserver kubelet runtime shim 1. kubectl exec -i 2. upgrade connection 3 stream api Design Doc
  • 26. Streaming (CRI version) kubectl apiserver kubelet runtime shim 1. kubectl exec -i 2. upgrade connection 3. stream api serving process 4. launch a http2 server 6. URL: <ip>:<port> 7. redirect responce 8. update connection CRI Design Doc 5. response
  • 27. Streaming in Runtime Shim kubelet frakti Streaming Server Runtime apiserver url of streaming server CRI Exec() url Exec() request $ kubectl exec … "/exec/{token}" stream resp runtime exec api Stream Runtime Exec() Attach() PortForward()
  • 28. CNI Network in Runtime Shim Workflow in runtime shim (may vary from different runtimes): 1. Create a network NS for sandbox 2. plugin.SetUpPod(NS, podID) to configure this NS 3. Also checkpoint the NS path for future usage (TearDown) 4. Infra container join this network namespace 1. or scanning /etc/cni/net.d/xxx.conf to configure sandbox Pod A B eth0 vethXXX
  • 29. Physical Server frakti Mixed Runtimes: IaaS-less Kubernetes Hypervisor container + Docker Handled by: https://github.com/kubernetes/frakti/ 1.Share the same CNI network 2.Fast responsiveness (no VM host needed) 3.High resource efficiency (k8s QoS classes) 4.Mixed run micro-services & legacy application 1. independent kernel + hardware virtualization 2. high I/O performance + host namespace hyper runtime dockershim CRI grpc hypervisor NFV monitor hypervisor NFV logger docker docker docker docker docker
  • 30. Node & Kubernetes is Moving Fast GPU isolation libnvidia-container is proposed CRI enhancement cri-containerd (promising default), cri-tools, hypervisor based secure container CPU pin (and update) and NUMA affinity (CPU sensitive workloads) HugePages support for large memory workloads Local storage management (disk, blkio, quota) “G on G”: run Google internal workloads on Google Kubernetes
  • 31. Recently Kubernetes CRI enhancement, equivalence class scheduler (Borg), NodeController, StatefulSet https://github.com/kubernetes/frakti (Secure container runtime in k8s) Mentoring cri-containerd, cri-tools Unikernels & LinuxKit + k8s (Google Summer of Code 2017) ovn-kubernetes (coming soon) Newly started: Stackube
  • 32. Stackube https://github.com/openstack/stackube (Hypernetes v2) 100% upstream Kubernetes + OpenStack plugins + Mixed Runtime A IaaS-less, multi-tenant, secure and production ready Kubernetes distro Milestone: 2017.9