Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It was originally developed by Google based on years of experience running production workloads at scale. Kubernetes groups containers into logical units called pods and handles tasks like scheduling, health checking, scaling and rollbacks. The main components include a master node that manages the cluster and worker nodes that run application containers scheduled by the master.
1. Evolution of Containers to
Krishna Kumar, CNCF Ambassador India & Shivram Srivastava, Open Source Maintainer k8s
A presentation at
Contents & pictures are taken from the web; Will be used for education purpose only
2. Contents
• Evolution of Containers!
• LXC to Docker …
• Kubernetes
• Architecture
• Components
• Use cases
• More…
3.
4. Where does all it comes from?
• LXC (LinuX Containers) was the first, most complete implementation of
Linux container manager. It was implemented in 2008 using cgroups
and Linux namespaces, and it works on a single Linux kernel without
requiring any patches. early stages and later replacing it with its own
implementation.
• The LXC containers that are now part of every Linux distribution were
founded on Google’s work in 2006 -
https://linuxacademy.com/blog/containers/history-of-container-
technology/
• When Docker emerged in 2013, containers exploded in popularity. It’s no
coincidence the growth of Docker and container use goes hand-in-
hand. Docker used LXC in its initial stages and later replaced that container
manager with its own library, libcontainer. But there’s no doubt that
Docker separated itself from the pack by offering an entire ecosystem for
container management.
• Brief history of Containers - https://dzone.com/articles/evolution-of-
linux-containers-future
https://content.pivotal.io/infographics/moments-in-container-history
5. Container Evolution: Moving from Chroot to Namespaces
chroot
Chroot jails were the earliest isolation
mechanisms. Chroot offers file system
level abstraction and prevents
applications from interfering with each
other.
There was problem with chroot: The
chroot can be broken by root users.
Applications in different namespaces
could still interfere in many ways. Also
Start some programs (with some
overlapping name) from two shells
chrooted to two different directories.
Execute ps –ef | grep “<overlapping
portion>” from one bash shell and kill
all the PID’s corresponding to the
matching results (a typical termination
program), will kill program in other
bash shell.
5
namespaces
The name spaces are IPC, PID, hostname, User,
Mount, network
Application isolation by manipulate certain system
attributes - e.g: clone(), unshared(), setns(), execute the
bash shell in separate PID, mount proc file system
namespaces, kill the init process of the child
namespace, etc.
6. Container Evolution: cgroups – Resource Limiting
• Handles resource quota allocation. It associates “subsystems” – which are kernel modules
that meter the system resources such as (cpu and memory) to “hierarchies of ‘cgroups’” –
which are user-defined rules to restrict resource usage of processes. Some of the cgroup
subsystems are:
o blkio — this subsystem sets limits on input/output access to and from block devices
such as physical drives (disk, solid state, or USB).
o cpu — this subsystem uses the scheduler to provide cgroup tasks access to the CPU.
o cpuacct — this subsystem generates automatic reports on CPU resources used by
tasks in a cgroup.
o cpuset — this subsystem assigns individual CPUs (on a multicore system) and
memory nodes to tasks in a cgroup.
o devices — this subsystem allows or denies access to devices by tasks in a cgroup.
o freezer — this subsystem suspends or resumes tasks in a cgroup.
o memory — this subsystem sets limits on memory use by tasks in a cgroup and
generates automatic reports on memory resources used by those tasks.
o net_cls — this subsystem tags network packets with a class identifier (classid) that
allows the Linux traffic controller (tc) to identify packets originating from a particular
cgroup task.
o net_prio — this subsystem provides a way to dynamically set the priority of network
traffic per network interface.
o ns — the namespace subsystem.
o perf_event — this subsystem identifies cgroup membership of tasks and can be used
for performance analysis.
6
A cgroup example:
7. Container Evolution: LXC Architecture
7
- LXC abstracts the complexities of configuring the cgroups
and namespaces and helps to automate the process.
- LXC (Linux Containers) is an operating-system-level
virtualization method for running multiple isolated Linux
systems (containers) on a control host using a single Linux
kernel. “LXC” refers to capabilities of the Linux kernel (specifically
namespaces and control groups) which allow sandboxing
processes from one another, and controlling their resource
allocations.
- Kernel features for containers: Isolated Namespaces: (Audit,
System, Device, Time, Taskcount), Watch mount points, root level
permissions, file system read only privilege for Copy-on-write, Use
resource limitation feature, Avoid running container with User ID
= 0; harden OS,
8. Major Container Types….
Docker: The most popular container OS - a lightweight, portable, self sufficient LXC container that can run virtually anywhere. It has layered container
image, global container registry, Cluster management, CLI/Rest API connections.
Rocket: From CoreOS/RedHat, Rocket has more rigorous security, app container specifications. Also developed flannel, etcd to support container
orchestration.
Photon: Photon OS is a minimal Linux container host, optimized to run on VMware platforms. Compatible withDocker, and Kubernetes.
Garden: From Pivotal Cloud Foundry. Garden (Warden) is a platform-agnostic Go API for container creation and management, with pluggable back ends
for different platforms and runtimes.
Mesos Containers: MesosContainerizer provides lightweight containerization and resource isolation of executors using Linux-specific functionality such
as control cgroups and namespaces.
Windows Containers: Two different run times - Windows Server Containers which uses shared kernel space and Hyper-V Isolation Containers which run
each container in a optimized virtual machine (e.g: Windows 10 containers)
IBM Nabla Container: Cut down OS system calls to a bare minimum with as little code as possible. This is expected to decrease the surface area available
for an attack. Make use of Library OS (unikernel techniques) and use only 9 system calls; the rest are blocked through linux seccomp policy.
Google gVisor: User-space kernel, written in Go, that implements a substantial portion of the Linux system surface. It includes an OCI runtime
called runsc that provides an isolation boundary between the application and the host kernel.
Kata Containers: standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload
isolation and security advantages of VMs. It is designed to be architecture agnostic, run on multiple hypervisors and be compatible with the OCI.
ConrtainerD: It is an industry-standard core container runtime available as a daemon for Linux and Windows, which can manage the complete container
lifecycle of its host system. It uses runC to run containers according to the OCI specification.
OCI: Open Container Initiative -currently contains two specifications: the Runtime Specification (runtime-spec) and the Image Specification (image-spec).
The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. At a high-level an OCI implementation would download an
OCI Image then unpack that image into an OCI Runtime filesystem bundle. 8
9. Container Platform - Docker
Docker is a platform and tool for building, distributing,
and running container.
Docker means a person who docks, shipping docks.
Docker Inc is a private company which handles all
aspects of Docker. https://www.docker.com/
Docker's open source components are
generally licensed under the Apache 2.0 license. Written in
GO language.
Docker also has Enterprise commercial product.
In 2008, Solomon Hykes and few others started
dotcloud and eventually it becomes Docker, Inc. the
company revolutionize the cloud industry.
9
10. How to Build Docker Image
• An image is an inert, immutable, file that's
essentially a snapshot of a
container.Images are created with the build
command, and they'll produce a container
when started with run. Images are stored in
a Docker registry such as
registry.hub.docker.com
• Docker can build images automatically by
reading the instructions from a Dockerfile, a
text document that contains all the
commands you would normally execute
manually in order to build a Docker image.
• https://docs.docker.com/engine/reference/co
mmandline/images/
12
FROM keyword tells which image is your image based on
COPY instruction copies new files/directories from <src> and
adds to the filesystem of the container at the path <dest>
EXPOSE listens on the specified network ports at runtime
(does not make the ports of the container accessible to the
host)
CMD provide defaults for an executing container. There can be
only one cmd in a Dockerfile. If multiple CMDs are there, then
only the last cmd will take effect
To build the image name “nginx:latest_image “ from current
directory where dockerfile is residing; run
docker build –t nginx:latest_image .
Sample Docker File
11. Docker Registry
• Create an account in Docker Hub. Push the image which you tagged. Docker hub account would contain the pushed image.
• One can pull the image and run a container using the image.
• https://docs.docker.com/registry/
• https://blog.octo.com/en/docker-registry-first-steps/
13
14. From Virtualization to Cloud Native
16
Containers
Cloud
Native
Open
Source
IaaS
PaaS
Open
Source
PaaS
Virtualiza-
tion
2000 2001 2006 2009 2010 2011
Non-
Virtualized
Hardware
2013 2015
IaaS
15. Container Orchestration Engines
Kubernetes is a orchestration system for running Docker and
other containers.
Some other leading container orchestration Engines (like Kubernetes):
Mesosphere Mesos
Docker Swarm
Hashicorp Nomad
Netflix Titus
Racher Cattle
Pivotal Cloud Foundry
Amazon ECS
Deis (Microsoft acquired it)
Cisco Mantl
Kontena Pharos
https://medium.freecodecamp.org/how-to-choose-the-right-container-orchestration-and-how-to-deploy-it-41844021c241
16. CNCF is an open source software foundation (under Linux foundation) dedicated to making cloud native computing universal and sustainable. Cloud
native computing uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and
dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great
products faster. The following projects are under the foundation:
18
• Platinum members:
Orchestration
Incubating Sandbox
Service
Mesh
Storage
Networking
API
Service
Discovery
Distributed
Tracing
Service Mesh
Software
Update Spec
StorageSecurity
Logging Remote
Procedure Call
Monitoring
Distributed
Tracing API
Container
Runtime
Messa
ging
Iden
tity
Spec
Identit
y
Poli
cy
Graduated
Server
less Tooling
Container
Runtime
Package
Manageme
nt
Harbor
Registry
Open
Metrics
TIKV
Key-Value
Store
20. Evolution of Kubernetes!!
• Borg – original paper -
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pd
f
• Omega – original paper -
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41684.pd
f
• Borg: The Predecessor to Kubernetes - https://kubernetes.io/blog/2015/04/borg-predecessor-
to-kubernetes/
• Lessons learned from three container management system over a decade at Google - Borg,
Omega, and Kubernetes - https://storage.googleapis.com/pub-tools-public-publication-
data/pdf/44843.pdf
• Kubernetes (κυβερνήτης, Greek for "governor", "helmsman" or "captain") was founded by Joe
Beda, Brendan Burns and Craig McLuckie, was quickly joined by other Google engineers
including Brian Grant and Tim Hockin, and was first announced by Google in mid-2014.
• Kubernetes is also referred to as k8s, as there are 8 characters between k and s.
• Kubernetes celebrates its birthday every year on 21st July. Kubernetes 1.0 was released on July
21 2015. Generally licensed under the Apache 2.0 license. Written in GO language.
• In July 2015, Google along with Twitter, IBM, Intel, Cisco, and Goldman Sachs, among others
announced creating CNCF and inducted kubernetes as the first project in it.
• Aug 2018, Google sets Kubernetes free with $9m in its pocket for expenses - Google shifting
the ownership of Kubernetes cloud resources to CNCF.
“Kubernetes is an open-source system for automating deployment, scaling, and
management of containerized applications. It groups containers that make up an
application into logical units for easy management and discovery.”
21. K8s - Main features
23
• Open source & run anywhere
• Group containers in to logical units
• Easy management & Discovery
• Self healing
• Scaling – Planet size (billions of containers in a week)
• Serverless with server
• Optimize resource usage
• Multi Clouds
Automatic bin-packing
Kubernetes automatically schedules the containers based on resource usage and constraints, without sacrificing the availability.
Self-healing
Kubernetes automatically replaces and reschedules the containers from failed nodes. It also kills and restarts the containers which do not respond to health checks, based on
existing rules/policy.
Horizontal scaling
Kubernetes can automatically scale applications based on resource usage like CPU and memory. In some cases, it also supports dynamic scaling based on customer metrics.
Service discovery and Load balancing
Kubernetes groups sets of containers and refers to them via a Domain Name System (DNS). This DNS is also called a Kubernetes service. Kubernetes can discover these services
automatically, and load-balance requests between containers of a given service
Automated rollouts and rollbacks
Kubernetes can roll out and roll back new versions/configurations of an application, without introducing any downtime.
Secrets and configuration management
Kubernetes can manage secrets and configuration details for an application without re-building the respective images. With secrets, we can share confidential information to
our application without exposing it to the stack configuration, like on GitHub.
Storage orchestration
With Kubernetes and its plugins, we can automatically mount local, external, and storage solutions to the containers in a seamless manner, based on software-defined storage
(SDS).
Batch execution
Besides long running jobs, Kubernetes also supports batch execution.
Every 3 months there is one release.
https://kubernetes.io/docs/setup/release/
notes/
22. K8s Architecture
One or more master nodes
One or more worker nodes
Distributed key-value store, like etcd
https://kubernetes.io/docs/concepts/
https://kubernetes.io/docs/home/
https://www.slideshare.net/RedHatDevelopers/kubernetes-introduction-86540172 24
23. Pod Concept…
25
• Pod – Multiple containers share
IP/storagevolume/resources/secrets/
labels/Lifecycle – Define as a YAML
• Service – Groups of pods act as one
service with IP and DNS name
• Label – Key/value pair associated
with k8s objects like Pod.
• https://kubernetes.io/docs/concepts
/workloads/pods/pod-overview/
24. 26
The master node is responsible for managing the Kubernetes cluster, and it
is the entry point for all administrative tasks.
1. API server
2. Scheduler
3. Controller manager
4. etcd.
A worker node is a machine (VM, physical server, etc.) which runs the
applications using Pods and is controlled by the master node. Pods are
scheduled on the worker nodes, which have the necessary tools to run and
connect them. A Pod is the scheduling unit in Kubernetes. It is a logical
collection of one or more containers which are always scheduled together.
To run and manage a container's lifecycle, we need a container runtime on
the worker node. Some container runtimes are: containerd, rkt, lxd, docker.
We will explore them further in later chapters.
1. Container runtime
2. kubelet
3. kube-proxy
kubectl – Command Line interface (CLI)
K8s Components
25. Master Node Components
27
1. API server
• All the administrative tasks are performed via the API
server within the master node. A user/operator sends REST
commands to the API server, which then validates and
processes the requests. After executing the requests, the
resulting state of the cluster is stored in the distributed key-
value store.
2. Scheduler
• schedules the work to different worker nodes. The scheduler
has the resource usage information for each worker node. It
also knows about the constraints that users/operators may
have set, such as scheduling work on a node that has the
label disk==ssd set. Before scheduling the work, the scheduler
also takes into account the quality of the service requirements,
data locality, affinity, anti-affinity, etc. The scheduler schedules
the work in terms of Pods and Services.
3. Controller manager
• Manages different non-terminating control loops, which
regulate the state of the Kubernetes cluster. Each one of these
control loops knows about the desired state of the objects it
manages, and watches their current state through the API
server. In a control loop, if the current state of the objects it
manages does not meet the desired state, then the control
loop takes corrective steps to make sure that the current state
is the same as the desired state.
4. etcd
• Is a distributed key-value store which is used to store the
cluster state. It can be part of the Kubernetes Master, or, it can
be configured externally, in which case, master nodes would
connect to it.
26. K8s etcd – more.…
28
• State Management with etcd
• Kubernetes uses etcd to store the cluster state.
• etcd is a distributed key-value store based on the Raft
Consensus Algorithm.
• Raft allows a collection of machines to work as a coherent
group that can survive the failures of some of its members.
• At any given time, one of the nodes in the group will be the
master, and the rest of them will be the followers. Any node
can be treated as a master
• etcd is written in the Go programming language.
• In Kubernetes, besides storing the cluster state, etcd is also
used to store configuration details such as subnets,
ConfigMaps, Secrets, etc.
27. Worker Node Components
29
1. The kubelet is an agent which runs on each worker node and
communicates with the master node. It receives the Pod definition via
various means (primarily, through the API server), and runs the
containers associated with the Pod. It also makes sure that the
containers which are part of the Pods are healthy at all times. The
kubelet connects to the container runtime using Container Runtime
Interface (CRI).
2. The Container Runtime Interface consists of protocol buffers, gRPC
API, and libraries.
3. Kube-proxy Instead of connecting directly to Pods to access the
applications, we use a logical construct called a Service as a
connection endpoint. A Service groups related Pods and, when
accessed, load balances to them. kube-proxy is the network proxy
which runs on each worker node and listens to the API server for each
Service endpoint creation/deletion. For each Service endpoint, kube-
proxy sets up the routes so that it can reach to it.
1. dockershim
• With dockershim, containers are created using Docker installed on the
worker nodes. Internally, Docker uses containerd to create and manage
containers.
2. cri-containerd
• With cri-containerd, we can directly use Docker's smaller offspring
containerd to create and manage containers
3. CRI-O
• CRI-O enables using any Open Container Initiative (OCI) compatible
runtimes with Kubernetes. Supports runC and Clear Containers as
container runtimes now.
28. Kubectl
• kubectl create -f is what we call Imperative
Management. On this approach you tell the
Kubernetes API what you want to create, replace
or delete, not how you want your K8s cluster
world to look like.
• kubectl apply is part of the Declarative
Management approach, where changes that you
may have applied to a live object (i.e. through
scale) are maintained even if you apply other
changes to the object.
• https://kubernetes.io/docs/reference/kubectl/o
verview/
30
31. K8s - Storage & Networking
33
Persistent Volume Providers
• GCE
• AWS
• Rook
• Glusterfs
• OpenStack Cinder
• CephRBD
• QuoByte
• Kube-Aliyun
• Portworx
• Rancher Longhorn
• Stork
• OpenEBS
• StorageOS
Networking Stacks
• Weave Net
• Canal by Tigera
• OpenContrail
• Nuage
• Kuryr
• Contiv
• Calico
• OpenVSwitch
• Kube-router
• Cilium
• Linen
• CNI-Genie
• Romana
• Infoblox
• External DNS - To control DNS records dynamically via Kube
resources
• cni-ipvlan-vpc-k8s
• kubernetes-network-policy-recipes
• Multus-cni - Multi-homed pod cni
• Network-Controller - Open vSwitch, Multiple network interfaces that
associate with Kubernetes pods
• AWS VPC CNI - Networking plugin using Elastic Network Interfaces
• NSX-T - CNI plugin supporting load balancing and distributed
firewalls.
• Container Network Model (CNM) - proposed by Docker
• Container Network Interface (CNI) - proposed by CoreOS.
32. K8s - Installers & Monitoring
34
Installers
• Minikube - Run Kubernetes locally
• Kops - OS Agnostique - AWS
• Kube-deploy
• Kubeadm - OS Agnostique - Cloud Agnostique
• Kubespray - OS Agnostique - Cloud Agnostique
• Bootkube - CoreOS - Cloud Agnostique
• Kube-aws - CoreOS - AWS
• Kismatic - CentOS - Cloud Agnostique
• Juju - Ubuntu - Cloud Agnostique
• Terraform - CoreOS - AWS
• Supergiant - CoreOS - Cloud Agnostique
• Archon - OS Agnostique - Cloud Agnostique
• KubeNow - Ubuntu - Cloud Agnostique
• Kubicorn - OS Agnostique - Cloud Agnostique
• Simplekube - systemd OS - Cloud Agnostique
• Conjure-up - Ubuntu - Cloud Agnostique
• Kube-ansible - OS Agnostique - Cloud Agnostique
• Kubernetes-Saltstack - systemD OS - Cloud Agnostique
• matchbox - CoreOS - Network boot and provision Container Linux clusters
(e.g. etcd3, Kubernetes, more).
• RKE - OS Agnostique - Cloud Agnostique
• Typhoon - Container Linux - Cloud Agnostique
• Docker for Windows - Run Kubernetes and Docker locally on your Windows
PC (Edge Channel)
• Docker for MAC - Run Kubernetes and Docker locally on your MAC (Edge
Channel)
• MetalK8s - CentOS - On-Prem - Cloud Agnostique - Apache-2.0
Monitoring
• Console
• Datadog
• eventrouter - simple introspective kubernetes service that forwards events
to a specified sink.
• Grafana Kubernetes App
• Heapster
• Instana
• Kubebox - Terminal console for Kubernetes
• Kubedash
• Kubernetes Operational View - read-only system dashboard for multiple
K8s clusters
• Kubetail
• Kubewatch
• Netsil
• Outcold Solutions - monitoring Kubernetes, OpenShift and Docker in Splunk
Enterprise and Splunk Cloud (metrics and log forwarding)
• Prometheus
• Sysdig Monitoring
• Sysdig Open Source
• Weave Scope
• Searchlight
• Ingress Monitor Controller - A Kubernetes Controller to watch your
ingresses and create liveness alerts for your endpoints
33. K8s - more elements…
35
Operators
• Prometheus
• Kong API
• Kubernetes Operators
• K8s Operator Workshop
• Cert Operator
• Cert manager
• cert-manager
• Operator Kit
• Container Linux Update Operator
• DB Operator
• KubeVirt
• Operator SDK
• kooper - Simple Go library to create
Kubernetes operators and controllers
CI / CD
•Jenkins X - CI/CD for Kubernetes using
Jenkins
•kb8or
•Wercker
•Shippable
•GitLab
•Kontinuous
•Kit
•Spinnaker
•CircleCI
•KubeCI
•Vili
•Keel
•Apollo - Open Source application
providing teams with self service UI for
creating and deploying their services to
Kubernetes.
•Weave Flux – GitOps reconcoliation
operator
•Codefresh - Kubernetes CI/CD platform
(with private Docker and Helm Chart
repositories)
Some Related Software..
•Argo The Workflow Engine for
Kubernetes
•Hypernetes
•Kubernetes Cluster Federation
(previously Ubernetes)
•kmachine
•Kubefuse
•KubeSpray
•Kubernetes Ec2 Autoscaler
•Kubeform
•kube-openvpn
•Archon
•Client Libraries
•Kubic-Project
•Telepresence - Locally develop/debug
services against a remote Kubernetes
cluster
•Fission Workflows - Workflow-based
serverless function composition
•Ambassador - API Gateway built on the
Envoy Proxy
34. Kubernetes – use cases …..
36
https://thenewstack.io/ebooks/us
e-cases/use-cases-for-
kubernetes/
Scaling services like large scale web applications for ecommerce
operation
Stateful workloads – Zookeper, MySQL, etc.
Serverless execution for short burst jobs (like AWS Lambda)
Optimized resource usage with resource quota
Hybrid and multi cloud scenarios
Solution providers for CRM, ERP, etc.
System Integrators for Enterprise/Legacy/Cloud
Specific needs – Advanced routing, network scanners, etc.
Kubernetes case studies - https://kubernetes.io/case-studies/
And now gaining momentum in Big data, AI/ML, Block chain,
Robotics operations and so on, almost all areas of computing..
The business case for running k8s -
https://thenewstack.io/containers-enable-companies-just-less/