This document provides an overview of Kubernetes including: - Kubernetes is an open source system for managing containerized applications and services across clusters of hosts. It provides tools to deploy, maintain, and scale applications. - Kubernetes objects include pods, services, deployments, jobs, and others to define application components and how they relate. - The Kubernetes architecture consists of a control plane running on the master including the API server, scheduler and controller manager. Nodes run the kubelet and kube-proxy to manage pods and services. - Kubernetes can be deployed on AWS using tools like CloudFormation templates to automate cluster creation and management for high availability and scalability.

1. DevOps with Kubernetes

2. Agenda
• Kubernetes overview
• Usage and demo
• Architecture
• Kubernetes on AWS with Cloud
Formation

3. Kubernetes Overview

4. What is Kubernetes?
Quick facts
• System for managing and orchestrating containerized
applications in
clusters, a.k.a. cluster management software
• Open source, MIT licensed, developed by Google
• Used in GCE, OpenShift, other projects

5. Kubernetes is
• portable: public, private, hybrid, multi-cloud, written in Go
• extensible: modular, pluggable, hookable, composable
• self-healing: auto-placement, auto-restart, auto-replication, auto-
scaling
• scalable and reliable: all components are scalable and clear setup
path
exists to setup scalable and reliable cluster
• documented: a lot of documentation, training materials,
community support
• open source: MIT license, large and active community

6. With Kubernetes you can
• Orchestrate complex application deployments quickly and
predictably
• Scale your applications on the fly
• Seamlessly roll out new features
• Easily setup complex operations scenarios, e.g. rolling update,
canary
deployments etc
• Optimize use of your hardware by using only the resources you
need
• Manage persistent storage
• Automate

7. Kubernetes solves
• application composition: co-
locating helper processes
preserving the “one-
application- per-container”
model,
• mounting storage systems,
• distributing configuration and
secrets,
• application health checking,
• replicating application
instances,
• horizontal (auto-)scaling,
• naming and discovery,
• load balancing,
• rolling updates,
• resource monitoring,
• log access and ingestion,
• support for introspection
and
debugging, and
• identity and authorization.

8. Kubernetes Usage

9. Kubernetes management
• Kubectl CLI
• Independent binaries for multiple platforms (Go)
• put config file to $HOME/.kube or set $KUBECONFIG
• Automation friendly with multiple output formats: text, json, yaml,
jsonpath
• Supports proxy into cluster network, container attachment and log
retrieval
• REST API
• Available at https://<master-ip>
• Self-documented, swagger documentation
• Supports proxy into cluster network
• Basic Web dashboard
• Available at https://<master-ip>/ui

10. Kubernetes objects
Primitives
• Namespace
• Node
• Pod
• Service
• Config Map
• Secret
• Volume
• Persistent
Volume
Controller
s• Replication
Controller
• Deployment
• Job
• Daemon Set
• Ingress
• ...

11. Kubernetes objects: common
• All objects include metadata with
• Name – unique
• Labels – searchable, selectable
• Annotations – arbitrary additional information
• Spec – object specific description/specification of the
object
• Status – object status within the cluster
• Object information may be received in different formats

12. Kubernetes objects: names and namespaces
• Namespaces are used to separate groups of objects, e.g. by
user, team,
project etc
• Namespaces are scopes for names; names are unique per
type within namespace
• Namespaces may also be a basis for access control separation
• Resource quotas may be associated with namespaces
kubectl get namespaces [ <ns> ... ]
kubectl describe namespaces [ <ns> ... ]

13. Kubernetes objects: nodes
• Nodes represent a physical or virtual worker machine where
kubelet,
kube proxy, and docker run
• Kubelet registers a node on the master and maintains keep-
alive check
• Nodes may be annotated and labeled to specify workload
affinity and
constraints
kubectl get nodes
kubectl describe
[ <nd> ... ]
nodes [ <nd> ... ]

14. Kubernetes objects: pods
• Pod is a group of containers
• Run on the same node – co-located and co-
scheduled
• Shared storage
• Shared localhost network and port space
• Unique IP within a cluster
• Example: app server and log shipper

15. Kubernetes objects: nodes and pods
Node 1 Node 2
Pod A-1
10.0.0.3
Cnt1
Cnt2
Pod A-2
10.0.0.5
Cnt1
Cnt2
Pod B-1
10.0.0.8
Cnt3

16. Kubernetes objects: volumes and persistent volumes
• Used to manage persistent
storage
• Multiple types supported:
• AWS EBS
• Azure block store
• Git
• NFS
• GlusterFS
• Ceph
• ...

17. Kubernetes objects: pods and volumes
Pod
Volume
Volume
Volume
Claim
Container 1
Volume
Container 2
Volume
Persistent
Mount Mount

18. Kubernetes objects: config maps and secrets
• Config maps and secrets are used for distribution of
configuration
information including secrets like password, certificates, keys etc
• Kubelet registers a node on the master and maintains keep-
alive check
• Nodes may be annotated and labeled to specify workload
affinity and constraintskubectl
kubectl
kubectl
kubectl
get configmaps [ <cm> ... ]
describe configmaps [ <cm> ... ]
get secrets [ <sc> ... ]
describe secrets [ <sc> ... ]

19. Kubernetes objects: services
• Service is an abstraction that defines a set of pods a policy to
access
them
• Service is a distributed L3 load balancer
• Single unique IP within a cluster
• Used to expose pods to the world:
• Default
• NodePort
• LoadBalancer

20. external
port
Kubernetes objects: pods and services abstraction
Cluster
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.
1
Srv
B
10.7.0.
3

21. Kubernetes objects: pods and services
Node 1 Node 2
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.
1
Srv
B
10.7.0.
3
SrvA
10.7.0.
1
Srv
B
10.7.0.
3
external
port
external
port
iptables iptables

22. Service Discovery: DNS
DNS
• <service-name>.<namespace-name>.svc.cluster.local
• <service-name>.<namespace-name>
• <service-name> - in the same namespace
• DNS SRV _<port>._<proto>.<service-name> - for port
number
e.g. “SRV _http._tcp.nginx”

23. Kubernetes Object: Controllers
• Deployment
• Daemon Set
• Job
• Ingress
• Replication
Controller
• Replication Set

24. Kubernetes Controller: Job
• Create one or more pods and ensure that specified number of
them
successfully terminates
• Jobs may be used for operations automation

25. Kubernetes Controller: Deployment
• Deployment provides declarative updates for Pods and
Replica Sets
• Orchestrate updates and rollbacks
• Scale up or down

26. Kubernetes Addons
• DNS
• UI
• Logging
• Monitorin
g

27. Kubernetes Architecture

28. Kubernetes architecture
.......
,.,:..'
"-
......,..,...,•..••..-.!
.,., . . N . . .O..,. . .
----"'-. '"".
...
-
EASTBA NC
T E C H N O L O G I E S
...w,.,..,,.,..,-.
( ' ¥ ( )! ( 0 ,<) ',(o.;.)l;o;"' '" <••·.,.,
••.:V"•• . . . .,. " " " ' " ' ·
.,.,....,,:U:
.....

29. Kubernetes architecture: node
• kubelet manages pods,
their containers, images,
volumes, network etc
• kube-proxy is a simple
network proxy and load
balancer responsible for
reflecting services on the
nodes

30. Kubernetes architecture: node
• kubelet manages pods,
their containers, images,
volumes, network etc
• kube-proxy is a simple
network proxy and load
balancer responsible for
reflecting services on the
nodes. Userspace (legacy)
or iptables (modern) modes
are supported.

31. Kubernetes architecture: control plane
• etcd is a reliable watchable
storage for all persistent
master state
• API Server is a CRUD-y
REST server with
most/all logic
implemented in plugins
that serves Kubernetes
API.
It mainly processes REST
operations, validates them,
and updates the
corresponding objects in
etcd.

32. Kubernetes architecture: control plane
• etcd is a reliable watchable
storage for all persistent
master state
• API Server is a CRUD-y
REST server with
most/all logic
implemented in plugins
that serves Kubernetes
API.
It mainly processes REST
operations, validates them,
and updates the
corresponding objects in
etcd.

33. Kubernetes architecture: control plane
• Scheduler binds
unscheduled pods to
nodes
• Control manager performs
all
other cluster-level functions,
e.g. deployments rollout,
job control, pod replication
control
etc

34. Kubernetes architecture: control plane
• Scheduler binds
unscheduled pods to
nodes
• Control manager performs
all
other cluster-level functions,
e.g. deployments rollout,
job control, pod replication
control
etc

35. Kubernetes architecture: control plane
• Kubectl client is CLI to
manage K8S cluster

36. Kubernetes architecture: security
• Authentication and
authorization are pluggable.
By default – file based, but
may be easily switched to
external resources (OAuth,
authorization service)
• Transport security is based
on TLS, key distribution is
deployment specific

37. Kubernetes architecture: security
• Authentication and
authorization are pluggable:
file based by default, but
may be easily switched to
external resources (OAuth,
authorization service)
• Transport security is based
on TLS, key distribution is
deployment specific

38. Example orchestration scenario
1. User creates a new
Deployment object via REST
2. Controller Manager sees a
Deployment object with no Pods
and creates Pod objects based
on the Deployment object
specification
3. Scheduler sees Pod objects
not assigned to Nodes and
allocates them according to
the Nodes load and the Pods
specifications
4. Kubelets running on Nodes see
Pod objects allocated to their
corresponding Nodes and start
Pods’ containers based on the
Pods’ specifications
1
4
3 2 4

39. Kubernetes Deployment on AWS

40. Deployment options
• kube-up.sh script
Available in k8s distro and supported by the developers
• Other methods as described in kubernetes
documentation
• Other projects and systems based on kubernetes, such
as GCE
• EBT AWS CloudFormation template

41. AWS Cloud Formation K8S Cluster Improvements
• Master is in auto scaling group for auto recovery
• Nodes are in multi-zone auto scaling group for high
availability
• Multiple auto scaling groups are supported for nodes
• Simple no-client cluster rollout and teardown
• Support for node EIP auto-assignment

42. Summary

43. The good, the bad, and the ugly
Pros
• Multi-platform
• Rich OTB abstractions
and
functionality
• Extensibility
Cons and problems
• Complex architecture and
setup (AWS CF Template
solves the problem for AWS)
• Manifest parameterization
is outside K8S

44. Future work
• Simple deployment to other
clouds (Azure) and on-prem
• Multi-zone master
• Single-node deployment
(reusable
• master)
• Multi-region, multi-cloud
and federated deployment
• Persistent volume management
and
• backup in prod
• Monitoring and log collection in
prod
• Integration with Jenkins
• Use Vault for key and
secret storage
• Packaged components:
HAProxy ingress, glusterfs,
elasticsearch, mongo DB,
MySql Cluster(?), Galena
Cluster(?), WildFly,
ActiveMQ, RabbitMQ (?),
HippoCMS, Keycloak,
OpenAM, Hadoop (?),
Rstudio Server, Jupyter, etc
• Web UI
• • ...

45. THANK YOU
Oleg Chunikhin
Chief Architect
ochunikhin@eastbanctech.c
om
202-295-3000
eastbanctech.com