This document is a presentation on safely running Kubernetes clusters. It discusses the need for soft multi-tenancy between namespaces, using a cloud-native approach to managing Kubernetes clusters, establishing clear roles and personas, handling stateful workloads, differences between Kubernetes implementations, and adopting new operational practices. The presentation provides examples and recommendations on each of these topics to help users run Kubernetes clusters safely and effectively.

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0
6 Things You Need to Know to
Safely Run Kubernetes
Cornelia Davis
Vice President, Technology, Pivotal
April 2019

2. Cover w/ Image
Me?
Developer (wasn’t Ops)
Web architectures for >10 years
Cloud-native for 6+ years
Cloud Foundry for 6+ years
Discount code 40% off!: 40cloudnat
https://www.manning.com/books/cloud-native-patterns
@cdavisafc

3. ❤
💕
💕
💕
❤
❤
💕
🌈
🦋

5. Let’s have a
look at some
things you
need to get
right…

6. #1 - Soft Multi-tenancy

7. K8s Cluster Kubernetes Master

8. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
…

9. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
Kubelet Kube Proxy Kubelet Kube Proxy

10. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
Namespace 1:
Kubelet Kube Proxy Kubelet Kube Proxy
Namespace 2:
These are all shared components
That are not
namespace aware!!!

11. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
Namespace 1:
Kubelet Kube Proxy Kubelet Kube Proxy
Namespace 2:
Limitations:
● Noisy neighbors (workloads can affect
other tenants)
● Tenants share the same network
● Tenants share DNS
● Tenants share Configuration
● …

12. https://blog.jessfraz.com/post/hard-multi-tenancy-in-kubernetes/

13. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
Namespace 1:
Kubelet Kube Proxy Kubelet Kube Proxy
Namespace 2:
The suggestion: Give each tenant
their own set of K8s controller
components.

14. Kubernetes Master
API Server
Controller Manager DNS
Scheduler
Kubelet Kube Proxy Kubelet Kube Proxy
Tenant 1:
Kubernetes Master
API Server
Controller Manager DNS
Scheduler
Kubelet Kube Proxy Kubelet Kube Proxy
Tenant 2:
Leverage 20
years of maturity
in hypervisor-
based security!

15. Multi-cluster
https://content.pivotal.io/blog/kubernetes-one-cluster-or-many

16. #2 - Cloud-native Cluster Management

17. Kubernetes Takes Care of Your Workloads
Image Cache
etcd
K8s Master
Worker Worker Worker
Kubernetes
Scheduler
Desired
State
Actual
State
Replica Set Controller
LoadBalancer

18. But who is taking care of your
Kubernetes?

19. VMs are Monitored
IaaSBOSH
Worker
AGENT
Master
AGENT
etcd
AGENT
Message Bus
Health Monitor
Responses:
pager
email
monitoring
ressurector
…
BOSH Director
Desired State Actual State
Message Bus

20. VMs are Monitored
IaaSBOSH
Worker
AGENT
Master
AGENT
etcd
AGENT
Message Bus
Health Monitor
Responses:
pager
email
monitoring
ressurector
…
BOSH Director
Desired State Actual State
Worker
AGENT
etcd
AGENT
BOSH Director
Message Bus

21. VMs are Monitored
IaaSBOSH
Worker
AGENT
Master
AGENT
etcd
AGENT
Message Bus
Health Monitor
Responses:
pager
email
monitoring
ressurector
…
BOSH Director
Desired State Actual State
CPI
BOSH Director
Message Bus
Worker
AGENT
etcd
AGENT

22. PKS does for your Kubernetes
what
Kubernetes does for your apps

23. #3 - Who are Your Personas?

24. Teams Delivering Outcomes
Platform Team
Application Team
Iteratively building and
delivering digital offerings
to the consumer
Enabling the app teams all
while maintaining
Security
Compliance
Resilience
Cost Efficiency
Your Application
Code
Virtualized
Infrastructure

26. Warning!!!
kubectl delete deploy kubectl delete node
This is workload related. This is cluster related!

27. #4 - Stateful Services?

28. https://twitter.com/kelseyhightower/status/963413508300812295
But a even more has happened since then!

29. Kubernetes Master
Stateless workloads:
● Can be moved around
● Aided by service discovery
● Don’t have a particular
start order
● Bind to backing services
for state

30. Kubernetes Master

31. Kubernetes Master
Stateful workloads:
● Start order dependencies
● Storage
%
&
%
&

32. Kubernetes Master
Stateful workloads:
● Start order dependencies
● Storage
Kubernetes abstractions:
● Stateful sets
● Persistent volumes/persistent volume claims
%
&
%
&

33. Kubernetes Master
Stateful workloads:
● Start order dependencies
● Storage
Kubernetes abstractions:
● Stateful sets
● Persistent volumes/persistent volume claims
%
&
%
&

34. Warning!!!
Availability
Zone 1
Availability
Zone2
? ?
Storage classes have
different behaviors
Data replication needs to
be handled out of band

35. #5 - K8s is K8s is K8s - right?
(Spoiler alert: Nope)

36. https://twitter.com/kelseyhightower/status/935252923721793536

37. You are probably doing multi-cloud
kubectl
PKS AKS GKE EKS

38. Kubernetes Master
API Server Controller Manager
DNS Scheduler …
…
There are 150 flags
you can set on
startup
There are many
different controllers
that affect workload
behaviors
Your chosen machine
types can affect your
workloads (i.e. standard
CPU vs. GPU)
Cluster addons affect
your workloads

39. You are probably doing multi-cloud
kubectl
PKS AKS GKE EKS
Different cluster configurations -> Different workload behaviors

41. You are probably doing multi-cloud
kubectl
PKS PKS PKS PKS
Same cluster configurations across all clouds

42. #6 - New Operational Practices

43. You are probably doing multi-cloud
kubectl
PKS AKS GKE EKS
pks az gcloud aws
4 times the:
- Skills
- Scripts
- Runbooks
- …

44. Where should the abstractions lie?
kubectl
PKS PKS PKS PKS
pks pks pks pks

45. https://content.pivotal.io/blog/happy-monday-how-to-fix-a-kubernetes-cve-before-your-boss-and-the-rest-of-the-world-reads-about-it

46. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
vSphere
Azure &
Azure StackGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Concourse
Repair
— CVEs
Repave Rotate
— Credhub
Pivotal Application
Service (PAS)
Pivotal Container
Service (PKS)
Pivotal Services
Marketplace
Pivotal Function
Service (PFS)
Do you trust your software
delivery supply chain?
https://www.youtube.com/watch?v=1qcTu2QUtrU

48. Slaying
Dragons

49. Slaying Dragons
1. Think about your tenancy needs
2.You need something to take care of your Kubernetes clusters
3.Great care needed in establishing roles and permissions
4.Stateful workloads work - need to be deliberate about persistence
5.There is no such thing as “Vanilla Kubernetes”
6.Establish new operational practices

50. Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.

51. Cover w/ Image
Me?
Developer (wasn’t Ops)
Web architectures for >10 years
Cloud-native for 6+ years
Cloud Foundry for 6+ years
Discount code 40% off!: 40cloudnat
https://www.manning.com/books/cloud-native-patterns
@cdavisafc