5. 5
Challenge #1 : Communication Cost
Doing it the traditional way –
1. Communication lag – takes too long to formulate requirements from developers
2. XY problem – no idea what the real problem is
3. Validation and policy injection is manually done
6. 6
Challenge #1 : Communication Cost
Solution: Create an opionated Internal Developer Platform and form an API based contract with
users
Philosophy :
• When you have APIs and their documentation users rarely need to communicate with you
• Easier to explicitly define what you provide and what you don’t
• Standardization = low re-invention of wheel, less pets, easier to propagate tech culture
Implementation :
• In CaaS we make use of K8s APIs to expose features to users. Custom Resource Definitions (CRDs)
and Operators fits us well.
• Admission control webhooks, podSecurityPolicy and networkPolicy
7. 7
Challenge #1 : Communication Cost
Jiange : Validation without human communication
Jiange
etcd K8s API
9. 9
Challenge #2 : Day 2 Ops
Day 1 Ops :
• Provisioning
• Step 1
• Step 2
• Step 3… N
• Procedural – easy to automate
Day 2 Ops:
• Maintainence
• Not always the same
• Improvements – need to keep an eye on various components
• Metrics
• Logs
• Traces
10. 10
Challenge #2 : Day 2 Ops
Solution: Infrastructure as Data instead of Infrastructure as Code
Script
for X
Script
for Y
Script
for Z
IaC – run scripts one by one
Data
Store Infra
Infra
Control
Loop
Reconcile Spec
Reconcile Status
IaD – Store the state as Data and
reconcile until state is achieved
11. 11
Challenge #2 : Day 2 Ops
Solution: Infrastructure as Data instead of Infrastructure as Code
In CaaS we have written controllers based on same approach
• Klone – Binary that provisions master nodes and system components based on git configs (written in
Go)
• Node operator – used for creating worker nodes
• Namespace operator – used for creating user namespaces with correct permissions, good defaults,
jenkins repositories, harbor projects etc when user on boards.
• Gateway controller – For creating istio ingress gateways
• Wildcard instant domain controller – For instantly creating simple domains to test out services
• Cloud controlller manager – for creating load balancers
• Endpoints controller – for creating container native load balancers
12. 12
Challenge #3 : Day 2 Ops
Internet
Load Balancer
K8s API
Node
List
Cloud
Controller
Manager
K8s cluster nodes
14. 14
Challenge #3 : Container Networks
• Kubernetes network != Host Network
• Pods are not first class citizens (not flat network)
• Pods are ephemeral
• Fair Load balancing does not happen when using NodePorts
• Additional hops (through K8s node Iptables)
• Source IP is not preserved
• Network is difficult to use
15. 15
Challenge #3 : Container Networks
Solution: No one size fits all, provide all
solutions with good defaults and let users
choose
Shared Gateway +
Auto Assigned
Domain
Dedicated Gateway +
Custom Domain
Domain Auto Assigned Any Domain
Performance Not isolated Isolated
Maintainence (for
users)
Zero High
Customization Low Fully customizable
Cost Low High
16. 16
Challenge #3 : Container Networks
Solution: Container Native Load balancing
Legacy Load
Balancer
Container Native
Load Balancer
Number of hops 2 1
IP preservation Remote IP lost Remote IP
preserved
Load Balancing Across nodes Across containers
Health checks Only for Nodes Application level
health checks
17. 17
Future Challenges:
Multicluster CaaS -
Network
Deployments
IPv4 not enough (need IPv6 and/or VPCs)
Stateful apps -
Local persistence
Remote persistence
GPU
SRIOV
CPU pinning
Single Data proxy