This document discusses strategies for rolling out application updates on Kubernetes in a way that keeps users happy. It describes how typical deployments and rollouts work, ways they can go wrong by dropping requests, and strategies to handle rollouts gracefully like using a sidecar container and preStop termination hooks. The strategies aim to allow applications to shutdown gracefully and drain connections during pod deletion to avoid dropped requests and improve the user experience during updates.

1. 1
Crossing the Streams!
Rollout Strategies to Keep
Your Users Happy!
Steve Sloka
Bill Koch

2. - K8s since early 2015
- Kubernetes Contributor
- Contour / Gimbal Maintainer
- Pittsburgh Kubernetes
Meetup Organizer
About Us
Sr. Member of Technical Staff
VMware
- K8s since early 2015
- Spring since 2007
Software Engineer - Expert
UPMC Enterprises

3. 3
Application
Rollout Strategies

4. 4

5. 5
Typical Kubernetes Deployment
Deployment
ReplicaSet
Pod
Pod
Pod
Pod
Pod
Pod
Service
label selector

6. 6
Typical Deployment Rollout
Deployment ReplicaSet (Old)
Pod
Pod
Pod
Pod
Pod
Pod
Service
ReplicaSet (New)
Pod
Pod
Pod
Pod
Pod
Pod

7. 7
Typical Deployment Rollout
Deployment ReplicaSet (Old)
Pod
Pod
Pod
Pod
Pod
Pod
Service
ReplicaSet (New)
Pod
Pod
Pod
Pod
Pod
Pod

8. 8
Typical Deployment Rollout
Deployment ReplicaSet (Old)
Pod
Pod
Pod
Pod
Pod
Pod
Service
Pod
ReplicaSet (New)
Pod
Pod
Pod
Pod
Pod
Pod

9. 9
How Can Deployment Rollouts Go Wrong?
https://images.app.goo.gl/n4dhosQnGp2QnVJi9

10. 10
- Kubernetes Control-Plane components not
updated
Dropped Requests

11. 11
Macro View
What Happens When A Pod Is Deleted?
kubectl delete pod
API Server
Endpoint
Controller
kube-proxy
API Server
Ingress
Time
Core DNS
iptables
API Server
kubelet
pod termination process
1
2

12. 12
Potential Timing Issues
What Happens When A Pod Is Deleted?
kubectl delete pod
API Server
Endpoint
Controller
kube-proxy
API Server
Ingress
Time
Core DNS
iptables
API Server
kubelet
pod termination process
1
2
Dropped Requests

13. 13
- In flight requests are dropped during
shutdown process
Dropped Requests

14. 14
Pod Termination Process
What Happens When A Pod Is Deleted?
kubectl delete pod
API Server
kubelet
graceful shutdown
preStop hook
(if defined)
API Server
terminationGracePeriodSeconds
(defaults to 30)
SIGTERM SIGKILL
Time
Pod Terminated

15. 15
Pod Termination Process
What Happens When A Pod Is Deleted?
kubectl delete pod
API Server
kubelet
graceful shutdown
preStop hook
API Server
terminationGracePeriodSeconds
(defaults to 30)
SIGTERM SIGKILL
Time
Dropped Requests

16. 16
- Application handles gracefully
- Sidecar
Types of Strategies

17. 17
DEMO

18. 18
Side-Car Pattern
preStop ExecHook
Kubernetes API Kubelet Envoy
Shutdown
Manager
/shutdown
Envoy drain connections
Poll Open Connections
PreStop HTTP Hook Response
Pod Terminated
Terminate Pod
Envoy Fail Readiness Probe

19. 19
Considerations
- How long to wait?
- Memory / CPU Usage
- Debugging preStop Hook

20. 20
Resources
- github.com/billkoch/cross-the-streams
- kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
- projectcontour.io/docs/v1.8.0/redeploy-envoy/

21. Thank You
@stevesloka
@wmkoch