Nowadays almost everybody knows about Kubernetes, some teams are using it and some are only dreaming about it. Despite the popularity, Kubernetes is not simple and there are so many ways of abuse this peaceful technology. In this talk I would like to present quite obvious set of tips, based on failures and ineffectiveness at different companies I have worked with during last several years.

1. Effectiveness tips from K8S
trenches by Captain Obvious
Mikalai Alimenkou
@xpinjection
https://t.me/xpinjection_channel
https://xpinjection.com

3. Telegram channel
https://t.me/xpinjection_channel

4. Disclaimer
This talk is based on
personal experience

5. K8S is an orchestration flagman

6. Everything is so simple…

7. But reality is not always the same

8. Feel the difference
Deploy single
microservice to
Kubernetes
Deploy and maintain
production workload
to Kubernetes

9. All tips are really obvious

10. #1. Don’t start with K8S ;)

11. Focus on business first
Don’t invest into complex infrastructure from start
Use PaaS cloud solutions if possible
“Buy” more time for business focus
Containerize your services for flexibility
Experiment with K8S on local dev environments

12. #2. Develop Cloud Native services

13. Use best practices and enablers
Choose Cloud Native platform if possible
Follow best practices for 12+ factor applications
Use available enablers and libraries
Focus on the most important aspects:
- flexible configuration;
- full observability;
- stateless.
You could migrate anywhere later

14. #3. Avoid K8S maintenance

15. Delegate complexity
Build on managed K8S where possible
Try K8S wrappers instead of vanilla K8S
Use public Helm charts for dependencies
Try public K8S operators to delegate maintenance for
dependencies as well
Use service mesh if don’t want to rely on developers
level and use cross-language microservices

16. #4. Switch to declarative mindset

17. Forget imperative habits from the past
All K8S resources are just declarations
OK response on API call means resource is accepted
Avoid imperative commands for all types of action
Deployment may fail even at container level
Use extended monitoring to control actual state of the
cluster
Log kubernetes.io/change-cause for deployment
traceability

18. #5. Don’t be hype-driven maniacs
NO ROOM
FOR LEGACY!

19. Keep you platform as simple as possible
Service mesh is great but not for everybody
Every new component increases maintenance cost
Start with plain K8S manifests
Use templating and customization if more flexibility is
required
Switch to Helm only in complex cases

20. #6. Define clear roles/responsibilities

21. Establish clear DevOps boundaries
Infrastructure engineers responsible for low level
infrastructure (cloud, bare metal, mixed)
Dedicated K8S engineers responsible for K8S platform
Developers responsible for their services deployment,
configuration and support
SRE engineers responsible for SRE practices and toolset
establishment

22. More balanced development

24. #7. Implement business health checks

25. Don’t be lazy developer
Don’t rely on simple HTTP ping
Distinguish liveness and readiness probes
Be careful with returned HTTP status
Add business context to liveness and readiness probes
Implement and configure graceful shutdown where
possible

26. #8. Simplify dev environments

27. Provide environments for all needs
Share data storages between environments if possible
Make environment lightweight to use everywhere
(feature branches, pull requests, CI builds,
dev/qa/demo)
Share the same K8S cluster with isolated namespaces
Use node level cache for quick start
Automate environment setup for developers
Use CD tools like scaffold for development speed

28. #9. Don’t forget about limits

29. Defaults bring hidden issues
Define RAM/CPU requirements for all services
Limit resources usage per namespace
Be careful with default network policy
Complex network setup requires special skills
Pay attention to security guidelines
Apply strict access policies
Use dedicated clusters for isolated environments

30. #10. Not all nodes are equal

31. K8S is not about “commodity hardware”
Use labels for nodes to control deployment specifics
(storage type, node size, CPU capacity, etc.)
Carefully choose node size, taking into account all
sidecar containers and daemon sets
Save money on spot instances for dev environments
Don’t forget about latency and volume speed for data
storages
Use node level caches to speed up deployments

33. Summary and take aways
K8S is great orchestration tool, but not given for free
Containerized Cloud Native services run everywhere
New roles needed to set proper DevOps boundaries
K8S ecosystem is hype-driven and growing very quickly
Environment should be really easy to start
Everything is still working on hardware
Pay special attention to security and network

34. @xpinjection
https://xpinjection.com
https://t.me/xpinjection_channel