In this session, we'll unravel the core and essential pillars of any 'secure' Kubernetes cluster, that you absolutely can't ignore if you are running Kubernetes in production (or plan to). You'll discover the key concepts and strategies pivotal to safeguarding your Kubernetes environments. Our focus will be on practical, real-world applications, demystifying complex security challenges. Regardless if you are from a large organisation or from a small start-up, a seasoned DevOps professiyou will walk away with foundational knowledge and actionable insights, ready to implement stronger security measures in their Kubernetes deployments. Whether you're a seasoned DevOps professional or new to the cloud native arena, this talk will enhance your understanding of Kubernetes security, ensuring you're prepared for the evolving landscape of cloud native security.

1. The Art of Cloud Native Defense
PILLARS OF KUBERNETES
SECURITY

2. Intro

3. About Me - Jacopo Nardiello
How to get in touch with me: @jnardiello
Reach out if you want to talk about:
- Infrastructure Automation & Cloud Native stuff (not just Kubernetes)
- Practices, TDD, Testing
- How to structure teams and your organization (Thanks! Stockholm Syndrome)
Feedback, feedback, feedback & I love to discuss. Find me around conference venue and let’s talk!
Founder & CEO SIGHUP
- CNCF Ambassador
- Linux Foundation Europe Founding Advisory Board
- Infra engineer, automation freak, cowboy entrepreneur
- Dad of 2 kiddos

4. About us
SIGHUP IS AN ENGINEERING FIRM AND ENTERPRISE OPEN SOURCE COMPANY
FOCUSED AND SPECIALIZED IN KUBERNETES AND INFRASTRUCTURE AUTOMATION.
We help enterprises and organizations during their Cloud Native journey towards software-defined
infrastructures and DevOps, leveraging the competitive advantage provided by automation.
We have been crazy enough, since 2017, to create Kubernetes Fury Distribution - a maintained
Kubernetes-based platform providing Enterprise-grade support on upstream CNCF technologies.
In March 2024, we have been fully acquired by ReeVo - Secure Cloud & CyberSecurity

5. First thing first…
Thank you for surviving the day (and for attending this session)
Please:
- Rate my talk
- Provide constructive feedback
- Don’t be a*hole
- I love human interactions (I’m a devops weirdo!), if you
have questions or there’s anything you want to discuss,
reach out after the talk!

6. https://community.cncf.io/kcd-italy/
KCD Italy

7. What you will learn in this session
- Understand exactly what “Security” means in the context of Cloud
Native and Kubernetes
- What matters, what doesn’t and where to focus
- Provide you with an high-level, yet actionable framework and
learning topics
- We will try to keep it open and simple
- I will not sell you anything
- I will not position any paid enterprise tool
- I will focus on open standards and avoid all confusing or exotic solutions/tools

8. The state of Cloud Native
- The great orchestration war (2016 a.c.)
- Kubernetes as a new standard and the rise of a new foundation:
CNCF (2017 a.c.)
- CNCF landscape and a new ecosystem of emerging standards
(2018-2019 a.c.)
- The mighty landscape consolidation (2019-2021 a.c.)
This brings us today…

9. The state of Cloud Native
2022/2023, the year of Platforms
- IDPs
- A natural evolution of all the things we
called devops for the past 10y
- Fueled by remote work, supposedly
leaner processes and digital
accelerationism demanded by
management
- Kubernetes-as-a-silver-bullet for digital
transformation (whatever that means)
The good news: we are in prod!
The bad news: we are in prod.
Kubernetes is reaching maturity.
Question is..

10. The maturity:
Credits: someone on linkedin

11. Kubernetes is reaching maturity
But there’s one thing that didn’t change…
WHAT NOW?
Credits: someone on linkedin

12. Sidenote: 💩 is hitting the fan
- World is getting at war 😔
- New regulatory push 󰎾
We (the infra people) can’t
ignore anymore security as a
key component of the
architectures we deliver and
maintain.

13. 2024: Chaos is now expanding to Security
teams
Standard security practices:
- Threat modeling
- Red/Blue/Purple/Whatever teams
- Threat analysis and response
Welcome to hell:
- Ephemeral elastic workloads
- Api-driven architectures
- New paradigms that didn’t exist before (IaC, PaC, Operators & CRDs)
- Microservice architectures pushed to the extreme
These new challenges drifts significantly from the standard set of skills of traditional
security teams.

14. What is Cloud
Native Security?

15. A GPT Definition

16. A Definition
Key Ideas:
- Incorporate security practices into SDLC
- Security as a FOUNDATIONAL component
- Beyond securing cloud infrastructure

17. Identifying Key Macro Groups
There are 3 macro areas to cover when it comes to Kubernetes
Security:
- Artefacts and dependencies management
How to track your dependencies (both application and system) and vulnerabilities.
How to patch them and reduce the probability of shipping compromised software.
- SDLC & pipelines
How to engineer your CI/CD pipelines so that they respect both your business and
regulatory requirements, while reducing the probability of shipping compromised
software to production.
- Security of Kubernetes workloads at runtime
Secure your productive and non-productive environments at runtime
Each macro-area have its own unique set of challenges and
solutions.
Supply Chain

18. Pillars of Cloud Native
Security

19. A description of the Golden Pillars of Cloud Native
Security
Follow the value chain
While we won’t deep dive on each pillar, we will focus on the key aspects and
provide a checklist of important stuff you need to address.

20. By definition your docker images isolate all the dependencies into a
single, standardized, artefact.
There are three set of dependencies you must be aware of:
- Your application dependencies (software)
- Your system dependencies
- The dependencies you don’t know you have, embedded
into your base images
Key critical facts to handle the security around your artefacts and
dependencies:
- Ensure you have your SBOM (Software Bill of Material)
Correctly track all your application and system container dependencies
- Use secure base images and invest in your own
Ensure that your standard base image catalog meets your security
requirements (community images might not be enough).
https://github.com/wolfi-dev
https://sighup.io/secure-containers/
https://www.chainguard.dev/chainguard-images
- Sign your images
Use technologies like cosign, notary, or more recently openpubkey to sign
your own artefacts and prevent tampering by malicious actors.
Artefacts & Dependencies Management
Investigating Pillar 1 and 2

21. Safe builds & pipelines
Pillar 3
Secure pipelines have been for a long time the holy grail of anything
devops/devsecops/secdevops/whateverops.
At @SIGHUP we have our own reference architecture (pictured left)
and implementation, vendor and technology agnostic. The key
universal role of cicd pipelines is one of gatekeeping and ensuring
quality and regulatory compliance.
Key Architectural aspects:
- Vulnerability checks on both your code and dependencies
https://trivy.dev
- Artifacts build and signing (see Pillars 1 & 2)
- Push to safe, certified, registries
- Linting, validation and other quality gatekeeping
- Policy enforcing, our tools of choice for this are OPA and
Kyverno, having a look at otterize as a new point-of-view on
policies management in Kubernetes
https://www.openpolicyagent.org
https://kyverno.io

22. In-cluster Security
Pillars 5-7, Starting from the basics
Basic security in Kubernetes can be achieved using the native components present in any
conformant Kubernetes cluster, more specifically you must put great effort in correctly using the
following primitives:
- RBAC
Set of permissions on Kubernetes objects about “who can do what” on which namespace.
- A CNI plugin that will introduce support to Network Policies
Regulating in-cluster network interactions and packet flow.
- Namespaces and quotas
Namespaces introduce segmentation on your cluster, introducing base configurations, quotas and rbac access to
resources
- Pod Security Standards & Admission
These define isolation policies for pods running into your cluster at namespace level
- Ingress tuning and exposure
Ingresses are meant to expose your service to the outside world. Depending on which ingress-controller you are
working with, you can enforce rules, validation and much more on incoming requests.
- ETCD data encryption and backups (self-managed Kubernetes only)
As ETCD holds the state of your cluster, it gets mandatory to make sure data is encrypted whenever stored.
RTFM:
https://kubernetes.io/docs/concepts/security/

23. Not-so-good-defaults
- Stored in base64 (basically plaintext) within Kubernetes etcd
- Lack of automated rotation mechanisms, increasing the risk of credential misuse or compromise
over time.
- Insufficient access controls and auditing
- Difficulty in managing secrets at scale, especially in distributed and complex environments.
Pay attention to your secrets

24. Pay attention to your secrets
What you really want: A secret manager!
- Management of all secrets (tokens, certificates, passwords) through a dedicated solution.
- Secrets stored at rest using strong symmetric cryptography.
- Granular access controls, allowing only authorized users or services to access specific secrets.
- Native integration with CI/CD pipelines and deployment workflows to securely provision and manage secrets throughout the
application lifecycle.
- Automated rotation, rollback, versioning and expiration policies to regularly update secrets and mitigate the risk of long-term
exposure.
- Integration with external identity providers and authentication mechanisms for enhanced security and identity management.

25. In-cluster Security: Advanced Topics
- Advanced runtime security
- Secure Container runtimes
- Tools to monitor privilege escalation, Storage and files exfiltration
- Anomaly detection on kubernetes audit logs, processes syscalls, PID
monitoring (to mention a few)
- Advanced dynamic secrets managements
Secrets are a first-class citizen in Kubernetes and while It’s not within the scope
of this presentation if you should use Kubernetes Secrets or not, running
secured workloads in production requires you to integrate your cluster secrets
management with external ad-hoc advanced tools.
https://www.conjur.org
https://secrets-store-csi-driver.sigs.k8s.io
- Policy controllers and admission webhooks
Kubernetes policies can become a super powerful tool to enforce security and
compliance checks, with several comprehensive ecosystems. Relevant projects
for this:
https://github.com/open-policy-agent/gatekeeper
https://github.com/sighupio/gatekeeper-policy-manager
https://kyverno.io
https://otterize.com
- Securing north-south / east-west
network traffic
This can be achieved in different ways or different
set of technologies, ranging from traditional service
meshes or newer ebpf-based CNIs.
- IAM and Identities
Both user identities and workloads identities are
crucial in ensuring correct validation of workloads.
SPIFFE and SPIRE to the rescue.
https://spiffe.io
- Workloads Isolation, multi-tenancy and
cluster isolation
Let’s dive into the key aspects of in-cluster security to highlight the main things we should pay
attention to when hardening any Kubernetes-based infrastructure:
https://falco.org
https://katacontainers.io

26. The case for multi-tenancy, workloads and cluster
isolation
- Soft multi-tenancy
- Virtualized control planes
- Dedicated clusters (managed or self-hosted)
Credits to pineapple pizza on unsplash

27. - Kubernetes is NOT multi-tenant
- You must/can implement
soft-multitenancy using native k8s
primitives
PRO
- Can be a great choice for partitioning
clusters on teams and tenants
- Can be leveraged within both
self-hosted and managed Kubernetes
- Great for platform engineering
CONS
- You can still (and likely will) be bitten by
noisy neighbours in weird ways
- From a security standpoint, there’s no
workload isolation beyond logical
isolation implemented with policies,
rbac and namespaces
- Not particularly recommended for
mission critical environments
https://github.com/clastix/capsule
The case for multi-tenancy, workloads and cluster
isolation
- Multi-tenancy
- Virtualized control planes
- Dedicated clusters (managed or self-hosted)

28. Meta-clusters of control planes, leveraging CAPI or virtual
kubeletes to automate cluster-creation
PRO
- It’s a really cool concept
- It leverages open standards with profuse efforts
from the community
- It ensures workload isolation and a high degree of
automation on multi-cluster management
- Stands on the shoulders of giants
CONS
- I feel like adoption is still in its infancy
- Not battle-tested
- Does not guarantee full cluster isolation
Promising technology, under heavy development and hence
not recommended if you MUST ensure full cluster isolation
in regulated environments.
If you want to play around:
- vclusters by loft
- Kamaji by Clastix
The case for multi-tenancy, workloads and cluster
isolation
- Multi-tenancy
- Virtualized control planes
- Dedicated clusters (managed or self-hosted)

29. Fully dedicated clusters (self-hosted or managed) are the
only solution, to date, that I feel like recommending if you
need full isolation for workloads and infrastructure.
PRO
- Full isolation
- Full complaints and strong security can be
applied
- Full configurability and customization of the
solution
CONS
- You need operators knowing what they are doing
- High operational burden
- You can automate, to a certain extent
This is where we (SIGHUP) shine, let me promote a bit our
own KFD and furyctl:
https://docs.kubernetesfury.com/docs/quickstart/quickstart
The case for multi-tenancy, workloads and cluster
isolation
- Multi-tenancy
- Virtualized control planes
- Dedicated clusters

30. Infrastructure Security
Infra hardening & Cloud configurations
Key recommendations:
- Private control planes
- Ensure as much as possible OS
immutability
- Verify signatures on Kubernetes
executables
- Make sure that your network
configurations allow for data
encryption for both traffic and storage
consumption
- If you are on cloud, managed services
and integrations might be a good idea
Don’t know where to start?
Here is a practical IaC example with sane
configurations:
https://github.com/sighupio/fury-eks-installer

31. The ugly duckling of Cloud Native Security
Pillar 8 - Compliance & Regulation
There are currently no standardised solutions enabling human operators to monitor and validate cluster resources state, violations and ensure
compliance.
We have been therefore developing over the past few months Permission Monitor, soon available with KFD or standalone on any
cncf-compliant distro. If you are interested, ping me to test it out.

32. Conclusions

33. It’s a wrap
What we have learnt:
- We have shared with you our core set of guiding pillars and
principles when it comes to securing Kubernetes-based
infrastructures.
- You now know what matters and what to focus on as a
starting learning point
- You have an actionable list of items, projects, or sets of
technologies to get started with, for each core pillar we
have explored.
- I’m aware that this is just scratching the surface, let’s talk
beer (or spritz) & talk 🍻

34. PS: It’s a 💩ton of work, I know.
Imagine, we do all this stuff for a
living..

35. @jnardiello
https://sighup.io