Sharing the experience about how to harden the Kubernetes cluster

1. Harden Your
Kubernetes Cluster

2. Hello!
I am smalltown
Maicoin Site Reliability Engineer
Taipei HashiCorp UG Organizer
AWS UG Taiwan Staff
2

3. 3

4. 4

5. Network RuntimeImage Credential PaC
5

6. Network RuntimeImage Credential PaC
6

7. 7
Image Build Practice (1/2)
◎ Minimal Base Images
◎ Least Privileged User
◎ Use Fixed Tags
◎ Sign/Verify Images (Docker Notary)
◎ Don’t Contain Sensitive Information
Ref

8. 8
Image Build Practice (2/2)
◎ Use COPY Instead of ADD
◎ Use Metadata Labels
◎ Multi-Stage Build Small/Secure Images
◎ Use Linter (hadolint)
◎ Find/Fix/Monitor Vulnerabilities
Ref

9. How to Find Vulnerabilities?
◎ Of Course! Scanning!
◎ Commercial V.S. Open Source
◎ Open Source: Anchor, Clair, Trivy...
9

10. Scan Coverage
◎ OS Packages
○ Alpine, Red Hat Universal Base Image, Red Hat
Enterprise Linux, CentOS, Debian and Ubuntu
◎ Application Dependencies
○ Bundler, Composer, Pipenv, Poetry, npm, yarn and
Cargo
10

11. Try to Use It! (1/2)
11
~$ docker run --rm -v $HOME/Library/Caches:/root/.cache/ aquasec/trivy
python:alpine3.10

12. Try to Use It! (2/2)
12
~$ docker run --rm -v $HOME/Library/Caches:/root/.cache/ aquasec/trivy
golang:1.12.9-buster

13. DevSecOps
◎ Integrate with CI/CD Framework, e.g. if Find
Critical Vulnerability then Fail the Pipeline
◎ Don’t Forget Store the Vulnerability
Database/Data
◎ Use Json Format Output to Generate
Report
13

14. 14
Competitive Analysis
Ref

15. Regular Vulnerability Assessment
◎ Process of Defining, Identifying, Classifying
and Prioritizing Vulnerabilities
◎ Information on the Security Weaknesses in
its environment
◎ Assess the Risks Associated with those
Weaknesses and Evolving Threats
15

16. Network RuntimeImage Credential PaC
16

17. Kubernetes Secret
17
Ref

18. HashiCorp Vault
◎ Secures, stores, and tightly controls access
to tokens, passwords, certificates, API keys
◎ Handles leasing, key revocation, key rolling,
and auditing
18

19. Ideal Credential Lifecycle
Service is Accessed
Application
1. Request Access Credential (Running)
2. Use the Credential to Access Service
3. Revoke the Credential
Credentials Only
Exist in Memory
19

20. Authentication
◎ Vault provide various auth method
○ Tokens, AppRole
○ AWS, Azure, Google Cloud
○ LDAP, GitHub ...etc
20

21. Authorization
◎ Vault store credentials like key/value DB, e.g.
○ /secret/stag/database/admin
○ /secret/prod/database/admin
◎ Hence, predefined policy grant appropriate
permission, e.g.
path "secret/stag/database/admin" {
capabilities = ["read"]
} 21

22. Dynamic Credentials
◎ Vault support many secret backend
○ AWS, Azure, GCP, Database...etc
◎ Take database for example, you could generate
dynamic database credentials
$ vault read database/creds/my-role
Key Value
--- -----
lease_duration 1h
password 8cab931c-d62e-a73d-60d3-5ee85139cd66
username v-root-e2978cd0- 22

23. HashiCorp Vault Workshop：幫
Credentials 找個窩
◎ Slide: Here
◎ GitHub: Here
23

24. Network RuntimeImage Credential PaC
24

25. Connection Between Pods/Endpoints
Out In Internet
- Port
- IP, FQDN
- Protocol
25

26. What is The Problem?
◎ By Default, Pods are Non-Isolated; They
Accept Traffic from any Source.
26

27. 27
Kubernetes Network Policy
podSelector
Ref
- podSelector
- namespaceSelector
- ipBlock
- podSelector
- namespaceSelector
- ipBlock
Ingress (From)
Egress(To)
Protocol
Port

28. 28
How to Enable Network Policy
Ref

29. Connection Between Pods/Endpoints
Out In Internet
- Port
- IP, FQDN
- Protocol
29

30. What is The Problem?
◎ MITM (Man-In-The-Middle Attack)
30

31. Istio
31

32. 32
Istio with Mutual TLS
Ref

33. 33
Policy & DestinationRule
Ref

34. Istio with Kiali
34

35. Network RuntimeImage Credential PaC
35

36. Runtime Security Tools
Enforcement
Auditing
36

37. Enforcement Runtime Security Tools
◎ Using the Policy to Change the Behavior of
a Process by Preventing System Calls from
Succeeding
○ Pod Security Policy
○ AppArmor
○ Seccomp
37

38. 38
Pod Security Policies
◎ Define a Set of Conditions that a Pod Must
Run With in Order to be Accepted Into the
System
○ Usage of volume types
○ Usage of host networking and ports
○ The user and group IDs of the container
○ ... Ref

39. How to Use PSP
(Cluster)Role
(Cluster)RoleBinding For Pod
For Deployment, ...
Create
Use
With
39

40. When a Pod Created...
◎ The PSP is Enabled, Pod Don’t Use the PSP
-> Fail
◎ The PSP is Enabled, Pod Can Use the PSP,
Pod Don’t Follow PSP -> Fail
◎ The PSP is Enabled, Pod Can Use the PSP,
Pod Follow PSP -> Success
40

41. Auditing Runtime Security Tools
◎ Using the Policy to Monitor the Behavior of
a Process and Notify when its behavior
steps outside the policy
○ Falco
○ Auditd
41

42. Falco
◎ A Behavioral Activity Monitor Designed to
Detect Anomalous Activity in Your
Applications
42

43. How Falco Take Effect
43

44. Falco Rules
- macro: access_file
condition: evt.type=open
- rule: program_accesses_file
desc: track whenever a set of programs opens a file
condition: proc.name in (cat, ls) and (access_file)
output: a tracked program opened a file (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: INFO
44

45. Falco V.S. Others
Falco Others
User Space Kernel Level
Kill/Suspend/Starve the Falco
Process
Replacing a loaded set of
policies or BPF program in the
kernel is probably more difficult
Much Richer Set of Information
Powering Its Policies
Those Types of Policies are
More Difficult to Implement at
the Kernel Level
45

46. Network RuntimeImage Credential PaC
46

47. Policy as Code
YAML Engineer 4NI ?!
47

48. Open Policy Agent
◎ A general-purpose Policy Engine that Helps Solve Use
Cases Ranging From Authorization and Admission
Control to Resource Placement.
○ Kubernetes Admission Control
○ HTTP API Authorization
○ Remote Access
○ Data Filtering with Partial Evaluation
48

49. OPA Flow
ContextPolicy
API Authorization
API
Retrieve Data from Target Service
49

50. Finally Not YAML
◎ Rego was Inspired by Datalog, Which is a
Well Understood, Decades Old Query
Language
◎ Rego Queries are Assertions on Data Stored
in OPA
50
Ref

51. 51
Rego Sample Code
package kubernetes.admission
import data.kubernetes.namespaces
deny[msg] {
input.request.kind.kind == "Ingress"
input.request.operation == "CREATE"
host := input.request.object.spec.rules[_].host
not fqdn_matches_any(host, valid_ingress_hosts)
msg := sprintf("invalid ingress host %q", [host])
} Ref

52. Thanks!
Any questions?
You can find me at:
facebook.com/smalltown0110
smalltown@awsug.tw
52

53. We’re Hiring!!
Software Engineer in
Test
Software Engineer
53