Slide Share of Veer Muchandi's presentation to the OpenShift Kubernetes Atlanta Meetup on 2017-08-31

1. @OpenShift
RHOpenShift
Container Security
Presenter: Veer Muchandi
Title: Principal Architect - Container Solutions
Social Handle: @VeerMuchandi
Blogs: https://blog.openshift.com/author/veermuchandi/

2. What Are Containers?
● Sandboxed application processes
on a shared Linux OS kernel
● Simpler, lighter, and denser than
virtual machines
● Portable across different
environments
● Package my application and all of
its dependencies
● Deploy to any environment in
seconds and enable CI/CD
● Easily access and share
containerized components
INFRASTRUCTURE APPLICATIONS
It Depends on Who You Ask
2

3. Understanding Containers

4. Virtualization vs Containerization

5. Container Host Security
Linux Technologies
- Namespaces for Isolation
- Seccomp
- Read Only Mounts
- SELinux
- MCS
- RHEL Atomic
Base Image build your applications - RHEL or RHEL-Atomic

6. Linux Containers Architecture

7. Kernel provides process isolation by creating separate namespaces for
containers
PID namespaces allow processes in different containers to have the same
PID
Network namespaces allows container to use separate virtual network
stack, loopback device and process space
Mount namespaces isolate the set of file system mount points
UTS namespaces isolate system identifiers – nodename and domainname
IPC namespaces isolate certain interprocess communication (IPC)
resources
User allow you to specify a range of host UIDs dedicated to the container.
A process can have full root privileges for operations inside the container.

8. Cgroups ensure that a single container cannot exhaust a large
amount of system resources
Cgroups allocate CPU time, system memory,
network bandwidth, or combinations of these
among user-defined groups of tasks

9. 9
SELinux and Multi Category Security
● Mandatory Access Control
● SElinux is a LABELING system
● Every Process has a Label
● Every file, Directory, System object has a Label
● Policy rules control access between labeled processes and labeled
objects
● The Kernel enforces the rules

10. 10
SELINUX - MAC - MCS - Process
system_u:system_r:container_runtime_t:s0
SElinux Policy module for the container
The OOTB SElinux policy container.te defines
what you can execute and access with the label
container_runtime_t
[root@osemaster ~]# ps -efZ | grep docker-containerd-shim-current
system_u:system_r:container_runtime_t:s0 root 3035 1479 0 Feb15 ? 00:00:01
/usr/bin/docker-containerd-shim-current
4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16
/var/run/docker/libcontainerd/4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16
/usr/libexec/docker/docker-runc-current

11. 11
SELINUX - MAC - MCS - Files
container_var_lib_t / svirt_sandbox_file_t
SElinux Policy module for the container
[root@osemaster ~]# ls -lZ
/var/lib/docker/containers/97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737
/
-rw-r-----. root root system_u:object_r:container_var_lib_t:s0
97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737-json.log
-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 config.v2.json
-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 hostconfig.json
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 hostname
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 hosts
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 resolv.conf
-rw-r--r--. root root system_u:object_r:container_var_lib_t:s0 resolv.conf.hash
drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 secrets
drwx------. root root system_u:object_r:container_var_lib_t:s0 shm

12. 12
SECCOMP - DROPPING PRIVILEGES FROM
CONTAINERS
CAP_SETPCAP
CAP_SYS_MODULE
CAP_SYS_RAWIO
CAP_SYS_PACCT
CAP_SYS_NICE
CAP_SYS_RESOURCE
CAP_SYS_TIME
CAP_SYS_TTY_CONFIG
CAP_AUDIT_WRITE
CAP_AUDIT_CONTROL
CAP_MAC_OVERRIDE
CAP_MAC_ADMIN
CAP_SYSLOG
CAP_NET_ADMIN
CAP_SYS_ADMIN
Modify process capabilities
Insert/Remove kernel modules
Modify Kernel Memory
Configure process accounting
Modify Priority of processes
Override Resource Limits
Modify the system clock
Configure tty devices
Write the audit log
Configure Audit Subsystem
Ignore Kernel MAC Policy
Configure MAC Configuration
Modify Kernel printk behaviour
Configure the network:
- Setting the hostname/domainname
- mount(),unmount()
- nfsservctl
- ….

13. 13
SECCOMP - REMOVE PRIVILEGES FROM
CONTAINERS
A root user inside a container
running in OpenShift has none of
the previous capabilities available!

14. 14
Read Only Mounts
/sys
/proc/sys
/proc/sysrg-trigger
/proc/irq
/proc/bus

15. Container Build Time Security
- Content in the containers, Trusted Containers
- Responsibility of maintaining Container Images
- Container Registry - RBAC, Public and Private Registries
- Scanning for Vulnerabilities
- Integrating Container Scanning and SAST into CI/CD
- Future Vulnerabilities
- Build Automation to rebuild during fixes

16. ● Red Hat Container
Registry
● Policies to control who
can deploy which
containers
● Certification Catalog
● Trusted content with
security updates
HOST OS
CONTAINER
OS
RUNTIME
APP
HOST OS
CONTAINER
OS
RUNTIME
APP
16
Image governance and private
registries
● What security meta-data is
available for your images?
● Are the images in the
registry updated regularly?
● Are there access controls
on the registry? How
strong are they?
Trusting Container Content

17. Red Hat provides Trusted Container Images
(registry.access.redhat.com)
Languages: PHP, Python, Ruby, Perl, Node.js, Java, .Net Core
Enterprise Grade JBoss Middleware: EWS, EAP, BPM, BRMS, RH SSO, Data
Grid, DataVirt, 3Scale
Databases: MySQL, Mongo, PostgreSQL, Maria
CICD: Jenkins
Partners Images: Container Certification by Red Hat.
Publish fixes to the content in this registry and notify you.

18. 18
Red Hat Registry: Container Health Index
https://access.redhat.com/articles/2803031

19. Private Registries
What if public registry (like DockerHub) is down?
Enterprise security policies may not allow your Container Images to be pushed
outside.
Caching images in Private Registry
- Atomic Registry with RBAC
- JFrog Artifactory
- Docker Trusted Registry

20. 20
RESTRICT WHERE YOUR CONTAINERS COME
FROM
- name: allow-images-from-internal-registry
onResources:
- resource: pods
- resource: builds
matchIntegratedRegistry: false
- name: allow-images-from-dockerhub
onResources:
- resource: pods
- resource: builds
matchRegistries:
- docker.io

21. Responsibilities in the new paradigm
Operations
- Maintain Trusted OS Base Images (RHEL, RHEL-Atomic)
- Ensure these are good with Linux Kernel
System Engineers/Architects
- Maintain Middleware Images
- Control build process - S2I, CICD
- Reference Architectures with Trusted middleware
Development teams
- Write code that layers on approved images
- Ensure code is clean, opensource software vullnerabilities are handled

22. Container Scanning
Frequency
Scan containers as soon as they are created
Scan containers that get into registry
Ongoing basis- Identify any new vulnerabilities
“Deny execution of containers that are vulnerable”

23. 23
Scanning containers when created
https://www.youtube.com/watch?v=65BnTLcDAJI
Run the scan at
this point

24. Scanning Tools
Atomic Scan - configurable with different scanners
OpenScap Scanner - also built into CloudForms
BlackDuck
JFrog XRay

25. 25
Identify Vulnerabilites on an ongoing basis

26. 26
Vulnerable? CloudForms takes Action!

27. 27
Prevent Image From Running

28. 28
Build Automation - when there are updated
images

29. Container Platform Security
- Authentication
- Authorization
- Running Privileged Containers
- Network Isolation
- Securing communication between hosts
- Storage Security
- Deploying workloads to specific hosts of your choice

30. API Authentication
• OAuth Access Token
– Obtained from OAuth server using endpoints
– Sent as
• X.509 Client Certs
– HTTPS connection
– API Server verifies against a trusted cert authority bundle
– API Server creates and distributes certs to Controllers to authenticate themselves
• Request with invalid token or cert gets a 401
• No cert or token gets system:anonymous user and system: unauthenticated virtual group
<master>/oauth/authorize <master>/oauth/token
Authorization: Bearer access_token=…​

31. Authentication
OAuth
Client
OAuth
Server
Master
Client Types:
openshift-web-console
openshift-browser-client
openshift-challenging-client
Identity
Provider
1. Authentication
Request
2. Determine
Identity
3. Access token
Configured Identity
Providers:
- LDAP
- GitHub
- GitLab
- Google
- OpenID Connect
- HTPasswd
Authentication Requests:
<master>/oauth/authorize
WWW-authenticate
<master>/oauth/token/request

32. Authorization
RoleBinding associates Users/Groups with Roles
- Cluster RoleBindings
- Project RoleBindings

33. 33
Container Deployment Permissions (SCC)

34. Network Isolation with Network Policy Objects
Add Network Policies To
Allow Specific Incoming
Traffic

35. 35
Secured Communications between Hosts
Secures cluster communications
with IPsec
● Encryption between all
Master and Node hosts (L3)
● Uses OpenShift CA and
existing certificates
● Simple setup via policy defn
○ Groups (e.g. subnets)
○ Individual hosts
Master
P1
Nodes
P2
172.16.0.0/16

36. 36
Secure storage by using
● SELinux access controls
● Secure mounts
● Supplemental group IDs for
shared storage
Securing Storage attached to Containers

37. 37
Isolate Workloads by labeling Nodes
Node 1
east
Node 2
east
Node 1
west
Node 2
west
Master /
Scheduler
$ oadm new-project myproject
--node-selector='type=user-node,region=east'
pod pod

38. 38
Securing federated clusters
across data centers or
environments
● Authentication and
authorization
● API endpoints
● Secrets
● Namespaces
FEDERATED CLUSTERS (Roadmap)
ROLES & ACCESS MANAGEMENT
Source: Building Globally Distributed Services using Kubernetes Cluster Federation. October 14, 2016

39. 39
FEDERATED CLUSTERS (Roadmap)
ROLES & ACCESS MANAGEMENT
Source: Building Globally Distributed Services using Kubernetes Cluster Federation. October 14, 2016
API
Repl Ctrl
Ubernetes
state
API
Repl Ctrl
Kubernetes Cluster
state
API
Repl Ctrl
Kubernetes Cluster
state

40. Application Security - Running as Containers
- API Management
- Red Hat Single Sign-on
- HTTP(s)
- Securing sensitivedata as Secrets
- Calling services running outside a Container Platform

41. 41
Container platform & application APIs
● Authentication and authorization
● LDAP integration
● End-point access controls
● Rate limiting
API Management

42. No SSL
My App
http://myapp.mydomain.com
Router
My App
My App

43. Edge Termination
My App
https://myapp.mydomain.com
Router
My App
My App

44. Passthrough Termination
My App
https://myapp.mydomain.com
Router
My App
My App

45. Reencrypt Termination
My App
https://myapp.mydomain.com
Router
My App
My App

46. Secrets
Sensitive Info: Passwords, Client Config files, dockercfg etc
used by application containers provided as secrets
Never come to rest on Nodes
Stored in ETCD. Encrypted starting OCP 3.6.1
kubernetesMasterConfig:
apiServerArguments:
experimental-encryption-provider-config:
- /path/to/encryption.config

47. 47
Calling External Services using Egress Router
The OpenShift egress
router runs a service that
redirects egress pod traffic
to one or more specified
remote servers, using a
pre-defined source IP
address that can be
whitelisted on the remote
server.
NODE
IP1
EGRESS
ROUTER
POD
IP1
EGRESS
SERVICE
INTERNAL-IP:8080
EXTERNAL
SERVICE
Whitelist: IP1
POD
POD
POD
...
- name: EGRESS_DESTINATION
value: |
80 tcp 1.2.3.4
8080 tcp 5.6.7.8 80
8443 tcp 9.10.11.12 443
13.14.15.16
...

48. Signing
GPG Keyring
policy.json
Signed OpenShift Node
OpenShift Registry
image-auditor role
gpg2 --gen-key
atomic push --sign-by
oc adm verify-image-signature
Cluster Admin
Unsecure Container
Ansible for Key Propagation
OpenShift Node
Tenant
Select from Content
imageStreams
templates
images
if/then
sandboxProject
productionProject
nodeLabelSelector
nodeLabelSelector
1
2
3
1
Tenant Usage Steps
Cluster Admin Setup Steps
2
3
Upcoming!!

49. Thank you!!