Presented by Rohini Rajaram at the Fedex Cloud-Native Conference in Pittsburgh on July 12th, 2019.

1. © Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
Pivotal Container Service
Overview
Rohini Rajaram
Sr. Platform Architect, Pivotal
rrajaram@pivotal.io
July 2019

2. Why PKS?
Main Capabilities

3. Operational
Efficiency
● Employ 500:1 developer
to operator ratio
● Perform zero-downtime
upgrades
● Runs the same way
on every public/private
cloud
Developer
Productivity
Comprehensive
Security
● Accelerate feedback
loops by improving
delivery velocity
● Focus on applications,
not infrastructure
● Give developers the
tools and frameworks
to build resilient apps
● Adopt a defense-in-
depth approach
● Continuously update
platforms to limit
threat impact
● Apply the 3 R’s →
repair, repave, rotate
● Run platforms that
stay online under
all circumstances
● Scale up and down,
in and out, through
automation
● Deploy multi-cloud
resilience patterns
High Availability
Platform Team Delivering Real Value

4. Can we realize these benefits for other workloads too?
MONOLITHIC
APPLICATIONS
More
MICROSERVICES
Stateful
or
Clusters
CONTAINERS COTS
DATA SERVICES
MICROSERVICES
MONOLITHIC
APPLICATIONS
Some Some .NET APPLICATIONS

5. Can we realize these benefits for other workloads too?
MONOLITHIC
APPLICATIONS
More
MICROSERVICES
Stateful
or
Clusters
CONTAINERS COTS
DATA SERVICES
MICROSERVICES
MONOLITHIC
APPLICATIONS
Some Some .NET APPLICATIONS

6. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
vSphere AzureGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Concourse
Repair
— CVEs
Repave Rotate
— Credhub
PCF

7. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
Java | .NET | NodeJS
Pivotal Application
Service (PAS)
Application Code & Frameworks
Buildpacks | Spring Boot | Spring Cloud |
Steeltoe
>cf push
WE build the container
vSphere AzureGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Github
Concourse
Concourse
Continuous
delivery
Repair
— CVEs
Repave Rotate
— Credhub
PCF

8. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
Java | .NET | NodeJS
Pivotal Application
Service (PAS)
Application Code & Frameworks
Buildpacks | Spring Boot | Spring Cloud |
Steeltoe
Elastic | Packaged Software | Spark
Pivotal Container
Service (PKS)
>cf push >kubectl run
YOU build the containerWE build the container
vSphere AzureGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Github
Concourse
Concourse
Continuous
delivery
Repair
— CVEs
Repave Rotate
— Credhub
PCF

9. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
Java | .NET | NodeJS
Pivotal Application
Service (PAS)
Application Code & Frameworks
Buildpacks | Spring Boot | Spring Cloud |
Steeltoe
Elastic | Packaged Software | Spark
Pivotal Container
Service (PKS)
>cf push >kubectl run
YOU build the containerWE build the container
vSphere AzureGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Github
Concourse
Concourse
Pivotal Services
Marketplace
Pivotal and
Partner Products
Continuous
delivery
Public Cloud
Services
Customer
Managed
Services
Repair
— CVEs
Repave Rotate
— Credhub
PCF

10. Overview
Pivotal Container Service

11. > kubectl
Storage NetworkingCompute
Kubernetes Dashboard
Dev / Apps IT / Ops
App User
Kubernetes is a Runtime for Containerized Workloads

12. Storage NetworkingCompute
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
Container Image
Registry
App Monitoring
App Logging
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster
Provisioning
Provision & Scale
Command Line
/ API
Management
GUI
Monitoring GUI
...but Kubernetes alone is not enough for enterprises

13. Storage NetworkingCompute
Pivotal Container Service (PKS) provides what’s missing
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster
Provisioning
Provision & Scale
PKS Control Plane
> pks
Operations Manager
vRealize Operations*
Container Image
Registry
vRealize
LogInsight

14. on any Cloud
Dev / Apps
App User
IT / Ops
> kubectl
Kubernetes Dashboard
Load Balancing / Routing
Container Image
Registry
OS Updates
OS Images
K8S Updates
K8S Images
Log & Monitor
Recover & Restart
Backup & Restore
External
Data Services
Cluster
Provisioning
Provision & Scale
vRealize
LogInsight
PKS Control Plane
> pks
Operations Manager
vRealize Operations*

15. What PKS adds to Kubernetes
PKS value-
added
features
Built into
Kubernetes
Multi-container pods
Stateful Sets of pods
Persistent disks
Single tenant ingress
Pod scaling and high availability
Rolling upgrades to pods
Cluster provisioning and scaling
Embedded, hardened Operating System
Monitoring and recovery of cluster VMs and processes
Rolling upgrades to cluster infrastructure
Secure multi-tenant ingress
Secure container registry

16. “On a good day our DIY k8s solution could provision a cluster
in 45 minutes. On a bad day it would fail to deploy ~30
times” - IT Operator

17. Cover w/ Image
Turnkey solution. All the things you need to use
and operate a container runtime. In one package.
On every cloud.
Enterprise readiness, control and security.
Continually updated platform, embedded OS,
secure container registry, policy-driven networking,
integrated IdM. Controllable and customizable by
service plans.
Developer empowerment. Consume app services,
popular tools “just work” with vanilla Kubernetes via
constant compatibility. Developers get the
Kubernetes they want, for their choice of workloads.
Multi cloud. Run it on any infrastructure of choice.
On premises or in the public cloud.
Our investment themes

18. Is your network ready?
(How mature is your SDN?)
Answer: Probably not; if you require
tickets/manual process for network or
firewall rules, you aren’t ready
● Microsegmentation with NSX-T
● Automated IP allocation and
load balancer provisioning
● Monitoring & troubleshooting
with familiar VMware tooling
How many clusters are
you going to need?
Answer: More than one required to
provide true multi-tenancy
● Self-service, on-demand
provisioning of clusters
● Pre-defined T-shirt size
clusters
● Scale clusters up and down
How many times does
K8s release per year?
Answer: Major releases quarterly;
minor releases/patches frequently
● Automated upgrades on-
demand with BOSH
● Automatic patching with
Concourse pipelines
● Self-healing nodes on failure
Upgrades Multi-tenancy Network
PIVOTAL’S OPINION PIVOTAL’S OPINION
Three Questions
PIVOTAL’S OPINION

19. What is PKS?
Main Capabilities

20. BOSH
Reliable and consistent operational experience for any cloud.
BOSH
Harbor
NSX-T
Kubernetes
K8s Cluster
K8s Cluster
K8s Cluster
PKS Control Plane
Use the PKS CLI and API to
create, operate, and scale your
clusters.
VMware GCP Azure Openstack AWS
PKSControlPlane
Built with open-source
Kubernetes
Constant compatibility with the
latest stable release Kubernetes
—no proprietary extensions.
Harbor
An enterprise-class container registry.
Includes vulnerability scanning,
identity management, and more.
NSX-T
Network management, security, and
load balancing out-of-the-box with
VMware NSX-T. Multi-cloud, multi-
hypervisor.
Enterprise-Grade Kubernetes

21. Multi-Cloud with BOSH + CPI

22. PKS does for your Kubernetes
what
Kubernetes does for your apps

23. Main Capability #1
BOSH

24. BOSH is an open source
tool for release
engineering, deployment,
lifecycle management,
and monitoring of
distributed systems.

25. ➔ Health monitoring (server & processes)
➔ Self-healing w/ Resurrector
➔ Storage management
➔ Rolling upgrades with canaries
➔ Easy scaling of clusters
➔ Repeatability and Consistency
➔ Packaging w/ embedded OS
➔ Server provisioning on any IaaS
➔ Software deployment across AZs

26. Main Capability #2
Multi Cloud

27. Pivotal Application
Service (PAS)
Pivotal Container
Service (PKS)
vSphere
Azure &
Azure StackGoogle CloudAWSOpenstack
Pivotal Services
Marketplace
Multicloud
CPI (15 methods)
Embedded OS
(Windows & Linux)
NSX-T
consistent
operational
experience

28. Availability Zone B
Availability Zone A
Bosh distributes deployments across AZ’s
Availability Zone A
Availability Zone B
Kubelet
Kube-proxy
Pod
Pod
K8s Worker
Pod
API Server
Kube Scheduler
K8s Master
Controller
Manager
Kubelet watches and restart containers
Bosh agent watches and restarts processes
Bosh director watches and restarts nodes
Bosh agent
Bosh agent
Bosh Health
Manager
PKS Health Management

29. Main Capability #3
Multi Cluster

30. Tenancy
Multi-tenancy models remain weak in
Kubernetes alone
We provide solutions for this today

31. Two models supported
Multi-tenant clusters
● Leverage Kubernetes namespaces
Limitations with Kubernetes alone
● Noisy neighbors (workloads can affect other
tenants)
● Share the same network
● Share DNS
● Shared Configuration
● ...
We add
● Network microsegmentation with NSX-T
○ Eliminating “Share the same network”
Multi (Single-tenant) clusters
It is having an API for creation and management
that enables this!!!
● Every tenant gets their own cluster
Addresses limitations
● Single tenant worker VMs (depend on the
hypervisor to ensure host is properly shared)
● Every cluster has own network segment
● Every cluster has own DNS
● Every cluster has own configuration
● ...
No other “on
prem” solution
has this!!!

32. Two models supported
Multi-tenant clusters
● Leverage Kubernetes namespaces
Limitations with Kubernetes alone
● Noisy neighbors (workloads can affect other
tenants)
● Share the same network
● Share DNS
● Shared Configuration
● ...
We add
● Network microsegmentation with NSX-T
○ Eliminating “Share the same network”
Multi (Single-tenant) clusters
It is having an API for creation and management
that enables this!!!
● Every tenant gets their own cluster
Addresses limitations
● Single tenant worker VMs (depend on the
hypervisor to ensure host is properly shared)
● Every cluster has own network segment
● Every cluster has own DNS
● Every cluster has own configuration
● ...
No other “on
prem” solution
has this!!!

34. Multi-Tenancy - Single Cluster vs Multi-Cluster
Shared Compute/Storage/Network
Tenant 1
Tenant 2
Kube API
Server
Kube SchedulerDNS Cont. Mgr.
Kubelet Kubelet Kubelet
Worker Worker Worker
PKS Control Plane
Master Node(s)
Shared Compute/Storage/Network
Kube API
Server
Kube
Scheduler
DNS
Cont. Mgr.
Kubelet Kubelet Kubelet
Worker Worker Worker
PKS Control Plane
Master Node(s)
Kube API
Server
Kube
Scheduler
DNS
Cont. Mgr.
Master Node(s)
Kubelet
Worker
Tenant 1 Tenant 2

35. Flexible Multi-
Tenancy
Infrastructure
StorageCompute Networking
Container
Mgmt.
NameSpac
e
Developers
Kubernetes Cluster 1
Developers
NameSpac
e
NameSpac
e
NameSpac
e
NameSpac
e
NameSpac
e
Platform Operator
Cluster 2
BOSH - Platform Lifecycle Management
Developers
NameSpac
e
NameSpac
e
Cluster 3
Developers

36. NSX-T
Main Capability #4

37. Unified VM to Container Networking
On-demand network virtualization
Microsegmentation
Full Network Visibility
Enterprise Support
Pod-Level Container Networking
Load Balancing
Network Security policies
Tenant - level isolation
Unique logical switch per K8s namespace

38. And…

39. Keeping the pace with k8s / giving Devs the latest
** Only Generally Available versions considered
Updated June 5th, 2019
K8s 1.9 K8s 1.10 K8s 1.11 K8s 1.12
Dec 2017 Mar 2018 Jun 2018 Sep 2018
Jan 2018
Feb 2018
K8s 1.9 K8s 1.10 K8s 1.11
May 2018
Jun 2018 Sep 2018 Jan 2019
PKS 1.0 (k8s 1.9) PKS 1.1 (k8s 1.10) PKS 1.2 (k8s 1.11) PKS 1.3 (k8s 1.12)
Sep 2018
Jun 2018 Dec 2018
K8s 1.10 K8s 1.11
Jun 2018
K8s 1.10 K8s 1.11
Sep 2018 Jan 2019
K8s 1.12
Jul 2018
OCP 3.10 (k8s 1.10) OCP 3.11 (k8s 1.11)
Oct 2018Mar 2018
OCP 3.9 (k8s 1.8 + 1.9)
Nov 2017
OCP 3.7 (k8s 1.7)
K8s 1.13
Dec 2018
K8s 1.12
Feb 2019
K8s 1.14
Mar 2019
K8s 1.13
Mar 2019
Apr 2019
PKS 1.4 (k8s 1.13)
Mar 2019
K8s 1.12
May 2019
K8s 1.13
Jun 2019
OCP 4.1 (k8s 1.13)

40. Private Registry
Harbor

41. Role-Based Access Control (RBAC)
LDAP/AD Integration
Image Vulnerability Scanning (Clair)
Notary Image Signing
Policy-Based Image Replication
Graphical User Portal & RESTful API
Image Deletion & Garbage Collection
Auditing
An enterprise-class registry server for
Docker images
Build Image
Push
Image
Scan
Image
for CVEs
Sign
Image
kubectl
run
Dev Team
Image
Registry
Clair Notary
R
B
A
C
UAA
Auth
R
E
P
L

42. Physical Infrastructure
Container
Registry
vSphere vSAN, NFS, iSCSI, FC Datastores
NSX-T
BOSH
masteretcd workermasteretcd worker
PKS Control Plane
Integrations w/ VMware
vRealize
Automation
vRealize
Log Insight
vRealize
Operations
vRealize
Network
Insight
Wavefront
by VMware

43. Deep Dive

44. ● Install/Manage PKS
● Configure cluster plans
● Apply a patch / update
● Onboard Cluster Owner
via RBAC
● Operate Bosh
Platform Operator (Alana)
A Day in Life with PKS
● Create a cluster
● Scale a cluster
● Create Network Policy
● Onboard App Dev via
RBAC
● ...
Cluster Owner (Cody)
● Deploy an app
● Expose app with service
type: LoadBalancer
● Expose app with Ingress
● ...
App Dev (Naomi)
● Health Management (server & process)
● Network Automation
Automation

45. BOSH
Pivotal Container Service
Platform Ops
Pivotal
Ops Manager
PKS tile
upload
and config
Pivotal
Network
Install
Installing PKS
NSX-T

46. PKSControlPlane
Harbor
BOSH
Pivotal Container Service
Platform Ops
deploy
Install
Pivotal
Ops Manager
Installing PKS
NSX-T

47. … or
...
Pivotal
NetworkConcourse
pipelinePlatform Ops
Execute
Verify pre-reqs
Provision
infrastructure
Download
binaries
Install
Product
Config
Install PKS

48. PKSControlPlane
Harbor
BOSH
Pivotal Container Service
NSX-T

49. PKS User Interaction
● The PKS Management VM runs the PKS API
together with the Broker, UAA and a MySQL DB.
● The PKS API orchestrates the initial kubernetes
cluster deployments and scaling of those clusters.
● A single PKS VM can manage hundreds of
Kubernetes cluster.
● The PKS CLI is a single binary that can be installed
on a Mac, Windows, or Linux to drive the PKS API.
PKS CLI
PKS
Control Plane
PKS API

50. Creating a new K8s Cluster
Platform User
PKSControlPlane
CLI
API
PKS CREATE CLUSTER
BOSH
deploy
Kubernetes cluster
Create
Harbor
NSX-T
Master
Worker
WorkerWorker
etcd Worker
Master
etcd

51. Scaling a Kubernetes Cluster
Platform User
PKSControlPlane
CLI
API
PKS SCALE CLUSTER
BOSH
deploy
Kubernetes cluster
Scale
Harbor
NSX-T
Master
Worker
WorkerWorker
etcd
Worker
Master
etcd
Worker

52. Demo