SpringOne Platform 2018 PKS Networking with NSX-T: You Focus on your App, We'll Take Care of the Rest! Gaetano Borgione, VMware; Angela Chin, Pivotal

1. PKS Networking with NSX-T
“you focus on your app, we'll take care of the rest !”
Gaetano Borgione, @TanoBorgione
Angela Chin, @AngelaSChin

2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
● What is PKS
● Networking in Kubernetes
● What is NSX
● NSX + PKS
● Cluster Creation
● Cluster Upgrade
2

3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Disclaimer

4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal
offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to
change. This information is provided without warranty or any kind, express or implied, and is not a commitment to
deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding
Pivotal's offerings. These purchasing decisions should only be based on features currently available. The
development, release, and timing of any features or functionality described for Pivotal's offerings in this
presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking
information in this presentation.
4

5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
What is PKS

6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
6

7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
7

8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
8

9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
9

10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
● Aimed at Day 2 Operations
1
0

11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
● Aimed at Day 2 Operations
● Uses BOSH!
1
1

12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
2
BOSH
Infrastructure

13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
3
BOSH
PKS
Infrastructure
watches

14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
4
BOSH
PKS K8s
Infrastructure
watches
deploys

15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
5
BOSH
PKS K8s
Infrastructure
watches watches
deploys

16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
6
BOSH
PKS K8s K8s
Infrastructure
watches watches
deploys

17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
7
BOSH
PKS K8s K8s
Infrastructure
watches watches watches
deploys

18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
1
8

19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/ 19

20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
2
0
● Deploys as a tile in Ops Manager
PKS

21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
2
1
PKS

22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
2
2
PKS

23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
2
3
PKS k8s

24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
2
4
PKS k8s

25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
2
5
PKS k8s
kubeconfig

26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
kubectl get pods
2
6
PKS k8s
kubeconfig

27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking in Kubernetes

28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
2
8

29. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
2
9

30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
3
0

31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
● NetworkPolicy for enforcing network rules
3
1

32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
● NetworkPolicy for enforcing network rules
○ ...but not provided with Flannel alone
3
2

33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
What is NSX-T ?

34. The NSX-T Platform
Single platform for Networking, Security & Monitoring for Containers & VMs
PKS PAS/PCFOpenShift VM
Networking, Security, Monitoring
Kubernetes
34

35. Address all K8s Networking Functions
Load Balancing, IPAM, Routing, Firewalling
Complete automation and dynamic provisioning of
Network Objects required for K8s and Workloads
Support for different topology choices for
Pod and Node Networks (NAT/No-NAT)
Network Security Policies for Kubernetes Clusters,
Namespaces and Individual Services
Full Network traceability/visibility using NSX-T in-built
operational tools for Kubernetes
How does PKS with NSX-T add value over opensource solutions
35

36. NSX-T: operational tools and automation
36
Traceflow, Port-Connection,
Topology View
Back-Up & Restore
Selective Tech Support Logs
Monitoring & Stats
REST API support
Dynamic libraries available in
multiple languages (Java, Python,
Terraform)
Upstream OpenStack Support and
Partner Ecosystem
Getting started wizards
Dashboards
IPFIX, Port Mirroring, Port
Connection
Granular RBAC
Upgrade Coordinator
vRealize Log Insight / Splunk
plugins
Automation Operations Troubleshooting
36

37. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
NSX-T + PKS

38. 38
Pivotal
Operations Manager
BOSH
Master Worker Worker Worker
Master Worker Worker Worker
Master Worker Worker Worker
PKS-API
UAA
ODB
kubo
service
adapter
MySQL
NSX-T
Proxy
Broker
Telemetry
Aggregator
PKS
Control Plane
adminserver
jobservice
ui registry
notary
clair
NSX-T
vSphere
vRLI
Wavefront
VAC
PKS: High Level Architecture
PKS CLI
38
38

39. PRINCIPAL
IDENTITY
39
K8s Cluster Management Nodes
T1
T1 for LB NSX-T LB
K8s cluster 1
K8s cluster 2
K8s cluster n
Ops-Manager BOSH PKS
Control Plane
Harbor
T1
PKS
mgmt
plane
NSX-T
Edge Cluster
NSX-T
Controllers
vCenter
NSX-T
Manager
vCenter
Physical
L2/L3
switches
Internet
K8s Cluster and NSX-T Provisioning
VIP
API Server https://api.cluster.pks.customer.com
Master VM#1, VM#2, VM#3
39
pks create-cluster my-cluster --plan small --num-nodes 3
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
Master ‘VM’
Master ‘VM’
Master ‘VM’
etcd
API
srv
sched
ctrlr
mgr
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
NSX
NCP
dash
board
NA NANA
Kube
DNS

40. Cluster Provisioning Workflow
41
PKS API
1
NSX-T
Proxy Broker
named_cloud_config
2
• pre-deployment provisioning
o allocate cluster subnet
o create named_cloud_config
o create Node Network
o create SNAT rule, if required, from k8s nodes to external
• post-destroy decommissioning
o delete SNAT rule, if required
o release Virtual IP for API Server
o delete Node Network
o delete named_cloud_config
o remove Principal Identity
o run clean-up script for NSX-T cluster resources
3
manifest
Service
Adapter
5
CFCR
release
pks-nsx-t
release
7
Ops Manager OD-Broker
cloud_config
4
vSphere CPI
6
BOSH
pks create-cluster my-cluster --plan small --num-nodes 3
41
41

41. Cluster Provisioning Workflow (cont.)
42
o NCP
o NSX-CNI
o OpenvSwitch
o NSX scripts
BOSH
CFCR
release
vSphere CPI
pks-nsx-t
release
7
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
Master ‘VM’
Master ‘VM’
Master ‘VM’
8
VMs tagged with BOSH ID
9
BOSH Job on Master Node to perform cluster-level provisioning
o create Principal Identity
o create Load Balancer for the new Cluster
o reserve Virtual IP for API Server
10
42
42

42. PKS and NSX-T integration: Key Components
NSX Container Plugin (NCP)
43
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
NS: foo NS: bar
NSX Example K8s topology
K8s master
etcd
API-Server
Scheduler
• NCP is a software component
provided by Vmware, running on a
K8s Node..
• NCP is built in a modular way, so
that individual adapters can be
added for different CaaS and
PaaS systems
• NCP ‘listens’ to Kubernetes/CF to
create the required NSX-T
constructs.
• CNI Integration is used.

43. PKS + NSX-T: built-in Load Balancing
Kubernetes Ingress and Svc Type LB support
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server
10.114.209.209HTTP and/or
HTTPS traffic
Server Pool 1
Server Pool 2Rule 2
/bar/
Rule 1
/foo/
LB Service
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server
10.114.209.212TCP and/or
UDP traffic
Server Pool
LB Service
Built-in support for Ingress (L7) and Svc Type LB (L4) w/t NSX-T integration. Most other K8s
networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX
from Ingress (L7).
44

44. admin@k8s-master:~$ kubectl create namespace foo
namespace ”foo" created
admin@k8s-master:~$ kubectl create namespace bar
namespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo
deployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar
deployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT
boundary
NAT
boundary
K8s nodesK8s Masters
45
PKS + NSX-T: Namespaces
45
45

45. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Cluster Upgrades

46. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● Install the new tile
● Update any new fields
● Deploy!
4
7

47. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
4
8
BOSH
PKS K8s K8s
Infrastructure
watches watches watches
deploys

48. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
4
9
BOSH
PKS K8s K8s
Infrastructure

49. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
0
BOSH
PKS K8s K8s
Infrastructure
upgrade

50. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
1
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade

51. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
52
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade upgrade

52. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● NCP components upgrade within PKS
5
3

53. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
4
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade upgrade

54. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
5
Master
Worker
MasterMaster
WorkerWorker

55. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
6
Master
Worker
MasterMaster
WorkerWorker

56. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
7
Master
Worker
MasterMaster
WorkerWorker

57. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
8
Master
Worker
MasterMaster
WorkerWorker
NCP

58. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
9
Master
Worker
MasterMaster
WorkerWorker
NCP

59. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● NCP components upgrade within PKS
● NSX-T can be upgraded independent of PKS
6
0