Tectonic Summit 2016: Networking for Kubernetes
Sreekanth Pothanis, Cloud Engineering, eBay shares a networking Kubernetes tale from the trenches. Networking is the hardest component in any ones infrastructure, everything depends on it. Specifically when we have web scale infrastructure with tens of thousands of servers. eBay is investing heavily in Kubernetes and networking again is one of the areas we have the most difficulty with. During the course of this talk we will go through various approaches we tried to make container networking conform to Kubernetes networking principles, while ensuring that it adapts to the existing networking models our infrastructure supports. We would also cover how we have automated the process of setting up networking for Kubernetes clusters and how it offers seamless integration with non-Kubernetes workloads. 12/12/16
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tectonic Summit 2016: Networking for Kubernetes
Similar to Tectonic Summit 2016: Networking for Kubernetes (20)
More from CoreOS
More from CoreOS (17)
Recently uploaded
Recently uploaded (20)
Tectonic Summit 2016: Networking for Kubernetes
- 1. Networking for Kubernetes A Tale from the Trenches Cloud Engineering, eBay Sreekanth Pothanis
- 2. Networking is inherently hard! Complexities of running on openstack Scale Multitenancy Interoperability with Legacy
- 3. Private Network model with Openstack SDN Dedicated kube router provisioned in neutron Private Networks Subnet per node
- 4. L3 Routed Model NIPAP as IPAM Subnet per node Fully routable pods
- 5. Network 2.0 Abstract out network boundaries from nodes to arbitrary network scopes IP blocks are allocated to these network scopes Scopes can represent a host or a higher level aggregation Supports legacy and other complex network zoning
- 6. Network 2.0 node pod pod Network Scope IPAM node pod pod Allocation Pools Network Scope Allocation Pool 1 uuid1 2 uuid2
- 7. IPAM controller Cluster admin creates network scopes + allocation pools Kubernetes Nodes are associated with Scopes IPAM Controller assigns IP based on scope of the node selected by Kube scheduler Pods are annotated with IPs Tessnet plugin configures the pods with annotated IP Kube Scheduler IPAM controller Tess NetPlugin Pod: myPod Host: A Pod:myPod Node A notMyPod myPod 10.10.11.4 Tessnet Pluginkubelet Network Scope1 Allocation Pools 10.10.12.0/22 10.11.1.0/24 Node: A Node: B Node: C Network Scope2 API Server Host: A IP: 10.10.1.4 Pod: myPod Host: A IP: 10.10.1.4 Pod: myPod “network_scope”: “netscope1”
- 8. Networking 2.0 -- host OVS ARP Proxy
- 9. Service to POD Kube’s default implementation creates LBs on Nodes Load balance on pods directly Neutron LBaaS Pool Neutron LBaaS VIP POD POD POD POD
- 10. eBay Ingress Application Topology POOL Application VIP VIP GTM Load Balanced Pool POOL VIP POOL VIP Region 1 Region 2 Region 3 Global Name (omg.g.ebay.com) MONITOR MONITOR MONITOR Application VIPApplication VIP
- 11. Ingress controller Ingress: myIngress Status: VIP-1 IP GTM name Ingress controller API Server LBMS DNS GTM Ingress: myIngress Ingress: myIngress Status: VIP-1 IP Ingress: myIngress Status: VIP-1 IP GTM name
- 13. Future work Network Policy Enforcement Globally federated Ingress -- SLB based
Editor's Notes
- ebays architecture, multiple generations of cloud, scale, use of lot of physical hardware and LBs have sdn and routed l3 netowrks Dont want to make it more harder why not private ips
- IPTables work because one end of veth was still in host We had a few options, we went with what we knew best that worked on an overlay and routed L3 network Issues: Floating IPs need to created for every pod that needs to be routable Complex router/network setup. Works only overlay
- Static allocation of per minion subnet Bootstrapping issues
- Why central IPAM
- Statically route to tor
- Finalize with IPAM and Network plugin
- Bonus: Hairpin ovs flow routes
- Fully software based implementation of ingress collab with community