Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Kubernetes to make cellular data plans cheaper for 50M users

126 views

Published on

Use case of Kubernetes based NFV infrastructure used in production to run an open source evolved packet core. Presented by Facebook Connectivity and Mirantis at KubeCon + CloudNativeCon Europe 2020.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Using Kubernetes to make cellular data plans cheaper for 50M users

  1. 1. Using Kubernetes to make cellular data plans cheaper for 50M users July 28th, 2020
  2. 2. 2 Introductions Amar Padmanabhan Lead Developer, Magma Project Facebook Connectivity Chandra Dodda Sr. DevOps Engineer Mirantis
  3. 3. The problem of bringing the next 50 million onto a faster internet is a problem of heterogeneity Heterogeneity in access, backhaul, scale and business models
  4. 4. 1. Edge vs Fabric decomposition of the network 2. State based orchestration of edges 4 Design Principles
  5. 5. Edge vs Fabric Modularization of the network
  6. 6. ● Lots of in-network processing appliances: IDS, Firewall, Proxies, Load balancers ● Chokepoint devices ○ Force a topology ○ Expensive: Fast pipes + rich policies 6 Traditional Datacenter: Hierarchical Networks
  7. 7. Modularize the network: Fabric responsible for moving packets faster. Distributed edge responsible for rich policy enforcement Modern datacenters: Fabric and policy rich edge
  8. 8. 8 Notes on edge services ● Fundamentally distributed ○ Good candidate for x86: Does complex things well but slowly. ● Software only policy enforcement allows for rapid iteration. ● Leverage programmable interfaces like openflow, e-BPF etc. ● Distributed services are harder to manage ○ Operationalizing the solution is key
  9. 9. X X XX SGW/PGW are chokepoint devices Today’s GSM/LTE architecture
  10. 10. ● Distribute policy enforcement point ○ Let the ideal topology decide the policy enforcement point ● Move policy enforcement to software ○ Leverage rapid iteration and programmability of software ● Keep core network simple ○ Allows for easy scale up/down ○ Cheap: Core network only needs to move packets fast ● Focus on operationalizing the distributed network ○ What used to be a single central node is now distributed services across multiple edge locations 10 Why Magma? Summary: Modularize the cellular network
  11. 11. 4G LTE / 5G / Wi-Fi INTERNETACCESS NETWORK MAGMA CONVERGED CORE REST APIs S1 Cell Site Private Cloud *Core Deployment Options Converged Core* Radio Backhaul SGi (User IP traffic) Public Cloud Orchestrator & NMS Magma Converged Core (OTS H/W) Federation Gateway CPE HTTP2 HTTP2
  12. 12. The core problem we built Orchestrator to tackle: Managing a fleet of heterogeneous, edge-deployed service meshes for core convergence.
  13. 13. ● Declaratively configure the edge fleet from centralized management ‒ Push state, not procedures ‒ Declare the desired topology for your fleet and the service meshes running on each device ● Delegate procedural control to the edge device ‒ The device always has the most up-to-date view of its operational state, so it’s the best place to perform procedural reconciliation of declared configuration and actual state Anchoring Principles 13 Declare new configuration Synchronize updates Monitor aggregated edge state Send new operating state Reconcile local and desired states Edge Cloud
  14. 14. ● 1 instance per managed device ● Applies configuration updates to the local service mesh based on updates from management ● Ships local state and operational history to management ‒ Service states, performance metrics, application timeseries, events, logs, etc. ● In Magma LTE, this is a custom solution built on top of systemd, apt, and custom Python code ‒ Plans are in place to migrate to a k8s-based control plane at the edge Edge Control Plane Orchestrator Components 14 magma cl updates state, timeseries fluent logs, events control state update local config store local state store Control: systemd / logs Config store: disk / Config Map local service mesh Redis 1 3 2
  15. 15. ● Singleton, multi-tenant cloud application built on k8s ● Aggregates state and history from edge fleet and sends configuration updates to fleet (GRPC) ● Exposes endpoints to query and set edge fleet configuration and query edge fleet state (REST) ● Batteries included: ‒ Multi-tenant wrappers around Grafana and Elasticsearch to expose timeseries dashboards, events, and logs ‒ ECDSA-based trust negotiation for authenticating edge devices ‒ REST authorization framework ‒ Integration points for external timeseries and event datasinks Central Management Plane Orchestrator Components 15 REST Core Orchestrator SQL Prom ES GRPC Domain Plugins Edge Fleet: ● OCN 5G ● Magma LTE ● Etc. stateconfig Field Cloud
  16. 16. 16 Edge Infrastructure Stack Mirantis Cloud Platform
  17. 17. Body Level One ● Body Level Two ‒ Body Level Three ‒ Body Level Four BODY LEVEL FIVE Optional subtitle (delete if not used) Title Text 17 MCP Overview Lifecycle Management (LCM) DriveTrain Version Control Artifact Store Code Review Node Classification CI/CD Orchestration Compute Storage Network Operations Support System (OSS) StackLight Web NFV AnalyticsMediaIoT Deployment Kubernetes Calico SDN OpenStack Tungsten Fabric Ceph Flexible Infrastructure containers Bare Metal VMs Notifications Logging Tungsten Fabric Neutron + OvS Monitoring Alerting Analytics Trending/Capacity Enterprise Integrations (e.g. Security & Single Sign-on)
  18. 18. 18 MCP Edge Architecture ● Minimum footprint: 4 nodes ○ 3 control plane nodes that run DriveTrain, StackLight, and K8s Masters in HA ○ 1 or more worker nodes that run workload ● Coexistence and seamless networking with container pods and VM pods ● Virtlet: use to implement VM pods in k8s environment ○ Open source project
  19. 19. 19 Virtlet
  20. 20. 20 Kubernetes CRI implementation for running VM workloads ● Targeted at VM workloads that need to behave as containers on the outside ● Run unmodified VM images using qcow2 format ● Build higher-level Kubernetes objects using VM pods ● Use familiar kubectl pod commands to work with your VMs ● Integrate with cluster networking using normal CNI plugins ● Easy to deploy - only need to install simple CRI Proxy package on the nodes What is Virtlet?
  21. 21. 21 ● Virtlet enables you to run unmodified QEMU/KVM virtual machines that do not include an additional Docker layer as in similar solutions in Kubernetes. ● Virtlet supports all standard Kubernetes objects, such as ReplicaSets, Deployments, DaemonSets, and so on, as well as their operations. ● This diagram describes the Virtlet components and interactions between them. What Does Virtlet Enable?
  22. 22. 22 Magma Integration for MCP Edge
  23. 23. 23 Optimizing Carrier-Grade Wi-Fi Offload with Magma Provide operators with a sustainable and efficient way to address consumer data demands through Wi-Fi offload Seamless User Experience ● Flexible distributed Core integration ● Extend user reach through FB app footprint ● Steer users to the best connection Optimized Business Platform ● Identify areas best suited for mobile data offload ● Analytics and management capabilities for large-scale Wi-Fi networks and hotspots Wi-Fi Ecosystem ● Vendor Integrations ● Offload ecosystem leveraging additional bandwidth and capacity for hungry applications
  24. 24. 24 Elevate subscriber’s quality of experience on Wi-Fi to match that on mobile data Mobile Core Integration
  25. 25. 25 MCP Edge Architecture Magma Worker 1 Control Plane Node 1 Control Plane Node 3Control Plane Node 2 DriveTrain StackLight K8s Master DriveTrain StackLight K8s Master DriveTrain StackLight K8s Master Docker Pod Orchestrator Virtlet Pod containerd } MCP Edge Access Gateway VM Pod VM Pod Fed. Gateway
  26. 26. 26 Covered in this demo : 1. Deploy MySQL pods in HA 2. Deploy Orchestrator in HA 3. Deploy Federation Gateway ( As VM using Virtlet ) 4. Deploy Wi-Fi Access Gateway ( As VM using Virtlet ) 5. Connect to hotspot and access Internet from mobile phone Carrier Wi-Fi Demo Not covered and out of scope: 1. Deploy and configure Wifi AP 2. Configure user in HSS/PCR and OCS
  27. 27. 27 MCP Edge Private Cloud Magma Worker Node User Gateway Wi-Fi Node 3 mysql03 containerd Carrier Wi-Fi Access Gateway Virtlet CNICNI CNI Orchestrator Service Wireless Controller Policy Control and Charging Rules Function Online Charging System Home Subscriber Server Node 2 mysql02 containerd Federation Gateway Virtlet CNI CNI SCTP EoGRE DHCP, DNS, NAT User MySQL Service Container VM Pod Node 1 mysql01 containerd Orchestrator Ctl + Proxy Network Monitoring
  28. 28. ● Flexibility: Modularize the network into a fast fabric and a policy rich edge to maximize deployment and business models ● Hierarchical orchestration: Distributed edge services require hierarchical orchestration ● Consistent runtime: k8s based edge clusters offer a consistent runtime for edge services ● Get involved! https://github.com/magma 28 Summarizing: Solving for heterogeneity
  29. 29. ● magma-dev@googlegroups.com : For more information on Magma ● magma-announce@googlegroups.com : Join our mailing list to receive updates and announcements ● https://connectivity.fb.com/magma : For a broader perspective on how Magma fits into Facebook’s other connectivity programs For more information
  30. 30. 30 Kontena Lens: Kubernetes IDE simplifies Kubernetes for application developers ●View demo at Mirantis booth ●Download from: k8slens.dev Kontena Lens
  31. 31. Thank You

×