![Enabling DPDK/SR-IOV for
containerized Virtual Network
Functions with Zun
Bin Zhou [NFV Researcher, Lenovo]
Hongbin Lu [Zun PTL,Huawei]
Yaguang Tang [NFV Researcher, Lenovo]
Shunli Zhou [Zun Core, Fiberhome]
November 2017](https://image.slidesharecdn.com/enabledpdk2fsr-iovforcontainerizedvirtualnetworkfunctionswithzun1-180108150314/85/enable-dpdk-and-sriov-for-containerized-virtual-network-functions-with-zun-1-320.jpg)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Enable DPDK and SR-IOV for containerized virtual network functions with zun
Similar to Enable DPDK and SR-IOV for containerized virtual network functions with zun (20)
Recently uploaded
Recently uploaded (20)
Enable DPDK and SR-IOV for containerized virtual network functions with zun
- 1. Enabling DPDK/SR-IOV for containerized Virtual Network Functions with Zun Bin Zhou [NFV Researcher, Lenovo] Hongbin Lu [Zun PTL,Huawei] Yaguang Tang [NFV Researcher, Lenovo] Shunli Zhou [Zun Core, Fiberhome] November 2017
- 2. ➡Introduction to Zun ➡Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡Performance Benchmark Testing • Setup • Results ➡Demo ➡Conclusion Agenda
- 3. Which Emerging Technologies Interest OpenStack Users? ● Containers are the most interesting emerging technologies. ● 75% of OpenStack users interests in containers.
- 4. ➡How to use containers on OpenStack? ➡Existing solutions • Integrate containers into Nova • Example: Nova-docker, Nova-lxd • Install Container Orchestration Engine (COEs) on VMs. • Example: Magnum, Kubespray • OpenStack Container service: Zun Introduce Zun
- 5. ● OpenStack Container service ● Provide API for provisioning and managing containers without VMs ○ Speed ○ Simplicity ● Arbitrary memory and vCPUs ● Containers as first class resource ○ Keystone RBAC for individual container ○ Neutron port(s) for each container ○ Cinder volume(s) bind-mount Introduce Zun
- 6. VMs Containers Create List Delete Run Exec ... SSH Migrate ... Nova Zun ➡Nova-docker • Use Nova to manage containers • Suitable if VMs and containers are the same ➡Obstacles • VMs and containers are different • Container specified features are not exposed Introduce Zun
- 7. Baremetal Tenant 1 Virtualization Tenant 2 Tenant 3 COE Baremetal Tenant 1 Virtualization (optional) Tenant 2 Tenant 3 Contain ers ZunCOE COE Contain ers Contain ers Contain ers Contain ers Contain ers Magnum Zun ➡Magnum • Provision Nova instances • Install a COE • Run containers on the COE ➡Pros: • Strong Isolation ➡Cons: • Low resource utilization • Virtualization penalty Introduce Zun
- 8. ➡Concepts: • Container: A single container • create, update, delete, start, stop, kill, … • network-attach, add-security-group, … • attach, exec, commit, log, ... • Capsule (Experimental): A group of containers that are co- located, have shared network and volumes. • create, list, delete, … Introduce Zun
- 9. Introduce Zun ➡Zun API • Provide REST APIs • Manage all compute nodes • Scheduling containers ➡Zun Compute • Compute node agent • Manage local containers • Track compute resources ➡Kuryr • Bind neutron ports to containers Zun API Zun Compute Docker Keystone KuryrNeutron Cinder
- 10. ➡Introduction to Zun ➡Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡Performance Benchmark Testing • Setup • Results ➡Demo ➡Conclusion Agenda
- 11. ➡What is NFV • A new way to design, deploy and manage network services • Replace hardware with software • Move network functions to commodity hardware ➡Benefits of NFV • Fast provisioning • Quick scale up and down • Easy upgrade and relocate • Reduce cost • No vendor hardware locked-in Container for NFV
- 12. ➡VM or Containers? • Time to provision: container boots faster • Resource consumption: container has less memory footprint • Package management: Docker makes it easy • Configurability: container is better • Portability: container image is smaller • Security: VM provides better isolation • Use Clear Container to improve security Container for NFV
- 13. Challenges & Gaps of using containers NFV Req features VM Container SR-IOV Yes Weak DPDK Yes Weak CPU pinning Yes Weak NUMA Yes Weak Hugepage Yes Weak ➡Lack of supports of NFV required features in container ecosystem • Container runtime • Container orchestration • OpenStack integration ➡Use Zun to reduce the gaps
- 14. Enable SR-IOV in Zun ➡What is SR-IOV? • A standardized mechanism to virtualize PCIe devices • Make a single PCIe Ethernet controller (PF) to appear as multiple PCIe devices (VF) • PF: Physical Function • VF: Virtual Function • Passthrough VF to container • Bypass virtual switch layer
- 15. Enable SR-IOV in Zun ➡Enable SR-IOV in Zun • Create VFs in compute nodes • Configure Neutron • Configure Zun • Whitelist PCI devices (e.g. pci_passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2"}) • Enable PCI filters (e.g. enabled_filters = ...,PciPassthroughFilter) • Configure Kuryr • Enable SR-IOV driver
- 16. Enable SR-IOV in Zun 1.Create a SR-IOV port 2.Create a container 3.Pick a host that has available VFs 4.Assign a VF to the port 5.Create a container 6.Docker calls its network plugin (Kuryr) to setup the network 7.Kuryr retrieve VF’s information from the neutron port and perform port binding Zun API Zun Compute Kuryr Neutron Docker User 1 2 3 5 6 7 4
- 17. Container with DPDK DPDK PMD ● physical nic ○ igb_uio ○ vfio-pci ● virtual hardware ○ virtio_user vhost software ● net_pcap (kernel stack)
- 18. Host kernel Container Container VF VFPF PF driver Host kernel Container DPDK DPDK DPDK DPDK & SR-IOV for container SR-IOV in userland SR-IOV in kernel VFVF VF driver VF driver Container netns ETHx netns ETHx Passthrough
- 19. ➡Introduction to Zun ➡Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡Performance Benchmark Testing • Setup • Results ➡Demo ➡Conclusion Agenda
- 20. Case 1 (non DPDK) ● Zun Container with SR-IOV ● Zun Container with OVS networking Performance Benchmark Testing Case 2 （SR-IOV & DPDK） ● Container with SR-IOV & DPDK (kernel land) ● Container with SR-IOV & DPDK (user land)
- 21. Role Hardware OS network CPU Controller Think system x3650 M5 Ubuntu 16.04.3 82599ES 10Gb Intel(R) E5- 2680 v3 @ 2.50GHz compute Think system x3650 M5 Ubuntu 16.04.3 82599ES 10Gb Intel(R) E5- 2680 v3 @ 2.50GHz Software version other DPDK 17.05 Openvswitch 2.8.1 Testing setup ● L2FWD as containerized VNF ● RFC 2544 standard throughput testing ● DPDK-pktgen as packet generator DPDK Testing non DPDK Testing ● iperf3 with udp
- 22. zun-compute Server1 zun-compute Server2 O V S O V S container container container container Linux bridge Linux bridge PF PF Zun networking without SR-IOV
- 25. ● Hugepage size ● PCIe NUMA ● Isolate CPU cores for tx/rx pktgen ● Disable isolated cpu core interrupts BOOT_IMAGE=/vmlinuz-4.4.0-87-generic root=/dev/mapper/docker2--vg-root ro default_hugepagesz=1G hugepagesz=2M hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on isolcpus=5,6,7,8,9,10 nohz=on nohz_full=5,6,7,8,9,10 rcu_nocbs=5,6,7,8,9,10 DPDK testing tuning
- 26. Server1 Server2 VF1 VF2 pktgen VNF l2fwd VF1 VF2 VF1 Testing scenario 1 ● Userland SR-IOV used by container ● DPDK application l2fwd inside container Container dpdk-devbind --bind=igb_uio 0000:06:10.2 docker run -v /dev/hugepages/:/dev/hug epages --net=none -- privileged --name test2 -dit 14ce48b74dd9 l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket- mem 1024,1024 -- -q 8 -p 1
- 27. Server1 Server2 VF1 VF2 pktgen VNF l2fwd VF1 VF2 VF1 Testing scenario 2 ● containers using SR-IOV by kernel netns ● DPDK application l2fwd inside container NETNS Container $ neutron port-create sriov -- name sriov_port -- binding:vnic_type direct $ zun run --net port=sriov_port dpdk-test l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket-mem 1024,1024 -- vdev=’eth_pcap0,iface=eth0’ -- -q 8 -p 1
- 30. ➡Introduction to Zun ➡Zun Container for NFV • Challenges & Gaps • SR-IOV support in Zun • Container with DPDK ➡Performance Benchmark Testing • Setup • Results ➡Demo ➡Conclusion Agenda
- 31. SR-IOV & DPDK can accelerate container networking performance Benefits High throughput Low latency Deterministic networking Conclusion ● DPDK & SR-IOV for container user land approaching physical server performance ● multi-tenancy issue ● security issue ● Container with SR-IOV for high throughput non DPDK application ● unified management of VF
- 32. @OpenStack Q&A Thank you! openstack openstack OpenStackFoundation