Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scalable networking in Apache CloudStack

Presentation I made at the Carrier Cloud Summit on June 20, 2012 in Paris

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Scalable networking in Apache CloudStack

  1. 1. Making a million firewalls sing Scalable Networking in Apache CloudStack June 19 2012 Chiradeep Vittal
  2. 2. Agenda• Who am I & what am I doing here?• Apache CloudStack• Networking modes in Apache CloudStack• Scaling challenges in Cloud Networking• Scale Up• Scale Out
  3. 3. WhoChiradeep Vittal (@chiradeep) – Founding engineer @ Cloud.com (2008) – Architect @ Citrix Systems (2011-) – Maintainer @ Apache CloudStack (2012-) – Not a Network NinjaWhy – Challenges of Cloud Networking – Apache CloudStack – Real World Cloud Networking
  4. 4. Apache CloudStack • Secure, multi-tenant cloud orchestration platform – Turnkey platform for delivering IaaS clouds – Over 100 commercial deployments: Build your cloud the way the private and publicworld’s most successful clouds – Full featured GUI, end-user API and are built admin API
  5. 5. Apache CloudStack • Open Source • Apache License • Incubating in the Apache Software Foundation since April 2012 Build your cloud the way theworld’s most successful clouds • Open Source since May 2010 are built • In production since 2009
  6. 6. Apache CloudStack • Flexibility and scale • Hypervisor agnostic • Flexible network topologies • Multiple storage options Build your cloud the way the • Proven to scale to tens ofworld’s most successful clouds thousands of hypervisors are built
  7. 7. Server Virtualization++ Cloud • 10x more scaleable • 2-5x lower cost • 100% more open Built for traditional Designed around big data, enterprise apps & client- massive scale & next-gen apps server compute •Cloud architecture for 1000s of • Enterprise arch for 100s of hosts hosts •Scale-out (multi-site server farms) • Scale-up (server clusters) •Apps assume failure • Apps assume reliability •Autonomic [1:1,000’s] • IT Mgmt-centric [1:Dozens] •Open, value-added stack • Proprietary vendor stack
  8. 8. Network Flexibility Network Services• L2 connectivity• IPAM• DNS• Routing• ACL• Firewall• NAT• VPN• LB• IDS• IPS
  9. 9. Network Flexibility Network Service Services Providers• L2  Virtual connectivity appliances• IPAM  Hardware• DNS firewalls• Routing  LB• ACL appliances• Firewall  SDN• NAT controllers• VPN  IDS /IPS• LB appliances• IDS  VRF• IPS  Hypervisor
  10. 10. Network Flexibility Network Service Network Services Providers Isolation• L2  Virtual • No isolation connectivity appliances• IPAM  Hardware • VLAN• DNS firewalls isolation• Routing  LB• ACL appliances • Overlays• Firewall  SDN • L3 isolation• NAT controllers• VPN  IDS /IPS• LB appliances• IDS  VRF• IPS  Hypervisor
  11. 11. End-user experience• Deploy a VM in a network – VM Template = Windows 2008 with Joomla on VMWare – Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB RAM – Disk Offering {Super fast} – Network Offering {Gold} = Source NAT + LB+ FW + 20 Mbps Internet access
  12. 12. End-user experience• Deploy a VM in a network – VM Template = Windows 2008 with Joomla on VMWare – Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB RAM – Disk Offering {Super fast} – Network Offering {Gold} = Source NAT + LB+ FW + 20 Mbps Internet access• Network Offering Gold is realized by – VLAN isolation – Source NAT & FW on Juniper SRX – LB on F5 BigIp – DHCP, DNS on virtual appliance
  13. 13. End-user experience• Deploy a VM in a network – VM Template = Windows 2008 with Joomla on VMWare – Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB RAM – Disk Offering {Super fast} – Network Offering {Gold} = Source NAT + LB+ FW + 20 Mbps Internet access• Network Offering Gold is realized by – VLAN isolation – Source NAT & FW on Juniper SRX – LB on F5 BigIp – DHCP, DNS on virtual appliance• CloudStack orchestration: – Pick a free VLAN, pick a free public IP, free private IP – Pick hypervisor with spare capacity – Pick primary storage of SSD type accessible in hypervisor cluster – Pick a Juniper SRX and F5 with spare capacity – Spin up a new virtual appliance if necessary that runs DHCP and DNS service • Pick hypervisor, call hypervisor APIs to provision virtual appliance on selected VLAN – Call hypervisor APIs to provision VM on selected VLAN – Call SRX and F5 APIs to place their internal interfaces on the VLAN, public interfaces on public VLAN – Call SRX API to provision source NAT, default FW rules
  14. 14. Networking StylesServer Virt ++• VLAN (or no) isolation• Multiple service levels• Interoperate with legacy networks at L2 or L3• Legacy workloads requiring multicast and broadcast• Assumes reliable infrastructure• Difficult / expensive to scale out• Bonding, multi-link, multi- path, redundant networks, STP
  15. 15. Networking StylesServer Virt ++ Cloud Style• VLAN (or no) isolation • L3 isolation or overlays• Multiple service levels • Single or few service levels• Interoperate with legacy networks at L2 or L3 • Interoperate with legacy• Legacy workloads requiring networks using gateways or multicast and broadcast at L3• Assumes reliable • Workloads assume infrastructure unreliable infrastructure• Difficult / expensive to scale out • Generally do not support• Bonding, multi-link, multi- multicast or broadcast path, redundant networks, STP • Scales out massively
  16. 16. Software Defined Networking• Built-in overlay controller (using vanilla GRE between Open vSwitch on hypervisor) Or• Integration hooks available – E.g., call SDN controller API to create logical switch when a network is created – Call SDN API when VM nic is added to a network – Nicira NVP, Midonet (more coming)
  17. 17. Physical Network Operations End Admin and Users Cloud API CloudStack Mgmt Server Cluster Router MySQL Edge Services Availability Zone L3/L2 Core Access LayerSwitches Secondary Servers … … … … … Storage Pod 1 Pod 2 Pod 3 Pod N
  18. 18. Network virtualization with VLANs Tenant 1 Virtual Network 10.1.1.0/24 Tenant 10.1.1.2 Gateway address 1 VM 1 10.1.1.1 Tenant 10.1.1.3 1 VM 2Internet Tenant 10.1.1.4 1 VM 3 Tenant 10.1.1.5 1 VM 4
  19. 19. Network virtualization with VLANs Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP address Tenant Network 65.37.141.11 10.1.1.2 Gateway address 1 VM 1 65.37.141.36 10.1.1.1 Tenant 1 Tenant 10.1.1.3 Edge Services 1 VM 2 Appliance(s) NATInternet DHCP Tenant 10.1.1.4 1 VM 3 FW Tenant 10.1.1.5 1 VM 4
  20. 20. Network virtualization with VLANs Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP address Tenant Network 65.37.141.11 10.1.1.2 Gateway address 1 VM 1 65.37.141.36 10.1.1.1 Tenant 1 Tenant 10.1.1.3 Edge Tenant 1 Services 1 VM 2 Edge Services Appliance(s) Appliance(s) NATInternet DHCP Tenant 10.1.1.4 1 VM 3 FW Load Balancing VPN Tenant 10.1.1.5 1 VM 4
  21. 21. Network virtualization with VLANs Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP address Tenant Network 65.37.141.11 10.1.1.2 Gateway address 1 VM 1 65.37.141.36 10.1.1.1 Tenant 1 Tenant 10.1.1.3 Edge Tenant 1 Services 1 VM 2 Edge Services Appliance(s) Appliance(s) NATInternet DHCP Tenant 10.1.1.4 1 VM 3 FW Load Balancing VPN Tenant 10.1.1.5 1 VM 4 Tenant 2 Virtual Network 10.1.1.0/24 Public IP address 65.37.141.24 Gateway address Tenant 10.1.1.2 65.37.141.80 10.1.1.1 2 VM 1 Tenant 2 Tenant 10.1.1.3 Edge Services 2 VM 2 Appliance VPN Tenant 10.1.1.4 NAT 2 VM 3 DHCP
  22. 22. Scaling with VLANs Scale out edge services using virtual appliances 10.1.1.0/24 VLAN 100 VM 1 10.1.1.265.37.141.11 10.1.1.11 CS65.37.141.11 Virtual VM 22 Router 10.1.1.3 DHCP, DNS NAT Load Balancing 10.1.1.4 VM 3 VPN VM 4 10.1.1.5
  23. 23. Scaling with VLANs Scale out edge services using virtual appliances Scale up using hardware devices 10.1.1.0/24 10.1.1.0/24 VLAN 100 VLAN 100 VM 1 65.37.141.11 10.1.1.1 10.1.1.2 VM 1 10.1.1.2 1 Juniper65.37.141.11 SRX 10.1.1.1 NAT,1 CS Firewall65.37.141.11 Virtual VM 2 VPN VM 2 10.1.1.3 10.1.1.32 Router 65.37.141.112 10.1.1.112 DHCP, DNS Netscaler NAT Load Load Balancing 10.1.1.4 VM 3 VM 3 Balancer 10.1.1.4 VPN VM 4 VM 4 10.1.1.5 10.1.1.5 CS DHCP, Virtual Router DNS
  24. 24. Multi-tier virtual networking Internet Loadbalancer Virtual appliance/ Hardware Devices (virtual or HW)Network Services• IPAM• DNS Web VM 1• LB [intra]• S-2-S VPN• Static Routes Web VM• ACLs 2• NAT, PF• FW [ingress & egress] VLAN 353 Web VM• BGP 3 Web VM 4 Web subnet 10.1.1.0/24 VLAN 101
  25. 25. Multi-tier virtual networking Internet Loadbalancer Virtual appliance/ Hardware Devices (virtual or HW) MPLS VLANNetwork Services App VM• IPAM 1• DNS Web VM 1• LB [intra]• S-2-S VPN App VM• Static Routes Web VM 2 VLAN 2724• ACLs 2• NAT, PF• FW [ingress & egress] VLAN 353 Web VM• BGP 3 DB VM 1 Web VM 4 Web subnet App subnet DB Subnet 10.1.1.0/24 VLAN 101 10.1.2.0/24 10.1.3.0/24
  26. 26. Multi-tier virtual networking Internet IPSec or SSL site-to-site VPN Loadbalancer Virtual appliance/ Customer Hardware Devices Premises (virtual or HW) MPLS VLANNetwork Services App VM• IPAM 1• DNS Web VM 1• LB [intra]• S-2-S VPN App VM• Static Routes Web VM 2 VLAN 2724• ACLs 2• NAT, PF• FW [ingress & egress] VLAN 353 Web VM• BGP 3 DB VM 1 Web VM 4 Web subnet App subnet DB Subnet 10.1.1.0/24 VLAN 101 10.1.2.0/24 10.1.3.0/24
  27. 27. Problem:Manage Configuration of 1000s of virtual appliances (or VRF) Dozens of HW appliances
  28. 28. Problem:Manage Configuration of 1000s of virtual appliances (or VRF) Dozens of HW appliancesSolution:Database-driven state management of appliances Message queues + Retry Logic Idempotent updates, Recreatable virtual appliances
  29. 29. Problem:Manage Configuration of 1000s of virtual appliances (or VRF) Dozens of HW appliancesSolution:Database-driven state management of appliances Message queues + Retry Logic Idempotent updates, Recreatable virtual appliancesProblem:Single-tenant HW appliancesSolution:CloudStack API layers multi-tenancy, provides abstraction No direct access to devices
  30. 30. Problem:Hardware appliances with no APIs CLI only Limited concurrent login sessionsSolution:Recommend appliances with APIsIntegrate with Network Orchestrators
  31. 31. Problem:4094 VLAN Limit
  32. 32. Layer 3 cloud networking Web DB Web VM VM VM Web DB Security Security Group Group Web Web DB VM VM VM … … … Web Web VM VMIngress Rule: Allow VMs in Web Security Group access to VMs in DB Security Group on Port 3306
  33. 33. L3 isolation with distributed firewalls Tenant 10.1.0.2Public Public IP address 1 VM 1Internet 65.37.141.11 65.37.141.24 65.37.141.36 10.1.0.1 Pod 1 L2 Tenant 10.1.0.3 65.37.141.80 Switch 2 VM 1 Tenant 10.1.0.4 1 VM 2 L3 Core Pod 2 L2 Switch 10.1.8.1 … 10.1.16.1 Load Pod 3 L2 Balancer Switch …
  34. 34. L3 isolation with distributed firewalls Tenant 10.1.0.2Public Public IP address 1 VM 1Internet 65.37.141.11 65.37.141.24 65.37.141.36 10.1.0.1 Pod 1 L2 Tenant 10.1.0.3 65.37.141.80 Switch 2 VM 1 Tenant 10.1.0.4 1 VM 2 L3 Core Pod 2 L2 Switch 10.1.8.1 … 10.1.16.1 Load Pod 3 L2 Balancer Switch … Tenant 1 VM 3 10.1.16.47 Tenant 10.1.16.85 1 VM 4
  35. 35. L3 isolation with distributed firewalls Tenant 10.1.0.2Public Public IP address 1 VM 1Internet 65.37.141.11 65.37.141.24 65.37.141.36 10.1.0.1 Pod 1 L2 Tenant 10.1.0.3 65.37.141.80 Switch 2 VM 1 Tenant 10.1.0.4 1 VM 2 L3 Core Pod 2 L2 Switch 10.1.8.1 … Tenant 10.1.16.12 10.1.16.1 2 VM 2 Load Pod 3 L2 Balancer Switch Tenant 2 VM 3 10.1.16.21 … Tenant 1 VM 3 10.1.16.47 Tenant 10.1.16.85 1 VM 4
  36. 36. 1 Firewall per Virtual Machine
  37. 37. A Million Firewalls?VM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM…VM VM … VM VM … VM … … VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VMVM VM VM… … VM VM … …VM VM … VM VMVM VM VM VM VM VM
  38. 38. Problem:Manage the state of 100s of thousands of firewalls
  39. 39. Problem:Manage the state of 100s of thousands of firewallsSolution:Well-known software scaling techniques• Message queues• Consistency tradeoffs• Idempotent configuration & retriesCloudStack uses• special purpose queues• optimized for large security groups• eventual consistency for rule updates
  40. 40. Problem:Firewall (iptables) rules explosion on the host firewall Allow Security Group {Web} on TCP port 3060 -A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.16.31 – j ACCEPT -A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.45.112 – j ACCEPT -A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.189.5 – j ACCEPT … -A FORWARD -m tcp –p tcp –dport 3060 –src 10.21.9.77 – j ACCEPT For large security groups, performance suffers
  41. 41. Problem:Firewall (iptables) rules explosion on the host firewallSolution:Use ipsets: ipset –N web_sg iptreemap ipset –A web_sg 10.1.16.31 ipset –A web_sg 10.1.16.112 ipset –A web_sg 10.1.189.5 … ipset –A web_sg 10.21.9.77 -A FORWARD –p tcp –m tcp –dport 3060 –m set –match-set web_sg src -j ACCEPT
  42. 42. Multi-tier networking with Overlay Internet IPSec or SSL site-to-site VPN Loadbalancer Customer Virtual Router Premises (virtual appliance) MPLS VLANNetwork Services App VM• IPAM Web VM 1• DNS 1• LB [intra] App VM• S-2-S VPN Web VM 2 GRE Key 2724• Static Routes 2• ACLs• NAT, PF• FW [ingress & egress] Web VM GRE Key 353 DB VM 1• BGP 3 Web VM 4 Web subnet App subnet DB Subnet 10.1.1.0/24 GRE Key 101 10.1.2.0/24 10.1.3.0/24
  43. 43. Multi-tier networking with Overlay Internet vswitches Loadbalancer (virtual appliance)Network Services App VM• IPAM 1 Web VM• DNS 1• LB [intra]• S-2-S VPN App VM Web VM 2 GRE Key 2724• Static Routes 2• ACLs• NAT, PF• FW [ingress & egress] Web VM GRE Key 353 DB VM 1• BGP 3 Web VM 4 Web subnet App subnet DB Subnet 10.1.1.0/24 GRE Key 101 10.1.2.0/24 10.1.3.0/24
  44. 44. Check it Out• Apache CloudStack – http://wiki.cloudstack.org – Download it – Use it – Contribute to it• Citrix CloudPlatform – Based on Apache CloudStack – Commercial support

    Be the first to comment

    Login to see the comments

  • acconet

    Oct. 21, 2012
  • thuraminnminn

    Oct. 30, 2012
  • rikuriku0224

    Nov. 7, 2012
  • happycoder

    Nov. 8, 2012
  • gkaul

    Dec. 13, 2012
  • wellhope

    Dec. 27, 2012
  • daniel.checchia

    Jan. 6, 2013
  • se_100

    Jan. 31, 2013
  • nontster

    Feb. 11, 2013
  • gavin_lee

    Apr. 10, 2013
  • chenwaydee

    May. 4, 2013
  • RoySo

    Jun. 19, 2013
  • rubyshen

    Oct. 7, 2013
  • tuninger

    Aug. 21, 2014
  • udomsakc

    Feb. 1, 2015
  • muthukumarvellayappan

    Feb. 3, 2015
  • pban02

    Mar. 20, 2015
  • megabyte875

    Dec. 28, 2015
  • ccasado

    Feb. 11, 2016
  • zeroproductionincidents

    Jan. 23, 2017

Presentation I made at the Carrier Cloud Summit on June 20, 2012 in Paris

Views

Total views

12,085

On Slideshare

0

From embeds

0

Number of embeds

5,606

Actions

Downloads

0

Shares

0

Comments

0

Likes

26

×